Skip to content

feat: build shared cell-group multi-Raft runtime - #160

Draft
forhappy wants to merge 300 commits into
mainfrom
codex/shared-cell-groups
Draft

forhappy wants to merge 300 commits into
mainfrom
codex/shared-cell-groups

Conversation

@forhappy

@forhappy forhappy commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

This draft replaces one-cell Raft ownership with physical cell groups. Independent SQLite cells share one Raft machine per group replica while retaining their histories, receipts, effects and versions. The full MVP and hard cutover remain unfinished; this PR is not merge-ready.

The implementation includes namespace/range routing, replicated enrollment, complete manifests/checkpoints, follower reads, membership send fences, archives/reclamation, fenced fresh-history recovery, indexed effects and bounded caches. Managed startup, directory refresh, capacity, peer/S3 replacement, maintenance and retained restart have scoped coverage. Automatic failed-voter recovery reaches terminal removal with independent S3 replay and retained obsolete data.

Latest change

Public SDK writes sent to a follower previously attempted local SQL and returned NotLeader despite a healthy known leader. The controller now owns one forwarding hop for a queued originating SDK command, retaining its command identity, captured inputs, admission and original deadline. Incoming commands and the lower-level controller command API remain local-only. A leader hint grants no execution authority. Any possible outbound transmission makes an error ambiguous; the owner does not invent a retry identity or automatically retry an uncertain mutation.

Managed peers can explicitly grant/revoke namespace command-origin permission after discovery verifies their actual boots and endpoints. The SDK checks its owned namespace; transport checks current process identity and monotonic permission revisions. Membership grants no write permission. These are trusted host configuration APIs; running-transport permissions must be reapplied from trusted policy after restart.

Command output distinguishes local inspected SQL receipts from authenticated remote results. Common result/observation/token/durability accessors work for either; local_receipt is present only for the local variant. Effects, wire encoding and physical qualification oracles require local proof. Existing physical assertions retain that requirement explicitly; no remote output is converted into a local image/archive proof.

The counter explicitly grants its configured peers permission for its namespace. Its process drill now requires the same original receipt to succeed through all six native/Axum addresses, both before and after retained restart; searching for a leader cannot hide broken forwarding.

Current-source qualification

Based on latest fetched origin/main 981a5417d9359d1f992e2dbf214eec7d221b6777.

  • Eleven complete native cell-group targets: 308 passed, 2 failed, 39 explicit skips. The full campaign is FAILED. The new managed HTTP case and strengthened public SDK/counter process/restart gate pass; eight-turn maintenance replacement settlement and successor readiness after S3 source-service closure fail. Their disks/logs and original bounds are retained.
  • The managed HTTP case covers cold capacity/group birth and replicated enrollment, independent cells, native/Axum follower writes, remote AtLeast/linearizable reads with actual responder identity, conflicting retries, cross-cell token refusal, grant/revocation and stale-process/foreign-namespace refusal, forwarded S3 acknowledgement, joined leader-stop failover, retained restart/old receipts/version floors/new writes, and quorum-loss refusals. Both image forms run.
  • Four exact real-RustFS invocations: managed request run1: PASS (53.37s); managed request run2: PASS (50.61s); archived maintenance: PASS (88.53s); automatic S3 recovery: PASS (45.41s). Backend matrix status: PASSED. Each request case runs both image forms. The failed native campaign remains attached as prerequisite evidence; component passes do not waive it. Signed readiness/conditional probes pass. The pinned one-CPU/two-GiB backend and dedicated VM are stopped with data retained.
  • Format/layout/dependency/docs/native-only Cellar/Python-library/diff checks pass. Strict library Clippy remains red: 220 reports, 202 dead-code reports, with no diagnostic-key additions. Plan tests/validate/preflight remain red for generated-index/Purpose and reviewed source drift. No suppression or silent anchor refresh is applied.

Code manifest: da3514ed9ea2dd614a45f3150f92d2280f75795287b7469a5883f72564b1a87c (2,726 Rust/Python/manifest inputs).
Native FAILED record: 6a4f24b43031c9b9cab3a996edd3b3bf280ecf494a2a231d997912b0e75098e7.
Managed request component record: 8f6e418e605c0d56a635b245ded5cc423d7657cf8f548bdc40109cdb3bdac1a8.
RustFS record: 646ebe709b457b540a97a12e85e43aade6aee1fbbb445a5d37d84a2650330695.

All results use the original local default stack, opt-level 0, assertions/overflow checks, quotas and existing operation/movement bounds. The leader-stop test joins shutdown; it is not a process-crash or partition drill. RustFS is constrained; voters remain host-native embedded Nodes with fixture capacity/isolation. These results do not qualify independent constrained voter containers, permanent launch fencing, sustained load or release.

The deterministic follower-write failure was reproduced before the fix. Earlier unseeded maintenance, ambiguous-write, transfer/readiness, eight-turn replacement, TCP, enrollment and inspection failures remain unwaived. Earlier disk-erasure RustFS recovery retains its recorded source and fence scope; it is not relabeled as this campaign.

Remaining priorities

  1. Complete bounded command leader discovery for ingress without a local replica. Current automatic selection uses local physical group status. Finish normal namespace provisioning and managed SDK/CLI/daemon/native/Axum/admin/example default wiring without weaker fallback.
  2. Deliver the four remaining original public gates: complete managed_group_lifecycle, ingress_cannot_cross_cell_authority, occupancy_one_uses_shared_path and feature_off_fleet_contract_unchanged. Finish retained surviving-voter debt and lifecycle failure/cancellation ownership.
  3. Resolve Workflow/Queue callbacks, WebSockets and mutable-container capability coverage before broad default enablement. Queries still default stale; quorum writes do not imply linearizable reads.
  4. Diagnose retained reliability failures; finish manifest-page collection, 128-cell enrollment, inspection, strict lint, plan reconciliation and SDK CI feature/profile modes.
  5. Qualify independent constrained Linux voters with real RustFS, permanent Docker launch fencing, crashes/partitions/quorum/archive loss and bounded mixed replication/reclamation load with complete ledgers.
  6. Complete CG-09–16: transfers, split/merge, placement, hibernation/wakeup, managed multi-group recovery, load/crash qualification and predecessor removal.

CG-07 remains IN_PROGRESS. RF3 quorum writes with asynchronous S3 archives are the intended default contract, with measured archive lag/RPO; explicit S3 acknowledgement requires complete coverage. Selecting the default backend for new cells remains unfinished. Existing Fleet data must not be reinterpreted as Raft data; no online conversion or legacy/shared switch is supplied.

The requested MR-13/MR-14 progress JSON files remain excluded from the entire PR diff.

See progress, remaining work and qualification.

@forhappy forhappy changed the title docs: select shared cell groups as the Raft replacement docs: add executable cell-group multi-Raft delivery plan Oct 29, 2026
@forhappy forhappy changed the title docs: add executable cell-group multi-Raft delivery plan feat: add cell-group identities and version contracts Oct 29, 2026
@forhappy forhappy changed the title feat: add cell-group identities and version contracts feat: define cell-group multi-Raft delivery and wire contracts Oct 29, 2026
@forhappy forhappy changed the title feat: define cell-group multi-Raft delivery and wire contracts feat: build cell-group contracts and manifest storage Oct 29, 2026
@forhappy forhappy changed the title feat: build cell-group contracts and manifest storage feat: persist cell-group manifests through Raft Engine Oct 29, 2026
@forhappy forhappy changed the title feat: persist cell-group manifests through Raft Engine feat: persist and incrementally prepare cell-group manifests Oct 29, 2026
@forhappy forhappy changed the title feat: persist and incrementally prepare cell-group manifests feat: persist cell-group manifests with retention and reclamation Oct 29, 2026
@forhappy forhappy changed the title feat: persist cell-group manifests with retention and reclamation feat: build cell-group durability and enrollment Oct 29, 2026
@forhappy forhappy changed the title feat: build cell-group durability and enrollment feat: build cell-group durability, enrollment and range directory Oct 29, 2026
@forhappy forhappy changed the title feat: build cell-group durability, enrollment and range directory feat: build cell-group durability, routing and lazy birth metadata Oct 29, 2026
@forhappy forhappy changed the title feat: build cell-group durability, routing and lazy birth metadata feat: build cell-group storage and routing foundations Oct 29, 2026
@forhappy forhappy changed the title feat: build cell-group storage and routing foundations feat: add physical cell-group bootstrap and storage foundations Oct 29, 2026
@forhappy forhappy changed the title feat: add physical cell-group bootstrap and storage foundations feat: deliver cell-group multi-Raft plan and foundations Oct 29, 2026
@forhappy forhappy changed the title feat: deliver cell-group multi-Raft plan and foundations feat: add cell-group routing and durable enrollment foundations Oct 29, 2026
@forhappy forhappy changed the title feat: add cell-group routing and durable enrollment foundations feat: add cell-group consensus and durable state foundations Oct 30, 2026
@forhappy
forhappy marked this pull request as ready for review October 30, 2026 00:50
@forhappy forhappy changed the title feat: add cell-group consensus and durable state foundations feat: add shared cell-group runtime foundations Oct 30, 2026
@forhappy forhappy changed the title feat: add shared cell-group runtime foundations feat: add shared cell-group consensus and SQL foundations Oct 30, 2026
@forhappy forhappy changed the title feat: add shared cell-group consensus and SQL foundations feat: add shared cell-group consensus and bounded SQL workers Oct 30, 2026
@forhappy forhappy changed the title feat: add shared cell-group consensus and bounded SQL workers feat: add shared cell-group consensus and SQL recovery Oct 30, 2026
@forhappy
forhappy marked this pull request as draft November 3, 2026 07:45
BREAKING CHANGE: canonical SDK and fleet consumers use shared physical
cell groups with independent logical cell histories. Retired per-cell
APIs have no compatibility path.

Publish the cutover checkpoint and its MVP acceptance scope. Public
consumer ports, managed S3 activation and complete build, lint and backend
gates remain unfinished; keep PR #160 in draft.

Refs #160
Keep replica attempts in one invocation, original deadline and attempt
bank. Physical SQL and transport retain admission until joined; leader
advice never replaces destination authorization.

Normalize the exact SQL deadline envelope so timer races cannot change
fallback behavior. Drain example Node producers before sampling the final
Raft quiescence report, preserving its existing success predicate.

BREAKING CHANGE: shared query replies carry authenticated leader advice
at marker 1. No predecessor wire reader is retained.

Refs #160
Use explicit namespace/archive provisioning and replicated enrollment.
Keep independent SQLite histories and receipts in one group per voter.
Preserve native/Axum read, token, retry and shutdown behavior.

Add the original public SDK shared-group gate with retained process
proofs. Other managed, ingress/admin and feature-off gates remain open.

BREAKING CHANGE: the counter uses logical histories and cell tokens.
Predecessor per-cell layouts are refused, without migration or fallback.
Replies distinguish cell sequence from physical commit witnesses.

Refs #160
Resolve the host conflict at the shared configuration surface. Adapt
upstream bulk-seed callbacks to logical cell history and retain the
complete metrics inventory assertions.

Preserve upstream replica, snapshot, shutdown and qualification changes.
Their applicability to the shared membership owner requires its own
qualification; a legacy replica fix is not proof for that path.

Refs #160
Freeze one finite Ready boundary before configuration Apply and retain
actual emitted-message admission through physical sends and producers.
Announce and join the committed configuration before route revocation.

Release the physical send fence before idle-owner advisory feedback to
avoid the reproduced wait cycle. Catalog closure remains independent.
Validate rooted declarations and immutable owner descriptors before
pinning storage or consuming a boot. Sync the exact boot before managed
attachment, and rebuild retained routes from current membership.

Own and join bounded directory refresh and install Node stop services
from the actual bound authorization and signer. Preserve advertised
endpoints independently of local listener addresses.
Record source-bound qualification and its failing merge gates. Keep
nonempty managed lifecycle, public consumers, S3 learner activation,
current-source backend qualification and CG-09 through CG-16 in scope.
Break the inventory/capacity/birth cycle with a pinned standby directory
and an exact first-range CAS. Revalidate boot-bound capacity before
initial placement while retaining existing groups across ordinary boots.

Journal import fences with the head and enter the owning execution domain
for birth reservation and materialization. Qualify real RF3 startup, two
independent cells, retained restart and capacity readmission in both image
forms.
Record immutable source-bound checks and the remaining inspection,
performance build and lint failures. Keep full managed lifecycle, public
consumer ports, S3/backend qualification and CG-09 through CG-16 in scope.
Provider I/O can fail after the terminal resource CAS commits. Report an
uncertain outcome so the next turn resolves the same permanent receipt.
Expose the nested startup cause so SDK callers can match boot refusal.

Qualify peer replacement, lost settlement confirmation, leader handoff,
retained joined-member restart and old-boot refusal in both image forms.
Sample complete physical group leadership and authorize retained startup on the exact returned boot. Keep temporary admission refusals distinct from permanent closure.

Release lifecycle transport waiters after rooted preflight and bounded worker enqueue so fresh stop and resume proofs fit the existing peer metadata budget. Preserve uncertainty until sync, actual join, fresh verification and capacity settlement.
Resume exact seeded histories through an explicit fencing authority and a joined activation worker. Keep admission pending until real Raft activation applies, preserving original receipts and catalog intent.

Join public control services before infrastructure teardown on late startup refusal. Cover disk loss, retained restart, S3 minima and held-worker ownership.

Refs #160
Use shared ranges, replicated enrollment, logical receipts and physical witnesses throughout the performance consumers.

Release disposable image references after terminal proof and adopt complete graph pins at startup. Preserve reader, checkpoint and uncertain-publication ownership.

254 component cases and six LTX apply cases pass. The wider-group S3 deadline gate still fails; keep the end-to-end work and draft PR open.
Adopt complete serving and verifier references before bounded collection. Retain control admission and registry ownership through physical completion and cancelled close waiters.

Record RF3 pressure, native regressions and real RustFS recovery. Wider-group enrollment deadlines and strict Cellar lint remain open.
Warm cached destinations bypassed buffered source validation.
Check source bytes before reuse, and reuse completed file/directory
barriers only within their open owner. Keep verifier files charged
under existing ceilings to avoid rebuilding every archive image.

Refs #160
Repeated ancestor syncs consumed rooted initialization time. Reuse the
locked inventory and exact workspace owner, while retaining child and
parent barriers, quarantine, physical ownership and cleanup charges.
Read-cache directories are disposable and need no durable ancestor walk.

Refs #160
Startup dropped verifier resources after registering its initial roster.
Retain them with provisioning and transfer archive owners after actual
attachment. Open existing identities before bootstrap and preserve the
original pools, budgets and joined shutdown.

Qualify dynamic managed RF3 S3 writes, missing-identity refusal/retry and
held archive reads through close in both image forms. Full MVP and
release qualification remain incomplete.
Keep quorum and boot fences through protected S3 handoff and joined cleanup.
Reauthorize source hints after failover without replacing destination ownership.
Seed retirement depended on another learner request. Reconcile exact
journal intents during idle operation and retained restart while keeping
live sources and entered I/O owned through joined shutdown.
Reject quarantined startup before opening mutable engine state.
A retained quarantined endpoint otherwise prevents publishing the
admitted learner route. Preserve durable votes and settlement debt.
Nested recovery futures exhausted default stacks before S3 HTTP polling.
Own catalog work and phase futures without extending locks or deadlines.
Prepare cold replay scratch and preserve typed receiver failures.

Qualify exact-boot isolation, joined cancellation, controller restart,
archive refusal and terminal removal with retained obsolete disks.
Reserve renewal time when a read-only member observation is lost.
Require all fresh RF3 proofs within the original deadline.

Exercise the lost-read boundary and reconcile guarded test handoffs
against actual leadership. Retain earlier failures and qualification
limits in the implementation checkpoint.
Follower ingress attempted local SQL despite a known leader. Retain
one SDK invocation and deadline through authenticated forwarding.
Require explicit namespace grants for actual managed peer processes.

Keep incoming and low-level controller commands local. Distinguish
remote output metadata from local SQL and archive proof. Qualify
native/Axum requests, retained restart and quorum loss against RustFS.
Retain the two failed native lifecycle cases and existing release gaps.

BREAKING CHANGE: low-level command output exposes optional
local_receipt; result and observation also cover remote outputs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant