diff --git a/.github/workflows/javascript-sdk.yml b/.github/workflows/javascript-sdk.yml index 97f9313a3..10799b8f1 100644 --- a/.github/workflows/javascript-sdk.yml +++ b/.github/workflows/javascript-sdk.yml @@ -221,7 +221,8 @@ jobs: - name: Quality and engine boundaries if: matrix.profile == 'adapters' run: | - cargo clippy --locked -p cellar-container -p cellar-guest -p cellar-javascript --features cellar-javascript/embed,cellar-javascript/node --all-targets -- -D warnings + # Keep dead code visible during the cell-group cutover. + cargo clippy --locked -p cellar-container -p cellar-guest -p cellar-javascript --features cellar-javascript/embed,cellar-javascript/node --all-targets -- -D warnings --force-warn dead_code cargo format --check cargo check-layout python3 tools/check_native_dependencies.py diff --git a/.github/workflows/native-sdk.yml b/.github/workflows/native-sdk.yml index a2dca52d6..104fdb116 100644 --- a/.github/workflows/native-sdk.yml +++ b/.github/workflows/native-sdk.yml @@ -336,7 +336,8 @@ jobs: - name: Doctests and Clippy run: | cargo test --workspace --exclude cellar-javascript --exclude cellar-rpc-qualification --all-features --doc --locked - cargo clippy --workspace --exclude cellar-javascript --exclude cellar-rpc-qualification --all-targets --all-features --locked -- -D warnings + # Keep dead code visible during the cell-group cutover. + cargo clippy --workspace --exclude cellar-javascript --exclude cellar-rpc-qualification --all-targets --all-features --locked -- -D warnings --force-warn dead_code - name: Qualify dense filesystem writes and cold recovery env: # Match the native campaign while retaining its correctness checks. diff --git a/.github/workflows/typed-rpc.yml b/.github/workflows/typed-rpc.yml index 13eace9f6..c00863f0b 100644 --- a/.github/workflows/typed-rpc.yml +++ b/.github/workflows/typed-rpc.yml @@ -87,7 +87,8 @@ jobs: cargo test --locked -p cellar-rpc-codegen -- --test-threads=1 python3 tools/python/test_bridge_report.py python3 -m unittest discover -s tools/perf -p test_python_fleet.py - cargo clippy --locked -p cellar-rpc-qualification --all-targets --features full -- -D warnings + # Keep dead code visible during the cell-group cutover. + cargo clippy --locked -p cellar-rpc-qualification --all-targets --features full -- -D warnings --force-warn dead_code cargo test --locked -p cellar-rpc-qualification --features full -- --test-threads=1 - name: Finish build writeback before live qualification run: | diff --git a/.gitignore b/.gitignore index 13268f9d0..6dbaf51ce 100644 --- a/.gitignore +++ b/.gitignore @@ -37,6 +37,8 @@ console-audit.sqlite* # Keep new local multi-Raft benchmark evidence out of commits. /docs/design/multi-raft/evidence/ +# Retain historical local evidence at its source-bound paths. +/plans/multi-raft/evidence/ # Reproducible streaming example builds. /examples/streaming/app.bundle.js* diff --git a/crates/cellar-admin/README.md b/crates/cellar-admin/README.md index 1665b6144..8595f9aa7 100644 --- a/crates/cellar-admin/README.md +++ b/crates/cellar-admin/README.md @@ -99,7 +99,14 @@ For an existing managed fleet, add `raft` to its private dashboard configuration "catalog_prefix": "raft/", "scope": "bucket", "managed": true, - "image_storage": "Materialized" + "image_storage": "Materialized", + "group_directory": { + "max_groups": 1000000, + "max_mutations": 1000000, + "max_cached_pages": 64, + "max_cache_bytes": 4194304, + "operation_timeout_ms": 5000 + } } ] } @@ -145,19 +152,28 @@ Graceful node departure adds independent `fleet_evacuate`, `fleet_remove` and then exclude it, prepare the complete-inventory finalization fence, explicitly request the exact process stop, and consume its signed joined closure in terminal removal. The embedding server must additionally configure -`RaftFleetControl::with_removal_verifier` with its private bounded replay store; -extent namespaces additionally require `ArchiveVerifier::with_extent_store` -with a separate exclusively owned, initially empty `NodeExtentStore`. Never -supply a live node's inventory. Each proof freshly downloads and seals the exact -remote extent closure in this scratch owner. Entered physical work retains the -owner after cancellation; a subsequent attempt reconciles scratch only after that -work exits. Physical inventory quarantine still refuses verification. +`RaftFleetControl::with_removal_verifier` with a `FleetGroupArchiveVerifier`. +Its explicit replay options supply private bounded materialized/extent inventories, +page limits and node-shared archive, checkpoint, tail and scratch pools. Never +supply a live node's inventory. Each proof downloads the complete group checkpoint +and replays its contiguous committed tail. An exact acquire intent is journaled +before S3 mutation, and restart restores its original fixed cut and allocation. +Entered physical work retains page/image ownership and memory admission after +cancellation; physical inventory quarantine still refuses verification. Persistent +holds remain retained until an explicit owner can fence users and join their work; +dropping a controller or timing out does not release remote retention. Standalone existing-root connections cannot perform live proof or dispatch stop. Status and exact committed receipt reconciliation remain metadata operations. See the [departure workflow](../../docs/design/multi-raft/fleet-operations/operator-workflows.md#evacuate-and-permanently-remove-a-node) for CLI syntax, restart behavior and retained reservation reporting. Removal retains volumes and archives. +Operator split and merge preparation uses `RaftFleetControl::with_transition_preparer`. +Configure a private `FleetGroupArchiveVerifier` and a shared `CellGroupRecoveryPool` +on the embedding management server. The request identifies an admitted plan. +The server selects its archive cuts and returns the preparation digest for the +separate seal authorization request. + The `fleet_cancel` grant also authorizes node `cancel-evacuation`. Before finalization this is a metadata operation. Afterwards the server owns an exact process-fence RPC and requires its synced acknowledgement before Ready and slot diff --git a/crates/cellar-admin/lib.rs b/crates/cellar-admin/lib.rs index d02b80f4b..f2043f4cc 100644 --- a/crates/cellar-admin/lib.rs +++ b/crates/cellar-admin/lib.rs @@ -10,9 +10,11 @@ mod cron; mod events; mod nodes; mod raft; +mod raft_first_group; mod raft_lifecycle; mod raft_maintenance; mod raft_nodes; +mod raft_park; pub use audit::{AuditStore, Operation}; pub use auth::{Permission, Principal}; @@ -109,6 +111,82 @@ pub fn router(console: Console) -> Result { "/fleets/{fleet}/raft/nodes", post(raft_nodes::register).layer(DefaultBodyLimit::max(16 * 1024)), ) + .route( + "/fleets/{fleet}/raft/namespaces/{namespace}/first-group/standby", + get(raft_first_group::standby), + ) + .route( + "/fleets/{fleet}/raft/namespaces/{namespace}/first-group/reserve", + post(raft_first_group::reserve), + ) + .route( + "/fleets/{fleet}/raft/namespaces/{namespace}/first-group/publish", + post(raft_first_group::publish), + ) + .route( + "/fleets/{fleet}/raft/namespaces/{namespace}/first-group/archive", + post(raft_first_group::archive), + ) + .route( + "/fleets/{fleet}/raft/namespaces/{namespace}/parks/{operation}/{action}", + post(raft_park::dispatch).layer(DefaultBodyLimit::max(16 * 1024)), + ) + .route( + "/fleets/{fleet}/raft/namespaces/{namespace}/transitions/{operation}", + get(raft::transition), + ) + .route( + "/fleets/{fleet}/raft/namespaces/{namespace}/transitions/{operation}/resume", + post(raft::resume_transition).layer(DefaultBodyLimit::max(16 * 1024)), + ) + .route( + "/fleets/{fleet}/raft/namespaces/{namespace}/transition-basis", + get(raft::transition_basis), + ) + .route( + "/fleets/{fleet}/raft/namespaces/{namespace}/transitions/{operation}/admit", + post(raft::admit_transition).layer(DefaultBodyLimit::max(16 * 1024)), + ) + .route( + "/fleets/{fleet}/raft/namespaces/{namespace}/transitions/{operation}/prepare", + post(raft::prepare_transition).layer(DefaultBodyLimit::max(16 * 1024)), + ) + .route( + "/fleets/{fleet}/raft/namespaces/{namespace}/transitions/{operation}/authorize-seal", + post(raft::authorize_transition_seal).layer(DefaultBodyLimit::max(16 * 1024)), + ) + .route( + "/fleets/{fleet}/raft/namespaces/{namespace}/transitions/{operation}/seal-source", + post(raft::seal_transition_source).layer(DefaultBodyLimit::max(16 * 1024)), + ) + .route( + "/fleets/{fleet}/raft/namespaces/{namespace}/transitions/{operation}/retire-source", + post(raft::retire_transition_source).layer(DefaultBodyLimit::max(16 * 1024)), + ) + .route( + "/fleets/{fleet}/raft/namespaces/{namespace}/transitions/{operation}/capture-source", + post(raft::capture_transition_source).layer(DefaultBodyLimit::max(16 * 1024)), + ) + .route( + "/fleets/{fleet}/raft/namespaces/{namespace}/transitions/{operation}/setup-child", + post(raft::setup_transition_child).layer(DefaultBodyLimit::max(16 * 1024)), + ) + .route( + "/fleets/{fleet}/raft/namespaces/{namespace}/transitions/{operation}/verify-archive", + post(raft::verify_transition_archive).layer(DefaultBodyLimit::max(16 * 1024)), + ) + .route( + "/fleets/{fleet}/raft/namespaces/{namespace}/transitions/{operation}/verify-child-archives", + post(raft::verify_transition_child_archives).layer(DefaultBodyLimit::max(16 * 1024)), + ) + .route( + "/fleets/{fleet}/raft/namespaces/{namespace}/transitions/{operation}/publish", + post(raft::publish_transition).layer(DefaultBodyLimit::max(16 * 1024)), + ) + .route( + "/fleets/{fleet}/raft/namespaces/{namespace}/transitions/{operation}/activate", + post(raft::activate_transition_child).layer(DefaultBodyLimit::max(16 * 1024)), + ) .route("/fleets/{fleet}/raft/nodes/{node}", get(raft_nodes::status)) .route( "/fleets/{fleet}/raft/nodes/{node}/qualify", diff --git a/crates/cellar-admin/raft.rs b/crates/cellar-admin/raft.rs index 83de3776d..f9b24d2ff 100644 --- a/crates/cellar-admin/raft.rs +++ b/crates/cellar-admin/raft.rs @@ -82,6 +82,577 @@ pub(crate) async fn operation( }) } +pub(crate) async fn transition( + State(console): State>, + Extension(principal): Extension, + Path((fleet, namespace, operation)): Path<(String, String, String)>, +) -> Result { + principal.require(&fleet, Permission::MetadataRead)?; + Ok(match backend(&console, &fleet) { + Ok(control) => match control + .transition_status(&namespace, &operation, TIMEOUT) + .await + { + Ok(value) => Json(value).into_response(), + Err(error) => failure(error), + }, + Err(error) => failure(error), + }) +} + +pub(crate) async fn transition_basis( + State(console): State>, + Extension(principal): Extension, + Path((fleet, namespace)): Path<(String, String)>, +) -> Result { + principal.require(&fleet, Permission::MetadataRead)?; + Ok(match backend(&console, &fleet) { + Ok(control) => match control.transition_basis(&namespace, TIMEOUT).await { + Ok(value) => Json(value).into_response(), + Err(error) => failure(error), + }, + Err(error) => failure(error), + }) +} + +pub(crate) async fn admit_transition( + State(console): State>, + Extension(principal): Extension, + Path((fleet, namespace, operation)): Path<(String, String, String)>, + body: Result, JsonRejection>, +) -> Result { + principal.require(&fleet, Permission::FleetTopology)?; + let Ok(Json(request)) = body else { + return Ok(failure(RaftFleetFailure::invalid( + "invalid transition admission", + ))); + }; + if let Err(error) = request.validate(&namespace, &operation) { + return Ok(failure(error)); + } + let fingerprint = match serde_json::to_vec(&request) { + Ok(bytes) => hex_digest(&bytes), + Err(_) => { + return Ok(failure(RaftFleetFailure::invalid( + "invalid transition admission", + ))) + } + }; + let mut audit = match console.audit.begin( + &format!("raft-{}-{}", request.catalog_id, request.request_id), + &fleet, + &principal.id, + &format!("raft/namespaces/{namespace}/transitions/{operation}/admit"), + "raft_transition_admit", + &fingerprint, + ) { + Ok(value) => value, + Err(error) => return Ok(audit_failure(error)), + }; + let result = match backend(&console, &fleet) { + Ok(control) => { + control + .admit_transition(&principal.id, &namespace, &operation, &request, TIMEOUT) + .await + } + Err(error) => Err(error), + }; + finish_with_status(&console, &mut audit, result, "admitted") +} + +pub(crate) async fn prepare_transition( + State(console): State>, + Extension(principal): Extension, + Path((fleet, namespace, operation)): Path<(String, String, String)>, + body: Result, JsonRejection>, +) -> Result { + principal.require(&fleet, Permission::FleetTopology)?; + let Ok(Json(request)) = body else { + return Ok(failure(RaftFleetFailure::invalid( + "invalid transition preparation", + ))); + }; + if let Err(error) = request.validate(&namespace, &operation) { + return Ok(failure(error)); + } + let fingerprint = match serde_json::to_vec(&request) { + Ok(bytes) => hex_digest(&bytes), + Err(_) => { + return Ok(failure(RaftFleetFailure::invalid( + "invalid transition preparation", + ))) + } + }; + let mut audit = match console.audit.begin( + &format!("raft-{}-{}", request.catalog_id, request.request_id), + &fleet, + &principal.id, + &format!("raft/namespaces/{namespace}/transitions/{operation}/prepare"), + "raft_transition_prepare", + &fingerprint, + ) { + Ok(value) => value, + Err(error) => return Ok(audit_failure(error)), + }; + let result = match backend(&console, &fleet) { + Ok(control) => { + control + .prepare_transition(&principal.id, &namespace, &operation, &request, TIMEOUT) + .await + } + Err(error) => Err(error), + }; + finish_with_status(&console, &mut audit, result, "prepared") +} + +pub(crate) async fn verify_transition_archive( + State(console): State>, + Extension(principal): Extension, + Path(selector): Path<(String, String, String)>, + body: Result, JsonRejection>, +) -> Result { + verify_archive(&console, principal, selector, body, false).await +} + +pub(crate) async fn verify_transition_child_archives( + State(console): State>, + Extension(principal): Extension, + Path(selector): Path<(String, String, String)>, + body: Result, JsonRejection>, +) -> Result { + verify_archive(&console, principal, selector, body, true).await +} + +async fn verify_archive( + console: &Console, + principal: Principal, + (fleet, namespace, operation): (String, String, String), + body: Result, JsonRejection>, + child_archives: bool, +) -> Result { + principal.require(&fleet, Permission::FleetTopology)?; + let Ok(Json(request)) = body else { + return Ok(failure(RaftFleetFailure::invalid( + "invalid transfer archive verification", + ))); + }; + if let Err(error) = request.validate(&namespace, &operation) { + return Ok(failure(error)); + } + let fingerprint = match serde_json::to_vec(&request) { + Ok(bytes) => hex_digest(&bytes), + Err(_) => { + return Ok(failure(RaftFleetFailure::invalid( + "invalid transfer archive verification", + ))) + } + }; + let (path, action) = if child_archives { + ( + "verify-child-archives", + "raft_transition_verify_child_archives", + ) + } else { + ("verify-archive", "raft_transition_verify_archive") + }; + let mut audit = match console.audit.begin( + &format!("raft-{}-{}", request.catalog_id, request.request_id), + &fleet, + &principal.id, + &format!("raft/namespaces/{namespace}/transitions/{operation}/{path}"), + action, + &fingerprint, + ) { + Ok(value) => value, + Err(error) => return Ok(audit_failure(error)), + }; + let result = match backend(console, &fleet) { + Ok(control) if child_archives => { + control + .verify_transition_child_archives( + &principal.id, + &namespace, + &operation, + &request, + TIMEOUT, + ) + .await + } + Ok(control) => { + control + .verify_transition_archive(&principal.id, &namespace, &operation, &request, TIMEOUT) + .await + } + Err(error) => Err(error), + }; + finish_with_status(console, &mut audit, result, "archive_verified") +} + +pub(crate) async fn authorize_transition_seal( + State(console): State>, + Extension(principal): Extension, + Path((fleet, namespace, operation)): Path<(String, String, String)>, + body: Result, JsonRejection>, +) -> Result { + principal.require(&fleet, Permission::FleetTopology)?; + let Ok(Json(request)) = body else { + return Ok(failure(RaftFleetFailure::invalid( + "invalid transition seal authorization", + ))); + }; + if let Err(error) = request.validate(&namespace, &operation) { + return Ok(failure(error)); + } + let fingerprint = match serde_json::to_vec(&request) { + Ok(bytes) => hex_digest(&bytes), + Err(_) => { + return Ok(failure(RaftFleetFailure::invalid( + "invalid transition seal authorization", + ))) + } + }; + let mut audit = match console.audit.begin( + &format!("raft-{}-{}", request.catalog_id, request.request_id), + &fleet, + &principal.id, + &format!("raft/namespaces/{namespace}/transitions/{operation}/authorize-seal"), + "raft_transition_authorize_seal", + &fingerprint, + ) { + Ok(value) => value, + Err(error) => return Ok(audit_failure(error)), + }; + let result = match backend(&console, &fleet) { + Ok(control) => { + control + .authorize_transition_seal(&principal.id, &namespace, &operation, &request, TIMEOUT) + .await + } + Err(error) => Err(error), + }; + finish_with_status(&console, &mut audit, result, "seal_authorized") +} + +pub(crate) async fn seal_transition_source( + State(console): State>, + Extension(principal): Extension, + Path((fleet, namespace, operation)): Path<(String, String, String)>, + body: Result, JsonRejection>, +) -> Result { + principal.require(&fleet, Permission::FleetTopology)?; + let Ok(Json(request)) = body else { + return Ok(failure(RaftFleetFailure::invalid( + "invalid transition source seal", + ))); + }; + if let Err(error) = request.validate(&namespace, &operation) { + return Ok(failure(error)); + } + let fingerprint = match serde_json::to_vec(&request) { + Ok(bytes) => hex_digest(&bytes), + Err(_) => { + return Ok(failure(RaftFleetFailure::invalid( + "invalid transition source seal", + ))) + } + }; + let mut audit = match console.audit.begin( + &format!("raft-{}-{}", request.catalog_id, request.request_id), + &fleet, + &principal.id, + &format!("raft/namespaces/{namespace}/transitions/{operation}/seal-source"), + "raft_transition_seal_source", + &fingerprint, + ) { + Ok(value) => value, + Err(error) => return Ok(audit_failure(error)), + }; + let result = match backend(&console, &fleet) { + Ok(control) => { + control + .seal_transition_source(&principal.id, &namespace, &operation, &request, TIMEOUT) + .await + } + Err(error) => Err(error), + }; + finish(&console, &mut audit, result) +} + +pub(crate) async fn retire_transition_source( + State(console): State>, + Extension(principal): Extension, + Path((fleet, namespace, operation)): Path<(String, String, String)>, + body: Result, JsonRejection>, +) -> Result { + principal.require(&fleet, Permission::FleetTopology)?; + let Ok(Json(request)) = body else { + return Ok(failure(RaftFleetFailure::invalid( + "invalid transition source retirement", + ))); + }; + if let Err(error) = request.validate(&namespace, &operation) { + return Ok(failure(error)); + } + let fingerprint = match serde_json::to_vec(&request) { + Ok(bytes) => hex_digest(&bytes), + Err(_) => { + return Ok(failure(RaftFleetFailure::invalid( + "invalid transition source retirement", + ))) + } + }; + let mut audit = match console.audit.begin( + &format!("raft-{}-{}", request.catalog_id, request.request_id), + &fleet, + &principal.id, + &format!("raft/namespaces/{namespace}/transitions/{operation}/retire-source"), + "raft_transition_retire_source", + &fingerprint, + ) { + Ok(value) => value, + Err(error) => return Ok(audit_failure(error)), + }; + let result = match backend(&console, &fleet) { + Ok(control) => { + control + .retire_transition_source(&principal.id, &namespace, &operation, &request, TIMEOUT) + .await + } + Err(error) => Err(error), + }; + finish(&console, &mut audit, result) +} + +pub(crate) async fn capture_transition_source( + State(console): State>, + Extension(principal): Extension, + Path((fleet, namespace, operation)): Path<(String, String, String)>, + body: Result, JsonRejection>, +) -> Result { + principal.require(&fleet, Permission::FleetTopology)?; + let Ok(Json(request)) = body else { + return Ok(failure(RaftFleetFailure::invalid( + "invalid transition source capture", + ))); + }; + if let Err(error) = request.validate(&namespace, &operation) { + return Ok(failure(error)); + } + let fingerprint = match serde_json::to_vec(&request) { + Ok(bytes) => hex_digest(&bytes), + Err(_) => { + return Ok(failure(RaftFleetFailure::invalid( + "invalid transition source capture", + ))) + } + }; + let mut audit = match console.audit.begin( + &format!("raft-{}-{}", request.catalog_id, request.request_id), + &fleet, + &principal.id, + &format!("raft/namespaces/{namespace}/transitions/{operation}/capture-source"), + "raft_transition_capture_source", + &fingerprint, + ) { + Ok(value) => value, + Err(error) => return Ok(audit_failure(error)), + }; + let result = match backend(&console, &fleet) { + Ok(control) => { + control + .capture_transition_source(&principal.id, &namespace, &operation, &request, TIMEOUT) + .await + } + Err(error) => Err(error), + }; + finish_with_status(&console, &mut audit, result, "source_captured") +} + +pub(crate) async fn setup_transition_child( + State(console): State>, + Extension(principal): Extension, + Path((fleet, namespace, operation)): Path<(String, String, String)>, + body: Result, JsonRejection>, +) -> Result { + principal.require(&fleet, Permission::FleetTopology)?; + let Ok(Json(request)) = body else { + return Ok(failure(RaftFleetFailure::invalid( + "invalid transition child setup", + ))); + }; + if let Err(error) = request.validate(&namespace, &operation) { + return Ok(failure(error)); + } + let fingerprint = match serde_json::to_vec(&request) { + Ok(bytes) => hex_digest(&bytes), + Err(_) => { + return Ok(failure(RaftFleetFailure::invalid( + "invalid transition child setup", + ))) + } + }; + let mut audit = match console.audit.begin( + &format!("raft-{}-{}", request.catalog_id, request.request_id), + &fleet, + &principal.id, + &format!("raft/namespaces/{namespace}/transitions/{operation}/setup-child"), + "raft_transition_setup_child", + &fingerprint, + ) { + Ok(value) => value, + Err(error) => return Ok(audit_failure(error)), + }; + let result = match backend(&console, &fleet) { + Ok(control) => { + control + .setup_transition_child(&principal.id, &namespace, &operation, &request, TIMEOUT) + .await + } + Err(error) => Err(error), + }; + finish(&console, &mut audit, result) +} + +pub(crate) async fn publish_transition( + State(console): State>, + Extension(principal): Extension, + Path((fleet, namespace, operation)): Path<(String, String, String)>, + body: Result, JsonRejection>, +) -> Result { + principal.require(&fleet, Permission::FleetTopology)?; + let Ok(Json(request)) = body else { + return Ok(failure(RaftFleetFailure::invalid( + "invalid transition publication", + ))); + }; + if let Err(error) = request.validate(&namespace, &operation) { + return Ok(failure(error)); + } + let fingerprint = match serde_json::to_vec(&request) { + Ok(bytes) => hex_digest(&bytes), + Err(_) => { + return Ok(failure(RaftFleetFailure::invalid( + "invalid transition publication", + ))) + } + }; + let mut audit = match console.audit.begin( + &format!("raft-{}-{}", request.catalog_id, request.request_id), + &fleet, + &principal.id, + &format!("raft/namespaces/{namespace}/transitions/{operation}/publish"), + "raft_transition_publish", + &fingerprint, + ) { + Ok(value) => value, + Err(error) => return Ok(audit_failure(error)), + }; + let result = match backend(&console, &fleet) { + Ok(control) => { + control + .publish_transition(&principal.id, &namespace, &operation, &request, TIMEOUT) + .await + } + Err(error) => Err(error), + }; + finish_with_status(&console, &mut audit, result, "published") +} + +pub(crate) async fn activate_transition_child( + State(console): State>, + Extension(principal): Extension, + Path((fleet, namespace, operation)): Path<(String, String, String)>, + body: Result, JsonRejection>, +) -> Result { + principal.require(&fleet, Permission::FleetTopology)?; + let Ok(Json(request)) = body else { + return Ok(failure(RaftFleetFailure::invalid( + "invalid child activation", + ))); + }; + if let Err(error) = request.validate(&namespace, &operation) { + return Ok(failure(error)); + } + let fingerprint = match serde_json::to_vec(&request) { + Ok(bytes) => hex_digest(&bytes), + Err(_) => { + return Ok(failure(RaftFleetFailure::invalid( + "invalid child activation", + ))) + } + }; + let mut audit = match console.audit.begin( + &format!("raft-{}-{}", request.catalog_id, request.request_id), + &fleet, + &principal.id, + &format!("raft/namespaces/{namespace}/transitions/{operation}/activate"), + "raft_transition_activate", + &fingerprint, + ) { + Ok(value) => value, + Err(error) => return Ok(audit_failure(error)), + }; + let result = match backend(&console, &fleet) { + Ok(control) => { + control + .activate_transition_child(&principal.id, &namespace, &operation, &request, TIMEOUT) + .await + } + Err(error) => Err(error), + }; + let status = if result.as_ref().is_ok_and(|value| value.applied.is_some()) { + "applied" + } else { + "accepted" + }; + finish_with_status(&console, &mut audit, result, status) +} + +pub(crate) async fn resume_transition( + State(console): State>, + Extension(principal): Extension, + Path((fleet, namespace, operation)): Path<(String, String, String)>, + body: Result, JsonRejection>, +) -> Result { + principal.require(&fleet, Permission::FleetTopology)?; + let Ok(Json(request)) = body else { + return Ok(failure(RaftFleetFailure::invalid( + "invalid transition resume", + ))); + }; + if let Err(error) = request.validate(&namespace, &operation) { + return Ok(failure(error)); + } + let fingerprint = match serde_json::to_vec(&request) { + Ok(bytes) => hex_digest(&bytes), + Err(_) => { + return Ok(failure(RaftFleetFailure::invalid( + "invalid transition resume", + ))) + } + }; + let mut audit = match console.audit.begin( + &format!("raft-{}-{}", request.catalog_id, request.request_id), + &fleet, + &principal.id, + &format!("raft/namespaces/{namespace}/transitions/{operation}"), + "raft_transition_resume", + &fingerprint, + ) { + Ok(value) => value, + Err(error) => return Ok(audit_failure(error)), + }; + let result = match backend(&console, &fleet) { + Ok(control) => { + control + .resume_transition(&principal.id, &namespace, &operation, &request, TIMEOUT) + .await + } + Err(error) => Err(error), + }; + finish_with_status(&console, &mut audit, result, "completed") +} + #[derive(Deserialize)] #[serde(deny_unknown_fields)] pub(crate) struct Page { diff --git a/crates/cellar-admin/raft_first_group.rs b/crates/cellar-admin/raft_first_group.rs new file mode 100644 index 000000000..61bdb709c --- /dev/null +++ b/crates/cellar-admin/raft_first_group.rs @@ -0,0 +1,177 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Bind first-group operations to the authenticated fleet and namespace. +use crate::{ + api::ApiError, + raft::{audit_failure, backend, failure, finish, hex_digest, TIMEOUT}, + Console, Permission, Principal, +}; +use axum::{ + extract::{rejection::JsonRejection, Path, State}, + response::{IntoResponse, Response}, + Extension, Json, +}; +use cellar::sdk::fleet_operations::*; +use serde::Serialize; +use std::sync::Arc; + +pub(crate) async fn standby( + State(console): State>, + Extension(principal): Extension, + Path((fleet, namespace)): Path<(String, String)>, +) -> Result { + principal.require(&fleet, Permission::MetadataRead)?; + Ok(match backend(&console, &fleet) { + Ok(control) => match control.first_group_standby(&namespace, TIMEOUT).await { + Ok(value) => Json(value).into_response(), + Err(error) => failure(error), + }, + Err(error) => failure(error), + }) +} + +pub(crate) async fn reserve( + State(console): State>, + Extension(principal): Extension, + Path((fleet, namespace)): Path<(String, String)>, + body: Result, JsonRejection>, +) -> Result { + principal.require(&fleet, Permission::FleetTopology)?; + let Ok(Json(plan)) = body else { + return Ok(failure(RaftFleetFailure::invalid( + "invalid first group plan", + ))); + }; + submit( + &console, + &principal, + &fleet, + &namespace, + Mutation::Reserve(plan), + ) + .await +} + +pub(crate) async fn publish( + State(console): State>, + Extension(principal): Extension, + Path((fleet, namespace)): Path<(String, String)>, + body: Result, JsonRejection>, +) -> Result { + principal.require(&fleet, Permission::FleetTopology)?; + let Ok(Json(request)) = body else { + return Ok(failure(RaftFleetFailure::invalid( + "invalid first group confirmation", + ))); + }; + submit( + &console, + &principal, + &fleet, + &namespace, + Mutation::Publish(request), + ) + .await +} + +pub(crate) async fn archive( + State(console): State>, + Extension(principal): Extension, + Path((fleet, namespace)): Path<(String, String)>, + body: Result, JsonRejection>, +) -> Result { + principal.require(&fleet, Permission::FleetTopology)?; + let Ok(Json(request)) = body else { + return Ok(failure(RaftFleetFailure::invalid( + "invalid first group archive request", + ))); + }; + submit( + &console, + &principal, + &fleet, + &namespace, + Mutation::Archive(request), + ) + .await +} + +#[derive(Serialize)] +#[serde(rename_all = "snake_case")] +enum Mutation { + Reserve(RaftFleetFirstGroupPlan), + Publish(RaftFleetFirstGroupPublish), + Archive(RaftFleetFirstGroupMutation), +} + +async fn submit( + console: &Console, + principal: &Principal, + fleet: &str, + namespace: &str, + mutation: Mutation, +) -> Result { + let (plan, request_id, action, valid) = match &mutation { + Mutation::Reserve(plan) => ( + plan, + &plan.request_id, + "raft_first_group_reserve", + plan.validate(namespace), + ), + Mutation::Publish(request) => ( + &request.plan, + &request.request_id, + "raft_first_group_publish", + request.validate(namespace), + ), + Mutation::Archive(request) => ( + &request.plan, + &request.request_id, + "raft_first_group_archive", + request.validate(namespace), + ), + }; + if let Err(error) = valid { + return Ok(failure(error)); + } + let fingerprint = match serde_json::to_vec(&mutation) { + Ok(bytes) => hex_digest(&bytes), + Err(_) => { + return Ok(failure(RaftFleetFailure::invalid( + "invalid first group request", + ))) + } + }; + let mut audit = match console.audit.begin( + &format!("raft-{}-{request_id}", plan.catalog_id), + fleet, + &principal.id, + &format!("raft/namespaces/{namespace}/first-group"), + action, + &fingerprint, + ) { + Ok(value) => value, + Err(error) => return Ok(audit_failure(error)), + }; + let result = match backend(console, fleet) { + Ok(control) => match &mutation { + Mutation::Reserve(plan) => { + control + .reserve_first_group(&principal.id, namespace, plan, TIMEOUT) + .await + } + Mutation::Publish(request) => { + control + .publish_first_group(&principal.id, namespace, request, TIMEOUT) + .await + } + Mutation::Archive(request) => { + control + .initialize_first_group_archive(&principal.id, namespace, request, TIMEOUT) + .await + } + }, + Err(error) => Err(error), + }; + finish(console, &mut audit, result) +} diff --git a/crates/cellar-admin/raft_park.rs b/crates/cellar-admin/raft_park.rs new file mode 100644 index 000000000..39a091b5a --- /dev/null +++ b/crates/cellar-admin/raft_park.rs @@ -0,0 +1,125 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use crate::{api::ApiError, raft::*, Console, Permission, Principal}; +use axum::{ + extract::{rejection::JsonRejection, Path, State}, + response::{IntoResponse, Response}, + Extension, Json, +}; +use cellar::sdk::fleet_operations::*; +use serde::{Deserialize, Serialize}; +use std::sync::Arc; + +#[derive(Deserialize)] +#[serde(rename_all = "snake_case")] +pub(crate) enum Action { + Park, + Status, + Wake, +} + +#[derive(Serialize)] +#[serde(untagged)] +enum Outcome { + Park(RaftFleetParkStatus), + Wake(RaftFleetWakeStatus), +} + +pub(crate) async fn dispatch( + State(console): State>, + Extension(principal): Extension, + Path((fleet, namespace, operation, action)): Path<(String, String, String, Action)>, + body: Result, JsonRejection>, +) -> Result { + let permission = match action { + Action::Status => Permission::MetadataRead, + Action::Park | Action::Wake => Permission::FleetTopology, + }; + principal.require(&fleet, permission)?; + let Ok(Json(request)) = body else { + return Ok(failure(RaftFleetFailure::invalid( + "invalid group park request", + ))); + }; + if let Err(error) = request.validate(&namespace, &operation) { + return Ok(failure(error)); + } + let control = match backend(&console, &fleet) { + Ok(control) => control, + Err(error) => return Ok(failure(error)), + }; + let (audit_id, verb) = match action { + Action::Status => { + return Ok( + match control + .park_group_status(&principal.id, &request, TIMEOUT) + .await + { + Ok(value) => Json(value).into_response(), + Err(error) => failure(error), + }, + ); + } + Action::Park => ( + format!("raft-{}-{}", request.catalog_id, request.request_id), + "park", + ), + Action::Wake => ( + format!( + "raft-wake-{}-{}", + request.catalog_id, request.settlement_request_id + ), + "wake", + ), + }; + let fingerprint = match serde_json::to_vec(&request) { + Ok(bytes) => hex_digest(&bytes), + Err(_) => { + return Ok(failure(RaftFleetFailure::invalid( + "invalid group park request", + ))) + } + }; + let owner = console.clone(); + let actor = principal.id.clone(); + let begin = console.domain.blocking(move || { + owner.audit.begin( + &audit_id, + &fleet, + &actor, + &format!("raft/namespaces/{namespace}/parks/{operation}/{verb}"), + &format!("raft_group_{verb}"), + &fingerprint, + ) + }); + let mut audit = match begin { + Ok(task) => match task.await { + Ok(Ok(audit)) => audit, + Ok(Err(error)) => return Ok(audit_failure(error)), + Err(error) => return Ok(audit_failure(error.into())), + }, + Err(error) => return Ok(audit_failure(error.into())), + }; + let result = match action { + Action::Park => control + .park_group(&principal.id, &request, TIMEOUT) + .await + .map(Outcome::Park), + Action::Wake => control + .wake_group(&principal.id, &request, TIMEOUT) + .await + .map(Outcome::Wake), + Action::Status => unreachable!("status returns before audit"), + }; + let owner = console.clone(); + let completed = console + .domain + .blocking(move || finish(&owner, &mut audit, result)); + match completed { + Ok(task) => match task.await { + Ok(result) => result, + Err(error) => Ok(audit_failure(error.into())), + }, + Err(error) => Ok(audit_failure(error.into())), + } +} diff --git a/crates/cellar-logic/cell_group.rs b/crates/cellar-logic/cell_group.rs new file mode 100644 index 000000000..fe19a4b4b --- /dev/null +++ b/crates/cellar-logic/cell_group.rs @@ -0,0 +1,351 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Logical cell identity is independent of the group that currently replicates it. +//! +//! These are pure contracts. Constructing a route, witness or version does not +//! prove ownership, persistence or quorum; adapters must supply those proofs. + +use crate::consensus::{ + ClusterId, GroupId, IncarnationId, LogPosition, NamespaceId, ProposalId, Refusal, +}; +use std::num::NonZeroU64; + +pub const MAX_CELL_ID_BYTES: usize = 1024; +pub const CELL_ROUTE_DOMAIN: &[u8] = b"cellar-cell-route-1\0"; + +/// Exact UTF-8 bytes, without Unicode or locale normalization. +#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub struct CellId(String); + +impl CellId { + pub fn new(value: impl AsRef) -> Result { + let value = value.as_ref(); + Self::validate(value)?; + Ok(Self(value.to_owned())) + } + + pub fn validate(value: &str) -> Result<(), Refusal> { + if value.is_empty() + || value.len() > MAX_CELL_ID_BYTES + || value.chars().any(char::is_control) + { + return Err(Refusal::InvalidIdentity); + } + Ok(()) + } + + pub fn as_str(&self) -> &str { + &self.0 + } + + /// Canonical SHA-256 input; the adapter performs hashing outside the core. + pub fn routing_preimage(&self, namespace: NamespaceId) -> Vec { + let mut bytes = Vec::with_capacity(CELL_ROUTE_DOMAIN.len() + 20 + self.0.len()); + bytes.extend_from_slice(CELL_ROUTE_DOMAIN); + bytes.extend_from_slice(&namespace.bytes()); + bytes.extend_from_slice(&(self.0.len() as u32).to_be_bytes()); + bytes.extend_from_slice(self.0.as_bytes()); + bytes + } +} + +/// Preserved across elections, placement changes and normal split/merge. +/// Disaster recovery must allocate a new cell incarnation. +#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub struct CellHistory { + pub cluster: ClusterId, + pub namespace: NamespaceId, + pub cell: CellId, + pub incarnation: IncarnationId, +} + +/// Projects the actual logical binding from the temporary single-cell host. +/// No physical group identity becomes part of the logical history. This does +/// not migrate stored state or confer execution authority. +impl TryFrom<&crate::consensus::GroupScope> for CellHistory { + type Error = Refusal; + + fn try_from(scope: &crate::consensus::GroupScope) -> Result { + scope.validate()?; + Ok(Self { + cluster: scope.cluster, + namespace: scope.namespace, + cell: CellId::new(&scope.cell)?, + incarnation: scope.incarnation, + }) + } +} + +/// A physical Raft history; deliberately contains no cell identifier. +#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub struct GroupHistory { + pub cluster: ClusterId, + pub namespace: NamespaceId, + pub group: GroupId, + pub incarnation: IncarnationId, +} + +impl GroupHistory { + pub fn same_namespace(&self, cell: &CellHistory) -> bool { + self.cluster == cell.cluster && self.namespace == cell.namespace + } +} + +/// Projects the physical history from the temporary single-cell adapter. +/// This does not validate or authorize the adapter's cell binding. +impl From<&crate::consensus::GroupScope> for GroupHistory { + fn from(scope: &crate::consensus::GroupScope) -> Self { + Self { + cluster: scope.cluster, + namespace: scope.namespace, + group: scope.group, + incarnation: scope.incarnation, + } + } +} + +impl From<&GroupHistory> for GroupHistory { + fn from(history: &GroupHistory) -> Self { + *history + } +} + +macro_rules! nonzero_counter { + ($($name:ident),+ $(,)?) => {$ ( + #[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)] + pub struct $name(NonZeroU64); + + impl $name { + pub fn new(value: u64) -> Result { + NonZeroU64::new(value).map(Self).ok_or(Refusal::InvalidIdentity) + } + + pub const fn get(self) -> u64 { + self.0.get() + } + + /// Overflow is a refusal, never a wraparound or saturated version. + pub fn checked_next(self) -> Result { + Self::new(self.get().checked_add(1).ok_or(Refusal::InvalidPosition)?) + } + } + )+}; +} + +nonzero_counter!( + MutationSequence, + RouteEpoch, + MembershipRevision, + ParkGeneration +); + +/// Non-reusable identity of a namespace topology operation. +#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub struct TransitionId([u8; 16]); + +impl TransitionId { + pub fn new(bytes: [u8; 16]) -> Result { + if bytes == [0; 16] { + return Err(Refusal::InvalidIdentity); + } + Ok(Self(bytes)) + } + + pub const fn bytes(self) -> [u8; 16] { + self.0 + } +} + +/// Ordered hash plus the full ID. Digest collisions never alias distinct cells. +#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub struct CellKey { + digest: [u8; 32], + cell: CellId, +} + +impl CellKey { + /// The adapter must verify this digest against the namespace and canonical + /// routing preimage before using a key received over the wire. + pub fn from_digest(digest: [u8; 32], cell: CellId) -> Self { + Self { digest, cell } + } + + pub fn digest(&self) -> [u8; 32] { + self.digest + } + + pub fn cell(&self) -> &CellId { + &self.cell + } +} + +/// The declaration order is the canonical ordering, including infinity bounds. +#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub enum CellBound { + NegativeInfinity, + Key(CellKey), + PositiveInfinity, +} + +/// A nonempty half-open interval of whole cell routing keys. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct CellRange { + lower: CellBound, + upper: CellBound, +} + +impl CellRange { + pub fn new(lower: CellBound, upper: CellBound) -> Result { + if lower >= upper { + return Err(Refusal::InvalidIdentity); + } + Ok(Self { lower, upper }) + } + + pub fn all() -> Self { + Self { + lower: CellBound::NegativeInfinity, + upper: CellBound::PositiveInfinity, + } + } + + pub fn lower(&self) -> &CellBound { + &self.lower + } + + pub fn upper(&self) -> &CellBound { + &self.upper + } + + pub fn contains(&self, key: &CellKey) -> bool { + let lower = match &self.lower { + CellBound::NegativeInfinity => true, + CellBound::Key(bound) => key >= bound, + CellBound::PositiveInfinity => false, + }; + let upper = match &self.upper { + CellBound::PositiveInfinity => true, + CellBound::Key(bound) => key < bound, + CellBound::NegativeInfinity => false, + }; + lower && upper + } + + pub fn split(&self, at: CellKey) -> Result<(Self, Self), Refusal> { + let at = CellBound::Key(at); + Ok(( + Self::new(self.lower.clone(), at.clone())?, + Self::new(at, self.upper.clone())?, + )) + } + + /// Merge only adjacent intervals in order; overlap and gaps both refuse. + pub fn merge(&self, right: &Self) -> Result { + if self.upper != right.lower { + return Err(Refusal::NonContiguous); + } + Self::new(self.lower.clone(), right.upper.clone()) + } +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct GroupRoute { + pub history: GroupHistory, + pub range: CellRange, + pub epoch: RouteEpoch, + pub birth_digest: [u8; 32], + pub schema_digest: [u8; 32], +} + +impl GroupRoute { + /// The supplied key must already have its namespace hash verified. + pub fn check_cell(&self, cell: &CellHistory, key: &CellKey) -> Result<(), Refusal> { + if !self.history.same_namespace(cell) + || key.cell() != &cell.cell + || !self.range.contains(key) + { + return Err(Refusal::WrongHistory); + } + Ok(()) + } +} + +/// An LTX position with a complete, domain-specific image commitment. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct CellImage { + pub epoch: u64, + pub txid: u64, + pub checksum: u64, + pub commitment: CellImageCommitment, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum CellImageCommitment { + Materialized([u8; 32]), + Extents { root: [u8; 32], manifest_bytes: u32 }, +} + +impl CellImage { + pub fn validate(self) -> Result<(), Refusal> { + if self.epoch == 0 + || matches!( + self.commitment, + CellImageCommitment::Extents { + manifest_bytes: 0, + .. + } + ) + { + return Err(Refusal::InvalidPosition); + } + Ok(()) + } +} + +/// Authenticated cell progress, independent of the current group's log index. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct CellVersion { + pub history: CellHistory, + pub sequence: MutationSequence, + pub image: CellImage, +} + +impl CellVersion { + /// Call only after authenticating the token and validating its schema and + /// installed lineage. A witness from an old group is not current authority. + pub fn covers(&self, required: &Self) -> Result { + self.image.validate()?; + required.image.validate()?; + if self.history != required.history + || (self.sequence == required.sequence && self.image != required.image) + { + return Err(Refusal::WrongHistory); + } + Ok(self.sequence >= required.sequence) + } +} + +/// Locates the original decision, including after transfer to a new group. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct CommitWitness { + pub group: GroupHistory, + pub position: LogPosition, + pub proposal: ProposalId, +} + +impl CommitWitness { + pub fn validate(&self, cell: &CellHistory) -> Result<(), Refusal> { + self.position.validate()?; + if !self.group.same_namespace(cell) { + return Err(Refusal::WrongHistory); + } + if self.position.index.get() == 0 { + return Err(Refusal::InvalidPosition); + } + Ok(()) + } +} + +#[cfg(test)] +#[path = "cell_group_tests.rs"] +mod tests; diff --git a/crates/cellar-logic/cell_group_placement.rs b/crates/cellar-logic/cell_group_placement.rs new file mode 100644 index 000000000..355e7776e --- /dev/null +++ b/crates/cellar-logic/cell_group_placement.rs @@ -0,0 +1,563 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Bounded recommendations for partition sizing. Every result is advisory. +//! Admission belongs to the current directory, Fleet, and Raft owners. + +use crate::{ + cell_group::{CellBound, CellId, CellKey, GroupRoute, MembershipRevision}, + consensus::{Durability, Term}, +}; + +const BASIS_POINTS: u16 = 10_000; +const MAX_SAMPLE_AGE_MS: u64 = 3_600_000; + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct PartitionLoad { + pub cells: u32, + pub logical_bytes: u64, + pub writes_per_second: u64, +} + +impl PartitionLoad { + fn within(self, ceiling: Self) -> bool { + self.cells <= ceiling.cells + && self.logical_bytes <= ceiling.logical_bytes + && self.writes_per_second <= ceiling.writes_per_second + } + + fn sum(self, other: Self) -> Option { + Some(Self { + cells: self.cells.checked_add(other.cells)?, + logical_bytes: self.logical_bytes.checked_add(other.logical_bytes)?, + writes_per_second: self + .writes_per_second + .checked_add(other.writes_per_second)?, + }) + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct PartitionLimits { + /// A split requires a load dimension above this ceiling or sustained + /// CPU/apply pressure. Merge ceilings must be strictly lower. + pub split_ceiling: PartitionLoad, + pub merge_ceiling: PartitionLoad, + pub split_cpu_bps: u16, + pub merge_cpu_bps: u16, + pub max_queue_age_ms: u64, + pub max_archive_lag_entries: u64, + pub max_failure_domain_load_bps: u16, + pub hot_cell_bytes: u64, + pub hot_cell_writes_per_second: u64, + pub split_samples: u8, + pub merge_samples: u8, + pub sample_interval_ms: u64, + pub max_sample_age_ms: u64, + pub cooldown_ms: u64, + pub max_active_operations: u16, +} + +/// A validated, explicit policy. Qualification determines deployable limits. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct PartitionPolicy(PartitionLimits); + +impl PartitionPolicy { + pub fn new(limits: PartitionLimits) -> Option { + let split = limits.split_ceiling; + let merge = limits.merge_ceiling; + (merge.cells < split.cells + && merge.logical_bytes < split.logical_bytes + && merge.writes_per_second < split.writes_per_second + && limits.merge_cpu_bps < limits.split_cpu_bps + && limits.split_cpu_bps <= BASIS_POINTS + && (1..=BASIS_POINTS).contains(&limits.max_failure_domain_load_bps) + && limits.max_queue_age_ms > 0 + && limits.hot_cell_bytes > 0 + && limits.hot_cell_writes_per_second > 0 + && limits.split_samples >= 2 + && limits.merge_samples >= 2 + && (1..=MAX_SAMPLE_AGE_MS).contains(&limits.max_sample_age_ms) + && (1..=limits.max_sample_age_ms).contains(&limits.sample_interval_ms) + && limits.cooldown_ms > 0 + && limits.max_active_operations > 0) + .then_some(Self(limits)) + } +} + +/// Exact observations an adapter must compare again before admitting work. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct PartitionBasis { + pub route: GroupRoute, + pub directory_digest: [u8; 32], + pub membership: MembershipRevision, + pub term: Term, + pub durability: Durability, + /// Fingerprint of placement pool, separation and failure-domain policy. + pub placement_class: [u8; 32], +} + +/// A bounded hot-cell hint. The adapter supplies the next occupied key in +/// this range and whether there are occupied keys before the hot cell. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct HotCell { + pub key: CellKey, + pub next: Option, + pub has_predecessor: bool, + pub logical_bytes: u64, + pub writes_per_second: u64, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct PartitionSample { + pub basis: PartitionBasis, + /// Receiver-local monotonic time. + pub sampled_at_ms: u64, + /// Receiver-local lower bound from observed topology progress. The owner + /// supplies a conservative bound when it begins tracking a new route. + pub topology_changed_at_ms: u64, + pub load: PartitionLoad, + pub applies_per_second: u64, + pub queue_age_ms: u64, + pub archive_lag_entries: Option, + pub leader_cpu_bps: u16, + pub failure_domain_load_bps: u16, + pub active_and_stable: bool, + /// A verified occupied key with a nonempty occupied partition on each side. + pub balanced_pivot: Option, + pub hot_cell: Option, + pub hot_writer: Option, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum SplitReason { + GroupLoad, + LeaderCpu, + ApplyBacklog, + IsolateHotCell, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum WaitReason { + NoFreshSample, + Incompatible, + InTransition, + ArchiveLag, + DestinationPressure, + QueuePressure, + Cooldown, + OperationLimit, + Hysteresis, + NoWholeCellPivot, + DedicatedCell, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum PartitionDecision<'a> { + Wait(WaitReason), + Split { + basis: &'a PartitionBasis, + pivot: &'a CellKey, + reason: SplitReason, + }, + Merge { + left: &'a PartitionBasis, + right: &'a PartitionBasis, + }, +} + +/// Bounded ephemeral state for one observed group. No samples are counted +/// twice. A changed basis, changed policy, or observation gap resets the window. +#[derive(Default)] +pub struct PartitionTracker { + last: Option, + policy: Option, + hot: u8, + quiet: u8, +} + +impl PartitionTracker { + pub fn observe(&mut self, policy: PartitionPolicy, sample: PartitionSample, now_ms: u64) { + if !sample.valid() || !fresh(&sample, policy, now_ms) { + *self = Self::default(); + return; + } + if self.policy != Some(policy) + || self + .last + .as_ref() + .is_some_and(|last| last.basis != sample.basis) + { + *self = Self::default(); + } + let mut continuity = false; + if let Some(last) = &self.last { + let Some(elapsed) = sample.sampled_at_ms.checked_sub(last.sampled_at_ms) else { + *self = Self::default(); + return; + }; + if sample.topology_changed_at_ms < last.topology_changed_at_ms { + *self = Self::default(); + return; + } + if elapsed < policy.0.sample_interval_ms { + if &sample != last { + self.hot = 0; + self.quiet = 0; + self.last = Some(sample); + } + return; + } + continuity = elapsed <= policy.0.max_sample_age_ms + && sample.topology_changed_at_ms == last.topology_changed_at_ms; + } + let stable = sample.ineligible(policy).is_none(); + let reason = sample.split_reason(policy); + let same_split = continuity + && self.last.as_ref().is_some_and(|last| { + last.split_reason(policy) == reason + && (reason != Some(SplitReason::IsolateHotCell) + || sample.hot_candidate(policy).map(|hot| &hot.key) + == last.hot_candidate(policy).map(|hot| &hot.key)) + }); + self.hot = if stable && reason.is_some() { + if same_split { + self.hot.saturating_add(1) + } else { + 1 + } + } else { + 0 + }; + self.quiet = if stable && sample.quiet(policy) { + if continuity { + self.quiet.saturating_add(1) + } else { + 1 + } + } else { + 0 + }; + self.policy = Some(policy); + self.last = Some(sample); + } + + pub fn recommend_split( + &self, + policy: PartitionPolicy, + now_ms: u64, + active_operations: u16, + ) -> PartitionDecision<'_> { + let sample = match self.eligible(policy, now_ms, active_operations) { + Ok(sample) => sample, + Err(reason) => return PartitionDecision::Wait(reason), + }; + if self.hot < policy.0.split_samples { + return PartitionDecision::Wait(WaitReason::Hysteresis); + } + if sample.load.cells == 1 { + return PartitionDecision::Wait(WaitReason::DedicatedCell); + } + let Some(reason) = sample.split_reason(policy) else { + return PartitionDecision::Wait(WaitReason::Hysteresis); + }; + let pivot = if reason == SplitReason::IsolateHotCell { + sample.hot_candidate(policy).and_then(|hot| { + if hot.has_predecessor { + Some(&hot.key) + } else { + hot.next.as_ref() + } + }) + } else { + sample.balanced_pivot.as_ref() + }; + let Some(pivot) = pivot else { + return PartitionDecision::Wait(WaitReason::NoWholeCellPivot); + }; + PartitionDecision::Split { + basis: &sample.basis, + pivot, + reason, + } + } + + pub fn recommend_merge<'a>( + &'a self, + right: &'a Self, + policy: PartitionPolicy, + now_ms: u64, + active_operations: u16, + ) -> PartitionDecision<'a> { + let left = match self.eligible(policy, now_ms, active_operations) { + Ok(sample) => sample, + Err(reason) => return PartitionDecision::Wait(reason), + }; + let other = match right.eligible(policy, now_ms, active_operations) { + Ok(sample) => sample, + Err(reason) => return PartitionDecision::Wait(reason), + }; + if !left.compatible_neighbor(other) { + return PartitionDecision::Wait(WaitReason::Incompatible); + } + if self.quiet < policy.0.merge_samples + || right.quiet < policy.0.merge_samples + || left + .load + .sum(other.load) + .is_none_or(|load| !load.within(policy.0.merge_ceiling)) + || left.leader_cpu_bps.saturating_add(other.leader_cpu_bps) > policy.0.merge_cpu_bps + { + return PartitionDecision::Wait(WaitReason::Hysteresis); + } + PartitionDecision::Merge { + left: &left.basis, + right: &other.basis, + } + } + + fn eligible( + &self, + policy: PartitionPolicy, + now_ms: u64, + active_operations: u16, + ) -> Result<&PartitionSample, WaitReason> { + let sample = self + .last + .as_ref() + .filter(|sample| self.policy == Some(policy) && fresh(sample, policy, now_ms)) + .ok_or(WaitReason::NoFreshSample)?; + if let Some(reason) = sample.ineligible(policy) { + return Err(reason); + } + if now_ms.saturating_sub(sample.topology_changed_at_ms) < policy.0.cooldown_ms { + return Err(WaitReason::Cooldown); + } + if active_operations >= policy.0.max_active_operations { + return Err(WaitReason::OperationLimit); + } + Ok(sample) + } +} + +fn fresh(sample: &PartitionSample, policy: PartitionPolicy, now_ms: u64) -> bool { + now_ms + .checked_sub(sample.sampled_at_ms) + .is_some_and(|age| age <= policy.0.max_sample_age_ms) +} + +impl PartitionSample { + fn valid(&self) -> bool { + let range = &self.basis.route.range; + let is_pivot = |key: &CellKey| { + range.contains(key) && !matches!(range.lower(), CellBound::Key(lower) if lower == key) + }; + self.basis.directory_digest != [0; 32] + && self.basis.placement_class != [0; 32] + && self.basis.route.birth_digest != [0; 32] + && self.basis.route.schema_digest != [0; 32] + && self.basis.term.get() > 0 + && self.topology_changed_at_ms <= self.sampled_at_ms + && self.leader_cpu_bps <= BASIS_POINTS + && self.failure_domain_load_bps <= BASIS_POINTS + && self + .balanced_pivot + .as_ref() + .is_none_or(|pivot| self.load.cells >= 2 && is_pivot(pivot)) + && self.hot_cell.iter().chain(&self.hot_writer).all(|hot| { + self.load.cells > 0 + && range.contains(&hot.key) + && hot.logical_bytes <= self.load.logical_bytes + && hot.writes_per_second <= self.load.writes_per_second + && (!hot.has_predecessor || is_pivot(&hot.key)) + && hot + .next + .as_ref() + .is_none_or(|next| next > &hot.key && is_pivot(next)) + && self.load.cells + >= 1 + u32::from(hot.has_predecessor) + u32::from(hot.next.is_some()) + && (self.load.cells == 1 || hot.has_predecessor || hot.next.is_some()) + }) + } + + fn ineligible(&self, policy: PartitionPolicy) -> Option { + if !self.active_and_stable { + Some(WaitReason::InTransition) + } else if self + .archive_lag_entries + .is_none_or(|lag| lag > policy.0.max_archive_lag_entries) + { + Some(WaitReason::ArchiveLag) + } else if self.failure_domain_load_bps > policy.0.max_failure_domain_load_bps { + Some(WaitReason::DestinationPressure) + } else if self.queue_age_ms > policy.0.max_queue_age_ms { + Some(WaitReason::QueuePressure) + } else { + None + } + } + + fn hot_candidate(&self, policy: PartitionPolicy) -> Option<&HotCell> { + self.hot_cell.iter().chain(&self.hot_writer).find(|hot| { + hot.logical_bytes > policy.0.hot_cell_bytes + || hot.writes_per_second > policy.0.hot_cell_writes_per_second + }) + } + + fn split_reason(&self, policy: PartitionPolicy) -> Option { + if self.hot_candidate(policy).is_some() { + Some(SplitReason::IsolateHotCell) + } else if !self.load.within(policy.0.split_ceiling) { + Some(SplitReason::GroupLoad) + } else if self.leader_cpu_bps >= policy.0.split_cpu_bps { + Some(SplitReason::LeaderCpu) + } else if self.queue_age_ms > 0 && self.load.writes_per_second > self.applies_per_second { + Some(SplitReason::ApplyBacklog) + } else { + None + } + } + + fn quiet(&self, policy: PartitionPolicy) -> bool { + self.split_reason(policy).is_none() + && self.load.within(policy.0.merge_ceiling) + && self.leader_cpu_bps <= policy.0.merge_cpu_bps + && self.queue_age_ms == 0 + && self.load.writes_per_second <= self.applies_per_second + } + + fn compatible_neighbor(&self, right: &Self) -> bool { + let left = &self.basis; + let right = &right.basis; + left.directory_digest == right.directory_digest + && left.route.history.cluster == right.route.history.cluster + && left.route.history.namespace == right.route.history.namespace + && left.route.history.group != right.route.history.group + && left.route.schema_digest == right.route.schema_digest + && left.durability == right.durability + && left.placement_class == right.placement_class + && left.route.range.upper() == right.route.range.lower() + } +} + +/// Fixed sampling resolution for the bounded per-owner write sketch. +pub const HOT_WRITE_WINDOW_MS: u64 = 1_000; +const HOT_WRITER_SLOTS: usize = 4; + +#[derive(Default)] +pub struct WriteHeat { + scope: Option<(Term, MembershipRevision)>, + active: Option, + previous: Option<(u64, CellId, u64)>, +} + +struct HeatWindow { + start_ms: u64, + writers: [Option<(CellId, u64)>; HOT_WRITER_SLOTS], +} + +impl HeatWindow { + fn record(&mut self, cell: &CellId) { + if let Some((_, writes)) = self + .writers + .iter_mut() + .flatten() + .find(|(key, _)| key == cell) + { + *writes = writes.saturating_add(1); + } else if let Some(free) = self.writers.iter_mut().find(|slot| slot.is_none()) { + *free = Some((cell.clone(), 1)); + } else { + // Misra-Gries cancels one observation of every tracked key with + // this new key. Each remaining count is a lower bound. + for slot in &mut self.writers { + if let Some((_, writes)) = slot { + *writes -= 1; + if *writes == 0 { + *slot = None; + } + } + } + } + } + + fn hottest(&self) -> Option<(&CellId, u64)> { + self.writers + .iter() + .flatten() + .max_by(|(left, lhs), (right, rhs)| lhs.cmp(rhs).then_with(|| right.cmp(left))) + .map(|(cell, writes)| (cell, *writes)) + } +} + +impl WriteHeat { + /// Track at most four current names and one previous winner. An epoch + /// change drops evidence that cannot contribute to the new rate. + pub fn record( + &mut self, + term: Term, + membership: MembershipRevision, + now_ms: u64, + cell: &CellId, + ) { + let scope = (term, membership); + let start_ms = now_ms / HOT_WRITE_WINDOW_MS * HOT_WRITE_WINDOW_MS; + if self.scope != Some(scope) { + *self = Self { + scope: Some(scope), + ..Self::default() + }; + } + if self + .active + .as_ref() + .is_none_or(|active| active.start_ms != start_ms) + { + self.previous = self.active.take().and_then(|active| { + if active.start_ms.checked_add(HOT_WRITE_WINDOW_MS) != Some(start_ms) { + return None; + } + active + .hottest() + .map(|(cell, writes)| (active.start_ms, cell.clone(), writes)) + }); + self.active = Some(HeatWindow { + start_ms, + writers: Default::default(), + }); + } + if let Some(active) = &mut self.active { + active.record(cell); + } + } + + /// Return the most recent *complete* window in the same leadership and + /// membership. Quiet windows and old epochs never reuse a candidate. + pub fn observed( + &self, + term: Term, + membership: MembershipRevision, + now_ms: u64, + ) -> Option<(u64, &CellId, u64)> { + if self.scope != Some((term, membership)) { + return None; + } + let complete = (now_ms / HOT_WRITE_WINDOW_MS).checked_sub(1)? * HOT_WRITE_WINDOW_MS; + if let Some(active) = self + .active + .as_ref() + .filter(|active| active.start_ms == complete) + { + return active + .hottest() + .map(|(cell, writes)| (complete, cell, writes)); + } + self.previous + .as_ref() + .filter(|(start_ms, _, _)| *start_ms == complete) + .map(|(_, cell, writes)| (complete, cell, *writes)) + } +} + +#[cfg(test)] +#[path = "cell_group_placement_tests.rs"] +mod tests; diff --git a/crates/cellar-logic/cell_group_placement_tests.rs b/crates/cellar-logic/cell_group_placement_tests.rs new file mode 100644 index 000000000..33d1d2ec6 --- /dev/null +++ b/crates/cellar-logic/cell_group_placement_tests.rs @@ -0,0 +1,310 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use crate::{ + cell_group::{CellId, CellRange, GroupHistory, RouteEpoch}, + consensus::{ClusterId, GroupId, IncarnationId, NamespaceId}, +}; + +fn policy() -> PartitionPolicy { + PartitionPolicy::new(PartitionLimits { + split_ceiling: PartitionLoad { + cells: 8, + logical_bytes: 1_000, + writes_per_second: 80, + }, + merge_ceiling: PartitionLoad { + cells: 4, + logical_bytes: 450, + writes_per_second: 20, + }, + split_cpu_bps: 8_000, + merge_cpu_bps: 3_000, + max_queue_age_ms: 100, + max_archive_lag_entries: 4, + max_failure_domain_load_bps: 8_500, + hot_cell_bytes: 600, + hot_cell_writes_per_second: 60, + split_samples: 2, + merge_samples: 3, + sample_interval_ms: 10, + max_sample_age_ms: 100, + cooldown_ms: 50, + max_active_operations: 1, + }) + .unwrap() +} + +// A deliberate digest collision. Full IDs still order the pivots. +fn key(value: &str) -> CellKey { + CellKey::from_digest([7; 32], CellId::new(value).unwrap()) +} + +fn sample(group: u8, range: CellRange) -> PartitionSample { + PartitionSample { + basis: PartitionBasis { + route: GroupRoute { + history: GroupHistory { + cluster: ClusterId::new([1; 16]).unwrap(), + namespace: NamespaceId::new([2; 16]).unwrap(), + group: GroupId::new([group; 16]).unwrap(), + incarnation: IncarnationId::new([group; 16]).unwrap(), + }, + range, + epoch: RouteEpoch::new(1).unwrap(), + birth_digest: [group; 32], + schema_digest: [3; 32], + }, + directory_digest: [4; 32], + membership: MembershipRevision::new(1).unwrap(), + term: Term::new(2), + durability: Durability::Quorum, + placement_class: [5; 32], + }, + sampled_at_ms: 100, + topology_changed_at_ms: 0, + load: PartitionLoad { + cells: 2, + logical_bytes: 150, + writes_per_second: 5, + }, + applies_per_second: 5, + queue_age_ms: 0, + archive_lag_entries: Some(0), + leader_cpu_bps: 400, + failure_domain_load_bps: 3_500, + active_and_stable: true, + balanced_pivot: None, + hot_cell: None, + hot_writer: None, + } +} + +fn observe(tracker: &mut PartitionTracker, mut sample: PartitionSample, times: &[u64]) { + for &at in times { + sample.sampled_at_ms = at; + tracker.observe(policy(), sample.clone(), at); + } +} + +#[test] +fn scheduler_hysteresis_and_hot_isolation() { + let mut tracker = PartitionTracker::default(); + let mut view = sample(10, CellRange::all()); + view.load.cells = 3; + view.load.logical_bytes = 800; + view.hot_cell = Some(HotCell { + key: key("middle"), + next: Some(key("right")), + has_predecessor: true, + logical_bytes: 700, + writes_per_second: 4, + }); + observe(&mut tracker, view.clone(), &[100, 100]); + assert_eq!( + tracker.recommend_split(policy(), 100, 0), + PartitionDecision::Wait(WaitReason::Hysteresis) + ); + observe(&mut tracker, view.clone(), &[110]); + assert_eq!( + tracker.recommend_split(policy(), 110, 0), + PartitionDecision::Split { + basis: &view.basis, + pivot: &key("middle"), + reason: SplitReason::IsolateHotCell, + } + ); + + // The right child inherits a recent change. Once cooled down, a second + // whole-key split can separate the now-leftmost hot cell. + let (_, right) = view.basis.route.range.split(key("middle")).unwrap(); + view.basis.route.range = right; + view.basis.route.epoch = RouteEpoch::new(2).unwrap(); + view.topology_changed_at_ms = 115; + view.load.cells = 2; + view.hot_cell.as_mut().unwrap().has_predecessor = false; + observe(&mut tracker, view.clone(), &[120, 130]); + assert_eq!( + tracker.recommend_split(policy(), 130, 0), + PartitionDecision::Wait(WaitReason::Cooldown) + ); + assert_eq!( + tracker.recommend_split(policy(), 170, 0), + PartitionDecision::Split { + basis: &view.basis, + pivot: &key("right"), + reason: SplitReason::IsolateHotCell, + } + ); + view.load.cells = 1; + view.hot_cell.as_mut().unwrap().next = None; + observe(&mut tracker, view, &[180, 190]); + assert_eq!( + tracker.recommend_split(policy(), 190, 0), + PartitionDecision::Wait(WaitReason::DedicatedCell) + ); +} + +#[test] +fn quiet_neighbors_need_a_shared_basis_and_combined_headroom() { + let (left, right) = CellRange::all().split(key("pivot")).unwrap(); + let l = sample(10, left); + let r = sample(11, right); + let (mut a, mut b) = (PartitionTracker::default(), PartitionTracker::default()); + observe(&mut a, l.clone(), &[100, 110, 120]); + observe(&mut b, r.clone(), &[100, 110]); + assert_eq!( + a.recommend_merge(&b, policy(), 120, 0), + PartitionDecision::Wait(WaitReason::Hysteresis) + ); + observe(&mut b, r.clone(), &[120]); + assert_eq!( + a.recommend_merge(&b, policy(), 120, 0), + PartitionDecision::Merge { + left: &l.basis, + right: &r.basis, + } + ); + assert_eq!( + b.recommend_merge(&a, policy(), 120, 0), + PartitionDecision::Wait(WaitReason::Incompatible) + ); + let mut too_large = r.clone(); + too_large.load.logical_bytes = 350; + observe(&mut b, too_large, &[130, 140, 150]); + assert_eq!( + a.recommend_merge(&b, policy(), 150, 0), + PartitionDecision::Wait(WaitReason::Hysteresis) + ); + for incompatible in [ + PartitionBasis { + durability: Durability::S3, + ..r.basis.clone() + }, + PartitionBasis { + placement_class: [8; 32], + ..r.basis.clone() + }, + PartitionBasis { + directory_digest: [9; 32], + ..r.basis.clone() + }, + ] { + let other = PartitionSample { + basis: incompatible, + ..r.clone() + }; + observe(&mut b, other, &[160, 170, 180]); + assert_eq!( + a.recommend_merge(&b, policy(), 180, 0), + PartitionDecision::Wait(WaitReason::Incompatible) + ); + } +} + +#[test] +fn new_term_and_unknown_archive_invalidate_an_old_recommendation() { + let mut tracker = PartitionTracker::default(); + let mut view = sample(10, CellRange::all()); + view.load.cells = 9; + view.balanced_pivot = Some(key("pivot")); + observe(&mut tracker, view.clone(), &[100, 110]); + assert!(matches!( + tracker.recommend_split(policy(), 110, 0), + PartitionDecision::Split { + reason: SplitReason::GroupLoad, + .. + } + )); + assert_eq!( + tracker.recommend_split(policy(), 110, 1), + PartitionDecision::Wait(WaitReason::OperationLimit) + ); + assert_eq!( + tracker.recommend_split(policy(), 211, 0), + PartitionDecision::Wait(WaitReason::NoFreshSample) + ); + view.basis.term = Term::new(3); + view.basis.membership = MembershipRevision::new(2).unwrap(); + observe(&mut tracker, view.clone(), &[120]); + assert_eq!( + tracker.recommend_split(policy(), 120, 0), + PartitionDecision::Wait(WaitReason::Hysteresis) + ); + view.archive_lag_entries = None; + observe(&mut tracker, view.clone(), &[121]); + assert_eq!( + tracker.recommend_split(policy(), 121, 0), + PartitionDecision::Wait(WaitReason::ArchiveLag) + ); + view.archive_lag_entries = Some(0); + view.active_and_stable = false; + observe(&mut tracker, view, &[130]); + assert_eq!( + tracker.recommend_split(policy(), 130, 0), + PartitionDecision::Wait(WaitReason::InTransition) + ); +} + +#[test] +fn write_heat_tracks_exact_keys_and_expires_completed_windows() { + let mut heat = WriteHeat::default(); + let term = Term::new(2); + let membership = MembershipRevision::new(1).unwrap(); + let hot = CellId::new("hot").unwrap(); + for n in 0..20 { + heat.record(term, membership, 10, &hot); + heat.record( + term, + membership, + 10, + &CellId::new(format!("other-{n}")).unwrap(), + ); + } + assert_eq!(heat.observed(term, membership, 999), None); + let (start, key, lower) = heat.observed(term, membership, 1_000).unwrap(); + assert_eq!((start, key), (0, &hot)); + assert!((1..=20).contains(&lower)); + heat.record(term, membership, 1_100, &CellId::new("next").unwrap()); + assert_eq!( + heat.observed(term, membership, 1_100), + Some((0, &hot, lower)) + ); + assert_eq!(heat.observed(term, membership, 3_000), None); + assert_eq!(heat.observed(Term::new(3), membership, 1_100), None); + assert_eq!( + heat.observed(term, MembershipRevision::new(2).unwrap(), 1_100), + None + ); +} + +#[test] +fn scheduler_isolates_a_writer_independently_of_image_size() { + let mut tracker = PartitionTracker::default(); + let mut view = sample(10, CellRange::all()); + view.load.cells = 3; + view.load.writes_per_second = 80; + view.hot_cell = Some(HotCell { + key: key("right"), + next: None, + has_predecessor: true, + logical_bytes: 70, + writes_per_second: 0, + }); + view.hot_writer = Some(HotCell { + key: key("middle"), + next: Some(key("right")), + has_predecessor: true, + logical_bytes: 10, + writes_per_second: 70, + }); + observe(&mut tracker, view.clone(), &[100, 110]); + assert_eq!( + tracker.recommend_split(policy(), 110, 0), + PartitionDecision::Split { + basis: &view.basis, + pivot: &key("middle"), + reason: SplitReason::IsolateHotCell, + } + ); +} diff --git a/crates/cellar-logic/cell_group_tests.rs b/crates/cellar-logic/cell_group_tests.rs new file mode 100644 index 000000000..7851b0e1d --- /dev/null +++ b/crates/cellar-logic/cell_group_tests.rs @@ -0,0 +1,180 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; + +fn key(digest: u8, id: &str) -> CellKey { + CellKey::from_digest([digest; 32], CellId::new(id).unwrap()) +} + +#[test] +fn routing_rejects_foreign_namespace_identity_and_range() { + let cell = CellHistory { + cluster: ClusterId::new([1; 16]).unwrap(), + namespace: NamespaceId::new([2; 16]).unwrap(), + cell: CellId::new("alpha").unwrap(), + incarnation: IncarnationId::new([3; 16]).unwrap(), + }; + let route = GroupRoute { + history: GroupHistory { + cluster: cell.cluster, + namespace: cell.namespace, + group: GroupId::new([4; 16]).unwrap(), + incarnation: IncarnationId::new([5; 16]).unwrap(), + }, + range: CellRange::all().split(key(3, "boundary")).unwrap().0, + epoch: RouteEpoch::new(1).unwrap(), + birth_digest: [7; 32], + schema_digest: [8; 32], + }; + assert_eq!(route.check_cell(&cell, &key(2, "alpha")), Ok(())); + assert_eq!( + route.check_cell(&cell, &key(2, "beta")), + Err(Refusal::WrongHistory) + ); + assert_eq!( + route.check_cell(&cell, &key(4, "alpha")), + Err(Refusal::WrongHistory) + ); + let mut foreign = cell.clone(); + foreign.namespace = NamespaceId::new([9; 16]).unwrap(); + assert_eq!( + route.check_cell(&foreign, &key(2, "alpha")), + Err(Refusal::WrongHistory) + ); + foreign.namespace = cell.namespace; + foreign.cluster = ClusterId::new([9; 16]).unwrap(); + assert_eq!( + route.check_cell(&foreign, &key(2, "alpha")), + Err(Refusal::WrongHistory) + ); +} + +#[test] +fn ranges_partition_digest_collisions_without_aliases() { + let first = key(2, "alpha"); + let second = key(2, "beta"); + let third = key(3, "alpha"); + assert!(first < second && second < third); + let all = CellRange::all(); + let (left, right) = all.split(second.clone()).unwrap(); + for value in [key(0, "a"), first, second, third, key(255, "z")] { + assert_ne!(left.contains(&value), right.contains(&value)); + } + assert_eq!(left.merge(&right), Ok(all)); + assert_eq!(right.merge(&left), Err(Refusal::NonContiguous)); + assert!(left.split(key(2, "beta")).is_err()); + assert!(right.split(key(2, "beta")).is_err()); + assert!(left.split(key(3, "outside")).is_err()); + for (lower, upper) in [ + (CellBound::PositiveInfinity, CellBound::PositiveInfinity), + (CellBound::PositiveInfinity, CellBound::NegativeInfinity), + (CellBound::NegativeInfinity, CellBound::NegativeInfinity), + ] { + assert!(CellRange::new(lower, upper).is_err()); + } +} + +#[test] +fn identifiers_and_counters_have_no_silent_normalization_or_wraparound() { + for value in ["", "\n", "a\0b", "\u{85}", "a\u{7f}"] { + assert_eq!(CellId::new(value), Err(Refusal::InvalidIdentity)); + } + assert!(CellId::new("é".repeat(512)).is_ok()); + assert!(CellId::new("é".repeat(513)).is_err()); + assert_ne!(CellId::new("é").unwrap(), CellId::new("e\u{301}").unwrap()); + assert!(MutationSequence::new(0).is_err()); + assert_eq!( + MutationSequence::new(1) + .unwrap() + .checked_next() + .unwrap() + .get(), + 2 + ); + assert_eq!( + MutationSequence::new(u64::MAX).unwrap().checked_next(), + Err(Refusal::InvalidPosition) + ); + assert!(RouteEpoch::new(0).is_err()); + assert!(RouteEpoch::new(u64::MAX).unwrap().checked_next().is_err()); + assert!(MembershipRevision::new(0).is_err()); + assert!(MembershipRevision::new(u64::MAX) + .unwrap() + .checked_next() + .is_err()); +} + +#[test] +fn cell_versions_require_exact_history_and_same_sequence_image() { + let version = CellVersion { + history: CellHistory { + cluster: ClusterId::new([1; 16]).unwrap(), + namespace: NamespaceId::new([2; 16]).unwrap(), + cell: CellId::new("alpha").unwrap(), + incarnation: IncarnationId::new([3; 16]).unwrap(), + }, + sequence: MutationSequence::new(2).unwrap(), + image: CellImage { + epoch: 1, + txid: 2, + checksum: 123, + commitment: CellImageCommitment::Materialized([7; 32]), + }, + }; + assert_eq!(version.covers(&version), Ok(true)); + let mut later = version.clone(); + later.sequence = later.sequence.checked_next().unwrap(); + later.image.txid += 1; + assert_eq!(later.covers(&version), Ok(true)); + assert_eq!(version.covers(&later), Ok(false)); + for field in 0..4 { + let mut other = later.clone(); + match field { + 0 => other.history.cluster = ClusterId::new([9; 16]).unwrap(), + 1 => other.history.namespace = NamespaceId::new([9; 16]).unwrap(), + 2 => other.history.cell = CellId::new("beta").unwrap(), + _ => other.history.incarnation = IncarnationId::new([9; 16]).unwrap(), + } + assert_eq!(other.covers(&version), Err(Refusal::WrongHistory)); + } + for field in 0..4 { + let mut conflicting = version.clone(); + match field { + 0 => conflicting.image.epoch += 1, + 1 => conflicting.image.txid += 1, + 2 => conflicting.image.checksum += 1, + _ => { + conflicting.image.commitment = CellImageCommitment::Extents { + root: [7; 32], + manifest_bytes: 112, + } + } + } + assert_eq!(conflicting.covers(&version), Err(Refusal::WrongHistory)); + } +} + +#[test] +fn physical_history_projection_preserves_every_group_identity_field() { + let original = crate::consensus::GroupScope { + cluster: ClusterId::new([1; 16]).unwrap(), + namespace: NamespaceId::new([2; 16]).unwrap(), + group: GroupId::new([3; 16]).unwrap(), + incarnation: IncarnationId::new([4; 16]).unwrap(), + cell: "alpha".into(), + }; + let history = GroupHistory::from(&original); + let mut other = original.clone(); + other.cell = "beta".into(); + assert_eq!(GroupHistory::from(&other), history); + for field in 0..4 { + let mut other = original.clone(); + match field { + 0 => other.cluster = ClusterId::new([9; 16]).unwrap(), + 1 => other.namespace = NamespaceId::new([9; 16]).unwrap(), + 2 => other.group = GroupId::new([9; 16]).unwrap(), + _ => other.incarnation = IncarnationId::new([9; 16]).unwrap(), + } + assert_ne!(GroupHistory::from(&other), history); + } +} diff --git a/crates/cellar-logic/consensus.rs b/crates/cellar-logic/consensus.rs index 6883648a6..ff12466c9 100644 --- a/crates/cellar-logic/consensus.rs +++ b/crates/cellar-logic/consensus.rs @@ -109,15 +109,15 @@ pub struct NodeIdentity { /// A process operating a particular durable member disk. #[derive(Clone, Debug, PartialEq, Eq)] -pub struct ReplicaIdentity { - pub scope: GroupScope, +pub struct ReplicaIdentity { + pub scope: S, pub node: NodeId, pub member: MemberId, pub disk: DiskLineage, pub process: ProcessGeneration, } -impl ReplicaIdentity { +impl ReplicaIdentity { pub fn node_identity(&self) -> NodeIdentity { NodeIdentity { node: self.node, diff --git a/crates/cellar-logic/lib.rs b/crates/cellar-logic/lib.rs index 6248188bf..4c06e229b 100644 --- a/crates/cellar-logic/lib.rs +++ b/crates/cellar-logic/lib.rs @@ -14,6 +14,8 @@ use std::collections::{BTreeMap, BTreeSet, VecDeque}; pub mod alarm; pub mod cache; pub mod cell; +pub mod cell_group; +pub mod cell_group_placement; pub mod consensus; pub mod cron; pub mod drain; diff --git a/crates/cellar-logic/raft_fleet/ledger.rs b/crates/cellar-logic/raft_fleet/ledger.rs index 69f765073..d41ee4b01 100644 --- a/crates/cellar-logic/raft_fleet/ledger.rs +++ b/crates/cellar-logic/raft_fleet/ledger.rs @@ -3,9 +3,8 @@ //! Conservative accounting before a membership move. Serving roles are not //! physical resources: a retired source still occupies its engine and disk. -use crate::consensus::{ - ClusterId, EngineInstanceId, GroupScope, IncarnationId, NodeIdentity, ProposalId, -}; +use crate::cell_group::GroupHistory; +use crate::consensus::{ClusterId, EngineInstanceId, IncarnationId, NodeIdentity, ProposalId}; use std::collections::{BTreeMap, BTreeSet}; const MAX_RESERVATIONS: usize = 4096; @@ -153,7 +152,9 @@ pub struct Materialization { #[derive(Clone, Debug, PartialEq, Eq)] pub struct Reservation { pub id: ProposalId, - pub group: GroupScope, + /// One physical replica, regardless of its cell count. All of its retained + /// images and staging work remain charged in peak/materialized resources. + pub group: GroupHistory, pub scope: CapacityScope, /// Increment beyond starting physical occupancy, including temporary work. pub peak: Resources, @@ -163,9 +164,6 @@ pub struct Reservation { impl Reservation { fn validate(&self, scope: CapacityScope) -> Result<(), LedgerError> { self.scope.validate()?; - self.group - .validate() - .map_err(|_| LedgerError::InvalidReservation)?; if !scope.same_engine(self.scope) || self.group.cluster != scope.cluster || self.peak == Resources::default() diff --git a/crates/cellar-logic/raft_fleet/mod.rs b/crates/cellar-logic/raft_fleet/mod.rs index 4b036c3a1..094f70c21 100644 --- a/crates/cellar-logic/raft_fleet/mod.rs +++ b/crates/cellar-logic/raft_fleet/mod.rs @@ -14,7 +14,7 @@ pub use ledger::{ CapacityScope, LedgerError, LedgerPolicy, LedgerSnapshot, Materialization, PhysicalSample, Reservation, ResourceCharge, ResourceKind, ResourceLedger, Resources, SampleAttribution, }; -pub use repair::{select_isolated_repair, IsolatedRepairView}; +pub use repair::{select_isolated_repair, GroupRepairPlacement, IsolatedRepairView}; pub use topology::{ balance_topology, preview_topology, NodeRecovery, ReplicaMove, TopologyError, TopologyGroup, TopologyNode, TopologyPeak, TopologyPolicy, TopologyPreview, TopologyTarget, TopologyView, diff --git a/crates/cellar-logic/raft_fleet/repair.rs b/crates/cellar-logic/raft_fleet/repair.rs index 53d89df37..408f3e895 100644 --- a/crates/cellar-logic/raft_fleet/repair.rs +++ b/crates/cellar-logic/raft_fleet/repair.rs @@ -3,12 +3,42 @@ //! One required replacement after the adapter proves exact-boot isolation. //! Selection grants no membership, health, reservation or release authority. use super::{ResourceLedger, Resources, TopologyError, TopologyNode, TopologyPolicy}; -use crate::consensus::{GroupScope, LogIndex, MemberId, NodeId}; -use crate::raft_placement::{PlacedMember, RepairPlacement}; +use crate::cell_group::GroupHistory; +use crate::consensus::{LogIndex, MemberId, NodeId}; +use crate::raft_placement::PlacedMember; use std::collections::BTreeSet; +/// A physical-group replacement proposal. It grants no membership or isolation +/// authority. Every cell in this history moves under the same revision fence. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct GroupRepairPlacement { + pub scope: GroupHistory, + pub policy_revision: u64, + pub expected_revision: LogIndex, + pub obsolete: MemberId, + pub replacement: PlacedMember, +} + +// Stable load tie-break for a complete physical history. Cell identities never +// enter placement scoring; the adapter accounts their combined resource peak. +fn rank_group(history: &GroupHistory, node: NodeId) -> u64 { + let mut hash = 0xcbf29ce484222325_u64; + for bytes in [ + history.cluster.bytes(), + history.namespace.bytes(), + history.group.bytes(), + history.incarnation.bytes(), + node.bytes(), + ] { + for byte in bytes { + hash = (hash ^ u64::from(byte)).wrapping_mul(0x100000001b3); + } + } + hash +} + pub struct IsolatedRepairView { - pub scope: GroupScope, + pub scope: GroupHistory, pub revision: LogIndex, pub high_water: u64, pub voters: [PlacedMember; 3], @@ -49,7 +79,7 @@ pub fn select_isolated_repair( nodes: &[TopologyNode], policy: TopologyPolicy, now_ms: u64, -) -> Result { +) -> Result { if policy.revision == 0 || policy.max_work == 0 || policy.max_work > 50_000_000 { return Err(TopologyError::InvalidPolicy); } @@ -59,8 +89,7 @@ pub fn select_isolated_repair( if policy.max_work < nodes.len() as u64 * 2 { return Err(TopologyError::SearchExhausted); } - if view.scope.validate().is_err() - || view.revision.get() == 0 + if view.revision.get() == 0 || view.revision.get() >= u64::MAX - 2 || !label(&view.failed_host) || view.peak.engine_slots != 1 @@ -191,9 +220,8 @@ pub fn select_isolated_repair( (an * bd) .cmp(&(bn * ad)) .then_with(|| { - crate::raft_placement::rank(&view.scope, b.ledger.scope.node.node).cmp( - &crate::raft_placement::rank(&view.scope, a.ledger.scope.node.node), - ) + rank_group(&view.scope, b.ledger.scope.node.node) + .cmp(&rank_group(&view.scope, a.ledger.scope.node.node)) }) .then_with(|| a.ledger.scope.node.node.cmp(&b.ledger.scope.node.node)) }); @@ -206,8 +234,8 @@ pub fn select_isolated_repair( .iter() .find(|v| v.node == view.failed) .ok_or(TopologyError::InvalidInput)?; - Ok(RepairPlacement { - scope: view.scope.clone(), + Ok(GroupRepairPlacement { + scope: view.scope, policy_revision: policy.revision, expected_revision: view.revision, obsolete: obsolete.member, diff --git a/crates/cellar-logic/raft_fleet/topology/mod.rs b/crates/cellar-logic/raft_fleet/topology/mod.rs index dead60fb1..0bb22b63a 100644 --- a/crates/cellar-logic/raft_fleet/topology/mod.rs +++ b/crates/cellar-logic/raft_fleet/topology/mod.rs @@ -10,7 +10,8 @@ mod schedule; pub use balance::balance_topology; use super::{CapacityScope, LedgerError, LedgerSnapshot, ResourceLedger, Resources}; -use crate::consensus::{GroupScope, LogIndex, NodeId}; +use crate::cell_group::GroupHistory; +use crate::consensus::{LogIndex, NodeId}; use std::collections::{BTreeMap, BTreeSet}; #[derive(Clone, Debug)] @@ -30,7 +31,7 @@ pub struct TopologyNode { #[derive(Clone, Debug)] pub struct TopologyGroup { - pub scope: GroupScope, + pub scope: GroupHistory, /// Initial stable RF3 revision. Re-read before each actual membership permit. pub revision: LogIndex, pub voters: [NodeId; 3], @@ -44,7 +45,7 @@ pub struct TopologyGroup { #[derive(Clone, Debug, PartialEq, Eq)] pub struct TopologyTarget { - pub group: GroupScope, + pub group: GroupHistory, pub voters: [NodeId; 3], } @@ -69,7 +70,7 @@ pub struct TopologyPolicy { #[derive(Clone, Debug, PartialEq, Eq)] pub struct ReplicaMove { - pub group: GroupScope, + pub group: GroupHistory, pub source: CapacityScope, pub target: CapacityScope, /// Serial dependency: complete the preceding move's terminal proof first. @@ -214,7 +215,7 @@ fn verify(model: &Model<'_>, work: &mut Work) -> Result Model<'a> { }; model.groups.sort_by_key(|group| &group.scope); let mut histories = BTreeSet::new(); - let mut cells = BTreeSet::new(); let mut occupancy = vec![0_u64; view.nodes.len()]; for group in &model.groups { - group - .scope - .validate() - .map_err(|_| TopologyError::InvalidInput)?; if group.scope.cluster != first.cluster || group.revision.get() == 0 || group.revision.get() >= u64::MAX - 2 - || !histories.insert(group.scope.group) - || !cells.insert((group.scope.namespace, &group.scope.cell)) + || !histories.insert((group.scope.namespace, group.scope.group)) || group.resident.engine_slots != 1 || group.resident.durable_bytes == 0 || group.resident.memory_bytes == 0 diff --git a/crates/cellar-logic/raft_fleet/topology/schedule.rs b/crates/cellar-logic/raft_fleet/topology/schedule.rs index 8bf60ce90..7d1069fb5 100644 --- a/crates/cellar-logic/raft_fleet/topology/schedule.rs +++ b/crates/cellar-logic/raft_fleet/topology/schedule.rs @@ -145,7 +145,7 @@ fn materialize( peaks[a.target].high_water(add(retained[a.target], model.groups[a.group].peak)?); retained[a.target] = add(retained[a.target], model.groups[a.group].resident)?; moves.push(ReplicaMove { - group: model.groups[a.group].scope.clone(), + group: model.groups[a.group].scope, source: model.view.nodes[a.source].ledger.scope, target: model.view.nodes[a.target].ledger.scope, after: i.checked_sub(1), diff --git a/crates/cellar-logic/tests/raft_fleet_ledger.rs b/crates/cellar-logic/tests/raft_fleet_ledger.rs index 8d6406c03..4d2bdb814 100644 --- a/crates/cellar-logic/tests/raft_fleet_ledger.rs +++ b/crates/cellar-logic/tests/raft_fleet_ledger.rs @@ -3,6 +3,7 @@ //! Pure resource-accounting boundary. These tests do not qualify catalog CAS, //! physical observations, membership execution or fleet performance. +use cellar_logic::cell_group::GroupHistory; use cellar_logic::{consensus::*, raft_fleet::*}; fn resources(slots: u64, disk: u64, memory: u64) -> Resources { @@ -41,10 +42,10 @@ fn sample(sequence: u64, physical: Resources) -> PhysicalSample { fn reservation(id: u8, peak: Resources) -> Reservation { Reservation { id: ProposalId::new([id; 16]).unwrap(), - group: GroupScope { + group: GroupHistory { cluster: scope().cluster, namespace: NamespaceId::new([9; 16]).unwrap(), - cell: format!("Counter:{id}"), + group: GroupId::new([id; 16]).unwrap(), incarnation: IncarnationId::new([10; 16]).unwrap(), }, @@ -438,3 +439,24 @@ fn malformed_inputs_and_bounded_inventory_refuse() { Err(LedgerError::Limit) )); } + +#[test] +fn group_incarnation_cannot_reuse_a_reservation_or_erase_old_replica_debt() { + let mut ledger = ResourceLedger::restore(snapshot()).unwrap(); + let first = reservation(11, resources(1, 100, 50)); + ledger.reserve(first.clone(), 100).unwrap(); + let mut fresh = first.clone(); + fresh.group.incarnation = IncarnationId::new([79; 16]).unwrap(); + assert_eq!( + ledger.reserve(fresh.clone(), 100), + Err(LedgerError::PayloadConflict) + ); + assert_eq!(ledger.charge(100).unwrap().unmaterialized, first.peak); + fresh.id = ProposalId::new([12; 16]).unwrap(); + ledger.reserve(fresh, 100).unwrap(); + assert_eq!( + ledger.charge(100).unwrap().unmaterialized, + resources(2, 200, 100) + ); + assert_eq!(ledger.snapshot().reservations.len(), 2); +} diff --git a/crates/cellar-logic/tests/raft_fleet_repair.rs b/crates/cellar-logic/tests/raft_fleet_repair.rs index 4411c494b..793e47240 100644 --- a/crates/cellar-logic/tests/raft_fleet_repair.rs +++ b/crates/cellar-logic/tests/raft_fleet_repair.rs @@ -1,6 +1,7 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. //! Required repair selection; the host owns isolation and quorum provenance. +use cellar_logic::cell_group::GroupHistory; use cellar_logic::{consensus::*, raft_fleet::*, raft_placement::PlacedMember}; fn id(n: u8) -> NodeId { @@ -22,10 +23,10 @@ fn fixture() -> (IsolatedRepairView, Vec, TopologyPolicy) { node: id(n), disk: DiskLineage::new([n + 10; 16]).unwrap(), }; - let group = GroupScope { + let group = GroupHistory { cluster, namespace: NamespaceId::new([2; 16]).unwrap(), - cell: "Counter:repair".into(), + group: GroupId::new([3; 16]).unwrap(), incarnation: IncarnationId::new([4; 16]).unwrap(), }; @@ -183,7 +184,7 @@ fn isolated_repair_refuses_unsettled_and_incomplete_ledgers() { let scope = n[2].ledger.scope; n[2].ledger.reservations.push(Reservation { id: ProposalId::new([6; 16]).unwrap(), - group: v.scope.clone(), + group: v.scope, scope, peak: bytes(1), materialized: None, @@ -244,9 +245,8 @@ fn isolated_repair_refuses_overflow_instead_of_saturating_capacity() { ); } -fn unrelated(node: &TopologyNode, group: &GroupScope, id: u8) -> Reservation { - let mut other = group.clone(); - other.cell = format!("Counter:unrelated-{id}"); +fn unrelated(node: &TopologyNode, group: &GroupHistory, id: u8) -> Reservation { + let mut other = *group; other.group = GroupId::new([id; 16]).unwrap(); Reservation { id: ProposalId::new([id; 16]).unwrap(), diff --git a/crates/cellar-logic/tests/raft_fleet_topology.rs b/crates/cellar-logic/tests/raft_fleet_topology.rs index 28c6a3ed1..db42fd718 100644 --- a/crates/cellar-logic/tests/raft_fleet_topology.rs +++ b/crates/cellar-logic/tests/raft_fleet_topology.rs @@ -3,8 +3,9 @@ //! Executable topology witnesses. No catalog, deployed fleet or performance //! qualification is implied by these pure decision-core cases. +use cellar_logic::cell_group::GroupHistory; use cellar_logic::{consensus::*, raft_fleet::*}; -use std::collections::BTreeMap; +use std::collections::{BTreeMap, BTreeSet}; fn node(n: usize) -> NodeId { NodeId::new([n as u8 + 1; 16]).unwrap() @@ -79,10 +80,10 @@ fn fixture(count: usize, triples: &[[usize; 3]]) -> (TopologyView, Vec (TopologyView, Vec Vec<[usize; 3]> { } fn replay(view: &TopologyView, targets: &[TopologyTarget], moves: &[ReplicaMove]) { - let mut voters: BTreeMap<_, _> = view - .groups - .iter() - .map(|g| (g.scope.clone(), g.voters)) - .collect(); + let mut voters: BTreeMap<_, _> = view.groups.iter().map(|g| (g.scope, g.voters)).collect(); for (i, m) in moves.iter().enumerate() { assert_eq!(m.after, i.checked_sub(1)); let set = voters.get_mut(&m.group).unwrap(); @@ -360,7 +357,7 @@ fn incomplete_stale_and_unsettled_inputs_cannot_authorize_preview() { let scope = view.nodes[3].ledger.scope; view.nodes[3].ledger.reservations.push(Reservation { id: ProposalId::new([5; 16]).unwrap(), - group: view.groups[0].scope.clone(), + group: view.groups[0].scope, scope, peak: resources(1, 10, 10), materialized: None, @@ -651,7 +648,7 @@ fn automatic_balance_requires_complete_stable_fresh_inputs() { let scope = view.nodes[3].ledger.scope; view.nodes[3].ledger.reservations.push(Reservation { id: ProposalId::new([5; 16]).unwrap(), - group: view.groups[0].scope.clone(), + group: view.groups[0].scope, scope, peak: resources(1, 10, 10), materialized: None, @@ -745,3 +742,33 @@ fn automatic_balance_all_small_two_group_layouts_preserve_rf3() { } } } + +#[test] +fn topology_refuses_two_current_incarnations_of_one_physical_group() { + let (mut view, mut targets) = fixture(4, &[[0, 1, 2], [1, 2, 3]]); + view.groups[1].scope = view.groups[0].scope; + view.groups[1].scope.incarnation = IncarnationId::new([77; 16]).unwrap(); + targets[1].group = view.groups[1].scope; + assert_eq!( + preview_topology(&view, policy(), &targets, 100), + Err(TopologyError::InvalidInput) + ); +} + +#[test] +fn topology_binds_opaque_group_ids_to_their_namespace() { + let (mut view, mut targets) = fixture(4, &[[0, 1, 2], [1, 2, 3]]); + view.groups[1].scope.group = view.groups[0].scope.group; + view.groups[1].scope.namespace = NamespaceId::new([78; 16]).unwrap(); + targets[1].group = view.groups[1].scope; + let plan = preview_topology(&view, policy(), &targets, 100).unwrap(); + assert!(plan.moves.is_empty()); + assert_eq!(plan.assignments.len(), 2); + assert_eq!( + plan.assignments + .iter() + .map(|a| a.group) + .collect::>(), + view.groups.iter().map(|g| g.scope).collect::>() + ); +} diff --git a/crates/cellar-ltx/README.md b/crates/cellar-ltx/README.md index 75f3f7e5b..994e3d582 100644 --- a/crates/cellar-ltx/README.md +++ b/crates/cellar-ltx/README.md @@ -37,6 +37,13 @@ identities, files, cached images and persistent storage live together in [`src/extent/`](src/extent/mod.rs), with the node inventory in `store/`. Public `Db`/`db` paths and the `internal` verification facade remain unchanged. +`GenerationStore::pin_prepared` reopens an exactly identified materialized image +for a preparation retry. It verifies cold image bytes and persistence barriers, +preserves the existing host retention flag, and refuses missing or mismatched +images. Its pin prevents reclamation while borrowed; it supplies no apply, +publication or serving authority. The caller must obtain the expected state from +its committed input and publish the resulting complete manifest separately. + `RootView::read_extent` supplies one complete dependency to an admitted blocking transfer owner. The caller reserves its at-most-4-MiB output. It refuses objects outside the pinned root and returns bytes only after the full extent identity @@ -46,15 +53,46 @@ reads. This does not create a recovery or native admission capability. image seal. An identical root from another directory cannot substitute for that local physical capability. +`NodeExtentStore::prepare_transient_image_root` runs the same bounded LTX, +geometry, closure and persistence checks as `prepare_image_root`, but a fresh +root does not keep an additional permanent host reference. Its returned image +pin protects the complete closure; an existing root keeps its prior retention. +The shared-group host transfers that pin into its complete manifest graph, +whose current, checkpoint and uncertain publication roots stay pinned. Startup +recovers every local group's selected graphs before releasing conservative +filename retention. Collection still requires a reconciled node inventory, +respects reader/archive pins, and refunds capacity only after synced deletion. +`NodeExtentStore::needs_reclaim` provides a metadata-only scheduling hint after +reference reconciliation. It notices the last reader's release without requiring +another write; an idle, already marked inventory needs no file scan. It supplies +no collection authority. `reclaim_working_bytes` computes checked temporary-memory +admission for a bounded physical turn. Marking includes only locally present +objects, preserving dependencies imported under a pinned, incomplete receive. +The shared-group host schedules one node-owned collector for the serving and +independent verifier stores. Its existing control admission and shutdown registry +retain inventory handles and working memory through physical completion. This +does not authorize manifest-page collection, Raft log truncation or S3 coverage, +and does not qualify sustained load. + +Root metadata keeps a volatile observation of completed file and directory +barriers on its exact open inventory owner. A repeated seal still reads and +checks the entire root file and SQLite image bytes, while reusing those completed +barriers for unchanged immutable files. Reopening clears the observation; a failed +root read also invalidates it. A metadata-only read supplies no persistence proof, +and an incomplete or failed barrier cannot be reused. This observation supplies +neither a SQLite image seal by itself nor remote archive coverage. + `NodeExtentStore::put_extent_bytes` borrows a complete, already admitted extent through the existing at-most-4-MiB positional write window. Native providers avoid splitting it into allocated 64 KiB source chunks; injected providers keep their bounded positional-write fallback. Capacity admission, exclusive creation, source identity, file sync, independent destination verification and directory -sync still precede a reusable local proof. Existing objects are independently -reverified rather than overwritten. Peer and archive receives share this path, -as does an image update whose changed extent already occupies its admitted -workspace. Streaming source callbacks retain the separate 64 KiB contract. +sync still precede a reusable local proof. Buffered imports check their supplied +bytes against the extent identity even when the destination already exists. +Existing objects are independently reverified rather than overwritten. Peer and +archive receives share this path, as does an image update whose changed extent +already occupies its admitted workspace. Streaming source callbacks retain the +separate 64 KiB contract. `RootView::image` caches read-only handles to its pinned local extents. Full independent verification derives bounded block hashes that the inventory retains @@ -101,6 +139,17 @@ proof before acknowledgement. Ordinary `Db` constructors retain durable capture. The Raft host uses this mode only for private command execution: its verified preview, committed immutable image and Raft log keep their persistence barriers. +`NodeWorkspace` is created under an exclusively locked extent inventory whose +open already completed the ancestor barriers. Creation still syncs the new child +directory and its parent before returning its owner. Rooted private execution +validates and retains that exact owner instead of repeating the ancestor walk; +arbitrary materialized paths establish their own barriers. Child or parent sync +failure quarantines the inventory and retains the complete reservation until +recovery. Last-pin cleanup still syncs deletion before refunding capacity. +Anonymous read-cache directories use ordinary creation because they are rebuilt +after a crash. Neither those files nor the directory owner supplies an immutable +image seal, a Raft acknowledgement or recovery authority. + `Db::require_incremental_capture` restricts disposable capture to incremental updates after its initial or seeded TXID. A lost WAL continuation or checkpoint boundary that needs a full image returns `CaptureRebuildRequired` before snapshot diff --git a/crates/cellar-ltx/src/extent/store/image.rs b/crates/cellar-ltx/src/extent/store/image.rs index 510489a52..25a8fbf34 100644 --- a/crates/cellar-ltx/src/extent/store/image.rs +++ b/crates/cellar-ltx/src/extent/store/image.rs @@ -176,6 +176,31 @@ impl NodeExtentStore { update: &ImageRootUpdate<'_>, limits: &ApplyLimits, buffer: &mut [u8], + ) -> Result { + self.prepare_image_root_retention(base, update, limits, buffer, false) + } + + /// Prepare an image whose complete host graph will own the returned pin. + /// A fresh root becomes reclaimable after its last pin/reference closes. + /// An existing root keeps its prior retention. The host must recover and + /// pin all durable selections before reconciling the shared inventory. + pub fn prepare_transient_image_root( + &self, + base: Option<&SealedImageRoot>, + update: &ImageRootUpdate<'_>, + limits: &ApplyLimits, + buffer: &mut [u8], + ) -> Result { + self.prepare_image_root_retention(base, update, limits, buffer, true) + } + + fn prepare_image_root_retention( + &self, + base: Option<&SealedImageRoot>, + update: &ImageRootUpdate<'_>, + limits: &ApplyLimits, + buffer: &mut [u8], + transient: bool, ) -> Result { check_buffer(buffer)?; let mut current = match base { @@ -250,10 +275,20 @@ impl NodeExtentStore { // Header/hole checks are independent of the proposed commitment. // A checksum-enabled result has already been checked at each file. self.check_image_geometry(¤t, buffer, &mut work)?; - Ok(PreparedImageRoot { - image: SealedImageRoot(self.seal_root_inner(current)?), - work, - }) + // The serialized physical operation keeps the existence test and + // retention change atomic with sealing. The new view pins the + // entire closure before any root can become reclaimable. + let existed = self.inner.state()?.roots.contains_key(¤t.root()); + let image = SealedImageRoot(self.seal_root_inner(current)?); + if transient && !existed { + self.inner + .state()? + .roots + .get_mut(&image.reference().root) + .ok_or(StoreError::Missing)? + .retained = false; + } + Ok(PreparedImageRoot { image, work }) })(); operation.finish()?; result diff --git a/crates/cellar-ltx/src/extent/store/inventory.rs b/crates/cellar-ltx/src/extent/store/inventory.rs index 0782ece1a..6d1625c75 100644 --- a/crates/cellar-ltx/src/extent/store/inventory.rs +++ b/crates/cellar-ltx/src/extent/store/inventory.rs @@ -145,6 +145,7 @@ impl NodeExtentStore { bytes: metadata.len, slots, retained: true, + persisted: false, token: Arc::new(()), data: None, }, @@ -194,6 +195,31 @@ impl NodeExtentStore { reference: ExtentRootReference, persist: bool, ) -> Result, StoreError> { + let result = self.read_root_inner(reference, persist); + if result.is_err() { + if let Some(entry) = self.inner.state()?.roots.get_mut(&reference.root) { + entry.persisted = false; + } + } + result + } + + fn read_root_inner( + &self, + reference: ExtentRootReference, + persist: bool, + ) -> Result, StoreError> { + let persisted = { + let state = self.inner.state()?; + let entry = state + .roots + .get(&reference.root) + .ok_or(StoreError::Missing)?; + if entry.bytes != u64::from(reference.length) { + return Err(StoreError::Integrity); + } + entry.persisted + }; let path = self.inner.directory.join(root_name(reference.root)); let mut file = self .inner @@ -211,9 +237,15 @@ impl NodeExtentStore { return Err(StoreError::Integrity); } let manifest = ExtentManifest::decode(&bytes, reference.root)?; - if persist { + if persist && !persisted { file.sync_all()?; self.inner.filesystem.sync_all(&self.inner.directory)?; + self.inner + .state()? + .roots + .get_mut(&reference.root) + .ok_or(StoreError::Missing)? + .persisted = true; } let mut objects = BTreeSet::new(); for index in 0..manifest.geometry().extent_count() { diff --git a/crates/cellar-ltx/src/extent/store/mod.rs b/crates/cellar-ltx/src/extent/store/mod.rs index a809e7ba0..4d56363b1 100644 --- a/crates/cellar-ltx/src/extent/store/mod.rs +++ b/crates/cellar-ltx/src/extent/store/mod.rs @@ -159,6 +159,9 @@ struct RootEntry { bytes: u64, slots: usize, retained: bool, + // Only this open owner's completed file and directory barriers. Bytes are + // still read and checked before another seal; restart clears this observation. + persisted: bool, token: Arc<()>, data: Option>, } @@ -180,6 +183,7 @@ struct State { busy: bool, quarantined: bool, references_reconciled: bool, + reclaim_scan_needed: bool, } struct Inner { diff --git a/crates/cellar-ltx/src/extent/store/publication.rs b/crates/cellar-ltx/src/extent/store/publication.rs index c7f365adf..8f152deda 100644 --- a/crates/cellar-ltx/src/extent/store/publication.rs +++ b/crates/cellar-ltx/src/extent/store/publication.rs @@ -84,13 +84,10 @@ impl NodeExtentStore { id: ExtentId, bytes: &[u8], ) -> Result<(), StoreError> { - if bytes.len() != id.length() as usize { + if bytes.len() != id.length() as usize || ExtentId::of(bytes)? != id { return Err(StoreError::Integrity); } self.put_extent_with(id, |file| { - if ExtentId::of(bytes)? != id { - return Err(StoreError::Integrity); - } // The caller already owns this bounded input. Avoid a second // extent allocation and preserve injected providers' fallback. file.write_all_at_window(0, bytes)?; @@ -166,6 +163,7 @@ impl NodeExtentStore { read_proof: None, }, ); + state.reclaim_scan_needed = true; } let path = self.inner.directory.join(object_name(id)); let mut file = match self.inner.filesystem.create_read_write(&path) { @@ -339,6 +337,7 @@ impl NodeExtentStore { bytes: bytes.len() as u64, slots, retained: true, + persisted: false, token: Arc::new(()), data: None, }, @@ -392,12 +391,15 @@ impl NodeExtentStore { .map(|index| manifest.extent(index).ok_or(StoreError::Layout)) .collect::, _>>()?; let data = Arc::new(RootData { manifest, objects }); - self.inner - .state()? - .roots - .get_mut(&reference.root) - .ok_or(StoreError::Missing)? - .data = Some(data.clone()); + { + let mut state = self.inner.state()?; + let entry = state + .roots + .get_mut(&reference.root) + .ok_or(StoreError::Missing)?; + entry.persisted = true; + entry.data = Some(data.clone()); + } self.view(reference, data) } } diff --git a/crates/cellar-ltx/src/extent/store/reclaim.rs b/crates/cellar-ltx/src/extent/store/reclaim.rs index 1feddef5b..ef5ab3387 100644 --- a/crates/cellar-ltx/src/extent/store/reclaim.rs +++ b/crates/cellar-ltx/src/extent/store/reclaim.rs @@ -4,6 +4,55 @@ use super::inventory::{object_name, root_name}; use super::*; impl NodeExtentStore { + /// Metadata-only hint. A reader's last root pin can make progress possible + /// without another write. Idle, already marked inventories need no file scan. + /// This grants neither collection authority nor a promise of deletion. + pub fn needs_reclaim(&self) -> Result { + let state = self.inner.state()?; + if state.quarantined { + return Err(StoreError::Quarantined); + } + if !state.references_reconciled { + return Err(StoreError::Unreconciled); + } + Ok(state.reclaim_scan_needed + || state.roots.iter().any(|(root, entry)| { + !entry.retained + && Arc::strong_count(&entry.token) == 1 + && !state + .references + .values() + .any(|r| r.active && r.reference.root == *root) + })) + } + + /// Conservative temporary-memory admission for one collection turn. Marks + /// include only objects present in this inventory, even when pinned receive + /// roots also name objects that have not arrived yet. The host retains this + /// admission through physical completion; this does not authorize collection. + pub fn reclaim_working_bytes(&self, max_files: usize) -> Result { + if !(1..=4096).contains(&max_files) { + return Err(StoreError::Limit); + } + // One encoded/decoded manifest, its persistent tree and unique IDs; + // complete root references; local object marks; deletion lists and I/O. + MAX_MANIFEST_BYTES + .checked_mul(16) + .and_then(|n| n.checked_add(self.inner.limits.max_roots.checked_mul(64)?)) + .and_then(|n| { + n.checked_add( + self.inner + .limits + .max_objects + .min(self.inner.limits.max_root_extents) + .checked_mul(128)?, + ) + }) + .and_then(|n| n.checked_add(max_files.checked_mul(128)?)) + .and_then(|n| n.checked_add(65536)) + .ok_or(StoreError::Limit) + } + /// Retain an already sealed image without I/O or waiting for a physical /// mutation. The live exact-owner capability pins its root throughout GC; /// native control loops can therefore publish retention using metadata only. @@ -73,6 +122,7 @@ impl NodeExtentStore { entry.retained = selected.contains(root); } state.references_reconciled = true; + state.reclaim_scan_needed = true; Ok(()) })(); operation.finish()?; @@ -150,23 +200,35 @@ impl NodeExtentStore { state.bytes -= entry.bytes; state.root_extents -= entry.slots; } + state.reclaim_scan_needed = true; } if roots.len() == limit { return Ok(roots.len()); } - let references = self - .inner - .state()? - .roots - .iter() - .map(|(root, entry)| ExtentRootReference::new(*root, entry.bytes as u32)) - .collect::, _>>()?; + let references = { + let state = self.inner.state()?; + let mut references = Vec::with_capacity(state.roots.len()); + for (root, entry) in &state.roots { + references.push(ExtentRootReference::new(*root, entry.bytes as u32)?); + } + references + }; let mut live = BTreeSet::new(); for reference in references { // Metadata-only traversal, one bounded manifest at a time. Do not // load every root tree into the persistent in-memory cache for GC. let data = self.read_root_file(reference)?; - live.extend(data.objects.iter().copied()); + // The physical operation excludes imports/publication throughout + // this scan. Missing receive dependencies cannot be deleted here; + // filter them before allocating marks, rather than charging the sum + // of every remote root's slots to a live node's collector. + let state = self.inner.state()?; + live.extend( + data.objects + .iter() + .filter(|id| state.objects.contains_key(*id)) + .copied(), + ); if live.len() > self.inner.limits.max_root_extents { return Err(StoreError::Limit); } @@ -199,6 +261,10 @@ impl NodeExtentStore { state.bytes -= entry.bytes; } } + // A full deletion batch may leave more orphans. Other physical writes + // cannot interleave this operation. Pin release is checked separately by + // needs_reclaim, so clearing this hint cannot hide a retired reader root. + self.inner.state()?.reclaim_scan_needed = objects.len() == limit - roots.len(); Ok(roots.len() + objects.len()) } diff --git a/crates/cellar-ltx/src/extent/store/workspace.rs b/crates/cellar-ltx/src/extent/store/workspace.rs index 08aced2c2..5e7dc7980 100644 --- a/crates/cellar-ltx/src/extent/store/workspace.rs +++ b/crates/cellar-ltx/src/extent/store/workspace.rs @@ -87,6 +87,9 @@ impl Drop for Directory { /// A clone retains both the directory's complete reservation and the store's /// exclusive lock. The last pin removes files, syncs deletion, then refunds. +/// Creation syncs the child and its parent before returning. This open store +/// already established the parent's ancestor barriers; reopening establishes +/// them again. This owner supplies no image or consensus authority. /// Call only on the owning blocking domain, including dropping the last pin. #[derive(Clone)] pub struct NodeWorkspace(Arc); @@ -164,7 +167,16 @@ impl NodeExtentStore { Err(error) => return Err(StoreError::Io(error)), Ok(_) => return Err(StoreError::Layout), } - crate::host::create_dir_all_durable(self.inner.filesystem.as_ref(), &path)?; + // Open established this inventory's complete ancestor barriers + // and retains its exclusive lock. This validated child name is + // new: persist its contents and parent entry, without walking + // the same durable ancestors for every disposable workspace. + self.inner.filesystem.create_dir_all(&path)?; + if !self.inner.filesystem.symlink_metadata(&path)?.is_dir { + return Err(StoreError::Layout); + } + self.inner.filesystem.sync_all(&path)?; + self.inner.filesystem.sync_all(&self.inner.directory)?; Ok(()) })(); if let Err(error) = created { diff --git a/crates/cellar-ltx/src/generation.rs b/crates/cellar-ltx/src/generation.rs index 9e9e86e48..af524fce5 100644 --- a/crates/cellar-ltx/src/generation.rs +++ b/crates/cellar-ltx/src/generation.rs @@ -11,6 +11,10 @@ use std::ops::{Deref, DerefMut}; use std::path::{Path, PathBuf}; use std::sync::{Arc, Mutex, MutexGuard}; +#[path = "generation_receive.rs"] +mod receive; +pub use receive::GenerationReceive; + pub use crate::generation_accounting::{GenerationAccounting, GenerationResources}; pub use crate::generation_vfs::{open_immutable, ImmutableDatabase, ReadServices}; @@ -199,6 +203,20 @@ pub struct GenerationPin { } impl GenerationPin { + /// Independently inspect and sync this pinned destination, including when + /// its seal is already cached. Run on its joined blocking owner. This + /// verifies physical bytes; it supplies no publication or consensus right. + pub fn verify_and_sync(&self) -> Result<(), GenerationError> { + let mut file = self + .inner + .filesystem + .open(&self.inner.directory.join(final_name(self.id())))?; + apply::inspect(&mut file, &self.state(), None)?; + file.sync_all()?; + self.inner.filesystem.sync_all(&self.inner.directory)?; + Ok(()) + } + pub fn id(&self) -> ImageId { self.image.id } @@ -625,6 +643,11 @@ impl GenerationStore { self.inner.limits.apply } + /// Compare exact open directory owners without performing I/O. + pub fn same_inventory(&self, other: &Self) -> bool { + Arc::ptr_eq(&self.inner, &other.inner) + } + /// Whether this pin belongs to this exact open directory owner. Logical /// image IDs and digests alone do not establish local file ownership. /// This does not select the image as an authoritative retained reference. @@ -644,6 +667,48 @@ impl GenerationStore { self.prepare_chain(id, base, std::slice::from_ref(update)) } + /// Apply and verify the supplied delta even if the destination already + /// exists. A cached result cannot substitute for replay of untrusted LTX. + /// Existing images use an independently admitted anonymous scratch image; + /// existing inventory retention is preserved. New private results are + /// reclaimable once their returned pins are released, like received images. + pub fn prepare_verified( + &self, + id: ImageId, + base: Option<&GenerationPin>, + update: &ImageUpdate<'_>, + ) -> Result { + match self.prepare(id, base, update) { + Ok(pin) => { + self.retire(pin.id())?; + return Ok(pin); + } + Err(GenerationError::Exists) => (), + Err(error) => return Err(error), + } + let pin = self.pin_prepared(id, update.result)?; + let updates = std::slice::from_ref(update); + let peak = apply::peak_chain_bytes(updates, &self.inner.limits.apply)?; + let _scratch = self.inner.snapshot_scratch.acquire(peak, 1)?; + let mut file = self + .inner + .filesystem + .temporary_file(Some(&self.inner.directory))?; + let source = base.map(|pin| self.inner.directory.join(final_name(pin.id()))); + let actual = apply::apply_private_chain( + self.inner.filesystem.as_ref(), + source.as_deref(), + &mut file, + updates, + &self.inner.limits.apply, + )?; + if actual != update.result { + return Err(GenerationError::Identity); + } + pin.verify_and_sync()?; + Ok(pin) + } + /// Captured physical limits; callers can partition replay without relaxing /// the store's aggregate decode, temporary-image or generation budgets. pub fn limits(&self) -> GenerationLimits { @@ -958,6 +1023,27 @@ impl GenerationStore { &self, id: ImageId, expected: ImageState, + ) -> Result { + self.pin_existing(id, expected, true) + } + + /// Reopen an exactly identified prepared image without changing its host + /// retention flag. Cold images are independently verified and synced before + /// returning a pin. This is physical preparation only: the caller must prove + /// the expected image from its committed input and separately publish it. + pub fn pin_prepared( + &self, + id: ImageId, + expected: ImageState, + ) -> Result { + self.pin_existing(id, expected, false) + } + + fn pin_existing( + &self, + id: ImageId, + expected: ImageState, + retain: bool, ) -> Result { expected.validate(&self.inner.limits.apply)?; let token = { @@ -970,7 +1056,7 @@ impl GenerationStore { if image.state != expected { return Err(GenerationError::Identity); } - entry.retained = true; + entry.retained |= retain; return Ok(GenerationPin { inner: self.inner.clone(), image: image.clone(), @@ -980,7 +1066,7 @@ impl GenerationStore { return Err(GenerationError::Missing); } entry.busy = true; - entry.retained = true; + entry.retained |= retain; entry.token.clone() }; let mut reservation = Reservation { diff --git a/crates/cellar-ltx/src/generation_receive.rs b/crates/cellar-ltx/src/generation_receive.rs new file mode 100644 index 000000000..c7a07d6e4 --- /dev/null +++ b/crates/cellar-ltx/src/generation_receive.rs @@ -0,0 +1,105 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Bounded raw-image receive material. A finished image is still private data; +//! only the owning host can publish its complete durable state marker. + +use super::*; +use crate::stream_snapshot::ScratchReservation; + +pub struct GenerationReceive { + store: GenerationStore, + file: HostFile, + id: ImageId, + expected: ImageState, + received: u64, + failed: bool, + _reservation: ScratchReservation, +} + +impl GenerationStore { + /// Reserve the complete anonymous spool before opening it. Filesystem work + /// belongs on the caller's joined blocking owner. Only one image is buffered + /// on disk; append never holds more than the supplied <=64 KiB chunk. + pub fn receive_image( + &self, + id: ImageId, + expected: ImageState, + scratch: &crate::stream_snapshot::ScratchPool, + directory: &Path, + ) -> Result { + expected.validate(&self.inner.limits.apply)?; + if expected.length == 0 || !directory.is_absolute() { + return Err(GenerationError::Identity); + } + let reservation = scratch.acquire(expected.length, 1)?; + let file = self.inner.filesystem.temporary_file(Some(directory))?; + Ok(GenerationReceive { + store: self.clone(), + file, + id, + expected, + received: 0, + failed: false, + _reservation: reservation, + }) + } +} + +impl GenerationReceive { + pub fn append(&mut self, offset: u64, bytes: &[u8]) -> Result<(), GenerationError> { + if self.failed { + return Err(GenerationError::Quarantined); + } + self.failed = true; + let next = offset + .checked_add(bytes.len() as u64) + .ok_or(GenerationError::Limit)?; + if offset != self.received + || bytes.is_empty() + || bytes.len() > 64 * 1024 + || next > self.expected.length + { + return Err(GenerationError::Identity); + } + self.file.write_all_at(offset, bytes)?; + self.received = next; + self.failed = false; + Ok(()) + } + + /// Independently inspect incoming bytes even if the inventory already has + /// this image. The final immutable destination is separately verified/synced. + /// New unselected images are reclaimable after the returned pin is released; + /// existing host retention is preserved. + pub fn finish(mut self) -> Result { + if self.failed || self.received != self.expected.length { + return Err(GenerationError::Identity); + } + apply::inspect(&mut self.file, &self.expected, None)?; + self.file.sync_all()?; + match self.store.pin_prepared(self.id, self.expected) { + Ok(pin) => { + // pin_prepared may reuse an in-memory seal. A transfer must + // independently inspect the existing destination as well as + // the incoming spool, while the pin prevents reclamation. + pin.verify_and_sync()?; + return Ok(pin); + } + Err(GenerationError::Missing) => {} + Err(error) => return Err(error), + } + let pin = self.store.prepare_private( + self.id, + self.expected.length, + None, + None, + |destination, _| { + apply::inspect(&mut self.file, &self.expected, Some(destination))?; + apply::inspect(destination, &self.expected, None)?; + Ok(self.expected) + }, + )?; + self.store.retire(pin.id())?; + Ok(pin) + } +} diff --git a/crates/cellar-ltx/tests/extent_apply.rs b/crates/cellar-ltx/tests/extent_apply.rs index bc6680019..41787370a 100644 --- a/crates/cellar-ltx/tests/extent_apply.rs +++ b/crates/cellar-ltx/tests/extent_apply.rs @@ -458,3 +458,60 @@ fn byte_closure_and_reopened_inventory_cannot_substitute_for_sqlite_image_valida }; fixture.query(&recovered, 7); } + +#[test] +fn transient_images_keep_live_readers_and_preserve_existing_retention() { + let (fixture, before) = Fixture::new(4096, 4096); + let mut buffer = vec![0; EXTENT_BYTES as usize]; + let base = fixture.seal(&before, false, &mut buffer); + fixture.store.retain_only(&[base.reference()]).unwrap(); + let after = fixture.mutate("UPDATE counter SET value=8"); + let encoded = delta(&before, &after, 4096, 2, false); + let update = fixture.update(&base, &encoded, &after, 2, false); + let transient = fixture + .store + .prepare_transient_image_root(Some(&base), &update, &ApplyLimits::default(), &mut buffer) + .unwrap() + .image; + let selected = transient.reference(); + assert_eq!( + fixture.store.reclaim(4096).unwrap(), + 0, + "live reader closure was collected" + ); + fixture.query(&transient, 8); + fixture.query(&base, 7); + drop(transient); + assert!( + fixture.store.reclaim(4096).unwrap() > 0, + "fresh unpinned root retained forever" + ); + assert!(matches!( + fixture.store.open_root(selected), + Err(StoreError::Missing) + )); + fixture.query(&base, 7); + + // An earlier native marker may retain exactly these bytes. Retrying the + // transient preparation must not withdraw that separate host reference. + let retained = fixture + .store + .prepare_image_root(Some(&base), &update, &ApplyLimits::default(), &mut buffer) + .unwrap() + .image; + assert_eq!(retained.reference(), selected); + drop(retained); + let reused = fixture + .store + .prepare_transient_image_root(Some(&base), &update, &ApplyLimits::default(), &mut buffer) + .unwrap() + .image; + fixture.query(&reused, 8); + drop(reused); + assert_eq!( + fixture.store.reclaim(4096).unwrap(), + 0, + "existing retention was withdrawn" + ); + assert!(fixture.store.open_root(selected).is_ok()); +} diff --git a/crates/cellar-ltx/tests/extent_store.rs b/crates/cellar-ltx/tests/extent_store.rs index 740f27957..74039072e 100644 --- a/crates/cellar-ltx/tests/extent_store.rs +++ b/crates/cellar-ltx/tests/extent_store.rs @@ -150,6 +150,180 @@ impl Fixture { } } +#[test] +fn warm_image_seals_recheck_bytes_and_reuse_only_completed_root_barriers() { + let f = Fixture::new(0); + let store = f.store(); + f.fill(&store); + let mut buffer = vec![0; EXTENT_BYTES as usize]; + let image = store + .seal_image_root(f.manifest.clone(), &mut buffer) + .unwrap(); + let reference = image.reference(); + let reads = f.files.metrics.read_bytes.load(Ordering::Acquire); + let files = f.files.metrics.file_sync_calls.load(Ordering::Acquire); + let directories = f.files.metrics.dir_sync_calls.load(Ordering::Acquire); + let again = store + .seal_image_root(f.manifest.clone(), &mut buffer) + .unwrap(); + assert_eq!(again.reference(), reference); + let required = f.manifest.encode().unwrap().len() + + f.objects + .iter() + .map(|(id, _)| id.length() as usize) + .sum::(); + assert!(f.files.metrics.read_bytes.load(Ordering::Acquire) - reads >= required); + assert_eq!( + f.files.metrics.file_sync_calls.load(Ordering::Acquire), + files + ); + assert_eq!( + f.files.metrics.dir_sync_calls.load(Ordering::Acquire), + directories + ); + + // Even a held seal cannot hide damaged root bytes. A failed recheck also + // invalidates the persistence observation before this fixture repairs bytes. + let path = f.root.path().join("store").join(format!( + "r-{}", + reference + .root() + .bytes() + .iter() + .map(|b| format!("{b:02x}")) + .collect::() + )); + let original = std::fs::read(&path).unwrap(); + let mut corrupt = original.clone(); + corrupt[0] ^= 0x80; + std::fs::write(&path, corrupt).unwrap(); + assert!(store + .seal_image_root(f.manifest.clone(), &mut buffer) + .is_err()); + std::fs::write(path, original).unwrap(); + f.files.metrics.fail_sync.store(true, Ordering::Release); + assert!(store + .seal_image_root(f.manifest.clone(), &mut buffer) + .is_err()); + store + .seal_image_root(f.manifest.clone(), &mut buffer) + .unwrap(); +} + +#[test] +fn reopened_root_seals_require_both_persistence_barriers_after_metadata_reads() { + let f = Fixture::new(0); + let store = f.store(); + f.fill(&store); + let mut buffer = vec![0; EXTENT_BYTES as usize]; + let image = store + .seal_image_root(f.manifest.clone(), &mut buffer) + .unwrap(); + let reference = image.reference(); + drop(image); + drop(store); + let reopened = f.store(); + let read = reopened.open_root(reference).unwrap(); + for (id, _) in &f.objects { + reopened.verify_extent(*id).unwrap(); + } + f.files.metrics.fail_sync.store(true, Ordering::Release); + assert!(reopened + .seal_image_root(f.manifest.clone(), &mut buffer) + .is_err()); + f.files.metrics.fail_dir_sync.store(true, Ordering::Release); + assert!(reopened + .seal_image_root(f.manifest.clone(), &mut buffer) + .is_err()); + let image = reopened + .seal_image_root(f.manifest.clone(), &mut buffer) + .unwrap(); + assert_eq!(read.reference(), image.reference()); +} + +#[test] +fn collection_hint_is_idle_after_marking_and_notices_reader_release_without_io() { + let f = Fixture::new(0); + let store = f.store(); + store.retain_only(&[]).unwrap(); + let receiving = store.stage_transient_root(f.manifest.clone()).unwrap(); + f.fill(&store); + assert_eq!(store.reclaim(32).unwrap(), 0); + let before = f.files.metrics.read_bytes.load(Ordering::Acquire); + assert!(!store.needs_reclaim().unwrap()); + assert_eq!(f.files.metrics.read_bytes.load(Ordering::Acquire), before); + drop(receiving); + assert!(store.needs_reclaim().unwrap()); + assert_eq!(f.files.metrics.read_bytes.load(Ordering::Acquire), before); + assert_eq!(store.reclaim(1).unwrap(), 1); + assert!( + store.needs_reclaim().unwrap(), + "synced root deletion hid its orphan objects" + ); + assert_eq!(store.reclaim(32).unwrap(), f.objects.len()); + assert!(!store.needs_reclaim().unwrap()); + let bytes = vec![9; 4096]; + store + .put_extent_bytes(ExtentId::of(&bytes).unwrap(), &bytes) + .unwrap(); + assert!( + store.needs_reclaim().unwrap(), + "a new unreferenced object did not request marking" + ); + assert_eq!(store.reclaim(32).unwrap(), 1); + assert!(!store.needs_reclaim().unwrap()); +} + +#[test] +fn collection_marks_local_objects_while_a_pinned_receive_is_still_missing_dependencies() { + let f = Fixture::new(EXTENT_BYTES as usize + 4096); + let store = f.store(); + store.retain_only(&[]).unwrap(); + let receiving = store.stage_transient_root(f.manifest.clone()).unwrap(); + let orphan = vec![9; 4096]; + store + .put_extent_bytes(ExtentId::of(&orphan).unwrap(), &orphan) + .unwrap(); + assert_eq!(store.reclaim(32).unwrap(), 1); + assert_eq!(store.usage().unwrap().roots, 1); + assert!(!store.needs_reclaim().unwrap()); + f.fill(&store); + assert!(store.needs_reclaim().unwrap()); + assert_eq!( + store.reclaim(32).unwrap(), + 0, + "partial receive lost an imported dependency" + ); + drop(receiving); + assert_eq!(store.reclaim(32).unwrap(), f.objects.len() + 1); +} + +#[test] +fn collection_memory_admission_refuses_overflow_without_allocating_an_inventory() { + let f = Fixture::new(0); + let mut limits = f.limits(); + limits.max_roots = usize::MAX; + let store = NodeExtentStore::open( + f.root.path().join("oversized-control"), + f.files.clone(), + limits, + ) + .unwrap(); + assert!(matches!( + store.reclaim_working_bytes(16), + Err(StoreError::Limit) + )); + assert!(matches!( + store.reclaim_working_bytes(0), + Err(StoreError::Limit) + )); + assert!(matches!( + store.reclaim_working_bytes(4097), + Err(StoreError::Limit) + )); + assert_eq!(store.usage().unwrap().bytes, 0); +} + #[test] fn reopen_uses_metadata_only_inventory_then_verifies_sqlite_extents_on_demand() { let f = Fixture::new(9 * 1024 * 1024); diff --git a/crates/cellar-ltx/tests/extent_workspace_admission.rs b/crates/cellar-ltx/tests/extent_workspace_admission.rs index d1815e6dd..ffcfdc56a 100644 --- a/crates/cellar-ltx/tests/extent_workspace_admission.rs +++ b/crates/cellar-ltx/tests/extent_workspace_admission.rs @@ -94,6 +94,68 @@ fn immutable_objects_and_workspaces_share_admission_before_source_or_directory_i assert!(!other.owns_workspace(&next)); } +#[test] +fn workspace_creation_reuses_only_the_open_inventory_ancestor_barriers() { + let directory = tempfile::tempdir().unwrap(); + let files = files(directory.path()); + let path = directory.path().join("nested/store"); + let store = NodeExtentStore::open(path.clone(), files.clone(), Default::default()).unwrap(); + let before = files.metrics.dir_sync_calls.load(Ordering::Acquire); + let workspace = store.create_workspace([1; 32], bounds()).unwrap(); + assert_eq!( + files.metrics.dir_sync_calls.load(Ordering::Acquire) - before, + 2, + "workspace creation must sync its child and owned parent, without resyncing durable ancestors" + ); + assert!(workspace.directory().is_dir()); + let retained = workspace.clone(); + drop(workspace); + assert_eq!(store.usage().unwrap().workspaces, 1); + drop(retained); + assert_eq!(store.usage().unwrap().workspaces, 0); + drop(store); + // Restart must establish its own ancestor barriers before issuing another + // workspace. A pathname or an observation from a former owner is no proof. + let before = files.metrics.dir_sync_calls.load(Ordering::Acquire); + let store = NodeExtentStore::open(path, files.clone(), Default::default()).unwrap(); + assert!(files.metrics.dir_sync_calls.load(Ordering::Acquire) > before); + let before = files.metrics.dir_sync_calls.load(Ordering::Acquire); + drop(store.create_workspace([2; 32], bounds()).unwrap()); + assert_eq!( + files.metrics.dir_sync_calls.load(Ordering::Acquire) - before, + 3, + "creation barriers and the final deletion barrier must remain" + ); +} + +#[test] +fn workspace_child_and_parent_sync_failures_keep_the_full_reservation() { + for barrier in [1, 2] { + let directory = tempfile::tempdir().unwrap(); + let files = files(directory.path()); + let path = directory.path().join("store"); + let store = NodeExtentStore::open(path.clone(), files.clone(), Default::default()).unwrap(); + files + .metrics + .fail_dir_sync_countdown + .store(barrier, Ordering::Release); + assert!(store.create_workspace([1; 32], bounds()).is_err()); + let usage = store.usage().unwrap(); + assert!(usage.quarantined); + assert_eq!(usage.workspaces, 1); + assert_eq!(usage.workspace_bytes, bounds().reservation().unwrap()); + assert!(matches!( + store.create_workspace([2; 32], bounds()), + Err(StoreError::Quarantined) + )); + drop(store); + let recovered = NodeExtentStore::open(path, files, Default::default()).unwrap(); + assert_eq!(recovered.usage().unwrap().workspace_bytes, 0); + assert_eq!(recovered.usage().unwrap().workspaces, 0); + assert!(!recovered.usage().unwrap().quarantined); + } +} + #[test] fn failed_cleanup_or_deletion_sync_retains_the_whole_charge_until_reopen() { for fault in 0..3 { diff --git a/crates/cellar-ltx/tests/private_clone.rs b/crates/cellar-ltx/tests/private_clone.rs index 10bcae2aa..376207f74 100644 --- a/crates/cellar-ltx/tests/private_clone.rs +++ b/crates/cellar-ltx/tests/private_clone.rs @@ -178,3 +178,76 @@ fn corrupt_clone_and_ambiguous_creation_never_publish_or_refund_uncertain_capaci .is_err()); assert_eq!(std::fs::read(original).unwrap(), bytes); } + +#[test] +fn prepared_image_retry_verifies_identity_without_restoring_retention() { + let root = tempfile::tempdir().unwrap(); + let original = root.path().join("source.sqlite"); + let bytes = source(&original); + let directory = root.path().join("images"); + let id = ImageId::new([1; 16]).unwrap(); + let expected = state(&bytes, 1); + let store = GenerationStore::open( + directory.clone(), + Arc::new(DirectFileSystem), + GenerationLimits::default(), + ) + .unwrap(); + drop(store.prepare_image_file(id, &original, expected).unwrap()); + store.retire(id).unwrap(); + assert_eq!(store.usage().unwrap().retained, 0); + let pin = store.pin_prepared(id, expected).unwrap(); + assert_eq!(store.usage().unwrap().retained, 0); + assert_eq!(store.reclaim(10).unwrap().files, 0); + assert!(matches!( + store.pin_prepared( + id, + ImageState { + txid: TXID(2), + ..expected + } + ), + Err(GenerationError::Identity) + )); + drop(pin); + drop(store); + // Restart must verify the file, not infer publication from directory order. + let store = GenerationStore::open( + directory.clone(), + Arc::new(DirectFileSystem), + GenerationLimits::default(), + ) + .unwrap(); + store.retire(id).unwrap(); + let pin = store.pin_prepared(id, expected).unwrap(); + assert_eq!(pin.state(), expected); + assert_eq!(store.usage().unwrap().retained, 0); + drop(pin); + assert!(store.reclaim(10).unwrap().files > 0); + assert!(matches!( + store.pin_prepared(id, expected), + Err(GenerationError::Missing) + )); + drop(store); + let store = GenerationStore::open( + directory.clone(), + Arc::new(DirectFileSystem), + GenerationLimits::default(), + ) + .unwrap(); + drop(store.prepare_image_file(id, &original, expected).unwrap()); + drop(store); + let image = std::fs::read_dir(&directory) + .unwrap() + .map(|entry| entry.unwrap().path()) + .find(|path| path.extension().is_some_and(|ext| ext == "sqlite")) + .unwrap(); + std::fs::write(image, b"corrupt immutable image").unwrap(); + let store = GenerationStore::open( + directory, + Arc::new(DirectFileSystem), + GenerationLimits::default(), + ) + .unwrap(); + assert!(store.pin_prepared(id, expected).is_err()); +} diff --git a/crates/cellar-ltx/tests/support/extent_buffered_import.rs b/crates/cellar-ltx/tests/support/extent_buffered_import.rs index 766753f3a..9e8dade9e 100644 --- a/crates/cellar-ltx/tests/support/extent_buffered_import.rs +++ b/crates/cellar-ltx/tests/support/extent_buffered_import.rs @@ -53,6 +53,33 @@ fn complete_input_uses_one_native_window_and_preserves_bounded_provider_fallback } } +#[test] +fn complete_input_checks_source_bytes_even_when_a_valid_local_object_exists() { + let f = Fixture::new(0); + let store = f.store(); + let bytes = vec![7; 4096]; + let id = ExtentId::of(&bytes).unwrap(); + store.put_extent_bytes(id, &bytes).unwrap(); + let before = store.usage().unwrap(); + let writes = f.files.metrics.random_write_calls.load(Ordering::Acquire); + let reads = f.files.metrics.read_bytes.load(Ordering::Acquire); + let corrupt = vec![8; bytes.len()]; + assert!( + matches!( + store.put_extent_bytes(id, &corrupt), + Err(StoreError::Integrity) + ), + "a warm destination hid invalid supplied source bytes" + ); + assert_eq!(store.usage().unwrap(), before); + assert_eq!( + f.files.metrics.random_write_calls.load(Ordering::Acquire), + writes + ); + assert_eq!(f.files.metrics.read_bytes.load(Ordering::Acquire), reads); + store.put_extent_bytes(id, &bytes).unwrap(); +} + #[test] fn complete_input_refuses_wrong_length_and_capacity_before_writes() { let f = Fixture::new(0); diff --git a/crates/cellar-ltx/tests/support/stream_snapshot_files.rs b/crates/cellar-ltx/tests/support/stream_snapshot_files.rs index 52b9fe6c2..f28d13f02 100644 --- a/crates/cellar-ltx/tests/support/stream_snapshot_files.rs +++ b/crates/cellar-ltx/tests/support/stream_snapshot_files.rs @@ -33,6 +33,7 @@ pub struct Metrics { pub corrupt_read_countdown: AtomicUsize, pub create_mode: AtomicUsize, pub fail_dir_sync: AtomicBool, + pub fail_dir_sync_countdown: AtomicUsize, pub fail_remove: AtomicBool, pub fail_dir_remove_after: AtomicBool, pub refuse_dir_remove_with_handles: AtomicBool, @@ -304,7 +305,12 @@ impl FileSystem for BoundedFiles { fn sync_all(&self, path: &Path) -> io::Result<()> { if DirectFileSystem.metadata(path)?.is_dir { self.metrics.dir_sync_calls.fetch_add(1, Ordering::AcqRel); - if self.metrics.fail_dir_sync.swap(false, Ordering::AcqRel) { + let countdown = self.metrics.fail_dir_sync_countdown.fetch_update( + Ordering::AcqRel, + Ordering::Acquire, + |n| n.checked_sub(1), + ); + if self.metrics.fail_dir_sync.swap(false, Ordering::AcqRel) || countdown == Ok(1) { return Err(io::Error::other("injected directory sync failure")); } } diff --git a/crates/cellar/Cargo.toml b/crates/cellar/Cargo.toml index 7607a0726..aecee43af 100644 --- a/crates/cellar/Cargo.toml +++ b/crates/cellar/Cargo.toml @@ -79,6 +79,10 @@ path = "../../tools/perf/raft_node.rs" name = "cellar-perf-baseline" path = "../../tools/perf/baseline_node.rs" +[[bin]] +name = "cellar-perf-cell-group" +path = "../../tools/multi-raft/node/main.rs" + [[bin]] name = "cellar-perf-storage" path = "../../tools/perf/storage_operations.rs" diff --git a/crates/cellar/application/consensus/admission.rs b/crates/cellar/application/consensus/admission.rs index d79b12a5f..8fa712556 100644 --- a/crates/cellar/application/consensus/admission.rs +++ b/crates/cellar/application/consensus/admission.rs @@ -15,6 +15,14 @@ pub(crate) enum Access { Cleanup, } +/// The rejection is captured while holding the registration barrier; callers +/// must not infer a cause by resampling flags after the barrier is released. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) enum RegistrationRefusal { + Closed, + Maintenance, +} + #[derive(Default)] pub(crate) struct Admission { draining: AtomicBool, @@ -31,15 +39,25 @@ impl Admission { &self, access: Access, ) -> Result, CommandFailure> { - let guard = self.gate.lock().map_err(|_| CommandFailure::Unavailable)?; - if self.is_draining() && !matches!(access, Access::Cleanup) - || matches!(access, Access::Application) - && self - .maintenance - .get() - .is_some_and(|gate| gate.raft_closed()) + self.registration_reason(access) + .map_err(|_| CommandFailure::Unavailable) + } + + pub(crate) fn registration_reason( + &self, + access: Access, + ) -> Result, RegistrationRefusal> { + let guard = self.gate.lock().map_err(|_| RegistrationRefusal::Closed)?; + if self.is_draining() && !matches!(access, Access::Cleanup) { + return Err(RegistrationRefusal::Closed); + } + if matches!(access, Access::Application) + && self + .maintenance + .get() + .is_some_and(|gate| gate.raft_closed()) { - return Err(CommandFailure::Unavailable); + return Err(RegistrationRefusal::Maintenance); } Ok(guard) } diff --git a/crates/cellar/application/consensus/admission_tests.rs b/crates/cellar/application/consensus/admission_tests.rs index 1dafde9f3..6c61982c2 100644 --- a/crates/cellar/application/consensus/admission_tests.rs +++ b/crates/cellar/application/consensus/admission_tests.rs @@ -103,6 +103,10 @@ fn maintenance_reopens_application_without_resetting_permanent_shutdown() { drop(admission.registration(Access::Application).unwrap()); gate.seal_raft(&selector).unwrap(); admission.barrier().unwrap(); + assert!(matches!( + admission.registration_reason(Access::Application), + Err(RegistrationRefusal::Maintenance) + )); assert!(matches!( admission.registration(Access::Application), Err(CommandFailure::Unavailable) @@ -120,6 +124,10 @@ fn maintenance_reopens_application_without_resetting_permanent_shutdown() { admission.seal().unwrap(); gate.fence(&next).unwrap(); assert!(admission.is_draining()); + assert!(matches!( + admission.registration_reason(Access::Application), + Err(RegistrationRefusal::Closed) + )); assert!(admission.application_paused()); assert!(!admission.maintenance_paused()); assert!(matches!( diff --git a/crates/cellar/application/consensus/class.rs b/crates/cellar/application/consensus/class.rs index d8197e5f3..bd41dd536 100644 --- a/crates/cellar/application/consensus/class.rs +++ b/crates/cellar/application/consensus/class.rs @@ -2,9 +2,10 @@ //! Compiled, synchronous Raft operations with no ordinary mutable cell factory. -use super::{CommandId, CommittedVersion, Durability, GroupScope}; +use super::{CommandId, CommittedVersion, Durability}; +use crate::raft::version::CellVersionObservation; use crate::storage::{CommandInputs, CommandTransaction, ImmutableQuery}; -use cellar_logic::NamespaceMode; +use cellar_logic::{cell_group::CellHistory, NamespaceMode}; use sha2::{Digest, Sha256}; use std::{ collections::{BTreeMap, BTreeSet}, @@ -14,7 +15,7 @@ use std::{ thread_local! { static CALLBACK_ACTIVE: std::cell::Cell = const { std::cell::Cell::new(false) }; } -pub(super) fn callback_active() -> bool { +pub(crate) fn callback_active() -> bool { CALLBACK_ACTIVE.with(std::cell::Cell::get) } @@ -105,7 +106,7 @@ impl std::error::Error for RegistrationError {} /// fn socket(context: &CommandContext<'_>) { let _ = context.websocket_pair(); } /// ``` pub struct CommandContext<'a> { - scope: &'a GroupScope, + scope: &'a CellHistory, command: CommandId, transaction: &'a CommandTransaction<'a>, blobs: Option, @@ -118,7 +119,7 @@ pub struct CommandContext<'a> { } impl<'a> CommandContext<'a> { - pub fn identity(&self) -> &GroupScope { + pub fn identity(&self) -> &CellHistory { self.scope } @@ -210,10 +211,21 @@ impl CommandContext<'_> { } } +/// Exact observation supplied by the owning query path. A group cut carries no +/// cell mutation sequence and must never be relabeled as a cell version. +#[derive(Clone, Copy, Debug)] +pub enum QueryVersion<'a> { + Cell(&'a CellVersionObservation), + /// Temporary single-cell host boundary, removed with that host in CG-16. + /// This is an in-memory observation, not an old-format recovery reader. + GroupCut(&'a CommittedVersion), +} + /// A pinned, read-only committed image. The host establishes consistency before /// constructing this context. Query code cannot open or migrate writable state. pub struct QueryContext<'a> { - version: &'a CommittedVersion, + identity: &'a CellHistory, + version: QueryVersion<'a>, database: &'a ImmutableQuery, blobs: Option, outbox: Option<&'a super::RaftOutbox>, @@ -229,11 +241,11 @@ impl QueryContext<'_> { ) } - pub fn identity(&self) -> &GroupScope { - &self.version.scope + pub fn identity(&self) -> &CellHistory { + self.identity } - pub fn version(&self) -> &CommittedVersion { + pub fn version(&self) -> QueryVersion<'_> { self.version } @@ -261,6 +273,13 @@ type CommandCallback = dyn for<'a> Fn(&CommandContext<'a>, &[u8]) -> anyhow::Result> + Send + Sync; type QueryCallback = dyn for<'a> Fn(&QueryContext<'a>, &[u8]) -> anyhow::Result> + Send + Sync; +type InitializerCallback = + dyn Fn(&CellHistory, &CommandTransaction<'_>) -> anyhow::Result<()> + Send + Sync; + +struct Initializer { + digest: [u8; 32], + callback: Arc, +} struct Command { digest: [u8; 32], @@ -286,6 +305,7 @@ pub struct RaftClass { base_schema_digest: [u8; 32], schema_digest: [u8; 32], commands: BTreeMap, + initializer: Option, system_commands: BTreeMap, queries: BTreeMap, blobs: Option, @@ -325,6 +345,7 @@ impl RaftClass { base_schema_digest: schema_digest, schema_digest, commands: BTreeMap::new(), + initializer: None, system_commands: BTreeMap::new(), queries: BTreeMap::new(), blobs: None, @@ -361,6 +382,48 @@ impl RaftClass { self.base_schema_digest } + /// Register schema initialization for shared-group enrollment. The digest + /// participates in voting compatibility. Initialization runs once on a + /// private SQL transaction; only its committed creation record is success. + pub fn initializer( + mut self, + digest: [u8; 32], + callback: impl Fn(&CellHistory, &CommandTransaction<'_>) -> anyhow::Result<()> + + Send + + Sync + + 'static, + ) -> Result { + if digest == [0; 32] || self.initializer.is_some() { + return Err(RegistrationError::Invalid("initializer registration")); + } + self.initializer = Some(Initializer { + digest, + callback: Arc::new(callback), + }); + self.bind_command_manifest(); + Ok(self) + } + + pub(crate) fn require_initializer(&self) -> Result<(), RegistrationError> { + self.initializer + .as_ref() + .map(|_| ()) + .ok_or(RegistrationError::Invalid("missing initializer")) + } + + pub(crate) fn invoke_initializer( + &self, + cell: &CellHistory, + transaction: &CommandTransaction<'_>, + ) -> anyhow::Result<()> { + let initializer = self + .initializer + .as_ref() + .ok_or(RegistrationError::Invalid("missing initializer"))?; + let _guard = CallbackGuard::enter()?; + (initializer.callback)(cell, transaction) + } + /// Opt in to bounded immutable bytes replicated with the cell. This changes /// the class identity and requires a fresh compatible namespace. pub fn immutable_blobs( @@ -454,7 +517,9 @@ impl RaftClass { self.outbox.as_ref() } - pub(crate) fn validate_host( + /// Shared groups support initializers. All other declared capabilities + /// retain the same execution, receipt, scheduler and archive bounds. + pub(crate) fn validate_shared_host( &self, host: &crate::raft::host::RaftHostOptions, ) -> anyhow::Result<()> { @@ -585,7 +650,7 @@ impl RaftClass { pub(crate) fn validate_command_identity( &self, - scope: &GroupScope, + scope: &CellHistory, contract: &OperationContract, id: CommandId, durability: Durability, @@ -690,10 +755,6 @@ impl RaftClass { !self.system_commands.is_empty() } - pub(crate) fn has_queries(&self) -> bool { - !self.queries.is_empty() - } - pub(crate) fn validate_system_operation( &self, contract: &OperationContract, @@ -735,7 +796,7 @@ impl RaftClass { pub(crate) fn invoke_command( &self, contract: &OperationContract, - scope: &GroupScope, + scope: &CellHistory, command: CommandId, durability: Durability, transaction: &CommandTransaction<'_>, @@ -754,7 +815,7 @@ impl RaftClass { pub(crate) fn invoke_system_command( &self, contract: &OperationContract, - scope: &GroupScope, + scope: &CellHistory, command: CommandId, durability: Durability, transaction: &CommandTransaction<'_>, @@ -773,7 +834,7 @@ impl RaftClass { fn invoke_registered_command( &self, registration: RegisteredCommand<'_>, - scope: &GroupScope, + scope: &CellHistory, command: CommandId, durability: Durability, transaction: &CommandTransaction<'_>, @@ -815,12 +876,46 @@ impl RaftClass { ) } - pub(crate) fn invoke_query( + pub(crate) fn invoke_group_query( &self, contract: &OperationContract, version: &CommittedVersion, database: &ImmutableQuery, input: &[u8], + ) -> anyhow::Result> { + let identity = CellHistory::try_from(&version.scope)?; + self.invoke_query( + contract, + &identity, + QueryVersion::GroupCut(version), + database, + input, + ) + } + + pub(crate) fn invoke_cell_query( + &self, + contract: &OperationContract, + observation: &CellVersionObservation, + database: &ImmutableQuery, + input: &[u8], + ) -> anyhow::Result> { + self.invoke_query( + contract, + &observation.version.history, + QueryVersion::Cell(observation), + database, + input, + ) + } + + fn invoke_query( + &self, + contract: &OperationContract, + identity: &CellHistory, + version: QueryVersion<'_>, + database: &ImmutableQuery, + input: &[u8], ) -> anyhow::Result> { self.validate_operation(contract)?; anyhow::ensure!( @@ -836,6 +931,7 @@ impl RaftClass { let _guard = CallbackGuard::enter()?; callback( &QueryContext { + identity, version, database, blobs: self.blobs, @@ -849,8 +945,12 @@ impl RaftClass { fn bind_command_manifest(&mut self) { let mut hash = Sha256::new(); - hash.update(b"cellar-raft-class-1"); + hash.update(b"cellar-cell-class-1"); hash.update(self.base_schema_digest); + if let Some(initializer) = &self.initializer { + hash.update(b"initializer-1"); + hash.update(initializer.digest); + } if let Some(limits) = self.blobs { hash.update(b"immutable-blobs-1"); for size in [ @@ -924,3 +1024,7 @@ pub(super) fn validate_name(name: &str) -> Result<(), RegistrationError> { } Ok(()) } + +#[cfg(test)] +#[path = "class_initializer_tests.rs"] +mod initializer_tests; diff --git a/crates/cellar/application/consensus/class_initializer_tests.rs b/crates/cellar/application/consensus/class_initializer_tests.rs new file mode 100644 index 000000000..ed945339e --- /dev/null +++ b/crates/cellar/application/consensus/class_initializer_tests.rs @@ -0,0 +1,31 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; + +#[test] +fn initializer_contract_binds_schema_and_admits_shared_host() -> anyhow::Result<()> { + let base = || RaftClass::new("Counter", 1, [81; 32]); + let ordinary = base()?.command("Add", [82; 32], |_, _| Ok(vec![]))?; + let initial = + base()? + .initializer([90; 32], |_, _| Ok(()))? + .command("Add", [82; 32], |_, _| Ok(vec![]))?; + let reordered = base()? + .command("Add", [82; 32], |_, _| Ok(vec![]))? + .initializer([90; 32], |_, _| Ok(()))?; + let different = + base()? + .initializer([91; 32], |_, _| Ok(()))? + .command("Add", [82; 32], |_, _| Ok(vec![]))?; + assert_eq!(initial.schema_digest(), reordered.schema_digest()); + assert_ne!(initial.schema_digest(), ordinary.schema_digest()); + assert_ne!(initial.schema_digest(), different.schema_digest()); + ordinary.validate_shared_host(&Default::default())?; + initial.validate_shared_host(&Default::default())?; + assert!(base()?.initializer([0; 32], |_, _| Ok(())).is_err()); + assert!(base()? + .initializer([90; 32], |_, _| Ok(()))? + .initializer([90; 32], |_, _| Ok(())) + .is_err()); + Ok(()) +} diff --git a/crates/cellar/application/consensus/client.rs b/crates/cellar/application/consensus/client.rs index 56e0ebe65..280d2bb4d 100644 --- a/crates/cellar/application/consensus/client.rs +++ b/crates/cellar/application/consensus/client.rs @@ -23,9 +23,6 @@ use tokio::sync::{mpsc, oneshot, Notify}; #[path = "client_fleet.rs"] mod fleet; pub(crate) use fleet::FleetRequest; -#[path = "client_execution.rs"] -mod execution; -pub(crate) use execution::{LocalFleetActionCall, PeerFleetActionCall}; #[path = "client_bindings.rs"] mod bindings; @@ -33,8 +30,7 @@ mod bindings; mod node; pub(crate) use bindings::Bindings; use node::send_failure; -pub(crate) use node::{ControlRequest, NodeKind, NodeRequest}; -pub(crate) use node::{PeerFleetRequest, PeerStopRequest}; +pub(crate) use node::{ControlRequest, NodeKind}; /// A compare-and-set admission request for a preconfigured fresh learner. #[derive(Clone, Debug)] @@ -93,64 +89,6 @@ pub struct CommandReceipt { pub durability: Durability, } -#[derive(Clone, Debug, PartialEq, Eq)] -pub enum QueryConsistency { - Stale, - AtLeast(VersionToken), - Linearizable, -} - -#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] -pub enum StaleReadPolicy { - /// Refuse after the configured deadline for observing a current source. - #[default] - RecentSource, - /// Permit a verified historical image during disconnection. This supplies - /// no wall-clock staleness bound and does not strengthen its consistency. - DisconnectedHistorical, -} - -#[derive(Clone, Debug, Default)] -pub enum QueryRouting { - #[default] - Local, - /// Try at most eight distinct members in priority order, then optionally - /// one untried leader hint. Each attempt retains the same consistency, - /// token and captured input under the original invocation deadline. - Replicas { - members: Vec, - attempt_timeout: Duration, - leader_fallback: bool, - }, -} - -#[derive(Clone, Debug)] -pub struct QueryOptions { - pub consistency: QueryConsistency, - pub timeout: Duration, - pub stale_policy: StaleReadPolicy, - pub routing: QueryRouting, -} - -impl Default for QueryOptions { - fn default() -> Self { - Self { - consistency: QueryConsistency::Stale, - timeout: Duration::from_secs(5), - stale_policy: StaleReadPolicy::RecentSource, - routing: QueryRouting::Local, - } - } -} - -#[derive(Clone, Debug, PartialEq, Eq)] -pub struct QueryResult { - pub result: Vec, - pub version: CommittedVersion, - pub version_token: VersionToken, - pub served_by: MemberId, -} - /// Discovery/readiness observation, never a grant of mutation authority. #[derive(Clone, Debug, PartialEq, Eq)] pub struct CellStatus { @@ -747,31 +685,6 @@ impl RaftCellHandle { } } - /// Queue a read immediately; the returned version names its pinned image. - /// Dropping the returned future cancels the waiter. - pub fn query( - &self, - contract: OperationContract, - input: Vec, - options: QueryOptions, - ) -> impl Future> + Send + 'static { - let timeout = options.timeout; - let call = self.call( - Kind::Query { - contract, - input, - options, - }, - timeout, - ); - async move { - match call.await? { - Response::Query(result) => Ok(result), - _ => Err(CommandFailure::CorruptState), - } - } - } - /// Queue a readiness observation immediately. Readiness may change before /// a subsequent command; the observation grants no write authority. pub fn status( @@ -858,7 +771,8 @@ impl RaftCellHandle { self.binding .class .validate_command_identity( - &self.binding.scope, + &cellar_logic::cell_group::CellHistory::try_from(&self.binding.scope) + .map_err(|_| CommandFailure::CorruptState)?, contract, options.id, options.durability, @@ -883,7 +797,12 @@ impl RaftCellHandle { .validate_system_operation(contract) .is_err() || options.id.client - != crate::raft::effects::system_client(&self.binding.scope)? + != crate::raft::effects::system_client( + &cellar_logic::cell_group::CellHistory::try_from( + &self.binding.scope, + ) + .map_err(|_| CommandFailure::CorruptState)?, + )? { return Err(CommandFailure::UnsupportedCapability); } @@ -891,50 +810,6 @@ impl RaftCellHandle { return Err(CommandFailure::Overloaded); } } - Kind::Query { - contract, - input, - options, - } => { - if matches!(options.routing, QueryRouting::Local) { - if self.binding.member().is_none() { - return Err(CommandFailure::UnsupportedCapability); - } - self.validate(contract, OperationKind::Query, input)?; - } else { - // A voter may forward a query whose implementation is - // only installed on another replica. Destination code - // compatibility remains mandatory before execution. - if contract.kind != OperationKind::Query - || contract.digest == [0; 32] - || super::class::validate_name(&contract.operation).is_err() - { - return Err(CommandFailure::UnsupportedCapability); - } - if input.len() > self.handle.max_input_bytes { - return Err(CommandFailure::Overloaded); - } - } - if let QueryRouting::Replicas { - members, - attempt_timeout, - .. - } = &options.routing - { - if members.is_empty() - || members.len() > 8 - || members - .iter() - .collect::>() - .len() - != members.len() - || attempt_timeout.as_millis() == 0 - || *attempt_timeout > Duration::from_secs(60) - { - return Err(CommandFailure::UnsupportedConsistency); - } - } - } Kind::Inspect => (), Kind::InspectPeer(_) => { if timeout > Duration::from_secs(60) { @@ -981,14 +856,6 @@ impl RaftCellHandle { return Err(CommandFailure::UnsupportedCapability); } } - Kind::ObserveFleet(authorization) => { - if self.binding.member().is_none() - || timeout > Duration::from_secs(60) - || !authorization.matches_scope(&self.binding.scope) - { - return Err(CommandFailure::UnsupportedCapability); - } - } Kind::AdvanceRepair { expected_revision, .. } => { @@ -1102,7 +969,6 @@ pub(crate) enum Kind { ActivateInitial(Box), Fleet(Box), FleetStatus, - ObserveFleet(Box), ActivateManaged { authorization: Box, activation: LearnerActivation, @@ -1142,24 +1008,17 @@ pub(crate) enum Kind { input: Vec, options: CommandOptions, }, - Query { - contract: OperationContract, - input: Vec, - options: QueryOptions, - }, Status, } impl Kind { fn is_control(&self) -> bool { - !matches!(self, Self::Command { .. } | Self::Query { .. }) + !matches!(self, Self::Command { .. }) } fn bytes(&self) -> usize { 4096 + match self { - Self::Command { input, .. } - | Self::SystemCommand { input, .. } - | Self::Query { input, .. } => input.len(), + Self::Command { input, .. } | Self::SystemCommand { input, .. } => input.len(), Self::Inspect | Self::InspectPeer(_) | Self::Status @@ -1175,7 +1034,6 @@ impl Kind { | Self::ActivateInitial(_) | Self::Fleet(_) | Self::FleetStatus - | Self::ObserveFleet(_) | Self::RepairStatus => 0, } } @@ -1185,7 +1043,7 @@ impl Kind { Self::Command { options, .. } | Self::SystemCommand { options, .. } => { CancelFailure::Command(options.id) } - Self::Query { .. } | Self::Seed { .. } => CancelFailure::Query, + Self::Seed { .. } => CancelFailure::Seed, Self::Inspect | Self::InspectPeer(_) | Self::Status @@ -1200,7 +1058,6 @@ impl Kind { | Self::ActivateInitial(_) | Self::Fleet(_) | Self::FleetStatus - | Self::ObserveFleet(_) | Self::RepairStatus => CancelFailure::Status, } } @@ -1208,7 +1065,7 @@ impl Kind { enum CancelFailure { Command(CommandId), - Query, + Seed, Status, } @@ -1217,30 +1074,23 @@ impl CancelFailure { let submitted = control.cancel(); match self { Self::Command(id) if submitted => CommandFailure::Ambiguous { command: *id }, - Self::Query => CommandFailure::VersionUnavailable, + Self::Seed => CommandFailure::VersionUnavailable, _ => CommandFailure::Unavailable, } } } pub(crate) enum Response { - FleetExecutionAction(Result<(), crate::raft::fleet_catalog::FleetCatalogError>), PeerFleetStop( Result< Box, crate::raft::fleet_catalog::FleetCatalogError, >, ), - PeerFleetObservation( - Result, super::InspectionError>, - ), - LocalFleetObservation(Box), PeerNodeInspection(Result, super::InspectionError>), PhysicalNode(Box), - HistoryRegistered, Node(Box), Fleet(Option>), - FleetObservation(Box), Drained(super::RaftDrainReport), Controller(super::RaftControllerStatus), PeerInspection( @@ -1253,7 +1103,6 @@ pub(crate) enum Response { Seed(Box), Learner(LearnerReceipt), Command(CommandReceipt), - Query(QueryResult), Status(CellStatus), } diff --git a/crates/cellar/application/consensus/client_execution.rs b/crates/cellar/application/consensus/client_execution.rs deleted file mode 100644 index 450d1dffc..000000000 --- a/crates/cellar/application/consensus/client_execution.rs +++ /dev/null @@ -1,122 +0,0 @@ -// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. - -//! Internal fleet actions preserve one original bounded waiter. -use super::*; -use crate::raft::fleet_catalog::{FleetCatalogError, FleetExecutionCommand, MovementCommand}; -use crate::raft::fleet_stop::BootBinding; - -pub(crate) struct PeerFleetActionCall { - pub command: FleetExecutionCommand, - pub minimum_revision: u64, -} - -pub(crate) struct LocalFleetActionCall { - pub command: FleetExecutionCommand, - pub origin: BootBinding, - pub minimum_revision: u64, -} - -impl RaftCells { - pub(crate) async fn start_restart_worker( - &self, - work: crate::raft::fleet_catalog::RestartServiceWork, - timeout: Duration, - ) -> Result<(), FleetCatalogError> { - match self - .node_call(NodeKind::StartRestartService(Box::new(work)), timeout) - .await - { - Ok(Response::FleetExecutionAction(result)) => result, - Err(CommandFailure::Overloaded) => Err(FleetCatalogError::Overloaded), - _ => Err(FleetCatalogError::Unconfirmed), - } - } - - pub(crate) async fn start_movement_worker( - &self, - work: crate::raft::fleet_catalog::MovementServiceWork, - timeout: Duration, - ) -> Result<(), FleetCatalogError> { - match self - .node_call(NodeKind::StartMovementService(Box::new(work)), timeout) - .await - { - Ok(Response::FleetExecutionAction(result)) => result, - Err(CommandFailure::Overloaded) => Err(FleetCatalogError::Overloaded), - _ => Err(FleetCatalogError::Unconfirmed), - } - } - - pub(crate) fn request_peer_movement( - &self, - command: MovementCommand, - minimum_revision: u64, - timeout: Duration, - ) -> Pin> + Send>> { - self.request_peer_fleet_action( - FleetExecutionCommand::Movement(command), - minimum_revision, - timeout, - ) - } - - pub(crate) fn request_peer_fleet_action( - &self, - command: FleetExecutionCommand, - minimum_revision: u64, - timeout: Duration, - ) -> Pin> + Send>> { - if command.validate().is_err() { - return Box::pin(async { Err(FleetCatalogError::Invalid("fleet execution command")) }); - } - self.fleet_action_call( - NodeKind::PeerFleetAction(Box::new(PeerFleetActionCall { - command, - minimum_revision, - })), - minimum_revision, - timeout, - ) - } - - pub(crate) fn execute_local_fleet_action( - &self, - command: FleetExecutionCommand, - origin: BootBinding, - minimum_revision: u64, - timeout: Duration, - ) -> Pin> + Send>> { - self.fleet_action_call( - NodeKind::LocalFleetAction(Box::new(LocalFleetActionCall { - command, - origin, - minimum_revision, - })), - minimum_revision, - timeout, - ) - } - - fn fleet_action_call( - &self, - kind: NodeKind, - minimum_revision: u64, - timeout: Duration, - ) -> Pin> + Send>> { - if minimum_revision == 0 - || minimum_revision >= u64::MAX - 2 - || timeout.as_millis() == 0 - || timeout > Duration::from_secs(60) - { - return Box::pin(async { Err(FleetCatalogError::Invalid("fleet action bounds")) }); - } - let call = self.node_call(kind, timeout); - Box::pin(async move { - match call.await { - Ok(Response::FleetExecutionAction(result)) => result, - Err(CommandFailure::Overloaded) => Err(FleetCatalogError::Overloaded), - _ => Err(FleetCatalogError::Unconfirmed), - } - }) - } -} diff --git a/crates/cellar/application/consensus/client_fleet.rs b/crates/cellar/application/consensus/client_fleet.rs index ed0b8c0b1..ea79a37c1 100644 --- a/crates/cellar/application/consensus/client_fleet.rs +++ b/crates/cellar/application/consensus/client_fleet.rs @@ -1,7 +1,6 @@ //! Managed configuration calls retain opaque catalog authorizations. use super::*; use crate::raft::fleet_catalog::{FleetGroupAuthorization, FleetPermitAuthorization}; -use crate::raft::host::FleetMembershipObservation; use crate::raft::member_state::{FleetMembershipStatus, RepairPlan, RepairStatus, RepairStep}; pub(crate) enum FleetRequest { @@ -48,27 +47,6 @@ impl FleetRequest { } impl RaftCellHandle { - /// Read the enrolled configuration after a new quorum ReadIndex barrier. - /// The result is one observation during this invocation, not a lease or a - /// certificate to release reservations or remove a node. Cached local fleet - /// status, an old observation and a successful enrollment retry cannot - /// substitute for this call's barrier. No application query or image is - /// required. The original deadline also bounds barrier retries. - pub fn observe_fleet_membership( - &self, - authorization: FleetGroupAuthorization, - timeout: Duration, - ) -> impl Future> + Send + 'static - { - let call = self.call(Kind::ObserveFleet(Box::new(authorization)), timeout); - async move { - match call.await? { - Response::FleetObservation(observation) => Ok(*observation), - _ => Err(CommandFailure::CorruptState), - } - } - } - /// Initialize an exact reserved initial voter inside this existing host. /// Enrollment and genesis share their first sync before attachment. The /// receipt is local durable state, never a live-quorum/write-ack proof. diff --git a/crates/cellar/application/consensus/client_node.rs b/crates/cellar/application/consensus/client_node.rs index abd6289a8..c14743be3 100644 --- a/crates/cellar/application/consensus/client_node.rs +++ b/crates/cellar/application/consensus/client_node.rs @@ -3,10 +3,6 @@ use super::*; pub(crate) enum NodeKind { - StartRestartService(Box), - StartMovementService(Box), - PeerFleetAction(Box), - LocalFleetAction(Box), Drain, Observe, ObservePhysical, @@ -15,23 +11,11 @@ pub(crate) enum NodeKind { minimum_revision: u64, physical: bool, }, - InspectPeerFleet(Box), PeerFleetStop(Box), - ObserveFleet { - history: Box, - minimum_revision: u64, - }, - RegisterHistory(Box), ControllerStatus, SetControllerEnabled(bool), } -pub(crate) struct PeerFleetRequest { - pub node: cellar_logic::consensus::NodeId, - pub minimum_revision: u64, - pub authorization: crate::raft::fleet_catalog::FleetGroupAuthorization, -} - pub(crate) struct PeerStopRequest { pub request: crate::raft::fleet_stop::FleetStopRequest, pub minimum_revision: u64, @@ -117,99 +101,6 @@ impl RaftCells { }) } - /// Obtain a fresh safe-ReadIndex observation from an enrolled group. A local - /// target uses the same guarded quorum observer without a loopback RPC. - /// The origin may have zero local groups. Both endpoints must install the - /// requested catalog cut; the target independently authorizes this exact - /// history. This creates no query, seeding or membership-mutation grant. - /// Retaining a response does not establish current absence or removal proof. - pub fn observe_peer_fleet_membership( - &self, - node: cellar_logic::consensus::NodeId, - authorization: crate::raft::fleet_catalog::FleetGroupAuthorization, - minimum_revision: u64, - timeout: Duration, - ) -> Pin< - Box< - dyn Future< - Output = Result< - super::super::RaftPeerFleetMembershipObservation, - super::super::InspectionError, - >, - > + Send, - >, - > { - use super::super::InspectionError; - if minimum_revision == 0 - || minimum_revision >= u64::MAX - 2 - || timeout.is_zero() - || timeout > Duration::from_secs(60) - { - return Box::pin(async { Err(InspectionError::Refused) }); - } - let domain = self.handle.domain.clone(); - let started = match domain.mono_ms() { - Ok(started) => started, - Err(_) => return Box::pin(async { Err(InspectionError::Unavailable) }), - }; - let call = self.node_call( - NodeKind::InspectPeerFleet(Box::new(PeerFleetRequest { - node, - minimum_revision, - authorization, - })), - timeout, - ); - Box::pin(async move { - let response = call.await.map_err(|error| match error { - CommandFailure::Overloaded => InspectionError::Overloaded, - CommandFailure::UnsupportedCapability => InspectionError::Refused, - _ => InspectionError::Unavailable, - })?; - let Response::PeerFleetObservation(result) = response else { - return Err(InspectionError::Unavailable); - }; - let mut observed = result?; - let age = domain - .mono_ms() - .ok() - .and_then(|now| now.checked_sub(started)) - .ok_or(InspectionError::Unavailable)?; - observed.record_delivery_age(age)?; - Ok(*observed) - }) - } - - pub(crate) fn observe_local_fleet_membership( - &self, - history: crate::raft::fleet_inspection_wire::History, - minimum_revision: u64, - timeout: Duration, - ) -> Pin< - Box< - dyn Future< - Output = Result< - Box, - CommandFailure, - >, - > + Send, - >, - > { - let call = self.node_call( - NodeKind::ObserveFleet { - history: Box::new(history), - minimum_revision, - }, - timeout, - ); - Box::pin(async move { - match call.await? { - Response::LocalFleetObservation(result) => Ok(result), - _ => Err(CommandFailure::CorruptState), - } - }) - } - /// Inspect an enrolled process without a cell. Both endpoints must /// acknowledge at least this trust revision; stale or missing metadata is /// not evidence of node failure. No placement or membership is activated. @@ -334,38 +225,6 @@ impl RaftCells { }) } - /// Register dormant metadata for an already enrolled history under this - /// host's declared class. This creates no member, image or voting authority. - /// A lost response can race publication; retry the same authorization. - pub fn register_managed_history( - &self, - authorization: crate::raft::fleet_catalog::FleetGroupAuthorization, - timeout: Duration, - ) -> Pin> + Send>> { - if timeout > Duration::from_secs(60) { - return Box::pin(async { Err(CommandFailure::UnsupportedCapability) }); - } - let scope = authorization - .binding() - .scope() - .map_err(|_| CommandFailure::CommandConflict); - let response = self.node_call(NodeKind::RegisterHistory(Box::new(authorization)), timeout); - let host = self.clone(); - Box::pin(async move { - let scope = scope?; - match response.await? { - Response::HistoryRegistered => { - let cell = host.cell(scope.namespace, &scope.cell)?; - if cell.binding.scope != scope { - return Err(CommandFailure::CommandConflict); - } - Ok(cell) - } - _ => Err(CommandFailure::CorruptState), - } - }) - } - /// Point-in-time local physical inventory with startup and installed current /// enrollment. This grants no capacity activation, quorum or removal proof. pub fn observe_node( diff --git a/crates/cellar/application/consensus/group_client/actions.rs b/crates/cellar/application/consensus/group_client/actions.rs new file mode 100644 index 000000000..73cb99282 --- /dev/null +++ b/crates/cellar/application/consensus/group_client/actions.rs @@ -0,0 +1,326 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Physical-group fleet controls share the installed node's real runtime and +//! authenticated transport. Lost responses do not establish mutation rollback. +use super::*; +use crate::raft::{ + completion::{ReplicationStatus, RestartLeadershipCheck, RuntimeError}, + fleet_catalog::{FleetCatalogError, FleetExecutionCommand, MovementCommand}, + fleet_stop::BootBinding, + transport::fleet_actions::Delivery, +}; +use std::pin::Pin; + +const ACTION_BYTES: usize = 128 * 1024; + +struct Local { + command: FleetExecutionCommand, + origin: BootBinding, +} + +enum Dispatch { + Local(Box), + Remote(Delivery), +} + +impl RaftCells { + pub(crate) fn request_peer_movement( + &self, + command: MovementCommand, + minimum_revision: u64, + timeout: Duration, + ) -> Pin> + Send>> { + self.request_peer_fleet_action( + FleetExecutionCommand::Movement(command), + minimum_revision, + timeout, + ) + } + + pub(crate) fn request_peer_fleet_action( + &self, + command: FleetExecutionCommand, + minimum_revision: u64, + timeout: Duration, + ) -> Pin> + Send>> { + // Admission and the original deadline precede first polling. The + // remote transport also starts its owned request at invocation. + let admitted: Result<_, FleetCatalogError> = (|| { + command.validate()?; + let (deadline, memory) = self.action_admission(minimum_revision, timeout)?; + let dispatch = if command.target().node == self.handle.core.node.node.bytes() { + let origin = self.action_boot(command.target())?; + Dispatch::Local(Box::new(Local { command, origin })) + } else { + Dispatch::Remote(self.handle.core.transport.request_fleet_action( + command, + minimum_revision, + deadline, + )?) + }; + Ok((deadline, memory, dispatch)) + })(); + let owner = self.internal_observer(); + Box::pin(async move { + let (deadline, _memory, dispatch) = admitted?; + owner.action_live(deadline)?; + let result = match dispatch { + Dispatch::Local(local) => { + owner + .local_action_at(local.command, local.origin, minimum_revision, deadline) + .await + } + Dispatch::Remote(delivery) => delivery.receive().await, + }; + // Expiry/close after dispatch is uncertain even when the command + // may already have reached its serialized owner. + owner + .action_live(deadline) + .map_err(|_| FleetCatalogError::Unconfirmed)?; + result + }) + } + + pub(crate) fn execute_local_fleet_action( + &self, + command: FleetExecutionCommand, + origin: BootBinding, + minimum_revision: u64, + timeout: Duration, + ) -> Pin> + Send>> { + let admitted: Result<_, FleetCatalogError> = (|| { + command.validate()?; + origin + .validate() + .map_err(|_| FleetCatalogError::Invalid("fleet action origin"))?; + self.action_boot(command.target())?; + self.action_admission(minimum_revision, timeout) + })(); + let owner = self.internal_observer(); + Box::pin(async move { + let (deadline, _memory) = admitted?; + owner + .local_action_at(command, origin, minimum_revision, deadline) + .await + }) + } + + fn action_admission( + &self, + minimum_revision: u64, + timeout: Duration, + ) -> Result<(u64, crate::raft::budget::Permit), FleetCatalogError> { + if self.is_closed() { + return Err(FleetCatalogError::Closed); + } + if self.handle.core.managed.is_none() { + return Err(FleetCatalogError::Invalid("no managed fleet action owner")); + } + if minimum_revision == 0 + || minimum_revision >= u64::MAX - 2 + || timeout.is_zero() + || timeout > Duration::from_secs(60) + { + return Err(FleetCatalogError::Invalid("fleet action bounds")); + } + self.metadata_admission(timeout, ACTION_BYTES) + .map_err(|error| match error { + cellar_logic::consensus::CommandFailure::Overloaded => { + FleetCatalogError::Overloaded + } + cellar_logic::consensus::CommandFailure::UnsupportedCapability => { + FleetCatalogError::Invalid("fleet action callback context") + } + _ => FleetCatalogError::Unconfirmed, + }) + } + + fn action_live(&self, deadline: u64) -> Result<(), FleetCatalogError> { + self.metadata_live(deadline) + .map_err(|_| FleetCatalogError::Unconfirmed) + } + + fn action_boot(&self, target: &BootBinding) -> Result { + let core = &self.handle.core; + let guard = core + .log + .lifecycle_guard() + .map_err(|_| FleetCatalogError::Closed)? + .ok_or(FleetCatalogError::Closed)?; + let installed = core + .managed + .as_ref() + .ok_or(FleetCatalogError::Invalid("no managed fleet receiver"))?; + if &guard.boot != target + || installed.boot_binding()? != guard.boot + || target.node != core.node.node.bytes() + || target.disk != core.node.disk.bytes() + || target.process != core.node.process.bytes() + || target.engine != core.engine.bytes() + || !guard.admits_group_start(target) && guard.quiescing().is_none() + { + return Err(FleetCatalogError::Conflict); + } + Ok(guard.boot) + } + + async fn local_action_at( + &self, + command: FleetExecutionCommand, + origin: BootBinding, + minimum_revision: u64, + deadline: u64, + ) -> Result<(), FleetCatalogError> { + self.action_live(deadline)?; + let target = command.target().clone(); + self.action_boot(&target)?; + let host = self + .handle + .core + .managed + .as_ref() + .ok_or(FleetCatalogError::Closed)?; + let remaining = deadline + .checked_sub( + self.handle + .domain + .mono_ms() + .map_err(|_| FleetCatalogError::Closed)?, + ) + .filter(|ms| *ms > 0) + .map(Duration::from_millis) + .ok_or(FleetCatalogError::Unconfirmed)?; + let timer = self + .handle + .domain + .sleep_until(deadline) + .map_err(|_| FleetCatalogError::Closed)?; + let work = self.handle.domain.run(host.execute_fleet_command( + command, + origin, + minimum_revision, + self, + remaining, + )); + let result = crate::asyncrt::select! { + _ = timer => Err(FleetCatalogError::Unconfirmed), + value = work => value.unwrap_or(Err(FleetCatalogError::Unconfirmed)), + }; + self.action_live(deadline) + .map_err(|_| FleetCatalogError::Unconfirmed)?; + self.action_boot(&target) + .map_err(|_| FleetCatalogError::Unconfirmed)?; + result + } +} + +impl RaftGroupHandle { + /// Admit a handoff only if the actual owner still matches the caller's term + /// and membership revision and the target is a caught-up incoming voter. + /// The invocation deadline is rechecked after queuing, before mutation. + /// Acceptance is not election completion. Expiry/close after dispatch does + /// not establish whether a handoff was admitted; observe the group again. + pub fn transfer_leadership( + &self, + transfer: super::super::LeadershipTransfer, + timeout: Duration, + ) -> impl Future> + Send + 'static { + let deadline = self.control_deadline(timeout); + let group = self.clone(); + async move { + let deadline = deadline?; + group.control_live(deadline)?; + super::drain::within( + &group.owner.handle.domain, + deadline, + group.owner.handle.core.runtime.transfer_leadership_before( + group.history, + transfer, + deadline, + ), + ) + .await?; + group.control_live(deadline) + } + } + + /// Leader-local metadata, without a quorum, applied image or health lease. + pub fn replication_status( + &self, + timeout: Duration, + ) -> impl Future, RuntimeError>> + Send + 'static + { + self.replication_status_for(timeout) + } + + pub(crate) fn replication_status_for( + &self, + timeout: Duration, + ) -> impl Future, RuntimeError>> + Send + 'static + { + let deadline = self.control_deadline(timeout); + let group = self.clone(); + async move { + let deadline = deadline?; + group.control_live(deadline)?; + let result = super::drain::within( + &group.owner.handle.domain, + deadline, + group + .owner + .handle + .core + .runtime + .replication_status(group.history), + ) + .await?; + group.control_live(deadline)?; + Ok(result) + } + } + + /// An exact restart observation or transfer, rechecked by the real group + /// owner after queuing. Acceptance never proves election completion. + pub(crate) fn restart_leadership( + &self, + mut check: RestartLeadershipCheck, + timeout: Duration, + ) -> impl Future> + Send + 'static { + let deadline = self.control_deadline(timeout); + let group = self.clone(); + async move { + let deadline = deadline?.min(check.deadline); + check.deadline = deadline; + group.control_live(deadline)?; + super::drain::within( + &group.owner.handle.domain, + deadline, + group + .owner + .handle + .core + .runtime + .restart_leadership(group.history, check), + ) + .await?; + group.control_live(deadline) + } + } + + pub(super) fn control_live(&self, deadline: u64) -> Result<(), RuntimeError> { + if self.owner.is_closed() { + return Err(RuntimeError::Closed); + } + if self + .owner + .handle + .domain + .mono_ms() + .map_err(|_| RuntimeError::Closed)? + >= deadline + { + return Err(RuntimeError::DeadlineExceeded); + } + Ok(()) + } +} diff --git a/crates/cellar/application/consensus/group_client/close.rs b/crates/cellar/application/consensus/group_client/close.rs new file mode 100644 index 000000000..ee6c483ba --- /dev/null +++ b/crates/cellar/application/consensus/group_client/close.rs @@ -0,0 +1,227 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Joined closure records complete physical group counts. A sampled empty queue +//! is accepted only behind sealed admission and idle physical SQL ownership. +use super::*; +use std::sync::atomic::{AtomicBool, Ordering}; +use tokio::sync::Notify; + +/// Local producer quiescence does not prove that callers received outcomes, +/// archive coverage is complete, or old deployments can be restarted/deleted. +#[derive(Clone, Copy, Debug, PartialEq, Eq, serde::Serialize, serde::Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum MutationOutcomes { + Unobserved, + NoUnresolvedLocalProducers, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct RaftGroupCloseReport { + pub history: GroupHistory, + pub member: MemberId, + pub committed: cellar_logic::consensus::LogIndex, + pub applied: cellar_logic::consensus::LogPosition, + /// None is unavailable coverage, never a measured zero-lag archive. + pub archived: Option, + /// Activation metadata in the final synced manifest, not a live quorum or + /// serving permission after shutdown. + pub active: bool, + pub failed: bool, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct RaftCloseReport { + pub node: NodeIdentity, + pub storage_instance: EngineInstanceId, + pub groups: Vec, + pub groups_total: usize, + pub omitted_groups: usize, + pub groups_with_archive_lag: usize, + pub groups_with_unknown_archive: usize, + pub retained_calls_at_shutdown: usize, + pub mutation_outcomes: MutationOutcomes, +} + +#[derive(Clone, Debug)] +pub struct RaftCloseError { + pub controller: Option>, + pub log: Option>, + pub inventory: Option>, +} + +impl std::fmt::Display for RaftCloseError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "shared Raft closure: {self:?}") + } +} + +impl std::error::Error for RaftCloseError {} + +type Closed = Result>; + +pub(super) struct Close { + pub closing: AtomicBool, + requested: Notify, + terminal: Mutex>, + finished: Notify, + admission: Arc, +} + +impl Close { + pub fn new(admission: Arc) -> Arc { + Arc::new(Self { + closing: AtomicBool::new(false), + requested: Notify::new(), + terminal: Mutex::new(None), + finished: Notify::new(), + admission, + }) + } + + pub fn request(&self) { + self.closing.store(true, Ordering::Release); + let _ = self.admission.seal(); + self.requested.notify_one(); + } + + pub async fn wait(&self) -> Closed { + loop { + let notified = self.finished.notified(); + tokio::pin!(notified); + notified.as_mut().enable(); + if let Some(result) = self + .terminal + .lock() + .unwrap_or_else(|e| e.into_inner()) + .clone() + { + return result; + } + notified.await; + } + } +} + +pub(super) fn start( + core: Arc, + close: Arc, + domain: crate::asyncrt::DomainToken, +) -> Result<(), CellGroupControllerError> { + domain + .spawn_cleanup(async move { + let requested = close.requested.notified(); + tokio::pin!(requested); + requested.as_mut().enable(); + if !close.closing.load(Ordering::Acquire) { + crate::asyncrt::select! { _ = requested => (), _ = core.controller.wait() => () } + } + close.request(); + core.services.close().await; + core.gateways + .lock() + .unwrap_or_else(|e| e.into_inner()) + .clear(); + let usage = core.controller.usage(); + // Acquire no new writer after sealing. Accepted physical work cannot + // disappear from these counters when its response times out or drops. + let quiesced = !close.admission.is_poisoned() + && core.controller.local_producers_quiesced() + && core.resources.sql.usage().is_ok_and(|sql| { + sql.active_cells == 0 + && sql.retained_operations == 0 + && sql.queued_or_running == 0 + }); + let controller = core.host.close().await.err().map(Arc::new); + // Read after persistence/SQL/runtime joins, before closing the engine. + // Include every durable group even if its detail row exceeds the cap. + let report = capture(&core, usage.retained_commands, quiesced); + let log = core.log.close().await.err().map(Arc::new); + let result = match (controller, log, report) { + (None, None, Ok(report)) => Ok(report), + (controller, log, report) => Err(Arc::new(RaftCloseError { + controller, + log, + inventory: report.err().map(Arc::new), + })), + }; + *close.terminal.lock().unwrap_or_else(|e| e.into_inner()) = Some(result); + close.finished.notify_waiters(); + }) + .map_err(|_| CellGroupControllerError::Closed)? + .detach(); + Ok(()) +} + +fn capture( + core: &Core, + retained_calls: usize, + quiesced: bool, +) -> Result { + let groups = core.log.cell_groups()?; + let mut report = RaftCloseReport { + node: core.node, + storage_instance: core.engine, + groups: Vec::new(), + groups_total: groups.len(), + omitted_groups: 0, + groups_with_archive_lag: 0, + groups_with_unknown_archive: 0, + retained_calls_at_shutdown: retained_calls, + mutation_outcomes: if quiesced { + MutationOutcomes::NoUnresolvedLocalProducers + } else { + MutationOutcomes::Unobserved + }, + }; + // Fixed-size domain values only. Charge generously for vector/allocation + // overhead; counts remain complete even when all detail rows are omitted. + let details = core + .close_limits + .max_groups + .min(core.close_limits.max_bytes / 512); + for group in groups { + let history = group.identity().scope; + let published = core + .controller + .archive_status(history) + .ok() + .flatten() + .and_then(|status| status.published); + let failed = core + .controller + .group_failure(history) + .map_or(true, |failure| failure.is_some()); + if failed { + report.mutation_outcomes = MutationOutcomes::Unobserved; + } + let cache = group.read()?; + let marker = cache + .group_manifest + .as_ref() + .ok_or(LogError::Corrupt("closed group manifest missing".into()))?; + let archived = published + .filter(|published| published.route() == marker.route()) + .map(|published| published.position()); + match archived { + None => report.groups_with_unknown_archive += 1, + Some(position) if position.index.get() < cache.hard_state.commit => { + report.groups_with_archive_lag += 1 + } + Some(_) => (), + } + if report.groups.len() < details { + report.groups.push(RaftGroupCloseReport { + history, + member: group.identity().member, + committed: cellar_logic::consensus::LogIndex::new(cache.hard_state.commit), + applied: marker.position(), + archived, + active: marker.is_active(), + failed, + }); + } else { + report.omitted_groups += 1; + } + } + Ok(report) +} diff --git a/crates/cellar/application/consensus/group_client/drain.rs b/crates/cellar/application/consensus/group_client/drain.rs new file mode 100644 index 000000000..8762dce61 --- /dev/null +++ b/crates/cellar/application/consensus/group_client/drain.rs @@ -0,0 +1,154 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! One bounded turn per physical group. The actual Raft owner rechecks term, +//! membership revision, recent activity and catch-up before admitting transfer. +use super::*; +use crate::raft::{ + completion::{LeadershipTransfer, RuntimeError}, + operations::RaftDrainReport, +}; +use cellar_logic::consensus::CommandFailure; + +impl RaftCells { + pub(crate) async fn drain_node( + &self, + deadline: u64, + ) -> Result { + self.begin_node_drain(); + let core = &self.handle.core; + let domain = &self.handle.domain; + let groups = core + .log + .cell_groups() + .map_err(|_| CommandFailure::Unavailable)?; + let mut report = RaftDrainReport::default(); + for group in groups { + let now = domain.mono_ms().map_err(|_| CommandFailure::Unavailable)?; + if now >= deadline { + report.deadline_elapsed = true; + break; + } + report.groups_examined += 1; + let through = deadline.min(now.saturating_add(1000)); + let history = group.identity().scope; + let local = group.identity().member.get(); + let mut turn = Turn::default(); + let outcome = within(domain, through, async { + loop { + let status = core.runtime.status(history).await?; + if status.leader != Some(local) { + if status.applied.index == status.committed { + return Ok(()); + } + pause(domain, through).await?; + continue; + } + turn.needed = true; + if status.ready && status.applied.index == status.committed { + break; + } + pause(domain, through).await?; + } + let progress = core.runtime.replication_status(history).await?; + for member in progress + .members + .iter() + .filter(|p| { + p.voter + && p.member.get() != local + && p.recent_active + && p.matched >= progress.committed + }) + .take(8) + { + turn.attempts += 1; + match core + .runtime + .transfer_leadership( + history, + LeadershipTransfer { + target: member.member, + expected_term: progress.term, + expected_revision: progress.membership_revision, + }, + ) + .await + { + Ok(()) => { + // Admission remains a separate observation if this + // deadline expires before a successor is visible. + turn.accepted = true; + within(domain, through, async { + loop { + let status = core.runtime.status(history).await?; + if status.term >= progress.term + && status.leader.is_some_and(|id| id != local) + { + turn.released = true; + // Observing a successor can precede local + // commit and apply of its fresh-term entry. + if status.applied.term == status.term + && status.applied.index == status.committed + { + return Ok(()); + } + } + pause(domain, through).await?; + } + }) + .await?; + return Ok(()); + } + Err(RuntimeError::NotReady) => (), + Err(error) => return Err(error), + } + } + Ok(()) + }) + .await; + report.handoff_attempts += turn.attempts; + report.handoffs_accepted += usize::from(turn.accepted); + report.leadership_released += usize::from(turn.released); + if outcome.is_err() || turn.needed && !turn.accepted { + report.unavailable_groups += 1; + } + } + report.deadline_elapsed |= + domain.mono_ms().map_err(|_| CommandFailure::Unavailable)? >= deadline; + Ok(report) + } +} + +#[derive(Default)] +struct Turn { + needed: bool, + attempts: usize, + accepted: bool, + released: bool, +} + +pub(super) async fn within( + domain: &crate::asyncrt::DomainToken, + deadline: u64, + future: impl Future>, +) -> Result { + if domain.mono_ms().map_err(|_| RuntimeError::Closed)? >= deadline { + return Err(RuntimeError::DeadlineExceeded); + } + let timer = domain + .sleep_until(deadline) + .map_err(|_| RuntimeError::Closed)?; + crate::asyncrt::select! { result = future => result, _ = timer => Err(RuntimeError::DeadlineExceeded) } +} + +async fn pause(domain: &crate::asyncrt::DomainToken, deadline: u64) -> Result<(), RuntimeError> { + let now = domain.mono_ms().map_err(|_| RuntimeError::Closed)?; + if now >= deadline { + return Err(RuntimeError::DeadlineExceeded); + } + domain + .sleep_until(deadline.min(now.saturating_add(25))) + .map_err(|_| RuntimeError::Closed)? + .await; + Ok(()) +} diff --git a/crates/cellar/application/consensus/group_client/fleet.rs b/crates/cellar/application/consensus/group_client/fleet.rs new file mode 100644 index 000000000..55d409d55 --- /dev/null +++ b/crates/cellar/application/consensus/group_client/fleet.rs @@ -0,0 +1,224 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Fleet reads are metadata-only and require the exact rooted group admission. +//! The receiver independently reauthorizes it; node transport trust is not enough. +use super::*; +use crate::raft::{ + fleet_catalog::FleetGroupAuthorization, + fleet_inspection_wire::{ + GroupSnapshot, History, LocalObservation, RaftPeerFleetMembershipObservation, + RESPONSE_BYTES, + }, + inspection::InspectionError, +}; +use cellar_logic::consensus::{CommandFailure, NodeId}; + +impl RaftCells { + /// Fresh complete-group membership/publication from an exact enrolled boot. + /// Followers use safe ReadIndex forwarding and their own durable apply; + /// this grants no leader, seed, mutation, archive or physical-release right. + pub fn observe_peer_fleet_membership( + &self, + node: NodeId, + authorization: FleetGroupAuthorization, + minimum_revision: u64, + timeout: Duration, + ) -> impl Future> + Send + 'static + { + let admitted = (|| { + if minimum_revision == 0 + || minimum_revision >= u64::MAX - 2 + || self.handle.core.managed.is_none() + { + return Err(InspectionError::Refused); + } + let local = node == self.handle.core.node.node; + let (deadline, memory) = self + .metadata_admission( + timeout, + if local { + crate::raft::physical_inventory::RESPONSE_BYTES + } else { + RESPONSE_BYTES + }, + ) + .map_err(inspection_error)?; + let started = deadline + .checked_sub(timeout.as_millis() as u64) + .ok_or(InspectionError::Unavailable)?; + let remote = if local { + None + } else { + Some(self.handle.core.transport.inspect_fleet_membership( + node, + &authorization, + minimum_revision, + deadline, + )?) + }; + Ok((deadline, started, memory, remote)) + })(); + let owner = self.internal_observer(); + async move { + let (deadline, started, _memory, remote) = admitted?; + owner.metadata_live(deadline).map_err(inspection_error)?; + let local = if let Some(delivery) = remote { + delivery.receive().await?.observation + } else { + owner + .local_fleet_at(History::new(&authorization), minimum_revision, deadline) + .await + .map_err(inspection_error)? + }; + owner.metadata_live(deadline).map_err(inspection_error)?; + owner + .handle + .core + .transport + .validate_fleet_node_observation(&local.node, minimum_revision)?; + let age = owner + .handle + .domain + .mono_ms() + .ok() + .and_then(|now| now.checked_sub(started)) + .ok_or(InspectionError::Unavailable)?; + local + .into_peer(&authorization, age) + .map_err(|_| InspectionError::Refused) + } + } + + pub(crate) fn observe_local_fleet_membership( + &self, + history: History, + minimum_revision: u64, + timeout: Duration, + ) -> impl Future> + Send + 'static { + let admitted = + self.metadata_admission(timeout, crate::raft::physical_inventory::RESPONSE_BYTES); + let owner = self.internal_observer(); + async move { + let (deadline, _memory) = admitted?; + owner + .local_fleet_at(history, minimum_revision, deadline) + .await + } + } + + async fn local_fleet_at( + &self, + history: History, + minimum_revision: u64, + deadline: u64, + ) -> Result { + self.metadata_live(deadline)?; + if minimum_revision == 0 || minimum_revision >= u64::MAX - 2 { + return Err(CommandFailure::CommandConflict); + } + let scope = history + .scope() + .map_err(|_| CommandFailure::CommandConflict)?; + let reader = self + .handle + .core + .fleet_inspection + .as_ref() + .ok_or(CommandFailure::UnsupportedCapability)?; + let remaining = deadline + .checked_sub( + self.handle + .domain + .mono_ms() + .map_err(|_| CommandFailure::Unavailable)?, + ) + .filter(|n| *n > 0) + .map(Duration::from_millis) + .ok_or(CommandFailure::VersionUnavailable)?; + let authorization = reader + .authorize_group_at_least(&scope, minimum_revision, remaining) + .await + .map_err(|error| match error { + crate::raft::fleet_catalog::FleetCatalogError::Overloaded + | crate::raft::fleet_catalog::FleetCatalogError::Limit => { + CommandFailure::Overloaded + } + _ => CommandFailure::Unavailable, + })?; + if !history.matches(&authorization) { + return Err(CommandFailure::CommandConflict); + } + let selected = self + .handle + .core + .runtime + .quorum_membership(scope, deadline) + .await + .map_err(|error| match error { + crate::raft::completion::RuntimeError::Overloaded => CommandFailure::Overloaded, + _ => CommandFailure::Unavailable, + })?; + let mut snapshot = GroupSnapshot::capture(&selected); + // This descriptor contains no image/page pins. Release the temporary + // metadata read reservation before sampling unrelated physical work. + drop(selected); + let load = match self.handle.core.runtime.leadership_sample(deadline).await { + Ok(sample) => Some(sample), + Err( + crate::raft::completion::RuntimeError::NotReady + | crate::raft::completion::RuntimeError::GroupQuarantined + | crate::raft::completion::RuntimeError::Overloaded + | crate::raft::completion::RuntimeError::DeadlineExceeded, + ) => None, + Err(_) => return Err(CommandFailure::Unavailable), + }; + let (node, _) = self.capture_physical_node()?; + // An unattached or initializing physical history cannot supply known + // headroom. Count complete groups, independently of manifest cell count. + if let Some(sample) = load.filter(|sample| { + sample.groups == node.node.groups as usize + && node.pending_history_registrations == 0 + && node.unattached_engine_histories == 0 + }) { + snapshot.load = Some(crate::raft::inspection::LeaderLoad { + leaders: sample.leaders, + capacity: node.node.profile.leader_capacity, + sample_age_ms: self + .handle + .domain + .mono_ms() + .map_err(|_| CommandFailure::Unavailable)? + .checked_sub(sample.started_at_ms) + .ok_or(CommandFailure::Unavailable)?, + }); + } + self.handle + .core + .transport + .validate_fleet_node_observation(&node, minimum_revision) + .map_err(|_| CommandFailure::Unavailable)?; + self.metadata_live(deadline)?; + let local = LocalObservation { + node: Box::new(node), + snapshot, + }; + local.validate().map_err(|_| CommandFailure::CorruptState)?; + // The same constructor checks binding, immutable birth/schema and the + // actual local replica before anything reaches the transport encoder. + crate::raft::host::FleetMembershipObservation::check_quorum_group( + &local.snapshot, + &authorization, + )?; + Ok(local) + } +} + +fn inspection_error(error: CommandFailure) -> InspectionError { + match error { + CommandFailure::Overloaded => InspectionError::Overloaded, + CommandFailure::CommandConflict | CommandFailure::UnsupportedCapability => { + InspectionError::Refused + } + _ => InspectionError::Unavailable, + } +} diff --git a/crates/cellar/application/consensus/group_client/hibernation.rs b/crates/cellar/application/consensus/group_client/hibernation.rs new file mode 100644 index 000000000..dd4759504 --- /dev/null +++ b/crates/cellar/application/consensus/group_client/hibernation.rs @@ -0,0 +1,211 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::{startup, *}; +use crate::raft::{ + catalog::CatalogError, + completion::RuntimeError, + fleet_catalog::{FleetCatalogError, ManagedParkPermit}, + transport::ParkDeliveryReceipt, +}; + +impl RaftGroupHandle { + pub(crate) async fn observe_park_admission( + &self, + timeout: Duration, + ) -> Result { + let (deadline, _memory) = self + .owner + .metadata_admission(timeout, 4096) + .map_err(startup::admission_error)?; + Ok(self + .owner + .handle + .core + .runtime + .park_admission(self.history, deadline) + .await?) + } + + pub(crate) async fn execute_managed_park( + &self, + permit: &ManagedParkPermit, + timeout: Duration, + ) -> Result { + let (deadline, _memory) = self + .owner + .metadata_admission(timeout, 64 * 1024) + .map_err(startup::admission_error)?; + let owner = self.owner.internal_observer(); + owner + .handle + .domain + .run(Box::pin(crate::asyncrt::timeout_at( + deadline, + Box::pin(owner.deliver_managed_park(self.history, permit, deadline)), + ))) + .await + .map_err(|_| CellGroupControllerError::Closed)? + .map_err(|_| CellGroupControllerError::Runtime(RuntimeError::DeadlineExceeded))? + } + + pub(crate) async fn wake_managed_park( + &self, + permit: &ManagedParkPermit, + unsettled: bool, + timeout: Duration, + ) -> Result { + let (deadline, _memory) = self + .owner + .metadata_admission(timeout, 4096) + .map_err(startup::admission_error)?; + let core = &self.owner.handle.core; + let host = core + .managed + .as_ref() + .ok_or_else(|| startup::catalog_error(CatalogError::ManagedAdmissionRequired))?; + self.owner.startup_live(host, deadline)?; + let boot = host.boot_binding().map_err(park_catalog_error)?; + if !permit.matches_fleet(self.history, &boot) { + return Err(RuntimeError::StaleCompletion.into()); + } + let generation = permit.generation().map_err(park_catalog_error)?; + if unsettled { + let term = permit.term(); + let domain = &self.owner.handle.domain; + domain + .run(Box::pin(crate::asyncrt::timeout_at( + deadline, + Box::pin(async { + loop { + self.owner.startup_live(host, deadline)?; + match core + .runtime + .fence_managed_park(self.history, generation, term, deadline) + .await + { + Ok(()) => return Ok(()), + Err(RuntimeError::NotReady | RuntimeError::Overloaded) => (), + Err(error) => return Err(CellGroupControllerError::from(error)), + } + let now = domain + .mono_ms() + .map_err(|_| CellGroupControllerError::Closed)?; + domain + .sleep_until(deadline.min(now.saturating_add(25))) + .map_err(|_| CellGroupControllerError::Closed)? + .await; + } + }), + ))) + .await + .map_err(|_| CellGroupControllerError::Closed)? + .map_err(|_| CellGroupControllerError::Runtime(RuntimeError::DeadlineExceeded))??; + } + let observation = core + .runtime + .wake_park(self.history, generation, deadline) + .await?; + self.owner.startup_live(host, deadline)?; + Ok(observation) + } +} + +impl RaftCells { + async fn deliver_managed_park( + &self, + history: GroupHistory, + permit: &ManagedParkPermit, + deadline: u64, + ) -> Result { + let core = &self.handle.core; + let host = core + .managed + .as_ref() + .ok_or_else(|| startup::catalog_error(CatalogError::ManagedAdmissionRequired))?; + self.startup_live(host, deadline)?; + let boot = host.boot_binding().map_err(park_catalog_error)?; + let group = core.log.cell_group(&history).map_err(RuntimeError::from)?; + if !permit.matches_owner(history, &boot, group.identity().member) { + return Err(RuntimeError::StaleCompletion.into()); + } + let current = host + .stop_catalog() + .reopen_managed_park( + permit.operation().map_err(park_catalog_error)?, + self.startup_remaining(deadline)?, + ) + .await + .map_err(park_catalog_error)?; + if !current.is_some_and(|current| permit.same_reservation(¤t)) { + return Err(RuntimeError::StaleCompletion.into()); + } + self.startup_live(host, deadline)?; + + let domain = &self.handle.domain; + let now = domain + .mono_ms() + .map_err(|_| CellGroupControllerError::Closed)?; + let mut end = deadline.min(now.saturating_add(MAX_PARK_NEGOTIATION_MS)); + let request = permit.prepare_request(end).map_err(park_catalog_error)?; + let generation = permit.generation().map_err(park_catalog_error)?; + let cut = loop { + self.startup_live(host, end)?; + let (marker, parked) = { + let cache = group.read().map_err(RuntimeError::from)?; + ( + cache.group_manifest.clone().ok_or(RuntimeError::NotReady)?, + cache + .local_park + .as_ref() + .is_some_and(LocalParkRecord::parked), + ) + }; + let applied = marker.position(); + if !marker.is_active() || applied.index.get() > generation.get() { + return Err(RuntimeError::StaleCompletion.into()); + } + if parked { + let cut = marker.park_cut().filter(|cut| { + cut.position() == applied + && cut.generation() == generation.get() + && cut.position().term == request.term + && cut.membership_revision() == request.membership.get() + }); + break cut.cloned().ok_or(RuntimeError::StaleCompletion)?; + } + match core.runtime.prepare_park(history, request).await { + Ok(prepared) => { + end = end.min(prepared.deadline_ms); + if let Some(cut) = marker.park_cut().filter(|cut| { + cut.position() == applied && cut.position() == prepared.proposal.position + }) { + break cut.clone(); + } + } + Err(RuntimeError::NotReady | RuntimeError::Overloaded) => (), + Err(error) => return Err(error.into()), + } + let now = domain + .mono_ms() + .map_err(|_| CellGroupControllerError::Closed)?; + domain + .sleep_until(end.min(now.saturating_add(25))) + .map_err(|_| CellGroupControllerError::Closed)? + .await; + }; + let selected = core + .transport + .collect_park_certificate(history, cut, end) + .await + .map_err(CellGroupControllerError::Transport)?; + self.startup_live(host, end)?; + core.transport + .deliver_park_certificate(history, &selected, end) + .await + .map_err(CellGroupControllerError::Transport) + } +} + +fn park_catalog_error(error: FleetCatalogError) -> CellGroupControllerError { + startup::catalog_error(CatalogError::from(error)) +} diff --git a/crates/cellar/application/consensus/group_client/learner.rs b/crates/cellar/application/consensus/group_client/learner.rs new file mode 100644 index 000000000..676d0f84a --- /dev/null +++ b/crates/cellar/application/consensus/group_client/learner.rs @@ -0,0 +1,203 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Managed learner startup belongs to the node's retained provisioning owner. +use super::startup::{admission_error, catalog_error}; +use super::*; +use crate::application::consensus::client::LearnerActivation; +use crate::raft::{ + catalog::CatalogError, + completion::RuntimeError, + fleet_catalog::{FleetGroupAuthorization, FleetPermitAuthorization}, + member_state::{FleetMembershipStatus, FleetPermitPhase}, +}; + +impl RaftCells { + /// Install and attach one catalog-selected physical learner. Success is a + /// local durable receipt; it cannot promote the learner or grant serving. + /// Once submitted, cancellation retains preparation and joins entered I/O. + pub fn activate_managed_learner_group( + &self, + group: FleetGroupAuthorization, + authorization: FleetPermitAuthorization, + activation: LearnerActivation, + ) -> impl Future> + Send + 'static + { + let admitted = (|| { + let (deadline, memory) = self + .metadata_admission(activation.timeout, 4 * 1024 * 1024) + .map_err(admission_error)?; + let host = self + .handle + .core + .managed + .clone() + .ok_or(CellGroupControllerError::Invalid("managed learner owner"))?; + let history = group + .binding() + .scope() + .map_err(|e| catalog_error(CatalogError::Log(e)))?; + if group.binding() != authorization.permit().binding() + || group.binding().fleet() != host.fleet() + { + return Err(CellGroupControllerError::Invalid( + "managed learner capability", + )); + } + let directory = + self.handle + .core + .host + .directory(history) + .map_err(|error| match error { + CellDispatchError::Runtime(error) => error.into(), + _ => CellGroupControllerError::Invalid("learner namespace capability"), + })?; + Ok((deadline, memory, host, history, directory)) + })(); + let owner = self.internal_observer(); + async move { + use CellGroupStartupError::{Refused, Unconfirmed}; + let (deadline, _memory, host, history, directory) = admitted.map_err(Refused)?; + let mut submitted = false; + let work = async { + owner.startup_live(&host, deadline).map_err(Refused)?; + let birth = directory + .materialize_selected_history(history) + .await + .map_err(catalog_error) + .map_err(Refused)?; + if !group.check_birth(&directory, &birth) { + return Err(Refused(CellGroupControllerError::Invalid("learner birth"))); + } + let fresh = host + .group_authorization( + &history, + owner.startup_remaining(deadline).map_err(Refused)?, + ) + .await + .map_err(CatalogError::from) + .map_err(catalog_error) + .map_err(Refused)?; + if fresh.binding() != group.binding() || !fresh.check_birth(&directory, &birth) { + return Err(Refused(CellGroupControllerError::Invalid( + "learner history changed", + ))); + } + owner.startup_live(&host, deadline).map_err(Refused)?; + let member = activation.member; + let call = owner + .handle + .core + .controller + .provision_managed_learner( + directory.clone(), + birth.route().clone(), + host.clone(), + authorization.clone(), + activation, + deadline, + ) + .map_err(Refused)?; + submitted = true; + call.response() + .await + .map_err(|e| Unconfirmed((*e).clone()))?; + owner.startup_live(&host, deadline).map_err(Unconfirmed)?; + let fresh = host + .group_authorization( + &history, + owner.startup_remaining(deadline).map_err(Unconfirmed)?, + ) + .await + .map_err(CatalogError::from) + .map_err(catalog_error) + .map_err(Unconfirmed)?; + if fresh.binding() != group.binding() || !fresh.check_birth(&directory, &birth) { + return Err(Unconfirmed(CellGroupControllerError::Invalid( + "learner history changed", + ))); + } + let core = &owner.handle.core; + if core + .controller + .group_failure(history) + .map_err(CellGroupControllerError::from) + .map_err(Unconfirmed)? + .is_some() + { + return Err(Unconfirmed(RuntimeError::GroupQuarantined.into())); + } + let local = core + .log + .cell_group(&history) + .map_err(|e| Unconfirmed(RuntimeError::from(e).into()))?; + let status = { + let cache = local + .read() + .map_err(|e| Unconfirmed(RuntimeError::from(e).into()))?; + let joined = cache + .membership + .joined(member) + .map_err(|e| Unconfirmed(RuntimeError::from(e).into()))?; + let status = cache.membership.fleet_status().ok_or(Unconfirmed( + CellGroupControllerError::Invalid("learner fleet guard"), + ))?; + if local.identity().member != member + || !fresh.matches_replica(local.identity(), &cache.membership) + || cache + .group_manifest + .as_ref() + .is_none_or(|m| m.route() != birth.route() || !m.is_active()) + || joined.is_none_or(|(binding, at)| { + binding.node != core.node.node + || binding.disk != core.node.disk + || cache.group_seed.is_none_or(|seed| seed < at.get()) + }) + || status.permit.as_ref() != Some(authorization.permit()) + || &status.binding != authorization.permit().binding() + || status.consumed_sequence != authorization.permit().sequence() + || !matches!( + status.phase, + Some( + FleetPermitPhase::Seeding + | FleetPermitPhase::Joint + | FleetPermitPhase::Complete + ) + ) + { + return Err(Unconfirmed(CellGroupControllerError::Invalid( + "learner installed receipt", + ))); + } + status + }; + owner.startup_live(&host, deadline).map_err(Unconfirmed)?; + Ok(status) + }; + match owner + .handle + .core + .transport + .seed_control_until(deadline, async { + owner + .handle + .domain + .run(work) + .await + .map_err(|_| crate::raft::transport::TransportError::Closed) + }) + .await + { + Ok(result) => result, + Err(error) => { + let error = CellGroupControllerError::Transport(error); + Err(if submitted { + Unconfirmed(error) + } else { + Refused(error) + }) + } + } + } + } +} diff --git a/crates/cellar/application/consensus/group_client/membership.rs b/crates/cellar/application/consensus/group_client/membership.rs new file mode 100644 index 000000000..af361f8a0 --- /dev/null +++ b/crates/cellar/application/consensus/group_client/membership.rs @@ -0,0 +1,301 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Administrative commands address a physical group. Durable apply and a fresh +//! quorum barrier, rather than proposal admission, establish their result. +use super::*; +use crate::raft::{ + completion::RuntimeError, + fleet_catalog::{FleetGroupAuthorization, FleetPermitAuthorization}, + member_state::FleetPermitPhase, +}; +use cellar_logic::consensus::LogIndex; + +/// Refused means this invocation submitted no configuration change. It does not +/// resolve an earlier lost response. Unconfirmed may follow actual commitment; +/// reconcile the same authorization instead of inventing another operation. +#[derive(Clone, Debug)] +pub enum CellGroupControlError { + Refused(RuntimeError), + Unconfirmed(RuntimeError), +} + +impl std::fmt::Display for CellGroupControlError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::Refused(error) => write!(f, "group control refused: {error}"), + Self::Unconfirmed(error) => write!(f, "group control outcome unconfirmed: {error}"), + } + } +} + +impl std::error::Error for CellGroupControlError { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + Some(match self { + Self::Refused(error) | Self::Unconfirmed(error) => error, + }) + } +} + +pub(super) enum Change { + Enroll { + authorization: FleetGroupAuthorization, + expected: LogIndex, + }, + Fence(Box), + Repair(Box), +} + +impl Change { + fn satisfied(&self, observed: &CellGroupQuorumMembership) -> Result { + let status = observed.membership().fleet_status(); + match self { + Self::Enroll { authorization, .. } => { + if authorization.history_digests() + != ( + observed.root().route().birth_digest, + observed.root().route().schema_digest, + ) + || !authorization.matches_replica(observed.replica(), observed.membership()) + { + return Err(RuntimeError::Invalid("fleet enrollment authority".into())); + } + let Some(status) = status else { + return Ok(false); + }; + if &status.binding != authorization.binding() { + return Err(RuntimeError::Invalid( + "group enrolled in another fleet".into(), + )); + } + Ok(true) + } + Self::Fence(authorization) => { + let status = + status.ok_or_else(|| RuntimeError::Invalid("group is not enrolled".into()))?; + let permit = authorization.permit(); + if &status.binding != permit.binding() { + return Err(RuntimeError::Invalid("foreign group fence".into())); + } + if status.consumed_sequence < permit.sequence() { + return Ok(false); + } + if status.consumed_sequence == permit.sequence() + && status.permit.as_ref() == Some(permit) + && status.phase == Some(FleetPermitPhase::Fenced) + { + return Ok(true); + } + // A later sequence or another consumed phase cannot prove that + // this exact cancellation fence committed, nor release its debt. + Err(RuntimeError::StaleCompletion) + } + Self::Repair(change) => change.satisfied(observed), + } + } + + fn prepare( + &self, + observed: &CellGroupQuorumMembership, + ) -> Result { + let route = observed.root().route(); + let membership = observed.membership(); + let configuration = observed.configuration(); + Ok(match self { + Self::Enroll { + authorization, + expected, + } => { + if membership.revision() != *expected { + return Err(RuntimeError::StaleCompletion); + } + CellGroupMembershipChange::fleet_enrollment( + route, + membership, + configuration, + authorization, + )? + } + Self::Fence(authorization) => CellGroupMembershipChange::fleet_fence( + route, + membership, + configuration, + authorization, + )?, + Self::Repair(change) => change.prepare(observed)?, + }) + } +} + +impl RaftGroupHandle { + pub(super) fn control_deadline(&self, timeout: Duration) -> Result { + if self.owner.is_closed() { + return Err(RuntimeError::Closed); + } + if timeout.is_zero() || timeout > Duration::from_secs(60) { + return Err(RuntimeError::Invalid("group control timeout".into())); + } + self.owner + .handle + .domain + .mono_ms() + .map_err(|_| RuntimeError::Closed)? + .checked_add(timeout.as_millis() as u64) + .ok_or(RuntimeError::Overloaded) + } + + /// A fresh quorum observation behind a durable physical apply floor. Time + /// before polling consumes the original invocation deadline. + pub fn quorum_membership_after( + &self, + minimum: LogIndex, + timeout: Duration, + ) -> impl Future> + Send + 'static + { + let deadline = self.control_deadline(timeout); + let owner = self.owner.clone(); + let history = self.history; + async move { + owner + .handle + .core + .runtime + .quorum_membership_after(history, minimum, deadline?) + .await + } + } + + /// The exact rooted fleet binding must already be committed by this group. + /// The result contains no cell alias, SQL image or physical-release lease. + pub fn observe_fleet_membership( + &self, + authorization: FleetGroupAuthorization, + timeout: Duration, + ) -> impl Future> + Send + 'static + { + let deadline = self.control_deadline(timeout); + let owner = self.owner.clone(); + let history = self.history; + async move { + if !authorization.matches_scope(&history) { + return Err(RuntimeError::Invalid("fleet observation history".into())); + } + let observed = owner + .handle + .core + .runtime + .quorum_membership(history, deadline?) + .await?; + let status = observed + .membership() + .fleet_status() + .ok_or_else(|| RuntimeError::Invalid("group is not enrolled".into()))?; + if &status.binding != authorization.binding() + || authorization.history_digests() + != ( + observed.root().route().birth_digest, + observed.root().route().schema_digest, + ) + || !authorization.matches_replica(observed.replica(), observed.membership()) + { + return Err(RuntimeError::Invalid("fleet observation authority".into())); + } + Ok(observed) + } + } + + /// Commit this physical history's exact fleet guard. An exact retry can + /// return a fresh observation of the already committed guard. + pub fn enroll_fleet( + &self, + authorization: FleetGroupAuthorization, + expected: LogIndex, + timeout: Duration, + ) -> impl Future> + Send + 'static + { + let deadline = self.control_deadline(timeout); + let owner = self.clone(); + async move { + if !authorization.matches_scope(&owner.history) { + return Err(CellGroupControlError::Refused(RuntimeError::Invalid( + "fleet enrollment history".into(), + ))); + } + owner + .change_membership( + Change::Enroll { + authorization, + expected, + }, + deadline.map_err(CellGroupControlError::Refused)?, + ) + .await + } + } + + /// Commit the exact unused permit's cancellation fence. Admission alone + /// never supplies a fence receipt or permits reservation release. + pub fn fence_fleet_permit( + &self, + authorization: FleetPermitAuthorization, + timeout: Duration, + ) -> impl Future> + Send + 'static + { + let deadline = self.control_deadline(timeout); + let owner = self.clone(); + async move { + if !authorization.matches_scope(&owner.history) { + return Err(CellGroupControlError::Refused(RuntimeError::Invalid( + "fleet fence history".into(), + ))); + } + owner + .change_membership( + Change::Fence(Box::new(authorization)), + deadline.map_err(CellGroupControlError::Refused)?, + ) + .await + } + } + + pub(super) async fn change_membership( + &self, + change: Change, + deadline: u64, + ) -> Result { + use CellGroupControlError::{Refused, Unconfirmed}; + let core = &self.owner.handle.core; + let observed = core + .runtime + .quorum_membership(self.history, deadline) + .await + .map_err(Refused)?; + if change.satisfied(&observed).map_err(Refused)? { + return Ok(observed); + } + if let Change::Repair(repair) = &change { + repair + .check_placement(&core.transport, &observed) + .map_err(Refused)?; + } + let proposal = change.prepare(&observed).map_err(Refused)?; + drop(observed); + // Runtime owns an accepted configuration independently of this waiter. + // A timed-out send/response is conservatively uncertain, never aborted. + let admitted = super::drain::within( + &self.owner.handle.domain, + deadline, + core.runtime.configure(self.history, proposal), + ) + .await + .map_err(Unconfirmed)?; + let observed = core + .runtime + .quorum_membership_after(self.history, admitted.position.index, deadline) + .await + .map_err(Unconfirmed)?; + if !change.satisfied(&observed).map_err(Unconfirmed)? { + return Err(Unconfirmed(RuntimeError::StaleCompletion)); + } + Ok(observed) + } +} diff --git a/crates/cellar/application/consensus/group_client/mod.rs b/crates/cellar/application/consensus/group_client/mod.rs new file mode 100644 index 000000000..1a0c18da5 --- /dev/null +++ b/crates/cellar/application/consensus/group_client/mod.rs @@ -0,0 +1,600 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Canonical SDK handles bind logical cell histories. Physical group controls +//! are separate values; no cell handle owns a group, worker or permanent map entry. +use super::{CommandId, Durability, OperationContract}; +use crate::raft::{ + cell_groups::*, + storage::{LogError, ReplicaLog}, + version::{CellVersionObservation, CellVersionToken}, +}; +use cellar_logic::{ + cell_group::{CellHistory, CellVersion, CommitWitness, GroupHistory}, + consensus::{EngineInstanceId, MemberId, NodeIdentity}, +}; +use std::sync::atomic::Ordering; +use std::{ + future::Future, + sync::{Arc, Mutex}, + time::Duration, +}; + +pub use super::client::CommandOptions; +pub use crate::raft::cell_groups::CellQueryRouting as QueryRouting; + +#[derive(Clone, Debug)] +pub enum QueryConsistency { + Stale, + AtLeast(CellVersionToken), + Linearizable, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum StaleReadPolicy { + RecentSource, + DisconnectedHistorical, +} + +#[derive(Clone, Debug)] +pub struct QueryOptions { + /// Defaults to a fresh quorum read. Historical reads require explicit opt-in. + pub consistency: QueryConsistency, + pub stale_policy: StaleReadPolicy, + pub timeout: Duration, + pub max_attempts: u8, + pub routing: QueryRouting, +} + +impl Default for QueryOptions { + fn default() -> Self { + Self { + consistency: QueryConsistency::Linearizable, + stale_policy: StaleReadPolicy::RecentSource, + timeout: Duration::from_secs(5), + max_attempts: 4, + routing: QueryRouting::Automatic, + } + } +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct CommandReceipt { + pub id: CommandId, + pub result: Vec, + /// Selected logical cell version covering this exact stored receipt. + pub version: CellVersion, + pub witness: CommitWitness, + pub version_token: CellVersionToken, + pub durability: Durability, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct QueryResult { + pub result: Vec, + pub version: CellVersion, + pub witness: CommitWitness, + pub version_token: CellVersionToken, + pub served_by: MemberId, + pub served_by_node: NodeIdentity, +} + +mod actions; +mod close; +mod drain; +mod fleet; +mod hibernation; +mod learner; +mod membership; +mod node; +mod observation; +mod origins; +mod recovery; +mod repair; +mod services; +mod startup; +mod stop; +mod transition; +pub use membership::CellGroupControlError; +pub(crate) use services::{ControlServiceCompletion, ControlServiceKind, ControlServiceLease}; +pub use startup::CellGroupStartupError; +pub(crate) mod resources; +use close::Close; +pub use close::{MutationOutcomes, RaftCloseError, RaftCloseReport, RaftGroupCloseReport}; +pub use resources::{ + CellGroupCollectionFailure, CellGroupCollectionUsage, CellGroupImageUsage, + CellGroupResourceUsage, +}; + +struct Core { + host: CellGroupHost, + controller: CellGroupController, + runtime: CellGroupRuntime, + log: ReplicaLog, + node: NodeIdentity, + engine: EngineInstanceId, + gateways: Mutex>, + resources: resources::Resources, + close_limits: crate::raft::operations::RaftCloseReportLimits, + services: Arc, + managed: Option, + fleet_inspection: Option, + transport: crate::raft::transport::RaftTransport, + archive_io: Option, + node_profile: Option, + startup_bounds: crate::raft::host::RaftHostStartupBounds, + stale_source_timeout_ms: u64, +} + +pub(crate) struct Handle { + pub(crate) domain: crate::asyncrt::DomainToken, + pub(crate) admission: Arc, + core: Arc, + close: Arc, +} + +impl Drop for Handle { + fn drop(&mut self) { + self.close.request(); + } +} + +struct ExternalLifetime { + close: Arc, +} + +impl Drop for ExternalLifetime { + fn drop(&mut self) { + self.close.request(); + } +} + +pub struct RaftCells { + pub(crate) handle: Arc, + // Internal service observers retain metadata without keeping the last + // external SDK owner alive. They cannot indefinitely self-own this node. + lifetime: Option>, +} + +impl Clone for RaftCells { + fn clone(&self) -> Self { + Self { + handle: self.handle.clone(), + lifetime: self.lifetime.clone(), + } + } +} + +/// Physical services transferred together into the SDK's joined close owner. +pub(crate) struct Infrastructure { + pub runtime: CellGroupRuntime, + pub log: ReplicaLog, + pub transport: crate::raft::transport::RaftTransport, + pub archive_io: Option, +} + +impl RaftCells { + pub(crate) fn from_shared( + host: CellGroupHost, + controller: CellGroupController, + infrastructure: Infrastructure, + admission: Arc, + resources: resources::Resources, + options: &crate::raft::host::RaftHostOptions, + ) -> Result { + let Infrastructure { + runtime, + log, + transport, + archive_io, + } = infrastructure; + let domain = log.domain(); + let node = NodeIdentity { + node: log.limits().node, + disk: log.limits().disk, + process: log.limits().process, + }; + let engine = log + .storage_instance() + .map_err(|_| CellGroupControllerError::Invalid("SDK engine identity"))?; + controller.bind_public_admission( + admission.clone(), + crate::asyncrt::services().application_work_registry(), + )?; + let close = Close::new(admission.clone()); + let startup_bounds = node::startup_bounds(&controller, &runtime, &log, options)?; + let core = Arc::new(Core { + host, + controller, + runtime, + log, + node, + engine, + gateways: Mutex::new(Vec::new()), + resources, + close_limits: options.close_report, + services: services::Services::new( + options.max_control_bytes, + options.max_control_requests, + ), + managed: options + .managed + .as_ref() + .map(|managed| managed.authorization.clone()), + fleet_inspection: options + .managed + .as_ref() + .map(|managed| managed.authorization.trust_reader()), + transport, + archive_io, + node_profile: options.node_profile.clone(), + startup_bounds, + stale_source_timeout_ms: options.stale_source_timeout_ms, + }); + let service = Self { + handle: Arc::new(Handle { + domain: domain.clone(), + admission, + core: core.clone(), + close: close.clone(), + }), + lifetime: Some(Arc::new(ExternalLifetime { + close: close.clone(), + })), + }; + close::start(core, close, domain)?; + service + .handle + .core + .transport + .install_group_fleet_gateway(&service) + .map_err(|_| CellGroupControllerError::Invalid("fleet metadata gateway"))?; + Ok(service) + } + + pub(crate) fn from_internal_handle(handle: Arc) -> Self { + Self { + handle, + lifetime: None, + } + } + + pub(crate) fn uses_transport(&self, transport: &crate::raft::transport::RaftTransport) -> bool { + transport.uses_shared_runtime(&self.handle.core.runtime) + } + + pub(crate) fn retain_gateways( + &self, + query: CellQueryGateway, + command: CellCommandGateway, + ) -> Result<(), CellGroupControllerError> { + self.handle + .core + .gateways + .lock() + .map_err(|_| CellGroupControllerError::Closed)? + .push((query, command)); + Ok(()) + } + + pub fn cell(&self, history: CellHistory) -> Result { + self.check_open()?; + let cell = self.handle.core.host.cell(history)?; + Ok(RaftCellHandle { + owner: self.clone(), + cell, + }) + } + + /// Administrative identity is an entire physical group, never a cell alias. + pub fn group( + &self, + history: GroupHistory, + ) -> Result { + self.handle + .core + .log + .cell_group(&history) + .map_err(|_| crate::raft::completion::RuntimeError::NotFound)?; + Ok(RaftGroupHandle { + owner: self.clone(), + history, + }) + } + + fn check_open(&self) -> Result<(), CellDispatchError> { + if self.is_closed() { + return Err(CellDispatchError::Runtime( + crate::raft::completion::RuntimeError::Closed, + )); + } + Ok(()) + } + + /// Immutable identity of this node's log owner. This is metadata, not a live + /// membership, serving, fencing or restart authorization. + pub fn identity(&self) -> NodeIdentity { + self.handle.core.node + } + + pub fn is_closed(&self) -> bool { + self.handle.close.closing.load(Ordering::Acquire) || self.handle.core.host.is_closed() + } + + pub fn resource_usage(&self) -> CellGroupResourceUsage { + self.handle.core.resources.capture(&self.handle.core) + } + + pub fn performance(&self) -> super::RaftPerformanceSnapshot { + self.handle.core.log.performance_snapshot() + } + + pub(crate) fn begin_node_drain(&self) { + let _ = self.handle.admission.seal(); + } + + /// Request joined closure before returning the waiter. Cancellation cannot + /// abandon the physical close or turn unknown mutations into failed writes. + pub fn close( + &self, + ) -> impl Future>> + Send + 'static { + self.handle.close.request(); + let close = self.handle.close.clone(); + async move { close.wait().await } + } +} + +#[derive(Clone)] +pub struct RaftCellHandle { + owner: RaftCells, + cell: CellGroupCellHandle, +} + +/// Effect routing cannot keep its own host alive. Recreate a disposable logical +/// handle from the same immutable namespace registry while that owner exists. +#[derive(Clone)] +pub(crate) struct WeakCellHandle { + owner: std::sync::Weak, + history: CellHistory, +} + +impl WeakCellHandle { + pub(crate) fn upgrade(&self) -> Result, CellDispatchError> { + let Some(handle) = self.owner.upgrade() else { + return Ok(None); + }; + RaftCells { + handle, + lifetime: None, + } + .cell(self.history.clone()) + .map(Some) + } +} + +impl RaftCellHandle { + pub(crate) fn downgrade(&self) -> WeakCellHandle { + WeakCellHandle { + owner: Arc::downgrade(&self.owner.handle), + history: self.history().clone(), + } + } + + pub(crate) fn definition_digest(&self) -> [u8; 32] { + self.cell.definition_digest() + } + + pub fn history(&self) -> &CellHistory { + self.cell.history() + } + + pub fn contract(&self, operation: &str) -> Result { + self.cell.contract(operation) + } + + pub fn command( + &self, + contract: &OperationContract, + input: Vec, + options: CommandOptions, + ) -> impl Future>> + Send + 'static { + let id = options.id; + let accepted = self + .owner + .check_open() + .map_err(|cause| CellCommandError { + command: id, + ambiguous: false, + cause: Box::new(cause), + }) + .and_then(|()| { + self.cell.command( + CellCommandRequest { + id, + contract: contract.clone(), + input, + durability: options.durability, + retry_until_ms: options.retry_until_ms, + }, + CellRoutingLimits { + timeout: options.timeout, + max_attempts: 4, + }, + ) + }); + let owner = self.owner.clone(); + async move { + let _owner = owner; + let response = accepted.map_err(Arc::new)?.response().await?; + let CellVersionObservation { version, witness } = response.observation(); + Ok(CommandReceipt { + id, + result: response.result().to_vec(), + version, + witness, + version_token: response.token().clone(), + durability: response.durability(), + }) + } + } + + pub fn query( + &self, + invocation: super::ProposalId, + contract: &OperationContract, + input: Vec, + options: QueryOptions, + ) -> impl Future>> + Send + 'static { + let accepted = self.owner.check_open().and_then(|()| { + if !matches!(options.consistency, QueryConsistency::Stale) + && options.stale_policy != StaleReadPolicy::RecentSource + { + return Err(CellDispatchError::Invalid( + "stale policy requires stale consistency", + )); + } + let consistency = match options.consistency { + QueryConsistency::Stale => match options.stale_policy { + StaleReadPolicy::RecentSource => CellQueryConsistency::RecentSource { + max_age_ms: self.owner.handle.core.stale_source_timeout_ms, + }, + StaleReadPolicy::DisconnectedHistorical => CellQueryConsistency::Stale, + }, + QueryConsistency::AtLeast(token) => CellQueryConsistency::AtLeast(token), + QueryConsistency::Linearizable => CellQueryConsistency::Linearizable, + }; + self.cell.query_routed( + CellQueryRequest { + invocation, + contract: contract.clone(), + input, + consistency, + }, + CellRoutingLimits { + timeout: options.timeout, + max_attempts: options.max_attempts, + }, + options.routing, + ) + }); + let owner = self.owner.clone(); + async move { + let _owner = owner; + let response = accepted.map_err(Arc::new)?.response().await?; + let CellVersionObservation { version, witness } = response.observation().clone(); + Ok(QueryResult { + result: response.result().to_vec(), + version, + witness, + version_token: response.token().clone(), + served_by: response.replica().member, + served_by_node: response.replica().node_identity(), + }) + } + } + + pub fn enroll( + &self, + request: CellEnrollmentRequest, + timeout: Duration, + ) -> impl Future>> + Send + 'static + { + let response = self.cell.enroll( + request, + CellRoutingLimits { + timeout, + max_attempts: 4, + }, + ); + let owner = self.owner.clone(); + async move { + let _owner = owner; + response.await + } + } + + pub fn receipt( + &self, + request: CellReceiptRequest, + timeout: Duration, + ) -> impl Future>> + Send + 'static + { + let response = self.cell.receipt( + request, + CellRoutingLimits { + timeout, + max_attempts: 4, + }, + ); + let owner = self.owner.clone(); + async move { + let _owner = owner; + response.await + } + } +} + +#[derive(Clone)] +pub struct RaftGroupHandle { + owner: RaftCells, + history: GroupHistory, +} + +impl RaftGroupHandle { + pub fn history(&self) -> GroupHistory { + self.history + } + + pub async fn status(&self) -> Result { + self.owner.handle.core.runtime.status(self.history).await + } + + /// A fresh quorum observation of the complete physical configuration. The + /// deadline starts at invocation, including time before this future polls. + /// The returned metadata is not a lease or permission to release fleet debt. + pub fn quorum_membership( + &self, + timeout: Duration, + ) -> impl Future> + + Send + + 'static { + use crate::raft::completion::RuntimeError; + let deadline = (|| { + if self.owner.is_closed() { + return Err(RuntimeError::Closed); + } + if timeout.is_zero() || timeout > Duration::from_secs(60) { + return Err(RuntimeError::Invalid("group observation timeout".into())); + } + self.owner + .handle + .domain + .mono_ms() + .map_err(|_| RuntimeError::Closed)? + .checked_add(timeout.as_millis() as u64) + .ok_or(RuntimeError::Overloaded) + })(); + let owner = self.owner.clone(); + let history = self.history; + async move { + owner + .handle + .core + .runtime + .quorum_membership(history, deadline?) + .await + } + } + + /// Admission only; observe subsequent election status independently. + pub async fn transfer_leader( + &self, + member: MemberId, + ) -> Result<(), crate::raft::completion::RuntimeError> { + self.owner + .handle + .core + .runtime + .transfer_leader(self.history, member.get()) + .await + } +} diff --git a/crates/cellar/application/consensus/group_client/node.rs b/crates/cellar/application/consensus/group_client/node.rs new file mode 100644 index 000000000..50cbae093 --- /dev/null +++ b/crates/cellar/application/consensus/group_client/node.rs @@ -0,0 +1,370 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Physical metadata uses the shared engine/roster, independently of cell count. +//! Installed transport trust, not a separate root refresh, acknowledges routing. +use super::*; +use crate::raft::{ + host::{ + RaftHostEnrollmentObservation, RaftHostObservation, RaftHostStartupBounds, + RaftPeerNodeObservation, + }, + inspection::{InspectionError, NodeInspection, RESPONSE_RESERVATION}, + physical_inventory::{PhysicalNodeObservation, RESPONSE_BYTES}, +}; +use cellar_logic::consensus::{CommandFailure, NodeId}; + +impl RaftCells { + /// A point-in-time local physical cut. No quorum, capacity activation, + /// physical absence or release permission follows from retaining it. + pub fn observe_node( + &self, + timeout: Duration, + ) -> impl Future> + Send + 'static { + let admitted = self.metadata_admission(timeout, RESPONSE_BYTES); + let owner = self.internal_observer(); + async move { + let (deadline, _bytes) = admitted?; + owner.metadata_live(deadline)?; + let (view, _) = owner.capture_physical_node()?; + owner.metadata_live(deadline)?; + Ok(view) + } + } + + pub(crate) fn observe_physical_node( + &self, + timeout: Duration, + ) -> impl Future> + Send + 'static + { + let admitted = self.metadata_admission(timeout, RESPONSE_BYTES); + let owner = self.internal_observer(); + async move { + let (deadline, _bytes) = admitted?; + owner.metadata_live(deadline)?; + let observed = owner.capture_physical_node()?; + owner.metadata_live(deadline)?; + Ok(observed) + } + } + + /// Inspect the exact enrolled process over the authenticated control lane. + /// A local target uses the same installed-trust checks without loopback I/O. + pub fn inspect_peer_node( + &self, + node: NodeId, + minimum_revision: u64, + timeout: Duration, + ) -> impl Future> + Send + 'static + { + self.inspect_peer_node_mode(node, minimum_revision, timeout, false) + } + + /// Include every durable engine slot, even if unattached or quarantined. + /// An omitted inventory is never interpreted as an empty physical node. + pub fn inspect_peer_physical_inventory( + &self, + node: NodeId, + minimum_revision: u64, + timeout: Duration, + ) -> impl Future> + Send + 'static + { + self.inspect_peer_node_mode(node, minimum_revision, timeout, true) + } + + fn inspect_peer_node_mode( + &self, + node: NodeId, + minimum_revision: u64, + timeout: Duration, + physical: bool, + ) -> impl Future> + Send + 'static + { + // Starting the remote transport at invocation preserves its original + // deadline even if the response future is retained without polling. + let admitted = (|| { + if minimum_revision == 0 + || minimum_revision >= u64::MAX - 2 + || timeout > Duration::from_secs(60) + || self.handle.core.managed.is_none() + { + return Err(InspectionError::Refused); + } + let (deadline, memory) = self + .metadata_admission( + timeout, + if physical || node == self.handle.core.node.node { + RESPONSE_BYTES + } else { + RESPONSE_RESERVATION + }, + ) + .map_err(inspection_error)?; + let started = deadline + .checked_sub(timeout.as_millis() as u64) + .ok_or(InspectionError::Unavailable)?; + let remote = if node != self.handle.core.node.node { + Some(self.handle.core.transport.inspect_fleet_node( + node, + minimum_revision, + deadline, + physical, + )?) + } else { + None + }; + Ok((deadline, started, memory, remote)) + })(); + let owner = self.internal_observer(); + async move { + let (deadline, started, _memory, remote) = admitted?; + owner.metadata_live(deadline).map_err(inspection_error)?; + let (view, inventory) = if let Some(delivery) = remote { + let received = delivery.receive().await?; + if received.inventory.is_some() != physical { + return Err(InspectionError::Refused); + } + (*received.view, received.inventory) + } else { + let (view, inventory) = owner.capture_physical_node().map_err(inspection_error)?; + (view, physical.then(|| Box::new(inventory))) + }; + owner.metadata_live(deadline).map_err(inspection_error)?; + owner + .handle + .core + .transport + .validate_fleet_node_observation(&view, minimum_revision)?; + let sample_age_ms = owner + .handle + .domain + .mono_ms() + .ok() + .and_then(|now| now.checked_sub(started)) + .ok_or(InspectionError::Unavailable)?; + Ok(RaftPeerNodeObservation { + view, + inventory, + sample_age_ms, + }) + } + } + + pub(super) fn metadata_admission( + &self, + timeout: Duration, + bytes: usize, + ) -> Result<(u64, crate::raft::budget::Permit), CommandFailure> { + if crate::application::consensus::class::callback_active() { + return Err(CommandFailure::UnsupportedCapability); + } + let _gate = self + .handle + .admission + .registration(super::super::admission::Access::Cleanup)?; + if self.is_closed() { + return Err(CommandFailure::Unavailable); + } + if timeout.as_millis() == 0 || timeout > Duration::from_secs(60) { + return Err(CommandFailure::VersionUnavailable); + } + let deadline = self + .handle + .domain + .mono_ms() + .ok() + .and_then(|now| now.checked_add(timeout.as_millis() as u64)) + .ok_or(CommandFailure::Unavailable)?; + let memory = self + .control_budget() + .acquire(bytes) + .ok_or(CommandFailure::Overloaded)?; + Ok((deadline, memory)) + } + + pub(super) fn metadata_live(&self, deadline: u64) -> Result<(), CommandFailure> { + if self.is_closed() { + return Err(CommandFailure::Unavailable); + } + if !self.handle.domain.mono_ms().is_ok_and(|now| now < deadline) { + return Err(CommandFailure::VersionUnavailable); + } + Ok(()) + } + + pub(super) fn capture_physical_node(&self) -> Result { + let core = &self.handle.core; + let roster = core + .controller + .registration_inventory() + .map_err(|_| CommandFailure::Unavailable)?; + let mut inventory = core + .log + .cell_group_inventory(None) + .map_err(|_| CommandFailure::Unavailable)?; + if roster + .attached + .iter() + .any(|history| inventory.group_history(history).is_none()) + { + return Err(CommandFailure::Unavailable); + } + inventory + .mark_parked_retired_sources(&roster.retired_sources) + .map_err(|_| CommandFailure::CorruptState)?; + let usage = core.controller.usage(); + let mut profile = core + .node_profile + .clone() + .ok_or(CommandFailure::UnsupportedCapability)?; + profile.draining |= self.handle.admission.application_paused(); + let count = |n| u32::try_from(n).map_err(|_| CommandFailure::Overloaded); + let groups = count(inventory.histories().len())?; + let reserved_bytes = profile + .bytes_per_group + .checked_mul(u64::from(groups)) + .ok_or(CommandFailure::Overloaded)?; + let current = core + .managed + .as_ref() + .map(|startup| { + let current = core + .transport + .installed_fleet_node(core.node.node) + .map_err(|_| CommandFailure::Unavailable)?; + let boot = current + .boot_binding() + .map_err(|_| CommandFailure::CommandConflict)?; + let expected = startup + .boot_binding() + .map_err(|_| CommandFailure::CommandConflict)?; + if boot != expected + || boot.node != core.node.node.bytes() + || boot.disk != core.node.disk.bytes() + || boot.process != core.node.process.bytes() + || boot.engine != core.engine.bytes() + || current.disk() != core.node.disk + || current.storage_instance() != Some(core.engine) + { + return Err(CommandFailure::CommandConflict); + } + Ok(RaftHostEnrollmentObservation::capture(¤t)) + }) + .transpose()?; + let transport = core + .transport + .resource_observer() + .capture() + .ok_or(CommandFailure::Unavailable)?; + let seed_workers = transport + .snapshot_source + .items + .checked_add(usage.active_checkpoints) + .and_then(|n| n.checked_add(usage.retained_preparations)) + .and_then(|n| n.checked_add(core.runtime.usage().read_pins.items)) + .ok_or(CommandFailure::Overloaded)?; + let view = RaftHostObservation { + node: NodeInspection { + identity: core.node, + storage_instance: core.engine, + profile, + groups, + reserved_bytes, + }, + fleet_boot: core + .log + .lifecycle_guard() + .map_err(|_| CommandFailure::Unavailable)? + .map(|guard| guard.boot), + startup_bounds: core.startup_bounds.clone(), + startup_enrollment: core + .managed + .as_ref() + .map(RaftHostEnrollmentObservation::capture), + fleet_trust_revision: current.as_ref().map(|n| n.revision), + current_enrollment: current, + declared_namespaces: count(core.host.usage().namespaces)?, + registered_histories: count(roster.attached.len())?, + dormant_histories: count(roster.quarantined)?, + pending_history_registrations: count( + roster + .pending + .checked_add(usage.provision_waiters) + .ok_or(CommandFailure::Overloaded)?, + )?, + unattached_engine_histories: count( + inventory + .histories() + .len() + .checked_sub(roster.attached.len()) + .ok_or(CommandFailure::CorruptState)?, + )?, + seed_workers: count(seed_workers)?, + }; + // The catalog is append-only for this engine. Refuse a changed slot + // count or roster rather than combine unrelated physical cuts. + if core + .log + .group_count() + .map_err(|_| CommandFailure::Unavailable)? + != inventory.histories().len() + || core + .controller + .registration_inventory() + .map_err(|_| CommandFailure::Unavailable)? + != roster + { + return Err(CommandFailure::Unavailable); + } + if core.managed.is_some() { + inventory + .validate(&view) + .map_err(|_| CommandFailure::CorruptState)?; + } + Ok((view, inventory)) + } +} + +pub(super) fn startup_bounds( + controller: &CellGroupController, + runtime: &CellGroupRuntime, + log: &ReplicaLog, + options: &crate::raft::host::RaftHostOptions, +) -> Result { + let c = controller.limits(); + let r = runtime.limits(); + let l = log.limits(); + let bytes = [ + c.max_command_bytes, + c.max_command_bytes, + options.max_control_bytes, + r.max_request_bytes, + r.max_event_bytes, + r.max_ready_bytes, + r.max_apply_bytes, + r.max_read_pin_bytes, + l.max_pending_bytes, + l.max_cache_bytes, + ]; + let bounded_buffer_bytes = bytes + .into_iter() + .try_fold(0u64, |sum, n| sum.checked_add(u64::try_from(n).ok()?)) + .ok_or(CellGroupControllerError::Invalid("node buffer capacities"))?; + let count = + |n| u32::try_from(n).map_err(|_| CellGroupControllerError::Invalid("node group capacity")); + Ok(RaftHostStartupBounds { + host_groups: count(c.max_groups)?, + consensus_groups: count(r.max_groups)?, + engine_groups: count(l.max_groups)?, + bounded_buffer_bytes, + }) +} + +fn inspection_error(error: CommandFailure) -> InspectionError { + match error { + CommandFailure::Overloaded => InspectionError::Overloaded, + CommandFailure::UnsupportedCapability | CommandFailure::CommandConflict => { + InspectionError::Refused + } + _ => InspectionError::Unavailable, + } +} diff --git a/crates/cellar/application/consensus/group_client/observation.rs b/crates/cellar/application/consensus/group_client/observation.rs new file mode 100644 index 000000000..5071f89bf --- /dev/null +++ b/crates/cellar/application/consensus/group_client/observation.rs @@ -0,0 +1,85 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Advisory physical-group metadata stays separate from logical cell versions. +use super::*; +use crate::raft::completion::RuntimeError; + +impl RaftGroupHandle { + /// Sample this local physical owner and its durable log cache. The deadline + /// starts at invocation. This does not perform ReadIndex or open cell images. + pub fn observe( + &self, + timeout: Duration, + ) -> impl Future> + Send + 'static { + let deadline = self.control_deadline(timeout); + let group = self.clone(); + async move { + let deadline = deadline?; + group.control_live(deadline)?; + let observation = super::drain::within( + &group.owner.handle.domain, + deadline, + group.owner.handle.core.runtime.observe(group.history), + ) + .await?; + group.control_live(deadline)?; + Ok(observation) + } + } + + /// Read bounded partition geometry from the selected in-memory manifest. + /// No manifest walk, SQLite open, or ReadIndex is performed. Archive lag + /// comes from the local archive owner's separate historical observation. + pub fn observe_partition( + &self, + timeout: Duration, + ) -> impl Future> + Send + 'static + { + let deadline = self.control_deadline(timeout); + let group = self.clone(); + async move { + let deadline = deadline?; + group.control_live(deadline)?; + let mut sample = super::drain::within( + &group.owner.handle.domain, + deadline, + group + .owner + .handle + .core + .runtime + .observe_partition(group.history), + ) + .await?; + sample.archive_lag_entries = group.archive_status()?.and_then(|status| { + if status.stopped || status.error.is_some() { + return None; + } + let archived = status + .published + .filter(|cut| cut.route() == &sample.route)?; + sample + .status + .committed + .get() + .checked_sub(archived.position().index.get()) + }); + group.control_live(deadline)?; + Ok(sample) + } + } + + /// Last archive-controller publication, sampled independently of the Raft + /// owner. None means no installed archive owner. Even a published cut grants + /// no retained physical coverage or authority to compact the local log. + pub fn archive_status(&self) -> Result, RuntimeError> { + if self.owner.is_closed() { + return Err(RuntimeError::Closed); + } + self.owner + .handle + .core + .controller + .archive_status(self.history) + } +} diff --git a/crates/cellar/application/consensus/group_client/origins.rs b/crates/cellar/application/consensus/group_client/origins.rs new file mode 100644 index 000000000..241152c6b --- /dev/null +++ b/crates/cellar/application/consensus/group_client/origins.rs @@ -0,0 +1,66 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Explicit namespace origin grants after managed peers publish their boots. +use super::*; +use crate::raft::transport_options::{NamespaceCommandOrigin, NamespaceQueryOrigin}; +use cellar_logic::consensus::{ClusterId, NamespaceId, NodeId}; + +impl RaftCells { + /// Permit this exact peer process to originate reads for an owned + /// namespace. The destination still proves the requested consistency. + /// Managed discovery must verify the current boot and endpoints first. + pub fn grant_query_origin(&self, grant: NamespaceQueryOrigin) -> Result<(), CellDispatchError> { + self.check_open()?; + self.handle + .core + .host + .namespace_directory(grant.cluster, grant.namespace)?; + self.handle + .core + .transport + .grant_query_origin(grant) + .map_err(CellDispatchError::Transport) + } + + /// Permit this exact peer process to originate commands for an owned + /// namespace. Managed discovery must already have verified its current boot + /// and endpoints. Membership supplies no permission. Repeating the exact + /// grant is idempotent; replacing or restoring one requires a newer revision. + /// This is trusted host configuration, not a client-facing authorization API. + pub fn grant_command_origin( + &self, + grant: NamespaceCommandOrigin, + ) -> Result<(), CellDispatchError> { + self.check_open()?; + self.handle + .core + .host + .namespace_directory(grant.cluster, grant.namespace)?; + self.handle + .core + .transport + .grant_command_origin(grant) + .map_err(CellDispatchError::Transport) + } + + /// Revoke namespace write permission while retaining its revision tombstone. + /// Already admitted commands remain owned; revocation is not rollback proof. + pub fn revoke_command_origin( + &self, + cluster: ClusterId, + namespace: NamespaceId, + node: NodeId, + revision: u64, + ) -> Result<(), CellDispatchError> { + self.check_open()?; + self.handle + .core + .host + .namespace_directory(cluster, namespace)?; + self.handle + .core + .transport + .revoke_command_origin(cluster, namespace, node, revision) + .map_err(CellDispatchError::Transport) + } +} diff --git a/crates/cellar/application/consensus/group_client/recovery.rs b/crates/cellar/application/consensus/group_client/recovery.rs new file mode 100644 index 000000000..ff4540ff9 --- /dev/null +++ b/crates/cellar/application/consensus/group_client/recovery.rs @@ -0,0 +1,427 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Node-owned managed recovery seeding and attachment. Shutdown joins work even if +//! the operator stopped waiting for the exact durable receipt. +use super::*; +use crate::{ + management::RaftFleetPreparedRecoveryRange, + raft::{ + catalog::{CatalogError, CellGroupRecoveryError, ManagedRecoveryVoterSeed}, + completion::RuntimeError, + fleet_catalog::{ + FleetFencedTransitionRecovery, FleetRecoveryAttachRequest, + FleetRecoveryRangeReservation, FleetTransitionRecoveryReplacement, + ManagedRecoverySeedSelection, + }, + }, +}; +use cellar_logic::consensus::LogPosition; + +impl RaftGroupHandle { + /// Submit or reconcile this range's exact managed recovery activation. + /// Returns its applied position once the durable receipt is confirmed. + /// A pending proposal returns None. Serving requires separate settlement. + pub fn activate_managed_recovery( + &self, + request: FleetRecoveryAttachRequest, + timeout: Duration, + ) -> impl Future, CellGroupControllerError>> + Send + 'static + { + use super::startup::{admission_error, catalog_error}; + let admitted = self + .owner + .metadata_admission(timeout, 4096) + .map_err(admission_error); + let owner = self.owner.internal_observer(); + let history = self.history; + async move { + let (deadline, _memory) = admitted?; + let host = owner + .handle + .core + .managed + .as_ref() + .ok_or_else(|| catalog_error(CatalogError::ManagedAdmissionRequired))?; + owner.startup_live(host, deadline)?; + let lease = owner + .register_control_service(ControlServiceKind::Recovery, 4096) + .map_err(CatalogError::from) + .map_err(catalog_error)?; + let (reply, result) = tokio::sync::oneshot::channel(); + let domain = owner.handle.domain.clone(); + domain + .spawn_cleanup(async move { + let stop = lease.stop_receiver(); + let activated = Box::pin( + owner.run_managed_recovery_activation(&request, history, deadline, &stop), + ) + .await; + drop(lease); + let _ = reply.send(activated); + }) + .map_err(|_| CellGroupControllerError::Closed)? + .detach(); + domain + .run(crate::asyncrt::timeout_at(deadline, result)) + .await + .map_err(|_| CellGroupControllerError::Closed)? + .map_err(|_| CellGroupControllerError::Runtime(RuntimeError::DeadlineExceeded))? + .map_err(|_| CellGroupControllerError::Closed)? + } + } +} + +impl RaftCells { + /// Open this boot's configured page and image stores and copy the exact + /// retained archive into them. The returned preparation is engine-bound. + #[doc(hidden)] + pub async fn prepare_managed_recovery_seed( + &self, + fenced: &FleetFencedTransitionRecovery, + plan: &FleetTransitionRecoveryReplacement, + reservation: &FleetRecoveryRangeReservation, + recovery: &CellGroupRecoveryPool, + timeout: Duration, + ) -> Result { + use super::startup::{admission_error, catalog_error}; + let (deadline, _memory) = self + .metadata_admission(timeout, 4096) + .map_err(admission_error)?; + let core = &self.handle.core; + let host = core + .managed + .as_ref() + .ok_or_else(|| catalog_error(CatalogError::ManagedAdmissionRequired))?; + let archives = core + .archive_io + .as_ref() + .ok_or(CellGroupControllerError::Invalid("recovery archive owner"))?; + self.startup_live(host, deadline)?; + let lease = self + .register_control_service(ControlServiceKind::Recovery, 4096) + .map_err(CatalogError::from) + .map_err(catalog_error)?; + let stop = lease.stop_receiver(); + let route = reservation.route(); + let index = plan + .births() + .iter() + .position(|birth| birth.route() == route) + .ok_or_else(|| catalog_error(CatalogError::Fenced))?; + let source = fenced + .archives() + .get(index) + .ok_or_else(|| catalog_error(CatalogError::Fenced))?; + let directory = core + .host + .namespace_directory(route.history.cluster, route.history.namespace) + .map_err(|_| CellGroupControllerError::Invalid("recovery seed namespace"))?; + let selected = ManagedRecoverySeedSelection { + fenced, + plan, + reservation, + }; + let (view, inventory) = self.capture_physical_node().map_err(admission_error)?; + Box::pin(host.authorize_managed_recovery_seed( + &directory, + selected, + (&view, inventory.group_history(&route.history).is_some()), + archives, + self.recovery_seed_remaining(host, deadline, &stop)?, + )) + .await + .map_err(CatalogError::from) + .map_err(catalog_error)?; + let domain = &self.handle.domain; + let work = Box::pin(core.controller.prepare_managed_recovery_seed( + &directory, + route, + source.clone(), + (plan.digest(), plan.mutation().operation()), + recovery, + )); + let data = domain + .run(Box::pin(crate::asyncrt::timeout_at(deadline, work))) + .await + .map_err(|_| CellGroupControllerError::Closed)? + .map_err(|_| CellGroupControllerError::Runtime(RuntimeError::DeadlineExceeded))??; + self.recovery_seed_remaining(host, deadline, &stop)?; + Ok(RaftFleetPreparedRecoveryRange::for_node( + plan.digest(), + data, + core.engine, + )) + } + + /// Sync this reserved boot's copy of the prepared recovery genesis. An + /// exact retry can reopen the same durable receipt. Serving is gated by + /// separate directory publication and quorum activation. + #[doc(hidden)] + pub fn seed_managed_recovery_range( + &self, + fenced: Arc, + plan: Arc, + reservation: Arc, + prepared: RaftFleetPreparedRecoveryRange, + timeout: Duration, + ) -> impl Future> + Send + 'static + { + let admitted = self + .metadata_admission(timeout, 4096) + .map_err(super::startup::admission_error); + let owner = self.internal_observer(); + async move { + let (deadline, _memory) = admitted?; + let core = &owner.handle.core; + let host = core.managed.as_ref().ok_or_else(|| { + super::startup::catalog_error(CatalogError::ManagedAdmissionRequired) + })?; + owner.startup_live(host, deadline)?; + let lease = owner + .register_control_service(ControlServiceKind::Recovery, 4096) + .map_err(CatalogError::from) + .map_err(super::startup::catalog_error)?; + let (reply, result) = tokio::sync::oneshot::channel(); + let domain = owner.handle.domain.clone(); + let worker_domain = domain.clone(); + domain + .spawn_cleanup(async move { + let selected = ManagedRecoverySeedSelection { + fenced: &fenced, + plan: &plan, + reservation: &reservation, + }; + let stop = lease.stop_receiver(); + let seeded = Box::pin(worker_domain.run(Box::pin( + owner.run_managed_recovery_seed(selected, prepared, deadline, &stop), + ))) + .await + .unwrap_or(Err(CellGroupControllerError::Closed)); + drop(lease); + let _ = reply.send(seeded); + }) + .map_err(|_| CellGroupControllerError::Closed)? + .detach(); + domain + .run(crate::asyncrt::timeout_at(deadline, result)) + .await + .map_err(|_| CellGroupControllerError::Closed)? + .map_err(|_| CellGroupControllerError::Runtime(RuntimeError::DeadlineExceeded))? + .map_err(|_| CellGroupControllerError::Closed)? + } + } + + /// Attach this voter to a published recovery range using its retained seed. + /// A successful attachment can participate in Raft. Application admission + /// requires a separate committed activation. + pub fn attach_managed_recovery_voter( + &self, + request: FleetRecoveryAttachRequest, + history: GroupHistory, + timeout: Duration, + ) -> impl Future> + Send + 'static { + use super::startup::{admission_error, catalog_error}; + let admitted = self + .metadata_admission(timeout, 4096) + .map_err(admission_error); + let owner = self.internal_observer(); + async move { + let (deadline, _memory) = admitted?; + let core = &owner.handle.core; + let host = core + .managed + .as_ref() + .ok_or_else(|| catalog_error(CatalogError::ManagedAdmissionRequired))?; + owner.startup_live(host, deadline)?; + let directory = core + .host + .namespace_directory(history.cluster, history.namespace) + .map_err(|_| CellGroupControllerError::Invalid("recovery seed namespace"))?; + let lease = owner + .register_control_service(ControlServiceKind::Recovery, 4096) + .map_err(CatalogError::from) + .map_err(catalog_error)?; + let (reply, result) = tokio::sync::oneshot::channel(); + let controller = core.controller.clone(); + let transport = core.transport.clone(); + let host = host.clone(); + let domain = &owner.handle.domain; + domain + .spawn_cleanup(async move { + let stop = lease.stop_receiver(); + let attached = Box::pin(controller.attach_managed_recovery_voter( + &directory, + &request, + history, + &host, + &transport, + (deadline, &stop), + )) + .await; + drop(lease); + let _ = reply.send(attached); + }) + .map_err(|_| CellGroupControllerError::Closed)? + .detach(); + domain + .run(crate::asyncrt::timeout_at(deadline, result)) + .await + .map_err(|_| CellGroupControllerError::Closed)? + .map_err(|_| CellGroupControllerError::Runtime(RuntimeError::DeadlineExceeded))? + .map_err(|_| CellGroupControllerError::Closed)? + } + } + + async fn run_managed_recovery_activation( + &self, + request: &FleetRecoveryAttachRequest, + history: GroupHistory, + deadline: u64, + stop: &tokio::sync::watch::Receiver, + ) -> Result, CellGroupControllerError> { + use super::startup::catalog_error; + let core = &self.handle.core; + let host = core + .managed + .as_ref() + .ok_or_else(|| catalog_error(CatalogError::ManagedAdmissionRequired))?; + let directory = core + .host + .namespace_directory(history.cluster, history.namespace) + .map_err(|_| CellGroupControllerError::Invalid("recovery activation namespace"))?; + let remaining = || self.recovery_seed_remaining(host, deadline, stop); + remaining()?; + let prepare = || { + directory + .prepare_managed_recovery_activation(request, history, &core.log, host, deadline) + }; + let permit = prepare().await.map_err(catalog_error)?; + remaining()?; + let applied = match core.runtime.activate_recovery(permit).await? { + CellGroupRecoveryActivation::Accepted(_) => return Ok(None), + CellGroupRecoveryActivation::Applied(applied) => applied, + }; + remaining()?; + let confirmed = prepare().await.map_err(catalog_error)?; + remaining()?; + directory + .record_managed_recovery_activation(&confirmed, &applied, &core.log) + .await + .map(Some) + .map_err(catalog_error) + } + + async fn run_managed_recovery_seed( + &self, + selected: ManagedRecoverySeedSelection<'_>, + prepared: RaftFleetPreparedRecoveryRange, + deadline: u64, + stop: &tokio::sync::watch::Receiver, + ) -> Result { + use super::startup::{admission_error, catalog_error}; + let core = &self.handle.core; + let host = core + .managed + .as_ref() + .ok_or_else(|| catalog_error(CatalogError::ManagedAdmissionRequired))?; + let archives = core + .archive_io + .as_ref() + .ok_or(CellGroupControllerError::Invalid("recovery archive owner"))?; + let route = selected.reservation.route(); + let target = selected + .plan + .births() + .iter() + .find(|birth| birth.route() == route) + .ok_or_else(|| catalog_error(CatalogError::Fenced))?; + if prepared.plan_digest() != selected.plan.digest() + || prepared.engine() != Some(core.engine) + || prepared.manifest().marker().route() != route + { + return Err(catalog_error(CatalogError::Fenced)); + } + let directory = core + .host + .namespace_directory(route.history.cluster, route.history.namespace) + .map_err(|_| CellGroupControllerError::Invalid("recovery seed namespace"))?; + let remaining = || self.recovery_seed_remaining(host, deadline, stop); + let (view, inventory) = self.capture_physical_node().map_err(admission_error)?; + let already_seeded = inventory.group_history(&route.history).is_some(); + if prepared.data().image_bytes().map_err(archive_error)? > view.node.profile.bytes_per_group + { + return Err(RuntimeError::Overloaded.into()); + } + let authority = Box::pin(host.authorize_managed_recovery_seed( + &directory, + selected, + (&view, already_seeded), + archives, + remaining()?, + )) + .await + .map_err(CatalogError::from) + .map_err(catalog_error)?; + remaining()?; + let (pending, applier) = Box::pin(directory.seed_managed_recovery_voter( + &self.handle.domain, + target, + prepared.data().clone(), + authority, + &core.log, + deadline, + )) + .await + .map_err(|error| match error { + CellGroupRecoveryError::Catalog(error) => catalog_error(error), + CellGroupRecoveryError::Archive(error) => archive_error(error), + })?; + core.controller + .retain_managed_recovery_seed_owner(applier)?; + let (view, inventory) = self.capture_physical_node().map_err(admission_error)?; + if inventory.group_history(&route.history).is_none() { + return Err(catalog_error(CatalogError::LostStableState)); + } + let authority = Box::pin(host.authorize_managed_recovery_seed( + &directory, + selected, + (&view, true), + archives, + remaining()?, + )) + .await + .map_err(CatalogError::from) + .map_err(catalog_error)?; + remaining()?; + directory + .confirm_managed_recovery_voter(target, pending, &authority) + .await + .map_err(catalog_error) + } + + fn recovery_seed_remaining( + &self, + host: &crate::raft::fleet_catalog::FleetHostAuthorization, + deadline: u64, + stop: &tokio::sync::watch::Receiver, + ) -> Result { + if *stop.borrow() { + return Err(CellGroupControllerError::Closed); + } + self.startup_live(host, deadline)?; + let now = self + .handle + .domain + .mono_ms() + .map_err(|_| CellGroupControllerError::Closed)?; + deadline + .checked_sub(now) + .filter(|left| *left > 0) + .map(Duration::from_millis) + .ok_or_else(|| RuntimeError::DeadlineExceeded.into()) + } +} + +fn archive_error(error: CellGroupArchiveError) -> CellGroupControllerError { + CellGroupControllerError::Archive(Arc::new(error.into())) +} diff --git a/crates/cellar/application/consensus/group_client/repair.rs b/crates/cellar/application/consensus/group_client/repair.rs new file mode 100644 index 000000000..4014a4c5d --- /dev/null +++ b/crates/cellar/application/consensus/group_client/repair.rs @@ -0,0 +1,274 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Whole-group repair controls use exact permits and fresh post-apply quorum. +use super::membership::{CellGroupControlError, Change as MembershipChange}; +use super::*; +use crate::raft::{ + completion::RuntimeError, + fleet_catalog::{FleetGroupAuthorization, FleetPermitAuthorization}, + member_state::{FleetPermitPhase, RepairPhase, RepairPlan, RepairStep}, + transport::{RaftTransport, TransportError}, +}; +use cellar_logic::consensus::{CommandFailure, LogIndex}; + +pub(super) enum Change { + Start { + group: FleetGroupAuthorization, + authorization: FleetPermitAuthorization, + plan: RepairPlan, + }, + Advance { + group: FleetGroupAuthorization, + authorization: FleetPermitAuthorization, + step: RepairStep, + expected: LogIndex, + }, +} + +impl Change { + fn authority(&self) -> (&FleetGroupAuthorization, &FleetPermitAuthorization) { + match self { + Self::Start { + group, + authorization, + .. + } + | Self::Advance { + group, + authorization, + .. + } => (group, authorization), + } + } + + pub(super) fn satisfied( + &self, + observed: &CellGroupQuorumMembership, + ) -> Result { + let (group, authorization) = self.authority(); + let permit = authorization.permit(); + if group.binding() != permit.binding() + || group.history_digests() + != ( + observed.root().route().birth_digest, + observed.root().route().schema_digest, + ) + || !group.matches_replica(observed.replica(), observed.membership()) + { + return Err(RuntimeError::Invalid( + "managed group repair authority".into(), + )); + } + let fleet = observed + .membership() + .fleet_status() + .ok_or_else(|| RuntimeError::Invalid("group is not enrolled".into()))?; + if &fleet.binding != permit.binding() { + return Err(RuntimeError::Invalid("foreign repair permit".into())); + } + if let Self::Start { plan, .. } = self { + permit.check_plan(plan)?; + } + if fleet.consumed_sequence < permit.sequence() { + return match self { + Self::Start { .. } => Ok(false), + Self::Advance { .. } => Err(RuntimeError::StaleCompletion), + }; + } + if fleet.consumed_sequence != permit.sequence() + || fleet.permit.as_ref() != Some(permit) + || fleet.phase == Some(FleetPermitPhase::Fenced) + { + return Err(RuntimeError::StaleCompletion); + } + let repair = observed + .membership() + .repair_status()? + .ok_or(RuntimeError::StaleCompletion)?; + permit.check_plan(&repair.plan)?; + let wanted = match self { + Self::Start { plan, .. } => { + if &repair.plan != plan { + return Err(RuntimeError::StaleCompletion); + } + RepairPhase::Seeding + } + Self::Advance { step, .. } => match step { + RepairStep::PromoteAndRemove => RepairPhase::Joint, + RepairStep::FinishJoint => RepairPhase::Complete, + RepairStep::AbortLearner => RepairPhase::Aborted, + }, + }; + if repair.phase == RepairPhase::Aborted && wanted != RepairPhase::Aborted + || wanted == RepairPhase::Aborted + && repair.phase != RepairPhase::Seeding + && repair.phase != RepairPhase::Aborted + { + return Err(RuntimeError::StaleCompletion); + } + Ok(repair.phase == wanted + || wanted == RepairPhase::Seeding + || wanted == RepairPhase::Joint && repair.phase == RepairPhase::Complete) + } + + pub(super) fn prepare( + &self, + observed: &CellGroupQuorumMembership, + ) -> Result { + let (group, authorization) = self.authority(); + if !group.matches_scope(&observed.replica().scope) + || !authorization.matches_scope(&observed.replica().scope) + { + return Err(RuntimeError::Invalid("group repair history".into())); + } + Ok(match self { + Self::Start { plan, .. } => CellGroupMembershipChange::managed_repair_start( + observed.root().route(), + observed.membership(), + observed.configuration(), + plan, + authorization, + )?, + Self::Advance { step, expected, .. } => { + if observed.membership().revision() != *expected { + return Err(RuntimeError::StaleCompletion); + } + CellGroupMembershipChange::managed_repair_step( + observed.root().route(), + observed.membership(), + observed.configuration(), + authorization.permit().repair_id()?, + *step, + authorization, + )? + } + }) + } + + pub(super) fn check_placement( + &self, + transport: &RaftTransport, + observed: &CellGroupQuorumMembership, + ) -> Result<(), RuntimeError> { + if let Self::Start { + group, + authorization, + plan, + } = self + { + let nodes = group.placement_nodes(observed)?; + transport + .check_fleet_placement( + &observed.replica().scope, + &nodes, + &plan.replacement, + authorization, + ) + .map_err(|error| match error { + TransportError::Overloaded => RuntimeError::Overloaded, + TransportError::StalePeer | TransportError::Refused => RuntimeError::NotReady, + TransportError::Runtime(error) => error, + TransportError::Closed => RuntimeError::Closed, + _ => RuntimeError::Invalid("managed repair placement".into()), + })?; + } + Ok(()) + } +} + +fn admission_error(error: CommandFailure) -> RuntimeError { + match error { + CommandFailure::Overloaded => RuntimeError::Overloaded, + CommandFailure::VersionUnavailable => RuntimeError::DeadlineExceeded, + CommandFailure::UnsupportedCapability => { + RuntimeError::Invalid("group repair callback context".into()) + } + _ => RuntimeError::Closed, + } +} + +impl RaftGroupHandle { + /// Start the exact catalog-selected learner repair. Admission is followed by + /// actual durable apply and a fresh quorum observation of the same permit. + pub fn begin_managed_repair( + &self, + group: FleetGroupAuthorization, + authorization: FleetPermitAuthorization, + plan: RepairPlan, + timeout: Duration, + ) -> impl Future> + Send + 'static + { + self.repair_change( + Change::Start { + group, + authorization, + plan, + }, + timeout, + ) + } + + /// Advance or abort the consumed plan at this observed revision. Promotion + /// still requires actual RawNode progress; a route or seed receipt is insufficient. + pub fn advance_managed_repair( + &self, + group: FleetGroupAuthorization, + authorization: FleetPermitAuthorization, + step: RepairStep, + expected: LogIndex, + timeout: Duration, + ) -> impl Future> + Send + 'static + { + self.repair_change( + Change::Advance { + group, + authorization, + step, + expected, + }, + timeout, + ) + } + + fn repair_change( + &self, + change: Change, + timeout: Duration, + ) -> impl Future> + Send + 'static + { + let admitted = self + .owner + .metadata_admission(timeout, 64 * 1024) + .map_err(admission_error); + let owner = self.clone(); + async move { + let (deadline, _memory) = admitted.map_err(CellGroupControlError::Refused)?; + let (group, authorization) = change.authority(); + if !group.matches_scope(&owner.history) || !authorization.matches_scope(&owner.history) + { + return Err(CellGroupControlError::Refused(RuntimeError::Invalid( + "group repair history".into(), + ))); + } + let result = owner + .change_membership(MembershipChange::Repair(Box::new(change)), deadline) + .await?; + if owner.owner.is_closed() { + return Err(CellGroupControlError::Unconfirmed(RuntimeError::Closed)); + } + if owner + .owner + .handle + .domain + .mono_ms() + .map_err(|_| CellGroupControlError::Unconfirmed(RuntimeError::Closed))? + >= deadline + { + return Err(CellGroupControlError::Unconfirmed( + RuntimeError::DeadlineExceeded, + )); + } + Ok(result) + } + } +} diff --git a/crates/cellar/application/consensus/group_client/resources.rs b/crates/cellar/application/consensus/group_client/resources.rs new file mode 100644 index 000000000..a1325a3ae --- /dev/null +++ b/crates/cellar/application/consensus/group_client/resources.rs @@ -0,0 +1,309 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Shared-group observations use the actual shared admissions. Missing physical +//! observers remain unavailable, including after their owners have exited. +use super::*; +use crate::application::consensus::{RaftExtentUsage, RaftImageUsage, RaftTransportResources}; +use cellar_ltx::{generation::GenerationAccounting, internal::NodeExtentObserver}; +use std::fmt::Write; + +#[derive(Clone, Copy, Debug, serde::Serialize)] +pub struct CellGroupImageUsage { + pub materialized: Option, + pub extents: Option, +} + +#[derive(Clone, Copy, Debug, serde::Serialize)] +#[serde(rename_all = "snake_case")] +pub enum CellGroupCollectionFailure { + Inventory, + Worker, + Domain, +} + +/// Advisory physical collection observations, never archive or commit proof. +#[derive(Clone, Copy, Debug, Default, serde::Serialize)] +pub struct CellGroupCollectionUsage { + pub inventories: usize, + pub working_bytes: usize, + pub running: bool, + pub turns: u64, + pub files_reclaimed: u64, + pub deferrals: u64, + pub failures: u64, + pub last_failure: Option, +} + +#[derive(Clone)] +pub(crate) struct CollectionObserver(Arc>); + +impl CollectionObserver { + pub(crate) fn new(inventories: usize, working_bytes: usize) -> Self { + Self(Arc::new(Mutex::new(CellGroupCollectionUsage { + inventories, + working_bytes, + ..Default::default() + }))) + } + + pub(crate) fn update(&self, update: impl FnOnce(&mut CellGroupCollectionUsage)) { + if let Ok(mut state) = self.0.lock() { + update(&mut state); + } + } + + pub(crate) fn capture(&self) -> Option { + self.0.lock().ok().map(|state| *state) + } +} + +#[derive(Clone, Copy, Debug, serde::Serialize)] +pub struct CellGroupResourceUsage { + pub host: CellGroupHostUsage, + pub runtime: CellGroupRuntimeUsage, + pub control: crate::application::consensus::RaftBudgetUsage, + pub sql: Option, + pub serving_images: CellGroupImageUsage, + pub archive_verifier_images: Option, + pub transport: Option, + /// Cached manifest metadata over the bounded durable group inventory. + /// This counts retained cell histories, not current serving authority. + pub retained_manifest_cells: Option, + pub collection: Option, +} + +#[derive(Clone)] +pub(crate) struct Images { + pub materialized: Option, + pub extents: Option, +} + +impl Images { + fn capture(&self) -> CellGroupImageUsage { + CellGroupImageUsage { + materialized: self + .materialized + .as_ref() + .and_then(|v| v.snapshot()) + .map(Into::into), + extents: self + .extents + .as_ref() + .and_then(|v| v.capture()) + .map(Into::into), + } + } +} + +pub(crate) struct Resources { + pub(super) sql: CellGroupSqlPool, + transport: crate::raft::transport::ResourceObserver, + serving: Images, + archive: Option, + collection: Option, +} + +impl Resources { + pub(crate) fn new( + sql: CellGroupSqlPool, + transport: &crate::raft::transport::RaftTransport, + serving: Images, + archive: Option, + collection: Option, + ) -> Self { + Self { + sql, + transport: transport.resource_observer(), + serving, + archive, + collection, + } + } + + pub(super) fn capture(&self, core: &Core) -> CellGroupResourceUsage { + let cells = core.log.cell_groups().ok().and_then(|groups| { + groups.iter().try_fold(0u64, |sum, group| { + let manifest = group.group_manifest().ok()??; + sum.checked_add(u64::from(manifest.root().cells())) + }) + }); + CellGroupResourceUsage { + host: core.host.usage(), + runtime: core.runtime.usage(), + control: crate::application::consensus::RaftBudgetUsage::capture(&core.services.memory), + sql: self.sql.usage().ok(), + serving_images: self.serving.capture(), + archive_verifier_images: self.archive.as_ref().map(Images::capture), + transport: self.transport.capture(), + retained_manifest_cells: cells, + collection: self + .collection + .as_ref() + .and_then(CollectionObserver::capture), + } + } +} + +impl CellGroupResourceUsage { + /// Fixed vocabularies only. Group, cell, tenant and peer identities never + /// enter labels. Unavailable observers emit availability without zero usage. + pub fn prometheus(&self) -> String { + let mut out = String::new(); + sample( + &mut out, + "collection_sample_available", + "", + usize::from(self.collection.is_some()), + ); + if let Some(gc) = self.collection { + for (name, value) in [ + ("collection_running", u64::from(gc.running)), + ("collection_inventories", gc.inventories as u64), + ("collection_working_bound_bytes", gc.working_bytes as u64), + ("collection_turns", gc.turns), + ("collection_reclaimed_files", gc.files_reclaimed), + ("collection_deferrals", gc.deferrals), + ("collection_failures", gc.failures), + ] { + sample(&mut out, name, "", value); + } + } + let c = self.host.controller; + for (name, value) in [ + ("namespaces", self.host.namespaces), + ("namespace_bytes", self.host.namespace_bytes), + ("namespace_capacity_bytes", self.host.namespace_byte_limit), + ("registered_groups", c.groups), + ("quarantined_groups", c.quarantined_groups), + ("retained_calls", c.retained_commands), + ("call_bytes", c.command_bytes), + ("retained_s3", c.retained_s3), + ("s3_bytes", c.s3_bytes), + ("active_apply", c.active_apply), + ( + "provision_configuration_bytes", + c.provision_configuration_bytes, + ), + ("pending_apply", c.pending_apply), + ("active_effects", c.active_effects), + ] { + sample(&mut out, name, "", value); + } + for (lane, usage) in [ + ("ordinary", c.ordinary), + ("archive_settlement", c.archive_settlement), + ] { + let labels = format!("{{lane=\"{lane}\"}}"); + for (name, value) in [ + ("call_admission_items", usage.items), + ("call_admission_bytes", usage.bytes), + ("call_admission_capacity_items", usage.max_items), + ("call_admission_capacity_bytes", usage.max_bytes), + ] { + sample(&mut out, name, &labels, value); + } + } + for (lane, usage) in [ + ("ordinary", self.runtime.ordinary), + ("completion", self.runtime.completions), + ("snapshot_feedback", self.runtime.snapshot_feedback), + ("read_pin", self.runtime.read_pins), + ] { + let labels = format!("{{lane=\"{lane}\"}}"); + for (name, value) in [ + ("runtime_items", usage.items), + ("runtime_bytes", usage.bytes), + ("runtime_capacity_items", usage.max_items), + ("runtime_capacity_bytes", usage.max_bytes), + ] { + sample(&mut out, name, &labels, value); + } + } + sample( + &mut out, + "sql_sample_available", + "", + usize::from(self.sql.is_some()), + ); + if let Some(sql) = self.sql { + for (name, value) in [ + ("sql_workers", sql.running_workers), + ("sql_worker_capacity", sql.worker_capacity), + ("sql_active_cells", sql.active_cells), + ("sql_retained_operations", sql.retained_operations), + ("sql_retained_bytes", sql.retained_bytes), + ("sql_queued_or_running", sql.queued_or_running), + ("sql_apply_operations", sql.apply_retained_operations), + ("sql_apply_bytes", sql.apply_retained_bytes), + ("sql_completion_reservations", sql.completion_reservations), + ("sql_capacity_bytes", sql.limits.node_bytes), + ("sql_group_capacity_bytes", sql.limits.group_bytes), + ("sql_capacity_cells", sql.limits.max_pending_cells), + ("sql_group_capacity_cells", sql.limits.group_pending_cells), + ("sql_queue_closed", usize::from(sql.closing)), + ] { + sample(&mut out, name, "", value); + } + } + for (kind, images) in [ + ("serving", Some(self.serving_images)), + ("archive_verifier", self.archive_verifier_images), + ] { + let labels = format!("{{kind=\"{kind}\"}}"); + sample( + &mut out, + "images_sample_available", + &labels, + usize::from(images.and_then(|i| i.materialized).is_some()), + ); + sample( + &mut out, + "extents_sample_available", + &labels, + usize::from(images.and_then(|i| i.extents).is_some()), + ); + if let Some(i) = images.and_then(|i| i.materialized) { + for (name, value) in [ + ("image_bytes", i.bytes), + ("image_capacity_bytes", i.max_bytes), + ("image_generations", i.generations as u64), + ("image_retained", i.retained as u64), + ] { + sample(&mut out, name, &labels, value); + } + } + if let Some(i) = images.and_then(|i| i.extents) { + for (name, value) in [ + ("extent_bytes", i.bytes), + ("extent_capacity_bytes", i.max_bytes), + ("extent_objects", i.objects as u64), + ("extent_roots", i.roots as u64), + ("extent_workspace_bytes", i.workspace_bytes), + ("extent_quarantined", u64::from(i.quarantined)), + ] { + sample(&mut out, name, &labels, value); + } + } + } + sample( + &mut out, + "manifest_cell_sample_available", + "", + usize::from(self.retained_manifest_cells.is_some()), + ); + if let Some(cells) = self.retained_manifest_cells { + sample(&mut out, "retained_manifest_cells", "", cells); + } + sample( + &mut out, + "transport_sample_available", + "", + usize::from(self.transport.is_some()), + ); + out + } +} + +fn sample(out: &mut String, name: &str, labels: &str, value: impl std::fmt::Display) { + let _ = writeln!(out, "cellar_cell_groups_{name}{labels} {value}"); +} diff --git a/crates/cellar/application/consensus/group_client/services.rs b/crates/cellar/application/consensus/group_client/services.rs new file mode 100644 index 000000000..ddb6da096 --- /dev/null +++ b/crates/cellar/application/consensus/group_client/services.rs @@ -0,0 +1,277 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Bounded per-node control services. Closing signals every admitted worker, +//! then joins its actual lifetime before consensus/storage shutdown proceeds. +use super::*; +use crate::raft::{ + budget::{Budget, Permit}, + fleet_catalog::FleetCatalogError, +}; +use tokio::sync::{watch, Notify}; + +#[derive(Clone, Copy)] +pub(crate) enum ControlServiceKind { + Movement, + Restart, + Directory, + Recovery, + Reclamation, + Transfer, +} + +impl ControlServiceKind { + fn slot(self) -> usize { + match self { + Self::Movement => 0, + Self::Restart => 1, + Self::Directory => 2, + Self::Recovery => 3, + Self::Reclamation => 4, + Self::Transfer => 5, + } + } +} + +struct Entry { + id: u64, + stop: watch::Sender, +} + +#[derive(Default)] +struct State { + closing: bool, + next: u64, + slots: [Option; 6], +} + +pub(super) struct Services { + state: Mutex, + finished: Notify, + pub(super) memory: Arc, +} + +pub(crate) struct ControlServiceLease { + owner: Arc, + kind: ControlServiceKind, + id: u64, + stop: watch::Receiver, + memory: Option, +} + +pub(crate) struct ControlServiceCompletion(watch::Sender); + +impl ControlServiceCompletion { + pub(crate) fn new(done: watch::Sender) -> Self { + Self(done) + } +} + +impl Drop for ControlServiceCompletion { + fn drop(&mut self) { + self.0.send_replace(true); + } +} + +impl ControlServiceLease { + pub(crate) fn stop_receiver(&self) -> watch::Receiver { + self.stop.clone() + } +} + +impl Drop for ControlServiceLease { + fn drop(&mut self) { + // Release the worker's memory before advertising completed physical + // ownership. A cancelled close waiter does not remove this entry. + self.memory.take(); + let mut state = self.owner.state.lock().unwrap_or_else(|p| p.into_inner()); + if state.slots[self.kind.slot()] + .as_ref() + .is_some_and(|e| e.id == self.id) + { + state.slots[self.kind.slot()] = None; + } + drop(state); + self.owner.finished.notify_waiters(); + } +} + +impl Services { + pub(super) fn new(bytes: usize, items: usize) -> Arc { + Arc::new(Self { + state: Mutex::new(State::default()), + finished: Notify::new(), + memory: Budget::new(bytes, items), + }) + } + + fn register( + self: &Arc, + kind: ControlServiceKind, + bytes: usize, + ) -> Result { + let mut state = self.state.lock().map_err(|_| FleetCatalogError::Closed)?; + if state.closing { + return Err(FleetCatalogError::Closed); + } + if state.slots[kind.slot()].is_some() { + return Err(FleetCatalogError::Overloaded); + } + let memory = self + .memory + .acquire(bytes) + .ok_or(FleetCatalogError::Overloaded)?; + state.next = state.next.checked_add(1).ok_or(FleetCatalogError::Limit)?; + let id = state.next; + let (stop, receiver) = watch::channel(false); + state.slots[kind.slot()] = Some(Entry { id, stop }); + Ok(ControlServiceLease { + owner: self.clone(), + kind, + id, + stop: receiver, + memory: Some(memory), + }) + } + + pub(super) async fn close(&self) { + loop { + let notified = self.finished.notified(); + tokio::pin!(notified); + notified.as_mut().enable(); + { + let mut state = self.state.lock().unwrap_or_else(|p| p.into_inner()); + state.closing = true; + for entry in state.slots.iter().flatten() { + entry.stop.send_replace(true); + } + if state.slots.iter().all(Option::is_none) { + return; + } + } + notified.await; + } + } +} + +impl RaftCells { + /// Node installs its existing stop owner from the actual bound boot, after + /// startup has synced the lifecycle guard. No caller-provided preboot clone. + pub(crate) fn fleet_stop_services( + &self, + ) -> Result, FleetCatalogError> { + let core = &self.handle.core; + let Some(authorization) = &core.managed else { + return Ok(None); + }; + let boot = authorization.boot_binding()?; + let signer = crate::asyncrt::services() + .peer_probe_signer() + .map_err(|_| FleetCatalogError::Closed)?; + if core + .log + .lifecycle_guard() + .map_err(|_| FleetCatalogError::Closed)? + .is_none_or(|guard| guard.boot != boot) + || boot.node != core.node.node.bytes() + || boot.disk != core.node.disk.bytes() + || boot.process != core.node.process.bytes() + || boot.engine != core.engine.bytes() + || boot.join_key != signer.public_key_bytes() + { + return Err(FleetCatalogError::Conflict); + } + Ok(Some(crate::raft::fleet_stop::Services { + authorization: authorization.clone(), + log: core.log.clone(), + signer, + })) + } + + // Internal loops must not retain the last external SDK lifetime. An active + // service is joined by the registry, independently of its public handle. + pub(crate) fn internal_observer(&self) -> Self { + Self { + handle: self.handle.clone(), + lifetime: None, + } + } + + pub(crate) fn control_budget(&self) -> &Arc { + &self.handle.core.services.memory + } + + pub(crate) fn register_control_service( + &self, + kind: ControlServiceKind, + bytes: usize, + ) -> Result { + if self.is_closed() || self.handle.admission.is_draining() { + return Err(FleetCatalogError::Closed); + } + self.handle.core.services.register(kind, bytes) + } + + async fn installed_host( + &self, + timeout: Duration, + ) -> Result { + if self.is_closed() { + return Err(FleetCatalogError::Closed); + } + let core = &self.handle.core; + let host = core.managed.as_ref().ok_or(FleetCatalogError::Invalid( + "control service requires managed host", + ))?; + let expected = host.boot_binding()?; + let current = host.refresh(timeout).await?; + let boot = current.boot_binding()?; + if boot != expected + || boot.node != core.node.node.bytes() + || boot.disk != core.node.disk.bytes() + || boot.process != core.node.process.bytes() + || boot.engine != core.engine.bytes() + || core + .log + .lifecycle_guard() + .map_err(|_| FleetCatalogError::Closed)? + .is_none_or(|guard| !guard.admits_group_start(&boot)) + { + return Err(FleetCatalogError::Conflict); + } + Ok(current) + } + + pub(crate) async fn start_movement_worker( + &self, + mut work: crate::raft::fleet_catalog::MovementServiceWork, + timeout: Duration, + ) -> Result<(), FleetCatalogError> { + let host = self.installed_host(timeout).await?; + work.admit(&host)?; + self.handle + .domain + .spawn_cleanup(async move { work.run().await }) + .map_err(|_| FleetCatalogError::Closed)? + .detach(); + Ok(()) + } + + pub(crate) async fn start_restart_worker( + &self, + mut work: crate::raft::fleet_catalog::RestartServiceWork, + timeout: Duration, + ) -> Result<(), FleetCatalogError> { + let host = self.installed_host(timeout).await?; + work.admit(&host)?; + self.handle + .domain + .spawn_cleanup(async move { work.run().await }) + .map_err(|_| FleetCatalogError::Closed)? + .detach(); + Ok(()) + } +} + +#[cfg(test)] +#[path = "services_tests.rs"] +mod tests; diff --git a/crates/cellar/application/consensus/group_client/services_tests.rs b/crates/cellar/application/consensus/group_client/services_tests.rs new file mode 100644 index 000000000..ce4b1b784 --- /dev/null +++ b/crates/cellar/application/consensus/group_client/services_tests.rs @@ -0,0 +1,37 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; + +#[tokio::test] +async fn close_retains_service_admission_until_the_actual_worker_exits() { + let services = Services::new(128, 1); + let worker = services.register(ControlServiceKind::Movement, 64).unwrap(); + let stop = worker.stop_receiver(); + assert!(matches!( + services.register(ControlServiceKind::Movement, 64), + Err(FleetCatalogError::Overloaded) + )); + assert!(matches!( + services.register(ControlServiceKind::Restart, 64), + Err(FleetCatalogError::Overloaded) + )); + let mut closing = Box::pin(services.close()); + assert!( + tokio::time::timeout(Duration::from_millis(30), &mut closing) + .await + .is_err() + ); + assert!(*stop.borrow()); + assert_eq!(services.memory.usage(), (64, 1)); + drop(closing); + // Cancelling a close waiter neither removes the live worker nor reopens + // admission. The independently owned cleanup can resume the same join. + assert!(matches!( + services.register(ControlServiceKind::Restart, 1), + Err(FleetCatalogError::Closed) + )); + assert_eq!(services.memory.usage(), (64, 1)); + drop(worker); + services.close().await; + assert_eq!(services.memory.usage(), (0, 0)); +} diff --git a/crates/cellar/application/consensus/group_client/startup.rs b/crates/cellar/application/consensus/group_client/startup.rs new file mode 100644 index 000000000..9606dff00 --- /dev/null +++ b/crates/cellar/application/consensus/group_client/startup.rs @@ -0,0 +1,460 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Initial voters use the controller's retained whole-group preparation owner. +//! Attachment receipts grant neither a fresh quorum nor application admission. +use super::*; +use crate::raft::{ + catalog::CatalogError, + completion::RuntimeError, + fleet_catalog::{FleetGroupAuthorization, FleetHostAuthorization}, + member_state::FleetMembershipStatus, +}; +use cellar_logic::consensus::{ClusterId, CommandFailure, NamespaceId}; + +const STARTUP_BYTES: usize = 4 * 1024 * 1024; + +/// A refusal submitted no preparation in this invocation. An unconfirmed result +/// can follow durable genesis or attachment; retry the same physical history. +#[derive(Clone, Debug)] +pub enum CellGroupStartupError { + Refused(CellGroupControllerError), + Unconfirmed(CellGroupControllerError), +} + +impl std::fmt::Display for CellGroupStartupError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::Refused(error) => write!(f, "group startup refused: {error}"), + Self::Unconfirmed(error) => write!(f, "group startup unconfirmed: {error}"), + } + } +} + +impl std::error::Error for CellGroupStartupError { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + Some(match self { + Self::Refused(error) | Self::Unconfirmed(error) => error, + }) + } +} + +pub(super) fn admission_error(error: CommandFailure) -> CellGroupControllerError { + match error { + CommandFailure::Overloaded => RuntimeError::Overloaded.into(), + CommandFailure::VersionUnavailable => RuntimeError::DeadlineExceeded.into(), + CommandFailure::UnsupportedCapability => { + CellGroupControllerError::Invalid("group startup callback context") + } + _ => CellGroupControllerError::Closed, + } +} + +pub(super) fn catalog_error(error: CatalogError) -> CellGroupControllerError { + CellGroupControllerError::Catalog(Arc::new(error)) +} + +fn startup_directory_error(error: CellDispatchError) -> CellGroupControllerError { + match error { + CellDispatchError::Runtime(error) => error.into(), + _ => CellGroupControllerError::Invalid("group startup namespace capability"), + } +} + +impl RaftCells { + /// Start the exact operator-provisioned first group on this initial voter. + /// Wait for the exact rooted archive and registry, attach or validate the + /// local replica, and establish fresh quorum behind one startup deadline. + /// Call this from daemon readiness before opening application ingress. + /// The result is a point-in-time barrier and supplies no serving lease. + pub fn start_managed_first_group( + &self, + history: GroupHistory, + timeout: Duration, + ) -> impl Future> + Send + 'static { + let startup = + self.start_first_group(history.cluster, history.namespace, Some(history), timeout); + async move { startup.await.map(|_| ()) } + } + + /// Wait for the managed namespace's first full-range group and establish + /// fresh quorum on this initial voter. The catalog selects the history, + /// and the rooted archive intent must match its live birth. Call this + /// before binding ingress for a newly provisioned namespace. + pub fn start_managed_namespace_first_group( + &self, + cluster: ClusterId, + namespace: NamespaceId, + timeout: Duration, + ) -> impl Future> + Send + 'static { + self.start_first_group(cluster, namespace, None, timeout) + } + + fn start_first_group( + &self, + cluster: ClusterId, + namespace: NamespaceId, + history: Option, + timeout: Duration, + ) -> impl Future> + Send + 'static { + use CellGroupStartupError::{Refused, Unconfirmed}; + let admitted = (|| { + let (deadline, memory) = self + .metadata_admission(timeout, STARTUP_BYTES) + .map_err(admission_error)?; + let core = &self.handle.core; + let host = core + .managed + .clone() + .ok_or(CellGroupControllerError::Invalid( + "managed group startup owner", + ))?; + let archives = core + .archive_io + .clone() + .ok_or(CellGroupControllerError::Invalid( + "managed first group archive pool", + ))?; + let directory = core + .host + .namespace_directory(cluster, namespace) + .map_err(startup_directory_error)?; + Ok((deadline, memory, host, directory, archives)) + })(); + let owner = self.internal_observer(); + async move { + let (deadline, memory, host, directory, archives) = admitted.map_err(Refused)?; + let domain = &owner.handle.domain; + let mut backoff_ms = 25u64; + let authorization = loop { + owner.startup_live(&host, deadline).map_err(Refused)?; + let remaining = owner.startup_remaining(deadline).map_err(Refused)?; + let result = domain + .run(host.observe_first_group_archive( + directory.clone(), + archives.clone(), + history, + remaining, + )) + .await + .map_err(|_| Refused(CellGroupControllerError::Closed))?; + match result { + Ok(Some(authorization)) => break authorization, + Ok(None) | Err(crate::raft::fleet_catalog::FleetCatalogError::Overloaded) => (), + Err(error) => return Err(Refused(catalog_error(CatalogError::from(error)))), + } + let now = domain + .mono_ms() + .map_err(|_| Refused(CellGroupControllerError::Closed))?; + domain + .sleep_until(deadline.min(now.saturating_add(backoff_ms))) + .map_err(|_| Refused(CellGroupControllerError::Closed))? + .await; + backoff_ms = (backoff_ms * 2).min(250); + }; + let history = authorization + .binding() + .scope() + .map_err(|e| Refused(catalog_error(CatalogError::Log(e))))?; + drop(memory); + let remaining = owner.startup_remaining(deadline).map_err(Refused)?; + let attachment = owner + .activate_managed_initial_group(authorization, remaining) + .await?; + let core = &owner.handle.core; + let observed = loop { + owner.startup_live(&host, deadline).map_err(Unconfirmed)?; + match core.runtime.quorum_membership(history, deadline).await { + Ok(observed) => break observed, + // An election invalidates the in-flight barrier. Start a + // new one under this invocation's original deadline. + Err(RuntimeError::NotLeader) => (), + Err(error) => return Err(Unconfirmed(error.into())), + } + }; + let remaining = owner.startup_remaining(deadline).map_err(Unconfirmed)?; + let fresh = owner + .handle + .domain + .run(host.group_authorization(&history, remaining)) + .await + .map_err(|_| Unconfirmed(CellGroupControllerError::Closed))? + .map_err(|error| Unconfirmed(catalog_error(CatalogError::from(error))))?; + let local = observed.replica(); + if fresh.binding() != &attachment.binding + || observed.engine() != core.engine + || local.node_identity() != core.node + || !observed + .configuration() + .voters + .contains(&local.member.get()) + || !fresh.matches_replica(local, observed.membership()) + || fresh.history_digests() + != ( + observed.root().route().birth_digest, + observed.root().route().schema_digest, + ) + { + return Err(Unconfirmed(CellGroupControllerError::Invalid( + "managed first group quorum membership", + ))); + } + owner.startup_live(&host, deadline).map_err(Unconfirmed)?; + Ok(history) + } + } + + /// Recover or initialize this exact admitted initial voter. Namespace lookup + /// allocates no cell/group identity; the rooted directory selects its birth. + /// Success describes a durable local attachment, never quorum readiness. + pub fn activate_managed_initial_group( + &self, + authorization: FleetGroupAuthorization, + timeout: Duration, + ) -> impl Future> + Send + 'static + { + let admitted = (|| { + let (deadline, memory) = self + .metadata_admission(timeout, STARTUP_BYTES) + .map_err(admission_error)?; + let host = + self.handle + .core + .managed + .clone() + .ok_or(CellGroupControllerError::Invalid( + "managed group startup owner", + ))?; + let history = authorization + .binding() + .scope() + .map_err(|e| catalog_error(CatalogError::Log(e)))?; + if !authorization.allows_managed_genesis() + || authorization.binding().fleet() != host.fleet() + { + return Err(CellGroupControllerError::Invalid( + "managed genesis authority", + )); + } + let directory = self + .handle + .core + .host + .directory(history) + .map_err(startup_directory_error)?; + Ok((deadline, memory, host, history, directory)) + })(); + let owner = self.internal_observer(); + async move { + use CellGroupStartupError::{Refused, Unconfirmed}; + let (deadline, _memory, host, history, directory) = admitted.map_err(Refused)?; + let mut submitted = false; + let work = async { + owner.startup_live(&host, deadline).map_err(Refused)?; + let boot = host + .boot_binding() + .map_err(CatalogError::from) + .map_err(catalog_error) + .map_err(Refused)?; + let namespace_owner = directory + .namespace() + .cell_group_owner() + .await + .map_err(catalog_error) + .map_err(Refused)?; + if namespace_owner + .is_none_or(|o| o.fleet != host.fleet() || o.catalog != boot.catalog) + { + return Err(Refused(catalog_error( + CatalogError::ManagedAdmissionRequired, + ))); + } + let birth = directory + .materialize_selected_history(history) + .await + .map_err(catalog_error) + .map_err(Refused)?; + if !authorization.check_birth(&directory, &birth) + || !birth + .voters() + .iter() + .any(|m| m.node == owner.handle.core.node.node) + { + return Err(Refused(CellGroupControllerError::Invalid( + "initial group voter birth", + ))); + } + let fresh = host + .group_authorization( + &history, + owner.startup_remaining(deadline).map_err(Refused)?, + ) + .await + .map_err(CatalogError::from) + .map_err(catalog_error) + .map_err(Refused)?; + if fresh.binding() != authorization.binding() + || !fresh.check_birth(&directory, &birth) + || !fresh.allows_managed_genesis() + { + return Err(Refused(CellGroupControllerError::Invalid( + "initial group authorization changed", + ))); + } + owner.startup_live(&host, deadline).map_err(Refused)?; + let call = owner + .handle + .core + .controller + .provision_managed_initial( + directory.clone(), + birth.route().clone(), + host.clone(), + deadline, + ) + .map_err(Refused)?; + submitted = true; + call.response() + .await + .map_err(|error| Unconfirmed((*error).clone()))?; + owner.startup_live(&host, deadline).map_err(Unconfirmed)?; + // Reauthorize after attachment. The saved admission and a successful + // controller reply cannot substitute for this current root check. + let fresh = host + .group_authorization( + &history, + owner.startup_remaining(deadline).map_err(Unconfirmed)?, + ) + .await + .map_err(CatalogError::from) + .map_err(catalog_error) + .map_err(Unconfirmed)?; + if fresh.binding() != authorization.binding() + || !fresh.check_birth(&directory, &birth) + || !fresh.allows_managed_genesis() + { + return Err(Unconfirmed(CellGroupControllerError::Invalid( + "initial group authorization changed", + ))); + } + let core = &owner.handle.core; + if core + .controller + .group_failure(history) + .map_err(CellGroupControllerError::from) + .map_err(Unconfirmed)? + .is_some() + { + return Err(Unconfirmed(RuntimeError::GroupQuarantined.into())); + } + let group = core + .log + .cell_group(&history) + .map_err(|e| Unconfirmed(RuntimeError::from(e).into()))?; + let marker = group + .group_manifest() + .map_err(|e| Unconfirmed(RuntimeError::from(e).into()))? + .ok_or(Unconfirmed(CellGroupControllerError::Invalid( + "missing attached group root", + )))?; + if marker.route() != birth.route() || !marker.is_active() { + return Err(Unconfirmed(CellGroupControllerError::Invalid( + "initial group root", + ))); + } + let status = { + let cache = group + .read() + .map_err(|e| Unconfirmed(RuntimeError::from(e).into()))?; + if !crate::raft::member_state::members(&cache.conf_state) + .contains(&group.identity().member.get()) + || !fresh.matches_replica(group.identity(), &cache.membership) + { + return Err(Unconfirmed(CellGroupControllerError::Invalid( + "initial group membership", + ))); + } + cache.membership.fleet_status().ok_or(Unconfirmed( + CellGroupControllerError::Invalid("missing initial fleet guard"), + ))? + }; + if &status.binding != authorization.binding() { + return Err(Unconfirmed(CellGroupControllerError::Invalid( + "initial fleet binding", + ))); + } + owner.startup_live(&host, deadline).map_err(Unconfirmed)?; + Ok(status) + }; + match owner + .handle + .core + .transport + .seed_control_until(deadline, async { + // Public callers may poll on another runtime. Catalog + // deadlines and I/O must still enter this installed owner. + owner + .handle + .domain + .run(work) + .await + .map_err(|_| crate::raft::transport::TransportError::Closed) + }) + .await + { + Ok(result) => result, + Err(error) => { + let error = CellGroupControllerError::Transport(error); + Err(if submitted { + Unconfirmed(error) + } else { + Refused(error) + }) + } + } + } + } + + pub(super) fn startup_remaining( + &self, + deadline: u64, + ) -> Result { + let now = self + .handle + .domain + .mono_ms() + .map_err(|_| CellGroupControllerError::Closed)?; + let remaining = deadline + .checked_sub(now) + .filter(|ms| *ms > 0 && *ms <= 60_000) + .ok_or(CellGroupControllerError::Runtime( + RuntimeError::DeadlineExceeded, + ))?; + Ok(Duration::from_millis(remaining)) + } + + pub(super) fn startup_live( + &self, + host: &FleetHostAuthorization, + deadline: u64, + ) -> Result<(), CellGroupControllerError> { + self.metadata_live(deadline).map_err(admission_error)?; + let boot = host + .boot_binding() + .map_err(CatalogError::from) + .map_err(catalog_error)?; + let core = &self.handle.core; + if boot.node != core.node.node.bytes() + || boot.disk != core.node.disk.bytes() + || boot.process != core.node.process.bytes() + || boot.engine != core.engine.bytes() + || core + .log + .lifecycle_guard() + .map_err(RuntimeError::from)? + .is_none_or(|guard| !guard.admits_group_start(&boot)) + { + return Err(catalog_error(CatalogError::Fenced)); + } + Ok(()) + } +} diff --git a/crates/cellar/application/consensus/group_client/stop.rs b/crates/cellar/application/consensus/group_client/stop.rs new file mode 100644 index 000000000..3dc3eba11 --- /dev/null +++ b/crates/cellar/application/consensus/group_client/stop.rs @@ -0,0 +1,66 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Exact-boot lifecycle RPCs use the node's authenticated transport. Its receiver +//! owns admitted mutations; losing the origin waiter cannot establish rollback. +use super::*; +use crate::raft::{ + fleet_catalog::FleetCatalogError, + fleet_stop::{FleetStopAcceptance, FleetStopRequest}, + transport::fleet_stops::STOP_REQUEST_BYTES, +}; + +impl RaftCells { + /// Submit the exact persisted stop/quiesce/fence request. No business cell + /// or local group is required. Acceptance proves historical guard admission, + /// independently of physical closure and terminal removal certificates. + /// A lost or expired response is Unconfirmed; reconcile the same request. + pub fn request_peer_fleet_stop( + &self, + request: FleetStopRequest, + minimum_revision: u64, + timeout: Duration, + ) -> impl Future> + Send + 'static { + // Capture deadline and admission before returning the waiter. The + // transport starts owned bounded work at invocation, not at first poll. + let domain = self.handle.domain.clone(); + let admitted = (|| { + if self.is_closed() { + return Err(FleetCatalogError::Closed); + } + if self.handle.core.managed.is_none() { + return Err(FleetCatalogError::Invalid("no managed fleet origin")); + } + if request.selector.validate().is_err() + || minimum_revision == 0 + || minimum_revision >= u64::MAX - 2 + || timeout.as_millis() == 0 + || timeout > Duration::from_secs(60) + { + return Err(FleetCatalogError::Invalid("peer stop request")); + } + let deadline = domain + .mono_ms() + .map_err(|_| FleetCatalogError::Closed)? + .checked_add(timeout.as_millis() as u64) + .ok_or(FleetCatalogError::Limit)?; + let memory = self + .control_budget() + .acquire(STOP_REQUEST_BYTES) + .ok_or(FleetCatalogError::Overloaded)?; + let delivery = self.handle.core.transport.request_fleet_stop( + request, + minimum_revision, + deadline, + )?; + Ok((deadline, memory, delivery)) + })(); + async move { + let (deadline, _memory, delivery) = admitted?; + let received = delivery.receive().await?; + if !domain.mono_ms().is_ok_and(|now| now < deadline) { + return Err(FleetCatalogError::Unconfirmed); + } + Ok(*received.acceptance) + } + } +} diff --git a/crates/cellar/application/consensus/group_client/transition.rs b/crates/cellar/application/consensus/group_client/transition.rs new file mode 100644 index 000000000..1a6d286e3 --- /dev/null +++ b/crates/cellar/application/consensus/group_client/transition.rs @@ -0,0 +1,352 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Bind managed transition decisions to an existing local voter. +use super::*; +use crate::raft::{ + catalog::CatalogError, completion::RuntimeError, + fleet_catalog::ReopenedManagedCellGroupTransition, +}; +use cellar_logic::cell_group::GroupRoute; + +const SEAL_BYTES: usize = 4 * 1024 * 1024; + +impl RaftGroupHandle { + /// Observe a durable fresh-term commit on a pending managed transfer child. + /// The resulting proof supplies no application or routing authority. + pub fn managed_pending_transfer_quorum( + &self, + timeout: Duration, + ) -> impl Future> + Send + 'static + { + let admitted = self + .owner + .metadata_admission(timeout, 4096) + .map_err(super::startup::admission_error); + let owner = self.owner.internal_observer(); + let history = self.history; + async move { + let (deadline, _memory) = admitted?; + let core = &owner.handle.core; + let host = core.managed.as_ref().ok_or_else(|| { + super::startup::catalog_error(CatalogError::ManagedAdmissionRequired) + })?; + owner.startup_live(host, deadline)?; + let proof = owner + .handle + .domain + .run(crate::asyncrt::timeout_at( + deadline, + core.runtime.pending_transfer_quorum(history), + )) + .await + .map_err(|_| CellGroupControllerError::Closed)? + .map_err(|_| CellGroupControllerError::Runtime(RuntimeError::DeadlineExceeded))??; + owner.startup_live(host, deadline)?; + Ok(proof) + } + } + + /// Submit or observe the exact activation of this published transfer child. + /// Accepted identifies a proposal. Applied proves durable activation. + /// Retry the same transition to resolve an unknown outcome. + pub fn activate_managed_transfer_child<'a>( + &self, + transition: &'a ReopenedManagedCellGroupTransition, + timeout: Duration, + ) -> impl Future> + Send + 'a + { + let admitted = self + .owner + .metadata_admission(timeout, 4096) + .map_err(super::startup::admission_error); + let owner = self.owner.internal_observer(); + let history = self.history; + async move { + let (deadline, _memory) = admitted?; + let core = &owner.handle.core; + let host = core.managed.as_ref().ok_or_else(|| { + super::startup::catalog_error(CatalogError::ManagedAdmissionRequired) + })?; + let directory = core + .host + .namespace_directory(history.cluster, history.namespace) + .map_err(|_| CellGroupControllerError::Invalid("transfer child namespace"))?; + owner + .handle + .domain + .run(Box::pin(crate::asyncrt::timeout_at( + deadline, + Box::pin(async { + owner.startup_live(host, deadline)?; + let permit = directory + .prepare_managed_transfer_activation( + transition, history, &core.log, host, deadline, + ) + .await + .map_err(super::startup::catalog_error)?; + owner.startup_live(host, deadline)?; + core.runtime + .activate_transfer(permit) + .await + .map_err(Into::into) + }), + ))) + .await + .map_err(|_| CellGroupControllerError::Closed)? + .map_err(|_| CellGroupControllerError::Runtime(RuntimeError::DeadlineExceeded))? + } + } + + /// Retire the sealed parent after the directory selects every child + /// activation. A durable tombstone grants no authority to release holds. + pub fn retire_managed_source<'a>( + &self, + transition: &'a ReopenedManagedCellGroupTransition, + timeout: Duration, + ) -> impl Future> + Send + 'a + { + let admitted = self + .owner + .metadata_admission(timeout, 4096) + .map_err(super::startup::admission_error); + let owner = self.owner.internal_observer(); + let history = self.history; + async move { + let (deadline, _memory) = admitted?; + let core = &owner.handle.core; + let host = core.managed.as_ref().ok_or_else(|| { + super::startup::catalog_error(CatalogError::ManagedAdmissionRequired) + })?; + let directory = core + .host + .namespace_directory(history.cluster, history.namespace) + .map_err(|_| CellGroupControllerError::Invalid("transfer source namespace"))?; + owner + .handle + .domain + .run(Box::pin(crate::asyncrt::timeout_at( + deadline, + Box::pin(async { + owner.startup_live(host, deadline)?; + let (permit, group) = host + .managed_source_authority( + transition, + &directory, + history, + owner.startup_remaining(deadline)?, + ) + .await + .map_err(CatalogError::from) + .map_err(super::startup::catalog_error)?; + let retire = directory + .prepare_managed_source_retirement( + &transition.plan, + history, + &core.log, + host, + (&permit, &group), + deadline, + ) + .await + .map_err(super::startup::catalog_error)?; + owner.startup_live(host, deadline)?; + core.runtime.retire_source(retire).await.map_err(Into::into) + }), + ))) + .await + .map_err(|_| CellGroupControllerError::Closed)? + .map_err(|_| CellGroupControllerError::Runtime(RuntimeError::DeadlineExceeded))? + } + } + + /// Submit or observe the exact seal authorized for this managed parent. + /// Accepted identifies a proposal. Applied returns the selected durable cut. + /// A lost response can follow either outcome. Retry the same fleet plan. + pub fn seal_managed_source<'a>( + &self, + transition: &'a ReopenedManagedCellGroupTransition, + timeout: Duration, + ) -> impl Future> + Send + 'a + { + let admission = self + .owner + .metadata_admission(timeout, SEAL_BYTES) + .map_err(super::startup::admission_error); + let owner = self.owner.internal_observer(); + let source = self.history; + async move { + let (deadline, _memory) = admission?; + let core = &owner.handle.core; + let host = core.managed.as_ref().ok_or_else(|| { + super::startup::catalog_error(CatalogError::ManagedAdmissionRequired) + })?; + let directory = core + .host + .namespace_directory(source.cluster, source.namespace) + .map_err(|_| CellGroupControllerError::Invalid("managed source namespace"))?; + owner + .handle + .domain + .run(Box::pin(crate::asyncrt::timeout_at( + deadline, + Box::pin(async { + let remaining = owner.startup_remaining(deadline)?; + owner.startup_live(host, deadline)?; + let (permit, group) = host + .managed_source_authority(transition, &directory, source, remaining) + .await + .map_err(CatalogError::from) + .map_err(super::startup::catalog_error)?; + let seal = directory + .prepare_managed_source_seal( + &transition.plan, + source, + &core.log, + host, + (&permit, &group), + deadline, + ) + .await + .map_err(super::startup::catalog_error)?; + owner.startup_live(host, deadline)?; + core.runtime.seal_source(seal).await.map_err(Into::into) + }), + ))) + .await + .map_err(|_| CellGroupControllerError::Closed)? + .map_err(|_| CellGroupControllerError::Runtime(RuntimeError::DeadlineExceeded))? + } + } +} + +impl RaftCells { + /// Attach this boot's exact installed voter to the pending child. The + /// majority of install receipts and current capacity are revalidated before + /// attachment. The child can vote but cannot serve application requests. + pub fn attach_managed_transfer_child( + &self, + transition: Arc, + route: GroupRoute, + timeout: Duration, + ) -> impl Future> + Send + 'static { + let admitted = self + .metadata_admission(timeout, 4096) + .map_err(super::startup::admission_error); + let owner = self.internal_observer(); + async move { + let (deadline, _memory) = admitted?; + let core = &owner.handle.core; + let host = core.managed.as_ref().ok_or_else(|| { + super::startup::catalog_error(CatalogError::ManagedAdmissionRequired) + })?; + owner.startup_live(host, deadline)?; + let directory = core + .host + .namespace_directory(route.history.cluster, route.history.namespace) + .map_err(|_| CellGroupControllerError::Invalid("transfer child namespace"))?; + let lease = owner + .register_control_service(ControlServiceKind::Transfer, 4096) + .map_err(CatalogError::from) + .map_err(super::startup::catalog_error)?; + let (reply, result) = tokio::sync::oneshot::channel(); + let controller = core.controller.clone(); + let transport = core.transport.clone(); + let host = host.clone(); + let domain = &owner.handle.domain; + domain + .spawn_cleanup(async move { + let stop = lease.stop_receiver(); + let attached = Box::pin(controller.attach_managed_transfer_child( + &directory, + transition, + &route, + &host, + &transport, + (deadline, &stop), + )) + .await; + drop(lease); + let _ = reply.send(attached); + }) + .map_err(|_| CellGroupControllerError::Closed)? + .detach(); + domain + .run(crate::asyncrt::timeout_at(deadline, result)) + .await + .map_err(|_| CellGroupControllerError::Closed)? + .map_err(|_| CellGroupControllerError::Runtime(RuntimeError::DeadlineExceeded))? + .map_err(|_| CellGroupControllerError::Closed)? + } + } + + /// Install one reserved transfer child into this node's retained stores. + /// The receipt proves local sync. Attachment and activation are separate. + /// After a timeout, retry the same transition, sources and child. + pub fn install_managed_transfer_child( + &self, + transition: Arc, + sources: VerifiedCellGroupTransitionSources, + child: GroupHistory, + recovery: CellGroupRecoveryPool, + timeout: Duration, + ) -> impl Future> + Send + 'static + { + let admitted = self + .metadata_admission(timeout, 4096) + .map_err(super::startup::admission_error); + let owner = self.internal_observer(); + async move { + let (deadline, _memory) = admitted?; + let core = &owner.handle.core; + let host = core.managed.as_ref().ok_or_else(|| { + super::startup::catalog_error(CatalogError::ManagedAdmissionRequired) + })?; + owner.startup_live(host, deadline)?; + if sources.transition_plan() != (transition.plan.operation(), transition.plan.digest()) + { + return Err(super::startup::catalog_error(CatalogError::Fenced)); + } + let route = sources + .children() + .iter() + .find(|route| route.history == child) + .cloned() + .ok_or_else(|| super::startup::catalog_error(CatalogError::Fenced))?; + let directory = core + .host + .namespace_directory(child.cluster, child.namespace) + .map_err(|_| CellGroupControllerError::Invalid("transfer child namespace"))?; + let lease = owner + .register_control_service(ControlServiceKind::Transfer, 4096) + .map_err(CatalogError::from) + .map_err(super::startup::catalog_error)?; + let (reply, result) = tokio::sync::oneshot::channel(); + let controller = core.controller.clone(); + let host = host.clone(); + let domain = &owner.handle.domain; + domain + .spawn_cleanup(async move { + let stop = lease.stop_receiver(); + let installed = Box::pin(controller.install_managed_transfer_child( + &directory, + (&transition, &sources), + &route, + &host, + &recovery, + (deadline, &stop), + )) + .await; + drop(lease); + let _ = reply.send(installed); + }) + .map_err(|_| CellGroupControllerError::Closed)? + .detach(); + domain + .run(crate::asyncrt::timeout_at(deadline, result)) + .await + .map_err(|_| CellGroupControllerError::Closed)? + .map_err(|_| CellGroupControllerError::Runtime(RuntimeError::DeadlineExceeded))? + .map_err(|_| CellGroupControllerError::Closed)? + } + } +} diff --git a/crates/cellar/application/consensus/group_contracts.rs b/crates/cellar/application/consensus/group_contracts.rs new file mode 100644 index 000000000..c89de0e08 --- /dev/null +++ b/crates/cellar/application/consensus/group_contracts.rs @@ -0,0 +1,60 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Host-side hashing for the dependency-free cell-group contracts. + +use super::protocol::{ImageCommitment, LtxImage, ProtocolError}; +use cellar_logic::{ + cell_group::{CellId, CellImage, CellImageCommitment, CellKey}, + consensus::NamespaceId, +}; +use cellar_ltx::internal::{ExtentRootReference, ImageRoot}; +use sha2::{Digest, Sha256}; + +pub fn cell_routing_key(namespace: NamespaceId, cell: &CellId) -> CellKey { + CellKey::from_digest( + Sha256::digest(cell.routing_preimage(namespace)).into(), + cell.clone(), + ) +} + +impl TryFrom for LtxImage { + type Error = ProtocolError; + + fn try_from(image: CellImage) -> Result { + image.validate().map_err(|_| ProtocolError::Chain)?; + let commitment = match image.commitment { + CellImageCommitment::Materialized(digest) => ImageCommitment::Sha256(digest), + CellImageCommitment::Extents { + root, + manifest_bytes, + } => ImageCommitment::Extents( + ExtentRootReference::new(ImageRoot::from_bytes(root), manifest_bytes) + .map_err(|_| ProtocolError::Identity)?, + ), + }; + Ok(Self { + epoch: image.epoch, + txid: image.txid, + checksum: image.checksum, + commitment, + }) + } +} + +impl From for CellImage { + fn from(image: LtxImage) -> Self { + let commitment = match image.commitment { + ImageCommitment::Sha256(digest) => CellImageCommitment::Materialized(digest), + ImageCommitment::Extents(reference) => CellImageCommitment::Extents { + root: reference.root().bytes(), + manifest_bytes: reference.encoded_length(), + }, + }; + Self { + epoch: image.epoch, + txid: image.txid, + checksum: image.checksum, + commitment, + } + } +} diff --git a/crates/cellar/application/consensus/group_protocol/README.md b/crates/cellar/application/consensus/group_protocol/README.md new file mode 100644 index 000000000..25b939b8d --- /dev/null +++ b/crates/cellar/application/consensus/group_protocol/README.md @@ -0,0 +1,81 @@ + + +# Cell-group entry encoding + +This module encodes the replacement group runtime's cell-addressed log entries. +The existing host is not switched over by the codec. Storage and runtime owners +must verify quorum, physical images, membership and transition proofs separately. + +## Frame and bounds + +Every integer is unsigned big-endian. The marker stays at 1; the new magic and +semantic domain refuse old per-cell entries in both directions. + +| Field | Bytes | +| --- | ---: | +| `CLRCGE01` | 8 | +| Marker, exactly 1 | 4 | +| Header length | 4 | +| Payload length | 8 | +| Header | Declared, bounded | +| Concatenated LTX files | Declared, bounded | +| SHA-256 of all preceding bytes | 32 | + +The header starts with SHA-256 of +`cellar-cell-group-entry-1/cell-history-1/range-route-1/administration-1`, followed +by group history (four 16-byte identities), route epoch (8), birth digest (32), +schema digest (32), proposal ID (16) and command tag (1). The receiving owner +supplies its expected route; mismatched history, epoch, birth or schema refuses. + +Cell addresses contain the cell incarnation (16), ID byte length (2), and exact +UTF-8 ID. Cluster and namespace occur once in the group binding and are inherited +by the cell address. A key is recomputed from its namespace and validated ID. +An encoded range boundary also carries its digest; it must match recomputation. +Ranges have explicit infinity tags and retain the full ID as a collision tie-break. + +An image is epoch/txid/checksum (three 8-byte integers), commitment tag (1), digest +(32), and manifest length (4). Materialized commitments require length zero. +Extent commitments use the extent store's actual root-reference validation. + +File tables begin with a 4-byte count. Each record is five 8-byte integers: +encoded length, minimum txid, maximum txid, page count and decoded bytes. Payload +spans are consecutive and exhaustive; overlapping offsets cannot be represented. +Counts, header/payload limits and accumulated page/decoded-byte limits are checked +before allocation or physical staging. The decoder borrows payload slices from +the original entry buffer. The LTX applier must independently enforce actual +page/decompression limits; envelope declarations are not proof of physical size. + +Fixed-position fields are mandatory and occur once. Unknown tags/domains, extra +fields, trailing bytes, zero identities, overflow, truncated declarations and +noncanonical image commitments refuse. The frame digest detects corruption; it +is not authentication or a quorum certificate. + +## Commands + +| Tag | Command | Required owner behavior beyond decoding | +| --- | --- | --- | +| 1 | Mutation | Check exact current cell history/base image and next sequence before staging; verify every LTX file and publish only a contiguous group prefix. | +| 2 | Enroll | Sequence starts at one. A nonempty image carries its full LTX chain from txid one in the log payload. Verify and sync the initial image before publication; empty enrollment must verify the canonical empty root. | +| 3 | Membership | Bind previous/next membership revision and physical member identities. Use the actual Raft ConfChangeV2 path and membership authorization; never treat it as an ordinary data entry that implicitly changes voters. | +| 4 | Checkpoint | Verify the complete manifest and its referenced image closure at the exact committed position and membership revision. | +| 5 | Seal | Verify the immutable seal authorization and complete parent/child topology union before irrevocably sealing the parent. Child routes must be distinct, compatible and contiguous. | +| 6 | Install | Verify all sealed source cuts and the destination manifest; sync the whole seed closure on each installing replica. | +| 7 | Activate | Verify publication, seed and current-term quorum before admission. | +| 8 | PreparePark | Bind park generation, term and membership; collect durable acknowledgements and retain wake ownership. | +| 9 | Retire | Verify publication and persist the parent tombstone; reclamation still needs archive and pin proofs. | + +A decoded mutation exposes `check_base` against the coherent manifest leaf. This +check refuses another cell/incarnation, a skipped or repeated mutation sequence, +or a different base commitment. It must run before image preparation, and remains +owned by the serial apply path so another completion cannot change the base. + +Enrollment carries bytes rather than only a root reference: majority persistence +of a root alone cannot protect data that exists solely on the proposing host. +Oversized initialization must refuse the configured envelope budget; it cannot +fall back to an unproven local or S3 reference. + +The native contract fixture uses SQLite-produced LTX and verifies replay of an +enrollment from the entry payload alone. It also checks a separately calculated +retirement byte vector, hostile declarations, duplicate identities, every +administrative command, old-domain rejection, and borrowed payload ownership. +These are codec tests, not qualification of a running shared Raft group. diff --git a/crates/cellar/application/consensus/group_protocol/codec.rs b/crates/cellar/application/consensus/group_protocol/codec.rs new file mode 100644 index 000000000..9d026fcc8 --- /dev/null +++ b/crates/cellar/application/consensus/group_protocol/codec.rs @@ -0,0 +1,625 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::super::protocol::{EnvelopeLimits, ImageCommitment, LtxImage, ProtocolError}; +use super::types::*; +use cellar_logic::{cell_group::*, consensus::*}; +use cellar_ltx::internal::{ExtentRootReference, ImageRoot}; +use sha2::{Digest, Sha256}; + +const MAGIC: &[u8; 8] = b"CLRCGE01"; +const MARKER: u32 = 1; +const PREFIX: usize = 24; +const DIGEST: usize = 32; +const DOMAIN: &[u8] = b"cellar-cell-group-entry-1/cell-history-1/range-route-1/administration-1"; + +impl<'a> CellGroupEntry<'a> { + pub fn encode( + &self, + route: &GroupRoute, + limits: GroupEntryLimits, + ) -> Result, ProtocolError> { + let payload_len = self.validate(route, limits)?; + let mut header = Output { + bytes: Vec::new(), + limit: limits.envelope.max_header_bytes, + }; + header.put(&Sha256::digest(DOMAIN))?; + header.group(self.binding.group)?; + header.u64(self.binding.route_epoch.get())?; + header.put(&self.binding.birth_digest)?; + header.put(&self.binding.schema_digest)?; + header.put(&self.binding.proposal.bytes())?; + match &self.command { + GroupCommand::Mutation(update) => { + header.tag(1)?; + header.cell(&update.cell)?; + header.u64(update.sequence.get())?; + header.image(update.base)?; + header.image(update.result)?; + header.files(&update.files)?; + } + GroupCommand::Enroll { cell, image, files } => { + header.tag(2)?; + header.cell(cell)?; + header.image(*image)?; + header.files(files)?; + } + GroupCommand::Membership { + previous, + next, + transition, + changes, + } => { + header.tag(3)?; + header.u64(previous.get())?; + header.u64(next.get())?; + header.tag(match transition { + GroupMembershipTransition::Auto => 1, + GroupMembershipTransition::JointImplicit => 2, + GroupMembershipTransition::JointExplicit => 3, + GroupMembershipTransition::LeaveJoint => 4, + })?; + header.u32(changes.len() as u32)?; + for change in changes { + header.u64(change.member.get())?; + header.put(&change.node.bytes())?; + header.put(&change.disk.bytes())?; + header.tag(match change.role { + GroupMemberRole::Voter => 1, + GroupMemberRole::Learner => 2, + GroupMemberRole::Remove => 3, + })?; + } + } + GroupCommand::Checkpoint { + position, + manifest, + membership, + } => { + header.tag(4)?; + header.position(*position)?; + header.put(manifest)?; + header.u64(membership.get())?; + } + GroupCommand::Seal { + transition, + authorization, + children, + } => { + header.tag(5)?; + header.put(&transition.bytes())?; + header.put(authorization)?; + header.u32(children.len() as u32)?; + for child in children { + header.route(child)?; + } + } + GroupCommand::Install { + transition, + sources, + manifest, + } => { + header.tag(6)?; + header.put(&transition.bytes())?; + header.put(manifest)?; + header.u32(sources.len() as u32)?; + for source in sources { + header.group(source.group)?; + header.position(source.position)?; + header.put(&source.manifest)?; + } + } + GroupCommand::Activate { + transition, + publication, + manifest, + } => { + header.tag(7)?; + header.put(&transition.bytes())?; + header.put(publication)?; + header.put(manifest)?; + } + GroupCommand::PreparePark { + generation, + term, + membership, + } => { + header.tag(8)?; + header.u64(generation.get())?; + header.u64(term.get())?; + header.u64(membership.get())?; + } + GroupCommand::Retire { + transition, + publication, + } => { + header.tag(9)?; + header.put(&transition.bytes())?; + header.put(publication)?; + } + } + let size = PREFIX + .checked_add(header.bytes.len()) + .and_then(|n| n.checked_add(payload_len)) + .and_then(|n| n.checked_add(DIGEST)) + .ok_or(ProtocolError::Limit)?; + let mut encoded = Vec::with_capacity(size); + encoded.extend_from_slice(MAGIC); + encoded.extend_from_slice(&MARKER.to_be_bytes()); + encoded.extend_from_slice(&(header.bytes.len() as u32).to_be_bytes()); + encoded.extend_from_slice(&(payload_len as u64).to_be_bytes()); + encoded.extend_from_slice(&header.bytes); + let files = match &self.command { + GroupCommand::Mutation(update) => update.files.as_slice(), + GroupCommand::Enroll { files, .. } => files.as_slice(), + _ => &[], + }; + for file in files { + encoded.extend_from_slice(file.bytes); + } + let digest = Sha256::digest(&encoded); + encoded.extend_from_slice(&digest); + Ok(encoded) + } + + /// No payload bytes are copied. Bounds, domain and the expected group route + /// are checked before decoding variable-length cell/file metadata. The + /// caller still checks the coherent cell base and all physical LTX content. + pub fn decode( + bytes: &'a [u8], + route: &GroupRoute, + limits: GroupEntryLimits, + ) -> Result { + limits.validate()?; + let mut prefix = Input(bytes); + if prefix.take(8)? != MAGIC { + return Err(ProtocolError::Protocol); + } + if prefix.u32()? != MARKER { + return Err(ProtocolError::Protocol); + } + let header_len = prefix.u32()? as usize; + let payload_len = prefix.u64()?; + if header_len > limits.envelope.max_header_bytes + || payload_len > limits.envelope.max_payload_bytes as u64 + { + return Err(ProtocolError::Limit); + } + let payload_len = usize::try_from(payload_len).map_err(|_| ProtocolError::Limit)?; + let payload_start = PREFIX.checked_add(header_len).ok_or(ProtocolError::Limit)?; + let end = payload_start + .checked_add(payload_len) + .ok_or(ProtocolError::Limit)?; + if end.checked_add(DIGEST) != Some(bytes.len()) { + return Err(ProtocolError::Framing); + } + if Sha256::digest(&bytes[..end])[..] != bytes[end..] { + return Err(ProtocolError::Digest); + } + let mut input = Input(&bytes[PREFIX..payload_start]); + let mut payload = Input(&bytes[payload_start..end]); + if input.take(DIGEST)? != &Sha256::digest(DOMAIN)[..] { + return Err(ProtocolError::Protocol); + } + let binding = GroupEntryBinding { + group: input.group()?, + route_epoch: input.counter(RouteEpoch::new)?, + birth_digest: input.array()?, + schema_digest: input.array()?, + proposal: input.id(ProposalId::new)?, + }; + binding.check(route)?; + let command = match input.tag()? { + 1 => { + let cell = input.cell(binding.group)?; + check_cell(route, &cell)?; + let sequence = input.counter(MutationSequence::new)?; + let base = input.image()?; + let result = input.image()?; + let files = input.files(&mut payload, limits.envelope)?; + GroupCommand::Mutation(GroupMutation { + cell, + sequence, + base, + result, + files, + }) + } + 2 => { + let cell = input.cell(binding.group)?; + check_cell(route, &cell)?; + GroupCommand::Enroll { + cell, + image: input.image()?, + files: input.files(&mut payload, limits.envelope)?, + } + } + 3 => { + let previous = input.counter(MembershipRevision::new)?; + let next = input.counter(MembershipRevision::new)?; + let transition = match input.tag()? { + 1 => GroupMembershipTransition::Auto, + 2 => GroupMembershipTransition::JointImplicit, + 3 => GroupMembershipTransition::JointExplicit, + 4 => GroupMembershipTransition::LeaveJoint, + _ => return Err(ProtocolError::Protocol), + }; + let count = input.count(limits.max_members, 41)?; + let mut changes = Vec::with_capacity(count); + for _ in 0..count { + let member = input.counter(MemberId::new)?; + let node = input.id(NodeId::new)?; + let disk = input.id(DiskLineage::new)?; + let role = match input.tag()? { + 1 => GroupMemberRole::Voter, + 2 => GroupMemberRole::Learner, + 3 => GroupMemberRole::Remove, + _ => return Err(ProtocolError::Protocol), + }; + changes.push(GroupMemberChange { + member, + node, + disk, + role, + }); + } + GroupCommand::Membership { + previous, + next, + transition, + changes, + } + } + 4 => GroupCommand::Checkpoint { + position: input.position()?, + manifest: input.array()?, + membership: input.counter(MembershipRevision::new)?, + }, + 5 => { + let transition = input.id(TransitionId::new)?; + let authorization = input.array()?; + let count = input.count(limits.max_transition_groups, 138)?; + let mut children = Vec::with_capacity(count); + for _ in 0..count { + children.push(input.route()?); + } + GroupCommand::Seal { + transition, + authorization, + children, + } + } + 6 => { + let transition = input.id(TransitionId::new)?; + let manifest = input.array()?; + let count = input.count(limits.max_transition_groups, 112)?; + let mut sources = Vec::with_capacity(count); + for _ in 0..count { + sources.push(GroupSeal { + group: input.group()?, + position: input.position()?, + manifest: input.array()?, + }); + } + GroupCommand::Install { + transition, + sources, + manifest, + } + } + 7 => GroupCommand::Activate { + transition: input.id(TransitionId::new)?, + publication: input.array()?, + manifest: input.array()?, + }, + 8 => GroupCommand::PreparePark { + generation: input.counter(ParkGeneration::new)?, + term: Term::new(input.u64()?), + membership: input.counter(MembershipRevision::new)?, + }, + 9 => GroupCommand::Retire { + transition: input.id(TransitionId::new)?, + publication: input.array()?, + }, + _ => return Err(ProtocolError::Protocol), + }; + if !input.0.is_empty() || !payload.0.is_empty() { + return Err(ProtocolError::Framing); + } + let entry = Self { binding, command }; + entry.validate(route, limits)?; + Ok(entry) + } +} + +struct Output { + bytes: Vec, + limit: usize, +} + +impl Output { + fn put(&mut self, value: &[u8]) -> Result<(), ProtocolError> { + if self + .bytes + .len() + .checked_add(value.len()) + .is_none_or(|n| n > self.limit) + { + return Err(ProtocolError::Limit); + } + self.bytes.extend_from_slice(value); + Ok(()) + } + + fn tag(&mut self, value: u8) -> Result<(), ProtocolError> { + self.put(&[value]) + } + + fn u32(&mut self, value: u32) -> Result<(), ProtocolError> { + self.put(&value.to_be_bytes()) + } + + fn u64(&mut self, value: u64) -> Result<(), ProtocolError> { + self.put(&value.to_be_bytes()) + } + + fn group(&mut self, value: GroupHistory) -> Result<(), ProtocolError> { + for id in [ + value.cluster.bytes(), + value.namespace.bytes(), + value.group.bytes(), + value.incarnation.bytes(), + ] { + self.put(&id)?; + } + Ok(()) + } + + fn text(&mut self, value: &CellId) -> Result<(), ProtocolError> { + self.put(&(value.as_str().len() as u16).to_be_bytes())?; + self.put(value.as_str().as_bytes()) + } + + fn cell(&mut self, value: &CellHistory) -> Result<(), ProtocolError> { + self.put(&value.incarnation.bytes())?; + self.text(&value.cell) + } + + fn image(&mut self, value: LtxImage) -> Result<(), ProtocolError> { + for n in [value.epoch, value.txid, value.checksum] { + self.u64(n)?; + } + match value.commitment { + ImageCommitment::Sha256(digest) => { + self.tag(1)?; + self.put(&digest)?; + self.u32(0) + } + ImageCommitment::Extents(root) => { + self.tag(2)?; + self.put(&root.root().bytes())?; + self.u32(root.encoded_length()) + } + } + } + + fn files(&mut self, files: &[GroupLtx<'_>]) -> Result<(), ProtocolError> { + self.u32(files.len() as u32)?; + for file in files { + for value in [ + file.bytes.len() as u64, + file.min_txid, + file.max_txid, + file.page_count, + file.decoded_bytes, + ] { + self.u64(value)?; + } + } + Ok(()) + } + + fn position(&mut self, value: LogPosition) -> Result<(), ProtocolError> { + self.u64(value.index.get())?; + self.u64(value.term.get()) + } + + fn bound(&mut self, value: &CellBound) -> Result<(), ProtocolError> { + match value { + CellBound::NegativeInfinity => self.tag(0), + CellBound::Key(key) => { + self.tag(1)?; + self.put(&key.digest())?; + self.text(key.cell()) + } + CellBound::PositiveInfinity => self.tag(2), + } + } + + fn route(&mut self, value: &GroupRoute) -> Result<(), ProtocolError> { + self.group(value.history)?; + self.u64(value.epoch.get())?; + self.put(&value.birth_digest)?; + self.put(&value.schema_digest)?; + self.bound(value.range.lower())?; + self.bound(value.range.upper()) + } +} + +struct Input<'a>(&'a [u8]); + +impl<'a> Input<'a> { + fn take(&mut self, n: usize) -> Result<&'a [u8], ProtocolError> { + let (value, rest) = self.0.split_at_checked(n).ok_or(ProtocolError::Framing)?; + self.0 = rest; + Ok(value) + } + + fn array(&mut self) -> Result<[u8; N], ProtocolError> { + self.take(N)?.try_into().map_err(|_| ProtocolError::Framing) + } + + fn tag(&mut self) -> Result { + Ok(self.array::<1>()?[0]) + } + + fn u32(&mut self) -> Result { + Ok(u32::from_be_bytes(self.array()?)) + } + + fn u64(&mut self) -> Result { + Ok(u64::from_be_bytes(self.array()?)) + } + + fn id( + &mut self, + make: impl FnOnce([u8; 16]) -> Result, + ) -> Result { + make(self.array()?).map_err(|_| ProtocolError::Identity) + } + + fn counter( + &mut self, + make: impl FnOnce(u64) -> Result, + ) -> Result { + make(self.u64()?).map_err(|_| ProtocolError::Identity) + } + + fn count(&mut self, maximum: usize, minimum_bytes: usize) -> Result { + let count = self.u32()? as usize; + if count > maximum { + return Err(ProtocolError::Limit); + } + if count + .checked_mul(minimum_bytes) + .is_none_or(|n| n > self.0.len()) + { + return Err(ProtocolError::Framing); + } + Ok(count) + } + + fn group(&mut self) -> Result { + Ok(GroupHistory { + cluster: self.id(ClusterId::new)?, + namespace: self.id(NamespaceId::new)?, + group: self.id(GroupId::new)?, + incarnation: self.id(IncarnationId::new)?, + }) + } + + fn text(&mut self) -> Result { + let n = u16::from_be_bytes(self.array()?) as usize; + if n > MAX_CELL_ID_BYTES { + return Err(ProtocolError::Limit); + } + CellId::new(std::str::from_utf8(self.take(n)?).map_err(|_| ProtocolError::Identity)?) + .map_err(|_| ProtocolError::Identity) + } + + fn cell(&mut self, group: GroupHistory) -> Result { + let incarnation = self.id(IncarnationId::new)?; + Ok(CellHistory { + cluster: group.cluster, + namespace: group.namespace, + incarnation, + cell: self.text()?, + }) + } + + fn image(&mut self) -> Result { + let epoch = self.u64()?; + let txid = self.u64()?; + let checksum = self.u64()?; + let tag = self.tag()?; + let digest = self.array()?; + let length = self.u32()?; + let commitment = match (tag, length) { + (1, 0) => ImageCommitment::Sha256(digest), + (2, _) => ImageCommitment::Extents( + ExtentRootReference::new(ImageRoot::from_bytes(digest), length) + .map_err(|_| ProtocolError::Identity)?, + ), + _ => return Err(ProtocolError::Protocol), + }; + let image = LtxImage { + epoch, + txid, + checksum, + commitment, + }; + LtxImage::try_from(CellImage::from(image))?; + Ok(image) + } + + fn files( + &mut self, + payload: &mut Input<'a>, + limits: EnvelopeLimits, + ) -> Result>, ProtocolError> { + let count = self.count(limits.max_files, 40)?; + let mut files = Vec::with_capacity(count); + let (mut decoded, mut pages) = (0_u64, 0_u64); + for _ in 0..count { + let length = self.u64()?; + if length == 0 || length > limits.max_payload_bytes as u64 { + return Err(ProtocolError::Limit); + } + let min_txid = self.u64()?; + let max_txid = self.u64()?; + let page_count = self.u64()?; + let decoded_bytes = self.u64()?; + pages = pages.checked_add(page_count).ok_or(ProtocolError::Limit)?; + decoded = decoded + .checked_add(decoded_bytes) + .ok_or(ProtocolError::Limit)?; + if pages > limits.max_pages || decoded > limits.max_decoded_bytes { + return Err(ProtocolError::Limit); + } + let length = usize::try_from(length).map_err(|_| ProtocolError::Limit)?; + files.push(GroupLtx { + min_txid, + max_txid, + page_count, + decoded_bytes, + bytes: payload.take(length)?, + }); + } + Ok(files) + } + + fn position(&mut self) -> Result { + Ok(LogPosition { + index: LogIndex::new(self.u64()?), + term: Term::new(self.u64()?), + }) + } + + fn bound(&mut self) -> Result { + match self.tag()? { + 0 => Ok(CellBound::NegativeInfinity), + 1 => { + let digest = self.array()?; + Ok(CellBound::Key(CellKey::from_digest(digest, self.text()?))) + } + 2 => Ok(CellBound::PositiveInfinity), + _ => Err(ProtocolError::Protocol), + } + } + + fn route(&mut self) -> Result { + let history = self.group()?; + let epoch = self.counter(RouteEpoch::new)?; + let birth_digest = self.array()?; + let schema_digest = self.array()?; + let range = + CellRange::new(self.bound()?, self.bound()?).map_err(|_| ProtocolError::Identity)?; + Ok(GroupRoute { + history, + epoch, + birth_digest, + schema_digest, + range, + }) + } +} diff --git a/crates/cellar/application/consensus/group_protocol/mod.rs b/crates/cellar/application/consensus/group_protocol/mod.rs new file mode 100644 index 000000000..6f5171421 --- /dev/null +++ b/crates/cellar/application/consensus/group_protocol/mod.rs @@ -0,0 +1,9 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Cell-addressed entries for one physical group. Decoding verifies the wire +//! binding and borrows payload bytes; it does not grant quorum or image proofs. + +mod codec; +mod types; + +pub use types::*; diff --git a/crates/cellar/application/consensus/group_protocol/types.rs b/crates/cellar/application/consensus/group_protocol/types.rs new file mode 100644 index 000000000..f19df02fb --- /dev/null +++ b/crates/cellar/application/consensus/group_protocol/types.rs @@ -0,0 +1,369 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::super::protocol::{ImageCommitment, LtxImage, ProtocolError}; +use super::super::{group_contracts::cell_routing_key, protocol::EnvelopeLimits}; +use cellar_logic::{ + cell_group::*, + consensus::{DiskLineage, LogPosition, MemberId, NodeId, ProposalId, Term}, +}; +use std::collections::BTreeSet; + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct GroupEntryLimits { + pub envelope: EnvelopeLimits, + pub max_members: usize, + pub max_transition_groups: usize, +} + +impl Default for GroupEntryLimits { + fn default() -> Self { + Self { + envelope: EnvelopeLimits::default(), + max_members: 64, + max_transition_groups: 64, + } + } +} + +impl GroupEntryLimits { + pub fn validate(self) -> Result<(), ProtocolError> { + self.envelope.validate()?; + if !(1..=64).contains(&self.max_members) || !(1..=64).contains(&self.max_transition_groups) + { + return Err(ProtocolError::Limit); + } + Ok(()) + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct GroupEntryBinding { + pub group: GroupHistory, + pub route_epoch: RouteEpoch, + pub birth_digest: [u8; 32], + pub schema_digest: [u8; 32], + pub proposal: ProposalId, +} + +impl GroupEntryBinding { + pub fn check(&self, route: &GroupRoute) -> Result<(), ProtocolError> { + if self.group != route.history + || self.route_epoch != route.epoch + || self.birth_digest != route.birth_digest + || self.schema_digest != route.schema_digest + { + return Err(ProtocolError::Identity); + } + check_range_keys(route)?; + Ok(()) + } +} + +/// The entry's original encoded buffer owns these bytes until staging finishes. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct GroupLtx<'a> { + pub min_txid: u64, + pub max_txid: u64, + pub page_count: u64, + pub decoded_bytes: u64, + pub bytes: &'a [u8], +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct GroupMutation<'a> { + pub cell: CellHistory, + pub sequence: MutationSequence, + pub base: LtxImage, + pub result: LtxImage, + pub files: Vec>, +} + +impl GroupMutation<'_> { + /// Call against the coherent manifest leaf before allocating image work. + /// Valid framing or another cell's progress cannot establish this base. + pub fn check_base(&self, current: &CellVersion) -> Result<(), ProtocolError> { + if self.cell != current.history + || current.sequence.checked_next().ok() != Some(self.sequence) + || CellImage::from(self.base) != current.image + { + return Err(ProtocolError::Chain); + } + Ok(()) + } + + pub(super) fn validate(&self, limits: EnvelopeLimits) -> Result { + LtxImage::try_from(CellImage::from(self.base))?; + LtxImage::try_from(CellImage::from(self.result))?; + if self.sequence.get() <= 1 + || self.base.epoch != self.result.epoch + || self.result.txid <= self.base.txid + || !matches!( + (self.base.commitment, self.result.commitment), + (ImageCommitment::Sha256(_), ImageCommitment::Sha256(_)) + | (ImageCommitment::Extents(_), ImageCommitment::Extents(_)) + ) + { + return Err(ProtocolError::Chain); + } + validate_files(&self.files, limits, self.base.txid, self.result.txid) + } +} + +pub(super) fn validate_files( + files: &[GroupLtx<'_>], + limits: EnvelopeLimits, + base: u64, + result: u64, +) -> Result { + if files.len() > limits.max_files || (files.is_empty() && (base != 0 || result != 0)) { + return Err(ProtocolError::Limit); + } + let (mut txid, mut payload, mut decoded, mut pages) = (base, 0_usize, 0_u64, 0_u64); + for file in files { + if txid.checked_add(1) != Some(file.min_txid) + || file.max_txid < file.min_txid + || file.bytes.is_empty() + { + return Err(ProtocolError::Chain); + } + txid = file.max_txid; + payload = payload + .checked_add(file.bytes.len()) + .ok_or(ProtocolError::Limit)?; + decoded = decoded + .checked_add(file.decoded_bytes) + .ok_or(ProtocolError::Limit)?; + pages = pages + .checked_add(file.page_count) + .ok_or(ProtocolError::Limit)?; + if payload > limits.max_payload_bytes + || decoded > limits.max_decoded_bytes + || pages > limits.max_pages + { + return Err(ProtocolError::Limit); + } + } + if txid != result { + return Err(ProtocolError::Chain); + } + Ok(payload) +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum GroupMemberRole { + Voter, + Learner, + Remove, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct GroupMemberChange { + pub member: MemberId, + pub node: NodeId, + pub disk: DiskLineage, + pub role: GroupMemberRole, +} + +/// Matches the raft-rs configuration transition semantics. The owning runtime +/// must encode this command as the actual ConfChangeV2 entry, not a data no-op. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum GroupMembershipTransition { + Auto, + JointImplicit, + JointExplicit, + LeaveJoint, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct GroupSeal { + pub group: GroupHistory, + pub position: LogPosition, + pub manifest: [u8; 32], +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum GroupCommand<'a> { + Mutation(GroupMutation<'a>), + /// Enrollment starts at sequence one. A nonempty image carries a complete + /// LTX chain starting at txid one, so majority log persistence protects its + /// bytes. Publication still requires independent image verification/sync. + Enroll { + cell: CellHistory, + image: LtxImage, + files: Vec>, + }, + Membership { + previous: MembershipRevision, + next: MembershipRevision, + transition: GroupMembershipTransition, + changes: Vec, + }, + Checkpoint { + position: LogPosition, + manifest: [u8; 32], + membership: MembershipRevision, + }, + Seal { + transition: TransitionId, + authorization: [u8; 32], + children: Vec, + }, + Install { + transition: TransitionId, + sources: Vec, + manifest: [u8; 32], + }, + Activate { + transition: TransitionId, + publication: [u8; 32], + manifest: [u8; 32], + }, + PreparePark { + generation: ParkGeneration, + term: Term, + membership: MembershipRevision, + }, + Retire { + transition: TransitionId, + publication: [u8; 32], + }, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct CellGroupEntry<'a> { + pub binding: GroupEntryBinding, + pub command: GroupCommand<'a>, +} + +impl CellGroupEntry<'_> { + pub(super) fn validate( + &self, + route: &GroupRoute, + limits: GroupEntryLimits, + ) -> Result { + limits.validate()?; + self.binding.check(route)?; + match &self.command { + GroupCommand::Mutation(update) => { + check_cell(route, &update.cell)?; + return update.validate(limits.envelope); + } + GroupCommand::Enroll { cell, image, files } => { + check_cell(route, cell)?; + LtxImage::try_from(CellImage::from(*image))?; + if image.txid == 0 && image.checksum != 0 { + return Err(ProtocolError::Chain); + } + return validate_files(files, limits.envelope, 0, image.txid); + } + GroupCommand::Membership { + previous, + next, + transition, + changes, + } => { + if previous.checked_next().ok() != Some(*next) { + return Err(ProtocolError::Chain); + } + if changes.len() > limits.max_members { + return Err(ProtocolError::Limit); + } + if (*transition == GroupMembershipTransition::LeaveJoint) != changes.is_empty() { + return Err(ProtocolError::Header); + } + let mut members = BTreeSet::new(); + for change in changes { + if !members.insert(change.member) { + return Err(ProtocolError::Identity); + } + } + } + GroupCommand::Checkpoint { position, .. } => check_position(*position)?, + GroupCommand::Seal { children, .. } => { + if children.is_empty() || children.len() > limits.max_transition_groups { + return Err(ProtocolError::Limit); + } + let mut groups = BTreeSet::new(); + let mut last: Option<&GroupRoute> = None; + for child in children { + check_related(self.binding.group, child.history)?; + if child.schema_digest != route.schema_digest { + return Err(ProtocolError::Identity); + } + if !groups.insert(child.history.group) { + return Err(ProtocolError::Identity); + } + if let Some(previous) = last { + if previous.range.upper() != child.range.lower() { + return Err(ProtocolError::Chain); + } + } + check_range_keys(child)?; + last = Some(child); + } + // A merge child may include neighboring parent ranges. The + // transition owner verifies the complete parent/child union. + if children[0].range.lower() > route.range.lower() + || children[children.len() - 1].range.upper() < route.range.upper() + { + return Err(ProtocolError::Chain); + } + } + GroupCommand::Install { sources, .. } => { + if sources.is_empty() || sources.len() > limits.max_transition_groups { + return Err(ProtocolError::Limit); + } + let mut groups = BTreeSet::new(); + for source in sources { + check_related(self.binding.group, source.group)?; + if !groups.insert(source.group.group) { + return Err(ProtocolError::Identity); + } + check_position(source.position)?; + } + } + GroupCommand::PreparePark { term, .. } if term.get() == 0 => { + return Err(ProtocolError::Identity) + } + GroupCommand::Activate { .. } + | GroupCommand::PreparePark { .. } + | GroupCommand::Retire { .. } => {} + } + Ok(0) + } +} + +pub(super) fn check_cell(route: &GroupRoute, cell: &CellHistory) -> Result<(), ProtocolError> { + route + .check_cell(cell, &cell_routing_key(cell.namespace, &cell.cell)) + .map_err(|_| ProtocolError::Identity) +} + +pub(super) fn check_range_keys(route: &GroupRoute) -> Result<(), ProtocolError> { + for bound in [route.range.lower(), route.range.upper()] { + if let CellBound::Key(key) = bound { + if &cell_routing_key(route.history.namespace, key.cell()) != key { + return Err(ProtocolError::Identity); + } + } + } + Ok(()) +} + +fn check_related(owner: GroupHistory, other: GroupHistory) -> Result<(), ProtocolError> { + if owner.cluster != other.cluster + || owner.namespace != other.namespace + || owner.group == other.group + { + return Err(ProtocolError::Identity); + } + Ok(()) +} + +fn check_position(position: LogPosition) -> Result<(), ProtocolError> { + position.validate().map_err(|_| ProtocolError::Chain)?; + if position.index.get() == 0 { + return Err(ProtocolError::Chain); + } + Ok(()) +} diff --git a/crates/cellar/application/consensus/mod.rs b/crates/cellar/application/consensus/mod.rs index 9a56d25de..1be031d87 100644 --- a/crates/cellar/application/consensus/mod.rs +++ b/crates/cellar/application/consensus/mod.rs @@ -9,7 +9,12 @@ pub(crate) mod admission; mod archive_status; pub use archive_status::{ArchiveFailure, ArchiveReadiness, ArchiveStatus}; mod class; +pub(crate) use class::callback_active; pub(crate) mod client; +pub(crate) mod group_client; +mod group_contracts; +pub(crate) use group_contracts::cell_routing_key; +pub(crate) mod group_protocol; mod performance; pub use performance::{ RaftDurationBucket, RaftOperationPerformance, RaftPerformanceCounter, RaftPerformanceSnapshot, @@ -26,7 +31,9 @@ pub use resources::{ pub(crate) mod protocol; pub use crate::raft::archive::ArchiveLimits; -pub use crate::raft::completion::{LeadershipTransfer, MemberProgress, ReplicationStatus}; +pub use crate::raft::completion::{ + LeadershipTransfer, MemberProgress, ReplicationStatus, RuntimeError, +}; pub use crate::raft::effects::{ RaftAlarmLimits, RaftBlobLimits, RaftBlobReference, RaftDeliveryRequest, RaftEffectOptions, RaftEffectSigningKey, RaftEffectSink, RaftEffectSourceGrant, RaftEffectTarget, @@ -37,20 +44,42 @@ pub use crate::raft::physical_inventory::{RaftPhysicalHistory, RaftPhysicalInven pub use crate::raft::receipt_maintenance::ReceiptMaintenanceLimits; pub use crate::raft::archive::ArchiveReference; -pub use crate::raft::catalog::CatalogError; pub use crate::raft::catalog::{ BirthDisposition, BirthProposal, CatalogOptions, DisasterFenceAuthority, DisasterFenceRequest, DisasterRecoveryRequest, InitialMember, ManagedBirthOwner, NamespaceConfig, RaftImageStorage, VoterPolicy, }; +pub use crate::raft::catalog::{Catalog, CatalogError}; pub use crate::raft::disaster::S3DisasterRecovery; +pub use cellar_logic::cell_group_placement::{ + HotCell, PartitionBasis, PartitionDecision, PartitionLimits, PartitionLoad, PartitionPolicy, + PartitionSample, PartitionTracker, SplitReason, WaitReason, +}; pub use cellar_logic::raft_leadership::LeaderPolicy; pub use cellar_logic::raft_placement::PlacementPolicy; +pub use crate::raft::cell_groups::{ + CellCommandError, CellDispatchError, CellDispatchLimits, CellEnrollmentError, + CellEnrollmentRequest, CellEnrollmentResponse, CellGroupArchiveControlError, + CellGroupArchiveLimits, CellGroupArchivePool, CellGroupArchiveStatus, + CellGroupArchiveTailLimits, CellGroupArchiveTailPool, CellGroupArchiveUploadLimits, + CellGroupBirthSpec, CellGroupCheckpointLimits, CellGroupCheckpointPolicy, + CellGroupControllerError, CellGroupControllerLimits, CellGroupDisasterFenceRequest, + CellGroupHostLimits, CellGroupObservation, CellGroupPartitionCell, CellGroupPartitionContext, + CellGroupPartitionObservation, CellGroupPartitionWriter, CellGroupProposal, + CellGroupProvisionLimits, CellGroupQuorumMembership, CellGroupRecoveryLimits, + CellGroupRecoveryPool, CellGroupSealOutcome, CellGroupSourceRetirement, + CellGroupSourceRetirementAck, CellGroupTransferActivation, CellGroupTransferActivationAck, + CellGroupTransferArchive, CellGroupTransitionSource, CellGroupWorkload, CellReceiptRequest, + CellReceiptResponse, CellSqlLimits, DirectoryLimits, DirectoryPublication, GroupSourceSeal, + InstalledTransferVoter, ManifestLimits, TransferChildQuorum, TransferInstallQuorum, + VerifiedCellGroupTransitionSources, +}; pub use crate::raft::fleet_catalog::{ fleet_capacity_profile_digest, FleetAbortedMovementCompletion, FleetCapacityAction, FleetCapacityAdmission, FleetCapacityAttestation, FleetCapacityAuthorization, - FleetCapacityRequest, FleetFencedMovementCompletion, FleetGroupAuthorization, + FleetCapacityRequest, FleetCatalog, FleetCatalogError, FleetFencedMovementCompletion, + FleetGroupArchiveReplayOptions, FleetGroupArchiveVerifier, FleetGroupAuthorization, FleetHostAuthorization, FleetIsolationAttestation, FleetMovementAdmission, FleetMovementCompletion, FleetMovementCompletionRequest, FleetMovementExecutionRequest, FleetMovementPhase, FleetMovementProgress, FleetMovementRequest, FleetNodeEligibility, @@ -79,47 +108,69 @@ pub use crate::raft::fleet_catalog::{ FleetTopologyIntentRequest, FleetTopologyMovementEntry, FleetTopologyMovementPage, FleetTopologyOperation, FleetTopologyPhase, }; +pub use crate::raft::fleet_catalog::{ + RaftManagedArchiveProvisioning, RaftManagedFirstGroup, RaftManagedFirstGroupRequest, + ReopenedManagedCellGroupTransition, +}; pub use crate::raft::fleet_stop::{ FleetBootBinding, FleetStopAcceptance, FleetStopAction, FleetStopRequest, VerifiedFleetClosure, }; +pub use crate::raft::group_host::{ + CellGroupNodeArchiveOptions, CellGroupNodeLimits, RaftCellsConfig, + RaftGroupArchiveInitialization, RaftGroupAssignment, RaftGroupPlacement, RaftNamespaceConfig, + RaftNamespaceInitialization, RaftNamespaceProvisioning, +}; pub use crate::raft::host::{ FleetImageState, FleetMembershipConfiguration, FleetMembershipObservation, FleetPublishedCut, - FleetPublishedImage, RaftArchiveOptions, RaftCellPlacement, RaftCellRegistration, - RaftCellsConfig, RaftCellsServices, RaftControllerOptions, RaftControllerStatus, - RaftExtentCacheLimits, RaftExtentLimits, RaftExtentOptions, RaftHostEnrollmentObservation, - RaftHostObservation, RaftHostOptions, RaftHostStartupBounds, RaftInspectionScanOptions, - RaftManagedHostOptions, RaftNamespaceRegistration, RaftNodeDirectoryOptions, - RaftPeerFleetMembershipObservation, RaftPeerNode, RaftPeerNodeObservation, + FleetPublishedImage, RaftArchiveOptions, RaftCellsServices, RaftControllerOptions, + RaftControllerStatus, RaftExtentCacheLimits, RaftExtentLimits, RaftExtentOptions, + RaftHostEnrollmentObservation, RaftHostObservation, RaftHostOptions, RaftHostStartupBounds, + RaftInspectionScanOptions, RaftManagedHostOptions, RaftNamespaceRegistration, + RaftNodeDirectoryOptions, RaftPeerFleetMembershipObservation, RaftPeerNode, + RaftPeerNodeObservation, }; pub use crate::raft::inspection::{ - CellInspection, InspectionError, LeaderLoad, MemberInspection, NodeInspection, RaftNodeProfile, - ReplicaOccupancy, + CellInspection, GroupInspection, InspectionError, LeaderLoad, MemberInspection, NodeInspection, + RaftNodeProfile, ReplicaOccupancy, }; pub use crate::raft::member_state::{ FleetGroupBinding, FleetMembershipPermit, FleetMembershipStatus, FleetPermitPhase, JoiningMember, RepairPhase, RepairPlan, RepairStatus, RepairStep, }; pub use crate::raft::operations::{ - RaftCloseReportLimits, RaftDrainReport, RaftGroupCloseReport, RaftNodeShutdownReport, + RaftCloseReportLimits, RaftDrainReport, RaftNodeShutdownError, RaftNodeShutdownReport, }; pub use crate::raft::options::{LogOpenMode, RaftRuntimeOptions, ReplicaLogOptions}; pub use crate::raft::transport_options::{ - NamespaceInspectionOrigin, NamespaceQueryOrigin, RaftTls, TransportOptions, + NamespaceCommandOrigin, NamespaceInspectionOrigin, NamespaceQueryOrigin, RaftTls, + TransportOptions, }; pub use crate::raft::version::{ VersionError, VersionKeys, VersionSigningKey, VersionToken, VersionVerificationKey, }; pub use crate::storage::{CommandInputs, CommandTransaction}; +pub use cellar_logic::cell_group::{CellBound, CellRange, RouteEpoch}; pub use class::{ command_request_digest, CommandContext, OperationContract, OperationKind, QueryContext, - RaftClass, RegistrationError, + QueryVersion, RaftClass, RegistrationError, }; pub use client::{ - CellStatus, CommandOptions, CommandReceipt, LearnerActivation, LearnerOptions, LearnerReceipt, - QueryConsistency, QueryOptions, QueryResult, QueryRouting, RaftCellHandle, RaftCells, - RaftCloseReport, StaleReadPolicy, UnfinishedCommand, + CellStatus, LearnerActivation, LearnerOptions, LearnerReceipt, UnfinishedCommand, +}; +pub use group_protocol::GroupEntryLimits; + +pub use crate::raft::version::CellVersionToken; +pub use cellar_logic::cell_group::{CommitWitness, GroupHistory}; +pub use group_client::{ + CellGroupCollectionFailure, CellGroupCollectionUsage, CellGroupControlError, + CellGroupImageUsage, CellGroupResourceUsage, CellGroupStartupError, CommandOptions, + CommandReceipt, MutationOutcomes, QueryConsistency, QueryOptions, QueryResult, QueryRouting, + RaftCellHandle, RaftCells, RaftCloseError, RaftCloseReport, RaftGroupCloseReport, + RaftGroupHandle, StaleReadPolicy, }; +pub use crate::raft::version::CellVersionObservation; +pub use cellar_logic::cell_group::{CellHistory, CellId, CellVersion}; pub use cellar_logic::consensus::{ ClientId, ClusterId, CommandFailure, CommandId, CommittedVersion, DiskLineage, Durability, EngineInstanceId, GroupId, GroupScope, IncarnationId, LogIndex, LogPosition, MemberId, @@ -138,7 +189,7 @@ pub mod verification { ArchiveDependency, ArchiveError, ArchiveLimits, ArchiveMetadata, ArchiveRecoverySet, ArchiveReference, ArchiveVerifier, CellArchive, CommittedArchiveCut, RootAllocation, RootArchiveReference, RootArchiveSnapshot, RootDescriptor, RootIntent, RootProtection, - RootReference, RootRegistry, RootRegistryId, RootReservation, VerifiedArchiveCoverage, + RootReference, RootRegistry, RootRegistryId, RootReservation, }; pub use crate::raft::cache::{ CommittedImage, DurableApply, GroupLog, GroupWrite, SnapshotChange, @@ -154,10 +205,12 @@ pub mod verification { RuntimeEventKind, SnapshotCompletion, }; pub use crate::raft::execution::{ - query_registered, AppliedExecution, CommandAttempt, CommandAuthority, CommandRequest, - CommittedReceipt, ExecutionError, ExecutionLimits, ExecutionStatus, PreparedCommand, - PrivateExecutor, PrivateOptions, RegisteredQueryOutput, RootPrivateOptions, StageOutcome, - StoredReceipt, + query_registered, AppliedExecution, CellCommittedReceipt, CellEnrollmentAttempt, + CellEnrollmentResolution, CellPreparedCommand, CellPrivateEnrollment, CellPrivateExecutor, + CellProposalRejection, CellReconciliation, CellStageOutcome, CellSubmissionOutcome, + CommandAttempt, CommandAuthority, CommandRequest, CommittedReceipt, ExecutionError, + ExecutionLimits, ExecutionStatus, PreparedCommand, PrivateExecutor, PrivateOptions, + RegisteredQueryOutput, RootPrivateOptions, StageOutcome, StoredReceipt, }; pub use crate::raft::fleet_catalog::{ FleetBirthAuthorization, FleetBirthReservation, FleetCatalog, FleetCatalogError, @@ -199,11 +252,27 @@ pub mod verification { PreparedSnapshot, RootSnapshotSet, RootSnapshotSource, SnapshotCheckpoint, SnapshotDownload, SnapshotError, SnapshotLimits, SnapshotPackage, SnapshotReference, }; - pub use crate::raft::storage::{GroupBootstrap, LogError, ReplicaLog, SyncReceipt}; + pub use crate::raft::storage::{ + CellGroupBootstrap, GroupBootstrap, LogError, ReplicaLog, SyncReceipt, + }; pub use crate::raft::transport::{ - Delivery, GroupRoute, Lane, RaftPeer, RaftTransport, TransportError, TransportUsage, + CellGroupCheckpointRegistration, CellGroupSeedError, CellGroupSeedSession, Delivery, + GroupRoute, Lane, RaftPeer, RaftTransport, TransportError, TransportUsage, }; pub use crate::raft::transport_options::{ - NamespaceInspectionOrigin, NamespaceQueryOrigin, RaftTls, TransportOptions, + NamespaceCommandOrigin, NamespaceInspectionOrigin, NamespaceQueryOrigin, RaftTls, + TransportOptions, }; + /// Contracts under integration into the replacement group runtime. + pub mod cell_groups { + pub use super::super::group_contracts::cell_routing_key; + pub use super::super::group_protocol::*; + pub use crate::raft::cell_groups::*; + pub use crate::raft::version::{CellVersionObservation, CellVersionToken}; + pub use cellar_logic::cell_group::{ + CellBound, CellHistory, CellId, CellImage, CellImageCommitment, CellKey, CellRange, + CellVersion, CommitWitness, GroupHistory, GroupRoute, MembershipRevision, + MutationSequence, ParkGeneration, RouteEpoch, TransitionId, + }; + } } diff --git a/crates/cellar/application/consensus/protocol.rs b/crates/cellar/application/consensus/protocol.rs index e38488e8f..c49dc7f15 100644 --- a/crates/cellar/application/consensus/protocol.rs +++ b/crates/cellar/application/consensus/protocol.rs @@ -18,7 +18,7 @@ const DIGEST_LEN: usize = 32; // Wire markers remain 1. The immutable namespace identity also binds member // metadata so incompatible binaries cannot attach to the same namespace. const PROTOCOL: &str = - "cellar-raft-ltx-1/members-1/seeds-1/s3-1/repairs-1/nodes-1/bundles-1/fleet-1/enrollment-1/node-probes-1/birth-admission-1/managed-genesis-1/group-reservations-1/birth-inventory-1/namespace-import-1/fleet-namespaces-1/catalog-trust-1/fleet-admission-1/fleet-quorum-1/fleet-cuts-1/empty-archives-1/archive-cuts-1/fleet-finalization-1/fleet-stop-1/fleet-stop-rpc-1/capacity-readiness-1/resource-reservations-1/physical-inventory-1/fleet-removal-1/resource-movement-1/movement-completion-1/node-isolation-1/isolated-removal-1/images-1/fenced-movement-1/fleet-execution-1/aborted-movement-1/topology-bindings-1/topology-intent-1/config-tails-1/restart-stop-1/restart-return-1/restart-resume-1/maintenance-operator-1/stop-consent-refresh-1/extent-owners-1/operator-lifecycle-1/extent-fleet-proofs-1/failure-recovery-1"; + "cellar-raft-ltx-1/members-1/seeds-1/s3-1/repairs-1/nodes-1/bundles-1/fleet-1/enrollment-1/node-probes-1/birth-admission-1/managed-genesis-1/group-reservations-1/birth-inventory-1/namespace-import-1/fleet-namespaces-1/catalog-trust-1/fleet-admission-1/fleet-quorum-1/fleet-cuts-1/empty-archives-1/archive-cuts-1/fleet-finalization-1/fleet-stop-1/fleet-stop-rpc-1/capacity-readiness-1/resource-reservations-1/physical-inventory-1/fleet-removal-1/resource-movement-1/movement-completion-1/node-isolation-1/isolated-removal-1/images-1/fenced-movement-1/fleet-execution-1/aborted-movement-1/topology-bindings-1/topology-intent-1/config-tails-1/restart-stop-1/restart-return-1/restart-resume-1/maintenance-operator-1/stop-consent-refresh-1/extent-owners-1/operator-lifecycle-1/extent-fleet-proofs-1/failure-recovery-1/cell-group-fleet-1/group-import-1"; /// Bounds apply before decoding the header or copying any payload. The LTX /// decoder must additionally enforce these bounds on actual decoded pages; diff --git a/crates/cellar/cli/raft_cli/first_group.rs b/crates/cellar/cli/raft_cli/first_group.rs new file mode 100644 index 000000000..c1021829b --- /dev/null +++ b/crates/cellar/cli/raft_cli/first_group.rs @@ -0,0 +1,162 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Operator receipts must match the exact first-group request. +use super::*; +use serde::Deserialize; + +#[derive(Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct ApplyPlan { + reservation: RaftFleetFirstGroupPlan, + publication: MutationIdentity, + archive: MutationIdentity, +} + +#[derive(Deserialize)] +#[serde(deny_unknown_fields)] +struct MutationIdentity { + request_id: String, + expected_revision: u64, +} + +impl ApplyPlan { + pub fn validate(&self, namespace: &str, catalog: &str) -> Result<()> { + if self.reservation.catalog_id != catalog { + return Err(invalid("first group plan belongs to another catalog")); + } + self.publication + .request(&self.reservation) + .validate(namespace)?; + self.archive + .request(&self.reservation) + .validate(namespace)?; + if self.archive.request_id == self.publication.request_id + || self.archive.expected_revision <= self.publication.expected_revision + { + return Err(invalid( + "first group archive must follow publication with its own identity", + )); + } + Ok(()) + } +} + +impl MutationIdentity { + fn request(&self, plan: &RaftFleetFirstGroupPlan) -> RaftFleetFirstGroupMutation { + RaftFleetFirstGroupMutation { + plan: plan.clone(), + request_id: self.request_id.clone(), + expected_revision: self.expected_revision, + } + } +} + +pub(super) async fn apply( + client: &http::Client, + plan: &ApplyPlan, + timeout: Duration, +) -> Result { + crate::asyncrt::timeout(timeout, async { + let reservation = reserve(client, &plan.reservation) + .await + .map_err(|error| stage_error("reserve", error))?; + let publication = mutate( + client, + &plan.publication.request(&plan.reservation), + "publish", + RaftFleetFirstGroupStage::Published, + ) + .await + .map_err(|error| stage_error("publish", error))?; + let archive = mutate( + client, + &plan.archive.request(&plan.reservation), + "archive", + RaftFleetFirstGroupStage::ArchiveInitialized, + ) + .await + .map_err(|error| stage_error("archive", error))?; + Ok(json!({ "reservation": reservation, "publication": publication, "archive": archive })) + }) + .await + .map_err(|_| unavailable())? +} + +fn stage_error(stage: &str, mut error: RaftFleetFailure) -> RaftFleetFailure { + error.message = format!("First group {stage} stage. {}", error.message); + error +} + +pub(super) async fn standby( + client: &http::Client, + catalog: &str, + namespace: &str, + timeout: Duration, +) -> Result { + let observed = client + .get::( + &format!("namespaces/{namespace}/first-group/standby"), + None, + None, + timeout, + ) + .await?; + observed.validate().map_err(|_| unavailable())?; + if observed.catalog_id != catalog || observed.namespace_id != namespace { + return Err(unavailable()); + } + serde_json::to_value(observed).map_err(|_| unavailable()) +} + +pub(super) async fn reserve( + client: &http::Client, + plan: &RaftFleetFirstGroupPlan, +) -> Result { + let response = client + .post::( + &format!("namespaces/{}/first-group/reserve", plan.namespace_id), + plan, + ) + .await?; + let expected = RaftFleetFirstGroupAcceptance::from_plan( + plan, + &plan.request_id, + plan.expected_revision, + RaftFleetFirstGroupStage::Reserved, + ); + checked(response, expected) +} + +pub(super) async fn mutate( + client: &http::Client, + request: &RaftFleetFirstGroupMutation, + verb: &str, + stage: RaftFleetFirstGroupStage, +) -> Result { + let response = client + .post::( + &format!( + "namespaces/{}/first-group/{verb}", + request.plan.namespace_id + ), + request, + ) + .await?; + let expected = RaftFleetFirstGroupAcceptance::from_plan( + &request.plan, + &request.request_id, + request.expected_revision, + stage, + ); + checked(response, expected) +} + +fn checked( + response: RaftFleetFirstGroupAcceptance, + expected: RaftFleetFirstGroupAcceptance, +) -> Result { + if response != expected { + return Err(unavailable()); + } + serde_json::to_value(response).map_err(|_| unavailable()) +} diff --git a/crates/cellar/cli/raft_cli/mod.rs b/crates/cellar/cli/raft_cli/mod.rs index 7352cf3a4..fedf15318 100644 --- a/crates/cellar/cli/raft_cli/mod.rs +++ b/crates/cellar/cli/raft_cli/mod.rs @@ -3,11 +3,13 @@ // This module is an explicit process boundary for environment, files and exit. #![allow(clippy::disallowed_methods)] //! Human operator commands use authenticated management, not bucket listing. +mod first_group; mod http; mod lifecycle; mod maintenance; mod nodes; mod options; +mod park; use crate::sdk::fleet_operations::*; use serde_json::{json, Value}; use std::time::Duration; @@ -40,6 +42,34 @@ async fn execute(options: &options::Options) -> Result { let client = http::Client::new(options)?; let timeout = Duration::from_secs(30); match &options.action { + Action::Park(action, request) => park::execute(&client, action, request).await, + Action::FirstGroupStandby(namespace) => { + let catalog = options + .catalog_id + .as_deref() + .ok_or_else(|| invalid("missing catalog ID"))?; + first_group::standby(&client, catalog, namespace, timeout).await + } + Action::FirstGroupReserve(plan) => first_group::reserve(&client, plan).await, + Action::FirstGroupApply(plan) => first_group::apply(&client, plan, timeout).await, + Action::FirstGroupPublish(request) => { + first_group::mutate( + &client, + request, + "publish", + RaftFleetFirstGroupStage::Published, + ) + .await + } + Action::FirstGroupArchive(request) => { + first_group::mutate( + &client, + request, + "archive", + RaftFleetFirstGroupStage::ArchiveInitialized, + ) + .await + } Action::GrantMaintenance(node, request) => { maintenance::submit_action(&client, node, request, options.wait, options.timeout).await } @@ -207,6 +237,296 @@ async fn execute(options: &options::Options) -> Result { } serde_json::to_value(view).map_err(|_| unavailable()) } + Action::TransitionStatus(namespace, operation) => { + let view = client + .get::( + &format!("namespaces/{namespace}/transitions/{operation}"), + None, + None, + timeout, + ) + .await?; + check_catalog(options.catalog_id.as_deref(), &view.catalog_id)?; + if view.namespace_id != *namespace + || view.operation_id != *operation + || view.observed_fleet_revision == 0 + { + return Err(unavailable()); + } + serde_json::to_value(view).map_err(|_| unavailable()) + } + Action::TransitionBasis(namespace) => { + let basis = client + .get::( + &format!("namespaces/{namespace}/transition-basis"), + None, + None, + timeout, + ) + .await?; + check_catalog(options.catalog_id.as_deref(), &basis.catalog_id)?; + check_catalog(None, &basis.directory_digest)?; + if basis.namespace_id != *namespace + || basis.fleet_revision == 0 + || basis.directory_revision == 0 + { + return Err(unavailable()); + } + serde_json::to_value(basis).map_err(|_| unavailable()) + } + Action::TransitionAdmit(request) => { + let path = format!( + "namespaces/{}/transitions/{}/admit", + request.namespace_id, request.operation_id, + ); + let admitted = client + .post::(&path, request.as_ref()) + .await?; + check_catalog(None, &admitted.plan_digest)?; + if admitted.request_id != request.request_id + || admitted.operation_id != request.operation_id + { + return Err(unavailable()); + } + serde_json::to_value(admitted).map_err(|_| unavailable()) + } + Action::TransitionPrepare(request) => { + use crate::raft::cell_groups::TransitionPhase; + let path = format!( + "namespaces/{}/transitions/{}/prepare", + request.namespace_id, request.operation_id, + ); + let prepared = client + .post::(&path, request.as_ref()) + .await?; + if prepared.request != **request + || matches!( + prepared.phase, + TransitionPhase::Planned | TransitionPhase::Cancelled + ) + || options::validate_catalog(&prepared.preparation_digest).is_err() + { + return Err(unavailable()); + } + serde_json::to_value(prepared).map_err(|_| unavailable()) + } + Action::TransitionSeal(request) => { + use crate::raft::cell_groups::TransitionPhase; + let path = format!( + "namespaces/{}/transitions/{}/authorize-seal", + request.namespace_id, request.operation_id, + ); + let authorized = client + .post::(&path, request.as_ref()) + .await?; + if authorized.request != **request + || matches!( + authorized.phase, + TransitionPhase::Planned + | TransitionPhase::Prepared + | TransitionPhase::Cancelled + ) + { + return Err(unavailable()); + } + serde_json::to_value(authorized).map_err(|_| unavailable()) + } + Action::TransitionSealSource(request) => { + let path = format!( + "namespaces/{}/transitions/{}/seal-source", + request.namespace_id, request.operation_id, + ); + let sealed = client + .post::(&path, request.as_ref()) + .await?; + if sealed.request != **request + || sealed.applied.as_ref().is_some_and(|cut| { + cut.index == 0 + || cut.term == 0 + || options::validate_catalog(&cut.authorization_digest).is_err() + }) + { + return Err(unavailable()); + } + serde_json::to_value(sealed).map_err(|_| unavailable()) + } + Action::TransitionRetireSource(request) => { + use crate::raft::cell_groups::TransitionPhase; + let path = format!( + "namespaces/{}/transitions/{}/retire-source", + request.namespace_id, request.operation_id, + ); + let retired = client + .post::(&path, request.as_ref()) + .await?; + if retired.request != **request + || !matches!( + retired.phase, + TransitionPhase::Active | TransitionPhase::Retired + ) + || retired + .applied + .as_ref() + .is_some_and(|position| position.index == 0 || position.term == 0) + || (retired.phase == TransitionPhase::Retired && retired.applied.is_none()) + { + return Err(unavailable()); + } + serde_json::to_value(retired).map_err(|_| unavailable()) + } + Action::TransitionCaptureSource(request) => { + let path = format!( + "namespaces/{}/transitions/{}/capture-source", + request.namespace_id, request.operation_id, + ); + let captured = client + .post::(&path, request.as_ref()) + .await?; + if captured.request != **request + || captured.sealed.index == 0 + || captured.sealed.term == 0 + || options::validate_catalog(&captured.sealed.authorization_digest).is_err() + { + return Err(unavailable()); + } + serde_json::to_value(captured).map_err(|_| unavailable()) + } + Action::TransitionSetupChild(request) => { + let path = format!( + "namespaces/{}/transitions/{}/setup-child", + request.namespace_id, request.operation_id, + ); + let setup = client + .post::(&path, request.as_ref()) + .await?; + let matches_step = match (request.step, &setup.result) { + ( + RaftFleetTransitionChildStep::Install, + RaftFleetTransitionChildSetupResult::Installed { + member, + node_id, + disk_lineage, + storage_instance, + }, + ) => { + *member > 0 + && [node_id, disk_lineage, storage_instance] + .into_iter() + .all(|id| options::validate_id(id).is_ok()) + } + ( + RaftFleetTransitionChildStep::Attach, + RaftFleetTransitionChildSetupResult::Attached, + ) => true, + ( + RaftFleetTransitionChildStep::InitializeArchive, + RaftFleetTransitionChildSetupResult::ArchiveInitialized, + ) => true, + ( + RaftFleetTransitionChildStep::Quorum, + RaftFleetTransitionChildSetupResult::Quorum { index, term, phase }, + ) => { + use crate::raft::cell_groups::TransitionPhase; + *index > 1 + && *term > 1 + && matches!( + phase, + TransitionPhase::Sealed + | TransitionPhase::Installed + | TransitionPhase::Archived + ) + } + _ => false, + }; + if setup.request != **request || !matches_step { + return Err(unavailable()); + } + serde_json::to_value(setup).map_err(|_| unavailable()) + } + Action::TransitionVerifyArchive(request) + | Action::TransitionVerifyChildArchives(request) => { + use crate::raft::cell_groups::TransitionPhase; + let child_archives = + matches!(&options.action, Action::TransitionVerifyChildArchives(_)); + let command = if child_archives { + "verify-child-archives" + } else { + "verify-archive" + }; + let path = format!( + "namespaces/{}/transitions/{}/{command}", + request.namespace_id, request.operation_id, + ); + let verified = client + .post::(&path, request.as_ref()) + .await?; + let valid_phase = match verified.phase { + TransitionPhase::Active | TransitionPhase::Retired => true, + TransitionPhase::Archived | TransitionPhase::Published => !child_archives, + _ => false, + }; + if verified.request != **request + || !valid_phase + || options::validate_catalog(&verified.archive_digest).is_err() + { + return Err(unavailable()); + } + serde_json::to_value(verified).map_err(|_| unavailable()) + } + Action::TransitionPublish(request) => { + let path = format!( + "namespaces/{}/transitions/{}/publish", + request.namespace_id, request.operation_id, + ); + let published = client + .post::(&path, request.as_ref()) + .await?; + check_catalog(None, &published.tree_digest)?; + check_catalog(None, &published.publication_digest)?; + if published.request != **request || published.directory_revision == 0 { + return Err(unavailable()); + } + serde_json::to_value(published).map_err(|_| unavailable()) + } + Action::TransitionActivate(request) => { + use crate::raft::cell_groups::TransitionPhase; + let path = format!( + "namespaces/{}/transitions/{}/activate", + request.namespace_id, request.operation_id, + ); + let accepted = client + .post::(&path, request.as_ref()) + .await?; + if accepted.request != **request + || !matches!( + accepted.phase, + TransitionPhase::Published | TransitionPhase::Active + ) + || accepted + .applied + .is_some_and(|position| position.index <= 1 || position.term <= 1) + || (accepted.phase == TransitionPhase::Active && accepted.applied.is_none()) + { + return Err(unavailable()); + } + serde_json::to_value(accepted).map_err(|_| unavailable()) + } + Action::TransitionResume(request) => { + let path = format!( + "namespaces/{}/transitions/{}/resume", + request.namespace_id, request.operation_id, + ); + let accepted = client + .post::(&path, request.as_ref()) + .await?; + if accepted.request != **request + || Some(accepted.applied_revision) != request.expected_revision.checked_add(1) + || accepted.observed_fleet_revision < accepted.applied_revision + { + return Err(unavailable()); + } + serde_json::to_value(accepted).map_err(|_| unavailable()) + } Action::Operation(id) => { let view = client .get::(&format!("operations/{id}"), None, None, timeout) diff --git a/crates/cellar/cli/raft_cli/options.rs b/crates/cellar/cli/raft_cli/options.rs index 017ddb290..ce59ccadd 100644 --- a/crates/cellar/cli/raft_cli/options.rs +++ b/crates/cellar/cli/raft_cli/options.rs @@ -20,8 +20,28 @@ cellar raft fleet nodes add --descriptor NODE.json --operation-id REQUEST_HEX -- cellar raft fleet nodes status NODE_HEX cellar raft fleet nodes qualify NODE_HEX --attestation CAPACITY.json --operation-id REQUEST_HEX --expected-revision R cellar raft fleet nodes activate NODE_HEX --attestation CAPACITY.json --operation-id REQUEST_HEX --expected-revision R +cellar raft fleet first-group standby NAMESPACE_HEX +cellar raft fleet first-group reserve NAMESPACE_HEX --plan PLAN.json +cellar raft fleet first-group publish NAMESPACE_HEX --plan PLAN.json --request-id ID --expected-revision R +cellar raft fleet first-group archive NAMESPACE_HEX --plan PLAN.json --request-id ID --expected-revision R +cellar raft fleet first-group apply NAMESPACE_HEX --plan SETUP.json cellar raft fleet status --fleet F --admin-url URL [--json] cellar raft fleet rebalance --keep NODE_HEX,... --policy POLICY.json --operation-id OP_HEX --expected-revision R [--wait] +cellar raft fleet transition status NAMESPACE_HEX --operation-id OP_HEX +cellar raft fleet transition basis NAMESPACE_HEX +cellar raft fleet transition admit NAMESPACE_HEX --plan REQUEST.json +cellar raft fleet transition prepare NAMESPACE_HEX --plan REQUEST.json +cellar raft fleet transition authorize-seal NAMESPACE_HEX --plan REQUEST.json +cellar raft fleet transition seal-source NAMESPACE_HEX --plan REQUEST.json +cellar raft fleet transition capture-source NAMESPACE_HEX --plan REQUEST.json +cellar raft fleet transition retire-source NAMESPACE_HEX --plan REQUEST.json +cellar raft fleet transition setup-child NAMESPACE_HEX --plan REQUEST.json +cellar raft fleet transition verify-archive NAMESPACE_HEX --plan REQUEST.json +cellar raft fleet transition verify-child-archives NAMESPACE_HEX --plan REQUEST.json +cellar raft fleet transition publish NAMESPACE_HEX --plan REQUEST.json +cellar raft fleet transition activate NAMESPACE_HEX --plan REQUEST.json +cellar raft fleet transition resume NAMESPACE_HEX --plan REQUEST.json +cellar raft fleet group park|park-status|wake NAMESPACE_HEX --plan REQUEST.json cellar raft fleet operation status OP_HEX cellar raft fleet operation progress OP_HEX [--cursor TOKEN] [--limit 64] cellar raft fleet operation wait OP_HEX [--timeout-seconds 600] @@ -37,12 +57,23 @@ it does not certify leader balance or authorize node/storage deletion. Exit codes: 0 success, 2 invalid input, 3 conflict, 4 blocked/cancelled, 5 unavailable/unknown outcome, 6 wait timeout."; +pub(super) enum ParkAction { + Park, + Status, + Wake, +} + pub(super) enum Action { GrantMaintenance(String, Box), MaintenanceStatus(String), PrepareMaintenance(String, Box), CancelMaintenance(String, Box), Status, + FirstGroupStandby(String), + FirstGroupReserve(Box), + FirstGroupPublish(Box), + FirstGroupArchive(Box), + FirstGroupApply(Box), NodeStatus(String), EvacuationStatus(String), Evacuate(String, Box), @@ -54,6 +85,21 @@ pub(super) enum Action { NodeCapacity(String, Box, bool), Begin(RaftFleetBegin), Operation(String), + Park(ParkAction, Box), + TransitionStatus(String, String), + TransitionResume(Box), + TransitionActivate(Box), + TransitionBasis(String), + TransitionAdmit(Box), + TransitionPrepare(Box), + TransitionSeal(Box), + TransitionSealSource(Box), + TransitionCaptureSource(Box), + TransitionRetireSource(Box), + TransitionSetupChild(Box), + TransitionVerifyArchive(Box), + TransitionVerifyChildArchives(Box), + TransitionPublish(Box), Progress(String, Option, usize), Wait(String), Cancel(String, RaftFleetCancel), @@ -85,6 +131,24 @@ pub(super) fn parse(arguments: Vec) -> Result { Some(args.next().ok_or_else(|| invalid("missing node ID"))?) }; (format!("nodes_{verb}"), target) + } else if command == "first-group" { + let verb = args + .next() + .ok_or_else(|| invalid("missing first group command"))?; + let namespace = args.next().ok_or_else(|| invalid("missing namespace ID"))?; + (format!("first_group_{verb}"), Some(namespace)) + } else if command == "transition" { + let verb = args + .next() + .ok_or_else(|| invalid("missing transition command"))?; + let namespace = args.next().ok_or_else(|| invalid("missing namespace ID"))?; + (format!("transition_{verb}"), Some(namespace)) + } else if command == "group" { + let verb = args + .next() + .ok_or_else(|| invalid("missing group park command"))?; + let namespace = args.next().ok_or_else(|| invalid("missing namespace ID"))?; + (format!("group_{verb}"), Some(namespace)) } else if command == "operation" { let command = args .next() @@ -112,6 +176,7 @@ pub(super) fn parse(arguments: Vec) -> Result { let mut policy = None; let mut descriptor = None; let mut attestation = None; + let mut first_group_plan = None; let mut cursor = None; let mut limit = 64usize; let mut wait = false; @@ -175,6 +240,7 @@ pub(super) fn parse(arguments: Vec) -> Result { &mut policy, json_file::(&value)?, )?, + "--plan" => set(&mut first_group_plan, value)?, "--cursor" => set(&mut cursor, value)?, "--limit" => limit = value.parse().map_err(|_| invalid("invalid page limit"))?, "--timeout-seconds" => { @@ -203,6 +269,229 @@ pub(super) fn parse(arguments: Vec) -> Result { validate_catalog(pin)?; } let action = match (command.as_str(), target) { + (verb @ ("group_park" | "group_park-status" | "group_wake"), Some(namespace)) + if !wait && cursor.is_none() => + { + let path = first_group_plan + .take() + .ok_or_else(|| invalid("--plan is required"))?; + let plan = json_file::(&path)?; + plan.validate(&namespace, &plan.operation_id)?; + let catalog = catalog_id + .as_deref() + .ok_or_else(|| invalid("--catalog-id from fleet status is required"))?; + check_catalog(Some(catalog), &plan.catalog_id)?; + let action = match verb { + "group_park" => ParkAction::Park, + "group_park-status" => ParkAction::Status, + _ => ParkAction::Wake, + }; + Action::Park(action, Box::new(plan)) + } + ("transition_status", Some(namespace)) if !wait && cursor.is_none() => { + validate_id(&namespace)?; + let operation = operation_id + .take() + .ok_or_else(|| invalid("--operation-id is required"))?; + validate_id(&operation)?; + if catalog_id.is_none() { + return Err(invalid("--catalog-id from fleet status is required")); + } + Action::TransitionStatus(namespace, operation) + } + ("transition_basis", Some(namespace)) if !wait && cursor.is_none() => { + validate_id(&namespace)?; + Action::TransitionBasis(namespace) + } + ("transition_admit", Some(namespace)) if !wait && cursor.is_none() => { + let path = first_group_plan + .take() + .ok_or_else(|| invalid("--plan is required"))?; + let plan = json_file::(&path)?; + plan.validate(&namespace, &plan.operation_id)?; + let catalog = catalog_id + .as_deref() + .ok_or_else(|| invalid("--catalog-id from fleet status is required"))?; + check_catalog(Some(catalog), &plan.catalog_id)?; + Action::TransitionAdmit(Box::new(plan)) + } + ("transition_prepare", Some(namespace)) if !wait && cursor.is_none() => { + let path = first_group_plan + .take() + .ok_or_else(|| invalid("--plan is required"))?; + let plan = json_file::(&path)?; + plan.validate(&namespace, &plan.operation_id)?; + let catalog = catalog_id + .as_deref() + .ok_or_else(|| invalid("--catalog-id from fleet status is required"))?; + check_catalog(Some(catalog), &plan.catalog_id)?; + Action::TransitionPrepare(Box::new(plan)) + } + ( + command @ ("transition_verify-archive" | "transition_verify-child-archives"), + Some(namespace), + ) if !wait && cursor.is_none() => { + let path = first_group_plan + .take() + .ok_or_else(|| invalid("--plan is required"))?; + let plan = json_file::(&path)?; + plan.validate(&namespace, &plan.operation_id)?; + let catalog = catalog_id + .as_deref() + .ok_or_else(|| invalid("--catalog-id from fleet status is required"))?; + check_catalog(Some(catalog), &plan.catalog_id)?; + if command == "transition_verify-child-archives" { + Action::TransitionVerifyChildArchives(Box::new(plan)) + } else { + Action::TransitionVerifyArchive(Box::new(plan)) + } + } + ("transition_authorize-seal", Some(namespace)) if !wait && cursor.is_none() => { + let path = first_group_plan + .take() + .ok_or_else(|| invalid("--plan is required"))?; + let plan = json_file::(&path)?; + plan.validate(&namespace, &plan.operation_id)?; + let catalog = catalog_id + .as_deref() + .ok_or_else(|| invalid("--catalog-id from fleet status is required"))?; + check_catalog(Some(catalog), &plan.catalog_id)?; + Action::TransitionSeal(Box::new(plan)) + } + ("transition_seal-source", Some(namespace)) if !wait && cursor.is_none() => { + let path = first_group_plan + .take() + .ok_or_else(|| invalid("--plan is required"))?; + let plan = json_file::(&path)?; + plan.validate(&namespace, &plan.operation_id)?; + let catalog = catalog_id + .as_deref() + .ok_or_else(|| invalid("--catalog-id from fleet status is required"))?; + check_catalog(Some(catalog), &plan.catalog_id)?; + Action::TransitionSealSource(Box::new(plan)) + } + (command @ ("transition_capture-source" | "transition_retire-source"), Some(namespace)) + if !wait && cursor.is_none() => + { + let path = first_group_plan + .take() + .ok_or_else(|| invalid("--plan is required"))?; + let plan = json_file::(&path)?; + plan.validate(&namespace, &plan.operation_id)?; + let catalog = catalog_id + .as_deref() + .ok_or_else(|| invalid("--catalog-id from fleet status is required"))?; + check_catalog(Some(catalog), &plan.catalog_id)?; + if command == "transition_retire-source" { + Action::TransitionRetireSource(Box::new(plan)) + } else { + Action::TransitionCaptureSource(Box::new(plan)) + } + } + ("transition_setup-child", Some(namespace)) if !wait && cursor.is_none() => { + let path = first_group_plan + .take() + .ok_or_else(|| invalid("--plan is required"))?; + let plan = json_file::(&path)?; + plan.validate(&namespace, &plan.operation_id)?; + let catalog = catalog_id + .as_deref() + .ok_or_else(|| invalid("--catalog-id from fleet status is required"))?; + check_catalog(Some(catalog), &plan.catalog_id)?; + Action::TransitionSetupChild(Box::new(plan)) + } + ("transition_publish", Some(namespace)) if !wait && cursor.is_none() => { + let path = first_group_plan + .take() + .ok_or_else(|| invalid("--plan is required"))?; + let plan = json_file::(&path)?; + plan.validate(&namespace, &plan.operation_id)?; + let catalog = catalog_id + .as_deref() + .ok_or_else(|| invalid("--catalog-id from fleet status is required"))?; + check_catalog(Some(catalog), &plan.catalog_id)?; + Action::TransitionPublish(Box::new(plan)) + } + ("transition_activate", Some(namespace)) if !wait && cursor.is_none() => { + let path = first_group_plan + .take() + .ok_or_else(|| invalid("--plan is required"))?; + let plan = json_file::(&path)?; + plan.validate(&namespace, &plan.operation_id)?; + let catalog = catalog_id + .as_deref() + .ok_or_else(|| invalid("--catalog-id from fleet status is required"))?; + check_catalog(Some(catalog), &plan.catalog_id)?; + Action::TransitionActivate(Box::new(plan)) + } + ("transition_resume", Some(namespace)) if !wait && cursor.is_none() => { + let path = first_group_plan + .take() + .ok_or_else(|| invalid("--plan is required"))?; + let plan = json_file::(&path)?; + plan.validate(&namespace, &plan.operation_id)?; + let catalog = catalog_id + .as_deref() + .ok_or_else(|| invalid("--catalog-id from fleet status is required"))?; + check_catalog(Some(catalog), &plan.catalog_id)?; + Action::TransitionResume(Box::new(plan)) + } + ("first_group_standby", Some(namespace)) if !wait && cursor.is_none() => { + validate_id(&namespace)?; + if catalog_id.is_none() { + return Err(invalid("--catalog-id from fleet status is required")); + } + Action::FirstGroupStandby(namespace) + } + ("first_group_apply", Some(namespace)) if !wait && cursor.is_none() => { + let path = first_group_plan + .take() + .ok_or_else(|| invalid("--plan is required"))?; + let plan = json_file::(&path)?; + let catalog = catalog_id + .as_deref() + .ok_or_else(|| invalid("--catalog-id from fleet status is required"))?; + plan.validate(&namespace, catalog)?; + Action::FirstGroupApply(Box::new(plan)) + } + ( + verb @ ("first_group_reserve" | "first_group_publish" | "first_group_archive"), + Some(namespace), + ) if !wait && cursor.is_none() && operation_id.is_none() => { + let path = first_group_plan + .take() + .ok_or_else(|| invalid("--plan is required"))?; + let plan = json_file::(&path)?; + plan.validate(&namespace)?; + let catalog = catalog_id + .as_deref() + .ok_or_else(|| invalid("--catalog-id from fleet status is required"))?; + if catalog != plan.catalog_id { + return Err(invalid("first group plan belongs to another catalog")); + } + if verb != "first_group_reserve" { + let request = RaftFleetFirstGroupMutation { + plan, + request_id: request_id + .take() + .ok_or_else(|| invalid("--request-id is required"))?, + expected_revision: expected_revision + .take() + .ok_or_else(|| invalid("--expected-revision is required"))?, + }; + request.validate(&namespace)?; + if verb == "first_group_archive" { + Action::FirstGroupArchive(Box::new(request)) + } else { + Action::FirstGroupPublish(Box::new(request)) + } + } else { + if request_id.is_some() || expected_revision.is_some() { + return Err(invalid("reservation identity belongs in the plan")); + } + Action::FirstGroupReserve(Box::new(plan)) + } + } ("nodes_maintenance-status", Some(operation)) if cursor.is_none() => { validate_id(&operation)?; Action::MaintenanceStatus(operation) @@ -435,6 +724,7 @@ pub(super) fn parse(arguments: Vec) -> Result { || cursor.is_some() || descriptor.is_some() || attestation.is_some() + || first_group_plan.is_some() || request_id.is_some() || expected_process.is_some() || next_process.is_some() @@ -505,7 +795,7 @@ fn json_file(path: &str) -> Result { serde_json::from_slice(&read_file(path, 16 * 1024)?).map_err(|_| invalid("invalid JSON input")) } -fn validate_id(value: &str) -> Result<()> { +pub(super) fn validate_id(value: &str) -> Result<()> { let mut bytes = [0; 16]; if value.len() != 32 || hex::decode_to_slice(value, &mut bytes).is_err() @@ -520,7 +810,7 @@ fn validate_id(value: &str) -> Result<()> { } } -fn validate_catalog(value: &str) -> Result<()> { +pub(super) fn validate_catalog(value: &str) -> Result<()> { let mut bytes = [0; 32]; if value.len() != 64 || hex::decode_to_slice(value, &mut bytes).is_err() diff --git a/crates/cellar/cli/raft_cli/park.rs b/crates/cellar/cli/raft_cli/park.rs new file mode 100644 index 000000000..d8ae28d49 --- /dev/null +++ b/crates/cellar/cli/raft_cli/park.rs @@ -0,0 +1,62 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::{options::ParkAction, *}; + +pub(super) async fn execute( + client: &http::Client, + action: &ParkAction, + request: &RaftFleetParkRequest, +) -> Result { + let verb = match action { + ParkAction::Park => "park", + ParkAction::Status => "status", + ParkAction::Wake => "wake", + }; + let path = format!( + "namespaces/{}/parks/{}/{verb}", + request.namespace_id, request.operation_id + ); + match action { + ParkAction::Park => { + let status = client.post::(&path, request).await?; + verify(request, &status)?; + if status.phase == RaftFleetParkPhase::Negotiating { + return Err(unavailable()); + } + serde_json::to_value(status).map_err(|_| unavailable()) + } + ParkAction::Status => { + let status = client + .post::>(&path, request) + .await?; + if let Some(status) = &status { + verify(request, status)?; + } + serde_json::to_value(status).map_err(|_| unavailable()) + } + ParkAction::Wake => { + let status = client.post::(&path, request).await?; + verify(request, &status.park)?; + if status.park.phase == RaftFleetParkPhase::Negotiating + || status.applied_index < status.park.generation + || status.applied_term == 0 + { + return Err(unavailable()); + } + serde_json::to_value(status).map_err(|_| unavailable()) + } + } +} + +fn verify(request: &RaftFleetParkRequest, status: &RaftFleetParkStatus) -> Result<()> { + check_catalog(Some(&request.catalog_id), &status.catalog_id)?; + if status.namespace_id != request.namespace_id + || status.group != request.group + || status.operation_id != request.operation_id + || status.generation == 0 + || status.observed_revision <= request.expected_revision + { + return Err(unavailable()); + } + Ok(()) +} diff --git a/crates/cellar/durability/raft/README.md b/crates/cellar/durability/raft/README.md index d2ce011ad..48ead9908 100644 --- a/crates/cellar/durability/raft/README.md +++ b/crates/cellar/durability/raft/README.md @@ -14,6 +14,189 @@ under implementation. The [technical design](../../../../docs/design/multi-raft/technical-design.md) defines the complete contract. +The replacement [cell-group design](../../../../docs/design/multi-raft/cell-groups/design.md) +is under implementation. `verification::cell_groups::CellGroupOwner` now owns one +real RawNode for a physical group history and consumes a verified complete root +plus a revalidated initial-member bootstrap permit. It emits persistence, +transport and ordered application effects; it performs no SQLite I/O or +per-cell task creation. The existing node log worker syncs each write before +the owner releases dependent messages or advances the selected group root. +The exclusive ownership lease remains with queued engine work after cancellation, +so a replacement owner cannot start while that write is still in flight. + +Application completions bind the owner instance, exact entry, preceding root, +engine and manifest store. A slow cell holds the contiguous apply prefix while +other entries commit. Bounded backlog applies transport/proposal backpressure; +already collected Ready/LightReady batches retain room for their largest entry. +ReadIndex output is a barrier only: serving still requires current authority and +a coherent cell image after application through that index. The adapter currently +refuses snapshot/configuration/topology administration until those complete-group +protocols are connected. `CellGroupRuntime` now multiplexes physical group owners +on one execution-domain worker per node. It reuses the bounded mailbox, byte/item +admission, joined terminal result and existing persistence engine. Attachment +reserves ordinary node admission before catalog revalidation or RawNode construction. A fair active +queue collects writes across histories into one engine batch; queued and in-flight +work stay charged. Peer-frame admission retains its node/group reservations until +execution or discard and grants no Raft acknowledgement. Completion requests have +reserved capacity independent from ordinary traffic. Physical settlement can +progress under full event admission, releasing the apply reservation before its +Applied event is admitted. Shutdown stops admission, joins the submitted batch, +then fences the engine before releasing exclusive runtime ownership. + +`CellPrivateExecutor` now stages a cell-addressed mutation from an exact pinned +cell base, with materialized or extent-backed SQLite images. It reuses the existing +receipt admission, maintenance and atomic business/receipt transaction. The actual +leader issues `CellExecutionAuthority`; another cell may advance while this base +remains selected. Losing leadership or dropping the owner revokes the observation. +A callback must recheck it after capture, and settlement requires a fresh authority, +the exact committed entry and that cell's expected image/version. A receipt retry +reads the selected immutable cell image and does not invoke the callback again. +The synchronous executor must run on an owning blocking worker. It currently +reopens after settlement. Receipt proof alone does not provide S3 coverage. + +`CellGroupSqlPool` now sends cell-history requests through the existing fixed +`host_sql::Pool` workers and fair request queue. A stable cell key chooses its +worker; the node does not create a thread or task for each enrolled cell. One +admission lease covers queued SQL, the physical callback and its unresolved +proposal even when the waiting caller is cancelled. Node/group byte and item +reservations cover capture and retained outputs. Finished capture refunds its +transient buffer budget while prepared/receipt responses retain their payload +charge, including clones. Extent workspace creation and final cleanup run on the +admitted blocking worker. An exclusive engine lease prevents a second node SQL +pool until workers, the handle and retained outputs release their ownership. A lost stage response can recover the same prepared proposal without +another callback. Admission checks the authority's exact engine owner. + +Each worker has one reserved progress slot, independent of its ordinary queue. +Submission marking is idempotent for the retained proposal. Exact-entry settlement +requires the committed receipt; group progress cannot release another cell's lease. Pool close stops admission and its domain cleanup owner joins every +physical callback/worker even if a close waiter is cancelled. Closing a writer or +pool does not establish a failed submitted outcome or authorize replay; the host +must reconcile pending commands against committed state before admitting again. + +`CellReconciliationAuthority` selects a verified cell pin from the durable +current-term applied group root, including on followers. It grants no write or +strong-read authority. The pool can reconcile its exact pending proposal against +the original command identity in that image even after a later cell mutation; +this lookup does not require retaining the original Raft entry. It validates +identity, sequence floors and result allocation bounds without admitting a new +client or receipt; a full identity table cannot obscure absence. New commands +still enforce both client and retained-receipt ceilings. A matching receipt +settles the command. Absence can release the producer only after a strictly newer +term's applied barrier excludes its old proposal. Replicated expiry/conflict +refusals remain distinct from a known uncommitted outcome. Foreign engines, +changed routes/histories, stale cell pins and same-term absence refuse without +releasing the producer. Missing producers return uncertainty after a lost response. +Reconciliation uses the reserved worker progress slot and closes the disposable +writer before releasing cell admission; retained receipt outputs keep their byte +reservation. A retry still needs fresh leader authority and receipt admission. +Actual snapshot/compaction integration and production retry dispatch remain pending. + +Enrollment now uses `CellEnrollmentAuthority`, issued by the actual leader after +its current-term committed prefix is applied. `initialize` runs one SQLite +initializer on the existing SQL workers, captures the full initial LTX image and +proposes a cell-addressed enrollment. Followers apply those bytes without running +the initializer. Admission keys the logical cluster/namespace/cell name, so another +incarnation cannot bypass an in-flight producer. Ordinary commands use the same +name reservation. Queued and running initialization retain it through cancellation. + +Prepared SQL output carries its original leader term and engine/group identity. +Use `CellSqlPrepared::submit` (or `CellGroupRuntime::submit` for standalone +verification producers). The actual RawNode owner checks those bindings before +new admission. Acceptance is remembered by that exact preparation, so a lost +response can return its original position even after apply; it never proves a +commit or S3 coverage. The first valid prepared submission irreversibly disables +byte-only `propose`/`propose_at_term` for that owner, including raw requests already +queued behind it. This prevents delayed byte-only retries bypassing a seal. + +Only a definitive owner capacity refusal seals the preparation as rejected. +Generic queue overload, stale authority and missing responses grant no such +proof. Every clone shares the seal and can recover it through `rejection()`, even +if the original rejection response was lost. Returning capacity cannot unseal +old output. The pool's `reject` progress operation matches the exact preparation, +closes its private writer, then releases logical-name admission. It rejects +foreign engines and proofs for another preparation, including fresh retries with +the same proposal ID. Retained prepared outputs, queued submissions and rejection +proofs keep their SQL byte reservations through cancellation and delivery. + +The owner retains a bounded acceptance record for each pending logical cell name. +An exact proposal retry returns its original position and digest; different bytes +under that pending identity refuse. Independent proposals for the same pending +name wait. `max_cells_per_group` counts applied cells plus accepted creations +before append, so concurrent enrollments cannot overfill the manifest. Existing +cells still accept mutations at capacity. Mutation admission checks its exact +base against the selected cell version before entering Raft. + +Acceptance records retain a separate node-wide budget of `max_requests` items and +`max_request_bytes`, charging 4 KiB per record without retaining LTX payloads or +image pins. This budget cannot block Ready or completion work. Records release +only after exact durable apply, role/term loss, terminal failure, or owner destruction. A new leader +cannot admit until its applied current-term barrier covers inherited entries. +Without retained acceptance for that preparation, stale mutation bases and +existing enrollments return `StaleCompletion`: callers must resolve the original +receipt or creation record. +That refusal does not establish that the prior attempt failed. + +A committed creation retry returns the immutable enrollment record without opening +SQLite or rerunning the callback, even after later cell mutations. An enrollment +state selected from a durable current-term applied root also permits resolution +on a follower: an exact creation record proves commitment; absence/conflict can +retire an older producer only after a strictly newer applied term. Resolution +closes the writer before releasing name admission, while retained responses keep +their byte charge. Unknown producers remain uncertain. Pool shutdown joins an +entered initializer even if its waiter and close waiter were both cancelled. +As with commands, shutdown is not proof that a submitted enrollment failed. + +`CellGroupRuntime::read_authority` now selects one coherent cell image for an +invocation under an absolute execution-domain deadline. Linearizable selection +uses a fresh nonce/counter-correlated ReadIndex, current leader term and contiguous +applied group prefix. A committed update to B can delay a read of unchanged A, +but cannot change A's version. Raw read requests and forwarded read messages +cannot use the internal context namespace. The returned authority is consumed by +one query and rejects a changed route, lost leader term, foreign engine or expired +deadline. It is not a reusable leadership lease. Each selected leaf retains its complete +commit witness independently of the Raft entry cache. Immutable-query results +expose `observation()` for the exact version and witness; cell receipt results +expose the corresponding selected-version observation, which can cover an older +command receipt. A later mutation of another cell cannot change that observation. +Registered dispatch must still authenticate contracts and issue tokens only from +these authoritative results; manifest observation data alone is not authority. + +Stale explicitly selects a local committed image, including on followers. AtLeast +compares logical cell history/sequence/image, not group log positions; the +production dispatcher must authenticate the token and schema/transfer lineage. +Neither path needs a new quorum. Future versions wait under the same original +deadline, while wrong histories or inconsistent equal-sequence images refuse. + +Waiting reads and returned pins use a separate node budget of `max_pending_reads` +items and `max_read_pin_bytes` bytes; each group admits at most +`(max_pending_reads / 8).clamp(1, 32)` items under the same byte ceiling. Selecting +an image extends the reservation to its logical bytes plus metadata, refusing +before delivery if the ceiling is exceeded. This is a conservative pin-retention +bound, not a claim that every image byte is resident. The pin-byte limit defaults +to 64 MiB and can cover the native image-size envelope independently of transport +limits; configure it for the largest admitted images and concurrent reads. +Reads release ordinary peer +queue slots before waiting for quorum/apply. Cancellation, deadlines and shutdown +retire pending waiters; returned pins keep their reservations until consumed or +dropped. + +`CellGroupSqlPool::query` consumes that authority on the existing fixed workers. +It opens the immutable materialized/extent view without a private workspace or +schema initialization. Queries may read committed state while a private writer +is unresolved. They retain the read-only SQL authorizer, bounded output admission, +worker-owned pins and original deadline. SQLite's progress hook checks domain +expiry, waiter cancellation and shutdown; arbitrary callback work remains owned +and must finish before shutdown joins. Callback panics are contained. Result +accessors borrow bytes so their output reservation cannot be detached. + +Registered SDK creation/command/query dispatch, catalog routing integration and +the production hard cutover remain required; the existing host still uses its +prior single-cell owner. The new pool's callback entry point remains a verification +boundary. Transport authentication and actual image preparation are also caller +responsibilities at that boundary. See the +[implementation status](../../../../docs/design/multi-raft/cell-groups/progress.md) +for the exact qualification boundary. + Managed empty hosts expose `RaftCells::inspect_peer_node` under rooted fleet trust. It authenticates both physical processes and a requested minimum installed trust revision without constructing a cell. Current enrollment state/epoch is @@ -544,7 +727,9 @@ pinned until the callback joins and its private mutations are discarded. A new term and applied startup barrier are required to reconcile an unresolved submission. Closing records unfinished identity and refuses further output. `Node::start_raft_cells` installs registered groups in the node's execution -domain. `host.rs` owns serial command producers and effect dispatch; +domain. `group_host/mod.rs` owns canonical namespace/group startup; the shared controller +owns serial group apply and bounded cell command producers. `host.rs` retains +physical configuration and observation values; `host_sql/mod.rs` owns SQLite/image resources in a fixed node-wide blocking pool. Its private `images` module selects physical command/read adapters beneath the shared controller. The extent branch is exercised by the worker integration @@ -710,70 +895,44 @@ After submission, the producer remains owned through consensus and publication. A newer committed/published term barrier can retire old uncertainty even on a follower, after disposing of its private shadow; that grants no write authority. -MR-08's local read controller holds each request's original deadline and byte/item -permit while waiting for an image. `AtLeast` verifies an opaque signed token -before comparing history positions. `Linearizable` uses fresh safe ReadIndex -contexts, waits for actual image publication, and never treats a runtime status -or cached commit position as a barrier. A checked role/term change or lost reply -can resubmit the invocation's barrier under the same deadline. Retries wait four -host polling intervals, clamped to 100–1000 ms, and still require a safe ReadIndex -result. Status and submission generations prevent late replies from regressing -an executing query. - -The SQL owner captures the durable applied marker with its exact immutable pin. -Protocol-only entries can cover that pin; an unrelated newer image cannot relabel -it. Queries retain the pin through callback completion, and their result metadata -names that captured cut. The default Stale policy requires a recent observed -source; `DisconnectedHistorical` explicitly permits an older verified image -without a time bound. This policy observation is never used for strong reads. -SQL progress hooks observe an atomic cancellation flag raised by the node's -deadline controller, including for an unpolled waiter. Native Rust callbacks -remain trusted/cooperative and are joined before resource release. -Completed responses retain their byte admission until consumed or dropped; -unpolled response channels cannot reuse capacity merely because SQL finished. - -`RaftCellsServices::versions` injects managed signing/verification keys. Ed25519 -tokens use bounded canonical marker-1 framing and bind full history, schema and -log position. Namespace startup reconciles public-key retention metadata through -conditional updates and exact reads; configuration cannot silently drop a live -verifier on restart. The catalog supplies no new trust roots or consensus -authority. Key rotation and token lifetime are described in the +The shared cell-group query controller fixes each invocation's original deadline +and bounded attempt count. `Stale`, `AtLeast` and `Linearizable` are distinct +contracts. `AtLeast` authenticates a CellVersionToken and compares the logical +cell's version, rather than an unrelated group's log index. `Linearizable` uses +a fresh safe ReadIndex barrier and waits for its actual applied root. Runtime +status and cached commit positions cannot substitute for that barrier. + +The SQL owner retains the selected cell's exact immutable image through callback +completion. Returned observations separate CellVersion from its physical +CommitWitness, and protocol-only entries cannot relabel a different image. +The public SDK's QueryResult names the serving NodeIdentity. The lower transport +retains the authenticated responding member as part of its own delivery result. +Completed responses retain their byte admission until consumed or dropped. +Native callbacks remain cooperative and are joined before resource release. + +RaftCellsServices::versions injects managed Ed25519 signing/verification keys. +The bounded canonical marker-1 cell token binds logical history, schema, version, +commit witness, issuance and expiry. Namespace startup reconciles verifier +retention through conditional updates and exact reads. The registry supplies no +new trust roots or consensus authority. See the [application contract](../../../../docs/design/multi-raft/application-contract.md). -`QueryRouting::Replicas` tries up to eight distinct configured members in priority -order, with one optional untried leader hint. The caller supplies a bounded -per-attempt timeout; every attempt keeps the same contract, captured input, -consistency/token and stale policy under the invocation's original deadline. -Forwarding does not require query code on the origin voter. The destination -still requires a matching query definition before execution. `served_by` names -the actual responding member, independently of the history token. - -`NodeRaftListeners::query` and `RaftPeer::query_address` provide the separate -query endpoint. A peer has one bounded query queue/connection alongside its -control and LTX connections. A shared pool owns the physical RaftPeerNode -endpoint; it retains no group or member identity. Each consensus request still -carries and validates its own full member route. Query MAC domains, exporter -label and marker-1 framing are independent of consensus frames. Requests/replies bind full route, -schema, sequence, request tag and payload digest. The receiver forces local -query execution; it cannot recursively route. The sender deducts queued and -handshake time before sending a remaining timeout, while the origin separately -enforces its deadline. Closing the connection cancels the destination waiter. -Request/response representations remain charged through delivery. - -Query headers use a distinct route representation: the caller carries its -node/disk/process identity and label, with no Raft member ID. The target carries -its complete replica/history identity. The response reverses that exact route; -its orientation cannot be reused as a request. Consensus headers retain their -separate member-bearing routes and validators. This is a marker-1 hard cutover -for experimental query peers; incompatible headers are refused without an old -reader or translation path. - -Each `RaftCellRegistration` selects `RaftCellPlacement::Replica` or -`QueryRouter { leader_hint }`. A query router reads an existing namespace and -birth record, verifies the expected history/initial placement, and installs -discovery without creating a local member or SQLite image. It cannot reinterpret -existing member storage as a query-only binding. Its SDK handle rejects commands, -local reads and local status; use explicit `QueryRouting::Replicas`. The optional -hint is discovery only and cannot supply a read proof. + +The separate peer query listener has its own bounded queue, connection and byte +banks alongside control and LTX traffic. The same authenticated framing carries +cell queries, commands, enrollments and receipt lookups, with distinct purposes. +MAC domains, TLS exporter, schema, sequence, exact reversed response route, +request tag and payload digest remain checked. A route binds the physical group, +logical cell and exact node/disk/process endpoints. Namespace read permission +never grants command, enrollment, receipt mutation or consensus authority. + +The receiver executes a selected cell query locally. It cannot recursively route +the request. Queued/handshake time consumes the sender's original deadline before +it transmits the remaining timeout. Connection closure cancels the waiter while +entered physical work keeps its owner and admission until it completes. +The retired QueryRouting/QueryRouter SDK path, per-cell query wire bodies and +Cell history reader are removed. Marker 1 remains unchanged; old peers refuse. +Public SDK/fleet startup qualification is tracked in the +[cell-group progress record](../../../../docs/design/multi-raft/cell-groups/progress.md). `RaftCellsConfig::query_origins` supplies trusted `NamespaceQueryOrigin` grants for exact cluster/namespace/node/disk/process endpoints. A receiving replica @@ -792,10 +951,9 @@ supplies current trusted grants on restart; durable control-plane distribution belongs to the placement/operations controllers. Routing-only views and peer pools are bounded and never initialize a voter or learner. The [read audit](../../../../docs/design/multi-raft/read-audit.md) retains qualification -scope and remaining delivery work. The counter's native -and Axum HTTP paths expose consistency, tokens and bounded member options, and -their executable restart drill includes remote strong reads and cross-cell refusal. -These paths do not claim completion of MR-08 or the full delivery scope. +scope and remaining delivery work for its captured source. Older counter/Axum +and restart-drill fixtures still need the shared SDK cutover; their former member +routing APIs no longer describe the canonical query contract. Fresh status checks precede staging, submission and successful output. Transient protocol Ready work defers preflight or duplicate-receipt settlement under the @@ -1591,3 +1749,207 @@ require their fleet guard. Earlier roots/state have no compatibility reader. Use a fresh experimental namespace/prefix and local state. The earlier MR-10 evidence retains its exact historical protocol and source scope. + +## Shared-group registered command boundary + +`CellGroupSqlPool::stage_registered` dispatches a namespace's compiled `RaftClass` +on the existing fixed SQL workers. The selected route and private-image options +must agree on the full class schema. Operation contract, normalized input digest, +inbox authentication and reserved system-client checks precede receipt lookup, +including duplicate retries. The callback receives the exact logical `CellHistory` +from the execution authority. Prepared output follows the existing proposal, +publication and receipt-settlement path; it is not a committed response. +Registered input bytes are retained within the producer budget while queued or +executing. Cancelling the waiter does not release the physical producer. + +`CommandContext::identity()` and `RaftEffectTarget` now use `CellHistory` without a +physical group ID. Alarm clients, outbox lane identities and signed requests use +that logical history; normal group reassignment preserves them, while a new cell +incarnation separates them. Signed effects use canonical marker-1 logical-history +records and distinct `cellar-cell-delivery-*` domains. Old group-bearing records +and old signature domains refuse. The full class manifest uses the +`cellar-cell-class-1` domain, so this identity change cannot silently reuse a +previous namespace schema. There is no migration reader. + +The temporary single-cell host projects its actual logical binding into +`CellHistory`; the shared command path does not manufacture `GroupScope` values. +Registered shared queries use the boundary below. Catalog/SDK routing, shared +effect scheduling and transfer qualification remain separate work. These APIs do not by themselves complete the production cutover. + +## Shared-group registered query boundary + +`CellGroupSqlPool::query_registered` consumes the invocation's `CellReadAuthority` +and dispatches a compiled query on the existing workers. Operation names and +input are bounded before queueing; retained input and output use read admission. +The worker checks the operation kind, contract and full class schema before +opening the selected immutable image. The same owner retains deadline, SQL +interruption, cancellation, panic containment and joined shutdown behavior. +A cancelled receiver cannot release a still-running callback or its image pins. + +`QueryContext::identity()` exposes logical `CellHistory`. On this shared path, +`version()` yields `QueryVersion::Cell` containing the exact selected +`CellVersionObservation`: mutation sequence, image commitment and decision witness. +The returned `CellSqlQueryResult` retains that same observation for token issuance. +Advancing another cell's group log position cannot relabel the selected version. +A previously pinned image may remain a coherent historical read after publication. + +The temporary old host invokes `invoke_group_query` and supplies an explicit +`QueryVersion::GroupCut`. It cannot supply or infer a logical mutation sequence. +This in-memory staging observation is not a persisted-format reader or a final +runtime choice. CG-16 must remove that variant and constructor with the old host, +leaving the selected cell observation directly exposed by the final query context. +Catalog/SDK dispatch into these workers remains required before public cutover. + +### Catalog-bound query dispatch + +`CellGroupDispatcher` binds one registered namespace directory, compiled class, +version-key set and execution domain. Its clones share a bounded invocation bank; +there are no per-cell worker pools. `query` freezes the operation, input and +consistency before discovery, normalizes retained input capacity and charges both +the retained request and one physical input copy. The canonical fingerprint binds +logical cell identity, invocation ID, contract, input, consistency/token and routing +limits. A non-cloneable `CellQueryInvocation` uses the existing +`CellRoutingSession` for initial lookup, cached routes, retries and confirmed +redirects. The original absolute deadline and total attempt count never reset. + +Each attempt is consumed before asynchronous execution. A caller must explicitly +retry after losing or cancelling a response. The local destination must use the +same execution-domain clock, and its read authority must match the entire +catalog-confirmed route and logical cell. The existing SQL pool checks engine and +schema ownership. AtLeast tokens are verified against the configured keys, exact +cell history, schema and current captured-domain wall time. The response retains +SQL output admission and signs the actual selected observation. Catalog mapping +alone cannot manufacture a leader, group attachment or read authority. + +Cancellation can release the invocation's reservation; entered SQL still owns its +input, image pin, worker and independent admission until physical cleanup joins. +A timeout includes discovery, authority selection, callback execution and output +signing. Cross-host transports must translate a remaining deadline through their +own bounded request protocol; unrelated monotonic timestamps are refused here. +This is the local query dispatch integration boundary. Command/enrollment +invocations, transport selection and replacing the public SDK's old host binding +remain required; it does not complete CG-04 or the public cutover. Group topology +fencing is still owned by the later transition tasks, not by directory metadata. + +### Catalog-bound command phases + +`CellGroupDispatcher::command` captures a complete immutable command invocation, +including the caller's stable command identity/retry window and domain-owned time +and randomness. Its fingerprint and original deadline include that capture. The +same namespace/class binding and shared invocation admission used by queries also +bound commands. Current shared-group command dispatch requires both the request +and namespace minimum to be Quorum. S3 refuses with `UnsupportedCapability` until +CG-06 supplies a real complete-group archive proof; no acknowledgement is lowered. + +`CellCommandInvocation` exposes preparation, provisional submission and settlement +as distinct phases for the production controller. Before entering SQL it retains +its originating runtime, pool, exact route and fresh preparation IDs. Cancelling +preparation loses only the response. Resuming retrieves that same worker-owned +producer, even if a routing retry selects another member. A catalog redirect never +releases an unresolved producer or runs its callback at a second destination. +The controller must retain invocations independently from ingress response waiters. + +Submission uses the existing shared acceptance/seal state and original SQL +ownership. Repeated acceptance cannot append twice. An exact pre-append rejection +seals every clone before cleanup. Settlement reads the original receipt through +current applied authority; same-term absence stays uncertain. Only the existing +strictly newer applied-term proof can produce `NotCommitted`. A subsequent +preparation uses fresh physical IDs with the same logical command and captured +inputs. Lost duplicate/settlement responses may use registered receipt lookup with +new command execution disabled. On a follower, a missing local producer still +requires routing to a leader for that fallback; it is never evidence of absence. + +Returned `CellCommandResponse` retains SQL output admission and signs the selected +cell observation. Acceptance itself is never a response carrying durability. +Deadline checks cover each phase and output delivery. Errors retain the command +ID and an explicit ambiguity bit; a deadline, lost reply or redirect is not a +non-commit proof. Dropping the invocation never releases its worker's unresolved +producer. The node controller still needs to own these phases through ingress +cancellation and drive shared transport/apply/completion events. Enrollment routing, +that production integration and final public SDK replacement remain unfinished. + +### Physical-group peer transport + +`RaftTransport::start_shared` starts directly on `CellGroupRuntime`; it never +constructs the old cell-scoped runtime. Physical routes use +`GroupRoute` and `RaftPeer`. Installing a route checks +an existing group, selected schema, committed member/disk bindings, local process +and listener addresses. It creates no storage or Raft membership and grants no +serving authority. Unknown groups and unsupported complete-group snapshots refuse. + +Physical groups share the existing per-node TLS connections, control/data lanes, +pipelining, bounded byte/item banks and joined shutdown. Group routes share the +same configured group limit with temporary old/query routes. Node-view refresh, +fleet revocation and pool pruning include physical routes. Query/inspection grants +cannot replace conflicting physical-group endpoints. Queue admission, I/O and +observer delivery recheck exact route/process and committed membership. A delivery +response acknowledges receiver queue admission, never append, commit or apply. +Failed sends report bounded unreachable hints without fabricating Raft responses. + +Marker-1 peer frames now carry an explicitly typed history under a new protocol +digest. Physical frames contain group history without a cell key. Old untagged +scope frames refuse; there is no compatibility reader. Both directions bind full +history, member, node, disk, process, schema, sequence, payload digest and TLS +exporter through the existing authenticated frame machinery. The temporary +`TransportRuntime::Legacy` and typed cell-history frame arm support the old host +only during integration; CG-16 must remove them with that host. They are not a +configuration switch or an alternate final serving architecture. + +The transport retains its exclusive runtime claim until physical workers and +sockets exit, including cancelled close waiters. A replacement may start only +after that join. Complete-group snapshot streaming, shared query RPC routing and +production controller ownership of apply/command settlement remain unfinished. +The TLS fixtures still drive runtime events themselves; they do not qualify +controller ownership after an ingress caller disconnects. + +### Managed complete-group peer seeding + +`RaftTransport::prepare_managed_group_seed` requires the destination's rooted +FleetHostAuthorization and a fresh active physical learner reservation. It uses +the existing complete-group TLS seed exchange, receiver pools and original +invocation deadline. The authenticated source checkpoint must carry the exact +consumed permit in Seeding phase and the same repair plan. The source must be a +catalog-selected seed owner with the exact current disk/process boot and a voter +in the authenticated checkpoint. Revalidation cannot regress the catalog revision. +Catalog-read and handshake time are deducted before transmitting the remaining +source timeout. Catalog preflight, staging and joined-permit delivery share the +original transport deadline and process/shutdown cancellation. Queued engine +work retains its grant independently of a timed-out metadata waiter. +Unmanaged seed calls +refuse managed source metadata and managed namespace ownership. + +Staging retains the permanent member/node/disk/engine claim. The staged learner +is absent from the birth configuration and has no startup or serving authority. +Its base preserves the source's actual enrollment origin: a guard installed at +genesis remains a genesis guard; an imported group retains its later committed +enrollment through ordinary snapshot successor validation. Neither path can +retrofit a different origin or drop an installed fleet guard. + +The seed grant carries the exact local boot. The serialized engine worker checks +its last synced lifecycle guard when staging and immediately before the install +fence. Quiescence, stop acceptance and fencing refuse further installation. +Actual accepted engine work keeps its owner through sync; late/closed delivery +cannot erase an already persisted install receipt or report late startup success. + +`RangeDirectory::bootstrap_managed_joined_group_member` requires that durable +receipt, its permanent engine claim, the current rooted Live group and the exact +running local boot. This supports reconciliation after a lost install response +and restart after repair completion without replaying a saved active permit. +The returned permit repeats these checks before attachment. Fresh quorum and +application activation remain separate requirements. SDK/fleet startup, S3-based +managed learner recovery and native/backend qualification remain tracked in the +[cell-group progress record](../../../../docs/design/multi-raft/cell-groups/progress.md). + +Canonical startup prepares every selected local group root before publishing +physical routes. An explicit startup error joins controller/transport/SQL/runtime +and engine cleanup before returning, including actual storage-lock release. +Cancellation schedules that same cleanup on the owning execution domain. The +predecessor per-cell dispatcher is no longer compiled. Its used option/observation +contracts and enabled storage-policy regressions remain while the final legacy +verification sweep is incomplete. + +Old inbound per-cell seed and per-cell inspection RPC dispatch is removed. New +shared checkpoint seeding and physical-node/group inspection keep their existing +deadline, process, shutdown and bounded socket helpers. Retained outbound test +utilities do not provide an inbound compatibility reader. The library compiles; +unit/native and complete managed startup qualification are still in progress. diff --git a/crates/cellar/durability/raft/archive/coverage.rs b/crates/cellar/durability/raft/archive/coverage.rs deleted file mode 100644 index 8c5f7d909..000000000 --- a/crates/cellar/durability/raft/archive/coverage.rs +++ /dev/null @@ -1,239 +0,0 @@ -// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. - -//! Physical verification of one exact immutable archive and fixed publication. -use super::*; -use crate::raft::host::{FleetImageState, FleetPublishedCut}; -use crate::raft::member_state::FleetGroupBinding; -use cellar_logic::consensus::LogIndex; -use std::time::Duration; - -/// Historical, physically replayed coverage of one selected immutable archive. -/// This supplies no current health, quorum, placement or removal authority. -/// A lifecycle owner must obtain fresh survivor observations and recheck its -/// rooted operation before committing a terminal receipt or releasing resources. -#[derive(Clone, Debug)] -pub struct VerifiedArchiveCoverage { - reference: ArchiveReference, - binding: FleetGroupBinding, - birth: [u8; 32], - schema: [u8; 32], - revision: LogIndex, - position: LogPosition, - image_position: Option, - image: Option, -} - -impl VerifiedArchiveCoverage { - fn capture(set: &ArchiveRecoverySet) -> Result { - let membership = set.membership(); - let status = membership - .fleet_status() - .ok_or(ArchiveError::Invalid("archive has no fleet enrollment"))?; - let published = set.published_image()?; - let image = published.as_ref().map(|image| image.state()); - // A full snapshot's applied position can include configuration-only - // progress. Preserve the producer's exact publication origin, including - // legitimate intact-restore aliases; never infer it from that snapshot. - let image_position = published.as_ref().map(|image| image.position()); - Ok(Self { - reference: set.reference(), - binding: status.binding, - birth: set.birth_digest(), - schema: set.schema_digest(), - revision: membership.revision(), - position: set.position(), - image_position, - image, - }) - } - - pub fn reference(&self) -> &ArchiveReference { - &self.reference - } - - pub fn binding(&self) -> &FleetGroupBinding { - &self.binding - } - - pub fn birth_digest(&self) -> [u8; 32] { - self.birth - } - - pub fn schema_digest(&self) -> [u8; 32] { - self.schema - } - - pub fn membership_revision(&self) -> LogIndex { - self.revision - } - - pub fn position(&self) -> LogPosition { - self.position - } - - /// None denotes a physically verified canonical-empty archive, never a - /// missing payload or a pending SQL publication. - pub fn image_state(&self) -> Option { - self.image - } - - pub fn image_position(&self) -> Option { - self.image_position - } - - /// Compare complete prefix coverage at the same exact fleet/history and - /// membership revision. A configuration change requires a new attempt. - pub fn covers(&self, minimum: &FleetPublishedCut) -> Result { - if &self.binding != minimum.binding() - || self.birth != minimum.birth_digest() - || self.schema != minimum.schema_digest() - || self.revision != minimum.membership_revision() - { - return Err(ArchiveError::Conflict); - } - if self - .image - .as_ref() - .zip(minimum.image()) - .is_some_and(|(a, b)| !a.same_representation(&b.state())) - { - return Err(ArchiveError::Corrupt("archive image commitment domain")); - } - if self.position.index < minimum.applied().index { - return Ok(false); - } - if !consistent(self.position, minimum.applied()) { - return Err(ArchiveError::Corrupt("archive publication prefix term")); - } - match (self.image, minimum.image()) { - (None, None) => Ok(true), - (None, Some(_)) => Err(ArchiveError::Corrupt("archive erased published image")), - (Some(state), prior) => { - let position = self - .image_position - .ok_or(ArchiveError::Corrupt("archive image position"))?; - if !consistent(position, minimum.applied()) - || prior.is_some_and(|image| !consistent(position, image.position())) - { - return Err(ArchiveError::Corrupt("archive data prefix term")); - } - if prior.is_some_and(|image| state == image.state()) { - return Ok(true); - } - if position.index <= minimum.applied().index { - return Err(ArchiveError::Corrupt("archive conflicting published image")); - } - Ok(true) - } - } - } -} - -fn consistent(a: LogPosition, b: LogPosition) -> bool { - match a.index.cmp(&b.index) { - std::cmp::Ordering::Equal => a.term == b.term, - std::cmp::Ordering::Less => a.term <= b.term, - std::cmp::Ordering::Greater => a.term >= b.term, - } -} - -impl CellArchive { - /// Download and physically replay exactly `reference`, then prove coverage - /// of a fixed quorum-backed publication cut. A missing selected object - /// refuses; this operation never substitutes a newer mutable head. - /// - /// The original deadline covers every object read, workspace wait and - /// replay. Blocking work retains its inputs until joined; expiry/cancellation - /// suppresses the proof even if that work finishes later. The caller owns - /// scratch storage and admission for the configured recovery-memory bound. - pub async fn verify_published_at( - &self, - reference: &ArchiveReference, - minimum: &FleetPublishedCut, - verifier: &ArchiveVerifier, - timeout: Duration, - ) -> Result { - let millis = u64::try_from(timeout.as_millis()).map_err(|_| ArchiveError::Limit)?; - if !(1..=60_000).contains(&millis) || timeout > Duration::from_secs(60) { - return Err(ArchiveError::Invalid("archive verification timeout")); - } - let mut archive = self.clone(); - let now = archive - .io_domain - .as_ref() - .ok_or(ArchiveError::Stopped)? - .mono_ms() - .map_err(|_| ArchiveError::Stopped)?; - let deadline = now.checked_add(millis).ok_or(ArchiveError::Limit)?; - archive.verification_deadline = Some( - archive - .verification_deadline - .map_or(deadline, |previous| previous.min(deadline)), - ); - archive.check_verification()?; - let metadata = archive.load_metadata(&reference.0).await?; - let lease = if matches!(&metadata.inner, super::representation::Metadata::Rooted(_)) { - let lease = verifier.begin_extent()?; - archive.root_store = Some(lease.store()); - archive.root_verification = Some(lease.clone()); - if lease.needs_recovery() { - let worker = archive.clone(); - let owner = lease.clone(); - archive - .host - .run_blocking(move || owner.reconcile(|| worker.check_verification())) - .await - .map_err(ArchiveError::Worker)??; - } - Some(lease) - } else { - None - }; - let set = archive.load_recovery_material(metadata).await?; - let worker = archive.clone(); - let verifier = verifier.clone(); - let minimum = minimum.clone(); - let proof = archive - .host - .run_blocking(move || { - worker.check_verification()?; - let proof = VerifiedArchiveCoverage::capture(&set)?; - let coverage = proof.covers(&minimum); - match set.inner { - super::representation::Recovery::Materialized(set) => { - if !coverage? { - return Err(ArchiveError::Unconfirmed); - } - verifier.verify_checked(&set, || worker.check_verification())?; - } - super::representation::Recovery::Rooted(set) => { - lease - .ok_or(ArchiveError::Invalid("extent verifier lease required"))? - .finish(set, || worker.check_verification())?; - if !coverage? { - return Err(ArchiveError::Unconfirmed); - } - } - } - worker.check_verification()?; - Ok(proof) - }) - .await - .map_err(ArchiveError::Worker)??; - // Delivery can be delayed after the worker completed, including by an - // unpolled caller. A completed replay is no exception to this deadline. - archive.check_verification()?; - Ok(proof) - } - - fn check_verification(&self) -> Result<(), ArchiveError> { - if self - .cancellation - .as_ref() - .is_some_and(|cancel| *cancel.borrow() || cancel.has_changed().is_err()) - { - return Err(ArchiveError::Stopped); - } - self.deadline().map(|_| ()) - } -} diff --git a/crates/cellar/durability/raft/archive/mod.rs b/crates/cellar/durability/raft/archive/mod.rs index 543482c15..2a296c68e 100644 --- a/crates/cellar/durability/raft/archive/mod.rs +++ b/crates/cellar/durability/raft/archive/mod.rs @@ -23,7 +23,6 @@ use std::{collections::BTreeMap, fmt, sync::Arc}; pub(crate) mod bundle; mod configuration; -mod coverage; mod physical; mod publication; mod replay; @@ -41,8 +40,6 @@ pub use retention::{ pub use rooted::{RootArchiveReference, RootArchiveSnapshot}; pub use verifier::ArchiveVerifier; -pub use coverage::VerifiedArchiveCoverage; - const READ_WINDOW: usize = 4; pub(crate) const fn physical_read_bytes() -> usize { diff --git a/crates/cellar/durability/raft/budget.rs b/crates/cellar/durability/raft/budget.rs index 9b158ab34..a750b8971 100644 --- a/crates/cellar/durability/raft/budget.rs +++ b/crates/cellar/durability/raft/budget.rs @@ -2,13 +2,17 @@ //! Byte and item admission retained through queued work and event delivery. -use std::sync::atomic::{AtomicUsize, Ordering}; -use std::sync::Arc; +use std::sync::{Arc, Mutex, MutexGuard}; + +#[derive(Default)] +struct Usage { + bytes: usize, + items: usize, +} pub(crate) struct Budget { pub wake: tokio::sync::Notify, - bytes: AtomicUsize, - items: AtomicUsize, + usage: Mutex, max_bytes: usize, max_items: usize, } @@ -19,38 +23,49 @@ impl Budget { } pub(crate) fn usage(&self) -> (usize, usize) { - ( - self.bytes.load(Ordering::Acquire), - self.items.load(Ordering::Acquire), - ) + let usage = self.lock(); + (usage.bytes, usage.items) + } + + fn lock(&self) -> MutexGuard<'_, Usage> { + self.usage.lock().unwrap_or_else(|error| error.into_inner()) + } + + fn released(&self) { + // Requests have different byte sizes. A waiter that still cannot fit + // must not consume another request's only opportunity to make progress. + self.wake.notify_waiters(); + // Workers also inspect capacity before registering their select arm. + // Keep a stored signal when release precedes that registration. + self.wake.notify_one(); } pub fn new(max_bytes: usize, max_items: usize) -> Arc { Arc::new(Self { wake: tokio::sync::Notify::new(), - bytes: AtomicUsize::new(0), - items: AtomicUsize::new(0), + usage: Mutex::new(Usage::default()), max_bytes, max_items, }) } pub fn acquire(self: &Arc, bytes: usize) -> Option { - self.items - .fetch_update(Ordering::AcqRel, Ordering::Acquire, |n| { - n.checked_add(1).filter(|n| *n <= self.max_items) - }) - .ok()?; - if self + // Reserve both dimensions together. A rejected byte request must never + // temporarily exhaust the item count seen by a fitting caller. + let mut usage = self.lock(); + let next_bytes = usage .bytes - .fetch_update(Ordering::AcqRel, Ordering::Acquire, |n| { - n.checked_add(bytes).filter(|n| *n <= self.max_bytes) - }) - .is_err() - { - self.items.fetch_sub(1, Ordering::AcqRel); - return None; - } + .checked_add(bytes) + .filter(|n| *n <= self.max_bytes)?; + let next_items = usage + .items + .checked_add(1) + .filter(|n| *n <= self.max_items)?; + *usage = Usage { + bytes: next_bytes, + items: next_items, + }; + drop(usage); Some(Permit { owner: self.clone(), bytes, @@ -63,10 +78,176 @@ pub(crate) struct Permit { bytes: usize, } +impl Permit { + /// Ensure an admitted read can retain its payload before allocating it. + /// Repeated or smaller requests keep the same bytes and item slot. + pub(crate) fn grow_at_least(&mut self, bytes: usize) -> bool { + bytes <= self.bytes || self.grow_to(bytes) + } + + /// Extend an admitted operation without consuming another item slot. + /// Failure leaves its existing reservation unchanged. + pub(crate) fn grow_to(&mut self, bytes: usize) -> bool { + let Some(extra) = bytes.checked_sub(self.bytes) else { + return false; + }; + let mut usage = self.owner.lock(); + let Some(next) = usage + .bytes + .checked_add(extra) + .filter(|n| *n <= self.owner.max_bytes) + else { + return false; + }; + usage.bytes = next; + self.bytes = bytes; + true + } + + /// Refund transient capture capacity while keeping the operation slot and + /// the retained payload charged. A reservation can never grow here. + pub(crate) fn shrink_to(&mut self, bytes: usize) -> bool { + if bytes > self.bytes { + return false; + } + let released = self.bytes - bytes; + let mut usage = self.owner.lock(); + usage.bytes -= released; + self.bytes = bytes; + drop(usage); + if released != 0 { + self.owner.released(); + } + true + } +} + impl Drop for Permit { fn drop(&mut self) { - self.owner.bytes.fetch_sub(self.bytes, Ordering::AcqRel); - self.owner.items.fetch_sub(1, Ordering::AcqRel); - self.owner.wake.notify_one(); + let mut usage = self.owner.lock(); + usage.bytes -= self.bytes; + usage.items -= 1; + drop(usage); + self.owner.released(); + } +} + +#[cfg(test)] +mod tests { + use super::*; + use futures_util::FutureExt; + + fn fitting_waiter_after_partial_release(shrink: bool) { + let budget = Budget::new(100, 3); + let mut held = budget.acquire(80).unwrap(); + let mut other = Some(budget.acquire(20).unwrap()); + let mut large = std::pin::pin!(budget.wake.notified()); + large.as_mut().enable(); + let mut small = std::pin::pin!(budget.wake.notified()); + small.as_mut().enable(); + assert!(budget.acquire(40).is_none()); + assert!(budget.acquire(20).is_none()); + if shrink { + assert!(held.shrink_to(60)); + } else { + drop(other.take()); + } + assert_eq!(budget.usage().0, 80); + // The first registered request still cannot use the released bytes. + // A fitting request must be woken without another capacity change. + assert!(large.as_mut().now_or_never().is_some()); + assert!(budget.acquire(40).is_none()); + assert!( + small.as_mut().now_or_never().is_some(), + "a non-fitting waiter must not consume the fitting request's only wake" + ); + let admitted = budget.acquire(20).unwrap(); + assert_eq!(budget.usage().0, 100); + drop(admitted); + drop(other); + drop(held); + assert_eq!(budget.usage(), (0, 0)); + } + + #[test] + fn release_wakes_fitting_waiter_behind_a_larger_request() { + fitting_waiter_after_partial_release(false); + } + + #[test] + fn shrink_wakes_fitting_waiter_behind_a_larger_request() { + fitting_waiter_after_partial_release(true); + } + + #[test] + fn late_worker_observes_release_without_prior_registration() { + let budget = Budget::new(100, 1); + let held = budget.acquire(100).unwrap(); + drop(held); + assert!(budget.wake.notified().now_or_never().is_some()); + assert_eq!(budget.usage(), (0, 0)); + } + + #[test] + fn nonfitting_attempts_do_not_consume_available_item_capacity() { + let budget = Budget::new(100, 2); + let held = budget.acquire(80).unwrap(); + let start = std::sync::Barrier::new(2); + std::thread::scope(|scope| { + let worker = scope.spawn(|| { + start.wait(); + for _ in 0..100_000 { + assert!(budget.acquire(30).is_none()); + } + }); + start.wait(); + for _ in 0..100_000 { + let fitting = budget.acquire(20); + assert!( + fitting.is_some(), + "a rejected request must not charge an item against a fitting caller" + ); + drop(fitting); + } + worker.join().unwrap(); + }); + drop(held); + assert_eq!(budget.usage(), (0, 0)); + } + + #[test] + fn grow_preserves_item_and_refuses_without_changing_accounting() { + let budget = Budget::new(100, 2); + let mut pin = budget.acquire(10).unwrap(); + let other = budget.acquire(70).unwrap(); + assert!(pin.grow_to(30)); + assert_eq!(budget.usage(), (100, 2)); + assert!(!pin.grow_to(31)); + assert!(!pin.grow_to(29)); + assert!(pin.grow_at_least(29)); + assert!(pin.grow_at_least(30)); + assert!(!pin.grow_at_least(31)); + assert_eq!(budget.usage(), (100, 2)); + drop(other); + assert!(pin.grow_to(100)); + drop(pin); + assert_eq!(budget.usage(), (0, 0)); + } + + #[test] + fn shrink_refunds_bytes_but_keeps_the_operation_slot() { + let budget = Budget::new(100, 2); + let mut capture = budget.acquire(100).unwrap(); + assert!(budget.acquire(1).is_none()); + assert!(capture.shrink_to(20)); + let next = budget.acquire(80).unwrap(); + assert_eq!(budget.usage(), (100, 2)); + assert!(budget.acquire(0).is_none()); + assert!(!capture.shrink_to(21)); + assert_eq!(budget.usage(), (100, 2)); + drop(capture); + assert_eq!(budget.usage(), (80, 1)); + drop(next); + assert_eq!(budget.usage(), (0, 0)); } } diff --git a/crates/cellar/durability/raft/cache.rs b/crates/cellar/durability/raft/cache.rs index 711490abe..50a20a942 100644 --- a/crates/cellar/durability/raft/cache.rs +++ b/crates/cellar/durability/raft/cache.rs @@ -2,12 +2,14 @@ //! Durable storage views. Raft reads these bounded caches without disk I/O. +use super::cell_groups::{GroupManifest, ManifestPins, PreparedGroupManifest}; use super::member_state::GroupMembership; use super::storage::LogError; use crate::application::consensus::ImageCommitment; use ::raft::prelude::{ConfState, Entry, EntryType, HardState, Snapshot}; use ::raft::{GetEntriesContext, RaftState, Storage, StorageError}; -use cellar_logic::consensus::{LogIndex, LogPosition, ReplicaIdentity, Term}; +use cellar_logic::cell_group::GroupHistory; +use cellar_logic::consensus::{GroupScope, LogIndex, LogPosition, ReplicaIdentity, Term}; use protobuf::Message as _; use serde::{Deserialize, Serialize}; use std::collections::BTreeMap; @@ -89,6 +91,11 @@ pub struct DurableApply { pub enum SnapshotChange { /// A verified local snapshot; retain entries beyond its included index. Checkpoint(Snapshot), + /// A complete receiver-local cell-group closure. Its producer supplies no + /// Raft authority; the consensus owner must bind this to its snapshot effect. + CellGroup(super::cell_groups::PreparedCellGroupCheckpoint), + /// A captured exact local applied cut; retains all later entries and state. + CellGroupCheckpoint(super::cell_groups::PreparedLocalCellGroupCheckpoint), /// A verified incoming snapshot and its already durable state image. Install { snapshot: Snapshot, @@ -97,29 +104,56 @@ pub enum SnapshotChange { } #[derive(Clone)] -pub struct GroupWrite { - pub group: GroupLog, +pub struct GroupWrite { + pub(crate) log_reservations: + BTreeMap>, + pub(crate) require_archive: + Option>, + pub(crate) checkpoint_admission: + Option>, + pub(crate) consensus_pin: Option>, + pub(crate) seed_grant: Option>, + pub(crate) local_park: Option, + pub group: GroupLog, pub entries: Vec, pub hard_state: Option, pub conf_state: Option, pub membership: Option, pub applied: Option, + pub group_manifest: Option, pub snapshot: Option, } -impl GroupWrite { - pub fn new(group: &GroupLog) -> Self { +impl GroupWrite { + pub fn new(group: &GroupLog) -> Self { Self { + log_reservations: BTreeMap::new(), + require_archive: None, + checkpoint_admission: None, + consensus_pin: None, + seed_grant: None, + local_park: None, group: group.clone(), entries: Vec::new(), hard_state: None, conf_state: None, membership: None, applied: None, + group_manifest: None, snapshot: None, } } + pub(crate) fn prepared_group_manifest(&self) -> Option<&PreparedGroupManifest> { + self.group_manifest + .as_ref() + .or_else(|| match &self.snapshot { + Some(SnapshotChange::CellGroup(prepared)) => Some(prepared.root()), + Some(SnapshotChange::CellGroupCheckpoint(prepared)) => Some(prepared.root()), + _ => None, + }) + } + pub(crate) fn estimated_bytes(&self) -> Result { let mut bytes = 1024_usize; for entry in &self.entries { @@ -130,6 +164,8 @@ impl GroupWrite { if let Some(snapshot) = &self.snapshot { let snapshot = match snapshot { SnapshotChange::Checkpoint(s) | SnapshotChange::Install { snapshot: s, .. } => s, + SnapshotChange::CellGroup(prepared) => prepared.snapshot(), + SnapshotChange::CellGroupCheckpoint(prepared) => prepared.snapshot(), }; if snapshot.compute_size() > 64 * 1024 { return Err(LogError::Overloaded); @@ -151,25 +187,43 @@ impl GroupWrite { .checked_add(membership.reserved_bytes()) .ok_or(LogError::Overloaded)?; } + if let Some(manifest) = self.prepared_group_manifest() { + bytes = bytes + .checked_add(16 * 1024 + manifest.reserved_bytes()) + .ok_or(LogError::Overloaded)?; + } + use super::cell_groups::LocalParkChange; + if let Some(LocalParkChange::Certify(certificate) | LocalParkChange::Park(certificate)) = + &self.local_park + { + bytes = bytes + .checked_add(certificate.reserved_bytes()) + .ok_or(LogError::Overloaded)?; + } Ok(bytes) } } +/// Engine-owned storage handle. `GroupHistory` is the cell-group scope; +/// `GroupScope` remains only for the staged single-cell application adapter. +/// The engine rejects mixed application domains and never casts between them. #[derive(Clone)] -pub struct GroupLog { +pub struct GroupLog { + pub(crate) archive_compaction: Arc, + pub(crate) consensus_claimed: Arc, pub(crate) scratch_directory: std::path::PathBuf, pub(crate) snapshot_scratch: cellar_ltx::stream_snapshot::ScratchPool, pub(crate) filesystem: Arc, pub(crate) key: u64, pub(crate) birth_digest: [u8; 32], - pub(crate) identity: ReplicaIdentity, + pub(crate) identity: ReplicaIdentity, pub(crate) owner: Arc<()>, pub(crate) closed: Arc, pub(crate) cache: Arc>, pub(crate) cache_revision: Arc, } -impl std::fmt::Debug for GroupLog { +impl std::fmt::Debug for GroupLog { fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { formatter .debug_struct("GroupLog") @@ -179,8 +233,32 @@ impl std::fmt::Debug for GroupLog { } } -impl GroupLog { - pub fn identity(&self) -> &ReplicaIdentity { +impl GroupLog { + /// Only the physical engine close owner may detach derived cache pins. + /// Independent readers retain their own pins until their physical work exits. + fn release_closed_pins(&self) -> Result<(), LogError> { + if !self.closed.load(Ordering::Acquire) { + return Err(LogError::Invalid("live manifest-pin release".into())); + } + match self.cache.write() { + Ok(mut cache) => { + cache.manifest_pins.take(); + Ok(()) + } + Err(error) => { + error.into_inner().manifest_pins.take(); + Err(LogError::Corrupt("closed group cache poisoned".into())) + } + } + } + + pub(crate) fn log_retention( + &self, + ) -> Option<&Arc> { + self.archive_compaction.log_retention.as_ref() + } + + pub fn identity(&self) -> &ReplicaIdentity { &self.identity } @@ -189,7 +267,30 @@ impl GroupLog { } pub fn applied(&self) -> Result { - Ok(self.read()?.applied.clone()) + let cache = self.read()?; + if cache.group_manifest.is_some() { + return Err(LogError::Invalid( + "cell-group root requires complete-manifest recovery".into(), + )); + } + Ok(cache.applied.clone()) + } + + /// Immutable mapping metadata, shared by the engine cache. This observation + /// grants neither complete physical recovery nor startup authority. + pub(crate) fn recovery_genesis(&self) -> Result>, LogError> { + Ok(self.read()?.recovery_genesis.clone()) + } + + /// Engine-selected metadata only. Reverify every page/image before admission. + pub fn group_manifest(&self) -> Result, LogError> { + Ok(self.read()?.group_manifest.clone()) + } + + /// Metadata only; startup must recover this older closure as well as the + /// applied root before any collection or snapshot service can begin. + pub fn checkpoint_manifest(&self) -> Result, LogError> { + Ok(self.read()?.checkpoint_manifest.clone()) } /// Last synced bindings for this group history. This observation alone is @@ -222,6 +323,48 @@ impl GroupLog { } } +/// One engine catalog, with explicit staged application domains. A caller must +/// select the right handle type; enumeration never silently skips another kind. +#[derive(Clone)] +pub(crate) enum StoredGroup { + Cell(GroupLog), + Cells(GroupLog), +} + +impl StoredGroup { + pub(crate) fn release_closed_pins(&self) -> Result<(), LogError> { + match self { + Self::Cell(group) => group.release_closed_pins(), + Self::Cells(group) => group.release_closed_pins(), + } + } + + pub(crate) fn as_cell(&self) -> Result<&GroupLog, LogError> { + match self { + Self::Cell(group) => Ok(group), + Self::Cells(_) => Err(LogError::WrongIdentity), + } + } + + pub(crate) fn as_cells(&self) -> Result<&GroupLog, LogError> { + match self { + Self::Cells(group) => Ok(group), + Self::Cell(_) => Err(LogError::WrongIdentity), + } + } + + pub(crate) fn history(&self) -> GroupHistory { + match self { + Self::Cell(group) => GroupHistory::from(&group.identity.scope), + Self::Cells(group) => group.identity.scope, + } + } + + pub(crate) fn is_cells(&self) -> bool { + matches!(self, Self::Cells(_)) + } +} + fn raft_error(error: LogError) -> ::raft::Error { ::raft::Error::Store(StorageError::Other(Box::new(error))) } @@ -243,7 +386,7 @@ impl ConsensusStorage { } } -impl GroupLog { +impl GroupLog { pub(crate) fn consensus_storage(&self) -> ConsensusStorage { ConsensusStorage { cache: self.cache.clone(), @@ -257,7 +400,7 @@ impl GroupLog { } } -impl Storage for GroupLog { +impl Storage for GroupLog { fn initial_state(&self) -> ::raft::Result { self.checked_storage()?.initial_state() } @@ -356,12 +499,21 @@ impl Storage for ConsensusStorage { #[derive(Clone)] pub(crate) struct CachedGroup { + pub archive_required: bool, pub hard_state: HardState, pub conf_state: ConfState, pub membership: GroupMembership, pub snapshot: Snapshot, pub entries: BTreeMap, pub applied: DurableApply, + pub group_manifest: Option, + /// Derived from the immutable mapping; never supplied by a write/snapshot. + pub recovery_genesis: Option>, + pub manifest_pins: Option, + pub checkpoint_manifest: Option, + /// Local receipt; never imported from another node's checkpoint. + pub group_seed: Option, + pub(crate) local_park: Option, } impl CachedGroup { @@ -400,7 +552,19 @@ impl CachedGroup { } pub fn bytes(&self) -> usize { - 1024 + self.membership.reserved_bytes() + 1024 + self.group_manifest.as_ref().map_or(0, |m| { + 16 * 1024 + + self.manifest_pins.as_ref().map_or( + ManifestPins::maximum_bytes(m.root().cells()), + ManifestPins::reserved_bytes, + ) + }) + self.checkpoint_manifest.as_ref().map_or(0, |m| { + 16 * 1024 + ManifestPins::maximum_bytes(m.root().cells()) + }) + self.membership.reserved_bytes() + + self + .local_park + .as_ref() + .map_or(0, |park| park.reserved_bytes()) + self.snapshot.compute_size() as usize + self.conf_state.compute_size() as usize + self @@ -411,6 +575,17 @@ impl CachedGroup { } pub fn validate(&self, max_entries: usize) -> Result<(), LogError> { + self.validate_local_park()?; + if self.archive_required && self.group_manifest.is_none() { + return Err(LogError::Corrupt( + "archive requirement without cell group".into(), + )); + } + if self.group_seed.is_some_and(|index| { + index <= 1 || index > self.applied.position.index.get() || self.group_manifest.is_none() + }) { + return Err(LogError::Corrupt("invalid local group seed receipt".into())); + } validate_conf(&self.conf_state)?; self.membership .validate(&self.conf_state, self.applied.position.index.get()) @@ -449,14 +624,87 @@ impl CachedGroup { if let Some(image) = &self.applied.image { image.validate()?; } + if let Some(manifest) = &self.group_manifest { + if super::cell_groups::checkpoint::persisted_marker( + &self.snapshot, + manifest, + &self.membership, + )? != self.checkpoint_manifest + { + return Err(LogError::Corrupt("group checkpoint selection".into())); + } + manifest + .check() + .map_err(|e| LogError::Corrupt(e.to_string()))?; + if self.applied.image.is_some() + || manifest.position() != self.applied.position + || manifest.membership_revision() != self.membership.revision().get() + || self + .manifest_pins + .as_ref() + .is_some_and(|pins| pins.root() != manifest.root()) + { + return Err(LogError::Corrupt("incoherent cell-group root".into())); + } + } else if self.manifest_pins.is_some() || self.checkpoint_manifest.is_some() { + return Err(LogError::Corrupt("image pins without group root".into())); + } Ok(()) } - pub fn changed(&self, write: &GroupWrite, max_entries: usize) -> Result { + pub fn changed( + &self, + write: &GroupWrite, + max_entries: usize, + history: GroupHistory, + ) -> Result { let mut next = self.clone(); + if write.require_archive.is_some() { + if self.group_manifest.is_none() { + return Err(LogError::WrongIdentity); + } + next.archive_required = true; + } + if let Some(grant) = &write.seed_grant { + grant.check(&write.group.owner, history, write.group.identity())?; + let Some(SnapshotChange::CellGroup(prepared)) = &write.snapshot else { + return Err(LogError::WrongIdentity); + }; + if prepared.snapshot() != grant.snapshot() || !write.entries.is_empty() { + return Err(LogError::WrongIdentity); + } + next.group_seed = Some(prepared.marker().position().index.get()); + } + let group_install = matches!(&write.snapshot, Some(SnapshotChange::CellGroup(_))); + let group_checkpoint = matches!( + &write.snapshot, + Some(SnapshotChange::CellGroupCheckpoint(_)) + ); + if group_checkpoint && (!write.entries.is_empty() || write.hard_state.is_some()) { + return Err(LogError::Invalid("mixed local checkpoint write".into())); + } + if (group_install || group_checkpoint) + && (self.group_manifest.is_none() + || write.group_manifest.is_some() + || write.applied.is_some() + || write.conf_state.is_some() + || write.membership.is_some()) + { + return Err(LogError::Invalid("mixed cell-group snapshot write".into())); + } + if (self.group_manifest.is_some() || write.group_manifest.is_some()) + && (write.applied.is_some() + || (write.snapshot.is_some() && !group_install && !group_checkpoint)) + { + return Err(LogError::Invalid( + "mixed image/group apply or snapshot".into(), + )); + } if let Some(change) = &write.snapshot { let snapshot = match change { SnapshotChange::Checkpoint(s) | SnapshotChange::Install { snapshot: s, .. } => s, + SnapshotChange::CellGroup(prepared) => prepared.snapshot(), + SnapshotChange::CellGroupCheckpoint(prepared) => prepared.snapshot(), }; let metadata = snapshot.get_metadata(); if metadata.index < self.snapshot.get_metadata().index @@ -467,6 +715,76 @@ impl CachedGroup { return Err(LogError::Invalid("conflicting or stale snapshot".into())); } match change { + SnapshotChange::CellGroup(prepared) => { + let root = prepared.root(); + let marker = root.marker(); + if !Arc::ptr_eq(&root.engine_owner, &write.group.owner) + || !marker.matches_group(&history, write.group.birth_digest) + || marker.route() != self.group_manifest.as_ref().unwrap().route() + || (root.base.as_ref() != self.group_manifest.as_ref() + && Some(marker) != self.group_manifest.as_ref()) + || metadata.index < self.hard_state.commit + || marker.position().index.get() != metadata.index + || marker.position().term.get() != metadata.term + || (marker.position().index == self.applied.position.index + && Some(marker) != self.group_manifest.as_ref()) + { + return Err(LogError::Invalid("cell-group snapshot binding/base".into())); + } + let previous = self.group_manifest.as_ref().unwrap(); + if !marker.preserves_recovery(previous) + && !(previous.recovery_request().is_none() + && previous.root().cells() == 0 + && previous.position().index.get() == 1 + && write.seed_grant.is_some()) + { + return Err(LogError::Invalid( + "snapshot recovery lineage changed".into(), + )); + } + if !marker.preserves_transfer(previous) { + return Err(LogError::Invalid("snapshot child transfer changed".into())); + } + if !marker.preserves_source_seal(previous) { + return Err(LogError::Invalid("snapshot source seal changed".into())); + } + if !marker.preserves_park_cut(previous) { + return Err(LogError::Invalid("snapshot park cut changed".into())); + } + prepared.membership().succeeds(&self.membership)?; + next.entries.clear(); + next.applied = DurableApply { + position: marker.position(), + image: None, + }; + next.group_manifest = Some(marker.clone()); + next.manifest_pins = Some(root.images.clone()); + next.checkpoint_manifest = Some(marker.clone()); + next.conf_state = metadata.get_conf_state().clone(); + next.membership = prepared.membership().clone(); + } + SnapshotChange::CellGroupCheckpoint(prepared) => { + let root = prepared.root(); + if !Arc::ptr_eq(&root.engine_owner, &write.group.owner) + || !root + .marker() + .matches_group(&history, write.group.birth_digest) + || self.group_manifest.as_ref() != Some(root.marker()) + || root.marker().position() != self.applied.position + || self.term(metadata.index) != Some(metadata.term) + || metadata.index != self.applied.position.index.get() + || metadata.index > self.hard_state.commit + || metadata.get_conf_state() != &self.conf_state + || prepared + .membership() + .map_err(|e| LogError::Invalid(e.to_string()))? + != self.membership + { + return Err(LogError::Invalid("local checkpoint binding/cut".into())); + } + next.entries = next.entries.split_off(&(metadata.index + 1)); + next.checkpoint_manifest = Some(root.marker().clone()); + } SnapshotChange::Checkpoint(_) => { if metadata.index > self.applied.position.index.get() || self.term(metadata.index) != Some(metadata.term) @@ -552,6 +870,101 @@ impl CachedGroup { } next.applied = applied.clone(); } + if let Some(prepared) = &write.group_manifest { + let marker = prepared.marker(); + if let Some(base) = &prepared.base { + let fresh = self.group_manifest.is_none() + && base.root().cells() == 0 + && base.position() == self.applied.position + && self.applied.position.index.get() == 1; + if self.group_manifest.as_ref() != Some(marker) + && self.group_manifest.as_ref() != Some(base) + && !fresh + { + return Err(LogError::Invalid( + "cell-group preparation base changed".into(), + )); + } + } + marker + .check() + .map_err(|e| LogError::Invalid(e.to_string()))?; + if !Arc::ptr_eq(&prepared.engine_owner, &write.group.owner) + || !marker.matches_group(&history, write.group.birth_digest) + || marker.position().index < self.applied.position.index + || (marker.position().index == self.applied.position.index + && self.group_manifest.as_ref() != Some(marker)) + || (marker.position().index > self.applied.position.index + && self.applied.position.index.get().checked_add(1) + != Some(marker.position().index.get())) + || self.applied.image.is_some() + { + return Err(LogError::Invalid( + "invalid cell-group apply binding or prefix".into(), + )); + } + if let Some(previous) = &self.group_manifest { + if !marker.preserves_recovery(previous) { + return Err(LogError::Invalid("applied recovery lineage changed".into())); + } + if !marker.preserves_transfer(previous) { + return Err(LogError::Invalid("applied child transfer changed".into())); + } + if !marker.preserves_source_seal(previous) { + return Err(LogError::Invalid("applied source seal changed".into())); + } + if !marker.preserves_park_cut(previous) { + return Err(LogError::Invalid("applied park cut changed".into())); + } + if marker.route() != previous.route() { + return Err(LogError::Invalid( + "group route changed outside topology protocol".into(), + )); + } + } else if self.applied.position.index.get() != 1 + || self.snapshot.get_metadata().index != 1 + || !self.snapshot.data.is_empty() + { + return Err(LogError::Invalid( + "cell-group state requires a fresh group".into(), + )); + } + next.applied = DurableApply { + position: marker.position(), + image: None, + }; + next.group_manifest = Some(marker.clone()); + next.manifest_pins = Some(prepared.images.clone()); + } + if let (Some(previous), Some(conf)) = (&self.group_manifest, &write.conf_state) { + let prepared = write.group_manifest.as_ref().ok_or_else(|| { + LogError::Invalid("group configuration lacks complete root".into()) + })?; + let marker = prepared.marker(); + let entry = self + .entries + .get(&marker.position().index.get()) + .filter(|entry| entry.index <= self.hard_state.commit) + .ok_or_else(|| { + LogError::Invalid("group configuration lacks committed entry".into()) + })?; + let expected = super::cell_groups::membership::transition( + entry, + previous.route(), + &self.conf_state, + &self.membership, + )?; + if *conf != expected.conf + || write.membership.as_ref() != Some(&expected.membership) + || marker.root() != previous.root() + || marker.membership_revision() != entry.index + || marker.position().term.get() != entry.term + { + return Err(LogError::Invalid( + "group configuration/root transition".into(), + )); + } + } if let Some(conf) = &write.conf_state { if next.applied.position.index <= self.applied.position.index && conf != &self.conf_state @@ -573,7 +986,7 @@ impl CachedGroup { membership.succeeds(&self.membership)?; next.membership = membership.clone(); } - if (next.conf_state != self.conf_state && write.membership.is_none()) + if (next.conf_state != self.conf_state && write.membership.is_none() && !group_install) || (write.conf_state.is_some() && (write.membership.is_none() || next.membership.revision() != next.applied.position.index)) @@ -593,8 +1006,12 @@ impl CachedGroup { "applied configuration lacks ConfState".into(), )); } - next.membership - .validate_scope(&write.group.identity().scope)?; + next.update_local_park( + self, + write.local_park.as_ref(), + write.group.identity().member, + write.group.identity().node_identity(), + )?; next.validate(max_entries) .map_err(|error| LogError::Invalid(error.to_string()))?; Ok(next) @@ -647,11 +1064,18 @@ pub(crate) fn genesis(conf_state: ConfState) -> CachedGroup { ..Default::default() }; CachedGroup { + archive_required: false, hard_state, membership: GroupMembership::initial(&conf_state), conf_state, snapshot, entries: BTreeMap::new(), + group_manifest: None, + recovery_genesis: None, + manifest_pins: None, + checkpoint_manifest: None, + group_seed: None, + local_park: None, applied: DurableApply { position: LogPosition { index: LogIndex::new(1), @@ -661,3 +1085,13 @@ pub(crate) fn genesis(conf_state: ConfState) -> CachedGroup { }, } } + +impl GroupLog { + /// Advisory logical entry debt and owned reservations; not physical file + /// usage, archive coverage or permission to admit a mutation. + pub fn log_retention_usage( + &self, + ) -> Result { + self.log_retention().ok_or(LogError::WrongIdentity)?.usage() + } +} diff --git a/crates/cellar/durability/raft/catalog.rs b/crates/cellar/durability/raft/catalog.rs index 363d1c35a..75d11a41c 100644 --- a/crates/cellar/durability/raft/catalog.rs +++ b/crates/cellar/durability/raft/catalog.rs @@ -22,9 +22,34 @@ use std::{collections::BTreeSet, fmt, sync::Arc}; mod births; pub use births::BirthDisposition; +#[path = "cell_groups/directory/mod.rs"] +mod range_directory; +pub(crate) use range_directory::ManagedTransitionRecoverySelection; +pub(crate) use range_directory::RouteCache; +pub use range_directory::{ + CellGroupArchiveBindingError, CellGroupBirth, CellGroupBirthDisposition, CellGroupBirthRequest, + CellGroupBirthSpec, CellGroupBootstrapPermit, CellGroupDirectoryCursor, CellGroupDirectoryPage, + CellGroupDisasterFenceRequest, CellGroupRecoveryActivationAck, + CellGroupRecoveryActivationPermit, CellGroupRecoveryError, CellGroupRecoveryRequest, + CellGroupSealPermit, CellGroupSourceRetirementAck, CellGroupSourceRetirementPermit, + CellGroupTransferActivationAck, CellGroupTransferActivationPermit, CellGroupTransferArchive, + CellGroupTransitionPreparationInputs, CellGroupTransitionSource, CellRouteAttempt, + CellRouteCacheUsage, CellRouteOperation, CellRouteRequest, CellRoutingError, + CellRoutingFailure, CellRoutingLimits, CellRoutingSession, DirectoryLimits, DirectoryMutation, + DirectoryPublication, DirectoryView, FencedCellGroupRecovery, InstalledTransferVoter, + ManagedRecoverySeedQuorum, ManagedRecoveryVoterSeed, PreparedCellGroupArchiveBinding, + PreparedManagedCellGroupTransition, RangeDirectory, ResolvedCellRoute, TransferInstallQuorum, + TransitionJournal, TransitionPhase, VerifiedCellGroupTransitionSources, +}; +pub(crate) use range_directory::{ + ManagedRecoveryPublicationInput, RecoverySeedPermit, TransferSeedPermit, +}; + #[path = "catalog_inventory/mod.rs"] mod inventory; -pub(crate) use inventory::{BirthImportCut, ManagedBirthInventory}; +pub(crate) use inventory::groups::group_key as cell_group_import_key; +pub(crate) use inventory::groups::CellGroupImportCut; +pub use inventory::groups::FrozenCellGroupInventory; pub use inventory::{ BirthInventoryCursor, BirthInventoryEntry, BirthInventoryPage, FrozenBirthInventory, }; @@ -48,6 +73,8 @@ pub enum CatalogError { BirthCancelled, BirthsFrozen, ManagedAdmissionRequired, + /// Preserve the owning fleet's conflict, pressure and uncertain outcome. + FleetAuthority(Box), LostStableState, Unconfirmed, Storage(object_store::Error), @@ -65,7 +92,20 @@ impl fmt::Display for CatalogError { } } -impl std::error::Error for CatalogError {} +impl std::error::Error for CatalogError { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + match self { + Self::FleetAuthority(error) => Some(error.as_ref()), + _ => None, + } + } +} + +impl From for CatalogError { + fn from(error: super::fleet_catalog::FleetCatalogError) -> Self { + Self::FleetAuthority(Box::new(error)) + } +} #[derive(Clone, Debug)] pub struct CatalogOptions { @@ -288,32 +328,12 @@ impl Catalog { config: NamespaceConfig, create: bool, ) -> Result { - if config.schema_digest == [0; 32] { - return Err(CatalogError::Invalid("namespace schema digest")); - } - let data_prefix = match config.mode { - NamespaceMode::Raft => format!( - "{}raft/{}/{}/", - self.inner.prefix, - hex::encode(config.cluster.bytes()), - hex::encode(config.namespace.bytes()) - ), - NamespaceMode::Fleet => format!("{}cells/", self.inner.prefix), - }; - let expected = wire::Root::new(&config, data_prefix.clone()); - expected.validate()?; + let (namespace, expected, bytes) = self.namespace_candidate(config)?; let key = Path::from(format!( "{}namespaces/{}/root", self.inner.prefix, - hex::encode(config.namespace.bytes()) + hex::encode(namespace.config.namespace.bytes()) )); - let bytes = wire::encode(wire::ROOT, &expected, self.inner.limit)?; - let namespace = NamespaceCatalog { - inner: self.inner.clone(), - config, - root_digest: wire::digest(&bytes), - data_prefix, - }; let found = self.inner.read(&key).await?; let observed = match found { Some(bytes) => bytes, @@ -343,6 +363,43 @@ impl Catalog { } Ok(namespace) } + + /// Derive the exact descriptor before publishing any namespace objects. + pub(super) fn describe_namespace( + &self, + config: NamespaceConfig, + ) -> Result { + self.namespace_candidate(config) + .map(|(namespace, _, _)| namespace) + } + + fn namespace_candidate( + &self, + config: NamespaceConfig, + ) -> Result<(NamespaceCatalog, wire::Root, Vec), CatalogError> { + if config.schema_digest == [0; 32] { + return Err(CatalogError::Invalid("namespace schema digest")); + } + let data_prefix = match config.mode { + NamespaceMode::Raft => format!( + "{}raft/{}/{}/", + self.inner.prefix, + hex::encode(config.cluster.bytes()), + hex::encode(config.namespace.bytes()) + ), + NamespaceMode::Fleet => format!("{}cells/", self.inner.prefix), + }; + let expected = wire::Root::new(&config, data_prefix.clone()); + expected.validate()?; + let bytes = wire::encode(wire::ROOT, &expected, self.inner.limit)?; + let namespace = NamespaceCatalog { + inner: self.inner.clone(), + config, + root_digest: wire::digest(&bytes), + data_prefix, + }; + Ok((namespace, expected, bytes)) + } } impl NamespaceCatalog { @@ -481,34 +538,6 @@ impl NamespaceCatalog { self.bootstrap_initial(birth, log, None).await } - pub(crate) async fn bootstrap_managed_member( - &self, - birth: &GroupBirth, - log: &ReplicaLog, - authorization: &super::fleet_catalog::FleetGroupAuthorization, - ) -> Result { - if !birth - .managed_owner() - .is_some_and(|owner| owner.fleet == authorization.binding().fleet()) - || !authorization.check_birth(self, birth) - || birth.voters.len() != 3 - || !birth.learners.is_empty() - { - return Err(CatalogError::Conflict( - "managed initial birth authorization", - )); - } - if !authorization.allows_managed_genesis() { - match log.group(birth.scope()) { - Ok(_) => (), - Err(LogError::NotFound) => return Err(CatalogError::ManagedAdmissionRequired), - Err(error) => return Err(CatalogError::Log(error)), - } - } - self.bootstrap_initial(birth, log, Some(authorization.binding())) - .await - } - async fn bootstrap_initial( &self, birth: &GroupBirth, diff --git a/crates/cellar/durability/raft/catalog_birth.rs b/crates/cellar/durability/raft/catalog_birth.rs index fcea2cd92..106f087ac 100644 --- a/crates/cellar/durability/raft/catalog_birth.rs +++ b/crates/cellar/durability/raft/catalog_birth.rs @@ -62,46 +62,6 @@ impl NamespaceCatalog { .await } - /// A reservation remains inventoried even if drain wins a later revision. - /// Cancellation contends on the same birth key and never erases a live birth. - pub async fn cancel_managed_birth( - &self, - authorization: &super::super::fleet_catalog::FleetBirthAuthorization, - operation: ProposalId, - timeout: Duration, - ) -> Result { - authorization.check_namespace(self)?; - self.cancel_selected_birth( - authorization.proposal(), - operation, - timeout, - Some(authorization.owner()), - ) - .await - } - - pub async fn resolve_managed_birth( - &self, - authorization: &super::super::fleet_catalog::FleetBirthAuthorization, - timeout: Duration, - ) -> Result { - authorization.check_namespace(self)?; - if timeout.is_zero() || timeout > Duration::from_secs(60) { - return Err(CatalogError::Invalid("managed birth timeout")); - } - let birth = crate::asyncrt::timeout( - timeout, - self.select_birth(authorization.proposal(), None, Some(authorization.owner())), - ) - .await - .map_err(|_| CatalogError::Unconfirmed)?? - .live()?; - if !authorization.check_birth(self, &birth) { - return Err(CatalogError::Conflict("reserved birth winner")); - } - Ok(birth) - } - async fn cancel_selected_birth( &self, proposal: BirthProposal, diff --git a/crates/cellar/durability/raft/catalog_disaster.rs b/crates/cellar/durability/raft/catalog_disaster.rs index 4584f77fe..db7782907 100644 --- a/crates/cellar/durability/raft/catalog_disaster.rs +++ b/crates/cellar/durability/raft/catalog_disaster.rs @@ -48,15 +48,12 @@ impl DisasterFenceRequest { /// controller and bound to `request.digest()`. Fail closed on partial/unknown /// outcomes. Fences are permanent; retries must be idempotent and cancellation /// must never undo a fence. Cellar supplies no permissive default implementation. -pub trait DisasterFenceAuthority: Send + Sync { - fn fence<'a>( - &'a self, - request: &'a DisasterFenceRequest, - ) -> BoxFuture<'a, anyhow::Result>>; +pub trait DisasterFenceAuthority: Send + Sync { + fn fence<'a>(&'a self, request: &'a Request) -> BoxFuture<'a, anyhow::Result>>; fn verify<'a>( &'a self, - request: &'a DisasterFenceRequest, + request: &'a Request, evidence: &'a [u8], ) -> BoxFuture<'a, anyhow::Result<()>>; } diff --git a/crates/cellar/durability/raft/catalog_inventory/groups.rs b/crates/cellar/durability/raft/catalog_inventory/groups.rs new file mode 100644 index 000000000..6c87defc5 --- /dev/null +++ b/crates/cellar/durability/raft/catalog_inventory/groups.rs @@ -0,0 +1,417 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Physical-group admission shares the namespace CAS with disaster exclusion. +//! Frozen intents include unpublished groups; listing never defines coverage. +use super::*; +use crate::raft::catalog::ManagedBirthOwner; +use crate::raft::cell_groups::{CellGroupBirthRequest, RangeDirectory}; +use serde::{Deserialize, Serialize}; + +const CUT_KIND: u8 = 196; + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(crate) struct CellGroupImportCut { + pub root: [u8; 32], + pub operation: [u8; 16], + pub revision: u64, + pub count: u64, + #[serde(deserialize_with = "required_index")] + pub index: Option<[u8; 32]>, + pub directory: [u8; 32], +} + +fn required_index<'de, D: serde::Deserializer<'de>>(d: D) -> Result, D::Error> { + Option::deserialize(d) +} + +impl CellGroupImportCut { + pub(crate) fn validate(&self, namespace: &NamespaceCatalog) -> Result<(), CatalogError> { + if self.root != namespace.root_digest + || self.revision <= self.count + || self.revision < 3 + || self.index.is_none() != (self.count == 0) + || self.directory == [0; 32] + || namespace.config.managed_fleet.is_some() + || ProposalId::new(self.operation).is_err() + { + return Err(CatalogError::Corrupt("group import cut")); + } + Ok(()) + } + + pub(crate) fn digest(&self) -> Result<[u8; 32], CatalogError> { + Ok(record::digest(&record::encode(CUT_KIND, self, 4096)?)) + } +} + +/// A frozen, authoritative admission set. This is neither complete enrollment +/// nor permission to start, activate, remove or release a physical replica. +#[derive(Clone)] +pub struct FrozenCellGroupInventory { + owner: Arc, + cut: CellGroupImportCut, +} + +impl FrozenCellGroupInventory { + pub fn len(&self) -> u64 { + self.cut.count + } + + pub fn is_empty(&self) -> bool { + self.cut.count == 0 + } + + pub fn operation(&self) -> ProposalId { + // Construction validates the durable cut. Retain no unchecked public + // constructor, including for a resumed inventory. + ProposalId::new(self.cut.operation).expect("validated group inventory operation") + } + + pub(crate) fn import_cut( + &self, + directory: &RangeDirectory, + ) -> Result { + if !Arc::ptr_eq(&self.owner, &directory.namespace().inner) { + return Err(CatalogError::Conflict("foreign frozen group inventory")); + } + self.cut.validate(directory.namespace())?; + Ok(self.cut.clone()) + } +} + +pub(crate) struct ImportCellGroupStep { + pub entry: Option, + pub next: Option, +} + +pub(crate) fn group_key(request: &CellGroupBirthRequest) -> String { + hex::encode(record::digest(&request.route().history.group.bytes())) +} + +impl NamespaceCatalog { + pub(crate) async fn cell_group_owner(&self) -> Result, CatalogError> { + let (head, _) = Store(self).current().await?; + Ok(match head.mode { + Mode::Owned { fleet, catalog } + | Mode::GroupImporting { fleet, catalog, .. } + | Mode::GroupAdopted { fleet, catalog, .. } => { + Some(ManagedBirthOwner { fleet, catalog }) + } + Mode::Open | Mode::GroupFreezing { .. } | Mode::GroupFrozen { .. } => None, + _ => return Err(CatalogError::Conflict("not a shared-group namespace")), + }) + } + + pub(crate) async fn require_cell_group_unmanaged(&self) -> Result<(), CatalogError> { + if self.cell_group_owner().await?.is_some() { + return Err(CatalogError::ManagedAdmissionRequired); + } + Ok(()) + } + + /// Called after the permanent global claim, before any topology publication + /// or materialization. A losing CAS can retain a harmless identity claim; + /// without this admission it can never become a group birth. + pub(crate) async fn admit_cell_group( + &self, + request: &CellGroupBirthRequest, + ) -> Result<(), CatalogError> { + let store = Store(self); + let index = Index::new(&store); + let key = group_key(request); + let id = request.route().birth_digest; + if request.route().history.cluster != self.config.cluster + || request.route().history.namespace != self.config.namespace + || request.route().schema_digest != self.config.schema_digest + || record::digest(request.encode()) != id + { + return Err(CatalogError::Conflict("group admission namespace")); + } + for _ in 0..MAX_ATTEMPTS { + let (mut head, token) = store.current().await?; + if let Some(prior) = index.lookup(head.index, &key).await? { + if pointer(&prior)? != id { + return Err(CatalogError::Conflict("group admission identity")); + } + return self.verify_cell_group_intent(request).await; + } + match head.mode { + Mode::Open => (), + Mode::GroupFreezing { .. } + | Mode::GroupFrozen { .. } + | Mode::GroupImporting { .. } + | Mode::Frozen { .. } + | Mode::Importing { .. } => return Err(CatalogError::BirthsFrozen), + _ => return Err(CatalogError::ManagedAdmissionRequired), + } + let intent_key = store.key(&format!("group-intents/{}", hex::encode(id))); + if self + .inner + .create_and_observe_bounded(&intent_key, request.encode(), INDEX_PAGE_LIMIT) + .await? + != request.encode() + { + return Err(CatalogError::Corrupt("group admission intent collision")); + } + head.index = index.update(head.index, &key, Some(id.to_vec()), 0).await?; + head.count = head.count.checked_add(1).ok_or(CatalogError::Limit)?; + head.revision = head.revision.checked_add(1).ok_or(CatalogError::Limit)?; + head.validate(self)?; + let result = store.cas(&head, token).await; + let (observed, _) = store.current().await?; + if index.lookup(observed.index, &key).await?.as_deref() == Some(id.as_slice()) { + return self.verify_cell_group_intent(request).await; + } + result?; + } + Err(CatalogError::Unconfirmed) + } + + pub(crate) async fn verify_cell_group_intent( + &self, + request: &CellGroupBirthRequest, + ) -> Result<(), CatalogError> { + let store = Store(self); + let (head, _) = store.current().await?; + let value = Index::new(&store) + .lookup(head.index, &group_key(request)) + .await? + .ok_or(CatalogError::Corrupt("untracked group birth"))?; + let id = pointer(&value)?; + let bytes = self + .inner + .read_versioned_bounded( + &store.key(&format!("group-intents/{}", hex::encode(id))), + INDEX_PAGE_LIMIT, + ) + .await? + .ok_or(CatalogError::Corrupt("missing group intent"))? + .0; + if id != request.route().birth_digest + || record::digest(&bytes) != id + || bytes != request.encode() + { + return Err(CatalogError::Corrupt("group intent identity")); + } + Ok(()) + } + + pub(crate) async fn begin_cell_group_freeze( + &self, + operation: ProposalId, + ) -> Result<(), CatalogError> { + let store = Store(self); + for _ in 0..MAX_ATTEMPTS { + let (mut head, token) = store.current().await?; + if head.disaster { + return Err(CatalogError::Conflict( + "disaster namespace cannot be imported", + )); + } + match &head.mode { + Mode::GroupFreezing { operation: prior } if *prior == operation.bytes() => { + return Ok(()); + } + Mode::GroupFrozen { cut } if cut.operation == operation.bytes() => return Ok(()), + Mode::Open => (), + _ => return Err(CatalogError::Conflict("group freeze owner/operation")), + } + head.mode = Mode::GroupFreezing { + operation: operation.bytes(), + }; + head.revision = head.revision.checked_add(1).ok_or(CatalogError::Limit)?; + head.validate(self)?; + let result = store.cas(&head, token).await; + if store.current().await?.0.mode == head.mode { + return Ok(()); + } + result?; + } + Err(CatalogError::Unconfirmed) + } + + pub(crate) async fn finish_cell_group_freeze( + &self, + operation: ProposalId, + directory: [u8; 32], + ) -> Result { + let store = Store(self); + for _ in 0..MAX_ATTEMPTS { + let (mut head, token) = store.current().await?; + if let Mode::GroupFrozen { cut } = &head.mode { + if cut.operation == operation.bytes() && cut.directory == directory { + cut.validate(self)?; + return Ok(FrozenCellGroupInventory { + owner: self.inner.clone(), + cut: cut.clone(), + }); + } + } + if head.mode + != (Mode::GroupFreezing { + operation: operation.bytes(), + }) + { + return Err(CatalogError::Conflict("group freeze continuation")); + } + head.revision = head.revision.checked_add(1).ok_or(CatalogError::Limit)?; + let cut = CellGroupImportCut { + root: self.root_digest, + operation: operation.bytes(), + revision: head.revision, + count: head.count, + index: head.index, + directory, + }; + cut.validate(self)?; + head.mode = Mode::GroupFrozen { cut }; + let result = store.cas(&head, token).await; + if store.current().await?.0.mode != head.mode { + result?; + } + } + Err(CatalogError::Unconfirmed) + } + + pub(crate) async fn change_cell_group_import( + &self, + cut: &CellGroupImportCut, + owner: &ManagedBirthOwner, + adopted: bool, + ) -> Result<(), CatalogError> { + cut.validate(self)?; + let store = Store(self); + let desired = if adopted { + Mode::GroupAdopted { + fleet: owner.fleet, + catalog: owner.catalog, + cut: cut.clone(), + } + } else { + Mode::GroupImporting { + fleet: owner.fleet, + catalog: owner.catalog, + cut: cut.clone(), + } + }; + for _ in 0..MAX_ATTEMPTS { + let (mut head, token) = store.current().await?; + if head.mode == desired + || !adopted + && matches!(&head.mode, + Mode::GroupAdopted { fleet, catalog, cut: prior } + if *fleet == owner.fleet && *catalog == owner.catalog && prior == cut) + { + return Ok(()); + } + let allowed = if adopted { + matches!(&head.mode, Mode::GroupImporting { fleet, catalog, cut: prior } + if *fleet == owner.fleet && *catalog == owner.catalog && prior == cut) + } else { + matches!(&head.mode, Mode::GroupFrozen { cut: prior } if prior == cut) + }; + if !allowed { + return Err(CatalogError::Conflict("group import owner/cut")); + } + head.mode = desired.clone(); + head.revision = head.revision.checked_add(1).ok_or(CatalogError::Limit)?; + head.validate(self)?; + let result = store.cas(&head, token).await; + if store.current().await?.0.mode == desired { + return Ok(()); + } + result?; + } + Err(CatalogError::Unconfirmed) + } + + pub(crate) async fn check_cell_group_import( + &self, + cut: &CellGroupImportCut, + owner: &ManagedBirthOwner, + adopted: bool, + ) -> Result<(), CatalogError> { + cut.validate(self)?; + let valid = match Store(self).current().await?.0.mode { + Mode::GroupAdopted { + fleet, + catalog, + cut: prior, + } => fleet == owner.fleet && catalog == owner.catalog && prior == *cut, + Mode::GroupImporting { + fleet, + catalog, + cut: prior, + } if !adopted => fleet == owner.fleet && catalog == owner.catalog && prior == *cut, + _ => false, + }; + if !valid { + return Err(CatalogError::Conflict("group import marker")); + } + Ok(()) + } + + pub(crate) async fn cell_group_import_entry( + &self, + directory: &RangeDirectory, + cut: &CellGroupImportCut, + owner: &ManagedBirthOwner, + key: &str, + ) -> Result { + self.check_cell_group_import(cut, owner, false).await?; + let store = Store(self); + let pointer = Index::new(&store) + .lookup(cut.index, key) + .await? + .ok_or(CatalogError::Corrupt("untracked group import witness"))?; + let id = super::pointer(&pointer)?; + let bytes = self + .inner + .read_versioned_bounded( + &store.key(&format!("group-intents/{}", hex::encode(id))), + INDEX_PAGE_LIMIT, + ) + .await? + .ok_or(CatalogError::Corrupt("missing group import intent"))? + .0; + if record::digest(&bytes) != id { + return Err(CatalogError::Corrupt("group import intent digest")); + } + let request = directory.decode_group_birth(&bytes)?; + if request.route().birth_digest != id || group_key(&request) != key { + return Err(CatalogError::Corrupt("group import index identity")); + } + Ok(request) + } + + pub(crate) async fn cell_group_import_step( + &self, + directory: &RangeDirectory, + cut: &CellGroupImportCut, + owner: &ManagedBirthOwner, + after: Option<&str>, + ) -> Result { + self.check_cell_group_import(cut, owner, false).await?; + let cursor = after + .map(|key| IndexCursor::after(cut.index, key.to_owned())) + .transpose()?; + let page = Index::new(&Store(self)) + .scan(cut.index, cursor.as_ref(), 2) + .await?; + let more = page.entries.len() > 1; + let Some(first) = page.entries.into_iter().next() else { + return Ok(ImportCellGroupStep { + entry: None, + next: None, + }); + }; + let entry = self + .cell_group_import_entry(directory, cut, owner, &first.key) + .await?; + Ok(ImportCellGroupStep { + entry: Some(entry), + next: more.then_some(first.key), + }) + } +} diff --git a/crates/cellar/durability/raft/catalog_inventory/mod.rs b/crates/cellar/durability/raft/catalog_inventory/mod.rs index 21f649bd6..fbe2de42b 100644 --- a/crates/cellar/durability/raft/catalog_inventory/mod.rs +++ b/crates/cellar/durability/raft/catalog_inventory/mod.rs @@ -10,9 +10,9 @@ use object_store::{path::Path, PutMode, PutOptions, UpdateVersion}; use std::{sync::Arc, time::Duration}; mod current; +pub(crate) mod groups; mod import; mod wire; -pub(crate) use current::ManagedBirthInventory; pub(crate) use import::BirthImportCut; use wire::{Head, Mode, Pin}; diff --git a/crates/cellar/durability/raft/catalog_inventory/wire.rs b/crates/cellar/durability/raft/catalog_inventory/wire.rs index 9f33b072a..25273573d 100644 --- a/crates/cellar/durability/raft/catalog_inventory/wire.rs +++ b/crates/cellar/durability/raft/catalog_inventory/wire.rs @@ -22,6 +22,22 @@ pub(super) enum Mode { catalog: [u8; 32], cut: BirthImportCut, }, + GroupFreezing { + operation: [u8; 16], + }, + GroupFrozen { + cut: groups::CellGroupImportCut, + }, + GroupImporting { + fleet: [u8; 16], + catalog: [u8; 32], + cut: groups::CellGroupImportCut, + }, + GroupAdopted { + fleet: [u8; 16], + catalog: [u8; 32], + cut: groups::CellGroupImportCut, + }, } #[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] @@ -68,6 +84,49 @@ impl Head { return Err(CatalogError::Corrupt("birth inventory identity/count")); } match (&self.mode, &namespace.config.managed_fleet) { + (Mode::GroupFreezing { operation }, None) if self.revision > 1 => { + ProposalId::new(*operation) + .map_err(|_| CatalogError::Corrupt("group freeze operation"))?; + } + (Mode::GroupFrozen { cut }, None) => { + cut.validate(namespace)?; + if self.revision != cut.revision + || self.count != cut.count + || self.index != cut.index + { + return Err(CatalogError::Corrupt("changed frozen group set")); + } + } + ( + Mode::GroupImporting { + fleet, + catalog, + cut, + }, + None, + ) + | ( + Mode::GroupAdopted { + fleet, + catalog, + cut, + }, + None, + ) => { + cut.validate(namespace)?; + validate_owner(*fleet, *catalog)?; + let delta = if matches!(self.mode, Mode::GroupAdopted { .. }) { + 2 + } else { + 1 + }; + if self.revision != cut.revision.checked_add(delta).ok_or(CatalogError::Limit)? + || self.count != cut.count + || self.index != cut.index + { + return Err(CatalogError::Corrupt("changed imported group set")); + } + } (Mode::Open, None) => (), (Mode::Owned { fleet, catalog }, Some(owner)) => { if fleet != &owner.fleet || catalog != &owner.catalog { diff --git a/crates/cellar/durability/raft/catalog_wire.rs b/crates/cellar/durability/raft/catalog_wire.rs index 6fe705b88..154efc89d 100644 --- a/crates/cellar/durability/raft/catalog_wire.rs +++ b/crates/cellar/durability/raft/catalog_wire.rs @@ -220,6 +220,7 @@ mod tests { let mut root = Root::new(&config, "test".into()); root.validate().unwrap(); for protocol in [ + "cellar-raft-ltx-1/members-1/seeds-1/s3-1/repairs-1/nodes-1/bundles-1/fleet-1/enrollment-1/node-probes-1/birth-admission-1/managed-genesis-1/group-reservations-1/birth-inventory-1/namespace-import-1/fleet-namespaces-1/catalog-trust-1/fleet-admission-1/fleet-quorum-1/fleet-cuts-1/empty-archives-1/archive-cuts-1/fleet-finalization-1/fleet-stop-1/fleet-stop-rpc-1/capacity-readiness-1/resource-reservations-1/physical-inventory-1/fleet-removal-1/resource-movement-1/movement-completion-1/node-isolation-1/isolated-removal-1/images-1/fenced-movement-1/fleet-execution-1/aborted-movement-1/topology-bindings-1/topology-intent-1/config-tails-1/restart-stop-1/restart-return-1/restart-resume-1/maintenance-operator-1/stop-consent-refresh-1/extent-owners-1/operator-lifecycle-1/extent-fleet-proofs-1/failure-recovery-1", "cellar-raft-ltx-1", "cellar-raft-ltx-1/members-1", "cellar-raft-ltx-1/members-1/seeds-1", diff --git a/crates/cellar/durability/raft/cell_groups/README.md b/crates/cellar/durability/raft/cell_groups/README.md new file mode 100644 index 000000000..d82c6beb9 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/README.md @@ -0,0 +1,885 @@ + + +# Cell-group state and execution + +This component owns physical-group manifests, execution, checkpoints and local +recovery. Canonical SDK hosts bind declared namespaces and admitted groups to +these owners. A stored root or local attachment is not fresh quorum authority. +See the [delivery status](../../../../../docs/design/multi-raft/cell-groups/remaining-work.md) +for the remaining public fleet and backend qualification gates. + +## Managed startup outcomes + +`cellar::sdk::consensus` exports `CellGroupStartupError` and its nested +`CellGroupControllerError`. `Refused` means this invocation submitted no +preparation. `Unconfirmed` can follow durable installation or attachment; retain +the original history, permit and resource debit, and reconcile from fresh +observations. Neither result resolves a prior lost reply. A locally installed +learner must still catch up through Raft before promotion. Reopening a joined +member uses its retained seed receipt and permanent member claim; its old-boot +learner grant cannot initialize work on the new boot. The movement reconciler +also reports provider I/O during terminal settlement as `Unconfirmed`: a failed +confirmation read cannot undo an already committed root CAS. Reconcile the same +reservation and verify its permanent receipt. Corruption and authority failures +keep their typed errors. Settlement removes the active reservation while +retained source storage remains charged. + +## Managed maintenance and retained return + +Authenticated fleet observations sample leader load from actual attached group +owners. Each runtime turn visits at most sixteen owners; a roster revision, +complete population check and original deadline span the scan. A failed owner, +roster change, pending attachment or incomplete physical inventory supplies no +known headroom. Counts describe physical groups, independently of cell count. +This advisory sample supplies neither a transfer grant nor serving authority. + +A catalog-authorized maintenance return starts paused. Its exact new boot may +reattach an existing voting member only after the directory verifies the current +Live group, settled membership, returned-boot lineage, retained state, permanent +member claim and complete image closure. Joined members also retain their seed +receipt. The directory rechecks the local quiescence guard after catalog I/O. +A paused return cannot create genesis or a missing claim, seed a learner or admit +business writes. Generic new-group startup continues to require the ordinary +open/resumed guard. Fresh catalog resume proof and capacity settlement remain +separate from local attachment. + +A public command refused by the maintenance barrier reports temporary +`Unavailable` with no submission by that invocation. The cause is captured under +the registration lock, rather than inferred by rereading a flag after resume. +Permanent closure remains a distinct cause. Background effects defer a captured +temporary refusal, including when resume races with their next flag read. +Neither this refusal nor a lost lifecycle response resolves an earlier mutation. +A remote maintenance stop or resume returns `Unconfirmed` after enqueueing its +bounded Node-owned proof work. Exact rooted authority and payload are checked +before enqueue, preserving typed refusals. The worker rechecks them before +collecting fresh physical/quorum evidence and syncing the guard. Releasing the +transport waiter lets the survivor leader +renew its inspection within the existing two-slot peer metadata bank. A later +exact retry can observe the synced effect; queueing alone is never acceptance. +Reconcile an uncertain stop using the same rooted child; signed closure requires +actual owned shutdown and join before offline confirmation. + +## Recovered public-node startup + +An offline recovery operator persists the exact recovery intent, obtains the +infrastructure fence, verifies the complete archive closure and seeds every +replacement voter with a fresh disk/engine and disjoint member/node identities. +It publishes that same intent only after every required seed receipt is durable. +Archive setup has its own one-use intent, which must be synced before commit. +None of these steps creates a live quorum. + +`RaftCellsServices::recovery` supplies the independent +`DisasterFenceAuthority` for unmanaged public +startup in `LogOpenMode::Resume`. Each assignment must already have the exact +local recovery genesis, seed receipt and retained engine/disk pins. Startup +locates the original request through the selected birth and immutable binding, +then independently verifies the external fence. A missing authority, rejected +fence, missing image or blank disk refuses; startup cannot regenerate these +proofs from object-store metadata. + +One node-owned control service requests activation from the actual shared Raft +owners. Its metadata uses the existing control bank, and queued permits retain +their original deadlines. Startup may return before a fresh quorum exists; +application admission stays closed until activation commits and applies. Only +that acknowledgement can complete the original catalog intent. Unknown storage +outcomes retain the same intent for reconciliation. An authority or corruption +failure quarantines that group. Shutdown signals and joins entered recovery +verification/publication before closing consensus and storage. + +A retained restart uses the same disks and independently verifies the fence +again. `RangeDirectory::group_recovery_pending` observes the exact published +intent; a completed status requires its activation receipt and selected route. +This metadata is neither current quorum proof nor serving permission. An +unrelated recovery lock cannot substitute for this group's activation receipt. +Managed disaster recovery still requires its fleet recovery ownership contract; +an unmanaged authority cannot enable that path. + +## Manifests + +A manifest is a persistent AVL tree ordered by the namespace routing digest and +full cell ID. Each bounded page holds one cell leaf and at most two child +references. References bind SHA-256, height, subtree count and earliest due-effect time. Insertions and +updates copy bounded paths; rotations preserve balance. Unchanged cell images +are reused. The format uses `CLRCGM01`, the marker-1 page hash domain, canonical +JSON and a 16 KiB decode bound. Tree depth and cell count are explicit limits. +These limits are admission ceilings, not qualified fleet capacities. + +The due-effect index stores each subtree's earliest alarm/outbox time in its +required nullable page reference. `ManifestRoot::next_due_unix_millis` is a cheap +hint after complete-root verification. `ManifestTree::next_due` visits bounded +paths to select one due cell after a routing-key cursor, wrapping once for fair +selection. Future-only subtrees need no SQLite hydration. A prepared physical root also +exposes `next_due` over its verified persistent pin tree, with no filesystem or +SQLite I/O. Its returned image pin is a data view and grants no execution rights. +The caller runs page +I/O on its blocking execution domain and retains the selected verified root; +returned metadata still requires fresh leader, route, image and private-command +admission before execution. Missing scheduling fields refuse at marker 1, with +no compatibility reader. The shared controller now integrates indexed discovery +and private system commands. Full CG-08 effect/cache qualification and the public +SDK/fleet cutover remain unfinished; qualification is scoped to its recorded source. + +Leaves bind logical cell history, mutation sequence, image commitment and +geometry, last group installation/mutation position, schema, effect summary, +immutable original enrollment identity and the selected version's complete +`CommitWitness` (original group history, term/index and proposal ID). A page +without either the enrollment or witness field refuses at marker 1; no old-layout +decoder or inferred identity is used. The witness is part of the authenticated +page and survives unrelated group progress, restart and image collection. A local +installation position remains separate from original decision provenance for +future transfers. Shape validation of foreign-group provenance grants no transfer +authority; the transfer protocol must establish its lineage. + +Ordinary `prepare_update` requires the witness to name the current group and +its exact mutation position. The Raft owner additionally matches enrollment and +mutation witnesses to the actual committed entry's proposal before publishing. +An otherwise correct image cannot supply a fabricated proposal witness. Repeated +enrollment and unchanged entries retain the existing leaf and witness exactly. +`CellManifestLeaf::observation` exposes version/witness data; only an authoritative +selected root establishes its commitment. Local image +filenames are excluded: replicas may store the same image under different local +names and still derive the same logical manifest. A physical image adapter must +resolve and pin each image by the selected logical leaf. Extent and materialized +commitments retain separate domains. + +The page store uses an injected filesystem, a lifetime exclusive lock, bounded +inventory and byte/page quotas. A new page crosses file and directory sync before +its reference is returned. Any ambiguous write/sync quarantines the owner until +reopen. Reopened filenames are only inventory; reads verify length and digest, +and reuse verifies and syncs bytes again. Every prepared pin-tree node holds a +page lease and the directory lock. Dropping a store handle cannot let a replacement +owner collect pages still selected by old readers. The store explicitly bounds +retained root slots (2–4096), reserving current and pending slots for each group. + +`verify` traverses the entire authenticated closure, checks ordering and the +selected applied cut, and delegates actual image verification/pinning to its +caller. The callback must release any partial pins if verification fails. A +successful page traversal alone is not an image seal, Raft commit or serving +capability. Updates compare the expected cell version and require increasing +sequence, LTX position and group mutation position. + +`prepare_complete` is the full-closure path for bootstrap, recovery and +checkpoint preparation. Every leaf must match a real materialized generation +pin, sealed extent image, or canonical empty image. Extent roots bind the stable +logical cell-history digest and schema. The method verifies and syncs all pages, +retains physical image pins and binds the preparation to the exact engine owner. +This full scan establishes the capability used by ordinary writes. + +`prepare_update` compares the prior cell version, requires the next group index, +verifies the changed image seal and synchronizes the changed manifest path. A +persistent in-memory AVL pin tree mirrors the manifest. Unchanged subtrees are +shared across prepared/published roots; changed paths and rotations reuse the +unchanged cell image pins. Preparation performs O(log cells) page work and node +allocation, with O(log² cells) bounded in-memory comparisons. Cloning a root is +constant-time. `pin_cell` selects a coherent leaf/image pair from that root; read +authority still belongs to the caller. Memory admission conservatively charges +each owner for its entire tree, including cell IDs and pin-node allocations. + +Incremental capabilities bind the exact previous manifest and page-store owner. +The engine refuses a staged successor of a different branch at the same index; +an identical already-selected result remains retryable. Reopening the page store +requires full preparation again. A live old root or individual image reader +prevents generation reclamation after a newer root publishes. Stores still need +authoritative retention reconciliation before retiring inventory, especially +after an uncertain engine outcome; a disposable request pin alone is insufficient. + +After validating an engine batch, storage reserves physical root pins before +starting the write. Unarmed reservations can be cancelled without persistence. +Once armed, dropping the reservation before cache publication finishes quarantines +the retention owner and keeps both old and candidate closures. Successful cache +publication selects the candidate and releases the old engine reference. Identical +retries use the same retention protocol. No retention mutex spans engine or file I/O. + +`prepare_complete` automatically registers a recovered root only when it matches +the engine-selected marker. An existing group cannot publish through an unrecovered +retention owner. Before collection, the owning service must prepare **every** +engine-selected root assigned to this directory and call `reconcile_retained` with +the complete additional snapshot/archive/transition root set. This is a trusted +inventory handoff, not discovery from page filenames or proof supplied by clients. +The future shared host must obtain that inventory from its authoritative catalog. +Replacing the additional set never releases the engine's current roots. Pending +publication, an uncertain outcome or engine closure prevents reconciliation and +collection until restart. Reopening requires recovery and reconciliation again. + +`reclaim` scans at most the explicit scan limit (1–4096) and resumes from its last +inventory key. Only pages without physical leases can be selected. Deletions and +the directory barrier finish before page/byte quotas are refunded; failure keeps +the prior charge and quarantines the owner. Reader, apply, snapshot and transition +pins independently protect their closures. Image-store collection still follows +its own complete durable-reference reconciliation contract; the retained manifest +pins prevent it from deleting selected or uncertain images in the live process. + +The engine persists group root, applied position, membership and log/hard state +in one synced batch. It requires contiguous apply, matching group/birth/range, +matching membership revision and the current engine owner. Cached readers see +the old root until sync completes. Old single-image accessors and snapshot writes +refuse a group root; future group snapshot installation uses its own complete +closure protocol. Recovery first observes the selected metadata and then repeats +full closure verification before any serving admission. Missing images or pages +cannot become empty cells. + +The local application record now has explicit image/group semantic domains at +marker 1. Old bare-image record shapes refuse; there is no old-layout decoder or +automatic migration. The staged image variant supports existing code while the +remaining production consumers are converted, and is removed at hard cutover. +Existing disks and archives must be retained, not reset by this component. + +Further integration must wire complete catalog/checkpoint/transition inventory +handoff into the shared host. Native recovery fixtures exercise both physical +image modes with independent SQLite value and receipt rows. Process-kill cuts +cover image/page writes and sync, engine sync, publication and reclamation; +restart verifies every selected cell before collecting orphaned objects. Missing +or corrupt pages, images, extent roots and non-header extents refuse complete-root +preparation. These are storage-boundary checks, not quorum, SDK deduplication or +RustFS qualification. The full CG-02 source-bound evidence gate is recorded in the +delivery plan. + +## Complete-group checkpoint storage + +`CellGroupCheckpointPool` admits both sources and receivers under one node-wide +item/byte budget. A source captures the exact engine-selected root, membership +and configuration. It retains immutable dependencies while later writes proceed, +and streams bounded chunks without aggregating the group images. + +A receiver binds the destination engine and its selected base. It validates +ordered page/image parts, route/history, cut, membership and complete tree closure. +Materialized images spool one at a time under a shared `ScratchPool`, then undergo +independent SQLite/checksum verification and destination sync. Extent manifests +and objects are verified against each leaf and independently sealed. Partial, +corrupt, foreign or over-budget transfers cannot publish. All physical work runs +on a joined blocking owner; reservations follow that owner through completion. + +`PreparedCellGroupCheckpoint` owns the sealed destination closure and receive +reservation. `SnapshotChange::CellGroup` atomically publishes its snapshot, +configuration, membership and applied root through the shared engine fence. +Mixed single-cell/group writes, conflicting bases and stale cuts refuse. The +capability establishes physical storage only: its future Raft consumer must bind +it to the exact accepted snapshot effect before submitting the engine write. + +The page owner retains the installed checkpoint separately from the advancing +current root. Checkpoint retention consumes a third root slot in addition to the +current/pending publication slots. Recovery reconstructs both complete closures; +a missing older checkpoint refuses startup and reclamation even when the current +applied root is complete. Physical recovery never renews quorum authority. + +Canonical startup recovers every retained physical group's complete current and +checkpoint image graphs before releasing conservative filename retention in the +shared serving inventory. It refuses an incomplete assignment and publishes no +peer/public routes before this adoption. Complete manifest pins keep both +selections alive; uncertain publication additionally retains its pending closure. +Fresh applied materialized/extent images transfer their live pins into that graph +without adding a permanent host reference for every entry. An existing image's +retention is preserved. Disposable command/enrollment previews retire only after +exact terminal owner reconciliation, retaining their physical pins until work +finishes. Unknown outcomes keep the pending owner. Materialized preparation uses +its existing bounded collection; transient extent roots still need node-owned +collection. The offline complete-inventory reclamation API remains offline. + +These are verification APIs. Ordinary peer ingress still rejects snapshot +messages. `step_checkpoint` on the owner/runtime accepts an authenticated envelope +only with its exact receiver-local closure, engine/page owner, selected base and +membership binding. It requires settled prior apply and delegates restoration to +RawNode. Admission returns `Restoring` or `Ignored`; neither is a persistence +receipt. A matching log can advance commit and replay entries without installing +the received image. A restored snapshot is attached to its exact Ready write; +only the matching engine completion advances the application view and releases +Raft responses and `CheckpointInstalled`. Queued work retains the receiver budget +through engine sync and runtime shutdown. + +An unstreamed captured source can become a `PreparedLocalCellGroupCheckpoint`. +`publish_checkpoint` admits only the owner's exact current applied cut and runs +publication on its persistence lane. The distinct `CellGroupCheckpoint` storage +operation preserves hard state, applied state and every later log entry; it +compacts only the prefix represented by the complete checkpoint. Mixed writes +and stale cuts refuse. `CheckpointPublished` is emitted after the matching engine +completion. Source admission and pins survive accepted publication and shutdown. + +Archive registration owns an activation intent across retries, closing later +checkpoint admission while earlier exact cut-bound leases drain. Owner and +storage share the same lease so entered work can finish under the preceding +policy. Rejected or unentered shutdown work releases its intent; entered metadata +I/O retains it after waiter cancellation and fsyncs the required retention bit +before object-store I/O. Only successful durable publication completes intent. +That bit is part of the required marker-1 group application record, remains +set through ordinary writes and snapshot installation, and is recovered on open. +The in-memory verified archive frontier is deliberately absent after restart. +Local checkpoint publication and the storage preflight refuse a cut beyond that +frontier. Only the controller's independently verified complete archive coverage +advances it; the public archive status cannot authorize compaction. Its journal +continues to own the current remote hold. The frontier retains bounded marker +metadata, without keeping a replay workspace per idle group. A behind-archive +checkpoint is skipped before capture so quorum apply and sibling checkpoint work +continue. This is a prerequisite for retained-log admission, not its byte/disk +bound or the physical Raft file reclamation gate. + +`capture_snapshot` pins the retained checkpoint named by the exact Raft snapshot, +including its original membership/configuration, after ordinary apply has advanced +the current root. A concurrent checkpoint replacement may refuse capture; a +successful source keeps its immutable closure until streaming ends. Recovery +reconstructs this retained closure before it can become an outbound source. + +Snapshot RPC transport, learner catch-up, joint membership and disk replacement +remain CG-05 work. Configuration entries remain closed. Neither checkpoint API +authorizes a new voter or establishes S3 durability. + +## Enrollment admission and ordered preparation + +`decide_enrollment` checks a bounded cell-addressed enrollment entry against the +current group route and committed lifecycle state. Sealed and retired ranges +refuse, including delayed entries prepared before the seal. Active admission +returns either a prospective creation record or the identical committed record. +A competing operation, cell incarnation, schema or initial image cannot replace +the winner. The same decision must run again at ordered log application; a +preparation-time check is not permission to bypass a later seal. + +`EnrollmentRecord` binds logical cell history, original creation operation, +schema and initial image. It deliberately excludes physical group placement, so +normal transfer preserves the record while the request adopts the current route. +Its bounded canonical marker-1 codec verifies data shape, not authority. The +record is now required in each authenticated manifest leaf and published with +the selected image. Ordinary mutation refuses any changed creation record. +The record matches the cell history/schema; sequence one must select its initial +image, and later sequences must advance that image's LTX position. A normal +transfer carries the same record. It records an initial image commitment without +pinning that superseded image forever. Complete snapshots and archives must +retain this metadata as part of their selected manifest closure. + +`prepare_enrollment` rechecks a bounded enrollment entry against the preceding +complete root and the caller's committed lifecycle phase. It verifies the next +contiguous group position before invoking the physical image producer. A new +creation prepares sequence one and its image seal together. An exact retry does +not invoke that producer or replace the latest image; `prepare_unchanged` advances +only the group position, sharing the same manifest and image pins without page +I/O. A quarantined page store refuses even this unchanged-root path. Competing creations refuse before physical preparation. The engine checks +the exact previous branch and committed position before publication. The future +shared owner must consume this boundary only for the actual ordered log entry; +preparation is not a quorum proof or command acknowledgement. + +Current-term quorum and read authority remain required even for a recognized +retry. Shared RawNode ownership, real concurrent enrollment and business callback +settlement remain CG-04 work. The storage fixtures exercise repeated enrollment +after mutation, restart and reclamation of the initial image in both physical +image formats; they do not qualify a live shared host. + +## Range directory + +`RangeDirectory` stores a namespace-bound, immutable AVL range tree. Each page +binds its route, child hashes, subtree bounds, height, count and coverage. A lookup +descends one path and verifies each page; the cache has independent page and byte +limits. A pinned `DirectoryView` can remain readable after a topology change and +requires no head refresh for lookup. A fully cached path performs no S3 requests. +Views are hints; the destination must check its own current route and quorum +authority. Observation does not traverse a complete recovery inventory. + +Initialization publishes at most 128 contiguous ranges covering the whole routing +domain. Every new route must match its permanently reserved group birth before +publication. A replacement removes up to 64 exact parent routes and inserts up to 64 +fresh groups covering the identical contiguous union. Children advance the maximum +parent epoch by one. Paths are copied incrementally; incomplete intermediate trees +cannot become a selected root. Group IDs remain reserved after retirement. + +`DirectoryMutation` contains canonical bounded bytes binding the namespace, +operation, expected head and exact routes. Persist these bytes before submission; +after an unknown result, decode and retry the identical request. One conditional +head update selects the tree, immutable journal and operation/group indexes. +Read-back confirms the selected operation even after another publisher advances +the head. Reusing an operation for different bytes refuses. The returned receipt +describes its historical decision, not the latest route or serving authority. + +Genesis writes the head before a permanent pin. An interrupted initializer can +finish the same genesis, including after another initializer pins and advances it. +A missing head with a retained pin refuses reinitialization. Head observation +verifies the pin, selected journal and operation-index binding. Missing or corrupt +objects refuse instead of returning an empty directory. No namespace-wide listing +is used, and historical pages/journals are retained; remote metadata GC is not +implemented here. + +Run async operations within the owning execution domain. Each operation has an +explicit timeout; a timed-out publication is unconfirmed and retains its original +identity for reconciliation. Cache locks never span I/O. Object and traversal +limits bound individual operations; defaults are not a qualified capacity claim. + +Directory publication is metadata only. The future topology owner must obtain +the seal, installation, archive and activation proofs before exposing replacement +ownership. Birth/claim admission and invocation-bounded route confirmation are +described below. Managed admission and production client integration remain open. Native directory tests exercise real conditional +object-store operations in memory, including races, dropped replies, corruption, +restart and bounded split/merge churn. They do not qualify live Raft ownership or real RustFS. CG-03 combines these +checks with the separately retained physical-bootstrap and routing evidence. + +## Cell-independent group birth + +`CellGroupBirthSpec` binds physical group history, range, route epoch, original +creation operation and initial members. It has no single-cell ownership field; +range boundaries still contain full cell keys. The namespace binds schema, +storage representation and durability policy. Preparation sorts members +canonically and enforces the namespace voter policy and distinct member/node IDs. +The bounded marker-1 birth encoding has its own semantic record kind; its digest +is the route's birth binding. Persist the returned request bytes before submission. + +Use `prepare_group_birth`, `reserve_group_birth`, directory publication, then +`materialize_group_birth`. Reservation conditionally creates the existing global +group-ID key with the complete canonical birth. Another namespace, incarnation, +placement or creation operation cannot reuse that ID. Existing records from the +old catalog also prevent reuse; they are not converted or reinterpreted. Losing +a directory CAS retains the reservation permanently. Directory publication checks +every child's exact reservation, but does not eagerly create its birth object. + +Materialization verifies the currently selected route and its global reservation, +conditionally writes the exact birth, then checks selection again. Missing +confirmation remains unconfirmed; reopen and decode the original request before +retrying. A retired or replaced range cannot materialize again. A birth observation +is bound to its catalog owner; `require_current_group_birth` checks current route, +reservation and selected immutable bytes, and refuses missing or changed state. + +These metadata methods expose no voter-start capability and write no permanent +member disk claim. The bootstrap path below connects the verified birth to real +engine genesis and permanent claims. Managed +namespaces refuse these unmanaged APIs until their rooted fleet authorization is +connected. This staged boundary does not permit blank-disk recovery or replace +current quorum membership. + +## Fresh-history disaster recovery + +Unmanaged recovery preserves one canonical `CellGroupRecoveryRequest` through +all unknown outcomes. `fence_group_recovery` verifies its exact held archive cut +and an independently verified permanent infrastructure fence. The authority must +stop every possible old writer and revoke restart; unreachable hosts and S3 +markers cannot supply that proof. Retain the source hold while its provenance is +referenced. + +`CellGroupRecoveryPool::prepare` restores complete data under fresh cell histories. +It grants no vote. `seed_group_recovery` binds the imported genesis to the actual +node, member, disk and engine, atomically syncs its mapping/root and then confirms +a permanent seed receipt. A lost waiter cannot release entered physical work. +`publish_group_recovery` requires every initial member to have the same complete +seed commitment and retains the topology lock after selecting the replacement. + +Ordinary bootstrap refuses recovery births. `bootstrap_recovered_group_member` +requires the existing seed, immutable engine genesis, storage pin and selected +publication, and rechecks the external fence. Physical reopen must verify the +complete selected and checkpoint closures before attachment. Startup alone grants +no serving authority: pending roots reject application work, including reads. + +`prepare_group_recovery_activation` issues an engine/process-bound permit for +`CellGroupRuntime::activate_recovery`. The owner requires a fresh-term quorum +barrier and admits one bounded activation proposal. Only committed, persisted +and applied activation opens application admission. An applied acknowledgement +allows `complete_group_recovery` to confirm the activation receipt and release +the topology lock. Repeated attempts preserve the first activation position. +Rare catalog operations own boxed futures to bound composed control polling +frames; they do not start per-cell tasks. + +Recovery origin, seed and activation records use the internal marker-1 hard +cutover, with no reader for older intermediate shapes. These low-level contracts +do not complete the public SDK/fleet cutover; implementation and qualification +status remains in the [design progress](../../../../../docs/design/multi-raft/cell-groups/progress.md). + +## Physical log identity cutover + +The engine registry uses `GroupHistory` (cluster, namespace, group and incarnation), +independent of application cell names. Marker-1 mappings use an explicit +`cellar-group-mapping-1` domain and a separate physical history. Old bare `scope` +mapping records refuse during recovery; there is no compatibility reader. + +The staged single-cell host still stores its real cell binding as application +adapter metadata. Its lookup, birth retries, pending slot accounting and physical +inventory must reject another cell binding for the same physical history. This +metadata is not part of the registry key and is removed with the remaining +single-cell runtime consumers. Cell-group genesis and claims use the separate +physical scope described below. + +Cell-independent engine handles now use `GroupLog` and +`ReplicaIdentity`. `create_cell_group` atomically syncs the physical +mapping, initial membership, Raft genesis and an empty route-bound group manifest. +`persist_cells` uses the same owned worker, quotas, batch sync and publication +path as the staged adapter. The engine rejects mixed application domains; old +single-cell enumeration and inventory cannot silently omit cell-group histories. +The mapping's required application tag also refuses the earlier untagged adapter +record shape. No old disk layout is decoded or migrated. + +`RangeDirectory::bootstrap_group_member` admits an initial unmanaged member only +after checking its current birth, permanent node-storage pin, actual synced +genesis and an exact immutable member claim. Claims bind namespace, history, +birth, member, node, disk and engine instance. Lost confirmation returns no permit; +retry uses those same identities. A claim with missing local group state, or a +replacement engine behind the same node identity, refuses. Retired routes and +foreign nodes/catalog owners refuse. Managed namespaces remain closed until their +rooted fleet authorization is integrated. + +The resulting permit is catalog startup evidence, not complete image recovery, +quorum, or application-serving authority. A future shared runtime must verify the +selected manifest closure, revalidate the permit and obtain current-term quorum +before accepting application work. The production host is not yet converted. +Native storage recovery fixtures exercise both image representations through real +cell-independent genesis without a placeholder cell. + +`ReplicaLog::cell_group_inventory` reads every physical group under the engine +catalog lock and short cache reads. It includes retained histories regardless of +current membership, plus an optional initializing history. Pending identities +already in the catalog count once. The existing bounded node-inventory wire carries +these rows without allocating cell names. A domain-separated fixed-width digest +binds cluster, namespace, group and incarnation; `group_history` performs exact +lookup. Legacy cell-scoped lookup cannot match that digest. The node observation +must still match the complete occupied count before encoding or accepting a +response. Inventory is an occupancy observation, not quorum or release proof. +The generic physical capture includes shared histories, while the group-native +accessor refuses a legacy application engine. + + +## Invocation-bounded route confirmation + +`RangeDirectory::routing_session` captures an explicit execution domain, original +monotonic deadline and 1–16 total attempts before discovery. The immutable request +binds logical cell history, command/enrollment/query identity and the dispatcher's +canonical request fingerprint. Initial discovery, redirects and same-group retries +share this budget. A previously resolved route can begin an invocation without +object-store I/O, even during a catalog outage; it is a potentially stale hint. + +A relocation hint is checked against the namespace's configured trusted catalog. +The session observes a pinned directory root and requires an exact lookup match +for the requested cell. It refuses foreign namespaces/schemas, wrong ranges, +non-advancing epochs, unpublished histories and stale attempt tickets. This +confirms the mapping through catalog trust; it does not authenticate a network +connection. The production transport must still authenticate peers, and every +destination must verify the complete route, logical cell incarnation and current +serving authority. No redirect URL or caller-supplied catalog is followed. + +An originating public command may use its local group's observed leader as a +destination hint. An ingress process with a `QueryRouter` placement can probe +its configured trusted group peers for the currently ready leader. These +read-only probes use the namespace command grant, authenticate the exact peer, +and check the selected group, route epoch, birth and cell range. Discovery shares +the original deadline and keeps time for the command. It owns no SQL producer +and marks no mutation submitted. + +The same controller invocation retains its admission, command identity, captured +inputs and original deadline when it sends the mutation to one peer. The +destination independently checks its current route and write authority. Voter +membership grants no write permission. An incoming command always executes locally. + +SDK queries use the local replica if present. On an ingress process with a +trusted group route, automatic selection tries configured members within the +original attempt limit and deadline. All attempts preserve the requested read +consistency and version floor. Callers can choose explicit local execution or +supply member preferences. + +For a managed router, the existing directory resolves group authority through +its isolated reader at the installed fleet root. It samples authenticated quorum +membership and resolves the full voter set through the rooted process views. +Joined voters must match their committed disk bindings. The transport installs +the route by committed membership revision. Discovery remains advisory and +preserves namespace grants and destination checks. + +Managed embedders can call `RaftCells::grant_query_origin` and +`RaftCells::grant_command_origin` after discovery has accepted the peer's actual +boot and endpoints. The host verifies that it owns the namespace. The transport +checks the current peer view and monotonic permission revision. Read and write +grants are independent. `revoke_command_origin` retains a revision tombstone and +seals the old grant while already admitted work remains owned. Reapply these +trusted host permissions when restarting the transport. +The lower-level controller `command` API retains local-only execution; SDK +`dispatch_command` owns the originating routed invocation. + +`CellCommandResponse::result`, `observation`, `token` and `durability` describe +either a local result or authenticated remote output. `local_receipt` returns an +inspected SQL receipt only for the local variant. Remote results retain transport +admission and cannot authorize local effects, local archive settlement or another +forwarding hop. Any possible outbound transmission makes an error ambiguous; +the owner does not automatically retry it or create a new command identity. + +SDK enrollment uses the same leader selection and command grant. One controller +invocation captures the creation ID, initializer inputs and preview before +discovery, then admits one local or remote creation. A destination rechecks +its route and leader authority before entering SQL. `CellEnrollmentResponse` +returns the exact committed creation record and achieved durability. +`local_enrollment` exposes the locally inspected result only when this node +owns that result. An S3 request still requires complete archive coverage at +the destination. + +Attempts are consumed before awaiting reads. Cancellation cannot refund a budget +or make a delayed ticket current. Read timeouts use the original domain clock, +even when a waiter is polled elsewhere. Marking a command potentially submitted +makes subsequent routing failures carry its original ambiguous command ID; a +redirect never proves the earlier proposal uncommitted. Physical command producers +remain the dispatcher's responsibility and must outlive cancelled waiters. + +The directory supplies routing observations, not serving authority. The shared +dispatcher and controller combine those observations with actual quorum, SQL and +transport ownership. The public host cutover remains unfinished. + + +## Dynamic controller registration + +`CellGroupController::register` adds a prepared group to a running node, including +one whose initial roster is empty. The caller supplies a bootstrap permit, +verified selected root, the applier owning that root's manifest page store, and +an authenticated physical peer route. Pending, active and quarantined groups share the group +limit; attachment and unconsumed replies also use the shared item/byte budget. +Admission failure returns the prepared owners. After admission, dropping the +response does not cancel attachment. + +The worker registers the applier before polling runtime attachment, revalidates +the catalog permit through the existing runtime, and activates roster membership +only after success. Current-term quorum is still required for serving. Physical +route registration refuses adopting an existing route; cleanup checks exact route +identity. Failed attachment releases its group slot before publishing a refusal. +Shutdown joins attachment before closing transport, SQL and consensus ownership. + +Group-local apply/runtime failure or an unresolved command failure quarantines +its group. `group_failure` preserves the original cause and `usage` reports active +and quarantined groups separately. A reserved runtime control request revokes +leader authority even while apply is waiting. New group work refuses; pending +apply and ambiguous command owners remain charged through physical shutdown. +Quarantine cannot be cleared by registration. Recovery requires joined restart; +a late persistence completion cannot restore the revoked serving fence. + +Concurrent apply can invalidate a reconciliation observation before SQL inspects +it. This returns `ExecutionError::Uncertain` with the original producer retained. +The owned controller obtains a fresh observation and retries receipt inspection +without executing the callback again. Identity validation precedes this check. + +After receipt reconciliation retires SQL, S3 output fencing is a separate owned +phase. It buffers the exact committed receipt and verified archive proof under +their existing admission. Later group apply can temporarily refuse current +serving authority; the controller waits for progress within the original deadline +and rechecks route, leader and catalog binding. It neither restarts receipt +reconciliation nor executes another callback. A duplicate receipt uses the same +output phase. Cancellation keeps the admitted owner; terminal errors release +output resources without quarantining a group whose SQL producer already closed. + +Shared log-engine I/O/corruption/closure and lost shared SQL worker replies still +close the node. Unknown events and controller ownership violations fail closed. +Initial-member provisioning and canonical startup use the same physical page +inventory under the configured node directory: +`groups/////pages`. Their common path +belongs to `ManifestPageStore`; both take its exclusive lock and recover the +engine-selected root. Independent archive replay uses its own node base with the +same layout. A restart must refuse a missing selected page rather than open a +second empty inventory. Provisioning fault/restart fixtures target this canonical +path for both image backends. No previous flat-path reader or migration is added. + +Provisioning binds immutable `CellGroupProvisionNamespace` profiles to exact +`RangeDirectory` owners and the declared image representation. Canonical startup +clones its node image inventories into these profiles. Duplicate namespaces, +foreign directory owners and mismatched backends refuse; a request cannot select +another image store. Profile caches and fixed peer metadata count once against +the configured preparation-byte total. Every namespace then shares the remaining +preparation bank and parallel worker limit. `provision_configuration_bytes` +reports retained configuration separately from active preparations. + +`activate_managed_initial_group` uses the rooted group birth and actual initial +voter identity. It refreshes catalog authorization and checks the exact synced +boot before submission and again after local attachment. The controller retains +preparation, selected pages, roster and transport ownership when a waiter leaves; +retries reuse prepared owners. Entered physical work is joined even after its +invocation expires. An attached retry revalidates under the same shared budget +without reopening pages or attaching consensus again; it does not adopt another +registration's physical lifecycle. Shared-engine uncertainty still closes the +node. An attachment receipt describes the persisted fleet guard; it supplies no +fresh quorum, seed, serving or physical-release authority. Quorum readiness is +checked independently by the existing Raft owner. + +Managed group repair controls take both the rooted physical history and its +exact admitted permit. They inspect a fresh quorum, preserve the complete plan, +consume/advance only the matching permit and revision, and observe durable apply +behind a new quorum before reporting success. Beginning placement checks the +installed trust revision, original target boot and physical-host separation over +all current occupants, including members omitted from advisory routing. Promotion +still uses actual RawNode catch-up; seed receipts and route discovery cannot +substitute for progress. Lost configuration replies remain unconfirmed. + +After committed configuration changes, managed routing can resolve newly joined +member/disk bindings through rooted node views. Missing discovery leaves known +voters usable; it grants no initialization, vote, quorum or release authority. +Ordinary cell writes do not scan routes. Ordinary frames retry discovery once +only for an unknown destination; checkpoint sends also reconcile before capture. +Advisory refresh preserves the physical registration and checkpoint factory, +while each transfer pins an exact route generation. Old stream generations still +refuse. Per-chunk validation allocates no new endpoint. Explicit replacement or +removal revokes the old factory; an old registration cannot remove its successor. +All work retains the existing peer queues, admission and physical join owners. + +Managed learner activation and full managed canonical startup remain unfinished. +See the delivery package for qualification status. + + +### Retained entry admission + +`ReplicaLogOptions.cell_group_retention` bounds node/group logical retained log +entries plus owned reservations. Its progress sublimits reserve room for empty +Raft election and automatic joint-leave entries; ordinary payloads cannot consume +that space. The charge includes 512 bytes of fixed bookkeeping per entry. These +are logical entry and admission ceilings, not physical engine file byte limits. +`log_retention_usage` reports debt and reservations without granting admission. + +New SQL commands reserve their bounded encoded envelope after duplicate lookup +and before application callbacks. Enrollment reserves before initialization. +Prepared commands retain that lease through submission and shrink it to actual +encoded bytes. Raft owners reserve inbound appends and local proposals before +mutation, including prospective election and automatic configuration entries. +Existing completion and read-index work does not consume ordinary entry admission. +Sealed pre-append rejection or authoritative receipt reconciliation retires an +unused preparation independently of client clones. + +Entered storage retains exact engine/group/entry-bound leases after waiter loss. +After sync and cache/root publication, one ledger operation transfers them into +actual selected entry debt without a release gap or double charge. Suffix +replacement and selected checkpoints update debt from actual retained entries; +archive observations cannot refund it. Reopen reconstructs debt from the selected +log, retaining entries even under a newly lowered admission limit. Mutations +refuse until a reservation fits. No serialized field or per-cell worker is added. + +The source-scoped archive/reclamation and local disaster-recovery qualification +is recorded in the [implementation progress](../../../../../docs/design/multi-raft/cell-groups/progress.md). +Sustained-load fairness and final SDK/fleet cutover have separate task gates. + + +## Per-cell effects on shared owners + +`CellGroupController::configure_effects` installs bounded compiled namespace/class +registrations once. Each `CellGroupEffectRegistration` supplies explicit +`RaftEffectOptions` and a synchronous SQL-options factory, executed on the owning +domain's blocking service. The factory derives private paths from the complete +cell history and shares bounded node resources; its captured configuration must +not grow with enrolled cells. Dispatcher, SQL, engine and namespace schema owners +must match. Configuration enforces private-input and metadata capacity, and each +selected SQL execution verifies class-specific image, receipt and retry bounds. + +One owned turn per node rotates ready groups and due cells. Its group queue is +initialized from the selected in-memory markers and updated only after synced +publication. Empty and future-only groups require no runtime probe or SQLite open. +Outbox summaries retain the minimum time among lane fronts; a blocked later +sequence with an earlier timestamp cannot keep a sleeping lane busy. Hints remain +advisory: the actual ready leader issues cell execution authority and a fresh +linearizable image supplies each private client sequence and due generation. + +Discovery runs on the existing SQL workers. A completed scan transfers its +payload into the effect metadata permit and releases SQL read admission before +staging a command, including when the node/group has one SQL slot. Alarm discovery +retains only its generation ID; its bounded payload stays in SQLite until the +private callback loads it. The private operation and reserved logical cell client +are checked before both new execution and duplicate lookup. Public and remote +commands cannot select the private execution path. The last committed private +receipt alternates alarm/outbox priority without a preference table for every cell. + +An outbox claim crosses ordinary quorum and requested S3 output settlement before +send. After consuming the claim result, the turn releases its response admission +before a loopback receiver or source completion needs the same bounded lane. +Published durability survives that consumer release. The durable Dispatched row, +bounded request metadata and fenced image custody retain the unknown outcome. +Delivery custody requires the same active route, authorized member and leader +term, and a current same-history cell version covering the committed claim. +Later business writes and unrelated apply lag do not revoke that custody. A term, +route, activation or lineage change does. The source rechecks custody before send +and reply acceptance, verifies the configured receiver signature, then replicates +conditional generation-matched completion through the normal durability gate. + +Cancellation does not retire admitted command producers. Closing stops new effect +admission, joins factories/scans/delivery waiters, then closes the existing physical +SQL/runtime owners. Pending callback work must finish before close succeeds; +cancelled external delivery preserves committed debt for retry. Controller usage +reports active turns, completed turns and retries; SQL usage separately reports +retained producers and operations. Root publication can precede those resources' +disposal, so inspection through a one-slot pool waits for their ownership to drain. +Settled materialized producers discard the private SQLite/WAL and LTX scratch +under their existing lock, retaining only its lock inode. Failed opens and revoked +post-open authority also discard scratch before releasing that lock. Cleanup is +bounded and its failure is reported. Sealed committed images, receipts and pending +proposal bytes remain independently owned. Extent workspaces retain their existing +bounded store ownership and joined release. + +The [native effect cases](../../../tests/native_cell_groups_effects.rs) exercise +due discovery among cold cells, exact committed-base reopening after disposable +state is removed, retained SQL/delivery ownership, and joined shutdown. Two +independent physical group bindings compare the exact signed request and receiver +proof while each loses a reply and suppresses duplicate receiver execution. Live +transfer sealing/publication belongs to CG-09. The S3 case requires claim coverage +before send, retains the receiver's S3 policy, and independently restores the final +complete image/receipt closure. It has an explicit local RustFS variant without a +memory fallback. Current results and remaining qualification are recorded in the +[implementation progress](../../../../../docs/design/multi-raft/cell-groups/progress.md). + +Managed peer learner activation enters the same retained provisioning manager as +initial group startup. An exact original-boot permit selects the joining member; +the authenticated seed exchange independently checks that same permit and obtains +a fresh source quorum grant. The manager saves the recovered page/applier owner +before starting installation and retains it after an unknown result. A durable +seed receipt resumes through the permanent joined claim and selected physical +closure; a staged mapping cannot bootstrap genesis. The parallel preparation +permit spans recovery and installation. Attachment remains separate from learner +promotion and application serving. + +The SDK and movement receiver now use this whole-group peer path. S3 learner +activation currently refuses before submission: exact-reference archive transfer +has not yet been integrated. These new managed paths have no executed native or +RustFS qualification while the ordinary library still has legacy consumers to +remove. + +The canonical group host now joins physical cleanup before an explicit startup +error returns and withholds physical routes until all local group roots have +attached. Its cancellation guard retains the same cleanup on the execution +domain. The two existing startup regressions now use real shared groups, engines +and TLS endpoints; their original lock-release and incomplete-inventory traffic +oracles remain. Their execution is not yet qualified. + +Canonical archive startup creates and syncs its node-owned archive spool directory +on the execution domain before registering an archive job. Materialized checkpoint +receivers open anonymous files there under the existing shared scratch admission; +they do not create a missing caller-supplied parent. Missing image data and failed +scratch preparation still refuse S3 admission rather than changing durability. + +A provisioning retry may reuse its retained, completely verified selected root +without scanning or syncing it again. Reuse checks the same live engine and page +owners, exact selected marker, settled publication and complete checkpoint pins. +Pending or quarantined retention refuses; marker equality cannot resolve an +uncertain write. Changed durable selection recovers on the same page owner. + +A peer seed requires a live source grant through the engine's pre-fence admission +check. After actual engine persistence succeeds, local completion checks the +receiving engine/member/disk/process, managed boot, transport shutdown, original +deadline and fresh catalog bootstrap. A later source restart does not revoke that +completed installation. Neither its receipt nor bootstrap establishes fresh +quorum or serving authority; an expired completion is refused even when cold +recovery later finds the installed root. Retained replicas register seed +endpoints when the checkpoint transport pool is configured. The periodic local +checkpoint policy controls checkpoint publication separately. + +Shared linearizable cell reads may be selected on followers after their own safe +ReadIndex round. Their private forwarded context binds the authenticated origin +member, process-local nonce and sequence, separately from membership metadata and +leader-local seed/execution contexts. A follower awaiting a cell or membership +barrier can retransmit the same context within the original term and deadline. +Selection waits for complete local apply through the correlated barrier. SQL rechecks the selected history, live member, +term and source generation; a later term/leader change or shutdown revokes the +retained proof. This changes read availability, not leader-only mutation or seed +admission. SQL results describe the actual selecting replica. + +Recent-source reads use an explicit age bound for the observed current source. +A stale-term message or catalog refresh cannot renew that observation. The selected +proof retains its original source expiry through queued and entered SQL work; +fresh later traffic cannot extend it. SDK stale reads use the injected host source +age by default; disconnected historical reads require explicit selection. Neither +policy supplies a quorum proof. The default request deadline and physical query +retention still apply. Public replica preferences use the same shared controller; +counter/performance entrypoint conversion remains unfinished CG-07 work. + +Public replica preferences now enter the same shared controller invocation as a +local query. One to eight distinct candidates have a bounded per-attempt timeout; +all attempts share the original request, route fingerprint, global deadline and +attempt bank. An explicit replica list may proceed past a local member that +has no callback for the requested query. The receiving replica validates the +operation name, kind and digest against its own registration before admission. +A final fallback may address one untried observed leader. Discovery +and the leader field inside an authenticated query reply remain advisory; a +receiving node independently checks the complete cell/group route and namespace +process grant. Incoming RPCs always execute locally. + +An expired physical query retains both dispatcher and controller admission until +its SQL callback exits. Outbound work retains dispatcher admission through the +physical transport exchange; successful remote results retain their node/peer +reservations and exact authenticated replica identity. A later attempt cannot +renew an earlier read proof or create a private replacement admission bank. +Both image forms pass the focused routing cases and the complete execution +suite; repeated deadline cases retain the original capacities and timeouts. +Consumer ports and the combined public/managed/backend gates remain unfinished. diff --git a/crates/cellar/durability/raft/cell_groups/apply/images.rs b/crates/cellar/durability/raft/cell_groups/apply/images.rs new file mode 100644 index 000000000..c6daab060 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/apply/images.rs @@ -0,0 +1,293 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use crate::application::consensus::{group_protocol::GroupLtx, ImageCommitment, LtxImage}; +use cellar_logic::cell_group::CellHistory; +use cellar_ltx::{ + apply::{ApplyLimits, ImageState, ImageUpdate, LtxSegment}, + generation::{GenerationStore, ImageId, ReadServices}, + internal::{ + CacheLimits, ExtentManifest, ExtentRootReference, Geometry, ImageBinding, ImageRootUpdate, + NodeExtentStore, EXTENT_BYTES, + }, + stream_snapshot::ScratchPool, + TXID, +}; +use sha2::{Digest, Sha256}; +use std::path::PathBuf; + +#[derive(Clone)] +pub enum CellGroupApplyImages { + Materialized { + images: GenerationStore, + services: ReadServices, + }, + Extents { + images: NodeExtentStore, + limits: ApplyLimits, + services: ReadServices, + cache_directory: PathBuf, + cache_pool: ScratchPool, + cache: CacheLimits, + }, +} + +pub(super) struct ImageApply<'a> { + pub cell: &'a CellHistory, + pub sequence: u64, + pub schema: [u8; 32], + pub base: Option<&'a CellImagePin>, + pub image: LtxImage, + pub files: &'a [GroupLtx<'a>], + pub max_effect_rows: u32, +} + +impl CellGroupApplyImages { + pub(crate) fn memory_bound(&self) -> usize { + let limits = match self { + Self::Materialized { images, .. } => images.apply_limits(), + Self::Extents { limits, .. } => *limits, + }; + // Limits validate decoded bytes to at most 256 MiB. Include one extent + // scratch buffer even for materialized mode to keep admission uniform. + limits.max_total_decoded_bytes as usize + EXTENT_BYTES as usize + } + + pub(crate) fn validate(&self) -> Result<(), ExecutionError> { + match self { + Self::Materialized { images, .. } => images.apply_limits().validate().map_err(storage), + Self::Extents { + limits, + cache_directory, + .. + } => { + limits.validate().map_err(storage)?; + if cache_directory.as_os_str().is_empty() { + return Err(ExecutionError::Invalid("apply cache directory")); + } + Ok(()) + } + } + } + + pub(super) fn prepare( + &self, + request: ImageApply<'_>, + verify_existing: bool, + ) -> Result { + let ImageApply { + cell, + sequence, + schema, + base, + image, + files, + max_effect_rows, + } = request; + if image.txid == 0 { + if base.is_some() + || !files.is_empty() + || image.checksum != 0 + || image.commitment != ImageCommitment::Sha256(Sha256::digest([]).into()) + { + return Err(ExecutionError::Invalid("noncanonical empty cell image")); + } + return Ok(EnrollmentImage { + geometry: CellImageGeometry { + length: 0, + page_size: 512, + }, + effects: EffectSummary::default(), + pin: CellImagePin::Empty, + }); + } + let header = cellar_ltx::ltx::Header::parse( + files + .last() + .ok_or(ExecutionError::Invalid("apply LTX missing"))? + .bytes, + ) + .map_err(storage)?; + let geometry = CellImageGeometry { + page_size: header.page_size, + length: u64::from(header.page_size) * u64::from(header.commit), + }; + let segments = files + .iter() + .map(|file| { + Ok(LtxSegment { + bytes: file.bytes, + min_txid: TXID(file.min_txid), + max_txid: TXID(file.max_txid), + page_count: usize::try_from(file.page_count) + .map_err(|_| ExecutionError::Invalid("apply page count"))?, + decoded_bytes: file.decoded_bytes, + }) + }) + .collect::, ExecutionError>>()?; + let pin = match (self, image.commitment) { + (Self::Materialized { images, .. }, ImageCommitment::Sha256(digest)) => { + let base = match base { + Some(CellImagePin::Materialized(pin)) => Some(pin), + Some(CellImagePin::Empty) | None => None, + _ => return Err(ExecutionError::Invalid("materialized apply base")), + }; + let update = ImageUpdate { + base: match base { + Some(pin) => pin.state(), + None => ImageState::empty(image.epoch).map_err(storage)?, + }, + result: ImageState { + epoch: image.epoch, + txid: TXID(image.txid), + checksum: image.checksum, + digest, + page_size: geometry.page_size, + length: geometry.length, + }, + segments, + }; + let id = image_id(cell, sequence, update.result)?; + if verify_existing { + CellImagePin::Materialized( + images + .prepare_verified(id, base, &update) + .map_err(storage)?, + ) + } else { + CellImagePin::Materialized(match images.prepare(id, base, &update) { + Ok(pin) => { + // A fresh result has no earlier durable selector. + // This pin transfers into the complete prepared tree; + // RootRetentions keeps current, checkpoint and + // uncertain publication closures pinned. Do not add + // an independent, permanent host reference per entry. + // Existing files preserve their prior retention below. + images.retire(pin.id()).map_err(storage)?; + pin + } + Err(cellar_ltx::generation::GenerationError::Exists) => { + images.pin_prepared(id, update.result).map_err(storage)? + } + Err(error) => return Err(storage(error)), + }) + } + } + (Self::Extents { images, limits, .. }, ImageCommitment::Extents(result)) => { + let base = match base { + Some(CellImagePin::Extents(pin)) => Some(pin.as_ref()), + Some(CellImagePin::Empty) | None => None, + _ => return Err(ExecutionError::Invalid("extent apply base")), + }; + let binding = ImageBinding { + history: cell_image_history_digest(cell), + schema, + epoch: image.epoch, + txid: image.txid, + checksum: image.checksum, + }; + let empty = ExtentManifest::new( + ImageBinding { + txid: 0, + checksum: 0, + ..binding + }, + Geometry::new(0, 0).map_err(storage)?, + &[], + ) + .map_err(storage)?; + let before = match base { + Some(pin) => pin.reference(), + None => ExtentRootReference::new( + empty.root(), + empty + .encode() + .map_err(storage)? + .len() + .try_into() + .map_err(storage)?, + ) + .map_err(storage)?, + }; + let update = ImageRootUpdate { + base: before, + result, + binding, + geometry: Geometry::new(geometry.page_size, geometry.length) + .map_err(storage)?, + segments, + }; + let prepared = images + .prepare_transient_image_root( + base, + &update, + limits, + &mut vec![0; EXTENT_BYTES as usize], + ) + .map_err(storage)?; + CellImagePin::Extents(Arc::new(prepared.image)) + } + _ => return Err(ExecutionError::Invalid("apply image representation")), + }; + let database = match (self, &pin) { + (Self::Materialized { services, .. }, CellImagePin::Materialized(pin)) => { + crate::storage::open_immutable_generation( + pin.clone(), + crate::storage::ImmutableQueryOptions { + services: services.clone(), + sqlite_vec: false, + }, + ) + } + ( + Self::Extents { + services, + cache_directory, + cache_pool, + cache, + .. + }, + CellImagePin::Extents(pin), + ) => { + let image = pin + .view() + .image(cache_directory.clone(), cache_pool.clone(), *cache) + .map_err(storage)?; + crate::storage::open_immutable_extent_image( + image, + crate::storage::ImmutableQueryOptions { + services: services.clone(), + sqlite_vec: false, + }, + ) + } + _ => return Err(ExecutionError::Invalid("apply image query")), + } + .map_err(storage)?; + let effects = summary::read(database.connection(), max_effect_rows)?; + drop(database); + Ok(EnrollmentImage { + pin, + geometry, + effects, + }) + } +} + +// Names are physical lookup hints. The selected manifest and exact image state +// still have to verify; a filename never supplies publication authority. +pub(in crate::raft::cell_groups) fn image_id( + cell: &CellHistory, + sequence: u64, + image: ImageState, +) -> Result { + let mut id = Sha256::new(); + id.update(b"cellar-cell-apply-image-1\0"); + id.update(cell_image_history_digest(cell)); + for value in [sequence, image.epoch, image.txid.0, image.checksum] { + id.update(value.to_be_bytes()); + } + id.update(image.digest); + let bytes: [u8; 32] = id.finalize().into(); + ImageId::new(bytes[..16].try_into().map_err(storage)?).map_err(storage) +} diff --git a/crates/cellar/durability/raft/cell_groups/apply/mod.rs b/crates/cellar/durability/raft/cell_groups/apply/mod.rs new file mode 100644 index 000000000..f28627b36 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/apply/mod.rs @@ -0,0 +1,232 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Worker-owned physical preparation; only the Raft owner can publish its result. + +pub(super) mod images; +mod recovery; +mod summary; +mod transition; +pub use images::CellGroupApplyImages; + +use super::*; +use crate::application::consensus::group_protocol::GroupEntryLimits; +use crate::raft::{ + cache::GroupLog, + completion::{entry_digest, entry_position}, + execution::{storage, ExecutionError}, +}; +use cellar_logic::cell_group::GroupHistory; +use std::sync::{Arc, Mutex}; + +pub(super) struct Storage { + pub(super) tree: ManifestTree, + pub(super) pages: ManifestPageStore, + pub(super) images: CellGroupApplyImages, + limits: GroupEntryLimits, + max_effect_rows: u32, +} + +impl Storage { + pub(in crate::raft::cell_groups) fn new( + route: cellar_logic::cell_group::GroupRoute, + pages: ManifestPageStore, + images: CellGroupApplyImages, + manifest_limits: ManifestLimits, + limits: GroupEntryLimits, + max_effect_rows: u32, + ) -> Result { + limits.validate().map_err(storage)?; + images.validate()?; + if !(1..=1_000_000).contains(&max_effect_rows) { + return Err(ExecutionError::Invalid("apply effect summary limit")); + } + let tree = ManifestTree::new(route, manifest_limits).map_err(storage)?; + Ok(Self { + tree, + pages, + images, + limits, + max_effect_rows, + }) + } +} + +/// Owns the already opened manifest directory and shared physical image stores. +/// Construction performs no I/O. Opening/recovering the stores belongs on the +/// owning execution domain; transfer the same page owner used for startup. +#[derive(Clone)] +pub struct CellGroupApplier { + pub(super) group: GroupLog, + pub(super) storage: Arc>, +} + +impl CellGroupApplier { + pub fn new( + group: GroupLog, + pages: ManifestPageStore, + images: CellGroupApplyImages, + manifest_limits: ManifestLimits, + limits: GroupEntryLimits, + max_effect_rows: u32, + ) -> Result { + limits.validate().map_err(storage)?; + images.validate()?; + if !(1..=1_000_000).contains(&max_effect_rows) { + return Err(ExecutionError::Invalid("apply effect summary limit")); + } + let marker = group + .group_manifest() + .map_err(storage)? + .ok_or(ExecutionError::Invalid("group manifest missing"))?; + let state = Storage::new( + marker.route().clone(), + pages, + images, + manifest_limits, + limits, + max_effect_rows, + )?; + Ok(Self { + group, + storage: Arc::new(Mutex::new(state)), + }) + } + + /// Capture on the blocking domain: only the exact retained snapshot from + /// this applier's engine and page owner may become a network source. + pub(crate) fn capture_checkpoint( + &self, + pool: &CellGroupCheckpointPool, + snapshot: &::raft::prelude::Snapshot, + ) -> Result { + let root = { + let storage = self + .storage + .lock() + .map_err(|_| CellGroupCheckpointError::Invalid("source page owner"))?; + PreparedGroupManifest::checkpoint_from_store(&self.group, &storage.pages)? + }; + let source = pool.capture_root(&self.group, &root, true)?; + if source.snapshot()? != *snapshot { + return Err(CellGroupCheckpointError::Invalid( + "retained snapshot changed", + )); + } + Ok(source) + } + + /// Advisory metadata from the already published in-memory marker. No page + /// or cell database is opened, and no serving authority is granted. + pub(crate) fn effect_hint( + &self, + ) -> Result<(cellar_logic::consensus::LogIndex, Option), ExecutionError> { + let marker = self + .group + .group_manifest() + .map_err(storage)? + .ok_or(ExecutionError::Invalid("effect group manifest missing"))?; + Ok(( + marker.position().index, + marker.root().next_due_unix_millis(), + )) + } + + pub fn history(&self) -> GroupHistory { + self.group.identity().scope + } + + pub(crate) fn belongs_to_engine(&self, engine: &Arc<()>) -> bool { + Arc::ptr_eq(&self.group.owner, engine) + } + + pub(crate) fn check_startup(&self, root: &PreparedGroupManifest) -> Result<(), ExecutionError> { + let owner = self.storage.try_lock().map_err(|error| match error { + std::sync::TryLockError::WouldBlock => ExecutionError::Busy, + std::sync::TryLockError::Poisoned(_) => { + ExecutionError::Invalid("poisoned registration applier") + } + })?; + if !self.belongs_to_engine(&root.engine_owner) + || self.history() != root.marker().route().history + || !Arc::ptr_eq(&owner.pages.owner, &root.page_owner) + || self.group.group_manifest().map_err(storage)?.as_ref() != Some(root.marker()) + { + return Err(ExecutionError::Invalid( + "registration selected root/page owner", + )); + } + root.require_checkpoint(&self.group).map_err(storage)?; + Ok(()) + } + + pub(crate) fn reservation_bytes(&self, work: &CellGroupApply) -> Result { + self.check(work)?; + let owner = self.storage.try_lock().map_err(|error| match error { + std::sync::TryLockError::WouldBlock => ExecutionError::Busy, + std::sync::TryLockError::Poisoned(_) => { + ExecutionError::Invalid("poisoned group applier") + } + })?; + work.base + .reserved_bytes() + .checked_add(work.entry.data.len()) + .and_then(|n| n.checked_add(owner.images.memory_bound())) + .and_then(|n| n.checked_add(4096)) + .ok_or(ExecutionError::Invalid("apply reservation overflow")) + } + + fn check(&self, work: &CellGroupApply) -> Result<(), ExecutionError> { + if !Arc::ptr_eq(&self.group.owner, &work.base.engine_owner) + || work.base.marker().route().history != self.history() + || work.ticket.position() != entry_position(&work.entry) + || !matches!( + work.entry.get_entry_type(), + ::raft::prelude::EntryType::EntryNormal + | ::raft::prelude::EntryType::EntryConfChangeV2 + ) + { + return Err(ExecutionError::Invalid("group apply identity/entry")); + } + let cache = self.group.read().map_err(storage)?; + if cache.group_manifest.as_ref() != Some(work.base.marker()) + || cache.hard_state.commit < work.entry.index + || cache.entries.get(&work.entry.index) != Some(&work.entry) + || !cache.membership.permits_group(self.group.identity()) + { + return Err(ExecutionError::Invalid("group apply committed base")); + } + drop(cache); + if work.ticket.entry_digest() != entry_digest(&work.entry).map_err(storage)? { + return Err(ExecutionError::Invalid("group apply entry digest")); + } + Ok(()) + } + + /// Called only by the shared blocking SQL worker. The try-lock detects an + /// invalid concurrent owner instead of blocking one worker behind another. + pub(crate) fn prepare( + &self, + work: &CellGroupApply, + ) -> Result { + self.check(work)?; + let mut owner = self.storage.try_lock().map_err(|error| match error { + std::sync::TryLockError::WouldBlock => ExecutionError::Busy, + std::sync::TryLockError::Poisoned(_) => { + ExecutionError::Invalid("poisoned group applier") + } + })?; + let next = if work.entry.get_entry_type() == ::raft::prelude::EntryType::EntryConfChangeV2 { + let cache = self.group.read().map_err(storage)?; + owner.prepare_entry( + &work.base, + &work.entry, + Some((&cache.conf_state, &cache.membership)), + false, + )? + } else { + owner.prepare_entry(&work.base, &work.entry, None, false)? + }; + self.check(work)?; + Ok(next) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/apply/recovery.rs b/crates/cellar/durability/raft/cell_groups/apply/recovery.rs new file mode 100644 index 000000000..8b776b3e0 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/apply/recovery.rs @@ -0,0 +1,177 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Recover only the complete root selected by the persisted engine marker. +use super::*; +use cellar_logic::cell_group::CellImageCommitment; +use cellar_ltx::{ + apply::ImageState, + internal::{ExtentRootReference, ImageBinding, ImageRoot, EXTENT_BYTES}, + TXID, +}; + +impl CellGroupApplier { + /// Verify the persisted selected root and all physical cell images. Run on + /// the owning blocking execution domain, before starting its Raft owner. + /// This grants no quorum or serving authority: startup still consumes the + /// catalog permit and must establish current-term quorum before admission. + /// Missing/corrupt closure refuses; unselected newer files are not recovery. + pub fn recover( + group: GroupLog, + pages: ManifestPageStore, + images: CellGroupApplyImages, + manifest_limits: ManifestLimits, + limits: GroupEntryLimits, + max_effect_rows: u32, + ) -> Result<(Self, PreparedGroupManifest), ExecutionError> { + let applier = Self::new( + group, + pages, + images, + manifest_limits, + limits, + max_effect_rows, + )?; + let prepared = applier.recover_selected()?; + Ok((applier, prepared)) + } + + /// Reconcile the current and retained checkpoint closures using this same + /// physical store owner after an unknown installation result. Run on the + /// owning blocking domain. Never reopen a page directory while its selected + /// roots still hold pins. This grants no catalog, quorum or serving authority. + pub fn recover_selected(&self) -> Result { + let mut owner = self + .storage + .lock() + .map_err(|_| ExecutionError::Invalid("poisoned recovery owner"))?; + Ok(self.recover_roots(&mut owner)?.0) + } + + /// A retry may retain the exact complete closure. Reuse only with the same + /// live page/engine owners and settled selected/checkpoint retention. A + /// changed durable selection is recovered on this owner as before. + pub(in crate::raft::cell_groups) fn recover_or_reuse_selected( + &self, + previous: &PreparedGroupManifest, + ) -> Result { + let mut owner = self + .storage + .lock() + .map_err(|_| ExecutionError::Invalid("poisoned recovery owner"))?; + owner.pages.require_live().map_err(storage)?; + if !self.belongs_to_engine(&previous.engine_owner) + || self.history() != previous.marker().route().history + || !Arc::ptr_eq(&owner.pages.owner, &previous.page_owner) + { + return Err(ExecutionError::Invalid("retained recovery owner")); + } + if self.group.group_manifest().map_err(storage)?.as_ref() == Some(previous.marker()) { + previous.require_selected(&self.group).map_err(storage)?; + if self.group.group_manifest().map_err(storage)?.as_ref() != Some(previous.marker()) { + return Err(ExecutionError::Busy); + } + return Ok(previous.clone()); + } + Ok(self.recover_roots(&mut owner)?.0) + } + + pub(in crate::raft::cell_groups) fn recover_roots( + &self, + owner: &mut Storage, + ) -> Result<(PreparedGroupManifest, Option), ExecutionError> { + let marker = self + .group + .group_manifest() + .map_err(storage)? + .ok_or(ExecutionError::Invalid("recovery manifest missing"))?; + let checkpoint = self.group.checkpoint_manifest().map_err(storage)?; + let Storage { + tree, + pages, + images, + .. + } = owner; + let mut prepare = |marker: &GroupManifest| { + tree.prepare_bound_complete(&self.group, pages, marker.clone(), &mut |leaf| { + images + .recover_image(leaf, marker.route().schema_digest) + .map_err(|error| ManifestError::Io(std::io::Error::other(error))) + }) + .map_err(storage) + }; + let current = prepare(&marker)?; + let retained = checkpoint.as_ref().map(&mut prepare).transpose()?; + if self.group.group_manifest().map_err(storage)?.as_ref() != Some(&marker) + || self.group.checkpoint_manifest().map_err(storage)? != checkpoint + { + return Err(ExecutionError::Invalid("recovery selection changed")); + } + Ok((current, retained)) + } +} + +impl CellGroupApplyImages { + fn recover_image( + &self, + leaf: &CellManifestLeaf, + schema: [u8; 32], + ) -> Result { + let image = leaf.version.image; + let pin = if image.txid == 0 { + CellImagePin::Empty + } else { + match (self, image.commitment) { + (Self::Materialized { images, .. }, CellImageCommitment::Materialized(digest)) => { + let expected = ImageState { + epoch: image.epoch, + txid: TXID(image.txid), + checksum: image.checksum, + digest, + page_size: leaf.geometry.page_size, + length: leaf.geometry.length, + }; + let id = images::image_id( + &leaf.version.history, + leaf.version.sequence.get(), + expected, + )?; + CellImagePin::Materialized(images.pin_prepared(id, expected).map_err(storage)?) + } + ( + Self::Extents { images, .. }, + CellImageCommitment::Extents { + root, + manifest_bytes, + }, + ) => { + let reference = + ExtentRootReference::new(ImageRoot::from_bytes(root), manifest_bytes) + .map_err(storage)?; + let view = images.open_root(reference).map_err(storage)?; + let manifest = view.manifest(); + let expected = ImageBinding { + history: cell_image_history_digest(&leaf.version.history), + schema, + epoch: image.epoch, + txid: image.txid, + checksum: image.checksum, + }; + if manifest.binding() != expected + || manifest.geometry().length() != leaf.geometry.length + || manifest.geometry().page_size() != leaf.geometry.page_size + { + return Err(ExecutionError::Invalid("recovery image binding")); + } + CellImagePin::Extents(Arc::new( + images + .seal_image_root(manifest.clone(), &mut vec![0; EXTENT_BYTES as usize]) + .map_err(storage)?, + )) + } + _ => return Err(ExecutionError::Invalid("recovery image representation")), + } + }; + pin.check(leaf, schema).map_err(storage)?; + Ok(pin) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/apply/summary.rs b/crates/cellar/durability/raft/cell_groups/apply/summary.rs new file mode 100644 index 000000000..00bf3fdb9 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/apply/summary.rs @@ -0,0 +1,97 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use rusqlite::OptionalExtension; + +pub(super) fn read( + connection: &rusqlite::Connection, + max: u32, +) -> Result { + let scan = |table: &str, column: &str| -> Result<(u64, Option), ExecutionError> { + let kind: Option = connection + .query_row( + "SELECT type FROM sqlite_schema WHERE name=?1", + [table], + |row| row.get(0), + ) + .optional() + .map_err(storage)?; + match kind.as_deref() { + None => return Ok((0, None)), + Some("table") => (), + _ => return Err(ExecutionError::Invalid("effect summary schema")), + } + // Only constant internal identifiers reach this function. Limit the scan + // before aggregation; corrupt/unbounded effect state cannot monopolize a worker. + let sql = format!("SELECT COUNT(*), MIN(value), COALESCE(SUM(invalid),0) FROM (SELECT {column} AS value, CASE WHEN typeof({column})='integer' AND {column}>=0 THEN 0 ELSE 1 END AS invalid FROM {table} LIMIT ?1)"); + let (count, earliest, invalid): (u64, Option, u64) = connection + .query_row(&sql, [u64::from(max) + 1], |row| { + Ok((row.get(0)?, row.get(1)?, row.get(2)?)) + }) + .map_err(storage)?; + if count > u64::from(max) || invalid != 0 { + return Err(ExecutionError::Invalid("effect summary bounds/state")); + } + Ok((count, earliest)) + }; + let (_, next_alarm_unix_millis) = scan("__sys_raft_alarms", "due")?; + let (pending_outbox, _) = scan("__sys_raft_outbox", "next_at")?; + let next_outbox_unix_millis = outbox_front(connection, pending_outbox)?; + Ok(EffectSummary { + next_alarm_unix_millis, + next_outbox_unix_millis, + pending_outbox, + // The current inbox is synchronous receipt/deduplication state, not an + // asynchronous work queue. Retained inbox receipts cannot prevent parking. + pending_inbox: 0, + }) +} + +fn outbox_front( + connection: &rusqlite::Connection, + count: u64, +) -> Result, ExecutionError> { + if count == 0 { + return Ok(None); + } + // The preceding scan has already bounded every row and validated its time. + // Read fixed-size metadata only; no payload allocation or unbounded sort. + // Later deliveries cannot run before their lane's oldest sequence, even + // when those later rows have next_at=0. Scheduling that zero spins on an + // otherwise sleeping cell until its dispatched head becomes retryable. + let mut statement = connection.prepare( + "SELECT CASE WHEN typeof(lane)='text' AND length(CAST(lane AS BLOB)) BETWEEN 1 AND 64 THEN lane ELSE NULL END,CASE WHEN typeof(sequence)='blob' AND length(sequence)=8 THEN sequence ELSE NULL END,next_at FROM __sys_raft_outbox LIMIT ?1", + ).map_err(storage)?; + let mut rows = statement.query([count]).map_err(storage)?; + let mut fronts = std::collections::BTreeMap::::new(); + while let Some(row) = rows.next().map_err(storage)? { + let lane: String = row.get(0).map_err(storage)?; + let sequence: Vec = row.get(1).map_err(storage)?; + let sequence = u64::from_be_bytes( + sequence + .try_into() + .map_err(|_| ExecutionError::Invalid("effect outbox sequence"))?, + ); + let next: u64 = row.get(2).map_err(storage)?; + if sequence == 0 || !crate::raft::effects::delivery::valid_name(&lane) { + return Err(ExecutionError::Invalid("effect outbox identity")); + } + if !fronts.contains_key(&lane) && fronts.len() == crate::raft::effects::outbox::MAX_LANES { + return Err(ExecutionError::Invalid("effect outbox lane bounds")); + } + let front = fronts.entry(lane).or_insert((sequence, next)); + if sequence < front.0 { + *front = (sequence, next); + } + } + let next = fronts + .values() + .map(|(_, next)| *next) + .min() + .ok_or(ExecutionError::Invalid("effect outbox row count changed"))?; + Ok(Some(next)) +} + +#[cfg(test)] +#[path = "summary_tests.rs"] +mod tests; diff --git a/crates/cellar/durability/raft/cell_groups/apply/summary_tests.rs b/crates/cellar/durability/raft/cell_groups/apply/summary_tests.rs new file mode 100644 index 000000000..552bcdfc8 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/apply/summary_tests.rs @@ -0,0 +1,61 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; + +#[test] +fn effects_are_bounded_and_inbox_receipts_do_not_schedule_work() { + let db = rusqlite::Connection::open_in_memory().unwrap(); + assert_eq!(read(&db, 2).unwrap(), EffectSummary::default()); + db.execute_batch("CREATE TABLE __sys_raft_inbox(sequence); INSERT INTO __sys_raft_inbox VALUES(1); CREATE TABLE __sys_raft_alarms(due); INSERT INTO __sys_raft_alarms VALUES(20),(0); CREATE TABLE __sys_raft_outbox(lane,sequence,next_at); INSERT INTO __sys_raft_outbox VALUES('A',x'0000000000000001',30),('B',x'0000000000000001',10);").unwrap(); + assert_eq!( + read(&db, 2).unwrap(), + EffectSummary { + next_alarm_unix_millis: Some(0), + next_outbox_unix_millis: Some(10), + pending_outbox: 2, + pending_inbox: 0, + } + ); + assert!(read(&db, 1).is_err()); +} + +#[test] +fn blocked_later_delivery_does_not_wake_a_sleeping_lane() { + let db = rusqlite::Connection::open_in_memory().unwrap(); + db.execute_batch( + "CREATE TABLE __sys_raft_outbox(lane,sequence,next_at); + INSERT INTO __sys_raft_outbox VALUES + ('Events',x'0000000000000002',0), + ('Events',x'0000000000000001',10000), + ('Alerts',x'0000000000000001',300);", + ) + .unwrap(); + let summary = read(&db, 3).unwrap(); + assert_eq!(summary.pending_outbox, 3); + assert_eq!(summary.next_outbox_unix_millis, Some(300)); + db.execute("DELETE FROM __sys_raft_outbox WHERE lane='Alerts'", []) + .unwrap(); + assert_eq!(read(&db, 3).unwrap().next_outbox_unix_millis, Some(10000)); + db.execute( + "DELETE FROM __sys_raft_outbox WHERE sequence=x'0000000000000001'", + [], + ) + .unwrap(); + assert_eq!(read(&db, 3).unwrap().next_outbox_unix_millis, Some(0)); +} + +#[test] +fn malformed_effect_state_refuses_instead_of_reporting_idle() { + for ddl in [ + "CREATE VIEW __sys_raft_alarms AS SELECT 10 AS due", + "CREATE TABLE __sys_raft_alarms(other)", + "CREATE TABLE __sys_raft_alarms(due); INSERT INTO __sys_raft_alarms VALUES(-1)", + "CREATE TABLE __sys_raft_outbox(next_at); INSERT INTO __sys_raft_outbox VALUES(NULL)", + "CREATE TABLE __sys_raft_outbox(next_at); INSERT INTO __sys_raft_outbox VALUES('10')", + "CREATE TABLE __sys_raft_outbox(next_at); INSERT INTO __sys_raft_outbox VALUES(1.5)", + ] { + let db = rusqlite::Connection::open_in_memory().unwrap(); + db.execute_batch(ddl).unwrap(); + assert!(read(&db, 100).is_err(), "{ddl}"); + } +} diff --git a/crates/cellar/durability/raft/cell_groups/apply/transition.rs b/crates/cellar/durability/raft/cell_groups/apply/transition.rs new file mode 100644 index 000000000..0dd141b60 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/apply/transition.rs @@ -0,0 +1,199 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Physical transitions shared by live committed apply and archive data replay. +//! This module supplies preparation, never a persistence ticket or serving right. +use super::*; +use crate::application::consensus::group_protocol::{CellGroupEntry, GroupCommand}; +use crate::raft::member_state::GroupMembership; +use cellar_logic::cell_group::CommitWitness; +use raft::prelude::{ConfState, Entry}; +use sha2::{Digest, Sha256}; + +impl Storage { + pub(in crate::raft::cell_groups) fn prepare_entry( + &mut self, + base: &PreparedGroupManifest, + entry: &Entry, + configuration: Option<(&ConfState, &GroupMembership)>, + verify_existing: bool, + ) -> Result { + let Storage { + tree, + pages, + images, + limits, + max_effect_rows, + } = self; + if entry.get_entry_type() == ::raft::prelude::EntryType::EntryConfChangeV2 { + let (conf, membership) = configuration.ok_or(ExecutionError::Invalid( + "configuration replay state missing", + ))?; + return tree + .prepare_membership(pages, base, entry, conf, membership) + .map_err(storage); + } + // Check page owner and contiguous prefix before any image allocation/I/O. + let unchanged = tree + .prepare_unchanged(pages, base, entry_position(entry)) + .map_err(storage)?; + if entry.data.is_empty() { + return Ok(unchanged); + } + let decoded = + CellGroupEntry::decode(&entry.data, base.marker().route(), *limits).map_err(storage)?; + if let GroupCommand::Activate { + transition, + publication, + manifest, + } = decoded.command + { + if decoded.binding.proposal.bytes() != transition.bytes() { + return Err(ExecutionError::Invalid("activation proposal binding")); + } + let mut next = unchanged; + next.marker + .activate(transition, publication, manifest) + .map_err(storage)?; + return Ok(next); + } + if let GroupCommand::Seal { + transition, + authorization, + .. + } = decoded.command + { + if decoded.binding.proposal.bytes() != transition.bytes() { + return Err(ExecutionError::Invalid("source seal proposal binding")); + } + let mut next = unchanged; + next.marker + .seal_source( + transition, + authorization, + Sha256::digest(&entry.data).into(), + ) + .map_err(storage)?; + return Ok(next); + } + if let GroupCommand::Retire { + transition, + publication, + } = decoded.command + { + if decoded.binding.proposal.bytes() != transition.bytes() { + return Err(ExecutionError::Invalid( + "source retirement proposal binding", + )); + } + let mut next = unchanged; + next.marker + .retire_source(transition, publication) + .map_err(storage)?; + return Ok(next); + } + if let GroupCommand::PreparePark { + generation, + term, + membership, + } = decoded.command + { + let mut next = unchanged; + next.marker + .prepare_park(generation, term, membership) + .map_err(storage)?; + return Ok(next); + } + if !base.marker().is_active() { + return Err(ExecutionError::Invalid("group is not active")); + } + let witness = CommitWitness { + group: base.marker().route().history, + position: entry_position(entry), + proposal: decoded.binding.proposal, + }; + let next = match decoded.command { + GroupCommand::Enroll { cell, image, files } => { + let decision = decide_enrollment( + tree.route(), + EnrollmentPhase::Active, + &entry.data, + base.pin_cell(&cell.cell).map(|(leaf, _)| &leaf.enrollment), + *limits, + ); + match decision { + Ok(EnrollmentDecision::Existing(_)) | Err(EnrollmentError::Conflict) => { + unchanged + } + Ok(EnrollmentDecision::Propose(record)) => { + let prepared = images.prepare( + images::ImageApply { + cell: &cell, + sequence: 1, + schema: tree.route().schema_digest, + base: None, + image, + files: &files, + max_effect_rows: *max_effect_rows, + }, + verify_existing, + )?; + let leaf = CellManifestLeaf { + enrollment: record, + version: cellar_logic::cell_group::CellVersion { + history: cell, + sequence: cellar_logic::cell_group::MutationSequence::new(1) + .map_err(storage)?, + image: image.into(), + }, + geometry: prepared.geometry, + last_mutation: entry_position(entry), + witness, + effects: prepared.effects, + }; + tree.prepare_update(pages, base, leaf, None, prepared.pin) + .map_err(storage)? + } + Err(error) => return Err(storage(error)), + } + } + GroupCommand::Mutation(mutation) => { + let (old, image_base) = base + .pin_cell(&mutation.cell.cell) + .ok_or(ExecutionError::Invalid("apply cell missing"))?; + mutation.check_base(&old.version).map_err(storage)?; + let prepared = images.prepare( + images::ImageApply { + cell: &mutation.cell, + sequence: mutation.sequence.get(), + schema: tree.route().schema_digest, + base: Some(&image_base), + image: mutation.result, + files: &mutation.files, + max_effect_rows: *max_effect_rows, + }, + verify_existing, + )?; + let leaf = CellManifestLeaf { + enrollment: old.enrollment.clone(), + version: cellar_logic::cell_group::CellVersion { + history: mutation.cell, + sequence: mutation.sequence, + image: mutation.result.into(), + }, + geometry: prepared.geometry, + last_mutation: entry_position(entry), + witness, + effects: prepared.effects, + }; + tree.prepare_update(pages, base, leaf, Some(&old.version), prepared.pin) + .map_err(storage)? + } + _ => { + return Err(ExecutionError::Invalid( + "unsupported group apply administration", + )) + } + }; + Ok(next) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/archive/checkpoint.rs b/crates/cellar/durability/raft/cell_groups/archive/checkpoint.rs new file mode 100644 index 000000000..0e99eec39 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/archive/checkpoint.rs @@ -0,0 +1,374 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::{objects::Object, *}; +use std::io::Write; + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Index { + domain: String, + pub marker: GroupManifest, + bytes: u64, + pub chunks: Vec, +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Chunk { + part: wire::Part, + offset: u64, + length: u64, + pub object: Object, +} + +impl Index { + fn validate(&self, archive: &CellGroupArchive) -> Result<()> { + self.marker + .check() + .map_err(CellGroupCheckpointError::from)?; + if self.domain != DOMAIN || self.marker.route().history != archive.history { + return Err(CellGroupArchiveError::Invalid("checkpoint domain/history")); + } + if self.chunks.is_empty() + || self.chunks.len() > archive.limits.max_chunks + || self.bytes > archive.limits.max_checkpoint_bytes + { + return Err(CellGroupArchiveError::Limit); + } + let mut total = 0u64; + let mut previous: Option<&Chunk> = None; + for chunk in &self.chunks { + if chunk.object.length == 0 + || chunk.object.length > CHUNK_BYTES as u64 + || chunk.length == 0 + || chunk.length > archive.limits.max_checkpoint_bytes + { + return Err(CellGroupArchiveError::Limit); + } + chunk + .offset + .checked_add(chunk.object.length) + .filter(|n| *n <= chunk.length) + .ok_or(CellGroupArchiveError::Invalid("chunk bounds"))?; + if let Some(before) = previous { + let before_end = before.offset + before.object.length; + if before_end < before.length { + if chunk.part != before.part + || chunk.length != before.length + || chunk.offset != before_end + { + return Err(CellGroupArchiveError::Invalid("chunk continuity")); + } + } else if chunk.offset != 0 { + return Err(CellGroupArchiveError::Invalid("part offset")); + } + } else if chunk.part != wire::Part::Header || chunk.offset != 0 { + return Err(CellGroupArchiveError::Invalid("first archive part")); + } + total = total + .checked_add(chunk.object.length) + .ok_or(CellGroupArchiveError::Limit)?; + previous = Some(chunk); + } + let last = previous.ok_or(CellGroupArchiveError::Invalid("empty archive"))?; + if total != self.bytes || last.offset + last.object.length != last.length { + return Err(CellGroupArchiveError::Invalid("incomplete archive index")); + } + Ok(()) + } +} + +pub(super) struct Bounded { + pub(super) bytes: Vec, + pub(super) limit: usize, +} + +impl Write for Bounded { + fn write(&mut self, bytes: &[u8]) -> std::io::Result { + if self + .bytes + .len() + .checked_add(bytes.len()) + .is_none_or(|n| n > self.limit) + { + return Err(std::io::Error::other("archive index limit")); + } + self.bytes.extend_from_slice(bytes); + Ok(bytes.len()) + } + + fn flush(&mut self) -> std::io::Result<()> { + Ok(()) + } +} + +impl CellGroupArchiveUpload { + /// Fetch the exact checkpoint protected by this still-pending upload and + /// its inherited parent graph. This streams data only: the caller must use + /// a physical receiver and a separately authenticated live install grant. + pub(crate) async fn download_seed_checkpoint( + &self, + domain: &impl CellGroupArchiveExecution, + reference: &CellGroupArchiveReference, + mut sink: F, + ) -> Result<()> + where + F: FnMut(wire::Chunk) -> Fut + Send, + Fut: std::future::Future> + Send, + { + domain + .run(async { + let permit = self.admit()?; + self.registry().pending(&self.ticket()).await?; + self.allows_object(super::graph::Kind::Indexes, &reference.object())?; + let index = self + .checkpoint_index(domain, reference, permit.clone()) + .await?; + for chunk in index.chunks { + self.registry().pending(&self.ticket()).await?; + self.allows_object(super::graph::Kind::Chunks, &chunk.object)?; + let bytes = self.get("chunks", &chunk.object, CHUNK_BYTES).await?; + sink(wire::Chunk { + part: chunk.part, + offset: chunk.offset, + length: chunk.length, + bytes, + }) + .await?; + } + self.registry().pending(&self.ticket()).await?; + Ok(()) + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } + + /// Capture is supplied by the caller's node checkpoint pool. The bounded + /// producer keeps the full source pinned through physical reads. Losing the + /// async caller closes its channel; domain shutdown joins the physical owner. + /// Success confirms immutable bytes, without granting group coverage. + pub async fn upload_checkpoint( + &self, + domain: &impl CellGroupArchiveExecution, + source: CellGroupCheckpoint, + ) -> Result { + domain + .run(self.upload(domain, source)) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } + + async fn upload( + &self, + domain: &impl CellGroupArchiveExecution, + mut source: CellGroupCheckpoint, + ) -> Result { + if source.marker().route().history != self.history { + return Err(CellGroupArchiveError::Invalid("source history")); + } + let permit = self.admit()?; + self.registry().pending(&self.ticket()).await?; + let retained = permit.clone(); + let mut index = Index { + domain: DOMAIN.into(), + marker: source.marker().clone(), + bytes: 0, + chunks: Vec::new(), + }; + let (sender, mut receiver) = tokio::sync::mpsc::channel(1); + let producer = domain + .blocking(move || { + let _retained = retained; + source.stream(|chunk| { + if chunk.bytes.len() > CHUNK_BYTES { + return Err(CellGroupCheckpointError::Limit); + } + sender.blocking_send(wire::Chunk::copy(chunk)).map_err(|_| { + CellGroupCheckpointError::Sink(std::io::Error::other("archive stopped")) + }) + }) + }) + .map_err(|_| CellGroupArchiveError::Stopped)?; + let upload = async { + while let Some(chunk) = receiver.recv().await { + index.bytes = index + .bytes + .checked_add(chunk.bytes.len() as u64) + .filter(|n| *n <= self.limits.max_checkpoint_bytes) + .ok_or(CellGroupArchiveError::Limit)?; + if index.chunks.len() >= self.limits.max_chunks { + return Err(CellGroupArchiveError::Limit); + } + let object = self.put("chunks", chunk.bytes, CHUNK_BYTES).await?; + index.chunks.push(Chunk { + part: chunk.part, + offset: chunk.offset, + length: chunk.length, + object, + }); + } + Ok(()) + } + .await; + drop(receiver); // Release a blocked producer before joining any error. + let produced = producer.await.map_err(CellGroupArchiveError::Worker)?; + upload?; + produced?; + let archive = self.clone(); + let retained = permit.clone(); + let bytes = domain + .blocking(move || { + let _retained = retained; + index.validate(&archive)?; + let mut output = Bounded { + bytes: Vec::new(), + limit: archive.limits.max_index_bytes, + }; + serde_json::to_writer(&mut output, &index) + .map_err(|_| CellGroupArchiveError::Limit)?; + Ok::<_, CellGroupArchiveError>(output.bytes) + }) + .map_err(|_| CellGroupArchiveError::Stopped)? + .await + .map_err(CellGroupArchiveError::Worker)??; + let object = self + .put("indexes", bytes, self.limits.max_index_bytes) + .await?; + drop(permit); + Ok(CellGroupArchiveReference { + registry: object.registry, + generation: object.generation, + digest: object.digest, + length: object.length, + }) + } +} + +impl CellGroupArchive { + /// Independently GET every object into the ordinary physical receiver. No + /// source pin or upload receipt substitutes for dependency verification. + /// A missing/corrupt cell refuses the whole cut. Ordinary errors join the + /// receiver; cancellation retains its permits until physical work finishes. + pub async fn verify_checkpoint( + &self, + domain: &impl CellGroupArchiveExecution, + reference: &CellGroupArchiveReference, + receiver: CellGroupArchiveCheckpointReceiver, + ) -> Result { + domain + .run(self.verify(domain, reference, receiver)) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } + + async fn verify( + &self, + domain: &impl CellGroupArchiveExecution, + reference: &CellGroupArchiveReference, + receiver: CellGroupArchiveCheckpointReceiver, + ) -> Result { + let permit = self.admit()?; + self.verify_admitted(domain, reference, receiver, permit) + .await + } + + pub(super) async fn checkpoint_marker( + &self, + domain: &impl CellGroupArchiveExecution, + reference: &CellGroupArchiveReference, + permit: Arc, + ) -> Result { + Ok(self + .checkpoint_index(domain, reference, permit) + .await? + .marker) + } + + pub(super) async fn checkpoint_index( + &self, + domain: &impl CellGroupArchiveExecution, + reference: &CellGroupArchiveReference, + permit: Arc, + ) -> Result { + let bytes = self + .get( + "indexes", + &Object { + registry: reference.registry, + generation: reference.generation, + digest: reference.digest, + length: reference.length, + }, + self.limits.max_index_bytes, + ) + .await?; + let archive = self.clone(); + domain + .blocking(move || { + let _permit = permit; + let index: Index = serde_json::from_slice(&bytes) + .map_err(|_| CellGroupArchiveError::Invalid("checkpoint index"))?; + index.validate(&archive)?; + Ok::<_, CellGroupArchiveError>(index) + }) + .map_err(|_| CellGroupArchiveError::Stopped)? + .await + .map_err(CellGroupArchiveError::Worker)? + } + + pub(super) async fn verify_admitted( + &self, + domain: &impl CellGroupArchiveExecution, + reference: &CellGroupArchiveReference, + mut receiver: CellGroupArchiveCheckpointReceiver, + permit: Arc, + ) -> Result { + let index = self + .checkpoint_index(domain, reference, permit.clone()) + .await?; + let marker = index.marker; + let (sender, mut incoming) = tokio::sync::mpsc::channel::(1); + let retained = permit.clone(); + let consumer = domain + .blocking(move || { + let _retained = retained; + while let Some(chunk) = incoming.blocking_recv() { + chunk.append(&mut receiver)?; + } + receiver.finish() + }) + .map_err(|_| CellGroupArchiveError::Stopped)?; + let download: Result<()> = async { + for chunk in index.chunks { + let bytes = self.get("chunks", &chunk.object, CHUNK_BYTES).await?; + if sender + .send(wire::Chunk { + part: chunk.part, + offset: chunk.offset, + length: chunk.length, + bytes, + }) + .await + .is_err() + { + // The consumer owns the checkpoint failure. Join it below + // so a rejected chunk keeps its typed cause. + break; + } + } + Ok(()) + } + .await; + drop(sender); + let received = consumer.await.map_err(CellGroupArchiveError::Worker)?; + download?; + let checkpoint = received?; + if checkpoint.marker() != &marker { + return Err(CellGroupArchiveError::Invalid("archive checkpoint binding")); + } + Ok(VerifiedCellGroupArchiveCheckpoint { + reference: reference.clone(), + checkpoint, + }) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/archive/collection.rs b/crates/cellar/durability/raft/cell_groups/archive/collection.rs new file mode 100644 index 000000000..e6ab75f15 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/archive/collection.rs @@ -0,0 +1,189 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Complete marks precede deletion. Revision watermarks, protected parents and +//! never-reused generation keys make concurrent publication/delayed DELETE safe. +use super::{ + graph::{Graph, Key, Kind}, + *, +}; +use futures_util::TryStreamExt; + +#[derive(Clone, Copy, Debug)] +pub struct CellGroupArchiveCollectionLimits { + pub max_scan_objects: usize, + pub max_delete_objects: usize, +} + +impl Default for CellGroupArchiveCollectionLimits { + fn default() -> Self { + Self { + max_scan_objects: 4096, + max_delete_objects: 256, + } + } +} + +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub struct CellGroupArchiveCollectionReport { + pub revision: u64, + pub protected_objects: usize, + pub scanned_objects: usize, + pub deleted_objects: usize, + pub deleted_bytes: u64, +} + +fn parse(path: &str, prefix: &Path, registry: [u8; 32]) -> Result { + let suffix = path + .strip_prefix(prefix.as_ref()) + .and_then(|s| s.strip_prefix('/')) + .ok_or(CellGroupArchiveError::Invalid( + "collection object namespace", + ))?; + let mut parts = suffix.split('/'); + let generation = parts + .next() + .ok_or(CellGroupArchiveError::Invalid("collection generation"))?; + let parsed: u64 = generation + .parse() + .map_err(|_| CellGroupArchiveError::Invalid("collection generation"))?; + if parsed < 2 || parsed == u64::MAX || parsed.to_string() != generation { + return Err(CellGroupArchiveError::Invalid("collection generation")); + } + let kind = parts + .next() + .and_then(Kind::parse) + .ok_or(CellGroupArchiveError::Invalid("collection object kind"))?; + let (length, digest) = parts + .next() + .and_then(|p| p.split_once('-')) + .ok_or(CellGroupArchiveError::Invalid("collection object identity"))?; + let bytes: u64 = length + .parse() + .map_err(|_| CellGroupArchiveError::Invalid("collection object length"))?; + if bytes == 0 || bytes.to_string() != length || digest.len() != 64 || parts.next().is_some() { + return Err(CellGroupArchiveError::Invalid("collection object identity")); + } + let mut hash = [0; 32]; + hex::decode_to_slice(digest, &mut hash) + .map_err(|_| CellGroupArchiveError::Invalid("collection digest"))?; + if hex::encode(hash) != digest { + return Err(CellGroupArchiveError::Invalid("collection digest encoding")); + } + Ok(Key { + kind, + object: super::objects::Object { + registry, + generation: parsed, + length: bytes, + digest: hash, + }, + }) +} + +impl CellGroupArchiveRegistry { + /// Mark selected roots, explicit holds and pending parents; retain entire + /// pending/new generations. Verify every marked object before any DELETE. + /// No unknown owner expires by time. Cancellation may leave partial garbage, + /// but cannot permit a deleted key to be rebound by a later allocation. + pub async fn collect( + &self, + domain: &impl CellGroupArchiveExecution, + owners: &CellGroupArchiveUploadPool, + tails: &CellGroupArchiveTailPool, + limits: CellGroupArchiveCollectionLimits, + ) -> Result { + if !(1..=65_536).contains(&limits.max_scan_objects) + || limits.max_delete_objects == 0 + || limits.max_delete_objects > limits.max_scan_objects + || limits.max_delete_objects > owners.limits.max_objects + { + return Err(CellGroupArchiveError::Limit); + } + domain + .run(async { + let retained = owners.admit()?; + // Hashing entry objects needs the same bounded byte windows as tail + // verification. This lease does not construct a physical replay. + let _tail = tails.admit()?; + let io = self.archive.admit()?; + let protection = self.protection().await?; + let mut marked = Graph::new(owners.limits.max_objects); + for root in &protection.roots { + let closure = Graph::load( + &self.archive, + domain, + root, + tails.limits, + io.clone(), + retained.clone(), + owners.limits.max_objects, + ) + .await?; + for key in closure.objects { + marked.insert(key.kind, key.object, u64::MAX - 1)?; + } + } + marked.verify(&self.archive, tails.limits).await?; + let mut report = CellGroupArchiveCollectionReport { + revision: protection.revision, + protected_objects: marked.objects.len(), + ..Default::default() + }; + let prefix = self + .archive + .prefix + .child("registries") + .child(hex::encode(self.id())) + .child("generations"); + let mut objects = self.archive.store.list(Some(&prefix)); + let mut candidates = Vec::new(); + self.archive + .io(async { + while report.scanned_objects < limits.max_scan_objects { + let Some(object) = objects + .try_next() + .await + .map_err(CellGroupArchiveError::Storage)? + else { + break; + }; + report.scanned_objects += 1; + let key = parse(object.location.as_ref(), &prefix, self.id())?; + if key.object.generation <= protection.revision + && !protection + .pending_generations + .contains(&key.object.generation) + && !marked.objects.contains(&key) + && candidates.len() < limits.max_delete_objects + { + candidates.push((key, object.size)); + } + } + Ok(()) + }) + .await?; + // New reservations can inherit only roots protected by this mark; + // their own generation is newer than its watermark. Released or + // aborted allocations cannot resurrect an unmarked old key. + for (key, bytes) in candidates { + let path = self.archive.object_path(key.kind.name(), &key.object); + self.archive + .io(async { + match self.archive.store.delete(&path).await { + Ok(()) | Err(object_store::Error::NotFound { .. }) => Ok(()), + Err(error) => Err(CellGroupArchiveError::Storage(error)), + } + }) + .await?; + report.deleted_objects += 1; + report.deleted_bytes = report + .deleted_bytes + .checked_add(bytes) + .ok_or(CellGroupArchiveError::Limit)?; + } + Ok(report) + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } +} diff --git a/crates/cellar/durability/raft/cell_groups/archive/data_store.rs b/crates/cellar/durability/raft/cell_groups/archive/data_store.rs new file mode 100644 index 000000000..d9d88a01e --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/archive/data_store.rs @@ -0,0 +1,62 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Private physical preparation, without a Raft mapping or voter identity. +use super::super::{apply::Storage, root::ArchiveManifestOwner, *}; +use crate::application::consensus::group_protocol::GroupEntryLimits; +use crate::raft::execution::ExecutionError; +use cellar_logic::cell_group::GroupRoute; +use std::sync::{Arc, Mutex}; + +/// Owns archive verification pages/images for one expected route. Open dedicated +/// physical directories on the owning blocking domain, separately from serving +/// stores. No ReplicaLog, member claim, +/// bootstrap permission or application callback is created or required. +#[derive(Clone)] +pub struct CellGroupArchiveStore { + route: GroupRoute, + pub(in crate::raft::cell_groups) storage: Arc>, + pub(in crate::raft::cell_groups) owner: Arc, +} + +impl CellGroupArchiveStore { + pub fn new( + route: GroupRoute, + pages: ManifestPageStore, + images: CellGroupApplyImages, + manifest_limits: ManifestLimits, + entry_limits: GroupEntryLimits, + max_effect_rows: u32, + ) -> Result { + let state = Storage::new( + route.clone(), + pages, + images, + manifest_limits, + entry_limits, + max_effect_rows, + )?; + Ok(Self { + route, + storage: Arc::new(Mutex::new(state)), + owner: Arc::new(ArchiveManifestOwner::new()), + }) + } + + pub fn route(&self) -> &GroupRoute { + &self.route + } + + /// Refuse further receives/replay. Entered blocking work keeps its physical + /// pins/admission until it exits; its execution domain owns the join. + pub fn close(&self) { + self.owner.close(); + } + + pub(in crate::raft::cell_groups) fn require_live( + &self, + ) -> Result<(), CellGroupCheckpointError> { + self.owner + .require_live() + .map_err(CellGroupCheckpointError::from) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/archive/enrollment.rs b/crates/cellar/durability/raft/cell_groups/archive/enrollment.rs new file mode 100644 index 000000000..289fce993 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/archive/enrollment.rs @@ -0,0 +1,33 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Original creation provenance in a complete independently recovered group. +use super::*; +use crate::raft::cell_groups::EnrollmentRecord; + +pub(in crate::raft::cell_groups) fn verify( + coverage: &CellGroupArchiveCoverage, + store: &CellGroupArchiveStore, + expected: &EnrollmentRecord, +) -> Result { + store.require_live()?; + let root = coverage.recovery().manifest(); + if root.marker().route() != store.route() || !store.owner.owns(root) { + return Err(CellGroupArchiveError::Invalid( + "S3 enrollment verifier owner/route", + )); + } + let Some((leaf, _)) = root.pin_cell(&expected.cell().cell) else { + // An older complete cut can precede this committed creation. + return Ok(false); + }; + if &leaf.enrollment != expected { + return Err(CellGroupArchiveError::Invalid( + "S3 enrollment creation record", + )); + } + // The full coverage already verified every cell image. The immutable + // creation record, including its initial commitment, survives later images; + // retaining superseded initial image bytes is not required. + store.require_live()?; + Ok(true) +} diff --git a/crates/cellar/durability/raft/cell_groups/archive/execution.rs b/crates/cellar/durability/raft/cell_groups/archive/execution.rs new file mode 100644 index 000000000..ebffe72b8 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/archive/execution.rs @@ -0,0 +1,141 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Scheduling capability without ownership of execution-domain shutdown. +use crate::asyncrt::{DomainClosed, DomainToken, ExecutionDomain, TaskHandle}; +use std::future::Future; +use std::sync::{ + atomic::{AtomicUsize, Ordering}, + Arc, +}; +use tokio::sync::Notify; + +mod sealed { + pub trait Sealed {} + + impl Sealed for super::ExecutionDomain {} + + impl Sealed for super::DomainToken {} + + impl Sealed for super::ControllerExecution {} +} + +#[derive(Default)] +struct Physical { + active: AtomicUsize, + finished: Notify, +} + +struct Entered(Arc); + +impl Drop for Entered { + fn drop(&mut self) { + self.0.active.fetch_sub(1, Ordering::AcqRel); + self.0.finished.notify_waiters(); + } +} + +/// One controller's physical joins, including closures whose async waiter has +/// already failed or disappeared. Does not join unrelated execution-domain work. +#[derive(Clone)] +pub(crate) struct ControllerExecution { + domain: DomainToken, + physical: Arc, +} + +impl ControllerExecution { + pub(crate) fn new(domain: DomainToken) -> Self { + Self { + domain, + physical: Arc::new(Physical::default()), + } + } + + pub(crate) async fn join(&self) { + loop { + let finished = self.physical.finished.notified(); + tokio::pin!(finished); + finished.as_mut().enable(); + if self.physical.active.load(Ordering::Acquire) == 0 { + return; + } + finished.await; + } + } +} + +impl CellGroupArchiveExecution for ControllerExecution { + fn run(&self, future: F) -> impl Future> + Send + where + F: Future + Send, + F::Output: Send, + { + self.domain.run(future) + } + + fn blocking(&self, operation: F) -> Result, DomainClosed> + where + T: Send + 'static, + F: FnOnce() -> T + Send + 'static, + { + self.physical.active.fetch_add(1, Ordering::AcqRel); + let entered = Entered(self.physical.clone()); + self.domain.blocking(move || { + let _entered = entered; + operation() + }) + } +} + +/// Archive I/O runs on its owning execution domain. Sealed implementations let +/// node controllers use their captured token without acquiring shutdown rights. +pub trait CellGroupArchiveExecution: sealed::Sealed + Send + Sync { + fn run(&self, future: F) -> impl Future> + Send + where + F: Future + Send, + F::Output: Send; + + fn blocking(&self, operation: F) -> Result, DomainClosed> + where + T: Send + 'static, + F: FnOnce() -> T + Send + 'static; +} + +impl CellGroupArchiveExecution for ExecutionDomain { + fn run(&self, future: F) -> impl Future> + Send + where + F: Future + Send, + F::Output: Send, + { + ExecutionDomain::run(self, future) + } + + fn blocking(&self, operation: F) -> Result, DomainClosed> + where + T: Send + 'static, + F: FnOnce() -> T + Send + 'static, + { + ExecutionDomain::blocking(self, operation) + } +} + +impl CellGroupArchiveExecution for DomainToken { + fn run(&self, future: F) -> impl Future> + Send + where + F: Future + Send, + F::Output: Send, + { + DomainToken::run(self, future) + } + + fn blocking(&self, operation: F) -> Result, DomainClosed> + where + T: Send + 'static, + F: FnOnce() -> T + Send + 'static, + { + DomainToken::blocking(self, operation) + } +} + +#[cfg(test)] +#[path = "execution_tests.rs"] +mod tests; diff --git a/crates/cellar/durability/raft/cell_groups/archive/execution_tests.rs b/crates/cellar/durability/raft/cell_groups/archive/execution_tests.rs new file mode 100644 index 000000000..910a77aca --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/archive/execution_tests.rs @@ -0,0 +1,32 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[allow(clippy::disallowed_methods)] +async fn dropped_async_waiter_still_requires_physical_join() { + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let execution = ControllerExecution::new(domain.services().domain_token().unwrap()); + let (entered, observed) = tokio::sync::oneshot::channel(); + let (release, blocked) = std::sync::mpsc::sync_channel(1); + let task = execution + .blocking(move || { + entered.send(()).unwrap(); + blocked.recv().unwrap(); + }) + .unwrap(); + observed.await.unwrap(); + drop(task); + let join = execution.join(); + tokio::pin!(join); + assert!( + tokio::time::timeout(std::time::Duration::from_millis(25), join.as_mut()) + .await + .is_err() + ); + release.send(()).unwrap(); + tokio::time::timeout(std::time::Duration::from_secs(5), join) + .await + .unwrap(); + domain.shutdown().await; +} diff --git a/crates/cellar/durability/raft/cell_groups/archive/fleet.rs b/crates/cellar/durability/raft/cell_groups/archive/fleet.rs new file mode 100644 index 000000000..8767e9d90 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/archive/fleet.rs @@ -0,0 +1,95 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Fleet coverage consumes complete independently recovered group closures. +//! A selected object, uploaded bytes or a single cell image is insufficient. +use super::*; +use crate::raft::{host::FleetPublishedCut, member_state::FleetGroupBinding}; +use cellar_logic::consensus::{LogIndex, LogPosition}; + +#[derive(Clone)] +pub struct VerifiedFleetGroupArchiveCoverage { + coverage: CellGroupArchiveCoverage, + binding: FleetGroupBinding, + configuration_digest: [u8; 32], +} + +impl VerifiedFleetGroupArchiveCoverage { + pub fn hold(&self) -> &CellGroupArchiveHold { + self.coverage.hold() + } + + pub fn root(&self) -> &GroupManifest { + self.coverage.marker() + } + + pub fn binding(&self) -> &FleetGroupBinding { + &self.binding + } + + pub fn position(&self) -> LogPosition { + self.root().position() + } + + pub fn membership_revision(&self) -> LogIndex { + LogIndex::new(self.root().membership_revision()) + } + + pub fn configuration_digest(&self) -> [u8; 32] { + self.configuration_digest + } + + pub fn covers(&self, minimum: &FleetPublishedCut) -> Result { + if self.binding() != minimum.binding() + || self.configuration_digest != minimum.configuration_digest() + { + return Err(CellGroupArchiveError::Conflict); + } + crate::raft::host::group_prefix(self.root(), minimum.root()) + .map_err(|_| CellGroupArchiveError::Conflict) + } +} + +impl CellGroupArchiveRegistry { + /// Re-read the exact persisted hold, independently restore every checkpoint + /// dependency and replay the contiguous tail, then compare the full root. + /// The execution owner retains entered physical work after cancellation. + /// The caller must enforce its original deadline around this read/replay and + /// suppress late delivery; a saved proof is historical, never a health lease. + pub async fn verify_fleet_published_at( + &self, + domain: &impl CellGroupArchiveExecution, + tails: &CellGroupArchiveTailPool, + hold: &CellGroupArchiveHold, + minimum: &FleetPublishedCut, + receiver: CellGroupArchiveCheckpointReceiver, + ) -> Result { + let coverage = self.verify_hold(domain, hold, tails, receiver).await?; + let membership = coverage.recovery().membership(); + let status = membership + .fleet_status() + .ok_or(CellGroupArchiveError::Invalid( + "archive has no fleet enrollment", + ))?; + if membership.revision().get() != coverage.marker().membership_revision() + || !status + .binding + .matches_scope(&coverage.marker().route().history) + { + return Err(CellGroupArchiveError::Invalid("fleet archive membership")); + } + let configuration_digest = crate::raft::host::group_membership_digest( + membership, + coverage.recovery().configuration(), + ) + .map_err(|_| CellGroupArchiveError::Invalid("fleet archive configuration commitment"))?; + let proof = VerifiedFleetGroupArchiveCoverage { + coverage, + binding: status.binding, + configuration_digest, + }; + if !proof.covers(minimum)? { + return Err(CellGroupArchiveError::Unconfirmed); + } + Ok(proof) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/archive/graph.rs b/crates/cellar/durability/raft/cell_groups/archive/graph.rs new file mode 100644 index 000000000..04e02d7f9 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/archive/graph.rs @@ -0,0 +1,167 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Exact dependency-key expansion. No collector may delete from a partial mark. +use super::{objects::Object, ownership::Descriptor, *}; +use std::collections::BTreeSet; + +pub(super) const ROOT_DOMAIN: &str = "cellar-cell-group-archive-root-1"; + +#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)] +pub(super) enum Kind { + Chunks, + Indexes, + Entries, + Tails, + Roots, +} + +impl Kind { + pub fn parse(name: &str) -> Option { + Some(match name { + "chunks" => Self::Chunks, + "indexes" => Self::Indexes, + "entries" => Self::Entries, + "tails" => Self::Tails, + "roots" => Self::Roots, + _ => return None, + }) + } + + pub fn name(self) -> &'static str { + match self { + Self::Chunks => "chunks", + Self::Indexes => "indexes", + Self::Entries => "entries", + Self::Tails => "tails", + Self::Roots => "roots", + } + } + + pub fn limit(self, archive: &CellGroupArchive, limits: CellGroupArchiveTailLimits) -> usize { + match self { + Self::Chunks => CHUNK_BYTES, + Self::Indexes | Self::Tails => archive.limits.max_index_bytes, + Self::Entries => limits.max_tail_bytes, + Self::Roots => 8192, + } + } +} + +#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord)] +pub(super) struct Key { + pub kind: Kind, + pub object: Object, +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Root { + pub domain: String, + pub marker: GroupManifest, + pub tail: CellGroupArchiveTailReference, +} + +pub(super) struct Graph { + pub objects: BTreeSet, + limit: usize, +} + +impl Graph { + pub fn new(limit: usize) -> Self { + Self { + objects: BTreeSet::new(), + limit, + } + } + + pub fn limit(&self) -> usize { + self.limit + } + + pub fn insert(&mut self, kind: Kind, object: Object, generation: u64) -> Result<()> { + if object.registry == [0; 32] + || object.generation < 2 + || object.generation > generation + || object.length == 0 + { + return Err(CellGroupArchiveError::Invalid( + "archive dependency generation", + )); + } + let key = Key { kind, object }; + if !self.objects.contains(&key) && self.objects.len() == self.limit { + return Err(CellGroupArchiveError::Limit); + } + self.objects.insert(key); + Ok(()) + } + + pub async fn load( + archive: &CellGroupArchive, + domain: &impl CellGroupArchiveExecution, + descriptor: &Descriptor, + limits: CellGroupArchiveTailLimits, + io: Arc, + retained: Arc, + max_objects: usize, + ) -> Result { + let root = archive.get("roots", &descriptor.object, 8192).await?; + let root: Root = serde_json::from_slice(&root) + .map_err(|_| CellGroupArchiveError::Invalid("archive root encoding"))?; + if root.domain != ROOT_DOMAIN || root.marker != descriptor.marker { + return Err(CellGroupArchiveError::Invalid("archive root binding")); + } + let tail = archive + .read_tail_index(domain, &root.tail, limits, io.clone(), retained.clone()) + .await?; + if tail.cut.marker != root.marker { + return Err(CellGroupArchiveError::Invalid("archive root/tail cut")); + } + let checkpoint = archive + .checkpoint_index(domain, &tail.checkpoint, io) + .await?; + if checkpoint.marker != tail.base { + return Err(CellGroupArchiveError::Invalid( + "archive root/checkpoint cut", + )); + } + let generation = descriptor.object.generation; + let mut graph = Self::new(max_objects); + graph.insert(Kind::Roots, descriptor.object.clone(), generation)?; + graph.insert(Kind::Tails, root.tail.object(), generation)?; + graph.insert(Kind::Indexes, tail.checkpoint.object(), generation)?; + for entry in tail.entries { + graph.insert(Kind::Entries, entry.object, generation)?; + } + for chunk in checkpoint.chunks { + graph.insert(Kind::Chunks, chunk.object, generation)?; + } + if graph + .objects + .iter() + .any(|key| key.object.registry != descriptor.object.registry) + { + return Err(CellGroupArchiveError::Invalid( + "archive dependency registry", + )); + } + Ok(graph) + } + + pub async fn verify( + &self, + archive: &CellGroupArchive, + limits: CellGroupArchiveTailLimits, + ) -> Result<()> { + for key in &self.objects { + archive + .get( + key.kind.name(), + &key.object, + key.kind.limit(archive, limits), + ) + .await?; + } + Ok(()) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/archive/mod.rs b/crates/cellar/durability/raft/cell_groups/archive/mod.rs new file mode 100644 index 000000000..90a9826d9 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/archive/mod.rs @@ -0,0 +1,298 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Complete group checkpoints and committed tails in immutable object storage. Upload confirmation +//! and independently recovered physical state are separate capabilities. Neither +//! authorizes S3 settlement, log truncation, membership or disaster recovery. + +pub use super::checkpoint::{ + CellGroupArchiveCheckpointReceiver, PreparedCellGroupArchiveCheckpoint, +}; +use super::{checkpoint::wire, CellGroupCheckpoint, CellGroupCheckpointError, GroupManifest}; +use crate::raft::budget::{Budget, Permit}; +use cellar_logic::cell_group::GroupHistory; +use object_store::{path::Path, ObjectStore}; +use serde::{Deserialize, Serialize}; +use sha2::{Digest, Sha256}; +use std::{fmt, sync::Arc}; + +mod execution; +pub use execution::CellGroupArchiveExecution; +pub(crate) use execution::ControllerExecution; + +mod checkpoint; +mod data_store; +mod fleet; +pub use data_store::CellGroupArchiveStore; +pub use fleet::VerifiedFleetGroupArchiveCoverage; +mod collection; +mod graph; +mod objects; +mod publication; +pub use collection::{CellGroupArchiveCollectionLimits, CellGroupArchiveCollectionReport}; +pub(in crate::raft::cell_groups) use publication::CoverageUse; +pub use publication::{CellGroupArchiveCoverage, PreparedCellGroupArchivePublication}; +mod upload; +pub use upload::{ + CellGroupArchiveUpload, CellGroupArchiveUploadLimits, CellGroupArchiveUploadPool, +}; +mod ownership; +pub(crate) use ownership::CellGroupArchiveAcquireOutcome; +pub use ownership::{ + CellGroupArchiveCreation, CellGroupArchiveHold, CellGroupArchiveIntent, + CellGroupArchiveRegistry, CellGroupArchiveUploadTicket, PreparedCellGroupArchiveCreation, +}; +pub(in crate::raft::cell_groups) mod enrollment; +pub(in crate::raft::cell_groups) mod receipt; +mod replay; +mod tail; +mod tail_admission; +mod tail_codec; +pub use replay::VerifiedCellGroupArchiveRecovery; +pub use tail::{CellGroupArchiveTailReference, CellGroupArchiveUploadedTail}; +pub use tail_admission::{CellGroupArchiveTailLimits, CellGroupArchiveTailPool}; + +const DOMAIN: &str = "cellar-cell-group-archive-checkpoint-1"; +const CHUNK_BYTES: usize = 64 * 1024; + +#[derive(Debug)] +pub enum CellGroupArchiveError { + Invalid(&'static str), + Limit, + Busy, + Stopped, + Timeout, + Conflict, + Expired, + Unconfirmed, + MissingRegistry, + Checkpoint(CellGroupCheckpointError), + Replay(crate::raft::execution::ExecutionError), + Enrollment(super::EnrollmentError), + Storage(object_store::Error), + Worker(crate::asyncrt::TaskPanic), +} + +impl fmt::Display for CellGroupArchiveError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "cell-group archive: {self:?}") + } +} + +impl std::error::Error for CellGroupArchiveError { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + match self { + Self::Checkpoint(e) => Some(e), + Self::Replay(e) => Some(e), + Self::Enrollment(e) => Some(e), + Self::Storage(e) => Some(e), + Self::Worker(e) => Some(e), + _ => None, + } + } +} + +impl From for CellGroupArchiveError { + fn from(error: CellGroupCheckpointError) -> Self { + Self::Checkpoint(error) + } +} + +type Result = std::result::Result; + +/// Share one archive pool across this node's group jobs. Clones share admission. +/// Checkpoint source/receiver pools separately account physical closures/scratch. +#[derive(Clone, Copy, Debug)] +pub struct CellGroupArchiveLimits { + pub max_operations: usize, + pub max_retained_bytes: usize, + pub max_index_bytes: usize, + pub max_chunks: usize, + pub max_checkpoint_bytes: u64, + /// Deadline for a complete individual PUT or GET, including its body stream. + pub io_timeout_ms: u64, +} + +impl Default for CellGroupArchiveLimits { + fn default() -> Self { + Self { + max_operations: 2, + max_retained_bytes: 32 * 1024 * 1024, + max_index_bytes: 1024 * 1024, + max_chunks: 4096, + max_checkpoint_bytes: 256 * 1024 * 1024, + io_timeout_ms: 10_000, + } + } +} + +impl CellGroupArchiveLimits { + fn workspace(self) -> Result { + if !(1..=4096).contains(&self.max_operations) + || !(1024..=8 * 1024 * 1024).contains(&self.max_index_bytes) + || !(1..=65_536).contains(&self.max_chunks) + || !(64 * 1024..=1024 * 1024 * 1024).contains(&self.max_checkpoint_bytes) + || !(1..=60_000).contains(&self.io_timeout_ms) + { + return Err(CellGroupArchiveError::Limit); + } + // Decoded records, bounded encoded index, GET/PUT windows and transport + // copies. The physical source or receiver has its own pool reservation. + self.max_index_bytes + .checked_mul(6) + .and_then(|n| n.checked_add(self.max_chunks.checked_mul(2048)?)) + .and_then(|n| n.checked_add(8 * CHUNK_BYTES)) + .ok_or(CellGroupArchiveError::Limit) + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct CellGroupArchiveUsage { + pub operations: usize, + pub retained_bytes: usize, +} + +/// An untrusted locator, suitable for persistence by a later archive-head owner. +/// Its digest authenticates the index bytes, not their complete recovery closure. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct CellGroupArchiveReference { + registry: [u8; 32], + generation: u64, + digest: [u8; 32], + length: u64, +} + +impl CellGroupArchiveReference { + pub fn registry(&self) -> [u8; 32] { + self.registry + } + + pub fn generation(&self) -> u64 { + self.generation + } + + fn object(&self) -> objects::Object { + objects::Object { + registry: self.registry, + generation: self.generation, + digest: self.digest, + length: self.length, + } + } + + pub fn digest(&self) -> [u8; 32] { + self.digest + } + + pub fn length(&self) -> u64 { + self.length + } +} + +/// Every referenced object was downloaded, hashed and physically recovered on +/// the destination. The contained checkpoint retains receiver-local pins. This +/// is a checkpoint-only proof, without a tail, published head or remote GC lease. +pub struct VerifiedCellGroupArchiveCheckpoint { + reference: CellGroupArchiveReference, + checkpoint: PreparedCellGroupArchiveCheckpoint, +} + +impl VerifiedCellGroupArchiveCheckpoint { + pub fn reference(&self) -> &CellGroupArchiveReference { + &self.reference + } + + pub fn checkpoint(&self) -> &PreparedCellGroupArchiveCheckpoint { + &self.checkpoint + } + + pub fn into_checkpoint(self) -> PreparedCellGroupArchiveCheckpoint { + self.checkpoint + } +} + +/// Group-owned archive namespace. No synthetic cell identity is involved. +#[derive(Clone)] +pub struct CellGroupArchive { + store: Arc, + prefix: Path, + history: GroupHistory, + limits: CellGroupArchiveLimits, + budget: Arc, + workspace: usize, +} + +/// Node-wide archive admission, shared by every group binding and both transfer +/// directions. Holding a group handle consumes no operation slot or I/O buffer. +#[derive(Clone)] +pub struct CellGroupArchivePool { + limits: CellGroupArchiveLimits, + budget: Arc, + workspace: usize, +} + +impl CellGroupArchivePool { + pub fn new(limits: CellGroupArchiveLimits) -> Result { + let workspace = limits.workspace()?; + if limits.max_retained_bytes < workspace { + return Err(CellGroupArchiveError::Limit); + } + Ok(Self { + limits, + budget: Budget::new(limits.max_retained_bytes, limits.max_operations), + workspace, + }) + } + + pub fn for_group( + &self, + store: Arc, + prefix: Path, + history: GroupHistory, + ) -> Result { + if prefix.as_ref().len() > 4096 { + return Err(CellGroupArchiveError::Limit); + } + let mut binding = Sha256::new(); + binding.update(b"cellar-cell-group-archive-history-1\0"); + binding.update(history.cluster.bytes()); + binding.update(history.namespace.bytes()); + binding.update(history.group.bytes()); + binding.update(history.incarnation.bytes()); + let prefix = prefix.child(DOMAIN).child(hex::encode(binding.finalize())); + Ok(CellGroupArchive { + store, + prefix, + history, + limits: self.limits, + budget: self.budget.clone(), + workspace: self.workspace, + }) + } + + pub fn usage(&self) -> CellGroupArchiveUsage { + let (retained_bytes, operations) = self.budget.usage(); + CellGroupArchiveUsage { + operations, + retained_bytes, + } + } +} + +impl CellGroupArchive { + /// Advisory node-wide counters shared with every binding from this pool. + pub fn usage(&self) -> CellGroupArchiveUsage { + let (retained_bytes, operations) = self.budget.usage(); + CellGroupArchiveUsage { + operations, + retained_bytes, + } + } + + fn admit(&self) -> Result> { + self.budget + .acquire(self.workspace) + .map(Arc::new) + .ok_or(CellGroupArchiveError::Busy) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/archive/objects.rs b/crates/cellar/durability/raft/cell_groups/archive/objects.rs new file mode 100644 index 000000000..443f1eb46 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/archive/objects.rs @@ -0,0 +1,202 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use futures_util::TryStreamExt; +use object_store::{PutMode, PutOptions}; +use std::time::Duration; + +#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Object { + pub registry: [u8; 32], + pub generation: u64, + pub digest: [u8; 32], + pub length: u64, +} + +impl Object { + pub fn of(registry: [u8; 32], generation: u64, bytes: &[u8]) -> Self { + Self { + registry, + generation, + digest: Sha256::digest(bytes).into(), + length: bytes.len() as u64, + } + } + + pub fn verify(&self, bytes: &[u8]) -> Result<()> { + if bytes.len() as u64 != self.length + || <[u8; 32]>::from(Sha256::digest(bytes)) != self.digest + { + return Err(CellGroupArchiveError::Invalid("object digest/length")); + } + Ok(()) + } +} + +impl CellGroupArchive { + pub(super) fn object_path(&self, kind: &str, object: &Object) -> Path { + self.prefix + .child("registries") + .child(hex::encode(object.registry)) + .child("generations") + .child(object.generation.to_string()) + .child(kind) + .child(format!("{}-{}", object.length, hex::encode(object.digest))) + } + + pub(super) async fn io( + &self, + future: impl std::future::Future>, + ) -> Result { + crate::asyncrt::timeout(Duration::from_millis(self.limits.io_timeout_ms), future) + .await + .map_err(|_| CellGroupArchiveError::Timeout)? + } + + // One deadline includes headers, every body chunk and EOF. Never allocate + // from unbounded remote metadata or accept a partial/ranged object response. + async fn read(&self, kind: &str, object: &Object, limit: usize) -> Result>> { + if object.registry == [0; 32] + || object.generation < 2 + || object.generation == u64::MAX + || object.length == 0 + || object.length > limit as u64 + { + return Err(CellGroupArchiveError::Limit); + } + let path = self.object_path(kind, object); + self.io(async { + let result = match self.store.get(&path).await { + Ok(result) => result, + Err(object_store::Error::NotFound { .. }) => return Ok(None), + Err(error) => return Err(CellGroupArchiveError::Storage(error)), + }; + if result.meta.size != object.length + || result.range.start != 0 + || result.range.end != object.length + { + return Err(CellGroupArchiveError::Invalid( + "object response length/range", + )); + } + let mut bytes = Vec::with_capacity(object.length as usize); + let mut stream = result.into_stream(); + let mut chunks = 0; + while let Some(chunk) = stream + .try_next() + .await + .map_err(CellGroupArchiveError::Storage)? + { + chunks += 1; + if chunks > 16_384 + || bytes + .len() + .checked_add(chunk.len()) + .is_none_or(|n| n > object.length as usize) + { + return Err(CellGroupArchiveError::Limit); + } + bytes.extend_from_slice(&chunk); + } + object.verify(&bytes)?; + Ok(Some(bytes)) + }) + .await + } + + pub(super) async fn get(&self, kind: &str, object: &Object, limit: usize) -> Result> { + self.read(kind, object, limit) + .await? + .ok_or(CellGroupArchiveError::Invalid("missing archive object")) + } + + pub(super) async fn put_generation( + &self, + registry: [u8; 32], + generation: u64, + kind: &str, + bytes: Vec, + limit: usize, + ) -> Result { + if bytes.is_empty() || bytes.len() > limit { + return Err(CellGroupArchiveError::Limit); + } + let object = Object::of(registry, generation, &bytes); + // Reuse verified immutable objects without another PUT. A corrupt + // existing object refuses; overwriting could erase evidence of damage. + if self.read(kind, &object, limit).await?.is_some() { + return Ok(object); + } + let path = self.object_path(kind, &object); + let put = self + .io(async { + self.store + .put_opts(&path, bytes.into(), PutOptions::from(PutMode::Create)) + .await + .map_err(CellGroupArchiveError::Storage) + }) + .await; + // Lost/failed PUT responses are ambiguous. Only an exact GET confirms + // publication, including a concurrent conditional-create winner. + match self.read(kind, &object, limit).await? { + Some(_) => Ok(object), + None => match put { + Err(error) => Err(error), + Ok(_) => Err(CellGroupArchiveError::Invalid("unconfirmed archive object")), + }, + } + } + + pub(super) async fn read_versioned( + &self, + path: &Path, + limit: usize, + ) -> Result, object_store::UpdateVersion)>> { + self.io(async { + let result = match self.store.get(path).await { + Ok(result) => result, + Err(object_store::Error::NotFound { .. }) => return Ok(None), + Err(error) => return Err(CellGroupArchiveError::Storage(error)), + }; + let length = result.meta.size; + if length == 0 + || length > limit as u64 + || result.range.start != 0 + || result.range.end != length + { + return Err(CellGroupArchiveError::Invalid( + "control response length/range", + )); + } + let version = object_store::UpdateVersion { + e_tag: result.meta.e_tag.clone(), + version: result.meta.version.clone(), + }; + let mut bytes = Vec::with_capacity(length as usize); + let mut stream = result.into_stream(); + let mut chunks = 0; + while let Some(chunk) = stream + .try_next() + .await + .map_err(CellGroupArchiveError::Storage)? + { + chunks += 1; + if chunks > 16_384 + || bytes + .len() + .checked_add(chunk.len()) + .is_none_or(|n| n > length as usize) + { + return Err(CellGroupArchiveError::Limit); + } + bytes.extend_from_slice(&chunk); + } + if bytes.len() != length as usize { + return Err(CellGroupArchiveError::Invalid("control body length")); + } + Ok(Some((bytes, version))) + }) + .await + } +} diff --git a/crates/cellar/durability/raft/cell_groups/archive/ownership/mod.rs b/crates/cellar/durability/raft/cell_groups/archive/ownership/mod.rs new file mode 100644 index 000000000..6694d0d18 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/archive/ownership/mod.rs @@ -0,0 +1,811 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Persistent ownership of group archive generations. Dropping local handles +//! never retires a remote owner. Allocating intents must be persisted before CAS. +mod state; +mod wire; +use wire::WireCreation; + +use super::{objects::Object, *}; +use cellar_logic::cell_group::GroupRoute; +use object_store::{PutMode, PutOptions, UpdateVersion}; +use rand::RngCore; +use state::State; + +const IDENTITY_DOMAIN: &str = "cellar-cell-group-archive-identity-1"; +const STATE_DOMAIN: &str = "cellar-cell-group-archive-ownership-1"; +const INTENT_DOMAIN: &str = "cellar-cell-group-archive-intent-1"; +const MAX_CONTROL_BYTES: usize = 64 * 1024; +const MAX_INTENT_BYTES: usize = 8192; + +/// Persist before creation. Retry reconciles an already complete creation. +/// An incomplete identity requires authority recovery; counters never reset. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(try_from = "WireCreation", into = "WireCreation")] +pub struct CellGroupArchiveCreation { + domain: String, + id: [u8; 32], + route: GroupRoute, +} + +/// Fresh, one-use initialization authority. Serialized creation metadata can +/// reopen established state but cannot recreate counters after metadata loss. +pub struct PreparedCellGroupArchiveCreation { + archive: CellGroupArchive, + identity: CellGroupArchiveCreation, +} + +impl PreparedCellGroupArchiveCreation { + pub fn intent(&self) -> &CellGroupArchiveCreation { + &self.identity + } + + pub async fn commit( + self, + domain: &impl CellGroupArchiveExecution, + ) -> Result { + self.archive.create_registry(domain, &self.identity).await + } +} + +impl CellGroupArchiveCreation { + pub fn route(&self) -> &GroupRoute { + &self.route + } + + fn check(&self, history: GroupHistory) -> Result<()> { + if self.domain != IDENTITY_DOMAIN || self.id == [0; 32] || self.route.history != history { + return Err(CellGroupArchiveError::Invalid("archive registry identity")); + } + super::super::ManifestTree::new(self.route.clone(), Default::default()) + .map_err(CellGroupCheckpointError::from)?; + Ok(()) + } +} + +/// A never-reused upload generation. It is protected until explicitly aborted +/// or consumed by publication, including after its local handle is dropped. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct CellGroupArchiveUploadTicket { + registry: [u8; 32], + generation: u64, + allocation: [u8; 32], +} + +impl CellGroupArchiveUploadTicket { + pub fn generation(&self) -> u64 { + self.generation + } +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Descriptor { + pub object: Object, + pub marker: GroupManifest, +} + +impl Descriptor { + fn check(&self, identity: &CellGroupArchiveCreation, revision: u64) -> Result<()> { + self.marker + .check() + .map_err(CellGroupCheckpointError::from)?; + if self.marker.route() != &identity.route + || self.object.registry != identity.id + || self.object.generation < 2 + || self.object.generation >= revision + || self.object.length == 0 + || self.object.length > 8192 + { + return Err(CellGroupArchiveError::Invalid( + "archive retained descriptor", + )); + } + Ok(()) + } +} + +/// Explicit remote hold, distinct from integrity, quorum and serving proofs. +/// Persist it with its owner; release only after that owner retires/fences all +/// users and joins entered work. Never release merely because a Rust value drops. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct CellGroupArchiveHold { + registry: [u8; 32], + id: u64, + allocation: [u8; 32], + root: Descriptor, +} + +impl CellGroupArchiveHold { + pub fn marker(&self) -> &GroupManifest { + &self.root.marker + } + + /// Descriptor shape only. Persistent registry membership must still be + /// checked by the owning registry before and after physical verification. + pub(crate) fn check_reference(&self) -> Result<()> { + self.root + .marker + .check() + .map_err(CellGroupCheckpointError::from)?; + if self.registry == [0; 32] + || self.allocation == [0; 32] + || self.id == 0 + || self.id >= u64::MAX - 2 + || self.root.object.registry != self.registry + || self.root.object.digest == [0; 32] + || self.root.object.generation < 2 + || self.root.object.generation >= u64::MAX - 2 + || self.root.object.length == 0 + || self.root.object.length > 8192 + { + return Err(CellGroupArchiveError::Invalid("archive hold descriptor")); + } + Ok(()) + } +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) enum Action { + Reserve { + parent: Option, + }, + Acquire { + root: Descriptor, + }, + Publish { + ticket: CellGroupArchiveUploadTicket, + root: Descriptor, + }, +} + +/// One exact allocation at one registry revision. Persist the entire intent +/// before sending its mutation. A lost response never justifies a new intent. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct CellGroupArchiveIntent { + domain: String, + registry: [u8; 32], + revision: u64, + nonce: [u8; 32], + action: Action, +} + +impl CellGroupArchiveIntent { + pub(crate) fn acquire_marker(&self) -> Result<&GroupManifest> { + self.digest()?; + match &self.action { + Action::Acquire { root } => Ok(&root.marker), + _ => Err(CellGroupArchiveError::Invalid("acquire intent purpose")), + } + } + + fn digest(&self) -> Result<[u8; 32]> { + if self.domain != INTENT_DOMAIN + || self.registry == [0; 32] + || self.nonce == [0; 32] + || self.revision < 2 + || self.revision == u64::MAX + { + return Err(CellGroupArchiveError::Invalid("archive allocation intent")); + } + Ok(Sha256::digest(encode(self, MAX_INTENT_BYTES)?).into()) + } +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Pending { + pub ticket: CellGroupArchiveUploadTicket, + pub parent: Option, +} + +pub(super) enum Allocation { + Upload(CellGroupArchiveUploadTicket), + Hold(Box), +} + +/// A fresh registry observation, distinct from an unsuccessful mutation. Only +/// Superseded proves that this allocation cannot still commit later. +pub(crate) enum CellGroupArchiveAcquireOutcome { + Held(Box), + Pending, + Superseded, +} + +pub(super) struct Protection { + pub revision: u64, + pub roots: Vec, + pub pending_generations: Vec, +} + +/// Conditional storage adapter for one exact group route and registry identity. +/// Owns no Raft engine and cannot grant membership or application durability. +#[derive(Clone)] +pub struct CellGroupArchiveRegistry { + pub(super) archive: CellGroupArchive, + identity: CellGroupArchiveCreation, +} + +fn encode(value: &impl Serialize, limit: usize) -> Result> { + let mut output = super::checkpoint::Bounded { + bytes: Vec::new(), + limit, + }; + serde_json::to_writer(&mut output, value).map_err(|_| CellGroupArchiveError::Limit)?; + Ok(output.bytes) +} + +fn check_registry_version(version: &UpdateVersion) -> Result<()> { + if version.e_tag.as_ref().is_some_and(String::is_empty) + || version.version.as_ref().is_some_and(String::is_empty) + || version.e_tag.as_ref().map_or(0, String::len) + + version.version.as_ref().map_or(0, String::len) + > 8192 + || (version.e_tag.is_none() && version.version.is_none()) + { + return Err(CellGroupArchiveError::Invalid( + "archive registry conditional version", + )); + } + Ok(()) +} + +impl CellGroupArchive { + /// Preparation performs no storage mutation. The execution domain supplies + /// randomness; the caller durably records the complete result before create. + pub async fn prepare_registry_creation( + &self, + domain: &impl CellGroupArchiveExecution, + route: GroupRoute, + ) -> Result { + domain + .run(async { + let mut id = [0; 32]; + crate::asyncrt::rng("cell-group-archive-registry").fill_bytes(&mut id); + let creation = CellGroupArchiveCreation { + domain: IDENTITY_DOMAIN.into(), + id, + route, + }; + creation.check(self.history)?; + Ok(PreparedCellGroupArchiveCreation { + archive: self.clone(), + identity: creation, + }) + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } + + async fn create_registry( + &self, + domain: &impl CellGroupArchiveExecution, + creation: &CellGroupArchiveCreation, + ) -> Result { + domain + .run(async { + let _io = self.admit()?; + creation.check(self.history)?; + let registry = CellGroupArchiveRegistry { + archive: self.clone(), + identity: creation.clone(), + }; + if self + .read_versioned(®istry.path("registry"), MAX_CONTROL_BYTES) + .await? + .is_some() + { + registry.load().await?; + return Ok(registry); + } + let bytes = encode(creation, MAX_CONTROL_BYTES)?; + // Only a confirmed exclusive identity creation can initialize the + // counter. An existing identity may be reconciled only by opening + // valid established state, never by recreating a missing registry. + match registry.write("identity", bytes, PutMode::Create).await { + Ok(()) => (), + Err(_) => { + registry + .load() + .await + .map_err(|_| CellGroupArchiveError::Unconfirmed)?; + return Ok(registry); + } + } + let desired = encode(&State::empty(creation.clone()), MAX_CONTROL_BYTES)?; + let outcome = registry.write("registry", desired, PutMode::Create).await; + match registry.load().await { + Ok(_) => Ok(registry), + Err(_) => outcome.and(Err(CellGroupArchiveError::Unconfirmed)), + } + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } + + /// Initialize a fresh replacement whose exact namespace binding is already + /// selected under the managed recovery fence. Write the empty counter + /// before publishing its immutable identity so either partial setup can + /// be reopened. An established identity still forbids counter recreation. + pub(crate) async fn initialize_recovery_registry( + &self, + domain: &impl CellGroupArchiveExecution, + expected: &CellGroupArchiveCreation, + ) -> Result { + domain + .run(async { + let _io = self.admit()?; + expected.check(self.history)?; + let registry = CellGroupArchiveRegistry { + archive: self.clone(), + identity: expected.clone(), + }; + if self + .read_versioned(®istry.path("identity"), MAX_CONTROL_BYTES) + .await? + .is_some() + { + registry.load().await?; + return Ok(registry); + } + + let empty = encode(&State::empty(expected.clone()), MAX_CONTROL_BYTES)?; + let key = registry.path("registry"); + let existing = self.read_versioned(&key, MAX_CONTROL_BYTES).await?; + let (state, version) = match existing { + Some(existing) => existing, + None => { + let outcome = registry + .write("registry", empty.clone(), PutMode::Create) + .await; + self.read_versioned(&key, MAX_CONTROL_BYTES) + .await? + .ok_or_else(|| { + outcome.err().unwrap_or(CellGroupArchiveError::Unconfirmed) + })? + } + }; + check_registry_version(&version)?; + // A concurrent controller may have established this identity + // and published later state. Only an exact empty state can + // receive a new identity; any other state must already open. + if state == empty { + let bytes = encode(expected, MAX_CONTROL_BYTES)?; + let outcome = registry.write("identity", bytes, PutMode::Create).await; + match registry.load().await { + Ok(_) => (), + Err(CellGroupArchiveError::MissingRegistry) => { + return outcome.and(Err(CellGroupArchiveError::Unconfirmed)); + } + Err(error) => return Err(error), + } + } else { + registry.load().await?; + } + Ok(registry) + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } + + /// Open established state. Missing identity/state and malformed conditional + /// versions refuse; opening never allocates or repairs a missing record. + pub async fn open_registry( + &self, + domain: &impl CellGroupArchiveExecution, + expected: &CellGroupArchiveCreation, + ) -> Result { + domain + .run(async { + let _io = self.admit()?; + expected.check(self.history)?; + let registry = CellGroupArchiveRegistry { + archive: self.clone(), + identity: expected.clone(), + }; + registry.load().await?; + Ok(registry) + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } +} + +impl CellGroupArchiveRegistry { + pub fn archive(&self) -> &CellGroupArchive { + &self.archive + } + + pub fn route(&self) -> &GroupRoute { + &self.identity.route + } + + pub fn creation(&self) -> &CellGroupArchiveCreation { + &self.identity + } + + pub(super) fn id(&self) -> [u8; 32] { + self.identity.id + } + + fn path(&self, name: &str) -> Path { + self.archive.prefix.child("ownership").child(name) + } + + async fn write(&self, name: &str, bytes: Vec, mode: PutMode) -> Result<()> { + self.archive + .io(async { + self.archive + .store + .put_opts(&self.path(name), bytes.into(), PutOptions::from(mode)) + .await + .map_err(CellGroupArchiveError::Storage)?; + Ok(()) + }) + .await + } + + async fn load(&self) -> Result<(State, UpdateVersion)> { + let identity = self + .archive + .read_versioned(&self.path("identity"), MAX_CONTROL_BYTES) + .await? + .ok_or(CellGroupArchiveError::MissingRegistry)? + .0; + if identity != encode(&self.identity, MAX_CONTROL_BYTES)? { + return Err(CellGroupArchiveError::Invalid( + "archive registry identity changed", + )); + } + let (bytes, version) = self + .archive + .read_versioned(&self.path("registry"), MAX_CONTROL_BYTES) + .await? + .ok_or(CellGroupArchiveError::MissingRegistry)?; + check_registry_version(&version)?; + let state: State = serde_json::from_slice(&bytes) + .map_err(|_| CellGroupArchiveError::Invalid("archive registry encoding"))?; + state.check(&self.identity)?; + Ok((state, version)) + } + + pub async fn prepare_upload( + &self, + domain: &impl CellGroupArchiveExecution, + ) -> Result { + domain + .run(async { + let _io = self.archive.admit()?; + let state = self.load().await?.0; + self.intent( + &state, + Action::Reserve { + parent: state.selected.clone(), + }, + ) + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } + + pub async fn reserve( + &self, + domain: &impl CellGroupArchiveExecution, + intent: &CellGroupArchiveIntent, + ) -> Result { + if !matches!(intent.action, Action::Reserve { .. }) { + return Err(CellGroupArchiveError::Invalid("upload intent purpose")); + } + domain + .run(async { + let _io = self.archive.admit()?; + match self.commit(intent).await? { + Allocation::Upload(ticket) => Ok(ticket), + _ => Err(CellGroupArchiveError::Invalid("upload allocation purpose")), + } + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } + + pub async fn prepare_acquire( + &self, + domain: &impl CellGroupArchiveExecution, + ) -> Result { + domain + .run(async { + let _io = self.archive.admit()?; + let state = self.load().await?.0; + let root = state + .selected + .clone() + .ok_or(CellGroupArchiveError::Unconfirmed)?; + self.intent(&state, Action::Acquire { root }) + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } + + pub async fn acquire( + &self, + domain: &impl CellGroupArchiveExecution, + intent: &CellGroupArchiveIntent, + ) -> Result { + if !matches!(intent.action, Action::Acquire { .. }) { + return Err(CellGroupArchiveError::Invalid("acquire intent purpose")); + } + domain + .run(async { + let _io = self.archive.admit()?; + match self.commit(intent).await? { + Allocation::Hold(hold) => Ok(*hold), + _ => Err(CellGroupArchiveError::Invalid("hold allocation purpose")), + } + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } + + pub(crate) async fn acquire_outcome( + &self, + domain: &impl CellGroupArchiveExecution, + intent: &CellGroupArchiveIntent, + ) -> Result { + intent.acquire_marker()?; + domain + .run(async { + let _io = self.archive.admit()?; + self.load().await?.0.acquire_outcome(intent) + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } + + fn intent(&self, state: &State, action: Action) -> Result { + let mut nonce = [0; 32]; + crate::asyncrt::rng("cell-group-archive-intent").fill_bytes(&mut nonce); + let intent = CellGroupArchiveIntent { + domain: INTENT_DOMAIN.into(), + registry: self.identity.id, + revision: state + .revision + .checked_add(1) + .ok_or(CellGroupArchiveError::Limit)?, + nonce, + action, + }; + let mut proposed = state.clone(); + proposed.allocate(&intent)?; + proposed.check(&self.identity)?; + Ok(intent) + } + + pub(super) async fn commit(&self, intent: &CellGroupArchiveIntent) -> Result { + if intent.registry != self.identity.id { + return Err(CellGroupArchiveError::Conflict); + } + let (mut state, version) = self.load().await?; + if let Some(result) = state.reconcile(intent)? { + return Ok(result); + } + state.allocate(intent)?; + state.check(&self.identity)?; + let outcome = self + .write( + "registry", + encode(&state, MAX_CONTROL_BYTES)?, + PutMode::Update(version), + ) + .await; + let observed = self + .load() + .await + .map_err(|_| CellGroupArchiveError::Unconfirmed)? + .0; + if let Some(result) = observed.reconcile(intent)? { + return Ok(result); + } + if matches!( + outcome, + Err(CellGroupArchiveError::Storage( + object_store::Error::Precondition { .. } + | object_store::Error::AlreadyExists { .. } + )) + ) { + return Err(CellGroupArchiveError::Conflict); + } + Err(CellGroupArchiveError::Unconfirmed) + } + + pub(super) async fn prepare_publish( + &self, + ticket: CellGroupArchiveUploadTicket, + root: Descriptor, + ) -> Result { + let state = self.load().await?.0; + self.intent(&state, Action::Publish { ticket, root }) + } + + pub(super) async fn publication_hold( + &self, + intent: &CellGroupArchiveIntent, + ) -> Result { + if !matches!(intent.action, Action::Publish { .. }) { + return Err(CellGroupArchiveError::Invalid("publication intent purpose")); + } + match self.load().await?.0.reconcile(intent)? { + Some(Allocation::Hold(hold)) => Ok(*hold), + _ => Err(CellGroupArchiveError::Expired), + } + } + + pub(super) async fn pending(&self, ticket: &CellGroupArchiveUploadTicket) -> Result { + if ticket.registry != self.identity.id { + return Err(CellGroupArchiveError::Conflict); + } + self.load() + .await? + .0 + .pending + .iter() + .find(|p| &p.ticket == ticket) + .cloned() + .ok_or(CellGroupArchiveError::Expired) + } + + pub(super) async fn protection(&self) -> Result { + let state = self.load().await?.0; + Ok(Protection { + revision: state.revision, + roots: state.roots(), + pending_generations: state.pending.iter().map(|p| p.ticket.generation).collect(), + }) + } + + pub(super) async fn referenced(&self, hold: &CellGroupArchiveHold) -> Result { + if hold.registry != self.identity.id { + return Err(CellGroupArchiveError::Conflict); + } + self.load() + .await? + .0 + .holds + .iter() + .find(|h| *h == hold) + .map(|h| h.root.clone()) + .ok_or(CellGroupArchiveError::Expired) + } + + /// A held historical root is insufficient when recovery must cover every + /// S3-acknowledged write. Require the exact independently retained head. + pub(crate) async fn require_selected_hold( + &self, + domain: &impl CellGroupArchiveExecution, + hold: &CellGroupArchiveHold, + ) -> Result<()> { + domain + .run(async { + let _io = self.archive.admit()?; + if hold.registry != self.identity.id { + return Err(CellGroupArchiveError::Conflict); + } + let state = self.load().await?.0; + if state.selected.as_ref() != Some(&hold.root) { + return Err(CellGroupArchiveError::Conflict); + } + if !state.holds.contains(hold) { + return Err(CellGroupArchiveError::Expired); + } + Ok(()) + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } + + pub(crate) async fn require_retained_hold( + &self, + domain: &impl CellGroupArchiveExecution, + hold: &CellGroupArchiveHold, + ) -> Result<()> { + domain + .run(async { + let _io = self.archive.admit()?; + self.referenced(hold).await?; + Ok(()) + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } + + pub async fn release( + &self, + domain: &impl CellGroupArchiveExecution, + hold: &CellGroupArchiveHold, + ) -> Result<()> { + if hold.registry != self.identity.id { + return Err(CellGroupArchiveError::Conflict); + } + domain + .run(self.retire(Some(hold), None)) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } + + pub(crate) async fn cancel_acquire( + &self, + domain: &impl CellGroupArchiveExecution, + intent: &CellGroupArchiveIntent, + ) -> Result<()> { + domain + .run(self.retire_allocation(|state| state.cancel_acquire(intent))) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } + + pub async fn abort( + &self, + domain: &impl CellGroupArchiveExecution, + ticket: &CellGroupArchiveUploadTicket, + ) -> Result<()> { + if ticket.registry != self.identity.id { + return Err(CellGroupArchiveError::Conflict); + } + domain + .run(self.retire(None, Some(ticket))) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } + + async fn retire( + &self, + hold: Option<&CellGroupArchiveHold>, + ticket: Option<&CellGroupArchiveUploadTicket>, + ) -> Result<()> { + self.retire_allocation(|state| state.retire(hold, ticket)) + .await + } + + async fn retire_allocation( + &self, + retire: impl Fn(&mut State) -> Result + Send, + ) -> Result<()> { + let _io = self.archive.admit()?; + for _ in 0..8 { + let (mut state, version) = self.load().await?; + if !retire(&mut state)? { + return Ok(()); + } + state.check(&self.identity)?; + let outcome = self + .write( + "registry", + encode(&state, MAX_CONTROL_BYTES)?, + PutMode::Update(version), + ) + .await; + let mut observed = self + .load() + .await + .map_err(|_| CellGroupArchiveError::Unconfirmed)? + .0; + if !retire(&mut observed)? { + return Ok(()); + } + if matches!( + outcome, + Err(CellGroupArchiveError::Storage( + object_store::Error::Precondition { .. } + | object_store::Error::AlreadyExists { .. } + )) + ) { + continue; + } + return Err(CellGroupArchiveError::Unconfirmed); + } + Err(CellGroupArchiveError::Conflict) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/archive/ownership/state.rs b/crates/cellar/durability/raft/cell_groups/archive/ownership/state.rs new file mode 100644 index 000000000..d3bc7db52 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/archive/ownership/state.rs @@ -0,0 +1,295 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Pure bounded transitions. All allocating mutations reconcile their exact +//! intent digest; retired revisions never reappear and generations never repeat. +use super::*; + +#[cfg(test)] +#[path = "state_tests.rs"] +mod tests; +use std::collections::{BTreeMap, BTreeSet}; + +#[derive(Clone, Debug, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct State { + domain: String, + identity: CellGroupArchiveCreation, + pub revision: u64, + pub pending: Vec, + pub holds: Vec, + pub selected: Option, +} + +impl State { + pub fn acquire_outcome( + &self, + intent: &CellGroupArchiveIntent, + ) -> Result { + intent.acquire_marker()?; + if let Some(Allocation::Hold(hold)) = self.reconcile(intent)? { + return Ok(CellGroupArchiveAcquireOutcome::Held(hold)); + } + Ok(if self.revision >= intent.revision { + CellGroupArchiveAcquireOutcome::Superseded + } else { + CellGroupArchiveAcquireOutcome::Pending + }) + } + + pub fn cancel_acquire(&mut self, intent: &CellGroupArchiveIntent) -> Result { + match self.acquire_outcome(intent)? { + CellGroupArchiveAcquireOutcome::Held(hold) => self.retire(Some(&hold), None), + CellGroupArchiveAcquireOutcome::Pending => { + if self.revision.checked_add(1) != Some(intent.revision) { + return Err(CellGroupArchiveError::Conflict); + } + // Consume the same revision as the pending allocation. A + // delayed acquire can no longer create a hold after cleanup. + self.advance()?; + Ok(true) + } + CellGroupArchiveAcquireOutcome::Superseded => Ok(false), + } + } + + pub fn empty(identity: CellGroupArchiveCreation) -> Self { + Self { + domain: STATE_DOMAIN.into(), + identity, + revision: 1, + pending: Vec::new(), + holds: Vec::new(), + selected: None, + } + } + + pub fn check(&self, identity: &CellGroupArchiveCreation) -> Result<()> { + if self.domain != STATE_DOMAIN + || &self.identity != identity + || self.revision == 0 + || self.revision == u64::MAX + { + return Err(CellGroupArchiveError::Invalid( + "archive registry domain/revision", + )); + } + if self.pending.len() > 8 || self.holds.len() > 64 { + return Err(CellGroupArchiveError::Limit); + } + let mut ids = BTreeSet::new(); + let mut previous = 0; + for pending in &self.pending { + let t = pending.ticket; + if t.registry != identity.id + || t.generation < 2 + || t.generation <= previous + || t.generation > self.revision + || t.allocation == [0; 32] + || !ids.insert(t.generation) + { + return Err(CellGroupArchiveError::Invalid("archive pending generation")); + } + previous = t.generation; + if let Some(parent) = &pending.parent { + parent.check(identity, t.generation)?; + } + } + previous = 0; + for hold in &self.holds { + if hold.registry != identity.id + || hold.id <= previous + || hold.id > self.revision + || hold.allocation == [0; 32] + || !ids.insert(hold.id) + { + return Err(CellGroupArchiveError::Invalid( + "archive hold identity/order", + )); + } + previous = hold.id; + hold.root.check(identity, hold.id)?; + } + let mut generations = BTreeMap::new(); + for root in self.roots() { + root.check(identity, self.revision)?; + if ids.contains(&root.object.generation) { + return Err(CellGroupArchiveError::Invalid( + "pending generation already bound", + )); + } + if generations + .insert(root.object.generation, root.clone()) + .is_some_and(|before| before != root) + { + return Err(CellGroupArchiveError::Invalid( + "archive generation descriptor conflict", + )); + } + if self.selected.as_ref().is_none_or(|selected| { + root.marker.position().index > selected.marker.position().index + }) { + return Err(CellGroupArchiveError::Invalid( + "archive retained root exceeds head", + )); + } + } + Ok(()) + } + + fn advance(&mut self) -> Result { + self.revision = self + .revision + .checked_add(1) + .filter(|n| *n != u64::MAX) + .ok_or(CellGroupArchiveError::Limit)?; + Ok(self.revision) + } + + pub fn reconcile(&self, intent: &CellGroupArchiveIntent) -> Result> { + let allocation = intent.digest()?; + if intent.registry != self.identity.id { + return Err(CellGroupArchiveError::Conflict); + } + if let Some(p) = self + .pending + .iter() + .find(|p| p.ticket.generation == intent.revision && p.ticket.allocation == allocation) + { + return Ok(Some(Allocation::Upload(p.ticket))); + } + if let Some(h) = self + .holds + .iter() + .find(|h| h.id == intent.revision && h.allocation == allocation) + { + return Ok(Some(Allocation::Hold(Box::new(h.clone())))); + } + Ok(None) + } + + pub fn allocate(&mut self, intent: &CellGroupArchiveIntent) -> Result { + if let Some(result) = self.reconcile(intent)? { + return Ok(result); + } + if intent.revision <= self.revision { + return Err(CellGroupArchiveError::Expired); + } + if self.revision.checked_add(1) != Some(intent.revision) { + return Err(CellGroupArchiveError::Conflict); + } + let allocation = intent.digest()?; + match &intent.action { + Action::Reserve { parent } => { + if parent != &self.selected { + return Err(CellGroupArchiveError::Conflict); + } + if self.pending.len() == 8 { + return Err(CellGroupArchiveError::Limit); + } + let ticket = CellGroupArchiveUploadTicket { + registry: self.identity.id, + generation: self.advance()?, + allocation, + }; + self.pending.push(Pending { + ticket, + parent: parent.clone(), + }); + Ok(Allocation::Upload(ticket)) + } + Action::Acquire { root } => { + if self.selected.as_ref() != Some(root) { + return Err(CellGroupArchiveError::Expired); + } + self.hold(root.clone(), allocation) + } + Action::Publish { ticket, root } => { + if !self.pending.iter().any(|p| &p.ticket == ticket) + || root.object.generation != ticket.generation + { + return Err(CellGroupArchiveError::Expired); + } + root.check(&self.identity, intent.revision)?; + if let Some(selected) = &self.selected { + let before = selected.marker.position(); + let after = root.marker.position(); + if after.index < before.index + || after.term < before.term + || (after.index == before.index && root.marker != selected.marker) + { + return Err(CellGroupArchiveError::Conflict); + } + } + let hold = self.hold(root.clone(), allocation)?; + self.pending.retain(|p| &p.ticket != ticket); + self.selected = Some(root.clone()); + Ok(hold) + } + } + } + + fn hold(&mut self, root: Descriptor, allocation: [u8; 32]) -> Result { + if self.holds.len() == 64 { + return Err(CellGroupArchiveError::Limit); + } + let hold = CellGroupArchiveHold { + registry: self.identity.id, + id: self.advance()?, + allocation, + root, + }; + self.holds.push(hold.clone()); + Ok(Allocation::Hold(Box::new(hold))) + } + + pub fn retire( + &mut self, + hold: Option<&CellGroupArchiveHold>, + ticket: Option<&CellGroupArchiveUploadTicket>, + ) -> Result { + let id = match (hold, ticket) { + (Some(h), None) => h.id, + (None, Some(t)) => t.generation, + _ => return Err(CellGroupArchiveError::Invalid("archive retirement purpose")), + }; + if id < 2 || id > self.revision { + return Err(CellGroupArchiveError::Conflict); + } + if hold.is_some_and(|h| self.holds.iter().any(|p| p.id == h.id && p != h)) + || ticket.is_some_and(|t| { + self.pending + .iter() + .any(|p| p.ticket.generation == t.generation && &p.ticket != t) + }) + { + return Err(CellGroupArchiveError::Conflict); + } + let before = self.pending.len() + self.holds.len(); + if let Some(hold) = hold { + self.holds.retain(|h| h != hold); + } + if let Some(ticket) = ticket { + self.pending.retain(|p| &p.ticket != ticket); + } + if before == self.pending.len() + self.holds.len() { + return Ok(false); + } + self.advance()?; + Ok(true) + } + + pub fn roots(&self) -> Vec { + let mut roots = Vec::new(); + for root in self + .selected + .iter() + .chain(self.holds.iter().map(|h| &h.root)) + .chain(self.pending.iter().filter_map(|p| p.parent.as_ref())) + { + if !roots.contains(root) { + roots.push(root.clone()); + } + } + roots + } +} diff --git a/crates/cellar/durability/raft/cell_groups/archive/ownership/state_tests.rs b/crates/cellar/durability/raft/cell_groups/archive/ownership/state_tests.rs new file mode 100644 index 000000000..428c6bcf7 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/archive/ownership/state_tests.rs @@ -0,0 +1,167 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Registry metadata tests grant no physical archive coverage. +use super::*; +use cellar_logic::cell_group::{CellRange, GroupRoute, RouteEpoch}; +use cellar_logic::consensus::{ClusterId, GroupId, IncarnationId, NamespaceId}; + +fn selected() -> (State, CellGroupArchiveIntent) { + let route = GroupRoute { + history: GroupHistory { + cluster: ClusterId::new([1; 16]).unwrap(), + namespace: NamespaceId::new([2; 16]).unwrap(), + group: GroupId::new([3; 16]).unwrap(), + incarnation: IncarnationId::new([4; 16]).unwrap(), + }, + range: CellRange::all(), + epoch: RouteEpoch::new(1).unwrap(), + birth_digest: [5; 32], + schema_digest: [6; 32], + }; + let identity = CellGroupArchiveCreation { + domain: IDENTITY_DOMAIN.into(), + id: [7; 32], + route: route.clone(), + }; + let root = Descriptor { + object: Object { + registry: identity.id, + generation: 2, + digest: [8; 32], + length: 128, + }, + marker: GroupManifest::genesis(route).unwrap(), + }; + let mut state = State::empty(identity.clone()); + state.revision = 3; + state.selected = Some(root.clone()); + state.check(&identity).unwrap(); + let intent = CellGroupArchiveIntent { + domain: INTENT_DOMAIN.into(), + registry: identity.id, + revision: 4, + nonce: [9; 32], + action: Action::Acquire { root }, + }; + (state, intent) +} + +#[test] +fn acquire_outcome_reconciles_a_lost_response_after_unrelated_allocations() { + let (mut state, intent) = selected(); + assert!(matches!( + state.acquire_outcome(&intent).unwrap(), + CellGroupArchiveAcquireOutcome::Pending + )); + let Allocation::Hold(hold) = state.allocate(&intent).unwrap() else { + panic!("acquire did not allocate a hold"); + }; + let unrelated = CellGroupArchiveIntent { + revision: 5, + nonce: [10; 32], + action: Action::Reserve { + parent: state.selected.clone(), + }, + ..intent.clone() + }; + state.allocate(&unrelated).unwrap(); + state.check(&state.identity).unwrap(); + let CellGroupArchiveAcquireOutcome::Held(observed) = state.acquire_outcome(&intent).unwrap() + else { + panic!("lost response did not reconcile"); + }; + assert_eq!(*observed, *hold); + assert_eq!(state.holds.len(), 1); +} + +#[test] +fn acquire_outcome_requires_a_consumed_revision_before_reallocation() { + let (mut state, intent) = selected(); + let competing = CellGroupArchiveIntent { + nonce: [11; 32], + action: Action::Reserve { + parent: state.selected.clone(), + }, + ..intent.clone() + }; + assert!(matches!( + state.acquire_outcome(&intent).unwrap(), + CellGroupArchiveAcquireOutcome::Pending + )); + state.allocate(&competing).unwrap(); + assert!(matches!( + state.acquire_outcome(&intent).unwrap(), + CellGroupArchiveAcquireOutcome::Superseded + )); + assert!(matches!( + state.allocate(&intent), + Err(CellGroupArchiveError::Expired) + )); + assert!(state.holds.is_empty()); +} + +#[test] +fn acquire_outcome_refuses_foreign_registry_and_other_allocation_purposes() { + let (state, mut intent) = selected(); + intent.registry = [12; 32]; + assert!(matches!( + state.acquire_outcome(&intent), + Err(CellGroupArchiveError::Conflict) + )); + intent.registry = state.identity.id; + intent.action = Action::Reserve { + parent: state.selected.clone(), + }; + assert!(state.acquire_outcome(&intent).is_err()); +} + +#[test] +fn cancelling_an_acquire_fences_a_delayed_allocation() { + let (mut state, intent) = selected(); + assert!(state.cancel_acquire(&intent).unwrap()); + assert_eq!(state.revision, intent.revision); + assert!(matches!( + state.allocate(&intent), + Err(CellGroupArchiveError::Expired) + )); + assert!(!state.cancel_acquire(&intent).unwrap()); + assert!(state.holds.is_empty()); + state.check(&state.identity).unwrap(); +} + +#[test] +fn cancelling_a_committed_acquire_releases_only_its_hold() { + let (mut state, intent) = selected(); + state.allocate(&intent).unwrap(); + let unrelated = CellGroupArchiveIntent { + revision: intent.revision + 1, + nonce: [10; 32], + ..intent.clone() + }; + let Allocation::Hold(other) = state.allocate(&unrelated).unwrap() else { + panic!("unrelated acquire did not allocate a hold"); + }; + assert!(state.cancel_acquire(&intent).unwrap()); + assert_eq!(state.holds, vec![*other]); + assert!(!state.cancel_acquire(&intent).unwrap()); + assert!(matches!( + state.allocate(&intent), + Err(CellGroupArchiveError::Expired) + )); + state.check(&state.identity).unwrap(); +} + +#[test] +fn cancelling_a_superseded_acquire_preserves_the_same_revision_winner() { + let (mut state, intent) = selected(); + let unrelated = CellGroupArchiveIntent { + nonce: [10; 32], + ..intent.clone() + }; + let Allocation::Hold(other) = state.allocate(&unrelated).unwrap() else { + panic!("unrelated acquire did not allocate a hold"); + }; + assert!(!state.cancel_acquire(&intent).unwrap()); + assert_eq!(state.holds, vec![*other]); + state.check(&state.identity).unwrap(); +} diff --git a/crates/cellar/durability/raft/cell_groups/archive/ownership/wire.rs b/crates/cellar/durability/raft/cell_groups/archive/ownership/wire.rs new file mode 100644 index 000000000..ba0b75d35 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/archive/ownership/wire.rs @@ -0,0 +1,80 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Explicit host-side route codec; the dependency-free decision types do not +//! acquire serde or I/O dependencies. Shares canonical range-bound validation. +use super::*; +use crate::raft::cell_groups::root::Bound; +use cellar_logic::{cell_group::*, consensus::*}; + +#[derive(Clone, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct WireCreation { + domain: String, + id: [u8; 32], + cluster: [u8; 16], + namespace: [u8; 16], + group: [u8; 16], + incarnation: [u8; 16], + lower: Bound, + upper: Bound, + epoch: u64, + birth: [u8; 32], + schema: [u8; 32], +} + +impl From for WireCreation { + fn from(value: CellGroupArchiveCreation) -> Self { + let route = value.route; + Self { + domain: value.domain, + id: value.id, + cluster: route.history.cluster.bytes(), + namespace: route.history.namespace.bytes(), + group: route.history.group.bytes(), + incarnation: route.history.incarnation.bytes(), + lower: Bound::encode(route.range.lower()), + upper: Bound::encode(route.range.upper()), + epoch: route.epoch.get(), + birth: route.birth_digest, + schema: route.schema_digest, + } + } +} + +impl TryFrom for CellGroupArchiveCreation { + type Error = CellGroupArchiveError; + + fn try_from(value: WireCreation) -> Result { + let invalid = |_| CellGroupArchiveError::Invalid("archive registry route identity"); + let namespace = NamespaceId::new(value.namespace).map_err(invalid)?; + let route = GroupRoute { + history: GroupHistory { + cluster: ClusterId::new(value.cluster).map_err(invalid)?, + namespace, + group: GroupId::new(value.group).map_err(invalid)?, + incarnation: IncarnationId::new(value.incarnation).map_err(invalid)?, + }, + range: CellRange::new( + value + .lower + .decode(namespace) + .map_err(CellGroupCheckpointError::from)?, + value + .upper + .decode(namespace) + .map_err(CellGroupCheckpointError::from)?, + ) + .map_err(invalid)?, + epoch: RouteEpoch::new(value.epoch).map_err(invalid)?, + birth_digest: value.birth, + schema_digest: value.schema, + }; + let result = Self { + domain: value.domain, + id: value.id, + route, + }; + result.check(result.route.history)?; + Ok(result) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/archive/publication.rs b/crates/cellar/durability/raft/cell_groups/archive/publication.rs new file mode 100644 index 000000000..12f7705aa --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/archive/publication.rs @@ -0,0 +1,334 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Source-authorized, physically verified publication. Neither a serialized +//! locator nor a registry allocation can substitute for these two proofs. +use super::super::PreparedGroupManifest; +use super::{ + graph::{Graph, Root, ROOT_DOMAIN}, + ownership::{Allocation, Descriptor}, + *, +}; +use crate::raft::{cache::GroupLog, member_state::GroupMembership}; +use raft::prelude::ConfState; + +pub(super) struct SourceCut { + group: GroupLog, + root: PreparedGroupManifest, + configuration: ConfState, + membership: GroupMembership, + _reservation: Arc, +} + +impl SourceCut { + pub(super) fn from_capture( + group: GroupLog, + root: PreparedGroupManifest, + configuration: ConfState, + membership: GroupMembership, + source_budget: Arc, + ) -> Result { + root.images.visit(&mut |_, _, pin, _, _| -> Result<()> { + if matches!(pin, super::super::CellImagePin::Extents(pin) if pin.reference().encoded_length() > 1024 * 1024) { + return Err(CellGroupArchiveError::Limit); + } + Ok(()) + })?; + let mut source_bytes = root + .reserved_bytes() + .checked_add( + crate::raft::manifest::encode_conf(&configuration) + .map_err(|_| CellGroupArchiveError::Invalid("tail configuration encoding"))? + .len() + .checked_mul(4) + .ok_or(CellGroupArchiveError::Limit)?, + ) + .and_then(|n| n.checked_add(64 * 1024)) + .ok_or(CellGroupArchiveError::Limit)?; + source_bytes = source_bytes + .checked_add( + serde_json::to_vec(&membership) + .map_err(|_| CellGroupArchiveError::Limit)? + .len() + .checked_mul(4) + .ok_or(CellGroupArchiveError::Limit)?, + ) + .ok_or(CellGroupArchiveError::Limit)?; + root.images.visit(&mut |_, _, pin, _, _| -> Result<()> { + if let super::super::CellImagePin::Extents(pin) = pin { + source_bytes = source_bytes + .checked_add( + (pin.reference().encoded_length() as usize) + .checked_mul(2) + .ok_or(CellGroupArchiveError::Limit)?, + ) + .ok_or(CellGroupArchiveError::Limit)?; + } + Ok(()) + })?; + let source_reservation = source_budget + .acquire(source_bytes) + .map(Arc::new) + .ok_or(CellGroupArchiveError::Busy)?; + Ok(Self { + group, + root, + configuration, + membership, + _reservation: source_reservation, + }) + } + + fn check(&self) -> Result<()> { + let cache = self.group.read().map_err(CellGroupCheckpointError::from)?; + let target = self.root.marker().position(); + let current = cache + .group_manifest + .as_ref() + .ok_or(CellGroupArchiveError::Invalid( + "publication source root missing", + ))?; + if cache.applied.position.index < target.index + || cache.hard_state.commit < target.index.get() + || current.route() != self.root.marker().route() + || (current.position().index == target.index && current != self.root.marker()) + || cache + .term(target.index.get()) + .is_some_and(|term| term != target.term.get()) + { + return Err(CellGroupArchiveError::Invalid( + "archive publication source changed", + )); + } + // A later local checkpoint may compact the captured prefix. Its initial + // committed-source observation remains sealed to this live engine. + Ok(()) + } +} + +/// Persist `intent()` before commit. Holds the exact source and independently +/// recovered closure through CAS/readback; it cannot be deserialized or forged. +pub struct PreparedCellGroupArchivePublication { + upload: CellGroupArchiveUpload, + intent: CellGroupArchiveIntent, + verified: VerifiedCellGroupArchiveRecovery, + source: Arc, +} + +impl PreparedCellGroupArchivePublication { + pub fn intent(&self) -> &CellGroupArchiveIntent { + &self.intent + } + + pub async fn commit( + self, + domain: &impl CellGroupArchiveExecution, + ) -> Result { + domain + .run(async { + let _io = self.upload.admit()?; + self.source.check()?; + let hold = match self.upload.registry().commit(&self.intent).await? { + Allocation::Hold(hold) => *hold, + _ => { + return Err(CellGroupArchiveError::Invalid( + "publication allocation purpose", + )) + } + }; + self.source.check()?; + Ok(CellGroupArchiveCoverage(Arc::new(Coverage { + hold, + verified: self.verified, + }))) + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } +} + +struct Coverage { + hold: CellGroupArchiveHold, + verified: VerifiedCellGroupArchiveRecovery, +} + +pub(in crate::raft::cell_groups) struct CoverageUse { + hold: CellGroupArchiveHold, + coverage: std::sync::Weak, +} + +impl CoverageUse { + pub(in crate::raft::cell_groups) fn retains(&self, hold: &CellGroupArchiveHold) -> bool { + &self.hold == hold && self.coverage.strong_count() != 0 + } + + pub(in crate::raft::cell_groups) fn live(&self) -> bool { + self.coverage.strong_count() != 0 + } +} + +/// Historical complete data coverage with an explicit remote hold. Clones keep +/// local verification admission/pins. The remote hold is retired explicitly by +/// its durable owner after joined use; this grants no quorum or serving right. +#[derive(Clone)] +pub struct CellGroupArchiveCoverage(Arc); + +impl CellGroupArchiveCoverage { + pub(in crate::raft::cell_groups) fn retain_use(&self) -> CoverageUse { + CoverageUse { + hold: self.0.hold.clone(), + coverage: Arc::downgrade(&self.0), + } + } + + pub fn hold(&self) -> &CellGroupArchiveHold { + &self.0.hold + } + + pub fn recovery(&self) -> &VerifiedCellGroupArchiveRecovery { + &self.0.verified + } + + pub fn marker(&self) -> &GroupManifest { + self.0.verified.manifest().marker() + } +} + +impl CellGroupArchiveUpload { + /// Recover every dependency and replay every entry, then prepare one exact + /// conditional publication intent. Missing B prevents coverage of later A. + /// Callbacks, engine installation and S3 command settlement do not run here. + pub async fn prepare_publication( + &self, + domain: &impl CellGroupArchiveExecution, + tails: &CellGroupArchiveTailPool, + uploaded: &CellGroupArchiveUploadedTail, + receiver: CellGroupArchiveCheckpointReceiver, + ) -> Result { + domain + .run(async { + let source = uploaded.source.clone(); + source.check()?; + if source.root.marker().route() != self.registry().route() { + return Err(CellGroupArchiveError::Invalid( + "publication registry/source route", + )); + } + { + let _io = self.admit()?; + self.registry().pending(&self.ticket()).await?; + } + let reference = uploaded.reference(); + let verified = self.verify_tail(domain, tails, reference, receiver).await?; + if verified.manifest().marker() != source.root.marker() + || verified.configuration() != &source.configuration + || verified.membership() != &source.membership + { + return Err(CellGroupArchiveError::Invalid( + "archive publication exact cut", + )); + } + source.check()?; + let io = self.admit()?; + let root = Root { + domain: ROOT_DOMAIN.into(), + marker: source.root.marker().clone(), + tail: reference.clone(), + }; + let object = self + .put_generation( + self.registry().id(), + self.ticket().generation(), + "roots", + serde_json::to_vec(&root).map_err(|_| CellGroupArchiveError::Limit)?, + 8192, + ) + .await?; + let descriptor = Descriptor { + object, + marker: root.marker, + }; + let graph = Graph::load( + self, + domain, + &descriptor, + tails.limits, + io.clone(), + self.retained(), + self.max_objects(), + ) + .await?; + self.allows(&graph)?; + let intent = self + .registry() + .prepare_publish(self.ticket(), descriptor) + .await?; + Ok(PreparedCellGroupArchivePublication { + upload: self.clone(), + intent, + verified, + source, + }) + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } +} + +impl CellGroupArchiveRegistry { + /// Reconcile only a publication already present under its exact persisted + /// intent. This never sends a publishing CAS or substitutes a newer head. + pub async fn reconcile_publication( + &self, + domain: &impl CellGroupArchiveExecution, + intent: &CellGroupArchiveIntent, + tails: &CellGroupArchiveTailPool, + receiver: CellGroupArchiveCheckpointReceiver, + ) -> Result { + let hold = domain + .run(async { + let _io = self.archive.admit()?; + self.publication_hold(intent).await + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)??; + self.verify_hold(domain, &hold, tails, receiver).await + } + + /// Read only the exact held immutable root, independently recover all data, + /// and recheck the persistent hold before returning historical coverage. + pub async fn verify_hold( + &self, + domain: &impl CellGroupArchiveExecution, + hold: &CellGroupArchiveHold, + tails: &CellGroupArchiveTailPool, + receiver: CellGroupArchiveCheckpointReceiver, + ) -> Result { + domain + .run(async { + let io = self.archive.admit()?; + let descriptor = self.referenced(hold).await?; + let bytes = self.archive.get("roots", &descriptor.object, 8192).await?; + let root: Root = serde_json::from_slice(&bytes) + .map_err(|_| CellGroupArchiveError::Invalid("held root encoding"))?; + if root.domain != ROOT_DOMAIN || root.marker != descriptor.marker { + return Err(CellGroupArchiveError::Invalid("held root binding")); + } + drop(io); + let verified = self + .archive + .verify_tail(domain, tails, &root.tail, receiver) + .await?; + if verified.manifest().marker() != &root.marker { + return Err(CellGroupArchiveError::Invalid("held recovery cut")); + } + let _io = self.archive.admit()?; + self.referenced(hold).await?; + Ok(CellGroupArchiveCoverage(Arc::new(Coverage { + hold: hold.clone(), + verified, + }))) + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } +} diff --git a/crates/cellar/durability/raft/cell_groups/archive/receipt.rs b/crates/cellar/durability/raft/cell_groups/archive/receipt.rs new file mode 100644 index 000000000..859cea6b2 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/archive/receipt.rs @@ -0,0 +1,181 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Read the exact receipt from independently recovered, immutable archive bytes. +use super::*; +use crate::raft::cell_groups::{CellGroupApplyImages, CellImagePin, CellSqlReceiptLookup}; +use crate::raft::execution::{ + CellCommittedReceipt, CommandRequest, ExecutionError, ExecutionLimits, +}; +use std::sync::atomic::{AtomicBool, Ordering}; + +pub(in crate::raft::cell_groups) fn command( + coverage: &CellGroupArchiveCoverage, + store: &CellGroupArchiveStore, + expected: &CellCommittedReceipt, + mut limits: ExecutionLimits, + deadline: u64, + cancelled: Arc, +) -> Result { + store.require_live()?; + let root = coverage.recovery().manifest(); + if root.marker().route() != store.route() || !store.owner.owns(root) { + return Err(CellGroupArchiveError::Invalid( + "S3 receipt verifier owner/route", + )); + } + if root.marker().position().index < expected.position.index { + return Ok(false); + } + let (leaf, pin) = root + .pin_cell(&expected.version.history.cell) + .ok_or(CellGroupArchiveError::Invalid("S3 receipt cell missing"))?; + if leaf.version.history != expected.version.history { + return Err(CellGroupArchiveError::Invalid("S3 receipt history")); + } + if leaf.version.sequence < expected.version.sequence { + return Ok(false); + } + if leaf.version == expected.version && leaf.witness != expected.witness { + return Err(CellGroupArchiveError::Invalid("S3 receipt decision")); + } + let database = open(store, pin, deadline, cancelled)?; + // The pending output already owns the original bytes. Bound the additional + // row allocation before lookup; identity and the entire retained row match. + limits.max_result_bytes = expected.receipt.result().len(); + let request = CommandRequest { + id: expected.receipt.id, + operation: expected.receipt.operation.clone(), + contract_digest: expected.receipt.contract_digest, + request_digest: expected.receipt.request_digest, + durability: expected.receipt.minimum_durability, + retry_until_ms: expected.receipt.retry_until_ms, + }; + let receipt = crate::raft::receipts::lookup_cell_identity( + database.connection(), + &request, + &limits, + leaf.version.sequence.get(), + ) + .map_err(CellGroupArchiveError::Replay)?; + if receipt.as_ref() != Some(&expected.receipt) { + return Err(CellGroupArchiveError::Invalid("S3 receipt identity/result")); + } + store.require_live()?; + Ok(true) +} + +pub(in crate::raft::cell_groups) struct LookupRead<'a> { + pub request: &'a CommandRequest, + pub limits: ExecutionLimits, + pub deadline: u64, + pub cancelled: Arc, +} + +/// Return the actual archived observation at or beyond the quorum selection. +/// Absence remains ambiguous; it never resolves a submitted producer. +pub(in crate::raft::cell_groups) fn lookup( + coverage: &CellGroupArchiveCoverage, + store: &CellGroupArchiveStore, + selected: &mut CellSqlReceiptLookup, + read: LookupRead<'_>, + reserve_result: impl FnMut(usize) -> std::result::Result<(), ExecutionError>, +) -> Result { + store.require_live()?; + let root = coverage.recovery().manifest(); + if root.marker().route() != store.route() || !store.owner.owns(root) { + return Err(CellGroupArchiveError::Invalid( + "S3 lookup verifier owner/route", + )); + } + let floor = selected.observation(); + let Some((leaf, pin)) = root.pin_cell(&floor.version.history.cell) else { + return Ok(false); + }; + if leaf.version.history != floor.version.history { + return Err(CellGroupArchiveError::Invalid("S3 lookup history")); + } + if leaf.version.sequence < floor.version.sequence { + return Ok(false); + } + if leaf.version == floor.version && leaf.witness != floor.witness { + return Err(CellGroupArchiveError::Invalid("S3 lookup decision")); + } + let database = open(store, pin, read.deadline, read.cancelled)?; + let receipt = crate::raft::receipts::lookup_cell_identity_admitted( + database.connection(), + read.request, + &read.limits, + leaf.version.sequence.get(), + reserve_result, + ) + .map_err(CellGroupArchiveError::Replay)?; + // An equal logical version cannot disagree with the quorum read. A known + // retained result also remains exact across later images; expiry stays typed. + if (leaf.version == floor.version || selected.receipt().is_some()) + && receipt.as_ref() != selected.receipt() + { + return Err(CellGroupArchiveError::Invalid("S3 lookup identity/result")); + } + store.require_live()?; + selected.select_archived(receipt, leaf.observation()); + Ok(true) +} + +fn open( + store: &CellGroupArchiveStore, + pin: CellImagePin, + deadline: u64, + cancelled: Arc, +) -> Result { + let images = store + .storage + .try_lock() + .map_err(|_| CellGroupArchiveError::Busy)? + .images + .clone(); + let database = match (images, pin) { + (CellGroupApplyImages::Materialized { services, .. }, CellImagePin::Materialized(pin)) => { + crate::storage::open_immutable_generation( + pin, + crate::storage::ImmutableQueryOptions { + services, + sqlite_vec: false, + }, + ) + } + ( + CellGroupApplyImages::Extents { + services, + cache_directory, + cache_pool, + cache, + .. + }, + CellImagePin::Extents(pin), + ) => { + let image = pin + .view() + .image(cache_directory, cache_pool, cache) + .map_err(|e| CellGroupArchiveError::Replay(crate::raft::execution::storage(e)))?; + crate::storage::open_immutable_extent_image( + image, + crate::storage::ImmutableQueryOptions { + services, + sqlite_vec: false, + }, + ) + } + _ => return Err(CellGroupArchiveError::Invalid("S3 receipt image")), + } + .map_err(|e| CellGroupArchiveError::Replay(crate::raft::execution::storage(e)))?; + database + .connection() + .progress_handler( + 1000, + Some(move || { + cancelled.load(Ordering::Acquire) || crate::asyncrt::mono_ms() >= deadline + }), + ) + .map_err(|e| CellGroupArchiveError::Replay(crate::raft::execution::storage(e)))?; + Ok(database) +} diff --git a/crates/cellar/durability/raft/cell_groups/archive/replay.rs b/crates/cellar/durability/raft/cell_groups/archive/replay.rs new file mode 100644 index 000000000..4fe102d16 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/archive/replay.rs @@ -0,0 +1,227 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Non-serving data replay. Never constructs Raft tickets, changes a hard state, +//! installs a snapshot, invokes application callbacks or grants a quorum fence. +use super::super::{apply::Storage, CellGroupApplyImages, CellImagePin, PreparedGroupManifest}; +use super::{tail::Cut, *}; +use crate::raft::member_state::GroupMembership; +use raft::prelude::{ConfState, Entry, EntryType}; +use std::sync::atomic::{AtomicBool, Ordering}; + +/// Complete receiver-local state after independently replaying the selected +/// checkpoint and contiguous tail. Keeps physical pins and replay admission; +/// this supplies no published head, GC lease, serving right or S3 settlement. +pub struct VerifiedCellGroupArchiveRecovery { + reference: CellGroupArchiveTailReference, + checkpoint: VerifiedCellGroupArchiveCheckpoint, + root: PreparedGroupManifest, + configuration: ConfState, + membership: GroupMembership, + _reservation: Arc, +} + +impl VerifiedCellGroupArchiveRecovery { + pub fn reference(&self) -> &CellGroupArchiveTailReference { + &self.reference + } + + pub fn checkpoint(&self) -> &VerifiedCellGroupArchiveCheckpoint { + &self.checkpoint + } + + pub fn manifest(&self) -> &PreparedGroupManifest { + &self.root + } + + pub fn configuration(&self) -> &ConfState { + &self.configuration + } + + pub fn membership(&self) -> &GroupMembership { + &self.membership + } + + /// Static unmanaged recovery only. A fence controller must independently + /// exclude later joins and retired hosts from its deployment inventory. + pub(crate) fn check_disaster_target( + &self, + target: &crate::raft::catalog::CellGroupBirthRequest, + ) -> Result<()> { + if !self.configuration.voters_outgoing.is_empty() + || !self.configuration.learners_next.is_empty() + || self.configuration.auto_leave + || self + .membership + .repair_status() + .map_err(CellGroupCheckpointError::from)? + .is_some() + || self.membership.fleet_status().is_some() + { + return Err(CellGroupArchiveError::Invalid( + "recovery requires static unmanaged membership", + )); + } + for member in target.voters().iter().chain(target.learners()) { + if member.member.get() <= self.membership.high_water() + || self.membership.has_node_binding(member.node) + { + return Err(CellGroupArchiveError::Invalid( + "reused recovery member or node", + )); + } + } + self.check_recovery_cell_incarnation(target.route().history.incarnation) + } + + pub(crate) fn check_recovery_cell_incarnation( + &self, + incarnation: cellar_logic::consensus::IncarnationId, + ) -> Result<()> { + self.root.images.visit(&mut |_, leaf, _, _, _| { + if leaf.version.history.incarnation == incarnation { + return Err(CellGroupArchiveError::Invalid( + "reused recovery cell incarnation", + )); + } + Ok(()) + }) + } +} + +pub(super) fn replay( + reference: CellGroupArchiveTailReference, + checkpoint: VerifiedCellGroupArchiveCheckpoint, + cut: Cut, + entries: Vec, + limits: CellGroupArchiveTailLimits, + reservation: Arc, + cancelled: Arc, +) -> Result { + let store = checkpoint.checkpoint().store().clone(); + store.require_live()?; + let mut root = checkpoint.checkpoint().manifest().clone(); + if !store.owner.owns(&root) { + return Err(CellGroupArchiveError::Invalid( + "archive replay physical owner", + )); + } + let mut conf = checkpoint + .checkpoint() + .snapshot() + .get_metadata() + .get_conf_state() + .clone(); + let mut membership = checkpoint.checkpoint().membership().clone(); + let mut owner = store.storage.try_lock().map_err(|error| match error { + std::sync::TryLockError::WouldBlock => CellGroupArchiveError::Busy, + std::sync::TryLockError::Poisoned(_) => { + CellGroupArchiveError::Invalid("poisoned archive replay owner") + } + })?; + if owner.images.memory_bound() > limits.max_apply_bytes + || root.pinned_images() > limits.max_cells as usize + { + return Err(CellGroupArchiveError::Limit); + } + for entry in &entries { + if cancelled.load(Ordering::Acquire) { + return Err(CellGroupArchiveError::Stopped); + } + store.require_live()?; + // Control replay is the same canonical pure transition used by the live + // owner; cell bytes still go through the ordinary LTX/base-chain checks. + let change = if entry.get_entry_type() == EntryType::EntryConfChangeV2 { + Some( + super::super::membership::transition( + entry, + root.marker().route(), + &conf, + &membership, + ) + .map_err(CellGroupCheckpointError::from)?, + ) + } else { + None + }; + root = owner + .prepare_entry(&root, entry, Some((&conf, &membership)), true) + .map_err(CellGroupArchiveError::Replay)?; + if root.pinned_images() > limits.max_cells as usize { + return Err(CellGroupArchiveError::Limit); + } + if let Some(change) = change { + conf = change.conf; + membership = change.membership; + } + } + if cancelled.load(Ordering::Acquire) { + return Err(CellGroupArchiveError::Stopped); + } + let encoded = crate::raft::manifest::encode_conf(&conf) + .map_err(|_| CellGroupArchiveError::Invalid("replayed configuration"))?; + if root.marker() != &cut.marker || encoded != cut.conf || membership != cut.membership { + return Err(CellGroupArchiveError::Invalid("replayed complete cut")); + } + // Recheck every final physical image and page, even after warm replay. An + // in-memory pin or a cached destination is never a substitute for bytes. + let Storage { + tree, + pages, + images, + .. + } = &mut *owner; + let verified = tree + .prepare_archive_complete(&store.owner, pages, root.marker().clone(), &mut |leaf| { + if cancelled.load(Ordering::Acquire) { + return Err(super::super::ManifestError::Invalid( + "archive replay stopped", + )); + } + let (_, pin) = root + .pin_cell(&leaf.version.history.cell) + .filter(|(expected, _)| *expected == leaf) + .ok_or(super::super::ManifestError::Invalid( + "replayed leaf missing", + ))?; + match (&*images, &pin) { + (_, CellImagePin::Empty) => (), + (CellGroupApplyImages::Materialized { .. }, CellImagePin::Materialized(pin)) => { + pin.verify_and_sync() + .map_err(|e| super::super::ManifestError::Io(std::io::Error::other(e)))?; + } + (CellGroupApplyImages::Extents { images, .. }, CellImagePin::Extents(pin)) => { + if pin.reference().encoded_length() > 1024 * 1024 { + return Err(super::super::ManifestError::Invalid( + "replay extent metadata limit", + )); + } + images + .seal_image_root( + pin.manifest().clone(), + &mut vec![0; cellar_ltx::internal::EXTENT_BYTES as usize], + ) + .map_err(|e| super::super::ManifestError::Io(std::io::Error::other(e)))?; + } + _ => { + return Err(super::super::ManifestError::Invalid( + "replay representation", + )) + } + } + Ok(pin) + }) + .map_err(CellGroupCheckpointError::from)?; + if cancelled.load(Ordering::Acquire) { + return Err(CellGroupArchiveError::Stopped); + } + store.require_live()?; + drop(owner); + Ok(VerifiedCellGroupArchiveRecovery { + reference, + checkpoint, + root: verified, + configuration: conf, + membership, + _reservation: reservation, + }) +} diff --git a/crates/cellar/durability/raft/cell_groups/archive/tail.rs b/crates/cellar/durability/raft/cell_groups/archive/tail.rs new file mode 100644 index 000000000..aa789c3ae --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/archive/tail.rs @@ -0,0 +1,605 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::super::PreparedGroupManifest; +use super::{objects::Object, *}; +use crate::raft::{cache::GroupLog, member_state::GroupMembership}; +use raft::prelude::ConfState; +use std::sync::atomic::{AtomicBool, Ordering}; + +const TAIL_DOMAIN: &str = "cellar-cell-group-archive-tail-1"; + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +/// An untrusted immutable-tail locator; recovery must verify its entire closure. +pub struct CellGroupArchiveTailReference { + registry: [u8; 32], + generation: u64, + digest: [u8; 32], + length: u64, +} + +impl CellGroupArchiveTailReference { + pub fn registry(&self) -> [u8; 32] { + self.registry + } + + pub fn generation(&self) -> u64 { + self.generation + } + + pub(super) fn object(&self) -> Object { + Object { + registry: self.registry, + generation: self.generation, + digest: self.digest, + length: self.length, + } + } + + pub fn digest(&self) -> [u8; 32] { + self.digest + } + + pub fn length(&self) -> u64 { + self.length + } +} + +/// An upload's exact committed-source witness, captured before asynchronous I/O. +/// This value cannot be deserialized. Persist its untrusted reference separately; +/// publication needs this witness or reconciliation of an existing publication. +#[derive(Clone)] +pub struct CellGroupArchiveUploadedTail { + reference: CellGroupArchiveTailReference, + pub(super) source: Arc, +} + +impl CellGroupArchiveUploadedTail { + pub fn reference(&self) -> &CellGroupArchiveTailReference { + &self.reference + } + + pub fn into_reference(self) -> CellGroupArchiveTailReference { + self.reference + } +} + +impl std::ops::Deref for CellGroupArchiveUploadedTail { + type Target = CellGroupArchiveTailReference; + + fn deref(&self) -> &Self::Target { + &self.reference + } +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Cut { + pub marker: GroupManifest, + pub conf: Vec, + pub membership: GroupMembership, +} + +impl Cut { + pub(super) fn configuration(&self) -> Result { + crate::raft::manifest::decode_conf(&self.conf) + .map_err(|_| CellGroupArchiveError::Invalid("tail configuration")) + } + + fn validate(&self, history: GroupHistory, limits: CellGroupArchiveTailLimits) -> Result<()> { + self.marker + .check() + .map_err(CellGroupCheckpointError::from)?; + if self.marker.route().history != history + || self.marker.root().cells() > limits.max_cells + || self.conf.len() > 64 * 1024 + || self.membership.revision().get() != self.marker.membership_revision() + { + return Err(CellGroupArchiveError::Invalid("tail cut binding/bounds")); + } + self.membership + .validate(&self.configuration()?, self.marker.position().index.get()) + .map_err(CellGroupCheckpointError::from)?; + Ok(()) + } +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Tail { + domain: String, + pub checkpoint: CellGroupArchiveReference, + pub base: GroupManifest, + pub cut: Cut, + bytes: u64, + pub entries: Vec, +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct EntryObject { + index: u64, + term: u64, + pub object: Object, +} + +impl Tail { + fn validate( + &self, + archive: &CellGroupArchive, + limits: CellGroupArchiveTailLimits, + ) -> Result<()> { + self.base.check().map_err(CellGroupCheckpointError::from)?; + self.cut.validate(archive.history, limits)?; + if self.domain != TAIL_DOMAIN + || self.base.route() != self.cut.marker.route() + || self.checkpoint.length == 0 + || self.checkpoint.length > archive.limits.max_index_bytes as u64 + || self.entries.len() > limits.max_entries + || self.bytes > limits.max_tail_bytes as u64 + || self + .cut + .marker + .position() + .index + .get() + .checked_sub(self.base.position().index.get()) + != Some(self.entries.len() as u64) + { + return Err(CellGroupArchiveError::Invalid("tail domain/cuts/limits")); + } + let mut index = self.base.position().index.get(); + let mut term = self.base.position().term.get(); + let mut bytes = 0u64; + for entry in &self.entries { + index = index.checked_add(1).ok_or(CellGroupArchiveError::Limit)?; + if entry.index != index + || entry.term < term + || entry.object.length == 0 + || entry.object.length > limits.max_tail_bytes as u64 + { + return Err(CellGroupArchiveError::Invalid("tail continuity")); + } + term = entry.term; + bytes = bytes + .checked_add(entry.object.length) + .ok_or(CellGroupArchiveError::Limit)?; + } + if bytes != self.bytes || term != self.cut.marker.position().term.get() { + return Err(CellGroupArchiveError::Invalid("tail total/final term")); + } + Ok(()) + } +} + +struct Capture { + pub cut: Cut, + entries: Vec<(u64, u64, Vec)>, + authority: Arc, +} + +fn capture( + group: GroupLog, + base: &GroupManifest, + current: PreparedGroupManifest, + limits: CellGroupArchiveTailLimits, + source_budget: Arc, +) -> Result { + if !Arc::ptr_eq(&group.owner, ¤t.engine_owner) + || base.route() != current.marker().route() + || current.marker().root().cells() > limits.max_cells + { + return Err(CellGroupArchiveError::Invalid("tail source owner/route")); + } + let cache = group.read().map_err(CellGroupCheckpointError::from)?; + let target = current.marker().position(); + let count = target + .index + .get() + .checked_sub(base.position().index.get()) + .filter(|n| *n <= limits.max_entries as u64) + .ok_or(CellGroupArchiveError::Limit)?; + if cache.group_manifest.as_ref() != Some(current.marker()) + || cache.applied.position != target + || cache.applied.image.is_some() + || cache.hard_state.commit < target.index.get() + || cache.term(target.index.get()) != Some(target.term.get()) + || cache.term(base.position().index.get()) != Some(base.position().term.get()) + { + return Err(CellGroupArchiveError::Invalid("tail source committed cut")); + } + let mut entries = Vec::with_capacity(count as usize); + let mut bytes = 0usize; + let mut index = base.position().index.get(); + for _ in 0..count { + index += 1; + let entry = cache + .entries + .get(&index) + .ok_or(CellGroupArchiveError::Invalid( + "tail source log hole/compaction", + ))?; + bytes = bytes + .checked_add(tail_codec::length(entry)?) + .filter(|n| *n <= limits.max_tail_bytes) + .ok_or(CellGroupArchiveError::Limit)?; + entries.push(entry.clone()); + } + let cut = Cut { + marker: current.marker().clone(), + conf: crate::raft::manifest::encode_conf(&cache.conf_state) + .map_err(|_| CellGroupArchiveError::Invalid("tail configuration encoding"))?, + membership: cache.membership.clone(), + }; + let configuration = cache.conf_state.clone(); + let membership = cache.membership.clone(); + drop(cache); // No engine/cache lock across encoding or physical/S3 work. + cut.validate(group.identity().scope, limits)?; + let entries = entries + .iter() + .map(|entry| { + Ok(( + entry.index, + entry.term, + tail_codec::encode(entry, limits.max_tail_bytes)?, + )) + }) + .collect::>>()?; + Ok(Capture { + cut, + entries, + authority: Arc::new(super::publication::SourceCut::from_capture( + group, + current, + configuration, + membership, + source_budget, + )?), + }) +} + +enum Origin<'a> { + Checkpoint(&'a CellGroupArchiveReference), + Tail(&'a CellGroupArchiveTailReference), +} + +struct Stop(Arc); + +impl Drop for Stop { + fn drop(&mut self) { + self.0.store(true, Ordering::Release); + } +} + +impl CellGroupArchiveUpload { + /// Upload one complete captured cut without recapturing the live engine + /// after object-store I/O. Quorum may continue applying throughout upload. + /// The checkpoint pool must admit the source before this call; its slot is + /// released before the caller starts independent recovery verification. + pub async fn upload_cut( + &self, + domain: &impl CellGroupArchiveExecution, + pool: &CellGroupArchiveTailPool, + checkpoint: CellGroupCheckpoint, + ) -> Result { + domain + .run(async { + let _retained = pool.admit()?; + let (group, root, configuration, membership) = checkpoint.archive_source()?; + let cut = Cut { + marker: root.marker().clone(), + conf: crate::raft::manifest::encode_conf(&configuration).map_err(|_| { + CellGroupArchiveError::Invalid("tail configuration encoding") + })?, + membership: membership.clone(), + }; + cut.validate(self.history, pool.limits)?; + let source = Arc::new(super::publication::SourceCut::from_capture( + group, + root, + configuration, + membership, + self.source_budget(), + )?); + let reference = self.upload_checkpoint(domain, checkpoint).await?; + let io = self.admit()?; + self.registry().pending(&self.ticket()).await?; + let tail = Tail { + domain: TAIL_DOMAIN.into(), + checkpoint: reference, + base: cut.marker.clone(), + cut, + bytes: 0, + entries: Vec::new(), + }; + tail.validate(self, pool.limits)?; + let mut output = checkpoint::Bounded { + bytes: Vec::new(), + limit: self.limits.max_index_bytes, + }; + serde_json::to_writer(&mut output, &tail) + .map_err(|_| CellGroupArchiveError::Limit)?; + let object = self + .put("tails", output.bytes, self.limits.max_index_bytes) + .await?; + drop(io); + Ok(CellGroupArchiveUploadedTail { + reference: CellGroupArchiveTailReference { + registry: object.registry, + generation: object.generation, + digest: object.digest, + length: object.length, + }, + source, + }) + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } + + /// Copy an exact contiguous committed suffix under the engine cache lock, + /// then upload immutable entry objects without holding that lock. This + /// returns an untrusted locator; all base bytes and replay must still verify. + pub async fn upload_tail( + &self, + domain: &impl CellGroupArchiveExecution, + pool: &CellGroupArchiveTailPool, + checkpoint: &CellGroupArchiveReference, + group: &GroupLog, + current: &PreparedGroupManifest, + ) -> Result { + self.upload_from(domain, pool, Origin::Checkpoint(checkpoint), group, current) + .await + } + + /// Extend an immutable archived prefix while copying only the local suffix + /// after its last cut. This works after local prefix compaction; references + /// to prior entry objects are reused. Full closure verification is still + /// required, and the caller's publication owner must protect the base/prefix. + pub async fn extend_tail( + &self, + domain: &impl CellGroupArchiveExecution, + pool: &CellGroupArchiveTailPool, + previous: &CellGroupArchiveTailReference, + group: &GroupLog, + current: &PreparedGroupManifest, + ) -> Result { + self.upload_from(domain, pool, Origin::Tail(previous), group, current) + .await + } + + async fn upload_from( + &self, + domain: &impl CellGroupArchiveExecution, + pool: &CellGroupArchiveTailPool, + origin: Origin<'_>, + group: &GroupLog, + current: &PreparedGroupManifest, + ) -> Result { + domain + .run(async { + let io = self.admit()?; + self.registry().pending(&self.ticket()).await?; + let retention = pool.admit()?; + let limits = pool.limits; + let (checkpoint, base, worker_base, prefix, prefix_bytes) = match origin { + Origin::Checkpoint(checkpoint) => { + let marker = self + .checkpoint_marker(domain, checkpoint, io.clone()) + .await?; + (checkpoint.clone(), marker.clone(), marker, Vec::new(), 0) + } + Origin::Tail(previous) => { + let tail = self + .read_tail_index( + domain, + previous, + limits, + io.clone(), + retention.clone(), + ) + .await?; + ( + tail.checkpoint, + tail.base, + tail.cut.marker, + tail.entries, + tail.bytes, + ) + } + }; + let group = group.clone(); + let root = current.clone(); + let retained = retention.clone(); + let retained_io = io.clone(); + let source_budget = self.source_budget(); + let captured = domain + .blocking(move || { + let (_retained, _io) = (retained, retained_io); + capture(group, &worker_base, root, limits, source_budget) + }) + .map_err(|_| CellGroupArchiveError::Stopped)? + .await + .map_err(CellGroupArchiveError::Worker)??; + let mut tail = Tail { + domain: TAIL_DOMAIN.into(), + checkpoint, + base, + cut: captured.cut, + bytes: prefix_bytes, + entries: prefix, + }; + let total_entries = tail + .entries + .len() + .checked_add(captured.entries.len()) + .ok_or(CellGroupArchiveError::Limit)?; + let total_bytes = captured + .entries + .iter() + .try_fold(tail.bytes, |sum, (_, _, bytes)| { + sum.checked_add(bytes.len() as u64) + }) + .ok_or(CellGroupArchiveError::Limit)?; + if total_entries > limits.max_entries || total_bytes > limits.max_tail_bytes as u64 + { + return Err(CellGroupArchiveError::Limit); + } + for (index, term, entry) in captured.entries { + let object = self.put("entries", entry, limits.max_tail_bytes).await?; + tail.bytes = tail + .bytes + .checked_add(object.length) + .ok_or(CellGroupArchiveError::Limit)?; + tail.entries.push(EntryObject { + index, + term, + object, + }); + } + tail.validate(self, limits)?; + let archive = self.clone(); + let retained = retention.clone(); + let retained_io = io.clone(); + let encoded = domain + .blocking(move || { + let (_retained, _io) = (retained, retained_io); + let mut output = checkpoint::Bounded { + bytes: Vec::new(), + limit: archive.limits.max_index_bytes, + }; + serde_json::to_writer(&mut output, &tail) + .map_err(|_| CellGroupArchiveError::Limit)?; + Ok::<_, CellGroupArchiveError>(output.bytes) + }) + .map_err(|_| CellGroupArchiveError::Stopped)? + .await + .map_err(CellGroupArchiveError::Worker)??; + let object = self + .put("tails", encoded, self.limits.max_index_bytes) + .await?; + Ok(CellGroupArchiveUploadedTail { + reference: CellGroupArchiveTailReference { + registry: object.registry, + generation: object.generation, + digest: object.digest, + length: object.length, + }, + source: captured.authority, + }) + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } +} + +impl CellGroupArchive { + pub(super) async fn read_tail_index( + &self, + domain: &impl CellGroupArchiveExecution, + reference: &CellGroupArchiveTailReference, + limits: CellGroupArchiveTailLimits, + io: Arc, + retention: Arc, + ) -> Result { + let bytes = self + .get( + "tails", + &Object { + registry: reference.registry, + generation: reference.generation, + digest: reference.digest, + length: reference.length, + }, + self.limits.max_index_bytes, + ) + .await?; + let archive = self.clone(); + let retained = retention.clone(); + let retained_io = io.clone(); + domain + .blocking(move || { + let (_retained, _io) = (retained, retained_io); + let tail: Tail = serde_json::from_slice(&bytes) + .map_err(|_| CellGroupArchiveError::Invalid("tail index"))?; + tail.validate(&archive, limits)?; + Ok::<_, CellGroupArchiveError>(tail) + }) + .map_err(|_| CellGroupArchiveError::Stopped)? + .await + .map_err(CellGroupArchiveError::Worker)? + } + + /// Independently recover the base and replay every cell/control entry. + /// Retains replay admission with the returned physical closure. No archive + /// head/GC protection, serving permission or S3 settlement is supplied here. + pub async fn verify_tail( + &self, + domain: &impl CellGroupArchiveExecution, + pool: &CellGroupArchiveTailPool, + reference: &CellGroupArchiveTailReference, + receiver: CellGroupArchiveCheckpointReceiver, + ) -> Result { + domain + .run(async { + let io = self.admit()?; + let retention = pool.admit()?; + let limits = pool.limits; + let tail = self + .read_tail_index(domain, reference, limits, io.clone(), retention.clone()) + .await?; + let mut encoded = Vec::with_capacity(tail.entries.len()); + for entry in &tail.entries { + encoded.push( + self.get("entries", &entry.object, limits.max_tail_bytes) + .await?, + ); + } + let retained = retention.clone(); + let retained_io = io.clone(); + let descriptors = tail.entries; + let entries = domain + .blocking(move || { + let (_retained, _io) = (retained, retained_io); + encoded + .into_iter() + .zip(descriptors) + .map(|(bytes, expected)| { + let entry = tail_codec::decode(&bytes, limits.max_tail_bytes)?; + if entry.index != expected.index || entry.term != expected.term { + return Err(CellGroupArchiveError::Invalid( + "tail entry descriptor", + )); + } + Ok(entry) + }) + .collect::>>() + }) + .map_err(|_| CellGroupArchiveError::Stopped)? + .await + .map_err(CellGroupArchiveError::Worker)??; + let base = self + .verify_admitted(domain, &tail.checkpoint, receiver, io.clone()) + .await?; + if base.checkpoint().marker() != &tail.base { + return Err(CellGroupArchiveError::Invalid("tail checkpoint binding")); + } + let stop = Stop(Arc::new(AtomicBool::new(false))); + let cancelled = stop.0.clone(); + let reference = reference.clone(); + let retained_io = io.clone(); + domain + .blocking(move || { + let _io = retained_io; + replay::replay( + reference, base, tail.cut, entries, limits, retention, cancelled, + ) + }) + .map_err(|_| CellGroupArchiveError::Stopped)? + .await + .map_err(CellGroupArchiveError::Worker)? + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } +} diff --git a/crates/cellar/durability/raft/cell_groups/archive/tail_admission.rs b/crates/cellar/durability/raft/cell_groups/archive/tail_admission.rs new file mode 100644 index 000000000..92c8628e0 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/archive/tail_admission.rs @@ -0,0 +1,89 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::super::ManifestPins; +use super::*; + +/// Additional node-wide admission for committed-tail capture and independent +/// LTX replay. Share this pool across all group bindings. The ordinary archive +/// pool still charges I/O and indexes; checkpoint pools still charge their cuts. +#[derive(Clone, Copy, Debug)] +pub struct CellGroupArchiveTailLimits { + pub max_operations: usize, + pub max_retained_bytes: usize, + pub max_entries: usize, + pub max_tail_bytes: usize, + pub max_cells: u32, + /// Must cover the destination store's captured image workspace bound. + pub max_apply_bytes: usize, +} + +impl Default for CellGroupArchiveTailLimits { + fn default() -> Self { + Self { + max_operations: 1, + max_retained_bytes: 512 * 1024 * 1024, + max_entries: 1024, + max_tail_bytes: 16 * 1024 * 1024, + max_cells: 64, + max_apply_bytes: 128 * 1024 * 1024, + } + } +} + +#[derive(Clone)] +pub struct CellGroupArchiveTailPool { + pub(super) limits: CellGroupArchiveTailLimits, + budget: Arc, + workspace: usize, +} + +impl CellGroupArchiveTailPool { + pub fn new(limits: CellGroupArchiveTailLimits) -> Result { + if !(1..=4096).contains(&limits.max_operations) + || !(1..=65_536).contains(&limits.max_entries) + || !(1024..=512 * 1024 * 1024).contains(&limits.max_tail_bytes) + || !(1..=1_000_000).contains(&limits.max_cells) + || !(4 * 1024 * 1024..=512 * 1024 * 1024).contains(&limits.max_apply_bytes) + { + return Err(CellGroupArchiveError::Limit); + } + // Base, predecessor and successor roots can coexist while one image + // transition executes. Each cell may retain a <=1 MiB extent manifest. + let metadata = ManifestPins::maximum_bytes(limits.max_cells) + .checked_add( + (limits.max_cells as usize) + .checked_mul(1024 * 1024) + .ok_or(CellGroupArchiveError::Limit)?, + ) + .and_then(|n| n.checked_mul(3)); + let workspace = metadata + .and_then(|n| n.checked_add(limits.max_tail_bytes.checked_mul(4)?)) + .and_then(|n| n.checked_add(limits.max_apply_bytes)) + .and_then(|n| n.checked_add(limits.max_entries.checked_mul(512)?)) + .and_then(|n| n.checked_add(1024 * 1024)) + .ok_or(CellGroupArchiveError::Limit)?; + if workspace > limits.max_retained_bytes { + return Err(CellGroupArchiveError::Limit); + } + Ok(Self { + limits, + budget: Budget::new(limits.max_retained_bytes, limits.max_operations), + workspace, + }) + } + + pub fn usage(&self) -> CellGroupArchiveUsage { + let (retained_bytes, operations) = self.budget.usage(); + CellGroupArchiveUsage { + retained_bytes, + operations, + } + } + + pub(super) fn admit(&self) -> Result> { + self.budget + .acquire(self.workspace) + .map(Arc::new) + .ok_or(CellGroupArchiveError::Busy) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/archive/tail_codec.rs b/crates/cellar/durability/raft/cell_groups/archive/tail_codec.rs new file mode 100644 index 000000000..86234c242 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/archive/tail_codec.rs @@ -0,0 +1,86 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Fixed canonical entry framing bounds allocation before decoding. Includes +//! context and sync_log; no unknown/repeated protobuf fields are accepted. +use super::*; +use protobuf::Message as _; +use raft::prelude::{Entry, EntryType}; + +const ENTRY_DOMAIN: &[u8] = b"cellar-cell-group-archive-entry-1\0"; +const FIXED: usize = ENTRY_DOMAIN.len() + 26; + +pub(super) fn length(entry: &Entry) -> Result { + if entry.index == 0 + || entry.term == 0 + || entry.get_unknown_fields().iter().next().is_some() + || !matches!( + entry.get_entry_type(), + EntryType::EntryNormal | EntryType::EntryConfChangeV2 + ) + { + return Err(CellGroupArchiveError::Invalid("archive entry metadata")); + } + entry + .data + .len() + .checked_add(entry.context.len()) + .and_then(|n| n.checked_add(FIXED)) + .ok_or(CellGroupArchiveError::Limit) +} + +pub(super) fn encode(entry: &Entry, limit: usize) -> Result> { + let length = length(entry)?; + if length > limit + || entry.data.len() > u32::MAX as usize + || entry.context.len() > u32::MAX as usize + { + return Err(CellGroupArchiveError::Limit); + } + let mut bytes = Vec::with_capacity(length); + bytes.extend_from_slice(ENTRY_DOMAIN); + bytes.extend_from_slice(&entry.index.to_be_bytes()); + bytes.extend_from_slice(&entry.term.to_be_bytes()); + bytes.push(entry.get_entry_type() as u8); + bytes.push(u8::from(entry.sync_log)); + bytes.extend_from_slice(&(entry.context.len() as u32).to_be_bytes()); + bytes.extend_from_slice(&(entry.data.len() as u32).to_be_bytes()); + bytes.extend_from_slice(&entry.context); + bytes.extend_from_slice(&entry.data); + Ok(bytes) +} + +pub(super) fn decode(bytes: &[u8], limit: usize) -> Result { + if bytes.len() < FIXED || bytes.len() > limit || !bytes.starts_with(ENTRY_DOMAIN) { + return Err(CellGroupArchiveError::Invalid("archive entry framing")); + } + let at = ENTRY_DOMAIN.len(); + // Every fixed slice below lies in the already checked header. + let index = u64::from_be_bytes(bytes[at..at + 8].try_into().unwrap()); + let term = u64::from_be_bytes(bytes[at + 8..at + 16].try_into().unwrap()); + let entry_type = match bytes[at + 16] { + 0 => EntryType::EntryNormal, + 2 => EntryType::EntryConfChangeV2, + _ => return Err(CellGroupArchiveError::Invalid("archive entry type")), + }; + let sync_log = match bytes[at + 17] { + 0 => false, + 1 => true, + _ => return Err(CellGroupArchiveError::Invalid("archive entry sync bit")), + }; + let context = u32::from_be_bytes(bytes[at + 18..at + 22].try_into().unwrap()) as usize; + let data = u32::from_be_bytes(bytes[at + 22..at + 26].try_into().unwrap()) as usize; + if context.checked_add(data).and_then(|n| n.checked_add(FIXED)) != Some(bytes.len()) { + return Err(CellGroupArchiveError::Invalid("archive entry lengths")); + } + let entry = Entry { + index, + term, + entry_type, + sync_log, + context: bytes[FIXED..FIXED + context].to_vec().into(), + data: bytes[FIXED + context..].to_vec().into(), + ..Default::default() + }; + length(&entry)?; + Ok(entry) +} diff --git a/crates/cellar/durability/raft/cell_groups/archive/upload.rs b/crates/cellar/durability/raft/cell_groups/archive/upload.rs new file mode 100644 index 000000000..04cc54b56 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/archive/upload.rs @@ -0,0 +1,242 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Node-admitted upload owners. Persistent reservations, rather than local +//! handles or clocks, protect upload generations and inherited dependencies. +use super::{ + graph::{Graph, Kind}, + ownership::*, + *, +}; +use std::{collections::BTreeMap, ops::Deref}; + +#[derive(Clone, Copy, Debug)] +pub struct CellGroupArchiveUploadLimits { + /// Counts upload maps, captured source witnesses and collector marks. + /// At least two owners are needed for a map and its publication witness. + pub max_owners: usize, + pub max_retained_bytes: usize, + pub max_objects: usize, +} + +impl Default for CellGroupArchiveUploadLimits { + fn default() -> Self { + Self { + max_owners: 8, + max_retained_bytes: 64 * 1024 * 1024, + max_objects: 16_384, + } + } +} + +/// Share this admission across every group on the node. Upload clones retain +/// their map and permit; dropping a clone never aborts its persistent reservation. +#[derive(Clone)] +pub struct CellGroupArchiveUploadPool { + pub(super) limits: CellGroupArchiveUploadLimits, + budget: Arc, + workspace: usize, +} + +impl CellGroupArchiveUploadPool { + pub fn new(limits: CellGroupArchiveUploadLimits) -> Result { + if !(2..=64).contains(&limits.max_owners) || !(1..=65_536).contains(&limits.max_objects) { + return Err(CellGroupArchiveError::Limit); + } + // Compact object marks plus the content lookup index, allocation/tree + // overhead and bounded control metadata. Archive I/O has its own pool. + let workspace = limits + .max_objects + .checked_mul(512) + .and_then(|n| n.checked_add(128 * 1024)) + .ok_or(CellGroupArchiveError::Limit)?; + if workspace > limits.max_retained_bytes { + return Err(CellGroupArchiveError::Limit); + } + Ok(Self { + limits, + budget: Budget::new(limits.max_retained_bytes, limits.max_owners), + workspace, + }) + } + + pub fn usage(&self) -> CellGroupArchiveUsage { + let (retained_bytes, operations) = self.budget.usage(); + CellGroupArchiveUsage { + operations, + retained_bytes, + } + } + + pub(super) fn admit(&self) -> Result> { + self.budget + .acquire(self.workspace) + .map(Arc::new) + .ok_or(CellGroupArchiveError::Busy) + } +} + +struct Owner { + registry: CellGroupArchiveRegistry, + ticket: CellGroupArchiveUploadTicket, + inherited: Graph, + reuse: BTreeMap<(Kind, [u8; 32], u64), super::objects::Object>, + _reservation: Arc, + source_budget: Arc, +} + +/// The only producer of immutable group archive objects. All new keys include a +/// never-reused reservation generation. Reuse is restricted to its protected +/// parent closure; an arbitrary old content address cannot be resurrected. +#[derive(Clone)] +pub struct CellGroupArchiveUpload(Arc); + +impl Deref for CellGroupArchiveUpload { + type Target = CellGroupArchive; + + fn deref(&self) -> &Self::Target { + self.0.registry.archive() + } +} + +impl CellGroupArchiveRegistry { + /// Resume only the exact still-pending persisted ticket. A replacement + /// process must fence its predecessor before sharing/retiring that owner. + pub async fn open_upload( + &self, + domain: &impl CellGroupArchiveExecution, + pool: &CellGroupArchiveUploadPool, + ticket: &CellGroupArchiveUploadTicket, + tails: &CellGroupArchiveTailPool, + ) -> Result { + domain + .run(async { + let reservation = pool.admit()?; + let io = self.archive.admit()?; + let pending = self.pending(ticket).await?; + let inherited = match pending.parent { + Some(root) => { + Graph::load( + &self.archive, + domain, + &root, + tails.limits, + io.clone(), + reservation.clone(), + pool.limits.max_objects, + ) + .await? + } + None => Graph::new(pool.limits.max_objects), + }; + // Recheck after metadata I/O; an abort cannot produce a new owner. + self.pending(ticket).await?; + let reuse = inherited + .objects + .iter() + .map(|key| { + ( + (key.kind, key.object.digest, key.object.length), + key.object.clone(), + ) + }) + .collect(); + Ok(CellGroupArchiveUpload(Arc::new(Owner { + registry: self.clone(), + ticket: *ticket, + inherited, + reuse, + _reservation: reservation, + source_budget: pool.budget.clone(), + }))) + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } +} + +impl CellGroupArchiveUpload { + pub(crate) async fn require_pending( + &self, + domain: &impl CellGroupArchiveExecution, + ) -> Result<()> { + domain + .run(async { self.registry().pending(&self.ticket()).await.map(|_| ()) }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } + + pub fn registry(&self) -> &CellGroupArchiveRegistry { + &self.0.registry + } + + pub fn ticket(&self) -> CellGroupArchiveUploadTicket { + self.0.ticket + } + + pub(super) fn retained(&self) -> Arc { + self.0._reservation.clone() + } + + pub(super) fn max_objects(&self) -> usize { + self.0.inherited.limit() + } + + pub(super) fn source_budget(&self) -> Arc { + self.0.source_budget.clone() + } + + pub(super) fn allows(&self, graph: &Graph) -> Result<()> { + if graph.objects.iter().any(|key| { + key.object.generation != self.0.ticket.generation() + && !self.0.inherited.objects.contains(key) + }) { + return Err(CellGroupArchiveError::Invalid( + "unprotected inherited archive object", + )); + } + Ok(()) + } + + pub(super) fn allows_object(&self, kind: Kind, object: &super::objects::Object) -> Result<()> { + if object.registry != self.registry().id() + || object.generation > self.ticket().generation() + || (object.generation != self.ticket().generation() + && !self.0.inherited.objects.contains(&super::graph::Key { + kind, + object: object.clone(), + })) + { + return Err(CellGroupArchiveError::Invalid( + "unprotected seed checkpoint object", + )); + } + Ok(()) + } + + pub(super) async fn put( + &self, + kind: &str, + bytes: Vec, + limit: usize, + ) -> Result { + if bytes.is_empty() || bytes.len() > limit { + return Err(CellGroupArchiveError::Limit); + } + let kind = + Kind::parse(kind).ok_or(CellGroupArchiveError::Invalid("archive object kind"))?; + let own = + super::objects::Object::of(self.registry().id(), self.ticket().generation(), &bytes); + if let Some(inherited) = self.0.reuse.get(&(kind, own.digest, own.length)) { + self.get(kind.name(), inherited, limit).await?; + return Ok(inherited.clone()); + } + self.put_generation( + self.registry().id(), + self.ticket().generation(), + kind.name(), + bytes, + limit, + ) + .await + } +} diff --git a/crates/cellar/durability/raft/cell_groups/archive_compaction.rs b/crates/cellar/durability/raft/cell_groups/archive_compaction.rs new file mode 100644 index 000000000..3186189f3 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/archive_compaction.rs @@ -0,0 +1,247 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Local checkpoint compaction follows independently verified archive coverage. +use super::{CellGroupArchiveCoverage, GroupManifest}; +use crate::raft::{cache::GroupLog, storage::LogError}; +use cellar_logic::cell_group::GroupHistory; +use std::sync::{ + atomic::{AtomicBool, Ordering}, + Arc, Mutex, +}; + +struct State { + required: bool, + covered: Option, + admitted: usize, + activations: usize, +} + +impl State { + fn permits(&self, marker: &GroupManifest) -> bool { + self.activations == 0 + && (!self.required + || self.covered.as_ref().is_some_and(|covered| { + covered.route() == marker.route() + && covered.position().index >= marker.position().index + && (covered.position().index != marker.position().index + || covered == marker) + })) + } +} + +/// Admission already entered under the preceding policy. Archive activation +/// retries until both the owner and entered storage work have released the same +/// cut-bound lease. Storage can finish this work after new admission is closed. +pub(crate) struct ArchiveCompactionAdmission { + gate: Arc, + cut: GroupManifest, +} + +// The existing 4 KiB checkpoint metadata charge covers this lease, including +// both bounded routing-key strings and the Arc allocation's reference counts. +const _: () = assert!( + std::mem::size_of::() + + 2 * cellar_logic::cell_group::MAX_CELL_ID_BYTES + + 2 * std::mem::size_of::() + <= 4096 +); + +impl ArchiveCompactionAdmission { + pub(crate) fn validate( + &self, + gate: &Arc, + cut: &GroupManifest, + ) -> Result<(), LogError> { + if !Arc::ptr_eq(&self.gate, gate) || &self.cut != cut { + return Err(LogError::WrongIdentity); + } + let state = self.gate.state.lock().map_err(|_| LogError::Closed)?; + if state.admitted == 0 { + return Err(LogError::WrongIdentity); + } + Ok(()) + } +} + +impl Drop for ArchiveCompactionAdmission { + fn drop(&mut self) { + if let Ok(mut state) = self.gate.state.lock() { + state.admitted -= 1; + } + } +} + +/// Owned by the controller's registration, and by an entered metadata write. +/// Dropping an unentered intent restores the previous policy. Successful sync +/// makes the requirement irreversible before releasing the admission barrier. +pub(crate) struct ArchiveActivation { + gate: Arc, + pending: AtomicBool, + prepared: AtomicBool, +} + +impl ArchiveActivation { + pub(crate) fn drained(&self, gate: &Arc) -> Result<(), LogError> { + if !Arc::ptr_eq(&self.gate, gate) { + return Err(LogError::WrongIdentity); + } + let mut state = self.gate.state.lock().map_err(|_| LogError::Closed)?; + if !self.pending.load(Ordering::Relaxed) { + return if state.required { + Ok(()) + } else { + Err(LogError::WrongIdentity) + }; + } + if state.activations == 0 { + return Err(LogError::WrongIdentity); + } + if state.admitted != 0 { + return Err(LogError::Overloaded); + } + if !self.prepared.swap(true, Ordering::Relaxed) { + // Reset before I/O, once per intent. A delayed duplicate metadata + // completion must not discard coverage verified in the meantime. + state.covered = None; + } + Ok(()) + } + + /// Call only after selecting the durable local requirement, or rechecking + /// an existing durable bit. No failed/uncertain write may complete intent. + pub(crate) fn published(&self, gate: &Arc) -> Result<(), LogError> { + if !Arc::ptr_eq(&self.gate, gate) { + return Err(LogError::WrongIdentity); + } + let mut state = self.gate.state.lock().map_err(|_| LogError::Closed)?; + if !self.pending.load(Ordering::Relaxed) { + return if state.required { + Ok(()) + } else { + Err(LogError::WrongIdentity) + }; + } + if state.activations == 0 || state.admitted != 0 || !self.prepared.load(Ordering::Relaxed) { + return Err(LogError::WrongIdentity); + } + state.required = true; + self.pending.store(false, Ordering::Relaxed); + state.activations -= 1; + Ok(()) + } +} + +impl Drop for ArchiveActivation { + fn drop(&mut self) { + if let Ok(mut state) = self.gate.state.lock() { + if self.pending.swap(false, Ordering::Relaxed) { + state.activations -= 1; + } + } + } +} + +/// Shared by every clone of one engine-local group. The requirement survives +/// restart in the group record; the frontier never survives without rechecking +/// the actual remote hold. No replay workspace or per-cell state is retained. +pub(crate) struct ArchiveCompaction { + pub(crate) log_retention: Option>, + state: Mutex, +} + +impl ArchiveCompaction { + pub(crate) fn new( + required: bool, + log_retention: Option>, + ) -> Self { + Self { + log_retention, + state: Mutex::new(State { + required, + covered: None, + admitted: 0, + activations: 0, + }), + } + } + + pub(crate) fn begin_activation(self: &Arc) -> Result, LogError> { + let mut state = self.state.lock().map_err(|_| LogError::Closed)?; + state.activations = state + .activations + .checked_add(1) + .ok_or(LogError::Overloaded)?; + Ok(Arc::new(ArchiveActivation { + gate: self.clone(), + pending: AtomicBool::new(true), + prepared: AtomicBool::new(false), + })) + } + + /// Called only while the controller owns its journaled current remote hold. + /// This historical frontier grants neither an S3 response nor serving rights. + pub(in crate::raft::cell_groups) fn confirm( + &self, + group: &GroupLog, + coverage: &CellGroupArchiveCoverage, + ) -> Result<(), LogError> { + let cache = group.read()?; + let current = cache + .group_manifest + .as_ref() + .ok_or(LogError::WrongIdentity)?; + let marker = coverage.marker(); + let position = marker.position(); + if !cache.archive_required + || marker.route() != current.route() + || position.index > current.position().index + || (position.index == current.position().index && marker != current) + { + return Err(LogError::WrongIdentity); + } + // An old pending intent can reconcile behind a checkpoint selected + // before archival was enabled. Keep compaction closed until a newer + // verified cut reaches the retained local history; do not quarantine a + // valid historical publication or invent its missing local term. + if position.index.get() < cache.snapshot.get_metadata().index { + return Ok(()); + } + if cache.term(position.index.get()) != Some(position.term.get()) { + return Err(LogError::WrongIdentity); + } + let mut state = self.state.lock().map_err(|_| LogError::Closed)?; + if !state.required { + return Err(LogError::WrongIdentity); + } + if let Some(previous) = &state.covered { + if previous.position().index > position.index { + return Ok(()); + } + if previous.position().index == position.index && previous != marker { + return Err(LogError::WrongIdentity); + } + } + state.covered = Some(marker.clone()); + Ok(()) + } + + pub(crate) fn permits(&self, marker: &GroupManifest) -> Result { + let state = self.state.lock().map_err(|_| LogError::Closed)?; + Ok(state.permits(marker)) + } + + pub(crate) fn admit( + self: &Arc, + marker: &GroupManifest, + ) -> Result>, LogError> { + let mut state = self.state.lock().map_err(|_| LogError::Closed)?; + if !state.permits(marker) { + return Ok(None); + } + state.admitted = state.admitted.checked_add(1).ok_or(LogError::Overloaded)?; + Ok(Some(Arc::new(ArchiveCompactionAdmission { + gate: self.clone(), + cut: marker.clone(), + }))) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/checkpoint/header.rs b/crates/cellar/durability/raft/cell_groups/checkpoint/header.rs new file mode 100644 index 000000000..8f4998c83 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/checkpoint/header.rs @@ -0,0 +1,100 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use raft::prelude::{ConfState, Snapshot}; +use serde::{Deserialize, Serialize}; + +const DOMAIN: &str = "cellar-cell-group-checkpoint-1"; + +#[cfg(test)] +#[path = "header_tests.rs"] +mod tests; + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub(super) enum Transition { + Active, +} + +#[derive(Clone, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Header { + pub domain: String, + pub root: GroupManifest, + pub conf: Vec, + pub membership: GroupMembership, + pub transition: Transition, +} + +impl Header { + pub(super) fn capture( + root: GroupManifest, + conf: &ConfState, + membership: GroupMembership, + ) -> Result { + root.check()?; + membership.validate(conf, root.position().index.get())?; + membership.validate_scope(&root.route().history)?; + if root.membership_revision() != membership.revision().get() { + return Err(CellGroupCheckpointError::Invalid( + "checkpoint capture membership", + )); + } + Ok(Self { + domain: DOMAIN.into(), + root, + conf: crate::raft::manifest::encode_conf(conf) + .map_err(|_| CellGroupCheckpointError::Invalid("configuration encoding"))?, + membership, + transition: Transition::Active, + }) + } + + pub(super) fn decode(bytes: &[u8]) -> Result { + if bytes.is_empty() || bytes.len() > HEADER_LIMIT { + return Err(CellGroupCheckpointError::Limit); + } + let header: Self = serde_json::from_slice(bytes) + .map_err(|_| CellGroupCheckpointError::Invalid("checkpoint header"))?; + if header.domain != DOMAIN + || header.root.membership_revision() != header.membership.revision().get() + { + return Err(CellGroupCheckpointError::Invalid( + "checkpoint domain/membership", + )); + } + header.root.check()?; + header + .membership + .validate(&header.configuration()?, header.root.position().index.get())?; + header + .membership + .validate_scope(&header.root.route().history)?; + Ok(header) + } + + pub(super) fn configuration(&self) -> Result { + crate::raft::manifest::decode_conf(&self.conf) + .map_err(|_| CellGroupCheckpointError::Invalid("checkpoint configuration")) + } + + pub(super) fn encode(&self) -> Result> { + let bytes = serde_json::to_vec(self) + .map_err(|_| CellGroupCheckpointError::Invalid("header encoding"))?; + if bytes.len() > HEADER_LIMIT { + return Err(CellGroupCheckpointError::Limit); + } + Ok(bytes) + } + + pub(super) fn snapshot(&self) -> Result { + let mut snapshot = Snapshot::default(); + snapshot.mut_metadata().index = self.root.position().index.get(); + snapshot.mut_metadata().term = self.root.position().term.get(); + snapshot + .mut_metadata() + .set_conf_state(self.configuration()?); + snapshot.data = self.encode()?.into(); + Ok(snapshot) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/checkpoint/header_tests.rs b/crates/cellar/durability/raft/cell_groups/checkpoint/header_tests.rs new file mode 100644 index 000000000..80f628bf0 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/checkpoint/header_tests.rs @@ -0,0 +1,44 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use crate::raft::member_state::FleetGroupBinding; +use cellar_logic::{ + cell_group::{CellRange, GroupHistory, GroupRoute, RouteEpoch}, + consensus::{ClusterId, GroupId, IncarnationId, NamespaceId}, +}; + +#[test] +fn checkpoint_refuses_a_foreign_managed_group_guard() { + let route = GroupRoute { + history: GroupHistory { + cluster: ClusterId::new([1; 16]).unwrap(), + namespace: NamespaceId::new([2; 16]).unwrap(), + group: GroupId::new([3; 16]).unwrap(), + incarnation: IncarnationId::new([4; 16]).unwrap(), + }, + range: CellRange::all(), + epoch: RouteEpoch::new(1).unwrap(), + birth_digest: [5; 32], + schema_digest: [6; 32], + }; + let root = GroupManifest::genesis(route.clone()).unwrap(); + let conf = ConfState { + voters: vec![1, 2, 3], + ..Default::default() + }; + let binding = |history: &GroupHistory| { + FleetGroupBinding::new(IncarnationId::new([7; 16]).unwrap(), history, [8; 32]).unwrap() + }; + let mut foreign = route.history; + foreign.group = GroupId::new([9; 16]).unwrap(); + let foreign_membership = GroupMembership::managed_initial(&conf, binding(&foreign)).unwrap(); + assert!(Header::capture(root.clone(), &conf, foreign_membership.clone()).is_err()); + let membership = GroupMembership::managed_initial(&conf, binding(&route.history)).unwrap(); + let header = Header::capture(root, &conf, membership).unwrap(); + Header::decode(&header.encode().unwrap()).unwrap(); + let malformed = Header { + membership: foreign_membership, + ..header + }; + assert!(Header::decode(&malformed.encode().unwrap()).is_err()); +} diff --git a/crates/cellar/durability/raft/cell_groups/checkpoint/local.rs b/crates/cellar/durability/raft/cell_groups/checkpoint/local.rs new file mode 100644 index 000000000..600efe51a --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/checkpoint/local.rs @@ -0,0 +1,51 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use raft::prelude::Snapshot; + +struct Inner { + source: CellGroupCheckpoint, + snapshot: Snapshot, +} + +/// A captured local cut. It retains source admission and physical pins through +/// publication; only the matching engine completion establishes a checkpoint. +#[derive(Clone)] +pub struct PreparedLocalCellGroupCheckpoint(Arc); + +impl std::fmt::Debug for PreparedLocalCellGroupCheckpoint { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("PreparedLocalCellGroupCheckpoint") + .field("marker", self.marker()) + .finish() + } +} + +pub(super) fn prepare(source: CellGroupCheckpoint) -> Result { + if source.started { + return Err(CellGroupCheckpointError::Invalid("source already streamed")); + } + let snapshot = source.snapshot()?; + Ok(PreparedLocalCellGroupCheckpoint(Arc::new(Inner { + source, + snapshot, + }))) +} + +impl PreparedLocalCellGroupCheckpoint { + pub fn marker(&self) -> &GroupManifest { + self.0.source.marker() + } + + pub fn snapshot(&self) -> &Snapshot { + &self.0.snapshot + } + + pub(crate) fn root(&self) -> &PreparedGroupManifest { + &self.0.source.root + } + + pub(crate) fn membership(&self) -> Result { + Ok(Header::decode(&self.0.source.header)?.membership) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs b/crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs new file mode 100644 index 000000000..b254d7c83 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs @@ -0,0 +1,576 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! A complete immutable source cut, independent of snapshot installation and +//! voter admission. Run capture and streaming on an admitted blocking owner. + +use super::{page_wire, CellImagePin, GroupManifest, ManifestLimits, PreparedGroupManifest}; +use crate::raft::{ + budget::{Budget, Permit}, + cache::GroupLog, + member_state::GroupMembership, +}; +use cellar_logic::cell_group::{CellId, GroupHistory}; +use cellar_ltx::internal::{ExtentId, EXTENT_BYTES}; +mod header; +mod local; +mod prepared; +mod receive; +mod receive_archive; +mod seed; +pub(crate) use seed::{seed_initial_membership, seed_profile, seed_source}; +pub(crate) mod wire; +use header::Header; +pub use local::PreparedLocalCellGroupCheckpoint; +pub(crate) use prepared::persisted_marker; +pub use prepared::PreparedCellGroupCheckpoint; +pub use receive::CellGroupCheckpointReceiver; +pub(crate) use receive_archive::PreparedArchiveReceive; +pub use receive_archive::{CellGroupArchiveCheckpointReceiver, PreparedCellGroupArchiveCheckpoint}; +use sha2::{Digest, Sha256}; +use std::sync::Arc; + +/// Validate authenticated seed metadata before any destination mapping exists. +pub(crate) fn seed_marker( + snapshot: &raft::prelude::Snapshot, + target: &cellar_logic::consensus::ReplicaIdentity, + barrier: cellar_logic::consensus::LogIndex, +) -> Result { + let header = Header::decode(&snapshot.data)?; + let conf = header.configuration()?; + let id = target.member.get(); + let joined = header.membership.joined(target.member)?; + if header.snapshot()? != *snapshot + || header.root.route().history != target.scope + || barrier.get() == 0 + || header.root.position().index < barrier + || header.membership.revision() > barrier + || !conf.learners.contains(&id) + || conf.voters.contains(&id) + || conf.voters_outgoing.contains(&id) + || conf.learners_next.contains(&id) + || !header.membership.permits_group(target) + || joined.is_none_or(|(binding, at)| { + at > barrier || binding.node != target.node || binding.disk != target.disk + }) + { + return Err(CellGroupCheckpointError::Invalid("seed descriptor/member")); + } + Ok(header.root) +} + +pub(crate) fn seed_birth( + snapshot: &raft::prelude::Snapshot, + route: &cellar_logic::cell_group::GroupRoute, + initial: &raft::prelude::ConfState, +) -> Result<()> { + let header = Header::decode(&snapshot.data)?; + if header.root.route() != route + || !header + .membership + .same_birth(&GroupMembership::initial(initial)) + { + return Err(CellGroupCheckpointError::Invalid("seed birth")); + } + Ok(()) +} + +const HEADER_LIMIT: usize = 64 * 1024; + +// Covers one verified extent, page/extent metadata encoding, serialization +// temporaries, one flat-image read, membership, traversal stack and fixed owners. +const STREAM_WORKSPACE: usize = EXTENT_BYTES as usize + 1024 * 1024; + +#[derive(Debug)] +pub enum CellGroupCheckpointError { + Authority(crate::raft::completion::RuntimeError), + Invalid(&'static str), + Limit, + Busy, + /// The requested current cut raced durable apply. Retry after progress; + /// unlike malformed source metadata, this does not establish corruption. + NotReady, + Manifest(super::ManifestError), + Log(crate::raft::storage::LogError), + Image(cellar_ltx::generation::GenerationError), + Extent(cellar_ltx::internal::StoreError), + Sink(std::io::Error), +} + +impl std::fmt::Display for CellGroupCheckpointError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "group checkpoint: {self:?}") + } +} + +impl std::error::Error for CellGroupCheckpointError { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + match self { + Self::Authority(error) => Some(error), + Self::Manifest(error) => Some(error), + Self::Log(error) => Some(error), + Self::Image(error) => Some(error), + Self::Extent(error) => Some(error), + Self::Sink(error) => Some(error), + _ => None, + } + } +} + +impl From for CellGroupCheckpointError { + fn from(error: super::ManifestError) -> Self { + Self::Manifest(error) + } +} + +impl From for CellGroupCheckpointError { + fn from(error: crate::raft::storage::LogError) -> Self { + Self::Log(error) + } +} + +impl From for CellGroupCheckpointError { + fn from(error: cellar_ltx::generation::GenerationError) -> Self { + Self::Image(error) + } +} + +impl From for CellGroupCheckpointError { + fn from(error: cellar_ltx::internal::StoreError) -> Self { + Self::Extent(error) + } +} + +type Result = std::result::Result; + +/// One pool must be shared by the node's checkpoint sources and receivers. Byte +/// accounting charges retained metadata and workspace; physical stores bound +/// disk retention. Receivers additionally share a scratch pool for image spools. +#[derive(Clone, Copy, Debug)] +pub struct CellGroupCheckpointLimits { + pub max_sources: usize, + pub max_retained_bytes: usize, + pub max_cells: u32, + pub max_transfer_bytes: u64, + pub max_chunk_bytes: usize, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct CellGroupCheckpointUsage { + pub sources: usize, + pub retained_bytes: usize, +} + +#[derive(Clone)] +pub struct CellGroupCheckpointPool { + budget: Arc, + limits: CellGroupCheckpointLimits, +} + +#[derive(Clone, Copy)] +enum SourceRoot { + Published, + Retained, + OwnerCurrent, +} + +impl CellGroupCheckpointPool { + pub fn new(limits: CellGroupCheckpointLimits) -> Result { + if !(1..=4096).contains(&limits.max_sources) + || limits.max_retained_bytes < STREAM_WORKSPACE + || !(1..=1_000_000).contains(&limits.max_cells) + || limits.max_transfer_bytes < HEADER_LIMIT as u64 + || !(1..=64 * 1024).contains(&limits.max_chunk_bytes) + { + return Err(CellGroupCheckpointError::Limit); + } + Ok(Self { + budget: Budget::new(limits.max_retained_bytes, limits.max_sources), + limits, + }) + } + + pub fn usage(&self) -> CellGroupCheckpointUsage { + let (retained_bytes, sources) = self.budget.usage(); + CellGroupCheckpointUsage { + sources, + retained_bytes, + } + } + + /// Select only the exact engine-published complete root. Membership and + /// configuration are copied under the same cache read lock as that check. + /// A prospective, stale or foreign preparation is not a checkpoint source. + /// O(cells) metadata admission runs off the consensus loop, without image I/O. + pub fn capture( + &self, + group: &GroupLog, + root: &PreparedGroupManifest, + ) -> Result { + self.capture_root(group, root, false) + } + + /// Pin exactly the retained checkpoint selected by Raft Storage, even after + /// the current applied root advances. A concurrent replacement may refuse; + /// a successfully captured source owns its immutable cut through streaming. + pub fn capture_snapshot( + &self, + group: &GroupLog, + current: &PreparedGroupManifest, + snapshot: &raft::prelude::Snapshot, + ) -> Result { + let root = current.retained_checkpoint(group)?; + let source = self.capture_root(group, &root, true)?; + if source.snapshot()? != *snapshot { + return Err(CellGroupCheckpointError::Invalid( + "retained snapshot changed", + )); + } + Ok(source) + } + + pub(super) fn capture_root( + &self, + group: &GroupLog, + root: &PreparedGroupManifest, + is_retained: bool, + ) -> Result { + self.capture_checked( + group, + root, + if is_retained { + SourceRoot::Retained + } else { + SourceRoot::Published + }, + ) + } + + pub(super) fn capture_owner( + &self, + group: &GroupLog, + root: &PreparedGroupManifest, + ) -> Result { + self.capture_checked(group, root, SourceRoot::OwnerCurrent) + } + + fn capture_checked( + &self, + group: &GroupLog, + root: &PreparedGroupManifest, + source: SourceRoot, + ) -> Result { + if !Arc::ptr_eq(&root.engine_owner, &group.owner) + || !root + .marker() + .matches_group(&group.identity().scope, group.birth_digest) + || root.pinned_images() != root.marker().root().cells() as usize + || root.pinned_images() > self.limits.max_cells as usize + { + return Err(CellGroupCheckpointError::Invalid("source root/owner")); + } + let mut retained = STREAM_WORKSPACE + .checked_add(root.reserved_bytes()) + .ok_or(CellGroupCheckpointError::Limit)?; + let mut transfer = HEADER_LIMIT as u64; + root.images.visit(&mut |_, leaf, pin, _, _| -> Result<()> { + // Account even shared extent trees and inventory identities at a + // conservative per-extent cost. Payloads remain in their stores. + let extra = match pin { + CellImagePin::Extents(pin) => { + let count = pin.manifest().geometry().extent_count() as usize; + let encoded = pin.reference().encoded_length() as usize; + transfer = transfer + .checked_add(encoded as u64) + .ok_or(CellGroupCheckpointError::Limit)?; + count + .checked_mul(1024) + .and_then(|n| n.checked_add(encoded)) + .ok_or(CellGroupCheckpointError::Limit)? + } + _ => 1024, + }; + retained = retained + .checked_add(extra) + .ok_or(CellGroupCheckpointError::Limit)?; + transfer = transfer + .checked_add(page_wire::MAX_PAGE_BYTES as u64) + .and_then(|n| n.checked_add(leaf.geometry.length)) + .ok_or(CellGroupCheckpointError::Limit)?; + Ok(()) + })?; + if transfer > self.limits.max_transfer_bytes { + return Err(CellGroupCheckpointError::Limit); + } + let reservation = self + .budget + .acquire(retained) + .ok_or(CellGroupCheckpointError::Busy)?; + let header = { + let cache = group.read()?; + let marker = root.marker(); + if matches!(source, SourceRoot::Retained) { + if cache.checkpoint_manifest.as_ref() != Some(marker) { + return Err(CellGroupCheckpointError::Invalid( + "retained checkpoint changed", + )); + } + let header = Header::decode(&cache.snapshot.data)?; + if header.snapshot()? != cache.snapshot || &header.root != marker { + return Err(CellGroupCheckpointError::Invalid( + "retained snapshot binding", + )); + } + header + } else { + if cache.group_manifest.as_ref() != Some(marker) + || cache.applied.position != marker.position() + || cache.applied.image.is_some() + || marker.position().index.get() > cache.hard_state.commit + || cache.term(marker.position().index.get()) + != Some(marker.position().term.get()) + || marker.membership_revision() != cache.membership.revision().get() + || cache + .manifest_pins + .as_ref() + .is_some_and(|pins| pins.root() != marker.root()) + { + // Only the owner's sealed prior applied root may race a + // newer engine publication. Public callers supplying stale + // or prospective preparations retain strict refusal. + if matches!(source, SourceRoot::OwnerCurrent) + && cache.applied.position.index > marker.position().index + && cache.applied.image.is_none() + && cache.hard_state.commit >= marker.position().index.get() + && cache.group_manifest.as_ref().is_some_and(|published| { + published.position() == cache.applied.position + && published.route() == marker.route() + }) + { + return Err(CellGroupCheckpointError::NotReady); + } + return Err(CellGroupCheckpointError::Invalid("source durable cut")); + } + cache + .membership + .validate(&cache.conf_state, marker.position().index.get())?; + Header::capture(marker.clone(), &cache.conf_state, cache.membership.clone())? + } + }; + let encoded = header.encode()?; + Ok(CellGroupCheckpoint { + group: group.clone(), + root: root.clone(), + header: encoded, + limits: self.limits, + started: false, + _reservation: reservation, + }) + } +} + +/// Borrowed part identity. Each manifest page carries the full cell leaf, +/// including empty enrollment, version, receipt-image commitment and effects. +/// Extents are emitted in logical order, including repeated content identities; +/// a receiver may reuse verified objects but must check every manifest binding. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum CellGroupCheckpointPart<'a> { + Header, + Page([u8; 32]), + Materialized(&'a CellId), + ExtentManifest(&'a CellId), + Extent { + cell: &'a CellId, + ordinal: u32, + id: ExtentId, + }, +} + +pub struct CellGroupCheckpointChunk<'a> { + pub part: CellGroupCheckpointPart<'a>, + pub offset: u64, + pub length: u64, + pub bytes: &'a [u8], +} + +/// Non-cloneable admitted source. A dropped waiter must leave this value on its +/// joined physical worker until streaming finishes. Borrows prevent parallel +/// streams or retaining a chunk without copying it into separately charged work. +/// Neither this source nor a successful stream grants install/vote authority. +pub struct CellGroupCheckpoint { + group: GroupLog, + root: PreparedGroupManifest, + header: Vec, + limits: CellGroupCheckpointLimits, + started: bool, + _reservation: Permit, +} + +impl CellGroupCheckpoint { + /// Only a locally captured, unstreamed source can retain committed-source + /// authority. Received checkpoints use a different type and cannot mint it. + pub(super) fn archive_source( + &self, + ) -> Result<( + GroupLog, + PreparedGroupManifest, + raft::prelude::ConfState, + GroupMembership, + )> { + if self.started { + return Err(CellGroupCheckpointError::Invalid("source already streamed")); + } + let header = Header::decode(&self.header)?; + Ok(( + self.group.clone(), + self.root.clone(), + header.configuration()?, + header.membership, + )) + } + + /// Metadata for this immutable source cut; it does not establish publication. + pub fn snapshot(&self) -> Result { + Header::decode(&self.header)?.snapshot() + } + + /// Consume an unstreamed captured cut into a bounded publication capability. + /// The owner and engine independently require that it is still current. + pub fn prepare_publication(self) -> Result { + local::prepare(self) + } + + pub fn marker(&self) -> &GroupManifest { + self.root.marker() + } + + /// A one-shot stream. On error, previously emitted bytes are incomplete and + /// must remain unselected on the receiver. Flat-image digest verification + /// completes only at EOF; extent reads verify their full content first. + /// No engine/storage mutex is held while invoking the sink. + pub fn stream( + &mut self, + mut sink: impl FnMut(CellGroupCheckpointChunk<'_>) -> Result<()>, + ) -> Result<()> { + if self.started { + return Err(CellGroupCheckpointError::Invalid("source already streamed")); + } + self.started = true; + stream_bytes( + &self.group, + self.limits.max_chunk_bytes, + CellGroupCheckpointPart::Header, + &self.header, + &mut sink, + )?; + // Reconstruct canonical pages from the immutable verified closure. + // Exact page hashes must match; current mutable page-store state is + // never consulted and cannot substitute a later cell at this cut. + self.root + .images + .visit(&mut |reference, leaf, pin, left, right| -> Result<()> { + let page = page_wire::Page { + leaf: leaf.clone(), + left, + right, + }; + let bytes = page_wire::encode( + &page, + self.root.marker().route(), + ManifestLimits::default(), + )?; + if page_wire::digest(&bytes) != reference.digest { + return Err(CellGroupCheckpointError::Invalid( + "captured page commitment", + )); + } + stream_bytes( + &self.group, + self.limits.max_chunk_bytes, + CellGroupCheckpointPart::Page(reference.digest), + &bytes, + &mut sink, + )?; + drop(bytes); + let cell = &leaf.version.history.cell; + match pin { + CellImagePin::Empty => {} + CellImagePin::Materialized(pin) => { + let mut reader = pin.open_reader()?; + let mut offset = 0; + let mut digest = Sha256::new(); + while offset < reader.length() { + self.group.cache_revision()?; + let size = (reader.length() - offset) + .min(self.limits.max_chunk_bytes as u64) + as usize; + let bytes = reader.read_exact_at(offset, size)?; + digest.update(&bytes); + sink(CellGroupCheckpointChunk { + part: CellGroupCheckpointPart::Materialized(cell), + offset, + length: reader.length(), + bytes: &bytes, + })?; + offset += size as u64; + } + if <[u8; 32]>::from(digest.finalize()) != pin.state().digest { + return Err(CellGroupCheckpointError::Invalid( + "materialized image digest", + )); + } + } + CellImagePin::Extents(pin) => { + let manifest = pin + .manifest() + .encode() + .map_err(cellar_ltx::internal::StoreError::from)?; + stream_bytes( + &self.group, + self.limits.max_chunk_bytes, + CellGroupCheckpointPart::ExtentManifest(cell), + &manifest, + &mut sink, + )?; + drop(manifest); + for ordinal in 0..pin.manifest().geometry().extent_count() { + self.group.cache_revision()?; + let id = pin + .manifest() + .extent(ordinal) + .ok_or(CellGroupCheckpointError::Invalid("extent coverage"))?; + let bytes = pin.view().read_extent(id)?; + stream_bytes( + &self.group, + self.limits.max_chunk_bytes, + CellGroupCheckpointPart::Extent { cell, ordinal, id }, + &bytes, + &mut sink, + )?; + } + } + } + Ok(()) + })?; + self.group.cache_revision()?; + Ok(()) + } +} + +fn stream_bytes( + group: &GroupLog, + chunk_bytes: usize, + part: CellGroupCheckpointPart<'_>, + bytes: &[u8], + sink: &mut impl FnMut(CellGroupCheckpointChunk<'_>) -> Result<()>, +) -> Result<()> { + for (n, chunk) in bytes.chunks(chunk_bytes).enumerate() { + group.cache_revision()?; + sink(CellGroupCheckpointChunk { + part, + offset: (n * chunk_bytes) as u64, + length: bytes.len() as u64, + bytes: chunk, + })?; + } + Ok(()) +} diff --git a/crates/cellar/durability/raft/cell_groups/checkpoint/prepared.rs b/crates/cellar/durability/raft/cell_groups/checkpoint/prepared.rs new file mode 100644 index 000000000..d4d417a73 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/checkpoint/prepared.rs @@ -0,0 +1,107 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::super::CellGroupApplier; +use super::*; +use raft::prelude::Snapshot; + +struct Inner { + applier: CellGroupApplier, + root: PreparedGroupManifest, + header: Header, + snapshot: Snapshot, + _reservation: Permit, +} + +/// Receiver-local complete closure. Clones retain the same admission and physical +/// owners through queued engine work, including a lost publication response. +/// This capability proves storage, not that Raft has accepted the snapshot. +#[derive(Clone)] +pub struct PreparedCellGroupCheckpoint(Arc); + +impl std::fmt::Debug for PreparedCellGroupCheckpoint { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("PreparedCellGroupCheckpoint") + .field("marker", self.0.root.marker()) + .finish() + } +} + +impl PreparedCellGroupCheckpoint { + pub(super) fn new( + applier: CellGroupApplier, + root: PreparedGroupManifest, + header: Header, + reservation: Permit, + ) -> Result { + let snapshot = header.snapshot()?; + Ok(Self(Arc::new(Inner { + applier, + root, + header, + snapshot, + _reservation: reservation, + }))) + } + + pub fn marker(&self) -> &GroupManifest { + self.0.root.marker() + } + + pub fn snapshot(&self) -> &Snapshot { + &self.0.snapshot + } + + pub fn applier(&self) -> &CellGroupApplier { + &self.0.applier + } + + /// Complete physical closure. Admission still requires the engine-selected + /// marker and the current catalog permit; preparation alone cannot serve. + pub fn manifest(&self) -> &PreparedGroupManifest { + &self.0.root + } + + pub(crate) fn root(&self) -> &PreparedGroupManifest { + &self.0.root + } + + pub(crate) fn membership(&self) -> &GroupMembership { + &self.0.header.membership + } +} + +/// Decode persisted group snapshot metadata. This does not recover its physical +/// closure; startup must independently recover every referenced page and image. +pub(crate) fn persisted_marker( + snapshot: &Snapshot, + current: &GroupManifest, + membership: &GroupMembership, +) -> std::result::Result, crate::raft::storage::LogError> { + use crate::raft::storage::LogError; + if snapshot.data.is_empty() { + if snapshot.get_metadata().index != 1 || snapshot.get_metadata().term != 1 { + return Err(LogError::Corrupt( + "group snapshot lacks complete descriptor".into(), + )); + } + return Ok(None); + } + let header = Header::decode(&snapshot.data).map_err(|e| LogError::Corrupt(e.to_string()))?; + let metadata = snapshot.get_metadata(); + let conf = header + .configuration() + .map_err(|e| LogError::Corrupt(e.to_string()))?; + if header.root.route() != current.route() + || header.root.position().index.get() != metadata.index + || header.root.position().term.get() != metadata.term + || crate::raft::manifest::encode_conf(&conf) + .map_err(|e| LogError::Corrupt(e.to_string()))? + != crate::raft::manifest::encode_conf(metadata.get_conf_state()) + .map_err(|e| LogError::Corrupt(e.to_string()))? + || header.root.position().index > current.position().index + { + return Err(LogError::Corrupt("group snapshot metadata binding".into())); + } + membership.succeeds(&header.membership)?; + Ok(Some(header.root)) +} diff --git a/crates/cellar/durability/raft/cell_groups/checkpoint/receive.rs b/crates/cellar/durability/raft/cell_groups/checkpoint/receive.rs new file mode 100644 index 000000000..6468cb1dd --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/checkpoint/receive.rs @@ -0,0 +1,681 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::super::{ + apply::images::image_id, CellGroupApplier, CellImageGeometry, CellManifestLeaf, ManifestTree, +}; +use super::*; +use crate::application::consensus::cell_routing_key; +use cellar_logic::cell_group::{CellImageCommitment, CellKey}; +use cellar_ltx::{ + apply::ImageState, + generation::GenerationReceive, + internal::{ExtentManifest, ImageBinding, ImageRoot, RootView}, + stream_snapshot::ScratchPool, + TXID, +}; +use std::{collections::BTreeMap, path::PathBuf}; + +// Includes simultaneously received/verified extent bytes, receiver metadata +// decoding, header/page encoding, scratch verification and traversal stacks. +const RECEIVE_WORKSPACE: usize = 2 * EXTENT_BYTES as usize + 2 * 1024 * 1024; + +pub(super) trait Target { + fn storage(&self) -> &Arc>; + + fn route(&self) -> &cellar_logic::cell_group::GroupRoute; + + fn require_live(&self) -> Result<()>; + + fn check_header(&self, header: &Header) -> Result<()>; + + fn check_finish(&self, header: &Header) -> Result<()>; + + fn prepare( + &self, + tree: &ManifestTree, + pages: &mut super::super::ManifestPageStore, + header: &Header, + image: &mut dyn FnMut( + &CellManifestLeaf, + ) + -> std::result::Result, + ) -> Result; +} + +struct EngineTarget { + applier: CellGroupApplier, + base: GroupManifest, +} + +impl Target for EngineTarget { + fn storage(&self) -> &Arc> { + &self.applier.storage + } + + fn route(&self) -> &cellar_logic::cell_group::GroupRoute { + self.base.route() + } + + fn require_live(&self) -> Result<()> { + self.applier.group.cache_revision()?; + Ok(()) + } + + fn check_header(&self, header: &Header) -> Result<()> { + if header.root.position().index < self.base.position().index { + return Err(invalid("receiver history/range/cut")); + } + header + .membership + .succeeds(&self.applier.group.membership()?)?; + Ok(()) + } + + fn check_finish(&self, header: &Header) -> Result<()> { + if self.applier.group.group_manifest()?.as_ref() != Some(&self.base) { + return Err(invalid("incomplete closure or changed receiver base")); + } + header + .membership + .succeeds(&self.applier.group.membership()?)?; + Ok(()) + } + + fn prepare( + &self, + tree: &ManifestTree, + pages: &mut super::super::ManifestPageStore, + header: &Header, + image: &mut dyn FnMut( + &CellManifestLeaf, + ) + -> std::result::Result, + ) -> Result { + let mut root = + tree.prepare_bound_complete(&self.applier.group, pages, header.root.clone(), image)?; + root.base = Some(self.base.clone()); + Ok(root) + } +} + +impl CellGroupCheckpointPool { + /// Engine-bound receiving retains the exact selected base and every + /// membership/installation guard. Share source/receiver and scratch pools. + pub fn receive( + &self, + applier: CellGroupApplier, + scratch: ScratchPool, + directory: PathBuf, + ) -> Result { + let base = applier + .group + .group_manifest()? + .ok_or(invalid("receiver base"))?; + Ok(CellGroupCheckpointReceiver(State::new( + self, + EngineTarget { applier, base }, + scratch, + directory, + )?)) + } +} + +/// Engine-bound closure receiver. Finishing supplies storage preparation only; +/// the Raft owner's exact snapshot effect still controls atomic installation. +pub struct CellGroupCheckpointReceiver(State); + +pub(crate) trait WireReceiver { + fn wire_extent( + &self, + cell: &CellId, + ordinal: u32, + digest: [u8; 32], + length: u64, + ) -> Result; + + fn append_chunk(&mut self, chunk: CellGroupCheckpointChunk<'_>) -> Result<()>; +} + +impl WireReceiver for CellGroupCheckpointReceiver { + fn wire_extent( + &self, + cell: &CellId, + ordinal: u32, + digest: [u8; 32], + length: u64, + ) -> Result { + self.0.wire_extent(cell, ordinal, digest, length) + } + + fn append_chunk(&mut self, chunk: CellGroupCheckpointChunk<'_>) -> Result<()> { + self.0.append(chunk) + } +} + +impl CellGroupCheckpointReceiver { + pub fn append(&mut self, chunk: CellGroupCheckpointChunk<'_>) -> Result<()> { + self.0.append(chunk) + } + + pub fn finish(self) -> Result { + let received = self.0.finish()?; + PreparedCellGroupCheckpoint::new( + received.target.applier, + received.root, + received.header, + received.reservation, + ) + } +} + +pub(super) struct Received { + pub target: T, + pub root: PreparedGroupManifest, + pub header: Header, + pub reservation: Permit, +} + +/// Common physical decoder. Destination-specific authority checks remain in +/// sealed typed targets, and each public wrapper has a distinct output type. +pub(super) struct State { + target: T, + scratch: ScratchPool, + directory: PathBuf, + limits: CellGroupCheckpointLimits, + header: Option
, + part: Option, + cell: Option, + images: BTreeMap, + pages: Vec, + last_key: Option, + transferred: u64, + failed: bool, + reservation: Permit, +} + +impl State { + pub(super) fn new( + pool: &CellGroupCheckpointPool, + target: T, + scratch: ScratchPool, + directory: PathBuf, + ) -> Result { + let reservation = Self::reserve(pool, 0)?; + Self::with_reservation(pool, target, scratch, directory, reservation) + } + + pub(super) fn reserve(pool: &CellGroupCheckpointPool, extra_bytes: usize) -> Result { + let cells = pool.limits.max_cells as usize; + let extents = usize::try_from(pool.limits.max_transfer_bytes.div_ceil(EXTENT_BYTES)) + .map_err(|_| CellGroupCheckpointError::Limit)?; + let bytes = super::super::ManifestPins::maximum_bytes(pool.limits.max_cells) + .checked_mul(3) + .and_then(|n| n.checked_add(cells.checked_mul(4096)?)) + .and_then(|n| n.checked_add(extents.checked_mul(2048)?)) + .and_then(|n| n.checked_add(RECEIVE_WORKSPACE)) + .and_then(|n| n.checked_add(extra_bytes)) + .ok_or(CellGroupCheckpointError::Limit)?; + pool.budget + .acquire(bytes) + .ok_or(CellGroupCheckpointError::Busy) + } + + pub(super) fn with_reservation( + pool: &CellGroupCheckpointPool, + target: T, + scratch: ScratchPool, + directory: PathBuf, + reservation: Permit, + ) -> Result { + if !directory.is_absolute() || directory.as_os_str().len() > 4096 { + return Err(CellGroupCheckpointError::Invalid( + "receive scratch directory", + )); + } + target.require_live()?; + target + .storage() + .lock() + .map_err(|_| invalid("receiver page owner"))? + .pages + .require_live()?; + Ok(Self { + target, + scratch, + directory, + limits: pool.limits, + header: None, + part: None, + cell: None, + images: BTreeMap::new(), + pages: Vec::new(), + last_key: None, + transferred: 0, + failed: false, + reservation, + }) + } +} + +struct ReceivingCell { + leaf: CellManifestLeaf, + view: Option, + next_extent: u32, +} + +#[derive(Clone, PartialEq, Eq)] +enum Kind { + Header, + Page([u8; 32]), + Materialized(CellId), + ExtentManifest(CellId), + Extent(CellId, u32, ExtentId), +} + +impl Kind { + fn of(part: CellGroupCheckpointPart<'_>) -> Self { + match part { + CellGroupCheckpointPart::Header => Self::Header, + CellGroupCheckpointPart::Page(id) => Self::Page(id), + CellGroupCheckpointPart::Materialized(cell) => Self::Materialized(cell.clone()), + CellGroupCheckpointPart::ExtentManifest(cell) => Self::ExtentManifest(cell.clone()), + CellGroupCheckpointPart::Extent { cell, ordinal, id } => { + Self::Extent(cell.clone(), ordinal, id) + } + } + } +} + +struct Part { + kind: Kind, + length: u64, + received: u64, + data: Data, +} + +enum Data { + Bytes(Vec), + Image(GenerationReceive), +} + +fn invalid(message: &'static str) -> CellGroupCheckpointError { + CellGroupCheckpointError::Invalid(message) +} + +impl State { + pub(super) fn wire_extent( + &self, + cell: &CellId, + ordinal: u32, + digest: [u8; 32], + length: u64, + ) -> Result { + let pending = self.cell.as_ref().ok_or(invalid("extent without leaf"))?; + let id = pending + .view + .as_ref() + .and_then(|view| view.manifest().extent(ordinal)) + .ok_or(invalid("extent without manifest"))?; + if &pending.leaf.version.history.cell != cell + || ordinal != pending.next_extent + || id.bytes() != digest + || u64::from(id.length()) != length + { + return Err(invalid("wire extent binding")); + } + Ok(id) + } + + pub fn append(&mut self, chunk: CellGroupCheckpointChunk<'_>) -> Result<()> { + if self.failed { + return Err(invalid("failed receiver")); + } + self.failed = true; + self.append_inner(chunk)?; + self.failed = false; + Ok(()) + } + + fn append_inner(&mut self, chunk: CellGroupCheckpointChunk<'_>) -> Result<()> { + self.target.require_live()?; + if chunk.bytes.is_empty() || chunk.bytes.len() > self.limits.max_chunk_bytes { + return Err(CellGroupCheckpointError::Limit); + } + self.transferred = self + .transferred + .checked_add(chunk.bytes.len() as u64) + .filter(|n| *n <= self.limits.max_transfer_bytes) + .ok_or(CellGroupCheckpointError::Limit)?; + let kind = Kind::of(chunk.part); + if self.part.is_none() { + if chunk.offset != 0 { + return Err(invalid("receive part offset")); + } + self.part = Some(self.start_part(kind.clone(), chunk.length)?); + } + let part = self.part.as_mut().ok_or(invalid("receive part"))?; + let next = chunk + .offset + .checked_add(chunk.bytes.len() as u64) + .ok_or(CellGroupCheckpointError::Limit)?; + if part.kind != kind + || part.length != chunk.length + || part.received != chunk.offset + || next > part.length + { + return Err(invalid("mixed or noncontiguous receive part")); + } + match &mut part.data { + Data::Bytes(bytes) => bytes.extend_from_slice(chunk.bytes), + Data::Image(image) => image.append(chunk.offset, chunk.bytes)?, + } + part.received = next; + if next == part.length { + let completed = self.part.take().ok_or(invalid("receive completion"))?; + self.complete_part(completed)?; + } + Ok(()) + } + + fn start_part(&self, kind: Kind, length: u64) -> Result { + if length == 0 || length > self.limits.max_transfer_bytes { + return Err(CellGroupCheckpointError::Limit); + } + let limit = match &kind { + Kind::Header if self.header.is_none() && self.pages.is_empty() => HEADER_LIMIT as u64, + Kind::Page(_) if self.header.is_some() && self.cell.is_none() => { + if self.pages.len() >= self.header.as_ref().unwrap().root.root().cells() as usize { + return Err(invalid("extra checkpoint page")); + } + page_wire::MAX_PAGE_BYTES as u64 + } + Kind::Materialized(cell) => { + let pending = self.cell.as_ref().ok_or(invalid("image without leaf"))?; + if &pending.leaf.version.history.cell != cell + || !matches!( + pending.leaf.version.image.commitment, + CellImageCommitment::Materialized(_) + ) + || length != pending.leaf.geometry.length + { + return Err(invalid("materialized part binding")); + } + let CellImageCommitment::Materialized(digest) = + pending.leaf.version.image.commitment + else { + unreachable!() + }; + let expected = ImageState { + epoch: pending.leaf.version.image.epoch, + txid: TXID(pending.leaf.version.image.txid), + checksum: pending.leaf.version.image.checksum, + digest, + length, + page_size: pending.leaf.geometry.page_size, + }; + let id = image_id( + &pending.leaf.version.history, + pending.leaf.version.sequence.get(), + expected, + ) + .map_err(|_| invalid("received image identity"))?; + let storage = self + .target + .storage() + .lock() + .map_err(|_| invalid("receiver storage"))?; + let super::super::CellGroupApplyImages::Materialized { images, .. } = + &storage.images + else { + return Err(invalid("receiver image representation")); + }; + return Ok(Part { + kind, + length, + received: 0, + data: Data::Image(images.receive_image( + id, + expected, + &self.scratch, + &self.directory, + )?), + }); + } + Kind::ExtentManifest(cell) => { + let pending = self + .cell + .as_ref() + .ok_or(invalid("extent manifest without leaf"))?; + let CellImageCommitment::Extents { manifest_bytes, .. } = + pending.leaf.version.image.commitment + else { + return Err(invalid("extent manifest representation")); + }; + if &pending.leaf.version.history.cell != cell + || pending.view.is_some() + || length != u64::from(manifest_bytes) + { + return Err(invalid("extent manifest binding")); + } + // Includes the largest supported extent tree, while staying + // inside the admitted metadata workspace before allocation. + u64::from(manifest_bytes).min(1024 * 1024) + } + Kind::Extent(cell, ordinal, id) => { + let pending = self.cell.as_ref().ok_or(invalid("extent without leaf"))?; + let view = pending + .view + .as_ref() + .ok_or(invalid("extent without manifest"))?; + if &pending.leaf.version.history.cell != cell + || *ordinal != pending.next_extent + || view.manifest().extent(*ordinal) != Some(*id) + || length != u64::from(id.length()) + { + return Err(invalid("extent part binding")); + } + EXTENT_BYTES + } + _ => return Err(invalid("unexpected checkpoint part")), + }; + if length == 0 || length > limit { + return Err(CellGroupCheckpointError::Limit); + } + Ok(Part { + kind, + length, + received: 0, + data: Data::Bytes(Vec::with_capacity(length as usize)), + }) + } + + fn complete_part(&mut self, part: Part) -> Result<()> { + match (part.kind, part.data) { + (Kind::Header, Data::Bytes(bytes)) => { + let header = Header::decode(&bytes)?; + if header.root.route() != self.target.route() + || header.root.root().cells() > self.limits.max_cells + { + return Err(invalid("receiver history/range/cut")); + } + self.target.check_header(&header)?; + self.header = Some(header); + } + (Kind::Page(digest), Data::Bytes(bytes)) => { + let header = self.header.as_ref().ok_or(invalid("page without header"))?; + if page_wire::digest(&bytes) != digest { + return Err(invalid("received page digest")); + } + let limits = ManifestLimits { + max_cells: self.limits.max_cells, + max_depth: 32, + }; + let page = page_wire::decode(&bytes, header.root.route(), limits)?; + page.reference(&bytes, limits)?; + let key = cell_routing_key( + header.root.route().history.namespace, + &page.leaf.version.history.cell, + ); + if self.last_key.as_ref().is_some_and(|last| last >= &key) + || page.leaf.last_mutation.index > header.root.position().index + || page.leaf.last_mutation.term > header.root.position().term + { + return Err(invalid("checkpoint leaf order/cut")); + } + let mut storage = self + .target + .storage() + .lock() + .map_err(|_| invalid("receiver storage"))?; + storage.pages.put(digest, &bytes)?; + self.pages.push(storage.pages.pin(digest)?); + self.last_key = Some(key.clone()); + if page.leaf.version.image.txid == 0 { + CellImagePin::Empty.check(&page.leaf, header.root.route().schema_digest)?; + self.images.insert(key, (page.leaf, CellImagePin::Empty)); + } else { + self.cell = Some(ReceivingCell { + leaf: page.leaf, + view: None, + next_extent: 0, + }); + } + } + (Kind::Materialized(_), Data::Image(image)) => { + self.finish_cell(CellImagePin::Materialized(image.finish()?))? + } + (Kind::ExtentManifest(_), Data::Bytes(bytes)) => { + let pending = self.cell.as_mut().ok_or(invalid("extent cell"))?; + let CellImageCommitment::Extents { root, .. } = + pending.leaf.version.image.commitment + else { + return Err(invalid("extent leaf")); + }; + let manifest = ExtentManifest::decode(&bytes, ImageRoot::from_bytes(root)) + .map_err(cellar_ltx::internal::StoreError::from)?; + let image = pending.leaf.version.image; + if manifest.binding() + != (ImageBinding { + history: super::super::cell_image_history_digest( + &pending.leaf.version.history, + ), + schema: self.target.route().schema_digest, + epoch: image.epoch, + txid: image.txid, + checksum: image.checksum, + }) + || (CellImageGeometry { + length: manifest.geometry().length(), + page_size: manifest.geometry().page_size(), + }) != pending.leaf.geometry + { + return Err(invalid("extent image binding")); + } + let storage = self + .target + .storage() + .lock() + .map_err(|_| invalid("receiver storage"))?; + let super::super::CellGroupApplyImages::Extents { images, .. } = &storage.images + else { + return Err(invalid("receiver image representation")); + }; + pending.view = Some(images.stage_transient_root(manifest)?); + } + (Kind::Extent(_, _, id), Data::Bytes(bytes)) => { + let pin = { + let pending = self.cell.as_mut().ok_or(invalid("extent cell"))?; + let view = pending.view.as_ref().ok_or(invalid("extent view"))?; + let storage = self + .target + .storage() + .lock() + .map_err(|_| invalid("receiver storage"))?; + let super::super::CellGroupApplyImages::Extents { images, .. } = + &storage.images + else { + return Err(invalid("receiver image representation")); + }; + images.put_extent_bytes(id, &bytes)?; + drop(bytes); + pending.next_extent += 1; + if pending.next_extent == view.manifest().geometry().extent_count() { + Some(CellImagePin::Extents(Arc::new(images.seal_image_root( + view.manifest().clone(), + &mut vec![0; EXTENT_BYTES as usize], + )?))) + } else { + None + } + }; + if let Some(pin) = pin { + self.finish_cell(pin)?; + } + } + _ => return Err(invalid("receive part type")), + } + Ok(()) + } + + fn finish_cell(&mut self, pin: CellImagePin) -> Result<()> { + let pending = self.cell.take().ok_or(invalid("missing receive cell"))?; + pin.check(&pending.leaf, self.target.route().schema_digest)?; + let key = cell_routing_key( + self.target.route().history.namespace, + &pending.leaf.version.history.cell, + ); + if self.images.insert(key, (pending.leaf, pin)).is_some() { + return Err(invalid("duplicate receive cell")); + } + Ok(()) + } + + pub(super) fn finish(self) -> Result> { + if self.failed || self.part.is_some() || self.cell.is_some() { + return Err(invalid("incomplete checkpoint")); + } + let header = self.header.ok_or(invalid("missing checkpoint header"))?; + if self.images.len() != header.root.root().cells() as usize + || self.pages.len() != self.images.len() + { + return Err(invalid("incomplete closure or changed receiver base")); + } + self.target.require_live()?; + self.target.check_finish(&header)?; + let root = { + let mut storage = self + .target + .storage() + .lock() + .map_err(|_| invalid("receiver storage"))?; + let tree = ManifestTree::new( + header.root.route().clone(), + ManifestLimits { + max_cells: self.limits.max_cells, + max_depth: 32, + }, + )?; + self.target + .prepare(&tree, &mut storage.pages, &header, &mut |leaf| { + let key = cell_routing_key( + header.root.route().history.namespace, + &leaf.version.history.cell, + ); + self.images + .get(&key) + .filter(|(expected, _)| expected == leaf) + .map(|(_, pin)| pin.clone()) + .ok_or(super::super::ManifestError::Invalid( + "missing received cell image", + )) + })? + }; + self.target.require_live()?; + Ok(Received { + target: self.target, + root, + header, + reservation: self.reservation, + }) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/checkpoint/receive_archive.rs b/crates/cellar/durability/raft/cell_groups/checkpoint/receive_archive.rs new file mode 100644 index 000000000..f2d3ef4a9 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/checkpoint/receive_archive.rs @@ -0,0 +1,176 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::super::{ + CellGroupArchiveStore, CellManifestLeaf, ManifestError, ManifestPageStore, ManifestTree, +}; +use super::receive::{State, Target, WireReceiver}; +use super::*; +use cellar_logic::cell_group::GroupRoute; +use cellar_ltx::stream_snapshot::ScratchPool; +use std::{path::PathBuf, sync::Mutex}; + +#[cfg(test)] +#[path = "receive_archive_tests.rs"] +mod tests; + +impl Target for CellGroupArchiveStore { + fn storage(&self) -> &Arc> { + &self.storage + } + + fn route(&self) -> &GroupRoute { + CellGroupArchiveStore::route(self) + } + + fn require_live(&self) -> Result<()> { + CellGroupArchiveStore::require_live(self) + } + + fn check_header(&self, _header: &Header) -> Result<()> { + self.require_live() + } + + fn check_finish(&self, _header: &Header) -> Result<()> { + self.require_live() + } + + fn prepare( + &self, + tree: &ManifestTree, + pages: &mut ManifestPageStore, + header: &Header, + image: &mut dyn FnMut( + &CellManifestLeaf, + ) -> std::result::Result, + ) -> Result { + Ok(tree.prepare_archive_complete(&self.owner, pages, header.root.clone(), image)?) + } +} + +impl CellGroupCheckpointPool { + /// Reserve before opening/scanning an independently owned page inventory. + /// Includes worst-case inventory and directory metadata in this same bank. + pub(crate) fn prepare_archive_receive( + &self, + max_pages: usize, + ) -> Result { + let bytes = max_pages + .checked_mul(512) + .and_then(|n| n.checked_add(16384)) + .ok_or(CellGroupCheckpointError::Limit)?; + Ok(PreparedArchiveReceive { + pool: self.clone(), + reservation: State::::reserve(self, bytes)?, + }) + } + + /// Receive into private data stores, without a Raft mapping, surviving voter + /// disk or applied-cut observation. Shares the node's source/receiver quota. + pub fn receive_archive( + &self, + store: CellGroupArchiveStore, + scratch: ScratchPool, + directory: PathBuf, + ) -> Result { + Ok(CellGroupArchiveCheckpointReceiver(State::new( + self, store, scratch, directory, + )?)) + } +} + +pub(crate) struct PreparedArchiveReceive { + pool: CellGroupCheckpointPool, + reservation: Permit, +} + +impl PreparedArchiveReceive { + pub(crate) fn receive( + self, + store: CellGroupArchiveStore, + scratch: ScratchPool, + directory: PathBuf, + ) -> Result { + Ok(CellGroupArchiveCheckpointReceiver(State::with_reservation( + &self.pool, + store, + scratch, + directory, + self.reservation, + )?)) + } +} + +/// Physical archive receiver. It cannot produce the engine-bound checkpoint +/// capability accepted by SnapshotChange or the Raft owner. +pub struct CellGroupArchiveCheckpointReceiver(State); + +impl WireReceiver for CellGroupArchiveCheckpointReceiver { + fn wire_extent( + &self, + cell: &CellId, + ordinal: u32, + digest: [u8; 32], + length: u64, + ) -> Result { + self.0.wire_extent(cell, ordinal, digest, length) + } + + fn append_chunk(&mut self, chunk: CellGroupCheckpointChunk<'_>) -> Result<()> { + self.0.append(chunk) + } +} + +impl CellGroupArchiveCheckpointReceiver { + pub fn append(&mut self, chunk: CellGroupCheckpointChunk<'_>) -> Result<()> { + self.0.append(chunk) + } + + pub fn finish(self) -> Result { + let received = self.0.finish()?; + let snapshot = received.header.snapshot()?; + Ok(PreparedCellGroupArchiveCheckpoint(Arc::new(Inner { + store: received.target, + root: received.root, + header: received.header, + snapshot, + _reservation: received.reservation, + }))) + } +} + +struct Inner { + store: CellGroupArchiveStore, + root: PreparedGroupManifest, + header: Header, + snapshot: raft::prelude::Snapshot, + _reservation: Permit, +} + +/// Complete archive-local closure with no engine publication or serving right. +/// Clones share physical owners and admission, and have a distinct type from a +/// received Raft snapshot. Private owner tokens refuse every live engine. +#[derive(Clone)] +pub struct PreparedCellGroupArchiveCheckpoint(Arc); + +impl PreparedCellGroupArchiveCheckpoint { + pub fn marker(&self) -> &GroupManifest { + self.0.root.marker() + } + + pub fn manifest(&self) -> &PreparedGroupManifest { + &self.0.root + } + + /// Snapshot metadata only; copying it does not grant installation permission. + pub fn snapshot(&self) -> &raft::prelude::Snapshot { + &self.0.snapshot + } + + pub fn store(&self) -> &CellGroupArchiveStore { + &self.0.store + } + + pub(crate) fn membership(&self) -> &GroupMembership { + &self.0.header.membership + } +} diff --git a/crates/cellar/durability/raft/cell_groups/checkpoint/receive_archive_tests.rs b/crates/cellar/durability/raft/cell_groups/checkpoint/receive_archive_tests.rs new file mode 100644 index 000000000..aaf4dc250 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/checkpoint/receive_archive_tests.rs @@ -0,0 +1,52 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; + +fn pool() -> CellGroupCheckpointPool { + CellGroupCheckpointPool::new(CellGroupCheckpointLimits { + max_sources: 1, + max_retained_bytes: 64 * 1024 * 1024, + max_cells: 4, + max_transfer_bytes: 4 * 1024 * 1024, + max_chunk_bytes: 64 * 1024, + }) + .unwrap() +} + +#[test] +fn archive_inventory_admission_is_retained_before_storage_opens() { + let pool = pool(); + let pending = pool.prepare_archive_receive(1024).unwrap(); + assert_eq!(pool.usage().sources, 1); + let reserved = pool.usage().retained_bytes; + assert!(reserved >= 1024 * 512 + 16384); + assert!(matches!( + pool.prepare_archive_receive(1024), + Err(CellGroupCheckpointError::Busy) + )); + assert_eq!(pool.usage().retained_bytes, reserved); + drop(pending); + assert_eq!( + pool.usage(), + CellGroupCheckpointUsage { + sources: 0, + retained_bytes: 0 + } + ); + assert!(pool.prepare_archive_receive(1024).is_ok()); +} + +#[test] +fn archive_inventory_cannot_bypass_existing_receiver_byte_bank() { + let pool = pool(); + assert!(matches!( + pool.prepare_archive_receive(1_000_000), + Err(CellGroupCheckpointError::Busy) + )); + assert!(matches!( + pool.prepare_archive_receive(usize::MAX), + Err(CellGroupCheckpointError::Limit) + )); + assert_eq!(pool.usage().sources, 0); + assert_eq!(pool.usage().retained_bytes, 0); +} diff --git a/crates/cellar/durability/raft/cell_groups/checkpoint/seed.rs b/crates/cellar/durability/raft/cell_groups/checkpoint/seed.rs new file mode 100644 index 000000000..5b539be44 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/checkpoint/seed.rs @@ -0,0 +1,77 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! A complete authenticated descriptor must carry the exact consumed fleet +//! repair permit. These metadata checks mint no transport or catalog authority. +use super::*; +use crate::raft::member_state::{FleetMembershipPermit, FleetPermitPhase, RepairPhase}; + +#[cfg(test)] +#[path = "seed_tests.rs"] +mod tests; + +pub(super) fn check_membership( + membership: &GroupMembership, + permit: Option<&FleetMembershipPermit>, +) -> Result<()> { + match (membership.fleet_status(), permit) { + (None, None) => Ok(()), + (Some(status), Some(permit)) => { + let repair = membership + .repair_status()? + .ok_or(CellGroupCheckpointError::Invalid( + "seed has no managed repair", + ))?; + if &status.binding != permit.binding() + || status.consumed_sequence != permit.sequence() + || status.permit.as_ref() != Some(permit) + || status.phase != Some(FleetPermitPhase::Seeding) + || repair.phase != RepairPhase::Seeding + { + return Err(CellGroupCheckpointError::Invalid("seed fleet permit")); + } + permit.check_plan(&repair.plan)?; + Ok(()) + } + _ => Err(CellGroupCheckpointError::Invalid("seed managed profile")), + } +} + +pub(crate) fn seed_profile( + snapshot: &raft::prelude::Snapshot, + permit: Option<&FleetMembershipPermit>, +) -> Result<()> { + let header = Header::decode(&snapshot.data)?; + if header.snapshot()? != *snapshot { + return Err(CellGroupCheckpointError::Invalid("seed snapshot encoding")); + } + check_membership(&header.membership, permit) +} + +pub(crate) fn seed_initial_membership( + snapshot: &raft::prelude::Snapshot, + conf: &raft::prelude::ConfState, +) -> Result { + let header = Header::decode(&snapshot.data)?; + Ok(header.membership.seed_initial(conf)?) +} + +/// The responding process must also be a current, physically bound voter in +/// the authenticated checkpoint, independently of the joining target. +pub(crate) fn seed_source( + snapshot: &raft::prelude::Snapshot, + source: &cellar_logic::consensus::ReplicaIdentity, +) -> Result<()> { + let header = Header::decode(&snapshot.data)?; + let conf = header.configuration()?; + if header.root.route().history != source.scope + || !header.membership.permits_group(source) + || !conf + .voters + .iter() + .chain(&conf.voters_outgoing) + .any(|id| *id == source.member.get()) + { + return Err(CellGroupCheckpointError::Invalid("seed source member")); + } + Ok(()) +} diff --git a/crates/cellar/durability/raft/cell_groups/checkpoint/seed_tests.rs b/crates/cellar/durability/raft/cell_groups/checkpoint/seed_tests.rs new file mode 100644 index 000000000..e232b7129 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/checkpoint/seed_tests.rs @@ -0,0 +1,148 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Descriptive membership fixtures exercise profile checks. These values grant +//! no catalog, network, physical-install, startup or serving authority. +use super::*; +use crate::raft::member_state::{FleetGroupBinding, JoiningMember, RepairPlan, RepairStep}; +use cellar_logic::{ + cell_group::GroupHistory, + consensus::{ + ClusterId, DiskLineage, GroupId, IncarnationId, LogIndex, MemberId, NamespaceId, NodeId, + ProposalId, + }, +}; +use raft::prelude::{ConfChangeV2, ConfState}; + +fn fixture(genesis: bool) -> (GroupMembership, ConfState, FleetMembershipPermit) { + let history = GroupHistory { + cluster: ClusterId::new([1; 16]).unwrap(), + namespace: NamespaceId::new([2; 16]).unwrap(), + group: GroupId::new([3; 16]).unwrap(), + incarnation: IncarnationId::new([4; 16]).unwrap(), + }; + let binding = + FleetGroupBinding::new(IncarnationId::new([5; 16]).unwrap(), &history, [6; 32]).unwrap(); + let conf = ConfState { + voters: vec![1, 2, 3], + ..Default::default() + }; + let mut state = if genesis { + GroupMembership::managed_initial(&conf, binding.clone()).unwrap() + } else { + GroupMembership::initial(&conf) + }; + if !genesis { + let mut change = ConfChangeV2::default(); + state + .bind_fleet_enrollment(&mut change, &conf, &binding) + .unwrap(); + state = state + .applied(&change, &conf, &conf, LogIndex::new(2)) + .unwrap(); + } + let plan = RepairPlan { + id: ProposalId::new([7; 16]).unwrap(), + policy_revision: 1, + expected_revision: state.revision(), + obsolete: MemberId::new(3).unwrap(), + replacement: JoiningMember { + member: MemberId::new(4).unwrap(), + node: NodeId::new([8; 16]).unwrap(), + disk: DiskLineage::new([9; 16]).unwrap(), + }, + }; + let permit = FleetMembershipPermit::new( + binding, + 1, + ProposalId::new([10; 16]).unwrap(), + ProposalId::new([11; 16]).unwrap(), + &plan, + ) + .unwrap(); + let mut change = ConfChangeV2::default(); + state + .bind_managed_repair_start(&mut change, &conf, &plan, &permit) + .unwrap(); + let mut next = conf.clone(); + next.learners.push(4); + let state = state + .applied( + &change, + &conf, + &next, + LogIndex::new(state.revision().get() + 1), + ) + .unwrap(); + state.validate(&next, state.revision().get()).unwrap(); + (state, next, permit) +} + +#[test] +fn managed_seed_profile_requires_exact_consumed_permit_and_profile() { + for genesis in [false, true] { + let (state, conf, permit) = fixture(genesis); + check_membership(&state, Some(&permit)).unwrap(); + assert!(check_membership(&state, None).is_err()); + assert!(check_membership(&GroupMembership::initial(&conf), Some(&permit)).is_err()); + let plan = state.repair_status().unwrap().unwrap().plan; + let other = FleetMembershipPermit::new( + permit.binding().clone(), + 2, + ProposalId::new([12; 16]).unwrap(), + ProposalId::new([13; 16]).unwrap(), + &plan, + ) + .unwrap(); + assert!(check_membership(&state, Some(&other)).is_err()); + } +} + +#[test] +fn managed_seed_profile_refuses_aborted_repair() { + let (state, conf, permit) = fixture(true); + let mut change = ConfChangeV2::default(); + state + .bind_managed_repair_step( + &mut change, + &conf, + permit.repair_id().unwrap(), + RepairStep::AbortLearner, + &permit, + ) + .unwrap(); + let mut next = conf.clone(); + next.learners.clear(); + let aborted = state + .applied( + &change, + &conf, + &next, + LogIndex::new(state.revision().get() + 1), + ) + .unwrap(); + aborted.validate(&next, aborted.revision().get()).unwrap(); + assert!(check_membership(&aborted, Some(&permit)).is_err()); +} + +#[test] +fn managed_seed_base_preserves_genesis_and_committed_enrollment_origins() { + for genesis in [false, true] { + let (source, _, _) = fixture(genesis); + let birth = ConfState { + voters: vec![1, 2, 3], + ..Default::default() + }; + let staged = source.seed_initial(&birth).unwrap(); + staged.validate(&birth, 1).unwrap(); + assert_eq!(staged.revision(), LogIndex::new(1)); + assert!(staged.joined(MemberId::new(4).unwrap()).unwrap().is_none()); + assert_eq!(staged.fleet_status().is_some(), genesis); + source.succeeds(&staged).unwrap(); + assert!(staged.succeeds(&source).is_err()); + let wrong = ConfState { + voters: vec![1, 2, 5], + ..Default::default() + }; + assert!(source.seed_initial(&wrong).is_err()); + } +} diff --git a/crates/cellar/durability/raft/cell_groups/checkpoint/wire.rs b/crates/cellar/durability/raft/cell_groups/checkpoint/wire.rs new file mode 100644 index 000000000..584dbd0dd --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/checkpoint/wire.rs @@ -0,0 +1,93 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Owned bounded chunk identifiers. Extent identities are resolved from the +//! receiver's already verified manifest, never reconstructed from wire claims. +use super::*; +use serde::{Deserialize, Serialize}; + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(tag = "kind", deny_unknown_fields)] +pub(crate) enum Part { + Header, + Page { + digest: [u8; 32], + }, + Materialized { + cell: String, + }, + ExtentManifest { + cell: String, + }, + Extent { + cell: String, + ordinal: u32, + digest: [u8; 32], + }, +} + +pub(crate) struct Chunk { + pub part: Part, + pub offset: u64, + pub length: u64, + pub bytes: Vec, +} + +impl Chunk { + pub fn copy(chunk: CellGroupCheckpointChunk<'_>) -> Self { + let part = match chunk.part { + CellGroupCheckpointPart::Header => Part::Header, + CellGroupCheckpointPart::Page(digest) => Part::Page { digest }, + CellGroupCheckpointPart::Materialized(cell) => Part::Materialized { + cell: cell.as_str().to_owned(), + }, + CellGroupCheckpointPart::ExtentManifest(cell) => Part::ExtentManifest { + cell: cell.as_str().to_owned(), + }, + CellGroupCheckpointPart::Extent { cell, ordinal, id } => Part::Extent { + cell: cell.as_str().to_owned(), + ordinal, + digest: id.bytes(), + }, + }; + Self { + part, + offset: chunk.offset, + length: chunk.length, + bytes: chunk.bytes.to_vec(), + } + } + + pub fn append(self, receiver: &mut impl super::receive::WireReceiver) -> Result<()> { + let cell = match &self.part { + Part::Materialized { cell } + | Part::ExtentManifest { cell } + | Part::Extent { cell, .. } => Some( + CellId::new(cell).map_err(|_| CellGroupCheckpointError::Invalid("wire cell"))?, + ), + _ => None, + }; + let part = match &self.part { + Part::Header => CellGroupCheckpointPart::Header, + Part::Page { digest } => CellGroupCheckpointPart::Page(*digest), + Part::Materialized { .. } => { + CellGroupCheckpointPart::Materialized(cell.as_ref().unwrap()) + } + Part::ExtentManifest { .. } => { + CellGroupCheckpointPart::ExtentManifest(cell.as_ref().unwrap()) + } + Part::Extent { + ordinal, digest, .. + } => CellGroupCheckpointPart::Extent { + cell: cell.as_ref().unwrap(), + ordinal: *ordinal, + id: receiver.wire_extent(cell.as_ref().unwrap(), *ordinal, *digest, self.length)?, + }, + }; + receiver.append_chunk(CellGroupCheckpointChunk { + part, + offset: self.offset, + length: self.length, + bytes: &self.bytes, + }) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/admission.rs b/crates/cellar/durability/raft/cell_groups/controller/admission.rs new file mode 100644 index 000000000..94fd6bae9 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/admission.rs @@ -0,0 +1,125 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Node maintenance and work accounting become visible in one short admission +//! section. Physical producers and undelivered responses share the work lease. +use super::*; +use crate::application::consensus::admission::{Access, Admission, RegistrationRefusal}; +use crate::host_services::{ApplicationWork, ApplicationWorkGuard}; +use std::sync::MutexGuard; + +pub(super) struct Public { + admission: Arc, + work: Arc, +} + +pub(super) struct Registration<'a> { + // Serializes initial binding with standalone admissions. Never crosses an + // await, filesystem operation or application callback. + _configuration: MutexGuard<'a, ()>, + _gate: Option>, + pub work: Option, +} + +pub(super) struct Retention { + pub _permit: crate::raft::budget::Permit, + pub _work: Option, +} + +impl Shared { + pub(super) fn registration(&self) -> Result, RuntimeError> { + let configuration = self + .admission_configuration + .lock() + .map_err(|_| RuntimeError::Closed)?; + if self.closing.load(Ordering::Acquire) { + return Err(RuntimeError::Closed); + } + let (gate, work) = match self.public.get() { + Some(public) => { + let gate = public + .admission + .registration_reason(Access::Application) + .map_err(|refusal| match refusal { + RegistrationRefusal::Closed => RuntimeError::Closed, + RegistrationRefusal::Maintenance => RuntimeError::NotReady, + })?; + (Some(gate), Some(public.work.enter())) + } + None => (None, None), + }; + Ok(Registration { + _configuration: configuration, + _gate: gate, + work, + }) + } + + pub(super) fn application_paused(&self) -> bool { + self.public + .get() + .is_some_and(|public| public.admission.application_paused()) + } +} + +impl CellGroupController { + /// Conservative shutdown observation behind the sealed public barrier. + /// Poison, an unbound owner, held responses, pending provisioning or any + /// accepted physical application work make this proof unavailable. + pub(crate) fn local_producers_quiesced(&self) -> bool { + let shared = &self.handle.0; + let Ok(_configuration) = shared.admission_configuration.lock() else { + return false; + }; + let Some(public) = shared.public.get() else { + return false; + }; + if !public.admission.is_draining() + || public.admission.is_poisoned() + || public.work.snapshot().pending_events != 0 + || shared.commands.usage().1 != 0 + || shared.active_effects.load(Ordering::Acquire) != 0 + || !shared.groups.producers_quiesced() + { + return false; + } + let Ok(provisioning) = shared.provisioning.lock() else { + return false; + }; + provisioning.as_ref().is_none_or(|config| { + config.preparations.usage().1 == 0 && config.waiters.usage().1 == 0 + }) + } + + /// Install the canonical SDK's node admission before exposing ingress. A + /// different owner, or previously admitted untracked work, cannot be rebound. + pub(crate) fn bind_public_admission( + &self, + admission: Arc, + work: Arc, + ) -> Result<(), CellGroupControllerError> { + let shared = &self.handle.0; + let _configuration = shared + .admission_configuration + .lock() + .map_err(|_| CellGroupControllerError::Closed)?; + if let Some(old) = shared.public.get() { + return if Arc::ptr_eq(&old.admission, &admission) && Arc::ptr_eq(&old.work, &work) { + Ok(()) + } else { + Err(CellGroupControllerError::Invalid("public admission owner")) + }; + } + if shared.closing.load(Ordering::Acquire) + || shared.commands.usage().1 != 0 + || shared.active_effects.load(Ordering::Acquire) != 0 + { + return Err(CellGroupControllerError::Invalid( + "public admission already active/closed", + )); + } + shared + .public + .set(Public { admission, work }) + .map_err(|_| CellGroupControllerError::Invalid("public admission already bound")) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/archive/job.rs b/crates/cellar/durability/raft/cell_groups/controller/archive/job.rs new file mode 100644 index 000000000..a85475c94 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/archive/job.rs @@ -0,0 +1,294 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Each remote mutation follows its exact durable intent. Restart either finds +//! that publication or fences its pending ticket before allocating again. +use super::*; +use journal::{Journal, Pending, Record}; + +#[derive(Clone)] +pub(super) struct Context { + pub journal: Arc, + pub registry: CellGroupArchiveRegistry, + pub uses: Arc>>, + pub authority: Arc, +} + +pub(super) struct Authority { + directory: RangeDirectory, + birth: CellGroupBirth, +} + +impl Authority { + pub(super) async fn require_current(&self) -> Result<()> { + self.directory + .require_current_group_birth(&self.birth) + .await + .map_err(|error| Error::Binding(error.into())) + } +} + +pub(super) async fn open( + registration: &CellGroupArchiveRegistration, + runtime: &CellGroupRuntime, + execution: &ControllerExecution, + activation: Arc, +) -> Result { + runtime + .log() + .require_cell_group_archive(®istration.birth.route().history, activation) + .await + .map_err(|error| match error { + crate::raft::storage::LogError::Overloaded => Error::Runtime(RuntimeError::Overloaded), + error => Error::Runtime(error.into()), + })?; + let registry = registration + .directory + .open_group_archive(execution, ®istration.io, ®istration.birth) + .await?; + let filesystem = execution + .run(async { crate::asyncrt::fs() }) + .await + .map_err(|_| Error::Stopped)?; + let history = registration.birth.route().history; + let mut hash = sha2::Sha256::new(); + use sha2::Digest; + hash.update(b"cellar-node-group-archive-journal-1"); + for bytes in [ + history.cluster.bytes(), + history.namespace.bytes(), + history.group.bytes(), + history.incarnation.bytes(), + ] { + hash.update(bytes); + } + let directory = runtime + .log() + .limits() + .directory + .join("cell-group-archive-journals") + .join(hex::encode(hash.finalize())); + let engine = runtime + .log() + .storage_instance() + .map_err(|error| Error::Runtime(RuntimeError::Storage(error)))? + .bytes(); + let creation = registry.creation().clone(); + let journal = execution + .blocking(move || Journal::open(filesystem, directory, engine, creation)) + .map_err(|_| Error::Stopped)? + .await + .map_err(Error::Worker)??; + Ok(Context { + journal: Arc::new(journal), + registry, + uses: Arc::new(Mutex::new(Vec::new())), + authority: Arc::new(Authority { + directory: registration.directory.clone(), + birth: registration.birth.clone(), + }), + }) +} + +async fn load(context: &Context, execution: &ControllerExecution) -> Result { + let journal = context.journal.clone(); + execution + .blocking(move || journal.load()) + .map_err(|_| Error::Stopped)? + .await + .map_err(Error::Worker)? +} + +async fn store(context: &Context, execution: &ControllerExecution, record: &Record) -> Result<()> { + let journal = context.journal.clone(); + let record = record.clone(); + execution + .blocking(move || journal.store(&record)) + .map_err(|_| Error::Stopped)? + .await + .map_err(Error::Worker)? +} + +fn receiver( + registration: &CellGroupArchiveRegistration, +) -> Result { + Ok(registration + .checkpoints + .receive_archive( + registration.store.clone(), + registration.scratch.clone(), + registration.scratch_directory.clone(), + ) + .map_err(CellGroupArchiveError::from)?) +} + +async fn select( + context: &Context, + execution: &ControllerExecution, + record: &mut Record, + coverage: CellGroupArchiveCoverage, +) -> Result { + let head = coverage.hold().clone(); + if record.retire.is_some() { + return Err(Error::Invalid("unfinished archive hold retirement")); + } + record.retire = record.head.replace(head); + record.pending = Pending::None; + store(context, execution, record).await?; + retire(context, execution, record).await?; + Ok(coverage) +} + +async fn retire( + context: &Context, + execution: &ControllerExecution, + record: &mut Record, +) -> Result<()> { + if let Some(previous) = &record.retire { + let retained = { + let mut uses = context + .uses + .lock() + .map_err(|_| Error::Invalid("archive use lock"))?; + uses.retain(|usage| usage.live()); + uses.iter().any(|usage| usage.retains(previous)) + }; + if retained { + return Err(CellGroupArchiveError::Busy.into()); + } + context.registry.release(execution, previous).await?; + record.retire = None; + store(context, execution, record).await?; + } + Ok(()) +} + +pub(super) async fn drive( + context: &Context, + registration: &CellGroupArchiveRegistration, + runtime: &CellGroupRuntime, + execution: &ControllerExecution, +) -> Result { + let node = runtime.log().limits().node; + let group = registration.birth.route().history.group; + tracing::debug!(target: "cellar::raft::group_archive", ?node, ?group, stage = "drive"); + registration + .directory + .require_current_group_birth(®istration.birth) + .await + .map_err(CellGroupArchiveBindingError::from)?; + let mut record = load(context, execution).await?; + retire(context, execution, &mut record).await?; + if let Pending::Publish { ticket, intent } = record.pending.clone() { + match context + .registry + .reconcile_publication( + execution, + &intent, + ®istration.tails, + receiver(registration)?, + ) + .await + { + Ok(coverage) => return select(context, execution, &mut record, coverage).await, + Err(CellGroupArchiveError::Expired) => (), + Err(error) => return Err(error.into()), + } + // Absence alone does not disprove a delayed publishing CAS. Abort the + // exact ticket to invalidate its old conditional version, then reconcile + // once more in case publication won the race with that abort. + context.registry.abort(execution, &ticket).await?; + match context + .registry + .reconcile_publication( + execution, + &intent, + ®istration.tails, + receiver(registration)?, + ) + .await + { + Ok(coverage) => return select(context, execution, &mut record, coverage).await, + Err(CellGroupArchiveError::Expired) => (), + Err(error) => return Err(error.into()), + } + record.pending = Pending::None; + store(context, execution, &record).await?; + } + if matches!(record.pending, Pending::None) { + let current = runtime + .status(registration.birth.route().history) + .await + .map_err(Error::Runtime)?; + if let Some(head) = &record.head { + if head.marker().position().index >= current.applied.index { + let coverage = context + .registry + .verify_hold( + execution, + head, + ®istration.tails, + receiver(registration)?, + ) + .await?; + return Ok(coverage); + } + } + let intent = context.registry.prepare_upload(execution).await?; + record.pending = Pending::Reserve(intent); + store(context, execution, &record).await?; + } + if let Pending::Reserve(intent) = &record.pending { + match context.registry.reserve(execution, intent).await { + Ok(ticket) => record.pending = Pending::Upload(ticket), + Err(CellGroupArchiveError::Expired | CellGroupArchiveError::Conflict) => { + record.pending = Pending::None; + store(context, execution, &record).await?; + return Err(CellGroupArchiveError::Expired.into()); + } + Err(error) => return Err(error.into()), + } + store(context, execution, &record).await?; + } + let Pending::Upload(ticket) = record.pending else { + return Err(Error::Invalid("archive upload stage")); + }; + let upload = context + .registry + .open_upload( + execution, + ®istration.owners, + &ticket, + ®istration.tails, + ) + .await?; + let source = runtime + .capture_checkpoint( + registration.birth.route().history, + ®istration.checkpoints, + ) + .await + .map_err(CellGroupArchiveError::from)?; + tracing::debug!(target: "cellar::raft::group_archive", ?node, ?group, stage = "captured", + index = source.marker().position().index.get()); + let uploaded = upload + .upload_cut(execution, ®istration.tails, source) + .await?; + tracing::debug!(target: "cellar::raft::group_archive", ?node, ?group, stage = "uploaded"); + let publication = upload + .prepare_publication( + execution, + ®istration.tails, + &uploaded, + receiver(registration)?, + ) + .await?; + tracing::debug!(target: "cellar::raft::group_archive", ?node, ?group, stage = "verified"); + record.pending = Pending::Publish { + ticket, + intent: publication.intent().clone(), + }; + store(context, execution, &record).await?; + let coverage = publication.commit(execution).await?; + tracing::debug!(target: "cellar::raft::group_archive", ?node, ?group, stage = "published"); + select(context, execution, &mut record, coverage).await +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/archive/journal.rs b/crates/cellar/durability/raft/cell_groups/controller/archive/journal.rs new file mode 100644 index 000000000..260a97707 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/archive/journal.rs @@ -0,0 +1,250 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! One atomically replaced, fsynced record under an exclusive node-disk lock. +use super::*; +use cellar_ltx::{FileSystem, HostFileLock}; +use serde::{Deserialize, Serialize}; + +pub(super) const LIMIT: usize = 32 * 1024; +const SEED_DOMAIN: &str = "cellar-cell-group-seed-archive-journal-1"; + +#[derive(Clone, Serialize, Deserialize, PartialEq, Eq)] +#[serde(deny_unknown_fields)] +struct Identity { + domain: String, + engine: [u8; 16], + creation: CellGroupArchiveCreation, +} + +#[derive(Clone, Default, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) enum Pending { + #[default] + None, + Reserve(CellGroupArchiveIntent), + Upload(CellGroupArchiveUploadTicket), + Publish { + ticket: CellGroupArchiveUploadTicket, + intent: CellGroupArchiveIntent, + }, +} + +#[derive(Clone, Default, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Record { + pub head: Option, + pub retire: Option, + pub pending: Pending, +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Envelope { + identity: Identity, + record: Record, +} + +pub(super) struct Journal { + filesystem: Arc, + directory: std::path::PathBuf, + identity: Identity, + _lock: HostFileLock, +} + +impl Journal { + pub(super) fn open( + filesystem: Arc, + directory: std::path::PathBuf, + engine: [u8; 16], + creation: CellGroupArchiveCreation, + ) -> Result { + Self::open_domain( + filesystem, + directory, + engine, + creation, + "cellar-cell-group-archive-journal-1", + true, + ) + } + + pub(super) fn open_seed( + filesystem: Arc, + directory: std::path::PathBuf, + engine: [u8; 16], + creation: CellGroupArchiveCreation, + ) -> Result { + Self::open_domain(filesystem, directory, engine, creation, SEED_DOMAIN, true) + } + + pub(super) fn open_seed_existing( + filesystem: Arc, + directory: std::path::PathBuf, + engine: [u8; 16], + creation: CellGroupArchiveCreation, + ) -> Result { + Self::open_domain(filesystem, directory, engine, creation, SEED_DOMAIN, false) + } + + pub(super) fn inspect_seed( + filesystem: &dyn FileSystem, + directory: &std::path::Path, + engine: [u8; 16], + ) -> Result { + let bytes = read(filesystem, directory, "identity")? + .ok_or(Error::Invalid("seed journal identity missing"))?; + let identity: Identity = serde_json::from_slice(&bytes) + .map_err(|_| Error::Invalid("seed journal identity encoding"))?; + if identity.domain != SEED_DOMAIN + || identity.engine != engine + || encode(&identity)? != bytes + { + return Err(Error::Invalid("seed journal identity binding")); + } + Ok(identity.creation) + } + + fn open_domain( + filesystem: Arc, + directory: std::path::PathBuf, + engine: [u8; 16], + creation: CellGroupArchiveCreation, + purpose: &str, + initialize: bool, + ) -> Result { + if initialize { + cellar_ltx::host::create_dir_all_durable(filesystem.as_ref(), &directory)?; + } else if !filesystem.symlink_metadata(&directory)?.is_dir { + return Err(Error::Invalid("seed journal directory type")); + } + let lock = filesystem.lock_file(&directory.join("owner.lock"))?; + let identity = Identity { + domain: purpose.into(), + engine, + creation, + }; + let journal = Self { + filesystem, + directory, + identity, + _lock: lock, + }; + let guard = journal.read("identity")?; + let record = journal.read("state")?; + match (guard, record) { + (None, None) if initialize => { + let bytes = encode(&journal.identity)?; + let mut file = journal + .filesystem + .create_new(&journal.directory.join("identity"))?; + file.write_all(&bytes)?; + file.sync_all()?; + journal.filesystem.sync_all(&journal.directory)?; + // Only this fresh, confirmed exclusive creation initializes a + // journal. An interrupted initializer is never reconstructed. + journal.store(&Record::default())?; + } + (Some(bytes), Some(_)) if bytes == encode(&journal.identity)? => { + journal.load()?; + } + _ => { + return Err(Error::Invalid( + "archive journal identity/state lost or changed", + )) + } + } + Ok(journal) + } + + fn read(&self, name: &str) -> Result>> { + read(self.filesystem.as_ref(), &self.directory, name) + } + + pub(super) fn load(&self) -> Result { + if self.read("identity")?.as_deref() != Some(encode(&self.identity)?.as_slice()) { + return Err(Error::Invalid("archive journal identity changed")); + } + let bytes = self + .read("state")? + .ok_or(Error::Invalid("archive journal state lost"))?; + let envelope: Envelope = serde_json::from_slice(&bytes) + .map_err(|_| Error::Invalid("archive journal encoding"))?; + if envelope.identity != self.identity || encode(&envelope)? != bytes { + return Err(Error::Invalid("archive journal binding/canonical bytes")); + } + Ok(envelope.record) + } + + pub(super) fn store(&self, record: &Record) -> Result<()> { + let bytes = encode(&Envelope { + identity: self.identity.clone(), + record: record.clone(), + })?; + let temporary = self.directory.join("next"); + // Only the locked owner removes an unselected interrupted temporary. + match self.filesystem.remove_file(&temporary) { + Ok(()) => (), + Err(error) if error.kind() == std::io::ErrorKind::NotFound => (), + Err(error) => return Err(error.into()), + } + let mut file = self.filesystem.create_new(&temporary)?; + file.write_all(&bytes)?; + file.sync_all()?; + self.filesystem + .rename(&temporary, &self.directory.join("state"))?; + self.filesystem.sync_all(&self.directory)?; + if self.read("state")?.as_deref() != Some(bytes.as_slice()) { + return Err(Error::Invalid("archive journal write readback")); + } + Ok(()) + } +} + +pub(super) fn seed_name(history: GroupHistory) -> String { + use sha2::Digest; + let mut hash = sha2::Sha256::new(); + for bytes in [ + history.cluster.bytes(), + history.namespace.bytes(), + history.group.bytes(), + history.incarnation.bytes(), + ] { + hash.update(bytes); + } + hex::encode(hash.finalize()) +} + +fn read( + filesystem: &dyn FileSystem, + directory: &std::path::Path, + name: &str, +) -> Result>> { + let path = directory.join(name); + let metadata = match filesystem.symlink_metadata(&path) { + Ok(metadata) => metadata, + Err(error) if error.kind() == std::io::ErrorKind::NotFound => return Ok(None), + Err(error) => return Err(error.into()), + }; + if !metadata.is_file || metadata.len == 0 || metadata.len > LIMIT as u64 { + return Err(Error::Invalid("archive journal file bounds/type")); + } + let mut file = filesystem.open(&path)?; + let length = file.file_len()?; + if length != metadata.len { + return Err(Error::Invalid("archive journal changed during read")); + } + Ok(Some(file.read_exact_at(0, length as usize)?)) +} + +fn encode(value: &impl Serialize) -> Result> { + let bytes = + serde_json::to_vec(value).map_err(|_| Error::Invalid("archive journal encoding"))?; + if bytes.len() > LIMIT { + return Err(Error::Invalid("archive journal record limit")); + } + Ok(bytes) +} + +#[cfg(test)] +#[path = "journal_tests.rs"] +mod tests; diff --git a/crates/cellar/durability/raft/cell_groups/controller/archive/journal_tests.rs b/crates/cellar/durability/raft/cell_groups/controller/archive/journal_tests.rs new file mode 100644 index 000000000..014bffd49 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/archive/journal_tests.rs @@ -0,0 +1,167 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use cellar_logic::{cell_group::*, consensus::*}; + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[allow(clippy::disallowed_methods)] +async fn seed_journal_reopens_only_existing_exact_engine_and_purpose() -> anyhow::Result<()> { + let domain = crate::sdk::ExecutionDomain::new(tokio::runtime::Handle::current()); + let history = GroupHistory { + cluster: ClusterId::new([21; 16])?, + namespace: NamespaceId::new([22; 16])?, + group: GroupId::new([23; 16])?, + incarnation: IncarnationId::new([24; 16])?, + }; + let archive = CellGroupArchivePool::new(Default::default())?.for_group( + Arc::new(object_store::memory::InMemory::new()), + object_store::path::Path::from("seed-journal"), + history, + )?; + let prepared = archive + .prepare_registry_creation( + &domain, + GroupRoute { + history, + range: CellRange::all(), + epoch: RouteEpoch::new(1)?, + birth_digest: [25; 32], + schema_digest: [26; 32], + }, + ) + .await?; + let creation = prepared.intent().clone(); + let registry = prepared.commit(&domain).await?; + let intent = registry.prepare_upload(&domain).await?; + let disk = tempfile::tempdir()?; + let root = disk.path().to_owned(); + domain.blocking(move || -> Result<()> { + let fs: Arc = Arc::new(cellar_ltx::DirectFileSystem); + let path = root.join(seed_name(history)); + assert!(Journal::open_seed_existing(fs.clone(), path.clone(), [27; 16], creation.clone()).is_err()); + assert!(matches!(fs.symlink_metadata(&path), Err(error) if error.kind() == std::io::ErrorKind::NotFound)); + let journal = Journal::open_seed(fs.clone(), path.clone(), [27; 16], creation.clone())?; + journal.store(&Record { pending: Pending::Reserve(intent.clone()), ..Default::default() })?; + let identity = fs.read(&path.join("identity"))?; + let saved = fs.read(&path.join("state"))?; + assert_eq!(Journal::inspect_seed(fs.as_ref(), &path, [27; 16])?, creation); + assert!(Journal::inspect_seed(fs.as_ref(), &path, [28; 16]).is_err()); + drop(journal); + assert!(Journal::open_seed_existing(fs.clone(), path.clone(), [28; 16], creation.clone()).is_err()); + assert_eq!(fs.read(&path.join("identity"))?, identity); + assert_eq!(fs.read(&path.join("state"))?, saved); + let reopened = Journal::open_seed_existing(fs.clone(), path.clone(), [27; 16], creation.clone())?; + let Pending::Reserve(recovered) = reopened.load()?.pending else { panic!("lost pending seed reserve") }; + assert_eq!(recovered, intent); + drop(reopened); + let ordinary = root.join("ordinary"); + let journal = Journal::open(fs.clone(), ordinary.clone(), [27; 16], creation.clone())?; + assert!(Journal::inspect_seed(fs.as_ref(), &ordinary, [27; 16]).is_err()); + drop(journal); + fs.remove_file(&path.join("state"))?; + assert!(Journal::open_seed_existing(fs.clone(), path.clone(), [27; 16], creation.clone()).is_err()); + assert!(matches!(fs.symlink_metadata(&path.join("state")), Err(error) if error.kind() == std::io::ErrorKind::NotFound)); + fs.remove_file(&path.join("identity"))?; + assert!(Journal::open_seed_existing(fs.clone(), path.clone(), [27; 16], creation).is_err()); + assert!(matches!(fs.symlink_metadata(&path.join("identity")), Err(error) if error.kind() == std::io::ErrorKind::NotFound)); + Ok(()) + })?.await??; + domain.shutdown().await; + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[allow(clippy::disallowed_methods)] +async fn archive_journal_reopens_exact_intent_and_refuses_lost_state() -> anyhow::Result<()> { + let domain = crate::sdk::ExecutionDomain::new(tokio::runtime::Handle::current()); + let history = GroupHistory { + cluster: ClusterId::new([1; 16])?, + namespace: NamespaceId::new([2; 16])?, + group: GroupId::new([3; 16])?, + incarnation: IncarnationId::new([4; 16])?, + }; + let archive = CellGroupArchivePool::new(Default::default())?.for_group( + Arc::new(object_store::memory::InMemory::new()), + object_store::path::Path::from("journal"), + history, + )?; + let creation = archive + .prepare_registry_creation( + &domain, + GroupRoute { + history, + range: CellRange::all(), + epoch: RouteEpoch::new(1)?, + birth_digest: [5; 32], + schema_digest: [6; 32], + }, + ) + .await?; + let expected = creation.intent().clone(); + let registry = creation.commit(&domain).await?; + let intent = registry.prepare_upload(&domain).await?; + let disk = tempfile::tempdir()?; + let directory = disk.path().join("owner"); + let path = directory.clone(); + let identity = expected.clone(); + let journal = domain + .blocking(move || { + Journal::open( + Arc::new(cellar_ltx::DirectFileSystem), + path, + [7; 16], + identity, + ) + })? + .await??; + let saved = intent.clone(); + domain + .blocking(move || -> Result<()> { + journal.store(&Record { + pending: Pending::Reserve(saved), + ..Default::default() + })?; + Ok(()) + })? + .await??; + let path = directory.clone(); + let identity = expected.clone(); + let reopened = domain + .blocking(move || { + Journal::open( + Arc::new(cellar_ltx::DirectFileSystem), + path, + [7; 16], + identity, + ) + })? + .await??; + let recovered = domain.blocking(move || reopened.load())?.await??; + let Pending::Reserve(recovered) = recovered.pending else { + anyhow::bail!("lost reserve intent") + }; + assert_eq!(recovered, intent); + let ticket = registry.reserve(&domain, &recovered).await?; + assert_eq!(registry.reserve(&domain, &intent).await?, ticket); + let path = directory.clone(); + let identity = expected.clone(); + domain + .blocking(move || -> Result<()> { + let filesystem = cellar_ltx::DirectFileSystem; + filesystem.remove_file(&path.join("state"))?; + let before = filesystem.read(&path.join("identity"))?; + assert!(Journal::open(Arc::new(filesystem), path.clone(), [7; 16], identity).is_err()); + assert_eq!( + cellar_ltx::DirectFileSystem.read(&path.join("identity"))?, + before + ); + assert!(cellar_ltx::DirectFileSystem + .metadata(&path.join("state")) + .is_err()); + Ok(()) + })? + .await??; + registry.abort(&domain, &ticket).await?; + domain.shutdown().await; + Ok(()) +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/archive/mod.rs b/crates/cellar/durability/raft/cell_groups/controller/archive/mod.rs new file mode 100644 index 000000000..ada23f85a --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/archive/mod.rs @@ -0,0 +1,615 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! One bounded, fair archive lane on the existing node controller. Published +//! markers are observations, never S3 command settlement or serving authority. +mod job; +mod journal; +mod provision; +pub(in crate::raft::cell_groups::controller) mod seed; +pub(crate) use provision::Provisioning; +pub(crate) use seed::{Client as ArchiveSeedClient, Descriptor as ArchiveSeedDescriptor}; +pub(in crate::raft::cell_groups::controller) mod settlement; + +use super::calls::Delivery; +use super::*; +use crate::raft::cell_groups::archive_compaction::ArchiveActivation; +use crate::raft::cell_groups::{archive::ControllerExecution, *}; +use cellar_logic::{cell_group::GroupHistory, consensus::LogIndex}; +use futures_util::future::BoxFuture; +use std::collections::VecDeque; +use tokio::sync::oneshot; + +#[derive(Debug)] +pub enum CellGroupArchiveControlError { + Invalid(&'static str), + Runtime(RuntimeError), + Archive(CellGroupArchiveError), + Binding(CellGroupArchiveBindingError), + Journal(std::io::Error), + Worker(crate::asyncrt::TaskPanic), + Stopped, +} + +impl std::fmt::Display for CellGroupArchiveControlError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "group archive controller: {self:?}") + } +} + +impl std::error::Error for CellGroupArchiveControlError { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + match self { + Self::Runtime(error) => Some(error), + Self::Archive(error) => Some(error), + Self::Binding(error) => Some(error), + Self::Journal(error) => Some(error), + Self::Worker(error) => Some(error), + _ => None, + } + } +} + +impl From for CellGroupArchiveControlError { + fn from(error: CellGroupArchiveError) -> Self { + Self::Archive(error) + } +} + +impl From for CellGroupArchiveControlError { + fn from(error: CellGroupArchiveBindingError) -> Self { + Self::Binding(error) + } +} + +impl From for CellGroupArchiveControlError { + fn from(error: std::io::Error) -> Self { + Self::Journal(error) + } +} + +impl From for CellGroupArchiveControlError { + fn from(_: crate::asyncrt::DomainClosed) -> Self { + Self::Stopped + } +} + +type Error = CellGroupArchiveControlError; +type Result = std::result::Result; + +/// Share all admission pools across this node's registrations. The binding must +/// already be initialized in the namespace catalog. The controller owns and +/// closes the private verifier store after joining archive work. Use fresh stores +/// on restart. No application callback is needed for background publication. +#[derive(Clone)] +pub struct CellGroupArchiveRegistration { + pub directory: RangeDirectory, + pub birth: CellGroupBirth, + pub io: CellGroupArchivePool, + pub owners: CellGroupArchiveUploadPool, + pub tails: CellGroupArchiveTailPool, + pub checkpoints: CellGroupCheckpointPool, + pub store: CellGroupArchiveStore, + pub scratch: cellar_ltx::stream_snapshot::ScratchPool, + pub scratch_directory: std::path::PathBuf, +} + +#[derive(Clone, Default)] +pub struct CellGroupArchiveStatus { + /// Last complete cut verified/published by this owner. Advisory historical + /// metadata only; a consumer still needs a retained physical coverage proof. + pub published: Option, + pub attempts: u64, + pub error: Option>, + pub stopped: bool, +} + +pub struct CellGroupArchiveCall { + reply: oneshot::Receiver>, +} + +impl CellGroupArchiveCall { + /// Caller cancellation does not cancel the admitted archive owner. + pub async fn response(self) -> std::result::Result<(), Arc> { + self.reply + .await + .map_err(|_| Arc::new(Error::Stopped))? + .result + } +} + +pub(super) struct Request { + registration: CellGroupArchiveRegistration, + activation: Arc, + reply: RegistrationReply, +} + +struct RegistrationReply { + sender: oneshot::Sender>, + permit: Arc, +} + +impl RegistrationReply { + fn send(self, result: std::result::Result<(), Arc>) { + let _ = self.sender.send(Delivery { + result, + _permit: self.permit, + }); + } +} + +impl CellGroupController { + /// Install background archival for an already attached group. Admission is + /// node-bounded and survives response cancellation. This never enables S3 + /// acknowledgements or authorizes local log reclamation. + pub fn archive( + &self, + registration: CellGroupArchiveRegistration, + ) -> Result { + let shared = &self.handle.0; + if shared.closing.load(Ordering::Acquire) { + return Err(Error::Stopped); + } + let group = registration.birth.route().history; + if !shared.groups.contains(&group).map_err(Error::Runtime)? { + return Err(Error::Runtime(RuntimeError::NotFound)); + } + let local = shared + .runtime + .log() + .cell_group(&group) + .map_err(|error| Error::Runtime(error.into()))?; + if local + .group_manifest() + .map_err(|error| Error::Runtime(error.into()))? + .as_ref() + .is_none_or(|marker| marker.route() != registration.birth.route()) + { + return Err(Error::Invalid("archive registration local group birth")); + } + if registration.store.route() != registration.birth.route() + || !registration.scratch_directory.is_absolute() + || registration.scratch_directory.as_os_str().len() > 4096 + { + return Err(Error::Invalid("archive registration binding/path")); + } + let permit = shared + .commands + .acquire(8192) + .ok_or(Error::Runtime(RuntimeError::Overloaded))?; + let activation = local + .archive_compaction + .begin_activation() + .map_err(|error| Error::Runtime(error.into()))?; + let (reply, receiver) = oneshot::channel(); + shared + .requests + .try_send(requests::Request::Archive(Box::new(Request { + registration, + activation, + reply: RegistrationReply { + sender: reply, + permit: Arc::new(permit), + }, + }))) + .map_err(|error| match error { + tokio::sync::mpsc::error::TrySendError::Closed(_) => Error::Stopped, + tokio::sync::mpsc::error::TrySendError::Full(_) => { + Error::Runtime(RuntimeError::Overloaded) + } + })?; + Ok(CellGroupArchiveCall { reply: receiver }) + } + + pub fn archive_status( + &self, + group: GroupHistory, + ) -> std::result::Result, RuntimeError> { + Ok(self + .handle + .0 + .archives + .lock() + .map_err(|_| RuntimeError::Closed)? + .get(&group) + .cloned()) + } +} + +struct Entry { + registration: CellGroupArchiveRegistration, + activation: Option>, + context: Option, + reply: Option, + queued: bool, + latest_applied: Option, + dirty: bool, + fatal: bool, + retry_at: u64, + waiters: Vec, +} + +pub(super) struct Completed { + group: GroupHistory, + context: Option, + result: Result, + observation: std::result::Result, + waiters: Vec, +} + +pub(super) struct Manager { + entries: BTreeMap, + queue: VecDeque, + job: Option>, + execution: ControllerExecution, +} + +impl Manager { + pub fn new(domain: crate::asyncrt::DomainToken) -> Self { + Self { + entries: BTreeMap::new(), + queue: VecDeque::new(), + job: None, + execution: ControllerExecution::new(domain), + } + } + + pub fn request(&mut self, request: Request, shared: &Shared) { + let group = request.registration.birth.route().history; + if self.entries.contains_key(&group) || self.entries.len() >= shared.limits.max_groups { + request.reply.send(Err(Arc::new(Error::Invalid( + "duplicate/full archive roster", + )))); + return; + } + shared + .archives + .lock() + .unwrap_or_else(|error| error.into_inner()) + .insert(group, CellGroupArchiveStatus::default()); + self.entries.insert( + group, + Entry { + registration: request.registration, + activation: Some(request.activation), + context: None, + reply: Some(request.reply), + queued: true, + latest_applied: None, + dirty: true, + fatal: false, + retry_at: 0, + waiters: Vec::new(), + }, + ); + self.queue.push_back(group); + } + + pub fn provision( + &mut self, + registration: CellGroupArchiveRegistration, + shared: &Shared, + ) -> std::result::Result<(), CellGroupControllerError> { + let group = registration.birth.route().history; + if self.entries.contains_key(&group) || self.entries.len() >= shared.limits.max_groups { + return Err(CellGroupControllerError::Invalid( + "duplicate/full provisioned archive roster", + )); + } + let local = shared + .runtime + .log() + .cell_group(&group) + .map_err(RuntimeError::from)?; + let activation = local + .archive_compaction + .begin_activation() + .map_err(RuntimeError::from)?; + shared + .archives + .lock() + .map_err(|_| CellGroupControllerError::Closed)? + .insert(group, CellGroupArchiveStatus::default()); + // The physical group/preparation already holds bounded admission. This + // internal handoff consumes no second ordinary command slot or waiter. + self.entries.insert( + group, + Entry { + registration, + activation: Some(activation), + context: None, + reply: None, + queued: true, + latest_applied: None, + dirty: true, + fatal: false, + retry_at: 0, + waiters: Vec::new(), + }, + ); + self.queue.push_back(group); + Ok(()) + } + + pub fn applied(&mut self, group: GroupHistory, index: LogIndex) { + if let Some(entry) = self.entries.get_mut(&group) { + entry.latest_applied = Some( + entry + .latest_applied + .map_or(index, |previous| previous.max(index)), + ); + if !entry.fatal { + entry.dirty = true; + if !entry.queued { + entry.queued = true; + self.queue.push_back(group); + } + } + } + } + + pub fn settle(&mut self, request: settlement::Request) { + let group = request.route.history; + let Some(entry) = self.entries.get_mut(&group) else { + request.fail(Arc::new(Error::Runtime(RuntimeError::NotReady))); + return; + }; + if entry.fatal || entry.registration.birth.route() != &request.route { + request.fail(Arc::new(Error::Invalid( + "S3 archive registration unavailable/changed", + ))); + return; + } + entry.waiters.push(request); + entry.dirty = true; + if !entry.queued { + entry.queued = true; + self.queue.push_back(group); + } + } + + pub fn running(&self) -> bool { + self.job.is_some() + } + + pub async fn next(&mut self) -> Completed { + self.job.as_mut().expect("active archive job").await + } + + pub fn launch(&mut self, runtime: &CellGroupRuntime, shared: &Shared, now: u64) { + if self.running() { + return; + } + for _ in 0..self.queue.len() { + let group = self.queue.pop_front().expect("archive roster queue"); + let entry = self.entries.get_mut(&group).expect("archive roster entry"); + entry.queued = false; + if !entry.dirty || entry.fatal { + continue; + } + if entry.retry_at > now { + entry.queued = true; + self.queue.push_back(group); + continue; + } + entry.dirty = false; + if let Some(status) = shared + .archives + .lock() + .unwrap_or_else(|error| error.into_inner()) + .get_mut(&group) + { + status.attempts = status.attempts.saturating_add(1); + } + let registration = entry.registration.clone(); + let activation = entry.activation.clone(); + let mut context = entry.context.clone(); + let execution = self.execution.clone(); + let runtime = runtime.clone(); + // Bound receipt SQL work per fair group turn, independently of the + // node's potentially larger total command admission. + let count = entry.waiters.len().min(32); + let mut waiters = entry.waiters.drain(..count).collect(); + tracing::debug!(target: "cellar::raft::group_archive", node = ?runtime.log().limits().node, + group = ?group.group, stage = "launch", waiters = count); + settlement::prune(&mut waiters, now); + self.job = Some(Box::pin(async move { + let result = std::panic::AssertUnwindSafe(async { + if context.is_none() { + context = Some( + job::open( + ®istration, + &runtime, + &execution, + activation.ok_or(Error::Invalid("missing archive activation"))?, + ) + .await?, + ); + } + let coverage = job::drive( + context.as_ref().expect("opened archive context"), + ®istration, + &runtime, + &execution, + ) + .await?; + let local = runtime + .log() + .cell_group(&group) + .map_err(|error| Error::Runtime(error.into()))?; + local + .archive_compaction + .confirm(&local, &coverage) + .map_err(|error| Error::Runtime(error.into()))?; + tracing::debug!(target: "cellar::raft::group_archive", node = ?runtime.log().limits().node, + group = ?group.group, stage = "coverage", index = coverage.marker().position().index.get()); + waiters = settlement::prove( + std::mem::take(&mut waiters), + context.as_ref().expect("opened archive context"), + ®istration, + &coverage, + &execution, + ) + .await; + tracing::debug!(target: "cellar::raft::group_archive", node = ?runtime.log().limits().node, + group = ?group.group, stage = "settled", remaining = waiters.len()); + Ok(coverage.marker().clone()) + }) + .catch_unwind() + .await + .unwrap_or(Err(Error::Invalid("archive job panicked"))); + // Reconciliation may have recovered an older head while all + // newer apply events were already consumed before this job. + // Observe the owner after the attempt so an idle final tail + // cannot disappear when the dirty flag was cleared at launch. + let observation = runtime + .status(group) + .await + .map(|status| status.applied.index); + Completed { + group, + context, + result, + observation, + waiters, + } + })); + break; + } + } + + pub fn complete(&mut self, completed: Completed, shared: &Shared, now: u64, closing: bool) { + self.job = None; + let entry = self + .entries + .get_mut(&completed.group) + .expect("completed archive owner"); + entry.context = completed.context; + entry.waiters.extend(completed.waiters); + settlement::prune(&mut entry.waiters, now); + let mut statuses = shared + .archives + .lock() + .unwrap_or_else(|error| error.into_inner()); + let status = statuses.get_mut(&completed.group).expect("archive status"); + match completed.result { + Ok(marker) => { + let covered = marker.position().index; + status.published = Some(marker); + status.error = None; + match completed.observation { + Ok(applied) => { + let latest = entry + .latest_applied + .map_or(applied, |seen| seen.max(applied)); + entry.dirty = latest > covered; + } + Err(error) => { + let error = Error::Runtime(error); + entry.fatal = !retryable(&error); + status.stopped = entry.fatal; + entry.dirty = true; + entry.retry_at = now.saturating_add(shared.limits.retry_interval_ms); + status.error = Some(Arc::new(error)); + } + } + } + Err(error) => { + entry.fatal = !retryable(&error); + status.stopped = entry.fatal; + entry.dirty = true; + entry.retry_at = now.saturating_add(shared.limits.retry_interval_ms); + status.error = Some(Arc::new(error)); + } + } + if entry.context.is_some() || entry.fatal { + entry.activation.take(); + } + if entry.fatal { + let error = status + .error + .clone() + .unwrap_or_else(|| Arc::new(Error::Stopped)); + for request in entry.waiters.drain(..) { + request.fail(error.clone()); + } + } else if !entry.waiters.is_empty() { + entry.dirty = true; + } + if entry.dirty && !entry.fatal && !entry.queued { + entry.queued = true; + self.queue.push_back(completed.group); + } + if closing || entry.context.is_some() || entry.fatal { + if let Some(reply) = entry.reply.take() { + let result = if closing { + Err(Arc::new(Error::Stopped)) + } else if entry.context.is_some() { + Ok(()) + } else { + Err(status.error.clone().expect("failed archive initialization")) + }; + reply.send(result); + } + } + } + + pub async fn finish(&mut self, shared: &Shared) { + if let Some(job) = self.job.take() { + let completed = job.await; + self.complete(completed, shared, 0, true); + } + self.execution.join().await; + for entry in self.entries.values_mut() { + for request in entry.waiters.drain(..) { + request.fail(Arc::new(Error::Stopped)); + } + entry.registration.store.close(); + if let Some(reply) = entry.reply.take() { + reply.send(Err(Arc::new(Error::Stopped))); + } + } + self.entries.clear(); + for status in shared + .archives + .lock() + .unwrap_or_else(|error| error.into_inner()) + .values_mut() + { + status.stopped = true; + } + } +} + +fn archive_retryable(error: &CellGroupArchiveError) -> bool { + matches!( + error, + CellGroupArchiveError::Busy + | CellGroupArchiveError::Timeout + | CellGroupArchiveError::Conflict + | CellGroupArchiveError::Expired + | CellGroupArchiveError::Unconfirmed + | CellGroupArchiveError::Storage(_) + | CellGroupArchiveError::Checkpoint( + CellGroupCheckpointError::NotReady + | CellGroupCheckpointError::Busy + | CellGroupCheckpointError::Authority( + RuntimeError::NotReady | RuntimeError::Overloaded + ) + ) + ) +} + +fn retryable(error: &Error) -> bool { + match error { + Error::Archive(error) | Error::Binding(CellGroupArchiveBindingError::Archive(error)) => { + archive_retryable(error) + } + Error::Binding(CellGroupArchiveBindingError::Catalog(error)) => matches!( + error, + crate::raft::catalog::CatalogError::Storage(_) + | crate::raft::catalog::CatalogError::Unconfirmed + ), + Error::Runtime(RuntimeError::Overloaded | RuntimeError::NotReady) => true, + _ => false, + } +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/archive/provision.rs b/crates/cellar/durability/raft/cell_groups/controller/archive/provision.rs new file mode 100644 index 000000000..5d590464e --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/archive/provision.rs @@ -0,0 +1,116 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Dynamic groups reuse the node's independent verifier inventory and pools. +//! Preparing an owner opens existing archive identity; it never initializes one. +use super::*; +use crate::raft::cell_groups::controller::provision::CellGroupProvisioning; + +pub(crate) struct Provisioning { + pub namespaces: Vec, + pub io: CellGroupArchivePool, + pub owners: CellGroupArchiveUploadPool, + pub tails: CellGroupArchiveTailPool, + pub checkpoints: CellGroupCheckpointPool, +} + +impl Provisioning { + pub(in crate::raft::cell_groups::controller) fn metadata( + &self, + settings: &CellGroupProvisioning, + ) -> std::result::Result { + let invalid = || CellGroupControllerError::Invalid("archive provisioning resources"); + if self.namespaces.len() != settings.namespaces.len() { + return Err(invalid()); + } + let mut bytes = seed::JOURNAL_WORKSPACE_BYTES; + for (profile, serving) in self.namespaces.iter().zip(&settings.namespaces) { + if !profile.directory.same_owner(&serving.directory) + || std::mem::discriminant(&profile.images) + != std::mem::discriminant(&serving.images) + { + return Err(invalid()); + } + profile + .images + .validate() + .map_err(|e| CellGroupControllerError::Apply(Arc::new(e)))?; + bytes = bytes + .checked_add(profile.directory.cache_capacity_bytes()) + .and_then(|n| n.checked_add(8192)) + .ok_or_else(invalid)?; + } + Ok(bytes) + } + + pub(in crate::raft::cell_groups::controller) async fn prepare( + &self, + directory: &RangeDirectory, + birth: &CellGroupBirth, + settings: &CellGroupProvisioning, + runtime: &CellGroupRuntime, + existing: Option, + ) -> std::result::Result { + let profile = self + .namespaces + .iter() + .find(|p| p.directory.same_owner(directory)) + .ok_or(CellGroupControllerError::Invalid( + "archive provisioning namespace owner", + ))?; + // Repeat identity/counter validation on retries, even when retaining the + // same local store after an uncertain attachment. No absence fallback. + directory + .open_group_archive(&runtime.log().domain(), &self.io, birth) + .await + .map_err(|e| CellGroupControllerError::Archive(Arc::new(Error::Binding(e))))?; + if let Some(existing) = existing { + if existing.birth.route() != birth.route() || !existing.directory.same_owner(directory) + { + return Err(CellGroupControllerError::Invalid( + "retained archive provisioning owner", + )); + } + return Ok(existing); + } + let route = birth.route().clone(); + let path = ManifestPageStore::group_pages_path( + &settings.directory.join("archive-verifier"), + route.history, + ); + let filesystem = settings.filesystem.clone(); + let images = profile.images.clone(); + let (max_pages, max_bytes, roots) = ( + settings.max_pages, + settings.max_page_bytes, + settings.max_retained_roots, + ); + let (manifest, entries, effects) = ( + settings.manifest, + settings.entries, + settings.max_effect_rows, + ); + let store = runtime + .log() + .domain() + .blocking(move || { + let pages = ManifestPageStore::open(&path, filesystem, max_pages, max_bytes, roots) + .map_err(crate::raft::execution::storage)?; + CellGroupArchiveStore::new(route, pages, images, manifest, entries, effects) + }) + .map_err(|_| CellGroupControllerError::Closed)? + .await + .map_err(|_| CellGroupControllerError::Closed)? + .map_err(|e| CellGroupControllerError::Apply(Arc::new(e)))?; + Ok(CellGroupArchiveRegistration { + directory: directory.clone(), + birth: birth.clone(), + io: self.io.clone(), + owners: self.owners.clone(), + tails: self.tails.clone(), + checkpoints: self.checkpoints.clone(), + store, + scratch: runtime.log().snapshot_scratch(), + scratch_directory: settings.directory.join("archive-spools"), + }) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/archive/seed/cleanup.rs b/crates/cellar/durability/raft/cell_groups/controller/archive/seed/cleanup.rs new file mode 100644 index 000000000..0523c721e --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/archive/seed/cleanup.rs @@ -0,0 +1,290 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Idle/restart retirement reads one existing journal at a time. A live source +//! owns the same group lock, so a sweep cannot retire an entered transfer. +use super::*; +use cellar_ltx::{FileSystem, HostDirEntry, HostDirIterator}; + +struct EnteredCleanup(Arc); + +impl Drop for EnteredCleanup { + fn drop(&mut self) { + self.0 + .state + .lock() + .unwrap_or_else(|e| e.into_inner()) + .cleaning = false; + self.0.finished.notify_waiters(); + } +} + +impl Jobs { + pub(in crate::raft::cell_groups::controller) fn maintain( + self: &Arc, + shared: &Arc, + now: u64, + ) { + { + let state = self.state.lock().unwrap_or_else(|e| e.into_inner()); + if state.cleaning || now < state.next_cleanup || self.stopping.load(Ordering::Acquire) { + return; + } + } + let config = match shared.provisioning.lock() { + Ok(slot) => slot.clone(), + Err(_) => { + self.state + .lock() + .unwrap_or_else(|e| e.into_inner()) + .discovery_error = Some(Arc::new(Error::Stopped)); + return; + } + }; + let Some(config) = config.filter(|config| config.archives.is_some()) else { + return; + }; + let entered = { + let mut state = self.state.lock().unwrap_or_else(|e| e.into_inner()); + if state.cleaning || self.stopping.load(Ordering::Acquire) { + return; + } + state.cleaning = true; + state.next_cleanup = now.saturating_add(shared.limits.retry_interval_ms.max(1000)); + EnteredCleanup(self.clone()) + }; + let runtime = shared.runtime.clone(); + let jobs = self.clone(); + let domain = shared.domain.clone(); + let s3 = shared.s3.clone(); + let interval = shared.limits.retry_interval_ms.max(1000); + let result = domain.spawn_cleanup(async move { + let _entered = entered; + let execution = ControllerExecution::new(runtime.log().domain()); + let result = + std::panic::AssertUnwindSafe(sweep(&config, &runtime, &jobs, &s3, &execution)) + .catch_unwind() + .await; + // A cancelled remote operation may still own physical I/O. Join it + // before releasing either the journal sweep or its bank reservation. + execution.join().await; + let error = match result { + Ok(result) => result.err(), + Err(_) => Some(Arc::new(Error::Invalid("seed journal sweep panicked"))), + }; + if let Some(error) = &error { + tracing::warn!(%error, "seed journal discovery retained cleanup debt"); + } + let next = runtime.log().domain().mono_ms(); + let mut state = jobs.state.lock().unwrap_or_else(|e| e.into_inner()); + state.discovery_error = error; + // Rest between complete passes, including after a long outage scan. + state.next_cleanup = match next { + Ok(now) => now.saturating_add(interval), + Err(_) => { + state + .discovery_error + .get_or_insert_with(|| Arc::new(Error::Stopped)); + u64::MAX + } + }; + }); + match result { + Ok(task) => task.detach(), + Err(_) => { + self.state + .lock() + .unwrap_or_else(|e| e.into_inner()) + .discovery_error = Some(Arc::new(Error::Stopped)); + } + } + } +} + +async fn sweep( + config: &Config, + runtime: &CellGroupRuntime, + jobs: &Jobs, + s3: &Arc, + execution: &ControllerExecution, +) -> std::result::Result<(), Arc> { + let mut first_error = None; + let result = async { + let filesystem = execution.run(async { crate::asyncrt::fs() }).await?; + let root = runtime + .log() + .limits() + .directory + .join("cell-group-seed-archives"); + let engine = runtime + .log() + .storage_instance() + .map_err(|e| Error::Runtime(e.into()))? + .bytes(); + let fs = filesystem.clone(); + let path = root.clone(); + let entries = execution + .blocking(move || enumerate(fs.as_ref(), &path))? + .await + .map_err(Error::Worker)??; + let Some(mut entries) = entries else { + return Ok(()); + }; + loop { + if jobs.stopping.load(Ordering::Acquire) { + return Ok(()); + } + let (next, item) = execution + .blocking(move || { + let item = entries.next(); + (entries, item) + })? + .await + .map_err(Error::Worker)?; + entries = next; + let Some(item) = item else { + break; + }; + let fs = filesystem.clone(); + let directory = root.clone(); + let inspected = execution + .blocking(move || inspect(fs.as_ref(), &directory, item?, engine))? + .await + .map_err(Error::Worker)?; + let (path, creation) = match inspected { + Ok(identity) => identity, + Err(error) => { + first_error.get_or_insert_with(|| Arc::new(error)); + continue; + } + }; + let history = creation.route().history; + let group = { + let mut state = jobs.state.lock().map_err(|_| Error::Stopped)?; + jobs.group(&mut state, history)? + }; + let Ok(_group) = group.try_lock_owned() else { + // No waiting behind a live producer, and no age-based expiry. + continue; + }; + // The configuration owns the bounded journal workspace. Take one + // ordinary S3 slot per journal, not for the whole disk inventory. + // A busy bank defers this entry to the next pass without expiry. + let Some(_permit) = s3.acquire(8192) else { + continue; + }; + let result = std::panic::AssertUnwindSafe(reconcile( + config, + filesystem.clone(), + path, + engine, + creation, + execution, + )) + .catch_unwind() + .await; + // Keep the group lock, admission and journal ownership until even + // cancelled/timed-out child I/O is terminal, before another source + // can reserve a new generation for this history. + execution.join().await; + let result = match result { + Ok(result) => result, + Err(_) => Err(Error::Invalid("seed journal retirement panicked")), + }; + if !record_cleanup(jobs, history, result) { + first_error + .get_or_insert_with(|| Arc::new(Error::Invalid("seed cleanup error capacity"))); + } + } + Ok::<_, Error>(()) + } + .await; + if let Err(error) = result { + first_error.get_or_insert_with(|| Arc::new(error)); + } + first_error.map_or(Ok(()), Err) +} + +fn enumerate( + filesystem: &dyn FileSystem, + root: &std::path::Path, +) -> Result> { + let metadata = match filesystem.symlink_metadata(root) { + Ok(metadata) => metadata, + Err(error) if error.kind() == std::io::ErrorKind::NotFound => return Ok(None), + Err(error) => return Err(error.into()), + }; + if !metadata.is_dir { + return Err(Error::Invalid("seed journal root type")); + } + Ok(Some(filesystem.read_dir_iter(root)?)) +} + +fn inspect( + filesystem: &dyn FileSystem, + root: &std::path::Path, + entry: HostDirEntry, + engine: [u8; 16], +) -> Result<(std::path::PathBuf, CellGroupArchiveCreation)> { + let name = entry + .file_name + .to_str() + .ok_or(Error::Invalid("seed journal name encoding"))?; + if !entry.is_dir + || name.len() != 64 + || entry.path != root.join(name) + || !filesystem.symlink_metadata(&entry.path)?.is_dir + { + return Err(Error::Invalid("seed journal entry type/path")); + } + let creation = journal::Journal::inspect_seed(filesystem, &entry.path, engine)?; + if name != journal::seed_name(creation.route().history) { + return Err(Error::Invalid("seed journal history path")); + } + Ok((entry.path, creation)) +} + +async fn reconcile( + config: &Config, + filesystem: Arc, + path: std::path::PathBuf, + engine: [u8; 16], + creation: CellGroupArchiveCreation, + execution: &ControllerExecution, +) -> Result<()> { + let expected = creation.clone(); + let (journal, pending) = execution + .blocking(move || { + let journal = Arc::new(journal::Journal::open_seed_existing( + filesystem, path, engine, expected, + )?); + let record = journal.load()?; + if record.head.is_some() + || record.retire.is_some() + || matches!(record.pending, journal::Pending::Publish { .. }) + { + return Err(Error::Invalid("seed cleanup journal purpose")); + } + let pending = !matches!(record.pending, journal::Pending::None); + Ok::<_, Error>((journal, pending)) + })? + .await + .map_err(Error::Worker)??; + if !pending { + return Ok(()); + } + let archives = config.archives.as_ref().ok_or(Error::Stopped)?; + let history = creation.route().history; + let profile = archives + .namespaces + .iter() + .find(|profile| { + let config = profile.directory.namespace_config(); + config.cluster == history.cluster && config.namespace == history.namespace + }) + .ok_or(Error::Invalid("seed cleanup namespace unavailable"))?; + let registry = profile + .directory + .open_seed_cleanup_archive(execution, &archives.io, &creation) + .await?; + retire(&journal, ®istry, execution).await +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/archive/seed/mod.rs b/crates/cellar/durability/raft/cell_groups/controller/archive/seed/mod.rs new file mode 100644 index 000000000..83732196d --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/archive/seed/mod.rs @@ -0,0 +1,474 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! A fresh quorum seed owns one journaled upload generation through handoff. +//! Transfer data never publishes a recovery head or supplies S3 output proof. +mod cleanup; + +use super::*; +use crate::raft::cell_groups::controller::provision::Config; +use std::sync::{atomic::AtomicBool, Weak}; + +// One bounded sweep may overlap the transport's single admitted source. Charge +// their journal codec workspace once to immutable provisioning configuration. +pub(super) const JOURNAL_WORKSPACE_BYTES: usize = 8 * journal::LIMIT + 16 * 1024; + +#[derive(Clone)] +pub(crate) struct Client(Weak); + +#[derive(Clone, Debug, serde::Serialize, serde::Deserialize)] +#[serde(deny_unknown_fields)] +pub(crate) struct Descriptor { + pub ticket: CellGroupArchiveUploadTicket, + pub checkpoint: CellGroupArchiveReference, +} + +pub(crate) struct Source { + authority: CellGroupSeedAuthority, + snapshot: ::raft::prelude::Snapshot, + descriptor: Descriptor, + lease: Arc, + jobs: Arc, +} + +impl Source { + pub(crate) fn check(&self) -> Result<()> { + if self.jobs.stopping.load(Ordering::Acquire) { + return Err(Error::Stopped); + } + self.authority.check().map_err(Error::Runtime) + } + + pub(crate) fn snapshot(&self) -> &::raft::prelude::Snapshot { + &self.snapshot + } + + pub(crate) fn descriptor(&self) -> &Descriptor { + &self.descriptor + } + + pub(crate) fn barrier(&self) -> LogIndex { + self.authority.barrier().index + } +} + +impl Drop for Source { + fn drop(&mut self) { + self.lease.released.store(true, Ordering::Release); + self.lease.changed.notify_waiters(); + } +} + +#[derive(Default)] +struct Lease { + released: AtomicBool, + changed: Notify, +} + +impl Lease { + async fn wait(&self) { + loop { + let changed = self.changed.notified(); + tokio::pin!(changed); + changed.as_mut().enable(); + if self.released.load(Ordering::Acquire) { + return; + } + changed.await; + } + } +} + +#[derive(Default)] +struct State { + active: usize, + groups: BTreeMap>>, + cleanup_errors: BTreeMap>, + cleaning: bool, + next_cleanup: u64, + discovery_error: Option>, +} + +pub(in crate::raft::cell_groups::controller) struct Jobs { + max_groups: usize, + stopping: AtomicBool, + state: Mutex, + finished: Notify, +} + +struct Entered(Arc); + +impl Drop for Entered { + fn drop(&mut self) { + self.0 + .state + .lock() + .unwrap_or_else(|e| e.into_inner()) + .active -= 1; + self.0.finished.notify_waiters(); + } +} + +impl Jobs { + pub(in crate::raft::cell_groups::controller) fn new(max_groups: usize) -> Self { + Self { + max_groups, + stopping: AtomicBool::new(false), + state: Mutex::new(State::default()), + finished: Notify::new(), + } + } + + fn group( + &self, + state: &mut State, + history: GroupHistory, + ) -> Result>> { + // The lock index bounds concurrent owners, not every history ever seen. + state.groups.retain(|_, group| group.strong_count() != 0); + if let Some(group) = state.groups.get(&history).and_then(Weak::upgrade) { + return Ok(group); + } + if state.groups.len() >= self.max_groups { + return Err(Error::Runtime(RuntimeError::Overloaded)); + } + let group = Arc::new(tokio::sync::Mutex::new(())); + state.groups.insert(history, Arc::downgrade(&group)); + Ok(group) + } + + fn enter( + self: &Arc, + history: GroupHistory, + ) -> Result<(Entered, Arc>)> { + let mut state = self.state.lock().map_err(|_| Error::Stopped)?; + if self.stopping.load(Ordering::Acquire) { + return Err(Error::Stopped); + } + let group = self.group(&mut state, history)?; + state.active += 1; + Ok((Entered(self.clone()), group)) + } + + pub(in crate::raft::cell_groups::controller) fn stop(&self) { + let _gate = self.state.lock().unwrap_or_else(|e| e.into_inner()); + self.stopping.store(true, Ordering::Release); + } + + pub(in crate::raft::cell_groups::controller) async fn join(&self) { + loop { + let finished = self.finished.notified(); + tokio::pin!(finished); + finished.as_mut().enable(); + let done = { + let state = self.state.lock().unwrap_or_else(|e| e.into_inner()); + state.active == 0 && !state.cleaning + }; + if done { + return; + } + finished.await; + } + } + + pub(in crate::raft::cell_groups::controller) fn usage(&self) -> (usize, usize, usize) { + let state = self.state.lock().unwrap_or_else(|e| e.into_inner()); + ( + state.active, + usize::from(state.cleaning), + state.cleanup_errors.len() + usize::from(state.discovery_error.is_some()), + ) + } +} + +impl Client { + pub(in crate::raft::cell_groups::controller) fn new(shared: &Arc) -> Self { + Self(Arc::downgrade(shared)) + } + + /// The private transport caller retains its already-admitted source lane, + /// wire workspace and metadata reservation throughout this owned operation. + pub(crate) async fn prepare( + &self, + authority: CellGroupSeedAuthority, + admitted_transport: Arc, + ) -> std::result::Result> { + let shared = self.0.upgrade().ok_or_else(|| Arc::new(Error::Stopped))?; + let admitted = (|| -> Result<_> { + if shared.closing.load(Ordering::Acquire) + || !authority.belongs_to_engine(&shared.runtime.log().owner()) + { + return Err(Error::Stopped); + } + let history = authority.target().scope; + if !shared.groups.contains(&history).map_err(Error::Runtime)? { + return Err(Error::Runtime(RuntimeError::NotFound)); + } + let config = shared + .provisioning + .lock() + .map_err(|_| Error::Stopped)? + .clone() + .ok_or(Error::Invalid("archive seed provisioning unavailable"))?; + if config.archives.is_none() { + return Err(Error::Invalid("archive seed resources unavailable")); + } + let permit = shared + .s3 + .acquire(8192) + .ok_or(Error::Runtime(RuntimeError::Overloaded))?; + let (entered, group) = shared.archive_seeds.enter(history)?; + Ok((config, permit, entered, group, history)) + })() + .map_err(Arc::new)?; + let (config, permit, entered, group, history) = admitted; + let runtime = shared.runtime.clone(); + let jobs = shared.archive_seeds.clone(); + let domain = shared.domain.clone(); + let (sender, receiver) = oneshot::channel(); + domain + .spawn_cleanup(async move { + let _entered = entered; + let _permit = permit; + let _transport = admitted_transport; + let _group = group.lock_owned().await; + let execution = ControllerExecution::new(runtime.log().domain()); + let outcome = std::panic::AssertUnwindSafe(async { + let result = Box::pin(prepare(&config, &runtime, authority, &execution)).await; + // A cancelled network waiter cannot release an entered producer or + // journal write. Join before reading its final intent for cleanup. + execution.join().await; + match result { + Ok((authority, snapshot, descriptor, journal, registry)) => { + let lease = Arc::new(Lease::default()); + let source = Source { + authority, + snapshot, + descriptor, + lease: lease.clone(), + jobs: jobs.clone(), + }; + let _ = sender.send(Ok(source)); + lease.wait().await; + let cleanup = retire(&journal, ®istry, &execution).await; + record_cleanup(&jobs, history, cleanup); + } + Err(failure) => { + let (error, journal) = *failure; + let _ = sender.send(Err(Arc::new(error))); + if let Some((journal, registry)) = journal { + let cleanup = retire(&journal, ®istry, &execution).await; + record_cleanup(&jobs, history, cleanup); + } + } + } + }) + .catch_unwind() + .await; + execution.join().await; + if outcome.is_err() { + record_cleanup( + &jobs, + history, + Err(Error::Invalid("archive seed worker panicked")), + ); + } + }) + .map_err(|_| Arc::new(Error::Stopped))? + .detach(); + receiver.await.map_err(|_| Arc::new(Error::Stopped))? + } +} + +fn record_cleanup(jobs: &Jobs, history: GroupHistory, result: Result<()>) -> bool { + if let Err(error) = &result { + tracing::warn!(%error, "archive seed cleanup retained its durable intent"); + } + let mut state = jobs.state.lock().unwrap_or_else(|e| e.into_inner()); + if let Err(error) = result { + if !state.cleanup_errors.contains_key(&history) + && state.cleanup_errors.len() >= jobs.max_groups + { + state.discovery_error = Some(Arc::new(Error::Invalid("seed cleanup error capacity"))); + return false; + } + state.cleanup_errors.insert(history, Arc::new(error)); + } else { + state.cleanup_errors.remove(&history); + } + true +} + +type JournalOwner = (Arc, CellGroupArchiveRegistry); +type PrepareFailure = Box<(Error, Option)>; + +fn failed(error: Error, journal: Option) -> PrepareFailure { + Box::new((error, journal)) +} + +type Prepared = ( + CellGroupSeedAuthority, + ::raft::prelude::Snapshot, + Descriptor, + Arc, + CellGroupArchiveRegistry, +); + +async fn prepare( + config: &Config, + runtime: &CellGroupRuntime, + authority: CellGroupSeedAuthority, + execution: &ControllerExecution, +) -> std::result::Result { + let archives = config + .archives + .as_ref() + .ok_or_else(|| failed(Error::Stopped, None))?; + let history = authority.target().scope; + let deadline = authority.deadline(); + let pool = archives.checkpoints.clone(); + // Capture before catalog I/O. The original exact-root selection and all + // existing fresh-quorum/membership checks remain unchanged. + let captured = execution + .blocking(move || { + let source = authority.capture(&pool)?; + let snapshot = source.snapshot()?; + let (authority, checkpoint) = source.into_archive_parts()?; + Ok::<_, CellGroupCheckpointError>((authority, checkpoint, snapshot)) + }) + .map_err(|_| failed(Error::Stopped, None))? + .await + .map_err(|e| failed(Error::Worker(e), None))? + .map_err(|e| failed(Error::Archive(e.into()), None))?; + let (authority, checkpoint, snapshot) = captured; + let profile = archives + .namespaces + .iter() + .find(|p| { + let n = p.directory.namespace_config(); + n.cluster == history.cluster && n.namespace == history.namespace + }) + .ok_or_else(|| failed(Error::Invalid("archive seed namespace"), None))?; + let birth = profile + .directory + .materialize_selected_history(history) + .await + .map_err(|e| failed(Error::Binding(e.into()), None))?; + if birth.route() != authority.marker().route() { + return Err(failed(Error::Invalid("archive seed birth"), None)); + } + let registry = profile + .directory + .open_group_archive(execution, &archives.io, &birth) + .await + .map_err(|e| failed(Error::Binding(e), None))?; + let filesystem = execution + .run(async { crate::asyncrt::fs() }) + .await + .map_err(|_| failed(Error::Stopped, None))?; + let path = runtime + .log() + .limits() + .directory + .join("cell-group-seed-archives") + .join(journal::seed_name(history)); + let creation = registry.creation().clone(); + let engine = runtime + .log() + .storage_instance() + .map_err(|e| failed(Error::Runtime(e.into()), None))? + .bytes(); + let journal = Arc::new( + execution + .blocking(move || journal::Journal::open_seed(filesystem, path, engine, creation)) + .map_err(|_| failed(Error::Stopped, None))? + .await + .map_err(|e| failed(Error::Worker(e), None))? + .map_err(|e| failed(e, None))?, + ); + let owner = (journal.clone(), registry.clone()); + let transfer = async { + retire(&journal, ®istry, execution).await?; + authority.check().map_err(Error::Runtime)?; + let intent = registry.prepare_upload(execution).await?; + let saved = intent.clone(); + let j = journal.clone(); + execution + .blocking(move || { + j.store(&journal::Record { + pending: journal::Pending::Reserve(saved), + ..Default::default() + }) + })? + .await + .map_err(Error::Worker)??; + let ticket = registry.reserve(execution, &intent).await?; + let j = journal.clone(); + execution + .blocking(move || { + j.store(&journal::Record { + pending: journal::Pending::Upload(ticket), + ..Default::default() + }) + })? + .await + .map_err(Error::Worker)??; + let upload = registry + .open_upload(execution, &archives.owners, &ticket, &archives.tails) + .await?; + let reference = upload.upload_checkpoint(execution, checkpoint).await?; + authority.check().map_err(Error::Runtime)?; + upload.require_pending(execution).await?; + Ok::<_, Error>(Descriptor { + ticket, + checkpoint: reference, + }) + }; + let timer = runtime + .log() + .domain() + .sleep_until(deadline) + .map_err(|_| failed(Error::Stopped, Some(owner.clone())))?; + let result = crate::asyncrt::select! { + _ = timer => Err(Error::Runtime(RuntimeError::DeadlineExceeded)), + result = transfer => result, + }; + result + .map(|descriptor| (authority, snapshot, descriptor, journal, registry)) + .map_err(|e| failed(e, Some(owner))) +} + +async fn retire( + journal: &Arc, + registry: &CellGroupArchiveRegistry, + execution: &ControllerExecution, +) -> Result<()> { + let j = journal.clone(); + let record = execution + .blocking(move || j.load())? + .await + .map_err(Error::Worker)??; + if record.head.is_some() || record.retire.is_some() { + return Err(Error::Invalid("archive seed journal purpose")); + } + let ticket = match record.pending { + journal::Pending::None => return Ok(()), + journal::Pending::Upload(ticket) => Some(ticket), + journal::Pending::Reserve(intent) => match registry.reserve(execution, &intent).await { + Ok(ticket) => Some(ticket), + Err(CellGroupArchiveError::Conflict | CellGroupArchiveError::Expired) => None, + Err(error) => return Err(error.into()), + }, + journal::Pending::Publish { .. } => { + return Err(Error::Invalid("archive seed publication intent")) + } + }; + if let Some(ticket) = ticket { + registry.abort(execution, &ticket).await?; + } + let j = journal.clone(); + execution + .blocking(move || j.store(&journal::Record::default()))? + .await + .map_err(Error::Worker)??; + Ok(()) +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/archive/settlement.rs b/crates/cellar/durability/raft/cell_groups/controller/archive/settlement.rs new file mode 100644 index 000000000..47928213a --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/archive/settlement.rs @@ -0,0 +1,564 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Operation proofs share the archive lane; waiting never owns a SQL producer. +use super::*; +use crate::raft::budget::Permit; +use crate::raft::cell_groups::{ + CellDispatchError, CellSqlEnrollment, CellSqlReceipt, CellSqlReceiptLookup, +}; +use crate::raft::execution::{CommandRequest, ExecutionError, ExecutionLimits}; +use cellar_logic::cell_group::GroupRoute; +use std::sync::{atomic::AtomicBool, Mutex, Weak}; + +#[derive(Clone)] +pub(crate) struct Client(Weak); + +pub(crate) struct OutputProof { + _coverage: CellGroupArchiveCoverage, + // The exclusive journal lock survives response buffering and node close. + // A replacement owner cannot retire this hold while a consumer still uses it. + _journal: Arc, + _permit: Arc>, + authority: Arc, +} + +impl OutputProof { + pub(crate) async fn require_current( + &self, + domain: &crate::asyncrt::DomainToken, + deadline: u64, + ) -> std::result::Result<(), CellDispatchError> { + let now = domain + .mono_ms() + .map_err(|_| CellDispatchError::Runtime(RuntimeError::Closed))?; + if now >= deadline { + return Err(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)); + } + let timer = domain + .sleep_until(deadline) + .map_err(|_| CellDispatchError::Runtime(RuntimeError::Closed))?; + let result = crate::asyncrt::select! { + result = domain.run(self.authority.require_current()) => result.map_err(|_| CellDispatchError::Runtime(RuntimeError::Closed))?, + _ = timer => return Err(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)), + }; + if domain + .mono_ms() + .map_err(|_| CellDispatchError::Runtime(RuntimeError::Closed))? + >= deadline + { + return Err(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)); + } + result.map_err(|error| CellDispatchError::Archive(Arc::new(error))) + } +} + +pub(crate) struct CommandSettled { + pub receipt: Box, + pub proof: Box, +} + +pub(crate) struct EnrollmentSettled { + pub enrollment: Box, + pub proof: Box, +} + +pub(crate) struct LookupSettled { + pub selected: Box, + pub proof: Box, +} + +struct Lookup { + selected: Box, + request: CommandRequest, +} + +/// Admitted immutable observation; no node work is queued until response. +pub(crate) struct LookupPrepared(Prepared); + +impl LookupPrepared { + pub(crate) async fn response(self) -> std::result::Result { + let settled = self.0.response().await?; + let Output::Lookup(lookup) = settled.output else { + return Err(CellDispatchError::Invalid("S3 lookup output")); + }; + Ok(LookupSettled { + selected: lookup.selected, + proof: settled.proof, + }) + } +} + +enum Output { + Command(Box), + Enrollment(Box), + Lookup(Box), +} + +impl Output { + fn retained_bytes(&self) -> std::result::Result { + match self { + Self::Command(receipt) => receipt.receipt().receipt.result().len(), + // Creation records are bounded by the protocol to 4096 bytes. Their + // variable identity bytes and the archive comparison stay admitted. + Self::Enrollment(_) => 4096, + Self::Lookup(lookup) => lookup + .selected + .receipt() + .map_or(0, |row| row.result().len()), + } + .checked_mul(2) + .and_then(|n| n.checked_add(8192)) + .ok_or(CellDispatchError::Runtime(RuntimeError::Overloaded)) + } + + fn retain(self, permit: Arc>) -> Self { + match self { + Self::Command(receipt) => Self::Command(receipt.retain_for_archive(permit)), + Self::Enrollment(enrollment) => Self::Enrollment(enrollment.retain_for_archive(permit)), + Self::Lookup(lookup) => Self::Lookup(Box::new(Lookup { + selected: lookup.selected.retain_for_archive(permit), + request: lookup.request, + })), + } + } +} + +struct Settled { + output: Output, + proof: Box, +} + +pub(in crate::raft::cell_groups::controller) struct Request { + pub route: GroupRoute, + output: Output, + limits: Option, + deadline: u64, + cancelled: Arc, + reply: oneshot::Sender>, + permit: Arc>, +} + +struct Cancel(Arc); + +impl Drop for Cancel { + fn drop(&mut self) { + self.0.store(true, Ordering::Release); + } +} + +impl CellGroupController { + pub(crate) fn s3_client( + &self, + domain: &crate::asyncrt::DomainToken, + ) -> std::result::Result { + if !self.handle.0.domain.same_owner(domain) { + return Err(RuntimeError::Invalid("S3 controller domain".into())); + } + Ok(Client(Arc::downgrade(&self.handle.0))) + } +} + +impl Client { + pub(crate) fn require_runtime( + &self, + runtime: &CellGroupRuntime, + ) -> std::result::Result<(), CellDispatchError> { + let shared = self + .0 + .upgrade() + .ok_or(CellDispatchError::Runtime(RuntimeError::Closed))?; + if !shared.runtime.same_owner(runtime) { + return Err(CellDispatchError::Runtime(RuntimeError::Invalid( + "S3 runtime owner".into(), + ))); + } + Ok(()) + } + + pub(crate) fn require_group( + &self, + route: &GroupRoute, + ) -> std::result::Result<(), CellDispatchError> { + let shared = self + .0 + .upgrade() + .ok_or(CellDispatchError::Runtime(RuntimeError::Closed))?; + if shared.closing.load(Ordering::Acquire) { + return Err(CellDispatchError::Runtime(RuntimeError::Closed)); + } + let states = shared + .archives + .lock() + .map_err(|_| CellDispatchError::Runtime(RuntimeError::Closed))?; + let state = states + .get(&route.history) + .ok_or(CellDispatchError::Runtime(RuntimeError::NotReady))?; + if state.stopped { + return Err(state.error.clone().map_or( + CellDispatchError::Runtime(RuntimeError::Closed), + CellDispatchError::Archive, + )); + } + Ok(()) + } + + pub(crate) async fn command( + &self, + runtime: &CellGroupRuntime, + route: GroupRoute, + receipt: Box, + limits: ExecutionLimits, + deadline: u64, + ) -> std::result::Result { + let settled = self + .settle( + runtime, + route, + Output::Command(receipt), + Some(limits), + deadline, + ) + .await?; + let Output::Command(receipt) = settled.output else { + return Err(CellDispatchError::Invalid("S3 command output")); + }; + Ok(CommandSettled { + receipt, + proof: settled.proof, + }) + } + + pub(crate) async fn enrollment( + &self, + runtime: &CellGroupRuntime, + route: GroupRoute, + enrollment: Box, + deadline: u64, + ) -> std::result::Result { + let settled = self + .settle( + runtime, + route, + Output::Enrollment(enrollment), + None, + deadline, + ) + .await?; + let Output::Enrollment(enrollment) = settled.output else { + return Err(CellDispatchError::Invalid("S3 enrollment output")); + }; + Ok(EnrollmentSettled { + enrollment, + proof: settled.proof, + }) + } + + pub(crate) fn prepare_lookup( + &self, + runtime: &CellGroupRuntime, + route: GroupRoute, + selected: Box, + request: CommandRequest, + limits: ExecutionLimits, + deadline: u64, + ) -> std::result::Result { + self.prepare( + runtime, + route, + Output::Lookup(Box::new(Lookup { selected, request })), + Some(limits), + deadline, + ) + .map(LookupPrepared) + } + + async fn settle( + &self, + runtime: &CellGroupRuntime, + route: GroupRoute, + output: Output, + limits: Option, + deadline: u64, + ) -> std::result::Result { + self.prepare(runtime, route, output, limits, deadline)? + .response() + .await + } + + fn prepare( + &self, + runtime: &CellGroupRuntime, + route: GroupRoute, + output: Output, + limits: Option, + deadline: u64, + ) -> std::result::Result { + self.require_runtime(runtime)?; + self.require_group(&route)?; + let shared = self + .0 + .upgrade() + .ok_or(CellDispatchError::Runtime(RuntimeError::Closed))?; + let permit = shared + .s3 + .acquire(output.retained_bytes()?) + .map(|permit| Arc::new(Mutex::new(permit))) + .ok_or(CellDispatchError::Runtime(RuntimeError::Overloaded))?; + let output = output.retain(permit.clone()); + let (reply, receiver) = oneshot::channel(); + let cancelled = Arc::new(AtomicBool::new(false)); + let now = shared + .domain + .mono_ms() + .map_err(|_| CellDispatchError::Runtime(RuntimeError::Closed))?; + if now >= deadline { + return Err(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)); + } + Ok(Prepared { + client: self.clone(), + request: Request { + route, + output, + limits, + deadline, + cancelled: cancelled.clone(), + reply, + permit, + }, + receiver, + _cancel: Cancel(cancelled), + }) + } +} + +struct Prepared { + client: Client, + request: Request, + receiver: oneshot::Receiver>, + _cancel: Cancel, +} + +impl Prepared { + async fn response(self) -> std::result::Result { + let Self { + client, + request, + receiver, + _cancel, + } = self; + let shared = client + .0 + .upgrade() + .ok_or(CellDispatchError::Runtime(RuntimeError::Closed))?; + let deadline = request.deadline; + if shared.closing.load(Ordering::Acquire) { + return Err(CellDispatchError::Runtime(RuntimeError::Closed)); + } + let now = shared + .domain + .mono_ms() + .map_err(|_| CellDispatchError::Runtime(RuntimeError::Closed))?; + if now >= deadline { + return Err(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)); + } + shared + .requests + .try_send(requests::Request::S3(Box::new(request))) + .map_err(|e| { + CellDispatchError::Runtime(match e { + tokio::sync::mpsc::error::TrySendError::Closed(_) => RuntimeError::Closed, + tokio::sync::mpsc::error::TrySendError::Full(_) => RuntimeError::Overloaded, + }) + })?; + let timer = shared + .domain + .sleep_until(deadline) + .map_err(|_| CellDispatchError::Runtime(RuntimeError::Closed))?; + let result = crate::asyncrt::select! { + result = receiver => result.map_err(|_| CellDispatchError::Runtime(RuntimeError::Closed))?, + _ = timer => return Err(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)), + }; + if shared.closing.load(Ordering::Acquire) { + return Err(CellDispatchError::Runtime(RuntimeError::Closed)); + } + if shared + .domain + .mono_ms() + .map_err(|_| CellDispatchError::Runtime(RuntimeError::Closed))? + >= deadline + { + return Err(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)); + } + result + } +} + +impl Request { + pub(super) fn fail(self, error: Arc) { + let _ = self.reply.send(Err(CellDispatchError::Archive(error))); + } + + fn expired(&self, now: u64) -> bool { + self.cancelled.load(Ordering::Acquire) || self.reply.is_closed() || now >= self.deadline + } +} + +pub(super) fn prune(requests: &mut Vec, now: u64) { + let mut live = Vec::new(); + for request in requests.drain(..) { + if now >= request.deadline { + let _ = request.reply.send(Err(CellDispatchError::Runtime( + RuntimeError::DeadlineExceeded, + ))); + } else if !request.expired(now) { + live.push(request); + } + } + *requests = live; +} + +pub(super) async fn prove( + requests: Vec, + context: &job::Context, + registration: &CellGroupArchiveRegistration, + coverage: &CellGroupArchiveCoverage, + execution: &ControllerExecution, +) -> Vec { + let mut waiting = Vec::new(); + { + let mut uses = context.uses.lock().unwrap_or_else(|e| e.into_inner()); + uses.retain(|usage| usage.live()); + uses.push(coverage.retain_use()); + } + for request in requests { + let now = execution + .run(async { crate::asyncrt::mono_ms() }) + .await + .unwrap_or(u64::MAX); + if request.expired(now) { + continue; + } + if request.route != *coverage.marker().route() { + request.fail(Arc::new(Error::Invalid("S3 output route"))); + continue; + } + let store = registration.store.clone(); + let physical = coverage.clone(); + let read = execution.blocking(move || { + let mut request = request; + let result = + std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { + match &mut request.output { + Output::Command(receipt) => { + crate::raft::cell_groups::archive::receipt::command( + &physical, + &store, + receipt.receipt(), + request + .limits + .ok_or(CellGroupArchiveError::Invalid("S3 command limits"))?, + request.deadline, + request.cancelled.clone(), + ) + } + Output::Enrollment(enrollment) => { + crate::raft::cell_groups::archive::enrollment::verify( + &physical, + &store, + enrollment.record(), + ) + } + Output::Lookup(lookup) => { + let original = lookup + .selected + .receipt() + .map_or(0, |row| row.result().len()); + crate::raft::cell_groups::archive::receipt::lookup( + &physical, + &store, + &mut lookup.selected, + crate::raft::cell_groups::archive::receipt::LookupRead { + request: &lookup.request, + limits: request.limits.ok_or( + CellGroupArchiveError::Invalid("S3 lookup limits"), + )?, + deadline: request.deadline, + cancelled: request.cancelled.clone(), + }, + |length| { + let bytes = length + .max(original) + .checked_mul(2) + .and_then(|n| n.checked_add(8192)) + .ok_or(ExecutionError::Refused( + cellar_logic::consensus::CommandFailure::Overloaded, + ))?; + let mut permit = request.permit.lock().map_err(|_| { + ExecutionError::Invalid("S3 result admission") + })?; + if !permit.grow_at_least(bytes) { + return Err(ExecutionError::Refused( + cellar_logic::consensus::CommandFailure::Overloaded, + )); + } + Ok(()) + }, + ) + } + } + })) + .unwrap_or_else(|panic| { + Err(CellGroupArchiveError::Replay( + crate::raft::execution::storage(crate::native_worker::panic_error(panic)), + )) + }); + (request, result) + }); + // A refused submission drops the closure and its owned request. Entered + // closures retain both until the controller-specific physical join. + let Ok(read) = read else { + continue; + }; + let Ok((request, result)) = read.await else { + continue; + }; + let now = execution + .run(async { crate::asyncrt::mono_ms() }) + .await + .unwrap_or(u64::MAX); + if request.expired(now) { + continue; + } + match result { + Ok(false) | Err(CellGroupArchiveError::Busy) => waiting.push(request), + Err(CellGroupArchiveError::Replay(error)) + if matches!(&request.output, Output::Lookup(_)) => + { + let _ = request.reply.send(Err(CellDispatchError::Execution(error))); + } + Err(error) => request.fail(Arc::new(Error::Archive(error))), + Ok(true) => { + if let Err(error) = context.authority.require_current().await { + request.fail(Arc::new(error)); + continue; + } + let Request { + output, + reply, + permit, + .. + } = request; + let _ = reply.send(Ok(Settled { + output, + proof: Box::new(OutputProof { + _coverage: coverage.clone(), + _journal: context.journal.clone(), + _permit: permit, + authority: context.authority.clone(), + }), + })); + } + } + } + waiting +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/calls.rs b/crates/cellar/durability/raft/cell_groups/controller/calls.rs new file mode 100644 index 000000000..84ebf54ea --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/calls.rs @@ -0,0 +1,49 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Response deadlines do not own the lifetime of admitted mutations. +use super::*; +use crate::raft::budget::Permit; +use tokio::sync::oneshot; + +pub(super) struct Delivery { + pub result: Result>, + pub _permit: Arc

, +} + +pub(super) struct Call { + pub receiver: oneshot::Receiver>, + pub identity: I, + pub deadline: u64, + pub domain: crate::asyncrt::DomainToken, + pub failure: fn(I, RuntimeError) -> E, +} + +impl Call { + pub async fn response(self) -> Result> { + let failure = |reason| Arc::new((self.failure)(self.identity, reason)); + let now = self + .domain + .mono_ms() + .map_err(|_| failure(RuntimeError::Closed))?; + if now >= self.deadline { + return Err(failure(RuntimeError::DeadlineExceeded)); + } + let timer = self + .domain + .sleep_until(self.deadline) + .map_err(|_| failure(RuntimeError::Closed))?; + let delivery = crate::asyncrt::select! { + result = self.receiver => result.map_err(|_| failure(RuntimeError::Closed))?, + _ = timer => return Err(failure(RuntimeError::DeadlineExceeded)), + }; + if self + .domain + .mono_ms() + .map_err(|_| failure(RuntimeError::Closed))? + >= self.deadline + { + return Err(failure(RuntimeError::DeadlineExceeded)); + } + delivery.result + } +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/checkpoints.rs b/crates/cellar/durability/raft/cell_groups/controller/checkpoints.rs new file mode 100644 index 000000000..45e4b28b0 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/checkpoints.rs @@ -0,0 +1,260 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Checkpoint the preceding applied cut while retaining its successor ticket. +//! One bounded slot spans capture, reserved admission and durable publication. +use super::*; +use crate::raft::cell_groups::{ + CellGroupApply, CellGroupCheckpointError, CellGroupCheckpointPool, + PreparedLocalCellGroupCheckpoint, +}; +use cellar_logic::cell_group::GroupHistory; +use futures_util::{future::BoxFuture, stream::FuturesUnordered, StreamExt}; + +#[derive(Clone, Copy, Debug)] +pub struct CellGroupCheckpointPolicy { + /// Entries since the persisted checkpoint (birth index before the first). + /// Selection occurs before a successor apply; an idle final tail is retained. + pub interval_entries: u64, + /// Node-wide captures, admissions and pending syncs combined. + pub max_parallel: usize, +} + +pub(super) struct Config { + policy: CellGroupCheckpointPolicy, + pool: CellGroupCheckpointPool, +} + +impl Config { + pub fn new( + limits: CellGroupControllerLimits, + transport: &RaftTransport, + ) -> Result, CellGroupControllerError> { + let Some(policy) = limits.checkpoints else { + return Ok(None); + }; + if policy.interval_entries == 0 + || policy.max_parallel == 0 + || policy.max_parallel > limits.max_groups.min(256) + { + return Err(CellGroupControllerError::Invalid("checkpoint policy")); + } + Ok(Some(Self { + policy, + pool: transport + .group_checkpoint_pool() + .map_err(CellGroupControllerError::Transport)?, + })) + } +} + +struct Entry { + work: Box, + admitted: bool, + published: bool, +} + +type Captured = ( + GroupHistory, + Result, +); +type Admitted = (GroupHistory, Result<(), RuntimeError>); + +pub(super) enum Completed { + Captured(Captured), + Admitted(Admitted), +} + +#[derive(Default)] +pub(super) struct Manager { + entries: BTreeMap, + retry_at: BTreeMap, + captures: FuturesUnordered>, + publications: FuturesUnordered>, +} + +impl Manager { + pub fn contains(&self, group: &GroupHistory) -> bool { + self.entries.contains_key(group) + } + + pub fn running(&self) -> bool { + !self.captures.is_empty() || !self.publications.is_empty() + } + + pub async fn next(&mut self) -> Completed { + crate::asyncrt::select! { + Some(done) = self.captures.next(), if !self.captures.is_empty() => Completed::Captured(done), + Some(done) = self.publications.next(), if !self.publications.is_empty() => Completed::Admitted(done), + } + } + + pub fn start( + &mut self, + work: &CellGroupApply, + now: u64, + shared: &Shared, + ) -> Result { + let Some(config) = &shared.checkpoints else { + return Ok(false); + }; + let history = work.base.marker().route().history; + if self.entries.len() >= config.policy.max_parallel + || self.retry_at.get(&history).is_some_and(|at| *at > now) + { + return Ok(false); + } + let group = shared + .runtime + .log() + .cell_group(&history) + .map_err(RuntimeError::from)?; + let previous = group + .checkpoint_manifest() + .map_err(RuntimeError::from)? + .map_or(1, |root| root.position().index.get()); + // An archive outage must not turn a due checkpoint into an apply stall. + // Recheck the exact captured cut again at owner and storage admission. + if !group + .archive_compaction + .permits(work.base.marker()) + .map_err(RuntimeError::from)? + { + return Ok(false); + } + let index = work.base.marker().position().index.get(); + if index.saturating_sub(previous) < config.policy.interval_entries { + return Ok(false); + } + let pool = config.pool.clone(); + let root = work.base.clone(); + let task = shared + .domain + .blocking(move || pool.capture(&group, &root)?.prepare_publication()) + .map_err(|_| CellGroupControllerError::Closed)?; + self.entries.insert( + history, + Entry { + work: Box::new(work.clone()), + admitted: false, + published: false, + }, + ); + self.captures.push(Box::pin(async move { + ( + history, + task.await + .map_err(|_| CellGroupControllerError::Closed) + .and_then(|result| { + result + .map_err(|error| CellGroupControllerError::Checkpoint(Arc::new(error))) + }), + ) + })); + shared + .active_checkpoints + .store(self.entries.len(), Ordering::Release); + Ok(true) + } + + pub fn publish( + &mut self, + group: GroupHistory, + checkpoint: PreparedLocalCellGroupCheckpoint, + runtime: &CellGroupRuntime, + ) -> Result<(), CellGroupControllerError> { + let entry = self + .entries + .get(&group) + .ok_or(CellGroupControllerError::Invalid( + "checkpoint capture ticket", + ))?; + let ticket = entry.work.ticket.clone(); + let runtime = runtime.clone(); + self.publications.push(Box::pin(async move { + ( + group, + runtime + .checkpoint_before_apply(group, ticket, checkpoint) + .await, + ) + })); + Ok(()) + } + + pub fn observed( + &mut self, + group: GroupHistory, + index: u64, + shared: &Shared, + ) -> Result>, CellGroupControllerError> { + let Some(entry) = self.entries.get_mut(&group) else { + return Ok(None); + }; + if entry.work.base.marker().position().index.get() != index { + return Err(CellGroupControllerError::Invalid( + "checkpoint publication cut", + )); + } + entry.published = true; + Ok(if entry.admitted { + self.remove(group, shared) + } else { + None + }) + } + + pub fn accept( + &mut self, + group: GroupHistory, + shared: &Shared, + ) -> Result>, CellGroupControllerError> { + let entry = self + .entries + .get_mut(&group) + .ok_or(CellGroupControllerError::Invalid( + "checkpoint admission ticket", + ))?; + entry.admitted = true; + Ok(if entry.published { + self.remove(group, shared) + } else { + None + }) + } + + pub fn retry( + &mut self, + group: GroupHistory, + now: u64, + shared: &Shared, + ) -> Option> { + self.retry_at + .insert(group, now.saturating_add(shared.limits.retry_interval_ms)); + shared.checkpoint_retries.fetch_add(1, Ordering::Relaxed); + self.remove(group, shared) + } + + pub fn remove(&mut self, group: GroupHistory, shared: &Shared) -> Option> { + let work = self.entries.remove(&group).map(|entry| entry.work); + shared + .active_checkpoints + .store(self.entries.len(), Ordering::Release); + work + } + + /// A cancelled controller waiter cannot detach a physical capture. Join it + /// before closing the runtime's engine claim. Admitted writes belong to the + /// runtime and are joined by runtime.close(). Retain tickets until both joins. + pub async fn join(&mut self) { + while self.captures.next().await.is_some() {} + self.publications.clear(); + } +} + +pub(super) fn retryable(error: &CellGroupControllerError) -> bool { + matches!(error, CellGroupControllerError::Checkpoint(error) if matches!(error.as_ref(), CellGroupCheckpointError::Busy | CellGroupCheckpointError::NotReady)) + || matches!( + error, + CellGroupControllerError::Runtime(RuntimeError::Overloaded) + ) +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/commands.rs b/crates/cellar/durability/raft/cell_groups/controller/commands.rs new file mode 100644 index 000000000..5dc833379 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/commands.rs @@ -0,0 +1,294 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Bounded invocation ownership is independent of response-waiter lifetime. +use super::admission::Retention; +use super::calls::Delivery; +use super::*; +use crate::raft::budget::Budget; +use crate::raft::cell_groups::{ + CellCommandError, CellCommandInvocation, CellCommandRequest, CellCommandResponse, + CellDispatchError, CellGroupDispatcher, CellRoutingLimits, CellSqlOptions, PendingCellCommand, +}; +use cellar_logic::consensus::CommandId; +use tokio::sync::{mpsc, oneshot, watch}; + +enum Invocation { + Ready(Box), + Pending(Box), +} + +pub(super) struct Request { + invocation: Invocation, + options: CellSqlOptions, + reply: oneshot::Sender>, + permit: Arc, +} + +pub(super) struct Completed { + pub invocation: Option, + pub error: Option>, + _permit: Arc, +} + +/// Dropping or timing out this response never cancels the admitted command. +/// An unpolled reply retains node admission, including error-only replies. +pub struct CellCommandCall { + inner: calls::Call, +} + +/// Admission did not transfer ownership. In particular, a rejected non-fresh +/// invocation is returned with its original recovery state intact. +pub struct CellCommandAdmissionError { + pub error: CellCommandError, + pub invocation: Box, +} + +impl std::fmt::Debug for CellCommandAdmissionError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("CellCommandAdmissionError") + .field("error", &self.error) + .finish_non_exhaustive() + } +} + +impl std::fmt::Display for CellCommandAdmissionError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + std::fmt::Display::fmt(&self.error, f) + } +} + +impl std::error::Error for CellCommandAdmissionError { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + Some(&self.error) + } +} + +fn error(command: CommandId, ambiguous: bool, reason: RuntimeError) -> CellCommandError { + CellCommandError { + command, + ambiguous, + cause: Box::new(CellDispatchError::Runtime(reason)), + } +} + +impl CellCommandCall { + pub async fn response(self) -> Result> { + self.inner.response().await + } +} + +pub(super) fn option_bytes(options: &mut CellSqlOptions) -> Result { + let (path, vfs) = match options { + CellSqlOptions::Materialized(options) => (&mut options.directory, &mut options.sqlite_vfs), + CellSqlOptions::Extents(options) => (&mut options.cache_directory, &mut options.sqlite_vfs), + }; + if path.as_os_str().len() > 4096 || vfs.as_ref().is_some_and(|vfs| vfs.len() > 128) { + return Err(RuntimeError::Invalid("command SQL option bounds".into())); + } + *path = std::mem::take(path).into_boxed_path().into_path_buf(); + *vfs = vfs.take().map(|vfs| vfs.into_boxed_str().into_string()); + Ok(2 * (path.as_os_str().len() + vfs.as_ref().map_or(0, String::len))) +} + +impl CellGroupController { + /// Admit a fresh local invocation. Discovery and destination authority still + /// run inside the owner; catalog routing never grants leader authority. + pub fn command( + &self, + invocation: CellCommandInvocation, + options: CellSqlOptions, + ) -> Result { + admit(&self.handle.0, invocation, options) + } + + /// Capture and queue a registered cell command before returning its response + /// waiter. Construction, routing, SQL and publication remain node-owned even + /// if that waiter is never polled or is dropped. The timeout starts here. + pub fn dispatch_command( + &self, + dispatcher: &CellGroupDispatcher, + cell: cellar_logic::cell_group::CellHistory, + request: CellCommandRequest, + routing: CellRoutingLimits, + mut options: CellSqlOptions, + ) -> Result { + let shared = &self.handle.0; + let command = request.id; + let mut registration = shared + .registration() + .map_err(|reason| error(command, false, reason))?; + if shared.closing.load(Ordering::Acquire) { + return Err(error(command, false, RuntimeError::Closed)); + } + let option_bytes = option_bytes(&mut options).map_err(|e| error(command, false, e))?; + let pending = dispatcher.prepare_command(cell, request, routing, None, None, false)?; + let (_, deadline, bytes) = pending.controller_admission(&shared.domain)?; + let bytes = bytes + .checked_add(option_bytes) + .ok_or_else(|| error(command, false, RuntimeError::Overloaded))?; + let permit = shared + .commands + .acquire(bytes) + .ok_or_else(|| error(command, false, RuntimeError::Overloaded))?; + let (reply, receiver) = oneshot::channel(); + shared + .requests + .try_send(requests::Request::Command(Box::new(Request { + invocation: Invocation::Pending(Box::new(pending)), + options, + reply, + permit: Arc::new(Retention { + _permit: permit, + _work: registration.work.take(), + }), + }))) + .map_err(|e| { + error( + command, + false, + match e { + mpsc::error::TrySendError::Closed(_) => RuntimeError::Closed, + mpsc::error::TrySendError::Full(_) => RuntimeError::Overloaded, + }, + ) + })?; + Ok(CellCommandCall { + inner: calls::Call { + receiver, + identity: command, + deadline, + domain: shared.domain.clone(), + failure: |id, reason| error(id, true, reason), + }, + }) + } +} + +pub(super) fn admit( + shared: &Arc, + invocation: CellCommandInvocation, + mut options: CellSqlOptions, +) -> Result { + let admitted = (|| { + let (command, deadline, bytes) = invocation.controller_admission(&shared.domain)?; + let registration = shared + .registration() + .map_err(|reason| error(command, false, reason))?; + if shared.closing.load(Ordering::Acquire) { + return Err(error(command, false, RuntimeError::Closed)); + } + let bytes = bytes + .checked_add(option_bytes(&mut options).map_err(|e| error(command, false, e))?) + .ok_or_else(|| error(command, false, RuntimeError::Overloaded))?; + let permit = shared + .commands + .acquire(bytes) + .ok_or_else(|| error(command, false, RuntimeError::Overloaded))?; + Ok((command, deadline, permit, registration)) + })(); + let (command, deadline, permit, mut registration) = match admitted { + Ok(admitted) => admitted, + Err(error) => { + return Err(CellCommandAdmissionError { + error, + invocation: Box::new(invocation), + }) + } + }; + let (reply, receiver) = oneshot::channel(); + shared + .requests + .try_send(requests::Request::Command(Box::new(Request { + invocation: Invocation::Ready(Box::new(invocation)), + options, + reply, + permit: Arc::new(Retention { + _permit: permit, + _work: registration.work.take(), + }), + }))) + .map_err(|e| { + let (request, reason) = match e { + mpsc::error::TrySendError::Closed(request) => (request, RuntimeError::Closed), + mpsc::error::TrySendError::Full(request) => (request, RuntimeError::Overloaded), + }; + CellCommandAdmissionError { + error: error(command, false, reason), + invocation: match request { + requests::Request::Command(request) => match request.invocation { + Invocation::Ready(invocation) => invocation, + Invocation::Pending(_) => unreachable!("ready command admission envelope"), + }, + _ => unreachable!("command admission envelope"), + }, + } + })?; + Ok(CellCommandCall { + inner: calls::Call { + receiver, + identity: command, + deadline, + domain: shared.domain.clone(), + failure: |id, reason| error(id, true, reason), + }, + }) +} + +pub(super) async fn drive( + request: Request, + runtime: CellGroupRuntime, + sql: CellGroupSqlPool, + shared: Arc, + mut progress: watch::Receiver<()>, + transport: RaftTransport, +) -> Completed { + let Request { + invocation, + options, + reply, + permit, + } = request; + let (mut invocation, transport) = match invocation { + Invocation::Ready(invocation) => (*invocation, None), + Invocation::Pending(pending) => match (*pending).finish().await { + Ok(invocation) => (invocation, Some(transport)), + Err(error) => { + let error = Arc::new(error); + let _ = reply.send(Delivery { + result: Err(error.clone()), + _permit: permit.clone(), + }); + return Completed { + invocation: None, + error: Some(error), + _permit: permit, + }; + } + }, + }; + let result = invocation + .run_owned( + &runtime, + &sql, + options, + &shared.groups, + &mut progress, + transport.as_ref(), + ) + .await + .map_err(Arc::new); + let error = result.as_ref().err().cloned(); + let _ = reply.send(Delivery { + result, + _permit: permit.clone(), + }); + Completed { + invocation: Some(invocation), + error, + _permit: permit, + } +} + +pub(super) fn budget(limits: CellGroupControllerLimits) -> Arc { + Budget::new(limits.max_command_bytes, limits.max_commands) +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/effects/mod.rs b/crates/cellar/durability/raft/cell_groups/controller/effects/mod.rs new file mode 100644 index 000000000..c437b7191 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/effects/mod.rs @@ -0,0 +1,372 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! One bounded node turn discovers committed work through the manifest index. +//! Cursors are hints per physical group; no cell owns a task, timer or cache. + +mod turn; + +use super::*; +use crate::application::consensus::RaftClass; +use crate::raft::budget::Budget; +use crate::raft::cell_groups::{CellGroupDispatcher, CellSqlOptions}; +use crate::raft::effects::RaftEffectOptions; +use cellar_logic::{ + cell_group::{CellHistory, CellId, GroupHistory}, + consensus::{ClusterId, LogIndex, NamespaceId}, +}; +use futures_util::{future::BoxFuture, stream::FuturesUnordered, FutureExt, StreamExt}; +use std::collections::BTreeSet; +use std::ops::Bound::{Excluded, Included, Unbounded}; + +type SqlFactory = Arc Result + Send + Sync>; + +/// A compiled namespace's effect adapter and explicit execution resources. +/// The factory runs on the owning execution domain's blocking service. It must +/// derive private paths from the complete cell history and must not spawn work. +/// Its captured configuration must be bounded independently of cell count. +pub struct CellGroupEffectRegistration { + dispatcher: CellGroupDispatcher, + options: RaftEffectOptions, + sql: SqlFactory, +} + +impl CellGroupEffectRegistration { + pub fn new( + dispatcher: CellGroupDispatcher, + options: RaftEffectOptions, + sql: impl Fn(&CellHistory) -> Result + Send + Sync + 'static, + ) -> Result { + if !(10..=60_000).contains(&options.interval_ms) + || !(1..=60_000).contains(&options.operation_timeout_ms) + || options + .operation_timeout_ms + .checked_add(100) + .is_none_or(|n| n > options.retry_window_ms) + || options.retry_window_ms > 600_000 + || !(64 * 1024..=8 * 1024 * 1024).contains(&options.metadata_bytes) + { + return Err(CellGroupControllerError::Invalid("effect scheduler bounds")); + } + Ok(Self { + dispatcher, + options, + sql: Arc::new(sql), + }) + } +} + +struct Registration { + dispatcher: CellGroupDispatcher, + class: Arc, + schema: [u8; 32], + options: RaftEffectOptions, + memory: Arc, + sql: SqlFactory, +} + +pub(super) struct Config { + namespaces: BTreeMap<(ClusterId, NamespaceId), Arc>, +} + +impl CellGroupController { + pub(crate) fn require_host_effects( + &self, + dispatcher: &CellGroupDispatcher, + ) -> Result<(), crate::raft::cell_groups::CellDispatchError> { + use crate::raft::cell_groups::CellDispatchError; + let (_, class, _) = dispatcher.host_binding()?; + if !class.has_system_commands() { + return Ok(()); + } + let effects = self + .handle + .0 + .effects + .lock() + .map_err(|_| CellDispatchError::Runtime(RuntimeError::Closed))?; + if effects + .as_ref() + .and_then(|config| config.namespaces.get(&dispatcher.namespace())) + .is_none_or(|entry| entry.schema != class.schema_digest()) + { + return Err(CellDispatchError::Execution(ExecutionError::Refused( + cellar_logic::consensus::CommandFailure::UnsupportedCapability, + ))); + } + Ok(()) + } +} + +impl CellGroupController { + /// Install the node's bounded compiled effect registrations once. All local + /// groups in these namespaces, including later registrations, participate. + /// Commands retain ordinary admission and S3 settlement. A scan or delivery + /// waiter cannot retire a pending command's physical SQL producer. + pub fn configure_effects( + &self, + registrations: Vec, + ) -> Result<(), CellGroupControllerError> { + let shared = &self.handle.0; + if registrations.is_empty() || registrations.len() > shared.limits.max_groups { + return Err(CellGroupControllerError::Invalid("effect registrations")); + } + let mut namespaces = BTreeMap::new(); + for registration in registrations { + let (namespace, class) = registration + .dispatcher + .effect_binding(&shared.domain) + .map_err(|_| CellGroupControllerError::Invalid("effect dispatcher owner/class"))?; + // Alarm discovery retains only Fire{id}; the payload stays on the + // selected SQLite image until the bounded SQL callback loads it. + if class.outbox_definition().is_some() + && registration.options.metadata_bytes < 512 * 1024 + { + return Err(CellGroupControllerError::Invalid( + "effect metadata capacity", + )); + } + let dispatcher = registration + .dispatcher + .with_s3(self) + .map_err(|_| CellGroupControllerError::Invalid("effect archive owner"))?; + let key = (namespace.cluster, namespace.namespace); + let registration = Arc::new(Registration { + dispatcher, + class, + schema: namespace.schema_digest, + options: registration.options, + memory: Budget::new(registration.options.metadata_bytes, 1), + sql: registration.sql, + }); + if namespaces.insert(key, registration).is_some() { + return Err(CellGroupControllerError::Invalid( + "duplicate effect namespace", + )); + } + } + let mut slot = shared + .effects + .lock() + .map_err(|_| CellGroupControllerError::Closed)?; + if slot.is_some() || shared.closing.load(Ordering::Acquire) { + return Err(CellGroupControllerError::Invalid( + "effect owner already configured/closed", + )); + } + *slot = Some(Arc::new(Config { namespaces })); + Ok(()) + } +} + +#[derive(Default)] +pub(super) struct Manager { + group: Option, + cells: BTreeMap, + hints: BTreeMap)>, + waiting: BTreeSet<(u64, GroupHistory)>, + ready: BTreeSet, + next_at: u64, + jobs: FuturesUnordered>, +} + +pub(super) struct Completed { + pub(super) group: GroupHistory, + cell: Option, + interval: u64, + result: Result<(), turn::Error>, +} + +impl Manager { + pub fn new( + roster: &BTreeMap, + ) -> Result { + let mut manager = Self::default(); + for applier in roster.values() { + manager.register(applier)?; + } + Ok(manager) + } + + pub fn register(&mut self, applier: &CellGroupApplier) -> Result<(), CellGroupControllerError> { + let (position, next) = applier + .effect_hint() + .map_err(|error| CellGroupControllerError::Apply(Arc::new(error)))?; + self.observe(applier.history(), position, next) + } + + pub fn observe( + &mut self, + group: GroupHistory, + position: LogIndex, + next: Option, + ) -> Result<(), CellGroupControllerError> { + if let Some((old_position, old_next)) = self.hints.get(&group) { + if position < *old_position { + return Ok(()); + } + if position == *old_position && next != *old_next { + return Err(CellGroupControllerError::Invalid( + "conflicting effect hint at one apply boundary", + )); + } + if let Some(old_next) = old_next { + self.waiting.remove(&(*old_next, group)); + } + } + self.ready.remove(&group); + self.hints.insert(group, (position, next)); + if let Some(next) = next { + self.waiting.insert((next, group)); + } + Ok(()) + } + + pub fn remove(&mut self, group: &GroupHistory) { + self.cells.remove(group); + self.ready.remove(group); + if let Some((_, Some(at))) = self.hints.remove(group) { + self.waiting.remove(&(at, *group)); + } + } + + pub fn running(&self) -> bool { + !self.jobs.is_empty() + } + + pub fn launch( + &mut self, + quarantined: &BTreeSet, + shared: &Arc, + now: u64, + ) -> Result<(), CellGroupControllerError> { + if self.running() || now < self.next_at || shared.closing.load(Ordering::Acquire) { + return Ok(()); + } + let Some(config) = shared + .effects + .lock() + .map_err(|_| CellGroupControllerError::Closed)? + .clone() + else { + return Ok(()); + }; + let Ok(wall) = u64::try_from(crate::asyncrt::wall_ms()) else { + return Ok(()); + }; + // Work scales with due groups. Empty and future-only groups incur no + // runtime RPC or SQLite open. Promotion is bounded per node turn. + for _ in 0..64 { + let Some((at, group)) = self.waiting.first().copied() else { + break; + }; + if at > wall { + break; + } + self.waiting.remove(&(at, group)); + self.ready.insert(group); + } + let after = self.group.map_or(Unbounded, Excluded); + let through = self.group.map_or(Unbounded, Included); + // Rotate ready groups rather than repeatedly choosing the oldest due + // timestamp. A hot cell/group cannot starve another due group. + for group in self + .ready + .range((after, Unbounded)) + .chain(self.ready.range((Unbounded, through))) + .take(4) + { + self.group = Some(*group); + if quarantined.contains(group) || !shared.groups.contains(group)? { + continue; + } + let Some(registration) = config.namespaces.get(&(group.cluster, group.namespace)) + else { + continue; + }; + let Some(reservation) = registration + .memory + .acquire(registration.options.metadata_bytes) + else { + continue; + }; + // Serialize work visibility with the public maintenance/drain + // barrier before scheduling a scan or invoking an effect adapter. + let mut admission = match shared.registration() { + Ok(admission) => admission, + // The reversible refusal was captured under the admission + // barrier. A concurrent resume cannot turn it into a failure. + Err(RuntimeError::NotReady) => return Ok(()), + Err(_) if shared.application_paused() || shared.closing.load(Ordering::Acquire) => { + return Ok(()) + } + Err(error) => return Err(error.into()), + }; + if shared.application_paused() { + return Ok(()); + } + let work = admission.work.take(); + let group = *group; + let after = self.cells.get(&group).cloned(); + let registration = registration.clone(); + drop(admission); + let shared = shared.clone(); + shared.active_effects.store(1, Ordering::Release); + self.jobs.push(Box::pin(async move { + let _work = work; + let mut cell = None; + let result = std::panic::AssertUnwindSafe(turn::run( + group, + after, + ®istration, + &shared, + reservation, + &mut cell, + )) + .catch_unwind() + .await + .unwrap_or(Err(turn::Error::Invalid("effect adapter panicked"))); + Completed { + group, + cell, + interval: registration.options.interval_ms, + result, + } + })); + return Ok(()); + } + self.next_at = now.saturating_add(shared.limits.retry_interval_ms); + Ok(()) + } + + pub async fn next(&mut self) -> Option { + self.jobs.next().await + } + + pub fn complete(&mut self, completed: Completed, shared: &Shared) { + if let Some(cell) = completed.cell { + if self.hints.contains_key(&completed.group) { + self.cells.insert(completed.group, cell); + } + } + self.next_at = shared + .domain + .mono_ms() + .unwrap_or(u64::MAX) + .saturating_add(completed.interval); + shared.active_effects.store(0, Ordering::Release); + shared.effect_turns.fetch_add(1, Ordering::Relaxed); + if let Err(error) = completed.result { + shared.effect_retries.fetch_add(1, Ordering::Relaxed); + tracing::debug!(group = ?completed.group, %error, "effect turn requires committed-state reconciliation"); + } + } + + pub async fn join(&mut self, shared: &Shared) { + // Closing stops admission; already admitted commands stay owned by the + // controller. Poll scans/factories/sink waiters to joined completion + // before the controller closes SQL and releases its physical claims. + while let Some(completed) = self.next().await { + self.complete(completed, shared); + } + } +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/effects/turn.rs b/crates/cellar/durability/raft/cell_groups/controller/effects/turn.rs new file mode 100644 index 000000000..7771bcc25 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/effects/turn.rs @@ -0,0 +1,352 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Every turn rereads committed state. Discovery hints never authorize sends. + +use super::*; +use crate::raft::cell_groups::{ + CellCommandRequest, CellCommandResponse, CellEffectAuthority, CellReadConsistency, + CellRoutingLimits, +}; +use crate::raft::effects::{outbox, scan::DueEffect, RaftDeliveryRequest, RaftEffectSink}; +use cellar_logic::consensus::{CommandFailure, CommandId, Durability}; +use std::time::Duration; + +#[derive(Debug)] +pub(super) enum Error { + Runtime(RuntimeError), + Execution(ExecutionError), + Command(Arc), + Effect(CommandFailure), + Invalid(&'static str), +} + +impl std::fmt::Display for Error { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::Runtime(e) => std::fmt::Display::fmt(e, f), + Self::Execution(e) => std::fmt::Display::fmt(e, f), + Self::Command(e) => std::fmt::Display::fmt(e, f), + Self::Effect(e) => std::fmt::Display::fmt(e, f), + Self::Invalid(e) => f.write_str(e), + } + } +} + +fn open(shared: &Shared) -> Result<(), Error> { + if shared.closing.load(Ordering::Acquire) { + Err(Error::Runtime(RuntimeError::Closed)) + } else { + Ok(()) + } +} + +fn clock(shared: &Shared) -> Result { + shared + .domain + .mono_ms() + .map_err(|_| Error::Runtime(RuntimeError::Closed)) +} + +fn wall() -> Result { + u64::try_from(crate::asyncrt::wall_ms()).map_err(|_| Error::Runtime(RuntimeError::Closed)) +} + +fn effect(error: anyhow::Error) -> Error { + Error::Effect( + error + .downcast_ref::() + .cloned() + .unwrap_or(CommandFailure::CorruptState), + ) +} + +pub(super) async fn run( + group: GroupHistory, + after: Option, + registration: &Registration, + shared: &Arc, + reservation: crate::raft::budget::Permit, + cursor: &mut Option, +) -> Result<(), Error> { + open(shared)?; + let Some(execution) = shared + .runtime + .next_effect(group, wall()?, after) + .await + .map_err(Error::Runtime)? + else { + return Ok(()); + }; + if execution.route().schema_digest != registration.schema { + return Err(Error::Invalid("effect group schema")); + } + let cell = execution.cell().version.history.clone(); + *cursor = Some(cell.cell.clone()); + // Retain this coherent image and original leader fence through delivery. + let sql = shared + .sql + .upgrade() + .ok_or(Error::Runtime(RuntimeError::Closed))?; + let factory = registration.sql.clone(); + let factory_cell = cell.clone(); + let options = shared + .domain + .blocking(move || factory(&factory_cell)) + .map_err(|_| Error::Runtime(RuntimeError::Closed))? + .await + .map_err(|_| Error::Runtime(RuntimeError::Closed))? + .map_err(Error::Execution)?; + options.limits().validate().map_err(Error::Execution)?; + let limits = options.limits(); + if registration.options.retry_window_ms > limits.max_retry_window_ms { + return Err(Error::Invalid( + "effect retry window exceeds execution limit", + )); + } + if let Some(alarms) = registration.class.alarm_limits() { + if limits.envelope.max_payload_bytes < 1024 + || limits.max_client_identities < 2 + || limits.max_retained_receipts < 2 + || alarms + .max_payload_bytes + .checked_mul(alarms.max_alarms) + .and_then(|n| n.checked_add(128 * 1024)) + .is_none_or(|n| n as u64 > limits.max_image_bytes) + { + return Err(Error::Invalid("alarm execution capacity")); + } + } + if let Some(outbox) = registration.class.outbox_definition() { + if limits.envelope.max_payload_bytes < 32 * 1024 + || limits.max_result_bytes < 33 + crate::raft::effects::delivery::MAX_REQUEST + || limits.max_client_identities < 2 + || limits.max_retained_receipts < 3 + || (outbox.limits.max_total_bytes as u64) + .checked_add(128 * 1024) + .is_none_or(|n| n > limits.max_image_bytes) + { + return Err(Error::Invalid("effect execution capacity")); + } + } + open(shared)?; + let deadline = clock(shared)? + .checked_add(registration.options.operation_timeout_ms) + .ok_or(Error::Invalid("effect deadline"))?; + let authority = shared + .runtime + .read_authority( + group, + cell.clone(), + CellReadConsistency::Linearizable, + deadline, + ) + .await + .map_err(Error::Runtime)?; + let scan = sql + .scan_effects( + options.clone(), + authority, + registration.class.clone(), + wall()?, + reservation, + ) + .await + .map_err(Error::Execution)?; + let crate::raft::effects::scan::EffectScan { + next, + due, + _reservation, + .. + } = *scan.into_scan(); + let _metadata = _reservation; + let Some(due) = due else { + return Ok(()); + }; + let minimum = due.minimum(); + let (operation, input) = due.command().map_err(effect)?; + let retry_until = wall()? + .checked_add(registration.options.retry_window_ms) + .ok_or(Error::Invalid("effect retry window"))?; + let receipt = command( + shared, + registration, + &cell, + options.clone(), + request(registration, next, &operation, input, minimum, retry_until)?, + ) + .await?; + let local = receipt + .local_receipt() + .ok_or(Error::Invalid("local effect receipt"))?; + match due { + DueEffect::Alarm(_) => check_result(receipt.result()), + DueEffect::Delivery { + selected, + definition, + .. + } => { + let Some(claimed) = outbox::claimed( + &definition, + &cell, + registration.schema, + &selected, + receipt.result(), + ) + .map_err(effect)? + else { + return Ok(()); + }; + let authority = execution + .effect_authority(&local.version, local.position.term) + .map_err(Error::Runtime)?; + // The committed claim has been consumed into bounded delivery + // metadata and fenced image custody. Release its response admission + // before a loopback receiver or source completion needs the same + // one-slot SQL/archive lane. Published S3 durability survives a + // consumer dropping its response; the durable Dispatched row stays. + drop(receipt); + let proof = deliver( + shared, + authority, + claimed.request.clone(), + claimed.sink, + registration.options.operation_timeout_ms, + ) + .await?; + claimed.request.verify_receipt(&proof).map_err(effect)?; + let input = + outbox::completion_input(selected, claimed.generation, &proof).map_err(effect)?; + let next = CommandId::new( + next.client, + next.sequence() + .checked_add(1) + .ok_or(Error::Effect(CommandFailure::Overloaded))?, + ) + .map_err(|_| Error::Effect(CommandFailure::Overloaded))?; + let receipt = command( + shared, + registration, + &cell, + options, + request( + registration, + next, + outbox::COMPLETE, + input, + minimum, + retry_until, + )?, + ) + .await?; + check_result(receipt.result()) + } + } +} + +fn check_result(result: &[u8]) -> Result<(), Error> { + if result == [0] || result == [1] { + Ok(()) + } else { + Err(Error::Effect(CommandFailure::CorruptState)) + } +} + +fn request( + registration: &Registration, + id: CommandId, + operation: &str, + input: Vec, + durability: Durability, + retry_until_ms: u64, +) -> Result { + let contract = registration + .class + .system_contract(operation) + .ok_or(Error::Invalid("effect system operation"))?; + Ok(CellCommandRequest { + id, + contract, + input, + durability, + retry_until_ms, + }) +} + +async fn command( + shared: &Arc, + registration: &Registration, + cell: &CellHistory, + options: CellSqlOptions, + request: CellCommandRequest, +) -> Result { + open(shared)?; + if wall()? >= request.retry_until_ms { + return Err(Error::Effect(CommandFailure::Unavailable)); + } + let invocation = registration + .dispatcher + .system_command( + cell.clone(), + request, + CellRoutingLimits { + timeout: Duration::from_millis(registration.options.operation_timeout_ms), + max_attempts: 4, + }, + ) + .await + .map_err(|error| Error::Command(Arc::new(error)))?; + let call = commands::admit(shared, invocation, options) + .map_err(|error| Error::Command(Arc::new(error.error)))?; + let response = call.response(); + tokio::pin!(response); + loop { + open(shared)?; + let timer = shared + .domain + .sleep_until(clock(shared)?.saturating_add(100)) + .map_err(|_| Error::Runtime(RuntimeError::Closed))?; + crate::asyncrt::select! { + result = &mut response => return result.map_err(Error::Command), + _ = timer => (), + } + } +} + +async fn deliver( + shared: &Shared, + authority: CellEffectAuthority, + request: RaftDeliveryRequest, + sink: Arc, + timeout: u64, +) -> Result, Error> { + let deadline = clock(shared)? + .checked_add(timeout) + .ok_or(Error::Invalid("delivery deadline"))?; + let mut send = None; + let mut proof = None; + loop { + open(shared)?; + if clock(shared)? >= deadline { + return Err(Error::Effect(CommandFailure::Unavailable)); + } + // Apply lag and unrelated business mutations do not revoke this + // committed dispatch. Term, route, activation, membership and cell + // lineage still must cover the claim before send and reply acceptance. + authority.check().map_err(Error::Runtime)?; + if let Some(proof) = proof { + return Ok(proof); + } + if send.is_none() { + send = Some(sink.deliver(request.clone())); + } + let timer = shared + .domain + .sleep_until(clock(shared)?.saturating_add(100).min(deadline)) + .map_err(|_| Error::Runtime(RuntimeError::Closed))?; + let pending = send.as_mut().ok_or(Error::Invalid("delivery waiter"))?; + crate::asyncrt::select_biased! { "source authority is rechecked before accepting a late delivery"; + _ = timer => (), + result = pending => { proof = Some(result.map_err(Error::Effect)?); } + } + } +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/enrollments.rs b/crates/cellar/durability/raft/cell_groups/controller/enrollments.rs new file mode 100644 index 000000000..25bb12c83 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/enrollments.rs @@ -0,0 +1,183 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Bounded invocation ownership is independent of response-waiter lifetime. +use super::admission::Retention; +use super::calls::Delivery; +use super::*; +use crate::raft::cell_groups::{ + CellDispatchError, CellEnrollmentError, CellEnrollmentInvocation, CellEnrollmentResponse, + CellSqlOptions, +}; +use cellar_logic::consensus::ProposalId; +use tokio::sync::{mpsc, oneshot, watch}; + +pub(super) struct Request { + invocation: CellEnrollmentInvocation, + options: CellSqlOptions, + reply: oneshot::Sender>, + permit: Arc, +} + +pub(super) struct Completed { + pub invocation: CellEnrollmentInvocation, + pub error: Option>, + _permit: Arc, +} + +/// Dropping or timing out this response never cancels the admitted enrollment. +/// An unpolled reply retains node admission, including error-only replies. +pub struct CellEnrollmentCall { + inner: calls::Call, +} + +/// Admission did not transfer ownership. In particular, a rejected non-fresh +/// invocation is returned with its original recovery state intact. +pub struct CellEnrollmentAdmissionError { + pub error: CellEnrollmentError, + pub invocation: Box, +} + +impl std::fmt::Debug for CellEnrollmentAdmissionError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("CellEnrollmentAdmissionError") + .field("error", &self.error) + .finish_non_exhaustive() + } +} + +impl std::fmt::Display for CellEnrollmentAdmissionError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + std::fmt::Display::fmt(&self.error, f) + } +} + +impl std::error::Error for CellEnrollmentAdmissionError { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + Some(&self.error) + } +} + +fn error(command: ProposalId, ambiguous: bool, reason: RuntimeError) -> CellEnrollmentError { + CellEnrollmentError { + creation: command, + ambiguous, + cause: Box::new(CellDispatchError::Runtime(reason)), + } +} + +impl CellEnrollmentCall { + pub async fn response(self) -> Result> { + self.inner.response().await + } +} + +impl CellGroupController { + /// Admit a fresh invocation. An SDK origin may select a remote destination + /// inside the owner. Incoming and verification invocations remain local. + pub fn enroll( + &self, + invocation: CellEnrollmentInvocation, + mut options: CellSqlOptions, + ) -> Result { + let shared = &self.handle.0; + let admitted = (|| { + let (command, deadline) = invocation.controller_admission(&shared.domain)?; + let registration = shared + .registration() + .map_err(|reason| error(command, false, reason))?; + if shared.closing.load(Ordering::Acquire) { + return Err(error(command, false, RuntimeError::Closed)); + } + let bytes = 8192_usize + .checked_add( + commands::option_bytes(&mut options).map_err(|e| error(command, false, e))?, + ) + .ok_or_else(|| error(command, false, RuntimeError::Overloaded))?; + let permit = shared + .commands + .acquire(bytes) + .ok_or_else(|| error(command, false, RuntimeError::Overloaded))?; + Ok((command, deadline, permit, registration)) + })(); + let (command, deadline, permit, mut registration) = match admitted { + Ok(admitted) => admitted, + Err(error) => { + return Err(CellEnrollmentAdmissionError { + error, + invocation: Box::new(invocation), + }) + } + }; + let (reply, receiver) = oneshot::channel(); + shared + .requests + .try_send(requests::Request::Enrollment(Box::new(Request { + invocation, + options, + reply, + permit: Arc::new(Retention { + _permit: permit, + _work: registration.work.take(), + }), + }))) + .map_err(|e| { + let (request, reason) = match e { + mpsc::error::TrySendError::Closed(request) => (request, RuntimeError::Closed), + mpsc::error::TrySendError::Full(request) => (request, RuntimeError::Overloaded), + }; + CellEnrollmentAdmissionError { + error: error(command, false, reason), + invocation: match request { + requests::Request::Enrollment(request) => Box::new(request.invocation), + _ => unreachable!("enrollment admission envelope"), + }, + } + })?; + Ok(CellEnrollmentCall { + inner: calls::Call { + receiver, + identity: command, + deadline, + domain: shared.domain.clone(), + failure: |id, reason| error(id, true, reason), + }, + }) + } +} + +pub(super) async fn drive( + request: Request, + runtime: CellGroupRuntime, + sql: CellGroupSqlPool, + shared: Arc, + mut progress: watch::Receiver<()>, + transport: RaftTransport, +) -> Completed { + let Request { + mut invocation, + options, + reply, + permit, + } = request; + let result = invocation + .run_owned( + &runtime, + &sql, + options, + &shared.groups, + &mut progress, + &transport, + ) + .await + .map_err(Arc::new); + let error = result.as_ref().err().cloned(); + let _ = reply.send(Delivery { + result, + _permit: permit.clone(), + }); + Completed { + invocation, + error, + _permit: permit, + } +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/failure.rs b/crates/cellar/durability/raft/cell_groups/controller/failure.rs new file mode 100644 index 000000000..66dcca351 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/failure.rs @@ -0,0 +1,68 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Group isolation never converts shared-engine uncertainty into availability. +use super::*; +use crate::raft::{cell_groups::CellDispatchError, storage::LogError}; + +fn runtime(error: &RuntimeError) -> bool { + matches!(error, RuntimeError::Closed) + || matches!(error, RuntimeError::Storage(error) if storage(error)) +} + +fn storage(error: &LogError) -> bool { + matches!( + error, + LogError::Io(_) | LogError::Corrupt(_) | LogError::LostStableState | LogError::Closed + ) +} + +fn execution(error: &ExecutionError) -> bool { + match error { + // A lost shared SQL worker reply cannot prove that the pool remains + // usable. Preserve its existing joined node shutdown boundary. + ExecutionError::Refused(cellar_logic::consensus::CommandFailure::Unavailable) => true, + ExecutionError::Storage(error) => { + error.downcast_ref::().is_some_and(storage) + || error.downcast_ref::().is_some_and(runtime) + } + _ => false, + } +} + +fn dispatch(error: &CellDispatchError) -> bool { + match error { + CellDispatchError::Runtime(error) => runtime(error), + CellDispatchError::Execution(error) => execution(error), + _ => false, + } +} + +pub(super) fn node_wide(error: &CellGroupControllerError) -> bool { + match error { + CellGroupControllerError::Catalog(error) => { + matches!(error.as_ref(), crate::raft::catalog::CatalogError::Log(error) if storage(error)) + } + CellGroupControllerError::Runtime(error) => runtime(error), + CellGroupControllerError::Apply(error) => execution(error), + CellGroupControllerError::Archive(error) => match error.as_ref() { + CellGroupArchiveControlError::Runtime(error) => runtime(error), + CellGroupArchiveControlError::Binding( + super::super::CellGroupArchiveBindingError::Catalog( + crate::raft::catalog::CatalogError::Log(error), + ), + ) => storage(error), + CellGroupArchiveControlError::Archive(super::super::CellGroupArchiveError::Replay( + error, + )) => execution(error), + CellGroupArchiveControlError::Stopped => true, + _ => false, + }, + CellGroupControllerError::Checkpoint(error) => { + matches!(error.as_ref(), super::super::CellGroupCheckpointError::Log(error) if storage(error)) + } + CellGroupControllerError::Command(error) => dispatch(&error.cause), + CellGroupControllerError::Enrollment(error) => dispatch(&error.cause), + CellGroupControllerError::Closed | CellGroupControllerError::Transport(_) => true, + CellGroupControllerError::Invalid(_) => false, + } +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/mod.rs b/crates/cellar/durability/raft/cell_groups/controller/mod.rs new file mode 100644 index 000000000..7f6d6d1b8 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/mod.rs @@ -0,0 +1,510 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Owns registered groups' consensus events, transport dispatch and physical apply. +//! Admitted command ownership survives response cancellation and expiry. + +mod admission; +mod archive; +pub(crate) use archive::settlement::{ + Client as S3Client, LookupPrepared as S3LookupPrepared, OutputProof as S3OutputProof, +}; +pub(crate) use archive::Provisioning as CellGroupArchiveProvisioning; +pub(crate) use archive::{ArchiveSeedClient, ArchiveSeedDescriptor}; +mod calls; +pub use archive::{ + CellGroupArchiveCall, CellGroupArchiveControlError, CellGroupArchiveRegistration, + CellGroupArchiveStatus, +}; +mod checkpoints; +pub use checkpoints::CellGroupCheckpointPolicy; +mod commands; +mod queries; +mod query_routes; +pub use queries::CellQueryCall; +mod effects; +mod enrollments; +pub use effects::CellGroupEffectRegistration; +mod failure; +mod provision; +mod registration; +pub use provision::{ + CellGroupProvisionCall, CellGroupProvisionLimits, CellGroupProvisionNamespace, + CellGroupProvisioning, +}; +mod requests; +pub use enrollments::{CellEnrollmentAdmissionError, CellEnrollmentCall}; +pub(crate) use registration::GroupRoster; +pub use registration::{ + CellGroupRegistration, CellGroupRegistrationCall, CellGroupRegistrationError, +}; +mod worker; +pub use commands::{CellCommandAdmissionError, CellCommandCall}; + +use super::{CellGroupApplier, CellGroupRuntime, CellGroupRuntimeEvents, CellGroupSqlPool}; +use crate::raft::{ + completion::RuntimeError, + execution::ExecutionError, + transport::{RaftTransport, TransportError}, +}; +use futures_util::FutureExt; +use std::collections::BTreeMap; +use std::sync::{ + atomic::{AtomicBool, AtomicU64, AtomicUsize, Ordering}, + Arc, Mutex, OnceLock, +}; +use tokio::sync::Notify; + +#[derive(Clone, Copy, Debug)] +pub struct CellGroupControllerLimits { + pub max_groups: usize, + pub max_parallel_apply: usize, + /// One node timer backs off transient SQL admission, never one timer per cell. + pub retry_interval_ms: u64, + /// Shared by commands, queries, enrollments and registrations, including queued work, resolution + /// and unconsumed responses. + pub max_commands: usize, + /// Invocation/input/options retention. The independent S3 output stage has + /// the same byte ceiling; configure their combined envelope as twice this + /// value. SQL capture/apply/results retain separate SQL pool admission. + pub max_command_bytes: usize, + /// Automatic compaction at an apply boundary. Requires the transport's + /// shared checkpoint pool. None leaves local publication to its caller. + pub checkpoints: Option, +} + +#[derive(Clone, Debug)] +pub enum CellGroupControllerError { + Invalid(&'static str), + Runtime(RuntimeError), + Apply(Arc), + Archive(Arc), + Checkpoint(Arc), + Transport(TransportError), + Catalog(Arc), + Command(Arc), + Enrollment(Arc), + Closed, +} + +impl std::fmt::Display for CellGroupControllerError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "cell-group controller: {self:?}") + } +} + +impl std::error::Error for CellGroupControllerError {} + +impl From for CellGroupControllerError { + fn from(error: RuntimeError) -> Self { + Self::Runtime(error) + } +} + +#[derive(Clone, Copy, Debug, serde::Serialize)] +pub struct CellGroupControllerUsage { + pub ordinary: super::CellGroupAdmissionUsage, + pub archive_settlement: super::CellGroupAdmissionUsage, + pub groups: usize, + pub pending_registrations: usize, + /// Retained failed groups still consume the registration capacity. + pub quarantined_groups: usize, + pub active_apply: usize, + pub pending_apply: usize, + pub applied: u64, + pub active_checkpoints: usize, + pub checkpoints_published: u64, + pub checkpoint_retries: u64, + pub apply_retries: u64, + /// Immediate queue/route refusals. The transport tracks later I/O failures. + pub send_refusals: u64, + pub closing: bool, + /// Ordinary node calls, including queries, enrollments and registrations. + pub retained_commands: usize, + pub command_bytes: usize, + pub retained_preparations: usize, + pub preparation_bytes: usize, + /// Immutable namespace/directory metadata charged to provisioning's total. + pub provision_configuration_bytes: usize, + pub provision_waiters: usize, + /// Independent archive-settlement admission, bounded by max_commands and + /// max_command_bytes. This lane cannot recursively consume command slots. + pub retained_s3: usize, + pub s3_bytes: usize, + /// At most one effect turn runs on the node. SQL and command admission are + /// shared with ordinary operations; configured metadata has its own permit. + pub active_effects: usize, + pub effect_turns: u64, + pub effect_retries: u64, + /// Includes response handoff and physical retirement of seed uploads. + pub active_archive_seeds: usize, + /// One node-owned streaming journal sweep; never supplies serving authority. + pub active_archive_seed_cleanup: usize, + pub archive_seed_cleanup_errors: usize, +} + +type HostResourceRelease = Box Result<(), CellGroupControllerError> + Send>; + +struct Shared { + admission_configuration: Mutex<()>, + public: OnceLock, + // One-shot physical release is opaque to break the ownership type graph: + // bindings contain weak archive clients back to this same shared owner. + host_resources: Mutex>, + runtime: CellGroupRuntime, + sql: crate::raft::host_sql::cell_groups::WeakCellGroupSqlPool, + groups: Arc, + limits: CellGroupControllerLimits, + domain: crate::asyncrt::DomainToken, + commands: Arc, + provisioning: Mutex>>, + effects: Mutex>>, + active_effects: AtomicUsize, + effect_turns: AtomicU64, + effect_retries: AtomicU64, + requests: tokio::sync::mpsc::Sender, + archives: Mutex>, + archive_seeds: Arc, + s3: Arc, + active_apply: AtomicUsize, + pending_apply: AtomicUsize, + applied: AtomicU64, + checkpoints: Option, + active_checkpoints: AtomicUsize, + checkpoints_published: AtomicU64, + checkpoint_retries: AtomicU64, + apply_retries: AtomicU64, + send_refusals: AtomicU64, + closing: AtomicBool, + stop: Notify, + terminal: Mutex>>, + finished: Notify, +} + +impl Shared { + fn stop(&self) { + self.closing.store(true, Ordering::Release); + self.stop.notify_one(); + } + + fn complete(&self, result: Result<(), CellGroupControllerError>) { + let mut terminal = self.terminal.lock().unwrap_or_else(|e| e.into_inner()); + if terminal.is_none() { + *terminal = Some(result); + } + drop(terminal); + self.finished.notify_waiters(); + } + + async fn wait(&self) -> Result<(), CellGroupControllerError> { + loop { + let finished = self.finished.notified(); + tokio::pin!(finished); + finished.as_mut().enable(); + let result = self + .terminal + .lock() + .unwrap_or_else(|e| e.into_inner()) + .clone(); + if let Some(result) = result { + return result; + } + finished.await; + } + } +} + +struct Handle(Arc); + +impl Drop for Handle { + fn drop(&mut self) { + self.0.stop(); + } +} + +/// One worker per node, with a bounded dynamic roster and no cell-specific tasks. +/// Dropping the final public handle requests joined shutdown. An explicit close +/// is required to observe its result. Runtime/transport/pool clones remain +/// usable during operation; closing this owner closes all three services. +#[derive(Clone)] +pub struct CellGroupController { + handle: Arc, +} + +impl CellGroupController { + pub(crate) fn limits(&self) -> CellGroupControllerLimits { + self.handle.0.limits + } + + pub(crate) fn belongs_to(&self, runtime: &CellGroupRuntime) -> bool { + self.handle.0.runtime.same_owner(runtime) + } + + pub(in crate::raft::cell_groups) fn own_host_resources( + &self, + resources: Arc, + ) -> Result<(), RuntimeError> { + let mut slot = self + .handle + .0 + .host_resources + .lock() + .map_err(|_| RuntimeError::Closed)?; + if self.handle.0.closing.load(Ordering::Acquire) { + return Err(RuntimeError::Closed); + } + if slot.is_some() { + return Err(RuntimeError::AlreadyAttached); + } + *slot = Some(Box::new(move || resources.release())); + Ok(()) + } + + /// Consume the exclusive event receiver after validating exact engine and + /// runtime ownership. Initial groups must already be attached. Later groups + /// enter through register() or provision(); unknown events still fail closed. + pub async fn start( + runtime: CellGroupRuntime, + events: CellGroupRuntimeEvents, + transport: RaftTransport, + sql: CellGroupSqlPool, + appliers: Vec, + limits: CellGroupControllerLimits, + ) -> Result { + if limits.max_groups == 0 + || limits.max_groups > runtime.log().limits().max_groups + || limits.max_parallel_apply == 0 + || limits.max_parallel_apply > limits.max_groups.min(256) + || !(1..=60_000).contains(&limits.retry_interval_ms) + || appliers.len() > limits.max_groups + || !(1..=65_536).contains(&limits.max_commands) + || limits.max_command_bytes < 8192 + { + return Err(CellGroupControllerError::Invalid( + "controller limits/roster", + )); + } + if !events.belongs_to(&runtime) + || !transport.uses_shared_runtime(&runtime) + || !sql.belongs_to_engine(&runtime.log().owner()) + { + return Err(CellGroupControllerError::Invalid("controller owner")); + } + let checkpoints = checkpoints::Config::new(limits, &transport)?; + let mut roster = BTreeMap::new(); + for applier in appliers { + if !applier.belongs_to_engine(&runtime.log().owner()) + || roster.insert(applier.history(), applier).is_some() + { + return Err(CellGroupControllerError::Invalid("controller applier")); + } + } + for history in roster.keys() { + runtime.status(*history).await?; + } + runtime.admit()?; + let domain = runtime.log().domain(); + let (requests, receiver) = tokio::sync::mpsc::channel(limits.max_commands); + let shared = Arc::new(Shared { + admission_configuration: Mutex::new(()), + public: OnceLock::new(), + host_resources: Mutex::new(None), + runtime: runtime.clone(), + sql: sql.downgrade(), + groups: GroupRoster::new(roster.keys().copied(), limits.max_groups), + limits, + domain: domain.clone(), + commands: commands::budget(limits), + provisioning: Mutex::new(None), + effects: Mutex::new(None), + active_effects: AtomicUsize::new(0), + effect_turns: AtomicU64::new(0), + effect_retries: AtomicU64::new(0), + requests, + archives: Mutex::new(BTreeMap::new()), + archive_seeds: Arc::new(archive::seed::Jobs::new(limits.max_groups)), + s3: commands::budget(limits), + active_apply: AtomicUsize::new(0), + pending_apply: AtomicUsize::new(0), + applied: AtomicU64::new(0), + checkpoints, + active_checkpoints: AtomicUsize::new(0), + checkpoints_published: AtomicU64::new(0), + checkpoint_retries: AtomicU64::new(0), + apply_retries: AtomicU64::new(0), + send_refusals: AtomicU64::new(0), + closing: AtomicBool::new(false), + stop: Notify::new(), + terminal: Mutex::new(None), + finished: Notify::new(), + }); + let handle = Self { + handle: Arc::new(Handle(shared.clone())), + }; + let worker_domain = domain.clone(); + // Infrastructure cleanup is not cancelled with application tasks. The + // worker retains tickets/appliers until all physical owners have joined. + let mut worker = worker::Worker::new( + runtime, + events, + transport, + sql, + roster, + receiver, + shared.clone(), + )?; + domain + .spawn_cleanup(async move { + let result = std::panic::AssertUnwindSafe(worker.run(&worker_domain)) + .catch_unwind() + .await + .unwrap_or(Err(CellGroupControllerError::Runtime( + RuntimeError::Protocol("controller worker panicked".into()), + ))); + shared.stop(); + let joined = worker.finish().await; + drop(worker); + shared + .provisioning + .lock() + .unwrap_or_else(|e| e.into_inner()) + .take(); + shared + .effects + .lock() + .unwrap_or_else(|e| e.into_inner()) + .take(); + shared.active_effects.store(0, Ordering::Release); + shared.active_checkpoints.store(0, Ordering::Release); + shared.active_apply.store(0, Ordering::Release); + shared.pending_apply.store(0, Ordering::Release); + // Resource release belongs to this joined owner, never the + // lifetime of an async close waiter or a retained cell handle. + let released = match shared.host_resources.lock() { + Ok(mut slot) => slot.take().map_or(Ok(()), |release| release()), + Err(error) => { + if let Some(release) = error.into_inner().take() { + let _ = release(); + } + Err(CellGroupControllerError::Invalid( + "host resource owner poisoned", + )) + } + }; + shared.complete(result.and(joined).and(released)); + }) + .map_err(|_| CellGroupControllerError::Closed)?; + Ok(handle) + } + + /// Original failure retained for a quarantined group. Recovery requires a + /// joined restart; registration cannot replace the failed owner in place. + pub fn group_failure( + &self, + group: cellar_logic::cell_group::GroupHistory, + ) -> Result, RuntimeError> { + self.handle.0.groups.failure(&group) + } + + pub fn usage(&self) -> CellGroupControllerUsage { + let shared = &self.handle.0; + let (command_bytes, retained_commands) = shared.commands.usage(); + let (s3_bytes, retained_s3) = shared.s3.usage(); + let (active_archive_seeds, active_archive_seed_cleanup, archive_seed_cleanup_errors) = + shared.archive_seeds.usage(); + let ( + preparation_bytes, + retained_preparations, + provision_waiters, + provision_configuration_bytes, + ) = shared + .provisioning + .lock() + .ok() + .and_then(|slot| { + slot.as_ref().map(|config| { + let (bytes, items) = config.preparations.usage(); + ( + bytes, + items, + config.waiters.usage().1, + config.configuration_bytes, + ) + }) + }) + .unwrap_or_default(); + CellGroupControllerUsage { + ordinary: super::CellGroupAdmissionUsage::read(&shared.commands), + archive_settlement: super::CellGroupAdmissionUsage::read(&shared.s3), + active_effects: shared.active_effects.load(Ordering::Acquire), + effect_turns: shared.effect_turns.load(Ordering::Relaxed), + effect_retries: shared.effect_retries.load(Ordering::Relaxed), + preparation_bytes, + provision_configuration_bytes, + retained_preparations, + provision_waiters, + groups: shared.groups.len(), + pending_registrations: shared.groups.pending(), + quarantined_groups: shared.groups.quarantined(), + command_bytes, + retained_commands, + s3_bytes, + retained_s3, + active_apply: shared.active_apply.load(Ordering::Acquire), + pending_apply: shared.pending_apply.load(Ordering::Acquire), + applied: shared.applied.load(Ordering::Relaxed), + active_checkpoints: shared.active_checkpoints.load(Ordering::Acquire), + checkpoints_published: shared.checkpoints_published.load(Ordering::Relaxed), + checkpoint_retries: shared.checkpoint_retries.load(Ordering::Relaxed), + apply_retries: shared.apply_retries.load(Ordering::Relaxed), + send_refusals: shared.send_refusals.load(Ordering::Relaxed), + closing: shared.closing.load(Ordering::Acquire), + active_archive_seeds, + active_archive_seed_cleanup, + archive_seed_cleanup_errors, + } + } + + /// Wait for terminal failure or joined shutdown without requesting a stop. + pub async fn wait(&self) -> Result<(), CellGroupControllerError> { + self.handle.0.wait().await + } + + /// Cancelling this waiter does not cancel transport, persistence or SQL joins. + pub async fn close(&self) -> Result<(), CellGroupControllerError> { + self.handle.0.stop(); + self.wait().await + } +} + +impl CellGroupController { + pub(crate) async fn lookup_receipt( + &self, + mut invocation: super::CellReceiptInvocation, + options: super::CellSqlOptions, + ) -> Result { + let shared = &self.handle.0; + let work = { + let mut registration = shared + .registration() + .map_err(super::CellDispatchError::Runtime)?; + registration.work.take() + }; + let _work = work; + invocation.begin().await?; + let attempt = invocation + .attempt() + .ok_or(super::CellDispatchError::Runtime(RuntimeError::NotFound))?; + if !shared + .groups + .contains(&attempt.route().history) + .map_err(super::CellDispatchError::Runtime)? + { + return Err(super::CellDispatchError::Runtime(RuntimeError::NotFound)); + } + let sql = shared + .sql + .upgrade() + .ok_or(super::CellDispatchError::Runtime(RuntimeError::Closed))?; + invocation.execute(&shared.runtime, &sql, options).await + } +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/provision/config.rs b/crates/cellar/durability/raft/cell_groups/controller/provision/config.rs new file mode 100644 index 000000000..41dd35501 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/provision/config.rs @@ -0,0 +1,278 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Explicit node resources; configuration itself performs no physical work. +use super::*; +use crate::application::consensus::group_protocol::GroupEntryLimits; +use crate::raft::cell_groups::{CellGroupApplyImages, ManifestLimits, ManifestPins}; +use crate::raft::{budget::Budget, transport_options::RaftPeerNode}; +use cellar_logic::cell_group::GroupHistory; +use std::path::PathBuf; + +/// Node-shared dynamic preparation ceilings; no namespace gets its own bank. +#[derive(Clone, Copy, Debug)] +pub struct CellGroupProvisionLimits { + pub max_preparations: usize, + pub max_preparation_bytes: usize, + pub max_parallel_preparations: usize, + pub max_waiters: usize, + pub peer_revision: u64, +} + +impl CellGroupProvisionLimits { + pub(crate) fn validate(&self, max_groups: usize) -> Result<(), CellGroupControllerError> { + if self.max_preparations == 0 + || self.max_preparations > max_groups + || self.max_preparation_bytes == 0 + || self.max_parallel_preparations == 0 + || self.max_parallel_preparations > self.max_preparations.min(256) + || !(1..=65_536).contains(&self.max_waiters) + || self.peer_revision == 0 + { + return Err(CellGroupControllerError::Invalid( + "group provisioning limits", + )); + } + Ok(()) + } +} + +/// Immutable namespace image resources. Directory identity is checked before +/// catalog reads or physical preparation; a matching name is not ownership. +pub struct CellGroupProvisionNamespace { + pub directory: RangeDirectory, + pub images: CellGroupApplyImages, +} + +/// Resources for initial members of already assigned ranges. Managed fleet +/// admission is not inferred from this configuration. Image stores retain their +/// own quotas; page inventory and recovery work have additional bounds here. +pub struct CellGroupProvisioning { + pub directory: PathBuf, + pub filesystem: Arc, + pub namespaces: Vec, + pub manifest: ManifestLimits, + pub entries: GroupEntryLimits, + pub max_effect_rows: u32, + pub max_pages: usize, + pub max_page_bytes: u64, + pub max_retained_roots: usize, + pub peers: Vec, + pub peer_revision: u64, + /// Includes recovered owners retained after a failed attachment. + pub max_preparations: usize, + pub max_preparation_bytes: usize, + pub max_parallel_preparations: usize, + /// Includes queued calls and unconsumed replies, independently of commands. + pub max_waiters: usize, +} + +pub(in super::super) struct Config { + pub settings: CellGroupProvisioning, + pub archives: Option, + pub preparations: Arc, + pub waiters: Arc, + pub parallel: Arc, + pub bytes: usize, + pub configuration_bytes: usize, + // Startup verified these physical owners before enabling image collection. + // They have no controller roster or transport registration. + pub(super) parked: Mutex>, +} + +impl Config { + pub fn new( + mut settings: CellGroupProvisioning, + shared: &Shared, + archives: Option, + parked: Vec<(CellGroupApplier, PreparedGroupManifest)>, + ) -> Result { + let invalid = || CellGroupControllerError::Invalid("group provisioning configuration"); + settings.manifest.validate().map_err(|_| invalid())?; + settings.entries.validate().map_err(|_| invalid())?; + if settings.namespaces.is_empty() || settings.namespaces.len() > 4096 { + return Err(invalid()); + } + let mut namespaces = std::collections::BTreeSet::new(); + // Trusted endpoints and fixed configuration metadata remain retained + // even when no preparation is active. Keep them in the same total. + let mut configuration_bytes = 2 * 1024 * 1024_usize; + let mut image_bytes = 0; + for profile in &mut settings.namespaces { + let namespace = profile.directory.namespace_config(); + if !namespaces.insert((namespace.cluster, namespace.namespace)) + || !matches!( + (&profile.images, namespace.image_storage), + ( + CellGroupApplyImages::Materialized { .. }, + crate::raft::catalog::RaftImageStorage::Materialized + ) | ( + CellGroupApplyImages::Extents { .. }, + crate::raft::catalog::RaftImageStorage::Extents + ) + ) + { + return Err(invalid()); + } + profile.images.validate().map_err(apply_error)?; + let mut metadata = profile + .directory + .cache_capacity_bytes() + .checked_add(8192) + .ok_or_else(invalid)?; + if let CellGroupApplyImages::Extents { + cache_directory, .. + } = &mut profile.images + { + if !cache_directory.is_absolute() + || cache_directory.as_os_str().len() > 4096 + || cache_directory + .components() + .any(|p| matches!(p, std::path::Component::ParentDir)) + { + return Err(invalid()); + } + metadata = metadata + .checked_add(cache_directory.as_os_str().len() * 2) + .ok_or_else(invalid)?; + *cache_directory = std::mem::take(cache_directory) + .into_boxed_path() + .into_path_buf(); + } + configuration_bytes = configuration_bytes + .checked_add(metadata) + .ok_or_else(invalid)?; + image_bytes = image_bytes.max(profile.images.memory_bound()); + } + if !settings.directory.is_absolute() + || settings.directory.as_os_str().len() > 4096 + || settings + .directory + .components() + .any(|p| matches!(p, std::path::Component::ParentDir)) + || !(1..=1_000_000).contains(&settings.max_effect_rows) + || !(1..=4_000_000).contains(&settings.max_pages) + || !(1..=68_719_476_736).contains(&settings.max_page_bytes) + || !(2..=4096).contains(&settings.max_retained_roots) + || !(1..=128).contains(&settings.peers.len()) + || settings.peer_revision == 0 + || settings.max_preparations == 0 + || settings.max_preparations > shared.limits.max_groups + || settings.max_parallel_preparations == 0 + || settings.max_parallel_preparations > settings.max_preparations.min(256) + || !(1..=65_536).contains(&settings.max_waiters) + { + return Err(invalid()); + } + let local = shared.runtime.log().limits(); + let mut nodes = std::collections::BTreeSet::new(); + let mut addresses = std::collections::BTreeSet::new(); + let mut labels = std::collections::BTreeSet::new(); + for peer in &mut settings.peers { + if peer.label.len() > 4096 + || peer.server_name.len() > 4096 + || !nodes.insert(peer.node) + || !labels.insert(peer.label.clone()) + || ![peer.control_address, peer.data_address, peer.query_address] + .into_iter() + .all(|a| addresses.insert(a)) + { + return Err(invalid()); + } + peer.validate_endpoint().map_err(|_| invalid())?; + peer.label = std::mem::take(&mut peer.label) + .into_boxed_str() + .into_string(); + peer.server_name = std::mem::take(&mut peer.server_name) + .into_boxed_str() + .into_string(); + } + if !settings + .peers + .iter() + .any(|p| p.node == local.node && p.disk == local.disk && p.process == local.process) + { + return Err(invalid()); + } + // Bound recovery pins, BTree inventory/leases, recursive page decoding, + // image scratch and all bounded catalog/peer/request metadata. The active + // roster retains at most max_groups stores of these configured sizes. + let mut bytes = ManifestPins::maximum_bytes(settings.manifest.max_cells) + .checked_add(settings.max_pages.checked_mul(256).ok_or_else(invalid)?) + .and_then(|n| { + n.checked_add( + settings.manifest.max_depth as usize + * crate::raft::cell_groups::page_wire::MAX_PAGE_BYTES, + ) + }) + .and_then(|n| n.checked_add(image_bytes)) + .and_then(|n| n.checked_add(2 * 1024 * 1024)) + .ok_or_else(invalid)?; + if let Some(archives) = &archives { + configuration_bytes = configuration_bytes + .checked_add(archives.metadata(&settings)?) + .ok_or_else(invalid)?; + // A preparation retains both serving and private verifier stores. + // Charge both complete bounds to the existing node-wide bank. + bytes = bytes.checked_mul(2).ok_or_else(invalid)?; + } + // Immutable namespace handles keep their advertised directory caches alive. + // Charge those once against the existing total, leaving one shared bank + // for every namespace's physical preparations. No per-namespace banks. + let available = settings + .max_preparation_bytes + .checked_sub(configuration_bytes) + .ok_or_else(invalid)?; + if available < bytes { + return Err(invalid()); + } + settings.directory = settings.directory.into_boxed_path().into_path_buf(); + settings.peers = settings.peers.into_boxed_slice().into_vec(); + settings.namespaces = settings.namespaces.into_boxed_slice().into_vec(); + Ok(Self { + preparations: Budget::new(available, settings.max_preparations), + waiters: Budget::new(settings.max_waiters * WAITER_BYTES, settings.max_waiters), + parallel: Arc::new(tokio::sync::Semaphore::new( + settings.max_parallel_preparations, + )), + bytes, + configuration_bytes, + parked: Mutex::new( + parked + .into_iter() + .map(|(applier, current)| { + ( + applier.history(), + PendingGroupOwner::Recovered(Arc::new(prepare::Prepared { + applier, + current, + })), + ) + }) + .collect(), + ), + archives, + settings, + }) + } + + pub fn images( + &self, + directory: &RangeDirectory, + ) -> Result { + self.profile(directory) + .map(|profile| profile.images.clone()) + } + + pub fn profile( + &self, + directory: &RangeDirectory, + ) -> Result<&CellGroupProvisionNamespace, CellGroupControllerError> { + self.settings + .namespaces + .iter() + .find(|profile| profile.directory.same_owner(directory)) + .ok_or(CellGroupControllerError::Invalid( + "unconfigured namespace owner", + )) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/provision/learner.rs b/crates/cellar/durability/raft/cell_groups/controller/provision/learner.rs new file mode 100644 index 000000000..41f191727 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/provision/learner.rs @@ -0,0 +1,378 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Managed learners keep one physical preparation through grant, install and +//! attachment. Neither a reservation nor a staged mapping can create genesis. +use super::*; +use crate::application::consensus::client::LearnerActivation; +use crate::raft::{ + cache::GroupLog, + cell_groups::CellGroupBirth, + fleet_catalog::{FleetHostAuthorization, FleetPermitAuthorization}, + member_state::{members, FleetPermitPhase}, + storage::LogError, + transport::{CellGroupSeedError, CellGroupSeedSession, GroupRoute as PhysicalRoute, RaftPeer}, +}; +use cellar_logic::{ + cell_group::GroupHistory, + consensus::{MemberId, NodeId, ReplicaIdentity}, +}; + +#[derive(Clone)] +pub(super) struct Admission { + authorization: FleetPermitAuthorization, + pub member: MemberId, + source: MemberId, + source_node: NodeId, + from_s3: bool, +} + +impl Admission { + pub fn new( + route: &GroupRoute, + host: &FleetHostAuthorization, + authorization: FleetPermitAuthorization, + activation: LearnerActivation, + ) -> Result { + let replacement = authorization + .permit() + .replacement() + .map_err(|e| catalog_error(CatalogError::Log(e)))?; + let boot = host + .boot_binding() + .map_err(CatalogError::from) + .map_err(catalog_error)?; + if !authorization.matches_scope(&route.history) + || authorization.permit().binding().fleet() != host.fleet() + || authorization.target_boot() != Some(&boot) + || replacement.member != activation.member + || replacement.node != host.node() + || replacement.disk != host.disk() + || activation.member == activation.source + || activation.source_node == host.node() + { + return Err(CellGroupControllerError::Invalid( + "managed learner admission", + )); + } + Ok(Self { + authorization, + member: activation.member, + source: activation.source, + source_node: activation.source_node, + from_s3: activation.from_s3, + }) + } + + pub(super) fn check_installed( + &self, + group: &GroupLog, + ) -> Result<(), CellGroupControllerError> { + let cache = group.read().map_err(RuntimeError::from)?; + let status = cache + .membership + .fleet_status() + .ok_or(CellGroupControllerError::Invalid( + "missing learner fleet guard", + ))?; + let permit = self.authorization.permit(); + if group.identity().member != self.member + || status.permit.as_ref() != Some(permit) + || &status.binding != permit.binding() + || status.consumed_sequence != permit.sequence() + || !matches!( + status.phase, + Some( + FleetPermitPhase::Seeding + | FleetPermitPhase::Joint + | FleetPermitPhase::Complete + ) + ) + { + return Err(CellGroupControllerError::Invalid( + "installed learner permit", + )); + } + Ok(()) + } +} + +pub(super) fn same_admission(a: Option<&Admission>, b: Option<&Admission>) -> bool { + match (a, b) { + (None, None) => true, + (Some(a), Some(b)) => { + // A source endpoint is a fresh-attempt hint, not destination + // ownership. Leader failure must not strand the exact retained + // boot/member/permit. A running attempt keeps its original hint; + // the next attempt reauthorizes its source quorum independently. + a.authorization.permit() == b.authorization.permit() + && a.authorization.target_boot() == b.authorization.target_boot() + && a.member == b.member + && a.from_s3 == b.from_s3 + } + _ => false, + } +} + +pub(super) struct Staged { + pub prepared: prepare::Prepared, + session: CellGroupSeedSession, + birth: CellGroupBirth, + config: Arc, + transport: RaftTransport, + admission: Admission, + _parallel: tokio::sync::OwnedSemaphorePermit, +} + +impl Staged { + pub async fn install(self) -> Result { + let (permit, current) = self + .session + .install(self.prepared.applier.clone()) + .await + .map_err(seed_error)?; + self.admission.check_installed(permit.group())?; + let route = physical_route( + &self.birth, + permit.group(), + &self.config, + &self.transport, + self.admission.authorization.minimum_trust_revision(), + )?; + Ok(CellGroupRegistration { + permit, + current, + applier: self.prepared.applier, + transport: route, + }) + } +} + +pub(super) struct Recovery<'a> { + pub directory: &'a RangeDirectory, + pub birth: &'a CellGroupBirth, + pub config: &'a Arc, + pub runtime: &'a CellGroupRuntime, + pub transport: &'a RaftTransport, + pub host: &'a FleetHostAuthorization, + pub admission: &'a Admission, + pub deadline: u64, +} + +impl Recovery<'_> { + pub(super) async fn recover( + self, + resources: prepare::Resources, + parallel: tokio::sync::OwnedSemaphorePermit, + ) -> Result { + let Self { + directory, + birth, + config, + runtime, + transport, + host, + admission, + deadline, + } = self; + let prepare::Resources { existing, archive } = resources; + let log = runtime.log(); + let installed = match log.cell_group(&birth.route().history) { + Ok(group) => group + .read() + .map_err(RuntimeError::from)? + .group_seed + .is_some(), + Err(LogError::NotFound) => false, + Err(error) => return Err(RuntimeError::from(error).into()), + }; + let images = config.images(directory)?; + if installed { + // A lost install response cannot authorize downloading/reinitializing a + // different cut. The durable receipt and permanent claim select restart. + let permit = directory + .bootstrap_managed_joined_group_member(birth, admission.member, log, host) + .await + .map_err(catalog_error)?; + admission.check_installed(permit.group())?; + let prepared = + prepare::recover_pages(config, runtime, permit.group().clone(), images, existing) + .await?; + let route = physical_route( + birth, + permit.group(), + config, + transport, + admission.authorization.minimum_trust_revision(), + )?; + return Ok(prepare::Recovery::Ready(CellGroupRegistration { + permit, + current: prepared.current, + applier: prepared.applier, + transport: route, + })); + } + let peer = transport + .managed_group_peer( + &birth.route().history, + admission.source_node, + admission.authorization.minimum_trust_revision(), + ) + .map_err(CellGroupControllerError::Transport)?; + let local = log.limits(); + let target = ReplicaIdentity { + scope: birth.route().history, + member: admission.member, + node: local.node, + disk: local.disk, + process: local.process, + }; + let source = RaftPeer { + replica: ReplicaIdentity { + scope: target.scope, + member: admission.source, + node: peer.node, + disk: peer.disk, + process: peer.process, + }, + label: peer.label, + server_name: peer.server_name, + control_address: peer.control_address, + data_address: peer.data_address, + query_address: peer.query_address, + }; + let session = if admission.from_s3 { + let archive = archive.ok_or(CellGroupControllerError::Invalid( + "S3 learner resources unavailable", + ))?; + transport + .prepare_managed_group_archive_seed( + target, + source, + directory.clone(), + birth.clone(), + deadline, + crate::raft::transport::ManagedArchiveSeed { + host: host.clone(), + authorization: admission.authorization.clone(), + archive, + }, + ) + .await + } else { + transport + .prepare_managed_group_seed( + target, + source, + directory.clone(), + birth.clone(), + deadline, + crate::raft::transport::ManagedPeerSeed { + host: host.clone(), + authorization: admission.authorization.clone(), + }, + ) + .await + } + .map_err(seed_error)?; + let prepared = + prepare::recover_pages(config, runtime, session.group().clone(), images, existing) + .await?; + Ok(prepare::Recovery::Seed(Staged { + prepared, + session, + birth: birth.clone(), + config: config.clone(), + transport: transport.clone(), + admission: admission.clone(), + _parallel: parallel, + })) + } +} + +fn physical_route( + birth: &CellGroupBirth, + group: &GroupLog, + config: &Config, + transport: &RaftTransport, + minimum_revision: u64, +) -> Result, CellGroupControllerError> { + let bindings = { + let cache = group.read().map_err(RuntimeError::from)?; + members(&cache.conf_state) + .into_iter() + .map(|id| { + let member = MemberId::new(id) + .map_err(|_| CellGroupControllerError::Invalid("member id"))?; + let binding = cache + .membership + .joined(member) + .map_err(RuntimeError::from)? + .map(|(b, _)| (b.node, b.disk)); + let node = binding + .map(|b| b.0) + .or_else(|| { + birth + .voters() + .iter() + .chain(birth.learners()) + .find(|b| b.member == member) + .map(|b| b.node) + }) + .ok_or(CellGroupControllerError::Invalid("member node binding"))?; + Ok((member, node, binding.map(|b| b.1))) + }) + .collect::, CellGroupControllerError>>()? + }; + // Resolve after the group read guard drops. Each endpoint is independently + // checked against rooted trust again by physical registration. + let mut peers = Vec::new(); + for (member, node, disk) in bindings { + let peer = + match transport.managed_group_peer(&birth.route().history, node, minimum_revision) { + Ok(peer) => peer, + Err(TransportError::StalePeer | TransportError::Refused) + if node != group.identity().node => + { + continue + } + Err(error) => return Err(CellGroupControllerError::Transport(error)), + }; + if disk.is_some_and(|disk| disk != peer.disk) { + return Err(CellGroupControllerError::Invalid( + "joined member disk changed", + )); + } + peers.push(RaftPeer { + replica: ReplicaIdentity { + scope: birth.route().history, + member, + node, + disk: peer.disk, + process: peer.process, + }, + label: peer.label, + server_name: peer.server_name, + control_address: peer.control_address, + data_address: peer.data_address, + query_address: peer.query_address, + }); + } + Ok(PhysicalRoute { + scope: birth.route().history, + schema_digest: birth.route().schema_digest, + revision: config.settings.peer_revision, + peers, + }) +} + +fn seed_error(error: CellGroupSeedError) -> CellGroupControllerError { + match error { + CellGroupSeedError::Transport(e) => CellGroupControllerError::Transport(e), + CellGroupSeedError::Catalog(e) => catalog_error(e), + CellGroupSeedError::Log(e) => RuntimeError::from(e).into(), + CellGroupSeedError::Checkpoint(e) => CellGroupControllerError::Checkpoint(Arc::new(e)), + CellGroupSeedError::Archive(e) => { + CellGroupControllerError::Archive(Arc::new(CellGroupArchiveControlError::Archive(*e))) + } + } +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/provision/manager.rs b/crates/cellar/durability/raft/cell_groups/controller/provision/manager.rs new file mode 100644 index 000000000..7048e464b --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/provision/manager.rs @@ -0,0 +1,428 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Retained preparations are retryable without reopening an engine-pinned store. +use super::*; +use crate::raft::transport::PhysicalRegistration; +use cellar_logic::cell_group::GroupHistory; +use futures_util::{future::BoxFuture, stream::FuturesUnordered, StreamExt}; + +struct Entry { + directory: RangeDirectory, + route: GroupRoute, + config: Arc, + authority: Authority, + reservation: registration::Reservation, + permit: Option, + prepared: Option, + archive: Option, + waiters: Vec, + running: bool, + active: bool, +} + +pub(in super::super) struct Recovered { + input: prepare::Recovery, + archive: Option, +} + +pub(in super::super) enum Completed { + Prepared( + GroupHistory, + Result, CellGroupControllerError>, + ), + Attached( + GroupHistory, + Result, + ), + Checked( + GroupHistory, + Result<(), CellGroupControllerError>, + Waiter, + Permit, + ), +} + +impl Completed { + pub(in super::super) fn group(&self) -> GroupHistory { + match self { + Self::Prepared(group, _) | Self::Attached(group, _) | Self::Checked(group, _, _, _) => { + *group + } + } + } +} + +#[derive(Default)] +pub(in super::super) struct Manager { + entries: BTreeMap, + jobs: FuturesUnordered>, +} + +impl Manager { + pub fn is_running(&self) -> bool { + !self.jobs.is_empty() + } + + pub async fn next(&mut self) -> Option { + self.jobs.next().await + } + + pub fn request(&mut self, request: Request, shared: &Shared, transport: &RaftTransport) { + let Request { + directory, + route, + config, + authority, + waiter, + } = request; + let group = route.history; + let ready = match shared.groups.contains(&group) { + Ok(ready) => ready, + Err(error) => { + waiter.finish(Err(error.into())); + return; + } + }; + if let Some(entry) = self.entries.get_mut(&group) { + if !entry.directory.same_owner(&directory) + || entry.route != route + || !entry.authority.same_owner(&authority) + { + waiter.finish(Err(CellGroupControllerError::Invalid( + "conflicting provision binding", + ))); + return; + } + if entry.active { + // This owner already attached the exact assignment. No network + // read or fresh quorum claim is implied by this idempotent reply. + if !ready { + waiter.finish(Err(RuntimeError::NotFound.into())); + return; + } + if matches!(authority, Authority::Initial) { + waiter.finish(Ok(())); + } else { + self.validate_attached(directory, route, config, authority, waiter, shared); + } + return; + } + entry.waiters.push(waiter); + if !entry.running { + // A completed attempt may be retried with a fresh invocation + // deadline. A still-entered attempt never extends its deadline. + entry.authority = authority; + self.launch(group, shared.runtime.clone(), transport.clone()); + } + return; + } + if ready && matches!(authority, Authority::Managed(_)) { + // Observe an existing controller-owned attachment without adopting + // its applier, roster reservation or physical route lifecycle. + self.validate_attached(directory, route, config, authority, waiter, shared); + return; + } + let admitted = (|| { + // Manual registrations cannot be adopted by a different lifecycle. + if ready { + return Err(RuntimeError::AlreadyAttached); + } + let permit = config + .preparations + .acquire(config.bytes) + .ok_or(RuntimeError::Overloaded)?; + let reservation = shared.groups.reserve(group)?; + Ok((permit, reservation)) + })(); + let (permit, reservation) = match admitted { + Ok(value) => value, + Err(error) => { + waiter.finish(Err(error.into())); + return; + } + }; + self.entries.insert( + group, + Entry { + directory, + route, + config, + authority, + reservation, + permit: Some(permit), + prepared: None, + archive: None, + waiters: vec![waiter], + running: false, + active: false, + }, + ); + self.launch(group, shared.runtime.clone(), transport.clone()); + } + + fn validate_attached( + &mut self, + directory: RangeDirectory, + route: GroupRoute, + config: Arc, + authority: Authority, + waiter: Waiter, + shared: &Shared, + ) { + let permit = config.profile(&directory).and_then(|_| { + config + .preparations + .acquire(config.bytes) + .ok_or(RuntimeError::Overloaded.into()) + }); + let permit = match permit { + Ok(permit) => permit, + Err(error) => { + waiter.finish(Err(error)); + return; + } + }; + let runtime = shared.runtime.clone(); + self.jobs.push(Box::pin(async move { + let result = prepare::validate(directory, &route, config, runtime, authority).await; + Completed::Checked(route.history, result, waiter, permit) + })); + } + + fn launch(&mut self, group: GroupHistory, runtime: CellGroupRuntime, transport: RaftTransport) { + let entry = self.entries.get_mut(&group).expect("provision entry"); + entry.running = true; + let directory = entry.directory.clone(); + let route = entry.route.clone(); + let config = entry.config.clone(); + let prepared = entry.prepared.clone(); + let authority = entry.authority.clone(); + let archive = entry.archive.clone(); + self.jobs.push(Box::pin(async move { + let result = async { + let archive = if let Some(archives) = &config.archives { + let _parallel = config + .parallel + .acquire() + .await + .map_err(|_| CellGroupControllerError::Closed)?; + let birth = prepare::resolve(&directory, &route, &runtime, &authority).await?; + let archive = archives + .prepare(&directory, &birth, &config.settings, &runtime, archive) + .await?; + authority.check_live(&runtime)?; + Some(archive) + } else { + None + }; + let input = prepare::recover( + directory, + route, + config, + runtime, + transport, + authority, + prepare::Resources { + existing: prepared, + archive: archive.clone(), + }, + ) + .await?; + Ok(Box::new(Recovered { input, archive })) + } + .await; + Completed::Prepared(group, result) + })); + } + + pub fn complete( + &mut self, + done: Completed, + shared: &Shared, + transport: &RaftTransport, + roster: &mut BTreeMap, + routes: &mut BTreeMap, + archives: &mut archive::Manager, + ) -> Result<(), CellGroupControllerError> { + match done { + Completed::Prepared(group, Ok(input)) => { + let Recovered { input, archive } = *input; + self.entries + .get_mut(&group) + .expect("prepared archive entry") + .archive = archive; + let input = match input { + prepare::Recovery::Seed(staged) => { + let entry = self.entries.get_mut(&group).expect("staged entry"); + // Save the physical owner before entering transfer. Even + // a lost/expired install reply must retain these pins. + entry.prepared = Some(staged.prepared.clone()); + if let Err(error) = entry.authority.check_live(&shared.runtime) { + return self.failed(group, error); + } + let archive = entry.archive.clone(); + self.jobs.push(Box::pin(async move { + Completed::Prepared( + group, + staged + .install() + .await + .map(|registration| Recovered { + input: prepare::Recovery::Ready(registration), + archive, + }) + .map(Box::new), + ) + })); + return Ok(()); + } + prepare::Recovery::Ready(input) => input, + }; + let entry = self.entries.get_mut(&group).expect("prepared entry"); + // Save the owners before polling attachment: early Raft events + // must see the applier, and a catalog refusal must retain pins. + entry.prepared = Some(prepare::Prepared { + applier: input.applier.clone(), + current: input.current.clone(), + }); + if let Err(error) = entry.authority.check_live(&shared.runtime) { + return self.failed(group, error); + } + roster.insert(group, input.applier.clone()); + let runtime = shared.runtime.clone(); + let transport = transport.clone(); + self.jobs.push(Box::pin(async move { + Completed::Attached( + group, + registration::attach(input, runtime, transport).await, + ) + })); + Ok(()) + } + Completed::Prepared(group, Err(error)) => self.failed(group, error), + Completed::Attached(group, result) => match result { + Ok(guard) => { + let entry = self.entries.get_mut(&group).expect("attached entry"); + if let Err(error) = entry.reservation.activate() { + // Quarantine may have raced an early runtime event. Keep + // the route and all owners until the joined shutdown. + routes.insert(group, guard); + return self.failed(group, error.into()); + } + routes.insert(group, guard); + if let Some(archive) = entry.archive.take() { + // Transfer the prepared owner directly into the actor's + // bounded archive lane. Never call and await this same + // controller's public registration queue from its actor. + archives.provision(archive, shared)?; + } + entry.active = true; + entry.running = false; + entry.prepared = None; + entry.permit = None; + // Attachment can have finished after the caller expired. + // Keep the real active slot/route, but suppress late success. + let result = entry.authority.check_live(&shared.runtime); + for waiter in std::mem::take(&mut entry.waiters) { + waiter.finish(result.clone()); + } + Ok(()) + } + Err(error) => self.failed(group, error), + }, + Completed::Checked(group, result, waiter, _permit) => { + let result = result.and_then(|()| { + if shared.groups.contains(&group)? { + Ok(()) + } else { + Err(RuntimeError::NotFound.into()) + } + }); + let fatal = result + .as_ref() + .err() + .filter(|error| failure::node_wide(error)) + .cloned(); + waiter.finish(result); + match fatal { + Some(error) => Err(error), + None => Ok(()), + } + } + } + } + + fn failed( + &mut self, + group: GroupHistory, + error: CellGroupControllerError, + ) -> Result<(), CellGroupControllerError> { + let entry = self.entries.get_mut(&group).expect("failed entry"); + entry.running = false; + let waiters = std::mem::take(&mut entry.waiters); + // Before recovery installs a physical owner, a failed attempt can give + // back its preparation/roster capacity. Immutable birth claims persist. + if entry.prepared.is_none() { + self.entries.remove(&group); + } + for waiter in waiters { + waiter.finish(Err(error.clone())); + } + // Registration refusal happens before any transport submission. An + // identity/capacity refusal is retryable without stopping healthy groups; + // an actual closed service or uncertain engine/runtime still is fatal. + let uncertain = match &error { + CellGroupControllerError::Transport(TransportError::Runtime(error)) => { + failure::node_wide(&CellGroupControllerError::Runtime(error.clone())) + || matches!(error, RuntimeError::AlreadyAttached) + } + CellGroupControllerError::Transport(error) => matches!(error, TransportError::Closed), + // Recovery checks the engine marker again after installing its pins. + // A competing selection cannot safely reopen the same locked files. + CellGroupControllerError::Apply(error) + if matches!( + error.as_ref(), + ExecutionError::Invalid("recovery selection changed") + ) => + { + true + } + _ => { + failure::node_wide(&error) + || matches!( + error, + CellGroupControllerError::Runtime(RuntimeError::AlreadyAttached) + ) + } + }; + if uncertain { + return Err(error); + } + Ok(()) + } + + pub async fn join(&mut self) { + // Do not launch attachment after close, but do join every entered + // blocking recovery and attachment before releasing dependencies. + while let Some(done) = self.jobs.next().await { + if let Completed::Prepared(group, Ok(input)) = done { + let Recovered { input, archive } = *input; + let entry = self.entries.get_mut(&group).expect("closing entry"); + entry.archive = archive; + let prepared = match input { + prepare::Recovery::Ready(input) => prepare::Prepared { + applier: input.applier, + current: input.current, + }, + prepare::Recovery::Seed(staged) => staged.prepared, + }; + entry.prepared = Some(prepared); + } + } + for entry in self.entries.values_mut() { + for waiter in std::mem::take(&mut entry.waiters) { + waiter.finish(Err(CellGroupControllerError::Closed)); + } + } + } +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/provision/mod.rs b/crates/cellar/durability/raft/cell_groups/controller/provision/mod.rs new file mode 100644 index 000000000..60e1f7e09 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/provision/mod.rs @@ -0,0 +1,338 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! One preparation per physical group, owned and joined by the node controller. +mod config; +mod learner; +mod manager; +mod prepare; +mod recovery; +mod transfer; +pub(super) use config::Config; +pub use config::{CellGroupProvisionLimits, CellGroupProvisionNamespace, CellGroupProvisioning}; +pub(super) use manager::Manager; + +use super::{calls::Delivery, *}; +use crate::raft::{ + budget::Permit, + catalog::CatalogError, + cell_groups::{PreparedCellGroupTransferChild, PreparedGroupManifest, RangeDirectory}, +}; +use cellar_logic::cell_group::GroupRoute; +use tokio::sync::oneshot; + +// Logical routes have fixed-width bounded keys. Directory handles share their +// separately bounded cache; no tree or cell list is copied into a request. +const WAITER_BYTES: usize = 4096; + +fn apply_error(error: ExecutionError) -> CellGroupControllerError { + CellGroupControllerError::Apply(Arc::new(error)) +} + +fn catalog_error(error: CatalogError) -> CellGroupControllerError { + CellGroupControllerError::Catalog(Arc::new(error)) +} + +/// Response loss never cancels physical recovery or attachment. Success means +/// locally attached, not elected, caught up, or authorized to serve a request. +pub struct CellGroupProvisionCall { + receiver: oneshot::Receiver>, + deadline: Option<(crate::asyncrt::DomainToken, u64)>, +} + +impl CellGroupProvisionCall { + pub async fn response(self) -> Result<(), Arc> { + let delivery = self + .receiver + .await + .map_err(|_| Arc::new(CellGroupControllerError::Closed))?; + delivery.result?; + check_delivery_deadline(&self.deadline).map_err(Arc::new) + } +} + +pub(super) struct Waiter { + reply: oneshot::Sender>, + permit: Arc, + deadline: Option<(crate::asyncrt::DomainToken, u64)>, +} + +impl Waiter { + fn finish(self, result: Result<(), CellGroupControllerError>) { + let result = result.and_then(|()| check_delivery_deadline(&self.deadline)); + let _ = self.reply.send(Delivery { + result: result.map_err(Arc::new), + _permit: self.permit, + }); + } +} + +#[derive(Clone)] +enum PendingGroupOwner { + Recovered(Arc), + Seeded(Arc), + Synced(Arc), +} + +async fn recover_pending_owner( + log: &crate::raft::storage::ReplicaLog, + applier: CellGroupApplier, +) -> Result { + log.domain() + .blocking(move || { + let current = applier.recover_selected()?; + Ok(prepare::Prepared { applier, current }) + }) + .map_err(|_| CellGroupControllerError::Closed)? + .await + .map_err(|_| CellGroupControllerError::Closed)? + .map_err(apply_error) +} + +fn check_delivery_deadline( + deadline: &Option<(crate::asyncrt::DomainToken, u64)>, +) -> Result<(), CellGroupControllerError> { + if let Some((domain, deadline)) = deadline { + if domain + .mono_ms() + .map_err(|_| CellGroupControllerError::Closed)? + >= *deadline + { + return Err(RuntimeError::DeadlineExceeded.into()); + } + } + Ok(()) +} + +pub(super) struct Request { + directory: RangeDirectory, + route: GroupRoute, + config: Arc, + authority: Authority, + waiter: Waiter, +} + +#[derive(Clone)] +enum Authority { + Initial, + Managed(Box), +} + +#[derive(Clone)] +struct ManagedAuthority { + learner: Option, + host: crate::raft::fleet_catalog::FleetHostAuthorization, + deadline: u64, +} + +impl Authority { + fn managed(&self) -> Option<&ManagedAuthority> { + match self { + Self::Initial => None, + Self::Managed(authority) => Some(authority), + } + } + + fn same_owner(&self, other: &Self) -> bool { + match (self, other) { + (Self::Initial, Self::Initial) => true, + (Self::Managed(a), Self::Managed(b)) => { + a.host.same_boot_owner(&b.host) + && learner::same_admission(a.learner.as_ref(), b.learner.as_ref()) + } + _ => false, + } + } + + fn check_live(&self, runtime: &CellGroupRuntime) -> Result<(), CellGroupControllerError> { + if let Some(ManagedAuthority { host, deadline, .. }) = self.managed() { + let now = runtime + .log() + .domain() + .mono_ms() + .map_err(|_| CellGroupControllerError::Closed)?; + let remaining = deadline + .checked_sub(now) + .filter(|ms| *ms > 0) + .ok_or(RuntimeError::DeadlineExceeded)?; + if remaining > 60_000 { + return Err(CellGroupControllerError::Invalid( + "group startup deadline bound", + )); + } + let boot = host + .boot_binding() + .map_err(CatalogError::from) + .map_err(catalog_error)?; + let log = runtime.log(); + if boot.node != log.limits().node.bytes() + || boot.disk != log.limits().disk.bytes() + || boot.process != log.limits().process.bytes() + || boot.engine + != log + .storage_instance() + .map_err(|e| catalog_error(CatalogError::Log(e)))? + .bytes() + || log + .lifecycle_guard() + .map_err(|e| catalog_error(CatalogError::Log(e)))? + .is_none_or(|guard| !guard.admits_group_start(&boot)) + { + return Err(catalog_error(CatalogError::Fenced)); + } + } + Ok(()) + } +} + +impl CellGroupController { + /// Install immutable node resources once. Reconfiguration requires a joined + /// restart so pending recovery cannot change filesystem or image owners. + pub fn configure_provisioning( + &self, + settings: CellGroupProvisioning, + ) -> Result<(), CellGroupControllerError> { + self.configure_provisioning_with_archives(settings, None, Vec::new()) + } + + pub(crate) fn configure_provisioning_with_archives( + &self, + settings: CellGroupProvisioning, + archives: Option, + parked: Vec<(CellGroupApplier, PreparedGroupManifest)>, + ) -> Result<(), CellGroupControllerError> { + let shared = &self.handle.0; + let config = Arc::new(Config::new(settings, shared, archives, parked)?); + let mut slot = shared + .provisioning + .lock() + .map_err(|_| CellGroupControllerError::Closed)?; + if shared.closing.load(Ordering::Acquire) { + return Err(CellGroupControllerError::Closed); + } + if slot.is_some() { + return Err(CellGroupControllerError::Invalid( + "provisioning already configured", + )); + } + *slot = Some(config); + Ok(()) + } + + /// Materialize and recover an already selected initial-member assignment. + /// Concurrent requests coalesce only for the same directory owner and exact + /// route. This never allocates a range or a Raft group for an individual cell. + pub fn provision( + &self, + directory: RangeDirectory, + route: GroupRoute, + ) -> Result { + self.provision_authorized(directory, route, Authority::Initial) + } + + /// Initial members require their exact managed boot and fresh rooted birth. + /// The caller's deadline controls admission and delivery; accepted physical + /// work remains owned and joined after response loss or deadline expiry. + pub(crate) fn provision_managed_initial( + &self, + directory: RangeDirectory, + route: GroupRoute, + host: crate::raft::fleet_catalog::FleetHostAuthorization, + deadline: u64, + ) -> Result { + self.provision_authorized( + directory, + route, + Authority::Managed(Box::new(ManagedAuthority { + host, + deadline, + learner: None, + })), + ) + } + + /// A learner must retain the exact repair admission and complete authenticated + /// seed. This path never creates genesis from a placement reservation. + pub(crate) fn provision_managed_learner( + &self, + directory: RangeDirectory, + route: GroupRoute, + host: crate::raft::fleet_catalog::FleetHostAuthorization, + authorization: crate::raft::fleet_catalog::FleetPermitAuthorization, + activation: crate::application::consensus::client::LearnerActivation, + deadline: u64, + ) -> Result { + if activation.from_s3 { + let configured = self + .handle + .0 + .provisioning + .lock() + .map_err(|_| CellGroupControllerError::Closed)?; + if configured.as_ref().is_none_or(|c| c.archives.is_none()) { + return Err(CellGroupControllerError::Invalid( + "S3 learner resources unavailable", + )); + } + } + let learner = learner::Admission::new(&route, &host, authorization, activation)?; + self.provision_authorized( + directory, + route, + Authority::Managed(Box::new(ManagedAuthority { + host, + deadline, + learner: Some(learner), + })), + ) + } + + fn provision_authorized( + &self, + directory: RangeDirectory, + route: GroupRoute, + authority: Authority, + ) -> Result { + let shared = &self.handle.0; + if shared.closing.load(Ordering::Acquire) { + return Err(CellGroupControllerError::Closed); + } + authority.check_live(&shared.runtime)?; + let config = shared + .provisioning + .lock() + .map_err(|_| CellGroupControllerError::Closed)? + .clone() + .ok_or(CellGroupControllerError::Invalid( + "provisioning not configured", + ))?; + let permit = config + .waiters + .acquire(WAITER_BYTES) + .ok_or(RuntimeError::Overloaded)?; + let (reply, receiver) = oneshot::channel(); + let deadline = match &authority { + Authority::Initial => None, + Authority::Managed(authority) => Some((shared.domain.clone(), authority.deadline)), + }; + shared + .requests + .try_send(requests::Request::Provision(Box::new(Request { + directory, + route, + config, + authority, + waiter: Waiter { + reply, + permit: Arc::new(permit), + deadline: deadline.clone(), + }, + }))) + .map_err(|e| match e { + tokio::sync::mpsc::error::TrySendError::Closed(_) => { + CellGroupControllerError::Closed + } + tokio::sync::mpsc::error::TrySendError::Full(_) => RuntimeError::Overloaded.into(), + })?; + Ok(CellGroupProvisionCall { receiver, deadline }) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/provision/prepare.rs b/crates/cellar/durability/raft/cell_groups/controller/provision/prepare.rs new file mode 100644 index 000000000..d9334b50d --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/provision/prepare.rs @@ -0,0 +1,272 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use crate::raft::cell_groups::{ManifestPageStore, PreparedGroupManifest}; +use crate::raft::transport::{GroupRoute as PhysicalRoute, RaftPeer}; +use cellar_logic::consensus::ReplicaIdentity; + +#[derive(Clone)] +pub(super) struct Prepared { + pub applier: CellGroupApplier, + pub current: PreparedGroupManifest, +} + +pub(super) struct Resources { + pub existing: Option, + pub archive: Option, +} + +pub(super) enum Recovery { + Ready(CellGroupRegistration), + Seed(learner::Staged), +} + +pub(super) async fn recover( + directory: RangeDirectory, + route: GroupRoute, + config: Arc, + runtime: CellGroupRuntime, + transport: RaftTransport, + authority: Authority, + resources: Resources, +) -> Result { + let Resources { existing, archive } = resources; + authority.check_live(&runtime)?; + let images = config.images(&directory)?; + let _parallel = config + .parallel + .clone() + .acquire_owned() + .await + .map_err(|_| CellGroupControllerError::Closed)?; + let birth = resolve(&directory, &route, &runtime, &authority).await?; + if let Some(ManagedAuthority { + host, + deadline, + learner: Some(admission), + }) = authority.managed() + { + return learner::Recovery { + directory: &directory, + birth: &birth, + config: &config, + runtime: &runtime, + transport: &transport, + host, + admission, + deadline: *deadline, + } + .recover(Resources { existing, archive }, _parallel) + .await; + } + let physical = transport_route( + &config, + &transport, + &route, + birth.voters().iter().chain(birth.learners()), + authority.managed().is_some(), + )?; + let permit = bootstrap(&directory, &birth, &runtime, &authority).await?; + let prepared = + recover_pages(&config, &runtime, permit.group().clone(), images, existing).await?; + Ok(Recovery::Ready(CellGroupRegistration { + permit, + current: prepared.current, + applier: prepared.applier, + transport: physical, + })) +} + +pub(super) fn transport_route<'a>( + config: &Config, + transport: &RaftTransport, + route: &GroupRoute, + members: impl Iterator, + managed: bool, +) -> Result, CellGroupControllerError> { + // Check complete trusted placement before creating genesis or page files. + let mut peers = Vec::new(); + for member in members { + let discovered; + let node = if managed { + discovered = transport + .observed_peer(member.node) + .map_err(CellGroupControllerError::Transport)?; + discovered + .as_ref() + .ok_or(CellGroupControllerError::Invalid( + "undiscovered managed birth member", + ))? + } else { + config + .settings + .peers + .iter() + .find(|p| p.node == member.node) + .ok_or(CellGroupControllerError::Invalid( + "unconfigured birth member", + ))? + }; + peers.push(RaftPeer { + replica: ReplicaIdentity { + scope: route.history, + member: member.member, + node: node.node, + disk: node.disk, + process: node.process, + }, + label: node.label.clone(), + server_name: node.server_name.clone(), + control_address: node.control_address, + data_address: node.data_address, + query_address: node.query_address, + }); + } + Ok(PhysicalRoute { + scope: route.history, + schema_digest: route.schema_digest, + revision: config.settings.peer_revision, + peers, + }) +} + +pub(super) async fn resolve( + directory: &RangeDirectory, + route: &GroupRoute, + runtime: &CellGroupRuntime, + authority: &Authority, +) -> Result { + authority.check_live(runtime)?; + if let Some(ManagedAuthority { host, .. }) = authority.managed() { + let boot = host + .boot_binding() + .map_err(CatalogError::from) + .map_err(catalog_error)?; + let owner = directory + .namespace() + .cell_group_owner() + .await + .map_err(catalog_error)?; + if owner.is_none_or(|o| o.fleet != host.fleet() || o.catalog != boot.catalog) { + return Err(catalog_error(CatalogError::ManagedAdmissionRequired)); + } + authority.check_live(runtime)?; + } + let birth = match authority { + Authority::Initial => directory.materialize_selected_group(route).await, + Authority::Managed(_) => directory.materialize_selected_history(route.history).await, + } + .map_err(catalog_error)?; + if birth.route() != route { + return Err(CellGroupControllerError::Invalid("provision birth route")); + } + authority.check_live(runtime)?; + Ok(birth) +} + +async fn bootstrap( + directory: &RangeDirectory, + birth: &crate::raft::cell_groups::CellGroupBirth, + runtime: &CellGroupRuntime, + authority: &Authority, +) -> Result { + authority.check_live(runtime)?; + let permit = match authority.managed() { + None => directory.bootstrap_group_member(birth, runtime.log()).await, + Some(ManagedAuthority { + host, + learner: Some(admission), + .. + }) => { + directory + .bootstrap_managed_joined_group_member(birth, admission.member, runtime.log(), host) + .await + } + Some(ManagedAuthority { + host, + learner: None, + .. + }) => { + directory + .bootstrap_managed_group_member(birth, runtime.log(), host) + .await + } + } + .map_err(catalog_error)?; + authority.check_live(runtime)?; + Ok(permit) +} + +pub(super) async fn recover_pages( + config: &Arc, + runtime: &CellGroupRuntime, + group: crate::raft::cache::GroupLog, + images: crate::raft::cell_groups::CellGroupApplyImages, + existing: Option, +) -> Result { + let config = config.clone(); + runtime + .log() + .domain() + .blocking(move || { + if let Some(mut prepared) = existing { + // Reuse the retained complete closure only while its exact + // selection/owners are live and publication is settled. A lost + // response with a changed selection recovers on this same owner. + prepared.current = prepared + .applier + .recover_or_reuse_selected(&prepared.current)?; + return Ok(prepared); + } + let settings = &config.settings; + let path = + ManifestPageStore::group_pages_path(&settings.directory, group.identity().scope); + let pages = ManifestPageStore::open( + &path, + settings.filesystem.clone(), + settings.max_pages, + settings.max_page_bytes, + settings.max_retained_roots, + ) + .map_err(crate::raft::execution::storage)?; + let (applier, current) = CellGroupApplier::recover( + group, + pages, + images, + settings.manifest, + settings.entries, + settings.max_effect_rows, + )?; + Ok::<_, ExecutionError>(Prepared { applier, current }) + }) + .map_err(|_| CellGroupControllerError::Closed)? + .await + .map_err(|_| CellGroupControllerError::Closed)? + .map_err(apply_error) +} + +/// An attached retry rechecks catalog/boot authority without opening a second +/// page owner or trying to attach its already claimed consensus slot again. +pub(super) async fn validate( + directory: RangeDirectory, + route: &GroupRoute, + config: Arc, + runtime: CellGroupRuntime, + authority: Authority, +) -> Result<(), CellGroupControllerError> { + let _parallel = config + .parallel + .acquire() + .await + .map_err(|_| CellGroupControllerError::Closed)?; + let birth = resolve(&directory, route, &runtime, &authority).await?; + let permit = bootstrap(&directory, &birth, &runtime, &authority).await?; + if let Some(ManagedAuthority { + learner: Some(admission), + .. + }) = authority.managed() + { + admission.check_installed(permit.group())?; + } + Ok(()) +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/provision/recovery.rs b/crates/cellar/durability/raft/cell_groups/controller/provision/recovery.rs new file mode 100644 index 000000000..8a032d427 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/provision/recovery.rs @@ -0,0 +1,144 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Reopen the published recovery voter's own synchronized page and image stores. +use super::*; +use crate::raft::fleet_catalog::{FleetHostAuthorization, FleetRecoveryAttachRequest}; +use cellar_logic::cell_group::GroupHistory; + +impl CellGroupController { + pub(crate) fn retain_managed_recovery_seed_owner( + &self, + applier: CellGroupApplier, + ) -> Result<(), CellGroupControllerError> { + let shared = &self.handle.0; + let config = shared + .provisioning + .lock() + .map_err(|_| CellGroupControllerError::Closed)? + .clone() + .ok_or(CellGroupControllerError::Invalid( + "recovery provisioning owner", + ))?; + let history = applier.history(); + let mut parked = config + .parked + .lock() + .map_err(|_| CellGroupControllerError::Closed)?; + // The engine pins this page owner through attachment and any lost + // receipt confirmation. An attached runtime already owns it. + if !shared.groups.contains(&history)? { + parked + .entry(history) + .or_insert_with(|| PendingGroupOwner::Synced(Arc::new(applier))); + } + Ok(()) + } + + pub(crate) async fn attach_managed_recovery_voter( + &self, + directory: &RangeDirectory, + request: &FleetRecoveryAttachRequest, + history: GroupHistory, + host: &FleetHostAuthorization, + transport: &RaftTransport, + control: (u64, &tokio::sync::watch::Receiver), + ) -> Result<(), CellGroupControllerError> { + let (deadline, stop) = control; + let shared = &self.handle.0; + let check_live = || { + if *stop.borrow() || shared.closing.load(Ordering::Acquire) { + return Err(CellGroupControllerError::Closed); + } + check_delivery_deadline(&Some((shared.runtime.log().domain(), deadline))) + }; + check_live()?; + let config = shared + .provisioning + .lock() + .map_err(|_| CellGroupControllerError::Closed)? + .clone() + .ok_or(CellGroupControllerError::Invalid( + "recovery provisioning owner", + ))?; + let images = config.images(directory)?; + let permit = directory + .bootstrap_managed_pending_recovery_voter( + request, + history, + shared.runtime.log(), + host, + deadline, + ) + .await + .map_err(catalog_error)?; + check_live()?; + if shared.groups.contains(&history)? { + return Ok(()); + } + let _preparation = config + .preparations + .acquire(config.bytes) + .ok_or(RuntimeError::Overloaded)?; + let _parallel = config + .parallel + .clone() + .try_acquire_owned() + .map_err(|_| RuntimeError::Overloaded)?; + let physical = prepare::transport_route( + &config, + transport, + permit.route(), + permit.initial_voters().iter(), + true, + )?; + let owner = config + .parked + .lock() + .map_err(|_| CellGroupControllerError::Closed)? + .get(&history) + .cloned(); + let prepared = match owner { + Some(PendingGroupOwner::Synced(applier)) => { + recover_pending_owner(shared.runtime.log(), applier.as_ref().clone()).await? + } + Some(PendingGroupOwner::Recovered(owner)) => { + prepare::recover_pages( + &config, + &shared.runtime, + permit.group().clone(), + images, + Some(owner.as_ref().clone()), + ) + .await? + } + Some(PendingGroupOwner::Seeded(_)) => return Err(catalog_error(CatalogError::Fenced)), + None => { + prepare::recover_pages( + &config, + &shared.runtime, + permit.group().clone(), + images, + None, + ) + .await? + } + }; + check_live()?; + self.register(CellGroupRegistration { + permit, + current: prepared.current, + applier: prepared.applier, + transport: physical, + }) + .map_err(|error| error.error)? + .response() + .await + .map_err(|error| error.as_ref().clone())?; + config + .parked + .lock() + .map_err(|_| CellGroupControllerError::Closed)? + .remove(&history); + Ok(()) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/provision/transfer.rs b/crates/cellar/durability/raft/cell_groups/controller/provision/transfer.rs new file mode 100644 index 000000000..82b4ad1d3 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/provision/transfer.rs @@ -0,0 +1,329 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Prepare exactly one managed child in the destination's retained stores. +use super::*; +use crate::raft::{ + cell_groups::{ + CellGroupApplyImages, CellGroupArchiveCoverage, CellGroupArchiveStore, + CellGroupRecoveryPool, CellGroupTransferTarget, InstalledTransferVoter, ManifestPageStore, + PreparedCellGroupTransferChild, PreparedRecoveryData, TransitionPhase, + VerifiedCellGroupTransitionSources, + }, + fleet_catalog::{FleetHostAuthorization, ReopenedManagedCellGroupTransition}, +}; +use std::time::Duration; + +impl Config { + async fn open_import_target( + self: &Arc, + log: &crate::raft::storage::ReplicaLog, + images: CellGroupApplyImages, + route: &GroupRoute, + spools: &'static str, + ) -> Result { + let domain = log.domain(); + let target_route = route.clone(); + let scratch = log.snapshot_scratch(); + let staging = self.clone(); + domain + .blocking(move || { + let settings = &staging.settings; + let directory = settings.directory.join(spools); + cellar_ltx::host::create_dir_all_durable(settings.filesystem.as_ref(), &directory) + .map_err(crate::raft::execution::storage)?; + let path = + ManifestPageStore::group_pages_path(&settings.directory, target_route.history); + let pages = ManifestPageStore::open( + &path, + settings.filesystem.clone(), + settings.max_pages, + settings.max_page_bytes, + settings.max_retained_roots, + ) + .map_err(crate::raft::execution::storage)?; + let store = CellGroupArchiveStore::new( + target_route, + pages, + images, + settings.manifest, + settings.entries, + settings.max_effect_rows, + ) + .map_err(crate::raft::execution::storage)?; + Ok::<_, ExecutionError>(CellGroupTransferTarget { + store, + scratch, + directory, + }) + }) + .map_err(|_| CellGroupControllerError::Closed)? + .await + .map_err(|_| CellGroupControllerError::Closed)? + .map_err(apply_error) + } + + fn retain_synced_transfer( + &self, + log: &crate::raft::storage::ReplicaLog, + child: &PreparedCellGroupTransferChild, + ) -> Result<(), CellGroupControllerError> { + let history = child.route().history; + let group = log.cell_group(&history).map_err(RuntimeError::from)?; + let applier = child.installed_applier_on(&group)?; + self.parked + .lock() + .map_err(|_| CellGroupControllerError::Closed)? + .insert(history, PendingGroupOwner::Synced(Arc::new(applier))); + Ok(()) + } +} + +impl CellGroupController { + pub(crate) async fn prepare_managed_recovery_seed( + &self, + directory: &RangeDirectory, + route: &GroupRoute, + source: CellGroupArchiveCoverage, + identity: ([u8; 32], cellar_logic::consensus::ProposalId), + recovery: &CellGroupRecoveryPool, + ) -> Result { + let shared = &self.handle.0; + let config = shared + .provisioning + .lock() + .map_err(|_| CellGroupControllerError::Closed)? + .clone() + .ok_or(CellGroupControllerError::Invalid( + "recovery provisioning owner", + ))?; + let images = config.images(directory)?; + let _preparation = config + .preparations + .acquire(config.bytes) + .ok_or(RuntimeError::Overloaded)?; + let _parallel = config + .parallel + .clone() + .try_acquire_owned() + .map_err(|_| RuntimeError::Overloaded)?; + let log = shared.runtime.log(); + let target = config + .open_import_target(log, images, route, "recovery-spools") + .await?; + Box::pin(recovery.prepare_cut(&log.domain(), source, target, identity.0, identity.1)) + .await + .map_err(|error| CellGroupControllerError::Archive(Arc::new(error.into()))) + } + + pub(crate) async fn attach_managed_transfer_child( + &self, + directory: &RangeDirectory, + transition: Arc, + route: &GroupRoute, + host: &FleetHostAuthorization, + transport: &RaftTransport, + control: (u64, &tokio::sync::watch::Receiver), + ) -> Result<(), CellGroupControllerError> { + let (deadline, stop) = control; + let shared = &self.handle.0; + let log = shared.runtime.log(); + let config = shared + .provisioning + .lock() + .map_err(|_| CellGroupControllerError::Closed)? + .clone() + .ok_or(CellGroupControllerError::Invalid( + "transfer provisioning owner", + ))?; + config.images(directory)?; + let permit = directory + .bootstrap_managed_pending_transfer_voter(transition, route, log, host, deadline) + .await + .map_err(catalog_error)?; + if shared.groups.contains(&route.history)? { + return Ok(()); + } + let physical = prepare::transport_route( + &config, + transport, + route, + permit.initial_voters().iter(), + true, + )?; + let owner = config + .parked + .lock() + .map_err(|_| CellGroupControllerError::Closed)? + .get(&route.history) + .cloned() + .ok_or_else(|| catalog_error(CatalogError::LostStableState))?; + let prepared = match owner { + PendingGroupOwner::Recovered(owner) => owner.as_ref().clone(), + PendingGroupOwner::Synced(applier) => { + recover_pending_owner(log, applier.as_ref().clone()).await? + } + PendingGroupOwner::Seeded(child) => { + recover_pending_owner(log, child.installed_applier(&permit)?).await? + } + }; + if *stop.borrow() || shared.closing.load(Ordering::Acquire) { + return Err(CellGroupControllerError::Closed); + } + self.register(CellGroupRegistration { + permit, + current: prepared.current, + applier: prepared.applier, + transport: physical, + }) + .map_err(|error| error.error)? + .response() + .await + .map_err(|error| error.as_ref().clone())?; + // The roster owns this page store after registration joins. Release + // the separate genesis reference so it cannot pin superseded images. + config + .parked + .lock() + .map_err(|_| CellGroupControllerError::Closed)? + .remove(&route.history); + Ok(()) + } + + pub(crate) async fn install_managed_transfer_child( + &self, + directory: &RangeDirectory, + transfer: ( + &ReopenedManagedCellGroupTransition, + &VerifiedCellGroupTransitionSources, + ), + route: &GroupRoute, + host: &FleetHostAuthorization, + recovery: &CellGroupRecoveryPool, + control: (u64, &tokio::sync::watch::Receiver), + ) -> Result { + let (transition, sources) = transfer; + let (deadline, stop) = control; + let shared = &self.handle.0; + let log = shared.runtime.log(); + let domain = log.domain(); + let remaining = || { + if *stop.borrow() || shared.closing.load(Ordering::Acquire) { + return Err(CellGroupControllerError::Closed); + } + let now = domain + .mono_ms() + .map_err(|_| CellGroupControllerError::Closed)?; + deadline + .checked_sub(now) + .filter(|ms| *ms > 0) + .map(Duration::from_millis) + .ok_or_else(|| RuntimeError::DeadlineExceeded.into()) + }; + let config = shared + .provisioning + .lock() + .map_err(|_| CellGroupControllerError::Closed)? + .clone() + .ok_or(CellGroupControllerError::Invalid( + "transfer provisioning owner", + ))?; + let images = config.images(directory)?; + let _preparation = config + .preparations + .acquire(config.bytes) + .ok_or(RuntimeError::Overloaded)?; + let _parallel = config + .parallel + .clone() + .try_acquire_owned() + .map_err(|_| RuntimeError::Overloaded)?; + let authority = host + .managed_transfer_install_authority(transition, directory, route, remaining()?) + .await + .map_err(CatalogError::from) + .map_err(catalog_error)?; + let selected = directory + .managed_transition_status(&transition.plan, &authority.transition) + .await + .map_err(catalog_error)? + .ok_or_else(|| catalog_error(CatalogError::Fenced))?; + if selected.phase() == TransitionPhase::Retired + || selected.sealed_sources() != sources.journal().sealed_sources() + { + return Err(catalog_error(CatalogError::Fenced)); + } + let previous = config + .parked + .lock() + .map_err(|_| CellGroupControllerError::Closed)? + .get(&route.history) + .cloned(); + let recovered = match &previous { + Some(PendingGroupOwner::Recovered(owner)) => Some(&owner.current), + _ => None, + }; + if let Some(receipt) = directory + .reconcile_managed_transfer_voter(transition, route, log, host, deadline, recovered) + .await + .map_err(catalog_error)? + { + if let Some(PendingGroupOwner::Seeded(prepared)) = &previous { + config.retain_synced_transfer(log, prepared)?; + } + return Ok(receipt); + } + match previous { + Some(PendingGroupOwner::Recovered(_) | PendingGroupOwner::Synced(_)) => { + return Err(catalog_error(CatalogError::LostStableState)) + } + Some(PendingGroupOwner::Seeded(prepared)) => { + let result = directory + .install_managed_transfer_voter(transition, &prepared, log, host, remaining()?) + .await + .map_err(catalog_error); + let owner = config.retain_synced_transfer(log, &prepared); + let receipt = result?; + owner?; + return Ok(receipt); + } + None => (), + } + if config + .parked + .lock() + .map_err(|_| CellGroupControllerError::Closed)? + .len() + >= shared.limits.max_groups + { + return Err(RuntimeError::Overloaded.into()); + } + remaining()?; + let target = config + .open_import_target(log, images, route, "transfer-spools") + .await?; + remaining()?; + let prepared = recovery + .prepare_transfer_child(&domain, sources, target) + .await + .map_err(|error| CellGroupControllerError::Archive(Arc::new(error.into())))?; + let remaining = remaining()?; + let prepared = Arc::new(prepared); + // Retain the exact page owner before the engine can enter sync. A lost + // receipt or timeout can follow the durable mapping. + config + .parked + .lock() + .map_err(|_| CellGroupControllerError::Closed)? + .insert(route.history, PendingGroupOwner::Seeded(prepared.clone())); + let result = directory + .install_managed_transfer_voter(transition, &prepared, log, host, remaining) + .await + .map_err(catalog_error); + // A lost receipt can follow durable sync. Keep the applier but release + // the sealed-source preparation's admission when a mapping exists. + let owner = config.retain_synced_transfer(log, &prepared); + let receipt = result?; + owner?; + Ok(receipt) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/queries.rs b/crates/cellar/durability/raft/cell_groups/controller/queries.rs new file mode 100644 index 000000000..5b0c14aee --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/queries.rs @@ -0,0 +1,111 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Read callbacks share the node queue and SQL owner, independently of waiters. +use super::admission::Retention; +use super::calls::Delivery; +use super::*; +use crate::raft::cell_groups::{ + CellDispatchError, CellQueryInvocation, CellQueryResponse, CellSqlOptions, +}; +use cellar_logic::consensus::ProposalId; +use tokio::sync::oneshot; + +pub(super) struct Request { + invocation: CellQueryInvocation, + options: CellSqlOptions, + reply: oneshot::Sender>, + permit: Arc, +} + +/// Dropping or expiring the response does not cancel an admitted query callback. +/// Its physical connection and reservation remain owned until the callback exits. +pub struct CellQueryCall { + inner: calls::Call, +} + +impl CellQueryCall { + pub async fn response(self) -> Result> { + self.inner.response().await + } +} + +impl CellGroupController { + /// Queue one fresh registered query synchronously. It retains its original + /// consistency, token and deadline through discovery, read barrier and SQL. + pub fn query( + &self, + invocation: CellQueryInvocation, + mut options: CellSqlOptions, + ) -> Result { + let shared = &self.handle.0; + let (identity, deadline, bytes) = invocation.controller_admission(&shared.domain)?; + let mut registration = shared.registration().map_err(CellDispatchError::Runtime)?; + if shared.closing.load(Ordering::Acquire) { + return Err(CellDispatchError::Runtime(RuntimeError::Closed)); + } + let bytes = bytes + .checked_add(commands::option_bytes(&mut options).map_err(CellDispatchError::Runtime)?) + .ok_or(CellDispatchError::Runtime(RuntimeError::Overloaded))?; + let permit = shared + .commands + .acquire(bytes) + .ok_or(CellDispatchError::Runtime(RuntimeError::Overloaded))?; + let (reply, receiver) = oneshot::channel(); + shared + .requests + .try_send(requests::Request::Query(Box::new(Request { + invocation, + options, + reply, + permit: Arc::new(Retention { + _permit: permit, + _work: registration.work.take(), + }), + }))) + .map_err(|error| { + CellDispatchError::Runtime(match error { + tokio::sync::mpsc::error::TrySendError::Closed(_) => RuntimeError::Closed, + tokio::sync::mpsc::error::TrySendError::Full(_) => RuntimeError::Overloaded, + }) + })?; + Ok(CellQueryCall { + inner: calls::Call { + receiver, + identity, + deadline, + domain: shared.domain.clone(), + failure: |_, reason| CellDispatchError::Runtime(reason), + }, + }) + } +} + +pub(super) async fn drive( + request: Request, + runtime: CellGroupRuntime, + sql: CellGroupSqlPool, + shared: Arc, + transport: RaftTransport, +) { + let Request { + mut invocation, + options, + reply, + permit, + } = request; + let result = super::query_routes::execute( + &mut invocation, + &runtime, + &sql, + &transport, + &shared, + options, + permit.clone(), + ) + .await + .map_err(Arc::new); + let _ = reply.send(Delivery { + result, + _permit: permit, + }); +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/query_routes.rs b/crates/cellar/durability/raft/cell_groups/controller/query_routes.rs new file mode 100644 index 000000000..aa4694dea --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/query_routes.rs @@ -0,0 +1,217 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! One controller invocation owns every replica attempt and its original budget. +use super::*; +use crate::application::consensus::RegistrationError; +use crate::raft::cell_groups::{ + CellDispatchError as Error, CellQueryInvocation, CellQueryResponse, CellQueryRouting, + CellSqlOptions, +}; +use crate::raft::host_sql::cell_groups::ReadRetention; +use crate::raft::storage::LogError; +use crate::raft::transport::TransportError; +use cellar_logic::consensus::{CommandFailure, MemberId}; +use std::time::Duration; + +struct Destination<'a> { + runtime: &'a CellGroupRuntime, + sql: &'a CellGroupSqlPool, + transport: &'a RaftTransport, + shared: &'a Shared, +} + +pub(super) async fn execute( + invocation: &mut CellQueryInvocation, + runtime: &CellGroupRuntime, + sql: &CellGroupSqlPool, + transport: &RaftTransport, + shared: &Shared, + options: CellSqlOptions, + permit: Arc, +) -> Result { + let destination = Destination { + runtime, + sql, + transport, + shared, + }; + invocation.begin().await?; + invocation.validate_minimum().await?; + let (members, attempt_timeout, leader_fallback) = match invocation.routing() { + CellQueryRouting::Local => return destination.local(invocation, options, permit).await, + CellQueryRouting::Automatic => { + let route = invocation + .attempt() + .ok_or(Error::Invalid("missing route"))? + .route(); + match runtime.log().cell_group(&route.history) { + Ok(_) => return destination.local(invocation, options, permit).await, + Err(LogError::NotFound) => (), + Err(error) => return Err(Error::Runtime(RuntimeError::Storage(error))), + } + let members = transport + .cell_query_router_members(route, invocation.maximum_attempts()) + .map_err(|error| match error { + TransportError::UnknownGroup => Error::Runtime(RuntimeError::NotFound), + other => Error::Transport(other), + })?; + (members, None, false) + } + CellQueryRouting::Replicas { + members, + attempt_timeout, + leader_fallback, + } => (members, Some(attempt_timeout), leader_fallback), + }; + let mut last = None; + for (n, member) in members.iter().enumerate() { + if n > 0 { + invocation.retry().await?; + } + let timeout = match attempt_timeout { + Some(timeout) => timeout, + None => { + let now = shared + .domain + .mono_ms() + .map_err(|_| Error::Runtime(RuntimeError::Closed))?; + let remaining = invocation.deadline_ms().saturating_sub(now); + // Leave time for other trusted replicas after a blackholed peer. + Duration::from_millis((remaining / (members.len() - n) as u64).max(1)) + } + }; + invocation.limit_attempt(timeout)?; + let result = destination + .attempt(invocation, *member, options.clone(), permit.clone()) + .await; + match result { + Ok(response) => return Ok(response), + Err(error) if retryable(&error) => last = Some(error), + Err(error) => return Err(error), + } + } + if let Some(attempt_timeout) = attempt_timeout.filter(|_| leader_fallback) { + let history = invocation + .attempt() + .ok_or(Error::Invalid("missing route"))? + .route() + .history; + let timer = shared + .domain + .sleep_until(invocation.deadline_ms()) + .map_err(|_| Error::Runtime(RuntimeError::Closed))?; + let status = crate::asyncrt::select! { + status = runtime.status(history) => status, + _ = timer => return Err(Error::Runtime(RuntimeError::DeadlineExceeded)), + }; + let hint = match status { + Ok(status) => status + .leader + .and_then(|member| MemberId::new(member).ok()) + .or(invocation.leader_hint()), + Err(RuntimeError::NotFound) => invocation.leader_hint(), + Err(error) => return Err(Error::Runtime(error)), + }; + if let Some(leader) = hint.filter(|leader| !members.contains(leader)) { + invocation.retry().await?; + invocation.limit_attempt(attempt_timeout)?; + return destination + .attempt(invocation, leader, options, permit) + .await; + } + } + Err(last.unwrap_or(Error::Invalid("empty query routing plan"))) +} + +impl Destination<'_> { + async fn attempt( + &self, + invocation: &mut CellQueryInvocation, + member: MemberId, + options: CellSqlOptions, + permit: Arc, + ) -> Result { + let history = invocation + .attempt() + .ok_or(Error::Invalid("missing route"))? + .route() + .history; + let is_local = match self.runtime.log().cell_group(&history) { + Ok(group) => group.identity().member == member, + Err(LogError::NotFound) => false, + Err(error) => return Err(Error::Runtime(RuntimeError::Storage(error))), + }; + if is_local { + self.local(invocation, options, permit).await + } else { + invocation + .execute_remote(self.transport, member) + .await + .map(CellQueryResponse::remote) + } + } + + async fn local( + &self, + invocation: &mut CellQueryInvocation, + options: CellSqlOptions, + permit: Arc, + ) -> Result { + let history = invocation + .attempt() + .ok_or(Error::Invalid("missing route"))? + .route() + .history; + if !self + .shared + .groups + .contains(&history) + .map_err(Error::Runtime)? + { + return Err(Error::Runtime(RuntimeError::NotFound)); + } + invocation + .execute_retained( + self.runtime, + self.sql, + options, + Some(ReadRetention::new(permit)), + ) + .await + } +} + +fn unavailable(failure: &CommandFailure) -> bool { + matches!( + failure, + CommandFailure::NotLeader { .. } + | CommandFailure::Unavailable + | CommandFailure::Overloaded + | CommandFailure::VersionUnavailable + ) +} + +fn retryable(error: &Error) -> bool { + match error { + Error::Runtime( + RuntimeError::NotLeader + | RuntimeError::NotReady + | RuntimeError::NotFound + | RuntimeError::DeadlineExceeded + | RuntimeError::Overloaded, + ) + | Error::Registration(RegistrationError::UnknownOperation(_)) => true, + Error::Execution(crate::raft::execution::ExecutionError::Refused(failure)) => { + unavailable(failure) + } + Error::Transport(TransportError::Query(failure)) => unavailable(failure), + Error::Transport( + TransportError::Io(_) + | TransportError::Timeout + | TransportError::UnknownGroup + | TransportError::StalePeer + | TransportError::Overloaded, + ) => true, + _ => false, + } +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/registration.rs b/crates/cellar/durability/raft/cell_groups/controller/registration.rs new file mode 100644 index 000000000..6a4a3d6bb --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/registration.rs @@ -0,0 +1,503 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Bounded attachment ownership survives response loss and joins before shutdown. +use super::calls::Delivery; +use super::*; +use crate::raft::budget::Permit; +use crate::raft::cell_groups::{CellGroupBootstrapPermit, PreparedGroupManifest}; +use crate::raft::transport::{GroupRoute, PhysicalRegistration}; +use cellar_logic::cell_group::GroupHistory; +use tokio::sync::{mpsc, oneshot}; + +enum GroupState { + Pending, + Active, + ParkedRetired, + Quarantined(CellGroupControllerError), +} + +pub(crate) struct GroupRoster { + groups: Mutex>, + limit: usize, +} + +impl GroupRoster { + pub(super) fn producers_quiesced(&self) -> bool { + self.groups.lock().is_ok_and(|groups| { + groups + .values() + .all(|state| matches!(state, GroupState::Active | GroupState::ParkedRetired)) + }) + } + + pub(super) fn new(groups: impl Iterator, limit: usize) -> Arc { + Arc::new(Self { + groups: Mutex::new(groups.map(|g| (g, GroupState::Active)).collect()), + limit, + }) + } + + pub(crate) fn contains(&self, group: &GroupHistory) -> Result { + match self + .groups + .lock() + .map_err(|_| RuntimeError::Closed)? + .get(group) + { + Some(GroupState::Active) => Ok(true), + Some(GroupState::Quarantined(_)) => Err(RuntimeError::GroupQuarantined), + _ => Ok(false), + } + } + + pub(super) fn quarantine( + &self, + group: GroupHistory, + error: CellGroupControllerError, + ) -> Result<(), RuntimeError> { + let mut groups = self.groups.lock().map_err(|_| RuntimeError::Closed)?; + let state = groups.get_mut(&group).ok_or(RuntimeError::NotFound)?; + if matches!(state, GroupState::ParkedRetired) { + return Err(RuntimeError::NotFound); + } + if !matches!(state, GroupState::Quarantined(_)) { + *state = GroupState::Quarantined(error); + } + Ok(()) + } + + pub(super) fn failure( + &self, + group: &GroupHistory, + ) -> Result, RuntimeError> { + Ok( + match self + .groups + .lock() + .map_err(|_| RuntimeError::Closed)? + .get(group) + { + Some(GroupState::Quarantined(error)) => Some(error.clone()), + _ => None, + }, + ) + } + + pub(super) fn len(&self) -> usize { + self.groups + .lock() + .map(|g| { + g.values() + .filter(|s| matches!(s, GroupState::Active)) + .count() + }) + .unwrap_or(0) + } + + pub(super) fn pending(&self) -> usize { + self.groups + .lock() + .map(|g| { + g.values() + .filter(|s| matches!(s, GroupState::Pending)) + .count() + }) + .unwrap_or(0) + } + + pub(super) fn quarantined(&self) -> usize { + self.groups + .lock() + .map(|g| { + g.values() + .filter(|s| matches!(s, GroupState::Quarantined(_))) + .count() + }) + .unwrap_or(0) + } + + pub(super) fn reserve( + self: &Arc, + group: GroupHistory, + ) -> Result { + let mut groups = self.groups.lock().map_err(|_| RuntimeError::Closed)?; + if groups.contains_key(&group) { + return Err(RuntimeError::AlreadyAttached); + } + if groups.len() >= self.limit { + return Err(RuntimeError::Overloaded); + } + groups.insert(group, GroupState::Pending); + Ok(Reservation { + roster: self.clone(), + group, + active: false, + }) + } +} + +pub(super) struct Reservation { + roster: Arc, + group: GroupHistory, + active: bool, +} + +impl Reservation { + pub(super) fn activate(&mut self) -> Result<(), RuntimeError> { + let mut groups = self + .roster + .groups + .lock() + .map_err(|_| RuntimeError::Closed)?; + let entry = groups.get_mut(&self.group).ok_or(RuntimeError::NotFound)?; + if !matches!(entry, GroupState::Pending) { + return Err(RuntimeError::GroupQuarantined); + } + *entry = GroupState::Active; + self.active = true; + Ok(()) + } +} + +impl Drop for Reservation { + fn drop(&mut self) { + if !self.active { + if let Ok(mut groups) = self.roster.groups.lock() { + if matches!(groups.get(&self.group), Some(GroupState::Pending)) { + groups.remove(&self.group); + } + } + } + } +} + +/// The applier must own the exact page store used to verify the selected root. +/// Bootstrap/recovery grants no serving authority; attachment revalidates the +/// catalog and application admission still requires a current-term quorum. +pub struct CellGroupRegistration { + pub permit: CellGroupBootstrapPermit, + pub current: PreparedGroupManifest, + pub applier: CellGroupApplier, + pub transport: GroupRoute, +} + +pub struct CellGroupRegistrationError { + pub error: CellGroupControllerError, + pub registration: Box, +} + +impl std::fmt::Debug for CellGroupRegistrationError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("CellGroupRegistrationError") + .field("error", &self.error) + .finish_non_exhaustive() + } +} + +impl std::fmt::Display for CellGroupRegistrationError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + std::fmt::Display::fmt(&self.error, f) + } +} + +impl std::error::Error for CellGroupRegistrationError {} + +/// Dropping this waiter does not cancel attachment or release its group slot. +pub struct CellGroupRegistrationCall { + receiver: oneshot::Receiver>, +} + +impl CellGroupRegistrationCall { + pub async fn response(self) -> Result<(), Arc> { + self.receiver + .await + .map_err(|_| Arc::new(CellGroupControllerError::Closed))? + .result + } +} + +pub(super) struct Request { + pub registration: CellGroupRegistration, + reservation: Reservation, + reply: oneshot::Sender>, + permit: Arc, +} + +pub(super) struct Completed { + pub group: GroupHistory, + result: Result, + reservation: Reservation, + reply: oneshot::Sender>, + permit: Arc, +} + +#[derive(PartialEq, Eq)] +pub(crate) struct GroupRegistrationInventory { + pub(crate) attached: Vec, + pub(crate) pending: usize, + pub(crate) quarantined: usize, + pub(crate) retired_sources: Vec, +} + +impl CellGroupController { + pub(crate) fn retain_parked_retired_sources( + &self, + histories: Vec, + ) -> Result<(), RuntimeError> { + let roster = &self.handle.0.groups; + let mut groups = roster.groups.lock().map_err(|_| RuntimeError::Closed)?; + if groups.len().saturating_add(histories.len()) > roster.limit { + return Err(RuntimeError::Overloaded); + } + if histories.windows(2).any(|pair| pair[0] >= pair[1]) + || histories.iter().any(|history| groups.contains_key(history)) + { + return Err(RuntimeError::AlreadyAttached); + } + groups.extend( + histories + .into_iter() + .map(|history| (history, GroupState::ParkedRetired)), + ); + Ok(()) + } + + /// One locked roster cut. Failed attachments remain registered physical + /// debt; pending slots are separate from attached histories. + pub(crate) fn registration_inventory( + &self, + ) -> Result { + let groups = self + .handle + .0 + .groups + .groups + .lock() + .map_err(|_| RuntimeError::Closed)?; + let mut inventory = GroupRegistrationInventory { + attached: Vec::new(), + pending: 0, + quarantined: 0, + retired_sources: Vec::new(), + }; + for (history, state) in groups.iter() { + match state { + GroupState::Pending => inventory.pending += 1, + GroupState::Active => inventory.attached.push(*history), + GroupState::ParkedRetired => inventory.retired_sources.push(*history), + GroupState::Quarantined(_) => { + inventory.attached.push(*history); + inventory.quarantined += 1; + } + } + } + Ok(inventory) + } + + /// Atomically reserve the controller slot and transfer attachment ownership. + /// Admission failure returns all prepared owners without starting attachment. + pub fn register( + &self, + mut registration: CellGroupRegistration, + ) -> Result { + let shared = &self.handle.0; + let admitted = (|| { + if shared.closing.load(Ordering::Acquire) { + return Err(CellGroupControllerError::Closed); + } + let route = registration.permit.route(); + if route != registration.current.marker().route() + || route.history != registration.applier.history() + || route.history != registration.transport.scope + || route.schema_digest != registration.transport.schema_digest + || !Arc::ptr_eq( + ®istration.current.engine_owner, + &shared.runtime.log().owner(), + ) + || !Arc::ptr_eq( + ®istration.permit.group().owner, + &shared.runtime.log().owner(), + ) + { + return Err(CellGroupControllerError::Invalid( + "registration identity/engine", + )); + } + registration + .applier + .check_startup(®istration.current) + .map_err(|e| CellGroupControllerError::Apply(Arc::new(e)))?; + if registration + .permit + .group() + .consensus_claimed + .load(Ordering::Acquire) + { + return Err(RuntimeError::AlreadyAttached.into()); + } + if registration.transport.peers.is_empty() || registration.transport.peers.len() > 128 { + return Err(CellGroupControllerError::Invalid( + "registration peer bounds", + )); + } + let mut bytes = registration + .current + .reserved_bytes() + .checked_add(16384) + .ok_or(RuntimeError::Overloaded)?; + for peer in &mut registration.transport.peers { + if peer.label.len() > 4096 || peer.server_name.len() > 4096 { + return Err(RuntimeError::Overloaded.into()); + } + bytes = bytes + .checked_add(peer.label.len() + peer.server_name.len() + 512) + .ok_or(RuntimeError::Overloaded)?; + peer.label = std::mem::take(&mut peer.label) + .into_boxed_str() + .into_string(); + peer.server_name = std::mem::take(&mut peer.server_name) + .into_boxed_str() + .into_string(); + } + registration.transport.peers = std::mem::take(&mut registration.transport.peers) + .into_boxed_slice() + .into_vec(); + let permit = shared + .commands + .acquire(bytes) + .ok_or(RuntimeError::Overloaded)?; + let reservation = shared.groups.reserve(route.history)?; + Ok((permit, reservation)) + })(); + let (permit, reservation) = match admitted { + Ok(pair) => pair, + Err(error) => { + return Err(CellGroupRegistrationError { + error, + registration: Box::new(registration), + }) + } + }; + let (reply, receiver) = oneshot::channel(); + shared + .requests + .try_send(requests::Request::Registration(Box::new(Request { + registration, + reservation, + reply, + permit: Arc::new(permit), + }))) + .map_err(|error| { + let (request, error) = match error { + mpsc::error::TrySendError::Closed(request) => { + (request, CellGroupControllerError::Closed) + } + mpsc::error::TrySendError::Full(request) => { + (request, RuntimeError::Overloaded.into()) + } + }; + let requests::Request::Registration(request) = request else { + unreachable!("registration envelope") + }; + CellGroupRegistrationError { + error, + registration: Box::new(request.registration), + } + })?; + Ok(CellGroupRegistrationCall { receiver }) + } +} + +pub(super) async fn drive( + request: Request, + runtime: CellGroupRuntime, + transport: RaftTransport, +) -> Completed { + let Request { + registration, + reservation, + reply, + permit, + } = request; + let group = registration.permit.route().history; + let result = attach(registration, runtime, transport).await; + Completed { + group, + result, + reservation, + reply, + permit, + } +} + +pub(super) async fn attach( + registration: CellGroupRegistration, + runtime: CellGroupRuntime, + transport: RaftTransport, +) -> Result { + registration + .applier + .check_startup(®istration.current) + .map_err(|e| CellGroupControllerError::Apply(Arc::new(e)))?; + let mut guard = transport + .register_physical_group(registration.transport) + .map_err(CellGroupControllerError::Transport)?; + guard.checkpoint = Some( + transport + .register_group_checkpoint(registration.applier) + .map_err(CellGroupControllerError::Transport)?, + ); + runtime + .attach(registration.permit, registration.current) + .await?; + Ok(guard) +} + +impl Completed { + /// A runtime-wide close or competing external owner is not a safely failed + /// registration. Preserve the conservative joined node shutdown boundary. + pub fn uncertain(&self) -> bool { + matches!( + &self.result, + Err(CellGroupControllerError::Runtime( + RuntimeError::Closed | RuntimeError::AlreadyAttached + )) + ) + } + + pub fn finish( + self, + closing: bool, + ) -> Result, CellGroupControllerError> { + let Self { + result, + mut reservation, + reply, + permit, + .. + } = self; + let outcome = if closing { + Err(CellGroupControllerError::Closed) + } else { + result.and_then(|guard| { + reservation.activate()?; + Ok(guard) + }) + }; + // A failed attachment's group capacity is free before publishing its + // refusal; success keeps the active roster entry after this guard drops. + drop(reservation); + let _ = reply.send(Delivery { + result: outcome + .as_ref() + .map(|_| ()) + .map_err(|e| Arc::new(e.clone())), + _permit: permit, + }); + match outcome { + Ok(guard) => Ok(Some(guard)), + Err(_) if closing => Ok(None), + Err(error) => Err(error), + } + } +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/requests.rs b/crates/cellar/durability/raft/cell_groups/controller/requests.rs new file mode 100644 index 000000000..126acf92f --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/requests.rs @@ -0,0 +1,74 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! One node queue and physical shutdown owner. Provisioning has independent +//! admission so command ownership need not recursively acquire its own budget. +use super::*; + +pub(super) enum Request { + Archive(Box), + S3(Box), + Registration(Box), + Provision(Box), + Command(Box), + Query(Box), + Enrollment(Box), +} + +pub(super) enum Completed { + Command(Box), + Enrollment(Box), + Query, +} + +impl Completed { + pub fn group(&self) -> Option { + match self { + Self::Command(done) => done.invocation.as_ref().and_then(|i| i.unresolved_group()), + Self::Enrollment(done) => done.invocation.unresolved_group(), + Self::Query => None, + } + } + + pub fn failure(&self) -> Option { + match self { + Self::Command(done) if done.invocation.as_ref().is_some_and(|i| i.unresolved()) => { + Some(done.error.clone().map_or( + CellGroupControllerError::Invalid("unresolved successful command"), + CellGroupControllerError::Command, + )) + } + Self::Enrollment(done) if done.invocation.unresolved() => { + Some(done.error.clone().map_or( + CellGroupControllerError::Invalid("unresolved successful enrollment"), + CellGroupControllerError::Enrollment, + )) + } + _ => None, + } + } +} + +pub(super) async fn drive( + request: Request, + runtime: CellGroupRuntime, + sql: CellGroupSqlPool, + shared: Arc, + progress: tokio::sync::watch::Receiver<()>, + transport: RaftTransport, +) -> Completed { + match request { + Request::Registration(_) | Request::Provision(_) | Request::Archive(_) | Request::S3(_) => { + unreachable!("registration has its own physical join owner") + } + Request::Command(request) => Completed::Command(Box::new( + commands::drive(*request, runtime, sql, shared, progress, transport).await, + )), + Request::Enrollment(request) => Completed::Enrollment(Box::new( + enrollments::drive(*request, runtime, sql, shared, progress, transport).await, + )), + Request::Query(request) => { + queries::drive(*request, runtime, sql, shared, transport).await; + Completed::Query + } + } +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/worker.rs b/crates/cellar/durability/raft/cell_groups/controller/worker.rs new file mode 100644 index 000000000..64d1b9ad4 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/worker.rs @@ -0,0 +1,473 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use crate::raft::cell_groups::{CellGroupApply, CellGroupEffect, CellGroupPreparedApply}; +use cellar_logic::{cell_group::GroupHistory, consensus::CommandFailure}; +use futures_util::{future::BoxFuture, stream::FuturesUnordered, StreamExt}; +use std::collections::{BTreeSet, VecDeque}; + +#[derive(Clone)] +enum Work { + Prepare(Box), + Publish(Box), +} + +struct Pending { + group: GroupHistory, + work: Work, + retry_at: u64, +} + +type Completed = ( + GroupHistory, + Result, CellGroupControllerError>, + Work, +); + +pub(super) struct Worker { + runtime: CellGroupRuntime, + events: CellGroupRuntimeEvents, + transport: RaftTransport, + sql: CellGroupSqlPool, + roster: BTreeMap, + limits: CellGroupControllerLimits, + shared: Arc, + pending: VecDeque, + active: BTreeSet, + jobs: FuturesUnordered>, + hints: FuturesUnordered>, + requests: tokio::sync::mpsc::Receiver, + commands: FuturesUnordered>, + failed_commands: Vec, + quarantined: BTreeSet, + retained_apply: Vec, + provisioning: provision::Manager, + registrations: FuturesUnordered>, + routes: BTreeMap, + progress: tokio::sync::watch::Sender<()>, + checkpoints: checkpoints::Manager, + archives: archive::Manager, + effects: effects::Manager, + _checkpoint_routes: Vec, +} + +impl Worker { + pub(super) fn new( + runtime: CellGroupRuntime, + events: CellGroupRuntimeEvents, + transport: RaftTransport, + sql: CellGroupSqlPool, + roster: BTreeMap, + requests: tokio::sync::mpsc::Receiver, + shared: Arc, + ) -> Result { + // Retained replicas serve peer and S3 seeds whenever the transport + // pool is configured, independently of periodic local checkpoints. + let serve_checkpoints = match transport.group_checkpoint_pool() { + Ok(_) => true, + Err(TransportError::Refused) => false, + Err(error) => return Err(CellGroupControllerError::Transport(error)), + }; + let mut checkpoint_routes = Vec::new(); + if serve_checkpoints { + for applier in roster.values() { + checkpoint_routes.push( + transport + .register_group_checkpoint(applier.clone()) + .map_err(CellGroupControllerError::Transport)?, + ); + } + } + let archives = archive::Manager::new(shared.domain.clone()); + transport + .enable_group_archive_seeds(archive::seed::Client::new(&shared)) + .map_err(CellGroupControllerError::Transport)?; + let effects = effects::Manager::new(&roster)?; + Ok(Self { + runtime, + events, + transport, + sql, + roster, + limits: shared.limits, + shared, + pending: VecDeque::new(), + active: BTreeSet::new(), + jobs: FuturesUnordered::new(), + hints: FuturesUnordered::new(), + requests, + commands: FuturesUnordered::new(), + failed_commands: Vec::new(), + quarantined: BTreeSet::new(), + retained_apply: Vec::new(), + registrations: FuturesUnordered::new(), + provisioning: provision::Manager::default(), + routes: BTreeMap::new(), + progress: tokio::sync::watch::channel(()).0, + checkpoints: checkpoints::Manager::default(), + archives, + effects, + _checkpoint_routes: checkpoint_routes, + }) + } + + fn launch(&mut self, now: u64) -> Result<(), CellGroupControllerError> { + // Rotate the bounded roster queue. A busy group cannot monopolize the + // next available apply slot, and successors never run concurrently. + for _ in 0..self.pending.len() { + let Some(job) = self.pending.pop_front() else { + break; + }; + if self.quarantined.contains(&job.group) { + self.retained_apply.push(job.work); + continue; + } + if job.retry_at > now + || self.active.contains(&job.group) + || self.checkpoints.contains(&job.group) + { + self.pending.push_back(job); + continue; + } + let checkpoint_started = match &job.work { + Work::Prepare(work) => self.checkpoints.start(work, now, &self.shared)?, + Work::Publish(_) => false, + }; + if checkpoint_started { + continue; + } + // Capture/publication has its own bounded capacity and requires + // no SQL slot. Continue scanning when apply is full so a slow + // physical apply cannot prevent another group's checkpoint. + if self.jobs.len() == self.limits.max_parallel_apply { + self.pending.push_back(job); + continue; + } + self.active.insert(job.group); + let sql = self.sql.clone(); + let applier = self.roster[&job.group].clone(); + let runtime = self.runtime.clone(); + self.jobs.push(Box::pin(async move { + let retained = job.work.clone(); + let result = apply(sql, applier, runtime, job.work).await; + (job.group, result, retained) + })); + } + self.shared + .active_apply + .store(self.jobs.len(), Ordering::Release); + self.shared + .pending_apply + .store(self.pending.len(), Ordering::Release); + Ok(()) + } + + async fn quarantine( + &mut self, + group: GroupHistory, + error: CellGroupControllerError, + ) -> Result<(), CellGroupControllerError> { + if failure::node_wide(&error) { + return Err(error); + } + if self.quarantined.insert(group) { + // Stop application admission first. Runtime revocation then bypasses + // any waiting apply, fences captured leader authority and refuses + // new direct/transport work without dropping the RawNode claim. + self.shared.groups.quarantine(group, error)?; + self.runtime.quarantine(group).await?; + self.progress.send_replace(()); + } + Ok(()) + } + + pub(super) async fn run( + &mut self, + domain: &crate::asyncrt::DomainToken, + ) -> Result<(), CellGroupControllerError> { + let mut wake_at = domain + .mono_ms() + .map_err(|_| CellGroupControllerError::Closed)? + .saturating_add(self.limits.retry_interval_ms); + loop { + if self.shared.closing.load(Ordering::Acquire) { + return Ok(()); + } + let now = domain + .mono_ms() + .map_err(|_| CellGroupControllerError::Closed)?; + self.launch(now)?; + self.archives.launch(&self.runtime, &self.shared, now); + self.shared.archive_seeds.maintain(&self.shared, now); + self.effects.launch(&self.quarantined, &self.shared, now)?; + let timer = domain + .sleep_until(wake_at) + .map_err(|_| CellGroupControllerError::Closed)?; + crate::asyncrt::select! { + _ = self.shared.stop.notified() => return Ok(()), + _ = domain.wait_for_work() => return Err(CellGroupControllerError::Closed), + _ = timer => { + self.progress.send_replace(()); + wake_at = domain.mono_ms().map_err(|_| CellGroupControllerError::Closed)? + .saturating_add(self.limits.retry_interval_ms); + }, + Some(request) = self.requests.recv() => { + match request { + requests::Request::Archive(request) => self.archives.request(*request, &self.shared), + requests::Request::S3(request) => self.archives.settle(*request), + requests::Request::Provision(request) => self.provisioning.request(*request, &self.shared, &self.transport), + requests::Request::Registration(request) => { + let group = request.registration.applier.history(); + if self.roster.insert(group, request.registration.applier.clone()).is_some() { + return Err(CellGroupControllerError::Invalid("duplicate registered applier")); + } + self.effects.register(&request.registration.applier)?; + self.registrations.push(Box::pin(registration::drive(*request, self.runtime.clone(), self.transport.clone()))); + } + other => self.commands.push(Box::pin(requests::drive(other, self.runtime.clone(), self.sql.clone(), self.shared.clone(), self.progress.subscribe(), self.transport.clone()))), + } + } + Some(done) = self.provisioning.next(), if self.provisioning.is_running() => { + let group = done.group(); + self.provisioning.complete(done, &self.shared, &self.transport, &mut self.roster, &mut self.routes, &mut self.archives)?; + if let Some(applier) = self.roster.get(&group) { self.effects.register(applier)?; } + } + Some(completed) = self.registrations.next(), if !self.registrations.is_empty() => { + let group = completed.group; + let uncertain = completed.uncertain(); + match completed.finish(false) { + Ok(Some(guard)) => { self.routes.insert(group, guard); } + Ok(None) => unreachable!("active registration completion"), + Err(error) => { + if !self.quarantined.contains(&group) { + self.roster.remove(&group); + self.effects.remove(&group); + } + if uncertain { return Err(error); } + } + } + } + Some(completed) = self.commands.next(), if !self.commands.is_empty() => { + if let Some(error) = completed.failure() { + // Retain the invocation and its permit through physical + // shutdown even after delivering the error to its caller. + let group = completed.group().ok_or(CellGroupControllerError::Invalid("unresolved command without group"))?; + self.failed_commands.push(completed); + self.quarantine(group, error).await?; + } + } + Some((group, result, retained)) = self.jobs.next(), if !self.jobs.is_empty() => { + self.active.remove(&group); + match result { + Err(error) => { + self.retained_apply.push(retained); + self.quarantine(group, error).await?; + } + Ok(Some(work)) if self.quarantined.contains(&group) => self.retained_apply.push(work), + Ok(Some(work)) => { + self.shared.apply_retries.fetch_add(1, Ordering::Relaxed); + self.pending.push_back(Pending { group, work, retry_at: now.saturating_add(self.limits.retry_interval_ms) }); + } + Ok(None) => (), + } + } + completed = self.checkpoints.next(), if self.checkpoints.running() => { + match completed { + checkpoints::Completed::Captured((group, result)) => match result { + Ok(checkpoint) if !self.quarantined.contains(&group) => self.checkpoints.publish(group, checkpoint, &self.runtime)?, + Ok(_) => { + if let Some(work) = self.checkpoints.remove(group, &self.shared) { self.retained_apply.push(Work::Prepare(work)); } + } + Err(error) => self.checkpoint_failed(group, error, now).await?, + }, + checkpoints::Completed::Admitted((group, result)) => match result { + Ok(()) => { + if let Some(work) = self.checkpoints.accept(group, &self.shared)? { self.resume_apply(group, work, now); } + } + Err(error) => self.checkpoint_failed(group, error.into(), now).await?, + }, + } + } + Some(()) = self.hints.next(), if !self.hints.is_empty() => (), + Some(completed) = self.effects.next(), if self.effects.running() => { + if let Some(applier) = self.roster.get(&completed.group) { self.effects.register(applier)?; } + self.effects.complete(completed, &self.shared); + } + completed = self.archives.next(), if self.archives.running() => { + self.archives.complete(completed, &self.shared, now, false); + self.progress.send_replace(()); + } + event = self.events.recv() => { + let event = event.ok_or(CellGroupControllerError::Closed)?; + let group = event.group(); + if !self.roster.contains_key(&group) { + return Err(CellGroupControllerError::Invalid("unregistered group event")); + } + if self.quarantined.contains(&group) { + if let Some(work) = event.retain_apply() { + self.retained_apply.push(Work::Prepare(Box::new(work))); + } + continue; + } + let effect = match event.effect() { + Ok(effect) => effect, + Err(error) => { + self.quarantine(group, error.clone().into()).await?; + continue; + } + }; + let installed_checkpoint = matches!(&effect, CellGroupEffect::CheckpointInstalled { .. }); + match effect { + CellGroupEffect::Message(message) => { + if let Err(error) = self.transport.submit_cell_group_event(&event) { + if !matches!(error, TransportError::Overloaded | TransportError::UnknownGroup | TransportError::StalePeer | TransportError::Refused) { + return Err(CellGroupControllerError::Transport(error)); + } + self.shared.send_refusals.fetch_add(1, Ordering::Relaxed); + // A hint is advisory, never delivery or commit evidence. + // Excess ordinary unreachable hints may be dropped. + let snapshot = message.get_msg_type() == ::raft::prelude::MessageType::MsgSnapshot; + // Snapshot failure cannot be dropped: Raft pauses + // that peer until this feedback. At most one can + // be pending per known peer; ordinary hints keep + // the stricter one-per-group aggregate cap. + if snapshot || self.hints.len() < self.limits.max_groups { + let runtime = self.runtime.clone(); + let member = message.to; + self.hints.push(Box::pin(async move { + if snapshot { + let _ = runtime.report_snapshot(group, member, false).await; + } else { + let _ = runtime.report_unreachable(group, member).await; + } + })); + } + } + } + CellGroupEffect::Apply(_) => { + // One pending successor per group is possible before a + // completed publication future has been polled here. + if self.pending.iter().any(|job| job.group == group) { + return Err(CellGroupControllerError::Invalid("duplicate group apply")); + } + let work = event.retain_apply().ok_or(CellGroupControllerError::Invalid("apply ticket"))?; + self.pending.push_back(Pending { group, work: Work::Prepare(Box::new(work)), retry_at: now }); + } + CellGroupEffect::Applied { manifest, .. } | CellGroupEffect::CheckpointInstalled { manifest } => { + // Configuration apply sets its revision to this + // entry's index. Ordinary cell writes do not scan routes. + if installed_checkpoint || manifest.membership_revision() == manifest.position().index.get() { + if let Err(error) = self.transport.reconcile_managed_physical(&group) { + if !matches!(error, TransportError::Overloaded | TransportError::UnknownGroup | TransportError::StalePeer | TransportError::Refused) { + return Err(CellGroupControllerError::Transport(error)); + } + } + } + self.archives.applied(group, manifest.position().index); + self.effects.observe(group, manifest.position().index, manifest.root().next_due_unix_millis())?; + // Only this post-sync event advances the published count. + self.shared.applied.fetch_add(1, Ordering::Relaxed); + self.progress.send_replace(()); + } + CellGroupEffect::CheckpointPublished { manifest } => { + self.shared.checkpoints_published.fetch_add(1, Ordering::Relaxed); + if let Some(work) = self.checkpoints.observed(group, manifest.position().index.get(), &self.shared)? { + self.resume_apply(group, work, now); + } + // Local compaction does not advance applied state. + self.progress.send_replace(()); + } + CellGroupEffect::ReadIndex { .. } => (), // Runtime-owned read authority. + CellGroupEffect::Persist(_) => return Err(CellGroupControllerError::Invalid("escaped persistence")), + } + } + } + } + } + + fn resume_apply(&mut self, group: GroupHistory, work: Box, now: u64) { + if self.quarantined.contains(&group) { + self.retained_apply.push(Work::Prepare(work)); + } else { + self.pending.push_back(Pending { + group, + work: Work::Prepare(work), + retry_at: now, + }); + } + } + + async fn checkpoint_failed( + &mut self, + group: GroupHistory, + error: CellGroupControllerError, + now: u64, + ) -> Result<(), CellGroupControllerError> { + if checkpoints::retryable(&error) { + if let Some(work) = self.checkpoints.retry(group, now, &self.shared) { + self.resume_apply(group, work, now); + } + } else { + if let Some(work) = self.checkpoints.remove(group, &self.shared) { + self.retained_apply.push(Work::Prepare(work)); + } + self.quarantine(group, error).await?; + } + Ok(()) + } + + pub(super) async fn finish(&mut self) -> Result<(), CellGroupControllerError> { + self.shared.archive_seeds.stop(); + self.requests.close(); + while self.requests.try_recv().is_ok() {} + // Attachment may already own a RawNode/claim or be inside catalog I/O. + // Poll it through completion before closing its physical dependencies. + while let Some(completed) = self.registrations.next().await { + let _ = completed.finish(true); + } + self.effects.join(&self.shared).await; + self.provisioning.join().await; + self.checkpoints.join().await; + self.archives.finish(&self.shared).await; + // A seed producer waits for its transport-held handoff lease before + // retiring the journaled generation. Drive both joins together. + futures_util::future::join(self.transport.close(), self.shared.archive_seeds.join()).await; + let sql = self + .sql + .close() + .await + .map_err(|error| CellGroupControllerError::Apply(Arc::new(error))); + // Retain the runtime's engine claim while an apply worker can still + // write. A replacement runtime must not overlap physical preparation. + let runtime = self + .runtime + .close() + .await + .map_err(CellGroupControllerError::Runtime); + // Keep all unfinished tickets and appliers until these physical joins. + sql.and(runtime) + } +} + +async fn apply( + sql: CellGroupSqlPool, + applier: CellGroupApplier, + runtime: CellGroupRuntime, + work: Work, +) -> Result, CellGroupControllerError> { + let prepared = match work { + Work::Prepare(work) => match sql.prepare_apply(&applier, (*work).clone()).await { + Ok(prepared) => Box::new(prepared), + Err(ExecutionError::Busy | ExecutionError::Refused(CommandFailure::Overloaded)) => { + return Ok(Some(Work::Prepare(work))) + } + Err(error) => return Err(CellGroupControllerError::Apply(Arc::new(error))), + }, + Work::Publish(prepared) => prepared, + }; + match (*prepared).clone().complete(&runtime).await { + Ok(()) => Ok(None), + Err(RuntimeError::Overloaded) => Ok(Some(Work::Publish(prepared))), + Err(error) => Err(error.into()), + } +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/archive.rs b/crates/cellar/durability/raft/cell_groups/directory/archive.rs new file mode 100644 index 000000000..829feab31 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/archive.rs @@ -0,0 +1,428 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Namespace-owned archive identity. Recovery reads the expected identity from +//! immutable catalog metadata, never by trusting a mutable registry's identity. +use super::*; +use crate::raft::cell_groups::{ + CellGroupArchive, CellGroupArchiveCreation, CellGroupArchiveError, CellGroupArchiveExecution, + CellGroupArchivePool, CellGroupArchiveRegistry, PreparedCellGroupArchiveCreation, +}; +use serde::{Deserialize, Serialize}; + +#[derive(Debug)] +pub enum CellGroupArchiveBindingError { + Catalog(CatalogError), + Archive(CellGroupArchiveError), +} + +impl std::fmt::Display for CellGroupArchiveBindingError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "group archive binding: {self:?}") + } +} + +impl std::error::Error for CellGroupArchiveBindingError { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + Some(match self { + Self::Catalog(error) => error, + Self::Archive(error) => error, + }) + } +} + +impl From for CellGroupArchiveBindingError { + fn from(error: CatalogError) -> Self { + Self::Catalog(error) + } +} + +impl From for CellGroupArchiveBindingError { + fn from(error: CellGroupArchiveError) -> Self { + Self::Archive(error) + } +} + +type Result = std::result::Result; + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Binding { + namespace: [u8; 32], + creation: CellGroupArchiveCreation, +} + +/// Persist `encode()` before commit. This one-use initializer first binds the +/// expected identity in the namespace catalog, then creates registry counters. +/// After a crash, open the existing binding; never allocate a replacement. +/// Incomplete initialization refuses admission and requires authority recovery. +pub struct PreparedCellGroupArchiveBinding { + directory: RangeDirectory, + birth: CellGroupBirth, + bytes: Vec, + creation: PreparedCellGroupArchiveCreation, +} + +impl PreparedCellGroupArchiveBinding { + pub fn encode(&self) -> &[u8] { + &self.bytes + } + + pub async fn commit( + self, + domain: &impl CellGroupArchiveExecution, + ) -> Result { + domain + .run(async { + self.directory + .require_current_group_birth(&self.birth) + .await?; + self.directory + .bounded(async { + let observed = self + .directory + .namespace + .inner + .create_and_observe_bounded( + &self.directory.archive_binding_key(self.birth.route()), + &self.bytes, + wire::LIMIT, + ) + .await?; + if observed != self.bytes { + return Err(CatalogError::Conflict("group archive binding")); + } + Ok(()) + }) + .await?; + self.directory + .require_current_group_birth(&self.birth) + .await?; + let registry = self.creation.commit(domain).await?; + self.directory + .require_current_group_birth(&self.birth) + .await?; + Ok(registry) + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } +} + +impl RangeDirectory { + fn archive_binding_key(&self, route: &GroupRoute) -> Path { + self.key(&format!( + "archives/{}/{}/binding", + hex::encode(route.history.group.bytes()), + hex::encode(route.history.incarnation.bytes()) + )) + } + + fn group_archive( + &self, + pool: &CellGroupArchivePool, + route: &GroupRoute, + ) -> Result { + Ok(pool.for_group( + self.namespace.inner.store.clone(), + Path::from(format!( + "{}cell-group-archives", + self.namespace.data_prefix() + )), + route.history, + )?) + } + + /// Prepare a fresh namespace-owned archive for a selected immutable birth. + /// Use one node-shared archive pool. No caller-selected backend or data key + /// can substitute another group's archive. This grants no serving authority. + pub async fn prepare_group_archive_binding( + &self, + domain: &impl CellGroupArchiveExecution, + pool: &CellGroupArchivePool, + birth: &CellGroupBirth, + ) -> Result { + domain + .run(async { + self.require_current_group_birth(birth).await?; + let archive = self.group_archive(pool, birth.route())?; + let creation = archive + .prepare_registry_creation(domain, birth.route().clone()) + .await?; + let bytes = record::encode( + wire::GROUP_ARCHIVE, + &Binding { + namespace: self.namespace.root_digest(), + creation: creation.intent().clone(), + }, + wire::LIMIT, + )?; + Ok(PreparedCellGroupArchiveBinding { + directory: self.clone(), + birth: birth.clone(), + bytes, + creation, + }) + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } + + // Called only while the exact managed recovery still fences this selected + // replacement. Reopen an existing registry without reinitializing it. + pub(super) async fn initialize_recovered_group_archive( + &self, + domain: &impl CellGroupArchiveExecution, + pool: &CellGroupArchivePool, + birth: &CellGroupBirth, + ) -> Result<()> { + let archive = self.group_archive(pool, birth.route())?; + let binding = match self.read_archive_binding(birth, None).await? { + Some(binding) => binding, + None => { + let creation = archive + .prepare_registry_creation(domain, birth.route().clone()) + .await?; + let bytes = record::encode( + wire::GROUP_ARCHIVE, + &Binding { + namespace: self.namespace.root_digest(), + creation: creation.intent().clone(), + }, + wire::LIMIT, + )?; + let key = self.archive_binding_key(birth.route()); + let observed = match self + .namespace + .inner + .create_and_observe_bounded(&key, &bytes, wire::LIMIT) + .await + { + Ok(observed) => observed, + Err(error @ (CatalogError::Storage(_) | CatalogError::Unconfirmed)) => { + self.namespace + .inner + .read_versioned_bounded(&key, wire::LIMIT) + .await? + .ok_or(error)? + .0 + } + Err(error) => return Err(error.into()), + }; + self.decode_archive_binding(birth, &observed)? + } + }; + archive + .initialize_recovery_registry(domain, &binding.creation) + .await?; + Ok(()) + } + + /// Reopen only the immutable expected identity. Missing binding, identity or + /// counters fail closed, including after every voter disk has disappeared. + /// This does not recreate metadata, fence old owners or authorize new voters. + pub async fn open_group_archive( + &self, + domain: &impl CellGroupArchiveExecution, + pool: &CellGroupArchivePool, + birth: &CellGroupBirth, + ) -> Result { + self.open_selected_archive(domain, pool, birth, None).await + } + + pub(crate) async fn has_group_archive_binding(&self, birth: &CellGroupBirth) -> Result { + self.require_current_group_birth(birth).await?; + Ok(self.read_archive_binding(birth, None).await?.is_some()) + } + + pub(crate) async fn verify_group_archive_binding( + &self, + domain: &impl CellGroupArchiveExecution, + pool: &CellGroupArchivePool, + birth: &CellGroupBirth, + intent: &[u8], + ) -> Result<()> { + self.open_selected_archive(domain, pool, birth, Some(intent)) + .await?; + Ok(()) + } + + /// Observe an exact rooted intent during operator startup. Absence only + /// delays admission. An invalid identity or binding refuses immediately. + pub(crate) async fn observe_group_archive_binding( + &self, + domain: &impl CellGroupArchiveExecution, + pool: &CellGroupArchivePool, + birth: &CellGroupBirth, + intent: &[u8], + ) -> Result { + domain + .run(async { + self.require_current_group_birth(birth).await?; + let Some(binding) = self.read_archive_binding(birth, Some(intent)).await? else { + return Ok(false); + }; + match self + .group_archive(pool, birth.route())? + .open_registry(domain, &binding.creation) + .await + { + Ok(_) => (), + Err(CellGroupArchiveError::MissingRegistry) => return Ok(false), + Err(error) => return Err(error.into()), + } + self.require_current_group_birth(birth).await?; + Ok(true) + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } + + async fn open_selected_archive( + &self, + domain: &impl CellGroupArchiveExecution, + pool: &CellGroupArchivePool, + birth: &CellGroupBirth, + intent: Option<&[u8]>, + ) -> Result { + domain + .run(async { + self.require_current_group_birth(birth).await?; + let registry = self + .open_archive_binding(domain, pool, birth, intent) + .await?; + self.require_current_group_birth(birth).await?; + Ok(registry) + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } + + // The recovery owner checks the exact intent, selected source, immutable + // birth and topology lock. Ordinary archive admission remains fenced. + pub(super) async fn open_recovery_archive( + &self, + domain: &impl CellGroupArchiveExecution, + pool: &CellGroupArchivePool, + birth: &CellGroupBirth, + ) -> Result { + self.open_archive_binding(domain, pool, birth, None).await + } + + /// Reopen a selected recovery source while ordinary startup is fenced. + pub(crate) async fn open_managed_recovery_archive( + &self, + domain: &impl CellGroupArchiveExecution, + pool: &CellGroupArchivePool, + route: &GroupRoute, + transition: ProposalId, + barrier: [u8; 32], + ) -> Result { + let birth = self + .bounded(async { + let (view, _) = self.current().await?; + let lock = view.head.recovery.as_ref().ok_or(CatalogError::Fenced)?; + if view.digest != barrier + || !lock.owns_transition(transition) + || !lock.blocks_startup(self, route)? + { + return Err(CatalogError::Fenced); + } + self.selected_route_in(&view, route).await?; + self.open_historical_birth(route).await + }) + .await?; + self.open_recovery_archive(domain, pool, &birth).await + } + + pub(super) async fn open_retired_source_archive( + &self, + domain: &impl CellGroupArchiveExecution, + pool: &CellGroupArchivePool, + route: &GroupRoute, + ) -> Result { + let birth = self.bounded(self.open_historical_birth(route)).await?; + self.open_archive_binding(domain, pool, &birth, None).await + } + + async fn open_archive_binding( + &self, + domain: &impl CellGroupArchiveExecution, + pool: &CellGroupArchivePool, + birth: &CellGroupBirth, + intent: Option<&[u8]>, + ) -> Result { + let binding = self + .read_archive_binding(birth, intent) + .await? + .ok_or(CatalogError::Corrupt("missing group archive binding"))?; + Ok(self + .group_archive(pool, birth.route())? + .open_registry(domain, &binding.creation) + .await?) + } + + async fn read_archive_binding( + &self, + birth: &CellGroupBirth, + intent: Option<&[u8]>, + ) -> Result> { + let Some((bytes, _)) = self + .bounded( + self.namespace + .inner + .read_versioned_bounded(&self.archive_binding_key(birth.route()), wire::LIMIT), + ) + .await? + else { + return Ok(None); + }; + if intent.is_some_and(|intent| intent != bytes.as_slice()) { + return Err(CatalogError::Conflict("group archive intent").into()); + } + self.decode_archive_binding(birth, &bytes).map(Some) + } + + fn decode_archive_binding(&self, birth: &CellGroupBirth, bytes: &[u8]) -> Result { + let binding: Binding = record::decode(wire::GROUP_ARCHIVE, bytes, wire::LIMIT)?; + if binding.namespace != self.namespace.root_digest() + || binding.creation.route() != birth.route() + { + return Err(CatalogError::Corrupt("group archive binding identity").into()); + } + Ok(binding) + } + + /// Cleanup may outlive the selected birth. Reopen only the exact immutable + /// catalog binding recorded by this engine; no initialization or serving + /// authority is supplied by an abandoned upload intent. + pub(in crate::raft) async fn open_seed_cleanup_archive( + &self, + domain: &impl CellGroupArchiveExecution, + pool: &CellGroupArchivePool, + expected: &CellGroupArchiveCreation, + ) -> Result { + let route = expected.route(); + let config = self.namespace_config(); + if route.history.cluster != config.cluster || route.history.namespace != config.namespace { + return Err(CatalogError::Corrupt("seed cleanup namespace").into()); + } + let bytes = self + .bounded(async { + self.namespace + .inner + .read_versioned_bounded(&self.archive_binding_key(route), wire::LIMIT) + .await? + .map(|(bytes, _)| bytes) + .ok_or(CatalogError::Corrupt("missing seed archive binding")) + }) + .await?; + let binding: Binding = record::decode(wire::GROUP_ARCHIVE, &bytes, wire::LIMIT)?; + if binding.namespace != self.namespace.root_digest() || &binding.creation != expected { + return Err(CatalogError::Corrupt("seed archive binding changed").into()); + } + Ok(self + .group_archive(pool, route)? + .open_registry(domain, expected) + .await?) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/birth.rs b/crates/cellar/durability/raft/cell_groups/directory/birth.rs new file mode 100644 index 000000000..ecece63a6 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/birth.rs @@ -0,0 +1,649 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Cell-independent immutable group births. A selected birth is metadata; only +//! durable genesis plus permanent member claims can authorize voter startup. + +use super::super::InitialMember; +use super::*; + +#[derive(Clone)] +pub struct CellGroupBirthSpec { + pub history: GroupHistory, + pub range: CellRange, + pub epoch: RouteEpoch, + pub creation: ProposalId, + pub voters: Vec, + pub learners: Vec, +} + +/// Persist the canonical request before reserving its global group ID. Retrying +/// must keep these bytes, including placement, creation identity and incarnation. +#[derive(Clone, Debug)] +pub struct CellGroupBirthRequest { + bytes: Vec, + route: GroupRoute, + creation: ProposalId, + voters: Vec, + learners: Vec, +} + +impl CellGroupBirthRequest { + pub fn encode(&self) -> &[u8] { + &self.bytes + } + + pub fn route(&self) -> &GroupRoute { + &self.route + } + + pub fn creation(&self) -> ProposalId { + self.creation + } + + pub fn voters(&self) -> &[InitialMember] { + &self.voters + } + + pub fn learners(&self) -> &[InitialMember] { + &self.learners + } +} + +/// Read-back of an immutable birth selected by an observed directory root. +/// This grants neither membership nor permission to initialize a blank disk. +#[derive(Clone)] +pub struct CellGroupBirth { + pub(super) request: Arc, + pub(super) owner: Arc<()>, +} + +impl CellGroupBirth { + pub fn encode(&self) -> &[u8] { + self.request.encode() + } + + pub fn route(&self) -> &GroupRoute { + self.request.route() + } + + pub fn creation(&self) -> ProposalId { + self.request.creation() + } + + pub fn voters(&self) -> &[InitialMember] { + self.request.voters() + } + + pub fn learners(&self) -> &[InitialMember] { + self.request.learners() + } +} + +impl RangeDirectory { + /// Resolve a physical placement by its complete history. The permanent + /// reservation only supplies candidate metadata: directory selection is + /// rechecked before materializing it, and bootstrap still requires a permit. + pub async fn materialize_selected_history( + &self, + history: GroupHistory, + ) -> Result { + let request = self.selected_history_request(history).await?; + if self.namespace.cell_group_owner().await?.is_some() { + self.observe_selected_birth(request).await + } else { + self.materialize_group_birth(&request).await + } + } + + /// Read an exact managed birth without creating a missing record. + pub(crate) async fn open_managed_selected_history( + &self, + history: GroupHistory, + owner: &crate::raft::catalog::ManagedBirthOwner, + ) -> Result { + if self.namespace.cell_group_owner().await?.as_ref() != Some(owner) { + return Err(CatalogError::ManagedAdmissionRequired); + } + let request = self.selected_history_request(history).await?; + self.observe_selected_birth(request).await + } + + async fn selected_history_request( + &self, + history: GroupHistory, + ) -> Result { + if history.cluster != self.namespace.config.cluster + || history.namespace != self.namespace.config.namespace + { + return Err(CatalogError::Conflict("group placement namespace")); + } + self.bounded(async { + let key = Path::from(format!( + "{}groups/{}", + self.namespace.inner.prefix, + hex::encode(history.group.bytes()) + )); + let (bytes, _) = self + .namespace + .inner + .read_versioned_bounded(&key, wire::LIMIT) + .await? + .ok_or(CatalogError::Corrupt("missing group birth reservation"))?; + let reservation: wire::BirthReservation = + record::decode(wire::GROUP_BIRTH_RESERVATION, &bytes, wire::LIMIT)?; + if reservation.recovery == Some([0; 32]) { + return Err(CatalogError::Corrupt("recovery reservation identity")); + } + let request = self.decode_group_birth(&reservation.birth)?; + if request.route().history != history { + return Err(CatalogError::Fenced); + } + Ok(request) + }) + .await + } + + async fn observe_selected_birth( + &self, + request: CellGroupBirthRequest, + ) -> Result { + let birth = CellGroupBirth { + request: Arc::new(request), + owner: self.namespace.inner.owner.clone(), + }; + self.require_current_group_birth(&birth).await?; + Ok(birth) + } + + /// Recover the immutable reservation for an already selected range. This + /// never chooses placement or allocates another group in response to a cell. + pub async fn materialize_selected_group( + &self, + route: &GroupRoute, + ) -> Result { + self.unmanaged_birth()?; + self.bounded(async { + self.selected_birth_route(route).await?; + let request = self.verify_birth_reservation(route).await?; + self.materialize_group_birth(&request).await + }) + .await + } + + pub(crate) fn same_owner(&self, other: &Self) -> bool { + Arc::ptr_eq(&self.cache, &other.cache) + } + + pub(super) fn unmanaged_birth(&self) -> Result<(), CatalogError> { + if self.namespace.config.managed_fleet.is_some() { + return Err(CatalogError::ManagedAdmissionRequired); + } + Ok(()) + } + + pub fn prepare_group_birth( + &self, + spec: CellGroupBirthSpec, + ) -> Result { + self.unmanaged_birth()?; + self.prepare_birth(spec) + } + + pub(crate) fn prepare_fleet_group_birth( + &self, + spec: CellGroupBirthSpec, + owner: &crate::raft::catalog::ManagedBirthOwner, + ) -> Result { + if self + .namespace + .config + .managed_fleet + .as_ref() + .is_some_and(|configured| configured != owner) + { + return Err(CatalogError::ManagedAdmissionRequired); + } + self.prepare_birth(spec) + } + + fn prepare_birth( + &self, + mut spec: CellGroupBirthSpec, + ) -> Result { + if spec.history.cluster != self.namespace.config.cluster + || spec.history.namespace != self.namespace.config.namespace + { + return Err(CatalogError::Conflict("group birth namespace")); + } + if spec.voters.len() > 32 || spec.learners.len() > 32 { + return Err(CatalogError::Limit); + } + spec.voters.sort_by_key(|m| m.member); + spec.learners.sort_by_key(|m| m.member); + self.namespace.check_members(&spec.voters, &spec.learners)?; + let value = wire::Birth { + namespace: self.namespace.root_digest, + group: spec.history.group.bytes(), + incarnation: spec.history.incarnation.bytes(), + lower: wire::Bound::from_bound(spec.range.lower()), + upper: wire::Bound::from_bound(spec.range.upper()), + epoch: spec.epoch.get(), + creation: spec.creation.bytes(), + voters: spec + .voters + .into_iter() + .map(record::Member::encode) + .collect(), + learners: spec + .learners + .into_iter() + .map(record::Member::encode) + .collect(), + }; + let request = + self.decode_group_birth(&record::encode(wire::GROUP_BIRTH, &value, wire::LIMIT)?)?; + if request.route.range != spec.range { + return Err(CatalogError::Invalid("noncanonical group birth range")); + } + Ok(request) + } + + pub fn decode_group_birth(&self, bytes: &[u8]) -> Result { + let value: wire::Birth = record::decode(wire::GROUP_BIRTH, bytes, wire::LIMIT)?; + if value.namespace != self.namespace.root_digest { + return Err(CatalogError::Conflict("group birth namespace")); + } + let voters = value + .voters + .iter() + .map(record::Member::decode) + .collect::, _>>()?; + let learners = value + .learners + .iter() + .map(record::Member::decode) + .collect::, _>>()?; + self.namespace.check_members(&voters, &learners)?; + let route = wire::Route { + group: value.group, + incarnation: value.incarnation, + lower: value.lower, + upper: value.upper, + epoch: value.epoch, + birth: record::digest(bytes), + } + .decode(self)?; + let creation = ProposalId::new(value.creation) + .map_err(|_| CatalogError::Corrupt("group birth creation"))?; + Ok(CellGroupBirthRequest { + bytes: bytes.to_vec(), + route, + creation, + voters, + learners, + }) + } + + fn reservation_key(&self, route: &GroupRoute) -> Path { + // Same permanent global key as the prior catalog: an occupied identity + // cannot be reused across namespaces, incarnations or semantic domains. + Path::from(format!( + "{}groups/{}", + self.namespace.inner.prefix, + hex::encode(route.history.group.bytes()) + )) + } + + pub(super) fn group_birth_key(&self, route: &GroupRoute) -> Path { + self.key(&format!( + "births/{}/{}", + hex::encode(route.history.group.bytes()), + hex::encode(route.history.incarnation.bytes()) + )) + } + + /// Permanently reserve before directory publication. An orphan reservation + /// is retained; a failed topology CAS does not release its identity. + pub async fn reserve_group_birth( + &self, + request: &CellGroupBirthRequest, + ) -> Result<(), CatalogError> { + self.unmanaged_birth()?; + self.reserve_birth_record( + request, + &wire::BirthReservation { + birth: request.encode().to_vec(), + recovery: None, + }, + ) + .await?; + self.bounded(self.namespace.admit_cell_group(request)).await + } + + pub(crate) async fn require_unreserved_fleet_group( + &self, + request: &CellGroupBirthRequest, + owner: &crate::raft::catalog::ManagedBirthOwner, + ) -> Result<(), CatalogError> { + if self.namespace.cell_group_owner().await?.as_ref() != Some(owner) { + return Err(CatalogError::ManagedAdmissionRequired); + } + self.bounded(async { + if self + .namespace + .inner + .read_versioned_bounded(&self.reservation_key(request.route()), wire::LIMIT) + .await? + .is_some() + { + return Err(CatalogError::Conflict("group birth identity reserved")); + } + Ok(()) + }) + .await + } + + /// A provisional permanent identity claim precedes the fleet root CAS. + /// Losing that CAS retains this claim; it grants no materialization, + /// bootstrap or quorum authority without the rooted birth capability. + pub(crate) async fn reserve_fleet_group_identity( + &self, + request: &CellGroupBirthRequest, + owner: &crate::raft::catalog::ManagedBirthOwner, + ) -> Result<(), CatalogError> { + if self.namespace.cell_group_owner().await?.as_ref() != Some(owner) { + return Err(CatalogError::ManagedAdmissionRequired); + } + self.reserve_birth_record( + request, + &wire::BirthReservation { + birth: request.encode().to_vec(), + recovery: None, + }, + ) + .await + } + + pub(crate) async fn reserve_managed_recovery_identity( + &self, + request: &CellGroupBirthRequest, + owner: &crate::raft::catalog::ManagedBirthOwner, + transition: ProposalId, + barrier: [u8; 32], + plan: [u8; 32], + ) -> Result<(), CatalogError> { + if plan == [0; 32] || self.namespace.cell_group_owner().await?.as_ref() != Some(owner) { + return Err(CatalogError::ManagedAdmissionRequired); + } + let view = self.observe().await?; + if view.digest != barrier + || !view + .head + .recovery + .as_ref() + .is_some_and(|lock| lock.owns_transition(transition)) + { + return Err(CatalogError::Fenced); + } + self.reserve_birth_record( + request, + &wire::BirthReservation { + birth: request.encode().to_vec(), + recovery: Some(plan), + }, + ) + .await + } + + pub(crate) async fn require_managed_recovery_identity( + &self, + request: &CellGroupBirthRequest, + plan: [u8; 32], + ) -> Result<(), CatalogError> { + self.bounded(async { + let (birth, origin) = self.birth_reservation(request.route()).await?; + if origin != Some(plan) || birth.encode() != request.encode() { + return Err(CatalogError::Fenced); + } + Ok(()) + }) + .await + } + + pub(super) async fn reserve_recovery_birth( + &self, + request: &super::CellGroupRecoveryRequest, + ) -> Result<(), CatalogError> { + self.reserve_birth_record( + request.replacement(), + &wire::BirthReservation { + birth: request.replacement().encode().to_vec(), + recovery: Some(request.digest()), + }, + ) + .await + } + + async fn reserve_birth_record( + &self, + request: &CellGroupBirthRequest, + reservation: &wire::BirthReservation, + ) -> Result<(), CatalogError> { + self.decode_group_birth(request.encode())?; + let bytes = record::encode(wire::GROUP_BIRTH_RESERVATION, reservation, wire::LIMIT)?; + self.bounded(async { + let observed = self + .namespace + .inner + .create_and_observe_bounded( + &self.reservation_key(request.route()), + &bytes, + wire::LIMIT, + ) + .await?; + if observed != bytes { + return Err(CatalogError::Conflict("group birth identity reserved")); + } + Ok(()) + }) + .await + } + + pub(super) async fn verify_birth_reservation( + &self, + route: &GroupRoute, + ) -> Result { + self.birth_reservation(route).await.map(|record| record.0) + } + + pub(super) async fn require_ordinary_birth_reservation( + &self, + route: &GroupRoute, + ) -> Result { + let (request, recovery) = self.birth_reservation(route).await?; + if recovery.is_some() { + return Err(CatalogError::Fenced); + } + if self.namespace.config.managed_fleet.is_none() + && self.namespace.cell_group_owner().await?.is_none() + { + self.namespace.verify_cell_group_intent(&request).await?; + } + Ok(request) + } + + pub(super) async fn birth_reservation( + &self, + route: &GroupRoute, + ) -> Result<(CellGroupBirthRequest, Option<[u8; 32]>), CatalogError> { + let bytes = self + .namespace + .inner + .read_versioned_bounded(&self.reservation_key(route), wire::LIMIT) + .await? + .ok_or(CatalogError::Corrupt("missing group birth reservation"))? + .0; + let reservation: wire::BirthReservation = + record::decode(wire::GROUP_BIRTH_RESERVATION, &bytes, wire::LIMIT)?; + if reservation.recovery == Some([0; 32]) { + return Err(CatalogError::Corrupt("recovery reservation identity")); + } + let request = self.decode_group_birth(&reservation.birth)?; + if request.route() != route { + return Err(CatalogError::Conflict("group birth reservation binding")); + } + Ok((request, reservation.recovery)) + } + + async fn selected_birth_route(&self, route: &GroupRoute) -> Result<(), CatalogError> { + let (view, _) = self.current().await?; + if let Some(lock) = &view.head.recovery { + if lock.blocks_startup(self, route)? { + return Err(CatalogError::Fenced); + } + } + self.selected_route_in(&view, route).await + } + + pub(super) async fn selected_route_in( + &self, + view: &DirectoryView, + route: &GroupRoute, + ) -> Result<(), CatalogError> { + let mut next = view.head.tree.clone(); + for _ in 0..32 { + let reference = next.take().ok_or(CatalogError::Fenced)?; + let page = self.page(&reference).await?; + let found = page.route.decode(self)?; + match route.range.lower().cmp(found.range.lower()) { + std::cmp::Ordering::Less => next = page.left, + std::cmp::Ordering::Greater => next = page.right, + std::cmp::Ordering::Equal => { + return if found == *route { + Ok(()) + } else { + Err(CatalogError::Fenced) + } + } + } + } + Err(CatalogError::Limit) + } + + /// Materialize the exact reserved birth only after directory selection. + /// This is lazy metadata creation, not local genesis or a bootstrap permit. + pub async fn materialize_group_birth( + &self, + request: &CellGroupBirthRequest, + ) -> Result { + self.unmanaged_birth()?; + self.namespace.require_cell_group_unmanaged().await?; + self.materialize_birth(request).await + } + + /// Only a rooted fleet reservation can create managed birth metadata. + /// Materialization supplies no voter, quorum or application authority. + pub async fn materialize_fleet_group_birth( + &self, + authorization: &crate::raft::fleet_catalog::FleetBirthAuthorization, + ) -> Result { + authorization.check_namespace(&self.namespace)?; + if self.namespace.cell_group_owner().await?.as_ref() != Some(authorization.owner()) { + return Err(CatalogError::ManagedAdmissionRequired); + } + let request = authorization.proposal(); + let decoded = self.decode_group_birth(request.encode())?; + if decoded.encode() != request.encode() { + return Err(CatalogError::Conflict("managed group birth request")); + } + self.bounded(self.require_ordinary_birth_reservation(request.route())) + .await?; + self.materialize_birth(&request).await + } + + pub(super) async fn materialize_import_birth( + &self, + request: &CellGroupBirthRequest, + ) -> Result { + self.materialize_birth(request).await + } + + async fn materialize_birth( + &self, + request: &CellGroupBirthRequest, + ) -> Result { + self.decode_group_birth(request.encode())?; + self.bounded(async { + self.selected_birth_route(request.route()).await?; + let birth = self.decide_live_birth(request).await?; + self.selected_birth_route(request.route()).await?; + Ok(birth) + }) + .await + } + + pub(super) async fn decide_live_birth( + &self, + request: &CellGroupBirthRequest, + ) -> Result { + self.verify_birth_reservation(request.route()).await?; + let observed = self + .namespace + .inner + .create_and_observe_bounded( + &self.group_birth_key(request.route()), + request.encode(), + wire::LIMIT, + ) + .await?; + match self.decode_group_disposition(request, &observed)? { + CellGroupBirthDisposition::Live(birth) => Ok(birth), + CellGroupBirthDisposition::Cancelled { .. } => Err(CatalogError::BirthCancelled), + } + } + + pub(super) async fn open_historical_birth( + &self, + route: &GroupRoute, + ) -> Result { + let request = self.verify_birth_reservation(route).await?; + let bytes = self + .namespace + .inner + .read_versioned_bounded(&self.group_birth_key(route), wire::LIMIT) + .await? + .ok_or(CatalogError::Corrupt("missing historical group birth"))? + .0; + match self.decode_group_disposition(&request, &bytes)? { + CellGroupBirthDisposition::Live(birth) => Ok(birth), + CellGroupBirthDisposition::Cancelled { .. } => Err(CatalogError::BirthCancelled), + } + } + + /// Recheck a birth observation's current directory and immutable metadata. + /// Future voter admission must additionally verify disk claims and quorum. + pub async fn require_current_group_birth( + &self, + birth: &CellGroupBirth, + ) -> Result<(), CatalogError> { + if !Arc::ptr_eq(&birth.owner, &self.namespace.inner.owner) { + return Err(CatalogError::Conflict("foreign group birth observation")); + } + self.bounded(async { + self.selected_birth_route(birth.route()).await?; + self.verify_birth_reservation(birth.route()).await?; + let bytes = self + .namespace + .inner + .read_versioned_bounded(&self.group_birth_key(birth.route()), wire::LIMIT) + .await? + .ok_or(CatalogError::Corrupt("missing selected group birth"))? + .0; + if bytes != birth.request.encode() { + return Err(CatalogError::Corrupt("selected group birth changed")); + } + Ok(()) + }) + .await + } +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/bootstrap.rs b/crates/cellar/durability/raft/cell_groups/directory/bootstrap.rs new file mode 100644 index 000000000..154c2ef55 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/bootstrap.rs @@ -0,0 +1,653 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Member admission after real genesis and a permanent physical claim. + +use super::*; +use crate::raft::{ + cache::GroupLog, + catalog::{DisasterFenceAuthority, InitialMember}, + storage::{CellGroupBootstrap, LogError, ReplicaLog}, +}; +use cellar_logic::consensus::MemberId; +use serde::{Deserialize, Serialize}; + +#[path = "joined_bootstrap.rs"] +mod joined; + +/// Durable member startup permission, not quorum or application-serving authority. +/// The shared runtime must verify the complete selected image/manifest closure +/// and revalidate this permit before attachment. It must obtain current-term +/// quorum authority separately before accepting application work. +pub struct CellGroupBootstrapPermit { + group: GroupLog, + birth: CellGroupBirth, + directory: RangeDirectory, + log: ReplicaLog, + origin: BootstrapOrigin, +} + +enum BootstrapOrigin { + Initial, + Managed(Box), + Joined, + ManagedJoined(Box), + Recovery(Box), + ManagedRecovery(Box), + PendingTransfer(PendingTransfer), + ManagedActivatedTransfer { + host: Box, + archive_pending: bool, + }, + ManagedSealedSource(Box), +} + +pub(super) enum PendingTransfer { + Unmanaged(Arc), + Managed { + transition: Arc, + host: Box, + deadline: u64, + }, +} + +impl CellGroupBootstrapPermit { + pub(in crate::raft::catalog::range_directory) fn recovered( + group: GroupLog, + birth: CellGroupBirth, + directory: RangeDirectory, + log: ReplicaLog, + startup: recovery::RecoveryStartup, + ) -> Self { + Self { + group, + birth, + directory, + log, + origin: BootstrapOrigin::Recovery(Box::new(startup)), + } + } + + pub(super) fn managed_pending_recovery( + group: GroupLog, + birth: CellGroupBirthRequest, + directory: RangeDirectory, + log: ReplicaLog, + startup: recovery::ManagedRecoveryStartup, + ) -> Self { + Self { + group, + birth: CellGroupBirth { + request: Arc::new(birth), + owner: directory.namespace.inner.owner.clone(), + }, + directory, + log, + origin: BootstrapOrigin::ManagedRecovery(Box::new(startup)), + } + } + + pub(super) fn pending_transfer( + group: GroupLog, + birth: CellGroupBirthRequest, + directory: RangeDirectory, + log: ReplicaLog, + origin: PendingTransfer, + ) -> Self { + Self { + group, + birth: CellGroupBirth { + request: Arc::new(birth), + owner: directory.namespace.inner.owner.clone(), + }, + directory, + log, + origin: BootstrapOrigin::PendingTransfer(origin), + } + } + + pub(super) fn managed_activated_transfer( + group: GroupLog, + birth: CellGroupBirth, + directory: RangeDirectory, + log: ReplicaLog, + host: crate::raft::fleet_catalog::FleetHostAuthorization, + archive_pending: bool, + ) -> Self { + Self { + group, + birth, + directory, + log, + origin: BootstrapOrigin::ManagedActivatedTransfer { + host: Box::new(host), + archive_pending, + }, + } + } + + pub(super) fn managed_sealed_source( + group: GroupLog, + birth: CellGroupBirth, + directory: RangeDirectory, + log: ReplicaLog, + host: crate::raft::fleet_catalog::FleetHostAuthorization, + ) -> Self { + Self { + group, + birth, + directory, + log, + origin: BootstrapOrigin::ManagedSealedSource(Box::new(host)), + } + } + + pub(crate) fn log(&self) -> &ReplicaLog { + &self.log + } + + pub(crate) fn managed_recovery_fenced(&self) -> bool { + matches!(self.origin, BootstrapOrigin::ManagedRecovery(_)) + } + + pub(crate) fn transfer_archive_pending(&self) -> bool { + matches!( + self.origin, + BootstrapOrigin::ManagedActivatedTransfer { + archive_pending: true, + .. + } + ) + } + + pub fn group(&self) -> &GroupLog { + &self.group + } + + pub fn route(&self) -> &GroupRoute { + self.birth.route() + } + + pub(crate) fn initial_voters(&self) -> &[InitialMember] { + self.birth.voters() + } + + pub(crate) fn recovery_request(&self) -> Option<&Arc> { + match &self.origin { + BootstrapOrigin::Recovery(startup) => Some(&startup.request), + _ => None, + } + } + + pub(crate) fn recovery_authority( + &self, + ) -> Option<&Arc>> { + match &self.origin { + BootstrapOrigin::Recovery(startup) => Some(&startup.authority), + _ => None, + } + } + + pub async fn revalidate(&self) -> Result<(), CatalogError> { + match &self.origin { + BootstrapOrigin::PendingTransfer(PendingTransfer::Unmanaged(plan)) => self + .directory + .bootstrap_pending_transfer_voter(plan, self.route().history, &self.log) + .await + .map(|_| ()), + BootstrapOrigin::PendingTransfer(PendingTransfer::Managed { + transition, + host, + deadline, + }) => self + .directory + .bootstrap_managed_pending_transfer_voter( + transition.clone(), + self.route(), + &self.log, + host, + *deadline, + ) + .await + .map(|_| ()), + BootstrapOrigin::ManagedActivatedTransfer { host, .. } => self + .directory + .bootstrap_managed_activated_transfer_voter(&self.birth, &self.log, host) + .await + .map(|_| ()), + BootstrapOrigin::ManagedSealedSource(host) => self + .directory + .bootstrap_managed_sealed_source(&self.birth, &self.log, host) + .await + .map(|_| ()), + BootstrapOrigin::Initial => self + .directory + .bootstrap_group_member(&self.birth, &self.log) + .await + .map(|_| ()), + BootstrapOrigin::Managed(host) => self + .directory + .bootstrap_managed_group_member(&self.birth, &self.log, host) + .await + .map(|_| ()), + BootstrapOrigin::Joined => self + .directory + .bootstrap_joined_group_member(&self.birth, self.group.identity().member, &self.log) + .await + .map(|_| ()), + BootstrapOrigin::ManagedJoined(host) => self + .directory + .bootstrap_managed_joined_group_member( + &self.birth, + self.group.identity().member, + &self.log, + host, + ) + .await + .map(|_| ()), + BootstrapOrigin::ManagedRecovery(startup) => self + .directory + .bootstrap_managed_pending_recovery_voter( + &startup.request, + self.route().history, + &self.log, + &startup.host, + startup.deadline, + ) + .await + .map(|_| ()), + BootstrapOrigin::Recovery(startup) => self + .directory + .bootstrap_recovered_group_member( + &startup.request, + &self.log, + startup.authority.clone(), + ) + .await + .map(|_| ()), + } + } +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Claim { + namespace: [u8; 32], + group: [u8; 16], + incarnation: [u8; 16], + birth: [u8; 32], + member: u64, + node: [u8; 16], + disk: [u8; 16], + engine: [u8; 16], +} + +pub(super) fn check_member(group: &GroupLog) -> Result<(), CatalogError> { + let cache = group.read().map_err(CatalogError::Log)?; + if !crate::raft::member_state::members(&cache.conf_state) + .contains(&group.identity().member.get()) + || !cache.membership.permits_group(group.identity()) + { + return Err(CatalogError::Conflict("group member is no longer admitted")); + } + Ok(()) +} + +impl RangeDirectory { + pub(crate) async fn verify_group_import_member_claim( + &self, + birth: &CellGroupBirth, + member: MemberId, + host: &crate::raft::fleet_catalog::FleetHostAuthorization, + ) -> Result<(), CatalogError> { + let history = birth.route().history; + let bytes = self + .namespace + .inner + .read_versioned_bounded(&self.group_claim_key(history, member), wire::LIMIT) + .await? + .ok_or(CatalogError::Conflict("missing group import member claim"))? + .0; + let claim: Claim = record::decode(wire::GROUP_CLAIM, &bytes, wire::LIMIT)?; + if claim.namespace != self.namespace.root_digest + || claim.group != history.group.bytes() + || claim.incarnation != history.incarnation.bytes() + || claim.birth != birth.route().birth_digest + || claim.member != member.get() + || claim.node != host.node().bytes() + || claim.disk != host.disk().bytes() + || Some(claim.engine) != host.storage_instance().map(|id| id.bytes()) + { + return Err(CatalogError::Conflict("group import member claim identity")); + } + Ok(()) + } + + fn group_claim_key(&self, history: GroupHistory, member: MemberId) -> Path { + self.key(&format!( + "claims/{}/{}/{:016x}", + hex::encode(history.group.bytes()), + hex::encode(history.incarnation.bytes()), + member.get() + )) + } + + /// Initialize or resume an initial member using its exact immutable birth. + /// Unknown outcomes retain the same history, disk and engine identities. + /// Managed admission remains closed until rooted fleet authorization exists. + pub async fn bootstrap_group_member( + &self, + birth: &CellGroupBirth, + log: &ReplicaLog, + ) -> Result { + self.unmanaged_birth()?; + self.bounded(async { + self.namespace.require_cell_group_unmanaged().await?; + self.bootstrap_initial_group_member(birth, log, None).await + }) + .await + } + + /// Recheck the exact running node/engine boot and the rooted Live group on + /// every startup and attachment. A saved birth or binding is insufficient. + pub async fn bootstrap_managed_group_member( + &self, + birth: &CellGroupBirth, + log: &ReplicaLog, + host: &crate::raft::fleet_catalog::FleetHostAuthorization, + ) -> Result { + self.bounded(async { + match log.cell_group(&birth.route().history) { + Ok(group) => { + if let Some(marker) = group + .group_manifest() + .map_err(CatalogError::Log)? + .filter(|marker| marker.recovery_request().is_some()) + { + Box::pin(self.require_serving_managed_recovery_member( + birth, &group, &marker, log, host, + )) + .await?; + return Ok(CellGroupBootstrapPermit { + group, + birth: birth.clone(), + directory: self.clone(), + log: log.clone(), + origin: BootstrapOrigin::Managed(Box::new(host.clone())), + }); + } + self.require_managed_retained_group_boot(birth, log, host) + .await?; + } + Err(LogError::NotFound) => { + self.require_managed_group_boot(log, host).await?; + } + Err(error) => return Err(CatalogError::Log(error)), + } + self.bootstrap_initial_group_member(birth, log, Some(host)) + .await + }) + .await + } + + pub(super) async fn require_managed_group_boot( + &self, + log: &ReplicaLog, + host: &crate::raft::fleet_catalog::FleetHostAuthorization, + ) -> Result { + let boot = self.require_managed_boot_identity(log, host).await?; + if log + .lifecycle_guard() + .map_err(CatalogError::Log)? + .is_none_or(|guard| !guard.admits_group_start(&boot)) + { + return Err(CatalogError::ManagedAdmissionRequired); + } + Ok(boot) + } + + pub(super) async fn require_managed_retained_group_boot( + &self, + birth: &CellGroupBirth, + log: &ReplicaLog, + host: &crate::raft::fleet_catalog::FleetHostAuthorization, + ) -> Result { + let boot = self.require_managed_boot_identity(log, host).await?; + if log + .lifecycle_guard() + .map_err(CatalogError::Log)? + .is_some_and(|guard| guard.admits_group_start(&boot)) + { + return Ok(boot); + } + let guard = log + .lifecycle_guard() + .map_err(CatalogError::Log)? + .ok_or(CatalogError::ManagedAdmissionRequired)?; + let selector = guard + .quiescing() + .ok_or(CatalogError::ManagedAdmissionRequired)? + .clone(); + let group = log + .cell_group(&birth.route().history) + .map_err(CatalogError::Log)?; + host.require_retained_group_start( + &selector, + group.identity(), + Duration::from_millis(self.limits.operation_timeout_ms), + ) + .await + .map_err(CatalogError::from)?; + let current = log + .lifecycle_guard() + .map_err(CatalogError::Log)? + .ok_or(CatalogError::Fenced)?; + if current.boot != boot || current.quiescing() != Some(&selector) { + return Err(CatalogError::Fenced); + } + Ok(boot) + } + + async fn require_managed_boot_identity( + &self, + log: &ReplicaLog, + host: &crate::raft::fleet_catalog::FleetHostAuthorization, + ) -> Result { + let owner = self + .namespace + .cell_group_owner() + .await? + .ok_or(CatalogError::ManagedAdmissionRequired)?; + let boot = host.boot_binding().map_err(CatalogError::from)?; + if owner.fleet != host.fleet() + || owner.catalog != boot.catalog + || host.cluster() != self.namespace.config.cluster + || host.node() != log.limits().node + || host.disk() != log.limits().disk + || host.storage_instance() != Some(log.storage_instance().map_err(CatalogError::Log)?) + || boot.process != log.limits().process.bytes() + || log + .lifecycle_guard() + .map_err(CatalogError::Log)? + .is_none_or(|guard| guard.boot != boot) + { + return Err(CatalogError::ManagedAdmissionRequired); + } + Ok(boot) + } + + async fn bootstrap_initial_group_member( + &self, + birth: &CellGroupBirth, + log: &ReplicaLog, + host: Option<&crate::raft::fleet_catalog::FleetHostAuthorization>, + ) -> Result { + self.bounded(async { + let authorization = if let Some(host) = host { + let authorization = host + .group_authorization( + &birth.route().history, + Duration::from_millis(self.limits.operation_timeout_ms), + ) + .await + .map_err(CatalogError::from)?; + if !authorization.check_birth(self, birth) { + return Err(CatalogError::ManagedAdmissionRequired); + } + Some(authorization) + } else { + None + }; + self.require_current_group_birth(birth).await?; + self.require_non_transfer_birth(birth.route()).await?; + self.require_ordinary_birth_reservation(birth.route()) + .await?; + let selected = birth + .voters() + .iter() + .chain(birth.learners()) + .find(|member| member.node == log.limits().node) + .ok_or(CatalogError::NotInitialMember)?; + // This permanent node pin refuses a recreated engine even before a + // member claim has been written. Keep the existing pin protocol. + let _node = self.namespace.require_node_storage(log).await?; + let history = birth.route().history; + let expected = record::encode( + wire::GROUP_CLAIM, + &Claim { + namespace: self.namespace.root_digest, + group: history.group.bytes(), + incarnation: history.incarnation.bytes(), + birth: birth.route().birth_digest, + member: selected.member.get(), + node: selected.node.bytes(), + disk: log.limits().disk.bytes(), + engine: log.storage_instance().map_err(CatalogError::Log)?.bytes(), + }, + wire::LIMIT, + )?; + let key = self.group_claim_key(history, selected.member); + let remote = self + .namespace + .inner + .read_versioned_bounded(&key, wire::LIMIT) + .await?; + if remote.as_ref().is_some_and(|(bytes, _)| bytes != &expected) { + return Err(CatalogError::Conflict("permanent group member claim")); + } + let local = match log.cell_group(&history) { + Ok(group) => Some(group), + Err(LogError::NotFound) => None, + Err(error) => return Err(CatalogError::Log(error)), + }; + if remote.is_some() && local.is_none() { + return Err(CatalogError::LostStableState); + } + if local.is_none() + && authorization + .as_ref() + .is_some_and(|authority| !authority.allows_managed_genesis()) + { + return Err(CatalogError::ManagedAdmissionRequired); + } + // The worker creates the complete empty root with genesis and only + // returns after actual engine sync. Existing mappings must match + // the original route, membership and birth, even after later apply. + let bootstrap = CellGroupBootstrap { + route: birth.route().clone(), + member: selected.member, + voters: birth.voters().iter().map(|m| m.member).collect(), + learners: birth.learners().iter().map(|m| m.member).collect(), + }; + let paused_retained = if let Some(host) = host { + let boot = host.boot_binding().map_err(CatalogError::from)?; + let guard = log + .lifecycle_guard() + .map_err(CatalogError::Log)? + .ok_or(CatalogError::ManagedAdmissionRequired)?; + !guard.admits_group_start(&boot) + } else { + false + }; + let group = if paused_retained { + // A returned paused boot only reattaches retained state. It + // cannot complete genesis or create a missing permanent claim. + if remote.is_none() { + return Err(CatalogError::LostStableState); + } + let host = host.ok_or(CatalogError::ManagedAdmissionRequired)?; + self.require_managed_retained_group_boot(birth, log, host) + .await?; + let group = local.ok_or(CatalogError::LostStableState)?; + { + let cache = group.read().map_err(CatalogError::Log)?; + let authorization = authorization + .as_ref() + .ok_or(CatalogError::ManagedAdmissionRequired)?; + if group.identity().member != selected.member + || cache + .group_manifest + .as_ref() + .is_none_or(|root| root.route() != birth.route()) + || cache + .membership + .fleet_status() + .is_none_or(|status| &status.binding != authorization.binding()) + || !authorization.matches_replica(group.identity(), &cache.membership) + { + return Err(CatalogError::ManagedAdmissionRequired); + } + } + group + } else { + match &authorization { + Some(authorization) => { + let boot = host + .ok_or(CatalogError::ManagedAdmissionRequired)? + .boot_binding() + .map_err(CatalogError::from)?; + log.create_managed_cell_group(bootstrap, authorization, &boot) + .await + } + None => log.create_cell_group(bootstrap).await, + } + .map_err(CatalogError::Log)? + }; + check_member(&group)?; + let observed = if paused_retained { + // Reobserve without a conditional create: a missing claim + // during return must refuse, including after the earlier read. + self.namespace + .inner + .read_versioned_bounded(&key, wire::LIMIT) + .await? + .ok_or(CatalogError::LostStableState)? + .0 + } else { + self.namespace + .inner + .create_and_observe_bounded(&key, &expected, wire::LIMIT) + .await? + }; + if observed != expected { + return Err(CatalogError::Conflict("permanent group member claim")); + } + self.require_current_group_birth(birth).await?; + check_member(&group)?; + if let Some(host) = host { + self.require_managed_retained_group_boot(birth, log, host) + .await?; + } + Ok(CellGroupBootstrapPermit { + group, + birth: birth.clone(), + directory: self.clone(), + log: log.clone(), + origin: match host { + Some(host) => BootstrapOrigin::Managed(Box::new(host.clone())), + None => BootstrapOrigin::Initial, + }, + }) + }) + .await + } +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/import.rs b/crates/cellar/durability/raft/cell_groups/directory/import.rs new file mode 100644 index 000000000..865bad870 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/import.rs @@ -0,0 +1,368 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Namespace import first closes admission, then CAS-locks topology. Delayed +//! topology writers lose their exact expected head; neither effect auto-expires. +use super::*; +use crate::raft::catalog::ManagedBirthOwner; +use crate::raft::catalog::{CellGroupImportCut, FrozenCellGroupInventory}; +use serde::{Deserialize, Serialize}; + +const CANCELLED: u8 = 197; + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Cancelled { + namespace: [u8; 32], + birth: [u8; 32], + operation: [u8; 16], +} + +/// An immutable exact-key decision. Cancellation retains the admission and +/// permanent identity; it grants no removal or physical-release permission. +pub enum CellGroupBirthDisposition { + Live(CellGroupBirth), + Cancelled { + history: GroupHistory, + operation: ProposalId, + }, +} + +impl RangeDirectory { + /// Settle an unselected managed reservation at the same immutable decision + /// key as materialization. A selected or already live birth remains Live; + /// cancellation never erases a possible replica or releases its resources. + pub async fn cancel_fleet_group_birth( + &self, + authorization: &crate::raft::fleet_catalog::FleetBirthAuthorization, + operation: ProposalId, + ) -> Result { + authorization.check_namespace(&self.namespace)?; + self.bounded(async { + if self.namespace.cell_group_owner().await?.as_ref() != Some(authorization.owner()) { + return Err(CatalogError::ManagedAdmissionRequired); + } + let request = authorization.proposal(); + self.verify_birth_reservation(request.route()).await?; + if let Some(winner) = self.group_birth_disposition(&request).await? { + return Ok(winner); + } + let (view, _) = self.current().await?; + match self.selected_route_in(&view, request.route()).await { + Ok(()) => { + return self + .materialize_import_birth(&request) + .await + .map(CellGroupBirthDisposition::Live) + } + Err(CatalogError::Fenced) => (), + Err(error) => return Err(error), + } + let cancelled = record::encode( + CANCELLED, + &Cancelled { + namespace: self.namespace.root_digest, + birth: request.route().birth_digest, + operation: operation.bytes(), + }, + wire::LIMIT, + )?; + let winner = self + .namespace + .inner + .create_and_observe_bounded( + &self.group_birth_key(request.route()), + &cancelled, + wire::LIMIT, + ) + .await?; + self.decode_group_disposition(&request, &winner) + }) + .await + } + + /// Freeze the complete admitted group set, including unselected and lazy + /// births. Retry the same operation after loss. Already running groups keep + /// their Raft write path; new topology and birth admission remain closed. + pub async fn freeze_group_births( + &self, + operation: ProposalId, + ) -> Result { + self.unmanaged_birth()?; + self.bounded(async { + let (initial, _) = self.current().await?; + if self.transition_active(&initial.head).await?.is_some() { + return Err(CatalogError::Fenced); + } + if self + .transition_record(&initial.head, operation) + .await? + .is_some() + { + return Err(CatalogError::Conflict("transition operation reused")); + } + self.namespace.begin_cell_group_freeze(operation).await?; + for _ in 0..8 { + let (mut view, version) = self.current().await?; + if view.head.recovery.is_some() + || self.transition_active(&view.head).await?.is_some() + { + return Err(CatalogError::Fenced); + } + if let Some(prior) = view.head.import { + if prior != operation.bytes() { + return Err(CatalogError::Conflict("directory import operation")); + } + let bytes = record::encode(wire::HEAD, &view.head, wire::LIMIT)?; + let digest = record::digest(&bytes); + self.immutable("import-cuts", digest, &bytes).await?; + return self + .namespace + .finish_cell_group_freeze(operation, digest) + .await; + } + if self + .transition_record(&view.head, operation) + .await? + .is_some() + { + return Err(CatalogError::Conflict("transition operation reused")); + } + let operation_key = hex::encode(operation.bytes()); + if Index::new(self) + .lookup(view.head.operations, &operation_key) + .await? + .is_some() + { + return Err(CatalogError::Conflict("directory import operation reused")); + } + let revision = view + .head + .revision + .checked_add(1) + .ok_or(CatalogError::Limit)?; + let journal = wire::Journal { + namespace: self.namespace.root_digest, + operation: operation.bytes(), + // The previous authenticated head is the exact freeze + // input. Its record domain differs from a range request. + request: view.digest, + revision, + previous: Some(view.head.journal), + transfer_archive: None, + tree: view.head.tree.clone(), + }; + let journal_bytes = record::encode(wire::JOURNAL, &journal, wire::LIMIT)?; + let journal_id = record::digest(&journal_bytes); + self.immutable("journal", journal_id, &journal_bytes) + .await?; + view.head.operations = Index::new(self) + .update( + view.head.operations, + &operation_key, + Some(journal_id.to_vec()), + 0, + ) + .await?; + view.head.import = Some(operation.bytes()); + view.head.revision = revision; + view.head.journal = journal_id; + self.check_head(&view.head)?; + let bytes = record::encode(wire::HEAD, &view.head, wire::LIMIT)?; + let _ = self + .namespace + .inner + .store + .put_opts( + &self.key("head"), + bytes.into(), + PutOptions::from(PutMode::Update(version)), + ) + .await; + // Read back even after a lost CAS response. A missing head is + // ambiguous; it never reopens admission or chooses another ID. + let (observed, _) = self.current().await?; + if observed + .head + .import + .is_some_and(|prior| prior != operation.bytes()) + { + return Err(CatalogError::Conflict("directory import winner")); + } + } + Err(CatalogError::Unconfirmed) + }) + .await + } + + pub(crate) async fn import_directory_view( + &self, + cut: &CellGroupImportCut, + ) -> Result { + cut.validate(&self.namespace)?; + let bytes = self + .read(&format!("import-cuts/{}", hex::encode(cut.directory))) + .await? + .ok_or(CatalogError::Corrupt("missing frozen group directory"))?; + if record::digest(&bytes) != cut.directory { + return Err(CatalogError::Corrupt("frozen group directory digest")); + } + let head: wire::Head = record::decode(wire::HEAD, &bytes, wire::LIMIT)?; + self.check_head(&head)?; + if head.import != Some(cut.operation) || head.recovery.is_some() { + return Err(CatalogError::Corrupt("frozen group directory operation")); + } + Ok(DirectoryView { + head, + digest: cut.directory, + owner: self.namespace.inner.owner.clone(), + }) + } + + pub(crate) async fn check_group_import( + &self, + cut: &CellGroupImportCut, + owner: &ManagedBirthOwner, + adopted: bool, + ) -> Result<(), CatalogError> { + self.namespace + .check_cell_group_import(cut, owner, adopted) + .await?; + self.import_directory_view(cut).await?; + if !adopted { + let (current, _) = self.current().await?; + if current.digest != cut.directory || current.head.import != Some(cut.operation) { + // A Handoff retry may observe the already adopted and unlocked + // directory after losing the final fleet-root response. + if current.head.import.is_some() { + return Err(CatalogError::Conflict("changed import topology")); + } + self.namespace + .check_cell_group_import(cut, owner, true) + .await?; + } + } + Ok(()) + } + + pub(crate) async fn finish_group_import( + &self, + cut: &CellGroupImportCut, + owner: &ManagedBirthOwner, + ) -> Result<(), CatalogError> { + self.namespace + .change_cell_group_import(cut, owner, true) + .await?; + for _ in 0..8 { + let (mut view, version) = self.current().await?; + if view.head.import.is_none() { + return self.check_group_import(cut, owner, true).await; + } + if view.digest != cut.directory || view.head.import != Some(cut.operation) { + return Err(CatalogError::Conflict("import handoff directory")); + } + view.head.import = None; + view.head.revision = view + .head + .revision + .checked_add(1) + .ok_or(CatalogError::Limit)?; + self.check_head(&view.head)?; + let bytes = record::encode(wire::HEAD, &view.head, wire::LIMIT)?; + let _ = self + .namespace + .inner + .store + .put_opts( + &self.key("head"), + bytes.into(), + PutOptions::from(PutMode::Update(version)), + ) + .await; + } + Err(CatalogError::Unconfirmed) + } + + pub(crate) fn decode_group_disposition( + &self, + request: &CellGroupBirthRequest, + bytes: &[u8], + ) -> Result { + if bytes == request.encode() { + return Ok(CellGroupBirthDisposition::Live(CellGroupBirth { + request: Arc::new(request.clone()), + owner: self.namespace.inner.owner.clone(), + })); + } + let cancelled: Cancelled = record::decode(CANCELLED, bytes, wire::LIMIT)?; + if cancelled.namespace != self.namespace.root_digest + || cancelled.birth != request.route().birth_digest + { + return Err(CatalogError::Corrupt("cancelled group birth identity")); + } + let operation = ProposalId::new(cancelled.operation) + .map_err(|_| CatalogError::Corrupt("group cancellation operation"))?; + Ok(CellGroupBirthDisposition::Cancelled { + history: request.route().history, + operation, + }) + } + + pub(crate) async fn group_birth_disposition( + &self, + request: &CellGroupBirthRequest, + ) -> Result, CatalogError> { + self.verify_birth_reservation(request.route()).await?; + let bytes = self + .namespace + .inner + .read_versioned_bounded(&self.group_birth_key(request.route()), wire::LIMIT) + .await?; + bytes + .map(|(bytes, _)| self.decode_group_disposition(request, &bytes)) + .transpose() + } + + pub(crate) async fn settle_import_group( + &self, + request: &CellGroupBirthRequest, + cut: &CellGroupImportCut, + owner: &ManagedBirthOwner, + ) -> Result { + self.check_group_import(cut, owner, false).await?; + self.namespace.verify_cell_group_intent(request).await?; + let view = self.import_directory_view(cut).await?; + // Every selected range must remain live. A lazy selected birth is + // materialized, never confused with an unselected cancelled intent. + match self.selected_route_in(&view, request.route()).await { + Ok(()) => { + return self + .materialize_import_birth(request) + .await + .map(CellGroupBirthDisposition::Live) + } + Err(CatalogError::Fenced) => (), + Err(error) => return Err(error), + } + let cancelled = record::encode( + CANCELLED, + &Cancelled { + namespace: self.namespace.root_digest, + birth: request.route().birth_digest, + operation: cut.operation, + }, + wire::LIMIT, + )?; + let bytes = self + .namespace + .inner + .create_and_observe_bounded( + &self.group_birth_key(request.route()), + &cancelled, + wire::LIMIT, + ) + .await?; + // A materializer that won before the topology lock keeps its original + // immutable live decision. It must be covered too, even if unselected. + self.decode_group_disposition(request, &bytes) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/inventory.rs b/crates/cellar/durability/raft/cell_groups/directory/inventory.rs new file mode 100644 index 000000000..52eb1c849 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/inventory.rs @@ -0,0 +1,86 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Bounded physical-range enumeration at one immutable directory observation. +//! A complete traversal is metadata inventory, never serving or release proof. +use super::*; + +/// Continuation for one exact observed tree. It cannot be rebased onto a later +/// root; restart an independent inventory after a topology change. +#[derive(Clone, Debug)] +pub struct CellGroupDirectoryCursor { + root: [u8; 32], + after: CellBound, +} + +pub struct CellGroupDirectoryPage { + pub routes: Vec, + pub next: Option, +} + +impl RangeDirectory { + /// Enumerate physical groups without scanning cells or listing S3 objects. + /// Each call retains at most `limit` routes and one tree path. The cursor + /// binds the immutable view, including its namespace and topology revision. + pub async fn group_page( + &self, + view: &DirectoryView, + after: Option<&CellGroupDirectoryCursor>, + limit: usize, + ) -> Result { + self.check_view(view)?; + if !(1..=128).contains(&limit) { + return Err(CatalogError::Limit); + } + if after.is_some_and(|cursor| cursor.root != view.digest) { + return Err(CatalogError::Conflict("directory inventory cursor")); + } + self.bounded(async { + let mut lower = after.map(|cursor| cursor.after.clone()); + let mut routes = Vec::with_capacity(limit); + for _ in 0..limit { + let Some(route) = self.successor(view, lower.as_ref()).await? else { + return Ok(CellGroupDirectoryPage { routes, next: None }); + }; + lower = Some(route.range.lower().clone()); + routes.push(route); + } + let next = if self.successor(view, lower.as_ref()).await?.is_some() { + Some(CellGroupDirectoryCursor { + root: view.digest, + after: lower.ok_or(CatalogError::Corrupt("empty directory inventory page"))?, + }) + } else { + None + }; + Ok(CellGroupDirectoryPage { routes, next }) + }) + .await + } + + async fn successor( + &self, + view: &DirectoryView, + after: Option<&CellBound>, + ) -> Result, CatalogError> { + let mut next = view.head.tree.clone(); + let mut candidate = None; + for _ in 0..32 { + let Some(reference) = next.take() else { + return Ok(candidate); + }; + let page = self.page(&reference).await?; + let route = page.route.decode(self)?; + if after.is_none_or(|lower| route.range.lower() > lower) { + candidate = Some(route); + next = page.left; + } else { + next = page.right; + } + } + if next.is_none() { + Ok(candidate) + } else { + Err(CatalogError::Limit) + } + } +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/joined_bootstrap.rs b/crates/cellar/durability/raft/cell_groups/directory/joined_bootstrap.rs new file mode 100644 index 000000000..ac4010c8b --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/joined_bootstrap.rs @@ -0,0 +1,330 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! A staging mapping never grants startup. Only a locally fenced authenticated +//! install plus the exact permanent remote claim can resume a joined member. +use super::*; +use crate::raft::{ + fleet_catalog::{FleetHostAuthorization, FleetPermitAuthorization}, + transport::CellGroupSeedGrant, +}; + +impl RangeDirectory { + pub(crate) async fn stage_group_learner( + &self, + birth: &CellGroupBirth, + log: &ReplicaLog, + grant: Arc, + ) -> Result, CatalogError> { + self.unmanaged_birth()?; + self.bounded(async { + self.namespace.require_cell_group_unmanaged().await?; + self.stage_joined_group(birth, log, grant, None).await + }) + .await + } + + pub(crate) async fn stage_managed_group_learner( + &self, + birth: &CellGroupBirth, + log: &ReplicaLog, + grant: Arc, + host: &FleetHostAuthorization, + ) -> Result, CatalogError> { + self.stage_joined_group(birth, log, grant, Some(host)).await + } + + /// The same active physical reservation must still name this boot. The + /// returned capability does not substitute for the authenticated seed grant. + pub(crate) async fn authorize_group_learner_seed( + &self, + birth: &CellGroupBirth, + log: &ReplicaLog, + target: &cellar_logic::consensus::ReplicaIdentity, + source: &cellar_logic::consensus::ReplicaIdentity, + host: &FleetHostAuthorization, + deadline: u64, + ) -> Result { + let remaining = || -> Result { + let now = log.domain().mono_ms().map_err(|_| CatalogError::Fenced)?; + deadline + .checked_sub(now) + .filter(|ms| *ms > 0) + .map(|ms| Duration::from_millis(ms.min(self.limits.operation_timeout_ms))) + .ok_or(CatalogError::Fenced) + }; + crate::asyncrt::timeout(remaining()?, async { + let boot = self.require_managed_group_boot(log, host).await?; + let authorization = host + .admitted_learner_from(&birth.route().history, source, remaining()?) + .await + .map_err(CatalogError::from)?; + let group = host + .group_authorization_at_least( + &birth.route().history, + authorization.minimum_trust_revision(), + remaining()?, + ) + .await + .map_err(CatalogError::from)?; + let replacement = authorization + .permit() + .replacement() + .map_err(CatalogError::Log)?; + if !group.check_birth(self, birth) + || !authorization.matches_scope(&target.scope) + || target.scope != birth.route().history + || authorization.target_boot() != Some(&boot) + || replacement.member != target.member + || replacement.node != target.node + || replacement.disk != target.disk + || target.node != log.limits().node + || target.disk != log.limits().disk + || target.process != log.limits().process + { + return Err(CatalogError::ManagedAdmissionRequired); + } + self.require_current_group_birth(birth).await?; + remaining()?; + self.require_managed_group_boot(log, host).await?; + remaining()?; + Ok(authorization) + }) + .await + .map_err(|_| CatalogError::Unconfirmed)? + } + + async fn stage_joined_group( + &self, + birth: &CellGroupBirth, + log: &ReplicaLog, + grant: Arc, + host: Option<&FleetHostAuthorization>, + ) -> Result, CatalogError> { + self.bounded(async { + grant.check_log(log).map_err(CatalogError::Log)?; + if let Some(host) = host { + let current = self + .authorize_group_learner_seed( + birth, + log, + grant.local(), + grant.source(), + host, + grant.deadline(), + ) + .await?; + grant + .check_authorization(¤t) + .map_err(CatalogError::Log)?; + } else if grant.managed_binding().is_some() { + return Err(CatalogError::ManagedAdmissionRequired); + } + self.require_current_group_birth(birth).await?; + let _node = self.namespace.require_node_storage(log).await?; + let member = grant.local().member; + let bootstrap = CellGroupBootstrap { + route: birth.route().clone(), + member, + voters: birth.voters().iter().map(|m| m.member).collect(), + learners: birth.learners().iter().map(|m| m.member).collect(), + }; + grant + .initial_configuration(&bootstrap) + .map_err(CatalogError::Log)?; + let expected = self.joined_claim(birth, member, log)?; + let key = self.group_claim_key(birth.route().history, member); + let remote = self + .namespace + .inner + .read_versioned_bounded(&key, wire::LIMIT) + .await?; + if remote.as_ref().is_some_and(|(bytes, _)| bytes != &expected) { + return Err(CatalogError::Conflict("permanent group member claim")); + } + match log.cell_group(&birth.route().history) { + Err(LogError::NotFound) if remote.is_some() => { + return Err(CatalogError::LostStableState) + } + Err(LogError::NotFound) | Ok(_) => (), + Err(error) => return Err(CatalogError::Log(error)), + } + // Stage before publishing the remote claim. An interrupted operation + // can reconcile this same engine; an existing claim can never reset it. + let group = log + .stage_cell_group(bootstrap, grant.clone()) + .await + .map_err(CatalogError::Log)?; + let observed = self + .namespace + .inner + .create_and_observe_bounded(&key, &expected, wire::LIMIT) + .await?; + if observed != expected { + return Err(CatalogError::Conflict("permanent group member claim")); + } + self.require_current_group_birth(birth).await?; + grant.check_log(log).map_err(CatalogError::Log)?; + if let Some(host) = host { + let current = self + .authorize_group_learner_seed( + birth, + log, + grant.local(), + grant.source(), + host, + grant.deadline(), + ) + .await?; + grant + .check_authorization(¤t) + .map_err(CatalogError::Log)?; + } + Ok(group) + }) + .await + } + + fn joined_claim( + &self, + birth: &CellGroupBirth, + member: MemberId, + log: &ReplicaLog, + ) -> Result, CatalogError> { + let history = birth.route().history; + record::encode( + wire::GROUP_CLAIM, + &Claim { + namespace: self.namespace.root_digest, + group: history.group.bytes(), + incarnation: history.incarnation.bytes(), + birth: birth.route().birth_digest, + member: member.get(), + node: log.limits().node.bytes(), + disk: log.limits().disk.bytes(), + engine: log.storage_instance().map_err(CatalogError::Log)?.bytes(), + }, + wire::LIMIT, + ) + } + + /// Resume a previously installed joined member. A staged empty mapping, an + /// imported snapshot or a recreated disk cannot substitute for the local + /// authenticated-install receipt and permanent engine claim. The runtime + /// still verifies the whole physical closure before attaching this permit. + pub async fn bootstrap_joined_group_member( + &self, + birth: &CellGroupBirth, + member: MemberId, + log: &ReplicaLog, + ) -> Result { + self.unmanaged_birth()?; + self.bounded(async { + self.namespace.require_cell_group_unmanaged().await?; + self.bootstrap_joined_member(birth, member, log, None).await + }) + .await + } + + /// A retained install receipt and permanent engine claim are mandatory even + /// after a repair completes. A saved active permit is not restart authority. + pub async fn bootstrap_managed_joined_group_member( + &self, + birth: &CellGroupBirth, + member: MemberId, + log: &ReplicaLog, + host: &FleetHostAuthorization, + ) -> Result { + self.bootstrap_joined_member(birth, member, log, Some(host)) + .await + } + + async fn bootstrap_joined_member( + &self, + birth: &CellGroupBirth, + member: MemberId, + log: &ReplicaLog, + host: Option<&FleetHostAuthorization>, + ) -> Result { + self.bounded(async { + let authorization = if let Some(host) = host { + self.require_managed_retained_group_boot(birth, log, host) + .await?; + let group = host + .group_authorization( + &birth.route().history, + Duration::from_millis(self.limits.operation_timeout_ms), + ) + .await + .map_err(CatalogError::from)?; + if !group.check_birth(self, birth) { + return Err(CatalogError::ManagedAdmissionRequired); + } + Some(group) + } else { + None + }; + self.require_current_group_birth(birth).await?; + let _node = self.namespace.require_node_storage(log).await?; + let group = log + .cell_group(&birth.route().history) + .map_err(CatalogError::Log)?; + let check = || -> Result<(), CatalogError> { + let cache = group.read().map_err(CatalogError::Log)?; + let joined = cache.membership.joined(member).map_err(CatalogError::Log)?; + if group.identity().member != member + || cache + .group_manifest + .as_ref() + .is_none_or(|m| m.route() != birth.route()) + || joined.is_none_or(|(binding, at)| { + binding.node != log.limits().node + || binding.disk != log.limits().disk + || cache.group_seed.is_none_or(|seed| seed < at.get()) + }) + { + return Err(CatalogError::Conflict( + "joined group has no durable seed receipt", + )); + } + match (&authorization, cache.membership.fleet_status()) { + (Some(authorization), Some(status)) + if &status.binding == authorization.binding() + && authorization + .matches_replica(group.identity(), &cache.membership) => {} + (None, None) => (), + _ => return Err(CatalogError::ManagedAdmissionRequired), + } + drop(cache); + check_member(&group) + }; + check()?; + let key = self.group_claim_key(birth.route().history, member); + let expected = self.joined_claim(birth, member, log)?; + let remote = self + .namespace + .inner + .read_versioned_bounded(&key, wire::LIMIT) + .await?; + if remote.is_none_or(|(bytes, _)| bytes != expected) { + return Err(CatalogError::Conflict("permanent group member claim")); + } + self.require_current_group_birth(birth).await?; + check()?; + if let Some(host) = host { + self.require_managed_retained_group_boot(birth, log, host) + .await?; + } + Ok(CellGroupBootstrapPermit { + group, + birth: birth.clone(), + directory: self.clone(), + log: log.clone(), + origin: match host { + Some(host) => BootstrapOrigin::ManagedJoined(Box::new(host.clone())), + None => BootstrapOrigin::Joined, + }, + }) + }) + .await + } +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/mod.rs b/crates/cellar/durability/raft/cell_groups/directory/mod.rs new file mode 100644 index 000000000..373efd559 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/mod.rs @@ -0,0 +1,967 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Range discovery and conditional topology metadata. Neither a route nor a +//! directory publication grants Raft, transfer, birth or serving authority. + +use super::{CatalogError, NamespaceCatalog}; +use crate::application::consensus::cell_routing_key; +use crate::raft::{catalog_wire as record, coordination_index::Index}; +use cellar_logic::{ + cell_group::*, + consensus::{GroupId, IncarnationId, NamespaceId, ProposalId}, +}; +use object_store::{path::Path, PutMode, PutOptions, UpdateVersion}; +use std::{ + collections::BTreeSet, + sync::{Arc, Mutex}, + time::Duration, +}; + +mod archive; +pub use archive::{CellGroupArchiveBindingError, PreparedCellGroupArchiveBinding}; +mod birth; +mod bootstrap; +mod import; +pub use import::CellGroupBirthDisposition; +mod inventory; +pub use inventory::{CellGroupDirectoryCursor, CellGroupDirectoryPage}; +mod recovery; +pub use recovery::{ + CellGroupDisasterFenceRequest, CellGroupRecoveryActivationAck, + CellGroupRecoveryActivationPermit, CellGroupRecoveryError, CellGroupRecoveryRequest, + FencedCellGroupRecovery, ManagedRecoverySeedQuorum, ManagedRecoveryVoterSeed, +}; +pub(crate) use recovery::{ManagedRecoveryPublicationInput, RecoverySeedPermit}; +mod route_cache; +mod routing; +pub use route_cache::CellRouteCacheUsage; +pub(crate) use route_cache::RouteCache; +mod store; +mod transition; +pub(crate) use transition::ManagedTransitionRecoverySelection; +mod transition_acquires; +mod transition_activation; +pub use transition_activation::{ + CellGroupTransferActivationAck, CellGroupTransferActivationPermit, +}; +mod transition_archive; +pub use transition_archive::CellGroupTransferArchive; +mod transition_child_archive; +mod transition_child_quorum; +pub use transition::{CellGroupSealPermit, TransitionJournal, TransitionPhase}; +mod transition_source_slots; +mod transition_sources; +pub use transition_sources::{CellGroupTransitionSource, VerifiedCellGroupTransitionSources}; +mod transition_install; +pub(crate) use transition_install::TransferSeedPermit; +pub use transition_install::{InstalledTransferVoter, TransferInstallQuorum}; +mod transition_preparation; +mod transition_retirement; +pub use transition_preparation::{ + CellGroupTransitionPreparationInputs, PreparedManagedCellGroupTransition, +}; +pub use transition_retirement::{CellGroupSourceRetirementAck, CellGroupSourceRetirementPermit}; +mod tree; +mod wire; + +pub use birth::{CellGroupBirth, CellGroupBirthRequest, CellGroupBirthSpec}; +pub use bootstrap::CellGroupBootstrapPermit; +pub use routing::{ + CellRouteAttempt, CellRouteOperation, CellRouteRequest, CellRoutingError, CellRoutingFailure, + CellRoutingLimits, CellRoutingSession, ResolvedCellRoute, +}; + +#[derive(Clone, Copy, Debug, serde::Serialize, serde::Deserialize)] +#[serde(deny_unknown_fields)] +pub struct DirectoryLimits { + pub max_groups: u32, + pub max_mutations: u64, + pub max_cached_pages: usize, + pub max_cache_bytes: usize, + pub operation_timeout_ms: u64, +} + +impl Default for DirectoryLimits { + fn default() -> Self { + Self { + max_groups: 1_000_000, + max_mutations: 1_000_000, + max_cached_pages: 64, + max_cache_bytes: 4 * 1024 * 1024, + operation_timeout_ms: 5000, + } + } +} + +#[derive(Clone)] +pub struct RangeDirectory { + namespace: NamespaceCatalog, + limits: DirectoryLimits, + cache: Arc>, +} + +impl RangeDirectory { + /// Namespace identity for the owning fleet's rooted declarations. This + /// exposes metadata inside the host, never an application-cell birth API. + pub(crate) fn namespace(&self) -> &NamespaceCatalog { + &self.namespace + } +} + +/// A pinned catalog observation, not authority or an automatic freshness claim. +#[derive(Clone)] +pub struct DirectoryView { + head: wire::Head, + digest: [u8; 32], + owner: Arc<()>, +} + +impl DirectoryView { + pub fn revision(&self) -> u64 { + self.head.revision + } + + pub fn groups(&self) -> u32 { + self.head.tree.as_ref().map_or(0, |tree| tree.groups) + } + + pub fn digest(&self) -> [u8; 32] { + self.digest + } +} + +/// Persist these canonical bytes before issuing a topology mutation. Resuming +/// decodes the same expected root and operation, rather than rebasing a retry. +pub struct DirectoryMutation { + request: wire::Request, + bytes: Vec, + digest: [u8; 32], + operation: ProposalId, +} + +impl DirectoryMutation { + pub fn encode(&self) -> &[u8] { + &self.bytes + } + + pub fn digest(&self) -> [u8; 32] { + self.digest + } + + pub(crate) fn expected_directory_digest(&self) -> Option<[u8; 32]> { + self.request.expected + } + + pub fn operation(&self) -> ProposalId { + self.operation + } +} + +/// Exact historical catalog decision. A later directory head may supersede its +/// ranges; destination admission still needs the replicated transition protocol. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct DirectoryPublication { + operation: ProposalId, + revision: u64, + tree: Option<[u8; 32]>, + journal: [u8; 32], + transfer_archive: Option<[u8; 32]>, +} + +enum PublicationAuthority<'a> { + Ordinary, + Recovery(&'a recovery::RecoveryPublication), + ManagedRecovery(&'a recovery::ManagedRecoveryPublication<'a>), + Transfer([u8; 32]), +} + +impl DirectoryPublication { + pub fn operation(&self) -> ProposalId { + self.operation + } + + pub fn revision(&self) -> u64 { + self.revision + } + + /// None confirms a standby genesis with no selected ranges. It grants no + /// cell route, birth, membership, capacity or serving authority. + pub fn tree_digest(&self) -> Option<[u8; 32]> { + self.tree + } + + pub fn journal_digest(&self) -> [u8; 32] { + self.journal + } + + pub fn transfer_archive(&self) -> Option<[u8; 32]> { + self.transfer_archive + } +} + +impl RangeDirectory { + pub(crate) fn cache_capacity_bytes(&self) -> usize { + self.limits.max_cache_bytes + } + + /// Immutable namespace properties for binding schema and image resources. + /// This does not refresh routing metadata or grant member admission. + pub fn namespace_config(&self) -> &crate::raft::catalog::NamespaceConfig { + self.namespace.config() + } + + pub fn new(namespace: NamespaceCatalog, limits: DirectoryLimits) -> Result { + namespace.require_mode(cellar_logic::NamespaceMode::Raft)?; + if !(1..=1_000_000).contains(&limits.max_groups) + || !(1..=1_000_000).contains(&limits.max_mutations) + || limits.max_cached_pages > 1024 + || limits.max_cache_bytes > 64 * 1024 * 1024 + || (limits.max_cached_pages == 0) != (limits.max_cache_bytes == 0) + || !(1..=60_000).contains(&limits.operation_timeout_ms) + { + return Err(CatalogError::Limit); + } + Ok(Self { + namespace, + limits, + cache: Arc::new(Mutex::new(store::Cache::default())), + }) + } + + fn check_view(&self, view: &DirectoryView) -> Result<(), CatalogError> { + if !Arc::ptr_eq(&view.owner, &self.namespace.inner.owner) + || view.head.namespace != self.namespace.root_digest + { + return Err(CatalogError::Conflict("foreign directory view")); + } + Ok(()) + } + + fn request( + &self, + operation: ProposalId, + expected: Option<[u8; 32]>, + parents: &[GroupRoute], + children: &[GroupRoute], + ) -> Result { + if parents.len() > 64 || children.len() > if expected.is_none() { 128 } else { 64 } { + return Err(CatalogError::Limit); + } + let request = wire::Request { + namespace: self.namespace.root_digest, + operation: operation.bytes(), + expected, + parents: parents + .iter() + .map(|r| wire::Route::encode(r, self)) + .collect::>()?, + children: children + .iter() + .map(|r| wire::Route::encode(r, self)) + .collect::>()?, + }; + self.decode_mutation(&record::encode(wire::REQUEST, &request, wire::LIMIT)?) + } + + pub fn prepare_initialization( + &self, + operation: ProposalId, + ranges: &[GroupRoute], + ) -> Result { + if ranges.is_empty() { + return Err(CatalogError::Invalid("empty initial range set")); + } + self.request(operation, None, &[], ranges) + } + + /// Explicitly initialize a new namespace before any range is selected. + /// Publish and retain this exact intent like any other topology mutation. + /// Inventory can then prove zero groups without treating absence as empty. + pub fn prepare_standby_initialization( + &self, + operation: ProposalId, + ) -> Result { + self.request(operation, None, &[], &[]) + } + + /// Select the first complete range map from an exact standby observation. + /// Every child still needs its permanent birth reservation before publication. + /// A concurrent initializer loses this CAS; retry the original intent after + /// an uncertain response rather than rebasing onto a later observation. + pub fn prepare_initialization_at( + &self, + standby: &DirectoryView, + operation: ProposalId, + ranges: &[GroupRoute], + ) -> Result { + self.check_view(standby)?; + if standby.head.tree.is_some() || ranges.is_empty() { + return Err(CatalogError::Conflict("not a standby directory")); + } + self.request(operation, Some(standby.digest), &[], ranges) + } + + pub(crate) fn prepare_managed_first_group( + &self, + standby: [u8; 32], + operation: ProposalId, + route: &GroupRoute, + ) -> Result { + if self.namespace.config().managed_fleet.is_none() || standby == [0; 32] { + return Err(CatalogError::Invalid("managed standby identity")); + } + self.request(operation, Some(standby), &[], std::slice::from_ref(route)) + } + + /// Accept the original empty root or the exact operation's durable receipt. + pub(crate) async fn check_managed_first_group( + &self, + mutation: &DirectoryMutation, + ) -> Result<(), CatalogError> { + self.check_request(&mutation.request)?; + self.bounded(self.first_group_publication(mutation)).await?; + Ok(()) + } + + async fn first_group_publication( + &self, + mutation: &DirectoryMutation, + ) -> Result, CatalogError> { + let (view, _) = self.current().await?; + if view.head.revision == 1 + && view.head.tree.is_none() + && mutation.request.expected == Some(view.digest) + { + return Ok(None); + } + self.confirmed(&view.head, mutation) + .await? + .map(Some) + .ok_or(CatalogError::Conflict("managed first group standby")) + } + + /// Resolve cancellation at the immutable birth decision before selecting the + /// range. A Live decision alone cannot admit a member. A losing range CAS + /// leaves the exact birth inventoried, without granting serving authority. + pub(crate) async fn publish_managed_first_group( + &self, + mutation: &DirectoryMutation, + authorization: &crate::raft::fleet_catalog::FleetBirthAuthorization, + ) -> Result<(), CatalogError> { + self.check_request(&mutation.request)?; + authorization.check_namespace(&self.namespace)?; + self.bounded(async { + let request = &mutation.request; + let [child] = request.children.as_slice() else { + return Err(CatalogError::Invalid("managed first group range")); + }; + if self.namespace.cell_group_owner().await?.as_ref() != Some(authorization.owner()) + || request.expected.is_none() + || !request.parents.is_empty() + || child.decode(self)? != *authorization.route() + { + return Err(CatalogError::ManagedAdmissionRequired); + } + let proposal = authorization.proposal(); + let birth = if self.first_group_publication(mutation).await?.is_some() { + match self.group_birth_disposition(&proposal).await? { + Some(CellGroupBirthDisposition::Live(birth)) => birth, + _ => return Err(CatalogError::Corrupt("missing published first group birth")), + } + } else { + let birth = self.decide_live_birth(&proposal).await?; + self.publish_inner(mutation, PublicationAuthority::Ordinary) + .await?; + birth + }; + self.require_current_group_birth(&birth).await + }) + .await + } + + /// Prepare metadata for one exact contiguous replacement. The topology + /// owner must separately obtain seal/install/archive/activation proofs before + /// using this publication; this staging API does not perform a transfer. + pub fn prepare_replacement( + &self, + view: &DirectoryView, + operation: ProposalId, + parents: &[GroupRoute], + children: &[GroupRoute], + ) -> Result { + self.check_view(view)?; + self.request(operation, Some(view.digest), parents, children) + } + + /// Prepare a split of one selected range into adjacent fresh child routes. + /// Bounds are whole cell keys. The ordered children must cover the exact + /// parent range and share its successor epoch. + pub fn prepare_split( + &self, + view: &DirectoryView, + operation: ProposalId, + parent: &GroupRoute, + children: &[GroupRoute], + ) -> Result { + let plan = + self.prepare_replacement(view, operation, std::slice::from_ref(parent), children)?; + self.check_transition_plan(&plan)?; + Ok(plan) + } + + /// Prepare a merge of adjacent selected ranges into one fresh child route. + /// The child covers their exact union at the highest parent epoch plus one. + pub fn prepare_merge( + &self, + view: &DirectoryView, + operation: ProposalId, + parents: &[GroupRoute], + child: &GroupRoute, + ) -> Result { + let plan = + self.prepare_replacement(view, operation, parents, std::slice::from_ref(child))?; + self.check_transition_plan(&plan)?; + Ok(plan) + } + + pub fn decode_mutation(&self, bytes: &[u8]) -> Result { + let request: wire::Request = record::decode(wire::REQUEST, bytes, wire::LIMIT)?; + self.check_request(&request)?; + let operation = ProposalId::new(request.operation) + .map_err(|_| CatalogError::Invalid("directory operation"))?; + Ok(DirectoryMutation { + request, + operation, + digest: record::digest(bytes), + bytes: bytes.to_vec(), + }) + } + + fn union(&self, routes: &[wire::Route]) -> Result { + let first = routes + .first() + .ok_or(CatalogError::Invalid("empty range set"))? + .decode(self)?; + let mut range = first.range; + let mut ids = BTreeSet::from([first.history.group]); + for route in &routes[1..] { + let route = route.decode(self)?; + if !ids.insert(route.history.group) { + return Err(CatalogError::Conflict("duplicate range group")); + } + range = range + .merge(&route.range) + .map_err(|_| CatalogError::Conflict("overlap or gap in range set"))?; + } + Ok(range) + } + + fn check_request(&self, request: &wire::Request) -> Result<(), CatalogError> { + if request.namespace != self.namespace.root_digest { + return Err(CatalogError::Conflict("directory namespace")); + } + if request.children.is_empty() { + return if request.expected.is_none() && request.parents.is_empty() { + Ok(()) + } else { + Err(CatalogError::Invalid("empty replacement range set")) + }; + } + if request.children.len() > if request.expected.is_none() { 128 } else { 64 } + || request.parents.len() > 64 + { + return Err(CatalogError::Limit); + } + let children = self.union(&request.children)?; + let epoch = if request.parents.is_empty() { + if children != CellRange::all() { + return Err(CatalogError::Invalid("initial range coverage")); + } + 1 + } else { + if request.expected.is_none() { + return Err(CatalogError::Invalid("initial parent ranges")); + } + if self.union(&request.parents)? != children { + return Err(CatalogError::Conflict("changed range union")); + } + request + .parents + .iter() + .map(|r| r.epoch) + .max() + .ok_or(CatalogError::Invalid("parent ranges"))? + .checked_add(1) + .ok_or(CatalogError::Limit)? + }; + if request.children.iter().any(|r| r.epoch != epoch) { + return Err(CatalogError::Conflict("range epoch successor")); + } + Ok(()) + } + + fn check_head(&self, head: &wire::Head) -> Result<(), CatalogError> { + if head.namespace != self.namespace.root_digest + || head.revision == 0 + || head.revision > self.limits.max_mutations + || head.genesis == [0; 32] + || head.journal == [0; 32] + || head.operations.is_none() + { + return Err(CatalogError::Corrupt("directory head")); + } + if head.import.is_some_and(|id| ProposalId::new(id).is_err()) + || (head.import.is_some() && head.recovery.is_some()) + { + return Err(CatalogError::Corrupt("directory import fence")); + } + if let Some(recovery) = &head.recovery { + recovery.check(self)?; + } + match &head.tree { + Some(tree) if head.groups.is_some() => tree.require_complete(self), + None if head.groups.is_none() + && ((head.revision == 1 && head.import.is_none()) + || (head.revision > 1 && head.import.is_some())) + && head.recovery.is_none() => + { + Ok(()) + } + _ => Err(CatalogError::Corrupt("directory range state")), + } + } + + async fn head(&self) -> Result, CatalogError> { + self.namespace + .inner + .read_versioned_bounded(&self.key("head"), wire::LIMIT) + .await? + .map(|(bytes, version)| { + let head: wire::Head = record::decode(wire::HEAD, &bytes, wire::LIMIT)?; + self.check_head(&head)?; + if !version.e_tag.as_ref().is_some_and(|s| !s.is_empty()) + && !version.version.as_ref().is_some_and(|s| !s.is_empty()) + { + return Err(CatalogError::Corrupt("directory CAS token")); + } + Ok(( + DirectoryView { + head, + digest: record::digest(&bytes), + owner: self.namespace.inner.owner.clone(), + }, + version, + )) + }) + .transpose() + } + + async fn current(&self) -> Result<(DirectoryView, UpdateVersion), CatalogError> { + let pin: wire::Pin = record::decode( + wire::PIN, + &self + .read("pin") + .await? + .ok_or(CatalogError::Corrupt("missing directory pin"))?, + wire::LIMIT, + )?; + let (view, version) = self + .head() + .await? + .ok_or(CatalogError::Corrupt("missing directory head"))?; + if pin.namespace != self.namespace.root_digest || pin.genesis != view.head.genesis { + return Err(CatalogError::Corrupt("directory genesis pin")); + } + let journal = self.journal(view.head.journal).await?; + if journal.revision != view.head.revision + || journal.tree != view.head.tree + || (journal.revision == 1 && view.head.genesis != view.head.journal) + || Index::new(self) + .lookup(view.head.operations, &hex::encode(journal.operation)) + .await? + != Some(view.head.journal.to_vec()) + { + return Err(CatalogError::Corrupt("directory head/journal disagreement")); + } + self.check_confirmed_transfer(&view.head, &journal, view.head.journal) + .await?; + Ok((view, version)) + } + + fn bounded( + &self, + work: impl std::future::Future>, + ) -> impl std::future::Future> { + let duration = Duration::from_millis(self.limits.operation_timeout_ms); + // Bootstrap revalidation nests catalog operations during attachment. + // Box before constructing the timeout future so each wrapper retains + // only a pointer instead of copying the complete nested operation. + let work = Box::pin(work); + async move { + crate::asyncrt::timeout(duration, work) + .await + .map_err(|_| CatalogError::Unconfirmed)? + } + } + + pub async fn observe(&self) -> Result { + self.bounded(async { Ok(self.current().await?.0) }).await + } + + /// Looks up only an immutable tree path. No head refresh, namespace listing + /// or object-store call is required when that path is already cached. + pub async fn lookup( + &self, + view: &DirectoryView, + cell: &CellId, + ) -> Result { + self.check_view(view)?; + let tree = view + .head + .tree + .as_ref() + .ok_or(CatalogError::Conflict("directory has no selected ranges"))?; + self.bounded(self.lookup_tree(tree, cell)).await + } + + async fn confirmed( + &self, + head: &wire::Head, + mutation: &DirectoryMutation, + ) -> Result, CatalogError> { + let key = hex::encode(mutation.request.operation); + let Some(value) = Index::new(self).lookup(head.operations, &key).await? else { + return Ok(None); + }; + let id: [u8; 32] = value + .try_into() + .map_err(|_| CatalogError::Corrupt("directory journal pointer"))?; + let journal = self.journal(id).await?; + if journal.operation != mutation.request.operation || journal.revision > head.revision { + return Err(CatalogError::Corrupt("directory journal identity")); + } + if journal.request != mutation.digest { + return Err(CatalogError::Conflict("directory operation reused")); + } + self.check_confirmed_transfer(head, &journal, id).await?; + Ok(Some(DirectoryPublication { + operation: mutation.operation, + revision: journal.revision, + tree: journal.tree.as_ref().map(|tree| tree.digest), + journal: id, + transfer_archive: journal.transfer_archive, + })) + } + + async fn journal(&self, id: [u8; 32]) -> Result { + let bytes = self + .read(&format!("journal/{}", hex::encode(id))) + .await? + .ok_or(CatalogError::Corrupt("missing directory journal"))?; + if record::digest(&bytes) != id { + return Err(CatalogError::Corrupt("directory journal hash")); + } + let journal: wire::Journal = record::decode(wire::JOURNAL, &bytes, wire::LIMIT)?; + if let Some(tree) = &journal.tree { + tree.require_complete(self)?; + } + if journal.namespace != self.namespace.root_digest + || journal.revision == 0 + || journal.revision > self.limits.max_mutations + || ProposalId::new(journal.operation).is_err() + || journal.previous.is_none() != (journal.revision == 1) + || journal.transfer_archive == Some([0; 32]) + || (journal.revision == 1 && journal.transfer_archive.is_some()) + { + return Err(CatalogError::Corrupt("directory journal identity")); + } + Ok(journal) + } + + /// `None` is an observation, never permission to replace the operation ID + /// of a potentially submitted mutation. Re-submit the identical bytes. + pub async fn publication( + &self, + mutation: &DirectoryMutation, + ) -> Result, CatalogError> { + self.check_request(&mutation.request)?; + self.bounded(async { + self.confirmed(&self.current().await?.0.head, mutation) + .await + }) + .await + } + + pub async fn publish( + &self, + mutation: &DirectoryMutation, + ) -> Result { + self.check_request(&mutation.request)?; + self.bounded(self.publish_inner(mutation, PublicationAuthority::Ordinary)) + .await + } + + async fn publish_inner( + &self, + mutation: &DirectoryMutation, + authority: PublicationAuthority<'_>, + ) -> Result { + let request = &mutation.request; + let (recovery, transfer_archive, managed_recovery) = match authority { + PublicationAuthority::Ordinary => (None, None, None), + PublicationAuthority::Recovery(recovery) => (Some(recovery), None, None), + PublicationAuthority::Transfer(archive) => (None, Some(archive), None), + PublicationAuthority::ManagedRecovery(recovery) => (None, None, Some(recovery)), + }; + let mut transition = None; + let current = if request.expected.is_some() { + Some(self.current().await?) + } else { + let pin = self.read("pin").await?; + let head = self.head().await?; + if pin.is_some() { + let (view, _) = self.current().await?; + return self + .confirmed(&view.head, mutation) + .await? + .ok_or(CatalogError::Conflict("directory already initialized")); + } + if head.as_ref().is_some_and(|(v, _)| v.head.revision != 1) { + return Err(CatalogError::Corrupt("missing active directory pin")); + } + head + }; + if let Some((view, _)) = ¤t { + if view.head.import.is_some() { + return Err(CatalogError::BirthsFrozen); + } + if let Some(publication) = self.confirmed(&view.head, mutation).await? { + if request.expected.is_some() { + if transfer_archive.is_some() + && publication.transfer_archive() != transfer_archive + { + return Err(CatalogError::Fenced); + } + if let Some(recovery) = managed_recovery { + recovery.check_selected(self, view, &publication).await?; + } + return Ok(publication); + } + // An interrupted genesis resumes only its identical first head. + } + if let Some(archive) = transfer_archive { + transition = Some( + self.check_transfer_publication(view, mutation, archive) + .await?, + ); + } else if let Some(recovery) = managed_recovery { + recovery.check(view, mutation)?; + if self + .transition_record(&view.head, mutation.operation) + .await? + .is_some() + { + return Err(CatalogError::Conflict("transition operation reused")); + } + } else { + if self.transition_active(&view.head).await?.is_some() { + return Err(CatalogError::Fenced); + } + if self + .transition_record(&view.head, mutation.operation) + .await? + .is_some() + { + return Err(CatalogError::Conflict("transition operation reused")); + } + if let Some(recovery) = recovery { + recovery.check(self, view, mutation)?; + } else { + if view.head.recovery.is_some() { + return Err(CatalogError::Fenced); + } + if request + .expected + .is_some_and(|expected| expected != view.digest) + { + return Err(CatalogError::Conflict("stale directory root")); + } + } + } + } + let base = if request.expected.is_some() { + current.as_ref().map(|(v, _)| &v.head) + } else { + None + }; + if request.expected.is_some() + && request.parents.is_empty() + && base.is_none_or(|head| head.tree.is_some()) + { + return Err(CatalogError::Conflict("directory already has ranges")); + } + let revision = base + .map_or(Some(1), |h| h.revision.checked_add(1)) + .ok_or(CatalogError::Limit)?; + if revision > self.limits.max_mutations { + return Err(CatalogError::Limit); + } + let mut tree = base.and_then(|h| h.tree.clone()); + // A route can be lazy, but its exact cell-independent birth and global + // group identity must be reserved before it can enter the directory. + for child in &request.children { + let route = child.decode(self)?; + if let Some(recovery) = recovery { + recovery.check_child(&route)?; + } else if let Some(recovery) = managed_recovery { + recovery.check_child(self, &route).await?; + } else { + let birth = self.require_ordinary_birth_reservation(&route).await?; + if transition.is_some() { + self.decide_live_birth(&birth).await?; + } + } + } + for parent in &request.parents { + tree = self.remove(tree, parent.clone(), 0).await?; + } + let mut groups = base.and_then(|h| h.groups); + for child in &request.children { + let key = hex::encode(child.group); + if Index::new(self).lookup(groups, &key).await?.is_some() { + return Err(CatalogError::Conflict("group identity already assigned")); + } + groups = Index::new(self) + .update(groups, &key, Some(child.incarnation.to_vec()), 0) + .await?; + tree = Some(self.insert(tree, child.clone(), 0).await?); + } + if let Some(tree) = &tree { + tree.require_complete(self)?; + } else if revision != 1 || request.expected.is_some() { + return Err(CatalogError::Invalid("empty directory replacement")); + } + let journal = wire::Journal { + namespace: self.namespace.root_digest, + operation: request.operation, + request: mutation.digest, + revision, + previous: base.map(|h| h.journal), + transfer_archive, + tree: tree.clone(), + }; + let bytes = record::encode(wire::JOURNAL, &journal, wire::LIMIT)?; + let id = record::digest(&bytes); + self.immutable("journal", id, &bytes).await?; + let mut operations = Index::new(self) + .update( + base.and_then(|h| h.operations), + &hex::encode(request.operation), + Some(id.to_vec()), + 0, + ) + .await?; + if let Some(transition) = transition { + operations = self + .root_transfer_publication(operations, mutation, transition, id) + .await?; + } + let head = wire::Head { + namespace: self.namespace.root_digest, + genesis: base.map_or(id, |h| h.genesis), + revision, + tree, + operations, + groups, + journal: id, + recovery: match managed_recovery { + Some(grant) => Some( + grant.published_lock(¤t.as_ref().ok_or(CatalogError::Fenced)?.0, id)?, + ), + None => recovery.map(|grant| grant.published_lock(id)).transpose()?, + }, + import: None, + }; + self.check_head(&head)?; + let bytes = record::encode(wire::HEAD, &head, wire::LIMIT)?; + if request.expected.is_none() { + // Unlike an immutable page, this object may advance after Create + // succeeds and before its read-back. Confirm via its pinned journal. + let _ = self + .namespace + .inner + .store + .put_opts( + &self.key("head"), + bytes.into(), + PutOptions::from(PutMode::Create), + ) + .await; + let (observed, _) = self.head().await?.ok_or(CatalogError::Unconfirmed)?; + if observed.head != head { + if observed.head.genesis != head.genesis { + return Err(CatalogError::Conflict("directory genesis winner")); + } + let (selected, _) = self.current().await?; + return self + .confirmed(&selected.head, mutation) + .await? + .ok_or(CatalogError::Corrupt("missing genesis operation")); + } + let pin = record::encode( + wire::PIN, + &wire::Pin { + namespace: head.namespace, + genesis: head.genesis, + }, + wire::LIMIT, + )?; + if self + .namespace + .inner + .create_and_observe_bounded(&self.key("pin"), &pin, wire::LIMIT) + .await? + != pin + { + return Err(CatalogError::Corrupt("directory pin collision")); + } + } else { + let (_, version) = current.ok_or(CatalogError::Conflict("missing directory base"))?; + // A transport error can follow a committed CAS. Reconcile by the + // root-selected operation index, including after subsequent writes. + let _ = self + .namespace + .inner + .store + .put_opts( + &self.key("head"), + bytes.into(), + PutOptions::from(PutMode::Update(version)), + ) + .await; + } + let (after, _) = self + .current() + .await + .map_err(|_| CatalogError::Unconfirmed)?; + match self.confirmed(&after.head, mutation).await { + Ok(Some(publication)) + if transfer_archive.is_none() + || publication.transfer_archive() == transfer_archive => + { + if let Some(recovery) = managed_recovery { + recovery.check_published(self, &after, &publication).await?; + } + Ok(publication) + } + Ok(Some(_)) => Err(CatalogError::Fenced), + Ok(None) + if transfer_archive.is_none() + && request.expected.is_some_and(|base| base != after.digest) => + { + Err(CatalogError::Conflict("directory CAS lost")) + } + _ => Err(CatalogError::Unconfirmed), + } + } +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/recovery/activation.rs b/crates/cellar/durability/raft/cell_groups/directory/recovery/activation.rs new file mode 100644 index 000000000..655f6f89b --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/recovery/activation.rs @@ -0,0 +1,478 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! A catalog grant admits one activation proposal. Only the real Raft owner can +//! produce an applied acknowledgement and release the topology recovery lock. +use super::*; +use crate::raft::{ + cache::GroupLog, + cell_groups::GroupManifest, + completion::RuntimeError, + fleet_catalog::{FleetHostAuthorization, FleetRecoveryAttachRequest}, + fleet_stop::BootBinding, + storage::ReplicaLog, +}; +use cellar_logic::consensus::{EngineInstanceId, LogIndex, LogPosition, Term}; +use std::sync::atomic::Ordering; + +pub struct CellGroupRecoveryActivationPermit { + route: GroupRoute, + request: [u8; 32], + operation: ProposalId, + group: GroupLog, + engine: EngineInstanceId, + managed_boot: Option, + publication: [u8; 32], + manifest: [u8; 32], + deadline: u64, +} + +impl CellGroupRecoveryActivationPermit { + pub(crate) fn deadline(&self) -> u64 { + self.deadline + } + + pub fn history(&self) -> GroupHistory { + self.group.identity().scope + } + + pub(crate) fn check( + &self, + log: &ReplicaLog, + group: &GroupLog, + ) -> Result<(), RuntimeError> { + if !Arc::ptr_eq(&group.owner, &self.group.owner) + || group.identity() != self.group.identity() + || log.storage_instance()? != self.engine + || log.domain().mono_ms().map_err(|_| RuntimeError::Closed)? >= self.deadline + { + return Err(RuntimeError::StaleCompletion); + } + if let Some(boot) = &self.managed_boot { + if log + .lifecycle_guard()? + .is_none_or(|guard| !guard.admits_group_start(boot)) + { + return Err(RuntimeError::StaleCompletion); + } + } + drop(group.read()?); + Ok(()) + } + + pub(crate) fn matches(&self, marker: &GroupManifest) -> Result { + if marker.route() != &self.route + || marker.recovery_request() != Some(self.request) + || marker + .recovery_commitment() + .map_err(|e| RuntimeError::Invalid(e.to_string()))? + != Some(self.manifest) + { + return Err(RuntimeError::StaleCompletion); + } + match marker.recovery_activation() { + Some((publication, manifest, _)) + if publication == self.publication && manifest == self.manifest => + { + Ok(true) + } + Some(_) => Err(RuntimeError::StaleCompletion), + None => Ok(false), + } + } + + pub(crate) fn entry(&self) -> Result, RuntimeError> { + use crate::application::consensus::group_protocol::{ + CellGroupEntry, GroupCommand, GroupEntryBinding, GroupEntryLimits, + }; + let route = &self.route; + CellGroupEntry { + binding: GroupEntryBinding { + group: route.history, + route_epoch: route.epoch, + birth_digest: route.birth_digest, + schema_digest: route.schema_digest, + proposal: self.operation, + }, + command: GroupCommand::Activate { + transition: TransitionId::new(self.operation.bytes()) + .map_err(|e| RuntimeError::Invalid(e.to_string()))?, + publication: self.publication, + manifest: self.manifest, + }, + } + .encode(route, GroupEntryLimits::default()) + .map_err(|e| RuntimeError::Invalid(e.to_string())) + } + + pub(crate) fn acknowledgement( + &self, + group: &GroupLog, + selected: &GroupManifest, + ) -> Result { + let cache = group.read()?; + let marker = cache + .group_manifest + .as_ref() + .ok_or(RuntimeError::NotReady)?; + if marker != selected || !self.matches(marker)? { + return Err(RuntimeError::NotReady); + } + let (_, _, position) = marker.recovery_activation().ok_or(RuntimeError::NotReady)?; + if cache.hard_state.commit < position.index.get() { + return Err(RuntimeError::NotReady); + } + Ok(CellGroupRecoveryActivationAck { + group: group.clone(), + request: self.request, + publication: self.publication, + manifest: self.manifest, + position, + }) + } +} + +/// An observation of activation committed and selected by an actual local Raft +/// owner. This is neither a transferable serving grant nor an archive witness. +pub struct CellGroupRecoveryActivationAck { + group: GroupLog, + request: [u8; 32], + publication: [u8; 32], + manifest: [u8; 32], + position: LogPosition, +} + +impl CellGroupRecoveryActivationAck { + pub fn position(&self) -> LogPosition { + self.position + } + + fn check( + &self, + route: &GroupRoute, + request: [u8; 32], + publication: [u8; 32], + manifest: [u8; 32], + ) -> Result<(), CatalogError> { + if self.request != request + || self.publication != publication + || self.manifest != manifest + || self.group.closed.load(Ordering::Acquire) + || self.group.identity().scope != route.history + { + return Err(CatalogError::Fenced); + } + let cache = self.group.read().map_err(CatalogError::Log)?; + let root = cache + .group_manifest + .as_ref() + .ok_or(CatalogError::LostStableState)?; + if root.route() != route + || root.recovery_request() != Some(self.request) + || root.recovery_activation() != Some((self.publication, self.manifest, self.position)) + || cache.hard_state.commit < self.position.index.get() + { + return Err(CatalogError::Fenced); + } + Ok(()) + } +} + +#[derive(Serialize, Deserialize, PartialEq, Eq)] +#[serde(deny_unknown_fields)] +struct Activated { + request: [u8; 32], + publication: [u8; 32], + manifest: [u8; 32], + index: u64, + term: u64, +} + +impl Activated { + fn position_for( + &self, + request: [u8; 32], + publication: [u8; 32], + manifest: [u8; 32], + ) -> Result { + let position = LogPosition { + index: LogIndex::new(self.index), + term: Term::new(self.term), + }; + if self.request != request + || self.publication != publication + || self.manifest != manifest + || self.index <= 1 + || self.term <= 1 + || position.validate().is_err() + { + return Err(CatalogError::Corrupt("recovery activation binding")); + } + Ok(position) + } +} + +impl RangeDirectory { + pub(super) async fn require_recovery_activation( + &self, + request: &CellGroupRecoveryRequest, + publication: &DirectoryPublication, + manifest: [u8; 32], + ) -> Result<(), CatalogError> { + let bytes = self + .namespace + .inner + .read_versioned_bounded(&self.recovery_key(request, "activation"), wire::LIMIT) + .await? + .ok_or(CatalogError::Corrupt("missing recovery activation"))? + .0; + let activated: Activated = + record::decode(wire::GROUP_RECOVERY_ACTIVATED, &bytes, wire::LIMIT)?; + activated.position_for(request.digest(), publication.journal_digest(), manifest)?; + Ok(()) + } + + /// Recheck publication, physical member identity and the permanent external + /// fence before admitting the exact activation to the actual Raft owner. + pub fn prepare_group_recovery_activation<'a>( + &'a self, + request: &'a CellGroupRecoveryRequest, + log: &'a ReplicaLog, + authority: &'a dyn DisasterFenceAuthority, + ) -> futures_util::future::BoxFuture<'a, Result> + { + Box::pin(async move { + let request = self.decode_group_recovery(request.encode())?; + self.bounded(async { + self.verify_recovery_fence(&request, authority).await?; + let group = self.recovered_member(&request, log).await?; + let (publication, binding, _) = self.published_recovery(&request).await?; + let deadline = log + .domain() + .mono_ms() + .map_err(|_| CatalogError::Unconfirmed)? + .checked_add(self.limits.operation_timeout_ms) + .ok_or(CatalogError::Limit)?; + Ok(CellGroupRecoveryActivationPermit { + route: request.replacement().route().clone(), + request: request.digest(), + operation: request.operation(), + group, + managed_boot: None, + engine: log.storage_instance().map_err(CatalogError::Log)?, + publication: publication.journal_digest(), + manifest: binding.commitment, + deadline, + }) + }) + .await + }) + } + + pub(crate) fn prepare_managed_recovery_activation<'a>( + &'a self, + request: &'a FleetRecoveryAttachRequest, + history: GroupHistory, + log: &'a ReplicaLog, + host: &'a FleetHostAuthorization, + deadline: u64, + ) -> futures_util::future::BoxFuture<'a, Result> + { + Box::pin(self.bounded(async move { + let (group, authority) = self + .verify_published_managed_recovery_member(request, history, log, host, deadline) + .await?; + Ok(CellGroupRecoveryActivationPermit { + route: authority.birth.route().clone(), + request: request.plan_digest, + operation: request.operation, + group, + engine: log.storage_instance().map_err(CatalogError::Log)?, + managed_boot: Some(authority.current_boot), + publication: authority.publication.journal_digest(), + manifest: authority.seeds.genesis_commitment(), + deadline, + }) + })) + } + + /// Record an applied activation under the exact selected range. The + /// recovery lock stays selected until the complete Fleet settlement. + pub(crate) async fn record_managed_recovery_activation( + &self, + permit: &CellGroupRecoveryActivationPermit, + acknowledgement: &CellGroupRecoveryActivationAck, + log: &ReplicaLog, + ) -> Result { + if permit.managed_boot.is_none() { + return Err(CatalogError::Fenced); + } + self.bounded(async { + permit + .check(log, &acknowledgement.group) + .map_err(CatalogError::Runtime)?; + acknowledgement.check( + &permit.route, + permit.request, + permit.publication, + permit.manifest, + )?; + let position = acknowledgement.position(); + if position.index.get() <= 1 || position.term.get() <= 1 { + return Err(CatalogError::Fenced); + } + let bytes = record::encode( + wire::GROUP_RECOVERY_ACTIVATED, + &Activated { + request: permit.request, + publication: permit.publication, + manifest: permit.manifest, + index: position.index.get(), + term: position.term.get(), + }, + wire::LIMIT, + )?; + let key = self.managed_seed_key(&permit.route, "activation"); + if self + .namespace + .inner + .create_and_observe_bounded(&key, &bytes, wire::LIMIT) + .await? + != bytes + { + return Err(CatalogError::Conflict( + "managed recovery activation receipt", + )); + } + Ok(position) + }) + .await + } + + pub(crate) async fn observe_managed_recovery_activation( + &self, + route: &GroupRoute, + request: [u8; 32], + publication: [u8; 32], + manifest: [u8; 32], + ) -> Result, CatalogError> { + self.bounded(async { + let key = self.managed_seed_key(route, "activation"); + let Some((bytes, _)) = self + .namespace + .inner + .read_versioned_bounded(&key, wire::LIMIT) + .await? + else { + return Ok(None); + }; + let observed: Activated = + record::decode(wire::GROUP_RECOVERY_ACTIVATED, &bytes, wire::LIMIT)?; + observed + .position_for(request, publication, manifest) + .map(Some) + }) + .await + } + + /// Release topology only after committed/applied activation. Unknown S3 + /// responses retain the original request and are reconciled by read-back. + pub fn complete_group_recovery<'a>( + &'a self, + request: &'a CellGroupRecoveryRequest, + acknowledgement: &'a CellGroupRecoveryActivationAck, + authority: &'a dyn DisasterFenceAuthority, + ) -> futures_util::future::BoxFuture<'a, Result> { + Box::pin(async move { + let request = self.decode_group_recovery(request.encode())?; + self.bounded(async { + self.verify_recovery_fence(&request, authority).await?; + let (publication, binding, _) = self.published_recovery(&request).await?; + acknowledgement.check( + request.replacement().route(), + request.digest(), + publication.journal_digest(), + binding.commitment, + )?; + let position = acknowledgement.position(); + let complete = record::encode( + wire::GROUP_RECOVERY_ACTIVATED, + &Activated { + request: request.digest(), + publication: publication.journal_digest(), + manifest: binding.commitment, + index: position.index.get(), + term: position.term.get(), + }, + wire::LIMIT, + )?; + if self + .namespace + .inner + .create_and_observe_bounded( + &self.recovery_key(&request, "activation"), + &complete, + wire::LIMIT, + ) + .await? + != complete + { + return Err(CatalogError::Conflict("recovery activation receipt")); + } + let (mut view, version) = self.current().await?; + self.selected_route_in(&view, request.replacement().route()) + .await?; + if view + .head + .recovery + .as_deref() + .is_none_or(|lock| lock.request != request.digest()) + { + self.require_recovery_activation(&request, &publication, binding.commitment) + .await?; + return Ok(publication); + } + if view.head.recovery.as_deref().is_none_or(|lock| { + lock.request != request.digest() + || lock.phase + != (RecoveryPhase::Published { + publication: publication.journal_digest(), + manifest: binding.commitment, + }) + }) { + return Err(CatalogError::Fenced); + } + acknowledgement.check( + request.replacement().route(), + request.digest(), + publication.journal_digest(), + binding.commitment, + )?; + view.head.recovery = None; + let bytes = record::encode(wire::HEAD, &view.head, wire::LIMIT)?; + let _ = self + .namespace + .inner + .store + .put_opts( + &self.key("head"), + bytes.into(), + PutOptions::from(PutMode::Update(version)), + ) + .await; + let (after, _) = self + .current() + .await + .map_err(|_| CatalogError::Unconfirmed)?; + self.require_recovery_activation(&request, &publication, binding.commitment) + .await?; + if after.head.recovery.is_some() { + return Err(CatalogError::Unconfirmed); + } + self.confirmed(&after.head, request.mutation()) + .await? + .ok_or(CatalogError::Unconfirmed) + }) + .await + }) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/recovery/bootstrap.rs b/crates/cellar/durability/raft/cell_groups/directory/recovery/bootstrap.rs new file mode 100644 index 000000000..ccacccc0f --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/recovery/bootstrap.rs @@ -0,0 +1,136 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Recovered startup never creates an empty mapping or a missing storage pin. +use super::*; +use crate::raft::{ + cache::GroupLog, + storage::{LogError, ReplicaLog}, +}; + +pub(in crate::raft::catalog::range_directory) struct RecoveryStartup { + pub(in crate::raft::catalog::range_directory) request: Arc, + pub(in crate::raft::catalog::range_directory) authority: + Arc>, +} + +impl RangeDirectory { + /// Locate data only through the selected birth and this disk's immutable + /// recovery genesis. The supplied authority still verifies the permanent + /// external fence; a copied S3 binding cannot create a local seed receipt. + pub(crate) async fn bootstrap_selected_recovered_group_member( + &self, + birth: &CellGroupBirth, + log: &ReplicaLog, + authority: Arc>, + ) -> Result { + self.require_current_group_birth(birth).await?; + let local = log + .cell_group(&birth.route().history) + .map_err(CatalogError::Log)?; + let genesis = local + .recovery_genesis() + .map_err(CatalogError::Log)? + .ok_or(CatalogError::LostStableState)?; + let bytes = self + .bounded(async { + self.namespace + .inner + .read_versioned_bounded(&self.recovery_binding_key(birth.route()), wire::LIMIT) + .await? + .map(|(bytes, _)| bytes) + .ok_or(CatalogError::LostStableState) + }) + .await?; + let binding: super::seed::Binding = + record::decode(wire::GROUP_RECOVERY_BINDING, &bytes, wire::LIMIT)?; + let request = self.decode_group_recovery(&binding.request)?; + if request.replacement().route() != birth.route() + || genesis.recovery_request() != Some(request.digest()) + || genesis.as_ref() != &binding.genesis + { + return Err(CatalogError::LostStableState); + } + self.bootstrap_recovered_group_member(&request, log, authority) + .await + } + + pub(super) async fn recovered_member( + &self, + request: &CellGroupRecoveryRequest, + log: &ReplicaLog, + ) -> Result, CatalogError> { + let (_, binding, digest) = self.published_recovery(request).await?; + let selected = request + .replacement() + .voters() + .iter() + .chain(request.replacement().learners()) + .find(|member| member.node == log.limits().node) + .ok_or(CatalogError::NotInitialMember)?; + let receipt = self + .recovery_seed_receipt(request, selected, digest) + .await?; + receipt.check_log(log)?; + let group = log + .cell_group(&request.replacement().route().history) + .map_err(|error| match error { + LogError::NotFound => CatalogError::LostStableState, + error => CatalogError::Log(error), + })?; + if group.identity().member != selected.member + || group + .recovery_genesis() + .map_err(CatalogError::Log)? + .as_deref() + != Some(&binding.genesis) + { + return Err(CatalogError::LostStableState); + } + let marker = group + .group_manifest() + .map_err(CatalogError::Log)? + .ok_or(CatalogError::LostStableState)?; + if marker.route() != request.replacement().route() + || marker.recovery_request() != Some(request.digest()) + || marker + .recovery_commitment() + .map_err(|_| CatalogError::Corrupt("recovery lineage"))? + != Some(binding.commitment) + { + return Err(CatalogError::LostStableState); + } + super::super::bootstrap::check_member(&group)?; + Ok(group) + } + + /// Resume only the exact physically seeded member of the selected recovery. + /// The owner must still verify all local bytes and establish a fresh-term + /// quorum barrier; the pending lifecycle refuses every application request. + pub fn bootstrap_recovered_group_member<'a>( + &'a self, + request: &'a CellGroupRecoveryRequest, + log: &'a ReplicaLog, + authority: Arc>, + ) -> futures_util::future::BoxFuture<'a, Result> { + Box::pin(async move { + let request = Arc::new(self.decode_group_recovery(request.encode())?); + self.bounded(async { + self.verify_recovery_fence(&request, authority.as_ref()) + .await?; + let group = self.recovered_member(&request, log).await?; + let birth = self.materialize_group_birth(request.replacement()).await?; + self.verify_recovery_fence(&request, authority.as_ref()) + .await?; + self.published_recovery(&request).await?; + Ok(CellGroupBootstrapPermit::recovered( + group, + birth, + self.clone(), + log.clone(), + RecoveryStartup { request, authority }, + )) + }) + .await + }) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/recovery/managed_bootstrap.rs b/crates/cellar/durability/raft/cell_groups/directory/recovery/managed_bootstrap.rs new file mode 100644 index 000000000..189179fa7 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/recovery/managed_bootstrap.rs @@ -0,0 +1,317 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Attach retained recovery state under the pending or released directory decision. +use super::*; +use crate::raft::{ + cache::GroupLog, + cell_groups::GroupManifest, + fleet_catalog::{ + FleetHistory, FleetHistoryState, FleetHostAuthorization, FleetPublishedRecoveryVoter, + FleetRecoveryAttachRequest, + }, + member_state::FleetGroupBinding, + storage::{LogError, ReplicaLog}, +}; +use cellar_logic::consensus::{EngineInstanceId, MemberId}; + +pub(in crate::raft::catalog::range_directory) struct ManagedRecoveryStartup { + pub(in crate::raft::catalog::range_directory) request: FleetRecoveryAttachRequest, + pub(in crate::raft::catalog::range_directory) host: FleetHostAuthorization, + pub(in crate::raft::catalog::range_directory) deadline: u64, +} + +impl RangeDirectory { + /// A retained seed can keep its verified images on boot without starting + /// Raft. Publication and attachment still require independent Fleet authority. + pub(crate) async fn park_retained_managed_recovery( + &self, + group: &GroupLog, + marker: &GroupManifest, + history: &FleetHistory, + engine: EngineInstanceId, + ) -> Result { + self.bounded(async { + let local = group.identity(); + let route = marker.route(); + let plan = marker.recovery_request().ok_or(CatalogError::Fenced)?; + let activated = marker.recovery_activation().is_some(); + let (view, _) = self.current().await?; + let phase = view + .head + .recovery + .as_deref() + .map(|lock| (&lock.phase, lock)); + let publication = match phase { + Some((RecoveryPhase::ManagedSource { .. }, lock)) + if !activated && !lock.blocks_startup(self, route)? => + { + None + } + Some(( + RecoveryPhase::ManagedPublished { + replacement, + publication, + .. + }, + lock, + )) if *replacement == plan && lock.blocks_startup(self, route)? => { + self.selected_route_in(&view, route).await?; + Some(*publication) + } + _ if activated => return Ok(false), + _ => return Err(CatalogError::Fenced), + }; + match history.state() { + FleetHistoryState::Reserved => (), + FleetHistoryState::Live if activated && publication.is_some() => (), + _ => return Err(CatalogError::Fenced), + } + if !history.allows_managed_genesis() + || history.birth_digest() != route.birth_digest + || !history.binding().matches_scope(&local.scope) + { + return Err(CatalogError::Fenced); + } + let (birth, origin) = self.birth_reservation(route).await?; + if origin != Some(plan) + || !birth + .voters() + .iter() + .any(|voter| voter.node == local.node && voter.member == local.member) + { + return Err(CatalogError::Fenced); + } + let (genesis, commitment) = self + .require_retained_managed_seed(group, route, plan, history.binding(), engine, false) + .await?; + check_managed_recovery_member( + group, + marker, + &genesis, + (commitment, publication), + local.member, + history.binding(), + )?; + Ok(true) + }) + .await + } + + pub(in crate::raft::catalog::range_directory) async fn require_serving_managed_recovery_member( + &self, + birth: &CellGroupBirth, + group: &GroupLog, + marker: &GroupManifest, + log: &ReplicaLog, + host: &FleetHostAuthorization, + ) -> Result<(), CatalogError> { + let boot = self + .require_managed_retained_group_boot(birth, log, host) + .await?; + let route = birth.route(); + let (publication, commitment, position) = + marker.recovery_activation().ok_or(CatalogError::Fenced)?; + let plan = marker.recovery_request().ok_or(CatalogError::Fenced)?; + let operation = ProposalId::new(marker.recovery_operation().ok_or(CatalogError::Fenced)?) + .map_err(|_| CatalogError::Corrupt("managed recovery operation"))?; + let (view, _) = self.current().await?; + let (released, _) = self + .managed_recovery_serving_record(&view, operation) + .await? + .ok_or(CatalogError::Fenced)?; + if released.plan != plan || released.publication != publication { + return Err(CatalogError::Fenced); + } + + let (reserved, origin) = self.birth_reservation(route).await?; + let local = group.identity(); + if origin != Some(plan) + || marker.route() != route + || !reserved + .voters() + .iter() + .any(|voter| voter.node == local.node && voter.member == local.member) + { + return Err(CatalogError::Fenced); + } + let authorization = host + .group_authorization( + &route.history, + Duration::from_millis(self.limits.operation_timeout_ms), + ) + .await + .map_err(CatalogError::from)?; + if !authorization.check_birth(self, birth) + || !authorization + .matches_replica(local, &group.membership().map_err(CatalogError::Log)?) + { + return Err(CatalogError::ManagedAdmissionRequired); + } + let (genesis, seeded) = self + .require_retained_managed_seed( + group, + route, + plan, + authorization.binding(), + log.storage_instance().map_err(CatalogError::Log)?, + true, + ) + .await?; + if commitment != seeded + || self + .observe_managed_recovery_activation(route, plan, publication, commitment) + .await? + != Some(position) + { + return Err(CatalogError::LostStableState); + } + check_managed_recovery_member( + group, + marker, + &genesis, + (commitment, Some(publication)), + local.member, + authorization.binding(), + )?; + self.require_current_group_birth(birth).await?; + if self + .require_managed_retained_group_boot(birth, log, host) + .await? + != boot + { + return Err(CatalogError::Fenced); + } + Ok(()) + } + + pub(crate) fn bootstrap_managed_pending_recovery_voter<'a>( + &'a self, + request: &'a FleetRecoveryAttachRequest, + history: GroupHistory, + log: &'a ReplicaLog, + host: &'a FleetHostAuthorization, + deadline: u64, + ) -> futures_util::future::BoxFuture<'a, Result> { + Box::pin(self.bounded(async move { + let (group, authority) = self + .verify_published_managed_recovery_member(request, history, log, host, deadline) + .await?; + Ok(CellGroupBootstrapPermit::managed_pending_recovery( + group, + authority.birth, + self.clone(), + log.clone(), + ManagedRecoveryStartup { + request: request.clone(), + host: host.clone(), + deadline, + }, + )) + })) + } + + pub(super) async fn verify_published_managed_recovery_member( + &self, + request: &FleetRecoveryAttachRequest, + history: GroupHistory, + log: &ReplicaLog, + host: &FleetHostAuthorization, + deadline: u64, + ) -> Result<(GroupLog, FleetPublishedRecoveryVoter), CatalogError> { + let now = log + .domain() + .mono_ms() + .map_err(|_| CatalogError::Runtime(crate::raft::completion::RuntimeError::Closed))?; + let timeout = deadline + .checked_sub(now) + .filter(|remaining| *remaining > 0) + .map(Duration::from_millis) + .ok_or(CatalogError::Unconfirmed)?; + let authority = host + .authorize_published_recovery_voter(self, request, history, timeout) + .await + .map_err(CatalogError::from)?; + let boot = self.require_managed_group_boot(log, host).await?; + if boot != authority.current_boot { + return Err(CatalogError::Fenced); + } + self.namespace + .verify_node_storage_identity( + log.limits().node, + log.limits().disk, + log.storage_instance().map_err(CatalogError::Log)?, + ) + .await?; + let voter = authority + .birth + .voters() + .iter() + .find(|voter| voter.node == log.limits().node) + .ok_or(CatalogError::NotInitialMember)?; + let genesis = self + .require_published_managed_voter_seed( + &authority.birth, + request.plan_digest, + &authority.seeds, + voter.member, + &authority.seeded_boot, + ) + .await?; + let group = log.cell_group(&history).map_err(|error| match error { + LogError::NotFound => CatalogError::LostStableState, + error => CatalogError::Log(error), + })?; + let marker = group + .group_manifest() + .map_err(CatalogError::Log)? + .ok_or(CatalogError::LostStableState)?; + check_managed_recovery_member( + &group, + &marker, + &genesis, + ( + authority.seeds.genesis_commitment(), + Some(authority.publication.journal_digest()), + ), + voter.member, + &authority.group, + )?; + Ok((group, authority)) + } +} + +fn check_managed_recovery_member( + group: &GroupLog, + marker: &GroupManifest, + genesis: &GroupManifest, + expected: ([u8; 32], Option<[u8; 32]>), + member: MemberId, + binding: &FleetGroupBinding, +) -> Result<(), CatalogError> { + let (commitment, publication) = expected; + if group.identity().member != member + || group + .recovery_genesis() + .map_err(CatalogError::Log)? + .as_deref() + != Some(genesis) + || marker.route() != genesis.route() + || marker.source_seal().is_some() + || marker.recovery_request() != genesis.recovery_request() + || marker + .recovery_activation() + .is_some_and(|(observed, _, _)| Some(observed) != publication) + || marker + .recovery_commitment() + .map_err(|_| CatalogError::Corrupt("managed recovery lineage"))? + != Some(commitment) + || group + .membership() + .map_err(CatalogError::Log)? + .fleet_status() + .is_none_or(|status| &status.binding != binding) + { + return Err(CatalogError::LostStableState); + } + super::super::bootstrap::check_member(group) +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/recovery/managed_completion.rs b/crates/cellar/durability/raft/cell_groups/directory/recovery/managed_completion.rs new file mode 100644 index 000000000..a36a0c2c7 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/recovery/managed_completion.rs @@ -0,0 +1,289 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Select the complete Fleet settlement and reopen routing through one directory CAS. +use super::{ + managed_publication::{ManagedRecoveryPublication, ManagedRecoveryPublicationInput}, + *, +}; +use crate::{ + asyncrt::DomainToken, + raft::{ + catalog::CellGroupArchiveBindingError, + cell_groups::CellGroupArchivePool, + fleet_catalog::{FleetCatalogError, FleetManagedRecoveryCompletion}, + }, +}; + +const KIND: u8 = 231; +const LIMIT: usize = 4096; + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Record { + namespace: [u8; 32], + operation: [u8; 16], + transition: [u8; 16], + transition_plan: [u8; 32], + intent: [u8; 32], + pub(super) plan: [u8; 32], + seeds: [u8; 32], + pub(super) publication: [u8; 32], + fleet: [u8; 32], + fleet_revision: u64, + fleet_settlement: [u8; 32], +} + +impl Record { + pub(super) fn matches_publication( + &self, + input: &ManagedRecoveryPublicationInput<'_>, + publication: [u8; 32], + ) -> bool { + self.transition == input.transition.bytes() + && self.transition_plan == input.transition_plan + && self.intent == input.intent + && self.plan == input.plan + && self.seeds == input.seeds + && self.publication == publication + } +} + +impl RangeDirectory { + /// A rooted release is historical evidence. Replica startup must also + /// verify the current route, Fleet boot and local activation. + pub(super) async fn managed_recovery_serving_record( + &self, + view: &DirectoryView, + operation: ProposalId, + ) -> Result, CatalogError> { + let Some(bytes) = Index::new(self) + .lookup(view.head.operations, &serving_key(operation)) + .await? + else { + return Ok(None); + }; + let selected: Record = record::decode(KIND, &bytes, LIMIT)?; + if selected.namespace != self.namespace.root_digest + || selected.operation != operation.bytes() + { + return Err(CatalogError::Corrupt("managed recovery serving receipt")); + } + let transition = ProposalId::new(selected.transition) + .map_err(|_| CatalogError::Corrupt("managed recovery transition identity"))?; + if view + .head + .recovery + .as_deref() + .is_some_and(|lock| lock.owns_transition(transition)) + { + return Err(CatalogError::Corrupt( + "managed recovery release still fenced", + )); + } + let digest = record::digest(&bytes); + self.verify_recovered_transition(&view.head, transition, selected.transition_plan, digest) + .await?; + Ok(Some((selected, digest))) + } + + pub(crate) async fn observe_released_managed_recovery_completion( + &self, + input: &ManagedRecoveryPublicationInput<'_>, + ) -> Result, CatalogError> { + self.bounded(async { + let (view, _) = self.current().await?; + let Some((release, digest)) = self + .managed_recovery_serving_record(&view, input.mutation.operation()) + .await? + else { + return Ok(None); + }; + let publication = self + .confirmed(&view.head, input.mutation) + .await? + .ok_or(CatalogError::Fenced)?; + if !release.matches_publication(input, publication.journal_digest()) { + return Err(CatalogError::Fenced); + } + Ok(Some(digest)) + }) + .await + } + + /// Verify the exact serving decision and, when supplied, the old routes + /// that a live retirement request is about to dispose of. + pub(crate) async fn check_managed_recovery_retirement( + &self, + recovery: ProposalId, + transition: ProposalId, + intent: [u8; 32], + completion: [u8; 32], + old_routes: &[GroupRoute], + ) -> Result<(), CatalogError> { + self.bounded(async { + let (view, _) = self.current().await?; + let (release, digest) = self + .managed_recovery_serving_record(&view, recovery) + .await? + .ok_or(CatalogError::Fenced)?; + if release.transition != transition.bytes() + || release.intent != intent + || digest != completion + { + return Err(CatalogError::Fenced); + } + for route in old_routes { + match self.selected_route_in(&view, route).await { + Ok(()) => return Err(CatalogError::Fenced), + Err(CatalogError::Fenced) => (), + Err(error) => return Err(error), + } + } + Ok(()) + }) + .await + } + + /// Root the settlement, or release the recovery lock using its exact + /// previously rooted receipt. Both decisions survive lost confirmation. + pub(crate) async fn record_managed_recovery_completion( + &self, + input: &ManagedRecoveryPublicationInput<'_>, + proof: &FleetManagedRecoveryCompletion, + release: Option<([u8; 32], &DomainToken, &CellGroupArchivePool)>, + ) -> Result<[u8; 32], CatalogError> { + self.bounded(async { + let (mut view, version) = self.current().await?; + let publication = self + .confirmed(&view.head, input.mutation) + .await? + .ok_or(CatalogError::Fenced)?; + let (fleet, fleet_revision, fleet_settlement) = + proof.directory_binding(self, input, publication.journal_digest())?; + let operation = input.mutation.operation(); + let selected = Record { + namespace: self.namespace.root_digest, + operation: operation.bytes(), + transition: input.transition.bytes(), + transition_plan: input.transition_plan, + intent: input.intent, + plan: input.plan, + seeds: input.seeds, + publication: publication.journal_digest(), + fleet, + fleet_revision, + fleet_settlement, + }; + let bytes = record::encode(KIND, &selected, LIMIT)?; + let completion = record::digest(&bytes); + if release.is_some_and(|(expected, _, _)| expected != completion) { + return Err(CatalogError::Conflict( + "managed recovery completion changed", + )); + } + if let Some((_, recorded)) = self + .managed_recovery_serving_record(&view, operation) + .await? + { + return if recorded == completion { + Ok(completion) + } else { + Err(CatalogError::Conflict( + "managed recovery serving receipt changed", + )) + }; + } + + let authority = ManagedRecoveryPublication::new(input); + authority.check_published(self, &view, &publication).await?; + let key = format!("recovery-completion/{}", hex::encode(operation.bytes())); + let index = Index::new(self); + let existing = index.lookup(view.head.operations, &key).await?; + if let Some(existing) = &existing { + if *existing != bytes { + return Err(CatalogError::Conflict( + "managed recovery completion changed", + )); + } + self.verify_recovered_transition( + &view.head, + input.transition, + selected.transition_plan, + completion, + ) + .await?; + } + if let Some((_, domain, archives)) = release { + if existing.is_none() { + return Err(CatalogError::Unconfirmed); + } + // Recovered histories need independent archives before normal + // voter boot can enable archiving or local log reclamation. + for birth in input.births { + let birth = self.open_historical_birth(birth.route()).await?; + Box::pin(self.initialize_recovered_group_archive(domain, archives, &birth)) + .await + .map_err(|error| match error { + CellGroupArchiveBindingError::Catalog(error) => error, + CellGroupArchiveBindingError::Archive(error) => { + CatalogError::from(FleetCatalogError::from(error)) + } + })?; + } + view.head.operations = index + .update( + view.head.operations, + &serving_key(operation), + Some(bytes), + 0, + ) + .await?; + view.head.recovery = None; + let after = self.commit_transition_head(&view.head, version).await?; + if self + .managed_recovery_serving_record(&after, operation) + .await? + .is_none_or(|(_, recorded)| recorded != completion) + { + return Err(CatalogError::Unconfirmed); + } + return Ok(completion); + } + if existing.is_some() { + return Ok(completion); + } + for birth in input.births { + self.open_historical_birth(birth.route()).await?; + } + self.supersede_transition_for_recovery( + &mut view.head, + input.transition, + selected.transition_plan, + completion, + ) + .await?; + view.head.operations = index + .update(view.head.operations, &key, Some(bytes.clone()), 0) + .await?; + let after = self.commit_transition_head(&view.head, version).await?; + if index.lookup(after.head.operations, &key).await?.as_deref() != Some(bytes.as_slice()) + { + return Err(CatalogError::Unconfirmed); + } + authority.check_selected(self, &after, &publication).await?; + self.verify_recovered_transition( + &after.head, + input.transition, + selected.transition_plan, + completion, + ) + .await?; + Ok(completion) + }) + .await + } +} + +fn serving_key(operation: ProposalId) -> String { + format!("recovery-serving/{}", hex::encode(operation.bytes())) +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/recovery/managed_publication.rs b/crates/cellar/durability/raft/cell_groups/directory/recovery/managed_publication.rs new file mode 100644 index 000000000..7c6b0b9dc --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/recovery/managed_publication.rs @@ -0,0 +1,229 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Atomically select all recovered ranges while retaining their startup fence. +use super::*; +use crate::raft::catalog::CellGroupBirthRequest; + +pub(crate) struct ManagedRecoveryPublicationInput<'a> { + pub(crate) intent: [u8; 32], + pub(crate) transition: ProposalId, + pub(crate) transition_plan: [u8; 32], + pub(crate) barrier: [u8; 32], + pub(crate) plan: [u8; 32], + pub(crate) seeds: [u8; 32], + pub(crate) mutation: &'a DirectoryMutation, + pub(crate) births: &'a [CellGroupBirthRequest], +} + +pub(in crate::raft::catalog::range_directory) struct ManagedRecoveryPublication<'a> { + input: &'a ManagedRecoveryPublicationInput<'a>, +} + +impl<'a> ManagedRecoveryPublication<'a> { + pub(super) fn new(input: &'a ManagedRecoveryPublicationInput<'a>) -> Self { + Self { input } + } + + pub(in crate::raft::catalog::range_directory) fn check( + &self, + view: &DirectoryView, + mutation: &DirectoryMutation, + ) -> Result<(), CatalogError> { + let input = self.input; + if mutation.encode() != input.mutation.encode() + || mutation.request.expected != Some(input.barrier) + || view.digest != input.barrier + || !view.head.recovery.as_ref().is_some_and(|lock| { + lock.request == input.intent + && matches!(&lock.phase, RecoveryPhase::ManagedSource { transition, plan, .. } + if *transition == input.transition.bytes() && *plan == input.transition_plan) + }) + { + return Err(CatalogError::Fenced); + } + Ok(()) + } + + pub(in crate::raft::catalog::range_directory) async fn check_child( + &self, + directory: &RangeDirectory, + route: &GroupRoute, + ) -> Result<(), CatalogError> { + let birth = self + .input + .births + .iter() + .find(|birth| birth.route() == route) + .ok_or(CatalogError::Fenced)?; + directory + .require_managed_recovery_identity(birth, self.input.plan) + .await + } + + pub(in crate::raft::catalog::range_directory) fn published_lock( + &self, + view: &DirectoryView, + publication: [u8; 32], + ) -> Result, CatalogError> { + let lock = view.head.recovery.as_ref().ok_or(CatalogError::Fenced)?; + let RecoveryPhase::ManagedSource { + transition, + plan, + groups, + } = &lock.phase + else { + return Err(CatalogError::Fenced); + }; + let mut groups = groups.clone(); + groups.extend(self.input.births.iter().map(|birth| { + let history = birth.route().history; + (history.group.bytes(), history.incarnation.bytes()) + })); + groups.sort_unstable(); + Ok(Box::new(RecoveryLock { + request: lock.request, + base: lock.base, + source: lock.source.clone(), + phase: RecoveryPhase::ManagedPublished { + transition: *transition, + plan: *plan, + groups, + replacement: self.input.plan, + seeds: self.input.seeds, + publication, + }, + })) + } + + pub(in crate::raft::catalog::range_directory) async fn check_selected( + &self, + directory: &RangeDirectory, + view: &DirectoryView, + publication: &DirectoryPublication, + ) -> Result<(), CatalogError> { + let input = self.input; + if let Some((released, _)) = directory + .managed_recovery_serving_record(view, input.mutation.operation()) + .await? + { + if !released.matches_publication(input, publication.journal_digest()) { + return Err(CatalogError::Fenced); + } + return Ok(()); + } + self.check_published(directory, view, publication).await + } + + pub(in crate::raft::catalog::range_directory) async fn check_published( + &self, + directory: &RangeDirectory, + view: &DirectoryView, + publication: &DirectoryPublication, + ) -> Result<(), CatalogError> { + let input = self.input; + if !view.head.recovery.as_ref().is_some_and(|lock| { + lock.request == input.intent + && matches!(&lock.phase, RecoveryPhase::ManagedPublished { + transition, plan, replacement, seeds, publication: published, .. + } if *transition == input.transition.bytes() && *plan == input.transition_plan + && *replacement == input.plan && *seeds == input.seeds + && *published == publication.journal_digest()) + }) { + return Err(CatalogError::Fenced); + } + for birth in input.births { + directory.selected_route_in(view, birth.route()).await?; + } + Ok(()) + } +} + +impl RangeDirectory { + pub(crate) async fn observe_pending_managed_recovery_publication( + &self, + input: &ManagedRecoveryPublicationInput<'_>, + ) -> Result, CatalogError> { + self.bounded(async { + let (view, _) = self.current().await?; + let publication = self.confirmed(&view.head, input.mutation).await?; + if let Some(publication) = &publication { + ManagedRecoveryPublication::new(input) + .check_published(self, &view, publication) + .await?; + } + Ok(publication) + }) + .await + } + + pub(crate) async fn observe_managed_recovery_publication( + &self, + input: &ManagedRecoveryPublicationInput<'_>, + ) -> Result, CatalogError> { + self.bounded(async { + let (view, _) = self.current().await?; + let publication = self.confirmed(&view.head, input.mutation).await?; + if let Some(publication) = &publication { + ManagedRecoveryPublication::new(input) + .check_selected(self, &view, publication) + .await?; + } + Ok(publication) + }) + .await + } + + /// Record the exact selected births before Fleet promotes their histories. + /// The recovery lock remains the exclusive startup and routing authority. + pub(crate) async fn materialize_managed_recovery_births( + &self, + input: &ManagedRecoveryPublicationInput<'_>, + batch: std::ops::Range, + ) -> Result<(), CatalogError> { + self.bounded(async { + let (view, _) = self.current().await?; + let publication = self + .confirmed(&view.head, input.mutation) + .await? + .ok_or(CatalogError::Fenced)?; + let permit = ManagedRecoveryPublication { input }; + permit.check_published(self, &view, &publication).await?; + let births = input.births.get(batch).ok_or(CatalogError::Limit)?; + for birth in births { + self.require_managed_recovery_identity(birth, input.plan) + .await?; + self.decide_live_birth(birth).await?; + } + let (after, _) = self.current().await?; + permit.check_published(self, &after, &publication).await + }) + .await + } + + pub(crate) async fn publish_managed_recovery( + &self, + input: &ManagedRecoveryPublicationInput<'_>, + ) -> Result { + self.check_request(&input.mutation.request)?; + if input.births.len() != input.mutation.request.children.len() + || input + .births + .iter() + .zip(&input.mutation.request.children) + .any(|(birth, child)| { + !child + .decode(self) + .is_ok_and(|route| route == *birth.route()) + }) + || [input.plan, input.seeds].contains(&[0; 32]) + { + return Err(CatalogError::Fenced); + } + let permit = ManagedRecoveryPublication::new(input); + self.bounded(Box::pin(self.publish_inner( + input.mutation, + PublicationAuthority::ManagedRecovery(&permit), + ))) + .await + } +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/recovery/managed_seed.rs b/crates/cellar/durability/raft/cell_groups/directory/recovery/managed_seed.rs new file mode 100644 index 000000000..785e16959 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/recovery/managed_seed.rs @@ -0,0 +1,478 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! One managed voter seed. The Fleet reservation and directory recovery origin +//! must both exist before the writer can sync a non-serving genesis. +use super::{ + seed::{ManagedRecoverySeed, RecoverySeedPermit}, + *, +}; +use crate::raft::{ + cache::GroupLog, + cell_groups::{CellGroupApplier, GroupManifest, PreparedRecoveryData, RecoverySeed}, + fleet_catalog::ManagedRecoverySeedAuthority, + fleet_stop::BootBinding, + member_state::FleetGroupBinding, + storage::{LogError, ReplicaLog}, +}; +use cellar_logic::consensus::{DiskLineage, EngineInstanceId, MemberId}; + +const BINDING: u8 = 226; +const INTENT: u8 = 227; +const COMPLETE: u8 = 228; + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Binding { + plan: [u8; 32], + genesis: GroupManifest, + commitment: [u8; 32], +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Receipt { + binding: [u8; 32], + member: u64, + boot: BootBinding, +} + +/// Exact durable receipt for a managed recovery voter. Publication and +/// activation require independent topology and quorum proofs. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct ManagedRecoveryVoterSeed { + pub route: GroupRoute, + pub plan_digest: [u8; 32], + pub genesis_commitment: [u8; 32], + pub member: MemberId, + pub boot: BootBinding, +} + +/// The immutable genesis and completed voter set for one replacement range. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct ManagedRecoverySeedQuorum { + birth: [u8; 32], + binding: [u8; 32], + commitment: [u8; 32], + members: Vec, +} + +impl ManagedRecoverySeedQuorum { + pub fn birth_digest(&self) -> [u8; 32] { + self.birth + } + + pub fn genesis_commitment(&self) -> [u8; 32] { + self.commitment + } + + pub fn members(&self) -> &[u64] { + &self.members + } +} + +pub(crate) struct PendingManagedRecoverySeed { + proof: ManagedRecoveryVoterSeed, + binding: [u8; 32], +} + +impl RangeDirectory { + pub(crate) async fn seed_managed_recovery_voter( + &self, + domain: &impl CellGroupArchiveExecution, + target: &crate::raft::catalog::CellGroupBirthRequest, + prepared: PreparedRecoveryData, + authority: ManagedRecoverySeedAuthority, + log: &ReplicaLog, + deadline: u64, + ) -> Result<(PendingManagedRecoverySeed, CellGroupApplier), CellGroupRecoveryError> { + let marker = prepared.manifest.marker(); + let plan = marker.recovery_request().ok_or(CatalogError::Fenced)?; + if !marker.is_recovery_genesis() + || marker.route() != target.route() + || prepared.store.route() != target.route() + || marker + .recovery_commitment() + .map_err(|_| CatalogError::Corrupt("managed seed commitment"))? + != Some(prepared.commitment) + || !authority.group.matches_scope(&target.route().history) + || authority.boot.node != log.limits().node.bytes() + || authority.boot.disk != log.limits().disk.bytes() + || authority.boot.process != log.limits().process.bytes() + || authority.boot.engine != log.storage_instance().map_err(CatalogError::Log)?.bytes() + { + return Err(CatalogError::Fenced.into()); + } + self.require_managed_recovery_identity(target, plan).await?; + let _node = self.namespace.require_node_storage(log).await?; + let member = target + .voters() + .iter() + .find(|member| member.node == log.limits().node) + .ok_or(CatalogError::NotInitialMember)?; + let binding_bytes = record::encode( + BINDING, + &Binding { + plan, + genesis: marker.clone(), + commitment: prepared.commitment, + }, + wire::LIMIT, + )?; + let binding = record::digest(&binding_bytes); + let proof = ManagedRecoveryVoterSeed { + route: target.route().clone(), + plan_digest: plan, + genesis_commitment: prepared.commitment, + member: member.member, + boot: authority.boot.clone(), + }; + let pending = PendingManagedRecoverySeed { proof, binding }; + let (intent, complete) = pending.receipts()?; + self.bounded(async { + self.put_exact_managed_seed( + &self.managed_seed_key(target.route(), "binding"), + binding_bytes, + ) + .await?; + let key = self.managed_seed_key(target.route(), &pending.member_key("complete")); + if let Some((bytes, _)) = self + .namespace + .inner + .read_versioned_bounded(&key, wire::LIMIT) + .await? + { + if bytes != complete { + return Err(CatalogError::Conflict("managed recovery seed receipt")); + } + log.cell_group(&target.route().history) + .map_err(|error| match error { + LogError::NotFound => CatalogError::LostStableState, + error => CatalogError::Log(error), + })?; + } + self.put_exact_managed_seed( + &self.managed_seed_key(target.route(), &pending.member_key("intent")), + intent, + ) + .await + }) + .await?; + + let permit = RecoverySeedPermit::for_local( + target, + marker, + prepared.commitment, + log, + Some(ManagedRecoverySeed { + group: authority.group, + boot: authority.boot, + deadline, + }), + )?; + let genesis = marker.clone(); + let seed = RecoverySeed::new(permit, prepared.clone()).map_err(CatalogError::Log)?; + let group = Box::pin(log.install_recovery_seed(seed)) + .await + .map_err(CatalogError::Log)?; + let (applier, _) = domain + .blocking(move || CellGroupApplier::recover_seeded(group, &prepared.store, &genesis)) + .map_err(|_| CellGroupArchiveError::Stopped)? + .await + .map_err(CellGroupArchiveError::Worker)? + .map_err(CellGroupArchiveError::Replay)?; + Ok((pending, applier)) + } + + pub(crate) async fn confirm_managed_recovery_voter( + &self, + target: &crate::raft::catalog::CellGroupBirthRequest, + pending: PendingManagedRecoverySeed, + authority: &ManagedRecoverySeedAuthority, + ) -> Result { + let proof = &pending.proof; + if proof.route != *target.route() + || proof.boot != authority.boot + || !authority.group.matches_scope(&proof.route.history) + { + return Err(CatalogError::Fenced); + } + self.require_managed_recovery_identity(target, proof.plan_digest) + .await?; + let (_, complete) = pending.receipts()?; + self.bounded(self.put_exact_managed_seed( + &self.managed_seed_key(&proof.route, &pending.member_key("complete")), + complete, + )) + .await?; + Ok(pending.proof) + } + + /// Read the immutable receipts for a quorum of the planned voters. + /// Reopening a selected set never silently includes a later completion. + pub(crate) async fn observe_managed_recovery_seed_quorum( + &self, + target: &crate::raft::catalog::CellGroupBirthRequest, + plan: [u8; 32], + boots: &[BootBinding], + selected: Option<&ManagedRecoverySeedQuorum>, + ) -> Result { + self.bounded(async { + self.require_managed_recovery_identity(target, plan).await?; + let route = target.route(); + let voters = target.voters(); + if voters.len() != 3 + || boots.len() != voters.len() + || !target.learners().is_empty() + || selected.is_some_and(|value| { + value.birth != route.birth_digest + || !(2..=voters.len()).contains(&value.members.len()) + || value.members.windows(2).any(|pair| pair[0] >= pair[1]) + }) + { + return Err(CatalogError::Fenced); + } + let (binding, digest) = self.managed_seed_binding(route, plan).await?; + let mut disks = BTreeSet::new(); + let mut engines = BTreeSet::new(); + let mut members = Vec::with_capacity(voters.len()); + for (voter, boot) in voters.iter().zip(boots) { + if boot.node != voter.node.bytes() { + return Err(CatalogError::Corrupt("managed seed voter boot binding")); + } + let member = voter.member.get(); + if selected.is_some_and(|value| value.members.binary_search(&member).is_err()) { + continue; + } + if !self + .has_managed_seed_receipt(route, member, boot, digest) + .await? + { + if selected.is_some() { + return Err(CatalogError::Unconfirmed); + } + continue; + } + boot.validate().map_err(CatalogError::Log)?; + let disk = DiskLineage::new(boot.disk) + .map_err(|_| CatalogError::Corrupt("managed seed disk"))?; + let engine = EngineInstanceId::new(boot.engine) + .map_err(|_| CatalogError::Corrupt("managed seed engine"))?; + if !disks.insert(disk) || !engines.insert(engine) { + return Err(CatalogError::Conflict("duplicate managed seed storage")); + } + self.namespace + .verify_node_storage_identity(voter.node, disk, engine) + .await?; + members.push(member); + } + if members.len() <= voters.len() / 2 { + return Err(CatalogError::Unconfirmed); + } + let quorum = ManagedRecoverySeedQuorum { + birth: route.birth_digest, + binding: digest, + commitment: binding.commitment, + members, + }; + if selected.is_some_and(|expected| expected != &quorum) { + return Err(CatalogError::Corrupt("managed seed quorum binding")); + } + Ok(quorum) + }) + .await + } + + pub(super) async fn require_published_managed_voter_seed( + &self, + target: &CellGroupBirthRequest, + plan: [u8; 32], + selected: &ManagedRecoverySeedQuorum, + voter: MemberId, + boot: &BootBinding, + ) -> Result { + self.require_managed_recovery_identity(target, plan).await?; + let route = target.route(); + let (binding, digest) = self.managed_seed_binding(route, plan).await?; + if selected.birth != route.birth_digest + || selected.binding != digest + || selected.commitment != binding.commitment + { + return Err(CatalogError::Corrupt("published recovery seed binding")); + } + // A voter may have completed its seed without being needed in the + // selected majority. It must still present its own exact receipt. + if !self + .has_managed_seed_receipt(route, voter.get(), boot, digest) + .await? + { + return Err(CatalogError::LostStableState); + } + Ok(binding.genesis) + } + + /// A crash may leave a synced genesis with only its seed intent. Retain + /// that owner, but require a complete receipt separately before voting. + pub(super) async fn require_retained_managed_seed( + &self, + group: &GroupLog, + route: &GroupRoute, + plan: [u8; 32], + fleet: &FleetGroupBinding, + engine: EngineInstanceId, + require_complete: bool, + ) -> Result<(GroupManifest, [u8; 32]), CatalogError> { + let local = group.identity(); + let (binding, digest) = self.managed_seed_binding(route, plan).await?; + let member = local.member.get(); + let intent = self.read_managed_seed_intent(route, member).await?; + let receipt: Receipt = record::decode(INTENT, &intent, wire::LIMIT)?; + let boot = &receipt.boot; + boot.validate().map_err(CatalogError::Log)?; + if receipt.binding != digest + || receipt.member != member + || boot.catalog != fleet.catalog() + || boot.fleet != fleet.fleet() + || boot.node != local.node.bytes() + || boot.disk != local.disk.bytes() + || boot.engine != engine.bytes() + { + return Err(CatalogError::LostStableState); + } + let expected = record::encode(COMPLETE, &receipt, wire::LIMIT)?; + match self.read_managed_seed_complete(route, member).await? { + Some(complete) if complete != expected => { + return Err(CatalogError::Corrupt("retained managed seed receipt")); + } + None if require_complete => return Err(CatalogError::LostStableState), + _ => (), + } + self.namespace + .verify_node_storage_identity(local.node, local.disk, engine) + .await?; + Ok((binding.genesis, binding.commitment)) + } + + async fn managed_seed_binding( + &self, + route: &GroupRoute, + plan: [u8; 32], + ) -> Result<(Binding, [u8; 32]), CatalogError> { + let binding_key = self.managed_seed_key(route, "binding"); + let (bytes, _) = self + .namespace + .inner + .read_versioned_bounded(&binding_key, wire::LIMIT) + .await? + .ok_or(CatalogError::Unconfirmed)?; + let binding: Binding = record::decode(BINDING, &bytes, wire::LIMIT)?; + if binding.plan != plan + || !binding.genesis.is_recovery_genesis() + || binding.genesis.route() != route + || binding.genesis.recovery_request() != Some(plan) + || binding + .genesis + .recovery_commitment() + .map_err(|_| CatalogError::Corrupt("managed seed genesis"))? + != Some(binding.commitment) + { + return Err(CatalogError::Corrupt("managed seed genesis binding")); + } + let digest = record::digest(&bytes); + Ok((binding, digest)) + } + + async fn has_managed_seed_receipt( + &self, + route: &GroupRoute, + member: u64, + boot: &BootBinding, + digest: [u8; 32], + ) -> Result { + let receipt = Receipt { + binding: digest, + member, + boot: boot.clone(), + }; + let Some(complete) = self.read_managed_seed_complete(route, member).await? else { + return Ok(false); + }; + let intent = self.read_managed_seed_intent(route, member).await?; + if complete != record::encode(COMPLETE, &receipt, wire::LIMIT)? + || intent != record::encode(INTENT, &receipt, wire::LIMIT)? + { + return Err(CatalogError::Corrupt("managed seed receipt binding")); + } + Ok(true) + } + + async fn read_managed_seed_complete( + &self, + route: &GroupRoute, + member: u64, + ) -> Result>, CatalogError> { + let key = self.managed_seed_key(route, &seed_member_key(member, "complete")); + Ok(self + .namespace + .inner + .read_versioned_bounded(&key, wire::LIMIT) + .await? + .map(|(bytes, _)| bytes)) + } + + async fn read_managed_seed_intent( + &self, + route: &GroupRoute, + member: u64, + ) -> Result, CatalogError> { + let key = self.managed_seed_key(route, &seed_member_key(member, "intent")); + self.namespace + .inner + .read_versioned_bounded(&key, wire::LIMIT) + .await? + .map(|(bytes, _)| bytes) + .ok_or(CatalogError::Corrupt("missing managed seed intent")) + } + + pub(super) fn managed_seed_key(&self, route: &GroupRoute, leaf: &str) -> Path { + self.key(&format!( + "managed-recovery-seeds/{}/{}/{leaf}", + hex::encode(route.history.group.bytes()), + hex::encode(route.history.incarnation.bytes()) + )) + } + + async fn put_exact_managed_seed(&self, key: &Path, bytes: Vec) -> Result<(), CatalogError> { + let actual = self + .namespace + .inner + .create_and_observe_bounded(key, &bytes, wire::LIMIT) + .await?; + if actual != bytes { + return Err(CatalogError::Conflict("managed recovery seed identity")); + } + Ok(()) + } +} + +impl PendingManagedRecoverySeed { + fn member_key(&self, leaf: &str) -> String { + seed_member_key(self.proof.member.get(), leaf) + } + + fn receipts(&self) -> Result<(Vec, Vec), CatalogError> { + let receipt = Receipt { + binding: self.binding, + member: self.proof.member.get(), + boot: self.proof.boot.clone(), + }; + Ok(( + record::encode(INTENT, &receipt, wire::LIMIT)?, + record::encode(COMPLETE, &receipt, wire::LIMIT)?, + )) + } +} + +fn seed_member_key(member: u64, leaf: &str) -> String { + format!("voters/{member:016x}/{leaf}") +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/recovery/mod.rs b/crates/cellar/durability/raft/cell_groups/directory/recovery/mod.rs new file mode 100644 index 000000000..e4ffddcfb --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/recovery/mod.rs @@ -0,0 +1,500 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Irreversible fencing, complete fresh genesis, publication and activation. +//! Archive data and catalog observations never grant quorum or serving authority. +//! Rare catalog operations own boxed futures so composing them cannot inflate +//! a controller polling frame with multiple complete recovery state machines. +mod activation; +mod bootstrap; +mod publication; +mod request; +mod seed; +use super::*; +use crate::raft::{ + catalog::DisasterFenceAuthority, + cell_groups::{ + CellGroupArchiveBindingError, CellGroupArchiveCheckpointReceiver, CellGroupArchiveCoverage, + CellGroupArchiveError, CellGroupArchiveExecution, CellGroupArchivePool, + CellGroupArchiveTailPool, + }, +}; +pub use activation::{CellGroupRecoveryActivationAck, CellGroupRecoveryActivationPermit}; +pub(in crate::raft::catalog::range_directory) use bootstrap::RecoveryStartup; +pub(in crate::raft::catalog::range_directory) use publication::RecoveryPublication; +pub use request::{CellGroupDisasterFenceRequest, CellGroupRecoveryRequest}; +pub(crate) use seed::RecoverySeedPermit; +mod managed_bootstrap; +mod managed_completion; +pub(super) use managed_bootstrap::ManagedRecoveryStartup; +mod managed_publication; +pub(in crate::raft::catalog::range_directory) use managed_publication::ManagedRecoveryPublication; +pub(crate) use managed_publication::ManagedRecoveryPublicationInput; +mod managed_seed; +pub use managed_seed::{ManagedRecoverySeedQuorum, ManagedRecoveryVoterSeed}; +use serde::{Deserialize, Serialize}; + +#[derive(Debug)] +pub enum CellGroupRecoveryError { + Catalog(CatalogError), + Archive(CellGroupArchiveError), +} + +impl std::fmt::Display for CellGroupRecoveryError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "cell group recovery: {self:?}") + } +} + +impl std::error::Error for CellGroupRecoveryError { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + Some(match self { + Self::Catalog(error) => error, + Self::Archive(error) => error, + }) + } +} + +impl From for CellGroupRecoveryError { + fn from(error: CatalogError) -> Self { + Self::Catalog(error) + } +} + +impl From for CellGroupRecoveryError { + fn from(error: CellGroupArchiveError) -> Self { + Self::Archive(error) + } +} + +impl From for CellGroupRecoveryError { + fn from(error: CellGroupArchiveBindingError) -> Self { + match error { + CellGroupArchiveBindingError::Catalog(error) => Self::Catalog(error), + CellGroupArchiveBindingError::Archive(error) => Self::Archive(error), + } + } +} + +/// A topology-head CAS closes old startup and all competing topology mutations +/// atomically. It never stands in for stopping a running deployment. +#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(in crate::raft::catalog::range_directory) struct RecoveryLock { + request: [u8; 32], + base: [u8; 32], + source: wire::Route, + phase: RecoveryPhase, +} + +#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +enum RecoveryPhase { + Source, + ManagedSource { + transition: [u8; 16], + plan: [u8; 32], + groups: Vec<([u8; 16], [u8; 16])>, + }, + ManagedPublished { + transition: [u8; 16], + plan: [u8; 32], + groups: Vec<([u8; 16], [u8; 16])>, + replacement: [u8; 32], + seeds: [u8; 32], + publication: [u8; 32], + }, + Published { + publication: [u8; 32], + manifest: [u8; 32], + }, +} + +impl RecoveryLock { + pub(in crate::raft::catalog::range_directory) fn managed( + directory: &RangeDirectory, + request: [u8; 32], + basis: &crate::raft::fleet_catalog::ManagedRecoveryIntentBasis, + source: &GroupRoute, + routes: impl Iterator, + ) -> Result { + let mut groups: Vec<_> = routes + .map(|history| (history.group.bytes(), history.incarnation.bytes())) + .collect(); + groups.sort_unstable(); + let lock = Self { + request, + base: basis.directory_root, + source: wire::Route::encode(source, directory)?, + phase: RecoveryPhase::ManagedSource { + transition: basis.transition, + plan: basis.plan, + groups, + }, + }; + lock.check(directory)?; + Ok(lock) + } + + pub(in crate::raft::catalog::range_directory) fn owns_transition( + &self, + operation: ProposalId, + ) -> bool { + matches!(&self.phase, + RecoveryPhase::ManagedSource { transition, .. } + | RecoveryPhase::ManagedPublished { transition, .. } + if *transition == operation.bytes()) + } + + pub(in crate::raft::catalog::range_directory) fn blocks_startup( + &self, + directory: &RangeDirectory, + route: &GroupRoute, + ) -> Result { + let source = self.source_route(directory)?; + Ok(match &self.phase { + RecoveryPhase::ManagedSource { groups, .. } + | RecoveryPhase::ManagedPublished { groups, .. } => { + route.history.cluster == source.history.cluster + && route.history.namespace == source.history.namespace + && groups + .binary_search(&( + route.history.group.bytes(), + route.history.incarnation.bytes(), + )) + .is_ok() + } + _ => source == *route, + }) + } + + pub(in crate::raft::catalog::range_directory) fn blocks_application_route( + &self, + directory: &RangeDirectory, + route: &GroupRoute, + ) -> Result { + Ok(matches!(self.phase, RecoveryPhase::ManagedPublished { .. }) + && self.blocks_startup(directory, route)?) + } + + pub(super) fn check(&self, directory: &RangeDirectory) -> Result<(), CatalogError> { + if self.request == [0; 32] || self.base == [0; 32] { + return Err(CatalogError::Corrupt("group recovery lock")); + } + if let RecoveryPhase::Published { + publication, + manifest, + } = self.phase + { + if publication == [0; 32] || manifest == [0; 32] { + return Err(CatalogError::Corrupt("published recovery lock")); + } + } + let source = self.source_route(directory)?; + let managed = match &self.phase { + RecoveryPhase::ManagedSource { + transition, + plan, + groups, + } => Some((transition, plan, groups, 3..=65)), + RecoveryPhase::ManagedPublished { + transition, + plan, + groups, + replacement, + seeds, + publication, + } => { + if [*replacement, *seeds, *publication].contains(&[0; 32]) { + return Err(CatalogError::Corrupt("managed recovery publication")); + } + Some((transition, plan, groups, 4..=129)) + } + _ => None, + }; + if let Some((transition, plan, groups, limits)) = managed { + if *transition == [0; 16] + || *plan == [0; 32] + || !limits.contains(&groups.len()) + || groups + .iter() + .any(|(group, incarnation)| *group == [0; 16] || *incarnation == [0; 16]) + || groups.windows(2).any(|pair| pair[0] >= pair[1]) + || groups + .binary_search(&( + source.history.group.bytes(), + source.history.incarnation.bytes(), + )) + .is_err() + { + return Err(CatalogError::Corrupt("managed recovery lock")); + } + } + Ok(()) + } + + pub(in crate::raft::catalog::range_directory) fn source_route( + &self, + directory: &RangeDirectory, + ) -> Result { + self.source.decode(directory) + } +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Complete { + request: [u8; 32], + evidence: Vec, +} + +/// A complete selected archive and independently verified permanent fence. +/// Retain its dedicated remote hold until recovery and entered work finish. +/// This deliberately cannot be supplied to ordinary bootstrap or live apply. +/// Clones share the same admitted cut when preparing multiple destinations. +#[derive(Clone)] +pub struct FencedCellGroupRecovery { + request: Arc, + coverage: CellGroupArchiveCoverage, +} + +impl FencedCellGroupRecovery { + pub fn request(&self) -> &CellGroupRecoveryRequest { + &self.request + } + + pub fn coverage(&self) -> &CellGroupArchiveCoverage { + &self.coverage + } +} + +impl RangeDirectory { + fn recovery_key(&self, request: &CellGroupRecoveryRequest, suffix: &str) -> Path { + self.key(&format!( + "recovery/{}/{suffix}", + hex::encode(request.digest()) + )) + } + + async fn recovery_source( + &self, + request: &CellGroupRecoveryRequest, + ) -> Result { + let (view, _) = self.current().await?; + self.check_recovery_view(&view, request)?; + if Index::new(self) + .lookup( + view.head.operations, + &hex::encode(request.operation().bytes()), + ) + .await? + .is_some() + { + return Err(CatalogError::Conflict( + "group recovery operation already published", + )); + } + self.selected_route_in(&view, request.source().route()) + .await?; + self.verify_birth_reservation(request.source().route()) + .await?; + let bytes = self + .namespace + .inner + .read_versioned_bounded(&self.group_birth_key(request.source().route()), wire::LIMIT) + .await? + .ok_or(CatalogError::Corrupt("missing recovery source birth"))? + .0; + if bytes != request.source().encode() { + return Err(CatalogError::Corrupt("recovery source birth binding")); + } + Ok(CellGroupBirth { + request: Arc::new(request.source().clone()), + owner: self.namespace.inner.owner.clone(), + }) + } + + fn check_recovery_view( + &self, + view: &DirectoryView, + request: &CellGroupRecoveryRequest, + ) -> Result<(), CatalogError> { + match &view.head.recovery { + Some(lock) + if lock.phase == RecoveryPhase::Source + && lock.request == request.digest() + && Some(lock.base) == request.mutation().request.expected + && lock.source_route(self)? == *request.source().route() => + { + Ok(()) + } + Some(_) => Err(CatalogError::Conflict( + "different group recovery owns topology", + )), + None if Some(view.digest) == request.mutation().request.expected => Ok(()), + None => Err(CatalogError::Conflict("stale group recovery topology")), + } + } + + async fn lock_recovery(&self, request: &CellGroupRecoveryRequest) -> Result<(), CatalogError> { + self.bounded(async { + let (mut view, version) = self.current().await?; + if self.transition_active(&view.head).await?.is_some() { + return Err(CatalogError::Fenced); + } + self.check_recovery_view(&view, request)?; + if view.head.recovery.is_some() { + return Ok(()); + } + self.selected_route_in(&view, request.source().route()) + .await?; + view.head.recovery = Some(Box::new(RecoveryLock { + request: request.digest(), + base: view.digest, + source: wire::Route::encode(request.source().route(), self)?, + phase: RecoveryPhase::Source, + })); + let bytes = record::encode(wire::HEAD, &view.head, wire::LIMIT)?; + // A lost response can follow a committed CAS. Observe the exact + // lock; never issue another request or rebase onto a newer head. + let _ = self + .namespace + .inner + .store + .put_opts( + &self.key("head"), + bytes.into(), + PutOptions::from(PutMode::Update(version)), + ) + .await; + let (after, _) = self + .current() + .await + .map_err(|_| CatalogError::Unconfirmed)?; + self.check_recovery_view(&after, request)?; + if after.head.recovery.is_none() { + return Err(CatalogError::Unconfirmed); + } + Ok(()) + }) + .await + } + + /// Verify the exact held closure, serialize recovery with fleet import and + /// topology, then permanently fence every possible old writer. Persist + /// `request.encode()` before calling. A failed/unknown outcome is resumable + /// ONLY with those bytes; there is no abort or unfence operation. + /// + /// The authority must use external deployment/launch inventory, including + /// later joins, removed hosts, routers and effects; independently verify + /// evidence bound to the request digest and exclude every replacement node + /// from the old lineage. A marker or caller assertion cannot satisfy it. + /// This static unmanaged boundary does not publish a replacement or seed a + /// Raft engine. Fresh genesis, restored histories and quorum remain required. + pub async fn fence_group_recovery( + &self, + domain: &impl CellGroupArchiveExecution, + archives: &CellGroupArchivePool, + tails: &CellGroupArchiveTailPool, + request: &CellGroupRecoveryRequest, + receiver: CellGroupArchiveCheckpointReceiver, + authority: &dyn DisasterFenceAuthority, + ) -> Result { + domain + .run(async { + let request = self.decode_group_recovery(request.encode())?; + let birth = self.bounded(self.recovery_source(&request)).await?; + let registry = self.open_recovery_archive(domain, archives, &birth).await?; + // No inventory claim, topology lock or infrastructure action until + // every cell/base/control dependency has been physically verified. + let coverage = registry + .verify_hold(domain, request.hold(), tails, receiver) + .await?; + coverage + .recovery() + .check_disaster_target(request.replacement())?; + self.namespace.claim_disaster_namespace().await?; + self.reserve_recovery_birth(&request).await?; + self.bounded(async { + let observed = self + .namespace + .inner + .create_and_observe_bounded( + &self.recovery_key(&request, "intent"), + request.encode(), + wire::LIMIT, + ) + .await?; + if observed != request.encode() { + return Err(CatalogError::Conflict("group recovery intent")); + } + Ok(()) + }) + .await?; + self.lock_recovery(&request).await?; + let fence = CellGroupDisasterFenceRequest::new(request); + let complete_key = self.recovery_key(fence.recovery(), "fence"); + let existing = self + .bounded(async { + Ok(self + .namespace + .inner + .read_versioned_bounded(&complete_key, wire::LIMIT) + .await? + .map(|v| v.0)) + }) + .await?; + let bytes = if let Some(bytes) = existing { + bytes + } else { + let evidence = authority + .fence(&fence) + .await + .map_err(|e| CatalogError::Fence(e.to_string()))?; + check_evidence(&evidence)?; + authority + .verify(&fence, &evidence) + .await + .map_err(|e| CatalogError::Fence(e.to_string()))?; + let bytes = record::encode( + wire::GROUP_FENCE, + &Complete { + request: fence.digest(), + evidence, + }, + wire::LIMIT, + )?; + self.bounded(self.namespace.inner.create_and_observe_bounded( + &complete_key, + &bytes, + wire::LIMIT, + )) + .await? + }; + // Concurrent controllers may return different valid evidence. Verify + // the winning durable receipt independently, including on reopen. + let complete: Complete = record::decode(wire::GROUP_FENCE, &bytes, wire::LIMIT)?; + if complete.request != fence.digest() { + return Err(CatalogError::Corrupt("group fence request binding").into()); + } + check_evidence(&complete.evidence)?; + authority + .verify(&fence, &complete.evidence) + .await + .map_err(|e| CatalogError::Fence(e.to_string()))?; + self.bounded(self.recovery_source(fence.recovery())).await?; + Ok(FencedCellGroupRecovery { + request: Arc::new(fence.into_recovery()), + coverage, + }) + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } +} + +fn check_evidence(evidence: &[u8]) -> Result<(), CatalogError> { + if evidence.is_empty() || evidence.len() > 4096 { + return Err(CatalogError::Invalid("group fence evidence length")); + } + Ok(()) +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/recovery/publication.rs b/crates/cellar/durability/raft/cell_groups/directory/recovery/publication.rs new file mode 100644 index 000000000..e94a3b96e --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/recovery/publication.rs @@ -0,0 +1,248 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! The same complete genesis must be durably seeded everywhere before the +//! topology CAS. Keep its lock until replicated activation has been confirmed. +use super::*; +use crate::raft::cell_groups::GroupManifest; + +pub(in crate::raft::catalog::range_directory) struct RecoveryPublication { + request: Arc, + manifest: [u8; 32], +} + +impl RecoveryPublication { + pub(in crate::raft::catalog::range_directory) fn check( + &self, + directory: &RangeDirectory, + view: &DirectoryView, + mutation: &DirectoryMutation, + ) -> Result<(), CatalogError> { + if mutation.encode() != self.request.mutation().encode() { + return Err(CatalogError::Fenced); + } + directory.check_recovery_view(view, &self.request)?; + if view.head.recovery.is_none() { + return Err(CatalogError::Fenced); + } + Ok(()) + } + + pub(in crate::raft::catalog::range_directory) fn check_child( + &self, + route: &GroupRoute, + ) -> Result<(), CatalogError> { + if route != self.request.replacement().route() { + return Err(CatalogError::Fenced); + } + Ok(()) + } + + pub(in crate::raft::catalog::range_directory) fn published_lock( + &self, + publication: [u8; 32], + ) -> Result, CatalogError> { + Ok(Box::new(RecoveryLock { + request: self.request.digest(), + base: self + .request + .mutation() + .request + .expected + .ok_or(CatalogError::Invalid("recovery base"))?, + source: self + .request + .mutation() + .request + .parents + .first() + .ok_or(CatalogError::Invalid("recovery source"))? + .clone(), + phase: RecoveryPhase::Published { + publication, + manifest: self.manifest, + }, + })) + } +} + +impl RangeDirectory { + pub(super) async fn recovery_binding( + &self, + request: &CellGroupRecoveryRequest, + ) -> Result<(seed::Binding, [u8; 32]), CatalogError> { + let (_, origin) = self + .birth_reservation(request.replacement().route()) + .await?; + if origin != Some(request.digest()) { + return Err(CatalogError::Fenced); + } + let bytes = self + .namespace + .inner + .read_versioned_bounded( + &self.recovery_binding_key(request.replacement().route()), + wire::LIMIT, + ) + .await? + .ok_or(CatalogError::Conflict("recovery genesis not bound"))? + .0; + let binding: seed::Binding = + record::decode(wire::GROUP_RECOVERY_BINDING, &bytes, wire::LIMIT)?; + let genesis: &GroupManifest = &binding.genesis; + if binding.request != request.encode() + || !genesis.is_recovery_genesis() + || genesis.route() != request.replacement().route() + || genesis.recovery_request() != Some(request.digest()) + || genesis + .recovery_commitment() + .map_err(|_| CatalogError::Corrupt("recovery genesis"))? + != Some(binding.commitment) + { + return Err(CatalogError::Corrupt("recovery genesis binding")); + } + Ok((binding, record::digest(&bytes))) + } + + pub(super) async fn recovery_seed_receipt( + &self, + request: &CellGroupRecoveryRequest, + selected: &crate::raft::catalog::InitialMember, + binding: [u8; 32], + ) -> Result { + let bytes = self + .namespace + .inner + .read_versioned_bounded( + &self.recovery_seed_key(request, selected.member.get(), "complete"), + wire::LIMIT, + ) + .await? + .ok_or(CatalogError::Conflict("recovery member not seeded"))? + .0; + let receipt: seed::MemberSeed = + record::decode(wire::GROUP_RECOVERY_SEED_COMPLETE, &bytes, wire::LIMIT)?; + receipt.check(selected, binding)?; + let intent = record::encode(wire::GROUP_RECOVERY_SEED_INTENT, &receipt, wire::LIMIT)?; + let observed = self + .namespace + .inner + .read_versioned_bounded( + &self.recovery_seed_key(request, selected.member.get(), "intent"), + wire::LIMIT, + ) + .await? + .ok_or(CatalogError::Corrupt("missing recovery seed intent"))? + .0; + if observed != intent { + return Err(CatalogError::Corrupt("recovery seed intent binding")); + } + self.namespace + .verify_node_storage_identity(selected.node, receipt.disk()?, receipt.engine()?) + .await?; + Ok(receipt) + } + + /// Publish only after all initial members have synced the same complete + /// pending genesis. This metadata result never grants voting or serving. + /// Retry the original request bytes after an unknown CAS outcome. + pub fn publish_group_recovery<'a>( + &'a self, + request: &'a CellGroupRecoveryRequest, + authority: &'a dyn DisasterFenceAuthority, + ) -> futures_util::future::BoxFuture<'a, Result> { + Box::pin(async move { + let request = Arc::new(self.decode_group_recovery(request.encode())?); + self.bounded(async { + if let Some(publication) = self.publication(request.mutation()).await? { + return Ok(publication); + } + let (view, _) = self.current().await?; + self.check_recovery_view(&view, &request)?; + if view.head.recovery.is_none() { + return Err(CatalogError::Fenced); + } + self.recovery_source(&request).await?; + self.verify_recovery_fence(&request, authority).await?; + let (binding, digest) = self.recovery_binding(&request).await?; + let mut disks = BTreeSet::new(); + let mut engines = BTreeSet::new(); + for member in request + .replacement() + .voters() + .iter() + .chain(request.replacement().learners()) + { + let receipt = self.recovery_seed_receipt(&request, member, digest).await?; + if !disks.insert(receipt.disk()?) || !engines.insert(receipt.engine()?) { + return Err(CatalogError::Conflict("duplicate recovery storage")); + } + } + let permit = RecoveryPublication { + request: request.clone(), + manifest: binding.commitment, + }; + self.publish_inner(request.mutation(), PublicationAuthority::Recovery(&permit)) + .await + }) + .await + }) + } + + pub(super) async fn published_recovery( + &self, + request: &CellGroupRecoveryRequest, + ) -> Result<(DirectoryPublication, seed::Binding, [u8; 32]), CatalogError> { + let (view, _) = self.current().await?; + let publication = self + .confirmed(&view.head, request.mutation()) + .await? + .ok_or(CatalogError::Fenced)?; + self.selected_route_in(&view, request.replacement().route()) + .await?; + let (binding, digest) = self.recovery_binding(request).await?; + match view.head.recovery.as_deref() { + Some(lock) + if lock.request == request.digest() + && Some(lock.base) == request.mutation().request.expected + && lock.source_route(self)? == *request.source().route() + && lock.phase + == (RecoveryPhase::Published { + publication: publication.journal_digest(), + manifest: binding.commitment, + }) => {} + // An unrelated recovery can own the topology lock after this + // group completed. Its exact activation receipt and selected route + // must still verify; pending groups cannot use this lane. + Some(_) | None => { + self.require_recovery_activation(request, &publication, binding.commitment) + .await? + } + } + Ok((publication, binding, digest)) + } + + /// Read-only status for this exact published intent. False requires its + /// durable activation receipt; another group's lock is never completion. + pub async fn group_recovery_pending( + &self, + request: &CellGroupRecoveryRequest, + ) -> Result { + self.bounded(async { + let (publication, binding, _) = self.published_recovery(request).await?; + let (view, _) = self.current().await?; + self.selected_route_in(&view, request.replacement().route()) + .await?; + let pending = view + .head + .recovery + .as_deref() + .is_some_and(|lock| lock.request == request.digest()); + if !pending { + self.require_recovery_activation(request, &publication, binding.commitment) + .await?; + } + Ok(pending) + }) + .await + } +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/recovery/request.rs b/crates/cellar/durability/raft/cell_groups/directory/recovery/request.rs new file mode 100644 index 000000000..c7ced2b2c --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/recovery/request.rs @@ -0,0 +1,173 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Persistable operator intent. No mutable-head selection or authority is +//! reconstructed from a fence receipt or an untrusted archive locator. +use super::*; +use crate::raft::cell_groups::CellGroupArchiveHold; + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Request { + namespace: [u8; 32], + source: Vec, + replacement: Vec, + mutation: Vec, + hold: CellGroupArchiveHold, +} + +/// Persist before submission, including the dedicated remote hold. Never +/// release that hold while the intent or any recovery user remains live. +/// Replacement cell histories use the replacement group's fresh incarnation; +/// old witnesses remain provenance, not authority in the replacement history. +pub struct CellGroupRecoveryRequest { + bytes: Vec, + source: CellGroupBirthRequest, + replacement: CellGroupBirthRequest, + mutation: DirectoryMutation, + hold: CellGroupArchiveHold, +} + +impl CellGroupRecoveryRequest { + pub fn encode(&self) -> &[u8] { + &self.bytes + } + + pub fn digest(&self) -> [u8; 32] { + record::digest(&self.bytes) + } + + pub fn source(&self) -> &CellGroupBirthRequest { + &self.source + } + + pub fn replacement(&self) -> &CellGroupBirthRequest { + &self.replacement + } + + pub fn hold(&self) -> &CellGroupArchiveHold { + &self.hold + } + + pub fn operation(&self) -> ProposalId { + self.mutation.operation() + } + + pub(super) fn mutation(&self) -> &DirectoryMutation { + &self.mutation + } +} + +/// Whole-group binding for `DisasterFenceAuthority`. It is created only after +/// complete archive verification and a durable topology/namespace claim. +pub struct CellGroupDisasterFenceRequest { + recovery: CellGroupRecoveryRequest, +} + +impl CellGroupDisasterFenceRequest { + pub fn recovery(&self) -> &CellGroupRecoveryRequest { + &self.recovery + } + + pub fn digest(&self) -> [u8; 32] { + self.recovery.digest() + } + + pub(super) fn new(recovery: CellGroupRecoveryRequest) -> Self { + Self { recovery } + } + + pub(super) fn into_recovery(self) -> CellGroupRecoveryRequest { + self.recovery + } +} + +impl RangeDirectory { + /// Prepare one exact static-group rollback. This allocates no metadata and + /// grants no fence, startup or serving permission. The hold must cover the + /// selected birth; callers explicitly choose its recovery point. + pub fn prepare_group_recovery( + &self, + view: &DirectoryView, + operation: ProposalId, + source: &CellGroupBirth, + replacement: &CellGroupBirthRequest, + hold: CellGroupArchiveHold, + ) -> Result { + self.unmanaged_birth()?; + self.check_view(view)?; + if !Arc::ptr_eq(&source.owner, &self.namespace.inner.owner) { + return Err(CatalogError::Conflict("foreign recovery source birth")); + } + if view.head.recovery.is_some() { + return Err(CatalogError::Fenced); + } + let mutation = self.prepare_replacement( + view, + operation, + std::slice::from_ref(source.route()), + std::slice::from_ref(replacement.route()), + )?; + let bytes = record::encode( + wire::GROUP_RECOVERY, + &Request { + namespace: self.namespace.root_digest, + source: source.request.encode().to_vec(), + replacement: replacement.encode().to_vec(), + mutation: mutation.encode().to_vec(), + hold, + }, + wire::LIMIT, + )?; + self.decode_group_recovery(&bytes) + } + + /// Decode exactly the persisted intent under this namespace. A new catalog + /// owner may resume it; it cannot reinterpret or rebase its selected cut. + pub fn decode_group_recovery( + &self, + bytes: &[u8], + ) -> Result { + self.unmanaged_birth()?; + let value: Request = record::decode(wire::GROUP_RECOVERY, bytes, wire::LIMIT)?; + if value.namespace != self.namespace.root_digest { + return Err(CatalogError::Conflict("group recovery namespace")); + } + let source = self.decode_group_birth(&value.source)?; + let replacement = self.decode_group_birth(&value.replacement)?; + let mutation = self.decode_mutation(&value.mutation)?; + let old = source.route(); + let new = replacement.route(); + if mutation.request.expected.is_none() + || mutation.request.parents.len() != 1 + || mutation.request.children.len() != 1 + || mutation.request.parents[0].decode(self)? != *old + || mutation.request.children[0].decode(self)? != *new + || value.hold.marker().route() != old + || old.range != new.range + || old.history.group == new.history.group + || old.history.incarnation == new.history.incarnation + || source.creation() == replacement.creation() + { + return Err(CatalogError::Conflict("group recovery identities or cut")); + } + for member in replacement.voters().iter().chain(replacement.learners()) { + if source + .voters() + .iter() + .chain(source.learners()) + .any(|old| old.node == member.node || old.member == member.member) + { + return Err(CatalogError::Conflict( + "reused initial recovery member or node", + )); + } + } + Ok(CellGroupRecoveryRequest { + bytes: bytes.to_vec(), + source, + replacement, + mutation, + hold: value.hold, + }) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/recovery/seed.rs b/crates/cellar/durability/raft/cell_groups/directory/recovery/seed.rs new file mode 100644 index 000000000..5dbdabda8 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/recovery/seed.rs @@ -0,0 +1,428 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! A permanent recovery origin and stable seed receipts precede topology +//! publication. Losing any response never authorizes a new disk or empty root. +use super::*; +use crate::raft::{ + cell_groups::{ + CellGroupApplier, GroupManifest, PreparedCellGroupRecovery, RecoverySeed, + SeededCellGroupRecovery, + }, + storage::{CellGroupBootstrap, LogError, ReplicaLog}, +}; +use cellar_logic::consensus::{EngineInstanceId, ReplicaIdentity}; + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Binding { + pub(super) request: Vec, + pub(super) genesis: GroupManifest, + pub(super) commitment: [u8; 32], +} + +#[derive(Serialize, Deserialize, PartialEq, Eq)] +#[serde(deny_unknown_fields)] +pub(super) struct MemberSeed { + binding: [u8; 32], + member: u64, + node: [u8; 16], + disk: [u8; 16], + engine: [u8; 16], +} + +impl MemberSeed { + pub(super) fn disk(&self) -> Result { + cellar_logic::consensus::DiskLineage::new(self.disk) + .map_err(|_| CatalogError::Corrupt("recovery seed disk")) + } + + pub(super) fn engine(&self) -> Result { + EngineInstanceId::new(self.engine) + .map_err(|_| CatalogError::Corrupt("recovery seed engine")) + } + + pub(super) fn check( + &self, + selected: &crate::raft::catalog::InitialMember, + binding: [u8; 32], + ) -> Result<(), CatalogError> { + if self.binding != binding + || self.member != selected.member.get() + || self.node != selected.node.bytes() + { + return Err(CatalogError::Corrupt("recovery seed identity")); + } + self.disk()?; + self.engine()?; + Ok(()) + } + + pub(super) fn check_log(&self, log: &ReplicaLog) -> Result<(), CatalogError> { + if self.node != log.limits().node.bytes() + || self.disk()? != log.limits().disk + || self.engine()? != log.storage_instance().map_err(CatalogError::Log)? + { + return Err(CatalogError::LostStableState); + } + Ok(()) + } +} + +/// Process-local engine grant, minted only after the durable fence/origin and +/// exact seed intent are confirmed. It is not a startup or serving permission. +pub(crate) struct RecoverySeedPermit { + owner: Arc<()>, + local: ReplicaIdentity, + engine: EngineInstanceId, + request: [u8; 32], + managed: Option, + bootstrap: CellGroupBootstrap, + genesis: GroupManifest, + commitment: [u8; 32], +} + +pub(super) struct ManagedRecoverySeed { + pub(super) group: crate::raft::member_state::FleetGroupBinding, + pub(super) boot: crate::raft::fleet_stop::BootBinding, + pub(super) deadline: u64, +} + +impl RecoverySeedPermit { + pub(crate) fn owner(&self) -> &Arc<()> { + &self.owner + } + + pub(crate) fn request(&self) -> [u8; 32] { + self.request + } + + pub(crate) fn managed_binding( + &self, + guard: Option<&crate::raft::fleet_stop::Guard>, + now: u64, + ) -> Result, LogError> { + let Some(managed) = &self.managed else { + return Ok(None); + }; + if now >= managed.deadline + || guard.is_none_or(|guard| !guard.admits_group_start(&managed.boot)) + || !managed.group.matches_scope(&self.local.scope) + { + return Err(LogError::WrongIdentity); + } + Ok(Some(&managed.group)) + } + + pub(super) fn for_local( + target: &crate::raft::catalog::CellGroupBirthRequest, + genesis: &GroupManifest, + commitment: [u8; 32], + log: &ReplicaLog, + managed: Option, + ) -> Result { + let member = target + .voters() + .iter() + .chain(target.learners()) + .find(|member| member.node == log.limits().node) + .ok_or(CatalogError::NotInitialMember)?; + let request = genesis.recovery_request().ok_or(CatalogError::Fenced)?; + Ok(Self { + owner: log.owner(), + local: ReplicaIdentity { + scope: target.route().history, + member: member.member, + node: member.node, + disk: log.limits().disk, + process: log.limits().process, + }, + engine: log.storage_instance().map_err(CatalogError::Log)?, + request, + managed, + bootstrap: CellGroupBootstrap { + route: target.route().clone(), + member: member.member, + voters: target.voters().iter().map(|voter| voter.member).collect(), + learners: target + .learners() + .iter() + .map(|learner| learner.member) + .collect(), + }, + genesis: genesis.clone(), + commitment, + }) + } + + pub(crate) fn bootstrap(&self) -> &CellGroupBootstrap { + &self.bootstrap + } + + pub(crate) fn genesis(&self) -> &GroupManifest { + &self.genesis + } + + pub(crate) fn commitment(&self) -> [u8; 32] { + self.commitment + } + + pub(crate) fn check( + &self, + owner: &Arc<()>, + local: &ReplicaIdentity, + engine: EngineInstanceId, + ) -> Result<(), LogError> { + if !Arc::ptr_eq(owner, &self.owner) || local != &self.local || engine != self.engine { + return Err(LogError::WrongIdentity); + } + Ok(()) + } +} + +impl RangeDirectory { + pub(in crate::raft::catalog::range_directory::recovery) fn recovery_binding_key( + &self, + route: &GroupRoute, + ) -> Path { + self.key(&format!( + "recovery-bindings/{}/{}", + hex::encode(route.history.group.bytes()), + hex::encode(route.history.incarnation.bytes()) + )) + } + + pub(super) fn recovery_seed_key( + &self, + request: &CellGroupRecoveryRequest, + member: u64, + leaf: &str, + ) -> Path { + self.recovery_key(request, &format!("seeds/{member:016x}/{leaf}")) + } + + async fn require_locked_recovery( + &self, + request: &CellGroupRecoveryRequest, + ) -> Result<(), CatalogError> { + let (view, _) = self.current().await?; + self.check_recovery_view(&view, request)?; + if view.head.recovery.is_none() { + return Err(CatalogError::Fenced); + } + let (_, origin) = self + .birth_reservation(request.replacement().route()) + .await?; + if origin != Some(request.digest()) { + return Err(CatalogError::Corrupt("recovery birth origin")); + } + self.recovery_source(request).await?; + Ok(()) + } + + pub(super) async fn verify_recovery_fence( + &self, + request: &CellGroupRecoveryRequest, + authority: &dyn DisasterFenceAuthority, + ) -> Result<(), CatalogError> { + let request = self.decode_group_recovery(request.encode())?; + let fence = CellGroupDisasterFenceRequest::new(request); + let bytes = self + .namespace + .inner + .read_versioned_bounded(&self.recovery_key(fence.recovery(), "fence"), wire::LIMIT) + .await? + .ok_or(CatalogError::Corrupt("missing recovery fence"))? + .0; + let complete: Complete = record::decode(wire::GROUP_FENCE, &bytes, wire::LIMIT)?; + if complete.request != fence.digest() { + return Err(CatalogError::Corrupt("recovery fence identity")); + } + check_evidence(&complete.evidence)?; + authority + .verify(&fence, &complete.evidence) + .await + .map_err(|e| CatalogError::Fence(e.to_string())) + } + + /// Sync the imported complete root as a fresh Raft genesis and confirm its + /// permanent physical seed receipt. The exact preparation can be cloned for + /// retry after a lost reply. The replacement remains closed to startup. + pub async fn seed_group_recovery( + &self, + domain: &impl CellGroupArchiveExecution, + prepared: PreparedCellGroupRecovery, + log: &ReplicaLog, + authority: &dyn DisasterFenceAuthority, + ) -> Result { + domain + .run(async { + let request = Arc::new(self.decode_group_recovery(prepared.request().encode())?); + self.bounded(self.require_locked_recovery(&request)).await?; + self.bounded(self.verify_recovery_fence(&request, authority)) + .await?; + if !prepared.manifest().marker().is_recovery_genesis() + || prepared.manifest().marker().route() != request.replacement().route() + || prepared.manifest().marker().recovery_request() != Some(request.digest()) + || prepared + .manifest() + .marker() + .recovery_commitment() + .map_err(|_| CatalogError::Corrupt("seed commitment"))? + != Some(prepared.commitment()) + { + return Err(CatalogError::Conflict("recovery preparation identity").into()); + } + let target = request.replacement(); + let member = target + .voters() + .iter() + .chain(target.learners()) + .find(|member| member.node == log.limits().node) + .ok_or(CatalogError::NotInitialMember)?; + if prepared + .source() + .coverage() + .recovery() + .membership() + .has_disk_binding(log.limits().disk) + { + return Err(CatalogError::Conflict("reused archived recovery disk").into()); + } + let nodes = request + .source() + .voters() + .iter() + .chain(request.source().learners()) + .map(|member| member.node) + .collect::>(); + self.bounded(self.namespace.verify_recovery_storage_fresh(log, &nodes)) + .await?; + let _node = self.namespace.require_node_storage(log).await?; + let binding_bytes = record::encode( + wire::GROUP_RECOVERY_BINDING, + &Binding { + request: request.encode().to_vec(), + genesis: prepared.manifest().marker().clone(), + commitment: prepared.commitment(), + }, + wire::LIMIT, + )?; + let binding = record::digest(&binding_bytes); + self.bounded(async { + let observed = self + .namespace + .inner + .create_and_observe_bounded( + &self.recovery_binding_key(target.route()), + &binding_bytes, + wire::LIMIT, + ) + .await?; + if observed != binding_bytes { + return Err(CatalogError::Conflict("recovery genesis binding")); + } + Ok(()) + }) + .await?; + let member_seed = MemberSeed { + binding, + member: member.member.get(), + node: member.node.bytes(), + disk: log.limits().disk.bytes(), + engine: log.storage_instance().map_err(CatalogError::Log)?.bytes(), + }; + let intent = + record::encode(wire::GROUP_RECOVERY_SEED_INTENT, &member_seed, wire::LIMIT)?; + let complete = record::encode( + wire::GROUP_RECOVERY_SEED_COMPLETE, + &member_seed, + wire::LIMIT, + )?; + self.bounded(async { + let existing = self + .namespace + .inner + .read_versioned_bounded( + &self.recovery_seed_key(&request, member_seed.member, "complete"), + wire::LIMIT, + ) + .await?; + if let Some((bytes, _)) = existing { + if bytes != complete { + return Err(CatalogError::Conflict("recovery seed receipt")); + } + log.cell_group(&target.route().history) + .map_err(|error| match error { + LogError::NotFound => CatalogError::LostStableState, + error => CatalogError::Log(error), + })?; + } + let observed = self + .namespace + .inner + .create_and_observe_bounded( + &self.recovery_seed_key(&request, member_seed.member, "intent"), + &intent, + wire::LIMIT, + ) + .await?; + if observed != intent { + return Err(CatalogError::Conflict("recovery seed identity")); + } + Ok(()) + }) + .await?; + let permit = RecoverySeedPermit::for_local( + target, + prepared.manifest().marker(), + prepared.commitment(), + log, + None, + )?; + let seed = RecoverySeed::new(permit, prepared.data().clone()) + .map_err(CatalogError::Log)?; + let group = log + .install_recovery_seed(seed) + .await + .map_err(CatalogError::Log)?; + let store = prepared.store().clone(); + let genesis = prepared.manifest().marker().clone(); + let selected = group.clone(); + let retained = prepared.clone(); + let (applier, root) = domain + .blocking(move || { + let _retained = retained; + CellGroupApplier::recover_seeded(selected, &store, &genesis) + }) + .map_err(|_| CellGroupArchiveError::Stopped)? + .await + .map_err(CellGroupArchiveError::Worker)? + .map_err(CellGroupArchiveError::Replay)?; + self.bounded(self.require_locked_recovery(&request)).await?; + self.bounded(self.verify_recovery_fence(&request, authority)) + .await?; + self.bounded(async { + let observed = self + .namespace + .inner + .create_and_observe_bounded( + &self.recovery_seed_key(&request, member_seed.member, "complete"), + &complete, + wire::LIMIT, + ) + .await?; + if observed != complete { + return Err(CatalogError::Conflict("recovery seed receipt")); + } + self.require_locked_recovery(&request).await?; + Ok(()) + }) + .await?; + Ok(SeededCellGroupRecovery::new( + request, group, applier, root, binding, + )) + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/route_cache.rs b/crates/cellar/durability/raft/cell_groups/directory/route_cache.rs new file mode 100644 index 000000000..fda6ed870 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/route_cache.rs @@ -0,0 +1,149 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Bounded namespace range hints. Cached observations never grant authority. +use super::*; +use std::collections::VecDeque; + +// Two bounded routing-key strings plus the route, deque slot and bookkeeping. +// Reserve the full capacity up front; idle/evicted slots cannot hide memory. +const ENTRY_BYTES: usize = 8192; + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct CellRouteCacheUsage { + pub entries: usize, + pub capacity: usize, + pub reserved_bytes: usize, + pub revision: Option, +} + +struct State { + head: Option<(u64, [u8; 32])>, + routes: VecDeque, +} + +pub(crate) struct RouteCache { + directory: RangeDirectory, + capacity: usize, + state: Mutex, +} + +impl RouteCache { + pub(crate) fn new( + directory: RangeDirectory, + entries: usize, + bytes: usize, + ) -> Result { + if !(1..=4096).contains(&entries) || !(ENTRY_BYTES..=64 * 1024 * 1024).contains(&bytes) { + return Err(CatalogError::Limit); + } + let capacity = entries.min(bytes / ENTRY_BYTES); + Ok(Self { + directory, + capacity, + state: Mutex::new(State { + head: None, + routes: VecDeque::with_capacity(capacity), + }), + }) + } + + pub(crate) fn usage(&self) -> Result { + let state = self + .state + .lock() + .map_err(|_| CatalogError::Corrupt("routing cache"))?; + Ok(CellRouteCacheUsage { + entries: state.routes.len(), + capacity: self.capacity, + reserved_bytes: self.capacity * ENTRY_BYTES, + revision: state.head.map(|h| h.0), + }) + } + + pub(super) fn get( + &self, + cell: &CellHistory, + ) -> Result, CatalogError> { + let config = self.directory.namespace_config(); + if cell.cluster != config.cluster || cell.namespace != config.namespace { + return Err(CatalogError::Invalid("routing cache namespace")); + } + let key = cell_routing_key(cell.namespace, &cell.cell); + let mut state = self + .state + .lock() + .map_err(|_| CatalogError::Corrupt("routing cache"))?; + let Some((revision, _)) = state.head else { + return Ok(None); + }; + let Some(index) = state + .routes + .iter() + .position(|route| route.range.contains(&key)) + else { + return Ok(None); + }; + let route = state + .routes + .remove(index) + .ok_or(CatalogError::Corrupt("routing cache entry"))?; + let result = route.clone(); + state.routes.push_back(route); + Ok(Some((result, revision))) + } + + pub(crate) fn observe(&self, view: &DirectoryView) -> Result<(), CatalogError> { + self.directory.check_view(view)?; + let mut state = self + .state + .lock() + .map_err(|_| CatalogError::Corrupt("routing cache"))?; + Self::advance(&mut state, view) + } + + fn advance(state: &mut State, view: &DirectoryView) -> Result<(), CatalogError> { + // Planning, preparation and cancellation share the directory CAS but + // preserve its validated range journal. Only routing publication + // advances this identity and invalidates cached range hints. + let selected = view.head.journal; + if let Some((revision, digest)) = state.head { + if view.revision() < revision { + return Err(CatalogError::Conflict("superseded routing observation")); + } + if view.revision() == revision { + return if selected == digest { + Ok(()) + } else { + Err(CatalogError::Corrupt("routing revision changed digest")) + }; + } + } + state.routes.clear(); + state.head = Some((view.revision(), selected)); + Ok(()) + } + + // Only the routing session calls this after lookup in this exact pinned view. + // An older in-flight lookup cannot repopulate an observed successor root. + pub(super) fn remember( + &self, + view: &DirectoryView, + route: &GroupRoute, + ) -> Result<(), CatalogError> { + self.directory.check_view(view)?; + wire::Route::encode(route, &self.directory)?; + let mut state = self + .state + .lock() + .map_err(|_| CatalogError::Corrupt("routing cache"))?; + Self::advance(&mut state, view)?; + if let Some(index) = state.routes.iter().position(|old| old == route) { + state.routes.remove(index); + } + if state.routes.len() == self.capacity { + state.routes.pop_front(); + } + state.routes.push_back(route.clone()); + Ok(()) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/routing.rs b/crates/cellar/durability/raft/cell_groups/directory/routing.rs new file mode 100644 index 000000000..82361d00c --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/routing.rs @@ -0,0 +1,422 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Invocation-bounded catalog confirmation of untrusted relocation hints. +//! Attempts are routing observations, never admission or quorum capabilities. + +use super::*; +use crate::asyncrt::{DomainToken, ExecutionDomain}; +use cellar_logic::consensus::CommandId; +use std::fmt; + +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum CellRouteOperation { + Command(CommandId), + Receipt(CommandId), + Enrollment(ProposalId), + Query(ProposalId), +} + +/// The dispatcher must compute the fingerprint over its complete canonical +/// request (contract, input, durability or consistency, and retry metadata). +/// The session retains this exact binding through every attempt; destination +/// admission still validates the actual request and logical cell incarnation. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct CellRouteRequest { + pub cell: CellHistory, + pub operation: CellRouteOperation, + pub fingerprint: [u8; 32], +} + +#[derive(Clone, Copy, Debug)] +pub struct CellRoutingLimits { + pub timeout: Duration, + /// Includes the initial attempt, same-group retries and relocations. + pub max_attempts: u8, +} + +impl CellRoutingLimits { + pub(crate) fn deadline_at(self, started_at: u64) -> Option { + if self.timeout.as_millis() == 0 + || self.timeout > Duration::from_secs(60) + || !(1..=16).contains(&self.max_attempts) + { + return None; + } + started_at.checked_add(self.timeout.as_millis() as u64) + } +} + +#[derive(Debug)] +pub enum CellRoutingFailure { + InvalidRequest, + Closed, + Deadline, + AttemptsExhausted, + StaleAttempt, + InvalidHint, + Catalog(CatalogError), +} + +#[derive(Debug)] +pub struct CellRoutingError { + pub reason: CellRoutingFailure, + /// Once dispatch may have submitted a command, neither a redirect nor a + /// failed catalog read proves it uncommitted. Retry this same command ID. + pub ambiguous_command: Option, +} + +impl fmt::Display for CellRoutingError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "cell-group routing: {:?}", self.reason) + } +} + +impl std::error::Error for CellRoutingError {} + +/// A cached observation obtained from the configured trusted catalog. It can +/// become stale; a serving destination must check the full route binding. +#[derive(Clone)] +pub struct ResolvedCellRoute { + cell: CellHistory, + route: GroupRoute, + revision: u64, + owner: Arc<()>, +} + +impl ResolvedCellRoute { + pub fn route(&self) -> &GroupRoute { + &self.route + } +} + +#[derive(Clone)] +pub struct CellRouteAttempt { + resolved: ResolvedCellRoute, + request: Arc, + deadline_ms: u64, + ordinal: u8, + owner: Arc<()>, +} + +impl CellRouteAttempt { + pub fn route(&self) -> &GroupRoute { + self.resolved.route() + } + + pub fn cached_route(&self) -> ResolvedCellRoute { + self.resolved.clone() + } + + pub fn request(&self) -> &CellRouteRequest { + &self.request + } + + /// Absolute monotonic deadline in the session's captured execution domain. + pub fn deadline_ms(&self) -> u64 { + self.deadline_ms + } + + pub fn ordinal(&self) -> u8 { + self.ordinal + } + + pub(crate) fn limit_query_deadline(&mut self, deadline: u64) -> Result<(), CellRoutingError> { + if !matches!(self.request.operation, CellRouteOperation::Query(_)) + || deadline > self.deadline_ms + { + return Err(CellRoutingError { + reason: CellRoutingFailure::InvalidRequest, + ambiguous_command: None, + }); + } + self.deadline_ms = deadline; + Ok(()) + } +} + +/// One invocation, with no automatic command submission or producer ownership. +/// Dispatchers retain physical producers independently of these routing waiters. +/// This owner is not cloneable: retries cannot copy or replenish its budget. +pub struct CellRoutingSession { + directory: RangeDirectory, + domain: DomainToken, + request: Arc, + deadline: u64, + maximum: u8, + attempts: u8, + active: bool, + submitted: bool, + cached: Option, + route_cache: Option>, + owner: Arc<()>, +} + +impl RangeDirectory { + /// Capture the deadline before initial discovery. A matching cached route + /// permits the first attempt without any object-store access. + pub fn routing_session( + &self, + domain: &ExecutionDomain, + request: CellRouteRequest, + limits: CellRoutingLimits, + cached: Option, + ) -> Result { + let domain = domain.services().domain_token().ok_or(CellRoutingError { + reason: CellRoutingFailure::Closed, + ambiguous_command: None, + })?; + let started_at = domain.mono_ms().map_err(|_| CellRoutingError { + reason: CellRoutingFailure::Closed, + ambiguous_command: None, + })?; + self.routing_session_in(domain, request, limits, cached, started_at) + } + + pub(crate) fn routing_session_in( + &self, + domain: DomainToken, + request: CellRouteRequest, + limits: CellRoutingLimits, + cached: Option, + started_at: u64, + ) -> Result { + let error = |reason| CellRoutingError { + reason, + ambiguous_command: None, + }; + let config = self.namespace.config(); + if request.cell.cluster != config.cluster + || request.cell.namespace != config.namespace + || request.fingerprint == [0; 32] + || cached.as_ref().is_some_and(|cached| { + cached.cell != request.cell + || !Arc::ptr_eq(&cached.owner, &self.namespace.inner.owner) + }) + { + return Err(error(CellRoutingFailure::InvalidRequest)); + } + let now = domain + .mono_ms() + .map_err(|_| error(CellRoutingFailure::Closed))?; + if started_at > now { + return Err(error(CellRoutingFailure::InvalidRequest)); + } + let deadline = limits + .deadline_at(started_at) + .ok_or_else(|| error(CellRoutingFailure::InvalidRequest))?; + if now >= deadline { + return Err(error(CellRoutingFailure::Deadline)); + } + Ok(CellRoutingSession { + directory: self.clone(), + domain, + request: Arc::new(request), + deadline, + maximum: limits.max_attempts, + attempts: 0, + active: false, + submitted: false, + cached, + route_cache: None, + owner: Arc::new(()), + }) + } +} + +impl CellRoutingSession { + pub(crate) fn using_cache(mut self, cache: Arc) -> Self { + self.route_cache = Some(cache); + self + } + + pub(crate) async fn begin_current(&mut self) -> Result { + if self.attempts != 0 { + return Err(self.error(CellRoutingFailure::StaleAttempt)); + } + self.cached = None; + self.next(None, false).await + } + + pub(crate) fn maximum_attempts(&self) -> usize { + usize::from(self.maximum) + } + + pub(crate) fn deadline_ms(&self) -> u64 { + self.deadline + } + + pub(crate) fn validate_attempt( + &self, + attempt: &CellRouteAttempt, + ) -> Result<(), CellRoutingError> { + self.check_attempt(attempt)?; + self.live() + } + + fn error(&self, reason: CellRoutingFailure) -> CellRoutingError { + CellRoutingError { + reason, + ambiguous_command: match self.request.operation { + CellRouteOperation::Command(id) if self.submitted => Some(id), + _ => None, + }, + } + } + + fn live(&self) -> Result<(), CellRoutingError> { + let now = self + .domain + .mono_ms() + .map_err(|_| self.error(CellRoutingFailure::Closed))?; + if now >= self.deadline { + return Err(self.error(CellRoutingFailure::Deadline)); + } + Ok(()) + } + + fn check_attempt(&self, attempt: &CellRouteAttempt) -> Result<(), CellRoutingError> { + if !self.active + || !Arc::ptr_eq(&self.owner, &attempt.owner) + || attempt.ordinal != self.attempts + { + return Err(self.error(CellRoutingFailure::StaleAttempt)); + } + Ok(()) + } + + /// Mark before handing a mutation to a producer that may submit it. Only + /// explicit request reconciliation can later establish its actual outcome. + pub fn mark_submitted(&mut self, attempt: &CellRouteAttempt) -> Result<(), CellRoutingError> { + self.check_attempt(attempt)?; + self.live()?; + self.submitted = true; + Ok(()) + } + + pub async fn begin(&mut self) -> Result { + if self.attempts != 0 { + return Err(self.error(CellRoutingFailure::StaleAttempt)); + } + self.next(None, true).await + } + + /// Same-group retry, including a different member, under the same budget. + pub async fn retry( + &mut self, + previous: &CellRouteAttempt, + ) -> Result { + self.check_attempt(previous)?; + self.next(None, true).await + } + + /// A relocation hint is untrusted until an exact lookup in the configured + /// catalog confirms its namespace, range, epoch, schema, birth and history. + /// No arbitrary URL or caller-supplied catalog is followed. + pub async fn redirect( + &mut self, + previous: &CellRouteAttempt, + hint: &GroupRoute, + ) -> Result { + self.check_attempt(previous)?; + self.next(Some(hint), true).await + } + + async fn next( + &mut self, + hint: Option<&GroupRoute>, + use_cache: bool, + ) -> Result { + // Consume the previous ticket and attempt before any await. Dropping a + // discovery future cannot refund attempts or authorize a late response. + self.active = false; + self.live()?; + if self.attempts >= self.maximum { + return Err(self.error(CellRoutingFailure::AttemptsExhausted)); + } + self.attempts += 1; + if let Some(hint) = hint { + let previous = self + .cached + .as_ref() + .ok_or_else(|| self.error(CellRoutingFailure::InvalidHint))?; + if wire::Route::encode(hint, &self.directory).is_err() + || hint.epoch <= previous.route.epoch + || !hint.range.contains(&cell_routing_key( + hint.history.namespace, + &self.request.cell.cell, + )) + { + return Err(self.error(CellRoutingFailure::InvalidHint)); + } + } + if use_cache && hint.is_none() && self.cached.is_none() { + if let Some(cache) = &self.route_cache { + if let Some((route, revision)) = cache + .get(&self.request.cell) + .map_err(|e| self.error(CellRoutingFailure::Catalog(e)))? + { + self.cached = Some(ResolvedCellRoute { + cell: self.request.cell.clone(), + route, + revision, + owner: self.directory.namespace.inner.owner.clone(), + }); + } + } + } + if self.cached.is_none() || hint.is_some() { + let minimum = self.cached.as_ref().map_or(0, |route| route.revision); + let work = async { + let view = self.directory.observe().await?; + let route = self + .directory + .lookup(&view, &self.request.cell.cell) + .await?; + if let Some(recovery) = &view.head.recovery { + // A published recovery range can vote before activation. + // Never cache it as an application route during that window. + if recovery.blocks_application_route(&self.directory, &route)? { + return Err(CatalogError::Fenced); + } + } + Ok::<_, CatalogError>((view, route)) + }; + let result = self + .domain + .run(crate::asyncrt::timeout_at(self.deadline, work)) + .await + .map_err(|_| self.error(CellRoutingFailure::Closed))?; + self.live()?; + let (view, route) = result + .map_err(|_| self.error(CellRoutingFailure::Deadline))? + .map_err(|error| self.error(CellRoutingFailure::Catalog(error)))?; + if view.revision() < minimum || hint.is_some_and(|hint| &route != hint) { + return Err(self.error(CellRoutingFailure::InvalidHint)); + } + if let Some(cache) = &self.route_cache { + cache + .remember(&view, &route) + .map_err(|e| self.error(CellRoutingFailure::Catalog(e)))?; + } + self.cached = Some(ResolvedCellRoute { + cell: self.request.cell.clone(), + route, + revision: view.revision(), + owner: self.directory.namespace.inner.owner.clone(), + }); + } + self.live()?; + let resolved = self + .cached + .clone() + .ok_or_else(|| self.error(CellRoutingFailure::InvalidHint))?; + self.active = true; + Ok(CellRouteAttempt { + resolved, + request: self.request.clone(), + deadline_ms: self.deadline, + ordinal: self.attempts, + owner: self.owner.clone(), + }) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/store.rs b/crates/cellar/durability/raft/cell_groups/directory/store.rs new file mode 100644 index 000000000..c0140f6a2 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/store.rs @@ -0,0 +1,128 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use crate::raft::coordination_index::PageStore; +use bytes::Bytes; +use futures_util::future::BoxFuture; + +#[derive(Default)] +pub(super) struct Cache { + pages: std::collections::BTreeMap<[u8; 32], Bytes>, + order: std::collections::VecDeque<[u8; 32]>, + bytes: usize, +} + +impl RangeDirectory { + pub(super) fn key(&self, suffix: &str) -> Path { + Path::from(format!( + "{}cell-groups/{suffix}", + self.namespace.namespace_key() + )) + } + + pub(super) async fn read(&self, suffix: &str) -> Result>, CatalogError> { + Ok(self + .namespace + .inner + .read_versioned_bounded(&self.key(suffix), wire::LIMIT) + .await? + .map(|r| r.0)) + } + + pub(super) async fn immutable( + &self, + category: &str, + id: [u8; 32], + bytes: &[u8], + ) -> Result<(), CatalogError> { + if record::digest(bytes) != id { + return Err(CatalogError::Invalid("immutable range object digest")); + } + let key = self.key(&format!("{category}/{}", hex::encode(id))); + let observed = self + .namespace + .inner + .create_and_observe_bounded(&key, bytes, wire::LIMIT) + .await?; + if observed != bytes { + return Err(CatalogError::Corrupt("immutable range object collision")); + } + Ok(()) + } + + pub(super) async fn range_bytes(&self, id: [u8; 32]) -> Result { + if let Some(bytes) = self + .cache + .lock() + .map_err(|_| CatalogError::Corrupt("range cache"))? + .pages + .get(&id) + .cloned() + { + return Ok(bytes); + } + let bytes = Bytes::from( + self.read(&format!("ranges/{}", hex::encode(id))) + .await? + .ok_or(CatalogError::Corrupt("missing range page"))?, + ); + if record::digest(&bytes) != id { + return Err(CatalogError::Corrupt("range page digest")); + } + let mut cache = self + .cache + .lock() + .map_err(|_| CatalogError::Corrupt("range cache"))?; + if self.limits.max_cached_pages > 0 + && bytes.len() <= self.limits.max_cache_bytes + && !cache.pages.contains_key(&id) + { + while cache.pages.len() >= self.limits.max_cached_pages + || cache.bytes + bytes.len() > self.limits.max_cache_bytes + { + let oldest = cache + .order + .pop_front() + .ok_or(CatalogError::Corrupt("range cache accounting"))?; + if let Some(removed) = cache.pages.remove(&oldest) { + cache.bytes -= removed.len(); + } + } + cache.bytes += bytes.len(); + cache.order.push_back(id); + cache.pages.insert(id, bytes.clone()); + } + Ok(bytes) + } + + /// Cached immutable bytes only. Returned routes remain hints and every + /// destination must verify the current route and its own quorum authority. + pub fn cache_usage(&self) -> Result<(usize, usize), CatalogError> { + let cache = self + .cache + .lock() + .map_err(|_| CatalogError::Corrupt("range cache"))?; + Ok((cache.pages.len(), cache.bytes)) + } +} + +impl PageStore for RangeDirectory { + type Error = CatalogError; + + fn page_limit(&self) -> usize { + wire::LIMIT + } + + fn read_page(&self, id: [u8; 32]) -> BoxFuture<'_, Result> { + Box::pin(async move { + self.read(&format!("index/{}", hex::encode(id))) + .await? + .map(Bytes::from) + .ok_or(CatalogError::Corrupt("missing range index page")) + }) + } + + fn create_page(&self, id: [u8; 32], bytes: Vec) -> BoxFuture<'_, Result<(), CatalogError>> { + Box::pin(async move { self.immutable("index", id, &bytes).await }) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/transition.rs b/crates/cellar/durability/raft/cell_groups/directory/transition.rs new file mode 100644 index 000000000..b1cfb8f0f --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/transition.rs @@ -0,0 +1,1818 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Split/merge decisions rooted in the same CAS as the range tree. +use super::{transition_sources::TransitionSourceCut, *}; +use crate::application::consensus::group_protocol::{ + CellGroupEntry, GroupCommand, GroupEntryBinding, GroupEntryLimits, +}; +use crate::raft::{ + cache::GroupLog, + cell_groups::{CellGroupArchiveHold, GroupManifest}, + completion::RuntimeError, + storage::ReplicaLog, +}; +use crate::raft::{ + fleet_catalog::{ + FleetGroupAuthorization, FleetHostAuthorization, ManagedRecoveryIntentBasis, + ManagedTransitionPermit, + }, + fleet_stop::BootBinding, +}; +use cellar_logic::consensus::EngineInstanceId; +use serde::{Deserialize, Serialize}; +use sha2::{Digest, Sha256}; + +// Normal operation keys are 32 lowercase hexadecimal digits. These keys share +// their immutable index and head CAS without changing its wire shape. +const ACTIVE: &str = "transition/active"; +const PLANS: &str = "transition-plans"; +const COUNT: &str = "transition/count"; +const JOURNAL: u8 = 198; + +// A retired journal can carry nine 32-byte digests. Each JSON byte array +// takes up to 129 bytes, before field names, phase and the record envelope. +const JOURNAL_LIMIT: usize = 2048; + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum TransitionPhase { + Planned, + Prepared, + SealAuthorized, + Sealed, + Installed, + Archived, + Published, + Active, + Retired, + Cancelled, +} + +/// A historical topology decision. A source still needs authenticated +/// admission and a committed seal before its cut can be used for transfer. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct TransitionJournal { + request: [u8; 32], + preparation: Option<[u8; 32]>, + #[serde(default, skip_serializing_if = "Option::is_none")] + preparation_sources: Option<[u8; 32]>, + #[serde(default, skip_serializing_if = "Option::is_none")] + preparation_acquires: Option<[u8; 32]>, + #[serde(default, skip_serializing_if = "Option::is_none")] + source_set: Option<[u8; 32]>, + #[serde(default, skip_serializing_if = "Option::is_none")] + installed_set: Option<[u8; 32]>, + #[serde(default, skip_serializing_if = "Option::is_none")] + archive: Option<[u8; 32]>, + #[serde(default, skip_serializing_if = "Option::is_none")] + publication: Option<[u8; 32]>, + #[serde(default, skip_serializing_if = "Option::is_none")] + activation_set: Option<[u8; 32]>, + #[serde(default, skip_serializing_if = "Option::is_none")] + child_archives: Option<[u8; 32]>, + #[serde(default, skip_serializing_if = "Option::is_none")] + retirement_set: Option<[u8; 32]>, + #[serde(default, skip_serializing_if = "std::ops::Not::not")] + parent_holds_released: bool, + #[serde(default, skip_serializing_if = "Option::is_none")] + recovery_settlement: Option<[u8; 32]>, + phase: TransitionPhase, +} + +pub(crate) struct ManagedTransitionRecoverySelection { + pub(crate) plan: DirectoryMutation, + pub(crate) directory_digest: [u8; 32], + pub(crate) journal: TransitionJournal, + pub(crate) publication: Option<[u8; 32]>, + pub(crate) parents: Vec, + pub(crate) children: Vec, +} + +impl ManagedTransitionRecoverySelection { + pub(crate) fn archive_routes(&self) -> &[GroupRoute] { + if self.publication.is_some() { + &self.children + } else { + &self.parents + } + } +} + +impl TransitionJournal { + pub fn phase(&self) -> TransitionPhase { + self.phase + } + + pub fn preparation(&self) -> Option<[u8; 32]> { + self.preparation + } + + pub(super) fn preparation_acquires(&self) -> Option<[u8; 32]> { + self.preparation_acquires + } + + pub(super) fn preparation_sources(&self) -> Option<[u8; 32]> { + self.preparation_sources + } + + pub fn sealed_sources(&self) -> Option<[u8; 32]> { + self.source_set + } + + pub fn installed_children(&self) -> Option<[u8; 32]> { + self.installed_set + } + + pub fn transfer_archive(&self) -> Option<[u8; 32]> { + self.archive + } + + pub fn activated_children(&self) -> Option<[u8; 32]> { + self.activation_set + } + + pub fn child_archives(&self) -> Option<[u8; 32]> { + self.child_archives + } + + pub fn retired_parents(&self) -> Option<[u8; 32]> { + self.retirement_set + } + + pub fn parent_holds_released(&self) -> bool { + self.parent_holds_released + } + + pub fn recovery_settlement(&self) -> Option<[u8; 32]> { + self.recovery_settlement + } + + pub(crate) fn owns_topology(&self) -> bool { + self.phase != TransitionPhase::Cancelled + && !self.parent_holds_released + && self.recovery_settlement.is_none() + } + + pub(super) fn seal_authorization(&self) -> Result<[u8; 32], CatalogError> { + if !matches!( + self.phase, + TransitionPhase::SealAuthorized + | TransitionPhase::Sealed + | TransitionPhase::Installed + | TransitionPhase::Archived + | TransitionPhase::Published + | TransitionPhase::Active + | TransitionPhase::Retired + ) { + return Err(CatalogError::Fenced); + } + let authorized = Self { + phase: TransitionPhase::SealAuthorized, + source_set: None, + installed_set: None, + archive: None, + publication: None, + activation_set: None, + child_archives: None, + retirement_set: None, + parent_holds_released: false, + recovery_settlement: None, + ..self.clone() + }; + Ok(record::digest(&record::encode( + JOURNAL, + &authorized, + JOURNAL_LIMIT, + )?)) + } +} + +/// Short-lived local capability minted only from the selected irreversible journal. +/// It binds the encoded seal to one physically open source replica. +pub struct CellGroupSealPermit { + group: GroupLog, + engine: EngineInstanceId, + deadline: u64, + transition: TransitionId, + authorization: [u8; 32], + payload: [u8; 32], + entry: Vec, + managed: Option, +} + +struct ManagedSealOwner { + boot: BootBinding, + membership_revision: u64, +} + +struct ManagedSealAdmission<'a> { + host: &'a FleetHostAuthorization, + group: &'a FleetGroupAuthorization, + deadline: u64, +} + +impl CellGroupSealPermit { + pub(crate) fn history(&self) -> GroupHistory { + self.group.identity().scope + } + + pub(crate) fn check( + &self, + log: &ReplicaLog, + group: &GroupLog, + ) -> Result<(), RuntimeError> { + if !Arc::ptr_eq(&group.owner, &self.group.owner) + || group.identity() != self.group.identity() + || log.storage_instance()? != self.engine + || log.domain().mono_ms().map_err(|_| RuntimeError::Closed)? >= self.deadline + { + return Err(RuntimeError::StaleCompletion); + } + if let Some(managed) = &self.managed { + if log + .lifecycle_guard()? + .is_none_or(|guard| !guard.admits_group_start(&managed.boot)) + { + return Err(RuntimeError::StaleCompletion); + } + let cache = group.read()?; + if cache.group_manifest.as_ref().is_none_or(|manifest| { + manifest.membership_revision() != managed.membership_revision + }) || !cache + .conf_state + .voters + .contains(&group.identity().member.get()) + { + return Err(RuntimeError::StaleCompletion); + } + } else { + drop(group.read()?); + } + Ok(()) + } + + pub(crate) fn matches(&self, seal: &crate::raft::cell_groups::GroupSourceSeal) -> bool { + seal.matches(self.transition, self.authorization, self.payload) + } + + pub(crate) fn payload(&self) -> [u8; 32] { + self.payload + } + + pub(crate) fn entry_bytes(&self) -> usize { + self.entry.len() + } + + pub(crate) fn into_entry(self) -> Vec { + self.entry + } +} + +#[derive(Clone, Copy)] +enum Advance { + ChoosePreparationAcquires { + previous: Option<[u8; 32]>, + next: [u8; 32], + }, + StagePreparationSources { + sources: [u8; 32], + acquires: Option<[u8; 32]>, + }, + Prepare([u8; 32], Option<[u8; 32]>), + Authorize, + RecordSources([u8; 32]), + RecordInstalled([u8; 32]), + RecordArchive { + sources: [u8; 32], + installed: [u8; 32], + digest: [u8; 32], + }, + RecordActivations { + publication: [u8; 32], + installed: [u8; 32], + digest: [u8; 32], + }, + RecordChildArchives { + publication: [u8; 32], + activations: [u8; 32], + digest: [u8; 32], + }, + RecordRetired { + publication: [u8; 32], + child_archives: [u8; 32], + digest: [u8; 32], + }, + ReleaseParentHolds([u8; 32]), + Cancel, +} + +impl Advance { + fn apply(self, mut journal: TransitionJournal) -> Result { + if journal.recovery_settlement.is_some() { + return Err(CatalogError::Fenced); + } + use TransitionPhase::*; + match (self, journal.phase) { + (Self::ChoosePreparationAcquires { previous, next }, Planned) + if next != [0; 32] + && journal.preparation_sources.is_none() + && journal.preparation_acquires == previous => + { + journal.preparation_acquires = Some(next); + } + (Self::ChoosePreparationAcquires { next, .. }, Planned | Prepared) + if journal.preparation_acquires == Some(next) => {} + (Self::StagePreparationSources { sources, acquires }, Planned) + if sources != [0; 32] + && journal.preparation_sources.is_none() + && journal.preparation_acquires == acquires => + { + journal.preparation_sources = Some(sources); + } + (Self::StagePreparationSources { sources, acquires }, Planned | Prepared) + if journal.preparation_sources == Some(sources) + && journal.preparation_acquires == acquires => {} + (Self::Prepare(digest, sources), Planned) + if digest != [0; 32] + && (journal.preparation_acquires.is_none() + || journal.preparation_sources.is_some()) + && journal + .preparation_sources + .is_none_or(|selected| Some(selected) == sources) => + { + journal.preparation = Some(digest); + journal.phase = Prepared; + } + ( + Self::Prepare(digest, _), + Prepared | SealAuthorized | Sealed | Installed | Archived, + ) if journal.preparation == Some(digest) => {} + (Self::Authorize, Prepared) => journal.phase = SealAuthorized, + (Self::RecordSources(digest), SealAuthorized) if digest != [0; 32] => { + journal.phase = Sealed; + journal.source_set = Some(digest); + } + (Self::RecordSources(digest), Sealed | Installed | Archived) + if journal.source_set == Some(digest) => {} + (Self::RecordInstalled(digest), Sealed) if digest != [0; 32] => { + journal.installed_set = Some(digest); + journal.phase = Installed; + } + (Self::RecordInstalled(digest), Installed | Archived) + if journal.installed_set == Some(digest) => {} + ( + Self::RecordArchive { + sources, + installed, + digest, + }, + Installed | Archived, + ) if journal.source_set == Some(sources) + && journal.installed_set == Some(installed) + && digest != [0; 32] + && journal.archive.is_none_or(|selected| selected == digest) => + { + journal.archive = Some(digest); + journal.phase = Archived; + } + ( + Self::RecordActivations { + publication, + installed, + digest, + }, + Published | Active, + ) if journal.publication == Some(publication) + && journal.installed_set == Some(installed) + && digest != [0; 32] + && journal + .activation_set + .is_none_or(|selected| selected == digest) => + { + journal.activation_set = Some(digest); + journal.phase = Active; + } + ( + Self::RecordChildArchives { + publication, + activations, + digest, + }, + Active, + ) if journal.publication == Some(publication) + && journal.activation_set == Some(activations) + && digest != [0; 32] + && journal + .child_archives + .is_none_or(|selected| selected == digest) => + { + journal.child_archives = Some(digest); + } + ( + Self::RecordRetired { + publication, + child_archives, + digest, + }, + Active | Retired, + ) if journal.publication == Some(publication) + && journal.child_archives == Some(child_archives) + && digest != [0; 32] + && journal + .retirement_set + .is_none_or(|selected| selected == digest) => + { + journal.retirement_set = Some(digest); + journal.phase = Retired; + } + (Self::ReleaseParentHolds(digest), Retired) + if journal.retirement_set == Some(digest) => + { + journal.parent_holds_released = true; + } + (Self::Cancel, Planned | Prepared) => journal.phase = Cancelled, + ( + Self::Authorize, + SealAuthorized | Sealed | Installed | Archived | Published | Active | Retired, + ) + | (Self::Cancel, Cancelled) => {} + _ => return Err(CatalogError::Conflict("transition decision")), + } + Ok(journal) + } +} + +fn key(operation: ProposalId) -> String { + format!("transition/{}", hex::encode(operation.bytes())) +} + +fn child_key(history: GroupHistory) -> String { + format!( + "transition/child/{}/{}", + hex::encode(history.group.bytes()), + hex::encode(history.incarnation.bytes()), + ) +} + +impl RangeDirectory { + pub(super) async fn transition_active( + &self, + head: &wire::Head, + ) -> Result, CatalogError> { + Index::new(self) + .lookup(head.operations, ACTIVE) + .await? + .map(|bytes| { + let id = bytes + .try_into() + .map_err(|_| CatalogError::Corrupt("transition owner"))?; + ProposalId::new(id).map_err(|_| CatalogError::Corrupt("transition owner")) + }) + .transpose() + } + + pub(super) async fn transition_record( + &self, + head: &wire::Head, + operation: ProposalId, + ) -> Result, CatalogError> { + Index::new(self) + .lookup(head.operations, &key(operation)) + .await? + .map(|bytes| { + let journal: TransitionJournal = record::decode(JOURNAL, &bytes, JOURNAL_LIMIT)?; + let required = matches!( + journal.phase, + TransitionPhase::Prepared + | TransitionPhase::SealAuthorized + | TransitionPhase::Sealed + | TransitionPhase::Installed + | TransitionPhase::Archived + | TransitionPhase::Published + | TransitionPhase::Active + | TransitionPhase::Retired + ); + if journal.request == [0; 32] + || journal.preparation == Some([0; 32]) + || journal.source_set == Some([0; 32]) + || journal.installed_set == Some([0; 32]) + || journal.archive == Some([0; 32]) + || journal.publication == Some([0; 32]) + || journal.activation_set == Some([0; 32]) + || journal.child_archives == Some([0; 32]) + || journal.retirement_set == Some([0; 32]) + || (journal.child_archives.is_some() + && !matches!( + journal.phase, + TransitionPhase::Active | TransitionPhase::Retired + )) + || (journal.phase == TransitionPhase::Retired) + != journal.retirement_set.is_some() + || (journal.phase == TransitionPhase::Retired + && journal.child_archives.is_none()) + || (journal.parent_holds_released && journal.phase != TransitionPhase::Retired) + || journal.recovery_settlement == Some([0; 32]) + || (journal.recovery_settlement.is_some() + && (journal.parent_holds_released + || journal.phase == TransitionPhase::Cancelled)) + || matches!( + journal.phase, + TransitionPhase::Sealed + | TransitionPhase::Installed + | TransitionPhase::Archived + | TransitionPhase::Published + | TransitionPhase::Active + | TransitionPhase::Retired + ) != journal.source_set.is_some() + || matches!( + journal.phase, + TransitionPhase::Installed + | TransitionPhase::Archived + | TransitionPhase::Published + | TransitionPhase::Active + | TransitionPhase::Retired + ) != journal.installed_set.is_some() + || matches!( + journal.phase, + TransitionPhase::Archived + | TransitionPhase::Published + | TransitionPhase::Active + | TransitionPhase::Retired + ) != journal.archive.is_some() + || matches!( + journal.phase, + TransitionPhase::Published + | TransitionPhase::Active + | TransitionPhase::Retired + ) != journal.publication.is_some() + || matches!( + journal.phase, + TransitionPhase::Active | TransitionPhase::Retired + ) != journal.activation_set.is_some() + || (required && journal.preparation.is_none()) + || (journal.phase == TransitionPhase::Planned && journal.preparation.is_some()) + { + return Err(CatalogError::Corrupt("transition journal")); + } + Ok(journal) + }) + .transpose() + } + + pub(super) fn check_transition_plan( + &self, + mutation: &DirectoryMutation, + ) -> Result<(), CatalogError> { + self.check_request(&mutation.request)?; + let parents = &mutation.request.parents; + let children = &mutation.request.children; + if !matches!((parents.len(), children.len()), (1, 2..=64) | (2..=64, 1)) { + return Err(CatalogError::Invalid("transition split or merge shape")); + } + if children + .iter() + .any(|child| parents.iter().any(|parent| parent.group == child.group)) + { + return Err(CatalogError::Conflict("transfer reuses a parent group")); + } + Ok(()) + } + + pub(crate) fn transition_routes( + &self, + mutation: &DirectoryMutation, + ) -> Result<(Vec, Vec), CatalogError> { + self.check_transition_plan(mutation)?; + let decode = |routes: &[wire::Route]| { + routes + .iter() + .map(|route| route.decode(self)) + .collect::>() + }; + Ok(( + decode(&mutation.request.parents)?, + decode(&mutation.request.children)?, + )) + } + + pub(super) async fn matching_transition( + &self, + head: &wire::Head, + mutation: &DirectoryMutation, + ) -> Result, CatalogError> { + if head + .recovery + .as_ref() + .is_some_and(|lock| lock.owns_transition(mutation.operation)) + { + return Err(CatalogError::Fenced); + } + let journal = self.matching_transition_journal(head, mutation).await?; + if journal + .as_ref() + .is_some_and(|journal| journal.recovery_settlement().is_some()) + { + return Err(CatalogError::Fenced); + } + Ok(journal) + } + + async fn matching_transition_journal( + &self, + head: &wire::Head, + mutation: &DirectoryMutation, + ) -> Result, CatalogError> { + let found = self.transition_record(head, mutation.operation).await?; + if let Some(journal) = &found { + if journal.request != mutation.digest { + return Err(CatalogError::Conflict("transition operation reused")); + } + let active = self.transition_active(head).await? == Some(mutation.operation); + if active != journal.owns_topology() { + return Err(CatalogError::Corrupt( + "transition owner and journal disagree", + )); + } + } + Ok(found) + } + + pub(super) async fn check_transfer_publication( + &self, + view: &DirectoryView, + mutation: &DirectoryMutation, + archive: [u8; 32], + ) -> Result { + if view.head.recovery.is_some() { + return Err(CatalogError::Fenced); + } + self.matching_transition(&view.head, mutation) + .await? + .filter(|journal| { + journal.phase == TransitionPhase::Archived && journal.archive == Some(archive) + }) + .ok_or(CatalogError::Fenced) + } + + pub(super) async fn root_transfer_publication( + &self, + operations: Option<[u8; 32]>, + mutation: &DirectoryMutation, + mut journal: TransitionJournal, + publication: [u8; 32], + ) -> Result, CatalogError> { + journal.publication = Some(publication); + journal.phase = TransitionPhase::Published; + // Root the birth fence with route selection. Startup must still refuse + // ordinary genesis if a detached install receipt goes missing. + let index = Index::new(self); + let mut operations = operations; + for child in &mutation.request.children { + operations = index + .update( + operations, + &child_key(child.decode(self)?.history), + Some(publication.to_vec()), + 0, + ) + .await?; + } + // Keep the operation active until activation and retirement settle. + index + .update( + operations, + &key(mutation.operation), + Some(record::encode(JOURNAL, &journal, JOURNAL_LIMIT)?), + 0, + ) + .await + } + + pub(super) async fn require_non_transfer_birth( + &self, + route: &GroupRoute, + ) -> Result<(), CatalogError> { + let (view, _) = self.current().await?; + if Index::new(self) + .lookup(view.head.operations, &child_key(route.history)) + .await? + .is_some() + { + return Err(CatalogError::Fenced); + } + Ok(()) + } + + pub(super) async fn require_selected_retired_transfer_child( + &self, + marker: &GroupManifest, + ) -> Result<(), CatalogError> { + let origin = marker.transfer_origin().ok_or(CatalogError::Fenced)?; + let (publication, _, _) = marker.transfer_activation().ok_or(CatalogError::Fenced)?; + let operation = ProposalId::new(origin.operation()) + .map_err(|_| CatalogError::Corrupt("transfer operation"))?; + let (view, _) = self.current().await?; + self.selected_route_in(&view, marker.route()).await?; + let journal = self + .transition_record(&view.head, operation) + .await? + .ok_or(CatalogError::Fenced)?; + if journal.phase() != TransitionPhase::Retired + || journal.request != origin.plan() + || journal.sealed_sources() != Some(origin.sealed_sources()) + || journal.publication != Some(publication) + || Index::new(self) + .lookup(view.head.operations, &child_key(marker.route().history)) + .await? + .as_deref() + != Some(publication.as_slice()) + { + return Err(CatalogError::Fenced); + } + Ok(()) + } + + pub(super) async fn check_confirmed_transfer( + &self, + head: &wire::Head, + journal: &wire::Journal, + publication: [u8; 32], + ) -> Result<(), CatalogError> { + let Some(archive) = journal.transfer_archive else { + return Ok(()); + }; + let operation = ProposalId::new(journal.operation) + .map_err(|_| CatalogError::Corrupt("transfer publication operation"))?; + let selected = self + .transition_record(head, operation) + .await? + .ok_or(CatalogError::Corrupt("missing published transfer decision"))?; + if !matches!( + selected.phase, + TransitionPhase::Published | TransitionPhase::Active | TransitionPhase::Retired + ) || selected.request != journal.request + || selected.archive != Some(archive) + || selected.publication != Some(publication) + || (self.transition_active(head).await? == Some(operation)) != selected.owns_topology() + { + return Err(CatalogError::Corrupt("transfer publication binding")); + } + Ok(()) + } + + pub(super) async fn supersede_transition_for_recovery( + &self, + head: &mut wire::Head, + operation: ProposalId, + plan: [u8; 32], + completion: [u8; 32], + ) -> Result<(), CatalogError> { + if self.transition_active(head).await? != Some(operation) { + return Err(CatalogError::Fenced); + } + let mut journal = self + .transition_record(head, operation) + .await? + .ok_or(CatalogError::Corrupt("missing recovery transition"))?; + if journal.request != plan || !journal.owns_topology() { + return Err(CatalogError::Fenced); + } + journal.recovery_settlement = Some(completion); + let index = Index::new(self); + let operations = index + .update( + head.operations, + &key(operation), + Some(record::encode(JOURNAL, &journal, JOURNAL_LIMIT)?), + 0, + ) + .await?; + head.operations = index.update(operations, ACTIVE, None, 0).await?; + Ok(()) + } + + pub(super) async fn verify_recovered_transition( + &self, + head: &wire::Head, + operation: ProposalId, + plan: [u8; 32], + completion: [u8; 32], + ) -> Result<(), CatalogError> { + let journal = self + .transition_record(head, operation) + .await? + .ok_or(CatalogError::Corrupt("missing recovered transition"))?; + if journal.request != plan + || journal.recovery_settlement != Some(completion) + || self.transition_active(head).await? == Some(operation) + { + return Err(CatalogError::Corrupt("recovery transition completion")); + } + Ok(()) + } + + pub(super) async fn commit_transition_head( + &self, + head: &wire::Head, + version: UpdateVersion, + ) -> Result { + self.check_head(head)?; + let bytes = record::encode(wire::HEAD, head, wire::LIMIT)?; + // A transport failure can follow the committed conditional update. + // Observe the rooted winner before returning a seal decision. + let _ = self + .namespace + .inner + .store + .put_opts( + &self.key("head"), + bytes.into(), + PutOptions::from(PutMode::Update(version)), + ) + .await; + self.current() + .await + .map(|(view, _)| view) + .map_err(|_| CatalogError::Unconfirmed) + } + + /// Reserve one exact split or merge against the selected range tree. + /// Children must already have immutable birth reservations. No voter or + /// resource admission is granted by the journal alone. + pub async fn plan_transition( + &self, + mutation: &DirectoryMutation, + ) -> Result { + self.unmanaged_birth()?; + self.plan_transition_inner(mutation, false).await + } + + pub(crate) async fn plan_managed_transition( + &self, + mutation: &DirectoryMutation, + permit: &ManagedTransitionPermit, + ) -> Result { + permit.check(self, mutation)?; + self.plan_transition_inner(mutation, true).await + } + + async fn plan_transition_inner( + &self, + mutation: &DirectoryMutation, + managed: bool, + ) -> Result { + self.check_transition_plan(mutation)?; + self.bounded(async { + let (mut view, version) = self.current().await?; + if let Some(journal) = self.matching_transition(&view.head, mutation).await? { + return Ok(journal); + } + if self.namespace.cell_group_owner().await?.is_some() != managed { + return Err(CatalogError::ManagedAdmissionRequired); + } + let count = self.check_new_transition(&view, mutation).await?; + let index = Index::new(self); + self.immutable(PLANS, mutation.digest, mutation.encode()) + .await?; + let journal = TransitionJournal { + request: mutation.digest, + preparation: None, + preparation_sources: None, + preparation_acquires: None, + source_set: None, + installed_set: None, + archive: None, + publication: None, + activation_set: None, + child_archives: None, + retirement_set: None, + parent_holds_released: false, + recovery_settlement: None, + phase: TransitionPhase::Planned, + }; + for (key, value) in [ + (COUNT, (count + 1).to_be_bytes().to_vec()), + (ACTIVE, mutation.operation.bytes().to_vec()), + ( + &key(mutation.operation), + record::encode(JOURNAL, &journal, JOURNAL_LIMIT)?, + ), + ] { + view.head.operations = index + .update(view.head.operations, key, Some(value), 0) + .await?; + } + let after = self.commit_transition_head(&view.head, version).await?; + self.matching_transition(&after.head, mutation) + .await? + .ok_or(CatalogError::Unconfirmed) + }) + .await + } + + pub(crate) async fn stage_managed_transition_plan( + &self, + mutation: &DirectoryMutation, + permit: &ManagedTransitionPermit, + ) -> Result<(), CatalogError> { + permit.check(self, mutation)?; + self.check_transition_plan(mutation)?; + self.bounded(async { + let (view, _) = self.current().await?; + if self + .matching_transition(&view.head, mutation) + .await? + .is_some() + { + return Err(CatalogError::Conflict("transition already decided")); + } + self.check_new_transition(&view, mutation).await?; + // Stage the bytes before the fleet can select their digest. A + // controller may lose its reply before entering the directory CAS. + self.immutable(PLANS, mutation.digest, mutation.encode()) + .await + }) + .await + } + + pub(crate) async fn reopen_managed_transition_plan( + &self, + operation: ProposalId, + digest: [u8; 32], + permit: &ManagedTransitionPermit, + ) -> Result<(DirectoryMutation, Option), CatalogError> { + self.bounded(async { + let plan = self.load_transition_plan(operation, digest).await?; + permit.check(self, &plan)?; + let (view, _) = self.current().await?; + let journal = self.matching_transition_journal(&view.head, &plan).await?; + Ok((plan, journal)) + }) + .await + } + + /// Resolve the source domain against a single rooted view. The immutable + /// publication and the selected tree must agree even after a lost CAS reply. + pub(crate) async fn observe_managed_transition_recovery( + &self, + operation: ProposalId, + digest: [u8; 32], + permit: &ManagedTransitionPermit, + ) -> Result { + self.bounded(async { + let plan = self.load_transition_plan(operation, digest).await?; + permit.check(self, &plan)?; + let (view, _) = self.current().await?; + if view.head.import.is_some() || view.head.recovery.is_some() { + return Err(CatalogError::Fenced); + } + let journal = self + .matching_transition(&view.head, &plan) + .await? + .ok_or(CatalogError::Unconfirmed)?; + if !journal.owns_topology() { + return Err(CatalogError::Fenced); + } + let publication = self.confirmed(&view.head, &plan).await?; + if publication + .as_ref() + .map(|decision| (decision.journal_digest(), decision.transfer_archive())) + != journal + .publication + .map(|id| (id, journal.transfer_archive())) + { + return Err(CatalogError::Corrupt("transition recovery publication")); + } + let (parents, children) = self.transition_routes(&plan)?; + let sources = if publication.is_some() { + &children + } else { + &parents + }; + for route in sources { + self.selected_route_in(&view, route).await?; + } + Ok(ManagedTransitionRecoverySelection { + plan, + directory_digest: view.digest, + publication: journal.publication, + journal, + parents, + children, + }) + }) + .await + } + + async fn managed_recovery_lock( + &self, + request: [u8; 32], + basis: &ManagedRecoveryIntentBasis, + holds: &[CellGroupArchiveHold], + permit: &ManagedTransitionPermit, + ) -> Result<(DirectoryMutation, recovery::RecoveryLock), CatalogError> { + let operation = ProposalId::new(basis.transition) + .map_err(|_| CatalogError::Corrupt("recovery transition operation"))?; + let plan = self.load_transition_plan(operation, basis.plan).await?; + permit.check(self, &plan)?; + let (parents, children) = self.transition_routes(&plan)?; + let sources = if basis.publication.is_some() { + &children + } else { + &parents + }; + if holds.len() != sources.len() + || sources + .iter() + .zip(holds) + .any(|(route, hold)| hold.marker().route() != route) + { + return Err(CatalogError::Conflict("managed recovery archive set")); + } + let gate = recovery::RecoveryLock::managed( + self, + request, + basis, + sources + .first() + .ok_or(CatalogError::Corrupt("managed recovery has no sources"))?, + parents.iter().chain(&children).map(|route| route.history), + )?; + Ok((plan, gate)) + } + + pub(crate) async fn observe_managed_recovery_barrier( + &self, + request: [u8; 32], + basis: &ManagedRecoveryIntentBasis, + holds: &[CellGroupArchiveHold], + permit: &ManagedTransitionPermit, + ) -> Result, CatalogError> { + self.bounded(async { + let (view, _) = self.current().await?; + match view.head.recovery.as_ref() { + Some(lock) => { + let (_, gate) = self + .managed_recovery_lock(request, basis, holds, permit) + .await?; + if **lock != gate { + return Err(CatalogError::Fenced); + } + Ok(Some(view.digest)) + } + None => Ok(None), + } + }) + .await + } + + /// Select recovery in the same directory CAS that orders transfer + /// publication, seal authorization and retirement. Fleet retains its + /// disruption slot while the managed recovery owns the directory. + pub(crate) async fn select_managed_recovery_barrier( + &self, + request: [u8; 32], + basis: &ManagedRecoveryIntentBasis, + holds: &[CellGroupArchiveHold], + permit: &ManagedTransitionPermit, + ) -> Result<[u8; 32], CatalogError> { + self.bounded(async { + let (plan, gate) = self + .managed_recovery_lock(request, basis, holds, permit) + .await?; + let (mut view, version) = self.current().await?; + if let Some(lock) = &view.head.recovery { + return if **lock == gate { + Ok(view.digest) + } else { + Err(CatalogError::Fenced) + }; + } + if view.digest != basis.directory_root || view.head.import.is_some() { + return Err(CatalogError::Conflict("stale managed recovery topology")); + } + let journal = self + .matching_transition_journal(&view.head, &plan) + .await? + .filter(TransitionJournal::owns_topology) + .ok_or(CatalogError::Fenced)?; + if journal.publication != basis.publication + || self + .confirmed(&view.head, &plan) + .await? + .as_ref() + .map(DirectoryPublication::journal_digest) + != basis.publication + { + return Err(CatalogError::Conflict("managed recovery publication")); + } + for hold in holds { + self.selected_route_in(&view, hold.marker().route()).await?; + } + view.head.recovery = Some(Box::new(gate.clone())); + let after = self.commit_transition_head(&view.head, version).await?; + if after.head.recovery.as_deref() != Some(&gate) { + return Err(CatalogError::Unconfirmed); + } + Ok(after.digest) + }) + .await + } + + pub(crate) async fn check_managed_transition_recovery_holds( + &self, + selection: &ManagedTransitionRecoverySelection, + permit: &ManagedTransitionPermit, + holds: &[CellGroupArchiveHold], + ) -> Result<(), CatalogError> { + let routes = selection.archive_routes(); + if holds.len() != routes.len() { + return Err(CatalogError::Invalid( + "incomplete transition recovery archives", + )); + } + permit.check(self, &selection.plan)?; + let installed = if selection.publication.is_some() { + self.installed_transfer_children_inner(&selection.plan, Some(permit)) + .await? + .ok_or(CatalogError::Unconfirmed)? + } else { + Vec::new() + }; + for (index, (route, hold)) in routes.iter().zip(holds).enumerate() { + let marker = hold.marker(); + if marker.route() != route { + return Err(CatalogError::Fenced); + } + if let Some(publication) = selection.publication { + if !marker.is_active() + || installed + .get(index) + .is_none_or(|prior| !marker.preserves_transfer(prior)) + || marker + .transfer_activation() + .is_none_or(|(bound, _, _)| bound != publication) + { + return Err(CatalogError::Fenced); + } + } else { + let cut = if marker.is_active() && selection.journal.sealed_sources().is_none() { + TransitionSourceCut::Active + } else { + TransitionSourceCut::Sealed + }; + self.check_source_marker(&selection.plan, &selection.journal, route, marker, cut)?; + } + } + Ok(()) + } + + pub(crate) async fn managed_transition_status( + &self, + mutation: &DirectoryMutation, + permit: &ManagedTransitionPermit, + ) -> Result, CatalogError> { + permit.check(self, mutation)?; + self.check_transition_plan(mutation)?; + self.bounded(async { + let (view, _) = self.current().await?; + self.matching_transition(&view.head, mutation).await + }) + .await + } + + async fn check_new_transition( + &self, + view: &DirectoryView, + mutation: &DirectoryMutation, + ) -> Result { + let count = self + .check_transition_admission_input(view, mutation) + .await?; + for child in &mutation.request.children { + self.require_ordinary_birth_reservation(&child.decode(self)?) + .await?; + } + Ok(count) + } + + /// Revalidate the exact directory cut before staging child reservations. + pub(crate) async fn validate_managed_transition_input( + &self, + mutation: &DirectoryMutation, + ) -> Result<(), CatalogError> { + self.check_transition_plan(mutation)?; + self.bounded(async { + let (view, _) = self.current().await?; + self.check_transition_admission_input(&view, mutation) + .await?; + Ok(()) + }) + .await + } + + async fn check_transition_admission_input( + &self, + view: &DirectoryView, + mutation: &DirectoryMutation, + ) -> Result { + if view.head.import.is_some() + || view.head.recovery.is_some() + || self.transition_active(&view.head).await?.is_some() + { + return Err(CatalogError::Fenced); + } + let index = Index::new(self); + if mutation.request.expected != Some(view.digest) + || index + .lookup( + view.head.operations, + &hex::encode(mutation.operation.bytes()), + ) + .await? + .is_some() + { + return Err(CatalogError::Conflict("stale transition root or operation")); + } + let count = self.available_transition_slot(&view.head).await?; + for parent in &mutation.request.parents { + self.selected_route_in(view, &parent.decode(self)?).await?; + } + for child in &mutation.request.children { + if index + .lookup(view.head.groups, &hex::encode(child.group)) + .await? + .is_some() + { + return Err(CatalogError::Conflict( + "transition child group is already assigned", + )); + } + } + Ok(count) + } + + async fn available_transition_slot(&self, head: &wire::Head) -> Result { + let index = Index::new(self); + let count = index + .lookup(head.operations, COUNT) + .await? + .map(|value| { + value + .try_into() + .map(u64::from_be_bytes) + .map_err(|_| CatalogError::Corrupt("transition count")) + }) + .transpose()? + .unwrap_or(0); + if count >= self.limits.max_mutations { + return Err(CatalogError::Limit); + } + Ok(count) + } + + /// Recover the original request and decision after losing the controller. + pub async fn active_transition( + &self, + ) -> Result, CatalogError> { + self.bounded(async { + let (view, _) = self.current().await?; + let Some(operation) = self.transition_active(&view.head).await? else { + return Ok(None); + }; + let journal = self + .transition_record(&view.head, operation) + .await? + .ok_or(CatalogError::Corrupt("missing active transition"))?; + let mutation = self + .load_transition_plan(operation, journal.request) + .await?; + if !journal.owns_topology() { + return Err(CatalogError::Corrupt("active transition plan binding")); + } + Ok(Some((mutation, journal))) + }) + .await + } + + async fn load_transition_plan( + &self, + operation: ProposalId, + digest: [u8; 32], + ) -> Result { + let bytes = self + .read(&format!("{PLANS}/{}", hex::encode(digest))) + .await? + .ok_or(CatalogError::Corrupt("missing transition plan"))?; + let mutation = self.decode_mutation(&bytes)?; + self.check_transition_plan(&mutation)?; + if mutation.operation != operation || mutation.digest != digest { + return Err(CatalogError::Corrupt("transition plan binding")); + } + Ok(mutation) + } + + /// Issue the selected transition's exact seal for a local source owner. + /// Retrying after a lost response uses the original plan and journal digest. + pub async fn prepare_source_seal( + &self, + mutation: &DirectoryMutation, + source: GroupHistory, + log: &ReplicaLog, + ) -> Result { + self.unmanaged_birth()?; + self.prepare_source_seal_inner(mutation, source, log, None) + .await + } + + pub(crate) async fn prepare_managed_source_seal( + &self, + mutation: &DirectoryMutation, + source: GroupHistory, + log: &ReplicaLog, + host: &FleetHostAuthorization, + authority: (&ManagedTransitionPermit, &FleetGroupAuthorization), + deadline: u64, + ) -> Result { + let (permit, group) = authority; + permit.check(self, mutation)?; + self.prepare_source_seal_inner( + mutation, + source, + log, + Some(ManagedSealAdmission { + host, + group, + deadline, + }), + ) + .await + } + + async fn prepare_source_seal_inner( + &self, + mutation: &DirectoryMutation, + source: GroupHistory, + log: &ReplicaLog, + managed: Option>, + ) -> Result { + self.check_transition_plan(mutation)?; + self.bounded(async { + let (view, _) = self.current().await?; + let journal = self + .matching_transition(&view.head, mutation) + .await? + .ok_or(CatalogError::Fenced)?; + let route = mutation + .request + .parents + .iter() + .find(|route| { + route.group == source.group.bytes() + && route.incarnation == source.incarnation.bytes() + }) + .ok_or(CatalogError::Fenced)? + .decode(self)?; + if route.history != source { + return Err(CatalogError::Fenced); + } + self.selected_route_in(&view, &route).await?; + let group = log.cell_group(&source).map_err(CatalogError::Log)?; + if group + .group_manifest() + .map_err(CatalogError::Log)? + .as_ref() + .is_none_or(|manifest| manifest.route() != &route) + { + return Err(CatalogError::Fenced); + } + let (transition, authorization, entry) = + self.source_seal_entry(mutation, &journal, &route)?; + let managed_owner = if let Some(admission) = &managed { + let birth = self.materialize_selected_history(source).await?; + let boot = self.require_managed_group_boot(log, admission.host).await?; + let cache = group.read().map_err(CatalogError::Log)?; + let manifest = cache.group_manifest.as_ref().ok_or(CatalogError::Fenced)?; + if birth.route() != &route + || manifest.route() != &route + || !admission.group.check_birth(self, &birth) + || cache + .membership + .fleet_status() + .is_none_or(|status| &status.binding != admission.group.binding()) + || !admission + .group + .matches_replica(group.identity(), &cache.membership) + || !cache + .conf_state + .voters + .contains(&group.identity().member.get()) + { + return Err(CatalogError::Fenced); + } + Some(ManagedSealOwner { + boot, + membership_revision: manifest.membership_revision(), + }) + } else { + None + }; + let now = log + .domain() + .mono_ms() + .map_err(|_| CatalogError::Unconfirmed)?; + let deadline = now + .checked_add(self.limits.operation_timeout_ms) + .ok_or(CatalogError::Limit)?; + let deadline = managed.map_or(deadline, |admission| deadline.min(admission.deadline)); + if deadline <= now { + return Err(CatalogError::Unconfirmed); + } + Ok(CellGroupSealPermit { + group, + engine: log.storage_instance().map_err(CatalogError::Log)?, + deadline, + transition, + authorization, + payload: Sha256::digest(&entry).into(), + entry, + managed: managed_owner, + }) + }) + .await + } + + pub(super) fn source_seal_entry( + &self, + mutation: &DirectoryMutation, + journal: &TransitionJournal, + route: &GroupRoute, + ) -> Result<(TransitionId, [u8; 32], Vec), CatalogError> { + let transition = TransitionId::new(mutation.operation.bytes()) + .map_err(|_| CatalogError::Invalid("transition identity"))?; + let authorization = journal.seal_authorization()?; + let children = mutation + .request + .children + .iter() + .map(|route| route.decode(self)) + .collect::>()?; + let entry = CellGroupEntry { + binding: GroupEntryBinding { + group: route.history, + route_epoch: route.epoch, + birth_digest: route.birth_digest, + schema_digest: route.schema_digest, + proposal: mutation.operation, + }, + command: GroupCommand::Seal { + transition, + authorization, + children, + }, + } + .encode(route, GroupEntryLimits::default()) + .map_err(|_| CatalogError::Invalid("source seal entry"))?; + Ok((transition, authorization, entry)) + } + + pub(super) async fn record_verified_sources( + &self, + mutation: &DirectoryMutation, + digest: [u8; 32], + permit: Option<&ManagedTransitionPermit>, + ) -> Result { + let advance = Advance::RecordSources(digest); + if let Some(permit) = permit { + permit.check(self, mutation)?; + self.advance_transition_inner(mutation, advance, false) + .await + } else { + self.advance_transition(mutation, advance).await + } + } + + pub(super) async fn record_verified_child_quorums( + &self, + mutation: &DirectoryMutation, + digest: [u8; 32], + permit: Option<&ManagedTransitionPermit>, + ) -> Result { + let advance = Advance::RecordInstalled(digest); + if let Some(permit) = permit { + permit.check(self, mutation)?; + self.advance_transition_inner(mutation, advance, false) + .await + } else { + self.advance_transition(mutation, advance).await + } + } + + pub(super) async fn record_verified_transfer_archive( + &self, + mutation: &DirectoryMutation, + sources: [u8; 32], + installed: [u8; 32], + digest: [u8; 32], + permit: Option<&ManagedTransitionPermit>, + ) -> Result { + let advance = Advance::RecordArchive { + sources, + installed, + digest, + }; + if let Some(permit) = permit { + permit.check(self, mutation)?; + self.advance_transition_inner(mutation, advance, false) + .await + } else { + self.advance_transition(mutation, advance).await + } + } + + pub(super) async fn record_verified_transfer_activations( + &self, + mutation: &DirectoryMutation, + publication: [u8; 32], + installed: [u8; 32], + digest: [u8; 32], + permit: Option<&ManagedTransitionPermit>, + ) -> Result { + let advance = Advance::RecordActivations { + publication, + installed, + digest, + }; + if let Some(permit) = permit { + permit.check(self, mutation)?; + self.advance_transition_inner(mutation, advance, false) + .await + } else { + self.advance_transition(mutation, advance).await + } + } + + pub(super) async fn record_verified_child_archives( + &self, + mutation: &DirectoryMutation, + publication: [u8; 32], + activations: [u8; 32], + digest: [u8; 32], + permit: Option<&ManagedTransitionPermit>, + ) -> Result { + let advance = Advance::RecordChildArchives { + publication, + activations, + digest, + }; + if let Some(permit) = permit { + permit.check(self, mutation)?; + self.advance_transition_inner(mutation, advance, false) + .await + } else { + self.advance_transition(mutation, advance).await + } + } + + pub(super) async fn record_verified_retirement( + &self, + mutation: &DirectoryMutation, + publication: [u8; 32], + child_archives: [u8; 32], + digest: [u8; 32], + permit: Option<&ManagedTransitionPermit>, + ) -> Result { + let advance = Advance::RecordRetired { + publication, + child_archives, + digest, + }; + if let Some(permit) = permit { + permit.check(self, mutation)?; + self.advance_transition_inner(mutation, advance, false) + .await + } else { + self.advance_transition(mutation, advance).await + } + } + + pub(super) async fn record_released_parent_holds( + &self, + mutation: &DirectoryMutation, + digest: [u8; 32], + permit: Option<&ManagedTransitionPermit>, + ) -> Result { + let advance = Advance::ReleaseParentHolds(digest); + if let Some(permit) = permit { + permit.check(self, mutation)?; + self.advance_transition_inner(mutation, advance, false) + .await + } else { + self.advance_transition(mutation, advance).await + } + } + + /// Record the exact digest of the controller's child preparation evidence. + /// This metadata boundary does not verify copied images or admit a child. + pub async fn prepare_transition( + &self, + mutation: &DirectoryMutation, + preparation: [u8; 32], + ) -> Result { + self.advance_transition(mutation, Advance::Prepare(preparation, None)) + .await + } + + pub(super) async fn record_managed_preparation_acquires( + &self, + mutation: &DirectoryMutation, + permit: &ManagedTransitionPermit, + previous: Option<[u8; 32]>, + next: [u8; 32], + ) -> Result { + permit.check(self, mutation)?; + self.advance_transition_inner( + mutation, + Advance::ChoosePreparationAcquires { previous, next }, + false, + ) + .await + } + + pub(super) async fn record_managed_preparation_sources( + &self, + mutation: &DirectoryMutation, + permit: &ManagedTransitionPermit, + sources: [u8; 32], + acquires: Option<[u8; 32]>, + ) -> Result { + permit.check(self, mutation)?; + self.advance_transition_inner( + mutation, + Advance::StagePreparationSources { sources, acquires }, + false, + ) + .await + } + + pub(super) async fn record_managed_preparation( + &self, + mutation: &DirectoryMutation, + permit: &ManagedTransitionPermit, + preparation: [u8; 32], + sources: [u8; 32], + ) -> Result { + permit.check(self, mutation)?; + self.advance_transition_inner( + mutation, + Advance::Prepare(preparation, Some(sources)), + false, + ) + .await + } + + pub async fn authorize_transition_seal( + &self, + mutation: &DirectoryMutation, + ) -> Result { + self.advance_transition(mutation, Advance::Authorize).await + } + + pub(crate) async fn authorize_managed_transition_seal( + &self, + mutation: &DirectoryMutation, + permit: &ManagedTransitionPermit, + ) -> Result { + permit.check(self, mutation)?; + self.advance_transition_inner(mutation, Advance::Authorize, false) + .await + } + + pub async fn cancel_transition( + &self, + mutation: &DirectoryMutation, + ) -> Result { + self.advance_transition(mutation, Advance::Cancel).await + } + + pub(crate) async fn cancel_managed_transition( + &self, + mutation: &DirectoryMutation, + permit: &ManagedTransitionPermit, + ) -> Result { + permit.check(self, mutation)?; + self.advance_transition_inner(mutation, Advance::Cancel, true) + .await + } + + async fn advance_transition( + &self, + mutation: &DirectoryMutation, + advance: Advance, + ) -> Result { + self.unmanaged_birth()?; + self.advance_transition_inner(mutation, advance, false) + .await + } + + async fn advance_transition_inner( + &self, + mutation: &DirectoryMutation, + advance: Advance, + cancel_unplanned: bool, + ) -> Result { + self.check_transition_plan(mutation)?; + self.bounded(async { + let (mut view, version) = self.current().await?; + if view.head.recovery.is_some() { + return Err(CatalogError::Fenced); + } + let Some(before) = self.matching_transition(&view.head, mutation).await? else { + if !cancel_unplanned { + return Err(CatalogError::Conflict("unplanned transition")); + } + // The fleet slot can commit before the directory plan. Root a + // cancellation even then, so a delayed planner cannot reopen it. + return self + .cancel_unplanned_transition(mutation, view, version) + .await; + }; + let requested = advance.apply(before.clone())?; + if requested == before { + return Ok(before); + } + let index = Index::new(self); + let bytes = record::encode(JOURNAL, &requested, JOURNAL_LIMIT)?; + view.head.operations = index + .update( + view.head.operations, + &key(mutation.operation), + Some(bytes), + 0, + ) + .await?; + if before.owns_topology() && !requested.owns_topology() { + view.head.operations = index.update(view.head.operations, ACTIVE, None, 0).await?; + } + let after = self.commit_transition_head(&view.head, version).await?; + let winner = self + .matching_transition(&after.head, mutation) + .await? + .ok_or(CatalogError::Corrupt("lost transition journal"))?; + if advance.apply(winner.clone())? != winner { + return Err(CatalogError::Unconfirmed); + } + Ok(winner) + }) + .await + } + + async fn cancel_unplanned_transition( + &self, + mutation: &DirectoryMutation, + mut view: DirectoryView, + version: UpdateVersion, + ) -> Result { + let index = Index::new(self); + let journal = TransitionJournal { + request: mutation.digest, + preparation: None, + preparation_sources: None, + preparation_acquires: None, + source_set: None, + installed_set: None, + archive: None, + publication: None, + activation_set: None, + child_archives: None, + retirement_set: None, + parent_holds_released: false, + recovery_settlement: None, + phase: TransitionPhase::Cancelled, + }; + let count = self.available_transition_slot(&view.head).await?; + view.head.operations = index + .update( + view.head.operations, + COUNT, + Some((count + 1).to_be_bytes().to_vec()), + 0, + ) + .await?; + view.head.operations = index + .update( + view.head.operations, + &key(mutation.operation), + Some(record::encode(JOURNAL, &journal, JOURNAL_LIMIT)?), + 0, + ) + .await?; + let after = self.commit_transition_head(&view.head, version).await?; + let winner = self.matching_transition(&after.head, mutation).await?; + if winner.as_ref() != Some(&journal) { + return Err(CatalogError::Unconfirmed); + } + Ok(journal) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/transition_acquires.rs b/crates/cellar/durability/raft/cell_groups/directory/transition_acquires.rs new file mode 100644 index 000000000..05917cce6 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/transition_acquires.rs @@ -0,0 +1,233 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Root operation-owned archive acquires before allocating pre-copy holds. +use super::*; +use crate::raft::{ + cell_groups::{ + CellGroupArchiveAcquireOutcome, CellGroupArchiveError, CellGroupArchiveExecution, + CellGroupArchiveIntent, CellGroupArchivePool, + }, + fleet_catalog::ManagedTransitionPermit, +}; +use serde::{Deserialize, Serialize}; + +const ACQUIRES: &str = "transition-preparation-acquires"; +const KIND: u8 = 214; + +type Result = std::result::Result; + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Acquires { + namespace: [u8; 32], + plan: [u8; 32], + intents: Vec, +} + +struct Selected { + digest: [u8; 32], + intents: Vec, +} + +impl RangeDirectory { + /// Acquire holds using only the intents selected by the directory. A + /// proven superseded intent can be replaced while the operation is Planned. + pub(crate) async fn allocate_managed_preparation_holds( + &self, + domain: &impl CellGroupArchiveExecution, + archives: &CellGroupArchivePool, + plan: &DirectoryMutation, + permit: &ManagedTransitionPermit, + ) -> Result { + permit.check(self, plan)?; + let (journal, selected) = self.preparation_acquires(plan).await?; + if !matches!( + journal.phase(), + TransitionPhase::Planned | TransitionPhase::Prepared + ) { + return Err(CatalogError::Fenced.into()); + } + if journal.preparation_sources().is_some() { + return if selected.is_some() { + Ok(journal) + } else { + Err(CatalogError::Conflict("preparation has caller-owned holds").into()) + }; + } + if journal.phase() != TransitionPhase::Planned { + return Err(CatalogError::Conflict("preparation already selected").into()); + } + + let (parents, _) = self.transition_routes(plan)?; + let mut registries = Vec::with_capacity(parents.len()); + for parent in &parents { + let birth = self.materialize_selected_history(parent.history).await?; + if birth.route() != parent { + return Err(CatalogError::Fenced.into()); + } + registries.push(self.open_group_archive(domain, archives, &birth).await?); + } + let selected = match selected { + Some(selected) => selected, + None => { + let mut intents = Vec::with_capacity(registries.len()); + for registry in ®istries { + intents.push(registry.prepare_acquire(domain).await?); + } + self.choose_preparation_acquires(plan, permit, None, intents) + .await? + } + }; + + let mut holds = Vec::with_capacity(parents.len()); + for (index, registry) in registries.iter().enumerate() { + let intent = &selected.intents[index]; + let hold = match registry.acquire(domain, intent).await { + Ok(hold) => hold, + Err(error @ (CellGroupArchiveError::Expired | CellGroupArchiveError::Conflict)) => { + match registry.acquire_outcome(domain, intent).await? { + CellGroupArchiveAcquireOutcome::Held(hold) => *hold, + CellGroupArchiveAcquireOutcome::Pending => return Err(error.into()), + CellGroupArchiveAcquireOutcome::Superseded => { + // The old revision is consumed and owns no hold. + // A successor controller must reopen the new set. + let mut intents = selected.intents; + intents[index] = registry.prepare_acquire(domain).await?; + self.choose_preparation_acquires( + plan, + permit, + Some(selected.digest), + intents, + ) + .await?; + return Err(CatalogError::Unconfirmed.into()); + } + } + } + Err(error) => return Err(error.into()), + }; + holds.push(hold); + } + self.stage_managed_preparation_holds( + domain, + archives, + plan, + permit, + &holds, + Some(selected.digest), + ) + .await + } + + /// Reconcile each selected acquire after cancellation. The caller joins + /// entered physical pre-copy work before using this cleanup boundary. + pub(crate) async fn release_cancelled_preparation_acquires( + &self, + domain: &impl CellGroupArchiveExecution, + archives: &CellGroupArchivePool, + plan: &DirectoryMutation, + ) -> Result<()> { + let (journal, selected) = self.preparation_acquires(plan).await?; + if journal.phase() != TransitionPhase::Cancelled { + return Err(CatalogError::Fenced.into()); + } + if let Some(selected) = selected { + let (parents, _) = self.transition_routes(plan)?; + for (route, intent) in parents.iter().zip(&selected.intents) { + let registry = self + .open_retired_source_archive(domain, archives, route) + .await?; + registry.cancel_acquire(domain, intent).await?; + } + } + Ok(()) + } + + async fn preparation_acquires( + &self, + plan: &DirectoryMutation, + ) -> Result<(TransitionJournal, Option)> { + if self.namespace_config().managed_fleet.is_none() { + return Err(CatalogError::ManagedAdmissionRequired.into()); + } + self.check_transition_plan(plan)?; + let (journal, stored) = self + .bounded(async { + let (view, _) = self.current().await?; + let journal = self + .matching_transition(&view.head, plan) + .await? + .ok_or(CatalogError::Fenced)?; + let stored = if let Some(digest) = journal.preparation_acquires() { + let bytes = self + .read(&format!("{ACQUIRES}/{}", hex::encode(digest))) + .await? + .ok_or(CatalogError::Corrupt("missing preparation acquires"))?; + if record::digest(&bytes) != digest { + return Err(CatalogError::Corrupt("preparation acquire digest")); + } + let acquires: Acquires = record::decode(KIND, &bytes, wire::LIMIT)?; + if acquires.namespace != self.namespace.root_digest + || acquires.plan != plan.digest() + { + return Err(CatalogError::Corrupt("preparation acquire binding")); + } + Some(Selected { + digest, + intents: acquires.intents, + }) + } else { + None + }; + Ok((journal, stored)) + }) + .await?; + if let Some(selected) = &stored { + self.check_preparation_acquires(plan, &selected.intents)?; + } + Ok((journal, stored)) + } + + async fn choose_preparation_acquires( + &self, + plan: &DirectoryMutation, + permit: &ManagedTransitionPermit, + previous: Option<[u8; 32]>, + intents: Vec, + ) -> Result { + self.check_preparation_acquires(plan, &intents)?; + let acquires = Acquires { + namespace: self.namespace.root_digest, + plan: plan.digest(), + intents, + }; + let bytes = record::encode(KIND, &acquires, wire::LIMIT)?; + let digest = record::digest(&bytes); + self.bounded(self.immutable(ACQUIRES, digest, &bytes)) + .await?; + self.record_managed_preparation_acquires(plan, permit, previous, digest) + .await?; + Ok(Selected { + digest, + intents: acquires.intents, + }) + } + + fn check_preparation_acquires( + &self, + plan: &DirectoryMutation, + intents: &[CellGroupArchiveIntent], + ) -> Result<()> { + let (parents, _) = self.transition_routes(plan)?; + if intents.len() != parents.len() { + return Err(CatalogError::Corrupt("preparation acquire count").into()); + } + for (route, intent) in parents.iter().zip(intents) { + let marker = intent.acquire_marker()?; + if marker.route() != route || !marker.is_active() { + return Err(CatalogError::Fenced.into()); + } + } + Ok(()) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/transition_activation.rs b/crates/cellar/durability/raft/cell_groups/directory/transition_activation.rs new file mode 100644 index 000000000..325b314be --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/transition_activation.rs @@ -0,0 +1,634 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use crate::application::consensus::group_protocol::{ + CellGroupEntry, GroupCommand, GroupEntryBinding, GroupEntryLimits, +}; +use crate::raft::{ + cache::GroupLog, + cell_groups::GroupManifest, + completion::RuntimeError, + fleet_catalog::{ + FleetHostAuthorization, ManagedTransitionPermit, ReopenedManagedCellGroupTransition, + TransferVoterAdmission, + }, + fleet_stop::BootBinding, + storage::ReplicaLog, +}; +use cellar_logic::consensus::{EngineInstanceId, LogIndex, LogPosition, Term}; +use serde::{Deserialize, Serialize}; + +const ACTIVATIONS: u8 = 208; +const SETS: &str = "transfer-activation-sets"; +const STAGED_ACTIVATION: u8 = 212; + +/// Historical proof from an engine-bound Raft owner after durable apply. +#[derive(Clone, Debug)] +pub struct CellGroupTransferActivationAck { + route: GroupRoute, + publication: [u8; 32], + manifest: [u8; 32], + position: LogPosition, +} + +impl CellGroupTransferActivationAck { + pub fn position(&self) -> LogPosition { + self.position + } +} + +/// Exact selected publication and seed, bound to one open voter engine. +pub struct CellGroupTransferActivationPermit { + group: GroupLog, + engine: EngineInstanceId, + publication: [u8; 32], + manifest: [u8; 32], + deadline: u64, + entry: Vec, + boot: Option, +} + +impl CellGroupTransferActivationPermit { + pub(crate) fn history(&self) -> GroupHistory { + self.group.identity().scope + } + + pub(crate) fn check( + &self, + log: &ReplicaLog, + group: &GroupLog, + ) -> Result<(), RuntimeError> { + if !Arc::ptr_eq(&group.owner, &self.group.owner) + || group.identity() != self.group.identity() + || log.storage_instance()? != self.engine + || log.domain().mono_ms().map_err(|_| RuntimeError::Closed)? >= self.deadline + { + return Err(RuntimeError::StaleCompletion); + } + if let Some(boot) = &self.boot { + if log + .lifecycle_guard()? + .is_none_or(|guard| !guard.admits_group_start(boot)) + { + return Err(RuntimeError::StaleCompletion); + } + let cache = group.read()?; + if cache.membership.revision().get() != 1 + || !cache + .conf_state + .voters + .contains(&group.identity().member.get()) + { + return Err(RuntimeError::StaleCompletion); + } + } else { + drop(group.read()?); + } + Ok(()) + } + + pub(crate) fn acknowledge( + &self, + group: &GroupLog, + marker: &GroupManifest, + ) -> Result, RuntimeError> { + if marker.route().history != self.history() + || marker + .transfer_commitment() + .map_err(|e| RuntimeError::Invalid(e.to_string()))? + != Some(self.manifest) + { + return Err(RuntimeError::StaleCompletion); + } + match marker.transfer_activation() { + Some((publication, manifest, position)) + if publication == self.publication && manifest == self.manifest => + { + let cache = group.read()?; + if cache.group_manifest.as_ref() != Some(marker) + || cache.hard_state.commit < position.index.get() + { + return Err(RuntimeError::NotReady); + } + Ok(Some(CellGroupTransferActivationAck { + route: marker.route().clone(), + publication, + manifest, + position, + })) + } + Some(_) => Err(RuntimeError::StaleCompletion), + None => Ok(None), + } + } + + pub(crate) fn into_entry(self) -> Vec { + self.entry + } +} + +impl RangeDirectory { + /// Bind the child voter and its retained seed to the selected publication. + /// Admission requires a separate current-term barrier and committed apply. + pub async fn prepare_transfer_activation( + &self, + plan: &DirectoryMutation, + history: GroupHistory, + log: &ReplicaLog, + ) -> Result { + self.unmanaged_birth()?; + let bootstrap = self + .bootstrap_pending_transfer_voter(plan, history, log) + .await?; + self.prepare_transfer_activation_inner( + plan, + bootstrap.route(), + bootstrap.group().clone(), + log, + None, + ) + .await + } + + pub(crate) async fn prepare_managed_transfer_activation( + &self, + transition: &ReopenedManagedCellGroupTransition, + history: GroupHistory, + log: &ReplicaLog, + host: &FleetHostAuthorization, + deadline: u64, + ) -> Result { + let (_, children) = self.transition_routes(&transition.plan)?; + let route = children + .iter() + .find(|route| route.history == history) + .ok_or(CatalogError::Fenced)?; + let (group, _, boot) = self + .require_managed_transfer_voter( + transition, + route, + log, + host, + TransferVoterAdmission::Install, + deadline, + ) + .await?; + self.prepare_transfer_activation_inner( + &transition.plan, + route, + group, + log, + Some((boot, deadline)), + ) + .await + } + + async fn prepare_transfer_activation_inner( + &self, + plan: &DirectoryMutation, + route: &GroupRoute, + group: GroupLog, + log: &ReplicaLog, + managed: Option<(BootBinding, u64)>, + ) -> Result { + self.bounded(async { + if managed.is_some() { + let (view, _) = self.current().await?; + self.selected_route_in(&view, route).await?; + } + let publication = self + .publication(plan) + .await? + .filter(|publication| publication.transfer_archive().is_some()) + .ok_or(CatalogError::Fenced)?; + let marker = group + .group_manifest() + .map_err(CatalogError::Log)? + .ok_or(CatalogError::LostStableState)?; + let manifest = marker + .transfer_commitment() + .map_err(|_| CatalogError::Corrupt("transfer seed commitment"))? + .ok_or(CatalogError::Fenced)?; + let transition = TransitionId::new(plan.operation().bytes()) + .map_err(|_| CatalogError::Invalid("transfer operation"))?; + let publication = publication.journal_digest(); + let entry = CellGroupEntry { + binding: GroupEntryBinding { + group: route.history, + route_epoch: route.epoch, + birth_digest: route.birth_digest, + schema_digest: route.schema_digest, + proposal: plan.operation(), + }, + command: GroupCommand::Activate { + transition, + publication, + manifest, + }, + } + .encode(route, GroupEntryLimits::default()) + .map_err(|_| CatalogError::Invalid("transfer activation entry"))?; + let now = log + .domain() + .mono_ms() + .map_err(|_| CatalogError::Unconfirmed)?; + let limit = now + .checked_add(self.limits.operation_timeout_ms) + .ok_or(CatalogError::Limit)?; + let (boot, deadline) = managed.map_or((None, limit), |(boot, deadline)| { + (Some(boot), deadline.min(limit)) + }); + if deadline <= now { + return Err(CatalogError::Unconfirmed); + } + Ok(CellGroupTransferActivationPermit { + group, + engine: log.storage_instance().map_err(CatalogError::Log)?, + publication, + manifest, + deadline, + entry, + boot, + }) + }) + .await + } +} + +#[derive(Deserialize, Serialize)] +#[serde(deny_unknown_fields)] +struct ActivationSet { + namespace: [u8; 32], + plan: [u8; 32], + publication: [u8; 32], + installed: [u8; 32], + children: Vec, +} + +#[derive(Deserialize, Serialize)] +#[serde(deny_unknown_fields)] +struct StagedActivation { + namespace: [u8; 32], + plan: [u8; 32], + publication: [u8; 32], + installed: [u8; 32], + child: ActivationReceipt, +} + +#[derive(Clone, Deserialize, Serialize)] +#[serde(deny_unknown_fields)] +struct ActivationReceipt { + group: [u8; 16], + incarnation: [u8; 16], + manifest: [u8; 32], + index: u64, + term: u64, +} + +impl ActivationReceipt { + fn new(ack: &CellGroupTransferActivationAck) -> Self { + Self { + group: ack.route.history.group.bytes(), + incarnation: ack.route.history.incarnation.bytes(), + manifest: ack.manifest, + index: ack.position.index.get(), + term: ack.position.term.get(), + } + } + + fn validate(&self, installed: &GroupManifest) -> Result { + let history = installed.route().history; + let position = LogPosition { + index: LogIndex::new(self.index), + term: Term::new(self.term), + }; + let commitment = installed + .transfer_commitment() + .map_err(|_| CatalogError::Corrupt("installed child seed commitment"))?; + if self.group != history.group.bytes() + || self.incarnation != history.incarnation.bytes() + || commitment != Some(self.manifest) + || position.index <= installed.position().index + || position.term < installed.position().term + { + return Err(CatalogError::Fenced); + } + Ok(position) + } +} + +impl RangeDirectory { + /// Select a complete set of durable child activations in plan order. + /// Keep the topology lane owned until the parents can safely retire. + pub async fn record_transfer_activations( + &self, + plan: &DirectoryMutation, + acknowledgements: Vec, + ) -> Result { + self.unmanaged_birth()?; + self.record_transfer_activations_inner(plan, acknowledgements, None) + .await + } + + pub(crate) async fn record_managed_transfer_activations( + &self, + plan: &DirectoryMutation, + acknowledgements: Vec, + permit: &ManagedTransitionPermit, + ) -> Result { + permit.check(self, plan)?; + self.record_transfer_activations_inner(plan, acknowledgements, Some(permit)) + .await + } + + /// Persist a local applied acknowledgment and select Active only after + /// every installed child has an exact staged acknowledgment. + pub(crate) async fn resume_managed_transfer_activation( + &self, + plan: &DirectoryMutation, + history: GroupHistory, + acknowledgement: Option, + permit: &ManagedTransitionPermit, + ) -> Result, CatalogError> { + permit.check(self, plan)?; + let (journal, position, complete) = self + .bounded(async { + let (view, _) = self.current().await?; + let journal = self + .matching_transition(&view.head, plan) + .await? + .filter(|journal| { + matches!( + journal.phase(), + TransitionPhase::Published | TransitionPhase::Active + ) + }) + .ok_or(CatalogError::Fenced)?; + let (publication, installed) = self + .transfer_activation_dependencies(plan, &view.head, &journal) + .await?; + let markers = self + .installed_transfer_children_inner(plan, Some(permit)) + .await? + .ok_or(CatalogError::Fenced)?; + let marker = markers + .iter() + .find(|marker| marker.route().history == history) + .ok_or(CatalogError::Fenced)?; + if let Some(ack) = acknowledgement { + let receipt = checked_activation(&ack, marker, publication)?; + self.claim_staged_activation(plan, publication, installed, receipt) + .await?; + } + + let mut position = None; + let mut children = Vec::with_capacity(markers.len()); + for marker in &markers { + let child = marker.route().history; + let key = + staged_activation_key(plan, child.group.bytes(), child.incarnation.bytes()); + let Some(bytes) = self.read(&key).await? else { + continue; + }; + let staged: StagedActivation = + record::decode(STAGED_ACTIVATION, &bytes, wire::LIMIT)?; + if staged.namespace != self.namespace.root_digest + || staged.plan != plan.digest() + || staged.publication != publication + || staged.installed != installed + { + return Err(CatalogError::Corrupt("staged transfer activation binding")); + } + let applied = staged.child.validate(marker)?; + if child == history { + position = Some(applied); + } + children.push(staged.child); + } + let complete = if children.len() == markers.len() { + let digest = self + .store_activation_set(plan, publication, installed, children) + .await?; + Some((publication, installed, digest)) + } else { + None + }; + Ok((journal, position, complete)) + }) + .await?; + let Some(position) = position else { + return Ok(None); + }; + let journal = if let Some((publication, installed, digest)) = complete { + self.record_verified_transfer_activations( + plan, + publication, + installed, + digest, + Some(permit), + ) + .await? + } else { + journal + }; + Ok(Some((position, journal))) + } + + async fn claim_staged_activation( + &self, + plan: &DirectoryMutation, + publication: [u8; 32], + installed: [u8; 32], + child: ActivationReceipt, + ) -> Result<(), CatalogError> { + let key = self.key(&staged_activation_key(plan, child.group, child.incarnation)); + let value = StagedActivation { + namespace: self.namespace.root_digest, + plan: plan.digest(), + publication, + installed, + child, + }; + let bytes = record::encode(STAGED_ACTIVATION, &value, wire::LIMIT)?; + let observed = self + .namespace + .inner + .create_and_observe_bounded(&key, &bytes, wire::LIMIT) + .await?; + if observed != bytes { + return Err(CatalogError::Conflict("staged transfer activation changed")); + } + Ok(()) + } + + async fn store_activation_set( + &self, + plan: &DirectoryMutation, + publication: [u8; 32], + installed: [u8; 32], + children: Vec, + ) -> Result<[u8; 32], CatalogError> { + let set = ActivationSet { + namespace: self.namespace.root_digest, + plan: plan.digest(), + publication, + installed, + children, + }; + let bytes = record::encode(ACTIVATIONS, &set, wire::LIMIT)?; + let digest = record::digest(&bytes); + self.immutable(SETS, digest, &bytes).await?; + Ok(digest) + } + + async fn record_transfer_activations_inner( + &self, + plan: &DirectoryMutation, + acknowledgements: Vec, + permit: Option<&ManagedTransitionPermit>, + ) -> Result { + self.check_transition_plan(plan)?; + if acknowledgements.len() != plan.request.children.len() { + return Err(CatalogError::Invalid("incomplete transfer activation set")); + } + let (publication, installed, digest) = self + .bounded(async { + let (view, _) = self.current().await?; + let journal = self + .matching_transition(&view.head, plan) + .await? + .ok_or(CatalogError::Fenced)?; + let (publication, installed) = self + .transfer_activation_dependencies(plan, &view.head, &journal) + .await?; + let markers = self + .installed_transfer_children_inner(plan, permit) + .await? + .ok_or(CatalogError::Fenced)?; + let children = acknowledgements + .iter() + .zip(&markers) + .map(|(ack, marker)| checked_activation(ack, marker, publication)) + .collect::, _>>()?; + if permit.is_some() && journal.phase() == TransitionPhase::Published { + for receipt in &children { + self.claim_staged_activation(plan, publication, installed, receipt.clone()) + .await?; + } + } + let digest = self + .store_activation_set(plan, publication, installed, children) + .await?; + Ok((publication, installed, digest)) + }) + .await?; + self.record_verified_transfer_activations(plan, publication, installed, digest, permit) + .await + } + + /// Reopen the historical applied positions selected by the Active CAS. + pub async fn transfer_activations( + &self, + plan: &DirectoryMutation, + ) -> Result>, CatalogError> { + self.unmanaged_birth()?; + self.transfer_activations_inner(plan, None).await + } + + pub(super) async fn transfer_activations_inner( + &self, + plan: &DirectoryMutation, + permit: Option<&ManagedTransitionPermit>, + ) -> Result>, CatalogError> { + if let Some(permit) = permit { + permit.check(self, plan)?; + } + self.check_transition_plan(plan)?; + self.bounded(async { + let (view, _) = self.current().await?; + let journal = self + .matching_transition(&view.head, plan) + .await? + .ok_or(CatalogError::Fenced)?; + let Some(digest) = journal.activated_children() else { + return Ok(None); + }; + let bytes = self + .read(&format!("{SETS}/{}", hex::encode(digest))) + .await? + .ok_or(CatalogError::Corrupt("missing transfer activation set"))?; + if record::digest(&bytes) != digest { + return Err(CatalogError::Corrupt("transfer activation set digest")); + } + let set: ActivationSet = record::decode(ACTIVATIONS, &bytes, wire::LIMIT)?; + let (publication, installed) = self + .transfer_activation_dependencies(plan, &view.head, &journal) + .await?; + if set.namespace != self.namespace.root_digest + || set.plan != plan.digest() + || set.publication != publication + || set.installed != installed + || set.children.len() != plan.request.children.len() + { + return Err(CatalogError::Corrupt("transfer activation set binding")); + } + let markers = self + .installed_transfer_children_inner(plan, permit) + .await? + .ok_or(CatalogError::Corrupt("missing activated child install set"))?; + set.children + .into_iter() + .zip(markers) + .map(|(receipt, marker)| Ok((marker.route().history, receipt.validate(&marker)?))) + .collect::, CatalogError>>() + .map(Some) + }) + .await + } + + async fn transfer_activation_dependencies( + &self, + plan: &DirectoryMutation, + head: &wire::Head, + journal: &TransitionJournal, + ) -> Result<([u8; 32], [u8; 32]), CatalogError> { + let publication = self + .confirmed(head, plan) + .await? + .filter(|publication| { + publication.transfer_archive() == journal.transfer_archive() + && publication.transfer_archive().is_some() + }) + .ok_or(CatalogError::Fenced)?; + Ok(( + publication.journal_digest(), + journal.installed_children().ok_or(CatalogError::Fenced)?, + )) + } +} + +fn checked_activation( + ack: &CellGroupTransferActivationAck, + installed: &GroupManifest, + publication: [u8; 32], +) -> Result { + if ack.publication != publication || &ack.route != installed.route() { + return Err(CatalogError::Fenced); + } + let receipt = ActivationReceipt::new(ack); + receipt.validate(installed)?; + Ok(receipt) +} + +fn staged_activation_key( + plan: &DirectoryMutation, + group: [u8; 16], + incarnation: [u8; 16], +) -> String { + format!( + "transfer-activation-progress/{}/{}/{}", + hex::encode(plan.digest()), + hex::encode(group), + hex::encode(incarnation) + ) +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/transition_archive.rs b/crates/cellar/durability/raft/cell_groups/directory/transition_archive.rs new file mode 100644 index 000000000..26da22ed6 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/transition_archive.rs @@ -0,0 +1,300 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! A remote transfer closure references the exact held parent data and the +//! independently reconstructed child roots selected by the installation CAS. +use super::{transition_sources::TransitionSourceCut, *}; +use crate::raft::cell_groups::{ + CellGroupArchiveError, CellGroupArchiveExecution, CellGroupArchiveHold, CellGroupArchivePool, + GroupManifest, +}; +use crate::raft::fleet_catalog::ManagedTransitionPermit; +use serde::{Deserialize, Serialize}; + +const CLOSURE: u8 = 207; +const CLOSURES: &str = "transfer-archive-closures"; + +#[derive(Deserialize, Serialize)] +#[serde(deny_unknown_fields)] +struct Closure { + namespace: [u8; 32], + plan: [u8; 32], + sealed_sources: [u8; 32], + installed_children: [u8; 32], +} + +/// Immutable archive references selected by the directory. Recovery must +/// independently replay the holds before using their data. +pub struct CellGroupTransferArchive { + pub digest: [u8; 32], + pub source_holds: Vec, + pub children: Vec, +} + +impl RangeDirectory { + /// Independently recover every held parent archive and rebuild the complete + /// child partition in dedicated verification stores. Persist the closure + /// only if the rebuilt roots match the installed children exactly. Keep + /// the original remote holds until the transition owner retires them. + pub async fn verify_and_record_transfer_archive( + &self, + domain: &impl CellGroupArchiveExecution, + plan: &DirectoryMutation, + inputs: CellGroupTransitionPreparationInputs, + ) -> Result { + self.unmanaged_birth()?; + self.verify_and_record_transfer_archive_inner(domain, plan, inputs, None) + .await + } + + pub(crate) async fn verify_and_record_managed_transfer_archive( + &self, + domain: &impl CellGroupArchiveExecution, + plan: &DirectoryMutation, + inputs: CellGroupTransitionPreparationInputs, + permit: &ManagedTransitionPermit, + ) -> Result { + permit.check(self, plan)?; + self.verify_and_record_transfer_archive_inner(domain, plan, inputs, Some(permit)) + .await + } + + async fn verify_and_record_transfer_archive_inner( + &self, + domain: &impl CellGroupArchiveExecution, + plan: &DirectoryMutation, + inputs: CellGroupTransitionPreparationInputs, + permit: Option<&ManagedTransitionPermit>, + ) -> Result { + domain + .run(Box::pin(async { + let children = self + .installed_transfer_children_inner(plan, permit) + .await? + .ok_or(CatalogError::Fenced)?; + let holds = inputs + .sources + .iter() + .map(|source| (source.birth.clone(), source.hold.clone())) + .collect::>(); + let sources = Box::pin(self.verify_and_record_sealed_sources_inner( + domain, + &inputs.archives, + &inputs.tails, + plan, + inputs.sources, + permit, + )) + .await?; + let journal = sources.journal(); + let sealed_sources = journal.sealed_sources().ok_or(CatalogError::Fenced)?; + let installed_children = + journal.installed_children().ok_or(CatalogError::Fenced)?; + let partition = inputs + .restore + .prepare_transfer_partition(domain, sources, inputs.targets) + .await?; + if partition.children().len() != children.len() + || partition + .children() + .iter() + .zip(&children) + .any(|(rebuilt, installed)| { + rebuilt.genesis().is_none_or(|genesis| { + let marker = genesis.marker(); + marker.route() != installed.route() + || !installed.preserves_transfer(marker) + }) + }) + { + return Err(CatalogError::Conflict("rebuilt transfer child root").into()); + } + + let closure = Closure { + namespace: self.namespace.root_digest, + plan: plan.digest(), + sealed_sources, + installed_children, + }; + let bytes = record::encode(CLOSURE, &closure, wire::LIMIT)?; + let digest = record::digest(&bytes); + self.bounded(self.immutable(CLOSURES, digest, &bytes)) + .await?; + for (birth, hold) in holds { + let registry = self + .open_group_archive(domain, &inputs.archives, &birth) + .await?; + registry.require_retained_hold(domain, &hold).await?; + } + self.record_verified_transfer_archive( + plan, + sealed_sources, + installed_children, + digest, + permit, + ) + .await + .map_err(Into::into) + })) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } + + /// Atomically select the archived child ranges. An exact retry confirms + /// the same publication, including when the first CAS response was lost. + /// Children remain fenced until their own replicated activation. + pub async fn publish_transfer( + &self, + domain: &impl CellGroupArchiveExecution, + archives: &CellGroupArchivePool, + plan: &DirectoryMutation, + ) -> Result { + self.unmanaged_birth()?; + self.publish_transfer_inner(domain, archives, plan, None) + .await + } + + pub(crate) async fn publish_managed_transfer( + &self, + domain: &impl CellGroupArchiveExecution, + archives: &CellGroupArchivePool, + plan: &DirectoryMutation, + permit: &ManagedTransitionPermit, + ) -> Result { + permit.check(self, plan)?; + self.publish_transfer_inner(domain, archives, plan, Some(permit)) + .await + } + + async fn publish_transfer_inner( + &self, + domain: &impl CellGroupArchiveExecution, + archives: &CellGroupArchivePool, + plan: &DirectoryMutation, + permit: Option<&ManagedTransitionPermit>, + ) -> Result { + self.check_transition_plan(plan)?; + domain + .run(Box::pin(async { + if permit.is_none() { + self.namespace.require_cell_group_unmanaged().await?; + } + if let Some(publication) = self.confirmed_transfer_publication(plan).await? { + return Ok(publication); + } + let attempt = async { + let closure = self + .transfer_archive_inner(plan, permit) + .await? + .ok_or(CatalogError::Fenced)?; + for hold in &closure.source_holds { + let birth = self + .materialize_selected_history(hold.marker().route().history) + .await?; + let registry = self.open_group_archive(domain, archives, &birth).await?; + registry.require_retained_hold(domain, hold).await?; + } + self.bounded( + self.publish_inner(plan, PublicationAuthority::Transfer(closure.digest)), + ) + .await + .map_err(Into::into) + } + .await; + match attempt { + Ok(publication) => Ok(publication), + // Another controller may have published this exact plan + // while this attempt was reopening the parent archives. + Err(error) => match self.confirmed_transfer_publication(plan).await { + Ok(Some(publication)) => Ok(publication), + _ => Err(error), + }, + } + })) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } + + pub(super) async fn confirmed_transfer_publication( + &self, + plan: &DirectoryMutation, + ) -> Result, CatalogError> { + let publication = self.publication(plan).await?; + if publication + .as_ref() + .is_some_and(|publication| publication.transfer_archive().is_none()) + { + return Err(CatalogError::Fenced); + } + Ok(publication) + } + + /// Reopen the exact closure selected by an earlier CAS, including after a + /// lost response. This reads remote references without opening voter disks. + pub async fn transfer_archive( + &self, + plan: &DirectoryMutation, + ) -> Result, CatalogError> { + self.unmanaged_birth()?; + self.transfer_archive_inner(plan, None).await + } + + pub(crate) async fn managed_transfer_archive( + &self, + plan: &DirectoryMutation, + permit: &ManagedTransitionPermit, + ) -> Result, CatalogError> { + permit.check(self, plan)?; + self.transfer_archive_inner(plan, Some(permit)).await + } + + async fn transfer_archive_inner( + &self, + plan: &DirectoryMutation, + permit: Option<&ManagedTransitionPermit>, + ) -> Result, CatalogError> { + self.check_transition_plan(plan)?; + self.bounded(async { + let (view, _) = self.current().await?; + let journal = self + .matching_transition(&view.head, plan) + .await? + .ok_or(CatalogError::Fenced)?; + let Some(digest) = journal.transfer_archive() else { + return Ok(None); + }; + let bytes = self + .read(&format!("{CLOSURES}/{}", hex::encode(digest))) + .await? + .ok_or(CatalogError::Corrupt("missing transfer archive closure"))?; + if record::digest(&bytes) != digest { + return Err(CatalogError::Corrupt("transfer archive closure digest")); + } + let closure: Closure = record::decode(CLOSURE, &bytes, wire::LIMIT)?; + if closure.namespace != self.namespace.root_digest + || closure.plan != plan.digest() + || journal.sealed_sources() != Some(closure.sealed_sources) + || journal.installed_children() != Some(closure.installed_children) + { + return Err(CatalogError::Corrupt("transfer archive closure binding")); + } + let source_holds = self + .read_transition_source_holds( + plan, + &journal, + closure.sealed_sources, + TransitionSourceCut::Sealed, + ) + .await?; + let children = self + .installed_transfer_children_inner(plan, permit) + .await? + .ok_or(CatalogError::Corrupt("missing archived child install set"))?; + Ok(Some(CellGroupTransferArchive { + digest, + source_holds, + children, + })) + }) + .await + } +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/transition_child_archive.rs b/crates/cellar/durability/raft/cell_groups/directory/transition_child_archive.rs new file mode 100644 index 000000000..46db3fce0 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/transition_child_archive.rs @@ -0,0 +1,433 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::{transition_source_slots::ArchiveSlot, *}; +use crate::raft::cell_groups::{ + CellGroupArchiveError, CellGroupArchiveExecution, CellGroupArchiveHold, CellGroupArchivePool, + CellGroupArchiveTailPool, GroupManifest, +}; +use crate::raft::fleet_catalog::ManagedTransitionPermit; +use cellar_logic::consensus::LogPosition; +use serde::{Deserialize, Serialize}; + +const HOLD: u8 = 209; +const SET: u8 = 210; +const HOLDS: &str = "transfer-child-archive-holds"; +const SETS: &str = "transfer-child-archive-sets"; + +#[derive(Deserialize, Serialize)] +#[serde(deny_unknown_fields)] +struct ChildArchiveSet { + namespace: [u8; 32], + plan: [u8; 32], + publication: [u8; 32], + activations: [u8; 32], + holds: Vec<[u8; 32]>, +} + +struct ChildArchiveBasis { + publication: [u8; 32], + activations: [u8; 32], + installed: Vec, + applied: Vec<(GroupHistory, LogPosition)>, + selected: Option<[u8; 32]>, +} + +impl ChildArchiveBasis { + fn check_marker(&self, index: usize, marker: &GroupManifest) -> Result<(), CatalogError> { + let installed = &self.installed[index]; + let (history, position) = self.applied[index]; + if marker.route() != installed.route() + || marker.route().history != history + || !marker.is_active() + || !marker.preserves_transfer(installed) + || marker + .transfer_activation() + .is_none_or(|(publication, _, applied)| { + publication != self.publication || applied != position + }) + { + return Err(CatalogError::Fenced); + } + Ok(()) + } +} + +impl RangeDirectory { + /// Independently replay an exact archive hold for every active child. + /// The selected coverage protects the transfer recovery boundary. Parent + /// holds and the topology lane remain owned until retirement settlement. + pub async fn verify_and_record_child_archives( + &self, + domain: &impl CellGroupArchiveExecution, + pool: &CellGroupArchivePool, + tails: &CellGroupArchiveTailPool, + plan: &DirectoryMutation, + children: Vec, + ) -> Result { + self.unmanaged_birth()?; + self.verify_and_record_child_archives_inner(domain, pool, tails, plan, children, None) + .await + } + + pub(crate) async fn verify_and_record_managed_child_archives( + &self, + domain: &impl CellGroupArchiveExecution, + pool: &CellGroupArchivePool, + tails: &CellGroupArchiveTailPool, + plan: &DirectoryMutation, + children: Vec, + permit: &ManagedTransitionPermit, + ) -> Result { + permit.check(self, plan)?; + self.verify_and_record_child_archives_inner( + domain, + pool, + tails, + plan, + children, + Some(permit), + ) + .await + } + + async fn verify_and_record_child_archives_inner( + &self, + domain: &impl CellGroupArchiveExecution, + pool: &CellGroupArchivePool, + tails: &CellGroupArchiveTailPool, + plan: &DirectoryMutation, + children: Vec, + permit: Option<&ManagedTransitionPermit>, + ) -> Result { + self.check_transition_plan(plan)?; + if children.len() != plan.request.children.len() { + return Err(CatalogError::Invalid("incomplete transfer child archives").into()); + } + domain + .run(Box::pin(async { + if permit.is_none() { + self.namespace.require_cell_group_unmanaged().await?; + } + let basis = self.bounded(self.child_archive_basis(plan, permit)).await?; + let mut hold_records = Vec::with_capacity(children.len()); + for (index, child) in children.iter().enumerate() { + if child.birth.route() != basis.installed[index].route() { + return Err(CatalogError::Fenced.into()); + } + child.hold.check_reference()?; + basis.check_marker(index, child.hold.marker())?; + let bytes = record::encode(HOLD, &child.hold, wire::LIMIT)?; + hold_records.push((record::digest(&bytes), bytes)); + } + let set = ChildArchiveSet { + namespace: self.namespace.root_digest, + plan: plan.digest(), + publication: basis.publication, + activations: basis.activations, + holds: hold_records.iter().map(|(digest, _)| *digest).collect(), + }; + let bytes = record::encode(SET, &set, wire::LIMIT)?; + let digest = record::digest(&bytes); + if basis.selected.is_some_and(|selected| selected != digest) { + return Err(CatalogError::Conflict("transfer child archive set changed").into()); + } + + let mut retained = Vec::with_capacity(children.len()); + for (index, child) in children.into_iter().enumerate() { + let registry = self.open_group_archive(domain, pool, &child.birth).await?; + let verified = registry + .verify_hold(domain, &child.hold, tails, child.receiver) + .await?; + basis.check_marker(index, verified.marker())?; + retained.push((registry, child.hold)); + } + self.bounded(async { + for (id, hold) in hold_records { + self.immutable(HOLDS, id, &hold).await?; + } + self.immutable(SETS, digest, &bytes).await + }) + .await?; + for (registry, hold) in retained { + registry.require_retained_hold(domain, &hold).await?; + } + self.record_verified_child_archives( + plan, + basis.publication, + basis.activations, + digest, + permit, + ) + .await + .map_err(Into::into) + })) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } + + /// Acquire one restart-reconcilable hold per child in plan order. Only + /// the independent physical verifier can select the complete archive set. + pub(crate) async fn acquire_managed_child_archives( + &self, + domain: &impl CellGroupArchiveExecution, + pool: &CellGroupArchivePool, + plan: &DirectoryMutation, + permit: &ManagedTransitionPermit, + ) -> Result, CellGroupArchiveBindingError> { + permit.check(self, plan)?; + let basis = self + .bounded(self.child_archive_basis(plan, Some(permit))) + .await?; + let selected = match basis.selected { + Some(digest) => Some( + self.bounded(self.load_child_archive_holds(plan, &basis, digest)) + .await?, + ), + None => None, + }; + let mut sources = Vec::with_capacity(basis.installed.len()); + for (index, marker) in basis.installed.iter().enumerate() { + let route = marker.route(); + let birth = self.materialize_selected_history(route.history).await?; + if birth.route() != route { + return Err(CatalogError::Fenced.into()); + } + let hold = if let Some(selected) = &selected { + selected[index].clone() + } else { + let key = format!( + "transfer-child-archive-acquires/{}/{index}", + hex::encode(plan.digest()) + ); + let staged = match self.bounded(self.read_archive_slot(&key)).await? { + Some((ArchiveSlot::Acquiring(intent), version)) => Some((intent, version)), + Some((ArchiveSlot::Verified(_), _)) => { + return Err(CatalogError::Corrupt("transfer child acquire slot").into()); + } + None => None, + }; + let registry = self.open_group_archive(domain, pool, &birth).await?; + let (hold, _) = self + .acquire_transition_archive_hold(domain, ®istry, &key, staged, |marker| { + basis + .check_marker(index, marker) + .map_err(|error| match error { + CatalogError::Fenced + if marker.route() == route && !marker.is_active() => + { + CatalogError::Unconfirmed.into() + } + error => error.into(), + }) + }) + .await?; + hold + }; + hold.check_reference()?; + basis.check_marker(index, hold.marker())?; + sources.push((birth, hold)); + } + Ok(sources) + } + + /// Reopen the exact selected child holds for recovery or retirement. + /// Physical replay and registry liveness still require their own checks. + pub async fn retirement_child_holds( + &self, + plan: &DirectoryMutation, + ) -> Result>, CatalogError> { + self.unmanaged_birth()?; + self.retirement_child_holds_inner(plan, None).await + } + + pub(crate) async fn managed_retirement_child_holds( + &self, + plan: &DirectoryMutation, + permit: &ManagedTransitionPermit, + ) -> Result>, CatalogError> { + permit.check(self, plan)?; + self.retirement_child_holds_inner(plan, Some(permit)).await + } + + async fn retirement_child_holds_inner( + &self, + plan: &DirectoryMutation, + permit: Option<&ManagedTransitionPermit>, + ) -> Result>, CatalogError> { + self.check_transition_plan(plan)?; + self.bounded(async { + let basis = self.child_archive_basis(plan, permit).await?; + let Some(digest) = basis.selected else { + return Ok(None); + }; + self.load_child_archive_holds(plan, &basis, digest) + .await + .map(Some) + }) + .await + } + + async fn load_child_archive_holds( + &self, + plan: &DirectoryMutation, + basis: &ChildArchiveBasis, + digest: [u8; 32], + ) -> Result, CatalogError> { + let bytes = self.read_child_archive_record(SETS, digest).await?; + let set: ChildArchiveSet = record::decode(SET, &bytes, wire::LIMIT)?; + if set.namespace != self.namespace.root_digest + || set.plan != plan.digest() + || set.publication != basis.publication + || set.activations != basis.activations + || set.holds.len() != basis.installed.len() + { + return Err(CatalogError::Corrupt("transfer child archive set binding")); + } + let mut holds = Vec::with_capacity(set.holds.len()); + for (index, id) in set.holds.into_iter().enumerate() { + let bytes = self.read_child_archive_record(HOLDS, id).await?; + let hold: CellGroupArchiveHold = record::decode(HOLD, &bytes, wire::LIMIT)?; + hold.check_reference() + .map_err(|_| CatalogError::Corrupt("transfer child archive hold"))?; + basis.check_marker(index, hold.marker())?; + holds.push(hold); + } + Ok(holds) + } + + /// Root the selected archive identity before its one-use initializer + /// writes counters. A new controller can reopen a completed setup. An + /// incomplete registry still requires archive authority recovery. + pub(crate) async fn initialize_managed_child_archive( + &self, + domain: &impl CellGroupArchiveExecution, + pool: &CellGroupArchivePool, + plan: &DirectoryMutation, + history: GroupHistory, + permit: &ManagedTransitionPermit, + ) -> Result<(), CellGroupArchiveBindingError> { + let (birth, coverage_selected) = self + .managed_transfer_child_archive_birth(plan, history, permit) + .await?; + let key = format!( + "transfer-child-archive-initialization/{}/{}", + hex::encode(plan.digest()), + hex::encode(history.group.bytes()), + ); + if let Some(intent) = self.bounded(self.read(&key)).await? { + return self + .verify_group_archive_binding(domain, pool, &birth, &intent) + .await; + } + if coverage_selected || self.has_group_archive_binding(&birth).await? { + return Err(CatalogError::Conflict("transfer child archive initialization").into()); + } + let prepared = self + .prepare_group_archive_binding(domain, pool, &birth) + .await?; + let selected = match self + .bounded(self.namespace.inner.create_and_observe_bounded( + &self.key(&key), + prepared.encode(), + wire::LIMIT, + )) + .await + { + Ok(selected) => selected, + Err(error @ (CatalogError::Storage(_) | CatalogError::Unconfirmed)) => { + // The create may have succeeded before its confirming read + // failed. Only the current owner still holds the initializer. + self.bounded(self.read(&key)).await?.ok_or(error)? + } + Err(error) => return Err(error.into()), + }; + if selected == prepared.encode() { + prepared.commit(domain).await?; + } else { + self.verify_group_archive_binding(domain, pool, &birth, &selected) + .await?; + } + Ok(()) + } + + pub(crate) async fn managed_transfer_child_archive_birth( + &self, + plan: &DirectoryMutation, + history: GroupHistory, + permit: &ManagedTransitionPermit, + ) -> Result<(CellGroupBirth, bool), CatalogError> { + permit.check(self, plan)?; + self.bounded(async { + let basis = self.child_archive_basis(plan, Some(permit)).await?; + let installed = basis + .installed + .iter() + .find(|marker| marker.route().history == history) + .ok_or(CatalogError::Fenced)?; + let birth = self.materialize_selected_history(history).await?; + if birth.route() != installed.route() { + return Err(CatalogError::Fenced); + } + Ok((birth, basis.selected.is_some())) + }) + .await + } + + async fn child_archive_basis( + &self, + plan: &DirectoryMutation, + permit: Option<&ManagedTransitionPermit>, + ) -> Result { + let (view, _) = self.current().await?; + let journal = self + .matching_transition(&view.head, plan) + .await? + .filter(|journal| { + matches!( + journal.phase(), + TransitionPhase::Active | TransitionPhase::Retired + ) + }) + .ok_or(CatalogError::Fenced)?; + let publication = self + .confirmed(&view.head, plan) + .await? + .filter(|publication| publication.transfer_archive().is_some()) + .ok_or(CatalogError::Fenced)? + .journal_digest(); + let activations = journal.activated_children().ok_or(CatalogError::Fenced)?; + let installed = self + .installed_transfer_children_inner(plan, permit) + .await? + .ok_or(CatalogError::Fenced)?; + let applied = self + .transfer_activations_inner(plan, permit) + .await? + .ok_or(CatalogError::Fenced)?; + if installed.len() != plan.request.children.len() || applied.len() != installed.len() { + return Err(CatalogError::Corrupt("transfer child archive basis")); + } + Ok(ChildArchiveBasis { + publication, + activations, + installed, + applied, + selected: journal.child_archives(), + }) + } + + async fn read_child_archive_record( + &self, + category: &str, + digest: [u8; 32], + ) -> Result, CatalogError> { + let bytes = self + .read(&format!("{category}/{}", hex::encode(digest))) + .await? + .ok_or(CatalogError::Corrupt("missing child archive record"))?; + if record::digest(&bytes) != digest { + return Err(CatalogError::Corrupt("child archive record digest")); + } + Ok(bytes) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/transition_child_quorum.rs b/crates/cellar/durability/raft/cell_groups/directory/transition_child_quorum.rs new file mode 100644 index 000000000..023180780 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/transition_child_quorum.rs @@ -0,0 +1,369 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use crate::raft::{ + cell_groups::{GroupManifest, TransferChildQuorum}, + fleet_catalog::ManagedTransitionPermit, +}; +use cellar_logic::consensus::LogPosition; +use serde::{Deserialize, Serialize}; + +const CHILD: u8 = 205; +const SET: u8 = 206; +const CHILDREN: &str = "transfer-child-quorums"; +const SETS: &str = "transfer-child-quorum-sets"; +const STAGED: &str = "transfer-child-quorum-progress"; + +#[derive(Deserialize, Serialize)] +#[serde(deny_unknown_fields)] +struct ChildQuorum { + marker: GroupManifest, + member: u64, + node: [u8; 16], + disk: [u8; 16], + engine: [u8; 16], +} + +impl From for ChildQuorum { + fn from(proof: TransferChildQuorum) -> Self { + let (marker, leader) = proof.into_parts(); + Self { + marker, + member: leader.member.get(), + node: leader.node.bytes(), + disk: leader.disk.bytes(), + engine: leader.engine.bytes(), + } + } +} + +#[derive(Deserialize, Serialize)] +#[serde(deny_unknown_fields)] +struct ChildQuorumSet { + namespace: [u8; 32], + request: [u8; 32], + children: Vec<[u8; 32]>, +} + +impl RangeDirectory { + /// Root one durable fresh-term witness for every child in plan order. + /// The selected set records installation. The parent routes stay selected. + pub async fn record_transfer_child_quorums( + &self, + plan: &DirectoryMutation, + proofs: Vec, + ) -> Result { + self.unmanaged_birth()?; + self.record_transfer_child_quorums_inner(plan, proofs, None) + .await + } + + pub(crate) async fn record_managed_transfer_child_quorums( + &self, + plan: &DirectoryMutation, + proofs: Vec, + permit: &ManagedTransitionPermit, + ) -> Result { + permit.check(self, plan)?; + self.record_transfer_child_quorums_inner(plan, proofs, Some(permit)) + .await + } + + async fn record_transfer_child_quorums_inner( + &self, + plan: &DirectoryMutation, + proofs: Vec, + permit: Option<&ManagedTransitionPermit>, + ) -> Result { + let (_, children) = self.transition_routes(plan)?; + if proofs.len() != children.len() { + return Err(CatalogError::Invalid( + "incomplete transfer child quorum set", + )); + } + let digest = self + .bounded(async { + let mut selected = Vec::with_capacity(children.len()); + for (route, proof) in children.iter().zip(proofs) { + let proof = ChildQuorum::from(proof); + selected.push( + self.store_checked_child_quorum(plan, route, &proof, permit) + .await?, + ); + } + // Validate the whole input before claiming any managed slot. + if permit.is_some() { + let (view, _) = self.current().await?; + let journal = self + .matching_transition(&view.head, plan) + .await? + .ok_or(CatalogError::Fenced)?; + if journal.phase() == TransitionPhase::Sealed { + for (index, digest) in selected.iter().enumerate() { + let winner = self.claim_child_quorum(plan, index, *digest).await?; + if winner != *digest { + return Err(CatalogError::Conflict( + "staged transfer child quorum changed", + )); + } + } + } + } + self.store_child_quorum_set(plan, selected).await + }) + .await?; + self.record_verified_child_quorums(plan, digest, permit) + .await + } + + /// Record or reopen one exact child witness. The complete ordered set is + /// selected only after every slot and the current Fleet claims verify. + pub(crate) async fn resume_managed_transfer_child_quorum( + &self, + plan: &DirectoryMutation, + history: GroupHistory, + proof: Option, + permit: &ManagedTransitionPermit, + all_installations_current: bool, + ) -> Result, CatalogError> { + permit.check(self, plan)?; + let (_, children) = self.transition_routes(plan)?; + let index = children + .iter() + .position(|route| route.history == history) + .ok_or(CatalogError::Fenced)?; + let (journal, position, complete) = self + .bounded(async { + let (view, _) = self.current().await?; + let journal = self + .matching_transition(&view.head, plan) + .await? + .ok_or(CatalogError::Fenced)?; + if let Some(digest) = journal.installed_children() { + let set = self + .read_child_quorum_set(plan, digest, children.len()) + .await?; + let proof = self.load_child_quorum(set.children[index]).await?; + self.check_child_quorum(plan, &children[index], &proof, Some(permit)) + .await?; + return Ok((journal, Some(proof.marker.position()), None)); + } + if journal.phase() != TransitionPhase::Sealed { + return Err(CatalogError::Fenced); + } + if let Some(proof) = proof { + let proof = ChildQuorum::from(proof); + let digest = self + .store_checked_child_quorum(plan, &children[index], &proof, Some(permit)) + .await?; + // Another ready leader can have supplied the original + // historical witness while this controller was observing. + self.claim_child_quorum(plan, index, digest).await?; + } + let mut position = None; + let mut selected = Vec::with_capacity(children.len()); + for (slot, route) in children.iter().enumerate() { + let Some(bytes) = self.read(&staged_key(plan, slot)).await? else { + continue; + }; + let digest = bytes.try_into().map_err(|_| { + CatalogError::Corrupt("staged transfer child quorum digest") + })?; + let proof = self.load_child_quorum(digest).await?; + self.check_child_quorum(plan, route, &proof, Some(permit)) + .await?; + if slot == index { + position = Some(proof.marker.position()); + } + selected.push(digest); + } + let complete = if all_installations_current && selected.len() == children.len() { + Some(self.store_child_quorum_set(plan, selected).await?) + } else { + None + }; + Ok((journal, position, complete)) + }) + .await?; + let Some(position) = position else { + return Ok(None); + }; + let journal = if let Some(digest) = complete { + self.record_verified_child_quorums(plan, digest, Some(permit)) + .await? + } else { + journal + }; + Ok(Some((position, journal))) + } + + async fn claim_child_quorum( + &self, + plan: &DirectoryMutation, + index: usize, + digest: [u8; 32], + ) -> Result<[u8; 32], CatalogError> { + self.namespace + .inner + .create_and_observe_bounded(&self.key(&staged_key(plan, index)), &digest, 32) + .await? + .try_into() + .map_err(|_| CatalogError::Corrupt("staged transfer child quorum digest")) + } + + async fn store_checked_child_quorum( + &self, + plan: &DirectoryMutation, + route: &GroupRoute, + proof: &ChildQuorum, + permit: Option<&ManagedTransitionPermit>, + ) -> Result<[u8; 32], CatalogError> { + self.check_child_quorum(plan, route, proof, permit).await?; + let bytes = record::encode(CHILD, proof, wire::LIMIT)?; + let digest = record::digest(&bytes); + self.immutable(CHILDREN, digest, &bytes).await?; + Ok(digest) + } + + async fn store_child_quorum_set( + &self, + plan: &DirectoryMutation, + children: Vec<[u8; 32]>, + ) -> Result<[u8; 32], CatalogError> { + let set = ChildQuorumSet { + namespace: self.namespace.root_digest, + request: plan.digest(), + children, + }; + let bytes = record::encode(SET, &set, wire::LIMIT)?; + let digest = record::digest(&bytes); + self.immutable(SETS, digest, &bytes).await?; + Ok(digest) + } + + /// Recover the exact child markers selected by the installation CAS. + /// Revalidates the physical install bindings and keeps admission fenced. + pub async fn installed_transfer_children( + &self, + plan: &DirectoryMutation, + ) -> Result>, CatalogError> { + self.installed_transfer_children_inner(plan, None).await + } + + pub(super) async fn installed_transfer_children_inner( + &self, + plan: &DirectoryMutation, + permit: Option<&ManagedTransitionPermit>, + ) -> Result>, CatalogError> { + match permit { + Some(permit) => permit.check(self, plan)?, + None => self.unmanaged_birth()?, + } + let (_, children) = self.transition_routes(plan)?; + self.bounded(async { + let (view, _) = self.current().await?; + let journal = self + .matching_transition(&view.head, plan) + .await? + .ok_or(CatalogError::Fenced)?; + let Some(digest) = journal.installed_children() else { + return Ok(None); + }; + let set = self + .read_child_quorum_set(plan, digest, children.len()) + .await?; + let mut markers = Vec::with_capacity(children.len()); + for (route, digest) in children.iter().zip(set.children) { + let proof = self.load_child_quorum(digest).await?; + self.check_child_quorum(plan, route, &proof, permit).await?; + markers.push(proof.marker); + } + Ok(Some(markers)) + }) + .await + } + + async fn read_child_quorum_set( + &self, + plan: &DirectoryMutation, + digest: [u8; 32], + child_count: usize, + ) -> Result { + let bytes = self.read_child_quorum_record(SETS, digest).await?; + let set: ChildQuorumSet = record::decode(SET, &bytes, wire::LIMIT)?; + if set.namespace != self.namespace.root_digest + || set.request != plan.digest() + || set.children.len() != child_count + { + return Err(CatalogError::Corrupt("transfer child quorum set binding")); + } + Ok(set) + } + + async fn load_child_quorum(&self, digest: [u8; 32]) -> Result { + let bytes = self.read_child_quorum_record(CHILDREN, digest).await?; + record::decode(CHILD, &bytes, wire::LIMIT) + } + + async fn read_child_quorum_record( + &self, + category: &str, + digest: [u8; 32], + ) -> Result, CatalogError> { + let bytes = self + .read(&format!("{category}/{}", hex::encode(digest))) + .await? + .ok_or(CatalogError::Corrupt( + "missing transfer child quorum record", + ))?; + if record::digest(&bytes) != digest { + return Err(CatalogError::Corrupt("transfer child quorum digest")); + } + Ok(bytes) + } + + async fn check_child_quorum( + &self, + plan: &DirectoryMutation, + route: &GroupRoute, + proof: &ChildQuorum, + permit: Option<&ManagedTransitionPermit>, + ) -> Result<(), CatalogError> { + let marker = &proof.marker; + let position = marker.position(); + if marker.route() != route + || marker.is_active() + || marker.membership_revision() != 1 + || position.term.get() <= 1 + || position.index.get() <= 1 + { + return Err(CatalogError::Fenced); + } + let installed = match permit { + Some(permit) => { + self.observe_managed_transfer_install_quorum(plan, permit, route.history) + .await? + } + None => { + self.observe_transfer_install_quorum(plan, route.history) + .await? + } + } + .ok_or(CatalogError::Unconfirmed)?; + if !marker.preserves_transfer(&installed.genesis) + || !installed.voters.iter().any(|voter| { + voter.member.get() == proof.member + && voter.node.bytes() == proof.node + && voter.disk.bytes() == proof.disk + && voter.engine.bytes() == proof.engine + }) + { + return Err(CatalogError::Fenced); + } + Ok(()) + } +} + +fn staged_key(plan: &DirectoryMutation, index: usize) -> String { + format!("{STAGED}/{}/{index}", hex::encode(plan.digest())) +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/transition_install.rs b/crates/cellar/durability/raft/cell_groups/directory/transition_install.rs new file mode 100644 index 000000000..85a50df66 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/transition_install.rs @@ -0,0 +1,1067 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! A selected sealed-source decision can install a child on its exact engine. +use super::*; +use crate::raft::{ + cache::GroupLog, + cell_groups::{ + GroupManifest, PreparedCellGroupTransferChild, PreparedGroupManifest, TransferSeed, + }, + fleet_catalog::{ + FleetCatalogError, FleetGroupAuthorization, FleetHostAuthorization, + ReopenedManagedCellGroupTransition, TransferVoterAdmission, + }, + fleet_stop::{BootBinding, Guard}, + member_state::FleetGroupBinding, + storage::{CellGroupBootstrap, LogError, ReplicaLog}, +}; +use cellar_logic::consensus::{DiskLineage, EngineInstanceId, MemberId, NodeId, ReplicaIdentity}; +use serde::{Deserialize, Serialize}; + +const BINDING: u8 = 203; +const INTENT: u8 = 204; + +#[derive(Deserialize, Serialize)] +#[serde(deny_unknown_fields)] +struct InstallBinding { + namespace: [u8; 32], + genesis: GroupManifest, +} + +#[derive(Deserialize, Serialize)] +#[serde(deny_unknown_fields)] +struct InstallIntent { + binding: [u8; 32], + member: u64, + node: [u8; 16], + disk: [u8; 16], + engine: [u8; 16], +} + +pub(crate) struct TransferSeedPermit { + pub(crate) owner: Arc<()>, + pub(crate) bootstrap: CellGroupBootstrap, + pub(crate) genesis: GroupManifest, + local: ReplicaIdentity, + engine: EngineInstanceId, + managed: Option, +} + +struct ManagedSeed { + group: FleetGroupBinding, + boot: BootBinding, + deadline: u64, +} + +enum InstallData<'a> { + Prepared(&'a PreparedCellGroupTransferChild), + Retained { + marker: &'a GroupManifest, + selected: &'a GroupManifest, + closure: Option<&'a PreparedGroupManifest>, + }, +} + +impl InstallData<'_> { + fn genesis(&self) -> Result<&GroupManifest, CatalogError> { + match self { + Self::Prepared(child) => { + child + .genesis() + .map(|root| root.marker()) + .ok_or(CatalogError::Conflict( + "transfer requires final child genesis", + )) + } + Self::Retained { marker, .. } => Ok(marker), + } + } +} + +struct ManagedInstall<'a> { + transition: &'a ReopenedManagedCellGroupTransition, + host: &'a FleetHostAuthorization, + deadline: u64, +} + +impl ManagedInstall<'_> { + fn remaining(&self, log: &ReplicaLog) -> Result { + let now = log + .domain() + .mono_ms() + .map_err(|_| CatalogError::Unconfirmed)?; + self.deadline + .checked_sub(now) + .filter(|ms| *ms > 0) + .map(Duration::from_millis) + .ok_or(CatalogError::Unconfirmed) + } +} + +impl TransferSeedPermit { + pub(crate) fn check( + &self, + owner: &Arc<()>, + local: &ReplicaIdentity, + engine: EngineInstanceId, + ) -> Result<(), LogError> { + if !Arc::ptr_eq(owner, &self.owner) || local != &self.local || engine != self.engine { + return Err(LogError::WrongIdentity); + } + Ok(()) + } + + pub(crate) fn managed_binding( + &self, + guard: Option<&Guard>, + now: u64, + ) -> Result, LogError> { + let Some(managed) = &self.managed else { + return Ok(None); + }; + if now >= managed.deadline + || guard.is_none_or(|guard| !guard.admits_group_start(&managed.boot)) + || !managed.group.matches_scope(&self.local.scope) + { + return Err(LogError::WrongIdentity); + } + Ok(Some(&managed.group)) + } +} + +/// One reserved voter whose engine sync has an exact durable receipt. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct InstalledTransferVoter { + pub member: MemberId, + pub node: NodeId, + pub disk: DiskLineage, + pub engine: EngineInstanceId, +} + +/// The selected sealed-source child and its confirmed physical voter majority. +/// Startup and current-term Raft commitment require separate proof. +pub struct TransferInstallQuorum { + pub genesis: GroupManifest, + pub voters: Vec, +} + +fn install_prefix(route: &GroupRoute) -> String { + format!( + "transfer-installs/{}/{}", + hex::encode(route.history.group.bytes()), + hex::encode(route.history.incarnation.bytes()), + ) +} + +fn pending_genesis(marker: &GroupManifest) -> Result { + marker + .pending_transfer_genesis() + .map_err(|_| CatalogError::Fenced) +} + +struct InstallRecords { + prefix: String, + member: String, + binding: Vec, + intent: Vec, +} + +impl RangeDirectory { + fn install_records( + &self, + genesis: &GroupManifest, + local: &ReplicaIdentity, + engine: EngineInstanceId, + ) -> Result { + let prefix = install_prefix(genesis.route()); + let binding = record::encode( + BINDING, + &InstallBinding { + namespace: self.namespace.root_digest, + genesis: genesis.clone(), + }, + wire::LIMIT, + )?; + let intent = record::encode( + INTENT, + &InstallIntent { + binding: record::digest(&binding), + member: local.member.get(), + node: local.node.bytes(), + disk: local.disk.bytes(), + engine: engine.bytes(), + }, + wire::LIMIT, + )?; + let member = format!("{prefix}/{:016x}", local.member.get()); + Ok(InstallRecords { + prefix, + member, + binding, + intent, + }) + } + + async fn require_install_intent(&self, records: &InstallRecords) -> Result<(), CatalogError> { + if self + .read(&format!("{}/binding", records.prefix)) + .await? + .as_deref() + != Some(records.binding.as_slice()) + || self + .read(&format!("{}/intent", records.member)) + .await? + .as_deref() + != Some(records.intent.as_slice()) + { + return Err(CatalogError::Fenced); + } + Ok(()) + } + + /// Before a managed boot can park a pending child, recheck the exact + /// operation and durable install intent. The receipt may still be missing + /// after a lost reply. This path issues no permit and verifies no images. + pub(crate) async fn require_parked_managed_transfer( + &self, + transition: &ReopenedManagedCellGroupTransition, + group: &GroupLog, + marker: &GroupManifest, + engine: EngineInstanceId, + ) -> Result<(), CatalogError> { + let owner = self + .namespace + .config + .managed_fleet + .as_ref() + .ok_or(CatalogError::ManagedAdmissionRequired)?; + let local = group.identity(); + if marker.route().history != local.scope + || group + .membership() + .map_err(CatalogError::Log)? + .fleet_status() + .is_none_or(|status| { + status.binding.fleet() != owner.fleet + || !status.binding.matches_scope(&local.scope) + }) + { + return Err(CatalogError::Fenced); + } + self.bounded(async { + let genesis = pending_genesis(marker)?; + self.require_open_transfer_install(&transition.plan, &genesis) + .await?; + let birth = self + .require_ordinary_birth_reservation(marker.route()) + .await?; + if !birth + .voters() + .iter() + .any(|voter| voter.node == local.node && voter.member == local.member) + { + return Err(CatalogError::NotInitialMember); + } + let records = self.install_records(&genesis, local, engine)?; + self.require_install_intent(&records).await?; + if self + .read(&format!("{}/receipt", records.member)) + .await? + .is_some_and(|receipt| receipt != records.intent) + { + return Err(CatalogError::Fenced); + } + Ok(()) + }) + .await + } + + /// Install one reserved voter from the final sealed-source partition. The + /// returned engine mapping grants no startup, voting or serving authority. + /// Retrying the exact disk and engine reconciles an unknown result. + pub async fn install_transfer_voter( + &self, + plan: &DirectoryMutation, + child: &PreparedCellGroupTransferChild, + log: &ReplicaLog, + ) -> Result, CatalogError> { + self.unmanaged_birth()?; + self.install_transfer_voter_inner(plan, InstallData::Prepared(child), log, None) + .await + } + + /// Sync a prepared child into its reserved managed voter engine. + /// The receiver must own the destination's retained page and image stores. + /// A durable install receipt grants no attachment or serving authority. + pub async fn install_managed_transfer_voter( + &self, + transition: &ReopenedManagedCellGroupTransition, + child: &PreparedCellGroupTransferChild, + log: &ReplicaLog, + host: &FleetHostAuthorization, + timeout: Duration, + ) -> Result { + if timeout.as_millis() == 0 || timeout > Duration::from_secs(60) { + return Err(CatalogError::Invalid("managed transfer install timeout")); + } + let deadline = log + .domain() + .mono_ms() + .map_err(|_| CatalogError::Unconfirmed)? + .checked_add(timeout.as_millis() as u64) + .ok_or(CatalogError::Limit)?; + self.sync_managed_transfer_voter( + transition, + InstallData::Prepared(child), + log, + host, + deadline, + ) + .await + } + + pub(crate) async fn reconcile_managed_transfer_voter( + &self, + transition: &ReopenedManagedCellGroupTransition, + route: &GroupRoute, + log: &ReplicaLog, + host: &FleetHostAuthorization, + deadline: u64, + closure: Option<&PreparedGroupManifest>, + ) -> Result, CatalogError> { + let group = match log.cell_group(&route.history) { + Ok(group) => group, + Err(LogError::NotFound) => return Ok(None), + Err(error) => return Err(CatalogError::Log(error)), + }; + let selected = group + .group_manifest() + .map_err(CatalogError::Log)? + .filter(|marker| marker.route() == route) + .ok_or(CatalogError::Fenced)?; + let marker = pending_genesis(&selected)?; + self.sync_managed_transfer_voter( + transition, + InstallData::Retained { + marker: &marker, + selected: &selected, + closure, + }, + log, + host, + deadline, + ) + .await + .map(Some) + } + + async fn sync_managed_transfer_voter( + &self, + transition: &ReopenedManagedCellGroupTransition, + data: InstallData<'_>, + log: &ReplicaLog, + host: &FleetHostAuthorization, + deadline: u64, + ) -> Result { + let managed = ManagedInstall { + transition, + host, + deadline, + }; + let group = log + .domain() + .run(Box::pin(crate::asyncrt::timeout_at( + deadline, + Box::pin(self.install_transfer_voter_inner( + &transition.plan, + data, + log, + Some(managed), + )), + ))) + .await + .map_err(|_| CatalogError::Unconfirmed)? + .map_err(|_| CatalogError::Unconfirmed)??; + let local = group.identity(); + Ok(InstalledTransferVoter { + member: local.member, + node: local.node, + disk: local.disk, + engine: log.storage_instance().map_err(CatalogError::Log)?, + }) + } + + async fn install_transfer_voter_inner( + &self, + plan: &DirectoryMutation, + data: InstallData<'_>, + log: &ReplicaLog, + managed: Option>, + ) -> Result, CatalogError> { + let genesis = data.genesis()?; + self.bounded(async { + let authority = if let Some(managed) = &managed { + let boot = self.require_managed_group_boot(log, managed.host).await?; + let authority = managed + .host + .managed_transfer_install_authority( + managed.transition, + self, + genesis.route(), + managed.remaining(log)?, + ) + .await + .map_err(CatalogError::from)?; + authority.transition.check(self, plan)?; + if authority.boot != boot { + return Err(CatalogError::Fenced); + } + Some((managed, authority)) + } else { + self.namespace.require_cell_group_unmanaged().await?; + None + }; + self.require_open_transfer_install(plan, genesis).await?; + let birth = self + .require_ordinary_birth_reservation(genesis.route()) + .await?; + let voter = birth + .voters() + .iter() + .find(|member| member.node == log.limits().node) + .ok_or(CatalogError::NotInitialMember)?; + self.namespace.require_node_storage(log).await?; + let engine = log.storage_instance().map_err(CatalogError::Log)?; + let route = genesis.route(); + let local = ReplicaIdentity { + scope: route.history, + member: voter.member, + node: voter.node, + disk: log.limits().disk, + process: log.limits().process, + }; + let records = self.install_records(genesis, &local, engine)?; + match &data { + InstallData::Prepared(_) => { + self.exact_install_record( + &format!("{}/binding", records.prefix), + &records.binding, + ) + .await?; + self.exact_install_record( + &format!("{}/intent", records.member), + &records.intent, + ) + .await?; + } + InstallData::Retained { .. } => self.require_install_intent(&records).await?, + } + let receipt = format!("{}/receipt", records.member); + if let Some(observed) = self.read(&receipt).await? { + if observed != records.intent { + return Err(CatalogError::Conflict("transfer install receipt")); + } + match log.cell_group(&route.history) { + Ok(_) => (), + Err(LogError::NotFound) => return Err(CatalogError::LostStableState), + Err(error) => return Err(CatalogError::Log(error)), + } + } + self.require_open_transfer_install(plan, genesis).await?; + let managed = if let Some((managed, original)) = authority { + let authority = managed + .host + .managed_transfer_install_authority( + managed.transition, + self, + route, + managed.remaining(log)?, + ) + .await + .map_err(CatalogError::from)?; + authority.transition.check(self, plan)?; + if authority.group != original.group || authority.boot != original.boot { + return Err(CatalogError::Fenced); + } + Some(ManagedSeed { + group: authority.group, + boot: authority.boot, + deadline: managed.deadline, + }) + } else { + None + }; + let permit = TransferSeedPermit { + owner: log.owner(), + bootstrap: CellGroupBootstrap { + route: route.clone(), + member: voter.member, + voters: birth.voters().iter().map(|m| m.member).collect(), + learners: birth.learners().iter().map(|m| m.member).collect(), + }, + genesis: genesis.clone(), + local, + engine, + managed, + }; + let group = match &data { + InstallData::Prepared(child) => { + let seed = TransferSeed::new(permit, child).map_err(CatalogError::Log)?; + log.install_transfer_seed(seed) + .await + .map_err(CatalogError::Log)? + } + InstallData::Retained { + selected, closure, .. + } => { + let group = log.cell_group(&route.history).map_err(CatalogError::Log)?; + let guard = log.lifecycle_guard().map_err(CatalogError::Log)?; + let now = log + .domain() + .mono_ms() + .map_err(|_| CatalogError::Unconfirmed)?; + let binding = permit + .managed_binding(guard.as_ref(), now) + .map_err(CatalogError::Log)? + .ok_or(CatalogError::ManagedAdmissionRequired)?; + let cache = group.read().map_err(CatalogError::Log)?; + if group.identity() != &permit.local + || cache.group_manifest.as_ref() != Some(*selected) + || cache + .membership + .fleet_status() + .is_none_or(|status| &status.binding != binding) + { + return Err(CatalogError::Fenced); + } + let reopened = closure.is_some_and(|verified| { + Arc::ptr_eq(&verified.engine_owner, &group.owner) + && verified.marker() == *selected + && verified.images.root() == genesis.root() + }); + if !reopened + && cache + .manifest_pins + .as_ref() + .is_none_or(|pins| pins.root() != genesis.root()) + { + return Err(CatalogError::Runtime( + crate::raft::completion::RuntimeError::NotReady, + )); + } + drop(cache); + group + } + }; + self.require_open_transfer_install(plan, genesis).await?; + self.exact_install_record(&receipt, &records.intent).await?; + Ok(group) + }) + .await + } + + /// Observe an exact majority of synced installs for the selected child. + /// Missing receipts leave a child below quorum. Corrupt receipts, stale + /// storage claims or two receipts sharing physical identity fail closed. + pub async fn observe_transfer_install_quorum( + &self, + plan: &DirectoryMutation, + history: GroupHistory, + ) -> Result, CatalogError> { + self.unmanaged_birth()?; + self.observe_transfer_installs(plan, history, false).await + } + + pub(crate) async fn observe_managed_transfer_install_quorum( + &self, + plan: &DirectoryMutation, + permit: &crate::raft::fleet_catalog::ManagedTransitionPermit, + history: GroupHistory, + ) -> Result, CatalogError> { + permit.check(self, plan)?; + self.observe_transfer_installs(plan, history, true).await + } + + async fn observe_transfer_installs( + &self, + plan: &DirectoryMutation, + history: GroupHistory, + managed: bool, + ) -> Result, CatalogError> { + let (_, children) = self.transition_routes(plan)?; + let route = children + .iter() + .find(|route| route.history == history) + .ok_or(CatalogError::Conflict("transfer child outside plan"))?; + self.bounded(async { + if !managed { + self.namespace.require_cell_group_unmanaged().await?; + } + let prefix = install_prefix(route); + let Some(binding) = self.read(&format!("{prefix}/binding")).await? else { + return Ok(None); + }; + let selected: InstallBinding = record::decode(BINDING, &binding, wire::LIMIT)?; + if selected.namespace != self.namespace.root_digest || selected.genesis.route() != route + { + return Err(CatalogError::Corrupt("transfer install genesis binding")); + } + self.require_transfer_install(plan, &selected.genesis) + .await?; + let birth = self.require_ordinary_birth_reservation(route).await?; + let digest = record::digest(&binding); + let mut voters = Vec::with_capacity(birth.voters().len()); + let mut disks = BTreeSet::new(); + let mut engines = BTreeSet::new(); + for voter in birth.voters() { + let member = format!("{prefix}/{:016x}", voter.member.get()); + let Some(receipt) = self.read(&format!("{member}/receipt")).await? else { + continue; + }; + if self.read(&format!("{member}/intent")).await?.as_ref() != Some(&receipt) { + return Err(CatalogError::Corrupt( + "transfer install receipt without exact intent", + )); + } + let intent: InstallIntent = record::decode(INTENT, &receipt, wire::LIMIT)?; + if intent.binding != digest + || intent.member != voter.member.get() + || intent.node != voter.node.bytes() + { + return Err(CatalogError::Corrupt("transfer install voter identity")); + } + let disk = DiskLineage::new(intent.disk) + .map_err(|_| CatalogError::Corrupt("transfer install disk"))?; + let engine = EngineInstanceId::new(intent.engine) + .map_err(|_| CatalogError::Corrupt("transfer install engine"))?; + if !disks.insert(disk) || !engines.insert(engine) { + return Err(CatalogError::Conflict("shared transfer install storage")); + } + self.namespace + .verify_node_storage_identity(voter.node, disk, engine) + .await?; + voters.push(InstalledTransferVoter { + member: voter.member, + node: voter.node, + disk, + engine, + }); + } + self.require_transfer_install(plan, &selected.genesis) + .await?; + Ok( + (voters.len() > birth.voters().len() / 2).then_some(TransferInstallQuorum { + genesis: selected.genesis, + voters, + }), + ) + }) + .await + } + + /// Admit a physically installed voter to the pending child Raft group. + /// The complete local image must still be recovered before attachment. + /// This grants no routing, application or activation authority. + pub fn bootstrap_pending_transfer_voter<'a>( + &'a self, + plan: &'a DirectoryMutation, + history: GroupHistory, + log: &'a ReplicaLog, + ) -> futures_util::future::BoxFuture<'a, Result> { + Box::pin(async move { + let proof = self + .observe_transfer_install_quorum(plan, history) + .await? + .ok_or(CatalogError::Unconfirmed)?; + let (group, birth) = self.pending_transfer_member(plan, &proof, log).await?; + Ok(CellGroupBootstrapPermit::pending_transfer( + group, + birth, + self.clone(), + log.clone(), + super::bootstrap::PendingTransfer::Unmanaged(Arc::new( + self.decode_mutation(plan.encode())?, + )), + )) + }) + } + + pub(crate) fn bootstrap_managed_pending_transfer_voter<'a>( + &'a self, + transition: Arc, + route: &'a GroupRoute, + log: &'a ReplicaLog, + host: &'a FleetHostAuthorization, + deadline: u64, + ) -> futures_util::future::BoxFuture<'a, Result> { + Box::pin(async move { + let (group, birth, _) = self + .require_managed_transfer_voter( + &transition, + route, + log, + host, + TransferVoterAdmission::Install, + deadline, + ) + .await?; + if group + .group_manifest() + .map_err(CatalogError::Log)? + .is_none_or(|marker| marker.transfer_activation().is_some()) + { + return Err(CatalogError::Fenced); + } + Ok(CellGroupBootstrapPermit::pending_transfer( + group, + birth, + self.clone(), + log.clone(), + super::bootstrap::PendingTransfer::Managed { + transition, + host: Box::new(host.clone()), + deadline, + }, + )) + }) + } + + /// Reattach an already activated voter from its permanently claimed engine. + /// A Live child reopens from fleet membership and permanent install and + /// publication proofs. If a later transfer has sealed its selected range, + /// the local seal continues to fence application work. A Reserved child + /// still needs the active transition and an active local marker. + pub(crate) async fn bootstrap_managed_activated_transfer_voter( + &self, + birth: &CellGroupBirth, + log: &ReplicaLog, + host: &FleetHostAuthorization, + ) -> Result { + self.bounded(async { + let route = birth.route(); + let group = log.cell_group(&route.history).map_err(CatalogError::Log)?; + let marker = group + .group_manifest() + .map_err(CatalogError::Log)? + .filter(|marker| { + marker.route() == route + && marker.transfer_activation().is_some() + && (marker.is_active() + || marker + .source_seal() + .is_some_and(|seal| seal.retirement().is_none())) + }) + .ok_or(CatalogError::Fenced)?; + let timeout = Duration::from_millis(self.limits.operation_timeout_ms); + match host.group_authorization(&route.history, timeout).await { + Ok(authority) => { + self.require_live_transfer_voter(birth, log, host, &authority, &group, &marker) + .await?; + return Ok(CellGroupBootstrapPermit::managed_activated_transfer( + group, + birth.clone(), + self.clone(), + log.clone(), + host.clone(), + false, + )); + } + // The exact active transition check below still requires a + // Reserved history and independently rechecks the current boot. + Err(FleetCatalogError::Conflict) if marker.is_active() => (), + Err(error) => return Err(CatalogError::from(error)), + } + let origin = marker.transfer_origin().ok_or(CatalogError::Fenced)?; + let operation = ProposalId::new(origin.operation()) + .map_err(|_| CatalogError::Corrupt("transfer operation"))?; + let deadline = log + .domain() + .mono_ms() + .map_err(|_| CatalogError::Unconfirmed)? + .checked_add(self.limits.operation_timeout_ms) + .ok_or(CatalogError::Limit)?; + let transition = host + .stop_catalog() + .reopen_managed_transition(self, operation, timeout) + .await + .map_err(CatalogError::from)? + .ok_or(CatalogError::Fenced)?; + let (group, _, _) = self + .require_managed_transfer_voter( + &transition, + route, + log, + host, + TransferVoterAdmission::Retained, + deadline, + ) + .await?; + self.require_current_group_birth(birth).await?; + Ok(CellGroupBootstrapPermit::managed_activated_transfer( + group, + birth.clone(), + self.clone(), + log.clone(), + host.clone(), + true, + )) + }) + .await + } + + async fn require_live_transfer_voter( + &self, + birth: &CellGroupBirth, + log: &ReplicaLog, + host: &FleetHostAuthorization, + authority: &FleetGroupAuthorization, + group: &GroupLog, + marker: &GroupManifest, + ) -> Result<(), CatalogError> { + if !authority.check_birth(self, birth) { + return Err(CatalogError::Fenced); + } + self.require_managed_retained_group_boot(birth, log, host) + .await?; + self.require_current_group_birth(birth).await?; + let prefix = install_prefix(birth.route()); + let bytes = self + .read(&format!("{prefix}/binding")) + .await? + .ok_or(CatalogError::LostStableState)?; + let binding: InstallBinding = record::decode(BINDING, &bytes, wire::LIMIT)?; + if binding.namespace != self.namespace.root_digest + || !binding.genesis.is_transfer_genesis() + || binding.genesis.route() != birth.route() + || !marker.preserves_transfer(&binding.genesis) + { + return Err(CatalogError::Fenced); + } + let engine = log.storage_instance().map_err(CatalogError::Log)?; + let records = self.install_records(&binding.genesis, group.identity(), engine)?; + self.require_install_intent(&records).await?; + if self + .read(&format!("{}/receipt", records.member)) + .await? + .as_deref() + != Some(records.intent.as_slice()) + { + return Err(CatalogError::LostStableState); + } + self.require_selected_retired_transfer_child(marker).await?; + self.require_managed_retained_group_boot(birth, log, host) + .await?; + super::bootstrap::check_member(group)?; + let (_, _, activation) = marker.transfer_activation().ok_or(CatalogError::Fenced)?; + let cache = group.read().map_err(CatalogError::Log)?; + if cache.hard_state.commit < activation.index.get() + || cache + .membership + .fleet_status() + .is_none_or(|status| &status.binding != authority.binding()) + || !authority.matches_replica(group.identity(), &cache.membership) + { + return Err(CatalogError::Fenced); + } + Ok(()) + } + + pub(super) async fn require_managed_transfer_voter( + &self, + transition: &ReopenedManagedCellGroupTransition, + route: &GroupRoute, + log: &ReplicaLog, + host: &FleetHostAuthorization, + admission: TransferVoterAdmission, + deadline: u64, + ) -> Result<(GroupLog, CellGroupBirthRequest, BootBinding), CatalogError> { + let domain = log.domain(); + let remaining = || { + let now = domain.mono_ms().map_err(|_| CatalogError::Unconfirmed)?; + deadline + .checked_sub(now) + .filter(|ms| *ms > 0) + .map(Duration::from_millis) + .ok_or(CatalogError::Runtime( + crate::raft::completion::RuntimeError::DeadlineExceeded, + )) + }; + let proof = host + .stop_catalog() + .observe_managed_transfer_install_quorum(transition, self, route, remaining()?) + .await + .map_err(CatalogError::from)? + .ok_or(CatalogError::Unconfirmed)?; + if admission == TransferVoterAdmission::Install { + self.require_open_transfer_install(&transition.plan, &proof.genesis) + .await?; + } + let (group, birth) = self + .pending_transfer_member(&transition.plan, &proof, log) + .await?; + let authority = host + .managed_transfer_voter_authority(transition, self, route, admission, remaining()?) + .await + .map_err(CatalogError::from)?; + let boot = self.require_managed_group_boot(log, host).await?; + remaining()?; + let cache = group.read().map_err(CatalogError::Log)?; + if boot != authority.boot + || cache + .membership + .fleet_status() + .is_none_or(|status| status.binding != authority.group) + || cache.membership.revision().get() != 1 + || !cache + .conf_state + .voters + .contains(&group.identity().member.get()) + { + return Err(CatalogError::Fenced); + } + if admission == TransferVoterAdmission::Retained + && cache + .group_manifest + .as_ref() + .filter(|marker| marker.is_active()) + .and_then(GroupManifest::transfer_activation) + .is_none_or(|(_, _, position)| cache.hard_state.commit < position.index.get()) + { + return Err(CatalogError::Fenced); + } + drop(cache); + Ok((group, birth, boot)) + } + + async fn pending_transfer_member( + &self, + plan: &DirectoryMutation, + proof: &TransferInstallQuorum, + log: &ReplicaLog, + ) -> Result<(GroupLog, CellGroupBirthRequest), CatalogError> { + self.bounded(async { + let local = log.limits(); + let voter = proof + .voters + .iter() + .find(|receipt| receipt.node == local.node) + .ok_or(CatalogError::NotInitialMember)?; + let engine = log.storage_instance().map_err(CatalogError::Log)?; + if voter.disk != local.disk || voter.engine != engine { + return Err(CatalogError::LostStableState); + } + let group = + log.cell_group(&proof.genesis.route().history) + .map_err(|error| match error { + LogError::NotFound => CatalogError::LostStableState, + error => CatalogError::Log(error), + })?; + let birth = self + .require_ordinary_birth_reservation(proof.genesis.route()) + .await?; + let activation = { + let cache = group.read().map_err(CatalogError::Log)?; + let marker = cache + .group_manifest + .as_ref() + .ok_or(CatalogError::LostStableState)?; + let expected_voters = birth.voters().iter().map(|member| member.member.get()); + let expected_learners = birth.learners().iter().map(|member| member.member.get()); + if group.identity().member != voter.member + || marker.route() != proof.genesis.route() + || !marker.preserves_transfer(&proof.genesis) + { + return Err(CatalogError::LostStableState); + } + let activation = marker.transfer_activation(); + if activation.is_none() + && (cache.membership.revision().get() != 1 + || !cache.conf_state.voters.iter().copied().eq(expected_voters) + || !cache + .conf_state + .learners + .iter() + .copied() + .eq(expected_learners) + || !cache.conf_state.voters_outgoing.is_empty() + || !cache.conf_state.learners_next.is_empty() + || cache.conf_state.auto_leave) + { + return Err(CatalogError::LostStableState); + } + activation + }; + if let Some((certificate, _, _)) = activation { + let published = self.publication(plan).await?.ok_or(CatalogError::Fenced)?; + if published.transfer_archive().is_none() + || published.journal_digest() != certificate + { + return Err(CatalogError::Fenced); + } + } + super::bootstrap::check_member(&group)?; + let phase = self.require_transfer_install(plan, &proof.genesis).await?; + if phase == TransitionPhase::Retired || activation.is_some() { + let (view, _) = self.current().await?; + self.selected_route_in(&view, proof.genesis.route()).await?; + } + Ok((group, birth)) + }) + .await + } + + async fn require_open_transfer_install( + &self, + plan: &DirectoryMutation, + genesis: &GroupManifest, + ) -> Result<(), CatalogError> { + if self.require_transfer_install(plan, genesis).await? == TransitionPhase::Retired { + return Err(CatalogError::Fenced); + } + Ok(()) + } + + async fn require_transfer_install( + &self, + plan: &DirectoryMutation, + genesis: &GroupManifest, + ) -> Result { + let (_, children) = self.transition_routes(plan)?; + let (view, _) = self.current().await?; + let journal = self + .matching_transition(&view.head, plan) + .await? + .ok_or(CatalogError::Fenced)?; + let origin = genesis.transfer_origin().ok_or(CatalogError::Fenced)?; + if !genesis.is_transfer_genesis() + || !matches!( + journal.phase(), + TransitionPhase::Sealed + | TransitionPhase::Installed + | TransitionPhase::Archived + | TransitionPhase::Published + | TransitionPhase::Active + | TransitionPhase::Retired + ) + || journal.sealed_sources() != Some(origin.sealed_sources()) + || origin.operation() != plan.operation().bytes() + || origin.plan() != plan.digest() + || !children.contains(genesis.route()) + { + return Err(CatalogError::Fenced); + } + Ok(journal.phase()) + } + + async fn exact_install_record(&self, path: &str, expected: &[u8]) -> Result<(), CatalogError> { + let observed = self + .namespace + .inner + .create_and_observe_bounded(&self.key(path), expected, wire::LIMIT) + .await?; + if observed != expected { + return Err(CatalogError::Conflict("transfer install binding")); + } + Ok(()) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/transition_preparation.rs b/crates/cellar/durability/raft/cell_groups/directory/transition_preparation.rs new file mode 100644 index 000000000..043edec18 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/transition_preparation.rs @@ -0,0 +1,188 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Managed child pre-copy from independently replayed active source holds. +use super::transition_sources::TransitionSourceCut; +use super::*; +use crate::raft::{ + cell_groups::{ + CellGroupArchiveCoverage, CellGroupArchiveExecution, CellGroupArchiveHold, + CellGroupArchivePool, CellGroupArchiveTailPool, CellGroupRecoveryPool, + CellGroupTransferTarget, PreparedCellGroupTransferChild, StagedCellGroupPartition, + }, + fleet_catalog::ManagedTransitionPermit, +}; +use serde::{Deserialize, Serialize}; + +const KIND: u8 = 202; +const PREPARATIONS: &str = "transition-preparations"; + +/// Existing archive and restore owners, plus exact parent holds and dedicated +/// child stores in plan order. Retain the remote holds through replay. +pub struct CellGroupTransitionPreparationInputs { + pub archives: CellGroupArchivePool, + pub tails: CellGroupArchiveTailPool, + pub restore: CellGroupRecoveryPool, + pub sources: Vec, + pub targets: Vec, +} + +/// Replayed active source data and the complete synced, non-serving child +/// partition. A persisted preparation describes the copy. A replacement +/// controller must reopen the holds and independently rebuild its physical pins. +pub struct PreparedManagedCellGroupTransition { + partition: StagedCellGroupPartition, + sources: Vec, + journal: TransitionJournal, +} + +impl PreparedManagedCellGroupTransition { + pub fn journal(&self) -> &TransitionJournal { + &self.journal + } + + pub fn children(&self) -> &[PreparedCellGroupTransferChild] { + &self.partition.children + } + + pub fn sources(&self) -> &[CellGroupArchiveCoverage] { + &self.sources + } +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Preparation { + namespace: [u8; 32], + plan: [u8; 32], + sources: [u8; 32], + children: Vec, +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Child { + root: Option<[u8; 32]>, + cells: u32, +} + +impl RangeDirectory { + pub(crate) async fn prepare_managed_transition_data( + &self, + domain: &impl CellGroupArchiveExecution, + plan: &DirectoryMutation, + permit: &ManagedTransitionPermit, + inputs: CellGroupTransitionPreparationInputs, + ) -> Result { + permit.check(self, plan)?; + let verified = Box::pin(self.verify_transition_source_set( + domain, + &inputs.archives, + &inputs.tails, + plan, + inputs.sources, + TransitionSourceCut::Active, + )) + .await?; + if let Some(expected) = verified.journal.preparation() { + let selected = self + .bounded(self.read_transition_preparation(plan, expected)) + .await?; + if selected.sources != verified.digest { + return Err( + CatalogError::Conflict("transition preparation sources changed").into(), + ); + } + } + let partition = Box::pin(inputs.restore.stage_partition( + domain, + verified.sources.clone(), + verified.children, + inputs.targets.into_iter().map(Some).collect(), + None, + )) + .await?; + let record = Preparation { + namespace: self.namespace.root_digest, + plan: plan.digest, + sources: verified.digest, + children: partition + .children + .iter() + .map(|child| Child { + root: child.root().digest(), + cells: child.root().cells(), + }) + .collect(), + }; + let bytes = record::encode(KIND, &record, wire::LIMIT)?; + let digest = record::digest(&bytes); + self.bounded(self.immutable(PREPARATIONS, digest, &bytes)) + .await?; + let journal = self + .record_managed_preparation(plan, permit, digest, verified.digest) + .await?; + Ok(PreparedManagedCellGroupTransition { + partition, + sources: verified.sources, + journal, + }) + } + + /// Reopen the exact active-source holds referenced by a managed preparation. + /// Replaying them into fresh child stores reestablishes physical readiness. + pub async fn managed_preparation_holds( + &self, + plan: &DirectoryMutation, + ) -> Result, CatalogError> { + if self.namespace_config().managed_fleet.is_none() { + return Err(CatalogError::ManagedAdmissionRequired); + } + self.check_transition_plan(plan)?; + self.bounded(async { + let (view, _) = self.current().await?; + let journal = self + .matching_transition(&view.head, plan) + .await? + .ok_or(CatalogError::Fenced)?; + if journal.phase() == TransitionPhase::Cancelled { + return Err(CatalogError::Fenced); + } + let digest = journal.preparation().ok_or(CatalogError::Fenced)?; + let record = self.read_transition_preparation(plan, digest).await?; + self.read_transition_source_holds( + plan, + &journal, + record.sources, + TransitionSourceCut::Active, + ) + .await + }) + .await + } + + async fn read_transition_preparation( + &self, + plan: &DirectoryMutation, + digest: [u8; 32], + ) -> Result { + let bytes = self + .read(&format!("{PREPARATIONS}/{}", hex::encode(digest))) + .await? + .ok_or(CatalogError::Corrupt("missing transition preparation"))?; + if record::digest(&bytes) != digest { + return Err(CatalogError::Corrupt("transition preparation digest")); + } + let decoded: Preparation = record::decode(KIND, &bytes, wire::LIMIT)?; + if decoded.namespace != self.namespace.root_digest + || decoded.plan != plan.digest + || decoded.sources == [0; 32] + || decoded.children.len() != plan.request.children.len() + || decoded.children.iter().any(|child| { + (child.cells == 0) != child.root.is_none() || child.root == Some([0; 32]) + }) + { + return Err(CatalogError::Corrupt("transition preparation binding")); + } + Ok(decoded) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/transition_retirement.rs b/crates/cellar/durability/raft/cell_groups/directory/transition_retirement.rs new file mode 100644 index 000000000..e188f2ece --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/transition_retirement.rs @@ -0,0 +1,1030 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use crate::application::consensus::group_protocol::{ + CellGroupEntry, GroupCommand, GroupEntryBinding, GroupEntryLimits, +}; +use crate::raft::{ + cache::GroupLog, + cell_groups::{ + CellGroupArchiveError, CellGroupArchiveExecution, CellGroupArchiveHold, + CellGroupArchivePool, GroupManifest, GroupSourceRetirement, + }, + completion::RuntimeError, + fleet_catalog::{FleetGroupAuthorization, FleetHostAuthorization, ManagedTransitionPermit}, + fleet_stop::BootBinding, + member_state::{members, GroupMembership}, + storage::ReplicaLog, +}; +use cellar_logic::consensus::{ + DiskLineage, EngineInstanceId, LogIndex, LogPosition, MemberId, NodeId, Term, +}; +use serde::{Deserialize, Serialize}; + +/// A durable parent tombstone witnessed by its engine-bound Raft owner. +#[derive(Clone, Debug)] +pub struct CellGroupSourceRetirementAck { + route: GroupRoute, + seal: LogPosition, + tombstone: GroupSourceRetirement, + members: Vec, +} + +impl CellGroupSourceRetirementAck { + pub fn tombstone(&self) -> &GroupSourceRetirement { + &self.tombstone + } +} + +/// One sealed parent and the exact replacement certificate, bound to its +/// existing voter engine. A permit does not authorize releasing any holds. +pub struct CellGroupSourceRetirementPermit { + group: GroupLog, + engine: EngineInstanceId, + sealed: GroupManifest, + birth: CellGroupBirth, + publication: [u8; 32], + deadline: u64, + entry: Vec, + managed: Option<(BootBinding, u64)>, +} + +struct ManagedRetirementAdmission<'a> { + permit: &'a ManagedTransitionPermit, + host: &'a FleetHostAuthorization, + authority: &'a FleetGroupAuthorization, + deadline: u64, +} + +impl CellGroupSourceRetirementPermit { + pub(crate) fn history(&self) -> GroupHistory { + self.group.identity().scope + } + + pub(crate) fn check( + &self, + log: &ReplicaLog, + group: &GroupLog, + ) -> Result<(), RuntimeError> { + if !Arc::ptr_eq(&group.owner, &self.group.owner) + || group.identity() != self.group.identity() + || log.storage_instance()? != self.engine + || log.domain().mono_ms().map_err(|_| RuntimeError::Closed)? >= self.deadline + { + return Err(RuntimeError::StaleCompletion); + } + if let Some((boot, revision)) = &self.managed { + if log + .lifecycle_guard()? + .is_none_or(|guard| !guard.admits_group_start(boot)) + { + return Err(RuntimeError::StaleCompletion); + } + let cache = group.read()?; + if cache.membership.revision().get() != *revision + || !cache + .conf_state + .voters + .contains(&group.identity().member.get()) + { + return Err(RuntimeError::StaleCompletion); + } + } else { + drop(group.read()?); + } + Ok(()) + } + + pub(crate) fn acknowledge( + &self, + group: &GroupLog, + marker: &GroupManifest, + ) -> Result, RuntimeError> { + if marker.route() != self.sealed.route() || !marker.preserves_source_seal(&self.sealed) { + return Err(RuntimeError::StaleCompletion); + } + let seal = marker.source_seal().ok_or(RuntimeError::StaleCompletion)?; + let Some(retired) = seal.retirement() else { + return Ok(None); + }; + if retired.publication() != self.publication { + return Err(RuntimeError::StaleCompletion); + } + let cache = group.read()?; + if cache.group_manifest.as_ref() != Some(marker) + || cache.membership.revision().get() != marker.membership_revision() + || cache.hard_state.commit < retired.position().index.get() + { + return Err(RuntimeError::NotReady); + } + Ok(Some(CellGroupSourceRetirementAck { + route: marker.route().clone(), + seal: seal.position(), + tombstone: retired.clone(), + members: final_source_members(&cache.conf_state, &cache.membership, &self.birth)?, + })) + } + + pub(crate) fn into_entry(self) -> Vec { + self.entry + } +} + +impl RangeDirectory { + /// Verify the exact synced local tombstone before preserving a retired + /// source as physical occupancy without an attached Raft owner. + pub(crate) async fn park_managed_retired_source( + &self, + plan: &DirectoryMutation, + group: &GroupLog, + marker: &GroupManifest, + ) -> Result<(), CatalogError> { + self.bounded(async { + let seal = marker.source_seal().ok_or(CatalogError::Fenced)?; + let tombstone = seal.retirement().ok_or(CatalogError::Fenced)?; + if seal.transition() != plan.operation().bytes() { + return Err(CatalogError::Fenced); + } + let basis = self.retirement_basis(plan).await?; + let digest = basis + .journal + .retired_parents() + .ok_or(CatalogError::Fenced)?; + let retirement = self.require_retirement_set(plan, &basis, digest).await?; + let receipt = basis + .parents + .iter() + .zip(&retirement.parents) + .find_map(|(hold, receipt)| { + (marker.route() == hold.marker().route() + && marker.preserves_source_seal(hold.marker())) + .then_some(receipt) + }) + .ok_or(CatalogError::Fenced)?; + let position = tombstone.position(); + if retirement.publication != tombstone.publication() + || receipt.index != position.index.get() + || receipt.term != position.term.get() + || group.identity().scope != marker.route().history + { + return Err(CatalogError::Fenced); + } + let birth = self.open_historical_birth(marker.route()).await?; + let cache = group.read().map_err(CatalogError::Log)?; + if cache.group_manifest.as_ref() != Some(marker) + || cache.hard_state.commit < position.index.get() + { + return Err(CatalogError::LostStableState); + } + let identity = group.identity(); + if cache.membership.revision().get() != marker.membership_revision() + || final_source_members(&cache.conf_state, &cache.membership, &birth) + .map_err(CatalogError::Runtime)? + != receipt.members + || !receipt.members.iter().any(|member| { + member.member == identity.member.get() + && member.node == identity.node.bytes() + && member.disk.is_none_or(|disk| disk == identity.disk.bytes()) + }) + { + return Err(CatalogError::Fenced); + } + Ok(()) + }) + .await + } + + pub(crate) async fn require_managed_transfer_retirement_coverage( + &self, + domain: &impl CellGroupArchiveExecution, + pool: &CellGroupArchivePool, + plan: &DirectoryMutation, + permit: &ManagedTransitionPermit, + ) -> Result<(), CellGroupArchiveBindingError> { + permit.check(self, plan)?; + let basis = self.bounded(self.retirement_basis(plan)).await?; + let digest = basis + .journal + .retired_parents() + .ok_or(CatalogError::Fenced)?; + if basis.journal.parent_holds_released() { + return Err(CatalogError::Fenced.into()); + } + self.bounded(self.require_retirement_set(plan, &basis, digest)) + .await?; + self.require_live_retirement_archives(domain, pool, plan, Some(permit)) + .await + } + + /// Discover a retained source after its ranges have been replaced. The + /// historical birth is usable only for this sealed replica's retirement. + pub(crate) async fn discover_managed_sealed_source( + &self, + marker: &GroupManifest, + ) -> Result, CatalogError> { + self.bounded(async { + let Some((plan, journal)) = self.active_transition().await? else { + return Ok(None); + }; + if !matches!( + journal.phase(), + TransitionPhase::Published | TransitionPhase::Active | TransitionPhase::Retired + ) || !self.transition_routes(&plan)?.0.contains(marker.route()) + { + return Ok(None); + } + let publication = self + .confirmed_transfer_publication(&plan) + .await? + .ok_or(CatalogError::Fenced)?; + if journal.transfer_archive() != publication.transfer_archive() { + return Err(CatalogError::Fenced); + } + let sealed = self + .transfer_source_marker(&plan, &journal, marker.route().history) + .await?; + if !marker.preserves_source_seal(&sealed) { + return Err(CatalogError::Fenced); + } + let birth = self.open_historical_birth(marker.route()).await?; + Ok(Some((birth, plan))) + }) + .await + } + + pub(crate) async fn bootstrap_managed_sealed_source( + &self, + birth: &CellGroupBirth, + log: &ReplicaLog, + host: &FleetHostAuthorization, + ) -> Result { + self.bounded(async { + let source = birth.route().history; + let group = log.cell_group(&source).map_err(CatalogError::Log)?; + let marker = group + .group_manifest() + .map_err(CatalogError::Log)? + .ok_or(CatalogError::LostStableState)?; + let (historical, plan) = self + .discover_managed_sealed_source(&marker) + .await? + .ok_or(CatalogError::Fenced)?; + if historical.route() != birth.route() { + return Err(CatalogError::Fenced); + } + let timeout = Duration::from_millis(self.limits.operation_timeout_ms); + let transition = host + .stop_catalog() + .reopen_managed_transition(self, plan.operation(), timeout) + .await + .map_err(CatalogError::from)? + .ok_or(CatalogError::Fenced)?; + if transition.plan.digest() != plan.digest() { + return Err(CatalogError::Fenced); + } + let (permit, authority) = host + .managed_source_authority(&transition, self, source, timeout) + .await + .map_err(CatalogError::from)?; + permit.check(self, &plan)?; + self.require_managed_source_voter(log, &group, &historical, host, &authority) + .await?; + Ok(CellGroupBootstrapPermit::managed_sealed_source( + group, + historical, + self.clone(), + log.clone(), + host.clone(), + )) + }) + .await + } + + async fn require_managed_source_voter( + &self, + log: &ReplicaLog, + group: &GroupLog, + birth: &CellGroupBirth, + host: &FleetHostAuthorization, + authority: &FleetGroupAuthorization, + ) -> Result<(BootBinding, u64), CatalogError> { + let boot = self.require_managed_group_boot(log, host).await?; + let cache = group.read().map_err(CatalogError::Log)?; + let marker = cache + .group_manifest + .as_ref() + .ok_or(CatalogError::LostStableState)?; + if !authority.check_birth(self, birth) + || marker.route() != birth.route() + || marker + .source_seal() + .is_none_or(|seal| cache.hard_state.commit < seal.position().index.get()) + || cache + .membership + .fleet_status() + .is_none_or(|status| &status.binding != authority.binding()) + || !authority.matches_replica(group.identity(), &cache.membership) + || !cache + .conf_state + .voters + .contains(&group.identity().member.get()) + { + return Err(CatalogError::Fenced); + } + Ok((boot, marker.membership_revision())) + } + + async fn transfer_source_marker( + &self, + plan: &DirectoryMutation, + journal: &TransitionJournal, + source: GroupHistory, + ) -> Result { + self.read_transition_source_holds( + plan, + journal, + journal.sealed_sources().ok_or(CatalogError::Fenced)?, + super::transition_sources::TransitionSourceCut::Sealed, + ) + .await? + .into_iter() + .find(|hold| hold.marker().route().history == source) + .map(|hold| hold.marker().clone()) + .ok_or(CatalogError::Fenced) + } +} + +impl RangeDirectory { + /// Admit the exact source tombstone only after every child activation has + /// been selected. The source remains sealed while the entry is pending. + pub async fn prepare_source_retirement( + &self, + plan: &DirectoryMutation, + source: GroupHistory, + log: &ReplicaLog, + ) -> Result { + self.unmanaged_birth()?; + self.prepare_source_retirement_inner(plan, source, log, None) + .await + } + + pub(crate) async fn prepare_managed_source_retirement( + &self, + plan: &DirectoryMutation, + source: GroupHistory, + log: &ReplicaLog, + host: &FleetHostAuthorization, + authority: (&ManagedTransitionPermit, &FleetGroupAuthorization), + deadline: u64, + ) -> Result { + self.prepare_source_retirement_inner( + plan, + source, + log, + Some(ManagedRetirementAdmission { + permit: authority.0, + host, + authority: authority.1, + deadline, + }), + ) + .await + } + + async fn prepare_source_retirement_inner( + &self, + plan: &DirectoryMutation, + source: GroupHistory, + log: &ReplicaLog, + managed: Option>, + ) -> Result { + self.check_transition_plan(plan)?; + self.bounded(async { + if managed.is_none() { + self.namespace.require_cell_group_unmanaged().await?; + } + self.transfer_activations_inner( + plan, + managed.as_ref().map(|admission| admission.permit), + ) + .await? + .ok_or(CatalogError::Fenced)?; + let publication = self + .confirmed_transfer_publication(plan) + .await? + .ok_or(CatalogError::Fenced)? + .journal_digest(); + let (view, _) = self.current().await?; + let journal = self + .matching_transition(&view.head, plan) + .await? + .filter(|journal| journal.phase() == TransitionPhase::Active) + .ok_or(CatalogError::Fenced)?; + let sealed = self.transfer_source_marker(plan, &journal, source).await?; + let route = sealed.route(); + let group = log.cell_group(&source).map_err(CatalogError::Log)?; + let marker = group + .group_manifest() + .map_err(CatalogError::Log)? + .ok_or(CatalogError::LostStableState)?; + if marker.route() != route || !marker.preserves_source_seal(&sealed) { + return Err(CatalogError::Fenced); + } + super::bootstrap::check_member(&group)?; + let engine = log.storage_instance().map_err(CatalogError::Log)?; + let identity = group.identity(); + self.namespace + .verify_node_storage_identity(identity.node, identity.disk, engine) + .await?; + let transition = TransitionId::new(plan.operation().bytes()) + .map_err(|_| CatalogError::Invalid("transfer operation"))?; + let entry = CellGroupEntry { + binding: GroupEntryBinding { + group: route.history, + route_epoch: route.epoch, + birth_digest: route.birth_digest, + schema_digest: route.schema_digest, + proposal: plan.operation(), + }, + command: GroupCommand::Retire { + transition, + publication, + }, + } + .encode(route, GroupEntryLimits::default()) + .map_err(|_| CatalogError::Invalid("source retirement entry"))?; + let now = log + .domain() + .mono_ms() + .map_err(|_| CatalogError::Unconfirmed)?; + let limit = now + .checked_add(self.limits.operation_timeout_ms) + .ok_or(CatalogError::Limit)?; + let birth = self.open_historical_birth(route).await?; + let (managed, deadline) = if let Some(admission) = managed { + let voter = self + .require_managed_source_voter( + log, + &group, + &birth, + admission.host, + admission.authority, + ) + .await?; + (Some(voter), limit.min(admission.deadline)) + } else { + (None, limit) + }; + if deadline <= now { + return Err(CatalogError::Unconfirmed); + } + Ok(CellGroupSourceRetirementPermit { + group, + engine, + sealed, + birth, + publication, + deadline, + entry, + managed, + }) + }) + .await + } +} + +const RETIREMENTS: u8 = 211; +const SETS: &str = "transfer-parent-retirement-sets"; +const PARENT_RECEIPT: u8 = 216; +const STAGED: &str = "transfer-parent-retirement-progress"; + +struct RetirementBasis { + journal: TransitionJournal, + publication: [u8; 32], + child_archives: [u8; 32], + parents: Vec, +} + +#[derive(Deserialize, Serialize)] +#[serde(deny_unknown_fields)] +struct RetirementSet { + namespace: [u8; 32], + plan: [u8; 32], + publication: [u8; 32], + child_archives: [u8; 32], + parents: Vec, +} + +#[derive(Deserialize, Serialize)] +#[serde(deny_unknown_fields)] +struct StagedRetirement { + plan: [u8; 32], + publication: [u8; 32], + receipt: RetirementReceipt, +} + +#[derive(PartialEq, Eq, Deserialize, Serialize)] +#[serde(deny_unknown_fields)] +struct RetirementReceipt { + group: [u8; 16], + incarnation: [u8; 16], + seal_index: u64, + seal_term: u64, + index: u64, + term: u64, + members: Vec, +} + +#[derive(Clone, Debug, PartialEq, Eq, Deserialize, Serialize)] +#[serde(deny_unknown_fields)] +pub(crate) struct RetiredSourceMember { + pub(crate) member: u64, + pub(crate) node: [u8; 16], + pub(crate) disk: Option<[u8; 16]>, +} + +pub(crate) struct ManagedRetiredSource { + pub(crate) route: GroupRoute, + pub(crate) members: Vec, +} + +fn final_source_members( + configuration: &::raft::prelude::ConfState, + membership: &GroupMembership, + birth: &CellGroupBirth, +) -> Result, RuntimeError> { + members(configuration) + .into_iter() + .map(|raw| { + let member = MemberId::new(raw).map_err(|_| RuntimeError::StaleCompletion)?; + let (node, disk) = if let Some((joined, _)) = membership.joined(member)? { + (joined.node, Some(joined.disk.bytes())) + } else { + let initial = birth + .voters() + .iter() + .chain(birth.learners()) + .find(|initial| initial.member == member) + .ok_or(RuntimeError::StaleCompletion)?; + (initial.node, None) + }; + Ok(RetiredSourceMember { + member: raw, + node: node.bytes(), + disk, + }) + }) + .collect() +} + +impl RetirementReceipt { + fn from_ack(ack: CellGroupSourceRetirementAck) -> Self { + let position = ack.tombstone.position(); + Self { + group: ack.route.history.group.bytes(), + incarnation: ack.route.history.incarnation.bytes(), + seal_index: ack.seal.index.get(), + seal_term: ack.seal.term.get(), + index: position.index.get(), + term: position.term.get(), + members: ack.members, + } + } + + fn position(&self) -> LogPosition { + LogPosition { + index: LogIndex::new(self.index), + term: Term::new(self.term), + } + } + + fn validate(&self, held: &GroupManifest) -> Result<(), CatalogError> { + let history = held.route().history; + let seal = held.source_seal().ok_or(CatalogError::Fenced)?.position(); + let position = self.position(); + if self.group != history.group.bytes() + || self.incarnation != history.incarnation.bytes() + || self.seal_index != seal.index.get() + || self.seal_term != seal.term.get() + || position.index <= held.position().index + || position.term < held.position().term + || self.members.is_empty() + || self.members.len() > crate::raft::member_state::MAX_MEMBERS + || self + .members + .windows(2) + .any(|pair| pair[0].member >= pair[1].member) + || self.members.iter().any(|member| { + MemberId::new(member.member).is_err() + || NodeId::new(member.node).is_err() + || member + .disk + .is_some_and(|disk| DiskLineage::new(disk).is_err()) + }) + { + return Err(CatalogError::Fenced); + } + Ok(()) + } +} + +impl RangeDirectory { + pub(crate) async fn managed_retired_sources( + &self, + plan: &DirectoryMutation, + permit: &ManagedTransitionPermit, + ) -> Result<(TransitionJournal, Vec), CatalogError> { + permit.check(self, plan)?; + self.bounded(async { + let basis = self.retirement_basis(plan).await?; + let digest = basis + .journal + .retired_parents() + .ok_or(CatalogError::Fenced)?; + let set = self.require_retirement_set(plan, &basis, digest).await?; + let sources = basis + .parents + .into_iter() + .zip(set.parents) + .map(|(hold, receipt)| ManagedRetiredSource { + route: hold.marker().route().clone(), + members: receipt.members, + }) + .collect(); + Ok((basis.journal, sources)) + }) + .await + } + + /// Reopen or persist a parent voter acknowledgement. Settlement requires + /// every parent and the independently verified child holds. + pub(crate) async fn resume_managed_source_retirement( + &self, + domain: &impl CellGroupArchiveExecution, + pool: &CellGroupArchivePool, + plan: &DirectoryMutation, + history: GroupHistory, + acknowledgement: Option, + permit: &ManagedTransitionPermit, + ) -> Result, CellGroupArchiveBindingError> { + permit.check(self, plan)?; + let basis = self.bounded(self.retirement_basis(plan)).await?; + let index = basis + .parents + .iter() + .position(|hold| hold.marker().route().history == history) + .ok_or(CatalogError::Fenced)?; + let receipt = acknowledgement + .map(|ack| Self::checked_retirement(&basis, index, ack)) + .transpose()?; + if let Some(selected) = basis.journal.retired_parents() { + let set = self + .bounded(self.require_retirement_set(plan, &basis, selected)) + .await?; + if receipt + .as_ref() + .is_some_and(|receipt| receipt != &set.parents[index]) + { + return Err(CatalogError::Conflict("selected retirement receipt changed").into()); + } + return Ok(Some((set.parents[index].position(), basis.journal))); + } + let (position, parents) = self + .bounded(async { + if let Some(receipt) = receipt { + let staged = StagedRetirement { + plan: plan.digest(), + publication: basis.publication, + receipt, + }; + let bytes = record::encode(PARENT_RECEIPT, &staged, wire::LIMIT)?; + let key = staged_retirement_key(plan, index); + let selected = self + .namespace + .inner + .create_and_observe_bounded(&self.key(&key), &bytes, wire::LIMIT) + .await?; + if selected != bytes { + return Err(CatalogError::Conflict("staged retirement receipt changed")); + } + } + let mut parents = Vec::with_capacity(basis.parents.len()); + let mut position = None; + for (slot, hold) in basis.parents.iter().enumerate() { + let Some(bytes) = self.read(&staged_retirement_key(plan, slot)).await? else { + continue; + }; + let staged: StagedRetirement = + record::decode(PARENT_RECEIPT, &bytes, wire::LIMIT)?; + if staged.plan != plan.digest() || staged.publication != basis.publication { + return Err(CatalogError::Corrupt("staged retirement receipt binding")); + } + staged.receipt.validate(hold.marker())?; + if slot == index { + position = Some(staged.receipt.position()); + } + parents.push(staged.receipt); + } + Ok((position, parents)) + }) + .await?; + let Some(position) = position else { + return Ok(None); + }; + let journal = if parents.len() == basis.parents.len() { + self.select_retirement(domain, pool, plan, basis, parents, Some(permit)) + .await? + } else { + basis.journal + }; + Ok(Some((position, journal))) + } + + fn checked_retirement( + basis: &RetirementBasis, + index: usize, + ack: CellGroupSourceRetirementAck, + ) -> Result { + let hold = &basis.parents[index]; + if ack.tombstone.publication() != basis.publication || &ack.route != hold.marker().route() { + return Err(CatalogError::Fenced); + } + let receipt = RetirementReceipt::from_ack(ack); + receipt.validate(hold.marker())?; + Ok(receipt) + } + + /// Record the durable parent tombstones after confirming the selected + /// child coverage is still held. Keep topology owned through hold release. + pub async fn settle_transfer_retirement( + &self, + domain: &impl CellGroupArchiveExecution, + pool: &CellGroupArchivePool, + plan: &DirectoryMutation, + acknowledgements: Vec, + ) -> Result { + self.unmanaged_birth()?; + self.settle_transfer_retirement_inner(domain, pool, plan, acknowledgements, None) + .await + } + + pub(crate) async fn settle_managed_transfer_retirement( + &self, + domain: &impl CellGroupArchiveExecution, + pool: &CellGroupArchivePool, + plan: &DirectoryMutation, + acknowledgements: Vec, + permit: &ManagedTransitionPermit, + ) -> Result { + permit.check(self, plan)?; + self.settle_transfer_retirement_inner(domain, pool, plan, acknowledgements, Some(permit)) + .await + } + + async fn settle_transfer_retirement_inner( + &self, + domain: &impl CellGroupArchiveExecution, + pool: &CellGroupArchivePool, + plan: &DirectoryMutation, + acknowledgements: Vec, + permit: Option<&ManagedTransitionPermit>, + ) -> Result { + self.check_transition_plan(plan)?; + if acknowledgements.len() != plan.request.parents.len() { + return Err(CatalogError::Invalid("incomplete transfer parent retirement").into()); + } + domain + .run(Box::pin(async { + if permit.is_none() { + self.namespace.require_cell_group_unmanaged().await?; + } + let basis = self.bounded(self.retirement_basis(plan)).await?; + let parents = acknowledgements + .into_iter() + .enumerate() + .map(|(index, ack)| Self::checked_retirement(&basis, index, ack)) + .collect::, _>>()?; + self.select_retirement(domain, pool, plan, basis, parents, permit) + .await + })) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } + + async fn select_retirement( + &self, + domain: &impl CellGroupArchiveExecution, + pool: &CellGroupArchivePool, + plan: &DirectoryMutation, + basis: RetirementBasis, + parents: Vec, + permit: Option<&ManagedTransitionPermit>, + ) -> Result { + let set = RetirementSet { + namespace: self.namespace.root_digest, + plan: plan.digest(), + publication: basis.publication, + child_archives: basis.child_archives, + parents, + }; + let bytes = record::encode(RETIREMENTS, &set, wire::LIMIT)?; + let digest = record::digest(&bytes); + if let Some(selected) = basis.journal.retired_parents() { + if selected != digest { + return Err(CatalogError::Conflict("transfer retirement set changed").into()); + } + self.bounded(self.require_retirement_set(plan, &basis, selected)) + .await?; + return Ok(basis.journal); + } + self.require_live_retirement_archives(domain, pool, plan, permit) + .await?; + self.bounded(self.immutable(SETS, digest, &bytes)).await?; + self.record_verified_retirement( + plan, + basis.publication, + basis.child_archives, + digest, + permit, + ) + .await + .map_err(Into::into) + } + + /// Release the exact transfer-owned parent holds after the Retired CAS. + /// Each archive CAS is idempotent. Retain topology ownership until all + /// releases are observed, then record completion and reopen the lane. + pub async fn release_retired_parent_holds( + &self, + domain: &impl CellGroupArchiveExecution, + pool: &CellGroupArchivePool, + plan: &DirectoryMutation, + ) -> Result { + self.unmanaged_birth()?; + self.release_retired_parent_holds_inner(domain, pool, plan, None) + .await + } + + pub(crate) async fn release_managed_retired_parent_holds( + &self, + domain: &impl CellGroupArchiveExecution, + pool: &CellGroupArchivePool, + plan: &DirectoryMutation, + permit: &ManagedTransitionPermit, + ) -> Result { + permit.check(self, plan)?; + self.release_retired_parent_holds_inner(domain, pool, plan, Some(permit)) + .await + } + + async fn release_retired_parent_holds_inner( + &self, + domain: &impl CellGroupArchiveExecution, + pool: &CellGroupArchivePool, + plan: &DirectoryMutation, + permit: Option<&ManagedTransitionPermit>, + ) -> Result { + self.check_transition_plan(plan)?; + domain + .run(Box::pin(async { + if permit.is_none() { + self.namespace.require_cell_group_unmanaged().await?; + } + let basis = self.bounded(self.retirement_basis(plan)).await?; + let digest = basis + .journal + .retired_parents() + .ok_or(CatalogError::Fenced)?; + self.bounded(self.require_retirement_set(plan, &basis, digest)) + .await?; + if basis.journal.parent_holds_released() { + return Ok(basis.journal); + } + let released = async { + self.require_live_retirement_archives(domain, pool, plan, permit) + .await?; + let preparation_holds = if permit.is_some() { + self.managed_preparation_holds(plan).await? + } else { + Vec::new() + }; + for hold in basis.parents.iter().chain(&preparation_holds) { + let registry = self + .open_retired_source_archive(domain, pool, hold.marker().route()) + .await?; + registry.release(domain, hold).await?; + } + self.record_released_parent_holds(plan, digest, permit) + .await + .map_err(Into::into) + } + .await; + if released.is_err() { + // Another controller may have completed cleanup and a later + // topology owner may already have replaced a child. + if let Ok(current) = self.bounded(self.retirement_basis(plan)).await { + if current.journal.parent_holds_released() + && current.journal.retired_parents() == Some(digest) + { + return Ok(current.journal); + } + } + } + released + })) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } + + async fn retirement_basis( + &self, + plan: &DirectoryMutation, + ) -> Result { + let (view, _) = self.current().await?; + let journal = self + .matching_transition(&view.head, plan) + .await? + .filter(|journal| { + matches!( + journal.phase(), + TransitionPhase::Active | TransitionPhase::Retired + ) + }) + .ok_or(CatalogError::Fenced)?; + let publication = self + .confirmed(&view.head, plan) + .await? + .filter(|publication| publication.transfer_archive().is_some()) + .ok_or(CatalogError::Fenced)? + .journal_digest(); + let child_archives = journal.child_archives().ok_or(CatalogError::Fenced)?; + let parents = self + .read_transition_source_holds( + plan, + &journal, + journal.sealed_sources().ok_or(CatalogError::Fenced)?, + super::transition_sources::TransitionSourceCut::Sealed, + ) + .await?; + Ok(RetirementBasis { + journal, + publication, + child_archives, + parents, + }) + } + + async fn require_retirement_set( + &self, + plan: &DirectoryMutation, + basis: &RetirementBasis, + digest: [u8; 32], + ) -> Result { + let bytes = self + .read(&format!("{SETS}/{}", hex::encode(digest))) + .await? + .ok_or(CatalogError::Corrupt("missing transfer retirement set"))?; + if record::digest(&bytes) != digest { + return Err(CatalogError::Corrupt("transfer retirement set digest")); + } + let set: RetirementSet = record::decode(RETIREMENTS, &bytes, wire::LIMIT)?; + if set.namespace != self.namespace.root_digest + || set.plan != plan.digest() + || set.publication != basis.publication + || set.child_archives != basis.child_archives + || set.parents.len() != basis.parents.len() + { + return Err(CatalogError::Corrupt("transfer retirement set binding")); + } + for (receipt, hold) in set.parents.iter().zip(&basis.parents) { + receipt.validate(hold.marker())?; + } + Ok(set) + } + + async fn require_live_retirement_archives( + &self, + domain: &impl CellGroupArchiveExecution, + pool: &CellGroupArchivePool, + plan: &DirectoryMutation, + permit: Option<&ManagedTransitionPermit>, + ) -> Result<(), CellGroupArchiveBindingError> { + let children = match permit { + Some(permit) => self.managed_retirement_child_holds(plan, permit).await?, + None => self.retirement_child_holds(plan).await?, + } + .ok_or(CatalogError::Fenced)?; + for hold in children { + let birth = self + .materialize_selected_history(hold.marker().route().history) + .await?; + let registry = self.open_group_archive(domain, pool, &birth).await?; + registry.require_retained_hold(domain, &hold).await?; + } + Ok(()) + } +} + +fn staged_retirement_key(plan: &DirectoryMutation, index: usize) -> String { + format!("{STAGED}/{}/{index}", hex::encode(plan.digest())) +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/transition_source_slots.rs b/crates/cellar/durability/raft/cell_groups/directory/transition_source_slots.rs new file mode 100644 index 000000000..d32da212a --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/transition_source_slots.rs @@ -0,0 +1,272 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Conditional ownership of archive acquisitions during a range transition. + +use super::transition_sources::TransitionSourceCut; +use super::*; +use crate::raft::{ + cell_groups::{ + CellGroupArchiveAcquireOutcome, CellGroupArchiveError, CellGroupArchiveExecution, + CellGroupArchiveHold, CellGroupArchiveIntent, CellGroupArchivePool, + CellGroupArchiveRegistry, GroupManifest, + }, + fleet_catalog::ManagedTransitionPermit, +}; +use serde::{Deserialize, Serialize}; + +const STAGED: &str = "transition-source-progress"; +const KIND: u8 = 215; + +type Result = std::result::Result; + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) enum ArchiveSlot { + Acquiring(CellGroupArchiveIntent), + Verified(CellGroupArchiveHold), +} + +impl RangeDirectory { + /// Root an exact acquire before touching the archive registry. An already + /// verified slot returns its hold with no further physical work required. + pub(crate) async fn acquire_managed_sealed_source( + &self, + domain: &impl CellGroupArchiveExecution, + archives: &CellGroupArchivePool, + plan: &DirectoryMutation, + permit: &ManagedTransitionPermit, + history: GroupHistory, + ) -> Result<(CellGroupArchiveHold, Option)> { + permit.check(self, plan)?; + let (parents, _) = self.transition_routes(plan)?; + let (index, route) = parents + .iter() + .enumerate() + .find(|(_, route)| route.history == history) + .ok_or(CatalogError::Fenced)?; + let (journal, staged) = self + .bounded(async { + let (view, _) = self.current().await?; + let journal = self + .matching_transition(&view.head, plan) + .await? + .ok_or(CatalogError::Fenced)?; + journal.seal_authorization()?; + self.selected_route_in(&view, route).await?; + let staged = self.read_source_slot(plan, index).await?; + Ok((journal, staged)) + }) + .await?; + let staged = match staged { + Some((ArchiveSlot::Verified(hold), _)) => { + self.check_sealed_hold(plan, &journal, route, &hold)?; + return Ok((hold, None)); + } + Some((ArchiveSlot::Acquiring(intent), version)) => Some((intent, version)), + None => None, + }; + if journal.phase() != TransitionPhase::SealAuthorized { + return Err(CatalogError::Fenced.into()); + } + + let birth = self.materialize_selected_history(history).await?; + if birth.route() != route { + return Err(CatalogError::Fenced.into()); + } + let registry = self.open_group_archive(domain, archives, &birth).await?; + let (hold, intent) = self + .acquire_transition_archive_hold( + domain, + ®istry, + &source_slot_key(plan, index), + staged, + |marker| self.check_sealed_marker(plan, &journal, route, marker), + ) + .await?; + self.check_sealed_hold(plan, &journal, route, &hold)?; + Ok((hold, Some(intent))) + } + + /// Root the exact acquire before issuing it. An expired intent can be + /// replaced only after the registry proves its nonce owns no hold. + pub(super) async fn acquire_transition_archive_hold( + &self, + domain: &impl CellGroupArchiveExecution, + registry: &CellGroupArchiveRegistry, + key: &str, + staged: Option<(CellGroupArchiveIntent, UpdateVersion)>, + check: impl Fn(&GroupManifest) -> Result<()> + Send + Sync, + ) -> Result<(CellGroupArchiveHold, CellGroupArchiveIntent)> { + let (intent, version) = match staged { + Some(staged) => staged, + None => { + let intent = registry.prepare_acquire(domain).await?; + check(intent.acquire_marker()?)?; + let version = self + .bounded(self.write_archive_slot( + key, + None, + &ArchiveSlot::Acquiring(intent.clone()), + )) + .await?; + (intent, version) + } + }; + check(intent.acquire_marker()?)?; + let hold = match registry.acquire(domain, &intent).await { + Ok(hold) => hold, + Err(error @ (CellGroupArchiveError::Expired | CellGroupArchiveError::Conflict)) => { + match registry.acquire_outcome(domain, &intent).await? { + CellGroupArchiveAcquireOutcome::Held(hold) => *hold, + CellGroupArchiveAcquireOutcome::Pending => return Err(error.into()), + CellGroupArchiveAcquireOutcome::Superseded => { + let next = registry.prepare_acquire(domain).await?; + check(next.acquire_marker()?)?; + self.bounded(self.write_archive_slot( + key, + Some(version), + &ArchiveSlot::Acquiring(next), + )) + .await?; + return Err(CatalogError::Unconfirmed.into()); + } + } + } + Err(error) => return Err(error.into()), + }; + Ok((hold, intent)) + } + + pub(super) fn check_sealed_hold( + &self, + plan: &DirectoryMutation, + journal: &TransitionJournal, + route: &GroupRoute, + hold: &CellGroupArchiveHold, + ) -> std::result::Result<(), CatalogError> { + hold.check_reference() + .map_err(|_| CatalogError::Corrupt("staged source hold"))?; + self.check_source_marker( + plan, + journal, + route, + hold.marker(), + TransitionSourceCut::Sealed, + ) + } + + pub(super) fn check_sealed_intent( + &self, + plan: &DirectoryMutation, + journal: &TransitionJournal, + route: &GroupRoute, + intent: &CellGroupArchiveIntent, + ) -> Result<()> { + self.check_sealed_marker(plan, journal, route, intent.acquire_marker()?) + } + + fn check_sealed_marker( + &self, + plan: &DirectoryMutation, + journal: &TransitionJournal, + route: &GroupRoute, + marker: &GroupManifest, + ) -> Result<()> { + self.check_source_marker(plan, journal, route, marker, TransitionSourceCut::Sealed) + .map_err(|error| match error { + // Archive publication can lag the already applied seal. + CatalogError::Fenced if marker.route() == route && marker.is_active() => { + CatalogError::Unconfirmed.into() + } + error => error.into(), + }) + } + + pub(super) async fn claim_staged_source_hold( + &self, + plan: &DirectoryMutation, + index: usize, + hold: &CellGroupArchiveHold, + intent: Option<&CellGroupArchiveIntent>, + ) -> std::result::Result<(), CatalogError> { + let version = match (self.read_source_slot(plan, index).await?, intent) { + (Some((ArchiveSlot::Verified(selected), _)), _) if &selected == hold => return Ok(()), + (Some((ArchiveSlot::Acquiring(selected), version)), Some(intent)) + if &selected == intent => + { + Some(version) + } + (None, None) => None, + _ => return Err(CatalogError::Conflict("staged source hold owner changed")), + }; + self.write_archive_slot( + &source_slot_key(plan, index), + version, + &ArchiveSlot::Verified(hold.clone()), + ) + .await?; + Ok(()) + } + + pub(super) async fn read_source_slot( + &self, + plan: &DirectoryMutation, + index: usize, + ) -> std::result::Result, CatalogError> { + self.read_archive_slot(&source_slot_key(plan, index)).await + } + + pub(super) async fn read_archive_slot( + &self, + key: &str, + ) -> std::result::Result, CatalogError> { + self.namespace + .inner + .read_versioned_bounded(&self.key(key), wire::LIMIT) + .await? + .map(|(bytes, version)| { + if version.e_tag.is_none() && version.version.is_none() { + return Err(CatalogError::Corrupt("source slot conditional version")); + } + Ok((record::decode(KIND, &bytes, wire::LIMIT)?, version)) + }) + .transpose() + } + + async fn write_archive_slot( + &self, + suffix: &str, + version: Option, + slot: &ArchiveSlot, + ) -> std::result::Result { + let bytes = record::encode(KIND, slot, wire::LIMIT)?; + let key = self.key(suffix); + let store = &self.namespace.inner; + if let Some(version) = version { + let _ = store + .store + .put_opts( + &key, + bytes.into(), + PutOptions::from(PutMode::Update(version)), + ) + .await; + } else { + store + .create_and_observe_bounded(&key, &bytes, wire::LIMIT) + .await?; + } + let (observed, version) = self + .read_archive_slot(suffix) + .await? + .ok_or(CatalogError::Unconfirmed)?; + if &observed != slot { + return Err(CatalogError::Conflict("staged source hold owner changed")); + } + Ok(version) + } +} + +fn source_slot_key(plan: &DirectoryMutation, index: usize) -> String { + format!("{STAGED}/{}/{index}", hex::encode(plan.digest())) +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/transition_sources.rs b/crates/cellar/durability/raft/cell_groups/directory/transition_sources.rs new file mode 100644 index 000000000..e01c69dc0 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/transition_sources.rs @@ -0,0 +1,584 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Independently replay each held parent closure before rooting the sealed set. +use super::{transition_source_slots::ArchiveSlot, *}; +use crate::raft::cell_groups::{ + CellGroupArchiveCheckpointReceiver, CellGroupArchiveCoverage, CellGroupArchiveExecution, + CellGroupArchiveHold, CellGroupArchiveIntent, CellGroupArchivePool, CellGroupArchiveTailPool, + GroupManifest, +}; +use crate::raft::fleet_catalog::ManagedTransitionPermit; +use serde::{Deserialize, Serialize}; +use sha2::{Digest, Sha256}; + +const SOURCE_HOLD: u8 = 199; +const SOURCE_SET: u8 = 200; +const HOLDS: &str = "transition-source-holds"; +const SETS: &str = "transition-source-sets"; + +type Result = std::result::Result; + +/// The explicit archive hold belongs to the transition owner until retirement. +/// The receiver must have its own physical recovery admission and stores. +pub struct CellGroupTransitionSource { + pub birth: CellGroupBirth, + pub hold: CellGroupArchiveHold, + pub receiver: CellGroupArchiveCheckpointReceiver, +} + +/// Independently recovered source images and their selected catalog decision. +/// This grants no child boot, installation, publication or serving authority. +#[derive(Clone)] +pub struct VerifiedCellGroupTransitionSources { + operation: ProposalId, + plan: [u8; 32], + journal: TransitionJournal, + sources: Vec, + children: Vec, +} + +impl VerifiedCellGroupTransitionSources { + pub(crate) fn transition_plan(&self) -> (ProposalId, [u8; 32]) { + (self.operation, self.plan) + } + + pub fn journal(&self) -> &TransitionJournal { + &self.journal + } + + pub fn sources(&self) -> &[CellGroupArchiveCoverage] { + &self.sources + } + + pub fn children(&self) -> &[GroupRoute] { + &self.children + } +} + +#[derive(Clone, Copy)] +pub(super) enum TransitionSourceCut { + Active, + Sealed, +} + +pub(super) struct VerifiedTransitionSourceSet { + pub(super) journal: TransitionJournal, + pub(super) digest: [u8; 32], + pub(super) sources: Vec, + pub(super) children: Vec, +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct SourceSet { + namespace: [u8; 32], + request: [u8; 32], + holds: Vec<[u8; 32]>, +} + +struct EncodedSourceSet { + bytes: Vec, + holds: Vec<([u8; 32], Vec)>, +} + +impl RangeDirectory { + pub(super) fn check_source_marker( + &self, + mutation: &DirectoryMutation, + journal: &TransitionJournal, + route: &GroupRoute, + marker: &GroupManifest, + cut: TransitionSourceCut, + ) -> std::result::Result<(), CatalogError> { + if marker.route() != route { + return Err(CatalogError::Fenced); + } + if matches!(cut, TransitionSourceCut::Active) { + return if marker.is_active() { + Ok(()) + } else { + Err(CatalogError::Fenced) + }; + } + let (transition, authorization, entry) = + self.source_seal_entry(mutation, journal, route)?; + let payload = Sha256::digest(&entry).into(); + if marker + .source_seal() + .is_none_or(|seal| !seal.matches(transition, authorization, payload)) + { + return Err(CatalogError::Fenced); + } + Ok(()) + } + + /// Select the exact active hold set before opening any physical receivers. + /// The caller supplies its own holds and retains responsibility for their + /// cleanup if the plan is cancelled. Selection and cancellation share a CAS. + pub(crate) async fn stage_managed_preparation_holds( + &self, + domain: &impl CellGroupArchiveExecution, + archives: &CellGroupArchivePool, + plan: &DirectoryMutation, + permit: &ManagedTransitionPermit, + holds: &[CellGroupArchiveHold], + acquires: Option<[u8; 32]>, + ) -> Result { + permit.check(self, plan)?; + let (parents, _) = self.transition_routes(plan)?; + if parents.len() != holds.len() { + return Err(CatalogError::Invalid("incomplete transition preparation inputs").into()); + } + let (view, journal) = self + .bounded(async { + let (view, _) = self.current().await?; + let journal = self + .matching_transition(&view.head, plan) + .await? + .filter(|journal| { + matches!( + journal.phase(), + TransitionPhase::Planned | TransitionPhase::Prepared + ) + }) + .ok_or(CatalogError::Fenced)?; + Ok((view, journal)) + }) + .await?; + if journal.preparation_acquires() != acquires { + return Err(CatalogError::Conflict("transition preparation hold owner changed").into()); + } + let set = self.encode_transition_source_set(plan, holds.iter())?; + let digest = record::digest(&set.bytes); + if journal + .preparation_sources() + .is_some_and(|selected| selected != digest) + { + return Err(CatalogError::Conflict("transition preparation inputs changed").into()); + } + for (route, hold) in parents.iter().zip(holds) { + self.bounded(self.selected_route_in(&view, route)).await?; + self.check_source_marker( + plan, + &journal, + route, + hold.marker(), + TransitionSourceCut::Active, + )?; + let birth = self.materialize_selected_history(route.history).await?; + if birth.route() != route { + return Err(CatalogError::Fenced.into()); + } + let registry = self.open_group_archive(domain, archives, &birth).await?; + registry.require_retained_hold(domain, hold).await?; + } + self.persist_transition_source_set(digest, set).await?; + self.record_managed_preparation_sources(plan, permit, digest, acquires) + .await + .map_err(Into::into) + } + + /// Reopen the staged input descriptors, including after cancellation. + /// A descriptor alone supplies neither a physical pin nor seal authority. + pub(crate) async fn staged_managed_preparation_holds( + &self, + plan: &DirectoryMutation, + ) -> Result>> { + if self.namespace_config().managed_fleet.is_none() { + return Err(CatalogError::ManagedAdmissionRequired.into()); + } + self.check_transition_plan(plan)?; + self.bounded(async { + let (view, _) = self.current().await?; + let journal = self + .matching_transition(&view.head, plan) + .await? + .ok_or(CatalogError::Fenced)?; + let Some(digest) = journal.preparation_sources() else { + return Ok(None); + }; + self.read_transition_source_holds(plan, &journal, digest, TransitionSourceCut::Active) + .await + .map(Some) + }) + .await + .map_err(Into::into) + } + + fn encode_transition_source_set<'a>( + &self, + plan: &DirectoryMutation, + holds: impl Iterator, + ) -> Result { + let holds = holds + .map(|hold| { + hold.check_reference()?; + let bytes = record::encode(SOURCE_HOLD, hold, wire::LIMIT)?; + Ok((record::digest(&bytes), bytes)) + }) + .collect::>>()?; + let set = SourceSet { + namespace: self.namespace.root_digest, + request: plan.digest, + holds: holds.iter().map(|(digest, _)| *digest).collect(), + }; + Ok(EncodedSourceSet { + bytes: record::encode(SOURCE_SET, &set, wire::LIMIT)?, + holds, + }) + } + + async fn persist_transition_source_set( + &self, + digest: [u8; 32], + set: EncodedSourceSet, + ) -> std::result::Result<(), CatalogError> { + self.bounded(async { + for (id, bytes) in set.holds { + self.immutable(HOLDS, id, &bytes).await?; + } + self.immutable(SETS, digest, &set.bytes).await + }) + .await + } + + /// Select a single verified parent hold for this irreversible operation. + /// Independent slots allow different parents to finish in either order. + pub(crate) async fn stage_managed_sealed_source( + &self, + domain: &impl CellGroupArchiveExecution, + archives: &CellGroupArchivePool, + tails: &CellGroupArchiveTailPool, + plan: &DirectoryMutation, + source: (CellGroupTransitionSource, Option), + permit: &ManagedTransitionPermit, + ) -> Result<()> { + let (source, intent) = source; + permit.check(self, plan)?; + let (parents, _) = self.transition_routes(plan)?; + let route = source.birth.route(); + let index = parents + .iter() + .position(|parent| parent == route) + .ok_or(CatalogError::Fenced)?; + let journal = self + .bounded(async { + let (view, _) = self.current().await?; + let journal = self + .matching_transition(&view.head, plan) + .await? + .filter(|journal| journal.phase() == TransitionPhase::SealAuthorized) + .ok_or(CatalogError::Fenced)?; + self.selected_route_in(&view, route).await?; + Ok(journal) + }) + .await?; + source.hold.check_reference()?; + self.check_source_marker( + plan, + &journal, + route, + source.hold.marker(), + TransitionSourceCut::Sealed, + )?; + let registry = self + .open_group_archive(domain, archives, &source.birth) + .await?; + // Keep the nested archive replay out of the staging future's + // stack frame while the object-store HTTP client is being polled. + let coverage = + Box::pin(registry.verify_hold(domain, &source.hold, tails, source.receiver)).await?; + self.check_source_marker( + plan, + &journal, + route, + coverage.marker(), + TransitionSourceCut::Sealed, + )?; + registry + .require_retained_hold(domain, coverage.hold()) + .await?; + self.bounded(self.claim_staged_source_hold(plan, index, coverage.hold(), intent.as_ref())) + .await?; + Ok(()) + } + + /// Bounded plan-order progress for an already admitted managed operation. + pub(crate) async fn managed_staged_source_holds( + &self, + plan: &DirectoryMutation, + ) -> Result)>> { + if self.namespace_config().managed_fleet.is_none() { + return Err(CatalogError::ManagedAdmissionRequired.into()); + } + let (parents, _) = self.transition_routes(plan)?; + self.bounded(async { + let (view, _) = self.current().await?; + let journal = self + .matching_transition(&view.head, plan) + .await? + .ok_or(CatalogError::Fenced)?; + journal.seal_authorization()?; + let mut holds = Vec::with_capacity(parents.len()); + for (index, route) in parents.into_iter().enumerate() { + let hold = match self.read_source_slot(plan, index).await? { + Some((ArchiveSlot::Verified(hold), _)) => { + self.check_sealed_hold(plan, &journal, &route, &hold)?; + Some(hold) + } + Some((ArchiveSlot::Acquiring(intent), _)) => { + self.check_sealed_intent(plan, &journal, &route, &intent) + .map_err(|_| CatalogError::Corrupt("staged source acquire"))?; + None + } + None => None, + }; + holds.push((route, hold)); + } + Ok(holds) + }) + .await + .map_err(Into::into) + } + + /// Verify every exact held parent cut in plan order, then select the whole + /// set in the namespace CAS. Persist the hold allocations before calling. + /// Keep those remote holds until the transition owner retires all users. + /// Retry with the identical set after a lost response and independently + /// verify the restored holds again before using their data. On reopen, + /// materialize the selected births with this directory before retrying. + pub async fn verify_and_record_sealed_sources( + &self, + domain: &impl CellGroupArchiveExecution, + archives: &CellGroupArchivePool, + tails: &CellGroupArchiveTailPool, + mutation: &DirectoryMutation, + sources: Vec, + ) -> Result { + self.unmanaged_birth()?; + self.verify_and_record_sealed_sources_inner( + domain, archives, tails, mutation, sources, None, + ) + .await + } + + pub(crate) async fn verify_and_record_managed_sealed_sources( + &self, + domain: &impl CellGroupArchiveExecution, + archives: &CellGroupArchivePool, + tails: &CellGroupArchiveTailPool, + mutation: &DirectoryMutation, + sources: Vec, + permit: &ManagedTransitionPermit, + ) -> Result { + permit.check(self, mutation)?; + self.verify_and_record_sealed_sources_inner( + domain, + archives, + tails, + mutation, + sources, + Some(permit), + ) + .await + } + + pub(super) async fn verify_and_record_sealed_sources_inner( + &self, + domain: &impl CellGroupArchiveExecution, + archives: &CellGroupArchivePool, + tails: &CellGroupArchiveTailPool, + mutation: &DirectoryMutation, + sources: Vec, + permit: Option<&ManagedTransitionPermit>, + ) -> Result { + domain + .run(Box::pin(async { + let verified = Box::pin(self.verify_transition_source_set( + domain, + archives, + tails, + mutation, + sources, + TransitionSourceCut::Sealed, + )) + .await?; + if permit.is_some() && verified.journal.phase() == TransitionPhase::SealAuthorized { + self.bounded(async { + for (index, coverage) in verified.sources.iter().enumerate() { + self.claim_staged_source_hold(mutation, index, coverage.hold(), None) + .await?; + } + Ok(()) + }) + .await?; + } + let journal = self + .record_verified_sources(mutation, verified.digest, permit) + .await?; + Ok(VerifiedCellGroupTransitionSources { + operation: mutation.operation, + plan: mutation.digest, + journal, + sources: verified.sources, + children: verified.children, + }) + })) + .await + .map_err(|_| crate::raft::cell_groups::CellGroupArchiveError::Stopped)? + } + + pub(super) async fn verify_transition_source_set( + &self, + domain: &impl CellGroupArchiveExecution, + archives: &CellGroupArchivePool, + tails: &CellGroupArchiveTailPool, + mutation: &DirectoryMutation, + sources: Vec, + cut: TransitionSourceCut, + ) -> Result { + let (parents, children) = self.transition_routes(mutation)?; + if sources.len() != parents.len() { + return Err(CatalogError::Invalid("incomplete transition source set").into()); + } + let (view, journal) = self + .bounded(async { + let (view, _) = self.current().await?; + let journal = self + .matching_transition(&view.head, mutation) + .await? + .ok_or(CatalogError::Fenced)?; + match cut { + TransitionSourceCut::Active + if !matches!( + journal.phase(), + TransitionPhase::Planned | TransitionPhase::Prepared + ) => + { + return Err(CatalogError::Fenced) + } + TransitionSourceCut::Sealed => { + journal.seal_authorization()?; + } + TransitionSourceCut::Active => (), + } + Ok((view, journal)) + }) + .await?; + for route in &parents { + self.bounded(self.selected_route_in(&view, route)).await?; + } + for (route, source) in parents.iter().zip(&sources) { + if source.birth.route() != route { + return Err(CatalogError::Fenced.into()); + } + self.check_source_marker(mutation, &journal, route, source.hold.marker(), cut)?; + } + let set = + self.encode_transition_source_set(mutation, sources.iter().map(|source| &source.hold))?; + let digest = record::digest(&set.bytes); + let selected = match cut { + TransitionSourceCut::Active => journal.preparation_sources(), + TransitionSourceCut::Sealed => journal.sealed_sources(), + }; + if selected.is_some_and(|selected| selected != digest) { + return Err(CatalogError::Conflict("transition source set changed").into()); + } + let mut verified = Vec::with_capacity(sources.len()); + let mut registries = Vec::with_capacity(sources.len()); + for source in sources { + let registry = self + .open_group_archive(domain, archives, &source.birth) + .await?; + let coverage = registry + .verify_hold(domain, &source.hold, tails, source.receiver) + .await?; + self.check_source_marker( + mutation, + &journal, + source.birth.route(), + coverage.marker(), + cut, + )?; + verified.push(coverage); + registries.push(registry); + } + self.persist_transition_source_set(digest, set).await?; + for (registry, coverage) in registries.iter().zip(&verified) { + registry + .require_retained_hold(domain, coverage.hold()) + .await?; + } + Ok(VerifiedTransitionSourceSet { + journal, + digest, + sources: verified, + children, + }) + } + + /// Reopen the original hold descriptors after a lost controller. They are + /// references to verify again, never recovered images or install authority. + pub async fn sealed_source_holds( + &self, + mutation: &DirectoryMutation, + ) -> Result> { + self.check_transition_plan(mutation)?; + self.bounded(async { + let (view, _) = self.current().await?; + let journal = self + .matching_transition(&view.head, mutation) + .await? + .ok_or(CatalogError::Fenced)?; + let digest = journal.sealed_sources().ok_or(CatalogError::Fenced)?; + self.read_transition_source_holds( + mutation, + &journal, + digest, + TransitionSourceCut::Sealed, + ) + .await + }) + .await + .map_err(Into::into) + } + + pub(super) async fn read_transition_source_holds( + &self, + mutation: &DirectoryMutation, + journal: &TransitionJournal, + digest: [u8; 32], + cut: TransitionSourceCut, + ) -> std::result::Result, CatalogError> { + let bytes = self + .read(&format!("{SETS}/{}", hex::encode(digest))) + .await? + .ok_or(CatalogError::Corrupt("missing transition source set"))?; + if record::digest(&bytes) != digest { + return Err(CatalogError::Corrupt("transition source set digest")); + } + let set: SourceSet = record::decode(SOURCE_SET, &bytes, wire::LIMIT)?; + if set.namespace != self.namespace.root_digest + || set.request != mutation.digest + || set.holds.len() != mutation.request.parents.len() + { + return Err(CatalogError::Corrupt("transition source set binding")); + } + let mut holds = Vec::with_capacity(set.holds.len()); + for (parent, id) in mutation.request.parents.iter().zip(set.holds) { + let bytes = self + .read(&format!("{HOLDS}/{}", hex::encode(id))) + .await? + .ok_or(CatalogError::Corrupt("missing transition source hold"))?; + if record::digest(&bytes) != id { + return Err(CatalogError::Corrupt("transition source hold digest")); + } + let hold: CellGroupArchiveHold = record::decode(SOURCE_HOLD, &bytes, wire::LIMIT)?; + hold.check_reference() + .map_err(|_| CatalogError::Corrupt("transition source hold"))?; + self.check_source_marker(mutation, journal, &parent.decode(self)?, hold.marker(), cut)?; + holds.push(hold); + } + Ok(holds) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/tree.rs b/crates/cellar/durability/raft/cell_groups/directory/tree.rs new file mode 100644 index 000000000..b3ca5e197 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/tree.rs @@ -0,0 +1,209 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Persistent ordered AVL pages. Intermediate edits may have gaps; only a +//! complete all-domain tree can enter a published namespace root. + +use super::{wire::*, *}; +use futures_util::future::BoxFuture; + +impl RangeDirectory { + pub(super) async fn page(&self, reference: &Reference) -> Result { + reference.range(self)?; + let bytes = self.range_bytes(reference.digest).await?; + let page: Page = record::decode(PAGE, &bytes, LIMIT)?; + if page.reference(&bytes, self)? != *reference { + return Err(CatalogError::Corrupt("range page reference")); + } + Ok(page) + } + + async fn save(&self, page: Page) -> Result { + let bytes = record::encode(PAGE, &page, LIMIT)?; + let reference = page.reference(&bytes, self)?; + self.immutable("ranges", reference.digest, &bytes).await?; + Ok(reference) + } + + pub(super) async fn lookup_tree( + &self, + root: &Reference, + cell: &CellId, + ) -> Result { + root.require_complete(self)?; + let key = cell_routing_key(self.namespace.config.namespace, cell); + let mut next = Some(root.clone()); + for _ in 0..32 { + let page = self + .page( + &next + .take() + .ok_or(CatalogError::Corrupt("range coverage hole"))?, + ) + .await?; + let route = page.route.decode(self)?; + if route.range.contains(&key) { + return Ok(route); + } + next = if CellBound::Key(key.clone()) < *route.range.lower() { + page.left + } else { + page.right + }; + } + Err(CatalogError::Limit) + } + + pub(super) fn insert<'a>( + &'a self, + root: Option, + route: Route, + depth: u8, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + if depth >= 32 { + return Err(CatalogError::Limit); + } + let Some(reference) = root else { + return self + .save(Page { + namespace: self.namespace.root_digest, + route, + left: None, + right: None, + }) + .await; + }; + let mut page = self.page(&reference).await?; + match route + .decode(self)? + .range + .lower() + .cmp(page.route.decode(self)?.range.lower()) + { + std::cmp::Ordering::Less => { + page.left = Some(self.insert(page.left.take(), route, depth + 1).await?) + } + std::cmp::Ordering::Greater => { + page.right = Some(self.insert(page.right.take(), route, depth + 1).await?) + } + std::cmp::Ordering::Equal => { + return Err(CatalogError::Conflict("duplicate range lower bound")) + } + } + self.balance(page).await + }) + } + + pub(super) fn remove<'a>( + &'a self, + root: Option, + expected: Route, + depth: u8, + ) -> BoxFuture<'a, Result, CatalogError>> { + Box::pin(async move { + if depth >= 32 { + return Err(CatalogError::Limit); + } + let reference = root.ok_or(CatalogError::Conflict("missing parent range"))?; + let mut page = self.page(&reference).await?; + match expected + .decode(self)? + .range + .lower() + .cmp(page.route.decode(self)?.range.lower()) + { + std::cmp::Ordering::Less => { + page.left = self.remove(page.left.take(), expected, depth + 1).await? + } + std::cmp::Ordering::Greater => { + page.right = self.remove(page.right.take(), expected, depth + 1).await? + } + std::cmp::Ordering::Equal => { + if page.route != expected { + return Err(CatalogError::Conflict("stale parent range")); + } + match (&page.left, &page.right) { + (None, _) => return Ok(page.right), + (_, None) => return Ok(page.left), + (Some(_), Some(right)) => { + let mut successor = self.page(right).await?; + for _ in depth..32 { + match &successor.left { + None => break, + Some(left) => successor = self.page(left).await?, + } + } + if successor.left.is_some() { + return Err(CatalogError::Limit); + } + page.route = successor.route; + page.right = self + .remove(page.right.take(), page.route.clone(), depth + 1) + .await?; + } + } + } + } + Ok(Some(self.balance(page).await?)) + }) + } + + async fn balance(&self, mut page: Page) -> Result { + let height = |r: &Option| r.as_ref().map_or(0, |r| i16::from(r.height)); + let balance = height(&page.left) - height(&page.right); + if balance > 1 { + let mut left = self + .page( + page.left + .as_ref() + .ok_or(CatalogError::Corrupt("range rotation"))?, + ) + .await?; + if height(&left.left) < height(&left.right) { + let mut middle = self + .page( + left.right + .as_ref() + .ok_or(CatalogError::Corrupt("range rotation"))?, + ) + .await?; + left.right = middle.left.take(); + middle.left = Some(self.save(left).await?); + page.left = middle.right.take(); + middle.right = Some(self.save(page).await?); + return self.save(middle).await; + } + page.left = left.right.take(); + left.right = Some(self.save(page).await?); + return self.save(left).await; + } + if balance < -1 { + let mut right = self + .page( + page.right + .as_ref() + .ok_or(CatalogError::Corrupt("range rotation"))?, + ) + .await?; + if height(&right.right) < height(&right.left) { + let mut middle = self + .page( + right + .left + .as_ref() + .ok_or(CatalogError::Corrupt("range rotation"))?, + ) + .await?; + right.left = middle.right.take(); + middle.right = Some(self.save(right).await?); + page.right = middle.left.take(); + middle.left = Some(self.save(page).await?); + return self.save(middle).await; + } + page.right = right.left.take(); + right.left = Some(self.save(page).await?); + return self.save(right).await; + } + self.save(page).await + } +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/wire.rs b/crates/cellar/durability/raft/cell_groups/directory/wire.rs new file mode 100644 index 000000000..810228694 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/wire.rs @@ -0,0 +1,337 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use serde::{Deserialize, Serialize}; + +pub(super) const PAGE: u8 = 81; +pub(super) const HEAD: u8 = 82; +pub(super) const JOURNAL: u8 = 83; +pub(super) const PIN: u8 = 84; +pub(super) const REQUEST: u8 = 85; + +// Fleet maintenance already uses 86 in the shared catalog envelope. +pub(super) const GROUP_BIRTH: u8 = 186; +pub(super) const GROUP_CLAIM: u8 = 187; +pub(super) const GROUP_ARCHIVE: u8 = 188; +pub(super) const GROUP_RECOVERY: u8 = 189; +pub(super) const GROUP_FENCE: u8 = 190; +pub(super) const GROUP_RECOVERY_BINDING: u8 = 191; +pub(super) const GROUP_RECOVERY_SEED_INTENT: u8 = 192; +pub(super) const GROUP_RECOVERY_SEED_COMPLETE: u8 = 193; +pub(super) const GROUP_BIRTH_RESERVATION: u8 = 194; +pub(super) const GROUP_RECOVERY_ACTIVATED: u8 = 195; +pub(super) const LIMIT: usize = 64 * 1024; + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct BirthReservation { + pub birth: Vec, + #[serde(deserialize_with = "required_origin")] + pub recovery: Option<[u8; 32]>, +} + +fn required_origin<'de, D: serde::Deserializer<'de>>( + decoder: D, +) -> Result, D::Error> { + Option::deserialize(decoder) +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(tag = "bound", content = "cell", deny_unknown_fields)] +pub(super) enum Bound { + NegativeInfinity, + Key(String), + PositiveInfinity, +} + +impl Bound { + pub fn from_bound(value: &CellBound) -> Self { + match value { + CellBound::NegativeInfinity => Self::NegativeInfinity, + CellBound::Key(k) => Self::Key(k.cell().as_str().into()), + CellBound::PositiveInfinity => Self::PositiveInfinity, + } + } + + pub fn decode(&self, ns: NamespaceId) -> Result { + Ok(match self { + Self::NegativeInfinity => CellBound::NegativeInfinity, + Self::PositiveInfinity => CellBound::PositiveInfinity, + Self::Key(cell) => CellBound::Key(cell_routing_key( + ns, + &CellId::new(cell).map_err(|_| CatalogError::Corrupt("range cell ID"))?, + )), + }) + } +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Route { + pub group: [u8; 16], + pub incarnation: [u8; 16], + pub lower: Bound, + pub upper: Bound, + pub epoch: u64, + pub birth: [u8; 32], +} + +impl Route { + pub fn encode(value: &GroupRoute, owner: &RangeDirectory) -> Result { + let route = Self { + group: value.history.group.bytes(), + incarnation: value.history.incarnation.bytes(), + lower: Bound::from_bound(value.range.lower()), + upper: Bound::from_bound(value.range.upper()), + epoch: value.epoch.get(), + birth: value.birth_digest, + }; + if route.decode(owner)? != *value { + return Err(CatalogError::Invalid("range route binding")); + } + Ok(route) + } + + pub fn decode(&self, owner: &RangeDirectory) -> Result { + let config = owner.namespace.config(); + if self.birth == [0; 32] { + return Err(CatalogError::Corrupt("range birth digest")); + } + Ok(GroupRoute { + history: GroupHistory { + cluster: config.cluster, + namespace: config.namespace, + group: GroupId::new(self.group) + .map_err(|_| CatalogError::Corrupt("range group"))?, + incarnation: IncarnationId::new(self.incarnation) + .map_err(|_| CatalogError::Corrupt("range incarnation"))?, + }, + range: CellRange::new( + self.lower.decode(config.namespace)?, + self.upper.decode(config.namespace)?, + ) + .map_err(|_| CatalogError::Corrupt("range bounds"))?, + epoch: RouteEpoch::new(self.epoch).map_err(|_| CatalogError::Corrupt("range epoch"))?, + birth_digest: self.birth, + schema_digest: config.schema_digest, + }) + } +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Reference { + pub digest: [u8; 32], + pub height: u8, + pub groups: u32, + pub lower: Bound, + pub upper: Bound, + pub complete: bool, +} + +impl Reference { + pub fn range(&self, owner: &RangeDirectory) -> Result { + if self.height == 0 + || self.height > 32 + || self.groups == 0 + || self.groups > owner.limits.max_groups + { + return Err(CatalogError::Limit); + } + CellRange::new( + self.lower.decode(owner.namespace.config.namespace)?, + self.upper.decode(owner.namespace.config.namespace)?, + ) + .map_err(|_| CatalogError::Corrupt("page bounds")) + } + + pub fn require_complete(&self, owner: &RangeDirectory) -> Result<(), CatalogError> { + if self.range(owner)? != CellRange::all() || !self.complete { + return Err(CatalogError::Corrupt("incomplete range directory")); + } + Ok(()) + } +} + +#[derive(Clone, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Page { + pub namespace: [u8; 32], + pub route: Route, + pub left: Option, + pub right: Option, +} + +impl Page { + pub fn reference( + &self, + bytes: &[u8], + owner: &RangeDirectory, + ) -> Result { + if self.namespace != owner.namespace.root_digest { + return Err(CatalogError::Corrupt("range page namespace")); + } + let route = self.route.decode(owner)?; + let height = |r: &Option| r.as_ref().map_or(0, |r| r.height); + if height(&self.left).abs_diff(height(&self.right)) > 1 { + return Err(CatalogError::Corrupt("unbalanced range page")); + } + let mut groups = 1_u32; + let mut lower = route.range.lower().clone(); + let mut upper = route.range.upper().clone(); + let mut complete = true; + if let Some(left) = &self.left { + let range = left.range(owner)?; + if range.upper() > route.range.lower() { + return Err(CatalogError::Corrupt("overlapping left ranges")); + } + complete &= left.complete && range.upper() == route.range.lower(); + lower = range.lower().clone(); + groups = groups.checked_add(left.groups).ok_or(CatalogError::Limit)?; + } + if let Some(right) = &self.right { + let range = right.range(owner)?; + if range.lower() < route.range.upper() { + return Err(CatalogError::Corrupt("overlapping right ranges")); + } + complete &= right.complete && range.lower() == route.range.upper(); + upper = range.upper().clone(); + groups = groups + .checked_add(right.groups) + .ok_or(CatalogError::Limit)?; + } + let result = Reference { + digest: record::digest(bytes), + height: height(&self.left) + .max(height(&self.right)) + .checked_add(1) + .ok_or(CatalogError::Limit)?, + groups, + lower: Bound::from_bound(&lower), + upper: Bound::from_bound(&upper), + complete, + }; + result.range(owner)?; + Ok(result) + } +} + +#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Head { + pub namespace: [u8; 32], + pub genesis: [u8; 32], + pub revision: u64, + #[serde(deserialize_with = "required_tree")] + pub tree: Option, + pub operations: Option<[u8; 32]>, + pub groups: Option<[u8; 32]>, + pub journal: [u8; 32], + #[serde(deserialize_with = "required_recovery")] + pub recovery: Option>, + #[serde(deserialize_with = "required_import")] + pub import: Option<[u8; 16]>, +} + +fn required_import<'de, D: serde::Deserializer<'de>>( + decoder: D, +) -> Result, D::Error> { + Option::deserialize(decoder) +} + +// Marker 1 has one record shape. Missing fields are not an older-reader mode. +fn required_recovery<'de, D: serde::Deserializer<'de>>( + decoder: D, +) -> Result>, D::Error> { + Option::deserialize(decoder) +} + +#[derive(Serialize, Deserialize, PartialEq, Eq)] +#[serde(deny_unknown_fields)] +pub(super) struct Pin { + pub namespace: [u8; 32], + pub genesis: [u8; 32], +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Request { + pub namespace: [u8; 32], + pub operation: [u8; 16], + pub expected: Option<[u8; 32]>, + pub parents: Vec, + pub children: Vec, +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Journal { + pub namespace: [u8; 32], + pub operation: [u8; 16], + pub request: [u8; 32], + pub revision: u64, + pub previous: Option<[u8; 32]>, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub transfer_archive: Option<[u8; 32]>, + #[serde(deserialize_with = "required_tree")] + pub tree: Option, +} + +// A deliberate null is the journaled standby genesis. An omitted tree is corrupt, +// never an older record shape or inferred empty namespace. +fn required_tree<'de, D: serde::Deserializer<'de>>( + decoder: D, +) -> Result, D::Error> { + Option::deserialize(decoder) +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Birth { + pub namespace: [u8; 32], + pub group: [u8; 16], + pub incarnation: [u8; 16], + pub lower: Bound, + pub upper: Bound, + pub epoch: u64, + pub creation: [u8; 16], + pub voters: Vec, + pub learners: Vec, +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn missing_tree_is_not_a_standby_record() { + let head = Head { + namespace: [1; 32], + genesis: [2; 32], + revision: 1, + tree: None, + operations: Some([3; 32]), + groups: None, + journal: [2; 32], + recovery: None, + import: None, + }; + let mut value = serde_json::to_value(head).unwrap(); + assert!(serde_json::from_value::(value.clone()).is_ok()); + value.as_object_mut().unwrap().remove("tree"); + assert!(serde_json::from_value::(value).is_err()); + let journal = Journal { + namespace: [1; 32], + operation: [4; 16], + request: [5; 32], + revision: 1, + previous: None, + tree: None, + transfer_archive: None, + }; + let mut value = serde_json::to_value(journal).unwrap(); + assert!(serde_json::from_value::(value.clone()).is_ok()); + value.as_object_mut().unwrap().remove("tree"); + assert!(serde_json::from_value::(value).is_err()); + } +} diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs b/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs new file mode 100644 index 000000000..99802a58b --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs @@ -0,0 +1,874 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Routed commands retain their originating producer until exact resolution. +//! The node controller drives publication; acceptance is only provisional. +mod output; +mod owned; +mod remote; +use output::PendingOutput; + +use super::*; +use crate::application::consensus::command_request_digest; +use crate::raft::{ + cell_groups::CellSqlStageMode, + execution::{CommandAttempt, CommandRequest}, +}; +use crate::storage::CommandInputs; +use cellar_logic::consensus::{CommandFailure, CommandId, Durability}; +use cellar_ltx::generation::ImageId; + +#[derive(Clone)] +pub struct CellCommandRequest { + pub id: CommandId, + pub contract: OperationContract, + pub input: Vec, + pub durability: Durability, + pub retry_until_ms: u64, +} + +#[derive(Debug)] +pub struct CellCommandError { + pub command: CommandId, + /// Conservatively sticky after SQL entry. Definitive resolution is returned + /// separately; an error is never a non-commit proof. Retry the same identity. + pub ambiguous: bool, + pub cause: Box, +} + +impl fmt::Display for CellCommandError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!( + f, + "cell command {:?} (ambiguous={}): {}", + self.command, self.ambiguous, self.cause + ) + } +} + +impl std::error::Error for CellCommandError {} + +pub enum CellCommandPreparation { + Prepared, + Committed(CellCommandResponse), +} + +pub enum CellCommandSubmission { + Accepted(CellGroupProposal), + /// An exact shared preparation was irreversibly sealed before append. + Rejected, +} + +pub enum CellCommandSettlement { + Committed(CellCommandResponse), + NotCommitted, + Refused(CommandFailure), +} + +pub struct CellCommandResponse { + output: CommandOutput, +} + +enum CommandOutput { + Local { + receipt: Box, + token: CellVersionToken, + s3: Option>, + }, + Remote(Box), +} + +impl CellCommandResponse { + /// A remote acknowledgement carries authenticated output metadata, never a + /// locally inspected SQL image or a local archive-settlement proof. + pub fn local_receipt(&self) -> Option<&crate::raft::execution::CellCommittedReceipt> { + match &self.output { + CommandOutput::Local { receipt, .. } => Some(receipt.receipt()), + CommandOutput::Remote(_) => None, + } + } + + pub fn result(&self) -> &[u8] { + match &self.output { + CommandOutput::Local { receipt, .. } => receipt.receipt().receipt.result(), + CommandOutput::Remote(response) => response.result(), + } + } + + pub fn observation(&self) -> crate::raft::version::CellVersionObservation { + match &self.output { + CommandOutput::Local { receipt, .. } => receipt.receipt().observation(), + CommandOutput::Remote(response) => response.observation().clone(), + } + } + + pub fn token(&self) -> &CellVersionToken { + match &self.output { + CommandOutput::Local { token, .. } => token, + CommandOutput::Remote(response) => response.token(), + } + } + + pub fn durability(&self) -> Durability { + match &self.output { + CommandOutput::Local { s3: Some(_), .. } => Durability::S3, + CommandOutput::Local { s3: None, .. } => Durability::Quorum, + CommandOutput::Remote(response) => response.durability(), + } + } +} + +struct Origin { + runtime: CellGroupRuntime, + sql: CellGroupSqlPool, + options: CellSqlOptions, + route: GroupRoute, + proposal: ProposalId, + preview: ImageId, + prepared: Option, + submitted: bool, +} + +/// The controller retains this non-cloneable invocation across waiter cancellation. +/// Dropping it never releases the SQL worker's unresolved physical producer. +pub struct CellCommandInvocation { + dispatcher: CellGroupDispatcher, + system: bool, + cell: CellHistory, + request: CellCommandRequest, + inputs: CommandInputs, + session: CellRoutingSession, + attempt: Option, + dispatched: bool, + ambiguous: bool, + origin: Option, + output: Option, + remote_route: Option, + _admission: Permit, +} + +/// Synchronous request capture for the node queue. No SQL or routing I/O has +/// begun; dropping this preparation releases only its invocation reservation. +pub(crate) struct PendingCellCommand { + dispatcher: CellGroupDispatcher, + cell: CellHistory, + request: CellCommandRequest, + limits: CellRoutingLimits, + cached: Option, + captured: Option, + system: bool, + started: u64, + deadline: u64, + admission: Permit, +} + +impl CellGroupDispatcher { + /// Capture invocation admission and the original deadline immediately. + /// Polling constructs routing state; it never restarts the timeout. + pub fn command( + &self, + cell: CellHistory, + request: CellCommandRequest, + limits: CellRoutingLimits, + cached: Option, + ) -> impl std::future::Future> + + Send + + 'static { + self.command_with_inputs(cell, request, limits, cached, None) + } + + pub(crate) fn command_with_inputs( + &self, + cell: CellHistory, + request: CellCommandRequest, + limits: CellRoutingLimits, + cached: Option, + captured: Option, + ) -> impl std::future::Future> + + Send + + 'static { + let pending = self.prepare_command(cell, request, limits, cached, captured, false); + async move { pending?.finish().await } + } + + pub(crate) async fn system_command( + &self, + cell: CellHistory, + request: CellCommandRequest, + limits: CellRoutingLimits, + ) -> Result { + self.prepare_command(cell, request, limits, None, None, true)? + .finish() + .await + } + + pub(crate) fn prepare_command( + &self, + cell: CellHistory, + mut request: CellCommandRequest, + limits: CellRoutingLimits, + cached: Option, + captured: Option, + system: bool, + ) -> Result { + let error = |cause| CellCommandError { + command: request.id, + ambiguous: false, + cause: Box::new(cause), + }; + let started = self.domain.mono_ms().map_err(|_| error(closed()))?; + let deadline = limits.deadline_at(started).ok_or_else(|| { + error(CellDispatchError::Routing(CellRoutingError { + reason: CellRoutingFailure::InvalidRequest, + ambiguous_command: None, + })) + })?; + request.durability = request + .durability + .max(self.directory.namespace_config().minimum_durability); + if request.durability == Durability::S3 && self.s3.is_none() { + return Err(error(CellDispatchError::Execution( + ExecutionError::Refused(CommandFailure::UnsupportedCapability), + ))); + } + if request.contract.operation.len() > 128 + || request.input.len() > self.limits.max_input_bytes + { + return Err(error(CellDispatchError::Runtime(RuntimeError::Overloaded))); + } + if request.contract.kind != OperationKind::Command { + return Err(error(CellDispatchError::Registration( + RegistrationError::KindMismatch, + ))); + } + if system { + self.class.validate_system_operation(&request.contract) + } else { + self.class.validate_operation(&request.contract) + } + .map_err(|e| error(CellDispatchError::Registration(e)))?; + if system + && request.id.client + != crate::raft::effects::system_client(&cell).map_err(|failure| { + error(CellDispatchError::Execution(ExecutionError::Refused( + failure, + ))) + })? + { + return Err(error(CellDispatchError::Execution( + ExecutionError::Refused(CommandFailure::UnsupportedCapability), + ))); + } + let size = request + .input + .len() + .checked_mul(2) + .and_then(|n| n.checked_add(8192)) + .ok_or_else(|| error(CellDispatchError::Runtime(RuntimeError::Overloaded)))?; + let admission = self + .budget + .acquire(size) + .ok_or_else(|| error(CellDispatchError::Runtime(RuntimeError::Overloaded)))?; + request.input = request.input.into_boxed_slice().into_vec(); + request.contract.operation = request.contract.operation.into_boxed_str().into_string(); + Ok(PendingCellCommand { + dispatcher: self.clone(), + cell, + request, + limits, + cached, + captured, + system, + started, + deadline, + admission, + }) + } +} + +impl PendingCellCommand { + pub(crate) fn controller_admission( + &self, + domain: &DomainToken, + ) -> Result<(CommandId, u64, usize), CellCommandError> { + let error = |cause| CellCommandError { + command: self.request.id, + ambiguous: false, + cause: Box::new(cause), + }; + if !self.dispatcher.uses_domain(domain) { + return Err(error(CellDispatchError::Invalid( + "command dispatcher domain", + ))); + } + if domain.mono_ms().map_err(|_| error(closed()))? >= self.deadline { + return Err(error(CellDispatchError::Runtime( + RuntimeError::DeadlineExceeded, + ))); + } + let bytes = self + .request + .input + .len() + .checked_mul(2) + .and_then(|n| n.checked_add(8192)) + .ok_or_else(|| error(CellDispatchError::Runtime(RuntimeError::Overloaded)))?; + Ok((self.request.id, self.deadline, bytes)) + } + + pub(crate) async fn finish(self) -> Result { + let Self { + dispatcher, + cell, + request, + limits, + cached, + captured, + system, + started, + deadline: _, + admission, + } = self; + let error = |cause| CellCommandError { + command: request.id, + ambiguous: false, + cause: Box::new(cause), + }; + let inputs = if let Some(inputs) = captured { + inputs + } else { + dispatcher + .domain + .run(async { + use rand::RngCore as _; + let mut random = [0; 32]; + crate::asyncrt::rng("cell-group-command-inputs").fill_bytes(&mut random); + CommandInputs { + now_unix_millis: crate::asyncrt::wall_ms().max(0) as u64, + random, + } + }) + .await + .map_err(|_| error(closed()))? + }; + let session = dispatcher + .directory + .routing_session_in( + dispatcher.domain.clone(), + CellRouteRequest { + cell: cell.clone(), + operation: CellRouteOperation::Command(request.id), + fingerprint: fingerprint(&cell, &request, inputs, limits), + }, + limits, + cached, + started, + ) + .map_err(|e| error(CellDispatchError::Routing(e)))? + .using_cache(dispatcher.routes.clone()); + Ok(CellCommandInvocation { + dispatcher, + system, + cell, + request, + inputs, + session, + attempt: None, + dispatched: false, + ambiguous: false, + origin: None, + output: None, + remote_route: None, + _admission: admission, + }) + } +} + +impl CellCommandInvocation { + fn error(&self, cause: CellDispatchError) -> CellCommandError { + CellCommandError { + command: self.request.id, + ambiguous: self.ambiguous, + cause: Box::new(cause), + } + } + + fn checked( + &self, + result: Result, + deadline: u64, + ) -> Result { + let output = result.map_err(|e| self.error(e))?; + if self + .dispatcher + .domain + .mono_ms() + .map_err(|_| self.error(closed()))? + >= deadline + { + return Err(self.error(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded))); + } + Ok(output) + } + + pub fn attempt(&self) -> Option<&CellRouteAttempt> { + self.attempt.as_ref() + } + + pub async fn begin(&mut self) -> Result<(), CellCommandError> { + self.dispatched = true; + let result = if self.remote_route.is_some() { + self.session.begin_current().await + } else { + self.session.begin().await + } + .map_err(CellDispatchError::Routing); + let attempt = result.map_err(|e| self.error(e))?; + if self + .remote_route + .as_ref() + .is_some_and(|expected| !expected.matches(attempt.route())) + { + return Err(self.error(CellDispatchError::Invalid( + "remote command destination route", + ))); + } + self.attempt = Some(attempt); + self.dispatched = false; + Ok(()) + } + + pub async fn retry(&mut self) -> Result<(), CellCommandError> { + let previous = self.current()?.clone(); + self.dispatched = true; + let result = self + .session + .retry(&previous) + .await + .map_err(CellDispatchError::Routing); + self.attempt = Some(result.map_err(|e| self.error(e))?); + self.dispatched = false; + Ok(()) + } + + pub async fn redirect(&mut self, hint: &GroupRoute) -> Result<(), CellCommandError> { + let previous = self.current()?.clone(); + self.dispatched = true; + let result = self + .session + .redirect(&previous, hint) + .await + .map_err(CellDispatchError::Routing); + self.attempt = Some(result.map_err(|e| self.error(e))?); + self.dispatched = false; + Ok(()) + } + + fn current(&self) -> Result<&CellRouteAttempt, CellCommandError> { + self.attempt + .as_ref() + .ok_or_else(|| self.error(CellDispatchError::Invalid("missing route"))) + } + + fn command_attempt(&self, origin: &Origin) -> CommandAttempt { + CommandAttempt { + request: CommandRequest { + id: self.request.id, + operation: self.request.contract.operation.clone(), + contract_digest: self.request.contract.digest, + request_digest: command_request_digest(&self.request.input), + durability: self.request.durability, + retry_until_ms: self.request.retry_until_ms, + }, + proposal: origin.proposal, + preview: origin.preview, + inputs: self.inputs, + } + } + + /// Retain the origin before entering SQL. A cancelled or lost preparation + /// response resumes that producer, never a second destination's callback. + pub async fn prepare( + &mut self, + runtime: &CellGroupRuntime, + sql: &CellGroupSqlPool, + options: CellSqlOptions, + ) -> Result { + let attempt = self.current()?.clone(); + self.session + .validate_attempt(&attempt) + .map_err(|e| self.error(CellDispatchError::Routing(e)))?; + if self.dispatched { + return Err(self.error(CellDispatchError::Invalid("consumed route attempt"))); + } + self.dispatched = true; + if self.origin.is_some() || self.output.is_some() { + return self.resume().await; + } + let domain = self.dispatcher.domain.clone(); + if !runtime.uses_domain(&domain) || !sql.uses_domain(&domain) { + return Err(self.error(CellDispatchError::Invalid("foreign execution domain"))); + } + let timer = domain + .sleep_until(attempt.deadline_ms()) + .map_err(|_| self.error(closed()))?; + let result = crate::asyncrt::select! { + result = self.prepare_at(&attempt, runtime, sql, options) => result, + _ = timer => Err(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)), + }; + self.checked(result, attempt.deadline_ms()) + } + + async fn prepare_at( + &mut self, + attempt: &CellRouteAttempt, + runtime: &CellGroupRuntime, + sql: &CellGroupSqlPool, + options: CellSqlOptions, + ) -> Result { + let authority = runtime + .execution_authority(attempt.route().history, self.cell.clone()) + .await + .map_err(CellDispatchError::Runtime)?; + if authority.route() != attempt.route() { + return Err(CellDispatchError::Invalid("destination route binding")); + } + if self.request.durability == Durability::S3 { + self.dispatcher + .s3 + .as_ref() + .ok_or(CellDispatchError::Invalid("S3 command owner"))? + .require_runtime(runtime)?; + self.dispatcher + .s3 + .as_ref() + .ok_or(CellDispatchError::Invalid("S3 command owner"))? + .require_group(attempt.route())?; + } + let (proposal, preview) = self + .dispatcher + .domain + .run(async { + use rand::RngCore as _; + let mut proposal = [0; 16]; + let mut preview = [0; 16]; + let mut rng = crate::asyncrt::rng("cell-group-command-preparation"); + rng.fill_bytes(&mut proposal); + rng.fill_bytes(&mut preview); + (ProposalId::new(proposal), ImageId::new(preview)) + }) + .await + .map_err(|_| closed())?; + let proposal = proposal.map_err(|_| CellDispatchError::Invalid("proposal ID"))?; + let preview = preview.map_err(|_| CellDispatchError::Invalid("preview ID"))?; + self.session + .mark_submitted(attempt) + .map_err(CellDispatchError::Routing)?; + self.ambiguous = true; + self.origin = Some(Origin { + runtime: runtime.clone(), + sql: sql.clone(), + options: options.clone(), + route: attempt.route().clone(), + proposal, + preview, + prepared: None, + submitted: false, + }); + let command = self.command_attempt( + self.origin + .as_ref() + .ok_or(CellDispatchError::Invalid("missing origin"))?, + ); + let outcome = if self.system { + sql.stage_system_owned( + options, + authority, + command, + self.dispatcher.class.clone(), + self.request.input.clone(), + CellSqlStageMode::InvokeBefore(attempt.deadline_ms()), + ) + .await + } else { + sql.stage_registered_owned( + options, + authority, + command, + self.dispatcher.class.clone(), + self.request.input.clone(), + CellSqlStageMode::InvokeBefore(attempt.deadline_ms()), + ) + .await + }; + match outcome { + Ok(outcome) => self.preparation(outcome).await, + Err(failure) => { + // Admission refusals and received Stage errors completed without + // installing this producer. Lost replies retain the exact origin. + if !failure.unresolved { + self.origin = None; + } + Err(CellDispatchError::Execution(failure.error)) + } + } + } + + async fn preparation( + &mut self, + outcome: CellSqlOutcome, + ) -> Result { + match outcome { + CellSqlOutcome::Prepared(prepared) => { + self.origin + .as_mut() + .ok_or(CellDispatchError::Invalid("missing origin"))? + .prepared = Some(prepared); + Ok(CellCommandPreparation::Prepared) + } + CellSqlOutcome::Receipt(receipt) => { + let response = self.response(receipt).await?; + Ok(CellCommandPreparation::Committed(response)) + } + _ => Err(CellDispatchError::Invalid("command preparation response")), + } + } + + pub async fn resume(&mut self) -> Result { + let deadline = self.current()?.deadline_ms(); + let domain = self.dispatcher.domain.clone(); + if domain.mono_ms().map_err(|_| self.error(closed()))? >= deadline { + return Err(self.error(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded))); + } + let timer = domain + .sleep_until(deadline) + .map_err(|_| self.error(closed()))?; + let result = crate::asyncrt::select! { + result = self.resume_at() => result, + _ = timer => Err(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)), + }; + self.checked(result, deadline) + } + + async fn resume_at(&mut self) -> Result { + if self.output.is_some() { + return self + .finish_output() + .await + .map(CellCommandPreparation::Committed); + } + let origin = self + .origin + .as_ref() + .ok_or(CellDispatchError::Invalid("missing origin"))?; + match origin.sql.resume(self.cell.clone(), origin.proposal).await { + Ok(prepared) => self.preparation(CellSqlOutcome::Prepared(prepared)).await, + Err(ExecutionError::Uncertain) => self.lookup().await, + Err(error) => Err(CellDispatchError::Execution(error)), + } + } + + // A lost duplicate/settlement response can leave no pending producer. The + // registered path with allow_new=false may return its receipt, never execute. + async fn lookup(&mut self) -> Result { + let origin = self + .origin + .as_ref() + .ok_or(CellDispatchError::Invalid("missing origin"))?; + let authority = origin + .runtime + .execution_authority(origin.route.history, self.cell.clone()) + .await + .map_err(CellDispatchError::Runtime)?; + if authority.route() != &origin.route { + return Err(CellDispatchError::Invalid("origin route binding")); + } + let outcome = if self.system { + origin + .sql + .stage_system_owned( + origin.options.clone(), + authority, + self.command_attempt(origin), + self.dispatcher.class.clone(), + self.request.input.clone(), + CellSqlStageMode::Lookup, + ) + .await + } else { + origin + .sql + .stage_registered_owned( + origin.options.clone(), + authority, + self.command_attempt(origin), + self.dispatcher.class.clone(), + self.request.input.clone(), + CellSqlStageMode::Lookup, + ) + .await + } + .map_err(|failure| CellDispatchError::Execution(failure.error))?; + self.preparation(outcome).await + } + + pub async fn submit(&mut self) -> Result { + let deadline = self.current()?.deadline_ms(); + let domain = self.dispatcher.domain.clone(); + if domain.mono_ms().map_err(|_| self.error(closed()))? >= deadline { + return Err(self.error(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded))); + } + let timer = domain + .sleep_until(deadline) + .map_err(|_| self.error(closed()))?; + let result = crate::asyncrt::select! { + result = self.submit_at() => result, + _ = timer => Err(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)), + }; + self.checked(result, deadline) + } + + async fn submit_at(&mut self) -> Result { + let origin = self + .origin + .as_mut() + .ok_or(CellDispatchError::Invalid("missing origin"))?; + let prepared = origin + .prepared + .as_ref() + .ok_or(CellDispatchError::Invalid("missing preparation"))?; + if let Some(proof) = prepared.rejection().map_err(CellDispatchError::Runtime)? { + return self.reject(proof).await; + } + origin.submitted = true; + origin + .sql + .mark_submitted(self.cell.clone(), origin.proposal) + .await + .map_err(CellDispatchError::Execution)?; + match prepared + .submit(&origin.runtime) + .await + .map_err(CellDispatchError::Runtime)? + { + CellSqlSubmission::Accepted(accepted) => Ok(CellCommandSubmission::Accepted(accepted)), + CellSqlSubmission::Rejected(proof) => self.reject(proof).await, + } + } + + async fn reject( + &mut self, + proof: CellSqlRejection, + ) -> Result { + let origin = self + .origin + .as_ref() + .ok_or(CellDispatchError::Invalid("missing origin"))?; + match origin.sql.reject(self.cell.clone(), proof).await { + // The retained seal proves this exact preparation can never append; + // Uncertain here means its worker cleanup has already removed it. + Ok(()) | Err(ExecutionError::Uncertain) => (), + Err(error) => return Err(CellDispatchError::Execution(error)), + } + self.origin = None; + Ok(CellCommandSubmission::Rejected) + } + + /// Call after durable apply progress. Same-term absence remains uncertain; + /// only the existing newer-term barrier can establish NotCommitted. + pub async fn settle(&mut self) -> Result { + let deadline = self.current()?.deadline_ms(); + let domain = self.dispatcher.domain.clone(); + if domain.mono_ms().map_err(|_| self.error(closed()))? >= deadline { + return Err(self.error(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded))); + } + let timer = domain + .sleep_until(deadline) + .map_err(|_| self.error(closed()))?; + let result = crate::asyncrt::select! { + result = self.settle_at() => result, + _ = timer => Err(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)), + }; + self.checked(result, deadline) + } + + async fn settle_at(&mut self) -> Result { + if self.output.is_some() { + return self + .finish_output() + .await + .map(CellCommandSettlement::Committed); + } + let origin = self + .origin + .as_ref() + .ok_or(CellDispatchError::Invalid("missing origin"))?; + if !origin.submitted { + return Err(CellDispatchError::Invalid("command not submitted")); + } + let authority = origin + .runtime + .reconciliation_authority(origin.route.history, self.cell.clone()) + .await + .map_err(CellDispatchError::Runtime)?; + if authority.route() != &origin.route { + return Err(CellDispatchError::Invalid("origin route binding")); + } + match origin.sql.reconcile(origin.proposal, authority).await { + Ok(CellSqlReconciliation::Committed(receipt)) => { + let response = self.response(receipt).await?; + Ok(CellCommandSettlement::Committed(response)) + } + Ok(CellSqlReconciliation::NotCommitted) => { + self.origin = None; + Ok(CellCommandSettlement::NotCommitted) + } + Ok(CellSqlReconciliation::Refused(failure)) => { + self.origin = None; + Ok(CellCommandSettlement::Refused(failure)) + } + Err(ExecutionError::Uncertain) => { + match origin.sql.resume(self.cell.clone(), origin.proposal).await { + Ok(_) => return Err(CellDispatchError::Execution(ExecutionError::Uncertain)), + Err(ExecutionError::Uncertain) => (), + Err(error) => return Err(CellDispatchError::Execution(error)), + } + match self.lookup().await? { + CellCommandPreparation::Committed(receipt) => { + Ok(CellCommandSettlement::Committed(receipt)) + } + CellCommandPreparation::Prepared => { + Err(CellDispatchError::Invalid("lookup executed a command")) + } + } + } + Err(error) => Err(CellDispatchError::Execution(error)), + } + } +} + +fn fingerprint( + cell: &CellHistory, + request: &CellCommandRequest, + inputs: CommandInputs, + limits: CellRoutingLimits, +) -> [u8; 32] { + let mut hash = Sha256::new(); + hash.update(b"cellar-cell-command-invocation-1\0"); + hash.update(cell_image_history_digest(cell)); + hash.update(request.id.client.bytes()); + hash.update(request.id.sequence().to_be_bytes()); + hash.update((request.contract.operation.len() as u32).to_be_bytes()); + hash.update(request.contract.operation.as_bytes()); + hash.update(request.contract.digest); + hash.update(command_request_digest(&request.input)); + hash.update([match request.durability { + Durability::Quorum => 1, + Durability::S3 => 2, + }]); + hash.update(request.retry_until_ms.to_be_bytes()); + hash.update(inputs.now_unix_millis.to_be_bytes()); + hash.update(inputs.random); + hash.update(limits.timeout.as_nanos().to_be_bytes()); + hash.update([limits.max_attempts]); + hash.finalize().into() +} diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/command/output.rs b/crates/cellar/durability/raft/cell_groups/dispatch/command/output.rs new file mode 100644 index 000000000..b933cde08 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/dispatch/command/output.rs @@ -0,0 +1,133 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Committed outputs retain proof admission, independently of retired SQL work. +use super::*; + +pub(super) struct PendingOutput { + runtime: CellGroupRuntime, + route: GroupRoute, + receipt: Box, + proof: Box, +} + +impl CellCommandInvocation { + pub(super) async fn response( + &mut self, + receipt: Box, + ) -> Result { + if receipt.receipt().receipt.command() != self.request.id + || receipt.receipt().receipt.operation != self.request.contract.operation + || receipt.receipt().receipt.contract_digest != self.request.contract.digest + || receipt.receipt().receipt.request_digest + != command_request_digest(&self.request.input) + || receipt.receipt().version.history != self.cell + { + return Err(CellDispatchError::Invalid("command receipt binding")); + } + // SQL has resolved its producer. Archive failure and output fencing + // cannot turn that committed result back into unresolved SQL ownership. + let mut origin = self + .origin + .take() + .ok_or(CellDispatchError::Invalid("resolved command origin"))?; + let required = self + .request + .durability + .max(receipt.receipt().receipt.minimum_durability()); + if required == Durability::S3 { + let owner = self + .dispatcher + .s3 + .as_ref() + .ok_or(CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::UnsupportedCapability, + )))?; + let limits = origin.options.limits(); + // Do not retain the preparation's SQL slot across archive waiting. + origin.prepared = None; + let settled = owner + .command( + &origin.runtime, + origin.route.clone(), + receipt, + limits, + self.session.deadline_ms(), + ) + .await?; + self.output = Some(PendingOutput { + runtime: origin.runtime, + route: origin.route, + receipt: settled.receipt, + proof: settled.proof, + }); + return self.finish_output().await; + } + let token = self.output_token(&receipt).await?; + Ok(CellCommandResponse { + output: CommandOutput::Local { + receipt, + token, + s3: None, + }, + }) + } + + pub(super) async fn finish_output(&mut self) -> Result { + let output = self + .output + .as_ref() + .ok_or(CellDispatchError::Invalid("missing committed output"))?; + // A later committed entry may temporarily put the group ahead of apply. + // Retain this exact result/proof until a fresh serving fence is available; + // never reconcile again or rerun SQL to reconstruct the response. + let authority = output + .runtime + .execution_authority(output.route.history, self.cell.clone()) + .await + .map_err(CellDispatchError::Runtime)?; + if authority.route() != &output.route { + return Err(CellDispatchError::Invalid("S3 response current route")); + } + output + .proof + .require_current(&self.dispatcher.domain, self.session.deadline_ms()) + .await?; + authority.check().map_err(CellDispatchError::Runtime)?; + let token = self.output_token(&output.receipt).await?; + authority.check().map_err(CellDispatchError::Runtime)?; + if self.dispatcher.domain.mono_ms().map_err(|_| closed())? >= self.session.deadline_ms() { + return Err(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)); + } + let output = self + .output + .take() + .ok_or(CellDispatchError::Invalid("consumed committed output"))?; + Ok(CellCommandResponse { + output: CommandOutput::Local { + receipt: output.receipt, + token, + s3: Some(output.proof), + }, + }) + } + + async fn output_token( + &self, + receipt: &CellSqlReceipt, + ) -> Result { + let now = self + .dispatcher + .domain + .run(async { crate::asyncrt::wall_ms().max(0) as u64 }) + .await + .map_err(|_| closed())?; + self.dispatcher + .keys + .issue_cell( + &receipt.receipt().observation(), + self.dispatcher.class.schema_digest(), + now, + ) + .map_err(CellDispatchError::Version) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/command/owned.rs b/crates/cellar/durability/raft/cell_groups/dispatch/command/owned.rs new file mode 100644 index 000000000..4f20382d3 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/dispatch/command/owned.rs @@ -0,0 +1,263 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Controller-only execution: response expiry never cancels a physical phase. +use super::*; +use tokio::sync::watch; + +impl CellCommandInvocation { + pub(crate) fn controller_admission( + &self, + domain: &DomainToken, + ) -> Result<(CommandId, u64, usize), CellCommandError> { + if !self.dispatcher.domain.same_owner(domain) + || self.attempt.is_some() + || self.dispatched + || self.origin.is_some() + || self.output.is_some() + { + return Err(self.error(CellDispatchError::Invalid( + "controller requires a fresh local invocation", + ))); + } + let deadline = self.session.deadline_ms(); + if domain.mono_ms().map_err(|_| self.error(closed()))? >= deadline { + return Err(self.error(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded))); + } + let bytes = self + .request + .input + .len() + .checked_mul(2) + .and_then(|n| n.checked_add(8192)) + .ok_or_else(|| self.error(CellDispatchError::Runtime(RuntimeError::Overloaded)))?; + Ok((self.request.id, deadline, bytes)) + } + + pub(crate) fn unresolved_group(&self) -> Option { + self.origin.as_ref().map(|origin| origin.route.history) + } + + pub(crate) fn unresolved(&self) -> bool { + self.origin.is_some() + } + + fn expired(&self) -> Result { + Ok(self.dispatcher.domain.mono_ms().map_err(|_| closed())? >= self.session.deadline_ms()) + } + + async fn cancel_unsubmitted( + &mut self, + progress: &mut watch::Receiver<()>, + ) -> Result { + let prepared = self + .origin + .as_ref() + .and_then(|origin| origin.prepared.as_ref()) + .ok_or(CellDispatchError::Invalid("missing controller preparation"))?; + let Some(proof) = prepared + .cancel_before_submission() + .map_err(CellDispatchError::Runtime)? + else { + return Ok(false); + }; + loop { + match self.reject(proof.clone()).await { + Ok(_) => return Ok(true), + Err(CellDispatchError::Execution( + ExecutionError::Busy | ExecutionError::Refused(CommandFailure::Overloaded), + )) => { + progress.changed().await.map_err(|_| closed())?; + } + Err(error) => return Err(error), + } + } + } + + /// The caller must keep this future/owner until resolution or joined node + /// shutdown. It never resets routing deadlines or re-executes a callback. + pub(crate) async fn run_owned( + &mut self, + runtime: &CellGroupRuntime, + sql: &CellGroupSqlPool, + options: CellSqlOptions, + roster: &crate::raft::cell_groups::controller::GroupRoster, + progress: &mut watch::Receiver<()>, + transport: Option<&crate::raft::transport::RaftTransport>, + ) -> Result { + self.begin().await?; + // Only an originating invocation may forward. An authenticated incoming + // command stays local, so stale hints cannot create forwarding cycles. + if let Some(transport) = transport.filter(|_| !self.system && self.remote_route.is_none()) { + let remote = Box::pin(super::super::mutation_route::mutation_peer( + runtime, + transport, + &self.session, + self.current()?, + &self.dispatcher.keys, + &self.dispatcher.domain, + )) + .await + .map_err(|cause| self.error(cause))?; + if let Some(member) = remote { + let response = self.execute_remote(transport, member).await?; + return Ok(CellCommandResponse { + output: CommandOutput::Remote(Box::new(response)), + }); + } + } + let result = self.run_at(runtime, sql, options, roster, progress).await; + result.map_err(|error| self.error(error)) + } + + async fn run_at( + &mut self, + runtime: &CellGroupRuntime, + sql: &CellGroupSqlPool, + options: CellSqlOptions, + roster: &crate::raft::cell_groups::controller::GroupRoster, + progress: &mut watch::Receiver<()>, + ) -> Result { + let attempt = self + .attempt + .as_ref() + .ok_or(CellDispatchError::Invalid("missing route"))? + .clone(); + self.session + .validate_attempt(&attempt) + .map_err(CellDispatchError::Routing)?; + if !roster + .contains(&attempt.route().history) + .map_err(CellDispatchError::Runtime)? + { + return Err(CellDispatchError::Runtime(RuntimeError::NotFound)); + } + self.dispatched = true; + // Unlike the response-facing phase API, do not drop this SQL waiter at + // the response deadline. It carries the exact preparation or refusal. + match self.prepare_at(&attempt, runtime, sql, options).await { + Ok(CellCommandPreparation::Committed(response)) => return Ok(response), + Ok(CellCommandPreparation::Prepared) => (), + Err(error) if self.output.is_some() && output_retryable(&error) => { + return self.finish_owned_output(progress).await; + } + Err(error) => return Err(error), + } + loop { + if self.expired()? { + if self.cancel_unsubmitted(progress).await? { + return Err(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)); + } + break; + } + match self.submit_at().await { + Ok(CellCommandSubmission::Accepted(_)) => break, + Ok(CellCommandSubmission::Rejected) => { + return Err(CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::Overloaded, + ))) + } + Err(CellDispatchError::Runtime( + RuntimeError::NotReady | RuntimeError::Overloaded, + )) + | Err(CellDispatchError::Execution( + ExecutionError::Busy | ExecutionError::Refused(CommandFailure::Overloaded), + )) => { + // Retry only the exact prepared submission; no new callback. + progress.changed().await.map_err(|_| closed())?; + } + Err(CellDispatchError::Runtime( + RuntimeError::NotLeader | RuntimeError::StaleCompletion, + )) => break, + Err(error) => { + // An untouched preparation can be sealed even if the SQL + // mark/submission mailbox refused. Other errors stay owned. + self.cancel_unsubmitted(progress).await?; + return Err(error); + } + } + } + // Submission may have entered the owner despite a lost/error response. + // Only an exact receipt or newer applied term can retire this producer. + self.origin + .as_mut() + .ok_or(CellDispatchError::Invalid("missing origin"))? + .submitted = true; + loop { + match self.settle_at().await { + Ok(CellCommandSettlement::Committed(response)) => return Ok(response), + Ok(CellCommandSettlement::NotCommitted) => { + return Err(CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::Unavailable, + ))) + } + Ok(CellCommandSettlement::Refused(reason)) => { + return Err(CellDispatchError::Execution(ExecutionError::Refused( + reason, + ))) + } + Err( + error @ CellDispatchError::Runtime( + RuntimeError::NotReady + | RuntimeError::NotLeader + | RuntimeError::Overloaded + | RuntimeError::StaleCompletion, + ), + ) + | Err( + error @ CellDispatchError::Execution( + ExecutionError::Uncertain + | ExecutionError::Busy + | ExecutionError::Refused(CommandFailure::Overloaded), + ), + ) => { + if self.output.is_some() && output_retryable(&error) { + return self.finish_owned_output(progress).await; + } + // SQL has already resolved before an archive/output error. + // Never restart reconciliation after retiring its origin. + if self.origin.is_none() { + return Err(error); + } + } + Err(error) => return Err(error), + } + progress.changed().await.map_err(|_| closed())?; + } + } + + async fn finish_owned_output( + &mut self, + progress: &mut watch::Receiver<()>, + ) -> Result { + let domain = self.dispatcher.domain.clone(); + let deadline = self.session.deadline_ms(); + loop { + let timer = domain.sleep_until(deadline).map_err(|_| closed())?; + let result = crate::asyncrt::select! { + result = self.finish_output() => result, + _ = timer => Err(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)), + }; + match result { + Ok(response) => return Ok(response), + Err(error) if output_retryable(&error) => (), + Err(error) => return Err(error), + } + // This is buffered committed output, not an unresolved SQL producer. + // Progress wakes a fresh fence check without callbacks or proposals. + let timer = domain.sleep_until(deadline).map_err(|_| closed())?; + crate::asyncrt::select! { + result = progress.changed() => result.map_err(|_| closed())?, + _ = timer => return Err(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)), + } + } + } +} + +fn output_retryable(error: &CellDispatchError) -> bool { + matches!( + error, + CellDispatchError::Runtime( + RuntimeError::NotReady | RuntimeError::Overloaded | RuntimeError::StaleCompletion + ) + ) +} diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/command/remote.rs b/crates/cellar/durability/raft/cell_groups/dispatch/command/remote.rs new file mode 100644 index 000000000..f210b39c0 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/dispatch/command/remote.rs @@ -0,0 +1,94 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! One remote attempt consumes the original route budget; loss stays ambiguous. +use super::*; +use crate::raft::query_wire::command::CellCommandCall; +use crate::raft::transport::{CellRemoteCommandResponse, RaftTransport, TransportError}; +use cellar_logic::consensus::MemberId; + +impl CellCommandInvocation { + pub(crate) fn bind_remote_route(&mut self, route: super::super::RemoteMutationRoute) { + self.remote_route = Some(route); + } + + /// Execute this catalog attempt on an explicitly selected remote member. + /// Any possible transmission makes failure ambiguous. Retry this same + /// invocation/identity; a transport response is never a non-commit proof. + pub async fn execute_remote( + &mut self, + transport: &RaftTransport, + member: MemberId, + ) -> Result { + if self.system { + return Err(self.error(CellDispatchError::Invalid( + "private commands require local ownership", + ))); + } + let attempt = self.current()?.clone(); + self.session + .validate_attempt(&attempt) + .map_err(|e| self.error(CellDispatchError::Routing(e)))?; + if self.dispatched + || self.origin.is_some() + || self.output.is_some() + || self.remote_route.is_some() + { + return Err(self.error(CellDispatchError::Invalid( + "consumed or local command attempt", + ))); + } + if !transport.uses_command_domain(&self.dispatcher.domain) { + return Err(self.error(CellDispatchError::Invalid("command transport domain"))); + } + let now = self + .dispatcher + .domain + .mono_ms() + .map_err(|_| self.error(closed()))?; + let timeout_ms = self + .session + .deadline_ms() + .checked_sub(now) + .filter(|n| *n > 0) + .ok_or_else(|| { + self.error(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)) + })?; + let call = CellCommandCall { + request: self.request.clone(), + inputs: self.inputs, + timeout_ms, + route_epoch: attempt.route().epoch.get(), + birth_digest: attempt.route().birth_digest, + }; + self.session + .mark_submitted(&attempt) + .map_err(|e| self.error(CellDispatchError::Routing(e)))?; + self.dispatched = true; + self.ambiguous = true; + let result = self + .dispatcher + .domain + .run(transport.command_cell( + attempt.route(), + &self.cell, + member, + call, + self.dispatcher.keys.clone(), + self.session.deadline_ms(), + )) + .await + .map_err(|_| self.error(closed()))?; + let response = result.map_err(|error| { + self.error(match error { + TransportError::Command(failure) => { + CellDispatchError::Execution(ExecutionError::Refused(failure)) + } + other => CellDispatchError::Transport(other), + }) + })?; + self.session + .validate_attempt(&attempt) + .map_err(|e| self.error(CellDispatchError::Routing(e)))?; + Ok(response) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs b/crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs new file mode 100644 index 000000000..491abd96d --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs @@ -0,0 +1,662 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Routed creation identity and controller-owned initialization, never a cell task. +#[path = "enrollment_remote.rs"] +mod remote; + +use super::*; +use crate::raft::execution::CellEnrollmentAttempt; +use crate::raft::transport::{CellRemoteEnrollmentResponse, RaftTransport}; +use crate::storage::CommandInputs; +use cellar_logic::consensus::{CommandFailure, Durability}; +use cellar_ltx::generation::ImageId; +use tokio::sync::watch; + +#[derive(Clone, Copy)] +pub struct CellEnrollmentRequest { + pub creation: ProposalId, + pub durability: Durability, +} + +/// Committed creation metadata. A remote result retains its transport proof +/// and cannot stand in for a locally inspected SQL or archive result. +pub struct CellEnrollmentResponse(EnrollmentOutput); + +enum EnrollmentOutput { + Local(Box), + Remote(Box), +} + +impl CellEnrollmentResponse { + pub fn record(&self) -> &EnrollmentRecord { + match &self.0 { + EnrollmentOutput::Local(response) => response.record(), + EnrollmentOutput::Remote(response) => response.record(), + } + } + + pub fn durability(&self) -> Durability { + match &self.0 { + EnrollmentOutput::Local(response) => response.durability(), + EnrollmentOutput::Remote(response) => response.durability(), + } + } + + pub fn local_enrollment(&self) -> Option<&CellSqlEnrollment> { + match &self.0 { + EnrollmentOutput::Local(response) => Some(response), + EnrollmentOutput::Remote(_) => None, + } + } + + pub(crate) fn into_local(self) -> Option { + match self.0 { + EnrollmentOutput::Local(response) => Some(*response), + EnrollmentOutput::Remote(_) => None, + } + } +} + +#[derive(Debug)] +pub struct CellEnrollmentError { + pub creation: ProposalId, + pub ambiguous: bool, + pub cause: Box, +} + +impl fmt::Display for CellEnrollmentError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!( + f, + "cell enrollment: {} (ambiguous={})", + self.cause, self.ambiguous + ) + } +} + +impl std::error::Error for CellEnrollmentError {} + +struct Origin { + runtime: CellGroupRuntime, + sql: CellGroupSqlPool, + route: GroupRoute, + prepared: Option, +} + +/// Captured once before discovery and transferred into the node controller. +/// The registered class initializer is shared; no per-request closure is retained. +pub struct CellEnrollmentInvocation { + dispatcher: CellGroupDispatcher, + cell: CellHistory, + creation: ProposalId, + durability: Durability, + preview: ImageId, + inputs: CommandInputs, + session: CellRoutingSession, + started: bool, + route_remote: bool, + attempt: Option, + dispatched: bool, + remote_route: Option, + ambiguous: bool, + origin: Option, + _admission: Permit, +} + +enum EnrollmentInput { + Local, + Originating, + Received(CommandInputs, ImageId), +} + +impl CellGroupDispatcher { + pub fn enroll( + &self, + cell: CellHistory, + request: CellEnrollmentRequest, + limits: CellRoutingLimits, + cached: Option, + ) -> impl std::future::Future> + + Send + + 'static { + self.prepare_enrollment(cell, request, limits, cached, EnrollmentInput::Local) + } + + pub(crate) fn enroll_routed( + &self, + cell: CellHistory, + request: CellEnrollmentRequest, + limits: CellRoutingLimits, + ) -> impl std::future::Future> + + Send + + 'static { + self.prepare_enrollment(cell, request, limits, None, EnrollmentInput::Originating) + } + + pub(crate) fn enroll_with_inputs( + &self, + cell: CellHistory, + request: CellEnrollmentRequest, + limits: CellRoutingLimits, + cached: Option, + captured: Option<(CommandInputs, ImageId)>, + ) -> impl std::future::Future> + + Send + + 'static { + let input = captured.map_or(EnrollmentInput::Local, |(inputs, preview)| { + EnrollmentInput::Received(inputs, preview) + }); + self.prepare_enrollment(cell, request, limits, cached, input) + } + + fn prepare_enrollment( + &self, + cell: CellHistory, + request: CellEnrollmentRequest, + limits: CellRoutingLimits, + cached: Option, + input: EnrollmentInput, + ) -> impl std::future::Future> + + Send + + 'static { + let dispatcher = self.clone(); + let started = self.domain.mono_ms(); + async move { + let started = started.map_err(|_| CellEnrollmentError { + creation: request.creation, + ambiguous: false, + cause: Box::new(closed()), + })?; + dispatcher + .enroll_captured(cell, request, limits, cached, input, started) + .await + } + } + + async fn enroll_captured( + &self, + cell: CellHistory, + mut request: CellEnrollmentRequest, + limits: CellRoutingLimits, + cached: Option, + input: EnrollmentInput, + started: u64, + ) -> Result { + let error = |cause| CellEnrollmentError { + creation: request.creation, + ambiguous: false, + cause: Box::new(cause), + }; + + request.durability = request + .durability + .max(self.directory.namespace_config().minimum_durability); + let route_remote = matches!(input, EnrollmentInput::Originating); + if request.durability == Durability::S3 && self.s3.is_none() && !route_remote { + return Err(error(CellDispatchError::Execution( + ExecutionError::Refused(CommandFailure::UnsupportedCapability), + ))); + } + self.class + .require_initializer() + .map_err(|e| error(CellDispatchError::Registration(e)))?; + let admission = self + .budget + .acquire(8192) + .ok_or_else(|| error(CellDispatchError::Runtime(RuntimeError::Overloaded)))?; + let (inputs, preview) = if let EnrollmentInput::Received(inputs, preview) = input { + (inputs, preview) + } else { + let (inputs, preview) = self + .domain + .run(async { + use rand::RngCore as _; + let mut rng = crate::asyncrt::rng("cell-group-enrollment-inputs"); + let mut random = [0; 32]; + let mut preview = [0; 16]; + rng.fill_bytes(&mut random); + rng.fill_bytes(&mut preview); + ( + CommandInputs { + now_unix_millis: crate::asyncrt::wall_ms().max(0) as u64, + random, + }, + ImageId::new(preview), + ) + }) + .await + .map_err(|_| error(closed()))?; + let preview = + preview.map_err(|_| error(CellDispatchError::Invalid("enrollment preview")))?; + (inputs, preview) + }; + let mut hash = Sha256::new(); + hash.update(b"cellar-cell-enrollment-invocation-1\0"); + hash.update(cell_image_history_digest(&cell)); + hash.update(request.creation.bytes()); + hash.update(self.class.schema_digest()); + hash.update(inputs.now_unix_millis.to_be_bytes()); + hash.update(inputs.random); + hash.update(limits.timeout.as_nanos().to_be_bytes()); + hash.update([ + limits.max_attempts, + match request.durability { + Durability::Quorum => 1, + Durability::S3 => 2, + }, + ]); + let session = self + .directory + .routing_session_in( + self.domain.clone(), + CellRouteRequest { + cell: cell.clone(), + operation: CellRouteOperation::Enrollment(request.creation), + fingerprint: hash.finalize().into(), + }, + limits, + cached, + started, + ) + .map_err(|e| error(CellDispatchError::Routing(e)))? + .using_cache(self.routes.clone()); + Ok(CellEnrollmentInvocation { + dispatcher: self.clone(), + cell, + creation: request.creation, + durability: request.durability, + preview, + inputs, + session, + started: false, + route_remote, + attempt: None, + dispatched: false, + remote_route: None, + ambiguous: false, + origin: None, + _admission: admission, + }) + } +} + +impl CellEnrollmentInvocation { + fn error(&self, cause: CellDispatchError) -> CellEnrollmentError { + CellEnrollmentError { + creation: self.creation, + ambiguous: self.ambiguous, + cause: Box::new(cause), + } + } + + pub(crate) fn controller_admission( + &self, + domain: &DomainToken, + ) -> Result<(ProposalId, u64), CellEnrollmentError> { + if self.started || self.origin.is_some() || !self.dispatcher.domain.same_owner(domain) { + return Err(self.error(CellDispatchError::Invalid( + "controller requires fresh local enrollment", + ))); + } + let deadline = self.session.deadline_ms(); + if domain.mono_ms().map_err(|_| self.error(closed()))? >= deadline { + return Err(self.error(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded))); + } + Ok((self.creation, deadline)) + } + + pub(crate) fn unresolved_group(&self) -> Option { + self.origin.as_ref().map(|origin| origin.route.history) + } + + pub(crate) fn unresolved(&self) -> bool { + self.origin.is_some() + } + + pub(crate) async fn run_owned( + &mut self, + runtime: &CellGroupRuntime, + sql: &CellGroupSqlPool, + options: CellSqlOptions, + roster: &crate::raft::cell_groups::controller::GroupRoster, + progress: &mut watch::Receiver<()>, + transport: &RaftTransport, + ) -> Result { + self.started = true; + let attempt = self.begin_owned().await.map_err(|e| self.error(e))?; + if self.route_remote && self.remote_route.is_none() { + let member = Box::pin(super::mutation_route::mutation_peer( + runtime, + transport, + &self.session, + &attempt, + &self.dispatcher.keys, + &self.dispatcher.domain, + )) + .await + .map_err(|e| self.error(e))?; + if let Some(member) = member { + self.attempt = Some(attempt); + let response = self.execute_remote(transport, member).await?; + return Ok(CellEnrollmentResponse(EnrollmentOutput::Remote(Box::new( + response, + )))); + } + } + self.run_at(runtime, sql, options, roster, progress, attempt) + .await + .map(|response| CellEnrollmentResponse(EnrollmentOutput::Local(Box::new(response)))) + .map_err(|e| self.error(e)) + } + + async fn begin_owned(&mut self) -> Result { + let attempt = if self.remote_route.is_some() { + self.session.begin_current().await + } else { + self.session.begin().await + } + .map_err(CellDispatchError::Routing)?; + if self + .remote_route + .as_ref() + .is_some_and(|expected| !expected.matches(attempt.route())) + { + return Err(CellDispatchError::Invalid( + "remote enrollment destination route", + )); + } + self.session + .validate_attempt(&attempt) + .map_err(CellDispatchError::Routing)?; + Ok(attempt) + } + + async fn run_at( + &mut self, + runtime: &CellGroupRuntime, + sql: &CellGroupSqlPool, + options: CellSqlOptions, + roster: &crate::raft::cell_groups::controller::GroupRoster, + progress: &mut watch::Receiver<()>, + attempt: CellRouteAttempt, + ) -> Result { + if !roster + .contains(&attempt.route().history) + .map_err(CellDispatchError::Runtime)? + { + return Err(CellDispatchError::Runtime(RuntimeError::NotFound)); + } + let authority = runtime + .enrollment_authority(attempt.route().history, self.cell.clone()) + .await + .map_err(CellDispatchError::Runtime)?; + if authority.state().route() != attempt.route() { + return Err(CellDispatchError::Invalid("enrollment destination route")); + } + if self.durability == Durability::S3 { + let owner = self + .dispatcher + .s3 + .as_ref() + .ok_or(CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::UnsupportedCapability, + )))?; + owner.require_runtime(runtime)?; + owner.require_group(attempt.route())?; + } + self.session + .mark_submitted(&attempt) + .map_err(CellDispatchError::Routing)?; + self.ambiguous = true; + self.origin = Some(Origin { + runtime: runtime.clone(), + sql: sql.clone(), + route: attempt.route().clone(), + prepared: None, + }); + let class = self.dispatcher.class.clone(); + let cell = self.cell.clone(); + let staged = sql + .initialize_owned( + options, + authority, + CellEnrollmentAttempt { + creation: self.creation, + preview: self.preview, + inputs: self.inputs, + }, + move |tx| class.invoke_initializer(&cell, tx), + ) + .await; + match staged { + Ok(CellSqlOutcome::Enrollment(enrolled)) => return self.committed(*enrolled).await, + Ok(CellSqlOutcome::Prepared(prepared)) => self.origin_mut()?.prepared = Some(prepared), + Ok(_) => { + return Err(CellDispatchError::Invalid( + "enrollment preparation response", + )) + } + Err(failure) => { + if !failure.unresolved { + self.origin = None; + } + return Err(CellDispatchError::Execution(failure.error)); + } + } + loop { + if self.dispatcher.domain.mono_ms().map_err(|_| closed())? >= self.session.deadline_ms() + { + let prepared = self.prepared()?; + if let Some(proof) = prepared + .cancel_before_submission() + .map_err(CellDispatchError::Runtime)? + { + self.reject(proof, progress).await?; + return Err(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)); + } + break; + } + let prepared = self.prepared()?; + if let Some(proof) = prepared.rejection().map_err(CellDispatchError::Runtime)? { + self.reject(proof, progress).await?; + return Err(CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::Overloaded, + ))); + } + let origin = self.origin_ref()?; + let submitted = async { + origin + .sql + .mark_submitted(self.cell.clone(), self.creation) + .await + .map_err(CellDispatchError::Execution)?; + prepared + .submit(&origin.runtime) + .await + .map_err(CellDispatchError::Runtime) + } + .await; + match submitted { + Ok(CellSqlSubmission::Accepted(_)) => break, + Ok(CellSqlSubmission::Rejected(proof)) => { + self.reject(proof, progress).await?; + return Err(CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::Overloaded, + ))); + } + Err(CellDispatchError::Runtime( + RuntimeError::NotLeader | RuntimeError::StaleCompletion, + )) => break, + Err(error) if transient(&error) => { + progress.changed().await.map_err(|_| closed())? + } + Err(error) => { + if let Some(proof) = prepared + .cancel_before_submission() + .map_err(CellDispatchError::Runtime)? + { + self.reject(proof, progress).await?; + } + return Err(error); + } + } + } + loop { + let origin = self.origin_ref()?; + let resolved = async { + let state = origin + .runtime + .enrollment_state(origin.route.history, self.cell.clone()) + .await + .map_err(CellDispatchError::Runtime)?; + if state.route() != &origin.route { + return Err(CellDispatchError::Invalid("enrollment resolution route")); + } + origin + .sql + .resolve_enrollment(self.creation, state) + .await + .map_err(CellDispatchError::Execution) + } + .await; + match resolved { + Ok(CellSqlEnrollmentResolution::Committed(enrolled)) => { + return self.committed(*enrolled).await + } + Ok(result) => { + self.origin = None; + return Err(CellDispatchError::Execution(ExecutionError::Refused( + match result { + CellSqlEnrollmentResolution::Conflict => { + CommandFailure::CommandConflict + } + _ => CommandFailure::Unavailable, + }, + ))); + } + Err(error) + if transient(&error) + || matches!( + error, + CellDispatchError::Execution(ExecutionError::Uncertain) + ) => {} + Err(error) => return Err(error), + } + progress.changed().await.map_err(|_| closed())?; + } + } + + fn origin_ref(&self) -> Result<&Origin, CellDispatchError> { + self.origin + .as_ref() + .ok_or(CellDispatchError::Invalid("missing enrollment origin")) + } + + fn origin_mut(&mut self) -> Result<&mut Origin, CellDispatchError> { + self.origin + .as_mut() + .ok_or(CellDispatchError::Invalid("missing enrollment origin")) + } + + fn prepared(&self) -> Result { + self.origin_ref()? + .prepared + .clone() + .ok_or(CellDispatchError::Invalid("missing enrollment preparation")) + } + + async fn committed( + &mut self, + enrolled: CellSqlEnrollment, + ) -> Result { + let record = enrolled.record(); + if record.cell() != &self.cell + || record.creation() != self.creation + || record.schema() != self.dispatcher.class.schema_digest() + { + return Err(CellDispatchError::Invalid("enrollment receipt binding")); + } + // SQL is resolved. Archive failures and deadlines must not quarantine + // this group or keep the initializer's producer/preparation admitted. + let mut origin = self + .origin + .take() + .ok_or(CellDispatchError::Invalid("resolved enrollment origin"))?; + if self.durability == Durability::Quorum { + return Ok(enrolled); + } + origin.prepared = None; + let owner = self + .dispatcher + .s3 + .as_ref() + .ok_or(CellDispatchError::Invalid("S3 enrollment owner"))?; + let settled = owner + .enrollment( + &origin.runtime, + origin.route.clone(), + Box::new(enrolled), + self.session.deadline_ms(), + ) + .await?; + let authority = origin + .runtime + .enrollment_authority(origin.route.history, self.cell.clone()) + .await + .map_err(CellDispatchError::Runtime)?; + if authority.state().route() != &origin.route + || authority.state().record() != Some(settled.enrollment.record()) + { + return Err(CellDispatchError::Invalid( + "S3 enrollment current creation/route", + )); + } + settled + .proof + .require_current(&self.dispatcher.domain, self.session.deadline_ms()) + .await?; + if self.dispatcher.domain.mono_ms().map_err(|_| closed())? >= self.session.deadline_ms() { + return Err(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)); + } + authority.check().map_err(CellDispatchError::Runtime)?; + Ok((*settled.enrollment).with_archive_proof(settled.proof)) + } + + async fn reject( + &mut self, + proof: CellSqlRejection, + progress: &mut watch::Receiver<()>, + ) -> Result<(), CellDispatchError> { + loop { + match self + .origin_ref()? + .sql + .reject(self.cell.clone(), proof.clone()) + .await + { + Ok(()) | Err(ExecutionError::Uncertain) => { + self.origin = None; + return Ok(()); + } + Err(ExecutionError::Busy | ExecutionError::Refused(CommandFailure::Overloaded)) => { + progress.changed().await.map_err(|_| closed())?; + } + Err(error) => return Err(CellDispatchError::Execution(error)), + } + } + } +} + +fn transient(error: &CellDispatchError) -> bool { + matches!( + error, + CellDispatchError::Runtime( + RuntimeError::NotReady + | RuntimeError::Overloaded + | RuntimeError::NotLeader + | RuntimeError::StaleCompletion + ) | CellDispatchError::Execution( + ExecutionError::Busy | ExecutionError::Refused(CommandFailure::Overloaded) + ) + ) +} diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/enrollment_remote.rs b/crates/cellar/durability/raft/cell_groups/dispatch/enrollment_remote.rs new file mode 100644 index 000000000..133fbe5c4 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/dispatch/enrollment_remote.rs @@ -0,0 +1,142 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Remote creation retries preserve the captured identity, inputs and route budget. +use super::*; +use crate::raft::query_wire::enrollment::CellEnrollmentCall; +use crate::raft::transport::{CellRemoteEnrollmentResponse, RaftTransport, TransportError}; +use cellar_logic::consensus::MemberId; + +impl CellEnrollmentInvocation { + pub(crate) fn bind_remote_route(&mut self, route: super::super::RemoteMutationRoute) { + self.remote_route = Some(route); + } + + pub fn attempt(&self) -> Option<&CellRouteAttempt> { + self.attempt.as_ref() + } + + /// Select a remote attempt. Once selected, this invocation cannot be handed + /// to a local controller as a fresh creation, even after transport failure. + pub async fn begin(&mut self) -> Result<(), CellEnrollmentError> { + if self.origin.is_some() || self.remote_route.is_some() { + return Err(self.error(CellDispatchError::Invalid("local enrollment ownership"))); + } + self.started = true; + self.dispatched = true; + let attempt = self + .session + .begin() + .await + .map_err(|e| self.error(CellDispatchError::Routing(e)))?; + self.attempt = Some(attempt); + self.dispatched = false; + Ok(()) + } + + pub async fn retry(&mut self) -> Result<(), CellEnrollmentError> { + let previous = self.current()?.clone(); + self.dispatched = true; + let next = self + .session + .retry(&previous) + .await + .map_err(|e| self.error(CellDispatchError::Routing(e)))?; + self.attempt = Some(next); + self.dispatched = false; + Ok(()) + } + + pub async fn redirect(&mut self, hint: &GroupRoute) -> Result<(), CellEnrollmentError> { + let previous = self.current()?.clone(); + self.dispatched = true; + let next = self + .session + .redirect(&previous, hint) + .await + .map_err(|e| self.error(CellDispatchError::Routing(e)))?; + self.attempt = Some(next); + self.dispatched = false; + Ok(()) + } + + fn current(&self) -> Result<&CellRouteAttempt, CellEnrollmentError> { + self.attempt + .as_ref() + .ok_or_else(|| self.error(CellDispatchError::Invalid("missing enrollment route"))) + } + + /// One consumed remote attempt. Possible transmission makes errors sticky + /// ambiguous; neither a lost reply nor admission refusal proves non-commit. + pub async fn execute_remote( + &mut self, + transport: &RaftTransport, + member: MemberId, + ) -> Result { + let attempt = self.current()?.clone(); + self.session + .validate_attempt(&attempt) + .map_err(|e| self.error(CellDispatchError::Routing(e)))?; + if self.dispatched || self.origin.is_some() || self.remote_route.is_some() { + return Err(self.error(CellDispatchError::Invalid( + "consumed or local enrollment attempt", + ))); + } + if !transport.uses_command_domain(&self.dispatcher.domain) { + return Err(self.error(CellDispatchError::Invalid("enrollment transport domain"))); + } + let now = self + .dispatcher + .domain + .mono_ms() + .map_err(|_| self.error(closed()))?; + let timeout_ms = self + .session + .deadline_ms() + .checked_sub(now) + .filter(|n| *n > 0) + .ok_or_else(|| { + self.error(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)) + })?; + let call = CellEnrollmentCall { + request: CellEnrollmentRequest { + creation: self.creation, + durability: self.durability, + }, + inputs: self.inputs, + preview: self.preview, + timeout_ms, + route_epoch: attempt.route().epoch.get(), + birth_digest: attempt.route().birth_digest, + }; + self.session + .mark_submitted(&attempt) + .map_err(|e| self.error(CellDispatchError::Routing(e)))?; + self.dispatched = true; + self.ambiguous = true; + let result = self + .dispatcher + .domain + .run(transport.enroll_cell( + attempt.route(), + &self.cell, + member, + call, + self.dispatcher.keys.clone(), + self.session.deadline_ms(), + )) + .await + .map_err(|_| self.error(closed()))?; + let response = result.map_err(|error| { + self.error(match error { + TransportError::Command(failure) => { + CellDispatchError::Execution(ExecutionError::Refused(failure)) + } + other => CellDispatchError::Transport(other), + }) + })?; + self.session + .validate_attempt(&attempt) + .map_err(|e| self.error(CellDispatchError::Routing(e)))?; + Ok(response) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs b/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs new file mode 100644 index 000000000..987b95c46 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs @@ -0,0 +1,806 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Catalog-bound invocations over the existing consensus and SQL owners. +//! Routing observations never replace destination authority or ingress grants. + +mod mutation_route; +mod query_routing; +mod receipt; +pub use query_routing::CellQueryRouting; +pub use receipt::{ + CellReceiptInvocation, CellReceiptRequest, CellReceiptResponse, CellReceiptSelection, +}; + +mod enrollment; +pub use enrollment::{ + CellEnrollmentError, CellEnrollmentInvocation, CellEnrollmentRequest, CellEnrollmentResponse, +}; + +mod command; +pub(crate) use command::PendingCellCommand; +pub use command::{ + CellCommandError, CellCommandInvocation, CellCommandPreparation, CellCommandRequest, + CellCommandResponse, CellCommandSettlement, CellCommandSubmission, +}; + +use super::*; +use crate::application::consensus::{ + OperationContract, OperationKind, RaftClass, RegistrationError, +}; +use crate::asyncrt::{DomainToken, ExecutionDomain}; +use crate::raft::budget::{Budget, Permit}; +use crate::raft::completion::RuntimeError; +use crate::raft::execution::ExecutionError; +use crate::raft::version::{CellVersionObservation, CellVersionToken, VersionError, VersionKeys}; +use cellar_logic::cell_group::{CellHistory, GroupRoute}; +use cellar_logic::consensus::ProposalId; +use sha2::{Digest, Sha256}; +use std::{fmt, sync::Arc}; + +#[derive(Clone, Copy)] +pub struct CellDispatchLimits { + /// Maximum retained group-range hints shared by this registration's clones. + pub max_cached_routes: usize, + /// Conservative reserved hint capacity; at least 8 KiB, at most 64 MiB. + pub route_cache_bytes: usize, + pub max_invocations: usize, + pub retained_bytes: usize, + pub max_input_bytes: usize, +} + +#[derive(Clone, Debug)] +pub enum CellQueryConsistency { + Stale, + RecentSource { max_age_ms: u64 }, + AtLeast(CellVersionToken), + Linearizable, +} + +#[derive(Clone)] +pub struct CellQueryRequest { + pub invocation: ProposalId, + pub contract: OperationContract, + pub input: Vec, + pub consistency: CellQueryConsistency, +} + +#[derive(Debug)] +pub enum CellDispatchError { + Invalid(&'static str), + Routing(CellRoutingError), + Registration(RegistrationError), + Runtime(RuntimeError), + Execution(ExecutionError), + Version(VersionError), + Transport(crate::raft::transport::TransportError), + Archive(Arc), +} + +impl CellDispatchError { + /// A typed application or readiness refusal from local execution or an + /// authenticated remote reply. This does not resolve a command error's ambiguity: callers + /// must inspect `CellCommandError::ambiguous` and preserve its command ID. + pub fn refusal(&self) -> Option<&cellar_logic::consensus::CommandFailure> { + match self { + Self::Runtime(RuntimeError::NotReady) => { + Some(&cellar_logic::consensus::CommandFailure::Unavailable) + } + Self::Execution(ExecutionError::Refused(failure)) + | Self::Transport( + crate::raft::transport::TransportError::Query(failure) + | crate::raft::transport::TransportError::Command(failure), + ) => Some(failure), + _ => None, + } + } +} + +impl fmt::Display for CellDispatchError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "cell dispatch: {self:?}") + } +} + +impl std::error::Error for CellDispatchError {} + +/// One authorized namespace/class registration. Clones share invocation admission; +/// no cell owns a worker, timer loop or independent admission bank. +#[derive(Clone)] +pub struct CellGroupDispatcher { + directory: RangeDirectory, + class: Arc, + keys: VersionKeys, + domain: DomainToken, + limits: CellDispatchLimits, + budget: Arc, + routes: Arc, + s3: Option, +} + +impl CellGroupDispatcher { + pub(in crate::raft::cell_groups) fn directory(&self) -> RangeDirectory { + self.directory.clone() + } + + pub(crate) fn host_binding( + &self, + ) -> Result< + ( + crate::application::consensus::NamespaceConfig, + Arc, + usize, + ), + CellDispatchError, + > { + // Reserve the configured invocation/cache envelopes, rather than only + // their current usage, when admitting a node namespace registration. + let bytes = self + .limits + .retained_bytes + .checked_add(self.limits.route_cache_bytes) + .and_then(|n| n.checked_add(self.directory.cache_capacity_bytes())) + .and_then(|n| n.checked_add(8192)) + .ok_or(CellDispatchError::Runtime(RuntimeError::Overloaded))?; + Ok(( + self.directory.namespace_config().clone(), + self.class.clone(), + bytes, + )) + } + + pub(crate) fn uses_domain(&self, domain: &DomainToken) -> bool { + self.domain.same_owner(domain) + } + + pub(crate) fn namespace( + &self, + ) -> ( + cellar_logic::consensus::ClusterId, + cellar_logic::consensus::NamespaceId, + ) { + let config = self.directory.namespace_config(); + (config.cluster, config.namespace) + } + + pub fn new( + domain: &ExecutionDomain, + directory: RangeDirectory, + class: Arc, + keys: VersionKeys, + limits: CellDispatchLimits, + ) -> Result { + let domain = domain.services().domain_token().ok_or_else(closed)?; + Self::from_domain(domain, directory, class, keys, limits) + } + + pub(crate) fn from_domain( + domain: DomainToken, + directory: RangeDirectory, + class: Arc, + keys: VersionKeys, + limits: CellDispatchLimits, + ) -> Result { + if class.schema_digest() != directory.namespace_config().schema_digest { + return Err(CellDispatchError::Registration( + RegistrationError::ContractMismatch, + )); + } + if !(1..=65_536).contains(&limits.max_invocations) + || limits.max_input_bytes == 0 + || limits + .max_input_bytes + .checked_mul(2) + .and_then(|n| n.checked_add(8192)) + .is_none_or(|n| n > limits.retained_bytes) + { + return Err(CellDispatchError::Invalid("invocation limits")); + } + let routes = Arc::new( + crate::raft::catalog::RouteCache::new( + directory.clone(), + limits.max_cached_routes, + limits.route_cache_bytes, + ) + .map_err(|_| CellDispatchError::Invalid("route cache limits"))?, + ); + Ok(Self { + routes, + directory, + class, + keys, + domain, + limits, + budget: Budget::new(limits.retained_bytes, limits.max_invocations), + s3: None, + }) + } + + /// Bind local command, enrollment and receipt S3 settlement to the node archive owner. + /// The destination group must already have an archive registration. Remote + /// response settlement has a separate gate. + pub fn with_s3(mut self, controller: &CellGroupController) -> Result { + self.s3 = Some( + controller + .s3_client(&self.domain) + .map_err(CellDispatchError::Runtime)?, + ); + Ok(self) + } + + pub(crate) fn effect_binding( + &self, + domain: &DomainToken, + ) -> Result< + ( + crate::application::consensus::NamespaceConfig, + Arc, + ), + CellDispatchError, + > { + let required_input = if self.class.outbox_definition().is_some() { + 32 * 1024 + } else { + 1024 + }; + if !self.domain.same_owner(domain) + || !self.class.has_system_commands() + || self.limits.max_input_bytes < required_input + { + return Err(CellDispatchError::Invalid( + "effect dispatcher domain/class/input capacity", + )); + } + Ok(( + self.directory.namespace_config().clone(), + self.class.clone(), + )) + } + + /// Retained namespace range hints, never per-cell execution or authority. + pub fn route_cache_usage(&self) -> Result { + self.routes.usage().map_err(catalog_failure) + } + + /// Observe a verified current directory head and retire all older range hints. + /// This is host-driven metadata refresh, not a serving lease or a timer task. + pub async fn refresh_routes(&self) -> Result { + let view = self + .domain + .run(self.directory.observe()) + .await + .map_err(|_| closed())? + .map_err(catalog_failure)?; + self.routes.observe(&view).map_err(catalog_failure)?; + Ok(view.revision()) + } + + pub fn usage(&self) -> (usize, usize) { + self.budget.usage() + } + + /// Freeze the complete request before discovery. Input copies for a physical + /// attempt are included in this reservation; SQL additionally retains its + /// own admission if the invocation waiter or router is dropped. + pub fn query( + &self, + cell: CellHistory, + request: CellQueryRequest, + limits: CellRoutingLimits, + cached: Option, + ) -> Result { + self.query_routed(cell, request, limits, cached, CellQueryRouting::Local) + } + + pub fn query_routed( + &self, + cell: CellHistory, + mut request: CellQueryRequest, + limits: CellRoutingLimits, + cached: Option, + mut routing: CellQueryRouting, + ) -> Result { + let started_at = self.domain.mono_ms().map_err(|_| closed())?; + routing.validate(limits)?; + if request.contract.operation.len() > 128 + || request.input.len() > self.limits.max_input_bytes + { + return Err(CellDispatchError::Runtime(RuntimeError::Overloaded)); + } + if matches!(&request.consistency, CellQueryConsistency::RecentSource { max_age_ms } + if !(1..=3_600_000).contains(max_age_ms)) + { + return Err(CellDispatchError::Invalid("stale source age")); + } + if request.contract.kind != OperationKind::Query { + return Err(CellDispatchError::Registration( + RegistrationError::KindMismatch, + )); + } + let local_registration_error = match self.class.validate_operation(&request.contract) { + Ok(()) => None, + Err(error @ RegistrationError::UnknownOperation(_)) + if !matches!(routing, CellQueryRouting::Local) => + { + // An authenticated remote replica checks its own registration + // before admitting this exact query contract. + Some(error) + } + Err(error) => return Err(CellDispatchError::Registration(error)), + }; + let bytes = request + .input + .len() + .checked_mul(2) + .and_then(|n| n.checked_add(8192)) + .ok_or(CellDispatchError::Runtime(RuntimeError::Overloaded))?; + let admission = self + .budget + .acquire(bytes) + .ok_or(CellDispatchError::Runtime(RuntimeError::Overloaded))?; + request.input = request.input.into_boxed_slice().into_vec(); + request.contract.operation = request.contract.operation.into_boxed_str().into_string(); + let session = self + .directory + .routing_session_in( + self.domain.clone(), + CellRouteRequest { + cell: cell.clone(), + operation: CellRouteOperation::Query(request.invocation), + fingerprint: fingerprint(&cell, &request, limits, &routing), + }, + limits, + cached, + started_at, + ) + .map_err(CellDispatchError::Routing)? + .using_cache(self.routes.clone()); + Ok(CellQueryInvocation { + dispatcher: self.clone(), + request, + routing, + local_registration_error, + leader_hint: None, + session, + attempt: None, + remote_route: None, + dispatched: false, + _admission: Arc::new(admission), + }) + } +} + +/// A non-cloneable invocation. Each ticket is consumed before entering async +/// destination work; retries and redirects share one original deadline/budget. +pub struct CellQueryInvocation { + dispatcher: CellGroupDispatcher, + request: CellQueryRequest, + routing: CellQueryRouting, + local_registration_error: Option, + leader_hint: Option, + session: CellRoutingSession, + attempt: Option, + remote_route: Option, + dispatched: bool, + _admission: Arc, +} + +impl CellQueryInvocation { + pub(crate) fn controller_admission( + &self, + domain: &DomainToken, + ) -> Result<(ProposalId, u64, usize), CellDispatchError> { + if !self.dispatcher.uses_domain(domain) || self.attempt.is_some() || self.dispatched { + return Err(CellDispatchError::Invalid( + "controller requires a fresh local query", + )); + } + let deadline = self.session.deadline_ms(); + if domain.mono_ms().map_err(|_| closed())? >= deadline { + return Err(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)); + } + let bytes = self + .request + .input + .len() + .checked_mul(2) + .and_then(|n| n.checked_add(8192)) + .ok_or(CellDispatchError::Runtime(RuntimeError::Overloaded))?; + Ok((self.request.invocation, deadline, bytes)) + } + + pub(crate) fn bind_remote_route(&mut self, route: RemoteMutationRoute) { + self.remote_route = Some(route); + } + + pub fn attempt(&self) -> Option<&CellRouteAttempt> { + self.attempt.as_ref() + } + + pub async fn begin(&mut self) -> Result<(), CellDispatchError> { + self.dispatched = true; + let attempt = if self.remote_route.is_some() { + self.session.begin_current().await + } else { + self.session.begin().await + } + .map_err(CellDispatchError::Routing)?; + if self + .remote_route + .as_ref() + .is_some_and(|route| !route.matches(attempt.route())) + { + return Err(CellDispatchError::Invalid("remote query destination route")); + } + self.attempt = Some(attempt); + self.dispatched = false; + Ok(()) + } + + pub async fn retry(&mut self) -> Result<(), CellDispatchError> { + let previous = self + .attempt + .as_ref() + .ok_or(CellDispatchError::Invalid("missing route"))?; + self.dispatched = true; + let attempt = self + .session + .retry(previous) + .await + .map_err(CellDispatchError::Routing)?; + self.attempt = Some(attempt); + self.dispatched = false; + Ok(()) + } + + pub async fn redirect(&mut self, hint: &GroupRoute) -> Result<(), CellDispatchError> { + let previous = self + .attempt + .as_ref() + .ok_or(CellDispatchError::Invalid("missing route"))?; + self.dispatched = true; + let attempt = self + .session + .redirect(previous, hint) + .await + .map_err(CellDispatchError::Routing)?; + self.attempt = Some(attempt); + self.dispatched = false; + Ok(()) + } + + /// Execute on a local destination with the same captured clock owner. Remote + /// transports must preserve the remaining deadline through their own protocol; + /// monotonic timestamps from unrelated hosts are never interchangeable. + pub async fn execute( + &mut self, + runtime: &CellGroupRuntime, + sql: &CellGroupSqlPool, + options: CellSqlOptions, + ) -> Result { + self.execute_retained(runtime, sql, options, None).await + } + + pub(crate) async fn execute_retained( + &mut self, + runtime: &CellGroupRuntime, + sql: &CellGroupSqlPool, + options: CellSqlOptions, + retention: Option, + ) -> Result { + let attempt = self + .attempt + .as_ref() + .ok_or(CellDispatchError::Invalid("missing route"))?; + self.session + .validate_attempt(attempt) + .map_err(CellDispatchError::Routing)?; + if self.dispatched { + return Err(CellDispatchError::Invalid("consumed route attempt")); + } + self.dispatched = true; + if let Some(error) = &self.local_registration_error { + return Err(CellDispatchError::Registration(error.clone())); + } + let domain = &self.dispatcher.domain; + if !runtime.uses_domain(domain) || !sql.uses_domain(domain) { + return Err(CellDispatchError::Invalid("foreign execution domain")); + } + let timer = domain + .sleep_until(attempt.deadline_ms()) + .map_err(|_| closed())?; + crate::asyncrt::select! { + result = self.execute_at(attempt, runtime, sql, options, retention) => { + let result = result?; + if domain.mono_ms().map_err(|_| closed())? >= attempt.deadline_ms() { + return Err(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)); + } + self.session.validate_attempt(attempt).map_err(CellDispatchError::Routing)?; + Ok(result) + } + _ = timer => Err(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)), + } + } + + /// One remote attempt over the node's existing bounded peer query lane. + /// The response owns its transport reservation until the caller drops it. + pub async fn execute_remote( + &mut self, + transport: &crate::raft::transport::RaftTransport, + member: cellar_logic::consensus::MemberId, + ) -> Result { + let attempt = self + .attempt + .as_ref() + .ok_or(CellDispatchError::Invalid("missing route"))?; + self.session + .validate_attempt(attempt) + .map_err(CellDispatchError::Routing)?; + if self.dispatched { + return Err(CellDispatchError::Invalid("consumed route attempt")); + } + self.dispatched = true; + if !transport.uses_query_domain(&self.dispatcher.domain) { + return Err(CellDispatchError::Invalid("foreign query domain")); + } + let now = self.dispatcher.domain.mono_ms().map_err(|_| closed())?; + let timeout_ms = attempt + .deadline_ms() + .checked_sub(now) + .filter(|ms| *ms > 0) + .ok_or(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded))?; + let call = crate::raft::query_wire::cell::CellQueryCall { + request: self.request.clone(), + timeout_ms, + route_epoch: attempt.route().epoch.get(), + birth_digest: attempt.route().birth_digest, + }; + let timer = self + .dispatcher + .domain + .sleep_until(attempt.deadline_ms()) + .map_err(|_| closed())?; + let response = crate::asyncrt::select! { + response = transport.query_cell(attempt.route(), &attempt.request().cell, member, call, crate::raft::transport::CellQueryContext { keys: self.dispatcher.keys.clone(), admission: self._admission.clone() }, attempt.deadline_ms()) => response, + _ = timer => return Err(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)), + }; + let response = response.map_err(CellDispatchError::Transport)?; + self.leader_hint = response.leader_hint; + let response = response.result.map_err(|failure| { + CellDispatchError::Transport(crate::raft::transport::TransportError::Query(failure)) + })?; + if self.dispatcher.domain.mono_ms().map_err(|_| closed())? >= attempt.deadline_ms() { + return Err(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)); + } + self.session + .validate_attempt(attempt) + .map_err(CellDispatchError::Routing)?; + // Authenticate again at the caller boundary, then enforce the requested + // minimum independently from the destination's execution path. + let observation = self + .dispatcher + .keys + .verify_cell( + response.token(), + &attempt.request().cell, + attempt.route().schema_digest, + self.now().await?, + ) + .map_err(CellDispatchError::Version)?; + if observation != *response.observation() { + return Err(CellDispatchError::Invalid("query observation")); + } + if let CellQueryConsistency::AtLeast(token) = &self.request.consistency { + let minimum = self + .dispatcher + .keys + .verify_cell( + token, + &attempt.request().cell, + attempt.route().schema_digest, + self.now().await?, + ) + .map_err(CellDispatchError::Version)?; + if !observation + .version + .covers(&minimum.version) + .map_err(|_| CellDispatchError::Version(VersionError::WrongHistory))? + { + return Err(CellDispatchError::Runtime(RuntimeError::NotReady)); + } + } + self.session + .validate_attempt(attempt) + .map_err(CellDispatchError::Routing)?; + Ok(response) + } + + async fn execute_at( + &self, + attempt: &CellRouteAttempt, + runtime: &CellGroupRuntime, + sql: &CellGroupSqlPool, + options: CellSqlOptions, + retention: Option, + ) -> Result { + let cell = &attempt.request().cell; + let schema = self.dispatcher.class.schema_digest(); + let consistency = match &self.request.consistency { + CellQueryConsistency::Stale => CellReadConsistency::Stale, + CellQueryConsistency::RecentSource { max_age_ms } => { + CellReadConsistency::RecentSource { + max_age_ms: *max_age_ms, + } + } + CellQueryConsistency::Linearizable => CellReadConsistency::Linearizable, + CellQueryConsistency::AtLeast(token) => { + let observation = self + .dispatcher + .keys + .verify_cell(token, cell, schema, self.now().await?) + .map_err(CellDispatchError::Version)?; + CellReadConsistency::AtLeast(observation.version) + } + }; + let authority = runtime + .read_authority( + attempt.route().history, + cell.clone(), + consistency, + attempt.deadline_ms(), + ) + .await + .map_err(CellDispatchError::Runtime)?; + if authority.route() != attempt.route() || authority.cell().version.history != *cell { + return Err(CellDispatchError::Invalid("destination route binding")); + } + let selected = sql + .query_registered_retained( + options, + authority, + self.dispatcher.class.clone(), + self.request.contract.clone(), + self.request.input.clone(), + Some(crate::raft::host_sql::cell_groups::ReadRetention::new( + Arc::new((retention, self._admission.clone())), + )), + ) + .await + .map_err(|error| { + // The SQL waiter uses the shared execution error envelope. Its + // exact deadline must have the same routing meaning as this + // invocation's outer timer; ordinary storage failures stay fatal. + if matches!(&error, ExecutionError::Storage(cause) + if matches!(cause.downcast_ref::(), Some(RuntimeError::DeadlineExceeded))) + { + CellDispatchError::Runtime(RuntimeError::DeadlineExceeded) + } else { + CellDispatchError::Execution(error) + } + })?; + let token = self + .dispatcher + .keys + .issue_cell(selected.observation(), schema, self.now().await?) + .map_err(CellDispatchError::Version)?; + Ok(CellQueryResponse { + selected: QuerySelection::Local(selected), + token, + }) + } + + async fn now(&self) -> Result { + self.dispatcher + .domain + .run(async { crate::asyncrt::wall_ms().max(0) as u64 }) + .await + .map_err(|_| closed()) + } +} + +pub struct CellQueryResponse { + selected: QuerySelection, + token: CellVersionToken, +} + +enum QuerySelection { + Local(CellSqlQueryResult), + Remote(crate::raft::transport::CellRemoteQueryResponse), +} + +impl CellQueryResponse { + pub fn result(&self) -> &[u8] { + match &self.selected { + QuerySelection::Local(selected) => selected.result(), + QuerySelection::Remote(selected) => selected.result(), + } + } + + pub fn observation(&self) -> &CellVersionObservation { + match &self.selected { + QuerySelection::Local(selected) => selected.observation(), + QuerySelection::Remote(selected) => selected.observation(), + } + } + + pub fn token(&self) -> &CellVersionToken { + &self.token + } + + pub fn replica( + &self, + ) -> &cellar_logic::consensus::ReplicaIdentity { + match &self.selected { + QuerySelection::Local(selected) => selected.replica(), + QuerySelection::Remote(selected) => selected.replica(), + } + } + + pub(crate) fn remote(selected: crate::raft::transport::CellRemoteQueryResponse) -> Self { + let token = selected.token().clone(); + Self { + selected: QuerySelection::Remote(selected), + token, + } + } +} + +fn closed() -> CellDispatchError { + CellDispatchError::Runtime(RuntimeError::Closed) +} + +fn fingerprint( + cell: &CellHistory, + request: &CellQueryRequest, + limits: CellRoutingLimits, + routing: &CellQueryRouting, +) -> [u8; 32] { + let mut hash = Sha256::new(); + hash.update(b"cellar-cell-query-invocation-1\0"); + hash.update(cell_image_history_digest(cell)); + hash.update(limits.timeout.as_nanos().to_be_bytes()); + hash.update([limits.max_attempts]); + routing.fingerprint(&mut hash); + hash.update(request.invocation.bytes()); + hash.update((request.contract.operation.len() as u32).to_be_bytes()); + hash.update(request.contract.operation.as_bytes()); + hash.update(request.contract.digest); + hash.update((request.input.len() as u64).to_be_bytes()); + hash.update(&request.input); + match &request.consistency { + CellQueryConsistency::Stale => hash.update([0]), + CellQueryConsistency::RecentSource { max_age_ms } => { + hash.update([3]); + hash.update(max_age_ms.to_be_bytes()); + } + CellQueryConsistency::Linearizable => hash.update([1]), + CellQueryConsistency::AtLeast(token) => { + hash.update([2]); + hash.update(token.as_str()); + } + } + hash.finalize().into() +} + +fn catalog_failure(error: crate::raft::catalog::CatalogError) -> CellDispatchError { + CellDispatchError::Routing(CellRoutingError { + reason: CellRoutingFailure::Catalog(error), + ambiguous_command: None, + }) +} + +/// Receiver-side destination constraint, checked against a fresh catalog lookup. +pub(crate) struct RemoteMutationRoute { + pub group: cellar_logic::cell_group::GroupHistory, + pub schema: [u8; 32], + pub epoch: u64, + pub birth: [u8; 32], +} + +impl RemoteMutationRoute { + fn matches(&self, route: &GroupRoute) -> bool { + self.group == route.history + && self.schema == route.schema_digest + && self.epoch == route.epoch.get() + && self.birth == route.birth_digest + } +} diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/mutation_route.rs b/crates/cellar/durability/raft/cell_groups/dispatch/mutation_route.rs new file mode 100644 index 000000000..197302344 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/dispatch/mutation_route.rs @@ -0,0 +1,85 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Select one destination for a mutation before submission. +use super::*; +use crate::raft::transport::{RaftTransport, TransportError}; +use cellar_logic::consensus::MemberId; + +pub(super) async fn mutation_peer( + runtime: &CellGroupRuntime, + transport: &RaftTransport, + session: &CellRoutingSession, + attempt: &CellRouteAttempt, + keys: &VersionKeys, + domain: &DomainToken, +) -> Result, CellDispatchError> { + let route = attempt.route(); + let timer = domain + .sleep_until(session.deadline_ms()) + .map_err(|_| closed())?; + let status = crate::asyncrt::select! { + status = runtime.status(route.history) => status, + _ = timer => return Err(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)), + }; + match status { + Ok(status) => { + let Some(leader) = status.leader else { + return Ok(None); + }; + let local = runtime + .log() + .cell_group(&route.history) + .map_err(|e| CellDispatchError::Runtime(RuntimeError::Storage(e)))? + .identity() + .member; + if leader == local.get() { + return Ok(None); + } + return MemberId::new(leader) + .map(Some) + .map_err(|_| CellDispatchError::Invalid("mutation leader identity")); + } + Err(RuntimeError::NotFound) => (), + Err(error) => return Err(CellDispatchError::Runtime(error)), + } + + let candidates = match transport.cell_query_router_members(route, session.maximum_attempts()) { + Ok(members) => members, + Err(TransportError::UnknownGroup) => return Ok(None), + Err(error) => return Err(CellDispatchError::Transport(error)), + }; + for (index, member) in candidates.iter().enumerate() { + session + .validate_attempt(attempt) + .map_err(CellDispatchError::Routing)?; + let now = domain.mono_ms().map_err(|_| closed())?; + let remaining = session.deadline_ms().saturating_sub(now); + if remaining < 2 { + return Err(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)); + } + // Keep a share of the original deadline for the mutation itself. + let deadline = now + (remaining / (candidates.len() - index + 1) as u64).max(1); + match transport + .probe_cell_command_leader( + route, + &attempt.request().cell, + *member, + keys.clone(), + deadline, + ) + .await + { + Ok(true) => return Ok(Some(*member)), + Ok(false) => (), + Err( + TransportError::Io(_) + | TransportError::Timeout + | TransportError::StalePeer + | TransportError::UnknownGroup + | TransportError::Overloaded, + ) => (), + Err(error) => return Err(CellDispatchError::Transport(error)), + } + } + Err(CellDispatchError::Runtime(RuntimeError::NotReady)) +} diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/query_routing.rs b/crates/cellar/durability/raft/cell_groups/dispatch/query_routing.rs new file mode 100644 index 000000000..b1b1fbf26 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/dispatch/query_routing.rs @@ -0,0 +1,126 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Immutable replica preferences are observations, never destination authority. +use super::*; +use cellar_logic::consensus::MemberId; +use std::time::Duration; + +/// Replica preferences for one logical cell query. Candidates and an optional +/// untried observed leader share the original invocation deadline and attempt +/// bank. Every destination independently authorizes the request. Incoming RPCs +/// always execute locally, regardless of the origin's preferences. +#[derive(Clone, Debug, Default)] +pub enum CellQueryRouting { + #[default] + Local, + /// Use the local replica when present. On an ingress-only node, try + /// the installed trusted group route within the original attempt budget. + Automatic, + /// One to eight distinct members; attempt_timeout is positive and no larger + /// than the original timeout. Missing leader advice never grants authority. + Replicas { + members: Vec, + attempt_timeout: Duration, + leader_fallback: bool, + }, +} + +impl CellQueryRouting { + pub(super) fn validate(&mut self, limits: CellRoutingLimits) -> Result<(), CellDispatchError> { + if let Self::Replicas { + members, + attempt_timeout, + .. + } = self + { + if members.is_empty() + || members.len() > 8 + || members + .iter() + .enumerate() + .any(|(n, member)| members[..n].contains(member)) + || attempt_timeout.as_millis() == 0 + || *attempt_timeout > limits.timeout + { + return Err(CellDispatchError::Invalid("query replica routing bounds")); + } + *members = std::mem::take(members).into_boxed_slice().into_vec(); + } + Ok(()) + } + + pub(super) fn fingerprint(&self, hash: &mut Sha256) { + match self { + Self::Local => hash.update([0]), + Self::Automatic => hash.update([2]), + Self::Replicas { + members, + attempt_timeout, + leader_fallback, + } => { + hash.update([1, members.len() as u8, u8::from(*leader_fallback)]); + for member in members { + hash.update(member.get().to_be_bytes()); + } + hash.update(attempt_timeout.as_nanos().to_be_bytes()); + } + } + } +} + +impl CellQueryInvocation { + /// Reject invalid minima before any replica attempt can reinterpret a + /// destination refusal as transient unavailability. + pub(crate) async fn validate_minimum(&self) -> Result<(), CellDispatchError> { + let attempt = self + .attempt + .as_ref() + .ok_or(CellDispatchError::Invalid("missing route"))?; + if let CellQueryConsistency::AtLeast(token) = &self.request.consistency { + self.dispatcher + .keys + .verify_cell( + token, + &attempt.request().cell, + self.dispatcher.class.schema_digest(), + self.now().await?, + ) + .map_err(CellDispatchError::Version)?; + } + self.session + .validate_attempt(attempt) + .map_err(CellDispatchError::Routing) + } + + pub(crate) fn deadline_ms(&self) -> u64 { + self.session.deadline_ms() + } + + pub(crate) fn maximum_attempts(&self) -> usize { + self.session.maximum_attempts() + } + + pub(crate) fn routing(&self) -> CellQueryRouting { + self.routing.clone() + } + + pub(crate) fn leader_hint(&self) -> Option { + self.leader_hint + } + + /// Shortening one ticket never renews the session or any earlier physical read. + pub(crate) fn limit_attempt(&mut self, timeout: Duration) -> Result<(), CellDispatchError> { + let now = self.dispatcher.domain.mono_ms().map_err(|_| closed())?; + let attempt = self + .attempt + .as_mut() + .ok_or(CellDispatchError::Invalid("missing route"))?; + let deadline = now + .checked_add(timeout.as_millis() as u64) + .ok_or(CellDispatchError::Invalid("query attempt deadline"))? + .min(attempt.deadline_ms()); + attempt + .limit_query_deadline(deadline) + .map_err(CellDispatchError::Routing) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/receipt.rs b/crates/cellar/durability/raft/cell_groups/dispatch/receipt.rs new file mode 100644 index 000000000..2e8b85bac --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/dispatch/receipt.rs @@ -0,0 +1,470 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Receipt-only dispatch never stages SQL or turns absence into non-commit. +use super::*; +use crate::application::consensus::command_request_digest; +use crate::raft::execution::{CommandRequest, StoredReceipt}; +use cellar_logic::consensus::{CommandFailure, CommandId, Durability}; + +#[derive(Clone)] +pub struct CellReceiptRequest { + pub id: CommandId, + pub contract: OperationContract, + pub input: Vec, + pub durability: Durability, +} + +pub struct CellReceiptInvocation { + dispatcher: CellGroupDispatcher, + cell: CellHistory, + request: CellReceiptRequest, + session: CellRoutingSession, + attempt: Option, + dispatched: bool, + remote_route: Option, + _admission: Permit, +} + +pub struct CellReceiptResponse { + command: CommandId, + selected: Box, + token: CellVersionToken, + s3: Option>, +} + +/// One coherent quorum selection, bound to its original operation and deadline. +/// No result escapes before `response` satisfies the required durability. S3 +/// admission takes SQL retention here; dropping this phase releases admission. +pub struct CellReceiptSelection<'a> { + invocation: &'a mut CellReceiptInvocation, + runtime: CellGroupRuntime, + attempt: CellRouteAttempt, + selected: Selected, +} + +enum Selected { + Quorum(Box), + S3(Box), +} + +impl CellReceiptSelection<'_> { + pub async fn response(self) -> Result { + self.invocation + .response(&self.runtime, self.attempt, self.selected) + .await + } +} + +impl CellReceiptResponse { + /// Achieved durability of this observation, including an ambiguous absence. + pub fn durability(&self) -> Durability { + if self.s3.is_some() { + Durability::S3 + } else { + Durability::Quorum + } + } + + pub fn command(&self) -> CommandId { + self.command + } + + /// None is ambiguous, even after a quorum or S3 read: an unresolved or delayed + /// original command may still commit. This response releases no producer. + pub fn receipt(&self) -> Option<&StoredReceipt> { + self.selected.receipt() + } + + pub fn observation(&self) -> &CellVersionObservation { + self.selected.observation() + } + + pub fn token(&self) -> &CellVersionToken { + &self.token + } +} + +impl CellGroupDispatcher { + pub fn lookup_receipt( + &self, + cell: CellHistory, + mut request: CellReceiptRequest, + limits: CellRoutingLimits, + cached: Option, + ) -> Result { + request.durability = request + .durability + .max(self.directory.namespace_config().minimum_durability); + if request.durability == Durability::S3 && self.s3.is_none() { + return Err(CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::UnsupportedCapability, + ))); + } + if request.contract.kind != OperationKind::Command { + return Err(CellDispatchError::Registration( + RegistrationError::KindMismatch, + )); + } + self.class + .validate_operation(&request.contract) + .map_err(CellDispatchError::Registration)?; + if request.contract.operation.len() > 128 + || request.input.len() > self.limits.max_input_bytes + { + return Err(CellDispatchError::Runtime(RuntimeError::Overloaded)); + } + let bytes = request + .input + .len() + .checked_mul(2) + .and_then(|n| n.checked_add(8192)) + .ok_or(CellDispatchError::Runtime(RuntimeError::Overloaded))?; + let permit = self + .budget + .acquire(bytes) + .ok_or(CellDispatchError::Runtime(RuntimeError::Overloaded))?; + request.contract.operation = request.contract.operation.into_boxed_str().into_string(); + request.input = request.input.into_boxed_slice().into_vec(); + let started = self.domain.mono_ms().map_err(|_| closed())?; + let mut hash = Sha256::new(); + hash.update(b"cellar-cell-receipt-invocation-1\0"); + hash.update(cell_image_history_digest(&cell)); + hash.update(request.id.client.bytes()); + hash.update(request.id.sequence().to_be_bytes()); + hash.update((request.contract.operation.len() as u32).to_be_bytes()); + hash.update(request.contract.operation.as_bytes()); + hash.update(request.contract.digest); + hash.update(command_request_digest(&request.input)); + hash.update([ + match request.durability { + Durability::Quorum => 1, + Durability::S3 => 2, + }, + limits.max_attempts, + ]); + hash.update(limits.timeout.as_nanos().to_be_bytes()); + let session = self + .directory + .routing_session_in( + self.domain.clone(), + CellRouteRequest { + cell: cell.clone(), + operation: CellRouteOperation::Receipt(request.id), + fingerprint: hash.finalize().into(), + }, + limits, + cached, + started, + ) + .map_err(CellDispatchError::Routing)? + .using_cache(self.routes.clone()); + Ok(CellReceiptInvocation { + dispatcher: self.clone(), + cell, + request, + session, + attempt: None, + dispatched: false, + remote_route: None, + _admission: permit, + }) + } +} + +impl CellReceiptInvocation { + pub fn attempt(&self) -> Option<&CellRouteAttempt> { + self.attempt.as_ref() + } + + pub(crate) fn bind_remote_route(&mut self, route: RemoteMutationRoute) { + self.remote_route = Some(route); + } + + pub async fn begin(&mut self) -> Result<(), CellDispatchError> { + self.dispatched = true; + let attempt = if self.remote_route.is_some() { + self.session.begin_current().await + } else { + self.session.begin().await + } + .map_err(CellDispatchError::Routing)?; + if self + .remote_route + .as_ref() + .is_some_and(|r| !r.matches(attempt.route())) + { + return Err(CellDispatchError::Invalid("receipt destination route")); + } + self.attempt = Some(attempt); + self.dispatched = false; + Ok(()) + } + + pub async fn retry(&mut self) -> Result<(), CellDispatchError> { + let previous = self.current()?.clone(); + self.dispatched = true; + self.attempt = Some( + self.session + .retry(&previous) + .await + .map_err(CellDispatchError::Routing)?, + ); + self.dispatched = false; + Ok(()) + } + + pub async fn redirect(&mut self, hint: &GroupRoute) -> Result<(), CellDispatchError> { + let previous = self.current()?.clone(); + self.dispatched = true; + self.attempt = Some( + self.session + .redirect(&previous, hint) + .await + .map_err(CellDispatchError::Routing)?, + ); + self.dispatched = false; + Ok(()) + } + + fn current(&self) -> Result<&CellRouteAttempt, CellDispatchError> { + self.attempt + .as_ref() + .ok_or(CellDispatchError::Invalid("missing receipt route")) + } + + pub async fn execute( + &mut self, + runtime: &CellGroupRuntime, + sql: &CellGroupSqlPool, + options: CellSqlOptions, + ) -> Result { + self.select(runtime, sql, options).await?.response().await + } + + /// Select one immutable quorum image under the original attempt/deadline. + /// S3 selection transfers result retention before returning this opaque phase; + /// only its response can expose the required-durability result. + pub async fn select( + &mut self, + runtime: &CellGroupRuntime, + sql: &CellGroupSqlPool, + options: CellSqlOptions, + ) -> Result, CellDispatchError> { + let attempt = self.current()?.clone(); + self.session + .validate_attempt(&attempt) + .map_err(CellDispatchError::Routing)?; + if self.dispatched { + return Err(CellDispatchError::Invalid("consumed receipt attempt")); + } + self.dispatched = true; + if !runtime.uses_domain(&self.dispatcher.domain) + || !sql.uses_domain(&self.dispatcher.domain) + { + return Err(CellDispatchError::Invalid("receipt execution domain")); + } + if self.request.durability == Durability::S3 { + let owner = self + .dispatcher + .s3 + .as_ref() + .ok_or(CellDispatchError::Invalid("S3 receipt owner"))?; + owner.require_runtime(runtime)?; + owner.require_group(attempt.route())?; + } + let limits = match &options { + CellSqlOptions::Materialized(o) => o.limits, + CellSqlOptions::Extents(o) => o.limits, + }; + let request = CommandRequest { + id: self.request.id, + operation: self.request.contract.operation.clone(), + contract_digest: self.request.contract.digest, + request_digest: command_request_digest(&self.request.input), + durability: self.request.durability, + retry_until_ms: 0, + }; + let authority = runtime + .read_authority( + attempt.route().history, + self.cell.clone(), + CellReadConsistency::Linearizable, + attempt.deadline_ms(), + ) + .await + .map_err(CellDispatchError::Runtime)?; + if authority.route() != attempt.route() || authority.cell().version.history != self.cell { + return Err(CellDispatchError::Invalid("receipt destination route")); + } + let selected = sql + .lookup_receipt_registered( + options, + authority, + self.dispatcher.class.clone(), + request.clone(), + self.request.input.clone(), + ) + .await + .map_err(CellDispatchError::Execution)?; + self.session + .validate_attempt(&attempt) + .map_err(CellDispatchError::Routing)?; + let required = self.request.durability.max( + selected + .receipt() + .map_or(Durability::Quorum, |row| row.minimum_durability()), + ); + let selected = if required == Durability::S3 { + let owner = self + .dispatcher + .s3 + .as_ref() + .ok_or(CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::UnsupportedCapability, + )))?; + Selected::S3(Box::new(owner.prepare_lookup( + runtime, + attempt.route().clone(), + Box::new(selected), + request, + limits, + attempt.deadline_ms(), + )?)) + } else { + Selected::Quorum(Box::new(selected)) + }; + Ok(CellReceiptSelection { + invocation: self, + runtime: runtime.clone(), + attempt, + selected, + }) + } + + async fn response( + &mut self, + runtime: &CellGroupRuntime, + attempt: CellRouteAttempt, + selected: Selected, + ) -> Result { + self.session + .validate_attempt(&attempt) + .map_err(CellDispatchError::Routing)?; + let (selected, s3, authority) = match selected { + Selected::Quorum(selected) => (selected, None, None), + Selected::S3(prepared) => { + let settled = prepared.response().await?; + settled + .proof + .require_current(&self.dispatcher.domain, attempt.deadline_ms()) + .await?; + // Use a fresh read authority, whose leader fence tolerates later + // cell writes, rather than retaining a private writer's base. + let authority = runtime + .read_authority( + attempt.route().history, + self.cell.clone(), + CellReadConsistency::Linearizable, + attempt.deadline_ms(), + ) + .await + .map_err(CellDispatchError::Runtime)?; + if authority.route() != attempt.route() + || authority.cell().version.history != self.cell + { + return Err(CellDispatchError::Invalid( + "S3 receipt current route/history", + )); + } + if authority.cell().version.sequence + < settled.selected.observation().version.sequence + { + return Err(CellDispatchError::Runtime(RuntimeError::NotReady)); + } + if authority.cell().version == settled.selected.observation().version + && authority.cell().witness != settled.selected.observation().witness + { + return Err(CellDispatchError::Invalid("S3 receipt current decision")); + } + (settled.selected, Some(settled.proof), Some(authority)) + } + }; + let now = self + .dispatcher + .domain + .run(async { crate::asyncrt::wall_ms().max(0) as u64 }) + .await + .map_err(|_| closed())?; + let token = self + .dispatcher + .keys + .issue_cell(selected.observation(), attempt.route().schema_digest, now) + .map_err(CellDispatchError::Version)?; + self.session + .validate_attempt(&attempt) + .map_err(CellDispatchError::Routing)?; + if let Some(authority) = authority { + authority + .check(self.dispatcher.domain.mono_ms().map_err(|_| closed())?) + .map_err(CellDispatchError::Runtime)?; + } + Ok(CellReceiptResponse { + command: self.request.id, + selected, + token, + s3, + }) + } + + pub async fn execute_remote( + &mut self, + transport: &crate::raft::transport::RaftTransport, + member: cellar_logic::consensus::MemberId, + ) -> Result { + let attempt = self.current()?.clone(); + self.session + .validate_attempt(&attempt) + .map_err(CellDispatchError::Routing)?; + if self.dispatched || self.remote_route.is_some() { + return Err(CellDispatchError::Invalid("consumed receipt attempt")); + } + self.dispatched = true; + if !transport.uses_command_domain(&self.dispatcher.domain) { + return Err(CellDispatchError::Invalid("receipt transport domain")); + } + let timeout_ms = attempt + .deadline_ms() + .checked_sub(self.dispatcher.domain.mono_ms().map_err(|_| closed())?) + .filter(|n| *n > 0) + .ok_or(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded))?; + let call = crate::raft::query_wire::receipt::CellReceiptCall { + request: self.request.clone(), + timeout_ms, + route_epoch: attempt.route().epoch.get(), + birth_digest: attempt.route().birth_digest, + }; + let response = self + .dispatcher + .domain + .run(transport.lookup_cell_receipt( + attempt.route(), + &self.cell, + member, + call, + self.dispatcher.keys.clone(), + attempt.deadline_ms(), + )) + .await + .map_err(|_| closed())? + .map_err(|error| match error { + crate::raft::transport::TransportError::Command(failure) => { + CellDispatchError::Execution(ExecutionError::Refused(failure)) + } + other => CellDispatchError::Transport(other), + })?; + self.session + .validate_attempt(&attempt) + .map_err(CellDispatchError::Routing)?; + Ok(response) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/effect_index.rs b/crates/cellar/durability/raft/cell_groups/effect_index.rs new file mode 100644 index 000000000..98b478fe8 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/effect_index.rs @@ -0,0 +1,68 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Authenticated next-due hints prune cold subtrees without hydrating SQLite. +//! Hints are scheduling data, never execution or delivery authority. +use super::page_wire::PageRef; +use super::{ + CellManifestLeaf, ManifestError, ManifestLimits, ManifestPageStore, ManifestRoot, ManifestTree, +}; +use crate::application::consensus::cell_routing_key; +use cellar_logic::cell_group::{CellId, CellKey}; + +pub(super) fn next( + tree: &ManifestTree, + store: &ManifestPageStore, + root: ManifestRoot, + now: u64, + after: Option<&CellId>, + limits: ManifestLimits, +) -> Result, ManifestError> { + let after = after.map(|cell| cell_routing_key(tree.route().history.namespace, cell)); + // One cursor path can precede a path to the successor; wrapping adds at + // most one more root-to-leaf path. Exhaustion is an error, never "no work". + let mut remaining = usize::from(limits.max_depth) * 3; + let selected = find(tree, store, root.0, now, after.as_ref(), &mut remaining)?; + if selected.is_some() || after.is_none() { + return Ok(selected); + } + find(tree, store, root.0, now, None, &mut remaining) +} + +fn find( + tree: &ManifestTree, + store: &ManifestPageStore, + reference: Option, + now: u64, + after: Option<&CellKey>, + remaining: &mut usize, +) -> Result, ManifestError> { + let Some(reference) = + reference.filter(|p| p.next_due_unix_millis.is_some_and(|due| due <= now)) + else { + return Ok(None); + }; + *remaining = remaining.checked_sub(1).ok_or(ManifestError::Limit)?; + let page = tree.read(store, reference)?; + let key = cell_routing_key( + tree.route().history.namespace, + &page.leaf.version.history.cell, + ); + if after.is_none_or(|after| &key > after) { + if let Some(selected) = find(tree, store, page.left, now, after, remaining)? { + return Ok(Some(selected)); + } + if page + .leaf + .effects + .next_due_unix_millis() + .is_some_and(|due| due <= now) + { + return Ok(Some(page.leaf)); + } + } + find(tree, store, page.right, now, after, remaining) +} + +#[cfg(test)] +#[path = "effect_index_tests.rs"] +mod tests; diff --git a/crates/cellar/durability/raft/cell_groups/effect_index_tests.rs b/crates/cellar/durability/raft/cell_groups/effect_index_tests.rs new file mode 100644 index 000000000..c8d7d2f02 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/effect_index_tests.rs @@ -0,0 +1,349 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Metadata-index tests only. They do not qualify effect execution or durability. +use super::*; +use crate::raft::cell_groups::{CellImageGeometry, EffectSummary, EnrollmentRecord}; +use cellar_logic::cell_group::*; +use cellar_logic::consensus::*; +use std::sync::Arc; + +struct Fixture { + _directory: tempfile::TempDir, + store: ManifestPageStore, + tree: ManifestTree, + root: ManifestRoot, + leaves: Vec, +} + +fn fixture(empty: bool) -> Fixture { + let directory = tempfile::tempdir().unwrap(); + let mut store = ManifestPageStore::open( + directory.path(), + Arc::new(cellar_ltx::DirectFileSystem), + 4096, + 32 * 1024 * 1024, + 16, + ) + .unwrap(); + let group = GroupHistory { + cluster: ClusterId::new([1; 16]).unwrap(), + namespace: NamespaceId::new([2; 16]).unwrap(), + group: GroupId::new([4; 16]).unwrap(), + incarnation: IncarnationId::new([5; 16]).unwrap(), + }; + let route = GroupRoute { + history: group, + range: CellRange::all(), + epoch: RouteEpoch::new(1).unwrap(), + birth_digest: [6; 32], + schema_digest: [7; 32], + }; + let tree = ManifestTree::new(route, ManifestLimits::default()).unwrap(); + let image = CellImage { + epoch: 1, + txid: if empty { 0 } else { 1 }, + checksum: 0, + commitment: CellImageCommitment::Materialized(if empty { + use sha2::Digest as _; + sha2::Sha256::digest([]).into() + } else { + [8; 32] + }), + }; + let mut root = ManifestRoot::default(); + let mut leaves = vec![]; + for n in 0..64 { + let history = CellHistory { + cluster: group.cluster, + namespace: group.namespace, + cell: CellId::new(format!("cell-{n}")).unwrap(), + incarnation: IncarnationId::new([3; 16]).unwrap(), + }; + let enrollment = EnrollmentRecord::new( + history.clone(), + ProposalId::new([9; 16]).unwrap(), + tree.route().schema_digest, + image, + ) + .unwrap(); + let position = LogPosition { + index: LogIndex::new(2), + term: Term::new(1), + }; + let effects = match n { + 3 => EffectSummary { + next_alarm_unix_millis: Some(10), + ..Default::default() + }, + 19 => EffectSummary { + next_outbox_unix_millis: Some(10), + pending_outbox: 1, + ..Default::default() + }, + 49 => EffectSummary { + next_alarm_unix_millis: Some(20), + ..Default::default() + }, + _ => EffectSummary { + next_alarm_unix_millis: Some(1000 + n), + ..Default::default() + }, + }; + let leaf = CellManifestLeaf { + enrollment: enrollment.clone(), + version: CellVersion { + history, + sequence: MutationSequence::new(1).unwrap(), + image, + }, + geometry: CellImageGeometry { + length: if empty { 0 } else { 512 }, + page_size: 512, + }, + last_mutation: position, + witness: CommitWitness { + group, + position, + proposal: enrollment.creation(), + }, + effects, + }; + root = tree.update(&mut store, root, leaf.clone(), None).unwrap(); + leaves.push(leaf); + } + Fixture { + _directory: directory, + store, + tree, + root, + leaves, + } +} + +#[test] +fn due_selection_matches_independent_order_and_wraps_fairly() { + let f = fixture(false); + assert_eq!(f.root.next_due_unix_millis(), Some(10)); + for now in [0, 9, 10, 19, 20, 1000, 9999] { + let mut eligible: Vec<_> = f + .leaves + .iter() + .filter(|leaf| { + leaf.effects + .next_due_unix_millis() + .is_some_and(|at| at <= now) + }) + .collect(); + eligible.sort_by_key(|leaf| { + cell_routing_key(f.tree.route().history.namespace, &leaf.version.history.cell) + }); + for cursor in f + .leaves + .iter() + .map(|leaf| Some(&leaf.version.history.cell)) + .chain([None]) + { + let key = cursor.map(|cell| cell_routing_key(f.tree.route().history.namespace, cell)); + let expected = eligible + .iter() + .copied() + .find(|leaf| { + key.as_ref().is_none_or(|key| { + &cell_routing_key( + f.tree.route().history.namespace, + &leaf.version.history.cell, + ) > key + }) + }) + .or_else(|| eligible.first().copied()); + let actual = f.tree.next_due(&f.store, f.root, now, cursor).unwrap(); + assert_eq!(actual.as_ref(), expected); + } + } + let mut cursor = None; + let mut seen = std::collections::BTreeMap::new(); + for _ in 0..20 { + let leaf = f + .tree + .next_due(&f.store, f.root, 10, cursor.as_ref()) + .unwrap() + .unwrap(); + *seen.entry(leaf.version.history.cell.clone()).or_insert(0) += 1; + cursor = Some(leaf.version.history.cell); + } + assert_eq!(seen.len(), 2); + assert!(seen.values().all(|count| *count == 10)); +} + +#[test] +fn changing_effects_updates_new_root_and_preserves_earlier_root() { + let mut f = fixture(false); + let original = f.root; + for n in [3, 19, 49] { + let previous = f.leaves[n].clone(); + let mut next = previous.clone(); + next.effects = EffectSummary::default(); + next.version.sequence = next.version.sequence.checked_next().unwrap(); + next.version.image.txid += 1; + next.version.image.commitment = CellImageCommitment::Materialized([10; 32]); + next.last_mutation.index = LogIndex::new(100 + n as u64); + next.witness.position = next.last_mutation; + next.witness.proposal = ProposalId::new([10; 16]).unwrap(); + f.root = f + .tree + .update(&mut f.store, f.root, next.clone(), Some(&previous.version)) + .unwrap(); + f.leaves[n] = next; + } + assert_eq!(f.root.next_due_unix_millis(), Some(1000)); + assert!(f + .tree + .next_due(&f.store, f.root, 999, None) + .unwrap() + .is_none()); + assert!(f + .tree + .next_due(&f.store, original, 10, None) + .unwrap() + .is_some()); + f.tree + .verify( + &f.store, + f.root, + LogPosition { + index: LogIndex::new(1000), + term: Term::new(1), + }, + |_| Ok(()), + ) + .unwrap(); + let mut forged = f.root; + forged.0.as_mut().unwrap().next_due_unix_millis = None; + assert!(f + .tree + .verify( + &f.store, + forged, + LogPosition { + index: LogIndex::new(1000), + term: Term::new(1) + }, + |_| Ok(()) + ) + .is_err()); +} + +#[test] +fn marker_one_references_require_nullable_due_summary() { + let f = fixture(false); + let reference = PageRef { + digest: [1; 32], + height: 1, + cells: 1, + last_mutation_index: 1, + last_mutation_term: 1, + next_due_unix_millis: None, + }; + let mut value = serde_json::to_value(reference).unwrap(); + assert!(value["next_due_unix_millis"].is_null()); + assert_eq!( + serde_json::from_value::(value.clone()).unwrap(), + reference + ); + value + .as_object_mut() + .unwrap() + .remove("next_due_unix_millis"); + assert!(serde_json::from_value::(value).is_err()); + for field in ["last_mutation_index", "last_mutation_term"] { + let mut value = serde_json::to_value(reference).unwrap(); + value.as_object_mut().unwrap().remove(field); + assert!(serde_json::from_value::(value).is_err()); + } + let mut invalid = f.leaves[3].clone(); + invalid.effects.next_outbox_unix_millis = Some(i64::MAX as u64 + 1); + invalid.effects.pending_outbox = 1; + assert!(invalid.validate(f.tree.route()).is_err()); +} + +#[test] +fn complete_manifest_authenticates_latest_mutation_across_other_cell_updates() { + let mut f = fixture(false); + let before = f.root; + assert_eq!( + before.last_mutation(), + Some(LogPosition { + index: LogIndex::new(2), + term: Term::new(1) + }) + ); + let mut changed = f.leaves[7].clone(); + let old = changed.version.clone(); + changed.version.sequence = MutationSequence::new(2).unwrap(); + changed.version.image.txid = 2; + changed.version.image.commitment = CellImageCommitment::Materialized([11; 32]); + changed.last_mutation = LogPosition { + index: LogIndex::new(40), + term: Term::new(2), + }; + changed.witness = CommitWitness { + group: f.tree.route().history, + position: changed.last_mutation, + proposal: ProposalId::new([12; 16]).unwrap(), + }; + let after = f + .tree + .update(&mut f.store, before, changed.clone(), Some(&old)) + .unwrap(); + assert_eq!(after.cells(), 64); + assert_eq!(after.last_mutation(), Some(changed.last_mutation)); + assert_eq!(before.last_mutation().unwrap().index.get(), 2); + let applied = LogPosition { + index: LogIndex::new(60), + term: Term::new(3), + }; + f.tree.verify(&f.store, after, applied, |_| Ok(())).unwrap(); + let mut forged = after; + forged.0.as_mut().unwrap().last_mutation_index = 39; + assert!(f + .tree + .verify(&f.store, forged, applied, |_| Ok(())) + .is_err()); + let mut forged = after; + forged.0.as_mut().unwrap().last_mutation_term = 1; + assert!(f + .tree + .verify(&f.store, forged, applied, |_| Ok(())) + .is_err()); +} + +#[test] +fn verified_pinned_index_matches_page_lookup_without_opening_images() { + let mut f = fixture(true); + let position = LogPosition { + index: LogIndex::new(1000), + term: Term::new(1), + }; + f.tree + .verify(&f.store, f.root, position, |_| Ok(())) + .unwrap(); + let pins = crate::raft::cell_groups::ManifestPins::complete( + &f.tree, + &mut f.store, + f.root, + &mut |_| Ok(crate::raft::cell_groups::CellImagePin::Empty), + ) + .unwrap(); + for now in [0, 10, 20, 1000, 9999] { + for leaf in &f.leaves { + let cell = &leaf.version.history.cell; + let key = cell_routing_key(f.tree.route().history.namespace, cell); + let expected = f.tree.next_due(&f.store, f.root, now, Some(cell)).unwrap(); + let actual = pins.next_due(now, Some(&key)); + assert_eq!(actual.as_ref().map(|(leaf, _)| *leaf), expected.as_ref()); + if let Some((leaf, image)) = actual { + image.check(leaf, f.tree.route().schema_digest).unwrap(); + } + } + } +} diff --git a/crates/cellar/durability/raft/cell_groups/enrollment.rs b/crates/cellar/durability/raft/cell_groups/enrollment.rs new file mode 100644 index 000000000..e41cf1c13 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/enrollment.rs @@ -0,0 +1,238 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Deterministic enrollment admission. Decisions and decoded records are data, +//! never proof of leadership, a committed enrollment or a published image. + +use crate::application::consensus::{ + group_protocol::{CellGroupEntry, GroupCommand, GroupEntryLimits}, + ImageCommitment, LtxImage, ProtocolError, +}; +use cellar_logic::{cell_group::*, consensus::*}; +use serde::{Deserialize, Serialize}; +use sha2::{Digest, Sha256}; + +const RECORD_DOMAIN: &str = "cellar-cell-enrollment-1"; +const MAX_RECORD_BYTES: usize = 4096; + +/// The group owner's committed lifecycle state. Recheck it at log application; +/// an earlier decision made while active cannot authorize an entry after a seal. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum EnrollmentPhase { + Active, + Sealed, + Retired, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum EnrollmentError { + Protocol(ProtocolError), + NotEnrollment, + Sealed, + Retired, + Conflict, + InvalidRecord, + Limit, +} + +impl std::fmt::Display for EnrollmentError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "cell enrollment: {self:?}") + } +} + +impl std::error::Error for EnrollmentError {} + +/// Immutable creation identity retained alongside the cell's committed state. +/// It survives subsequent cell mutations and normal group transfers. The host +/// must publish it atomically with the initial cell image and carry it through +/// snapshots/archives; the latest image cannot stand in for this original image. +/// Decoding only checks shape and canonical encoding, not commitment authority. +#[derive(Clone, Debug, PartialEq, Eq, Deserialize)] +#[serde(try_from = "WireRecord")] +pub struct EnrollmentRecord { + cell: CellHistory, + creation: ProposalId, + schema: [u8; 32], + initial: CellImage, +} + +impl EnrollmentRecord { + /// Construct bounded creation data. This grants no admission or commitment; + /// the owner must still validate the replicated enrollment against its route. + pub fn new( + cell: CellHistory, + creation: ProposalId, + schema: [u8; 32], + initial: CellImage, + ) -> Result { + let record = Self { + cell, + creation, + schema, + initial, + }; + record.encode()?; + Ok(record) + } + + pub fn cell(&self) -> &CellHistory { + &self.cell + } + + pub fn creation(&self) -> ProposalId { + self.creation + } + + pub fn schema(&self) -> [u8; 32] { + self.schema + } + + pub fn initial(&self) -> CellImage { + self.initial + } + + pub fn encode(&self) -> Result, EnrollmentError> { + let image = LtxImage::try_from(self.initial).map_err(EnrollmentError::Protocol)?; + if image.txid == 0 + && (image.checksum != 0 + || image.commitment != ImageCommitment::Sha256(Sha256::digest([]).into())) + { + return Err(EnrollmentError::InvalidRecord); + } + let bytes = serde_json::to_vec(self).map_err(|_| EnrollmentError::InvalidRecord)?; + if bytes.len() > MAX_RECORD_BYTES { + return Err(EnrollmentError::Limit); + } + Ok(bytes) + } + + pub fn decode(bytes: &[u8]) -> Result { + if bytes.len() > MAX_RECORD_BYTES { + return Err(EnrollmentError::Limit); + } + let record: Self = + serde_json::from_slice(bytes).map_err(|_| EnrollmentError::InvalidRecord)?; + if record.encode()? != bytes { + return Err(EnrollmentError::InvalidRecord); + } + Ok(record) + } +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum EnrollmentDecision { + /// No committed creation exists. The host still needs current-term authority, + /// physical image verification and replicated atomic publication. + Propose(EnrollmentRecord), + /// Exact retry of the supplied committed creation. Do not rerun initialization + /// or replace the cell's current image with its original enrollment image. + Existing(EnrollmentRecord), +} + +/// Validate the bounded cell-addressed wire request against the current route +/// and the group's committed lifecycle and creation record. Payload is borrowed +/// by the codec; admission never allocates another copy of the LTX image bytes. +/// The caller supplies only committed records, not pending proposals. Concurrent +/// proposals are rechecked in log order, so the first publication fixes identity. +pub fn decide_enrollment( + route: &GroupRoute, + phase: EnrollmentPhase, + request: &[u8], + committed: Option<&EnrollmentRecord>, + limits: GroupEntryLimits, +) -> Result { + match phase { + EnrollmentPhase::Sealed => return Err(EnrollmentError::Sealed), + EnrollmentPhase::Retired => return Err(EnrollmentError::Retired), + EnrollmentPhase::Active => {} + } + let entry = + CellGroupEntry::decode(request, route, limits).map_err(EnrollmentError::Protocol)?; + let GroupCommand::Enroll { cell, image, .. } = entry.command else { + return Err(EnrollmentError::NotEnrollment); + }; + let requested = EnrollmentRecord { + cell, + creation: entry.binding.proposal, + schema: entry.binding.schema_digest, + initial: image.into(), + }; + // Bound the durable creation record before an owner reserves image work. + requested.encode()?; + match committed { + Some(existing) if existing == &requested => { + Ok(EnrollmentDecision::Existing(existing.clone())) + } + Some(_) => Err(EnrollmentError::Conflict), + None => Ok(EnrollmentDecision::Propose(requested)), + } +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct WireRecord { + domain: String, + cluster: [u8; 16], + namespace: [u8; 16], + cell: String, + incarnation: [u8; 16], + creation: [u8; 16], + schema: [u8; 32], + initial_epoch: u64, + initial_txid: u64, + initial_checksum: u64, + initial_commitment: ImageCommitment, +} + +impl TryFrom for EnrollmentRecord { + type Error = EnrollmentError; + + fn try_from(wire: WireRecord) -> Result { + if wire.domain != RECORD_DOMAIN { + return Err(EnrollmentError::InvalidRecord); + } + let record = EnrollmentRecord { + cell: CellHistory { + cluster: ClusterId::new(wire.cluster) + .map_err(|_| EnrollmentError::InvalidRecord)?, + namespace: NamespaceId::new(wire.namespace) + .map_err(|_| EnrollmentError::InvalidRecord)?, + cell: CellId::new(wire.cell).map_err(|_| EnrollmentError::InvalidRecord)?, + incarnation: IncarnationId::new(wire.incarnation) + .map_err(|_| EnrollmentError::InvalidRecord)?, + }, + creation: ProposalId::new(wire.creation).map_err(|_| EnrollmentError::InvalidRecord)?, + schema: wire.schema, + initial: LtxImage { + epoch: wire.initial_epoch, + txid: wire.initial_txid, + checksum: wire.initial_checksum, + commitment: wire.initial_commitment, + } + .into(), + }; + record.encode()?; + Ok(record) + } +} + +impl Serialize for EnrollmentRecord { + fn serialize(&self, serializer: S) -> Result { + let record = self; + let image = LtxImage::try_from(record.initial).map_err(serde::ser::Error::custom)?; + WireRecord { + domain: RECORD_DOMAIN.into(), + cluster: record.cell.cluster.bytes(), + namespace: record.cell.namespace.bytes(), + cell: record.cell.cell.as_str().into(), + incarnation: record.cell.incarnation.bytes(), + creation: record.creation.bytes(), + schema: record.schema, + initial_epoch: record.initial.epoch, + initial_txid: record.initial.txid, + initial_checksum: record.initial.checksum, + initial_commitment: image.commitment, + } + .serialize(serializer) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/enrollment_apply.rs b/crates/cellar/durability/raft/cell_groups/enrollment_apply.rs new file mode 100644 index 000000000..a3de27884 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/enrollment_apply.rs @@ -0,0 +1,128 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Ordered enrollment preparation against one coherent group root. Publication +//! still belongs to the engine and requires the actual committed log position. + +use super::{ + decide_enrollment, CellImageGeometry, CellImagePin, CellManifestLeaf, EffectSummary, + EnrollmentDecision, EnrollmentError, EnrollmentPhase, EnrollmentRecord, ManifestError, + ManifestPageStore, ManifestTree, PreparedGroupManifest, +}; +use crate::application::consensus::group_protocol::{ + CellGroupEntry, GroupCommand, GroupEntryLimits, +}; +use cellar_logic::{cell_group::*, consensus::LogPosition}; + +pub struct EnrollmentApply<'a> { + pub position: LogPosition, + pub phase: EnrollmentPhase, + pub entry: &'a [u8], + pub limits: GroupEntryLimits, +} + +/// Independently prepared and synced initial image, returned only for a new +/// enrollment. A repeated creation must never invoke its image producer again. +pub struct EnrollmentImage { + pub geometry: CellImageGeometry, + pub effects: EffectSummary, + pub pin: CellImagePin, +} + +#[derive(Debug)] +pub enum EnrollmentApplyError { + Enrollment(EnrollmentError), + Manifest(ManifestError), +} + +impl std::fmt::Display for EnrollmentApplyError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::Enrollment(error) => error.fmt(f), + Self::Manifest(error) => error.fmt(f), + } + } +} + +impl std::error::Error for EnrollmentApplyError { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + match self { + Self::Enrollment(error) => Some(error), + Self::Manifest(error) => Some(error), + } + } +} + +impl From for EnrollmentApplyError { + fn from(error: EnrollmentError) -> Self { + Self::Enrollment(error) + } +} + +impl From for EnrollmentApplyError { + fn from(error: ManifestError) -> Self { + Self::Manifest(error) + } +} + +impl ManifestTree { + /// Recheck enrollment against the complete preceding root at ordered apply. + /// Exactly matching retries advance the group prefix while retaining the + /// current image, original creation and cell sequence. Conflicts or invalid + /// prefixes refuse before invoking the physical image producer. + /// + /// Run this synchronous preparation on the owning blocking execution domain. + /// The returned decision is not an acknowledgement: publish its manifest + /// through `persist_cells` only for the corresponding committed entry. + pub fn prepare_enrollment( + &self, + store: &mut ManifestPageStore, + previous: &PreparedGroupManifest, + apply: EnrollmentApply<'_>, + image: impl FnOnce(&EnrollmentRecord) -> Result, + ) -> Result<(PreparedGroupManifest, EnrollmentDecision), EnrollmentApplyError> { + if !previous.marker().is_active() { + return Err(ManifestError::Invalid("recovery is not activated").into()); + } + // Validate route, store ownership and contiguous position before image + // I/O. The unchanged candidate also supplies the idempotent apply path. + let unchanged = self.prepare_unchanged(store, previous, apply.position)?; + let entry = CellGroupEntry::decode(apply.entry, &self.route, apply.limits) + .map_err(EnrollmentError::Protocol)?; + let GroupCommand::Enroll { cell, .. } = entry.command else { + return Err(EnrollmentError::NotEnrollment.into()); + }; + let committed = previous.pin_cell(&cell.cell); + let decision = decide_enrollment( + &self.route, + apply.phase, + apply.entry, + committed.as_ref().map(|(leaf, _)| &leaf.enrollment), + apply.limits, + )?; + match &decision { + EnrollmentDecision::Existing(_) => Ok((unchanged, decision)), + EnrollmentDecision::Propose(record) => { + let prepared = image(record)?; + let leaf = CellManifestLeaf { + enrollment: record.clone(), + version: CellVersion { + history: record.cell().clone(), + sequence: MutationSequence::new(1) + .map_err(|_| ManifestError::Invalid("initial sequence"))?, + image: record.initial(), + }, + geometry: prepared.geometry, + last_mutation: apply.position, + witness: CommitWitness { + group: self.route.history, + position: apply.position, + proposal: record.creation(), + }, + effects: prepared.effects, + }; + let next = self.prepare_update(store, previous, leaf, None, prepared.pin)?; + Ok((next, decision)) + } + } + } +} diff --git a/crates/cellar/durability/raft/cell_groups/enrollment_authority.rs b/crates/cellar/durability/raft/cell_groups/enrollment_authority.rs new file mode 100644 index 000000000..f6765d6ed --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/enrollment_authority.rs @@ -0,0 +1,164 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Enrollment observations bind logical names before a cell image exists. + +use super::{EnrollmentRecord, PreparedGroupManifest}; +use crate::application::consensus::cell_routing_key; +use crate::raft::{cache::GroupLog, completion::RuntimeError}; +use cellar_logic::{cell_group::*, consensus::Term}; +use std::sync::{ + atomic::{AtomicU64, Ordering}, + Arc, +}; + +/// A verified current-term applied enrollment observation, including on a +/// follower. It permits resolution of old producers, never new initialization. +pub struct CellEnrollmentState { + group: GroupLog, + route: GroupRoute, + cell: CellHistory, + term: Term, + record: Option, +} + +impl CellEnrollmentState { + pub(crate) fn reserve_log( + &self, + envelope: crate::application::consensus::EnvelopeLimits, + ) -> Result, RuntimeError> { + self.group + .log_retention() + .ok_or_else(|| RuntimeError::Protocol("missing cell-group log retention".into()))? + .reserve_execution(envelope) + .map_err(|error| match error { + crate::raft::storage::LogError::Overloaded => RuntimeError::Overloaded, + other => RuntimeError::Storage(other), + }) + } + + pub(crate) fn engine_owner(&self) -> &Arc<()> { + &self.group.owner + } + + pub(super) fn new( + group: GroupLog, + current: &PreparedGroupManifest, + cell: CellHistory, + ) -> Result { + let route = current.marker().route(); + if !route.history.same_namespace(&cell) + || !route + .range + .contains(&cell_routing_key(cell.namespace, &cell.cell)) + { + return Err(RuntimeError::Invalid("enrollment cell route".into())); + } + let state = Self { + record: current + .pin_cell(&cell.cell) + .map(|(leaf, _)| leaf.enrollment.clone()), + group, + route: route.clone(), + cell, + term: current.marker().position().term, + }; + state.check()?; + Ok(state) + } + + pub fn cell(&self) -> &CellHistory { + &self.cell + } + + pub fn route(&self) -> &GroupRoute { + &self.route + } + + pub fn term(&self) -> Term { + self.term + } + + pub fn record(&self) -> Option<&EnrollmentRecord> { + self.record.as_ref() + } + + pub(crate) fn belongs_to_engine(&self, owner: &Arc<()>) -> bool { + Arc::ptr_eq(&self.group.owner, owner) + } + + pub(crate) fn same_binding(&self, other: &Self) -> bool { + Arc::ptr_eq(&self.group.owner, &other.group.owner) + && self.route == other.route + && self.cell == other.cell + } + + pub fn check(&self) -> Result<(), RuntimeError> { + let cache = self.group.read()?; + let current = cache + .group_manifest + .as_ref() + .ok_or(RuntimeError::NotReady)?; + let key = cell_routing_key(self.cell.namespace, &self.cell.cell); + if !current.is_active() + || self.term.get() == 0 + || cache.hard_state.term != self.term.get() + || current.position().term != self.term + || current.position().index.get() > cache.hard_state.commit + || current.route() != &self.route + || !cache.membership.permits_group(self.group.identity()) + || cache + .manifest_pins + .as_ref() + .ok_or(RuntimeError::NotReady)? + .get(&key) + .map(|(leaf, _)| &leaf.enrollment) + != self.record.as_ref() + { + return Err(RuntimeError::NotReady); + } + Ok(()) + } +} + +/// Issued by the actual leader only after its current-term committed prefix is +/// applied. A node SQL admission lease must additionally serialize the name. +pub struct CellEnrollmentAuthority { + state: CellEnrollmentState, + leader_term: Arc, +} + +impl CellEnrollmentAuthority { + pub(super) fn new( + state: CellEnrollmentState, + leader_term: Arc, + ) -> Result { + let authority = Self { state, leader_term }; + authority.check()?; + Ok(authority) + } + + pub fn state(&self) -> &CellEnrollmentState { + &self.state + } + + pub fn check(&self) -> Result<(), RuntimeError> { + if self.leader_term.load(Ordering::Acquire) != self.state.term.get() { + return Err(RuntimeError::NotReady); + } + self.state.check()?; + let cache = self.state.group.read()?; + if !cache + .conf_state + .voters + .iter() + .chain(&cache.conf_state.voters_outgoing) + .any(|id| *id == self.state.group.identity().member.get()) + { + return Err(RuntimeError::NotReady); + } + if self.leader_term.load(Ordering::Acquire) != self.state.term.get() { + return Err(RuntimeError::NotReady); + } + Ok(()) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/execution_authority.rs b/crates/cellar/durability/raft/cell_groups/execution_authority.rs new file mode 100644 index 000000000..3b03ee635 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/execution_authority.rs @@ -0,0 +1,302 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! A private-execution observation, never an acknowledgement or query lease. + +use super::{CellImagePin, CellManifestLeaf, PreparedGroupManifest}; +use crate::application::consensus::cell_routing_key; +use crate::raft::{cache::GroupLog, completion::RuntimeError}; +use cellar_logic::{cell_group::*, consensus::Term}; +use std::sync::{ + atomic::{AtomicU64, Ordering}, + Arc, +}; + +/// Issued only by the actual leader after its current-term applied barrier. +/// It pins one cell, not every image in the group. Other cells may advance while +/// this cell prepares; its own selected base, route or leader term may not. +/// Proposal admission must still recheck authority after physical preparation. +pub struct CellExecutionAuthority { + group: GroupLog, + route: GroupRoute, + term: Term, + leader_term: Arc, + leaf: CellManifestLeaf, + image: CellImagePin, +} + +impl CellExecutionAuthority { + pub(crate) fn reserve_log( + &self, + envelope: crate::application::consensus::EnvelopeLimits, + ) -> Result, RuntimeError> { + self.group + .log_retention() + .ok_or_else(|| RuntimeError::Protocol("missing cell-group log retention".into()))? + .reserve_execution(envelope) + .map_err(|error| match error { + crate::raft::storage::LogError::Overloaded => RuntimeError::Overloaded, + other => RuntimeError::Storage(other), + }) + } + + pub(crate) fn engine_owner(&self) -> &Arc<()> { + &self.group.owner + } + + pub(crate) fn accepts_reconciliation(&self, proof: &CellReconciliationAuthority) -> bool { + Arc::ptr_eq(&self.group.owner, &proof.group.owner) + && self.route == proof.route + && self.leaf.version.history == proof.leaf.version.history + } + + pub(super) fn new( + group: GroupLog, + current: &PreparedGroupManifest, + cell: &CellHistory, + term: Term, + leader_term: Arc, + ) -> Result { + let (leaf, image) = current.pin_cell(&cell.cell).ok_or(RuntimeError::NotFound)?; + if &leaf.version.history != cell { + return Err(RuntimeError::Invalid("cell execution history".into())); + } + let authority = Self { + group, + route: current.marker().route().clone(), + term, + leader_term, + leaf: leaf.clone(), + image, + }; + authority.check()?; + Ok(authority) + } + + pub(crate) fn belongs_to_engine(&self, owner: &Arc<()>) -> bool { + Arc::ptr_eq(&self.group.owner, owner) + } + + pub fn route(&self) -> &GroupRoute { + &self.route + } + + pub fn term(&self) -> Term { + self.term + } + + pub fn cell(&self) -> &CellManifestLeaf { + &self.leaf + } + + pub fn image(&self) -> &CellImagePin { + &self.image + } + + pub(crate) fn check_committed_entry(&self, bytes: &[u8]) -> Result<(), RuntimeError> { + self.check()?; + let cache = self.group.read()?; + let position = self.leaf.last_mutation; + let entry = cache + .entries + .get(&position.index.get()) + .ok_or(RuntimeError::NotReady)?; + if entry.term != position.term.get() + || entry.data.as_ref() != bytes + || cache.hard_state.commit < position.index.get() + { + return Err(RuntimeError::StaleCompletion); + } + Ok(()) + } + + pub fn check(&self) -> Result<(), RuntimeError> { + if self.leader_term.load(Ordering::Acquire) != self.term.get() { + return Err(RuntimeError::NotReady); + } + let cache = self.group.read()?; + let current = cache + .group_manifest + .as_ref() + .ok_or(RuntimeError::NotReady)?; + let key = cell_routing_key( + self.route.history.namespace, + &self.leaf.version.history.cell, + ); + if !current.is_active() + || cache.hard_state.term != self.term.get() + || !cache + .conf_state + .voters + .iter() + .chain(&cache.conf_state.voters_outgoing) + .any(|id| *id == self.group.identity().member.get()) + || current.route() != &self.route + || !cache.membership.permits_group(self.group.identity()) + || cache + .manifest_pins + .as_ref() + .and_then(|pins| pins.get(&key)) + .is_none_or(|(leaf, _)| leaf != &self.leaf) + || self.leader_term.load(Ordering::Acquire) != self.term.get() + { + return Err(RuntimeError::NotReady); + } + Ok(()) + } +} + +/// Custody of an already committed outbox claim. It retains the selected +/// image and leader fence while unrelated cells or later business writes may +/// advance. It grants neither SQL execution nor an acknowledgement. +pub(crate) struct CellEffectAuthority { + execution: CellExecutionAuthority, + minimum: cellar_logic::cell_group::CellVersion, +} + +impl CellExecutionAuthority { + pub(crate) fn effect_authority( + self, + minimum: &cellar_logic::cell_group::CellVersion, + term: Term, + ) -> Result { + // The execution observation was minted at a ready applied barrier. + // Its own cell has since advanced through the exact committed claim; + // checking equality with that older leaf would invalidate every claim. + if term != self.term + || !minimum + .covers(&self.leaf.version) + .map_err(|_| RuntimeError::Invalid("effect claim history".into()))? + { + return Err(RuntimeError::NotReady); + } + let authority = CellEffectAuthority { + execution: self, + minimum: minimum.clone(), + }; + authority.check()?; + Ok(authority) + } +} + +impl CellEffectAuthority { + pub(crate) fn check(&self) -> Result<(), RuntimeError> { + let execution = &self.execution; + if execution.leader_term.load(Ordering::Acquire) != execution.term.get() { + return Err(RuntimeError::NotReady); + } + let cache = execution.group.read()?; + let current = cache + .group_manifest + .as_ref() + .ok_or(RuntimeError::NotReady)?; + let key = cell_routing_key( + execution.route.history.namespace, + &self.minimum.history.cell, + ); + let (leaf, _pin) = cache + .manifest_pins + .as_ref() + .and_then(|pins| pins.get(&key)) + .ok_or(RuntimeError::NotReady)?; + if !current.is_active() + || current.route() != &execution.route + || cache.hard_state.term != execution.term.get() + || !cache + .conf_state + .voters + .iter() + .chain(&cache.conf_state.voters_outgoing) + .any(|id| *id == execution.group.identity().member.get()) + || !cache.membership.permits_group(execution.group.identity()) + || !leaf + .version + .covers(&self.minimum) + .map_err(|_| RuntimeError::NotReady)? + || execution.leader_term.load(Ordering::Acquire) != execution.term.get() + { + return Err(RuntimeError::NotReady); + } + Ok(()) + } +} + +/// A durable current-term applied cut, including on a follower. This permits +/// receipt inspection and retirement of older-term producers, never new SQL +/// execution, a linearizable query, or an S3 acknowledgement. +pub struct CellReconciliationAuthority { + group: GroupLog, + route: GroupRoute, + term: Term, + leaf: CellManifestLeaf, + image: CellImagePin, +} + +impl CellReconciliationAuthority { + pub fn route(&self) -> &GroupRoute { + &self.route + } + + pub(super) fn new( + group: GroupLog, + current: &PreparedGroupManifest, + cell: &CellHistory, + ) -> Result { + let (leaf, image) = current.pin_cell(&cell.cell).ok_or(RuntimeError::NotFound)?; + if &leaf.version.history != cell { + return Err(RuntimeError::Invalid("cell reconciliation history".into())); + } + let proof = Self { + group, + route: current.marker().route().clone(), + term: current.marker().position().term, + leaf: leaf.clone(), + image, + }; + proof.check()?; + Ok(proof) + } + + pub(crate) fn belongs_to_engine(&self, owner: &Arc<()>) -> bool { + Arc::ptr_eq(&self.group.owner, owner) + } + + pub fn term(&self) -> Term { + self.term + } + + pub fn cell(&self) -> &CellManifestLeaf { + &self.leaf + } + + pub fn image(&self) -> &CellImagePin { + &self.image + } + + pub fn check(&self) -> Result<(), RuntimeError> { + let cache = self.group.read()?; + let current = cache + .group_manifest + .as_ref() + .ok_or(RuntimeError::NotReady)?; + let key = cell_routing_key( + self.route.history.namespace, + &self.leaf.version.history.cell, + ); + if !current.is_active() + || self.term.get() == 0 + || cache.hard_state.term != self.term.get() + || current.position().term != self.term + || current.position().index.get() > cache.hard_state.commit + || current.route() != &self.route + || !cache.membership.permits_group(self.group.identity()) + || cache + .manifest_pins + .as_ref() + .and_then(|pins| pins.get(&key)) + .is_none_or(|(leaf, _)| leaf != &self.leaf) + { + return Err(RuntimeError::NotReady); + } + Ok(()) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/hibernate.rs b/crates/cellar/durability/raft/cell_groups/hibernate.rs new file mode 100644 index 000000000..fa7b833bf --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/hibernate.rs @@ -0,0 +1,435 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Replicated park preparation evidence. Local park and wake authority +//! require separate durable records and acknowledgements from the voters. + +use super::{CellGroupProposal, GroupManifest, ManifestError, ParkCertificate}; +use crate::raft::{ + cache::{CachedGroup, GroupLog, GroupWrite}, + storage::{LogError, ReplicaLog}, +}; +use cellar_logic::{ + cell_group::{GroupHistory, MembershipRevision, ParkGeneration}, + consensus::{ + LogIndex, LogPosition, MemberId, NodeIdentity, OperationId, ProcessGeneration, ProposalId, + ReplicaIdentity, Term, + }, +}; +use serde::{Deserialize, Serialize}; + +/// Historical cut selected by applying a committed PreparePark entry. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct GroupParkCut { + generation: u64, + term: u64, + membership_revision: u64, +} + +impl GroupParkCut { + pub(super) fn prepare( + generation: ParkGeneration, + term: Term, + membership: MembershipRevision, + applied: LogPosition, + applied_membership: u64, + ) -> Result { + // The committed index is the generation, so a successor leader + // cannot commit a second prepare with an already committed identity. + if generation.get() != applied.index.get() + || term != applied.term + || membership.get() != applied_membership + { + return Err(ManifestError::Invalid("park entry binding")); + } + let cut = Self { + generation: generation.get(), + term: term.get(), + membership_revision: membership.get(), + }; + if !cut.valid_at(applied, applied_membership) { + return Err(ManifestError::Invalid("park cut")); + } + Ok(cut) + } + + pub fn generation(&self) -> u64 { + self.generation + } + + pub fn position(&self) -> LogPosition { + LogPosition { + index: LogIndex::new(self.generation), + term: Term::new(self.term), + } + } + + pub fn membership_revision(&self) -> u64 { + self.membership_revision + } + + pub(super) fn valid_at(&self, applied: LogPosition, membership: u64) -> bool { + self.membership_revision > 0 + && self.membership_revision < self.generation + && self.membership_revision <= membership + && self.position().validate().is_ok() + && self.generation <= applied.index.get() + && self.term <= applied.term.get() + && (self.generation != applied.index.get() + || (self.term == applied.term.get() && self.membership_revision == membership)) + } +} + +pub(crate) const MAX_PARK_NEGOTIATION_MS: u64 = 5_000; + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) struct ParkPrepareRequest { + pub proposal: ProposalId, + pub term: Term, + pub membership: MembershipRevision, + pub expected_applied: LogPosition, + pub deadline_ms: u64, +} + +/// Leader-local admission, followed by ordinary Raft commitment and apply. +#[derive(Clone, Debug)] +pub(crate) struct ParkPreparation { + pub proposal: CellGroupProposal, + pub next_due_unix_millis: Option, + pub deadline_ms: u64, +} + +/// Physical evidence for one voter. Selection of a park certificate still +/// requires authenticated acknowledgements from the complete voter set. +#[derive(Debug, PartialEq, Eq)] +pub(crate) struct ParkAcknowledgement { + pub voter: ReplicaIdentity, + pub leader: MemberId, + pub cut: GroupParkCut, + pub sync_sequence: u64, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +enum LocalParkPhase { + Acknowledged, + Certified, + Parked, + Closed, + WakeClosed, +} + +/// Engine-local acknowledgement and irreversible close floor. Snapshots +/// carry the replicated cut, never a different disk's acknowledgement. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(crate) struct LocalParkRecord { + cut: GroupParkCut, + process: [u8; 16], + phase: LocalParkPhase, + #[serde(default, skip_serializing_if = "Option::is_none")] + certificate: Option>, +} + +#[derive(Clone)] +pub(crate) enum LocalParkChange { + Acknowledge(GroupParkCut), + Certify(Box), + Park(Box), + Close(ParkGeneration), +} + +impl LocalParkRecord { + pub(crate) fn is_open(&self) -> bool { + matches!( + self.phase, + LocalParkPhase::Acknowledged | LocalParkPhase::Certified | LocalParkPhase::Parked + ) + } + + pub(super) fn matches_acknowledgement( + &self, + cut: &GroupParkCut, + process: ProcessGeneration, + ) -> bool { + self.is_open() && &self.cut == cut && self.process == process.bytes() + } + + fn closed(&self) -> Self { + Self { + cut: self.cut.clone(), + process: self.process, + phase: if self.phase == LocalParkPhase::Parked + || self.phase == LocalParkPhase::WakeClosed + { + LocalParkPhase::WakeClosed + } else { + LocalParkPhase::Closed + }, + certificate: None, + } + } + + pub(crate) fn reserved_bytes(&self) -> usize { + self.certificate + .as_ref() + .map_or(0, |certificate| certificate.reserved_bytes()) + } + + pub(crate) fn certificate(&self) -> Option<&ParkCertificate> { + self.certificate.as_deref() + } + + pub(crate) fn matches_member(&self, member: u64) -> bool { + self.certificate.as_ref().is_none_or(|certificate| { + certificate.matches_journal(member, self.process) + && (self.phase == LocalParkPhase::Parked + || certificate + .leader() + .is_ok_and(|leader| leader.get() == member)) + }) + } + + pub(crate) fn parked(&self) -> bool { + self.phase == LocalParkPhase::Parked + } + + pub(super) fn wake_floor(&self) -> Option { + (self.phase == LocalParkPhase::WakeClosed).then(|| self.cut.position().index) + } + + fn validate_history(&self, manifest: &GroupManifest) -> Result<(), LogError> { + let latest = manifest.park_cut().ok_or(LogError::WrongIdentity)?; + if ProcessGeneration::new(self.process).is_err() + || !self + .cut + .valid_at(manifest.position(), manifest.membership_revision()) + || self.cut.generation() > latest.generation() + || self.cut.position().term > latest.position().term + || self.cut.membership_revision() > latest.membership_revision() + || (self.cut.generation() == latest.generation() && &self.cut != latest) + { + return Err(LogError::WrongIdentity); + } + Ok(()) + } +} + +impl CachedGroup { + pub(crate) fn quiescent_park_cut(&self, cut: &GroupParkCut) -> bool { + let at = cut.position(); + self.group_manifest.as_ref().is_some_and(|marker| { + marker.is_active() && marker.park_cut() == Some(cut) && marker.position() == at + }) && self.applied.position == at + && self.last_index() == at.index.get() + && self.hard_state.commit == at.index.get() + && self.hard_state.term == at.term.get() + && self.membership.revision().get() == cut.membership_revision() + && self.conf_state.voters_outgoing.is_empty() + && self.conf_state.learners.is_empty() + && self.conf_state.learners_next.is_empty() + && !self.conf_state.auto_leave + } + + pub(super) fn admit_park_acknowledgement( + &self, + cut: &GroupParkCut, + member: MemberId, + process: ProcessGeneration, + ) -> Result<(), LogError> { + if !self.quiescent_park_cut(cut) + || !self.conf_state.voters.contains(&member.get()) + || self.local_park.as_ref().is_some_and(|current| { + if current.is_open() { + !current.matches_acknowledgement(cut, process) + } else { + current.cut.generation() >= cut.generation() + } + }) + { + return Err(LogError::WrongIdentity); + } + Ok(()) + } + + pub(crate) fn validate_local_park(&self) -> Result<(), LogError> { + let Some(record) = &self.local_park else { + return Ok(()); + }; + let manifest = self + .group_manifest + .as_ref() + .ok_or(LogError::WrongIdentity)?; + record.validate_history(manifest)?; + if record.is_open() && !self.quiescent_park_cut(&record.cut) { + return Err(LogError::Corrupt("unclosed park acknowledgement".into())); + } + match (&record.phase, record.certificate()) { + (LocalParkPhase::Certified | LocalParkPhase::Parked, Some(certificate)) + if certificate.cut() == &record.cut => + { + certificate.validate(manifest.route(), &self.conf_state, &self.membership) + } + ( + LocalParkPhase::Acknowledged | LocalParkPhase::Closed | LocalParkPhase::WakeClosed, + None, + ) => Ok(()), + _ => Err(LogError::Corrupt("park certificate phase".into())), + } + } + + pub(super) fn admit_park_certificate( + &self, + certificate: &ParkCertificate, + member: MemberId, + identity: NodeIdentity, + ) -> Result<(), LogError> { + if !certificate.matches_leader(member, identity) + || self + .local_park + .as_ref() + .is_some_and(LocalParkRecord::parked) + { + return Err(LogError::WrongIdentity); + } + self.admit_park_selection(certificate, member, identity) + } + + pub(super) fn admit_park( + &self, + certificate: &ParkCertificate, + member: MemberId, + identity: NodeIdentity, + ) -> Result<(), LogError> { + if certificate.leader()? == member + && self + .local_park + .as_ref() + .and_then(LocalParkRecord::certificate) + != Some(certificate) + { + return Err(LogError::WrongIdentity); + } + self.admit_park_selection(certificate, member, identity) + } + + fn admit_park_selection( + &self, + certificate: &ParkCertificate, + member: MemberId, + identity: NodeIdentity, + ) -> Result<(), LogError> { + let cut = certificate.cut(); + let manifest = self + .group_manifest + .as_ref() + .ok_or(LogError::WrongIdentity)?; + if !self.quiescent_park_cut(cut) + || !certificate.matches_voter(member, identity) + || !self.local_park.as_ref().is_some_and(|record| { + record.matches_acknowledgement(cut, identity.process) + && record + .certificate() + .is_none_or(|current| current == certificate) + }) + { + return Err(LogError::WrongIdentity); + } + certificate.validate(manifest.route(), &self.conf_state, &self.membership) + } + + pub(crate) fn update_local_park( + &mut self, + previous: &Self, + change: Option<&LocalParkChange>, + member: MemberId, + identity: NodeIdentity, + ) -> Result<(), LogError> { + let process = identity.process; + let current = previous.local_park.as_ref(); + match change { + Some(LocalParkChange::Acknowledge(cut)) => { + previous.admit_park_acknowledgement(cut, member, process)?; + if !self.quiescent_park_cut(cut) + || !self.conf_state.voters.contains(&member.get()) + || self.hard_state.vote != previous.hard_state.vote + { + return Err(LogError::WrongIdentity); + } + if current.is_none_or(|record| !record.matches_acknowledgement(cut, process)) { + self.local_park = Some(LocalParkRecord { + cut: cut.clone(), + process: process.bytes(), + phase: LocalParkPhase::Acknowledged, + certificate: None, + }); + } + } + Some(LocalParkChange::Certify(certificate) | LocalParkChange::Park(certificate)) => { + let phase = if matches!(change, Some(LocalParkChange::Park(_))) { + previous.admit_park(certificate, member, identity)?; + self.admit_park(certificate, member, identity)?; + LocalParkPhase::Parked + } else { + previous.admit_park_certificate(certificate, member, identity)?; + self.admit_park_certificate(certificate, member, identity)?; + LocalParkPhase::Certified + }; + if self.hard_state.vote != previous.hard_state.vote { + return Err(LogError::WrongIdentity); + } + self.local_park = Some(LocalParkRecord { + cut: certificate.cut().clone(), + process: process.bytes(), + phase, + certificate: Some(certificate.clone()), + }); + } + Some(LocalParkChange::Close(generation)) => { + let record = current + .filter(|record| record.cut.generation() == generation.get()) + .ok_or(LogError::WrongIdentity)?; + self.local_park = Some(record.closed()); + } + None => { + if let Some(record) = current { + if record.is_open() + && (!self.quiescent_park_cut(&record.cut) + || self.hard_state.vote != previous.hard_state.vote) + { + self.local_park = Some(record.closed()); + } + } + } + } + Ok(()) + } +} + +impl ReplicaLog { + /// Close an earlier process's local acknowledgement before starting any + /// retained group owner. This never rewrites the selected Raft state. + pub(crate) async fn close_retained_park( + &self, + group: &GroupLog, + ) -> Result<(), LogError> { + let acknowledged = group + .read()? + .local_park + .as_ref() + .filter(|record| record.is_open()) + .map(|record| record.cut.generation()); + if let Some(generation) = acknowledged { + let generation = + ParkGeneration::new(generation).map_err(|_| LogError::WrongIdentity)?; + let mut write = GroupWrite::new(group); + write.local_park = Some(LocalParkChange::Close(generation)); + self.persist_cells(OperationId::new(1), vec![write]).await?; + } + Ok(()) + } +} + +#[cfg(test)] +#[path = "hibernate_tests.rs"] +mod tests; diff --git a/crates/cellar/durability/raft/cell_groups/hibernate_tests.rs b/crates/cellar/durability/raft/cell_groups/hibernate_tests.rs new file mode 100644 index 000000000..6d764968b --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/hibernate_tests.rs @@ -0,0 +1,787 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use crate::{ + asyncrt::ExecutionDomain, + raft::{ + catalog::{Catalog, CatalogOptions, InitialMember, NamespaceConfig, VoterPolicy}, + cell_groups::{ + CellGroupBirthSpec, CellGroupBootstrapPermit, DirectoryLimits, RangeDirectory, + }, + }, +}; +use cellar_logic::{ + cell_group::{CellRange, GroupHistory, GroupRoute, RouteEpoch}, + consensus::{ClusterId, GroupId, IncarnationId, NamespaceId, NodeId, ProposalId}, +}; +use std::sync::Arc; + +fn marker(index: u64) -> GroupManifest { + let mut marker = GroupManifest::genesis(GroupRoute { + history: GroupHistory { + cluster: ClusterId::new([1; 16]).unwrap(), + namespace: NamespaceId::new([2; 16]).unwrap(), + group: GroupId::new([3; 16]).unwrap(), + incarnation: IncarnationId::new([4; 16]).unwrap(), + }, + range: CellRange::all(), + epoch: RouteEpoch::new(1).unwrap(), + birth_digest: [5; 32], + schema_digest: [6; 32], + }) + .unwrap(); + marker.position = LogPosition { + index: LogIndex::new(index), + term: Term::new(2), + }; + marker +} + +#[test] +fn park_prepare_binds_committed_position_and_membership() { + let mut marker = marker(3); + let generation = ParkGeneration::new(3).unwrap(); + let term = Term::new(2); + let membership = MembershipRevision::new(1).unwrap(); + for (generation, term, membership) in [ + (ParkGeneration::new(2).unwrap(), term, membership), + (generation, Term::new(3), membership), + (generation, term, MembershipRevision::new(2).unwrap()), + ] { + assert!(marker.prepare_park(generation, term, membership).is_err()); + assert!(marker.park_cut().is_none()); + } + marker.prepare_park(generation, term, membership).unwrap(); + let cut = marker.park_cut().unwrap(); + assert_eq!(cut.generation(), generation.get()); + assert_eq!(cut.position(), marker.position()); + assert_eq!(cut.membership_revision(), membership.get()); + assert!(marker.is_active()); + + let bytes = serde_json::to_vec(&marker).unwrap(); + assert_eq!( + serde_json::from_slice::(&bytes).unwrap(), + marker + ); + let mut value = serde_json::to_value(&marker).unwrap(); + value["park"]["term"] = 1.into(); + assert!(serde_json::from_value::(value).is_err()); +} + +#[test] +fn park_cut_survives_advance_and_refuses_checkpoint_regression() { + let mut prepared = marker(3); + prepared + .prepare_park( + ParkGeneration::new(3).unwrap(), + Term::new(2), + MembershipRevision::new(1).unwrap(), + ) + .unwrap(); + let mut advanced = prepared.clone(); + advanced.position.index = LogIndex::new(4); + advanced.membership_revision = 4; + advanced.check().unwrap(); + assert!(advanced.preserves_park_cut(&prepared)); + assert!(!marker(4).preserves_park_cut(&prepared)); + assert!(advanced + .prepare_park( + ParkGeneration::new(3).unwrap(), + Term::new(2), + MembershipRevision::new(1).unwrap(), + ) + .is_err()); + + let mut forged = marker(5); + forged + .prepare_park( + ParkGeneration::new(5).unwrap(), + Term::new(2), + MembershipRevision::new(1).unwrap(), + ) + .unwrap(); + assert!(!forged.preserves_park_cut(&advanced)); + + let mut next = advanced.clone(); + next.position.index = LogIndex::new(5); + next.prepare_park( + ParkGeneration::new(5).unwrap(), + Term::new(2), + MembershipRevision::new(4).unwrap(), + ) + .unwrap(); + assert!(next.preserves_park_cut(&advanced)); + assert!(!advanced.preserves_park_cut(&next)); +} + +#[test] +fn park_certificate_requires_exact_complete_voter_evidence() -> anyhow::Result<()> { + use crate::raft::member_state::GroupMembership; + use ::raft::prelude::ConfState; + use cellar_logic::consensus::DiskLineage; + + let mut marker = marker(3); + marker.prepare_park( + ParkGeneration::new(3)?, + Term::new(2), + MembershipRevision::new(1)?, + )?; + let cut = marker.park_cut().unwrap(); + let route = marker.route(); + let voters: Vec<_> = (1..=3) + .map(|id| -> anyhow::Result<_> { + Ok(ReplicaIdentity { + scope: route.history, + member: MemberId::new(u64::from(id))?, + node: NodeId::new([id; 16])?, + disk: DiskLineage::new([id + 3; 16])?, + process: ProcessGeneration::new([id + 6; 16])?, + }) + }) + .collect::>()?; + let leader = &voters[0]; + let evidence = || { + voters + .iter() + .rev() + .map(|voter| ParkAcknowledgement { + voter: voter.clone(), + leader: leader.member, + cut: cut.clone(), + sync_sequence: 10, + }) + .collect::>() + }; + let certificate = ParkCertificate::assemble(route, cut.clone(), leader, &voters, evidence())?; + let conf = ConfState { + voters: vec![3, 1, 2], + ..Default::default() + }; + let membership = GroupMembership::initial(&conf); + certificate.validate(route, &conf, &membership)?; + assert_eq!( + serde_json::from_slice::(&serde_json::to_vec(&certificate)?)?, + certificate + ); + let mut other_route = route.clone(); + other_route.history.incarnation = IncarnationId::new([11; 16])?; + assert!(certificate + .validate(&other_route, &conf, &membership) + .is_err()); + let mut joint = conf.clone(); + joint.voters_outgoing.push(1); + assert!(certificate.validate(route, &joint, &membership).is_err()); + + let mut partial = evidence(); + partial.pop(); + assert!(ParkCertificate::assemble(route, cut.clone(), leader, &voters, partial).is_err()); + for field in 0..6 { + let mut replies = evidence(); + match field { + 0 => replies[0].voter = replies[1].voter.clone(), + 1 => replies[0].voter.process = ProcessGeneration::new([12; 16])?, + 2 => replies[0].voter.disk = DiskLineage::new([12; 16])?, + 3 => replies[0].leader = voters[1].member, + 4 => replies[0].sync_sequence = 0, + _ => { + replies[0].cut = GroupParkCut::prepare( + ParkGeneration::new(4)?, + Term::new(2), + MembershipRevision::new(1)?, + LogPosition { + term: Term::new(2), + index: LogIndex::new(4), + }, + 1, + )? + } + } + assert!(ParkCertificate::assemble(route, cut.clone(), leader, &voters, replies).is_err()); + } + Ok(()) +} + +async fn bootstrap( + domain: &ExecutionDomain, + log: &ReplicaLog, + voters: u8, +) -> anyhow::Result { + let route = marker(1).route().clone(); + let catalog = Catalog::new( + Arc::new(object_store::memory::InMemory::new()), + CatalogOptions::default(), + )?; + let mut config = NamespaceConfig::new( + route.history.cluster, + route.history.namespace, + cellar_logic::NamespaceMode::Raft, + route.schema_digest, + ); + if voters == 1 { + config.voters = VoterPolicy::SingleVoterDevelopment; + } + let namespace = domain.run(catalog.open_namespace(config)).await??; + let directory = RangeDirectory::new(namespace, DirectoryLimits::default())?; + let request = directory.prepare_group_birth(CellGroupBirthSpec { + history: route.history, + range: route.range, + epoch: route.epoch, + creation: ProposalId::new([16; 16])?, + voters: (1..=voters) + .map(|id| -> anyhow::Result<_> { + Ok(InitialMember { + member: MemberId::new(u64::from(id))?, + node: NodeId::new([id + 10; 16])?, + }) + }) + .collect::>()?, + learners: vec![], + })?; + domain + .run(directory.reserve_group_birth(&request)) + .await??; + let initialize = + directory.prepare_initialization(ProposalId::new([17; 16])?, &[request.route().clone()])?; + domain.run(directory.publish(&initialize)).await??; + let birth = domain + .run(directory.materialize_group_birth(&request)) + .await??; + Ok(domain + .run(directory.bootstrap_group_member(&birth, log)) + .await??) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn park_acknowledgement_waits_for_sync_and_reopens_closed() -> anyhow::Result<()> { + use crate::{ + application::consensus::group_protocol::{ + CellGroupEntry, GroupCommand, GroupEntryBinding, GroupEntryLimits, + }, + raft::{ + cell_groups::{CellGroupEffect, CellGroupRuntime, ManifestPageStore, ManifestTree}, + completion::RuntimeError, + options::{LogOpenMode, RaftRuntimeOptions, ReplicaLogOptions}, + storage_fault as fault, + }, + }; + use ::raft::prelude::{Entry, HardState, Message, MessageType}; + use cellar_logic::consensus::DiskLineage; + use cellar_ltx::DirectFileSystem; + use std::time::Duration; + + let directory = tempfile::tempdir()?; + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let mut options = ReplicaLogOptions::new( + directory.path().join("log"), + NodeId::new([11; 16])?, + DiskLineage::new([12; 16])?, + ProcessGeneration::new([13; 16])?, + LogOpenMode::Create, + ); + options.purge_interval_ms = None; + let gate = fault::Gate::new(None); + let log = domain + .run(ReplicaLog::open( + options.clone(), + Arc::new(fault::FaultFileSystem { gate: gate.clone() }), + )) + .await??; + let member = MemberId::new(1)?; + let leader = MemberId::new(2)?; + let permit = bootstrap(&domain, &log, 3).await?; + let group = permit.group().clone(); + let route = permit.route().clone(); + let at = marker(2).position(); + let generation = ParkGeneration::new(at.index.get())?; + let membership = MembershipRevision::new(1)?; + let genesis = group.group_manifest()?.unwrap(); + let store_group = group.clone(); + let page_path = directory.path().join("pages"); + let (pages, prepared) = domain + .blocking(move || -> anyhow::Result<_> { + let mut pages = ManifestPageStore::open( + &page_path, + Arc::new(DirectFileSystem), + 128, + 1024 * 1024, + 16, + )?; + let tree = ManifestTree::new(genesis.route().clone(), Default::default())?; + let base = tree.prepare_complete( + &store_group, + &mut pages, + genesis.root(), + genesis.position(), + genesis.membership_revision(), + |_| unreachable!("empty genesis"), + )?; + let mut prepared = tree.prepare_unchanged(&pages, &base, at)?; + prepared + .marker + .prepare_park(generation, at.term, membership)?; + Ok((pages, prepared)) + })? + .await??; + let cut = prepared.marker().park_cut().unwrap().clone(); + let acknowledge = |group: &GroupLog| { + let mut write = GroupWrite::new(group); + write.local_park = Some(LocalParkChange::Acknowledge(cut.clone())); + write + }; + let operation = OperationId::new(1); + assert!(matches!( + log.persist_cells(operation, vec![acknowledge(&group)]) + .await, + Err(LogError::WrongIdentity) + )); + + let mut apply = GroupWrite::new(&group); + let data = CellGroupEntry { + binding: GroupEntryBinding { + group: route.history, + route_epoch: route.epoch, + birth_digest: route.birth_digest, + schema_digest: route.schema_digest, + proposal: ProposalId::new([14; 16])?, + }, + command: GroupCommand::PreparePark { + generation, + term: at.term, + membership, + }, + } + .encode(&route, GroupEntryLimits::default())?; + apply.entries.push(Entry { + index: at.index.get(), + term: at.term.get(), + data: data.into(), + ..Default::default() + }); + let hard = HardState { + term: at.term.get(), + vote: leader.get(), + commit: at.index.get(), + ..Default::default() + }; + apply.hard_state = Some(hard.clone()); + apply.group_manifest = Some(prepared.clone()); + log.persist_cells(operation, vec![apply]).await?; + let (runtime, mut events) = CellGroupRuntime::start( + log.clone(), + RaftRuntimeOptions { + tick_interval_ms: None, + election_seed: Some([18; 32]), + ..Default::default() + }, + ) + .await?; + runtime.attach(permit, prepared).await?; + assert!(matches!( + runtime + .acknowledge_park(route.history, leader, cut.clone()) + .await, + Err(RuntimeError::NotReady) + )); + let mut heartbeat = Message { + from: leader.get(), + to: member.get(), + term: at.term.get(), + commit: at.index.get(), + ..Default::default() + }; + heartbeat.set_msg_type(MessageType::MsgHeartbeat); + runtime.step(route.history, heartbeat).await?; + let event = domain + .run(crate::asyncrt::timeout( + Duration::from_secs(5), + events.recv(), + )) + .await?? + .expect("heartbeat response"); + assert!(matches!( + event.effect(), + Ok(CellGroupEffect::Message(message)) + if message.get_msg_type() == MessageType::MsgHeartbeatResponse + )); + drop(event); + assert!(matches!( + runtime + .acknowledge_park(route.history, member, cut.clone()) + .await, + Err(RuntimeError::NotReady) + )); + + let release = gate.arm(fault::HOLD_SYNC); + let requester = runtime.clone(); + let history = route.history; + let requested_cut = cut.clone(); + let mut pending = domain.spawn(async move { + requester + .acknowledge_park(history, leader, requested_cut) + .await + })?; + let entered = domain + .run(crate::asyncrt::timeout( + Duration::from_secs(5), + gate.entered.notified(), + )) + .await?; + let hidden = group.read()?.local_park.is_none(); + let unresolved = futures_util::poll!(&mut pending).is_pending(); + drop(release); + entered?; + assert!(hidden); + assert!(unresolved); + let receipt = pending.await??; + assert_eq!(receipt.voter, *group.identity()); + assert_eq!(receipt.leader, leader); + assert_eq!(receipt.cut, cut); + assert_ne!(receipt.sync_sequence, 0); + let duplicate = runtime + .acknowledge_park(route.history, leader, cut.clone()) + .await?; + assert_eq!(duplicate.voter, receipt.voter); + assert_eq!(duplicate.cut, receipt.cut); + assert!(duplicate.sync_sequence > receipt.sync_sequence); + runtime.close().await?; + let acknowledged = LocalParkRecord { + cut: cut.clone(), + process: options.process.bytes(), + phase: LocalParkPhase::Acknowledged, + certificate: None, + }; + assert_eq!(group.read()?.local_park.as_ref(), Some(&acknowledged)); + + // An ordinary Ready that advances the term closes the acknowledgement + // in the same engine batch, even when the applied park cut has not moved. + let mut higher_term = GroupWrite::new(&group); + higher_term.hard_state = Some(HardState { + term: at.term.get() + 1, + vote: 0, + ..hard.clone() + }); + let next = group.read()?.changed(&higher_term, 16, route.history)?; + assert_eq!(next.local_park.as_ref(), Some(&acknowledged.closed())); + + log.close().await?; + drop(group); + drop(pages); + options.mode = LogOpenMode::Resume; + options.process = ProcessGeneration::new([15; 16])?; + let log = domain + .run(ReplicaLog::open( + options.clone(), + Arc::new(DirectFileSystem), + )) + .await??; + let group = log.cell_group(&route.history)?; + assert_eq!(group.read()?.local_park.as_ref(), Some(&acknowledged)); + assert_ne!(options.process.bytes(), acknowledged.process); + assert!(matches!( + log.persist_cells(operation, vec![acknowledge(&group)]) + .await, + Err(LogError::WrongIdentity) + )); + + log.close_retained_park(&group).await?; + log.close_retained_park(&group).await?; + assert_eq!(group.read()?.hard_state, hard); + assert_eq!( + group.read()?.local_park.as_ref(), + Some(&acknowledged.closed()) + ); + assert!(matches!( + log.persist_cells(operation, vec![acknowledge(&group)]) + .await, + Err(LogError::WrongIdentity) + )); + log.close().await?; + + let log = domain + .run(ReplicaLog::open(options, Arc::new(DirectFileSystem))) + .await??; + let group = log.cell_group(&route.history)?; + let record = group.read()?.local_park.clone().unwrap(); + assert_eq!(record, acknowledged.closed()); + assert_eq!(group.group_manifest()?.unwrap().park_cut(), Some(&cut)); + log.close().await?; + domain.shutdown().await; + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn leader_fences_park_preparation_through_timeout_apply() -> anyhow::Result<()> { + exercise_park(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn parked_voter_closes_before_wake_and_requires_fresh_apply() -> anyhow::Result<()> { + exercise_park(true).await +} + +async fn exercise_park(enter_park: bool) -> anyhow::Result<()> { + use crate::raft::{ + cell_groups::{ + CellGroupApplier, CellGroupApplyImages, CellGroupEffect, CellGroupRuntime, + ManifestPageStore, ManifestTree, + }, + completion::RuntimeError, + options::{LogOpenMode, RaftRuntimeOptions, ReplicaLogOptions}, + }; + use anyhow::Context as _; + use cellar_logic::consensus::DiskLineage; + use cellar_ltx::{ + generation::{GenerationStore, ReadServices}, + DirectFileSystem, + }; + use std::time::Duration; + + let root = tempfile::tempdir()?; + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let log = domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + root.path().join("log"), + NodeId::new([11; 16])?, + DiskLineage::new([12; 16])?, + ProcessGeneration::new([13; 16])?, + LogOpenMode::Create, + ), + Arc::new(DirectFileSystem), + )) + .await??; + let permit = bootstrap(&domain, &log, 1).await?; + let history = permit.route().history; + let member = permit.group().identity().member; + let group = permit.group().clone(); + let page_directory = root.path().join("pages"); + let images_directory = root.path().join("images"); + let (applier, genesis) = domain + .blocking(move || -> anyhow::Result<_> { + let mut pages = ManifestPageStore::open( + &page_directory, + Arc::new(DirectFileSystem), + 128, + 1024 * 1024, + 16, + )?; + let marker = group.group_manifest()?.context("genesis marker")?; + let tree = ManifestTree::new(marker.route().clone(), Default::default())?; + let genesis = tree.prepare_complete( + &group, + &mut pages, + marker.root(), + marker.position(), + marker.membership_revision(), + |_| unreachable!("empty genesis"), + )?; + let images = CellGroupApplyImages::Materialized { + images: GenerationStore::open( + images_directory, + Arc::new(DirectFileSystem), + Default::default(), + )?, + services: ReadServices { + now_unix_millis: Arc::new(crate::asyncrt::wall_ms), + fill_random: Arc::new(|bytes| bytes.fill(17)), + }, + }; + let applier = CellGroupApplier::new( + group, + pages, + images, + Default::default(), + Default::default(), + 32, + )?; + Ok((applier, genesis)) + })? + .await??; + let (runtime, mut events) = CellGroupRuntime::start( + log.clone(), + RaftRuntimeOptions { + tick_interval_ms: None, + election_seed: Some([19; 32]), + ..Default::default() + }, + ) + .await?; + runtime.attach(permit, genesis).await?; + runtime.campaign(history).await?; + + let mut cut = None; + let mut parked_certificate = None; + for phase in ["election", "park", "resume"] { + let event = domain + .run(crate::asyncrt::timeout( + Duration::from_secs(10), + events.recv(), + )) + .await?? + .context("expected committed apply")?; + let work = event.retain_apply().context("apply effect")?; + drop(event); + if phase == "resume" { + assert!(work.entry.data.is_empty()); + assert!(!runtime.status(history).await?.ready); + if enter_park { + let (_, position): (ParkPrepareRequest, LogPosition) = cut.context("park cut")?; + let group = log.cell_group(&history)?; + let cache = group.read()?; + let journal = cache.local_park.as_ref().context("durable wake closure")?; + assert_eq!(journal.wake_floor(), Some(position.index)); + assert!(journal.certificate().is_none()); + assert!(work.ticket.position().term > position.term); + } + } + let prepare = work.clone(); + let applier = applier.clone(); + let next = domain + .blocking(move || applier.prepare(&prepare))? + .await??; + runtime.complete_apply(history, work, next).await?; + let event = domain + .run(crate::asyncrt::timeout( + Duration::from_secs(10), + events.recv(), + )) + .await?? + .context("expected durable apply")?; + let Ok(CellGroupEffect::Applied { manifest, .. }) = event.effect() else { + anyhow::bail!("unexpected cell-group effect"); + }; + let selected = manifest.park_cut().cloned(); + let status = runtime.status(history).await?; + match phase { + "election" => { + assert!(status.ready); + let request = ParkPrepareRequest { + proposal: ProposalId::new([20; 16])?, + term: status.term, + membership: MembershipRevision::new(manifest.membership_revision())?, + expected_applied: status.applied, + deadline_ms: log.domain().mono_ms()? + MAX_PARK_NEGOTIATION_MS, + }; + let stale = ParkPrepareRequest { + term: Term::new(status.term.get() - 1), + ..request + }; + assert!(matches!( + runtime.prepare_park(history, stale).await, + Err(RuntimeError::StaleCompletion) + )); + let prepared = runtime.prepare_park(history, request).await?; + let retry = ParkPrepareRequest { + deadline_ms: request.deadline_ms - 1, + ..request + }; + let duplicate = runtime.prepare_park(history, retry).await?; + assert_eq!(prepared.proposal.position, duplicate.proposal.position); + assert_eq!(duplicate.deadline_ms, request.deadline_ms); + assert!(prepared.next_due_unix_millis.is_none()); + assert!(!runtime.status(history).await?.ready); + cut = Some((request, prepared.proposal.position)); + } + "park" => { + let (request, position) = cut.context("park preparation")?; + let selected = selected.context("committed park cut")?; + assert_eq!(selected.position(), position); + assert!(!status.ready); + assert!(matches!( + runtime.read_index(history, vec![1]).await, + Err(RuntimeError::NotReady) + )); + let receipt = runtime.acknowledge_park(history, member, selected).await?; + assert_eq!(receipt.cut.position(), position); + let replica = receipt.voter.clone(); + let sequence = receipt.sync_sequence; + let certificate = ParkCertificate::assemble( + manifest.route(), + receipt.cut.clone(), + &replica, + std::slice::from_ref(&replica), + vec![receipt], + )?; + let certified = runtime.certify_park(history, certificate.clone()).await?; + assert_eq!(certified.certificate(), &certificate); + assert!(certified.sync_sequence() > sequence); + let retry = runtime.certify_park(history, certificate.clone()).await?; + assert_eq!(retry.certificate(), &certificate); + + let later = runtime + .acknowledge_park(history, member, certificate.cut().clone()) + .await?; + let conflict = ParkCertificate::assemble( + manifest.route(), + later.cut.clone(), + &replica, + std::slice::from_ref(&replica), + vec![later], + )?; + assert_ne!(conflict, certificate); + assert!(matches!( + runtime.certify_park(history, conflict).await, + Err(RuntimeError::Storage(LogError::WrongIdentity)) + )); + let group = log.cell_group(&history)?; + assert_eq!( + group + .read()? + .local_park + .as_ref() + .and_then(LocalParkRecord::certificate), + Some(&certificate) + ); + if enter_park { + runtime.enter_park(history, certificate.clone()).await?; + assert!(group + .read()? + .local_park + .as_ref() + .is_some_and(LocalParkRecord::parked)); + assert!(!runtime.status(history).await?.ready); + parked_certificate = Some(certificate); + } + domain + .run(async { crate::asyncrt::sleep_until(request.deadline_ms).await }) + .await?; + runtime.tick().await?; + if enter_park { + let quiet = domain + .run(crate::asyncrt::timeout( + Duration::from_millis(30), + events.recv(), + )) + .await?; + assert!(quiet.is_err(), "parked voter emitted tick work"); + assert_eq!(runtime.status(history).await?.applied, position); + assert!(matches!( + runtime.read_index(history, vec![2]).await, + Err(RuntimeError::NotReady) + )); + } + } + _ => { + let (_, position) = cut.context("park preparation")?; + let selected = selected.context("retained park cut")?; + assert_eq!(selected.position(), position); + assert!(status.ready); + assert_eq!(status.applied.index.get(), position.index.get() + 1); + if let Some(certificate) = parked_certificate.take() { + assert!(matches!( + runtime.enter_park(history, certificate).await, + Err(RuntimeError::NotReady) + )); + } + assert!(matches!( + runtime.acknowledge_park(history, member, selected).await, + Err(RuntimeError::NotReady) + )); + } + } + } + runtime.close().await?; + log.close().await?; + domain.shutdown().await; + Ok(()) +} diff --git a/crates/cellar/durability/raft/cell_groups/host/mod.rs b/crates/cellar/durability/raft/cell_groups/host/mod.rs new file mode 100644 index 000000000..4e5338be3 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/host/mod.rs @@ -0,0 +1,407 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Namespace registrations over one shared node controller. Cell handles are +//! disposable values; obtaining or cloning one neither enrolls nor caches a cell. +mod storage; +pub use storage::CellGroupNamespaceStorage; + +use super::*; +use crate::application::consensus::{OperationContract, RaftClass, RegistrationError}; +use crate::raft::{ + budget::{Budget, Permit}, + completion::RuntimeError, + transport::RaftTransport, +}; +use cellar_logic::{ + cell_group::CellHistory, + consensus::{ClusterId, NamespaceId}, +}; +use std::{ + collections::BTreeMap, + sync::{ + atomic::{AtomicBool, Ordering}, + Arc, Mutex, + }, +}; + +pub struct CellGroupHostRegistration { + pub dispatcher: CellGroupDispatcher, + pub storage: CellGroupNamespaceStorage, +} + +#[derive(Clone, Copy, Debug)] +pub struct CellGroupHostLimits { + pub max_namespaces: usize, + /// Includes each registration's configured invocation, route-cache and + /// directory-cache capacities, not only their current occupancy. + pub namespace_bytes: usize, +} + +struct Binding { + dispatcher: CellGroupDispatcher, + class: Arc, + storage: Mutex>, + _admission: Permit, +} + +struct Inner { + controller: CellGroupController, + domain: crate::asyncrt::DomainToken, + namespaces: Arc>>, + budget: Arc, + closed: AtomicBool, +} + +/// Bind compiled namespaces to an already initialized shared node controller. +/// Group placement and physical bootstrap remain node-startup responsibilities; +/// these registrations grant no voting, leader or serving authority. +#[derive(Clone)] +pub struct CellGroupHost { + inner: Arc, +} + +#[derive(Clone, Copy, Debug, serde::Serialize)] +pub struct CellGroupHostUsage { + pub namespaces: usize, + pub namespace_bytes: usize, + pub namespace_byte_limit: usize, + pub controller: CellGroupControllerUsage, +} + +impl CellGroupHost { + pub fn bind( + runtime: &CellGroupRuntime, + sql: &CellGroupSqlPool, + transport: &RaftTransport, + controller: CellGroupController, + registrations: Vec, + limits: CellGroupHostLimits, + ) -> Result { + let domain = runtime.log().domain(); + if !controller.belongs_to(runtime) + || !transport.uses_shared_runtime(runtime) + || !sql.belongs_to_engine(&runtime.log().owner()) + || !sql.uses_domain(&domain) + { + return Err(CellDispatchError::Invalid("shared host resource owner")); + } + if !(1..=4096).contains(&limits.max_namespaces) + || registrations.is_empty() + || registrations.len() > limits.max_namespaces + || limits.namespace_bytes < 8192 + { + return Err(CellDispatchError::Invalid("shared host namespace limits")); + } + let budget = Budget::new(limits.namespace_bytes, limits.max_namespaces); + let mut namespaces = BTreeMap::new(); + for registration in registrations { + let dispatcher = registration.dispatcher; + if !dispatcher.uses_domain(&domain) { + return Err(CellDispatchError::Invalid("shared host namespace domain")); + } + let (config, class, bytes) = dispatcher.host_binding()?; + controller.require_host_effects(&dispatcher)?; + registration.storage.validate_namespace(&config)?; + let bytes = bytes + .checked_add(registration.storage.bytes()) + .ok_or(CellDispatchError::Runtime(RuntimeError::Overloaded))?; + let admission = budget + .acquire(bytes) + .ok_or(CellDispatchError::Runtime(RuntimeError::Overloaded))?; + let key = (config.cluster, config.namespace); + if namespaces.contains_key(&key) { + return Err(CellDispatchError::Invalid("duplicate host namespace")); + } + namespaces.insert( + key, + Arc::new(Binding { + dispatcher, + class, + storage: Mutex::new(Some(registration.storage)), + _admission: admission, + }), + ); + } + let namespaces = Arc::new(namespaces); + controller + .own_host_resources(Arc::new(HostResources { + namespaces: namespaces.clone(), + })) + .map_err(CellDispatchError::Runtime)?; + Ok(Self { + inner: Arc::new(Inner { + controller, + domain, + namespaces, + budget, + closed: AtomicBool::new(false), + }), + }) + } + + /// Selecting a history allocates no permanent binding and opens no SQL. + /// Each invocation still obtains exact runtime/catalog authority. + pub fn cell(&self, history: CellHistory) -> Result { + self.inner.check_open()?; + let binding = self + .inner + .namespaces + .get(&(history.cluster, history.namespace)) + .ok_or(CellDispatchError::Runtime(RuntimeError::NotFound))? + .clone(); + Ok(CellGroupCellHandle { + host: self.inner.clone(), + binding, + history, + }) + } + + /// Namespace capability lookup only; supplies no group placement or startup authority. + pub(crate) fn directory( + &self, + history: cellar_logic::cell_group::GroupHistory, + ) -> Result { + self.namespace_directory(history.cluster, history.namespace) + } + + pub(crate) fn namespace_directory( + &self, + cluster: cellar_logic::consensus::ClusterId, + namespace: cellar_logic::consensus::NamespaceId, + ) -> Result { + self.inner.check_open()?; + self.inner + .namespaces + .get(&(cluster, namespace)) + .map(|binding| binding.dispatcher.directory()) + .ok_or(CellDispatchError::Runtime(RuntimeError::NotFound)) + } + + pub fn is_closed(&self) -> bool { + self.inner.closed.load(Ordering::Acquire) + || self.inner.domain.mono_ms().is_err() + || self.inner.controller.usage().closing + } + + pub fn usage(&self) -> CellGroupHostUsage { + CellGroupHostUsage { + namespaces: self.inner.namespaces.len(), + namespace_bytes: self.inner.budget.usage().0, + namespace_byte_limit: self.inner.budget.capacity().0, + controller: self.inner.controller.usage(), + } + } + + /// Seal all handles, then join the node controller's physical owners. + /// Cancellation cannot reopen admission or cancel those physical joins. + pub async fn close(&self) -> Result<(), CellGroupControllerError> { + self.inner.closed.store(true, Ordering::Release); + self.inner.controller.close().await + } +} + +/// Registered exactly once with the controller's physical join owner. This +/// retains no public controller handle and cannot prevent last-handle shutdown. +pub(in crate::raft::cell_groups) struct HostResources { + namespaces: Arc>>, +} + +impl HostResources { + pub(in crate::raft::cell_groups) fn release(&self) -> Result<(), CellGroupControllerError> { + let mut released = Ok(()); + for binding in self.namespaces.values() { + match binding.storage.lock() { + Ok(mut storage) => { + storage.take(); + } + Err(error) => { + error.into_inner().take(); + released = Err(CellGroupControllerError::Invalid( + "namespace resource owner poisoned", + )); + } + } + } + released + } +} + +impl Binding { + fn options(&self, history: &CellHistory) -> Result { + self.storage + .lock() + .map_err(|_| CellDispatchError::Runtime(RuntimeError::Closed))? + .as_ref() + .map(|storage| storage.for_cell(history)) + .ok_or(CellDispatchError::Runtime(RuntimeError::Closed)) + } +} + +impl Inner { + fn check_open(&self) -> Result<(), CellDispatchError> { + if crate::application::consensus::callback_active() { + return Err(CellDispatchError::Execution( + crate::raft::execution::ExecutionError::Refused( + cellar_logic::consensus::CommandFailure::UnsupportedCapability, + ), + )); + } + if self.closed.load(Ordering::Acquire) + || self.domain.mono_ms().is_err() + || self.controller.usage().closing + { + return Err(CellDispatchError::Runtime(RuntimeError::Closed)); + } + Ok(()) + } +} + +/// A logical cell history, independent of its current physical group binding. +#[derive(Clone)] +pub struct CellGroupCellHandle { + host: Arc, + binding: Arc, + history: CellHistory, +} + +impl CellGroupCellHandle { + pub(crate) fn definition_digest(&self) -> [u8; 32] { + self.binding.class.definition_digest() + } + + pub fn history(&self) -> &CellHistory { + &self.history + } + + pub fn contract(&self, operation: &str) -> Result { + if operation.len() > 128 { + return Err(RegistrationError::Invalid("operation bounds")); + } + self.binding + .class + .contract(operation) + .ok_or_else(|| RegistrationError::UnknownOperation(operation.into())) + } + + /// Queue before returning the response waiter. No group ID is retained by + /// this handle, and cancellation cannot release an admitted SQL producer. + pub fn command( + &self, + request: CellCommandRequest, + limits: CellRoutingLimits, + ) -> Result { + self.host.check_open().map_err(|cause| CellCommandError { + command: request.id, + ambiguous: false, + cause: Box::new(cause), + })?; + let options = self + .binding + .options(&self.history) + .map_err(|cause| CellCommandError { + command: request.id, + ambiguous: false, + cause: Box::new(cause), + })?; + self.host.controller.dispatch_command( + &self.binding.dispatcher, + self.history.clone(), + request, + limits, + options, + ) + } + + /// Queue a read with its explicit consistency and original invocation deadline. + pub fn query( + &self, + request: CellQueryRequest, + limits: CellRoutingLimits, + ) -> Result { + self.query_routed(request, limits, CellQueryRouting::Local) + } + + pub fn query_routed( + &self, + request: CellQueryRequest, + limits: CellRoutingLimits, + routing: CellQueryRouting, + ) -> Result { + self.host.check_open()?; + let invocation = self.binding.dispatcher.query_routed( + self.history.clone(), + request, + limits, + None, + routing, + )?; + self.host + .controller + .query(invocation, self.binding.options(&self.history)?) + } + + pub fn enroll( + &self, + request: CellEnrollmentRequest, + limits: CellRoutingLimits, + ) -> impl std::future::Future>> + + Send + + 'static { + let opened = self.host.check_open().map_err(|cause| { + Arc::new(CellEnrollmentError { + creation: request.creation, + ambiguous: false, + cause: Box::new(cause), + }) + }); + let creation = request.creation; + let invocation = + self.binding + .dispatcher + .enroll_routed(self.history.clone(), request, limits); + let host = self.host.clone(); + let binding = self.binding.clone(); + let history = self.history.clone(); + async move { + opened?; + let invocation = invocation.await.map_err(Arc::new)?; + let options = binding.options(&history).map_err(|cause| { + Arc::new(CellEnrollmentError { + creation, + ambiguous: false, + cause: Box::new(cause), + }) + })?; + let call = host + .controller + .enroll(invocation, options) + .map_err(|e| Arc::new(e.error))?; + call.response().await + } + } + + /// Absence remains ambiguous and never releases an original producer. + pub fn receipt( + &self, + request: CellReceiptRequest, + limits: CellRoutingLimits, + ) -> impl std::future::Future>> + + Send + + 'static { + let invocation = self.host.check_open().and_then(|()| { + self.binding + .dispatcher + .lookup_receipt(self.history.clone(), request, limits, None) + }); + let host = self.host.clone(); + let binding = self.binding.clone(); + let history = self.history.clone(); + async move { + let options = binding.options(&history).map_err(Arc::new)?; + host.controller + .lookup_receipt(invocation.map_err(Arc::new)?, options) + .await + .map_err(Arc::new) + } + } +} diff --git a/crates/cellar/durability/raft/cell_groups/host/storage.rs b/crates/cellar/durability/raft/cell_groups/host/storage.rs new file mode 100644 index 000000000..6ba4b9962 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/host/storage.rs @@ -0,0 +1,79 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Immutable namespace resources; private paths bind logical history, not placement. +use super::*; +use crate::application::consensus::RaftImageStorage; +use std::path::{Component, Path}; + +/// Namespace-level SQL/image resources. Creating a handle performs no I/O and +/// opens no SQLite connection. Each invocation derives a private history path. +#[derive(Clone)] +pub struct CellGroupNamespaceStorage { + options: CellSqlOptions, + bytes: usize, +} + +impl CellGroupNamespaceStorage { + pub fn new(mut options: CellSqlOptions) -> Result { + options + .limits() + .validate() + .map_err(CellDispatchError::Execution)?; + let (path, vfs) = match &mut options { + CellSqlOptions::Materialized(o) => (&mut o.directory, &mut o.sqlite_vfs), + CellSqlOptions::Extents(o) => (&mut o.cache_directory, &mut o.sqlite_vfs), + }; + validate_path(path)?; + if vfs.as_ref().is_some_and(|name| name.len() > 128) { + return Err(CellDispatchError::Invalid("namespace SQL VFS bounds")); + } + let bytes = 8192 + 2 * (path.as_os_str().len() + vfs.as_ref().map_or(0, String::len)); + *path = std::mem::take(path).into_boxed_path().into_path_buf(); + *vfs = vfs.take().map(|v| v.into_boxed_str().into_string()); + Ok(Self { options, bytes }) + } + + pub(crate) fn validate_namespace( + &self, + config: &crate::raft::catalog::NamespaceConfig, + ) -> Result<(), CellDispatchError> { + let (kind, schema) = match &self.options { + CellSqlOptions::Materialized(o) => { + (RaftImageStorage::Materialized, o.application_digest) + } + CellSqlOptions::Extents(o) => (RaftImageStorage::Extents, o.application_digest), + }; + if config.image_storage != kind || config.schema_digest != schema { + return Err(CellDispatchError::Invalid( + "namespace SQL image/schema binding", + )); + } + Ok(()) + } + + pub(super) fn bytes(&self) -> usize { + self.bytes + } + + pub(crate) fn for_cell(&self, history: &CellHistory) -> CellSqlOptions { + let key = hex::encode(super::super::cell_image_history_digest(history)); + let mut options = self.options.clone(); + match &mut options { + CellSqlOptions::Materialized(o) => o.directory.push(&key), + CellSqlOptions::Extents(o) => o.cache_directory.push(&key), + } + options + } +} + +fn validate_path(path: &Path) -> Result<(), CellDispatchError> { + // Reserve a separator plus the full 64-byte history digest under the + // existing 4096-byte invocation option bound. Never use cell IDs as paths. + if !path.is_absolute() + || path.as_os_str().len() > 4000 + || path.components().any(|p| matches!(p, Component::ParentDir)) + { + return Err(CellDispatchError::Invalid("namespace SQL directory")); + } + Ok(()) +} diff --git a/crates/cellar/durability/raft/cell_groups/image_pin.rs b/crates/cellar/durability/raft/cell_groups/image_pin.rs new file mode 100644 index 000000000..c7c94cd4d --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/image_pin.rs @@ -0,0 +1,79 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Physical image proofs used while preparing complete group roots. + +use super::{CellManifestLeaf, ManifestError}; +use cellar_logic::cell_group::{CellHistory, CellImageCommitment}; +use cellar_ltx::{generation::GenerationPin, internal::SealedImageRoot}; +use sha2::{Digest, Sha256}; +use std::sync::Arc; + +#[derive(Clone)] +pub enum CellImagePin { + Materialized(GenerationPin), + Extents(Arc), + /// Only the canonical zero-length image; never a missing-image fallback. + Empty, +} + +/// Stable across group transfer, distinct from a physical group birth digest. +pub fn cell_image_history_digest(history: &CellHistory) -> [u8; 32] { + let mut hash = Sha256::new(); + hash.update(b"cellar-cell-image-history-1\0"); + hash.update(history.cluster.bytes()); + hash.update(history.namespace.bytes()); + hash.update(history.incarnation.bytes()); + hash.update((history.cell.as_str().len() as u32).to_be_bytes()); + hash.update(history.cell.as_str().as_bytes()); + hash.finalize().into() +} + +impl CellImagePin { + pub(super) fn check( + &self, + leaf: &CellManifestLeaf, + schema: [u8; 32], + ) -> Result<(), ManifestError> { + let image = leaf.version.image; + let valid = match self { + Self::Materialized(pin) => { + let state = pin.state(); + image.commitment == CellImageCommitment::Materialized(state.digest) + && image.epoch == state.epoch + && image.txid == state.txid.0 + && image.checksum == state.checksum + && leaf.geometry.length == state.length + && leaf.geometry.page_size == state.page_size + } + Self::Extents(pin) => { + let manifest = pin.manifest(); + let binding = manifest.binding(); + let reference = pin.reference(); + binding.history == cell_image_history_digest(&leaf.version.history) + && binding.schema == schema + && binding.epoch == image.epoch + && binding.txid == image.txid + && binding.checksum == image.checksum + && manifest.geometry().length() == leaf.geometry.length + && manifest.geometry().page_size() == leaf.geometry.page_size + && image.commitment + == CellImageCommitment::Extents { + root: reference.root().bytes(), + manifest_bytes: reference.encoded_length(), + } + } + Self::Empty => { + image.txid == 0 + && image.checksum == 0 + && leaf.geometry.length == 0 + && image.commitment + == CellImageCommitment::Materialized(Sha256::digest([]).into()) + } + }; + if valid { + Ok(()) + } else { + Err(ManifestError::Invalid("cell image seal")) + } + } +} diff --git a/crates/cellar/durability/raft/cell_groups/log_admission.rs b/crates/cellar/durability/raft/cell_groups/log_admission.rs new file mode 100644 index 000000000..20e73ff46 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/log_admission.rs @@ -0,0 +1,220 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Reserve before RawNode can append; retain exact entries through persistence. +use super::log_retention::LogReservation; +use crate::raft::{ + cache::{ConsensusStorage, GroupLog}, + completion::{entry_digest, RuntimeError}, +}; +use ::raft::{prelude::*, RawNode, StateRole}; +use cellar_logic::cell_group::GroupHistory; +use protobuf::Message as _; +use std::{collections::BTreeMap, sync::Arc}; + +#[derive(Default)] +pub(super) struct LogAdmission { + entries: BTreeMap)>, + automatic_leave: Option>, +} + +#[derive(Default)] +pub(super) struct Prospective { + entries: BTreeMap)>, + control: Option>, + automatic_leave: Option>, +} + +fn retention( + group: &GroupLog, +) -> Result<&Arc, RuntimeError> { + group + .log_retention() + .ok_or_else(|| RuntimeError::Protocol("missing group log admission".into())) +} + +impl LogAdmission { + pub(super) fn prepare( + &self, + group: &GroupLog, + entries: &[Entry], + supplied: Option<&Arc>, + ) -> Result { + let mut next = Prospective::default(); + if supplied.is_some() && entries.len() != 1 { + return Err(RuntimeError::Invalid( + "prepared log reservation shape".into(), + )); + } + let cache = group.read()?; + for entry in entries { + let digest = entry_digest(entry)?; + if self + .entries + .get(&entry.index) + .is_some_and(|(old, _)| old == &digest) + { + continue; + } + if cache.entries.get(&entry.index) == Some(entry) { + continue; + } + let lease = if let Some(lease) = supplied { + lease.bind(entry)?; + lease.validate(retention(group)?, entry)?; + lease.clone() + } else { + retention(group)? + .reserve_entry(entry) + .map_err(admission_error)? + }; + next.entries.insert(entry.index, (digest, lease)); + } + Ok(next) + } + + pub(super) fn election( + &self, + group: &GroupLog, + node: &RawNode, + mut next: Prospective, + ) -> Result { + next.control = Some( + retention(group)? + .reserve(1024, true) + .map_err(admission_error)?, + ); + if self.automatic_leave.is_none() && needs_automatic_leave(group, node)? { + next.automatic_leave = Some( + retention(group)? + .reserve(1024, true) + .map_err(admission_error)?, + ); + } + Ok(next) + } + + pub(super) fn configuration( + &self, + group: &GroupLog, + mut next: Prospective, + auto_leave: bool, + ) -> Result { + if auto_leave && self.automatic_leave.is_none() { + next.automatic_leave = Some( + retention(group)? + .reserve(1024, true) + .map_err(admission_error)?, + ); + } + Ok(next) + } + + /// A promised automatic entry must never seek admission after collected + /// apply work has already selected its new durable configuration. + pub(super) fn applying(&mut self, automatic: bool) -> Result { + let mut next = Prospective::default(); + if automatic { + next.control = Some(self.automatic_leave.take().ok_or_else(|| { + RuntimeError::Protocol("unreserved automatic joint leave".into()) + })?); + } + Ok(next) + } + + pub(super) fn finish( + &mut self, + group: &GroupLog, + node: &RawNode, + mut next: Prospective, + ) -> Result<(), RuntimeError> { + let mut entries = BTreeMap::new(); + let cache = group.read()?; + for entry in node.raft.raft_log.unstable_entries() { + let digest = entry_digest(entry)?; + if cache.entries.get(&entry.index) == Some(entry) { + continue; + } + let old = self + .entries + .get(&entry.index) + .filter(|(old, _)| old == &digest); + let fresh = next + .entries + .remove(&entry.index) + .filter(|(new, _)| new == &digest); + let lease = if let Some((_, lease)) = old { + lease.clone() + } else if let Some((_, lease)) = fresh { + lease + } else if entry.data.is_empty() && entry.context.is_empty() { + let lease = next.control.take().ok_or_else(|| { + RuntimeError::Protocol("unreserved Raft control entry".into()) + })?; + lease.bind(entry)?; + lease + } else { + return Err(RuntimeError::Protocol("unreserved Raft entry".into())); + }; + entries.insert(entry.index, (digest, lease)); + } + drop(cache); + self.entries = entries; + if node.raft.state == StateRole::Leader && needs_automatic_leave(group, node)? { + if let Some(lease) = next.automatic_leave { + self.automatic_leave = Some(lease); + } + } else { + self.automatic_leave = None; + } + Ok(()) + } + + pub(super) fn persistence( + &self, + entries: &[Entry], + ) -> Result>, RuntimeError> { + let mut leases = BTreeMap::new(); + for entry in entries { + if let Some((digest, lease)) = self.entries.get(&entry.index) { + if *digest != entry_digest(entry)? { + return Err(RuntimeError::Protocol("changed reserved entry".into())); + } + leases.insert(entry.index, lease.clone()); + } + } + Ok(leases) + } +} + +fn needs_automatic_leave( + group: &GroupLog, + node: &RawNode, +) -> Result { + if *node.raft.prs().conf().auto_leave() { + return Ok(true); + } + let cache = group.read()?; + for entry in cache + .entries + .values() + .chain(node.raft.raft_log.unstable_entries()) + { + if entry.index > node.raft.raft_log.applied + && entry.get_entry_type() == EntryType::EntryConfChangeV2 + { + let change = ConfChangeV2::parse_from_bytes(&entry.data) + .map_err(|e| RuntimeError::Invalid(e.to_string()))?; + if change.enter_joint() == Some(true) { + return Ok(true); + } + } + } + Ok(false) +} + +fn admission_error(error: crate::raft::storage::LogError) -> RuntimeError { + match error { + crate::raft::storage::LogError::Overloaded => RuntimeError::Overloaded, + other => RuntimeError::Storage(other), + } +} diff --git a/crates/cellar/durability/raft/cell_groups/log_retention.rs b/crates/cellar/durability/raft/cell_groups/log_retention.rs new file mode 100644 index 000000000..e0f1f0789 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/log_retention.rs @@ -0,0 +1,563 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Engine-local retained entry debt and owned reservations. Remote archive +//! observations never release this accounting; only selected storage state can. +use crate::raft::storage::LogError; +use ::raft::prelude::{Entry, EntryType}; +use protobuf::Message as _; +use sha2::{Digest, Sha256}; +use std::collections::{BTreeMap, BTreeSet}; +use std::sync::{ + atomic::{AtomicBool, Ordering}, + Arc, Mutex, MutexGuard, +}; + +// Includes the cached entry's fixed metadata and an outstanding lease/map node. +const ENTRY_OVERHEAD: usize = 512; + +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub struct CellGroupLogCapacity { + pub bytes: usize, + pub entries: usize, +} + +impl CellGroupLogCapacity { + fn add(self, other: Self) -> Result { + Ok(Self { + bytes: self + .bytes + .checked_add(other.bytes) + .ok_or(LogError::Overloaded)?, + entries: self + .entries + .checked_add(other.entries) + .ok_or(LogError::Overloaded)?, + }) + } + + fn sub(self, other: Self) -> Result { + Ok(Self { + bytes: self + .bytes + .checked_sub(other.bytes) + .ok_or_else(accounting_error)?, + entries: self + .entries + .checked_sub(other.entries) + .ok_or_else(accounting_error)?, + }) + } + + fn fits(self, limit: Self) -> bool { + self.bytes <= limit.bytes && self.entries <= limit.entries + } +} + +/// Hard node/group entry admission, including separately reserved capacity for +/// Raft's empty election and automatic joint-leave entries. This bounds logical +/// retained entries and reservations, not physical engine file amplification. +#[derive(Clone, Copy, Debug)] +pub struct CellGroupLogRetentionLimits { + pub node: CellGroupLogCapacity, + pub group: CellGroupLogCapacity, + pub progress_node: CellGroupLogCapacity, + pub progress_group: CellGroupLogCapacity, +} + +impl Default for CellGroupLogRetentionLimits { + fn default() -> Self { + Self { + node: CellGroupLogCapacity { + bytes: 192 * 1024 * 1024, + entries: 32_768, + }, + group: CellGroupLogCapacity { + bytes: 64 * 1024 * 1024, + entries: 8192, + }, + progress_node: CellGroupLogCapacity { + bytes: 2 * 1024 * 1024, + entries: 1024, + }, + progress_group: CellGroupLogCapacity { + bytes: 128 * 1024, + entries: 64, + }, + } + } +} + +impl CellGroupLogRetentionLimits { + pub(crate) fn validate(self) -> Result<(), LogError> { + if self.node.bytes > 2 * 1024 * 1024 * 1024 + || self.node.entries > 1_048_576 + || !self.group.fits(self.node) + || !self.progress_group.fits(self.progress_node) + || [ + self.node, + self.group, + self.progress_node, + self.progress_group, + ] + .iter() + .any(|x| x.bytes == 0 || x.entries == 0) + || self.progress_node.bytes >= self.node.bytes + || self.progress_node.entries >= self.node.entries + || self.progress_group.bytes >= self.group.bytes + || self.progress_group.entries >= self.group.entries + { + return Err(LogError::Invalid("cell-group log retention limits".into())); + } + Ok(()) + } +} + +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub struct CellGroupLogRetentionUsage { + pub retained_bytes: usize, + pub retained_entries: usize, + pub reserved_bytes: usize, + pub reserved_entries: usize, +} + +#[derive(Clone, Copy, Debug, Default)] +struct Charge { + data: CellGroupLogCapacity, + progress: CellGroupLogCapacity, +} + +impl Charge { + fn entry(entry: &Entry) -> Result { + let capacity = CellGroupLogCapacity { + bytes: (entry.compute_size() as usize) + .checked_add(ENTRY_OVERHEAD) + .ok_or(LogError::Overloaded)?, + entries: 1, + }; + Ok(Self::new(capacity, progress_entry(entry))) + } + + fn new(capacity: CellGroupLogCapacity, progress: bool) -> Self { + if progress { + Self { + progress: capacity, + ..Self::default() + } + } else { + Self { + data: capacity, + ..Self::default() + } + } + } + + fn add(self, other: Self) -> Result { + Ok(Self { + data: self.data.add(other.data)?, + progress: self.progress.add(other.progress)?, + }) + } + + fn sub(self, other: Self) -> Result { + Ok(Self { + data: self.data.sub(other.data)?, + progress: self.progress.sub(other.progress)?, + }) + } + + fn fits( + self, + limit: CellGroupLogCapacity, + progress: CellGroupLogCapacity, + ) -> Result { + Ok(self.data.fits(limit.sub(progress)?) + && self.progress.fits(progress) + && self.data.add(self.progress)?.fits(limit)) + } + + fn total(self) -> Result { + self.data.add(self.progress) + } +} + +fn accounting_error() -> LogError { + LogError::Corrupt("cell-group retained log accounting".into()) +} + +fn progress_entry(entry: &Entry) -> bool { + entry.data.is_empty() + && entry.context.is_empty() + && matches!( + entry.get_entry_type(), + EntryType::EntryNormal | EntryType::EntryConfChangeV2 + ) +} + +fn binding(entry: &Entry) -> Result<[u8; 32], LogError> { + Ok(Sha256::digest( + entry + .write_to_bytes() + .map_err(|e| LogError::Invalid(e.to_string()))?, + ) + .into()) +} + +fn retained<'a>(mut entries: impl Iterator) -> Result { + entries.try_fold(Charge::default(), |sum, entry| { + sum.add(Charge::entry(entry)?) + }) +} + +#[derive(Default)] +struct Group { + durable: Charge, + usage: Charge, +} + +struct Hold { + group: u64, + charge: Charge, + binding: Option<[u8; 32]>, +} + +#[derive(Default)] +struct State { + groups: BTreeMap, + holds: BTreeMap, + next: u64, + durable: Charge, + usage: Charge, +} + +pub(crate) struct NodeLogRetention { + state: Mutex, + limits: CellGroupLogRetentionLimits, + closed: Arc, +} + +pub(crate) struct GroupLogRetention { + node: Arc, + key: u64, +} + +/// Clones share one charge. Storage transfers it into actual durable debt in +/// the same accounting turn; losing the waiter cannot refund entered work. +pub(crate) struct LogReservation { + group: Arc, + id: u64, +} + +impl NodeLogRetention { + pub(crate) fn new(limits: CellGroupLogRetentionLimits, closed: Arc) -> Arc { + Arc::new(Self { + state: Mutex::new(State::default()), + limits, + closed, + }) + } + + fn lock(&self) -> Result, LogError> { + if self.closed.load(Ordering::Acquire) { + return Err(LogError::Closed); + } + self.state.lock().map_err(|_| { + self.closed.store(true, Ordering::Release); + LogError::Closed + }) + } + + /// Reconstruct debt from actual selected entries, including when reopening + /// under a smaller ceiling. New reservations refuse until the debt fits. + pub(crate) fn register<'a>( + self: &Arc, + key: u64, + entries: impl Iterator, + ) -> Result, LogError> { + let durable = retained(entries)?; + let mut state = self.lock()?; + if state.groups.contains_key(&key) { + return Err(LogError::WrongIdentity); + } + let usage = state.usage.add(durable)?; + let all_durable = state.durable.add(durable)?; + state.groups.insert( + key, + Group { + durable, + usage: durable, + }, + ); + state.usage = usage; + state.durable = all_durable; + Ok(Arc::new(GroupLogRetention { + node: self.clone(), + key, + })) + } + + pub(crate) fn usage(&self) -> Result { + let state = self.lock()?; + usage(state.durable, state.usage) + } +} + +fn usage(durable: Charge, all: Charge) -> Result { + let retained = durable.total()?; + let reserved = all.sub(durable)?.total()?; + Ok(CellGroupLogRetentionUsage { + retained_bytes: retained.bytes, + retained_entries: retained.entries, + reserved_bytes: reserved.bytes, + reserved_entries: reserved.entries, + }) +} + +impl GroupLogRetention { + pub(crate) fn usage(&self) -> Result { + let state = self.node.lock()?; + let group = state.groups.get(&self.key).ok_or(LogError::WrongIdentity)?; + usage(group.durable, group.usage) + } + + pub(crate) fn reserve( + self: &Arc, + bytes: usize, + progress: bool, + ) -> Result, LogError> { + if bytes < ENTRY_OVERHEAD { + return Err(LogError::Invalid("entry reservation overhead".into())); + } + let charge = Charge::new(CellGroupLogCapacity { bytes, entries: 1 }, progress); + let mut state = self.node.lock()?; + let group = state.groups.get(&self.key).ok_or(LogError::WrongIdentity)?; + let next_group = group.usage.add(charge)?; + let next_node = state.usage.add(charge)?; + let limits = self.node.limits; + if !next_group.fits(limits.group, limits.progress_group)? + || !next_node.fits(limits.node, limits.progress_node)? + { + return Err(LogError::Overloaded); + } + let id = state.next.checked_add(1).ok_or(LogError::Overloaded)?; + state.next = id; + state.holds.insert( + id, + Hold { + group: self.key, + charge, + binding: None, + }, + ); + state + .groups + .get_mut(&self.key) + .ok_or(LogError::WrongIdentity)? + .usage = next_group; + state.usage = next_node; + Ok(Arc::new(LogReservation { + group: self.clone(), + id, + })) + } + + pub(crate) fn reserve_execution( + self: &Arc, + envelope: crate::application::consensus::EnvelopeLimits, + ) -> Result, LogError> { + let bytes = envelope + .max_header_bytes + .checked_add(envelope.max_payload_bytes) + .and_then(|n| n.checked_add(1024)) + .ok_or(LogError::Overloaded)?; + self.reserve(bytes, false) + } + + pub(crate) fn reserve_entry( + self: &Arc, + entry: &Entry, + ) -> Result, LogError> { + let lease = self.reserve(Charge::entry(entry)?.total()?.bytes, progress_entry(entry))?; + lease.bind(entry)?; + Ok(lease) + } + + /// Call only after the exact cache/root state is durably selected. Every new + /// entry must have an entered lease; accounting can only stay equal or fall. + pub(crate) fn publish<'a>( + self: &Arc, + entries: impl Iterator, + leases: &[Arc], + ) -> Result<(), LogError> { + let next = retained(entries)?; + let mut state = self.node.lock()?; + let old = state + .groups + .get(&self.key) + .ok_or(LogError::WrongIdentity)? + .durable; + let mut consumed = Charge::default(); + let mut ids = BTreeSet::new(); + for lease in leases { + if !Arc::ptr_eq(self, &lease.group) || !ids.insert(lease.id) { + return Err(LogError::WrongIdentity); + } + let hold = state.holds.get(&lease.id).ok_or(LogError::WrongIdentity)?; + if hold.group != self.key || hold.binding.is_none() { + return Err(LogError::WrongIdentity); + } + consumed = consumed.add(hold.charge)?; + } + let admitted = old.add(consumed)?; + if !next.data.fits(admitted.data) || !next.progress.fits(admitted.progress) { + return Err(accounting_error()); + } + let next_group = state + .groups + .get(&self.key) + .ok_or(LogError::WrongIdentity)? + .usage + .sub(admitted)? + .add(next)?; + let next_node = state.usage.sub(admitted)?.add(next)?; + let next_durable = state.durable.sub(old)?.add(next)?; + for id in ids { + state.holds.remove(&id); + } + state.groups.insert( + self.key, + Group { + durable: next, + usage: next_group, + }, + ); + state.usage = next_node; + state.durable = next_durable; + Ok(()) + } +} + +impl LogReservation { + /// Shrink the pre-callback envelope to the actual encoded proposal, retaining + /// enough space for its later Raft term/index and fixed entry bookkeeping. + pub(crate) fn encoded(&self, bytes: usize) -> Result<(), LogError> { + let charge = Charge::new( + CellGroupLogCapacity { + bytes: bytes + .checked_add(64 + ENTRY_OVERHEAD) + .ok_or(LogError::Overloaded)?, + entries: 1, + }, + false, + ); + let mut state = self.group.node.lock()?; + let hold = state.holds.get(&self.id).ok_or(LogError::WrongIdentity)?; + if hold.binding.is_some() + || !charge.data.fits(hold.charge.data) + || !charge.progress.fits(hold.charge.progress) + { + return Err(LogError::WrongIdentity); + } + let released = hold.charge.sub(charge)?; + let next_group = state + .groups + .get(&self.group.key) + .ok_or(LogError::WrongIdentity)? + .usage + .sub(released)?; + state + .groups + .get_mut(&self.group.key) + .ok_or(LogError::WrongIdentity)? + .usage = next_group; + state.usage = state.usage.sub(released)?; + state + .holds + .get_mut(&self.id) + .ok_or(LogError::WrongIdentity)? + .charge = charge; + Ok(()) + } + + /// Only a sealed pre-append rejection or authoritative reconciliation may + /// release a live lease independently of its remaining client clones. + pub(crate) fn release(&self) -> Result<(), LogError> { + let mut state = self.group.node.lock()?; + if let Some(hold) = state.holds.remove(&self.id) { + let group = state + .groups + .get_mut(&self.group.key) + .ok_or(LogError::WrongIdentity)?; + group.usage = group.usage.sub(hold.charge)?; + state.usage = state.usage.sub(hold.charge)?; + } + Ok(()) + } + + pub(crate) fn bind(&self, entry: &Entry) -> Result<(), LogError> { + let digest = binding(entry)?; + let charge = Charge::entry(entry)?; + let mut state = self.group.node.lock()?; + let hold = state.holds.get(&self.id).ok_or(LogError::WrongIdentity)?; + if hold.group != self.group.key + || hold.binding.is_some_and(|old| old != digest) + || !charge.data.fits(hold.charge.data) + || !charge.progress.fits(hold.charge.progress) + { + return Err(LogError::WrongIdentity); + } + let released = hold.charge.sub(charge)?; + let group = state + .groups + .get_mut(&self.group.key) + .ok_or(LogError::WrongIdentity)?; + group.usage = group.usage.sub(released)?; + state.usage = state.usage.sub(released)?; + let hold = state + .holds + .get_mut(&self.id) + .ok_or(LogError::WrongIdentity)?; + hold.charge = charge; + hold.binding = Some(digest); + Ok(()) + } + + pub(crate) fn validate( + &self, + group: &Arc, + entry: &Entry, + ) -> Result<(), LogError> { + if !Arc::ptr_eq(&self.group, group) { + return Err(LogError::WrongIdentity); + } + let digest = binding(entry)?; + let state = self.group.node.lock()?; + let hold = state.holds.get(&self.id).ok_or(LogError::WrongIdentity)?; + if hold.group != group.key || hold.binding != Some(digest) { + return Err(LogError::WrongIdentity); + } + Ok(()) + } +} + +impl Drop for LogReservation { + fn drop(&mut self) { + // Even a closed engine may release an unselected reservation. No new + // admission is possible there; reopen reconstructs actual durable debt. + if let Ok(mut state) = self.group.node.state.lock() { + if let Some(hold) = state.holds.remove(&self.id) { + let result = (|| { + let group = state + .groups + .get_mut(&self.group.key) + .ok_or(LogError::WrongIdentity)?; + group.usage = group.usage.sub(hold.charge)?; + state.usage = state.usage.sub(hold.charge)?; + Ok::<_, LogError>(()) + })(); + if result.is_err() { + self.group.node.closed.store(true, Ordering::Release); + } + } + } + } +} diff --git a/crates/cellar/durability/raft/cell_groups/manifest.rs b/crates/cellar/durability/raft/cell_groups/manifest.rs new file mode 100644 index 000000000..7455303d1 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/manifest.rs @@ -0,0 +1,503 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Immutable authenticated AVL pages. Every cell occupies one bounded page; +//! changing a cell copies O(log cells) pages, preserving all other image roots. + +use super::page_store::ManifestPageStore; +use super::page_wire::{decode, encode, Page, PageRef}; +use crate::application::consensus::{cell_routing_key, LtxImage}; +use cellar_logic::{cell_group::*, consensus::LogPosition}; +use std::cmp::Ordering; + +#[derive(Debug)] +pub enum ManifestError { + Invalid(&'static str), + Limit, + Conflict, + Quarantined, + Io(std::io::Error), +} + +impl std::fmt::Display for ManifestError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "group manifest: {self:?}") + } +} + +impl std::error::Error for ManifestError { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + match self { + Self::Io(error) => Some(error), + _ => None, + } + } +} + +impl From for ManifestError { + fn from(error: std::io::Error) -> Self { + Self::Io(error) + } +} + +/// Geometry is logical data. Local generation names never enter a replicated +/// manifest commitment; each replica resolves and pins its own image storage. +#[derive(Clone, Copy, Debug, PartialEq, Eq, serde::Serialize, serde::Deserialize)] +#[serde(deny_unknown_fields)] +pub struct CellImageGeometry { + pub length: u64, + pub page_size: u32, +} + +#[derive(Clone, Copy, Debug)] +pub struct ManifestLimits { + pub max_cells: u32, + pub max_depth: u8, +} + +impl Default for ManifestLimits { + fn default() -> Self { + Self { + max_cells: 1_000_000, + max_depth: 32, + } + } +} + +impl ManifestLimits { + pub(crate) fn validate(self) -> Result<(), ManifestError> { + if self.max_cells == 0 + || self.max_cells > 1_000_000 + || self.max_depth == 0 + || self.max_depth > 32 + { + return Err(ManifestError::Limit); + } + Ok(()) + } +} + +/// Authenticated scheduling hints. Callbacks and delivery identity still come +/// from the selected cell image; a summary grants no execution authority. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, serde::Serialize, serde::Deserialize)] +#[serde(deny_unknown_fields)] +pub struct EffectSummary { + pub next_alarm_unix_millis: Option, + pub next_outbox_unix_millis: Option, + pub pending_outbox: u64, + pub pending_inbox: u64, +} + +impl EffectSummary { + /// Earliest durable work hint. It grants no callback or delivery authority. + pub fn next_due_unix_millis(self) -> Option { + [self.next_alarm_unix_millis, self.next_outbox_unix_millis] + .into_iter() + .flatten() + .min() + } +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct CellManifestLeaf { + /// Original creation identity, retained atomically with every selected image. + /// Mutations and ordinary group transfers must preserve this record exactly. + pub enrollment: super::EnrollmentRecord, + pub version: CellVersion, + pub geometry: CellImageGeometry, + /// Position that installed this image in the current group. + pub last_mutation: LogPosition, + /// Original decision for this logical version, retained independently of + /// log entries and of later group progress or transfer installation. + pub witness: CommitWitness, + pub effects: EffectSummary, +} + +impl CellManifestLeaf { + /// Observation data from this leaf. Only an authoritative selected root + /// establishes commitment; constructing a leaf grants no serving authority. + pub fn observation(&self) -> crate::raft::version::CellVersionObservation { + crate::raft::version::CellVersionObservation { + version: self.version.clone(), + witness: self.witness, + } + } + + pub(super) fn validate(&self, route: &GroupRoute) -> Result { + if [ + self.effects.next_alarm_unix_millis, + self.effects.next_outbox_unix_millis, + ] + .into_iter() + .flatten() + .any(|due| due > i64::MAX as u64) + || (self.effects.pending_outbox == 0) != self.effects.next_outbox_unix_millis.is_none() + { + return Err(ManifestError::Invalid("effect summary")); + } + let key = cell_routing_key(route.history.namespace, &self.version.history.cell); + route + .check_cell(&self.version.history, &key) + .map_err(|_| ManifestError::Invalid("cell route"))?; + self.last_mutation + .validate() + .map_err(|_| ManifestError::Invalid("mutation position"))?; + if self.last_mutation.index.get() == 0 { + return Err(ManifestError::Invalid("zero mutation index")); + } + self.witness + .validate(&self.version.history) + .map_err(|_| ManifestError::Invalid("cell commit witness"))?; + if (self.witness.group == route.history + && (self.witness.position.index > self.last_mutation.index + || self.witness.position.term > self.last_mutation.term + || (self.witness.position.index == self.last_mutation.index + && self.witness.position.term != self.last_mutation.term))) + || (self.version.sequence.get() == 1 + && self.witness.proposal != self.enrollment.creation()) + { + return Err(ManifestError::Invalid("cell witness position/creation")); + } + // The immutable record's constructor/decoder validates its bounded shape. + // Do not allocate its canonical encoding on every tree lookup. + let initial = self.enrollment.initial(); + if self.enrollment.cell() != &self.version.history + || self.enrollment.schema() != route.schema_digest + || initial.epoch != self.version.image.epoch + || (self.version.sequence.get() == 1 && initial != self.version.image) + || (self.version.sequence.get() > 1 && initial.txid >= self.version.image.txid) + { + return Err(ManifestError::Invalid("enrollment/image binding")); + } + let image = LtxImage::try_from(self.version.image) + .map_err(|_| ManifestError::Invalid("image commitment"))?; + if !cellar_ltx::ltx::is_valid_page_size(self.geometry.page_size) + || self.geometry.length > cellar_ltx::stream_snapshot::MAX_IMAGE_BYTES + || !self + .geometry + .length + .is_multiple_of(u64::from(self.geometry.page_size)) + || (image.txid == 0) != (self.geometry.length == 0) + || (image.txid == 0 && image.checksum != 0) + { + return Err(ManifestError::Invalid("image geometry")); + } + if let CellImageCommitment::Extents { manifest_bytes, .. } = self.version.image.commitment { + let expected = 112 + + 32 * self + .geometry + .length + .div_ceil(cellar_ltx::internal::EXTENT_BYTES); + if self.geometry.length == 0 || u64::from(manifest_bytes) != expected { + return Err(ManifestError::Invalid("extent geometry")); + } + } + Ok(key) + } +} + +/// An untrusted tree descriptor until checked against an authoritative group +/// root and traversed by `verify`. It is never an applied marker or image seal. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub struct ManifestRoot(pub(super) Option); + +impl ManifestRoot { + pub fn cells(self) -> u32 { + self.0.map_or(0, |p| p.cells) + } + + pub fn depth(self) -> u8 { + self.0.map_or(0, |p| p.height) + } + + /// Authenticated subtree hint after complete-root verification. No SQLite + /// database needs to be opened to decide whether this root has due work. + pub fn next_due_unix_millis(self) -> Option { + self.0.and_then(|p| p.next_due_unix_millis) + } + + pub fn digest(self) -> Option<[u8; 32]> { + self.0.map(|p| p.digest) + } + + /// Maximum physical installation position across the authenticated subtree. + /// Metadata-only apply leaves this unchanged. Complete-root verification + /// checks it against the actual pages and each cell's original mutation. + pub fn last_mutation(self) -> Option { + self.0.map(|p| LogPosition { + index: cellar_logic::consensus::LogIndex::new(p.last_mutation_index), + term: cellar_logic::consensus::Term::new(p.last_mutation_term), + }) + } +} + +/// Synchronous operations belong on the owning blocking execution domain. +/// Retain every root used by apply/read/snapshot/transfer before reclamation. +/// This layer performs no page deletion and cannot publish serving authority. +pub struct ManifestTree { + pub(super) route: GroupRoute, + limits: ManifestLimits, +} + +impl ManifestTree { + pub(crate) fn route(&self) -> &GroupRoute { + &self.route + } + + pub fn new(route: GroupRoute, limits: ManifestLimits) -> Result { + limits.validate()?; + for bound in [route.range.lower(), route.range.upper()] { + if let CellBound::Key(key) = bound { + if &cell_routing_key(route.history.namespace, key.cell()) != key { + return Err(ManifestError::Invalid("range hash")); + } + } + } + Ok(Self { route, limits }) + } + + /// Select one due cell after the cursor in routing-key order, wrapping once. + /// Run on the owning blocking domain while retaining the verified selected + /// root. Reads at most three tree depths of pages, and opens no SQLite DB. + /// The returned leaf is advisory: execution still needs fresh leader, route, + /// exact-image and private-command admission checks. + pub fn next_due( + &self, + store: &ManifestPageStore, + root: ManifestRoot, + now_unix_millis: u64, + after: Option<&CellId>, + ) -> Result, ManifestError> { + super::effect_index::next(self, store, root, now_unix_millis, after, self.limits) + } + + pub fn get( + &self, + store: &ManifestPageStore, + root: ManifestRoot, + cell: &CellId, + ) -> Result, ManifestError> { + let key = cell_routing_key(self.route.history.namespace, cell); + let mut cursor = root.0; + let mut remaining = self.limits.max_depth; + while let Some(reference) = cursor { + if remaining == 0 { + return Err(ManifestError::Limit); + } + remaining -= 1; + let page = self.read(store, reference)?; + match key.cmp(&page.leaf.validate(&self.route)?) { + Ordering::Less => cursor = page.left, + Ordering::Greater => cursor = page.right, + Ordering::Equal => return Ok(Some(page.leaf)), + } + } + Ok(None) + } + + /// Compare-and-replace one logical cell. `None` requires absence; transfer + /// installation may insert a retained sequence. Enrollment authorization, + /// image sealing and the group's contiguous apply proof belong to the caller. + pub fn update( + &self, + store: &mut ManifestPageStore, + root: ManifestRoot, + leaf: CellManifestLeaf, + expected: Option<&CellVersion>, + ) -> Result { + let key = leaf.validate(&self.route)?; + let current = self.get(store, root, &leaf.version.history.cell)?; + match (¤t, expected) { + (None, None) => { + if root.cells() >= self.limits.max_cells { + return Err(ManifestError::Limit); + } + } + (Some(old), Some(expected)) if &old.version == expected => { + if old.enrollment != leaf.enrollment + || old.version.history != leaf.version.history + || old.version.sequence.checked_next().ok() != Some(leaf.version.sequence) + || leaf.version.image.epoch != old.version.image.epoch + || leaf.version.image.txid <= old.version.image.txid + || leaf.last_mutation.index <= old.last_mutation.index + || leaf.last_mutation.term < old.last_mutation.term + { + return Err(ManifestError::Conflict); + } + } + _ => return Err(ManifestError::Conflict), + } + self.insert(store, root.0, leaf, &key, 0) + .map(|p| ManifestRoot(Some(p))) + } + + /// Walk the entire authenticated closure before recovery admission. The + /// callback must independently verify/pin each actual image and receipt state. + /// No success or partial callback traversal is itself a quorum proof. + pub fn verify( + &self, + store: &ManifestPageStore, + root: ManifestRoot, + applied: LogPosition, + mut image: impl FnMut(&CellManifestLeaf) -> Result<(), ManifestError>, + ) -> Result<(), ManifestError> { + applied + .validate() + .map_err(|_| ManifestError::Invalid("applied position"))?; + let mut visited = 0; + self.walk( + store, + root.0, + (None, None), + applied, + &mut visited, + &mut image, + )?; + if visited != root.cells() { + return Err(ManifestError::Invalid("cell count")); + } + Ok(()) + } + + fn walk( + &self, + store: &ManifestPageStore, + reference: Option, + bounds: (Option<&CellKey>, Option<&CellKey>), + applied: LogPosition, + visited: &mut u32, + image: &mut impl FnMut(&CellManifestLeaf) -> Result<(), ManifestError>, + ) -> Result<(), ManifestError> { + let (lower, upper) = bounds; + let Some(reference) = reference else { + return Ok(()); + }; + *visited = visited.checked_add(1).ok_or(ManifestError::Limit)?; + if *visited > self.limits.max_cells { + return Err(ManifestError::Limit); + } + let page = self.read(store, reference)?; + let key = page.leaf.validate(&self.route)?; + if lower.is_some_and(|l| key <= *l) + || upper.is_some_and(|u| key >= *u) + || page.leaf.last_mutation.index > applied.index + || page.leaf.last_mutation.term > applied.term + || (page.leaf.last_mutation.index == applied.index + && page.leaf.last_mutation.term != applied.term) + { + return Err(ManifestError::Invalid("tree order or apply cut")); + } + self.walk( + store, + page.left, + (lower, Some(&key)), + applied, + visited, + image, + )?; + image(&page.leaf)?; + self.walk( + store, + page.right, + (Some(&key), upper), + applied, + visited, + image, + ) + } + + pub(super) fn read( + &self, + store: &ManifestPageStore, + reference: PageRef, + ) -> Result { + let bytes = store.read(reference.digest)?; + let page = decode(&bytes, &self.route, self.limits)?; + if page.reference(&bytes, self.limits)? != reference { + return Err(ManifestError::Invalid("page reference")); + } + Ok(page) + } + + fn save(&self, store: &mut ManifestPageStore, page: Page) -> Result { + let bytes = encode(&page, &self.route, self.limits)?; + let reference = page.reference(&bytes, self.limits)?; + store.put(reference.digest, &bytes)?; + Ok(reference) + } + + fn insert( + &self, + store: &mut ManifestPageStore, + root: Option, + leaf: CellManifestLeaf, + key: &CellKey, + depth: u8, + ) -> Result { + if depth >= self.limits.max_depth { + return Err(ManifestError::Limit); + } + let Some(reference) = root else { + return self.save( + store, + Page { + leaf, + left: None, + right: None, + }, + ); + }; + let mut page = self.read(store, reference)?; + match key.cmp(&page.leaf.validate(&self.route)?) { + Ordering::Less => { + page.left = Some(self.insert(store, page.left, leaf, key, depth + 1)?) + } + Ordering::Greater => { + page.right = Some(self.insert(store, page.right, leaf, key, depth + 1)?) + } + Ordering::Equal => page.leaf = leaf, + } + self.balance(store, page) + } + + fn balance( + &self, + store: &mut ManifestPageStore, + mut page: Page, + ) -> Result { + let height = |r: Option| r.map_or(0, |p| i16::from(p.height)); + let balance = height(page.left) - height(page.right); + if balance > 1 { + let mut left = + self.read(store, page.left.ok_or(ManifestError::Invalid("rotation"))?)?; + if height(left.left) < height(left.right) { + let mut middle = + self.read(store, left.right.ok_or(ManifestError::Invalid("rotation"))?)?; + left.right = middle.left; + middle.left = Some(self.save(store, left)?); + page.left = middle.right; + middle.right = Some(self.save(store, page)?); + return self.save(store, middle); + } + page.left = left.right; + left.right = Some(self.save(store, page)?); + return self.save(store, left); + } + if balance < -1 { + let mut right = + self.read(store, page.right.ok_or(ManifestError::Invalid("rotation"))?)?; + if height(right.right) < height(right.left) { + let mut middle = + self.read(store, right.left.ok_or(ManifestError::Invalid("rotation"))?)?; + right.left = middle.right; + middle.right = Some(self.save(store, right)?); + page.right = middle.left; + middle.left = Some(self.save(store, page)?); + return self.save(store, middle); + } + page.right = right.left; + right.left = Some(self.save(store, page)?); + return self.save(store, right); + } + self.save(store, page) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/membership.rs b/crates/cellar/durability/raft/cell_groups/membership.rs new file mode 100644 index 000000000..2366c8e10 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/membership.rs @@ -0,0 +1,324 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Group-bound configuration entries and deterministic, authority-free replay. +//! Raft progress changes only after the complete configuration/root write syncs. +use super::GroupManifest; +use crate::raft::{ + member_state::{GroupMembership, JoiningMember}, + storage::LogError, +}; +use cellar_logic::{cell_group::GroupRoute, consensus::LogIndex}; +use protobuf::Message as _; +use raft::{ + prelude::{ConfChangeV2, ConfState, Entry, EntryType}, + restore_configuration, Changer, ProgressTracker, +}; +use serde::{Deserialize, Serialize}; +use sha2::{Digest, Sha256}; + +pub(crate) const MAX_CHANGE_BYTES: usize = 40 * 1024; +const MAX_CONTEXT_BYTES: usize = 36 * 1024; + +fn invalid() -> LogError { + LogError::Invalid("cell-group configuration binding or transition".into()) +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Intent { + domain: String, + route: [u8; 32], + intent: String, +} + +fn binding(route: &GroupRoute) -> Result<[u8; 32], LogError> { + // Canonical marker-1 genesis projects the complete route, including range, + // epoch, birth, schema and all history IDs, without a mutable applied cut. + let genesis = GroupManifest::genesis(route.clone()).map_err(|_| invalid())?; + Ok(Sha256::digest(serde_json::to_vec(&genesis).map_err(|_| invalid())?).into()) +} + +/// An explicit proposal bound to one route and observed membership revision. +/// Losing the admission reply requires inspecting committed membership; callers +/// must not manufacture a new revision to blindly repeat an ambiguous change. +#[derive(Clone, Debug)] +pub struct CellGroupMembershipChange { + pub(super) change: ConfChangeV2, +} + +impl CellGroupMembershipChange { + fn wrap(route: &GroupRoute, mut change: ConfChangeV2) -> Result { + let intent = Intent { + domain: "cellar-cell-group-membership-1".into(), + route: binding(route)?, + intent: String::from_utf8(change.context.to_vec()).map_err(|_| invalid())?, + }; + change.context = serde_json::to_vec(&intent).map_err(|_| invalid())?.into(); + validate(&change, route)?; + Ok(Self { change }) + } + + pub fn new( + route: &GroupRoute, + membership: &GroupMembership, + mut change: ConfChangeV2, + joins: &[JoiningMember], + ) -> Result { + if !change.context.is_empty() || change.changes.len() > 32 { + return Err(invalid()); + } + membership.bind_change(&mut change, joins)?; + Self::wrap(route, change) + } + + /// Enrollment requires the exact catalog capability. A serialized binding + /// is descriptive and cannot authorize an unmanaged group to join a fleet. + pub fn fleet_enrollment( + route: &GroupRoute, + membership: &GroupMembership, + conf: &ConfState, + authorization: &crate::raft::fleet_catalog::FleetGroupAuthorization, + ) -> Result { + if !authorization.matches_scope(&route.history) + || authorization.history_digests() != (route.birth_digest, route.schema_digest) + { + return Err(invalid()); + } + let mut change = ConfChangeV2::default(); + membership.bind_fleet_enrollment(&mut change, conf, authorization.binding())?; + Self::wrap(route, change) + } + + /// Consume the next catalog-selected permit without changing membership. + /// Completion must be observed after durable apply; admission is not a fence. + pub fn fleet_fence( + route: &GroupRoute, + membership: &GroupMembership, + conf: &ConfState, + authorization: &crate::raft::fleet_catalog::FleetPermitAuthorization, + ) -> Result { + if !authorization.matches_scope(&route.history) { + return Err(invalid()); + } + let mut change = ConfChangeV2::default(); + membership.bind_fleet_fence(&mut change, conf, authorization.permit())?; + Self::wrap(route, change) + } + + /// Start one whole-group learner repair under its rooted physical debt. + /// The owning RawNode additionally checks readiness and actual catch-up. + pub fn managed_repair_start( + route: &GroupRoute, + membership: &GroupMembership, + conf: &ConfState, + plan: &crate::raft::member_state::RepairPlan, + authorization: &crate::raft::fleet_catalog::FleetPermitAuthorization, + ) -> Result { + if !authorization.matches_scope(&route.history) + || authorization.target_boot().is_none_or(|boot| { + boot.node != plan.replacement.node.bytes() + || boot.disk != plan.replacement.disk.bytes() + || boot.validate().is_err() + }) + { + return Err(invalid()); + } + let mut change = ConfChangeV2::default(); + membership.bind_managed_repair_start(&mut change, conf, plan, authorization.permit())?; + Self::wrap(route, change) + } + + /// Continue the same consumed permit through joint consensus or abort. + pub fn managed_repair_step( + route: &GroupRoute, + membership: &GroupMembership, + conf: &ConfState, + id: cellar_logic::consensus::ProposalId, + step: crate::raft::member_state::RepairStep, + authorization: &crate::raft::fleet_catalog::FleetPermitAuthorization, + ) -> Result { + if !authorization.matches_scope(&route.history) { + return Err(invalid()); + } + let mut change = ConfChangeV2::default(); + membership.bind_managed_repair_step(&mut change, conf, id, step, authorization.permit())?; + Self::wrap(route, change) + } + + pub(crate) fn bytes(&self) -> usize { + self.change.compute_size() as usize + 4096 + } +} + +/// Bound repeated fields before protobuf allocation. Unknown fields and duplicate +/// singular fields refuse; each nested member change has a fixed small bound. +fn scan(bytes: &[u8]) -> Result<(), LogError> { + fn varint(bytes: &[u8], at: &mut usize) -> Result { + let mut value = 0; + for shift in (0..70).step_by(7) { + let byte = *bytes.get(*at).ok_or_else(invalid)?; + *at += 1; + if shift == 63 && byte > 1 { + return Err(invalid()); + } + value |= u64::from(byte & 127) << shift; + if byte < 128 { + return Ok(value); + } + } + Err(invalid()) + } + if bytes.len() > MAX_CHANGE_BYTES { + return Err(LogError::Overloaded); + } + let (mut at, mut count, mut transition, mut context) = (0, 0, false, false); + while at < bytes.len() { + match varint(bytes, &mut at)? { + 8 if !transition => { + transition = true; + if varint(bytes, &mut at)? > 2 { + return Err(invalid()); + } + } + tag @ (18 | 26) => { + let length = usize::try_from(varint(bytes, &mut at)?).map_err(|_| invalid())?; + if tag == 18 { + count += 1; + if count > 32 || length > 32 { + return Err(invalid()); + } + } else { + if context || length > MAX_CONTEXT_BYTES { + return Err(invalid()); + } + context = true; + } + at = at + .checked_add(length) + .filter(|end| *end <= bytes.len()) + .ok_or_else(invalid)?; + } + _ => return Err(invalid()), + } + } + Ok(()) +} + +fn validate(change: &ConfChangeV2, route: &GroupRoute) -> Result { + if change.compute_size() as usize > MAX_CHANGE_BYTES + || change.changes.len() > 32 + || change.get_unknown_fields().iter().next().is_some() + || change + .changes + .iter() + .any(|item| item.get_unknown_fields().iter().next().is_some()) + { + return Err(invalid()); + } + let mut decoded = change.clone(); + if change.context.is_empty() { + if !change.changes.is_empty() + || change.get_transition() != raft::prelude::ConfChangeTransition::Auto + { + return Err(invalid()); + } + return Ok(decoded); // Only a committed auto-leave is accepted by transition(). + } + if change.context.len() > MAX_CONTEXT_BYTES { + return Err(invalid()); + } + let intent: Intent = serde_json::from_slice(&change.context).map_err(|_| invalid())?; + if intent.domain != "cellar-cell-group-membership-1" + || intent.route != binding(route)? + || serde_json::to_vec(&intent).map_err(|_| invalid())? != change.context + { + return Err(invalid()); + } + decoded.context = intent.intent.into_bytes().into(); + Ok(decoded) +} + +pub(crate) fn decode(entry: &Entry, route: &GroupRoute) -> Result { + if entry.get_entry_type() != EntryType::EntryConfChangeV2 { + return Err(invalid()); + } + scan(&entry.data)?; + let change = ConfChangeV2::parse_from_bytes(&entry.data).map_err(|_| invalid())?; + // Reject duplicate nested fields and noncanonical encodings as well as + // the outer framing checked before allocation. All local writers use this + // same protobuf serializer, including Raft's empty automatic leave. + if change.write_to_bytes().map_err(|_| invalid())? != entry.data { + return Err(invalid()); + } + validate(&change, route) +} + +pub(crate) struct Transition { + pub conf: ConfState, + pub membership: GroupMembership, +} + +impl super::ManifestTree { + /// Prepare the unchanged cell root and new configuration revision for one + /// canonical membership entry. This proves data preparation only; the Raft + /// owner must validate its committed entry/ticket and atomically persist the + /// matching configuration and membership before acknowledging application. + pub fn prepare_membership( + &self, + pages: &super::ManifestPageStore, + previous: &super::PreparedGroupManifest, + entry: &Entry, + before: &ConfState, + membership: &GroupMembership, + ) -> Result { + if !previous.marker().is_active() { + return Err(super::ManifestError::Invalid("recovery is not activated")); + } + let mut prepared = self.prepare_unchanged( + pages, + previous, + crate::raft::completion::entry_position(entry), + )?; + if previous.marker().membership_revision() != membership.revision().get() { + return Err(super::ManifestError::Invalid( + "membership preparation revision", + )); + } + membership + .validate(before, previous.marker().position().index.get()) + .map_err(|e| super::ManifestError::Io(std::io::Error::other(e)))?; + transition(entry, self.route(), before, membership) + .map_err(|e| super::ManifestError::Io(std::io::Error::other(e)))?; + prepared.marker.membership_revision = entry.index; + prepared.marker.check()?; + Ok(prepared) + } +} + +pub(crate) fn transition( + entry: &Entry, + route: &GroupRoute, + before: &ConfState, + membership: &GroupMembership, +) -> Result { + let change = decode(entry, route)?; + if change.context.is_empty() && (before.voters_outgoing.is_empty() || !before.auto_leave) { + return Err(invalid()); + } + membership.admit_group(&change, before, &route.history)?; + let mut tracker = ProgressTracker::new(1); + restore_configuration(&mut tracker, 1, before).map_err(|_| invalid())?; + let mut changer = Changer::new(&tracker); + let (conf, progress) = if change.leave_joint() { + changer.leave_joint() + } else if let Some(auto) = change.enter_joint() { + changer.enter_joint(auto, &change.changes) + } else { + changer.simple(&change.changes) + } + .map_err(|_| invalid())?; + tracker.apply_conf(conf, progress, 1); + let conf = tracker.conf().to_conf_state(); + let membership = membership.applied(&change, before, &conf, LogIndex::new(entry.index))?; + Ok(Transition { conf, membership }) +} diff --git a/crates/cellar/durability/raft/cell_groups/membership_observation.rs b/crates/cellar/durability/raft/cell_groups/membership_observation.rs new file mode 100644 index 000000000..404a22f05 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/membership_observation.rs @@ -0,0 +1,102 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Invocation-owned safe-ReadIndex observations for whole physical groups. +//! These contain no application image and grant no mutation or release lease. +use super::GroupManifest; +use crate::raft::{budget::Permit, completion::RuntimeError, member_state::GroupMembership}; +use cellar_logic::{ + cell_group::GroupHistory, + consensus::{EngineInstanceId, LogPosition, ReplicaIdentity}, +}; +use raft::prelude::ConfState; + +// Charged to the existing node/group read banks before sending ReadIndex or +// cloning membership metadata. No admission ceiling is enlarged for this lane. +pub(super) const MEMBERSHIP_BYTES: usize = 64 * 1024; + +pub(super) struct MembershipSelection { + pub replica: ReplicaIdentity, + pub engine: EngineInstanceId, + pub root: GroupManifest, + pub barrier: LogPosition, + pub membership: GroupMembership, + pub configuration: ConfState, + pub status: super::CellGroupStatus, +} + +/// Exact durable configuration observed behind this invocation's fresh quorum +/// barrier. Retaining this snapshot supplies no current quorum, fleet eligibility, +/// complete inventory, archive coverage or permission to reclaim physical debt. +pub struct CellGroupQuorumMembership { + selected: MembershipSelection, + _node: Permit, + _group: Permit, +} + +impl CellGroupQuorumMembership { + pub(super) fn new( + selected: MembershipSelection, + node: Permit, + group: Permit, + ) -> Result { + selected.membership.validate( + &selected.configuration, + selected.root.position().index.get(), + )?; + selected + .membership + .validate_scope(&selected.replica.scope)?; + if selected.root.route().history != selected.replica.scope + || selected.root.membership_revision() != selected.membership.revision().get() + || selected.barrier.index.get() == 0 + || selected.barrier.term.get() == 0 + || selected.root.position().index < selected.barrier.index + || selected.root.position().term != selected.barrier.term + || selected.status.term != selected.barrier.term + || selected.status.applied != selected.root.position() + || selected.status.committed < selected.root.position().index + || selected.status.leader.is_none() + || selected.membership.revision() > selected.barrier.index + || !selected.membership.permits_group(&selected.replica) + { + return Err(RuntimeError::Invalid( + "quorum group membership observation".into(), + )); + } + Ok(Self { + selected, + _node: node, + _group: group, + }) + } + + pub fn replica(&self) -> &ReplicaIdentity { + &self.selected.replica + } + + pub fn engine(&self) -> EngineInstanceId { + self.selected.engine + } + + pub fn root(&self) -> &GroupManifest { + &self.selected.root + } + + pub fn barrier(&self) -> LogPosition { + self.selected.barrier + } + + pub fn membership(&self) -> &GroupMembership { + &self.selected.membership + } + + pub fn configuration(&self) -> &ConfState { + &self.selected.configuration + } + + /// Same-owner progress at selection. A follower observation establishes no + /// local leader, execution, transfer or application-serving authority. + pub fn status(&self) -> &super::CellGroupStatus { + &self.selected.status + } +} diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs new file mode 100644 index 000000000..cc708452c --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -0,0 +1,175 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Complete group-state storage. A synced page is data, not Raft authority. + +pub(crate) use super::host_sql::cell_groups::CellSqlStageMode; +pub use super::host_sql::cell_groups::{ + CellGroupPreparedApply, CellGroupSqlPool, CellSqlEnrollment, CellSqlEnrollmentResolution, + CellSqlLimits, CellSqlOptions, CellSqlOutcome, CellSqlPrepared, CellSqlQueryResult, + CellSqlReceipt, CellSqlReceiptLookup, CellSqlReconciliation, CellSqlRejection, + CellSqlRootOptions, CellSqlSubmission, CellSqlUsage, +}; + +pub use crate::raft::transport::{ + CellCommandGateway, CellQueryGateway, CellRemoteCommandResponse, CellRemoteEnrollmentResponse, + CellRemoteQueryResponse, CellRemoteReceiptResponse, +}; + +mod apply; +mod archive; +pub(crate) use archive::CellGroupArchiveAcquireOutcome; +pub(crate) use archive::ControllerExecution as ArchiveExecution; +pub(crate) mod archive_compaction; +pub use archive::{ + CellGroupArchive, CellGroupArchiveCheckpointReceiver, CellGroupArchiveCollectionLimits, + CellGroupArchiveCollectionReport, CellGroupArchiveCoverage, CellGroupArchiveCreation, + CellGroupArchiveError, CellGroupArchiveExecution, CellGroupArchiveHold, CellGroupArchiveIntent, + CellGroupArchiveLimits, CellGroupArchivePool, CellGroupArchiveReference, + CellGroupArchiveRegistry, CellGroupArchiveStore, CellGroupArchiveTailLimits, + CellGroupArchiveTailPool, CellGroupArchiveTailReference, CellGroupArchiveUpload, + CellGroupArchiveUploadLimits, CellGroupArchiveUploadPool, CellGroupArchiveUploadTicket, + CellGroupArchiveUploadedTail, CellGroupArchiveUsage, PreparedCellGroupArchiveCheckpoint, + PreparedCellGroupArchiveCreation, PreparedCellGroupArchivePublication, + VerifiedCellGroupArchiveCheckpoint, VerifiedCellGroupArchiveRecovery, + VerifiedFleetGroupArchiveCoverage, +}; +pub(crate) mod checkpoint; +pub use checkpoint::{ + CellGroupCheckpoint, CellGroupCheckpointChunk, CellGroupCheckpointError, + CellGroupCheckpointLimits, CellGroupCheckpointPart, CellGroupCheckpointPool, + CellGroupCheckpointReceiver, CellGroupCheckpointUsage, PreparedCellGroupCheckpoint, + PreparedLocalCellGroupCheckpoint, +}; +mod controller; +pub(crate) use controller::CellGroupArchiveProvisioning; +pub(crate) use controller::S3OutputProof; +pub(crate) use controller::{ArchiveSeedClient, ArchiveSeedDescriptor}; +pub use controller::{ + CellCommandAdmissionError, CellCommandCall, CellEnrollmentAdmissionError, CellEnrollmentCall, + CellGroupArchiveCall, CellGroupArchiveControlError, CellGroupArchiveRegistration, + CellGroupArchiveStatus, CellGroupCheckpointPolicy, CellGroupController, + CellGroupControllerError, CellGroupControllerLimits, CellGroupControllerUsage, + CellGroupEffectRegistration, CellGroupProvisionCall, CellGroupProvisionLimits, + CellGroupProvisionNamespace, CellGroupProvisioning, CellGroupRegistration, + CellGroupRegistrationCall, CellGroupRegistrationError, CellQueryCall, +}; +mod dispatch; +mod host; +pub use apply::{CellGroupApplier, CellGroupApplyImages}; +pub(crate) use dispatch::PendingCellCommand; +pub use dispatch::{ + CellCommandError, CellCommandInvocation, CellCommandPreparation, CellCommandRequest, + CellCommandResponse, CellCommandSettlement, CellCommandSubmission, CellDispatchError, + CellDispatchLimits, CellEnrollmentError, CellEnrollmentInvocation, CellEnrollmentRequest, + CellEnrollmentResponse, CellGroupDispatcher, CellQueryConsistency, CellQueryInvocation, + CellQueryRequest, CellQueryResponse, CellQueryRouting, CellReceiptInvocation, + CellReceiptRequest, CellReceiptResponse, CellReceiptSelection, +}; +pub use host::{ + CellGroupCellHandle, CellGroupHost, CellGroupHostLimits, CellGroupHostRegistration, + CellGroupHostUsage, CellGroupNamespaceStorage, +}; + +mod effect_index; +mod execution_authority; +pub(crate) use execution_authority::CellEffectAuthority; +pub use execution_authority::{CellExecutionAuthority, CellReconciliationAuthority}; + +mod enrollment_authority; +pub use enrollment_authority::{CellEnrollmentAuthority, CellEnrollmentState}; + +mod enrollment; +mod enrollment_apply; +mod hibernate; +pub use hibernate::GroupParkCut; +pub(crate) use hibernate::ParkPrepareRequest; +pub(crate) use hibernate::{ + LocalParkChange, LocalParkRecord, ParkAcknowledgement, MAX_PARK_NEGOTIATION_MS, +}; +mod image_pin; +mod manifest; +pub(crate) mod membership; +pub use membership::CellGroupMembershipChange; +mod log_admission; +pub(crate) mod log_retention; +mod membership_observation; +mod observation; +mod owner; +pub use membership_observation::CellGroupQuorumMembership; +pub use observation::{ + CellGroupObservation, CellGroupPartitionCell, CellGroupPartitionContext, + CellGroupPartitionObservation, CellGroupPartitionWriter, CellGroupWorkload, +}; +mod park_certificate; +pub(crate) use park_certificate::ParkCertificate; +mod page_store; +mod page_wire; +mod pins; +mod proposal_admission; +pub use log_retention::{ + CellGroupLogCapacity, CellGroupLogRetentionLimits, CellGroupLogRetentionUsage, +}; +mod read_authority; +mod recovery; +pub(crate) use recovery::StagedCellGroupPartition; +pub use recovery::{ + CellGroupRecoveryLimits, CellGroupRecoveryPool, CellGroupTransferTarget, + PreparedCellGroupRecovery, PreparedCellGroupTransfer, PreparedCellGroupTransferChild, + SeededCellGroupRecovery, +}; +pub(crate) use recovery::{PreparedRecoveryData, RecoverySeed, TransferSeed}; +pub(crate) mod reclaim; +pub use read_authority::{CellReadAuthority, CellReadConsistency}; +pub use reclaim::{CellGroupReclaimError, CellGroupReclaimLimits, CellGroupReclaimReport}; +mod seed_authority; +pub use seed_authority::{CellGroupSeedAuthority, CellGroupSeedSource}; +mod retention; +mod root; +mod runtime; + +pub use runtime::{ + CellGroupAdmissionUsage, CellGroupRuntime, CellGroupRuntimeEvent, CellGroupRuntimeEvents, + CellGroupRuntimeUsage, +}; + +pub use manifest::{ + CellImageGeometry, CellManifestLeaf, EffectSummary, ManifestError, ManifestLimits, + ManifestRoot, ManifestTree, +}; +pub use page_store::{ManifestPageStore, ManifestReclaimReport, ManifestStoreUsage}; + +pub use crate::raft::catalog::{ + CellGroupArchiveBindingError, CellGroupBirth, CellGroupBirthDisposition, CellGroupBirthRequest, + CellGroupBirthSpec, CellGroupBootstrapPermit, CellGroupDirectoryCursor, CellGroupDirectoryPage, + CellGroupDisasterFenceRequest, CellGroupRecoveryActivationAck, + CellGroupRecoveryActivationPermit, CellGroupRecoveryError, CellGroupRecoveryRequest, + CellGroupSealPermit, CellGroupSourceRetirementAck, CellGroupSourceRetirementPermit, + CellGroupTransferActivationAck, CellGroupTransferActivationPermit, CellGroupTransferArchive, + CellGroupTransitionPreparationInputs, CellGroupTransitionSource, CellRouteAttempt, + CellRouteCacheUsage, CellRouteOperation, CellRouteRequest, CellRoutingError, + CellRoutingFailure, CellRoutingLimits, CellRoutingSession, DirectoryLimits, DirectoryMutation, + DirectoryPublication, DirectoryView, FencedCellGroupRecovery, FrozenCellGroupInventory, + InstalledTransferVoter, PreparedCellGroupArchiveBinding, PreparedManagedCellGroupTransition, + RangeDirectory, ResolvedCellRoute, TransferInstallQuorum, TransitionJournal, TransitionPhase, + VerifiedCellGroupTransitionSources, +}; +pub use enrollment::{ + decide_enrollment, EnrollmentDecision, EnrollmentError, EnrollmentPhase, EnrollmentRecord, +}; +pub use enrollment_apply::{EnrollmentApply, EnrollmentApplyError, EnrollmentImage}; +pub use image_pin::{cell_image_history_digest, CellImagePin}; +pub(crate) use pins::ManifestPins; +pub use root::{ + GroupManifest, GroupSourceRetirement, GroupSourceSeal, GroupTransferOrigin, + PreparedGroupManifest, +}; + +pub(crate) use owner::SelectedParkCertificate; +pub use owner::{ + CellGroupApply, CellGroupEffect, CellGroupOwner, CellGroupPersist, CellGroupPersisted, + CellGroupProposal, CellGroupRecoveryActivation, CellGroupSealOutcome, + CellGroupSnapshotAdmission, CellGroupSourceRetirement, CellGroupStatus, CellGroupTicket, + CellGroupTransferActivation, TransferChildQuorum, +}; + +pub(crate) use dispatch::RemoteMutationRoute; diff --git a/crates/cellar/durability/raft/cell_groups/observation.rs b/crates/cellar/durability/raft/cell_groups/observation.rs new file mode 100644 index 000000000..c131532ca --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/observation.rs @@ -0,0 +1,187 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Bounded, same-owner physical log metadata for administration and measurement. +use super::CellGroupStatus; +use crate::application::consensus::RaftReplicaRole; +use cellar_logic::{ + cell_group::{CellKey, GroupHistory, GroupRoute, MembershipRevision}, + cell_group_placement::{ + HotCell, PartitionBasis, PartitionLoad, PartitionSample, HOT_WRITE_WINDOW_MS, + }, + consensus::{Durability, LogIndex, LogPosition, ReplicaIdentity}, +}; + +/// A local sample of one physical group, independent of its cell occupancy. +/// Selected in one Raft-owner turn with a single durable-cache read. It grants +/// no quorum, serving lease, archive coverage or permission to reclaim logs. +#[derive(Clone, Debug)] +pub struct CellGroupObservation { + pub replica: ReplicaIdentity, + pub status: CellGroupStatus, + pub role: RaftReplicaRole, + pub membership_revision: LogIndex, + /// Last locally persisted entry or checkpoint, including uncommitted data. + pub persisted: LogIndex, + /// Installed local checkpoint, which can lag both apply and archive output. + pub checkpoint: LogPosition, + /// Entries retained after the checkpoint; this is not a byte/disk estimate. + pub retained_entries: usize, + /// Logical cells in the currently applied manifest, including sealed cells. + pub cells: u32, + /// Compaction requires independently verified archive coverage on this log. + pub archive_required: bool, +} + +/// Geometry and whole-key neighbors from one selected local manifest. +#[derive(Clone, Debug)] +pub struct CellGroupPartitionCell { + pub key: CellKey, + pub next: Option, + pub has_predecessor: bool, + pub logical_bytes: u64, +} + +/// A lower bound for one writer in a complete one-second owner window. +#[derive(Clone, Debug)] +pub struct CellGroupPartitionWriter { + pub cell: CellGroupPartitionCell, + pub started_at_ms: u64, + pub admitted_writes: u64, +} + +/// Cumulative work on this physical owner and its monotonic sample time. +#[derive(Clone, Copy, Debug)] +pub struct CellGroupWorkload { + pub(super) owner: [u8; 16], + pub sampled_at_ms: u64, + /// New Raft proposals for enrollments and mutations. Reusing a proposal + /// or serving a durable receipt does not increase this counter. + pub admitted_writes: u64, + /// Fresh cell leaf installations after durable apply. Metadata-only + /// entries and received snapshots do not increase this counter. + pub applied_writes: u64, + /// Age of the oldest committed entry still in the local apply queue. + pub queue_age_ms: u64, +} + +/// Placement observations from the controller that owns this sample. Supply +/// an actual directory digest and resource readings for the selected route. +#[derive(Clone, Copy, Debug)] +pub struct CellGroupPartitionContext { + pub directory_digest: [u8; 32], + pub topology_changed_at_ms: u64, + pub durability: Durability, + pub placement_class: [u8; 32], + pub leader_cpu_bps: u16, + pub failure_domain_load_bps: u16, + pub active_and_stable: bool, +} + +/// Bounded scheduler input from one local owner. Advisory only. +#[derive(Clone, Debug)] +pub struct CellGroupPartitionObservation { + pub route: GroupRoute, + pub membership: MembershipRevision, + pub status: CellGroupStatus, + /// Local leadership and membership are stable, including while apply runs. + pub stable_leader: bool, + pub workload: CellGroupWorkload, + pub cells: u32, + pub logical_bytes: u64, + pub last_mutation: Option, + pub balanced_pivot: Option, + pub largest_cell: Option, + pub hot_writer: Option, + /// Independently sampled archive owner, bound to this exact route. + pub archive_lag_entries: Option, +} + +impl CellGroupPartitionObservation { + /// Convert two samples from the same stable local owner into a policy + /// observation. The context's topology time uses this owner's monotonic + /// clock. Sample and recommendation carry no authority to change routes. + pub fn sample_since( + &self, + earlier: &Self, + context: CellGroupPartitionContext, + ) -> Option { + if !self.stable_leader + || !earlier.stable_leader + || self.route != earlier.route + || self.membership != earlier.membership + || self.status.term != earlier.status.term + || self.workload.owner != earlier.workload.owner + { + return None; + } + let elapsed_ms = self + .workload + .sampled_at_ms + .checked_sub(earlier.workload.sampled_at_ms)?; + let rate = |current: u64, previous| { + if current == u64::MAX || elapsed_ms == 0 { + return None; + } + let count = current.checked_sub(previous)?; + u64::try_from(u128::from(count) * 1_000 / u128::from(elapsed_ms)).ok() + }; + let writes_per_second = rate( + self.workload.admitted_writes, + earlier.workload.admitted_writes, + )?; + let applies_per_second = rate( + self.workload.applied_writes, + earlier.workload.applied_writes, + )?; + let hot_writer = self.hot_writer.as_ref().and_then(|writer| { + // Include only a complete window contained in this interval. + // Consecutive policy observations never count one window twice. + let end = writer.started_at_ms.checked_add(HOT_WRITE_WINDOW_MS)?; + if writer.started_at_ms < earlier.workload.sampled_at_ms + || end > self.workload.sampled_at_ms + { + return None; + } + let writes = rate(writer.admitted_writes, 0)?; + Some(writer.cell.hot(writes.min(writes_per_second))) + }); + Some(PartitionSample { + basis: PartitionBasis { + route: self.route.clone(), + directory_digest: context.directory_digest, + membership: self.membership, + term: self.status.term, + durability: context.durability, + placement_class: context.placement_class, + }, + sampled_at_ms: self.workload.sampled_at_ms, + topology_changed_at_ms: context.topology_changed_at_ms, + load: PartitionLoad { + cells: self.cells, + logical_bytes: self.logical_bytes, + writes_per_second, + }, + applies_per_second, + queue_age_ms: self.workload.queue_age_ms, + archive_lag_entries: self.archive_lag_entries, + leader_cpu_bps: context.leader_cpu_bps, + failure_domain_load_bps: context.failure_domain_load_bps, + active_and_stable: context.active_and_stable, + balanced_pivot: self.balanced_pivot.clone(), + hot_cell: self.largest_cell.as_ref().map(|cell| cell.hot(0)), + hot_writer, + }) + } +} + +impl CellGroupPartitionCell { + fn hot(&self, writes_per_second: u64) -> HotCell { + HotCell { + key: self.key.clone(), + next: self.next.clone(), + has_predecessor: self.has_predecessor, + logical_bytes: self.logical_bytes, + writes_per_second, + } + } +} diff --git a/crates/cellar/durability/raft/cell_groups/owner.rs b/crates/cellar/durability/raft/cell_groups/owner.rs new file mode 100644 index 000000000..fb07d8c4d --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/owner.rs @@ -0,0 +1,3350 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! One physical group owns one RawNode. The node scheduler executes the emitted +//! effects; this owner performs no image I/O and starts no per-cell tasks. + +use super::{ + hibernate::{ + LocalParkChange, LocalParkRecord, ParkAcknowledgement, ParkPreparation, ParkPrepareRequest, + MAX_PARK_NEGOTIATION_MS, + }, + CellGroupBootstrapPermit, GroupManifest, GroupParkCut, ParkCertificate, + PreparedCellGroupCheckpoint, PreparedGroupManifest, PreparedLocalCellGroupCheckpoint, +}; +use crate::application::consensus::group_protocol::{ + CellGroupEntry, GroupCommand, GroupEntryBinding, GroupEntryLimits, +}; +use crate::raft::{ + cache::{ConsensusStorage, GroupLog, GroupWrite, SnapshotChange}, + completion::{entry_digest, entry_position, RuntimeError}, + options::RaftRuntimeOptions, + storage::ReplicaLog, +}; +use ::raft::{prelude::*, Config, LightReady, RawNode, Ready, StateRole, Storage}; +use cellar_logic::{ + cell_group::*, + cell_group_placement::WriteHeat, + consensus::{LogIndex, LogPosition, MemberId, OperationId, ReplicaIdentity, Term}, +}; +use protobuf::Message as _; +use rand::RngCore as _; +use std::{ + collections::VecDeque, + sync::{ + atomic::{AtomicBool, AtomicU64, Ordering}, + Arc, Weak, + }, +}; + +struct Lease(Arc); + +impl Drop for Lease { + fn drop(&mut self) { + self.0.store(false, Ordering::Release); + } +} + +#[derive(Clone)] +pub struct CellGroupTicket { + lease: Arc, + sequence: u64, + position: LogPosition, + digest: [u8; 32], + pub(super) admission: Option>, + pub(crate) preparation: Option>, +} + +impl CellGroupTicket { + pub fn position(&self) -> LogPosition { + self.position + } + + pub fn entry_digest(&self) -> [u8; 32] { + self.digest + } + + fn matches(&self, other: &Self) -> bool { + Arc::ptr_eq(&self.lease, &other.lease) + && self.sequence == other.sequence + && self.position == other.position + && self.digest == other.digest + } +} + +/// Physical work retains the owner lease even if its awaiting client is dropped. +/// The engine request also retains that lease until sync/publication finishes. +pub struct CellGroupPersist { + log: ReplicaLog, + pub(super) ticket: CellGroupTicket, + pub(super) write: GroupWrite, +} + +pub struct CellGroupPersisted { + pub(super) ticket: CellGroupTicket, + pub(super) sequence: u64, +} + +impl CellGroupPersist { + /// Execute through the existing node-wide persistence worker. A lost waiter + /// cannot advance the owner; its pending work must settle or restart recover. + pub async fn run(self) -> Result { + let operation = OperationId::new(self.ticket.sequence); + let receipt = self.log.persist_cells(operation, vec![self.write]).await?; + Ok(CellGroupPersisted { + ticket: self.ticket, + sequence: receipt.sequence, + }) + } +} + +#[derive(Clone)] +pub struct CellGroupApply { + pub ticket: CellGroupTicket, + pub entry: Entry, + pub base: PreparedGroupManifest, +} + +/// Snapshot admission does not establish persistence or serving authority. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum CellGroupSnapshotAdmission { + /// Raft selected the verified snapshot; persistence is still outstanding. + Restoring, + /// Raft retained its log. Any resulting hard state/messages still need sync. + Ignored, +} + +pub enum CellGroupEffect { + Persist(Box), + Message(Message), + Apply(Box), + Applied { + ticket: CellGroupTicket, + manifest: Box, + }, + CheckpointInstalled { + manifest: Box, + }, + CheckpointPublished { + manifest: Box, + }, + /// A barrier to wait for, not permission to serve a cell. The dispatcher + /// must recheck owner/term and pin an applied coherent cell view. + ReadIndex { + term: Term, + index: LogIndex, + context: Vec, + }, +} + +impl CellGroupEffect { + pub(super) fn bytes(&self) -> usize { + 4096 + match self { + Self::Persist(work) => work.write.estimated_bytes().unwrap_or(usize::MAX - 4096), + Self::Message(message) => message.compute_size() as usize, + Self::Apply(work) => work.entry.compute_size() as usize + work.base.reserved_bytes(), + Self::ReadIndex { context, .. } => context.len(), + Self::Applied { .. } + | Self::CheckpointInstalled { .. } + | Self::CheckpointPublished { .. } => 1024, + } + } +} + +#[derive(Clone, Debug)] +pub struct CellGroupProposal { + pub group: GroupHistory, + pub position: LogPosition, + pub digest: [u8; 32], +} + +#[derive(Clone, Debug)] +pub struct CellGroupStatus { + pub term: Term, + pub leader: Option, + pub applied: LogPosition, + pub committed: LogIndex, + pub ready: bool, +} + +/// A historical witness that a pending child committed and durably applied +/// a fresh-term barrier. Only its Raft owner can construct this proof. +#[derive(Clone)] +pub struct TransferChildQuorum { + marker: GroupManifest, + leader: super::InstalledTransferVoter, +} + +impl TransferChildQuorum { + pub(crate) fn into_parts(self) -> (GroupManifest, super::InstalledTransferVoter) { + (self.marker, self.leader) + } +} + +/// Admission is distinct from a seal committed and selected by this owner. +pub enum CellGroupSealOutcome { + Accepted(CellGroupProposal), + Applied(super::GroupSourceSeal), +} + +/// The exact retirement is either proposed or durably applied. +pub enum CellGroupSourceRetirement { + Accepted(CellGroupProposal), + Applied(Box), +} + +/// Admission is distinct from activation committed and selected by this owner. +pub enum CellGroupRecoveryActivation { + Accepted(CellGroupProposal), + Applied(Box), +} + +/// An admitted activation proposal or its durable applied position. +pub enum CellGroupTransferActivation { + Accepted(CellGroupProposal), + Applied(Box), +} + +enum Phase { + Ready(Box, Term), + Light(LightReady), + Apply(CellGroupTicket, Box, bool), + Checkpoint( + PreparedLocalCellGroupCheckpoint, + bool, + Arc, + ), +} + +struct ConfigurationFence { + index: u64, + announced: bool, +} + +/// Local evidence that the leader synced its exact certificate selection. +pub(crate) struct SelectedParkCertificate { + certificate: Box, + sync_sequence: u64, +} + +impl SelectedParkCertificate { + pub(crate) fn certificate(&self) -> &ParkCertificate { + &self.certificate + } + + pub(crate) fn sync_sequence(&self) -> u64 { + self.sync_sequence + } +} + +struct PendingPark { + request: ParkPrepareRequest, + preparation: ParkPreparation, + abort: Option, +} + +struct PendingApply { + entry: Entry, + queued_at_ms: u64, +} + +pub struct CellGroupOwner { + log_admission: super::log_admission::LogAdmission, + leader_term: Arc, + source_epoch: Arc, + source_identity: (u64, u64), + source_observed: Option<(u64, u64, u64)>, + lease: Arc, + log: ReplicaLog, + group: GroupLog, + node: RawNode, + current: PreparedGroupManifest, + checkpoint: Option, + options: RaftRuntimeOptions, + phase: Option, + waiting: Option, + effects: VecDeque, + deliveries: Vec>, + configuration_fence: Option, + apply: VecDeque, + workload_owner: [u8; 16], + admitted_writes: u64, + applied_writes: u64, + write_heat: WriteHeat, + apply_bytes: usize, + applying: Option, + next_ticket: u64, + last_sync: u64, + failed: Option, + proposals: super::proposal_admission::ProposalAdmission, + pub(super) proposal_budget: Arc, + prepared_only: bool, + managed_recovery_fenced: bool, + activation_proposal: Option>, + retirement_proposal: Option>, + pending_seal: Option>, + pending_park: Option>, + parked: Option, + waking_after: Option, +} + +impl CellGroupOwner { + /// Attach a durably claimed member with a completely verified selected + /// root. The catalog permit is revalidated; serving additionally waits for + /// the current term's committed/applied Raft barrier. Managed members and + /// ordinary snapshot messages remain closed; step_checkpoint requires the + /// exact independently verified destination closure. + pub async fn start( + permit: CellGroupBootstrapPermit, + current: PreparedGroupManifest, + options: RaftRuntimeOptions, + seed: [u8; 32], + ) -> Result { + options.validate(permit.log().limits())?; + let group = permit.group().clone(); + group + .consensus_claimed + .compare_exchange(false, true, Ordering::AcqRel, Ordering::Acquire) + .map_err(|_| RuntimeError::AlreadyAttached)?; + let lease = Arc::new(Lease(group.consensus_claimed.clone())); + Box::pin(permit.revalidate()) + .await + .map_err(|e| RuntimeError::Invalid(e.to_string()))?; + Box::pin(permit.log().close_retained_park(&group)).await?; + let cache = group.read()?; + if !Arc::ptr_eq(¤t.engine_owner, &group.owner) + || cache.group_manifest.as_ref() != Some(current.marker()) + || current.marker().route() != permit.route() + || !cache.membership.permits_group(group.identity()) + || cache.conf_state.voters.len() + cache.conf_state.learners.len() + > options.max_members_per_group + { + return Err(RuntimeError::Invalid( + "cell-group startup closure/member".into(), + )); + } + let waking_after = cache + .local_park + .as_ref() + .and_then(LocalParkRecord::wake_floor) + .filter(|floor| { + current.marker().position().index <= *floor + || current.marker().position().term.get() != cache.hard_state.term + }); + drop(cache); + current + .require_checkpoint(&group) + .map_err(|e| RuntimeError::Invalid(e.to_string()))?; + let config = Config { + id: group.identity().member.get(), + applied: current.marker().position().index.get(), + election_tick: 20, + heartbeat_tick: 2, + check_quorum: true, + pre_vote: true, + read_only_option: ::raft::ReadOnlyOption::Safe, + max_size_per_msg: options.max_message_bytes as u64, + max_inflight_msgs: 8, + max_uncommitted_size: options.max_group_apply_bytes as u64, + // A valid entry has at least four encoded bytes (term/index). Its + // retained accounting adds 128 bytes. Budget for those small + // entries as well as payloads; raft-rs may still return one entry + // larger than this soft byte cap to guarantee progress. + max_committed_size_per_ready: (options.max_message_bytes / 66).max(1) as u64, + ..Default::default() + }; + let logger = slog::Logger::root(slog::Discard, slog::o!()); + let node = RawNode::new_with_election_seed( + &config, + group.consensus_storage(), + &logger, + Some(seed), + )?; + let mut workload_owner = [0; 16]; + crate::asyncrt::rng("cell-group-workload-owner").fill_bytes(&mut workload_owner); + Ok(Self { + log_admission: Default::default(), + leader_term: Arc::new(AtomicU64::new(0)), + source_epoch: Arc::new(AtomicU64::new(1)), + source_identity: (0, 0), + source_observed: None, + lease, + log: permit.log().clone(), + group, + node, + current, + checkpoint: None, + proposal_budget: crate::raft::budget::Budget::new( + options.max_request_bytes, + options.max_requests, + ), + options, + phase: None, + waiting: None, + effects: VecDeque::new(), + deliveries: Vec::new(), + configuration_fence: None, + apply: VecDeque::new(), + workload_owner, + admitted_writes: 0, + applied_writes: 0, + write_heat: WriteHeat::default(), + apply_bytes: 0, + applying: None, + next_ticket: 0, + last_sync: 0, + failed: None, + proposals: Default::default(), + prepared_only: false, + managed_recovery_fenced: permit.managed_recovery_fenced(), + activation_proposal: None, + retirement_proposal: None, + pending_seal: None, + pending_park: None, + parked: None, + waking_after, + }) + } + + fn finish_log_admission( + &mut self, + next: super::log_admission::Prospective, + ) -> Result<(), RuntimeError> { + if let Err(error) = self.log_admission.finish(&self.group, &self.node, next) { + self.failed = Some(error.clone()); + self.revoke_execution(); + return Err(error); + } + Ok(()) + } + + pub(super) fn revoke_execution(&mut self) { + self.leader_term.store(0, Ordering::Release); + self.source_epoch.store(0, Ordering::Release); + self.proposals.clear(); + self.activation_proposal = None; + self.retirement_proposal = None; + self.pending_seal = None; + self.pending_park = None; + } + + fn advance_source_epoch(&self) { + if self + .source_epoch + .fetch_update(Ordering::AcqRel, Ordering::Acquire, |epoch| { + (epoch != 0).then(|| epoch.checked_add(1)).flatten() + }) + .is_err() + { + // Exhaustion or revocation never rearms an old read capability. + self.source_epoch.store(0, Ordering::Release); + } + } + + fn refresh_execution_fence(&mut self) { + let source = (self.node.raft.term, self.node.raft.leader_id); + if source != self.source_identity { + self.source_identity = source; + self.source_observed = None; + self.advance_source_epoch(); + } + if self.pending_seal.as_ref().is_some_and(|pending| { + pending.0.get() != self.node.raft.term || self.node.raft.state != StateRole::Leader + }) { + self.pending_seal = None; + } + if self.pending_park.as_ref().is_some_and(|pending| { + pending.request.term.get() != self.node.raft.term + || self.node.raft.state != StateRole::Leader + }) { + self.pending_park = None; + } + let at = self.current.marker().position(); + if self + .waking_after + .is_some_and(|floor| at.index > floor && at.term.get() == self.node.raft.term) + { + self.waking_after = None; + } + let term = if !self.managed_recovery_fenced + && !self.park_fenced() + && self.pending_seal.is_none() + && self.current.marker().source_seal().is_none() + && self.node.raft.state == StateRole::Leader + && self.node.raft.lead_transferee.is_none() + && self + .node + .raft + .prs() + .conf() + .voters() + .contains(self.node.raft.id) + { + self.node.raft.term + } else { + 0 + }; + if self.leader_term.swap(term, Ordering::AcqRel) != term { + self.proposals.clear(); + self.activation_proposal = None; + } + } + + pub fn execution_authority( + &self, + cell: &CellHistory, + ) -> Result { + let status = self.status()?; + // Role refusal is distinct from a leader still awaiting durable apply. + // Remote callers must not retry a follower as generic unavailability. + if self.node.raft.state != StateRole::Leader { + return Err(RuntimeError::NotLeader); + } + if !status.ready || status.applied.index != status.committed { + return Err(RuntimeError::NotReady); + } + super::CellExecutionAuthority::new( + self.group.clone(), + &self.current, + cell, + status.term, + self.leader_term.clone(), + ) + } + + pub(crate) fn next_effect( + &self, + now: u64, + after: Option<&CellId>, + ) -> Result, RuntimeError> { + let status = self.status()?; + if self.node.raft.state != StateRole::Leader { + return Err(RuntimeError::NotLeader); + } + if !status.ready || status.applied.index != status.committed { + return Err(RuntimeError::NotReady); + } + let Some((leaf, _pin)) = self.current.next_due(now, after) else { + return Ok(None); + }; + self.execution_authority(&leaf.version.history).map(Some) + } + + pub fn manifest(&self) -> &PreparedGroupManifest { + &self.current + } + + pub(super) fn require_application_active(&self) -> Result<(), RuntimeError> { + self.live()?; + if self.managed_recovery_fenced + || self.park_fenced() + || self.pending_seal.is_some() + || !self.current.marker().is_active() + { + return Err(RuntimeError::NotReady); + } + Ok(()) + } + + pub(super) fn capture_checkpoint( + &self, + pool: &super::CellGroupCheckpointPool, + ) -> Result { + self.live() + .map_err(super::CellGroupCheckpointError::Authority)?; + pool.capture_owner(&self.group, &self.current) + } + + pub(super) fn select_read( + &self, + cell: &CellHistory, + consistency: &super::CellReadConsistency, + term: Option, + barrier: Option, + ) -> Result { + self.require_application_active()?; + let status = self.status()?; + let mut source = None; + let mut source_deadline = None; + let leader = if matches!(consistency, super::CellReadConsistency::Linearizable) { + if term.is_none() { + return Err(RuntimeError::NotReady); + } + if Some(status.term) != term { + return Err(RuntimeError::NotLeader); + } + let index = barrier.ok_or(RuntimeError::NotReady)?; + if status.applied.index < index { + return Err(RuntimeError::NotReady); + } + if self.node.raft.state == StateRole::Leader { + if !status.ready { + return Err(RuntimeError::NotReady); + } + Some((status.term, self.leader_term.clone())) + } else if self.node.raft.state == StateRole::Follower && self.membership_read_ready()? { + source = Some(self.read_source_epoch(status.term)?); + None + } else { + return Err(RuntimeError::NotLeader); + } + } else { + None + }; + if let super::CellReadConsistency::RecentSource { max_age_ms } = consistency { + if !(1..=3_600_000).contains(max_age_ms) { + return Err(RuntimeError::Invalid("stale source age".into())); + } + let now = self + .log + .domain() + .mono_ms() + .map_err(|_| RuntimeError::Closed)?; + let observed = if self.node.raft.state == StateRole::Leader + && self.current.marker().position().term == status.term + { + Some(now) + } else { + self.source_observed + .filter(|(member, term, _)| { + *member == self.node.raft.leader_id && *term == status.term.get() + }) + .map(|(_, _, at)| at) + }; + source_deadline = Some( + crate::raft::query::recent_source_deadline(observed, now, *max_age_ms) + .ok_or(RuntimeError::NotReady)?, + ); + source = Some(self.read_source_epoch(status.term)?); + } + let (leaf, image) = self + .current + .pin_cell(&cell.cell) + .ok_or(RuntimeError::NotFound)?; + if &leaf.version.history != cell { + return Err(RuntimeError::Invalid("cell read history".into())); + } + if let super::CellReadConsistency::AtLeast(minimum) = consistency { + if !leaf + .version + .covers(minimum) + .map_err(|_| RuntimeError::Invalid("cell read version".into()))? + { + return Err(RuntimeError::NotReady); + } + } + Ok(super::read_authority::ReadSelection { + group: self.group.clone(), + route: self.current.marker().route().clone(), + cell: leaf.clone(), + image, + applied: status.applied, + barrier: term + .zip(barrier) + .map(|(term, index)| LogPosition { term, index }), + leader, + source, + source_deadline, + }) + } + + fn read_source_epoch(&self, term: Term) -> Result<(Term, u64, Arc), RuntimeError> { + let epoch = self.source_epoch.load(Ordering::Acquire); + if epoch == 0 || self.group.read()?.hard_state.term != term.get() { + return Err(RuntimeError::NotReady); + } + Ok((term, epoch, self.source_epoch.clone())) + } + + pub(super) fn check_seed_target( + &self, + target: &ReplicaIdentity, + ) -> Result<(), RuntimeError> { + self.live()?; + super::seed_authority::check_target(&self.group, target).map(|_| ()) + } + + pub(super) fn select_membership( + &self, + minimum: LogIndex, + term: Option, + barrier: Option, + ) -> Result { + self.require_application_active()?; + let status = self.status()?; + if Some(status.term) != term { + return Err(RuntimeError::NotLeader); + } + let index = barrier.ok_or(RuntimeError::NotReady)?; + if !self.membership_read_ready()? + || status.applied.index < index + || status.applied.index < minimum + { + return Err(RuntimeError::NotReady); + } + let cache = self.group.read()?; + let root = self.current.marker(); + if cache.group_manifest.as_ref() != Some(root) + || root.membership_revision() != cache.membership.revision().get() + || cache.membership.revision() > index + { + return Err(RuntimeError::NotReady); + } + cache + .membership + .validate(&cache.conf_state, root.position().index.get())?; + cache.membership.validate_scope(&root.route().history)?; + let bytes = cache + .membership + .reserved_bytes() + .checked_add(cache.conf_state.compute_size() as usize) + .and_then(|n| n.checked_add(serde_json::to_vec(root).ok()?.len())) + .and_then(|n| n.checked_add(4096)) + .ok_or(RuntimeError::Overloaded)?; + if bytes > super::membership_observation::MEMBERSHIP_BYTES { + return Err(RuntimeError::Overloaded); + } + Ok(super::membership_observation::MembershipSelection { + replica: self.group.identity().clone(), + engine: self.log.storage_instance()?, + root: root.clone(), + barrier: LogPosition { + term: status.term, + index, + }, + membership: cache.membership.clone(), + configuration: cache.conf_state.clone(), + status, + }) + } + + pub(super) fn select_seed( + &self, + target: &ReplicaIdentity, + term: Option, + barrier: Option, + ) -> Result { + self.live()?; + let status = self.status()?; + if self.node.raft.state != raft::StateRole::Leader || Some(status.term) != term { + return Err(RuntimeError::NotLeader); + } + let index = barrier.ok_or(RuntimeError::NotReady)?; + if !status.ready + || status.applied.index < index + || self.current.marker().membership_revision() > index.get() + || super::seed_authority::check_target(&self.group, target)? > index + { + return Err(RuntimeError::NotReady); + } + Ok(super::seed_authority::SeedSelection { + group: self.group.clone(), + root: self.current.clone(), + target: target.clone(), + barrier: LogPosition { + term: status.term, + index, + }, + leader: self.leader_term.clone(), + }) + } + + pub fn enrollment_state( + &self, + cell: CellHistory, + ) -> Result { + self.live()?; + if self.managed_recovery_fenced { + return Err(RuntimeError::NotReady); + } + super::CellEnrollmentState::new(self.group.clone(), &self.current, cell) + } + + pub fn enrollment_authority( + &self, + cell: CellHistory, + ) -> Result { + let status = self.status()?; + if self.node.raft.state != StateRole::Leader { + return Err(RuntimeError::NotLeader); + } + if !status.ready || status.applied.index != status.committed { + return Err(RuntimeError::NotReady); + } + super::CellEnrollmentAuthority::new(self.enrollment_state(cell)?, self.leader_term.clone()) + } + + pub fn reconciliation_authority( + &self, + cell: &CellHistory, + ) -> Result { + self.live()?; + if self.managed_recovery_fenced { + return Err(RuntimeError::NotReady); + } + super::CellReconciliationAuthority::new(self.group.clone(), &self.current, cell) + } + + /// Count even a leader awaiting its current-term apply barrier. Headroom + /// must never mistake a non-serving leader for an unused leadership slot. + pub(super) fn local_leader(&self) -> Result { + self.live()?; + Ok(self.node.raft.state == StateRole::Leader) + } + + pub fn status(&self) -> Result { + self.live()?; + let cache = self.group.read()?; + Ok(self.status_at(&cache)) + } + + fn status_at(&self, cache: &crate::raft::cache::CachedGroup) -> CellGroupStatus { + CellGroupStatus { + term: Term::new(self.node.raft.term), + leader: (self.node.raft.leader_id != 0).then_some(self.node.raft.leader_id), + applied: self.current.marker().position(), + committed: LogIndex::new(cache.hard_state.commit), + ready: !self.managed_recovery_fenced + && !self.park_fenced() + && self.pending_seal.is_none() + && self.current.marker().is_active() + && self.protocol_ready(cache), + } + } + + pub(super) fn observe(&self) -> Result { + use crate::application::consensus::RaftReplicaRole; + self.live()?; + let cache = self.group.read()?; + let replica = self.group.identity().clone(); + let member = replica.member.get(); + let role = if cache.conf_state.voters.contains(&member) + || cache.conf_state.voters_outgoing.contains(&member) + { + if self.node.raft.state == StateRole::Leader { + RaftReplicaRole::Leader + } else { + RaftReplicaRole::Voter + } + } else if cache.conf_state.learners.contains(&member) + || cache.conf_state.learners_next.contains(&member) + { + RaftReplicaRole::Learner + } else { + RaftReplicaRole::Removed + }; + let checkpoint = cache.snapshot.get_metadata(); + Ok(super::CellGroupObservation { + replica, + status: self.status_at(&cache), + role, + membership_revision: cache.membership.revision(), + persisted: LogIndex::new(cache.last_index()), + checkpoint: LogPosition { + term: Term::new(checkpoint.term), + index: LogIndex::new(checkpoint.index), + }, + retained_entries: cache.entries.len(), + cells: self.current.marker().root().cells(), + archive_required: cache.archive_required, + }) + } + + pub(super) fn observe_partition( + &self, + ) -> Result { + self.live()?; + let cache = self.group.read()?; + let marker = self.current.marker(); + if cache.group_manifest.as_ref() != Some(marker) { + return Err(RuntimeError::NotReady); + } + let pins = &self.current.images; + let sampled_at_ms = self + .log + .domain() + .mono_ms() + .map_err(|_| RuntimeError::Closed)?; + let cells = marker.root().cells(); + let geometry = |(rank, logical_bytes)| { + Some(super::CellGroupPartitionCell { + key: pins.key_at_rank(rank)?.clone(), + next: pins.key_at_rank(rank + 1).cloned(), + has_predecessor: rank > 0, + logical_bytes, + }) + }; + let membership = MembershipRevision::new(cache.membership.revision().get()) + .map_err(|_| RuntimeError::NotReady)?; + let status = self.status_at(&cache); + let hot_writer = self + .write_heat + .observed(status.term, membership, sampled_at_ms) + .and_then(|(started_at_ms, cell, admitted_writes)| { + let key = crate::application::consensus::cell_routing_key( + marker.route().history.namespace, + cell, + ); + let cell = geometry(pins.image_rank(&key)?)?; + Some(super::CellGroupPartitionWriter { + cell, + started_at_ms, + admitted_writes, + }) + }); + Ok(super::CellGroupPartitionObservation { + route: marker.route().clone(), + membership, + status, + stable_leader: !self.managed_recovery_fenced + && marker.is_active() + && !self.park_fenced() + && self.pending_seal.is_none() + && self.configuration_fence.is_none() + && cache.conf_state.voters_outgoing.is_empty() + && self.node.raft.pending_conf_index <= marker.position().index.get() + && self.selected_leader(&cache), + workload: super::CellGroupWorkload { + owner: self.workload_owner, + sampled_at_ms, + admitted_writes: self.admitted_writes, + applied_writes: self.applied_writes, + queue_age_ms: self.apply.front().map_or(0, |pending| { + sampled_at_ms.saturating_sub(pending.queued_at_ms) + }), + }, + cells, + logical_bytes: pins.logical_bytes(), + last_mutation: marker.root().last_mutation(), + balanced_pivot: if cells > 1 { + pins.key_at_rank(cells as usize / 2).cloned() + } else { + None + }, + largest_cell: pins.largest_image_rank().and_then(geometry), + hot_writer, + archive_lag_entries: None, + }) + } + + /// Observe the real leader's progress tracker. This is advisory; transfer + /// admission rechecks the same owner against the caller's term and revision. + pub fn replication_status( + &self, + ) -> Result, RuntimeError> { + use crate::raft::completion::{MemberProgress, ReplicationStatus}; + self.live()?; + if self.node.raft.state != StateRole::Leader { + return Err(RuntimeError::NotLeader); + } + let cache = self.group.read()?; + let mut members = self + .node + .raft + .prs() + .iter() + .map(|(&id, progress)| { + Ok(MemberProgress { + member: MemberId::new(id).map_err(|e| RuntimeError::Protocol(e.to_string()))?, + matched: LogIndex::new(progress.matched), + voter: cache.conf_state.voters.contains(&id) + || cache.conf_state.voters_outgoing.contains(&id), + recent_active: progress.recent_active, + }) + }) + .collect::, RuntimeError>>()?; + members.sort_by_key(|member| member.member); + Ok(ReplicationStatus { + replica: self.group.identity().clone(), + term: Term::new(self.node.raft.term), + membership_revision: cache.membership.revision(), + committed: LogIndex::new(self.node.raft.raft_log.committed), + members, + }) + } + + fn protocol_ready(&self, cache: &crate::raft::cache::CachedGroup) -> bool { + self.phase.is_none() + && self.checkpoint.is_none() + && self.apply_bytes < self.options.max_group_apply_bytes + && self.selected_leader(cache) + } + + fn selected_leader(&self, cache: &crate::raft::cache::CachedGroup) -> bool { + self.node.raft.state == StateRole::Leader + && self.node.raft.lead_transferee.is_none() + && self + .node + .raft + .prs() + .conf() + .voters() + .contains(self.node.raft.id) + && self.current.marker().position().term.get() == self.node.raft.term + && cache.hard_state.term == self.node.raft.term + && cache.group_manifest.as_ref() == Some(self.current.marker()) + } + + pub(super) fn pending_transfer_quorum(&self) -> Result { + self.live()?; + let marker = self.current.marker(); + let cache = self.group.read()?; + if marker.transfer_origin().is_none() + || marker.transfer_activation().is_some() + || marker.membership_revision() != 1 + || cache.membership.revision().get() != 1 + || !cache.conf_state.voters_outgoing.is_empty() + || !cache.conf_state.learners_next.is_empty() + || cache.conf_state.auto_leave + { + return Err(RuntimeError::NotReady); + } + if self.node.raft.state != StateRole::Leader { + return Err(RuntimeError::NotLeader); + } + let position = marker.position(); + if !self.protocol_ready(&cache) + || position.term.get() <= 1 + || position.index.get() <= 1 + || position.index.get() != cache.hard_state.commit + || !self.apply.is_empty() + || self.applying.is_some() + { + return Err(RuntimeError::NotReady); + } + let local = self.group.identity(); + Ok(TransferChildQuorum { + marker: marker.clone(), + leader: super::InstalledTransferVoter { + member: local.member, + node: local.node, + disk: local.disk, + engine: self.log.storage_instance()?, + }, + }) + } + + /// Admit only the selected recovery's exact activation. Retries in the same + /// term share one bounded proposal; a new leader must apply its fresh-term + /// barrier before it can decide whether an inherited activation survived. + pub fn activate_recovery( + &mut self, + permit: super::CellGroupRecoveryActivationPermit, + ) -> Result { + self.live()?; + permit.check(&self.log, &self.group)?; + if permit.matches(self.current.marker())? { + self.activation_proposal = None; + return Ok(CellGroupRecoveryActivation::Applied(Box::new( + permit.acknowledgement(&self.group, self.current.marker())?, + ))); + } + self.propose_activation(|| permit.entry()) + .map(CellGroupRecoveryActivation::Accepted) + } + + pub fn activate_transfer( + &mut self, + permit: super::CellGroupTransferActivationPermit, + ) -> Result { + self.live()?; + permit.check(&self.log, &self.group)?; + let marker = self.current.marker(); + if let Some(acknowledgement) = permit.acknowledge(&self.group, marker)? { + self.activation_proposal = None; + return Ok(CellGroupTransferActivation::Applied(Box::new( + acknowledgement, + ))); + } + self.propose_activation(|| Ok(permit.into_entry())) + .map(CellGroupTransferActivation::Accepted) + } + + fn propose_activation( + &mut self, + entry: impl FnOnce() -> Result, RuntimeError>, + ) -> Result { + if self.node.raft.state != StateRole::Leader { + return Err(RuntimeError::NotLeader); + } + if let Some(pending) = &self.activation_proposal { + if pending.0.get() == self.node.raft.term { + return Ok(pending.1.clone()); + } + } + self.require_control_prefix()?; + let proposal = self.append_control(entry()?, "group activation")?; + self.activation_proposal = + Some(Box::new((Term::new(self.node.raft.term), proposal.clone()))); + Ok(proposal) + } + + pub fn retire_source( + &mut self, + permit: super::CellGroupSourceRetirementPermit, + ) -> Result { + self.live()?; + permit.check(&self.log, &self.group)?; + let marker = self.current.marker(); + if let Some(retirement) = permit.acknowledge(&self.group, marker)? { + self.retirement_proposal = None; + return Ok(CellGroupSourceRetirement::Applied(Box::new(retirement))); + } + if self.node.raft.state != StateRole::Leader { + return Err(RuntimeError::NotLeader); + } + let term = Term::new(self.node.raft.term); + if let Some(pending) = &self.retirement_proposal { + if pending.0 == term { + return Ok(CellGroupSourceRetirement::Accepted(pending.1.clone())); + } + } + self.require_control_prefix()?; + let proposal = self.append_control(permit.into_entry(), "source retirement")?; + self.retirement_proposal = Some(Box::new((term, proposal.clone()))); + Ok(CellGroupSourceRetirement::Accepted(proposal)) + } + + fn require_control_prefix(&self) -> Result<(), RuntimeError> { + let cache = self.group.read()?; + if self.park_fenced() + || !self.protocol_ready(&cache) + || self.current.marker().position().index.get() != cache.hard_state.commit + || !self.apply.is_empty() + || self.applying.is_some() + { + return Err(RuntimeError::NotReady); + } + Ok(()) + } + + /// Drain the selected prefix before appending the irreversible source cut. + /// Only the rooted transition lane can supply a permit for this entry. + pub fn seal_source( + &mut self, + permit: super::CellGroupSealPermit, + ) -> Result { + self.live()?; + permit.check(&self.log, &self.group)?; + let marker = self.current.marker(); + if let Some(seal) = marker.source_seal() { + let cache = self.group.read()?; + if !permit.matches(seal) + || cache.group_manifest.as_ref() != Some(marker) + || cache.hard_state.commit < seal.position().index.get() + { + return Err(RuntimeError::StaleCompletion); + } + return Ok(CellGroupSealOutcome::Applied(seal.clone())); + } + if self.node.raft.state != StateRole::Leader { + return Err(RuntimeError::NotLeader); + } + let term = Term::new(self.node.raft.term); + let payload = permit.payload(); + if let Some(pending) = &self.pending_seal { + if pending.0 == term && pending.1 == payload { + return Ok(CellGroupSealOutcome::Accepted(pending.2.clone())); + } + return Err(RuntimeError::StaleCompletion); + } + let cache = self.group.read()?; + let index = marker.position().index.get(); + if self.park_fenced() + || !marker.is_active() + || !self.protocol_ready(&cache) + || index != cache.hard_state.commit + || index != self.node.raft.raft_log.last_index() + || !cache.conf_state.voters_outgoing.is_empty() + || !cache.conf_state.learners_next.is_empty() + || !self.apply.is_empty() + || self.applying.is_some() + { + return Err(RuntimeError::NotReady); + } + drop(cache); + let proposal = self.append_control(permit.into_entry(), "source seal")?; + self.pending_seal = Some(Box::new((term, payload, proposal.clone()))); + self.leader_term.store(0, Ordering::Release); + self.advance_source_epoch(); + Ok(CellGroupSealOutcome::Accepted(proposal)) + } + + pub(super) fn prepare_park( + &mut self, + request: ParkPrepareRequest, + ) -> Result { + self.live()?; + if self.node.raft.state != StateRole::Leader { + return Err(RuntimeError::NotLeader); + } + let now = self + .log + .domain() + .mono_ms() + .map_err(|_| RuntimeError::Closed)?; + if request.deadline_ms <= now { + return Err(RuntimeError::DeadlineExceeded); + } + let remaining = request.deadline_ms - now; + if remaining > MAX_PARK_NEGOTIATION_MS { + return Err(RuntimeError::Invalid("park negotiation deadline".into())); + } + if let Some(pending) = &self.pending_park { + let same = ParkPrepareRequest { + deadline_ms: request.deadline_ms, + ..pending.request + }; + return if same == request + && pending.abort.is_none() + && now < pending.request.deadline_ms + { + Ok(pending.preparation.clone()) + } else { + Err(RuntimeError::NotReady) + }; + } + let cache = self.group.read()?; + let marker = self.current.marker(); + let at = marker.position(); + let conf = &cache.conf_state; + let next_due_unix_millis = marker.root().next_due_unix_millis(); + let wall = u64::try_from(crate::asyncrt::wall_ms()).map_err(|_| RuntimeError::NotReady)?; + if request.term.get() != self.node.raft.term + || request.membership.get() != cache.membership.revision().get() + || request.expected_applied != at + { + return Err(RuntimeError::StaleCompletion); + } + if !self.idle() + || !self.status_at(&cache).ready + || cache.hard_state.commit != at.index.get() + || self.node.raft.raft_log.last_index() != at.index.get() + || self.node.raft.pending_conf_index > at.index.get() + || self.configuration_fence.is_some() + || !conf.voters_outgoing.is_empty() + || !conf.learners.is_empty() + || !conf.learners_next.is_empty() + || conf.auto_leave + || self.applying.is_some() + || !self.apply.is_empty() + || self.node.raft.read_only.pending_read_count() != 0 + // Execution, enrollment and claimed-effect handles pin this fence. + || Arc::strong_count(&self.leader_term) != 1 + || next_due_unix_millis.is_some_and(|due| due <= wall.saturating_add(remaining)) + { + return Err(RuntimeError::NotReady); + } + let route = marker.route(); + let index = at + .index + .get() + .checked_add(1) + .ok_or(RuntimeError::Overloaded)?; + let entry = CellGroupEntry { + binding: GroupEntryBinding { + group: route.history, + route_epoch: route.epoch, + birth_digest: route.birth_digest, + schema_digest: route.schema_digest, + proposal: request.proposal, + }, + command: GroupCommand::PreparePark { + generation: ParkGeneration::new(index) + .map_err(|error| RuntimeError::Invalid(error.to_string()))?, + term: request.term, + membership: request.membership, + }, + } + .encode(route, GroupEntryLimits::default()) + .map_err(|error| RuntimeError::Invalid(error.to_string()))?; + drop(cache); + let proposal = self.append_control(entry, "park preparation")?; + let preparation = ParkPreparation { + proposal, + next_due_unix_millis, + deadline_ms: request.deadline_ms, + }; + self.pending_park = Some(Box::new(PendingPark { + request, + preparation: preparation.clone(), + abort: None, + })); + // A same-term abort can mint fresh capabilities. Any old handle must + // remain fenced even if the leader keeps its term. + self.leader_term.store(0, Ordering::Release); + self.leader_term = Arc::new(AtomicU64::new(0)); + self.source_observed = None; + self.advance_source_epoch(); + Ok(preparation) + } + + fn expire_park_preparation(&mut self) -> Result<(), RuntimeError> { + let Some(pending) = self.pending_park.as_ref() else { + return Ok(()); + }; + let now = self + .log + .domain() + .mono_ms() + .map_err(|_| RuntimeError::Closed)?; + if pending.abort.is_none() && now >= pending.request.deadline_ms { + match self.abort_park_preparation() { + // A full log must still tick and let check-quorum run. + Ok(()) | Err(RuntimeError::Overloaded) => (), + Err(error) => return Err(error), + } + } + Ok(()) + } + + fn abort_park_preparation(&mut self) -> Result<(), RuntimeError> { + // Persistence closes local acknowledgements. Apply proves fresh + // quorum progress before the same leader can reopen admission. + let proposal = self.append_control(Vec::new(), "abort park preparation")?; + if let Some(pending) = self.pending_park.as_mut() { + pending.abort = Some(proposal.position); + } + Ok(()) + } + + pub(super) fn fence_managed_park( + &mut self, + generation: ParkGeneration, + term: Term, + deadline: u64, + ) -> Result<(), RuntimeError> { + self.check_park_wake(generation)?; + let now = self + .log + .domain() + .mono_ms() + .map_err(|_| RuntimeError::Closed)?; + if now >= deadline { + return Err(RuntimeError::DeadlineExceeded); + } + let status = self.status()?; + let at = status.applied; + if self.node.raft.state != StateRole::Leader && status.leader.is_some() { + return Err(RuntimeError::NotLeader); + } + if at.term > term || (at.term == term && at.index.get() > generation.get()) { + return if status.ready { + Ok(()) + } else { + Err(RuntimeError::NotReady) + }; + } + if self.node.raft.state != StateRole::Leader || status.term != term { + return Err(RuntimeError::NotReady); + } + if let Some(pending) = self.pending_park.as_ref() { + if pending.preparation.proposal.position.index.get() != generation.get() { + return Err(RuntimeError::StaleCompletion); + } + if pending.abort.is_none() { + self.abort_park_preparation()?; + } + } else if !self.park_fenced() + && at.term == term + && at.index.get().saturating_add(1) >= generation.get() + && self.node.raft.raft_log.last_index() == at.index.get() + && status.ready + { + // A controller can stop after rooting the reservation but before + // proposing the cut. Occupy the reserved index with a no-op, then + // advance beyond it so the catalog can settle on durable proof. + self.append_control(Vec::new(), "fence reserved park cut")?; + } + Err(RuntimeError::NotReady) + } + + fn append_control( + &mut self, + data: Vec, + kind: &str, + ) -> Result { + if data.len() > self.options.max_proposal_bytes { + return Err(RuntimeError::Overloaded); + } + let before = self.node.raft.raft_log.last_index(); + let expected = Entry { + index: before + 1, + term: self.node.raft.term, + data: data.clone().into(), + ..Default::default() + }; + let admission = + self.log_admission + .prepare(&self.group, std::slice::from_ref(&expected), None)?; + let result = self.node.propose(Vec::new(), data); + self.finish_log_admission(admission)?; + if let Err(error) = result { + let error = RuntimeError::from(error); + if self.node.raft.raft_log.last_index() != before { + self.failed = Some(error.clone()); + self.revoke_execution(); + } + return Err(error); + } + let observed = (|| { + let entry = self + .node + .raft + .raft_log + .unstable_entries() + .last() + .ok_or_else(|| RuntimeError::Protocol(format!("missing {kind}")))?; + if entry != &expected { + return Err(RuntimeError::Protocol(format!("changed {kind}"))); + } + Ok(CellGroupProposal { + group: self.group.identity().scope, + position: entry_position(entry), + digest: entry_digest(entry)?, + }) + })(); + let proposal = match observed { + Ok(proposal) => proposal, + Err(error) => { + self.failed = Some(error.clone()); + self.revoke_execution(); + return Err(error); + } + }; + Ok(proposal) + } + + pub fn campaign(&mut self) -> Result<(), RuntimeError> { + self.live()?; + if self.pending_park.is_some() + || self.parked.is_some() + || self.phase.is_some() + || self.checkpoint.is_some() + { + return Err(RuntimeError::NotReady); + } + let admission = self + .log_admission + .election(&self.group, &self.node, Default::default())?; + let result = self.node.campaign(); + self.refresh_execution_fence(); + self.finish_log_admission(admission)?; + result?; + Ok(()) + } + + /// Admit a handoff to a caught-up incoming voter. Completion is observed + /// through leadership status; admission alone cannot prove an election. + pub fn transfer_leader(&mut self, member: u64) -> Result<(), RuntimeError> { + let progress = self.replication_status()?; + self.transfer_leadership(crate::raft::completion::LeadershipTransfer { + target: MemberId::new(member).map_err(|e| RuntimeError::Invalid(e.to_string()))?, + expected_term: progress.term, + expected_revision: progress.membership_revision, + }) + } + + /// Recheck whole-group matched progress immediately before restart + /// handoff/observation. The runtime separately checks the synced stop gate. + pub(crate) fn restart_leadership( + &mut self, + check: crate::raft::completion::RestartLeadershipCheck, + ) -> Result<(), RuntimeError> { + let progress = self.replication_status()?; + let cache = self.group.read()?; + let transfer = crate::raft::restart_leadership::check_progress( + &progress, + &cache.conf_state, + &check, + self.node.raft.pending_conf_index > self.node.raft.raft_log.applied, + self.node.raft.lead_transferee.is_some(), + )?; + drop(cache); + if let Some(transfer) = transfer { + self.transfer_leadership(transfer)?; + } + Ok(()) + } + + pub fn transfer_leadership( + &mut self, + transfer: crate::raft::completion::LeadershipTransfer, + ) -> Result<(), RuntimeError> { + self.live()?; + if self.node.raft.state != StateRole::Leader { + return Err(RuntimeError::NotLeader); + } + let status = self.status()?; + let cache = self.group.read()?; + let member = transfer.target.get(); + if transfer.expected_term != status.term + || transfer.expected_revision != cache.membership.revision() + || member == self.node.raft.id + || !cache.conf_state.voters.contains(&member) + { + return Err(RuntimeError::Invalid( + "leadership target is not another incoming voter".into(), + )); + } + if !status.ready + || !cache.conf_state.voters_outgoing.is_empty() + || self.node.raft.lead_transferee.is_some() + || cache.membership.repair_status()?.is_some_and(|repair| { + matches!( + repair.phase, + crate::raft::member_state::RepairPhase::Seeding + | crate::raft::member_state::RepairPhase::Joint + ) + }) + || status.applied.index != status.committed + || !self.apply.is_empty() + || self.applying.is_some() + || self.node.raft.pending_conf_index > self.node.raft.raft_log.applied + || self.node.raft.prs().get(member).is_none_or(|p| { + !p.recent_active || p.matched < self.node.raft.raft_log.last_index() + }) + { + return Err(RuntimeError::NotReady); + } + drop(cache); + self.node.transfer_leader(member); + self.refresh_execution_fence(); + Ok(()) + } + + pub fn tick(&mut self) -> Result<(), RuntimeError> { + self.live()?; + if self.parked.is_some() { + return Ok(()); + } + // The scheduler coalesces a deferred tick until pending Ready work is + // drained. In particular, an arbitrarily slow fsync must not accumulate + // an unbounded vector of heartbeat/election messages inside RawNode. + if self.phase.is_some() || self.node.has_ready() { + return Err(RuntimeError::NotReady); + } + self.expire_park_preparation()?; + let admission = if self.node.raft.state == StateRole::Leader { + Default::default() + } else { + self.log_admission + .election(&self.group, &self.node, Default::default())? + }; + self.node.tick(); + self.refresh_execution_fence(); + self.finish_log_admission(admission)?; + Ok(()) + } + + pub(super) fn validate_snapshot_member(&self, member: u64) -> Result<(), RuntimeError> { + self.live()?; + if member == 0 || self.node.raft.prs().get(member).is_none() { + return Err(RuntimeError::Protocol("unknown snapshot member".into())); + } + Ok(()) + } + + pub(super) fn report_snapshot( + &mut self, + member: u64, + delivered: bool, + ) -> Result<(), RuntimeError> { + self.validate_snapshot_member(member)?; + if self.phase.is_some() || self.checkpoint.is_some() { + return Err(RuntimeError::NotReady); + } + self.node.report_snapshot( + member, + if delivered { + ::raft::SnapshotStatus::Finish + } else { + ::raft::SnapshotStatus::Failure + }, + ); + Ok(()) + } + + pub(super) fn report_unreachable(&mut self, member: u64) -> Result<(), RuntimeError> { + self.live()?; + if self.phase.is_some() || self.checkpoint.is_some() { + return Err(RuntimeError::NotReady); + } + if member == 0 || self.node.raft.prs().get(member).is_none() { + return Err(RuntimeError::Protocol("unknown unreachable member".into())); + } + self.node.report_unreachable(member); + Ok(()) + } + + /// Queue publication of the exact applied cut without changing Raft progress + /// or the applied view. A later CheckpointPublished event proves persistence. + pub fn publish_checkpoint( + &mut self, + checkpoint: PreparedLocalCellGroupCheckpoint, + ) -> Result<(), RuntimeError> { + self.live()?; + if !self.idle() || self.applying.is_some() { + return Err(RuntimeError::NotReady); + } + self.select_local_checkpoint(checkpoint) + } + + /// The controller holds this exact successor ticket without starting apply. + /// Publishing its base cannot race another applied root or snapshot install. + /// Pending Ready work may persist a later suffix; it cannot change this cut. + pub(super) fn checkpoint_before_apply( + &mut self, + ticket: &CellGroupTicket, + checkpoint: PreparedLocalCellGroupCheckpoint, + ) -> Result<(), RuntimeError> { + self.live()?; + if self.phase.is_some() || self.checkpoint.is_some() || self.waiting.is_some() { + return Err(RuntimeError::NotReady); + } + if self + .applying + .as_ref() + .is_none_or(|active| !active.matches(ticket)) + { + return Err(RuntimeError::StaleCompletion); + } + self.select_local_checkpoint(checkpoint) + } + + fn select_local_checkpoint( + &mut self, + checkpoint: PreparedLocalCellGroupCheckpoint, + ) -> Result<(), RuntimeError> { + let root = checkpoint.root(); + if root.marker() != self.current.marker() + || !Arc::ptr_eq(&root.engine_owner, &self.group.owner) + || !Arc::ptr_eq(&root.page_owner, &self.current.page_owner) + || root.marker().position().index.get() <= 1 + || root.marker().position().index.get() >= u64::MAX - 2 + || root.marker().position().term.get() >= u64::MAX - 2 + { + return Err(RuntimeError::Invalid("local checkpoint owner/cut".into())); + } + if root + .reserved_bytes() + .saturating_add(checkpoint.snapshot().compute_size() as usize) + .saturating_add(4096) + > self.options.max_ready_bytes + { + return Err(RuntimeError::Overloaded); + } + let admission = self + .group + .archive_compaction + .admit(root.marker())? + .ok_or(RuntimeError::Overloaded)?; + self.phase = Some(Phase::Checkpoint(checkpoint, false, admission)); + Ok(()) + } + + /// Admit an authenticated snapshot envelope only with its exact complete + /// receiver-local closure. This returns protocol admission, never a durable + /// acknowledgement. Snapshot messages are emitted after the Ready fence. + pub fn step_checkpoint( + &mut self, + message: Message, + checkpoint: PreparedCellGroupCheckpoint, + ) -> Result { + self.live()?; + if !self.idle() || self.applying.is_some() || !self.apply.is_empty() { + return Err(RuntimeError::NotReady); + } + self.validate_peer_message(&message, true)?; + let root = checkpoint.manifest(); + if message.get_msg_type() != MessageType::MsgSnapshot + || message.get_snapshot() != checkpoint.snapshot() + || checkpoint.marker().position().index.get() >= u64::MAX - 2 + || checkpoint.marker().position().term.get() >= u64::MAX - 2 + || message.term < self.node.raft.term + || message.term < checkpoint.marker().position().term.get() + || root.base.as_ref() != Some(self.current.marker()) + || root.marker().route() != self.current.marker().route() + || !Arc::ptr_eq(&root.engine_owner, &self.group.owner) + || !Arc::ptr_eq(&root.page_owner, &self.current.page_owner) + || !checkpoint.membership().permits_group(self.group.identity()) + || root.marker().position().index < self.current.marker().position().index + || root.marker().root().cells() > self.options.max_cells_per_group + || crate::raft::member_state::members( + checkpoint.snapshot().get_metadata().get_conf_state(), + ) + .len() + > self.options.max_members_per_group + { + return Err(RuntimeError::Invalid( + "checkpoint envelope/owner/base".into(), + )); + } + if root + .reserved_bytes() + .saturating_add(message.compute_size() as usize) + .saturating_add(4096) + > self.options.max_ready_bytes + { + return Err(RuntimeError::Overloaded); + } + let result = self.node.step(message); + self.refresh_execution_fence(); + self.finish_log_admission(Default::default())?; + result?; + if let Some(snapshot) = self.node.snap() { + if snapshot != checkpoint.snapshot() { + self.failed = Some(RuntimeError::StaleCompletion); + self.revoke_execution(); + return Err(RuntimeError::StaleCompletion); + } + self.checkpoint = Some(checkpoint); + Ok(CellGroupSnapshotAdmission::Restoring) + } else { + Ok(CellGroupSnapshotAdmission::Ignored) + } + } + + pub fn step(&mut self, message: Message) -> Result<(), RuntimeError> { + self.live()?; + if self.phase.is_some() || self.checkpoint.is_some() { + return Err(RuntimeError::NotReady); + } + if self.apply_bytes >= self.options.max_group_apply_bytes { + return Err(RuntimeError::NotReady); + } + self.validate_message(&message)?; + let entries = if message.get_msg_type() == MessageType::MsgAppend { + message.entries.as_slice() + } else { + &[] + }; + let mut admission = self.log_admission.prepare(&self.group, entries, None)?; + if self.node.raft.state != StateRole::Leader + && matches!( + message.get_msg_type(), + MessageType::MsgRequestVoteResponse + | MessageType::MsgRequestPreVoteResponse + | MessageType::MsgTimeoutNow + ) + { + admission = self + .log_admission + .election(&self.group, &self.node, admission)?; + } + let (from, term, kind) = (message.from, message.term, message.get_msg_type()); + let result = self.node.step(message); + self.refresh_execution_fence(); + self.finish_log_admission(admission)?; + result?; + if matches!(kind, MessageType::MsgAppend | MessageType::MsgHeartbeat) + && from == self.node.raft.leader_id + && term == self.node.raft.term + { + self.source_observed = Some(( + from, + term, + self.log + .domain() + .mono_ms() + .map_err(|_| RuntimeError::Closed)?, + )); + } + Ok(()) + } + + /// Propose a route/revision-bound whole-group configuration. The returned + /// position is admission only; the persisted applied root proves completion. + pub fn configure( + &mut self, + change: super::CellGroupMembershipChange, + ) -> Result { + self.live()?; + if self.node.raft.state != StateRole::Leader { + return Err(RuntimeError::NotLeader); + } + if !self.status()?.ready + || !self.apply.is_empty() + || self.applying.is_some() + || self.node.raft.pending_conf_index > self.node.raft.raft_log.applied + { + return Err(RuntimeError::NotReady); + } + let before = self.node.raft.raft_log.last_index(); + let entry = Entry { + entry_type: EntryType::EntryConfChangeV2, + index: before + 1, + term: self.node.raft.term, + data: change + .change + .write_to_bytes() + .map_err(|e| RuntimeError::Invalid(e.to_string()))? + .into(), + ..Default::default() + }; + if entry.data.len() > self.options.max_proposal_bytes { + return Err(RuntimeError::Overloaded); + } + let cache = self.group.read()?; + let next = super::membership::transition( + &entry, + self.current.marker().route(), + &cache.conf_state, + &cache.membership, + )?; + if crate::raft::member_state::members(&next.conf).len() > self.options.max_members_per_group + { + return Err(RuntimeError::Overloaded); + } + for item in &change.change.changes { + if item.get_change_type() == ConfChangeType::AddNode + && !cache.conf_state.voters.contains(&item.node_id) + && self + .node + .raft + .prs() + .get(item.node_id) + .is_none_or(|p| p.matched < cache.hard_state.commit) + { + return Err(RuntimeError::NotReady); + } + } + drop(cache); + let expected = entry.data.clone(); + let admission = + self.log_admission + .prepare(&self.group, std::slice::from_ref(&entry), None)?; + let admission = self.log_admission.configuration( + &self.group, + admission, + change.change.enter_joint() == Some(true), + )?; + let result = self.node.propose_conf_change(Vec::new(), change.change); + self.finish_log_admission(admission)?; + if let Err(error) = result { + let error = RuntimeError::from(error); + if self.node.raft.raft_log.last_index() != before { + self.failed = Some(error.clone()); + self.revoke_execution(); + } + return Err(error); + } + let result = (|| { + let observed = self + .node + .raft + .raft_log + .unstable_entries() + .last() + .ok_or_else(|| RuntimeError::Protocol("missing configuration proposal".into()))?; + if observed.index != before + 1 + || observed.get_entry_type() != EntryType::EntryConfChangeV2 + || observed.data != expected + { + return Err(RuntimeError::Protocol( + "configuration proposal changed".into(), + )); + } + Ok(CellGroupProposal { + group: self.group.identity().scope, + position: entry_position(observed), + digest: entry_digest(observed)?, + }) + })(); + if let Err(error) = &result { + self.failed = Some(error.clone()); + self.revoke_execution(); + } + result + } + + /// Protocol admission only. The shared cell executor must serialize each + /// cell's preparation, reserve physical work and retain ambiguous outcomes. + /// The first prepared submission permanently closes this byte-only entry + /// point for this owner, including already queued raw retries. + pub fn propose(&mut self, data: Vec) -> Result { + if self.prepared_only { + return Err(RuntimeError::Invalid("prepared submission required".into())); + } + self.try_propose(data, None)? + .ok_or(RuntimeError::Overloaded) + } + + /// Seals a definitively refused preparation against every delayed clone. + /// Queue admission errors never manufacture a rejection proof. + pub fn submit( + &mut self, + prepared: crate::raft::execution::CellPreparedCommand, + ) -> Result { + let engine = self.group.owner.clone(); + let group = self.group.identity().scope; + if !prepared.belongs_to(&engine, group) { + return Err(RuntimeError::Invalid("prepared submission owner".into())); + } + self.prepared_only = true; + prepared.submit(&engine, group, || { + self.live()?; + if prepared.term().get() != self.node.raft.term { + return Err(RuntimeError::StaleCompletion); + } + self.try_propose(prepared.bytes().to_vec(), Some(prepared.log_reservation()?)) + }) + } + + fn try_propose( + &mut self, + data: Vec, + reservation: Option<&Arc>, + ) -> Result, RuntimeError> { + if !self.status()?.ready { + return Err(RuntimeError::NotReady); + } + if data.len() > self.options.max_proposal_bytes { + return Err(RuntimeError::Overloaded); + } + let entry = CellGroupEntry::decode( + &data, + self.current.marker().route(), + GroupEntryLimits::default(), + ) + .map_err(|e| RuntimeError::Invalid(e.to_string()))?; + let admission = match self.proposals.check( + &entry, + &data, + &self.current, + self.options.max_cells_per_group, + self.options.max_requests, + &self.proposal_budget, + ) { + Ok(admission) => admission, + Err(RuntimeError::Overloaded) => return Ok(None), + Err(error) => return Err(error), + }; + let (candidate, permit) = match admission { + super::proposal_admission::Admission::Existing(accepted) => return Ok(Some(accepted)), + super::proposal_admission::Admission::New(candidate, permit) => (candidate, permit), + }; + let before = self.node.raft.raft_log.last_index(); + let entry = Entry { + index: before + 1, + term: self.node.raft.term, + data: data.clone().into(), + ..Default::default() + }; + let admission = + match self + .log_admission + .prepare(&self.group, std::slice::from_ref(&entry), reservation) + { + Ok(admission) => admission, + Err(RuntimeError::Overloaded) => return Ok(None), + Err(error) => return Err(error), + }; + let result = self.node.propose(Vec::new(), data); + self.finish_log_admission(admission)?; + if let Err(error) = result { + let error = RuntimeError::from(error); + if self.node.raft.raft_log.last_index() != before { + self.failed = Some(error.clone()); + self.revoke_execution(); + } + return Err(error); + } + let observed = (|| { + let entry = self + .node + .raft + .raft_log + .unstable_entries() + .last() + .ok_or_else(|| RuntimeError::Protocol("missing proposed entry".into()))?; + if entry.index != before + 1 { + return Err(RuntimeError::Protocol("proposal append position".into())); + } + Ok(CellGroupProposal { + group: self.group.identity().scope, + position: entry_position(entry), + digest: entry_digest(entry)?, + }) + })(); + let accepted = match observed { + Ok(accepted) => accepted, + Err(error) => { + // Append was attempted. Never turn an uncertain local result + // into permission to append again or a rejection proof. + self.failed = Some(error.clone()); + self.revoke_execution(); + return Err(error); + } + }; + // Track only newly appended mutations. A duplicate, rejected + // preparation or enrollment does not contribute to per-cell heat. + if let (Some(cell), Ok(now_ms), Ok(membership)) = ( + candidate.mutation_cell(), + self.log.domain().mono_ms(), + MembershipRevision::new(self.current.marker().membership_revision()), + ) { + self.write_heat + .record(accepted.position.term, membership, now_ms, cell); + } + self.proposals.accept(candidate, accepted.clone(), permit); + self.admitted_writes = self.admitted_writes.saturating_add(1); + Ok(Some(accepted)) + } + + /// Check the SQL preparation fence on the actual RawNode owner in the same + /// turn as proposal admission. A sampled status before queueing is not enough. + pub fn propose_at_term( + &mut self, + term: Term, + data: Vec, + ) -> Result { + self.live()?; + if term.get() != self.node.raft.term { + return Err(RuntimeError::StaleCompletion); + } + self.propose(data) + } + + pub fn read_index(&mut self, context: Vec) -> Result<(), RuntimeError> { + if super::runtime::reads::reserved_context(&context) { + return Err(RuntimeError::Invalid("reserved cell read context".into())); + } + self.read_cell_index(context) + } + + pub(super) fn read_cell_index(&mut self, context: Vec) -> Result<(), RuntimeError> { + if !self.status()?.ready { + return Err(RuntimeError::NotReady); + } + if context.is_empty() || context.len() > 4096 { + return Err(RuntimeError::Invalid("read context".into())); + } + if self.node.raft.read_only.pending_read_count() >= self.options.max_pending_reads { + return Err(RuntimeError::Overloaded); + } + self.node.read_index(context); + Ok(()) + } + + /// Business reads have their own authenticated forwarding context; seed + /// contexts remain leader-local and cannot enter this path. + pub(super) fn read_forwarded_cell_index( + &mut self, + context: Vec, + ) -> Result<(), RuntimeError> { + if super::runtime::reads::cell_origin(&context) != Some(self.membership_read_origin()) { + return Err(RuntimeError::Invalid("cell read origin".into())); + } + if !self.membership_read_ready()? + || (self.node.raft.state == StateRole::Leader && !self.status()?.ready) + { + return Err(RuntimeError::NotReady); + } + if self.node.raft.read_only.pending_read_count() >= self.options.max_pending_reads { + return Err(RuntimeError::Overloaded); + } + self.node.read_index(context); + Ok(()) + } + + pub(super) fn membership_read_origin(&self) -> u64 { + self.group.identity().member.get() + } + + /// Safe forwarding requires an active coherent member and known leader. + /// This does not change leader-only mutation or seed readiness. + pub(super) fn membership_read_ready(&self) -> Result { + self.require_application_active()?; + let cache = self.group.read()?; + let member = self.group.identity().member.get(); + let conf = &cache.conf_state; + // Retained history still binds its old identity. A replica removed + // from the durable configuration cannot complete a quorum read. + if !conf.voters.contains(&member) + && !conf.voters_outgoing.contains(&member) + && !conf.learners.contains(&member) + && !conf.learners_next.contains(&member) + { + return Err(RuntimeError::NotFound); + } + // Leaders drop ReadIndex before committing their first entry in the + // current term. Local durable apply proves that point on followers too. + Ok(matches!( + self.node.raft.state, + StateRole::Leader | StateRole::Follower + ) && self.node.raft.leader_id != 0 + && self.node.raft.term > 0 + && self.phase.is_none() + && self.checkpoint.is_none() + && cache.hard_state.term == self.node.raft.term + && self.current.marker().position().term.get() == self.node.raft.term + && cache.group_manifest.as_ref() == Some(self.current.marker()) + && cache.membership.permits_group(self.group.identity())) + } + + pub(super) fn read_membership_index(&mut self, context: Vec) -> Result<(), RuntimeError> { + if super::runtime::reads::membership_origin(&context) != Some(self.membership_read_origin()) + { + return Err(RuntimeError::Invalid("membership read origin".into())); + } + if !self.membership_read_ready()? { + return Err(RuntimeError::NotReady); + } + if self.node.raft.read_only.pending_read_count() >= self.options.max_pending_reads { + return Err(RuntimeError::Overloaded); + } + self.node.read_index(context); + Ok(()) + } + + fn park_fenced(&self) -> bool { + self.pending_park.is_some() || self.parked.is_some() || self.waking_after.is_some() + } + + pub(super) fn check_park_wake(&self, generation: ParkGeneration) -> Result<(), RuntimeError> { + self.live()?; + let marker = self.current.marker(); + if !marker.is_active() + || self.parked.is_some_and(|current| current != generation) + || marker + .park_cut() + .is_some_and(|cut| cut.generation() > generation.get()) + || self.pending_park.as_ref().is_some_and(|pending| { + pending.preparation.proposal.position.index.get() > generation.get() + }) + { + return Err(RuntimeError::StaleCompletion); + } + Ok(()) + } + + pub(super) fn is_parked(&self) -> bool { + self.parked.is_some() + } + + pub(super) fn parked_term(&self) -> Option { + self.parked.map(|_| self.node.raft.term) + } + + pub(super) fn parked_effect_deadline(&self) -> Option { + self.parked?; + self.current.marker().root().next_due_unix_millis() + } + + pub(super) fn begin_park( + &mut self, + certificate: &ParkCertificate, + ) -> Result<(Box, usize), RuntimeError> { + if self.parked.is_none() { + self.require_park_acknowledgement(certificate.leader()?, certificate.cut())?; + if Arc::strong_count(&self.leader_term) != 1 + || self + .deliveries + .iter() + .any(|delivery| delivery.strong_count() != 0) + { + return Err(RuntimeError::NotReady); + } + } else if !self.idle() { + return Err(RuntimeError::NotReady); + } + let voter = self.group.identity(); + self.group + .read()? + .admit_park(certificate, voter.member, voter.node_identity())?; + self.persist_park_change(LocalParkChange::Park(Box::new(certificate.clone()))) + } + + pub(super) fn finish_park( + &mut self, + certificate: &ParkCertificate, + ) -> Result<(), RuntimeError> { + let cache = self.group.read()?; + if !cache + .local_park + .as_ref() + .is_some_and(|park| park.parked() && park.certificate() == Some(certificate)) + { + return Err(RuntimeError::StaleCompletion); + } + drop(cache); + let generation = ParkGeneration::new(certificate.cut().generation()) + .map_err(|_| RuntimeError::StaleCompletion)?; + self.parked = Some(generation); + self.pending_park = None; + self.leader_term.store(0, Ordering::Release); + self.leader_term = Arc::new(AtomicU64::new(0)); + self.source_observed = None; + self.advance_source_epoch(); + Ok(()) + } + + pub(super) fn begin_wake(&mut self) -> Result<(Box, usize), RuntimeError> { + self.live()?; + if !self.idle() { + return Err(RuntimeError::NotReady); + } + let generation = self.parked.ok_or(RuntimeError::NotReady)?; + self.persist_park_change(LocalParkChange::Close(generation)) + } + + pub(super) fn finish_wake( + &mut self, + generation: ParkGeneration, + campaign: bool, + ) -> Result<(), RuntimeError> { + let floor = LogIndex::new(generation.get()); + if self.parked != Some(generation) + || self + .group + .read()? + .local_park + .as_ref() + .and_then(LocalParkRecord::wake_floor) + != Some(floor) + { + return Err(RuntimeError::StaleCompletion); + } + self.waking_after = Some(floor); + self.parked = None; + // Clear soft leadership without changing the retained term or vote. + // The next election, or a live leader's later entry, supplies authority. + self.node.raft.become_follower(self.node.raft.term, 0); + self.refresh_execution_fence(); + if campaign { + match self.campaign() { + Ok(()) | Err(RuntimeError::NotReady | RuntimeError::Overloaded) => (), + Err(error) => return Err(error), + } + } + Ok(()) + } + + fn require_park_acknowledgement( + &self, + leader: MemberId, + cut: &GroupParkCut, + ) -> Result<(), RuntimeError> { + self.live()?; + let position = cut.position(); + if self.node.raft.state == StateRole::Leader { + let now = self + .log + .domain() + .mono_ms() + .map_err(|_| RuntimeError::Closed)?; + if self.pending_park.as_ref().is_none_or(|pending| { + pending.preparation.proposal.position != position + || pending.abort.is_some() + || now >= pending.request.deadline_ms + }) { + return Err(RuntimeError::NotReady); + } + } + if !self.idle() + || self.parked.is_some() + || self.waking_after.is_some() + || self.pending_seal.is_some() + || self.node.raft.leader_id != leader.get() + || self.node.raft.term != position.term.get() + || self.node.raft.raft_log.last_index() != position.index.get() + || self.node.raft.raft_log.committed != position.index.get() + || self.node.raft.raft_log.applied != position.index.get() + || self.current.marker().park_cut() != Some(cut) + || self.applying.is_some() + || !self.apply.is_empty() + || self.node.raft.read_only.pending_read_count() != 0 + { + return Err(RuntimeError::NotReady); + } + let cache = self.group.read()?; + let voter = self.group.identity(); + if cache.group_manifest.as_ref() != Some(self.current.marker()) + || !cache.conf_state.voters.contains(&leader.get()) + { + return Err(RuntimeError::NotReady); + } + cache.admit_park_acknowledgement(cut, voter.member, voter.process)?; + Ok(()) + } + + pub(super) fn begin_park_acknowledgement( + &mut self, + leader: MemberId, + cut: &GroupParkCut, + ) -> Result<(Box, usize), RuntimeError> { + self.require_park_acknowledgement(leader, cut)?; + self.persist_park_change(LocalParkChange::Acknowledge(cut.clone())) + } + + pub(super) fn begin_park_certificate( + &mut self, + certificate: &ParkCertificate, + ) -> Result<(Box, usize), RuntimeError> { + self.require_park_certificate(certificate)?; + self.persist_park_change(LocalParkChange::Certify(Box::new(certificate.clone()))) + } + + fn persist_park_change( + &mut self, + change: LocalParkChange, + ) -> Result<(Box, usize), RuntimeError> { + let mut write = GroupWrite::new(&self.group); + write.local_park = Some(change); + let bytes = write.estimated_bytes()?; + let work = self.prepare_persistence(write)?; + Ok((work, bytes)) + } + + fn require_park_certificate(&self, certificate: &ParkCertificate) -> Result<(), RuntimeError> { + if self.node.raft.state != StateRole::Leader { + return Err(RuntimeError::NotLeader); + } + self.require_park_acknowledgement(certificate.leader()?, certificate.cut())?; + let voter = self.group.identity(); + self.group.read()?.admit_park_certificate( + certificate, + voter.member, + voter.node_identity(), + )?; + Ok(()) + } + + pub(super) fn finish_park_certificate( + &self, + certificate: &ParkCertificate, + ) -> Result { + if self.is_parked() { + self.live()?; + if self.last_sync == 0 { + return Err(RuntimeError::StaleCompletion); + } + } else { + self.require_park_certificate(certificate)?; + } + if self + .group + .read()? + .local_park + .as_ref() + .and_then(|park| park.certificate()) + != Some(certificate) + { + return Err(RuntimeError::StaleCompletion); + } + Ok(SelectedParkCertificate { + certificate: Box::new(certificate.clone()), + sync_sequence: self.last_sync, + }) + } + + pub(super) fn finish_park_acknowledgement( + &self, + leader: MemberId, + cut: GroupParkCut, + ) -> Result { + self.require_park_acknowledgement(leader, &cut)?; + let voter = self.group.identity().clone(); + let cache = self.group.read()?; + if !cache + .local_park + .as_ref() + .is_some_and(|park| park.matches_acknowledgement(&cut, voter.process)) + { + return Err(RuntimeError::StaleCompletion); + } + Ok(ParkAcknowledgement { + voter, + leader, + cut, + sync_sequence: self.last_sync, + }) + } + + pub fn complete_persist(&mut self, completed: CellGroupPersisted) -> Result<(), RuntimeError> { + self.live()?; + if self + .waiting + .as_ref() + .is_none_or(|t| !t.matches(&completed.ticket)) + || completed.sequence <= self.last_sync + { + return Err(RuntimeError::StaleCompletion); + } + self.last_sync = completed.sequence; + self.waiting = None; + Ok(()) + } + + pub fn complete_apply( + &mut self, + ticket: CellGroupTicket, + next: PreparedGroupManifest, + ) -> Result<(), RuntimeError> { + self.live()?; + if self.phase.is_some() || self.checkpoint.is_some() { + return Err(RuntimeError::NotReady); + } + if self.applying.as_ref().is_none_or(|t| !t.matches(&ticket)) + || next.marker().position() != ticket.position + || next.base.as_ref() != Some(self.current.marker()) + || next.marker().route() != self.current.marker().route() + || !Arc::ptr_eq(&next.engine_owner, &self.group.owner) + || !Arc::ptr_eq(&next.page_owner, &self.current.page_owner) + { + return Err(RuntimeError::StaleCompletion); + } + let pending = self.apply.front().ok_or(RuntimeError::StaleCompletion)?; + self.validate_apply(&pending.entry, &next)?; + self.phase = Some(Phase::Apply(ticket, Box::new(next), false)); + Ok(()) + } + + /// Return one bounded effect. No Ready-dependent message or applied event is + /// released until its exact physical persistence completion is consumed. + pub fn poll(&mut self) -> Result, RuntimeError> { + self.poll_budget(true, true) + } + + pub(super) fn poll_budget( + &mut self, + collect: bool, + emit: bool, + ) -> Result, RuntimeError> { + self.live()?; + let result = self.poll_inner(collect, emit); + if let Err(error) = &result { + self.failed = Some(error.clone()); + self.revoke_execution(); + } + result + } + + pub(super) fn idle(&self) -> bool { + self.phase.is_none() + && self.checkpoint.is_none() + && self.waiting.is_none() + && self.effects.is_empty() + && !self.node.has_ready() + } + + pub(super) fn waiting(&self) -> bool { + self.waiting.is_some() + } + + pub(super) fn control_pending(&self) -> bool { + self.phase.is_some() + } + + pub(super) fn has_work(&self) -> bool { + !self.effects.is_empty() + || self.phase.is_some() + || self.node.has_ready() + || (self.applying.is_none() && !self.apply.is_empty()) + } + + pub(super) fn usage(&self) -> (usize, usize, usize) { + let entries = |entries: &[Entry]| { + entries + .iter() + .map(|e| e.compute_size() as usize + 128) + .sum::() + }; + let messages = |messages: &[Message]| { + messages + .iter() + .map(|m| m.compute_size() as usize + 128) + .sum::() + }; + let ready = match &self.phase { + Some(Phase::Ready(ready, _)) => { + entries(ready.entries()) + + entries(ready.committed_entries()) + + messages(ready.messages()) + + messages(ready.persisted_messages()) + + ready + .read_states() + .iter() + .map(|r| r.request_ctx.len() + 128) + .sum::() + } + Some(Phase::Light(light)) => { + entries(light.committed_entries()) + messages(light.messages()) + } + Some(Phase::Apply(_, next, _)) => next.reserved_bytes() + 4096, + Some(Phase::Checkpoint(checkpoint, _, _)) => { + checkpoint.root().reserved_bytes() + + checkpoint.snapshot().compute_size() as usize + + 4096 + } + None => 0, + }; + ( + ready + + self.checkpoint.as_ref().map_or(0, |c| { + c.manifest().reserved_bytes() + c.snapshot().compute_size() as usize + }) + + self + .effects + .iter() + .map(CellGroupEffect::bytes) + .sum::(), + self.apply_bytes, + self.node.raft.read_only.pending_read_count(), + ) + } + + fn live(&self) -> Result<(), RuntimeError> { + if let Some(error) = &self.failed { + return Err(error.clone()); + } + if self.group.closed.load(Ordering::Acquire) { + return Err(RuntimeError::Closed); + } + if self.node.raft.term >= u64::MAX - 2 + || self.node.raft.raft_log.last_index() >= u64::MAX - 2 + { + return Err(RuntimeError::Overloaded); + } + Ok(()) + } + + fn ticket( + &mut self, + position: LogPosition, + digest: [u8; 32], + ) -> Result { + self.next_ticket = self + .next_ticket + .checked_add(1) + .ok_or(RuntimeError::Overloaded)?; + Ok(CellGroupTicket { + lease: self.lease.clone(), + sequence: self.next_ticket, + position, + digest, + admission: None, + preparation: None, + }) + } + + fn persistence( + &mut self, + write: GroupWrite, + ) -> Result { + self.prepare_persistence(write) + .map(CellGroupEffect::Persist) + } + + fn prepare_persistence( + &mut self, + mut write: GroupWrite, + ) -> Result, RuntimeError> { + let ticket = self.ticket(LogPosition::default(), [0; 32])?; + write.log_reservations = self.log_admission.persistence(&write.entries)?; + write.consensus_pin = Some(self.lease.clone()); + self.waiting = Some(ticket.clone()); + Ok(Box::new(CellGroupPersist { + log: self.log.clone(), + ticket, + write, + })) + } + + fn enqueue(&mut self, entries: Vec) -> Result<(), RuntimeError> { + if entries.is_empty() { + return Ok(()); + } + let queued_at_ms = self + .log + .domain() + .mono_ms() + .map_err(|_| RuntimeError::Closed)?; + let mut previous = self + .apply + .back() + .map_or(self.current.marker().position().index.get(), |pending| { + pending.entry.index + }); + let committed = self.group.initial_state()?.hard_state.commit; + for entry in entries { + if entry.index != previous + 1 || entry.index > committed { + return Err(RuntimeError::Protocol( + "noncontiguous/unpersisted group apply".into(), + )); + } + previous = entry.index; + self.apply_bytes = self + .apply_bytes + .checked_add(entry.compute_size() as usize + 128) + .ok_or(RuntimeError::Overloaded)?; + // Ready and its LightReady can each retain one oversize entry. + // Collection stops at the soft allowance; these two bounded + // in-flight batches must be drained, not treated as data failure. + if self.apply_bytes + > self.options.max_group_apply_bytes + 2 * self.options.max_message_bytes + 256 + { + return Err(RuntimeError::Overloaded); + } + self.apply.push_back(PendingApply { + entry, + queued_at_ms, + }); + } + Ok(()) + } + + fn messages(&mut self, messages: Vec) -> Result<(), RuntimeError> { + if self.effects.len() + messages.len() > self.options.max_events { + return Err(RuntimeError::Overloaded); + } + let existing: usize = self + .effects + .iter() + .map(|event| match event { + CellGroupEffect::Message(message) => message.compute_size() as usize + 128, + CellGroupEffect::ReadIndex { context, .. } => context.len() + 128, + _ => 4096, + }) + .sum(); + let added: usize = messages + .iter() + .map(|m| m.compute_size() as usize + 128) + .sum(); + if existing.saturating_add(added) > self.options.max_event_bytes { + return Err(RuntimeError::Overloaded); + } + self.effects + .extend(messages.into_iter().map(CellGroupEffect::Message)); + Ok(()) + } + + /// Register the exact output admission before exposing a Message. A physical + /// sender retains it independently of its caller's response waiter. + pub(super) fn track_delivery(&mut self, permit: &Arc) { + self.deliveries + .retain(|delivery| delivery.strong_count() != 0); + self.deliveries.push(Arc::downgrade(permit)); + } + + fn announce_configuration(&mut self, index: u64) -> Result<(), RuntimeError> { + if self + .configuration_fence + .as_ref() + .is_some_and(|f| f.index == index && f.announced) + { + return Ok(()); + } + self.configuration_fence = Some(ConfigurationFence { + index, + announced: true, + }); + if self.node.raft.state != StateRole::Leader { + return Ok(()); + } + let hard = self.group.initial_state()?.hard_state; + if hard.term != self.node.raft.term || hard.vote != self.node.raft.vote { + return Ok(()); + } + let entry = self + .apply + .front() + .map(|pending| &pending.entry) + .filter(|e| e.index == index) + .ok_or_else(|| RuntimeError::Protocol("missing configuration apply".into()))?; + let messages = self + .node + .raft + .prs() + .iter() + .filter(|(member, _)| **member != self.node.raft.id) + .map(|(&member, _)| Message { + msg_type: MessageType::MsgAppend, + from: self.node.raft.id, + to: member, + term: self.node.raft.term, + index, + log_term: entry.term, + commit: index, + ..Default::default() + }) + .collect(); + // Paused peers may miss the normal commit broadcast. Announce this exact + // committed index/term before removing their old route. Acceptance is + // advisory; the catalog still requires actual physical closure. + self.messages(messages) + } + + fn poll_inner( + &mut self, + collect: bool, + emit: bool, + ) -> Result, RuntimeError> { + loop { + if self.waiting.is_some() { + return Ok(if emit { self.effects.pop_front() } else { None }); + } + match self.phase.take() { + Some(Phase::Ready(mut ready, term)) => { + let installed = if !ready.snapshot().is_empty() { + let checkpoint = self + .checkpoint + .take() + .ok_or(RuntimeError::StaleCompletion)?; + if checkpoint.snapshot() != ready.snapshot() + || self.group.group_manifest()?.as_ref() != Some(checkpoint.marker()) + || self.group.checkpoint_manifest()?.as_ref() + != Some(checkpoint.marker()) + { + return Err(RuntimeError::StaleCompletion); + } + checkpoint + .manifest() + .require_checkpoint(&self.group) + .map_err(|e| RuntimeError::Invalid(e.to_string()))?; + self.current = checkpoint.manifest().clone(); + if self.waking_after.is_some() { + self.refresh_execution_fence(); + } + self.configuration_fence = None; + Some(self.current.marker().position().index.get()) + } else { + None + }; + self.messages(ready.take_messages())?; + self.messages(ready.take_persisted_messages())?; + let reads = ready.take_read_states(); + let queued: usize = self + .effects + .iter() + .map(|event| match event { + CellGroupEffect::Message(message) => { + message.compute_size() as usize + 128 + } + _ => 4096, + }) + .sum(); + let read_bytes: usize = reads.iter().map(|r| r.request_ctx.len() + 128).sum(); + if self.effects.len() + reads.len() > self.options.max_events + || queued.saturating_add(read_bytes) > self.options.max_event_bytes + { + return Err(RuntimeError::Overloaded); + } + for read in reads { + self.effects.push_back(CellGroupEffect::ReadIndex { + term, + index: LogIndex::new(read.index), + context: read.request_ctx.to_vec(), + }); + } + self.enqueue(ready.take_committed_entries())?; + let light = self.node.advance_append(*ready); + self.finish_log_admission(Default::default())?; + if let Some(index) = installed { + self.node.advance_apply_to(index); + self.proposals.clear(); + if self.effects.len() >= self.options.max_events + || self + .effects + .iter() + .map(CellGroupEffect::bytes) + .sum::() + .saturating_add(5120) + > self.options.max_event_bytes + { + return Err(RuntimeError::Overloaded); + } + self.effects + .push_back(CellGroupEffect::CheckpointInstalled { + manifest: Box::new(self.current.marker().clone()), + }); + } + let commit = light.commit_index(); + self.phase = Some(Phase::Light(light)); + if let Some(commit) = commit { + let mut write = GroupWrite::new(&self.group); + let mut hard = self.group.initial_state()?.hard_state; + hard.commit = commit; + write.hard_state = Some(hard); + return self.persistence(write).map(Some); + } + } + Some(Phase::Light(mut light)) => { + self.messages(light.take_messages())?; + self.enqueue(light.take_committed_entries())?; + } + Some(Phase::Checkpoint(checkpoint, submitted, admission)) => { + if !submitted { + let mut write = GroupWrite::new(&self.group); + write.snapshot = + Some(SnapshotChange::CellGroupCheckpoint(checkpoint.clone())); + write.checkpoint_admission = Some(admission.clone()); + self.phase = Some(Phase::Checkpoint(checkpoint, true, admission)); + return self.persistence(write).map(Some); + } + let cache = self.group.read()?; + if cache.snapshot != *checkpoint.snapshot() + || cache.checkpoint_manifest.as_ref() != Some(checkpoint.marker()) + || cache.group_manifest.as_ref() != Some(self.current.marker()) + { + return Err(RuntimeError::StaleCompletion); + } + drop(cache); + self.current + .require_checkpoint(&self.group) + .map_err(|e| RuntimeError::Invalid(e.to_string()))?; + return Ok(Some(CellGroupEffect::CheckpointPublished { + manifest: Box::new(checkpoint.marker().clone()), + })); + } + Some(Phase::Apply(mut ticket, next, submitted)) => { + if !submitted { + let mut write = GroupWrite::new(&self.group); + write.group_manifest = Some((*next).clone()); + let entry = &self + .apply + .front() + .ok_or(RuntimeError::StaleCompletion)? + .entry; + if entry.get_entry_type() == EntryType::EntryConfChangeV2 { + let cache = self.group.read()?; + let transition = super::membership::transition( + entry, + self.current.marker().route(), + &cache.conf_state, + &cache.membership, + )?; + write.conf_state = Some(transition.conf); + write.membership = Some(transition.membership); + } + self.phase = Some(Phase::Apply(ticket, next, true)); + return self.persistence(write).map(Some); + } + if self.group.group_manifest()?.as_ref() != Some(next.marker()) { + return Err(RuntimeError::StaleCompletion); + } + let entry = &self + .apply + .front() + .ok_or(RuntimeError::StaleCompletion)? + .entry; + if entry.get_entry_type() == EntryType::EntryConfChangeV2 { + let change = + super::membership::decode(entry, self.current.marker().route())?; + let conf = self.node.apply_conf_change(&change)?; + // RawNode retains its historical insertion order while + // pure replay restores the persisted configuration first. + // Compare validated sets, including joint/auto-leave + // state, rather than hash iteration order after a move. + let durable = self.group.initial_state()?.conf_state; + if crate::raft::manifest::encode_conf(&conf) + .map_err(|e| RuntimeError::Invalid(e.to_string()))? + != crate::raft::manifest::encode_conf(&durable) + .map_err(|e| RuntimeError::Invalid(e.to_string()))? + { + return Err(RuntimeError::StaleCompletion); + } + self.refresh_execution_fence(); + self.configuration_fence = None; + } + let last = self.node.raft.raft_log.last_index(); + let automatic = self.node.raft.state == StateRole::Leader + && *self.node.raft.prs().conf().auto_leave() + && self.node.raft.raft_log.applied <= self.node.raft.pending_conf_index + && ticket.position.index.get() >= self.node.raft.pending_conf_index; + let admission = self.log_admission.applying(automatic)?; + self.node.advance_apply_to(ticket.position.index.get()); + self.finish_log_admission(admission)?; + if self.node.raft.raft_log.last_index() > last { + // Raft appended an automatic joint-leave entry. Drive + // replication without waiting for a later write/tick; + // Ready still fences messages behind its log sync. + self.node.raft.bcast_append(); + } + let entry = self + .apply + .pop_front() + .ok_or(RuntimeError::StaleCompletion)?; + self.apply_bytes -= entry.entry.compute_size() as usize + 128; + self.applying = None; + // A fresh installed leaf has this exact local Raft + // position. Metadata entries and snapshots do not. + if next.marker().root().last_mutation() == Some(ticket.position) { + self.applied_writes = self.applied_writes.saturating_add(1); + } + self.current = *next; + if self + .pending_park + .as_ref() + .is_some_and(|pending| pending.abort == Some(ticket.position)) + { + self.pending_park = None; + self.refresh_execution_fence(); + } else if self.waking_after.is_some() { + self.refresh_execution_fence(); + } + self.proposals.applied(ticket.position); + // Physical preparation and publication have settled. Its + // reservation must not deadlock admission of Applied. + ticket.admission = None; + return Ok(Some(CellGroupEffect::Applied { + ticket, + manifest: Box::new(self.current.marker().clone()), + })); + } + None => { + // Complete already-persisted protocol phases even when + // output admission is full. Otherwise held Apply events + // can prevent the completion which releases their quota. + if emit { + if let Some(event) = self.effects.pop_front() { + return Ok(Some(event)); + } + } else { + return Ok(None); + } + // Include one already-pending Ready, then freeze the finite + // send boundary. Later heartbeat work cannot starve Apply. + // An admitted snapshot keeps its existing priority. + let configuration_index = self + .apply + .front() + .map(|pending| &pending.entry) + .filter(|e| e.get_entry_type() == EntryType::EntryConfChangeV2) + .filter(|_| self.node.raft.raft_log.unstable_snapshot().is_none()) + .map(|e| e.index); + let fenced = self + .configuration_fence + .as_ref() + .is_some_and(|f| configuration_index == Some(f.index)); + if !fenced + && collect + && self.node.has_ready() + && self.apply_bytes < self.options.max_group_apply_bytes + { + if let Some(index) = configuration_index { + self.configuration_fence = Some(ConfigurationFence { + index, + announced: false, + }); + } + let ready = self.node.ready(); + if !ready.snapshot().is_empty() + && self + .checkpoint + .as_ref() + .is_none_or(|c| c.snapshot() != ready.snapshot()) + { + return Err(RuntimeError::Invalid( + "group snapshot is not attached".into(), + )); + } + let size: usize = ready + .entries() + .iter() + .chain(ready.committed_entries()) + .map(|e| e.compute_size() as usize + 128) + .sum::() + + ready + .messages() + .iter() + .chain(ready.persisted_messages()) + .map(|m| m.compute_size() as usize + 128) + .sum::() + + ready + .read_states() + .iter() + .map(|r| r.request_ctx.len() + 128) + .sum::() + + ready.snapshot().compute_size() as usize + + self + .checkpoint + .as_ref() + .map_or(0, |c| c.manifest().reserved_bytes()); + if size > self.options.max_ready_bytes { + return Err(RuntimeError::Overloaded); + } + let changed = !ready.entries().is_empty() + || ready.hs().is_some() + || !ready.snapshot().is_empty(); + let mut write = GroupWrite::new(&self.group); + write.entries = ready.entries().clone(); + write.hard_state = ready.hs().cloned(); + if !ready.snapshot().is_empty() { + write.snapshot = Some(SnapshotChange::CellGroup( + self.checkpoint + .as_ref() + .ok_or(RuntimeError::StaleCompletion)? + .clone(), + )); + } + self.phase = Some(Phase::Ready( + Box::new(ready), + Term::new(self.node.raft.term), + )); + if changed { + return self.persistence(write).map(Some); + } + continue; + } + if self.applying.is_some() { + return Ok(None); + } + if let Some(index) = configuration_index { + if self + .configuration_fence + .as_ref() + .is_none_or(|f| f.index != index) + { + self.configuration_fence = Some(ConfigurationFence { + index, + announced: false, + }); + } + self.deliveries + .retain(|delivery| delivery.strong_count() != 0); + if !self.deliveries.is_empty() { + return Ok(None); + } + self.announce_configuration(index)?; + if !self.effects.is_empty() { + continue; + } + } + let Some(entry) = self.apply.front().map(|pending| pending.entry.clone()) + else { + return Ok(None); + }; + let ticket = self.ticket(entry_position(&entry), entry_digest(&entry)?)?; + self.applying = Some(ticket.clone()); + return Ok(Some(CellGroupEffect::Apply(Box::new(CellGroupApply { + ticket, + entry, + base: self.current.clone(), + })))); + } + } + } + } + + fn validate_apply( + &self, + entry: &Entry, + next: &PreparedGroupManifest, + ) -> Result<(), RuntimeError> { + let invalid = || RuntimeError::Invalid("group apply does not match committed entry".into()); + if !next.marker().preserves_transfer(self.current.marker()) { + return Err(invalid()); + } + let same_transfer = + next.marker().transfer_origin() == self.current.marker().transfer_origin(); + let same_recovery = next.marker().recovery == self.current.marker().recovery; + let same_seal = next.marker().source_seal() == self.current.marker().source_seal(); + let same_park = next.marker().park_cut() == self.current.marker().park_cut(); + if entry.get_entry_type() == EntryType::EntryConfChangeV2 { + if !self.current.marker().is_active() + || !same_recovery + || !same_seal + || !same_transfer + || !same_park + { + return Err(invalid()); + } + let cache = self.group.read()?; + super::membership::transition( + entry, + self.current.marker().route(), + &cache.conf_state, + &cache.membership, + )?; + return if next.marker().root() == self.current.marker().root() + && next.marker().membership_revision() == entry.index + { + Ok(()) + } else { + Err(invalid()) + }; + } + if entry.get_entry_type() != EntryType::EntryNormal + || next.marker().membership_revision() != self.current.marker().membership_revision() + { + return Err(invalid()); + } + if entry.data.is_empty() { + return if next.marker().root() == self.current.marker().root() + && same_recovery + && same_seal + && same_transfer + && same_park + { + Ok(()) + } else { + Err(invalid()) + }; + } + let decoded = CellGroupEntry::decode( + &entry.data, + self.current.marker().route(), + GroupEntryLimits::default(), + ) + .map_err(|_| invalid())?; + let retirement = matches!(&decoded.command, GroupCommand::Retire { .. }); + if !matches!(&decoded.command, GroupCommand::PreparePark { .. }) && !same_park { + return Err(invalid()); + } + if !matches!(&decoded.command, GroupCommand::Activate { .. }) + && ((!self.current.marker().is_active() && !retirement) + || !same_recovery + || !same_transfer) + { + return Err(invalid()); + } + if !matches!(&decoded.command, GroupCommand::Seal { .. }) && !retirement && !same_seal { + return Err(invalid()); + } + let witness = cellar_logic::cell_group::CommitWitness { + group: self.current.marker().route().history, + position: entry_position(entry), + proposal: decoded.binding.proposal, + }; + match decoded.command { + GroupCommand::PreparePark { + generation, + term, + membership, + } => { + let mut expected = self.current.marker().clone(); + expected.position = entry_position(entry); + expected + .prepare_park(generation, term, membership) + .map_err(|_| invalid())?; + if next.marker() != &expected { + return Err(invalid()); + } + } + GroupCommand::Activate { + transition, + publication, + manifest, + } => { + if decoded.binding.proposal.bytes() != transition.bytes() { + return Err(invalid()); + } + let mut expected = self.current.marker().clone(); + expected.position = entry_position(entry); + expected + .activate(transition, publication, manifest) + .map_err(|_| invalid())?; + if next.marker() != &expected { + return Err(invalid()); + } + } + GroupCommand::Seal { + transition, + authorization, + .. + } => { + if decoded.binding.proposal.bytes() != transition.bytes() { + return Err(invalid()); + } + use sha2::{Digest, Sha256}; + let mut expected = self.current.marker().clone(); + expected.position = entry_position(entry); + expected + .seal_source( + transition, + authorization, + Sha256::digest(&entry.data).into(), + ) + .map_err(|_| invalid())?; + if next.marker() != &expected { + return Err(invalid()); + } + } + GroupCommand::Retire { + transition, + publication, + } => { + if decoded.binding.proposal.bytes() != transition.bytes() { + return Err(invalid()); + } + let mut expected = self.current.marker().clone(); + expected.position = entry_position(entry); + expected + .retire_source(transition, publication) + .map_err(|_| invalid())?; + if next.marker() != &expected { + return Err(invalid()); + } + } + GroupCommand::Enroll { ref cell, .. } => { + let old = self.current.pin_cell(&cell.cell); + let decision = super::decide_enrollment( + self.current.marker().route(), + super::EnrollmentPhase::Active, + &entry.data, + old.as_ref().map(|(leaf, _)| &leaf.enrollment), + GroupEntryLimits::default(), + ); + match decision { + Ok(super::EnrollmentDecision::Propose(record)) => { + let (leaf, _) = next.pin_cell(&cell.cell).ok_or_else(invalid)?; + if leaf.enrollment != record + || leaf.version.sequence.get() != 1 + || leaf.version.image != record.initial() + || leaf.last_mutation != entry_position(entry) + || leaf.witness != witness + { + return Err(invalid()); + } + } + Ok(super::EnrollmentDecision::Existing(_)) + | Err(super::EnrollmentError::Conflict) => { + if next.marker().root() != self.current.marker().root() { + return Err(invalid()); + } + } + Err(_) => return Err(invalid()), + } + } + GroupCommand::Mutation(mutation) => { + let (old, _) = self + .current + .pin_cell(&mutation.cell.cell) + .ok_or_else(invalid)?; + mutation.check_base(&old.version).map_err(|_| invalid())?; + let (new, _) = next.pin_cell(&mutation.cell.cell).ok_or_else(invalid)?; + if new.version.history != mutation.cell + || new.version.sequence != mutation.sequence + || new.version.image != mutation.result.into() + || new.enrollment != old.enrollment + || new.last_mutation != entry_position(entry) + || new.witness != witness + { + return Err(invalid()); + } + } + _ => return Err(invalid()), + } + Ok(()) + } + + pub(super) fn validate_message(&self, message: &Message) -> Result<(), RuntimeError> { + self.validate_peer_message(message, false) + } + + fn validate_peer_message( + &self, + message: &Message, + checkpoint: bool, + ) -> Result<(), RuntimeError> { + let invalid = || RuntimeError::Invalid("cell-group peer message".into()); + let kind = message.get_msg_type(); + if message.to != self.group.identity().member.get() + || message.from == message.to + || message.from == 0 + || self.node.raft.prs().get(message.from).is_none() + || ::raft::raw_node::is_local_msg(kind) + || matches!( + kind, + MessageType::MsgPropose | MessageType::MsgTransferLeader + ) + || (!checkpoint + && (kind == MessageType::MsgSnapshot || !message.get_snapshot().is_empty())) + || message.entries.len() > 65_536 + || message.compute_size() as usize > self.options.max_message_bytes + || [ + message.term, + message.log_term, + message.index, + message.commit, + message.reject_hint, + message.commit_term, + message.request_snapshot, + ] + .iter() + .any(|&n| n >= u64::MAX - 2) + { + return Err(invalid()); + } + match kind { + MessageType::MsgAppend => { + let mut index = message.index; + let mut term = message.log_term; + if (index == 0) != (term == 0) { + return Err(invalid()); + } + for entry in &message.entries { + index = index.checked_add(1).ok_or_else(invalid)?; + if entry.index != index + || entry.index >= u64::MAX - 2 + || entry.term == 0 + || entry.term < term + || entry.term > message.term + || !matches!( + entry.get_entry_type(), + EntryType::EntryNormal | EntryType::EntryConfChangeV2 + ) + || entry.data.len() > self.options.max_proposal_bytes + { + return Err(invalid()); + } + if entry.get_entry_type() == EntryType::EntryConfChangeV2 { + super::membership::decode(entry, self.current.marker().route())?; + } + term = entry.term; + } + } + MessageType::MsgReadIndex | MessageType::MsgReadIndexResp => { + if message.entries.len() != 1 { + return Err(invalid()); + } + let entry = &message.entries[0]; + if entry.data.is_empty() + || entry.data.len() > 4096 + || entry.index != 0 + || entry.term != 0 + || !entry.context.is_empty() + || entry.get_entry_type() != EntryType::EntryNormal + { + return Err(invalid()); + } + if super::runtime::reads::reserved_context(&entry.data) { + let origin = super::runtime::reads::membership_origin(&entry.data) + .or_else(|| super::runtime::reads::cell_origin(&entry.data)) + .ok_or_else(invalid)?; + // Seed/private execution contexts remain local. Forwarded + // cell/metadata requests bind their authenticated origin; + // replies come only from this follower's current leader. + if (kind == MessageType::MsgReadIndex && origin != message.from) + || (kind == MessageType::MsgReadIndexResp + && (origin != self.group.identity().member.get() + || message.from != self.node.raft.leader_id)) + { + return Err(invalid()); + } + } + if kind == MessageType::MsgReadIndex { + if !self.is_parked() { + self.require_application_active()?; + } + if self.node.raft.read_only.pending_read_count() + >= self.options.max_pending_reads + { + return Err(RuntimeError::Overloaded); + } + } + } + _ if !message.entries.is_empty() => return Err(invalid()), + _ => (), + } + Ok(()) + } +} + +impl Drop for CellGroupOwner { + fn drop(&mut self) { + self.leader_term.store(0, Ordering::Release); + self.source_epoch.store(0, Ordering::Release); + } +} diff --git a/crates/cellar/durability/raft/cell_groups/page_store.rs b/crates/cellar/durability/raft/cell_groups/page_store.rs new file mode 100644 index 000000000..48af2464c --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/page_store.rs @@ -0,0 +1,389 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Exclusively owned, bounded content-addressed page files. No filesystem work +//! escapes the supplied host. Call on a blocking execution-domain worker. + +use super::{ + manifest::ManifestError, + page_wire::{digest, MAX_PAGE_BYTES}, + retention::RootRetentions, + PreparedGroupManifest, +}; +use cellar_ltx::{FileSystem, HostFileLock}; +use std::{ + collections::BTreeMap, + path::{Path, PathBuf}, + sync::{ + atomic::{AtomicBool, Ordering}, + Arc, + }, +}; + +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub struct ManifestStoreUsage { + pub pages: usize, + pub bytes: u64, + pub written_pages: u64, +} + +pub struct ManifestPageStore { + pub(super) owner: Arc, + pub(super) retention: Arc, + directory: PathBuf, + filesystem: Arc, + inventory: BTreeMap<[u8; 32], PageEntry>, + usage: ManifestStoreUsage, + max_pages: usize, + max_bytes: u64, + quarantined: bool, + collection_cursor: Option<[u8; 32]>, +} + +pub(super) struct PageOwner { + _lock: HostFileLock, +} + +struct PageEntry { + length: u64, + lease: Arc<()>, + // Only this open owner's completed file and directory barriers. Every + // subsequent preparation still rechecks the file's length and digest. + persisted: AtomicBool, +} + +/// A page lease also keeps its directory locked. A replacement store cannot +/// collect pages still selected by readers from a previous owner. +pub(super) struct ManifestPagePin { + _owner: Arc, + _lease: Arc<()>, +} + +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub struct ManifestReclaimReport { + pub scanned: usize, + pub pages: usize, + pub bytes: u64, +} + +impl ManifestPageStore { + /// Startup and controller provisioning must lock the same page inventory. + /// Archive replay uses its independent node base with this same layout. + pub(crate) fn group_pages_path( + base: &Path, + history: cellar_logic::cell_group::GroupHistory, + ) -> PathBuf { + base.join("groups") + .join(hex::encode(history.cluster.bytes())) + .join(hex::encode(history.namespace.bytes())) + .join(hex::encode(history.group.bytes())) + .join(hex::encode(history.incarnation.bytes())) + .join("pages") + } + + pub(crate) fn validate_limits( + max_pages: usize, + max_bytes: u64, + max_retained_roots: usize, + ) -> Result<(), ManifestError> { + if max_pages == 0 + || max_pages > 4_000_000 + || max_bytes == 0 + || max_bytes > 64 * 1024 * 1024 * 1024 + || !(2..=4096).contains(&max_retained_roots) + { + return Err(ManifestError::Limit); + } + Ok(()) + } + + pub(super) fn require_live(&self) -> Result<(), ManifestError> { + if self.quarantined { + return Err(ManifestError::Quarantined); + } + Ok(()) + } + + pub fn open( + directory: &Path, + filesystem: Arc, + max_pages: usize, + max_bytes: u64, + max_retained_roots: usize, + ) -> Result { + Self::validate_limits(max_pages, max_bytes, max_retained_roots)?; + cellar_ltx::host::create_dir_all_durable(filesystem.as_ref(), directory)?; + let lock = filesystem.lock_file(&directory.join("LOCK"))?; + let mut store = Self { + owner: Arc::new(PageOwner { _lock: lock }), + retention: Arc::new(RootRetentions::new(max_retained_roots)), + directory: directory.to_owned(), + filesystem, + inventory: BTreeMap::new(), + usage: ManifestStoreUsage::default(), + max_pages, + max_bytes, + quarantined: false, + collection_cursor: None, + }; + for entry in store.filesystem.read_dir_iter(directory)? { + let entry = entry?; + let name = entry + .file_name + .to_str() + .ok_or(ManifestError::Invalid("page name"))?; + if name == "LOCK" { + continue; + } + if store.inventory.len() >= max_pages { + return Err(ManifestError::Limit); + } + let hex = name + .strip_suffix(".page") + .filter(|s| { + s.len() == 64 + && s.bytes() + .all(|b| b.is_ascii_digit() || (b'a'..=b'f').contains(&b)) + }) + .ok_or(ManifestError::Invalid("page name"))?; + let mut key = [0; 32]; + hex::decode_to_slice(hex, &mut key).map_err(|_| ManifestError::Invalid("page name"))?; + // Use our own joined basename, never a provider's arbitrary path. + let metadata = store.filesystem.symlink_metadata(&directory.join(name))?; + if !metadata.is_file || metadata.len > MAX_PAGE_BYTES as u64 { + return Err(ManifestError::Invalid("page file")); + } + store.usage.bytes = store + .usage + .bytes + .checked_add(metadata.len) + .ok_or(ManifestError::Limit)?; + if store.usage.bytes > max_bytes { + return Err(ManifestError::Limit); + } + if store + .inventory + .insert( + key, + PageEntry { + length: metadata.len, + lease: Arc::new(()), + persisted: AtomicBool::new(false), + }, + ) + .is_some() + { + return Err(ManifestError::Invalid("duplicate page")); + } + } + store.usage.pages = store.inventory.len(); + Ok(store) + } + + pub fn usage(&self) -> ManifestStoreUsage { + self.usage + } + + /// Reconcile this directory's complete authoritative root inventory before + /// collection. First prepare every engine-selected root through + /// `prepare_complete`; the engine retains those roots automatically. Supply + /// all additional snapshot/archive/transition roots here, even without live + /// readers. Calling again replaces only that additional retained set. + /// Missing roots must refuse recovery, never be omitted to permit collection. + pub fn reconcile_retained( + &mut self, + roots: &[PreparedGroupManifest], + ) -> Result<(), ManifestError> { + if self.quarantined { + return Err(ManifestError::Quarantined); + } + if roots.len() > 4096 { + return Err(ManifestError::Limit); + } + if roots + .iter() + .any(|root| !Arc::ptr_eq(&root.page_owner, &self.owner)) + { + return Err(ManifestError::Invalid("retained root/page owner")); + } + self.retention + .reconcile(roots.iter().map(|root| root.images.clone()).collect()) + } + + /// Scan a bounded inventory window and remove unpinned orphan pages. Root + /// preparation requires this store's exclusive mutable borrow, so no lease + /// can appear for a selected page during deletion. Quotas are refunded only + /// after the directory barrier; any uncertain failure quarantines the owner. + pub fn reclaim(&mut self, scan_limit: usize) -> Result { + use std::ops::Bound::{Excluded, Unbounded}; + if self.quarantined { + return Err(ManifestError::Quarantined); + } + if !(1..=4096).contains(&scan_limit) { + return Err(ManifestError::Limit); + } + self.retention.can_reclaim()?; + let start = self.collection_cursor.map_or(Unbounded, Excluded); + let mut report = ManifestReclaimReport::default(); + let mut last = None; + let mut selected = Vec::new(); + for (key, entry) in self.inventory.range((start, Unbounded)).take(scan_limit) { + report.scanned += 1; + last = Some(*key); + if Arc::strong_count(&entry.lease) == 1 { + selected.push(*key); + report.bytes += entry.length; + } + } + if !selected.is_empty() { + self.quarantined = true; + for key in &selected { + self.filesystem + .remove_file(&self.directory.join(format!("{}.page", hex::encode(key))))?; + } + self.filesystem.sync_all(&self.directory)?; + for key in &selected { + self.inventory.remove(key); + } + report.pages = selected.len(); + self.usage.pages -= report.pages; + self.usage.bytes -= report.bytes; + self.quarantined = false; + } + self.collection_cursor = if report.scanned < scan_limit { + None + } else { + last + }; + Ok(report) + } + + pub(super) fn pin(&self, key: [u8; 32]) -> Result { + if self.quarantined { + return Err(ManifestError::Quarantined); + } + let entry = self + .inventory + .get(&key) + .ok_or(ManifestError::Invalid("missing page"))?; + Ok(ManifestPagePin { + _owner: self.owner.clone(), + _lease: entry.lease.clone(), + }) + } + + pub(super) fn read(&self, key: [u8; 32]) -> Result, ManifestError> { + if self.quarantined { + return Err(ManifestError::Quarantined); + } + let expected = self + .inventory + .get(&key) + .ok_or(ManifestError::Invalid("missing page"))?; + let result = (|| { + let path = self.directory.join(format!("{}.page", hex::encode(key))); + let mut file = self.filesystem.open(&path)?; + let length = file.file_len()?; + if length != expected.length || length > MAX_PAGE_BYTES as u64 { + return Err(ManifestError::Invalid("page length")); + } + let bytes = file.read_exact_at(0, length as usize)?; + if digest(&bytes) != key { + return Err(ManifestError::Invalid("page digest")); + } + Ok(bytes) + })(); + if result.is_err() { + expected.persisted.store(false, Ordering::Release); + } + result + } + + pub(super) fn sync_page(&mut self, key: [u8; 32]) -> Result<(), ManifestError> { + self.read(key)?; + self.sync_checked_page(key) + } + + // Both callers have just read and validated the immutable page. Filename + // inventory and metadata-only reads never establish persistence after open. + fn sync_checked_page(&mut self, key: [u8; 32]) -> Result<(), ManifestError> { + if self + .inventory + .get(&key) + .ok_or(ManifestError::Invalid("missing page"))? + .persisted + .load(Ordering::Acquire) + { + return Ok(()); + } + self.quarantined = true; + self.filesystem + .open(&self.directory.join(format!("{}.page", hex::encode(key))))? + .sync_all()?; + self.filesystem.sync_all(&self.directory)?; + self.inventory + .get(&key) + .ok_or(ManifestError::Invalid("missing page"))? + .persisted + .store(true, Ordering::Release); + self.quarantined = false; + Ok(()) + } + + pub(super) fn sync_directory(&mut self) -> Result<(), ManifestError> { + if self.quarantined { + return Err(ManifestError::Quarantined); + } + self.quarantined = true; + self.filesystem.sync_all(&self.directory)?; + self.quarantined = false; + Ok(()) + } + + pub(super) fn put(&mut self, key: [u8; 32], bytes: &[u8]) -> Result<(), ManifestError> { + if self.quarantined { + return Err(ManifestError::Quarantined); + } + if bytes.len() > MAX_PAGE_BYTES || digest(bytes) != key { + return Err(ManifestError::Invalid("page bytes")); + } + let path = self.directory.join(format!("{}.page", hex::encode(key))); + if self.inventory.contains_key(&key) { + if self.read(key)? != bytes { + return Err(ManifestError::Invalid("page collision")); + } + return self.sync_checked_page(key); + } + let next_bytes = self + .usage + .bytes + .checked_add(bytes.len() as u64) + .ok_or(ManifestError::Limit)?; + if self.inventory.len() >= self.max_pages || next_bytes > self.max_bytes { + return Err(ManifestError::Limit); + } + // Any uncertain write/sync leaves the owner quarantined. No refunds or + // deletion before a new owner reconstructs authoritative root references. + self.quarantined = true; + let mut file = self.filesystem.create_new(&path)?; + file.write_all(bytes)?; + file.sync_all()?; + self.filesystem.sync_all(&self.directory)?; + self.inventory.insert( + key, + PageEntry { + length: bytes.len() as u64, + lease: Arc::new(()), + persisted: AtomicBool::new(true), + }, + ); + self.usage.pages += 1; + self.usage.bytes = next_bytes; + self.usage.written_pages = self + .usage + .written_pages + .checked_add(1) + .ok_or(ManifestError::Limit)?; + self.quarantined = false; + Ok(()) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/page_wire.rs b/crates/cellar/durability/raft/cell_groups/page_wire.rs new file mode 100644 index 000000000..e68844633 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/page_wire.rs @@ -0,0 +1,287 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::manifest::{ + CellImageGeometry, CellManifestLeaf, EffectSummary, ManifestError, ManifestLimits, +}; +use crate::application::consensus::{ImageCommitment, LtxImage}; +use cellar_logic::{cell_group::*, consensus::*}; +use serde::{Deserialize, Serialize}; +use sha2::{Digest, Sha256}; + +#[cfg(test)] +#[path = "page_wire_tests.rs"] +mod tests; + +const MAGIC: &[u8; 8] = b"CLRCGM01"; +pub(super) const MAX_PAGE_BYTES: usize = 16 * 1024; + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct PageRef { + pub digest: [u8; 32], + pub height: u8, + pub cells: u32, + pub last_mutation_index: u64, + pub last_mutation_term: u64, + // Required even when null: internal marker-1 hard cutover, no old reader. + #[serde(deserialize_with = "required_due")] + pub next_due_unix_millis: Option, +} + +fn required_due<'de, D: serde::Deserializer<'de>>( + deserializer: D, +) -> Result, D::Error> { + Option::::deserialize(deserializer) +} + +pub(super) struct Page { + pub leaf: CellManifestLeaf, + pub left: Option, + pub right: Option, +} + +impl Page { + pub fn reference( + &self, + bytes: &[u8], + limits: ManifestLimits, + ) -> Result { + let height = |p: Option| p.map_or(0, |p| p.height); + let count = |p: Option| p.map_or(0, |p| p.cells); + for child in [self.left, self.right].into_iter().flatten() { + if child.height == 0 + || child.height > limits.max_depth + || child.cells == 0 + || child.cells > limits.max_cells + || child.last_mutation_index == 0 + || child.last_mutation_term == 0 + || child.last_mutation_index >= u64::MAX - 2 + || child.last_mutation_term >= u64::MAX - 2 + || child + .next_due_unix_millis + .is_some_and(|due| due > i64::MAX as u64) + { + return Err(ManifestError::Invalid("child bounds")); + } + } + if height(self.left).abs_diff(height(self.right)) > 1 { + return Err(ManifestError::Invalid("unbalanced page")); + } + let height = height(self.left) + .max(height(self.right)) + .checked_add(1) + .ok_or(ManifestError::Limit)?; + let cells = count(self.left) + .checked_add(count(self.right)) + .and_then(|n| n.checked_add(1)) + .ok_or(ManifestError::Limit)?; + if height > limits.max_depth || cells > limits.max_cells { + return Err(ManifestError::Limit); + } + let mut latest = self.leaf.last_mutation; + for child in [self.left, self.right].into_iter().flatten() { + let next = LogPosition { + index: LogIndex::new(child.last_mutation_index), + term: Term::new(child.last_mutation_term), + }; + if (next.index < latest.index && next.term > latest.term) + || (next.index > latest.index && next.term < latest.term) + || (next.index == latest.index && next.term != latest.term) + { + return Err(ManifestError::Invalid("subtree mutation order")); + } + if next.index > latest.index { + latest = next; + } + } + Ok(PageRef { + digest: digest(bytes), + height, + cells, + last_mutation_index: latest.index.get(), + last_mutation_term: latest.term.get(), + next_due_unix_millis: [ + self.leaf.effects.next_due_unix_millis(), + self.left.and_then(|p| p.next_due_unix_millis), + self.right.and_then(|p| p.next_due_unix_millis), + ] + .into_iter() + .flatten() + .min(), + }) + } +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct WirePage { + enrollment: super::EnrollmentRecord, + schema: [u8; 32], + cluster: [u8; 16], + namespace: [u8; 16], + cell: String, + incarnation: [u8; 16], + sequence: u64, + commitment: ImageCommitment, + epoch: u64, + txid: u64, + checksum: u64, + geometry: CellImageGeometry, + mutation_index: u64, + mutation_term: u64, + witness: WireWitness, + effects: EffectSummary, + left: Option, + right: Option, +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct WireWitness { + cluster: [u8; 16], + namespace: [u8; 16], + group: [u8; 16], + incarnation: [u8; 16], + index: u64, + term: u64, + proposal: [u8; 16], +} + +impl From for WireWitness { + fn from(witness: CommitWitness) -> Self { + Self { + cluster: witness.group.cluster.bytes(), + namespace: witness.group.namespace.bytes(), + group: witness.group.group.bytes(), + incarnation: witness.group.incarnation.bytes(), + index: witness.position.index.get(), + term: witness.position.term.get(), + proposal: witness.proposal.bytes(), + } + } +} + +impl TryFrom for CommitWitness { + type Error = ManifestError; + + fn try_from(wire: WireWitness) -> Result { + let invalid = |_| ManifestError::Invalid("cell commit witness"); + Ok(Self { + group: GroupHistory { + cluster: ClusterId::new(wire.cluster).map_err(invalid)?, + namespace: NamespaceId::new(wire.namespace).map_err(invalid)?, + group: GroupId::new(wire.group).map_err(invalid)?, + incarnation: IncarnationId::new(wire.incarnation).map_err(invalid)?, + }, + position: LogPosition { + index: LogIndex::new(wire.index), + term: Term::new(wire.term), + }, + proposal: ProposalId::new(wire.proposal).map_err(invalid)?, + }) + } +} + +pub(super) fn digest(bytes: &[u8]) -> [u8; 32] { + let mut hash = Sha256::new(); + hash.update(b"cellar-cell-group-manifest-page-1\0"); + hash.update(bytes); + hash.finalize().into() +} + +pub(super) fn encode( + page: &Page, + route: &GroupRoute, + limits: ManifestLimits, +) -> Result, ManifestError> { + page.leaf.validate(route)?; + let leaf = &page.leaf; + let history = &leaf.version.history; + let wire = WirePage { + enrollment: leaf.enrollment.clone(), + schema: route.schema_digest, + cluster: history.cluster.bytes(), + namespace: history.namespace.bytes(), + cell: history.cell.as_str().to_owned(), + incarnation: history.incarnation.bytes(), + sequence: leaf.version.sequence.get(), + commitment: LtxImage::try_from(leaf.version.image) + .map_err(|_| ManifestError::Invalid("image commitment"))? + .commitment, + epoch: leaf.version.image.epoch, + txid: leaf.version.image.txid, + checksum: leaf.version.image.checksum, + geometry: leaf.geometry, + mutation_index: leaf.last_mutation.index.get(), + mutation_term: leaf.last_mutation.term.get(), + witness: leaf.witness.into(), + effects: leaf.effects, + left: page.left, + right: page.right, + }; + let mut bytes = MAGIC.to_vec(); + bytes.extend(serde_json::to_vec(&wire).map_err(|_| ManifestError::Invalid("page encoding"))?); + if bytes.len() > MAX_PAGE_BYTES { + return Err(ManifestError::Limit); + } + page.reference(&bytes, limits)?; + Ok(bytes) +} + +pub(super) fn decode( + bytes: &[u8], + route: &GroupRoute, + limits: ManifestLimits, +) -> Result { + // Reject length before JSON allocation. The schema has no unbounded vectors + // or recursive fields; serde's recursion limit remains enabled. + if bytes.len() > MAX_PAGE_BYTES { + return Err(ManifestError::Limit); + } + if !bytes.starts_with(MAGIC) { + return Err(ManifestError::Invalid("page domain")); + } + let wire: WirePage = serde_json::from_slice(&bytes[MAGIC.len()..]) + .map_err(|_| ManifestError::Invalid("page decoding"))?; + if wire.schema != route.schema_digest { + return Err(ManifestError::Invalid("page schema")); + } + let history = CellHistory { + cluster: ClusterId::new(wire.cluster).map_err(|_| ManifestError::Invalid("cluster"))?, + namespace: NamespaceId::new(wire.namespace) + .map_err(|_| ManifestError::Invalid("namespace"))?, + cell: CellId::new(wire.cell).map_err(|_| ManifestError::Invalid("cell ID"))?, + incarnation: IncarnationId::new(wire.incarnation) + .map_err(|_| ManifestError::Invalid("incarnation"))?, + }; + let image = LtxImage { + epoch: wire.epoch, + txid: wire.txid, + checksum: wire.checksum, + commitment: wire.commitment, + }; + let page = Page { + leaf: CellManifestLeaf { + enrollment: wire.enrollment, + version: CellVersion { + history, + sequence: MutationSequence::new(wire.sequence) + .map_err(|_| ManifestError::Invalid("sequence"))?, + image: image.into(), + }, + geometry: wire.geometry, + last_mutation: LogPosition { + index: LogIndex::new(wire.mutation_index), + term: Term::new(wire.mutation_term), + }, + witness: wire.witness.try_into()?, + effects: wire.effects, + }, + left: wire.left, + right: wire.right, + }; + if encode(&page, route, limits)? != bytes { + return Err(ManifestError::Invalid("noncanonical page")); + } + Ok(page) +} diff --git a/crates/cellar/durability/raft/cell_groups/page_wire_tests.rs b/crates/cellar/durability/raft/cell_groups/page_wire_tests.rs new file mode 100644 index 000000000..e3afe0d04 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/page_wire_tests.rs @@ -0,0 +1,180 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use crate::raft::cell_groups::EnrollmentRecord; + +fn fixture() -> (GroupRoute, Page) { + let history = CellHistory { + cluster: ClusterId::new([1; 16]).unwrap(), + namespace: NamespaceId::new([2; 16]).unwrap(), + cell: CellId::new("cell").unwrap(), + incarnation: IncarnationId::new([3; 16]).unwrap(), + }; + let route = GroupRoute { + history: GroupHistory { + cluster: history.cluster, + namespace: history.namespace, + group: GroupId::new([4; 16]).unwrap(), + incarnation: IncarnationId::new([5; 16]).unwrap(), + }, + range: CellRange::all(), + epoch: RouteEpoch::new(1).unwrap(), + birth_digest: [6; 32], + schema_digest: [7; 32], + }; + let initial = CellImage { + epoch: 1, + txid: 1, + checksum: 0, + commitment: CellImageCommitment::Materialized([8; 32]), + }; + let enrollment = EnrollmentRecord::new( + history.clone(), + ProposalId::new([9; 16]).unwrap(), + route.schema_digest, + initial, + ) + .unwrap(); + let witness = CommitWitness { + group: route.history, + position: LogPosition { + index: LogIndex::new(2), + term: Term::new(1), + }, + proposal: enrollment.creation(), + }; + ( + route, + Page { + leaf: CellManifestLeaf { + enrollment, + version: CellVersion { + history, + sequence: MutationSequence::new(1).unwrap(), + image: initial, + }, + geometry: CellImageGeometry { + length: 512, + page_size: 512, + }, + last_mutation: LogPosition { + index: LogIndex::new(2), + term: Term::new(1), + }, + witness, + effects: EffectSummary::default(), + }, + left: None, + right: None, + }, + ) +} + +#[test] +fn creation_record_is_required_in_marker_one_pages() { + let (route, page) = fixture(); + let limits = ManifestLimits::default(); + let encoded = encode(&page, &route, limits).unwrap(); + assert_eq!(decode(&encoded, &route, limits).unwrap().leaf, page.leaf); + let original = String::from_utf8(encoded[MAGIC.len()..].to_vec()).unwrap(); + let record = String::from_utf8(page.leaf.enrollment.encode().unwrap()).unwrap(); + let replace = |field: &str, value: serde_json::Value| { + let parsed: serde_json::Value = serde_json::from_str(&record).unwrap(); + record.replacen( + &format!("\"{field}\":{}", parsed[field]), + &format!("\"{field}\":{value}"), + 1, + ) + }; + // Keep every unaffected field in canonical order. A reordered JSON map + // must not be the reason these malformed identities refuse. + for changed in [ + None, + Some("null".to_owned()), + Some(replace("cell", "another-cell".into())), + Some(replace("schema", serde_json::json!(vec![0; 32]))), + Some(replace("initial_txid", 2.into())), + Some(replace("creation", serde_json::json!(vec![0; 16]))), + ] { + let changed = match changed { + Some(value) => original.replacen(&record, &value, 1), + None => original.replacen(&format!("\"enrollment\":{record},"), "", 1), + }; + assert_ne!(changed, original); + let mut bytes = MAGIC.to_vec(); + bytes.extend(changed.as_bytes()); + assert!(decode(&bytes, &route, limits).is_err()); + } +} + +#[test] +fn creation_binding_survives_later_versions_and_bounds_all_cell_names() { + let (route, mut page) = fixture(); + let original = page.leaf.enrollment.clone(); + page.leaf.version.sequence = MutationSequence::new(2).unwrap(); + assert!(encode(&page, &route, ManifestLimits::default()).is_err()); + page.leaf.version.image.txid = 2; + page.leaf.version.image.commitment = CellImageCommitment::Materialized([10; 32]); + page.leaf.last_mutation.index = LogIndex::new(3); + page.leaf.witness.position = page.leaf.last_mutation; + page.leaf.witness.proposal = ProposalId::new([10; 16]).unwrap(); + let bytes = encode(&page, &route, ManifestLimits::default()).unwrap(); + let restored = decode(&bytes, &route, ManifestLimits::default()).unwrap(); + assert_eq!(restored.leaf.enrollment, original); + assert_eq!(restored.leaf.version, page.leaf.version); + + let id = CellId::new("\\".repeat(MAX_CELL_ID_BYTES)).unwrap(); + page.leaf.version.history.cell = id; + page.leaf.enrollment = EnrollmentRecord::new( + page.leaf.version.history.clone(), + original.creation(), + original.schema(), + original.initial(), + ) + .unwrap(); + let bytes = encode(&page, &route, ManifestLimits::default()).unwrap(); + assert!(bytes.len() <= MAX_PAGE_BYTES); + assert_eq!( + decode(&bytes, &route, ManifestLimits::default()) + .unwrap() + .leaf, + page.leaf + ); +} + +#[test] +fn commit_witness_is_required_and_bounded_at_marker_one() { + let (route, page) = fixture(); + let limits = ManifestLimits::default(); + let encoded = encode(&page, &route, limits).unwrap(); + let original = String::from_utf8(encoded[MAGIC.len()..].to_vec()).unwrap(); + let witness = serde_json::to_string(&WireWitness::from(page.leaf.witness)).unwrap(); + let replace = |field: &str, value: serde_json::Value| { + let parsed: serde_json::Value = serde_json::from_str(&witness).unwrap(); + witness.replacen( + &format!("\"{field}\":{}", parsed[field]), + &format!("\"{field}\":{value}"), + 1, + ) + }; + for changed in [ + None, + Some("null".to_owned()), + Some(replace("namespace", serde_json::json!(vec![80; 16]))), + Some(replace("group", serde_json::json!(vec![0; 16]))), + Some(replace("proposal", serde_json::json!(vec![0; 16]))), + Some(replace("proposal", serde_json::json!(vec![80; 16]))), + Some(replace("index", 0.into())), + Some(replace("index", 3.into())), + Some(replace("term", 2.into())), + ] { + let changed = match changed { + Some(value) => original.replacen(&witness, &value, 1), + None => original.replacen(&format!("\"witness\":{witness},"), "", 1), + }; + assert_ne!(changed, original); + let mut bytes = MAGIC.to_vec(); + bytes.extend(changed.as_bytes()); + assert!(decode(&bytes, &route, limits).is_err()); + } +} diff --git a/crates/cellar/durability/raft/cell_groups/park_certificate.rs b/crates/cellar/durability/raft/cell_groups/park_certificate.rs new file mode 100644 index 000000000..3153272bb --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/park_certificate.rs @@ -0,0 +1,218 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Complete voter evidence for one exact park generation. Transport must +//! authenticate each remote reply before it enters certificate assembly. + +use super::hibernate::{GroupParkCut, ParkAcknowledgement}; +use crate::raft::{member_state::GroupMembership, storage::LogError}; +use ::raft::prelude::ConfState; +use cellar_logic::{ + cell_group::{CellBound, GroupHistory, GroupRoute}, + consensus::{DiskLineage, MemberId, NodeId, NodeIdentity, ProcessGeneration, ReplicaIdentity}, +}; +use serde::{Deserialize, Serialize}; +use sha2::{Digest, Sha256}; +use std::collections::{BTreeMap, BTreeSet}; + +const MAX_PARK_VOTERS: usize = 16; + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(crate) struct ParkCertificate { + route: [u8; 32], + cut: GroupParkCut, + leader: u64, + voters: Vec, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct VoterWitness { + member: u64, + node: [u8; 16], + disk: [u8; 16], + process: [u8; 16], + sequence: u64, +} + +impl VoterWitness { + fn replica(&self, scope: GroupHistory) -> Result, LogError> { + Ok(ReplicaIdentity { + scope, + member: MemberId::new(self.member).map_err(|_| LogError::WrongIdentity)?, + node: NodeId::new(self.node).map_err(|_| LogError::WrongIdentity)?, + disk: DiskLineage::new(self.disk).map_err(|_| LogError::WrongIdentity)?, + process: ProcessGeneration::new(self.process).map_err(|_| LogError::WrongIdentity)?, + }) + } +} + +impl ParkCertificate { + /// The roster comes from the active physical registration. Each + /// acknowledgement is supplied by the local runtime or an authenticated + /// reply tied to that same registration. + pub(crate) fn assemble( + route: &GroupRoute, + cut: GroupParkCut, + leader: &ReplicaIdentity, + roster: &[ReplicaIdentity], + acknowledgements: Vec, + ) -> Result { + if !(1..=MAX_PARK_VOTERS).contains(&roster.len()) || roster.len() != acknowledgements.len() + { + return Err(LogError::WrongIdentity); + } + let mut expected = BTreeMap::new(); + let mut nodes = BTreeSet::new(); + let mut disks = BTreeSet::new(); + for replica in roster { + if replica.scope != route.history + || expected.insert(replica.member, replica).is_some() + || !nodes.insert(replica.node) + || !disks.insert(replica.disk) + { + return Err(LogError::WrongIdentity); + } + } + if expected.get(&leader.member).copied() != Some(leader) { + return Err(LogError::WrongIdentity); + } + + let mut voters = Vec::with_capacity(roster.len()); + for acknowledgement in acknowledgements { + let voter = acknowledgement.voter; + if acknowledgement.leader != leader.member + || acknowledgement.cut != cut + || acknowledgement.sync_sequence == 0 + || expected.remove(&voter.member) != Some(&voter) + { + return Err(LogError::WrongIdentity); + } + voters.push(VoterWitness { + member: voter.member.get(), + node: voter.node.bytes(), + disk: voter.disk.bytes(), + process: voter.process.bytes(), + sequence: acknowledgement.sync_sequence, + }); + } + voters.sort_unstable_by_key(|voter| voter.member); + Ok(Self { + route: route_binding(route), + cut, + leader: leader.member.get(), + voters, + }) + } + + pub(crate) fn cut(&self) -> &GroupParkCut { + &self.cut + } + + pub(crate) fn leader(&self) -> Result { + MemberId::new(self.leader).map_err(|_| LogError::WrongIdentity) + } + + pub(crate) fn reserved_bytes(&self) -> usize { + 1024_usize.saturating_add(self.voters.len().saturating_mul(512)) + } + + pub(super) fn validate( + &self, + route: &GroupRoute, + conf: &ConfState, + membership: &GroupMembership, + ) -> Result<(), LogError> { + if self.route != route_binding(route) + || self.cut.membership_revision() != membership.revision().get() + || !(1..=MAX_PARK_VOTERS).contains(&self.voters.len()) + || self.voters.len() != conf.voters.len() + || !conf.voters.contains(&self.leader) + || !conf.voters_outgoing.is_empty() + || !conf.learners.is_empty() + || !conf.learners_next.is_empty() + || conf.auto_leave + { + return Err(LogError::WrongIdentity); + } + let mut previous = 0; + let mut nodes = BTreeSet::new(); + let mut disks = BTreeSet::new(); + for witness in &self.voters { + let replica = witness.replica(route.history)?; + if witness.member <= previous + || witness.sequence == 0 + || !conf.voters.contains(&witness.member) + || !membership.permits_group(&replica) + || !nodes.insert(replica.node) + || !disks.insert(replica.disk) + { + return Err(LogError::WrongIdentity); + } + previous = witness.member; + } + Ok(()) + } + + pub(crate) fn matches_roster( + &self, + route: &GroupRoute, + leader: &ReplicaIdentity, + roster: &[ReplicaIdentity], + ) -> bool { + leader.scope == route.history + && self.route == route_binding(route) + && self.matches_leader(leader.member, leader.node_identity()) + && self.voters.len() == roster.len() + && self.voters.iter().all(|witness| { + witness + .replica(route.history) + .is_ok_and(|replica| roster.contains(&replica)) + }) + } + + pub(super) fn matches_leader(&self, member: MemberId, identity: NodeIdentity) -> bool { + self.leader == member.get() && self.matches_voter(member, identity) + } + + pub(super) fn matches_voter(&self, member: MemberId, identity: NodeIdentity) -> bool { + self.voters.iter().any(|voter| { + voter.member == member.get() + && voter.node == identity.node.bytes() + && voter.disk == identity.disk.bytes() + && voter.process == identity.process.bytes() + }) + } + + pub(super) fn matches_journal(&self, member: u64, process: [u8; 16]) -> bool { + self.voters + .iter() + .any(|voter| voter.member == member && voter.process == process) + } +} + +fn route_binding(route: &GroupRoute) -> [u8; 32] { + let mut digest = Sha256::new(); + digest.update(b"cellar-park-certificate-route-1"); + digest.update(route.history.cluster.bytes()); + digest.update(route.history.namespace.bytes()); + digest.update(route.history.group.bytes()); + digest.update(route.history.incarnation.bytes()); + digest.update(route.epoch.get().to_be_bytes()); + for bound in [route.range.lower(), route.range.upper()] { + match bound { + CellBound::NegativeInfinity => digest.update([0]), + CellBound::Key(key) => { + digest.update([1]); + digest.update(key.digest()); + let cell = key.cell().as_str().as_bytes(); + digest.update((cell.len() as u64).to_be_bytes()); + digest.update(cell); + } + CellBound::PositiveInfinity => digest.update([2]), + } + } + digest.update(route.birth_digest); + digest.update(route.schema_digest); + digest.finalize().into() +} diff --git a/crates/cellar/durability/raft/cell_groups/pins.rs b/crates/cellar/durability/raft/cell_groups/pins.rs new file mode 100644 index 000000000..6e35695cd --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/pins.rs @@ -0,0 +1,309 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Persistent physical ownership follows the authenticated AVL topology. Cloning +//! a root is constant-time; changing a cell copies only its path and rotations. + +use super::{ + page_wire::PageRef, CellImagePin, CellManifestLeaf, ManifestError, ManifestPageStore, + ManifestRoot, ManifestTree, +}; +use crate::application::consensus::cell_routing_key; +use cellar_logic::cell_group::{CellKey, MAX_CELL_ID_BYTES}; +use std::{cmp::Ordering, sync::Arc}; + +#[derive(Clone, Default)] +pub(crate) struct ManifestPins(Option>); + +struct PinnedPage { + _page: super::page_store::ManifestPagePin, + reference: PageRef, + key: CellKey, + leaf: CellManifestLeaf, + image: CellImagePin, + left: ManifestPins, + right: ManifestPins, + bytes: usize, + logical_bytes: u64, + largest_image: u64, +} + +impl ManifestPins { + /// Walk the immutable closure without cloning its cells or allocating a + /// second inventory. Verified AVL height bounds the recursive stack to 32. + pub(super) fn visit( + &self, + visitor: &mut impl FnMut( + PageRef, + &CellManifestLeaf, + &CellImagePin, + Option, + Option, + ) -> Result<(), E>, + ) -> Result<(), E> { + if let Some(page) = &self.0 { + page.left.visit(visitor)?; + visitor( + page.reference, + &page.leaf, + &page.image, + page.left.root().0, + page.right.root().0, + )?; + page.right.visit(visitor)?; + } + Ok(()) + } + + pub(crate) fn len(&self) -> usize { + self.0.as_ref().map_or(0, |p| p.reference.cells as usize) + } + + pub(super) fn logical_bytes(&self) -> u64 { + self.0.as_ref().map_or(0, |page| page.logical_bytes) + } + + fn largest_image(&self) -> u64 { + self.0.as_ref().map_or(0, |page| page.largest_image) + } + + pub(super) fn key_at_rank(&self, mut rank: usize) -> Option<&CellKey> { + let mut tree = self; + while let Some(page) = tree.0.as_deref() { + let left = page.left.len(); + match rank.cmp(&left) { + Ordering::Less => tree = &page.left, + Ordering::Equal => return Some(&page.key), + Ordering::Greater => { + rank -= left + 1; + tree = &page.right; + } + } + } + None + } + + pub(super) fn image_rank(&self, key: &CellKey) -> Option<(usize, u64)> { + let mut tree = self; + let mut preceding = 0; + while let Some(page) = tree.0.as_deref() { + let left = page.left.len(); + match key.cmp(&page.key) { + Ordering::Less => tree = &page.left, + Ordering::Equal => return Some((preceding + left, page.leaf.geometry.length)), + Ordering::Greater => { + preceding += left + 1; + tree = &page.right; + } + } + } + None + } + + /// Largest logical image, breaking ties in whole-cell key order. + pub(super) fn largest_image_rank(&self) -> Option<(usize, u64)> { + let mut tree = self; + let mut preceding = 0; + while let Some(page) = tree.0.as_deref() { + if page.left.0.is_some() && page.left.largest_image() == page.largest_image { + tree = &page.left; + continue; + } + let left = page.left.len(); + if page.leaf.geometry.length == page.largest_image { + return Some((preceding + left, page.largest_image)); + } + preceding += left + 1; + tree = &page.right; + } + None + } + + pub(crate) fn root(&self) -> ManifestRoot { + ManifestRoot(self.0.as_ref().map(|p| p.reference)) + } + + pub(crate) fn maximum_bytes(cells: u32) -> usize { + cells as usize + * (std::mem::size_of::() + + 2 * std::mem::size_of::() + + 3 * MAX_CELL_ID_BYTES) + } + + /// Conservatively charge shared subtrees to every owner. Include all three cell + /// ID allocations and Arc control blocks, not just the image handle size. + pub(crate) fn reserved_bytes(&self) -> usize { + self.0.as_ref().map_or(0, |p| p.bytes) + } + + pub(super) fn get(&self, key: &CellKey) -> Option<(&CellManifestLeaf, CellImagePin)> { + self.find(key).map(|p| (&p.leaf, p.image.clone())) + } + + /// The complete verified mirror already owns every referenced page/image. + /// Discovery uses immutable metadata only: no filesystem or SQLite work. + pub(super) fn next_due( + &self, + now: u64, + after: Option<&CellKey>, + ) -> Option<(&CellManifestLeaf, CellImagePin)> { + let page = self + .find_due(now, after) + .or_else(|| after.and_then(|_| self.find_due(now, None)))?; + Some((&page.leaf, page.image.clone())) + } + + fn find_due(&self, now: u64, after: Option<&CellKey>) -> Option<&PinnedPage> { + let page = self.0.as_deref()?; + if !page + .reference + .next_due_unix_millis + .is_some_and(|due| due <= now) + { + return None; + } + if after.is_none_or(|after| &page.key > after) { + if let Some(found) = page.left.find_due(now, after) { + return Some(found); + } + if page + .leaf + .effects + .next_due_unix_millis() + .is_some_and(|due| due <= now) + { + return Some(page); + } + } + page.right.find_due(now, after) + } + + fn find(&self, key: &CellKey) -> Option<&Arc> { + let mut cursor = &self.0; + while let Some(page) = cursor { + match key.cmp(&page.key) { + Ordering::Less => cursor = &page.left.0, + Ordering::Greater => cursor = &page.right.0, + Ordering::Equal => return Some(page), + } + } + None + } + + fn node( + reference: PageRef, + key: CellKey, + leaf: CellManifestLeaf, + image: CellImagePin, + left: Self, + right: Self, + page: super::page_store::ManifestPagePin, + ) -> Self { + let bytes = std::mem::size_of::() + + 2 * std::mem::size_of::() + + key.cell().as_str().len() + + leaf.version.history.cell.as_str().len() + + leaf.enrollment.cell().cell.as_str().len() + + left.reserved_bytes() + + right.reserved_bytes(); + // The verified manifest bounds every image and caps occupancy at a + // million cells, so the logical sum fits in u64. + let logical_bytes = left.logical_bytes() + leaf.geometry.length + right.logical_bytes(); + let largest_image = leaf + .geometry + .length + .max(left.largest_image()) + .max(right.largest_image()); + Self(Some(Arc::new(PinnedPage { + _page: page, + reference, + key, + leaf, + image, + left, + right, + bytes, + logical_bytes, + largest_image, + }))) + } + + /// The caller first verifies the complete root's ordering and applied cut. + /// Every read rechecks content addressing; the second walk cannot silently + /// substitute different leaves after that verification. + pub(super) fn complete( + tree: &ManifestTree, + store: &mut ManifestPageStore, + root: ManifestRoot, + image: &mut impl FnMut(&CellManifestLeaf) -> Result, + ) -> Result { + let Some(reference) = root.0 else { + return Ok(Self::default()); + }; + let page = tree.read(store, reference)?; + let left = Self::complete(tree, store, ManifestRoot(page.left), image)?; + let pin = image(&page.leaf)?; + pin.check(&page.leaf, tree.route.schema_digest)?; + let right = Self::complete(tree, store, ManifestRoot(page.right), image)?; + store.sync_page(reference.digest)?; + let key = cell_routing_key( + tree.route.history.namespace, + &page.leaf.version.history.cell, + ); + Ok(Self::node( + reference, + key, + page.leaf, + pin, + left, + right, + store.pin(reference.digest)?, + )) + } + + /// Only accepts the root produced by the tree's compare-and-replace method. + /// Search the old tree by the new page's key so AVL rotations also find and + /// reuse unchanged subtrees. Disk reads/node allocations stay O(log cells); + /// the bounded in-memory searches take O(log² cells) comparisons. + pub(super) fn updated( + &self, + tree: &ManifestTree, + store: &ManifestPageStore, + root: ManifestRoot, + changed: &CellManifestLeaf, + image: &CellImagePin, + ) -> Result { + let Some(reference) = root.0 else { + return Ok(Self::default()); + }; + let page = tree.read(store, reference)?; + let key = cell_routing_key( + tree.route.history.namespace, + &page.leaf.version.history.cell, + ); + let previous = self.find(&key); + if let Some(old) = previous { + if old.reference == reference { + return Ok(Self(Some(old.clone()))); + } + } + let pin = if page.leaf == *changed { + image.clone() + } else { + let old = previous + .filter(|old| old.leaf == page.leaf) + .ok_or(ManifestError::Invalid("unexpected changed leaf"))?; + old.image.clone() + }; + let left = self.updated(tree, store, ManifestRoot(page.left), changed, image)?; + let right = self.updated(tree, store, ManifestRoot(page.right), changed, image)?; + Ok(Self::node( + reference, + key, + page.leaf, + pin, + left, + right, + store.pin(reference.digest)?, + )) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/proposal_admission.rs b/crates/cellar/durability/raft/cell_groups/proposal_admission.rs new file mode 100644 index 000000000..3ca1f2a1a --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/proposal_admission.rs @@ -0,0 +1,166 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Bounded, leader-local acceptance; never a commit or receipt proof. + +use super::{CellGroupProposal, PreparedGroupManifest}; +use crate::application::consensus::group_protocol::{CellGroupEntry, GroupCommand}; +use crate::raft::budget::{Budget, Permit}; +use crate::raft::completion::RuntimeError; +use cellar_logic::{ + cell_group::CellId, + consensus::{LogIndex, LogPosition, ProposalId}, +}; +use sha2::{Digest, Sha256}; +use std::collections::BTreeMap; +use std::sync::Arc; + +pub(super) struct Candidate { + cell: CellId, + proposal: ProposalId, + payload: [u8; 32], + enrollment: bool, +} + +impl Candidate { + pub(super) fn mutation_cell(&self) -> Option<&CellId> { + (!self.enrollment).then_some(&self.cell) + } +} + +struct Accepted { + candidate: Candidate, + acceptance: CellGroupProposal, + _permit: Permit, +} + +pub(super) enum Admission { + Existing(CellGroupProposal), + New(Candidate, Permit), +} + +/// At most one record per pending logical name, bounded by max_requests. No LTX +/// payload or image pin is retained. Clearing on role loss is safe because new +/// admission requires a current-term applied barrier covering inherited entries. +#[derive(Default)] +pub(super) struct ProposalAdmission { + cells: BTreeMap, + positions: BTreeMap, + enrollments: u32, +} + +impl ProposalAdmission { + pub(super) fn check( + &self, + entry: &CellGroupEntry<'_>, + bytes: &[u8], + current: &PreparedGroupManifest, + max_cells: u32, + max_pending: usize, + budget: &Arc, + ) -> Result { + let (cell, enrollment) = match &entry.command { + GroupCommand::Enroll { cell, .. } => (&cell.cell, true), + GroupCommand::Mutation(mutation) => (&mutation.cell.cell, false), + _ => { + return Err(RuntimeError::Invalid( + "group administration is not attached".into(), + )) + } + }; + let candidate = Candidate { + cell: cell.clone(), + proposal: entry.binding.proposal, + payload: Sha256::digest(bytes).into(), + enrollment, + }; + if let Some(accepted) = self.cells.get(cell) { + if accepted.candidate.proposal != candidate.proposal { + return Err(RuntimeError::NotReady); + } + if accepted.candidate.payload != candidate.payload { + return Err(RuntimeError::Invalid( + "proposal identity reused with different bytes".into(), + )); + } + return Ok(Admission::Existing(accepted.acceptance.clone())); + } + let existing = current.pin_cell(cell); + match &entry.command { + GroupCommand::Enroll { .. } => { + // Once applied, the immutable creation record is the retry + // authority. Do not append another enrollment, even at capacity. + if existing.is_some() { + return Err(RuntimeError::StaleCompletion); + } + if current + .marker() + .root() + .cells() + .saturating_add(self.enrollments) + >= max_cells + { + return Err(RuntimeError::Overloaded); + } + } + GroupCommand::Mutation(mutation) => { + let (leaf, _) = existing.ok_or(RuntimeError::StaleCompletion)?; + mutation + .check_base(&leaf.version) + .map_err(|_| RuntimeError::StaleCompletion)?; + } + _ => unreachable!("validated data command"), + } + if self.cells.len() >= max_pending { + return Err(RuntimeError::Overloaded); + } + // Three bounded cell-ID copies, both map nodes and acceptance metadata. + // Shared across runtime groups, separately from Ready/completion budgets + // so saturation cannot prevent accepted work from draining. + let permit = budget.acquire(4096).ok_or(RuntimeError::Overloaded)?; + Ok(Admission::New(candidate, permit)) + } + + pub(super) fn accept( + &mut self, + candidate: Candidate, + acceptance: CellGroupProposal, + permit: Permit, + ) { + self.enrollments += u32::from(candidate.enrollment); + self.positions + .insert(acceptance.position.index, candidate.cell.clone()); + self.cells.insert( + candidate.cell.clone(), + Accepted { + candidate, + acceptance, + _permit: permit, + }, + ); + } + + pub(super) fn applied(&mut self, position: LogPosition) { + let Some(cell) = self.positions.get(&position.index) else { + return; + }; + if self + .cells + .get(cell) + .is_none_or(|accepted| accepted.acceptance.position != position) + { + return; + } + let cell = self + .positions + .remove(&position.index) + .expect("accepted position"); + let accepted = self.cells.remove(&cell).expect("accepted cell"); + self.enrollments -= u32::from(accepted.candidate.enrollment); + } + + pub(super) fn clear(&mut self) { + self.cells.clear(); + self.positions.clear(); + self.enrollments = 0; + } +} diff --git a/crates/cellar/durability/raft/cell_groups/read_authority.rs b/crates/cellar/durability/raft/cell_groups/read_authority.rs new file mode 100644 index 000000000..0940256b9 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/read_authority.rs @@ -0,0 +1,139 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! One invocation's coherent cell pin, never a reusable leadership lease. + +use super::{CellImagePin, CellManifestLeaf}; +use crate::raft::{budget::Permit, cache::GroupLog, completion::RuntimeError}; +use cellar_logic::{ + cell_group::*, + consensus::{LogPosition, Term}, +}; +use std::sync::{ + atomic::{AtomicU64, Ordering}, + Arc, +}; + +#[derive(Clone)] +pub enum CellReadConsistency { + /// Explicitly allows a locally committed older image, including on followers. + Stale, + /// A bounded-age observation of the current source; never a quorum proof. + RecentSource { + max_age_ms: u64, + }, + /// The dispatcher must first authenticate the version and its schema/lineage. + AtLeast(CellVersion), + Linearizable, +} + +pub(super) struct ReadSelection { + pub group: GroupLog, + pub route: GroupRoute, + pub cell: CellManifestLeaf, + pub image: CellImagePin, + pub applied: LogPosition, + pub barrier: Option, + pub leader: Option<(Term, Arc)>, + pub source: Option<(Term, u64, Arc)>, + pub source_deadline: Option, +} + +/// Consumed by one SQL query. The image and its admission cannot be cloned out +/// of this proof. A strong selection requires a fresh correlated ReadIndex and +/// contiguous apply; unrelated cell progress never becomes a cell version. +pub struct CellReadAuthority { + selection: ReadSelection, + deadline: u64, + _node: Permit, + _group: Permit, +} + +impl CellReadAuthority { + pub(super) fn new( + selection: ReadSelection, + deadline: u64, + mut node: Permit, + mut group: Permit, + ) -> Result { + let bytes = usize::try_from(selection.cell.geometry.length) + .ok() + .and_then(|n| n.checked_add(4096)) + .ok_or(RuntimeError::Overloaded)?; + if !node.grow_to(bytes) || !group.grow_to(bytes) { + return Err(RuntimeError::Overloaded); + } + let deadline = selection + .source_deadline + .map_or(deadline, |source| deadline.min(source)); + Ok(Self { + selection, + deadline, + _node: node, + _group: group, + }) + } + + pub fn cell(&self) -> &CellManifestLeaf { + &self.selection.cell + } + + pub fn replica(&self) -> &cellar_logic::consensus::ReplicaIdentity { + self.selection.group.identity() + } + + pub fn route(&self) -> &GroupRoute { + &self.selection.route + } + + pub fn applied(&self) -> LogPosition { + self.selection.applied + } + + pub fn barrier(&self) -> Option { + self.selection.barrier + } + + pub fn deadline(&self) -> u64 { + self.deadline + } + + pub(crate) fn image(&self) -> &CellImagePin { + &self.selection.image + } + + pub(crate) fn belongs_to_engine(&self, engine: &Arc<()>) -> bool { + Arc::ptr_eq(engine, &self.selection.group.owner) + } + + pub(crate) fn check(&self, now: u64) -> Result<(), RuntimeError> { + if now >= self.deadline { + return Err(RuntimeError::DeadlineExceeded); + } + let s = &self.selection; + let cache = s.group.read()?; + if cache + .group_manifest + .as_ref() + .is_none_or(|m| m.route() != &s.route || !m.is_active()) + || !cache.membership.permits_group(s.group.identity()) + || s.source.as_ref().is_some_and(|(term, epoch, source)| { + cache.hard_state.term != term.get() + || *epoch == 0 + || source.load(Ordering::Acquire) != *epoch + }) + || s.leader.as_ref().is_some_and(|(term, leader)| { + cache.hard_state.term != term.get() + || leader.load(Ordering::Acquire) != term.get() + || !cache + .conf_state + .voters + .iter() + .chain(&cache.conf_state.voters_outgoing) + .any(|id| *id == s.group.identity().member.get()) + }) + { + return Err(RuntimeError::NotReady); + } + Ok(()) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/reclaim.rs b/crates/cellar/durability/raft/cell_groups/reclaim.rs new file mode 100644 index 000000000..105a889f4 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/reclaim.rs @@ -0,0 +1,346 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Reconcile all engine-selected closures before bounded orphan collection. +use super::*; +use crate::raft::{cache::GroupLog, execution::ExecutionError, storage::LogError}; +use cellar_logic::cell_group::GroupHistory; +use cellar_ltx::{ + generation::{GenerationError, GenerationStore, ImageId}, + internal::{ExtentRootReference, NodeExtentStore, StoreError, EXTENT_BYTES}, +}; +use std::{ + collections::{BTreeMap, BTreeSet}, + sync::{Arc, TryLockError}, +}; + +/// One maintenance request's aggregate bounds, including duplicate retained cuts. +#[derive(Clone, Copy, Debug)] +pub struct CellGroupReclaimLimits { + pub max_selected_cells: u32, + pub max_files: usize, + pub max_page_scan: usize, +} + +impl Default for CellGroupReclaimLimits { + fn default() -> Self { + Self { + max_selected_cells: 4096, + max_files: 256, + max_page_scan: 256, + } + } +} + +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub struct CellGroupReclaimReport { + pub groups: usize, + pub selected_roots: usize, + pub selected_cells: u32, + pub pages_scanned: usize, + pub page_files: usize, + pub image_files: usize, +} + +#[derive(Debug)] +pub enum CellGroupReclaimError { + Invalid(&'static str), + Limit, + Busy, + Log(LogError), + Recovery(ExecutionError), + Manifest(ManifestError), + Image(GenerationError), + Extent(StoreError), +} + +impl std::fmt::Display for CellGroupReclaimError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "cell-group storage reclamation: {self:?}") + } +} + +impl std::error::Error for CellGroupReclaimError { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + match self { + Self::Log(e) => Some(e), + Self::Recovery(e) => Some(e), + Self::Manifest(e) => Some(e), + Self::Image(e) => Some(e), + Self::Extent(e) => Some(e), + _ => None, + } + } +} + +type Result = std::result::Result; + +pub(crate) struct ReclaimCells { + appliers: Vec, + retained: Vec, + limits: CellGroupReclaimLimits, +} + +impl ReclaimCells { + pub(crate) fn new( + mut appliers: Vec, + retained: Vec, + limits: CellGroupReclaimLimits, + ) -> Result { + if appliers.is_empty() + || appliers.len() > 4096 + || retained.len() > 4096 + || !(1..=1_000_000).contains(&limits.max_selected_cells) + || !(1..=4096).contains(&limits.max_files) + || !(1..=4096).contains(&limits.max_page_scan) + { + return Err(CellGroupReclaimError::Limit); + } + appliers.sort_by_key(CellGroupApplier::history); + if appliers + .windows(2) + .any(|a| a[0].history() == a[1].history()) + { + return Err(CellGroupReclaimError::Invalid("duplicate group applier")); + } + Ok(Self { + appliers, + retained, + limits, + }) + } + + pub(crate) fn reservation_bytes(&self) -> Result { + // Full recovered trees, reference inventories, one extent verifier and + // bounded group/retained-root owners stay charged through physical work. + ManifestPins::maximum_bytes(self.limits.max_selected_cells) + .checked_mul(2) + .and_then(|n| n.checked_add(self.appliers.len().checked_mul(4096)?)) + .and_then(|n| n.checked_add(self.retained.len().checked_mul(1024)?)) + .and_then(|n| n.checked_add(EXTENT_BYTES as usize + 65536)) + .ok_or(CellGroupReclaimError::Limit) + } + + /// Runs on the engine worker: group creation/publication cannot interleave. + pub(crate) fn run( + self, + engine: &Arc<()>, + groups: Vec>, + ) -> Result { + if groups.len() != self.appliers.len() + || groups.iter().zip(&self.appliers).any(|(g, a)| { + g.identity() != a.group.identity() + || !Arc::ptr_eq(&g.owner, engine) + || !a.belongs_to_engine(engine) + }) + { + return Err(CellGroupReclaimError::Invalid( + "incomplete or foreign engine inventory", + )); + } + let mut owners = self + .appliers + .iter() + .map(|a| { + a.storage.try_lock().map_err(|e| match e { + TryLockError::WouldBlock => CellGroupReclaimError::Busy, + TryLockError::Poisoned(_) => { + CellGroupReclaimError::Invalid("poisoned group storage") + } + }) + }) + .collect::>>()?; + let mut report = CellGroupReclaimReport { + groups: groups.len(), + ..Default::default() + }; + // Refuse the aggregate cell bound before recovery allocates any trees. + for group in &groups { + let current = group + .group_manifest() + .map_err(CellGroupReclaimError::Log)? + .ok_or(CellGroupReclaimError::Invalid("selected manifest missing"))?; + let checkpoint = group + .checkpoint_manifest() + .map_err(CellGroupReclaimError::Log)?; + for marker in std::iter::once(¤t).chain(checkpoint.as_ref()) { + report.selected_cells = report + .selected_cells + .checked_add(marker.root().cells()) + .filter(|n| *n <= self.limits.max_selected_cells) + .ok_or(CellGroupReclaimError::Limit)?; + report.selected_roots += 1; + } + } + for root in &self.retained { + report.selected_cells = report + .selected_cells + .checked_add(root.marker().root().cells()) + .filter(|n| *n <= self.limits.max_selected_cells) + .ok_or(CellGroupReclaimError::Limit)?; + report.selected_roots += 1; + } + let mut roots = Vec::with_capacity(groups.len()); + let mut stores = Vec::::new(); + for (applier, owner) in self.appliers.iter().zip(&mut owners) { + let (current, checkpoint) = applier + .recover_roots(owner) + .map_err(CellGroupReclaimError::Recovery)?; + let mut selected = vec![current]; + selected.extend(checkpoint); + let extras: Vec<_> = self + .retained + .iter() + .filter(|r| r.marker().route().history == applier.history()) + .cloned() + .collect(); + for extra in &extras { + if !Arc::ptr_eq(&extra.engine_owner, engine) + || !Arc::ptr_eq(&extra.page_owner, &owner.pages.owner) + { + return Err(CellGroupReclaimError::Invalid("retained root owner")); + } + } + selected.extend(extras); + let index = match stores.iter().position(|s| s.same(&owner.images)) { + Some(index) => index, + None => { + stores.push(Images::new(&owner.images)); + stores.len() - 1 + } + }; + for root in &selected { + stores[index].retain(root)?; + } + roots.push(selected); + } + if self.retained.iter().any(|r| { + !self + .appliers + .iter() + .any(|a| a.history() == r.marker().route().history) + }) { + return Err(CellGroupReclaimError::Invalid( + "retained root outside engine", + )); + } + // No deletion until every selected closure and owner has been checked. + for store in &stores { + store.reconcile()?; + } + for (applier, owner) in self.appliers.iter().zip(&mut owners) { + let extras: Vec<_> = self + .retained + .iter() + .filter(|r| r.marker().route().history == applier.history()) + .cloned() + .collect(); + owner + .pages + .reconcile_retained(&extras) + .map_err(CellGroupReclaimError::Manifest)?; + } + let mut files = self.limits.max_files; + let mut scan = self.limits.max_page_scan; + for owner in &mut owners { + if scan == 0 || files == 0 { + break; + } + let pages = owner + .pages + .reclaim(scan.min(files)) + .map_err(CellGroupReclaimError::Manifest)?; + scan -= pages.scanned; + files -= pages.pages; + report.pages_scanned += pages.scanned; + report.page_files += pages.pages; + } + for store in &stores { + if files == 0 { + break; + } + let removed = store.reclaim(files)?; + files -= removed; + report.image_files += removed; + } + // Retain every recovered tree until all file operations finish. + drop(roots); + Ok(report) + } +} + +enum Images { + Materialized(GenerationStore, BTreeSet), + Extents(NodeExtentStore, BTreeMap<[u8; 32], ExtentRootReference>), +} + +impl Images { + fn new(images: &CellGroupApplyImages) -> Self { + match images { + CellGroupApplyImages::Materialized { images, .. } => { + Self::Materialized(images.clone(), BTreeSet::new()) + } + CellGroupApplyImages::Extents { images, .. } => { + Self::Extents(images.clone(), BTreeMap::new()) + } + } + } + + fn same(&self, images: &CellGroupApplyImages) -> bool { + match (self, images) { + (Self::Materialized(a, _), CellGroupApplyImages::Materialized { images: b, .. }) => { + a.same_inventory(b) + } + (Self::Extents(a, _), CellGroupApplyImages::Extents { images: b, .. }) => { + a.same_inventory(b) + } + _ => false, + } + } + + fn retain(&mut self, root: &PreparedGroupManifest) -> Result<()> { + root.images.visit(&mut |_, _, pin, _, _| { + match (&mut *self, pin) { + (_, CellImagePin::Empty) => (), + (Self::Materialized(store, refs), CellImagePin::Materialized(pin)) + if store.owns(pin) => + { + refs.insert(pin.id()); + } + (Self::Extents(store, refs), CellImagePin::Extents(pin)) + if store.owns_image(pin) => + { + let reference = pin.reference(); + if refs + .insert(reference.root().bytes(), reference) + .is_some_and(|old| old != reference) + { + return Err(CellGroupReclaimError::Invalid("conflicting extent root")); + } + } + _ => return Err(CellGroupReclaimError::Invalid("image inventory owner")), + } + Ok(()) + }) + } + + fn reconcile(&self) -> Result<()> { + match self { + Self::Materialized(store, refs) => store + .retain_only(&refs.iter().copied().collect::>()) + .map_err(CellGroupReclaimError::Image), + Self::Extents(store, refs) => store + .retain_only(&refs.values().copied().collect::>()) + .map_err(CellGroupReclaimError::Extent), + } + } + + fn reclaim(&self, files: usize) -> Result { + match self { + Self::Materialized(store, _) => store + .reclaim(files) + .map(|r| r.files) + .map_err(CellGroupReclaimError::Image), + Self::Extents(store, _) => store.reclaim(files).map_err(CellGroupReclaimError::Extent), + } + } +} diff --git a/crates/cellar/durability/raft/cell_groups/recovery/admission.rs b/crates/cellar/durability/raft/cell_groups/recovery/admission.rs new file mode 100644 index 000000000..830aac927 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/recovery/admission.rs @@ -0,0 +1,89 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Separate import admission: the held source already owns archive replay and +//! checkpoint permits. Reacquiring either single-slot pool would deadlock. +use super::super::{CellGroupArchiveError, CellGroupArchiveUsage, ManifestPins}; +use crate::raft::budget::{Budget, Permit}; +use std::sync::Arc; + +#[derive(Clone, Copy, Debug)] +pub struct CellGroupRecoveryLimits { + pub max_operations: usize, + pub max_retained_bytes: usize, + pub max_cells: u32, + pub max_image_bytes: u64, + pub max_restore_bytes: u64, +} + +impl Default for CellGroupRecoveryLimits { + fn default() -> Self { + Self { + max_operations: 1, + max_retained_bytes: 256 * 1024 * 1024, + max_cells: 64, + max_image_bytes: 256 * 1024 * 1024, + max_restore_bytes: 1024 * 1024 * 1024, + } + } +} + +/// Share across the node. The returned preparation retains admission until its +/// physical pins are dropped; cancellation does not release entered worker debt. +#[derive(Clone)] +pub struct CellGroupRecoveryPool { + pub(super) limits: CellGroupRecoveryLimits, + pub(super) budget: Arc, + workspace: usize, +} + +impl CellGroupRecoveryPool { + pub fn new(limits: CellGroupRecoveryLimits) -> Result { + if !(1..=4096).contains(&limits.max_operations) + || !(1..=1_000_000).contains(&limits.max_cells) + || !(512..=cellar_ltx::stream_snapshot::MAX_IMAGE_BYTES) + .contains(&limits.max_image_bytes) + || limits.max_restore_bytes < limits.max_image_bytes + { + return Err(CellGroupArchiveError::Limit); + } + // Source metadata has its own retained archive admission. Account the + // target tree, temporary AVL paths, both verification walks, all extent + // references and two full extent buffers without duplicating an inventory. + let extents_per_image = limits + .max_image_bytes + .div_ceil(cellar_ltx::internal::EXTENT_BYTES); + let workspace = usize::try_from(extents_per_image) + .ok() + .and_then(|count| count.checked_mul(1024)) + .and_then(|bytes| bytes.checked_add(1024 * 1024)) + .and_then(|bytes| bytes.checked_mul(limits.max_cells as usize)) + .and_then(|bytes| { + bytes.checked_add(ManifestPins::maximum_bytes(limits.max_cells).checked_mul(4)?) + }) + .and_then(|bytes| bytes.checked_add(3 * cellar_ltx::internal::EXTENT_BYTES as usize)) + .ok_or(CellGroupArchiveError::Limit)?; + if workspace > limits.max_retained_bytes { + return Err(CellGroupArchiveError::Limit); + } + Ok(Self { + limits, + budget: Budget::new(limits.max_retained_bytes, limits.max_operations), + workspace, + }) + } + + pub fn usage(&self) -> CellGroupArchiveUsage { + let (retained_bytes, operations) = self.budget.usage(); + CellGroupArchiveUsage { + retained_bytes, + operations, + } + } + + pub(super) fn admit(&self) -> Result, CellGroupArchiveError> { + self.budget + .acquire(self.workspace) + .map(Arc::new) + .ok_or(CellGroupArchiveError::Busy) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/recovery/genesis.rs b/crates/cellar/durability/raft/cell_groups/recovery/genesis.rs new file mode 100644 index 000000000..53586e680 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/recovery/genesis.rs @@ -0,0 +1,204 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Catalog-authorized, fresh Raft genesis. The node writer owns the sync and +//! physical root publication; neither an archive owner nor an image is a grant. +use super::super::*; +use crate::raft::{cache::GroupLog, catalog::RecoverySeedPermit, storage::LogError}; +use cellar_logic::{ + cell_group::GroupHistory, + consensus::{EngineInstanceId, ReplicaIdentity}, +}; +use std::sync::{atomic::AtomicBool, Arc}; + +pub(crate) struct RecoverySeed { + permit: RecoverySeedPermit, + prepared: super::PreparedRecoveryData, +} + +impl RecoverySeed { + pub(crate) fn new( + permit: RecoverySeedPermit, + prepared: super::PreparedRecoveryData, + ) -> Result { + if prepared.manifest.marker().recovery_request() != Some(permit.request()) + || prepared.manifest.marker() != permit.genesis() + || prepared.commitment != permit.commitment() + || !prepared.store.owner.owns(&prepared.manifest) + { + return Err(LogError::WrongIdentity); + } + Ok(Self { permit, prepared }) + } + + pub(crate) fn bootstrap(&self) -> &crate::raft::storage::CellGroupBootstrap { + self.permit.bootstrap() + } + + pub(crate) fn genesis(&self) -> &GroupManifest { + self.permit.genesis() + } + + pub(crate) fn check( + &self, + owner: &Arc<()>, + identity: &ReplicaIdentity, + engine: EngineInstanceId, + ) -> Result<(), LogError> { + self.permit.check(owner, identity, engine) + } + + /// Reverify the complete closure on this actual engine's blocking writer. + /// Hold the source/import proof until the writer finishes even on a lost reply. + pub(crate) fn managed_binding( + &self, + guard: Option<&crate::raft::fleet_stop::Guard>, + now: u64, + ) -> Result, LogError> { + self.permit.managed_binding(guard, now) + } + + pub(crate) fn prepare( + &self, + closed: &Arc, + ) -> Result< + ( + PreparedGroupManifest, + crate::raft::cell_groups::retention::PublicationPin, + ), + LogError, + > { + let mut owner = self + .prepared + .store + .storage + .try_lock() + .map_err(|error| match error { + std::sync::TryLockError::WouldBlock => LogError::Overloaded, + std::sync::TryLockError::Poisoned(_) => { + LogError::Io("poisoned recovery store".into()) + } + })?; + let apply::Storage { tree, pages, .. } = &mut *owner; + let root = tree + .prepare_recovery_genesis(&self.permit, pages, &self.prepared.manifest) + .map_err(preparation_error)?; + let publication = root + .begin_genesis(&self.permit, closed) + .map_err(preparation_error)?; + // Revoke further archive admission. Entered immutable I/O retains its + // own permits/pins, while the engine now owns this page-store lifetime. + self.prepared.store.close(); + Ok((root, publication)) + } +} + +/// One synced physical seed and independently confirmed catalog receipt. +/// It has no startup, voting or application permission. The directory must +/// select the replacement, then startup and a fresh committed activation follow. +#[derive(Clone)] +pub struct SeededCellGroupRecovery { + request: Arc, + group: GroupLog, + applier: CellGroupApplier, + root: PreparedGroupManifest, + binding: [u8; 32], +} + +impl SeededCellGroupRecovery { + pub(crate) fn new( + request: Arc, + group: GroupLog, + applier: CellGroupApplier, + root: PreparedGroupManifest, + binding: [u8; 32], + ) -> Self { + Self { + request, + group, + applier, + root, + binding, + } + } + + pub fn request(&self) -> &CellGroupRecoveryRequest { + &self.request + } + + pub fn group(&self) -> &GroupLog { + &self.group + } + + pub fn applier(&self) -> &CellGroupApplier { + &self.applier + } + + pub fn manifest(&self) -> &PreparedGroupManifest { + &self.root + } + + pub fn binding(&self) -> [u8; 32] { + self.binding + } +} + +impl CellGroupApplier { + /// Preserve the already opened physical owner during archive-to-engine + /// transfer. Only a checked recovery grant invokes this boundary. + pub(crate) fn recover_seeded( + group: GroupLog, + store: &CellGroupArchiveStore, + genesis: &GroupManifest, + ) -> Result<(Self, PreparedGroupManifest), crate::raft::execution::ExecutionError> { + if group + .group_manifest() + .map_err(crate::raft::execution::storage)? + .as_ref() + != Some(genesis) + || store.route() != genesis.route() + || !genesis.is_recovery_genesis() + { + return Err(crate::raft::execution::ExecutionError::Invalid( + "recovery seed selection", + )); + } + store.close(); + let applier = Self { + group, + storage: store.storage.clone(), + }; + let root = applier.recover_selected()?; + // A same-process retry can encounter cached seals. Confirm bytes and + // persistence independently before a permanent seed receipt is written. + root.images.visit(&mut |_, _, pin, _, _| { + match pin { + CellImagePin::Materialized(pin) => pin + .verify_and_sync() + .map_err(crate::raft::execution::storage)?, + CellImagePin::Extents(pin) => { + for index in 0..pin.manifest().geometry().extent_count() { + let extent = pin.manifest().extent(index).ok_or( + crate::raft::execution::ExecutionError::Invalid("recovery extent"), + )?; + pin.view() + .read_extent(extent) + .map_err(crate::raft::execution::storage)?; + } + } + CellImagePin::Empty => (), + } + Ok::<_, crate::raft::execution::ExecutionError>(()) + })?; + Ok((applier, root)) + } +} + +pub(super) fn preparation_error(error: ManifestError) -> LogError { + match error { + ManifestError::Limit => LogError::Overloaded, + ManifestError::Conflict => LogError::WrongIdentity, + ManifestError::Invalid(message) => LogError::Invalid(message.into()), + ManifestError::Quarantined => LogError::LostStableState, + ManifestError::Io(error) => LogError::Io(error.to_string()), + } +} diff --git a/crates/cellar/durability/raft/cell_groups/recovery/import.rs b/crates/cellar/durability/raft/cell_groups/recovery/import.rs new file mode 100644 index 000000000..fe00428b3 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/recovery/import.rs @@ -0,0 +1,365 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Fresh-history data preparation. No engine mapping, catalog publication, +//! membership, callback, log acknowledgement or serving permission is created. +use super::{super::*, CellGroupRecoveryPool}; +use cellar_logic::{cell_group::*, consensus::*}; +use cellar_ltx::{ + internal::{ImageBinding, EXTENT_BYTES}, + stream_snapshot::ScratchPool, +}; +use std::{ + path::PathBuf, + sync::{ + atomic::{AtomicBool, Ordering}, + Arc, + }, +}; + +type Result = std::result::Result; + +/// Complete imported data under fresh histories. Clones retain the same +/// physical owners and admission. This grants no engine or serving authority; +/// only a catalog-authorized genesis installation may select it in a Raft log. +#[derive(Clone)] +pub struct PreparedCellGroupRecovery { + fence: FencedCellGroupRecovery, + data: PreparedRecoveryData, +} + +#[derive(Clone)] +pub(crate) struct PreparedRecoveryData { + pub(crate) store: CellGroupArchiveStore, + pub(crate) manifest: PreparedGroupManifest, + pub(crate) commitment: [u8; 32], + _source: CellGroupArchiveCoverage, +} + +impl PreparedRecoveryData { + pub(crate) fn image_bytes(&self) -> Result { + let mut bytes = 0u64; + self.manifest + .images + .visit(&mut |_, leaf, _, _, _| -> Result<()> { + bytes = bytes + .checked_add(leaf.geometry.length) + .ok_or(CellGroupArchiveError::Limit)?; + Ok(()) + })?; + Ok(bytes) + } +} + +impl PreparedCellGroupRecovery { + pub(crate) fn data(&self) -> &PreparedRecoveryData { + &self.data + } + + pub fn request(&self) -> &CellGroupRecoveryRequest { + self.fence.request() + } + + pub fn source(&self) -> &FencedCellGroupRecovery { + &self.fence + } + + pub fn store(&self) -> &CellGroupArchiveStore { + &self.data.store + } + + pub fn manifest(&self) -> &PreparedGroupManifest { + &self.data.manifest + } + + pub fn commitment(&self) -> [u8; 32] { + self.data.commitment + } +} + +pub(super) struct Stop(pub(super) Arc); + +impl Drop for Stop { + fn drop(&mut self) { + self.0.store(true, Ordering::Release); + } +} + +impl CellGroupRecoveryPool { + /// Copy the complete fenced cut into dedicated replacement stores, with + /// new cell histories and genesis witnesses. SQLite bytes (including receipt + /// results, client floors and effect identity) remain exact. The source hold + /// is provenance and must remain retained after this preparation is dropped. + pub async fn prepare( + &self, + domain: &impl CellGroupArchiveExecution, + fence: FencedCellGroupRecovery, + target: CellGroupArchiveStore, + scratch: ScratchPool, + directory: PathBuf, + ) -> Result { + if target.route() != fence.request().replacement().route() { + return Err(CellGroupArchiveError::Invalid( + "recovery destination binding", + )); + } + let data = Box::pin(self.prepare_cut( + domain, + fence.coverage().clone(), + CellGroupTransferTarget { + store: target, + scratch, + directory, + }, + fence.request().digest(), + fence.request().operation(), + )) + .await?; + Ok(PreparedCellGroupRecovery { fence, data }) + } + + pub(crate) async fn prepare_cut( + &self, + domain: &impl CellGroupArchiveExecution, + source_cut: CellGroupArchiveCoverage, + target: CellGroupTransferTarget, + binding: [u8; 32], + operation: ProposalId, + ) -> Result { + let CellGroupTransferTarget { + store: target, + scratch, + directory, + } = target; + domain + .run(async { + let admission = self.admit()?; + let stop = Stop(Arc::new(AtomicBool::new(false))); + let cancelled = stop.0.clone(); + let limits = self.limits; + domain + .blocking(move || { + let check = || { + if cancelled.load(Ordering::Acquire) { + return Err(CellGroupArchiveError::Stopped); + } + target.require_live()?; + source_cut + .recovery() + .checkpoint() + .checkpoint() + .store() + .require_live()?; + Ok(()) + }; + check()?; + let source = source_cut.recovery().manifest(); + let old = source.marker().route(); + let new = target.route(); + if !directory.is_absolute() + || old.range != new.range + || old.schema_digest != new.schema_digest + || old.history.cluster != new.history.cluster + || old.history.namespace != new.history.namespace + || old.history.group == new.history.group + || old.history.incarnation == new.history.incarnation + { + return Err(CellGroupArchiveError::Invalid( + "recovery destination binding", + )); + } + source_cut + .recovery() + .check_recovery_cell_incarnation(new.history.incarnation)?; + if source.pinned_images() > limits.max_cells as usize { + return Err(CellGroupArchiveError::Limit); + } + let mut total = 0u64; + source.images.visit(&mut |_, leaf, _, _, _| -> Result<()> { + if leaf.geometry.length > limits.max_image_bytes { + return Err(CellGroupArchiveError::Limit); + } + total = total + .checked_add(leaf.geometry.length) + .filter(|n| *n <= limits.max_restore_bytes) + .ok_or(CellGroupArchiveError::Limit)?; + Ok(()) + })?; + let mut owner = target.storage.try_lock().map_err(|error| match error { + std::sync::TryLockError::WouldBlock => CellGroupArchiveError::Busy, + std::sync::TryLockError::Poisoned(_) => { + CellGroupArchiveError::Invalid("poisoned recovery destination") + } + })?; + let apply::Storage { + tree, + pages, + images, + .. + } = &mut *owner; + let mut pins = ManifestPins::default(); + let mut root = ManifestRoot::default(); + source + .images + .visit(&mut |_, old, pin, _, _| -> Result<()> { + check()?; + let mut leaf = old.clone(); + leaf.version.history.incarnation = + target.route().history.incarnation; + leaf.version.sequence = MutationSequence::new(1).map_err(|_| { + CellGroupArchiveError::Invalid("recovery sequence") + })?; + let pin = copy_image( + images, pin, &mut leaf, &scratch, &directory, &check, + )?; + leaf.enrollment = EnrollmentRecord::new( + leaf.version.history.clone(), + operation, + target.route().schema_digest, + leaf.version.image, + ) + .map_err(CellGroupArchiveError::Enrollment)?; + leaf.last_mutation = LogPosition { + index: LogIndex::new(1), + term: Term::new(1), + }; + leaf.witness = CommitWitness { + group: target.route().history, + position: leaf.last_mutation, + proposal: operation, + }; + root = tree + .update(pages, root, leaf.clone(), None) + .map_err(manifest_error)?; + pins = pins + .updated(tree, pages, root, &leaf, &pin) + .map_err(manifest_error)?; + Ok(()) + })?; + let marker = GroupManifest::recovery_genesis( + target.route().clone(), + root, + binding, + operation, + ) + .map_err(manifest_error)?; + let commitment = marker + .recovery + .as_deref() + .ok_or(CellGroupArchiveError::Invalid("recovery lifecycle missing"))? + .seed_commitment(&marker) + .map_err(manifest_error)?; + let mut manifest = tree + .prepare_archive_complete(&target.owner, pages, marker, &mut |leaf| { + let key = crate::application::consensus::cell_routing_key( + target.route().history.namespace, + &leaf.version.history.cell, + ); + pins.get(&key) + .filter(|(selected, _)| *selected == leaf) + .map(|(_, pin)| pin) + .ok_or(ManifestError::Invalid("recovery closure changed")) + }) + .map_err(manifest_error)?; + manifest.recovery_admission = Some(admission); + check()?; + drop(owner); + Ok(PreparedRecoveryData { + _source: source_cut, + store: target, + manifest, + commitment, + }) + }) + .map_err(|_| CellGroupArchiveError::Stopped)? + .await + .map_err(CellGroupArchiveError::Worker)? + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } +} + +fn manifest_error(error: ManifestError) -> CellGroupArchiveError { + CellGroupCheckpointError::from(error).into() +} + +fn physical(error: impl Into) -> CellGroupArchiveError { + error.into().into() +} + +pub(super) fn copy_image( + images: &CellGroupApplyImages, + source: &CellImagePin, + leaf: &mut CellManifestLeaf, + scratch: &ScratchPool, + directory: &std::path::Path, + check: &impl Fn() -> Result<()>, +) -> Result { + match (images, source) { + (_, CellImagePin::Empty) => Ok(CellImagePin::Empty), + (CellGroupApplyImages::Materialized { images, .. }, CellImagePin::Materialized(source)) => { + source.verify_and_sync().map_err(physical)?; + let state = source.state(); + let id = + apply::images::image_id(&leaf.version.history, leaf.version.sequence.get(), state) + .map_err(CellGroupArchiveError::Replay)?; + let mut receive = images + .receive_image(id, state, scratch, directory) + .map_err(physical)?; + let mut reader = source.open_reader().map_err(physical)?; + let mut offset = 0; + while offset < state.length { + check()?; + let bytes = reader + .read_exact_at(offset, (state.length - offset).min(64 * 1024) as usize) + .map_err(physical)?; + receive.append(offset, &bytes).map_err(physical)?; + offset += bytes.len() as u64; + } + Ok(CellImagePin::Materialized( + receive.finish().map_err(physical)?, + )) + } + (CellGroupApplyImages::Extents { images, .. }, CellImagePin::Extents(source)) => { + let old = source.manifest(); + let manifest = old + .with_updates( + ImageBinding { + history: cell_image_history_digest(&leaf.version.history), + ..old.binding() + }, + old.geometry(), + &[], + ) + .map_err(cellar_ltx::internal::StoreError::from) + .map_err(physical)?; + // Protect introduced object identities from concurrent collection + // before the first copy, including deduplicated existing extents. + let staged = images + .stage_transient_root(manifest.clone()) + .map_err(physical)?; + for index in 0..old.geometry().extent_count() { + check()?; + let id = old + .extent(index) + .ok_or(CellGroupArchiveError::Invalid("source extent missing"))?; + let bytes = source.view().read_extent(id).map_err(physical)?; + images.put_extent_bytes(id, &bytes).map_err(physical)?; + } + let pin = Arc::new( + images + .seal_image_root(manifest, &mut vec![0; EXTENT_BYTES as usize]) + .map_err(physical)?, + ); + drop(staged); + leaf.version.image.commitment = CellImageCommitment::Extents { + root: pin.reference().root().bytes(), + manifest_bytes: pin.reference().encoded_length(), + }; + Ok(CellImagePin::Extents(pin)) + } + _ => Err(CellGroupArchiveError::Invalid( + "recovery image representation", + )), + } +} diff --git a/crates/cellar/durability/raft/cell_groups/recovery/mod.rs b/crates/cellar/durability/raft/cell_groups/recovery/mod.rs new file mode 100644 index 000000000..2a535465e --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/recovery/mod.rs @@ -0,0 +1,25 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Fresh-history restoration and activation. Archive integrity, infrastructure +//! fencing, physical genesis and a committed serving decision are distinct proofs. +mod admission; +mod genesis; +mod transfer_genesis; +pub(crate) use transfer_genesis::TransferSeed; +mod import; +mod partition; +mod state; +pub use admission::{CellGroupRecoveryLimits, CellGroupRecoveryPool}; +pub(crate) use genesis::RecoverySeed; +pub use genesis::SeededCellGroupRecovery; +pub use import::PreparedCellGroupRecovery; +pub(crate) use import::PreparedRecoveryData; +pub(crate) use partition::StagedCellGroupPartition; +pub use partition::{ + CellGroupTransferTarget, PreparedCellGroupTransfer, PreparedCellGroupTransferChild, +}; +pub(super) use state::Activation; +pub(super) use state::RecoveryState; + +#[cfg(test)] +mod tests; diff --git a/crates/cellar/durability/raft/cell_groups/recovery/partition.rs b/crates/cellar/durability/raft/cell_groups/recovery/partition.rs new file mode 100644 index 000000000..c507b0d05 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/recovery/partition.rs @@ -0,0 +1,442 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Copy exact sealed cell data into bounded, non-serving child stores. +use super::{ + super::*, + import::{copy_image, Stop}, + CellGroupRecoveryLimits, CellGroupRecoveryPool, +}; +use crate::application::consensus::cell_routing_key; +use cellar_logic::{ + cell_group::{CellId, GroupHistory, GroupRoute}, + consensus::{LogIndex, LogPosition, Term}, +}; +use cellar_ltx::stream_snapshot::ScratchPool; +use std::{ + path::PathBuf, + sync::{ + atomic::{AtomicBool, Ordering}, + Arc, OnceLock, + }, +}; + +type Result = std::result::Result; + +/// Dedicated child preparation stores. The store route must be an exact +/// child of the selected transition, in the original plan order. +pub struct CellGroupTransferTarget { + pub store: CellGroupArchiveStore, + pub scratch: ScratchPool, + pub directory: PathBuf, +} + +/// Prepared physical data under the child's route. No log, membership, +/// engine or application owner exists at this boundary. +pub struct PreparedCellGroupTransferChild { + data: ChildData, + store: CellGroupArchiveStore, +} + +enum ChildData { + PreCopy(ManifestPins), + Sealed { + genesis: Box, + engine: Arc>>, + }, +} + +impl PreparedCellGroupTransferChild { + pub(super) fn store(&self) -> &CellGroupArchiveStore { + &self.store + } + + pub fn route(&self) -> &GroupRoute { + self.store.route() + } + + pub fn root(&self) -> ManifestRoot { + match &self.data { + ChildData::PreCopy(pins) => pins.root(), + ChildData::Sealed { genesis, .. } => genesis.marker().root(), + } + } + + pub fn pin_cell(&self, cell: &CellId) -> Option<(&CellManifestLeaf, CellImagePin)> { + match &self.data { + ChildData::PreCopy(pins) => { + pins.get(&cell_routing_key(self.route().history.namespace, cell)) + } + ChildData::Sealed { genesis, .. } => genesis.pin_cell(cell), + } + } + + /// A synced non-serving genesis exists only for the final sealed-source + /// partition. The archive store owns it until an engine-bound install. + pub fn genesis(&self) -> Option<&PreparedGroupManifest> { + match &self.data { + ChildData::PreCopy(_) => None, + ChildData::Sealed { genesis, .. } => Some(genesis), + } + } + + pub(super) fn install_owner(&self) -> Option<&Arc>>> { + match &self.data { + ChildData::PreCopy(_) => None, + ChildData::Sealed { engine, .. } => Some(engine), + } + } + + /// Open the applier on the exact page store handed to the durable engine. + /// Recover and verify the selected root on a blocking owner before attach. + pub fn installed_applier( + &self, + permit: &CellGroupBootstrapPermit, + ) -> std::result::Result { + if permit.route() != self.route() { + return Err(crate::raft::completion::RuntimeError::StaleCompletion); + } + self.installed_applier_on(permit.group()) + } + + pub(crate) fn installed_applier_on( + &self, + group: &crate::raft::cache::GroupLog, + ) -> std::result::Result { + use crate::raft::completion::RuntimeError; + if group.identity().scope != self.route().history + || !self + .install_owner() + .and_then(|engine| engine.get()) + .is_some_and(|engine| Arc::ptr_eq(engine, &group.owner)) + { + return Err(RuntimeError::StaleCompletion); + } + Ok(CellGroupApplier { + group: group.clone(), + storage: self.store.storage.clone(), + }) + } +} + +/// Complete staged partition with both the original source coverage and the +/// new child pins held until joined use ends. The transition owner retains +/// the original remote holds separately until their durable retirement. +pub struct PreparedCellGroupTransfer { + partition: StagedCellGroupPartition, + sources: VerifiedCellGroupTransitionSources, +} + +pub(crate) struct StagedCellGroupPartition { + pub(crate) children: Vec, + _admission: Arc, +} + +impl PreparedCellGroupTransfer { + pub fn children(&self) -> &[PreparedCellGroupTransferChild] { + &self.partition.children + } + + pub fn sources(&self) -> &VerifiedCellGroupTransitionSources { + &self.sources + } +} + +impl CellGroupRecoveryPool { + /// Materialize every child from the directory-selected held parent cuts. + /// Preserve the logical image, version witness, enrollment and effect + /// summary. Rebase only the physical installation position to genesis. + /// A prepared child still requires a separate committed install and activation. + pub async fn prepare_transfer_partition( + &self, + domain: &impl CellGroupArchiveExecution, + sources: VerifiedCellGroupTransitionSources, + targets: Vec, + ) -> Result { + let origin = sealed_origin(&sources)?; + let partition = self + .stage_partition( + domain, + sources.sources().to_vec(), + sources.children().to_vec(), + targets.into_iter().map(Some).collect(), + Some(origin), + ) + .await?; + Ok(PreparedCellGroupTransfer { partition, sources }) + } + + /// Use the complete verified partition to prepare only the local child. + /// Cells in other ranges are validated against the plan without copying them. + pub(crate) async fn prepare_transfer_child( + &self, + domain: &impl CellGroupArchiveExecution, + sources: &VerifiedCellGroupTransitionSources, + target: CellGroupTransferTarget, + ) -> Result { + let origin = sealed_origin(sources)?; + let routes = sources.children(); + let selected = routes + .iter() + .position(|route| route == target.store.route()) + .ok_or(CellGroupArchiveError::Invalid("unselected transfer child"))?; + let mut targets: Vec<_> = (0..routes.len()).map(|_| None).collect(); + targets[selected] = Some(target); + let mut partition = self + .stage_partition( + domain, + sources.sources().to_vec(), + routes.to_vec(), + targets, + Some(origin), + ) + .await?; + partition + .children + .pop() + .ok_or(CellGroupArchiveError::Invalid("missing prepared child")) + } + + pub(crate) async fn stage_partition( + &self, + domain: &impl CellGroupArchiveExecution, + sources: Vec, + routes: Vec, + targets: Vec>, + origin: Option, + ) -> Result { + domain + .run(Box::pin(async { + let admission = self.admit()?; + let stop = Stop(Arc::new(AtomicBool::new(false))); + let cancelled = stop.0.clone(); + let limits = self.limits; + let work = domain + .blocking(move || { + let children = partition( + &sources, + &routes, + targets, + limits, + &cancelled, + origin.map(|origin| (origin, admission.clone())), + )?; + Ok(StagedCellGroupPartition { + children, + _admission: admission, + }) + }) + .map_err(|_| CellGroupArchiveError::Stopped)?; + work.await.map_err(CellGroupArchiveError::Worker)? + })) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } +} + +fn sealed_origin(sources: &VerifiedCellGroupTransitionSources) -> Result { + let (operation, plan) = sources.transition_plan(); + let sealed = sources + .journal() + .sealed_sources() + .ok_or(CellGroupArchiveError::Invalid( + "missing sealed transfer sources", + ))?; + Ok(GroupTransferOrigin::new(operation, plan, sealed)) +} + +fn partition( + sources: &[CellGroupArchiveCoverage], + routes: &[GroupRoute], + targets: Vec>, + limits: CellGroupRecoveryLimits, + cancelled: &AtomicBool, + transfer: Option<(GroupTransferOrigin, Arc)>, +) -> Result> { + if targets.len() != routes.len() + || targets.iter().zip(routes).any(|(target, route)| { + target.as_ref().is_some_and(|target| { + target.store.route() != route || !target.directory.is_absolute() + }) + }) + { + return Err(CellGroupArchiveError::Invalid( + "transition child destinations", + )); + } + let check = || { + if cancelled.load(Ordering::Acquire) { + return Err(CellGroupArchiveError::Stopped); + } + Ok(()) + }; + check()?; + let mut total_cells = 0usize; + let mut total_bytes = 0u64; + for source in sources { + let root = source.recovery().manifest(); + total_cells = total_cells + .checked_add(root.pinned_images()) + .filter(|n| *n <= limits.max_cells as usize) + .ok_or(CellGroupArchiveError::Limit)?; + root.images.visit(&mut |_, leaf, _, _, _| -> Result<()> { + check()?; + if leaf.geometry.length > limits.max_image_bytes { + return Err(CellGroupArchiveError::Limit); + } + total_bytes = total_bytes + .checked_add(leaf.geometry.length) + .filter(|bytes| *bytes <= limits.max_restore_bytes) + .ok_or(CellGroupArchiveError::Limit)?; + Ok(()) + })?; + } + // Both input and output ranges are ordered by the rooted plan. Lock each + // private child store in that order without waiting behind another owner. + let mut stores = Vec::with_capacity(targets.len()); + for target in &targets { + let store = if let Some(target) = target { + target.store.require_live()?; + Some( + target + .store + .storage + .try_lock() + .map_err(|error| match error { + std::sync::TryLockError::WouldBlock => CellGroupArchiveError::Busy, + std::sync::TryLockError::Poisoned(_) => { + CellGroupArchiveError::Invalid("poisoned transfer destination") + } + })?, + ) + } else { + None + }; + stores.push(store); + } + let mut pins = vec![ManifestPins::default(); targets.len()]; + let genesis = LogPosition { + index: LogIndex::new(1), + term: Term::new(1), + }; + let mut child = 0; + let mut selected_cells = 0; + for source in sources { + let source_store = source.recovery().checkpoint().checkpoint().store(); + source_store.require_live()?; + source + .recovery() + .manifest() + .images + .visit(&mut |_, old, pin, _, _| -> Result<()> { + let key = + cell_routing_key(old.version.history.namespace, &old.version.history.cell); + while routes + .get(child) + .is_some_and(|route| !route.range.contains(&key)) + { + child += 1; + } + let Some(target) = targets + .get(child) + .ok_or(CellGroupArchiveError::Invalid("unassigned transition cell"))? + else { + return Ok(()); + }; + selected_cells += 1; + let live = || { + check()?; + source_store.require_live()?; + target.store.require_live()?; + Ok(()) + }; + live()?; + let apply::Storage { + tree, + pages, + images, + .. + } = stores[child] + .as_deref_mut() + .ok_or(CellGroupArchiveError::Invalid("missing transfer store"))?; + let mut leaf = old.clone(); + leaf.last_mutation = genesis; + let pin = copy_image( + images, + pin, + &mut leaf, + &target.scratch, + &target.directory, + &live, + )?; + if leaf.version != old.version { + return Err(CellGroupArchiveError::Invalid( + "transfer changed cell image", + )); + } + pin.check(&leaf, target.store.route().schema_digest) + .map_err(CellGroupCheckpointError::from)?; + let root = tree + .update(pages, pins[child].root(), leaf.clone(), None) + .map_err(CellGroupCheckpointError::from)?; + pins[child] = pins[child] + .updated(tree, pages, root, &leaf, &pin) + .map_err(CellGroupCheckpointError::from)?; + Ok(()) + })?; + } + if pins.iter().map(ManifestPins::len).sum::() != selected_cells { + return Err(CellGroupArchiveError::Invalid( + "transition partition cell count", + )); + } + let mut children = Vec::with_capacity(targets.len()); + for ((store, pins), target) in stores.iter_mut().zip(pins).zip(&targets) { + let (Some(store), Some(target)) = (store.as_deref_mut(), target) else { + continue; + }; + check()?; + target.store.require_live()?; + let apply::Storage { tree, pages, .. } = store; + let data = match &transfer { + Some((origin, admission)) => { + let marker = GroupManifest::transfer_genesis( + target.store.route().clone(), + pins.root(), + origin.clone(), + ) + .map_err(CellGroupCheckpointError::from)?; + let mut prepared = tree + .prepare_archive_complete(&target.store.owner, pages, marker, &mut |leaf| { + check().map_err(|_| ManifestError::Quarantined)?; + let key = cell_routing_key( + leaf.version.history.namespace, + &leaf.version.history.cell, + ); + pins.get(&key) + .filter(|(selected, _)| *selected == leaf) + .map(|(_, pin)| pin) + .ok_or(ManifestError::Invalid("child transfer image closure")) + }) + .map_err(CellGroupCheckpointError::from)?; + prepared.recovery_admission = Some(admission.clone()); + ChildData::Sealed { + genesis: Box::new(prepared), + engine: Arc::new(OnceLock::new()), + } + } + None => { + tree.verify(pages, pins.root(), genesis, |_| Ok(())) + .map_err(CellGroupCheckpointError::from)?; + pages + .sync_directory() + .map_err(CellGroupCheckpointError::from)?; + ChildData::PreCopy(pins) + } + }; + children.push(PreparedCellGroupTransferChild { + data, + store: target.store.clone(), + }); + } + Ok(children) +} diff --git a/crates/cellar/durability/raft/cell_groups/recovery/state.rs b/crates/cellar/durability/raft/cell_groups/recovery/state.rs new file mode 100644 index 000000000..6dbef169c --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/recovery/state.rs @@ -0,0 +1,194 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Applied recovery lifecycle. A source intent is provenance, never authority +//! to serve its replacement. Only replicated activation opens admission. +use crate::raft::cell_groups::{GroupManifest, ManifestError}; +use cellar_logic::{cell_group::TransitionId, consensus::*}; +use serde::{Deserialize, Serialize}; + +/// The immutable request commits to the held source archive and all its original +/// cell witnesses. Retain that source hold while this provenance is referenced; +/// no old witness is installed as authority in the replacement history. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(in crate::raft::cell_groups) struct RecoveryState { + request: [u8; 32], + operation: [u8; 16], + #[serde(deserialize_with = "required_activation")] + activation: Option, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(in crate::raft::cell_groups) struct Activation { + publication: [u8; 32], + manifest: [u8; 32], + index: u64, + term: u64, +} + +impl Activation { + pub(in crate::raft::cell_groups) fn new( + publication: [u8; 32], + manifest: [u8; 32], + position: LogPosition, + ) -> Self { + Self { + publication, + manifest, + index: position.index.get(), + term: position.term.get(), + } + } + + pub(in crate::raft::cell_groups) fn parts(&self) -> ([u8; 32], [u8; 32], LogPosition) { + ( + self.publication, + self.manifest, + LogPosition { + index: LogIndex::new(self.index), + term: Term::new(self.term), + }, + ) + } + + pub(in crate::raft::cell_groups) fn validate( + &self, + position: LogPosition, + ) -> Result<(), ManifestError> { + if self.publication == [0; 32] + || self.manifest == [0; 32] + || self.index <= 1 + || self.term <= 1 + || self.index > position.index.get() + || self.term > position.term.get() + || (self.index == position.index.get() && self.term != position.term.get()) + { + return Err(ManifestError::Invalid("activation position")); + } + Ok(()) + } +} + +fn required_activation<'de, D: serde::Deserializer<'de>>( + decoder: D, +) -> Result, D::Error> { + Option::deserialize(decoder) +} + +impl RecoveryState { + pub(in crate::raft::cell_groups) fn pending( + request: [u8; 32], + operation: ProposalId, + ) -> Result { + if request == [0; 32] { + return Err(ManifestError::Invalid("recovery request digest")); + } + Ok(Self { + request, + operation: operation.bytes(), + activation: None, + }) + } + + pub(in crate::raft::cell_groups) fn request(&self) -> [u8; 32] { + self.request + } + + pub(in crate::raft::cell_groups) fn operation(&self) -> [u8; 16] { + self.operation + } + + pub(in crate::raft::cell_groups) fn active(&self) -> bool { + self.activation.is_some() + } + + pub(in crate::raft::cell_groups) fn activation_manifest(&self) -> Option<[u8; 32]> { + self.activation + .as_ref() + .map(|activation| activation.manifest) + } + + pub(in crate::raft::cell_groups) fn activation( + &self, + ) -> Option<([u8; 32], [u8; 32], LogPosition)> { + self.activation.as_ref().map(Activation::parts) + } + + pub(in crate::raft::cell_groups) fn succeeds(&self, previous: &Self) -> bool { + self.request == previous.request + && self.operation == previous.operation + && previous + .activation + .as_ref() + .is_none_or(|activation| self.activation.as_ref() == Some(activation)) + } + + pub(in crate::raft::cell_groups) fn validate( + &self, + position: LogPosition, + ) -> Result<(), ManifestError> { + if self.request == [0; 32] || ProposalId::new(self.operation).is_err() { + return Err(ManifestError::Invalid("recovery identity")); + } + if let Some(activation) = &self.activation { + activation.validate(position)?; + } + Ok(()) + } + + pub(in crate::raft::cell_groups) fn activate( + &mut self, + transition: TransitionId, + publication: [u8; 32], + manifest: [u8; 32], + expected_manifest: [u8; 32], + position: LogPosition, + ) -> Result<(), ManifestError> { + if transition.bytes() != self.operation + || publication == [0; 32] + || manifest == [0; 32] + || manifest != expected_manifest + { + return Err(ManifestError::Invalid("recovery activation binding")); + } + if let Some(existing) = &self.activation { + if existing.publication != publication || existing.manifest != manifest { + return Err(ManifestError::Conflict); + } + return self.validate(position); + } + let next = Self { + activation: Some(Activation::new(publication, manifest, position)), + ..self.clone() + }; + next.validate(position)?; + *self = next; + Ok(()) + } + + pub(in crate::raft::cell_groups) fn seed_commitment( + &self, + marker: &GroupManifest, + ) -> Result<[u8; 32], ManifestError> { + use sha2::{Digest, Sha256}; + if self.active() + || marker.membership_revision() != 1 + || marker.recovery.as_deref() != Some(self) + { + return Err(ManifestError::Invalid("recovery seed commitment phase")); + } + let mut genesis = marker.clone(); + genesis.position = LogPosition { + index: LogIndex::new(1), + term: Term::new(1), + }; + genesis.check()?; + let bytes = serde_json::to_vec(&genesis) + .map_err(|_| ManifestError::Invalid("recovery seed encoding"))?; + let mut digest = Sha256::new(); + digest.update(b"cellar-cell-group-recovery-genesis-1\0"); + digest.update(bytes); + Ok(digest.finalize().into()) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/recovery/tests.rs b/crates/cellar/durability/raft/cell_groups/recovery/tests.rs new file mode 100644 index 000000000..4347783c6 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/recovery/tests.rs @@ -0,0 +1,450 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Storage preparation tests; no marker here grants catalog or quorum authority. +use super::super::*; +use crate::application::consensus::group_protocol::{ + CellGroupEntry, GroupCommand, GroupEntryBinding, GroupEntryLimits, +}; +use cellar_logic::{cell_group::*, consensus::*}; +use cellar_ltx::{ + generation::{GenerationStore, ReadServices}, + DirectFileSystem, +}; +use raft::prelude::Entry; +use std::sync::Arc; + +fn route() -> GroupRoute { + GroupRoute { + history: GroupHistory { + cluster: ClusterId::new([1; 16]).unwrap(), + namespace: NamespaceId::new([2; 16]).unwrap(), + group: GroupId::new([3; 16]).unwrap(), + incarnation: IncarnationId::new([4; 16]).unwrap(), + }, + range: CellRange::all(), + epoch: RouteEpoch::new(2).unwrap(), + birth_digest: [5; 32], + schema_digest: [6; 32], + } +} + +fn operation() -> ProposalId { + ProposalId::new([7; 16]).unwrap() +} + +fn position(index: u64) -> LogPosition { + LogPosition { + index: LogIndex::new(index), + term: Term::new(2), + } +} + +fn pending() -> GroupManifest { + GroupManifest::recovery_genesis(route(), ManifestRoot::default(), [8; 32], operation()).unwrap() +} + +#[test] +fn recovery_marker_requires_explicit_lifecycle_and_valid_activation() { + let marker = pending(); + assert!(!marker.is_active()); + let value = serde_json::to_value(&marker).unwrap(); + assert_eq!( + serde_json::from_value::(value.clone()).unwrap(), + marker + ); + let mut malformed = value.clone(); + malformed.as_object_mut().unwrap().remove("recovery"); + assert!(serde_json::from_value::(malformed).is_err()); + let mut malformed = value.clone(); + malformed["recovery"] + .as_object_mut() + .unwrap() + .remove("activation"); + assert!(serde_json::from_value::(malformed).is_err()); + for invalid in [ + serde_json::json!({"publication":vec![0; 32],"manifest":vec![9; 32],"index":2,"term":2}), + serde_json::json!({"publication":vec![9; 32],"manifest":vec![9; 32],"index":1,"term":1}), + serde_json::json!({"publication":vec![9; 32],"manifest":vec![9; 32],"index":3,"term":2}), + serde_json::json!({"publication":vec![9; 32],"manifest":vec![9; 32],"index":2,"term":1}), + ] { + let mut malformed = value.clone(); + malformed["recovery"]["activation"] = invalid; + assert!(serde_json::from_value::(malformed).is_err()); + } + let ordinary = GroupManifest::genesis(route()).unwrap(); + assert!(ordinary.is_active()); + assert_eq!( + serde_json::to_value(ordinary).unwrap()["recovery"], + serde_json::Value::Null + ); + let mut invalid = pending(); + invalid.position = position(2); + invalid.membership_revision = 2; + assert!(invalid.check().is_err()); +} + +#[test] +fn recovery_commitment_is_bound_and_activation_cannot_be_replaced() { + let mut marker = pending(); + let expected = marker + .recovery + .as_ref() + .unwrap() + .seed_commitment(&marker) + .unwrap(); + let original = marker.clone(); + marker.position = position(2); + assert_eq!( + marker + .recovery + .as_ref() + .unwrap() + .seed_commitment(&marker) + .unwrap(), + expected + ); + let foreign = super::RecoveryState::pending([11; 32], operation()).unwrap(); + assert!(foreign.seed_commitment(&marker).is_err()); + let transition = TransitionId::new(operation().bytes()).unwrap(); + let state = marker.recovery.as_mut().unwrap(); + for (id, publication, manifest) in [ + (TransitionId::new([12; 16]).unwrap(), [13; 32], expected), + (transition, [0; 32], expected), + (transition, [13; 32], [14; 32]), + ] { + assert!(state + .activate(id, publication, manifest, expected, position(2)) + .is_err()); + assert!(!state.active()); + } + state + .activate(transition, [13; 32], expected, expected, position(2)) + .unwrap(); + let activated = state.clone(); + state + .activate(transition, [13; 32], expected, expected, position(3)) + .unwrap(); + assert_eq!(*state, activated); + assert!(state + .activate(transition, [15; 32], expected, expected, position(3)) + .is_err()); + assert!(marker.is_active()); + assert!(marker.preserves_recovery(&original)); + assert!(!original.preserves_recovery(&marker)); + assert!(!GroupManifest::genesis(route()) + .unwrap() + .preserves_recovery(&marker)); + assert!(!marker.preserves_recovery(&GroupManifest::genesis(route()).unwrap())); + marker.check().unwrap(); +} + +fn activation(marker: &GroupManifest, index: u64, manifest: [u8; 32]) -> Entry { + let bytes = CellGroupEntry { + binding: GroupEntryBinding { + group: marker.route().history, + route_epoch: marker.route().epoch, + birth_digest: marker.route().birth_digest, + schema_digest: marker.route().schema_digest, + proposal: operation(), + }, + command: GroupCommand::Activate { + transition: TransitionId::new(operation().bytes()).unwrap(), + publication: [9; 32], + manifest, + }, + } + .encode(marker.route(), GroupEntryLimits::default()) + .unwrap(); + Entry { + index, + term: 2, + data: bytes.into(), + ..Default::default() + } +} + +#[test] +fn recovery_apply_requires_exact_activation_and_keeps_noop_prefix_closed() -> anyhow::Result<()> { + let root = tempfile::tempdir()?; + let pages = ManifestPageStore::open( + &root.path().join("pages"), + Arc::new(DirectFileSystem), + 128, + 1024 * 1024, + 16, + )?; + let images = CellGroupApplyImages::Materialized { + images: GenerationStore::open( + root.path().join("images"), + Arc::new(DirectFileSystem), + Default::default(), + )?, + services: ReadServices { + now_unix_millis: Arc::new(|| 10), + fill_random: Arc::new(|bytes| bytes.fill(17)), + }, + }; + let mut storage = apply::Storage::new( + route(), + pages, + images, + Default::default(), + Default::default(), + 100, + )?; + let marker = pending(); + let commitment = marker.recovery.as_ref().unwrap().seed_commitment(&marker)?; + let owner = root::ArchiveManifestOwner::new(); + let base = + storage + .tree + .prepare_archive_complete(&owner, &mut storage.pages, marker, &mut |_| { + unreachable!("empty root") + })?; + let noop = storage.prepare_entry( + &base, + &Entry { + index: 2, + term: 2, + ..Default::default() + }, + None, + false, + )?; + assert!(!noop.marker().is_active()); + assert_eq!( + noop.marker().recovery_request(), + base.marker().recovery_request() + ); + assert!(storage + .prepare_entry(&noop, &activation(noop.marker(), 3, [99; 32]), None, false) + .is_err()); + assert!(!noop.marker().is_active()); + let active = storage.prepare_entry( + &noop, + &activation(noop.marker(), 3, commitment), + None, + false, + )?; + assert!(active.marker().is_active()); + assert_eq!(active.marker().root(), base.marker().root()); + assert!(active.marker().preserves_recovery(base.marker())); + let duplicate = storage.prepare_entry( + &active, + &activation(active.marker(), 4, commitment), + None, + false, + )?; + assert_eq!(duplicate.marker().recovery, active.marker().recovery); + + let prepare = |index| -> anyhow::Result { + let route = active.marker().route(); + let data = CellGroupEntry { + binding: GroupEntryBinding { + group: route.history, + route_epoch: route.epoch, + birth_digest: route.birth_digest, + schema_digest: route.schema_digest, + proposal: ProposalId::new([10; 16])?, + }, + command: GroupCommand::PreparePark { + generation: ParkGeneration::new(index)?, + term: Term::new(2), + membership: MembershipRevision::new(1)?, + }, + } + .encode(route, GroupEntryLimits::default())?; + Ok(Entry { + index, + term: 2, + data: data.into(), + ..Default::default() + }) + }; + assert!(storage + .prepare_entry(&noop, &prepare(3)?, None, false) + .is_err()); + let entry = prepare(4)?; + let live = storage.prepare_entry(&active, &entry, None, false)?; + let replayed = storage.prepare_entry(&active, &entry, None, true)?; + assert_eq!(live.marker(), replayed.marker()); + assert_eq!(live.marker().park_cut().unwrap().position(), position(4)); + let after = storage.prepare_entry( + &replayed, + &Entry { + index: 5, + term: 2, + ..Default::default() + }, + None, + true, + )?; + assert_eq!(after.marker().park_cut(), live.marker().park_cut()); + assert_eq!(after.marker().root(), active.marker().root()); + + let ordinary = storage.tree.prepare_archive_complete( + &owner, + &mut storage.pages, + GroupManifest::genesis(route())?, + &mut |_| unreachable!("empty root"), + )?; + assert!(storage + .prepare_entry( + &ordinary, + &activation(ordinary.marker(), 2, commitment), + None, + false + ) + .is_err()); + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn recovery_pending_root_denies_stale_and_execution_authorities() -> anyhow::Result<()> { + use crate::{ + asyncrt::ExecutionDomain, + raft::{ + budget::Budget, + options::{LogOpenMode, ReplicaLogOptions}, + storage::{CellGroupBootstrap, ReplicaLog}, + }, + }; + use sha2::{Digest, Sha256}; + use std::sync::atomic::AtomicU64; + let directory = tempfile::tempdir()?; + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let log = domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + directory.path().join("log"), + NodeId::new([21; 16])?, + DiskLineage::new([22; 16])?, + ProcessGeneration::new([23; 16])?, + LogOpenMode::Create, + ), + Arc::new(DirectFileSystem), + )) + .await??; + let group = log + .create_cell_group(CellGroupBootstrap { + route: route(), + member: MemberId::new(1)?, + voters: vec![MemberId::new(1)?], + learners: vec![], + }) + .await?; + domain + .blocking(move || -> anyhow::Result<()> { + let cell = CellHistory { + cluster: route().history.cluster, + namespace: route().history.namespace, + cell: CellId::new("restored")?, + incarnation: route().history.incarnation, + }; + let image = CellImage { + epoch: 1, + txid: 0, + checksum: 0, + commitment: CellImageCommitment::Materialized(Sha256::digest([]).into()), + }; + let leaf = CellManifestLeaf { + enrollment: EnrollmentRecord::new( + cell.clone(), + operation(), + route().schema_digest, + image, + )?, + version: CellVersion { + history: cell.clone(), + sequence: MutationSequence::new(1)?, + image, + }, + geometry: CellImageGeometry { + length: 0, + page_size: 512, + }, + last_mutation: pending().position(), + witness: CommitWitness { + group: route().history, + position: pending().position(), + proposal: operation(), + }, + effects: Default::default(), + }; + let mut pages = ManifestPageStore::open( + &directory.path().join("pages"), + Arc::new(DirectFileSystem), + 128, + 1024 * 1024, + 16, + )?; + let tree = ManifestTree::new(route(), Default::default())?; + let root = tree.update(&mut pages, ManifestRoot::default(), leaf.clone(), None)?; + let marker = GroupManifest::recovery_genesis(route(), root, [8; 32], operation())?; + let prepared = + tree.prepare_bound_complete(&group, &mut pages, marker.clone(), &mut |_| { + Ok(CellImagePin::Empty) + })?; + // This is an admission-negative fixture, not a supported genesis installer: + // emulate the selected cache to exercise every actual authority boundary. + group.cache.write().unwrap().group_manifest = Some(marker.clone()); + let leader = Arc::new(AtomicU64::new(1)); + assert!(CellExecutionAuthority::new( + group.clone(), + &prepared, + &cell, + Term::new(1), + leader.clone() + ) + .is_err()); + assert!(CellReconciliationAuthority::new(group.clone(), &prepared, &cell).is_err()); + assert!(CellEnrollmentState::new(group.clone(), &prepared, cell.clone()).is_err()); + let node = Budget::new(1024 * 1024, 1); + let local = Budget::new(1024 * 1024, 1); + let read = CellReadAuthority::new( + read_authority::ReadSelection { + group: group.clone(), + route: route(), + cell: leaf.clone(), + image: CellImagePin::Empty, + applied: marker.position(), + barrier: None, + leader: None, + source: None, + source_deadline: None, + }, + 100, + node.acquire(4096).unwrap(), + local.acquire(4096).unwrap(), + )?; + assert!(matches!( + read.check(10), + Err(crate::raft::completion::RuntimeError::NotReady) + )); + assert!(tree + .prepare_complete(&group, &mut pages, root, position(2), 1, |_| Ok( + CellImagePin::Empty + )) + .is_err()); + let reopened = + tree.prepare_complete(&group, &mut pages, root, marker.position(), 1, |_| { + Ok(CellImagePin::Empty) + })?; + assert_eq!(reopened.marker(), &marker); + assert!(!reopened.marker().is_active()); + let mut changed = leaf; + changed.last_mutation = position(2); + changed.witness.position = position(2); + assert!(tree + .prepare_update(&mut pages, &prepared, changed, None, CellImagePin::Empty) + .is_err()); + drop(read); + drop(reopened); + drop(prepared); + Ok(()) + })? + .await??; + log.close().await?; + domain.shutdown().await; + Ok(()) +} diff --git a/crates/cellar/durability/raft/cell_groups/recovery/transfer_genesis.rs b/crates/cellar/durability/raft/cell_groups/recovery/transfer_genesis.rs new file mode 100644 index 000000000..96e0b412f --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/recovery/transfer_genesis.rs @@ -0,0 +1,98 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Local physical preparation for an authorized, non-serving transfer child. +use super::super::*; +use crate::raft::{catalog::TransferSeedPermit, storage::LogError}; +use cellar_logic::{ + cell_group::GroupHistory, + consensus::{EngineInstanceId, ReplicaIdentity}, +}; +use std::sync::{atomic::AtomicBool, Arc, OnceLock}; + +pub(crate) struct TransferSeed { + permit: TransferSeedPermit, + prepared: PreparedGroupManifest, + store: CellGroupArchiveStore, + installed_engine: Arc>>, +} + +impl TransferSeed { + pub(crate) fn new( + permit: TransferSeedPermit, + child: &PreparedCellGroupTransferChild, + ) -> Result { + let prepared = child.genesis().ok_or(LogError::WrongIdentity)?; + let store = child.store(); + if prepared.marker() != &permit.genesis || !store.owner.owns(prepared) { + return Err(LogError::WrongIdentity); + } + Ok(Self { + permit, + prepared: prepared.clone(), + store: store.clone(), + installed_engine: child + .install_owner() + .ok_or(LogError::WrongIdentity)? + .clone(), + }) + } + + pub(crate) fn bootstrap(&self) -> &crate::raft::storage::CellGroupBootstrap { + &self.permit.bootstrap + } + + pub(crate) fn genesis(&self) -> &GroupManifest { + &self.permit.genesis + } + + pub(crate) fn check( + &self, + owner: &Arc<()>, + identity: &ReplicaIdentity, + engine: EngineInstanceId, + ) -> Result<(), LogError> { + self.permit.check(owner, identity, engine) + } + + pub(crate) fn managed_binding( + &self, + guard: Option<&crate::raft::fleet_stop::Guard>, + now: u64, + ) -> Result, LogError> { + self.permit.managed_binding(guard, now) + } + + pub(crate) fn prepare( + &self, + closed: &Arc, + ) -> Result< + ( + PreparedGroupManifest, + super::super::retention::PublicationPin, + ), + LogError, + > { + self.store + .require_live() + .map_err(|error| LogError::Invalid(error.to_string()))?; + let mut owner = self.store.storage.try_lock().map_err(|error| match error { + std::sync::TryLockError::WouldBlock => LogError::Overloaded, + std::sync::TryLockError::Poisoned(_) => LogError::Io("poisoned transfer store".into()), + })?; + self.store + .require_live() + .map_err(|error| LogError::Invalid(error.to_string()))?; + let apply::Storage { tree, pages, .. } = &mut *owner; + let root = tree + .prepare_transfer_install(&self.permit, pages, &self.prepared) + .map_err(super::genesis::preparation_error)?; + let publication = root + .begin_transfer_genesis(&self.permit, closed) + .map_err(super::genesis::preparation_error)?; + self.installed_engine + .set(self.permit.owner.clone()) + .map_err(|_| LogError::WrongIdentity)?; + self.store.close(); + Ok((root, publication)) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/retention.rs b/crates/cellar/durability/raft/cell_groups/retention.rs new file mode 100644 index 000000000..6a6184ec0 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/retention.rs @@ -0,0 +1,442 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Physical retention around engine publication. An uncertain write keeps both +//! closures and disables collection until a new owner recovers durable markers. + +use super::{GroupManifest, ManifestError, ManifestPins, PreparedGroupManifest}; +use crate::raft::cache::GroupLog; +use std::{ + collections::BTreeMap, + sync::{ + atomic::{AtomicBool, Ordering}, + Arc, Mutex, MutexGuard, + }, +}; + +type GroupKey = [[u8; 16]; 4]; + +struct Root { + marker: GroupManifest, + _pins: ManifestPins, +} + +struct Group { + engine: Arc<()>, + closed: Arc, + current: Option, + pending: Option, + checkpoint: Option, + expected_checkpoint: Option, + pending_checkpoint: bool, +} + +#[derive(Default)] +struct State { + groups: BTreeMap, + retained: Vec, + reconciled: bool, + quarantined: bool, +} + +pub(super) struct RootRetentions { + state: Mutex, + max_roots: usize, +} + +fn key(marker: &GroupManifest) -> GroupKey { + let h = marker.route().history; + [ + h.cluster.bytes(), + h.namespace.bytes(), + h.group.bytes(), + h.incarnation.bytes(), + ] +} + +impl RootRetentions { + pub(super) fn new(max_roots: usize) -> Self { + Self { + state: Mutex::new(State::default()), + max_roots, + } + } + + fn state(&self) -> Result, ManifestError> { + let state = self.state.lock().map_err(|_| ManifestError::Quarantined)?; + if state.quarantined { + return Err(ManifestError::Quarantined); + } + Ok(state) + } + + fn used_roots(state: &State) -> usize { + state.groups.len() * 2 + + state.retained.len() + + state + .groups + .values() + .filter(|g| g.checkpoint.is_some() || g.pending_checkpoint) + .count() + } + + fn admit_group(&self, state: &State, key: GroupKey) -> Result<(), ManifestError> { + // Reserve current + pending slots before a group can submit a write. + if !state.groups.contains_key(&key) && Self::used_roots(state) + 2 > self.max_roots { + return Err(ManifestError::Limit); + } + Ok(()) + } + + pub(super) fn recover( + &self, + group: &GroupLog, + marker: &GroupManifest, + pins: &ManifestPins, + ) -> Result<(), ManifestError> { + if group.closed.load(Ordering::Acquire) { + return Err(ManifestError::Quarantined); + } + let expected_checkpoint = group + .checkpoint_manifest() + .map_err(|_| ManifestError::Invalid("closed engine"))?; + let mut state = self.state()?; + let key = key(marker); + self.admit_group(&state, key)?; + let selected = state.groups.entry(key).or_insert_with(|| Group { + engine: group.owner.clone(), + closed: group.closed.clone(), + current: None, + pending: None, + checkpoint: None, + expected_checkpoint, + pending_checkpoint: false, + }); + if !Arc::ptr_eq(&selected.engine, &group.owner) + || selected.pending.is_some() + || selected + .current + .as_ref() + .is_some_and(|r| r.marker != *marker) + { + return Err(ManifestError::Conflict); + } + if selected.current.is_none() { + selected.current = Some(Root { + marker: marker.clone(), + _pins: pins.clone(), + }); + } + Ok(()) + } + + pub(super) fn recover_checkpoint( + &self, + group: &GroupLog, + marker: &GroupManifest, + pins: &ManifestPins, + ) -> Result<(), ManifestError> { + let mut state = self.state()?; + let key = key(marker); + let selected = state + .groups + .get(&key) + .ok_or(ManifestError::Invalid("recover applied root first"))?; + if !Arc::ptr_eq(&selected.engine, &group.owner) + || selected.pending.is_some() + || selected.closed.load(Ordering::Acquire) + || selected.expected_checkpoint.as_ref() != Some(marker) + || selected + .checkpoint + .as_ref() + .is_some_and(|r| r.marker != *marker) + { + return Err(ManifestError::Conflict); + } + if selected.checkpoint.is_none() { + if Self::used_roots(&state) >= self.max_roots { + return Err(ManifestError::Limit); + } + state + .groups + .get_mut(&key) + .ok_or(ManifestError::Conflict)? + .checkpoint = Some(Root { + marker: marker.clone(), + _pins: pins.clone(), + }); + } + Ok(()) + } + + pub(super) fn begin( + self: &Arc, + group: &GroupLog, + prepared: &PreparedGroupManifest, + previous: Option<&GroupManifest>, + checkpoint: bool, + ) -> Result { + self.begin_owned(&group.owner, &group.closed, prepared, previous, checkpoint) + } + + pub(super) fn begin_genesis( + self: &Arc, + permit: &crate::raft::catalog::RecoverySeedPermit, + closed: &Arc, + prepared: &PreparedGroupManifest, + ) -> Result { + if prepared.marker() != permit.genesis() + || !Arc::ptr_eq(&prepared.engine_owner, permit.owner()) + || !prepared.marker().is_recovery_genesis() + { + return Err(ManifestError::Invalid("genesis retention grant")); + } + self.begin_owned(permit.owner(), closed, prepared, None, false) + } + + pub(super) fn begin_transfer_genesis( + self: &Arc, + permit: &crate::raft::catalog::TransferSeedPermit, + closed: &Arc, + prepared: &PreparedGroupManifest, + ) -> Result { + if prepared.marker() != &permit.genesis + || !Arc::ptr_eq(&prepared.engine_owner, &permit.owner) + || !prepared.marker().is_transfer_genesis() + { + return Err(ManifestError::Invalid("transfer genesis retention grant")); + } + self.begin_owned(&permit.owner, closed, prepared, None, false) + } + + fn begin_owned( + self: &Arc, + engine: &Arc<()>, + closed: &Arc, + prepared: &PreparedGroupManifest, + previous: Option<&GroupManifest>, + checkpoint: bool, + ) -> Result { + if closed.load(Ordering::Acquire) { + return Err(ManifestError::Quarantined); + } + let mut state = self.state()?; + let key = key(prepared.marker()); + self.admit_group(&state, key)?; + if !state.groups.contains_key(&key) && previous.is_some() { + return Err(ManifestError::Invalid( + "selected root has not been recovered", + )); + } + if checkpoint + && state + .groups + .get(&key) + .is_none_or(|g| g.checkpoint.is_none()) + && Self::used_roots(&state) + usize::from(!state.groups.contains_key(&key)) * 2 + >= self.max_roots + { + return Err(ManifestError::Limit); + } + let selected = state.groups.entry(key).or_insert_with(|| Group { + engine: engine.clone(), + closed: closed.clone(), + current: None, + pending: None, + checkpoint: None, + expected_checkpoint: None, + pending_checkpoint: false, + }); + if !Arc::ptr_eq(&selected.engine, engine) + || selected.pending.is_some() + || selected.current.as_ref().map(|r| &r.marker) != previous + { + return Err(ManifestError::Conflict); + } + selected.pending_checkpoint = checkpoint; + selected.pending = Some(Root { + marker: prepared.marker().clone(), + _pins: prepared.images.clone(), + }); + Ok(PublicationPin { + owner: self.clone(), + key, + checkpoint, + armed: false, + finished: false, + }) + } + + /// Validate an already verified physical closure without rebuilding it. + /// Marker equality alone cannot clear pending or uncertain publication. + pub(super) fn require_selected( + &self, + group: &GroupLog, + marker: &GroupManifest, + ) -> Result<(), ManifestError> { + let checkpoint = group + .checkpoint_manifest() + .map_err(|_| ManifestError::Invalid("closed engine"))?; + let state = self.state()?; + let selected = state + .groups + .get(&key(marker)) + .ok_or(ManifestError::Invalid("selected root not recovered"))?; + if !Arc::ptr_eq(&selected.engine, &group.owner) + || group.closed.load(Ordering::Acquire) + || selected.closed.load(Ordering::Acquire) + || selected.pending.is_some() + || selected.pending_checkpoint + || selected.current.as_ref().map(|root| &root.marker) != Some(marker) + || selected.expected_checkpoint != checkpoint + || selected.checkpoint.as_ref().map(|root| &root.marker) != checkpoint.as_ref() + { + return Err(ManifestError::Conflict); + } + Ok(()) + } + + pub(super) fn require_checkpoint(&self, group: &GroupLog) -> Result<(), ManifestError> { + let expected = group + .checkpoint_manifest() + .map_err(|_| ManifestError::Invalid("closed engine"))?; + let Some(expected) = expected else { + return Ok(()); + }; + let state = self.state()?; + let selected = state + .groups + .get(&key(&expected)) + .ok_or(ManifestError::Invalid("checkpoint not recovered"))?; + if !Arc::ptr_eq(&selected.engine, &group.owner) + || selected.checkpoint.as_ref().map(|r| &r.marker) != Some(&expected) + { + return Err(ManifestError::Invalid("checkpoint not recovered")); + } + Ok(()) + } + + pub(super) fn checkpoint( + &self, + group: &GroupLog, + marker: &GroupManifest, + ) -> Result { + let state = self.state()?; + let selected = state + .groups + .get(&key(marker)) + .ok_or(ManifestError::Invalid("checkpoint not recovered"))?; + if !Arc::ptr_eq(&selected.engine, &group.owner) + || selected.closed.load(Ordering::Acquire) + || selected.expected_checkpoint.as_ref() != Some(marker) + { + return Err(ManifestError::Conflict); + } + let root = selected + .checkpoint + .as_ref() + .filter(|root| &root.marker == marker) + .ok_or(ManifestError::Invalid("checkpoint not recovered"))?; + Ok(root._pins.clone()) + } + + pub(super) fn reconcile(&self, retained: Vec) -> Result<(), ManifestError> { + let mut state = self.state()?; + if state + .groups + .values() + .any(|g| g.closed.load(Ordering::Acquire)) + { + return Err(ManifestError::Quarantined); + } + if state.groups.values().any(|g| g.pending.is_some()) { + return Err(ManifestError::Conflict); + } + if Self::used_roots(&state) - state.retained.len() + retained.len() > self.max_roots { + return Err(ManifestError::Limit); + } + state.retained = retained; + state.reconciled = true; + Ok(()) + } + + pub(super) fn can_reclaim(&self) -> Result<(), ManifestError> { + let state = self.state()?; + if state + .groups + .values() + .any(|g| g.closed.load(Ordering::Acquire)) + { + return Err(ManifestError::Quarantined); + } + if state + .groups + .values() + .any(|g| g.checkpoint.as_ref().map(|r| &r.marker) != g.expected_checkpoint.as_ref()) + { + return Err(ManifestError::Invalid("checkpoint not recovered")); + } + if !state.reconciled { + return Err(ManifestError::Invalid("root inventory not reconciled")); + } + if state.groups.values().any(|g| g.pending.is_some()) { + return Err(ManifestError::Conflict); + } + Ok(()) + } +} + +pub(crate) struct PublicationPin { + owner: Arc, + key: GroupKey, + checkpoint: bool, + armed: bool, + finished: bool, +} + +impl PublicationPin { + pub(crate) fn arm(&mut self) { + self.armed = true; + } + + pub(crate) fn commit(mut self) -> Result<(), ManifestError> { + { + let mut state = self.owner.state()?; + let group = state + .groups + .get_mut(&self.key) + .ok_or(ManifestError::Conflict)?; + let pending = group.pending.take().ok_or(ManifestError::Conflict)?; + if self.checkpoint { + group.expected_checkpoint = Some(pending.marker.clone()); + group.checkpoint = Some(Root { + marker: pending.marker.clone(), + _pins: pending._pins.clone(), + }); + } + group.pending_checkpoint = false; + group.current = Some(pending); + } + self.finished = true; + Ok(()) + } +} + +impl Drop for PublicationPin { + fn drop(&mut self) { + if self.finished { + return; + } + let mut state = self.owner.state.lock().unwrap_or_else(|e| e.into_inner()); + if self.armed { + // Do not release either old or new image/page pins after an unknown + // engine outcome, including unwinding between sync and publication. + state.quarantined = true; + } else if let Some(group) = state.groups.get_mut(&self.key) { + group.pending = None; + group.pending_checkpoint = false; + if group.current.is_none() { + state.groups.remove(&self.key); + } + } + } +} diff --git a/crates/cellar/durability/raft/cell_groups/root.rs b/crates/cellar/durability/raft/cell_groups/root.rs new file mode 100644 index 000000000..0a7707842 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/root.rs @@ -0,0 +1,1160 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Complete-root preparation and engine metadata. Neither a marker observation +//! nor physical image seals establish current-term quorum/serving authority. + +use super::{ + CellImagePin, CellManifestLeaf, ManifestError, ManifestLimits, ManifestPageStore, ManifestPins, + ManifestRoot, ManifestTree, +}; +use crate::application::consensus::cell_routing_key; +use cellar_logic::{cell_group::*, consensus::*}; +use serde::{Deserialize, Serialize}; +use std::sync::Arc; + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(try_from = "WireRoot", into = "WireRoot")] +pub struct GroupManifest { + pub(super) route: GroupRoute, + pub(super) position: LogPosition, + pub(super) membership_revision: u64, + pub(super) root: ManifestRoot, + pub(super) recovery: Option>, + seal: Option>, + transfer: Option>, + park: Option>, +} + +/// Exact sealed-source provenance for a child prepared before activation. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct GroupTransferOrigin { + operation: [u8; 16], + plan: [u8; 32], + sealed_sources: [u8; 32], + #[serde(default, skip_serializing_if = "Option::is_none")] + activation: Option, +} + +impl GroupTransferOrigin { + pub(super) fn new(operation: ProposalId, plan: [u8; 32], sealed_sources: [u8; 32]) -> Self { + Self { + operation: operation.bytes(), + plan, + sealed_sources, + activation: None, + } + } + + pub fn operation(&self) -> [u8; 16] { + self.operation + } + + pub fn plan(&self) -> [u8; 32] { + self.plan + } + + pub fn sealed_sources(&self) -> [u8; 32] { + self.sealed_sources + } + + fn same_provenance(&self, prior: &Self) -> bool { + self.operation == prior.operation + && self.plan == prior.plan + && self.sealed_sources == prior.sealed_sources + } + + fn check(&self, position: LogPosition) -> Result<(), ManifestError> { + if let Some(activation) = &self.activation { + activation.validate(position)?; + } + if ProposalId::new(self.operation).is_err() + || self.plan == [0; 32] + || self.sealed_sources == [0; 32] + { + return Err(ManifestError::Invalid("child transfer provenance")); + } + Ok(()) + } +} + +/// Exact, irreversible cut selected by committed source apply. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct GroupSourceSeal { + transition: [u8; 16], + authorization: [u8; 32], + payload: [u8; 32], + index: u64, + term: u64, + #[serde(default, skip_serializing_if = "Option::is_none")] + retirement: Option, +} + +/// A committed source tombstone. The certificate identifies the replacement +/// partition, and the original seal keeps the source permanently fenced. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct GroupSourceRetirement { + publication: [u8; 32], + index: u64, + term: u64, +} + +impl GroupSourceRetirement { + pub fn publication(&self) -> [u8; 32] { + self.publication + } + + pub fn position(&self) -> LogPosition { + LogPosition { + index: LogIndex::new(self.index), + term: Term::new(self.term), + } + } + + fn valid_at(&self, sealed: LogPosition, applied: LogPosition) -> bool { + let retired = self.position(); + self.publication != [0; 32] + && retired.index > sealed.index + && retired.term >= sealed.term + && retired.index <= applied.index + && retired.term <= applied.term + && (retired.index != applied.index || retired.term == applied.term) + } +} + +impl GroupSourceSeal { + pub(crate) fn transition(&self) -> [u8; 16] { + self.transition + } + + pub fn position(&self) -> LogPosition { + LogPosition { + index: LogIndex::new(self.index), + term: Term::new(self.term), + } + } + + pub fn authorization(&self) -> [u8; 32] { + self.authorization + } + + pub fn retirement(&self) -> Option<&GroupSourceRetirement> { + self.retirement.as_ref() + } + + fn same_cut(&self, previous: &Self) -> bool { + self.transition == previous.transition + && self.authorization == previous.authorization + && self.payload == previous.payload + && self.position() == previous.position() + } + + pub(crate) fn matches( + &self, + transition: TransitionId, + authorization: [u8; 32], + payload: [u8; 32], + ) -> bool { + self.transition == transition.bytes() + && self.authorization == authorization + && self.payload == payload + } +} + +impl GroupManifest { + pub fn route(&self) -> &GroupRoute { + &self.route + } + + pub fn position(&self) -> LogPosition { + self.position + } + + pub fn membership_revision(&self) -> u64 { + self.membership_revision + } + + pub fn root(&self) -> ManifestRoot { + self.root + } + + /// Data imported into a fresh history remains unavailable to application + /// reads/writes until its replicated activation decision is selected. + pub fn is_active(&self) -> bool { + self.transfer + .as_ref() + .is_none_or(|state| state.activation.is_some()) + && self.seal.is_none() + && self.recovery.as_ref().is_none_or(|state| state.active()) + } + + pub fn transfer_origin(&self) -> Option<&GroupTransferOrigin> { + self.transfer.as_deref() + } + + pub(crate) fn is_transfer_genesis(&self) -> bool { + self.transfer + .as_ref() + .is_some_and(|state| state.activation.is_none()) + && self.position.index.get() == 1 + && self.position.term.get() == 1 + && self.membership_revision == 1 + } + + pub(crate) fn pending_transfer_genesis(&self) -> Result { + let origin = self.transfer_origin().ok_or(ManifestError::Conflict)?; + if self.transfer_activation().is_some() || self.membership_revision != 1 { + return Err(ManifestError::Conflict); + } + Self::transfer_genesis(self.route.clone(), self.root, origin.clone()) + } + + pub(crate) fn preserves_transfer(&self, previous: &Self) -> bool { + match (previous.transfer_origin(), self.transfer_origin()) { + (None, None) => true, + (Some(prior), Some(next)) if next.same_provenance(prior) => { + if prior.activation.is_some() { + return next.activation == prior.activation; + } + match self.transfer_activation() { + Some((_, manifest, position)) => { + position.index > previous.position.index + && position.term >= previous.position.term + && previous.transfer_commitment().ok().flatten() == Some(manifest) + } + None => self.root == previous.root, + } + } + _ => false, + } + } + + pub(crate) fn transfer_activation(&self) -> Option<([u8; 32], [u8; 32], LogPosition)> { + self.transfer_origin()? + .activation + .as_ref() + .map(super::recovery::Activation::parts) + } + + pub(crate) fn transfer_commitment(&self) -> Result, ManifestError> { + use sha2::{Digest, Sha256}; + + let Some(origin) = self.transfer_origin() else { + return Ok(None); + }; + if let Some(activation) = &origin.activation { + return Ok(Some(activation.parts().1)); + } + let genesis = Self::transfer_genesis(self.route.clone(), self.root, origin.clone())?; + let bytes = serde_json::to_vec(&genesis) + .map_err(|_| ManifestError::Invalid("transfer genesis encoding"))?; + let mut digest = Sha256::new(); + digest.update(b"cellar-cell-group-transfer-genesis-1\0"); + digest.update(bytes); + Ok(Some(digest.finalize().into())) + } + + pub(super) fn activate( + &mut self, + transition: TransitionId, + publication: [u8; 32], + manifest: [u8; 32], + ) -> Result<(), ManifestError> { + if let Some(expected) = self.recovery_commitment()? { + return self + .recovery + .as_deref_mut() + .ok_or(ManifestError::Invalid("recovery activation state"))? + .activate(transition, publication, manifest, expected, self.position); + } + let expected = self.transfer_commitment()?; + let origin = self + .transfer + .as_deref_mut() + .ok_or(ManifestError::Invalid("missing transfer activation state"))?; + if origin.operation != transition.bytes() + || expected != Some(manifest) + || publication == [0; 32] + { + return Err(ManifestError::Invalid("transfer activation binding")); + } + if let Some(existing) = &origin.activation { + if existing.parts().0 != publication { + return Err(ManifestError::Conflict); + } + } else { + origin.activation = Some(super::recovery::Activation::new( + publication, + manifest, + self.position, + )); + } + self.check() + } + + /// The latest applied PreparePark cut. A later entry can make it + /// ineligible for a park acknowledgement without deleting its history. + pub fn park_cut(&self) -> Option<&super::GroupParkCut> { + self.park.as_deref() + } + + pub(super) fn prepare_park( + &mut self, + generation: ParkGeneration, + term: Term, + membership: MembershipRevision, + ) -> Result<(), ManifestError> { + if !self.is_active() + || self + .park_cut() + .is_some_and(|cut| cut.generation() >= generation.get()) + { + return Err(ManifestError::Conflict); + } + let cut = super::GroupParkCut::prepare( + generation, + term, + membership, + self.position, + self.membership_revision, + )?; + self.park = Some(Box::new(cut)); + self.check() + } + + pub(crate) fn preserves_park_cut(&self, previous: &Self) -> bool { + match (previous.park_cut(), self.park_cut()) { + (None, None) => true, + (Some(prior), Some(next)) if prior == next => true, + (_, Some(next)) => { + next.position().index > previous.position.index + && next.position().term >= previous.position.term + && next.membership_revision() >= previous.membership_revision + } + (Some(_), None) => false, + } + } + + pub fn source_seal(&self) -> Option<&GroupSourceSeal> { + self.seal.as_deref() + } + + pub(super) fn seal_source( + &mut self, + transition: TransitionId, + authorization: [u8; 32], + payload: [u8; 32], + ) -> Result<(), ManifestError> { + if !self.is_active() { + return Err(ManifestError::Invalid("source is not active")); + } + self.seal = Some(Box::new(GroupSourceSeal { + transition: transition.bytes(), + authorization, + payload, + index: self.position.index.get(), + term: self.position.term.get(), + retirement: None, + })); + self.check() + } + + pub(super) fn retire_source( + &mut self, + transition: TransitionId, + publication: [u8; 32], + ) -> Result<(), ManifestError> { + let seal = self + .seal + .as_deref_mut() + .ok_or(ManifestError::Invalid("source retirement without seal"))?; + if seal.transition != transition.bytes() || publication == [0; 32] { + return Err(ManifestError::Invalid("source retirement binding")); + } + if let Some(retired) = &seal.retirement { + if retired.publication != publication { + return Err(ManifestError::Conflict); + } + } else { + seal.retirement = Some(GroupSourceRetirement { + publication, + index: self.position.index.get(), + term: self.position.term.get(), + }); + } + self.check() + } + + pub(crate) fn preserves_source_seal(&self, previous: &Self) -> bool { + match (previous.source_seal(), self.source_seal()) { + (Some(previous_seal), Some(next_seal)) => { + next_seal.same_cut(previous_seal) + && match (previous_seal.retirement(), next_seal.retirement()) { + (Some(prior), next) => next == Some(prior), + (None, Some(next)) => { + next.position().index > previous.position.index + && next.position().term >= previous.position.term + } + (None, None) => true, + } + && previous.root == self.root + && previous.membership_revision == self.membership_revision + } + (Some(_), None) => false, + (None, Some(next_seal)) => { + next_seal.index > previous.position.index.get() + && next_seal.term >= previous.position.term.get() + } + (None, None) => true, + } + } + + /// Commitment to the immutable recovery intent and retained source archive. + /// This is provenance, never a fence, membership or serving permission. + pub fn recovery_request(&self) -> Option<[u8; 32]> { + self.recovery.as_ref().map(|state| state.request()) + } + + pub(crate) fn recovery_operation(&self) -> Option<[u8; 16]> { + self.recovery.as_ref().map(|state| state.operation()) + } + + pub(crate) fn is_recovery_genesis(&self) -> bool { + self.recovery.is_some() + && !self.is_active() + && self.position.index.get() == 1 + && self.position.term.get() == 1 + && self.membership_revision == 1 + } + + pub(crate) fn recovery_commitment(&self) -> Result, ManifestError> { + self.recovery + .as_deref() + .map(|state| match state.activation_manifest() { + Some(commitment) => Ok(commitment), + None => state.seed_commitment(self), + }) + .transpose() + } + + pub(crate) fn recovery_activation(&self) -> Option<([u8; 32], [u8; 32], LogPosition)> { + self.recovery.as_ref().and_then(|state| state.activation()) + } + + /// A retained history cannot forget or replace its recovery lineage. Only + /// an authenticated empty learner may acquire that lineage via a snapshot. + pub(crate) fn preserves_recovery(&self, previous: &Self) -> bool { + match (&previous.recovery, &self.recovery) { + (None, None) => true, + (Some(previous), Some(next)) => next.succeeds(previous), + _ => false, + } + } + + pub(crate) fn check(&self) -> Result<(), ManifestError> { + self.position + .validate() + .map_err(|_| ManifestError::Invalid("root position"))?; + if self.position.index.get() == 0 + || self.membership_revision == 0 + || self.membership_revision > self.position.index.get() + { + return Err(ManifestError::Invalid("root membership")); + } + ManifestTree::new(self.route.clone(), ManifestLimits::default())?; + if let Some(recovery) = &self.recovery { + recovery.validate(self.position)?; + if !recovery.active() && self.membership_revision != 1 { + return Err(ManifestError::Invalid( + "pending recovery membership changed", + )); + } + } + if let Some(transfer) = &self.transfer { + transfer.check(self.position)?; + if self.recovery.is_some() + || (transfer.activation.is_none() + && (self.seal.is_some() || self.membership_revision != 1)) + { + return Err(ManifestError::Invalid("pending child transfer lifecycle")); + } + } + if let Some(seal) = self.source_seal() { + if TransitionId::new(seal.transition).is_err() + || seal.authorization == [0; 32] + || seal.payload == [0; 32] + || seal.index < 2 + || seal.term == 0 + || seal.index > self.position.index.get() + || seal.term > self.position.term.get() + || (seal.index == self.position.index.get() + && seal.term != self.position.term.get()) + || self.membership_revision >= seal.index + || seal + .retirement() + .is_some_and(|retired| !retired.valid_at(seal.position(), self.position)) + || self.recovery.as_ref().is_some_and(|state| !state.active()) + { + return Err(ManifestError::Invalid("source seal")); + } + } + if let Some(park) = self.park_cut() { + let at = park.position(); + if !park.valid_at(self.position, self.membership_revision) + || self + .source_seal() + .is_some_and(|seal| seal.position().index <= at.index) + || self.transfer.as_ref().is_some_and(|transfer| { + transfer + .activation + .as_ref() + .is_none_or(|a| a.parts().2.index >= at.index) + }) + || self.recovery.as_ref().is_some_and(|recovery| { + recovery + .activation() + .is_none_or(|(_, _, activation)| activation.index >= at.index) + }) + || self + .root + .last_mutation() + .is_some_and(|mutation| mutation.index == at.index) + { + return Err(ManifestError::Invalid("historical park cut")); + } + } + if let Some(root) = self.root.0 { + if root.cells == 0 + || root.cells > 1_000_000 + || root.height == 0 + || root.height > 32 + || root.last_mutation_index == 0 + || root.last_mutation_term == 0 + || root.last_mutation_index > self.position.index.get() + || root.last_mutation_term > self.position.term.get() + || root.last_mutation_index == self.position.index.get() + && root.last_mutation_term != self.position.term.get() + || root + .next_due_unix_millis + .is_some_and(|due| due > i64::MAX as u64) + { + return Err(ManifestError::Invalid("root bounds")); + } + } + Ok(()) + } + + pub(crate) fn matches_group(&self, scope: &S, birth: [u8; 32]) -> bool + where + for<'a> GroupHistory: From<&'a S>, + { + self.route.history == GroupHistory::from(scope) && self.route.birth_digest == birth + } + + pub(crate) fn genesis(route: GroupRoute) -> Result { + let marker = Self { + route, + position: LogPosition { + index: LogIndex::new(1), + term: Term::new(1), + }, + membership_revision: 1, + root: ManifestRoot::default(), + recovery: None, + seal: None, + transfer: None, + park: None, + }; + marker.check()?; + Ok(marker) + } + + pub(super) fn transfer_genesis( + route: GroupRoute, + root: ManifestRoot, + origin: GroupTransferOrigin, + ) -> Result { + let marker = Self { + root, + transfer: Some(Box::new(origin)), + ..Self::genesis(route)? + }; + marker.check()?; + Ok(marker) + } + + pub(crate) fn recovery_genesis( + route: GroupRoute, + root: ManifestRoot, + request: [u8; 32], + operation: ProposalId, + ) -> Result { + let marker = Self { + root, + recovery: Some(Box::new(super::recovery::RecoveryState::pending( + request, operation, + )?)), + ..Self::genesis(route)? + }; + marker.check()?; + Ok(marker) + } +} + +/// A complete, synced closure. Image pins remain alive while the engine request +/// is queued or syncing. Durable-root retention must also survive uncertain +/// outcomes; physical stores must reconcile engine roots before any reclamation. +#[derive(Clone)] +pub struct PreparedGroupManifest { + pub(crate) marker: GroupManifest, + // The pins are deliberately inaccessible to mutation/publication callers. + pub(crate) images: ManifestPins, + pub(crate) engine_owner: Arc<()>, + pub(crate) base: Option, + /// Import admission follows every clone of this private physical closure. + pub(super) recovery_admission: Option>, + pub(super) page_owner: Arc, + retention: Arc, +} + +/// Private storage identity for archive preparation, with no Raft engine, +/// member, disk lineage or publication permission. Never accept a caller's token. +pub(super) struct ArchiveManifestOwner { + identity: Arc<()>, + closed: std::sync::atomic::AtomicBool, +} + +impl ArchiveManifestOwner { + pub(super) fn new() -> Self { + Self { + identity: Arc::new(()), + closed: std::sync::atomic::AtomicBool::new(false), + } + } + + pub(super) fn require_live(&self) -> Result<(), ManifestError> { + if self.closed.load(std::sync::atomic::Ordering::Acquire) { + return Err(ManifestError::Invalid("closed archive store")); + } + Ok(()) + } + + pub(super) fn close(&self) { + self.closed + .store(true, std::sync::atomic::Ordering::Release); + } + + pub(super) fn owns(&self, root: &PreparedGroupManifest) -> bool { + Arc::ptr_eq(&self.identity, &root.engine_owner) + } +} + +impl PreparedGroupManifest { + pub fn marker(&self) -> &GroupManifest { + &self.marker + } + + pub fn pinned_images(&self) -> usize { + self.images.len() + } + + /// Pin the exact image/version selected by this coherent root. This is a + /// physical read view; the caller still needs the requested read authority. + pub fn pin_cell(&self, cell: &CellId) -> Option<(&CellManifestLeaf, CellImagePin)> { + self.images + .get(&cell_routing_key(self.marker.route.history.namespace, cell)) + } + + /// Select one due cell from this immutable physical closure without I/O, + /// wrapping after the routing-key cursor. Like `pin_cell`, this is a data + /// view only; it grants no execution, leadership or delivery permission. + pub fn next_due( + &self, + now_unix_millis: u64, + after: Option<&CellId>, + ) -> Option<(&CellManifestLeaf, CellImagePin)> { + let after = after.map(|cell| cell_routing_key(self.marker.route.history.namespace, cell)); + self.images.next_due(now_unix_millis, after.as_ref()) + } + + pub(super) fn require_selected( + &self, + group: &crate::raft::cache::GroupLog, + ) -> Result<(), ManifestError> { + self.retention.require_selected(group, &self.marker) + } + + pub(crate) fn require_checkpoint( + &self, + group: &crate::raft::cache::GroupLog, + ) -> Result<(), ManifestError> { + self.retention.require_checkpoint(group) + } + + pub(crate) fn retained_checkpoint( + &self, + group: &crate::raft::cache::GroupLog, + ) -> Result { + if !Arc::ptr_eq(&self.engine_owner, &group.owner) { + return Err(ManifestError::Conflict); + } + let marker = group + .checkpoint_manifest() + .map_err(|_| ManifestError::Invalid("closed engine"))? + .ok_or(ManifestError::Invalid("checkpoint not published"))?; + let images = self.retention.checkpoint(group, &marker)?; + Ok(Self { + marker, + images, + base: None, + ..self.clone() + }) + } + + pub(super) fn checkpoint_from_store( + group: &crate::raft::cache::GroupLog, + store: &ManifestPageStore, + ) -> Result { + store.require_live()?; + let marker = group + .checkpoint_manifest() + .map_err(|_| ManifestError::Invalid("closed engine"))? + .ok_or(ManifestError::Invalid("checkpoint not published"))?; + let images = store.retention.checkpoint(group, &marker)?; + Ok(Self { + marker, + images, + engine_owner: group.owner.clone(), + base: None, + recovery_admission: None, + page_owner: store.owner.clone(), + retention: store.retention.clone(), + }) + } + + pub(crate) fn reserved_bytes(&self) -> usize { + self.images.reserved_bytes() + } + + pub(crate) fn begin_publication( + &self, + group: &crate::raft::cache::GroupLog, + previous: Option<&GroupManifest>, + checkpoint: bool, + ) -> Result { + self.retention.begin(group, self, previous, checkpoint) + } + + pub(super) fn begin_genesis( + &self, + permit: &crate::raft::catalog::RecoverySeedPermit, + closed: &Arc, + ) -> Result { + self.retention.begin_genesis(permit, closed, self) + } + + pub(super) fn begin_transfer_genesis( + &self, + permit: &crate::raft::catalog::TransferSeedPermit, + closed: &Arc, + ) -> Result { + self.retention.begin_transfer_genesis(permit, closed, self) + } +} + +impl ManifestTree { + /// Advance one committed no-op or duplicate enrollment without changing any + /// cell version. The owner must supply the actual next committed position; + /// this preparation alone proves neither commitment nor serving authority. + pub fn prepare_unchanged( + &self, + store: &ManifestPageStore, + previous: &PreparedGroupManifest, + position: LogPosition, + ) -> Result { + store.require_live()?; + if previous.marker.route != self.route || !Arc::ptr_eq(&previous.page_owner, &store.owner) { + return Err(ManifestError::Invalid("root/page owner")); + } + if previous.marker.position.index.get().checked_add(1) != Some(position.index.get()) + || position.term < previous.marker.position.term + { + return Err(ManifestError::Invalid("noncontiguous no-op")); + } + let mut prepared = previous.clone(); + prepared.base = Some(previous.marker.clone()); + prepared.marker.position = position; + prepared.marker.check()?; + Ok(prepared) + } + + /// Full closure verification for bootstrap, recovery and checkpoint install. + /// Ordinary cell mutations use `prepare_update` instead. + /// The callback returns actual sealed images, never merely accepted markers. + pub fn prepare_complete( + &self, + group: &crate::raft::cache::GroupLog, + store: &mut ManifestPageStore, + root: ManifestRoot, + position: LogPosition, + membership_revision: u64, + mut image: impl FnMut(&CellManifestLeaf) -> Result, + ) -> Result + where + for<'a> GroupHistory: From<&'a S>, + { + let mut marker = GroupManifest { + route: self.route.clone(), + position, + membership_revision, + root, + recovery: None, + seal: None, + transfer: None, + park: None, + }; + // Reopening must retain the selected recovery lifecycle, rather than + // silently turning a pending imported root into an ordinary active one. + let mut lifecycle_selected = false; + let mut matched = false; + for selected in [group.group_manifest(), group.checkpoint_manifest()] { + if let Some(selected) = selected.map_err(|_| ManifestError::Invalid("closed engine"))? { + lifecycle_selected |= selected.recovery.is_some() + || selected.seal.is_some() + || selected.transfer.is_some() + || selected.park.is_some(); + if selected.route == marker.route + && selected.position == marker.position + && selected.membership_revision == marker.membership_revision + && selected.root == marker.root + { + marker = selected; + matched = true; + break; + } + } + } + if lifecycle_selected && !matched { + return Err(ManifestError::Invalid("complete lifecycle marker required")); + } + self.prepare_bound_complete(group, store, marker, &mut image) + } + + pub(super) fn prepare_bound_complete( + &self, + group: &crate::raft::cache::GroupLog, + store: &mut ManifestPageStore, + marker: GroupManifest, + image: &mut dyn FnMut(&CellManifestLeaf) -> Result, + ) -> Result + where + for<'a> GroupHistory: From<&'a S>, + { + if marker.route != self.route { + return Err(ManifestError::Conflict); + } + marker.check()?; + group + .cache_revision() + .map_err(|_| ManifestError::Invalid("closed engine"))?; + if !marker.matches_group(&group.identity().scope, group.birth_digest) { + return Err(ManifestError::Invalid("root/engine identity")); + } + let prepared = self.prepare_physical(store, marker.clone(), group.owner.clone(), image)?; + // Recover physical ownership only from the engine's selected marker. + // A prospective checkpoint or mutation remains a preparation capability. + if group + .group_manifest() + .map_err(|_| ManifestError::Invalid("closed engine"))? + .as_ref() + == Some(&marker) + { + store.retention.recover(group, &marker, &prepared.images)?; + } + if group + .checkpoint_manifest() + .map_err(|_| ManifestError::Invalid("closed engine"))? + .as_ref() + == Some(&marker) + { + store + .retention + .recover_checkpoint(group, &marker, &prepared.images)?; + } + Ok(prepared) + } + + pub(super) fn prepare_archive_complete( + &self, + owner: &ArchiveManifestOwner, + store: &mut ManifestPageStore, + marker: GroupManifest, + image: &mut dyn FnMut(&CellManifestLeaf) -> Result, + ) -> Result { + owner.require_live()?; + if marker.route() != self.route() { + return Err(ManifestError::Invalid("archive root route")); + } + self.prepare_physical(store, marker, owner.identity.clone(), image) + } + + /// Called only with the catalog's process-local engine grant. No engine + /// selection/retention is recovered before sync. + pub(super) fn prepare_recovery_genesis( + &self, + permit: &crate::raft::catalog::RecoverySeedPermit, + store: &mut ManifestPageStore, + source: &PreparedGroupManifest, + ) -> Result { + if !source.marker.is_recovery_genesis() + || source.marker.route != self.route + || !Arc::ptr_eq(&source.page_owner, &store.owner) + || source.marker != *permit.genesis() + { + return Err(ManifestError::Invalid("recovery genesis physical binding")); + } + self.prepare_imported_genesis(store, source, permit.owner()) + } + + pub(super) fn prepare_transfer_install( + &self, + permit: &crate::raft::catalog::TransferSeedPermit, + store: &mut ManifestPageStore, + source: &PreparedGroupManifest, + ) -> Result { + if !source.marker.is_transfer_genesis() + || source.marker.route != self.route + || !Arc::ptr_eq(&source.page_owner, &store.owner) + || source.marker != permit.genesis + { + return Err(ManifestError::Invalid("transfer genesis physical binding")); + } + self.prepare_imported_genesis(store, source, &permit.owner) + } + + fn prepare_imported_genesis( + &self, + store: &mut ManifestPageStore, + source: &PreparedGroupManifest, + owner: &Arc<()>, + ) -> Result { + self.prepare_physical(store, source.marker.clone(), owner.clone(), &mut |leaf| { + source + .pin_cell(&leaf.version.history.cell) + .filter(|(selected, _)| *selected == leaf) + .map(|(_, pin)| pin) + .ok_or(ManifestError::Invalid("imported genesis closure")) + }) + } + + fn prepare_physical( + &self, + store: &mut ManifestPageStore, + marker: GroupManifest, + owner: Arc<()>, + image: &mut dyn FnMut(&CellManifestLeaf) -> Result, + ) -> Result { + marker.check()?; + store.require_live()?; + self.verify(store, marker.root(), marker.position(), |_| Ok(()))?; + let pins = ManifestPins::complete(self, store, marker.root(), &mut |leaf| image(leaf))?; + store.sync_directory()?; + Ok(PreparedGroupManifest { + marker, + images: pins, + engine_owner: owner, + base: None, + recovery_admission: None, + page_owner: store.owner.clone(), + retention: store.retention.clone(), + }) + } + + /// Prepare one cell mutation/enrollment at the next contiguous group index. + /// The previous complete capability supplies unchanged physical image pins; + /// only the changed image and O(log cells) page path need preparation. It + /// may be chained for staging, but engine publication still checks order. + pub fn prepare_update( + &self, + store: &mut ManifestPageStore, + previous: &PreparedGroupManifest, + leaf: CellManifestLeaf, + expected: Option<&CellVersion>, + image: CellImagePin, + ) -> Result { + if previous.marker.route != self.route || !Arc::ptr_eq(&previous.page_owner, &store.owner) { + return Err(ManifestError::Invalid("root/page owner")); + } + if !previous.marker.is_active() { + return Err(ManifestError::Invalid("group is not active")); + } + if previous.marker.position.index.get().checked_add(1) + != Some(leaf.last_mutation.index.get()) + || leaf.last_mutation.term < previous.marker.position.term + || leaf.witness.group != self.route.history + || leaf.witness.position != leaf.last_mutation + { + return Err(ManifestError::Invalid("noncontiguous mutation")); + } + image.check(&leaf, self.route.schema_digest)?; + let root = self.update(store, previous.marker.root, leaf.clone(), expected)?; + // update synchronizes every newly referenced page before returning. All + // reused subtrees belong to a complete preparation on this store owner. + let pins = previous.images.updated(self, store, root, &leaf, &image)?; + let marker = GroupManifest { + route: self.route.clone(), + position: leaf.last_mutation, + membership_revision: previous.marker.membership_revision, + root, + recovery: previous.marker.recovery.clone(), + seal: None, + transfer: previous.marker.transfer.clone(), + park: previous.marker.park.clone(), + }; + marker.check()?; + Ok(PreparedGroupManifest { + marker, + images: pins, + engine_owner: previous.engine_owner.clone(), + base: Some(previous.marker.clone()), + recovery_admission: previous.recovery_admission.clone(), + page_owner: store.owner.clone(), + retention: store.retention.clone(), + }) + } +} + +#[derive(Clone, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct WireRoot { + domain: String, + cluster: [u8; 16], + namespace: [u8; 16], + group: [u8; 16], + incarnation: [u8; 16], + lower: Bound, + upper: Bound, + route_epoch: u64, + birth: [u8; 32], + schema: [u8; 32], + index: u64, + term: u64, + membership_revision: u64, + #[serde(deserialize_with = "required_root")] + root: Option, + #[serde(deserialize_with = "required_recovery")] + recovery: Option>, + #[serde(default, skip_serializing_if = "Option::is_none")] + seal: Option>, + #[serde(default, skip_serializing_if = "Option::is_none")] + transfer: Option>, + #[serde(default, skip_serializing_if = "Option::is_none")] + park: Option>, +} + +fn required_recovery<'de, D: serde::Deserializer<'de>>( + decoder: D, +) -> Result>, D::Error> { + Option::deserialize(decoder) +} + +fn required_root<'de, D: serde::Deserializer<'de>>( + decoder: D, +) -> Result, D::Error> { + Option::deserialize(decoder) +} + +#[derive(Clone, Serialize, Deserialize)] +#[serde(tag = "bound", content = "cell", deny_unknown_fields)] +pub(super) enum Bound { + NegativeInfinity, + Key(String), + PositiveInfinity, +} + +impl Bound { + pub(super) fn encode(value: &CellBound) -> Self { + match value { + CellBound::NegativeInfinity => Self::NegativeInfinity, + CellBound::Key(k) => Self::Key(k.cell().as_str().into()), + CellBound::PositiveInfinity => Self::PositiveInfinity, + } + } + + pub(super) fn decode(self, namespace: NamespaceId) -> Result { + Ok(match self { + Self::NegativeInfinity => CellBound::NegativeInfinity, + Self::PositiveInfinity => CellBound::PositiveInfinity, + Self::Key(s) => CellBound::Key(cell_routing_key( + namespace, + &CellId::new(s).map_err(|_| ManifestError::Invalid("range ID"))?, + )), + }) + } +} + +impl From for WireRoot { + fn from(m: GroupManifest) -> Self { + Self { + domain: "cellar-group-root-1".into(), + cluster: m.route.history.cluster.bytes(), + namespace: m.route.history.namespace.bytes(), + group: m.route.history.group.bytes(), + incarnation: m.route.history.incarnation.bytes(), + lower: Bound::encode(m.route.range.lower()), + upper: Bound::encode(m.route.range.upper()), + route_epoch: m.route.epoch.get(), + birth: m.route.birth_digest, + schema: m.route.schema_digest, + index: m.position.index.get(), + term: m.position.term.get(), + membership_revision: m.membership_revision, + root: m.root.0, + recovery: m.recovery, + seal: m.seal, + transfer: m.transfer, + park: m.park, + } + } +} + +impl TryFrom for GroupManifest { + type Error = ManifestError; + + fn try_from(w: WireRoot) -> Result { + if w.domain != "cellar-group-root-1" { + return Err(ManifestError::Invalid("root domain")); + } + let invalid = |_| ManifestError::Invalid("root identity"); + let namespace = NamespaceId::new(w.namespace).map_err(invalid)?; + let result = Self { + route: GroupRoute { + history: GroupHistory { + cluster: ClusterId::new(w.cluster).map_err(invalid)?, + namespace, + group: GroupId::new(w.group).map_err(invalid)?, + incarnation: IncarnationId::new(w.incarnation).map_err(invalid)?, + }, + range: CellRange::new(w.lower.decode(namespace)?, w.upper.decode(namespace)?) + .map_err(invalid)?, + epoch: RouteEpoch::new(w.route_epoch).map_err(invalid)?, + birth_digest: w.birth, + schema_digest: w.schema, + }, + position: LogPosition { + index: LogIndex::new(w.index), + term: Term::new(w.term), + }, + membership_revision: w.membership_revision, + root: ManifestRoot(w.root), + recovery: w.recovery, + seal: w.seal, + transfer: w.transfer, + park: w.park, + }; + result.check()?; + Ok(result) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/runtime/leadership.rs b/crates/cellar/durability/raft/cell_groups/runtime/leadership.rs new file mode 100644 index 000000000..5e1f926ae --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/runtime/leadership.rs @@ -0,0 +1,147 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! A complete node sample uses bounded owner turns and one roster revision. +use super::*; +use std::collections::BTreeMap; + +pub(super) const BATCH_GROUPS: usize = 16; + +pub(super) struct LeadershipBatch { + pub revision: u64, + pub population: usize, + pub count: usize, + pub leaders: u32, + pub next: Option, +} + +pub(crate) struct LeadershipSample { + pub groups: usize, + pub leaders: u32, + pub started_at_ms: u64, +} + +/// Only the bounded prefix invokes the physical owner callback. Looking ahead +/// for another key must not sample a seventeenth owner in the same turn. +pub(super) fn capture( + groups: &BTreeMap, + revision: u64, + after: Option, + mut leader: impl FnMut(&T) -> Result, +) -> Result { + use std::ops::Bound::{Excluded, Unbounded}; + let mut rows = groups.range((after.map_or(Unbounded, Excluded), Unbounded)); + let mut count = 0; + let mut leaders = 0; + let mut last = None; + for _ in 0..BATCH_GROUPS { + let Some((history, group)) = rows.next() else { + break; + }; + leaders += u32::from(leader(group)?); + count += 1; + last = Some(*history); + } + Ok(LeadershipBatch { + revision, + population: groups.len(), + count, + leaders, + next: rows.next().and(last), + }) +} + +struct Scan { + started_at_ms: u64, + max_groups: usize, + population: Option<(u64, usize)>, + after: Option, + visited: usize, + leaders: u32, +} + +impl Scan { + fn new(started_at_ms: u64, max_groups: usize) -> Self { + Self { + started_at_ms, + max_groups, + population: None, + after: None, + visited: 0, + leaders: 0, + } + } + + fn accept(&mut self, batch: LeadershipBatch) -> Result, RuntimeError> { + let current = (batch.revision, batch.population); + if self.population.is_some_and(|expected| expected != current) { + return Err(RuntimeError::NotReady); + } + self.population = Some(current); + if batch.count > BATCH_GROUPS + || batch.leaders as usize > batch.count + || batch.population > self.max_groups + { + return Err(RuntimeError::Protocol("leadership batch bounds".into())); + } + self.visited = self + .visited + .checked_add(batch.count) + .ok_or(RuntimeError::Overloaded)?; + self.leaders = self + .leaders + .checked_add(batch.leaders) + .ok_or(RuntimeError::Overloaded)?; + if self.visited > batch.population { + return Err(RuntimeError::Protocol("leadership population".into())); + } + match batch.next { + Some(next) if batch.count > 0 && self.after.is_none_or(|previous| next > previous) => { + self.after = Some(next); + Ok(None) + } + Some(_) => Err(RuntimeError::Protocol("leadership cursor".into())), + None if self.visited == batch.population => Ok(Some(LeadershipSample { + groups: self.visited, + leaders: self.leaders, + started_at_ms: self.started_at_ms, + })), + None => Err(RuntimeError::NotReady), + } + } +} + +impl CellGroupRuntime { + /// Advisory node-wide physical-group leadership, never a serving or + /// transfer grant. A failed owner, changed roster or incomplete scan + /// refuses the sample instead of publishing partial headroom. + pub(crate) async fn leadership_sample( + &self, + deadline: u64, + ) -> Result { + let domain = self.inner.log.domain(); + let started_at_ms = domain.mono_ms().map_err(|_| RuntimeError::Closed)?; + let mut scan = Scan::new(started_at_ms, self.inner.options.max_groups); + loop { + if domain.mono_ms().map_err(|_| RuntimeError::Closed)? >= deadline { + return Err(RuntimeError::DeadlineExceeded); + } + let timer = domain + .sleep_until(deadline) + .map_err(|_| RuntimeError::Closed)?; + let response = crate::asyncrt::select! { + _ = timer => return Err(RuntimeError::DeadlineExceeded), + result = self.call(None, Command::LeadershipBatch(scan.after), 4096) => result?, + }; + let Response::LeadershipBatch(batch) = response else { + return Err(RuntimeError::Protocol("leadership batch reply".into())); + }; + if let Some(sample) = scan.accept(batch)? { + return Ok(sample); + } + } + } +} + +#[cfg(test)] +#[path = "leadership_tests.rs"] +mod tests; diff --git a/crates/cellar/durability/raft/cell_groups/runtime/leadership_tests.rs b/crates/cellar/durability/raft/cell_groups/runtime/leadership_tests.rs new file mode 100644 index 000000000..780ec73e7 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/runtime/leadership_tests.rs @@ -0,0 +1,121 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use cellar_logic::consensus::{ClusterId, GroupId, IncarnationId, NamespaceId}; + +fn history(id: u8) -> GroupHistory { + GroupHistory { + cluster: ClusterId::new([1; 16]).unwrap(), + namespace: NamespaceId::new([2; 16]).unwrap(), + group: GroupId::new([id; 16]).unwrap(), + incarnation: IncarnationId::new([7; 16]).unwrap(), + } +} + +fn groups() -> BTreeMap { + (1..=33) + .map(|id| (history(id), id.is_multiple_of(7))) + .collect() +} + +#[test] +fn leadership_scan_bounds_owner_work_and_publishes_only_complete_population() { + let groups = groups(); + let mut scan = Scan::new(10, 33); + let mut total = 0; + loop { + let mut calls = 0; + let batch = capture(&groups, 4, scan.after, |leader| { + calls += 1; + Ok(*leader) + }) + .unwrap(); + assert!(calls <= BATCH_GROUPS); + assert_eq!(batch.count, calls); + total += calls; + if let Some(sample) = scan.accept(batch).unwrap() { + assert_eq!( + (sample.groups, sample.leaders, sample.started_at_ms), + (33, 4, 10) + ); + break; + } + assert!(total < 33); + } + assert_eq!(total, 33); +} + +#[test] +fn leadership_scan_same_count_different_roster_revision_refuses() { + let groups = groups(); + let mut scan = Scan::new(10, 33); + assert!(scan + .accept(capture(&groups, 4, None, |l| Ok(*l)).unwrap()) + .unwrap() + .is_none()); + assert!(matches!( + scan.accept(capture(&groups, 5, scan.after, |l| Ok(*l)).unwrap()), + Err(RuntimeError::NotReady) + )); +} + +#[test] +fn leadership_scan_failed_owner_never_emits_partial_headroom() { + let groups = groups(); + let mut scan = Scan::new(10, 33); + assert!(scan + .accept(capture(&groups, 4, None, |l| Ok(*l)).unwrap()) + .unwrap() + .is_none()); + let mut calls = 0; + let result = capture(&groups, 4, scan.after, |_| { + calls += 1; + if calls == 3 { + Err(RuntimeError::GroupQuarantined) + } else { + Ok(false) + } + }); + assert!(matches!(result, Err(RuntimeError::GroupQuarantined))); + assert_eq!(calls, 3); +} + +#[test] +fn leadership_scan_missing_suffix_and_repeated_cursor_refuse() { + let groups = groups(); + let mut scan = Scan::new(10, 33); + let mut batch = capture(&groups, 4, None, |l| Ok(*l)).unwrap(); + batch.next = None; + assert!(matches!(scan.accept(batch), Err(RuntimeError::NotReady))); + let mut scan = Scan::new(10, 33); + let first = capture(&groups, 4, None, |l| Ok(*l)).unwrap(); + let repeated = first.next; + assert!(scan.accept(first).unwrap().is_none()); + let mut second = capture(&groups, 4, scan.after, |l| Ok(*l)).unwrap(); + second.next = repeated; + assert!(matches!( + scan.accept(second), + Err(RuntimeError::Protocol(_)) + )); +} + +#[test] +fn leadership_scan_excess_counts_and_empty_continuations_refuse() { + for (population, count, leaders, next) in [ + (34, 16, 1, Some(history(16))), + (33, 16, 17, Some(history(16))), + (33, 0, 0, Some(history(1))), + ] { + let mut scan = Scan::new(10, 33); + assert!(matches!( + scan.accept(LeadershipBatch { + revision: 4, + population, + count, + leaders, + next + }), + Err(RuntimeError::Protocol(_)) + )); + } +} diff --git a/crates/cellar/durability/raft/cell_groups/runtime/mod.rs b/crates/cellar/durability/raft/cell_groups/runtime/mod.rs new file mode 100644 index 000000000..2d27d4488 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/runtime/mod.rs @@ -0,0 +1,1309 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! One execution-domain worker multiplexes physical cell-group owners per node. + +mod leadership; +mod park_deadlines; +pub(super) mod reads; +mod worker; +use leadership::LeadershipBatch; + +use super::{ + hibernate::{ParkAcknowledgement, ParkPreparation, ParkPrepareRequest}, + owner::SelectedParkCertificate, + *, +}; +use crate::raft::{ + budget::{Budget, Permit}, + completion::RuntimeError, + mailbox, + options::RaftRuntimeOptions, + runtime::Terminal, + storage::ReplicaLog, +}; +use ::raft::prelude::Message; +use cellar_logic::{ + cell_group::{GroupHistory, ParkGeneration}, + consensus::{MemberId, Term}, +}; +use protobuf::Message as _; +use sha2::{Digest, Sha256}; +use std::sync::{ + atomic::{AtomicBool, Ordering}, + Arc, +}; +use tokio::sync::{mpsc, oneshot, Notify}; + +const COMPLETION_SLOTS: usize = 8; +const SNAPSHOT_HINT_SLOTS: usize = 256; + +/// Advisory admission counters. Different lanes may be observed at different +/// moments; these values neither reserve capacity nor establish durability or +/// serving authority. +#[derive(Clone, Copy, Debug, PartialEq, Eq, serde::Serialize)] +pub struct CellGroupAdmissionUsage { + pub items: usize, + pub bytes: usize, + pub max_items: usize, + pub max_bytes: usize, +} + +impl CellGroupAdmissionUsage { + pub(crate) fn read(budget: &Budget) -> Self { + let (bytes, items) = budget.usage(); + let (max_bytes, max_items) = budget.capacity(); + Self { + items, + bytes, + max_items, + max_bytes, + } + } +} + +/// Node-wide request/pin admission, retained through each lane's owned work. +#[derive(Clone, Copy, Debug, PartialEq, Eq, serde::Serialize)] +pub struct CellGroupRuntimeUsage { + pub ordinary: CellGroupAdmissionUsage, + /// Eight slots and max_ready_bytes of request metadata, independently of + /// the worker's Ready/persistence accounting. Refunded at owner admission; + /// physical work retains its other source, root and storage owners. + pub completions: CellGroupAdmissionUsage, + pub snapshot_feedback: CellGroupAdmissionUsage, + pub read_pins: CellGroupAdmissionUsage, +} + +pub struct CellGroupRuntimeEvent { + group: GroupHistory, + effect: Result, + _permit: Arc, +} + +impl CellGroupRuntimeEvent { + pub(crate) fn delivery_admission(&self) -> Arc { + self._permit.clone() + } + + pub fn group(&self) -> GroupHistory { + self.group + } + + pub fn effect(&self) -> Result<&CellGroupEffect, &RuntimeError> { + self.effect.as_ref() + } + + /// The cloned ticket retains event admission through physical preparation + /// and publication, even if the receiving task drops this event. + pub fn retain_apply(&self) -> Option { + match &self.effect { + Ok(CellGroupEffect::Apply(work)) => Some((**work).clone()), + _ => None, + } + } +} + +pub struct CellGroupRuntimeEvents { + receiver: mpsc::Receiver, + owner: std::sync::Weak, +} + +impl CellGroupRuntimeEvents { + pub(crate) fn belongs_to(&self, runtime: &CellGroupRuntime) -> bool { + self.owner.ptr_eq(&Arc::downgrade(&runtime.inner)) + } + + pub async fn recv(&mut self) -> Option { + self.receiver.recv().await + } + + pub fn try_recv(&mut self) -> Result { + self.receiver.try_recv() + } +} + +#[derive(Clone)] +pub struct CellGroupRuntime { + inner: Arc, +} + +struct Handle { + log: ReplicaLog, + domain: crate::asyncrt::DomainToken, + options: RaftRuntimeOptions, + requests: mailbox::Sender, + budget: Arc, + completions: Arc, + snapshot_hints: Arc, + read_pins: Arc, + transport_claimed: AtomicBool, + closing: Arc, + terminal: Arc, + stop: Arc, +} + +enum Command { + Attach(Box), + Quarantine, + Campaign, + TransferLeader(u64), + TransferLeadership(crate::raft::completion::LeadershipTransfer), + TransferLeadershipBefore(crate::raft::completion::LeadershipTransfer, u64), + RestartLeadership(Box), + Configure(CellGroupMembershipChange), + ActivateRecovery(Box), + ActivateTransfer(Box), + SealSource(Box), + PreparePark(ParkPrepareRequest), + FencePark { + generation: ParkGeneration, + term: Term, + deadline: u64, + }, + CertifyPark(Box), + EnterPark(Box), + AcknowledgePark { + leader: MemberId, + cut: GroupParkCut, + }, + RetireSource(Box), + Execution(cellar_logic::cell_group::CellHistory), + NextEffect { + now: u64, + after: Option, + }, + Enrollment(cellar_logic::cell_group::CellHistory), + EnrollmentState(cellar_logic::cell_group::CellHistory), + Reconciliation(cellar_logic::cell_group::CellHistory), + Step(Message), + Checkpoint(Message, PreparedCellGroupCheckpoint), + PublishCheckpoint(PreparedLocalCellGroupCheckpoint), + CheckpointBeforeApply(CellGroupTicket, PreparedLocalCellGroupCheckpoint), + AdmitStep(Message), + Propose(Vec), + ProposeAtTerm(cellar_logic::consensus::Term, Vec), + Submit( + crate::raft::execution::CellPreparedCommand, + Option>, + ), + ReadIndex(Vec), + Unreachable(u64), + SnapshotStatus(u64, bool), + Read(Box), + Apply(CellGroupTicket, Box), + CaptureCheckpoint(CellGroupCheckpointPool), + Status, + PendingTransferQuorum, + Observe, + ObservePartition, + LeadershipBatch(Option), + ReplicationStatus, + Tick, +} + +enum Response { + Unit, + Checkpoint(CellGroupSnapshotAdmission), + Proposal(CellGroupProposal), + RecoveryActivation(Box), + TransferActivation(CellGroupTransferActivation), + SourceSeal(Box), + ParkPrepared(ParkPreparation), + ParkCertified(SelectedParkCertificate), + ParkAcknowledgement(ParkAcknowledgement), + SourceRetirement(CellGroupSourceRetirement), + Submission(crate::raft::execution::CellSubmissionOutcome), + Execution(Box), + NextEffect(Option>), + Enrollment(Box), + EnrollmentState(Box), + Reconciliation(Box), + Status(CellGroupStatus), + PendingTransferQuorum(Box), + Observation(Box), + PartitionObservation(Box), + LeadershipBatch(LeadershipBatch), + ReplicationStatus(crate::raft::completion::ReplicationStatus), + Read(Box), + Seed(Box), + Membership(Box), + CheckpointSource(Box>), +} + +struct Request { + group: Option, + command: Command, + reply: oneshot::Sender>, + bytes: usize, + _permit: Permit, + inbound: Option, +} + +pub(crate) struct TransportClaim(Arc); + +impl Drop for TransportClaim { + fn drop(&mut self) { + self.0.transport_claimed.store(false, Ordering::Release); + } +} + +impl CellGroupRuntime { + /// Observe admission without requiring a free request slot. These gauges + /// may include work whose response waiter has disconnected. + pub fn usage(&self) -> CellGroupRuntimeUsage { + CellGroupRuntimeUsage { + ordinary: CellGroupAdmissionUsage::read(&self.inner.budget), + completions: CellGroupAdmissionUsage::read(&self.inner.completions), + snapshot_feedback: CellGroupAdmissionUsage::read(&self.inner.snapshot_hints), + read_pins: CellGroupAdmissionUsage::read(&self.inner.read_pins), + } + } + + pub(crate) fn same_owner(&self, other: &Self) -> bool { + Arc::ptr_eq(&self.inner, &other.inner) + } + + pub(crate) fn log(&self) -> &ReplicaLog { + &self.inner.log + } + + pub(crate) fn limits(&self) -> &RaftRuntimeOptions { + &self.inner.options + } + + pub(crate) fn claim_transport(&self) -> Result { + self.admit()?; + self.inner + .transport_claimed + .compare_exchange(false, true, Ordering::AcqRel, Ordering::Acquire) + .map_err(|_| RuntimeError::AlreadyAttached)?; + Ok(TransportClaim(self.inner.clone())) + } + + pub(crate) fn uses_domain(&self, domain: &crate::asyncrt::DomainToken) -> bool { + self.inner.domain.same_owner(domain) + } + + pub async fn submit( + &self, + prepared: crate::raft::execution::CellPreparedCommand, + ) -> Result { + self.submit_retained(prepared, None).await + } + + pub(crate) async fn submit_retained( + &self, + prepared: crate::raft::execution::CellPreparedCommand, + retained: Option>, + ) -> Result { + let bytes = prepared.bytes().len(); + if bytes > self.inner.options.max_proposal_bytes { + return Err(RuntimeError::Overloaded); + } + let group = prepared.group(); + match self + .call( + Some(group), + Command::Submit(prepared, retained), + bytes + 4096, + ) + .await? + { + Response::Submission(outcome) => Ok(outcome), + _ => Err(RuntimeError::Protocol("prepared submission reply".into())), + } + } + + /// Start parked: no recovered group is attached without its verified root + /// and catalog/member permit. The engine admits only one node runtime. + pub async fn start( + log: ReplicaLog, + options: RaftRuntimeOptions, + ) -> Result<(Self, CellGroupRuntimeEvents), RuntimeError> { + options.validate(log.limits())?; + // Reject an engine containing the superseded application family. + log.cell_groups()?; + let lease = log.claim_runtime()?; + let domain = log.domain(); + let (sender, requests) = + mailbox::channel(options.max_requests + COMPLETION_SLOTS + SNAPSHOT_HINT_SLOTS); + let (events, receiver) = mpsc::channel(options.max_events); + let closing = Arc::new(AtomicBool::new(false)); + let terminal = Arc::new(Terminal::new()); + let stop = Arc::new(Notify::new()); + let (ready, wait) = oneshot::channel(); + let handle = Self { + inner: Arc::new(Handle { + log: log.clone(), + domain: domain.clone(), + budget: Budget::new(options.max_request_bytes, options.max_requests), + completions: Budget::new(options.max_ready_bytes, COMPLETION_SLOTS), + snapshot_hints: Budget::new(SNAPSHOT_HINT_SLOTS * 4096, SNAPSHOT_HINT_SLOTS), + read_pins: Budget::new(options.max_read_pin_bytes, options.max_pending_reads), + options: options.clone(), + requests: sender, + transport_claimed: AtomicBool::new(false), + closing: closing.clone(), + terminal: terminal.clone(), + stop: stop.clone(), + }), + }; + let worker_domain = domain.clone(); + domain + .spawn_detached("cell-group-consensus", async move { + struct Closed(Arc, Arc); + + impl Drop for Closed { + fn drop(&mut self) { + self.0.store(true, Ordering::Release); + self.1.complete(Err(RuntimeError::Closed)); + } + } + let _closed = Closed(closing.clone(), terminal.clone()); + let mut worker = worker::Worker::new(log, options, worker_domain, events); + if ready.send(()).is_err() { + lease.release(); + return; + } + let result = worker.run(requests, &closing, &stop).await; + closing.store(true, Ordering::Release); + let fence = worker.finish().await; + let result = result.and(fence); + drop(worker); + if result.is_ok() { + lease.release(); + } + terminal.complete(result); + }) + .map_err(|_| RuntimeError::Closed)?; + wait.await.map_err(|_| RuntimeError::Closed)?; + handle.admit()?; + let owner = Arc::downgrade(&handle.inner); + Ok((handle, CellGroupRuntimeEvents { receiver, owner })) + } + + pub(crate) fn admit(&self) -> Result<(), RuntimeError> { + if self.inner.closing.load(Ordering::Acquire) || self.inner.domain.mono_ms().is_err() { + return Err(RuntimeError::Closed); + } + Ok(()) + } + + async fn call( + &self, + group: Option, + command: Command, + bytes: usize, + ) -> Result { + self.admit()?; + let completion = matches!( + command, + Command::Apply(..) | Command::CheckpointBeforeApply(..) | Command::Quarantine + ); + let snapshot_hint = matches!(command, Command::SnapshotStatus(..)); + let budget = if snapshot_hint { + &self.inner.snapshot_hints + } else if completion { + &self.inner.completions + } else { + &self.inner.budget + }; + if bytes > budget.capacity().0 { + return Err(RuntimeError::Overloaded); + } + let permit = loop { + let wake = budget.wake.notified(); + tokio::pin!(wake); + wake.as_mut().enable(); + self.admit()?; + if let Some(permit) = budget.acquire(bytes) { + break permit; + } + if !completion && !snapshot_hint { + return Err(RuntimeError::Overloaded); + } + crate::asyncrt::select! { + _ = wake => (), + _ = self.inner.terminal.wait() => return Err(RuntimeError::Closed), + } + }; + self.send_reserved(group, command, bytes, permit).await + } + + async fn send_reserved( + &self, + group: Option, + command: Command, + bytes: usize, + permit: Permit, + ) -> Result { + self.admit()?; + let (reply, wait) = oneshot::channel(); + self.inner + .requests + .try_send(Request { + group, + command, + reply, + bytes, + _permit: permit, + inbound: None, + }) + .map_err(|error| match error { + mpsc::error::TrySendError::Full(_) => RuntimeError::Overloaded, + mpsc::error::TrySendError::Closed(_) => RuntimeError::Closed, + })?; + let result = wait.await.map_err(|_| RuntimeError::Closed)?; + self.admit()?; + result + } + + /// Revoke this group's authority without retiring its physical owners. This + /// control request bypasses the waiting apply/ordinary queues and shares the + /// reserved completion lane. Only joined node shutdown releases the claim. + pub(crate) async fn quarantine(&self, group: GroupHistory) -> Result<(), RuntimeError> { + match self.call(Some(group), Command::Quarantine, 4096).await? { + Response::Unit => Ok(()), + _ => Err(RuntimeError::Protocol("quarantine reply".into())), + } + } + + pub async fn attach( + &self, + permit: CellGroupBootstrapPermit, + current: PreparedGroupManifest, + ) -> Result<(), RuntimeError> { + self.admit()?; + // Reserve before catalog revalidation and RawNode construction, keeping + // concurrent attachment work inside the node's ordinary admission cap. + let bytes = current.reserved_bytes() + 4096; + let admission = self + .inner + .budget + .acquire(bytes) + .ok_or(RuntimeError::Overloaded)?; + let history = permit.route().history; + let local = self.inner.log.cell_group(&history)?; + if !Arc::ptr_eq(&local.owner, &permit.group().owner) { + return Err(RuntimeError::StaleCompletion); + } + let mut seed = Sha256::new(); + let election_seed = self + .inner + .domain + .run(async { + self.inner.options.election_seed.unwrap_or_else(|| { + use rand::RngCore as _; + let mut seed = [0; 32]; + crate::asyncrt::rng("cell-group-election").fill_bytes(&mut seed); + seed + }) + }) + .await + .map_err(|_| RuntimeError::Closed)?; + seed.update(election_seed); + seed.update(history.group.bytes()); + seed.update(history.incarnation.bytes()); + seed.update(local.identity().member.get().to_be_bytes()); + seed.update(local.identity().node.bytes()); + let owner = self + .inner + .domain + // Retained attachment can revalidate through S3 on the caller's + // stack. Keep the nested owner startup future in one allocation. + .run(Box::pin(CellGroupOwner::start( + permit, + current, + self.inner.options.clone(), + seed.finalize().into(), + ))) + .await + .map_err(|_| RuntimeError::Closed)??; + self.send_reserved( + Some(history), + Command::Attach(Box::new(owner)), + bytes, + admission, + ) + .await + .map(|_| ()) + } + + /// Select one coherent image for this invocation. The deadline uses the + /// owning execution domain's monotonic clock and includes queue/barrier wait. + /// Returned pins retain separate node/group read admission until dropped. + pub async fn read_authority( + &self, + history: GroupHistory, + cell: cellar_logic::cell_group::CellHistory, + consistency: CellReadConsistency, + deadline: u64, + ) -> Result { + match self + .select_authority(history, reads::ReadKind::Cell(cell, consistency), deadline) + .await? + { + Response::Read(authority) => { + authority.check( + self.inner + .domain + .mono_ms() + .map_err(|_| RuntimeError::Closed)?, + )?; + Ok(*authority) + } + _ => Err(RuntimeError::Protocol("cell read authority reply".into())), + } + } + + /// Authorize one complete-group seed from a fresh quorum ReadIndex. Only a + /// committed, node/disk-bound new learner is eligible. This source authority + /// does not admit the destination, authenticate a peer, or grant a vote. + pub async fn seed_authority( + &self, + target: cellar_logic::consensus::ReplicaIdentity, + deadline: u64, + ) -> Result { + match self + .select_authority(target.scope, reads::ReadKind::Seed(target), deadline) + .await? + { + Response::Seed(authority) => { + authority.check()?; + Ok(*authority) + } + _ => Err(RuntimeError::Protocol("group seed authority reply".into())), + } + } + + async fn select_authority( + &self, + history: GroupHistory, + kind: reads::ReadKind, + deadline: u64, + ) -> Result { + self.admit()?; + if self + .inner + .domain + .mono_ms() + .map_err(|_| RuntimeError::Closed)? + >= deadline + { + return Err(RuntimeError::DeadlineExceeded); + } + let bytes = kind.reservation_bytes(); + let reservation = self + .inner + .read_pins + .acquire(bytes) + .ok_or(RuntimeError::Overloaded)?; + let request = reads::ReadRequest { + kind, + deadline, + reservation, + }; + let timer = self + .inner + .domain + .sleep_until(deadline) + .map_err(|_| RuntimeError::Closed)?; + crate::asyncrt::select! { + result = self.call(Some(history), Command::Read(Box::new(request)), bytes) => { + result + } + _ = timer => Err(RuntimeError::DeadlineExceeded), + } + } + + /// Observe one actual physical configuration after a fresh safe ReadIndex + /// and contiguous durable apply, without selecting an application cell. + /// A follower forwards the metadata request to its current leader and waits + /// for its own durable apply. No leader/execution authority is returned. + pub async fn quorum_membership( + &self, + history: GroupHistory, + deadline: u64, + ) -> Result { + self.quorum_membership_after(history, cellar_logic::consensus::LogIndex::new(0), deadline) + .await + } + + /// Wait for this durable apply floor before issuing the fresh safe ReadIndex. + /// Admission of a configuration proposal alone cannot satisfy this read. + pub async fn quorum_membership_after( + &self, + history: GroupHistory, + minimum: cellar_logic::consensus::LogIndex, + deadline: u64, + ) -> Result { + self.membership_for(history, minimum, reads::MembershipRead::Ordinary, deadline) + .await + } + + pub(crate) async fn park_admission( + &self, + history: GroupHistory, + deadline: u64, + ) -> Result { + self.membership_for( + history, + cellar_logic::consensus::LogIndex::new(0), + reads::MembershipRead::ParkAdmission, + deadline, + ) + .await + } + + pub(crate) async fn wake_park( + &self, + history: GroupHistory, + generation: ParkGeneration, + deadline: u64, + ) -> Result { + self.membership_for( + history, + cellar_logic::consensus::LogIndex::new(generation.get()), + reads::MembershipRead::Wake(generation), + deadline, + ) + .await + } + + async fn membership_for( + &self, + history: GroupHistory, + minimum: cellar_logic::consensus::LogIndex, + mode: reads::MembershipRead, + deadline: u64, + ) -> Result { + let response = self + .select_authority( + history, + reads::ReadKind::Membership { minimum, mode }, + deadline, + ) + .await?; + if self + .inner + .domain + .mono_ms() + .map_err(|_| RuntimeError::Closed)? + >= deadline + { + return Err(RuntimeError::DeadlineExceeded); + } + match response { + Response::Membership(observation) => Ok(*observation), + _ => Err(RuntimeError::Protocol( + "group quorum membership reply".into(), + )), + } + } + + pub async fn execution_authority( + &self, + history: GroupHistory, + cell: cellar_logic::cell_group::CellHistory, + ) -> Result { + let bytes = cell.cell.as_str().len() + 4096; + match self + .call(Some(history), Command::Execution(cell), bytes) + .await? + { + Response::Execution(authority) => Ok(*authority), + _ => Err(RuntimeError::Protocol("execution authority reply".into())), + } + } + + pub(crate) async fn next_effect( + &self, + history: GroupHistory, + now: u64, + after: Option, + ) -> Result, RuntimeError> { + let bytes = after.as_ref().map_or(0, |cell| cell.as_str().len()) + 4096; + match self + .call(Some(history), Command::NextEffect { now, after }, bytes) + .await? + { + Response::NextEffect(authority) => Ok(authority.map(|authority| *authority)), + _ => Err(RuntimeError::Protocol("effect discovery reply".into())), + } + } + + pub async fn reconciliation_authority( + &self, + history: GroupHistory, + cell: cellar_logic::cell_group::CellHistory, + ) -> Result { + let bytes = cell.cell.as_str().len() + 4096; + match self + .call(Some(history), Command::Reconciliation(cell), bytes) + .await? + { + Response::Reconciliation(authority) => Ok(*authority), + _ => Err(RuntimeError::Protocol( + "reconciliation authority reply".into(), + )), + } + } + + pub async fn enrollment_authority( + &self, + history: GroupHistory, + cell: cellar_logic::cell_group::CellHistory, + ) -> Result { + let bytes = cell.cell.as_str().len() + 4096; + match self + .call(Some(history), Command::Enrollment(cell), bytes) + .await? + { + Response::Enrollment(authority) => Ok(*authority), + _ => Err(RuntimeError::Protocol("enrollment authority reply".into())), + } + } + + pub async fn enrollment_state( + &self, + history: GroupHistory, + cell: cellar_logic::cell_group::CellHistory, + ) -> Result { + let bytes = cell.cell.as_str().len() + 4096; + match self + .call(Some(history), Command::EnrollmentState(cell), bytes) + .await? + { + Response::EnrollmentState(state) => Ok(*state), + _ => Err(RuntimeError::Protocol("enrollment state reply".into())), + } + } + + pub async fn configure( + &self, + history: GroupHistory, + change: CellGroupMembershipChange, + ) -> Result { + let bytes = change.bytes(); + match self + .call(Some(history), Command::Configure(change), bytes) + .await? + { + Response::Proposal(proposal) => Ok(proposal), + _ => Err(RuntimeError::Protocol("configuration reply".into())), + } + } + + /// Admit whole-group leadership handoff; observe status for completion. + /// Admit the catalog-bound activation on the multiplexed physical owner. + pub async fn activate_transfer( + &self, + permit: CellGroupTransferActivationPermit, + ) -> Result { + let history = permit.history(); + match self + .call( + Some(history), + Command::ActivateTransfer(Box::new(permit)), + 4096, + ) + .await? + { + Response::TransferActivation(result) => Ok(result), + _ => Err(RuntimeError::Protocol("transfer activation reply".into())), + } + } + + pub async fn activate_recovery( + &self, + permit: CellGroupRecoveryActivationPermit, + ) -> Result { + let history = permit.history(); + match self + .call( + Some(history), + Command::ActivateRecovery(Box::new(permit)), + 4096, + ) + .await? + { + Response::RecoveryActivation(result) => Ok(*result), + _ => Err(RuntimeError::Protocol( + "recovery activation response".into(), + )), + } + } + + /// Submit the exact local source seal to the physical Raft owner. + pub async fn retire_source( + &self, + permit: CellGroupSourceRetirementPermit, + ) -> Result { + let history = permit.history(); + match self + .call(Some(history), Command::RetireSource(Box::new(permit)), 4096) + .await? + { + Response::SourceRetirement(result) => Ok(result), + _ => Err(RuntimeError::Protocol("source retirement reply".into())), + } + } + + pub(crate) async fn prepare_park( + &self, + history: GroupHistory, + request: ParkPrepareRequest, + ) -> Result { + match self + .call(Some(history), Command::PreparePark(request), 4096) + .await? + { + Response::ParkPrepared(preparation) => Ok(preparation), + _ => Err(RuntimeError::Protocol("park preparation reply".into())), + } + } + + pub(crate) async fn fence_managed_park( + &self, + history: GroupHistory, + generation: ParkGeneration, + term: Term, + deadline: u64, + ) -> Result<(), RuntimeError> { + match self + .call( + Some(history), + Command::FencePark { + generation, + term, + deadline, + }, + 4096, + ) + .await? + { + Response::Unit => Ok(()), + _ => Err(RuntimeError::Protocol("park fence reply".into())), + } + } + + pub(crate) async fn certify_park( + &self, + history: GroupHistory, + certificate: ParkCertificate, + ) -> Result { + let bytes = certificate.reserved_bytes() + 4096; + match self + .call( + Some(history), + Command::CertifyPark(Box::new(certificate)), + bytes, + ) + .await? + { + Response::ParkCertified(certificate) => Ok(certificate), + _ => Err(RuntimeError::Protocol("park certificate reply".into())), + } + } + + pub(crate) async fn enter_park( + &self, + history: GroupHistory, + certificate: ParkCertificate, + ) -> Result<(), RuntimeError> { + let bytes = certificate.reserved_bytes() + 4096; + self.call( + Some(history), + Command::EnterPark(Box::new(certificate)), + bytes, + ) + .await + .map(|_| ()) + } + + /// Ask the local RawNode owner to sync its voter acknowledgement. The + /// caller still owns peer authentication and certificate assembly. + pub(crate) async fn acknowledge_park( + &self, + history: GroupHistory, + leader: MemberId, + cut: GroupParkCut, + ) -> Result { + match self + .call( + Some(history), + Command::AcknowledgePark { leader, cut }, + 4096, + ) + .await? + { + Response::ParkAcknowledgement(acknowledgement) => Ok(acknowledgement), + _ => Err(RuntimeError::Protocol("park acknowledgement reply".into())), + } + } + + pub async fn seal_source( + &self, + permit: CellGroupSealPermit, + ) -> Result { + let history = permit.history(); + let bytes = permit.entry_bytes().saturating_add(4096); + match self + .call(Some(history), Command::SealSource(Box::new(permit)), bytes) + .await? + { + Response::SourceSeal(result) => Ok(*result), + _ => Err(RuntimeError::Protocol("source seal response".into())), + } + } + + pub async fn transfer_leader( + &self, + history: GroupHistory, + member: u64, + ) -> Result<(), RuntimeError> { + self.call(Some(history), Command::TransferLeader(member), 4096) + .await + .map(|_| ()) + } + + pub async fn campaign(&self, history: GroupHistory) -> Result<(), RuntimeError> { + self.call(Some(history), Command::Campaign, 4096) + .await + .map(|_| ()) + } + + pub async fn step(&self, history: GroupHistory, message: Message) -> Result<(), RuntimeError> { + let bytes = message.compute_size() as usize; + if bytes > self.inner.options.max_message_bytes { + return Err(RuntimeError::Overloaded); + } + self.call(Some(history), Command::Step(message), bytes + 4096) + .await + .map(|_| ()) + } + + /// Admit local checkpoint publication. Completion is delivered through the + /// CheckpointPublished event after engine persistence, not this return value. + pub async fn publish_checkpoint( + &self, + history: GroupHistory, + checkpoint: PreparedLocalCellGroupCheckpoint, + ) -> Result<(), RuntimeError> { + let bytes = checkpoint + .root() + .reserved_bytes() + .saturating_add(checkpoint.snapshot().compute_size() as usize) + .saturating_add(4096); + self.call(Some(history), Command::PublishCheckpoint(checkpoint), bytes) + .await + .map(|_| ()) + } + + /// Admit publication of the exact captured base of a runtime-issued apply + /// ticket, while its owner retains the successor without preparing it. + /// Uses reserved completion admission, waiting with its source/pins when + /// capacity is occupied. Success proves admission; CheckpointPublished or + /// selected engine recovery proves persistence. The same ticket must then + /// complete its successor. Cancelling before admission publishes no work; + /// admitted work stays owned by the runtime through joined shutdown. + pub async fn checkpoint_before_apply( + &self, + history: GroupHistory, + ticket: CellGroupTicket, + checkpoint: PreparedLocalCellGroupCheckpoint, + ) -> Result<(), RuntimeError> { + let bytes = checkpoint + .root() + .reserved_bytes() + .saturating_add(checkpoint.snapshot().compute_size() as usize) + .saturating_add(4096); + self.call( + Some(history), + Command::CheckpointBeforeApply(ticket, checkpoint), + bytes, + ) + .await + .map(|_| ()) + } + + /// Queue a verified snapshot on this group owner. Admission is not a + /// persistence receipt; only its post-fence Raft response acknowledges it. + pub async fn step_checkpoint( + &self, + history: GroupHistory, + message: Message, + checkpoint: PreparedCellGroupCheckpoint, + ) -> Result { + let size = message.compute_size() as usize; + if size > self.inner.options.max_message_bytes { + return Err(RuntimeError::Overloaded); + } + let bytes = checkpoint + .manifest() + .reserved_bytes() + .saturating_add(size) + .saturating_add(4096); + match self + .call( + Some(history), + Command::Checkpoint(message, checkpoint), + bytes, + ) + .await? + { + Response::Checkpoint(outcome) => Ok(outcome), + _ => Err(RuntimeError::Protocol("checkpoint admission reply".into())), + } + } + + pub async fn propose( + &self, + history: GroupHistory, + data: Vec, + ) -> Result { + let bytes = data.len(); + if bytes > self.inner.options.max_proposal_bytes { + return Err(RuntimeError::Overloaded); + } + match self + .call(Some(history), Command::Propose(data), bytes + 4096) + .await? + { + Response::Proposal(proposal) => Ok(proposal), + _ => Err(RuntimeError::Protocol("proposal reply".into())), + } + } + + pub async fn propose_at_term( + &self, + history: GroupHistory, + term: cellar_logic::consensus::Term, + data: Vec, + ) -> Result { + let bytes = data.len(); + if bytes > self.inner.options.max_proposal_bytes { + return Err(RuntimeError::Overloaded); + } + match self + .call( + Some(history), + Command::ProposeAtTerm(term, data), + bytes + 4096, + ) + .await? + { + Response::Proposal(proposal) => Ok(proposal), + _ => Err(RuntimeError::Protocol("fenced proposal reply".into())), + } + } + + /// Queue an authenticated peer frame without waiting for protocol work. + /// Node/group admission remains owned until execution or discard. Success + /// is neither a Raft acknowledgement nor persistence/application proof. + pub async fn admit_step( + &self, + history: GroupHistory, + message: Message, + ) -> Result<(), RuntimeError> { + let bytes = message.compute_size() as usize; + if bytes > self.inner.options.max_message_bytes { + return Err(RuntimeError::Overloaded); + } + self.call(Some(history), Command::AdmitStep(message), bytes + 4096) + .await + .map(|_| ()) + } + + pub(crate) async fn report_snapshot( + &self, + history: GroupHistory, + member: u64, + delivered: bool, + ) -> Result<(), RuntimeError> { + self.call( + Some(history), + Command::SnapshotStatus(member, delivered), + 4096, + ) + .await + .map(|_| ()) + } + + pub(crate) async fn report_unreachable( + &self, + history: GroupHistory, + member: u64, + ) -> Result<(), RuntimeError> { + self.call(Some(history), Command::Unreachable(member), 4096) + .await + .map(|_| ()) + } + + pub async fn read_index( + &self, + history: GroupHistory, + context: Vec, + ) -> Result<(), RuntimeError> { + let bytes = context.len(); + if bytes > 4096 { + return Err(RuntimeError::Overloaded); + } + self.call(Some(history), Command::ReadIndex(context), bytes + 4096) + .await + .map(|_| ()) + } + + pub async fn complete_apply( + &self, + history: GroupHistory, + work: CellGroupApply, + next: PreparedGroupManifest, + ) -> Result<(), RuntimeError> { + let bytes = next.reserved_bytes() + 4096; + self.call( + Some(history), + Command::Apply(work.ticket, Box::new(next)), + bytes, + ) + .await + .map(|_| ()) + } + + pub async fn status(&self, history: GroupHistory) -> Result { + match self.call(Some(history), Command::Status, 4096).await? { + Response::Status(status) => Ok(status), + _ => Err(RuntimeError::Protocol("status reply".into())), + } + } + + /// Observe the pending child's durable fresh-term commit. The witness can + /// record installation and never grants serving authority. + pub async fn pending_transfer_quorum( + &self, + history: GroupHistory, + ) -> Result { + match self + .call(Some(history), Command::PendingTransferQuorum, 4096) + .await? + { + Response::PendingTransferQuorum(proof) => Ok(*proof), + _ => Err(RuntimeError::Protocol("pending child quorum reply".into())), + } + } + + pub async fn replication_status( + &self, + history: GroupHistory, + ) -> Result, RuntimeError> { + match self + .call(Some(history), Command::ReplicationStatus, 4096) + .await? + { + Response::ReplicationStatus(status) => Ok(status), + _ => Err(RuntimeError::Protocol( + "group replication status reply".into(), + )), + } + } + + pub(crate) async fn observe( + &self, + history: GroupHistory, + ) -> Result { + match self.call(Some(history), Command::Observe, 4096).await? { + Response::Observation(observation) => Ok(*observation), + _ => Err(RuntimeError::Protocol("group observation reply".into())), + } + } + + pub(crate) async fn observe_partition( + &self, + history: GroupHistory, + ) -> Result { + match self + .call(Some(history), Command::ObservePartition, 12 * 1024) + .await? + { + Response::PartitionObservation(observation) => Ok(*observation), + _ => Err(RuntimeError::Protocol("partition observation reply".into())), + } + } + + /// The deadline is absolute in this runtime's execution domain. The owner + /// checks it after its serialized queue, before admitting the handoff. + pub async fn transfer_leadership_before( + &self, + history: GroupHistory, + transfer: crate::raft::completion::LeadershipTransfer, + deadline: u64, + ) -> Result<(), RuntimeError> { + self.call( + Some(history), + Command::TransferLeadershipBefore(transfer, deadline), + 4096, + ) + .await + .map(|_| ()) + } + + pub async fn transfer_leadership( + &self, + history: GroupHistory, + transfer: crate::raft::completion::LeadershipTransfer, + ) -> Result<(), RuntimeError> { + self.call(Some(history), Command::TransferLeadership(transfer), 4096) + .await + .map(|_| ()) + } + + pub(crate) async fn restart_leadership( + &self, + history: GroupHistory, + check: crate::raft::completion::RestartLeadershipCheck, + ) -> Result<(), RuntimeError> { + self.call( + Some(history), + Command::RestartLeadership(Box::new(check)), + 4096, + ) + .await + .map(|_| ()) + } + + /// Capture the owner's exact durable applied cut on the existing worker. + /// This performs bounded metadata work only; streaming and archive I/O run + /// outside the worker. Concurrent persistence may transiently invalidate a + /// cut, which the archive producer can retry after the next apply event. + pub async fn capture_checkpoint( + &self, + history: GroupHistory, + pool: &CellGroupCheckpointPool, + ) -> Result { + match self + .call( + Some(history), + Command::CaptureCheckpoint(pool.clone()), + 4096, + ) + .await + .map_err(CellGroupCheckpointError::Authority)? + { + Response::CheckpointSource(source) => *source, + _ => Err(CellGroupCheckpointError::Authority(RuntimeError::Protocol( + "checkpoint source reply".into(), + ))), + } + } + + pub async fn tick(&self) -> Result<(), RuntimeError> { + self.call(None, Command::Tick, 4096).await.map(|_| ()) + } + + /// Close admission, join submitted persistence and release runtime ownership + /// only after a final engine fence. Cancelling this waiter does not cancel closure. + pub async fn close(&self) -> Result<(), RuntimeError> { + self.inner.closing.store(true, Ordering::Release); + self.inner.stop.notify_one(); + self.inner.terminal.wait().await + } +} diff --git a/crates/cellar/durability/raft/cell_groups/runtime/park_deadlines.rs b/crates/cellar/durability/raft/cell_groups/runtime/park_deadlines.rs new file mode 100644 index 000000000..770e76e78 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/runtime/park_deadlines.rs @@ -0,0 +1,86 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Node-owned earliest effect deadlines for parked voters. + +use cellar_logic::cell_group::GroupHistory; +use std::collections::{BTreeMap, BTreeSet}; + +#[derive(Default)] +pub(super) struct ParkDeadlines { + by_group: BTreeMap, + by_deadline: BTreeSet<(u64, GroupHistory)>, +} + +impl ParkDeadlines { + pub(super) fn set(&mut self, group: GroupHistory, deadline: Option) { + self.remove(&group); + if let Some(deadline) = deadline { + self.by_group.insert(group, deadline); + self.by_deadline.insert((deadline, group)); + } + } + + pub(super) fn remove(&mut self, group: &GroupHistory) { + if let Some(deadline) = self.by_group.remove(group) { + self.by_deadline.remove(&(deadline, *group)); + } + } + + pub(super) fn next(&self) -> Option { + self.by_deadline.first().map(|(deadline, _)| *deadline) + } + + pub(super) fn pop_due(&mut self, now: u64) -> Option { + let &(deadline, group) = self.by_deadline.first()?; + if deadline > now { + return None; + } + self.remove(&group); + Some(group) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use cellar_logic::consensus::{ClusterId, GroupId, IncarnationId, NamespaceId}; + + fn history(id: u8) -> GroupHistory { + GroupHistory { + cluster: ClusterId::new([1; 16]).unwrap(), + namespace: NamespaceId::new([2; 16]).unwrap(), + group: GroupId::new([id; 16]).unwrap(), + incarnation: IncarnationId::new([4; 16]).unwrap(), + } + } + + #[test] + fn park_deadlines_pop_only_due_voters_and_order_ties() { + let mut deadlines = ParkDeadlines::default(); + deadlines.set(history(1), Some(12)); + deadlines.set(history(3), Some(9)); + deadlines.set(history(2), Some(9)); + assert_eq!(deadlines.next(), Some(9)); + assert_eq!(deadlines.pop_due(8), None); + assert_eq!(deadlines.pop_due(9), Some(history(2))); + assert_eq!(deadlines.pop_due(9), Some(history(3))); + assert_eq!(deadlines.pop_due(9), None); + assert_eq!(deadlines.pop_due(12), Some(history(1))); + assert_eq!(deadlines.next(), None); + } + + #[test] + fn park_deadlines_replacement_and_early_wake_remove_old_entries() { + let mut deadlines = ParkDeadlines::default(); + deadlines.set(history(1), Some(9)); + deadlines.set(history(1), Some(21)); + deadlines.set(history(2), Some(15)); + deadlines.remove(&history(2)); + deadlines.set(history(3), Some(19)); + deadlines.set(history(3), None); + assert_eq!(deadlines.next(), Some(21)); + assert_eq!(deadlines.pop_due(20), None); + assert_eq!(deadlines.pop_due(21), Some(history(1))); + assert_eq!(deadlines.pop_due(u64::MAX), None); + } +} diff --git a/crates/cellar/durability/raft/cell_groups/runtime/reads.rs b/crates/cellar/durability/raft/cell_groups/runtime/reads.rs new file mode 100644 index 000000000..d64964bcc --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/runtime/reads.rs @@ -0,0 +1,372 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Bounded invocation-owned reads. No pending read retains peer/apply capacity. + +use super::super::read_authority::ReadSelection; +use super::super::seed_authority::{CellGroupSeedAuthority, SeedSelection}; +use super::*; +use cellar_logic::{ + cell_group::CellHistory, + consensus::{LogIndex, Term}, +}; +use std::{collections::BTreeMap, sync::Weak}; + +const PREFIX: &[u8] = b"cellar-cell-read-1\0"; +const CELL_FORWARD_PREFIX: &[u8] = b"cellar-cell-forward-read-1\0"; +const MEMBERSHIP_PREFIX: &[u8] = b"cellar-group-membership-read-1\0"; +const FORWARDED_RETRY_MS: u64 = 100; + +pub(in crate::raft::cell_groups) fn reserved_context(context: &[u8]) -> bool { + context.starts_with(PREFIX) + || context.starts_with(MEMBERSHIP_PREFIX) + || context.starts_with(CELL_FORWARD_PREFIX) +} + +/// Forwarded metadata reads have a separate namespace from local execution +/// authority. Bind the exact origin member as well as this invocation's nonce. +pub(in crate::raft::cell_groups) fn membership_origin(context: &[u8]) -> Option { + forwarded_origin(context, MEMBERSHIP_PREFIX) +} + +pub(in crate::raft::cell_groups) fn cell_origin(context: &[u8]) -> Option { + forwarded_origin(context, CELL_FORWARD_PREFIX) +} + +fn forwarded_origin(context: &[u8], prefix: &[u8]) -> Option { + let rest = context.strip_prefix(prefix)?; + if rest.len() != 32 { + return None; + } + let member = u64::from_be_bytes(rest[..8].try_into().ok()?); + let sequence = u64::from_be_bytes(rest[24..].try_into().ok()?); + cellar_logic::consensus::MemberId::new(member).ok()?; + (sequence > 0).then_some(member) +} + +pub(super) struct ReadRequest { + pub kind: ReadKind, + pub deadline: u64, + pub reservation: Permit, +} + +pub(super) enum ReadKind { + Cell(CellHistory, CellReadConsistency), + Seed(cellar_logic::consensus::ReplicaIdentity), + Membership { + minimum: LogIndex, + mode: MembershipRead, + }, +} + +#[derive(Clone, Copy, PartialEq, Eq)] +pub(super) enum MembershipRead { + Ordinary, + ParkAdmission, + Wake(cellar_logic::cell_group::ParkGeneration), +} + +impl ReadKind { + pub(super) fn membership_mode(&self) -> MembershipRead { + match self { + Self::Membership { mode, .. } => *mode, + _ => MembershipRead::Ordinary, + } + } + + pub fn reservation_bytes(&self) -> usize { + match self { + Self::Membership { .. } => super::super::membership_observation::MEMBERSHIP_BYTES, + _ => 4096, + } + } + + pub fn linearizable(&self) -> bool { + matches!(self, Self::Seed(_)) || self.forwards_read_index() + } + + fn forwards_read_index(&self) -> bool { + matches!( + self, + Self::Membership { .. } | Self::Cell(_, CellReadConsistency::Linearizable) + ) + } +} + +pub(super) enum Selection { + Cell(ReadSelection), + Seed(SeedSelection), + Membership(super::super::membership_observation::MembershipSelection), +} + +struct Pending { + history: GroupHistory, + request: ReadRequest, + reply: oneshot::Sender>, + group: Permit, + term: Option, + barrier: Option, + retry_at: Option, +} + +pub(super) struct Reads { + pending: BTreeMap, Pending>, + groups: BTreeMap>, + nonce: [u8; 16], + sequence: u64, + max_bytes: usize, + max_items: usize, +} + +impl Reads { + pub fn new(options: &RaftRuntimeOptions) -> Self { + use rand::RngCore as _; + let mut nonce = [0; 16]; + crate::asyncrt::rng("cell-group-read-context").fill_bytes(&mut nonce); + Self { + pending: BTreeMap::new(), + groups: BTreeMap::new(), + nonce, + sequence: 0, + max_bytes: options.max_read_pin_bytes, + max_items: (options.max_pending_reads / 8).clamp(1, 32), + } + } + + pub fn start( + &mut self, + history: GroupHistory, + owner: &mut CellGroupOwner, + request: ReadRequest, + reply: oneshot::Sender>, + now: u64, + ) { + if reply.is_closed() { + return; + } + let begin = (|| { + if now >= request.deadline { + return Err(RuntimeError::DeadlineExceeded); + } + match &request.kind { + ReadKind::Cell(cell, consistency) => { + if !history.same_namespace(cell) { + return Err(RuntimeError::Invalid("cell read namespace".into())); + } + if let CellReadConsistency::AtLeast(minimum) = consistency { + if &minimum.history != cell { + return Err(RuntimeError::Invalid("cell read version history".into())); + } + } + } + ReadKind::Seed(target) => owner.check_seed_target(target)?, + ReadKind::Membership { .. } => (), + } + // A pending recovery is an admission refusal, not a read barrier + // that may occupy the ordinary wait queue until its deadline. + if let MembershipRead::Wake(generation) = request.kind.membership_mode() { + owner.check_park_wake(generation)?; + } else { + owner.require_application_active()?; + } + self.groups.retain(|_, budget| budget.strong_count() != 0); + let budget = self + .groups + .get(&history) + .and_then(Weak::upgrade) + .unwrap_or_else(|| Budget::new(self.max_bytes, self.max_items)); + let group = budget + .acquire(request.kind.reservation_bytes()) + .ok_or(RuntimeError::Overloaded)?; + self.groups.insert(history, Arc::downgrade(&budget)); + self.sequence = self + .sequence + .checked_add(1) + .ok_or(RuntimeError::Overloaded)?; + let membership = matches!(request.kind, ReadKind::Membership { .. }); + let cell_read = matches!( + request.kind, + ReadKind::Cell(_, CellReadConsistency::Linearizable) + ); + let mut context = if membership || cell_read { + let mut context = if membership { + MEMBERSHIP_PREFIX + } else { + CELL_FORWARD_PREFIX + } + .to_vec(); + context.extend_from_slice(&owner.membership_read_origin().to_be_bytes()); + context + } else { + PREFIX.to_vec() + }; + context.extend_from_slice(&self.nonce); + context.extend_from_slice(&self.sequence.to_be_bytes()); + let term = if request.kind.linearizable() { + let status = owner.status()?; + if matches!(request.kind, ReadKind::Membership { minimum, .. } if status.applied.index < minimum) + { + // The configuration proposal is only admitted. Start this + // invocation's quorum barrier after its durable apply floor. + None + } else { + if membership { + match owner.read_membership_index(context.clone()) { + Ok(()) => Some(status.term), + Err(RuntimeError::NotReady) => None, + Err(error) => return Err(error), + } + } else if cell_read { + match owner.read_forwarded_cell_index(context.clone()) { + Ok(()) => Some(status.term), + Err(RuntimeError::NotReady) => None, + Err(error) => return Err(error), + } + } else { + if !status.ready { + return Err(RuntimeError::NotReady); + } + owner.read_cell_index(context.clone())?; + Some(status.term) + } + } + } else { + None + }; + Ok((context, group, term)) + })(); + match begin { + Ok((context, group, term)) => { + let retry_at = request + .kind + .forwards_read_index() + .then(|| now.saturating_add(FORWARDED_RETRY_MS)); + self.pending.insert( + context, + Pending { + history, + request, + reply, + group, + term, + barrier: None, + retry_at, + }, + ); + } + Err(error) => { + let _ = reply.send(Err(error)); + } + } + } + + pub fn observe(&mut self, history: GroupHistory, term: Term, index: LogIndex, context: &[u8]) { + if let Some(pending) = self.pending.get_mut(context) { + if pending.history == history && pending.term == Some(term) { + pending.barrier = Some(pending.barrier.map_or(index, |seen| seen.max(index))); + } + } + } + + /// Pending reads and issued read pins both retain this group's permit. + pub(super) fn busy(&self, history: GroupHistory) -> bool { + self.groups + .get(&history) + .is_some_and(|budget| budget.strong_count() != 0) + } + + pub fn deadline(&self) -> Option { + self.pending + .values() + .map(|pending| match (pending.barrier, pending.retry_at) { + (None, Some(retry)) => pending.request.deadline.min(retry), + _ => pending.request.deadline, + }) + .min() + } + + pub fn progress( + &mut self, + now: u64, + domain: &crate::asyncrt::DomainToken, + mut select: impl FnMut( + GroupHistory, + &ReadRequest, + &mut Option, + &mut Option, + &[u8], + bool, + ) -> Result, + ) -> bool { + let mut made = false; + self.pending.retain(|_, pending| { + // Taking a sender requires removing the entry; collect below instead. + if pending.reply.is_closed() { + made = true; + false + } else { + true + } + }); + let mut finished = Vec::new(); + for (context, pending) in &mut self.pending { + let before = (pending.term, pending.barrier); + let result = if now >= pending.request.deadline { + Err(RuntimeError::DeadlineExceeded) + } else { + let retry = + pending.barrier.is_none() && pending.retry_at.is_some_and(|retry| now >= retry); + if retry { + pending.retry_at = Some(now.saturating_add(FORWARDED_RETRY_MS)); + // A retransmitted context may have produced another Ready. + made = true; + } + select( + pending.history, + &pending.request, + &mut pending.term, + &mut pending.barrier, + context, + retry, + ) + }; + // A newly issued barrier must drive the owner's Ready work even + // when selection still waits. Waiting without a state change sleeps. + made |= before != (pending.term, pending.barrier); + if !matches!(result, Err(RuntimeError::NotReady)) { + finished.push((context.clone(), result)); + } + } + for (context, result) in finished { + let pending = self + .pending + .remove(&context) + .expect("selected pending read"); + let result = result.and_then(|selection| match selection { + Selection::Cell(selection) => CellReadAuthority::new( + selection, + pending.request.deadline, + pending.request.reservation, + pending.group, + ) + .map(|authority| Response::Read(Box::new(authority))), + Selection::Seed(selection) => CellGroupSeedAuthority::new( + selection, + pending.request.deadline, + domain.clone(), + pending.request.reservation, + pending.group, + ) + .map(|authority| Response::Seed(Box::new(authority))), + Selection::Membership(selection) => CellGroupQuorumMembership::new( + selection, + pending.request.reservation, + pending.group, + ) + .map(|observation| Response::Membership(Box::new(observation))), + }); + let _ = pending.reply.send(result); + made = true; + } + made + } +} diff --git a/crates/cellar/durability/raft/cell_groups/runtime/worker.rs b/crates/cellar/durability/raft/cell_groups/runtime/worker.rs new file mode 100644 index 000000000..51316c593 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/runtime/worker.rs @@ -0,0 +1,1214 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Fair group turns, node-wide persistence batches and joined shutdown. + +use super::{park_deadlines::ParkDeadlines, *}; +use crate::raft::storage::{LogError, SyncReceipt}; +use cellar_logic::consensus::OperationId; +use futures_util::future::BoxFuture; +use std::collections::{BTreeMap, BTreeSet, VecDeque}; + +const PARK_WAKE_BATCH: usize = 32; + +fn wakes_parked_peer(message: &::raft::prelude::Message, term: u64) -> bool { + use ::raft::prelude::MessageType::*; + message.term > term + || matches!( + message.get_msg_type(), + MsgAppend + | MsgHeartbeat + | MsgRequestVote + | MsgRequestPreVote + | MsgReadIndex + | MsgTimeoutNow + ) +} + +fn wakes_parked_group(command: &Command, term: u64) -> bool { + match command { + Command::Step(message) | Command::AdmitStep(message) => wakes_parked_peer(message, term), + Command::SnapshotStatus(..) | Command::Unreachable(_) => false, + _ => true, + } +} + +struct Group { + owner: Box, + queued: bool, + deferred: VecDeque, + completion: Option, + park_write: Option, + // One status per current peer, retaining node-wide hint item/byte admission. + snapshot_hints: BTreeMap, + inbound: Arc, + tick: bool, + wake_due: bool, + write: Option>, + write_bytes: usize, + pending: Option>, + settled: Option, + failed: Option, + usage: (usize, usize, usize), +} + +impl Group { + fn wake(&mut self, history: GroupHistory, order: &mut VecDeque) { + if !self.queued && (!self.owner.waiting() || self.failed.is_some()) { + self.queued = true; + order.push_back(history); + } + } + + fn usage(&self) -> (usize, usize, usize) { + let (ready, apply, reads) = self.owner.usage(); + let pending = self + .pending + .as_ref() + .map_or(0, |e| e.as_ref().map_or(4096, CellGroupEffect::bytes)); + ( + ready + .saturating_add(pending) + .saturating_add(self.write_bytes) + .saturating_add(self.settled.as_ref().map_or(0, CellGroupEffect::bytes)), + apply, + reads, + ) + } + + fn fail(&mut self, error: RuntimeError) { + self.owner.revoke_execution(); + self.failed = Some(error.clone()); + self.pending = Some(Err(error.clone())); + self.settled = None; + self.write = None; + self.write_bytes = 0; + self.tick = false; + self.wake_due = false; + self.snapshot_hints.clear(); + for request in self.deferred.drain(..) { + reply(request, Err(error.clone())); + } + if let Some(request) = self.park_write.take() { + reply(request, Err(error.clone())); + } + if let Some(request) = self.completion.take() { + reply(request, Err(error)); + } + } +} + +fn reply(request: Request, result: Result) { + let Request { + reply, + command, + _permit, + inbound, + .. + } = request; + drop(command); + drop(_permit); + drop(inbound); + let _ = reply.send(result); +} + +struct Job { + history: GroupHistory, + work: Box, + bytes: usize, +} + +type Batch = ( + Vec, + OperationId, + Result, LogError>, +); + +pub(super) struct Worker { + log: ReplicaLog, + options: RaftRuntimeOptions, + domain: crate::asyncrt::DomainToken, + groups: BTreeMap, + ticking: BTreeSet, + park_deadlines: ParkDeadlines, + // Every physical roster insertion changes the scan token. Future detach + // paths must also advance it, so an ABA roster cannot validate a sample. + roster_revision: u64, + order: VecDeque, + writes: VecDeque, + events: mpsc::Sender, + outbox: VecDeque, + output: Arc, + proposals: Arc, + inflight: Option>, + inflight_bytes: usize, + sequence: u64, + retry_at: u64, + ready: usize, + apply: usize, + reads: usize, + cell_reads: reads::Reads, +} + +impl Worker { + pub(super) fn new( + log: ReplicaLog, + options: RaftRuntimeOptions, + domain: crate::asyncrt::DomainToken, + events: mpsc::Sender, + ) -> Self { + Self { + log, + output: Budget::new(options.max_event_bytes, options.max_events), + proposals: Budget::new(options.max_request_bytes, options.max_requests), + cell_reads: reads::Reads::new(&options), + options, + domain, + groups: BTreeMap::new(), + ticking: BTreeSet::new(), + park_deadlines: ParkDeadlines::default(), + roster_revision: 0, + order: VecDeque::new(), + writes: VecDeque::new(), + events, + outbox: VecDeque::new(), + inflight: None, + inflight_bytes: 0, + sequence: 0, + retry_at: 0, + ready: 0, + apply: 0, + reads: 0, + } + } + + fn refresh(&mut self, history: GroupHistory) { + let group = self + .groups + .get_mut(&history) + .expect("scheduled group exists"); + if group.failed.is_some() { + self.ticking.remove(&history); + self.park_deadlines.remove(&history); + } + let usage = group.usage(); + self.ready = self.ready - group.usage.0 + usage.0; + self.apply = self.apply - group.usage.1 + usage.1; + self.reads = self.reads - group.usage.2 + usage.2; + group.usage = usage; + } + + fn leadership_batch( + &self, + after: Option, + ) -> Result { + leadership::capture(&self.groups, self.roster_revision, after, |group| { + if let Some(error) = &group.failed { + return Err(error.clone()); + } + group.owner.local_leader() + }) + } + + fn dispatch(&mut self, mut request: Request) { + if matches!(request.command, Command::Tick) { + self.tick(); + reply(request, Ok(Response::Unit)); + return; + } + if let Command::LeadershipBatch(after) = &request.command { + let result = self.leadership_batch(*after).map(Response::LeadershipBatch); + reply(request, result); + return; + } + let Some(history) = request.group else { + reply(request, Err(RuntimeError::NotFound)); + return; + }; + if matches!(request.command, Command::Attach(_)) { + if self.groups.contains_key(&history) { + reply(request, Err(RuntimeError::AlreadyAttached)); + return; + } + if self.groups.len() >= self.options.max_groups { + reply(request, Err(RuntimeError::Overloaded)); + return; + } + let Some(revision) = self.roster_revision.checked_add(1) else { + reply( + request, + Err(RuntimeError::Invalid( + "runtime roster revision overflow".into(), + )), + ); + return; + }; + let Request { + command: Command::Attach(mut owner), + reply: answer, + _permit, + .. + } = request + else { + unreachable!() + }; + // attach constructs a fresh follower; it has no accepted proposals. + // All owners retain acceptance against the same node-wide budget. + owner.proposal_budget = self.proposals.clone(); + let mut group = Group { + owner, + queued: false, + deferred: VecDeque::new(), + completion: None, + park_write: None, + snapshot_hints: BTreeMap::new(), + inbound: Budget::new( + (self.options.max_request_bytes / 4).max(self.options.max_message_bytes + 4096), + (self.options.max_requests / 8).clamp(1, 32), + ), + tick: false, + wake_due: false, + write: None, + write_bytes: 0, + pending: None, + settled: None, + failed: None, + usage: (0, 0, 0), + }; + group.wake(history, &mut self.order); + self.groups.insert(history, group); + self.ticking.insert(history); + self.roster_revision = revision; + self.refresh(history); + drop(_permit); + let _ = answer.send(Ok(Response::Unit)); + return; + } + let Some(group) = self.groups.get_mut(&history) else { + reply(request, Err(RuntimeError::NotFound)); + return; + }; + if matches!(request.command, Command::Quarantine) { + if group.failed.is_none() { + group.fail(RuntimeError::GroupQuarantined); + group.wake(history, &mut self.order); + } + reply(request, Ok(Response::Unit)); + self.refresh(history); + return; + } + if let Some(error) = &group.failed { + reply(request, Err(error.clone())); + return; + } + // Prepare is a park-only operation. A retry may race certificate + // delivery and must never enter the deferred work that wakes a voter. + if matches!(request.command, Command::PreparePark(_)) + && (group.owner.is_parked() || group.park_write.is_some()) + { + reply(request, Err(RuntimeError::NotReady)); + return; + } + let wake = match &request.command { + Command::FencePark { generation, .. } => reads::MembershipRead::Wake(*generation), + Command::Read(read) => read.kind.membership_mode(), + _ => reads::MembershipRead::Ordinary, + }; + let check = match wake { + reads::MembershipRead::Wake(generation) => group.owner.check_park_wake(generation), + reads::MembershipRead::ParkAdmission + if group.owner.is_parked() || group.park_write.is_some() => + { + Err(RuntimeError::NotReady) + } + _ => Ok(()), + }; + if let Err(error) = check { + reply(request, Err(error)); + return; + } + if let Command::CertifyPark(certificate) = &request.command { + if group.owner.is_parked() { + let selected = group + .owner + .finish_park_certificate(certificate) + .map(Response::ParkCertified); + reply(request, selected); + return; + } + } + if matches!( + &request.command, + Command::AcknowledgePark { .. } | Command::CertifyPark(_) | Command::EnterPark(_) + ) { + // A metadata write can share the physical batch, but must never + // overtake an already admitted Raft turn or application completion. + if group.tick + || !group.deferred.is_empty() + || group.completion.is_some() + || !group.snapshot_hints.is_empty() + || group.pending.is_some() + || group.settled.is_some() + || group.write.is_some() + || self.cell_reads.busy(history) + { + reply(request, Err(RuntimeError::NotReady)); + return; + } + let preparation = match &request.command { + Command::AcknowledgePark { leader, cut } => { + group.owner.begin_park_acknowledgement(*leader, cut) + } + Command::CertifyPark(certificate) => { + group.owner.begin_park_certificate(certificate) + } + Command::EnterPark(certificate) => group.owner.begin_park(certificate), + _ => Err(RuntimeError::Protocol("park write queue".into())), + }; + match preparation { + Ok((work, bytes)) => { + group.write = Some(work); + group.write_bytes = bytes; + group.park_write = Some(request); + self.writes.push_back(history); + self.refresh(history); + } + Err(error) => reply(request, Err(error)), + } + return; + } + if matches!(request.command, Command::Status) { + reply(request, group.owner.status().map(Response::Status)); + return; + } + if matches!(request.command, Command::PendingTransferQuorum) { + reply( + request, + group + .owner + .pending_transfer_quorum() + .map(|proof| Response::PendingTransferQuorum(Box::new(proof))), + ); + return; + } + if matches!(request.command, Command::Observe) { + reply( + request, + group + .owner + .observe() + .map(|value| Response::Observation(Box::new(value))), + ); + return; + } + if matches!(request.command, Command::ObservePartition) { + reply( + request, + group + .owner + .observe_partition() + .map(|value| Response::PartitionObservation(Box::new(value))), + ); + return; + } + if matches!(request.command, Command::ReplicationStatus) { + reply( + request, + group + .owner + .replication_status() + .map(Response::ReplicationStatus), + ); + return; + } + if let Command::CaptureCheckpoint(pool) = &request.command { + let source = group.owner.capture_checkpoint(pool); + reply(request, Ok(Response::CheckpointSource(Box::new(source)))); + return; + } + if let Some(term) = group.owner.parked_term() { + match &request.command { + Command::Step(message) | Command::AdmitStep(message) + if !wakes_parked_peer(message, term) => + { + // A delayed reply to the last active term carries no new + // protocol work. It must not undo completed parking. + let result = group + .owner + .validate_message(message) + .map(|()| Response::Unit); + reply(request, result); + return; + } + Command::SnapshotStatus(..) | Command::Unreachable(_) => { + reply(request, Ok(Response::Unit)); + return; + } + _ => (), + } + } + if let Command::SnapshotStatus(member, delivered) = request.command { + if let Err(error) = group.owner.validate_snapshot_member(member) { + reply(request, Err(error)); + return; + } + let Request { + reply: answer, + _permit, + .. + } = request; + group.snapshot_hints.insert(member, (delivered, _permit)); + group.wake(history, &mut self.order); + let _ = answer.send(Ok(Response::Unit)); + return; + } + if let Command::AdmitStep(message) = &request.command { + if let Err(error) = group.owner.validate_message(message) { + reply(request, Err(error)); + return; + } + let Some(inbound) = group.inbound.acquire(request.bytes) else { + reply(request, Err(RuntimeError::Overloaded)); + return; + }; + let Request { + command: Command::AdmitStep(message), + reply: answer, + bytes, + _permit, + .. + } = request + else { + unreachable!() + }; + let (reply, _) = oneshot::channel(); + group.deferred.push_back(Request { + group: Some(history), + command: Command::Step(message), + reply, + bytes, + _permit, + inbound: Some(inbound), + }); + group.wake(history, &mut self.order); + let _ = answer.send(Ok(Response::Unit)); + return; + } + if matches!( + request.command, + Command::Apply(..) | Command::CheckpointBeforeApply(..) + ) { + if group.owner.is_parked() { + reply(request, Err(RuntimeError::StaleCompletion)); + return; + } + if group.completion.is_some() { + reply(request, Err(RuntimeError::Overloaded)); + return; + } + group.completion = Some(request); + } else { + let Some(permit) = group.inbound.acquire(request.bytes) else { + reply(request, Err(RuntimeError::Overloaded)); + return; + }; + request.inbound = Some(permit); + group.deferred.push_back(request); + } + group.wake(history, &mut self.order); + } + + fn tick(&mut self) { + for history in &self.ticking { + if let Some(group) = self.groups.get_mut(history) { + group.tick = true; + group.wake(*history, &mut self.order); + } + } + } + + fn schedule_park_wakes(&mut self) -> Result, RuntimeError> { + if self.park_deadlines.next().is_none() { + return Ok(None); + } + let now = self.domain.mono_ms().map_err(|_| RuntimeError::Closed)?; + let wall = crate::asyncrt::wall_ms().max(0) as u64; + for _ in 0..PARK_WAKE_BATCH { + let Some(history) = self.park_deadlines.pop_due(wall) else { + break; + }; + if let Some(group) = self.groups.get_mut(&history) { + if group.failed.is_none() && group.owner.is_parked() { + group.wake_due = true; + group.wake(history, &mut self.order); + } + } + } + // Convert the next wall deadline at each scheduler turn. The periodic + // monotonic timer also resamples the wall clock after a clock change. + Ok(self + .park_deadlines + .next() + .map(|deadline| now.saturating_add(deadline.saturating_sub(wall)))) + } + + fn progress(&mut self) -> Result { + let mut made = false; + for _ in 0..self.order.len() { + let Some(history) = self.order.pop_front() else { + break; + }; + let collect = self.ready.saturating_add(self.inflight_bytes) + < self.options.max_ready_bytes + && self.apply < self.options.max_apply_bytes; + let group = self + .groups + .get_mut(&history) + .ok_or(RuntimeError::NotFound)?; + group.queued = false; + if let Some(term) = group.owner.parked_term().filter(|_| group.failed.is_none()) { + // Responses can arrive during the park fsync, before dispatch + // knows the voter is parked. They carry no new work. + group.snapshot_hints.clear(); + while group + .deferred + .front() + .is_some_and(|request| !wakes_parked_group(&request.command, term)) + { + let Some(request) = group.deferred.pop_front() else { + break; + }; + reply(request, Ok(Response::Unit)); + made = true; + } + if !group.owner.waiting() + && (group.wake_due + || !group.deferred.is_empty() + || !group.snapshot_hints.is_empty()) + { + match group.owner.begin_wake() { + Ok((work, bytes)) => { + self.park_deadlines.remove(&history); + group.write = Some(work); + group.write_bytes = bytes; + self.writes.push_back(history); + made = true; + } + Err(error) => group.fail(error), + } + } + } else if group.failed.is_none() { + if let Some(request) = group.completion.take() { + let result = match &request.command { + Command::Apply(ticket, next) => { + group.owner.complete_apply(ticket.clone(), (**next).clone()) + } + Command::CheckpointBeforeApply(ticket, checkpoint) => group + .owner + .checkpoint_before_apply(ticket, checkpoint.clone()), + _ => return Err(RuntimeError::Protocol("completion queue".into())), + }; + if matches!(result, Err(RuntimeError::NotReady)) { + group.completion = Some(request); + } else { + reply(request, result.map(|()| Response::Unit)); + made = true; + } + } + // Hints never wait behind ordinary admission or consume an + // apply-completion slot. Their protocol mutation still waits + // until the exact in-flight Ready/persistence phase has settled. + if group.owner.idle() { + if let Some((member, (delivered, _permit))) = group.snapshot_hints.pop_first() { + // A committed membership change may have removed this + // peer while the hint was queued; it then has no progress. + if group.owner.validate_snapshot_member(member).is_ok() { + group.owner.report_snapshot(member, delivered)?; + } + made = true; + } + } + if collect && group.owner.idle() && group.pending.is_none() { + if group.tick { + match group.owner.tick() { + Ok(()) => { + group.tick = false; + made = true; + } + Err(RuntimeError::NotReady) => (), + Err(RuntimeError::Overloaded) => { + group.tick = false; + made = true; + } + Err(error) => group.fail(error), + } + } else if let Some(request) = group.deferred.pop_front() { + let Request { + command, + reply: answer, + _permit, + inbound, + .. + } = request; + let mut answer = Some(answer); + let new_read = matches!(command, Command::ReadIndex(_)) + || matches!(&command, Command::Read(request) if request.kind.linearizable()) + || matches!(&command, Command::Step(m) if m.get_msg_type()==::raft::prelude::MessageType::MsgReadIndex); + let result = if new_read && self.reads >= self.options.max_pending_reads { + Err(RuntimeError::Overloaded) + } else { + match command { + Command::Read(request) => { + let now = + self.domain.mono_ms().map_err(|_| RuntimeError::Closed)?; + self.cell_reads.start( + history, + &mut group.owner, + *request, + answer.take().expect("read reply"), + now, + ); + Ok(Response::Unit) + } + Command::Execution(cell) => group + .owner + .execution_authority(&cell) + .map(|authority| Response::Execution(Box::new(authority))), + Command::NextEffect { now, after } => group + .owner + .next_effect(now, after.as_ref()) + .map(|authority| Response::NextEffect(authority.map(Box::new))), + Command::Enrollment(cell) => group + .owner + .enrollment_authority(cell) + .map(|authority| Response::Enrollment(Box::new(authority))), + Command::EnrollmentState(cell) => group + .owner + .enrollment_state(cell) + .map(|state| Response::EnrollmentState(Box::new(state))), + Command::Reconciliation(cell) => group + .owner + .reconciliation_authority(&cell) + .map(|authority| Response::Reconciliation(Box::new(authority))), + Command::ActivateRecovery(permit) => group + .owner + .activate_recovery(*permit) + .map(|result| Response::RecoveryActivation(Box::new(result))), + Command::ActivateTransfer(permit) => group + .owner + .activate_transfer(*permit) + .map(Response::TransferActivation), + Command::RetireSource(permit) => group + .owner + .retire_source(*permit) + .map(Response::SourceRetirement), + Command::PreparePark(request) => { + if self.cell_reads.busy(history) || group.completion.is_some() { + Err(RuntimeError::NotReady) + } else { + group + .owner + .prepare_park(request) + .map(Response::ParkPrepared) + } + } + Command::FencePark { + generation, + term, + deadline, + } => group + .owner + .fence_managed_park(generation, term, deadline) + .map(|()| Response::Unit), + Command::SealSource(permit) => group + .owner + .seal_source(*permit) + .map(|result| Response::SourceSeal(Box::new(result))), + Command::Configure(change) => { + group.owner.configure(change).map(Response::Proposal) + } + Command::TransferLeader(member) => { + group.owner.transfer_leader(member).map(|()| Response::Unit) + } + Command::TransferLeadership(transfer) => group + .owner + .transfer_leadership(transfer) + .map(|()| Response::Unit), + Command::TransferLeadershipBefore(transfer, deadline) => self + .domain + .mono_ms() + .map_err(|_| RuntimeError::Closed) + .and_then(|now| { + if now >= deadline { + return Err(RuntimeError::DeadlineExceeded); + } + group.owner.transfer_leadership(transfer) + }) + .map(|()| Response::Unit), + Command::RestartLeadership(check) => self + .domain + .mono_ms() + .map_err(|_| RuntimeError::Closed) + .and_then(|now| { + crate::raft::restart_leadership::check_gate( + &self.log, now, &check, + ) + }) + .and_then(|()| group.owner.restart_leadership(*check)) + .map(|()| Response::Unit), + Command::Campaign => { + group.owner.campaign().map(|()| Response::Unit) + } + Command::PublishCheckpoint(checkpoint) => group + .owner + .publish_checkpoint(checkpoint) + .map(|()| Response::Unit), + Command::Checkpoint(message, checkpoint) => group + .owner + .step_checkpoint(message, checkpoint) + .map(Response::Checkpoint), + Command::Step(message) => { + group.owner.step(message).map(|()| Response::Unit) + } + Command::Propose(data) => { + group.owner.propose(data).map(Response::Proposal) + } + Command::ProposeAtTerm(term, data) => group + .owner + .propose_at_term(term, data) + .map(Response::Proposal), + Command::Submit(prepared, _retained) => { + group.owner.submit(prepared).map(Response::Submission) + } + Command::Unreachable(member) => group + .owner + .report_unreachable(member) + .map(|_| Response::Unit), + Command::ReadIndex(context) => { + group.owner.read_index(context).map(|()| Response::Unit) + } + _ => Err(RuntimeError::Protocol("ordinary queue".into())), + } + }; + drop(_permit); + drop(inbound); + if let Some(answer) = answer { + let _ = answer.send(result); + } + made = true; + } + } + if group.write.is_none() + && (group.pending.is_none() && group.settled.is_none() + || group.owner.control_pending()) + { + let emit = group.pending.is_none() && group.settled.is_none(); + let control = group.owner.control_pending(); + match group.owner.poll_budget(collect && emit, emit) { + Ok(Some(CellGroupEffect::Persist(work))) => { + group.write_bytes = work.write.estimated_bytes()?; + group.write = Some(work); + self.writes.push_back(history); + made = true; + } + Ok(Some(CellGroupEffect::ReadIndex { + term, + index, + context, + })) if reads::reserved_context(&context) => { + self.cell_reads.observe(history, term, index, &context); + made = true; + } + Ok(Some(effect)) => { + if group.pending.is_none() { + group.pending = Some(Ok(effect)); + } else { + if !matches!( + effect, + CellGroupEffect::Applied { .. } + | CellGroupEffect::CheckpointInstalled { .. } + | CellGroupEffect::CheckpointPublished { .. } + ) || group.settled.is_some() + { + return Err(RuntimeError::Protocol( + "reserved physical completion output".into(), + )); + } + group.settled = Some(effect); + } + made = true; + } + Ok(None) => (), + Err(RuntimeError::Closed) => return Err(RuntimeError::Closed), + Err(error) => group.fail(error), + } + made |= control && !group.owner.control_pending(); + } + } + if group.pending.is_none() { + group.pending = group.settled.take().map(Ok); + } + if let Some(mut pending) = group.pending.take() { + let bytes = pending.as_ref().map_or(4096, CellGroupEffect::bytes); + if bytes > self.options.max_event_bytes { + group.fail(RuntimeError::Overloaded); + } else if let Some(permit) = self.output.acquire(bytes) { + let permit = Arc::new(permit); + if let Ok(CellGroupEffect::Message(_)) = &pending { + group.owner.track_delivery(&permit); + } + if let Ok(CellGroupEffect::Apply(work)) = &mut pending { + work.ticket.admission = Some(permit.clone()); + } + self.outbox.push_back(CellGroupRuntimeEvent { + group: history, + effect: pending, + _permit: permit, + }); + made = true; + } else { + group.pending = Some(pending); + } + } + if group.pending.is_some() + || group.settled.is_some() + || (group.failed.is_none() + && (group.tick + || group.wake_due + || !group.snapshot_hints.is_empty() + || !group.deferred.is_empty() + || group.completion.is_some() + || group.owner.has_work())) + { + group.wake(history, &mut self.order); + } + self.refresh(history); + } + Ok(made) + } + + fn operation(&mut self) -> Result { + self.sequence = self + .sequence + .checked_add(1) + .ok_or(RuntimeError::Overloaded)?; + Ok(OperationId::new(self.sequence)) + } + + fn start_batch(&mut self) -> Result { + if self.inflight.is_some() + || self.domain.mono_ms().map_err(|_| RuntimeError::Closed)? < self.retry_at + { + return Ok(false); + } + let mut jobs = Vec::new(); + let mut bytes = 1024; + for _ in 0..self.writes.len() { + let Some(history) = self.writes.pop_front() else { + break; + }; + let group = self + .groups + .get_mut(&history) + .ok_or(RuntimeError::NotFound)?; + let Some(work) = group.write.take() else { + continue; + }; + let size = std::mem::take(&mut group.write_bytes); + if size + 1024 > self.log.limits().max_batch_bytes { + group.fail(RuntimeError::Overloaded); + // The unsubmitted write is discarded; release its waiting ticket + // only by retiring this failed owner at node shutdown. + group.queued = true; + self.order.push_back(history); + } else if size + bytes <= self.log.limits().max_batch_bytes { + bytes += size; + jobs.push(Job { + history, + work, + bytes: size, + }); + } else { + group.write = Some(work); + group.write_bytes = size; + self.writes.push_front(history); + break; + } + self.refresh(history); + if jobs.len() >= self.options.max_batch_groups { + break; + } + } + if jobs.is_empty() { + return Ok(false); + } + let operation = self.operation()?; + let log = self.log.clone(); + self.inflight_bytes = bytes; + self.inflight = Some(Box::pin(async move { + let writes = jobs.iter().map(|job| job.work.write.clone()).collect(); + let result = log.persist_cells(operation, writes).await; + (jobs, operation, result) + })); + Ok(true) + } + + fn finish_batch(&mut self, batch: Batch) -> Result<(), RuntimeError> { + self.inflight = None; + self.inflight_bytes = 0; + let (jobs, operation, result) = batch; + match result { + Ok(receipt) => { + if receipt.operation != operation + || receipt.groups + != jobs + .iter() + .map(|j| j.work.write.group.identity().clone()) + .collect::>() + { + return Err(RuntimeError::StaleCompletion); + } + for job in jobs { + let group = self + .groups + .get_mut(&job.history) + .ok_or(RuntimeError::NotFound)?; + // The physical batch must join even after quarantine. Its + // successful sync cannot restore the revoked execution fence. + if group.failed.is_some() { + continue; + } + group.owner.complete_persist(CellGroupPersisted { + ticket: job.work.ticket, + sequence: receipt.sequence, + })?; + if let Some(LocalParkChange::Close(generation)) = job.work.write.local_park { + let campaign = std::mem::take(&mut group.wake_due) + || group.deferred.front().is_some_and(|request| { + !matches!( + request.command, + Command::Step(_) | Command::Checkpoint(..) + ) + }); + group.owner.finish_wake(generation, campaign)?; + self.ticking.insert(job.history); + } + if let Some(request) = group.park_write.take() { + let result = match &request.command { + Command::AcknowledgePark { leader, cut } => group + .owner + .finish_park_acknowledgement(*leader, cut.clone()) + .map(Response::ParkAcknowledgement), + Command::CertifyPark(certificate) => group + .owner + .finish_park_certificate(certificate) + .map(Response::ParkCertified), + Command::EnterPark(certificate) => { + match group.owner.finish_park(certificate) { + Ok(()) => { + group.tick = false; + self.ticking.remove(&job.history); + self.park_deadlines + .set(job.history, group.owner.parked_effect_deadline()); + Ok(Response::Unit) + } + Err(error) => { + group.fail(error.clone()); + Err(error) + } + } + } + _ => Err(RuntimeError::Protocol("park write queue".into())), + }; + reply(request, result); + } + group.wake(job.history, &mut self.order); + self.refresh(job.history); + } + } + Err(LogError::Overloaded) => { + self.retry_at = self + .domain + .mono_ms() + .map_err(|_| RuntimeError::Closed)? + .saturating_add(10); + for job in jobs.into_iter().rev() { + let group = self + .groups + .get_mut(&job.history) + .ok_or(RuntimeError::NotFound)?; + if group.failed.is_some() { + continue; + } + group.write = Some(job.work); + group.write_bytes = job.bytes; + self.writes.push_front(job.history); + self.refresh(job.history); + } + } + Err(error) => return Err(error.into()), + } + Ok(()) + } + + fn flush(&mut self) -> bool { + let mut made = false; + while let Some(event) = self.outbox.pop_front() { + match self.events.try_send(event) { + Ok(()) => made = true, + Err(error) => { + self.outbox.push_front(error.into_inner()); + break; + } + } + } + made + } + + pub(super) async fn run( + &mut self, + mut requests: mailbox::Receiver, + closing: &AtomicBool, + stop: &Notify, + ) -> Result<(), RuntimeError> { + let mut tick_at = self + .domain + .mono_ms() + .map_err(|_| RuntimeError::Closed)? + .saturating_add(self.options.tick_interval_ms.unwrap_or(1000)); + loop { + if closing.load(Ordering::Acquire) { + return Ok(()); + } + let wake_at = self.schedule_park_wakes()?; + let mut made = self.progress()? | self.start_batch()? | self.flush(); + let now = self.domain.mono_ms().map_err(|_| RuntimeError::Closed)?; + made |= self.cell_reads.progress( + now, + &self.domain, + |history, request, term, barrier, context, retry| { + let group = self + .groups + .get_mut(&history) + .ok_or(RuntimeError::NotFound)?; + if let Some(error) = &group.failed { + return Err(error.clone()); + } + match &request.kind { + reads::ReadKind::Cell(cell, consistency) => { + if matches!(consistency, CellReadConsistency::Linearizable) + && (term.is_none() || retry) + { + let status = group.owner.status()?; + if term.is_some_and(|observed| observed != status.term) { + return Err(RuntimeError::NotLeader); + } + // Raft already retries a leader's heartbeat. + // A follower must resend a lost forwarded request. + if term.is_none() + || status.leader != Some(group.owner.membership_read_origin()) + { + group.owner.read_forwarded_cell_index(context.to_vec())?; + *term = Some(status.term); + } + } + group + .owner + .select_read(cell, consistency, *term, *barrier) + .map(reads::Selection::Cell) + } + reads::ReadKind::Seed(target) => group + .owner + .select_seed(target, *term, *barrier) + .map(reads::Selection::Seed), + reads::ReadKind::Membership { minimum, mode } => { + if let reads::MembershipRead::Wake(generation) = mode { + group.owner.check_park_wake(*generation)?; + } + let status = group.owner.status()?; + if term.is_some_and(|observed| observed != status.term) { + return Err(RuntimeError::NotLeader); + } + if status.applied.index < *minimum + || !group.owner.membership_read_ready()? + { + return Err(RuntimeError::NotReady); + } + if barrier.is_some_and(|index| { + group.owner.manifest().marker().membership_revision() > index.get() + }) { + // A configuration overtook this read's barrier. + // Re-arm within the same invocation, reservation + // and original deadline; never spin a new call. + *term = None; + *barrier = None; + } + if term.is_none() + || retry + && status.leader != Some(group.owner.membership_read_origin()) + { + group.owner.read_membership_index(context.to_vec())?; + *term = Some(status.term); + return Err(RuntimeError::NotReady); + } + group + .owner + .select_membership(*minimum, *term, *barrier) + .map(reads::Selection::Membership) + } + } + }, + ); + let deadline = if self.retry_at > now { + tick_at.min(self.retry_at) + } else { + tick_at + }; + let deadline = self + .cell_reads + .deadline() + .map_or(deadline, |read| deadline.min(read)); + let deadline = wake_at.map_or(deadline, |wake| deadline.min(wake)); + let timer = self + .domain + .sleep_until(deadline) + .map_err(|_| RuntimeError::Closed)?; + let events = self.events.clone(); + let output = self.output.clone(); + let io = async { + match self.inflight.as_mut() { + Some(io) => io.await, + None => std::future::pending().await, + } + }; + crate::asyncrt::select! { + request = requests.recv() => { + let Some(request) = request else { return Ok(()); }; self.dispatch(request); + for _ in 1..64 { let Ok(request) = requests.try_recv() else { break; }; self.dispatch(request); } + } + _ = stop.notified() => return Ok(()), + result = io => self.finish_batch(result)?, + _ = timer => { let now=self.domain.mono_ms().map_err(|_| RuntimeError::Closed)?; if now>=tick_at { if self.options.tick_interval_ms.is_some() { self.tick(); } tick_at=now.saturating_add(self.options.tick_interval_ms.unwrap_or(1000)); } } + _ = events.closed() => return Ok(()), + capacity = events.reserve(), if !self.outbox.is_empty() => { if capacity.is_err() { return Ok(()); } } + _ = output.wake.notified() => (), + _ = std::future::ready(()), if made => (), + } + } + } + + pub(super) async fn finish(&mut self) -> Result<(), RuntimeError> { + for group in self.groups.values_mut() { + if let Some(request) = group.park_write.take() { + reply(request, Err(RuntimeError::Closed)); + } + } + if let Some(io) = self.inflight.take() { + let batch = io.await; + batch.2.as_ref().map_err(Clone::clone)?; + self.finish_batch(batch)?; + } + let operation = self.operation()?; + self.log.persist_cells(operation, Vec::new()).await?; + Ok(()) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/seed_authority.rs b/crates/cellar/durability/raft/cell_groups/seed_authority.rs new file mode 100644 index 000000000..d0eafc35d --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/seed_authority.rs @@ -0,0 +1,230 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Fresh quorum authorization for a complete-group learner seed. Destination +//! authentication, permanent claims and installation are separate boundaries. + +use super::{ + CellGroupCheckpoint, CellGroupCheckpointChunk, CellGroupCheckpointError, + CellGroupCheckpointPool, GroupManifest, PreparedGroupManifest, +}; +use crate::{ + asyncrt::DomainToken, + raft::{budget::Permit, cache::GroupLog, completion::RuntimeError}, +}; +use cellar_logic::{ + cell_group::GroupHistory, + consensus::{LogIndex, LogPosition, ReplicaIdentity}, +}; +use std::sync::{ + atomic::{AtomicU64, Ordering}, + Arc, +}; + +pub(super) struct SeedSelection { + pub group: GroupLog, + pub root: PreparedGroupManifest, + pub target: ReplicaIdentity, + pub barrier: LogPosition, + pub leader: Arc, +} + +/// Reject birth members, retired IDs, other disks, and members in either voter +/// set. A joining learner has one permanent physical binding in the applied log. +pub(super) fn check_target( + group: &GroupLog, + target: &ReplicaIdentity, +) -> Result { + let cache = group.read()?; + let local = group.identity(); + let id = target.member.get(); + if target.scope != local.scope + || target.node == local.node + || !cache.conf_state.learners.contains(&id) + || cache.conf_state.voters.contains(&id) + || cache.conf_state.voters_outgoing.contains(&id) + || cache.conf_state.learners_next.contains(&id) + || !cache.membership.permits_group(target) + { + return Err(RuntimeError::Invalid("group seed target".into())); + } + let (binding, joined_at) = cache + .membership + .joined(target.member)? + .ok_or_else(|| RuntimeError::Invalid("group seed requires a joined learner".into()))?; + if binding.node != target.node || binding.disk != target.disk { + return Err(RuntimeError::Invalid("group seed member binding".into())); + } + Ok(joined_at) +} + +/// One non-cloneable, deadline-bound ReadIndex result. Holding this value pins a +/// complete applied root and charges the existing node and group read budgets. +/// Capture and stream on an admitted blocking owner, retaining it through actual +/// completion even when a network waiter disconnects. +pub struct CellGroupSeedAuthority { + selection: SeedSelection, + deadline: u64, + domain: DomainToken, + _node: Permit, + _group: Permit, +} + +impl CellGroupSeedAuthority { + pub(crate) fn belongs_to_engine(&self, owner: &Arc<()>) -> bool { + Arc::ptr_eq(&self.selection.group.owner, owner) + } + + pub(crate) fn marker(&self) -> &GroupManifest { + self.selection.root.marker() + } + + pub(super) fn new( + selection: SeedSelection, + deadline: u64, + domain: DomainToken, + mut node: Permit, + mut group: Permit, + ) -> Result { + let bytes = selection + .root + .reserved_bytes() + .checked_add(4096) + .ok_or(RuntimeError::Overloaded)?; + if !node.grow_to(bytes) || !group.grow_to(bytes) { + return Err(RuntimeError::Overloaded); + } + let authority = Self { + selection, + deadline, + domain, + _node: node, + _group: group, + }; + authority.check()?; + Ok(authority) + } + + pub fn target(&self) -> &ReplicaIdentity { + &self.selection.target + } + + pub fn barrier(&self) -> LogPosition { + self.selection.barrier + } + + pub fn deadline(&self) -> u64 { + self.deadline + } + + /// Leadership, applied membership and deadline remain live constraints. + /// This check cannot replace the fresh ReadIndex that created the value. + pub fn check(&self) -> Result<(), RuntimeError> { + if self.domain.mono_ms().map_err(|_| RuntimeError::Closed)? >= self.deadline { + return Err(RuntimeError::DeadlineExceeded); + } + let s = &self.selection; + let cache = s.group.read()?; + let marker = s.root.marker(); + let local = s.group.identity(); + let term = s.barrier.term.get(); + if !marker.is_active() + || s.leader.load(Ordering::Acquire) != term + || cache.hard_state.term != term + || cache + .group_manifest + .as_ref() + .is_none_or(|m| m.route() != marker.route() || !m.is_active()) + || cache.membership.revision().get() != marker.membership_revision() + || !cache.membership.permits_group(local) + || !cache + .conf_state + .voters + .iter() + .chain(&cache.conf_state.voters_outgoing) + .any(|id| *id == local.member.get()) + { + return Err(RuntimeError::NotReady); + } + drop(cache); + if check_target(&s.group, &s.target)? > s.barrier.index { + return Err(RuntimeError::NotReady); + } + Ok(()) + } + + /// Capture the exact selected root. If apply advanced before capture, refuse + /// and obtain a fresh authority. Ordinary writes after capture may continue; + /// the admitted source retains its immutable cut. Membership changes revoke it. + pub fn capture( + self, + pool: &CellGroupCheckpointPool, + ) -> Result { + self.check().map_err(CellGroupCheckpointError::Authority)?; + let checkpoint = pool.capture(&self.selection.group, &self.selection.root)?; + self.check().map_err(CellGroupCheckpointError::Authority)?; + Ok(CellGroupSeedSource { + authority: self, + checkpoint, + }) + } +} + +/// A one-shot complete checkpoint bound to one joining process and fresh quorum +/// barrier. Neither its metadata nor a completed stream grants installation or +/// voting authority. A receiver must require an authenticated successful End. +pub struct CellGroupSeedSource { + authority: CellGroupSeedAuthority, + checkpoint: CellGroupCheckpoint, +} + +impl CellGroupSeedSource { + /// Retain the same fresh authority while its exact complete cut is uploaded. + /// This is data preparation, never a publication or installation permission. + pub(crate) fn into_archive_parts( + self, + ) -> Result<(CellGroupSeedAuthority, CellGroupCheckpoint), CellGroupCheckpointError> { + self.check().map_err(CellGroupCheckpointError::Authority)?; + Ok((self.authority, self.checkpoint)) + } + + pub fn target(&self) -> &ReplicaIdentity { + self.authority.target() + } + + pub fn barrier(&self) -> LogPosition { + self.authority.barrier() + } + + pub fn marker(&self) -> &GroupManifest { + self.checkpoint.marker() + } + + pub fn check(&self) -> Result<(), RuntimeError> { + self.authority.check() + } + + pub fn snapshot(&self) -> Result { + self.check().map_err(CellGroupCheckpointError::Authority)?; + self.checkpoint.snapshot() + } + + /// No lock is held across the sink. A revoked or expired stream cannot report + /// success, including when the last sink call stalls beyond the deadline. + pub fn stream( + &mut self, + mut sink: impl FnMut(CellGroupCheckpointChunk<'_>) -> Result<(), CellGroupCheckpointError>, + ) -> Result<(), CellGroupCheckpointError> { + self.check().map_err(CellGroupCheckpointError::Authority)?; + let authority = &self.authority; + self.checkpoint.stream(|chunk| { + authority + .check() + .map_err(CellGroupCheckpointError::Authority)?; + sink(chunk)?; + authority + .check() + .map_err(CellGroupCheckpointError::Authority) + })?; + self.check().map_err(CellGroupCheckpointError::Authority) + } +} diff --git a/crates/cellar/durability/raft/command_cell_transport.rs b/crates/cellar/durability/raft/command_cell_transport.rs new file mode 100644 index 000000000..b6788b236 --- /dev/null +++ b/crates/cellar/durability/raft/command_cell_transport.rs @@ -0,0 +1,882 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Commands and enrollment share bounded RPC admission and explicit write grants. +use super::*; +use crate::application::consensus::CommandFailure; +use crate::raft::cell_groups::{ + CellDispatchError, CellGroupController, CellGroupDispatcher, CellGroupNamespaceStorage, + CellRoutingLimits, EnrollmentRecord, +}; +use crate::raft::query_identity::QueryRoute; +use crate::raft::query_wire::{ + self as wire, + command::{CellCommandCall, CellCommandProbe, CellCommandValue}, + enrollment::{CellEnrollmentCall, CellEnrollmentValue}, + receipt::{CellReceiptCall, CellReceiptValue}, +}; +use crate::raft::version::{CellVersionObservation, CellVersionToken, VersionKeys}; +use cellar_logic::{ + cell_group::CellHistory, + consensus::{MemberId, NodeIdentity}, +}; +use queries::{QueryConnection, SendQuery}; +use std::time::Duration; + +/// Host-owned registration. The transport retains only a weak reference, so it +/// cannot extend the host's lifetime or create a runtime/transport ownership cycle. +#[derive(Clone)] +pub struct CellCommandGateway { + inner: Arc, +} + +// Accepted requests retain Gateway, never this admission owner. In-flight work +// cannot keep accepting new commands after the final host registration is dropped. +struct Registration { + gateway: Arc, +} + +impl Drop for Registration { + fn drop(&mut self) { + self.gateway.accepting.store(false, Ordering::Release); + } +} + +pub(super) struct Gateway { + accepting: AtomicBool, + controller: CellGroupController, + dispatcher: CellGroupDispatcher, + storage: CellGroupNamespaceStorage, +} + +/// Retains the same node/peer query admission through consumption of the result. +pub struct CellRemoteCommandResponse { + value: CellCommandValue, + _node: Permit, + _peer: Permit, +} + +impl CellRemoteCommandResponse { + pub fn command(&self) -> cellar_logic::consensus::CommandId { + self.value.command + } + + pub fn retry_until_ms(&self) -> u64 { + self.value.retry_until_ms + } + + pub fn durability(&self) -> cellar_logic::consensus::Durability { + self.value.durability + } + + pub fn result(&self) -> &[u8] { + &self.value.value.result + } + + pub fn observation(&self) -> &CellVersionObservation { + &self.value.value.observation + } + + pub fn token(&self) -> &CellVersionToken { + &self.value.value.token + } + + pub fn served_by(&self) -> MemberId { + self.value.value.served_by + } +} + +/// An authenticated creation receipt, retaining transport admission until dropped. +/// The original enrollment image is not a claim about the latest cell version. +pub struct CellRemoteEnrollmentResponse { + record: EnrollmentRecord, + durability: cellar_logic::consensus::Durability, + served_by: MemberId, + _node: Permit, + _peer: Permit, +} + +impl CellRemoteEnrollmentResponse { + pub fn record(&self) -> &EnrollmentRecord { + &self.record + } + + pub fn served_by(&self) -> MemberId { + self.served_by + } + + pub fn durability(&self) -> cellar_logic::consensus::Durability { + self.durability + } +} + +/// Receipt inspection retains reply admission but never owns a producer. +pub struct CellRemoteReceiptResponse { + value: CellReceiptValue, + _node: Permit, + _peer: Permit, +} + +impl CellRemoteReceiptResponse { + pub fn durability(&self) -> cellar_logic::consensus::Durability { + self.value.value.durability + } + + pub fn command(&self) -> cellar_logic::consensus::CommandId { + self.value.value.command + } + + /// None remains ambiguous; no command or initializer was invoked by lookup. + pub fn result(&self) -> Option<&[u8]> { + self.value + .retained + .then_some(self.value.value.value.result.as_slice()) + } + + pub fn retry_until_ms(&self) -> Option { + self.value + .retained + .then_some(self.value.value.retry_until_ms) + } + + pub fn observation(&self) -> &CellVersionObservation { + &self.value.value.value.observation + } + + pub fn token(&self) -> &CellVersionToken { + &self.value.value.value.token + } + + pub fn served_by(&self) -> MemberId { + self.value.value.value.served_by + } +} + +enum WriteCall { + Command(CellCommandCall), + Probe(CellCommandProbe), + Enrollment(CellEnrollmentCall), + Receipt(CellReceiptCall), +} + +impl WriteCall { + fn encode(&self, limit: usize) -> Result, TransportError> { + match self { + Self::Command(c) => c.encode(limit), + Self::Probe(c) => c.encode(limit), + Self::Enrollment(c) => c.encode(limit), + Self::Receipt(c) => c.encode(limit), + } + } + + fn shorten_timeout(&mut self, remaining: u64) { + let timeout = match self { + Self::Command(c) => &mut c.timeout_ms, + Self::Probe(c) => &mut c.timeout_ms, + Self::Enrollment(c) => &mut c.timeout_ms, + Self::Receipt(c) => &mut c.timeout_ms, + }; + *timeout = (*timeout).min(remaining); + } + + fn input_bytes(&self) -> usize { + match self { + Self::Command(c) => c.request.input.len(), + Self::Enrollment(_) | Self::Probe(_) => 0, + Self::Receipt(c) => c.request.input.len(), + } + } +} + +enum WriteValue { + Command(CellCommandValue), + Probe(bool), + Enrollment(CellEnrollmentValue), + Receipt(CellReceiptValue), +} + +struct WriteResponse { + value: WriteValue, + served_by: MemberId, + node: Permit, + peer: Permit, +} + +pub(super) struct SendCellWrite { + route: Arc>, + peer: RaftPeer, + cell: CellHistory, + call: WriteCall, + keys: VersionKeys, + deadline: u64, + cancel: watch::Receiver, + reply: oneshot::Sender>, + node: Permit, + peer_bytes: Permit, +} + +struct Cancel(watch::Sender); + +impl Drop for Cancel { + fn drop(&mut self) { + self.0.send_replace(true); + } +} + +impl RaftTransport { + /// Register compiled commands and initialization on the exact controller/runtime owner. Retain + /// this host-owned handle while accepting commands. Dropping it refuses new + /// admission; commands already transferred to the controller remain owned. + /// Namespace write grants are independently required even for voter peers. + pub fn install_cell_command_gateway( + &self, + dispatcher: CellGroupDispatcher, + controller: CellGroupController, + storage: CellGroupNamespaceStorage, + ) -> Result { + let state = &self.handle.0; + let runtime = state.runtime.shared()?; + if !dispatcher.uses_domain(&state.domain) || !controller.belongs_to(runtime) { + return Err(TransportError::Identity); + } + let (namespace, _, _) = dispatcher + .host_binding() + .map_err(|_| TransportError::Identity)?; + storage + .validate_namespace(&namespace) + .map_err(|_| TransportError::Identity)?; + let key = dispatcher.namespace(); + let mut gateways = state + .cell_command_gateways + .lock() + .map_err(|_| TransportError::Closed)?; + let registry = state.registry.lock().map_err(|_| TransportError::Closed)?; + state.admit(®istry)?; + gateways.retain(|_, value| value.strong_count() > 0); + if gateways.contains_key(&key) { + return Err(TransportError::Refused); + } + if gateways.len() >= state.options.max_groups { + return Err(TransportError::Overloaded); + } + let inner = Arc::new(Gateway { + accepting: AtomicBool::new(true), + controller, + dispatcher, + storage, + }); + gateways.insert(key, Arc::downgrade(&inner)); + Ok(CellCommandGateway { + inner: Arc::new(Registration { gateway: inner }), + }) + } + + pub(crate) fn uses_command_domain(&self, domain: &DomainToken) -> bool { + self.handle.0.domain.same_owner(domain) + } + + /// Called only by a consumed catalog attempt. No forwarding or retry occurs + /// here; the caller keeps its original deadline and attempt budget. + pub(in crate::raft) async fn command_cell( + &self, + route: &cellar_logic::cell_group::GroupRoute, + cell: &CellHistory, + member: MemberId, + call: CellCommandCall, + keys: VersionKeys, + deadline: u64, + ) -> Result { + let response = self + .write_cell( + route, + cell, + member, + WriteCall::Command(call), + keys, + deadline, + ) + .await?; + let WriteValue::Command(value) = response.value else { + return Err(TransportError::Protocol); + }; + Ok(CellRemoteCommandResponse { + value, + _node: response.node, + _peer: response.peer, + }) + } + + pub(in crate::raft) async fn probe_cell_command_leader( + &self, + route: &cellar_logic::cell_group::GroupRoute, + cell: &CellHistory, + member: MemberId, + keys: VersionKeys, + deadline: u64, + ) -> Result { + let timeout_ms = deadline + .checked_sub( + self.handle + .0 + .domain + .mono_ms() + .map_err(|_| TransportError::Closed)?, + ) + .filter(|ms| *ms > 0) + .ok_or(TransportError::Timeout)?; + let call = CellCommandProbe { + timeout_ms, + route_epoch: route.epoch.get(), + birth_digest: route.birth_digest, + }; + let timer = self + .handle + .0 + .domain + .sleep_until(deadline) + .map_err(|_| TransportError::Closed)?; + let response = crate::asyncrt::select! { + result = self.write_cell(route, cell, member, WriteCall::Probe(call), keys, deadline) => result?, + _ = timer => return Err(TransportError::Timeout), + }; + match response.value { + WriteValue::Probe(leader) => Ok(leader), + _ => Err(TransportError::Protocol), + } + } + + pub(in crate::raft) async fn enroll_cell( + &self, + route: &cellar_logic::cell_group::GroupRoute, + cell: &CellHistory, + member: MemberId, + call: CellEnrollmentCall, + keys: VersionKeys, + deadline: u64, + ) -> Result { + let response = self + .write_cell( + route, + cell, + member, + WriteCall::Enrollment(call), + keys, + deadline, + ) + .await?; + let WriteValue::Enrollment(value) = response.value else { + return Err(TransportError::Protocol); + }; + Ok(CellRemoteEnrollmentResponse { + record: value.record, + durability: value.durability, + served_by: response.served_by, + _node: response.node, + _peer: response.peer, + }) + } + + pub(in crate::raft) async fn lookup_cell_receipt( + &self, + route: &cellar_logic::cell_group::GroupRoute, + cell: &CellHistory, + member: MemberId, + call: CellReceiptCall, + keys: VersionKeys, + deadline: u64, + ) -> Result { + let response = self + .write_cell( + route, + cell, + member, + WriteCall::Receipt(call), + keys, + deadline, + ) + .await?; + let WriteValue::Receipt(value) = response.value else { + return Err(TransportError::Protocol); + }; + Ok(CellRemoteReceiptResponse { + value, + _node: response.node, + _peer: response.peer, + }) + } + + async fn write_cell( + &self, + route: &cellar_logic::cell_group::GroupRoute, + cell: &CellHistory, + member: MemberId, + call: WriteCall, + keys: VersionKeys, + deadline: u64, + ) -> Result { + let state = &self.handle.0; + // Discovery never creates a local replica or grants voting authority. + state.runtime.shared()?; + let (view, peer, pool) = { + let registry = state.registry.lock().map_err(|_| TransportError::Closed)?; + state.admit(®istry)?; + let view = registry + .physical_groups + .get(&route.history) + .or_else(|| registry.cell_query_groups.get(&route.history)) + .ok_or(TransportError::UnknownGroup)? + .clone(); + if view.schema_digest != route.schema_digest || !route.history.same_namespace(cell) { + return Err(TransportError::Identity); + } + let peer = view + .peers + .iter() + .find(|p| { + p.replica.member == member + && p.replica.node != state.runtime.log().limits().node + }) + .ok_or(TransportError::Identity)? + .clone(); + let pool = registry + .pools + .get(&peer.replica.node) + .ok_or(TransportError::StalePeer)? + .clone(); + (view, peer, pool) + }; + if state.domain.mono_ms().map_err(|_| TransportError::Closed)? >= deadline { + return Err(TransportError::Timeout); + } + let charge = queries::reservation( + call.input_bytes() + .checked_add(8192) + .ok_or(TransportError::Limit)?, + &state.options, + )?; + let node = state + .outgoing + .query + .acquire(charge) + .ok_or(TransportError::Overloaded)?; + let peer_bytes = pool + .outgoing + .query + .acquire(charge) + .ok_or(TransportError::Overloaded)?; + drop(call.encode(state.options.max_query_request_bytes)?); + if !state.current_cell_query_route(&view, &peer) { + return Err(TransportError::StalePeer); + } + let (reply, result) = oneshot::channel(); + let (cancel, cancelled) = watch::channel(false); + let _cancel = Cancel(cancel); + pool.query + .try_send(SendQuery::Write(Box::new(SendCellWrite { + route: view, + peer, + cell: cell.clone(), + call, + keys, + deadline, + cancel: cancelled, + reply, + node, + peer_bytes, + }))) + .map_err(|e| match e { + mpsc::error::TrySendError::Full(_) => TransportError::Overloaded, + mpsc::error::TrySendError::Closed(_) => TransportError::Closed, + })?; + result.await.map_err(|_| TransportError::Closed)? + } +} + +impl CellCommandGateway { + /// The cluster and namespace bound to this host-owned registration. + pub fn namespace(&self) -> (ClusterId, NamespaceId) { + self.inner.gateway.dispatcher.namespace() + } +} + +impl State { + pub(super) fn authorize_cell_command( + &self, + wire: &QueryRoute, + ) -> Result<(Arc, Option>), TransportError> { + let runtime = self.runtime.shared()?; + let (source, target, _) = wire.decode_shared()?; + let registry = self.registry.lock().map_err(|_| TransportError::Closed)?; + self.admit(®istry)?; + let route = registry + .physical_groups + .get(&target.scope) + .ok_or(TransportError::UnknownGroup)?; + let group = runtime + .log() + .cell_group(&target.scope) + .map_err(|_| TransportError::UnknownGroup)?; + if *group.identity() != target + || wire.target_label() != self.auth.source() + || wire.schema() != route.schema_digest + { + return Err(TransportError::StalePeer); + } + shared::permitted(&group, &[&target])?; + let grant = registry + .command_origins + .get(&(target.scope.cluster, target.scope.namespace, source.node)) + .filter(|grant| !*grant.stop.borrow()) + .filter(|grant| { + grant.peer.as_ref().is_some_and(|peer| { + peer.node == source.node + && peer.disk == source.disk + && peer.process == source.process + && peer.label == wire.source_label() + }) + }) + .ok_or(TransportError::Refused)?; + let pool = registry + .pools + .get(&source.node) + .filter(|pool| grant.peer.as_ref() == Some(&pool.peer)) + .cloned() + .ok_or(TransportError::Refused)?; + Ok((pool, Some(grant.stop.subscribe()))) + } + + pub(super) async fn send_cell_write( + &self, + pool: &Pool, + mut request: SendCellWrite, + connection: &mut Option, + ) -> bool { + let result = self + .query_until( + pool, + request.deadline, + request.cancel.clone(), + None, + async { + if !self.current_cell_query_route(&request.route, &request.peer) { + return Err(TransportError::StalePeer); + } + let remaining = request + .deadline + .checked_sub(self.domain.mono_ms().map_err(|_| TransportError::Closed)?) + .filter(|n| *n > 0) + .ok_or(TransportError::Timeout)?; + request.call.shorten_timeout(remaining); + let body = request.call.encode(self.options.max_query_request_bytes)?; + let local = self.runtime.log().limits(); + let route = QueryRoute::new_shared( + NodeIdentity { + node: local.node, + disk: local.disk, + process: local.process, + }, + self.auth.source(), + &request.peer.replica, + &request.peer.label, + &request.cell, + request.route.schema_digest, + )?; + let route = match &request.call { + WriteCall::Command(_) => route.for_command()?, + WriteCall::Probe(_) => route.for_command_probe()?, + WriteCall::Enrollment(_) => route.for_enrollment()?, + WriteCall::Receipt(_) => route.for_receipt()?, + }; + let body = self + .exchange_query(connection, &RaftPeerNode::from(&request.peer), route, body) + .await?; + if !self.current_cell_query_route(&request.route, &request.peer) { + return Err(TransportError::StalePeer); + } + match &request.call { + WriteCall::Command(call) => { + let now = u64::try_from(crate::asyncrt::wall_ms()) + .map_err(|_| TransportError::Authentication)?; + wire::command::decode_result( + &body, + (call.request.id, &request.cell, call.request.durability), + request.route.schema_digest, + &request.keys, + now, + self.options.max_query_response_bytes, + request.peer.replica.member, + )? + .map(WriteValue::Command) + } + WriteCall::Probe(_) => { + return wire::command::decode_probe_result(&body) + .map(WriteValue::Probe); + } + WriteCall::Receipt(call) => { + let now = u64::try_from(crate::asyncrt::wall_ms()) + .map_err(|_| TransportError::Authentication)?; + wire::receipt::decode_result( + &body, + (call.request.id, &request.cell, call.request.durability), + request.route.schema_digest, + &request.keys, + now, + self.options.max_query_response_bytes, + request.peer.replica.member, + )? + .map(WriteValue::Receipt) + } + WriteCall::Enrollment(call) => wire::enrollment::decode_result( + &body, + (call.request.creation, call.request.durability), + &request.cell, + request.route.schema_digest, + self.options.max_query_response_bytes, + )? + .map(WriteValue::Enrollment), + } + .map_err(TransportError::Command) + }, + ) + .await; + // A fully authenticated application refusal consumed a complete reply; + // retain the peer connection just as the legacy query lane does. + let failed = matches!(&result, Err(error) if !matches!(error, TransportError::Command(_))); + let _ = request.reply.send(result.map(|value| WriteResponse { + value, + served_by: request.peer.replica.member, + node: request.node, + peer: request.peer_bytes, + })); + failed + } +} + +impl State { + pub(super) async fn execute_cell_receipt( + &self, + route: &QueryRoute, + call: CellReceiptCall, + ) -> Result { + let (_, target, cell) = route.decode_shared()?; + let gateway = self + .cell_command_gateways + .lock() + .map_err(|_| TransportError::Closed)? + .get(&(cell.cluster, cell.namespace)) + .and_then(|g| g.upgrade()) + .ok_or(TransportError::Refused)?; + if !gateway.accepting.load(Ordering::Acquire) { + return Err(TransportError::Refused); + } + let options = gateway.storage.for_cell(&cell); + let binding = crate::raft::cell_groups::RemoteMutationRoute { + group: target.scope, + schema: route.schema(), + epoch: call.route_epoch, + birth: call.birth_digest, + }; + let result = async { + let mut invocation = gateway + .dispatcher + .lookup_receipt( + cell, + call.request, + CellRoutingLimits { + timeout: Duration::from_millis(call.timeout_ms), + max_attempts: 1, + }, + None, + ) + .map_err(|e| command_failure(&e))?; + invocation.bind_remote_route(binding); + self.authorize_cell_command(route) + .map_err(|_| CommandFailure::Unavailable)?; + gateway + .controller + .lookup_receipt(invocation, options) + .await + .map_err(|e| command_failure(&e)) + } + .await; + super::query_replies::Reply::receipt(result, self.options.max_query_response_bytes) + } + + pub(super) async fn execute_cell_enrollment( + &self, + route: &QueryRoute, + call: CellEnrollmentCall, + ) -> Result { + let (_, target, cell) = route.decode_shared()?; + let gateway = self + .cell_command_gateways + .lock() + .map_err(|_| TransportError::Closed)? + .get(&(cell.cluster, cell.namespace)) + .and_then(|g| g.upgrade()) + .ok_or(TransportError::Refused)?; + if !gateway.accepting.load(Ordering::Acquire) { + return Err(TransportError::Refused); + } + let options = gateway.storage.for_cell(&cell); + let binding = crate::raft::cell_groups::RemoteMutationRoute { + group: target.scope, + schema: route.schema(), + epoch: call.route_epoch, + birth: call.birth_digest, + }; + let result = async { + let mut invocation = gateway + .dispatcher + .enroll_with_inputs( + cell, + call.request, + CellRoutingLimits { + timeout: Duration::from_millis(call.timeout_ms), + max_attempts: 1, + }, + None, + Some((call.inputs, call.preview)), + ) + .await + .map_err(|error| command_failure(&error.cause))?; + invocation.bind_remote_route(binding); + if !gateway.accepting.load(Ordering::Acquire) { + return Err(CommandFailure::Unavailable); + } + self.authorize_cell_command(route) + .map_err(|_| CommandFailure::Unavailable)?; + let call = gateway + .controller + .enroll(invocation, options) + .map_err(|error| command_failure(&error.error.cause))?; + call.response() + .await + .map_err(|error| command_failure(&error.cause))? + .into_local() + .ok_or(CommandFailure::CorruptState) + } + .await; + super::query_replies::Reply::enrollment(result, self.options.max_query_response_bytes) + } + + pub(super) async fn execute_cell_command_probe( + &self, + route: &QueryRoute, + call: CellCommandProbe, + ) -> Result { + let (_, target, cell) = route.decode_shared()?; + let gateway = self + .cell_command_gateways + .lock() + .map_err(|_| TransportError::Closed)? + .get(&(cell.cluster, cell.namespace)) + .and_then(|g| g.upgrade()) + .ok_or(TransportError::Refused)?; + let runtime = self.runtime.shared()?; + let status = runtime + .status(target.scope) + .await + .map_err(TransportError::Runtime)?; + self.authorize_cell_command(route)?; + if !gateway.accepting.load(Ordering::Acquire) { + return Err(TransportError::Refused); + } + let manifest = runtime + .log() + .cell_group(&target.scope) + .and_then(|group| group.group_manifest()) + .map_err(|error| TransportError::Runtime(error.into()))? + .ok_or(TransportError::UnknownGroup)?; + let selected = manifest.route(); + if selected.history != target.scope + || selected.schema_digest != route.schema() + || selected.epoch.get() != call.route_epoch + || selected.birth_digest != call.birth_digest + || selected + .check_cell( + &cell, + &crate::application::consensus::cell_routing_key(cell.namespace, &cell.cell), + ) + .is_err() + { + return Err(TransportError::StalePeer); + } + // The sample selects a destination for one attempt. Command admission + // must independently check current local Raft authority. + let leader = status.ready && status.leader == Some(target.member.get()); + Ok(super::query_replies::Reply::plain( + wire::command::encode_probe_result(leader), + )) + } + + pub(super) async fn execute_cell_command( + &self, + route: &QueryRoute, + call: CellCommandCall, + ) -> Result { + let (_, target, cell) = route.decode_shared()?; + let gateway = self + .cell_command_gateways + .lock() + .map_err(|_| TransportError::Closed)? + .get(&(cell.cluster, cell.namespace)) + .and_then(|g| g.upgrade()) + .ok_or(TransportError::Refused)?; + if !gateway.accepting.load(Ordering::Acquire) { + return Err(TransportError::Refused); + } + let options = gateway.storage.for_cell(&cell); + let binding = crate::raft::cell_groups::RemoteMutationRoute { + group: target.scope, + schema: route.schema(), + epoch: call.route_epoch, + birth: call.birth_digest, + }; + let result = async { + let mut invocation = gateway + .dispatcher + .command_with_inputs( + cell, + call.request, + CellRoutingLimits { + timeout: Duration::from_millis(call.timeout_ms), + max_attempts: 1, + }, + None, + Some(call.inputs), + ) + .await + .map_err(|error| command_failure(&error.cause))?; + invocation.bind_remote_route(binding); + if !gateway.accepting.load(Ordering::Acquire) { + return Err(CommandFailure::Unavailable); + } + // Recheck after asynchronous request construction, before ownership + // transfers. Later revocation cannot undo an admitted mutation. + self.authorize_cell_command(route) + .map_err(|_| CommandFailure::Unavailable)?; + let call = gateway + .controller + .command(invocation, options) + .map_err(|error| command_failure(&error.error.cause))?; + call.response() + .await + .map_err(|error| command_failure(&error.cause)) + } + .await; + super::query_replies::Reply::command(result, self.options.max_query_response_bytes) + } +} + +fn command_failure(error: &CellDispatchError) -> CommandFailure { + use crate::raft::execution::ExecutionError; + match error { + CellDispatchError::Execution(ExecutionError::Refused(failure)) => failure.clone(), + CellDispatchError::Runtime(RuntimeError::NotLeader) => { + CommandFailure::NotLeader { leader: None } + } + CellDispatchError::Runtime(RuntimeError::Overloaded) => CommandFailure::Overloaded, + CellDispatchError::Registration(_) => CommandFailure::UnsupportedCapability, + _ => CommandFailure::Unavailable, + } +} diff --git a/crates/cellar/durability/raft/command_cell_wire.rs b/crates/cellar/durability/raft/command_cell_wire.rs new file mode 100644 index 000000000..2a982910a --- /dev/null +++ b/crates/cellar/durability/raft/command_cell_wire.rs @@ -0,0 +1,307 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Command payloads are accepted only under an authenticated command-purpose route. +use super::*; +use crate::raft::cell_groups::{CellCommandRequest, CellCommandResponse}; +use crate::storage::CommandInputs; +use cellar_logic::{ + cell_group::CellHistory, + consensus::{ClientId, CommandId, Durability}, +}; + +pub(in crate::raft) struct CellCommandCall { + pub request: CellCommandRequest, + pub inputs: CommandInputs, + pub timeout_ms: u64, + pub route_epoch: u64, + pub birth_digest: [u8; 32], +} + +/// A read-only probe under the same origin permission as a command. +pub(in crate::raft) struct CellCommandProbe { + pub timeout_ms: u64, + pub route_epoch: u64, + pub birth_digest: [u8; 32], +} + +impl CellCommandProbe { + pub fn encode(&self, limit: usize) -> Result, TransportError> { + self.validate()?; + if limit < 56 { + return Err(TransportError::Limit); + } + let mut body = b"CLGCPQ01".to_vec(); + body.extend_from_slice(&self.route_epoch.to_be_bytes()); + body.extend_from_slice(&self.birth_digest); + body.extend_from_slice(&self.timeout_ms.to_be_bytes()); + Ok(body) + } + + pub fn decode(bytes: &[u8], limit: usize) -> Result { + if bytes.len() > limit { + return Err(TransportError::Limit); + } + let mut body = Reader(bytes); + if body.take(8)? != b"CLGCPQ01" { + return Err(TransportError::Protocol); + } + let probe = Self { + route_epoch: u64::from_be_bytes(body.array()?), + birth_digest: body.array()?, + timeout_ms: u64::from_be_bytes(body.array()?), + }; + if !body.0.is_empty() { + return Err(TransportError::Protocol); + } + probe.validate()?; + Ok(probe) + } + + fn validate(&self) -> Result<(), TransportError> { + if self.route_epoch == 0 || self.birth_digest == [0; 32] { + return Err(TransportError::Identity); + } + if !(1..=60_000).contains(&self.timeout_ms) { + return Err(TransportError::Protocol); + } + Ok(()) + } +} + +pub(in crate::raft) fn encode_probe_result(leader: bool) -> Vec { + let mut body = b"CLGCPS01".to_vec(); + body.push(u8::from(leader)); + body +} + +pub(in crate::raft) fn decode_probe_result(bytes: &[u8]) -> Result { + match bytes.strip_prefix(b"CLGCPS01") { + Some([0]) => Ok(false), + Some([1]) => Ok(true), + _ => Err(TransportError::Protocol), + } +} + +pub(in crate::raft) struct CellCommandValue { + pub command: CommandId, + pub durability: Durability, + pub retry_until_ms: u64, + pub value: cell::CellQueryValue, +} + +impl CellCommandCall { + pub fn encode(&self, limit: usize) -> Result, TransportError> { + let r = &self.request; + if self.route_epoch == 0 || self.birth_digest == [0; 32] { + return Err(TransportError::Identity); + } + if r.contract.kind != OperationKind::Command + || r.contract.operation.is_empty() + || r.contract.operation.len() > 128 + || r.contract.operation.chars().any(char::is_control) + || !(1..=3_600_000).contains(&self.timeout_ms) + { + return Err(TransportError::Protocol); + } + let mut body = b"CLGCCQ01".to_vec(); + body.extend_from_slice(&r.id.client.bytes()); + body.extend_from_slice(&r.id.sequence().to_be_bytes()); + body.extend_from_slice(&self.route_epoch.to_be_bytes()); + body.extend_from_slice(&self.birth_digest); + string(&mut body, &r.contract.operation)?; + body.extend_from_slice(&r.contract.digest); + body.push(durability_code(r.durability)); + body.extend_from_slice(&r.retry_until_ms.to_be_bytes()); + body.extend_from_slice(&self.inputs.now_unix_millis.to_be_bytes()); + body.extend_from_slice(&self.inputs.random); + body.extend_from_slice(&self.timeout_ms.to_be_bytes()); + let length = u32::try_from(r.input.len()).map_err(|_| TransportError::Limit)?; + if body + .len() + .checked_add(4) + .and_then(|n| n.checked_add(r.input.len())) + .is_none_or(|n| n > limit) + { + return Err(TransportError::Limit); + } + body.extend_from_slice(&length.to_be_bytes()); + body.extend_from_slice(&r.input); + Ok(body) + } + + pub fn decode(bytes: &[u8], limit: usize) -> Result { + if bytes.len() > limit { + return Err(TransportError::Limit); + } + let mut body = Reader(bytes); + if body.take(8)? != b"CLGCCQ01" { + return Err(TransportError::Protocol); + } + let id = command_id(&mut body)?; + let route_epoch = u64::from_be_bytes(body.array()?); + let birth_digest = body.array()?; + if route_epoch == 0 || birth_digest == [0; 32] { + return Err(TransportError::Identity); + } + let operation = body.string(128)?; + if operation.is_empty() || operation.chars().any(char::is_control) { + return Err(TransportError::Protocol); + } + let contract = OperationContract { + operation, + kind: OperationKind::Command, + digest: body.array()?, + }; + let durability = read_durability(&mut body)?; + let retry_until_ms = u64::from_be_bytes(body.array()?); + let inputs = CommandInputs { + now_unix_millis: u64::from_be_bytes(body.array()?), + random: body.array()?, + }; + let timeout_ms = u64::from_be_bytes(body.array()?); + if !(1..=3_600_000).contains(&timeout_ms) { + return Err(TransportError::Protocol); + } + let length = u32::from_be_bytes(body.array()?) as usize; + let input = body.take(length)?.to_vec(); + if !body.0.is_empty() { + return Err(TransportError::Protocol); + } + Ok(Self { + request: CellCommandRequest { + id, + contract, + input, + durability, + retry_until_ms, + }, + inputs, + timeout_ms, + route_epoch, + birth_digest, + }) + } +} + +pub(super) fn command_id(body: &mut Reader<'_>) -> Result { + let client = ClientId::new(body.array()?).map_err(|_| TransportError::Identity)?; + CommandId::new(client, u64::from_be_bytes(body.array()?)).map_err(|_| TransportError::Identity) +} + +pub(in crate::raft) fn encode_result( + result: &Result, + limit: usize, +) -> Result, TransportError> { + let mut body = b"CLGCCS01".to_vec(); + match result { + Ok(response) => { + // Incoming commands execute locally; remote output is not a local + // SQL image or this receiver's archive-settlement proof. + let receipt = &response + .local_receipt() + .ok_or(TransportError::Protocol)? + .receipt; + body.push(0); + body.push(durability_code(response.durability())); + body.extend_from_slice(&receipt.command().client.bytes()); + body.extend_from_slice(&receipt.command().sequence().to_be_bytes()); + body.extend_from_slice(&receipt.retry_until_ms().to_be_bytes()); + string(&mut body, response.token().as_str())?; + let length = + u32::try_from(receipt.result().len()).map_err(|_| TransportError::Limit)?; + if body + .len() + .checked_add(4) + .and_then(|n| n.checked_add(receipt.result().len())) + .is_none_or(|n| n > limit) + { + return Err(TransportError::Limit); + } + body.extend_from_slice(&length.to_be_bytes()); + body.extend_from_slice(receipt.result()); + } + Err(CommandFailure::CommandConflict) => body.push(9), + Err(CommandFailure::CommandExpired) => body.push(10), + Err(CommandFailure::SequenceGap { next }) => { + body.push(11); + body.extend_from_slice(&next.to_be_bytes()); + } + Err(CommandFailure::NotLeader { leader }) => { + body.push(12); + body.extend_from_slice(&leader.map_or(0, MemberId::get).to_be_bytes()); + } + Err(error) => body.push(failure_code(error)), + } + if body.len() > limit { + return Err(TransportError::Limit); + } + Ok(body) +} + +pub(in crate::raft) fn decode_result( + bytes: &[u8], + expected: (CommandId, &CellHistory, Durability), + schema: [u8; 32], + keys: &VersionKeys, + now: u64, + limit: usize, + served_by: MemberId, +) -> Result, TransportError> { + if bytes.len() > limit { + return Err(TransportError::Limit); + } + let mut body = Reader(bytes); + if body.take(8)? != b"CLGCCS01" { + return Err(TransportError::Protocol); + } + let result = match body.byte()? { + 0 => { + let durability = read_durability(&mut body)?; + if durability < expected.2 { + return Err(TransportError::Authentication); + } + let command = command_id(&mut body)?; + if command != expected.0 { + return Err(TransportError::Authentication); + } + let retry_until_ms = u64::from_be_bytes(body.array()?); + let token = crate::raft::version::CellVersionToken::from_encoded(&body.string(4096)?) + .map_err(|_| TransportError::Authentication)?; + let observation = keys + .verify_cell(&token, expected.1, schema, now) + .map_err(|_| TransportError::Authentication)?; + let length = u32::from_be_bytes(body.array()?) as usize; + Ok(CellCommandValue { + command, + durability, + retry_until_ms, + value: cell::CellQueryValue { + result: body.take(length)?.to_vec(), + observation, + token, + served_by, + }, + }) + } + 9 => Err(CommandFailure::CommandConflict), + 10 => Err(CommandFailure::CommandExpired), + 11 => Err(CommandFailure::SequenceGap { + next: u64::from_be_bytes(body.array()?), + }), + 12 => { + let id = u64::from_be_bytes(body.array()?); + Err(CommandFailure::NotLeader { + leader: if id == 0 { + None + } else { + Some(MemberId::new(id).map_err(|_| TransportError::Identity)?) + }, + }) + } + status => Err(decode_failure(status)?), + }; + if !body.0.is_empty() { + return Err(TransportError::Protocol); + } + Ok(result) +} diff --git a/crates/cellar/durability/raft/command_origins.rs b/crates/cellar/durability/raft/command_origins.rs new file mode 100644 index 000000000..cd157183c --- /dev/null +++ b/crates/cellar/durability/raft/command_origins.rs @@ -0,0 +1,120 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Explicit write grants remain independent from query and voting permission. +use super::*; +use crate::raft::transport_options::NamespaceCommandOrigin; + +impl RaftTransport { + /// Install trusted namespace command permission. It cannot create a group or + /// authorize a consensus message. Revisions include revocation tombstones. + pub fn grant_command_origin( + &self, + grant: NamespaceCommandOrigin, + ) -> Result<(), TransportError> { + let state = &self.handle.0; + grant.peer.validate_endpoint()?; + if grant.revision == 0 || grant.peer.node == state.runtime.log().limits().node { + return Err(TransportError::Identity); + } + let key = (grant.cluster, grant.namespace, grant.peer.node); + let mut registry = state.registry.lock().map_err(|_| TransportError::Closed)?; + state.admit(®istry)?; + registry.check_node_view(&grant.peer)?; + registry.check_physical_endpoint(&grant.peer)?; + if let Some(old) = registry.command_origins.get(&key) { + if old.revision == grant.revision && old.peer.as_ref() == Some(&grant.peer) { + return Ok(()); + } + if old.revision >= grant.revision { + return Err(TransportError::StalePeer); + } + } else if registry.command_origins.len() >= state.options.max_command_origins { + return Err(TransportError::Overloaded); + } + if registry + .groups + .values() + .chain(registry.query_groups.values()) + .flat_map(|route| &route.peers) + .any(|peer| peer.replica.node == grant.peer.node && !grant.peer.same_host(peer)) + || registry + .command_origins + .iter() + .filter(|(other, _)| **other != key) + .filter_map(|(_, old)| old.peer.as_ref()) + .any(|peer| peer.node == grant.peer.node && *peer != grant.peer) + { + return Err(TransportError::StalePeer); + } + if registry + .origins + .values() + .filter_map(|grant| grant.peer.as_ref()) + .any(|old| old.node == grant.peer.node && old != &grant.peer) + { + return Err(TransportError::StalePeer); + } + if registry + .inspection_origins + .values() + .filter_map(|origin| origin.peer.as_ref()) + .any(|peer| peer.node == grant.peer.node && peer != &grant.peer) + { + return Err(TransportError::StalePeer); + } + let starts = state.prepare_pools(&mut registry, std::slice::from_ref(&grant.peer))?; + let (stop, _) = watch::channel(false); + if let Some(old) = registry.command_origins.insert( + key, + Arc::new(queries::OriginGrant { + revision: grant.revision, + peer: Some(grant.peer), + stop, + }), + ) { + old.stop.send_replace(true); + } + registry.prune_pools(); + drop(registry); + state.start_pools(starts) + } + + pub fn revoke_command_origin( + &self, + cluster: ClusterId, + namespace: NamespaceId, + node: NodeId, + revision: u64, + ) -> Result<(), TransportError> { + if revision == 0 { + return Err(TransportError::Identity); + } + let state = &self.handle.0; + let key = (cluster, namespace, node); + let mut registry = state.registry.lock().map_err(|_| TransportError::Closed)?; + state.admit(®istry)?; + if let Some(old) = registry.command_origins.get(&key) { + if old.revision == revision && old.peer.is_none() { + return Ok(()); + } + if old.revision >= revision { + return Err(TransportError::StalePeer); + } + } else if registry.command_origins.len() >= state.options.max_command_origins { + return Err(TransportError::Overloaded); + } + let (stop, _) = watch::channel(true); + if let Some(old) = registry.command_origins.insert( + key, + Arc::new(queries::OriginGrant { + revision, + peer: None, + stop, + }), + ) { + old.stop.send_replace(true); + } + registry.prune_pools(); + Ok(()) + } +} diff --git a/crates/cellar/durability/raft/completion.rs b/crates/cellar/durability/raft/completion.rs index 1cb8fe69d..40bc0e0d2 100644 --- a/crates/cellar/durability/raft/completion.rs +++ b/crates/cellar/durability/raft/completion.rs @@ -21,10 +21,13 @@ pub enum RuntimeError { Protocol(String), NotLeader, NotReady, + DeadlineExceeded, NotFound, AlreadyAttached, Overloaded, StaleCompletion, + /// This process retains the failed group; serving requires recovery on restart. + GroupQuarantined, Closed, } @@ -144,8 +147,8 @@ pub(crate) struct RestartLeadershipCheck { /// Leader-local replication observations. These are neither apply nor quorum /// proofs; the owner rechecks actual progress when accepting a transfer. #[derive(Clone, Debug, PartialEq, Eq)] -pub struct ReplicationStatus { - pub replica: ReplicaIdentity, +pub struct ReplicationStatus { + pub replica: ReplicaIdentity, pub term: Term, pub membership_revision: LogIndex, pub committed: LogIndex, diff --git a/crates/cellar/durability/raft/effect_delivery.rs b/crates/cellar/durability/raft/effect_delivery.rs index 8f511c98e..434c76228 100644 --- a/crates/cellar/durability/raft/effect_delivery.rs +++ b/crates/cellar/durability/raft/effect_delivery.rs @@ -3,21 +3,19 @@ //! Authenticated, immutable delivery envelopes. Transport success alone cannot //! settle an intent: the destination signs its transaction result and releases //! it through the ordinary command durability gate. +use super::history::WireHistory; use crate::application::consensus::{ - protocol::WireScope, ClientId, CommandFailure, CommandId, Durability, GroupScope, - OperationContract, OperationKind, RaftCellHandle, + ClientId, CommandFailure, CommandId, Durability, OperationContract, OperationKind, + RaftCellHandle, }; use anyhow::{ensure, Result}; use base64::{engine::general_purpose::URL_SAFE_NO_PAD, Engine as _}; +use cellar_logic::cell_group::CellHistory; use ed25519_dalek::{Signature, Signer, SigningKey, VerifyingKey}; use futures_util::future::BoxFuture; use serde::{de::DeserializeOwned, Deserialize, Serialize}; use sha2::{Digest, Sha256}; -use std::{ - fmt, - sync::{Arc, Weak}, - time::Duration, -}; +use std::{fmt, sync::Arc, time::Duration}; #[cfg(all(test, not(cellar_internal_tests)))] #[path = "effect_delivery_tests.rs"] @@ -27,12 +25,12 @@ pub(crate) const MAX_REQUEST: usize = 60 * 1024; pub(crate) const MAX_RECEIPT: usize = 16 * 1024; pub(crate) const MAX_PAYLOAD: usize = 32 * 1024; pub(crate) const MAX_RESULT: usize = 4 * 1024; -const REQUEST_DOMAIN: &[u8] = b"cellar-raft-delivery-request-1"; -const RECEIPT_DOMAIN: &[u8] = b"cellar-raft-delivery-receipt-1"; +const REQUEST_DOMAIN: &[u8] = b"cellar-cell-delivery-request-1"; +const RECEIPT_DOMAIN: &[u8] = b"cellar-cell-delivery-receipt-1"; const CLIENT_PREFIX: &[u8; 4] = b"RFX1"; pub(crate) fn protocol_identity() -> [u8; 32] { - Sha256::digest(b"cellar-raft-delivery-1/definition-target-1").into() + Sha256::digest(b"cellar-cell-delivery-1/definition-target-1").into() } /// Deployment-owned material shared by compatible replicas. Debug never @@ -80,7 +78,7 @@ impl RaftEffectVerificationKey { /// compiled schema; that manifest can contain self/cyclic delivery bindings. #[derive(Clone, Debug, PartialEq, Eq)] pub struct RaftEffectTarget { - pub(crate) scope: GroupScope, + pub(crate) scope: CellHistory, pub(crate) definition: [u8; 32], pub(crate) contract: OperationContract, pub(crate) key: RaftEffectVerificationKey, @@ -89,13 +87,12 @@ pub struct RaftEffectTarget { impl RaftEffectTarget { pub fn new( - scope: GroupScope, + scope: CellHistory, definition: [u8; 32], contract: OperationContract, receipt_key: RaftEffectVerificationKey, durability: Durability, ) -> Result { - scope.validate()?; ensure!( definition != [0; 32] && contract.kind == OperationKind::Command @@ -114,7 +111,7 @@ impl RaftEffectTarget { }) } - pub fn scope(&self) -> &GroupScope { + pub fn scope(&self) -> &CellHistory { &self.scope } @@ -132,7 +129,7 @@ impl RaftEffectTarget { pub(crate) fn wire(&self) -> Target { Target { - scope: WireScope::from(&self.scope), + scope: WireHistory::from(&self.scope), definition: self.definition, operation: self.contract.operation.clone(), contract: self.contract.digest, @@ -194,11 +191,12 @@ impl RaftEffectSourceGrant { fn accepts(&self, request: &Request) -> Result { let scope = request.origin.clone().decode()?; - scope.validate()?; + Ok(scope.cluster == self.cluster && scope.namespace == self.namespace && scope .cell + .as_str() .split_once(':') .is_some_and(|(class, _)| class == self.class) && request.binding == self.binding @@ -283,7 +281,7 @@ fn verify( #[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] #[serde(deny_unknown_fields)] pub(crate) struct Target { - scope: WireScope, + scope: WireHistory, definition: [u8; 32], operation: String, contract: [u8; 32], @@ -295,7 +293,7 @@ pub(crate) struct Target { #[serde(deny_unknown_fields)] pub(crate) struct Request { marker: u8, - pub(crate) origin: WireScope, + origin: WireHistory, origin_schema: [u8; 32], pub(crate) binding: String, binding_digest: [u8; 32], @@ -330,7 +328,7 @@ impl fmt::Debug for RaftDeliveryRequest { } pub(crate) struct DeliveryOrigin<'a> { - pub scope: &'a GroupScope, + pub scope: &'a CellHistory, pub schema: [u8; 32], pub binding: &'a str, pub digest: [u8; 32], @@ -351,7 +349,7 @@ impl RaftDeliveryRequest { digest, sequence, } = origin; - origin.validate()?; + ensure!( schema != [0; 32] && valid_name(binding) @@ -362,7 +360,7 @@ impl RaftDeliveryRequest { ); let request = Request { marker: 1, - origin: WireScope::from(origin), + origin: WireHistory::from(origin), origin_schema: schema, binding: binding.into(), binding_digest: digest, @@ -446,7 +444,7 @@ fn identity(request: &Request) -> Result<[u8; 32]> { &request.binding_digest, &request.target, ))?; - Ok(Sha256::digest(message(b"cellar-raft-delivery-lane-1", &body)).into()) + Ok(Sha256::digest(message(b"cellar-cell-delivery-lane-1", &body)).into()) } fn command_id(request: &Request) -> Result { @@ -473,7 +471,7 @@ pub(crate) struct Incoming { pub(crate) fn authenticate( bytes: &[u8], grants: &[RaftEffectSourceGrant], - target: &GroupScope, + target: &CellHistory, definition: [u8; 32], contract: &OperationContract, key: RaftEffectVerificationKey, @@ -567,8 +565,7 @@ pub struct RaftNativeEffectSink { #[derive(Clone)] struct NativeEndpoint { - handle: Weak, - scope: GroupScope, + handle: crate::application::consensus::group_client::WeakCellHandle, } impl RaftNativeEffectSink { @@ -590,8 +587,7 @@ impl RaftNativeEffectSink { endpoints: endpoints .into_iter() .map(|endpoint| NativeEndpoint { - handle: Arc::downgrade(&endpoint.handle), - scope: endpoint.scope().clone(), + handle: endpoint.downgrade(), }) .collect(), timeout, @@ -615,20 +611,12 @@ impl RaftEffectSink for RaftNativeEffectSink { .ok_or(CommandFailure::Unavailable)?; let mut last = CommandFailure::Unavailable; for hint in endpoints { - let Some(handle) = hint.handle.upgrade() else { - continue; + let endpoint = match hint.handle.upgrade() { + Ok(Some(endpoint)) => endpoint, + Ok(None) | Err(_) => continue, }; - let binding = match handle - .bindings - .get(&(hint.scope.namespace, hint.scope.cell)) - { - Ok(binding) => binding, - Err(CommandFailure::Unavailable) => continue, - Err(error) => return Err(error), - }; - let endpoint = RaftCellHandle { handle, binding }; - if endpoint.scope() != request.inner.target.scope() - || endpoint.binding.class.definition_digest() != request.inner.target.definition + if endpoint.history() != request.inner.target.scope() + || endpoint.definition_digest() != request.inner.target.definition { continue; } @@ -640,11 +628,12 @@ impl RaftEffectSink for RaftNativeEffectSink { options.durability = request.inner.target.durability; match endpoint .command( - request.inner.target.contract.clone(), + &request.inner.target.contract, request.inner.bytes.clone(), options, ) .await + .map_err(|error| native_command_failure(&error)) { Ok(receipt) => { request @@ -668,3 +657,28 @@ impl RaftEffectSink for RaftNativeEffectSink { }) } } + +fn native_command_failure(error: &crate::raft::cell_groups::CellCommandError) -> CommandFailure { + use crate::raft::{ + cell_groups::CellDispatchError, completion::RuntimeError, execution::ExecutionError, + }; + if let CellDispatchError::Execution(ExecutionError::Refused(failure)) = error.cause.as_ref() { + return failure.clone(); + } + if error.ambiguous { + return CommandFailure::Ambiguous { + command: error.command, + }; + } + match error.cause.as_ref() { + CellDispatchError::Runtime(RuntimeError::NotLeader) => { + CommandFailure::NotLeader { leader: None } + } + CellDispatchError::Runtime(RuntimeError::Overloaded) => CommandFailure::Overloaded, + CellDispatchError::Registration(_) => CommandFailure::UnsupportedCapability, + CellDispatchError::Version(crate::raft::version::VersionError::WrongHistory) => { + CommandFailure::WrongIncarnation + } + _ => CommandFailure::Unavailable, + } +} diff --git a/crates/cellar/durability/raft/effect_delivery_tests.rs b/crates/cellar/durability/raft/effect_delivery_tests.rs index fab1d381d..5881f6813 100644 --- a/crates/cellar/durability/raft/effect_delivery_tests.rs +++ b/crates/cellar/durability/raft/effect_delivery_tests.rs @@ -1,20 +1,19 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. use super::*; -use crate::application::consensus::{ClusterId, GroupId, IncarnationId, NamespaceId}; +use crate::application::consensus::{ClusterId, IncarnationId, NamespaceId}; -fn scope(id: u8, class: &str) -> Result { - Ok(GroupScope { +fn scope(id: u8, class: &str) -> Result { + Ok(CellHistory { cluster: ClusterId::new([id; 16])?, namespace: NamespaceId::new([id + 1; 16])?, - cell: format!("{class}:one"), - group: GroupId::new([id + 2; 16])?, + cell: cellar_logic::cell_group::CellId::new(format!("{class}:one"))?, incarnation: IncarnationId::new([id + 3; 16])?, }) } struct Fixture { - origin: GroupScope, + origin: CellHistory, target: RaftEffectTarget, source: RaftEffectSigningKey, receiver: RaftEffectSigningKey, @@ -222,3 +221,63 @@ fn delivery_bounds_and_secret_debug() -> Result<()> { ensure!(reserved_client(f.request(1, b"first")?.command_id().client)); Ok(()) } + +#[test] +fn logical_effect_identity_survives_group_reassignment() -> Result<()> { + use cellar_logic::consensus::{GroupId, GroupScope}; + let mut physical = GroupScope { + cluster: ClusterId::new([1; 16])?, + namespace: NamespaceId::new([2; 16])?, + cell: "Source:one".into(), + group: GroupId::new([3; 16])?, + incarnation: IncarnationId::new([4; 16])?, + }; + let before = CellHistory::try_from(&physical)?; + physical.group = GroupId::new([8; 16])?; + let after = CellHistory::try_from(&physical)?; + let mut f = Fixture::new()?; + f.origin = before.clone(); + let first = f.request(1, b"same")?; + f.origin = after.clone(); + assert_eq!(first.encoded(), f.request(1, b"same")?.encoded()); + assert_eq!( + super::super::system_client(&before)?, + super::super::system_client(&after)? + ); + f.origin.incarnation = IncarnationId::new([7; 16])?; + assert_ne!(first.command_id(), f.request(1, b"same")?.command_id()); + assert_ne!( + super::super::system_client(&before)?, + super::super::system_client(&f.origin)? + ); + f.origin = before; + f.origin.cell = cellar_logic::cell_group::CellId::new("Source:two")?; + assert_ne!(first.command_id(), f.request(1, b"same")?.command_id()); + Ok(()) +} + +#[test] +fn cell_effect_wire_refuses_group_scope_and_old_signature_domain() -> Result<()> { + let f = Fixture::new()?; + let request = f.request(1, b"same")?; + let signed: Signed = parse(request.encoded(), MAX_REQUEST)?; + let old_domain = sign( + signed.body.clone(), + &f.source, + b"cellar-raft-delivery-request-1", + MAX_REQUEST, + )?; + assert!(f.incoming(&old_domain).is_err()); + for field in ["origin", "target"] { + let mut body = serde_json::to_value(&signed.body)?; + let scope = if field == "origin" { + &mut body["origin"] + } else { + &mut body["target"]["scope"] + }; + scope["group"] = hex::encode([3; 16]).into(); + let bytes = sign(body, &f.source, REQUEST_DOMAIN, MAX_REQUEST)?; + assert!(f.incoming(&bytes).is_err()); + } + Ok(()) +} diff --git a/crates/cellar/durability/raft/effect_history.rs b/crates/cellar/durability/raft/effect_history.rs new file mode 100644 index 000000000..a78dabbad --- /dev/null +++ b/crates/cellar/durability/raft/effect_history.rs @@ -0,0 +1,53 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Canonical logical history for signed effects; physical placement is absent. +use anyhow::{ensure, Result}; +use cellar_logic::{ + cell_group::{CellHistory, CellId}, + consensus::{ClusterId, IncarnationId, NamespaceId}, +}; +use serde::{Deserialize, Serialize}; + +#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct WireHistory { + cluster: String, + namespace: String, + cell: String, + incarnation: String, +} + +impl From<&CellHistory> for WireHistory { + fn from(history: &CellHistory) -> Self { + Self { + cluster: hex::encode(history.cluster.bytes()), + namespace: hex::encode(history.namespace.bytes()), + cell: history.cell.as_str().to_owned(), + incarnation: hex::encode(history.incarnation.bytes()), + } + } +} + +impl WireHistory { + pub(super) fn decode(self) -> Result { + Ok(CellHistory { + cluster: ClusterId::new(identity(&self.cluster)?)?, + namespace: NamespaceId::new(identity(&self.namespace)?)?, + cell: CellId::new(self.cell)?, + incarnation: IncarnationId::new(identity(&self.incarnation)?)?, + }) + } +} + +fn identity(value: &str) -> Result<[u8; 16]> { + ensure!( + value.len() == 32 + && value + .bytes() + .all(|b| b.is_ascii_digit() || (b'a'..=b'f').contains(&b)), + "noncanonical effect identity" + ); + let mut bytes = [0; 16]; + hex::decode_to_slice(value, &mut bytes)?; + Ok(bytes) +} diff --git a/crates/cellar/durability/raft/effect_outbox.rs b/crates/cellar/durability/raft/effect_outbox.rs index 0d44ef8e7..195781e34 100644 --- a/crates/cellar/durability/raft/effect_outbox.rs +++ b/crates/cellar/durability/raft/effect_outbox.rs @@ -16,6 +16,7 @@ use std::{collections::BTreeMap, sync::Arc}; pub(crate) const CLAIM: &str = "__sys_raft_outbox_claim"; pub(crate) const COMPLETE: &str = "__sys_raft_outbox_complete"; +pub(crate) const MAX_LANES: usize = 32; #[derive(Clone, Copy, Debug)] pub struct RaftOutboxLimits { @@ -80,7 +81,7 @@ impl Default for RaftOutboxLimits { impl RaftOutboxLimits { fn validate(self) -> Result<()> { ensure!( - (1..=32).contains(&self.max_lanes) + (1..=MAX_LANES).contains(&self.max_lanes) && (1..=128).contains(&self.max_pending) && (1..=delivery::MAX_PAYLOAD).contains(&self.max_payload_bytes) && (delivery::MAX_REQUEST..=16 * 1024 * 1024).contains(&self.max_total_bytes) @@ -536,7 +537,7 @@ pub(crate) struct ClaimedDelivery { pub(crate) fn claimed( outbox: &RaftOutbox, - scope: &crate::application::consensus::GroupScope, + scope: &cellar_logic::cell_group::CellHistory, schema: [u8; 32], selected: &Selected, bytes: &[u8], diff --git a/crates/cellar/durability/raft/effect_scan.rs b/crates/cellar/durability/raft/effect_scan.rs index b584b1f45..128872ce6 100644 --- a/crates/cellar/durability/raft/effect_scan.rs +++ b/crates/cellar/durability/raft/effect_scan.rs @@ -12,6 +12,10 @@ use cellar_logic::consensus::{CommandFailure, CommandId, Durability, LogPosition use rusqlite::OptionalExtension; use std::sync::Arc; +#[cfg(all(test, not(cellar_internal_tests)))] +#[path = "effect_scan_tests.rs"] +mod tests; + pub(crate) enum DueEffect { Alarm(alarm::DueAlarm), Delivery { @@ -63,7 +67,7 @@ pub(crate) fn scan( )?; scan_connection( database.connection(), - image.scope(), + &cellar_logic::cell_group::CellHistory::try_from(image.scope())?, image.position(), class, now, @@ -74,7 +78,7 @@ pub(crate) fn scan( pub(crate) fn scan_connection( connection: &rusqlite::Connection, - scope: &cellar_logic::consensus::GroupScope, + scope: &cellar_logic::cell_group::CellHistory, position: LogPosition, class: &RaftClass, now: u64, @@ -129,3 +133,31 @@ pub(crate) fn scan_connection( _reservation: reservation, }) } + +/// Alternate adapter preference using the last committed private operation. +/// Placement, process restarts and the number of other cells do not affect it. +pub(crate) fn scan_connection_fair( + connection: &rusqlite::Connection, + scope: &cellar_logic::cell_group::CellHistory, + position: LogPosition, + class: &RaftClass, + now: u64, + reservation: Permit, +) -> Result { + let client = super::system_client(scope)?; + let previous: Option = connection.query_row( + "SELECT CASE WHEN typeof(r.operation)='text' AND length(CAST(r.operation AS BLOB)) BETWEEN 1 AND 128 THEN r.operation ELSE NULL END FROM __sys_raft_clients c JOIN __sys_raft_receipts r ON r.client_id=c.client_id AND r.sequence=c.last_sequence WHERE c.client_id=?1", + [client.bytes().as_slice()], |row| row.get(0), + ).optional()?; + let prefer_delivery = + previous.is_some_and(|operation| operation.starts_with("__sys_raft_alarm_")); + scan_connection( + connection, + scope, + position, + class, + now, + prefer_delivery, + reservation, + ) +} diff --git a/crates/cellar/durability/raft/effect_scan_tests.rs b/crates/cellar/durability/raft/effect_scan_tests.rs new file mode 100644 index 000000000..b68ab5428 --- /dev/null +++ b/crates/cellar/durability/raft/effect_scan_tests.rs @@ -0,0 +1,71 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use crate::raft::budget::Budget; +use cellar_logic::{cell_group::*, consensus::*}; +use rusqlite::types::Value; + +fn scan_operation(operation: Value) -> Result<()> { + let database = rusqlite::Connection::open_in_memory()?; + database.execute_batch( + "CREATE TABLE __sys_raft_clients(client_id BLOB, last_sequence BLOB); + CREATE TABLE __sys_raft_receipts(client_id BLOB, sequence BLOB, operation);", + )?; + let history = CellHistory { + cluster: ClusterId::new([1; 16])?, + namespace: NamespaceId::new([2; 16])?, + cell: CellId::new("Counter:one")?, + incarnation: IncarnationId::new([3; 16])?, + }; + let client = super::super::system_client(&history)?; + let sequence = 1_u64.to_be_bytes(); + database.execute( + "INSERT INTO __sys_raft_clients VALUES(?1, ?2)", + rusqlite::params![client.bytes().as_slice(), sequence.as_slice()], + )?; + database.execute( + "INSERT INTO __sys_raft_receipts VALUES(?1, ?2, ?3)", + rusqlite::params![client.bytes().as_slice(), sequence.as_slice(), operation], + )?; + let memory = Budget::new(64 * 1024, 1); + let result = scan_connection_fair( + &database, + &history, + LogPosition { + index: LogIndex::new(1), + term: Term::new(1), + }, + &RaftClass::new("Counter", 1, [4; 32])?, + 1, + memory.acquire(64 * 1024).unwrap(), + ); + match result { + Ok(scan) => { + assert_eq!(scan.next, CommandId::new(client, 2)?); + assert!(scan.due.is_none()); + drop(scan); + assert_eq!(memory.usage(), (0, 0)); + Ok(()) + } + Err(error) => { + assert_eq!(memory.usage(), (0, 0)); + Err(error) + } + } +} + +#[test] +fn previous_private_operation_is_bounded_before_allocation() -> Result<()> { + scan_operation(Value::Text("__sys_raft_alarm_Wake".into()))?; + scan_operation(Value::Text("x".repeat(128)))?; + for invalid in [ + Value::Null, + Value::Integer(1), + Value::Blob(vec![1; 128]), + Value::Text(String::new()), + Value::Text("x".repeat(129)), + ] { + assert!(scan_operation(invalid).is_err()); + } + Ok(()) +} diff --git a/crates/cellar/durability/raft/effects.rs b/crates/cellar/durability/raft/effects.rs index 24a25b215..59c2aecc7 100644 --- a/crates/cellar/durability/raft/effects.rs +++ b/crates/cellar/durability/raft/effects.rs @@ -11,6 +11,8 @@ pub(crate) mod alarm; mod blob; #[path = "effect_delivery.rs"] pub(crate) mod delivery; +#[path = "effect_history.rs"] +mod history; #[path = "effect_inbox.rs"] mod inbox; #[path = "effect_outbox.rs"] @@ -32,21 +34,20 @@ pub use options::RaftEffectOptions; /// One stable private command client per history, independent of node boot and /// leadership. Public commands cannot consume it in a class with system work. pub(crate) fn system_client( - scope: &cellar_logic::consensus::GroupScope, + scope: &cellar_logic::cell_group::CellHistory, ) -> Result { use sha2::{Digest, Sha256}; let mut hash = Sha256::new(); - hash.update(b"cellar-raft-system-client-1"); + hash.update(b"cellar-cell-system-client-1"); for identity in [ scope.cluster.bytes(), scope.namespace.bytes(), - scope.group.bytes(), scope.incarnation.bytes(), ] { hash.update(identity); } - hash.update((scope.cell.len() as u64).to_be_bytes()); - hash.update(scope.cell.as_bytes()); + hash.update((scope.cell.as_str().len() as u64).to_be_bytes()); + hash.update(scope.cell.as_str().as_bytes()); let digest: [u8; 32] = hash.finalize().into(); let mut id = [0; 16]; id.copy_from_slice(&digest[..16]); diff --git a/crates/cellar/durability/raft/enrollment_cell_wire.rs b/crates/cellar/durability/raft/enrollment_cell_wire.rs new file mode 100644 index 000000000..ac823bf75 --- /dev/null +++ b/crates/cellar/durability/raft/enrollment_cell_wire.rs @@ -0,0 +1,162 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Creation payloads use a distinct authenticated RPC purpose and exact receipt. +use super::*; +use crate::raft::cell_groups::{CellEnrollmentRequest, EnrollmentRecord}; +use crate::storage::CommandInputs; +use cellar_logic::{ + cell_group::CellHistory, + consensus::{Durability, ProposalId}, +}; +use cellar_ltx::generation::ImageId; + +pub(in crate::raft) struct CellEnrollmentCall { + pub request: CellEnrollmentRequest, + pub inputs: CommandInputs, + pub preview: ImageId, + pub timeout_ms: u64, + pub route_epoch: u64, + pub birth_digest: [u8; 32], +} + +#[derive(Debug, PartialEq, Eq)] +pub(in crate::raft) struct CellEnrollmentValue { + pub record: EnrollmentRecord, + pub durability: Durability, +} + +impl CellEnrollmentCall { + pub fn encode(&self, limit: usize) -> Result, TransportError> { + if self.route_epoch == 0 || self.birth_digest == [0; 32] { + return Err(TransportError::Identity); + } + if !(1..=3_600_000).contains(&self.timeout_ms) { + return Err(TransportError::Protocol); + } + let mut body = b"CLGCEQ01".to_vec(); + body.extend_from_slice(&self.request.creation.bytes()); + body.extend_from_slice(&self.route_epoch.to_be_bytes()); + body.extend_from_slice(&self.birth_digest); + body.push(durability_code(self.request.durability)); + body.extend_from_slice(&self.inputs.now_unix_millis.to_be_bytes()); + body.extend_from_slice(&self.inputs.random); + body.extend_from_slice(&self.preview.bytes()); + body.extend_from_slice(&self.timeout_ms.to_be_bytes()); + if body.len() > limit { + return Err(TransportError::Limit); + } + Ok(body) + } + + pub fn decode(bytes: &[u8], limit: usize) -> Result { + if bytes.len() > limit { + return Err(TransportError::Limit); + } + let mut body = Reader(bytes); + if body.take(8)? != b"CLGCEQ01" { + return Err(TransportError::Protocol); + } + let creation = ProposalId::new(body.array()?).map_err(|_| TransportError::Identity)?; + let route_epoch = u64::from_be_bytes(body.array()?); + let birth_digest = body.array()?; + let durability = read_durability(&mut body)?; + let inputs = CommandInputs { + now_unix_millis: u64::from_be_bytes(body.array()?), + random: body.array()?, + }; + let preview = ImageId::new(body.array()?).map_err(|_| TransportError::Identity)?; + let call = Self { + request: CellEnrollmentRequest { + creation, + durability, + }, + inputs, + preview, + route_epoch, + birth_digest, + timeout_ms: u64::from_be_bytes(body.array()?), + }; + if !body.0.is_empty() { + return Err(TransportError::Protocol); + } + // Fixed-size encoding validates the same domain constraints in both directions. + call.encode(limit)?; + Ok(call) + } +} + +pub(in crate::raft) fn encode_result( + result: &Result<(&EnrollmentRecord, Durability), CommandFailure>, + limit: usize, +) -> Result, TransportError> { + let mut body = b"CLGCES01".to_vec(); + match result { + Ok((record, durability)) => { + let record = record.encode().map_err(|_| TransportError::Protocol)?; + body.push(0); + body.push(durability_code(*durability)); + body.extend_from_slice(&(record.len() as u32).to_be_bytes()); + body.extend_from_slice(&record); + } + Err(CommandFailure::CommandConflict) => body.push(9), + Err(CommandFailure::NotLeader { leader }) => { + body.push(12); + body.extend_from_slice(&leader.map_or(0, MemberId::get).to_be_bytes()); + } + Err(failure) => body.push(failure_code(failure)), + } + if body.len() > limit { + return Err(TransportError::Limit); + } + Ok(body) +} + +/// Only the authenticated exchange may call this decoder. A record alone is +/// creation data, never a signed version token or proof of the current cell image. +pub(in crate::raft) fn decode_result( + bytes: &[u8], + expected: (ProposalId, Durability), + cell: &CellHistory, + schema: [u8; 32], + limit: usize, +) -> Result, TransportError> { + if bytes.len() > limit { + return Err(TransportError::Limit); + } + let mut body = Reader(bytes); + if body.take(8)? != b"CLGCES01" { + return Err(TransportError::Protocol); + } + let result = match body.byte()? { + 0 => { + let durability = read_durability(&mut body)?; + if durability < expected.1 { + return Err(TransportError::Authentication); + } + let length = u32::from_be_bytes(body.array()?) as usize; + let record = EnrollmentRecord::decode(body.take(length)?) + .map_err(|_| TransportError::Protocol)?; + if record.creation() != expected.0 || record.cell() != cell || record.schema() != schema + { + return Err(TransportError::Authentication); + } + Ok(CellEnrollmentValue { record, durability }) + } + 9 => Err(CommandFailure::CommandConflict), + 12 => { + let member = u64::from_be_bytes(body.array()?); + Err(CommandFailure::NotLeader { + leader: if member == 0 { + None + } else { + Some(MemberId::new(member).map_err(|_| TransportError::Identity)?) + }, + }) + } + status => Err(decode_failure(status)?), + }; + if !body.0.is_empty() { + return Err(TransportError::Protocol); + } + Ok(result) +} diff --git a/crates/cellar/durability/raft/execution/cell.rs b/crates/cellar/durability/raft/execution/cell.rs new file mode 100644 index 000000000..c6f4e6b8d --- /dev/null +++ b/crates/cellar/durability/raft/execution/cell.rs @@ -0,0 +1,498 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! One cell's private SQL/receipt producer under a physical group's authority. +//! The node's SQL pool must own this synchronous object and join its callbacks. + +use super::*; +use crate::application::consensus::group_protocol::{ + CellGroupEntry, GroupCommand, GroupEntryBinding, GroupEntryLimits, GroupLtx, GroupMutation, +}; +use crate::raft::cell_groups::{ + cell_image_history_digest, CellExecutionAuthority, CellReconciliationAuthority, +}; +use crate::raft::completion::RuntimeError; +use cellar_logic::cell_group::CellVersion; + +/// A receipt read from the exact committed cell image. This is not S3 coverage. +pub struct CellCommittedReceipt { + pub receipt: StoredReceipt, + pub version: CellVersion, + pub position: LogPosition, + /// Decision for the selected version, which may cover a prior receipt. + pub witness: cellar_logic::cell_group::CommitWitness, +} + +impl CellCommittedReceipt { + pub fn observation(&self) -> crate::raft::version::CellVersionObservation { + crate::raft::version::CellVersionObservation { + version: self.version.clone(), + witness: self.witness, + } + } +} + +pub enum CellStageOutcome { + Prepared(CellPreparedCommand), + Duplicate(Box), +} + +pub enum CellReconciliation { + /// The original request's receipt, from the inspected committed image. + /// Its position identifies that image, not necessarily the original entry. + Committed(Box), + /// A newer-term applied barrier excludes the old proposal and the original + /// request has no receipt. A retry must still pass fresh command admission. + NotCommitted, + /// A replicated receipt floor or conflicting identity prevents a retry. + /// Expiration does not establish whether the original command committed. + Refused(CommandFailure), +} + +struct PendingCell { + prepared: CellPreparedCommand, + request: CommandRequest, + result: CellVersion, + _preview: Preview, + submitted: bool, +} + +pub struct CellPrivateExecutor { + options: Options, + authority: CellExecutionAuthority, + working: Option, + pending: Option, + poisoned: bool, + receipt_cursor: Option, + _lock: PrivateLock, +} + +impl CellPrivateExecutor { + pub fn open( + options: PrivateOptions, + authority: CellExecutionAuthority, + ) -> Result { + Self::open_image(options.into(), authority) + } + + pub fn open_rooted( + options: RootPrivateOptions, + authority: CellExecutionAuthority, + ) -> Result { + Self::open_image(options.into(), authority) + } + + fn open_image( + options: Options, + authority: CellExecutionAuthority, + ) -> Result { + options.limits.validate()?; + options.images.validate_workspace(options.limits)?; + authority.check().map_err(authority_failure)?; + if authority.route().schema_digest != options.application_digest { + return Err(ExecutionError::Invalid("cell execution schema")); + } + let filesystem = options.host.filesystem(); + options.prepare_directory()?; + let lock = PrivateLock { + _lock: filesystem + .lock_file(&options.directory.join("working.lock")) + .map_err(storage)?, + _workspace: options.images.workspace().cloned(), + }; + let working = match WorkingImage::open_cell( + &options, + authority.image(), + cell_image_history_digest(&authority.cell().version.history), + ) { + Ok(working) => working, + Err(error) => { + options.discard_cell_scratch().map_err(storage)?; + return Err(storage(error)); + } + }; + if let Err(error) = authority.check() { + working.close_cell(&options).map_err(storage)?; + return Err(authority_failure(error)); + } + Ok(Self { + options, + authority, + working: Some(working), + pending: None, + poisoned: false, + receipt_cursor: None, + _lock: lock, + }) + } + + /// Invoke a registered application operation with this cell's logical + /// identity. Validation precedes receipt lookup, including duplicate retries. + pub fn stage_registered( + &mut self, + attempt: CommandAttempt, + class: &RaftClass, + input: &[u8], + allow_new: bool, + ) -> Result { + self.stage_registered_kind(attempt, class, input, allow_new, false) + } + + pub(crate) fn stage_system_registered( + &mut self, + attempt: CommandAttempt, + class: &RaftClass, + input: &[u8], + allow_new: bool, + ) -> Result { + self.stage_registered_kind(attempt, class, input, allow_new, true) + } + + fn stage_registered_kind( + &mut self, + attempt: CommandAttempt, + class: &RaftClass, + input: &[u8], + allow_new: bool, + system: bool, + ) -> Result { + let history = self.authority.cell().version.history.clone(); + let contract = super::registered::validate( + self.options.limits, + self.options.application_digest, + &history, + &attempt.request, + class, + input, + system, + )?; + let command = attempt.request.id; + let durability = attempt.request.durability; + self.stage(attempt, allow_new, |transaction| { + if system { + class.invoke_system_command( + &contract, + &history, + command, + durability, + transaction, + input, + ) + } else { + class.invoke_command(&contract, &history, command, durability, transaction, input) + } + }) + } + + /// Stage on an owning blocking worker. Callback output is retained only in + /// the private receipt transaction; it cannot escape as a committed result. + /// The host serializes unresolved producers by CellHistory, independently + /// from the group's consensus owner and other cells' SQL work. + pub fn stage( + &mut self, + attempt: CommandAttempt, + allow_new: bool, + callback: impl FnOnce(&CommandTransaction<'_>) -> anyhow::Result>, + ) -> Result { + if self.pending.is_some() { + return Err(ExecutionError::Busy); + } + if self.poisoned { + return Err(ExecutionError::RebuildRequired); + } + self.authority.check().map_err(authority_failure)?; + let working = self + .working + .as_ref() + .ok_or(ExecutionError::RebuildRequired)?; + let maintenance = match command::prepare( + working.database().framework(), + self.options.limits, + &mut self.receipt_cursor, + &attempt, + allow_new, + )? { + command::Admission::Duplicate => { + return self + .read_receipt(&self.authority, &attempt.request) + .map(|receipt| CellStageOutcome::Duplicate(Box::new(receipt))) + } + command::Admission::New(maintenance) => maintenance, + }; + let log_reservation = self + .authority + .reserve_log(self.options.limits.envelope) + .map_err(authority_failure)?; + let sequence = self + .authority + .cell() + .version + .sequence + .checked_next() + .map_err(|_| ExecutionError::Invalid("cell mutation sequence exhausted"))?; + self.poisoned = true; + let working = self + .working + .as_mut() + .ok_or(ExecutionError::RebuildRequired)?; + working.begin_mutation(); + command::apply( + working.database(), + self.options.limits, + &attempt, + maintenance, + callback, + )?; + let update = working + .capture_cell(&self.options, self.authority.image()) + .map_err(private_failure)?; + if cellar_logic::cell_group::CellImage::from(update.base) + != self.authority.cell().version.image + { + return Err(ExecutionError::Invalid("captured cell base")); + } + let route = self.authority.route(); + let entry = CellGroupEntry { + binding: GroupEntryBinding { + group: route.history, + route_epoch: route.epoch, + birth_digest: route.birth_digest, + schema_digest: route.schema_digest, + proposal: attempt.proposal, + }, + command: GroupCommand::Mutation(GroupMutation { + cell: self.authority.cell().version.history.clone(), + sequence, + base: update.base, + result: update.result, + files: update + .files + .iter() + .map(|file| GroupLtx { + min_txid: file.min_txid, + max_txid: file.max_txid, + page_count: file.page_count, + decoded_bytes: file.decoded_bytes, + bytes: &file.bytes, + }) + .collect(), + }), + }; + let bytes = entry + .encode( + route, + GroupEntryLimits { + envelope: self.options.limits.envelope, + ..Default::default() + }, + ) + .map_err(ExecutionError::Protocol)?; + let preview = self + .options + .images + .preview_cell( + &self.options, + self.authority.image(), + attempt.preview, + &update, + cell_image_history_digest(&self.authority.cell().version.history), + ) + .map_err(storage)?; + self.authority.check().map_err(authority_failure)?; + log_reservation + .encoded(bytes.len()) + .map_err(|error| authority_failure(error.into()))?; + let prepared = CellPreparedCommand::new( + bytes, + attempt.proposal, + self.authority.term(), + route.history, + self.authority.engine_owner().clone(), + ) + .with_log_reservation(log_reservation); + self.pending = Some(PendingCell { + prepared: prepared.clone(), + request: attempt.request, + result: CellVersion { + history: self.authority.cell().version.history.clone(), + sequence, + image: update.result.into(), + }, + _preview: preview, + submitted: false, + }); + self.poisoned = false; + Ok(CellStageOutcome::Prepared(prepared)) + } + + /// Set before awaiting proposal submission. A dropped response never frees + /// this producer for another callback or establishes that its effect failed. + pub fn mark_submitted(&mut self, proposal: ProposalId) -> Result<(), ExecutionError> { + self.authority.check().map_err(authority_failure)?; + let pending = self + .pending + .as_mut() + .ok_or(ExecutionError::Invalid("no cell proposal"))?; + if pending.submitted || pending.prepared.proposal() != proposal { + return Err(ExecutionError::Invalid("cell proposal submission")); + } + pending.submitted = true; + Ok(()) + } + + /// Settle only the exact committed entry and cell result, using a fresh + /// current-term authority from the actual owner. Another cell's progress, + /// no-op or selected image cannot settle this receipt. Compacted entries + /// require later receipt reconciliation, not an inferred proposal proof. + pub fn finish( + &mut self, + committed: CellExecutionAuthority, + ) -> Result { + let pending = self + .pending + .as_ref() + .ok_or(ExecutionError::Invalid("no cell proposal"))?; + if !pending.submitted + || committed.route() != self.authority.route() + || committed.cell().version != pending.result + { + return Err(ExecutionError::Invalid("committed cell proposal/image")); + } + committed + .check_committed_entry(pending.prepared.bytes()) + .map_err(authority_failure)?; + let receipt = self.read_receipt(&committed, &pending.request)?; + // Only exact committed/reconciliation authority retires a private + // preview. Pending retains its live pin through physical settlement. + self.options + .images + .retire(&pending._preview) + .map_err(storage)?; + pending + .prepared + .release_log_reservation() + .map_err(authority_failure)?; + self.pending = None; + self.poisoned = true; + Ok(receipt) + } + + /// Resolve the original request without requiring the original log entry. + /// Missing receipts cannot retire an ambiguous same-term proposal. The + /// caller must close this disposable producer after successful resolution. + /// A selected image that advances during inspection leaves the original + /// producer uncertain; retry with a fresh reconciliation observation. + pub fn reconcile( + &mut self, + proposal: ProposalId, + authority: CellReconciliationAuthority, + ) -> Result { + let pending = self.pending.as_ref().ok_or(ExecutionError::Uncertain)?; + if pending.prepared.proposal() != proposal + || !self.authority.accepts_reconciliation(&authority) + { + return Err(ExecutionError::Invalid("cell reconciliation identity")); + } + let check = || { + authority.check().map_err(|error| match error { + RuntimeError::NotReady => ExecutionError::Uncertain, + error => authority_failure(error), + }) + }; + check()?; + // Staging already established absence in this exact immutable base. + // In particular an enrollment image need not yet have receipt tables. + let receipt = if authority.cell().version == self.authority.cell().version { + Ok(None) + } else { + let database = self + .options + .images + .read_cell(&self.options, authority.image()) + .map_err(storage)?; + receipts::lookup_identity( + database.connection(), + &pending.request, + &self.options.limits, + ) + }; + let resolution = match receipt { + Ok(Some(receipt)) => CellReconciliation::Committed(Box::new(CellCommittedReceipt { + receipt, + version: authority.cell().version.clone(), + position: authority.cell().last_mutation, + witness: authority.cell().witness, + })), + Ok(None) if authority.term() > self.authority.term() => { + CellReconciliation::NotCommitted + } + Err(ExecutionError::Refused( + failure @ (CommandFailure::CommandExpired | CommandFailure::CommandConflict), + )) if authority.term() > self.authority.term() => CellReconciliation::Refused(failure), + Err(error) => return Err(error), + Ok(None) => return Err(ExecutionError::Uncertain), + }; + check()?; + // Only exact committed/reconciliation authority retires a private + // preview. Pending retains its live pin through physical settlement. + self.options + .images + .retire(&pending._preview) + .map_err(storage)?; + pending + .prepared + .release_log_reservation() + .map_err(authority_failure)?; + self.pending = None; + self.poisoned = true; + Ok(resolution) + } + + fn read_receipt( + &self, + authority: &CellExecutionAuthority, + request: &CommandRequest, + ) -> Result { + authority.check().map_err(authority_failure)?; + let database = self + .options + .images + .read_cell(&self.options, authority.image()) + .map_err(storage)?; + let receipt = receipts::preflight(database.connection(), request, &self.options.limits)? + .ok_or(ExecutionError::Invalid("committed cell receipt missing"))?; + authority.check().map_err(authority_failure)?; + Ok(CellCommittedReceipt { + receipt, + version: authority.cell().version.clone(), + position: authority.cell().last_mutation, + witness: authority.cell().witness, + }) + } + + /// Join and close the disposable writer. It never changes the outcome of + /// an already submitted entry; recovery reads the committed cell image. + pub fn close(mut self) -> Result<(), ExecutionError> { + let store = self.options.images.root_store().cloned(); + if let Some(working) = self.working.take() { + working.close_cell(&self.options).map_err(storage)?; + } + drop(self.options); + drop(self._lock); + if let Some(store) = store { + if store.usage().map_err(storage)?.quarantined { + return Err(storage(cellar_ltx::internal::StoreError::Quarantined)); + } + } + Ok(()) + } +} + +pub(crate) fn authority_failure(error: RuntimeError) -> ExecutionError { + match error { + RuntimeError::Overloaded => ExecutionError::Refused(CommandFailure::Overloaded), + RuntimeError::NotReady | RuntimeError::NotLeader | RuntimeError::Closed => { + ExecutionError::Refused(CommandFailure::Unavailable) + } + other => storage(other), + } +} diff --git a/crates/cellar/durability/raft/execution/command.rs b/crates/cellar/durability/raft/execution/command.rs new file mode 100644 index 000000000..fcadeb28e --- /dev/null +++ b/crates/cellar/durability/raft/execution/command.rs @@ -0,0 +1,100 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Receipt admission and one atomic business/receipt transaction for both owners. + +use super::*; + +pub(super) enum Admission { + Duplicate, + New(Option), +} + +pub(super) fn prepare( + connection: &rusqlite::Connection, + limits: ExecutionLimits, + cursor: &mut Option, + attempt: &CommandAttempt, + allow_new: bool, +) -> Result { + if attempt.request.operation.is_empty() || attempt.request.operation.len() > 128 { + return Err(ExecutionError::Invalid("command operation")); + } + if receipts::preflight_identity(connection, &attempt.request, &limits)?.is_some() { + return Ok(Admission::Duplicate); + } + let count = receipts::retained_count(connection)?; + let ceiling = limits.max_retained_receipts; + let maintenance = match limits.receipt_maintenance { + Some(limits) if count >= ceiling - ceiling / 4 => { + let plan = crate::raft::receipt_maintenance::plan( + connection, + attempt.inputs.now_unix_millis, + limits, + *cursor, + )?; + *cursor = plan.next_cursor; + Some(plan) + } + _ => None, + }; + let removed = maintenance.as_ref().map_or(0, |plan| plan.rows_removed); + if count + .checked_sub(removed) + .is_none_or(|count| count >= ceiling) + { + return Err(ExecutionError::Refused(CommandFailure::Overloaded)); + } + if attempt.request.retry_until_ms < attempt.inputs.now_unix_millis + || attempt + .request + .retry_until_ms + .checked_sub(attempt.inputs.now_unix_millis) + .is_none_or(|window| window > limits.max_retry_window_ms) + { + return Err(ExecutionError::Invalid("command retry window")); + } + // Duplicate receipt lookup above remains available under archive + // pressure. A known refusal has not mutated SQL or capture state and + // must not force the next request to reconstruct the private image. + if !allow_new { + return Err(ExecutionError::Refused(CommandFailure::Overloaded)); + } + Ok(Admission::New(maintenance)) +} + +pub(super) fn apply( + database: &crate::storage::CommandDatabase, + limits: ExecutionLimits, + attempt: &CommandAttempt, + maintenance: Option, + command: impl FnOnce(&CommandTransaction<'_>) -> anyhow::Result>, +) -> Result<(), ExecutionError> { + database + .transaction(attempt.inputs, |connection, transaction| { + if let Some(maintenance) = maintenance { + maintenance + .apply(connection, attempt.inputs.now_unix_millis) + .map_err(anyhow::Error::new)?; + } + let result = command(transaction)?; + if result.len() > limits.max_result_bytes { + return Err(anyhow::Error::new(CommandFailure::Overloaded)); + } + receipts::record(connection, &attempt.request, result)?; + let limits = &limits; + if limits.receipt_reserve_bytes != 0 { + let usage = + crate::storage::CommandDatabase::capacity(connection, limits.max_image_bytes)?; + limits + .check_capacity( + usage.application_bytes, + usage.receipt_bytes, + usage.structural_reserve_bytes, + ) + .map_err(anyhow::Error::new)?; + } + Ok(()) + }) + .map_err(private_failure)?; + Ok(()) +} diff --git a/crates/cellar/durability/raft/execution/enrollment.rs b/crates/cellar/durability/raft/execution/enrollment.rs new file mode 100644 index 000000000..9f13844ed --- /dev/null +++ b/crates/cellar/durability/raft/execution/enrollment.rs @@ -0,0 +1,265 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Disposable SQLite initialization under a physical group's leader fence. + +use super::cell::authority_failure; +use super::*; +use crate::application::consensus::group_protocol::{ + CellGroupEntry, GroupCommand, GroupEntryBinding, GroupEntryLimits, GroupLtx, +}; +use crate::raft::cell_groups::{ + cell_image_history_digest, CellEnrollmentAuthority, CellEnrollmentState, EnrollmentRecord, +}; + +pub struct CellEnrollmentAttempt { + pub creation: ProposalId, + pub preview: ImageId, + pub inputs: CommandInputs, +} + +pub enum CellEnrollmentResolution { + Committed(Box), + NotCommitted, + Conflict, +} + +struct Pending { + log_reservation: std::sync::Arc, + record: EnrollmentRecord, + _preview: Preview, + submitted: bool, +} + +pub struct CellPrivateEnrollment { + options: Options, + authority: CellEnrollmentAuthority, + working: Option, + pending: Option, + staged: bool, + lock: PrivateLock, +} + +impl CellPrivateEnrollment { + pub fn open( + options: PrivateOptions, + authority: CellEnrollmentAuthority, + ) -> Result { + Self::open_image(options.into(), authority) + } + + pub fn open_rooted( + options: RootPrivateOptions, + authority: CellEnrollmentAuthority, + ) -> Result { + Self::open_image(options.into(), authority) + } + + fn open_image( + options: Options, + authority: CellEnrollmentAuthority, + ) -> Result { + options.limits.validate()?; + options.images.validate_workspace(options.limits)?; + authority.check().map_err(authority_failure)?; + if authority.state().record().is_some() + || authority.state().route().schema_digest != options.application_digest + { + return Err(ExecutionError::Invalid("initialization binding")); + } + let filesystem = options.host.filesystem(); + options.prepare_directory()?; + let lock = PrivateLock { + _lock: filesystem + .lock_file(&options.directory.join("working.lock")) + .map_err(storage)?, + _workspace: options.images.workspace().cloned(), + }; + let working = match WorkingImage::open( + &options, + None, + cell_image_history_digest(authority.state().cell()), + ) { + Ok(working) => working, + Err(error) => { + options.discard_cell_scratch().map_err(storage)?; + return Err(storage(error)); + } + }; + if let Err(error) = authority.check() { + working.close_cell(&options).map_err(storage)?; + return Err(authority_failure(error)); + } + Ok(Self { + options, + authority, + working: Some(working), + pending: None, + staged: false, + lock, + }) + } + + /// Run once on the admitted blocking worker. Followers consume the encoded + /// initial LTX image and never execute the initializer. + pub fn prepare( + &mut self, + attempt: CellEnrollmentAttempt, + initialize: impl FnOnce(&CommandTransaction<'_>) -> anyhow::Result<()>, + ) -> Result { + if self.staged { + return Err(ExecutionError::Busy); + } + self.authority.check().map_err(authority_failure)?; + let log_reservation = self + .authority + .state() + .reserve_log(self.options.limits.envelope) + .map_err(authority_failure)?; + self.staged = true; + let working = self + .working + .as_mut() + .ok_or(ExecutionError::RebuildRequired)?; + working.begin_mutation(); + working + .database() + .transaction(attempt.inputs, |_, tx| initialize(tx)) + .map_err(private_failure)?; + let update = working + .capture_initial(&self.options) + .map_err(private_failure)?; + let state = self.authority.state(); + let route = state.route(); + let entry = CellGroupEntry { + binding: GroupEntryBinding { + group: route.history, + route_epoch: route.epoch, + birth_digest: route.birth_digest, + schema_digest: route.schema_digest, + proposal: attempt.creation, + }, + command: GroupCommand::Enroll { + cell: state.cell().clone(), + image: update.result, + files: update + .files + .iter() + .map(|file| GroupLtx { + min_txid: file.min_txid, + max_txid: file.max_txid, + page_count: file.page_count, + decoded_bytes: file.decoded_bytes, + bytes: &file.bytes, + }) + .collect(), + }, + }; + let bytes = entry + .encode( + route, + GroupEntryLimits { + envelope: self.options.limits.envelope, + ..Default::default() + }, + ) + .map_err(ExecutionError::Protocol)?; + let record = EnrollmentRecord::new( + state.cell().clone(), + attempt.creation, + route.schema_digest, + update.result.into(), + ) + .map_err(storage)?; + let preview = self + .options + .images + .preview_initial( + &self.options, + attempt.preview, + &update, + cell_image_history_digest(state.cell()), + ) + .map_err(storage)?; + self.authority.check().map_err(authority_failure)?; + log_reservation + .encoded(bytes.len()) + .map_err(|error| authority_failure(error.into()))?; + self.pending = Some(Pending { + log_reservation: log_reservation.clone(), + record, + _preview: preview, + submitted: false, + }); + Ok(CellPreparedCommand::new( + bytes, + attempt.creation, + state.term(), + state.route().history, + state.engine_owner().clone(), + ) + .with_log_reservation(log_reservation)) + } + + pub fn mark_submitted(&mut self, creation: ProposalId) -> Result<(), ExecutionError> { + self.authority.check().map_err(authority_failure)?; + let pending = self.pending.as_mut().ok_or(ExecutionError::Uncertain)?; + if pending.record.creation() != creation || pending.submitted { + return Err(ExecutionError::Invalid("enrollment submission")); + } + pending.submitted = true; + Ok(()) + } + + /// The immutable creation record survives later cell mutations and original + /// log compaction. Absence/conflict retires an old producer only behind a + /// strictly newer applied term. No callback or initial-image replay occurs. + pub fn resolve( + &mut self, + creation: ProposalId, + state: CellEnrollmentState, + ) -> Result { + let pending = self.pending.as_ref().ok_or(ExecutionError::Uncertain)?; + if pending.record.creation() != creation || !self.authority.state().same_binding(&state) { + return Err(ExecutionError::Invalid("enrollment resolution binding")); + } + state.check().map_err(authority_failure)?; + let outcome = match state.record() { + Some(record) if record == &pending.record => { + CellEnrollmentResolution::Committed(Box::new(record.clone())) + } + _ if state.term() <= self.authority.state().term() => { + return Err(ExecutionError::Uncertain) + } + Some(_) => CellEnrollmentResolution::Conflict, + None => CellEnrollmentResolution::NotCommitted, + }; + // A terminal owner observation resolves this disposable preview. Its + // live pin still protects physical work until pending is dropped; the + // independently applied image owns every durable serving reference. + self.options + .images + .retire(&pending._preview) + .map_err(storage)?; + pending + .log_reservation + .release() + .map_err(|error| authority_failure(error.into()))?; + self.pending = None; + Ok(outcome) + } + + pub fn close(mut self) -> Result<(), ExecutionError> { + let store = self.options.images.root_store().cloned(); + if let Some(working) = self.working.take() { + working.close_cell(&self.options).map_err(storage)?; + } + drop(self.options); + drop(self.lock); + if let Some(store) = store { + if store.usage().map_err(storage)?.quarantined { + return Err(storage(cellar_ltx::internal::StoreError::Quarantined)); + } + } + Ok(()) + } +} diff --git a/crates/cellar/durability/raft/execution/images.rs b/crates/cellar/durability/raft/execution/images.rs index 2140af734..669e45429 100644 --- a/crates/cellar/durability/raft/execution/images.rs +++ b/crates/cellar/durability/raft/execution/images.rs @@ -109,6 +109,40 @@ impl From for Options { } impl Options { + pub(super) fn prepare_directory(&self) -> Result<(), ExecutionError> { + let filesystem = self.host.filesystem(); + if let Some(workspace) = self.images.workspace() { + // This exact live owner already created and synced the directory + // under its locked inventory. Keep its pin; neither a pathname nor + // materialized execution can reuse that completed creation proof. + if workspace.directory() != self.directory.as_path() + || !filesystem + .symlink_metadata(workspace.directory()) + .map_err(storage)? + .is_dir + { + return Err(ExecutionError::Invalid("private workspace directory")); + } + return Ok(()); + } + cellar_ltx::host::create_dir_all_durable(filesystem.as_ref(), &self.directory) + .map_err(storage) + } + + /// Only shared-cell private scratch is disposable here. Keep the lock inode + /// and sealed image stores; an unresolved producer still owns this directory + /// until its physical callback/settlement has joined. + pub(super) fn discard_cell_scratch(&self) -> anyhow::Result<()> { + if let Images::Materialized(_) = &self.images { + crate::raft::working::clean( + self.host.filesystem().as_ref(), + &self.directory, + &self.directory.join("working.sqlite"), + )?; + } + Ok(()) + } + pub fn materialized(&self) -> anyhow::Result { let Images::Materialized(images) = &self.images else { anyhow::bail!("materialized private storage required"); @@ -256,6 +290,58 @@ impl WorkingImage { } } + pub(super) fn open_cell( + options: &Options, + base: &crate::raft::cell_groups::CellImagePin, + history: [u8; 32], + ) -> anyhow::Result { + use crate::raft::cell_groups::CellImagePin; + match (&options.images, base) { + (Images::Materialized(_), CellImagePin::Materialized(pin)) => { + Ok(Self::Materialized(Box::new( + crate::raft::working::Working::open_pin(&options.materialized()?, Some(pin))?, + ))) + } + (Images::Materialized(_), CellImagePin::Empty) => Ok(Self::Materialized(Box::new( + crate::raft::working::Working::open_pin(&options.materialized()?, None)?, + ))), + (Images::Extents { .. }, CellImagePin::Extents(pin)) => Ok(Self::Rooted(Box::new( + super::rooted_working::RootWorking::open_pin(options, Some(pin.clone()), history)?, + ))), + _ => anyhow::bail!("cell image storage differs"), + } + } + + pub(super) fn capture_cell( + &mut self, + options: &Options, + base: &crate::raft::cell_groups::CellImagePin, + ) -> anyhow::Result { + use crate::raft::cell_groups::CellImagePin; + match (self, base) { + (Self::Materialized(working), CellImagePin::Materialized(pin)) => { + working.capture_pin(&options.materialized()?, Some(pin)) + } + (Self::Materialized(working), CellImagePin::Empty) => { + working.capture_pin(&options.materialized()?, None) + } + (Self::Rooted(working), CellImagePin::Extents(pin)) => { + working.capture_pin(options, Some(pin)) + } + _ => anyhow::bail!("cell image storage differs"), + } + } + + pub(super) fn capture_initial( + &mut self, + options: &Options, + ) -> anyhow::Result { + match self { + Self::Materialized(working) => working.capture_pin(&options.materialized()?, None), + Self::Rooted(working) => working.capture_pin(options, None), + } + } + pub fn database(&self) -> &crate::storage::CommandDatabase { match self { Self::Materialized(working) => &working.database, @@ -277,6 +363,14 @@ impl WorkingImage { } } + pub(super) fn close_cell(self, options: &Options) -> anyhow::Result<()> { + // Close SQLite/WAL capture before removing any path, including when its + // close reports an error. The caller still owns the private lock. + let closed = self.close(); + options.discard_cell_scratch()?; + closed + } + pub fn trim(&self, options: &Options, base: Option<&Published>) -> anyhow::Result<()> { match self { Self::Materialized(working) => working.trim( @@ -404,37 +498,101 @@ impl Images { update: &CellUpdate, history: [u8; 32], ) -> anyhow::Result { + let pin = base.map(|base| match base { + Published::Materialized(image) => { + crate::raft::cell_groups::CellImagePin::Materialized(image.pin()) + } + Published::Rooted(image) => { + crate::raft::cell_groups::CellImagePin::Extents(image.image()) + } + }); + self.preview_payload( + options, + pin.as_ref(), + id, + crate::raft::working::CapturedUpdateRef { + base: update.base, + result: update.result, + files: &update.files, + }, + history, + ) + } + + pub(super) fn preview_cell( + &self, + options: &Options, + base: &crate::raft::cell_groups::CellImagePin, + id: ImageId, + update: &crate::raft::working::CapturedUpdate, + history: [u8; 32], + ) -> anyhow::Result { + self.preview_payload(options, Some(base), id, update.as_ref(), history) + } + + pub(super) fn preview_initial( + &self, + options: &Options, + id: ImageId, + update: &crate::raft::working::CapturedUpdate, + history: [u8; 32], + ) -> anyhow::Result { + self.preview_payload(options, None, id, update.as_ref(), history) + } + + fn preview_payload( + &self, + options: &Options, + base: Option<&crate::raft::cell_groups::CellImagePin>, + id: ImageId, + update: crate::raft::working::CapturedUpdateRef<'_>, + history: [u8; 32], + ) -> anyhow::Result { + let crate::raft::working::CapturedUpdateRef { + base: before, + result: after, + files, + } = update; match self { - Self::Materialized(_) => { - Ok(Preview::Materialized(crate::raft::working::verify_preview( + Self::Materialized(_) => Ok(Preview::Materialized( + crate::raft::working::verify_preview_pin( &options.materialized()?, - base.map(Published::flat).transpose()?, + match base { + None | Some(crate::raft::cell_groups::CellImagePin::Empty) => None, + Some(crate::raft::cell_groups::CellImagePin::Materialized(pin)) => { + Some(pin) + } + _ => anyhow::bail!("materialized base required"), + }, id, - update, - )?)) - } + before, + after, + files, + )?, + )), Self::Extents { store, .. } => { - let result = match update.result.commitment { + let result = match after.commitment { ImageCommitment::Extents(reference) => reference, _ => anyhow::bail!("extent result required"), }; - let before = match update.base.commitment { + let before = match before.commitment { ImageCommitment::Extents(reference) => reference, _ => anyhow::bail!("extent base required"), }; let header = cellar_ltx::ltx::Header::parse( - &update - .files + &files .last() .ok_or_else(|| anyhow::anyhow!("empty update"))? .bytes, )?; - let base = base - .map(Published::rooted) - .transpose()? - .map(RootPublishedImage::image); - let segments = update - .files + let base = match base { + Some(crate::raft::cell_groups::CellImagePin::Extents(pin)) => { + Some(pin.as_ref()) + } + None => None, + _ => anyhow::bail!("extent base required"), + }; + let segments = files .iter() .map(|file| { Ok(cellar_ltx::apply::LtxSegment { @@ -447,16 +605,16 @@ impl Images { }) .collect::>>()?; let prepared = store.prepare_image_root( - base.as_deref(), + base, &ImageRootUpdate { base: before, result, binding: ImageBinding { history, schema: options.application_digest, - epoch: update.result.epoch, - txid: update.result.txid, - checksum: update.result.checksum, + epoch: after.epoch, + txid: after.txid, + checksum: after.checksum, }, geometry: Geometry::new( header.page_size, @@ -477,10 +635,27 @@ impl Images { options: &Options, image: &Published, ) -> anyhow::Result { + let image = match image { + Published::Materialized(image) => { + crate::raft::cell_groups::CellImagePin::Materialized(image.pin()) + } + Published::Rooted(image) => { + crate::raft::cell_groups::CellImagePin::Extents(image.image()) + } + }; + self.read_cell(options, &image) + } + + pub(super) fn read_cell( + &self, + options: &Options, + image: &crate::raft::cell_groups::CellImagePin, + ) -> anyhow::Result { + use crate::raft::cell_groups::CellImagePin; match (self, image) { - (Self::Materialized(_), Published::Materialized(image)) => { + (Self::Materialized(_), CellImagePin::Materialized(image)) => { Ok(cellar_ltx::generation::open_immutable( - image.pin(), + image.clone(), options.query_services.clone(), )?) } @@ -491,10 +666,9 @@ impl Images { cache, .. }, - Published::Rooted(image), + CellImagePin::Extents(image), ) => Ok(cellar_ltx::internal::open_extent_image( image - .image() .view() .image(cache_directory.clone(), cache_pool.clone(), *cache)?, options.query_services.clone(), diff --git a/crates/cellar/durability/raft/execution/mod.rs b/crates/cellar/durability/raft/execution/mod.rs index 5168198ad..96114aae0 100644 --- a/crates/cellar/durability/raft/execution/mod.rs +++ b/crates/cellar/durability/raft/execution/mod.rs @@ -7,7 +7,17 @@ use super::apply::{marker, PublishedImage}; use super::cache::{DurableApply, GroupLog}; use super::completion::GroupStatus; use super::receipts; +mod cell; +mod submission; +pub(crate) use cell::authority_failure; +pub use submission::{CellPreparedCommand, CellProposalRejection, CellSubmissionOutcome}; +mod command; +mod enrollment; +mod registered; +pub use enrollment::{CellEnrollmentAttempt, CellEnrollmentResolution, CellPrivateEnrollment}; +pub(crate) use registered::validate as validate_registered_command; mod images; +pub use cell::{CellCommittedReceipt, CellPrivateExecutor, CellReconciliation, CellStageOutcome}; mod rooted_working; use crate::application::consensus::{ command_request_digest, CommittedVersion, EnvelopeLimits, OperationContract, OperationKind, @@ -183,7 +193,7 @@ pub(crate) fn query_registered_controlled( position: published.position(), }; let result = class - .invoke_query(contract, &version, &database, input) + .invoke_group_query(contract, &version, &database, input) .map_err(storage)?; if result.len() > limits.max_result_bytes { return Err(ExecutionError::Refused(CommandFailure::Overloaded)); @@ -625,43 +635,19 @@ impl PrivateExecutor { allow_new: bool, system: bool, ) -> Result { - let contract = OperationContract { - operation: attempt.request.operation.clone(), - kind: OperationKind::Command, - digest: attempt.request.contract_digest, - }; - if system { - class.validate_system_operation(&contract) - } else { - class.validate_operation(&contract) - } - .map_err(ExecutionError::Registration)?; - if class.schema_digest() != self.options.application_digest { - return Err(ExecutionError::Registration( - RegistrationError::ContractMismatch, - )); - } - if input.len() > self.options.limits.envelope.max_payload_bytes { - return Err(ExecutionError::Refused(CommandFailure::Overloaded)); - } - if command_request_digest(input) != attempt.request.request_digest { - return Err(ExecutionError::Refused(CommandFailure::CommandConflict)); - } - let scope = self.authority.replica.scope.clone(); + let scope = cellar_logic::cell_group::CellHistory::try_from(&self.authority.replica.scope) + .map_err(storage)?; + let contract = registered::validate( + self.options.limits, + self.options.application_digest, + &scope, + &attempt.request, + class, + input, + system, + )?; let command = attempt.request.id; let durability = attempt.request.durability; - if !system { - class - .validate_command_identity(&scope, &contract, command, durability, input) - .map_err(|error| { - ExecutionError::Refused( - error - .downcast_ref::() - .cloned() - .unwrap_or(CommandFailure::UnsupportedCapability), - ) - })?; - } self.stage_admitted(attempt, allow_new, |transaction| { if system { class.invoke_system_command( @@ -706,8 +692,7 @@ impl PrivateExecutor { } authority.check_base(base.as_ref(), options.application_digest)?; let filesystem = options.host.filesystem(); - cellar_ltx::host::create_dir_all_durable(filesystem.as_ref(), &options.directory) - .map_err(storage)?; + options.prepare_directory()?; let lock = filesystem .lock_file(&options.directory.join("working.lock")) .map_err(storage)?; @@ -821,102 +806,41 @@ impl PrivateExecutor { command: impl FnOnce(&CommandTransaction<'_>) -> anyhow::Result>, ) -> Result { self.ready()?; - if attempt.request.operation.is_empty() || attempt.request.operation.len() > 128 { - return Err(ExecutionError::Invalid("command operation")); - } let working = self .working .as_ref() .ok_or(ExecutionError::RebuildRequired)?; - if receipts::preflight_identity( + let maintenance = match command::prepare( working.database().framework(), - &attempt.request, - &self.options.limits, - )? - .is_some() - { - let receipt = self.read_committed( - &attempt.request, - self.base - .as_ref() - .ok_or(ExecutionError::Invalid("receipt without base"))?, - )?; - return Ok(StageOutcome::Duplicate(receipt)); - } - let count = receipts::retained_count(working.database().framework())?; - let ceiling = self.options.limits.max_retained_receipts; - let maintenance = match self.options.limits.receipt_maintenance { - Some(limits) if count >= ceiling - ceiling / 4 => { - let plan = super::receipt_maintenance::plan( - working.database().framework(), - attempt.inputs.now_unix_millis, - limits, - self.receipt_cursor, + self.options.limits, + &mut self.receipt_cursor, + &attempt, + allow_new, + )? { + command::Admission::Duplicate => { + let receipt = self.read_committed( + &attempt.request, + self.base + .as_ref() + .ok_or(ExecutionError::Invalid("receipt without base"))?, )?; - self.receipt_cursor = plan.next_cursor; - Some(plan) + return Ok(StageOutcome::Duplicate(receipt)); } - _ => None, + command::Admission::New(maintenance) => maintenance, }; - let removed = maintenance.as_ref().map_or(0, |plan| plan.rows_removed); - if count - .checked_sub(removed) - .is_none_or(|count| count >= ceiling) - { - return Err(ExecutionError::Refused(CommandFailure::Overloaded)); - } - if attempt.request.retry_until_ms < attempt.inputs.now_unix_millis - || attempt - .request - .retry_until_ms - .checked_sub(attempt.inputs.now_unix_millis) - .is_none_or(|window| window > self.options.limits.max_retry_window_ms) - { - return Err(ExecutionError::Invalid("command retry window")); - } - // Duplicate receipt lookup above remains available under archive - // pressure. A known refusal has not mutated SQL or capture state and - // must not force the next request to reconstruct the private image. - if !allow_new { - return Err(ExecutionError::Refused(CommandFailure::Overloaded)); - } self.rebuild_required = true; - self.working + let working = self + .working .as_mut() - .ok_or(ExecutionError::RebuildRequired)? - .begin_mutation(); - self.working - .as_ref() - .ok_or(ExecutionError::RebuildRequired)? - .database() - .transaction(attempt.inputs, |connection, transaction| { - if let Some(maintenance) = maintenance { - maintenance - .apply(connection, attempt.inputs.now_unix_millis) - .map_err(anyhow::Error::new)?; - } - let result = command(transaction)?; - if result.len() > self.options.limits.max_result_bytes { - return Err(anyhow::Error::new(CommandFailure::Overloaded)); - } - receipts::record(connection, &attempt.request, result)?; - let limits = &self.options.limits; - if limits.receipt_reserve_bytes != 0 { - let usage = crate::storage::CommandDatabase::capacity( - connection, - limits.max_image_bytes, - )?; - limits - .check_capacity( - usage.application_bytes, - usage.receipt_bytes, - usage.structural_reserve_bytes, - ) - .map_err(anyhow::Error::new)?; - } - Ok(()) - }) - .map_err(private_failure)?; + .ok_or(ExecutionError::RebuildRequired)?; + working.begin_mutation(); + command::apply( + working.database(), + self.options.limits, + &attempt, + maintenance, + command, + )?; let prepared = self.freeze( attempt.proposal, attempt.preview, diff --git a/crates/cellar/durability/raft/execution/registered.rs b/crates/cellar/durability/raft/execution/registered.rs new file mode 100644 index 000000000..1f2067599 --- /dev/null +++ b/crates/cellar/durability/raft/execution/registered.rs @@ -0,0 +1,57 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Registration and request identity are checked even for a receipt retry. +use super::*; +use cellar_logic::cell_group::CellHistory; + +pub(crate) fn validate( + limits: ExecutionLimits, + schema: [u8; 32], + history: &CellHistory, + request: &CommandRequest, + class: &RaftClass, + input: &[u8], + system: bool, +) -> Result { + let contract = OperationContract { + operation: request.operation.clone(), + kind: OperationKind::Command, + digest: request.contract_digest, + }; + if system { + class.validate_system_operation(&contract) + } else { + class.validate_operation(&contract) + } + .map_err(ExecutionError::Registration)?; + if class.schema_digest() != schema { + return Err(ExecutionError::Registration( + RegistrationError::ContractMismatch, + )); + } + if input.len() > limits.envelope.max_payload_bytes { + return Err(ExecutionError::Refused(CommandFailure::Overloaded)); + } + if command_request_digest(input) != request.request_digest { + return Err(ExecutionError::Refused(CommandFailure::CommandConflict)); + } + if !system { + class + .validate_command_identity(history, &contract, request.id, request.durability, input) + .map_err(|error| { + ExecutionError::Refused( + error + .downcast_ref::() + .cloned() + .unwrap_or(CommandFailure::UnsupportedCapability), + ) + })?; + } else if request.id.client + != crate::raft::effects::system_client(history).map_err(ExecutionError::Refused)? + { + return Err(ExecutionError::Refused( + CommandFailure::UnsupportedCapability, + )); + } + Ok(contract) +} diff --git a/crates/cellar/durability/raft/execution/rooted_working.rs b/crates/cellar/durability/raft/execution/rooted_working.rs index 2af25995b..22c2fac82 100644 --- a/crates/cellar/durability/raft/execution/rooted_working.rs +++ b/crates/cellar/durability/raft/execution/rooted_working.rs @@ -37,6 +37,14 @@ impl RootWorking { options: &Options, base: Option<&crate::raft::rooted::RootPublishedImage>, history: [u8; 32], + ) -> Result { + Self::open_pin(options, base.map(|base| base.image()), history) + } + + pub fn open_pin( + options: &Options, + base: Option>, + history: [u8; 32], ) -> Result { let Images::Extents { store, @@ -56,9 +64,9 @@ impl RootWorking { let path = options.directory.join("working.sqlite"); crate::raft::working::clean(filesystem.as_ref(), &options.directory, &path)?; let image = match base { - Some(image) => image.image(), - None => Arc::new(store.seal_image_root( - ExtentManifest::new( + Some(image) => image, + None => { + let manifest = ExtentManifest::new( ImageBinding { history, schema: options.application_digest, @@ -68,9 +76,14 @@ impl RootWorking { }, Geometry::new(0, 0)?, &[], - )?, - &mut vec![0; EXTENT_BYTES as usize], - )?), + )?; + // This empty base belongs only to the disposable workspace. + // Pin before sealing so concurrent collection cannot remove it. + let receiving = store.stage_transient_root(manifest.clone())?; + let image = store.seal_image_root(manifest, &mut vec![0; EXTENT_BYTES as usize])?; + drop(receiving); + Arc::new(image) + } }; ensure!( image.manifest().binding().history == history @@ -78,7 +91,10 @@ impl RootWorking { "working extent history/schema differs" ); let initial = RootState::of(&image); - cellar_ltx::host::create_dir_all_durable(filesystem.as_ref(), cache_directory)?; + // Anonymous read-cache files are disposable and never supply recovery + // authority. Their directory need not survive a crash; immutable image + // publication still requires the inventory's own persistence barriers. + filesystem.create_dir_all(cache_directory)?; let workspace = ExtentWorkspace::new( image, WorkspaceOptions { @@ -170,9 +186,17 @@ impl RootWorking { proposal: ProposalId, base: Option<&crate::raft::rooted::RootPublishedImage>, ) -> Result { - let before = base - .map(|base| RootState::of(&base.image())) - .unwrap_or(self.initial); + Ok(self + .capture_pin(options, base.map(|base| base.image()).as_deref())? + .bind(scope.clone(), options.application_digest, proposal)) + } + + pub fn capture_pin( + &mut self, + options: &Options, + base: Option<&cellar_ltx::internal::SealedImageRoot>, + ) -> Result { + let before = base.map(RootState::of).unwrap_or(self.initial); let (position, files) = crate::raft::working::capture_update( &mut self.capture, &self.database, @@ -213,10 +237,7 @@ impl RootWorking { geometry: observed.manifest().geometry(), }); self.observation = Some(observed); - Ok(CellUpdate { - scope: scope.clone(), - application_digest: options.application_digest, - proposal, + Ok(crate::raft::working::CapturedUpdate { base: before.image, result, files, diff --git a/crates/cellar/durability/raft/execution/submission.rs b/crates/cellar/durability/raft/execution/submission.rs new file mode 100644 index 000000000..4840a18b7 --- /dev/null +++ b/crates/cellar/durability/raft/execution/submission.rs @@ -0,0 +1,219 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! A prepared SQL attempt is accepted once or permanently sealed before append. + +use crate::raft::{cell_groups::CellGroupProposal, completion::RuntimeError}; +use cellar_logic::{ + cell_group::GroupHistory, + consensus::{ProposalId, Term}, +}; +use std::sync::{Arc, Mutex}; + +#[derive(Default)] +enum Decision { + #[default] + Open, + Submitting, + Uncertain, + Accepted(CellGroupProposal), + Rejected, +} + +struct Submission { + engine: Arc<()>, + decision: Mutex, +} + +#[derive(Clone)] +pub struct CellPreparedCommand { + log_reservation: Option>, + bytes: Arc>, + proposal: ProposalId, + term: Term, + group: GroupHistory, + submission: Arc, +} + +/// This preparation was sealed before append. It is not a claim about other +/// preparations of the same business request. All clones share the same seal. +#[derive(Clone)] +pub struct CellProposalRejection { + submission: Arc, +} + +pub enum CellSubmissionOutcome { + /// Acceptance remains provisional, never a commit or S3 acknowledgement. + Accepted(CellGroupProposal), + Rejected(CellProposalRejection), +} + +impl CellPreparedCommand { + pub(super) fn new( + bytes: Vec, + proposal: ProposalId, + term: Term, + group: GroupHistory, + engine: Arc<()>, + ) -> Self { + Self { + log_reservation: None, + bytes: Arc::new(bytes), + proposal, + term, + group, + submission: Arc::new(Submission { + engine, + decision: Mutex::new(Decision::Open), + }), + } + } + + pub(super) fn with_log_reservation( + mut self, + lease: Arc, + ) -> Self { + self.log_reservation = Some(lease); + self + } + + pub(crate) fn log_reservation( + &self, + ) -> Result<&Arc, RuntimeError> { + self.log_reservation + .as_ref() + .ok_or_else(|| RuntimeError::Invalid("missing prepared log reservation".into())) + } + + pub(super) fn release_log_reservation(&self) -> Result<(), RuntimeError> { + if let Some(lease) = &self.log_reservation { + lease.release()?; + } + Ok(()) + } + + pub fn term(&self) -> Term { + self.term + } + + pub fn bytes(&self) -> &[u8] { + &self.bytes + } + + pub fn proposal(&self) -> ProposalId { + self.proposal + } + + pub(crate) fn group(&self) -> GroupHistory { + self.group + } + + pub(crate) fn belongs_to(&self, engine: &Arc<()>, group: GroupHistory) -> bool { + Arc::ptr_eq(engine, &self.submission.engine) && group == self.group + } + + pub(crate) fn submit( + &self, + engine: &Arc<()>, + group: GroupHistory, + attempt: impl FnOnce() -> Result, RuntimeError>, + ) -> Result { + if !self.belongs_to(engine, group) { + return Err(RuntimeError::Invalid("prepared submission owner".into())); + } + let mut decision = self + .submission + .decision + .lock() + .map_err(|_| RuntimeError::Protocol("submission state poisoned".into()))?; + let uncertain = matches!(*decision, Decision::Uncertain); + match &*decision { + Decision::Accepted(accepted) => { + return Ok(CellSubmissionOutcome::Accepted(accepted.clone())) + } + Decision::Rejected => return Ok(CellSubmissionOutcome::Rejected(self.rejected())), + Decision::Submitting => return Err(RuntimeError::NotReady), + Decision::Open | Decision::Uncertain => (), + } + *decision = Decision::Submitting; + drop(decision); + // No lock spans decoding, hashing, RawNode work or async operations. + // Only the owner's definitive pre-append capacity refusal returns None. + let outcome = attempt(); + let mut decision = self + .submission + .decision + .lock() + .map_err(|_| RuntimeError::Protocol("submission state poisoned".into()))?; + match outcome { + Ok(Some(accepted)) => { + *decision = Decision::Accepted(accepted.clone()); + Ok(CellSubmissionOutcome::Accepted(accepted)) + } + Ok(None) if !uncertain => { + *decision = Decision::Rejected; + self.release_log_reservation()?; + Ok(CellSubmissionOutcome::Rejected(self.rejected())) + } + Ok(None) => { + // This refusal says nothing about an earlier attempted append. + *decision = Decision::Uncertain; + Err(RuntimeError::Overloaded) + } + Err(error) => { + *decision = Decision::Uncertain; + Err(error) + } + } + } + + fn rejected(&self) -> CellProposalRejection { + CellProposalRejection { + submission: self.submission.clone(), + } + } + + /// Only a never-attempted preparation may be cancelled locally. The same + /// lock serializes the seal with every delayed or currently entering clone. + pub(crate) fn cancel_before_submission( + &self, + ) -> Result, RuntimeError> { + let mut decision = self + .submission + .decision + .lock() + .map_err(|_| RuntimeError::Protocol("submission state poisoned".into()))?; + if matches!(*decision, Decision::Open) { + *decision = Decision::Rejected; + } + if matches!(*decision, Decision::Rejected) { + self.release_log_reservation()?; + } + Ok(matches!(*decision, Decision::Rejected).then(|| self.rejected())) + } + + /// Recover a lost rejection response without asking a closed or replaced + /// runtime. Open, entering, uncertain and accepted attempts never yield a + /// rejection proof without an explicit seal of a never-attempted preparation. + pub fn rejection(&self) -> Result, RuntimeError> { + let decision = self + .submission + .decision + .lock() + .map_err(|_| RuntimeError::Protocol("submission state poisoned".into()))?; + Ok(matches!(*decision, Decision::Rejected).then(|| self.rejected())) + } + + pub(crate) fn matches_rejection(&self, rejection: &CellProposalRejection) -> bool { + Arc::ptr_eq(&self.submission, &rejection.submission) + } +} + +#[cfg(test)] +#[path = "submission_tests.rs"] +mod tests; + +impl CellProposalRejection { + pub(crate) fn belongs_to_engine(&self, engine: &Arc<()>) -> bool { + Arc::ptr_eq(engine, &self.submission.engine) + } +} diff --git a/crates/cellar/durability/raft/execution/submission_tests.rs b/crates/cellar/durability/raft/execution/submission_tests.rs new file mode 100644 index 000000000..ae6499c3d --- /dev/null +++ b/crates/cellar/durability/raft/execution/submission_tests.rs @@ -0,0 +1,73 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use cellar_logic::consensus::{ + ClusterId, GroupId, IncarnationId, LogIndex, LogPosition, NamespaceId, +}; + +fn prepared() -> CellPreparedCommand { + CellPreparedCommand::new( + vec![1], + ProposalId::new([1; 16]).unwrap(), + Term::new(1), + GroupHistory { + cluster: ClusterId::new([2; 16]).unwrap(), + namespace: NamespaceId::new([3; 16]).unwrap(), + group: GroupId::new([4; 16]).unwrap(), + incarnation: IncarnationId::new([5; 16]).unwrap(), + }, + Arc::new(()), + ) +} + +#[test] +fn cancellation_seals_every_delayed_submission_clone() { + let command = prepared(); + let delayed = command.clone(); + let proof = command.cancel_before_submission().unwrap().unwrap(); + assert!(delayed.matches_rejection(&proof)); + assert!(matches!( + delayed.submit(&command.submission.engine, command.group, || panic!( + "sealed clone reached append" + )), + Ok(CellSubmissionOutcome::Rejected(_)) + )); +} + +#[test] +fn attempted_error_cannot_turn_into_cancellation_or_capacity_proof() { + let command = prepared(); + assert!(command + .submit(&command.submission.engine, command.group, || Err( + RuntimeError::Protocol("uncertain append".into()) + )) + .is_err()); + assert!(command.cancel_before_submission().unwrap().is_none()); + assert!(matches!( + command.submit(&command.submission.engine, command.group, || Ok(None)), + Err(RuntimeError::Overloaded) + )); + assert!(command.rejection().unwrap().is_none()); + assert!(command.cancel_before_submission().unwrap().is_none()); +} + +#[test] +fn entering_or_accepted_submission_cannot_be_cancelled() { + let command = prepared(); + let accepted = CellGroupProposal { + group: command.group, + position: LogPosition { + index: LogIndex::new(2), + term: Term::new(1), + }, + digest: [9; 32], + }; + assert!(matches!( + command.submit(&command.submission.engine, command.group, || { + assert!(command.cancel_before_submission().unwrap().is_none()); + Ok(Some(accepted)) + }), + Ok(CellSubmissionOutcome::Accepted(_)) + )); + assert!(command.cancel_before_submission().unwrap().is_none()); +} diff --git a/crates/cellar/durability/raft/fleet_action_transport.rs b/crates/cellar/durability/raft/fleet_action_transport.rs index 356b328b3..1781f0570 100644 --- a/crates/cellar/durability/raft/fleet_action_transport.rs +++ b/crates/cellar/durability/raft/fleet_action_transport.rs @@ -387,11 +387,11 @@ impl State { stream.read_exact(&mut bytes).await?; let command = request.decode_body(&bytes)?; self.authorize_node_inspection(&request.route)?; - let handle = self.query_gateway.lock().map_err(|_| TransportError::Closed)? + let handle = self.fleet_gateway.lock().map_err(|_| TransportError::Closed)? .as_ref().and_then(|handle| handle.upgrade()).ok_or(TransportError::Refused)?; let remaining = deadline.checked_sub(self.domain.mono_ms().map_err(|_| TransportError::Closed)?) .filter(|ms| *ms > 0).ok_or(TransportError::Timeout)?; - let work = RaftCells { handle }.execute_local_fleet_action(command, request.origin.clone(), + let work = RaftCells::from_internal_handle(handle).execute_local_fleet_action(command, request.origin.clone(), request.route.minimum_revision()?, std::time::Duration::from_millis(remaining)); let mut extra = [0; 1]; let result = crate::asyncrt::select_biased! { "disconnect cancels the waiter, not committed movement"; diff --git a/crates/cellar/durability/raft/fleet_catalog/README.md b/crates/cellar/durability/raft/fleet_catalog/README.md index b51d6c4a9..09b9d23ed 100644 --- a/crates/cellar/durability/raft/fleet_catalog/README.md +++ b/crates/cellar/durability/raft/fleet_catalog/README.md @@ -573,6 +573,14 @@ placement and original-member identity, then invokes the existing guarded registration and initial-member pipeline. An imported birth, dynamic member, retired slot or claimed blank engine cannot grant fresh genesis. +Live voting verification divides its remaining observation budget among the +unobserved members and final validation. A lost read-only membership reply can +therefore return Unavailable before consuming the complete turn. The pending +history remains unverified; the owning walk must obtain all three fresh safe +ReadIndex/durable-apply observations again. Root-input, boot, configuration and +proof-age checks remain mandatory. This allocation extends no deadline and +introduces no mutation retry or stale-proof fallback. + The shared fleet action lane now carries explicit Movement and RestartGenesis variants. Their receiving authorities remain separate; a movement permit cannot authorize genesis. TLS exporter/purpose binding, route, nonce, body digest, @@ -880,12 +888,25 @@ generic mutations cannot erase the receipt, clear quarantine or change its boot. A lost confirmation reconciles the exact original journal before current live prerequisites. Changed evidence under the same request identity conflicts. +Shared-host startup checks the current host authorization together with namespace +declarations against one rooted cut before opening the mutable Raft engine. A +saved pre-isolation capability therefore refuses without changing an already +quarantined engine. This is an early refusal check, not a disk fence against a +concurrent platform action: the later engine-bound boot CAS still rechecks current +authority before publishing endpoints or attaching groups. + A resource-backed repair can pin the quarantined obsolete voter while obtaining fresh quorum from surviving voters. Route construction omits only an explicitly quarantined endpoint; placement validation still uses the full durable membership, including that voter's node and failure domain. Raft configuration and quorum thresholds do not change until the membership protocol commits them. +Shared physical route refresh also removes explicitly quarantined initial +endpoints when membership is otherwise unchanged. Keeping such an endpoint would +make the refreshed route fail trust validation and prevent an admitted learner +from receiving its seed. Missing discovery or an unreachable surviving voter does +not establish isolation and cannot supply its settlement closure. + Successful movement completion can substitute the admitted isolation proof only for the exact original obsolete boot, after fresh target quorum proves that member absent from stable RF3. All surviving original voters and the target @@ -1463,7 +1484,10 @@ the durable handoff; neither a timeout nor Unavailable can evict a voter. `start_failure_recovery_service(observer, movement_evidence, repair_evidence, platform, options, recovery)` uses the same exclusive Node-owned worker and -existing probe bounds. It is explicitly enabled and default-off. Installing +existing probe bounds. It is explicitly enabled and default-off. The worker +must have an exact rooted boot whose state permits admitted movement actions. +Each recovery turn revalidates that authority before changing the catalog or +calling the platform. Installing `FleetFailureRecoveryPlatform` authorizes permanent isolation of a failed boot; operators must select and authenticate that deployment integration. `FleetDockerIsolation` supplies a Linux Docker/Colima adapter for exclusively @@ -1489,10 +1513,13 @@ deployment; they are not inferred from Engine block-I/O counters or a filesystem scan. This homogeneous-policy provider covers Ready/Active recovery targets; departure/return settlement still needs its own retained evidence. Complete deployment wiring and resource-limit qualification remain separate gates. -`FleetFailureRecoveryOptions` supplies detection bounds plus an explicit LTX host, -archive verifier and archive limits. The embedding runtime owns the verifier's -private scratch store and disk/memory admission, including extent storage when -required by its namespace. Startup validates both detection and archive bounds. +`FleetFailureRecoveryOptions` supplies detection bounds and an explicit +`FleetGroupArchiveVerifier`. The embedding runtime owns its private image/page +inventories and shared checkpoint, archive, tail and scratch admission, including +extent storage when required by its namespace. Receiver setup creates and syncs +the private spool directory on the execution domain before materialized image +replay. Startup validates both detection and archive bounds. Serving image pins +cannot replace physical archive verification. The worker first advances due admitted movements. During their retry/backoff windows it collects direct failure observations and complete namespace handles, @@ -1546,6 +1573,16 @@ VerifyingRemoval and Removed, with checked/total history counts and the confirme terminal revision. Only Removed follows an accepted terminal receipt. Status grants no current health lease; observation timestamps remain local. +Catalog turns box their work before constructing the turn future, keeping large +replay state out of enclosing poll frames before object-store HTTP calls. +Control turns, recovery/removal phases and complete live verification retain +boxed futures under the existing worker admission, stop owner and original +deadline. This bounds nested poll-frame stack use without increasing the +worker stack or changing failure grace. Receiver rejection returns its typed +checkpoint failure after joining the consumer, rather than hiding it behind a +closed-channel message. An archive outage retains the intent and disruption; +controller restart repeats physical replay before finalization/removal. + The SDK handoff includes the opt-in Docker/Colima isolation adapter described above. Simultaneous-failure policy, deployment startup integration and deployed fleet qualification remain required. The complete final survivor renewal still shares diff --git a/crates/cellar/durability/raft/fleet_catalog/archive_replay.rs b/crates/cellar/durability/raft/fleet_catalog/archive_replay.rs new file mode 100644 index 000000000..114d45c64 --- /dev/null +++ b/crates/cellar/durability/raft/fleet_catalog/archive_replay.rs @@ -0,0 +1,244 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Complete physical group replay for managed lifecycle verification. The +//! supplied inventories belong to the verifier, separately from serving data. +use super::*; +use crate::application::consensus::group_protocol::GroupEntryLimits; +use crate::raft::catalog::{CellGroupArchiveBindingError, RaftImageStorage, RangeDirectory}; +use crate::raft::cell_groups::{ + CellGroupApplyImages, CellGroupArchiveCheckpointReceiver, CellGroupArchivePool, + CellGroupArchiveRegistry, CellGroupArchiveStore, CellGroupArchiveTailPool, + CellGroupCheckpointError, CellGroupCheckpointPool, CellGroupTransferTarget, ManifestLimits, + ManifestPageStore, +}; +use cellar_logic::cell_group::GroupRoute; +use cellar_ltx::{stream_snapshot::ScratchPool, FileSystem}; +use sha2::{Digest, Sha256}; +use std::path::{Component, PathBuf}; + +/// Explicit verifier-owned physical stores and existing per-group page bounds. +/// Image, checkpoint, tail and scratch pools are shared across all histories. +/// Use private inventories: serving stores cannot substitute for S3 replay. +pub struct FleetGroupArchiveReplayOptions { + pub directory: PathBuf, + pub filesystem: Arc, + pub materialized: Option, + pub extents: Option, + pub manifest: ManifestLimits, + pub entries: GroupEntryLimits, + pub max_pages: usize, + pub max_page_bytes: u64, + pub max_retained_roots: usize, + pub max_effect_rows: u32, +} + +/// Opens independent receivers on the caller's execution domain. This owns no +/// Raft log, application callback, admission authority or remote retention lease. +#[derive(Clone)] +pub struct FleetGroupArchiveVerifier { + archives: CellGroupArchivePool, + tails: CellGroupArchiveTailPool, + checkpoints: CellGroupCheckpointPool, + scratch: ScratchPool, + options: Arc, +} + +impl FleetGroupArchiveVerifier { + pub fn new( + archives: CellGroupArchivePool, + tails: CellGroupArchiveTailPool, + checkpoints: CellGroupCheckpointPool, + scratch: ScratchPool, + options: FleetGroupArchiveReplayOptions, + ) -> Result { + if !options.directory.is_absolute() + || options.directory.as_os_str().len() > 3800 + || options + .directory + .components() + .any(|p| matches!(p, Component::ParentDir)) + || options.materialized.is_none() && options.extents.is_none() + || options + .materialized + .as_ref() + .is_some_and(|i| !matches!(i, CellGroupApplyImages::Materialized { .. })) + || options + .extents + .as_ref() + .is_some_and(|i| !matches!(i, CellGroupApplyImages::Extents { .. })) + || !(1..=1_000_000).contains(&options.max_effect_rows) + { + return Err(FleetCatalogError::Invalid("group archive replay storage")); + } + ManifestPageStore::validate_limits( + options.max_pages, + options.max_page_bytes, + options.max_retained_roots, + ) + .and_then(|()| options.manifest.validate()) + .map_err(|_| FleetCatalogError::Limit)?; + options + .entries + .validate() + .map_err(|_| FleetCatalogError::Invalid("group archive entry limits"))?; + for images in options.materialized.iter().chain(options.extents.iter()) { + images + .validate() + .map_err(|e| FleetCatalogError::Storage(e.into()))?; + } + Ok(Self { + archives, + tails, + checkpoints, + scratch, + options: Arc::new(options), + }) + } + + pub(super) fn archives(&self) -> &CellGroupArchivePool { + &self.archives + } + + pub(crate) fn tails(&self) -> &CellGroupArchiveTailPool { + &self.tails + } + + /// Reopen the immutable namespace-owned registry for this exact rooted + /// physical history. Never initialize counters or select a cell image. + pub(crate) async fn open( + &self, + domain: &DomainToken, + namespace: &RangeDirectory, + history: &FleetHistory, + ) -> Result { + domain + .run(async { + let scope = history + .binding() + .scope() + .map_err(|_| FleetCatalogError::Corrupt("archive history"))?; + let birth = namespace.materialize_selected_history(scope).await?; + if birth.route().history != scope + || birth.route().birth_digest != history.birth_digest() + || birth.route().schema_digest != history.schema_digest() + { + return Err(FleetCatalogError::Conflict); + } + namespace + .open_group_archive(domain, &self.archives, &birth) + .await + .map_err(|e| match e { + CellGroupArchiveBindingError::Catalog(e) => FleetCatalogError::from(e), + CellGroupArchiveBindingError::Archive(e) => FleetCatalogError::from(e), + }) + }) + .await + .map_err(|_| FleetCatalogError::Closed)? + } + + /// Allocate a complete receiver under the existing checkpoint/scratch + /// admissions. Deterministic group paths keep retries bounded; a prior + /// entered worker retains its directory lock until physical work exits. + pub(crate) async fn receiver( + &self, + domain: &DomainToken, + route: &GroupRoute, + storage: RaftImageStorage, + ) -> Result { + let admission = self + .checkpoints + .prepare_archive_receive(self.options.max_pages) + .map_err(checkpoint_error)?; + self.with_private_target(domain, route, storage, move |target| { + admission + .receive(target.store, target.scratch, target.directory) + .map_err(checkpoint_error) + }) + .await + } + + pub(super) async fn transfer_target( + &self, + domain: &DomainToken, + route: &GroupRoute, + storage: RaftImageStorage, + ) -> Result { + self.with_private_target(domain, route, storage, Ok).await + } + + async fn with_private_target( + &self, + domain: &DomainToken, + route: &GroupRoute, + storage: RaftImageStorage, + finish: impl FnOnce(CellGroupTransferTarget) -> Result + Send + 'static, + ) -> Result { + let route = route.clone(); + let mut images = match storage { + RaftImageStorage::Materialized => self.options.materialized.as_ref(), + RaftImageStorage::Extents => self.options.extents.as_ref(), + } + .ok_or(FleetCatalogError::Invalid( + "missing group archive image inventory", + ))? + .clone(); + let mut digest = Sha256::new(); + digest.update(b"cellar-fleet-group-replay-directory-1"); + digest.update(route.history.cluster.bytes()); + digest.update(route.history.namespace.bytes()); + digest.update(route.history.group.bytes()); + digest.update(route.history.incarnation.bytes()); + let key = hex::encode(digest.finalize()); + if let CellGroupApplyImages::Extents { + cache_directory, .. + } = &mut images + { + *cache_directory = self.options.directory.join("cache").join(&key); + } + let options = self.options.clone(); + let pages = options.directory.join("groups").join(key).join("pages"); + let scratch = self.scratch.clone(); + domain + .blocking(move || { + let directory = options.directory.join("scratch"); + // Image receivers require an existing spool parent. Initialize + // our private inventory before admitting any physical replay. + cellar_ltx::host::create_dir_all_durable(options.filesystem.as_ref(), &directory) + .map_err(|e| FleetCatalogError::Storage(e.into()))?; + let pages = ManifestPageStore::open( + &pages, + options.filesystem.clone(), + options.max_pages, + options.max_page_bytes, + options.max_retained_roots, + ) + .map_err(|e| FleetCatalogError::Storage(e.into()))?; + let store = CellGroupArchiveStore::new( + route, + pages, + images, + options.manifest, + options.entries, + options.max_effect_rows, + ) + .map_err(|e| FleetCatalogError::Storage(e.into()))?; + finish(CellGroupTransferTarget { + store, + scratch, + directory, + }) + }) + .map_err(|_| FleetCatalogError::Closed)? + .await + .map_err(|e| FleetCatalogError::Storage(e.into()))? + } +} + +fn checkpoint_error(error: CellGroupCheckpointError) -> FleetCatalogError { + match error { + CellGroupCheckpointError::Invalid(reason) => FleetCatalogError::Invalid(reason), + CellGroupCheckpointError::Limit => FleetCatalogError::Limit, + CellGroupCheckpointError::Busy => FleetCatalogError::Overloaded, + error => FleetCatalogError::Storage(error.into()), + } +} diff --git a/crates/cellar/durability/raft/fleet_catalog/boots.rs b/crates/cellar/durability/raft/fleet_catalog/boots.rs index bc8078072..14cedef34 100644 --- a/crates/cellar/durability/raft/fleet_catalog/boots.rs +++ b/crates/cellar/durability/raft/fleet_catalog/boots.rs @@ -118,7 +118,9 @@ impl FleetHostAuthorization { }], }; mutation.validate()?; - let committed = self.catalog.commit_inner(&snapshot, &mutation).await?; + // The boot caller retains startup state while this commit + // traverses Fleet index pages and polls the object-store client. + let committed = Box::pin(self.catalog.commit_inner(&snapshot, &mutation)).await?; let observed = self .catalog .host_from(&committed.snapshot, self.node()) diff --git a/crates/cellar/durability/raft/fleet_catalog/capacity.rs b/crates/cellar/durability/raft/fleet_catalog/capacity.rs index e54ef9476..eae2e1a64 100644 --- a/crates/cellar/durability/raft/fleet_catalog/capacity.rs +++ b/crates/cellar/durability/raft/fleet_catalog/capacity.rs @@ -6,7 +6,7 @@ use super::histories::validate_request; use super::nodes::NODE_KIND; use super::*; use crate::application::consensus::RaftCells; -use crate::raft::catalog::NamespaceCatalog; +use crate::raft::catalog::RangeDirectory; use crate::raft::fleet_stop::BootBinding; use crate::raft::host::RaftPeerNodeObservation; use crate::raft::inspection::RaftNodeProfile; @@ -203,7 +203,7 @@ impl FleetCatalog { snapshot: &FleetSnapshot, request: &FleetCapacityRequest, observer: &RaftCells, - namespaces: &[NamespaceCatalog], + namespaces: &[RangeDirectory], timeout: Duration, ) -> Result { self.check_snapshot(snapshot)?; diff --git a/crates/cellar/durability/raft/fleet_catalog/disruption.rs b/crates/cellar/durability/raft/fleet_catalog/disruption.rs index 7ec33f5a0..243526d08 100644 --- a/crates/cellar/durability/raft/fleet_catalog/disruption.rs +++ b/crates/cellar/durability/raft/fleet_catalog/disruption.rs @@ -8,6 +8,8 @@ pub(super) const SLOT_KEY: &str = "active-disruption"; pub(super) enum Disruption { Evacuation(evacuation::Record), Restart(maintenance::Record), + Transition(managed_transitions::Record), + Park(managed_park::Record), } impl Disruption { @@ -17,13 +19,25 @@ impl Disruption { if let Ok(record) = decode::(evacuation::ADMISSION_KIND, bytes, 4096) { return Ok(Self::Evacuation(record)); } - Ok(Self::Restart(decode(maintenance::KIND, bytes, 4096)?)) + if let Ok(record) = decode::(maintenance::KIND, bytes, 4096) { + return Ok(Self::Restart(record)); + } + if let Ok(record) = decode::(managed_park::KIND, bytes, 4096) { + return Ok(Self::Park(record)); + } + Ok(Self::Transition(decode( + managed_transitions::KIND, + bytes, + 4096, + )?)) } fn validate(&self, catalog: &FleetCatalog) -> Result<()> { match self { Self::Evacuation(record) => record.validate(catalog), Self::Restart(record) => record.validate(catalog), + Self::Transition(record) => record.validate(catalog), + Self::Park(record) => record.validate(catalog), } } @@ -31,14 +45,16 @@ impl Disruption { match self { Self::Evacuation(record) => record.operation(), Self::Restart(record) => record.request.operation, + Self::Transition(record) => record.operation(), + Self::Park(record) => record.operation(), } } pub(super) fn evacuation(self) -> Result { match self { Self::Evacuation(record) => Ok(record), - Self::Restart(record) => Err(FleetCatalogError::DisruptionInProgress { - operation: record.request.operation, + other => Err(FleetCatalogError::DisruptionInProgress { + operation: other.operation(), }), } } @@ -47,6 +63,8 @@ impl Disruption { match self { Self::Evacuation(record) => record.mutation(), Self::Restart(record) => record.mutation(), + Self::Transition(record) => record.mutation(), + Self::Park(record) => record.mutation(), } } @@ -54,6 +72,8 @@ impl Disruption { match self { Self::Evacuation(record) => record.check_basis(snapshot), Self::Restart(record) => record.check_basis(snapshot), + Self::Transition(record) => record.check_basis(snapshot), + Self::Park(record) => record.check_basis(snapshot), } } } @@ -72,12 +92,14 @@ impl FleetCatalog { (Some(digest), Some(bytes)) if hash(&bytes) == digest => { let record = Disruption::decode(&bytes)?; record.validate(self)?; + let key = match &record { + Disruption::Transition(record) => record.key(), + Disruption::Park(record) => record.key(), + _ => hex::encode(record.operation()), + }; let operation = self .inner - .lookup( - snapshot.head.tables[FleetTable::Operations.slot()], - &hex::encode(record.operation()), - ) + .lookup(snapshot.head.tables[FleetTable::Operations.slot()], &key) .await?; if operation.as_deref() != Some(bytes.as_slice()) { return Err(FleetCatalogError::Corrupt("disruption operation reference")); @@ -89,6 +111,12 @@ impl FleetCatalog { Disruption::Restart(record) => { self.confirm_restart(snapshot, record).await?; } + Disruption::Transition(record) => { + self.confirm_managed_transition(snapshot, record).await?; + } + Disruption::Park(record) => { + self.confirm_managed_park(snapshot, record).await?; + } } Ok(Some(record)) } @@ -103,12 +131,21 @@ impl FleetCatalog { mutation: &FleetMutation, previous: Option<[u8; 32]>, bytes: Option<&[u8]>, + authority: resources::Authority, ) -> Result<[u8; 32]> { if previous.is_some() { return Err(FleetCatalogError::Invalid("unsettled disruption slot")); } let bytes = bytes.ok_or(FleetCatalogError::Invalid("disruption slot deletion"))?; let record = Disruption::decode(bytes)?; + if matches!(record, Disruption::Transition(_)) + && authority != resources::Authority::ManagedTransition + { + return Err(FleetCatalogError::Invalid("managed transition authority")); + } + if matches!(record, Disruption::Park(_)) && authority != resources::Authority::ManagedPark { + return Err(FleetCatalogError::Invalid("managed park authority")); + } record.validate(self)?; record.check_basis(snapshot)?; if record.mutation()? != *mutation { diff --git a/crates/cellar/durability/raft/fleet_catalog/docker_isolation/mod.rs b/crates/cellar/durability/raft/fleet_catalog/docker_isolation/mod.rs index e1d333d9a..4416060ba 100644 --- a/crates/cellar/durability/raft/fleet_catalog/docker_isolation/mod.rs +++ b/crates/cellar/durability/raft/fleet_catalog/docker_isolation/mod.rs @@ -5,7 +5,7 @@ //! Immutable IDs never authorize deletion or replacement-container discovery. use super::*; use crate::docker::{ContainerId, Docker}; -use crate::raft::catalog::NamespaceCatalog; +use crate::raft::catalog::RangeDirectory; use crate::raft::fleet_stop::BootBinding; use cellar_logic::consensus::{NodeId, ProposalId}; use std::pin::Pin; @@ -42,7 +42,7 @@ pub struct FleetDockerIsolation { struct Adapter { catalog: FleetCatalog, engine: Engine, - namespaces: Vec, + namespaces: Vec, turn: Mutex<()>, } @@ -54,7 +54,7 @@ impl FleetDockerIsolation { catalog: FleetCatalog, docker: Docker, owner: &str, - namespaces: Vec, + namespaces: Vec, ) -> Result { if namespaces.len() > super::namespaces::MAX_NAMESPACES { return Err(FleetCatalogError::Limit); @@ -176,7 +176,7 @@ impl FleetFailureRecoveryPlatform for FleetDockerIsolation { fn namespaces<'a>( &'a self, timeout: Duration, - ) -> Pin>> + Send + 'a>> { + ) -> Pin>> + Send + 'a>> { Box::pin(async move { self.run(timeout, async { Ok(self.inner.namespaces.clone()) }) .await diff --git a/crates/cellar/durability/raft/fleet_catalog/docker_isolation/resources.rs b/crates/cellar/durability/raft/fleet_catalog/docker_isolation/resources.rs index 336f2f81b..34c1cf859 100644 --- a/crates/cellar/durability/raft/fleet_catalog/docker_isolation/resources.rs +++ b/crates/cellar/durability/raft/fleet_catalog/docker_isolation/resources.rs @@ -201,10 +201,6 @@ impl FleetIsolatedRepairEvidence for FleetDockerResources { let timeout = timeout.min(Duration::from_secs(5)); self.isolation .run(timeout, async { - context - .group - .validate() - .map_err(|_| FleetCatalogError::Invalid("Docker repair scope"))?; context .failed_boot .validate() @@ -212,6 +208,9 @@ impl FleetIsolatedRepairEvidence for FleetDockerResources { let catalog = &self.isolation.inner.catalog; let deadline = Deadline::new(catalog, timeout)?; let cut = catalog.read(deadline.remaining()?).await?; + catalog + .authorize_group(&cut, &context.group, deadline.remaining()?) + .await?; let nodes = catalog .run(deadline.remaining()?, catalog.node_records(&cut)) .await?; @@ -234,7 +233,7 @@ impl FleetIsolatedRepairEvidence for FleetDockerResources { } Ok(FleetIsolatedRepairInputs { history: FleetTopologyHistoryResources { - group: context.group.clone(), + group: context.group, resident: self.policy.resident, peak: self.policy.peak, }, diff --git a/crates/cellar/durability/raft/fleet_catalog/evacuation.rs b/crates/cellar/durability/raft/fleet_catalog/evacuation.rs index 88852e77e..abfac4f4b 100644 --- a/crates/cellar/durability/raft/fleet_catalog/evacuation.rs +++ b/crates/cellar/durability/raft/fleet_catalog/evacuation.rs @@ -3,7 +3,7 @@ use super::histories::{history_mutation, validate_request}; use super::nodes::{NodeRecord, NODE_KIND}; use super::*; -use crate::raft::catalog::NamespaceCatalog; +use crate::raft::catalog::RangeDirectory; use crate::raft::fleet_stop::BootBinding; use cellar_logic::consensus::{DiskLineage, EngineInstanceId, NodeId, ProposalId}; use serde::Deserialize; @@ -291,7 +291,7 @@ impl FleetCatalog { snapshot: &FleetSnapshot, request: &FleetRequestIdentity, node: NodeId, - namespaces: &[NamespaceCatalog], + namespaces: &[RangeDirectory], timeout: Duration, ) -> Result { self.check_snapshot(snapshot)?; @@ -323,7 +323,7 @@ impl FleetCatalog { snapshot: &FleetSnapshot, request: &FleetRequestIdentity, node: NodeId, - namespaces: &[NamespaceCatalog], + namespaces: &[RangeDirectory], timeout: Duration, ) -> Result { self.check_snapshot(snapshot)?; @@ -352,7 +352,7 @@ impl FleetCatalog { snapshot: &FleetSnapshot, request: &FleetRequestIdentity, failure: (BootBinding, u64), - namespaces: &[NamespaceCatalog], + namespaces: &[RangeDirectory], timeout: Duration, ) -> Result { self.check_snapshot(snapshot)?; @@ -384,7 +384,7 @@ impl FleetCatalog { snapshot: &FleetSnapshot, request: &FleetRequestIdentity, node: NodeId, - namespaces: &[NamespaceCatalog], + namespaces: &[RangeDirectory], mode: AdmissionMode, ) -> Result { let current = self.inner.current().await?; diff --git a/crates/cellar/durability/raft/fleet_catalog/finalization/archive_holds.rs b/crates/cellar/durability/raft/fleet_catalog/finalization/archive_holds.rs new file mode 100644 index 000000000..c42f53d9b --- /dev/null +++ b/crates/cellar/durability/raft/fleet_catalog/finalization/archive_holds.rs @@ -0,0 +1,228 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! One durable acquire allocation for an exact verification basis and fixed cut. +//! A timeout leaves the allocation untouched. These holds outlive local handles; +//! terminal cleanup must fence all users and join physical work before release. +use super::*; +use crate::raft::cell_groups::{ + CellGroupArchiveAcquireOutcome, CellGroupArchiveCreation, CellGroupArchiveError, + CellGroupArchiveIntent, +}; +use receipts::Publication; +use serde::Deserialize; + +const KIND: u8 = 101; + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Record { + basis: Basis, + fixed: Publication, + registry: CellGroupArchiveCreation, + attempt: u64, + intent: CellGroupArchiveIntent, +} + +impl Record { + fn check( + &self, + basis: &Basis, + fixed: &FleetPublishedCut, + registry: &CellGroupArchiveRegistry, + ) -> Result<()> { + if &self.basis != basis || &self.registry != registry.creation() || self.attempt == u64::MAX + { + return Err(FleetCatalogError::PayloadConflict); + } + if self.fixed != Publication::from(fixed) { + return Err(FleetCatalogError::Conflict); + } + check_marker(self.intent.acquire_marker()?, fixed) + } +} + +fn check_marker( + marker: &crate::raft::cell_groups::GroupManifest, + fixed: &FleetPublishedCut, +) -> Result<()> { + if !crate::raft::host::group_prefix(marker, fixed.root()) + .map_err(|_| FleetCatalogError::Conflict)? + { + return Err(FleetCatalogError::Unconfirmed); + } + Ok(()) +} + +impl FleetEvacuationVerification { + fn archive_hold_key(&self) -> Result { + let pending = &self + .pending + .as_ref() + .ok_or(FleetCatalogError::InventoryIncomplete)? + .0; + Ok(format!( + "removal-archive-acquires/{}/{}", + hex::encode(hash(&encode(KIND, &self.book.basis, PAGE_LIMIT)?)), + pending.key + )) + } + + /// Recover a fixed cut even if the last controller disappeared between + /// acquiring retention and appending a live receipt. A new observation must + /// cover the stored cut; it cannot silently select a new acquire journal. + pub(super) async fn archive_fixed( + &self, + current: &FleetPublishedCut, + ) -> Result> { + let Some((record, _)) = self + .catalog + .inner + .read::(&self.archive_hold_key()?, KIND, PAGE_LIMIT) + .await? + else { + return Ok(None); + }; + let fixed = record.fixed.restore(current)?; + if record.basis != self.book.basis + || record.registry.route() != fixed.root().route() + || record.attempt == u64::MAX + { + return Err(FleetCatalogError::PayloadConflict); + } + check_marker(record.intent.acquire_marker()?, &fixed)?; + Ok(Some(fixed)) + } + + /// Acquire this walk's own remote retention. Call freeze_live first. The + /// exact intent is stored in the fleet catalog before any registry mutation. + /// Unknown outcomes reconcile the same allocation after controller loss. + /// No publisher-owned hold, current-health lease or terminal authority is + /// borrowed. Local cancellation/drop never retires this persistent owner. + pub async fn acquire_archive_hold( + &mut self, + registry: &CellGroupArchiveRegistry, + timeout: Duration, + ) -> Result { + let catalog = &self.catalog; + let deadline = Deadline::new(catalog, timeout)?; + let fixed = &self + .frozen + .as_ref() + .ok_or(FleetCatalogError::Unconfirmed)? + .fixed; + let basis = &self.book.basis; + if registry.route() != fixed.root().route() { + return Err(FleetCatalogError::Conflict); + } + let key = self.archive_hold_key()?; + let result = catalog + .run(deadline.remaining()?, async { + basis.check_current(catalog).await?; + let (record, token) = + match catalog.inner.read::(&key, KIND, PAGE_LIMIT).await? { + Some(stored) => stored, + None => { + let candidate = prepare(catalog, registry, basis, fixed, 0).await?; + persist(catalog, &key, &candidate, None).await? + } + }; + record.check(basis, fixed, registry)?; + basis.check_current(catalog).await?; + match registry + .acquire(&catalog.inner.domain, &record.intent) + .await + { + Ok(hold) => Ok(hold), + Err( + error @ (CellGroupArchiveError::Expired | CellGroupArchiveError::Conflict), + ) => { + match registry + .acquire_outcome(&catalog.inner.domain, &record.intent) + .await? + { + CellGroupArchiveAcquireOutcome::Held(hold) => Ok(*hold), + CellGroupArchiveAcquireOutcome::Pending => Err(error.into()), + CellGroupArchiveAcquireOutcome::Superseded => { + // A fresh consistent registry read proved that the + // old revision is consumed and its nonce owns no + // hold. Its delayed conditional write cannot win. + let next = record + .attempt + .checked_add(1) + .ok_or(FleetCatalogError::Limit)?; + let candidate = + prepare(catalog, registry, basis, fixed, next).await?; + basis.check_current(catalog).await?; + let (winner, _) = + persist(catalog, &key, &candidate, Some(&token)).await?; + winner.check(basis, fixed, registry)?; + // Reconcile this persisted winner on a later turn; + // never hide another allocation retry in this call. + Err(FleetCatalogError::Unconfirmed) + } + } + } + Err(error) => Err(error.into()), + } + }) + .await; + let hold = match result { + Ok(hold) => hold, + Err(error) => { + // Another controller may have persisted an earlier fixed cut + // since this one's quorum capture. Re-enter fresh quorum proof + // and restore that winner; do not move its journal or nonce. + self.frozen = None; + return Err(error); + } + }; + check_marker(hold.marker(), fixed)?; + catalog + .run(deadline.remaining()?, basis.check_current(catalog)) + .await?; + deadline.remaining()?; + Ok(hold) + } +} + +async fn prepare( + catalog: &FleetCatalog, + registry: &CellGroupArchiveRegistry, + basis: &Basis, + fixed: &FleetPublishedCut, + attempt: u64, +) -> Result { + let intent = registry.prepare_acquire(&catalog.inner.domain).await?; + let record = Record { + basis: basis.clone(), + fixed: fixed.into(), + registry: registry.creation().clone(), + attempt, + intent, + }; + record.check(basis, fixed, registry)?; + Ok(record) +} + +/// Conditional journal storage chooses one candidate even when simultaneous +/// controllers prepare different nonces. Only the observed winner may be sent. +async fn persist( + catalog: &FleetCatalog, + key: &str, + candidate: &Record, + token: Option<&str>, +) -> Result<(Record, String)> { + let bytes = encode(KIND, candidate, PAGE_LIMIT)?; + let outcome = catalog + .inner + .bucket + .put_cas(&catalog.inner.key(key), bytes, token) + .await; + match catalog.inner.read(key, KIND, PAGE_LIMIT).await? { + Some(stored) => Ok(stored), + None => match outcome { + Err(error) => Err(FleetCatalogError::Storage(error)), + Ok(_) => Err(FleetCatalogError::Unconfirmed), + }, + } +} diff --git a/crates/cellar/durability/raft/fleet_catalog/finalization/fencing.rs b/crates/cellar/durability/raft/fleet_catalog/finalization/fencing.rs index 2af4f1fae..50ac7e155 100644 --- a/crates/cellar/durability/raft/fleet_catalog/finalization/fencing.rs +++ b/crates/cellar/durability/raft/fleet_catalog/finalization/fencing.rs @@ -262,7 +262,7 @@ impl FleetEvacuationCandidate { snapshot: &FleetSnapshot, request: &FleetRequestIdentity, observer: &RaftCells, - namespaces: &[NamespaceCatalog], + namespaces: &[RangeDirectory], timeout: Duration, ) -> Result { self.begin_finalization_inner( @@ -284,7 +284,7 @@ impl FleetEvacuationCandidate { snapshot: &FleetSnapshot, request: &FleetRequestIdentity, isolation: ProposalId, - namespaces: &[NamespaceCatalog], + namespaces: &[RangeDirectory], timeout: Duration, ) -> Result { self.begin_finalization_inner( @@ -306,7 +306,7 @@ impl FleetEvacuationCandidate { snapshot: &FleetSnapshot, request: &FleetRequestIdentity, source: FinalizationSource<'_>, - namespaces: &[NamespaceCatalog], + namespaces: &[RangeDirectory], timeout: Duration, ) -> Result { let catalog = &self.catalog; diff --git a/crates/cellar/durability/raft/fleet_catalog/finalization/mod.rs b/crates/cellar/durability/raft/fleet_catalog/finalization/mod.rs index f5ea5c5e6..4b47726a4 100644 --- a/crates/cellar/durability/raft/fleet_catalog/finalization/mod.rs +++ b/crates/cellar/durability/raft/fleet_catalog/finalization/mod.rs @@ -2,6 +2,7 @@ //! Bounded receipt material for inventory-consuming terminal verification. //! Stored descriptors grant no completion or disruption-release authority. +mod archive_holds; mod cancellation; mod fencing; pub(super) mod receipts; @@ -19,8 +20,11 @@ pub use terminal::{ use super::*; use crate::application::consensus::RaftCells; -use crate::raft::archive::{ArchiveReference, ArchiveVerifier, CellArchive}; -use crate::raft::catalog::NamespaceCatalog; +use crate::raft::catalog::RangeDirectory; +use crate::raft::cell_groups::{ + CellGroupArchiveCheckpointReceiver, CellGroupArchiveHold, CellGroupArchiveRegistry, + CellGroupArchiveTailPool, +}; use crate::raft::host::FleetPublishedCut; use cellar_logic::consensus::{NodeId, ProposalId}; use receipts::{Basis, Book, Candidate, Entry, Evidence, LiveEvidence}; @@ -28,9 +32,10 @@ use receipts::{Basis, Book, Candidate, Entry, Evidence, LiveEvidence}; /// Services and the exact immutable archive selected for one history. Scratch /// storage and recovery-memory admission remain owned by the embedding runtime. pub struct FleetEvacuationArchive<'a> { - pub archive: &'a CellArchive, - pub reference: &'a ArchiveReference, - pub verifier: &'a ArchiveVerifier, + pub registry: &'a CellGroupArchiveRegistry, + pub hold: &'a CellGroupArchiveHold, + pub tails: &'a CellGroupArchiveTailPool, + pub receiver: CellGroupArchiveCheckpointReceiver, } /// A bounded, complete-inventory walk that owns its authenticated quorum calls @@ -107,7 +112,7 @@ impl FleetCatalog { &self, snapshot: &FleetSnapshot, operation: ProposalId, - namespaces: &[NamespaceCatalog], + namespaces: &[RangeDirectory], timeout: Duration, ) -> Result { self.check_snapshot(snapshot)?; @@ -247,6 +252,32 @@ impl FleetEvacuationVerification { .await } + /// Consume a retired history already checked against the closed transfer + /// and its exact source retirement set by complete inventory. + pub async fn verify_retired(&mut self, timeout: Duration) -> Result<()> { + let catalog = self.catalog.clone(); + let deadline = Deadline::new(&catalog, timeout)?; + let history = self + .next_history(deadline.remaining()?) + .await? + .ok_or(FleetCatalogError::InventoryIncomplete)?; + let FleetHistoryState::Retired { + operation, + retirement, + } = *history.state() + else { + return Err(FleetCatalogError::Conflict); + }; + self.append( + Evidence::Retired { + operation, + retirement, + }, + &deadline, + ) + .await + } + /// Freeze this pending history's fixed cut through three actual quorum /// responses. Repeated calls return that same historical cut, allowing /// archive catch-up while writes continue. Rooted-input changes invalidate @@ -327,13 +358,12 @@ impl FleetEvacuationVerification { .try_into() .map_err(|_| FleetCatalogError::Corrupt("receipt observation count"))?; let observations: [crate::raft::host::RaftPeerFleetMembershipObservation; 3] = observations; - let current = observations[0] - .published_cut() - .ok_or(FleetCatalogError::Unconfirmed)?; + let current = observations[0].published_cut(); let (fixed, survivors) = catalog .run(deadline.remaining()?, async { self.book.basis.check_current(&catalog).await?; let mut fixed = current.clone(); + let mut from_prior = false; if let Some(prior) = &self.prior { let (pending, _) = self .pending @@ -342,13 +372,25 @@ impl FleetEvacuationVerification { if let Some(entry) = prior.entry(&catalog, &pending.key).await? { if entry.record == hash(&pending.value) { if let Evidence::Live(live) = entry.evidence { - if live.fixed.revision == current.membership_revision().get() { + if live.fixed.configuration_revision()? + == current.membership_revision() + { fixed = live.fixed.restore(current)?; + from_prior = true; } } } } } + if let Some(saved) = self.archive_fixed(current).await? { + if from_prior + && (saved.root() != fixed.root() + || saved.configuration_digest() != fixed.configuration_digest()) + { + return Err(FleetCatalogError::Conflict); + } + fixed = saved; + } let survivors = survivors::verify(&catalog, &self.snapshot, history, &observations, &fixed) .await?; @@ -370,51 +412,59 @@ impl FleetEvacuationVerification { /// needed, capture that cut first through actual quorum calls under this /// same method deadline. Prior freeze_live work remains historical; retry /// does not move its business-write boundary or grant a freshness lease. - pub async fn verify_live( - &mut self, - observer: &RaftCells, + pub fn verify_live<'a>( + &'a mut self, + observer: &'a RaftCells, nodes: [NodeId; 3], - archive: FleetEvacuationArchive<'_>, + archive: FleetEvacuationArchive<'a>, timeout: Duration, - ) -> Result<()> { - let catalog = self.catalog.clone(); - let deadline = Deadline::new(&catalog, timeout)?; - let fixed = self - .freeze_live(observer, nodes, deadline.remaining()?) - .await?; - let coverage = catalog - .inner - .domain - .run(archive.archive.verify_published_at( - archive.reference, - &fixed, - archive.verifier, - deadline.remaining()?, - )) + ) -> futures_util::future::BoxFuture<'a, Result<()>> { + // A caller retains this entire physical verification operation. Its + // state must not also occupy the caller's control-service poll frame. + // Keep the same fixed cut, deadline and post-replay authority checks. + Box::pin(async move { + let catalog = self.catalog.clone(); + let deadline = Deadline::new(&catalog, timeout)?; + let fixed = self + .freeze_live(observer, nodes, deadline.remaining()?) + .await?; + let coverage = catalog + .run(deadline.remaining()?, async { + archive + .registry + .verify_fleet_published_at( + &catalog.inner.domain, + archive.tails, + archive.hold, + &fixed, + archive.receiver, + ) + .await + .map_err(FleetCatalogError::from) + }) + .await?; + deadline.remaining()?; + if !coverage + .covers(&fixed) + .map_err(|_| FleetCatalogError::Conflict)? + { + return Err(FleetCatalogError::Unconfirmed); + } + self.append( + Evidence::Live(Box::new(LiveEvidence { + fixed: (&fixed).into(), + archive: (&coverage).into(), + survivors: self + .frozen + .as_ref() + .ok_or(FleetCatalogError::Unconfirmed)? + .survivors + .clone(), + })), + &deadline, + ) .await - .map_err(|_| FleetCatalogError::Closed)? - .map_err(|error| FleetCatalogError::Storage(error.into()))?; - deadline.remaining()?; - if !coverage - .covers(&fixed) - .map_err(|_| FleetCatalogError::Conflict)? - { - return Err(FleetCatalogError::Unconfirmed); - } - self.append( - Evidence::Live(Box::new(LiveEvidence { - fixed: (&fixed).into(), - archive: (&coverage).into(), - survivors: self - .frozen - .as_ref() - .ok_or(FleetCatalogError::Unconfirmed)? - .survivors - .clone(), - })), - &deadline, - ) - .await + }) } async fn append(&mut self, evidence: Evidence, deadline: &Deadline<'_>) -> Result<()> { diff --git a/crates/cellar/durability/raft/fleet_catalog/finalization/receipts.rs b/crates/cellar/durability/raft/fleet_catalog/finalization/receipts.rs index e5f6fa769..ed00173a4 100644 --- a/crates/cellar/durability/raft/fleet_catalog/finalization/receipts.rs +++ b/crates/cellar/durability/raft/fleet_catalog/finalization/receipts.rs @@ -4,8 +4,10 @@ //! from these records; the runtime must independently establish every proof. use super::super::evacuation::ActiveEvacuation; use super::super::*; -use crate::raft::archive::{ArchiveReference, VerifiedArchiveCoverage}; -use crate::raft::host::{FleetImageState, FleetPublishedCut}; +use crate::raft::cell_groups::{ + CellGroupArchiveHold, GroupManifest, VerifiedFleetGroupArchiveCoverage, +}; +use crate::raft::host::FleetPublishedCut; use crate::raft::manifest::Position; use cellar_logic::consensus::{DiskLineage, EngineInstanceId, NodeId, ProcessGeneration}; use serde::Deserialize; @@ -13,6 +15,11 @@ use std::collections::BTreeSet; const CANDIDATE_KIND: u8 = 55; const ENTRY_KIND: u8 = 56; +const LIVE_KIND: u8 = 59; +#[cfg(test)] +#[path = "receipts_tests.rs"] +mod tests; + const ENTRY_LIMIT: usize = 4096; const APPEND_LIMIT: usize = 16; const BASIS_TABLES: [FleetTable; 4] = [ @@ -218,21 +225,15 @@ impl Basis { #[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] #[serde(deny_unknown_fields)] pub(in crate::raft::fleet_catalog) struct Publication { - pub(super) applied: Position, - pub(super) revision: u64, - #[serde(deserialize_with = "nullable")] - image_position: Option, - #[serde(deserialize_with = "nullable")] - image: Option, + root: GroupManifest, + configuration_digest: [u8; 32], } impl From<&FleetPublishedCut> for Publication { fn from(cut: &FleetPublishedCut) -> Self { Self { - applied: cut.applied().into(), - revision: cut.membership_revision().get(), - image_position: cut.image().map(|image| image.position().into()), - image: cut.image().map(|image| image.state()), + root: cut.root().clone(), + configuration_digest: cut.configuration_digest(), } } } @@ -241,94 +242,42 @@ impl Publication { pub(in crate::raft::fleet_catalog) fn applied_position( &self, ) -> Result { - self.applied - .decode() - .map_err(|_| FleetCatalogError::Corrupt("receipt publication position")) + self.root + .check() + .map_err(|_| FleetCatalogError::Corrupt("receipt group root"))?; + Ok(self.root.position()) } pub(in crate::raft::fleet_catalog) fn configuration_revision( &self, ) -> Result { - if self.revision == 0 || self.revision >= u64::MAX - 2 { - return Err(FleetCatalogError::Corrupt("receipt configuration revision")); - } - Ok(cellar_logic::consensus::LogIndex::new(self.revision)) + self.root + .check() + .map_err(|_| FleetCatalogError::Corrupt("receipt configuration root"))?; + Ok(cellar_logic::consensus::LogIndex::new( + self.root.membership_revision(), + )) } pub(super) fn restore(&self, current: &FleetPublishedCut) -> Result { - validate_image(self.applied, self.image_position, self.image)?; - let image = match (self.image_position, self.image) { - (Some(position), Some(image)) => Some( - crate::raft::host::FleetPublishedImage::new_state( - position - .decode() - .map_err(|_| FleetCatalogError::Corrupt("receipt publication"))?, - image, - ) - .map_err(|_| FleetCatalogError::Corrupt("receipt image"))?, - ), - (None, None) => None, - _ => return Err(FleetCatalogError::Corrupt("receipt image binding")), - }; current - .checked_prefix( - self.applied - .decode() - .map_err(|_| FleetCatalogError::Corrupt("receipt position"))?, - cellar_logic::consensus::LogIndex::new(self.revision), - image, - ) + .checked_prefix(self.root.clone(), self.configuration_digest) .map_err(|_| FleetCatalogError::Unconfirmed) } } -fn validate_image( - position: Position, - origin: Option, - image: Option, -) -> Result<()> { - position - .decode() - .map_err(|_| FleetCatalogError::Corrupt("receipt position"))?; - match (origin, image) { - (None, None) => Ok(()), - (Some(origin), Some(image)) => { - origin - .decode() - .map_err(|_| FleetCatalogError::Corrupt("receipt publication"))?; - image - .validate() - .map_err(|_| FleetCatalogError::Corrupt("receipt image"))?; - if origin.index > position.index - || origin.term > position.term - || origin.index == position.index && origin.term != position.term - { - return Err(FleetCatalogError::Corrupt("receipt publication prefix")); - } - Ok(()) - } - _ => Err(FleetCatalogError::Corrupt("receipt image binding")), - } -} - #[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] #[serde(deny_unknown_fields)] pub(super) struct Archive { - reference: ArchiveReference, - position: Position, - #[serde(deserialize_with = "nullable")] - image_position: Option, - #[serde(deserialize_with = "nullable")] - image: Option, + hold: CellGroupArchiveHold, + configuration_digest: [u8; 32], } -impl From<&VerifiedArchiveCoverage> for Archive { - fn from(proof: &VerifiedArchiveCoverage) -> Self { +impl From<&VerifiedFleetGroupArchiveCoverage> for Archive { + fn from(proof: &VerifiedFleetGroupArchiveCoverage) -> Self { Self { - reference: proof.reference().clone(), - position: proof.position().into(), - image_position: proof.image_position().map(Into::into), - image: proof.image_state(), + hold: proof.hold().clone(), + configuration_digest: proof.configuration_digest(), } } } @@ -351,7 +300,13 @@ pub(super) struct Survivor { #[serde(tag = "state", rename_all = "snake_case", deny_unknown_fields)] pub(super) enum Evidence { Live(Box), - Cancelled { operation: [u8; 16] }, + Cancelled { + operation: [u8; 16], + }, + Retired { + operation: [u8; 16], + retirement: [u8; 32], + }, } #[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] @@ -378,46 +333,35 @@ impl Entry { match &self.evidence { Evidence::Cancelled { operation } if *operation != [0; 16] => Ok(()), Evidence::Cancelled { .. } => Err(FleetCatalogError::Corrupt("receipt cancellation")), + Evidence::Retired { + operation, + retirement, + } if *operation != [0; 16] && *retirement != [0; 32] => Ok(()), + Evidence::Retired { .. } => Err(FleetCatalogError::Corrupt("receipt retirement")), Evidence::Live(live) => { let LiveEvidence { fixed, archive, survivors, } = live.as_ref(); - validate_image(fixed.applied, fixed.image_position, fixed.image)?; - validate_image(archive.position, archive.image_position, archive.image)?; - let selected = archive - .reference - .position() - .map_err(|_| FleetCatalogError::Corrupt("receipt archive reference"))?; - if Position::from(selected) != archive.position - || fixed.revision == 0 - || archive.position.index < fixed.applied.index - || archive.position.term < fixed.applied.term - || archive.position.index == fixed.applied.index - && archive.position.term != fixed.applied.term - { - return Err(FleetCatalogError::Corrupt("receipt archive cut")); - } - if fixed - .image - .as_ref() - .zip(archive.image.as_ref()) - .is_some_and(|(a, b)| !a.same_representation(b)) + fixed + .root + .check() + .and_then(|()| archive.hold.marker().check()) + .map_err(|_| FleetCatalogError::Corrupt("receipt complete group root"))?; + archive + .hold + .check_reference() + .map_err(|_| FleetCatalogError::Corrupt("receipt archive hold"))?; + if fixed.configuration_digest == [0; 32] + || fixed.configuration_digest != archive.configuration_digest + || fixed.root.route().history.cluster.bytes() != basis.cluster + || !crate::raft::host::group_prefix(archive.hold.marker(), &fixed.root) + .map_err(|_| FleetCatalogError::Corrupt("receipt archive group prefix"))? { - return Err(FleetCatalogError::Corrupt( - "receipt image commitment domain", - )); - } - if fixed.image.is_some() && archive.image.is_none() - || archive.image.is_some() - && archive.image != fixed.image - && archive - .image_position - .is_none_or(|origin| origin.index <= fixed.applied.index) - { - return Err(FleetCatalogError::Corrupt("receipt archive publication")); + return Err(FleetCatalogError::Corrupt("receipt archive group cut")); } + let fixed_position = fixed.root.position(); let mut nodes = BTreeSet::new(); let mut members = BTreeSet::new(); let mut hosts = BTreeSet::new(); @@ -442,10 +386,10 @@ impl Entry { || survivor.publication_digest == [0; 32] || !nodes.insert(survivor.node) || !members.insert(survivor.member) - || survivor.applied.index < fixed.applied.index - || survivor.applied.term < fixed.applied.term - || survivor.applied.index == fixed.applied.index - && survivor.applied.term != fixed.applied.term + || survivor.applied.index < fixed_position.index.get() + || survivor.applied.term < fixed_position.term.get() + || survivor.applied.index == fixed_position.index.get() + && survivor.applied.term != fixed_position.term.get() { return Err(FleetCatalogError::Corrupt("receipt survivor")); } @@ -460,6 +404,70 @@ impl Entry { } } +// Index values stay within the original 4 KiB bound. Complete roots and +// persistent archive holds live in one independently hashed bounded object. +#[derive(Serialize, Deserialize)] +#[serde(tag = "state", rename_all = "snake_case", deny_unknown_fields)] +enum StoredEvidence { + Live { + object: [u8; 32], + }, + Cancelled { + operation: [u8; 16], + }, + Retired { + operation: [u8; 16], + retirement: [u8; 32], + }, +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct StoredEntry { + history: [u8; 32], + record: [u8; 32], + evidence: StoredEvidence, +} + +type StoredEntryObjects = (Vec, Option<([u8; 32], Vec)>); + +impl Entry { + fn stored(&self) -> Result { + let (evidence, object) = match &self.evidence { + Evidence::Cancelled { operation } => ( + StoredEvidence::Cancelled { + operation: *operation, + }, + None, + ), + Evidence::Retired { + operation, + retirement, + } => ( + StoredEvidence::Retired { + operation: *operation, + retirement: *retirement, + }, + None, + ), + Evidence::Live(live) => { + let bytes = encode(LIVE_KIND, live, PAGE_LIMIT)?; + let digest = hash(&bytes); + ( + StoredEvidence::Live { object: digest }, + Some((digest, bytes)), + ) + } + }; + let stored = StoredEntry { + history: self.history, + record: self.record, + evidence, + }; + Ok((encode(ENTRY_KIND, &stored, ENTRY_LIMIT)?, object)) + } +} + /// An incremental immutable index, not an inventory or verification capability. #[derive(Clone)] pub(super) struct Book { @@ -493,12 +501,31 @@ impl Book { if entries.is_empty() || entries.len() > APPEND_LIMIT { return Err(FleetCatalogError::Limit); } - let mut index = self.index; - let mut count = self.count; + let mut prepared = Vec::with_capacity(entries.len()); + let mut bytes_total = 0usize; for entry in entries { entry.validate(&self.basis)?; - let key = hex::encode(entry.history); - let bytes = encode(ENTRY_KIND, entry, ENTRY_LIMIT)?; + let (stored, object) = entry.stored()?; + bytes_total = bytes_total + .checked_add(stored.len()) + .and_then(|n| n.checked_add(object.as_ref().map_or(0, |(_, bytes)| bytes.len()))) + .filter(|n| *n <= APPEND_LIMIT * ENTRY_LIMIT) + .ok_or(FleetCatalogError::Limit)?; + prepared.push((hex::encode(entry.history), stored, object)); + } + let mut index = self.index; + let mut count = self.count; + for (key, bytes, object) in prepared { + if let Some((digest, payload)) = object { + catalog + .inner + .create_exact( + &format!("receipt-live/{}", hex::encode(digest)), + &payload, + PAGE_LIMIT, + ) + .await?; + } if let Some(previous) = catalog.inner.lookup(index, &key).await? { if previous != bytes { return Err(FleetCatalogError::PayloadConflict); @@ -549,7 +576,40 @@ impl Candidate { let Some(bytes) = catalog.inner.lookup(self.index, key).await? else { return Ok(None); }; - let entry: Entry = decode(ENTRY_KIND, &bytes, ENTRY_LIMIT)?; + let stored: StoredEntry = decode(ENTRY_KIND, &bytes, ENTRY_LIMIT)?; + let evidence = match stored.evidence { + StoredEvidence::Cancelled { operation } => Evidence::Cancelled { operation }, + StoredEvidence::Retired { + operation, + retirement, + } => Evidence::Retired { + operation, + retirement, + }, + StoredEvidence::Live { object } => { + if object == [0; 32] { + return Err(FleetCatalogError::Corrupt("receipt live object")); + } + let (live, _) = catalog + .inner + .read::>( + &format!("receipt-live/{}", hex::encode(object)), + LIVE_KIND, + PAGE_LIMIT, + ) + .await? + .ok_or(FleetCatalogError::Corrupt("missing receipt live object"))?; + if hash(&encode(LIVE_KIND, &live, PAGE_LIMIT)?) != object { + return Err(FleetCatalogError::Corrupt("receipt live object digest")); + } + Evidence::Live(live) + } + }; + let entry = Entry { + history: stored.history, + record: stored.record, + evidence, + }; entry.validate(&self.basis)?; if hex::encode(entry.history) != key { return Err(FleetCatalogError::Corrupt("receipt entry key")); diff --git a/crates/cellar/durability/raft/fleet_catalog/finalization/receipts_tests.rs b/crates/cellar/durability/raft/fleet_catalog/finalization/receipts_tests.rs new file mode 100644 index 000000000..107ecab7f --- /dev/null +++ b/crates/cellar/durability/raft/fleet_catalog/finalization/receipts_tests.rs @@ -0,0 +1,93 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Storage-shape tests; decoded receipt material cannot mint runtime proofs. +use super::*; +use crate::application::consensus::cell_routing_key; +use cellar_logic::{cell_group::*, consensus::*}; + +fn large_entry() -> Entry { + let history = GroupHistory { + cluster: ClusterId::new([240; 16]).unwrap(), + namespace: NamespaceId::new([241; 16]).unwrap(), + group: GroupId::new([242; 16]).unwrap(), + incarnation: IncarnationId::new([243; 16]).unwrap(), + }; + let route = GroupRoute { + history, + range: CellRange::new( + CellBound::Key(cell_routing_key( + history.namespace, + &CellId::new("\\\"".repeat(500)).unwrap(), + )), + CellBound::PositiveInfinity, + ) + .unwrap(), + epoch: RouteEpoch::new(1).unwrap(), + birth_digest: [244; 32], + schema_digest: [245; 32], + }; + let root = GroupManifest::genesis(route).unwrap(); + let hold:CellGroupArchiveHold=serde_json::from_value(serde_json::json!({"registry":vec![246;32],"id":3,"allocation":vec![247;32],"root":{"object":{"registry":vec![246;32],"generation":2,"digest":vec![248;32],"length":128},"marker":root}})).unwrap(); + hold.check_reference().unwrap(); + let survivors = std::array::from_fn(|i| Survivor { + node: [i as u8 + 1; 16], + disk: [i as u8 + 10; 16], + engine: [i as u8 + 20; 16], + process: [i as u8 + 30; 16], + physical_host: [i as u8 + 40; 16], + epoch: 1, + member: i as u64 + 1, + applied: root.position().into(), + publication_digest: [249; 32], + }); + Entry { + history: [250; 32], + record: [251; 32], + evidence: Evidence::Live(Box::new(LiveEvidence { + fixed: Publication { + root, + configuration_digest: [252; 32], + }, + archive: Archive { + hold, + configuration_digest: [252; 32], + }, + survivors, + })), + } +} + +#[test] +fn complete_group_receipt_keeps_original_index_row_bound() { + let entry = large_entry(); + assert!(encode(ENTRY_KIND, &entry, ENTRY_LIMIT).is_err()); + let (row, object) = entry.stored().unwrap(); + assert!(row.len() <= ENTRY_LIMIT); + let (digest, payload) = object.unwrap(); + assert_eq!(digest, hash(&payload)); + assert!(payload.len() <= PAGE_LIMIT); + let stored: StoredEntry = decode(ENTRY_KIND, &row, ENTRY_LIMIT).unwrap(); + assert!(matches!(stored.evidence,StoredEvidence::Live {object} if object==digest)); + let live: Box = decode(LIVE_KIND, &payload, PAGE_LIMIT).unwrap(); + assert_eq!(Evidence::Live(live), entry.evidence); + assert!(row.len() + payload.len() <= APPEND_LIMIT * ENTRY_LIMIT); +} + +#[test] +fn cancelled_receipt_uses_no_external_live_payload() { + let entry = Entry { + history: [1; 32], + record: [2; 32], + evidence: Evidence::Cancelled { operation: [3; 16] }, + }; + let (row, object) = entry.stored().unwrap(); + assert!(object.is_none()); + let stored: StoredEntry = decode(ENTRY_KIND, &row, ENTRY_LIMIT).unwrap(); + assert!(matches!(stored.evidence,StoredEvidence::Cancelled {operation} if operation==[3;16])); + assert!(decode::( + ENTRY_KIND, + &encode(ENTRY_KIND, &large_entry(), PAGE_LIMIT).unwrap(), + ENTRY_LIMIT + ) + .is_err()); +} diff --git a/crates/cellar/durability/raft/fleet_catalog/finalization/survivors.rs b/crates/cellar/durability/raft/fleet_catalog/finalization/survivors.rs index 7d830fa76..41e2a71cd 100644 --- a/crates/cellar/durability/raft/fleet_catalog/finalization/survivors.rs +++ b/crates/cellar/durability/raft/fleet_catalog/finalization/survivors.rs @@ -93,6 +93,19 @@ pub(super) async fn refresh_terminal( return Err(FleetCatalogError::Conflict); } } + Evidence::Retired { + operation, + retirement, + } => { + if *history.state() + != (FleetHistoryState::Retired { + operation, + retirement, + }) + { + return Err(FleetCatalogError::Conflict); + } + } Evidence::Live(live) => { let observer = verification .observer @@ -137,9 +150,7 @@ pub(super) async fn refresh_terminal( let observations: [RaftPeerFleetMembershipObservation; 3] = observations .try_into() .map_err(|_| FleetCatalogError::Corrupt("terminal survivor count"))?; - let current = observations[0] - .published_cut() - .ok_or(FleetCatalogError::Unconfirmed)?; + let current = observations[0].published_cut(); let fixed = live.fixed.restore(current)?; catalog .run(deadline.remaining()?, async { @@ -212,9 +223,7 @@ pub(super) async fn verify( let proof = observed.membership(); let replica = proof.replica(); let node = observed.node(); - let cut = observed - .published_cut() - .ok_or(FleetCatalogError::Unconfirmed)?; + let cut = observed.published_cut(); let record = records .iter() .find(|record| record.node == replica.node.bytes()) @@ -295,7 +304,7 @@ pub(super) async fn verify( epoch: auth.epoch(), member: replica.member.get(), applied: proof.applied().into(), - publication_digest: hash(&encode(57, &Publication::from(cut), 4096)?), + publication_digest: hash(&encode(57, &Publication::from(cut), PAGE_LIMIT)?), }); } if snapshot.replica_separation() == FleetReplicaSeparation::PhysicalHosts && hosts.len() != 3 { diff --git a/crates/cellar/durability/raft/fleet_catalog/finalization/terminal.rs b/crates/cellar/durability/raft/fleet_catalog/finalization/terminal.rs index a1a5c33d9..a9d7414e9 100644 --- a/crates/cellar/durability/raft/fleet_catalog/finalization/terminal.rs +++ b/crates/cellar/durability/raft/fleet_catalog/finalization/terminal.rs @@ -388,7 +388,7 @@ impl FleetCatalog { snapshot: &FleetSnapshot, request: &FleetRemovalRequest, verification: Option<&mut FleetEvacuationVerification>, - namespaces: &[NamespaceCatalog], + namespaces: &[RangeDirectory], timeout: Duration, ) -> Result { self.complete_evacuation_attempt(snapshot, request, verification, namespaces, timeout) @@ -401,7 +401,7 @@ impl FleetCatalog { snapshot: &FleetSnapshot, request: &FleetRemovalRequest, verification: Option<&mut FleetEvacuationVerification>, - namespaces: &[NamespaceCatalog], + namespaces: &[RangeDirectory], timeout: Duration, ) -> Result> { match self @@ -437,7 +437,7 @@ impl FleetCatalog { snapshot: &FleetSnapshot, request: &FleetIsolatedRemovalRequest, verification: Option<&mut FleetEvacuationVerification>, - namespaces: &[NamespaceCatalog], + namespaces: &[RangeDirectory], timeout: Duration, ) -> Result { let completed = self @@ -466,7 +466,7 @@ impl FleetCatalog { request: &FleetRequestIdentity, proof: RemovalProof, verification: Option<&mut FleetEvacuationVerification>, - namespaces: &[NamespaceCatalog], + namespaces: &[RangeDirectory], timeout: Duration, ) -> Result> { self.check_snapshot(snapshot)?; diff --git a/crates/cellar/durability/raft/fleet_catalog/histories.rs b/crates/cellar/durability/raft/fleet_catalog/histories.rs index 5a126e587..decafd963 100644 --- a/crates/cellar/durability/raft/fleet_catalog/histories.rs +++ b/crates/cellar/durability/raft/fleet_catalog/histories.rs @@ -3,21 +3,48 @@ //! empty runtime scan proves that those references can be released. use super::*; use crate::raft::catalog::{ - BirthDisposition, BirthProposal, CatalogError, GroupBirth, InitialMember, ManagedBirthOwner, - NamespaceCatalog, NamespaceConfig, VoterPolicy, + CatalogError, InitialMember, ManagedBirthOwner, NamespaceCatalog, NamespaceConfig, VoterPolicy, +}; +use crate::raft::cell_groups::{ + CellGroupBirth, CellGroupBirthRequest, CellGroupBirthSpec, RangeDirectory, }; use crate::raft::member_state::FleetGroupBinding; -use cellar_logic::consensus::{GroupScope, MemberId, NodeId, ProposalId}; +use cellar_logic::{ + cell_group::{CellBound, CellId, CellRange, GroupHistory, GroupRoute, RouteEpoch}, + consensus::{MemberId, NodeId, ProposalId}, +}; use serde::Deserialize; use std::collections::BTreeSet; const HISTORY_KIND: u8 = 48; +#[cfg(test)] +#[path = "histories_tests.rs"] +mod tests; + +fn bound_key(bound: &CellBound) -> Option { + match bound { + CellBound::Key(key) => Some(key.cell().as_str().into()), + CellBound::NegativeInfinity | CellBound::PositiveInfinity => None, + } +} + +fn required_bound<'de, D: serde::Deserializer<'de>>( + decoder: D, +) -> std::result::Result, D::Error> { + Option::deserialize(decoder) +} + #[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] pub enum FleetHistoryState { Reserved, /// The immutable birth was confirmed; this does not attest to live quorum. Live, + /// The transfer closed. Any retained engine occupancy still counts. + Retired { + operation: [u8; 16], + retirement: [u8; 32], + }, Cancelled { operation: [u8; 16], }, @@ -43,6 +70,12 @@ pub struct FleetHistory { pub(super) schema_digest: [u8; 32], pub(super) namespace_root: [u8; 32], pub(super) birth_digest: [u8; 32], + #[serde(deserialize_with = "required_bound")] + lower: Option, + #[serde(deserialize_with = "required_bound")] + upper: Option, + epoch: u64, + creation: [u8; 16], pub(super) admission: [u8; 16], pub(super) operation: [u8; 16], pub(super) voters: Vec, @@ -52,21 +85,96 @@ pub struct FleetHistory { } impl FleetHistory { + pub(super) fn reserved_group( + binding: FleetGroupBinding, + directory: &RangeDirectory, + proposal: &CellGroupBirthRequest, + request: &FleetRequestIdentity, + ) -> Result { + if !binding.matches_scope(&proposal.route().history) { + return Err(FleetCatalogError::Conflict); + } + let namespace = directory.namespace(); + let history = Self { + binding, + schema_digest: namespace.config().schema_digest, + namespace_root: namespace.root_digest(), + birth_digest: proposal.route().birth_digest, + lower: bound_key(proposal.route().range.lower()), + upper: bound_key(proposal.route().range.upper()), + epoch: proposal.route().epoch.get(), + creation: proposal.creation().bytes(), + admission: request.id, + operation: request.operation, + voters: proposal + .voters() + .iter() + .copied() + .map(Member::from) + .collect(), + learners: vec![], + state: FleetHistoryState::Reserved, + origin: Origin::Managed, + }; + history.validate()?; + Ok(history) + } + + pub(super) fn cancelled_group( + binding: FleetGroupBinding, + directory: &RangeDirectory, + request: &CellGroupBirthRequest, + admission: ProposalId, + operation: ProposalId, + cancellation: ProposalId, + ) -> Result { + let decoded = directory + .decode_group_birth(request.encode()) + .map_err(catalog_error)?; + if !binding.matches_scope(&decoded.route().history) { + return Err(FleetCatalogError::Conflict); + } + let value = Self { + binding, + schema_digest: directory.namespace_config().schema_digest, + namespace_root: directory.namespace().root_digest(), + birth_digest: decoded.route().birth_digest, + lower: bound_key(decoded.route().range.lower()), + upper: bound_key(decoded.route().range.upper()), + epoch: decoded.route().epoch.get(), + creation: decoded.creation().bytes(), + admission: admission.bytes(), + operation: operation.bytes(), + voters: decoded.voters().iter().copied().map(Member::from).collect(), + learners: decoded + .learners() + .iter() + .copied() + .map(Member::from) + .collect(), + state: FleetHistoryState::Cancelled { + operation: cancellation.bytes(), + }, + origin: Origin::Observed, + }; + value.validate()?; + Ok(value) + } + /// An exact observed birth is import material, not proof that the namespace /// inventory is complete or that legacy creation has been quiesced. pub fn observed( binding: FleetGroupBinding, - namespace: &NamespaceCatalog, - birth: &GroupBirth, + directory: &RangeDirectory, + birth: &CellGroupBirth, admission: ProposalId, operation: ProposalId, ) -> Result { - namespace.check_birth(birth).map_err(catalog_error)?; - if !binding.matches_scope(birth.scope()) - || birth.managed_owner().is_some_and(|owner| { - owner.fleet != binding.fleet() || owner.catalog != binding.catalog() - }) - { + let namespace = directory.namespace(); + let decoded = directory + .decode_group_birth(birth.encode()) + .map_err(catalog_error)?; + if !binding.matches_scope(&birth.route().history) || decoded.route() != birth.route() { return Err(FleetCatalogError::Conflict); } if namespace @@ -83,7 +191,11 @@ impl FleetHistory { binding, schema_digest: namespace.config().schema_digest, namespace_root: namespace.root_digest(), - birth_digest: birth.digest(), + birth_digest: birth.route().birth_digest, + lower: bound_key(birth.route().range.lower()), + upper: bound_key(birth.route().range.upper()), + epoch: birth.route().epoch.get(), + creation: birth.creation().bytes(), admission: admission.bytes(), operation: operation.bytes(), voters: birth.voters().iter().copied().map(Member::from).collect(), @@ -127,6 +239,7 @@ impl FleetHistory { } pub(super) fn validate(&self) -> Result<()> { + self.initial_placement()?; self.binding .scope() .map_err(|_| FleetCatalogError::Corrupt("history binding"))?; @@ -138,6 +251,18 @@ impl FleetHistory { ProposalId::new(operation) .map_err(|_| FleetCatalogError::Corrupt("birth cancellation"))?; } + if let FleetHistoryState::Retired { + operation, + retirement, + } = self.state + { + if ProposalId::new(operation).is_err() + || retirement == [0; 32] + || self.origin != Origin::Managed + { + return Err(FleetCatalogError::Corrupt("history retirement")); + } + } if self.schema_digest == [0; 32] || self.namespace_root == [0; 32] || self.birth_digest == [0; 32] @@ -163,12 +288,39 @@ impl FleetHistory { } /// Recorded genesis references, never a current-membership or absence proof. - pub fn initial_placement(&self) -> Result { - Ok(BirthProposal { - scope: self - .binding - .scope() - .map_err(|_| FleetCatalogError::Corrupt("history scope"))?, + pub fn initial_placement(&self) -> Result { + let history = self + .binding + .scope() + .map_err(|_| FleetCatalogError::Corrupt("history scope"))?; + let key = |name: &str| { + CellId::new(name) + .map(|cell| { + crate::application::consensus::cell_routing_key(history.namespace, &cell) + }) + .map(CellBound::Key) + .map_err(|_| FleetCatalogError::Corrupt("history range")) + }; + let range = CellRange::new( + self.lower + .as_deref() + .map(key) + .transpose()? + .unwrap_or(CellBound::NegativeInfinity), + self.upper + .as_deref() + .map(key) + .transpose()? + .unwrap_or(CellBound::PositiveInfinity), + ) + .map_err(|_| FleetCatalogError::Corrupt("history range"))?; + Ok(CellGroupBirthSpec { + history, + range, + epoch: RouteEpoch::new(self.epoch) + .map_err(|_| FleetCatalogError::Corrupt("history route epoch"))?, + creation: ProposalId::new(self.creation) + .map_err(|_| FleetCatalogError::Corrupt("history creation"))?, voters: self .voters .iter() @@ -182,15 +334,26 @@ impl FleetHistory { }) } + // Multiple admitted candidates can share a range/epoch before the directory + // selects one. Every losing physical identity must remain inventoried too. pub(super) fn name_key(&self) -> Result { let scope = self .binding .scope() .map_err(|_| FleetCatalogError::Corrupt("history scope"))?; Ok(format!( - "birth/{}/{}", + "range/{}/{}", hex::encode(scope.namespace.bytes()), - hex::encode(hash(scope.cell.as_bytes())) + hex::encode(hash( + &serde_json::to_vec(&( + &self.lower, + &self.upper, + self.epoch, + scope.group.bytes(), + scope.incarnation.bytes(), + )) + .map_err(|_| FleetCatalogError::Corrupt("history range key"))? + )) )) } @@ -202,17 +365,39 @@ impl FleetHistory { Ok(format!("group/{}", hex::encode(scope.group.bytes()))) } - pub(crate) fn check_birth(&self, namespace: &NamespaceCatalog, birth: &GroupBirth) -> bool { - namespace.check_birth(birth).is_ok() - && namespace - .config() - .managed_fleet - .as_ref() - .is_none_or(|owner| owner.fleet == self.binding.fleet()) - && self.binding.matches_scope(birth.scope()) + pub(super) fn check_route(&self, directory: &RangeDirectory, route: &GroupRoute) -> bool { + let namespace = directory.namespace(); + namespace + .config() + .managed_fleet + .as_ref() + .is_none_or(|owner| owner.fleet == self.binding.fleet()) + && self.binding.matches_scope(&route.history) && self.namespace_root == namespace.root_digest() && self.schema_digest == namespace.config().schema_digest - && self.birth_digest == birth.digest() + && self.schema_digest == route.schema_digest + && self.birth_digest == route.birth_digest + && self.lower == bound_key(route.range.lower()) + && self.upper == bound_key(route.range.upper()) + && self.epoch == route.epoch.get() + } + + pub(crate) fn check_birth(&self, directory: &RangeDirectory, birth: &CellGroupBirth) -> bool { + directory.decode_group_birth(birth.encode()).is_ok() + && self.check_route(directory, birth.route()) + && self.creation == birth.creation().bytes() + && self.voters.len() == birth.voters().len() + && self.learners.len() == birth.learners().len() + && self + .voters + .iter() + .zip(birth.voters()) + .all(|(a, b)| a.member == b.member.get() && a.node == b.node.bytes()) + && self + .learners + .iter() + .zip(birth.learners()) + .all(|(a, b)| a.member == b.member.get() && a.node == b.node.bytes()) } } @@ -241,20 +426,38 @@ impl Member { #[derive(Clone, Debug)] pub struct FleetBirthAuthorization { history: FleetHistory, - proposal: BirthProposal, + proposal: CellGroupBirthRequest, owner: ManagedBirthOwner, } impl FleetBirthAuthorization { - pub(crate) fn check_birth(&self, namespace: &NamespaceCatalog, birth: &GroupBirth) -> bool { - self.history.check_birth(namespace, birth) + pub(crate) fn check_birth(&self, directory: &RangeDirectory, birth: &CellGroupBirth) -> bool { + self.history.check_birth(directory, birth) } - pub fn scope(&self) -> &GroupScope { - &self.proposal.scope + pub(super) fn matches_active_reservation(&self, history: &FleetHistory) -> bool { + match history.state() { + FleetHistoryState::Reserved => history == &self.history, + FleetHistoryState::Live => { + let mut expected = self.history.clone(); + expected.state = FleetHistoryState::Live; + history == &expected + } + FleetHistoryState::Cancelled { .. } | FleetHistoryState::Retired { .. } => false, + } } - pub(crate) fn proposal(&self) -> BirthProposal { + pub fn scope(&self) -> &GroupHistory { + &self.proposal.route().history + } + + /// Exact reserved metadata for the first range publication. Selection still + /// requires the directory CAS; this grants no member or quorum authority. + pub fn route(&self) -> &GroupRoute { + self.proposal.route() + } + + pub(crate) fn proposal(&self) -> CellGroupBirthRequest { self.proposal.clone() } @@ -330,25 +533,22 @@ impl FleetCatalog { &self, snapshot: &FleetSnapshot, request: &FleetRequestIdentity, - namespace: &NamespaceCatalog, - birth: &GroupBirth, + directory: &RangeDirectory, + birth: &CellGroupBirth, + class: &str, timeout: Duration, ) -> Result { self.check_snapshot(snapshot)?; validate_request(request)?; - if let Some(owner) = birth.managed_owner() { - if self.birth_owner()? != *owner { - return Err(FleetCatalogError::Conflict); - } - } + let namespace = directory.namespace(); if namespace.config().managed_fleet.is_some() && self.bind_namespace(namespace.config().clone())? != *namespace.config() { return Err(FleetCatalogError::Conflict); } let history = FleetHistory::observed( - self.binding(birth.scope())?, - namespace, + self.binding(&birth.route().history)?, + directory, birth, ProposalId::new(request.id).map_err(|_| FleetCatalogError::Invalid("admission"))?, ProposalId::new(request.operation) @@ -356,7 +556,14 @@ impl FleetCatalog { )?; self.run( timeout, - self.insert_history(snapshot, request, &history, namespace, false), + self.insert_history( + snapshot, + request, + &history, + directory, + class, + HistoryAdmission::Observed, + ), ) .await } @@ -365,12 +572,35 @@ impl FleetCatalog { &self, snapshot: &FleetSnapshot, request: &FleetRequestIdentity, - namespace: &NamespaceCatalog, - proposal: BirthProposal, + directory: &RangeDirectory, + proposal: CellGroupBirthSpec, + class: &str, + timeout: Duration, + ) -> Result { + self.reserve_birth_with_intent( + snapshot, + request, + directory, + (proposal, None), + class, + timeout, + ) + .await + } + + pub(super) async fn reserve_birth_with_intent( + &self, + snapshot: &FleetSnapshot, + request: &FleetRequestIdentity, + directory: &RangeDirectory, + setup: (CellGroupBirthSpec, Option<[u8; 32]>), + class: &str, timeout: Duration, ) -> Result { + let (proposal, intent) = setup; self.check_snapshot(snapshot)?; validate_request(request)?; + let namespace = directory.namespace(); if namespace.config().cluster.bytes() != self.inner.cluster || namespace.config().voters != VoterPolicy::ThreeVoters || proposal.voters.len() != 3 @@ -379,25 +609,25 @@ impl FleetCatalog { return Err(FleetCatalogError::Invalid("managed initial placement")); } self.run(timeout, async { - let owner = self.managed_owner_for_birth(snapshot, namespace).await?; - let (proposal, digest) = namespace - .prepare_birth_with_owner(proposal, Some(&owner)) + let owner = self.managed_owner_for_birth(snapshot, directory).await?; + let proposal = directory + .prepare_fleet_group_birth(proposal, &owner) .map_err(catalog_error)?; - let history = FleetHistory { - binding: self.binding(&proposal.scope)?, - schema_digest: namespace.config().schema_digest, - namespace_root: namespace.root_digest(), - birth_digest: digest, - admission: request.id, - operation: request.operation, - voters: proposal.voters.iter().copied().map(Member::from).collect(), - learners: vec![], - state: FleetHistoryState::Reserved, - origin: Origin::Managed, - }; - history.validate()?; + let history = FleetHistory::reserved_group( + self.binding(&proposal.route().history)?, + directory, + &proposal, + request, + )?; let commit = self - .insert_history(snapshot, request, &history, namespace, true) + .insert_history( + snapshot, + request, + &history, + directory, + class, + HistoryAdmission::Reserved(intent), + ) .await?; Ok(FleetBirthReservation { commit, @@ -418,11 +648,24 @@ impl FleetCatalog { pub async fn recover_birth_reservation( &self, snapshot: &FleetSnapshot, - namespace: &NamespaceCatalog, - scope: &GroupScope, + directory: &RangeDirectory, + scope: &GroupHistory, + timeout: Duration, + ) -> Result { + self.recover_birth_reservation_matching(snapshot, directory, scope, None, timeout) + .await + } + + pub(super) async fn recover_birth_reservation_matching( + &self, + snapshot: &FleetSnapshot, + directory: &RangeDirectory, + scope: &GroupHistory, + expected: Option<(&FleetRequestIdentity, [u8; 32])>, timeout: Duration, ) -> Result { self.check_snapshot(snapshot)?; + let namespace = directory.namespace(); let binding = self.binding(scope)?; let key = binding .key() @@ -454,26 +697,11 @@ impl FleetCatalog { principal: mutation.principal.clone(), expected_revision: mutation.expected_revision, }; - let expected = history_mutation( - &request, - vec![ - FleetChange { - table: FleetTable::Histories, - key: key.clone(), - value: Some(history.encode()?), - }, - FleetChange { - table: FleetTable::Policy, - key: history.name_key()?, - value: Some(key.as_bytes().to_vec()), - }, - FleetChange { - table: FleetTable::Policy, - key: history.group_key()?, - value: Some(key.as_bytes().to_vec()), - }, - ], - ); + let intent = self.first_group_intent(snapshot, &request.id).await?; + if expected.is_some_and(|(saved, digest)| saved != &request || Some(digest) != intent) { + return Err(FleetCatalogError::PayloadConflict); + } + let expected = birth_mutation(&history, &request, &key, intent)?; if mutation.id != history.admission || mutation.operation != history.operation || mutation != expected @@ -491,10 +719,11 @@ impl FleetCatalog { if applied.is_none() || applied != request.expected_revision.checked_add(1) { return Err(FleetCatalogError::Corrupt("unadmitted birth reservation")); } - let owner = self.managed_owner_for_birth(snapshot, namespace).await?; - let (proposal, digest) = namespace - .prepare_birth_with_owner(history.initial_placement()?, Some(&owner)) + let owner = self.managed_owner_for_birth(snapshot, directory).await?; + let proposal = directory + .prepare_fleet_group_birth(history.initial_placement()?, &owner) .map_err(catalog_error)?; + let digest = proposal.route().birth_digest; let authorization = FleetBirthAuthorization { history, proposal, @@ -517,12 +746,13 @@ impl FleetCatalog { &self, snapshot: &FleetSnapshot, request: &FleetRequestIdentity, - namespace: &NamespaceCatalog, + directory: &RangeDirectory, authorization: &FleetBirthAuthorization, timeout: Duration, ) -> Result { self.check_snapshot(snapshot)?; validate_request(request)?; + let namespace = directory.namespace(); authorization .check_namespace(namespace) .map_err(catalog_error)?; @@ -530,27 +760,23 @@ impl FleetCatalog { return Err(FleetCatalogError::Conflict); } self.run(timeout, async { - if self.managed_owner_for_birth(snapshot, namespace).await? != authorization.owner { + if self.managed_owner_for_birth(snapshot, directory).await? != authorization.owner { return Err(FleetCatalogError::Conflict); } let mut history = authorization.history.clone(); - match namespace - .birth_disposition(&authorization.scope().cell) + let birth = self + .inner + .domain + .run(Box::pin( + directory.materialize_fleet_group_birth(authorization), + )) .await - .map_err(catalog_error)? - { - Some(BirthDisposition::Live(birth)) if history.check_birth(namespace, &birth) => { - history.state = FleetHistoryState::Live - } - Some(BirthDisposition::Cancelled { scope, operation }) - if scope == *authorization.scope() => - { - history.state = FleetHistoryState::Cancelled { - operation: operation.bytes(), - } - } - _ => return Err(FleetCatalogError::Conflict), + .map_err(|_| FleetCatalogError::Closed)? + .map_err(catalog_error)?; + if !history.check_birth(directory, &birth) { + return Err(FleetCatalogError::Conflict); } + history.state = FleetHistoryState::Live; let key = history .binding .key() @@ -585,7 +811,7 @@ impl FleetCatalog { pub async fn history( &self, snapshot: &FleetSnapshot, - scope: &GroupScope, + scope: &GroupHistory, timeout: Duration, ) -> Result { self.check_snapshot(snapshot)?; @@ -633,14 +859,71 @@ impl FleetCatalog { Ok(history) } + pub(super) async fn first_group_intent( + &self, + snapshot: &FleetSnapshot, + request: &[u8; 16], + ) -> Result> { + self.inner + .lookup( + snapshot.head.tables[FleetTable::Policy.slot()], + &first_group_intent_key(request), + ) + .await? + .map(|value| { + value + .try_into() + .map_err(|_| FleetCatalogError::Corrupt("first group intent")) + }) + .transpose() + } + + pub(super) async fn require_unoccupied_birth( + &self, + snapshot: &FleetSnapshot, + history: &FleetHistory, + key: &str, + ) -> Result<()> { + if snapshot.head.history_count >= HISTORY_LIMIT { + return Err(FleetCatalogError::Limit); + } + if self + .inner + .lookup(snapshot.head.tables[FleetTable::Histories.slot()], key) + .await? + .is_some() + || self + .inner + .lookup( + snapshot.head.tables[FleetTable::Policy.slot()], + &history.name_key()?, + ) + .await? + .is_some() + || self + .inner + .lookup( + snapshot.head.tables[FleetTable::Policy.slot()], + &history.group_key()?, + ) + .await? + .is_some() + { + return Err(FleetCatalogError::Conflict); + } + Ok(()) + } + async fn insert_history( &self, snapshot: &FleetSnapshot, request: &FleetRequestIdentity, history: &FleetHistory, - namespace: &NamespaceCatalog, - placement: bool, + directory: &RangeDirectory, + class: &str, + admission: HistoryAdmission, ) -> Result { + let namespace = directory.namespace(); let key = history .binding .key() @@ -652,26 +935,11 @@ impl FleetCatalog { if self.binding(&scope)? != history.binding { return Err(FleetCatalogError::Conflict); } - let mutation = history_mutation( - request, - vec![ - FleetChange { - table: FleetTable::Histories, - key: key.clone(), - value: Some(history.encode()?), - }, - FleetChange { - table: FleetTable::Policy, - key: history.name_key()?, - value: Some(key.as_bytes().to_vec()), - }, - FleetChange { - table: FleetTable::Policy, - key: history.group_key()?, - value: Some(key.as_bytes().to_vec()), - }, - ], - ); + let intent = match admission { + HistoryAdmission::Observed => None, + HistoryAdmission::Reserved(intent) => intent, + }; + let mutation = birth_mutation(history, request, &key, intent)?; mutation.validate()?; let current = self.inner.current().await?; let digest = hash(&encode(REQUEST_KIND, &mutation, REQUEST_LIMIT)?); @@ -681,44 +949,14 @@ impl FleetCatalog { .await? .is_none() { - let class = scope - .cell - .split_once(':') - .ok_or(FleetCatalogError::Invalid("history class"))? - .0; self.require_namespace_declaration(snapshot, namespace, Some(class)) .await?; if current.token != snapshot.token || request.expected_revision != snapshot.revision() { return Err(FleetCatalogError::Conflict); } - if snapshot.head.history_count >= HISTORY_LIMIT { - return Err(FleetCatalogError::Limit); - } - if self - .inner - .lookup(snapshot.head.tables[FleetTable::Histories.slot()], &key) - .await? - .is_some() - || self - .inner - .lookup( - snapshot.head.tables[FleetTable::Policy.slot()], - &history.name_key()?, - ) - .await? - .is_some() - || self - .inner - .lookup( - snapshot.head.tables[FleetTable::Policy.slot()], - &history.group_key()?, - ) - .await? - .is_some() - { - return Err(FleetCatalogError::Conflict); - } - if placement { + self.require_unoccupied_birth(snapshot, history, &key) + .await?; + if let HistoryAdmission::Reserved(_) = admission { let nodes = self.node_records(snapshot).await?; let mut hosts = BTreeSet::new(); for member in &history.voters { @@ -735,13 +973,33 @@ impl FleetCatalog { { return Err(FleetCatalogError::Conflict); } + // Active is a placement hint. An ordinary retained boot + // invalidates the prior capacity receipt without erasing + // that state or its existing group ownership. + let target = NodeId::new(member.node) + .map_err(|_| FleetCatalogError::Corrupt("birth capacity node"))?; + if !self + .capacity_from(snapshot, target) + .await? + .receive_replicas() + { + return Err(FleetCatalogError::Conflict); + } } // This claim grants no birth authority. Confirming it before // the root CAS closes races with other namespace catalogs that // share the global group-ID key, including unmanaged imports. - namespace - .reserve_group_identity(&scope) + let owner = self.birth_owner()?; + let proposal = directory + .prepare_fleet_group_birth(history.initial_placement()?, &owner) + .map_err(catalog_error)?; + self.inner + .domain + .run(Box::pin( + directory.reserve_fleet_group_identity(&proposal, &owner), + )) .await + .map_err(|_| FleetCatalogError::Closed)? .map_err(catalog_error)?; } } @@ -749,6 +1007,48 @@ impl FleetCatalog { } } +enum HistoryAdmission { + Observed, + Reserved(Option<[u8; 32]>), +} + +fn first_group_intent_key(request: &[u8; 16]) -> String { + format!("first-group-intents/{}", hex::encode(request)) +} + +pub(super) fn birth_mutation( + history: &FleetHistory, + request: &FleetRequestIdentity, + key: &str, + intent: Option<[u8; 32]>, +) -> Result { + let mut changes = vec![ + FleetChange { + table: FleetTable::Histories, + key: key.into(), + value: Some(history.encode()?), + }, + FleetChange { + table: FleetTable::Policy, + key: history.name_key()?, + value: Some(key.as_bytes().to_vec()), + }, + FleetChange { + table: FleetTable::Policy, + key: history.group_key()?, + value: Some(key.as_bytes().to_vec()), + }, + ]; + if let Some(digest) = intent { + changes.push(FleetChange { + table: FleetTable::Policy, + key: first_group_intent_key(&request.id), + value: Some(digest.to_vec()), + }); + } + Ok(history_mutation(request, changes)) +} + pub(super) fn history_mutation( request: &FleetRequestIdentity, changes: Vec, @@ -782,6 +1082,7 @@ pub(super) fn catalog_error(error: CatalogError) -> FleetCatalogError { CatalogError::Limit => FleetCatalogError::Limit, CatalogError::Unconfirmed => FleetCatalogError::Unconfirmed, CatalogError::Storage(error) => FleetCatalogError::Storage(error.into()), + CatalogError::FleetAuthority(error) => *error, _ => FleetCatalogError::Conflict, } } diff --git a/crates/cellar/durability/raft/fleet_catalog/histories_tests.rs b/crates/cellar/durability/raft/fleet_catalog/histories_tests.rs new file mode 100644 index 000000000..27b4347be --- /dev/null +++ b/crates/cellar/durability/raft/fleet_catalog/histories_tests.rs @@ -0,0 +1,104 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use cellar_logic::consensus::{ClusterId, GroupId, NamespaceId}; + +#[test] +fn whole_group_history_retains_range_and_original_creation() { + let history = fixture(); + let restored = FleetHistory::decode(&history.encode().unwrap()).unwrap(); + let placement = restored.initial_placement().unwrap(); + assert_eq!(placement.history, history.binding.scope().unwrap()); + assert_eq!(placement.creation.bytes(), history.creation); + assert_eq!(placement.epoch.get(), 7); + assert_eq!(placement.voters.len(), 3); + assert_eq!( + bound_key(placement.range.lower()).as_deref(), + history.lower.as_deref() + ); + assert_eq!( + bound_key(placement.range.upper()).as_deref(), + history.upper.as_deref() + ); + let mut changed = restored; + changed.upper = Some("Counter:zz".into()); + assert_ne!(history.name_key().unwrap(), changed.name_key().unwrap()); + assert_eq!(history.group_key().unwrap(), changed.group_key().unwrap()); +} + +#[test] +fn pending_groups_with_the_same_range_keep_distinct_inventory_keys() { + let first = fixture(); + let mut second = first.clone(); + let mut history = second.binding.scope().unwrap(); + history.group = GroupId::new([13; 16]).unwrap(); + history.incarnation = IncarnationId::new([14; 16]).unwrap(); + second.binding = FleetGroupBinding::new( + IncarnationId::new(first.binding.fleet()).unwrap(), + &history, + first.binding.catalog(), + ) + .unwrap(); + assert_eq!(first.lower, second.lower); + assert_eq!(first.upper, second.upper); + assert_eq!(first.epoch, second.epoch); + assert_ne!(first.name_key().unwrap(), second.name_key().unwrap()); + assert_ne!(first.group_key().unwrap(), second.group_key().unwrap()); +} + +#[test] +fn history_refuses_missing_bounds_and_invalid_ranges() { + let history = fixture(); + for field in ["lower", "upper", "epoch", "creation"] { + let mut value = serde_json::to_value(&history).unwrap(); + value.as_object_mut().unwrap().remove(field); + let bytes = encode(HISTORY_KIND, &value, 4096).unwrap(); + assert!( + FleetHistory::decode(&bytes).is_err(), + "accepted missing {field}" + ); + } + let mut inverted = history.clone(); + inverted.lower = history.upper; + inverted.upper = history.lower; + assert!(inverted.encode().is_err()); +} + +fn fixture() -> FleetHistory { + let namespace = NamespaceId::new([2; 16]).unwrap(); + let mut names = ["Counter:a", "Counter:z"]; + names.sort_by_key(|name| { + crate::application::consensus::cell_routing_key(namespace, &CellId::new(name).unwrap()) + }); + FleetHistory { + binding: FleetGroupBinding::new( + IncarnationId::new([5; 16]).unwrap(), + &GroupHistory { + cluster: ClusterId::new([1; 16]).unwrap(), + namespace, + group: GroupId::new([3; 16]).unwrap(), + incarnation: IncarnationId::new([4; 16]).unwrap(), + }, + [6; 32], + ) + .unwrap(), + schema_digest: [7; 32], + namespace_root: [8; 32], + birth_digest: [9; 32], + lower: Some(names[0].into()), + upper: Some(names[1].into()), + epoch: 7, + creation: [10; 16], + admission: [11; 16], + operation: [12; 16], + voters: (1..=3) + .map(|member| Member { + member, + node: [member as u8; 16], + }) + .collect(), + learners: vec![], + state: FleetHistoryState::Reserved, + origin: Origin::Managed, + } +} diff --git a/crates/cellar/durability/raft/fleet_catalog/imports/README.md b/crates/cellar/durability/raft/fleet_catalog/imports/README.md index 1340cc905..381cee1dd 100644 --- a/crates/cellar/durability/raft/fleet_catalog/imports/README.md +++ b/crates/cellar/durability/raft/fleet_catalog/imports/README.md @@ -1,24 +1,34 @@ # Namespace import runtime -This component imports one authoritative frozen namespace admission set. Fleet +This component imports one authoritative frozen physical-group admission set. Fleet capacity activation, global namespace coverage, maintenance/removal certificates, automatic reconciliation and operator surfaces remain separate delivery work. Keep automatic fleet management default-off until its qualification gates pass. -The namespace's immutable configuration and schema stay fixed. The mutable birth -inventory is claimed by one exact fleet/catalog scope, then adopted only after -every frozen intent is settled and every live group has committed enrollment. -Business writes continue through their existing Raft acknowledgement path. +The namespace's immutable configuration and schema stay fixed. Call +`RangeDirectory::freeze_group_births` with a persisted operation identity. The +namespace admission CAS first closes new group intents and arbitrates against +disaster recovery. The directory-head CAS then locks topology; a writer holding +an earlier expected head loses. The exact locked directory is retained by digest. +Retrying that operation recovers the same cut after a lost reply or reopen. + +The cut includes every admitted physical request, including never-served, +unselected reservations. An unadmitted global identity claim remains permanently +reserved and grants no birth or voter permission. Fleet import claims this exact +cut under one catalog scope, then adopts only after every intent is settled and +every live group has committed enrollment. Business writes continue through their +existing Raft acknowledgement path. This does not import predecessor per-cell +inventories or create a compatibility path. ## Durable steps | Root phase | Effect and admission boundary | | --- | --- | | `Preparing` | `begin_namespace_import` roots the exact cut, operation and catalog owner. The next step conditionally claims that same frozen namespace. A competing catalog can retain a harmless Preparing intent but cannot replace the winner's claim. | -| `Inventory` | Traverse the rooted cursor, one canonical intent at a time. Reconcile the immutable birth decision; conditionally cancel a still-pending decision. Root the history and permanent name/group references with progress in one CAS. Existing exact histories retain their original admission identity. | -| `Coverage` | Traverse the same cut again. Cancelled histories retain their tombstones. Each live history requires its own opaque safe-ReadIndex observation with the exact catalog binding, birth/schema, stable RF3 configuration, committed guard and no active repair. Verify current voter enrollment, engine pins and permanent member/disk claims. | -| `Handoff` | The last coverage CAS admits irreversible handoff. Before changing namespace ownership, read and validate the complete retained witness set against the canonical cut, actual histories and immutable decisions/claims. Unresolved namespace admissions block handoff. | -| `Complete` | After namespace adoption is confirmed, CAS the terminal root state. Only this state plus the exact namespace adoption marker permits new managed birth reservations in the imported namespace. | +| `Inventory` | Traverse the rooted cursor, one canonical intent at a time. Reconcile the immutable birth decision; conditionally cancel an unselected pending group at the same exact key that materialization uses. A selected lazy birth stays live. Root the history and permanent range/group references with progress in one CAS. Existing exact histories retain their original admission identity. | +| `Coverage` | Traverse the same cut again. Cancelled histories retain their tombstones. Each live physical history requires its own opaque `CellGroupQuorumMembership` safe-ReadIndex observation with the exact catalog binding, birth/schema, stable RF3 configuration, committed guard and no active repair. Verify current voter enrollment, engine pins and permanent member/disk claims. | +| `Handoff` | The last coverage CAS admits irreversible handoff. Before changing namespace ownership, read and validate the complete retained witness set against the canonical cut, actual histories and immutable decisions/claims. Independently traverse every frozen selected range and require its exact admission and coverage witness. Unresolved namespace admissions block handoff. | +| `Complete` | After namespace adoption is confirmed, unlock topology with another CAS, then CAS the terminal fleet-root state. Both head changes advance revision, so an old expected head cannot become current again. Only this state plus the exact namespace adoption marker permits new managed birth reservations in the imported namespace. | Progress comes from the fleet root, never a caller-provided manifest, cursor or reachable-node scan. Every turn is deadline-bound and owns one bounded catalog @@ -48,16 +58,19 @@ admission. An unadmitted stale candidate cannot overwrite a later root revision. Imported original births retain their original unmanaged birth identity. Their enrollment is a committed Raft configuration entry, which intact original engines -can replay during recovery. New births after adoption carry an explicit managed -owner in their canonical birth record. They require an inventoried reservation -and initialize with their fleet guard in the first engine sync, even though the -namespace's immutable `managed_fleet` configuration remains absent. +can replay during recovery. New births after adoption require a rooted fleet reservation and the exact +adopted catalog owner; ordinary birth and bootstrap calls refuse that namespace. +They initialize with their configuration guard in the first engine sync, even +though the immutable `managed_fleet` configuration remains absent. Observed +histories may only resume an existing matching local guard: import does not turn +a missing disk into permission to recreate genesis. A selected lazy group without +an enrolled quorum therefore blocks completion; timeout does not cancel it. Group bindings now include the catalog-scope fingerprint as well as fleet and -cell-history identity. Use `FleetCatalog::group_binding` for that descriptor. +physical group/incarnation identity. Use `FleetCatalog::group_binding` for that descriptor. Constructing a descriptor grants no authority. Changing catalog prefixes cannot reuse a cached authorization or permit to mutate the already-enrolled group. -The experimental protocol hard-cuts to `/fleet-quorum-1/fleet-cuts-1/empty-archives-1/archive-cuts-1/fleet-finalization-1`; earlier namespace, +The experimental protocol includes `/cell-group-fleet-1/group-import-1`; earlier namespace, fleet and group records require fresh experimental prefixes and matching binaries. Numeric format markers remain 1, with no migration or compatibility reader. diff --git a/crates/cellar/durability/raft/fleet_catalog/imports/coverage.rs b/crates/cellar/durability/raft/fleet_catalog/imports/coverage.rs index 15bdb4624..08300e517 100644 --- a/crates/cellar/durability/raft/fleet_catalog/imports/coverage.rs +++ b/crates/cellar/durability/raft/fleet_catalog/imports/coverage.rs @@ -1,29 +1,30 @@ use super::*; -use crate::raft::catalog::BirthDisposition; +use crate::raft::cell_groups::CellGroupBirthDisposition; +use crate::raft::host::FleetMembershipConfiguration; use crate::raft::member_state::FleetMembershipPermit; use records::{Decision, Voter, Witness}; use std::collections::BTreeSet; impl FleetCatalog { /// Confirm that every retained proof and history is still readable before - /// the irreversible namespace effect. Missing pages are never empty sets. + /// the irreversible directory effect. Missing pages are never empty sets. pub(super) async fn verify_complete_import( &self, snapshot: &FleetSnapshot, - namespace: &NamespaceCatalog, + directory: &RangeDirectory, record: &Record, ) -> Result<()> { - self.verify_import_coverage(snapshot, namespace, record, true) + self.verify_import_coverage(snapshot, directory, record, true) .await } pub(in crate::raft::fleet_catalog) async fn verify_imported_inventory( &self, snapshot: &FleetSnapshot, - namespace: &NamespaceCatalog, + directory: &RangeDirectory, ) -> Result<()> { let record = self - .import_record(snapshot, namespace) + .import_record(snapshot, directory) .await .map_err(|e| match e { FleetCatalogError::Conflict => FleetCatalogError::InventoryIncomplete, @@ -32,21 +33,63 @@ impl FleetCatalog { if record.phase != records::FleetImportPhase::Complete || record.stopped { return Err(FleetCatalogError::InventoryIncomplete); } - namespace - .check_birth_import(&record.cut, &self.birth_owner()?, true) + directory + .check_group_import(&record.cut, &self.birth_owner()?, true) .await .map_err(catalog_error)?; - self.verify_import_coverage(snapshot, namespace, &record, false) + self.verify_import_coverage(snapshot, directory, &record, false) .await } async fn verify_import_coverage( &self, snapshot: &FleetSnapshot, - namespace: &NamespaceCatalog, + directory: &RangeDirectory, record: &Record, handoff: bool, ) -> Result<()> { + directory + .check_group_import(&record.cut, &self.birth_owner()?, !handoff) + .await + .map_err(catalog_error)?; + let frozen = directory + .import_directory_view(&record.cut) + .await + .map_err(catalog_error)?; + let mut cursor = None; + let mut selected = 0u64; + loop { + let page = directory + .group_page(&frozen, cursor.as_ref(), 16) + .await + .map_err(catalog_error)?; + for route in page.routes { + selected = selected + .checked_add(1) + .filter(|n| *n <= record.cut.count) + .ok_or(FleetCatalogError::Corrupt( + "untracked selected import group", + ))?; + let key = hex::encode(hash(&route.history.group.bytes())); + let admitted = directory + .namespace() + .cell_group_import_entry(directory, &record.cut, &self.birth_owner()?, &key) + .await + .map_err(catalog_error)?; + if admitted.route() != &route + || self.inner.lookup(record.coverage, &key).await?.is_none() + { + return Err(FleetCatalogError::Corrupt("selected import group coverage")); + } + } + cursor = page.next; + if cursor.is_none() { + break; + } + } + if selected != u64::from(frozen.groups()) { + return Err(FleetCatalogError::Corrupt("missing selected import group")); + } let mut after = None; let mut seen = 0u64; loop { @@ -60,18 +103,24 @@ impl FleetCatalog { .filter(|n| *n <= record.cut.count) .ok_or(FleetCatalogError::Corrupt("extra import coverage"))?; let witness: Witness = decode(records::WITNESS_KIND, &value.value, 4096)?; - let entry = namespace - .import_birth_entry(&record.cut, &self.birth_owner()?, &value.key) + let entry = directory + .namespace() + .cell_group_import_entry( + directory, + &record.cut, + &self.birth_owner()?, + &value.key, + ) .await .map_err(catalog_error)?; - let scope = &entry.proposal().scope; + let scope = &entry.route().history; let binding = self.binding(scope)?; let history_key = binding .key() .map_err(|_| FleetCatalogError::Corrupt("import witness binding"))?; let history = self.history_inner(snapshot, &history_key).await?; if witness.history != history_key - || witness.birth != entry.expected_digest() + || witness.birth != entry.route().birth_digest || history.birth_digest != witness.birth || history.namespace_root != record.cut.root || history.schema_digest != record.schema @@ -80,15 +129,15 @@ impl FleetCatalog { } match ( witness.decision, - namespace - .birth_disposition(&scope.cell) + directory + .group_birth_disposition(&entry) .await .map_err(catalog_error)?, ) { ( Decision::Cancelled { operation }, - Some(BirthDisposition::Cancelled { - scope: actual, + Some(CellGroupBirthDisposition::Cancelled { + history: actual, operation: op, }), ) => { @@ -113,9 +162,9 @@ impl FleetCatalog { phase, voters, }, - Some(BirthDisposition::Live(birth)), + Some(CellGroupBirthDisposition::Live(birth)), ) if history.state == FleetHistoryState::Live - && history.check_birth(namespace, &birth) => + && history.check_birth(directory, &birth) => { let valid_id = |id| cellar_logic::consensus::IncarnationId::new(id).is_ok(); if !valid_id(node) @@ -164,19 +213,18 @@ impl FleetCatalog { { return Err(FleetCatalogError::Conflict); } - namespace + directory + .namespace() .verify_fleet_storage_pin(&host) .await .map_err(catalog_error)?; - namespace - .verify_import_member_claim( + directory + .verify_group_import_member_claim( &birth, cellar_logic::consensus::MemberId::new(voter.member).map_err( |_| FleetCatalogError::Corrupt("import witness member"), )?, - cellar_logic::consensus::DiskLineage::new(voter.disk).map_err( - |_| FleetCatalogError::Corrupt("import witness disk"), - )?, + &host, ) .await .map_err(catalog_error)?; @@ -200,7 +248,7 @@ impl FleetCatalog { return Err(FleetCatalogError::Corrupt("missing import coverage")); } if handoff { - self.no_namespace_admissions(snapshot, namespace.config().namespace) + self.no_namespace_admissions(snapshot, directory.namespace_config().namespace) .await } else { Ok(()) @@ -210,61 +258,77 @@ impl FleetCatalog { pub(super) async fn import_coverage_step( &self, snapshot: &FleetSnapshot, - namespace: &NamespaceCatalog, + directory: &RangeDirectory, record: &mut Record, - observation: Option<&FleetMembershipObservation>, + observation: Option<&CellGroupQuorumMembership>, ) -> Result> { - let step = namespace - .import_birth_step(&record.cut, &self.birth_owner()?, record.cursor.as_deref()) + directory + .check_group_import(&record.cut, &self.birth_owner()?, false) + .await + .map_err(catalog_error)?; + let step = directory + .namespace() + .cell_group_import_step( + directory, + &record.cut, + &self.birth_owner()?, + record.cursor.as_deref(), + ) .await .map_err(catalog_error)?; if let Some(entry) = step.entry { - let proposal = entry.proposal(); - let binding = self.binding(&proposal.scope)?; + let proposal = &entry; + let binding = self.binding(&proposal.route().history)?; let history_key = binding .key() .map_err(|_| FleetCatalogError::Corrupt("import binding key"))?; let history = self.history_inner(snapshot, &history_key).await?; if history.namespace_root != record.cut.root || history.schema_digest != record.schema - || history.birth_digest != entry.expected_digest() + || history.birth_digest != entry.route().birth_digest { return Err(FleetCatalogError::Conflict); } - let decision = match namespace - .birth_disposition(&proposal.scope.cell) + let decision = match directory + .group_birth_disposition(proposal) .await .map_err(catalog_error)? { - Some(BirthDisposition::Cancelled { scope, operation }) - if scope == proposal.scope - && history.state - == (FleetHistoryState::Cancelled { - operation: operation.bytes(), - }) - && observation.is_none() => + Some(CellGroupBirthDisposition::Cancelled { + history: scope, + operation, + }) if scope == proposal.route().history + && history.state + == (FleetHistoryState::Cancelled { + operation: operation.bytes(), + }) + && observation.is_none() => { Decision::Cancelled { operation: operation.bytes(), } } - Some(BirthDisposition::Live(birth)) + Some(CellGroupBirthDisposition::Live(birth)) if history.state == FleetHistoryState::Live - && history.check_birth(namespace, &birth) => + && history.check_birth(directory, &birth) => { let observed = observation.ok_or(FleetCatalogError::Invalid( "missing live import observation", ))?; - let status = observed.status(); - let conf = observed.configuration(); - if observed.replica().scope != proposal.scope + let status = observed + .membership() + .fleet_status() + .ok_or(FleetCatalogError::Conflict)?; + let conf = FleetMembershipConfiguration::capture(observed.configuration()) + .map_err(|_| FleetCatalogError::Corrupt("import configuration"))?; + if observed.replica().scope != proposal.route().history || status.binding != binding - || observed.birth_digest() != history.birth_digest - || observed.schema_digest() != record.schema - || observed.barrier().get() == 0 - || observed.applied().index < observed.barrier() - || observed.membership_revision() > observed.applied().index - || observed.membership_revision() < status.changed_at + || observed.root().route().birth_digest != history.birth_digest + || observed.root().route().schema_digest != record.schema + || observed.barrier().index.get() == 0 + || observed.root().position().index < observed.barrier().index + || observed.membership().revision() > observed.root().position().index + || observed.membership().revision() < status.changed_at || conf.voters().len() != 3 || !conf.outgoing_voters().is_empty() || !conf.learners().is_empty() @@ -272,6 +336,7 @@ impl FleetCatalog { || conf.auto_leave() || status.phase.is_some_and(|phase| phase.active()) || observed + .membership() .repair_status() .map_err(|_| FleetCatalogError::Corrupt("import repair status"))? .is_some_and(|repair| repair.phase.active()) @@ -283,15 +348,16 @@ impl FleetCatalog { let mut nodes = BTreeSet::new(); for member in conf.voters() { let (node, disk) = if let Some((joined, _)) = observed - .joined_member(*member) + .membership() + .joined(*member) .map_err(|_| FleetCatalogError::Corrupt("import joined member"))? { (joined.node, Some(joined.disk)) } else { let initial = proposal - .voters + .voters() .iter() - .chain(&proposal.learners) + .chain(proposal.learners()) .find(|m| m.member == *member) .ok_or(FleetCatalogError::Conflict)?; (initial.node, None) @@ -308,12 +374,13 @@ impl FleetCatalog { { return Err(FleetCatalogError::Conflict); } - namespace + directory + .namespace() .verify_fleet_storage_pin(&host) .await .map_err(catalog_error)?; - namespace - .verify_import_member_claim(&birth, *member, host.disk()) + directory + .verify_group_import_member_claim(&birth, *member, &host) .await .map_err(catalog_error)?; voters.push(Voter { @@ -328,6 +395,7 @@ impl FleetCatalog { } let source = self.host_from(snapshot, observed.replica().node).await?; if source.disk() != observed.replica().disk + || source.storage_instance() != Some(observed.engine()) || !voters.iter().any(|v| { v.member == observed.replica().member.get() && v.node == observed.replica().node.bytes() @@ -341,10 +409,10 @@ impl FleetCatalog { disk: observed.replica().disk.bytes(), process: observed.replica().process.bytes(), member: observed.replica().member.get(), - barrier: observed.barrier().get(), - applied: observed.applied().index.get(), - term: observed.applied().term.get(), - membership_revision: observed.membership_revision().get(), + barrier: observed.barrier().index.get(), + applied: observed.root().position().index.get(), + term: observed.root().position().term.get(), + membership_revision: observed.membership().revision().get(), consumed_sequence: status.consumed_sequence, phase: status.phase, voters, @@ -352,7 +420,7 @@ impl FleetCatalog { } _ => return Err(FleetCatalogError::Conflict), }; - let name = hex::encode(hash(proposal.scope.cell.as_bytes())); + let name = crate::raft::catalog::cell_group_import_key(proposal); if self.inner.lookup(record.coverage, &name).await?.is_some() { return Err(FleetCatalogError::Corrupt("duplicate import coverage")); } @@ -360,7 +428,7 @@ impl FleetCatalog { records::WITNESS_KIND, &Witness { history: history_key, - birth: entry.expected_digest(), + birth: entry.route().birth_digest, decision, }, 4096, @@ -389,7 +457,7 @@ impl FleetCatalog { return Err(FleetCatalogError::Corrupt("incomplete import coverage")); } if record.covered == record.cut.count { - self.no_namespace_admissions(snapshot, namespace.config().namespace) + self.no_namespace_admissions(snapshot, directory.namespace_config().namespace) .await?; record.phase = FleetImportPhase::Handoff; } @@ -399,7 +467,7 @@ impl FleetCatalog { async fn no_namespace_admissions( &self, snapshot: &FleetSnapshot, - namespace: NamespaceId, + directory: NamespaceId, ) -> Result<()> { let index = snapshot.head.tables[FleetTable::Admissions.slot()]; let mut after = None; @@ -417,7 +485,7 @@ impl FleetCatalog { .binding() .scope() .map_err(|_| FleetCatalogError::Corrupt("import permit scope"))?; - if scope.namespace == namespace { + if scope.namespace == directory { return Err(FleetCatalogError::Conflict); } } diff --git a/crates/cellar/durability/raft/fleet_catalog/imports/inventory.rs b/crates/cellar/durability/raft/fleet_catalog/imports/inventory.rs index b5821146f..693a8c6eb 100644 --- a/crates/cellar/durability/raft/fleet_catalog/imports/inventory.rs +++ b/crates/cellar/durability/raft/fleet_catalog/imports/inventory.rs @@ -1,98 +1,80 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! One rooted physical intent per step, including cancelled pending groups. use super::*; -use crate::raft::catalog::{BirthDisposition, GroupBirth}; -use crate::raft::fleet_catalog::histories::{Member, Origin}; +use crate::raft::cell_groups::{CellGroupBirth, CellGroupBirthDisposition}; impl FleetCatalog { pub(super) async fn import_inventory_step( &self, snapshot: &FleetSnapshot, request: &FleetRequestIdentity, - namespace: &NamespaceCatalog, + directory: &RangeDirectory, record: &mut Record, - timeout: Duration, ) -> Result> { - let step = namespace - .import_birth_step(&record.cut, &self.birth_owner()?, record.cursor.as_deref()) + directory + .check_group_import(&record.cut, &self.birth_owner()?, false) .await .map_err(catalog_error)?; - let Some(entry) = step.entry else { + let step = directory + .namespace() + .cell_group_import_step( + directory, + &record.cut, + &self.birth_owner()?, + record.cursor.as_deref(), + ) + .await + .map_err(catalog_error)?; + let Some(proposal) = step.entry else { if record.cut.count != 0 || record.inventoried != 0 || step.next.is_some() { - return Err(FleetCatalogError::Corrupt("incomplete import inventory")); + return Err(FleetCatalogError::Corrupt( + "incomplete group import inventory", + )); } record.phase = FleetImportPhase::Coverage; return Ok(Vec::new()); }; - let proposal = entry.proposal(); - let class = proposal - .scope - .cell - .split_once(':') - .ok_or(FleetCatalogError::Invalid("import history class"))? - .0; - self.require_namespace_declaration(snapshot, namespace, Some(class)) + self.require_namespace_declaration(snapshot, directory.namespace(), None) .await?; - let decision = match namespace - .birth_disposition(&proposal.scope.cell) + let decision = directory + .settle_import_group(&proposal, &record.cut, &self.birth_owner()?) .await - .map_err(catalog_error)? - { - Some(decision) => decision, - None => namespace - .cancel_birth( - proposal.clone(), - ProposalId::new(record.cut.operation) - .map_err(|_| FleetCatalogError::Corrupt("import cancellation"))?, - timeout, - ) - .await - .map_err(catalog_error)?, - }; + .map_err(catalog_error)?; + let admission = ProposalId::new(request.id) + .map_err(|_| FleetCatalogError::Invalid("group import admission"))?; + let operation = ProposalId::new(request.operation) + .map_err(|_| FleetCatalogError::Invalid("group import operation"))?; let history = match &decision { - BirthDisposition::Live(birth) - if birth.scope() == &proposal.scope - && birth.digest() == entry.expected_digest() => - { + CellGroupBirthDisposition::Live(birth) if birth.route() == proposal.route() => { FleetHistory::observed( - self.binding(birth.scope())?, - namespace, + self.binding(&proposal.route().history)?, + directory, birth, - ProposalId::new(request.id) - .map_err(|_| FleetCatalogError::Invalid("import admission"))?, - ProposalId::new(request.operation) - .map_err(|_| FleetCatalogError::Invalid("import operation"))?, + admission, + operation, )? } - BirthDisposition::Cancelled { scope, operation } if scope == &proposal.scope => { - FleetHistory { - binding: self.binding(scope)?, - schema_digest: namespace.config().schema_digest, - namespace_root: namespace.root_digest(), - birth_digest: entry.expected_digest(), - admission: request.id, - operation: request.operation, - voters: proposal.voters.iter().copied().map(Member::from).collect(), - learners: proposal - .learners - .iter() - .copied() - .map(Member::from) - .collect(), - state: FleetHistoryState::Cancelled { - operation: operation.bytes(), - }, - origin: Origin::Observed, - } - } + CellGroupBirthDisposition::Cancelled { + history, + operation: cancellation, + } if *history == proposal.route().history => FleetHistory::cancelled_group( + self.binding(history)?, + directory, + &proposal, + admission, + operation, + *cancellation, + )?, _ => return Err(FleetCatalogError::Conflict), }; - history.validate()?; let changes = self .import_history_changes( snapshot, - namespace, + directory, &history, match &decision { - BirthDisposition::Live(birth) => Some(birth), + CellGroupBirthDisposition::Live(birth) => Some(birth), _ => None, }, ) @@ -102,10 +84,10 @@ impl FleetCatalog { .checked_add(1) .ok_or(FleetCatalogError::Limit)?; if record.inventoried > record.cut.count - || (step.next.is_none() != (record.inventoried == record.cut.count)) + || step.next.is_none() != (record.inventoried == record.cut.count) { return Err(FleetCatalogError::Corrupt( - "import inventory count/continuation", + "group import count/continuation", )); } record.cursor = step.next; @@ -118,9 +100,9 @@ impl FleetCatalog { async fn import_history_changes( &self, snapshot: &FleetSnapshot, - namespace: &NamespaceCatalog, + directory: &RangeDirectory, history: &FleetHistory, - birth: Option<&GroupBirth>, + birth: Option<&CellGroupBirth>, ) -> Result> { let binding_key = history .binding @@ -143,7 +125,7 @@ impl FleetCatalog { || saved.voters != history.voters || saved.learners != history.learners || saved.state != history.state - || birth.is_some_and(|birth| !saved.check_birth(namespace, birth)) + || birth.is_some_and(|birth| !saved.check_birth(directory, birth)) { return Err(FleetCatalogError::Conflict); } diff --git a/crates/cellar/durability/raft/fleet_catalog/imports/mod.rs b/crates/cellar/durability/raft/fleet_catalog/imports/mod.rs index ba037b269..a459891a3 100644 --- a/crates/cellar/durability/raft/fleet_catalog/imports/mod.rs +++ b/crates/cellar/durability/raft/fleet_catalog/imports/mod.rs @@ -1,12 +1,11 @@ -//! Resumable namespace import. The fleet root owns progress; namespace effects +//! Resumable directory import. The fleet root owns progress; directory effects //! retain frozen references until every canonical history has a committed guard. use super::histories::{catalog_error, history_mutation, validate_request}; use super::*; use crate::raft::catalog::{ - BirthImportCut, BirthInventoryEntry, FrozenBirthInventory, ManagedBirthOwner, NamespaceCatalog, - VoterPolicy, + CellGroupImportCut, FrozenCellGroupInventory, ManagedBirthOwner, VoterPolicy, }; -use crate::raft::host::FleetMembershipObservation; +use crate::raft::cell_groups::{CellGroupBirthRequest, CellGroupQuorumMembership, RangeDirectory}; use cellar_logic::consensus::{NamespaceId, ProposalId}; mod coverage; @@ -55,34 +54,34 @@ impl FleetNamespaceImport { } } -fn key(namespace: NamespaceId) -> String { - format!("namespace/{}", hex::encode(namespace.bytes())) +fn key(directory: NamespaceId) -> String { + format!("namespace/{}", hex::encode(directory.bytes())) } impl FleetCatalog { - /// Root the exact frozen cut before attempting its exclusive namespace + /// Root the exact frozen cut before attempting its exclusive directory /// claim. Preparing/stopped imports grant no managed birth or maintenance. pub async fn begin_namespace_import( &self, snapshot: &FleetSnapshot, request: &FleetRequestIdentity, - namespace: &NamespaceCatalog, - frozen: &FrozenBirthInventory, + directory: &RangeDirectory, + frozen: &FrozenCellGroupInventory, timeout: Duration, ) -> Result { self.check_snapshot(snapshot)?; validate_request(request)?; - let cut = frozen.import_cut(namespace).map_err(catalog_error)?; + let cut = frozen.import_cut(directory).map_err(catalog_error)?; if request.operation != cut.operation || cut.count > HISTORY_LIMIT - || namespace.config().cluster.bytes() != self.inner.cluster - || namespace.config().voters != VoterPolicy::ThreeVoters + || directory.namespace_config().cluster.bytes() != self.inner.cluster + || directory.namespace_config().voters != VoterPolicy::ThreeVoters { - return Err(FleetCatalogError::Invalid("namespace import cut/scope")); + return Err(FleetCatalogError::Invalid("directory import cut/scope")); } let record = Record { - namespace: namespace.config().namespace.bytes(), - schema: namespace.config().schema_digest, + namespace: directory.namespace_config().namespace.bytes(), + schema: directory.namespace_config().schema_digest, catalog: self.birth_owner()?.catalog, cut, phase: FleetImportPhase::Preparing, @@ -92,15 +91,19 @@ impl FleetCatalog { cursor: None, coverage: None, }; - record.validate(self, namespace)?; + record.validate(self, directory)?; self.run(timeout, async { - self.require_namespace_declaration(snapshot, namespace, None) + directory + .import_directory_view(&record.cut) + .await + .map_err(catalog_error)?; + self.require_namespace_declaration(snapshot, directory.namespace(), None) .await?; let mutation = history_mutation( request, vec![FleetChange { table: FleetTable::Policy, - key: key(namespace.config().namespace), + key: key(directory.namespace_config().namespace), value: Some(record.encode()?), }], ); @@ -116,7 +119,7 @@ impl FleetCatalog { .inner .lookup( snapshot.head.tables[FleetTable::Policy.slot()], - &key(namespace.config().namespace), + &key(directory.namespace_config().namespace), ) .await? .is_some() @@ -131,13 +134,13 @@ impl FleetCatalog { pub async fn namespace_import( &self, snapshot: &FleetSnapshot, - namespace: &NamespaceCatalog, + directory: &RangeDirectory, timeout: Duration, ) -> Result { self.check_snapshot(snapshot)?; self.run(timeout, async { Ok(FleetNamespaceImport { - record: self.import_record(snapshot, namespace).await?, + record: self.import_record(snapshot, directory).await?, }) }) .await @@ -148,14 +151,14 @@ impl FleetCatalog { pub async fn next_namespace_import_birth( &self, snapshot: &FleetSnapshot, - namespace: &NamespaceCatalog, + directory: &RangeDirectory, timeout: Duration, - ) -> Result> { + ) -> Result> { self.check_snapshot(snapshot)?; self.run_import( timeout, Box::pin(async { - let record = self.import_record(snapshot, namespace).await?; + let record = self.import_record(snapshot, directory).await?; if record.stopped { return Err(FleetCatalogError::Conflict); } @@ -165,8 +168,14 @@ impl FleetCatalog { ) { return Ok(None); } - Ok(namespace - .import_birth_step(&record.cut, &self.birth_owner()?, record.cursor.as_deref()) + Ok(directory + .namespace() + .cell_group_import_step( + directory, + &record.cut, + &self.birth_owner()?, + record.cursor.as_deref(), + ) .await .map_err(catalog_error)? .entry) @@ -182,8 +191,8 @@ impl FleetCatalog { &self, snapshot: &FleetSnapshot, request: &FleetRequestIdentity, - namespace: &NamespaceCatalog, - observation: Option<&FleetMembershipObservation>, + directory: &RangeDirectory, + observation: Option<&CellGroupQuorumMembership>, timeout: Duration, ) -> Result { self.check_snapshot(snapshot)?; @@ -191,7 +200,7 @@ impl FleetCatalog { self.run_import( timeout, Box::pin(async { - let mut record = self.import_record(snapshot, namespace).await?; + let mut record = self.import_record(snapshot, directory).await?; if request.operation != record.cut.operation || record.stopped { return Err(FleetCatalogError::Conflict); } @@ -200,32 +209,27 @@ impl FleetCatalog { } let changes = match record.phase { FleetImportPhase::Preparing => { - namespace - .claim_birth_import(&record.cut, &self.birth_owner()?) + directory + .namespace() + .change_cell_group_import(&record.cut, &self.birth_owner()?, false) .await .map_err(catalog_error)?; record.phase = FleetImportPhase::Inventory; Vec::new() } FleetImportPhase::Inventory => { - self.import_inventory_step( - snapshot, - request, - namespace, - &mut record, - timeout, - ) - .await? + self.import_inventory_step(snapshot, request, directory, &mut record) + .await? } FleetImportPhase::Coverage => { - self.import_coverage_step(snapshot, namespace, &mut record, observation) + self.import_coverage_step(snapshot, directory, &mut record, observation) .await? } FleetImportPhase::Handoff => { - self.verify_complete_import(snapshot, namespace, &record) + self.verify_complete_import(snapshot, directory, &record) .await?; - namespace - .finish_birth_import(&record.cut, &self.birth_owner()?) + directory + .finish_group_import(&record.cut, &self.birth_owner()?) .await .map_err(catalog_error)?; record.phase = FleetImportPhase::Complete; @@ -233,7 +237,7 @@ impl FleetCatalog { } FleetImportPhase::Complete => return Err(FleetCatalogError::Conflict), }; - self.commit_import_record(snapshot, request, namespace, record, changes) + self.commit_import_record(snapshot, request, directory, record, changes) .await }), ) @@ -247,14 +251,14 @@ impl FleetCatalog { &self, snapshot: &FleetSnapshot, request: &FleetRequestIdentity, - namespace: &NamespaceCatalog, + directory: &RangeDirectory, stopped: bool, timeout: Duration, ) -> Result { self.check_snapshot(snapshot)?; validate_request(request)?; self.run(timeout, async { - let mut record = self.import_record(snapshot, namespace).await?; + let mut record = self.import_record(snapshot, directory).await?; if request.operation != record.cut.operation || matches!( record.phase, @@ -264,7 +268,7 @@ impl FleetCatalog { return Err(FleetCatalogError::Conflict); } record.stopped = stopped; - self.commit_import_record(snapshot, request, namespace, record, Vec::new()) + self.commit_import_record(snapshot, request, directory, record, Vec::new()) .await }) .await @@ -276,7 +280,7 @@ impl FleetCatalog { pub async fn retry_namespace_import_request( &self, request: &FleetRequestIdentity, - namespace: &NamespaceCatalog, + directory: &RangeDirectory, timeout: Duration, ) -> Result { validate_request(request)?; @@ -304,7 +308,8 @@ impl FleetCatalog { .changes .last() .ok_or(FleetCatalogError::Corrupt("empty import retry"))?; - if last.table != FleetTable::Policy || last.key != key(namespace.config().namespace) + if last.table != FleetTable::Policy + || last.key != key(directory.namespace_config().namespace) { return Err(FleetCatalogError::Conflict); } @@ -313,7 +318,7 @@ impl FleetCatalog { last.value.as_deref().ok_or(FleetCatalogError::Conflict)?, 4096, )?; - record.validate(self, namespace)?; + record.validate(self, directory)?; if record.cut.operation != request.operation || mutation.changes.len() > 4 || mutation @@ -332,7 +337,7 @@ impl FleetCatalog { .await? .is_none() { - self.verify_complete_import(¤t, namespace, &record) + self.verify_complete_import(¤t, directory, &record) .await?; } self.commit_inner(¤t, &mutation).await @@ -344,18 +349,18 @@ impl FleetCatalog { async fn import_record( &self, snapshot: &FleetSnapshot, - namespace: &NamespaceCatalog, + directory: &RangeDirectory, ) -> Result { let bytes = self .inner .lookup( snapshot.head.tables[FleetTable::Policy.slot()], - &key(namespace.config().namespace), + &key(directory.namespace_config().namespace), ) .await? .ok_or(FleetCatalogError::Conflict)?; let record: Record = decode(records::KIND, &bytes, 4096)?; - record.validate(self, namespace)?; + record.validate(self, directory)?; Ok(record) } @@ -363,14 +368,14 @@ impl FleetCatalog { &self, snapshot: &FleetSnapshot, request: &FleetRequestIdentity, - namespace: &NamespaceCatalog, + directory: &RangeDirectory, record: Record, mut changes: Vec, ) -> Result { - record.validate(self, namespace)?; + record.validate(self, directory)?; changes.push(FleetChange { table: FleetTable::Policy, - key: key(namespace.config().namespace), + key: key(directory.namespace_config().namespace), value: Some(record.encode()?), }); let mutation = history_mutation(request, changes); @@ -399,20 +404,22 @@ impl FleetCatalog { pub(super) async fn managed_owner_for_birth( &self, snapshot: &FleetSnapshot, - namespace: &NamespaceCatalog, + directory: &RangeDirectory, ) -> Result { let owner = self.birth_owner()?; - if namespace.config().managed_fleet.is_some() { - if self.bind_namespace(namespace.config().clone())? != *namespace.config() { + if directory.namespace_config().managed_fleet.is_some() { + if self.bind_namespace(directory.namespace_config().clone())? + != *directory.namespace_config() + { return Err(FleetCatalogError::Conflict); } } else { - let record = self.import_record(snapshot, namespace).await?; + let record = self.import_record(snapshot, directory).await?; if record.phase != FleetImportPhase::Complete || record.stopped { return Err(FleetCatalogError::Conflict); } - namespace - .check_birth_import(&record.cut, &owner, true) + directory + .check_group_import(&record.cut, &owner, true) .await .map_err(catalog_error)?; } diff --git a/crates/cellar/durability/raft/fleet_catalog/imports/records.rs b/crates/cellar/durability/raft/fleet_catalog/imports/records.rs index 6a461a5a2..2257f2098 100644 --- a/crates/cellar/durability/raft/fleet_catalog/imports/records.rs +++ b/crates/cellar/durability/raft/fleet_catalog/imports/records.rs @@ -19,7 +19,7 @@ pub(super) struct Record { pub namespace: [u8; 16], pub schema: [u8; 32], pub catalog: [u8; 32], - pub cut: BirthImportCut, + pub cut: CellGroupImportCut, pub phase: FleetImportPhase, pub stopped: bool, pub inventoried: u64, @@ -37,12 +37,14 @@ fn required_nullable<'de, T: Deserialize<'de>, D: serde::Deserializer<'de>>( } impl Record { - pub fn validate(&self, catalog: &FleetCatalog, namespace: &NamespaceCatalog) -> Result<()> { - self.cut.validate(namespace).map_err(catalog_error)?; - if self.namespace != namespace.config().namespace.bytes() - || namespace.config().cluster.bytes() != catalog.inner.cluster - || namespace.config().voters != VoterPolicy::ThreeVoters - || self.schema != namespace.config().schema_digest + pub fn validate(&self, catalog: &FleetCatalog, directory: &RangeDirectory) -> Result<()> { + self.cut + .validate(directory.namespace()) + .map_err(catalog_error)?; + if self.namespace != directory.namespace_config().namespace.bytes() + || directory.namespace_config().cluster.bytes() != catalog.inner.cluster + || directory.namespace_config().voters != VoterPolicy::ThreeVoters + || self.schema != directory.namespace_config().schema_digest || self.catalog != catalog.birth_owner()?.catalog || self.cut.count > HISTORY_LIMIT || self.inventoried > self.cut.count @@ -55,7 +57,7 @@ impl Record { .all(|b| b.is_ascii_hexdigit() && !b.is_ascii_uppercase()) }) { - return Err(FleetCatalogError::Corrupt("namespace import record")); + return Err(FleetCatalogError::Corrupt("directory import record")); } let valid = match self.phase { FleetImportPhase::Preparing => { @@ -84,7 +86,7 @@ impl Record { }; if !valid { return Err(FleetCatalogError::Corrupt( - "namespace import phase/progress", + "directory import phase/progress", )); } Ok(()) diff --git a/crates/cellar/durability/raft/fleet_catalog/maintenance/birth_action.rs b/crates/cellar/durability/raft/fleet_catalog/maintenance/birth_action.rs index 0052400d0..723d114fa 100644 --- a/crates/cellar/durability/raft/fleet_catalog/maintenance/birth_action.rs +++ b/crates/cellar/durability/raft/fleet_catalog/maintenance/birth_action.rs @@ -344,7 +344,7 @@ impl FleetCatalog { &self, snapshot: &FleetSnapshot, operation: ProposalId, - scope: &cellar_logic::consensus::GroupScope, + scope: &cellar_logic::cell_group::GroupHistory, resolution: FleetRestartBirthResolution, timeout: Duration, ) -> Result<(Action, FleetSnapshot)> { @@ -395,8 +395,8 @@ impl FleetCatalog { &self, snapshot: &FleetSnapshot, parent: &Action, - namespace: &NamespaceCatalog, - winner: crate::raft::catalog::BirthDisposition, + namespace: &RangeDirectory, + winner: crate::raft::catalog::CellGroupBirthDisposition, timeout: Duration, ) -> Result { self.run(timeout, async { @@ -423,13 +423,13 @@ impl FleetCatalog { action.check_history(&history)?; let scope = history.binding().scope().map_err(|_| corrupt())?; history.state = match winner { - crate::raft::catalog::BirthDisposition::Live(birth) + crate::raft::catalog::CellGroupBirthDisposition::Live(birth) if history.check_birth(namespace, &birth) => { FleetHistoryState::Live } - crate::raft::catalog::BirthDisposition::Cancelled { - scope: actual, + crate::raft::catalog::CellGroupBirthDisposition::Cancelled { + history: actual, operation, } if actual == scope => FleetHistoryState::Cancelled { operation: operation.bytes(), diff --git a/crates/cellar/durability/raft/fleet_catalog/maintenance/mod.rs b/crates/cellar/durability/raft/fleet_catalog/maintenance/mod.rs index 042cde44a..15c1ab227 100644 --- a/crates/cellar/durability/raft/fleet_catalog/maintenance/mod.rs +++ b/crates/cellar/durability/raft/fleet_catalog/maintenance/mod.rs @@ -17,7 +17,7 @@ mod stop; use super::evacuation::CanonicalRequest; use super::histories::validate_request; use super::*; -use crate::raft::catalog::NamespaceCatalog; +use crate::raft::catalog::RangeDirectory; use cellar_logic::consensus::{NodeId, ProcessGeneration, ProposalId}; pub use gate::{FleetRestartGateAdmission, FleetRestartGateRequest}; pub use gate_cancel::FleetRestartGateCancelRequest; @@ -125,7 +125,7 @@ impl FleetCatalog { &self, snapshot: &FleetSnapshot, request: &FleetRestartRequest, - namespaces: &[NamespaceCatalog], + namespaces: &[RangeDirectory], timeout: Duration, ) -> Result { self.check_snapshot(snapshot)?; diff --git a/crates/cellar/durability/raft/fleet_catalog/maintenance/operator/driver.rs b/crates/cellar/durability/raft/fleet_catalog/maintenance/operator/driver.rs index 0f21c00e2..56b9ddbc1 100644 --- a/crates/cellar/durability/raft/fleet_catalog/maintenance/operator/driver.rs +++ b/crates/cellar/durability/raft/fleet_catalog/maintenance/operator/driver.rs @@ -52,7 +52,7 @@ pub struct FleetRestartOperatorProgress { /// controller retains Root work; a recreated driver repeats ephemeral proof. pub struct FleetRestartOperatorDriver { pub(super) catalog: FleetCatalog, - pub(super) namespaces: Vec, + pub(super) namespaces: Vec, pub(super) observer: RaftCells, pub(super) evidence: Option>, session: Mutex>, @@ -107,7 +107,7 @@ impl Stage { impl FleetRestartOperatorDriver { pub fn new( catalog: FleetCatalog, - namespaces: Vec, + namespaces: Vec, observer: RaftCells, evidence: Option>, ) -> Result { diff --git a/crates/cellar/durability/raft/fleet_catalog/maintenance/operator/mod.rs b/crates/cellar/durability/raft/fleet_catalog/maintenance/operator/mod.rs index 08b89eee8..91f6ea4e8 100644 --- a/crates/cellar/durability/raft/fleet_catalog/maintenance/operator/mod.rs +++ b/crates/cellar/durability/raft/fleet_catalog/maintenance/operator/mod.rs @@ -14,7 +14,7 @@ mod service; mod status; use super::super::resources::Authority; use super::*; -use crate::raft::catalog::NamespaceCatalog; +use crate::raft::catalog::RangeDirectory; pub use cancellation::{ FleetRestartOperatorCancelAdmission, FleetRestartOperatorCancelRequest, FleetRestartOperatorCancelled, @@ -80,7 +80,7 @@ impl FleetCatalog { &'a self, snapshot: &'a FleetSnapshot, request: &'a FleetRestartOperatorRequest, - namespaces: &'a [NamespaceCatalog], + namespaces: &'a [RangeDirectory], timeout: Duration, ) -> BoxFuture<'a, Result> { Box::pin(async move { @@ -234,7 +234,7 @@ impl FleetCatalog { })) } - async fn active_restart_operator_inner( + pub(in crate::raft::fleet_catalog) async fn active_restart_operator_inner( &self, snapshot: &FleetSnapshot, ) -> Result> { diff --git a/crates/cellar/durability/raft/fleet_catalog/maintenance/operator/service/mod.rs b/crates/cellar/durability/raft/fleet_catalog/maintenance/operator/service/mod.rs index d4502f822..807d9665a 100644 --- a/crates/cellar/durability/raft/fleet_catalog/maintenance/operator/service/mod.rs +++ b/crates/cellar/durability/raft/fleet_catalog/maintenance/operator/service/mod.rs @@ -93,7 +93,7 @@ impl FleetCatalog { pub async fn start_restart_service( &self, observer: crate::application::consensus::RaftCells, - namespaces: Vec, + namespaces: Vec, evidence: Option>, options: FleetRestartServiceOptions, ) -> Result { @@ -115,7 +115,12 @@ impl FleetCatalog { }; let timeout = options.turn_timeout; let work = RestartServiceWork::new( - FleetRestartOperatorDriver::new(self.clone(), namespaces, observer.clone(), evidence)?, + FleetRestartOperatorDriver::new( + self.clone(), + namespaces, + observer.internal_observer(), + evidence, + )?, options, wait, done, diff --git a/crates/cellar/durability/raft/fleet_catalog/maintenance/operator/service/worker.rs b/crates/cellar/durability/raft/fleet_catalog/maintenance/operator/service/worker.rs index 8823dca4a..f89853b58 100644 --- a/crates/cellar/durability/raft/fleet_catalog/maintenance/operator/service/worker.rs +++ b/crates/cellar/durability/raft/fleet_catalog/maintenance/operator/service/worker.rs @@ -2,19 +2,19 @@ //! Node job-set lifetime, bounded collection and independent shutdown signals. use super::*; -use crate::raft::budget::Permit; +use crate::application::consensus::group_client::{ + ControlServiceCompletion, ControlServiceKind, ControlServiceLease, +}; use crate::raft::fleet_catalog::deadline::Deadline; -use std::sync::atomic::Ordering; pub(crate) struct RestartServiceWork { driver: FleetRestartOperatorDriver, options: FleetRestartServiceOptions, stop: watch::Receiver, - done: watch::Sender, status: watch::Sender, node_stop: Option>, - memory: Option, - admitted: bool, + _registration: Option, + _completion: ControlServiceCompletion, } impl RestartServiceWork { @@ -29,11 +29,10 @@ impl RestartServiceWork { driver, options, stop, - done, status, node_stop: None, - memory: None, - admitted: false, + _registration: None, + _completion: ControlServiceCompletion::new(done), } } @@ -42,23 +41,14 @@ impl RestartServiceWork { return Err(FleetCatalogError::Invalid("restart service fleet")); } let handle = &self.driver.observer.handle; - if handle.closing.load(Ordering::Acquire) - || handle.admission.is_draining() - || *self.stop.borrow() + if self.driver.observer.is_closed() || handle.admission.is_draining() || *self.stop.borrow() { return Err(FleetCatalogError::Closed); } - let mut slot = handle - .fleet_restart_service - .lock() - .map_err(|_| FleetCatalogError::Closed)?; - if slot.is_some() { - return Err(FleetCatalogError::Overloaded); - } - let memory = handle - .control_budget - .acquire(2 * ROOT_LIMIT) - .ok_or(FleetCatalogError::Overloaded)?; + let registration = self + .driver + .observer + .register_control_service(ControlServiceKind::Restart, 2 * ROOT_LIMIT)?; // Select the installed backend and Node domain. A caller cannot replace // either by constructing another client with the same descriptive ID. let source = &host.catalog.inner; @@ -72,11 +62,8 @@ impl RestartServiceWork { turn: Mutex::new(()), }), }; - let (stop, wait) = watch::channel(false); - *slot = Some(stop); - self.node_stop = Some(wait); - self.memory = Some(memory); - self.admitted = true; + self.node_stop = Some(registration.stop_receiver()); + self._registration = Some(registration); self.status.send_modify(|s| s.running = true); Ok(()) } @@ -99,7 +86,7 @@ impl RestartServiceWork { async fn poll(&mut self) { loop { let handle = &self.driver.observer.handle; - if handle.closing.load(Ordering::Acquire) || handle.admission.is_draining() { + if self.driver.observer.is_closed() || handle.admission.is_draining() { return; } self.status @@ -193,13 +180,6 @@ impl RestartServiceWork { impl Drop for RestartServiceWork { fn drop(&mut self) { - if self.admitted { - if let Ok(mut slot) = self.driver.observer.handle.fleet_restart_service.lock() { - slot.take(); - } - } - self.memory.take(); self.status.send_modify(|s| s.running = false); - self.done.send_replace(true); } } diff --git a/crates/cellar/durability/raft/fleet_catalog/maintenance/rejoin/boot.rs b/crates/cellar/durability/raft/fleet_catalog/maintenance/rejoin/boot.rs index 81af73881..f2ea34a0e 100644 --- a/crates/cellar/durability/raft/fleet_catalog/maintenance/rejoin/boot.rs +++ b/crates/cellar/durability/raft/fleet_catalog/maintenance/rejoin/boot.rs @@ -403,3 +403,67 @@ impl FleetCatalog { }) } } + +impl FleetHostAuthorization { + /// Verify the exact returned-boot lineage for an existing voting member. + /// The directory independently checks local retention, claims and closure; + /// this metadata proof cannot create genesis or grant application serving. + pub(crate) fn require_retained_group_start<'a>( + &'a self, + selector: &'a StopSelector, + replica: &'a cellar_logic::consensus::ReplicaIdentity< + cellar_logic::cell_group::GroupHistory, + >, + timeout: Duration, + ) -> BoxFuture<'a, Result<()>> { + Box::pin(async move { + let boot = self.boot_binding()?; + if selector.boot != boot + || replica.node != self.node() + || replica.disk != self.disk() + || replica.process.bytes() != boot.process + { + return Err(FleetCatalogError::Conflict); + } + self.catalog + .run(timeout, async { + let snapshot = self.catalog.inner.current().await?; + let current = self + .catalog + .host_from_record(&snapshot, self.node()) + .await?; + self.require_running_successor(¤t)?; + let cut = self + .catalog + .checked_returned_cut(&snapshot, selector.operation) + .await?; + if cut.quiescence != *selector + || cut.node != current.record + || current.boot_binding()? != boot + || current.state() != FleetNodeState::RestartPending + { + return Err(FleetCatalogError::Conflict); + } + let key = self + .catalog + .binding(&replica.scope)? + .key() + .map_err(|_| FleetCatalogError::Invalid("retained group scope"))?; + let rooted = self.catalog.history_inner(&snapshot, &key).await?; + if *rooted.state() != crate::raft::fleet_catalog::FleetHistoryState::Live { + return Err(FleetCatalogError::Conflict); + } + let (voters, _) = self + .catalog + .restart_voters(&snapshot, &rooted, self.node()) + .await? + .ok_or(FleetCatalogError::Conflict)?; + if voters.get(&self.node()) != Some(&replica.member) { + return Err(FleetCatalogError::Conflict); + } + Ok(()) + }) + .await + }) + } +} diff --git a/crates/cellar/durability/raft/fleet_catalog/maintenance/resume/leadership.rs b/crates/cellar/durability/raft/fleet_catalog/maintenance/resume/leadership.rs index 5e061fae4..b87606143 100644 --- a/crates/cellar/durability/raft/fleet_catalog/maintenance/resume/leadership.rs +++ b/crates/cellar/durability/raft/fleet_catalog/maintenance/resume/leadership.rs @@ -168,13 +168,8 @@ impl FleetHostAuthorization { .binding() .scope() .map_err(|_| FleetCatalogError::Corrupt("resume scope"))?; - let cell = gateway - .cell(scope.namespace, &scope.cell) - .map_err(failure)?; - if cell.scope() != &scope { - return Err(FleetCatalogError::Conflict); - } - let progress = cell + let group = gateway.group(scope).map_err(failure)?; + let progress = group .replication_status_for(deadline.remaining()?) .await .map_err(failure)?; @@ -310,20 +305,21 @@ impl FleetHostAuthorization { .checked_add(remaining) .ok_or(FleetCatalogError::Limit)? .min(owner_limit); - cell.restart_leadership( - RestartLeadershipCheck { - deadline: expires, - expected_term: Term::new(command.term), - expected_revision: LogIndex::new(command.revision), - through: progress.committed, - survivors, - all_voters: true, - transfer, - }, - deadline.remaining()?, - ) - .await - .map_err(failure)?; + group + .restart_leadership( + RestartLeadershipCheck { + deadline: expires, + expected_term: Term::new(command.term), + expected_revision: LogIndex::new(command.revision), + through: progress.committed, + survivors, + all_voters: true, + transfer, + }, + deadline.remaining()?, + ) + .await + .map_err(failure)?; catalog .run(deadline.remaining()?, input.basis.check_current(catalog)) .await?; @@ -438,12 +434,13 @@ pub(super) async fn reconcile( ) } -fn failure(error: crate::application::consensus::CommandFailure) -> FleetCatalogError { - use crate::application::consensus::CommandFailure as C; +fn failure(error: crate::raft::completion::RuntimeError) -> FleetCatalogError { + use crate::raft::completion::RuntimeError as R; match error { - C::Overloaded => FleetCatalogError::Overloaded, - C::CommandConflict => FleetCatalogError::Conflict, - C::CorruptState => FleetCatalogError::Corrupt("resume leader state"), + R::Overloaded => FleetCatalogError::Overloaded, + R::Invalid(_) | R::NotFound | R::StaleCompletion => FleetCatalogError::Conflict, + R::Protocol(_) | R::GroupQuarantined => FleetCatalogError::Corrupt("resume leader state"), + R::Closed => FleetCatalogError::Closed, _ => FleetCatalogError::Unconfirmed, } } diff --git a/crates/cellar/durability/raft/fleet_catalog/maintenance/resume/mod.rs b/crates/cellar/durability/raft/fleet_catalog/maintenance/resume/mod.rs index c0596eca5..b83847b1d 100644 --- a/crates/cellar/durability/raft/fleet_catalog/maintenance/resume/mod.rs +++ b/crates/cellar/durability/raft/fleet_catalog/maintenance/resume/mod.rs @@ -10,14 +10,15 @@ use super::rejoin::ReturnedCut; use super::runtime::{proof, records as runtime_records}; use super::*; use crate::application::consensus::RaftCells; -use crate::raft::catalog::NamespaceCatalog; +use crate::raft::catalog::RangeDirectory; use crate::raft::fleet_catalog::deadline::Deadline; use crate::raft::fleet_catalog::resources::Authority; use crate::raft::fleet_stop::{ FleetStopAction, FleetStopRequest, LocalResume, ResumeMode, StopSelector, }; use crate::raft::physical_inventory::RaftPhysicalInventory; -use cellar_logic::consensus::{GroupScope, MemberId}; +use cellar_logic::cell_group::GroupHistory; +use cellar_logic::consensus::MemberId; use futures_util::future::BoxFuture; pub(crate) use leadership::ResumeLeadershipCommand; use records::{Completion, Intent}; @@ -60,16 +61,16 @@ impl From for FleetRestartResumePhase { #[derive(Clone, Debug)] pub enum FleetRestartResumeProgress { Unrelated { - group: GroupScope, + group: GroupHistory, }, Verified { - group: GroupScope, + group: GroupHistory, current_member: Option, retained_member: Option, replaced: bool, }, HandoffRequested { - group: GroupScope, + group: GroupHistory, from: MemberId, to: MemberId, }, @@ -161,7 +162,7 @@ impl FleetCatalog { &'a self, snapshot: &'a FleetSnapshot, request: &'a FleetRestartResumeRequest, - namespaces: &'a [NamespaceCatalog], + namespaces: &'a [RangeDirectory], observer: &'a RaftCells, timeout: Duration, ) -> BoxFuture<'a, Result> { diff --git a/crates/cellar/durability/raft/fleet_catalog/maintenance/resume/verification/mod.rs b/crates/cellar/durability/raft/fleet_catalog/maintenance/resume/verification/mod.rs index 548cbce88..b53c6f4b4 100644 --- a/crates/cellar/durability/raft/fleet_catalog/maintenance/resume/verification/mod.rs +++ b/crates/cellar/durability/raft/fleet_catalog/maintenance/resume/verification/mod.rs @@ -187,11 +187,7 @@ impl FleetRestartResumeVerification { .binding() .scope() .map_err(|_| FleetCatalogError::Corrupt("resume scope"))?; - let physical = self - .inventory - .history(&group) - .map_err(|_| FleetCatalogError::Conflict)? - .copied(); + let physical = self.inventory.group_history(&group).copied(); let target = NodeId::new(self.cut.node.node) .map_err(|_| FleetCatalogError::Corrupt("resume target"))?; let key = history @@ -247,7 +243,8 @@ impl FleetRestartResumeVerification { .await?; let mut proof_value = None; let mut sample = None; - let retained = physical.is_some_and(|p| !p.in_configuration()); + let retained = + physical.is_some_and(|p| !p.in_configuration() || p.parked_retired_source()); let mut replaced = false; let progress = match history.state() { FleetHistoryState::Reserved => { @@ -269,6 +266,21 @@ impl FleetRestartResumeVerification { } FleetRestartResumeProgress::Unrelated { group } } + FleetHistoryState::Retired { .. } => match (physical, previous) { + (Some(slot), Some((member, false))) + if (slot.parked_retired_source() || !slot.in_configuration()) + && slot.member() == member => + { + FleetRestartResumeProgress::Verified { + group, + current_member: None, + retained_member: Some(member), + replaced: false, + } + } + (None, None) => FleetRestartResumeProgress::Unrelated { group }, + _ => return Err(FleetCatalogError::Conflict), + }, FleetHistoryState::Live => { if let Some((voters, permit)) = voters { let current_member = voters.get(&target).copied(); diff --git a/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/genesis/materialize.rs b/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/genesis/materialize.rs index 6bf68b423..d8eba18ab 100644 --- a/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/genesis/materialize.rs +++ b/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/genesis/materialize.rs @@ -59,9 +59,7 @@ pub(in crate::raft::fleet_catalog::maintenance::runtime) async fn next( .as_ref(), None => verification.inventory.as_ref(), }; - let slot = inventory - .history(&scope) - .map_err(|_| FleetCatalogError::Conflict)?; + let slot = inventory.group_history(&scope); if let Some(slot) = slot { check_slot(history, member, *slot)?; let unregistered = physical @@ -121,8 +119,7 @@ pub(in crate::raft::fleet_catalog::maintenance::runtime) async fn next( ) .await?; let slot = actual - .history(&scope) - .map_err(|_| FleetCatalogError::Conflict)? + .group_history(&scope) .ok_or(FleetCatalogError::Unconfirmed)?; check_slot(history, member, *slot)?; refresh_target(verification, history, &voters, actual)?; @@ -132,8 +129,7 @@ pub(in crate::raft::fleet_catalog::maintenance::runtime) async fn next( .inventory .as_ref() .ok_or(FleetCatalogError::InventoryIncomplete)? - .history(&scope) - .map_err(|_| FleetCatalogError::Conflict)? + .group_history(&scope) .copied() .ok_or(FleetCatalogError::Unconfirmed)?; check_slot(history, member, slot)?; @@ -184,8 +180,7 @@ fn refresh_target( .map_err(|_| FleetCatalogError::Corrupt("genesis maintenance node"))?; let member = *voters.get(&main).ok_or(FleetCatalogError::Conflict)?; let slot = actual - .history(&scope) - .map_err(|_| FleetCatalogError::Conflict)? + .group_history(&scope) .copied() .ok_or(FleetCatalogError::Conflict)?; check_slot(history, member, slot)?; diff --git a/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/genesis/receive.rs b/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/genesis/receive.rs index a3545e1df..3ef823cb4 100644 --- a/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/genesis/receive.rs +++ b/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/genesis/receive.rs @@ -79,21 +79,12 @@ impl FleetHostAuthorization { let authorization = catalog .authorize_group(&snapshot, &scope, deadline.remaining()?) .await?; - let cell = match gateway.cell(scope.namespace, &scope.cell) { - Ok(cell) if cell.scope() == &scope => cell, - Ok(_) => return Err(FleetCatalogError::Conflict), - Err(_) => gateway - .register_managed_history(authorization.clone(), deadline.remaining()?) - .await - .map_err(command_failure)?, - }; - // This pipeline owns initialization after submission, even when the - // control waiter disconnects. It refuses an already-claimed blank - // engine, occupied/retired slot, wrong schema or substituted boot. - let receipt = cell - .activate_managed_initial_member(authorization, deadline.remaining()?) + // The controller owns complete-group recovery and attachment after + // admission. A receipt is descriptive; fresh quorum is checked later. + let receipt = gateway + .activate_managed_initial_group(authorization, deadline.remaining()?) .await - .map_err(command_failure)?; + .map_err(startup_failure)?; if &receipt.binding != history.binding() { return Err(FleetCatalogError::Conflict); } @@ -110,13 +101,50 @@ impl FleetHostAuthorization { } } -fn command_failure(error: crate::application::consensus::CommandFailure) -> FleetCatalogError { - use crate::application::consensus::CommandFailure as C; +fn startup_failure( + error: crate::application::consensus::CellGroupStartupError, +) -> FleetCatalogError { + use crate::application::consensus::CellGroupStartupError; + use crate::raft::cell_groups::CellGroupControllerError as C; match error { - C::Overloaded => FleetCatalogError::Overloaded, - C::CommandConflict => FleetCatalogError::Conflict, - C::CorruptState => FleetCatalogError::Corrupt("genesis host state"), - C::UnsupportedCapability => FleetCatalogError::Invalid("genesis host capability"), - _ => FleetCatalogError::Unconfirmed, + // Entered recovery may have synced or attached even if its delivery failed. + CellGroupStartupError::Unconfirmed(_) => FleetCatalogError::Unconfirmed, + CellGroupStartupError::Refused(C::Runtime( + crate::raft::completion::RuntimeError::Overloaded, + )) => FleetCatalogError::Overloaded, + CellGroupStartupError::Refused(C::Catalog(error)) => { + use crate::raft::catalog::CatalogError as E; + match error.as_ref() { + E::Invalid(reason) => FleetCatalogError::Invalid(reason), + E::Corrupt(reason) => FleetCatalogError::Corrupt(reason), + E::Limit => FleetCatalogError::Limit, + E::Log(crate::raft::storage::LogError::Corrupt(_)) | E::LostStableState => { + FleetCatalogError::Corrupt("genesis group state") + } + E::Conflict(_) | E::ManagedAdmissionRequired | E::NotInitialMember | E::Fenced => { + FleetCatalogError::Conflict + } + E::FleetAuthority(authority) => match authority.as_ref() { + FleetCatalogError::Invalid(reason) => FleetCatalogError::Invalid(reason), + FleetCatalogError::Corrupt(reason) => FleetCatalogError::Corrupt(reason), + FleetCatalogError::Overloaded => FleetCatalogError::Overloaded, + FleetCatalogError::Limit => FleetCatalogError::Limit, + FleetCatalogError::Conflict => FleetCatalogError::Conflict, + FleetCatalogError::Closed => FleetCatalogError::Closed, + // Preserve the complete typed cause for local diagnosis; + // transport delivery still classifies unknown outcomes. + _ => FleetCatalogError::Storage( + CellGroupStartupError::Refused(C::Catalog(error)).into(), + ), + }, + _ => FleetCatalogError::Storage( + CellGroupStartupError::Refused(C::Catalog(error)).into(), + ), + } + } + CellGroupStartupError::Refused(C::Invalid(_)) => { + FleetCatalogError::Invalid("genesis group capability") + } + CellGroupStartupError::Refused(_) => FleetCatalogError::Unconfirmed, } } diff --git a/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/handoff/receive.rs b/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/handoff/receive.rs index 7085d96c4..e79b221f0 100644 --- a/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/handoff/receive.rs +++ b/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/handoff/receive.rs @@ -103,13 +103,8 @@ impl FleetHostAuthorization { .binding() .scope() .map_err(|_| FleetCatalogError::Corrupt("restart scope"))?; - let cell = gateway - .cell(scope.namespace, &scope.cell) - .map_err(failure)?; - if cell.scope() != &scope { - return Err(FleetCatalogError::Conflict); - } - let progress = cell + let group = gateway.group(scope).map_err(failure)?; + let progress = group .replication_status_for(deadline.remaining()?) .await .map_err(failure)?; @@ -243,20 +238,21 @@ impl FleetHostAuthorization { .checked_add(budget) .ok_or(FleetCatalogError::Limit)?, ); - cell.restart_leadership( - RestartLeadershipCheck { - deadline: evidence_deadline, - expected_term: Term::new(command.term), - expected_revision: LogIndex::new(command.revision), - through: progress.committed, - survivors, - all_voters: false, - transfer, - }, - deadline.remaining()?, - ) - .await - .map_err(failure)?; + group + .restart_leadership( + RestartLeadershipCheck { + deadline: evidence_deadline, + expected_term: Term::new(command.term), + expected_revision: LogIndex::new(command.revision), + through: progress.committed, + survivors, + all_voters: false, + transfer, + }, + deadline.remaining()?, + ) + .await + .map_err(failure)?; catalog .run(deadline.remaining()?, basis.check_current(catalog)) .await?; @@ -267,12 +263,15 @@ impl FleetHostAuthorization { } } -fn failure(error: crate::application::consensus::CommandFailure) -> FleetCatalogError { - use crate::application::consensus::CommandFailure as C; +fn failure(error: crate::raft::completion::RuntimeError) -> FleetCatalogError { + use crate::raft::completion::RuntimeError as R; match error { - C::Overloaded => FleetCatalogError::Overloaded, - C::CommandConflict => FleetCatalogError::Conflict, - C::CorruptState => FleetCatalogError::Corrupt("restart leadership host state"), + R::Overloaded => FleetCatalogError::Overloaded, + R::Invalid(_) | R::NotFound | R::StaleCompletion => FleetCatalogError::Conflict, + R::Protocol(_) | R::GroupQuarantined => { + FleetCatalogError::Corrupt("restart leadership host state") + } + R::Closed => FleetCatalogError::Closed, _ => FleetCatalogError::Unconfirmed, } } diff --git a/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/mod.rs b/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/mod.rs index 4cdf1bcd7..cbdcea771 100644 --- a/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/mod.rs +++ b/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/mod.rs @@ -19,7 +19,8 @@ use crate::raft::fleet_catalog::finalization::receipts::Publication; use crate::raft::physical_inventory::{RaftPhysicalHistory, RaftPhysicalInventory}; pub(crate) use action::RestartActionContext; use basis::Basis; -use cellar_logic::consensus::{GroupScope, MemberId}; +use cellar_logic::cell_group::GroupHistory; +use cellar_logic::consensus::MemberId; use futures_util::future::BoxFuture; pub(super) use handoff::LeadershipAction; pub(crate) use handoff::RestartLeadershipCommand; @@ -50,17 +51,17 @@ pub struct FleetRestartVerification { #[derive(Clone, Debug)] pub enum FleetRestartHistoryProgress { Cancelled { - group: GroupScope, + group: GroupHistory, }, Unrelated { - group: GroupScope, + group: GroupHistory, }, Retained { - group: GroupScope, + group: GroupHistory, member: MemberId, }, Voting { - group: GroupScope, + group: GroupHistory, member: MemberId, survivors: [NodeId; 2], }, @@ -117,7 +118,7 @@ impl FleetCatalog { &'a self, snapshot: &'a FleetSnapshot, operation: ProposalId, - namespaces: &'a [NamespaceCatalog], + namespaces: &'a [RangeDirectory], observer: &'a RaftCells, timeout: Duration, ) -> BoxFuture<'a, Result> { @@ -263,11 +264,7 @@ impl FleetRestartVerification { .binding() .scope() .map_err(|_| FleetCatalogError::Corrupt("restart scope"))?; - let physical = self - .inventory - .history(&group) - .map_err(|_| FleetCatalogError::Conflict)? - .copied(); + let physical = self.inventory.group_history(&group).copied(); let target = NodeId::new(self.gate.node.node) .map_err(|_| FleetCatalogError::Corrupt("restart target"))?; let voters = catalog @@ -319,6 +316,20 @@ impl FleetRestartVerification { FleetRestartHistoryProgress::Cancelled { group }, ) } + FleetHistoryState::Retired { .. } => match physical { + Some(slot) if slot.parked_retired_source() || !slot.in_configuration() => ( + Evidence::Retained, + FleetRestartHistoryProgress::Retained { + group, + member: slot.member(), + }, + ), + Some(_) => return Err(FleetCatalogError::Conflict), + None => ( + Evidence::Unrelated, + FleetRestartHistoryProgress::Unrelated { group }, + ), + }, FleetHistoryState::Live => { if let Some((voters, permit)) = voters { let member = *voters.get(&target).ok_or(FleetCatalogError::Conflict)?; diff --git a/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/proof.rs b/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/proof.rs index da12b2623..d69785381 100644 --- a/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/proof.rs +++ b/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/proof.rs @@ -340,6 +340,9 @@ pub(in crate::raft::fleet_catalog::maintenance) fn observe_current_voting<'a>( observer, paused, } = cut; + if expected.len() != 3 { + return Err(FleetCatalogError::InventoryIncomplete); + } let first = if expected.contains_key(&target) { target } else { @@ -361,12 +364,22 @@ pub(in crate::raft::fleet_catalog::maintenance) fn observe_current_voting<'a>( for peer in std::iter::once(first).chain(expected.keys().copied().filter(|peer| *peer != first)) { + // One lost metadata reply must leave time for the owning walk to + // retry. Divide only this read-only observation budget; the final + // validation and original freshness/deadline checks stay intact. + let shares = + u32::try_from(4 - observations.len()).map_err(|_| FleetCatalogError::Limit)?; + let timeout = deadline + .remaining()? + .checked_div(shares) + .filter(|budget| budget.as_millis() > 0) + .ok_or(FleetCatalogError::Unconfirmed)?; let observation = observer .observe_peer_fleet_membership( peer, authorization.clone(), snapshot.revision(), - deadline.remaining()?, + timeout, ) .await .map_err(FleetCatalogError::Observation)?; @@ -380,9 +393,7 @@ pub(in crate::raft::fleet_catalog::maintenance) fn observe_current_voting<'a>( if observations.len() != 3 { return Err(FleetCatalogError::InventoryIncomplete); } - let fixed = observations[0] - .published_cut() - .ok_or(FleetCatalogError::Unconfirmed)?; + let fixed = observations[0].published_cut(); let result = catalog .run(deadline.remaining()?, async { let first = observations[0].membership(); @@ -415,9 +426,7 @@ pub(in crate::raft::fleet_catalog::maintenance) fn observe_current_voting<'a>( state, replica.node == target && paused, )?; - let cut = observed - .published_cut() - .ok_or(FleetCatalogError::Unconfirmed)?; + let cut = observed.published_cut(); let member = observed .progress() .member @@ -476,7 +485,7 @@ pub(in crate::raft::fleet_catalog::maintenance) fn observe_current_voting<'a>( ), epoch: auth.epoch(), member: replica.member.get(), - publication: hash(&encode(57, &Publication::from(cut), records::LIMIT)?), + publication: hash(&encode(57, &Publication::from(cut), PAGE_LIMIT)?), leader_load: observed .progress() .leader_load diff --git a/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/reconcile.rs b/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/reconcile.rs index 03f69b0cd..5278cb8b8 100644 --- a/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/reconcile.rs +++ b/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/reconcile.rs @@ -9,13 +9,13 @@ use super::*; pub enum FleetRestartRuntimeProgress { /// Transfer admission only. A later turn must observe the survivor leader. HandoffRequested { - group: GroupScope, + group: GroupHistory, from: MemberId, to: MemberId, term: cellar_logic::consensus::Term, }, Materialized { - group: GroupScope, + group: GroupHistory, node: NodeId, member: MemberId, previously_present: bool, diff --git a/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/records.rs b/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/records.rs index c49ae6b3e..e86584bd2 100644 --- a/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/records.rs +++ b/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/records.rs @@ -20,7 +20,7 @@ impl From for Physical { fn from(value: RaftPhysicalHistory) -> Self { Self { member: value.member().get(), - in_configuration: value.in_configuration(), + in_configuration: value.in_configuration() && !value.parked_retired_source(), completed_sequence: value.completed_sequence(), } } diff --git a/crates/cellar/durability/raft/fleet_catalog/maintenance/settle.rs b/crates/cellar/durability/raft/fleet_catalog/maintenance/settle.rs index 08f972647..21e941f2f 100644 --- a/crates/cellar/durability/raft/fleet_catalog/maintenance/settle.rs +++ b/crates/cellar/durability/raft/fleet_catalog/maintenance/settle.rs @@ -3,8 +3,8 @@ //! Explicit bounded controller turns; durable roots own ambiguous work. use super::*; use crate::application::consensus::RaftCells; -use crate::raft::catalog::NamespaceCatalog; -use cellar_logic::consensus::GroupScope; +use crate::raft::catalog::RangeDirectory; +use cellar_logic::cell_group::GroupHistory; use rand::RngCore; use std::pin::Pin; @@ -22,7 +22,7 @@ pub enum FleetRestartBirthResolution { #[derive(Clone, Debug, PartialEq, Eq)] pub struct FleetRestartBirthAction { pub admission: [u8; 16], - pub group: GroupScope, + pub group: GroupHistory, pub resolution: FleetRestartBirthResolution, } @@ -31,11 +31,11 @@ pub struct FleetRestartBirthAction { pub enum FleetRestartSettlementBlocker { ResourceOnly { reservation: [u8; 16], - group: GroupScope, + group: GroupHistory, }, MovementServicesUnavailable { reservation: [u8; 16], - group: GroupScope, + group: GroupHistory, }, } @@ -46,7 +46,7 @@ pub enum FleetRestartSettlementProgress { Birth { admission: [u8; 16], resolution: FleetRestartBirthResolution, - group: GroupScope, + group: GroupHistory, state: FleetHistoryState, applied_revision: u64, observed_revision: u64, @@ -86,7 +86,7 @@ impl FleetCatalog { &'a self, operation: ProposalId, resolution: FleetRestartBirthResolution, - namespaces: &'a [NamespaceCatalog], + namespaces: &'a [RangeDirectory], observer: Option<&'a RaftCells>, evidence: Option<&'a dyn FleetMovementEvidence>, timeout: Duration, @@ -115,7 +115,7 @@ impl FleetCatalog { } else if let Some(scope) = work.next_birth { let namespace = namespaces .iter() - .find(|n| n.config().namespace == scope.namespace) + .find(|n| n.namespace_config().namespace == scope.namespace) .ok_or(FleetCatalogError::InventoryIncomplete)?; self.recover_birth_reservation(&snapshot, namespace, &scope, deadline.remaining()?) .await?; @@ -132,7 +132,7 @@ impl FleetCatalog { return Ok(FleetRestartSettlementProgress::Blocked( FleetRestartSettlementBlocker::MovementServicesUnavailable { reservation: movement.reservation, - group: movement.group.clone(), + group: movement.group, }, )); }; @@ -157,7 +157,7 @@ impl FleetCatalog { return Ok(FleetRestartSettlementProgress::Blocked( FleetRestartSettlementBlocker::ResourceOnly { reservation: reservation.reservation, - group: reservation.group.clone(), + group: reservation.group, }, )); } else { @@ -177,36 +177,38 @@ impl FleetCatalog { .await?; let namespace = namespaces .iter() - .find(|n| n.config().namespace == scope.namespace) + .find(|n| n.namespace_config().namespace == scope.namespace) .ok_or(FleetCatalogError::InventoryIncomplete)?; let authorization = self .recover_birth_reservation(&snapshot, namespace, &scope, deadline.remaining()?) .await?; let namespace_timeout = deadline.remaining()?; let decision = async { - use crate::raft::catalog::{BirthDisposition, CatalogError}; + use crate::raft::catalog::{CatalogError, CellGroupBirthDisposition}; match action.resolution { FleetRestartBirthResolution::Cancel => { namespace - .cancel_managed_birth( + .cancel_fleet_group_birth( &authorization, ProposalId::new(action.request.id) .map_err(|_| CatalogError::Corrupt("restart birth ID"))?, - namespace_timeout, ) .await } FleetRestartBirthResolution::Finish => { - if let Some(winner) = namespace.birth_disposition(&scope.cell).await? { + if let Some(winner) = namespace + .group_birth_disposition(&authorization.proposal()) + .await? + { return Ok(winner); } match namespace - .resolve_managed_birth(&authorization, namespace_timeout) + .materialize_fleet_group_birth(&authorization) .await { - Ok(birth) => Ok(BirthDisposition::Live(birth)), + Ok(birth) => Ok(CellGroupBirthDisposition::Live(birth)), Err(CatalogError::BirthCancelled) => namespace - .birth_disposition(&scope.cell) + .group_birth_disposition(&authorization.proposal()) .await? .ok_or(CatalogError::Unconfirmed), Err(error) => Err(error), diff --git a/crates/cellar/durability/raft/fleet_catalog/maintenance/settlement.rs b/crates/cellar/durability/raft/fleet_catalog/maintenance/settlement.rs index feaa86bdd..ba0f67096 100644 --- a/crates/cellar/durability/raft/fleet_catalog/maintenance/settlement.rs +++ b/crates/cellar/durability/raft/fleet_catalog/maintenance/settlement.rs @@ -2,7 +2,7 @@ //! Complete rooted pending work, distinct from live survivor/stop readiness. use super::*; -use cellar_logic::consensus::GroupScope; +use cellar_logic::cell_group::GroupHistory; /// Descriptive work at one current root. Memory is bounded to the next birth, /// one owned birth action and the active resource book (at most two rows). An empty result proves no @@ -11,7 +11,7 @@ use cellar_logic::consensus::GroupScope; pub struct FleetRestartPendingWork { pub observed_revision: u64, pub pending_births: u64, - pub next_birth: Option, + pub next_birth: Option, pub active_birth: Option, pub movements: Vec, pub reservations: Vec, @@ -25,7 +25,7 @@ impl FleetCatalog { &self, snapshot: &FleetSnapshot, operation: ProposalId, - namespaces: &[NamespaceCatalog], + namespaces: &[RangeDirectory], timeout: Duration, ) -> Result { self.check_snapshot(snapshot)?; diff --git a/crates/cellar/durability/raft/fleet_catalog/managed_park.rs b/crates/cellar/durability/raft/fleet_catalog/managed_park.rs new file mode 100644 index 000000000..0adfc6fe9 --- /dev/null +++ b/crates/cellar/durability/raft/fleet_catalog/managed_park.rs @@ -0,0 +1,550 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! The rooted exclusion for one exact managed park negotiation. + +use super::disruption::{Disruption, SLOT_KEY}; +use super::histories::{history_mutation, validate_request}; +use super::managed_transitions::root_digest; +use super::resources::Authority; +use super::*; +use crate::raft::{ + cell_groups::{CellGroupQuorumMembership, ParkPrepareRequest}, + fleet_stop::BootBinding, + member_state::FleetGroupBinding, + transport::ParkDeliveryReceipt, +}; +use cellar_logic::{ + cell_group::{GroupHistory, MembershipRevision, ParkGeneration}, + consensus::{LogIndex, LogPosition, MemberId, ProposalId, Term}, +}; +use serde::Deserialize; + +pub(super) const KIND: u8 = 202; +const SETTLEMENT_KIND: u8 = 203; +const OPERATION_PREFIX: &str = "managed-park/"; +const SETTLEMENT_PREFIX: &str = "managed-park-settled/"; +const RECORD_LIMIT: usize = 4096; + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Record { + request: FleetRequestIdentity, + input_root: [u8; 32], + group: FleetGroupBinding, + birth: [u8; 32], + schema: [u8; 32], + boot: BootBinding, + member: u64, + base: u64, + term: u64, + membership: u64, +} + +impl Record { + pub(super) fn validate(&self, fleet: &FleetCatalog) -> Result<()> { + validate_request(&self.request)?; + self.boot + .validate() + .map_err(|_| FleetCatalogError::Invalid("park boot"))?; + let history = self.history()?; + if self.group != fleet.binding(&history)? + || self.boot.catalog != self.group.catalog() + || self.boot.fleet != self.group.fleet() + || [self.input_root, self.birth, self.schema].contains(&[0; 32]) + || self.member == 0 + || self.term == 0 + || self.term >= u64::MAX - 2 + || self.base >= u64::MAX - 2 + || self.membership == 0 + || self.membership > self.base + { + return Err(FleetCatalogError::Invalid("managed park binding")); + } + Ok(()) + } + + fn history(&self) -> Result { + self.group + .scope() + .map_err(|_| FleetCatalogError::Invalid("park history")) + } + + pub(super) fn operation(&self) -> [u8; 16] { + self.request.operation + } + + pub(super) fn key(&self) -> String { + key(OPERATION_PREFIX, self.operation()) + } + + pub(super) fn check_basis(&self, snapshot: &FleetSnapshot) -> Result<()> { + if self.request.expected_revision != snapshot.revision() + || self.input_root != root_digest(snapshot)? + { + return Err(FleetCatalogError::Conflict); + } + Ok(()) + } + + pub(super) fn mutation(&self) -> Result { + let value = encode(KIND, self, RECORD_LIMIT)?; + Ok(history_mutation( + &self.request, + vec![ + FleetChange { + table: FleetTable::Operations, + key: self.key(), + value: Some(value.clone()), + }, + FleetChange { + table: FleetTable::Policy, + key: SLOT_KEY.into(), + value: Some(value), + }, + ], + )) + } + + fn settlement(&self, proof: ManagedParkSettlement<'_>) -> Result { + match proof { + ManagedParkSettlement::Delivered(receipt) => { + let leader = receipt.leader(); + let cut = receipt.cut(); + if !self.group.matches_scope(&leader.scope) + || leader.member.get() != self.member + || leader.node.bytes() != self.boot.node + || leader.disk.bytes() != self.boot.disk + || leader.process.bytes() != self.boot.process + || receipt.engine().bytes() != self.boot.engine + || cut.generation() != self.base + 1 + || cut.position().term.get() != self.term + || cut.membership_revision() != self.membership + { + return Err(FleetCatalogError::Conflict); + } + Ok(SettlementEvidence::Delivered) + } + ManagedParkSettlement::Advanced(observed) => { + let root = observed.root(); + let at = root.position(); + if !self.group.matches_scope(&observed.replica().scope) + || root.route().birth_digest != self.birth + || root.route().schema_digest != self.schema + || !observed.status().ready + || observed.status().leader != Some(observed.replica().member.get()) + || !fences_park(self.base, self.term, at) + { + return Err(FleetCatalogError::Conflict); + } + Ok(SettlementEvidence::Advanced { + index: at.index.get(), + term: at.term.get(), + }) + } + } + } +} + +fn fences_park(base: u64, term: u64, position: LogPosition) -> bool { + position.term.get() > term + || (position.term.get() == term && position.index.get() > base.saturating_add(1)) +} + +fn key(prefix: &str, operation: [u8; 16]) -> String { + format!("{prefix}{}", hex::encode(operation)) +} + +pub(crate) struct ManagedParkPermit(Record); + +impl ManagedParkPermit { + /// A later actor turn rechecks the exact selected base. The rooted + /// operation cannot be reused at a different park generation. + pub(crate) fn prepare_request(&self, deadline_ms: u64) -> Result { + let term = self.term(); + Ok(ParkPrepareRequest { + proposal: ProposalId::new(self.0.operation()) + .map_err(|_| FleetCatalogError::Invalid("park operation"))?, + term, + membership: MembershipRevision::new(self.0.membership) + .map_err(|_| FleetCatalogError::Invalid("park membership revision"))?, + expected_applied: LogPosition { + index: LogIndex::new(self.0.base), + term, + }, + deadline_ms, + }) + } + + pub(crate) fn term(&self) -> Term { + Term::new(self.0.term) + } + + pub(crate) fn operation(&self) -> Result { + ProposalId::new(self.0.operation()) + .map_err(|_| FleetCatalogError::Invalid("park operation")) + } + + pub(crate) fn generation(&self) -> Result { + ParkGeneration::new(self.0.base + 1) + .map_err(|_| FleetCatalogError::Invalid("park generation")) + } + + pub(crate) fn matches_request( + &self, + history: GroupHistory, + request: &FleetRequestIdentity, + ) -> bool { + self.0.group.matches_scope(&history) && &self.0.request == request + } + + pub(crate) fn same_reservation(&self, other: &Self) -> bool { + self.0 == other.0 + } + + pub(crate) fn matches_fleet(&self, history: GroupHistory, boot: &BootBinding) -> bool { + self.0.group.matches_scope(&history) + && self.0.boot.fleet == boot.fleet + && self.0.boot.catalog == boot.catalog + } + + pub(crate) fn matches_owner( + &self, + history: GroupHistory, + boot: &BootBinding, + member: MemberId, + ) -> bool { + self.0.group.matches_scope(&history) + && &self.0.boot == boot + && self.0.member == member.get() + } +} + +#[derive(Clone, Copy, PartialEq, Eq)] +pub(crate) enum ManagedParkPhase { + Negotiating, + Delivered, + Superseded, +} + +pub(crate) struct ManagedParkOperation { + pub(crate) permit: ManagedParkPermit, + pub(crate) phase: ManagedParkPhase, + pub(crate) revision: u64, +} + +pub(crate) enum ManagedParkSettlement<'a> { + Delivered(&'a ParkDeliveryReceipt), + Advanced(&'a CellGroupQuorumMembership), +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case", tag = "state", deny_unknown_fields)] +enum SettlementEvidence { + Delivered, + Advanced { index: u64, term: u64 }, +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Settlement { + admission: [u8; 32], + request: FleetRequestIdentity, + evidence: SettlementEvidence, +} + +impl Settlement { + fn mutation(&self) -> Result { + Ok(history_mutation( + &self.request, + vec![ + FleetChange { + table: FleetTable::Operations, + key: key(SETTLEMENT_PREFIX, self.request.operation), + value: Some(encode(SETTLEMENT_KIND, self, RECORD_LIMIT)?), + }, + FleetChange { + table: FleetTable::Policy, + key: SLOT_KEY.into(), + value: None, + }, + ], + )) + } +} + +impl FleetCatalog { + pub(crate) async fn admit_managed_park( + &self, + snapshot: &FleetSnapshot, + request: &FleetRequestIdentity, + observed: &CellGroupQuorumMembership, + timeout: Duration, + ) -> Result { + self.check_snapshot(snapshot)?; + validate_request(request)?; + let deadline = deadline::Deadline::new(self, timeout)?; + if self + .active_topology_operation(snapshot, deadline.remaining()?) + .await? + .is_some() + { + return Err(FleetCatalogError::Conflict); + } + self.run( + deadline.remaining()?, + Box::pin(async { + self.require_transition_membership_idle(snapshot).await?; + if let Some(active) = self.active_disruption(snapshot).await? { + return Err(FleetCatalogError::DisruptionInProgress { + operation: active.operation(), + }); + } + let replica = observed.replica(); + let group = self.binding(&replica.scope)?; + let group_key = group.key().map_err(|_| FleetCatalogError::Conflict)?; + let authorization = self + .authorize_live_group(snapshot, group.clone(), &group_key) + .await?; + let host = self.host_from(snapshot, replica.node).await?; + let boot = host.boot_binding()?; + let root = observed.root(); + let at = root.position(); + let conf = observed.configuration(); + if host.state() != FleetNodeState::Active + || host.is_isolated() + || boot.disk != replica.disk.bytes() + || boot.process != replica.process.bytes() + || boot.engine != observed.engine().bytes() + || authorization.history_digests() + != (root.route().birth_digest, root.route().schema_digest) + || !authorization.matches_replica(replica, observed.membership()) + || observed + .membership() + .fleet_status() + .is_none_or(|status| status.binding != group) + || !root.is_active() + || !observed.status().ready + || observed.status().leader != Some(replica.member.get()) + || observed.status().committed != at.index + || !conf.voters.contains(&replica.member.get()) + || !conf.voters_outgoing.is_empty() + || !conf.learners.is_empty() + || !conf.learners_next.is_empty() + || conf.auto_leave + { + return Err(FleetCatalogError::Conflict); + } + let record = Record { + request: request.clone(), + input_root: root_digest(snapshot)?, + group, + birth: root.route().birth_digest, + schema: root.route().schema_digest, + boot, + member: replica.member.get(), + base: at.index.get(), + term: at.term.get(), + membership: observed.membership().revision().get(), + }; + record.validate(self)?; + record.check_basis(snapshot)?; + if self + .inner + .lookup( + snapshot.head.tables[FleetTable::Operations.slot()], + &record.key(), + ) + .await? + .is_some() + { + return Err(FleetCatalogError::PayloadConflict); + } + let mutation = record.mutation()?; + mutation.validate()?; + self.commit_with_disruption( + snapshot, + &mutation, + None, + Authority::ManagedPark, + || Ok(()), + ) + .await?; + Ok(ManagedParkPermit(record)) + }), + ) + .await + } + + pub(crate) async fn reopen_managed_park( + &self, + operation: ProposalId, + timeout: Duration, + ) -> Result> { + self.run(timeout, async { + let current = self.inner.current().await?; + match self.active_disruption(¤t).await? { + Some(Disruption::Park(record)) if record.operation() == operation.bytes() => { + Ok(Some(ManagedParkPermit(record))) + } + _ => Ok(None), + } + }) + .await + } + + pub(crate) async fn observe_managed_park( + &self, + operation: ProposalId, + timeout: Duration, + ) -> Result> { + self.run(timeout, Box::pin(async { + let current = self.inner.current().await?; + let operations = current.head.tables[FleetTable::Operations.slot()]; + let Some(bytes) = self.inner.lookup(operations, &key(OPERATION_PREFIX, operation.bytes())).await? else { + return Ok(None); + }; + let record: Record = decode(KIND, &bytes, RECORD_LIMIT)?; + record.validate(self)?; + self.confirm_managed_park(¤t, &record).await?; + if record.operation() != operation.bytes() { + return Err(FleetCatalogError::Corrupt("park operation index")); + } + let terminal = self.inner.lookup(operations, &key(SETTLEMENT_PREFIX, operation.bytes())).await?; + let phase = if let Some(terminal) = terminal { + let terminal: Settlement = decode(SETTLEMENT_KIND, &terminal, RECORD_LIMIT)?; + validate_request(&terminal.request)?; + let mutation = terminal.mutation()?; + let digest = hash(&encode(REQUEST_KIND, &mutation, REQUEST_LIMIT)?); + if terminal.admission != hash(&bytes) + || terminal.request.operation != operation.bytes() + || terminal.request.principal != record.request.principal + || terminal.request.id == record.request.id + || terminal.request.expected_revision <= record.request.expected_revision + || self.inner.committed(¤t.head, &hex::encode(terminal.request.id), digest).await?.is_none() + { + return Err(FleetCatalogError::Corrupt("park settlement index")); + } + match terminal.evidence { + SettlementEvidence::Delivered => ManagedParkPhase::Delivered, + SettlementEvidence::Advanced { index, term } if fences_park( + record.base, record.term, + LogPosition { index: LogIndex::new(index), term: Term::new(term) }, + ) => ManagedParkPhase::Superseded, + SettlementEvidence::Advanced { .. } => return Err(FleetCatalogError::Corrupt("park settlement cut")), + } + } else { + if current.head.disruption != Some(hash(&bytes)) + || !matches!(self.active_disruption(¤t).await?, Some(Disruption::Park(ref active)) if *active == record) + { + return Err(FleetCatalogError::Corrupt("unsettled park missing disruption")); + } + ManagedParkPhase::Negotiating + }; + Ok(Some(ManagedParkOperation { permit: ManagedParkPermit(record), phase, revision: current.revision() })) + })).await + } + + pub(crate) async fn settle_managed_park( + &self, + snapshot: &FleetSnapshot, + request: &FleetRequestIdentity, + permit: &ManagedParkPermit, + proof: ManagedParkSettlement<'_>, + timeout: Duration, + ) -> Result { + self.check_snapshot(snapshot)?; + validate_request(request)?; + let record = &permit.0; + record.validate(self)?; + if request.operation != record.operation() + || request.principal != record.request.principal + || request.id == record.request.id + || request.expected_revision <= record.request.expected_revision + { + return Err(FleetCatalogError::Invalid("park settlement request")); + } + let settlement = Settlement { + admission: hash(&encode(KIND, record, RECORD_LIMIT)?), + request: request.clone(), + evidence: record.settlement(proof)?, + }; + let mutation = settlement.mutation()?; + mutation.validate()?; + self.run(timeout, Box::pin(async { + let current = self.inner.current().await?; + let digest = hash(&encode(REQUEST_KIND, &mutation, REQUEST_LIMIT)?); + if let Some(applied_revision) = self.inner.committed(¤t.head, &hex::encode(request.id), digest).await? { + return Ok(FleetCommit { snapshot: current, applied_revision }); + } + if current.token != snapshot.token || request.expected_revision != current.revision() { + return Err(FleetCatalogError::Conflict); + } + if !matches!(self.active_disruption(¤t).await?, Some(Disruption::Park(ref active)) if active == record) + || current.head.disruption != Some(settlement.admission) + { + return Err(FleetCatalogError::Conflict); + } + self.commit_with_disruption( + snapshot, &mutation, Some(settlement.admission), Authority::ManagedPark, || Ok(()), + ).await + })).await + } + + pub(super) async fn confirm_managed_park( + &self, + snapshot: &FleetSnapshot, + record: &Record, + ) -> Result<()> { + let digest = hash(&encode(REQUEST_KIND, &record.mutation()?, REQUEST_LIMIT)?); + if self + .inner + .committed(&snapshot.head, &hex::encode(record.request.id), digest) + .await? + .is_none() + { + return Err(FleetCatalogError::Corrupt("managed park journal")); + } + Ok(()) + } + + pub(super) async fn check_park_mutation( + &self, + snapshot: &FleetSnapshot, + mutation: &FleetMutation, + authority: Authority, + ) -> Result<()> { + if authority == Authority::ManagedPark { + return Ok(()); + } + if mutation.changes.iter().any(|change| { + change.table == FleetTable::Operations + && (change.key.starts_with(OPERATION_PREFIX) + || change.key.starts_with(SETTLEMENT_PREFIX)) + }) { + return Err(FleetCatalogError::Invalid( + "managed park operation authority", + )); + } + if snapshot.head.disruption.is_some() + && (matches!( + authority, + Authority::Topology | Authority::TopologyAdmission + ) || mutation.changes.iter().any(|change| match change.table { + FleetTable::Histories | FleetTable::Admissions => true, + FleetTable::Policy => change.key == SLOT_KEY || change.key == namespaces::KEY, + _ => false, + })) + { + if let Some(Disruption::Park(record)) = self.active_disruption(snapshot).await? { + return Err(FleetCatalogError::DisruptionInProgress { + operation: record.operation(), + }); + } + } + Ok(()) + } +} + +#[cfg(test)] +#[path = "managed_park_tests.rs"] +mod tests; diff --git a/crates/cellar/durability/raft/fleet_catalog/managed_park_tests.rs b/crates/cellar/durability/raft/fleet_catalog/managed_park_tests.rs new file mode 100644 index 000000000..19a647e74 --- /dev/null +++ b/crates/cellar/durability/raft/fleet_catalog/managed_park_tests.rs @@ -0,0 +1,163 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; + +#[test] +fn park_settlement_needs_a_later_entry_or_term() { + let position = |index, term| LogPosition { + index: LogIndex::new(index), + term: Term::new(term), + }; + assert!(!fences_park(10, 7, position(12, 6))); + assert!(!fences_park(10, 7, position(10, 7))); + assert!(!fences_park(10, 7, position(11, 7))); + assert!(fences_park(10, 7, position(12, 7))); + assert!(fences_park(10, 7, position(11, 8))); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn managed_park_root_refuses_generic_writers_and_reopens_exact_owner() -> anyhow::Result<()> { + use crate::{asyncrt::ExecutionDomain, bucket::StorageBackend}; + use cellar_logic::consensus::{ClusterId, GroupId, IncarnationId, NamespaceId}; + use object_store::list::{PaginatedListOptions, PaginatedListResult, PaginatedListStore}; + + struct NoListing; + + #[async_trait::async_trait] + impl PaginatedListStore for NoListing { + async fn list_paginated( + &self, + _: Option<&str>, + _: PaginatedListOptions, + ) -> object_store::Result { + Err(object_store::Error::NotSupported { + source: "park admission uses exact keys".into(), + }) + } + } + + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let memory = Arc::new(object_store::memory::InMemory::new()); + let bucket = Bucket::from_clients( + memory.clone(), + memory, + Arc::new(NoListing), + StorageBackend::S3, + "park-catalog".into(), + "tenant/".into(), + )?; + let history = GroupHistory { + cluster: ClusterId::new([1; 16])?, + namespace: NamespaceId::new([2; 16])?, + group: GroupId::new([3; 16])?, + incarnation: IncarnationId::new([4; 16])?, + }; + let catalog = FleetCatalog::new( + &domain, + bucket, + history.cluster, + IncarnationId::new([5; 16])?, + "nodes/", + )?; + let timeout = Duration::from_secs(5); + let initial = catalog.initialize(timeout).await?; + let group = catalog.binding(&history)?; + let record = Record { + request: FleetRequestIdentity { + id: [6; 16], + operation: [7; 16], + expected_revision: initial.revision(), + principal: "operator".into(), + }, + input_root: root_digest(&initial)?, + group: group.clone(), + birth: [8; 32], + schema: [9; 32], + boot: BootBinding { + catalog: group.catalog(), + fleet: group.fleet(), + node: [10; 16], + disk: [11; 16], + engine: [12; 16], + boot: [13; 16], + process: [14; 16], + epoch: 1, + join_key: ed25519_dalek::SigningKey::from_bytes(&[15; 32]) + .verifying_key() + .to_bytes(), + }, + member: 1, + base: 10, + term: 7, + membership: 1, + }; + let mutation = record.mutation()?; + assert!(matches!( + catalog.commit(&initial, &mutation, timeout).await, + Err(FleetCatalogError::Invalid( + "managed park operation authority" + )) + )); + + // Exercise the actual catalog CAS with an internal, authorized record. + // The production entry point additionally requires a live quorum sample. + let committed = catalog + .run( + timeout, + catalog.commit_with_disruption( + &initial, + &mutation, + None, + Authority::ManagedPark, + || Ok(()), + ), + ) + .await?; + let operation = ProposalId::new(record.operation())?; + let reopened = catalog + .reopen_managed_park(operation, timeout) + .await? + .expect("active rooted park"); + assert!(reopened.matches_owner(history, &record.boot, MemberId::new(1)?)); + let observed = catalog + .observe_managed_park(operation, timeout) + .await? + .expect("rooted park history"); + assert!(observed.phase == ManagedParkPhase::Negotiating); + assert_eq!(observed.revision, committed.snapshot.revision()); + assert!(observed.permit.same_reservation(&reopened)); + assert!(observed.permit.matches_request(history, &record.request)); + assert_eq!(observed.permit.generation()?.get(), 11); + let request = reopened.prepare_request(100)?; + assert_eq!(request.proposal, operation); + assert_eq!(request.expected_applied.index, LogIndex::new(10)); + assert_eq!(request.expected_applied.term, Term::new(7)); + assert_eq!(request.membership.get(), 1); + + let mutate = |table, key: &str| FleetMutation { + id: [16; 16], + operation: [17; 16], + principal: "other".into(), + expected_revision: committed.snapshot.revision(), + changes: vec![FleetChange { + table, + key: key.into(), + value: None, + }], + }; + for (table, key) in [ + (FleetTable::Policy, SLOT_KEY), + (FleetTable::Admissions, "membership"), + ] { + assert!(matches!( + catalog.commit(&committed.snapshot, &mutate(table, key), timeout).await, + Err(FleetCatalogError::DisruptionInProgress { operation }) if operation == record.operation() + )); + } + assert_eq!( + catalog.read(timeout).await?.revision(), + committed.snapshot.revision() + ); + domain.shutdown().await; + Ok(()) +} diff --git a/crates/cellar/durability/raft/fleet_catalog/managed_recovery_fence.rs b/crates/cellar/durability/raft/fleet_catalog/managed_recovery_fence.rs new file mode 100644 index 000000000..33c9215bf --- /dev/null +++ b/crates/cellar/durability/raft/fleet_catalog/managed_recovery_fence.rs @@ -0,0 +1,345 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Exact infrastructure receipts for an interrupted managed transition. +use super::{histories::catalog_error, managed_transitions::transfer_archive_error, *}; +use crate::raft::{ + catalog::RangeDirectory, + cell_groups::{CellGroupArchiveCoverage, CellGroupArchivePool, CellGroupArchiveRegistry}, +}; +use cellar_logic::consensus::ProposalId; +use serde::Deserialize; + +const KIND: u8 = 223; +const LIMIT: usize = 20 * 1024; +const MAX_EVIDENCE: usize = 4096; + +/// Bind the infrastructure fence to one durable intent and its directory barrier. +/// The authority must also consult its complete inventory of possible old writers. +pub struct FleetTransitionRecoveryFenceRequest { + intent: FleetTransitionRecoveryIntent, + barrier: FleetTransitionRecoveryBarrier, + digest: [u8; 32], +} + +impl FleetTransitionRecoveryFenceRequest { + pub fn intent(&self) -> &FleetTransitionRecoveryIntent { + &self.intent + } + + pub fn barrier(&self) -> FleetTransitionRecoveryBarrier { + self.barrier + } + + pub fn digest(&self) -> [u8; 32] { + self.digest + } +} + +/// Read-back proof of a verified permanent infrastructure fence. Retains the +/// independently replayed source images for a subsequent recovery stage. +pub struct FleetFencedTransitionRecovery { + request: FleetTransitionRecoveryFenceRequest, + receipt_digest: [u8; 32], + archives: Vec, +} + +impl FleetFencedTransitionRecovery { + pub fn request(&self) -> &FleetTransitionRecoveryFenceRequest { + &self.request + } + + pub fn receipt_digest(&self) -> [u8; 32] { + self.receipt_digest + } + + pub fn archives(&self) -> &[CellGroupArchiveCoverage] { + &self.archives + } +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Receipt { + request: [u8; 32], + evidence: Vec, +} + +pub(crate) struct PreparedManagedRecoveryFence { + request: FleetTransitionRecoveryFenceRequest, + cuts: Vec<(CellGroupArchiveRegistry, CellGroupArchiveCoverage)>, + receipt: Option, +} + +impl PreparedManagedRecoveryFence { + pub(crate) fn request(&self) -> &FleetTransitionRecoveryFenceRequest { + &self.request + } + + pub(crate) fn evidence(&self) -> Option<&[u8]> { + self.receipt + .as_ref() + .map(|receipt| receipt.evidence.as_slice()) + } + + pub(crate) fn check_evidence(evidence: &[u8]) -> Result<()> { + if evidence.is_empty() || evidence.len() > MAX_EVIDENCE { + return Err(FleetCatalogError::Invalid( + "transition fence evidence length", + )); + } + Ok(()) + } +} + +impl FleetCatalog { + pub(crate) async fn prepare_managed_recovery_fence( + &self, + directory: &RangeDirectory, + operation: ProposalId, + principal: &str, + barrier: FleetTransitionRecoveryBarrier, + verifier: &FleetGroupArchiveVerifier, + timeout: Duration, + ) -> Result { + self.run(timeout, async { + let intent = self + .read_managed_recovery_intent(directory, operation, principal) + .await? + .ok_or(FleetCatalogError::Conflict)?; + if intent.digest() != barrier.intent_digest() { + return Err(FleetCatalogError::PayloadConflict); + } + let digest = managed_fence_digest(intent.digest(), barrier.directory_digest()); + let request = FleetTransitionRecoveryFenceRequest { + intent, + barrier, + digest, + }; + self.require_managed_fence_barrier(directory, &request) + .await?; + let receipt = self + .inner + .read::(&receipt_key(operation.bytes()), KIND, LIMIT) + .await? + .map(|(receipt, _)| receipt); + if let Some(receipt) = &receipt { + if receipt.request != request.digest { + return Err(FleetCatalogError::PayloadConflict); + } + PreparedManagedRecoveryFence::check_evidence(&receipt.evidence)?; + } + let transition = ProposalId::new(request.intent.transition()) + .map_err(|_| FleetCatalogError::Corrupt("recovery transition operation"))?; + let domain = &self.inner.domain; + let storage = directory.namespace_config().image_storage; + let mut cuts = Vec::with_capacity(request.intent.archive_holds().len()); + for hold in request.intent.archive_holds() { + let route = hold.marker().route(); + let registry = domain + .run(Box::pin(directory.open_managed_recovery_archive( + domain, + verifier.archives(), + route, + transition, + barrier.directory_digest(), + ))) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(transfer_archive_error)?; + registry.require_selected_hold(domain, hold).await?; + let receiver = verifier.receiver(domain, route, storage).await?; + let coverage = + Box::pin(registry.verify_hold(domain, hold, verifier.tails(), receiver)) + .await?; + cuts.push((registry, coverage)); + } + self.require_managed_fence_cuts(&cuts).await?; + self.require_managed_fence_barrier(directory, &request) + .await?; + Ok(PreparedManagedRecoveryFence { + request, + cuts, + receipt, + }) + }) + .await + } + + /// Persist independently verified evidence before checking the cuts again. + /// A late archive write may make the cut unusable, even though the fence + /// completed. Retain its receipt so that this outcome survives a restart. + pub(crate) async fn confirm_managed_recovery_fence( + &self, + directory: &RangeDirectory, + prepared: PreparedManagedRecoveryFence, + evidence: Vec, + timeout: Duration, + ) -> Result { + self.run(timeout, async { + let PreparedManagedRecoveryFence { + request, + cuts, + receipt, + } = prepared; + let record = Receipt { + request: request.digest, + evidence, + }; + let bytes = encode(KIND, &record, LIMIT)?; + if receipt.is_none() { + self.inner + .create_exact(&receipt_key(request.intent.operation()), &bytes, LIMIT) + .await?; + } + self.require_managed_fence_cuts(&cuts).await?; + self.require_managed_fence_barrier(directory, &request) + .await?; + Ok(FleetFencedTransitionRecovery { + request, + receipt_digest: hash(&bytes), + archives: cuts.into_iter().map(|(_, coverage)| coverage).collect(), + }) + }) + .await + } + + pub(super) async fn require_confirmed_managed_fence( + &self, + directory: &RangeDirectory, + fenced: &FleetFencedTransitionRecovery, + archives: &CellGroupArchivePool, + ) -> Result<()> { + let request = fenced.request(); + self.require_managed_fence_barrier(directory, request) + .await?; + self.require_durable_managed_fence( + request.intent.operation(), + request.digest, + fenced.receipt_digest(), + ) + .await?; + let transition = ProposalId::new(request.intent.transition()) + .map_err(|_| FleetCatalogError::Corrupt("recovery transition operation"))?; + let domain = &self.inner.domain; + for hold in request.intent.archive_holds() { + let registry = domain + .run(Box::pin(directory.open_managed_recovery_archive( + domain, + archives, + hold.marker().route(), + transition, + request.barrier.directory_digest(), + ))) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(transfer_archive_error)?; + registry.require_selected_hold(domain, hold).await?; + } + Ok(()) + } + + // Reconstruct the signed challenge after the directory has released its + // live fence. No archive, placement or mutation authority is returned. + pub(super) async fn read_recovery_retirement_fence( + &self, + intent: FleetTransitionRecoveryIntent, + plan: &FleetTransitionRecoveryReplacement, + ) -> Result<(FleetTransitionRecoveryFenceRequest, Vec)> { + let barrier = plan + .mutation() + .expected_directory_digest() + .ok_or(FleetCatalogError::Conflict)?; + let digest = managed_fence_digest(intent.digest(), barrier); + let (receipt, _) = self + .inner + .read::(&receipt_key(intent.operation()), KIND, LIMIT) + .await? + .ok_or(FleetCatalogError::Unconfirmed)?; + if receipt.request != digest + || hash(&encode(KIND, &receipt, LIMIT)?) != plan.receipt_digest() + { + return Err(FleetCatalogError::PayloadConflict); + } + PreparedManagedRecoveryFence::check_evidence(&receipt.evidence)?; + let barrier = FleetTransitionRecoveryBarrier::from_selection(intent.digest(), barrier); + Ok(( + FleetTransitionRecoveryFenceRequest { + intent, + barrier, + digest, + }, + receipt.evidence, + )) + } + + pub(super) async fn require_durable_managed_fence( + &self, + operation: [u8; 16], + request: [u8; 32], + expected: [u8; 32], + ) -> Result<()> { + let (receipt, _) = self + .inner + .read::(&receipt_key(operation), KIND, LIMIT) + .await? + .ok_or(FleetCatalogError::Unconfirmed)?; + if receipt.request != request || hash(&encode(KIND, &receipt, LIMIT)?) != expected { + return Err(FleetCatalogError::PayloadConflict); + } + Ok(()) + } + + async fn require_managed_fence_barrier( + &self, + directory: &RangeDirectory, + request: &FleetTransitionRecoveryFenceRequest, + ) -> Result<()> { + let intent = &request.intent; + let snapshot = self.inner.current().await?; + let permit = self + .managed_recovery_permit_at(&snapshot, directory, intent.basis()) + .await?; + let selected = self + .inner + .domain + .run(Box::pin(directory.observe_managed_recovery_barrier( + intent.digest(), + intent.basis(), + intent.archive_holds(), + &permit, + ))) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(catalog_error)?; + if selected != Some(request.barrier.directory_digest()) { + return Err(FleetCatalogError::Conflict); + } + Ok(()) + } + + async fn require_managed_fence_cuts( + &self, + cuts: &[(CellGroupArchiveRegistry, CellGroupArchiveCoverage)], + ) -> Result<()> { + for (registry, coverage) in cuts { + registry + .require_selected_hold(&self.inner.domain, coverage.hold()) + .await?; + } + Ok(()) + } +} + +pub(super) fn managed_fence_digest(intent: [u8; 32], barrier: [u8; 32]) -> [u8; 32] { + let mut binding = Vec::from(&b"cellar-managed-transition-fence-1"[..]); + binding.extend_from_slice(&intent); + binding.extend_from_slice(&barrier); + hash(&binding) +} + +fn receipt_key(operation: [u8; 16]) -> String { + format!( + "managed-transition-recovery-fences/{}", + hex::encode(operation) + ) +} diff --git a/crates/cellar/durability/raft/fleet_catalog/managed_recovery_intent.rs b/crates/cellar/durability/raft/fleet_catalog/managed_recovery_intent.rs new file mode 100644 index 000000000..5e77e0241 --- /dev/null +++ b/crates/cellar/durability/raft/fleet_catalog/managed_recovery_intent.rs @@ -0,0 +1,401 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Immutable recovery selections that survive controller loss. +use super::{ + histories::{catalog_error, validate_request}, + managed_transitions::root_digest, + *, +}; +use crate::raft::{ + catalog::RangeDirectory, + cell_groups::{CellGroupArchiveHold, CellGroupArchivePool}, +}; +use cellar_logic::{ + cell_group::GroupHistory, + consensus::{DiskLineage, GroupId, MemberId, NamespaceId, NodeId, ProposalId}, +}; +use serde::Deserialize; +use std::collections::BTreeSet; + +const KIND: u8 = 222; +const LIMIT: usize = 4 * 1024 * 1024; + +#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(crate) struct ManagedRecoveryIntentBasis { + pub(crate) transition: [u8; 16], + pub(crate) plan: [u8; 32], + pub(crate) fleet_root: [u8; 32], + pub(crate) fleet_revision: u64, + pub(crate) directory_root: [u8; 32], + pub(crate) publication: Option<[u8; 32]>, +} + +#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Lineage { + group: [u8; 16], + incarnation: [u8; 16], + member: u64, + node: [u8; 16], + disk: [u8; 16], + host: [u8; 16], +} + +impl From<&FleetRecoveryVoterLineage> for Lineage { + fn from(value: &FleetRecoveryVoterLineage) -> Self { + Self { + group: value.group.group.bytes(), + incarnation: value.group.incarnation.bytes(), + member: value.member.get(), + node: value.node.bytes(), + disk: value.disk.bytes(), + host: value.physical_host.bytes(), + } + } +} + +impl Lineage { + fn decode( + &self, + cluster: ClusterId, + namespace: NamespaceId, + ) -> Result { + let invalid = |_| FleetCatalogError::Corrupt("transition recovery lineage"); + Ok(FleetRecoveryVoterLineage { + group: GroupHistory { + cluster, + namespace, + group: GroupId::new(self.group).map_err(invalid)?, + incarnation: IncarnationId::new(self.incarnation).map_err(invalid)?, + }, + member: MemberId::new(self.member).map_err(invalid)?, + node: NodeId::new(self.node).map_err(invalid)?, + disk: DiskLineage::new(self.disk).map_err(invalid)?, + physical_host: IncarnationId::new(self.host).map_err(invalid)?, + }) + } +} + +#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Record { + catalog: [u8; 32], + namespace: [u8; 32], + operation: [u8; 16], + principal: String, + basis: ManagedRecoveryIntentBasis, + holds: Vec, + lineages: Vec, +} + +/// Exact durable input to a future fencing operation. Starting a fence +/// requires its own ownership barrier and a fresh verification of these cuts. +pub struct FleetTransitionRecoveryIntent { + record: Record, + digest: [u8; 32], + lineages: Vec, +} + +/// Read-back proof of the exact recovery barrier selected in the directory. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct FleetTransitionRecoveryBarrier { + intent_digest: [u8; 32], + directory_digest: [u8; 32], +} + +impl FleetTransitionRecoveryBarrier { + pub(super) fn from_selection(intent_digest: [u8; 32], directory_digest: [u8; 32]) -> Self { + Self { + intent_digest, + directory_digest, + } + } + + pub fn intent_digest(&self) -> [u8; 32] { + self.intent_digest + } + + pub fn directory_digest(&self) -> [u8; 32] { + self.directory_digest + } +} + +impl FleetTransitionRecoveryIntent { + pub(super) fn principal(&self) -> &str { + &self.record.principal + } + + pub(super) fn basis(&self) -> &ManagedRecoveryIntentBasis { + &self.record.basis + } + + pub fn operation(&self) -> [u8; 16] { + self.record.operation + } + + pub fn transition(&self) -> [u8; 16] { + self.record.basis.transition + } + + pub fn digest(&self) -> [u8; 32] { + self.digest + } + + pub fn archive_holds(&self) -> &[CellGroupArchiveHold] { + &self.record.holds + } + + pub fn required_voter_lineages(&self) -> &[FleetRecoveryVoterLineage] { + &self.lineages + } +} + +impl Record { + fn validate( + self, + fleet: &FleetCatalog, + directory: &RangeDirectory, + operation: ProposalId, + principal: &str, + ) -> Result { + validate_request(&FleetRequestIdentity { + id: self.operation, + operation: self.operation, + principal: self.principal.clone(), + expected_revision: self.basis.fleet_revision, + })?; + if self.catalog != fleet.catalog_id()? + || self.namespace != directory.namespace().root_digest() + || self.operation != operation.bytes() + || self.principal != principal + { + return Err(FleetCatalogError::Conflict); + } + let basis = &self.basis; + if basis.transition == [0; 16] + || basis.transition == self.operation + || [basis.plan, basis.fleet_root, basis.directory_root].contains(&[0; 32]) + || basis.publication == Some([0; 32]) + || self.holds.is_empty() + || self.holds.len() > 64 + || self.lineages.is_empty() + || self.lineages.len() > super::managed_recovery_inventory::MAX_RECOVERY_LINEAGES + { + return Err(FleetCatalogError::Invalid("transition recovery intent")); + } + let config = directory.namespace_config(); + let lineages: Vec<_> = self + .lineages + .iter() + .map(|lineage| lineage.decode(config.cluster, config.namespace)) + .collect::>()?; + if lineages.windows(2).any(|pair| pair[0] >= pair[1]) { + return Err(FleetCatalogError::Corrupt( + "transition recovery lineage order", + )); + } + let groups: BTreeSet<_> = lineages.iter().map(|lineage| lineage.group).collect(); + let mut held = BTreeSet::new(); + for hold in &self.holds { + hold.check_reference()?; + let group = hold.marker().route().history; + if !groups.contains(&group) || !held.insert(group) { + return Err(FleetCatalogError::Conflict); + } + } + let digest = hash(&encode(KIND, &self, LIMIT)?); + Ok(FleetTransitionRecoveryIntent { + record: self, + digest, + lineages, + }) + } +} + +pub(crate) struct ManagedRecoveryIntentInput<'a> { + pub(crate) operation: ProposalId, + pub(crate) principal: &'a str, + pub(crate) basis: &'a ManagedRecoveryIntentBasis, + pub(crate) holds: Vec, + pub(crate) lineages: &'a [FleetRecoveryVoterLineage], +} + +impl FleetCatalog { + pub(crate) async fn stage_managed_transition_recovery_intent( + &self, + directory: &RangeDirectory, + input: ManagedRecoveryIntentInput<'_>, + archives: &CellGroupArchivePool, + timeout: Duration, + ) -> Result { + self.run( + timeout, + Box::pin(async { + let record = Record { + catalog: self.catalog_id()?, + namespace: directory.namespace().root_digest(), + operation: input.operation.bytes(), + principal: input.principal.into(), + basis: input.basis.clone(), + holds: input.holds, + lineages: input.lineages.iter().map(Lineage::from).collect(), + }; + let intent = record.validate(self, directory, input.operation, input.principal)?; + let key = intent_key(input.operation); + if let Some((stored, _)) = self.inner.read::(&key, KIND, LIMIT).await? { + return if stored == intent.record { + Ok(intent) + } else { + Err(FleetCatalogError::PayloadConflict) + }; + } + let domain = &self.inner.domain; + let view = domain + .run(directory.observe()) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(catalog_error)?; + let current = self.inner.current().await?; + if view.digest() != input.basis.directory_root + || current.revision() != input.basis.fleet_revision + || root_digest(¤t)? != input.basis.fleet_root + { + return Err(FleetCatalogError::Conflict); + } + self.require_recovery_intent_holds(directory, archives, &intent) + .await?; + self.inner + .create_exact(&key, &encode(KIND, &intent.record, LIMIT)?, LIMIT) + .await?; + Ok(intent) + }), + ) + .await + } + + async fn require_recovery_intent_holds( + &self, + directory: &RangeDirectory, + archives: &CellGroupArchivePool, + intent: &FleetTransitionRecoveryIntent, + ) -> Result<()> { + let domain = &self.inner.domain; + for hold in &intent.record.holds { + let history = hold.marker().route().history; + let birth = domain + .run(directory.materialize_selected_history(history)) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(catalog_error)?; + let registry = directory + .open_group_archive(domain, archives, &birth) + .await + .map_err(super::managed_transitions::transfer_archive_error)?; + registry.require_selected_hold(domain, hold).await?; + } + Ok(()) + } + + pub(crate) async fn select_managed_transition_recovery_barrier( + &self, + directory: &RangeDirectory, + operation: ProposalId, + principal: &str, + expected: [u8; 32], + archives: &CellGroupArchivePool, + timeout: Duration, + ) -> Result { + self.run( + timeout, + Box::pin(async { + let intent = self + .read_managed_recovery_intent(directory, operation, principal) + .await? + .ok_or(FleetCatalogError::Conflict)?; + if intent.digest() != expected { + return Err(FleetCatalogError::PayloadConflict); + } + let basis = &intent.record.basis; + let snapshot = self.inner.current().await?; + let permit = self + .managed_recovery_permit_at(&snapshot, directory, basis) + .await?; + let domain = &self.inner.domain; + let selected = domain + .run(Box::pin(directory.observe_managed_recovery_barrier( + expected, + basis, + intent.archive_holds(), + &permit, + ))) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(catalog_error)?; + let directory_digest = if let Some(selected) = selected { + selected + } else { + if snapshot.revision() != basis.fleet_revision + || root_digest(&snapshot)? != basis.fleet_root + { + return Err(FleetCatalogError::Conflict); + } + self.require_recovery_intent_holds(directory, archives, &intent) + .await?; + if self.inner.current().await?.token != snapshot.token { + return Err(FleetCatalogError::Conflict); + } + domain + .run(Box::pin(directory.select_managed_recovery_barrier( + expected, + basis, + intent.archive_holds(), + &permit, + ))) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(catalog_error)? + }; + Ok(FleetTransitionRecoveryBarrier { + intent_digest: expected, + directory_digest, + }) + }), + ) + .await + } + + pub(crate) async fn reopen_managed_transition_recovery_intent( + &self, + directory: &RangeDirectory, + operation: ProposalId, + principal: &str, + timeout: Duration, + ) -> Result> { + self.run( + timeout, + self.read_managed_recovery_intent(directory, operation, principal), + ) + .await + } + + pub(super) async fn read_managed_recovery_intent( + &self, + directory: &RangeDirectory, + operation: ProposalId, + principal: &str, + ) -> Result> { + self.inner + .read::(&intent_key(operation), KIND, LIMIT) + .await? + .map(|(record, _)| record.validate(self, directory, operation, principal)) + .transpose() + } +} + +fn intent_key(operation: ProposalId) -> String { + format!( + "managed-transition-recovery/{}", + hex::encode(operation.bytes()) + ) +} diff --git a/crates/cellar/durability/raft/fleet_catalog/managed_recovery_inventory.rs b/crates/cellar/durability/raft/fleet_catalog/managed_recovery_inventory.rs new file mode 100644 index 000000000..77fa5e907 --- /dev/null +++ b/crates/cellar/durability/raft/fleet_catalog/managed_recovery_inventory.rs @@ -0,0 +1,116 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Catalog-required voter lineages for a managed transition fence. +use super::*; +use crate::raft::{ + catalog::{ManagedTransitionRecoverySelection, RangeDirectory}, + cell_groups::CellGroupArchiveCoverage, +}; +use cellar_logic::{ + cell_group::GroupHistory, + consensus::{DiskLineage, IncarnationId, MemberId, NodeId}, +}; +use std::collections::{BTreeMap, BTreeSet}; + +pub(super) const MAX_RECOVERY_LINEAGES: usize = 4096; + +/// Permanent voter or learner placement a deployment fence must cover. A +/// platform also needs its complete launch inventory for this disk and host. +#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)] +pub struct FleetRecoveryVoterLineage { + pub group: GroupHistory, + pub member: MemberId, + pub node: NodeId, + pub disk: DiskLineage, + pub physical_host: IncarnationId, +} + +pub(super) struct RecoveryLineages { + nodes: BTreeMap, + required: BTreeSet, +} + +impl RecoveryLineages { + pub(super) fn include( + &mut self, + group: GroupHistory, + member: MemberId, + node: NodeId, + expected_disk: Option, + ) -> Result<()> { + let &(disk, physical_host) = self + .nodes + .get(&node) + .ok_or(FleetCatalogError::InventoryIncomplete)?; + if expected_disk.is_some_and(|expected| expected != disk) { + return Err(FleetCatalogError::Conflict); + } + self.required.insert(FleetRecoveryVoterLineage { + group, + member, + node, + disk, + physical_host, + }); + if self.required.len() > MAX_RECOVERY_LINEAGES { + return Err(FleetCatalogError::Limit); + } + Ok(()) + } +} + +impl FleetCatalog { + /// Collect both sides from one exact Fleet root. Keep retired and reserved + /// placements, including admitted replacements that never became voters. + pub(super) async fn transition_recovery_voter_lineages<'a>( + &self, + snapshot: &FleetSnapshot, + directory: &RangeDirectory, + selection: &ManagedTransitionRecoverySelection, + archives: impl Iterator, + ) -> Result> { + let mut nodes = BTreeMap::new(); + for record in self.node_records(snapshot).await? { + let node = self.host_record(snapshot, record)?; + nodes.insert(node.node(), (node.disk(), node.physical_host())); + } + let mut lineages = RecoveryLineages { + nodes, + required: BTreeSet::new(), + }; + let mut groups = BTreeMap::new(); + for route in selection.parents.iter().chain(&selection.children) { + let binding = self.binding(&route.history)?; + let key = binding + .key() + .map_err(|_| FleetCatalogError::Corrupt("recovery history key"))?; + let history = self.history_inner(snapshot, &key).await?; + if history.binding() != &binding || !history.check_route(directory, route) { + return Err(FleetCatalogError::Conflict); + } + for member in history.voters.iter().chain(&history.learners) { + let id = MemberId::new(member.member) + .map_err(|_| FleetCatalogError::Corrupt("recovery initial member"))?; + let node = NodeId::new(member.node) + .map_err(|_| FleetCatalogError::Corrupt("recovery initial node"))?; + lineages.include(route.history, id, node, None)?; + } + groups.insert(key, route.history); + } + for archive in archives { + for member in archive.recovery().membership().retained_lineages() { + let member = member + .map_err(|_| FleetCatalogError::Corrupt("recovery archive membership"))?; + lineages.include( + archive.marker().route().history, + member.member, + member.node, + Some(member.disk), + )?; + } + } + self.collect_recovery_movements(snapshot, &groups, &mut lineages) + .await?; + Ok(lineages.required.into_iter().collect()) + } +} diff --git a/crates/cellar/durability/raft/fleet_catalog/managed_recovery_quorum.rs b/crates/cellar/durability/raft/fleet_catalog/managed_recovery_quorum.rs new file mode 100644 index 000000000..6ac23502b --- /dev/null +++ b/crates/cellar/durability/raft/fleet_catalog/managed_recovery_quorum.rs @@ -0,0 +1,495 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Select the exact seed receipts used to admit a managed recovery publication. +use super::{histories::catalog_error, *}; +use crate::raft::{ + catalog::{ + CatalogError, CellGroupBirthRequest, DirectoryPublication, ManagedRecoveryPublicationInput, + ManagedRecoverySeedQuorum, RangeDirectory, + }, + cell_groups::CellGroupArchivePool, + fleet_stop::BootBinding, + member_state::FleetGroupBinding, +}; +use cellar_logic::{ + cell_group::GroupHistory, + consensus::{LogPosition, NodeId, ProposalId}, +}; +use serde::Deserialize; +use std::collections::BTreeMap; + +const KIND: u8 = 229; +const LIMIT: usize = 64 * 1024; + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Record { + plan: [u8; 32], + receipt: [u8; 32], + pub(super) ranges: Vec, +} + +/// A verified durable seed set for every replacement range, in plan order. +/// Actual Raft quorum and activation require separate proofs. +pub struct FleetRecoverySeedQuorums { + record: Record, + digest: [u8; 32], +} + +impl FleetRecoverySeedQuorums { + pub fn plan_digest(&self) -> [u8; 32] { + self.record.plan + } + + pub fn digest(&self) -> [u8; 32] { + self.digest + } + + pub fn ranges(&self) -> &[ManagedRecoverySeedQuorum] { + &self.record.ranges + } +} + +/// Durable recovery identifiers used to reopen a published range on a voter. +#[derive(Clone, Debug)] +pub struct FleetRecoveryAttachRequest { + pub operation: ProposalId, + pub principal: String, + pub plan_digest: [u8; 32], + pub seed_digest: [u8; 32], +} + +pub(crate) struct FleetPublishedRecoveryVoter { + pub(crate) birth: CellGroupBirthRequest, + pub(crate) seeds: ManagedRecoverySeedQuorum, + pub(crate) publication: DirectoryPublication, + pub(crate) group: FleetGroupBinding, + pub(crate) seeded_boot: BootBinding, + pub(crate) current_boot: BootBinding, +} + +impl FleetHostAuthorization { + pub(crate) async fn authorize_published_recovery_voter( + &self, + directory: &RangeDirectory, + request: &FleetRecoveryAttachRequest, + history: GroupHistory, + timeout: Duration, + ) -> Result { + let catalog = &self.catalog; + catalog + .run(timeout, async { + let (intent, plan, record) = catalog + .read_managed_recovery_publication_inputs( + directory, + request.operation, + &request.principal, + (request.plan_digest, request.seed_digest), + ) + .await?; + let snapshot = catalog.inner.current().await?; + catalog + .managed_recovery_permit_at(&snapshot, directory, intent.basis()) + .await?; + let index = plan + .births() + .iter() + .position(|birth| birth.route().history == history) + .ok_or(FleetCatalogError::Conflict)?; + let birth = &plan.births()[index]; + let voter = birth + .voters() + .iter() + .position(|voter| voter.node == self.node()) + .ok_or(FleetCatalogError::Conflict)?; + let placement = catalog + .managed_recovery_placement_at(&snapshot, directory, &intent, &plan, birth) + .await? + .ok_or(FleetCatalogError::Conflict)?; + let seeded_boot = &placement.reservation.voter_boots()[voter]; + let current_boot = self.boot_binding()?; + if !seeded_boot.same_engine_owner(¤t_boot) + || catalog + .qualified_recovery_boot(&snapshot, self.node()) + .await? + .as_ref() + != Some(¤t_boot) + { + return Err(FleetCatalogError::Conflict); + } + let input = publication_input(&intent, &plan, request.seed_digest)?; + let publication = catalog + .inner + .domain + .run(Box::pin( + directory.observe_pending_managed_recovery_publication(&input), + )) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(catalog_error)? + .ok_or(FleetCatalogError::Conflict)?; + if placement.history.state == FleetHistoryState::Live { + catalog + .require_managed_recovery_promotion( + &snapshot, + request, + &intent, + publication.journal_digest(), + index, + ) + .await?; + } + if catalog.inner.current().await?.token != snapshot.token { + return Err(FleetCatalogError::Conflict); + } + Ok(FleetPublishedRecoveryVoter { + birth: birth.clone(), + seeds: record.ranges[index].clone(), + publication, + group: catalog.binding(&history)?, + seeded_boot: seeded_boot.clone(), + current_boot, + }) + }) + .await + } +} + +impl FleetCatalog { + pub(crate) async fn select_managed_recovery_seed_quorums( + &self, + directory: &RangeDirectory, + fenced: &FleetFencedTransitionRecovery, + plan: &FleetTransitionRecoveryReplacement, + archives: &CellGroupArchivePool, + timeout: Duration, + ) -> Result { + self.run(timeout, async { + self.require_managed_replacement(fenced, plan).await?; + self.require_confirmed_managed_fence(directory, fenced, archives) + .await?; + let intent = fenced.request().intent(); + let snapshot = self.inner.current().await?; + self.managed_recovery_permit_at(&snapshot, directory, intent.basis()) + .await?; + let key = seed_quorum_key(intent.operation()); + let existing = self.inner.read::(&key, KIND, LIMIT).await?; + if let Some((record, _)) = &existing { + if record.plan != plan.digest() + || record.receipt != fenced.receipt_digest() + || record.ranges.len() != plan.births().len() + { + return Err(FleetCatalogError::PayloadConflict); + } + } + + let ranges = self + .verify_managed_seed_quorums( + &snapshot, + directory, + fenced, + plan, + existing.as_ref().map(|(record, _)| record), + ) + .await?; + if self.inner.current().await?.token != snapshot.token { + return Err(FleetCatalogError::Conflict); + } + self.require_confirmed_managed_fence(directory, fenced, archives) + .await?; + if self.inner.current().await?.token != snapshot.token { + return Err(FleetCatalogError::Conflict); + } + let record = Record { + plan: plan.digest(), + receipt: fenced.receipt_digest(), + ranges, + }; + let bytes = encode(KIND, &record, LIMIT)?; + if existing.is_none() { + self.inner.create_exact(&key, &bytes, LIMIT).await?; + } + Ok(FleetRecoverySeedQuorums { + record, + digest: hash(&bytes), + }) + }) + .await + } + + pub(crate) async fn publish_managed_recovery_replacement( + &self, + directory: &RangeDirectory, + fenced: &FleetFencedTransitionRecovery, + plan: &FleetTransitionRecoveryReplacement, + selected: &FleetRecoverySeedQuorums, + archives: &CellGroupArchivePool, + timeout: Duration, + ) -> Result { + self.run(timeout, async { + self.require_managed_replacement(fenced, plan).await?; + let intent = fenced.request().intent(); + let key = seed_quorum_key(intent.operation()); + let (record, _) = self + .inner + .read::(&key, KIND, LIMIT) + .await? + .ok_or(FleetCatalogError::Unconfirmed)?; + if record.plan != plan.digest() + || record.receipt != fenced.receipt_digest() + || record.ranges.len() != plan.births().len() + || hash(&encode(KIND, &record, LIMIT)?) != selected.digest() + { + return Err(FleetCatalogError::PayloadConflict); + } + let publication = publication_input(intent, plan, selected.digest())?; + let domain = &self.inner.domain; + if let Some(existing) = domain + .run(Box::pin( + directory.observe_managed_recovery_publication(&publication), + )) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(catalog_error)? + { + return Ok(existing); + } + // A new selection still needs the active disruption. The + // directory's rooted historical receipt can be replayed after + // the original Fleet slot has been released. + let snapshot = self.inner.current().await?; + self.managed_recovery_permit_at(&snapshot, directory, intent.basis()) + .await?; + self.require_confirmed_managed_fence(directory, fenced, archives) + .await?; + self.verify_managed_seed_quorums(&snapshot, directory, fenced, plan, Some(&record)) + .await?; + self.require_confirmed_managed_fence(directory, fenced, archives) + .await?; + if self.inner.current().await?.token != snapshot.token { + return Err(FleetCatalogError::Conflict); + } + domain + .run(Box::pin(directory.publish_managed_recovery(&publication))) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(catalog_error) + }) + .await + } + + pub(crate) async fn reopen_managed_recovery_publication( + &self, + directory: &RangeDirectory, + operation: ProposalId, + principal: &str, + expected: ([u8; 32], [u8; 32]), + timeout: Duration, + ) -> Result> { + self.run(timeout, async { + let (intent, plan, _) = self + .read_managed_recovery_publication_inputs(directory, operation, principal, expected) + .await?; + let (_, seed_digest) = expected; + let input = publication_input(&intent, &plan, seed_digest)?; + self.inner + .domain + .run(Box::pin( + directory.observe_managed_recovery_publication(&input), + )) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(catalog_error) + }) + .await + } + + pub(crate) async fn reopen_managed_recovery_activation( + &self, + directory: &RangeDirectory, + request: &FleetRecoveryAttachRequest, + history: GroupHistory, + timeout: Duration, + ) -> Result> { + self.run(timeout, async { + let (intent, plan, record) = self + .read_managed_recovery_publication_inputs( + directory, + request.operation, + &request.principal, + (request.plan_digest, request.seed_digest), + ) + .await?; + let index = plan + .births() + .iter() + .position(|birth| birth.route().history == history) + .ok_or(FleetCatalogError::Conflict)?; + let input = publication_input(&intent, &plan, request.seed_digest)?; + self.inner + .domain + .run(Box::pin(async { + let publication = directory + .observe_managed_recovery_publication(&input) + .await? + .ok_or(CatalogError::Fenced)?; + directory + .observe_managed_recovery_activation( + plan.births()[index].route(), + request.plan_digest, + publication.journal_digest(), + record.ranges[index].genesis_commitment(), + ) + .await + })) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(catalog_error) + }) + .await + } + + pub(super) async fn read_managed_recovery_publication_inputs( + &self, + directory: &RangeDirectory, + operation: ProposalId, + principal: &str, + expected: ([u8; 32], [u8; 32]), + ) -> Result<( + FleetTransitionRecoveryIntent, + FleetTransitionRecoveryReplacement, + Record, + )> { + let intent = self + .read_managed_recovery_intent(directory, operation, principal) + .await? + .ok_or(FleetCatalogError::Conflict)?; + let (plan_digest, seed_digest) = expected; + let plan = self + .read_recovery_publication_plan(directory, &intent, plan_digest) + .await?; + let (record, _) = self + .inner + .read::(&seed_quorum_key(intent.operation()), KIND, LIMIT) + .await? + .ok_or(FleetCatalogError::Unconfirmed)?; + if record.plan != plan_digest + || record.receipt != plan.receipt_digest() + || record.ranges.len() != plan.births().len() + || hash(&encode(KIND, &record, LIMIT)?) != seed_digest + { + return Err(FleetCatalogError::PayloadConflict); + } + Ok((intent, plan, record)) + } + + async fn verify_managed_seed_quorums( + &self, + snapshot: &FleetSnapshot, + directory: &RangeDirectory, + fenced: &FleetFencedTransitionRecovery, + plan: &FleetTransitionRecoveryReplacement, + existing: Option<&Record>, + ) -> Result> { + let domain = &self.inner.domain; + let mut qualified = BTreeMap::new(); + let mut ranges = Vec::with_capacity(plan.births().len()); + for (index, birth) in plan.births().iter().enumerate() { + let reserved = self + .managed_recovery_reservation_at( + snapshot, + directory, + fenced.request().intent(), + plan, + birth, + ) + .await? + .ok_or(FleetCatalogError::Unconfirmed)?; + let boots = reserved.voter_boots(); + let selected = existing.map(|record| &record.ranges[index]); + let quorum = domain + .run(Box::pin(directory.observe_managed_recovery_seed_quorum( + birth, + plan.digest(), + boots, + selected, + ))) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(catalog_error)?; + let mut eligible = 0; + for (voter, boot) in birth.voters().iter().zip(boots) { + if quorum.members().binary_search(&voter.member.get()).is_err() { + continue; + } + let current = match qualified.entry(voter.node) { + std::collections::btree_map::Entry::Occupied(entry) => entry.into_mut(), + std::collections::btree_map::Entry::Vacant(entry) => { + entry.insert(self.qualified_recovery_boot(snapshot, voter.node).await?) + } + }; + // The seed is bound to durable storage. A newly qualified + // boot can retain it only on the exact same disk and engine. + if current + .as_ref() + .is_some_and(|current| boot.same_engine_owner(current)) + { + eligible += 1; + } + } + if eligible <= birth.voters().len() / 2 { + return Err(FleetCatalogError::Unconfirmed); + } + ranges.push(quorum); + } + Ok(ranges) + } + + async fn qualified_recovery_boot( + &self, + snapshot: &FleetSnapshot, + node: NodeId, + ) -> Result> { + let host = self.host_from_record(snapshot, node).await?; + if host.record.isolation.is_some() || !host.state().eligibility().receive_replicas { + return Ok(None); + } + match self.capacity_from(snapshot, node).await { + Ok(capacity) if capacity.receive_replicas() => { + Ok(Some(capacity.attestation().boot.clone())) + } + Ok(_) | Err(FleetCatalogError::Conflict) => Ok(None), + Err(error) => Err(error), + } + } +} + +pub(super) fn publication_input<'a>( + intent: &FleetTransitionRecoveryIntent, + plan: &'a FleetTransitionRecoveryReplacement, + seeds: [u8; 32], +) -> Result> { + let transition = ProposalId::new(intent.transition()) + .map_err(|_| FleetCatalogError::Corrupt("managed recovery transition"))?; + let barrier = plan + .mutation() + .expected_directory_digest() + .ok_or(FleetCatalogError::Conflict)?; + Ok(ManagedRecoveryPublicationInput { + intent: intent.digest(), + transition, + transition_plan: intent.basis().plan, + barrier, + plan: plan.digest(), + seeds, + mutation: plan.mutation(), + births: plan.births(), + }) +} + +fn seed_quorum_key(operation: [u8; 16]) -> String { + format!( + "managed-transition-recovery-seed-quorums/{}", + hex::encode(operation) + ) +} diff --git a/crates/cellar/durability/raft/fleet_catalog/managed_recovery_replacement.rs b/crates/cellar/durability/raft/fleet_catalog/managed_recovery_replacement.rs new file mode 100644 index 000000000..482da2d34 --- /dev/null +++ b/crates/cellar/durability/raft/fleet_catalog/managed_recovery_replacement.rs @@ -0,0 +1,318 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Immutable, range-preserving replacement plans under a managed recovery fence. +use super::{histories::catalog_error, *}; +use crate::raft::{ + catalog::{ + CellGroupBirthRequest, CellGroupBirthSpec, DirectoryMutation, DirectoryView, RangeDirectory, + }, + cell_groups::CellGroupArchivePool, +}; +use cellar_logic::consensus::ProposalId; +use serde::Deserialize; +use std::collections::BTreeSet; + +const KIND: u8 = 224; +const LIMIT: usize = 4 * 1024 * 1024; + +#[derive(PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Record { + fence: [u8; 32], + receipt: [u8; 32], + births: Vec>, + mutation: Vec, +} + +/// One durable, exact replacement per recovered range in source order. The +/// plan retains placement intent but grants no resource reservation or startup. +pub struct FleetTransitionRecoveryReplacement { + fence: [u8; 32], + receipt: [u8; 32], + births: Vec, + mutation: DirectoryMutation, + digest: [u8; 32], +} + +impl FleetTransitionRecoveryReplacement { + pub fn digest(&self) -> [u8; 32] { + self.digest + } + + pub fn births(&self) -> &[CellGroupBirthRequest] { + &self.births + } + + pub fn mutation(&self) -> &DirectoryMutation { + &self.mutation + } + + pub(super) fn receipt_digest(&self) -> [u8; 32] { + self.receipt + } + + pub(crate) fn matches_fence(&self, fenced: &FleetFencedTransitionRecovery) -> bool { + self.fence == fenced.request().digest() && self.receipt == fenced.receipt_digest() + } +} + +impl FleetCatalog { + pub(crate) async fn stage_managed_recovery_replacement( + &self, + directory: &RangeDirectory, + fenced: &FleetFencedTransitionRecovery, + replacements: Vec, + archives: &CellGroupArchivePool, + timeout: Duration, + ) -> Result { + self.run(timeout, async { + let view = self + .managed_replacement_view(directory, fenced, archives) + .await?; + if replacements.len() != fenced.archives().len() { + return Err(FleetCatalogError::Invalid("recovery replacement count")); + } + let owner = self.birth_owner()?; + let births = replacements + .into_iter() + .map(|spec| { + directory + .prepare_fleet_group_birth(spec, &owner) + .map_err(catalog_error) + }) + .collect::>>()?; + let mutation = replacement_mutation(directory, &view, fenced, &births)?; + let record = Record { + fence: fenced.request().digest(), + receipt: fenced.receipt_digest(), + births: births.iter().map(|birth| birth.encode().to_vec()).collect(), + mutation: mutation.encode().to_vec(), + }; + let bytes = encode(KIND, &record, LIMIT)?; + let key = replacement_key(fenced); + if let Some((selected, _)) = self.inner.read::(&key, KIND, LIMIT).await? { + if selected != record { + return Err(FleetCatalogError::PayloadConflict); + } + } else { + let domain = &self.inner.domain; + for birth in &births { + domain + .run(directory.require_unreserved_fleet_group(birth, &owner)) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(catalog_error)?; + } + self.inner.create_exact(&key, &bytes, LIMIT).await?; + } + Ok(FleetTransitionRecoveryReplacement { + fence: record.fence, + receipt: record.receipt, + births, + mutation, + digest: hash(&bytes), + }) + }) + .await + } + + pub(crate) async fn reopen_managed_recovery_replacement( + &self, + directory: &RangeDirectory, + fenced: &FleetFencedTransitionRecovery, + archives: &CellGroupArchivePool, + timeout: Duration, + ) -> Result> { + self.run(timeout, async { + let view = self + .managed_replacement_view(directory, fenced, archives) + .await?; + let Some((record, _)) = self + .inner + .read::(&replacement_key(fenced), KIND, LIMIT) + .await? + else { + return Ok(None); + }; + if record.fence != fenced.request().digest() + || record.receipt != fenced.receipt_digest() + { + return Err(FleetCatalogError::PayloadConflict); + } + let births = record + .births + .iter() + .map(|bytes| directory.decode_group_birth(bytes).map_err(catalog_error)) + .collect::>>()?; + let mutation = replacement_mutation(directory, &view, fenced, &births)?; + if mutation.encode() != record.mutation { + return Err(FleetCatalogError::Corrupt("recovery replacement mutation")); + } + Ok(Some(FleetTransitionRecoveryReplacement { + fence: record.fence, + receipt: record.receipt, + births, + mutation, + digest: hash(&encode(KIND, &record, LIMIT)?), + })) + }) + .await + } + + pub(super) async fn read_recovery_publication_plan( + &self, + directory: &RangeDirectory, + intent: &FleetTransitionRecoveryIntent, + expected: [u8; 32], + ) -> Result { + let operation = intent.operation(); + let (record, _) = self + .inner + .read::(&replacement_key_for(operation), KIND, LIMIT) + .await? + .ok_or(FleetCatalogError::Unconfirmed)?; + if hash(&encode(KIND, &record, LIMIT)?) != expected { + return Err(FleetCatalogError::PayloadConflict); + } + let mutation = directory + .decode_mutation(&record.mutation) + .map_err(catalog_error)?; + let barrier = mutation + .expected_directory_digest() + .ok_or(FleetCatalogError::Conflict)?; + if mutation.operation().bytes() != operation + || record.fence + != super::managed_recovery_fence::managed_fence_digest(intent.digest(), barrier) + { + return Err(FleetCatalogError::PayloadConflict); + } + self.require_durable_managed_fence(operation, record.fence, record.receipt) + .await?; + let births = record + .births + .iter() + .map(|bytes| directory.decode_group_birth(bytes).map_err(catalog_error)) + .collect::>>()?; + Ok(FleetTransitionRecoveryReplacement { + fence: record.fence, + receipt: record.receipt, + births, + mutation, + digest: expected, + }) + } + + pub(super) async fn require_managed_replacement( + &self, + fenced: &FleetFencedTransitionRecovery, + plan: &FleetTransitionRecoveryReplacement, + ) -> Result<()> { + let (record, _) = self + .inner + .read::(&replacement_key(fenced), KIND, LIMIT) + .await? + .ok_or(FleetCatalogError::Unconfirmed)?; + if !plan.matches_fence(fenced) + || record.fence != fenced.request().digest() + || record.receipt != fenced.receipt_digest() + || hash(&encode(KIND, &record, LIMIT)?) != plan.digest() + { + return Err(FleetCatalogError::PayloadConflict); + } + Ok(()) + } + + async fn managed_replacement_view( + &self, + directory: &RangeDirectory, + fenced: &FleetFencedTransitionRecovery, + archives: &CellGroupArchivePool, + ) -> Result { + self.require_confirmed_managed_fence(directory, fenced, archives) + .await?; + let view = self + .inner + .domain + .run(directory.observe()) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(catalog_error)?; + if view.digest() != fenced.request().barrier().directory_digest() { + return Err(FleetCatalogError::Conflict); + } + Ok(view) + } +} + +fn replacement_mutation( + directory: &RangeDirectory, + view: &DirectoryView, + fenced: &FleetFencedTransitionRecovery, + births: &[CellGroupBirthRequest], +) -> Result { + let intent = fenced.request().intent(); + if births.len() != fenced.archives().len() { + return Err(FleetCatalogError::Invalid("recovery replacement count")); + } + let old = intent.required_voter_lineages(); + let old_groups: BTreeSet<_> = old.iter().map(|member| member.group.group).collect(); + let old_incarnations: BTreeSet<_> = old.iter().map(|member| member.group.incarnation).collect(); + let old_nodes: BTreeSet<_> = old.iter().map(|member| member.node).collect(); + let member_floor = old + .iter() + .map(|member| member.member.get()) + .chain( + fenced + .archives() + .iter() + .map(|cut| cut.recovery().membership().high_water()), + ) + .max() + .ok_or(FleetCatalogError::Corrupt("recovery voter inventory"))?; + let mut groups = BTreeSet::new(); + let mut incarnations = BTreeSet::new(); + let mut creations = BTreeSet::new(); + let mut sources = Vec::with_capacity(births.len()); + let mut targets = Vec::with_capacity(births.len()); + for (source, birth) in fenced.archives().iter().zip(births) { + let source_route = source.marker().route(); + let target = birth.route(); + if target.range != source_route.range + || target.schema_digest != source_route.schema_digest + || old_groups.contains(&target.history.group) + || old_incarnations.contains(&target.history.incarnation) + || !groups.insert(target.history.group) + || !incarnations.insert(target.history.incarnation) + || !creations.insert(birth.creation()) + || [intent.operation(), intent.transition()].contains(&birth.creation().bytes()) + || birth.voters().len() != 3 + || !birth.learners().is_empty() + || birth.voters().iter().any(|member| { + old_nodes.contains(&member.node) || member.member.get() <= member_floor + }) + { + return Err(FleetCatalogError::Conflict); + } + source + .recovery() + .check_recovery_cell_incarnation(target.history.incarnation)?; + sources.push(source_route.clone()); + targets.push(target.clone()); + } + let operation = ProposalId::new(intent.operation()) + .map_err(|_| FleetCatalogError::Corrupt("recovery replacement operation"))?; + directory + .prepare_replacement(view, operation, &sources, &targets) + .map_err(catalog_error) +} + +fn replacement_key(fenced: &FleetFencedTransitionRecovery) -> String { + replacement_key_for(fenced.request().intent().operation()) +} + +fn replacement_key_for(operation: [u8; 16]) -> String { + format!( + "managed-transition-recovery-replacements/{}", + hex::encode(operation) + ) +} diff --git a/crates/cellar/durability/raft/fleet_catalog/managed_recovery_reservation.rs b/crates/cellar/durability/raft/fleet_catalog/managed_recovery_reservation.rs new file mode 100644 index 000000000..a1cdecda9 --- /dev/null +++ b/crates/cellar/durability/raft/fleet_catalog/managed_recovery_reservation.rs @@ -0,0 +1,485 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Root replacement history and its original voter boots while the recovery owns topology. +use super::{ + histories::{birth_mutation, catalog_error, validate_request}, + managed_transition_admission::TransitionOperatorStep, + *, +}; +use crate::raft::{ + catalog::{CellGroupBirthRequest, RangeDirectory}, + cell_groups::CellGroupArchivePool, + fleet_stop::BootBinding, +}; +use cellar_logic::{ + cell_group::{GroupHistory, GroupRoute}, + consensus::{NodeId, ProposalId}, +}; +use serde::Deserialize; +use std::collections::{BTreeMap, BTreeSet}; + +pub(super) const PREFIX: &str = "managed-recovery-birth/"; +const KIND: u8 = 225; +const LIMIT: usize = 4096; + +#[derive(PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Record { + plan: [u8; 32], + request: FleetRequestIdentity, + voters: Vec, +} + +/// A rooted placement reservation for one recovery range. Its voter boots +/// are historical evidence. The seeding owner must recheck current boot and +/// engine admission before performing physical work. +pub struct FleetRecoveryRangeReservation { + record: Record, + route: GroupRoute, + applied_revision: u64, +} + +impl FleetRecoveryRangeReservation { + pub fn plan_digest(&self) -> [u8; 32] { + self.record.plan + } + + pub fn route(&self) -> &GroupRoute { + &self.route + } + + pub fn voter_boots(&self) -> &[BootBinding] { + &self.record.voters + } + + pub fn applied_revision(&self) -> u64 { + self.applied_revision + } +} + +pub(super) struct ManagedRecoveryPlacement { + pub(super) reservation: FleetRecoveryRangeReservation, + pub(super) history: FleetHistory, + pub(super) key: String, +} + +#[derive(Clone, Copy)] +pub(crate) struct ManagedRecoverySeedSelection<'a> { + pub(crate) fenced: &'a FleetFencedTransitionRecovery, + pub(crate) plan: &'a FleetTransitionRecoveryReplacement, + pub(crate) reservation: &'a FleetRecoveryRangeReservation, +} + +pub(crate) struct ManagedRecoverySeedAuthority { + pub(crate) group: crate::raft::member_state::FleetGroupBinding, + pub(crate) boot: BootBinding, +} + +impl FleetHostAuthorization { + pub(crate) async fn authorize_managed_recovery_seed( + &self, + directory: &RangeDirectory, + selected: ManagedRecoverySeedSelection<'_>, + observed: (&crate::raft::host::RaftHostObservation, bool), + archives: &CellGroupArchivePool, + timeout: Duration, + ) -> Result { + let catalog = &self.catalog; + let ManagedRecoverySeedSelection { + fenced, + plan, + reservation, + } = selected; + let (view, already_seeded) = observed; + catalog + .run(timeout, async { + catalog.require_managed_replacement(fenced, plan).await?; + catalog + .require_confirmed_managed_fence(directory, fenced, archives) + .await?; + let birth = plan + .births() + .iter() + .find(|birth| birth.route() == reservation.route()) + .ok_or(FleetCatalogError::Conflict)?; + if reservation.plan_digest() != plan.digest() { + return Err(FleetCatalogError::PayloadConflict); + } + let intent = fenced.request().intent(); + let snapshot = catalog.inner.current().await?; + catalog + .managed_recovery_permit_at(&snapshot, directory, intent.basis()) + .await?; + let rooted = catalog + .managed_recovery_reservation_at(&snapshot, directory, intent, plan, birth) + .await? + .ok_or(FleetCatalogError::Conflict)?; + if rooted.record != reservation.record + || rooted.applied_revision != reservation.applied_revision + { + return Err(FleetCatalogError::Conflict); + } + let group = catalog.binding(&birth.route().history)?; + let boot = self.boot_binding()?; + if !rooted.voter_boots().contains(&boot) { + return Err(FleetCatalogError::Conflict); + } + let current = catalog.host_from(&snapshot, self.node()).await?; + let capacity = catalog.capacity_from(&snapshot, self.node()).await?; + let attestation = capacity.attestation(); + if !capacity.receive_replicas() + || boot != attestation.boot + || current.boot_binding()? != boot + { + return Err(FleetCatalogError::Conflict); + } + super::capacity::check_restart_capacity_view(¤t, attestation, view, false)?; + if !already_seeded { + let profile = &view.node.profile; + let bytes = view + .node + .reserved_bytes + .max(attestation.used_durable_bytes) + .checked_add(profile.bytes_per_group) + .ok_or(FleetCatalogError::Limit)?; + if view.node.groups >= profile.group_capacity + || bytes > profile.byte_capacity.min(attestation.durable_bytes) + { + return Err(FleetCatalogError::Conflict); + } + } + let old = fenced.request().intent().required_voter_lineages(); + let old_nodes: BTreeSet<_> = old.iter().map(|member| member.node.bytes()).collect(); + for node in catalog.node_records(&snapshot).await? { + if old_nodes.contains(&node.node) && node.storage_instance == Some(boot.engine) + { + return Err(FleetCatalogError::Conflict); + } + } + catalog + .inner + .domain + .run(directory.require_managed_recovery_identity(birth, plan.digest())) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(catalog_error)?; + if catalog.inner.current().await?.token != snapshot.token { + return Err(FleetCatalogError::Conflict); + } + Ok(ManagedRecoverySeedAuthority { group, boot }) + }) + .await + } +} + +impl FleetCatalog { + pub(crate) async fn reserve_managed_recovery_range( + &self, + directory: &RangeDirectory, + fenced: &FleetFencedTransitionRecovery, + plan: &FleetTransitionRecoveryReplacement, + scope: GroupHistory, + archives: &CellGroupArchivePool, + timeout: Duration, + ) -> Result { + self.run(timeout, async { + self.require_managed_replacement(fenced, plan).await?; + self.require_confirmed_managed_fence(directory, fenced, archives) + .await?; + let birth = plan + .births() + .iter() + .find(|birth| birth.route().history == scope) + .ok_or(FleetCatalogError::Conflict)?; + let intent = fenced.request().intent(); + let step = recovery_reservation_step(intent, plan, birth); + let binding = self.binding(&scope)?; + let history_key = binding + .key() + .map_err(|_| FleetCatalogError::Invalid("recovery history key"))?; + let key = format!("{PREFIX}{history_key}"); + let snapshot = self.inner.current().await?; + self.managed_recovery_permit_at(&snapshot, directory, intent.basis()) + .await?; + let domain = &self.inner.domain; + if let Some(reserved) = self + .managed_recovery_reservation_at(&snapshot, directory, intent, plan, birth) + .await? + { + domain + .run(directory.require_managed_recovery_identity(birth, plan.digest())) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(catalog_error)?; + return Ok(reserved); + } + self.require_namespace_declaration(&snapshot, directory.namespace(), None) + .await?; + let owner = self.managed_owner_for_birth(&snapshot, directory).await?; + let request = step.identity(snapshot.revision()); + validate_request(&request)?; + let history = FleetHistory::reserved_group(binding, directory, birth, &request)?; + self.require_unoccupied_birth(&snapshot, &history, &history_key) + .await?; + let voters = self.recovery_voter_boots(&snapshot, fenced, birth).await?; + let record = Record { + plan: plan.digest(), + request, + voters, + }; + check_boots(birth, &record, self.catalog_id()?)?; + let mutation = reservation_mutation(&record, &history, &history_key, key)?; + mutation.validate()?; + + let transition = ProposalId::new(intent.transition()) + .map_err(|_| FleetCatalogError::Corrupt("recovery transition operation"))?; + domain + .run(Box::pin(directory.reserve_managed_recovery_identity( + birth, + &owner, + transition, + fenced.request().barrier().directory_digest(), + plan.digest(), + ))) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(catalog_error)?; + let committed = Box::pin(self.commit_with_disruption( + &snapshot, + &mutation, + None, + resources::Authority::ManagedRecoveryReservation, + || Ok(()), + )) + .await?; + let applied_revision = self + .recovery_birth_commit(&committed.snapshot, &mutation) + .await?; + Ok(FleetRecoveryRangeReservation { + record, + route: birth.route().clone(), + applied_revision, + }) + }) + .await + } + + pub(super) async fn managed_recovery_reservation_at( + &self, + snapshot: &FleetSnapshot, + directory: &RangeDirectory, + intent: &FleetTransitionRecoveryIntent, + plan: &FleetTransitionRecoveryReplacement, + birth: &CellGroupBirthRequest, + ) -> Result> { + match self + .managed_recovery_placement_at(snapshot, directory, intent, plan, birth) + .await? + { + Some(placement) if placement.history.state == FleetHistoryState::Reserved => { + Ok(Some(placement.reservation)) + } + Some(_) => Err(FleetCatalogError::Conflict), + None => Ok(None), + } + } + + pub(super) async fn managed_recovery_placement_at( + &self, + snapshot: &FleetSnapshot, + directory: &RangeDirectory, + intent: &FleetTransitionRecoveryIntent, + plan: &FleetTransitionRecoveryReplacement, + birth: &CellGroupBirthRequest, + ) -> Result> { + let binding = self.binding(&birth.route().history)?; + let history_key = binding + .key() + .map_err(|_| FleetCatalogError::Invalid("recovery history key"))?; + let key = format!("{PREFIX}{history_key}"); + let Some(bytes) = self + .inner + .lookup(snapshot.head.tables[FleetTable::Policy.slot()], &key) + .await? + else { + return Ok(None); + }; + let record: Record = decode(KIND, &bytes, LIMIT)?; + let step = recovery_reservation_step(intent, plan, birth); + if record.plan != plan.digest() + || record.request != step.identity(record.request.expected_revision) + { + return Err(FleetCatalogError::PayloadConflict); + } + check_boots(birth, &record, self.catalog_id()?)?; + let history = self.history_inner(snapshot, &history_key).await?; + let reserved = FleetHistory::reserved_group(binding, directory, birth, &record.request)?; + if !matches!( + history.state, + FleetHistoryState::Reserved | FleetHistoryState::Live + ) { + return Err(FleetCatalogError::Conflict); + } + let mut expected = reserved.clone(); + expected.state = history.state.clone(); + if expected != history { + return Err(FleetCatalogError::Conflict); + } + let mutation = reservation_mutation(&record, &reserved, &history_key, key)?; + let applied_revision = self.recovery_birth_commit(snapshot, &mutation).await?; + Ok(Some(ManagedRecoveryPlacement { + reservation: FleetRecoveryRangeReservation { + record, + route: birth.route().clone(), + applied_revision, + }, + history, + key: history_key, + })) + } + + async fn recovery_birth_commit( + &self, + snapshot: &FleetSnapshot, + mutation: &FleetMutation, + ) -> Result { + let digest = hash(&encode(REQUEST_KIND, mutation, REQUEST_LIMIT)?); + self.inner + .committed(&snapshot.head, &hex::encode(mutation.id), digest) + .await? + .filter(|applied| Some(*applied) == mutation.expected_revision.checked_add(1)) + .ok_or(FleetCatalogError::Corrupt("recovery reservation commit")) + } + + async fn recovery_voter_boots( + &self, + snapshot: &FleetSnapshot, + fenced: &FleetFencedTransitionRecovery, + birth: &CellGroupBirthRequest, + ) -> Result> { + let old = fenced.request().intent().required_voter_lineages(); + let old_hosts: BTreeSet<_> = old.iter().map(|member| member.physical_host).collect(); + let old_disks: BTreeSet<_> = old.iter().map(|member| member.disk).collect(); + let old_nodes: BTreeSet<_> = old.iter().map(|member| member.node).collect(); + let mut separation = BTreeSet::new(); + let mut voters = Vec::with_capacity(birth.voters().len()); + let mut slots = self.recovery_history_slots(snapshot, birth).await?; + self.charge_recovery_movement_slots(snapshot, &mut slots) + .await?; + for voter in birth.voters() { + let host = self.host_from(snapshot, voter.node).await?; + let capacity = self.capacity_from(snapshot, voter.node).await?; + let partition = match snapshot.replica_separation() { + FleetReplicaSeparation::Nodes => host.node().bytes(), + FleetReplicaSeparation::PhysicalHosts => host.physical_host().bytes(), + }; + let occupied = slots + .get(&voter.node) + .ok_or(FleetCatalogError::Corrupt("recovery voter slots"))?; + if old_hosts.contains(&host.physical_host()) + || old_disks.contains(&host.disk()) + || old_nodes.contains(&host.node()) + || !separation.insert(partition) + || !capacity.receive_replicas() + || *occupied >= capacity.attestation().engine_slots + { + return Err(FleetCatalogError::Conflict); + } + voters.push(capacity.attestation().boot.clone()); + } + Ok(voters) + } + + async fn recovery_history_slots( + &self, + snapshot: &FleetSnapshot, + birth: &CellGroupBirthRequest, + ) -> Result> { + let mut slots: BTreeMap<_, _> = birth + .voters() + .iter() + .map(|voter| (voter.node, 0u32)) + .collect(); + let mut cursor = None; + let mut seen = 0u64; + loop { + let page = self + .inner + .scan( + snapshot.head.tables[FleetTable::Histories.slot()], + cursor.as_ref(), + 128, + ) + .await?; + for entry in page.entries { + seen = seen + .checked_add(1) + .filter(|seen| *seen <= HISTORY_LIMIT) + .ok_or(FleetCatalogError::Limit)?; + let history = FleetHistory::decode(&entry.value)?; + if matches!(history.state(), FleetHistoryState::Cancelled { .. }) { + continue; + } + // Include reserved and retired occupancy until its owner + // settles it. Historical initial placements are conservative. + for member in history.voters.iter().chain(&history.learners) { + let node = NodeId::new(member.node) + .map_err(|_| FleetCatalogError::Corrupt("recovery placement node"))?; + if let Some(count) = slots.get_mut(&node) { + *count = count.checked_add(1).ok_or(FleetCatalogError::Limit)?; + } + } + } + cursor = page.next; + if cursor.is_none() { + break; + } + } + if seen != snapshot.head.history_count { + return Err(FleetCatalogError::Corrupt("recovery placement inventory")); + } + Ok(slots) + } +} + +fn recovery_reservation_step( + intent: &FleetTransitionRecoveryIntent, + plan: &FleetTransitionRecoveryReplacement, + birth: &CellGroupBirthRequest, +) -> TransitionOperatorStep { + TransitionOperatorStep { + operation: plan.mutation().operation(), + principal: intent.principal().into(), + request_digest: plan.digest(), + step: birth.creation().bytes(), + } +} + +fn check_boots(birth: &CellGroupBirthRequest, record: &Record, catalog: [u8; 32]) -> Result<()> { + if birth.voters().len() != record.voters.len() { + return Err(FleetCatalogError::Corrupt("recovery voter count")); + } + for (voter, boot) in birth.voters().iter().zip(&record.voters) { + boot.validate() + .map_err(|_| FleetCatalogError::Corrupt("recovery voter boot"))?; + if boot.catalog != catalog || boot.node != voter.node.bytes() { + return Err(FleetCatalogError::Corrupt("recovery voter binding")); + } + } + Ok(()) +} + +fn reservation_mutation( + record: &Record, + history: &FleetHistory, + history_key: &str, + key: String, +) -> Result { + let mut mutation = birth_mutation(history, &record.request, history_key, None)?; + mutation.changes.push(FleetChange { + table: FleetTable::Policy, + key, + value: Some(encode(KIND, record, LIMIT)?), + }); + Ok(mutation) +} diff --git a/crates/cellar/durability/raft/fleet_catalog/managed_recovery_retirement.rs b/crates/cellar/durability/raft/fleet_catalog/managed_recovery_retirement.rs new file mode 100644 index 000000000..bc19a7b1d --- /dev/null +++ b/crates/cellar/durability/raft/fleet_catalog/managed_recovery_retirement.rs @@ -0,0 +1,374 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Close the original disruption after a permanent fence and serving release. +use super::{ + disruption::SLOT_KEY, + histories::{catalog_error, history_mutation, Origin}, + *, +}; +use crate::raft::catalog::{DirectoryMutation, DisasterFenceAuthority, RangeDirectory}; +use cellar_logic::{ + cell_group::{GroupHistory, GroupRoute}, + consensus::ProposalId, +}; +use serde::Deserialize; +use std::{collections::BTreeSet, ops::Range}; + +pub(super) const PREFIX: &str = "managed-recovery-retirement/"; +const KIND: u8 = 232; +const LIMIT: usize = 4096; + +// Leave room for the protected cursor and final disruption-slot deletion. +const BATCH: usize = 14; + +#[derive(PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Facts { + recovery: [u8; 16], + transition: [u8; 16], + intent: [u8; 32], + completion: [u8; 32], + total: usize, +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Record { + facts: Facts, + retired: usize, + request: FleetRequestIdentity, +} + +/// Bounded, resumable retirement of histories fenced by disaster recovery. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct FleetRecoveryRetirement { + pub retired_histories: usize, + pub total_histories: usize, + pub applied_revision: u64, + pub fleet_slot_released: bool, +} + +struct Sources { + routes: Vec, + children: BTreeSet, +} + +impl Sources { + fn new(directory: &RangeDirectory, plan: &DirectoryMutation) -> Result { + let (mut routes, children) = directory.transition_routes(plan).map_err(catalog_error)?; + let scopes = children.iter().map(|route| route.history).collect(); + routes.extend(children); + routes.sort_unstable_by_key(|route| route.history); + if !(3..=65).contains(&routes.len()) + || routes + .windows(2) + .any(|pair| pair[0].history == pair[1].history) + { + return Err(FleetCatalogError::Corrupt("recovery retirement sources")); + } + Ok(Self { + routes, + children: scopes, + }) + } +} + +impl Facts { + fn digest(&self) -> Result<[u8; 32]> { + Ok(hash(&encode(KIND, self, LIMIT)?)) + } + + fn identity( + &self, + principal: &str, + revision: u64, + retired: usize, + ) -> Result { + let step = u8::try_from(retired).map_err(|_| FleetCatalogError::Limit)?; + Ok(TransitionOperatorStep { + operation: ProposalId::new(self.recovery) + .map_err(|_| FleetCatalogError::Corrupt("recovery retirement operation"))?, + principal: principal.into(), + request_digest: self.digest()?, + step: [step; 16], + } + .identity(revision)) + } +} + +impl Record { + fn validate(&self, transition: [u8; 16], routes: usize) -> Result<()> { + if self.facts.transition != transition + || self.facts.total != routes + || [self.facts.intent, self.facts.completion].contains(&[0; 32]) + || !(1..=routes).contains(&self.retired) + || (self.retired != routes && !self.retired.is_multiple_of(BATCH)) + || self.request.expected_revision == 0 + || self.request + != self.facts.identity( + &self.request.principal, + self.request.expected_revision, + self.retired, + )? + { + return Err(FleetCatalogError::Corrupt("recovery retirement cursor")); + } + Ok(()) + } + + fn complete(&self) -> bool { + self.retired == self.facts.total + } + + fn last_batch(&self) -> Range { + ((self.retired - 1) / BATCH * BATCH)..self.retired + } + + fn status(&self, applied_revision: u64) -> FleetRecoveryRetirement { + FleetRecoveryRetirement { + retired_histories: self.retired, + total_histories: self.facts.total, + applied_revision, + fleet_slot_released: self.complete(), + } + } +} + +impl FleetCatalog { + pub(crate) async fn advance_managed_recovery_retirement( + &self, + directory: &RangeDirectory, + request: &FleetRecoveryAttachRequest, + completion: [u8; 32], + authority: &dyn DisasterFenceAuthority, + timeout: Duration, + ) -> Result { + self.run( + timeout, + Box::pin(async { + let (intent, replacement, _) = self + .read_managed_recovery_publication_inputs( + directory, + request.operation, + &request.principal, + (request.plan_digest, request.seed_digest), + ) + .await?; + let transition_id = ProposalId::new(intent.transition()) + .map_err(|_| FleetCatalogError::Corrupt("recovery retirement transition"))?; + let snapshot = self.inner.current().await?; + let transition = self + .observe_managed_transition_at(&snapshot, directory, transition_id) + .await? + .ok_or(FleetCatalogError::Conflict)? + .transition; + if transition.plan.digest() != intent.basis().plan + || transition + .journal + .as_ref() + .and_then(|journal| journal.recovery_settlement()) + != Some(completion) + { + return Err(FleetCatalogError::Conflict); + } + let sources = Sources::new(directory, &transition.plan)?; + let fenced: BTreeSet<_> = intent + .required_voter_lineages() + .iter() + .map(|lineage| lineage.group) + .collect(); + if !sources.routes.iter().map(|route| route.history).eq(fenced) { + return Err(FleetCatalogError::Conflict); + } + let facts = Facts { + recovery: request.operation.bytes(), + transition: transition_id.bytes(), + intent: intent.digest(), + completion, + total: sources.routes.len(), + }; + let mut start = 0; + if let Some(record) = self + .recovery_retirement_at(&snapshot, facts.transition) + .await? + { + record.validate(facts.transition, facts.total)?; + if record.facts != facts || record.request.principal != request.principal { + return Err(FleetCatalogError::PayloadConflict); + } + let previous = self + .recovery_retirement_mutation(&snapshot, directory, &sources, &record, true) + .await?; + let revision = self + .inner + .committed( + &snapshot.head, + &hex::encode(record.request.id), + hash(&encode(REQUEST_KIND, &previous, REQUEST_LIMIT)?), + ) + .await? + .filter(|applied| { + Some(*applied) == record.request.expected_revision.checked_add(1) + }) + .ok_or(FleetCatalogError::Corrupt("recovery retirement journal"))?; + if record.complete() { + return Ok(record.status(revision)); + } + start = record.retired; + } + + self.managed_recovery_permit_at(&snapshot, directory, intent.basis()) + .await?; + self.inner + .domain + .run(Box::pin(directory.check_managed_recovery_retirement( + request.operation, + transition_id, + facts.intent, + completion, + &sources.routes, + ))) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(catalog_error)?; + let (challenge, evidence) = self + .read_recovery_retirement_fence(intent, &replacement) + .await?; + authority + .verify(&challenge, &evidence) + .await + .map_err(FleetCatalogError::Storage)?; + + let retired = (start + BATCH).min(facts.total); + let record = Record { + request: facts.identity(&request.principal, snapshot.revision(), retired)?, + facts, + retired, + }; + let mutation = self + .recovery_retirement_mutation(&snapshot, directory, &sources, &record, false) + .await?; + let release = if record.complete() { + Some( + snapshot + .head + .disruption + .ok_or(FleetCatalogError::Conflict)?, + ) + } else { + None + }; + let committed = Box::pin(self.commit_with_disruption( + &snapshot, + &mutation, + release, + resources::Authority::ManagedRecoveryRetirement, + || Ok(()), + )) + .await?; + Ok(record.status(committed.applied_revision)) + }), + ) + .await + } + + async fn recovery_retirement_mutation( + &self, + snapshot: &FleetSnapshot, + directory: &RangeDirectory, + sources: &Sources, + record: &Record, + replay: bool, + ) -> Result { + let retired = FleetHistoryState::Retired { + operation: record.facts.transition, + retirement: record.facts.digest()?, + }; + let mut changes = Vec::with_capacity(BATCH + 2); + for route in &sources.routes[record.last_batch()] { + let key = self + .binding(&route.history)? + .key() + .map_err(|_| FleetCatalogError::Corrupt("recovery retirement history key"))?; + let mut history = self.history_inner(snapshot, &key).await?; + let allowed = if replay { + history.state == retired + } else { + history.state == FleetHistoryState::Live + || (history.state == FleetHistoryState::Reserved + && sources.children.contains(&route.history)) + }; + if !allowed + || history.origin != Origin::Managed + || !history.check_route(directory, route) + { + return Err(FleetCatalogError::Conflict); + } + history.state = retired.clone(); + changes.push(FleetChange { + table: FleetTable::Histories, + key, + value: Some(history.encode()?), + }); + } + changes.push(FleetChange { + table: FleetTable::Policy, + key: retirement_key(record.facts.transition), + value: Some(encode(KIND, record, LIMIT)?), + }); + if record.complete() { + changes.push(FleetChange { + table: FleetTable::Policy, + key: SLOT_KEY.into(), + value: None, + }); + } + let mutation = history_mutation(&record.request, changes); + mutation.validate()?; + Ok(mutation) + } + + async fn recovery_retirement_at( + &self, + snapshot: &FleetSnapshot, + operation: [u8; 16], + ) -> Result> { + self.inner + .lookup( + snapshot.head.tables[FleetTable::Policy.slot()], + &retirement_key(operation), + ) + .await? + .map(|bytes| decode(KIND, &bytes, LIMIT)) + .transpose() + } + + pub(super) async fn require_recovery_retired_history( + &self, + snapshot: &FleetSnapshot, + directory: &RangeDirectory, + plan: &DirectoryMutation, + route: &GroupRoute, + completion: [u8; 32], + retirement: [u8; 32], + ) -> Result<()> { + let operation = plan.operation().bytes(); + let sources = Sources::new(directory, plan)?; + let record = self + .recovery_retirement_at(snapshot, operation) + .await? + .ok_or(FleetCatalogError::InventoryIncomplete)?; + record.validate(operation, sources.routes.len())?; + if record.facts.completion != completion + || record.facts.digest()? != retirement + || !sources.routes[..record.retired].contains(route) + { + return Err(FleetCatalogError::InventoryIncomplete); + } + Ok(()) + } +} + +fn retirement_key(operation: [u8; 16]) -> String { + format!("{PREFIX}{}", hex::encode(operation)) +} diff --git a/crates/cellar/durability/raft/fleet_catalog/managed_recovery_settlement.rs b/crates/cellar/durability/raft/fleet_catalog/managed_recovery_settlement.rs new file mode 100644 index 000000000..2c25020e1 --- /dev/null +++ b/crates/cellar/durability/raft/fleet_catalog/managed_recovery_settlement.rs @@ -0,0 +1,483 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Promote the exact replacement histories while both recovery fences remain held. +use super::{ + histories::{catalog_error, history_mutation}, + managed_recovery_quorum::publication_input, + *, +}; +use crate::raft::{ + catalog::{CatalogError, ManagedRecoveryPublicationInput, RangeDirectory}, + cell_groups::CellGroupArchivePool, +}; +use cellar_logic::consensus::ProposalId; +use serde::Deserialize; +use std::ops::Range; + +pub(super) const PREFIX: &str = "managed-recovery-settlement/"; +const KIND: u8 = 230; +const LIMIT: usize = 4096; + +// The Fleet mutation also carries one protected cursor update. +const BATCH: usize = 15; + +#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Facts { + intent: [u8; 32], + plan: [u8; 32], + seeds: [u8; 32], + publication: [u8; 32], + activations: [u8; 32], + total: usize, +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Record { + facts: Facts, + promoted: usize, + request: FleetRequestIdentity, +} + +/// Rooted progress through the selected recovery's replacement histories. +/// Serving still requires the recovery fence to be released. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct FleetRecoverySettlement { + pub promoted_ranges: usize, + pub total_ranges: usize, + pub applied_revision: u64, + pub publication_digest: [u8; 32], +} + +/// Opaque proof minted from the complete rooted Fleet settlement. +pub(crate) struct FleetManagedRecoveryCompletion { + catalog: [u8; 32], + revision: u64, + digest: [u8; 32], + facts: Facts, +} + +impl FleetManagedRecoveryCompletion { + pub(crate) fn directory_binding( + &self, + directory: &RangeDirectory, + input: &ManagedRecoveryPublicationInput<'_>, + publication: [u8; 32], + ) -> std::result::Result<([u8; 32], u64, [u8; 32]), CatalogError> { + if directory + .namespace_config() + .managed_fleet + .as_ref() + .map(|owner| owner.catalog) + != Some(self.catalog) + || self.facts.intent != input.intent + || self.facts.plan != input.plan + || self.facts.seeds != input.seeds + || self.facts.publication != publication + || self.facts.total != input.births.len() + { + return Err(CatalogError::Fenced); + } + Ok((self.catalog, self.revision, self.digest)) + } +} + +struct Selection<'a> { + directory: &'a RangeDirectory, + intent: &'a FleetTransitionRecoveryIntent, + plan: &'a FleetTransitionRecoveryReplacement, +} + +impl Record { + fn identity( + facts: &Facts, + request: &FleetRecoveryAttachRequest, + revision: u64, + promoted: usize, + ) -> Result { + let step = u8::try_from(promoted).map_err(|_| FleetCatalogError::Limit)?; + Ok(TransitionOperatorStep { + operation: request.operation, + principal: request.principal.clone(), + request_digest: hash(&encode(KIND, facts, LIMIT)?), + step: [step; 16], + } + .identity(revision)) + } + + fn check( + &self, + request: &FleetRecoveryAttachRequest, + intent: &FleetTransitionRecoveryIntent, + publication: [u8; 32], + ) -> Result<()> { + if self.facts.intent != intent.digest() + || self.facts.plan != request.plan_digest + || self.facts.seeds != request.seed_digest + || self.facts.publication != publication + { + return Err(FleetCatalogError::PayloadConflict); + } + if !(1..=64).contains(&self.facts.total) + || !(1..=self.facts.total).contains(&self.promoted) + || (self.promoted != self.facts.total && !self.promoted.is_multiple_of(BATCH)) + || self.facts.activations == [0; 32] + || self.request.expected_revision == 0 + || self.request + != Self::identity( + &self.facts, + request, + self.request.expected_revision, + self.promoted, + )? + { + return Err(FleetCatalogError::Corrupt( + "managed recovery settlement cursor", + )); + } + Ok(()) + } + + fn status(&self, applied_revision: u64) -> FleetRecoverySettlement { + FleetRecoverySettlement { + promoted_ranges: self.promoted, + total_ranges: self.facts.total, + applied_revision, + publication_digest: self.facts.publication, + } + } + + fn last_batch(&self) -> Range { + ((self.promoted - 1) / BATCH * BATCH)..self.promoted + } +} + +impl FleetCatalog { + pub(crate) async fn advance_managed_recovery_settlement( + &self, + directory: &RangeDirectory, + request: &FleetRecoveryAttachRequest, + timeout: Duration, + ) -> Result { + self.run( + timeout, + Box::pin(async { + let (intent, plan, seeds) = self + .read_managed_recovery_publication_inputs( + directory, + request.operation, + &request.principal, + (request.plan_digest, request.seed_digest), + ) + .await?; + let snapshot = self.inner.current().await?; + self.managed_recovery_permit_at(&snapshot, directory, intent.basis()) + .await?; + let input = publication_input(&intent, &plan, request.seed_digest)?; + let facts = self + .verify_recovery_settlement_facts(directory, &input, &seeds) + .await?; + let publication = facts.publication; + let selection = Selection { + directory, + intent: &intent, + plan: &plan, + }; + let mut start = 0; + if let Some(record) = self + .managed_recovery_settlement_at(&snapshot, request.operation) + .await? + { + record.check(request, &intent, publication)?; + if record.facts != facts { + return Err(FleetCatalogError::PayloadConflict); + } + let applied_revision = self + .verify_recovery_settlement_commit(&snapshot, &selection, &record) + .await?; + start = record.promoted; + if start == facts.total { + return Ok(record.status(applied_revision)); + } + } + + let promoted = (start + BATCH).min(facts.total); + let request = Record::identity(&facts, request, snapshot.revision(), promoted)?; + let record = Record { + facts, + promoted, + request, + }; + let mutation = self + .recovery_settlement_mutation( + &snapshot, + &selection, + &record, + FleetHistoryState::Reserved, + ) + .await?; + self.inner + .domain + .run(Box::pin(directory.materialize_managed_recovery_births( + &input, + record.last_batch(), + ))) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(catalog_error)?; + let committed = Box::pin(self.commit_with_disruption( + &snapshot, + &mutation, + None, + resources::Authority::ManagedRecoverySettlement, + || Ok(()), + )) + .await?; + Ok(record.status(committed.applied_revision)) + }), + ) + .await + } + + pub(crate) async fn record_managed_recovery_directory_completion( + &self, + directory: &RangeDirectory, + request: &FleetRecoveryAttachRequest, + release: Option<([u8; 32], &CellGroupArchivePool)>, + timeout: Duration, + ) -> Result<[u8; 32]> { + self.run( + timeout, + Box::pin(async { + let (intent, plan, seeds) = self + .read_managed_recovery_publication_inputs( + directory, + request.operation, + &request.principal, + (request.plan_digest, request.seed_digest), + ) + .await?; + let input = publication_input(&intent, &plan, request.seed_digest)?; + if let Some(completion) = self + .inner + .domain + .run(Box::pin( + directory.observe_released_managed_recovery_completion(&input), + )) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(catalog_error)? + { + if release.is_some_and(|(expected, _)| expected != completion) { + return Err(FleetCatalogError::PayloadConflict); + } + return Ok(completion); + } + let snapshot = self.inner.current().await?; + self.managed_recovery_permit_at(&snapshot, directory, intent.basis()) + .await?; + let record = self + .managed_recovery_settlement_at(&snapshot, request.operation) + .await? + .ok_or(FleetCatalogError::Unconfirmed)?; + if record.promoted != record.facts.total { + return Err(FleetCatalogError::Unconfirmed); + } + let facts = self + .verify_recovery_settlement_facts(directory, &input, &seeds) + .await?; + record.check(request, &intent, facts.publication)?; + if record.facts != facts { + return Err(FleetCatalogError::PayloadConflict); + } + let selection = Selection { + directory, + intent: &intent, + plan: &plan, + }; + let revision = self + .verify_recovery_settlement_commit(&snapshot, &selection, &record) + .await?; + for birth in &plan.births()[..record.last_batch().start] { + let placement = self + .managed_recovery_placement_at(&snapshot, directory, &intent, &plan, birth) + .await? + .ok_or(FleetCatalogError::Conflict)?; + if placement.history.state != FleetHistoryState::Live { + return Err(FleetCatalogError::Conflict); + } + } + let proof = FleetManagedRecoveryCompletion { + catalog: self.catalog_id()?, + revision, + digest: hash(&encode(KIND, &record, LIMIT)?), + facts, + }; + if self.inner.current().await?.token != snapshot.token { + return Err(FleetCatalogError::Conflict); + } + self.inner + .domain + .run(Box::pin(directory.record_managed_recovery_completion( + &input, + &proof, + release.map(|(completion, archives)| { + (completion, &self.inner.domain, archives) + }), + ))) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(catalog_error) + }), + ) + .await + } + + async fn verify_recovery_settlement_facts( + &self, + directory: &RangeDirectory, + input: &ManagedRecoveryPublicationInput<'_>, + seeds: &super::managed_recovery_quorum::Record, + ) -> Result { + let (publication, activations) = self + .inner + .domain + .run(Box::pin(async { + let publication = directory + .observe_managed_recovery_publication(input) + .await? + .ok_or(CatalogError::Fenced)? + .journal_digest(); + let mut positions = Vec::with_capacity(input.births.len()); + for (birth, seed) in input.births.iter().zip(&seeds.ranges) { + if birth.route().birth_digest != seed.birth_digest() { + return Err(CatalogError::Corrupt("managed recovery settlement birth")); + } + let position = directory + .observe_managed_recovery_activation( + birth.route(), + input.plan, + publication, + seed.genesis_commitment(), + ) + .await? + .ok_or(CatalogError::Unconfirmed)?; + positions.push((position.index.get(), position.term.get())); + } + Ok::<_, CatalogError>((publication, positions)) + })) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(catalog_error)?; + + let facts = Facts { + intent: input.intent, + plan: input.plan, + seeds: input.seeds, + publication, + activations: hash(&encode(KIND, &activations, LIMIT)?), + total: input.births.len(), + }; + if !(1..=64).contains(&facts.total) { + return Err(FleetCatalogError::Limit); + } + Ok(facts) + } + + async fn verify_recovery_settlement_commit( + &self, + snapshot: &FleetSnapshot, + selection: &Selection<'_>, + record: &Record, + ) -> Result { + let previous = self + .recovery_settlement_mutation(snapshot, selection, record, FleetHistoryState::Live) + .await?; + let digest = hash(&encode(REQUEST_KIND, &previous, REQUEST_LIMIT)?); + let applied_revision = self + .inner + .committed(&snapshot.head, &hex::encode(record.request.id), digest) + .await? + .filter(|applied| Some(*applied) == record.request.expected_revision.checked_add(1)) + .ok_or(FleetCatalogError::Corrupt( + "managed recovery settlement journal", + ))?; + Ok(applied_revision) + } + + pub(super) async fn require_managed_recovery_promotion( + &self, + snapshot: &FleetSnapshot, + request: &FleetRecoveryAttachRequest, + intent: &FleetTransitionRecoveryIntent, + publication: [u8; 32], + index: usize, + ) -> Result<()> { + let record = self + .managed_recovery_settlement_at(snapshot, request.operation) + .await? + .ok_or(FleetCatalogError::Conflict)?; + record.check(request, intent, publication)?; + if index >= record.promoted { + return Err(FleetCatalogError::Conflict); + } + Ok(()) + } + + async fn managed_recovery_settlement_at( + &self, + snapshot: &FleetSnapshot, + operation: ProposalId, + ) -> Result> { + let key = settlement_key(operation.bytes()); + self.inner + .lookup(snapshot.head.tables[FleetTable::Policy.slot()], &key) + .await? + .map(|bytes| decode(KIND, &bytes, LIMIT)) + .transpose() + } + + async fn recovery_settlement_mutation( + &self, + snapshot: &FleetSnapshot, + selection: &Selection<'_>, + record: &Record, + expected: FleetHistoryState, + ) -> Result { + let mut changes = Vec::with_capacity(BATCH + 1); + for birth in &selection.plan.births()[record.last_batch()] { + let mut placement = self + .managed_recovery_placement_at( + snapshot, + selection.directory, + selection.intent, + selection.plan, + birth, + ) + .await? + .ok_or(FleetCatalogError::Conflict)?; + if placement.history.state != expected { + return Err(FleetCatalogError::Conflict); + } + placement.history.state = FleetHistoryState::Live; + changes.push(FleetChange { + table: FleetTable::Histories, + key: placement.key, + value: Some(placement.history.encode()?), + }); + } + changes.push(FleetChange { + table: FleetTable::Policy, + key: settlement_key(record.request.operation), + value: Some(encode(KIND, record, LIMIT)?), + }); + let mutation = history_mutation(&record.request, changes); + mutation.validate()?; + Ok(mutation) + } +} + +fn settlement_key(operation: [u8; 16]) -> String { + format!("{PREFIX}{}", hex::encode(operation)) +} diff --git a/crates/cellar/durability/raft/fleet_catalog/managed_transition_admission.rs b/crates/cellar/durability/raft/fleet_catalog/managed_transition_admission.rs new file mode 100644 index 000000000..49415c666 --- /dev/null +++ b/crates/cellar/durability/raft/fleet_catalog/managed_transition_admission.rs @@ -0,0 +1,247 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Persist the exact operator plan before reserving any child. + +use super::{ + histories::{catalog_error, validate_request}, + *, +}; +use crate::raft::{ + catalog::RangeDirectory, + cell_groups::{CellGroupBirthSpec, DirectoryMutation}, +}; +use cellar_logic::{cell_group::GroupRoute, consensus::ProposalId}; +use serde::Deserialize; +use sha2::{Digest, Sha256}; + +const ADMISSION: u8 = 213; +const ADMISSION_LIMIT: usize = 32 * 1024; + +/// Stable audit request and step, with a distinct Fleet attempt for each +/// conditional input revision. A consumed revision cannot later commit. +pub(crate) struct TransitionOperatorStep { + pub operation: ProposalId, + pub principal: String, + pub request_digest: [u8; 32], + pub step: [u8; 16], +} + +impl TransitionOperatorStep { + pub(crate) fn identity(&self, revision: u64) -> FleetRequestIdentity { + let mut digest = Sha256::new(); + digest.update(b"cellar-transition-operator-step-1"); + digest.update(self.request_digest); + digest.update(self.step); + digest.update(revision.to_be_bytes()); + let mut id = [0; 16]; + id.copy_from_slice(&digest.finalize()[..16]); + FleetRequestIdentity { + id, + operation: self.operation.bytes(), + principal: self.principal.clone(), + expected_revision: revision, + } + } +} + +#[derive(Deserialize, Serialize)] +#[serde(deny_unknown_fields)] +struct OperatorPlan { + catalog: [u8; 32], + namespace: [u8; 32], + request: [u8; 32], + plan: Vec, +} + +impl FleetCatalog { + pub(crate) async fn prepare_transition_child_routes( + &self, + snapshot: &FleetSnapshot, + directory: &RangeDirectory, + class: &str, + proposals: Vec, + timeout: Duration, + ) -> Result> { + self.check_snapshot(snapshot)?; + self.run(timeout, async { + self.require_namespace_declaration(snapshot, directory.namespace(), Some(class)) + .await?; + let owner = self.managed_owner_for_birth(snapshot, directory).await?; + proposals + .into_iter() + .map(|proposal| { + Ok(directory + .prepare_fleet_group_birth(proposal, &owner) + .map_err(catalog_error)? + .route() + .clone()) + }) + .collect() + }) + .await + } + + pub(crate) async fn reopen_transition_operator_plan( + &self, + directory: &RangeDirectory, + operation: ProposalId, + request: [u8; 32], + timeout: Duration, + ) -> Result> { + self.run(timeout, async { + let Some((record, _)) = self + .inner + .read::(&operator_plan_key(operation), ADMISSION, ADMISSION_LIMIT) + .await? + else { + return Ok(None); + }; + if record.catalog != self.catalog_id()? + || record.namespace != directory.namespace().root_digest() + || record.request != request + { + return Err(FleetCatalogError::PayloadConflict); + } + let plan = directory + .decode_mutation(&record.plan) + .map_err(catalog_error)?; + if plan.operation() != operation { + return Err(FleetCatalogError::Corrupt( + "operator transition plan operation", + )); + } + Ok(Some(plan)) + }) + .await + } + + /// Reopen a child reservation only when the indexed history and its + /// committed receipt belong to this exact operator step. + pub(crate) async fn reopen_transition_operator_child( + &self, + snapshot: &FleetSnapshot, + directory: &RangeDirectory, + route: &GroupRoute, + step: &TransitionOperatorStep, + timeout: Duration, + ) -> Result { + self.check_snapshot(snapshot)?; + let deadline = deadline::Deadline::new(self, timeout)?; + let binding = self.binding(&route.history)?; + let key = binding + .key() + .map_err(|_| FleetCatalogError::Invalid("transition child binding"))?; + let found = self + .run(deadline.remaining()?, async { + let Some(bytes) = self + .inner + .lookup(snapshot.head.tables[FleetTable::Histories.slot()], &key) + .await? + else { + return Ok(false); + }; + let history = FleetHistory::decode(&bytes)?; + if history.binding != binding + || history.operation != step.operation.bytes() + || history.state != FleetHistoryState::Reserved + || !history.check_route(directory, route) + { + return Err(FleetCatalogError::Conflict); + } + let (mutation, _) = self + .inner + .read::( + &format!("requests/{}", hex::encode(history.admission)), + REQUEST_KIND, + REQUEST_LIMIT, + ) + .await? + .ok_or(FleetCatalogError::Corrupt( + "missing transition child receipt", + ))?; + let expected = step.identity(mutation.expected_revision); + if history.admission != expected.id + || mutation.id != expected.id + || mutation.principal != expected.principal + || mutation.operation != expected.operation + { + return Err(FleetCatalogError::PayloadConflict); + } + Ok(true) + }) + .await?; + if !found { + return Ok(false); + } + let authorization = self + .recover_birth_reservation(snapshot, directory, &route.history, deadline.remaining()?) + .await?; + if authorization.route() != route { + return Err(FleetCatalogError::Corrupt( + "operator child reservation route", + )); + } + Ok(true) + } + + pub(crate) async fn stage_transition_operator_plan( + &self, + snapshot: &FleetSnapshot, + directory: &RangeDirectory, + identity: &FleetRequestIdentity, + request: [u8; 32], + plan: &DirectoryMutation, + timeout: Duration, + ) -> Result<()> { + self.check_snapshot(snapshot)?; + validate_request(identity)?; + self.run( + timeout, + Box::pin(async { + let current = self.inner.current().await?; + if current.token != snapshot.token { + return Err(FleetCatalogError::Conflict); + } + if let Some(active) = self.active_disruption(¤t).await? { + return Err(FleetCatalogError::DisruptionInProgress { + operation: active.operation(), + }); + } + if let Some(restart) = self.active_restart_operator_inner(¤t).await? { + return Err(FleetCatalogError::DisruptionInProgress { + operation: restart.identity().operation, + }); + } + self.require_transition_membership_idle(¤t).await?; + self.inner + .domain + .run(Box::pin(directory.validate_managed_transition_input(plan))) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(catalog_error)?; + let record = OperatorPlan { + catalog: self.catalog_id()?, + namespace: directory.namespace().root_digest(), + request, + plan: plan.encode().to_vec(), + }; + let bytes = encode(ADMISSION, &record, ADMISSION_LIMIT)?; + self.inner + .create_exact( + &operator_plan_key(plan.operation()), + &bytes, + ADMISSION_LIMIT, + ) + .await + }), + ) + .await + } +} + +fn operator_plan_key(operation: ProposalId) -> String { + format!( + "transition-operator-plans/{}", + hex::encode(operation.bytes()) + ) +} diff --git a/crates/cellar/durability/raft/fleet_catalog/managed_transition_replay.rs b/crates/cellar/durability/raft/fleet_catalog/managed_transition_replay.rs new file mode 100644 index 000000000..843b58895 --- /dev/null +++ b/crates/cellar/durability/raft/fleet_catalog/managed_transition_replay.rs @@ -0,0 +1,581 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Resume a managed transfer from its exact retained source cuts. + +use super::{histories::validate_request, managed_transitions::transfer_archive_error, *}; +use crate::raft::{ + catalog::RangeDirectory, + cell_groups::{ + CellGroupArchiveHold, CellGroupArchivePool, CellGroupArchiveTailPool, + CellGroupRecoveryPool, CellGroupTransitionPreparationInputs, CellGroupTransitionSource, + PreparedManagedCellGroupTransition, TransitionJournal, TransitionPhase, + VerifiedCellGroupTransitionSources, + }, +}; +use cellar_logic::{ + cell_group::{GroupHistory, GroupRoute}, + consensus::ProposalId, +}; + +impl FleetCatalog { + /// Stage operation-owned holds from the selected active parent archives. + /// A retry reconciles the rooted acquire set before any new allocation. + pub async fn allocate_managed_preparation_holds( + &self, + directory: &RangeDirectory, + operation: ProposalId, + archives: &CellGroupArchivePool, + timeout: Duration, + ) -> Result { + let deadline = deadline::Deadline::new(self, timeout)?; + let transition = self + .observe_transfer_replay( + directory, + operation, + &[TransitionPhase::Planned, TransitionPhase::Prepared], + deadline.remaining()?, + ) + .await?; + let permit = self + .run( + deadline.remaining()?, + Box::pin(self.active_managed_transition_permit( + &transition.request, + directory, + &transition.plan, + )), + ) + .await?; + let domain = &self.inner.domain; + domain + .run(Box::pin(crate::asyncrt::timeout( + deadline.remaining()?, + Box::pin(directory.allocate_managed_preparation_holds( + domain, + archives, + &transition.plan, + &permit, + )), + ))) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(|_| FleetCatalogError::Unconfirmed)? + .map_err(transfer_archive_error) + } + + /// Clean up only the cancelled transition's own acquire intents. The + /// controller must join entered pre-copy work before invoking cleanup. + pub async fn release_cancelled_preparation_holds( + &self, + directory: &RangeDirectory, + operation: ProposalId, + archives: &CellGroupArchivePool, + timeout: Duration, + ) -> Result<()> { + let deadline = deadline::Deadline::new(self, timeout)?; + let transition = self + .reopen_managed_transition(directory, operation, deadline.remaining()?) + .await? + .ok_or(FleetCatalogError::Conflict)?; + let domain = &self.inner.domain; + domain + .run(Box::pin(crate::asyncrt::timeout( + deadline.remaining()?, + Box::pin(directory.release_cancelled_preparation_acquires( + domain, + archives, + &transition.plan, + )), + ))) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(|_| FleetCatalogError::Unconfirmed)? + .map_err(transfer_archive_error) + } + + /// Persist the controller's own active source holds in plan order before + /// opening physical pre-copy receivers. Keep the selected holds alive + /// through retirement, or join pre-copy work before cancelled cleanup. + pub async fn stage_managed_preparation_holds( + &self, + transition: &ReopenedManagedCellGroupTransition, + directory: &RangeDirectory, + archives: &CellGroupArchivePool, + holds: &[CellGroupArchiveHold], + timeout: Duration, + ) -> Result { + validate_request(&transition.request)?; + let deadline = deadline::Deadline::new(self, timeout)?; + let permit = self + .run( + deadline.remaining()?, + Box::pin(self.active_managed_transition_permit( + &transition.request, + directory, + &transition.plan, + )), + ) + .await?; + let domain = &self.inner.domain; + domain + .run(Box::pin(crate::asyncrt::timeout( + deadline.remaining()?, + Box::pin(directory.stage_managed_preparation_holds( + domain, + archives, + &transition.plan, + &permit, + holds, + None, + )), + ))) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(|_| FleetCatalogError::Unconfirmed)? + .map_err(transfer_archive_error) + } + + /// Observe the selected inputs without opening physical data or changing + /// the decision. Cancelled transitions retain the descriptors so the hold + /// owner can reconcile and clean up after joining its work. + pub async fn staged_managed_preparation_holds( + &self, + directory: &RangeDirectory, + operation: ProposalId, + timeout: Duration, + ) -> Result>> { + let deadline = deadline::Deadline::new(self, timeout)?; + let transition = self + .reopen_managed_transition(directory, operation, deadline.remaining()?) + .await? + .ok_or(FleetCatalogError::Conflict)?; + self.inner + .domain + .run(Box::pin(crate::asyncrt::timeout( + deadline.remaining()?, + Box::pin(directory.staged_managed_preparation_holds(&transition.plan)), + ))) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(|_| FleetCatalogError::Unconfirmed)? + .map_err(transfer_archive_error) + } + + /// Independently rebuild the selected pre-copy after a controller restart. + /// Staged inputs can advance a Planned transition. Reopening a Prepared + /// transition must reproduce the same physical partition and journal. + pub async fn resume_managed_preparation( + &self, + directory: &RangeDirectory, + operation: ProposalId, + verifier: &FleetGroupArchiveVerifier, + restore: CellGroupRecoveryPool, + timeout: Duration, + ) -> Result { + let deadline = deadline::Deadline::new(self, timeout)?; + let transition = self + .observe_transfer_replay( + directory, + operation, + &[TransitionPhase::Planned, TransitionPhase::Prepared], + deadline.remaining()?, + ) + .await?; + let domain = &self.inner.domain; + let inputs = domain + .run(Box::pin(crate::asyncrt::timeout( + deadline.remaining()?, + Box::pin(async { + let holds = match directory + .staged_managed_preparation_holds(&transition.plan) + .await + .map_err(transfer_archive_error)? + { + Some(holds) => holds, + None if transition.journal.as_ref().is_some_and(|journal| { + journal.phase() == TransitionPhase::Prepared + }) => + { + directory + .managed_preparation_holds(&transition.plan) + .await + .map_err(histories::catalog_error)? + } + None => return Err(FleetCatalogError::Unconfirmed), + }; + let sources = + open_sources_for_holds(domain, directory, &transition, verifier, holds) + .await?; + transfer_inputs(domain, directory, &transition, verifier, restore, sources) + .await + }), + ))) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(|_| FleetCatalogError::Unconfirmed)??; + self.prepare_managed_transition( + &transition.request, + directory, + &transition.plan, + inputs, + deadline.remaining()?, + ) + .await + } + + /// Independently verify and durably select one sealed parent hold. Retain + /// the exact hold through retirement. After an uncertain result, observe + /// the staged slots before resolving the original hold allocation. + pub async fn stage_managed_sealed_source( + &self, + transition: &ReopenedManagedCellGroupTransition, + directory: &RangeDirectory, + archives: &CellGroupArchivePool, + tails: &CellGroupArchiveTailPool, + source: CellGroupTransitionSource, + timeout: Duration, + ) -> Result<()> { + validate_request(&transition.request)?; + let deadline = deadline::Deadline::new(self, timeout)?; + let permit = self + .run( + deadline.remaining()?, + Box::pin(self.active_managed_transition_permit( + &transition.request, + directory, + &transition.plan, + )), + ) + .await?; + let domain = &self.inner.domain; + domain + .run(Box::pin(crate::asyncrt::timeout( + deadline.remaining()?, + Box::pin(directory.stage_managed_sealed_source( + domain, + archives, + tails, + &transition.plan, + (source, None), + &permit, + )), + ))) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(|_| FleetCatalogError::Unconfirmed)? + .map_err(transfer_archive_error) + } + + /// Capture the selected parent's archive under the irreversible plan. + /// An uncertain reply leaves an exact operation-owned slot for the next + /// controller to reconcile before opening another physical receiver. + pub async fn capture_managed_sealed_source( + &self, + directory: &RangeDirectory, + operation: ProposalId, + history: GroupHistory, + verifier: &FleetGroupArchiveVerifier, + timeout: Duration, + ) -> Result { + let deadline = deadline::Deadline::new(self, timeout)?; + let transition = self + .observe_transfer_replay( + directory, + operation, + &[ + TransitionPhase::SealAuthorized, + TransitionPhase::Sealed, + TransitionPhase::Installed, + TransitionPhase::Archived, + ], + deadline.remaining()?, + ) + .await?; + let permit = self + .run( + deadline.remaining()?, + Box::pin(self.active_managed_transition_permit( + &transition.request, + directory, + &transition.plan, + )), + ) + .await?; + let domain = &self.inner.domain; + domain + .run(Box::pin(crate::asyncrt::timeout( + deadline.remaining()?, + Box::pin(async { + let (hold, intent) = directory + .acquire_managed_sealed_source( + domain, + verifier.archives(), + &transition.plan, + &permit, + history, + ) + .await + .map_err(transfer_archive_error)?; + if let Some(intent) = intent { + let birth = directory + .materialize_selected_history(history) + .await + .map_err(histories::catalog_error)?; + let receiver = verifier + .receiver( + domain, + hold.marker().route(), + directory.namespace_config().image_storage, + ) + .await?; + Box::pin(directory.stage_managed_sealed_source( + domain, + verifier.archives(), + verifier.tails(), + &transition.plan, + ( + CellGroupTransitionSource { + birth, + hold: hold.clone(), + receiver, + }, + Some(intent), + ), + &permit, + )) + .await + .map_err(transfer_archive_error)?; + } + Ok(hold) + }), + ))) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(|_| FleetCatalogError::Unconfirmed)? + } + + /// Read the plan-order staged holds for the active operation. Missing + /// slots identify parents that still need a verified archive cut. + pub async fn staged_managed_sealed_sources( + &self, + directory: &RangeDirectory, + operation: ProposalId, + timeout: Duration, + ) -> Result)>> { + let deadline = deadline::Deadline::new(self, timeout)?; + let transition = self + .observe_transfer_replay( + directory, + operation, + &[ + TransitionPhase::SealAuthorized, + TransitionPhase::Sealed, + TransitionPhase::Installed, + TransitionPhase::Archived, + ], + deadline.remaining()?, + ) + .await?; + self.inner + .domain + .run(Box::pin(crate::asyncrt::timeout( + deadline.remaining()?, + Box::pin(directory.managed_staged_source_holds(&transition.plan)), + ))) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(|_| FleetCatalogError::Unconfirmed)? + .map_err(transfer_archive_error) + } + + /// Rebuild the exact sealed input proofs for a replacement controller. + /// The verifier must have room to pin one complete cut per parent. Reuse + /// the returned proofs for destination install while the holds stay live. + /// A complete set of staged holds can establish the Sealed decision. + pub async fn replay_managed_sealed_sources( + &self, + directory: &RangeDirectory, + operation: ProposalId, + verifier: &FleetGroupArchiveVerifier, + timeout: Duration, + ) -> Result { + let deadline = deadline::Deadline::new(self, timeout)?; + let transition = self + .observe_transfer_replay( + directory, + operation, + &[ + TransitionPhase::SealAuthorized, + TransitionPhase::Sealed, + TransitionPhase::Installed, + TransitionPhase::Archived, + ], + deadline.remaining()?, + ) + .await?; + let domain = &self.inner.domain; + let sources = domain + .run(Box::pin(crate::asyncrt::timeout( + deadline.remaining()?, + open_transfer_sources(domain, directory, &transition, verifier), + ))) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(|_| FleetCatalogError::Unconfirmed)??; + self.verify_managed_sealed_sources( + &transition, + directory, + verifier.archives(), + verifier.tails(), + sources, + deadline.remaining()?, + ) + .await + } + + /// Resume transfer-archive verification after the complete installed set + /// was selected. Rebuild every child in the supplied verifier's private + /// inventory and commit only if all installed roots match. + pub async fn resume_managed_transfer_archive( + &self, + directory: &RangeDirectory, + operation: ProposalId, + verifier: &FleetGroupArchiveVerifier, + restore: CellGroupRecoveryPool, + timeout: Duration, + ) -> Result { + let deadline = deadline::Deadline::new(self, timeout)?; + let transition = self + .observe_transfer_replay( + directory, + operation, + &[TransitionPhase::Installed, TransitionPhase::Archived], + deadline.remaining()?, + ) + .await?; + let domain = &self.inner.domain; + let inputs = domain + .run(Box::pin(crate::asyncrt::timeout( + deadline.remaining()?, + Box::pin(async { + let sources = + open_transfer_sources(domain, directory, &transition, verifier).await?; + transfer_inputs(domain, directory, &transition, verifier, restore, sources) + .await + }), + ))) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(|_| FleetCatalogError::Unconfirmed)??; + self.verify_managed_transfer_archive(&transition, directory, inputs, deadline.remaining()?) + .await + } + + async fn observe_transfer_replay( + &self, + directory: &RangeDirectory, + operation: ProposalId, + phases: &[TransitionPhase], + timeout: Duration, + ) -> Result { + let observation = self + .observe_managed_transition(directory, operation, timeout) + .await? + .filter(|observation| observation.fleet_slot_held) + .ok_or(FleetCatalogError::Conflict)?; + if !observation + .transition + .journal + .as_ref() + .is_some_and(|journal| phases.contains(&journal.phase())) + { + return Err(FleetCatalogError::Conflict); + } + Ok(observation.transition) + } +} + +async fn open_transfer_sources( + domain: &DomainToken, + directory: &RangeDirectory, + transition: &ReopenedManagedCellGroupTransition, + verifier: &FleetGroupArchiveVerifier, +) -> Result> { + let holds = if transition + .journal + .as_ref() + .is_some_and(|journal| journal.phase() == TransitionPhase::SealAuthorized) + { + directory + .managed_staged_source_holds(&transition.plan) + .await + .map_err(transfer_archive_error)? + .into_iter() + .map(|(_, hold)| hold.ok_or(FleetCatalogError::Unconfirmed)) + .collect::>>()? + } else { + directory + .sealed_source_holds(&transition.plan) + .await + .map_err(transfer_archive_error)? + }; + open_sources_for_holds(domain, directory, transition, verifier, holds).await +} + +async fn open_sources_for_holds( + domain: &DomainToken, + directory: &RangeDirectory, + transition: &ReopenedManagedCellGroupTransition, + verifier: &FleetGroupArchiveVerifier, + holds: Vec, +) -> Result> { + let (parents, _) = directory + .transition_routes(&transition.plan) + .map_err(histories::catalog_error)?; + if parents.len() != holds.len() { + return Err(FleetCatalogError::Corrupt("managed transition hold count")); + } + let storage = directory.namespace_config().image_storage; + let mut sources = Vec::with_capacity(parents.len()); + for (route, hold) in parents.into_iter().zip(holds) { + let birth = directory + .materialize_selected_history(route.history) + .await + .map_err(histories::catalog_error)?; + if birth.route() != &route || hold.marker().route() != &route { + return Err(FleetCatalogError::Conflict); + } + let receiver = verifier.receiver(domain, &route, storage).await?; + sources.push(CellGroupTransitionSource { + birth, + hold, + receiver, + }); + } + Ok(sources) +} + +async fn transfer_inputs( + domain: &DomainToken, + directory: &RangeDirectory, + transition: &ReopenedManagedCellGroupTransition, + verifier: &FleetGroupArchiveVerifier, + restore: CellGroupRecoveryPool, + sources: Vec, +) -> Result { + let (_, children) = directory + .transition_routes(&transition.plan) + .map_err(histories::catalog_error)?; + let storage = directory.namespace_config().image_storage; + let mut targets = Vec::with_capacity(children.len()); + for route in &children { + targets.push(verifier.transfer_target(domain, route, storage).await?); + } + Ok(CellGroupTransitionPreparationInputs { + archives: verifier.archives().clone(), + tails: verifier.tails().clone(), + restore, + sources, + targets, + }) +} diff --git a/crates/cellar/durability/raft/fleet_catalog/managed_transitions.rs b/crates/cellar/durability/raft/fleet_catalog/managed_transitions.rs new file mode 100644 index 000000000..36a5597cd --- /dev/null +++ b/crates/cellar/durability/raft/fleet_catalog/managed_transitions.rs @@ -0,0 +1,2234 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Rooted fleet ownership of a managed split or merge before any seal. +use super::disruption::{Disruption, SLOT_KEY}; +use super::histories::{catalog_error, history_mutation, validate_request, Origin}; +use super::*; +use crate::application::consensus::RaftCells; +use crate::raft::{ + catalog::CatalogError, + cell_groups::{ + CellGroupArchiveBindingError, CellGroupArchiveCoverage, CellGroupArchiveHold, + CellGroupArchivePool, CellGroupArchiveTailPool, CellGroupSourceRetirementAck, + CellGroupTransferActivationAck, CellGroupTransferArchive, + CellGroupTransitionPreparationInputs, CellGroupTransitionSource, DirectoryMutation, + DirectoryPublication, PreparedManagedCellGroupTransition, RangeDirectory, + TransferChildQuorum, TransferInstallQuorum, TransitionJournal, + VerifiedCellGroupTransitionSources, + }, + fleet_stop::BootBinding, + group_host::RaftGroupArchiveInitialization, + member_state::FleetGroupBinding, +}; + +use cellar_logic::{ + cell_group::{GroupHistory, GroupRoute}, + consensus::{LogPosition, NodeId, ProposalId}, +}; +use serde::Deserialize; +use std::collections::BTreeMap; + +pub(super) const KIND: u8 = 201; +const OPERATION_PREFIX: &str = "managed-transition/"; +const RECORD_LIMIT: usize = 4096; + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Record { + catalog: [u8; 32], + namespace_root: [u8; 32], + input_root: [u8; 32], + plan: [u8; 32], + request: FleetRequestIdentity, +} + +impl Record { + pub(super) fn validate(&self, fleet: &FleetCatalog) -> Result<()> { + validate_request(&self.request)?; + if self.catalog != fleet.catalog_id()? + || [self.namespace_root, self.input_root, self.plan].contains(&[0; 32]) + { + return Err(FleetCatalogError::Invalid("managed transition binding")); + } + Ok(()) + } + + pub(super) fn operation(&self) -> [u8; 16] { + self.request.operation + } + + pub(super) fn key(&self) -> String { + operation_key(self.operation()) + } + + pub(super) fn check_basis(&self, snapshot: &FleetSnapshot) -> Result<()> { + if self.request.expected_revision != snapshot.revision() + || self.input_root != root_digest(snapshot)? + { + return Err(FleetCatalogError::Conflict); + } + Ok(()) + } + + pub(super) fn mutation(&self) -> Result { + let value = encode(KIND, self, RECORD_LIMIT)?; + Ok(history_mutation( + &self.request, + vec![ + FleetChange { + table: FleetTable::Operations, + key: self.key(), + value: Some(value.clone()), + }, + FleetChange { + table: FleetTable::Policy, + key: SLOT_KEY.into(), + value: Some(value), + }, + ], + )) + } +} + +fn operation_key(operation: [u8; 16]) -> String { + format!("{OPERATION_PREFIX}{}", hex::encode(operation)) +} + +pub(super) fn transfer_archive_error(error: CellGroupArchiveBindingError) -> FleetCatalogError { + match error { + CellGroupArchiveBindingError::Catalog(error) => catalog_error(error), + CellGroupArchiveBindingError::Archive(error) => error.into(), + } +} + +pub(super) fn root_digest(snapshot: &FleetSnapshot) -> Result<[u8; 32]> { + Ok(hash(&encode(HEAD_KIND, &snapshot.head, ROOT_LIMIT)?)) +} + +/// Private capability recovered only from an exact committed fleet intent. +pub(crate) struct ManagedTransitionPermit(Record); + +impl ManagedTransitionPermit { + pub(crate) fn check( + &self, + directory: &RangeDirectory, + plan: &DirectoryMutation, + ) -> std::result::Result<(), CatalogError> { + self.check_directory(directory)?; + let record = &self.0; + if plan.operation().bytes() != record.operation() || plan.digest() != record.plan { + return Err(CatalogError::ManagedAdmissionRequired); + } + Ok(()) + } + + fn check_directory(&self, directory: &RangeDirectory) -> std::result::Result<(), CatalogError> { + let record = &self.0; + if directory + .namespace_config() + .managed_fleet + .as_ref() + .map(|owner| owner.catalog) + != Some(record.catalog) + || directory.namespace().root_digest() != record.namespace_root + { + return Err(CatalogError::ManagedAdmissionRequired); + } + Ok(()) + } +} + +/// Exact admitted plan and any directory decision. A missing decision means +/// the fleet selected the plan before directory publication was confirmed. +/// This observation grants no sealing, installation or serving authority. +pub struct ReopenedManagedCellGroupTransition { + pub request: FleetRequestIdentity, + pub plan: DirectoryMutation, + pub journal: Option, +} + +enum TransferCompletion { + Replayed(Box), + Pending { + mutation: FleetMutation, + disruption: [u8; 32], + release_holds: bool, + }, +} + +pub(crate) struct ManagedTransitionObservation { + pub(crate) fleet_revision: u64, + pub(crate) fleet_slot_held: bool, + pub(crate) transition: ReopenedManagedCellGroupTransition, +} + +pub(crate) struct VerifiedManagedTransitionArchives { + pub(crate) intent_basis: ManagedRecoveryIntentBasis, + pub(crate) archives: Vec, + pub(crate) lineages: Vec, +} + +pub(crate) struct ManagedTransitionRecoveryObservation { + pub(crate) fleet_revision: u64, + pub(crate) fleet_digest: [u8; 32], + pub(crate) selection: crate::raft::catalog::ManagedTransitionRecoverySelection, +} + +impl FleetCatalog { + /// Reopen an admitted transition by operation ID. This reads the original + /// fleet request and canonical plan bytes even after the caller loses its + /// local copy. Resume planning by resubmitting the returned exact request. + pub async fn reopen_managed_transition( + &self, + directory: &RangeDirectory, + operation: ProposalId, + timeout: Duration, + ) -> Result> { + Ok(self + .observe_managed_transition(directory, operation, timeout) + .await? + .map(|observation| observation.transition)) + } + + pub(crate) async fn observe_managed_transition( + &self, + directory: &RangeDirectory, + operation: ProposalId, + timeout: Duration, + ) -> Result> { + self.run( + timeout, + Box::pin(async { + let current = self.inner.current().await?; + self.observe_managed_transition_at(¤t, directory, operation) + .await + }), + ) + .await + } + + // Reuse the caller's exact rooted view when a Fleet turn is already held. + pub(super) async fn observe_managed_transition_at( + &self, + current: &FleetSnapshot, + directory: &RangeDirectory, + operation: ProposalId, + ) -> Result> { + let Some(record) = self + .managed_transition_record(current, operation.bytes()) + .await? + else { + return Ok(None); + }; + let permit = ManagedTransitionPermit(record); + permit.check_directory(directory).map_err(catalog_error)?; + let record = &permit.0; + let active = matches!( + self.active_disruption(current).await?, + Some(Disruption::Transition(active)) if active == *record + ); + let (plan, journal) = self + .inner + .domain + .run(Box::pin(directory.reopen_managed_transition_plan( + operation, + record.plan, + &permit, + ))) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(catalog_error)?; + if !active + && !journal + .as_ref() + .is_some_and(|decision| !decision.owns_topology()) + { + return Err(FleetCatalogError::Corrupt( + "released managed transition has no terminal decision", + )); + } + Ok(Some(ManagedTransitionObservation { + fleet_revision: current.revision(), + fleet_slot_held: active, + transition: ReopenedManagedCellGroupTransition { + request: record.request.clone(), + plan, + journal, + }, + })) + } + + pub(crate) async fn observe_managed_transition_recovery( + &self, + directory: &RangeDirectory, + operation: ProposalId, + timeout: Duration, + ) -> Result { + self.run( + timeout, + Box::pin(async { + let snapshot = self.inner.current().await?; + let fleet_digest = root_digest(&snapshot)?; + let Some(Disruption::Transition(record)) = + self.active_disruption(&snapshot).await? + else { + return Err(FleetCatalogError::Conflict); + }; + if record.operation() != operation.bytes() { + return Err(FleetCatalogError::Conflict); + } + let digest = record.plan; + let permit = ManagedTransitionPermit(record); + permit.check_directory(directory).map_err(catalog_error)?; + let selection = self + .inner + .domain + .run(Box::pin(directory.observe_managed_transition_recovery( + operation, digest, &permit, + ))) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(catalog_error)?; + if root_digest(&self.inner.current().await?)? != fleet_digest { + return Err(FleetCatalogError::Conflict); + } + Ok(ManagedTransitionRecoveryObservation { + fleet_revision: snapshot.revision(), + fleet_digest, + selection, + }) + }), + ) + .await + } + + pub(crate) async fn verify_managed_transition_recovery_archives( + &self, + directory: &RangeDirectory, + observation: &ManagedTransitionRecoveryObservation, + request: &FleetRequestIdentity, + holds: Vec, + verifier: &FleetGroupArchiveVerifier, + timeout: Duration, + ) -> Result { + self.run( + timeout, + Box::pin(async { + let snapshot = self.inner.current().await?; + if root_digest(&snapshot)? != observation.fleet_digest { + return Err(FleetCatalogError::Conflict); + } + let selection = &observation.selection; + let permit = self + .managed_transition_permit_at(&snapshot, request, directory, &selection.plan) + .await?; + let domain = &self.inner.domain; + domain + .run(Box::pin(directory.check_managed_transition_recovery_holds( + selection, &permit, &holds, + ))) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(catalog_error)?; + let storage = directory.namespace_config().image_storage; + let mut verified = Vec::with_capacity(holds.len()); + for (route, hold) in selection.archive_routes().iter().zip(&holds) { + let birth = domain + .run(directory.materialize_selected_history(route.history)) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(catalog_error)?; + if birth.route() != route { + return Err(FleetCatalogError::Conflict); + } + let registry = directory + .open_group_archive(domain, verifier.archives(), &birth) + .await + .map_err(transfer_archive_error)?; + registry.require_selected_hold(domain, hold).await?; + let receiver = verifier.receiver(domain, route, storage).await?; + let coverage = registry + .verify_hold(domain, hold, verifier.tails(), receiver) + .await?; + verified.push((registry, coverage)); + } + let lineages = self + .transition_recovery_voter_lineages( + &snapshot, + directory, + selection, + verified.iter().map(|(_, coverage)| coverage), + ) + .await?; + for (registry, coverage) in &verified { + registry + .require_selected_hold(domain, coverage.hold()) + .await?; + } + let current = domain + .run(directory.observe()) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(catalog_error)?; + if current.digest() != selection.directory_digest + || root_digest(&self.inner.current().await?)? != observation.fleet_digest + { + return Err(FleetCatalogError::Conflict); + } + Ok(VerifiedManagedTransitionArchives { + intent_basis: ManagedRecoveryIntentBasis { + transition: selection.plan.operation().bytes(), + plan: selection.plan.digest(), + fleet_root: observation.fleet_digest, + fleet_revision: observation.fleet_revision, + directory_root: selection.directory_digest, + publication: selection.publication, + }, + archives: verified.into_iter().map(|(_, coverage)| coverage).collect(), + lineages, + }) + }), + ) + .await + } + + /// Acquire the fleet disruption slot and plan the exact range mutation. + /// Reserve all child births against the same operation before calling. + /// Persist the request and plan before submission. Replaying both after + /// controller loss recovers the original admission or cancellation. + /// This step grants no seal, installation or serving authority. + pub async fn plan_managed_transition( + &self, + request: &FleetRequestIdentity, + directory: &RangeDirectory, + plan: &DirectoryMutation, + timeout: Duration, + ) -> Result { + validate_request(request)?; + let (parents, children) = directory.transition_routes(plan).map_err(catalog_error)?; + self.run( + timeout, + Box::pin(async { + let current = self.inner.current().await?; + let original = self + .managed_transition_record(¤t, request.operation) + .await?; + let replay = original.is_some(); + let record = match original { + Some(record) if record.request == *request => record, + Some(_) => return Err(FleetCatalogError::PayloadConflict), + None => Record { + catalog: self.catalog_id()?, + namespace_root: directory.namespace().root_digest(), + input_root: root_digest(¤t)?, + plan: plan.digest(), + request: request.clone(), + }, + }; + let permit = ManagedTransitionPermit(record); + permit.check(directory, plan).map_err(catalog_error)?; + let record = &permit.0; + if replay { + if !matches!( + self.active_disruption(¤t).await?, + Some(Disruption::Transition(active)) if active == *record + ) { + let status = self + .inner + .domain + .run(Box::pin(directory.managed_transition_status(plan, &permit))) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(catalog_error)?; + return status.filter(|journal| !journal.owns_topology()).ok_or( + FleetCatalogError::Corrupt("missing terminal managed transition"), + ); + } + } else { + let snapshot = ¤t; + record.validate(self)?; + record.check_basis(snapshot)?; + if let Some(restart) = self.active_restart_operator_inner(snapshot).await? { + return Err(FleetCatalogError::DisruptionInProgress { + operation: restart.identity().operation, + }); + } + if let Some(active) = self.active_disruption(snapshot).await? { + return Err(FleetCatalogError::DisruptionInProgress { + operation: active.operation(), + }); + } + self.require_namespace_declaration(snapshot, directory.namespace(), None) + .await?; + self.require_transition_membership_idle(snapshot).await?; + for (route, state) in parents + .iter() + .map(|route| (route, FleetHistoryState::Live)) + .chain( + children + .iter() + .map(|route| (route, FleetHistoryState::Reserved)), + ) + { + let binding = self.binding(&route.history)?; + let key = binding + .key() + .map_err(|_| FleetCatalogError::Invalid("transition group binding"))?; + let history = self.history_inner(snapshot, &key).await?; + if history.origin != Origin::Managed + || history.state != state + || !history.check_route(directory, route) + || state == FleetHistoryState::Reserved + && history.operation != record.operation() + { + return Err(FleetCatalogError::Conflict); + } + } + self.inner + .domain + .run(Box::pin( + directory.stage_managed_transition_plan(plan, &permit), + )) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(catalog_error)?; + let mutation = record.mutation()?; + mutation.validate()?; + // The request journal persists this exact root-bound + // attempt. An uncommitted candidate owns no operation key. + self.commit_with_disruption( + snapshot, + &mutation, + None, + resources::Authority::ManagedTransition, + || Ok(()), + ) + .await?; + } + self.inner + .domain + .run(Box::pin(directory.plan_managed_transition(plan, &permit))) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(catalog_error) + }), + ) + .await + } + + /// The supplied archive tail pool must admit a pinned cut per parent. + /// Independently replay the planned active parent holds and sync all + /// non-serving child partitions. The directory selects the resulting + /// immutable preparation under the same CAS as pre-seal cancellation. + /// A replacement controller replays the exact recorded source holds. + pub async fn prepare_managed_transition( + &self, + request: &FleetRequestIdentity, + directory: &RangeDirectory, + plan: &DirectoryMutation, + inputs: CellGroupTransitionPreparationInputs, + timeout: Duration, + ) -> Result { + validate_request(request)?; + let deadline = deadline::Deadline::new(self, timeout)?; + let permit = self + .run( + deadline.remaining()?, + Box::pin(self.active_managed_transition_permit(request, directory, plan)), + ) + .await?; + // The catalog turn ends before replay/copy. The directory's final CAS + // still arbitrates cancellation, including release by another controller. + let remaining = deadline.remaining()?; + let domain = &self.inner.domain; + domain + .run(Box::pin(async { + crate::asyncrt::timeout( + remaining, + Box::pin( + directory.prepare_managed_transition_data(domain, plan, &permit, inputs), + ), + ) + .await + })) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(|_| FleetCatalogError::Unconfirmed)? + .map_err(transfer_archive_error) + } + + /// Verify and select the exact committed parent cuts of a managed transition. + /// Supply an independently configured receiver per held archive in plan + /// order. Retain every hold through retirement and retry with the same + /// descriptors after an unknown directory outcome. + pub async fn verify_managed_sealed_sources( + &self, + transition: &ReopenedManagedCellGroupTransition, + directory: &RangeDirectory, + archives: &CellGroupArchivePool, + tails: &CellGroupArchiveTailPool, + sources: Vec, + timeout: Duration, + ) -> Result { + validate_request(&transition.request)?; + let deadline = deadline::Deadline::new(self, timeout)?; + let permit = self + .run( + deadline.remaining()?, + Box::pin(self.active_managed_transition_permit( + &transition.request, + directory, + &transition.plan, + )), + ) + .await?; + let remaining = deadline.remaining()?; + let domain = &self.inner.domain; + domain + .run(Box::pin(async { + crate::asyncrt::timeout( + remaining, + Box::pin(directory.verify_and_record_managed_sealed_sources( + domain, + archives, + tails, + &transition.plan, + sources, + &permit, + )), + ) + .await + })) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(|_| FleetCatalogError::Unconfirmed)? + .map_err(transfer_archive_error) + } + + /// Independently rebuild all installed child roots from the retained + /// parent archives and select their durable transfer closure. The + /// directory keeps the parent ranges selected after verification. + pub async fn verify_managed_transfer_archive( + &self, + transition: &ReopenedManagedCellGroupTransition, + directory: &RangeDirectory, + inputs: CellGroupTransitionPreparationInputs, + timeout: Duration, + ) -> Result { + validate_request(&transition.request)?; + let deadline = deadline::Deadline::new(self, timeout)?; + let permit = self + .run( + deadline.remaining()?, + Box::pin(self.verified_managed_transfer_permit(transition, directory)), + ) + .await?; + // Replay and partition work use the caller's original deadline. The + // directory CAS binds the resulting closure to the installed set. + let remaining = deadline.remaining()?; + let domain = &self.inner.domain; + domain + .run(Box::pin(crate::asyncrt::timeout( + remaining, + Box::pin(directory.verify_and_record_managed_transfer_archive( + domain, + &transition.plan, + inputs, + &permit, + )), + ))) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(|_| FleetCatalogError::Unconfirmed)? + .map_err(transfer_archive_error) + } + + /// Reopen the archive closure selected for the exact active managed plan. + /// This reads the durable references and their install bindings. + pub async fn reopen_managed_transfer_archive( + &self, + transition: &ReopenedManagedCellGroupTransition, + directory: &RangeDirectory, + timeout: Duration, + ) -> Result> { + validate_request(&transition.request)?; + self.run( + timeout, + Box::pin(async { + let current = self.inner.current().await?; + let permit = self + .managed_transition_permit_at( + ¤t, + &transition.request, + directory, + &transition.plan, + ) + .await?; + let closure = self + .inner + .domain + .run(Box::pin( + directory.managed_transfer_archive(&transition.plan, &permit), + )) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(catalog_error)?; + if closure.is_some() { + self.verify_managed_transfer_storage_at( + ¤t, transition, directory, &permit, + ) + .await?; + } + if self.inner.current().await?.token != current.token { + return Err(FleetCatalogError::Conflict); + } + Ok(closure) + }), + ) + .await + } + + /// Select the archived children in one directory decision after rechecking + /// their reserved storage and all source holds. Replay uses the exact plan + /// and returns the selected publication even after a lost CAS response. + /// Publication grants no application admission to a pending child. + pub async fn publish_managed_transfer( + &self, + transition: &ReopenedManagedCellGroupTransition, + directory: &RangeDirectory, + archives: &CellGroupArchivePool, + timeout: Duration, + ) -> Result { + validate_request(&transition.request)?; + self.run( + timeout, + Box::pin(async { + let permit = self + .verified_managed_transfer_permit(transition, directory) + .await?; + let domain = &self.inner.domain; + domain + .run(Box::pin(directory.publish_managed_transfer( + domain, + archives, + &transition.plan, + &permit, + ))) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(|error| match error { + CellGroupArchiveBindingError::Catalog(error) => catalog_error(error), + CellGroupArchiveBindingError::Archive(error) => error.into(), + }) + }), + ) + .await + } + + /// Select a complete ordered set of durable activation acknowledgements. + /// The operation continues owning the topology lane for parent retirement. + pub async fn record_managed_transfer_activations( + &self, + transition: &ReopenedManagedCellGroupTransition, + directory: &RangeDirectory, + acknowledgements: Vec, + timeout: Duration, + ) -> Result { + validate_request(&transition.request)?; + self.run( + timeout, + Box::pin(async { + let permit = self + .verified_managed_transfer_permit(transition, directory) + .await?; + self.inner + .domain + .run(Box::pin(directory.record_managed_transfer_activations( + &transition.plan, + acknowledgements, + &permit, + ))) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(catalog_error) + }), + ) + .await + } + + pub(crate) async fn resume_managed_transfer_activation( + &self, + transition: &ReopenedManagedCellGroupTransition, + directory: &RangeDirectory, + history: GroupHistory, + acknowledgement: Option, + timeout: Duration, + ) -> Result> { + validate_request(&transition.request)?; + self.run( + timeout, + Box::pin(async { + let permit = self + .verified_managed_transfer_permit(transition, directory) + .await?; + self.inner + .domain + .run(Box::pin(directory.resume_managed_transfer_activation( + &transition.plan, + history, + acknowledgement, + &permit, + ))) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(catalog_error) + }), + ) + .await + } + + pub(crate) async fn initialize_managed_transfer_child_archive( + &self, + transition: &ReopenedManagedCellGroupTransition, + directory: &RangeDirectory, + history: GroupHistory, + archives: &CellGroupArchivePool, + timeout: Duration, + ) -> Result<()> { + validate_request(&transition.request)?; + let deadline = deadline::Deadline::new(self, timeout)?; + let permit = self + .run( + deadline.remaining()?, + Box::pin(self.verified_managed_transfer_permit(transition, directory)), + ) + .await?; + let domain = &self.inner.domain; + domain + .run(Box::pin(crate::asyncrt::timeout( + deadline.remaining()?, + Box::pin(directory.initialize_managed_child_archive( + domain, + archives, + &transition.plan, + history, + &permit, + )), + ))) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(|_| FleetCatalogError::Unconfirmed)? + .map_err(transfer_archive_error) + } + + /// Prepare a one-use archive identity for a child of the active transfer. + /// Persist the exact intent before committing it. Reopen or verify that + /// binding after an uncertain outcome. + pub async fn prepare_managed_transfer_child_archive( + &self, + transition: &ReopenedManagedCellGroupTransition, + directory: &RangeDirectory, + history: GroupHistory, + archives: &CellGroupArchivePool, + timeout: Duration, + ) -> Result { + validate_request(&transition.request)?; + self.run( + timeout, + Box::pin(async { + let permit = self + .verified_managed_transfer_permit(transition, directory) + .await?; + let domain = &self.inner.domain; + domain + .run(Box::pin(async { + let (birth, coverage_selected) = directory + .managed_transfer_child_archive_birth( + &transition.plan, + history, + &permit, + ) + .await + .map_err(catalog_error)?; + if coverage_selected + || directory + .has_group_archive_binding(&birth) + .await + .map_err(transfer_archive_error)? + { + return Err(FleetCatalogError::Conflict); + } + let prepared = directory + .prepare_group_archive_binding(domain, archives, &birth) + .await + .map_err(transfer_archive_error)?; + Ok(RaftGroupArchiveInitialization { prepared }) + })) + .await + .map_err(|_| FleetCatalogError::Closed)? + }), + ) + .await + } + + /// Verify the exact persisted child archive binding after an uncertain + /// setup reply. An incomplete registry refuses without creating state. + pub async fn verify_managed_transfer_child_archive( + &self, + transition: &ReopenedManagedCellGroupTransition, + directory: &RangeDirectory, + history: GroupHistory, + archives: &CellGroupArchivePool, + intent: &[u8], + timeout: Duration, + ) -> Result<()> { + validate_request(&transition.request)?; + self.run( + timeout, + Box::pin(async { + let permit = self + .verified_managed_transfer_permit(transition, directory) + .await?; + let domain = &self.inner.domain; + domain + .run(Box::pin(async { + let (birth, _) = directory + .managed_transfer_child_archive_birth( + &transition.plan, + history, + &permit, + ) + .await + .map_err(catalog_error)?; + directory + .verify_group_archive_binding(domain, archives, &birth, intent) + .await + .map_err(transfer_archive_error) + })) + .await + .map_err(|_| FleetCatalogError::Closed)? + }), + ) + .await + } + + /// Acquire the plan-owned child holds and replay each one into a private + /// receiver. A retry resumes the exact acquires selected before replay. + pub async fn verify_managed_transfer_child_archives( + &self, + transition: &ReopenedManagedCellGroupTransition, + directory: &RangeDirectory, + verifier: &FleetGroupArchiveVerifier, + timeout: Duration, + ) -> Result { + validate_request(&transition.request)?; + let deadline = deadline::Deadline::new(self, timeout)?; + let permit = self + .run( + deadline.remaining()?, + Box::pin(self.verified_managed_transfer_permit(transition, directory)), + ) + .await?; + let domain = &self.inner.domain; + domain + .run(Box::pin(crate::asyncrt::timeout( + deadline.remaining()?, + Box::pin(async { + let archives = verifier.archives(); + let held = directory + .acquire_managed_child_archives(domain, archives, &transition.plan, &permit) + .await + .map_err(transfer_archive_error)?; + let storage = directory.namespace_config().image_storage; + let mut children = Vec::with_capacity(held.len()); + for (birth, hold) in held { + let receiver = verifier.receiver(domain, birth.route(), storage).await?; + children.push(CellGroupTransitionSource { + birth, + hold, + receiver, + }); + } + directory + .verify_and_record_managed_child_archives( + domain, + archives, + verifier.tails(), + &transition.plan, + children, + &permit, + ) + .await + .map_err(transfer_archive_error) + }), + ))) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(|_| FleetCatalogError::Unconfirmed)? + } + + /// Reopen the selected references for retirement reconciliation. + pub async fn reopen_managed_transfer_child_holds( + &self, + transition: &ReopenedManagedCellGroupTransition, + directory: &RangeDirectory, + timeout: Duration, + ) -> Result>> { + validate_request(&transition.request)?; + self.run( + timeout, + Box::pin(async { + let permit = self + .verified_managed_transfer_permit(transition, directory) + .await?; + self.inner + .domain + .run(Box::pin( + directory.managed_retirement_child_holds(&transition.plan, &permit), + )) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(catalog_error) + }), + ) + .await + } + + pub(crate) async fn resume_managed_source_retirement( + &self, + transition: &ReopenedManagedCellGroupTransition, + directory: &RangeDirectory, + archives: &CellGroupArchivePool, + history: GroupHistory, + acknowledgement: Option, + timeout: Duration, + ) -> Result> { + validate_request(&transition.request)?; + let deadline = deadline::Deadline::new(self, timeout)?; + let permit = self + .run( + deadline.remaining()?, + Box::pin(self.verified_managed_transfer_permit(transition, directory)), + ) + .await?; + let domain = &self.inner.domain; + domain + .run(Box::pin(crate::asyncrt::timeout( + deadline.remaining()?, + Box::pin(directory.resume_managed_source_retirement( + domain, + archives, + &transition.plan, + history, + acknowledgement, + &permit, + )), + ))) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(|_| FleetCatalogError::Unconfirmed)? + .map_err(transfer_archive_error) + } + + /// Root the exact committed parent tombstones after verifying that every + /// selected child archive hold is still live. Both topology and the parent + /// archive holds remain owned pending cleanup. + pub async fn settle_managed_transfer_retirement( + &self, + transition: &ReopenedManagedCellGroupTransition, + directory: &RangeDirectory, + archives: &CellGroupArchivePool, + acknowledgements: Vec, + timeout: Duration, + ) -> Result { + validate_request(&transition.request)?; + let deadline = deadline::Deadline::new(self, timeout)?; + let permit = self + .run( + deadline.remaining()?, + Box::pin(self.verified_managed_transfer_permit(transition, directory)), + ) + .await?; + let domain = &self.inner.domain; + domain + .run(Box::pin(crate::asyncrt::timeout( + deadline.remaining()?, + Box::pin(directory.settle_managed_transfer_retirement( + domain, + archives, + &transition.plan, + acknowledgements, + &permit, + )), + ))) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(|_| FleetCatalogError::Unconfirmed)? + .map_err(transfer_archive_error) + } + + /// Promote every child of the retired transfer in one fleet CAS. The + /// exact request can be retried after a lost response. The disruption + /// slot and all parent capacity references remain owned. + pub async fn confirm_managed_transfer_children( + &self, + snapshot: &FleetSnapshot, + request: &FleetRequestIdentity, + transition: &ReopenedManagedCellGroupTransition, + directory: &RangeDirectory, + archives: &CellGroupArchivePool, + timeout: Duration, + ) -> Result { + self.check_snapshot(snapshot)?; + validate_request(request)?; + validate_request(&transition.request)?; + if request.operation != transition.request.operation + || request.id == transition.request.id + || request.principal != transition.request.principal + || request.expected_revision <= transition.request.expected_revision + { + return Err(FleetCatalogError::Invalid( + "managed transfer child confirmation", + )); + } + let (_, children) = directory + .transition_routes(&transition.plan) + .map_err(catalog_error)?; + self.run( + timeout, + Box::pin(async { + let mut all_reserved = true; + let mut changes = Vec::with_capacity(children.len()); + for route in &children { + let (key, mut history) = self + .transfer_child_history(snapshot, transition, directory, route) + .await?; + all_reserved &= history.state == FleetHistoryState::Reserved; + history.state = FleetHistoryState::Live; + changes.push(FleetChange { + table: FleetTable::Histories, + key, + value: Some(history.encode()?), + }); + } + let mutation = history_mutation(request, changes); + mutation.validate()?; + let current = self.inner.current().await?; + let digest = hash(&encode(REQUEST_KIND, &mutation, REQUEST_LIMIT)?); + let key = hex::encode(request.id); + if let Some(applied_revision) = + self.inner.committed(¤t.head, &key, digest).await? + { + return Ok(FleetCommit { + snapshot: current, + applied_revision, + }); + } + if !all_reserved + || current.token != snapshot.token + || request.expected_revision != snapshot.revision() + { + return Err(FleetCatalogError::Conflict); + } + let permit = self + .managed_transition_permit_at( + ¤t, + &transition.request, + directory, + &transition.plan, + ) + .await?; + self.verify_managed_transfer_storage_at(¤t, transition, directory, &permit) + .await?; + let domain = &self.inner.domain; + domain + .run(Box::pin( + directory.require_managed_transfer_retirement_coverage( + domain, + archives, + &transition.plan, + &permit, + ), + )) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(transfer_archive_error)?; + self.commit_with_disruption( + snapshot, + &mutation, + None, + resources::Authority::ManagedTransferChildren, + || Ok(()), + ) + .await + }), + ) + .await + } + + /// Release the prepared and sealed parent holds after authenticating every + /// parked replica named by the final parent configurations. The fleet + /// disruption slot and physical capacity references stay owned. + pub async fn release_managed_transfer_parent_holds( + &self, + transition: &ReopenedManagedCellGroupTransition, + directory: &RangeDirectory, + archives: &CellGroupArchivePool, + observer: &RaftCells, + timeout: Duration, + ) -> Result { + self.release_managed_transfer_parent_holds_at( + None, transition, directory, archives, observer, timeout, + ) + .await + } + + async fn release_managed_transfer_parent_holds_at( + &self, + expected: Option<&FleetSnapshot>, + transition: &ReopenedManagedCellGroupTransition, + directory: &RangeDirectory, + archives: &CellGroupArchivePool, + observer: &RaftCells, + timeout: Duration, + ) -> Result { + validate_request(&transition.request)?; + let deadline = deadline::Deadline::new(self, timeout)?; + let (_, children) = directory + .transition_routes(&transition.plan) + .map_err(catalog_error)?; + let (current, permit) = self + .run( + deadline.remaining()?, + Box::pin(async { + let current = self.inner.current().await?; + if expected.is_some_and(|expected| expected.token != current.token) { + return Err(FleetCatalogError::Conflict); + } + let permit = self + .managed_transition_permit_at( + ¤t, + &transition.request, + directory, + &transition.plan, + ) + .await?; + for route in &children { + let (_, child) = self + .transfer_child_history(¤t, transition, directory, route) + .await?; + if child.state != FleetHistoryState::Live { + return Err(FleetCatalogError::Conflict); + } + } + Ok((current, permit)) + }), + ) + .await?; + let domain = &self.inner.domain; + let (journal, parents) = domain + .run(Box::pin(crate::asyncrt::timeout( + deadline.remaining()?, + Box::pin(directory.managed_retired_sources(&transition.plan, &permit)), + ))) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(|_| FleetCatalogError::Unconfirmed)? + .map_err(catalog_error)?; + if journal.parent_holds_released() { + return Ok(journal); + } + // One authenticated physical cut per node covers all its parent + // slots. Drop each bounded inventory before sampling the next node. + let mut placements = BTreeMap::new(); + for parent in &parents { + for member in &parent.members { + let node = NodeId::new(member.node) + .map_err(|_| FleetCatalogError::Corrupt("retired source node"))?; + placements + .entry(node) + .or_insert_with(Vec::new) + .push((&parent.route, member)); + } + } + for (node, expected) in placements { + let boot = self + .run(deadline.remaining()?, self.host_from(¤t, node)) + .await? + .boot_binding()?; + let sample = observer + .inspect_peer_physical_inventory(node, current.revision(), deadline.remaining()?) + .await + .map_err(FleetCatalogError::Observation)?; + if sample.view.fleet_boot.as_ref() != Some(&boot) + || sample.view.node.identity.node != node + || sample.view.node.identity.disk.bytes() != boot.disk + || sample.view.node.identity.process.bytes() != boot.process + || sample.view.node.storage_instance.bytes() != boot.engine + { + return Err(FleetCatalogError::Unconfirmed); + } + let inventory = sample + .inventory + .ok_or(FleetCatalogError::InventoryIncomplete)?; + for (route, member) in expected { + let row = inventory + .group_history(&route.history) + .ok_or(FleetCatalogError::InventoryIncomplete)?; + if member.disk.is_some_and(|disk| disk != boot.disk) + || row.birth_digest() != route.birth_digest + || row.member().get() != member.member + || !row.parked_retired_source() + { + return Err(FleetCatalogError::Unconfirmed); + } + } + } + self.run(deadline.remaining()?, async { + if self.inner.current().await?.token != current.token { + return Err(FleetCatalogError::Conflict); + } + Ok(()) + }) + .await?; + domain + .run(Box::pin(crate::asyncrt::timeout( + deadline.remaining()?, + Box::pin(directory.release_managed_retired_parent_holds( + domain, + archives, + &transition.plan, + &permit, + )), + ))) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(|_| FleetCatalogError::Unconfirmed)? + .map_err(transfer_archive_error) + } + + /// Retire every historical parent and release the disruption slot in one + /// fleet CAS after directory cleanup. The permanent history rows continue + /// to identify occupied engine slots. Reconcile with the exact request. + pub async fn complete_managed_transfer( + &self, + snapshot: &FleetSnapshot, + request: &FleetRequestIdentity, + transition: &ReopenedManagedCellGroupTransition, + directory: &RangeDirectory, + timeout: Duration, + ) -> Result { + self.complete_managed_transfer_with_parent_release( + snapshot, request, transition, directory, None, timeout, + ) + .await + } + + pub(crate) async fn complete_managed_transfer_with_parent_release( + &self, + snapshot: &FleetSnapshot, + request: &FleetRequestIdentity, + transition: &ReopenedManagedCellGroupTransition, + directory: &RangeDirectory, + parent_release: Option<(&CellGroupArchivePool, &RaftCells)>, + timeout: Duration, + ) -> Result { + self.check_snapshot(snapshot)?; + validate_request(request)?; + validate_request(&transition.request)?; + if request.operation != transition.request.operation + || request.id == transition.request.id + || request.principal != transition.request.principal + || request.expected_revision <= transition.request.expected_revision + { + return Err(FleetCatalogError::Invalid("managed transfer completion")); + } + let (parents, children) = directory + .transition_routes(&transition.plan) + .map_err(catalog_error)?; + let deadline = deadline::Deadline::new(self, timeout)?; + let prepare = Box::pin(async { + let current = self.inner.current().await?; + let record = self + .managed_transition_record(¤t, request.operation) + .await? + .ok_or(FleetCatalogError::Conflict)?; + if record.request != transition.request { + return Err(FleetCatalogError::PayloadConflict); + } + let permit = ManagedTransitionPermit(record); + permit + .check(directory, &transition.plan) + .map_err(catalog_error)?; + let (journal, _) = self + .inner + .domain + .run(Box::pin( + directory.managed_retired_sources(&transition.plan, &permit), + )) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(catalog_error)?; + let retirement = journal + .retired_parents() + .ok_or(FleetCatalogError::Conflict)?; + let release_holds = !journal.parent_holds_released(); + if release_holds && parent_release.is_none() { + return Err(FleetCatalogError::Conflict); + } + let retired = FleetHistoryState::Retired { + operation: request.operation, + retirement, + }; + let mut changes = Vec::with_capacity(parents.len() + 1); + let mut all_live = true; + for route in &parents { + let binding = self.binding(&route.history)?; + let key = binding + .key() + .map_err(|_| FleetCatalogError::Corrupt("retired source binding"))?; + let mut history = self.history_inner(snapshot, &key).await?; + if history.binding != binding + || history.origin != Origin::Managed + || !history.check_route(directory, route) + || (history.state != FleetHistoryState::Live && history.state != retired) + { + return Err(FleetCatalogError::Conflict); + } + all_live &= history.state == FleetHistoryState::Live; + history.state = retired.clone(); + changes.push(FleetChange { + table: FleetTable::Histories, + key, + value: Some(history.encode()?), + }); + } + changes.push(FleetChange { + table: FleetTable::Policy, + key: SLOT_KEY.into(), + value: None, + }); + let mutation = history_mutation(request, changes); + mutation.validate()?; + let bytes = encode(REQUEST_KIND, &mutation, REQUEST_LIMIT)?; + let key = hex::encode(request.id); + if let Some(applied_revision) = self + .inner + .committed(¤t.head, &key, hash(&bytes)) + .await? + { + return Ok(TransferCompletion::Replayed(Box::new(FleetCommit { + snapshot: current, + applied_revision, + }))); + } + if !all_live + || current.token != snapshot.token + || request.expected_revision != snapshot.revision() + { + return Err(FleetCatalogError::Conflict); + } + let Some(Disruption::Transition(active)) = self.active_disruption(¤t).await? + else { + return Err(FleetCatalogError::Conflict); + }; + if active != permit.0 { + return Err(FleetCatalogError::Conflict); + } + for route in &children { + let (_, child) = self + .transfer_child_history(¤t, transition, directory, route) + .await?; + if child.state != FleetHistoryState::Live { + return Err(FleetCatalogError::Conflict); + } + } + let disruption = current.head.disruption.ok_or(FleetCatalogError::Conflict)?; + if release_holds { + // Claim the exact completion payload before releasing any + // remote hold. An interrupted call must use this same ID. + self.inner + .create_exact(&format!("requests/{key}"), &bytes, REQUEST_LIMIT) + .await?; + } + Ok(TransferCompletion::Pending { + mutation, + disruption, + release_holds, + }) + }); + let completion = self.run(deadline.remaining()?, prepare).await?; + let (mutation, disruption, release_holds) = match completion { + TransferCompletion::Replayed(commit) => return Ok(*commit), + TransferCompletion::Pending { + mutation, + disruption, + release_holds, + } => (mutation, disruption, release_holds), + }; + if release_holds { + let (archives, observer) = parent_release.ok_or(FleetCatalogError::Conflict)?; + let journal = self + .release_managed_transfer_parent_holds_at( + Some(snapshot), + transition, + directory, + archives, + observer, + deadline.remaining()?, + ) + .await?; + if !journal.parent_holds_released() { + return Err(FleetCatalogError::Unconfirmed); + } + } + self.run( + deadline.remaining()?, + Box::pin(self.commit_with_disruption( + snapshot, + &mutation, + Some(disruption), + resources::Authority::ManagedTransferCompletion, + || Ok(()), + )), + ) + .await + } + + pub(super) async fn require_retired_history( + &self, + snapshot: &FleetSnapshot, + directory: &RangeDirectory, + route: &GroupRoute, + operation: [u8; 16], + retirement: [u8; 32], + inventory: (&[super::nodes::NodeRecord], bool), + ) -> Result<()> { + let record = self + .managed_transition_record(snapshot, operation) + .await? + .ok_or(FleetCatalogError::InventoryIncomplete)?; + let permit = ManagedTransitionPermit(record); + permit.check_directory(directory).map_err(catalog_error)?; + let operation = ProposalId::new(operation) + .map_err(|_| FleetCatalogError::Corrupt("retired source operation"))?; + let (plan, observed) = directory + .reopen_managed_transition_plan(operation, permit.0.plan, &permit) + .await + .map_err(catalog_error)?; + if let Some(completion) = observed + .as_ref() + .and_then(TransitionJournal::recovery_settlement) + { + return self + .require_recovery_retired_history( + snapshot, directory, &plan, route, completion, retirement, + ) + .await; + } + let (known_nodes, materialized) = inventory; + if !materialized { + return Err(FleetCatalogError::InventoryIncomplete); + } + let (journal, parents) = directory + .managed_retired_sources(&plan, &permit) + .await + .map_err(catalog_error)?; + if !journal.parent_holds_released() + || journal.retired_parents() != Some(retirement) + || !parents.iter().any(|source| { + &source.route == route + && source + .members + .iter() + .all(|member| known_nodes.iter().any(|node| node.node == member.node)) + }) + { + return Err(FleetCatalogError::InventoryIncomplete); + } + Ok(()) + } + + async fn verified_managed_transfer_permit( + &self, + transition: &ReopenedManagedCellGroupTransition, + directory: &RangeDirectory, + ) -> Result { + let current = self.inner.current().await?; + let permit = self + .managed_transition_permit_at( + ¤t, + &transition.request, + directory, + &transition.plan, + ) + .await?; + self.verify_managed_transfer_storage_at(¤t, transition, directory, &permit) + .await?; + if self.inner.current().await?.token != current.token { + return Err(FleetCatalogError::Conflict); + } + Ok(permit) + } + + async fn verify_managed_transfer_storage_at( + &self, + current: &FleetSnapshot, + transition: &ReopenedManagedCellGroupTransition, + directory: &RangeDirectory, + permit: &ManagedTransitionPermit, + ) -> Result<()> { + let (_, children) = directory + .transition_routes(&transition.plan) + .map_err(catalog_error)?; + for route in &children { + self.managed_transfer_install_quorum_at(current, transition, directory, route, permit) + .await? + .ok_or(FleetCatalogError::Unconfirmed)?; + } + Ok(()) + } + + pub(super) async fn managed_recovery_permit_at( + &self, + snapshot: &FleetSnapshot, + directory: &RangeDirectory, + basis: &ManagedRecoveryIntentBasis, + ) -> Result { + let Some(Disruption::Transition(active)) = self.active_disruption(snapshot).await? else { + return Err(FleetCatalogError::Conflict); + }; + if active.operation() != basis.transition || active.plan != basis.plan { + return Err(FleetCatalogError::Conflict); + } + let permit = ManagedTransitionPermit(active); + permit.check_directory(directory).map_err(catalog_error)?; + Ok(permit) + } + + pub(super) async fn active_managed_transition_permit( + &self, + request: &FleetRequestIdentity, + directory: &RangeDirectory, + plan: &DirectoryMutation, + ) -> Result { + let current = self.inner.current().await?; + self.managed_transition_permit_at(¤t, request, directory, plan) + .await + } + + async fn managed_transition_permit_at( + &self, + current: &FleetSnapshot, + request: &FleetRequestIdentity, + directory: &RangeDirectory, + plan: &DirectoryMutation, + ) -> Result { + let Some(Disruption::Transition(active)) = self.active_disruption(current).await? else { + return Err(FleetCatalogError::Conflict); + }; + if active.request != *request { + return Err(FleetCatalogError::PayloadConflict); + } + let permit = ManagedTransitionPermit(active); + permit.check(directory, plan).map_err(catalog_error)?; + Ok(permit) + } + + /// Select the irreversible seal decision for a physically prepared plan. + /// The exact admitted request is retryable. Cancellation competes on the + /// same directory head. This decision alone grants no replica authority. + pub async fn authorize_managed_transition_seal( + &self, + request: &FleetRequestIdentity, + directory: &RangeDirectory, + plan: &DirectoryMutation, + timeout: Duration, + ) -> Result { + validate_request(request)?; + self.run( + timeout, + Box::pin(async { + let permit = self + .active_managed_transition_permit(request, directory, plan) + .await?; + self.inner + .domain + .run(Box::pin( + directory.authorize_managed_transition_seal(plan, &permit), + )) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(catalog_error) + }), + ) + .await + } + + /// Cancel the exact pre-seal plan and only then release its fleet slot. + /// Child identities and their existing capacity debits remain inventoried. + pub async fn cancel_managed_transition( + &self, + snapshot: &FleetSnapshot, + request: &FleetRequestIdentity, + directory: &RangeDirectory, + plan: &DirectoryMutation, + timeout: Duration, + ) -> Result { + self.check_snapshot(snapshot)?; + validate_request(request)?; + self.run( + timeout, + Box::pin(async { + let current = self.inner.current().await?; + let record = self + .managed_transition_record(¤t, request.operation) + .await? + .ok_or(FleetCatalogError::Conflict)?; + if request.id == record.request.id + || request.principal != record.request.principal + || request.expected_revision <= record.request.expected_revision + { + return Err(FleetCatalogError::Invalid( + "managed transition cancellation", + )); + } + let permit = ManagedTransitionPermit(record); + permit.check(directory, plan).map_err(catalog_error)?; + let mutation = history_mutation( + request, + vec![FleetChange { + table: FleetTable::Policy, + key: SLOT_KEY.into(), + value: None, + }], + ); + mutation.validate()?; + let digest = hash(&encode(REQUEST_KIND, &mutation, REQUEST_LIMIT)?); + let key = hex::encode(request.id); + if let Some(applied_revision) = + self.inner.committed(¤t.head, &key, digest).await? + { + return Ok(FleetCommit { + snapshot: current, + applied_revision, + }); + } + if current.token != snapshot.token + || request.expected_revision != snapshot.revision() + { + return Err(FleetCatalogError::Conflict); + } + let Some(Disruption::Transition(active)) = self.active_disruption(snapshot).await? + else { + return Err(FleetCatalogError::Conflict); + }; + if active != permit.0 { + return Err(FleetCatalogError::Conflict); + } + let release = snapshot + .head + .disruption + .ok_or(FleetCatalogError::Conflict)?; + self.inner + .domain + .run(Box::pin(directory.cancel_managed_transition(plan, &permit))) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(catalog_error)?; + self.commit_with_disruption( + snapshot, + &mutation, + Some(release), + resources::Authority::ManagedTransition, + || Ok(()), + ) + .await + }), + ) + .await + } + + async fn managed_transition_record( + &self, + snapshot: &FleetSnapshot, + operation: [u8; 16], + ) -> Result> { + let Some(bytes) = self + .inner + .lookup( + snapshot.head.tables[FleetTable::Operations.slot()], + &operation_key(operation), + ) + .await? + else { + return Ok(None); + }; + let record: Record = decode(KIND, &bytes, RECORD_LIMIT)?; + record.validate(self)?; + if record.operation() != operation { + return Err(FleetCatalogError::Corrupt("managed transition index")); + } + self.confirm_managed_transition(snapshot, &record).await?; + Ok(Some(record)) + } + + pub(super) async fn confirm_managed_transition( + &self, + snapshot: &FleetSnapshot, + record: &Record, + ) -> Result<()> { + let digest = hash(&encode(REQUEST_KIND, &record.mutation()?, REQUEST_LIMIT)?); + if self + .inner + .committed(&snapshot.head, &hex::encode(record.request.id), digest) + .await? + .is_none() + { + return Err(FleetCatalogError::Corrupt("managed transition journal")); + } + Ok(()) + } + + pub(super) async fn check_managed_transition_mutation( + &self, + snapshot: &FleetSnapshot, + mutation: &FleetMutation, + authority: resources::Authority, + ) -> Result<()> { + if authority != resources::Authority::ManagedRecoveryReservation + && mutation.changes.iter().any(|change| { + change.table == FleetTable::Policy + && change + .key + .starts_with(super::managed_recovery_reservation::PREFIX) + }) + { + return Err(FleetCatalogError::Invalid( + "managed recovery reservation authority", + )); + } + if authority != resources::Authority::ManagedRecoverySettlement + && mutation.changes.iter().any(|change| { + change.table == FleetTable::Policy + && change + .key + .starts_with(super::managed_recovery_settlement::PREFIX) + }) + { + return Err(FleetCatalogError::Invalid( + "managed recovery settlement authority", + )); + } + if authority != resources::Authority::ManagedRecoveryRetirement + && mutation.changes.iter().any(|change| { + change.table == FleetTable::Policy + && change + .key + .starts_with(super::managed_recovery_retirement::PREFIX) + }) + { + return Err(FleetCatalogError::Invalid( + "managed recovery retirement authority", + )); + } + for change in mutation + .changes + .iter() + .filter(|change| change.table == FleetTable::Histories) + { + if !matches!( + authority, + resources::Authority::ManagedTransferCompletion + | resources::Authority::ManagedRecoveryRetirement + ) && change + .value + .as_deref() + .and_then(|bytes| FleetHistory::decode(bytes).ok()) + .is_some_and(|history| matches!(history.state, FleetHistoryState::Retired { .. })) + { + return Err(FleetCatalogError::Invalid( + "managed history retirement authority", + )); + } + if self + .inner + .lookup( + snapshot.head.tables[FleetTable::Histories.slot()], + &change.key, + ) + .await? + .as_deref() + .and_then(|bytes| FleetHistory::decode(bytes).ok()) + .is_some_and(|history| matches!(history.state, FleetHistoryState::Retired { .. })) + { + return Err(FleetCatalogError::Conflict); + } + } + if authority != resources::Authority::ManagedTransition + && mutation.changes.iter().any(|change| { + change.table == FleetTable::Operations && change.key.starts_with(OPERATION_PREFIX) + }) + { + return Err(FleetCatalogError::Invalid( + "managed transition operation authority", + )); + } + if snapshot.head.disruption.is_some() + && mutation.changes.iter().any(|change| match change.table { + FleetTable::Histories => !matches!( + authority, + resources::Authority::ManagedTransferChildren + | resources::Authority::ManagedTransferCompletion + | resources::Authority::ManagedRecoveryReservation + | resources::Authority::ManagedRecoverySettlement + | resources::Authority::ManagedRecoveryRetirement + ), + FleetTable::Admissions => true, + FleetTable::Policy => { + !matches!( + authority, + resources::Authority::ManagedTransition + | resources::Authority::ManagedTransferCompletion + | resources::Authority::ManagedRecoveryRetirement + ) && change.key == SLOT_KEY + } + _ => false, + }) + { + if let Some(Disruption::Transition(record)) = self.active_disruption(snapshot).await? { + return Err(FleetCatalogError::DisruptionInProgress { + operation: record.operation(), + }); + } + } + Ok(()) + } +} + +pub(crate) struct ManagedTransferInstallAuthority { + pub(crate) transition: ManagedTransitionPermit, + pub(crate) group: FleetGroupBinding, + pub(crate) boot: BootBinding, +} + +impl FleetCatalog { + async fn reserved_transfer_child( + &self, + current: &FleetSnapshot, + transition: &ReopenedManagedCellGroupTransition, + directory: &RangeDirectory, + route: &GroupRoute, + ) -> Result { + let (_, history) = self + .transfer_child_history(current, transition, directory, route) + .await?; + if history.state != FleetHistoryState::Reserved { + return Err(FleetCatalogError::Conflict); + } + Ok(history) + } + + async fn transfer_child_history( + &self, + current: &FleetSnapshot, + transition: &ReopenedManagedCellGroupTransition, + directory: &RangeDirectory, + route: &GroupRoute, + ) -> Result<(String, FleetHistory)> { + let key = self + .binding(&route.history)? + .key() + .map_err(|_| FleetCatalogError::Invalid("transfer child binding"))?; + let history = self.history_inner(current, &key).await?; + if history.origin != Origin::Managed + || !matches!( + history.state, + FleetHistoryState::Reserved | FleetHistoryState::Live + ) + || history.operation != transition.request.operation + || !history.check_route(directory, route) + { + return Err(FleetCatalogError::Conflict); + } + Ok((key, history)) + } + + /// Observe a majority of physically installed voters of the exact reserved + /// managed child. Receipt storage identities must match permanent fleet + /// claims. An installation receipt grants no live boot or quorum authority. + pub async fn observe_managed_transfer_install_quorum( + &self, + transition: &ReopenedManagedCellGroupTransition, + directory: &RangeDirectory, + route: &GroupRoute, + timeout: Duration, + ) -> Result> { + validate_request(&transition.request)?; + self.run( + timeout, + Box::pin(async { + let current = self.inner.current().await?; + let permit = self + .managed_transition_permit_at( + ¤t, + &transition.request, + directory, + &transition.plan, + ) + .await?; + let proof = self + .managed_transfer_install_quorum_at( + ¤t, transition, directory, route, &permit, + ) + .await?; + if self.inner.current().await?.token != current.token { + return Err(FleetCatalogError::Conflict); + } + Ok(proof) + }), + ) + .await + } + + async fn managed_transfer_install_quorum_at( + &self, + current: &FleetSnapshot, + transition: &ReopenedManagedCellGroupTransition, + directory: &RangeDirectory, + route: &GroupRoute, + permit: &ManagedTransitionPermit, + ) -> Result> { + let child = self + .reserved_transfer_child(current, transition, directory, route) + .await?; + let proof = self + .inner + .domain + .run(Box::pin(directory.observe_managed_transfer_install_quorum( + &transition.plan, + permit, + route.history, + ))) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(catalog_error)?; + if let Some(proof) = &proof { + for voter in &proof.voters { + let host = self.host_from(current, voter.node).await?; + if host.disk() != voter.disk + || host.storage_instance() != Some(voter.engine) + || !child.voters.iter().any(|member| { + member.node == voter.node.bytes() && member.member == voter.member.get() + }) + { + return Err(FleetCatalogError::Conflict); + } + } + } + Ok(proof) + } + + pub(crate) async fn resume_managed_transfer_child_quorum( + &self, + transition: &ReopenedManagedCellGroupTransition, + directory: &RangeDirectory, + history: GroupHistory, + proof: Option, + timeout: Duration, + ) -> Result> { + validate_request(&transition.request)?; + let (_, children) = directory + .transition_routes(&transition.plan) + .map_err(catalog_error)?; + let route = children + .iter() + .find(|route| route.history == history) + .ok_or(FleetCatalogError::Conflict)?; + self.run( + timeout, + Box::pin(async { + let current = self.inner.current().await?; + let permit = self + .managed_transition_permit_at( + ¤t, + &transition.request, + directory, + &transition.plan, + ) + .await?; + self.managed_transfer_install_quorum_at( + ¤t, transition, directory, route, &permit, + ) + .await? + .ok_or(FleetCatalogError::Unconfirmed)?; + let all_current = match self + .verify_managed_transfer_storage_at(¤t, transition, directory, &permit) + .await + { + Ok(()) => true, + Err(FleetCatalogError::Unconfirmed) => false, + Err(error) => return Err(error), + }; + if self.inner.current().await?.token != current.token { + return Err(FleetCatalogError::Conflict); + } + self.inner + .domain + .run(Box::pin(directory.resume_managed_transfer_child_quorum( + &transition.plan, + history, + proof, + &permit, + all_current, + ))) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(catalog_error) + }), + ) + .await + } + + /// Select the exact ordered set of managed child fresh-term witnesses. + /// The fleet reservation and physical storage claims must still match. + /// The resulting Installed decision leaves routing and serving fenced. + pub async fn record_managed_transfer_child_quorums( + &self, + transition: &ReopenedManagedCellGroupTransition, + directory: &RangeDirectory, + proofs: Vec, + timeout: Duration, + ) -> Result { + validate_request(&transition.request)?; + let (_, children) = directory + .transition_routes(&transition.plan) + .map_err(catalog_error)?; + if proofs.len() != children.len() { + return Err(FleetCatalogError::Invalid( + "incomplete transfer child quorum set", + )); + } + self.run( + timeout, + Box::pin(async { + let permit = self + .verified_managed_transfer_permit(transition, directory) + .await?; + self.inner + .domain + .run(Box::pin(directory.record_managed_transfer_child_quorums( + &transition.plan, + proofs, + &permit, + ))) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(catalog_error) + }), + ) + .await + } +} + +#[derive(Clone, Copy, PartialEq, Eq)] +pub(crate) enum TransferVoterAdmission { + Install, + Retained, +} + +impl FleetHostAuthorization { + pub(crate) async fn managed_transfer_install_authority( + &self, + transition: &ReopenedManagedCellGroupTransition, + directory: &RangeDirectory, + route: &GroupRoute, + timeout: Duration, + ) -> Result { + self.managed_transfer_voter_authority( + transition, + directory, + route, + TransferVoterAdmission::Install, + timeout, + ) + .await + } + + pub(crate) async fn managed_transfer_voter_authority( + &self, + transition: &ReopenedManagedCellGroupTransition, + directory: &RangeDirectory, + route: &GroupRoute, + admission: TransferVoterAdmission, + timeout: Duration, + ) -> Result { + validate_request(&transition.request)?; + let catalog = &self.catalog; + catalog + .run( + timeout, + Box::pin(async { + let current = catalog.inner.current().await?; + let host = catalog.host_from_record(¤t, self.node()).await?; + self.require_running_successor(&host)?; + let boot = self.boot_binding()?; + let permit = catalog + .managed_transition_permit_at( + ¤t, + &transition.request, + directory, + &transition.plan, + ) + .await?; + let history = catalog + .reserved_transfer_child(¤t, transition, directory, route) + .await?; + if !history.voters.iter().any(|voter| voter.node == boot.node) { + return Err(FleetCatalogError::Conflict); + } + if admission == TransferVoterAdmission::Install { + let capacity = catalog.capacity_from(¤t, self.node()).await?; + if !capacity.receive_replicas() || capacity.attestation().boot != boot { + return Err(FleetCatalogError::Conflict); + } + } + if catalog.inner.current().await?.token != current.token { + return Err(FleetCatalogError::Conflict); + } + Ok(ManagedTransferInstallAuthority { + transition: permit, + group: history.binding, + boot, + }) + }), + ) + .await + } + + pub(crate) async fn managed_source_authority( + &self, + transition: &ReopenedManagedCellGroupTransition, + directory: &RangeDirectory, + source: GroupHistory, + timeout: Duration, + ) -> Result<(ManagedTransitionPermit, FleetGroupAuthorization)> { + validate_request(&transition.request)?; + let deadline = deadline::Deadline::new(&self.catalog, timeout)?; + let permit = self + .catalog + .run( + deadline.remaining()?, + Box::pin(self.catalog.active_managed_transition_permit( + &transition.request, + directory, + &transition.plan, + )), + ) + .await?; + let group = self + .group_authorization(&source, deadline.remaining()?) + .await?; + Ok((permit, group)) + } +} diff --git a/crates/cellar/durability/raft/fleet_catalog/mod.rs b/crates/cellar/durability/raft/fleet_catalog/mod.rs index 5cef81800..4aec61d06 100644 --- a/crates/cellar/durability/raft/fleet_catalog/mod.rs +++ b/crates/cellar/durability/raft/fleet_catalog/mod.rs @@ -1,7 +1,9 @@ //! Bounded CAS coordination. These records never grant Raft membership or votes. //! A retained mutation journal resolves ambiguous writes after later revisions. +mod archive_replay; mod boots; +pub use archive_replay::{FleetGroupArchiveReplayOptions, FleetGroupArchiveVerifier}; mod capacity; mod deadline; mod disruption; @@ -15,6 +17,43 @@ mod imports; mod index; mod isolation; mod maintenance; +mod managed_park; +pub(crate) use managed_park::{ + ManagedParkOperation, ManagedParkPermit, ManagedParkPhase, ManagedParkSettlement, +}; +mod managed_transition_admission; +pub(crate) use managed_transition_admission::TransitionOperatorStep; +mod managed_recovery_fence; +pub(crate) use managed_recovery_fence::PreparedManagedRecoveryFence; +pub use managed_recovery_fence::{ + FleetFencedTransitionRecovery, FleetTransitionRecoveryFenceRequest, +}; +mod managed_recovery_intent; +pub use managed_recovery_intent::{FleetTransitionRecoveryBarrier, FleetTransitionRecoveryIntent}; +pub(crate) use managed_recovery_intent::{ManagedRecoveryIntentBasis, ManagedRecoveryIntentInput}; +mod managed_recovery_inventory; +mod managed_recovery_quorum; +pub(crate) use managed_recovery_quorum::FleetPublishedRecoveryVoter; +pub use managed_recovery_quorum::{FleetRecoveryAttachRequest, FleetRecoverySeedQuorums}; +mod managed_recovery_replacement; +mod managed_recovery_reservation; +mod managed_recovery_retirement; +pub use managed_recovery_retirement::FleetRecoveryRetirement; +mod managed_recovery_settlement; +pub use managed_recovery_replacement::FleetTransitionRecoveryReplacement; +pub use managed_recovery_reservation::FleetRecoveryRangeReservation; +pub(crate) use managed_recovery_reservation::{ + ManagedRecoverySeedAuthority, ManagedRecoverySeedSelection, +}; +pub(crate) use managed_recovery_settlement::FleetManagedRecoveryCompletion; +pub use managed_recovery_settlement::FleetRecoverySettlement; +mod managed_transition_replay; +pub use managed_recovery_inventory::FleetRecoveryVoterLineage; +mod managed_transitions; +pub use managed_transitions::ReopenedManagedCellGroupTransition; +pub(crate) use managed_transitions::{ + ManagedTransitionPermit, ManagedTransitionRecoveryObservation, TransferVoterAdmission, +}; mod namespaces; mod nodes; mod permits; @@ -64,7 +103,10 @@ pub use maintenance::{ pub(crate) use maintenance::{ RestartActionContext, RestartLeadershipCommand, ResumeLeadershipCommand, }; -pub use namespaces::FleetInventory; +pub use namespaces::{ + FleetInventory, RaftManagedArchiveProvisioning, RaftManagedFirstGroup, + RaftManagedFirstGroupRequest, +}; pub use nodes::{ FleetHostAuthorization, FleetNodeEligibility, FleetNodeSpec, FleetNodeState, FleetRequestIdentity, @@ -148,11 +190,29 @@ impl From for FleetCatalogError { super::catalog::CatalogError::Corrupt(reason) => Self::Corrupt(reason), super::catalog::CatalogError::Limit => Self::Limit, super::catalog::CatalogError::Storage(error) => Self::Storage(error.into()), + super::catalog::CatalogError::FleetAuthority(error) => *error, _ => Self::Corrupt("index storage contract"), } } } +impl From for FleetCatalogError { + fn from(error: crate::raft::cell_groups::CellGroupArchiveError) -> Self { + use crate::raft::cell_groups::CellGroupArchiveError; + match error { + CellGroupArchiveError::Invalid(reason) => Self::Invalid(reason), + CellGroupArchiveError::Limit => Self::Limit, + CellGroupArchiveError::Busy => Self::Overloaded, + CellGroupArchiveError::Conflict => Self::Conflict, + CellGroupArchiveError::Timeout + | CellGroupArchiveError::Expired + | CellGroupArchiveError::Unconfirmed => Self::Unconfirmed, + CellGroupArchiveError::Stopped => Self::Closed, + error => Self::Storage(error.into()), + } + } +} + impl fmt::Display for FleetCatalogError { fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { write!(f, "fleet catalog: {self:?}") @@ -355,36 +415,46 @@ impl FleetCatalog { Ok(()) } - async fn run(&self, timeout: Duration, work: impl Future>) -> Result { - if timeout.is_zero() || timeout > Duration::from_secs(60) { - return Err(FleetCatalogError::Invalid("timeout")); + fn run<'a, T: 'a>( + &'a self, + timeout: Duration, + work: impl Future> + 'a, + ) -> impl Future> + 'a { + // Retain admitted work behind one owned pointer before constructing the + // catalog turn. Large replay futures must not inflate the turn and each + // caller's poll frame on the way into the object-store HTTP client. + let work = Box::pin(work); + async move { + if timeout.is_zero() || timeout > Duration::from_secs(60) { + return Err(FleetCatalogError::Invalid("timeout")); + } + let _turn = self + .inner + .turn + .try_lock() + .map_err(|_| FleetCatalogError::Overloaded)?; + let deadline = self + .inner + .domain + .mono_ms() + .map_err(|_| FleetCatalogError::Closed)? + .checked_add(timeout.as_millis() as u64) + .ok_or(FleetCatalogError::Limit)?; + let timer = self + .inner + .domain + .sleep_until(deadline) + .map_err(|_| FleetCatalogError::Closed)?; + let result = crate::asyncrt::select! { + _=timer=>Err(FleetCatalogError::Unconfirmed), + result=work=>result, + }; + self.inner + .domain + .mono_ms() + .map_err(|_| FleetCatalogError::Closed)?; + result } - let _turn = self - .inner - .turn - .try_lock() - .map_err(|_| FleetCatalogError::Overloaded)?; - let deadline = self - .inner - .domain - .mono_ms() - .map_err(|_| FleetCatalogError::Closed)? - .checked_add(timeout.as_millis() as u64) - .ok_or(FleetCatalogError::Limit)?; - let timer = self - .inner - .domain - .sleep_until(deadline) - .map_err(|_| FleetCatalogError::Closed)?; - let result = crate::asyncrt::select! { - _=timer=>Err(FleetCatalogError::Unconfirmed), - result=work=>result, - }; - self.inner - .domain - .mono_ms() - .map_err(|_| FleetCatalogError::Closed)?; - result } async fn commit_inner( @@ -392,7 +462,7 @@ impl FleetCatalog { snapshot: &FleetSnapshot, mutation: &FleetMutation, ) -> Result { - self.commit_checked(snapshot, mutation, || Ok(())).await + Box::pin(self.commit_checked(snapshot, mutation, || Ok(()))).await } async fn commit_checked( @@ -401,8 +471,16 @@ impl FleetCatalog { mutation: &FleetMutation, check: impl FnOnce() -> Result<()>, ) -> Result { - self.commit_with_disruption(snapshot, mutation, None, resources::Authority::None, check) - .await + // The full root transaction contains nested S3 and index futures. + // Keep them off the startup polling frame when a voter binds a boot. + Box::pin(self.commit_with_disruption( + snapshot, + mutation, + None, + resources::Authority::None, + check, + )) + .await } // Only owned proof paths supply terminal release or movement authority. @@ -428,6 +506,9 @@ impl FleetCatalog { if current.token != snapshot.token || mutation.expected_revision != snapshot.revision() { return Err(FleetCatalogError::Conflict); } + Box::pin(self.check_managed_transition_mutation(snapshot, mutation, movement)).await?; + self.check_park_mutation(snapshot, mutation, movement) + .await?; self.check_failure_recovery_mutation(snapshot, mutation, movement) .await?; self.check_isolation_mutation(snapshot, mutation, movement) @@ -464,6 +545,7 @@ impl FleetCatalog { mutation, next.disruption, change.value.as_deref(), + movement, )?) }; } diff --git a/crates/cellar/durability/raft/fleet_catalog/namespaces/README.md b/crates/cellar/durability/raft/fleet_catalog/namespaces/README.md index e1cbf623d..5b8074c9a 100644 --- a/crates/cellar/durability/raft/fleet_catalog/namespaces/README.md +++ b/crates/cellar/durability/raft/fleet_catalog/namespaces/README.md @@ -20,6 +20,52 @@ and journal determine its original applied revision. A changed request payload conflicts. An unadmitted stale candidate cannot overwrite a newer root. A different request cannot replace an earlier declaration. +## Fresh managed standby + +`FleetCatalog::provision_managed_namespace` checks the compiled class and +binds the immutable namespace to this fleet. It pins the canonical declaration +request before namespace writes, publishes the standby directory and commits +the declaration. The returned registration selects a namespace for managed +startup. Capacity admission and an authorized birth select the first group. + +Persist the request identity and catalog settings before submitting. Replay +the same request after an uncertain reply. If an unrelated fleet CAS advances +the root before declaration, a new request ID at the current revision may use +the same logical operation and configuration. The operation identifies the +original standby publication. A confirmed replay verifies existing metadata, +including the historical standby receipt if a group was selected later. + +## First full-range group + +Use `observe_managed_standby` to read the exact empty root. Persist a +`RaftManagedFirstGroupRequest` with that digest, the original fleet reservation +identity, the range operation and the full-range RF3 birth. The reservation API +rechecks current capacity and placement against the fleet root. Reopening uses +the same request, including the original range operation. + +`publish_managed_first_group` first resolves the immutable birth decision, +then selects the range and confirms the fleet history. The directory cannot +select a birth whose cancellation won. If the range CAS loses, the exact birth +remains inventoried and cannot start a replica outside the selected map. A +confirmed range with missing birth metadata refuses replay. Persist and retry +the exact confirmation request after an uncertain reply. A confirmed revision +conflict can use a new confirmation request at the new fleet revision with the +same operation and prepared reservation. Activation requires the existing +managed host boot and local member claim. + +## Managed group archives + +After fleet confirmation, use `FleetCatalog::open_managed_archive_provisioning` +with the same declared namespace. `prepare_archive` resolves the current live +fleet history and selected birth. Persist and sync the returned +`encoded_intent()` before committing the one-use initializer. + +After restart or an uncertain commit, reopen the namespace and call +`verify_archive` with the saved bytes. It verifies that exact binding, identity +and counters without creating metadata. A different intent conflicts. Missing +or incomplete metadata refuses and requires archive authority recovery. The +archive result does not admit voters or establish coverage of a command. + ## Compiled host and birth admission Managed startup checks every actual prepared namespace and compiled class against diff --git a/crates/cellar/durability/raft/fleet_catalog/namespaces/archive.rs b/crates/cellar/durability/raft/fleet_catalog/namespaces/archive.rs new file mode 100644 index 000000000..6d18efd5c --- /dev/null +++ b/crates/cellar/durability/raft/fleet_catalog/namespaces/archive.rs @@ -0,0 +1,414 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Archive setup over the declared namespace and its selected live histories. +use super::super::deadline::Deadline; +use super::*; +use crate::application::consensus::RaftNamespaceRegistration; +use crate::raft::{ + catalog::{Catalog, CellGroupArchiveBindingError}, + cell_groups::{ + CellGroupArchiveLimits, CellGroupArchivePool, CellGroupBirth, DirectoryLimits, + RangeDirectory, + }, + group_host::RaftGroupArchiveInitialization, +}; +use cellar_logic::cell_group::{CellRange, GroupHistory}; + +const FIRST_GROUP_ARCHIVE_KIND: u8 = 103; +const INTENT_LIMIT: usize = 4096; + +#[derive(PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct FirstGroupArchiveIntent { + request: FleetRequestIdentity, + publication: [u8; 32], + binding: String, +} + +impl FirstGroupArchiveIntent { + fn binding(&self) -> Result> { + hex::decode(&self.binding) + .map_err(|_| FleetCatalogError::Corrupt("first group archive intent")) + } + + fn mutation(&self, key: &str) -> Result { + Ok(history_mutation( + &self.request, + vec![FleetChange { + table: FleetTable::Policy, + key: key.into(), + value: Some(encode(FIRST_GROUP_ARCHIVE_KIND, self, INTENT_LIMIT)?), + }], + )) + } +} + +/// Operator archive setup for one exact managed namespace. Every operation +/// checks the current fleet history and selected birth. This handle grants +/// neither local replica activation nor archive coverage. +pub struct RaftManagedArchiveProvisioning { + fleet: FleetCatalog, + directory: RangeDirectory, + archives: CellGroupArchivePool, +} + +impl FleetCatalog { + /// Root one archive intent for the exact published first group before + /// entering the one-use initializer. The active owner can reconcile a + /// lost fleet confirmation. A later replay is verification only. + pub(crate) async fn initialize_managed_first_group_archive( + &self, + group: &RaftManagedFirstGroup, + request: &FleetRequestIdentity, + archives: &CellGroupArchivePool, + timeout: Duration, + ) -> Result<()> { + validate_request(request)?; + if request.id == group.reservation.id + || request.operation != group.reservation.operation + || request.principal != group.reservation.principal + || request.expected_revision <= group.reservation.expected_revision + || self.birth_owner()? != *group.authorization.owner() + { + return Err(FleetCatalogError::Invalid("first group archive request")); + } + let provisioning = RaftManagedArchiveProvisioning { + fleet: self.clone(), + directory: group.directory.clone(), + archives: archives.clone(), + }; + let history = *group.authorization.scope(); + let key = self.first_group_archive_key(history)?; + let publication = group.mutation.digest(); + self.run(timeout, async { + let snapshot = self.inner.current().await?; + let (birth, _) = provisioning.selected_birth_at(&snapshot, history).await?; + if let Some(intent) = self.first_group_archive_intent(&snapshot, &key).await? { + if intent.request != *request || intent.publication != publication { + return Err(FleetCatalogError::PayloadConflict); + } + return provisioning + .verify_birth_archive(&birth, &intent.binding()?) + .await; + } + if snapshot.revision() != request.expected_revision { + return Err(FleetCatalogError::Conflict); + } + if let Some((prior, _)) = self + .inner + .read::( + &format!("requests/{}", hex::encode(request.id)), + REQUEST_KIND, + REQUEST_LIMIT, + ) + .await? + { + // A previous owner may still be resolving its CAS. Its one-use + // initializer cannot be reconstructed from persisted bytes. + let [change] = prior.changes.as_slice() else { + return Err(FleetCatalogError::PayloadConflict); + }; + let intent: FirstGroupArchiveIntent = decode( + FIRST_GROUP_ARCHIVE_KIND, + change + .value + .as_deref() + .ok_or(FleetCatalogError::PayloadConflict)?, + INTENT_LIMIT, + ) + .map_err(|_| FleetCatalogError::PayloadConflict)?; + return Err( + if intent.request == *request + && intent.publication == publication + && prior == intent.mutation(&key)? + { + FleetCatalogError::Unconfirmed + } else { + FleetCatalogError::PayloadConflict + }, + ); + } + let domain = &self.inner.domain; + let prepared = provisioning + .directory + .prepare_group_archive_binding(domain, archives, &birth) + .await + .map_err(binding_error)?; + let intent = FirstGroupArchiveIntent { + request: request.clone(), + publication, + binding: hex::encode(prepared.encode()), + }; + let mutation = intent.mutation(&key)?; + match self.commit_inner(&snapshot, &mutation).await { + Ok(_) => (), + Err(error @ (FleetCatalogError::Storage(_) | FleetCatalogError::Unconfirmed)) => { + // The fleet CAS may have committed despite the lost reply + // or confirmation read. Only this invocation still owns + // the fresh initializer, and it needs the exact rooted + // journal entry before creating any archive metadata. + let observed = self.inner.current().await?; + if self + .first_group_archive_intent(&observed, &key) + .await? + .as_ref() + != Some(&intent) + { + return Err(error); + } + } + Err(error) => return Err(error), + } + prepared.commit(domain).await.map_err(binding_error)?; + Ok(()) + }) + .await + } + + fn first_group_archive_key(&self, history: GroupHistory) -> Result { + self.binding(&history)? + .key() + .map(|key| format!("first-group-archives/{key}")) + .map_err(|_| FleetCatalogError::Invalid("archive history binding")) + } + + async fn first_group_archive_intent( + &self, + snapshot: &FleetSnapshot, + key: &str, + ) -> Result> { + let Some(bytes) = self + .inner + .lookup(snapshot.head.tables[FleetTable::Policy.slot()], key) + .await? + else { + return Ok(None); + }; + let intent: FirstGroupArchiveIntent = + decode(FIRST_GROUP_ARCHIVE_KIND, &bytes, INTENT_LIMIT)?; + let mutation = intent.mutation(key)?; + let digest = hash(&encode(REQUEST_KIND, &mutation, REQUEST_LIMIT)?); + if self + .inner + .committed(&snapshot.head, &hex::encode(mutation.id), digest) + .await? + .is_none() + { + return Err(FleetCatalogError::Corrupt("first group archive journal")); + } + Ok(Some(intent)) + } + + /// Open the existing, declared namespace for explicit archive setup. + pub async fn open_managed_archive_provisioning( + &self, + catalog: &Catalog, + registration: &RaftNamespaceRegistration, + directory_limits: DirectoryLimits, + archive_limits: CellGroupArchiveLimits, + timeout: Duration, + ) -> Result { + let archives = CellGroupArchivePool::new(archive_limits)?; + let deadline = Deadline::new(self, timeout)?; + let directory = self + .open_managed_directory(catalog, registration, directory_limits, &deadline) + .await?; + Ok(RaftManagedArchiveProvisioning { + fleet: self.clone(), + directory, + archives, + }) + } +} + +impl FleetHostAuthorization { + pub(crate) async fn observe_first_group_archive( + &self, + directory: RangeDirectory, + archives: CellGroupArchivePool, + history: Option, + timeout: Duration, + ) -> Result> { + self.check_namespace(directory.namespace_config())?; + RaftManagedArchiveProvisioning { + fleet: self.catalog.clone(), + directory, + archives, + } + .verified_first_group(history, timeout) + .await + } +} + +impl RaftManagedArchiveProvisioning { + /// Prepare the one-use initializer for a confirmed managed group. Persist + /// and sync the encoded intent before committing it. + pub async fn prepare_archive( + &self, + history: GroupHistory, + timeout: Duration, + ) -> Result { + self.fleet + .run(timeout, async { + let domain = &self.fleet.inner.domain; + let birth = self.selected_birth(history).await?; + let prepared = self + .directory + .prepare_group_archive_binding(domain, &self.archives, &birth) + .await + .map_err(binding_error)?; + Ok(RaftGroupArchiveInitialization { prepared }) + }) + .await + } + + /// Verify the exact persisted intent after a restart or uncertain commit. + /// Missing binding, identity or counters refuse without creating metadata. + pub async fn verify_archive( + &self, + history: GroupHistory, + intent: &[u8], + timeout: Duration, + ) -> Result<()> { + self.fleet + .run(timeout, async { + let birth = self.selected_birth(history).await?; + self.verify_birth_archive(&birth, intent).await + }) + .await + } + + /// Verify the operator-owned archive against its rooted intent. A missing + /// intent or incomplete initialization refuses startup without allocating. + pub async fn verify_first_group_archive( + &self, + history: GroupHistory, + timeout: Duration, + ) -> Result<()> { + self.verified_first_group(Some(history), timeout) + .await? + .ok_or(FleetCatalogError::Unconfirmed)?; + Ok(()) + } + + async fn verified_first_group( + &self, + history: Option, + timeout: Duration, + ) -> Result> { + self.fleet + .run(timeout, async { + let history = match history { + Some(history) => history, + None => { + let Some(history) = self.selected_full_range_history().await? else { + return Ok(None); + }; + history + } + }; + let key = self.fleet.first_group_archive_key(history)?; + let snapshot = self.fleet.inner.current().await?; + let Some(intent) = self + .fleet + .first_group_archive_intent(&snapshot, &key) + .await? + else { + return Ok(None); + }; + let (birth, authority) = self.selected_birth_at(&snapshot, history).await?; + let verified = self + .directory + .observe_group_archive_binding( + &self.fleet.inner.domain, + &self.archives, + &birth, + &intent.binding()?, + ) + .await + .map_err(binding_error)?; + Ok(verified.then_some(authority)) + }) + .await + } + + async fn selected_full_range_history(&self) -> Result> { + let directory = &self.directory; + let (view, page) = self + .fleet + .inner + .domain + .run(async { + let view = directory.observe().await?; + let page = directory.group_page(&view, None, 1).await?; + Ok::<_, crate::raft::catalog::CatalogError>((view, page)) + }) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(catalog_error)?; + match page.routes.as_slice() { + [] if view.groups() == 0 => Ok(None), + [route] if view.groups() == 1 && route.range == CellRange::all() => { + Ok(Some(route.history)) + } + _ => Err(FleetCatalogError::Conflict), + } + } + + async fn verify_birth_archive(&self, birth: &CellGroupBirth, intent: &[u8]) -> Result<()> { + self.directory + .verify_group_archive_binding(&self.fleet.inner.domain, &self.archives, birth, intent) + .await + .map_err(|error| match error { + CellGroupArchiveBindingError::Archive( + crate::raft::cell_groups::CellGroupArchiveError::Invalid(reason), + ) => FleetCatalogError::Corrupt(reason), + error => binding_error(error), + }) + } + + async fn selected_birth(&self, history: GroupHistory) -> Result { + let snapshot = self.fleet.inner.current().await?; + self.selected_birth_at(&snapshot, history) + .await + .map(|(birth, _)| birth) + } + + async fn selected_birth_at( + &self, + snapshot: &FleetSnapshot, + history: GroupHistory, + ) -> Result<(CellGroupBirth, FleetGroupAuthorization)> { + let binding = self.fleet.binding(&history)?; + let key = binding + .key() + .map_err(|_| FleetCatalogError::Invalid("archive history binding"))?; + let authority = self + .fleet + .authorize_live_group(snapshot, binding, &key) + .await?; + let owner = self.fleet.birth_owner()?; + let birth = self + .fleet + .inner + .domain + .run( + self.directory + .open_managed_selected_history(history, &owner), + ) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(catalog_error)?; + if !authority.check_birth(&self.directory, &birth) { + return Err(FleetCatalogError::Conflict); + } + Ok((birth, authority)) + } +} + +fn binding_error(error: CellGroupArchiveBindingError) -> FleetCatalogError { + match error { + CellGroupArchiveBindingError::Catalog(error) => catalog_error(error), + CellGroupArchiveBindingError::Archive(error) => error.into(), + } +} diff --git a/crates/cellar/durability/raft/fleet_catalog/namespaces/first_group.rs b/crates/cellar/durability/raft/fleet_catalog/namespaces/first_group.rs new file mode 100644 index 000000000..9b3f90ec9 --- /dev/null +++ b/crates/cellar/durability/raft/fleet_catalog/namespaces/first_group.rs @@ -0,0 +1,315 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! SDK first-group provisioning over the existing fleet and directory journals. +use super::super::deadline::Deadline; +use super::*; +use crate::application::consensus::RaftNamespaceRegistration; +use crate::raft::{ + catalog::Catalog, + cell_groups::{CellGroupBirthSpec, DirectoryLimits, DirectoryMutation, RangeDirectory}, +}; +use cellar_logic::{cell_group::GroupRoute, consensus::ProposalId}; + +/// Persist this exact setup before reserving the first group. Confirmation +/// has its own fleet request because an unrelated CAS may advance that root. +#[derive(Clone)] +pub struct RaftManagedFirstGroupRequest { + pub reservation: FleetRequestIdentity, + pub standby: [u8; 32], + pub publication: ProposalId, + pub group: CellGroupBirthSpec, +} + +/// Exact admitted reservation and immutable range mutation. Reopen a lost +/// handle by replaying the original reservation and setup request. +pub struct RaftManagedFirstGroup { + pub(super) directory: RangeDirectory, + pub(super) authorization: FleetBirthAuthorization, + pub(super) mutation: DirectoryMutation, + pub(super) reservation: FleetRequestIdentity, +} + +impl FleetCatalog { + /// Read the selected empty standby for an exact managed declaration. + pub async fn observe_managed_standby( + &self, + catalog: &Catalog, + registration: &RaftNamespaceRegistration, + limits: DirectoryLimits, + timeout: Duration, + ) -> Result<[u8; 32]> { + let deadline = Deadline::new(self, timeout)?; + let directory = self + .open_managed_directory(catalog, registration, limits, &deadline) + .await?; + self.observe_first_group_standby(&directory, &deadline) + .await + } + + pub(crate) async fn observe_managed_standby_at( + &self, + directory: &RangeDirectory, + timeout: Duration, + ) -> Result<[u8; 32]> { + let deadline = Deadline::new(self, timeout)?; + self.first_group_class(directory, &deadline).await?; + self.observe_first_group_standby(directory, &deadline).await + } + + async fn observe_first_group_standby( + &self, + directory: &RangeDirectory, + deadline: &Deadline<'_>, + ) -> Result<[u8; 32]> { + self.run(deadline.remaining()?, async { + let view = self + .inner + .domain + .run(directory.observe()) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(catalog_error)?; + if view.revision() != 1 || view.groups() != 0 { + return Err(FleetCatalogError::Conflict); + } + Ok(view.digest()) + }) + .await + } + + /// Reserve the one full-range RF3 group against the original standby. The + /// regular birth admission checks current boot capacity and separation. + /// Replay accepts the exact already-published directory operation as well. + pub async fn reserve_managed_first_group( + &self, + catalog: &Catalog, + registration: &RaftNamespaceRegistration, + request: RaftManagedFirstGroupRequest, + limits: DirectoryLimits, + timeout: Duration, + ) -> Result { + let deadline = Deadline::new(self, timeout)?; + validate_request(&request.reservation)?; + let directory = self + .open_managed_directory(catalog, registration, limits, &deadline) + .await?; + self.reserve_first_group(&directory, request, ®istration.class, &deadline) + .await + } + + pub(crate) async fn reserve_managed_first_group_at( + &self, + directory: &RangeDirectory, + request: RaftManagedFirstGroupRequest, + timeout: Duration, + ) -> Result { + let deadline = Deadline::new(self, timeout)?; + validate_request(&request.reservation)?; + let class = self.first_group_class(directory, &deadline).await?; + self.reserve_first_group(directory, request, &class, &deadline) + .await + } + + async fn reserve_first_group( + &self, + directory: &RangeDirectory, + request: RaftManagedFirstGroupRequest, + class: &str, + deadline: &Deadline<'_>, + ) -> Result { + let (mutation, _) = self + .prepare_managed_first_group(directory, &request, deadline) + .await?; + let snapshot = self.read(deadline.remaining()?).await?; + let reservation = self + .reserve_birth_with_intent( + &snapshot, + &request.reservation, + directory, + (request.group, Some(mutation.digest())), + class, + deadline.remaining()?, + ) + .await?; + Ok(RaftManagedFirstGroup { + directory: directory.clone(), + authorization: reservation.authorization, + mutation, + reservation: request.reservation, + }) + } + + /// Reopen an already admitted first-group plan from its original inputs. + /// This checks the immutable reservation and range mutation without submitting + /// another fleet or directory mutation. + pub async fn reopen_managed_first_group( + &self, + catalog: &Catalog, + registration: &RaftNamespaceRegistration, + request: RaftManagedFirstGroupRequest, + limits: DirectoryLimits, + timeout: Duration, + ) -> Result { + let deadline = Deadline::new(self, timeout)?; + validate_request(&request.reservation)?; + let directory = self + .open_managed_directory(catalog, registration, limits, &deadline) + .await?; + self.reopen_first_group(&directory, request, &deadline) + .await + } + + pub(crate) async fn reopen_managed_first_group_at( + &self, + directory: &RangeDirectory, + request: RaftManagedFirstGroupRequest, + timeout: Duration, + ) -> Result { + let deadline = Deadline::new(self, timeout)?; + validate_request(&request.reservation)?; + self.first_group_class(directory, &deadline).await?; + self.reopen_first_group(directory, request, &deadline).await + } + + async fn reopen_first_group( + &self, + directory: &RangeDirectory, + request: RaftManagedFirstGroupRequest, + deadline: &Deadline<'_>, + ) -> Result { + let (mutation, route) = self + .prepare_managed_first_group(directory, &request, deadline) + .await?; + let snapshot = self.read(deadline.remaining()?).await?; + let authorization = self + .recover_birth_reservation_matching( + &snapshot, + directory, + &request.group.history, + Some((&request.reservation, mutation.digest())), + deadline.remaining()?, + ) + .await?; + if authorization.route() != &route { + return Err(FleetCatalogError::PayloadConflict); + } + Ok(RaftManagedFirstGroup { + directory: directory.clone(), + authorization, + mutation, + reservation: request.reservation, + }) + } + + async fn first_group_class( + &self, + directory: &RangeDirectory, + deadline: &Deadline<'_>, + ) -> Result { + if directory.namespace_config().managed_fleet.as_ref() != Some(&self.birth_owner()?) { + return Err(FleetCatalogError::Conflict); + } + self.run(deadline.remaining()?, async { + let snapshot = self.inner.current().await?; + let declaration = self + .namespace_declaration(&snapshot, directory.namespace(), None) + .await?; + Ok(declaration.class) + }) + .await + } + + async fn prepare_managed_first_group( + &self, + directory: &RangeDirectory, + request: &RaftManagedFirstGroupRequest, + deadline: &Deadline<'_>, + ) -> Result<(DirectoryMutation, GroupRoute)> { + let preview = directory + .prepare_fleet_group_birth(request.group.clone(), &self.birth_owner()?) + .map_err(catalog_error)?; + let mutation = directory + .prepare_managed_first_group(request.standby, request.publication, preview.route()) + .map_err(catalog_error)?; + self.run(deadline.remaining()?, async { + self.inner + .domain + .run(directory.check_managed_first_group(&mutation)) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(catalog_error) + }) + .await?; + Ok((mutation, preview.route().clone())) + } + + /// Publish the exact first range and confirm its birth. If the fleet CAS + /// conflicts after range publication, use a new confirmation request at + /// the observed revision with the same operation and prepared reservation. + /// The returned group authorization still requires physical activation. + pub async fn publish_managed_first_group( + &self, + prepared: &RaftManagedFirstGroup, + confirmation: &FleetRequestIdentity, + timeout: Duration, + ) -> Result { + let deadline = Deadline::new(self, timeout)?; + validate_request(confirmation)?; + if confirmation.id == prepared.reservation.id + || confirmation.operation != prepared.reservation.operation + || confirmation.principal != prepared.reservation.principal + || confirmation.expected_revision <= prepared.reservation.expected_revision + || self.birth_owner()? != *prepared.authorization.owner() + { + return Err(FleetCatalogError::Invalid( + "managed first group confirmation", + )); + } + self.run(deadline.remaining()?, async { + let current = self.inner.current().await?; + let key = self + .binding(prepared.authorization.scope())? + .key() + .map_err(|_| FleetCatalogError::Invalid("first group history binding"))?; + let history = self.history_inner(¤t, &key).await?; + if !prepared.authorization.matches_active_reservation(&history) { + return Err(FleetCatalogError::Conflict); + } + if self + .first_group_intent(¤t, &prepared.reservation.id) + .await? + != Some(prepared.mutation.digest()) + { + return Err(FleetCatalogError::Corrupt( + "first group intent disagrees with reservation", + )); + } + self.inner + .domain + .run(Box::pin(prepared.directory.publish_managed_first_group( + &prepared.mutation, + &prepared.authorization, + ))) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(catalog_error) + }) + .await?; + let snapshot = self.read(deadline.remaining()?).await?; + let committed = self + .confirm_birth( + &snapshot, + confirmation, + &prepared.directory, + &prepared.authorization, + deadline.remaining()?, + ) + .await?; + self.authorize_group( + &committed.snapshot, + prepared.authorization.scope(), + deadline.remaining()?, + ) + .await + } +} diff --git a/crates/cellar/durability/raft/fleet_catalog/namespaces/inventory.rs b/crates/cellar/durability/raft/fleet_catalog/namespaces/inventory.rs index b535f9b57..b792c4020 100644 --- a/crates/cellar/durability/raft/fleet_catalog/namespaces/inventory.rs +++ b/crates/cellar/durability/raft/fleet_catalog/namespaces/inventory.rs @@ -1,26 +1,29 @@ -use super::super::histories::Origin; +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Bidirectional physical-group inventory. Selected ranges must be rooted in +//! the fleet, and all rooted reservations keep their node/debt references. use super::*; -use crate::raft::catalog::{BirthDisposition, ManagedBirthInventory}; +use crate::raft::cell_groups::{CellGroupBirthDisposition, CellGroupBirthRequest, RangeDirectory}; use std::collections::BTreeMap; impl FleetCatalog { - /// Verify the entire rooted namespace set and all histories. Caller handles - /// select adapters, never the coverage set. Missing/offline namespaces refuse - /// success; pending births retain references and do not become absence proof. + /// Verify every declared namespace and every retained physical history at + /// this fleet cut. Caller adapters select neither the coverage set nor its + /// count. An inventory is not a current eligibility or release certificate. pub async fn check_inventory( &self, snapshot: &FleetSnapshot, - namespaces: &[NamespaceCatalog], + directories: &[RangeDirectory], timeout: Duration, ) -> Result { self.check_snapshot(snapshot)?; - if namespaces.len() > MAX_NAMESPACES { + if directories.len() > MAX_NAMESPACES { return Err(FleetCatalogError::Limit); } self.run(timeout, async { self.inner .domain - .run(Box::pin(self.check_inventory_inner(snapshot, namespaces))) + .run(Box::pin(self.check_inventory_inner(snapshot, directories))) .await .map_err(|_| FleetCatalogError::Closed)? }) @@ -30,13 +33,13 @@ impl FleetCatalog { pub(in super::super) async fn check_inventory_inner( &self, snapshot: &FleetSnapshot, - namespaces: &[NamespaceCatalog], + directories: &[RangeDirectory], ) -> Result { let declarations = self.declarations(snapshot).await?; let mut catalogs = BTreeMap::new(); - for namespace in namespaces { + for directory in directories { if catalogs - .insert(namespace.config().namespace.bytes(), namespace) + .insert(directory.namespace_config().namespace.bytes(), directory) .is_some() { return Err(FleetCatalogError::Invalid("duplicate inventory namespace")); @@ -45,53 +48,59 @@ impl FleetCatalog { if catalogs.len() != declarations.len() { return Err(FleetCatalogError::InventoryIncomplete); } + let owner = self.birth_owner()?; for declaration in &declarations { - let namespace = catalogs + let directory = *catalogs .get(&declaration.namespace) .ok_or(FleetCatalogError::InventoryIncomplete)?; - declaration.check(self, namespace, None)?; - if namespace.config().managed_fleet.is_none() { - self.verify_imported_inventory(snapshot, namespace).await?; + declaration.check(self, directory.namespace(), None)?; + // Import must pin and certify the complete range directory before + // it can acquire this owner. The predecessor cell-birth inventory + // cannot certify a shared namespace. + if self.managed_owner_for_birth(snapshot, directory).await? != owner { + return Err(FleetCatalogError::InventoryIncomplete); } - let cut: ManagedBirthInventory = namespace - .managed_birth_inventory(&self.birth_owner()?) - .await - .map_err(catalog_error)?; - if cut.count > HISTORY_LIMIT { + if directory.namespace_config().managed_fleet.is_none() { + self.verify_imported_inventory(snapshot, directory).await?; + } + let view = directory.observe().await.map_err(catalog_error)?; + if u64::from(view.groups()) > HISTORY_LIMIT { return Err(FleetCatalogError::Limit); } let mut cursor = None; let mut count = 0u64; loop { - let page = namespace - .managed_birth_page(&cut, cursor.as_ref()) + let page = directory + .group_page(&view, cursor.as_ref(), 16) .await .map_err(catalog_error)?; - for entry in page.entries { + for route in page.routes { count = count .checked_add(1) - .filter(|n| *n <= cut.count) - .ok_or(FleetCatalogError::Corrupt("extra managed birth intent"))?; - let binding = self.binding(&entry.proposal().scope)?; - let history = self - .history_inner( - snapshot, - &binding - .key() - .map_err(|_| FleetCatalogError::Corrupt("inventory binding"))?, + .filter(|n| *n <= u64::from(view.groups())) + .ok_or(FleetCatalogError::Corrupt("directory inventory count"))?; + let binding = self.binding(&route.history)?; + let key = binding + .key() + .map_err(|_| FleetCatalogError::Corrupt("inventory binding"))?; + let history = + self.history_inner(snapshot, &key) + .await + .map_err(|error| match error { + FleetCatalogError::Conflict => { + FleetCatalogError::InventoryIncomplete + } + error => error, + })?; + let expected = self.inventory_birth(directory, &history)?; + if history.binding != binding + || expected.route() != &route + || !matches!( + history.state, + FleetHistoryState::Reserved | FleetHistoryState::Live ) - .await - .map_err(|error| match error { - FleetCatalogError::Conflict => FleetCatalogError::InventoryIncomplete, - error => error, - })?; - let initial = history.initial_placement()?; - if history.birth_digest != entry.expected_digest() - || initial.scope != entry.proposal().scope - || initial.voters != entry.proposal().voters - || initial.learners != entry.proposal().learners { - return Err(FleetCatalogError::Corrupt("uninventoried canonical birth")); + return Err(FleetCatalogError::Corrupt("uninventoried selected group")); } } cursor = page.next; @@ -99,8 +108,8 @@ impl FleetCatalog { break; } } - if count != cut.count { - return Err(FleetCatalogError::Corrupt("missing managed birth intent")); + if count != u64::from(view.groups()) { + return Err(FleetCatalogError::Corrupt("missing selected group")); } } let nodes = self.node_records(snapshot).await?; @@ -129,7 +138,7 @@ impl FleetCatalog { .iter() .find(|d| d.namespace == scope.namespace.bytes()) .ok_or(FleetCatalogError::InventoryIncomplete)?; - let namespace = catalogs[&declaration.namespace]; + let directory = catalogs[&declaration.namespace]; if entry.key != history .binding @@ -138,8 +147,6 @@ impl FleetCatalog { || history.binding != self.binding(&scope)? || history.namespace_root != declaration.root || history.schema_digest != declaration.schema - || scope.cell.split_once(':').map(|(class, _)| class) - != Some(declaration.class.as_str()) { return Err(FleetCatalogError::Corrupt("inventory history declaration")); } @@ -155,48 +162,57 @@ impl FleetCatalog { } } for member in history.voters.iter().chain(&history.learners) { - if !nodes.iter().any(|n| n.node == member.node) { + if !nodes.iter().any(|node| node.node == member.node) { return Err(FleetCatalogError::InventoryIncomplete); } } - namespace - .verify_group_identity(&scope) - .await - .map_err(catalog_error)?; - let decision = namespace - .birth_disposition(&scope.cell) + let request = self.inventory_birth(directory, &history)?; + let birth = directory + .group_birth_disposition(&request) .await .map_err(catalog_error)?; - match (&history.state, decision) { - (FleetHistoryState::Reserved, decision) => { - let (_, digest) = namespace - .prepare_birth_with_owner( - history.initial_placement()?, - Some(&self.birth_owner()?), - ) - .map_err(catalog_error)?; - if history.origin != Origin::Managed - || digest != history.birth_digest - || decision.as_ref().is_some_and(|d| match d { - BirthDisposition::Live(b) => !history.check_birth(namespace, b), - BirthDisposition::Cancelled { scope: actual, .. } => { - actual != &scope - } - }) - { - return Err(FleetCatalogError::Corrupt("reserved inventory history")); - } + match (&history.state, birth) { + (FleetHistoryState::Reserved, None) if history.allows_managed_genesis() => (), + ( + FleetHistoryState::Reserved | FleetHistoryState::Live, + Some(CellGroupBirthDisposition::Live(birth)), + ) if history.check_birth(directory, &birth) => (), + ( + FleetHistoryState::Retired { + operation, + retirement, + }, + disposition, + ) => { + let materialized = match disposition { + Some(CellGroupBirthDisposition::Live(birth)) + if history.check_birth(directory, &birth) => + { + true + } + None => false, + _ => return Err(FleetCatalogError::InventoryIncomplete), + }; + self.require_retired_history( + snapshot, + directory, + request.route(), + *operation, + *retirement, + (&nodes, materialized), + ) + .await?; } - (FleetHistoryState::Live, Some(BirthDisposition::Live(birth))) - if history.check_birth(namespace, &birth) => {} ( FleetHistoryState::Cancelled { operation }, - Some(BirthDisposition::Cancelled { - scope: actual, - operation: actual_op, + Some(CellGroupBirthDisposition::Cancelled { + history: actual, + operation: decision, }), - ) if actual == scope && actual_op.bytes() == *operation => {} - _ => return Err(FleetCatalogError::Corrupt("inventory birth disposition")), + ) if actual == scope && decision.bytes() == *operation => (), + // Cancellation is a terminal directory decision, not a + // caller-supplied enum value or an observation of absence. + _ => return Err(FleetCatalogError::InventoryIncomplete), } } cursor = page.next; @@ -213,4 +229,23 @@ impl FleetCatalog { histories, }) } + + fn inventory_birth( + &self, + directory: &RangeDirectory, + history: &FleetHistory, + ) -> Result { + let request = directory + .prepare_fleet_group_birth(history.initial_placement()?, &self.birth_owner()?) + .map_err(catalog_error)?; + if request.route().birth_digest != history.birth_digest + || directory.namespace().root_digest() != history.namespace_root + || request.route().schema_digest != history.schema_digest + { + return Err(FleetCatalogError::Corrupt( + "inventory immutable group birth", + )); + } + Ok(request) + } } diff --git a/crates/cellar/durability/raft/fleet_catalog/namespaces/mod.rs b/crates/cellar/durability/raft/fleet_catalog/namespaces/mod.rs index a12c52cb1..9c0d7aefc 100644 --- a/crates/cellar/durability/raft/fleet_catalog/namespaces/mod.rs +++ b/crates/cellar/durability/raft/fleet_catalog/namespaces/mod.rs @@ -6,8 +6,13 @@ use crate::raft::catalog::{NamespaceCatalog, VoterPolicy}; use cellar_logic::consensus::NamespaceId; use serde::Deserialize; +mod archive; +mod first_group; mod inventory; +mod provisioning; mod records; +pub use archive::RaftManagedArchiveProvisioning; +pub use first_group::{RaftManagedFirstGroup, RaftManagedFirstGroupRequest}; use records::{Declaration, Registry}; pub(super) const KEY: &str = "compiled-namespaces"; @@ -67,80 +72,91 @@ impl FleetCatalog { validate_request(request)?; let declaration = Declaration::new(self, namespace, class)?; self.run(timeout, async { - // A reopened adapter may reconcile the original immutable candidate - // after later root changes, without reconstructing its former index. - let prior = self - .inner - .read::( - &format!("requests/{}", hex::encode(request.id)), - REQUEST_KIND, - REQUEST_LIMIT, - ) + let mutation = self + .prepare_namespace_declaration(snapshot, request, &declaration) .await?; - if let Some((candidate, _)) = prior { - candidate.validate()?; - if candidate.id != request.id - || candidate.operation != request.operation - || candidate.principal != request.principal - || candidate.expected_revision != request.expected_revision - || candidate.changes.len() != 1 - || candidate.changes[0].table != FleetTable::Policy - || candidate.changes[0].key != KEY - { - return Err(FleetCatalogError::PayloadConflict); - } - let registry = Registry::decode( - candidate.changes[0] - .value - .as_deref() - .ok_or(FleetCatalogError::PayloadConflict)?, - )?; - let bytes = self - .inner - .lookup(registry.index, &declaration.key()) - .await? - .ok_or(FleetCatalogError::PayloadConflict)?; - if decode::(DECLARATION_KIND, &bytes, 4096)? != declaration { - return Err(FleetCatalogError::PayloadConflict); - } - return self.commit_inner(snapshot, &candidate).await; + self.commit_inner(snapshot, &mutation).await + }) + .await + } + + async fn prepare_namespace_declaration( + &self, + snapshot: &FleetSnapshot, + request: &FleetRequestIdentity, + declaration: &Declaration, + ) -> Result { + // A reopened adapter may reconcile the original immutable candidate + // after later root changes, without reconstructing its former index. + let prior = self + .inner + .read::( + &format!("requests/{}", hex::encode(request.id)), + REQUEST_KIND, + REQUEST_LIMIT, + ) + .await?; + if let Some((candidate, _)) = prior { + candidate.validate()?; + if candidate.id != request.id + || candidate.operation != request.operation + || candidate.principal != request.principal + || candidate.expected_revision != request.expected_revision + || candidate.changes.len() != 1 + || candidate.changes[0].table != FleetTable::Policy + || candidate.changes[0].key != KEY + { + return Err(FleetCatalogError::PayloadConflict); } - let mut registry = self.registry(snapshot).await?.unwrap_or(Registry { - count: 0, - index: None, - }); - if self + let registry = Registry::decode( + candidate.changes[0] + .value + .as_deref() + .ok_or(FleetCatalogError::PayloadConflict)?, + )?; + let bytes = self .inner .lookup(registry.index, &declaration.key()) .await? - .is_some() - { - return Err(FleetCatalogError::Conflict); - } - if registry.count >= MAX_NAMESPACES { - return Err(FleetCatalogError::Limit); + .ok_or(FleetCatalogError::PayloadConflict)?; + if decode::(DECLARATION_KIND, &bytes, 4096)? != *declaration { + return Err(FleetCatalogError::PayloadConflict); } - registry.index = self - .inner - .update( - registry.index, - &declaration.key(), - Some(encode(DECLARATION_KIND, &declaration, 4096)?), - 0, - ) - .await?; - registry.count += 1; - let mutation = history_mutation( - request, - vec![FleetChange { - table: FleetTable::Policy, - key: KEY.into(), - value: Some(encode(REGISTRY_KIND, ®istry, 4096)?), - }], - ); - self.commit_inner(snapshot, &mutation).await - }) - .await + return Ok(candidate); + } + let mut registry = self.registry(snapshot).await?.unwrap_or(Registry { + count: 0, + index: None, + }); + if self + .inner + .lookup(registry.index, &declaration.key()) + .await? + .is_some() + { + return Err(FleetCatalogError::Conflict); + } + if registry.count >= MAX_NAMESPACES { + return Err(FleetCatalogError::Limit); + } + registry.index = self + .inner + .update( + registry.index, + &declaration.key(), + Some(encode(DECLARATION_KIND, declaration, 4096)?), + 0, + ) + .await?; + registry.count += 1; + Ok(history_mutation( + request, + vec![FleetChange { + table: FleetTable::Policy, + key: KEY.into(), + value: Some(encode(REGISTRY_KIND, ®istry, 4096)?), + }], + )) } async fn registry(&self, snapshot: &FleetSnapshot) -> Result> { @@ -186,6 +202,17 @@ impl FleetCatalog { namespace: &NamespaceCatalog, class: Option<&str>, ) -> Result<()> { + self.namespace_declaration(snapshot, namespace, class) + .await?; + Ok(()) + } + + async fn namespace_declaration( + &self, + snapshot: &FleetSnapshot, + namespace: &NamespaceCatalog, + class: Option<&str>, + ) -> Result { let registry = self .registry(snapshot) .await? @@ -199,30 +226,38 @@ impl FleetCatalog { .await? .ok_or(FleetCatalogError::InventoryIncomplete)?; let declaration: Declaration = decode(DECLARATION_KIND, &bytes, 4096)?; - declaration.check(self, namespace, class) + declaration.check(self, namespace, class)?; + Ok(declaration) } } -impl FleetCatalog { - /// Verify the actual prepared catalogs, not only the caller's digest. This - /// check is independent of inventory completion so import hosts can boot. +impl FleetHostAuthorization { + /// Verify this host and the prepared catalogs against one current root + /// before the caller opens mutable engine state. A saved capability cannot + /// reopen an already quarantined/finalized boot. This remains independent + /// of inventory completion so import hosts can boot; the later engine-bound + /// boot CAS must still recheck authority before publishing any endpoint. pub(crate) async fn check_declared_namespaces( &self, entries: &[crate::raft::host::NamespaceRegistration], timeout: Duration, ) -> Result<()> { - self.run(timeout, async { - let root = self.inner.current().await?; - for entry in entries { - self.require_namespace_declaration( - &root, - &entry.namespace, - Some(entry.class.name()), - ) - .await?; - } - Ok(()) - }) - .await + self.catalog + .run(timeout, async { + let root = self.catalog.inner.current().await?; + let current = self.catalog.host_from(&root, self.node()).await?; + self.require_successor(¤t)?; + for entry in entries { + self.catalog + .require_namespace_declaration( + &root, + &entry.namespace, + Some(entry.class.name()), + ) + .await?; + } + Ok(()) + }) + .await } } diff --git a/crates/cellar/durability/raft/fleet_catalog/namespaces/provisioning.rs b/crates/cellar/durability/raft/fleet_catalog/namespaces/provisioning.rs new file mode 100644 index 000000000..d2ad4b22a --- /dev/null +++ b/crates/cellar/durability/raft/fleet_catalog/namespaces/provisioning.rs @@ -0,0 +1,149 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Publish an exact managed standby before confirming its fleet declaration. +use super::super::deadline::Deadline; +use super::*; +use crate::application::consensus::{RaftClass, RaftNamespaceRegistration}; +use crate::raft::{ + catalog::{Catalog, CatalogError, NamespaceConfig}, + cell_groups::{DirectoryLimits, RangeDirectory}, +}; +use cellar_logic::{ + consensus::{ClusterId, NamespaceId, ProposalId}, + NamespaceMode, +}; + +impl FleetCatalog { + /// Declare a new managed SQL namespace using the fleet's cluster and the + /// compiled Raft class. Defaults are three voters, quorum durability and + /// materialized images. The declaration creates an empty standby and + /// retains the supplied request identity through retries. + pub async fn provision_managed_sql_namespace( + &self, + request: &FleetRequestIdentity, + catalog: &Catalog, + namespace: NamespaceId, + class: &RaftClass, + directory_limits: DirectoryLimits, + timeout: Duration, + ) -> Result { + let cluster = ClusterId::new(self.inner.cluster) + .map_err(|_| FleetCatalogError::Corrupt("fleet cluster identity"))?; + let config = NamespaceConfig::new( + cluster, + namespace, + NamespaceMode::Raft, + class.schema_digest(), + ); + self.provision_managed_namespace(request, catalog, config, class, directory_limits, timeout) + .await + } + + /// Provision a declared managed standby using the compiled class and the + /// original fleet request. Persist that request and the catalog settings + /// before calling. An exact retry resumes the same immutable setup. If + /// the fleet root advances before declaration, a new request at the new + /// revision can continue the same standby operation with the same config. + /// This publishes an empty directory and does not allocate a group or voter. + pub async fn provision_managed_namespace( + &self, + request: &FleetRequestIdentity, + catalog: &Catalog, + config: NamespaceConfig, + class: &RaftClass, + directory_limits: DirectoryLimits, + timeout: Duration, + ) -> Result { + validate_request(request)?; + let config = self.bind_namespace(config)?; + if class.schema_digest() != config.schema_digest { + return Err(FleetCatalogError::Invalid("compiled namespace schema")); + } + let namespace = catalog + .describe_namespace(config.clone()) + .map_err(catalog_error)?; + let declaration = Declaration::new(self, &namespace, class.name())?; + let directory = RangeDirectory::new(namespace, directory_limits).map_err(catalog_error)?; + let operation = ProposalId::new(request.operation) + .map_err(|_| FleetCatalogError::Invalid("standby operation"))?; + let standby = directory + .prepare_standby_initialization(operation) + .map_err(catalog_error)?; + self.run(timeout, async { + let snapshot = self.inner.current().await?; + let mutation = self + .prepare_namespace_declaration(&snapshot, request, &declaration) + .await?; + mutation.validate()?; + let bytes = encode(REQUEST_KIND, &mutation, REQUEST_LIMIT)?; + let key = hex::encode(request.id); + let committed = self + .inner + .committed(&snapshot.head, &key, hash(&bytes)) + .await? + .is_some(); + if !committed { + if snapshot.revision() != request.expected_revision { + return Err(FleetCatalogError::Conflict); + } + // Pin the exact declaration before the first namespace write. + // A losing root CAS retains the intent but grants no authority. + self.inner + .create_exact(&format!("requests/{key}"), &bytes, REQUEST_LIMIT) + .await?; + } + self.inner + .domain + .run(Box::pin(async { + if committed { + catalog.existing_namespace(config.clone()).await?; + directory + .publication(&standby) + .await? + .ok_or(CatalogError::Conflict( + "missing managed standby publication", + ))?; + } else { + catalog.open_namespace(config.clone()).await?; + directory.publish(&standby).await?; + } + Ok::<_, CatalogError>(()) + })) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(catalog_error)?; + self.commit_inner(&snapshot, &mutation).await?; + Ok(RaftNamespaceRegistration { + namespace: config, + class: declaration.class, + }) + }) + .await + } + + pub(super) async fn open_managed_directory( + &self, + catalog: &Catalog, + registration: &RaftNamespaceRegistration, + limits: DirectoryLimits, + deadline: &Deadline<'_>, + ) -> Result { + if registration.namespace.managed_fleet.as_ref() != Some(&self.birth_owner()?) { + return Err(FleetCatalogError::Conflict); + } + self.run(deadline.remaining()?, async { + let namespace = self + .inner + .domain + .run(catalog.existing_namespace(registration.namespace.clone())) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(catalog_error)?; + let snapshot = self.inner.current().await?; + self.require_namespace_declaration(&snapshot, &namespace, Some(®istration.class)) + .await?; + RangeDirectory::new(namespace, limits).map_err(catalog_error) + }) + .await + } +} diff --git a/crates/cellar/durability/raft/fleet_catalog/nodes.rs b/crates/cellar/durability/raft/fleet_catalog/nodes.rs index 45aedd073..14102f6db 100644 --- a/crates/cellar/durability/raft/fleet_catalog/nodes.rs +++ b/crates/cellar/durability/raft/fleet_catalog/nodes.rs @@ -360,6 +360,15 @@ impl fmt::Debug for FleetHostAuthorization { } impl FleetHostAuthorization { + pub(crate) fn same_boot_owner(&self, other: &Self) -> bool { + Arc::ptr_eq(&self.catalog.inner, &other.catalog.inner) + && self + .boot_binding() + .ok() + .zip(other.boot_binding().ok()) + .is_some_and(|(left, right)| left == right) + } + pub(crate) fn is_isolated(&self) -> bool { self.record.isolation.is_some() } @@ -373,16 +382,6 @@ impl FleetHostAuthorization { self.catalog.clone() } - pub(crate) async fn check_declared_namespaces( - &self, - entries: &[crate::raft::host::NamespaceRegistration], - timeout: Duration, - ) -> Result<()> { - self.catalog - .check_declared_namespaces(entries, timeout) - .await - } - pub(crate) fn check_namespace( &self, config: &crate::raft::catalog::NamespaceConfig, @@ -396,10 +395,10 @@ impl FleetHostAuthorization { Ok(()) } - /// The one owned discovery worker has its own bounded read turn. Sharing - /// the mutation adapter's turn would let a periodic read reject foreground - /// history admission while waiting on storage. Its caller retains the - /// separate physical metadata reservation through this reader's completion. + /// Give a metadata owner a private bounded catalog turn. Sharing the + /// mutation adapter's turn would let discovery or a remote read reject + /// foreground work while waiting on storage. The owner retains its metadata + /// reservation through the read. pub(crate) fn trust_reader(&self) -> FleetTrustReader { let source = &self.catalog.inner; let mut authorization = self.clone(); @@ -488,11 +487,24 @@ impl FleetHostAuthorization { &self, peer: &crate::raft::transport_options::RaftPeerNode, ) -> bool { - self.permits_transport() - && peer.node == self.node - && peer.disk == self.disk - && peer.label == self.record.label - && peer.server_name == self.record.server_name + self.permits_transport() && self.check_boot_endpoint(peer).is_ok() + } + + /// Validate the immutable host descriptor before claiming a new boot. The + /// process and advertised addresses belong to that boot; transport admission + /// independently requires a currently pinned, permitted process. + pub(crate) fn check_boot_endpoint( + &self, + peer: &crate::raft::transport_options::RaftPeerNode, + ) -> Result<()> { + if peer.node != self.node + || peer.disk != self.disk + || peer.label != self.record.label + || peer.server_name != self.record.server_name + { + return Err(FleetCatalogError::Conflict); + } + Ok(()) } pub fn node(&self) -> NodeId { @@ -555,7 +567,7 @@ impl FleetHostAuthorization { pub(crate) async fn group_authorization( &self, - scope: &cellar_logic::consensus::GroupScope, + scope: &cellar_logic::cell_group::GroupHistory, timeout: Duration, ) -> Result { self.group_authorization_at_least(scope, 0, timeout).await @@ -563,7 +575,7 @@ impl FleetHostAuthorization { pub(crate) async fn group_authorization_at_least( &self, - scope: &cellar_logic::consensus::GroupScope, + scope: &cellar_logic::cell_group::GroupHistory, minimum_revision: u64, timeout: Duration, ) -> Result { @@ -644,7 +656,7 @@ impl FleetHostAuthorization { } } -/// Read-only surface of the single separately budgeted directory adapter. +/// Read-only surface of a separately budgeted metadata adapter. #[derive(Clone)] pub(crate) struct FleetTrustReader { authorization: FleetHostAuthorization, @@ -658,6 +670,46 @@ impl FleetTrustReader { ) -> Result { self.authorization.refresh_trust(previous, timeout).await } + + pub(crate) async fn authorize_group_at_least( + &self, + scope: &cellar_logic::cell_group::GroupHistory, + minimum_revision: u64, + timeout: Duration, + ) -> Result { + self.authorization + .group_authorization_at_least(scope, minimum_revision, timeout) + .await + } + + /// Resolve one router group using the directory's isolated read turn. + pub(crate) async fn authorize_router_group( + &self, + trust: &FleetTrustSnapshot, + scope: &cellar_logic::cell_group::GroupHistory, + timeout: Duration, + ) -> Result> { + let catalog = &self.authorization.catalog; + let expected = catalog.binding(scope)?; + let key = expected + .key() + .map_err(|_| FleetCatalogError::Invalid("group binding"))?; + catalog + .run(timeout, async { + let cut = catalog.inner.current().await?; + if cut.revision() != trust.revision { + return Ok(None); + } + if hash(&encode(HEAD_KIND, &cut.head, ROOT_LIMIT)?) != trust.root_digest { + return Err(FleetCatalogError::Conflict); + } + catalog + .authorize_live_group(&cut, expected, &key) + .await + .map(Some) + }) + .await + } } /// Opaque complete node-table cut, produced only by a bounded current-root read. diff --git a/crates/cellar/durability/raft/fleet_catalog/permits.rs b/crates/cellar/durability/raft/fleet_catalog/permits.rs index a3289e331..7527bc1a3 100644 --- a/crates/cellar/durability/raft/fleet_catalog/permits.rs +++ b/crates/cellar/durability/raft/fleet_catalog/permits.rs @@ -2,7 +2,10 @@ //! A capability never replaces the group's current quorum or consumed sequence. use super::*; use crate::raft::member_state::{FleetGroupBinding, FleetMembershipPermit}; -use cellar_logic::consensus::GroupScope; +use cellar_logic::{ + cell_group::GroupHistory, + consensus::{DiskLineage, MemberId, NodeId}, +}; #[derive(Clone, Debug)] pub struct FleetGroupAuthorization { @@ -11,29 +14,113 @@ pub struct FleetGroupAuthorization { owner: super::super::catalog::ManagedBirthOwner, } +pub(crate) struct RouterVoter { + pub(crate) member: MemberId, + pub(crate) node: NodeId, + pub(crate) joined_disk: Option, +} + impl FleetGroupAuthorization { - pub(crate) fn matches_replica( + pub(crate) fn matches_replica( &self, - replica: &cellar_logic::consensus::ReplicaIdentity, + replica: &cellar_logic::consensus::ReplicaIdentity, membership: &crate::raft::member_state::GroupMembership, - ) -> bool { + ) -> bool + where + for<'a> GroupHistory: From<&'a S>, + { if !self.matches_scope(&replica.scope) || !membership.permits(replica) { return false; } match membership.joined(replica.member) { Ok(Some((joined, _))) => joined.node == replica.node && joined.disk == replica.disk, - Ok(None) => self - .history - .voters - .iter() - .chain(&self.history.learners) - .any(|initial| { - initial.member == replica.member.get() && initial.node == replica.node.bytes() - }), + Ok(None) => self.initial_member_node(replica.member) == Some(replica.node), Err(_) => false, } } + /// Resolve all current physical occupants from the immutable birth or the + /// committed joining bindings. An advisory route may omit an isolated peer. + pub(crate) fn placement_nodes( + &self, + observed: &crate::raft::cell_groups::CellGroupQuorumMembership, + ) -> std::result::Result, crate::raft::storage::LogError> { + let invalid = + || crate::raft::storage::LogError::Invalid("group placement authority".into()); + if !self.matches_replica(observed.replica(), observed.membership()) + || self.history_digests() + != ( + observed.root().route().birth_digest, + observed.root().route().schema_digest, + ) + { + return Err(invalid()); + } + crate::raft::member_state::members(observed.configuration()) + .into_iter() + .map(|id| { + let member = cellar_logic::consensus::MemberId::new(id).map_err(|_| invalid())?; + if let Some((binding, _)) = observed.membership().joined(member)? { + return Ok(binding.node); + } + self.initial_member_node(member).ok_or_else(invalid) + }) + .collect() + } + + /// Resolve initial or joined voters from one authenticated quorum sample. + /// A member number alone supplies no endpoint or boot authority. + pub(crate) fn router_voters( + &self, + observed: &crate::raft::host::FleetMembershipObservation, + ) -> Result> { + if !self.matches_scope(&observed.replica().scope) + || self.binding() != &observed.status().binding + || self.history_digests() != (observed.birth_digest(), observed.schema_digest()) + { + return Err(FleetCatalogError::Conflict); + } + let conf = observed.configuration(); + let members = conf + .voters() + .iter() + .chain(conf.outgoing_voters()) + .copied() + .collect::>(); + members + .into_iter() + .map(|member| { + if let Some((joined, _)) = observed + .joined_member(member) + .map_err(|_| FleetCatalogError::Invalid("router member binding"))? + { + return Ok(RouterVoter { + member, + node: joined.node, + joined_disk: Some(joined.disk), + }); + } + let node = self + .initial_member_node(member) + .ok_or(FleetCatalogError::Conflict)?; + Ok(RouterVoter { + member, + node, + joined_disk: None, + }) + }) + .collect() + } + + fn initial_member_node(&self, member: MemberId) -> Option { + self.history + .voters + .iter() + .chain(&self.history.learners) + .find(|initial| initial.member == member.get()) + .and_then(|initial| NodeId::new(initial.node).ok()) + } + pub(crate) fn history_digests(&self) -> ([u8; 32], [u8; 32]) { (self.history.birth_digest(), self.history.schema_digest()) } @@ -44,33 +131,36 @@ impl FleetGroupAuthorization { pub(crate) fn check_birth( &self, - namespace: &crate::raft::catalog::NamespaceCatalog, - birth: &crate::raft::catalog::GroupBirth, + directory: &crate::raft::cell_groups::RangeDirectory, + birth: &crate::raft::cell_groups::CellGroupBirth, ) -> bool { - self.history.check_birth(namespace, birth) - && birth - .managed_owner() + self.history.check_birth(directory, birth) + && directory + .namespace() + .config() + .managed_fleet + .as_ref() .is_none_or(|owner| owner == &self.owner) } pub(crate) fn matches_birth( &self, - birth: &crate::raft::catalog::GroupBirth, + birth: &crate::raft::cell_groups::CellGroupBirth, schema: [u8; 32], ) -> bool { - self.matches_scope(birth.scope()) - && self.history.birth_digest() == birth.digest() + self.matches_scope(&birth.route().history) + && self.history.birth_digest() == birth.route().birth_digest && self.history.schema_digest() == schema - && birth - .managed_owner() - .is_none_or(|owner| owner == &self.owner) } pub(crate) fn binding(&self) -> &FleetGroupBinding { &self.binding } - pub(crate) fn matches_scope(&self, scope: &GroupScope) -> bool { + pub(crate) fn matches_scope(&self, scope: &S) -> bool + where + for<'a> GroupHistory: From<&'a S>, + { self.binding.matches_scope(scope) } } @@ -109,7 +199,10 @@ impl FleetPermitAuthorization { &self.permit } - pub(crate) fn matches_scope(&self, scope: &GroupScope) -> bool { + pub(crate) fn matches_scope(&self, scope: &S) -> bool + where + for<'a> GroupHistory: From<&'a S>, + { self.permit.binding().matches_scope(scope) } } @@ -120,7 +213,7 @@ impl FleetCatalog { pub async fn authorize_group( &self, snapshot: &FleetSnapshot, - scope: &GroupScope, + scope: &GroupHistory, timeout: Duration, ) -> Result { self.check_snapshot(snapshot)?; @@ -128,27 +221,31 @@ impl FleetCatalog { let key = expected .key() .map_err(|_| FleetCatalogError::Invalid("group binding"))?; - self.run(timeout, async { - let bytes = self - .inner - .lookup(snapshot.head.tables[FleetTable::Histories.slot()], &key) - .await? - .ok_or(FleetCatalogError::Conflict)?; - let history = super::histories::FleetHistory::decode(&bytes)?; - let actual = history.binding().clone(); - if actual != expected { - return Err(FleetCatalogError::Conflict); - } - if *history.state() != FleetHistoryState::Live { - return Err(FleetCatalogError::Conflict); - } - Ok(FleetGroupAuthorization { - binding: actual, - history: Box::new(history), - owner: self.birth_owner()?, - }) + self.run(timeout, self.authorize_live_group(snapshot, expected, &key)) + .await + } + + pub(super) async fn authorize_live_group( + &self, + snapshot: &FleetSnapshot, + expected: FleetGroupBinding, + key: &str, + ) -> Result { + let bytes = self + .inner + .lookup(snapshot.head.tables[FleetTable::Histories.slot()], key) + .await? + .ok_or(FleetCatalogError::Conflict)?; + let history = super::histories::FleetHistory::decode(&bytes)?; + let actual = history.binding().clone(); + if actual != expected || *history.state() != FleetHistoryState::Live { + return Err(FleetCatalogError::Conflict); + } + Ok(FleetGroupAuthorization { + binding: actual, + history: Box::new(history), + owner: self.birth_owner()?, }) - .await } /// Resolve the exact rooted admission for a registered history. Replaying @@ -156,7 +253,7 @@ impl FleetCatalog { pub async fn authorize_membership( &self, snapshot: &FleetSnapshot, - scope: &GroupScope, + scope: &GroupHistory, timeout: Duration, ) -> Result { self.check_snapshot(snapshot)?; @@ -189,21 +286,25 @@ impl FleetCatalog { .await } - pub(super) fn binding(&self, scope: &GroupScope) -> Result { + pub(super) fn binding(&self, scope: &S) -> Result + where + for<'a> GroupHistory: From<&'a S>, + { + let scope = GroupHistory::from(scope); if scope.cluster.bytes() != self.inner.cluster { return Err(FleetCatalogError::Conflict); } FleetGroupBinding::new( IncarnationId::new(self.inner.incarnation) .map_err(|_| FleetCatalogError::Corrupt("fleet incarnation"))?, - scope, + &scope, self.birth_owner()?.catalog, ) .map_err(|_| FleetCatalogError::Invalid("group scope")) } /// Exact catalog-scoped descriptor. The value itself grants no authority. - pub fn group_binding(&self, scope: &GroupScope) -> Result { + pub fn group_binding(&self, scope: &GroupHistory) -> Result { self.binding(scope) } } diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/admission.rs b/crates/cellar/durability/raft/fleet_catalog/resources/admission.rs index d6c5219c8..61fb70401 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/admission.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/admission.rs @@ -5,7 +5,7 @@ use super::records::{Book, Record, Reference, Request}; use super::*; use crate::application::consensus::RaftCells; -use crate::raft::catalog::NamespaceCatalog; +use crate::raft::catalog::RangeDirectory; use crate::raft::host::RaftPeerNodeObservation; use cellar_logic::consensus::{ ClusterId, DiskLineage, EngineInstanceId, IncarnationId, NodeIdentity, ProcessGeneration, @@ -51,7 +51,7 @@ impl FleetCatalog { snapshot: &FleetSnapshot, request: &FleetResourceRequest, observer: &RaftCells, - namespaces: &[NamespaceCatalog], + namespaces: &[RangeDirectory], timeout: Duration, ) -> Result { self.reserve_resource_request( @@ -70,7 +70,7 @@ impl FleetCatalog { snapshot: &FleetSnapshot, canonical: Request, observer: &RaftCells, - namespaces: &[NamespaceCatalog], + namespaces: &[RangeDirectory], options: AdmissionOptions, timeout: Duration, ) -> Result { @@ -156,11 +156,7 @@ impl FleetCatalog { .binding() .scope() .map_err(|_| FleetCatalogError::Corrupt("resource scope"))?; - if inventory - .history(&scope) - .map_err(|_| FleetCatalogError::Corrupt("resource destination scope"))? - .is_some() - { + if inventory.group_history(&scope).is_some() { return Err(FleetCatalogError::Invalid("occupied resource destination")); } if *history.state() != FleetHistoryState::Live { @@ -416,12 +412,14 @@ impl FleetCatalog { .filter(|n| *n <= HISTORY_LIMIT) .ok_or(FleetCatalogError::Limit)?; let history = FleetHistory::decode(&entry.value)?; - if !matches!(history.state(), FleetHistoryState::Cancelled { .. }) - && history - .voters - .iter() - .chain(&history.learners) - .any(|m| m.node == target.bytes()) + if !matches!( + history.state(), + FleetHistoryState::Cancelled { .. } | FleetHistoryState::Retired { .. } + ) && history + .voters + .iter() + .chain(&history.learners) + .any(|m| m.node == target.bytes()) { if !matches!(history.state(), FleetHistoryState::Live) { return Err(resource_error(LedgerError::Incomplete)); @@ -431,8 +429,7 @@ impl FleetCatalog { .scope() .map_err(|_| FleetCatalogError::Corrupt("resource genesis scope"))?; let occupied = inventory - .history(&scope) - .map_err(|_| FleetCatalogError::Corrupt("resource genesis scope"))? + .group_history(&scope) .ok_or(resource_error(LedgerError::Incomplete))?; if occupied.birth_digest() != history.birth_digest() || !history.voters.iter().chain(&history.learners).any(|m| { diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/completion/mod.rs b/crates/cellar/durability/raft/fleet_catalog/resources/completion/mod.rs index 251bd3f69..bc7090415 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/completion/mod.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/completion/mod.rs @@ -471,8 +471,7 @@ impl FleetCatalog { .inventory .as_ref() .ok_or(FleetCatalogError::Unconfirmed)? - .history(&scope) - .map_err(|_| invalid())? + .group_history(&scope) .is_some(), membership_revision: quorum.membership().membership_revision().get(), barrier: quorum.membership().barrier().get(), diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/completion/proofs.rs b/crates/cellar/durability/raft/fleet_catalog/resources/completion/proofs.rs index 4b25a33af..358c2f473 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/completion/proofs.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/completion/proofs.rs @@ -298,7 +298,7 @@ impl FleetCatalog { .ok_or(FleetCatalogError::Unconfirmed)?; let history = self.history_inner(snapshot, &row.request.group).await?; let scope = history.binding().scope().map_err(|_| invalid())?; - let physical = inventory.history(&scope).map_err(|_| invalid())?; + let physical = inventory.group_history(&scope); let matches = match (role, physical) { (PhysicalRole::EmptyTarget, None) => true, (PhysicalRole::AbortedTarget { .. }, None) => true, diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/execution/mod.rs b/crates/cellar/durability/raft/fleet_catalog/resources/execution/mod.rs index acbd56223..d6a946c50 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/execution/mod.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/execution/mod.rs @@ -60,7 +60,7 @@ pub struct FleetMovementProgress { #[derive(Clone, Debug)] pub struct FleetPendingMovement { pub reservation: [u8; 16], - pub group: GroupScope, + pub group: GroupHistory, pub plan: crate::raft::member_state::RepairPlan, } @@ -69,7 +69,7 @@ pub struct FleetPendingMovement { #[derive(Clone, Debug)] pub struct FleetPendingReservation { pub reservation: [u8; 16], - pub group: cellar_logic::consensus::GroupScope, + pub group: cellar_logic::cell_group::GroupHistory, pub target: cellar_logic::consensus::NodeId, pub boot: crate::raft::fleet_stop::FleetBootBinding, pub peak: FleetResourceBounds, @@ -144,7 +144,8 @@ impl FleetCatalog { /// Reconcile one exact admitted move through the authenticated control lane. /// Recreated callers discover the same work from pending_movements. A lost /// action reply requires another fresh observation; it never releases debt. - /// No internal retry loop or application traffic is needed for progress. + /// Read-only leader discovery may resample within the turn deadline. + /// Actions are not retried within a turn; progress needs no application traffic. pub fn reconcile_movement<'a>( &'a self, snapshot: &'a FleetSnapshot, @@ -282,7 +283,14 @@ impl FleetCatalog { deadline.remaining()?, kind, ) - .await? + .await + .map_err(|error| match error { + // Provider I/O may fail after the terminal CAS has + // committed. This turn cannot claim rollback; the + // next turn must inspect the permanent receipt. + FleetCatalogError::Storage(_) => FleetCatalogError::Unconfirmed, + other => other, + })? .observed_revision(); return Ok(progress(phase, true, observed)); } diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/execution/receive.rs b/crates/cellar/durability/raft/fleet_catalog/resources/execution/receive.rs index d9156adbf..75fba7098 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/execution/receive.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/execution/receive.rs @@ -23,7 +23,25 @@ impl FleetHostAuthorization { /// exclusion. A history descriptor or old member permit alone is insufficient. pub(crate) async fn admitted_learner( &self, - scope: &GroupScope, + scope: &GroupHistory, + timeout: Duration, + ) -> Result { + self.admitted_seed(scope, None, timeout).await + } + + pub(crate) async fn admitted_learner_from( + &self, + scope: &GroupHistory, + source: &cellar_logic::consensus::ReplicaIdentity, + timeout: Duration, + ) -> Result { + self.admitted_seed(scope, Some(source), timeout).await + } + + async fn admitted_seed( + &self, + scope: &GroupHistory, + source: Option<&cellar_logic::consensus::ReplicaIdentity>, timeout: Duration, ) -> Result { let catalog = &self.catalog; @@ -50,6 +68,22 @@ impl FleetHostAuthorization { { return Err(FleetCatalogError::Conflict); } + if let Some(source) = source { + if source.scope != *scope + || !row.seed_owners.as_ref().is_some_and(|owners| { + owners.iter().any(|owner| { + owner.member == source.member.get() + && owner.node == source.node.bytes() + }) + }) + { + return Err(FleetCatalogError::Conflict); + } + let boot = catalog.movement_boot(&snapshot, &row, source.node).await?; + if boot.disk != source.disk.bytes() || boot.process != source.process.bytes() { + return Err(FleetCatalogError::Conflict); + } + } Ok(FleetPermitAuthorization::admitted( permit, snapshot.revision(), @@ -131,35 +165,56 @@ impl FleetHostAuthorization { let authorization = catalog .authorize_membership(&snapshot, &scope, deadline.remaining()?) .await?; - let cell = match gateway.cell(scope.namespace, &scope.cell) { - Ok(cell) if cell.scope() == &scope => cell, - _ if matches!(command.action, MovementAction::Activate { .. }) => { - let group = catalog - .authorize_group(&snapshot, &scope, deadline.remaining()?) - .await?; - gateway - .register_managed_history(group, deadline.remaining()?) + if !matches!(command.action, MovementAction::Activate { .. }) { + let group = gateway + .group(scope) + .map_err(|_| FleetCatalogError::Conflict)?; + let history = catalog + .authorize_group(&snapshot, &scope, deadline.remaining()?) + .await?; + return match command.action { + MovementAction::Begin => group + .begin_managed_repair( + history, + authorization, + row.request.movement.as_ref().ok_or_else(invalid)?.plan()?, + deadline.remaining()?, + ) .await - .map_err(command_failure)? - } - _ => return Err(FleetCatalogError::Conflict), - }; + .map(|_| ()) + .map_err(control_failure), + MovementAction::Promote { revision } | MovementAction::Finish { revision } => { + let step = if matches!(command.action, MovementAction::Promote { .. }) { + RepairStep::PromoteAndRemove + } else { + RepairStep::FinishJoint + }; + group + .advance_managed_repair( + history, + authorization, + step, + LogIndex::new(revision), + deadline.remaining()?, + ) + .await + .map(|_| ()) + .map_err(control_failure) + } + MovementAction::Activate { .. } => Err(invalid()), + }; + } + let history = catalog + .authorize_group(&snapshot, &scope, deadline.remaining()?) + .await?; match command.action { - MovementAction::Begin => cell - .begin_managed_repair( - authorization, - row.request.movement.as_ref().ok_or_else(invalid)?.plan()?, - deadline.remaining()?, - ) - .await - .map(|_| ()) - .map_err(command_failure), MovementAction::Activate { source, source_node, from_s3, - } => cell - .activate_managed_learner( + } => gateway + .activate_managed_learner_group( + history, authorization, LearnerActivation { member: permit.replacement().map_err(|_| invalid())?.member, @@ -171,23 +226,74 @@ impl FleetHostAuthorization { ) .await .map(|_| ()) - .map_err(command_failure), - MovementAction::Promote { revision } | MovementAction::Finish { revision } => { - let step = if matches!(command.action, MovementAction::Promote { .. }) { - RepairStep::PromoteAndRemove - } else { - RepairStep::FinishJoint - }; - cell.advance_managed_repair( - authorization, - step, - LogIndex::new(revision), - deadline.remaining()?, - ) - .await - .map(|_| ()) - .map_err(command_failure) + .map_err(startup_failure), + _ => Err(invalid()), + } + } +} + +fn control_failure( + error: crate::application::consensus::CellGroupControlError, +) -> FleetCatalogError { + use crate::application::consensus::CellGroupControlError; + use crate::raft::completion::RuntimeError; + match error { + CellGroupControlError::Unconfirmed(_) => FleetCatalogError::Unconfirmed, + CellGroupControlError::Refused(RuntimeError::Overloaded) => FleetCatalogError::Overloaded, + CellGroupControlError::Refused( + RuntimeError::Invalid(_) | RuntimeError::StaleCompletion, + ) => FleetCatalogError::Conflict, + CellGroupControlError::Refused(RuntimeError::Storage( + crate::raft::storage::LogError::Corrupt(_), + )) => FleetCatalogError::Corrupt("group repair state"), + CellGroupControlError::Refused(_) => FleetCatalogError::Unconfirmed, + } +} + +fn startup_failure( + error: crate::application::consensus::CellGroupStartupError, +) -> FleetCatalogError { + use crate::application::consensus::CellGroupStartupError; + use crate::raft::cell_groups::CellGroupControllerError as C; + match error { + // Entered recovery may have synced or attached even if its delivery failed. + CellGroupStartupError::Unconfirmed(_) => FleetCatalogError::Unconfirmed, + CellGroupStartupError::Refused(C::Runtime( + crate::raft::completion::RuntimeError::Overloaded, + )) => FleetCatalogError::Overloaded, + CellGroupStartupError::Refused(C::Catalog(error)) => { + use crate::raft::catalog::CatalogError as E; + match error.as_ref() { + E::Invalid(reason) => FleetCatalogError::Invalid(reason), + E::Corrupt(reason) => FleetCatalogError::Corrupt(reason), + E::Limit => FleetCatalogError::Limit, + E::Log(crate::raft::storage::LogError::Corrupt(_)) | E::LostStableState => { + FleetCatalogError::Corrupt("learner group state") + } + E::Conflict(_) | E::ManagedAdmissionRequired | E::NotInitialMember | E::Fenced => { + FleetCatalogError::Conflict + } + E::FleetAuthority(authority) => match authority.as_ref() { + FleetCatalogError::Invalid(reason) => FleetCatalogError::Invalid(reason), + FleetCatalogError::Corrupt(reason) => FleetCatalogError::Corrupt(reason), + FleetCatalogError::Overloaded => FleetCatalogError::Overloaded, + FleetCatalogError::Limit => FleetCatalogError::Limit, + FleetCatalogError::Conflict => FleetCatalogError::Conflict, + FleetCatalogError::Closed => FleetCatalogError::Closed, + // Preserve the complete typed cause for local diagnosis; + // transport delivery still classifies unknown outcomes. + _ => FleetCatalogError::Storage( + CellGroupStartupError::Refused(C::Catalog(error)).into(), + ), + }, + _ => FleetCatalogError::Storage( + CellGroupStartupError::Refused(C::Catalog(error)).into(), + ), } } + CellGroupStartupError::Refused(C::Invalid(_)) => { + FleetCatalogError::Invalid("learner group capability") + } + CellGroupStartupError::Refused(_) => FleetCatalogError::Unconfirmed, } } diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/isolated.rs b/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/isolated.rs index d8226de47..d18e1c5a4 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/isolated.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/isolated.rs @@ -206,7 +206,9 @@ impl IsolatedRepairService { let failed = NodeId::new(boot.node) .map_err(|_| FleetCatalogError::Corrupt("isolated repair boot"))?; match history.state() { - FleetHistoryState::Cancelled { .. } => return self.advance(status), + FleetHistoryState::Cancelled { .. } | FleetHistoryState::Retired { .. } => { + return self.advance(status) + } FleetHistoryState::Live => (), _ if !history .voters @@ -316,7 +318,7 @@ impl IsolatedRepairService { .and_then(|n| n.checked_add(nodes.checked_mul(512 * 1024)?)) .ok_or(FleetCatalogError::Limit)?; let _memory = - super::topology::reserve_collection_memory(&observer.handle.control_budget, bytes, 0)?; + super::topology::reserve_collection_memory(observer.control_budget(), bytes, 0)?; let group = Group { history, voters, @@ -366,7 +368,7 @@ impl IsolatedRepairService { fn block( &self, - group: GroupScope, + group: GroupHistory, error: &FleetCatalogError, status: &watch::Sender, ) { diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/mod.rs b/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/mod.rs index 146540a65..b467729fc 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/mod.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/mod.rs @@ -49,7 +49,7 @@ pub trait FleetTopologyEvidence: Send + Sync { pub struct FleetTopologyServiceInputs { pub request: FleetTopologyRequest, - pub namespaces: Vec, + pub namespaces: Vec, } /// Local diagnostics, not a convergence, membership or physical-release proof. @@ -162,7 +162,7 @@ pub struct FleetIsolatedRepairServiceStatus { pub histories_checked: u64, pub blocked_histories: u64, pub inventory_exhausted: bool, - pub last_blocked_group: Option, + pub last_blocked_group: Option, pub last_blocked_failure: Option, } diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/recovery.rs b/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/recovery.rs index 50f976894..944e2279f 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/recovery.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/recovery.rs @@ -2,7 +2,7 @@ //! Durable failure exclusion precedes a retryable trusted platform effect. use super::*; -use crate::raft::catalog::NamespaceCatalog; +use crate::raft::catalog::RangeDirectory; use crate::raft::fleet_catalog::deadline::Deadline; use crate::raft::fleet_catalog::disruption::Disruption; use crate::raft::fleet_stop::BootBinding; @@ -42,7 +42,7 @@ pub trait FleetFailureRecoveryPlatform: Send + Sync { fn namespaces<'a>( &'a self, timeout: Duration, - ) -> Pin>> + Send + 'a>>; + ) -> Pin>> + Send + 'a>>; fn isolate<'a>( &'a self, @@ -78,6 +78,7 @@ pub(super) struct Target { } pub(super) struct Recovery { + origin: Option, pub detection: super::detection::Detection, platform: Arc, evidence: Arc, @@ -93,6 +94,7 @@ impl Recovery { options: FleetFailureRecoveryOptions, ) -> Self { Self { + origin: None, detection: super::detection::Detection::new(options.detection.clone()), removal: super::removal::Removal::new(options, platform.clone()), last: None, @@ -102,6 +104,15 @@ impl Recovery { } } + pub(super) fn admit_origin(&mut self, host: &FleetHostAuthorization) -> Result<()> { + if !host.state().permits_admitted_movement() { + return Err(FleetCatalogError::Conflict); + } + self.origin = Some(host.boot_binding()?); + self.detection.set_origin(host.node()); + Ok(()) + } + async fn reconcile_completed( &mut self, catalog: &FleetCatalog, @@ -141,209 +152,226 @@ impl Recovery { .and_then(|(_, r)| r.pass_delay(options)) } - pub async fn round( - &mut self, - catalog: &FleetCatalog, - observer: &RaftCells, - cut: &FleetSnapshot, - options: &FleetMovementServiceOptions, - deadline: &Deadline<'_>, - status: &watch::Sender, - ) -> Result<()> { - let active = catalog - .run(deadline.remaining()?, async { - if catalog.inner.current().await?.token != cut.token { - return Err(FleetCatalogError::Conflict); - } - let Some(disruption) = catalog.active_disruption(cut).await? else { - return Ok(None); + pub fn round<'a>( + &'a mut self, + catalog: &'a FleetCatalog, + observer: &'a RaftCells, + cut: &'a FleetSnapshot, + options: &'a FleetMovementServiceOptions, + deadline: &'a Deadline<'_>, + status: &'a watch::Sender, + ) -> Pin> + Send + 'a>> { + // The complete replay/removal turn must not inflate every enclosing + // service poll frame. Keep one owned turn under the existing service + // admission and original deadline; cancellation drops it before join. + Box::pin(async move { + let active = catalog + .run(deadline.remaining()?, async { + if catalog.inner.current().await?.token != cut.token { + return Err(FleetCatalogError::Conflict); + } + let origin = self.origin.as_ref().ok_or(FleetCatalogError::Closed)?; + let node = NodeId::new(origin.node) + .map_err(|_| FleetCatalogError::Corrupt("recovery origin"))?; + let host = catalog.host_from(cut, node).await?; + if host.boot_binding()? != *origin || !host.state().permits_admitted_movement() + { + return Err(FleetCatalogError::Conflict); + } + let Some(disruption) = catalog.active_disruption(cut).await? else { + return Ok(None); + }; + let operation = disruption.operation(); + let Disruption::Evacuation(record) = disruption else { + return Err(FleetCatalogError::DisruptionInProgress { operation }); + }; + let boot = record + .failure_boot() + .cloned() + .ok_or(FleetCatalogError::DisruptionInProgress { operation })?; + let operation = ProposalId::new(operation) + .map_err(|_| FleetCatalogError::Corrupt("failure operation"))?; + let node = NodeId::new(boot.node) + .map_err(|_| FleetCatalogError::Corrupt("failure node"))?; + let host = catalog.host_from_record(cut, node).await?; + if host.record.operation() != Some(operation.bytes()) { + return Err(FleetCatalogError::Conflict); + } + let finalizing = match host.state() { + FleetNodeState::Draining => { + catalog.checked_active_evacuation(cut, operation).await?; + false + } + FleetNodeState::Finalizing => true, + _ => return Err(FleetCatalogError::Conflict), + }; + if host.record.boot_binding(catalog.birth_owner()?.catalog)? != boot { + return Err(FleetCatalogError::Conflict); + } + let isolation = host + .record + .isolation + .map(ProposalId::new) + .transpose() + .map_err(|_| FleetCatalogError::Corrupt("failure isolation"))?; + if isolation.is_some() { + catalog.confirm_node_isolation(cut, &host.record).await?; + } + if finalizing && isolation.is_none() { + return Err(FleetCatalogError::Conflict); + } + Ok(Some(Target { + context: FleetFailureRecoveryContext { operation, boot }, + isolation, + finalizing, + })) + }) + .await; + if matches!(&active, Err(FleetCatalogError::DisruptionInProgress { .. })) { + // A different operation may claim the slot after our terminal CAS + // committed but before its acknowledgement was reconciled. + self.reconcile_completed(catalog, cut, deadline, status) + .await?; + } + let Some(target) = active? else { + self.repair = None; + self.removal.clear(); + self.reconcile_completed(catalog, cut, deadline, status) + .await?; + self.detection + .round(catalog, observer, cut, deadline) + .await?; + let now = catalog + .inner + .domain + .mono_ms() + .map_err(|_| FleetCatalogError::Closed)?; + let view = self.detection.view(now); + let Some(failed) = view + .nodes + .iter() + .find(|n| n.health == FleetNodeHealth::Unavailable) + else { + return Ok(()); }; - let operation = disruption.operation(); - let Disruption::Evacuation(record) = disruption else { - return Err(FleetCatalogError::DisruptionInProgress { operation }); + let expires_at = now + .checked_add( + 5000u64 + .checked_sub( + failed.sample_age_ms.ok_or(FleetCatalogError::Unconfirmed)?, + ) + .ok_or(FleetCatalogError::Unconfirmed)?, + ) + .ok_or(FleetCatalogError::Limit)?; + let namespaces = self.platform.namespaces(deadline.remaining()?).await?; + let operation = identity(b"operation", &failed.boot, cut.revision())?; + let request = FleetRequestIdentity { + id: identity(b"exclusion", &failed.boot, cut.revision())?, + operation, + principal: "fleet-failure-recovery".into(), + expected_revision: cut.revision(), }; - let boot = record - .failure_boot() - .cloned() - .ok_or(FleetCatalogError::DisruptionInProgress { operation })?; - let operation = ProposalId::new(operation) - .map_err(|_| FleetCatalogError::Corrupt("failure operation"))?; - let node = NodeId::new(boot.node) - .map_err(|_| FleetCatalogError::Corrupt("failure node"))?; - let host = catalog.host_from_record(cut, node).await?; - if host.record.operation() != Some(operation.bytes()) { - return Err(FleetCatalogError::Conflict); + catalog + .begin_failure_evacuation( + cut, + &request, + (failed.boot.clone(), expires_at), + &namespaces, + deadline.remaining()?, + ) + .await?; + // Do not dispatch an external effect in the admission turn. The + // next turn must recover and authenticate the durable intent. + return Ok(()); + }; + let context = &target.context; + if self.last.as_ref() != Some(context) { + self.removal.clear(); + self.repair = None; + self.last = Some(context.clone()); + status.send_modify(|s| s.isolated_repair = None); + } + status.send_modify(|s| { + if s.failure_recovery + .as_ref() + .is_none_or(|r| r.context != *context) + { + s.failure_recovery = Some(FleetFailureRecoveryStatus { + context: context.clone(), + phase: FleetFailureRecoveryPhase::IsolationPending, + verified_histories: 0, + total_histories: 0, + removed_revision: None, + }); } - let finalizing = match host.state() { - FleetNodeState::Draining => { - catalog.checked_active_evacuation(cut, operation).await?; - false + if target.isolation.is_some() { + if let Some(r) = &mut s.failure_recovery { + if r.phase == FleetFailureRecoveryPhase::IsolationPending { + r.phase = FleetFailureRecoveryPhase::Repairing; + } } - FleetNodeState::Finalizing => true, - _ => return Err(FleetCatalogError::Conflict), - }; - if host.record.boot_binding(catalog.birth_owner()?.catalog)? != boot { - return Err(FleetCatalogError::Conflict); - } - let isolation = host - .record - .isolation - .map(ProposalId::new) - .transpose() - .map_err(|_| FleetCatalogError::Corrupt("failure isolation"))?; - if isolation.is_some() { - catalog.confirm_node_isolation(cut, &host.record).await?; } - if finalizing && isolation.is_none() { - return Err(FleetCatalogError::Conflict); + }); + if target.isolation.is_none() { + let attestation = self + .platform + .isolate(context, deadline.remaining()?) + .await?; + if attestation.boot != context.boot { + return Err(FleetCatalogError::Invalid("platform isolation boot")); } - Ok(Some(Target { - context: FleetFailureRecoveryContext { operation, boot }, - isolation, - finalizing, - })) - }) - .await; - if matches!(&active, Err(FleetCatalogError::DisruptionInProgress { .. })) { - // A different operation may claim the slot after our terminal CAS - // committed but before its acknowledgement was reconciled. - self.reconcile_completed(catalog, cut, deadline, status) - .await?; - } - let Some(target) = active? else { - self.repair = None; - self.removal.clear(); - self.reconcile_completed(catalog, cut, deadline, status) - .await?; - self.detection - .round(catalog, observer, cut, deadline) - .await?; - let now = catalog - .inner - .domain - .mono_ms() - .map_err(|_| FleetCatalogError::Closed)?; - let view = self.detection.view(now); - let Some(failed) = view - .nodes - .iter() - .find(|n| n.health == FleetNodeHealth::Unavailable) - else { + let request = FleetNodeIsolationRequest { + identity: FleetRequestIdentity { + id: identity(b"isolation", &context.boot, cut.revision())?, + operation: context.operation.bytes(), + principal: "fleet-failure-recovery".into(), + expected_revision: cut.revision(), + }, + attestation, + }; + catalog + .attest_node_isolation(cut, &request, deadline.remaining()?) + .await?; return Ok(()); - }; - let expires_at = now - .checked_add( - 5000u64 - .checked_sub(failed.sample_age_ms.ok_or(FleetCatalogError::Unconfirmed)?) - .ok_or(FleetCatalogError::Unconfirmed)?, - ) - .ok_or(FleetCatalogError::Limit)?; - let namespaces = self.platform.namespaces(deadline.remaining()?).await?; - let operation = identity(b"operation", &failed.boot, cut.revision())?; - let request = FleetRequestIdentity { - id: identity(b"exclusion", &failed.boot, cut.revision())?, - operation, - principal: "fleet-failure-recovery".into(), - expected_revision: cut.revision(), - }; - catalog - .begin_failure_evacuation( - cut, - &request, - (failed.boot.clone(), expires_at), - &namespaces, - deadline.remaining()?, - ) - .await?; - // Do not dispatch an external effect in the admission turn. The - // next turn must recover and authenticate the durable intent. - return Ok(()); - }; - let context = &target.context; - if self.last.as_ref() != Some(context) { - self.removal.clear(); - self.repair = None; - self.last = Some(context.clone()); - status.send_modify(|s| s.isolated_repair = None); - } - status.send_modify(|s| { - if s.failure_recovery + } + if target.finalizing { + // The durable finalization record is revalidated by preparation; + // the Draining-only repair admission must not gate this phase. + return self + .removal + .round(catalog, observer, cut, &target, deadline, status) + .await; + } + if self + .repair .as_ref() - .is_none_or(|r| r.context != *context) + .is_none_or(|(operation, _)| *operation != context.operation) { - s.failure_recovery = Some(FleetFailureRecoveryStatus { - context: context.clone(), - phase: FleetFailureRecoveryPhase::IsolationPending, - verified_histories: 0, - total_histories: 0, - removed_revision: None, - }); + self.repair = Some(( + context.operation, + super::isolated::IsolatedRepairService::new( + context.operation, + self.evidence.clone(), + ), + )); } - if target.isolation.is_some() { - if let Some(r) = &mut s.failure_recovery { - if r.phase == FleetFailureRecoveryPhase::IsolationPending { - r.phase = FleetFailureRecoveryPhase::Repairing; - } - } + if let Some((_, repair)) = &mut self.repair { + repair + .round(catalog, observer, cut, options, deadline, status) + .await?; } - }); - if target.isolation.is_none() { - let attestation = self - .platform - .isolate(context, deadline.remaining()?) - .await?; - if attestation.boot != context.boot { - return Err(FleetCatalogError::Invalid("platform isolation boot")); + let ready = status.borrow().isolated_repair.as_ref().is_some_and(|r| { + r.operation == context.operation + && r.inventory_exhausted + && r.blocked_histories == 0 + }); + if ready { + self.removal + .round(catalog, observer, cut, &target, deadline, status) + .await?; } - let request = FleetNodeIsolationRequest { - identity: FleetRequestIdentity { - id: identity(b"isolation", &context.boot, cut.revision())?, - operation: context.operation.bytes(), - principal: "fleet-failure-recovery".into(), - expected_revision: cut.revision(), - }, - attestation, - }; - catalog - .attest_node_isolation(cut, &request, deadline.remaining()?) - .await?; - return Ok(()); - } - if target.finalizing { - // The durable finalization record is revalidated by preparation; - // the Draining-only repair admission must not gate this phase. - return self - .removal - .round(catalog, observer, cut, &target, deadline, status) - .await; - } - if self - .repair - .as_ref() - .is_none_or(|(operation, _)| *operation != context.operation) - { - self.repair = Some(( - context.operation, - super::isolated::IsolatedRepairService::new( - context.operation, - self.evidence.clone(), - ), - )); - } - if let Some((_, repair)) = &mut self.repair { - repair - .round(catalog, observer, cut, options, deadline, status) - .await?; - } - let ready = status.borrow().isolated_repair.as_ref().is_some_and(|r| { - r.operation == context.operation && r.inventory_exhausted && r.blocked_histories == 0 - }); - if ready { - self.removal - .round(catalog, observer, cut, &target, deadline, status) - .await?; - } - Ok(()) + Ok(()) + }) } } diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/removal.rs b/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/removal.rs index d08513d01..2635ba9f2 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/removal.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/removal.rs @@ -4,30 +4,25 @@ //! the slot. A restarted worker repeats replay and current survivor checks. use super::recovery::Target; use super::*; -use crate::raft::archive::{ArchiveError, ArchiveLimits, ArchiveVerifier}; use crate::raft::fleet_catalog::deadline::Deadline; use crate::raft::fleet_catalog::{ - FleetEvacuationArchive, FleetEvacuationVerification, FleetHistoryState, - FleetIsolatedRemovalRequest, + FleetEvacuationArchive, FleetEvacuationVerification, FleetGroupArchiveVerifier, + FleetHistoryState, FleetIsolatedRemovalRequest, }; use cellar_logic::consensus::{NodeId, ProposalId}; -use cellar_ltx::LtxHost; use sha2::{Digest, Sha256}; /// Explicit replay ownership for automatic repair and terminal removal. The -/// embedding runtime owns the scratch store and its disk/memory limits. These -/// services grant no archive proof until physical verification succeeds. +/// embedding runtime owns shared private image inventories and replay pools. +/// These services grant no archive proof until complete physical replay succeeds. pub struct FleetFailureRecoveryOptions { pub detection: FleetFailureDetectionOptions, - pub archive_host: LtxHost, - pub archive_verifier: ArchiveVerifier, - pub archive_limits: ArchiveLimits, + pub archive: FleetGroupArchiveVerifier, } impl FleetFailureRecoveryOptions { pub(super) fn validate(&self) -> Result<()> { - self.detection.validate()?; - self.archive_limits.validate().map_err(archive_error) + self.detection.validate() } } @@ -39,9 +34,7 @@ struct Session { } pub(super) struct Removal { - host: LtxHost, - verifier: ArchiveVerifier, - limits: ArchiveLimits, + archive: FleetGroupArchiveVerifier, platform: Arc, session: Option, } @@ -52,9 +45,7 @@ impl Removal { platform: Arc, ) -> Self { Self { - host: options.archive_host, - verifier: options.archive_verifier, - limits: options.archive_limits, + archive: options.archive, platform, session: None, } @@ -64,185 +55,186 @@ impl Removal { self.session = None; } - pub async fn round( - &mut self, - catalog: &FleetCatalog, - observer: &RaftCells, - cut: &FleetSnapshot, - target: &Target, - deadline: &Deadline<'_>, - status: &watch::Sender, - ) -> Result<()> { - let isolation = target.isolation.ok_or(FleetCatalogError::Unconfirmed)?; - if self.session.as_ref().is_some_and(|s| { - s.snapshot.token != cut.token || s.operation != target.context.operation() - }) { - self.clear(); - } - let phase = if target.finalizing { - FleetFailureRecoveryPhase::VerifyingRemoval - } else { - FleetFailureRecoveryPhase::VerifyingFinalization - }; - status.send_modify(|s| { - if let Some(r) = &mut s.failure_recovery { - r.phase = phase; - r.verified_histories = self.session.as_ref().map_or(0, |s| s.checked); - r.total_histories = cut.head.history_count; - } - }); - let namespaces = self.platform.namespaces(deadline.remaining()?).await?; - if self.session.is_none() { - let walk = catalog - .prepare_evacuation_verification( - cut, - target.context.operation(), - &namespaces, - deadline.remaining()?, - ) - .await?; - self.session = Some(Session { - snapshot: cut.clone(), - operation: target.context.operation(), - walk, - checked: 0, - }); - } - let session = self.session.as_mut().ok_or(FleetCatalogError::Conflict)?; - if let Some(history) = session.walk.next_history(deadline.remaining()?).await? { - match history.state() { - FleetHistoryState::Cancelled { .. } => { - session.walk.verify_cancelled(deadline.remaining()?).await? - } - FleetHistoryState::Live => { - // Rooted current membership chooses candidates. The walk - // independently authenticates all three and excludes the - // failed node before accepting actual archive replay. - let (voters, _) = catalog - .run( - deadline.remaining()?, - catalog.settled_group_voters(cut, &history), - ) - .await?; - let nodes: [NodeId; 3] = voters - .into_keys() - .collect::>() - .try_into() - .map_err(|_| FleetCatalogError::InventoryIncomplete)?; - let scope = history - .binding() - .scope() - .map_err(|_| FleetCatalogError::Corrupt("recovery removal scope"))?; - let namespace = namespaces - .iter() - .find(|n| n.config().namespace == scope.namespace) - .ok_or(FleetCatalogError::InventoryIncomplete)?; - let birth = catalog - .inner - .domain - .run(namespace.birth(&scope.cell)) - .await - .map_err(|_| FleetCatalogError::Closed)?? - .ok_or(FleetCatalogError::InventoryIncomplete)?; - if birth.scope() != &scope || birth.digest() != history.birth_digest() { - return Err(FleetCatalogError::Conflict); - } - let archive = catalog - .inner - .domain - .run(async { - namespace.cell_archive(&birth, self.host.clone(), self.limits) - }) - .await - .map_err(|_| FleetCatalogError::Closed)??; - session - .walk - .freeze_live(observer, nodes, deadline.remaining()?) - .await?; - let metadata = catalog - .inner - .domain - .run(archive.recovery_metadata()) - .await - .map_err(|_| FleetCatalogError::Closed)? - .map_err(archive_error)? - .ok_or(FleetCatalogError::Unconfirmed)?; - let reference = metadata.reference(); - // The selected immutable reference must cover the fixed - // cut; archive publication may catch up on a later turn. - session - .walk - .verify_live( - observer, - nodes, - FleetEvacuationArchive { - archive: &archive, - reference: &reference, - verifier: &self.verifier, - }, - deadline.remaining()?, - ) - .await?; - } - _ => return Err(FleetCatalogError::InventoryIncomplete), + pub fn round<'a>( + &'a mut self, + catalog: &'a FleetCatalog, + observer: &'a RaftCells, + cut: &'a FleetSnapshot, + target: &'a Target, + deadline: &'a Deadline<'_>, + status: &'a watch::Sender, + ) -> Pin> + Send + 'a>> { + // Inventory and archive replay own a complete phase. Keep their future + // state out of the enclosing recovery poll frames while preserving the + // worker's existing admission, original deadline and joined lifetime. + Box::pin(async move { + let isolation = target.isolation.ok_or(FleetCatalogError::Unconfirmed)?; + if self.session.as_ref().is_some_and(|s| { + s.snapshot.token != cut.token || s.operation != target.context.operation() + }) { + self.clear(); } - session.checked = session - .checked - .checked_add(1) - .ok_or(FleetCatalogError::Limit)?; + let phase = if target.finalizing { + FleetFailureRecoveryPhase::VerifyingRemoval + } else { + FleetFailureRecoveryPhase::VerifyingFinalization + }; status.send_modify(|s| { if let Some(r) = &mut s.failure_recovery { - r.verified_histories = session.checked; + r.phase = phase; + r.verified_histories = self.session.as_ref().map_or(0, |s| s.checked); + r.total_histories = cut.head.history_count; } }); - return Ok(()); - } - let candidate = session - .walk - .publish_candidate(deadline.remaining()?) - .await?; - // Include immutable candidate bytes in the request identity. A new - // proof after controller loss cannot reuse an earlier payload's ID. - // Competing attempts at the same Root can only admit one successor. - let request = request(target, cut.revision(), candidate.digest()); - if target.finalizing { - let receipt = catalog - .complete_isolated_evacuation( - cut, - &FleetIsolatedRemovalRequest { - identity: request, - isolation, - }, - Some(&mut session.walk), - &namespaces, - deadline.remaining()?, - ) - .await?; - status.send_modify(|s| { - if let Some(r) = &mut s.failure_recovery { - r.phase = FleetFailureRecoveryPhase::Removed; - r.removed_revision = Some(receipt.applied_revision()); + let namespaces = self.platform.namespaces(deadline.remaining()?).await?; + if self.session.is_none() { + let walk = catalog + .prepare_evacuation_verification( + cut, + target.context.operation(), + &namespaces, + deadline.remaining()?, + ) + .await?; + self.session = Some(Session { + snapshot: cut.clone(), + operation: target.context.operation(), + walk, + checked: 0, + }); + } + let session = self.session.as_mut().ok_or(FleetCatalogError::Conflict)?; + if let Some(history) = session.walk.next_history(deadline.remaining()?).await? { + match history.state() { + FleetHistoryState::Cancelled { .. } => { + session.walk.verify_cancelled(deadline.remaining()?).await? + } + FleetHistoryState::Retired { .. } => { + session.walk.verify_retired(deadline.remaining()?).await? + } + FleetHistoryState::Live => { + // Rooted current membership chooses candidates. The walk + // independently authenticates all three and excludes the + // failed node before accepting actual archive replay. + let (voters, _) = catalog + .run( + deadline.remaining()?, + catalog.settled_group_voters(cut, &history), + ) + .await?; + let nodes: [NodeId; 3] = voters + .into_keys() + .collect::>() + .try_into() + .map_err(|_| FleetCatalogError::InventoryIncomplete)?; + let scope = history + .binding() + .scope() + .map_err(|_| FleetCatalogError::Corrupt("recovery removal scope"))?; + let namespace = namespaces + .iter() + .find(|n| n.namespace_config().namespace == scope.namespace) + .ok_or(FleetCatalogError::InventoryIncomplete)?; + let fixed = session + .walk + .freeze_live(observer, nodes, deadline.remaining()?) + .await?; + let registry = catalog + .run( + deadline.remaining()?, + self.archive + .open(&catalog.inner.domain, namespace, &history), + ) + .await?; + let hold = session + .walk + .acquire_archive_hold(®istry, deadline.remaining()?) + .await?; + let receiver = catalog + .run( + deadline.remaining()?, + self.archive.receiver( + &catalog.inner.domain, + fixed.root().route(), + namespace.namespace_config().image_storage, + ), + ) + .await?; + session + .walk + .verify_live( + observer, + nodes, + FleetEvacuationArchive { + registry: ®istry, + hold: &hold, + tails: self.archive.tails(), + receiver, + }, + deadline.remaining()?, + ) + .await?; + } + _ => return Err(FleetCatalogError::InventoryIncomplete), } - }); - } else { - candidate - .begin_isolated_finalization( - cut, - &request, - isolation, - &namespaces, - deadline.remaining()?, - ) + session.checked = session + .checked + .checked_add(1) + .ok_or(FleetCatalogError::Limit)?; + status.send_modify(|s| { + if let Some(r) = &mut s.failure_recovery { + r.verified_histories = session.checked; + } + }); + return Ok(()); + } + let candidate = session + .walk + .publish_candidate(deadline.remaining()?) .await?; - status.send_modify(|s| { - if let Some(r) = &mut s.failure_recovery { - r.phase = FleetFailureRecoveryPhase::VerifyingRemoval; - r.verified_histories = 0; - } - }); - } - self.clear(); - Ok(()) + // Include immutable candidate bytes in the request identity. A new + // proof after controller loss cannot reuse an earlier payload's ID. + // Competing attempts at the same Root can only admit one successor. + let request = request(target, cut.revision(), candidate.digest()); + if target.finalizing { + let receipt = catalog + .complete_isolated_evacuation( + cut, + &FleetIsolatedRemovalRequest { + identity: request, + isolation, + }, + Some(&mut session.walk), + &namespaces, + deadline.remaining()?, + ) + .await?; + status.send_modify(|s| { + if let Some(r) = &mut s.failure_recovery { + r.phase = FleetFailureRecoveryPhase::Removed; + r.removed_revision = Some(receipt.applied_revision()); + } + }); + } else { + candidate + .begin_isolated_finalization( + cut, + &request, + isolation, + &namespaces, + deadline.remaining()?, + ) + .await?; + status.send_modify(|s| { + if let Some(r) = &mut s.failure_recovery { + r.phase = FleetFailureRecoveryPhase::VerifyingRemoval; + r.verified_histories = 0; + } + }); + } + self.clear(); + Ok(()) + }) } } @@ -263,15 +255,3 @@ fn request(target: &Target, revision: u64, candidate: [u8; 32]) -> FleetRequestI principal: "fleet-failure-recovery".into(), } } - -fn archive_error(error: ArchiveError) -> FleetCatalogError { - match error { - ArchiveError::Invalid(reason) => FleetCatalogError::Invalid(reason), - ArchiveError::Corrupt(reason) => FleetCatalogError::Corrupt(reason), - ArchiveError::Conflict => FleetCatalogError::Conflict, - ArchiveError::Limit => FleetCatalogError::Limit, - ArchiveError::Unconfirmed | ArchiveError::Timeout => FleetCatalogError::Unconfirmed, - ArchiveError::Stopped => FleetCatalogError::Closed, - error => FleetCatalogError::Storage(error.into()), - } -} diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/tests.rs b/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/tests.rs index 80542b72b..dcec2be36 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/tests.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/tests.rs @@ -10,10 +10,9 @@ use cellar_logic::consensus::{ fn pending(id: u8) -> FleetPendingMovement { FleetPendingMovement { reservation: [id; 16], - group: GroupScope { + group: GroupHistory { cluster: ClusterId::new([1; 16]).unwrap(), namespace: NamespaceId::new([2; 16]).unwrap(), - cell: format!("Counter:{id}"), group: GroupId::new([id; 16]).unwrap(), incarnation: IncarnationId::new([3; 16]).unwrap(), }, diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/topology.rs b/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/topology.rs index 6acfb5347..eba74a495 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/topology.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/topology.rs @@ -245,7 +245,7 @@ impl TopologyService { ) .and_then(|n| n.checked_add(histories.checked_mul(4096)?)) .ok_or(FleetCatalogError::Limit)?; - let input = reserve_collection_memory(&observer.handle.control_budget, input_bytes, 0)?; + let input = reserve_collection_memory(observer.control_budget(), input_bytes, 0)?; let inputs = self .evidence .refresh(&operation, deadline.remaining()?) @@ -263,11 +263,8 @@ impl TopologyService { .checked_mul(28 * 1024) .and_then(|n| n.checked_add(actions.checked_mul(48)?)) .ok_or(FleetCatalogError::Limit)?; - let planning = reserve_collection_memory( - &observer.handle.control_budget, - planning_bytes, - input_bytes, - )?; + let planning = + reserve_collection_memory(observer.control_budget(), planning_bytes, input_bytes)?; let memory = Arc::new(vec![input, planning]); let verification = catalog .prepare_topology_verification( diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/worker.rs b/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/worker.rs index bfe84a8e0..8e64cc328 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/worker.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/worker.rs @@ -2,9 +2,10 @@ //! The node job set polls and joins this worker, including on physical closure. use super::*; -use crate::raft::budget::Permit; +use crate::application::consensus::group_client::{ + ControlServiceCompletion, ControlServiceKind, ControlServiceLease, +}; use crate::raft::fleet_catalog::deadline::Deadline; -use std::sync::atomic::Ordering; pub(crate) struct MovementServiceWork { catalog: FleetCatalog, @@ -12,16 +13,15 @@ pub(crate) struct MovementServiceWork { evidence: Arc, options: FleetMovementServiceOptions, stop: watch::Receiver, - done: watch::Sender, status: watch::Sender, schedule: schedule::Schedule, - admitted: bool, - memory: Option, node_stop: Option>, topology: Option, detection: Option, isolated: Option, recovery: Option, + _registration: Option, + _completion: ControlServiceCompletion, } impl MovementServiceWork { @@ -36,20 +36,19 @@ impl MovementServiceWork { ) -> Self { Self { catalog, - observer, + observer: observer.internal_observer(), evidence, options, stop, - done, status, schedule: schedule::Schedule::default(), - admitted: false, - memory: None, node_stop: None, topology: None, detection: None, isolated: None, recovery: None, + _registration: None, + _completion: ControlServiceCompletion::new(done), } } @@ -93,29 +92,18 @@ impl MovementServiceWork { if host.catalog.birth_owner()? != self.catalog.birth_owner()? { return Err(FleetCatalogError::Invalid("movement service fleet")); } - if self.observer.handle.closing.load(Ordering::Acquire) + if self.observer.is_closed() || self.observer.handle.admission.is_draining() || *self.stop.borrow() { return Err(FleetCatalogError::Closed); } - let mut slot = self - .observer - .handle - .fleet_movement_service - .lock() - .map_err(|_| FleetCatalogError::Closed)?; - if slot.is_some() { - return Err(FleetCatalogError::Overloaded); + if let Some(recovery) = &mut self.recovery { + recovery.admit_origin(host)?; } - // A retained service plus one bounded catalog turn. Transport/physical - // work separately acquires its existing request, seed and image pools. - let memory = self + let registration = self .observer - .handle - .control_budget - .acquire(2 * ROOT_LIMIT) - .ok_or(FleetCatalogError::Overloaded)?; + .register_control_service(ControlServiceKind::Movement, 2 * ROOT_LIMIT)?; // Use the enrolled node's installed client, never a caller's alternate // backend with the same descriptive fleet identity. Periodic I/O has a // private turn under the node's domain, so it cannot occupy foreground @@ -131,20 +119,14 @@ impl MovementServiceWork { turn: Mutex::new(()), }), }; - let (stop, wait) = watch::channel(false); - *slot = Some(stop); - self.memory = Some(memory); + self.node_stop = Some(registration.stop_receiver()); + self._registration = Some(registration); if let Some(detection) = &mut self.detection { detection.set_origin(host.node()); } - if let Some(recovery) = &mut self.recovery { - recovery.detection.set_origin(host.node()); - } - self.admitted = true; self.status.send_modify(|status| status.running = true); // Keep public cancellation and node drain independent. The node-owned // receiver is selected for the worker's entire lifetime, including I/O. - self.node_stop = Some(wait); Ok(()) } @@ -169,15 +151,16 @@ impl MovementServiceWork { Ok(now) => now, Err(_) => return, }; - if self.observer.handle.closing.load(Ordering::Acquire) - || self.observer.handle.admission.is_draining() - { + if self.observer.is_closed() || self.observer.handle.admission.is_draining() { return; } self.status .send_modify(|status| status.poll_rounds = status.poll_rounds.saturating_add(1)); let timeout = self.options.turn_timeout; - let result = crate::asyncrt::timeout(timeout, self.round()) + // One complete control turn belongs to this admitted worker. Keep + // its mode-specific future state off the enclosing run/select + // frames; the same timeout and stop owner retain it through await. + let result = crate::asyncrt::timeout(timeout, Box::pin(self.round())) .await .unwrap_or(Err(FleetCatalogError::Unconfirmed)); let delay = match result { @@ -210,6 +193,7 @@ impl MovementServiceWork { } self.publish_work(); let kind = schedule::classify(&error); + tracing::debug!(failure = ?kind, error = %error, "fleet service round failed"); self.status.send_modify(|status| { status.consecutive_round_failures = status.consecutive_round_failures.saturating_add(1); @@ -398,13 +382,6 @@ impl MovementServiceWork { impl Drop for MovementServiceWork { fn drop(&mut self) { - if self.admitted { - if let Ok(mut slot) = self.observer.handle.fleet_movement_service.lock() { - slot.take(); - } - } - self.memory.take(); self.status.send_modify(|status| status.running = false); - self.done.send_replace(true); } } diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/execution/work.rs b/crates/cellar/durability/raft/fleet_catalog/resources/execution/work.rs index 7292b7e7c..451fa4744 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/execution/work.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/execution/work.rs @@ -3,6 +3,7 @@ //! Reconstruct exact admitted work; cached descriptors never mint a lease. use super::*; use crate::raft::fleet_catalog::deadline::Deadline; +use crate::raft::inspection::InspectionError; use cellar_logic::consensus::MemberId; #[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] @@ -145,46 +146,58 @@ impl FleetCatalog { } } nodes.push(row.node); - for node in nodes { - let boot = match self - .run( - deadline.remaining()?, - self.movement_boot(snapshot, row, node), - ) - .await - { - Ok(boot) => boot, - Err(FleetCatalogError::Conflict) => continue, - Err(error) => return Err(error), - }; - let observed = observer - .observe_peer_fleet_membership( - node, - authorization.clone(), - snapshot.revision(), - deadline.remaining()?.min(Duration::from_millis(500)), - ) - .await; - let sample = match observed { - Ok(sample) => sample, - Err(crate::raft::inspection::InspectionError::Overloaded) => { - return Err(FleetCatalogError::Overloaded) + // A lost observation or a concurrent leader change can leave one + // sweep empty despite a live quorum. Resample once within the original + // turn budget; the caller still dispatches each action only once. + for _ in 0..2 { + let mut retryable = false; + for &node in &nodes { + let boot = match self + .run( + deadline.remaining()?, + self.movement_boot(snapshot, row, node), + ) + .await + { + Ok(boot) => boot, + Err(FleetCatalogError::Conflict) => continue, + Err(error) => return Err(error), + }; + let sample = match observer + .observe_peer_fleet_membership( + node, + authorization.clone(), + snapshot.revision(), + deadline.remaining()?.min(Duration::from_millis(500)), + ) + .await + { + Ok(sample) => sample, + Err(InspectionError::Overloaded) => return Err(FleetCatalogError::Overloaded), + Err(InspectionError::Unreachable | InspectionError::Unavailable) => { + retryable = true; + continue; + } + Err(InspectionError::Refused | InspectionError::StalePeer) => continue, + }; + if sample.sample_age_ms() > MAX_SAMPLE_AGE_MS + || sample.node().fleet_boot.as_ref() != Some(&boot) + || sample.membership().status().binding != *permit.binding() + { + return Err(FleetCatalogError::Unconfirmed); } - Err(_) => continue, - }; - if sample.sample_age_ms() > MAX_SAMPLE_AGE_MS - || sample.node().fleet_boot.as_ref() != Some(&boot) - || sample.membership().status().binding != *permit.binding() - { - return Err(FleetCatalogError::Unconfirmed); + if sample + .progress() + .member + .as_ref() + .is_some_and(|member| member.is_leader) + { + return Ok(sample); + } + retryable = true; } - if sample - .progress() - .member - .as_ref() - .is_some_and(|member| member.is_leader) - { - return Ok(sample); + if !retryable { + break; } } Err(FleetCatalogError::Unconfirmed) diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/intent/admission.rs b/crates/cellar/durability/raft/fleet_catalog/resources/intent/admission.rs index edc75cece..e8d369d32 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/intent/admission.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/intent/admission.rs @@ -12,7 +12,7 @@ impl FleetCatalog { &self, snapshot: &FleetSnapshot, request: &FleetTopologyIntentRequest, - namespaces: &[NamespaceCatalog], + namespaces: &[RangeDirectory], timeout: Duration, ) -> Result { self.check_snapshot(snapshot)?; diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/intent/mod.rs b/crates/cellar/durability/raft/fleet_catalog/resources/intent/mod.rs index f9916df4a..8e0cb8365 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/intent/mod.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/intent/mod.rs @@ -10,7 +10,7 @@ mod records; use super::*; use crate::application::consensus::RaftCells; -use crate::raft::catalog::NamespaceCatalog; +use crate::raft::catalog::RangeDirectory; use cellar_logic::consensus::ProposalId; use records::{Definition, IntentRecord, Policy}; use std::collections::BTreeSet; @@ -60,7 +60,7 @@ pub struct FleetTopologyOperation { #[derive(Clone, Debug)] pub struct FleetTopologyMovementEntry { pub reservation: [u8; 16], - pub group: GroupScope, + pub group: GroupHistory, pub source: NodeId, pub target: NodeId, pub admitted_revision: u64, diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/intent/progress.rs b/crates/cellar/durability/raft/fleet_catalog/resources/intent/progress.rs index 9f7a0bfd0..41e9fbd56 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/intent/progress.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/intent/progress.rs @@ -222,7 +222,7 @@ impl FleetCatalog { preview: &FleetTopologyPreview, identity: FleetRequestIdentity, observer: &RaftCells, - namespaces: &[NamespaceCatalog], + namespaces: &[RangeDirectory], timeout: Duration, ) -> Result> { self.check_snapshot(&preview.snapshot)?; diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/maintenance.rs b/crates/cellar/durability/raft/fleet_catalog/resources/maintenance.rs index d9d4afda4..02fea5628 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/maintenance.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/maintenance.rs @@ -123,7 +123,7 @@ impl FleetCatalog { if let Some(movement) = &row.request.movement { result.movements.push(FleetPendingMovement { reservation: row.request.identity.id, - group: group.clone(), + group, plan: movement.plan()?, }); } diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/membership.rs b/crates/cellar/durability/raft/fleet_catalog/resources/membership.rs index 27a00bc6a..227469c81 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/membership.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/membership.rs @@ -8,6 +8,87 @@ use cellar_logic::consensus::MemberId; use std::collections::{BTreeMap, BTreeSet}; impl FleetCatalog { + /// Revalidate a scheduler's source against the exact fleet root and a + /// purpose-specific quorum observation. The subsequent admission still + /// owns the topology slot, capacity checks and directory CAS. + pub(crate) async fn require_partition_source( + &self, + snapshot: &FleetSnapshot, + directory: &crate::raft::catalog::RangeDirectory, + observer: &crate::application::consensus::RaftCells, + basis: &cellar_logic::cell_group_placement::PartitionBasis, + timeout: Duration, + ) -> Result<()> { + use crate::raft::inspection::InspectionError; + let deadline = super::super::deadline::Deadline::new(self, timeout)?; + let scope = &basis.route.history; + let history = self.history(snapshot, scope, deadline.remaining()?).await?; + if !history.check_route(directory, &basis.route) + || directory.namespace_config().minimum_durability != basis.durability + { + return Err(FleetCatalogError::Conflict); + } + let (voters, _) = self + .run(deadline.remaining()?, async { + if self.inner.current().await?.token != snapshot.token { + return Err(FleetCatalogError::Conflict); + } + self.require_transition_membership_idle(snapshot).await?; + self.settled_group_voters(snapshot, &history).await + }) + .await?; + let authorization = self + .authorize_group(snapshot, scope, deadline.remaining()?) + .await?; + let expected: BTreeSet<_> = voters.values().copied().collect(); + let mut last = FleetCatalogError::Unconfirmed; + for (node, member) in voters { + let sample = match observer + .observe_peer_fleet_membership( + node, + authorization.clone(), + snapshot.revision(), + deadline.remaining()?.min(Duration::from_millis(500)), + ) + .await + { + Ok(sample) => sample, + Err(InspectionError::Overloaded) => return Err(FleetCatalogError::Overloaded), + Err(error) => { + last = FleetCatalogError::Observation(error); + continue; + } + }; + let started = self + .resource_now()? + .checked_sub(sample.sample_age_ms()) + .ok_or(FleetCatalogError::Unconfirmed)?; + self.require_resource_age(started)?; + let proof = sample.membership(); + let configuration = proof.configuration(); + if proof.replica().node != node + || proof.replica().member != member + || proof.root().route() != &basis.route + || proof.applied().term != basis.term + || proof.membership_revision().get() != basis.membership.get() + || configuration + .voters() + .iter() + .copied() + .collect::>() + != expected + || !configuration.outgoing_voters().is_empty() + || !configuration.learners().is_empty() + || !configuration.next_learners().is_empty() + || configuration.auto_leave() + { + return Err(FleetCatalogError::Conflict); + } + return Ok(()); + } + Err(last) + } + pub(in crate::raft::fleet_catalog) async fn resource_permit_authorization( &self, snapshot: &FleetSnapshot, @@ -43,7 +124,7 @@ impl FleetCatalog { // Called within an owned catalog turn. Return only the configuration pinned // by immutable birth/permit/settlement journals, never a current-health lease. - pub(in crate::raft::fleet_catalog) async fn settled_group_voters( + pub(crate) async fn settled_group_voters( &self, snapshot: &FleetSnapshot, history: &super::super::histories::FleetHistory, diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/mod.rs b/crates/cellar/durability/raft/fleet_catalog/resources/mod.rs index b174d374f..b4f17b03c 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/mod.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/mod.rs @@ -54,9 +54,17 @@ pub(super) enum Authority { RestartReturn, RestartResume, RestartOperator, + ManagedPark, + ManagedTransition, + ManagedTransferChildren, + ManagedTransferCompletion, + ManagedRecoveryReservation, + ManagedRecoveryRetirement, + ManagedRecoverySettlement, } mod movement; mod records; +mod recovery; mod topology; pub use topology::{ @@ -68,12 +76,31 @@ pub use movement::{FleetMovementAdmission, FleetMovementRequest}; use super::*; use crate::raft::fleet_stop::BootBinding; -use cellar_logic::consensus::{GroupScope, NodeId}; +use cellar_logic::cell_group::GroupHistory; +use cellar_logic::consensus::NodeId; use cellar_logic::raft_fleet::{LedgerError, Resources}; use serde::Deserialize; // Initial execution profile: two active membership moves fleet-wide and at // most two on any node. Queued plan entries are not active resource admissions. +impl FleetCatalog { + pub(super) async fn charge_recovery_movement_slots( + &self, + snapshot: &FleetSnapshot, + slots: &mut std::collections::BTreeMap, + ) -> Result<()> { + let book = self.resource_book(snapshot).await?; + for record in self.resource_rows(snapshot, &book).await? { + if let Some(count) = slots.get_mut(&record.node) { + *count = count + .checked_add(record.request.peak.engine_slots) + .ok_or(FleetCatalogError::Limit)?; + } + } + Ok(()) + } +} + const MAX_MOVES: usize = 2; const MAX_SAMPLE_AGE_MS: u64 = 5000; const BOOK_KEY: &str = "resource-reservations"; @@ -140,7 +167,7 @@ impl FleetResourceAttestation { #[derive(Clone, Debug, PartialEq, Eq)] pub struct FleetResourceRequest { pub identity: FleetRequestIdentity, - pub group: GroupScope, + pub group: GroupHistory, pub attestation: FleetResourceAttestation, /// Conservative incremental learner bound, including seed, image, tail /// and worker obligations. Actual host admission can still refuse it. diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/movement.rs b/crates/cellar/durability/raft/fleet_catalog/resources/movement.rs index 955de3873..81076a500 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/movement.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/movement.rs @@ -172,7 +172,7 @@ impl FleetCatalog { snapshot: &FleetSnapshot, request: &FleetMovementRequest, observer: &RaftCells, - namespaces: &[crate::raft::catalog::NamespaceCatalog], + namespaces: &[crate::raft::catalog::RangeDirectory], timeout: Duration, ) -> Result { self.reserve_movement_inner(snapshot, request, observer, namespaces, false, timeout) @@ -184,7 +184,7 @@ impl FleetCatalog { snapshot: &FleetSnapshot, request: &FleetMovementRequest, observer: &RaftCells, - namespaces: &[crate::raft::catalog::NamespaceCatalog], + namespaces: &[crate::raft::catalog::RangeDirectory], topology: bool, timeout: Duration, ) -> Result { diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/records.rs b/crates/cellar/durability/raft/fleet_catalog/resources/records.rs index 6e903af3d..e4b85e87a 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/records.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/records.rs @@ -267,6 +267,17 @@ impl Book { } impl FleetCatalog { + pub(in crate::raft::fleet_catalog) async fn require_transition_membership_idle( + &self, + snapshot: &FleetSnapshot, + ) -> Result<()> { + let book = self.resource_book(snapshot).await?; + if !book.rows.is_empty() { + return Err(FleetCatalogError::Conflict); + } + Ok(()) + } + pub(in super::super) async fn check_resource_mutation( &self, snapshot: &FleetSnapshot, diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/recovery.rs b/crates/cellar/durability/raft/fleet_catalog/resources/recovery.rs new file mode 100644 index 000000000..12d2e6552 --- /dev/null +++ b/crates/cellar/durability/raft/fleet_catalog/resources/recovery.rs @@ -0,0 +1,68 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Keep permanent movement admissions in the required recovery fence scope. +use super::records::Record; +use super::*; +use crate::raft::fleet_catalog::managed_recovery_inventory::RecoveryLineages; +use cellar_logic::consensus::MemberId; +use std::collections::BTreeMap; + +impl FleetCatalog { + pub(in crate::raft::fleet_catalog) async fn collect_recovery_movements( + &self, + snapshot: &FleetSnapshot, + groups: &BTreeMap, + lineages: &mut RecoveryLineages, + ) -> Result<()> { + let operations = snapshot.head.tables[FleetTable::Operations.slot()]; + let mut cursor = None; + loop { + let page = self.inner.scan(operations, cursor.as_ref(), 32).await?; + for entry in page.entries { + if !entry.key.starts_with(ROW_PREFIX) { + continue; + } + let candidate = Record::decode(self, &entry.value)?; + let Some(&history) = groups.get(&candidate.request.group) else { + continue; + }; + let id = candidate.request.identity.id; + if entry.key != row_key(id) { + return Err(FleetCatalogError::Corrupt("recovery movement key")); + } + if candidate.request.movement.is_none() { + continue; + } + let admitted = self.admitted_resource(snapshot, id).await?; + let replacement = admitted + .request + .movement + .as_ref() + .ok_or(FleetCatalogError::Corrupt("recovery movement plan"))? + .plan()? + .replacement; + lineages.include( + history, + replacement.member, + replacement.node, + Some(replacement.disk), + )?; + for owner in admitted + .seed_owners + .as_ref() + .ok_or(FleetCatalogError::Corrupt("recovery movement owners"))? + { + let node = NodeId::new(owner.node) + .map_err(|_| FleetCatalogError::Corrupt("recovery seed node"))?; + let member = MemberId::new(owner.member) + .map_err(|_| FleetCatalogError::Corrupt("recovery seed member"))?; + lineages.include(history, member, node, None)?; + } + } + let Some(next) = page.next else { + return Ok(()); + }; + cursor = Some(next); + } + } +} diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/repair/membership.rs b/crates/cellar/durability/raft/fleet_catalog/resources/repair/membership.rs index 4b5546441..789a5f191 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/repair/membership.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/repair/membership.rs @@ -62,9 +62,7 @@ impl FleetCatalog { .current_enrollment .as_ref() .ok_or(FleetCatalogError::Unconfirmed)?; - let published = sample - .published_cut() - .ok_or(FleetCatalogError::Unconfirmed)?; + let published = sample.published_cut(); let expected = group.voters.values().copied().collect::>(); let settled = match &group.latest { None => { diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/repair/mod.rs b/crates/cellar/durability/raft/fleet_catalog/resources/repair/mod.rs index 039e02c38..f49d388fa 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/repair/mod.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/repair/mod.rs @@ -5,7 +5,7 @@ mod membership; mod planning; use super::*; -use crate::raft::catalog::NamespaceCatalog; +use crate::raft::catalog::RangeDirectory; use cellar_logic::consensus::ProposalId; pub(super) use planning::Planning; use std::pin::Pin; @@ -14,7 +14,7 @@ use std::pin::Pin; pub struct FleetIsolatedRepairContext { pub operation: ProposalId, pub failed_boot: BootBinding, - pub group: GroupScope, + pub group: GroupHistory, } /// Conservative deployment inputs for one history and at most 64 candidate or @@ -23,7 +23,7 @@ pub struct FleetIsolatedRepairContext { pub struct FleetIsolatedRepairInputs { pub history: FleetTopologyHistoryResources, pub nodes: Vec, - pub namespaces: Vec, + pub namespaces: Vec, } /// The owned service cancels/joins the callback future with its original round @@ -54,7 +54,7 @@ impl FleetCatalog { cut: &FleetSnapshot, selected: &Selected, observer: &crate::application::consensus::RaftCells, - namespaces: &[NamespaceCatalog], + namespaces: &[RangeDirectory], timeout: Duration, ) -> Result { let mut canonical = super::records::Request::new(self, &selected.request.resources)?; diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/repair/planning.rs b/crates/cellar/durability/raft/fleet_catalog/resources/repair/planning.rs index d20cf1333..25c59346b 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/repair/planning.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/repair/planning.rs @@ -170,11 +170,7 @@ impl Planning<'_, '_> { .inventory .as_ref() .ok_or(FleetCatalogError::InventoryIncomplete)?; - if inventory - .history(&context.group) - .map_err(|_| FleetCatalogError::Corrupt("isolated repair physical scope"))? - .is_some() - { + if inventory.group_history(&context.group).is_some() { occupied.insert(node); } let observed_ms = catalog @@ -203,10 +199,8 @@ impl Planning<'_, '_> { if let Some(initializing) = inventory.initializing() { let mut attributed = false; for reservation in &charged.reservations { - attributed |= - crate::raft::physical_inventory::scope_digest(&reservation.group).map_err( - |_| FleetCatalogError::Corrupt("isolated repair initializer scope"), - )? == initializing; + attributed |= crate::raft::physical_inventory::group_digest(&reservation.group) + == initializing; } if !attributed { if group.voters.contains_key(&node) { @@ -270,7 +264,7 @@ impl Planning<'_, '_> { max_work: 1_000_000, }); let view = IsolatedRepairView { - scope: context.group.clone(), + scope: context.group, revision: sample.membership().membership_revision(), high_water: sample.membership().member_high_water(), voters: voters @@ -318,7 +312,7 @@ impl Planning<'_, '_> { request: FleetMovementRequest { resources: FleetResourceRequest { identity, - group: context.group.clone(), + group: context.group, attestation, peak: bounds.peak, }, diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/topology/inputs.rs b/crates/cellar/durability/raft/fleet_catalog/resources/topology/inputs.rs index c6763d161..7469cad85 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/topology/inputs.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/topology/inputs.rs @@ -115,22 +115,23 @@ impl FleetTopologyVerification { let live = self .groups .iter() - .map(|group| { - crate::raft::physical_inventory::scope_digest(&group.resources.group) - .map_err(|_| FleetCatalogError::Corrupt("topology scope digest")) + .map(|group| crate::raft::physical_inventory::group_digest(&group.resources.group)) + .collect::>(); + let known = |inventory: &crate::raft::physical_inventory::RaftPhysicalInventory| { + inventory.histories().iter().all(|row| { + let scope = row.scope_digest(); + live.contains(&scope) + || self.retired.get(&scope) == Some(&row.birth_digest()) + && (row.parked_retired_source() || !row.in_configuration()) }) - .collect::>>()?; + }; for node in &self.nodes { let original = node .physical .inventory .as_ref() .ok_or(FleetCatalogError::InventoryIncomplete)?; - if original - .histories() - .iter() - .any(|h| !live.contains(&h.scope_digest())) - { + if !known(original) { return Err(FleetCatalogError::InventoryIncomplete); } let observed = self @@ -143,7 +144,9 @@ impl FleetTopologyVerification { .await .map_err(FleetCatalogError::Observation)?; check_node(&node.host, &node.capacity, &observed)?; - if identity_digest(&observed)? != identity_digest(&node.physical)? { + if !observed.inventory.as_deref().is_some_and(&known) + || identity_digest(&observed)? != identity_digest(&node.physical)? + { return Err(FleetCatalogError::Conflict); } let now = catalog.resource_now()?; diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/topology/membership.rs b/crates/cellar/durability/raft/fleet_catalog/resources/topology/membership.rs index 787c2ba14..a3011b04d 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/topology/membership.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/topology/membership.rs @@ -43,10 +43,7 @@ impl FleetTopologyVerification { .chain(&history.learners) .any(|m| m.node == input.host.node().bytes()) || voters.contains_key(&input.host.node())) - && inventory - .history(&scope) - .map_err(|_| FleetCatalogError::Corrupt("topology genesis"))? - .is_none() + && inventory.group_history(&scope).is_none() { return Err(FleetCatalogError::InventoryIncomplete); } @@ -83,9 +80,7 @@ impl FleetTopologyVerification { )?; let proof = sample.membership(); let actual = proof.configuration(); - let published = sample - .published_cut() - .ok_or(FleetCatalogError::Unconfirmed)?; + let published = sample.published_cut(); let current = sample .node() .current_enrollment @@ -165,9 +160,7 @@ impl FleetTopologyVerification { .inventory .as_ref() .ok_or(FleetCatalogError::InventoryIncomplete)?; - let physical = inventory - .history(&scope) - .map_err(|_| FleetCatalogError::Corrupt("topology physical binding"))?; + let physical = inventory.group_history(&scope); if (history .voters .iter() diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/topology/mod.rs b/crates/cellar/durability/raft/fleet_catalog/resources/topology/mod.rs index 1c7d77152..3f82de471 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/topology/mod.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/topology/mod.rs @@ -7,7 +7,7 @@ mod membership; use super::*; use crate::application::consensus::RaftCells; -use crate::raft::catalog::NamespaceCatalog; +use crate::raft::catalog::RangeDirectory; use crate::raft::host::RaftPeerNodeObservation; use crate::raft::member_state::{JoiningMember, RepairPlan}; use cellar_logic::consensus::{MemberId, ProposalId}; @@ -40,7 +40,7 @@ pub struct FleetTopologyNodeResources { /// values are not measurements of image size or estimated command throughput. #[derive(Clone, Debug)] pub struct FleetTopologyHistoryResources { - pub group: GroupScope, + pub group: GroupHistory, pub resident: FleetResourceBounds, pub peak: FleetResourceBounds, } @@ -84,6 +84,7 @@ pub struct FleetTopologyVerification { nodes: Vec, bounds: BTreeMap, groups: Vec, + retired: BTreeMap<[u8; 32], [u8; 32]>, cursor: Option, exhausted: bool, seen: u64, @@ -164,7 +165,7 @@ impl FleetTopologyPreview { Ok(Some(FleetMovementRequest { resources: FleetResourceRequest { identity, - group: selected.group.clone(), + group: selected.group, attestation: attestation.clone(), peak: group.resources.peak, }, @@ -183,7 +184,7 @@ impl FleetCatalog { snapshot: &FleetSnapshot, request: &FleetTopologyRequest, observer: &RaftCells, - namespaces: &[NamespaceCatalog], + namespaces: &[RangeDirectory], timeout: Duration, ) -> Result { self.check_snapshot(snapshot)?; @@ -289,6 +290,7 @@ impl FleetCatalog { nodes, bounds, groups: Vec::new(), + retired: BTreeMap::new(), cursor: None, exhausted: false, seen: 0, @@ -407,6 +409,7 @@ impl FleetTopologyVerification { return Err(FleetCatalogError::Corrupt("topology inventory page")); } let mut group = None; + let mut retired = None; let mut transcript = self.transcript.clone(); if let Some(row) = page.entries.first() { let history = FleetHistory::decode(&row.value)?; @@ -421,6 +424,16 @@ impl FleetTopologyVerification { transcript.update(hash(&row.value)); match *history.state() { FleetHistoryState::Cancelled { .. } => (), + FleetHistoryState::Retired { .. } => { + let scope = history + .binding() + .scope() + .map_err(|_| FleetCatalogError::Corrupt("retired topology history"))?; + retired = Some(( + crate::raft::physical_inventory::group_digest(&scope), + history.birth_digest(), + )); + } FleetHistoryState::Live => { let bounds = self .bounds @@ -453,6 +466,9 @@ impl FleetTopologyVerification { if let Some(group) = group { self.groups.push(group); } + if let Some((scope, birth)) = retired { + self.retired.insert(scope, birth); + } if !page.entries.is_empty() { self.seen = self.seen.checked_add(1).ok_or(FleetCatalogError::Limit)?; } @@ -491,7 +507,7 @@ impl FleetTopologyVerification { .groups .iter() .map(|group| { - let scope = group.resources.group.clone(); + let scope = group.resources.group; let mut occupied = Vec::new(); for node in &self.nodes { if node @@ -499,8 +515,7 @@ impl FleetTopologyVerification { .inventory .as_ref() .ok_or(FleetCatalogError::InventoryIncomplete)? - .history(&scope) - .map_err(|_| FleetCatalogError::Corrupt("topology physical scope"))? + .group_history(&scope) .is_some() { occupied.push(node.host.node()); diff --git a/crates/cellar/durability/raft/fleet_inspection_auth_tests.rs b/crates/cellar/durability/raft/fleet_inspection_auth_tests.rs index 11546b960..58f3259b7 100644 --- a/crates/cellar/durability/raft/fleet_inspection_auth_tests.rs +++ b/crates/cellar/durability/raft/fleet_inspection_auth_tests.rs @@ -9,12 +9,11 @@ fn request() -> RequestHead { route.identities().unwrap(); let binding = crate::raft::member_state::FleetGroupBinding::new( cellar_logic::consensus::IncarnationId::new([80; 16]).unwrap(), - &cellar_logic::consensus::GroupScope { + &cellar_logic::cell_group::GroupHistory { cluster: cellar_logic::consensus::ClusterId::new([1; 16]).unwrap(), namespace: cellar_logic::consensus::NamespaceId::new([2; 16]).unwrap(), group: cellar_logic::consensus::GroupId::new([3; 16]).unwrap(), incarnation: cellar_logic::consensus::IncarnationId::new([4; 16]).unwrap(), - cell: "Counter:quorum".into(), }, [81; 32], ) diff --git a/crates/cellar/durability/raft/fleet_inspection_transport.rs b/crates/cellar/durability/raft/fleet_inspection_transport.rs index 667990ba3..c7f18cf26 100644 --- a/crates/cellar/durability/raft/fleet_inspection_transport.rs +++ b/crates/cellar/durability/raft/fleet_inspection_transport.rs @@ -279,7 +279,7 @@ impl State { .checked_add(request.timeout_ms) .ok_or(TransportError::Limit)?; let handle = self - .query_gateway + .fleet_gateway .lock() .map_err(|_| TransportError::Closed)? .as_ref() @@ -290,7 +290,7 @@ impl State { .checked_sub(self.domain.mono_ms().map_err(|_| TransportError::Closed)?) .filter(|ms| *ms > 0) .ok_or(TransportError::Timeout)?; - let work = RaftCells { handle }.observe_local_fleet_membership( + let work = RaftCells::from_internal_handle(handle).observe_local_fleet_membership( request.history.clone(), request.route.minimum_revision()?, std::time::Duration::from_millis(remaining), ); let mut extra = [0; 1]; diff --git a/crates/cellar/durability/raft/fleet_inspection_wire.rs b/crates/cellar/durability/raft/fleet_inspection_wire.rs index 402b4686c..67d08f07b 100644 --- a/crates/cellar/durability/raft/fleet_inspection_wire.rs +++ b/crates/cellar/durability/raft/fleet_inspection_wire.rs @@ -1,24 +1,24 @@ -//! Quorum metadata binds one rooted history to one authenticated node route. -//! The body reuses the bounded node and membership inspection codecs. +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Authenticated fresh quorum metadata for a complete physical group. The body +//! never chooses a cell image and cannot be decoded as an ordinary inspection. +use super::cell_groups::{CellGroupQuorumMembership, CellGroupStatus, GroupManifest}; use super::fleet_catalog::FleetGroupAuthorization; use super::host::{FleetMembershipObservation, FleetPublishedCut, RaftHostObservation}; -use super::inspection::{CellInspection, InspectionSnapshot}; -use super::member_state::FleetGroupBinding; +use super::inspection::{GroupInspection, LeaderLoad, MemberInspection}; +use super::member_state::{FleetGroupBinding, GroupMembership}; use super::transport::TransportError; use super::{inspection_wire, node_inspection_wire}; -use cellar_logic::consensus::{GroupScope, LogIndex}; +use cellar_logic::{ + cell_group::GroupHistory, + consensus::{EngineInstanceId, LogIndex, LogPosition, MemberId, ReplicaIdentity, Term}, +}; +use raft::prelude::ConfState; use serde::{Deserialize, Serialize}; -#[path = "fleet_cut_wire.rs"] -mod publication; - -pub(crate) const MAX_BODY: usize = - 20 + node_inspection_wire::MAX_BODY + inspection_wire::MAX_BODY + publication::MAX_BODY; -pub(crate) const RESPONSE_BYTES: usize = 128 * 1024; - -#[cfg(test)] -#[path = "fleet_inspection_wire_tests.rs"] -mod tests; +pub(crate) const MAX_BODY: usize = 12 + node_inspection_wire::MAX_BODY + inspection_wire::MAX_BODY; +pub(crate) const RESPONSE_BYTES: usize = super::physical_inventory::RESPONSE_BYTES; +const DOMAIN: &str = "cellar-fleet-group-quorum-1"; #[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] #[serde(deny_unknown_fields)] @@ -38,7 +38,7 @@ impl History { } } - pub(crate) fn scope(&self) -> Result { + pub(crate) fn scope(&self) -> Result { if self.birth == [0; 32] || self.schema == [0; 32] { return Err(TransportError::Identity); } @@ -50,33 +50,90 @@ impl History { } } +#[derive(Clone)] +pub(crate) struct GroupSnapshot { + pub replica: ReplicaIdentity, + pub engine: EngineInstanceId, + pub root: GroupManifest, + pub barrier: LogPosition, + pub membership: GroupMembership, + pub configuration: ConfState, + pub status: CellGroupStatus, + pub load: Option, +} + +impl GroupSnapshot { + pub(crate) fn capture(selected: &CellGroupQuorumMembership) -> Self { + Self { + replica: selected.replica().clone(), + engine: selected.engine(), + root: selected.root().clone(), + barrier: selected.barrier(), + membership: selected.membership().clone(), + configuration: selected.configuration().clone(), + status: selected.status().clone(), + load: None, + } + } + + pub(crate) fn validate(&self) -> Result<(), TransportError> { + self.root.check().map_err(|_| TransportError::Protocol)?; + self.membership + .validate(&self.configuration, self.root.position().index.get()) + .map_err(|_| TransportError::Protocol)?; + self.membership + .validate_scope(&self.replica.scope) + .map_err(|_| TransportError::Identity)?; + let fleet = self + .membership + .fleet_status() + .ok_or(TransportError::Identity)?; + let leader = self + .status + .leader + .and_then(|id| MemberId::new(id).ok()) + .ok_or(TransportError::Protocol)?; + if self.root.route().history != self.replica.scope + || !fleet.binding.matches_scope(&self.replica.scope) + || self.root.membership_revision() != self.membership.revision().get() + || self.barrier.index.get() == 0 + || self.barrier.term.get() == 0 + || self.barrier.index < self.membership.revision() + || self.root.position().index < self.barrier.index + || self.root.position().term != self.barrier.term + || self.status.term != self.barrier.term + || self.status.applied != self.root.position() + || self.status.committed < self.root.position().index + || !self.membership.permits_group(&self.replica) + || !(self.configuration.voters.contains(&leader.get()) + || self.configuration.voters_outgoing.contains(&leader.get())) + || self.status.ready && self.status.leader != Some(self.replica.member.get()) + || !self.root.is_active() + { + return Err(TransportError::Identity); + } + Ok(()) + } +} + pub(crate) struct LocalObservation { pub node: Box, - pub snapshot: Box, - pub barrier: LogIndex, + pub snapshot: GroupSnapshot, } impl LocalObservation { - fn validate(&self) -> Result<(), TransportError> { + pub(crate) fn validate(&self) -> Result<(), TransportError> { node_inspection_wire::validate(&self.node)?; - self.snapshot - .validate() - .map_err(|_| TransportError::Protocol)?; - let view = &self.snapshot.view; - if view.node.identity != self.node.node.identity - || view.node.storage_instance != self.node.node.storage_instance - || self.barrier.get() == 0 - || self.snapshot.publication.is_none() - || view - .member - .as_ref() - .is_none_or(|m| m.applied.index < self.barrier) - || self - .snapshot - .membership - .as_ref() - .and_then(|m| m.fleet_status()) - .is_none() + self.snapshot.validate()?; + let s = &self.snapshot; + if s.replica.node != self.node.node.identity.node + || s.replica.disk != self.node.node.identity.disk + || s.replica.process != self.node.node.identity.process + || s.engine != self.node.node.storage_instance + || s.load.as_ref().is_some_and(|load| { + load.capacity != self.node.node.profile.leader_capacity + || load.leaders > self.node.node.groups + }) { return Err(TransportError::Identity); } @@ -87,97 +144,192 @@ impl LocalObservation { self, authorization: &FleetGroupAuthorization, sample_age_ms: u64, - ) -> Result { + ) -> Result { self.validate()?; - let membership = FleetMembershipObservation::from_quorum_snapshot( - &self.snapshot, - authorization, - self.barrier, - ) - .map_err(|_| TransportError::Identity)?; - let published = FleetPublishedCut::capture( - &membership, - self.node.node.storage_instance, - self.snapshot - .publication - .as_ref() - .ok_or(TransportError::Protocol)?, - ); - Ok(super::host::RaftPeerFleetMembershipObservation::new( - *self.node, + let s = self.snapshot; + let progress = GroupInspection { + history: s.replica.scope, + schema_digest: s.root.route().schema_digest, + birth_digest: s.root.route().birth_digest, + node: self.node.node.clone(), + member: Some(MemberInspection { + member: s.replica.member, + term: s.status.term, + leader: s + .status + .leader + .map(|id| MemberId::new(id).map_err(|_| TransportError::Protocol)) + .transpose()?, + is_leader: s.status.leader == Some(s.replica.member.get()), + membership_revision: s.membership.revision(), + member_high_water: s.membership.high_water(), + committed: s.status.committed, + applied: s.status.applied, + published: Some(s.root.position()), + repair: s + .membership + .repair_status() + .map_err(|_| TransportError::Protocol)?, + sample_age_ms: 0, + }), + leader_load: s.load.clone(), + }; + let engine = s.engine; + let membership = FleetMembershipObservation::from_quorum_group(s, authorization) + .map_err(|_| TransportError::Identity)?; + let published = FleetPublishedCut::capture_group(&membership, engine); + let mut observed = RaftPeerFleetMembershipObservation { + node: *self.node, membership, - self.snapshot.view, + progress, published, - sample_age_ms, - )) + sample_age_ms: 0, + }; + observed + .record_delivery_age(sample_age_ms) + .map_err(|_| TransportError::Limit)?; + Ok(observed) } } +fn nullable<'de, T: Deserialize<'de>, D: serde::Deserializer<'de>>( + decoder: D, +) -> Result, D::Error> { + Option::deserialize(decoder) +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Body { + domain: String, + root: GroupManifest, + member: String, + barrier_index: String, + barrier_term: String, + committed: String, + #[serde(deserialize_with = "nullable")] + leader: Option, + ready: bool, + membership: GroupMembership, + configuration: Vec, + #[serde(deserialize_with = "nullable")] + load: Option, +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Load { + leaders: u32, + capacity: u32, + age: u64, +} + pub(crate) fn encode(observation: &LocalObservation) -> Result, TransportError> { observation.validate()?; let node = node_inspection_wire::encode(&observation.node)?; - let group = inspection_wire::encode(&observation.snapshot)?; - let mut bytes = Vec::with_capacity(20 + node.len() + group.len()); - bytes.extend_from_slice(&1_u32.to_be_bytes()); - bytes.extend_from_slice(&(node.len() as u32).to_be_bytes()); - bytes.extend_from_slice(&(group.len() as u32).to_be_bytes()); - bytes.extend_from_slice(&observation.barrier.get().to_be_bytes()); + let s = &observation.snapshot; + let group = serde_json::to_vec(&Body { + domain: DOMAIN.into(), + root: s.root.clone(), + member: s.replica.member.get().to_string(), + barrier_index: s.barrier.index.get().to_string(), + barrier_term: s.barrier.term.get().to_string(), + committed: s.status.committed.get().to_string(), + leader: s.status.leader.map(|id| id.to_string()), + ready: s.status.ready, + membership: s.membership.clone(), + configuration: super::manifest::encode_conf(&s.configuration) + .map_err(|_| TransportError::Protocol)?, + load: s.load.as_ref().map(|l| Load { + leaders: l.leaders, + capacity: l.capacity, + age: l.sample_age_ms, + }), + }) + .map_err(|_| TransportError::Protocol)?; + if group.len() > inspection_wire::MAX_BODY { + return Err(TransportError::Limit); + } + let mut bytes = Vec::with_capacity(12 + node.len() + group.len()); + for word in [1, node.len() as u32, group.len() as u32] { + bytes.extend_from_slice(&word.to_be_bytes()); + } bytes.extend_from_slice(&node); bytes.extend_from_slice(&group); - bytes.extend_from_slice(&publication::encode( - observation - .snapshot - .publication - .as_ref() - .ok_or(TransportError::Protocol)?, - )); Ok(bytes) } pub(crate) fn decode(bytes: &[u8]) -> Result { - if bytes.len() < 20 || bytes.len() > MAX_BODY { + if bytes.len() < 12 || bytes.len() > MAX_BODY { return Err(TransportError::Limit); } - // The fixed prefix is checked above before reading any word or lengths. - let word = |start| { - u32::from_be_bytes([ - bytes[start], - bytes[start + 1], - bytes[start + 2], - bytes[start + 3], - ]) - }; - let node = word(4) as usize; - let group = word(8) as usize; + let word = |i| u32::from_be_bytes([bytes[i], bytes[i + 1], bytes[i + 2], bytes[i + 3]]); + let (node_len, group_len) = (word(4) as usize, word(8) as usize); if word(0) != 1 - || node > node_inspection_wire::MAX_BODY - || group > inspection_wire::MAX_BODY - || 20 + node + group >= bytes.len() - || bytes.len() - (20 + node + group) > publication::MAX_BODY + || node_len > node_inspection_wire::MAX_BODY + || group_len > inspection_wire::MAX_BODY + || 12 + node_len + group_len != bytes.len() { return Err(TransportError::Protocol); } - let mut snapshot = inspection_wire::decode(&bytes[20 + node..20 + node + group])?; - snapshot.publication = Some(publication::decode(&bytes[20 + node + group..])?); - let observation = LocalObservation { - node: node_inspection_wire::decode(&bytes[20..20 + node])?, - snapshot, - barrier: LogIndex::new(u64::from_be_bytes([ - bytes[12], bytes[13], bytes[14], bytes[15], bytes[16], bytes[17], bytes[18], bytes[19], - ])), + let node = node_inspection_wire::decode(&bytes[12..12 + node_len])?; + let body: Body = + serde_json::from_slice(&bytes[12 + node_len..]).map_err(|_| TransportError::Protocol)?; + if body.domain != DOMAIN || body.configuration.len() > 4096 { + return Err(TransportError::Protocol); + } + let positive = |value: &str| { + let n = super::transport_frame::decimal(value)?; + if n == 0 || n >= u64::MAX - 2 { + return Err(TransportError::Protocol); + } + Ok(n) + }; + let barrier = LogPosition { + index: LogIndex::new(positive(&body.barrier_index)?), + term: Term::new(positive(&body.barrier_term)?), + }; + let leader = body.leader.as_deref().map(positive).transpose()?; + let snapshot = GroupSnapshot { + replica: ReplicaIdentity { + scope: body.root.route().history, + member: MemberId::new(positive(&body.member)?).map_err(|_| TransportError::Identity)?, + node: node.node.identity.node, + disk: node.node.identity.disk, + process: node.node.identity.process, + }, + engine: node.node.storage_instance, + status: CellGroupStatus { + term: barrier.term, + leader, + applied: body.root.position(), + committed: LogIndex::new(positive(&body.committed)?), + ready: body.ready, + }, + root: body.root, + barrier, + membership: body.membership, + configuration: super::manifest::decode_conf(&body.configuration) + .map_err(|_| TransportError::Protocol)?, + load: body.load.map(|l| LeaderLoad { + leaders: l.leaders, + capacity: l.capacity, + sample_age_ms: l.age, + }), }; + let observation = LocalObservation { node, snapshot }; observation.validate()?; Ok(observation) } -/// Opaque authenticated result. Keeping it does not extend its freshness or -/// confer placement, reservation-release or removal authority. +/// Opaque result from the purpose-specific authenticated safe-ReadIndex lane. +/// Retaining it neither extends freshness nor permits mutation or reclamation. #[derive(Clone, Debug)] pub struct RaftPeerFleetMembershipObservation { node: RaftHostObservation, membership: FleetMembershipObservation, - progress: CellInspection, - published: Option, + progress: GroupInspection, + published: FleetPublishedCut, sample_age_ms: u64, } @@ -187,38 +339,24 @@ impl RaftPeerFleetMembershipObservation { age: u64, ) -> Result<(), super::inspection::InspectionError> { use super::inspection::InspectionError; - self.sample_age_ms = self.sample_age_ms.max(age); + let next = self.sample_age_ms.max(age); + let additional = next - self.sample_age_ms; if let Some(member) = &mut self.progress.member { member.sample_age_ms = member .sample_age_ms - .checked_add(self.sample_age_ms) + .checked_add(additional) .ok_or(InspectionError::Unavailable)?; } if let Some(load) = &mut self.progress.leader_load { load.sample_age_ms = load .sample_age_ms - .checked_add(self.sample_age_ms) + .checked_add(additional) .ok_or(InspectionError::Unavailable)?; } + self.sample_age_ms = next; Ok(()) } - pub(crate) fn new( - node: RaftHostObservation, - membership: FleetMembershipObservation, - progress: CellInspection, - published: Option, - sample_age_ms: u64, - ) -> Self { - Self { - node, - membership, - progress, - published, - sample_age_ms, - } - } - pub fn node(&self) -> &RaftHostObservation { &self.node } @@ -227,20 +365,20 @@ impl RaftPeerFleetMembershipObservation { &self.membership } - pub fn progress(&self) -> &CellInspection { + pub fn progress(&self) -> &GroupInspection { &self.progress } - /// A fixed data/configuration cut only when publication matched durable - /// apply at capture. None means publication is pending; an empty cut is an - /// explicit Some value whose image() is None. Neither supplies archive or - /// current-health proof. - pub fn published_cut(&self) -> Option<&FleetPublishedCut> { - self.published.as_ref() + /// Atomic complete-root publication. The zero-cell root is explicit. + pub fn published_cut(&self) -> &FleetPublishedCut { + &self.published } - /// Whole invocation through SDK delivery, including retained reply time. pub fn sample_age_ms(&self) -> u64 { self.sample_age_ms } } + +#[cfg(test)] +#[path = "fleet_inspection_wire_tests.rs"] +mod tests; diff --git a/crates/cellar/durability/raft/fleet_inspection_wire_tests.rs b/crates/cellar/durability/raft/fleet_inspection_wire_tests.rs index 946dd8740..25bef2ecf 100644 --- a/crates/cellar/durability/raft/fleet_inspection_wire_tests.rs +++ b/crates/cellar/durability/raft/fleet_inspection_wire_tests.rs @@ -1,281 +1,180 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + use super::*; -use crate::raft::fleet_catalog::FleetNodeState; -use crate::raft::host::RaftHostEnrollmentObservation; -use crate::raft::inspection::*; -use crate::raft::member_state::GroupMembership; -use cellar_logic::consensus::*; +use cellar_logic::{ + cell_group::{CellRange, GroupRoute, RouteEpoch}, + consensus::*, +}; -fn snapshot() -> InspectionSnapshot { - let conf = ::raft::prelude::ConfState { - voters: vec![1, 2, 3], - ..Default::default() - }; - let scope = GroupScope { +fn observation(cells: bool) -> LocalObservation { + let mut node = super::super::node_inspection_wire::tests::view(); + node.node.groups = 1; + node.node.reserved_bytes = node.node.profile.bytes_per_group; + node.registered_histories = 1; + let history = GroupHistory { cluster: ClusterId::new([1; 16]).unwrap(), namespace: NamespaceId::new([2; 16]).unwrap(), - cell: "Counter:inspection".into(), group: GroupId::new([3; 16]).unwrap(), incarnation: IncarnationId::new([4; 16]).unwrap(), }; + let route = GroupRoute { + history, + range: CellRange::all(), + epoch: RouteEpoch::new(1).unwrap(), + birth_digest: [6; 32], + schema_digest: [5; 32], + }; + let mut root = serde_json::to_value(GroupManifest::genesis(route).unwrap()).unwrap(); + root["index"] = serde_json::json!(5); + root["term"] = serde_json::json!(2); + if cells { + root["root"] = serde_json::json!({"digest":vec![7;32],"height":2,"cells":2,"last_mutation_index":4,"last_mutation_term":2,"next_due_unix_millis":null}); + } + let root: GroupManifest = serde_json::from_value(root).unwrap(); + let conf = ConfState { + voters: vec![1, 2, 3], + ..Default::default() + }; let binding = - FleetGroupBinding::new(IncarnationId::new([80; 16]).unwrap(), &scope, [81; 32]).unwrap(); + FleetGroupBinding::new(IncarnationId::new([80; 16]).unwrap(), &history, [81; 32]).unwrap(); let membership = GroupMembership::managed_initial(&conf, binding).unwrap(); - InspectionSnapshot { - view: CellInspection { - scope: cellar_logic::consensus::GroupScope { - cluster: ClusterId::new([1; 16]).unwrap(), - namespace: NamespaceId::new([2; 16]).unwrap(), - cell: "Counter:inspection".into(), - group: GroupId::new([3; 16]).unwrap(), - incarnation: IncarnationId::new([4; 16]).unwrap(), - }, - schema_digest: [5; 32], - birth_digest: [6; 32], - node: NodeInspection { - identity: NodeIdentity { - node: NodeId::new([7; 16]).unwrap(), - disk: DiskLineage::new([8; 16]).unwrap(), - process: ProcessGeneration::new([9; 16]).unwrap(), - }, - storage_instance: EngineInstanceId::new([10; 16]).unwrap(), - profile: RaftNodeProfile { - pool: "native".into(), - region: "local".into(), - zone: "a".into(), - group_capacity: 8, - byte_capacity: 8192, - bytes_per_group: 1024, - leader_capacity: 8, - draining: false, - }, - groups: 1, - reserved_bytes: 1024, - }, - occupancy: ReplicaOccupancy::Member(MemberId::new(1).unwrap()), - member: Some(MemberInspection { - member: MemberId::new(1).unwrap(), - term: Term::new(2), - leader: Some(MemberId::new(1).unwrap()), - is_leader: true, - membership_revision: LogIndex::new(1), - member_high_water: 3, - committed: LogIndex::new(3), - applied: LogPosition { - index: LogIndex::new(2), - term: Term::new(1), - }, - published: Some(LogPosition { - index: LogIndex::new(1), - term: Term::new(1), - }), - repair: None, - sample_age_ms: 17, - }), - leader_load: None, + let snapshot = GroupSnapshot { + replica: ReplicaIdentity { + scope: history, + member: MemberId::new(1).unwrap(), + node: node.node.identity.node, + disk: node.node.identity.disk, + process: node.node.identity.process, }, - membership: Some(membership), - configuration: Some(conf), - publication: Some(crate::raft::host::Publication::Image( - crate::raft::host::FleetPublishedImage::new( - LogPosition { - index: LogIndex::new(1), - term: Term::new(1), - }, - cellar_ltx::apply::ImageState { - epoch: 1, - txid: cellar_ltx::TXID(1), - checksum: 0, - page_size: 4096, - length: 4096, - digest: [7; 32], - }, - ) - .unwrap(), - )), - } -} - -fn observation() -> LocalObservation { - let snapshot = Box::new(snapshot()); - let enrollment = RaftHostEnrollmentObservation { - fleet: [80; 16], - revision: 3, - epoch: 2, - physical_host: IncarnationId::new([90; 16]).unwrap(), - state: FleetNodeState::Draining, - capability_digest: [9; 32], - }; - let node = Box::new(RaftHostObservation { - fleet_boot: Some(super::super::fleet_stop::BootBinding { - catalog: [81; 32], - fleet: [80; 16], - node: snapshot.view.node.identity.node.bytes(), - disk: snapshot.view.node.identity.disk.bytes(), - process: snapshot.view.node.identity.process.bytes(), - engine: snapshot.view.node.storage_instance.bytes(), - boot: [41; 16], - epoch: 1, - join_key: crate::peer_probe::PeerProbeSigner::from_secret([4; 32]).public_key_bytes(), - }), - startup_bounds: crate::raft::host::RaftHostStartupBounds { - host_groups: 4096, - consensus_groups: 4096, - engine_groups: 4096, - bounded_buffer_bytes: 1024, + engine: node.node.storage_instance, + barrier: LogPosition { + index: LogIndex::new(5), + term: Term::new(2), + }, + status: CellGroupStatus { + term: Term::new(2), + leader: Some(1), + applied: root.position(), + committed: LogIndex::new(5), + ready: true, }, - node: snapshot.view.node.clone(), - startup_enrollment: Some(enrollment.clone()), - current_enrollment: Some(enrollment), - fleet_trust_revision: Some(3), - declared_namespaces: 1, - registered_histories: 1, - dormant_histories: 0, - pending_history_registrations: 0, - unattached_engine_histories: 0, - seed_workers: 0, - }); + root, + membership, + configuration: conf, + load: None, + }; LocalObservation { - node, + node: Box::new(node), snapshot, - barrier: LogIndex::new(2), } } #[test] -fn fleet_quorum_wire_retains_exact_configuration_and_publication() { - let view = observation(); - let restored = decode(&encode(&view).unwrap()).unwrap(); - assert_eq!(restored.node, view.node); - assert_eq!(restored.barrier, view.barrier); - assert_eq!(restored.snapshot.view, view.snapshot.view); - assert_eq!(restored.snapshot.membership, view.snapshot.membership); - assert_eq!(restored.snapshot.configuration, view.snapshot.configuration); - assert_eq!(restored.snapshot.publication, view.snapshot.publication); - assert!( - restored - .snapshot - .view - .member - .unwrap() - .published - .unwrap() - .index - < restored.barrier - ); +fn fleet_quorum_body_roundtrips_complete_group_roots_and_explicit_empty() { + for cells in [false, true] { + let observed = observation(cells); + let bytes = encode(&observed).unwrap(); + let decoded = decode(&bytes).unwrap(); + assert_eq!(*decoded.node, *observed.node); + assert_eq!(decoded.snapshot.root, observed.snapshot.root); + assert_eq!( + decoded.snapshot.root.root().cells(), + if cells { 2 } else { 0 } + ); + assert_eq!(decoded.snapshot.membership, observed.snapshot.membership); + assert_eq!( + decoded.snapshot.configuration, + observed.snapshot.configuration + ); + assert_eq!(decoded.snapshot.barrier, observed.snapshot.barrier); + assert!(bytes.len() <= MAX_BODY); + assert!(decoded.snapshot.load.is_none()); + } } #[test] -fn fleet_cut_wire_requires_explicit_coherent_publication() { - use crate::raft::host::Publication; - let mut view = observation(); - let valid = encode(&view).unwrap(); - let ordinary = crate::raft::inspection_wire::decode( - &crate::raft::inspection_wire::encode(&view.snapshot).unwrap(), - ) - .unwrap(); - assert!(ordinary.publication.is_none()); - let node = u32::from_be_bytes(valid[4..8].try_into().unwrap()) as usize; - let group = u32::from_be_bytes(valid[8..12].try_into().unwrap()) as usize; - let start = 20 + node + group; - assert!(decode(&valid[..start]).is_err()); - // Tags 2 and 3 are the distinct materialized and extent publications. - for tag in [4, 255] { - let mut malformed = valid.clone(); - malformed[start] = tag; - assert!(decode(&malformed).is_err()); - } - for offset in [1, 9, 17, 25, 45] { - let mut malformed = valid.clone(); - malformed[start + offset..start + offset + 8].fill(0); - assert!(decode(&malformed).is_err()); - } - let mut contradictory = valid[..start].to_vec(); - contradictory.push(0); - assert!(decode(&contradictory).is_err()); - view.snapshot.publication = Some(Publication::Pending); +fn fleet_configuration_commitment_is_order_independent_but_role_exact() { + let observed = observation(false); + let original = &observed.snapshot.configuration; + let mut reordered = original.clone(); + reordered.voters = vec![3, 1, 2]; assert_eq!( - decode(&encode(&view).unwrap()) + crate::raft::host::group_membership_digest(&observed.snapshot.membership, original) + .unwrap(), + crate::raft::host::group_membership_digest(&observed.snapshot.membership, &reordered) .unwrap() - .snapshot - .publication, - Some(Publication::Pending) ); - view.snapshot.view.member.as_mut().unwrap().published = None; - view.snapshot.publication = Some(Publication::Empty); assert_eq!( - decode(&encode(&view).unwrap()) + crate::raft::host::FleetMembershipConfiguration::capture(original).unwrap(), + crate::raft::host::FleetMembershipConfiguration::capture(&reordered).unwrap() + ); + reordered.voters = vec![1, 2]; + reordered.learners = vec![3]; + assert_ne!( + crate::raft::host::group_membership_digest(&observed.snapshot.membership, original) + .unwrap(), + crate::raft::host::group_membership_digest(&observed.snapshot.membership, &reordered) .unwrap() - .snapshot - .publication, - Some(Publication::Empty) ); - view.snapshot.publication = None; - assert!(encode(&view).is_err()); } #[test] -fn fleet_quorum_wire_refuses_bounds_trailing_bytes_and_impossible_barriers() { - let valid = encode(&observation()).unwrap(); - for end in 0..valid.len() { - assert!(decode(&valid[..end]).is_err()); - } - assert!(decode(&vec![0; MAX_BODY + 1]).is_err()); - let mut trailing = valid.clone(); - trailing.push(0); - assert!(decode(&trailing).is_err()); - for offset in [0, 4, 8] { - let mut changed = valid.clone(); - changed[offset..offset + 4].copy_from_slice(&u32::MAX.to_be_bytes()); - assert!(decode(&changed).is_err()); - } - for barrier in [0_u64, 3, u64::MAX] { - let mut changed = valid.clone(); - changed[12..20].copy_from_slice(&barrier.to_be_bytes()); - assert!(decode(&changed).is_err()); - } - let mut wrong_process = observation(); - wrong_process.node.node.identity.process = ProcessGeneration::new([99; 16]).unwrap(); - assert!(encode(&wrong_process).is_err()); - let mut wrong_engine = observation(); - wrong_engine.node.node.storage_instance = EngineInstanceId::new([99; 16]).unwrap(); - assert!(encode(&wrong_engine).is_err()); - let mut bare = observation(); - bare.snapshot.membership = Some(GroupMembership::initial( - bare.snapshot.configuration.as_ref().unwrap(), - )); - assert!(encode(&bare).is_err()); +fn fleet_quorum_body_rejects_mixed_boot_root_barrier_and_configuration() { + let mut observed = observation(true); + observed.snapshot.replica.process = ProcessGeneration::new([99; 16]).unwrap(); + assert!(encode(&observed).is_err()); + observed = observation(true); + observed.snapshot.barrier.index = LogIndex::new(6); + assert!(encode(&observed).is_err()); + observed = observation(true); + observed.snapshot.barrier.term = Term::new(1); + assert!(encode(&observed).is_err()); + observed = observation(true); + observed.snapshot.configuration.voters = vec![1, 4]; + assert!(encode(&observed).is_err()); + observed = observation(true); + observed.snapshot.status.leader = Some(4); + assert!(encode(&observed).is_err()); + observed = observation(true); + observed.snapshot.status.applied.index = LogIndex::new(4); + assert!(encode(&observed).is_err()); + observed = observation(true); + observed.snapshot.engine = EngineInstanceId::new([98; 16]).unwrap(); + assert!(encode(&observed).is_err()); } #[test] -fn extent_publication_round_trip_keeps_domain_in_the_existing_wire_bound() { - use crate::application::consensus::ImageCommitment; - use crate::raft::host::{FleetImageState, FleetPublishedImage, Publication}; - use cellar_ltx::internal::{ExtentRootReference, ImageRoot}; - let commitment = ImageCommitment::Extents( - ExtentRootReference::new(ImageRoot::from_bytes([7; 32]), 144).unwrap(), - ); - let image = FleetPublishedImage::new_state( - LogPosition { - index: LogIndex::new(1), - term: Term::new(1), - }, - FleetImageState::new(1, 1, 0, 4096, 4096, commitment).unwrap(), - ) - .unwrap(); - let statement = Publication::Image(image.clone()); - assert_eq!(publication::encode(&statement).len(), 85); - assert_eq!( - publication::decode(&publication::encode(&statement)).unwrap(), - statement - ); - let mut observed = observation(); - observed.snapshot.publication = Some(statement); - let decoded = decode(&encode(&observed).unwrap()).unwrap(); - assert_eq!( - decoded.snapshot.publication, - Some(Publication::Image(image)) - ); - for invalid in [vec![3], vec![3; 84], vec![3; 86]] { - assert!(publication::decode(&invalid).is_err()); +fn fleet_quorum_body_refuses_legacy_single_image_shapes_missing_fields_and_lengths() { + let original = encode(&observation(true)).unwrap(); + for len in [0, 11, 12, original.len() - 1] { + assert!(decode(&original[..len]).is_err()); + } + let mut extra = original.clone(); + extra.push(0); + assert!(decode(&extra).is_err()); + assert!(decode(&vec![0; MAX_BODY + 1]).is_err()); + let node_len = u32::from_be_bytes(original[4..8].try_into().unwrap()) as usize; + let body: serde_json::Value = serde_json::from_slice(&original[12 + node_len..]).unwrap(); + for field in [ + "root", + "membership", + "configuration", + "barrier_index", + "barrier_term", + "leader", + "load", + ] { + let mut changed = body.clone(); + changed.as_object_mut().unwrap().remove(field); + assert!(serde_json::from_value::(changed).is_err()); } + let mut old = body.clone(); + old["image"] = serde_json::json!(null); + assert!(serde_json::from_value::(old).is_err()); + let mut old_root = body; + old_root["root"]["root"] + .as_object_mut() + .unwrap() + .remove("last_mutation_index"); + assert!(serde_json::from_value::(old_root).is_err()); } diff --git a/crates/cellar/durability/raft/fleet_stop/closure.rs b/crates/cellar/durability/raft/fleet_stop/closure.rs index 4e6314a5b..81c488edc 100644 --- a/crates/cellar/durability/raft/fleet_stop/closure.rs +++ b/crates/cellar/durability/raft/fleet_stop/closure.rs @@ -142,7 +142,8 @@ impl Sink { .as_ref() .map_err(|_| FleetCatalogError::Unconfirmed)?; if raft.startup_incomplete - || !closed.unfinished.is_empty() + || closed.mutation_outcomes + != crate::application::consensus::MutationOutcomes::NoUnresolvedLocalProducers || closed.node.node.bytes() != selector.boot.node || closed.node.disk.bytes() != selector.boot.disk || closed.node.process.bytes() != selector.boot.process diff --git a/crates/cellar/durability/raft/fleet_stop/records.rs b/crates/cellar/durability/raft/fleet_stop/records.rs index 5588bf575..7f69965e2 100644 --- a/crates/cellar/durability/raft/fleet_stop/records.rs +++ b/crates/cellar/durability/raft/fleet_stop/records.rs @@ -40,7 +40,7 @@ impl BootBinding { Ok(()) } - fn same_engine_owner(&self, other: &Self) -> bool { + pub(crate) fn same_engine_owner(&self, other: &Self) -> bool { self.catalog == other.catalog && self.fleet == other.fleet && self.node == other.node @@ -180,6 +180,13 @@ pub(crate) struct Guard { } impl Guard { + /// New group attachments must order behind this exact locally synced boot. + /// Quiescence, stop acceptance and fencing close this admission as well as + /// business traffic; only a committed resume reopens the same boot. + pub(crate) fn admits_group_start(&self, boot: &BootBinding) -> bool { + self.boot == *boot && matches!(&self.phase, Phase::Open | Phase::Resumed { .. }) + } + pub fn accepted(&self) -> Option<&StopSelector> { match &self.phase { Phase::StopAccepted { selector } => Some(selector), diff --git a/crates/cellar/durability/raft/fleet_stop/reporter.rs b/crates/cellar/durability/raft/fleet_stop/reporter.rs index 74674c7f8..5c370f28e 100644 --- a/crates/cellar/durability/raft/fleet_stop/reporter.rs +++ b/crates/cellar/durability/raft/fleet_stop/reporter.rs @@ -142,6 +142,46 @@ impl Reporter { if self.remaining(deadline)? > Duration::from_secs(60) { return Err(FleetCatalogError::Invalid("stop deadline")); } + // Peer maintenance stop/resume renew proof via the survivor leader, + // which in turn inspects this node. Holding this transport waiter + // through that exchange consumes a third slot in the same two-slot + // peer metadata bank. The existing bounded worker owns proof and guard + // sync after enqueue; return uncertainty, never acceptance, until an + // exact retry observes the synced effect. + let release_peer_waiter = if origin.is_some() + && request.selector.purpose == LifecyclePurpose::Maintenance + && matches!( + request.action, + FleetStopAction::Stop | FleetStopAction::Resume + ) { + let guard = self + .log + .lifecycle_guard() + .map_err(log_error)? + .ok_or(FleetCatalogError::Corrupt("missing local guard"))?; + match request.action { + FleetStopAction::Stop => guard.accepted() != Some(&request.selector), + FleetStopAction::Resume => guard + .resumed() + .is_none_or(|(selector, _)| *selector != request.selector), + _ => false, + } + } else { + false + }; + if release_peer_waiter { + // Preserve typed authority/payload refusals before handing work to + // the bounded owner. The worker rechecks this exact context after + // queueing, then gathers fresh physical/quorum evidence before sync. + self.catalog + .verify_local_stop( + &request.selector, + request.action, + origin.as_deref(), + self.remaining(deadline)?, + ) + .await?; + } let (reply, response) = oneshot::channel(); self.sender .try_send(Job { @@ -154,6 +194,9 @@ impl Reporter { mpsc::error::TrySendError::Full(_) => FleetCatalogError::Overloaded, mpsc::error::TrySendError::Closed(_) => FleetCatalogError::Closed, })?; + if release_peer_waiter { + return Err(FleetCatalogError::Unconfirmed); + } let result = crate::asyncrt::timeout(self.remaining(deadline)?, response) .await .map_err(|_| FleetCatalogError::Unconfirmed)? diff --git a/crates/cellar/durability/raft/fleet_stop_transport.rs b/crates/cellar/durability/raft/fleet_stop_transport.rs index 5c78251a3..64fc020db 100644 --- a/crates/cellar/durability/raft/fleet_stop_transport.rs +++ b/crates/cellar/durability/raft/fleet_stop_transport.rs @@ -21,6 +21,9 @@ const REQUEST_PURPOSE: &[u8] = b"cellar-fleet-stop-request-1"; const RESPONSE_PURPOSE: &[u8] = b"cellar-fleet-stop-response-1"; const MAX_BODY: usize = 4096; +// SDK request/reply retention is separate from the transport's actual I/O banks. +pub(crate) const STOP_REQUEST_BYTES: usize = MAX_BODY + 4096; + #[derive(Serialize, Deserialize)] #[serde(deny_unknown_fields)] struct RequestHead { diff --git a/crates/cellar/durability/raft/group_host/archive.rs b/crates/cellar/durability/raft/group_host/archive.rs new file mode 100644 index 000000000..21c74cec7 --- /dev/null +++ b/crates/cellar/durability/raft/group_host/archive.rs @@ -0,0 +1,142 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Startup opens preprovisioned archive identities. An activated transfer may +//! still rely on its source archive closure while its own archive is pending. +use super::*; + +pub(super) struct Pools { + io: CellGroupArchivePool, + owners: CellGroupArchiveUploadPool, + tails: CellGroupArchiveTailPool, + checkpoints: CellGroupCheckpointPool, +} + +impl Pools { + pub fn new( + options: Option, + checkpoint: Option, + ) -> anyhow::Result> { + options + .map(|o| { + Ok(Self { + io: CellGroupArchivePool::new(o.io)?, + owners: CellGroupArchiveUploadPool::new(o.uploads)?, + tails: CellGroupArchiveTailPool::new(o.tails)?, + checkpoints: checkpoint.ok_or_else(|| { + anyhow::anyhow!("archive requires complete group checkpoints") + })?, + }) + }) + .transpose() + } + + pub async fn register( + &self, + prepared: &Prepared, + controller: &CellGroupController, + log: &super::super::storage::ReplicaLog, + directory: &std::path::Path, + limits: CellGroupNodeLimits, + options: &super::super::host::RaftHostOptions, + ) -> anyhow::Result<( + crate::application::consensus::group_client::resources::Images, + Option, + CellGroupArchiveProvisioning, + )> { + // All verifier groups share separate node-wide physical inventories. + // Serving pins or voter images cannot satisfy independent archive replay. + // The checkpoint receiver creates anonymous image spools in this + // node-owned directory. Its receive API expects an existing parent; + // initialize and sync it before any archive job can publish admission. + let scratch_directory = directory.join("archive-spools"); + let scratch = scratch_directory.clone(); + let filesystem = crate::asyncrt::fs(); + log.domain() + .blocking(move || { + cellar_ltx::host::create_dir_all_durable(filesystem.as_ref(), &scratch) + })? + .await??; + let base = directory.join("archive-verifier"); + let images = + resources::Images::open(log, base.clone(), options, &prepared.namespaces).await?; + let collectible = images.adopt_verifier(log).await?; + for assignment in prepared + .groups + .iter() + .filter(|a| a.placement == RaftGroupPlacement::Replica) + { + // The replacement closure owns the sealed source archive. A + // historical voter participates in Raft only for retirement. + if assignment.sealed_transfer_source { + continue; + } + let namespace = &prepared.namespaces[assignment.namespace]; + if assignment.pending_transfer_archive + && !namespace + .directory + .has_group_archive_binding(&assignment.birth) + .await? + { + // The verified transfer closure still owns recovery. Leave + // archive publication and explicit S3 coverage unavailable + // until the operator creates this child's archive binding. + continue; + } + namespace + .directory + .open_group_archive(&log.domain(), &self.io, &assignment.birth) + .await?; + let route = assignment.birth.route().clone(); + let path = ManifestPageStore::group_pages_path(&base, route.history); + let backend = images.apply(namespace.directory.namespace_config(), options)?; + let store = log + .domain() + .blocking(move || { + Ok::<_, anyhow::Error>(CellGroupArchiveStore::new( + route, + resources::pages(&path, limits)?, + backend, + limits.manifest, + limits.entries, + limits.max_effect_rows, + )?) + })? + .await??; + controller + .archive(CellGroupArchiveRegistration { + directory: namespace.directory.clone(), + birth: assignment.birth.clone(), + io: self.io.clone(), + owners: self.owners.clone(), + tails: self.tails.clone(), + checkpoints: self.checkpoints.clone(), + store, + scratch: log.snapshot_scratch(), + scratch_directory: scratch_directory.clone(), + })? + .response() + .await?; + } + let namespaces = prepared + .namespaces + .iter() + .map(|namespace| { + Ok(CellGroupProvisionNamespace { + directory: namespace.directory.clone(), + images: images.apply(namespace.directory.namespace_config(), options)?, + }) + }) + .collect::>>()?; + Ok(( + images.observer(), + collectible, + CellGroupArchiveProvisioning { + namespaces, + io: self.io.clone(), + owners: self.owners.clone(), + tails: self.tails.clone(), + checkpoints: self.checkpoints.clone(), + }, + )) + } +} diff --git a/crates/cellar/durability/raft/group_host/collection.rs b/crates/cellar/durability/raft/group_host/collection.rs new file mode 100644 index 000000000..cbf808a19 --- /dev/null +++ b/crates/cellar/durability/raft/group_host/collection.rs @@ -0,0 +1,201 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! One node-owned physical collection service. Reference adoption is a separate +//! startup proof; turns never select roots or relax the offline engine collector. +use super::resources::CollectibleExtents; +use crate::application::consensus::{ + group_client::{resources::CollectionObserver, ControlServiceKind, ControlServiceLease}, + CellGroupCollectionFailure, RaftCells, +}; +use crate::asyncrt::DomainToken; +use crate::raft::budget::{Budget, Permit}; +use cellar_ltx::internal::StoreError; +use std::{sync::Arc, time::Duration}; + +const FILES_PER_TURN: usize = 16; +const METADATA_BYTES: usize = 4096; + +pub(super) struct Collection { + inventories: Vec, + working_bytes: usize, + observer: CollectionObserver, +} + +// Field drop order is part of the join: release inventory handles before the +// last registry lease can advertise completed physical ownership. +struct Owner { + inventories: Vec, + _lease: ControlServiceLease, +} + +struct Turn { + _reservation: Permit, + owner: Arc, + inventory: usize, +} + +impl Collection { + pub(super) fn new(inventories: Vec) -> anyhow::Result> { + if inventories.is_empty() { + return Ok(None); + } + anyhow::ensure!( + inventories.len() <= 2, + "collection inventories exceed node roles" + ); + let working_bytes = inventories.iter().try_fold(0usize, |bound, inventory| { + inventory + .store() + .reclaim_working_bytes(FILES_PER_TURN) + .map(|n| bound.max(n)) + })?; + let observer = CollectionObserver::new(inventories.len(), working_bytes); + Ok(Some(Self { + inventories, + working_bytes, + observer, + })) + } + + pub(super) fn observer(&self) -> CollectionObserver { + self.observer.clone() + } + + pub(super) fn start(self, service: &RaftCells, poll_ms: u64) -> anyhow::Result<()> { + let budget = service.control_budget().clone(); + let (bytes, items) = budget.capacity(); + anyhow::ensure!( + items >= 2 + && self + .working_bytes + .checked_add(METADATA_BYTES) + .is_some_and(|required| required <= bytes), + "collection lacks control admission backing" + ); + let lease = + service.register_control_service(ControlServiceKind::Reclamation, METADATA_BYTES)?; + let stop = lease.stop_receiver(); + let owner = Arc::new(Owner { + inventories: self.inventories, + _lease: lease, + }); + let domain = service.handle.domain.clone(); + let observer = self.observer.clone(); + self.observer.update(|s| s.running = true); + let worker = domain.clone(); + match domain.spawn_cleanup(async move { + run( + owner, + budget, + worker, + stop, + self.observer, + self.working_bytes, + poll_ms, + ) + .await; + }) { + Ok(task) => { + task.detach(); + Ok(()) + } + Err(error) => { + failed(&observer, CellGroupCollectionFailure::Domain); + observer.update(|s| s.running = false); + Err(error.into()) + } + } + } +} + +fn physical(turn: Turn) -> Result { + // The turn, its memory and the registry lifetime remain in this physical + // closure if an async observer disappears while a filesystem call is held. + turn.owner.inventories[turn.inventory] + .store() + .reclaim(FILES_PER_TURN) +} + +fn failed(observer: &CollectionObserver, cause: CellGroupCollectionFailure) { + observer.update(|s| { + s.failures = s.failures.saturating_add(1); + s.last_failure = Some(cause); + }); +} + +async fn run( + owner: Arc, + budget: Arc, + domain: DomainToken, + mut stop: tokio::sync::watch::Receiver, + observer: CollectionObserver, + working_bytes: usize, + poll_ms: u64, +) { + 'service: loop { + let mut progress = false; + for inventory in 0..owner.inventories.len() { + if *stop.borrow() { + break 'service; + } + match owner.inventories[inventory].needs_reclaim() { + Ok(false) => continue, + Ok(true) => (), + Err(error) => { + tracing::warn!(%error, "node image collection refused inventory"); + failed(&observer, CellGroupCollectionFailure::Inventory); + break 'service; + } + } + let Some(reservation) = budget.acquire(working_bytes) else { + observer.update(|s| s.deferrals = s.deferrals.saturating_add(1)); + continue; + }; + let turn = Turn { + _reservation: reservation, + owner: owner.clone(), + inventory, + }; + let entered = match domain.blocking(move || physical(turn)) { + Ok(task) => task, + Err(_) => { + failed(&observer, CellGroupCollectionFailure::Domain); + break 'service; + } + }; + observer.update(|s| s.turns = s.turns.saturating_add(1)); + // Stop signals never cancel admitted physical work or its memory. + match entered.await { + Ok(Ok(files)) => { + progress |= files != 0; + observer.update(|s| { + s.files_reclaimed = s.files_reclaimed.saturating_add(files as u64) + }); + } + Ok(Err(StoreError::Busy)) => { + observer.update(|s| s.deferrals = s.deferrals.saturating_add(1)) + } + Ok(Err(error)) => { + tracing::warn!(%error, "node image collection stopped after physical failure"); + failed(&observer, CellGroupCollectionFailure::Inventory); + break 'service; + } + Err(_) => { + failed(&observer, CellGroupCollectionFailure::Worker); + break 'service; + } + } + } + if *stop.borrow() { + break; + } + // Drain bounded deletion batches promptly, yielding between turns. An + // idle marked inventory is checked using metadata without file I/O. + crate::asyncrt::select! { + _ = stop.changed() => break, + _ = crate::asyncrt::sleep(Duration::from_millis(if progress { 1 } else { poll_ms })) => (), + } + } + observer.update(|s| s.running = false); + drop(owner); +} diff --git a/crates/cellar/durability/raft/group_host/directory.rs b/crates/cellar/durability/raft/group_host/directory.rs new file mode 100644 index 000000000..df4738cd0 --- /dev/null +++ b/crates/cellar/durability/raft/group_host/directory.rs @@ -0,0 +1,378 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! One bounded physical-node discovery owner, joined by the shared SDK service. +//! Endpoints and rooted membership are independent of current-term Raft authority. +use super::*; +use crate::application::consensus::group_client::ControlServiceKind; +use crate::raft::{ + catalog::NodeRegistration, + fleet_catalog::{FleetHostAuthorization, FleetTrustReader, FleetTrustSnapshot}, + host::RaftNodeDirectoryOptions, + storage::ReplicaLog, + transport::RaftTransport, + transport_options::RaftPeerNode, +}; +use rand::RngCore; +use std::{collections::BTreeMap, time::Duration}; + +const METADATA_BYTES: usize = 4 * 1024 * 1024; + +#[derive(Clone)] +enum Trusted { + Configured(RaftPeerNode), + Enrolled(Box), +} + +pub(super) struct Directory { + options: RaftNodeDirectoryOptions, + registrations: Vec, + trusted: Vec, + reader: Option, + rooted: Option, + cursor: usize, + own_revision: u64, + routers: Option, +} + +pub(super) fn remaining(log: &ReplicaLog, deadline: u64) -> anyhow::Result { + let ms = deadline + .checked_sub(log.domain().mono_ms()?) + .filter(|ms| *ms > 0) + .ok_or_else(|| anyhow::anyhow!("node directory original deadline"))?; + Ok(Duration::from_millis(ms)) +} + +fn trusted(trust: &FleetTrustSnapshot) -> Vec { + trust + .nodes() + .iter() + .filter(|n| n.permits_transport()) + .cloned() + .map(|node| Trusted::Enrolled(Box::new(node))) + .collect() +} + +pub(super) async fn prepare( + prepared: &mut Prepared, + options: &mut crate::raft::host::RaftHostOptions, + log: &ReplicaLog, + transport: &RaftTransport, +) -> anyhow::Result> { + let Some(settings) = options.node_directory.clone() else { + return Ok(None); + }; + let local = log.limits(); + let mut peers: BTreeMap<_, _> = prepared + .peers + .iter() + .cloned() + .map(|p| (p.node, p)) + .collect(); + let own = peers + .get(&local.node) + .cloned() + .ok_or_else(|| anyhow::anyhow!("node directory requires an explicit self endpoint"))?; + anyhow::ensure!( + own.disk == local.disk + && own.process == local.process + && own.label == transport.local_label(), + "directory self endpoint disagrees with engine owner" + ); + for peer in peers.values() { + peer.validate_endpoint()?; + } + if let Some(managed) = &options.managed { + // Check the rooted descriptor before either a permanent storage pin or + // a boot CAS. Advertised addresses may differ from local bind addresses. + managed.authorization.check_boot_endpoint(&own)?; + } + let deadline = log + .domain() + .mono_ms()? + .checked_add(settings.timeout_ms) + .ok_or_else(|| anyhow::anyhow!("directory deadline overflow"))?; + let mut namespaces = BTreeMap::new(); + for ns in &prepared.namespaces { + namespaces + .entry(ns.directory.namespace_config().cluster) + .or_insert_with(|| ns.directory.namespace().clone()); + } + let mut registrations = Vec::new(); + for namespace in namespaces.into_values() { + registrations.push( + namespace + .register_node_storage(log, remaining(log, deadline)?) + .await?, + ); + } + let mut rooted = None; + let mut reader = None; + // Bind once for the complete host, before publishing any process endpoint or + // attaching a group. The engine update must actually sync before admission. + if let Some(managed) = options.managed.as_mut() { + let mut id = [0; 16]; + crate::asyncrt::rng("fleet-host-boot").fill_bytes(&mut id); + let id = cellar_logic::consensus::ProposalId::new(id)?; + let join_key = crate::asyncrt::services() + .peer_probe_signer()? + .public_key_bytes(); + let host = &managed.authorization; + let (bound, update) = + if host.state() == crate::raft::fleet_catalog::FleetNodeState::RestartPending { + host.bind_restart_boot(log, id, join_key, deadline, remaining(log, deadline)?) + .await? + } else { + let bound = host + .bind_boot( + log.storage_instance()?, + local.process, + id, + join_key, + remaining(log, deadline)?, + ) + .await?; + let update = crate::raft::fleet_stop::LifecycleUpdate::Boot { + boot: bound.boot_binding()?, + deadline, + }; + (bound, update) + }; + log.update_lifecycle(update).await?; + remaining(log, deadline)?; + managed.authorization = bound; + if let Some(selector) = log.lifecycle_guard()?.and_then(|g| g.quiescing().cloned()) { + crate::asyncrt::services() + .raft_lifecycle() + .install_startup_maintenance(&selector)?; + } + let trust = managed + .authorization + .trust_snapshot(remaining(log, deadline)?) + .await?; + require_local(&trust, log)?; + anyhow::ensure!( + trust.nodes().len() <= transport.peer_limit(), + "fleet exceeds transport peer bound" + ); + transport.install_fleet_trust(trust.clone())?; + reader = Some(managed.authorization.trust_reader()); + rooted = Some(trust); + } + let mut own_revision = 0; + for registration in ®istrations { + let view = registration + .publication(own.clone(), remaining(log, deadline)?) + .await? + .publish(remaining(log, deadline)?) + .await?; + own_revision = own_revision.max(view.revision()); + transport.accept_node_view(view, &own)?; + if let Some(trust) = &rooted { + for node in trust + .nodes() + .iter() + .filter(|n| n.node() != local.node && n.permits_transport()) + { + if let Some(view) = registration + .observe_fleet_node(node, remaining(log, deadline)?) + .await? + { + let peer = view.peer().clone(); + transport.accept_node_view(view, &peer)?; + peers.insert(peer.node, peer); + } + } + } + } + let selected = rooted.as_ref().map_or_else( + || peers.values().cloned().map(Trusted::Configured).collect(), + trusted, + ); + let routers = reader.as_ref().and_then(|_| { + super::router_directory::RouterDirectory::new( + prepared + .groups + .iter() + .filter(|assignment| assignment.placement == RaftGroupPlacement::QueryRouter) + .map(|assignment| assignment.birth.route().history), + ) + }); + prepared.peers = peers.into_values().collect(); + Ok(Some(Directory { + options: settings, + registrations, + trusted: selected, + reader, + rooted, + cursor: 0, + own_revision, + routers, + })) +} + +fn require_local(trust: &FleetTrustSnapshot, log: &ReplicaLog) -> anyhow::Result<()> { + let limits = log.limits(); + let boot = log + .lifecycle_guard()? + .ok_or_else(|| anyhow::anyhow!("missing synced managed boot"))? + .boot; + let host = trust + .node(limits.node) + .ok_or_else(|| anyhow::anyhow!("local node removed from fleet"))?; + anyhow::ensure!( + host.permits_transport() + && host.boot_binding()? == boot + && boot.disk == limits.disk.bytes() + && boot.process == limits.process.bytes() + && boot.engine == log.storage_instance()?.bytes(), + "local fleet boot superseded" + ); + Ok(()) +} + +impl Directory { + pub(super) fn start( + self, + service: &RaftCells, + log: ReplicaLog, + transport: RaftTransport, + ) -> anyhow::Result<()> { + let lease = + service.register_control_service(ControlServiceKind::Directory, METADATA_BYTES)?; + let mut stop = lease.stop_receiver(); + let observer = service.internal_observer(); + log.domain().spawn_cleanup(async move { + let _lease = lease; + let mut directory = self; + loop { + if *stop.borrow() { break; } + // Entered discovery work is retained until its original bounded + // operation finishes. Shutdown never drops it at a select edge. + if let Err(error) = directory.refresh(&log, &transport, &observer).await { + tracing::debug!(%error, "shared node directory fenced local owner"); + drop(observer.close()); // requests independently owned joined closure + break; + } + crate::asyncrt::select! { + _ = stop.changed() => break, + _ = crate::asyncrt::sleep(Duration::from_millis(directory.options.poll_ms)) => (), + } + } + })?.detach(); + Ok(()) + } + + async fn refresh( + &mut self, + log: &ReplicaLog, + transport: &RaftTransport, + observer: &RaftCells, + ) -> anyhow::Result<()> { + let deadline = log + .domain() + .mono_ms()? + .checked_add(self.options.timeout_ms) + .ok_or_else(|| anyhow::anyhow!("directory deadline overflow"))?; + if let Some(reader) = &self.reader { + let result = transport + .seed_control_until(deadline, async { + let timeout = remaining(log, deadline) + .map_err(|_| crate::raft::transport::TransportError::Timeout)?; + Ok(reader.refresh(self.rooted.as_ref(), timeout).await) + }) + .await; + match result { + Ok(Ok(trust)) => { + require_local(&trust, log)?; + transport.install_fleet_trust(trust.clone())?; + self.trusted = trusted(&trust); + self.rooted = Some(trust); + } + // Missing metadata is not revocation or loss of a live quorum. + Ok(Err( + error @ (crate::raft::fleet_catalog::FleetCatalogError::Conflict + | crate::raft::fleet_catalog::FleetCatalogError::Corrupt(_)), + )) => return Err(error.into()), + Ok(Err(error)) => tracing::debug!(%error, "fleet trust read unavailable"), + Err(error) => tracing::debug!(%error, "fleet trust read refused"), + } + } + let total = self + .registrations + .len() + .checked_mul(self.trusted.len()) + .ok_or_else(|| anyhow::anyhow!("directory inventory overflow"))?; + if total == 0 { + return Ok(()); + } + let mut batch = Vec::new(); + for _ in 0..self.options.batch_nodes.min(total) { + self.cursor %= total; + batch.push(( + self.registrations[self.cursor / self.trusted.len()].clone(), + self.trusted[self.cursor % self.trusted.len()].clone(), + )); + self.cursor = (self.cursor + 1) % total; + } + let results = futures_util::future::join_all(batch.into_iter().map( + |(registration, trusted)| async move { + let result = transport + .seed_control_until(deadline, async { + let timeout = remaining(log, deadline) + .map_err(|_| crate::raft::transport::TransportError::Timeout)?; + Ok(match &trusted { + Trusted::Configured(peer) => { + registration.observe_node(peer, timeout).await + } + Trusted::Enrolled(node) => { + registration.observe_fleet_node(node, timeout).await + } + }) + }) + .await; + (trusted, result) + }, + )) + .await; + for (trusted, result) in results { + let view = match result { + Ok(Ok(Some(view))) => view, + Ok(Ok(None)) => continue, + Ok(Err(error)) => { + tracing::debug!(%error, "node view unavailable"); + continue; + } + Err(error) => { + tracing::debug!(%error, "node discovery refused"); + continue; + } + }; + let peer = view.peer().clone(); + let local = log.limits(); + anyhow::ensure!( + peer.node != local.node + || (view.storage_instance() == log.storage_instance()? + && !(peer.process != local.process && view.revision() > self.own_revision)), + "local process view superseded" + ); + let expected = match &trusted { + Trusted::Configured(p) => p, + Trusted::Enrolled(_) => &peer, + }; + if let Err(error) = transport.accept_node_view(view, expected) { + tracing::debug!(%error, "node process refresh refused"); + } + } + if let (Some(routers), Some(reader), Some(trust)) = + (&mut self.routers, &self.reader, &self.rooted) + { + if let Err(error) = routers + .refresh(log, transport, observer, reader, trust, deadline) + .await + { + tracing::debug!(%error, "router membership refresh unavailable"); + } + } + Ok(()) + } +} diff --git a/crates/cellar/durability/raft/group_host/mod.rs b/crates/cellar/durability/raft/group_host/mod.rs new file mode 100644 index 000000000..6071e18e0 --- /dev/null +++ b/crates/cellar/durability/raft/group_host/mod.rs @@ -0,0 +1,603 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Canonical startup uses namespace capabilities and physical group placement. +//! Logical cells are enrolled through the controller, never through node config. +mod archive; +mod collection; +mod directory; +mod provisioning; +mod recovery; +mod registration; +mod router_directory; +pub use provisioning::{RaftGroupArchiveInitialization, RaftNamespaceProvisioning}; +mod resources; +#[cfg(all(test, not(cellar_internal_tests)))] +#[path = "../host_startup_tests.rs"] +mod startup_tests; +#[cfg(test)] +pub(crate) mod test_support; + +use super::cell_groups::*; +use crate::application::consensus::{RaftCells, RaftClass, VersionKeys}; +use cellar_logic::cell_group::GroupHistory; +use std::{path::PathBuf, sync::Arc}; + +/// An explicit, immutable initial range plan. Retrying preserves every birth +/// and the directory operation identity; startup never allocates a group per cell. +pub struct RaftNamespaceInitialization { + pub operation: cellar_logic::consensus::ProposalId, + pub groups: Vec, +} + +pub struct RaftNamespaceConfig { + pub namespace: super::catalog::NamespaceConfig, + pub class: String, + /// None opens an existing namespace only. A published initialization is + /// idempotent; it cannot replace a later topology or recovery incarnation. + pub initialize: Option, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum RaftGroupPlacement { + Replica, + QueryRouter, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct RaftGroupAssignment { + pub history: GroupHistory, + pub placement: RaftGroupPlacement, +} + +/// Explicit shared capacities. Physical SQL, archive and image admissions are +/// node-wide; manifest directories additionally have per-group disk ceilings. +#[derive(Clone, Copy)] +pub struct CellGroupNodeLimits { + pub namespaces: CellGroupHostLimits, + pub controller: CellGroupControllerLimits, + pub sql: CellSqlLimits, + pub provisioning: CellGroupProvisionLimits, + pub directory: DirectoryLimits, + pub dispatch: CellDispatchLimits, + pub manifest: ManifestLimits, + pub entries: crate::application::consensus::group_protocol::GroupEntryLimits, + pub manifest_pages: usize, + pub manifest_bytes: u64, + pub retained_roots: usize, + pub max_effect_rows: u32, + pub checkpoints: Option, + /// Bindings must have been durably provisioned in the namespace catalog. + /// Missing archive identity/counters are never initialized during restart. + pub archive: Option, +} + +#[derive(Clone, Copy)] +pub struct CellGroupNodeArchiveOptions { + pub io: CellGroupArchiveLimits, + pub uploads: CellGroupArchiveUploadLimits, + pub tails: CellGroupArchiveTailLimits, +} + +pub struct RaftCellsConfig { + pub infrastructure: crate::daemon::NodeRaftConfig, + pub catalog: super::catalog::CatalogOptions, + pub namespaces: Vec, + /// Explicit routes and placements. A managed host discovers retained local + /// replicas within its declared namespaces from the opened engine. + pub groups: Vec, + pub peers: Vec, + pub query_origins: Vec, + pub command_origins: Vec, + pub host: super::host::RaftHostOptions, + pub shared: CellGroupNodeLimits, +} + +pub(crate) struct Namespace { + directory: RangeDirectory, + class: Arc, +} + +pub(crate) struct Assignment { + namespace: usize, + birth: CellGroupBirth, + placement: RaftGroupPlacement, + pending_transfer_archive: bool, + sealed_transfer_source: bool, + transport: super::transport::GroupRoute, +} + +pub(crate) struct Prepared { + namespaces: Vec, + peers: Vec, + groups: Vec, + query_origins: Vec, + command_origins: Vec, +} + +pub(crate) use registration::prepare; + +/// Own infrastructure while assembling resources. Any error or cancelled startup +/// schedules joined closure on the same domain; SQL outlives its physical work. +struct Startup { + service: Option, + public: Option, + sql: Option, + controller: Option, +} + +struct Infrastructure { + log: super::storage::ReplicaLog, + runtime: CellGroupRuntime, + transport: super::transport::RaftTransport, +} + +impl Startup { + fn cleanup( + &mut self, + ) -> Result>, crate::asyncrt::DomainClosed> { + let Some(service) = self.service.take() else { + return Ok(None); + }; + let domain = service.log.domain(); + let public = self.public.take(); + let sql = self.sql.take(); + let controller = self.controller.take(); + domain + .spawn_cleanup(async move { + // A late admission/grant failure can follow entered control + // work. Join that owner before tearing down its dependencies. + if let Some(public) = public { + let _ = public.close().await; + } + if let Some(controller) = controller { + let _ = controller.close().await; + } else { + let _ = service.runtime.close().await; + service.transport.close().await; + if let Some(sql) = sql { + let _ = sql.close().await; + } + } + let _ = service.log.close().await; + }) + .map(Some) + } +} + +impl Drop for Startup { + fn drop(&mut self) { + if let Ok(Some(task)) = self.cleanup() { + task.detach(); + } + } +} + +pub(crate) async fn start( + service: crate::daemon::NodeRaftService, + mut prepared: Prepared, + directory: PathBuf, + mut options: super::host::RaftHostOptions, + limits: CellGroupNodeLimits, + keys: VersionKeys, + recovery_authority: Option< + Arc>, + >, +) -> anyhow::Result { + let crate::daemon::NodeRaftService { + log, + runtime, + events, + transport, + } = service; + let mut startup = Startup { + service: Some(Infrastructure { + log: log.clone(), + runtime: runtime.clone(), + transport: transport.clone(), + }), + public: None, + sql: None, + controller: None, + }; + // This initialization owns several nested catalog/recovery futures. Keep + // that state off callers' native stacks, including nested node startup. + let result = Box::pin(async { + let fleet = options + .managed + .as_ref() + .map(|managed| managed.authorization.stop_catalog()); + let retained = + registration::discover_retained_placements(&mut prepared, &log, fleet.as_ref(), limits) + .await?; + // Boot binding commits the Fleet root through the object-store client. + // Keep its nested futures off the caller's startup polling frame. + let discovery = Box::pin(directory::prepare( + &mut prepared, + &mut options, + &log, + &transport, + )) + .await?; + registration::refresh_routes(&mut prepared, &log, options.managed.is_some())?; + let sql = CellGroupSqlPool::start(&log, limits.sql)?; + startup.sql = Some(sql.clone()); + let checkpoint = limits + .checkpoints + .map(CellGroupCheckpointPool::new) + .transpose()?; + if let Some(pool) = &checkpoint { + transport.enable_group_checkpoints(pool.clone())?; + } + let images = + resources::Images::open(&log, directory.clone(), &options, &prepared.namespaces) + .await?; + let mut appliers = Vec::new(); + let mut recovering = Vec::new(); + let mut recovery_bytes = 4096usize; + for assignment in &mut prepared.groups { + match assignment.placement { + RaftGroupPlacement::QueryRouter => { + // Publish every route together after all local roots attach. + } + RaftGroupPlacement::Replica => { + let namespace = &prepared.namespaces[assignment.namespace]; + let retained = match log.cell_group(&assignment.birth.route().history) { + Ok(group) => Some(group), + Err(super::storage::LogError::NotFound) => None, + Err(error) => return Err(error.into()), + }; + let joined = retained + .as_ref() + .map(|group| { + group + .membership()? + .joined(group.identity().member) + .map(|binding| binding.map(|_| group.identity().member)) + }) + .transpose()? + .flatten(); + let recovered = retained + .as_ref() + .map(|group| group.recovery_genesis()) + .transpose()? + .flatten() + .is_some(); + let activated_transfer = retained + .as_ref() + .map(|group| group.group_manifest()) + .transpose()? + .flatten() + .is_some_and(|marker| marker.transfer_activation().is_some()); + let permit = if assignment.sealed_transfer_source { + let managed = options.managed.as_ref().ok_or_else(|| { + anyhow::anyhow!("sealed source needs managed boot authority") + })?; + namespace + .directory + .bootstrap_managed_sealed_source( + &assignment.birth, + &log, + &managed.authorization, + ) + .await? + } else if recovered && options.managed.is_none() { + let authority = recovery_authority.clone().ok_or_else(|| { + anyhow::anyhow!( + "recovered startup requires an infrastructure fencing authority" + ) + })?; + namespace + .directory + .bootstrap_selected_recovered_group_member( + &assignment.birth, + &log, + authority, + ) + .await? + } else if let Some(member) = joined { + if let Some(managed) = &options.managed { + namespace + .directory + .bootstrap_managed_joined_group_member( + &assignment.birth, + member, + &log, + &managed.authorization, + ) + .await? + } else { + namespace + .directory + .bootstrap_joined_group_member(&assignment.birth, member, &log) + .await? + } + } else if let Some(managed) = &options.managed { + if activated_transfer { + namespace + .directory + .bootstrap_managed_activated_transfer_voter( + &assignment.birth, + &log, + &managed.authorization, + ) + .await? + } else { + namespace + .directory + .bootstrap_managed_group_member( + &assignment.birth, + &log, + &managed.authorization, + ) + .await? + } + } else { + namespace + .directory + .bootstrap_group_member(&assignment.birth, &log) + .await? + }; + assignment.pending_transfer_archive = permit.transfer_archive_pending(); + let group = permit.group().clone(); + if let Some(request) = permit.recovery_request() { + // Charge the original bytes plus decoded records and + // both startup/activation references against the + // existing node control bank. No new bank is added. + recovery_bytes = + recovery_bytes + .checked_add( + request.encode().len().checked_mul(16).ok_or_else(|| { + anyhow::anyhow!("recovery metadata overflow") + })?, + ) + .ok_or_else(|| anyhow::anyhow!("recovery metadata overflow"))?; + anyhow::ensure!( + recovery_bytes <= options.max_control_bytes, + "recovery metadata exceeds node control admission" + ); + recovering.push(recovery::Registration { + directory: namespace.directory.clone(), + request: request.clone(), + authority: permit + .recovery_authority() + .ok_or_else(|| anyhow::anyhow!("recovery authority missing"))? + .clone(), + }); + } + let (applier, current) = images + .recover_group( + &log, + group, + namespace.directory.namespace_config(), + &options, + limits, + ) + .await?; + runtime.attach(permit, current).await?; + + appliers.push(applier); + } + } + } + // Recover parked images before clearing conservative references. + // Parked replicas stay out of the serving roster and transport. + let mut parked_replicas = Vec::with_capacity(retained.parked.len()); + for (namespace, group) in retained.parked { + parked_replicas.push( + images + .recover_group( + &log, + group, + prepared.namespaces[namespace].directory.namespace_config(), + &options, + limits, + ) + .await?, + ); + } + let mut collectible = Vec::new(); + collectible.extend( + images + .adopt_recovered(&log, &appliers, &parked_replicas) + .await?, + ); + // No physical peer or query route is reachable while a later local group + // still waits on its permanent claim or physical recovery. The attached + // owners retain bounded Ready work until the controller starts draining it. + for assignment in &prepared.groups { + match assignment.placement { + RaftGroupPlacement::Replica => { + transport.install_physical_group(assignment.transport.clone())? + } + RaftGroupPlacement::QueryRouter => { + let mut route = assignment.transport.clone(); + if options.managed.is_some() { + // Router revisions track the physical configuration log. + route.revision = 1; + if route.peers.is_empty() { + continue; + } + } + match transport.install_cell_query_group(route) { + Ok(()) => (), + Err( + super::transport::TransportError::Refused + | super::transport::TransportError::StalePeer, + ) if options.managed.is_some() => (), + Err(error) => return Err(error.into()), + } + } + } + } + let controller = CellGroupController::start( + runtime.clone(), + events, + transport.clone(), + sql.clone(), + appliers, + limits.controller, + ) + .await?; + startup.controller = Some(controller.clone()); + controller.retain_parked_retired_sources(retained.retired_sources)?; + let admission = Arc::new(crate::application::consensus::admission::Admission::default()); + controller.bind_public_admission( + admission.clone(), + crate::asyncrt::services().application_work_registry(), + )?; + let namespaces = prepared + .namespaces + .iter() + .map(|namespace| { + Ok(CellGroupProvisionNamespace { + directory: namespace.directory.clone(), + images: images.apply(namespace.directory.namespace_config(), &options)?, + }) + }) + .collect::>>()?; + let archive = archive::Pools::new(limits.archive, checkpoint)?; + let (archive_images, archive_provisioning) = if let Some(archive) = archive { + let (observer, inventory, provisioning) = archive + .register(&prepared, &controller, &log, &directory, limits, &options) + .await?; + collectible.extend(inventory); + (Some(observer), Some(provisioning)) + } else { + (None, None) + }; + let archive_io = archive_provisioning + .as_ref() + .map(|archive| archive.io.clone()); + controller.configure_provisioning_with_archives( + CellGroupProvisioning { + directory: directory.clone(), + filesystem: crate::asyncrt::fs(), + namespaces, + manifest: limits.manifest, + entries: limits.entries, + max_effect_rows: limits.max_effect_rows, + max_pages: limits.manifest_pages, + max_page_bytes: limits.manifest_bytes, + max_retained_roots: limits.retained_roots, + peers: prepared.peers.clone(), + peer_revision: limits.provisioning.peer_revision, + max_preparations: limits.provisioning.max_preparations, + max_preparation_bytes: limits.provisioning.max_preparation_bytes, + max_parallel_preparations: limits.provisioning.max_parallel_preparations, + max_waiters: limits.provisioning.max_waiters, + }, + archive_provisioning, + parked_replicas, + )?; + let mut registrations = Vec::new(); + let mut effects = Vec::new(); + let mut dispatchers = Vec::new(); + for namespace in &prepared.namespaces { + let dispatcher = CellGroupDispatcher::from_domain( + log.domain(), + namespace.directory.clone(), + namespace.class.clone(), + keys.clone(), + limits.dispatch, + )? + .with_s3(&controller)?; + let config = namespace.directory.namespace_config(); + let storage = CellGroupNamespaceStorage::new(images.sql(config, &options)?)?; + if namespace.class.has_system_commands() { + let resources = storage.clone(); + effects.push(CellGroupEffectRegistration::new( + dispatcher.clone(), + options + .effects + .ok_or_else(|| anyhow::anyhow!("effects require a scheduler"))?, + move |cell| Ok(resources.for_cell(cell)), + )?); + } + dispatchers.push((dispatcher.clone(), storage.clone())); + registrations.push(CellGroupHostRegistration { + dispatcher, + storage, + }); + } + if !effects.is_empty() { + controller.configure_effects(effects)?; + } + let host = CellGroupHost::bind( + &runtime, + &sql, + &transport, + controller.clone(), + registrations, + limits.namespaces, + )?; + let collection = collection::Collection::new(collectible)?; + let resources = crate::application::consensus::group_client::resources::Resources::new( + sql, + &transport, + images.observer(), + archive_images, + collection.as_ref().map(collection::Collection::observer), + ); + let service = RaftCells::from_shared( + host, + controller.clone(), + crate::application::consensus::group_client::Infrastructure { + runtime: runtime.clone(), + log: log.clone(), + transport: transport.clone(), + archive_io, + }, + admission, + resources, + &options, + )?; + startup.public = Some(service.clone()); + // Admission/work accounting is bound before gateways or grants become + // externally reachable. Returned capabilities own these weak registrations. + for (dispatcher, storage) in dispatchers { + service.retain_gateways( + transport.install_cell_query_gateway( + dispatcher.clone(), + controller.clone(), + storage.clone(), + )?, + transport.install_cell_command_gateway(dispatcher, controller.clone(), storage)?, + )?; + } + for origin in prepared.query_origins { + transport.grant_query_origin(origin)?; + } + for origin in prepared.command_origins { + transport.grant_command_origin(origin)?; + } + if let Some(discovery) = discovery { + discovery.start(&service, log.clone(), transport.clone())?; + } + recovery::start( + &service, + log.clone(), + runtime, + recovering, + recovery_bytes, + options.poll_ms, + )?; + if let Some(collection) = collection { + collection.start(&service, options.poll_ms)?; + } + startup.public.take(); + startup.service.take(); + startup.sql.take(); + startup.controller.take(); + Ok(service) + }) + .await; + if result.is_err() { + // An explicit startup error is a physical join boundary even when the + // caller retains log/runtime/transport clones. Cancellation detaches + // this same cleanup task, whose dependencies remain node-owned. + if let Some(task) = startup.cleanup()? { + task.await?; + } + } + result +} diff --git a/crates/cellar/durability/raft/group_host/provisioning.rs b/crates/cellar/durability/raft/group_host/provisioning.rs new file mode 100644 index 000000000..eba030118 --- /dev/null +++ b/crates/cellar/durability/raft/group_host/provisioning.rs @@ -0,0 +1,131 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Explicit catalog provisioning, separate from node boot and serving authority. +use super::*; +use crate::asyncrt::ExecutionDomain; +use crate::raft::catalog::{Catalog, CatalogError}; +use object_store::ObjectStore; + +/// Operator-owned namespace setup using the same immutable range initialization +/// as node startup. It creates no local Raft member, image or serving capability. +/// Retained setup must pass `initialize: None` to open existing metadata only. +pub struct RaftNamespaceProvisioning { + directory: RangeDirectory, + archives: CellGroupArchivePool, +} + +impl RaftNamespaceProvisioning { + pub async fn open( + domain: &ExecutionDomain, + store: Arc, + catalog: crate::raft::catalog::CatalogOptions, + namespace: RaftNamespaceConfig, + directory_limits: DirectoryLimits, + archive_limits: CellGroupArchiveLimits, + ) -> anyhow::Result { + domain + .run(async { + anyhow::ensure!( + namespace.namespace.managed_fleet.is_none(), + CatalogError::ManagedAdmissionRequired + ); + anyhow::ensure!( + namespace.namespace.mode == cellar_logic::NamespaceMode::Raft, + "namespace is not Raft" + ); + let catalog = Catalog::new(store, catalog)?; + let selected = if namespace.initialize.is_some() { + catalog.open_namespace(namespace.namespace).await? + } else { + catalog.existing_namespace(namespace.namespace).await? + }; + let directory = RangeDirectory::new(selected, directory_limits)?; + if let Some(initialization) = namespace.initialize { + initialize(&directory, initialization, directory_limits.max_groups).await?; + } + directory.observe().await?; + Ok(Self { + directory, + archives: CellGroupArchivePool::new(archive_limits)?, + }) + }) + .await? + } + + /// Allocate a one-use archive initialization for an existing selected group. + /// Persist and sync `encoded_intent()` before `commit`. On any uncertain + /// outcome, retain that exact intent and use recovery; never recreate it. + pub async fn prepare_archive( + &self, + domain: &ExecutionDomain, + history: GroupHistory, + ) -> anyhow::Result { + domain + .run(async { + let birth = self.directory.materialize_selected_history(history).await?; + let prepared = self + .directory + .prepare_group_archive_binding(domain, &self.archives, &birth) + .await?; + Ok(RaftGroupArchiveInitialization { prepared }) + }) + .await? + } + + /// Verify only the catalog-bound identity and existing registry counters. + /// Missing or incomplete setup refuses; this never initializes replacements. + pub async fn open_archive( + &self, + domain: &ExecutionDomain, + history: GroupHistory, + ) -> anyhow::Result<()> { + domain + .run(async { + let birth = self.directory.materialize_selected_history(history).await?; + self.directory + .open_group_archive(domain, &self.archives, &birth) + .await?; + Ok(()) + }) + .await? + } +} + +/// One-use archive setup. An encoded intent is recovery evidence, not a seed, +/// active membership, local reclamation proof or disaster fencing authority. +pub struct RaftGroupArchiveInitialization { + pub(crate) prepared: PreparedCellGroupArchiveBinding, +} + +impl RaftGroupArchiveInitialization { + pub fn encoded_intent(&self) -> &[u8] { + self.prepared.encode() + } + + /// The caller must have synced this exact encoded intent before entering. + /// Failure can be uncertain; do not retry by allocating another initializer. + pub async fn commit(self, domain: &ExecutionDomain) -> anyhow::Result<()> { + self.prepared.commit(domain).await?; + Ok(()) + } +} + +pub(super) async fn initialize( + directory: &RangeDirectory, + initialization: RaftNamespaceInitialization, + max_groups: u32, +) -> anyhow::Result<()> { + anyhow::ensure!( + !initialization.groups.is_empty() && initialization.groups.len() <= max_groups as usize, + "invalid initial range count" + ); + let mut routes = Vec::new(); + for spec in initialization.groups { + let request = directory.prepare_group_birth(spec)?; + directory.reserve_group_birth(&request).await?; + routes.push(request.route().clone()); + } + let mutation = directory.prepare_initialization(initialization.operation, &routes)?; + directory.publish(&mutation).await?; + Ok(()) +} diff --git a/crates/cellar/durability/raft/group_host/recovery.rs b/crates/cellar/durability/raft/group_host/recovery.rs new file mode 100644 index 000000000..b1ab5de6a --- /dev/null +++ b/crates/cellar/durability/raft/group_host/recovery.rs @@ -0,0 +1,127 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! One joined node service finishes the actual recovered groups' activation. +//! Pending genesis is never an application capability or a replacement quorum. +use super::*; +use crate::application::consensus::group_client::ControlServiceKind; +use crate::raft::{ + catalog::{CatalogError, DisasterFenceAuthority}, + completion::RuntimeError, + storage::ReplicaLog, +}; +use std::time::Duration; + +pub(super) struct Registration { + pub directory: RangeDirectory, + pub request: Arc, + pub authority: Arc>, +} + +pub(super) fn start( + service: &RaftCells, + log: ReplicaLog, + runtime: CellGroupRuntime, + mut groups: Vec, + bytes: usize, + poll_ms: u64, +) -> anyhow::Result<()> { + if groups.is_empty() { + return Ok(()); + } + let lease = service.register_control_service(ControlServiceKind::Recovery, bytes)?; + let mut stop = lease.stop_receiver(); + log.domain() + .spawn_cleanup(async move { + let _lease = lease; + while !groups.is_empty() && !*stop.borrow() { + let mut ordinal = 0; + while ordinal < groups.len() && !*stop.borrow() { + let pending = &groups[ordinal]; + let history = pending.request.replacement().route().history; + match activate(pending, &log, &runtime).await { + Ok(true) => { + groups.remove(ordinal); + } + Ok(false) => { + ordinal += 1; + } + Err(error) if retryable(&error) => { + ordinal += 1; + } + Err(error) => { + tracing::warn!(%error, "recovered group activation refused"); + // Refuse this entire group while retaining its original + // disk, seed and catalog intent for exact reconciliation. + let _ = runtime.quarantine(history).await; + groups.remove(ordinal); + } + } + } + if groups.is_empty() { + break; + } + crate::asyncrt::select! { + _ = stop.changed() => break, + _ = crate::asyncrt::sleep(Duration::from_millis(poll_ms)) => (), + } + } + })? + .detach(); + Ok(()) +} + +async fn activate( + group: &Registration, + log: &ReplicaLog, + runtime: &CellGroupRuntime, +) -> Result { + let permit = group + .directory + .prepare_group_recovery_activation(&group.request, log, group.authority.as_ref()) + .await?; + let timer = log + .domain() + .sleep_until(permit.deadline()) + .map_err(|_| CatalogError::Unconfirmed)?; + // Runtime queuing retains the permit and checks its original deadline. + // Cancelling this waiter never creates a new recovery operation. + let result = crate::asyncrt::select! { + _ = timer => return Err(CatalogError::Unconfirmed), + result = runtime.activate_recovery(permit) => result.map_err(CatalogError::Runtime)?, + }; + match result { + CellGroupRecoveryActivation::Accepted(_) => Ok(false), + CellGroupRecoveryActivation::Applied(acknowledgement) => { + if group + .directory + .group_recovery_pending(&group.request) + .await? + { + group + .directory + .complete_group_recovery( + &group.request, + &acknowledgement, + group.authority.as_ref(), + ) + .await?; + } + Ok(true) + } + } +} + +fn retryable(error: &CatalogError) -> bool { + matches!( + error, + CatalogError::Storage(_) + | CatalogError::Unconfirmed + | CatalogError::Runtime( + RuntimeError::NotLeader + | RuntimeError::NotReady + | RuntimeError::Overloaded + | RuntimeError::DeadlineExceeded + | RuntimeError::StaleCompletion + ) + ) +} diff --git a/crates/cellar/durability/raft/group_host/registration.rs b/crates/cellar/durability/raft/group_host/registration.rs new file mode 100644 index 000000000..f5f44200b --- /dev/null +++ b/crates/cellar/durability/raft/group_host/registration.rs @@ -0,0 +1,531 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Catalog observation resolves group placement. It does not bypass member +//! claims, create per-cell births or grant a destination leader authority. +use super::*; +use crate::raft::catalog::{Catalog, CatalogError}; +use cellar_logic::{consensus::ReplicaIdentity, NamespaceMode}; +use std::collections::{BTreeMap, BTreeSet}; + +pub(crate) async fn prepare( + config: &mut RaftCellsConfig, + application: &dyn crate::application::Application, + store: Arc, + keys: &VersionKeys, +) -> anyhow::Result { + config.host.validate()?; + if let Some(managed) = &config.host.managed { + anyhow::ensure!( + managed.authorization.node() == config.infrastructure.log.node + && managed.authorization.disk() == config.infrastructure.log.disk, + CatalogError::ManagedAdmissionRequired + ); + // Validate the actual startup declarations, not just their duplicated + // copy inside managed options, before any namespace or boot mutation. + anyhow::ensure!( + config.namespaces.len() == managed.namespaces.len() + && config.namespaces.iter().all(|n| n.initialize.is_none() + && managed + .namespaces + .iter() + .any(|d| d.namespace == n.namespace && d.class == n.class)), + "managed startup declarations disagree with rooted capabilities" + ); + } else { + anyhow::ensure!( + config + .namespaces + .iter() + .all(|n| n.namespace.managed_fleet.is_none()), + CatalogError::ManagedAdmissionRequired + ); + } + config.infrastructure.transport.validate()?; + config.shared.entries.validate()?; + config + .shared + .provisioning + .validate(config.shared.controller.max_groups)?; + anyhow::ensure!( + !config.namespaces.is_empty() + && config.namespaces.len() <= config.shared.namespaces.max_namespaces + && config.groups.len() <= config.shared.controller.max_groups + && config.shared.controller.max_groups <= config.infrastructure.log.max_groups + && config.shared.controller.max_groups == config.host.max_groups + && config.shared.dispatch.max_input_bytes == config.host.max_input_bytes + && config.shared.entries.envelope == config.host.execution.envelope + && config.shared.manifest.max_cells + <= config.infrastructure.consensus.max_cells_per_group, + "incoherent shared host capacities" + ); + anyhow::ensure!( + config.shared.archive.is_none() || config.shared.checkpoints.is_some(), + "archive requires checkpoint admission" + ); + anyhow::ensure!( + config.shared.controller.checkpoints.is_none() || config.shared.checkpoints.is_some(), + "checkpoint policy requires physical source admission" + ); + // Until these consumers are converted they cannot silently start a legacy + // placement/controller or publish a misleading managed node inventory. + anyhow::ensure!( + config.host.controller.is_none() && config.host.inspection_origins.is_empty(), + "whole-group placement and inspection integration is required" + ); + let mut definitions = Vec::new(); + let mut seen = BTreeSet::new(); + for registration in &config.namespaces { + anyhow::ensure!( + seen.insert(( + registration.namespace.cluster, + registration.namespace.namespace + )), + "duplicate namespace capability" + ); + anyhow::ensure!( + registration.namespace.mode == NamespaceMode::Raft, + "namespace is not Raft" + ); + anyhow::ensure!( + !application.supports_cell(®istration.class) + && !application.supports_workflow(®istration.class), + crate::application::consensus::RegistrationError::UnsupportedFactory( + registration.class.clone() + ) + ); + let class = application.resolve_raft_class(®istration.class)?; + class.validate_shared_host(&config.host)?; + anyhow::ensure!( + class.schema_digest() == registration.namespace.schema_digest, + "compiled namespace schema mismatch" + ); + anyhow::ensure!( + registration.namespace.image_storage + != super::super::catalog::RaftImageStorage::Extents + || config.host.extents.is_some(), + "extent namespace requires node storage" + ); + anyhow::ensure!( + registration.namespace.minimum_durability != cellar_logic::consensus::Durability::S3 + || config.shared.archive.is_some(), + "S3 minimum requires an archive owner" + ); + definitions.push(class); + } + let mut peers = BTreeMap::new(); + for peer in &config.peers { + anyhow::ensure!( + peers.insert(peer.node, peer).is_none(), + "duplicate physical node endpoint" + ); + } + let catalog = Catalog::new(store, config.catalog.clone())?; + let mut namespaces = Vec::new(); + for (registration, class) in config.namespaces.drain(..).zip(definitions) { + let namespace = if registration.initialize.is_some() { + catalog.open_namespace(registration.namespace).await? + } else { + catalog.existing_namespace(registration.namespace).await? + }; + namespace + .require_version_keys(keys, u64::try_from(crate::asyncrt::wall_ms())?) + .await?; + let directory = RangeDirectory::new(namespace, config.shared.directory)?; + let owner = directory.namespace().cell_group_owner().await?; + if let Some(managed) = &config.host.managed { + let bound = managed + .authorization + .stop_catalog() + .bind_namespace(directory.namespace_config().clone())?; + anyhow::ensure!( + owner == bound.managed_fleet, + CatalogError::ManagedAdmissionRequired + ); + } else { + anyhow::ensure!(owner.is_none(), CatalogError::ManagedAdmissionRequired); + } + if let Some(initialization) = registration.initialize { + provisioning::initialize( + &directory, + initialization, + config.shared.directory.max_groups, + ) + .await?; + } + // A declared managed standby may precede the first range publication. + // It attaches no group and invents no empty root or serving authority. + // Every explicit placement below still requires a selected full birth. + if config.host.managed.is_none() { + directory.observe().await?; + } + namespaces.push(Namespace { directory, class }); + } + if let Some(managed) = &config.host.managed { + let declarations = namespaces + .iter() + .map(|n| crate::raft::host::NamespaceRegistration { + namespace: n.directory.namespace().clone(), + class: n.class.clone(), + }) + .collect::>(); + let timeout = config + .host + .node_directory + .as_ref() + .ok_or_else(|| anyhow::anyhow!("managed directory required"))? + .timeout_ms; + managed + .authorization + .check_declared_namespaces(&declarations, std::time::Duration::from_millis(timeout)) + .await?; + } + let mut groups = Vec::new(); + let mut seen = BTreeSet::new(); + for assignment in &config.groups { + anyhow::ensure!( + seen.insert(assignment.history), + "duplicate physical group placement" + ); + let mut group = select_assignment(&namespaces, *assignment).await?; + let mut members = Vec::new(); + for member in group.birth.voters().iter().chain(group.birth.learners()) { + let endpoint = match peers.get(&member.node) { + Some(peer) => peer, + None if config.host.managed.is_some() + || config.infrastructure.log.mode + == crate::raft::options::LogOpenMode::Resume => + { + continue + } + None => anyhow::bail!("missing trusted group peer endpoint"), + }; + members.push(super::super::transport::RaftPeer { + replica: ReplicaIdentity { + scope: assignment.history, + member: member.member, + node: endpoint.node, + disk: endpoint.disk, + process: endpoint.process, + }, + label: endpoint.label.clone(), + server_name: endpoint.server_name.clone(), + control_address: endpoint.control_address, + data_address: endpoint.data_address, + query_address: endpoint.query_address, + }); + } + group.transport.peers = members; + groups.push(group); + } + Ok(Prepared { + namespaces, + peers: std::mem::take(&mut config.peers), + groups, + query_origins: std::mem::take(&mut config.query_origins), + command_origins: std::mem::take(&mut config.command_origins), + }) +} + +async fn select_assignment( + namespaces: &[Namespace], + assignment: RaftGroupAssignment, +) -> anyhow::Result { + let namespace = namespaces + .iter() + .position(|n| { + let config = n.directory.namespace_config(); + config.cluster == assignment.history.cluster + && config.namespace == assignment.history.namespace + }) + .ok_or_else(|| anyhow::anyhow!("group is outside namespace capabilities"))?; + let birth = namespaces[namespace] + .directory + .materialize_selected_history(assignment.history) + .await?; + Ok(assignment_from_birth( + namespace, + assignment.placement, + birth, + )) +} + +fn assignment_from_birth( + namespace: usize, + placement: RaftGroupPlacement, + birth: CellGroupBirth, +) -> Assignment { + Assignment { + namespace, + placement, + pending_transfer_archive: false, + sealed_transfer_source: false, + transport: super::super::transport::GroupRoute { + scope: birth.route().history, + schema_digest: birth.route().schema_digest, + revision: birth.route().epoch.get(), + peers: Vec::new(), + }, + birth, + } +} + +/// Recover local placement from the opened engine and selected namespace +/// histories. Boot, permanent claims and membership are checked at attachment. +#[derive(Default)] +pub(super) struct RetainedPlacements { + pub(super) parked: Vec<(usize, crate::raft::cache::GroupLog)>, + pub(super) retired_sources: Vec, +} + +pub(super) async fn discover_retained_placements( + prepared: &mut Prepared, + log: &crate::raft::storage::ReplicaLog, + fleet: Option<&crate::raft::fleet_catalog::FleetCatalog>, + limits: CellGroupNodeLimits, +) -> anyhow::Result { + let configured: BTreeMap<_, _> = prepared + .groups + .iter() + .map(|assignment| (assignment.birth.route().history, assignment.placement)) + .collect(); + let mut placements = RetainedPlacements::default(); + for retained in log.cell_groups()? { + let history = retained.identity().scope; + match configured.get(&history) { + Some(RaftGroupPlacement::Replica) => continue, + Some(RaftGroupPlacement::QueryRouter) => { + anyhow::bail!("retained replica conflicts with router placement"); + } + None => (), + } + anyhow::ensure!( + prepared + .groups + .len() + .saturating_add(placements.parked.len()) + < limits.controller.max_groups, + "retained group capacity exceeded" + ); + let namespace = || { + prepared + .namespaces + .iter() + .position(|namespace| { + let config = namespace.directory.namespace_config(); + config.cluster == history.cluster && config.namespace == history.namespace + }) + .ok_or(CatalogError::ManagedAdmissionRequired) + }; + let Some(fleet) = fleet else { + // Explicit unmanaged assignments control serving. An omitted + // replica in a declared namespace still owns its durable images. + placements.parked.push((namespace()?, retained)); + continue; + }; + let marker = retained.group_manifest()?; + if let Some(marker) = marker + .as_ref() + .filter(|marker| marker.recovery_request().is_some()) + { + let namespace = namespace()?; + let directory = &prepared.namespaces[namespace].directory; + let timeout = std::time::Duration::from_secs(5); + let snapshot = fleet.read(timeout).await?; + let birth = fleet.history(&snapshot, &history, timeout).await?; + if Box::pin(directory.park_retained_managed_recovery( + &retained, + marker, + &birth, + log.storage_instance()?, + )) + .await? + { + placements.parked.push((namespace, retained)); + continue; + } + } + if let Some(marker) = marker.as_ref().filter(|marker| { + marker.transfer_origin().is_some() && marker.transfer_activation().is_none() + }) { + let namespace = namespace()?; + let directory = &prepared.namespaces[namespace].directory; + let origin = marker.transfer_origin().ok_or(CatalogError::Fenced)?; + let operation = cellar_logic::consensus::ProposalId::new(origin.operation())?; + let transition = fleet + .reopen_managed_transition(directory, operation, std::time::Duration::from_secs(5)) + .await? + .ok_or(CatalogError::Fenced)?; + directory + .require_parked_managed_transfer( + &transition, + &retained, + marker, + log.storage_instance()?, + ) + .await?; + placements.parked.push((namespace, retained)); + continue; + } + if let Some(marker) = marker + .as_ref() + .filter(|marker| marker.source_seal().is_some()) + { + let namespace = namespace()?; + let directory = &prepared.namespaces[namespace].directory; + if let Some(seal) = marker + .source_seal() + .filter(|seal| seal.retirement().is_some()) + { + let operation = cellar_logic::consensus::ProposalId::new(seal.transition())?; + let transition = fleet + .reopen_managed_transition( + directory, + operation, + std::time::Duration::from_secs(5), + ) + .await? + .ok_or(CatalogError::Fenced)?; + if transition + .journal + .as_ref() + .is_some_and(|journal| journal.retired_parents().is_some()) + { + directory + .park_managed_retired_source(&transition.plan, &retained, marker) + .await?; + placements.retired_sources.push(history); + placements.parked.push((namespace, retained)); + continue; + } + } + if let Some((birth, _)) = directory.discover_managed_sealed_source(marker).await? { + let mut assignment = + assignment_from_birth(namespace, RaftGroupPlacement::Replica, birth); + assignment.sealed_transfer_source = true; + prepared.groups.push(assignment); + continue; + } + } + let group = select_assignment( + &prepared.namespaces, + RaftGroupAssignment { + history, + placement: RaftGroupPlacement::Replica, + }, + ) + .await?; + prepared.groups.push(group); + } + Ok(placements) +} + +/// Resolve the complete physical placement after managed discovery has accepted +/// current process views, before genesis, attachment or route publication. +pub(super) fn refresh_routes( + prepared: &mut Prepared, + log: &crate::raft::storage::ReplicaLog, + managed: bool, +) -> anyhow::Result<()> { + for assignment in &mut prepared.groups { + let managed_router = managed && assignment.placement == RaftGroupPlacement::QueryRouter; + let retained = if assignment.placement == RaftGroupPlacement::Replica { + match log.cell_group(&assignment.birth.route().history) { + Ok(group) => Some(group), + Err(crate::raft::storage::LogError::NotFound) => None, + Err(error) => return Err(error.into()), + } + } else { + None + }; + let members = if let Some(group) = &retained { + let cache = group.read()?; + let members = crate::raft::member_state::members(&cache.conf_state); + anyhow::ensure!( + members.contains(&group.identity().member.get()), + CatalogError::Fenced + ); + members + .into_iter() + .map(|id| { + let member = cellar_logic::consensus::MemberId::new(id)?; + let node = if let Some((binding, _)) = cache.membership.joined(member)? { + binding.node + } else { + assignment + .birth + .voters() + .iter() + .chain(assignment.birth.learners()) + .find(|m| m.member == member) + .ok_or_else(|| { + anyhow::anyhow!( + "retained membership has no birth or joined binding" + ) + })? + .node + }; + Ok((member, node)) + }) + .collect::>>()? + } else { + let learners = if managed_router { + &[] + } else { + assignment.birth.learners() + }; + assignment + .birth + .voters() + .iter() + .chain(learners) + .map(|m| (m.member, m.node)) + .collect() + }; + if managed_router + && members + .iter() + .any(|(_, node)| !prepared.peers.iter().any(|peer| peer.node == *node)) + { + // An offline router may return after every birth voter retired. + // Its directory must first observe the current quorum. + assignment.transport.peers.clear(); + continue; + } + assignment.transport.peers = members + .into_iter() + .map(|(member, node)| { + let endpoint = prepared + .peers + .iter() + .find(|peer| peer.node == node) + .ok_or_else(|| anyhow::anyhow!("missing trusted group peer endpoint"))?; + Ok(super::super::transport::RaftPeer { + replica: ReplicaIdentity { + scope: assignment.birth.route().history, + member, + node: endpoint.node, + disk: endpoint.disk, + process: endpoint.process, + }, + label: endpoint.label.clone(), + server_name: endpoint.server_name.clone(), + control_address: endpoint.control_address, + data_address: endpoint.data_address, + query_address: endpoint.query_address, + }) + }) + .collect::>()?; + if let Some(group) = retained { + let membership = group.membership()?; + anyhow::ensure!( + assignment + .transport + .peers + .iter() + .all(|p| membership.permits_group(&p.replica)), + CatalogError::Fenced + ); + } + } + Ok(()) +} diff --git a/crates/cellar/durability/raft/group_host/resources.rs b/crates/cellar/durability/raft/group_host/resources.rs new file mode 100644 index 000000000..bb5baa604 --- /dev/null +++ b/crates/cellar/durability/raft/group_host/resources.rs @@ -0,0 +1,347 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! One serving image inventory and one independent archive-verifier inventory +//! per node. Namespace/cell handles never open another physical store. +use super::*; +use crate::raft::{ + catalog::{NamespaceConfig, RaftImageStorage}, + execution::PrivateOptions, + host::RaftHostOptions, + storage::ReplicaLog, +}; +use cellar_ltx::{ + generation::{GenerationStore, ReadServices}, + internal::NodeExtentStore, + stream_snapshot::ScratchPool, +}; +use rand::RngCore; +use std::path::Path; + +#[derive(Clone)] +pub(super) struct Images { + directory: PathBuf, + materialized: Option, + extents: Option, + cache: Option, + accounting: Option, +} + +/// Constructed only after this owner's complete reference recovery. The live +/// collector can delete eligible physical files, but cannot change references. +pub(super) struct CollectibleExtents { + store: NodeExtentStore, + warm_cache: bool, +} + +impl CollectibleExtents { + fn serving(store: NodeExtentStore) -> Self { + Self { + store, + warm_cache: false, + } + } + + fn verifier(store: NodeExtentStore) -> Self { + Self { + store, + warm_cache: true, + } + } + + pub(super) fn store(&self) -> &NodeExtentStore { + &self.store + } + + pub(super) fn needs_reclaim(&self) -> Result { + if !self.store.needs_reclaim()? { + return Ok(false); + } + if !self.warm_cache { + return Ok(true); + } + // Keep eligible verifier files as a charged cache below a quarter of + // each existing inventory ceiling. The remaining headroom backs active + // receives/replay. These files grant no proof: every job still GETs and + // independently checks its source and local destination bytes. + let (usage, limits) = self + .store + .observer() + .capture() + .ok_or(cellar_ltx::internal::StoreError::Quarantined)?; + Ok(usage.roots == 0 + || usage.roots >= (limits.max_roots / 4).max(1) + || usage.objects >= (limits.max_objects / 4).max(1) + || usage.root_extents >= (limits.max_root_extents / 4).max(1) + || usage.bytes >= (limits.max_bytes / 4).max(1)) + } +} + +impl Images { + pub async fn recover_group( + &self, + log: &ReplicaLog, + group: crate::raft::cache::GroupLog, + namespace: &NamespaceConfig, + options: &RaftHostOptions, + limits: CellGroupNodeLimits, + ) -> anyhow::Result<(CellGroupApplier, PreparedGroupManifest)> { + let path = ManifestPageStore::group_pages_path(&self.directory, group.identity().scope); + let backend = self.apply(namespace, options)?; + log.domain() + .blocking(move || { + Ok::<_, anyhow::Error>(CellGroupApplier::recover( + group, + pages(&path, limits)?, + backend, + limits.manifest, + limits.entries, + limits.max_effect_rows, + )?) + })? + .await? + } + + pub async fn open( + log: &ReplicaLog, + directory: PathBuf, + options: &RaftHostOptions, + namespaces: &[Namespace], + ) -> anyhow::Result { + let materialized = namespaces.iter().any(|n| { + n.directory.namespace_config().image_storage == RaftImageStorage::Materialized + }); + let extents = namespaces + .iter() + .any(|n| n.directory.namespace_config().image_storage == RaftImageStorage::Extents); + let limits = options.images; + let extent_options = options.extents; + let scratch = log.snapshot_scratch(); + let fs = crate::asyncrt::fs(); + log.domain() + .blocking(move || { + let accounting = + materialized.then(cellar_ltx::generation::GenerationAccounting::default); + let materialized = materialized + .then(|| { + GenerationStore::open_with_scratch( + directory.join("images"), + fs.clone(), + limits, + accounting.as_ref(), + scratch, + ) + }) + .transpose()?; + let (extents, cache) = if extents { + let options = extent_options.ok_or_else(|| { + anyhow::anyhow!("extent namespace requires node inventory") + })?; + ( + Some(NodeExtentStore::open_serialized( + directory.join("extent-inventory"), + fs, + options.inventory, + )?), + Some(ScratchPool::new(options.cache_scratch)?), + ) + } else { + (None, None) + }; + Ok(Self { + directory, + materialized, + extents, + cache, + accounting, + }) + })? + .await? + } + + /// Startup has recovered every physical group's current/checkpoint graph, + /// and no peer or public route can enter yet. Those complete graph pins + /// replace the conservative host references loaded from filenames. Keep + /// the appliers alive through physical completion, including cancellation. + pub async fn adopt_recovered( + &self, + log: &ReplicaLog, + appliers: &[CellGroupApplier], + parked: &[(CellGroupApplier, PreparedGroupManifest)], + ) -> anyhow::Result> { + let inventory = log.cell_groups()?; + let owners: Vec<_> = appliers + .iter() + .chain(parked.iter().map(|(applier, _)| applier)) + .cloned() + .collect(); + anyhow::ensure!( + inventory.len() == owners.len() + && inventory + .iter() + .all(|group| owners.iter().any(|a| a.history() == group.identity().scope)), + "image adoption requires complete physical group inventory" + ); + let images = self.clone(); + log.domain() + .blocking(move || -> anyhow::Result<_> { + if let Some(store) = &images.materialized { + store.retain_only(&[])?; + } + if let Some(store) = &images.extents { + store.retain_only(&[])?; + } + let inventory = images.extents.map(CollectibleExtents::serving); + drop(owners); + Ok(inventory) + })? + .await? + } + + /// This separately locked verifier has no durable local selectors. Call + /// before registering its first receive/replay owner. Remote holds still + /// require fresh verification; cached filenames supply no recovery proof. + pub async fn adopt_verifier( + &self, + log: &ReplicaLog, + ) -> anyhow::Result> { + let images = self.clone(); + log.domain() + .blocking(move || -> anyhow::Result<_> { + if let Some(store) = &images.materialized { + store.retain_only(&[])?; + } + if let Some(store) = &images.extents { + store.retain_only(&[])?; + } + Ok(images.extents.map(CollectibleExtents::verifier)) + })? + .await? + } + + pub fn observer(&self) -> crate::application::consensus::group_client::resources::Images { + crate::application::consensus::group_client::resources::Images { + materialized: self.accounting.clone(), + extents: self.extents.as_ref().map(NodeExtentStore::observer), + } + } + + pub fn sql( + &self, + namespace: &NamespaceConfig, + options: &RaftHostOptions, + ) -> anyhow::Result { + let directory = self + .directory + .join("sql") + .join(hex::encode(namespace.cluster.bytes())) + .join(hex::encode(namespace.namespace.bytes())); + Ok(match namespace.image_storage { + RaftImageStorage::Materialized => CellSqlOptions::Materialized(PrivateOptions { + directory, + host: host(), + sqlite_vfs: None, + images: self + .materialized + .clone() + .ok_or_else(|| anyhow::anyhow!("missing materialized inventory"))?, + query_services: services(), + application_digest: namespace.schema_digest, + limits: options.execution, + }), + RaftImageStorage::Extents => CellSqlOptions::Extents(CellSqlRootOptions { + host: host(), + sqlite_vfs: None, + images: self + .extents + .clone() + .ok_or_else(|| anyhow::anyhow!("missing extent inventory"))?, + query_services: services(), + application_digest: namespace.schema_digest, + limits: options.execution, + cache_directory: directory, + cache_pool: self + .cache + .clone() + .ok_or_else(|| anyhow::anyhow!("missing extent cache admission"))?, + cache: options + .extents + .ok_or_else(|| anyhow::anyhow!("missing extent cache limits"))? + .cache, + }), + }) + } + + pub fn apply( + &self, + namespace: &NamespaceConfig, + options: &RaftHostOptions, + ) -> anyhow::Result { + Ok(match self.sql(namespace, options)? { + CellSqlOptions::Materialized(o) => CellGroupApplyImages::Materialized { + images: o.images, + services: o.query_services, + }, + CellSqlOptions::Extents(o) => CellGroupApplyImages::Extents { + images: o.images, + services: o.query_services, + limits: options.images.apply, + cache_directory: o.cache_directory, + cache_pool: o.cache_pool, + cache: o.cache, + }, + }) + } +} + +pub(super) fn pages(path: &Path, limits: CellGroupNodeLimits) -> anyhow::Result { + Ok(ManifestPageStore::open( + path, + crate::asyncrt::fs(), + limits.manifest_pages, + limits.manifest_bytes, + limits.retained_roots, + )?) +} + +fn services() -> ReadServices { + ReadServices { + now_unix_millis: Arc::new(crate::asyncrt::wall_ms), + fill_random: Arc::new(|b| crate::asyncrt::rng("cell-group-image-read").fill_bytes(b)), + } +} + +fn host() -> cellar_ltx::LtxHost { + let fs = crate::asyncrt::fs(); + let age = fs.clone(); + let read = fs.clone(); + cellar_ltx::LtxHost::new( + crate::asyncrt::wall_ms, + move |path| { + Ok(std::time::Duration::from_millis( + crate::asyncrt::wall_ms() + .saturating_sub(age.metadata(path)?.modified_unix_millis) + .max(0) as u64, + )) + }, + move |path| { + let fs = read.clone(); + async move { + crate::asyncrt::try_blocking(move || fs.read(&path)) + .map_err(|e| std::io::Error::other(e.to_string()))? + .await + .map_err(|e| std::io::Error::other(e.to_string()))? + } + }, + |job| async move { + crate::asyncrt::try_blocking(job) + .map_err(|e| cellar_ltx::HostTaskError::new(e.to_string()))? + .await + .map_err(|e| cellar_ltx::HostTaskError::new(e.to_string())) + }, + ) + .with_filesystem(fs) +} + +#[cfg(all(test, not(cellar_internal_tests)))] +#[path = "resources_tests.rs"] +mod tests; diff --git a/crates/cellar/durability/raft/group_host/resources_tests.rs b/crates/cellar/durability/raft/group_host/resources_tests.rs new file mode 100644 index 000000000..f7c15dcdb --- /dev/null +++ b/crates/cellar/durability/raft/group_host/resources_tests.rs @@ -0,0 +1,139 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Real public closure must join collection past its physical deletion barrier. +#![allow(clippy::disallowed_methods)] +use super::*; +use crate::raft::{ + catalog::Catalog, + group_host::{collection::Collection, startup_tests::StartupNode}, + storage_fault as fault, + version::{VersionKeys, VersionSigningKey}, +}; +use cellar_logic::{ + consensus::{ClusterId, NamespaceId}, + NamespaceMode, +}; +use cellar_ltx::internal::ExtentId; +use std::time::Duration; + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn collection_close_joins_held_deletion_and_retains_admission() -> anyhow::Result<()> { + let mut node = StartupNode::open(1).await?; + let class = Arc::new(RaftClass::new("CollectionOnly", 1, [65; 32])?); + let catalog = Catalog::new( + Arc::new(object_store::memory::InMemory::new()), + Default::default(), + )?; + let namespace = node + .domain + .run(catalog.open_namespace(NamespaceConfig::new( + ClusterId::new([1; 16])?, + NamespaceId::new([2; 16])?, + NamespaceMode::Raft, + class.schema_digest(), + ))) + .await??; + let prepared = Prepared { + namespaces: vec![Namespace { + directory: RangeDirectory::new(namespace, Default::default())?, + class, + }], + peers: vec![node.peer()], + groups: vec![], + query_origins: vec![], + command_origins: vec![], + }; + let input = node.service(); + let public = node + .domain + .run(start( + input, + prepared, + node.root.path().join("cells"), + test_support::host_options(), + test_support::limits(), + VersionKeys::new( + VersionSigningKey::new([31; 32], u64::MAX)?, + vec![], + Duration::from_secs(86400), + )?, + None, + )) + .await??; + + let gate = fault::Gate::new(None); + let directory = node.root.path().join("collection-inventory"); + let files = Arc::new(fault::FaultFileSystem { gate: gate.clone() }); + let path = directory.clone(); + let store = node + .domain + .blocking(move || -> anyhow::Result<_> { + let store = NodeExtentStore::open_serialized(path, files, Default::default())?; + let bytes = vec![7; 4096]; + store.put_extent_bytes(ExtentId::of(&bytes)?, &bytes)?; + Ok(store) + })? + .await??; + let images = Images { + directory: directory.clone(), + materialized: None, + extents: Some(store.clone()), + cache: None, + accounting: None, + }; + // The engine has no groups: its complete recovered reference set is empty. + let inventories = images + .adopt_recovered(&node.log, &[], &[]) + .await? + .into_iter() + .collect(); + let collection = Collection::new(inventories)?.ok_or_else(|| anyhow::anyhow!("collection"))?; + let observer = collection.observer(); + *gate.sync_path.lock().unwrap() = Some(directory); + let entered = gate.entered.notified(); + tokio::pin!(entered); + entered.as_mut().enable(); + let release = gate.arm(fault::HOLD_SELECTED_DIRECTORY_SYNC); + collection.start(&public, 20)?; + tokio::time::timeout(Duration::from_secs(5), entered).await?; + let charged = store.usage()?; + assert_eq!( + charged.objects, 1, + "deletion refunded before its physical call returned" + ); + let budget = public.control_budget().clone(); + let pending = budget.usage(); + assert!(pending.0 > 4096 && pending.1 == 2); + let mut closing = Box::pin(public.close()); + assert!( + tokio::time::timeout(Duration::from_millis(30), &mut closing) + .await + .is_err() + ); + drop(closing); + assert_eq!( + budget.usage(), + pending, + "cancelled close released admitted physical collection" + ); + assert!( + node.log.storage_instance().is_ok(), + "storage closed before collector joined" + ); + drop(release); + tokio::time::timeout(Duration::from_secs(5), public.close()).await??; + assert_eq!(budget.usage(), (0, 0)); + assert_eq!(store.usage()?.objects, 0); + let finished = observer + .capture() + .ok_or_else(|| anyhow::anyhow!("collection observation"))?; + assert!(!finished.running && finished.files_reclaimed == 1 && finished.failures == 0); + assert!(matches!( + node.log.storage_instance(), + Err(crate::raft::storage::LogError::Closed) + )); + drop(images); + drop(store); + drop(public); + node.close().await +} diff --git a/crates/cellar/durability/raft/group_host/router_directory.rs b/crates/cellar/durability/raft/group_host/router_directory.rs new file mode 100644 index 000000000..d36df0523 --- /dev/null +++ b/crates/cellar/durability/raft/group_host/router_directory.rs @@ -0,0 +1,109 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Managed router membership discovery uses the existing joined directory owner. +use super::*; +use crate::raft::{ + fleet_catalog::{FleetTrustReader, FleetTrustSnapshot}, + storage::ReplicaLog, + transport::{GroupRoute, RaftPeer, RaftTransport}, +}; +use cellar_logic::consensus::ReplicaIdentity; +use std::time::Duration; + +struct Work { + group: GroupHistory, + next_node: usize, +} + +pub(super) struct RouterDirectory { + work: Vec, + next_group: usize, +} + +impl RouterDirectory { + pub(super) fn new(groups: impl Iterator) -> Option { + let work = groups + .map(|group| Work { + group, + next_node: 0, + }) + .collect::>(); + (!work.is_empty()).then_some(Self { + work, + next_group: 0, + }) + } + + pub(super) async fn refresh( + &mut self, + log: &ReplicaLog, + transport: &RaftTransport, + observer: &RaftCells, + reader: &FleetTrustReader, + trust: &FleetTrustSnapshot, + deadline: u64, + ) -> anyhow::Result<()> { + let remaining = || super::directory::remaining(log, deadline); + let nodes = trust.nodes(); + let local = log.limits().node; + let index = self.next_group; + self.next_group = (index + 1) % self.work.len(); + let work = &mut self.work[index]; + let source = (0..nodes.len()).find_map(|_| { + let node = &nodes[work.next_node % nodes.len()]; + work.next_node = (work.next_node + 1) % nodes.len(); + (node.node() != local && node.permits_transport()).then_some(node.node()) + }); + let Some(source) = source else { + return Ok(()); + }; + // Sample one physical node and group per turn. The rotating fleet scan + // can recover even if this router was offline through all replacements. + let Some(authority) = reader + .authorize_router_group(trust, &work.group, remaining()?) + .await? + else { + return Ok(()); + }; + let revision = trust.revision(); + let proof = observer + .observe_peer_fleet_membership( + source, + authority.clone(), + revision, + remaining()?.min(Duration::from_millis(500)), + ) + .await?; + let observed = proof.membership(); + let mut peers = Vec::new(); + for voter in authority.router_voters(observed)? { + let peer = transport.managed_group_peer(&work.group, voter.node, revision)?; + anyhow::ensure!( + voter.joined_disk.is_none_or(|disk| peer.disk == disk), + "router member disk changed" + ); + peers.push(RaftPeer { + replica: ReplicaIdentity { + scope: work.group, + member: voter.member, + node: voter.node, + disk: peer.disk, + process: peer.process, + }, + label: peer.label, + server_name: peer.server_name, + control_address: peer.control_address, + data_address: peer.data_address, + query_address: peer.query_address, + }); + } + remaining()?; + transport.install_cell_query_group(GroupRoute { + scope: work.group, + revision: observed.membership_revision().get(), + schema_digest: observed.schema_digest(), + peers, + })?; + Ok(()) + } +} diff --git a/crates/cellar/durability/raft/group_host/test_support.rs b/crates/cellar/durability/raft/group_host/test_support.rs new file mode 100644 index 000000000..a4a895b31 --- /dev/null +++ b/crates/cellar/durability/raft/group_host/test_support.rs @@ -0,0 +1,63 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Explicit bounded startup capacities shared by the native host regressions. +use super::*; + +pub(crate) fn limits() -> CellGroupNodeLimits { + CellGroupNodeLimits { + namespaces: CellGroupHostLimits { + max_namespaces: 1, + namespace_bytes: 16 * 1024 * 1024, + }, + controller: CellGroupControllerLimits { + max_groups: 2, + max_parallel_apply: 2, + retry_interval_ms: 5, + max_commands: 2, + max_command_bytes: 64 * 1024, + checkpoints: None, + }, + sql: CellSqlLimits { + workers: 2, + queue_per_worker: 4, + max_pending_cells: 8, + group_pending_cells: 8, + node_bytes: 512 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + }, + provisioning: CellGroupProvisionLimits { + max_preparations: 1, + max_preparation_bytes: 512 * 1024 * 1024, + max_parallel_preparations: 1, + max_waiters: 2, + peer_revision: 1, + }, + directory: DirectoryLimits::default(), + dispatch: CellDispatchLimits { + max_cached_routes: 32, + route_cache_bytes: 256 * 1024, + max_invocations: 8, + retained_bytes: 1024 * 1024, + max_input_bytes: 1024, + }, + manifest: ManifestLimits { + max_cells: 32, + max_depth: 16, + }, + entries: crate::application::consensus::group_protocol::GroupEntryLimits::default(), + manifest_pages: 1024, + manifest_bytes: 16 * 1024 * 1024, + retained_roots: 16, + max_effect_rows: 100, + checkpoints: None, + archive: None, + } +} + +pub(crate) fn host_options() -> crate::raft::host::RaftHostOptions { + crate::raft::host::RaftHostOptions { + max_groups: 2, + max_input_bytes: 1024, + ..Default::default() + } +} diff --git a/crates/cellar/durability/raft/group_snapshot_transport/codec.rs b/crates/cellar/durability/raft/group_snapshot_transport/codec.rs new file mode 100644 index 000000000..a1081f1a5 --- /dev/null +++ b/crates/cellar/durability/raft/group_snapshot_transport/codec.rs @@ -0,0 +1,61 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. +use super::*; +use crate::raft::cell_groups::checkpoint::wire::Part; +use sha2::{Digest, Sha256}; + +pub(in crate::raft::transport) const START_MAGIC: &[u8; 4] = b"CLGS"; +pub(super) const DATA_MAGIC: &[u8; 4] = b"CLGD"; +pub(super) const ACK_MAGIC: &[u8; 4] = b"CLGA"; +pub(super) const START: &[u8] = b"cellar-group-snapshot-start-1"; +pub(super) const DATA: &[u8] = b"cellar-group-snapshot-data-1"; +pub(super) const ACK: &[u8] = b"cellar-group-snapshot-admitted-1"; +pub(super) const CHUNK_BYTES: usize = 64 * 1024; +pub(super) const MESSAGE_BYTES: usize = 128 * 1024; + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Start { + pub route: Route, + pub digest: [u8; 32], +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Packet { + pub sequence: u64, + pub part: Option, + pub offset: u64, + pub length: u64, + pub digest: [u8; 32], +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Ack { + pub sequence: u64, +} + +pub(super) fn digest(body: &[u8]) -> [u8; 32] { + Sha256::digest(body).into() +} + +pub(super) fn message(body: &[u8], route: &Route) -> Result { + if body.len() > MESSAGE_BYTES { + return Err(TransportError::Limit); + } + frame::scan_message(body, 0)?; + let message = Message::parse_from_bytes(body).map_err(|_| TransportError::Protocol)?; + let (source, target) = route.decode_group()?; + if message.get_msg_type() != MessageType::MsgSnapshot + || !message.has_snapshot() + || message.from != source.member.get() + || message.to != target.member.get() + || message.term == 0 + || message.get_snapshot().is_empty() + || !message.entries.is_empty() + || !message.context.is_empty() + { + return Err(TransportError::Identity); + } + Ok(message) +} diff --git a/crates/cellar/durability/raft/group_snapshot_transport/mod.rs b/crates/cellar/durability/raft/group_snapshot_transport/mod.rs new file mode 100644 index 000000000..7c4e476af --- /dev/null +++ b/crates/cellar/durability/raft/group_snapshot_transport/mod.rs @@ -0,0 +1,262 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Complete checkpoints on dedicated authenticated data connections. A stream +//! reply proves admission only; Raft emits its own response after persistence. +use super::*; +use crate::raft::cell_groups::{ + CellGroupApplier, CellGroupCheckpointError, CellGroupCheckpointPool, +}; +use std::sync::Weak; +mod codec; +mod receive; +#[cfg(all(test, not(cellar_internal_tests)))] +mod route_tests; +pub(crate) mod seed; +mod send; +mod workers; +pub(super) use codec::START_MAGIC; +use codec::*; +pub use seed::{CellGroupSeedError, CellGroupSeedSession}; + +pub(super) struct Endpoint { + applier: CellGroupApplier, + route: Arc>, + active: Arc, + owner: Arc<()>, + registration: Arc<()>, +} + +/// Retain with the group's controller registration. Dropping this handle revokes +/// new and in-flight streams; already admitted Raft work keeps its own owners. +pub struct CellGroupCheckpointRegistration { + state: Weak, + endpoint: Arc, +} + +impl Drop for CellGroupCheckpointRegistration { + fn drop(&mut self) { + self.endpoint.active.store(false, Ordering::Release); + let Some(state) = self.state.upgrade() else { + return; + }; + if let Ok(mut registry) = state.registry.lock() { + let history = self.endpoint.applier.history(); + if registry + .group_checkpoints + .get(&history) + .is_some_and(|old| old.ptr_eq(&Arc::downgrade(&self.endpoint))) + { + registry.group_checkpoints.remove(&history); + } + }; + } +} + +impl RaftTransport { + pub(crate) fn enable_group_archive_seeds( + &self, + client: crate::raft::cell_groups::ArchiveSeedClient, + ) -> Result<(), TransportError> { + let mut registry = self + .handle + .0 + .registry + .lock() + .map_err(|_| TransportError::Closed)?; + self.handle.0.admit(®istry)?; + if registry.group_archive_seeds.is_some() { + return Err(TransportError::Identity); + } + registry.group_archive_seeds = Some(client); + Ok(()) + } + + /// Configure once per node, before accepting snapshots. Share this same pool + /// with local checkpoint publication. Scratch comes from the owning log. + pub fn enable_group_checkpoints( + &self, + pool: CellGroupCheckpointPool, + ) -> Result<(), TransportError> { + let state = &self.handle.0; + state.runtime.shared()?; + let mut registry = state.registry.lock().map_err(|_| TransportError::Closed)?; + state.admit(®istry)?; + if registry.group_checkpoint_pool.is_some() { + return Err(TransportError::Identity); + } + registry.group_checkpoint_pool = Some(pool); + Ok(()) + } + + pub(crate) fn group_checkpoint_pool(&self) -> Result { + let state = &self.handle.0; + let registry = state.registry.lock().map_err(|_| TransportError::Closed)?; + state.admit(®istry)?; + registry + .group_checkpoint_pool + .clone() + .ok_or(TransportError::Refused) + } + + pub fn register_group_checkpoint( + &self, + applier: CellGroupApplier, + ) -> Result { + let state = &self.handle.0; + let runtime = state.runtime.shared()?; + if !applier.belongs_to_engine(&runtime.log().owner()) { + return Err(TransportError::Identity); + } + let mut registry = state.registry.lock().map_err(|_| TransportError::Closed)?; + state.admit(®istry)?; + let history = applier.history(); + let route = registry + .physical_groups + .get(&history) + .ok_or(TransportError::UnknownGroup)? + .clone(); + if registry + .group_checkpoints + .get(&history) + .and_then(Weak::upgrade) + .is_some() + { + return Err(TransportError::Identity); + } + if registry.group_checkpoints.len() >= state.options.max_groups { + return Err(TransportError::Overloaded); + } + let owner = registry + .physical_owners + .get(&history) + .cloned() + .ok_or(TransportError::UnknownGroup)?; + let endpoint = Arc::new(Endpoint { + applier, + route, + active: Arc::new(AtomicBool::new(true)), + owner, + registration: Arc::new(()), + }); + registry + .group_checkpoints + .insert(history, Arc::downgrade(&endpoint)); + Ok(CellGroupCheckpointRegistration { + state: Arc::downgrade(state), + endpoint, + }) + } +} + +impl State { + fn checkpoint_endpoint( + &self, + history: &GroupHistory, + ) -> Result<(Arc, CellGroupCheckpointPool), TransportError> { + let registry = self.registry.lock().map_err(|_| TransportError::Closed)?; + self.admit(®istry)?; + let pool = registry + .group_checkpoint_pool + .clone() + .ok_or(TransportError::Refused)?; + let endpoint = registry + .group_checkpoints + .get(history) + .and_then(Weak::upgrade) + .ok_or(TransportError::Refused)?; + if !endpoint.active.load(Ordering::Acquire) + || !registry + .physical_owners + .get(history) + .is_some_and(|owner| Arc::ptr_eq(owner, &endpoint.owner)) + { + return Err(TransportError::StalePeer); + } + let route = registry + .physical_groups + .get(history) + .ok_or(TransportError::UnknownGroup)?; + // Keep one factory/physical owner across advisory refreshes, while each + // admitted stream captures one exact immutable route generation. + let selected = if Arc::ptr_eq(route, &endpoint.route) { + endpoint + } else { + Arc::new(Endpoint { + applier: endpoint.applier.clone(), + route: route.clone(), + active: endpoint.active.clone(), + owner: endpoint.owner.clone(), + registration: endpoint.registration.clone(), + }) + }; + Ok((selected, pool)) + } + + fn checkpoint_current( + &self, + endpoint: &Arc, + wire: &Route, + ) -> Result, TransportError> { + let (_, target) = wire.decode_group()?; + { + let registry = self.registry.lock().map_err(|_| TransportError::Closed)?; + self.admit(®istry)?; + let current = registry + .group_checkpoints + .get(&target.scope) + .and_then(Weak::upgrade) + .ok_or(TransportError::Refused)?; + // Per-chunk validation allocates no selected endpoint or route. + // Drop this guard before peer authorization takes the registry lock. + if !endpoint.active.load(Ordering::Acquire) + || !Arc::ptr_eq(¤t.registration, &endpoint.registration) + || !registry + .physical_owners + .get(&target.scope) + .is_some_and(|owner| Arc::ptr_eq(owner, &endpoint.owner)) + || !registry + .physical_groups + .get(&target.scope) + .is_some_and(|route| Arc::ptr_eq(route, &endpoint.route)) + { + return Err(TransportError::StalePeer); + } + } + self.authorize_physical(wire) + } +} + +fn checkpoint_error(error: CellGroupCheckpointError) -> TransportError { + match error { + CellGroupCheckpointError::Busy => TransportError::Overloaded, + CellGroupCheckpointError::NotReady => TransportError::Runtime(RuntimeError::NotReady), + CellGroupCheckpointError::Authority(error) => TransportError::Runtime(error), + CellGroupCheckpointError::Limit => TransportError::Limit, + other => TransportError::Invalid(other.to_string()), + } +} + +// Covers two in-flight chunk windows, the producer's copy, protobuf decoding, +// authentication/header copies and reply framing. Physical metadata and image +// workspaces are separately charged by the node-wide checkpoint pool. +fn wire_charge(options: &TransportOptions) -> usize { + 4 * CHUNK_BYTES + 4 * MESSAGE_BYTES + 4 * options.max_header_bytes +} + +struct Admission { + // Source producers and transfer work share this owner through actual exit. + _source: Option>, + _node: Permit, + _peer: Permit, + _transfer: Permit, +} + +pub(super) fn revoke(registry: &mut Registry, history: &GroupHistory) { + if let Some(endpoint) = registry + .group_checkpoints + .remove(history) + .and_then(|weak| weak.upgrade()) + { + endpoint.active.store(false, Ordering::Release); + } +} diff --git a/crates/cellar/durability/raft/group_snapshot_transport/receive.rs b/crates/cellar/durability/raft/group_snapshot_transport/receive.rs new file mode 100644 index 000000000..d94e6b197 --- /dev/null +++ b/crates/cellar/durability/raft/group_snapshot_transport/receive.rs @@ -0,0 +1,181 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. +use super::*; +use crate::raft::cell_groups::checkpoint::wire::Chunk; + +impl State { + pub(in crate::raft::transport) async fn receive_group_checkpoint< + S: AsyncRead + AsyncWrite + Unpin, + >( + self: &Arc, + stream: &mut S, + exporter: &[u8; 32], + prefix: [u8; frame::PREFIX], + ) -> Result<(), TransportError> { + let _connection = self + .snapshot_connections + .clone() + .try_acquire_owned() + .map_err(|_| TransportError::Overloaded)?; + let wire = snapshots::Wire { + auth: &self.auth, + exporter, + options: &self.options, + max_body: MESSAGE_BYTES, + }; + let head = self + .bounded( + None, + snapshots::read_bulk_head( + stream, + Some(prefix), + START_MAGIC, + MESSAGE_BYTES, + &self.options, + ), + ) + .await?; + let start: Start = wire.decode(START_MAGIC, START, &head, None)?; + let binding = head.tag; + let pool = self.authorize_physical(&start.route)?; + let (_, target) = start.route.decode_group()?; + let (endpoint, checkpoints) = self.checkpoint_endpoint(&target.scope)?; + let charge = wire_charge(&self.options); + let admission = Arc::new(Admission { + _source: None, + _node: self + .incoming + .lane(Lane::Data) + .acquire(charge) + .ok_or(TransportError::Overloaded)?, + _peer: pool + .incoming + .lane(Lane::Data) + .acquire(charge) + .ok_or(TransportError::Overloaded)?, + _transfer: self + .snapshot_receive_budget + .acquire(snapshots::memory_bound(&self.options)) + .ok_or(TransportError::Overloaded)?, + }); + let deadline = self + .domain + .mono_ms() + .map_err(|_| TransportError::Closed)? + .saturating_add(self.options.snapshot_timeout_ms); + let mut body = vec![0; head.payload_len]; + self.snapshot_io(&pool, deadline, async { + stream.read_exact(&mut body).await?; + Ok(()) + }) + .await?; + if digest(&body) != start.digest { + return Err(TransportError::Authentication); + } + let message = codec::message(&body, &start.route)?; + drop(body); + self.checkpoint_current(&endpoint, &start.route)?; + let receiver = workers::receiver(self, endpoint.clone(), checkpoints, admission.clone())?; + let mut sequence = 0_u64; + let mut result = receiver.result; + let receiving = async { + loop { + self.checkpoint_current(&endpoint, &start.route)?; + let head = self + .snapshot_io( + &pool, + deadline, + snapshots::read_bulk_head( + stream, + None, + DATA_MAGIC, + CHUNK_BYTES.min(self.options.snapshot_chunk_bytes), + &self.options, + ), + ) + .await?; + let packet: Packet = wire.decode(DATA_MAGIC, DATA, &head, Some(&binding))?; + if packet.sequence != sequence { + return Err(TransportError::Replay); + } + let mut bytes = vec![0; head.payload_len]; + self.snapshot_io(&pool, deadline, async { + stream.read_exact(&mut bytes).await?; + Ok(()) + }) + .await?; + if digest(&bytes) != packet.digest { + return Err(TransportError::Authentication); + } + self.checkpoint_current(&endpoint, &start.route)?; + let input = match packet.part { + Some(part) => { + if bytes.is_empty() { + return Err(TransportError::Protocol); + } + workers::Input::Chunk(Chunk { + part, + offset: packet.offset, + length: packet.length, + bytes, + }) + } + None => { + if !bytes.is_empty() || packet.offset != 0 || packet.length != 0 { + return Err(TransportError::Protocol); + } + workers::Input::End + } + }; + let ending = matches!(input, workers::Input::End); + self.snapshot_io(&pool, deadline, async { + receiver + .input + .send(input) + .await + .map_err(|_| TransportError::Closed) + }) + .await?; + if ending { + break; + } + sequence = sequence.checked_add(1).ok_or(TransportError::Limit)?; + } + Ok::<(), TransportError>(()) + }; + // Verification runs on a physical worker. Its failure must also end a + // peer that keeps the next header or body open without an explicit End. + crate::asyncrt::select_biased! { "physical checkpoint verification failure precedes more network input"; + outcome = &mut result => { + outcome.map_err(|_| TransportError::Closed)??; + return Err(TransportError::Protocol); + }, + outcome = receiving => outcome?, + } + drop(receiver.input); + let prepared = self + .snapshot_io(&pool, deadline, async { + result.await.map_err(|_| TransportError::Closed)? + }) + .await?; + self.checkpoint_current(&endpoint, &start.route)?; + // The prepared closure carries physical pins and checkpoint admission + // into the bounded runtime request and its exact persistence completion. + self.snapshot_io(&pool, deadline, async { + self.runtime + .shared()? + .step_checkpoint(target.scope, message, prepared) + .await?; + Ok(()) + }) + .await?; + self.checkpoint_current(&endpoint, &start.route)?; + self.accepted.fetch_add(1, Ordering::Relaxed); + let ack = wire.encode(ACK_MAGIC, ACK, &binding, &Ack { sequence }, &[])?; + self.snapshot_io(&pool, deadline, async { + write_head(stream, &ack).await?; + stream.flush().await?; + Ok(()) + }) + .await + } +} diff --git a/crates/cellar/durability/raft/group_snapshot_transport/route_tests.rs b/crates/cellar/durability/raft/group_snapshot_transport/route_tests.rs new file mode 100644 index 000000000..a286f529a --- /dev/null +++ b/crates/cellar/durability/raft/group_snapshot_transport/route_tests.rs @@ -0,0 +1,211 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Real engine/registration ownership, not quorum or network-stream qualification. +use super::*; +use crate::raft::transport::shared_fixture::Fixture; +use crate::raft::{cell_groups::*, storage::CellGroupBootstrap}; +use cellar_logic::{ + cell_group::{CellRange, RouteEpoch}, + consensus::{DiskLineage, GroupId, IncarnationId, MemberId, ProcessGeneration}, +}; + +async fn fixture() -> anyhow::Result<( + Fixture, + GroupRoute, + CellGroupApplier, + tempfile::TempDir, +)> { + let history = GroupHistory { + cluster: ClusterId::new([1; 16])?, + namespace: NamespaceId::new([31; 16])?, + group: GroupId::new([32; 16])?, + incarnation: IncarnationId::new([33; 16])?, + }; + let fixture = Fixture::start( + TransportOptions::default(), + vec![CellGroupBootstrap { + route: cellar_logic::cell_group::GroupRoute { + history, + range: CellRange::all(), + epoch: RouteEpoch::new(1)?, + birth_digest: [10; 32], + schema_digest: [9; 32], + }, + member: MemberId::new(1)?, + voters: (1..=3).map(MemberId::new).collect::, _>>()?, + learners: vec![], + }], + ) + .await?; + let peers = (1..=3) + .map(|member| { + Ok(RaftPeer { + replica: ReplicaIdentity { + scope: history, + member: MemberId::new(member)?, + node: NodeId::new([member as u8; 16])?, + disk: DiskLineage::new([member as u8 + 1; 16])?, + process: ProcessGeneration::new([member as u8 + 2; 16])?, + }, + label: format!("node-{member}"), + server_name: "localhost".into(), + control_address: if member == 1 { + fixture.transport.control_address() + } else { + format!("127.0.0.1:{}", 42000 + member * 3).parse()? + }, + data_address: if member == 1 { + fixture.transport.data_address() + } else { + format!("127.0.0.1:{}", 42001 + member * 3).parse()? + }, + query_address: if member == 1 { + fixture.transport.query_address() + } else { + format!("127.0.0.1:{}", 42002 + member * 3).parse()? + }, + }) + }) + .collect::>>()?; + let group = fixture.log.cell_group(&history)?; + let directory = tempfile::tempdir()?; + let base = directory.path().to_owned(); + let (applier, current) = fixture + .domain + .blocking(move || { + let images = cellar_ltx::generation::GenerationStore::open( + base.join("images"), + Arc::new(cellar_ltx::DirectFileSystem), + Default::default(), + )?; + let pages = ManifestPageStore::open( + &base.join("pages"), + Arc::new(cellar_ltx::DirectFileSystem), + 1024, + 16 * 1024 * 1024, + 16, + )?; + Ok::<_, anyhow::Error>(CellGroupApplier::recover( + group, + pages, + CellGroupApplyImages::Materialized { + images, + services: cellar_ltx::generation::ReadServices { + now_unix_millis: Arc::new(crate::asyncrt::wall_ms), + fill_random: Arc::new(|bytes| bytes.fill(1)), + }, + }, + ManifestLimits::default(), + crate::application::consensus::group_protocol::GroupEntryLimits::default(), + 100, + )?) + })? + .await??; + drop(current); + // Keep the directory until every real page/image owner has been released. + fixture + .transport + .enable_group_checkpoints(CellGroupCheckpointPool::new(CellGroupCheckpointLimits { + max_sources: 1, + max_retained_bytes: 32 * 1024 * 1024, + max_cells: 64, + max_transfer_bytes: 32 * 1024 * 1024, + max_chunk_bytes: 4093, + })?)?; + Ok(( + fixture, + GroupRoute { + scope: history, + schema_digest: [9; 32], + revision: 1, + peers, + }, + applier, + directory, + )) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn advisory_refresh_keeps_factory_and_physical_owner_but_revokes_old_stream( +) -> anyhow::Result<()> { + let (fixture, route, applier, _directory) = fixture().await?; + let registration = fixture.transport.register_physical_group(route.clone())?; + let checkpoint = fixture + .transport + .register_group_checkpoint(applier.clone())?; + let state = &fixture.transport.handle.0; + let (old, _) = state.checkpoint_endpoint(&route.scope)?; + let owner = state.registry.lock().unwrap().physical_owners[&route.scope].clone(); + let mut refreshed = route.clone(); + refreshed.revision += 1; + fixture + .transport + .install_physical(refreshed, false, Some(owner))?; + let wire = Route::new_group( + &route.peers[1].replica, + &route.peers[1].label, + &route.peers[0].replica, + &route.peers[0].label, + route.schema_digest, + )?; + assert!(matches!( + state.checkpoint_current(&old, &wire), + Err(TransportError::StalePeer) + )); + let (current, _) = state.checkpoint_endpoint(&route.scope)?; + assert!(Arc::ptr_eq(&old.registration, ¤t.registration)); + assert!(state.checkpoint_current(¤t, &wire).is_ok()); + drop(registration); + assert!(!current.active.load(Ordering::Acquire)); + assert!(!state + .registry + .lock() + .unwrap() + .physical_groups + .contains_key(&route.scope)); + assert!(!state + .registry + .lock() + .unwrap() + .physical_owners + .contains_key(&route.scope)); + assert!(state.checkpoint_endpoint(&route.scope).is_err()); + drop((checkpoint, applier, old, current)); + fixture.close().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn explicit_route_replacement_is_not_owned_by_old_registration() -> anyhow::Result<()> { + let (fixture, route, applier, _directory) = fixture().await?; + let old_registration = fixture.transport.register_physical_group(route.clone())?; + let old_checkpoint = fixture + .transport + .register_group_checkpoint(applier.clone())?; + let state = &fixture.transport.handle.0; + let old_owner = state.registry.lock().unwrap().physical_owners[&route.scope].clone(); + let mut replacement = route.clone(); + replacement.revision += 1; + fixture + .transport + .install_physical_group(replacement.clone())?; + let checkpoint = fixture + .transport + .register_group_checkpoint(applier.clone())?; + assert!(!old_checkpoint.endpoint.active.load(Ordering::Acquire)); + let mut stale_refresh = replacement.clone(); + stale_refresh.revision += 1; + assert!(matches!( + fixture + .transport + .install_physical(stale_refresh, false, Some(old_owner)), + Err(TransportError::StalePeer) + )); + drop(old_registration); + drop(old_checkpoint); + let (current, _) = state.checkpoint_endpoint(&route.scope)?; + assert_eq!(*current.route, replacement); + fixture.transport.remove_physical_group(&route.scope)?; + assert!(!current.active.load(Ordering::Acquire)); + drop((checkpoint, applier, current)); + fixture.close().await +} diff --git a/crates/cellar/durability/raft/group_snapshot_transport/seed/archive.rs b/crates/cellar/durability/raft/group_snapshot_transport/seed/archive.rs new file mode 100644 index 000000000..619c9bd8c --- /dev/null +++ b/crates/cellar/durability/raft/group_snapshot_transport/seed/archive.rs @@ -0,0 +1,144 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! The archive supplies bytes; the authenticated source and existing engine +//! grant still supply every installation, boot and membership fence. +use super::*; +use crate::raft::cell_groups::{ + ArchiveExecution, ArchiveSeedDescriptor, CellGroupArchiveError, CellGroupArchiveRegistration, + PreparedCellGroupCheckpoint, +}; + +impl Exchange { + pub(super) async fn receive_archive_checkpoint( + &mut self, + applier: CellGroupApplier, + descriptor: ArchiveSeedDescriptor, + registration: CellGroupArchiveRegistration, + ) -> Result { + let state = self.state.clone(); + let log = state.runtime.log(); + self.grant.check_log(log)?; + if registration.birth.route() != self.grant.root.route() { + return Err(TransportError::Identity.into()); + } + let wire = snapshots::Wire { + auth: &state.auth, + exporter: &self.exporter, + options: &state.options, + max_body: CHUNK_BYTES, + }; + let ready = wire.encode(READY_MAGIC, READY, &self.binding, &Ack { sequence: 0 }, &[])?; + state + .seed_process_io(&self.grant.process, self.grant.deadline, async { + write_head(&mut self.stream, &ready).await?; + self.stream.flush().await?; + Ok(()) + }) + .await?; + let execution = ArchiveExecution::new(state.domain.clone()); + let data = async { + let registry = self + .directory + .open_group_archive(&execution, ®istration.io, ®istration.birth) + .await?; + let upload = registry + .open_upload( + &execution, + ®istration.owners, + &descriptor.ticket, + ®istration.tails, + ) + .await?; + let receiver = work::receiver( + &state, + applier, + self.pool.clone(), + self.admission.clone(), + self.grant.clone(), + )?; + let input = receiver.input; + let download = async { + let tx = input.clone(); + let grant = self.grant.clone(); + let local = log.clone(); + upload + .download_seed_checkpoint(&execution, &descriptor.checkpoint, move |chunk| { + let tx = tx.clone(); + let grant = grant.clone(); + let local = local.clone(); + async move { + grant + .check_log(&local) + .map_err(CellGroupCheckpointError::from)?; + tx.send(workers::Input::Chunk(chunk)).await.map_err(|_| { + CellGroupArchiveError::Invalid("seed receiver closed") + })?; + grant + .check_log(&local) + .map_err(CellGroupCheckpointError::from)?; + Ok(()) + } + }) + .await?; + self.grant.check_log(log)?; + input + .send(workers::Input::End) + .await + .map_err(|_| TransportError::Closed)?; + drop(input); + Ok::<_, CellGroupSeedError>(()) + }; + let physical = async { + receiver + .result + .await + .map_err(|_| TransportError::Closed)? + .map_err(CellGroupSeedError::Transport) + }; + let (_, prepared) = futures_util::future::try_join(download, physical).await?; + if prepared.snapshot() != self.grant.snapshot() { + return Err(TransportError::Authentication.into()); + } + Ok::<_, CellGroupSeedError>(prepared) + }; + let result = state + .seed_process_until(&self.grant.process, self.grant.deadline, async { + Ok::<_, TransportError>(data.await) + }) + .await; + // Timeout/cancellation cannot advertise physical release while archive + // metadata/receiver work still owns pins or a blocking task. + execution.join().await; + let prepared = result??; + self.grant.check_log(log)?; + let ready = wire.encode(READY_MAGIC, READY, &self.binding, &Ack { sequence: 1 }, &[])?; + state + .seed_process_io(&self.grant.process, self.grant.deadline, async { + write_head(&mut self.stream, &ready).await?; + self.stream.flush().await?; + Ok(()) + }) + .await?; + // There is no peer-byte fallback. Only the source's authenticated End + // after its final live-authority check can authorize the engine fence. + let head = state + .seed_process_until( + &self.grant.process, + self.grant.deadline, + snapshots::read_bulk_head(&mut self.stream, None, CHUNK_MAGIC, 0, &state.options), + ) + .await?; + let packet: Packet = wire.decode(CHUNK_MAGIC, CHUNK, &head, Some(&self.binding))?; + if packet.sequence != 0 + || packet.part.is_some() + || packet.offset != 0 + || packet.length != 0 + || packet.digest != digest(&[]) + || head.payload_len != 0 + { + return Err(TransportError::Authentication.into()); + } + self.grant.check_log(log)?; + Ok(prepared) + } +} diff --git a/crates/cellar/durability/raft/group_snapshot_transport/seed/client.rs b/crates/cellar/durability/raft/group_snapshot_transport/seed/client.rs new file mode 100644 index 000000000..62147054c --- /dev/null +++ b/crates/cellar/durability/raft/group_snapshot_transport/seed/client.rs @@ -0,0 +1,331 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. +use super::*; + +impl RaftTransport { + /// Authenticate a live-quorum source, then reconcile a non-serving staging + /// mapping and permanent claim. Retry failures with a fresh exchange and the + /// same identities; an existing remote claim never permits recreating a disk. + pub async fn prepare_group_seed( + &self, + local: ReplicaIdentity, + source: RaftPeer, + directory: RangeDirectory, + birth: CellGroupBirth, + deadline: u64, + ) -> Result { + self.prepare_seed(local, source, directory, birth, deadline, None) + .await + } + + /// Seed an admitted managed learner under this exact installed physical + /// boot. A fresh catalog reservation and live quorum grant are both required. + pub(crate) async fn prepare_managed_group_seed( + &self, + local: ReplicaIdentity, + source: RaftPeer, + directory: RangeDirectory, + birth: CellGroupBirth, + deadline: u64, + authority: ManagedPeerSeed, + ) -> Result { + self.prepare_seed( + local, + source, + directory, + birth, + deadline, + Some(ManagedSeed { + host: authority.host, + expected: authority.authorization, + archive: None, + }), + ) + .await + } + + pub(crate) async fn prepare_managed_group_archive_seed( + &self, + local: ReplicaIdentity, + source: RaftPeer, + directory: RangeDirectory, + birth: CellGroupBirth, + deadline: u64, + managed: ManagedArchiveSeed, + ) -> Result { + self.prepare_seed( + local, + source, + directory, + birth, + deadline, + Some(ManagedSeed { + host: managed.host, + expected: managed.authorization, + archive: Some(managed.archive), + }), + ) + .await + } + + async fn prepare_seed( + &self, + local: ReplicaIdentity, + source: RaftPeer, + directory: RangeDirectory, + birth: CellGroupBirth, + deadline: u64, + managed_options: Option, + ) -> Result { + let (host, expected, archive) = match managed_options { + Some(managed) => (Some(managed.host), Some(managed.expected), managed.archive), + None => (None, None, None), + }; + let state = self.handle.0.clone(); + let log = state.runtime.shared()?.log().clone(); + let options = log.limits(); + if local.node != options.node + || local.disk != options.disk + || local.process != options.process + || local.scope != source.replica.scope + || local.scope != birth.route().history + || local.node == source.replica.node + || local.member == source.replica.member + { + return Err(TransportError::Identity.into()); + } + let deadline = deadline.min(self.seed_deadline()?); + let guard = state.reserve_task()?; + let process = state.seed_peer_epoch(&RaftPeerNode::from(&source))?; + let metadata = state + .seed_metadata + .acquire(seeds::META_BYTES) + .ok_or(TransportError::Overloaded)?; + let admission = Arc::new(ReceiveAdmission { + _wire: state + .incoming + .lane(Lane::Data) + .acquire(wire_charge(&state.options)) + .ok_or(TransportError::Overloaded)?, + _transfer: state + .snapshot_receive_budget + .acquire(snapshots::memory_bound(&state.options)) + .ok_or(TransportError::Overloaded)?, + }); + // Refuse before remote or local mutations if the node has no receiver pool. + let pool = self.group_checkpoint_pool()?; + let managed = if let Some(host) = &host { + let authorization = state + .seed_process_until(&process, deadline, async { + Ok::<_, TransportError>( + directory + .authorize_group_learner_seed( + &birth, + &log, + &local, + &source.replica, + host, + deadline, + ) + .await, + ) + }) + .await??; + if expected.as_ref().is_none_or(|expected| { + expected.permit() != authorization.permit() + || expected.target_boot() != authorization.target_boot() + }) { + return Err(TransportError::Refused.into()); + } + Some(grant::Managed { + authorization, + boot: host.boot_binding().map_err(CatalogError::from)?, + }) + } else { + None + }; + let now = state.domain.mono_ms().map_err(|_| TransportError::Closed)?; + let timeout_ms = deadline + .checked_sub(now) + .filter(|ms| *ms > 0) + .ok_or(TransportError::Timeout)?; + let nonce = state + .sequence + .fetch_update(Ordering::AcqRel, Ordering::Acquire, |n| n.checked_add(1)) + .map_err(|_| TransportError::Closed)? + .checked_add(1) + .ok_or(TransportError::Closed)?; + let mut request = Request { + route: Route::new_group( + &local, + state.auth.source(), + &source.replica, + &source.label, + birth.route().schema_digest, + )?, + nonce: nonce.to_string(), + timeout_ms, + from_s3: archive.is_some(), + }; + let connection = state.outgoing_connections.acquire(Lane::Data); + let mut stream = state + .seed_process_io(&process, deadline, async { + let tcp = TcpStream::connect(source.data_address).await?; + tcp.set_nodelay(true)?; + let socket = state.observed_socket(tcp, metrics::Lane::Data); + let name = rustls::pki_types::ServerName::try_from(source.server_name.clone()) + .map_err(|_| TransportError::Identity)?; + tokio_rustls::TlsConnector::from(state.tls.client.clone()) + .connect(name, socket) + .await + .map_err(|e| TransportError::Tls(e.to_string())) + }) + .await?; + let exporter = stream + .get_ref() + .1 + .export_keying_material([0; 32], EXPORT_LABEL, None) + .map_err(|e| TransportError::Tls(e.to_string()))?; + request.timeout_ms = deadline + .checked_sub(state.domain.mono_ms().map_err(|_| TransportError::Closed)?) + .filter(|ms| *ms > 0) + .ok_or(TransportError::Timeout)?; + let wire = snapshots::Wire { + auth: &state.auth, + exporter: &exporter, + options: &state.options, + max_body: META_BYTES, + }; + let head = wire.encode(REQUEST_MAGIC, REQUEST, &[], &request, &[])?; + let (reply, body, binding) = state + .seed_process_until(&process, deadline, async { + write_head(&mut stream, &head).await?; + stream.flush().await?; + let received = snapshots::read_bulk_head( + &mut stream, + None, + REPLY_MAGIC, + META_BYTES, + &state.options, + ) + .await?; + let reply: Reply = wire.decode(REPLY_MAGIC, REPLY, &received, Some(&head.tag))?; + if reply.route != request.route.reversed() + || reply.request != head.tag + || reply.nonce != request.nonce + || matches!(&reply.data, SeedData::Archive { .. }) != request.from_s3 + { + return Err(TransportError::Authentication); + } + let mut body = vec![0; received.payload_len]; + stream.read_exact(&mut body).await?; + if reply.digest != digest(&body) { + return Err(TransportError::Authentication); + } + Ok((reply, body, received.tag)) + }) + .await?; + let barrier = LogIndex::new(frame::decimal(&reply.barrier)?); + let snapshot = ::raft::prelude::Snapshot::parse_from_bytes(&body) + .map_err(|_| TransportError::Protocol)?; + if snapshot + .write_to_bytes() + .map_err(|_| TransportError::Protocol)? + != body + { + return Err(TransportError::Protocol.into()); + } + let root = checkpoint::seed_marker(&snapshot, &local, barrier)?; + checkpoint::seed_profile( + &snapshot, + managed.as_ref().map(|m| m.authorization.permit()), + )?; + checkpoint::seed_source(&snapshot, &source.replica)?; + let archive = match (reply.data, archive) { + (SeedData::Archive { descriptor }, Some(registration)) => { + Some((descriptor, registration)) + } + (SeedData::Peer, None) => None, + _ => return Err(TransportError::Authentication.into()), + }; + let grant = Arc::new(Grant { + source: source.replica.clone(), + managed, + local, + root, + snapshot, + deadline, + owner: log.owner(), + domain: state.domain.clone(), + process, + stop: state.stop.subscribe(), + _metadata: metadata, + _request: Arc::new(guard), + }); + let group = state + .seed_process_until(&grant.process, deadline, async { + state + .domain + .run(async { + match &host { + Some(host) => { + directory + .stage_managed_group_learner(&birth, &log, grant.clone(), host) + .await + } + None => { + directory + .stage_group_learner(&birth, &log, grant.clone()) + .await + } + } + }) + .await + .map_err(|_| TransportError::Closed) + }) + .await??; + grant.check_log(&log)?; + group + .consensus_claimed + .compare_exchange(false, true, Ordering::AcqRel, Ordering::Acquire) + .map_err(|_| RuntimeError::AlreadyAttached) + .map_err(TransportError::from)?; + let lease = Arc::new(Lease(group.consensus_claimed.clone())); + let (command, ready) = oneshot::channel(); + let (response, result) = oneshot::channel(); + let exchange = Exchange { + host, + state: state.clone(), + stream, + exporter, + binding, + grant, + group: group.clone(), + lease, + directory, + birth, + pool, + admission, + archive, + _connection: connection, + operation: OperationId::new(nonce), + }; + let guard = state.reserve_task()?; + state.clone().spawn_reserved(guard, async move { + let outcome = async { + let applier = state + .seed_process_until(&exchange.grant.process, deadline, async { + ready.await.map_err(|_| TransportError::Closed) + }) + .await?; + exchange.install(applier).await + } + .await; + let _ = response.send(outcome); + })?; + Ok(CellGroupSeedSession { + group, + command, + result, + }) + } +} diff --git a/crates/cellar/durability/raft/group_snapshot_transport/seed/grant.rs b/crates/cellar/durability/raft/group_snapshot_transport/seed/grant.rs new file mode 100644 index 000000000..8eb6c4302 --- /dev/null +++ b/crates/cellar/durability/raft/group_snapshot_transport/seed/grant.rs @@ -0,0 +1,181 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. +use super::*; +use crate::raft::{ + fleet_catalog::FleetPermitAuthorization, + fleet_stop::{BootBinding, Guard}, + member_state::FleetGroupBinding, +}; + +pub(super) struct Managed { + pub authorization: FleetPermitAuthorization, + pub boot: BootBinding, +} + +pub(super) fn check_boot(guard: Option<&Guard>, expected: &BootBinding) -> Result<(), LogError> { + if guard.is_none_or(|guard| !guard.admits_group_start(expected)) { + return Err(LogError::WrongIdentity); + } + Ok(()) +} + +/// Only a verified TLS response constructs this capability. It remains live +/// through queued physical installation, including cancelled callers. +pub(crate) struct Grant { + pub(super) managed: Option, + pub(super) local: ReplicaIdentity, + pub(super) source: ReplicaIdentity, + pub(super) root: GroupManifest, + pub(super) snapshot: ::raft::prelude::Snapshot, + pub(super) deadline: u64, + pub(super) owner: Arc<()>, + pub(super) domain: DomainToken, + pub(super) process: watch::Receiver, + pub(super) stop: watch::Receiver, + pub(super) _metadata: Permit, + // Erase the transport graph while retaining its joined task ownership. + pub(super) _request: Arc, +} + +impl Grant { + pub(crate) fn local(&self) -> &ReplicaIdentity { + &self.local + } + + pub(crate) fn source(&self) -> &ReplicaIdentity { + &self.source + } + + pub(crate) fn snapshot(&self) -> &::raft::prelude::Snapshot { + &self.snapshot + } + + pub(crate) fn check( + &self, + owner: &Arc<()>, + history: GroupHistory, + local: &ReplicaIdentity, + ) -> Result<(), LogError> { + self.check_target(owner, history, local)?; + if *self.process.borrow() { + return Err(LogError::WrongIdentity); + } + Ok(()) + } + + // Source liveness authorizes submission. After actual engine sync succeeds, + // only the destination and original operation bounds govern local completion. + fn check_target( + &self, + owner: &Arc<()>, + history: GroupHistory, + local: &ReplicaIdentity, + ) -> Result<(), LogError> { + if !Arc::ptr_eq(owner, &self.owner) + || history != self.local.scope + || local.member != self.local.member + || local.node != self.local.node + || local.disk != self.local.disk + || local.process != self.local.process + || *self.stop.borrow() + || !self.domain.mono_ms().is_ok_and(|now| now < self.deadline) + { + return Err(LogError::WrongIdentity); + } + Ok(()) + } + + pub(crate) fn check_lifecycle(&self, guard: Option<&Guard>) -> Result<(), LogError> { + match &self.managed { + Some(managed) => check_boot(guard, &managed.boot), + None => Ok(()), + } + } + + pub(crate) fn initial_membership( + &self, + conf: &::raft::prelude::ConfState, + ) -> Result { + checkpoint::seed_initial_membership(&self.snapshot, conf) + .map_err(|e| LogError::Invalid(e.to_string())) + } + + pub(crate) fn deadline(&self) -> u64 { + self.deadline + } + + pub(crate) fn managed_binding(&self) -> Option<&FleetGroupBinding> { + self.managed + .as_ref() + .map(|m| m.authorization.permit().binding()) + } + + pub(crate) fn check_authorization( + &self, + authorization: &FleetPermitAuthorization, + ) -> Result<(), LogError> { + if self.managed.as_ref().is_none_or(|managed| { + managed.authorization.permit() != authorization.permit() + || authorization.minimum_trust_revision() + < managed.authorization.minimum_trust_revision() + || authorization.target_boot() != Some(&managed.boot) + }) { + return Err(LogError::WrongIdentity); + } + Ok(()) + } + + pub(crate) fn check_log(&self, log: &ReplicaLog) -> Result<(), LogError> { + self.check_target_log(log)?; + if *self.process.borrow() { + return Err(LogError::WrongIdentity); + } + Ok(()) + } + + pub(super) fn check_target_log(&self, log: &ReplicaLog) -> Result<(), LogError> { + let options = log.limits(); + self.check_target( + &log.owner(), + self.local.scope, + &ReplicaIdentity { + scope: self.local.scope, + node: options.node, + disk: options.disk, + process: options.process, + member: self.local.member, + }, + )?; + self.check_lifecycle(log.lifecycle_guard()?.as_ref()) + } + + pub(crate) fn initial_configuration( + &self, + bootstrap: &CellGroupBootstrap, + ) -> Result<::raft::prelude::ConfState, LogError> { + let conf = ::raft::prelude::ConfState { + voters: bootstrap.voters.iter().map(|id| id.get()).collect(), + learners: bootstrap.learners.iter().map(|id| id.get()).collect(), + ..Default::default() + }; + crate::raft::cache::validate_conf(&conf)?; + crate::raft::member_state::GroupMembership::initial(&conf).validate(&conf, 1)?; + if bootstrap.member != self.local.member + || &bootstrap.route != self.root.route() + || crate::raft::member_state::members(&conf).contains(&bootstrap.member.get()) + { + return Err(LogError::WrongIdentity); + } + checkpoint::seed_profile( + &self.snapshot, + self.managed.as_ref().map(|m| m.authorization.permit()), + ) + .map_err(|e| LogError::Invalid(e.to_string()))?; + checkpoint::seed_birth(&self.snapshot, &bootstrap.route, &conf) + .map_err(|e| LogError::Invalid(e.to_string()))?; + Ok(conf) + } +} + +#[cfg(test)] +#[path = "grant_tests.rs"] +mod tests; diff --git a/crates/cellar/durability/raft/group_snapshot_transport/seed/grant_tests.rs b/crates/cellar/durability/raft/group_snapshot_transport/seed/grant_tests.rs new file mode 100644 index 000000000..ccf7949d1 --- /dev/null +++ b/crates/cellar/durability/raft/group_snapshot_transport/seed/grant_tests.rs @@ -0,0 +1,77 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Local guard transition fixtures do not establish a sync or seed authority. +use super::*; +use crate::raft::fleet_stop::{LifecyclePurpose, LifecycleRecord, LifecycleUpdate, StopSelector}; + +fn boot() -> BootBinding { + BootBinding { + catalog: [1; 32], + fleet: [2; 16], + node: [3; 16], + disk: [4; 16], + engine: [5; 16], + boot: [6; 16], + process: [7; 16], + epoch: 1, + join_key: crate::peer_probe::PeerProbeSigner::from_secret([8; 32]).public_key_bytes(), + } +} + +#[test] +fn managed_seed_guard_requires_exact_open_boot() { + let boot = boot(); + assert!(check_boot(None, &boot).is_err()); + let open = LifecycleRecord::empty() + .change(LifecycleUpdate::Boot { + boot: boot.clone(), + deadline: 1000, + }) + .unwrap(); + check_boot(open.guard.as_ref(), &boot).unwrap(); + for part in 0..3 { + let mut changed = boot.clone(); + match part { + 0 => changed.process = [9; 16], + 1 => changed.engine = [10; 16], + _ => changed.boot = [11; 16], + } + assert!(check_boot(open.guard.as_ref(), &changed).is_err()); + } +} + +#[test] +fn managed_seed_guard_refuses_quiescence_and_fence() { + let boot = boot(); + let open = LifecycleRecord::empty() + .change(LifecycleUpdate::Boot { + boot: boot.clone(), + deadline: 1000, + }) + .unwrap(); + let selector = StopSelector { + purpose: LifecyclePurpose::Maintenance, + boot: boot.clone(), + epoch: 2, + operation: [12; 16], + finalization: [13; 16], + request: [14; 16], + }; + let quiescing = open + .change(LifecycleUpdate::Quiesce { + selector: selector.clone(), + deadline: 1000, + }) + .unwrap(); + assert!(check_boot(quiescing.guard.as_ref(), &boot).is_err()); + let fenced = quiescing + .change(LifecycleUpdate::Fence { + boot: boot.clone(), + epoch: 2, + operation: selector.operation, + request: [15; 16], + deadline: 1000, + }) + .unwrap(); + assert!(check_boot(fenced.guard.as_ref(), &boot).is_err()); +} diff --git a/crates/cellar/durability/raft/group_snapshot_transport/seed/install.rs b/crates/cellar/durability/raft/group_snapshot_transport/seed/install.rs new file mode 100644 index 000000000..32aaa7c9e --- /dev/null +++ b/crates/cellar/durability/raft/group_snapshot_transport/seed/install.rs @@ -0,0 +1,205 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. +use super::*; +use crate::raft::cell_groups::checkpoint::wire::Chunk; + +impl Exchange { + pub(super) async fn install( + mut self, + applier: CellGroupApplier, + ) -> Result { + let state = self.state.clone(); + let log = state.runtime.log(); + self.grant.check_log(log)?; + if applier.history() != self.group.identity().scope + || !applier.belongs_to_engine(&log.owner()) + { + return Err(TransportError::Identity.into()); + } + let prepared = if let Some((descriptor, registration)) = self.archive.take() { + Box::pin(self.receive_archive_checkpoint(applier, descriptor, registration)).await? + } else { + let receiver = work::receiver( + &state, + applier, + self.pool.clone(), + self.admission.clone(), + self.grant.clone(), + )?; + let wire = snapshots::Wire { + auth: &state.auth, + exporter: &self.exporter, + options: &state.options, + max_body: CHUNK_BYTES, + }; + let mut result = receiver.result; + let receiving = async { + let head = + wire.encode(READY_MAGIC, READY, &self.binding, &Ack { sequence: 0 }, &[])?; + state + .seed_process_io(&self.grant.process, self.grant.deadline, async { + write_head(&mut self.stream, &head).await?; + self.stream.flush().await?; + Ok(()) + }) + .await?; + let mut sequence = 0_u64; + loop { + self.grant.check_log(log)?; + let head = state + .seed_process_io( + &self.grant.process, + self.grant.deadline, + snapshots::read_bulk_head( + &mut self.stream, + None, + CHUNK_MAGIC, + CHUNK_BYTES.min(state.options.snapshot_chunk_bytes), + &state.options, + ), + ) + .await?; + let packet: Packet = + wire.decode(CHUNK_MAGIC, CHUNK, &head, Some(&self.binding))?; + if packet.sequence != sequence { + return Err(TransportError::Replay.into()); + } + let mut bytes = vec![0; head.payload_len]; + state + .seed_process_io(&self.grant.process, self.grant.deadline, async { + self.stream.read_exact(&mut bytes).await?; + Ok(()) + }) + .await?; + if digest(&bytes) != packet.digest { + return Err(TransportError::Authentication.into()); + } + let input = match packet.part { + Some(part) => { + if bytes.is_empty() { + return Err(TransportError::Protocol.into()); + } + workers::Input::Chunk(Chunk { + part, + offset: packet.offset, + length: packet.length, + bytes, + }) + } + None => { + if !bytes.is_empty() || packet.offset != 0 || packet.length != 0 { + return Err(TransportError::Protocol.into()); + } + workers::Input::End + } + }; + let ending = matches!(input, workers::Input::End); + self.grant.check_log(log)?; + state + .seed_process_until(&self.grant.process, self.grant.deadline, async { + receiver + .input + .send(input) + .await + .map_err(|_| TransportError::Closed) + }) + .await?; + if ending { + break; + } + sequence = sequence.checked_add(1).ok_or(TransportError::Limit)?; + } + Ok::<(), CellGroupSeedError>(()) + }; + // A physical verifier can reject a part while the peer keeps the next + // header/body open. Observe it throughout network work rather than + // retaining the connection and admission until the peer deadline. + crate::asyncrt::select_biased! { "physical seed verification failure precedes more network input"; + outcome = &mut result => { + outcome.map_err(|_| TransportError::Closed)??; + // Successful physical completion requires the explicit End + // sent by the receiving branch, which completes in that poll. + return Err(TransportError::Protocol.into()); + }, + outcome = receiving => outcome?, + } + drop(receiver.input); + let prepared = state + .seed_process_until(&self.grant.process, self.grant.deadline, async { + result.await.map_err(|_| TransportError::Closed)? + }) + .await?; + prepared + }; + self.grant.check_log(log)?; + if prepared.snapshot() != self.grant.snapshot() { + return Err(TransportError::Authentication.into()); + } + if let Some(host) = &self.host { + let current = state + .seed_process_until(&self.grant.process, self.grant.deadline, async { + Ok::<_, TransportError>( + self.directory + .authorize_group_learner_seed( + &self.birth, + log, + self.grant.local(), + self.grant.source(), + host, + self.grant.deadline, + ) + .await, + ) + }) + .await??; + self.grant.check_authorization(¤t)?; + } + let current = prepared.manifest().clone(); + let mut write = GroupWrite::new(&self.group); + let metadata = prepared.snapshot().get_metadata(); + write.hard_state = Some(::raft::prelude::HardState { + term: metadata.term, + commit: metadata.index, + ..Default::default() + }); + write.snapshot = Some(SnapshotChange::CellGroup(prepared)); + write.seed_grant = Some(self.grant.clone()); + write.consensus_pin = Some(self.lease.clone()); + // Once accepted, retain the network task, grant and consensus lease until + // actual persistence finishes. Caller cancellation/transport close cannot + // turn an in-flight fence into an unowned or successful empty bootstrap. + log.persist_cells(self.operation, vec![write]).await?; + self.grant.check_target_log(log)?; + // The engine checked source liveness immediately before entering its + // fence. A later source restart cannot revoke that completed durable + // result. Still require this destination's exact live boot, transport, + // original deadline, and freshly authorized catalog bootstrap. + let permit = state + .seed_until(self.grant.deadline, async { + Ok::<_, TransportError>(match &self.host { + Some(host) => { + self.directory + .bootstrap_managed_joined_group_member( + &self.birth, + self.grant.local.member, + log, + host, + ) + .await + } + None => { + self.directory + .bootstrap_joined_group_member( + &self.birth, + self.grant.local.member, + log, + ) + .await + } + }) + }) + .await??; + self.grant.check_target_log(log)?; + drop(self.lease); + Ok((permit, current)) + } +} diff --git a/crates/cellar/durability/raft/group_snapshot_transport/seed/mod.rs b/crates/cellar/durability/raft/group_snapshot_transport/seed/mod.rs new file mode 100644 index 000000000..be31f4f36 --- /dev/null +++ b/crates/cellar/durability/raft/group_snapshot_transport/seed/mod.rs @@ -0,0 +1,210 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Pull one authenticated complete-group seed into a non-serving staged member. +use super::*; +use crate::raft::{ + cache::{GroupLog, GroupWrite, SnapshotChange}, + catalog::{CatalogError, CellGroupBirth, CellGroupBootstrapPermit, RangeDirectory}, + cell_groups::{checkpoint, CellGroupSeedAuthority, GroupManifest, PreparedGroupManifest}, + storage::{CellGroupBootstrap, LogError, ReplicaLog}, +}; +use cellar_logic::consensus::{LogIndex, OperationId}; +mod archive; +mod client; +mod grant; +mod install; +mod server; +mod work; +pub(crate) use grant::Grant; + +pub(in crate::raft::transport) const REQUEST_MAGIC: &[u8; 4] = b"CLGJ"; +pub(in crate::raft::transport) const EXPORT_LABEL: &[u8] = b"EXPORTER-cellar-group-seed-1"; +const REPLY_MAGIC: &[u8; 4] = b"CLGK"; +const READY_MAGIC: &[u8; 4] = b"CLGL"; +const CHUNK_MAGIC: &[u8; 4] = b"CLGM"; +const REQUEST: &[u8] = b"cellar-group-seed-request-1"; +const REPLY: &[u8] = b"cellar-group-seed-grant-1"; +const READY: &[u8] = b"cellar-group-seed-ready-1"; +const CHUNK: &[u8] = b"cellar-group-seed-chunk-1"; +const META_BYTES: usize = 64 * 1024; + +type Stream = tokio_rustls::client::TlsStream>; +type Installed = (CellGroupBootstrapPermit, PreparedGroupManifest); + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Request { + route: Route, + nonce: String, + timeout_ms: u64, + from_s3: bool, +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Reply { + route: Route, + request: [u8; 32], + nonce: String, + barrier: String, + digest: [u8; 32], + data: SeedData, +} + +#[derive(Serialize, Deserialize)] +#[serde(tag = "kind", rename_all = "snake_case", deny_unknown_fields)] +enum SeedData { + Peer, + Archive { + descriptor: crate::raft::cell_groups::ArchiveSeedDescriptor, + }, +} + +#[derive(Debug)] +pub enum CellGroupSeedError { + Transport(TransportError), + Catalog(CatalogError), + Log(LogError), + Checkpoint(CellGroupCheckpointError), + Archive(Box), +} + +impl std::fmt::Display for CellGroupSeedError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "cell-group seed: {self:?}") + } +} + +impl std::error::Error for CellGroupSeedError { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + Some(match self { + Self::Transport(e) => e, + Self::Catalog(e) => e, + Self::Log(e) => e, + Self::Checkpoint(e) => e, + Self::Archive(e) => e.as_ref(), + }) + } +} + +impl From for CellGroupSeedError { + fn from(e: TransportError) -> Self { + Self::Transport(e) + } +} + +impl From for CellGroupSeedError { + fn from(e: CatalogError) -> Self { + Self::Catalog(e) + } +} + +impl From for CellGroupSeedError { + fn from(e: LogError) -> Self { + Self::Log(e) + } +} + +impl From for CellGroupSeedError { + fn from(e: CellGroupCheckpointError) -> Self { + Self::Checkpoint(e) + } +} + +impl From for CellGroupSeedError { + fn from(error: crate::raft::cell_groups::CellGroupArchiveError) -> Self { + Self::Archive(Box::new(error)) + } +} + +impl From for CellGroupSeedError { + fn from(error: crate::raft::cell_groups::CellGroupArchiveBindingError) -> Self { + match error { + crate::raft::cell_groups::CellGroupArchiveBindingError::Catalog(error) => { + Self::Catalog(error) + } + crate::raft::cell_groups::CellGroupArchiveBindingError::Archive(error) => { + Self::Archive(Box::new(error)) + } + } + } +} + +pub(crate) struct ManagedPeerSeed { + pub host: crate::raft::fleet_catalog::FleetHostAuthorization, + pub authorization: crate::raft::fleet_catalog::FleetPermitAuthorization, +} + +pub(crate) struct ManagedArchiveSeed { + pub host: crate::raft::fleet_catalog::FleetHostAuthorization, + pub authorization: crate::raft::fleet_catalog::FleetPermitAuthorization, + pub archive: crate::raft::cell_groups::CellGroupArchiveRegistration, +} + +struct ManagedSeed { + host: crate::raft::fleet_catalog::FleetHostAuthorization, + expected: crate::raft::fleet_catalog::FleetPermitAuthorization, + archive: Option, +} + +/// A staged, non-serving destination. Dropping it before install closes the +/// exchange; retry with a fresh grant and the same permanent engine/member IDs. +/// Recover the staged group with `CellGroupApplier::recover` before install so +/// the selected base and its physical retention owner are established, even for +/// an empty genesis. Once install is accepted, cancellation does not abandon +/// queued engine work. +/// An unknown result must be reconciled through bootstrap_joined_group_member. +pub struct CellGroupSeedSession { + group: GroupLog, + command: oneshot::Sender, + result: oneshot::Receiver>, +} + +impl CellGroupSeedSession { + pub fn group(&self) -> &GroupLog { + &self.group + } + + /// Use the same stores on retry. Only a complete authenticated End, verified + /// physical closure and actual engine fence can return the startup permit. + pub async fn install(self, applier: CellGroupApplier) -> Result { + self.command + .send(applier) + .map_err(|_| TransportError::Closed)?; + self.result.await.map_err(|_| TransportError::Closed)? + } +} + +struct Lease(Arc); + +impl Drop for Lease { + fn drop(&mut self) { + self.0.store(false, Ordering::Release); + } +} + +struct ReceiveAdmission { + _wire: Permit, + _transfer: Permit, +} + +struct Exchange { + host: Option, + state: Arc, + stream: Stream, + exporter: [u8; 32], + binding: [u8; 32], + grant: Arc, + group: GroupLog, + lease: Arc, + directory: RangeDirectory, + birth: CellGroupBirth, + pool: CellGroupCheckpointPool, + admission: Arc, + archive: Option<( + crate::raft::cell_groups::ArchiveSeedDescriptor, + crate::raft::cell_groups::CellGroupArchiveRegistration, + )>, + _connection: resources::ConnectionOwner, + operation: OperationId, +} diff --git a/crates/cellar/durability/raft/group_snapshot_transport/seed/server.rs b/crates/cellar/durability/raft/group_snapshot_transport/seed/server.rs new file mode 100644 index 000000000..63db366e8 --- /dev/null +++ b/crates/cellar/durability/raft/group_snapshot_transport/seed/server.rs @@ -0,0 +1,295 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. +use super::*; + +impl State { + pub(in crate::raft::transport) async fn receive_group_seed< + S: AsyncRead + AsyncWrite + Unpin, + >( + self: &Arc, + stream: &mut S, + exporter: &[u8; 32], + prefix: [u8; frame::PREFIX], + ) -> Result<(), TransportError> { + let _connection = self + .snapshot_connections + .clone() + .try_acquire_owned() + .map_err(|_| TransportError::Overloaded)?; + let _metadata = Arc::new( + self.seed_metadata + .acquire(seeds::META_BYTES) + .ok_or(TransportError::Overloaded)?, + ); + let wire = snapshots::Wire { + auth: &self.auth, + exporter, + options: &self.options, + max_body: META_BYTES, + }; + let head = self + .bounded( + None, + snapshots::read_bulk_head(stream, Some(prefix), REQUEST_MAGIC, 0, &self.options), + ) + .await?; + let request: Request = wire.decode(REQUEST_MAGIC, REQUEST, &head, None)?; + if frame::decimal(&request.nonce)? == 0 + || request.timeout_ms == 0 + || request.timeout_ms > 3_600_000 + { + return Err(TransportError::Protocol); + } + let peer = self.authorize_physical(&request.route)?; + let (target, source) = request.route.decode_group()?; + let (endpoint, pool) = self.checkpoint_endpoint(&source.scope)?; + let deadline = self + .domain + .mono_ms() + .map_err(|_| TransportError::Closed)? + .saturating_add(request.timeout_ms.min(self.options.snapshot_timeout_ms)); + let charge = wire_charge(&self.options); + let admission = Arc::new(Admission { + _source: None, + _node: self + .outgoing + .lane(Lane::Data) + .acquire(charge) + .ok_or(TransportError::Overloaded)?, + _peer: peer + .outgoing + .lane(Lane::Data) + .acquire(charge) + .ok_or(TransportError::Overloaded)?, + _transfer: self + .snapshot_source_budget + .acquire(snapshots::memory_bound(&self.options)) + .ok_or(TransportError::Overloaded)?, + }); + let authority = self + .seed_until(deadline, async { + Ok(self + .runtime + .shared()? + .seed_authority(target.clone(), deadline) + .await?) + }) + .await?; + let barrier = authority.barrier().index; + self.checkpoint_current(&endpoint, &request.route)?; + if request.from_s3 { + let client = self + .registry + .lock() + .map_err(|_| TransportError::Closed)? + .group_archive_seeds + .clone() + .ok_or(TransportError::Refused)?; + let source = self + .seed_until(deadline, async { + client + .prepare(authority, Arc::new((admission.clone(), _metadata.clone()))) + .await + .map_err(|error| { + tracing::warn!(%error, "archive seed source refused"); + match error.as_ref() { + crate::raft::cell_groups::CellGroupArchiveControlError::Stopped => { + TransportError::Closed + } + crate::raft::cell_groups::CellGroupArchiveControlError::Runtime( + error, + ) => TransportError::Runtime(error.clone()), + crate::raft::cell_groups::CellGroupArchiveControlError::Invalid( + _, + ) => TransportError::Refused, + _ => TransportError::Io(error.to_string()), + } + }) + }) + .await?; + source + .check() + .map_err(|e| TransportError::Io(e.to_string()))?; + let body = source + .snapshot() + .write_to_bytes() + .map_err(|_| TransportError::Protocol)?; + if body.len() > META_BYTES { + return Err(TransportError::Limit); + } + let reply = wire.encode( + REPLY_MAGIC, + REPLY, + &head.tag, + &Reply { + route: request.route.reversed(), + request: head.tag, + nonce: request.nonce, + barrier: source.barrier().get().to_string(), + digest: digest(&body), + data: SeedData::Archive { + descriptor: source.descriptor().clone(), + }, + }, + &body, + )?; + let binding = reply.tag; + self.checkpoint_current(&endpoint, &request.route)?; + self.snapshot_io(&peer, deadline, async { + write_head(stream, &reply).await?; + stream.write_all(&body).await?; + stream.flush().await?; + Ok(()) + }) + .await?; + for sequence in [0, 1] { + let ready = self + .seed_until( + deadline, + snapshots::read_bulk_head(stream, None, READY_MAGIC, 0, &self.options), + ) + .await?; + let ready: Ack = wire.decode(READY_MAGIC, READY, &ready, Some(&binding))?; + if ready.sequence != sequence { + return Err(TransportError::Replay); + } + source + .check() + .map_err(|e| TransportError::Io(e.to_string()))?; + self.checkpoint_current(&endpoint, &request.route)?; + } + // The second authenticated acknowledgement is sent only after all + // S3 bytes are independently prepared in the destination's stores. + // Its local pins then own durability through the engine fence. + let packet = Packet { + sequence: 0, + part: None, + offset: 0, + length: 0, + digest: digest(&[]), + }; + let end = wire.encode(CHUNK_MAGIC, CHUNK, &binding, &packet, &[])?; + source + .check() + .map_err(|e| TransportError::Io(e.to_string()))?; + self.snapshot_io(&peer, deadline, async { + write_head(stream, &end).await?; + stream.flush().await?; + Ok(()) + }) + .await?; + source + .check() + .map_err(|e| TransportError::Io(e.to_string()))?; + return Ok(()); + } + let work::Source { + metadata, + mut chunks, + completed, + } = work::source(self, endpoint.clone(), authority, pool, admission.clone())?; + let snapshot = self + .seed_until(deadline, async { + metadata.await.map_err(|_| TransportError::Closed)? + }) + .await?; + self.checkpoint_current(&endpoint, &request.route)?; + let body = snapshot + .write_to_bytes() + .map_err(|_| TransportError::Protocol)?; + if body.len() > META_BYTES { + return Err(TransportError::Limit); + } + let reply = wire.encode( + REPLY_MAGIC, + REPLY, + &head.tag, + &Reply { + route: request.route.reversed(), + request: head.tag, + nonce: request.nonce, + barrier: barrier.get().to_string(), + digest: digest(&body), + data: SeedData::Peer, + }, + &body, + )?; + let binding = reply.tag; + self.snapshot_io(&peer, deadline, async { + write_head(stream, &reply).await?; + stream.write_all(&body).await?; + stream.flush().await?; + Ok(()) + }) + .await?; + drop(body); + // The destination may need catalog I/O and store creation before it can + // accept bytes. Wait within the original total deadline, not one I/O tick. + let ready = self + .seed_until( + deadline, + snapshots::read_bulk_head(stream, None, READY_MAGIC, 0, &self.options), + ) + .await?; + let ready: Ack = wire.decode(READY_MAGIC, READY, &ready, Some(&binding))?; + if ready.sequence != 0 { + return Err(TransportError::Replay); + } + let _admission = admission; + let mut sequence = 0_u64; + let mut completed = Some(completed); + loop { + self.checkpoint_current(&endpoint, &request.route)?; + let next = self + .snapshot_io(&peer, deadline, async { + chunks.recv().await.ok_or(TransportError::Closed) + }) + .await?; + let (part, offset, length, bytes) = match next { + workers::Output::Chunk(chunk) => { + (Some(chunk.part), chunk.offset, chunk.length, chunk.bytes) + } + workers::Output::End(result) => { + result?; + (None, 0, 0, Vec::new()) + } + }; + if bytes.len() > CHUNK_BYTES.min(self.options.snapshot_chunk_bytes) { + return Err(TransportError::Limit); + } + let ending = part.is_none(); + let final_source = if ending { + let completed = completed.take().ok_or(TransportError::Protocol)?; + let source = self + .seed_until(deadline, async { + completed.await.map_err(|_| TransportError::Closed) + }) + .await?; + source.check()?; + Some(source) + } else { + None + }; + let packet = Packet { + sequence, + part, + offset, + length, + digest: digest(&bytes), + }; + let head = wire.encode(CHUNK_MAGIC, CHUNK, &binding, &packet, &bytes)?; + self.checkpoint_current(&endpoint, &request.route)?; + self.snapshot_io(&peer, deadline, async { + write_head(stream, &head).await?; + stream.write_all(&bytes).await?; + stream.flush().await?; + Ok(()) + }) + .await?; + if let Some(source) = final_source { + source.check()?; + return Ok(()); + } + sequence = sequence.checked_add(1).ok_or(TransportError::Limit)?; + } + } +} diff --git a/crates/cellar/durability/raft/group_snapshot_transport/seed/work.rs b/crates/cellar/durability/raft/group_snapshot_transport/seed/work.rs new file mode 100644 index 000000000..30feeaa93 --- /dev/null +++ b/crates/cellar/durability/raft/group_snapshot_transport/seed/work.rs @@ -0,0 +1,119 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. +use super::*; +use crate::raft::cell_groups::checkpoint::wire::Chunk; + +pub(super) struct Source { + pub metadata: oneshot::Receiver>, + pub chunks: mpsc::Receiver, + pub completed: oneshot::Receiver, +} + +pub(super) fn source( + state: &Arc, + endpoint: Arc, + authority: CellGroupSeedAuthority, + pool: CellGroupCheckpointPool, + admission: Arc, +) -> Result { + let guard = state.reserve_task()?; + let (metadata, result) = oneshot::channel(); + let (tx, rx) = mpsc::channel(1); + let (finished, completed) = oneshot::channel(); + state + .domain + .blocking(move || { + let _guard = guard; + let _admission = admission; + let captured = authority + .capture(&pool) + .and_then(|source| { + let snapshot = source.snapshot()?; + Ok((source, snapshot)) + }) + .map_err(checkpoint_error); + let (mut source, snapshot) = match captured { + Ok(value) => value, + Err(error) => { + let _ = metadata.send(Err(error)); + return; + } + }; + if metadata.send(Ok(snapshot)).is_err() { + return; + } + let result = source + .stream(|chunk| { + if !endpoint.active.load(Ordering::Acquire) { + return Err(CellGroupCheckpointError::Invalid("revoked seed endpoint")); + } + tx.blocking_send(workers::Output::Chunk(Chunk::copy(chunk))) + .map_err(|_| { + CellGroupCheckpointError::Sink(std::io::Error::new( + std::io::ErrorKind::BrokenPipe, + "seed connection closed", + )) + }) + }) + .map_err(checkpoint_error); + let success = result.is_ok(); + if tx.blocking_send(workers::Output::End(result)).is_ok() && success { + let _ = finished.send(source); + } + }) + .map_err(|_| TransportError::Closed)? + .detach(); + Ok(Source { + metadata: result, + chunks: rx, + completed, + }) +} + +pub(super) fn receiver( + state: &Arc, + applier: CellGroupApplier, + pool: CellGroupCheckpointPool, + admission: Arc, + grant: Arc, +) -> Result { + let guard = state.reserve_task()?; + let log = state.runtime.log().clone(); + let scratch = log.snapshot_scratch(); + let directory = log.limits().directory.clone(); + let (input, mut rx) = mpsc::channel(1); + let (tx, result) = oneshot::channel(); + state + .domain + .blocking(move || { + let _guard = guard; + let _admission = admission; + let outcome = (|| { + let mut receiver = pool.receive(applier, scratch, directory)?; + loop { + grant.check_log(&log)?; + match rx.blocking_recv() { + Some(workers::Input::Chunk(chunk)) => chunk.append(&mut receiver)?, + Some(workers::Input::End) => { + grant.check_log(&log)?; + let prepared = receiver.finish()?; + grant.check_log(&log)?; + if prepared.snapshot() != grant.snapshot() { + return Err(CellGroupCheckpointError::Invalid( + "seed descriptor changed", + )); + } + return Ok(prepared); + } + None => { + return Err(CellGroupCheckpointError::Invalid("incomplete seed stream")) + } + } + } + })() + .map_err(checkpoint_error); + let _ = tx.send(outcome); + }) + .map_err(|_| TransportError::Closed)? + .detach(); + Ok(workers::Receive { input, result }) +} diff --git a/crates/cellar/durability/raft/group_snapshot_transport/send.rs b/crates/cellar/durability/raft/group_snapshot_transport/send.rs new file mode 100644 index 000000000..f4575095a --- /dev/null +++ b/crates/cellar/durability/raft/group_snapshot_transport/send.rs @@ -0,0 +1,242 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. +use super::*; + +impl RaftTransport { + pub(in crate::raft::transport) fn submit_group_checkpoint( + &self, + history: &GroupHistory, + message: Message, + source: Option>, + ) -> Result { + let state = &self.handle.0; + let (endpoint, checkpoints) = state.checkpoint_endpoint(history)?; + let group = state + .runtime + .shared()? + .log() + .cell_group(history) + .map_err(|_| TransportError::UnknownGroup)?; + let local = group.identity(); + let peer = endpoint + .route + .peers + .iter() + .find(|p| p.replica.member.get() == message.to) + .cloned() + .ok_or(TransportError::StalePeer)?; + if peer.replica.node == local.node { + return Err(TransportError::Identity); + } + shared::permitted(&group, &[local, &peer.replica])?; + let route = Route::new_group( + local, + state.auth.source(), + &peer.replica, + &peer.label, + endpoint.route.schema_digest, + )?; + if message.compute_size() as usize > MESSAGE_BYTES { + return Err(TransportError::Limit); + } + let body = message + .write_to_bytes() + .map_err(|_| TransportError::Protocol)?; + codec::message(&body, &route)?; + let pool = state + .registry + .lock() + .map_err(|_| TransportError::Closed)? + .pools + .get(&peer.replica.node) + .cloned() + .ok_or(TransportError::StalePeer)?; + let charge = wire_charge(&state.options); + let admission = Arc::new(Admission { + _source: source, + _node: state + .outgoing + .lane(Lane::Data) + .acquire(charge) + .ok_or(TransportError::Overloaded)?, + _peer: pool + .outgoing + .lane(Lane::Data) + .acquire(charge) + .ok_or(TransportError::Overloaded)?, + _transfer: state + .snapshot_source_budget + .acquire(snapshots::memory_bound(&state.options)) + .ok_or(TransportError::Overloaded)?, + }); + let guard = state.reserve_task()?; + let mut chunks = workers::source( + state, + endpoint.clone(), + checkpoints, + message.get_snapshot().clone(), + admission.clone(), + )?; + let (result, wait) = oneshot::channel(); + let state = state.clone(); + state.clone().spawn_reserved(guard, async move { + let _admission = admission; + let outcome = state + .send_group_checkpoint(&endpoint, &peer, &pool, route, body, &mut chunks) + .await; + // Closing the channel wakes a blocked physical producer before any + // runtime hint can await a busy consensus owner. + drop(chunks); + // The producer independently retains admission through physical + // source exit. The completed network task must not extend its send + // fence while awaiting advisory runtime feedback. + drop(_admission); + let delivered = outcome.is_ok(); + if !delivered { + state.failed.fetch_add(1, Ordering::Relaxed); + } + let _ = result.send(outcome); + let hint = SendRoute::Physical { + route: endpoint.route.clone(), + peer: peer.clone(), + }; + // Failure feedback is required for Raft to leave Snapshot state. + // Retain this admitted transfer through reserved hint admission; + // only transport shutdown may cancel it, not an I/O reply timeout. + let mut stop = state.stop.subscribe(); + if !*stop.borrow() { + crate::asyncrt::select! { + _ = stop.changed() => (), + _ = state.runtime.delivery_hint(&hint, peer.replica.member.get(), delivered, true) => (), + } + } + })?; + Ok(Delivery { result: wait }) + } +} + +impl State { + async fn send_group_checkpoint( + &self, + endpoint: &Arc, + peer: &RaftPeer, + pool: &Arc, + route: Route, + body: Vec, + chunks: &mut mpsc::Receiver, + ) -> Result<(), TransportError> { + let deadline = self + .domain + .mono_ms() + .map_err(|_| TransportError::Closed)? + .saturating_add(self.options.snapshot_timeout_ms); + let _connection = self.outgoing_connections.acquire(Lane::Data); + let mut stream = self + .snapshot_io(pool, deadline, async { + let tcp = TcpStream::connect(peer.data_address).await?; + tcp.set_nodelay(true)?; + let tcp = self.observed_socket(tcp, metrics::Lane::Data); + let name = rustls::pki_types::ServerName::try_from(peer.server_name.clone()) + .map_err(|_| TransportError::Identity)?; + tokio_rustls::TlsConnector::from(self.tls.client.clone()) + .connect(name, tcp) + .await + .map_err(|e| TransportError::Tls(e.to_string())) + }) + .await?; + let exporter = stream + .get_ref() + .1 + .export_keying_material([0; 32], frame::EXPORT_LABEL, None) + .map_err(|e| TransportError::Tls(e.to_string()))?; + let wire = snapshots::Wire { + auth: &self.auth, + exporter: &exporter, + options: &self.options, + max_body: MESSAGE_BYTES, + }; + if !endpoint.active.load(Ordering::Acquire) || !self.current_physical(&endpoint.route, peer) + { + return Err(TransportError::StalePeer); + } + let start = wire.encode( + START_MAGIC, + START, + &[], + &Start { + route, + digest: digest(&body), + }, + &body, + )?; + let binding = start.tag; + self.snapshot_io(pool, deadline, async { + write_head(&mut stream, &start).await?; + stream.write_all(&body).await?; + stream.flush().await?; + Ok(()) + }) + .await?; + drop(body); + let mut sequence = 0_u64; + loop { + if !endpoint.active.load(Ordering::Acquire) + || !self.current_physical(&endpoint.route, peer) + { + return Err(TransportError::StalePeer); + } + let next = self + .snapshot_io(pool, deadline, async { + chunks.recv().await.ok_or(TransportError::Closed) + }) + .await?; + let (part, offset, length, bytes) = match next { + workers::Output::Chunk(chunk) => { + (Some(chunk.part), chunk.offset, chunk.length, chunk.bytes) + } + workers::Output::End(result) => { + result?; + (None, 0, 0, Vec::new()) + } + }; + if bytes.len() > CHUNK_BYTES.min(self.options.snapshot_chunk_bytes) { + return Err(TransportError::Limit); + } + let ending = part.is_none(); + let header = Packet { + sequence, + part, + offset, + length, + digest: digest(&bytes), + }; + let head = wire.encode(DATA_MAGIC, DATA, &binding, &header, &bytes)?; + self.snapshot_io(pool, deadline, async { + write_head(&mut stream, &head).await?; + stream.write_all(&bytes).await?; + stream.flush().await?; + Ok(()) + }) + .await?; + if ending { + break; + } + sequence = sequence.checked_add(1).ok_or(TransportError::Limit)?; + } + let head = self + .snapshot_io( + pool, + deadline, + snapshots::read_bulk_head(&mut stream, None, ACK_MAGIC, 0, &self.options), + ) + .await?; + let ack: Ack = wire.decode(ACK_MAGIC, ACK, &head, Some(&binding))?; + if ack.sequence != sequence { + return Err(TransportError::Replay); + } + if !endpoint.active.load(Ordering::Acquire) || !self.current_physical(&endpoint.route, peer) + { + return Err(TransportError::StalePeer); + } + Ok(()) + } +} diff --git a/crates/cellar/durability/raft/group_snapshot_transport/workers.rs b/crates/cellar/durability/raft/group_snapshot_transport/workers.rs new file mode 100644 index 000000000..2440ee798 --- /dev/null +++ b/crates/cellar/durability/raft/group_snapshot_transport/workers.rs @@ -0,0 +1,95 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Blocking owners retain task accounting and wire admission through actual I/O. +//! Channel closure aborts a partial receive; only an explicit End can seal it. +use super::*; +use crate::raft::cell_groups::{checkpoint::wire::Chunk, PreparedCellGroupCheckpoint}; + +pub(super) enum Input { + Chunk(Chunk), + End, +} + +pub(super) enum Output { + Chunk(Chunk), + End(Result<(), TransportError>), +} + +pub(super) fn source( + state: &Arc, + endpoint: Arc, + pool: CellGroupCheckpointPool, + snapshot: ::raft::prelude::Snapshot, + admission: Arc, +) -> Result, TransportError> { + let guard = state.reserve_task()?; + let (tx, rx) = mpsc::channel(1); + state + .domain + .blocking(move || { + let _guard = guard; + let _admission = admission; + let result = (|| { + let mut source = endpoint.applier.capture_checkpoint(&pool, &snapshot)?; + source.stream(|chunk| { + if !endpoint.active.load(Ordering::Acquire) { + return Err(CellGroupCheckpointError::Invalid("revoked source")); + } + tx.blocking_send(Output::Chunk(Chunk::copy(chunk))) + .map_err(|_| { + CellGroupCheckpointError::Sink(std::io::Error::new( + std::io::ErrorKind::BrokenPipe, + "snapshot connection closed", + )) + }) + }) + })() + .map_err(checkpoint_error); + let _ = tx.blocking_send(Output::End(result)); + }) + .map_err(|_| TransportError::Closed)? + .detach(); + Ok(rx) +} + +pub(super) struct Receive { + pub input: mpsc::Sender, + pub result: oneshot::Receiver>, +} + +pub(super) fn receiver( + state: &Arc, + endpoint: Arc, + pool: CellGroupCheckpointPool, + admission: Arc, +) -> Result { + let guard = state.reserve_task()?; + let scratch = state.runtime.log().snapshot_scratch(); + let directory = state.runtime.log().limits().directory.clone(); + let (input, mut rx) = mpsc::channel(1); + let (tx, result) = oneshot::channel(); + state + .domain + .blocking(move || { + let _guard = guard; + let _admission = admission; + let outcome = (|| { + let mut receiver = pool.receive(endpoint.applier.clone(), scratch, directory)?; + loop { + if !endpoint.active.load(Ordering::Acquire) { + return Err(CellGroupCheckpointError::Invalid("revoked receiver")); + } + match rx.blocking_recv() { + Some(Input::Chunk(chunk)) => chunk.append(&mut receiver)?, + Some(Input::End) => return receiver.finish(), + None => return Err(CellGroupCheckpointError::Invalid("incomplete stream")), + } + } + })() + .map_err(checkpoint_error); + let _ = tx.send(outcome); + }) + .map_err(|_| TransportError::Closed)? + .detach(); + Ok(Receive { input, result }) +} diff --git a/crates/cellar/durability/raft/host.rs b/crates/cellar/durability/raft/host.rs index 6fece2022..523b6fc3f 100644 --- a/crates/cellar/durability/raft/host.rs +++ b/crates/cellar/durability/raft/host.rs @@ -1,119 +1,71 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. -//! Node-owned command producers and Raft/SQL effect dispatch. Waiters never own -//! a submitted producer, its private image or its progress through publication. +//! Physical host configuration and observations shared by the cell-group runtime. +//! The only production dispatcher/startup is group_host and its node controller. pub use super::host_archive::RaftArchiveOptions; -use super::{ - archive::{ArchiveError, CellArchive}, - budget::{Budget, Permit}, - cache::GroupLog, - catalog::*, - completion::*, - execution::*, - host_sql::{self as sql, Action, ImageView, Pool, Value}, - mailbox, - query::{Minimum, PendingRead, Phase as ReadPhase, ReadId}, - runtime::{RaftRuntime, RuntimeEvents}, - transport::{GroupRoute, RaftPeer, RaftTransport}, -}; -use crate::application::consensus::client::{ - self, Binding, ControlRequest, Kind, NodeKind, NodeRequest, Request, Response, -}; -use crate::application::consensus::{command_request_digest, OperationKind, RaftCells, RaftClass}; +use super::{catalog::*, completion::*, execution::*}; +use crate::application::consensus::RaftClass; use cellar_logic::consensus::*; -use cellar_logic::raft_command::CommandState; -use cellar_ltx::generation::{GenerationLimits, ImageId, ReadServices}; -use futures_util::{future::BoxFuture, stream::FuturesUnordered, StreamExt}; -use rand::RngCore; -use std::collections::{BTreeMap, BTreeSet, VecDeque}; -use std::path::PathBuf; -use std::sync::{ - atomic::{AtomicBool, Ordering}, - Arc, Mutex, -}; -use tokio::sync::{mpsc, Notify}; - -#[path = "host_image_limits.rs"] -mod image_limits; +use cellar_ltx::generation::GenerationLimits; +use std::sync::{Arc, Mutex}; #[path = "host_extents.rs"] mod extents; +#[path = "host_image_limits.rs"] +mod image_limits; pub use extents::{RaftExtentCacheLimits, RaftExtentLimits, RaftExtentOptions}; - -#[cfg(all(test, not(cellar_internal_tests)))] -#[path = "host_startup_tests.rs"] -mod startup_tests; - -#[cfg(test)] -use super::storage_fault; - -#[path = "host_archive_status.rs"] -mod archive_status; - -#[path = "host_admission.rs"] -mod admission; - #[path = "host_controller.rs"] mod controller; -#[path = "host_drain.rs"] -mod drain; -#[path = "host_effects.rs"] -mod effect_scheduler; pub use controller::{RaftControllerOptions, RaftControllerStatus}; #[path = "host_directory.rs"] mod directory; +pub use directory::RaftNodeDirectoryOptions; #[path = "host_enrollment.rs"] mod enrollment; pub(crate) use enrollment::NamespaceRegistration; pub use enrollment::{RaftManagedHostOptions, RaftNamespaceRegistration}; -#[path = "host_inspection.rs"] -mod inspection; #[path = "host_leader_load.rs"] mod load; -#[path = "host_readiness.rs"] -mod readiness; pub(crate) use load::GroupResourceObserver; pub use load::RaftInspectionScanOptions; -#[path = "host_fleet_admission.rs"] -mod fleet_admission; -#[path = "host_fleet.rs"] -mod fleet_control; #[path = "host_fleet_observation.rs"] mod fleet_observation; +pub(crate) use fleet_observation::group_membership_digest; pub use fleet_observation::{FleetMembershipConfiguration, FleetMembershipObservation}; #[path = "host_fleet_cut.rs"] mod fleet_cut; -pub(crate) use fleet_cut::Publication; -pub use fleet_cut::{FleetImageState, FleetPublishedCut, FleetPublishedImage}; -#[path = "host_fleet_inspection.rs"] -mod fleet_inspection; pub use super::fleet_inspection_wire::RaftPeerFleetMembershipObservation; -#[path = "host_initial.rs"] -mod initial; +pub(crate) use fleet_cut::{group_prefix, Publication}; +pub use fleet_cut::{FleetImageState, FleetPublishedCut, FleetPublishedImage}; #[path = "host_node.rs"] -mod node_control; -#[path = "host_registration.rs"] -mod registration; -use node_control::reply_node; -pub use node_control::{ +mod node; +use super::group_host::RaftCellsConfig; +pub use super::transport_options::RaftPeerNode; +pub use node::{ RaftHostEnrollmentObservation, RaftHostObservation, RaftHostStartupBounds, RaftPeerNodeObservation, }; -#[path = "host_fleet_execution.rs"] -mod fleet_execution; -#[path = "host_fleet_stop.rs"] -mod fleet_stop; -#[path = "host_membership.rs"] -mod membership; -#[path = "host_node_inspection.rs"] -mod node_inspection; -pub use directory::RaftNodeDirectoryOptions; -#[path = "host_seed.rs"] -mod seed; -#[path = "host_snapshot.rs"] -mod snapshots; -#[path = "host_standby.rs"] -mod standby; + +// These storage-observation regressions remain enabled while their owning +// generic cache/archive primitives are retained. No predecessor dispatcher runs. +#[cfg(test)] +use super::host_sql::ImageView; +#[cfg(test)] +use super::storage_fault; +#[cfg(test)] +#[path = "host_admission.rs"] +mod admission; +#[cfg(test)] +#[path = "host_archive_status.rs"] +mod archive_status; +#[cfg(test)] +#[path = "host_inspection.rs"] +mod inspection; +#[cfg(test)] +#[path = "host_readiness.rs"] +mod readiness; +#[cfg(test)] +use super::archive::ArchiveError; #[derive(Clone, Debug)] pub struct RaftHostOptions { @@ -320,3619 +272,23 @@ impl RaftHostOptions { } } -use super::transport_options::NamespaceQueryOrigin; -pub use super::transport_options::RaftPeerNode; - -/// Local participation is explicit. A router never creates or attaches a -/// member and never initializes a SQLite cell image. -#[derive(Clone, Copy, Debug, PartialEq, Eq)] -pub enum RaftCellPlacement { - Replica, - /// Register an eligible local standby without creating a group or image. - /// A retained handle gains local admission only after authorized learner - /// initialization in this node's existing engine and SQL pool. - Standby, - /// Resume a stable joined member or seed a new committed learner from the - /// specified trusted peer. A claimed member with lost state is refused. - /// Fresh managed learners require authorized activation on a `Standby`; - /// this placement can still resume their intact retained engines. - Joining { - member: MemberId, - source: MemberId, - source_node: NodeId, - }, - /// Seed a fresh learner from an exact S3 manifest selected by a live peer - /// read barrier. Existing stable joined state resumes without reseeding. - /// Fresh managed learners use authorized `Standby` activation instead. - JoiningFromS3 { - member: MemberId, - source: MemberId, - source_node: NodeId, - }, - QueryRouter { - leader_hint: Option, - }, -} - -#[derive(Clone, Debug)] -pub struct RaftCellRegistration { - pub placement: RaftCellPlacement, - pub namespace: NamespaceConfig, - pub class: String, - pub birth: BirthProposal, -} - -pub struct RaftCellsConfig { - pub infrastructure: crate::daemon::NodeRaftConfig, - pub catalog: CatalogOptions, - pub cells: Vec, - pub peers: Vec, - pub query_origins: Vec, - pub host: RaftHostOptions, -} - pub struct RaftCellsServices { + /// Explicit infrastructure authority for resuming physically seeded fresh + /// histories. Archive metadata alone never enables recovered startup. + /// Requires unmanaged Resume mode and the exact retained seed/engine pins. + /// Startup returns while quorum activation may still be pending; ordinary + /// requests remain refused until the actual owner applies activation. + /// Node shutdown joins any entered recovery verification/publication work. + pub recovery: Option< + Arc< + dyn super::catalog::DisasterFenceAuthority< + super::cell_groups::CellGroupDisasterFenceRequest, + >, + >, + >, pub listeners: crate::daemon::NodeRaftListeners, pub tls: super::transport_options::RaftTls, pub store: Arc, pub application: Arc, pub versions: super::version::VersionKeys, } - -pub(crate) struct Registration { - pub placement: RaftCellPlacement, - pub namespace: NamespaceCatalog, - pub birth: GroupBirth, - pub class: Arc, -} - -pub(crate) struct Registrations { - pub cells: Vec, - pub namespaces: Vec, - pub peers: Vec, - pub origins: Vec, -} - -pub(crate) async fn start( - service: crate::daemon::NodeRaftService, - registrations: Registrations, - directory: PathBuf, - label: String, - options: RaftHostOptions, - versions: super::version::VersionKeys, -) -> anyhow::Result { - let log = service.log.clone(); - let runtime = service.runtime.clone(); - let transport = service.transport.clone(); - let result = Box::pin(initialize( - service, - registrations, - directory, - label, - options, - versions, - )) - .await; - if result.is_err() { - // This includes preflight and pool-construction errors before the - // initialized-cell cleanup exists. Retained handles must observe real - // closure; dropping the last caller is not a physical join boundary. - let _ = runtime.close().await; - transport.close().await; - let _ = log.close().await; - } - result -} - -async fn initialize( - service: crate::daemon::NodeRaftService, - registrations: Registrations, - directory: PathBuf, - label: String, - mut options: RaftHostOptions, - versions: super::version::VersionKeys, -) -> anyhow::Result { - let Registrations { - cells: registrations, - namespaces: declared_namespaces, - peers, - origins, - } = registrations; - options.validate()?; - for registration in ®istrations { - registration.class.validate_host(&options)?; - } - let snapshot_limits = service - .transport - .snapshot_limits(options.execution.max_image_bytes)?; - anyhow::ensure!( - (!registrations.is_empty() || options.managed.is_some()) - && registrations.len() <= options.max_groups, - "invalid registered group count" - ); - let domain = service.log.domain(); - let budget = Budget::new(options.max_pending_bytes, options.max_requests); - let archive_capture_budget = Budget::new( - options.archive.limits.max_capture_bytes * options.archive.max_producers, - options.archive.max_producers, - ); - let query_pins = Budget::new(0, options.max_query_pins); - let mut pool = Pool::start( - domain.clone(), - options.blocking_workers, - options.worker_queue, - options.max_groups, - super::host_sql::Budgets { - business: budget.clone(), - archive_capture: archive_capture_budget.clone(), - query_pins: query_pins.clone(), - }, - service.log.performance(), - )?; - let mut cells = BTreeMap::new(); - let mut standbys = BTreeMap::new(); - let mut bindings = BTreeMap::new(); - let mut routes = Vec::new(); - let mut archive_owner = None; - let mut node_directory = None; - let mut extent_owner = None; - let peer_count = peers.len(); - let mut peers: BTreeMap<_, _> = peers.into_iter().map(|peer| (peer.node, peer)).collect(); - let initialized = async { - let needs_extents = declared_namespaces.iter().any(|registration| { - registration.namespace.config().image_storage == RaftImageStorage::Extents - }) || registrations.iter().any(|registration| { - registration.namespace.config().image_storage == RaftImageStorage::Extents - && !matches!( - registration.placement, - RaftCellPlacement::QueryRouter { .. } - ) - }); - extent_owner = extents::Owner::open( - &service.log, - directory.clone(), - ltx_host(), - options.extents, - needs_extents, - ) - .await?; - anyhow::ensure!( - peer_count == peers.len() && peer_count <= service.transport.peer_limit(), - "invalid peer discovery count" - ); - let authorizations = enrollment::prepare( - &mut options, - &service, - ®istrations, - &declared_namespaces, - &peers, - &label, - ) - .await?; - let mut grants = BTreeSet::new(); - for origin in origins { - anyhow::ensure!( - grants.insert((origin.cluster, origin.namespace, origin.peer.node)), - "duplicate namespace query origin" - ); - service.transport.grant_query_origin(origin)?; - } - node_directory = directory::prepare( - options.node_directory.as_ref(), - ®istrations, - &declared_namespaces, - options.managed.as_mut().map(|m| &mut m.authorization), - &mut peers, - &service, - &label, - ) - .await?; - let namespaces: BTreeSet<_> = registrations - .iter() - .map(|registration| { - ( - registration.namespace.config().cluster, - registration.namespace.config().namespace, - ) - }) - .chain(declared_namespaces.iter().map(|registration| { - ( - registration.namespace.config().cluster, - registration.namespace.config().namespace, - ) - })) - .collect(); - let mut grants = BTreeSet::new(); - for origin in &options.inspection_origins { - anyhow::ensure!( - namespaces.contains(&(origin.cluster, origin.namespace)) - && peers.get(&origin.peer.node) == Some(&origin.peer) - && grants.insert((origin.cluster, origin.namespace, origin.peer.node)), - "inspection origin must match a unique registered namespace and trusted peer" - ); - service.transport.grant_inspection_origin(origin.clone())?; - } - for (index, registration) in registrations.into_iter().enumerate() { - let Registration { - mut placement, - namespace, - birth, - class, - } = registration; - namespace.require_mode(cellar_logic::NamespaceMode::Raft)?; - anyhow::ensure!( - namespace.config().schema_digest == class.schema_digest(), - "registered schema disagrees with namespace" - ); - let scope = birth.scope().clone(); - let worker = index % options.blocking_workers; - let cell_directory = directory - .join(hex::encode(scope.namespace.bytes())) - .join(hex::encode(scope.group.bytes())) - .join(hex::encode(scope.incarnation.bytes())); - let rooted = match namespace.config().image_storage { - RaftImageStorage::Materialized => None, - RaftImageStorage::Extents - if matches!(placement, RaftCellPlacement::QueryRouter { .. }) => - { - None - } - RaftImageStorage::Extents => Some( - extent_owner - .as_ref() - .ok_or_else(|| anyhow::anyhow!("missing node extent owner"))? - .cell_storage(&cell_directory), - ), - }; - let storage = sql::CellStorage { - directory: cell_directory, - host: ltx_host(), - services: read_services(), - images: options.images, - rooted, - snapshot_scratch: service.log.snapshot_scratch(), - execution: options.execution, - }; - if placement == RaftCellPlacement::Standby { - anyhow::ensure!( - options.enable_snapshots, - "standby activation requires snapshots" - ); - anyhow::ensure!( - !birth - .voters() - .iter() - .chain(birth.learners()) - .any(|m| m.node == service.log.limits().node), - "standby is an initial member" - ); - anyhow::ensure!( - !standbys.contains_key(&scope) - && !bindings.contains_key(&(scope.namespace, scope.cell.clone())), - "duplicate standby registration" - ); - namespace.require_node_storage(&service.log).await?; - let existing = match service.log.group(&scope) { - Ok(group) => Some(group), - Err(super::storage::LogError::NotFound) => None, - Err(error) => return Err(error.into()), - }; - let resume = existing.as_ref().is_some_and(|group| { - group.read().is_ok_and(|cache| { - cache.membership.permits(group.identity()) - && super::member_state::members(&cache.conf_state) - .contains(&group.identity().member.get()) - }) - }); - standbys.insert( - scope.clone(), - standby::Dormant { - namespace: namespace.clone(), - birth: birth.clone(), - class: class.clone(), - storage: storage.clone(), - worker, - failed: existing.is_some() && !resume, - }, - ); - if resume { - let group = existing - .as_ref() - .ok_or_else(|| anyhow::anyhow!("standby resume state missing"))?; - let source = birth - .voters() - .first() - .ok_or_else(|| anyhow::anyhow!("standby birth has no voter"))?; - placement = RaftCellPlacement::Joining { - member: group.identity().member, - source: source.member, - source_node: source.node, - }; - } else { - bindings.insert( - (scope.namespace, scope.cell.clone()), - Binding { - scope, - class, - minimum: namespace.minimum_durability(), - member: client::LocalMember::new(None), - leader_hint: None, - }, - ); - continue; - } - } - if let RaftCellPlacement::QueryRouter { leader_hint } = placement { - let scope = birth.scope().clone(); - let local = service.log.limits().node; - anyhow::ensure!( - !birth - .voters() - .iter() - .chain(birth.learners()) - .any(|member| member.node == local), - "query router is an initial group member" - ); - anyhow::ensure!( - matches!( - service.log.group(&scope), - Err(super::storage::LogError::NotFound) - ), - "query router has local member state" - ); - anyhow::ensure!( - leader_hint.is_none_or(|hint| birth - .voters() - .iter() - .any(|member| member.member == hint)), - "invalid query leader hint" - ); - anyhow::ensure!( - !bindings.contains_key(&(scope.namespace, scope.cell.clone())), - "duplicate registered cell" - ); - let routes = birth - .voters() - .iter() - .chain(birth.learners()) - .map(|member| { - let peer = peers - .get(&member.node) - .ok_or_else(|| anyhow::anyhow!("missing query peer route"))?; - Ok(RaftPeer { - replica: ReplicaIdentity { - scope: scope.clone(), - node: peer.node, - member: member.member, - disk: peer.disk, - process: peer.process, - }, - label: peer.label.clone(), - server_name: peer.server_name.clone(), - control_address: peer.control_address, - data_address: peer.data_address, - query_address: peer.query_address, - }) - }) - .collect::>>()?; - service.transport.install_query_group(GroupRoute { - scope: scope.clone(), - schema_digest: class.schema_digest(), - revision: 1, - peers: routes, - })?; - bindings.insert( - (scope.namespace, scope.cell.clone()), - Binding { - scope, - class, - minimum: namespace.minimum_durability(), - member: client::LocalMember::new(None), - leader_hint, - }, - ); - continue; - } - anyhow::ensure!( - !cells.contains_key(&scope) - && !bindings.contains_key(&(scope.namespace, scope.cell.clone())), - "duplicate registered cell" - ); - // Member bootstrap has large archive and seed futures. Keep their - // inline state out of the node enrollment/discovery poll frame. - let permit = Box::pin(async { - let permit = match placement { - RaftCellPlacement::Replica => { - if birth.managed_owner().is_some() { - let authorization = authorizations.get(&scope).ok_or_else(|| { - anyhow::anyhow!("managed initial host authorization required") - })?; - namespace - .bootstrap_managed_member(&birth, &service.log, authorization) - .await? - } else { - // Keep disaster-only archive reconstruction out of every - // ordinary managed startup poll frame. - Box::pin(async { - if let Some(permit) = - namespace.disaster_seed_permit(&birth, &service.log).await? - { - anyhow::ensure!( - options.enable_snapshots, - "disaster recovery requires snapshots" - ); - let (source, reference) = - birth.disaster_source().ok_or_else(|| { - anyhow::anyhow!("missing disaster source") - })?; - let archive = namespace - .cell_archive( - source, - storage.host.clone(), - options.archive.limits, - )? - .with_scratch_directory( - service.log.limits().directory.clone(), - ) - .with_snapshot_scratch(service.log.snapshot_scratch()); - let archive = match &storage.rooted { - Some(rooted) => { - archive.with_root_store(rooted.store.clone()) - } - None => archive, - }; - let bytes = archive.read_memory_bound(); - let reservation = - service.transport.reserve_disaster_seed(bytes)?; - let set = archive - .with_read_retention(reservation.clone()) - .recovery_set_at(reference) - .await?; - let output = pool - .call( - worker, - scope.clone(), - Action::SealDisasterSeed { - received: Box::new(super::snapshot::DisasterSeed { - permit, - set, - schema: class.schema_digest(), - limits: snapshot_limits, - reservation, - }), - storage: storage.clone(), - }, - None, - None, - ) - .await?; - let Value::Seeded(seed) = output.value else { - anyhow::bail!("disaster SQL seed response") - }; - service.log.create_seeded(*seed).await?; - } - Ok::<(), anyhow::Error>(()) - }) - .await?; - namespace.bootstrap_member(&birth, &service.log).await? - } - } - RaftCellPlacement::Joining { - member, - source, - source_node, - } - | RaftCellPlacement::JoiningFromS3 { - member, - source, - source_node, - } => { - let from_s3 = matches!(placement, RaftCellPlacement::JoiningFromS3 { .. }); - anyhow::ensure!( - options.enable_snapshots, - "joining requires snapshot recovery" - ); - match namespace - .resume_joined(&birth, &service.log, member) - .await? - { - Some(permit) => permit, - None => { - let peer = peers.get(&source_node).ok_or_else(|| { - anyhow::anyhow!("missing seed source endpoint") - })?; - peer.validate_endpoint()?; - let local = service.log.limits(); - let identity = ReplicaIdentity { - scope: scope.clone(), - node: local.node, - member, - disk: local.disk, - process: local.process, - }; - let source = RaftPeer { - replica: ReplicaIdentity { - scope: scope.clone(), - node: peer.node, - member: source, - disk: peer.disk, - process: peer.process, - }, - label: peer.label.clone(), - server_name: peer.server_name.clone(), - control_address: peer.control_address, - data_address: peer.data_address, - query_address: peer.query_address, - }; - let limits = service - .transport - .snapshot_limits(options.execution.max_image_bytes)?; - let action = if from_s3 { - let archive = namespace - .cell_archive( - &birth, - storage.host.clone(), - options.archive.limits, - )? - .with_scratch_directory( - service.log.limits().directory.clone(), - ) - .with_snapshot_scratch(service.log.snapshot_scratch()); - let archive = match &storage.rooted { - Some(rooted) => { - archive.with_root_store(rooted.store.clone()) - } - None => archive, - }; - let received = service - .transport - .seed_from_archive( - identity, - source, - &birth, - class.schema_digest(), - limits, - archive, - ) - .await?; - Action::SealArchiveSeed { - received: Box::new(received), - storage: storage.clone(), - } - } else { - let received = service - .transport - .seed_from_peer( - identity, - source, - &birth, - class.schema_digest(), - limits, - storage.rooted.as_ref().map(|root| root.store.clone()), - ) - .await?; - Action::SealSeed { - received: Box::new(received), - storage: storage.clone(), - } - }; - let output = - pool.call(worker, scope.clone(), action, None, None).await?; - let Value::Seeded(seed) = output.value else { - anyhow::bail!("SQL seed response") - }; - // Managed fresh learners use registered standby activation - // with an opaque catalog authorization. Retained joined - // state above resumes without creating a new membership. - standby::check_seed_guard( - seed.membership(), - seed.local(), - None, - false, - )?; - service - .log - .performance() - .measure( - super::performance::Operation::RecoveryInstall, - service.log.create_seeded(*seed), - ) - .await?; - namespace - .resume_joined(&birth, &service.log, member) - .await? - .ok_or_else(|| { - anyhow::anyhow!("seed state missing after fence") - })? - } - } - } - RaftCellPlacement::QueryRouter { .. } | RaftCellPlacement::Standby => { - unreachable!("non-replica handled above") - } - }; - Ok::<_, anyhow::Error>(permit) - }) - .await?; - let group = service.log.group(&scope)?; - let extent_store = storage.rooted.as_ref().map(|root| root.store.clone()); - let output = pool - .call( - worker, - scope.clone(), - Action::Initialize { - group: group.clone(), - class: class.clone(), - storage, - }, - None, - None, - ) - .await?; - let Value::Initialized(image) = output.value else { - anyhow::bail!("SQL initialization response") - }; - routes.push(membership::route( - &group, - &birth, - &peers, - class.schema_digest(), - &label, - &service.transport, - )?); - bindings.insert( - (scope.namespace, scope.cell.clone()), - Binding { - scope: scope.clone(), - class: class.clone(), - minimum: namespace.minimum_durability(), - member: client::LocalMember::new(Some(group.identity().member)), - leader_hint: None, - }, - ); - let archive = namespace - .cell_archive(&birth, ltx_host(), options.archive.limits)? - .with_scratch_directory(service.log.limits().directory.clone()) - .with_snapshot_scratch(service.log.snapshot_scratch()) - .with_publication_coordination( - super::host_archive::publication_owner(group.identity().node_identity()), - options.archive.publication_stale_ms, - 64 * 1024 * 1024, - )?; - cells.insert( - scope.clone(), - Cell::recovered( - group, - birth.clone(), - archive, - image, - worker, - namespace.minimum_durability(), - class.schema_digest(), - )? - .with_extents(extent_store), - ); - permit.attach(&service.runtime).await?; - } - if !cells.is_empty() || !standbys.is_empty() || options.managed.is_some() { - let mut nonce = [0; 16]; - crate::asyncrt::rng("raft-archive-verifier").fill_bytes(&mut nonce); - let owner = super::host_archive::Owner::open( - directory.join("archive-verifier"), - ltx_host(), - ImageId::new(nonce)?, - options.images.apply, - pool.image_accounting(), - service.log.snapshot_scratch(), - super::host_archive::PhysicalOptions { - max_jobs: options.archive.max_blocking_jobs, - performance: service.log.performance(), - }, - ) - .await?; - for cell in cells.values_mut() { - cell.archive = cell - .archive - .clone() - .with_cancellation(owner.cancellation.subscribe()); - } - archive_owner = Some(owner); - } - // The host cannot drain runtime events until initialization completes. - // Publishing earlier lets peer traffic retain the ordinary request - // budget behind that drain and reject later group attachments. Keep - // routes private until every local group and the archive owner are ready. - for route in routes { - service.transport.install_group(route)?; - } - Ok::<_, anyhow::Error>(()) - } - .await; - if let Err(error) = initialized { - let _ = service.runtime.close().await; - service.transport.close().await; - let _ = pool.join().await; - let _ = service.log.close().await; - // A failed join can still own sealed roots in cells or standby storage. - // Release their final directory locks on a joined blocking owner too. - let cleanup = service.log.domain().blocking(move || { - drop((cells, standbys, extent_owner, archive_owner)); - }); - let cleanup = match cleanup { - Ok(job) => job.await.map_err(anyhow::Error::from), - Err(error) => Err(error.into()), - }; - return match cleanup { - Ok(()) => Err(error), - Err(cleanup) => { - Err(error.context(format!("physical startup cleanup failed: {cleanup}"))) - } - }; - } - let (sender, requests) = mailbox::channel(options.max_requests); - let (control_sender, controls) = mailbox::channel(options.max_control_requests); - let control_budget = Budget::new(options.max_control_bytes, options.max_control_requests); - let seed_archive_work = Budget::new(0, options.max_control_requests); - let (close, closes) = mpsc::unbounded_channel(); - let closed = Arc::new(AtomicBool::new(false)); - let closing = Arc::new(AtomicBool::new(false)); - let terminal = Arc::new(client::Terminal { - result: Mutex::new(None), - changed: Notify::new(), - }); - let controller_bindings = if options.controller.is_some() { - bindings.values().cloned().collect() - } else { - Vec::new() - }; - let has_effects = bindings - .values() - .any(|b| b.class.alarm_limits().is_some() || b.class.outbox_definition().is_some()); - let handle = Arc::new(client::Handle { - performance: service.log.performance(), - requests: sender, - control_requests: control_sender, - control_budget: control_budget.clone(), - close, - domain: domain.clone(), - closed: closed.clone(), - closing: closing.clone(), - admission: Arc::new(crate::application::consensus::admission::Admission::default()), - terminal: terminal.clone(), - budget: budget.clone(), - archive_capture_budget, - query_pins, - sql: Some(pool.resource_observer()), - extents: extent_owner.as_ref().map(|owner| owner.store.observer()), - groups: load::GroupResourceObserver::default(), - controller: Mutex::new(None), - fleet_restart_service: Mutex::new(None), - fleet_movement_service: Mutex::new(None), - transport: service.transport.resource_observer(), - max_input_bytes: options.max_input_bytes, - bindings: client::Bindings::new(bindings, options.max_groups), - work: crate::asyncrt::services().application_work_registry(), - }); - let installation = (|| -> anyhow::Result<()> { - if options.managed.is_some() { - crate::asyncrt::services() - .raft_lifecycle() - .bind_startup_admission(&handle.admission)?; - } - service.transport.install_query_gateway(handle.clone())?; - Ok(()) - })(); - if let Err(error) = installation { - let _ = service.runtime.close().await; - service.transport.close().await; - let _ = pool.join().await; - let _ = service.log.close().await; - return Err(error); - } - let local = handle.clone(); - let crate::daemon::NodeRaftService { - log, - runtime, - events, - transport, - } = service; - domain.spawn_detached("raft-cell-host", async move { - struct Closed(Arc, Arc); - - impl Drop for Closed { - fn drop(&mut self) { - self.0.store(true, Ordering::Release); - self.1.changed.notify_waiters(); - } - } - let _closed = Closed(closed, terminal.clone()); - let mut rng = crate::asyncrt::rng("raft-cell-commands"); - let mut read_nonce = [0; 16]; - rng.fill_bytes(&mut read_nonce); - let mut driver = Driver { - drain: None, - effect_scheduler: options - .effects - .filter(|_| has_effects) - .map(effect_scheduler::Scheduler::new), - controller: options - .controller - .as_ref() - .map(|options| controller::Controller::new(options.clone(), controller_bindings)), - leader_load: Some(load::Scanner::new(options.inspection_scan.clone())), - directory: node_directory, - peers, - label, - archive_owner, - extents: extent_owner, - log, - runtime, - events, - transport, - pool, - cells, - standbys, - declared_namespaces, - history_registration: None, - directory_path: directory, - activating: None, - active: BTreeSet::new(), - applying: BTreeSet::new(), - uncertain: BTreeMap::new(), - jobs: FuturesUnordered::new(), - options, - rng, - closing, - versions, - reads: BTreeMap::new(), - read_nonce, - read_sequence: 0, - budget, - control_budget, - seed_archive_work, - local, - snapshot_limits, - snapshot_source: None, - checkpoint_cursor: None, - maintenance_at: 0, - admission_at: 0, - admission_waiters: VecDeque::new(), - }; - driver.run(requests, controls, closes).await; - let result = driver.shutdown().await; - if let Ok(mut target) = terminal.result.lock() { - *target = Some(result); - } - terminal.changed.notify_waiters(); - })?; - Ok(RaftCells { handle }) -} - -fn ltx_host() -> cellar_ltx::LtxHost { - let fs = crate::asyncrt::fs(); - let age = fs.clone(); - let read = fs.clone(); - cellar_ltx::LtxHost::new( - crate::asyncrt::wall_ms, - move |path| { - Ok(std::time::Duration::from_millis( - crate::asyncrt::wall_ms() - .saturating_sub(age.metadata(path)?.modified_unix_millis) - .max(0) as u64, - )) - }, - move |path| { - let fs = read.clone(); - async move { - crate::asyncrt::try_blocking(move || fs.read(&path)) - .map_err(|e| std::io::Error::other(e.to_string()))? - .await - .map_err(|e| std::io::Error::other(e.to_string()))? - } - }, - |job| async move { - crate::asyncrt::try_blocking(job) - .map_err(|e| cellar_ltx::HostTaskError::new(e.to_string()))? - .await - .map_err(|e| cellar_ltx::HostTaskError::new(e.to_string())) - }, - ) - .with_filesystem(fs) -} - -fn read_services() -> ReadServices { - ReadServices { - now_unix_millis: Arc::new(crate::asyncrt::wall_ms), - fill_random: Arc::new(|bytes| crate::asyncrt::rng("raft-image-read").fill_bytes(bytes)), - } -} - -fn progress( - group: &GroupLog, - image: Option<&ImageView>, - archived: LogIndex, -) -> Result, RuntimeError> { - let cache = group.read()?; - if cache.applied.image.as_ref() != image.map(|image| &image.image) { - return Ok(None); - } - let mut progress = ReplicaProgress::checkpoint( - group.identity().clone(), - cache.applied.position.index, - archived, - ) - .map_err(|e| RuntimeError::Invalid(e.to_string()))?; - progress - .persist(LogIndex::new(cache.last_index())) - .map_err(|e| RuntimeError::Invalid(e.to_string()))?; - progress - .commit(LogIndex::new(cache.hard_state.commit)) - .map_err(|e| RuntimeError::Invalid(e.to_string()))?; - Ok(Some(progress)) -} - -#[derive(Clone, Copy, PartialEq, Eq)] -enum Phase { - AwaitAdmission, - Start, - AwaitStart, - Staging, - BeforeSubmit, - AwaitSubmit, - Marking, - Proposing, - Waiting, - Finish, - AwaitFinish, - Discard, - Retiring, -} - -struct Pending { - allow_new: bool, - // Unknown until the callback's physical preview is verified. Keep this - // bound after proposal submission takes `prepared`, through proof/reply. - archive_bytes: Option, - // Only transient admission deferral is timed here, until grant/refusal. - // Dropping a cancelled/retired waiter records abandonment, never a grant. - admission_wait: Option, - system: bool, - proof: Option<(LogPosition, [u8; 32])>, - request: Request, - options: client::CommandOptions, - contract: crate::application::consensus::OperationContract, - input: Option>, - identity: Option, - prepared: Option, - result: Option, - duplicate: bool, - phase: Phase, - _bytes: Option, -} - -struct Cell { - extents: Option, - birth: GroupBirth, - route_revision: LogIndex, - configuration: Option, - retired: bool, - archive: CellArchive, - archived: LogIndex, - archive_retained: admission::Retained, - archive_observation: archive_status::Observation, - archive_retry_at: u64, - group: GroupLog, - schema_digest: [u8; 32], - worker: usize, - core: CommandState, - image: Option, - pending: Option, - sequence: u64, - status_busy: bool, - prefer_delivery: bool, - sql_busy: bool, - apply: Option, - publication: Option, - apply_ticket: Option, - apply_retry_at: u64, - failed: bool, - incoming: Option, - source_request: Option, - apply_inflight: bool, - checkpoint_busy: bool, - checkpoint_retry_at: u64, - archive_checkpoint_at: u64, -} - -enum StatusPurpose { - Start, - Submit, - Finish, - Poll, - Observe(Box), -} - -enum SqlPurpose { - EffectScan(u64), - Seed(ReadId), - Checkpoint, - SnapshotSource(Box<::raft::prelude::Message>), - SnapshotPrepare(ApplyTicket), - SnapshotPublish(ApplyTicket), - CaptureArchive, - Stage, - Mark, - Discard, - Retire, - Prepare(RuntimeEvent), - Publish(ApplyTicket), - Query(ReadId), -} - -enum Job { - MessageDelivery { - scope: GroupScope, - member: u64, - result: Result<(), super::transport::TransportError>, - }, - FleetMovementServiceStopped, - FleetRestartServiceStopped, - FleetAction(Box), - PeerFleetStop(Box), - PeerFleetInspection(Box), - LocalFleetInspection(Box), - FleetAdmission(Box), - NodeInspection(Box), - HistoryRegistered(Box), - Drained(Box), - Effect(Box), - Controller(Box), - LeaderLoad(Box), - Inspection(Box), - PeerInspection(Box), - Directory(Box), - Standby(Box), - Leadership { - request: Box, - result: Result, - }, - SeedArchive { - id: ReadId, - result: Box, ArchiveError>>, - }, - Configured { - scope: GroupScope, - result: Result, - }, - SnapshotSent { - lease: u64, - result: Result<(), super::transport::TransportError>, - }, - SnapshotFetched { - ticket: ApplyTicket, - result: Box< - Result, - >, - }, - SnapshotAuthorized { - ticket: ApplyTicket, - result: Result<(), RuntimeError>, - }, - SnapshotAbort { - ticket: ApplyTicket, - result: Result<(), RuntimeError>, - }, - Checkpoint { - scope: GroupScope, - result: Result<(), RuntimeError>, - }, - Archived(Vec), - ArchiveBatchJoined, - Routed { - request: Box, - result: Result, - }, - ReadStarted { - id: ReadId, - attempt: u64, - result: Result<(), RuntimeError>, - }, - ReadSource { - id: ReadId, - result: Result, - }, - ReadLeader { - id: ReadId, - check: u64, - result: Result, - }, - Status { - scope: GroupScope, - sequence: u64, - purpose: StatusPurpose, - result: Result, - }, - Sql { - scope: GroupScope, - purpose: SqlPurpose, - result: Box>, - work: Option>, - }, - SnapshotEncoded { - scope: GroupScope, - purpose: SqlPurpose, - result: Box>, - }, - Proposed { - scope: GroupScope, - identity: CommandIdentity, - result: Result, - }, - Installed { - scope: GroupScope, - result: Result<(), RuntimeError>, - }, - Hint, - ExtentGc(anyhow::Result), -} - -struct Driver { - maintenance_at: u64, - admission_at: u64, - admission_waiters: VecDeque, - drain: Option, - effect_scheduler: Option, - controller: Option, - leader_load: Option, - directory: Option, - standbys: BTreeMap, - declared_namespaces: Vec, - history_registration: Option>, - directory_path: PathBuf, - activating: Option, - peers: BTreeMap, - label: String, - snapshot_limits: super::snapshot::SnapshotLimits, - snapshot_source: Option, - checkpoint_cursor: Option, - archive_owner: Option, - extents: Option, - local: Arc, - log: super::storage::ReplicaLog, - runtime: RaftRuntime, - events: RuntimeEvents, - transport: RaftTransport, - pool: Pool, - cells: BTreeMap, - active: BTreeSet, - applying: BTreeSet, - uncertain: BTreeMap, - jobs: FuturesUnordered>, - options: RaftHostOptions, - rng: crate::asyncrt::rng::Stream, - closing: Arc, - versions: super::version::VersionKeys, - reads: BTreeMap, - read_nonce: [u8; 16], - read_sequence: u64, - budget: Arc, - control_budget: Arc, - seed_archive_work: Arc, -} - -fn reply(request: &mut Request, result: Result) { - reply_charged(request, result, None); -} - -fn reply_charged( - request: &mut Request, - result: Result, - bytes: Option, -) { - if let Some(reply) = request.reply.take() { - let _ = reply.send(result.map(|response| client::ChargedResponse { - response, - _bytes: bytes, - })); - } -} - -fn failure(error: ExecutionError) -> CommandFailure { - match error { - ExecutionError::Refused(failure) => failure, - ExecutionError::Registration(_) => CommandFailure::UnsupportedCapability, - _ => CommandFailure::Unavailable, - } -} - -impl Driver { - fn finish_read(&mut self, id: &ReadId, result: Result) { - if let Some(mut read) = self.reads.remove(id) { - if result.is_err() { - read.request.control.cancel(); - } - if let Some(timer) = read.archive_wait.take() { - timer.finish(result.is_ok()); - } - reply(&mut read.request, result); - } - } - - fn pump_reads(&mut self) { - let now = crate::asyncrt::mono_ms(); - for id in self.reads.keys().copied().collect::>() { - let Some(read) = self.reads.get_mut(&id) else { - continue; - }; - let scope = read.request.scope.clone(); - let Some(cell) = self.cells.get(&scope) else { - self.finish_read(&id, Err(CommandFailure::Unavailable)); - continue; - }; - if read.expired() || cell.failed || cell.retired { - self.finish_read(&id, Err(CommandFailure::VersionUnavailable)); - continue; - } - if cell.incoming.is_some() { - continue; - } - if now < read.retry_at { - continue; - } - match read.phase { - ReadPhase::Source => { - read.phase = ReadPhase::Probing; - let runtime = self.runtime.clone(); - self.jobs.push(Box::pin(async move { - Job::ReadSource { - id, - result: runtime.status(&scope).await, - } - })); - } - ReadPhase::StartBarrier => { - let Some(check) = read.leader_check.checked_add(1) else { - self.finish_read(&id, Err(CommandFailure::Overloaded)); - continue; - }; - read.leader_check = check; - read.phase = ReadPhase::CheckingLeader; - let runtime = self.runtime.clone(); - self.jobs.push(Box::pin(async move { - Job::ReadLeader { - id, - check, - result: runtime.status(&scope).await, - } - })); - } - ReadPhase::Barrier if !read.checking_leader => { - let Some(check) = read.leader_check.checked_add(1) else { - self.finish_read(&id, Err(CommandFailure::Overloaded)); - continue; - }; - read.leader_check = check; - read.checking_leader = true; - let runtime = self.runtime.clone(); - self.jobs.push(Box::pin(async move { - Job::ReadLeader { - id, - check, - result: runtime.status(&scope).await, - } - })); - } - ReadPhase::Image => { - if matches!(&read.operation, super::query::ReadOperation::Fleet { .. }) { - self.finish_fleet_observation(id); - continue; - } - let metadata_only = matches!( - &read.operation, - super::query::ReadOperation::Seed { - kind: super::transport::seeds::SeedKind::S3, - .. - } - ); - if !metadata_only - && (cell.sql_busy || cell.apply.is_some() || cell.publication.is_some()) - { - continue; - } - if let super::query::ReadOperation::Seed { - target, - kind, - charge, - } = &read.operation - { - let Some(Minimum::ReadIndex(barrier)) = read.minimum.as_ref() else { - self.finish_read(&id, Err(CommandFailure::CorruptState)); - continue; - }; - if !seed::valid_target(cell, target, &self.peers) { - self.finish_read(&id, Err(CommandFailure::Unavailable)); - continue; - } - let ready = !cell.apply_inflight - && cell.group.read().is_ok_and(|cache| { - cache.applied.position.index >= *barrier - && cache.applied.image.as_ref() - == cell.image.as_ref().map(|image| &image.image) - }); - if !ready { - continue; - } - if *kind == super::transport::seeds::SeedKind::S3 { - // One owner spans all metadata retries and waiting - // for the archive to cover this exact ReadIndex. - read.archive_wait.get_or_insert_with(|| { - self.local - .performance - .timer(super::performance::Operation::RecoveryArchiveWait) - }); - let archive = cell.archive.clone().with_read_retention(charge.clone()); - let Some(work) = self.seed_archive_work.acquire(0) else { - self.finish_read(&id, Err(CommandFailure::Overloaded)); - continue; - }; - read.phase = ReadPhase::Executing; - self.jobs.push(Box::pin(async move { - // Ownership survives expiry/removal of the read waiter. - let _work = work; - Job::SeedArchive { - id, - result: Box::new(archive.recovery_metadata().await), - } - })); - continue; - } - let reservation = match self.transport.reserve_snapshot_source() { - Ok(reservation) => reservation, - Err(_) => { - self.finish_read(&id, Err(CommandFailure::Overloaded)); - continue; - } - }; - read.phase = ReadPhase::Executing; - self.sql( - &scope, - SqlPurpose::Seed(id), - Action::CaptureSeed { - limits: self.snapshot_limits, - reservation, - }, - ); - continue; - } - let Some(image) = cell.image.as_ref() else { - continue; - }; - let candidate = cell.group.read().ok().and_then(|cache| { - (cache.applied.image.as_ref() == Some(&image.image)).then(|| { - CommittedVersion { - scope: scope.clone(), - position: cache.applied.position, - } - }) - }); - let Some(candidate) = candidate else { - continue; - }; - let Some(minimum) = read.minimum.clone() else { - self.finish_read(&id, Err(CommandFailure::CorruptState)); - continue; - }; - match minimum.satisfied(&candidate, &scope) { - Ok(false) => continue, - Err(error) => { - self.finish_read(&id, Err(error)); - continue; - } - Ok(true) => (), - } - read.phase = ReadPhase::Executing; - let super::query::ReadOperation::Query { contract, input } = &read.operation - else { - continue; - }; - let action = Action::Query { - contract: contract.clone(), - input: input.clone(), - minimum, - control: read.request.control.clone(), - source_deadline: read.source_deadline, - }; - self.sql(&scope, SqlPurpose::Query(id), action); - } - _ => (), - } - } - } - - fn on_read_source(&mut self, id: ReadId, result: Result) { - let Some(read) = self.reads.get_mut(&id) else { - return; - }; - if read.phase != ReadPhase::Probing { - return; - } - let now = crate::asyncrt::mono_ms(); - match result { - Err(RuntimeError::NotReady | RuntimeError::Overloaded) => { - read.phase = ReadPhase::Source; - read.retry_at = now.saturating_add(self.options.poll_ms); - } - Ok(status) - if self - .cells - .get(&read.request.scope) - .is_some_and(|cell| status.replica == *cell.group.identity()) => - { - if let Some(deadline) = super::query::recent_source_deadline( - status.source_observed_ms, - now, - self.options.stale_source_timeout_ms, - ) { - read.source_deadline = Some(deadline); - read.phase = ReadPhase::Image; - } else { - self.finish_read(&id, Err(CommandFailure::VersionUnavailable)); - } - } - _ => self.finish_read(&id, Err(CommandFailure::VersionUnavailable)), - } - } - - fn on_read_leader( - &mut self, - id: ReadId, - check: u64, - result: Result, - ) { - let Some(read) = self.reads.get_mut(&id) else { - return; - }; - if check != read.leader_check - || !matches!(read.phase, ReadPhase::CheckingLeader | ReadPhase::Barrier) - { - return; - } - read.checking_leader = false; - read.retry_at = crate::asyncrt::mono_ms().saturating_add(self.options.poll_ms); - let status = match result { - Ok(status) - if self - .cells - .get(&read.request.scope) - .is_some_and(|cell| status.replica == *cell.group.identity()) => - { - status - } - Err(RuntimeError::NotReady | RuntimeError::Overloaded) => { - if read.phase == ReadPhase::CheckingLeader { - read.phase = ReadPhase::StartBarrier; - } - return; - } - _ => { - self.finish_read(&id, Err(CommandFailure::VersionUnavailable)); - return; - } - }; - let leader = (status.term, status.leader, status.is_leader); - if read.phase == ReadPhase::Barrier { - // RawNode may clear a pending read on role change, or its reply - // may be lost without any leadership change. Retry the same - // invocation after a checked observation and bounded delay. The - // original deadline remains fixed; status never supplies proof. - if read.leader != Some(leader) || crate::asyncrt::mono_ms() >= read.barrier_retry_at { - read.phase = ReadPhase::StartBarrier; - } - return; - } - if status.leader.is_none() - || status.applied.term != status.term - || (status.is_leader && !status.protocol_ready) - { - read.phase = ReadPhase::StartBarrier; - return; - } - let Some(attempt) = read.barrier_attempt.checked_add(1) else { - self.finish_read(&id, Err(CommandFailure::Overloaded)); - return; - }; - read.barrier_attempt = attempt; - read.leader = Some(leader); - read.phase = ReadPhase::Barrier; - read.barrier_retry_at = crate::asyncrt::mono_ms() - .saturating_add(self.options.poll_ms.saturating_mul(4).clamp(100, 1000)); - let scope = read.request.scope.clone(); - let runtime = self.runtime.clone(); - self.jobs.push(Box::pin(async move { - Job::ReadStarted { - id, - attempt, - result: runtime.read_index(&scope, id.to_vec()).await, - } - })); - } - - fn status(&mut self, scope: &GroupScope, purpose: StatusPurpose) { - let Some(cell) = self.cells.get_mut(scope) else { - return; - }; - if cell.status_busy { - if let StatusPurpose::Observe(mut request) = purpose { - reply(&mut request, Err(CommandFailure::Overloaded)); - } - return; - } - let Some(sequence) = cell.sequence.checked_add(1) else { - self.fail(scope); - return; - }; - cell.sequence = sequence; - cell.status_busy = true; - let runtime = self.runtime.clone(); - let scope = scope.clone(); - self.jobs.push(Box::pin(async move { - let result = runtime.status(&scope).await; - Job::Status { - scope, - sequence, - purpose, - result, - } - })); - } - - fn sql(&mut self, scope: &GroupScope, purpose: SqlPurpose, action: Action) { - let Some(cell) = self.cells.get_mut(scope) else { - return; - }; - cell.sql_busy = true; - let work = match &purpose { - SqlPurpose::Query(id) | SqlPurpose::Seed(id) => { - self.reads.get(id).map(|read| read.request.work.clone()) - } - _ => cell.pending.as_ref().map(|p| p.request.work.clone()), - }; - let cancel = match &purpose { - SqlPurpose::Query(id) | SqlPurpose::Seed(id) => { - self.reads.get(id).map(|read| read.request.control.clone()) - } - SqlPurpose::Stage => cell - .pending - .as_ref() - .map(|pending| pending.request.control.clone()), - _ => None, - }; - let encoding_work = work.clone(); - let job = self - .pool - .call(cell.worker, scope.clone(), action, work, cancel); - let scope = scope.clone(); - self.jobs.push(Box::pin(async move { - Job::Sql { - scope, - purpose, - result: Box::new(job.await), - work: encoding_work, - } - })); - } - - fn finish_pending(&mut self, scope: &GroupScope, result: Result) { - self.finish_pending_charged(scope, result, None); - } - - fn finish_pending_charged( - &mut self, - scope: &GroupScope, - result: Result, - bytes: Option, - ) { - if let Some(cell) = self.cells.get_mut(scope) { - if let Some(mut pending) = cell.pending.take() { - let result = if pending.request.control.cancelled() - && (result.is_ok() - || pending.duplicate && matches!(result, Err(CommandFailure::Unavailable))) - { - Err( - if pending.request.control.submitted() || pending.duplicate { - CommandFailure::Ambiguous { - command: pending.options.id, - } - } else { - CommandFailure::Unavailable - }, - ) - } else { - result - }; - reply_charged(&mut pending.request, result, bytes); - } - } - self.active.remove(scope); - self.admission_waiters.retain(|queued| queued != scope); - self.admission_at = 0; - } - - #[track_caller] - fn fail(&mut self, scope: &GroupScope) { - if let Some(cell) = self.cells.get_mut(scope) { - if !cell.failed { - tracing::error!(group = ?scope.group, location = %std::panic::Location::caller(), "Raft host closed group admission after a failure"); - } - cell.failed = true; - cell.core.close(); - if let Some(pending) = cell.pending.as_mut() { - if pending.request.control.submitted() { - self.uncertain.insert(scope.clone(), pending.options.id); - } - } - } - let result = self - .cells - .get(scope) - .and_then(|c| c.pending.as_ref()) - .map(|p| { - if p.request.control.submitted() { - CommandFailure::Ambiguous { - command: p.options.id, - } - } else { - CommandFailure::CorruptState - } - }) - .unwrap_or(CommandFailure::CorruptState); - self.finish_pending(scope, Err(result)); - } - - fn refuse(&mut self, scope: &GroupScope, error: CommandFailure) { - if let Some(pending) = self - .cells - .get_mut(scope) - .and_then(|cell| cell.pending.as_mut()) - { - if pending.request.control.submitted() { - reply( - &mut pending.request, - Err(CommandFailure::Ambiguous { - command: pending.options.id, - }), - ); - pending.phase = Phase::Waiting; - } else { - reply(&mut pending.request, Err(error)); - pending.phase = Phase::Discard; - } - } - } - - fn on_routed_query(&mut self, mut request: Request) { - let scope = request.scope.clone(); - let Ok(binding) = self - .local - .bindings - .get(&(scope.namespace, scope.cell.clone())) - .and_then(|binding| { - if binding.scope == scope { - Ok(binding) - } else { - Err(CommandFailure::CommandConflict) - } - }) - else { - reply(&mut request, Err(CommandFailure::Unavailable)); - return; - }; - let Kind::Query { - contract, - input, - options, - } = std::mem::replace(&mut request.kind, Kind::Status) - else { - reply(&mut request, Err(CommandFailure::UnsupportedCapability)); - return; - }; - if contract.kind != OperationKind::Query { - reply(&mut request, Err(CommandFailure::UnsupportedCapability)); - return; - } - let minimum = if let client::QueryConsistency::AtLeast(token) = &options.consistency { - match u64::try_from(crate::asyncrt::wall_ms()) - .map_err(|_| super::version::VersionError::Expired) - .and_then(|now| { - self.versions - .verify(token, &scope, binding.class.schema_digest(), now) - }) { - Ok(version) => Some(version), - Err(error) => { - let failure = if error == super::version::VersionError::WrongHistory { - CommandFailure::WrongIncarnation - } else { - CommandFailure::VersionUnavailable - }; - reply(&mut request, Err(failure)); - return; - } - } - } else { - None - }; - let context = super::query_route::RouteContext { - member: binding.member(), - leader_hint: binding.leader_hint, - local: crate::application::consensus::RaftCellHandle { - handle: self.local.clone(), - binding, - }, - runtime: self.runtime.clone(), - transport: self.transport.clone(), - keys: self.versions.clone(), - control: request.control.clone(), - minimum, - }; - self.jobs.push(Box::pin(async move { - Job::Routed { - request: Box::new(request), - result: context.run(contract, input, options).await, - } - })); - } - - fn on_request(&mut self, mut request: Request) { - self.reconcile_membership(&request.scope); - let scope = request.scope.clone(); - if matches!(&request.kind, Kind::Inspect | Kind::InspectPeer(_)) { - if self.closing.load(Ordering::Acquire) { - reply(&mut request, Err(CommandFailure::Unavailable)); - } else if !request.control.cancelled() { - match request.kind { - Kind::InspectPeer(peer) if peer != self.log.limits().node => { - self.start_peer_inspection(request, peer) - } - _ => self.start_inspection(request), - } - } - return; - } - let system = matches!(&request.kind, Kind::SystemCommand { .. }); - let admission = if matches!( - &request.kind, - Kind::Command { .. } | Kind::SystemCommand { .. } - ) { - match self.archive_admission() { - admission::Decision::Granted if !self.admission_waiters.is_empty() => { - admission::Decision::Deferred - } - decision => decision, - } - } else { - admission::Decision::Refused - }; - if self.closing.load(Ordering::Acquire) - || self - .cells - .get(&scope) - .is_some_and(|cell| cell.failed || cell.retired) - { - reply(&mut request, Err(CommandFailure::Unavailable)); - return; - } - if request.control.cancelled() { - return; - } - if matches!( - &request.kind, - Kind::ActivateLearner(_) | Kind::ActivateManaged { .. } | Kind::ActivateInitial(_) - ) { - match std::mem::replace(&mut request.kind, Kind::Status) { - Kind::ActivateLearner(activation) => { - self.activate_learner(request, activation, None) - } - Kind::ActivateManaged { - activation, - authorization, - } => self.activate_learner(request, activation, Some(*authorization)), - Kind::ActivateInitial(authorization) => { - self.activate_initial(request, *authorization) - } - _ => reply(&mut request, Err(CommandFailure::CorruptState)), - } - return; - } - if matches!(&request.kind, Kind::Seed { .. }) { - if let Kind::Seed { - target, - kind, - charge, - } = std::mem::replace(&mut request.kind, Kind::Status) - { - self.start_seed(request, target, kind, charge); - } - return; - } - if matches!(&request.kind, Kind::Query { options, .. } if !matches!(options.routing, client::QueryRouting::Local)) - { - self.on_routed_query(request); - return; - } - if matches!( - &request.kind, - Kind::BeginRepair { .. } | Kind::AdvanceRepair { .. } | Kind::Fleet(_) - ) { - match std::mem::replace(&mut request.kind, Kind::Status) { - Kind::BeginRepair { plan } => self.begin_repair(request, plan, None), - Kind::AdvanceRepair { - id, - step, - expected_revision, - } => self.advance_repair(request, id, step, expected_revision, None), - Kind::Fleet(action) => self.fleet_configuration(request, *action), - _ => reply(&mut request, Err(CommandFailure::CorruptState)), - } - return; - } - if matches!(&request.kind, Kind::AddLearner { .. }) { - if let Kind::AddLearner { - member, - expected_revision, - } = std::mem::replace(&mut request.kind, Kind::Status) - { - self.add_learner(request, member, expected_revision); - } - return; - } - let Some(cell) = self.cells.get_mut(&scope) else { - reply(&mut request, Err(CommandFailure::Unavailable)); - return; - }; - if cell.failed || self.closing.load(Ordering::Acquire) { - reply(&mut request, Err(CommandFailure::Unavailable)); - return; - } - if cell.incoming.is_some() - && matches!( - &request.kind, - Kind::Command { .. } | Kind::SystemCommand { .. } - ) - { - reply(&mut request, Err(CommandFailure::Unavailable)); - return; - } - if request.control.cancelled() { - return; - } - match std::mem::replace(&mut request.kind, Kind::Status) { - Kind::ReplicationStatus => { - let runtime = self.runtime.clone(); - self.jobs.push(Box::pin(async move { - let result = runtime - .replication_status(&scope) - .await - .map(Response::Replication); - Job::Leadership { - request: Box::new(request), - result, - } - })); - } - Kind::TransferLeadership(transfer) => { - if cell.pending.is_some() || cell.configuration.is_some() || cell.incoming.is_some() - { - reply(&mut request, Err(CommandFailure::Unavailable)); - return; - } - if !request.control.begin_submission() { - return; - } - let runtime = self.runtime.clone(); - self.jobs.push(Box::pin(async move { - let result = if request.control.cancelled() { - Err(RuntimeError::Closed) - } else { - runtime - .transfer_leadership(&scope, transfer) - .await - .map(|()| Response::LeadershipRequested) - }; - Job::Leadership { - request: Box::new(request), - result, - } - })); - } - Kind::RestartLeadership(mut check) => { - if check.transfer.is_some() - && (cell.pending.is_some() - || cell.configuration.is_some() - || cell.incoming.is_some()) - { - reply(&mut request, Err(CommandFailure::Unavailable)); - return; - } - if !request.control.begin_submission() { - return; - } - check.deadline = check.deadline.min(request.control.deadline()); - let runtime = self.runtime.clone(); - self.jobs.push(Box::pin(async move { - let result = if request.control.cancelled() { - Err(RuntimeError::Closed) - } else { - runtime - .restart_leadership(&scope, check) - .await - .map(|()| Response::LeadershipRequested) - }; - Job::Leadership { - request: Box::new(request), - result, - } - })); - } - Kind::Inspect - | Kind::InspectPeer(_) - | Kind::AddLearner { .. } - | Kind::ActivateLearner(_) - | Kind::ActivateManaged { .. } - | Kind::ActivateInitial(_) - | Kind::Fleet(_) - | Kind::Seed { .. } - | Kind::BeginRepair { .. } - | Kind::AdvanceRepair { .. } => { - reply(&mut request, Err(CommandFailure::UnsupportedCapability)) - } - Kind::Status => self.status(&scope, StatusPurpose::Observe(Box::new(request))), - Kind::RepairStatus => { - let result = cell - .group - .membership() - .map_err(|_| CommandFailure::CorruptState) - .and_then(|membership| { - membership - .repair_status() - .map_err(|_| CommandFailure::CorruptState) - }) - .map(Response::Repair); - reply(&mut request, result); - } - Kind::FleetStatus => { - let result = cell - .group - .membership() - .map(|membership| Response::Fleet(membership.fleet_status().map(Box::new))) - .map_err(|_| CommandFailure::CorruptState); - reply(&mut request, result); - } - Kind::ObserveFleet(authorization) => { - self.start_fleet_observation(request, *authorization); - } - Kind::Query { - contract, - input, - options, - } => { - if contract.kind != OperationKind::Query { - reply(&mut request, Err(CommandFailure::UnsupportedCapability)); - return; - } - let minimum = if let client::QueryConsistency::AtLeast(token) = &options.consistency - { - let decoded = u64::try_from(crate::asyncrt::wall_ms()) - .map_err(|_| super::version::VersionError::Expired) - .and_then(|now| { - self.versions.verify(token, &scope, cell.schema_digest, now) - }); - match decoded { - Ok(version) => Some(version), - Err(error) => { - let failure = if error == super::version::VersionError::WrongHistory { - CommandFailure::WrongIncarnation - } else { - CommandFailure::VersionUnavailable - }; - reply(&mut request, Err(failure)); - return; - } - } - } else { - None - }; - let Some(sequence) = self.read_sequence.checked_add(1) else { - reply(&mut request, Err(CommandFailure::Overloaded)); - return; - }; - self.read_sequence = sequence; - let mut id = [0; 32]; - id[..8].copy_from_slice(b"CLRRD001"); - id[8..24].copy_from_slice(&self.read_nonce); - id[24..].copy_from_slice(&sequence.to_be_bytes()); - self.reads.insert( - id, - PendingRead::new(request, contract, input, &options, minimum), - ); - } - Kind::Command { - contract, - input, - options, - } - | Kind::SystemCommand { - contract, - input, - options, - } => { - // Background SQL does not consume the one business-command slot. - // Keep the bounded request until the physical worker completes; - // pump() rechecks its original deadline before staging. - if cell.pending.is_some() { - reply(&mut request, Err(CommandFailure::Overloaded)); - return; - } - cell.pending = Some(Pending { - allow_new: admission == admission::Decision::Granted, - archive_bytes: None, - admission_wait: (admission == admission::Decision::Deferred).then(|| { - self.local - .performance - .timer(super::performance::Operation::ArchiveAdmissionWait) - }), - system, - proof: None, - request, - options, - contract, - input: Some(input), - identity: None, - prepared: None, - result: None, - duplicate: false, - phase: if admission == admission::Decision::Deferred { - Phase::AwaitAdmission - } else { - Phase::Start - }, - _bytes: None, - }); - let sql_idle = !cell.sql_busy; - self.active.insert(scope.clone()); - if admission == admission::Decision::Deferred { - self.admission_waiters.push_back(scope); - } else if sql_idle { - self.status(&scope, StatusPurpose::Start); - } - } - } - } - - fn on_status( - &mut self, - scope: GroupScope, - sequence: u64, - purpose: StatusPurpose, - result: Result, - ) { - let Some(cell) = self.cells.get_mut(&scope) else { - return; - }; - cell.status_busy = false; - let accepted = result.and_then(|status| { - if status.replica != *cell.group.identity() - || cell.group.read()?.hard_state.term > status.term.get() - { - return Err(RuntimeError::NotReady); - } - if let Some(progress) = progress(&cell.group, cell.image.as_ref(), cell.archived)? { - cell.core - .observe_progress(progress) - .map_err(|_| RuntimeError::StaleCompletion)?; - } - let change = cell - .core - .observe_leadership(status.leadership_observation(sequence)?) - .map_err(|_| RuntimeError::StaleCompletion)?; - if change.retired.is_some() { - if let Some(pending) = cell.pending.as_mut() { - let error = if pending.request.control.submitted() { - CommandFailure::Ambiguous { - command: pending.options.id, - } - } else { - CommandFailure::NotLeader { - leader: status.leader.and_then(|id| MemberId::new(id).ok()), - } - }; - reply(&mut pending.request, Err(error)); - } - } - cell.archive_observation.observe( - status.committed, - cell.archived, - crate::asyncrt::mono_ms(), - ); - Ok(status) - }); - if let StatusPurpose::Observe(mut request) = purpose { - let result = accepted.and_then(|status| { - let now = self - .log - .domain() - .mono_ms() - .map_err(|_| RuntimeError::Closed)?; - let registered_queries = self - .local - .bindings - .get(&(scope.namespace, scope.cell.clone())) - .map_err(|_| RuntimeError::Closed)? - .class - .has_queries(); - let (role, queries) = readiness::capture( - cell, - &status, - now, - self.options.stale_source_timeout_ms, - self.local.admission.is_draining(), - registered_queries, - )?; - Ok(Response::Status(client::CellStatus { - role, - queries, - leader: status.leader.and_then(|id| MemberId::new(id).ok()), - term: status.term, - command_ready: role == crate::application::consensus::RaftReplicaRole::Leader - && cell.pending.is_none() - && cell.incoming.is_none() - && cell.core.authority().is_some() - && status.protocol_ready - && !status.awaiting_apply - && !status.awaiting_ready - && status.persisted == status.committed - && status.committed == status.applied.index, - version: cell.image.as_ref().map(|image| CommittedVersion { - scope: scope.clone(), - position: image.position, - }), - persisted: status.persisted, - committed: status.committed, - applied: status.applied, - checkpoint: status.checkpoint, - archived: cell.archived, - archive: cell.archive_observation.capture( - status.committed, - cell.archived, - now, - cell.archive_retained.bytes(&cell.group, cell.archived)?, - self.archive_owner - .as_ref() - .is_some_and(|owner| owner.inflight.contains(&scope)), - ), - membership_revision: status.membership_revision, - member_high_water: status.member_high_water, - })) - }); - reply( - &mut request, - result.map_err(|_| CommandFailure::Unavailable), - ); - return; - } - let status = match accepted { - Ok(status) => status, - Err(_) => { - if matches!(purpose, StatusPurpose::Start | StatusPurpose::Submit) { - self.refuse(&scope, CommandFailure::Unavailable); - } - return; - } - }; - let Some(pending) = cell.pending.as_mut() else { - return; - }; - if pending.request.control.cancelled() && !pending.request.control.submitted() { - pending.phase = Phase::Discard; - return; - } - match purpose { - StatusPurpose::Start => { - // A reader or archive capture may acquire SQL while status is - // in flight. Keep one bounded command and recheck authority - // after that work physically completes. - if cell.sql_busy { - pending.phase = Phase::Start; - return; - } - let Some(authority) = cell.core.authority().cloned() else { - if status.is_leader { - pending.phase = Phase::AwaitStart; - return; - } - self.refuse( - &scope, - CommandFailure::NotLeader { - leader: status.leader.and_then(|id| MemberId::new(id).ok()), - }, - ); - return; - }; - let mut bytes = [0; 16]; - self.rng.fill_bytes(&mut bytes); - let Ok(proposal) = ProposalId::new(bytes) else { - self.refuse(&scope, CommandFailure::Unavailable); - return; - }; - let identity = match cell.core.admit( - &authority, - pending.options.id, - proposal, - pending.options.durability, - ) { - Ok(identity) => identity, - Err(Refusal::LeaderNotReady) => { - pending.phase = Phase::AwaitStart; - return; - } - Err(_) => { - self.refuse(&scope, CommandFailure::Unavailable); - return; - } - }; - pending.identity = Some(identity.clone()); - let Some(input) = pending.input.take() else { - self.refuse(&scope, CommandFailure::Unavailable); - return; - }; - let mut preview = [0; 16]; - self.rng.fill_bytes(&mut preview); - let Ok(preview) = ImageId::new(preview) else { - self.refuse(&scope, CommandFailure::Unavailable); - return; - }; - let durability = match cell.core.receipt_durability(&identity) { - Ok(value) => value, - Err(_) => { - self.refuse(&scope, CommandFailure::Unavailable); - return; - } - }; - let mut random = [0; 32]; - self.rng.fill_bytes(&mut random); - let attempt = CommandAttempt { - request: CommandRequest { - id: pending.options.id, - operation: pending.contract.operation.clone(), - contract_digest: pending.contract.digest, - request_digest: command_request_digest(&input), - durability, - retry_until_ms: pending.options.retry_until_ms, - }, - proposal, - preview, - inputs: crate::storage::CommandInputs { - now_unix_millis: crate::asyncrt::wall_ms().max(0) as u64, - random, - }, - }; - pending.phase = Phase::Staging; - let allow_new = pending.allow_new; - let system = pending.system; - self.sql( - &scope, - SqlPurpose::Stage, - Action::Stage { - identity, - status: Box::new(status), - attempt, - input, - allow_new, - system, - }, - ); - } - StatusPurpose::Submit => { - let Some(identity) = pending.identity.clone() else { - self.refuse(&scope, CommandFailure::Unavailable); - return; - }; - if !pending.request.control.begin_submission() { - pending.phase = Phase::Discard; - return; - } - if let Err(reason) = cell.core.mark_submitted(&identity) { - pending.request.control.refuse_preflight(); - if reason == Refusal::LeaderNotReady { - // Ready persistence or protocol-only work may be in - // flight between two status requests. Keep the frozen - // private result; retry the authority check on a tick, - // under the original attempt deadline, before any RPC. - tracing::debug!(?reason, "Raft private submission preflight deferred"); - pending.phase = Phase::AwaitSubmit; - return; - } - self.refuse(&scope, CommandFailure::Unavailable); - return; - } - pending.phase = Phase::Marking; - self.sql(&scope, SqlPurpose::Mark, Action::Submit { identity }); - } - StatusPurpose::Finish => { - let (Some(identity), Some(receipt)) = - (pending.identity.as_ref(), pending.result.as_ref()) - else { - return; - }; - let result = if pending.duplicate { - cell.core.recover_receipt( - identity, - receipt.receipt().command(), - CommittedVersion { - scope: scope.clone(), - position: receipt.position(), - }, - receipt.receipt().minimum_durability(), - ) - } else { - cell.core.verify_receipt( - identity, - receipt.receipt().command(), - receipt.position(), - receipt.receipt().minimum_durability(), - ) - }; - if let Err(reason) = result { - if reason == Refusal::LeaderNotReady { - pending.phase = Phase::AwaitFinish; - return; - } - self.refuse(&scope, CommandFailure::Unavailable); - return; - } - if cell.archived >= receipt.position().index { - if let Some((position, entry_digest)) = pending.proof { - if cell - .core - .observe_proof(&OutputProof::Raft(Box::new(RaftProof { - command: identity.clone(), - position, - entry_digest, - stage: ProofStage::Archived, - }))) - .is_err() - { - self.fail(&scope); - return; - } - } - } - match cell.core.finish(identity) { - Ok(permit) => { - let token = u64::try_from(crate::asyncrt::wall_ms()) - .ok() - .and_then(|now| { - self.versions - .issue(permit.version(), cell.schema_digest, now) - .ok() - }); - let Some(version_token) = token else { - let id = permit.command(); - self.finish_pending( - &scope, - Err(CommandFailure::Ambiguous { command: id }), - ); - return; - }; - let Some(bytes) = self.budget.acquire(receipt.receipt().result().len()) - else { - let id = permit.command(); - self.finish_pending( - &scope, - Err(CommandFailure::Ambiguous { command: id }), - ); - return; - }; - let response = client::CommandReceipt { - id: permit.command(), - version: permit.version().clone(), - durability: permit.durability(), - result: receipt.receipt().result().to_vec(), - version_token, - }; - self.finish_pending_charged( - &scope, - Ok(Response::Command(response)), - Some(bytes), - ); - } - Err(Refusal::Unproven) => pending.phase = Phase::AwaitFinish, - Err(_) => self.refuse(&scope, CommandFailure::Unavailable), - } - } - StatusPurpose::Poll => { - if let Some(identity) = pending.identity.clone() { - if identity.authority.term < status.term { - if let Ok(barrier) = cell.core.begin_retirement(&identity) { - pending.phase = Phase::Retiring; - self.sql( - &scope, - SqlPurpose::Retire, - Action::Retire { identity, barrier }, - ); - } - return; - } - } - if pending.result.is_some() { - pending.phase = Phase::Finish; - } - } - StatusPurpose::Observe(_) => (), - } - } - - fn on_sql( - &mut self, - scope: GroupScope, - purpose: SqlPurpose, - result: Result, - work: Option>, - ) { - let result = match result { - Ok(sql::Output { - value: - Value::SnapshotCapture { - capture, - reservation, - }, - _bytes, - }) => { - let Some(cell) = self.cells.get_mut(&scope) else { - return; - }; - cell.sql_busy = false; - if !matches!( - &purpose, - SqlPurpose::Seed(_) | SqlPurpose::Checkpoint | SqlPurpose::SnapshotSource(_) - ) { - self.fail(&scope); - return; - } - self.encode_snapshot(scope, purpose, capture, reservation, work, _bytes); - return; - } - result => result, - }; - if let SqlPurpose::EffectScan(sequence) = purpose { - self.finish_effect_scan(scope, sequence, result); - return; - } - if let SqlPurpose::Seed(id) = purpose { - self.seed_sql(scope, id, result); - return; - } - if matches!( - &purpose, - SqlPurpose::Checkpoint - | SqlPurpose::SnapshotSource(_) - | SqlPurpose::SnapshotPrepare(_) - | SqlPurpose::SnapshotPublish(_) - ) { - self.on_snapshot_sql(scope, purpose, result); - return; - } - let Some(cell) = self.cells.get_mut(&scope) else { - return; - }; - cell.sql_busy = false; - // A scan while this cell was busy cannot predict when SQL finishes. - // Wake archive selection for newly idle retained work instead of - // waiting a maintenance interval and missing the collection window. - if !matches!(purpose, SqlPurpose::CaptureArchive) - && cell - .group - .applied() - .is_ok_and(|applied| applied.position.index > cell.archived) - { - if let Some(owner) = self.archive_owner.as_mut() { - if !owner.inflight.contains(&scope) { - owner.scan_at = 0; - } - } - } - if let SqlPurpose::CaptureArchive = purpose { - let Some(owner) = self.archive_owner.as_mut() else { - return; - }; - if self.closing.load(Ordering::Acquire) { - owner.inflight.remove(&scope); - return; - } - match result { - Ok(sql::Output { - value: Value::ArchiveCut(Some(cut)), - _bytes, - }) => { - if owner.pending.is_empty() { - owner.publish_at = crate::asyncrt::mono_ms() - .saturating_add(super::archive::bundle::COALESCE_MS); - } - owner.pending.push(super::host_archive::Captured { - scope, - archive: cell.archive.clone(), - cut: *cut, - reservation: _bytes, - }); - } - _ => { - owner.inflight.remove(&scope); - owner.scan_at = 0; - cell.archive_retry_at = - crate::asyncrt::mono_ms().saturating_add(self.options.archive.retry_ms); - } - } - return; - } - if let SqlPurpose::Query(id) = purpose { - let Some(mut read) = self.reads.remove(&id) else { - return; - }; - if read.expired() { - read.request.control.cancel(); - reply(&mut read.request, Err(CommandFailure::VersionUnavailable)); - return; - } - if matches!( - &result, - Err(ExecutionError::Refused(CommandFailure::VersionUnavailable)) - ) && !self.closing.load(Ordering::Acquire) - { - read.phase = ReadPhase::Image; - read.retry_at = crate::asyncrt::mono_ms().saturating_add(self.options.poll_ms); - self.reads.insert(id, read); - return; - } - let mut bytes = None; - let result = match result { - Ok(sql::Output { - value: Value::Queried(value), - _bytes, - }) => { - bytes = _bytes; - if read.minimum.as_ref().is_some_and(|minimum| { - minimum.satisfied(&value.version, &scope) == Ok(true) - }) { - u64::try_from(crate::asyncrt::wall_ms()) - .ok() - .and_then(|now| { - self.versions - .issue(&value.version, cell.schema_digest, now) - .ok() - }) - .map(|version_token| { - Response::Query(client::QueryResult { - result: value.result, - version: value.version, - version_token, - served_by: cell.group.identity().member, - }) - }) - .ok_or(CommandFailure::VersionUnavailable) - } else { - Err(CommandFailure::CorruptState) - } - } - Err(error) => Err(failure(error)), - _ => Err(CommandFailure::CorruptState), - }; - if !self.closing.load(Ordering::Acquire) { - reply_charged(&mut read.request, result, bytes); - } - return; - } - let sql::Output { value, _bytes } = match result { - Ok(result) => result, - Err(error) => { - tracing::debug!(event = "raft_sql_failure", cell = %scope.cell, - operation = match &purpose { SqlPurpose::Stage => "stage", SqlPurpose::Prepare(_) => "apply", SqlPurpose::Publish(_) => "publish", _ => "lifecycle" }, - error = ?error); - if let ( - SqlPurpose::Prepare(event), - ExecutionError::Refused(CommandFailure::Overloaded), - ) = (&purpose, &error) - { - let Some(retry) = event.clone_apply() else { - self.fail(&scope); - return; - }; - cell.apply = Some(retry); - cell.apply_retry_at = - crate::asyncrt::mono_ms().saturating_add(self.options.poll_ms); - self.applying.insert(scope); - return; - } - if matches!( - purpose, - SqlPurpose::Discard | SqlPurpose::Prepare(_) | SqlPurpose::Publish(_) - ) { - tracing::error!(group = ?scope.group, ?error, "Raft image operation failed"); - self.fail(&scope); - } else { - self.refuse(&scope, failure(error)); - } - return; - } - }; - match (purpose, value) { - (SqlPurpose::Stage, Value::Staged(outcome)) => { - self.admission_at = 0; - if let Some(pending) = cell.pending.as_mut() { - pending._bytes = _bytes; - match outcome { - StageOutcome::Prepared(prepared) => { - pending.archive_bytes = Some(prepared.archive_bytes()); - pending.prepared = Some(prepared); - pending.phase = Phase::BeforeSubmit; - } - StageOutcome::Duplicate(receipt) => { - pending.request.control.observe_committed(); - pending.result = Some(receipt); - pending.duplicate = true; - pending.phase = Phase::Finish; - } - } - } - } - (SqlPurpose::Mark, Value::Submitted) => { - let Some(pending) = cell.pending.as_mut() else { - return; - }; - let (Some(prepared), Some(identity)) = - (pending.prepared.take(), pending.identity.clone()) - else { - self.fail(&scope); - return; - }; - pending.phase = Phase::Proposing; - let runtime = self.runtime.clone(); - self.jobs.push(Box::pin(async move { - let result = runtime - .propose(&scope, prepared.context(), prepared.bytes().to_vec()) - .await; - Job::Proposed { - scope, - identity, - result, - } - })); - } - (SqlPurpose::Discard, Value::Discarded) => { - if let Some(identity) = cell.pending.as_ref().and_then(|p| p.identity.as_ref()) { - if cell.core.discard_completed(identity).is_err() { - self.fail(&scope); - return; - } - } - self.finish_pending(&scope, Err(CommandFailure::Unavailable)); - } - (SqlPurpose::Retire, Value::Retired(barrier)) => { - if let Some(identity) = cell - .pending - .as_ref() - .and_then(|pending| pending.identity.as_ref()) - { - if cell.core.retirement_completed(identity, barrier).is_err() { - self.fail(&scope); - return; - } - } - self.finish_pending(&scope, Err(CommandFailure::Unavailable)); - } - (SqlPurpose::Prepare(_), Value::Prepared(completion)) => { - cell.apply_ticket = Some(completion.ticket.clone()); - let runtime = self.runtime.clone(); - self.jobs.push(Box::pin(async move { - Job::Installed { - scope, - result: runtime.complete_apply(completion).await, - } - })); - } - (SqlPurpose::Publish(ticket), Value::Published { view, receipt }) => { - cell.image = Some(view); - // The runtime can deliver the next committed entry while this - // image is being published. Publication consumed the scheduler - // slot, so restore it for that queued entry after SQL joins. - cell.apply_inflight = cell.apply.is_some(); - if cell.apply_inflight { - self.applying.insert(scope.clone()); - } - cell.apply_ticket = None; - // New durable work invalidates an earlier empty inventory. - // Keep per-cell failure backoff and the single producer; only - // idle scanning should wait for the maintenance interval. - if let Some(owner) = &mut self.archive_owner { - owner.scan_at = 0; - } - if let Ok(Some(progress)) = - progress(&cell.group, cell.image.as_ref(), cell.archived) - { - let _ = cell.core.observe_progress(progress); - } - if let Some((identity, receipt)) = receipt { - if cell.pending.as_ref().and_then(|p| p.identity.as_ref()) == Some(&identity) { - let proven = cell - .core - .bind_entry(&identity, ticket.position(), ticket.entry_digest()) - .and_then(|()| { - for stage in [ - ProofStage::Committed, - ProofStage::Applied, - ProofStage::Queryable, - ] { - cell.core.observe_proof(&OutputProof::Raft(Box::new( - RaftProof { - command: identity.clone(), - entry_digest: ticket.entry_digest(), - position: ticket.position(), - stage, - }, - )))?; - } - Ok(()) - }); - if proven.is_ok() { - if let Some(pending) = cell.pending.as_mut() { - pending.request.control.observe_committed(); - pending.proof = Some((ticket.position(), ticket.entry_digest())); - pending.result = Some(receipt); - pending.phase = Phase::Finish; - } - } else { - self.refuse(&scope, CommandFailure::Unavailable); - } - } - } - } - _ => self.fail(&scope), - } - } - - fn on_event(&mut self, event: RuntimeEvent) { - match &event.kind { - RuntimeEventKind::Message { replica, message } => { - if message.get_msg_type() == ::raft::prelude::MessageType::MsgSnapshot { - self.send_snapshot(&replica.scope, message.clone()); - return; - } - match self.transport.submit(&replica.scope, message.clone()) { - Ok(delivery) => { - let scope = replica.scope.clone(); - let member = message.to; - let permit = event._permit.clone(); - self.jobs.push(Box::pin(async move { - let result = delivery.accepted().await; - // Keep the runtime's send boundary owned until the - // transport accepts or refuses this exact message. - drop(permit); - Job::MessageDelivery { - scope, - member, - result, - } - })); - } - Err(_) => { - let runtime = self.runtime.clone(); - let scope = replica.scope.clone(); - let member = message.to; - let snapshot = - message.get_msg_type() == ::raft::prelude::MessageType::MsgSnapshot; - self.jobs.push(Box::pin(async move { - let _ = runtime.admit_unreachable(&scope, member).await; - if snapshot { - let _ = runtime.admit_snapshot_status(&scope, member, false).await; - } - Job::Hint - })); - } - } - } - RuntimeEventKind::Apply { ticket, .. } => { - let scope = ticket.replica().scope.clone(); - if let Some(cell) = self.cells.get_mut(&scope) { - if cell.apply.is_some() { - self.fail(&scope); - } else { - cell.archive_observation.observe( - ticket.position().index, - cell.archived, - crate::asyncrt::mono_ms(), - ); - cell.apply_inflight = true; - cell.apply = Some(event); - self.applying.insert(scope); - } - } - } - RuntimeEventKind::Applied { ticket, .. } => { - let scope = ticket.replica().scope.clone(); - if let Some(cell) = self.cells.get_mut(&scope) { - let protocol = cell.group.read().ok().is_some_and(|cache| { - cache - .entries - .get(&ticket.position().index.get()) - .is_some_and(|entry| { - (entry.get_entry_type() != raft::prelude::EntryType::EntryNormal - || entry.data.is_empty()) - && entry.index == ticket.position().index.get() - && entry.term == ticket.position().term.get() - && super::completion::entry_digest(entry).ok() - == Some(ticket.entry_digest()) - }) - }); - if protocol && ticket.replica() == cell.group.identity() { - if cell.image.is_some() { - if let Some(owner) = &mut self.archive_owner { - owner.scan_at = 0; - } - } - } else if cell.apply_ticket.as_ref() == Some(ticket) { - cell.publication = Some(ticket.clone()); - self.applying.insert(scope); - } else { - self.fail(&scope); - } - } - } - RuntimeEventKind::Failed { replica, error } => { - tracing::error!(group = ?replica.scope.group, ?error, "Raft runtime failed"); - self.fail(&replica.scope); - } - RuntimeEventKind::Snapshot { .. } => self.receive_snapshot(event), - RuntimeEventKind::SnapshotInstalled { ticket, .. } => self.snapshot_installed(ticket), - RuntimeEventKind::SnapshotAborted { ticket, .. } => self.snapshot_aborted(ticket), - RuntimeEventKind::Checkpointed { ticket } => { - if let Some(cell) = self.cells.get_mut(&ticket.replica().scope) { - cell.checkpoint_busy = false; - cell.archive_checkpoint_at = crate::asyncrt::mono_ms() - .saturating_add(self.options.archive_checkpoint_ms); - } - } - RuntimeEventKind::ReadIndex { - replica, - index, - context, - } => { - let Ok(id) = ::try_from(context.as_slice()) else { - return; - }; - let Some(read) = self.reads.get_mut(&id) else { - return; - }; - if read.request.scope == replica.scope - && self - .cells - .get(&replica.scope) - .is_some_and(|cell| replica == cell.group.identity()) - && matches!( - read.phase, - ReadPhase::StartBarrier | ReadPhase::CheckingLeader | ReadPhase::Barrier - ) - { - if index.get() == 0 { - self.finish_read(&id, Err(CommandFailure::VersionUnavailable)); - } else { - read.minimum = Some(Minimum::ReadIndex(*index)); - read.phase = ReadPhase::Image; - } - } - } - _ => (), - } - } - - fn pump(&mut self) { - self.pump_directory(); - self.pump_admission(); - // Inventory scans are node maintenance, not per-frame work. Repeating - // them for every heartbeat makes N groups cost O(N²) per tick. - let now = crate::asyncrt::mono_ms(); - if now >= self.maintenance_at { - self.maintenance_at = now.saturating_add(self.options.poll_ms); - self.pump_extent_gc(); - self.pump_membership(); - self.pump_snapshots(); - } - for scope in self.applying.clone() { - let Some(cell) = self.cells.get_mut(&scope) else { - continue; - }; - if cell.failed { - self.applying.remove(&scope); - continue; - } - if cell - .incoming - .as_ref() - .is_some_and(|incoming| !incoming.allows_apply()) - { - continue; - } - if cell.sql_busy || cell.apply_retry_at > crate::asyncrt::mono_ms() { - continue; - } - if let Some(ticket) = cell.publication.take() { - self.sql( - &scope, - SqlPurpose::Publish(ticket.clone()), - Action::Publish { ticket }, - ); - self.applying.remove(&scope); - } else if let Some(event) = cell.apply.take() { - let mut bytes = [0; 16]; - self.rng.fill_bytes(&mut bytes); - let Ok(image) = ImageId::new(bytes) else { - self.fail(&scope); - continue; - }; - let Some(retry) = event.clone_apply() else { - self.fail(&scope); - continue; - }; - self.sql( - &scope, - SqlPurpose::Prepare(retry), - Action::Prepare { event, image }, - ); - self.applying.remove(&scope); - } - } - for scope in self.active.clone() { - let Some(cell) = self.cells.get_mut(&scope) else { - continue; - }; - if cell.failed || cell.sql_busy || cell.status_busy || cell.incoming.is_some() { - continue; - } - let Some(pending) = cell.pending.as_mut() else { - self.active.remove(&scope); - continue; - }; - if pending.request.control.cancelled() && !pending.request.control.submitted() { - pending.phase = Phase::Discard; - } - match pending.phase { - Phase::Start => self.status(&scope, StatusPurpose::Start), - Phase::BeforeSubmit => self.status(&scope, StatusPurpose::Submit), - Phase::Finish => self.status(&scope, StatusPurpose::Finish), - Phase::Discard => { - if let Some(identity) = pending.identity.clone() { - self.sql(&scope, SqlPurpose::Discard, Action::Discard { identity }); - } else { - self.finish_pending(&scope, Err(CommandFailure::Unavailable)); - } - } - _ => (), - } - } - self.pump_archive(); - self.pump_reads(); - self.pump_leader_load(); - self.pump_controller(); - self.publish_controller_resources(); - self.pump_effects(); - self.pump_node_drain(); - } - - fn pump_archive(&mut self) { - if self.closing.load(Ordering::Acquire) { - return; - } - let now = crate::asyncrt::mono_ms(); - // Already captured cuts must be published even when every slot is - // occupied. A partial batch's deadline cannot be renewed by slow SQL. - let publish = self.archive_owner.as_ref().is_some_and(|owner| { - !owner.pending.is_empty() - && (owner.pending.len() - >= self - .options - .archive - .max_batch_groups - .min(self.options.archive.max_producers) - || now >= owner.publish_at) - }); - if publish { - self.publish_archive_batch(); - } - let Some(owner) = self.archive_owner.as_mut() else { - return; - }; - if owner.inflight.len() >= self.options.archive.max_producers { - return; - } - if now < owner.scan_at { - return; - } - let eligible = |cell: &Cell| { - !cell.failed - && !cell.sql_busy - && cell.archive_retry_at <= now - && cell - .group - .applied() - .is_ok_and(|applied| applied.position.index > cell.archived) - }; - let mut captures = Vec::new(); - while owner.inflight.len() < self.options.archive.max_producers { - let next = super::host_archive::next_scope( - &self.cells, - owner.cursor.as_ref(), - |scope, cell| !owner.inflight.contains(scope) && eligible(cell), - ) - .cloned(); - let Some(scope) = next else { - owner.scan_at = now.saturating_add(self.options.poll_ms); - break; - }; - owner.inflight.insert(scope.clone()); - // Advance once per selected cell, independently of how completed - // captures are partitioned into compatible upload batches. - owner.cursor = Some(scope.clone()); - captures.push(scope); - } - let limits = self.options.archive.limits; - for scope in captures { - self.sql( - &scope, - SqlPurpose::CaptureArchive, - Action::CaptureArchive { limits }, - ); - } - } - - fn publish_archive_batch(&mut self) { - let Some(owner) = self.archive_owner.as_mut() else { - return; - }; - while !owner.pending.is_empty() { - let mut pending = vec![owner.pending.remove(0)]; - while pending.len() < self.options.archive.max_batch_groups { - let next = owner - .pending - .iter() - .position(|candidate| pending[0].archive.can_batch_with(&candidate.archive)); - let Some(next) = next else { - break; - }; - pending.push(owner.pending.remove(next)); - } - // Each cut has one bounded completion channel. The physical batch - // remains in jobs through joined completion, while a finished cell - // can release its own slot without waiting for another history. - let mut senders = Vec::with_capacity(pending.len()); - for capture in &pending { - let scope = capture.scope.clone(); - let position = capture.cut.position(); - let (send, receive) = tokio::sync::oneshot::channel(); - senders.push(Some(send)); - self.jobs.push(Box::pin(async move { - let completed = receive.await.unwrap_or(super::host_archive::Completed { - scope, - position, - result: Err(super::archive::ArchiveError::Unconfirmed), - }); - Job::Archived(vec![completed]) - })); - } - let verifier = owner.verifier.clone(); - let physical = owner.physical.clone(); - let prepared = self.transport.prepared_snapshot_cache(); - let timer = self - .local - .performance - .timer(super::performance::Operation::ArchivePublicationBatch); - self.jobs.push(Box::pin(async move { - let succeeded = super::host_archive::publish( - pending, - verifier, - prepared, - physical, - move |number, completed| { - if let Some(sender) = senders.get_mut(number).and_then(Option::take) { - let _ = sender.send(completed); - } - }, - ) - .await; - timer.finish(succeeded); - Job::ArchiveBatchJoined - })); - } - } - - fn on_job(&mut self, job: Job) { - match job { - Job::MessageDelivery { - scope, - member, - result, - } => { - if let Err(error) = result { - tracing::debug!(?error, "Raft message was not accepted"); - let runtime = self.runtime.clone(); - self.jobs.push(Box::pin(async move { - let _ = runtime.admit_unreachable(&scope, member).await; - Job::Hint - })); - } - } - Job::FleetMovementServiceStopped | Job::FleetRestartServiceStopped => (), - Job::FleetAction(completed) => self.finish_fleet_action(*completed), - Job::PeerFleetStop(completed) => self.finish_peer_fleet_stop(*completed), - Job::FleetAdmission(completed) => self.finish_fleet_admission(*completed), - Job::HistoryRegistered(completed) => self.finish_history_registration(*completed), - Job::ExtentGc(result) => self.extent_gc_completed(result), - Job::Drained(completed) => self.finish_node_drain(*completed), - Job::Controller(completed) => self.finish_controller(*completed), - Job::Effect(completed) => self.finish_effect(*completed), - Job::LeaderLoad(completed) => self.finish_leader_load(*completed), - Job::Inspection(completed) => self.finish_inspection(*completed), - Job::PeerInspection(completed) => self.finish_peer_inspection(*completed), - Job::NodeInspection(completed) => self.finish_node_inspection(*completed), - Job::PeerFleetInspection(completed) => self.finish_peer_fleet_inspection(*completed), - Job::LocalFleetInspection(completed) => self.finish_local_fleet_inspection(*completed), - Job::Directory(observation) => self.on_directory(*observation), - Job::Standby(completed) => self.on_standby_job(*completed), - Job::Leadership { - mut request, - result, - } => { - let result = if request.control.cancelled() || self.closing.load(Ordering::Acquire) - { - Err(CommandFailure::Unavailable) - } else { - result.map_err(|error| match error { - RuntimeError::NotLeader => CommandFailure::NotLeader { leader: None }, - RuntimeError::Overloaded | RuntimeError::NotReady => { - CommandFailure::Overloaded - } - _ => CommandFailure::Unavailable, - }) - }; - reply(&mut request, result); - } - Job::SeedArchive { id, result } => self.seed_archive(id, *result), - Job::Configured { scope, result } => self.configured(scope, result), - job @ (Job::SnapshotSent { .. } - | Job::SnapshotFetched { .. } - | Job::SnapshotAuthorized { .. } - | Job::SnapshotAbort { .. } - | Job::Checkpoint { .. }) => self.on_snapshot_job(job), - Job::ArchiveBatchJoined => (), - Job::Archived(completed) => { - self.admission_at = 0; - if let Some(owner) = self.archive_owner.as_mut() { - owner.scan_at = 0; - for result in &completed { - owner.inflight.remove(&result.scope); - } - } - for super::host_archive::Completed { - scope, - position, - result, - } in completed - { - use super::performance::Counter; - if let Err(error) = &result { - // Expected publication contention is common. Keep the - // typed cause available for diagnosis without warning - // on each advisory deferral or changing retry policy. - tracing::debug!( - event = "raft_archive_publication_failure", - node = ?self.log.limits().node, - group = ?scope.group, - index = position.index.get(), - error = ?error, - "Raft archive publication did not confirm the cut" - ); - } - self.local.performance.increment( - if result.is_ok() { - Counter::ArchivePublicationCutSucceeded - } else { - Counter::ArchivePublicationCutFailed - }, - 1, - ); - let Some(cell) = self.cells.get_mut(&scope) else { - if result.is_ok() { - self.local - .performance - .increment(Counter::ArchiveCoverageMissingCell, 1); - } - continue; - }; - if result.is_ok() && position.index > cell.archived { - // The proof names the captured local cut, even if another - // replica has already archived a later cut of this history. - // Never import that replica's later applied progress. - match progress(&cell.group, cell.image.as_ref(), position.index) { - Ok(Some(progress)) => { - if cell.core.observe_progress(progress).is_ok() { - self.local.performance.increment( - Counter::ArchiveCoverageIndexAdvance, - position.index.get() - cell.archived.get(), - ); - self.local - .performance - .increment(Counter::ArchiveCoverageAdvanced, 1); - cell.archived = position.index; - // The confirmed cut resolves this wait. Re-run - // the normal fresh authority/output checks now; - // it is not itself permission to send a result. - if let Some(pending) = cell.pending.as_mut() { - if pending.phase == Phase::AwaitFinish - && pending.result.as_ref().is_some_and(|receipt| { - receipt.position().index <= cell.archived - }) - { - pending.phase = Phase::Finish; - } - } - } else { - self.local - .performance - .increment(Counter::ArchiveCoverageRejected, 1); - } - } - Ok(None) => self - .local - .performance - .increment(Counter::ArchiveCoverageImagePending, 1), - Err(_) => self - .local - .performance - .increment(Counter::ArchiveCoverageRejected, 1), - } - } else if result.is_ok() { - self.local - .performance - .increment(Counter::ArchiveCoverageUnchanged, 1); - } - // A confirmed cut may already have newer work waiting. Keep it - // eligible for the next fair scan; back off failures or a proof - // that the local state machine could not accept. - let now = crate::asyncrt::mono_ms(); - if let Err(error) = &result { - cell.archive_observation.failed(error); - } - cell.archive_retry_at = if result.is_ok() && position.index <= cell.archived { - cell.archive_observation.confirmed(cell.archived, now); - now - } else { - now.saturating_add(self.options.archive.retry_ms) - }; - } - } - Job::Routed { - mut request, - result, - } => { - if request.control.cancelled() || self.closing.load(Ordering::Acquire) { - reply(&mut request, Err(CommandFailure::VersionUnavailable)); - } else { - match result { - Ok(output) => reply_charged( - &mut request, - Ok(Response::Query(output.result)), - Some(output.bytes), - ), - Err(error) => reply(&mut request, Err(error)), - } - } - } - Job::ReadSource { id, result } => self.on_read_source(id, result), - Job::ReadLeader { id, check, result } => self.on_read_leader(id, check, result), - Job::ReadStarted { - id, - attempt, - result, - } => { - if let Some(read) = self.reads.get_mut(&id) { - // A barrier event can arrive before the RPC acknowledgement. - // That late reply cannot regress an executing read. - if read.phase == ReadPhase::Barrier && read.barrier_attempt == attempt { - match result { - Ok(()) => (), - Err(RuntimeError::NotReady | RuntimeError::Overloaded) => { - read.phase = ReadPhase::StartBarrier; - read.checking_leader = false; - read.retry_at = - crate::asyncrt::mono_ms().saturating_add(self.options.poll_ms); - } - Err(_) => { - self.finish_read(&id, Err(CommandFailure::VersionUnavailable)) - } - } - } - } - } - Job::Status { - scope, - sequence, - purpose, - result, - } => self.on_status(scope, sequence, purpose, result), - Job::Sql { - scope, - purpose, - result, - work, - } => self.on_sql(scope, purpose, *result, work), - Job::SnapshotEncoded { - scope, - purpose, - result, - } => match purpose { - SqlPurpose::Seed(id) => self.seed_encoded(scope, id, *result), - SqlPurpose::Checkpoint | SqlPurpose::SnapshotSource(_) => { - self.on_snapshot_encoded(scope, purpose, *result) - } - _ => self.fail(&scope), - }, - Job::Installed { scope, result } => { - if let Err(error) = result { - tracing::error!(group = ?scope.group, ?error, "Raft durable apply completion failed"); - self.fail(&scope); - } - } - Job::Hint => (), - Job::Proposed { - scope, - identity, - result, - } => { - let Some(cell) = self.cells.get_mut(&scope) else { - return; - }; - if cell.pending.as_ref().and_then(|p| p.identity.as_ref()) != Some(&identity) { - return; - } - // Publication or a newer-term retirement may finish before this - // RPC reply. It cannot put that producer back into Waiting. - let waiting = cell - .pending - .as_ref() - .is_some_and(|p| p.phase == Phase::Proposing); - if waiting { - if let Some(pending) = cell.pending.as_mut() { - pending.phase = Phase::Waiting; - } - } - match result { - Ok(accepted) => { - if accepted.replica != identity.authority.replica { - self.fail(&scope); - } else { - // A retired gate may refuse this late observation. - // Only the verified committed receipt can release it. - let _ = cell.core.bind_entry( - &identity, - accepted.position, - accepted.entry_digest, - ); - } - } - Err(_) if waiting => self.refuse(&scope, CommandFailure::Unavailable), - Err(_) => (), - } - } - } - } - - async fn run( - &mut self, - mut requests: mailbox::Receiver, - mut controls: mailbox::Receiver, - mut closes: mpsc::UnboundedReceiver<()>, - ) { - let mut requests_open = true; - let mut controls_open = true; - let mut closes_open = true; - let Ok(now) = self.log.domain().mono_ms() else { - return; - }; - let mut poll_at = now.saturating_add(self.options.poll_ms); - loop { - if self.closing.load(Ordering::Acquire) { - return; - } - // Continue an admitted command before another administrative - // observer can occupy its serialized authority-status check. - self.pump(); - // Give admitted control work a bounded turn even while business - // traffic is continuously ready. The main select still services - // business, completions and consensus between these small batches. - for _ in 0..4 { - match controls.try_recv() { - Ok(request) => self.on_control(request), - Err(mpsc::error::TryRecvError::Empty) => break, - Err(mpsc::error::TryRecvError::Disconnected) => { - controls_open = false; - break; - } - } - } - self.pump(); - let wake_at = self - .archive_owner - .as_ref() - .filter(|owner| !owner.pending.is_empty()) - .map_or(poll_at, |owner| poll_at.min(owner.publish_at)); - let Ok(timer) = self.log.domain().sleep_until(wake_at) else { - return; - }; - crate::asyncrt::select! { - request=controls.recv(),if controls_open=>match request{Some(request)=>self.on_control(request),None=>controls_open=false}, - request=requests.recv(),if requests_open=>match request{Some(request)=>self.on_request(request),None=>requests_open=false}, - close=closes.recv(),if closes_open=>if close.is_some(){return;}else{closes_open=false;}, - event=self.events.recv()=>match event{Some(event)=>self.on_event(event),None=>return}, - job=self.jobs.next(),if !self.jobs.is_empty()=>if let Some(job)=job{self.on_job(job)}, - _=timer=>{let Ok(now)=self.log.domain().mono_ms() else{return;}; - if now < poll_at { continue; } - poll_at=now.saturating_add(self.options.poll_ms); - for scope in self.active.clone(){ - let purpose=self.cells.get(&scope).filter(|cell|!cell.status_busy).and_then(|cell|cell.pending.as_ref()).and_then(|pending|match pending.phase { - Phase::AwaitStart=>Some(StatusPurpose::Start),Phase::AwaitSubmit=>Some(StatusPurpose::Submit),Phase::AwaitFinish=>Some(StatusPurpose::Finish),Phase::Waiting=>Some(StatusPurpose::Poll),_=>None, - }); - if let Some(purpose)=purpose {self.status(&scope,purpose);} - }}, - } - } - } - - async fn shutdown(&mut self) -> Result { - self.closing.store(true, Ordering::Release); - RaftCells { - handle: self.local.clone(), - } - .stop_fleet_movement_service(); - RaftCells { - handle: self.local.clone(), - } - .stop_fleet_restart_service(); - if let Some(control) = self.history_registration.take() { - control.cancel(); - } - self.snapshot_source.take(); - if let Some(controller) = &self.controller { - controller.stop(); - } - self.close_configurations(); - if let Some(owner) = &mut self.archive_owner { - let _ = owner.cancellation.send(true); - owner.pending.clear(); - } - for (_, mut read) in std::mem::take(&mut self.reads) { - read.request.control.cancel(); - reply(&mut read.request, Err(CommandFailure::Unavailable)); - } - let mut unfinished = self - .uncertain - .iter() - .map(|(scope, id)| client::UnfinishedCommand { - scope: scope.clone(), - id: *id, - }) - .collect::>(); - for (scope, cell) in &mut self.cells { - cell.core.close(); - if let Some(pending) = cell.pending.as_mut() { - let submitted = pending.request.control.submitted(); - if submitted { - unfinished.push(client::UnfinishedCommand { - scope: scope.clone(), - id: pending.options.id, - }); - } - let error = if submitted { - CommandFailure::Ambiguous { - command: pending.options.id, - } - } else { - CommandFailure::Unavailable - }; - reply(&mut pending.request, Err(error)); - } - } - self.pool.stop(); - let runtime = self.runtime.close().await; - self.transport.close().await; - while self.jobs.next().await.is_some() {} - let sql = self.pool.join().await; - self.archive_owner.take(); - let node = NodeIdentity { - node: self.log.limits().node, - disk: self.log.limits().disk, - process: self.log.limits().process, - }; - let instance = self.log.storage_instance(); - let observations = (|| -> Result<_, CommandFailure> { - let mut groups = Vec::new(); - let mut groups_with_archive_lag = 0; - let mut bytes = 0usize; - for (scope, cell) in &self.cells { - let cache = cell - .group - .read() - .map_err(|_| CommandFailure::CorruptState)?; - let committed = LogIndex::new(cache.hard_state.commit); - if cell.archived < committed { - groups_with_archive_lag += 1; - } - let charge = scope - .cell - .len() - .checked_add(2048) - .ok_or(CommandFailure::CorruptState)?; - if groups.len() < self.options.close_report.max_groups - && bytes - .checked_add(charge) - .is_some_and(|n| n <= self.options.close_report.max_bytes) - { - bytes += charge; - groups.push(super::operations::RaftGroupCloseReport { - scope: scope.clone(), - member: cell.group.identity().member, - committed, - applied: cache.applied.position, - archived: cell.archived, - retired: cell.retired, - failed: cell.failed, - }); - } - } - Ok((groups, groups_with_archive_lag)) - })(); - let groups_total = self - .log - .group_count() - .map_err(|_| CommandFailure::CorruptState); - let attached = self.cells.len(); - self.cells.clear(); - self.standbys.clear(); - let log = self.log.close().await; - let closed_extents = if let Some(extents) = self.extents.take() { - match self.log.domain().blocking(move || drop(extents)) { - Ok(job) => job.await.is_ok(), - Err(_) => false, - } - } else { - true - }; - if runtime.is_err() || sql.is_err() || log.is_err() || !closed_extents { - return Err(CommandFailure::CorruptState); - } - if self.local.admission.is_poisoned() { - return Err(CommandFailure::Unavailable); - } - let (groups, groups_with_archive_lag) = observations?; - let groups_total = groups_total?; - let groups_with_unknown_archive = groups_total - .checked_sub(attached) - .ok_or(CommandFailure::CorruptState)?; - let omitted_groups = groups_total - groups.len(); - Ok(client::RaftCloseReport { - node, - storage_instance: instance.map_err(|_| CommandFailure::CorruptState)?, - groups, - groups_total, - omitted_groups, - groups_with_archive_lag, - groups_with_unknown_archive, - unfinished, - }) - } -} diff --git a/crates/cellar/durability/raft/host_admission.rs b/crates/cellar/durability/raft/host_admission.rs index 3771b2027..95b09e9bd 100644 --- a/crates/cellar/durability/raft/host_admission.rs +++ b/crates/cellar/durability/raft/host_admission.rs @@ -1,10 +1,8 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. //! Retained history refuses new work; temporary reservations defer callbacks. -use super::{Driver, Phase}; use crate::raft::{ cache::{CachedGroup, GroupLog}, - performance::{Counter as MeasuredCounter, Operation as MeasuredOperation}, storage::LogError, }; use cellar_logic::consensus::LogIndex; @@ -80,127 +78,3 @@ impl Retained { Ok(usage) } } - -#[derive(Clone, Copy, PartialEq, Eq)] -pub(super) enum Decision { - Granted, - Deferred, - Refused, -} - -impl Driver { - pub(super) fn archive_admission(&mut self) -> Decision { - let timer = self - .local - .performance - .timer(MeasuredOperation::ArchiveAdmissionCheck); - let decision = self.check_archive_admission(); - // This measures completed policy evaluation, including refusal and - // deferral. Decision counters retain the reason; success is not a grant. - timer.finish(true); - decision - } - - fn check_archive_admission(&mut self) -> Decision { - let reservation = self - .options - .execution - .max_image_bytes - .saturating_add(self.options.execution.envelope.max_payload_bytes as u64) - .saturating_add(self.options.execution.envelope.max_header_bytes as u64) - .saturating_add(1024); - let mut bytes = reservation; - let mut entries = 1u64; - let mut reserved_bytes = 0u64; - let mut reserved_entries = 0u64; - for cell in self.cells.values_mut() { - let Ok(usage) = cell.archive_retained.observe(&cell.group, cell.archived) else { - self.local - .performance - .increment(MeasuredCounter::ArchiveAdmissionObservationRefused, 1); - return Decision::Refused; - }; - bytes = bytes.saturating_add(usage.bytes); - entries = entries.saturating_add(usage.entries); - if let Some(pending) = cell.pending.as_ref().filter(|pending| pending.allow_new) { - reserved_bytes = - reserved_bytes.saturating_add(pending.archive_bytes.unwrap_or(reservation)); - reserved_entries = reserved_entries.saturating_add(1); - } - // Record the first encountered limit. These count decisions, - // including waiter rechecks, rather than terminal client outcomes. - if bytes > self.options.archive.max_unarchived_bytes { - self.local - .performance - .increment(MeasuredCounter::ArchiveAdmissionRetainedBytesRefused, 1); - return Decision::Refused; - } - if entries > self.options.archive.max_unarchived_entries as u64 { - self.local - .performance - .increment(MeasuredCounter::ArchiveAdmissionRetainedEntriesRefused, 1); - return Decision::Refused; - } - } - let (decision, counter) = - if bytes.saturating_add(reserved_bytes) > self.options.archive.max_unarchived_bytes { - ( - Decision::Deferred, - MeasuredCounter::ArchiveAdmissionReservedBytesDeferred, - ) - } else if entries.saturating_add(reserved_entries) - > self.options.archive.max_unarchived_entries as u64 - { - ( - Decision::Deferred, - MeasuredCounter::ArchiveAdmissionReservedEntriesDeferred, - ) - } else { - (Decision::Granted, MeasuredCounter::ArchiveAdmissionGranted) - }; - self.local.performance.increment(counter, 1); - decision - } - - pub(super) fn pump_admission(&mut self) { - let now = crate::asyncrt::mono_ms(); - if self.admission_waiters.is_empty() || now < self.admission_at { - return; - } - self.admission_at = now.saturating_add(self.options.poll_ms); - // These waiters retain the existing bounded request/input permits. - // They acquire no execution reservation and run no callback until - // granted. FIFO order and a bounded turn avoid starving other work. - for _ in 0..4 { - let Some(scope) = self.admission_waiters.pop_front() else { - break; - }; - let Some(pending) = self - .cells - .get(&scope) - .and_then(|cell| cell.pending.as_ref()) - else { - continue; - }; - if pending.phase != Phase::AwaitAdmission || pending.request.control.cancelled() { - continue; - } - let decision = self.archive_admission(); - if decision == Decision::Deferred { - self.admission_waiters.push_front(scope); - break; - } - if let Some(pending) = self - .cells - .get_mut(&scope) - .and_then(|cell| cell.pending.as_mut()) - { - pending.allow_new = decision == Decision::Granted; - if let Some(timer) = pending.admission_wait.take() { - timer.finish(pending.allow_new); - } - pending.phase = Phase::Start; - } - } - } -} diff --git a/crates/cellar/durability/raft/host_admission_tests.rs b/crates/cellar/durability/raft/host_admission_tests.rs index f21d53fda..1f73d1d5d 100644 --- a/crates/cellar/durability/raft/host_admission_tests.rs +++ b/crates/cellar/durability/raft/host_admission_tests.rs @@ -392,7 +392,28 @@ async fn cached_accounting_refuses_a_poisoned_group_cache() -> anyhow::Result<() .await? ); let result = retained.observe(&group, LogIndex::new(0)); - log.close().await?; + let closed = log.close().await; + assert!( + matches!(closed, Err(LogError::Corrupt(_))), + "poisoned cache must not report a healthy engine close: {closed:?}" + ); + assert!(matches!(log.storage_instance(), Err(LogError::Closed))); + // A typed corruption error still joins physical engine teardown. Reopen + // before shutting down the domain or dropping retained old group handles. + let mut resumed = options(root.path()); + resumed.mode = LogOpenMode::Resume; + resumed.process = ProcessGeneration::new([9; 16])?; + let reopened = domain + .run(ReplicaLog::open( + resumed, + Arc::new(cellar_ltx::DirectFileSystem), + )) + .await??; + assert_eq!( + reopened.group(&bootstrap(1).scope)?.identity().process, + ProcessGeneration::new([9; 16])? + ); + reopened.close().await?; domain.shutdown().await; assert!(matches!(result, Err(LogError::Corrupt(_)))); Ok(()) diff --git a/crates/cellar/durability/raft/host_controller.rs b/crates/cellar/durability/raft/host_controller.rs index 27f310919..7e88fa498 100644 --- a/crates/cellar/durability/raft/host_controller.rs +++ b/crates/cellar/durability/raft/host_controller.rs @@ -1,21 +1,8 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. -//! Owned reconciliation. Inspection supplies hints; ordinary revision-bound -//! host/runtime operations retain every quorum, seeding and progress check. -use super::*; -use crate::application::consensus::client::RaftCellHandle; -use crate::raft::inspection::{InspectionError, InspectionSnapshot}; -use crate::raft::member_state::{RepairPhase, RepairPlan, RepairStep}; -use cellar_logic::raft_leadership::LeaderPolicy; -use cellar_logic::raft_placement::PlacementPolicy; -use tokio::sync::watch; - -#[path = "controller_action.rs"] -mod action; -#[path = "placement.rs"] -mod policy; -#[path = "controller_probe.rs"] -mod probe; +//! Explicit automatic-placement policy. Runtime integration belongs to groups. +use super::RaftHostOptions; +use cellar_logic::{raft_leadership::LeaderPolicy, raft_placement::PlacementPolicy}; const PROBE_MEMORY: usize = 1024 * 1024; @@ -97,405 +84,3 @@ pub struct RaftControllerStatus { pub transfer_attempts: u64, pub completed_repairs: u64, } - -struct FailedProbe { - peer: RaftPeerNode, - since: u64, - last: u64, -} - -struct History { - binding: Binding, - candidate_cursor: Option, - pull_cursor: Option, - epoch: u64, - failures: BTreeMap, - leader_term: Option, - gained_at: u64, - seeding: Option<(ProposalId, u64)>, - last_completed: Option, -} - -pub(super) struct Controller { - options: RaftControllerOptions, - status: RaftControllerStatus, - histories: Vec, - inventory: usize, - active: BTreeSet, - cursor: usize, - active_cursor: Option, - priority_turn: bool, - intent: Option<(usize, RepairPlan)>, - started: Option, - last_transfer: u64, - next_at: u64, - sequence: u64, - epoch: u64, - busy: Option, - stop: watch::Sender, - memory: Arc, -} - -pub(super) enum Completed { - Observed { - sequence: u64, - epoch: u64, - history: usize, - result: Result, CommandFailure>, - }, - Acted { - sequence: u64, - epoch: u64, - history: usize, - result: Result<(), CommandFailure>, - }, -} - -impl Controller { - pub(super) fn new(options: RaftControllerOptions, bindings: Vec) -> Self { - let (stop, _) = watch::channel(false); - let histories = bindings - .into_iter() - .map(|binding| History { - binding, - candidate_cursor: None, - pull_cursor: None, - epoch: 0, - failures: BTreeMap::new(), - leader_term: None, - gained_at: 0, - seeding: None, - last_completed: None, - }) - .collect(); - Self { - status: RaftControllerStatus { - enabled: !options.start_paused, - recovering: true, - probe_rounds: 0, - repair_requests: 0, - activation_requests: 0, - transfer_attempts: 0, - completed_repairs: 0, - }, - memory: Budget::new(options.metadata_bytes, 1), - options, - histories, - inventory: 0, - active: BTreeSet::new(), - cursor: 0, - active_cursor: None, - priority_turn: false, - intent: None, - started: None, - last_transfer: 0, - next_at: 0, - sequence: 0, - epoch: 1, - busy: None, - stop, - } - } - - pub(super) fn stop(&self) { - self.stop.send_replace(true); - } - - fn next_sequence(&mut self) -> Option { - match self.sequence.checked_add(1) { - Some(sequence) => { - self.sequence = sequence; - Some(sequence) - } - None => { - self.status.enabled = false; - self.stop.send_replace(true); - None - } - } - } - - fn history(&mut self, index: usize, now: u64) -> Option<&mut History> { - let history = self.histories.get_mut(index)?; - if history.epoch != self.epoch { - history.epoch = self.epoch; - history.failures.clear(); - history.leader_term = None; - history.gained_at = now; - history.seeding = None; - } - Some(history) - } - - fn next_history(&mut self) -> Option { - if self.histories.is_empty() { - return None; - } - self.priority_turn = !self.priority_turn; - if self.priority_turn && !self.active.is_empty() { - let after = self - .active_cursor - .map_or(std::ops::Bound::Unbounded, std::ops::Bound::Excluded); - let index = self - .active - .range((after, std::ops::Bound::Unbounded)) - .next() - .or_else(|| self.active.first()) - .copied()?; - self.active_cursor = Some(index); - return Some(index); - } - let index = self.cursor; - self.cursor = (self.cursor + 1) % self.histories.len(); - Some(index) - } -} - -impl Driver { - pub(super) fn publish_controller_resources(&self) { - let sample = self.controller.as_ref().and_then(|controller| { - Some(( - self.log.domain().mono_ms().ok()?, - crate::application::consensus::RaftControllerResources { - enabled: controller.status.enabled, - recovering: controller.status.recovering, - busy: controller.busy.is_some(), - metadata: crate::application::consensus::RaftBudgetUsage::capture( - &controller.memory, - ), - probe_rounds: controller.status.probe_rounds, - repair_requests: controller.status.repair_requests, - activation_requests: controller.status.activation_requests, - transfer_attempts: controller.status.transfer_attempts, - completed_repairs: controller.status.completed_repairs, - sample_age_ms: 0, - }, - )) - }); - if let Ok(mut source) = self.local.controller.lock() { - *source = sample; - } - } - - pub(super) fn controller_control(&mut self, mut request: NodeRequest, enabled: Option) { - let result = (|| { - let controller = self - .controller - .as_mut() - .ok_or(CommandFailure::UnsupportedCapability)?; - if let Some(enabled) = enabled { - if enabled != controller.status.enabled { - let now = self - .log - .domain() - .mono_ms() - .map_err(|_| CommandFailure::Unavailable)?; - let epoch = controller - .epoch - .checked_add(1) - .ok_or(CommandFailure::Unavailable)?; - controller.stop.send_replace(true); - let (stop, _) = watch::channel(false); - controller.stop = stop; - controller.epoch = epoch; - controller.status.enabled = enabled; - controller.status.recovering = true; - controller.inventory = 0; - controller.active.clear(); - controller.last_transfer = now; - controller.next_at = now.saturating_add(controller.options.interval_ms); - } - } - Ok(Response::Controller(controller.status.clone())) - })(); - reply_node(&mut request, result); - } - - pub(super) fn pump_controller(&mut self) { - if self.local.admission.application_paused() { - if let Some(controller) = &self.controller { - controller.stop(); - } - return; - } - let Ok(now) = self.log.domain().mono_ms() else { - return; - }; - let Some(controller) = self.controller.as_mut() else { - return; - }; - if !controller.status.enabled || controller.busy.is_some() || now < controller.next_at { - return; - } - if controller.started.is_none() { - controller.started = Some(now); - controller.last_transfer = now; - } - // Reconstruct existing durable plans in bounded batches before any new - // automatic plan. Directory state cannot supply or clear this intent. - if controller.status.recovering { - let end = (controller.inventory + 4).min(controller.histories.len()); - for index in controller.inventory..end { - let scope = &controller.histories[index].binding.scope; - let group = match self.log.group(scope) { - Ok(group) => group, - Err(super::super::storage::LogError::NotFound) => continue, - Err(_) => { - controller.next_at = now.saturating_add(controller.options.interval_ms); - return; - } - }; - let active = match group - .membership() - .and_then(|membership| membership.repair_status()) - { - Ok(status) => status.is_some_and(|status| status.phase.active()), - Err(_) => { - controller.next_at = now.saturating_add(controller.options.interval_ms); - return; - } - }; - if active { - controller.active.insert(index); - } - } - controller.inventory = end; - if end != controller.histories.len() { - return; - } - controller.status.recovering = false; - if let Some((index, _)) = controller.intent.as_ref() { - controller.active.insert(*index); - } - } - let Some(index) = controller.next_history() else { - return; - }; - let options = controller.options.clone(); - let Some(memory) = controller - .memory - .acquire(options.probes_per_group * PROBE_MEMORY) - else { - return; - }; - let Some(sequence) = controller.next_sequence() else { - return; - }; - let epoch = controller.epoch; - let stop = controller.stop.subscribe(); - let Some(history) = controller.history(index, now) else { - return; - }; - let binding = history.binding.clone(); - let cursor = if binding.member().is_some() { - &mut history.candidate_cursor - } else { - &mut history.pull_cursor - }; - let setup = probe::targets( - &binding.scope, - &self.cells, - &self.peers, - self.log.limits().node, - *cursor, - options.probes_per_group, - ); - let (targets, next) = match setup { - Ok(setup) => setup, - Err(_) => { - controller.next_at = now.saturating_add(options.interval_ms); - return; - } - }; - *cursor = next; - controller.busy = Some(sequence); - controller.next_at = now.saturating_add(options.interval_ms); - let Some(count) = controller.status.probe_rounds.checked_add(1) else { - controller.status.enabled = false; - controller.stop.send_replace(true); - return; - }; - controller.status.probe_rounds = count; - let work = probe::Work { - local: RaftCellHandle { - handle: self.local.clone(), - binding, - }, - transport: self.transport.clone(), - runtime: self.runtime.clone(), - domain: self.log.domain(), - options, - targets, - memory, - }; - let performance = self.local.performance.clone(); - self.jobs.push(Box::pin(async move { - let timer = performance.timer(crate::raft::performance::Operation::ControllerProbe); - let result = action::until_stopped(stop, work.run()).await; - timer.finish(result.is_ok()); - Job::Controller(Box::new(Completed::Observed { - sequence, - epoch, - history: index, - result: result.map(Box::new), - })) - })); - } - - pub(super) fn finish_controller(&mut self, completed: Completed) { - let (sequence, epoch, index) = match &completed { - Completed::Observed { - sequence, - epoch, - history, - .. - } - | Completed::Acted { - sequence, - epoch, - history, - .. - } => (*sequence, *epoch, *history), - }; - let Some(controller) = self.controller.as_mut() else { - return; - }; - if controller.busy != Some(sequence) { - return; - } - controller.busy = None; - if !controller.status.enabled - || controller.epoch != epoch - || self.closing.load(Ordering::Acquire) - || self.local.admission.application_paused() - { - return; - } - let observed = match completed { - Completed::Observed { - result: Ok(observed), - .. - } => observed, - Completed::Acted { - result: Err(error), .. - } => { - tracing::debug!(%error, "automatic operation needs committed-state reconciliation"); - return; - } - _ => return, - }; - let Ok(now) = self.log.domain().mono_ms() else { - return; - }; - let context = policy::Context { - cells: &self.cells, - peers: &self.peers, - local: self.log.limits().node, - rng: &mut self.rng, - }; - let decision = policy::decide(controller, index, *observed, now, context); - if let Ok(Some(action)) = decision { - self.start_controller_action(index, action, now); - } - } -} diff --git a/crates/cellar/durability/raft/host_directory.rs b/crates/cellar/durability/raft/host_directory.rs index 28c140f40..d9f65a6fb 100644 --- a/crates/cellar/durability/raft/host_directory.rs +++ b/crates/cellar/durability/raft/host_directory.rs @@ -1,27 +1,6 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. -//! Owned exact-key discovery. Directory observations refresh endpoints only; -//! existing quorum authority survives missing metadata or an object-store outage. -use super::super::fleet_catalog::{FleetHostAuthorization, FleetTrustReader, FleetTrustSnapshot}; -use super::*; -use std::time::Duration; - -const FLEET_READ_BYTES: usize = 4 * 1024 * 1024; - -#[derive(Clone)] -enum DiscoveryNode { - Configured(RaftPeerNode), - Enrolled(Box), -} - -impl DiscoveryNode { - fn node(&self) -> NodeId { - match self { - Self::Configured(peer) => peer.node, - Self::Enrolled(node) => node.node(), - } - } -} +//! Explicit bounds for the physical-node directory owner. #[derive(Clone, Debug)] pub struct RaftNodeDirectoryOptions { @@ -54,421 +33,3 @@ impl RaftNodeDirectoryOptions { Ok(()) } } - -pub(super) struct Directory { - options: RaftNodeDirectoryOptions, - registrations: Vec, - trusted: Vec, - enrollment: Option, - rooted: Option, - namespace: usize, - node: usize, - next_at: u64, - busy: bool, - own_revision: u64, -} - -fn remaining(domain: &crate::asyncrt::DomainToken, deadline: u64) -> anyhow::Result { - let millis = deadline - .checked_sub(domain.mono_ms()?) - .filter(|ms| *ms > 0) - .ok_or_else(|| anyhow::anyhow!("node directory original deadline"))?; - Ok(Duration::from_millis(millis)) -} - -pub(super) async fn prepare( - options: Option<&RaftNodeDirectoryOptions>, - cells: &[Registration], - declared: &[NamespaceRegistration], - mut enrollment: Option<&mut super::super::fleet_catalog::FleetHostAuthorization>, - peers: &mut BTreeMap, - service: &crate::daemon::NodeRaftService, - label: &str, -) -> anyhow::Result> { - let Some(options) = options else { - return Ok(None); - }; - options.validate()?; - anyhow::ensure!( - !declared.is_empty() - || cells - .iter() - .any(|cell| !matches!(cell.placement, RaftCellPlacement::QueryRouter { .. })), - "node directory requires stable replica or standby storage" - ); - let local = service.log.limits(); - let own = peers - .get(&local.node) - .ok_or_else(|| anyhow::anyhow!("node directory requires an explicit self endpoint"))? - .clone(); - anyhow::ensure!( - own.node == local.node - && own.disk == local.disk - && own.process == local.process - && own.label == label, - "node directory self endpoint disagrees with engine owner" - ); - for peer in peers.values() { - peer.validate_endpoint()?; - } - let mut namespaces = BTreeMap::new(); - for (namespace, class) in cells - .iter() - .map(|c| (&c.namespace, &c.class)) - .chain(declared.iter().map(|c| (&c.namespace, &c.class))) - { - namespace.require_mode(cellar_logic::NamespaceMode::Raft)?; - anyhow::ensure!( - namespace.config().schema_digest == class.schema_digest(), - "registered schema disagrees with namespace" - ); - namespaces - .entry(namespace.config().cluster) - .or_insert_with(|| namespace.clone()); - } - let domain = service.log.domain(); - let deadline = domain.mono_ms()?.saturating_add(options.timeout_ms); - let boot = if enrollment.is_some() { - let mut id = [0; 16]; - crate::asyncrt::rng("fleet-host-boot").fill_bytes(&mut id); - Some(cellar_logic::consensus::ProposalId::new(id)?) - } else { - None - }; - let mut registrations = Vec::with_capacity(namespaces.len()); - let mut own_revision = 0; - let mut rooted = None; - for namespace in namespaces.into_values() { - let registration = namespace - .register_node_storage(&service.log, remaining(&domain, deadline)?) - .await?; - if let Some(authorization) = enrollment.as_mut() { - let boot = boot.ok_or_else(|| anyhow::anyhow!("missing managed boot identity"))?; - let join_key = crate::asyncrt::services() - .peer_probe_signer()? - .public_key_bytes(); - let (bound, update) = if authorization.state() - == crate::raft::fleet_catalog::FleetNodeState::RestartPending - { - authorization - .bind_restart_boot( - &service.log, - boot, - join_key, - deadline, - remaining(&domain, deadline)?, - ) - .await? - } else { - // Preserve the original ordinary-boot poll path. Polling a - // combined maintenance wrapper adds stack even when boxed. - let bound = authorization - .bind_boot( - service.log.storage_instance()?, - local.process, - boot, - join_key, - remaining(&domain, deadline)?, - ) - .await?; - let update = crate::raft::fleet_stop::LifecycleUpdate::Boot { - boot: bound.boot_binding()?, - deadline, - }; - (bound, update) - }; - **authorization = bound; - crate::asyncrt::timeout( - remaining(&domain, deadline)?, - service.log.update_lifecycle(update), - ) - .await??; - if let Some(selector) = service - .log - .lifecycle_guard()? - .and_then(|g| g.quiescing().cloned()) - { - crate::asyncrt::services() - .raft_lifecycle() - .install_startup_maintenance(&selector)?; - } - let trust = authorization - .trust_snapshot(remaining(&domain, deadline)?) - .await?; - service.transport.install_fleet_trust(trust.clone())?; - anyhow::ensure!( - trust.nodes().len() <= service.transport.peer_limit(), - "fleet exceeds transport peer bound" - ); - rooted = Some(trust); - } - let publication = registration - .publication(own.clone(), remaining(&domain, deadline)?) - .await?; - let view = publication.publish(remaining(&domain, deadline)?).await?; - own_revision = own_revision.max(view.revision()); - service.transport.accept_node_view(view, &own)?; - // Cold managed members have only a self hint. Resolve exact rooted - // process views before constructing their current membership routes; - // periodic discovery then refreshes restarts without static addresses. - if let Some(trust) = &rooted { - for node in trust - .nodes() - .iter() - .filter(|node| node.node() != local.node) - { - let result = registration - .observe_fleet_node(node, remaining(&domain, deadline)?) - .await; - match result { - Ok(Some(view)) => { - let peer = view.peer().clone(); - service.transport.accept_node_view(view, &peer)?; - peers.insert(peer.node, peer); - } - Ok(None) => (), - Err(error) => tracing::debug!(%error, "startup node view unavailable"), - } - } - } - registrations.push(registration); - } - Ok(Some(Directory { - options: options.clone(), - registrations, - trusted: rooted.as_ref().map_or_else( - || { - peers - .values() - .cloned() - .map(DiscoveryNode::Configured) - .collect() - }, - trusted_nodes, - ), - enrollment: enrollment - .as_deref() - .map(FleetHostAuthorization::trust_reader), - rooted, - namespace: 0, - node: 0, - next_at: domain.mono_ms()?, - busy: false, - own_revision, - })) -} - -enum DiscoveryError { - Catalog(CatalogError), - Transport(super::super::transport::TransportError), -} - -struct PeerObservation { - trusted: DiscoveryNode, - result: Result, DiscoveryError>, -} - -pub(super) struct Observation { - peers: Vec, - trust: Option>, - _metadata: Option, -} - -fn trusted_nodes(trust: &FleetTrustSnapshot) -> Vec { - trust - .nodes() - .iter() - .filter(|node| node.permits_transport()) - .cloned() - .map(|node| DiscoveryNode::Enrolled(Box::new(node))) - .collect() -} - -impl Directory { - pub(super) fn authorization(&self, node: NodeId) -> Option<&FleetHostAuthorization> { - self.rooted.as_ref()?.node(node) - } - - pub(super) fn rooted_revision(&self) -> Option { - self.rooted.as_ref().map(FleetTrustSnapshot::revision) - } -} - -impl Driver { - pub(super) fn pump_directory(&mut self) { - let Some(directory) = self.directory.as_mut() else { - return; - }; - let Ok(now) = self.log.domain().mono_ms() else { - return; - }; - if directory.busy || now < directory.next_at { - return; - } - let metadata = if directory.enrollment.is_some() { - let Some(permit) = self.control_budget.acquire(FLEET_READ_BYTES) else { - directory.next_at = now.saturating_add(directory.options.poll_ms); - return; - }; - Some(permit) - } else { - None - }; - directory.busy = true; - directory.next_at = now.saturating_add(directory.options.poll_ms); - let deadline = now.saturating_add(directory.options.timeout_ms); - let count = directory.options.batch_nodes.min( - directory - .registrations - .len() - .saturating_mul(directory.trusted.len()), - ); - let mut batch = Vec::with_capacity(count); - for _ in 0..count { - batch.push(( - directory.registrations[directory.namespace].clone(), - directory.trusted[directory.node].clone(), - )); - directory.node += 1; - if directory.node == directory.trusted.len() { - directory.node = 0; - directory.namespace = (directory.namespace + 1) % directory.registrations.len(); - } - } - let transport = self.transport.clone(); - let domain = self.log.domain(); - let enrollment = directory.enrollment.clone(); - let previous = directory.rooted.clone(); - self.jobs.push(Box::pin(async move { - let trust = if let Some(enrollment) = enrollment { - Some( - transport - .seed_control_until(deadline, async { - let timeout = remaining(&domain, deadline) - .map_err(|_| super::super::transport::TransportError::Timeout)?; - Ok(enrollment.refresh(previous.as_ref(), timeout).await) - }) - .await - .map_err(|error| error.to_string()) - .and_then(|result| result.map_err(|error| error.to_string())), - ) - } else { - None - }; - let observations = - futures_util::future::join_all(batch.into_iter().map(|(registration, trusted)| { - let transport = transport.clone(); - let domain = domain.clone(); - async move { - let result = transport - .seed_control_until(deadline, async { - let timeout = remaining(&domain, deadline).map_err(|_| { - super::super::transport::TransportError::Timeout - })?; - Ok(match &trusted { - DiscoveryNode::Configured(peer) => { - registration.observe_node(peer, timeout).await - } - DiscoveryNode::Enrolled(node) => { - registration.observe_fleet_node(node, timeout).await - } - }) - }) - .await - .map_err(DiscoveryError::Transport) - .and_then(|result| result.map_err(DiscoveryError::Catalog)); - PeerObservation { trusted, result } - } - })) - .await; - Job::Directory(Box::new(Observation { - peers: observations, - trust, - _metadata: metadata, - })) - })); - } - - pub(super) fn on_directory(&mut self, observation: Observation) { - if let Some(directory) = self.directory.as_mut() { - directory.busy = false; - } - if let Some(result) = observation.trust { - match result { - Ok(trust) => { - let local = self.log.limits(); - if trust - .node(local.node) - .is_none_or(|node| !node.permits_transport()) - { - self.closing.store(true, Ordering::Release); - return; - } - match self.transport.install_fleet_trust(trust.clone()) { - Ok(()) => { - if let Some(directory) = &mut self.directory { - if directory.rooted_revision() != Some(trust.revision()) { - directory.rooted = Some(trust.clone()); - directory.trusted = trusted_nodes(&trust); - directory.node = directory - .node - .min(directory.trusted.len().saturating_sub(1)); - } - } - self.peers.retain(|node, peer| { - trust - .node(*node) - .is_some_and(|entry| entry.matches_peer(peer)) - }); - } - Err(error) => tracing::debug!(%error, "Raft fleet trust cut refused"), - } - } - Err(error) => tracing::debug!(%error, "Raft fleet trust refresh unavailable"), - } - } - for observation in observation.peers { - let view = match observation.result { - Ok(Some(view)) => view, - // No observation supplies authority or invalidates a live - // quorum. Keep the last accepted routes and retry boundedly. - Ok(None) => continue, - Err(DiscoveryError::Catalog(error)) => { - tracing::debug!(node = ?observation.trusted.node(), %error, "Raft node directory observation refused"); - continue; - } - Err(DiscoveryError::Transport(error)) => { - tracing::debug!(node = ?observation.trusted.node(), %error, "Raft node directory observation cancelled"); - continue; - } - }; - let peer = view.peer().clone(); - let local = self.log.limits(); - let newer_boot = self.directory.as_ref().is_some_and(|directory| { - peer.process != local.process && view.revision() > directory.own_revision - }); - if peer.node == local.node - && (newer_boot || self.log.storage_instance().ok() != Some(view.storage_instance())) - { - // The shared closing flag immediately fences retained handles; - // the host loop then joins every owned service and physical job. - self.closing.store(true, Ordering::Release); - return; - } - let trusted = match &observation.trusted { - DiscoveryNode::Configured(peer) => peer, - // The transport's installed rooted cut independently checks - // registration identity and storage pin before accepting this. - DiscoveryNode::Enrolled(_) => &peer, - }; - match self.transport.accept_node_view(view, trusted) { - Ok(()) => { - self.peers.insert(peer.node, peer); - } - Err(error) => { - tracing::debug!(node = ?peer.node, %error, "Raft node process refresh refused") - } - } - } - } -} diff --git a/crates/cellar/durability/raft/host_effects.rs b/crates/cellar/durability/raft/host_effects.rs index 0bfcec3a0..8532015e2 100644 --- a/crates/cellar/durability/raft/host_effects.rs +++ b/crates/cellar/durability/raft/host_effects.rs @@ -295,7 +295,7 @@ impl Driver { let mut options = client::CommandOptions::new(scan.next, retry_until); options.durability = due.minimum(); options.timeout = std::time::Duration::from_millis(owner.options.operation_timeout_ms); - let local = crate::application::consensus::RaftCellHandle { + let local = crate::application::consensus::client::RaftCellHandle { handle: self.local.clone(), binding, }; @@ -342,7 +342,8 @@ impl Driver { sink, }) = super::super::effects::outbox::claimed( &definition, - &scope, + &cellar_logic::cell_group::CellHistory::try_from(&scope) + .map_err(|_| CommandFailure::CorruptState)?, local.binding.class.schema_digest(), &selected, &receipt.result, diff --git a/crates/cellar/durability/raft/host_enrollment.rs b/crates/cellar/durability/raft/host_enrollment.rs index 1336489f6..2051329d8 100644 --- a/crates/cellar/durability/raft/host_enrollment.rs +++ b/crates/cellar/durability/raft/host_enrollment.rs @@ -84,86 +84,3 @@ pub(crate) struct NamespaceRegistration { pub namespace: NamespaceCatalog, pub class: Arc, } - -pub(super) async fn prepare( - options: &mut RaftHostOptions, - service: &crate::daemon::NodeRaftService, - cells: &[Registration], - namespaces: &[NamespaceRegistration], - peers: &BTreeMap, - label: &str, -) -> anyhow::Result> { - let Some(managed) = &mut options.managed else { - anyhow::ensure!(namespaces.is_empty(), "undeclared managed namespaces"); - return Ok(BTreeMap::new()); - }; - managed.authorization = managed - .authorization - .refresh(std::time::Duration::from_secs(5)) - .await?; - managed.validate()?; - anyhow::ensure!( - namespaces.len() == managed.namespaces.len(), - "incomplete managed namespace preparation" - ); - for (prepared, declared) in namespaces.iter().zip(&managed.namespaces) { - anyhow::ensure!( - prepared.namespace.config() == &declared.namespace - && prepared.class.name() == declared.class - && prepared.class.schema_digest() == declared.namespace.schema_digest, - "managed namespace preparation mismatch" - ); - } - let auth = &managed.authorization; - let local = service.log.limits(); - let own = peers - .get(&local.node) - .ok_or_else(|| anyhow::anyhow!("managed host self endpoint missing"))?; - anyhow::ensure!( - auth.node() == local.node - && auth.disk() == local.disk - && auth.label() == label - && own.label == auth.label() - && own.server_name == auth.server_name() - && auth.instance_matches(service.log.storage_instance()?), - "managed host physical identity mismatch" - ); - managed - .authorization - .check_declared_namespaces(namespaces, std::time::Duration::from_secs(5)) - .await?; - - let mut authorizations = BTreeMap::new(); - for cell in cells { - let scope = cell.birth.scope(); - let authorization = auth - .group_authorization(scope, std::time::Duration::from_secs(5)) - .await?; - anyhow::ensure!( - authorization.check_birth(&cell.namespace, &cell.birth), - "managed history birth/schema mismatch" - ); - match service.log.group(scope) { - Ok(group) => anyhow::ensure!( - group - .membership()? - .fleet_status() - .is_some_and(|status| status.binding == *authorization.binding()), - "managed host requires committed group enrollment" - ), - Err(super::super::storage::LogError::NotFound) => anyhow::ensure!( - cell.placement == RaftCellPlacement::Replica - && cell.birth.managed_owner().is_some() - && authorization.allows_managed_genesis() - || matches!( - cell.placement, - RaftCellPlacement::Standby | RaftCellPlacement::QueryRouter { .. } - ), - "managed host cannot bootstrap an unguarded member" - ), - Err(error) => return Err(error.into()), - } - authorizations.insert(scope.clone(), authorization); - } - Ok(authorizations) -} diff --git a/crates/cellar/durability/raft/host_extents.rs b/crates/cellar/durability/raft/host_extents.rs index 57c42447f..ee2ba5201 100644 --- a/crates/cellar/durability/raft/host_extents.rs +++ b/crates/cellar/durability/raft/host_extents.rs @@ -4,11 +4,7 @@ //! The node supplies physical policy; immutable namespaces select commitments. use super::*; -use cellar_ltx::{ - internal::{ExtentRootReference, NodeExtentStore, EXTENT_BYTES}, - stream_snapshot::{ScratchLimits, ScratchPool}, -}; -use std::path::Path; +use cellar_ltx::{internal::EXTENT_BYTES, stream_snapshot::ScratchLimits}; pub use cellar_ltx::internal::{ CacheLimits as RaftExtentCacheLimits, NodeExtentLimits as RaftExtentLimits, @@ -66,139 +62,3 @@ impl RaftExtentOptions { Ok(()) } } - -pub(super) struct Owner { - pub store: NodeExtentStore, - cache: ScratchPool, - options: RaftExtentOptions, - next_gc: u64, - busy: bool, - failed: bool, -} - -impl Owner { - /// Startup is still parked. Rebuild retention from every durable engine - /// group, including unregistered/closed groups, before allowing any GC. - pub async fn open( - log: &super::super::storage::ReplicaLog, - directory: PathBuf, - host: cellar_ltx::LtxHost, - options: Option, - needed: bool, - ) -> anyhow::Result> { - if !needed && options.is_none() { - return Ok(None); - } - let options = options - .ok_or_else(|| anyhow::anyhow!("extent namespace requires node extent storage"))?; - let groups = log.groups()?; - let owner = log - .domain() - .blocking(move || { - let store = NodeExtentStore::open_serialized( - directory.join("extent-inventory"), - host.filesystem(), - options.inventory, - )?; - let mut retained = BTreeMap::<[u8; 32], ExtentRootReference>::new(); - for group in groups { - let applied = group.applied()?; - let snapshot_index = group.read()?.snapshot.get_metadata().index; - for (role, revision, image) in [ - (1, applied.position.index.get(), applied.image), - ( - 2, - snapshot_index, - super::super::snapshot::retained_image(&group)?, - ), - ] { - let Some(image) = image else { - continue; - }; - image.validate()?; - let crate::application::consensus::ImageCommitment::Extents(reference) = - image.commitment - else { - continue; - }; - store.retain_reference( - super::super::rooted::marker::reference_key(&group, role), - revision, - reference, - )?; - retained.insert(reference.root().bytes(), reference); - } - } - store.retain_only(&retained.into_values().collect::>())?; - store.reclaim(options.gc_batch_files)?; - Ok::<_, anyhow::Error>(Self { - store, - cache: ScratchPool::new(options.cache_scratch)?, - options, - next_gc: 0, - busy: false, - failed: false, - }) - })? - .await??; - Ok(Some(owner)) - } - - pub fn cell_storage(&self, directory: &Path) -> sql::RootStorage { - sql::RootStorage { - store: self.store.clone(), - cache_pool: self.cache.clone(), - cache: self.options.cache, - cache_directory: directory.join("extent-cache"), - } - } -} - -impl Driver { - pub(super) fn pump_extent_gc(&mut self) { - let Some(owner) = &mut self.extents else { - return; - }; - let now = crate::asyncrt::mono_ms(); - if owner.busy || owner.failed { - return; - } - let pressure = owner.store.usage().is_ok_and(|usage| { - usage.root_extents >= owner.options.inventory.max_root_extents.saturating_mul(3) / 4 - || usage.roots >= owner.options.inventory.max_roots.saturating_mul(3) / 4 - || usage.bytes >= owner.options.inventory.max_bytes.saturating_mul(3) / 4 - }); - if now < owner.next_gc && !pressure { - return; - } - owner.busy = true; - owner.next_gc = now.saturating_add(owner.options.gc_interval_ms); - let store = owner.store.clone(); - let limit = owner.options.gc_batch_files; - let job = self.log.domain().blocking(move || store.reclaim(limit)); - self.jobs.push(Box::pin(async move { - let result = match job { - Ok(job) => job - .await - .map_err(anyhow::Error::from) - .and_then(|value| value.map_err(anyhow::Error::from)), - Err(error) => Err(error.into()), - }; - Job::ExtentGc(result) - })); - } - - pub(super) fn extent_gc_completed(&mut self, result: anyhow::Result) { - if let Some(owner) = &mut self.extents { - owner.busy = false; - match result { - Ok(removed) if removed == owner.options.gc_batch_files => owner.next_gc = 0, - Ok(_) => (), - Err(error) => { - owner.failed = true; - tracing::error!(event = "raft_extent_gc_failed", %error); - } - } - } - } -} diff --git a/crates/cellar/durability/raft/host_fleet_cut.rs b/crates/cellar/durability/raft/host_fleet_cut.rs index cf3a876b6..a8920125e 100644 --- a/crates/cellar/durability/raft/host_fleet_cut.rs +++ b/crates/cellar/durability/raft/host_fleet_cut.rs @@ -1,5 +1,5 @@ -//! Fixed publication cuts pair the durable configuration with its SQL image. -//! A data position can precede configuration-only applied progress. +//! Fixed complete-group roots pair durable membership with all cell state. +//! Authenticated mutation summaries distinguish data from protocol-only apply. use super::*; use crate::application::consensus::ImageCommitment; use crate::raft::cache::{CommittedImage, DurableApply}; @@ -287,34 +287,42 @@ fn marker_state(image: &CommittedImage) -> Result, storage_instance: EngineInstanceId, - birth: [u8; 32], - schema: [u8; 32], - revision: LogIndex, - applied: LogPosition, - image: Option, + root: crate::raft::cell_groups::GroupManifest, + configuration_digest: [u8; 32], } impl FleetPublishedCut { - /// Re-establish a stored historical prefix from an actual current quorum - /// publication. Serialized receipt material alone cannot construct a cut. + pub(crate) fn capture_group( + membership: &FleetMembershipObservation, + storage_instance: EngineInstanceId, + ) -> Self { + Self { + binding: membership.status().binding.clone(), + replica: membership.replica().clone(), + storage_instance, + root: membership.root().clone(), + configuration_digest: membership.configuration_digest(), + } + } + + /// Stored journal data is historical material, never an independently minted + /// proof. Re-establish its prefix through an actual current quorum cut. pub(crate) fn checked_prefix( &self, - applied: LogPosition, - revision: LogIndex, - image: Option, + root: crate::raft::cell_groups::GroupManifest, + configuration_digest: [u8; 32], ) -> Result { + root.check().map_err(|_| CommandFailure::CorruptState)?; let prefix = Self { - applied, - revision, - image, + root, + configuration_digest, ..self.clone() }; if !self.covers(&prefix)? { @@ -323,33 +331,11 @@ impl FleetPublishedCut { Ok(prefix) } - pub(crate) fn capture( - membership: &FleetMembershipObservation, - storage_instance: EngineInstanceId, - publication: &Publication, - ) -> Option { - let image = match publication { - Publication::Pending => return None, - Publication::Empty => None, - Publication::Image(image) => Some(image.clone()), - }; - Some(Self { - binding: membership.status().binding.clone(), - replica: membership.replica().clone(), - storage_instance, - birth: membership.birth_digest(), - schema: membership.schema_digest(), - revision: membership.membership_revision(), - applied: membership.applied(), - image, - }) - } - pub fn binding(&self) -> &FleetGroupBinding { &self.binding } - pub fn replica(&self) -> &ReplicaIdentity { + pub fn replica(&self) -> &ReplicaIdentity { &self.replica } @@ -358,83 +344,80 @@ impl FleetPublishedCut { } pub fn birth_digest(&self) -> [u8; 32] { - self.birth + self.root.route().birth_digest } pub fn schema_digest(&self) -> [u8; 32] { - self.schema + self.root.route().schema_digest } pub fn membership_revision(&self) -> LogIndex { - self.revision + LogIndex::new(self.root.membership_revision()) } - /// Includes configuration-only progress even when the SQL data position - /// has not advanced. The image state was checked against this durable cut. pub fn applied(&self) -> LogPosition { - self.applied + self.root.position() } - /// None is proven canonical empty state, not missing publication metadata. - pub fn image(&self) -> Option<&FleetPublishedImage> { - self.image.as_ref() + /// The complete group commitment includes the route, every cell's image, + /// enrollment/receipt summary and recovery lineage. Empty is an explicit + /// zero-cell manifest; missing/pending publication cannot construct a cut. + pub fn root(&self) -> &crate::raft::cell_groups::GroupManifest { + &self.root + } + + pub fn configuration_digest(&self) -> [u8; 32] { + self.configuration_digest } - /// Compare committed prefix coverage at the same exact configuration. - /// Physical replicas may differ. Configuration changes require a fresh - /// verification attempt; inconsistent same-position images are corruption. pub fn covers(&self, minimum: &Self) -> Result { if self.binding != minimum.binding - || self.birth != minimum.birth - || self.schema != minimum.schema - || self.revision != minimum.revision + || self.membership_revision() != minimum.membership_revision() + || self.configuration_digest != minimum.configuration_digest + || self.root.route() != minimum.root.route() { return Err(CommandFailure::CommandConflict); } - if self - .image - .as_ref() - .zip(minimum.image.as_ref()) - .is_some_and(|(a, b)| !a.state.same_representation(&b.state)) - { - return Err(CommandFailure::CorruptState); - } - if self.applied.index < minimum.applied.index { - return Ok(false); - } - if self.applied.term < minimum.applied.term - || self.applied.index == minimum.applied.index - && self.applied.term != minimum.applied.term - { + group_prefix(&self.root, &minimum.root) + } +} + +pub(crate) fn group_prefix( + current: &crate::raft::cell_groups::GroupManifest, + minimum: &crate::raft::cell_groups::GroupManifest, +) -> Result { + current + .check() + .and_then(|()| minimum.check()) + .map_err(|_| CommandFailure::CorruptState)?; + if current.route() != minimum.route() + || current.membership_revision() != minimum.membership_revision() + { + return Err(CommandFailure::CommandConflict); + } + let (a, b) = (current.position(), minimum.position()); + if a.index < b.index { + return Ok(false); + } + if a.term < b.term + || a.index == b.index && current != minimum + || !current.preserves_recovery(minimum) + || !current.preserves_transfer(minimum) + || !current.preserves_source_seal(minimum) + || !current.preserves_park_cut(minimum) + { + return Err(CommandFailure::CorruptState); + } + if current.root() != minimum.root() { + let mutation = current + .root() + .last_mutation() + .ok_or(CommandFailure::CorruptState)?; + if mutation.index <= b.index || mutation.term < b.term { return Err(CommandFailure::CorruptState); } - match (&self.image, &minimum.image) { - (None, None) => Ok(true), - (None, Some(_)) => Err(CommandFailure::CorruptState), - (Some(current), prior) => { - if !consistent_positions(current.position, minimum.applied) - || prior.as_ref().is_some_and(|prior| { - !consistent_positions(current.position, prior.position) - }) - { - return Err(CommandFailure::CorruptState); - } - // Live and restored replicas can publish exactly the same - // state at different protocol-only positions. Changed state - // must instead follow the entire minimum applied prefix. - if prior - .as_ref() - .is_some_and(|prior| current.state == prior.state) - { - return Ok(true); - } - if current.position.index <= minimum.applied.index { - return Err(CommandFailure::CorruptState); - } - Ok(true) - } - } } + Ok(true) } fn consistent_positions(a: LogPosition, b: LogPosition) -> bool { diff --git a/crates/cellar/durability/raft/host_fleet_cut_tests.rs b/crates/cellar/durability/raft/host_fleet_cut_tests.rs index b3539fb32..ae09a52ea 100644 --- a/crates/cellar/durability/raft/host_fleet_cut_tests.rs +++ b/crates/cellar/durability/raft/host_fleet_cut_tests.rs @@ -1,3 +1,5 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + use super::*; fn position(index: u64, term: u64) -> LogPosition { @@ -19,45 +21,52 @@ fn marker() -> CommittedImage { } } -fn image(index: u64, term: u64, digest: u8) -> FleetPublishedImage { - FleetPublishedImage::new( - position(index, term), - ImageState { - epoch: 1, - txid: TXID(index), - checksum: 0, - page_size: 4096, - length: 4096, - digest: [digest; 32], +fn root( + index: u64, + term: u64, + mutation: u64, + digest: u8, +) -> crate::raft::cell_groups::GroupManifest { + use cellar_logic::cell_group::*; + let route = GroupRoute { + history: GroupHistory { + cluster: ClusterId::new([1; 16]).unwrap(), + namespace: NamespaceId::new([2; 16]).unwrap(), + group: GroupId::new([3; 16]).unwrap(), + incarnation: IncarnationId::new([4; 16]).unwrap(), }, - ) - .unwrap() + range: CellRange::all(), + epoch: RouteEpoch::new(1).unwrap(), + birth_digest: [8; 32], + schema_digest: [9; 32], + }; + let mut value = + serde_json::to_value(crate::raft::cell_groups::GroupManifest::genesis(route).unwrap()) + .unwrap(); + value["index"] = serde_json::json!(index); + value["term"] = serde_json::json!(term); + if mutation > 0 { + value["root"] = serde_json::json!({"digest":vec![digest;32],"height":2,"cells":2,"last_mutation_index":mutation,"last_mutation_term":1,"next_due_unix_millis":null}); + } + serde_json::from_value(value).unwrap() } -fn cut(node: u8, index: u64, term: u64, image: Option) -> FleetPublishedCut { - let scope = GroupScope { - cluster: ClusterId::new([1; 16]).unwrap(), - namespace: NamespaceId::new([2; 16]).unwrap(), - cell: "Counter:cut".into(), - group: GroupId::new([3; 16]).unwrap(), - incarnation: IncarnationId::new([4; 16]).unwrap(), - }; +fn cut(node: u8, index: u64, term: u64, mutation: u64, digest: u8) -> FleetPublishedCut { + let root = root(index, term, mutation, digest); + let history = root.route().history; FleetPublishedCut { - binding: FleetGroupBinding::new(IncarnationId::new([5; 16]).unwrap(), &scope, [6; 32]) + binding: FleetGroupBinding::new(IncarnationId::new([5; 16]).unwrap(), &history, [6; 32]) .unwrap(), replica: ReplicaIdentity { - scope, + scope: history, member: MemberId::new(node as u64).unwrap(), node: NodeId::new([node; 16]).unwrap(), disk: DiskLineage::new([node + 10; 16]).unwrap(), process: ProcessGeneration::new([node + 20; 16]).unwrap(), }, storage_instance: EngineInstanceId::new([node + 30; 16]).unwrap(), - birth: [8; 32], - schema: [9; 32], - revision: LogIndex::new(3), - applied: position(index, term), - image, + root, + configuration_digest: [10; 32], } } @@ -124,8 +133,8 @@ fn publication_preserves_extent_domain_and_refuses_materialized_substitution() { published.digest(), Err(CommandFailure::UnsupportedCapability) ); - let mut rooted_cut = cut(1, 5, 2, Some(published)); - let flat_cut = cut(2, 5, 2, Some(image(2, 1, 7))); + let rooted_cut = cut(1, 5, 2, 2, 7); + let flat_cut = cut(2, 5, 2, 2, 8); assert_eq!( rooted_cut.covers(&flat_cut), Err(CommandFailure::CorruptState) @@ -134,10 +143,6 @@ fn publication_preserves_extent_domain_and_refuses_materialized_substitution() { flat_cut.covers(&rooted_cut), Err(CommandFailure::CorruptState) ); - let mut later = rooted_cut.clone(); - later.applied = position(8, 3); - rooted_cut.image.as_mut().unwrap().position = position(4, 2); - assert!(later.covers(&rooted_cut).unwrap()); assert_eq!( Publication::capture(&durable, Some((&marker(), position(2, 1)))), Err(CommandFailure::CorruptState) @@ -146,70 +151,77 @@ fn publication_preserves_extent_domain_and_refuses_materialized_substitution() { #[test] fn fixed_publication_cut_covers_configuration_only_progress_and_later_writes() { - let minimum = cut(1, 5, 2, Some(image(2, 1, 7))); - let protocol_only = cut(2, 7, 3, Some(image(2, 1, 7))); - assert!(protocol_only.covers(&minimum).unwrap()); - assert!(!minimum.covers(&protocol_only).unwrap()); - let later = cut(3, 9, 3, Some(image(8, 3, 8))); + let minimum = cut(1, 5, 2, 2, 7); + let protocol = cut(2, 7, 3, 2, 7); + assert!(protocol.covers(&minimum).unwrap()); + assert!(!minimum.covers(&protocol).unwrap()); + let mut later = cut(3, 9, 3, 8, 8); + let mut value = serde_json::to_value(later.root()).unwrap(); + value["root"]["last_mutation_term"] = serde_json::json!(3); + later.root = serde_json::from_value(value).unwrap(); assert!(later.covers(&minimum).unwrap()); assert_eq!(minimum.applied(), position(5, 2)); - assert_eq!(minimum.image().unwrap().position(), position(2, 1)); - let mut restored = minimum.clone(); - restored.image.as_mut().unwrap().position = restored.applied; - assert!(protocol_only.covers(&restored).unwrap()); - assert!(restored.covers(&minimum).unwrap()); + assert_eq!(minimum.root().root().last_mutation(), Some(position(2, 1))); + assert_eq!(minimum.root().root().cells(), 2); } #[test] fn fixed_publication_cut_rejects_foreign_configuration_and_inconsistent_prefix() { - let minimum = cut(1, 5, 2, Some(image(2, 1, 7))); - let mut changed = cut(2, 7, 3, Some(image(2, 1, 7))); - changed.revision = LogIndex::new(6); - assert!(matches!( + let minimum = cut(1, 5, 2, 2, 7); + let mut changed = cut(2, 7, 3, 2, 7); + changed.configuration_digest = [99; 32]; + assert_eq!( changed.covers(&minimum), Err(CommandFailure::CommandConflict) - )); - changed.revision = minimum.revision; - changed.birth = [99; 32]; - assert!(matches!( + ); + let mut changed = minimum.clone(); + let mut value = serde_json::to_value(&changed.root).unwrap(); + value["birth"] = serde_json::json!(vec![99; 32]); + changed.root = serde_json::from_value(value).unwrap(); + assert_eq!( changed.covers(&minimum), Err(CommandFailure::CommandConflict) - )); - let mut foreign = minimum.clone(); - foreign.binding = FleetGroupBinding::new( + ); + changed = minimum.clone(); + changed.binding = FleetGroupBinding::new( IncarnationId::new([5; 16]).unwrap(), - &foreign.replica.scope, + &changed.replica.scope, [99; 32], ) .unwrap(); - assert!(matches!( - foreign.covers(&minimum), + assert_eq!( + changed.covers(&minimum), Err(CommandFailure::CommandConflict) - )); - for inconsistent in [ - cut(2, 5, 1, Some(image(2, 1, 7))), - cut(2, 7, 3, Some(image(2, 1, 99))), - cut(2, 7, 3, Some(image(4, 2, 8))), - ] { - assert!(matches!( + ); + for inconsistent in [cut(2, 5, 1, 2, 7), cut(2, 7, 3, 2, 99), cut(2, 7, 3, 4, 8)] { + assert_eq!( inconsistent.covers(&minimum), Err(CommandFailure::CorruptState) - )); + ); } + let mut changed = minimum.clone(); + let mut value = serde_json::to_value(&changed.root).unwrap(); + value["root"]["cells"] = serde_json::json!(3); + changed.root = serde_json::from_value(value).unwrap(); + assert_eq!(changed.covers(&minimum), Err(CommandFailure::CorruptState)); } #[test] fn canonical_empty_cut_is_distinct_from_missing_publication() { - let empty = cut(1, 5, 2, None); - assert!(cut(2, 7, 3, None).covers(&empty).unwrap()); - assert!(cut(2, 9, 3, Some(image(8, 3, 7))).covers(&empty).unwrap()); - assert!(matches!( - cut(2, 7, 3, Some(image(4, 2, 7))).covers(&empty), + let empty = cut(1, 5, 2, 0, 0); + assert_eq!(empty.root().root().cells(), 0); + assert!(cut(2, 7, 3, 0, 0).covers(&empty).unwrap()); + let mut later = cut(2, 9, 3, 8, 7); + let mut value = serde_json::to_value(&later.root).unwrap(); + value["root"]["last_mutation_term"] = serde_json::json!(3); + later.root = serde_json::from_value(value).unwrap(); + assert!(later.covers(&empty).unwrap()); + assert_eq!( + cut(2, 7, 3, 4, 7).covers(&empty), Err(CommandFailure::CorruptState) - )); - assert!(!empty.covers(&cut(2, 9, 3, Some(image(8, 3, 7)))).unwrap()); + ); + assert!(!empty.covers(&later).unwrap()); assert_eq!( - Publication::Empty, Publication::capture( &DurableApply { position: position(5, 2), @@ -217,7 +229,8 @@ fn canonical_empty_cut_is_distinct_from_missing_publication() { }, None ) - .unwrap() + .unwrap(), + Publication::Empty ); assert_ne!(Publication::Empty, Publication::Pending); } diff --git a/crates/cellar/durability/raft/host_fleet_execution.rs b/crates/cellar/durability/raft/host_fleet_execution.rs deleted file mode 100644 index e35789ffb..000000000 --- a/crates/cellar/durability/raft/host_fleet_execution.rs +++ /dev/null @@ -1,187 +0,0 @@ -// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. - -//! All nested work belongs to the host job set. Cancellation drops its waiter; -//! committed configuration and physical initialization retain their owners. -use super::*; -use crate::raft::fleet_catalog::FleetCatalogError; -use crate::raft::transport::fleet_actions::Delivery; -use std::time::Duration; - -pub(super) struct Completed { - request: Box, - result: Result<(), FleetCatalogError>, - _memory: Option, -} - -impl Driver { - pub(super) fn start_restart_service( - &mut self, - mut request: NodeRequest, - mut work: crate::raft::fleet_catalog::RestartServiceWork, - ) { - let result = self - .options - .managed - .as_ref() - .ok_or(FleetCatalogError::Invalid( - "no managed restart service origin", - )) - .and_then(|managed| work.admit(&managed.authorization)); - if result.is_ok() { - self.jobs.push(Box::pin(async move { - work.run().await; - Job::FleetRestartServiceStopped - })); - } - reply_node(&mut request, Ok(Response::FleetExecutionAction(result))); - } - - pub(super) fn start_movement_service( - &mut self, - mut request: NodeRequest, - mut work: crate::raft::fleet_catalog::MovementServiceWork, - ) { - let result = self - .options - .managed - .as_ref() - .ok_or(FleetCatalogError::Invalid( - "no managed movement service origin", - )) - .and_then(|managed| work.admit(&managed.authorization)); - if result.is_ok() { - self.jobs.push(Box::pin(async move { - work.run().await; - Job::FleetMovementServiceStopped - })); - } - reply_node(&mut request, Ok(Response::FleetExecutionAction(result))); - } - - pub(super) fn start_peer_fleet_action( - &mut self, - mut request: NodeRequest, - call: client::PeerFleetActionCall, - ) { - if call.command.target().node == self.log.limits().node.bytes() { - let origin = self - .log - .lifecycle_guard() - .ok() - .flatten() - .map(|guard| guard.boot); - if let Some(origin) = origin { - self.start_local_fleet_action( - request, - client::LocalFleetActionCall { - command: call.command, - minimum_revision: call.minimum_revision, - origin, - }, - ); - } else { - reply_node( - &mut request, - Ok(Response::FleetExecutionAction(Err( - FleetCatalogError::Closed, - ))), - ); - } - return; - } - let delivery = if self.options.managed.is_some() { - self.transport.request_fleet_action( - call.command, - call.minimum_revision, - request.control.deadline(), - ) - } else { - Err(FleetCatalogError::Invalid("no managed movement origin")) - }; - let delivery: Delivery = match delivery { - Ok(delivery) => delivery, - Err(error) => { - reply_node(&mut request, Ok(Response::FleetExecutionAction(Err(error)))); - return; - } - }; - self.jobs.push(Box::pin(async move { - let result = crate::asyncrt::select_biased! { "cancel movement origin without assuming rollback"; - _ = request.control.cancellation() => Err(FleetCatalogError::Unconfirmed), - result = delivery.receive() => result, - }; - Job::FleetAction(Box::new(Completed { request: Box::new(request), result, _memory: None })) - })); - } - - pub(super) fn start_local_fleet_action( - &mut self, - mut request: NodeRequest, - call: client::LocalFleetActionCall, - ) { - let setup = (|| { - call.command.validate()?; - let host = self - .options - .managed - .as_ref() - .ok_or(FleetCatalogError::Invalid("no managed receiver"))? - .authorization - .clone(); - let boot = self - .log - .lifecycle_guard() - .map_err(|_| FleetCatalogError::Closed)? - .ok_or(FleetCatalogError::Closed)? - .boot; - if &boot != call.command.target() { - return Err(FleetCatalogError::Conflict); - } - let memory = self - .control_budget - .acquire(128 * 1024) - .ok_or(FleetCatalogError::Overloaded)?; - Ok((host, memory)) - })(); - let (host, memory) = match setup { - Ok(setup) => setup, - Err(error) => { - reply_node(&mut request, Ok(Response::FleetExecutionAction(Err(error)))); - return; - } - }; - let gateway = RaftCells { - handle: self.local.clone(), - }; - let domain = self.log.domain(); - self.jobs.push(Box::pin(async move { - let work = async { - let remaining = request.control.deadline().checked_sub(domain.mono_ms().map_err(|_| FleetCatalogError::Closed)?) - .filter(|ms| *ms > 0).map(Duration::from_millis).ok_or(FleetCatalogError::Unconfirmed)?; - host.execute_fleet_command(call.command, call.origin, call.minimum_revision, &gateway, remaining).await - }; - let result = crate::asyncrt::select_biased! { "cancel receiver waiter while physical owners retain admitted work"; - _ = request.control.cancellation() => Err(FleetCatalogError::Unconfirmed), - result = work => result, - }; - Job::FleetAction(Box::new(Completed { request: Box::new(request), result, _memory: Some(memory) })) - })); - } - - pub(super) fn finish_fleet_action(&mut self, completed: Completed) { - let Completed { - mut request, - result, - _memory, - } = completed; - if request.control.cancelled() || self.closing.load(Ordering::Acquire) { - return; - } - let result = if request.control.before(request.control.deadline()) { - result - } else { - Err(FleetCatalogError::Unconfirmed) - }; - reply_node(&mut request, Ok(Response::FleetExecutionAction(result))); - } -} diff --git a/crates/cellar/durability/raft/host_fleet_inspection.rs b/crates/cellar/durability/raft/host_fleet_inspection.rs deleted file mode 100644 index 0f7c2ddd3..000000000 --- a/crates/cellar/durability/raft/host_fleet_inspection.rs +++ /dev/null @@ -1,237 +0,0 @@ -//! Remote quorum metadata owns nested SDK calls through cancellation and join. -use super::*; -use crate::application::consensus::client::PeerFleetRequest; -use crate::raft::fleet_inspection_wire::{History, LocalObservation, RESPONSE_BYTES}; -use crate::raft::inspection::{InspectionError, InspectionSnapshot}; -use crate::raft::transport::fleet_inspections::Received; -use std::time::Duration; - -pub(super) struct PeerCompleted { - request: Box, - call: PeerFleetRequest, - bytes: Permit, - result: Result, -} - -pub(super) struct LocalCompleted { - request: Box, - bytes: Permit, - result: Result<(Box, LogIndex), CommandFailure>, - peer: Option, -} - -impl Driver { - pub(super) fn start_peer_fleet_inspection( - &mut self, - mut request: NodeRequest, - call: PeerFleetRequest, - ) { - if call.node == self.log.limits().node { - self.start_fleet_inspection( - request, - History::new(&call.authorization), - call.minimum_revision, - Some(call), - ); - return; - } - let admitted = (|| { - if self.options.managed.is_none() { - return Err(InspectionError::Refused); - } - let bytes = self - .control_budget - .acquire(RESPONSE_BYTES) - .ok_or(InspectionError::Overloaded)?; - let delivery = self.transport.inspect_fleet_membership( - call.node, - &call.authorization, - call.minimum_revision, - request.control.deadline(), - )?; - Ok((bytes, delivery)) - })(); - let (bytes, delivery) = match admitted { - Ok(admitted) => admitted, - Err(error) => { - reply_node(&mut request, Ok(Response::PeerFleetObservation(Err(error)))); - return; - } - }; - self.jobs.push(Box::pin(async move { - let result = crate::asyncrt::select_biased! { "cancel remote quorum observation before delivery"; - _ = request.control.cancellation() => Err(InspectionError::Unavailable), - result = delivery.receive() => result, - }; - Job::PeerFleetInspection(Box::new(PeerCompleted { - request: Box::new(request), call, bytes, result, - })) - })); - } - - pub(super) fn finish_peer_fleet_inspection(&mut self, completed: PeerCompleted) { - let PeerCompleted { - mut request, - call, - bytes, - result, - } = completed; - if request.control.cancelled() || self.closing.load(Ordering::Acquire) { - return; - } - let result = result.and_then(|received| { - self.transport.validate_fleet_node_observation( - &received.observation.node, - call.minimum_revision, - )?; - let age = self - .log - .domain() - .mono_ms() - .ok() - .and_then(|now| now.checked_sub(request.started)) - .ok_or(InspectionError::Unavailable)?; - received - .observation - .into_peer(&call.authorization, age) - .map(Box::new) - .map_err(|_| InspectionError::Refused) - }); - node_control::reply_node_charged( - &mut request, - Ok(Response::PeerFleetObservation(result)), - Some(bytes), - ); - } - - pub(super) fn start_local_fleet_inspection( - &mut self, - request: NodeRequest, - history: History, - minimum_revision: u64, - ) { - self.start_fleet_inspection(request, history, minimum_revision, None); - } - - fn start_fleet_inspection( - &mut self, - mut request: NodeRequest, - history: History, - minimum_revision: u64, - peer: Option, - ) { - let admitted = (|| { - let host = self - .options - .managed - .as_ref() - .ok_or(CommandFailure::UnsupportedCapability)? - .authorization - .clone(); - let scope = history - .scope() - .map_err(|_| CommandFailure::CommandConflict)?; - let cell = self - .cells - .get(&scope) - .ok_or(CommandFailure::UnsupportedCapability)?; - if cell.failed || cell.retired { - return Err(CommandFailure::Unavailable); - } - let gateway = RaftCells { - handle: self.local.clone(), - }; - let handle = gateway.cell(scope.namespace, &scope.cell)?; - if handle.scope() != &scope { - return Err(CommandFailure::CommandConflict); - } - let bytes = self - .control_budget - .acquire(RESPONSE_BYTES) - .ok_or(CommandFailure::Overloaded)?; - Ok((host, scope, handle, bytes)) - })(); - let (host, scope, handle, bytes) = match admitted { - Ok(admitted) => admitted, - Err(error) => { - reply_node(&mut request, Err(error)); - return; - } - }; - let domain = self.log.domain(); - self.jobs.push(Box::pin(async move { - let work = async { - let remaining = || { - let now = domain.mono_ms().map_err(|_| CommandFailure::Unavailable)?; - request.control.deadline().checked_sub(now) - .filter(|ms| *ms > 0 && *ms <= 60_000) - .map(Duration::from_millis).ok_or(CommandFailure::Unavailable) - }; - let authorization = host.group_authorization_at_least(&scope, minimum_revision, remaining()?).await - .map_err(|error| match error { - crate::raft::fleet_catalog::FleetCatalogError::Overloaded - | crate::raft::fleet_catalog::FleetCatalogError::Limit => CommandFailure::Overloaded, - _ => CommandFailure::Unavailable, - })?; - if !history.matches(&authorization) { return Err(CommandFailure::CommandConflict); } - let proof = handle.observe_fleet_membership(authorization, remaining()?).await?; - let snapshot = handle.inspect_snapshot(remaining()?).await?; - if !proof.matches_snapshot(&snapshot) { return Err(CommandFailure::Unavailable); } - Ok((snapshot, proof.barrier())) - }; - let result = crate::asyncrt::select_biased! { "cancel nested quorum wait before retaining metadata"; - _ = request.control.cancellation() => Err(CommandFailure::Unavailable), - result = work => result, - }; - Job::LocalFleetInspection(Box::new(LocalCompleted { - request: Box::new(request), bytes, result, peer, - })) - })); - } - - pub(super) fn finish_local_fleet_inspection(&mut self, completed: LocalCompleted) { - let LocalCompleted { - mut request, - bytes, - result, - peer, - } = completed; - if request.control.cancelled() || self.closing.load(Ordering::Acquire) { - return; - } - let result = result.and_then(|(snapshot, barrier)| { - let node = self.node_observation()?; - if snapshot.view.node.identity != node.node.identity - || snapshot.view.node.storage_instance != node.node.storage_instance - { - return Err(CommandFailure::Unavailable); - } - let local = LocalObservation { - node: Box::new(node), - snapshot, - barrier, - }; - if let Some(call) = peer { - self.transport - .validate_fleet_node_observation(&local.node, call.minimum_revision) - .map_err(|_| CommandFailure::Unavailable)?; - let age = self - .log - .domain() - .mono_ms() - .ok() - .and_then(|now| now.checked_sub(request.started)) - .ok_or(CommandFailure::Unavailable)?; - Ok(Response::PeerFleetObservation( - local - .into_peer(&call.authorization, age) - .map(Box::new) - .map_err(|_| InspectionError::Refused), - )) - } else { - Ok(Response::LocalFleetObservation(Box::new(local))) - } - }); - node_control::reply_node_charged(&mut request, result, Some(bytes)); - } -} diff --git a/crates/cellar/durability/raft/host_fleet_observation.rs b/crates/cellar/durability/raft/host_fleet_observation.rs index 49e5aceb8..5b2ad5d1f 100644 --- a/crates/cellar/durability/raft/host_fleet_observation.rs +++ b/crates/cellar/durability/raft/host_fleet_observation.rs @@ -1,14 +1,15 @@ -//! Fresh safe-ReadIndex membership observations, independent of SQL images. -//! Catalog and group lifecycles must still prove complete inventory separately. -use super::*; -use crate::raft::fleet_catalog::FleetGroupAuthorization; -use crate::raft::member_state::{ - FleetMembershipStatus, GroupMembership, JoiningMember, RepairStatus, -}; +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. -// Includes the bounded membership ledger, its fleet status, all four voter/ -// learner sets and reply retention. Acquired before reading/cloning the ledger. -const OBSERVATION_BYTES: usize = 64 * 1024; +//! Fresh physical-group metadata. No application cell is selected or aliased. +use crate::raft::{ + cell_groups::GroupManifest, + fleet_catalog::FleetGroupAuthorization, + member_state::{FleetMembershipStatus, GroupMembership, JoiningMember, RepairStatus}, +}; +use cellar_logic::{ + cell_group::GroupHistory, + consensus::{CommandFailure, LogIndex, LogPosition, MemberId, ReplicaIdentity}, +}; /// The exact configuration at a quorum-backed membership observation. Lists /// retain joint-consensus roles; outgoing voters are never silently omitted. @@ -42,13 +43,15 @@ impl FleetMembershipConfiguration { self.auto_leave } - fn capture(conf: &::raft::prelude::ConfState) -> Result { + pub(crate) fn capture(conf: &::raft::prelude::ConfState) -> Result { let ids = |values: &[u64]| { - values + let mut ids = values .iter() .copied() .map(|id| MemberId::new(id).map_err(|_| CommandFailure::CorruptState)) - .collect::, _>>() + .collect::, _>>()?; + ids.sort_unstable(); + Ok::<_, CommandFailure>(ids) }; Ok(Self { voters: ids(&conf.voters)?, @@ -60,107 +63,90 @@ impl FleetMembershipConfiguration { } } -/// An opaque observation minted after this invocation's safe ReadIndex and -/// local durable apply. It binds the physical process, exact history and full -/// configuration. Retaining it supplies no lease, complete inventory, archive -/// coverage or permission to remove a node or release a reservation. +/// A fresh safe-ReadIndex observation of the complete durable group root and +/// configuration. Its process/history bindings grant no current health, lease, +/// placement, archive coverage, or permission to release physical debt. #[derive(Clone, Debug)] pub struct FleetMembershipObservation { - replica: ReplicaIdentity, - birth_digest: [u8; 32], - schema_digest: [u8; 32], - barrier: LogIndex, - applied: LogPosition, + replica: ReplicaIdentity, + root: GroupManifest, + barrier: LogPosition, membership: GroupMembership, configuration: FleetMembershipConfiguration, status: FleetMembershipStatus, + configuration_digest: [u8; 32], } impl FleetMembershipObservation { - pub(crate) fn matches_snapshot( - &self, - snapshot: &crate::raft::inspection::InspectionSnapshot, - ) -> bool { - snapshot.membership.as_ref() == Some(&self.membership) - && snapshot.configuration.as_ref().is_some_and(|conf| { - FleetMembershipConfiguration::capture(conf).as_ref() == Ok(&self.configuration) - }) - && snapshot - .view - .member - .as_ref() - .is_some_and(|m| m.applied.index >= self.barrier) - } - - // Only the purpose-specific authenticated quorum RPC calls this decoder. - // An ordinary inspection snapshot alone cannot establish a read barrier. - pub(crate) fn from_quorum_snapshot( - snapshot: &crate::raft::inspection::InspectionSnapshot, + // Only the authenticated quorum RPC (or the same local owner) invokes this + // constructor. Decoded status inspection cannot mint a quorum observation. + pub(crate) fn check_quorum_group( + snapshot: &crate::raft::fleet_inspection_wire::GroupSnapshot, authorization: &FleetGroupAuthorization, - barrier: LogIndex, - ) -> Result { - snapshot.validate()?; - let view = &snapshot.view; - let member = view.member.as_ref().ok_or(CommandFailure::Unavailable)?; - let membership = snapshot + ) -> Result<(), CommandFailure> { + snapshot + .validate() + .map_err(|_| CommandFailure::CorruptState)?; + let route = snapshot.root.route(); + let status = snapshot .membership - .as_ref() - .ok_or(CommandFailure::Unavailable)?; - let configuration = snapshot - .configuration - .as_ref() - .ok_or(CommandFailure::Unavailable)?; - let status = membership .fleet_status() .ok_or(CommandFailure::UnsupportedCapability)?; - if member.applied.index < barrier - || barrier.get() == 0 - || &status.binding != authorization.binding() - || !authorization.matches_scope(&view.scope) - || authorization.history_digests() != (view.birth_digest, view.schema_digest) + if &status.binding != authorization.binding() + || !authorization.matches_scope(&route.history) + || authorization.history_digests() != (route.birth_digest, route.schema_digest) + || !authorization.matches_replica(&snapshot.replica, &snapshot.membership) { return Err(CommandFailure::CommandConflict); } - let replica = ReplicaIdentity { - scope: view.scope.clone(), - member: member.member, - node: view.node.identity.node, - disk: view.node.identity.disk, - process: view.node.identity.process, - }; - if !authorization.matches_replica(&replica, membership) { - return Err(CommandFailure::CommandConflict); - } + Ok(()) + } + + pub(crate) fn from_quorum_group( + snapshot: crate::raft::fleet_inspection_wire::GroupSnapshot, + authorization: &FleetGroupAuthorization, + ) -> Result { + Self::check_quorum_group(&snapshot, authorization)?; + let status = snapshot + .membership + .fleet_status() + .ok_or(CommandFailure::UnsupportedCapability)?; + let configuration = FleetMembershipConfiguration::capture(&snapshot.configuration)?; + let configuration_digest = + group_membership_digest(&snapshot.membership, &snapshot.configuration)?; Ok(Self { - replica, - birth_digest: view.birth_digest, - schema_digest: view.schema_digest, - barrier, - applied: member.applied, - membership: membership.clone(), - configuration: FleetMembershipConfiguration::capture(configuration)?, + replica: snapshot.replica, + root: snapshot.root, + barrier: snapshot.barrier, + membership: snapshot.membership, + configuration, status, + configuration_digest, }) } - pub fn replica(&self) -> &ReplicaIdentity { + pub fn replica(&self) -> &ReplicaIdentity { &self.replica } + pub fn root(&self) -> &GroupManifest { + &self.root + } + pub fn birth_digest(&self) -> [u8; 32] { - self.birth_digest + self.root.route().birth_digest } pub fn schema_digest(&self) -> [u8; 32] { - self.schema_digest + self.root.route().schema_digest } pub fn barrier(&self) -> LogIndex { - self.barrier + self.barrier.index } pub fn applied(&self) -> LogPosition { - self.applied + self.root.position() } pub(crate) fn member_high_water(&self) -> u64 { @@ -179,8 +165,10 @@ impl FleetMembershipObservation { &self.status } - /// Only dynamically joined bindings have a committed disk here. Original - /// members retain their immutable birth identity and separate engine pins. + pub fn configuration_digest(&self) -> [u8; 32] { + self.configuration_digest + } + pub fn joined_member( &self, member: MemberId, @@ -195,111 +183,21 @@ impl FleetMembershipObservation { .repair_status() .map_err(|_| CommandFailure::CorruptState) } - - fn capture( - cell: &Cell, - authorization: &FleetGroupAuthorization, - barrier: LogIndex, - ) -> Result, CommandFailure> { - let cache = cell - .group - .read() - .map_err(|_| CommandFailure::CorruptState)?; - if cache.applied.position.index < barrier - || cell.route_revision < cache.membership.revision() - { - return Ok(None); - } - cache - .membership - .validate(&cache.conf_state, cache.applied.position.index.get()) - .and_then(|()| cache.membership.validate_scope(cell.birth.scope())) - .map_err(|_| CommandFailure::CorruptState)?; - let status = cache - .membership - .fleet_status() - .ok_or(CommandFailure::UnsupportedCapability)?; - if &status.binding != authorization.binding() - || !authorization.matches_birth(&cell.birth, cell.schema_digest) - { - return Err(CommandFailure::CommandConflict); - } - Ok(Some(Self { - replica: cell.group.identity().clone(), - birth_digest: cell.birth.digest(), - schema_digest: cell.schema_digest, - barrier, - applied: cache.applied.position, - membership: cache.membership.clone(), - configuration: FleetMembershipConfiguration::capture(&cache.conf_state)?, - status, - })) - } } -impl Driver { - pub(super) fn start_fleet_observation( - &mut self, - mut request: Request, - authorization: FleetGroupAuthorization, - ) { - let valid = self.cells.get(&request.scope).is_some_and(|cell| { - !cell.failed - && !cell.retired - && authorization.matches_scope(&request.scope) - && authorization.matches_birth(&cell.birth, cell.schema_digest) - }); - if !valid { - reply(&mut request, Err(CommandFailure::CommandConflict)); - return; - } - let Some(charge) = self.control_budget.acquire(OBSERVATION_BYTES) else { - reply(&mut request, Err(CommandFailure::Overloaded)); - return; - }; - let Some(id) = self.next_read_id() else { - reply(&mut request, Err(CommandFailure::Overloaded)); - return; - }; - self.reads - .insert(id, PendingRead::fleet(request, authorization, charge)); - } - - pub(super) fn finish_fleet_observation(&mut self, id: ReadId) { - use crate::raft::query::ReadOperation; - let outcome = (|| { - let read = self.reads.get(&id).ok_or(CommandFailure::CorruptState)?; - let ReadOperation::Fleet { authorization, .. } = &read.operation else { - return Err(CommandFailure::CorruptState); - }; - let Some(Minimum::ReadIndex(barrier)) = read.minimum.as_ref() else { - return Err(CommandFailure::CorruptState); - }; - let cell = self - .cells - .get(&read.request.scope) - .ok_or(CommandFailure::Unavailable)?; - FleetMembershipObservation::capture(cell, authorization, *barrier) - })(); - if matches!(outcome, Ok(None)) { - return; - } - let Some(mut read) = self.reads.remove(&id) else { - return; - }; - let ReadOperation::Fleet { charge, .. } = read.operation else { - reply(&mut read.request, Err(CommandFailure::CorruptState)); - return; - }; - let result = if read.request.control.cancelled() { - Err(CommandFailure::VersionUnavailable) - } else { - match outcome { - Ok(Some(observation)) => Ok(Response::FleetObservation(Box::new(observation))), - Err(error) => Err(error), - Ok(None) => Err(CommandFailure::CorruptState), - } - }; - reply_charged(&mut read.request, result, Some(charge)); - } +pub(crate) fn group_membership_digest( + membership: &GroupMembership, + configuration: &::raft::prelude::ConfState, +) -> Result<[u8; 32], CommandFailure> { + use sha2::{Digest, Sha256}; + let ledger = serde_json::to_vec(membership).map_err(|_| CommandFailure::CorruptState)?; + let conf = crate::raft::manifest::encode_conf(configuration) + .map_err(|_| CommandFailure::CorruptState)?; + let mut digest = Sha256::new(); + digest.update(b"cellar-fleet-group-membership-1\0"); + digest.update((ledger.len() as u64).to_be_bytes()); + digest.update(ledger); + digest.update((conf.len() as u64).to_be_bytes()); + digest.update(conf); + Ok(digest.finalize().into()) } diff --git a/crates/cellar/durability/raft/host_image_limits.rs b/crates/cellar/durability/raft/host_image_limits.rs index 3dbc17a41..23f375d74 100644 --- a/crates/cellar/durability/raft/host_image_limits.rs +++ b/crates/cellar/durability/raft/host_image_limits.rs @@ -27,9 +27,10 @@ impl RaftCellsConfig { options: super::RaftExtentOptions, ) -> anyhow::Result { anyhow::ensure!( - self.cells + self.namespaces .iter() - .all(|cell| cell.namespace.image_storage == super::RaftImageStorage::Extents), + .all(|namespace| namespace.namespace.image_storage + == super::RaftImageStorage::Extents), "extent image policy requires extent namespaces" ); self.host.extents = Some(options); diff --git a/crates/cellar/durability/raft/host_image_limits_tests.rs b/crates/cellar/durability/raft/host_image_limits_tests.rs index ddc5d3df8..65e6eb87b 100644 --- a/crates/cellar/durability/raft/host_image_limits_tests.rs +++ b/crates/cellar/durability/raft/host_image_limits_tests.rs @@ -20,7 +20,10 @@ fn config() -> RaftCellsConfig { transport: Default::default(), }, catalog: Default::default(), - cells: vec![], + namespaces: vec![], + groups: vec![], + command_origins: vec![], + shared: crate::raft::group_host::test_support::limits(), peers: vec![], query_origins: vec![], host: Default::default(), @@ -158,31 +161,18 @@ fn extent_profile_refuses_inadequate_shared_inventory() { #[test] fn extent_profile_requires_an_explicit_namespace_choice() { let mut configured = config(); - let scope = GroupScope { - cluster: ClusterId::new([1; 16]).unwrap(), - namespace: NamespaceId::new([2; 16]).unwrap(), - cell: "Counter:policy".into(), - group: GroupId::new([4; 16]).unwrap(), - incarnation: IncarnationId::new([5; 16]).unwrap(), - }; - configured.cells.push(RaftCellRegistration { - placement: RaftCellPlacement::Replica, - namespace: NamespaceConfig::new( - scope.cluster, - scope.namespace, - cellar_logic::NamespaceMode::Raft, - [6; 32], - ), - class: "Counter".into(), - birth: BirthProposal { - scope, - voters: vec![InitialMember { - member: MemberId::new(1).unwrap(), - node: NodeId::new([1; 16]).unwrap(), - }], - learners: vec![], - }, - }); + configured + .namespaces + .push(crate::raft::group_host::RaftNamespaceConfig { + namespace: NamespaceConfig::new( + cellar_logic::consensus::ClusterId::new([1; 16]).unwrap(), + cellar_logic::consensus::NamespaceId::new([2; 16]).unwrap(), + cellar_logic::NamespaceMode::Raft, + [6; 32], + ), + class: "Counter".into(), + initialize: None, + }); assert!(configured .with_extent_image_limit(10_000_000_000, Default::default()) .is_err()); diff --git a/crates/cellar/durability/raft/host_inspection.rs b/crates/cellar/durability/raft/host_inspection.rs index 240a21d94..9a0c8d7d1 100644 --- a/crates/cellar/durability/raft/host_inspection.rs +++ b/crates/cellar/durability/raft/host_inspection.rs @@ -1,348 +1,8 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. -//! Capture bounded registered-history metadata through reserved control work. +//! Exact physical image publication observations, independent of a dispatcher. use super::*; use crate::raft::cache::CachedGroup; -use crate::raft::inspection::{ - CellInspection, InspectionError, InspectionSnapshot, MemberInspection, NodeInspection, - ReplicaOccupancy, RESPONSE_RESERVATION, -}; - -pub(super) struct Completed { - request: Box, - started: u64, - result: Result, -} - -pub(super) struct PeerCompleted { - request: Box, - started: u64, - bytes: Permit, - result: Result, -} - -impl Driver { - pub(super) fn start_peer_inspection(&mut self, mut request: Request, peer: NodeId) { - let admitted = (|| { - if self.options.node_profile.is_none() || self.directory.is_none() { - return Err(InspectionError::Refused); - } - let endpoint = self - .peers - .get(&peer) - .cloned() - .ok_or(InspectionError::Refused)?; - let binding = self - .local - .bindings - .get(&(request.scope.namespace, request.scope.cell.clone())) - .map_err(|_| InspectionError::Unavailable)?; - let birth = self - .cells - .get(&request.scope) - .map(|cell| cell.birth.digest()) - .or_else(|| { - self.standbys - .get(&request.scope) - .map(|cell| cell.birth.digest()) - }) - .ok_or(InspectionError::Refused)?; - let bytes = self - .control_budget - .acquire(RESPONSE_RESERVATION) - .ok_or(InspectionError::Overloaded)?; - let started = self - .log - .domain() - .mono_ms() - .map_err(|_| InspectionError::Unavailable)?; - let delivery = self.transport.inspect_peer( - request.scope.clone(), - binding.class.schema_digest(), - birth, - endpoint, - request.control.deadline(), - )?; - Ok((started, bytes, delivery)) - })(); - let (started, bytes, delivery) = match admitted { - Ok(admitted) => admitted, - Err(error) => { - reply(&mut request, Ok(Response::PeerInspection(Err(error)))); - return; - } - }; - self.jobs.push(Box::pin(async move { - let result = crate::asyncrt::select_biased! { "cancel an inspection before retaining its response"; - _ = request.control.cancellation() => Err(InspectionError::Unavailable), - result = delivery.receive() => result, - }; - Job::PeerInspection(Box::new(PeerCompleted { request: Box::new(request), started, bytes, result })) - })); - } - - pub(super) fn finish_peer_inspection(&mut self, completed: PeerCompleted) { - let PeerCompleted { - mut request, - started, - bytes, - result, - } = completed; - if request.control.cancelled() || self.closing.load(Ordering::Acquire) { - return; - } - let result = result.and_then(|mut received| { - let elapsed = self - .log - .domain() - .mono_ms() - .ok() - .and_then(|now| now.checked_sub(started)) - .ok_or(InspectionError::Unavailable)?; - if let Some(member) = received.snapshot.view.member.as_mut() { - member.sample_age_ms = member - .sample_age_ms - .checked_add(elapsed) - .ok_or(InspectionError::Unavailable)?; - } - if let Some(load) = received.snapshot.view.leader_load.as_mut() { - load.sample_age_ms = load - .sample_age_ms - .checked_add(elapsed) - .ok_or(InspectionError::Unavailable)?; - } - Ok(received.snapshot) - }); - reply_charged( - &mut request, - Ok(Response::PeerInspection(result)), - Some(bytes), - ); - } - - pub(super) fn start_inspection(&mut self, request: Request) { - if self.options.node_profile.is_none() { - let mut request = request; - reply(&mut request, Err(CommandFailure::UnsupportedCapability)); - return; - } - let scope = request.scope.clone(); - if self - .cells - .get(&scope) - .is_some_and(|cell| !cell.failed && !cell.retired) - { - let runtime = self.runtime.clone(); - let domain = self.log.domain(); - self.jobs.push(Box::pin(async move { - let (started, result) = match domain.mono_ms() { - Ok(started) => (started, runtime.status(&scope).await), - Err(_) => (0, Err(RuntimeError::Closed)), - }; - Job::Inspection(Box::new(Completed { - request: Box::new(request), - started, - result, - })) - })); - } else { - self.reply_inspection(request, None); - } - } - - pub(super) fn finish_inspection(&mut self, completed: Completed) { - let Completed { - request, - started, - result, - } = completed; - let mut request = *request; - if request.control.cancelled() { - return; - } - match result { - Ok(status) => self.reply_inspection(request, Some((started, status))), - Err(error) => reply( - &mut request, - Err(match error { - RuntimeError::Overloaded | RuntimeError::NotReady => CommandFailure::Overloaded, - _ => CommandFailure::Unavailable, - }), - ), - } - } - - fn reply_inspection(&self, mut request: Request, status: Option<(u64, GroupStatus)>) { - let Some(bytes) = self.control_budget.acquire(RESPONSE_RESERVATION) else { - reply(&mut request, Err(CommandFailure::Overloaded)); - return; - }; - let result = self - .capture_inspection(&request.scope, status) - .map(|snapshot| Response::Inspection(Box::new(snapshot))); - reply_charged(&mut request, result, Some(bytes)); - } - - fn capture_inspection( - &self, - scope: &GroupScope, - status: Option<(u64, GroupStatus)>, - ) -> Result { - let node = self.capture_node_profile()?; - let groups = node.groups; - let leader_load = self.log.domain().mono_ms().ok().and_then(|now| { - self.leader_load.as_ref()?.view( - now, - load::Population { - cells: self.cells.len(), - groups: groups as usize, - }, - self.activating.is_some(), - node.profile.leader_capacity, - ) - }); - let mut snapshot = InspectionSnapshot { - view: CellInspection { - scope: scope.clone(), - schema_digest: [0; 32], - birth_digest: [0; 32], - node, - occupancy: ReplicaOccupancy::Failed, - member: None, - leader_load, - }, - membership: None, - configuration: None, - publication: None, - }; - self.fill_inspection(scope, status, &mut snapshot)?; - Ok(snapshot) - } - - pub(super) fn capture_node_profile(&self) -> Result { - let mut profile = self - .options - .node_profile - .as_ref() - .ok_or(CommandFailure::UnsupportedCapability)? - .clone(); - profile.draining |= self.local.admission.application_paused(); - let groups = self - .log - .group_count_with_pending(self.activating.as_ref()) - .map_err(|_| CommandFailure::Unavailable)?; - let groups = u32::try_from(groups).map_err(|_| CommandFailure::Overloaded)?; - let reserved_bytes = profile - .bytes_per_group - .checked_mul(u64::from(groups)) - .ok_or(CommandFailure::Overloaded)?; - let local = self.log.limits(); - Ok(NodeInspection { - identity: NodeIdentity { - node: local.node, - disk: local.disk, - process: local.process, - }, - storage_instance: self - .log - .storage_instance() - .map_err(|_| CommandFailure::Unavailable)?, - profile, - groups, - reserved_bytes, - }) - } - - fn fill_inspection( - &self, - scope: &GroupScope, - status: Option<(u64, GroupStatus)>, - snapshot: &mut InspectionSnapshot, - ) -> Result<(), CommandFailure> { - if let Some(cell) = self.cells.get(scope) { - snapshot.view.schema_digest = cell.schema_digest; - snapshot.view.birth_digest = cell.birth.digest(); - let member = cell.group.identity().member; - let cache = cell.group.read().map_err(|_| CommandFailure::Unavailable)?; - if cell.failed { - return Ok(()); - } - if cell.retired - || !crate::raft::member_state::members(&cache.conf_state).contains(&member.get()) - { - snapshot.view.occupancy = ReplicaOccupancy::Retired(member); - return Ok(()); - } - let (started, status) = status.ok_or(CommandFailure::Overloaded)?; - if status.replica != *cell.group.identity() - || status.term.get() != cache.hard_state.term - || status.membership_revision != cache.membership.revision() - || status.committed.get() != cache.hard_state.commit - || status.applied != cache.applied.position - { - return Err(CommandFailure::Overloaded); - } - let published = cell.image.as_ref().map(|image| image.position); - if cell - .image - .as_ref() - .is_some_and(|image| !valid_published_position(&cache, image)) - { - return Err(CommandFailure::Overloaded); - } - let sample_age_ms = self - .log - .domain() - .mono_ms() - .map_err(|_| CommandFailure::Unavailable)? - .checked_sub(started) - .ok_or(CommandFailure::Unavailable)?; - snapshot.view.occupancy = ReplicaOccupancy::Member(member); - snapshot.view.member = Some(MemberInspection { - member, - term: status.term, - leader: status.leader.and_then(|id| MemberId::new(id).ok()), - is_leader: status.is_leader, - membership_revision: cache.membership.revision(), - member_high_water: cache.membership.high_water(), - committed: status.committed, - applied: status.applied, - published, - repair: cache - .membership - .repair_status() - .map_err(|_| CommandFailure::CorruptState)?, - sample_age_ms, - }); - snapshot.membership = Some(cache.membership.clone()); - snapshot.configuration = Some(cache.conf_state.clone()); - snapshot.publication = Some(Publication::capture( - &cache.applied, - cell.image - .as_ref() - .map(|image| (&image.image, image.position)), - )?); - return Ok(()); - } - if let Some(dormant) = self.standbys.get(scope) { - snapshot.view.schema_digest = dormant.class.schema_digest(); - snapshot.view.birth_digest = dormant.birth.digest(); - snapshot.view.occupancy = if dormant.failed { - ReplicaOccupancy::Failed - } else if self.activating.as_ref() == Some(scope) { - ReplicaOccupancy::Initializing - } else { - match self.log.group(scope) { - Ok(group) => ReplicaOccupancy::Occupied(group.identity().member), - Err(super::super::storage::LogError::NotFound) => ReplicaOccupancy::Standby, - Err(_) => return Err(CommandFailure::Unavailable), - } - }; - return Ok(()); - } - Err(CommandFailure::UnsupportedCapability) - } -} fn valid_published_position(cache: &CachedGroup, image: &ImageView) -> bool { let position = image.position; diff --git a/crates/cellar/durability/raft/host_leader_load.rs b/crates/cellar/durability/raft/host_leader_load.rs index 706dc0677..cbb83f90b 100644 --- a/crates/cellar/durability/raft/host_leader_load.rs +++ b/crates/cellar/durability/raft/host_leader_load.rs @@ -3,9 +3,8 @@ //! Node-wide leadership observations, collected in bounded owned batches. //! A count is published only after every attached history was sampled. use super::*; -use crate::application::consensus::{RaftGroupResources, RaftReplicaRole}; +use crate::application::consensus::RaftGroupResources; use crate::raft::inspection::LeaderLoad; -use std::ops::Bound::{Excluded, Unbounded}; /// Metadata only: retaining an SDK handle cannot retain a group or engine. #[derive(Default)] @@ -284,157 +283,6 @@ impl Scanner { } } -impl Driver { - pub(super) fn pump_leader_load(&mut self) { - let Ok(now) = self.log.domain().mono_ms() else { - return; - }; - let Ok(groups) = self.log.group_count() else { - return; - }; - let population = Population { - cells: self.cells.len(), - groups, - }; - let Some(scanner) = self.leader_load.as_mut() else { - return; - }; - let begins = scanner.begin(now, population, self.activating.is_some()); - if let Ok(mut source) = self.local.groups.0.lock() { - *source = scanner - .cached - .as_ref() - .filter(|cached| cached.population == population && self.activating.is_none()) - .map(|cached| (cached.started, cached.resources)); - } - if !begins { - return; - } - let Some(cycle) = scanner.cycle.as_ref() else { - return; - }; - let after = cycle.cursor.as_ref().map_or(Unbounded, Excluded); - let batch: Vec<_> = self - .cells - .range((after, Unbounded)) - .take(scanner.options.groups_per_batch) - .map(|(_, cell)| cell.group.identity().clone()) - .collect(); - if batch.is_empty() { - scanner.failed(now); - return; - } - let sequence = cycle.sequence; - let deadline = cycle.deadline; - scanner.busy = true; - let runtime = self.runtime.clone(); - let domain = self.log.domain(); - self.jobs.push(Box::pin(async move { - let observed = batch.into_iter().map(|identity| { - let runtime = runtime.clone(); - let domain = domain.clone(); - async move { - let result = match domain.sleep_until(deadline) { - Ok(timer) => crate::asyncrt::select_biased! { "the original complete-scan deadline precedes a late status"; - _ = timer => Err(RuntimeError::NotReady), - result = runtime.status(&identity.scope) => result, - }, - Err(_) => Err(RuntimeError::Closed), - }; - (identity, result) - } - }); - Job::LeaderLoad(Box::new(Completed { sequence, result: futures_util::future::join_all(observed).await })) - })); - } - - pub(super) fn finish_leader_load(&mut self, completed: Completed) { - let Ok(now) = self.log.domain().mono_ms() else { - return; - }; - let Ok(groups) = self.log.group_count() else { - return; - }; - let population = Population { - cells: self.cells.len(), - groups, - }; - if let Some(scanner) = self.leader_load.as_mut() { - scanner.accept( - completed, - now, - population, - self.activating.is_some(), - |status| { - let cell = self.cells.get(&status.replica.scope)?; - if cell.group.identity() != &status.replica { - return None; - } - let registered = self - .local - .bindings - .get(&( - status.replica.scope.namespace, - status.replica.scope.cell.clone(), - )) - .ok()? - .class - .has_queries(); - let (role, queries) = readiness::capture( - cell, - status, - now, - self.options.stale_source_timeout_ms, - self.local.admission.is_draining(), - registered, - ) - .ok()?; - let apply_lag = status - .committed - .get() - .checked_sub(status.applied.index.get())?; - let archive_lag = status.committed.get().saturating_sub(cell.archived.get()); - Some(RaftGroupResources { - leaders: usize::from(role == RaftReplicaRole::Leader), - voters: usize::from(role == RaftReplicaRole::Voter), - learners: usize::from(role == RaftReplicaRole::Learner), - removed: usize::from(role == RaftReplicaRole::Removed), - unknown: usize::from(role == RaftReplicaRole::Unknown), - coherent_images: usize::from(queries.coherent_image), - historical_queries: usize::from(queries.historical), - recent_source_queries: usize::from(queries.recent_source), - failed: usize::from(cell.failed), - retired: usize::from(cell.retired), - sql_busy: usize::from(cell.sql_busy), - pending_commands: usize::from(cell.pending.is_some()), - incoming_seeds: usize::from(cell.incoming.is_some()), - applying: usize::from( - cell.apply.is_some() - || cell.publication.is_some() - || cell.apply_inflight, - ), - archive_pending: usize::from(archive_lag > 0), - archive_current: usize::from(cell.archived.get() > 0 && archive_lag == 0), - archive_unconfirmed: usize::from(cell.archived.get() == 0), - apply_lag_entries: apply_lag, - max_apply_lag_entries: apply_lag, - archive_lag_entries: archive_lag, - max_archive_lag_entries: archive_lag, - ..Default::default() - }) - }, - ); - if let Ok(mut source) = self.local.groups.0.lock() { - *source = scanner - .cached - .as_ref() - .filter(|cached| cached.population == population && self.activating.is_none()) - .map(|cached| (cached.started, cached.resources)); - } - } - } -} - #[cfg(test)] #[path = "host_leader_load_tests.rs"] mod tests; diff --git a/crates/cellar/durability/raft/host_node.rs b/crates/cellar/durability/raft/host_node.rs index 6902989d8..314fd7c53 100644 --- a/crates/cellar/durability/raft/host_node.rs +++ b/crates/cellar/durability/raft/host_node.rs @@ -50,7 +50,10 @@ pub struct RaftHostObservation { pub fleet_trust_revision: Option, pub declared_namespaces: u32, pub registered_histories: u32, + /// Registered groups unavailable for application work, including quarantine. pub dormant_histories: u32, + /// Pending physical attachment/provisioning operations. This is independent + /// of durable engine occupancy and is not limited to one business cell. pub pending_history_registrations: u32, /// Physical engine slots not attached as running cells, including retained /// or retired histories. This count never claims inventory completeness. @@ -72,191 +75,3 @@ pub struct RaftPeerNodeObservation { /// residence and retention by an unpolled caller. pub sample_age_ms: u64, } - -impl Driver { - pub(super) fn on_control(&mut self, request: ControlRequest) { - let mut request = match request { - ControlRequest::Cell(request) => { - self.on_request(*request); - return; - } - ControlRequest::Node(request) => *request, - }; - if matches!(&request.kind, NodeKind::Drain) { - self.start_node_drain(request); - return; - } - if self.closing.load(Ordering::Acquire) || request.control.cancelled() { - reply_node(&mut request, Err(CommandFailure::Unavailable)); - return; - } - match std::mem::replace(&mut request.kind, NodeKind::ControllerStatus) { - NodeKind::StartRestartService(work) => self.start_restart_service(request, *work), - NodeKind::StartMovementService(work) => self.start_movement_service(request, *work), - NodeKind::PeerFleetAction(call) => self.start_peer_fleet_action(request, *call), - NodeKind::LocalFleetAction(call) => self.start_local_fleet_action(request, *call), - NodeKind::PeerFleetStop(call) => self.start_peer_fleet_stop(request, *call), - NodeKind::InspectPeerFleet(call) => self.start_peer_fleet_inspection(request, *call), - NodeKind::ObserveFleet { - history, - minimum_revision, - } => self.start_local_fleet_inspection(request, *history, minimum_revision), - NodeKind::InspectPeer { - node, - minimum_revision, - physical, - } => self.start_node_inspection(request, node, minimum_revision, physical), - NodeKind::RegisterHistory(authorization) => { - self.start_history_registration(request, *authorization) - } - NodeKind::Observe => { - let result = self.node_observation().map(|o| Response::Node(Box::new(o))); - reply_node(&mut request, result); - } - NodeKind::ObservePhysical => { - let Some(bytes) = self - .control_budget - .acquire(super::super::physical_inventory::RESPONSE_BYTES) - else { - reply_node(&mut request, Err(CommandFailure::Overloaded)); - return; - }; - let result = self - .physical_node_observation() - .map(|(node, inventory)| Response::PhysicalNode(Box::new((node, inventory)))); - reply_node_charged(&mut request, result, Some(bytes)); - } - NodeKind::ControllerStatus => self.controller_control(request, None), - NodeKind::SetControllerEnabled(enabled) => { - self.controller_control(request, Some(enabled)) - } - NodeKind::Drain => reply_node(&mut request, Err(CommandFailure::CorruptState)), - } - } - - pub(super) fn node_observation(&self) -> Result { - let node = self.capture_node_profile()?; - let physical = self - .log - .group_count() - .map_err(|_| CommandFailure::Unavailable)?; - let unattached = physical - .checked_sub(self.cells.len()) - .ok_or(CommandFailure::CorruptState)?; - let bounded = |n| u32::try_from(n).map_err(|_| CommandFailure::Overloaded); - let dormant = self - .standbys - .keys() - .filter(|scope| !self.cells.contains_key(*scope)) - .count(); - let current_enrollment = self - .directory - .as_ref() - .and_then(|d| d.authorization(node.identity.node)) - .map(RaftHostEnrollmentObservation::capture); - Ok(RaftHostObservation { - node, - fleet_boot: self - .log - .lifecycle_guard() - .map_err(|_| CommandFailure::Unavailable)? - .map(|guard| guard.boot), - startup_bounds: self.startup_bounds()?, - startup_enrollment: self - .options - .managed - .as_ref() - .map(|managed| RaftHostEnrollmentObservation::capture(&managed.authorization)), - current_enrollment, - fleet_trust_revision: self - .directory - .as_ref() - .and_then(directory::Directory::rooted_revision), - declared_namespaces: bounded(self.declared_namespaces.len())?, - registered_histories: bounded( - self.cells - .len() - .checked_add(dormant) - .ok_or(CommandFailure::Overloaded)?, - )?, - dormant_histories: bounded(dormant)?, - pending_history_registrations: u32::from(self.history_registration.is_some()), - unattached_engine_histories: bounded(unattached)?, - seed_workers: bounded( - self.seed_archive_work - .usage() - .1 - .checked_add( - self.transport - .resource_observer() - .capture() - .ok_or(CommandFailure::Unavailable)? - .snapshot_source - .items, - ) - .ok_or(CommandFailure::Overloaded)?, - )?, - }) - } - - pub(super) fn physical_node_observation( - &self, - ) -> Result { - let inventory = self - .log - .physical_inventory(self.activating.as_ref()) - .map_err(|_| CommandFailure::Unavailable)?; - let node = self.node_observation()?; - // Engine sync can insert a group between the two lock acquisitions. - // Refuse a mixed cut rather than silently omitting an occupied slot. - inventory - .validate(&node) - .map_err(|_| CommandFailure::Overloaded)?; - Ok((node, inventory)) - } - - fn startup_bounds(&self) -> Result { - let runtime = self.runtime.limits(); - let log = self.log.limits(); - let buffers = [ - self.options.max_pending_bytes, - self.options.max_control_bytes, - runtime.max_request_bytes, - runtime.max_event_bytes, - runtime.max_ready_bytes, - runtime.max_apply_bytes, - log.max_pending_bytes, - log.max_cache_bytes, - ]; - let bounded_buffer_bytes = buffers - .into_iter() - .try_fold(0u64, |sum, bytes| { - sum.checked_add(u64::try_from(bytes).ok()?) - }) - .ok_or(CommandFailure::CorruptState)?; - let count = |n| u32::try_from(n).map_err(|_| CommandFailure::CorruptState); - Ok(RaftHostStartupBounds { - host_groups: count(self.options.max_groups)?, - consensus_groups: count(runtime.max_groups)?, - engine_groups: count(log.max_groups)?, - bounded_buffer_bytes, - }) - } -} - -pub(super) fn reply_node(request: &mut NodeRequest, result: Result) { - reply_node_charged(request, result, None); -} - -pub(super) fn reply_node_charged( - request: &mut NodeRequest, - result: Result, - bytes: Option, -) { - if let Some(reply) = request.reply.take() { - let _ = reply.send(result.map(|response| client::ChargedResponse { - response, - _bytes: bytes, - })); - } -} diff --git a/crates/cellar/durability/raft/host_readiness.rs b/crates/cellar/durability/raft/host_readiness.rs index 26f69303e..b5c94720e 100644 --- a/crates/cellar/durability/raft/host_readiness.rs +++ b/crates/cellar/durability/raft/host_readiness.rs @@ -2,7 +2,7 @@ //! Conservative local observations, separate from per-invocation proof gates. use super::*; -use crate::application::consensus::{RaftQueryReadiness, RaftReplicaRole}; +use crate::application::consensus::RaftReplicaRole; fn role(cache: &super::super::cache::CachedGroup, status: &GroupStatus) -> RaftReplicaRole { if cache.membership.revision() != status.membership_revision @@ -38,45 +38,3 @@ fn coherent_image( cache.coherent_image_position(&image.image, image.position) == Some(applied) }) } - -pub(super) fn capture( - cell: &Cell, - status: &GroupStatus, - now: u64, - source_timeout_ms: u64, - draining: bool, - registered_queries: bool, -) -> Result<(RaftReplicaRole, RaftQueryReadiness), RuntimeError> { - let cache = cell.group.read()?; - let role = role(&cache, status); - let coherent_image = coherent_image(&cache, cell.image.as_ref(), status.applied); - let historical = registered_queries - && coherent_image - && !matches!(role, RaftReplicaRole::Removed | RaftReplicaRole::Unknown) - && !draining - && !cell.failed - && !cell.retired - && cell.incoming.is_none() - && !cell.sql_busy - && cell.apply.is_none() - && cell.publication.is_none(); - let source_age_ms = status.source_observed_ms.and_then(|at| now.checked_sub(at)); - let recent_source = historical - && super::super::query::recent_source_deadline( - status.source_observed_ms, - now, - source_timeout_ms, - ) - .is_some(); - Ok(( - role, - RaftQueryReadiness { - registered_queries, - coherent_image, - historical, - recent_source, - source_age_ms, - source_timeout_ms, - }, - )) -} diff --git a/crates/cellar/durability/raft/host_seed.rs b/crates/cellar/durability/raft/host_seed.rs index 5c32ca603..2105e9ed5 100644 --- a/crates/cellar/durability/raft/host_seed.rs +++ b/crates/cellar/durability/raft/host_seed.rs @@ -4,7 +4,6 @@ //! archive is never a substitute for this request's live-quorum authorization. use super::*; -use crate::raft::query::ReadOperation; use crate::raft::transport::seeds::SourceSeed; pub(super) fn valid_target( @@ -103,11 +102,8 @@ impl Driver { let Some(mut read) = self.reads.remove(&id) else { return; }; - let ReadOperation::Seed { target, .. } = &read.operation else { - reply(&mut read.request, Err(CommandFailure::CorruptState)); - return; - }; - let Some(Minimum::ReadIndex(barrier)) = read.minimum.clone() else { + let target = &read.target; + let Some(barrier) = read.minimum else { reply(&mut read.request, Err(CommandFailure::CorruptState)); return; }; @@ -171,15 +167,13 @@ impl Driver { return; }; let scope = read.request.scope.clone(); - let ReadOperation::Seed { target, kind, .. } = &read.operation else { - self.finish_read(&id, Err(CommandFailure::CorruptState)); - return; - }; - let Some(Minimum::ReadIndex(barrier)) = read.minimum.as_ref() else { + let target = &read.target; + let kind = read.kind; + let Some(barrier) = read.minimum.as_ref() else { self.finish_read(&id, Err(CommandFailure::CorruptState)); return; }; - if *kind != crate::raft::transport::seeds::SeedKind::S3 + if kind != crate::raft::transport::seeds::SeedKind::S3 || read.expired() || self .cells diff --git a/crates/cellar/durability/raft/host_sql/cell_groups/admission.rs b/crates/cellar/durability/raft/host_sql/cell_groups/admission.rs new file mode 100644 index 000000000..c8c7d8a9f --- /dev/null +++ b/crates/cellar/durability/raft/host_sql/cell_groups/admission.rs @@ -0,0 +1,131 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use cellar_logic::{ + cell_group::CellId, + consensus::{ClusterId, NamespaceId}, +}; +use std::collections::{BTreeMap, BTreeSet}; +use std::sync::{Mutex, Weak}; + +type LogicalCell = (ClusterId, NamespaceId, CellId); + +pub(super) struct Admission { + lease: Arc, + node: Arc, + limits: CellSqlLimits, + state: Mutex, +} + +#[derive(Default)] +struct State { + cells: BTreeSet, + groups: BTreeMap>, +} + +pub(super) struct Bytes { + _lease: Arc, + _node: Permit, + _group: Permit, +} + +impl Bytes { + pub fn shrink(bytes: &mut Arc, retained: usize) -> Result<(), ExecutionError> { + let unique = + Arc::get_mut(bytes).ok_or(ExecutionError::Invalid("shared capture admission"))?; + if !unique._node.shrink_to(retained) || !unique._group.shrink_to(retained) { + return Err(ExecutionError::Invalid("retained SQL output bound")); + } + Ok(()) + } +} + +pub(super) struct CellLease { + owner: Arc, + cell: LogicalCell, +} + +impl Drop for CellLease { + fn drop(&mut self) { + if let Ok(mut state) = self.owner.state.lock() { + state.cells.remove(&self.cell); + } + } +} + +impl Admission { + pub fn limits(&self) -> CellSqlLimits { + self.limits + } + + pub fn new(limits: CellSqlLimits, lease: Arc) -> Arc { + Arc::new(Self { + lease, + node: Budget::new(limits.node_bytes, limits.max_pending_cells), + limits, + state: Mutex::new(State::default()), + }) + } + + pub fn reserve( + self: &Arc, + cell: &CellHistory, + group: GroupHistory, + bytes: usize, + ) -> Result<(CellLease, Arc), ExecutionError> { + let mut state = self.state.lock().map_err(|_| unavailable())?; + let key = (cell.cluster, cell.namespace, cell.cell.clone()); + if state.cells.contains(&key) { + return Err(ExecutionError::Busy); + } + let bytes = self.reserve_bytes(&mut state, group, bytes)?; + state.cells.insert(key.clone()); + Ok(( + CellLease { + owner: self.clone(), + cell: key, + }, + bytes, + )) + } + + pub fn reserve_read( + &self, + group: GroupHistory, + bytes: usize, + ) -> Result, ExecutionError> { + let mut state = self.state.lock().map_err(|_| unavailable())?; + self.reserve_bytes(&mut state, group, bytes) + } + + fn reserve_bytes( + &self, + state: &mut State, + group: GroupHistory, + bytes: usize, + ) -> Result, ExecutionError> { + let node = self.node.acquire(bytes).ok_or_else(overloaded)?; + // Reclaim metadata only, never authority or pending physical ownership. + state.groups.retain(|_, budget| budget.strong_count() != 0); + let group_budget = state + .groups + .get(&group) + .and_then(Weak::upgrade) + .unwrap_or_else(|| { + Budget::new(self.limits.group_bytes, self.limits.group_pending_cells) + }); + let group_permit = group_budget.acquire(bytes).ok_or_else(overloaded)?; + state.groups.insert(group, Arc::downgrade(&group_budget)); + Ok(Arc::new(Bytes { + _lease: self.lease.clone(), + _node: node, + _group: group_permit, + })) + } + + pub fn usage(&self) -> Result<(usize, usize, usize), ExecutionError> { + let cells = self.state.lock().map_err(|_| unavailable())?.cells.len(); + let (bytes, reservations) = self.node.usage(); + Ok((cells, bytes, reservations)) + } +} diff --git a/crates/cellar/durability/raft/host_sql/cell_groups/apply.rs b/crates/cellar/durability/raft/host_sql/cell_groups/apply.rs new file mode 100644 index 000000000..57b773c10 --- /dev/null +++ b/crates/cellar/durability/raft/host_sql/cell_groups/apply.rs @@ -0,0 +1,128 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Apply uses the existing workers' reserved progress lane, independently of SQL producers. + +use super::*; +use crate::raft::cell_groups::{ + CellGroupApplier, CellGroupApply, CellGroupRuntime, GroupManifest, PreparedGroupManifest, +}; +use sha2::{Digest, Sha256}; + +pub(in crate::raft::host_sql) struct ApplyRequest { + pub group: GroupHistory, + applier: CellGroupApplier, + work: CellGroupApply, + reply: oneshot::Sender>, + _slot: tokio::sync::OwnedSemaphorePermit, +} + +/// Retains physical preparation and its admission through delivery, retries and +/// owner completion. Successful submission is not a durable apply acknowledgement. +#[derive(Clone)] +pub struct CellGroupPreparedApply { + work: CellGroupApply, + prepared: PreparedGroupManifest, +} + +impl CellGroupPreparedApply { + pub fn manifest(&self) -> &GroupManifest { + self.prepared.marker() + } + + pub async fn complete(self, runtime: &CellGroupRuntime) -> Result<(), RuntimeError> { + runtime + .complete_apply( + self.prepared.marker().route().history, + self.work, + self.prepared, + ) + .await + } +} + +impl CellGroupSqlPool { + /// Runs image and manifest I/O on the existing fixed workers. A cancelled + /// waiter does not cancel physical work or release its ticket reservations. + pub async fn prepare_apply( + &self, + applier: &CellGroupApplier, + mut work: CellGroupApply, + ) -> Result { + self.admit()?; + if !applier.belongs_to_engine(&self.inner.engine_owner) { + return Err(ExecutionError::Invalid("apply pool engine")); + } + let bytes = applier.reservation_bytes(&work)?; + let reservation = self + .inner + .apply_admission + .acquire(bytes) + .ok_or_else(overloaded)?; + work.ticket.preparation = Some(Arc::new(reservation)); + let group = applier.history(); + let mut hash = Sha256::new(); + for id in [ + group.cluster.bytes(), + group.namespace.bytes(), + group.group.bytes(), + group.incarnation.bytes(), + ] { + hash.update(id); + } + let hash: [u8; 32] = hash.finalize().into(); + let index = u64::from_le_bytes( + hash[..8] + .try_into() + .map_err(|_| ExecutionError::Invalid("apply worker key"))?, + ) % self.inner.workers.len() as u64; + let worker = &self.inner.workers[index as usize]; + worker.domain.mono_ms().map_err(|_| unavailable())?; + let slot = worker + .completion_slots + .clone() + .try_acquire_owned() + .map_err(|_| overloaded())?; + self.admit()?; + let (reply, wait) = oneshot::channel(); + worker + .sender + .try_send(Envelope::GroupApply(Box::new(ApplyRequest { + group, + applier: applier.clone(), + work, + reply, + _slot: slot, + }))) + .map_err(|_| unavailable())?; + wait.await.map_err(|_| unavailable())? + } +} + +impl ApplyRequest { + pub fn execute(self, domain: &crate::asyncrt::DomainToken, stopped: &AtomicBool) { + let Self { + applier, + work, + reply, + _slot, + .. + } = self; + let result = if stopped.load(Ordering::Acquire) || domain.mono_ms().is_err() { + Err(unavailable()) + } else { + std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| applier.prepare(&work))) + .unwrap_or_else(|panic| { + Err(crate::raft::execution::storage( + crate::native_worker::panic_error(panic), + )) + }) + .and_then(|prepared| { + if stopped.load(Ordering::Acquire) || domain.mono_ms().is_err() { + return Err(unavailable()); + } + Ok(CellGroupPreparedApply { work, prepared }) + }) + }; + let _ = reply.send(result); + } +} diff --git a/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs b/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs new file mode 100644 index 000000000..2790aad50 --- /dev/null +++ b/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs @@ -0,0 +1,967 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Cell-history admission and commands on the existing fixed node SQL workers. + +mod admission; +mod apply; +pub(in crate::raft::host_sql) use apply::ApplyRequest; +pub use apply::CellGroupPreparedApply; +mod query; +pub(crate) use query::ReadRetention; +mod worker; +pub use query::{CellSqlEffectScan, CellSqlQueryResult, CellSqlReceiptLookup}; + +use super::{Budget, Budgets, Envelope, Permit, Pool, ResourceObserver, Worker}; +use crate::raft::cell_groups::{ + cell_image_history_digest, CellEnrollmentAuthority, CellEnrollmentState, + CellExecutionAuthority, CellReconciliationAuthority, EnrollmentRecord, +}; +use crate::raft::completion::RuntimeError; +use crate::raft::execution::{ + CellCommittedReceipt, CellEnrollmentAttempt, CellEnrollmentResolution, CellPreparedCommand, + CellPrivateEnrollment, CellPrivateExecutor, CellProposalRejection, CellReconciliation, + CellStageOutcome, CellSubmissionOutcome, CommandAttempt, ExecutionError, ExecutionLimits, + PrivateOptions, RootPrivateOptions, +}; +use crate::raft::runtime::Terminal; +use crate::raft::storage::ReplicaLog; +use crate::storage::CommandTransaction; +use admission::{Admission, Bytes, CellLease}; +use cellar_logic::cell_group::{CellHistory, GroupHistory}; +use cellar_logic::consensus::{CommandFailure, ProposalId}; +use std::sync::{ + atomic::{AtomicBool, Ordering}, + Arc, Mutex, +}; +use tokio::sync::{oneshot, Notify}; +pub(super) use worker::{Cells, Request}; + +#[derive(Clone, Copy, Debug, serde::Serialize)] +pub struct CellSqlLimits { + pub workers: usize, + pub queue_per_worker: usize, + pub max_pending_cells: usize, + pub group_pending_cells: usize, + /// Each of the SQL producer and independent apply budgets has this ceiling. + /// Configure their combined envelope as twice this value. + pub node_bytes: usize, + pub group_bytes: usize, +} + +impl CellSqlLimits { + fn validate(self) -> Result<(), ExecutionError> { + if !(1..=256).contains(&self.workers) + || !(1..=4096).contains(&self.queue_per_worker) + || !(1..=65_536).contains(&self.max_pending_cells) + || self.group_pending_cells == 0 + || self.group_pending_cells > self.max_pending_cells + || self.group_bytes == 0 + || self.group_bytes > self.node_bytes + { + return Err(ExecutionError::Invalid("cell SQL pool limits")); + } + Ok(()) + } +} + +/// Extent configuration without a live private workspace. The admitted worker +/// creates and destroys that physical owner; rejected/queued requests do no I/O. +#[derive(Clone)] +pub struct CellSqlRootOptions { + pub host: cellar_ltx::LtxHost, + pub sqlite_vfs: Option, + pub images: cellar_ltx::internal::NodeExtentStore, + pub query_services: cellar_ltx::generation::ReadServices, + pub application_digest: [u8; 32], + pub limits: ExecutionLimits, + pub cache_directory: std::path::PathBuf, + pub cache_pool: cellar_ltx::stream_snapshot::ScratchPool, + pub cache: cellar_ltx::internal::CacheLimits, +} + +#[derive(Clone)] +pub enum CellSqlOptions { + Materialized(PrivateOptions), + Extents(CellSqlRootOptions), +} + +impl CellSqlOptions { + pub(crate) fn limits(&self) -> ExecutionLimits { + match self { + Self::Materialized(o) => o.limits, + Self::Extents(o) => o.limits, + } + } + + fn application_digest(&self) -> [u8; 32] { + match self { + Self::Materialized(o) => o.application_digest, + Self::Extents(o) => o.application_digest, + } + } + + fn open_enrollment( + self, + authority: CellEnrollmentAuthority, + ) -> Result { + match self { + Self::Materialized(o) => CellPrivateEnrollment::open(o, authority), + Self::Extents(o) => { + CellPrivateEnrollment::open_rooted(o.prepare(authority.state().cell())?, authority) + } + } + } + + fn open( + self, + authority: CellExecutionAuthority, + ) -> Result { + match self { + Self::Materialized(o) => CellPrivateExecutor::open(o, authority), + Self::Extents(o) => CellPrivateExecutor::open_rooted( + o.prepare(&authority.cell().version.history)?, + authority, + ), + } + } +} + +impl CellSqlRootOptions { + fn prepare(self, cell: &CellHistory) -> Result { + let workspace = self + .images + .create_workspace( + cell_image_history_digest(cell), + RootPrivateOptions::workspace_bounds(self.limits)?, + ) + .map_err(crate::raft::execution::storage)?; + Ok(RootPrivateOptions { + workspace, + host: self.host, + sqlite_vfs: self.sqlite_vfs, + images: self.images, + query_services: self.query_services, + application_digest: self.application_digest, + limits: self.limits, + cache_directory: self.cache_directory, + cache_pool: self.cache_pool, + cache: self.cache, + }) + } +} + +pub struct CellSqlEnrollment { + record: EnrollmentRecord, + _retention: OutputRetention, + s3: Option>, +} + +impl CellSqlEnrollment { + pub fn record(&self) -> &EnrollmentRecord { + &self.record + } + + /// Achieved output durability, never inferred from creation metadata alone. + pub fn durability(&self) -> cellar_logic::consensus::Durability { + if self.s3.is_some() { + cellar_logic::consensus::Durability::S3 + } else { + cellar_logic::consensus::Durability::Quorum + } + } + + pub(crate) fn retain_for_archive( + mut self: Box, + permit: Arc>, + ) -> Box { + self._retention = OutputRetention::Archive { _permit: permit }; + self + } + + pub(crate) fn with_archive_proof( + mut self, + proof: Box, + ) -> Self { + self.s3 = Some(proof); + self + } +} + +pub enum CellSqlEnrollmentResolution { + Committed(Box), + NotCommitted, + Conflict, +} + +/// Clones retain both node and group admission through delivery and submission. +#[derive(Clone)] +pub struct CellSqlPrepared { + prepared: CellPreparedCommand, + _bytes: Arc, +} + +impl CellSqlPrepared { + pub(crate) fn cancel_before_submission( + &self, + ) -> Result, RuntimeError> { + Ok(self + .prepared + .cancel_before_submission()? + .map(|proof| CellSqlRejection { + proof, + _bytes: self._bytes.clone(), + })) + } + + pub async fn submit( + &self, + runtime: &crate::raft::cell_groups::CellGroupRuntime, + ) -> Result { + match runtime + .submit_retained(self.prepared.clone(), Some(self._bytes.clone())) + .await? + { + CellSubmissionOutcome::Accepted(accepted) => Ok(CellSqlSubmission::Accepted(accepted)), + CellSubmissionOutcome::Rejected(proof) => { + Ok(CellSqlSubmission::Rejected(CellSqlRejection { + proof, + _bytes: self._bytes.clone(), + })) + } + } + } + + pub fn rejection(&self) -> Result, RuntimeError> { + Ok(self.prepared.rejection()?.map(|proof| CellSqlRejection { + proof, + _bytes: self._bytes.clone(), + })) + } + + pub fn term(&self) -> cellar_logic::consensus::Term { + self.prepared.term() + } + + pub fn bytes(&self) -> &[u8] { + self.prepared.bytes() + } + + pub fn proposal(&self) -> ProposalId { + self.prepared.proposal() + } +} + +pub enum CellSqlSubmission { + Accepted(crate::raft::cell_groups::CellGroupProposal), + Rejected(CellSqlRejection), +} + +/// Retains the original SQL output budget through delivery and lost responses. +#[derive(Clone)] +pub struct CellSqlRejection { + proof: CellProposalRejection, + _bytes: Arc, +} + +pub struct CellSqlReceipt { + receipt: CellCommittedReceipt, + _retention: OutputRetention, +} + +enum OutputRetention { + Sql { + _bytes: Arc, + }, + Archive { + _permit: Arc>, + }, +} + +impl CellSqlReceipt { + /// Only a resolved output can transfer its bytes to an independently admitted + /// archive owner. Acquire that reservation before releasing SQL retention. + pub(crate) fn retain_for_archive( + mut self: Box, + permit: Arc>, + ) -> Box { + self._retention = OutputRetention::Archive { _permit: permit }; + self + } + + pub fn receipt(&self) -> &CellCommittedReceipt { + &self.receipt + } +} + +// A stage error after a received worker reply cannot hide an installed producer: +// Stage and Initialize insert pending ownership only on successful Prepared branches. +// A lost reply is deliberately different and requires recovery/joined shutdown. +pub(crate) struct CellSqlStageFailure { + pub error: ExecutionError, + pub unresolved: bool, +} + +impl CellSqlStageFailure { + fn known(error: ExecutionError) -> Self { + Self { + error, + unresolved: false, + } + } +} + +type SqlReply = Result; + +pub enum CellSqlOutcome { + Prepared(CellSqlPrepared), + Receipt(Box), + Submitted, + Reconciled(CellSqlReconciliation), + Enrollment(Box), + EnrollmentResolved(CellSqlEnrollmentResolution), + Rejected, + Queried(CellSqlQueryResult), + ReceiptLookedUp(CellSqlReceiptLookup), + EffectsScanned(CellSqlEffectScan), +} + +pub enum CellSqlReconciliation { + Committed(Box), + NotCommitted, + Refused(CommandFailure), +} + +#[derive(Clone, Copy, Debug, serde::Serialize)] +pub struct CellSqlUsage { + pub limits: CellSqlLimits, + pub closing: bool, + /// Independent progress budget; not included in SQL producer retention. + pub apply_retained_bytes: usize, + pub apply_retained_operations: usize, + pub active_cells: usize, + pub retained_bytes: usize, + pub retained_operations: usize, + pub running_workers: usize, + pub worker_capacity: usize, + pub queued_or_running: usize, + pub completion_reservations: usize, +} + +#[derive(Clone, Copy)] +pub(crate) enum CellSqlStageMode { + Lookup, + Verification, + InvokeBefore(u64), +} + +impl CellSqlStageMode { + fn verification(allow_new: bool) -> Self { + if allow_new { + Self::Verification + } else { + Self::Lookup + } + } + + fn allows_new(self) -> bool { + !matches!(self, Self::Lookup) + } + + fn expired(self, now: u64) -> bool { + matches!(self, Self::InvokeBefore(deadline) if now >= deadline) + } +} + +#[derive(Clone)] +pub struct CellGroupSqlPool { + inner: Arc, +} + +pub(crate) struct WeakCellGroupSqlPool(std::sync::Weak); + +impl WeakCellGroupSqlPool { + pub(crate) fn upgrade(&self) -> Option { + self.0.upgrade().map(|inner| CellGroupSqlPool { inner }) + } +} + +impl CellGroupSqlPool { + pub(crate) fn downgrade(&self) -> WeakCellGroupSqlPool { + WeakCellGroupSqlPool(Arc::downgrade(&self.inner)) + } +} + +struct Handle { + engine_owner: Arc<()>, + workers: Vec, + resources: ResourceObserver, + admission: Arc, + apply_admission: Arc, + closing: Arc, + stop: Arc, + terminal: Arc, +} + +impl Drop for Handle { + fn drop(&mut self) { + self.closing.store(true, Ordering::Release); + self.stop.notify_one(); + } +} + +fn unavailable() -> ExecutionError { + ExecutionError::Refused(CommandFailure::Unavailable) +} + +fn overloaded() -> ExecutionError { + ExecutionError::Refused(CommandFailure::Overloaded) +} + +impl CellGroupSqlPool { + pub(crate) fn belongs_to_engine(&self, owner: &Arc<()>) -> bool { + Arc::ptr_eq(&self.inner.engine_owner, owner) + } + + pub(crate) fn uses_domain(&self, domain: &crate::asyncrt::DomainToken) -> bool { + self.inner.workers[0].domain.same_owner(domain) + } + + pub fn start(log: &ReplicaLog, limits: CellSqlLimits) -> Result { + limits.validate()?; + log.cell_groups().map_err(crate::raft::execution::storage)?; + let lease = log + .claim_sql_pool() + .map_err(crate::raft::execution::storage)?; + let domain = log.domain(); + let mut pool = Pool::start( + domain.clone(), + limits.workers, + limits.queue_per_worker, + limits.max_pending_cells, + Budgets { + business: Budget::new(limits.node_bytes, limits.max_pending_cells), + archive_capture: Budget::new(1, 1), + query_pins: Budget::new(1, 1), + }, + log.performance(), + )?; + let workers = (0..limits.workers) + .map(|n| pool.worker(n)) + .collect::, _>>()?; + let resources = pool.resource_observer(); + let closing = pool.stopped.clone(); + let stop = Arc::new(Notify::new()); + let terminal = Arc::new(Terminal::new()); + let inner = Arc::new(Handle { + engine_owner: log.owner(), + workers, + resources, + admission: Admission::new(limits, lease.clone()), + apply_admission: Budget::new(limits.node_bytes, log.limits().max_groups), + closing: closing.clone(), + stop: stop.clone(), + terminal: terminal.clone(), + }); + let cleanup_domain = domain.clone(); + // Cleanup owns the join handles, so cancelling a close waiter cannot + // release physical workers. Domain shutdown also joins this cleanup. + domain.spawn_cleanup(async move { + struct Complete(Arc); + + impl Drop for Complete { + fn drop(&mut self) { self.0.complete(Err(RuntimeError::Closed)); } + } + let _complete = Complete(terminal.clone()); + let _physical_lease = lease; + crate::asyncrt::select! { _ = stop.notified() => (), _ = cleanup_domain.wait_for_work() => () } + closing.store(true, Ordering::Release); + let result = pool.join().await.map_err(|e| RuntimeError::Protocol(e.to_string())); + drop(pool); + drop(_physical_lease); + terminal.complete(result); + }).map_err(|_| unavailable())?; + Ok(Self { inner }) + } + + pub fn usage(&self) -> Result { + let (active_cells, retained_bytes, retained_operations) = self.inner.admission.usage()?; + let resources = self.inner.resources.capture().ok_or_else(unavailable)?; + let (apply_retained_bytes, apply_retained_operations) = self.inner.apply_admission.usage(); + Ok(CellSqlUsage { + limits: self.inner.admission.limits(), + closing: self.inner.closing.load(Ordering::Acquire), + apply_retained_bytes, + apply_retained_operations, + active_cells, + retained_bytes, + retained_operations, + running_workers: resources.running_workers, + worker_capacity: resources.worker_capacity, + queued_or_running: resources.work_reservations, + completion_reservations: self + .inner + .workers + .iter() + .map(|w| 1 - w.completion_slots.available_permits()) + .sum(), + }) + } + + fn admit(&self) -> Result<(), ExecutionError> { + if self.inner.closing.load(Ordering::Acquire) { + return Err(unavailable()); + } + Ok(()) + } + + async fn call(&self, cell: CellHistory, action: worker::Action) -> SqlReply { + self.send(cell, action)?.await.map_err(|_| unavailable())? + } + + fn send( + &self, + cell: CellHistory, + action: worker::Action, + ) -> Result, ExecutionError> { + self.admit()?; + let hash = cell_image_history_digest(&cell); + let index = u64::from_le_bytes( + hash[..8] + .try_into() + .map_err(|_| ExecutionError::Invalid("cell worker key"))?, + ) % self.inner.workers.len() as u64; + let worker = &self.inner.workers[index as usize]; + worker.domain.mono_ms().map_err(|_| unavailable())?; + let slots = if action.progress_priority() { + &worker.completion_slots + } else { + &worker.slots + }; + let slot = slots + .clone() + .try_acquire_owned() + .map_err(|error| match error { + tokio::sync::TryAcquireError::Closed => unavailable(), + tokio::sync::TryAcquireError::NoPermits => overloaded(), + })?; + self.admit()?; + let (reply, wait) = oneshot::channel(); + worker + .sender + .try_send(Envelope::Shared(Request { + cell, + action, + reply, + _slot: slot, + queued: worker + .performance + .timer(crate::raft::performance::Operation::SqlQueue), + })) + .map_err(|_| unavailable())?; + Ok(wait) + } + + /// The owning worker retains the callback and admission even if this waiter + /// is cancelled. Caller-provided callbacks are the verification boundary; + /// the production host must dispatch registered operations here. + pub async fn stage( + &self, + options: CellSqlOptions, + authority: CellExecutionAuthority, + attempt: CommandAttempt, + allow_new: bool, + callback: impl FnOnce(&CommandTransaction<'_>) -> anyhow::Result> + Send + 'static, + ) -> Result { + self.stage_callback( + options, + authority, + attempt, + allow_new, + worker::Callback::Verification(Box::new(callback)), + ) + .await + } + + /// Dispatch an authorized namespace's compiled class on the same fixed + /// workers and receipt path. The class is checked against the selected route + /// schema; contract, input digest and inbox identity are checked on the worker + /// before receipt lookup. Cancellation retains the same physical ownership. + pub async fn stage_registered( + &self, + options: CellSqlOptions, + authority: CellExecutionAuthority, + attempt: CommandAttempt, + class: Arc, + input: Vec, + allow_new: bool, + ) -> Result { + self.stage_registered_owned( + options, + authority, + attempt, + class, + input, + CellSqlStageMode::verification(allow_new), + ) + .await + .map_err(|failure| failure.error) + } + + pub(crate) async fn stage_registered_owned( + &self, + options: CellSqlOptions, + authority: CellExecutionAuthority, + attempt: CommandAttempt, + class: Arc, + input: Vec, + mode: CellSqlStageMode, + ) -> Result { + self.stage_compiled_owned( + options, + authority, + attempt, + mode, + worker::Callback::Registered { + class, + input: input.into_boxed_slice(), + system: false, + }, + ) + .await + } + + pub(crate) async fn stage_system_owned( + &self, + options: CellSqlOptions, + authority: CellExecutionAuthority, + attempt: CommandAttempt, + class: Arc, + input: Vec, + mode: CellSqlStageMode, + ) -> Result { + self.stage_compiled_owned( + options, + authority, + attempt, + mode, + worker::Callback::Registered { + class, + input: input.into_boxed_slice(), + system: true, + }, + ) + .await + } + + async fn stage_compiled_owned( + &self, + options: CellSqlOptions, + authority: CellExecutionAuthority, + attempt: CommandAttempt, + mode: CellSqlStageMode, + callback: worker::Callback, + ) -> Result { + let worker::Callback::Registered { input, .. } = &callback else { + return Err(CellSqlStageFailure::known(ExecutionError::Invalid( + "compiled command callback", + ))); + }; + if input.len() > options.limits().envelope.max_payload_bytes { + return Err(CellSqlStageFailure::known(overloaded())); + } + self.stage_owned(options, authority, attempt, mode, callback) + .await + } + + async fn stage_callback( + &self, + options: CellSqlOptions, + authority: CellExecutionAuthority, + attempt: CommandAttempt, + allow_new: bool, + callback: worker::Callback, + ) -> SqlReply { + self.stage_owned( + options, + authority, + attempt, + CellSqlStageMode::verification(allow_new), + callback, + ) + .await + .map_err(|failure| failure.error) + } + + async fn stage_owned( + &self, + options: CellSqlOptions, + authority: CellExecutionAuthority, + attempt: CommandAttempt, + mode: CellSqlStageMode, + callback: worker::Callback, + ) -> Result { + let (cell, action) = self + .stage_request(options, authority, attempt, mode, callback) + .map_err(CellSqlStageFailure::known)?; + let wait = self + .send(cell, action) + .map_err(CellSqlStageFailure::known)?; + wait.await + .map_err(|_| CellSqlStageFailure { + error: unavailable(), + unresolved: true, + })? + .map_err(CellSqlStageFailure::known) + } + + fn stage_request( + &self, + options: CellSqlOptions, + authority: CellExecutionAuthority, + attempt: CommandAttempt, + mode: CellSqlStageMode, + callback: worker::Callback, + ) -> Result<(CellHistory, worker::Action), ExecutionError> { + self.admit()?; + if !authority.belongs_to_engine(&self.inner.engine_owner) { + return Err(ExecutionError::Invalid("cell SQL engine owner")); + } + if attempt.request.operation.is_empty() || attempt.request.operation.len() > 128 { + return Err(ExecutionError::Invalid("command operation")); + } + let limits = options.limits(); + limits.validate()?; + let input_bytes = match &callback { + worker::Callback::Verification(_) => 0, + worker::Callback::Registered { input, .. } => input.len(), + }; + // Bound simultaneous encoded capture, entry, preview verification and + // result buffers before queueing or opening SQLite. Disk/workspace + // reservation is separately enforced by the chosen image owner. + let bytes = limits + .envelope + .max_payload_bytes + .checked_mul(4) + .and_then(|n| n.checked_add(input_bytes)) + .and_then(|n| n.checked_add(limits.max_result_bytes)) + .and_then(|n| n.checked_add(4096)) + .ok_or_else(overloaded)?; + let cell = authority.cell().version.history.clone(); + let (lease, bytes) = + self.inner + .admission + .reserve(&cell, authority.route().history, bytes)?; + Ok(( + cell, + worker::Action::Stage(Box::new(worker::Stage { + options, + authority, + attempt, + mode, + callback, + lease, + bytes, + })), + )) + } + + /// Initialize one logical cell on the existing blocking pool. The creation + /// ID is immutable retry identity; a committed retry returns its original + /// enrollment record without opening SQLite or rerunning the callback. + pub async fn initialize( + &self, + options: CellSqlOptions, + authority: CellEnrollmentAuthority, + attempt: CellEnrollmentAttempt, + initialize: impl FnOnce(&CommandTransaction<'_>) -> anyhow::Result<()> + Send + 'static, + ) -> Result { + self.initialize_owned(options, authority, attempt, initialize) + .await + .map_err(|failure| failure.error) + } + + pub(crate) async fn initialize_owned( + &self, + options: CellSqlOptions, + authority: CellEnrollmentAuthority, + attempt: CellEnrollmentAttempt, + initialize: impl FnOnce(&CommandTransaction<'_>) -> anyhow::Result<()> + Send + 'static, + ) -> Result { + let (cell, action) = self + .initialize_request(options, authority, attempt, initialize) + .map_err(CellSqlStageFailure::known)?; + let wait = self + .send(cell, action) + .map_err(CellSqlStageFailure::known)?; + wait.await + .map_err(|_| CellSqlStageFailure { + error: unavailable(), + unresolved: true, + })? + .map_err(CellSqlStageFailure::known) + } + + fn initialize_request( + &self, + options: CellSqlOptions, + authority: CellEnrollmentAuthority, + attempt: CellEnrollmentAttempt, + initialize: impl FnOnce(&CommandTransaction<'_>) -> anyhow::Result<()> + Send + 'static, + ) -> Result<(CellHistory, worker::Action), ExecutionError> { + self.admit()?; + if !authority + .state() + .belongs_to_engine(&self.inner.engine_owner) + { + return Err(ExecutionError::Invalid("cell SQL engine owner")); + } + authority + .check() + .map_err(crate::raft::execution::authority_failure)?; + if options.application_digest() != authority.state().route().schema_digest { + return Err(ExecutionError::Invalid("enrollment schema")); + } + let limits = options.limits(); + limits.validate()?; + let bytes = if authority.state().record().is_some() { + 4096 + } else { + limits + .envelope + .max_payload_bytes + .checked_mul(4) + .and_then(|n| n.checked_add(4096)) + .ok_or_else(overloaded)? + }; + let cell = authority.state().cell().clone(); + let (lease, bytes) = + self.inner + .admission + .reserve(&cell, authority.state().route().history, bytes)?; + Ok(( + cell, + worker::Action::Initialize(Box::new(worker::Initialize { + options, + authority, + attempt, + callback: Box::new(initialize), + lease, + bytes, + })), + )) + } + + pub async fn resolve_enrollment( + &self, + creation: ProposalId, + state: CellEnrollmentState, + ) -> Result { + if !state.belongs_to_engine(&self.inner.engine_owner) { + return Err(ExecutionError::Invalid("cell SQL engine owner")); + } + let cell = state.cell().clone(); + match self + .call( + cell, + worker::Action::ResolveEnrollment(creation, Box::new(state)), + ) + .await? + { + CellSqlOutcome::EnrollmentResolved(resolution) => Ok(resolution), + _ => Err(ExecutionError::Invalid("enrollment resolution response")), + } + } + + /// Recover a lost stage response without another callback or producer. + pub async fn resume( + &self, + cell: CellHistory, + proposal: ProposalId, + ) -> Result { + match self.call(cell, worker::Action::Resume(proposal)).await? { + CellSqlOutcome::Prepared(prepared) => Ok(prepared), + _ => Err(ExecutionError::Invalid("cell SQL resume response")), + } + } + + pub async fn mark_submitted( + &self, + cell: CellHistory, + proposal: ProposalId, + ) -> Result<(), ExecutionError> { + match self.call(cell, worker::Action::Submit(proposal)).await? { + CellSqlOutcome::Submitted => Ok(()), + _ => Err(ExecutionError::Invalid("cell SQL submit response")), + } + } + + /// Close the private writer only for this exact, irreversibly sealed + /// preparation. A generic error or another producer's proof cannot do this. + pub async fn reject( + &self, + cell: CellHistory, + rejection: CellSqlRejection, + ) -> Result<(), ExecutionError> { + if !rejection.proof.belongs_to_engine(&self.inner.engine_owner) { + return Err(ExecutionError::Invalid("cell rejection engine")); + } + match self + .call(cell, worker::Action::Reject(Box::new(rejection))) + .await? + { + CellSqlOutcome::Rejected => Ok(()), + _ => Err(ExecutionError::Invalid("cell rejection response")), + } + } + + pub async fn finish( + &self, + authority: CellExecutionAuthority, + ) -> Result { + if !authority.belongs_to_engine(&self.inner.engine_owner) { + return Err(ExecutionError::Invalid("cell SQL engine owner")); + } + let cell = authority.cell().version.history.clone(); + match self + .call(cell, worker::Action::Finish(Box::new(authority))) + .await? + { + CellSqlOutcome::Receipt(receipt) => Ok(*receipt), + _ => Err(ExecutionError::Invalid("cell SQL finish response")), + } + } + + /// Release a producer only after receipt resolution or a newer applied + /// term excludes its proposal. Cancellation leaves resolution on the worker; + /// a lost response must be reconciled against receipts, never inferred failed. + pub async fn reconcile( + &self, + proposal: ProposalId, + authority: CellReconciliationAuthority, + ) -> Result { + if !authority.belongs_to_engine(&self.inner.engine_owner) { + return Err(ExecutionError::Invalid("cell SQL engine owner")); + } + let cell = authority.cell().version.history.clone(); + match self + .call( + cell, + worker::Action::Reconcile(proposal, Box::new(authority)), + ) + .await? + { + CellSqlOutcome::Reconciled(resolution) => Ok(resolution), + _ => Err(ExecutionError::Invalid("cell SQL reconciliation response")), + } + } + + pub async fn close(&self) -> Result<(), ExecutionError> { + self.inner.closing.store(true, Ordering::Release); + self.inner.stop.notify_one(); + self.inner + .terminal + .wait() + .await + .map_err(crate::raft::execution::storage) + } +} diff --git a/crates/cellar/durability/raft/host_sql/cell_groups/query.rs b/crates/cellar/durability/raft/host_sql/cell_groups/query.rs new file mode 100644 index 000000000..e9598bf78 --- /dev/null +++ b/crates/cellar/durability/raft/host_sql/cell_groups/query.rs @@ -0,0 +1,563 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Immutable cell queries on the existing node SQL workers. + +use super::*; +use crate::raft::cell_groups::{CellImagePin, CellReadAuthority}; +use cellar_logic::cell_group::CellVersion; +use std::panic::AssertUnwindSafe; + +type VerificationCallback = Box< + dyn FnOnce(&crate::storage::ImmutableQuery, &CellVersion) -> anyhow::Result> + Send, +>; + +enum Callback { + Effects { + class: Arc, + now: u64, + reservation: crate::raft::budget::Permit, + }, + Receipt { + class: Arc, + request: crate::raft::execution::CommandRequest, + input: Box<[u8]>, + }, + Verification(VerificationCallback), + Registered { + class: Arc, + contract: crate::application::consensus::OperationContract, + input: Box<[u8]>, + }, +} + +/// Keeps upstream admission alive on the physical reader after async timeout. +/// The SQL owner does not interpret or release that admission early. +pub(crate) struct ReadRetention { + _owner: Arc, +} + +impl ReadRetention { + pub(crate) fn new(owner: Arc) -> Self { + Self { _owner: owner } + } +} + +pub(super) struct Query { + _retention: Option, + // SQL can outlive both its response deadline and the controller's reply. + // Node drain must still observe the physical callback until it exits. + _work: crate::host_services::ApplicationWorkGuard, + options: CellSqlOptions, + authority: CellReadAuthority, + callback: Callback, + cancelled: Arc, + closing: Arc, + bytes: Arc, +} + +/// Opaque committed effect observation, charged to the node scheduler until +/// the complete claim/delivery/settlement turn releases it. +pub struct CellSqlEffectScan { + scan: Box, + _bytes: Arc, +} + +impl CellSqlEffectScan { + /// SQL discovery has joined. Its payload is already charged to the effect + /// metadata permit, so release the read slot before admitting the private + /// command. A group with one SQL slot must not recursively acquire it. + pub(crate) fn into_scan(self) -> Box { + self.scan + } +} + +pub struct CellSqlQueryResult { + result: Vec, + replica: cellar_logic::consensus::ReplicaIdentity, + observation: crate::raft::version::CellVersionObservation, + _bytes: Arc, +} + +impl CellSqlQueryResult { + pub fn replica( + &self, + ) -> &cellar_logic::consensus::ReplicaIdentity { + &self.replica + } + + pub fn result(&self) -> &[u8] { + &self.result + } + + pub fn observation(&self) -> &crate::raft::version::CellVersionObservation { + &self.observation + } + + pub fn version(&self) -> &CellVersion { + &self.observation.version + } +} + +/// None means the selected image has no retained receipt, never non-commit. +pub struct CellSqlReceiptLookup { + receipt: Option, + observation: crate::raft::version::CellVersionObservation, + _retention: OutputRetention, +} + +impl CellSqlReceiptLookup { + pub(crate) fn retain_for_archive( + mut self: Box, + permit: Arc>, + ) -> Box { + self._retention = OutputRetention::Archive { _permit: permit }; + self + } + + pub(crate) fn select_archived( + &mut self, + receipt: Option, + observation: crate::raft::version::CellVersionObservation, + ) { + self.receipt = receipt; + self.observation = observation; + } + + pub fn receipt(&self) -> Option<&crate::raft::execution::StoredReceipt> { + self.receipt.as_ref() + } + + pub fn observation(&self) -> &crate::raft::version::CellVersionObservation { + &self.observation + } +} + +enum Output { + Effects(Box), + Query(Vec), + Receipt(Option), +} + +impl Output { + fn retained_bytes(&self) -> usize { + match self { + // The scheduler's metadata permit accounts for scan payloads and + // remains owned by EffectScan, independently of SQL read admission. + Self::Effects(_) => 0, + Self::Query(bytes) => bytes.len(), + Self::Receipt(receipt) => receipt.as_ref().map_or(0, |r| r.result().len()), + } + } +} + +struct Cancel(Arc); + +impl Drop for Cancel { + fn drop(&mut self) { + self.0.store(true, Ordering::Release); + } +} + +impl CellGroupSqlPool { + pub(crate) async fn scan_effects( + &self, + options: CellSqlOptions, + authority: CellReadAuthority, + class: Arc, + now: u64, + reservation: crate::raft::budget::Permit, + ) -> Result { + if authority.barrier().is_none() + || !class.has_system_commands() + || class.schema_digest() != authority.route().schema_digest + { + return Err(ExecutionError::Invalid("effect scan authority/class")); + } + match self + .query_callback( + options, + authority, + Callback::Effects { + class, + now, + reservation, + }, + ) + .await? + { + CellSqlOutcome::EffectsScanned(output) => Ok(output), + _ => Err(ExecutionError::Invalid("effect scan reply")), + } + } + + /// Consume an invocation's coherent pin. Queries can coexist with an + /// unresolved private writer but read only the selected committed image. + /// The registered dispatcher must enforce operation contracts before here. + pub async fn query( + &self, + options: CellSqlOptions, + authority: CellReadAuthority, + callback: impl FnOnce(&crate::storage::ImmutableQuery, &CellVersion) -> anyhow::Result> + + Send + + 'static, + ) -> Result { + match self + .query_callback( + options, + authority, + Callback::Verification(Box::new(callback)), + ) + .await? + { + CellSqlOutcome::Queried(output) => Ok(output), + _ => Err(ExecutionError::Invalid("cell query reply")), + } + } + + /// Invoke the namespace's compiled query against the exact selected cell + /// image and version witness. The existing query owner enforces the original + /// deadline, cancellation, read-only SQL, output budget and joined cleanup. + pub async fn query_registered( + &self, + options: CellSqlOptions, + authority: CellReadAuthority, + class: Arc, + contract: crate::application::consensus::OperationContract, + input: Vec, + ) -> Result { + self.query_registered_retained(options, authority, class, contract, input, None) + .await + } + + pub(crate) async fn query_registered_retained( + &self, + options: CellSqlOptions, + authority: CellReadAuthority, + class: Arc, + mut contract: crate::application::consensus::OperationContract, + input: Vec, + retention: Option, + ) -> Result { + if contract.operation.len() > 128 { + return Err(ExecutionError::Registration( + crate::application::consensus::RegistrationError::Invalid("class/operation name"), + )); + } + contract.operation = contract.operation.into_boxed_str().into_string(); + if input.len() > options.limits().envelope.max_payload_bytes { + return Err(overloaded()); + } + match self + .query_callback_retained( + options, + authority, + Callback::Registered { + class, + contract, + input: input.into_boxed_slice(), + }, + retention, + ) + .await? + { + CellSqlOutcome::Queried(output) => Ok(output), + _ => Err(ExecutionError::Invalid("cell query reply")), + } + } + + /// Inspect a committed image without a private writer or command callback. + /// Contract, payload digest and authenticated command identity are checked + /// before returning even a duplicate receipt. Missing state is not a proof. + pub async fn lookup_receipt_registered( + &self, + options: CellSqlOptions, + authority: CellReadAuthority, + class: Arc, + mut request: crate::raft::execution::CommandRequest, + input: Vec, + ) -> Result { + if authority.barrier().is_none() { + return Err(ExecutionError::Invalid( + "receipt lookup requires quorum read", + )); + } + if request.operation.len() > 128 + || input.len() > options.limits().envelope.max_payload_bytes + { + return Err(overloaded()); + } + request.operation = request.operation.into_boxed_str().into_string(); + match self + .query_callback( + options, + authority, + Callback::Receipt { + class, + request, + input: input.into_boxed_slice(), + }, + ) + .await? + { + CellSqlOutcome::ReceiptLookedUp(output) => Ok(output), + _ => Err(ExecutionError::Invalid("cell receipt lookup reply")), + } + } + + async fn query_callback( + &self, + options: CellSqlOptions, + authority: CellReadAuthority, + callback: Callback, + ) -> Result { + self.query_callback_retained(options, authority, callback, None) + .await + } + + async fn query_callback_retained( + &self, + options: CellSqlOptions, + authority: CellReadAuthority, + callback: Callback, + retention: Option, + ) -> Result { + self.admit()?; + let domain = &self.inner.workers[0].domain; + if !authority.belongs_to_engine(&self.inner.engine_owner) + || options.application_digest() != authority.route().schema_digest + { + return Err(ExecutionError::Invalid("cell query engine/schema")); + } + authority + .check(domain.mono_ms().map_err(|_| unavailable())?) + .map_err(crate::raft::execution::authority_failure)?; + options.limits().validate()?; + let input_bytes = match &callback { + Callback::Verification(_) | Callback::Effects { .. } => 0, + Callback::Registered { input, .. } | Callback::Receipt { input, .. } => input.len(), + }; + let bytes = self.inner.admission.reserve_read( + authority.route().history, + options + .limits() + .max_result_bytes + .checked_add(4096) + .and_then(|n| n.checked_add(input_bytes)) + .ok_or_else(overloaded)?, + )?; + let cell = authority.cell().version.history.clone(); + let deadline = authority.deadline(); + let cancelled = Arc::new(AtomicBool::new(false)); + let _cancel = Cancel(cancelled.clone()); + let work = domain + .run(async { + crate::asyncrt::services() + .application_work_registry() + .enter() + }) + .await + .map_err(|_| unavailable())?; + let query = Query { + _retention: retention, + _work: work, + options, + authority, + callback, + cancelled, + closing: self.inner.closing.clone(), + bytes, + }; + let timer = domain.sleep_until(deadline).map_err(|_| unavailable())?; + crate::asyncrt::select! { + result = self.call(cell, worker::Action::Query(Box::new(query))) => { + match result? { + output @ (CellSqlOutcome::Queried(_) | CellSqlOutcome::ReceiptLookedUp(_) | CellSqlOutcome::EffectsScanned(_)) => { + if domain.mono_ms().map_err(|_| unavailable())? >= deadline { + return Err(crate::raft::execution::storage(RuntimeError::DeadlineExceeded)); + } + Ok(output) + } + _ => Err(ExecutionError::Invalid("cell query reply")), + } + } + _ = timer => Err(crate::raft::execution::storage(RuntimeError::DeadlineExceeded)), + } + } +} + +// The SQLite wrapper catches progress-hook panics. This wrapper allows the +// domain clock to be captured without claiming that arbitrary clock backends +// themselves implement unwind-safety; a clock failure interrupts the query. +struct QueryClock(AssertUnwindSafe); + +impl QueryClock { + fn expired(&self, deadline: u64) -> bool { + self.0.mono_ms().map_or(true, |now| now >= deadline) + } +} + +pub(super) fn run( + mut query: Query, + domain: &crate::asyncrt::DomainToken, +) -> Result { + let check = |query: &Query| { + if query.cancelled.load(Ordering::Acquire) || query.closing.load(Ordering::Acquire) { + return Err(unavailable()); + } + query + .authority + .check(domain.mono_ms().map_err(|_| unavailable())?) + .map_err(crate::raft::execution::authority_failure) + }; + check(&query)?; + if let Callback::Registered { + class, + contract, + input, + } = &query.callback + { + crate::raft::execution::validate_registered_query( + class, + contract, + input, + query.authority.route().schema_digest, + query.options.limits(), + )?; + } + let database = match (&query.options, query.authority.image()) { + (CellSqlOptions::Materialized(options), CellImagePin::Materialized(pin)) => { + crate::storage::open_immutable_generation( + pin.clone(), + crate::storage::ImmutableQueryOptions { + services: options.query_services.clone(), + sqlite_vec: false, + }, + ) + } + (CellSqlOptions::Extents(options), CellImagePin::Extents(pin)) => { + let image = pin + .view() + .image( + options.cache_directory.clone(), + options.cache_pool.clone(), + options.cache, + ) + .map_err(crate::raft::execution::storage)?; + crate::storage::open_immutable_extent_image( + image, + crate::storage::ImmutableQueryOptions { + services: options.query_services.clone(), + sqlite_vec: false, + }, + ) + } + _ => return Err(ExecutionError::Invalid("cell query image storage")), + } + .map_err(crate::raft::execution::storage)?; + let clock = QueryClock(AssertUnwindSafe(domain.clone())); + let deadline = query.authority.deadline(); + let cancelled = query.cancelled.clone(); + let closing = query.closing.clone(); + database + .connection() + .progress_handler( + 1000, + Some(move || { + cancelled.load(Ordering::Acquire) + || closing.load(Ordering::Acquire) + || clock.expired(deadline) + }), + ) + .map_err(crate::raft::execution::storage)?; + check(&query)?; + let observation = query.authority.cell().observation(); + let result = std::panic::catch_unwind(AssertUnwindSafe(|| match query.callback { + Callback::Effects { + class, + now, + reservation, + } => crate::raft::effects::scan::scan_connection_fair( + database.connection(), + &observation.version.history, + query.authority.cell().last_mutation, + &class, + now, + reservation, + ) + .map(Box::new) + .map(Output::Effects) + .map_err(crate::raft::execution::storage), + Callback::Verification(callback) => callback(&database, &observation.version) + .map(Output::Query) + .map_err(crate::raft::execution::storage), + Callback::Registered { + class, + contract, + input, + } => class + .invoke_cell_query(&contract, &observation, &database, &input) + .map(Output::Query) + .map_err(crate::raft::execution::storage), + Callback::Receipt { + class, + request, + input, + } => { + crate::raft::execution::validate_registered_command( + query.options.limits(), + query.authority.route().schema_digest, + &observation.version.history, + &request, + &class, + &input, + false, + )?; + let receipt = crate::raft::receipts::lookup_cell_identity( + database.connection(), + &request, + &query.options.limits(), + observation.version.sequence.get(), + )?; + Ok(Output::Receipt(receipt)) + } + })) + .unwrap_or_else(|panic| { + Err(crate::raft::execution::storage( + crate::native_worker::panic_error(panic), + )) + })?; + // The callback has been consumed, so check the remaining owners explicitly. + if query.cancelled.load(Ordering::Acquire) || query.closing.load(Ordering::Acquire) { + return Err(unavailable()); + } + query + .authority + .check(domain.mono_ms().map_err(|_| unavailable())?) + .map_err(crate::raft::execution::authority_failure)?; + if result.retained_bytes() > query.options.limits().max_result_bytes { + return Err(overloaded()); + } + drop(database); + let replica = query.authority.replica().clone(); + drop(query.authority); + Bytes::shrink(&mut query.bytes, result.retained_bytes() + 4096)?; + Ok(match result { + Output::Effects(scan) => CellSqlOutcome::EffectsScanned(CellSqlEffectScan { + scan, + _bytes: query.bytes, + }), + Output::Query(result) => CellSqlOutcome::Queried(CellSqlQueryResult { + result, + replica, + observation, + _bytes: query.bytes, + }), + Output::Receipt(receipt) => CellSqlOutcome::ReceiptLookedUp(CellSqlReceiptLookup { + receipt, + observation, + _retention: OutputRetention::Sql { + _bytes: query.bytes, + }, + }), + }) +} diff --git a/crates/cellar/durability/raft/host_sql/cell_groups/worker.rs b/crates/cellar/durability/raft/host_sql/cell_groups/worker.rs new file mode 100644 index 000000000..5e0bfa48b --- /dev/null +++ b/crates/cellar/durability/raft/host_sql/cell_groups/worker.rs @@ -0,0 +1,446 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use std::collections::BTreeMap; + +type VerificationCallback = + Box) -> anyhow::Result> + Send>; + +pub(super) enum Callback { + Verification(VerificationCallback), + Registered { + class: Arc, + input: Box<[u8]>, + system: bool, + }, +} + +pub(super) struct Stage { + pub options: CellSqlOptions, + pub authority: CellExecutionAuthority, + pub attempt: CommandAttempt, + pub mode: CellSqlStageMode, + pub callback: Callback, + pub lease: CellLease, + pub bytes: Arc, +} + +type Initializer = Box) -> anyhow::Result<()> + Send>; + +pub(super) struct Initialize { + pub options: CellSqlOptions, + pub authority: CellEnrollmentAuthority, + pub attempt: CellEnrollmentAttempt, + pub callback: Initializer, + pub lease: CellLease, + pub bytes: Arc, +} + +pub(super) enum Action { + Stage(Box), + Initialize(Box), + ResolveEnrollment(ProposalId, Box), + Resume(ProposalId), + Submit(ProposalId), + Finish(Box), + Reconcile(ProposalId, Box), + Reject(Box), + Query(Box), +} + +impl Action { + pub(super) fn progress_priority(&self) -> bool { + !matches!(self, Self::Stage(_) | Self::Initialize(_) | Self::Query(_)) + } +} + +pub(in crate::raft::host_sql) struct Request { + pub cell: CellHistory, + pub(super) queued: crate::raft::performance::Timer, + pub(super) action: Action, + pub(super) reply: oneshot::Sender>, + pub(super) _slot: tokio::sync::OwnedSemaphorePermit, +} + +impl Request { + pub(in crate::raft::host_sql) fn progress_priority(&self) -> bool { + self.action.progress_priority() + } +} + +struct Pending { + producer: Producer, + prepared: CellSqlPrepared, + submitted: bool, + _lease: CellLease, +} + +enum Producer { + Command(Box), + Enrollment(Box), +} + +impl Producer { + fn mark_submitted(&mut self, proposal: ProposalId) -> Result<(), ExecutionError> { + match self { + Self::Command(executor) => executor.mark_submitted(proposal), + Self::Enrollment(executor) => executor.mark_submitted(proposal), + } + } + + fn command(&mut self) -> Result<&mut CellPrivateExecutor, ExecutionError> { + match self { + Self::Command(executor) => Ok(executor), + Self::Enrollment(_) => Err(ExecutionError::Invalid("command producer required")), + } + } + + fn close(self) -> Result<(), ExecutionError> { + match self { + Self::Command(executor) => (*executor).close(), + Self::Enrollment(executor) => (*executor).close(), + } + } +} + +#[derive(Default)] +pub(in crate::raft::host_sql) struct Cells { + pending: BTreeMap, +} + +impl Cells { + pub fn len(&self) -> usize { + self.pending.len() + } + + pub fn execute( + &mut self, + request: Request, + max: usize, + domain: &crate::asyncrt::DomainToken, + stopped: &AtomicBool, + performance: &Arc, + ) { + let Request { + cell, + action, + reply, + _slot, + queued, + } = request; + let operation = match &action { + Action::Stage(_) | Action::Initialize(_) => { + crate::raft::performance::Operation::SqlStage + } + Action::Finish(_) + | Action::Reconcile(..) + | Action::ResolveEnrollment(..) + | Action::Reject(_) => crate::raft::performance::Operation::SqlPublish, + Action::Query(_) => crate::raft::performance::Operation::SqlQuery, + _ => crate::raft::performance::Operation::SqlOther, + }; + let timer = performance.timer(operation); + let result = match domain.mono_ms() { + Ok(now) if !stopped.load(Ordering::Acquire) => { + queued.finish(true); + match action { + // The public waiter can expire while unrelated physical + // work holds this worker. Keep the original admission + // fence through dequeue, before opening a private writer. + Action::Stage(stage) if stage.mode.expired(now) => Err( + crate::raft::execution::authority_failure(RuntimeError::DeadlineExceeded), + ), + Action::Query(query) => super::query::run(*query, domain), + action => self.run(cell, action, max), + } + } + _ => { + queued.finish(false); + Err(unavailable()) + } + }; + // A lost receiver drops only the delivery. Pending owns both the + // private writer and per-cell reservation until exact settlement. + timer.finish(result.is_ok()); + let _ = reply.send(result); + } + + fn run( + &mut self, + cell: CellHistory, + action: Action, + max: usize, + ) -> Result { + match action { + Action::Query(_) => Err(ExecutionError::Invalid("query worker dispatch")), + Action::Stage(stage) => { + if self.pending.contains_key(&cell) { + return Err(ExecutionError::Busy); + } + if self.pending.len() >= max { + return Err(overloaded()); + } + let Stage { + options, + authority, + attempt, + mode, + callback, + lease, + mut bytes, + } = *stage; + if authority.cell().version.history != cell { + return Err(ExecutionError::Invalid("SQL cell history")); + } + let allow_new = mode.allows_new(); + let max_result = options.limits().max_result_bytes; + let mut executor = options.open(authority)?; + let result = match callback { + Callback::Verification(callback) => { + executor.stage(attempt, allow_new, callback) + } + Callback::Registered { + class, + input, + system, + } => { + if system { + executor.stage_system_registered(attempt, &class, &input, allow_new) + } else { + executor.stage_registered(attempt, &class, &input, allow_new) + } + } + }; + match result { + Ok(CellStageOutcome::Prepared(prepared)) => { + let retained = prepared + .bytes() + .len() + .checked_add(max_result) + .and_then(|n| n.checked_add(4096)) + .ok_or_else(overloaded)?; + Bytes::shrink(&mut bytes, retained)?; + let prepared = CellSqlPrepared { + prepared, + _bytes: bytes, + }; + self.pending.insert( + cell, + Pending { + producer: Producer::Command(Box::new(executor)), + prepared: prepared.clone(), + submitted: false, + _lease: lease, + }, + ); + Ok(CellSqlOutcome::Prepared(prepared)) + } + Ok(CellStageOutcome::Duplicate(receipt)) => { + executor.close()?; + let retained = receipt + .receipt + .result() + .len() + .checked_add(4096) + .ok_or_else(overloaded)?; + Bytes::shrink(&mut bytes, retained)?; + Ok(CellSqlOutcome::Receipt(Box::new(CellSqlReceipt { + receipt: *receipt, + _retention: OutputRetention::Sql { _bytes: bytes }, + }))) + } + Err(error) => { + executor.close()?; + Err(error) + } + } + } + Action::Initialize(request) => { + if self.pending.contains_key(&cell) { + return Err(ExecutionError::Busy); + } + if self.pending.len() >= max { + return Err(overloaded()); + } + let Initialize { + options, + authority, + attempt, + callback, + lease, + mut bytes, + } = *request; + if authority.state().cell() != &cell { + return Err(ExecutionError::Invalid("initialization cell history")); + } + authority + .check() + .map_err(crate::raft::execution::authority_failure)?; + if let Some(record) = authority.state().record() { + if record.cell() != &cell || record.creation() != attempt.creation { + return Err(ExecutionError::Refused(CommandFailure::CommandConflict)); + } + return Ok(CellSqlOutcome::Enrollment(Box::new(CellSqlEnrollment { + record: record.clone(), + _retention: OutputRetention::Sql { _bytes: bytes }, + s3: None, + }))); + } + let mut executor = options.open_enrollment(authority)?; + let prepared = match executor.prepare(attempt, callback) { + Ok(prepared) => prepared, + Err(error) => { + executor.close()?; + return Err(error); + } + }; + let retained = prepared + .bytes() + .len() + .checked_add(4096) + .ok_or_else(overloaded)?; + Bytes::shrink(&mut bytes, retained)?; + let prepared = CellSqlPrepared { + prepared, + _bytes: bytes, + }; + self.pending.insert( + cell, + Pending { + producer: Producer::Enrollment(Box::new(executor)), + prepared: prepared.clone(), + submitted: false, + _lease: lease, + }, + ); + Ok(CellSqlOutcome::Prepared(prepared)) + } + Action::ResolveEnrollment(creation, state) => { + let pending = self + .pending + .get_mut(&cell) + .ok_or(ExecutionError::Uncertain)?; + let Producer::Enrollment(executor) = &mut pending.producer else { + return Err(ExecutionError::Invalid("enrollment producer required")); + }; + let outcome = executor.resolve(creation, *state)?; + let pending = self + .pending + .remove(&cell) + .ok_or(ExecutionError::Uncertain)?; + pending.producer.close()?; + let outcome = match outcome { + CellEnrollmentResolution::Committed(record) => { + CellSqlEnrollmentResolution::Committed(Box::new(CellSqlEnrollment { + record: *record, + _retention: OutputRetention::Sql { + _bytes: pending.prepared._bytes, + }, + s3: None, + })) + } + CellEnrollmentResolution::NotCommitted => { + CellSqlEnrollmentResolution::NotCommitted + } + CellEnrollmentResolution::Conflict => CellSqlEnrollmentResolution::Conflict, + }; + Ok(CellSqlOutcome::EnrollmentResolved(outcome)) + } + Action::Resume(proposal) => { + let pending = self.pending.get(&cell).ok_or(ExecutionError::Uncertain)?; + if pending.prepared.proposal() != proposal { + return Err(ExecutionError::Invalid("cell producer proposal")); + } + Ok(CellSqlOutcome::Prepared(pending.prepared.clone())) + } + Action::Reject(rejection) => { + let pending = self.pending.get(&cell).ok_or(ExecutionError::Uncertain)?; + if !pending + .prepared + .prepared + .matches_rejection(&rejection.proof) + { + return Err(ExecutionError::Invalid("cell rejection preparation")); + } + let pending = self + .pending + .remove(&cell) + .ok_or(ExecutionError::Uncertain)?; + // The proof permanently seals every submission clone. Close + // physical SQL ownership before dropping logical-name admission. + pending.producer.close()?; + Ok(CellSqlOutcome::Rejected) + } + Action::Submit(proposal) => { + let pending = self + .pending + .get_mut(&cell) + .ok_or(ExecutionError::Uncertain)?; + if pending.prepared.proposal() != proposal { + return Err(ExecutionError::Invalid("cell producer proposal")); + } + if !pending.submitted { + pending.producer.mark_submitted(proposal)?; + pending.submitted = true; + } + Ok(CellSqlOutcome::Submitted) + } + Action::Finish(authority) => { + let pending = self + .pending + .get_mut(&cell) + .ok_or(ExecutionError::Uncertain)?; + let receipt = pending.producer.command()?.finish(*authority)?; + let pending = self + .pending + .remove(&cell) + .ok_or(ExecutionError::Invalid("cell producer changed"))?; + pending.producer.close()?; + Ok(CellSqlOutcome::Receipt(Box::new(CellSqlReceipt { + receipt, + _retention: OutputRetention::Sql { + _bytes: pending.prepared._bytes, + }, + }))) + } + Action::Reconcile(proposal, authority) => { + let pending = self + .pending + .get_mut(&cell) + .ok_or(ExecutionError::Uncertain)?; + let resolution = pending + .producer + .command()? + .reconcile(proposal, *authority)?; + let pending = self + .pending + .remove(&cell) + .ok_or(ExecutionError::Invalid("cell producer changed"))?; + pending.producer.close()?; + let resolution = match resolution { + CellReconciliation::Committed(receipt) => { + CellSqlReconciliation::Committed(Box::new(CellSqlReceipt { + receipt: *receipt, + _retention: OutputRetention::Sql { + _bytes: pending.prepared._bytes, + }, + })) + } + CellReconciliation::NotCommitted => CellSqlReconciliation::NotCommitted, + CellReconciliation::Refused(failure) => CellSqlReconciliation::Refused(failure), + }; + Ok(CellSqlOutcome::Reconciled(resolution)) + } + } + } + + pub fn close(&mut self) -> Result<(), ExecutionError> { + let mut result = Ok(()); + for (_, pending) in std::mem::take(&mut self.pending) { + if let Err(error) = pending.producer.close() { + result = Err(error); + } + } + result + } +} diff --git a/crates/cellar/durability/raft/host_sql/images.rs b/crates/cellar/durability/raft/host_sql/images.rs index cdebac2ec..8a41af95b 100644 --- a/crates/cellar/durability/raft/host_sql/images.rs +++ b/crates/cellar/durability/raft/host_sql/images.rs @@ -505,7 +505,8 @@ impl ImageOwner { let database = options.database(&image)?; scan::scan_connection( database.connection(), - image.scope(), + &cellar_logic::cell_group::CellHistory::try_from(image.scope()) + .map_err(storage)?, image.position(), class, now, diff --git a/crates/cellar/durability/raft/host_sql/mod.rs b/crates/cellar/durability/raft/host_sql/mod.rs index d58b0c91d..8c211915d 100644 --- a/crates/cellar/durability/raft/host_sql/mod.rs +++ b/crates/cellar/durability/raft/host_sql/mod.rs @@ -4,6 +4,7 @@ //! cancellation joins a running callback before these owners release locks. use super::apply::ImageApplier; +pub(crate) mod cell_groups; mod images; use super::archive::{ArchiveLimits, CommittedArchiveCut}; use super::budget::{Budget, Permit}; @@ -211,15 +212,56 @@ struct Request { _slot: tokio::sync::OwnedSemaphorePermit, } -struct RequestQueue { - receiver: mpsc::Receiver, - pending: VecDeque, +trait Scheduled { + fn same_cell(&self, other: &Self) -> bool; + + fn progress_priority(&self) -> bool; +} + +impl Scheduled for Request { + fn same_cell(&self, other: &Self) -> bool { + self.scope == other.scope + } + + fn progress_priority(&self) -> bool { + self.action.progress_priority() + } +} + +enum Envelope { + Legacy(Box), + Shared(cell_groups::Request), + GroupApply(Box), +} + +impl Scheduled for Envelope { + fn same_cell(&self, other: &Self) -> bool { + match (self, other) { + (Self::Legacy(a), Self::Legacy(b)) => a.same_cell(b), + (Self::Shared(a), Self::Shared(b)) => a.cell == b.cell, + (Self::GroupApply(a), Self::GroupApply(b)) => a.group == b.group, + _ => false, + } + } + + fn progress_priority(&self) -> bool { + match self { + Self::Legacy(request) => request.progress_priority(), + Self::Shared(request) => request.progress_priority(), + Self::GroupApply(_) => true, + } + } +} + +struct RequestQueue { + receiver: mpsc::Receiver, + pending: VecDeque, capacity: usize, progress_streak: usize, } -impl RequestQueue { - fn new(receiver: mpsc::Receiver, capacity: usize) -> Self { +impl RequestQueue { + fn new(receiver: mpsc::Receiver, capacity: usize) -> Self { Self { receiver, pending: VecDeque::new(), @@ -228,7 +270,7 @@ impl RequestQueue { } } - fn next(&mut self) -> Result { + fn next(&mut self) -> Result { if self.pending.is_empty() { self.pending .push_back(self.receiver.recv_timeout(Duration::from_millis(10))?); @@ -250,12 +292,12 @@ impl RequestQueue { .iter() .enumerate() .find(|(index, request)| { - request.action.progress_priority() + request.progress_priority() && !self .pending .iter() .take(*index) - .any(|earlier| earlier.scope == request.scope) + .any(|earlier| earlier.same_cell(request)) }) .map_or(0, |(index, _)| index) } else { @@ -265,7 +307,7 @@ impl RequestQueue { .pending .remove(index) .ok_or(mpsc::RecvTimeoutError::Disconnected)?; - self.progress_streak = if request.action.progress_priority() { + self.progress_streak = if request.progress_priority() { self.progress_streak.saturating_add(1).min(3) } else { 0 @@ -291,7 +333,8 @@ impl Action { pub(crate) struct Pool { performance: Arc, domain: crate::asyncrt::DomainToken, - senders: Vec>, + senders: Vec>, + completion_slots: Vec>, slots: Vec>, workers: Vec>>, stopped: Arc, @@ -363,7 +406,8 @@ pub(crate) struct Budgets { pub(crate) struct Worker { performance: Arc, domain: crate::asyncrt::DomainToken, - sender: mpsc::SyncSender, + sender: mpsc::SyncSender, + completion_slots: Arc, slots: Arc, stopped: Arc, } @@ -388,6 +432,7 @@ impl Pool { performance: performance.clone(), domain: domain.clone(), senders: vec![], + completion_slots: vec![], slots: vec![], workers: vec![], stopped: stopped.clone(), @@ -396,7 +441,10 @@ impl Pool { images: images.clone(), }; for _ in 0..count { - let (sender, receiver) = mpsc::sync_channel::(queue); + let capacity = queue + .checked_add(1) + .ok_or(ExecutionError::Invalid("SQL queue capacity"))?; + let (sender, receiver) = mpsc::sync_channel::(capacity); let owner = domain.clone(); let performance = performance.clone(); let stopped = stopped.clone(); @@ -412,7 +460,8 @@ impl Pool { // retire this physical worker count. let _running = Running::enter(running); let mut cells = BTreeMap::::new(); - let mut requests = RequestQueue::new(receiver, queue); + let mut shared = cell_groups::Cells::default(); + let mut requests = RequestQueue::new(receiver, capacity); let result = loop { if stopped.load(Ordering::Acquire) || owner.mono_ms().is_err() { break Ok(()); @@ -422,6 +471,23 @@ impl Pool { Err(mpsc::RecvTimeoutError::Timeout) => continue, Err(mpsc::RecvTimeoutError::Disconnected) => break Ok(()), }; + let request = match request { + Envelope::GroupApply(request) => { + request.execute(&owner, &stopped); + continue; + } + Envelope::Shared(request) => { + shared.execute( + request, + max_cells.saturating_sub(cells.len()), + &owner, + &stopped, + &performance, + ); + continue; + } + Envelope::Legacy(request) => *request, + }; let Request { queued, scope, @@ -484,7 +550,7 @@ impl Pool { let timer = performance.timer(measured); let result = Self::execute( &mut cells, - max_cells, + max_cells.saturating_sub(shared.len()), &scope, action, &query_pins, @@ -547,7 +613,7 @@ impl Pool { }; // Keep joining/closing every resource even when one reports an // error. Their locks are released on this blocking owner. - let mut cleanup = Ok(()); + let mut cleanup = shared.close(); for (_, mut cell) in cells { if let Some(private) = cell.private.take() { if let Err(error) = private.close() { @@ -560,6 +626,7 @@ impl Pool { .map_err(|_| ExecutionError::Refused(CommandFailure::Unavailable))?; pool.senders.push(sender); pool.slots.push(Arc::new(Semaphore::new(queue))); + pool.completion_slots.push(Arc::new(Semaphore::new(1))); pool.workers.push(task); } Ok(pool) @@ -599,6 +666,7 @@ impl Pool { performance: self.performance.clone(), domain: self.domain.clone(), sender, + completion_slots: self.completion_slots[worker].clone(), slots: self.slots[worker].clone(), stopped: self.stopped.clone(), }) @@ -606,7 +674,7 @@ impl Pool { pub fn stop(&self) { self.stopped.store(true, Ordering::Release); - for slots in &self.slots { + for slots in self.slots.iter().chain(&self.completion_slots) { slots.close(); } } @@ -764,6 +832,7 @@ impl Worker { sender, slots, stopped, + completion_slots: _, } = self.clone(); Box::pin(async move { let queued = performance.timer(Measured::SqlQueue); @@ -776,7 +845,7 @@ impl Worker { } let (reply, result) = oneshot::channel(); sender - .try_send(Request { + .try_send(Envelope::Legacy(Box::new(Request { queued, scope, action, @@ -784,7 +853,7 @@ impl Worker { _work: work, cancel, _slot: slot, - }) + }))) .map_err(|_| ExecutionError::Refused(CommandFailure::Unavailable))?; result .await diff --git a/crates/cellar/durability/raft/host_startup_tests.rs b/crates/cellar/durability/raft/host_startup_tests.rs index 9a6d95dae..094c10fdb 100644 --- a/crates/cellar/durability/raft/host_startup_tests.rs +++ b/crates/cellar/durability/raft/host_startup_tests.rs @@ -2,8 +2,14 @@ //! A startup error owns infrastructure until actual file-lock release. #![allow(clippy::disallowed_methods)] -use super::storage_fault as fault; use super::*; +use crate::raft::storage_fault as fault; +use crate::raft::{ + catalog::{Catalog, CatalogOptions, InitialMember, NamespaceConfig}, + completion::RuntimeError, + transport::{GroupRoute as PhysicalRoute, RaftPeer, RaftTransport}, + transport_options::RaftPeerNode, +}; use crate::raft::{ options::{LogOpenMode, RaftRuntimeOptions, ReplicaLogOptions}, storage::ReplicaLog, @@ -11,6 +17,12 @@ use crate::raft::{ version::{VersionKeys, VersionSigningKey}, }; use anyhow::Context as _; +use cellar_logic::{ + cell_group::{CellId, CellRange, GroupHistory, RouteEpoch}, + consensus::*, +}; +use std::collections::BTreeMap; +use std::sync::atomic::Ordering; use std::time::Duration; #[tokio::test(flavor = "multi_thread", worker_threads = 4)] @@ -32,7 +44,7 @@ async fn failed_preflight_joins_infrastructure_with_retained_handles() -> anyhow )) .await??; let (runtime, events) = - RaftRuntime::start_parked(log.clone(), RaftRuntimeOptions::default()).await?; + CellGroupRuntime::start(log.clone(), RaftRuntimeOptions::default()).await?; let tls = RaftTls::new( vec![include_bytes!("../../transport/tcp/fixtures/ca.der") .to_vec() @@ -45,7 +57,7 @@ async fn failed_preflight_joins_infrastructure_with_retained_handles() -> anyhow ) .into(), )?; - let transport = RaftTransport::start( + let transport = RaftTransport::start_shared( runtime.clone(), Arc::new(crate::peer_auth::PeerAuth::new([8; 32], "startup")?), tls, @@ -64,20 +76,22 @@ async fn failed_preflight_joins_infrastructure_with_retained_handles() -> anyhow let release = gate.arm(fault::HOLD_LOCK_DROP); let mut startup = Box::pin(domain.run(start( service, - Registrations { - cells: vec![], + Prepared { namespaces: vec![], + groups: vec![], peers: vec![], - origins: vec![], + query_origins: vec![], + command_origins: vec![], }, root.path().join("cells"), - "startup".into(), - RaftHostOptions::default(), + test_support::host_options(), + test_support::limits(), VersionKeys::new( VersionSigningKey::new([31; 32], u64::MAX)?, vec![], Duration::from_secs(86400), )?, + None, ))); let early = tokio::time::timeout(Duration::from_millis(30), &mut startup).await; let returned_early = early.is_ok(); @@ -102,7 +116,7 @@ async fn failed_preflight_joins_infrastructure_with_retained_handles() -> anyhow }?; let admission_closed = matches!( log.storage_instance(), - Err(super::super::storage::LogError::Closed) + Err(crate::raft::storage::LogError::Closed) ); if !returned_early { // This is deliberately before cleanup through the retained handles. @@ -140,17 +154,17 @@ async fn failed_preflight_joins_infrastructure_with_retained_handles() -> anyhow #[path = "../../tests/support/raft_catalog_fault.rs"] mod catalog_fault; -struct StartupNode { - root: tempfile::TempDir, - domain: crate::asyncrt::ExecutionDomain, - log: ReplicaLog, - runtime: RaftRuntime, - events: Option, +pub(super) struct StartupNode { + pub(super) root: tempfile::TempDir, + pub(super) domain: crate::asyncrt::ExecutionDomain, + pub(super) log: ReplicaLog, + runtime: CellGroupRuntime, + events: Option, transport: RaftTransport, } impl StartupNode { - async fn open(id: u8) -> anyhow::Result { + pub(super) async fn open(id: u8) -> anyhow::Result { let root = tempfile::tempdir()?; let domain = crate::asyncrt::ExecutionDomain::new(tokio::runtime::Handle::current()); let log = domain @@ -165,7 +179,7 @@ impl StartupNode { Arc::new(cellar_ltx::DirectFileSystem), )) .await??; - let (runtime, events) = RaftRuntime::start_parked( + let (runtime, events) = CellGroupRuntime::start( log.clone(), RaftRuntimeOptions { max_requests: 1, @@ -187,7 +201,7 @@ impl StartupNode { ) .into(), )?; - let transport = RaftTransport::start( + let transport = RaftTransport::start_shared( runtime.clone(), Arc::new(crate::peer_auth::PeerAuth::new( [8; 32], @@ -210,7 +224,7 @@ impl StartupNode { }) } - fn peer(&self) -> RaftPeerNode { + pub(super) fn peer(&self) -> RaftPeerNode { let limits = self.log.limits(); RaftPeerNode { node: limits.node, @@ -224,7 +238,7 @@ impl StartupNode { } } - fn service(&mut self) -> crate::daemon::NodeRaftService { + pub(super) fn service(&mut self) -> crate::daemon::NodeRaftService { crate::daemon::NodeRaftService { log: self.log.clone(), runtime: self.runtime.clone(), @@ -233,7 +247,7 @@ impl StartupNode { } } - async fn close(self) -> anyhow::Result<()> { + pub(super) async fn close(self) -> anyhow::Result<()> { self.transport.close().await; let _ = self.runtime.close().await; self.log.close().await?; @@ -275,65 +289,144 @@ async fn peer_traffic_waits_until_all_local_groups_attach() -> anyhow::Result<() class.schema_digest(), )) .await?; + let directory = RangeDirectory::new(namespace.clone(), DirectoryLimits::default())?; + let key = crate::application::consensus::cell_routing_key( + namespace.config().namespace, + &CellId::new("Counter:startup-boundary").map_err(|_| anyhow::anyhow!("cell id"))?, + ); + let (a, b) = CellRange::all() + .split(key) + .map_err(|_| anyhow::anyhow!("ranges"))?; + let mut requests = Vec::new(); + for (id, range) in [(1, a), (2, b)] { + let request = directory.prepare_group_birth(CellGroupBirthSpec { + history: GroupHistory { + cluster: namespace.config().cluster, + namespace: namespace.config().namespace, + group: GroupId::new([id; 16])?, + incarnation: IncarnationId::new([9; 16])?, + }, + range, + epoch: RouteEpoch::new(1).map_err(|_| anyhow::anyhow!("epoch"))?, + creation: ProposalId::new([id; 16])?, + voters: (1..=3) + .map(|member| { + Ok(InitialMember { + member: MemberId::new(member)?, + node: NodeId::new([member as u8; 16])?, + }) + }) + .collect::>()?, + learners: vec![], + })?; + local + .domain + .run(directory.reserve_group_birth(&request)) + .await??; + requests.push(request); + } + let initialization = directory.prepare_initialization( + ProposalId::new([33; 16])?, + &requests + .iter() + .map(|r| r.route().clone()) + .collect::>(), + )?; + local + .domain + .run(directory.publish(&initialization)) + .await??; let mut births = Vec::new(); - for id in 1..=2 { + for request in requests { births.push( - namespace - .resolve_birth(BirthProposal { - scope: GroupScope { - cluster: namespace.config().cluster, - namespace: namespace.config().namespace, - cell: format!("Counter:startup-{id}"), - group: GroupId::new([id; 16])?, - incarnation: IncarnationId::new([9; 16])?, - }, - voters: (1..=3) - .map(|member| { - Ok(InitialMember { - member: MemberId::new(member)?, - node: NodeId::new([member as u8; 16])?, - }) - }) - .collect::>()?, - learners: vec![], - }) - .await?, + local + .domain + .run(directory.materialize_selected_history(request.route().history)) + .await??, ); } - let first = births[0].scope().clone(); - let second = births[1].scope().clone(); - namespace - .bootstrap_member(&births[0], &remote.log) - .await? - .attach(&remote.runtime) - .await - .context("attach remote startup member")?; - remote.transport.install_group(membership::route( - &remote.log.group(&first)?, - &births[0], - &peers, - class.schema_digest(), - "node-2", - &remote.transport, - )?)?; + let first = births[0].route().history; + let second = births[1].route().history; + let assignment = |birth: &CellGroupBirth| -> anyhow::Result { + Ok(Assignment { + namespace: 0, + placement: RaftGroupPlacement::Replica, + pending_transfer_archive: false, + sealed_transfer_source: false, + birth: birth.clone(), + transport: PhysicalRoute { + scope: birth.route().history, + schema_digest: birth.route().schema_digest, + revision: birth.route().epoch.get(), + peers: birth + .voters() + .iter() + .map(|member| { + let peer = &peers[&member.node]; + RaftPeer { + replica: ReplicaIdentity { + scope: birth.route().history, + member: member.member, + node: peer.node, + disk: peer.disk, + process: peer.process, + }, + label: peer.label.clone(), + server_name: peer.server_name.clone(), + control_address: peer.control_address, + data_address: peer.data_address, + query_address: peer.query_address, + } + }) + .collect(), + }, + }) + }; + let mut remote = remote; + let remote_prepared = Prepared { + namespaces: vec![Namespace { + directory: directory.clone(), + class: class.clone(), + }], + groups: vec![assignment(&births[0])?], + peers: peers.values().cloned().collect(), + query_origins: vec![], + command_origins: vec![], + }; + let remote_service = remote.service(); + let remote_host = remote + .domain + .run(start( + remote_service, + remote_prepared, + remote.root.path().join("cells"), + test_support::host_options(), + test_support::limits(), + VersionKeys::new( + VersionSigningKey::new([31; 32], u64::MAX)?, + vec![], + Duration::from_secs(86400), + )?, + None, + )) + .await??; // Hold the second local member's permanent claim after the first member // has attached. Its real peer keeps sending while no host drains events. store.hold_nth_claim.store(2, Ordering::Release); store.claim_hang.store(true, Ordering::Release); let release = ReleaseClaim(store.clone()); - let registrations = Registrations { - cells: births - .into_iter() - .map(|birth| Registration { - placement: RaftCellPlacement::Replica, - namespace: namespace.clone(), - birth, - class: class.clone(), - }) - .collect(), + let registrations = Prepared { + namespaces: vec![Namespace { + directory: directory.clone(), + class: class.clone(), + }], + groups: births + .iter() + .map(assignment) + .collect::>()?, peers: peers.into_values().collect(), - namespaces: vec![], - origins: vec![], + query_origins: vec![], + command_origins: vec![], }; let service = local.service(); let directory = local.root.path().join("cells"); @@ -344,13 +437,14 @@ async fn peer_traffic_waits_until_all_local_groups_attach() -> anyhow::Result<() service, registrations, directory, - "node-1".into(), - RaftHostOptions::default(), + test_support::host_options(), + test_support::limits(), VersionKeys::new( VersionSigningKey::new([31; 32], u64::MAX)?, vec![], Duration::from_secs(86400), )?, + None, )) .await? }); @@ -368,7 +462,10 @@ async fn peer_traffic_waits_until_all_local_groups_attach() -> anyhow::Result<() responses.push( tokio::time::timeout( Duration::from_secs(3), - remote.transport.submit(&first, message)?.accepted(), + remote + .transport + .submit_physical(&first, message)? + .accepted(), ) .await?, ); @@ -390,13 +487,13 @@ async fn peer_traffic_waits_until_all_local_groups_attach() -> anyhow::Result<() .as_ref() .map_err(|error| anyhow::anyhow!("startup failed: {error:#}"))?; for scope in [&first, &second] { - anyhow::ensure!(host.cell(scope.namespace, &scope.cell).is_ok()); + anyhow::ensure!(host.group(*scope).is_ok()); // The active host's background inspection shares this fixture's // single request slot. Read-only overload is temporary contention, // not evidence that startup failed to attach the group. tokio::time::timeout(Duration::from_secs(3), async { loop { - match local.runtime.status(scope).await { + match local.runtime.status(*scope).await { Err(RuntimeError::Overloaded) => { tokio::time::sleep(Duration::from_millis(10)).await; } @@ -405,8 +502,8 @@ async fn peer_traffic_waits_until_all_local_groups_attach() -> anyhow::Result<() } }) .await - .with_context(|| format!("startup status deadline for {}", scope.cell))? - .with_context(|| format!("inspect attached startup group {}", scope.cell))?; + .with_context(|| format!("startup status deadline for {:?}", scope.group))? + .with_context(|| format!("inspect attached startup group {:?}", scope.group))?; } let mut message = Message { from: 2, @@ -419,7 +516,7 @@ async fn peer_traffic_waits_until_all_local_groups_attach() -> anyhow::Result<() // The active host's one-item runtime queue also serves inspection. // Retry only explicit refusal and still require successful admission. loop { - let result = match remote.transport.submit(&first, message.clone()) { + let result = match remote.transport.submit_physical(&first, message.clone()) { Ok(delivery) => delivery.accepted().await, Err(error) => Err(error), }; @@ -438,8 +535,113 @@ async fn peer_traffic_waits_until_all_local_groups_attach() -> anyhow::Result<() if let Ok(host) = started { host.close().await?; } + remote_host.close().await?; remote.close().await?; third.close().await?; local.close().await?; result } + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn late_startup_refusal_joins_control_services_before_storage_close() -> anyhow::Result<()> { + use crate::application::consensus::group_client::ControlServiceKind; + let mut node = StartupNode::open(1).await?; + let class = Arc::new(RaftClass::new("StartupOnly", 1, [65; 32])?); + let catalog = Catalog::new( + Arc::new(object_store::memory::InMemory::new()), + Default::default(), + )?; + let namespace = node + .domain + .run(catalog.open_namespace(NamespaceConfig::new( + ClusterId::new([1; 16])?, + NamespaceId::new([2; 16])?, + cellar_logic::NamespaceMode::Raft, + class.schema_digest(), + ))) + .await??; + let prepared = Prepared { + namespaces: vec![Namespace { + directory: RangeDirectory::new(namespace, Default::default())?, + class, + }], + peers: vec![node.peer()], + groups: vec![], + query_origins: vec![], + command_origins: vec![], + }; + let options = test_support::host_options(); + let capacity = options.max_control_bytes; + let input = node.service(); + let host = node + .domain + .run(start( + input, + prepared, + node.root.path().join("cells"), + options, + test_support::limits(), + VersionKeys::new( + VersionSigningKey::new([31; 32], u64::MAX)?, + vec![], + Duration::from_secs(86400), + )?, + None, + )) + .await??; + // Reproduce late registration refusal after another control worker entered. + // The node's existing bank and worker lifecycle supply the real boundary. + let lease = host.register_control_service(ControlServiceKind::Directory, capacity)?; + let mut stop = lease.stop_receiver(); + let (entered, stopped) = tokio::sync::oneshot::channel(); + let (release, released) = tokio::sync::oneshot::channel(); + node.log + .domain() + .spawn_cleanup(async move { + let _lease = lease; + let _ = stop.changed().await; + let _ = entered.send(()); + let _ = released.await; + })? + .detach(); + assert!(matches!( + host.register_control_service(ControlServiceKind::Recovery, 4096), + Err(crate::raft::fleet_catalog::FleetCatalogError::Overloaded) + )); + let mut startup = Startup { + service: Some(Infrastructure { + log: node.log.clone(), + runtime: node.runtime.clone(), + transport: node.transport.clone(), + }), + public: Some(host), + sql: None, + controller: None, + }; + let mut closing = startup.cleanup()?.context("owned cleanup task")?; + tokio::time::timeout(Duration::from_secs(5), stopped).await??; + assert!( + tokio::time::timeout(Duration::from_millis(30), &mut closing) + .await + .is_err(), + "startup error returned before its entered control work joined" + ); + let _ = release.send(()); + tokio::time::timeout(Duration::from_secs(5), closing).await??; + assert!(matches!( + node.log.storage_instance(), + Err(crate::raft::storage::LogError::Closed) + )); + let mut options = node.log.limits().clone(); + options.mode = LogOpenMode::Resume; + options.process = ProcessGeneration::new([9; 16])?; + let resumed = node + .domain + .run(ReplicaLog::open( + options, + Arc::new(cellar_ltx::DirectFileSystem), + )) + .await??; + resumed.close().await?; + node.close().await +} diff --git a/crates/cellar/durability/raft/inspection.rs b/crates/cellar/durability/raft/inspection.rs index cdbf0b778..7bead86ff 100644 --- a/crates/cellar/durability/raft/inspection.rs +++ b/crates/cellar/durability/raft/inspection.rs @@ -133,6 +133,19 @@ pub struct CellInspection { pub leader_load: Option, } +/// Progress captured with one physical group's complete quorum-selected root. +/// This never substitutes a business cell identity or grants leader authority. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct GroupInspection { + pub history: cellar_logic::cell_group::GroupHistory, + pub schema_digest: [u8; 32], + pub birth_digest: [u8; 32], + pub node: NodeInspection, + pub member: Option, + /// None means no complete node-wide leadership scan was captured. + pub leader_load: Option, +} + // The transport uses the validated ledger/configuration pair to encode a strict // bounded response. The public SDK returns only the observation above. #[derive(Clone, Debug)] diff --git a/crates/cellar/durability/raft/inspection_transport.rs b/crates/cellar/durability/raft/inspection_transport.rs index b84c676d1..e4f5f9597 100644 --- a/crates/cellar/durability/raft/inspection_transport.rs +++ b/crates/cellar/durability/raft/inspection_transport.rs @@ -4,7 +4,6 @@ //! transport task registry; dropping a caller cancels its read-only RPC. use super::snapshots::{read_bulk_head, Wire}; use super::*; -use crate::application::consensus::{CommandFailure, RaftCells}; use crate::raft::inspection::{InspectionError, InspectionSnapshot}; use crate::raft::inspection_identity::{InspectionRoute, PhysicalNode}; use crate::raft::inspection_wire as body; @@ -142,6 +141,7 @@ impl RaftTransport { let mut registry = state.registry.lock().map_err(|_| TransportError::Closed)?; state.admit(®istry)?; registry.check_node_view(&grant.peer)?; + registry.check_physical_endpoint(&grant.peer)?; if let Some(old) = registry.inspection_origins.get(&key) { if old.revision == grant.revision && old.peer.as_ref() == Some(&grant.peer) { return Ok(()); @@ -439,158 +439,6 @@ impl State { validate_response(route, &snapshot)?; Ok(snapshot) } - - pub(super) async fn receive_inspection( - &self, - stream: &mut S, - exporter: &[u8; 32], - prefix: [u8; frame::PREFIX], - ) -> Result<(), TransportError> { - let _node = self - .inspection_metadata - .acquire(META_BYTES) - .ok_or(TransportError::Overloaded)?; - let head = self - .bounded( - None, - read_bulk_head(stream, Some(prefix), REQUEST_MAGIC, 0, &self.options), - ) - .await?; - let wire = Wire { - auth: &self.auth, - exporter, - options: &self.options, - max_body: body::MAX_BODY, - }; - let request: RequestHead = wire.decode(REQUEST_MAGIC, REQUEST_PURPOSE, &head, None)?; - if frame::decimal(&request.nonce)? == 0 || !(1..=60_000).contains(&request.timeout_ms) { - return Err(TransportError::Protocol); - } - let (scope, source, target) = request.route.decode()?; - let local = self.runtime.log().limits(); - if target.identity - != (NodeIdentity { - node: local.node, - disk: local.disk, - process: local.process, - }) - || target.storage_instance - != self - .runtime - .log() - .storage_instance() - .map_err(|_| TransportError::Closed)? - || request.route.target_label() != self.auth.source() - { - return Err(TransportError::StalePeer); - } - let authorized = (|| { - let mut registry = self.registry.lock().map_err(|_| TransportError::Closed)?; - self.admit(®istry)?; - let grant = registry - .inspection_origins - .get(&(scope.cluster, scope.namespace, source.identity.node)) - .cloned() - .ok_or(TransportError::Refused)?; - let peer = grant.peer.as_ref().ok_or(TransportError::Refused)?; - registry.check_node_view(peer)?; - let view = registry - .node_views - .get(&peer.node) - .ok_or(TransportError::Refused)?; - if source.identity - != (NodeIdentity { - node: peer.node, - disk: peer.disk, - process: peer.process, - }) - || source.storage_instance != view.storage_instance() - || request.route.source_label() != peer.label - { - return Err(TransportError::StalePeer); - } - let budget = registry.inspection_peer_budget(peer.node, self.options.max_peers)?; - Ok::<_, TransportError>((grant, budget)) - })(); - let (grant, peer_budget) = match authorized { - Ok(authorized) => authorized, - Err(TransportError::Refused | TransportError::UnknownGroup) => { - let reply = ReplyHead { - route: request.route, - nonce: request.nonce, - status: 2, - digest: Sha256::digest([]).into(), - }; - let response = - wire.encode(RESPONSE_MAGIC, RESPONSE_PURPOSE, &head.tag, &reply, &[])?; - return self.bounded(None, write_head(stream, &response)).await; - } - Err(error) => return Err(error), - }; - let _peer = peer_budget - .acquire(META_BYTES) - .ok_or(TransportError::Overloaded)?; - let deadline = self - .domain - .mono_ms() - .map_err(|_| TransportError::Closed)? - .checked_add(request.timeout_ms) - .ok_or(TransportError::Limit)?; - let handle = self - .query_gateway - .lock() - .map_err(|_| TransportError::Closed)? - .as_ref() - .and_then(|h| h.upgrade()) - .ok_or(TransportError::Refused)?; - let binding = handle - .bindings - .get(&(scope.namespace, scope.cell.clone())) - .map_err(|_| TransportError::UnknownGroup)?; - if binding.scope != scope || binding.class.schema_digest() != request.route.schema() { - return Err(TransportError::Refused); - } - let cell = RaftCells { handle } - .cell(scope.namespace, &scope.cell) - .map_err(|_| TransportError::UnknownGroup)?; - let epoch = grant.stop.subscribe(); - self.seed_process_until(&epoch, deadline, async { - let now = self.domain.mono_ms().map_err(|_| TransportError::Closed)?; - let duration = std::time::Duration::from_millis( - deadline - .checked_sub(now) - .filter(|ms| *ms > 0) - .ok_or(TransportError::Timeout)?, - ); - let result = cell.inspect_snapshot(duration).await; - if *epoch.borrow() { - return Err(TransportError::StalePeer); - } - if *self.stop.borrow() { - return Err(TransportError::Closed); - } - let (status, bytes) = match result { - Ok(snapshot) if snapshot.view.birth_digest == request.route.birth() => { - (0, body::encode(&snapshot)?) - } - Err(CommandFailure::Overloaded) => (1, vec![]), - _ => (2, vec![]), - }; - let reply = ReplyHead { - route: request.route, - nonce: request.nonce, - status, - digest: Sha256::digest(&bytes).into(), - }; - let response = - wire.encode(RESPONSE_MAGIC, RESPONSE_PURPOSE, &head.tag, &reply, &bytes)?; - write_head(stream, &response).await?; - stream.write_all(&bytes).await?; - stream.flush().await?; - Ok(()) - }) - .await - } } #[cfg(test)] diff --git a/crates/cellar/durability/raft/member_fleet.rs b/crates/cellar/durability/raft/member_fleet.rs index 45f6d7d9c..dc47deff3 100644 --- a/crates/cellar/durability/raft/member_fleet.rs +++ b/crates/cellar/durability/raft/member_fleet.rs @@ -2,16 +2,15 @@ //! only ordered, committed changes can consume or fence a permit sequence. use super::*; use ::raft::prelude::ConfChangeSingle; -use cellar_logic::consensus::{ - ClusterId, GroupId, GroupScope, IncarnationId, NamespaceId, ProposalId, -}; +use cellar_logic::cell_group::GroupHistory; +use cellar_logic::consensus::{ClusterId, GroupId, IncarnationId, NamespaceId, ProposalId}; use sha2::{Digest, Sha256}; #[cfg(test)] #[path = "member_fleet_tests.rs"] mod tests; -/// Exact fleet and cell-history identity. Encoding this descriptor does not +/// Exact fleet and physical group-history identity. Encoding this descriptor does not /// enroll a group; enrollment requires a rooted catalog authorization. #[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] #[serde(deny_unknown_fields)] @@ -20,7 +19,6 @@ pub struct FleetGroupBinding { catalog: [u8; 32], cluster: [u8; 16], namespace: [u8; 16], - cell: String, group: [u8; 16], incarnation: [u8; 16], } @@ -28,10 +26,9 @@ pub struct FleetGroupBinding { impl FleetGroupBinding { pub fn new( fleet: IncarnationId, - scope: &GroupScope, + scope: &GroupHistory, catalog: [u8; 32], ) -> Result { - scope.validate().map_err(|_| invalid())?; if catalog == [0; 32] { return Err(invalid()); } @@ -40,7 +37,6 @@ impl FleetGroupBinding { catalog, cluster: scope.cluster.bytes(), namespace: scope.namespace.bytes(), - cell: scope.cell.clone(), group: scope.group.bytes(), incarnation: scope.incarnation.bytes(), }) @@ -54,12 +50,11 @@ impl FleetGroupBinding { self.catalog } - pub fn scope(&self) -> Result { + pub fn scope(&self) -> Result { self.validate()?; - Ok(GroupScope { + Ok(GroupHistory { cluster: ClusterId::new(self.cluster).map_err(|_| invalid())?, namespace: NamespaceId::new(self.namespace).map_err(|_| invalid())?, - cell: self.cell.clone(), group: GroupId::new(self.group).map_err(|_| invalid())?, incarnation: IncarnationId::new(self.incarnation).map_err(|_| invalid())?, }) @@ -68,11 +63,11 @@ impl FleetGroupBinding { pub fn key(&self) -> Result { self.validate()?; let bytes = serde_json::to_vec(&( + "cellar-cell-group-fleet-binding-1", self.fleet, self.catalog, self.cluster, self.namespace, - &self.cell, self.group, self.incarnation, )) @@ -94,16 +89,16 @@ impl FleetGroupBinding { NamespaceId::new(self.namespace).map_err(|_| invalid())?; GroupId::new(self.group).map_err(|_| invalid())?; IncarnationId::new(self.incarnation).map_err(|_| invalid())?; - if !cellar_logic::cell::valid_cell_scope(&self.cell) { - return Err(invalid()); - } Ok(()) } - pub(crate) fn matches_scope(&self, scope: &GroupScope) -> bool { + pub(crate) fn matches_scope(&self, scope: &S) -> bool + where + for<'a> GroupHistory: From<&'a S>, + { + let scope = GroupHistory::from(scope); self.cluster == scope.cluster.bytes() && self.namespace == scope.namespace.bytes() - && self.cell == scope.cell && self.group == scope.group.bytes() && self.incarnation == scope.incarnation.bytes() } @@ -374,6 +369,23 @@ impl GroupMembership { Ok(initial) } + /// A non-serving seed base preserves whether the source was fleet guarded + /// at genesis or enrolled later. The real full snapshot must still succeed + /// this base; this projection grants no startup or membership authority. + pub(crate) fn seed_initial(&self, conf: &ConfState) -> Result { + let initial = Self::initial(conf); + if !self.same_birth(&initial) { + return Err(invalid()); + } + match &self.fleet { + Some(state) => match state.origin { + EnrollmentOrigin::Genesis => Self::managed_initial(conf, state.binding.clone()), + EnrollmentOrigin::Committed => Ok(initial), + }, + None => Ok(initial), + } + } + pub fn fleet_status(&self) -> Option { self.fleet.as_ref().map(|state| FleetMembershipStatus { binding: state.binding.clone(), @@ -462,7 +474,10 @@ impl GroupMembership { ) } - pub(crate) fn validate_scope(&self, scope: &GroupScope) -> Result<(), LogError> { + pub(crate) fn validate_scope(&self, scope: &S) -> Result<(), LogError> + where + for<'a> GroupHistory: From<&'a S>, + { if self .fleet .as_ref() @@ -473,12 +488,15 @@ impl GroupMembership { Ok(()) } - pub(crate) fn admit_scoped( + pub(crate) fn admit_scoped( &self, change: &ConfChangeV2, conf: &ConfState, - scope: &GroupScope, - ) -> Result<(), LogError> { + scope: &S, + ) -> Result<(), LogError> + where + for<'a> GroupHistory: From<&'a S>, + { self.validate_scope(scope)?; let binding = match self.intent(change)?.fleet { Some(FleetIntent::Enroll { binding }) => Some(binding), diff --git a/crates/cellar/durability/raft/member_fleet_tests.rs b/crates/cellar/durability/raft/member_fleet_tests.rs index 5ccc37389..bf2e814b0 100644 --- a/crates/cellar/durability/raft/member_fleet_tests.rs +++ b/crates/cellar/durability/raft/member_fleet_tests.rs @@ -1,5 +1,56 @@ use super::*; +#[test] +fn fleet_group_binding_refuses_cell_scoped_records() { + let (_, _, history, binding) = fixture(); + let mut record = serde_json::to_value(&binding).unwrap(); + assert!(record.get("cell").is_none()); + record["cell"] = serde_json::json!("Counter:legacy"); + assert!(serde_json::from_value::(record).is_err()); + let mut foreign = history; + foreign.incarnation = IncarnationId::new([91; 16]).unwrap(); + let other = + FleetGroupBinding::new(IncarnationId::new([5; 16]).unwrap(), &foreign, [90; 32]).unwrap(); + assert_ne!(binding.key().unwrap(), other.key().unwrap()); + assert!(!binding.matches_scope(&foreign)); +} + +#[test] +fn group_membership_retains_its_managed_guard_and_permit_sequence() { + let (state, conf, history, binding) = fixture(); + let (permit, _) = permit(binding); + let mut change = ConfChangeV2::default(); + state.bind_fleet_fence(&mut change, &conf, &permit).unwrap(); + state.admit_group(&change, &conf, &history).unwrap(); + let mut foreign = history; + foreign.group = GroupId::new([92; 16]).unwrap(); + assert!(state.admit_group(&change, &conf, &foreign).is_err()); + let next = state + .applied(&change, &conf, &conf, LogIndex::new(3)) + .unwrap(); + let restored: GroupMembership = + serde_json::from_slice(&serde_json::to_vec(&next).unwrap()).unwrap(); + restored.validate_scope(&history).unwrap(); + assert_eq!(restored.fleet_status().unwrap().consumed_sequence, 1); + assert!(restored + .bind_fleet_fence(&mut ConfChangeV2::default(), &conf, &permit) + .is_err()); + let replica = cellar_logic::consensus::ReplicaIdentity { + scope: history, + member: MemberId::new(1).unwrap(), + node: NodeId::new([61; 16]).unwrap(), + disk: DiskLineage::new([62; 16]).unwrap(), + process: cellar_logic::consensus::ProcessGeneration::new([63; 16]).unwrap(), + }; + assert!(restored.permits_group(&replica)); + assert!( + !restored.permits_group(&cellar_logic::consensus::ReplicaIdentity { + scope: foreign, + ..replica + }) + ); +} + #[test] fn fleet_genesis_origin_is_explicit_and_cannot_be_retrofitted() { let (_, conf, scope, binding) = fixture(); @@ -59,11 +110,10 @@ fn fleet_genesis_fences_first_permit_before_any_business_write() { assert_ne!(unused.expected_revision(), permit.expected_revision()); } -fn fixture() -> (GroupMembership, ConfState, GroupScope, FleetGroupBinding) { - let scope = GroupScope { +fn fixture() -> (GroupMembership, ConfState, GroupHistory, FleetGroupBinding) { + let scope = GroupHistory { cluster: ClusterId::new([1; 16]).unwrap(), namespace: NamespaceId::new([2; 16]).unwrap(), - cell: "Counter:fleet".into(), group: GroupId::new([3; 16]).unwrap(), incarnation: IncarnationId::new([4; 16]).unwrap(), }; diff --git a/crates/cellar/durability/raft/member_state.rs b/crates/cellar/durability/raft/member_state.rs index f8c58eee3..642f100c5 100644 --- a/crates/cellar/durability/raft/member_state.rs +++ b/crates/cellar/durability/raft/member_state.rs @@ -23,7 +23,7 @@ pub use fleet::{ use fleet::{FleetIntent, FleetState}; const MAX_BYTES: usize = 16 * 1024; -const MAX_MEMBERS: usize = 32; +pub(crate) const MAX_MEMBERS: usize = 32; // A configuration entry carries one bounded intent plus at most 32 small // protobuf changes. Bound decoding before allocating their repeated fields. @@ -51,6 +51,16 @@ struct Binding { joined_at: u64, } +impl Binding { + fn joining_member(&self) -> Result { + Ok(JoiningMember { + member: MemberId::new(self.member).map_err(|_| invalid())?, + node: NodeId::new(self.node).map_err(|_| invalid())?, + disk: DiskLineage::new(self.disk).map_err(|_| invalid())?, + }) + } +} + /// Persisted with the applied configuration, including across compaction. IDs /// at or below high_water can never be assigned to a different member again. #[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] @@ -125,13 +135,39 @@ impl GroupMembership { self.high_water } - pub(crate) fn permits(&self, replica: &cellar_logic::consensus::ReplicaIdentity) -> bool { + pub(crate) fn has_node_binding(&self, node: NodeId) -> bool { + self.bindings + .iter() + .any(|binding| binding.node == node.bytes()) + } + + pub(crate) fn has_disk_binding(&self, disk: cellar_logic::consensus::DiskLineage) -> bool { + self.bindings + .iter() + .any(|binding| binding.disk == disk.bytes()) + } + + pub(crate) fn permits(&self, replica: &cellar_logic::consensus::ReplicaIdentity) -> bool + where + for<'a> cellar_logic::cell_group::GroupHistory: From<&'a S>, + { if self.validate_scope(&replica.scope).is_err() { return false; } + self.permits_binding(replica) + } + + pub(crate) fn permits_group( + &self, + replica: &cellar_logic::consensus::ReplicaIdentity, + ) -> bool { + self.validate_scope(&replica.scope).is_ok() && self.permits_binding(replica) + } + + fn permits_binding(&self, replica: &cellar_logic::consensus::ReplicaIdentity) -> bool { self.bindings .iter() - .find(|b| b.member == replica.member.get()) + .find(|binding| binding.member == replica.member.get()) .map_or_else( || self.initial.contains(&replica.member.get()), |binding| { @@ -144,19 +180,16 @@ impl GroupMembership { self.bindings .iter() .find(|b| b.member == member.get()) - .map(|b| { - Ok(( - JoiningMember { - member, - node: NodeId::new(b.node).map_err(|_| invalid())?, - disk: DiskLineage::new(b.disk).map_err(|_| invalid())?, - }, - LogIndex::new(b.joined_at), - )) - }) + .map(|binding| Ok((binding.joining_member()?, LogIndex::new(binding.joined_at)))) .transpose() } + pub(crate) fn retained_lineages( + &self, + ) -> impl Iterator> + '_ { + self.bindings.iter().map(Binding::joining_member) + } + pub(crate) fn reserved_bytes(&self) -> usize { // Retirement can leave spare vector capacity until this value is // cloned or released. Keep that allocation charged too. @@ -287,6 +320,19 @@ impl GroupMembership { Ok(intent) } + pub(crate) fn admit_group( + &self, + change: &ConfChangeV2, + conf: &ConfState, + history: &cellar_logic::cell_group::GroupHistory, + ) -> Result<(), LogError> { + let intent = self.intent(change)?; + if self.fleet.is_none() && (self.repair.is_some() || intent.repair.is_some()) { + return Err(invalid()); + } + self.admit_scoped(change, conf, history) + } + pub(crate) fn admit(&self, change: &ConfChangeV2, conf: &ConfState) -> Result<(), LogError> { if change.changes.len() > MAX_MEMBERS { return Err(LogError::Overloaded); diff --git a/crates/cellar/durability/raft/mod.rs b/crates/cellar/durability/raft/mod.rs index e81f7618c..46745f5e5 100644 --- a/crates/cellar/durability/raft/mod.rs +++ b/crates/cellar/durability/raft/mod.rs @@ -8,6 +8,7 @@ pub(crate) mod budget; pub(crate) mod cache; pub(crate) mod catalog; mod catalog_wire; +pub(crate) mod cell_groups; pub(crate) mod completion; pub(crate) mod coordination_index; pub(crate) mod disaster; @@ -16,9 +17,11 @@ pub(crate) mod execution; mod filesystem; pub(crate) mod fleet_catalog; pub(crate) mod fleet_stop; +pub(crate) mod group_host; pub(crate) mod host; mod host_archive; mod host_sql; +pub(crate) mod restart_leadership; pub(crate) use host_sql::ResourceObserver as SqlResourceObserver; pub(crate) mod fleet_inspection_wire; pub(crate) mod inspection; @@ -36,7 +39,6 @@ pub(crate) mod physical_inventory; mod query; mod query_identity; -mod query_route; mod query_wire; pub(crate) mod receipt_maintenance; mod receipts; diff --git a/crates/cellar/durability/raft/node_directory.rs b/crates/cellar/durability/raft/node_directory.rs index 7c73cf81f..052793cdb 100644 --- a/crates/cellar/durability/raft/node_directory.rs +++ b/crates/cellar/durability/raft/node_directory.rs @@ -172,6 +172,63 @@ pub struct NodePublication { } impl NamespaceCatalog { + /// Initial members' immutable pins remain available after host loss. A new + /// node ID cannot disguise reuse of an old disk lineage or engine instance. + pub(crate) async fn verify_recovery_storage_fresh( + &self, + log: &ReplicaLog, + source_nodes: &[NodeId], + ) -> Result<(), CatalogError> { + if source_nodes.len() > 64 { + return Err(CatalogError::Limit); + } + let local = log.limits(); + let instance = log.storage_instance().map_err(CatalogError::Log)?; + for node in source_nodes { + let bytes = self + .inner + .read_versioned_bounded( + &node_key(&self.inner, self.config.cluster, *node, "pin"), + self.inner.limit, + ) + .await? + .ok_or(CatalogError::Corrupt("missing recovery source storage pin"))? + .0; + let pin: Pin = wire::decode(NODE_PIN, &bytes, self.inner.limit)?; + let disk = DiskLineage::new(pin.disk) + .map_err(|_| CatalogError::Corrupt("source storage lineage"))?; + let old_instance = pin.validate(self.config.cluster, *node, disk)?; + if *node == local.node || disk == local.disk || old_instance == instance { + return Err(CatalogError::Conflict("reused recovery storage")); + } + } + Ok(()) + } + + /// Re-admission requires the existing immutable pin; it must not recreate a + /// missing physical claim from a caller's proposed engine identity. + pub(crate) async fn verify_node_storage_identity( + &self, + node: NodeId, + disk: DiskLineage, + engine: EngineInstanceId, + ) -> Result<(), CatalogError> { + let bytes = self + .inner + .read_versioned_bounded( + &node_key(&self.inner, self.config.cluster, node, "pin"), + self.inner.limit, + ) + .await? + .ok_or(CatalogError::LostStableState)? + .0; + let pin: Pin = wire::decode(NODE_PIN, &bytes, self.inner.limit)?; + if pin.validate(self.config.cluster, node, disk)? != engine { + return Err(CatalogError::LostStableState); + } + Ok(()) + } + /// Import reads the permanent physical pin without requiring a live /// advisory process view. Presence of an arbitrary fleet pin is insufficient. pub(crate) async fn verify_fleet_storage_pin( diff --git a/crates/cellar/durability/raft/node_inspection_transport.rs b/crates/cellar/durability/raft/node_inspection_transport.rs index 6896f27e6..b7c3fc5fd 100644 --- a/crates/cellar/durability/raft/node_inspection_transport.rs +++ b/crates/cellar/durability/raft/node_inspection_transport.rs @@ -47,6 +47,48 @@ pub(crate) struct Received { } impl RaftTransport { + pub(crate) fn install_group_fleet_gateway( + &self, + cells: &RaftCells, + ) -> Result<(), TransportError> { + let state = &self.handle.0; + let mut gateway = state + .fleet_gateway + .lock() + .map_err(|_| TransportError::Closed)?; + let registry = state.registry.lock().map_err(|_| TransportError::Closed)?; + state.admit(®istry)?; + if gateway + .as_ref() + .and_then(std::sync::Weak::upgrade) + .is_some() + || cells.is_closed() + || !cells.handle.domain.same_owner(&state.domain) + || !cells.uses_transport(self) + { + return Err(TransportError::Refused); + } + *gateway = Some(Arc::downgrade(&cells.handle)); + Ok(()) + } + + /// Read the exact complete trust cut already installed in this transport. + /// A separate catalog refresh cannot acknowledge installed routing state. + pub(crate) fn installed_fleet_node( + &self, + node: NodeId, + ) -> Result { + let state = &self.handle.0; + let registry = state.registry.lock().map_err(|_| TransportError::Closed)?; + state.admit(®istry)?; + registry + .fleet_trust + .as_ref() + .and_then(|trust| trust.node(node)) + .cloned() + .ok_or(TransportError::Refused) + } + pub(crate) fn inspect_fleet_node( &self, node: NodeId, @@ -459,7 +501,7 @@ impl State { .checked_add(request.timeout_ms) .ok_or(TransportError::Limit)?; let handle = self - .query_gateway + .fleet_gateway .lock() .map_err(|_| TransportError::Closed)? .as_ref() @@ -471,7 +513,7 @@ impl State { .filter(|ms| *ms > 0) .ok_or(TransportError::Timeout)?; let result = if request.physical { - match (RaftCells { handle }) + match RaftCells::from_internal_handle(handle) .observe_physical_node(std::time::Duration::from_millis(remaining)) .await { @@ -482,7 +524,7 @@ impl State { Err(error) => Err(error), } } else { - match (RaftCells { handle }) + match RaftCells::from_internal_handle(handle) .observe_node(std::time::Duration::from_millis(remaining)) .await { diff --git a/crates/cellar/durability/raft/node_inspection_wire.rs b/crates/cellar/durability/raft/node_inspection_wire.rs index 91a1c1610..21f2fe25c 100644 --- a/crates/cellar/durability/raft/node_inspection_wire.rs +++ b/crates/cellar/durability/raft/node_inspection_wire.rs @@ -162,6 +162,7 @@ struct Body { declared_namespaces: u32, registered_histories: u32, dormant_histories: u32, + #[serde(rename = "pending_group_registrations")] pending_history_registrations: u32, unattached_engine_histories: u32, seed_workers: u32, @@ -211,7 +212,6 @@ pub(crate) fn validate(view: &RaftHostObservation) -> Result<(), TransportError> || !(1..=64).contains(&view.declared_namespaces) || view.registered_histories > 4096 || view.dormant_histories > view.registered_histories - || view.pending_history_registrations > 1 || view.node.groups > 4097 || view.unattached_engine_histories > view.node.groups || view diff --git a/crates/cellar/durability/raft/node_inspection_wire_tests.rs b/crates/cellar/durability/raft/node_inspection_wire_tests.rs index 6b14d72c2..216b650c7 100644 --- a/crates/cellar/durability/raft/node_inspection_wire_tests.rs +++ b/crates/cellar/durability/raft/node_inspection_wire_tests.rs @@ -84,7 +84,7 @@ fn node_metadata_rejects_missing_invalid_and_unbounded_fields() { "startup", "current", "profile", - "pending_history_registrations", + "pending_group_registrations", "boot", "bounds", "seed_workers", @@ -122,3 +122,27 @@ fn node_metadata_rejects_missing_invalid_and_unbounded_fields() { assert!(decode(&serde_json::to_vec(&changed).unwrap()).is_err()); assert!(decode(&vec![0; MAX_BODY + 1]).is_err()); } + +#[test] +fn node_metadata_preserves_concurrent_group_admission_debt() { + let mut observed = view(); + observed.pending_history_registrations = 7; + let bytes = encode(&observed).unwrap(); + let decoded = decode(&bytes).unwrap(); + assert_eq!(decoded.pending_history_registrations, 7); + assert_eq!(decoded.node.groups, 0); + assert_eq!(decoded.registered_histories, 0); + // A durable engine scan with zero slots cannot erase admitted provisioning + // debt. The old single-history counter is rejected by the hard cutover. + let mut old: serde_json::Value = serde_json::from_slice(&bytes).unwrap(); + let n = old + .as_object_mut() + .unwrap() + .remove("pending_group_registrations") + .unwrap(); + old["pending_history_registrations"] = n; + assert!(decode(&serde_json::to_vec(&old).unwrap()).is_err()); + let mut overflow: serde_json::Value = serde_json::from_slice(&bytes).unwrap(); + overflow["pending_group_registrations"] = serde_json::json!(u64::from(u32::MAX) + 1); + assert!(decode(&serde_json::to_vec(&overflow).unwrap()).is_err()); +} diff --git a/crates/cellar/durability/raft/operations.rs b/crates/cellar/durability/raft/operations.rs index ab395b80e..ce5488069 100644 --- a/crates/cellar/durability/raft/operations.rs +++ b/crates/cellar/durability/raft/operations.rs @@ -75,13 +75,59 @@ pub struct RaftCloseReport { /// joining the complete domain. Missing startup/close proof remains explicit. #[derive(Clone, Debug, PartialEq, Eq)] pub struct RaftNodeShutdownReport { - pub result: Result, + pub result: + Result, pub startup_incomplete: bool, /// Best-effort handoff under the original Node shutdown deadline. Missing /// evidence is not a graceful drain, even when physical closure succeeded. pub drain: Option>, } +/// Preserve the joined closure failure, including both controller and log +/// failures. Cloned waiters share one failure identity; admission errors remain +/// comparable by value. This error never resolves an individual mutation. +#[derive(Clone, Debug)] +pub enum RaftNodeShutdownError { + Admission(CommandFailure), + Closure(Arc), +} + +impl From for RaftNodeShutdownError { + fn from(error: CommandFailure) -> Self { + Self::Admission(error) + } +} + +impl PartialEq for RaftNodeShutdownError { + fn eq(&self, other: &Self) -> bool { + match (self, other) { + (Self::Admission(a), Self::Admission(b)) => a == b, + (Self::Closure(a), Self::Closure(b)) => Arc::ptr_eq(a, b), + _ => false, + } + } +} + +impl Eq for RaftNodeShutdownError {} + +impl std::fmt::Display for RaftNodeShutdownError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::Admission(error) => std::fmt::Display::fmt(error, f), + Self::Closure(error) => std::fmt::Display::fmt(error, f), + } + } +} + +impl std::error::Error for RaftNodeShutdownError { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + Some(match self { + Self::Admission(error) => error, + Self::Closure(error) => error.as_ref(), + }) + } +} + /// Each registered local replica receives at most one bounded handoff turn. /// Waiting for admitted group work and inspecting a successor share that turn's /// deadline; a busy group cannot renew its budget or delay every other group. @@ -101,7 +147,7 @@ impl RaftNodeShutdownReport { pub(crate) fn cli_summary(&self) -> serde_json::Value { match &self.result { Ok(report) => { - serde_json::json!({"owners_closed":true,"startup_incomplete":self.startup_incomplete,"groups":report.groups_total,"omitted_group_details":report.omitted_groups,"groups_with_archive_lag":report.groups_with_archive_lag,"groups_with_unknown_archive":report.groups_with_unknown_archive,"unfinished_commands":report.unfinished.len(),"retain_replica_disks":true,"handoff":self.drain.as_ref().map(|result|match result {Ok(drain)=>serde_json::json!({"groups_examined":drain.groups_examined,"attempts":drain.handoff_attempts,"accepted":drain.handoffs_accepted,"leadership_released":drain.leadership_released,"unavailable_groups":drain.unavailable_groups,"deadline_elapsed":drain.deadline_elapsed}),Err(error)=>serde_json::json!({"error":error.to_string()})})}) + serde_json::json!({"owners_closed":true,"startup_incomplete":self.startup_incomplete,"groups":report.groups_total,"omitted_group_details":report.omitted_groups,"groups_with_archive_lag":report.groups_with_archive_lag,"groups_with_unknown_archive":report.groups_with_unknown_archive,"mutation_outcomes":report.mutation_outcomes,"retained_calls_at_shutdown":report.retained_calls_at_shutdown,"retain_replica_disks":true,"handoff":self.drain.as_ref().map(|result|match result {Ok(drain)=>serde_json::json!({"groups_examined":drain.groups_examined,"attempts":drain.handoff_attempts,"accepted":drain.handoffs_accepted,"leadership_released":drain.leadership_released,"unavailable_groups":drain.unavailable_groups,"deadline_elapsed":drain.deadline_elapsed}),Err(error)=>serde_json::json!({"error":error.to_string()})})}) } Err(error) => { serde_json::json!({"owners_closed":false,"startup_incomplete":self.startup_incomplete,"error":error.to_string(),"retain_replica_disks":true}) @@ -258,7 +304,10 @@ impl NodeLifecycle { pub(crate) fn resource_usage( &self, - ) -> Result, CommandFailure> { + ) -> Result< + Option, + CommandFailure, + > { Ok(self .sample_service()? .map(|service| service.resource_usage())) @@ -312,7 +361,7 @@ impl NodeLifecycle { Ok(state) => state, Err(_) => { return Some(RaftNodeShutdownReport { - result: Err(CommandFailure::Unavailable), + result: Err(CommandFailure::Unavailable.into()), startup_incomplete: true, drain: None, }) @@ -333,13 +382,16 @@ impl NodeLifecycle { None => None, }; RaftNodeShutdownReport { - result: service.close().await, + result: service + .close() + .await + .map_err(RaftNodeShutdownError::Closure), startup_incomplete: false, drain, } } None => RaftNodeShutdownReport { - result: Err(CommandFailure::Unavailable), + result: Err(CommandFailure::Unavailable.into()), startup_incomplete: true, drain: None, }, diff --git a/crates/cellar/durability/raft/options.rs b/crates/cellar/durability/raft/options.rs index f1b641ad9..9fe3ca813 100644 --- a/crates/cellar/durability/raft/options.rs +++ b/crates/cellar/durability/raft/options.rs @@ -25,6 +25,7 @@ pub struct ReplicaLogOptions { pub max_batch_bytes: usize, pub max_cache_bytes: usize, pub max_entries_per_group: usize, + pub cell_group_retention: super::cell_groups::CellGroupLogRetentionLimits, pub target_file_bytes: u64, pub purge_threshold_bytes: u64, /// Minimum interval between dirty-engine purge passes on the storage worker. @@ -56,6 +57,7 @@ impl ReplicaLogOptions { max_batch_bytes: 32 * 1024 * 1024, max_cache_bytes: 256 * 1024 * 1024, max_entries_per_group: 16_384, + cell_group_retention: Default::default(), target_file_bytes: 16 * 1024 * 1024, purge_threshold_bytes: 64 * 1024 * 1024, purge_interval_ms: Some(1000), @@ -64,6 +66,7 @@ impl ReplicaLogOptions { } pub(crate) fn validate(&self) -> Result<(), LogError> { + self.cell_group_retention.validate()?; self.snapshot_scratch .validate() .map_err(|e| LogError::Invalid(e.to_string()))?; @@ -100,8 +103,9 @@ pub struct RaftRuntimeOptions { pub tick_interval_ms: Option, pub election_seed: Option<[u8; 32]>, pub max_groups: usize, - /// Ordinary requests. Eight additional queue slots and 32 KiB are reserved - /// for physical apply/snapshot completions so saturated work can settle. + /// Ordinary requests. Eight additional queue slots are reserved for physical + /// completions so saturated work can settle. Cell-group completion requests + /// bound retained manifest metadata with max_ready_bytes. /// Transport hint admission has a separate 256 slots/1 MiB; the worker /// coalesces retained hints to one small record per current group member. pub max_requests: usize, @@ -121,7 +125,18 @@ pub struct RaftRuntimeOptions { pub max_message_bytes: usize, pub max_proposal_bytes: usize, pub max_members_per_group: usize, + /// Bounds protocol read states and a separate shared-cell read/pin budget. + /// At most (max_pending_reads / 8).clamp(1, 32) read pins belong to one group. pub max_pending_reads: usize, + /// Logical image bytes plus 4 KiB per waiting read or selected pin, retained + /// through query execution. Separate from transport request capacity; both + /// node and group pin admission use this byte ceiling. No memory is allocated + /// merely by configuring this limit. + pub max_read_pin_bytes: usize, + /// Shared-group creation admission counts applied cells and pending accepted + /// enrollments. A lower limit does not prevent existing cells from serving. + /// max_requests also bounds leader-local pending cell acceptance records. + pub max_cells_per_group: u32, } impl Default for RaftRuntimeOptions { @@ -142,6 +157,8 @@ impl Default for RaftRuntimeOptions { max_proposal_bytes: 16 * 1024 * 1024, max_members_per_group: 7, max_pending_reads: 256, + max_read_pin_bytes: 64 * 1024 * 1024, + max_cells_per_group: 1_000_000, } } } @@ -177,6 +194,12 @@ impl RaftRuntimeOptions { || self.max_members_per_group > 16 || self.max_pending_reads == 0 || self.max_pending_reads > 4096 + || self.max_read_pin_bytes < 4096 + || self.max_read_pin_bytes as u128 + > (cellar_ltx::stream_snapshot::MAX_IMAGE_BYTES as u128 + 4096) + * self.max_pending_reads as u128 + || self.max_cells_per_group == 0 + || self.max_cells_per_group > 1_000_000 { return Err(super::completion::RuntimeError::Invalid( "invalid runtime bounds".into(), @@ -185,3 +208,39 @@ impl RaftRuntimeOptions { Ok(()) } } + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn read_pin_capacity_is_independent_of_transport_and_bounds_images() { + let log = ReplicaLogOptions::new( + PathBuf::from("read-bounds"), + NodeId::new([1; 16]).unwrap(), + DiskLineage::new([2; 16]).unwrap(), + ProcessGeneration::new([3; 16]).unwrap(), + LogOpenMode::Create, + ); + let mut options = RaftRuntimeOptions::default(); + assert!(options.validate(&log).is_ok()); + let maximum_pin = cellar_ltx::stream_snapshot::MAX_IMAGE_BYTES as u128 + 4096; + // On narrower platforms the accounting type itself bounds the image. + options.max_read_pin_bytes = usize::try_from(maximum_pin).unwrap_or(usize::MAX); + assert_eq!(options.max_request_bytes, 64 * 1024 * 1024); + assert!(options.validate(&log).is_ok()); + options.max_read_pin_bytes = 4095; + assert!(options.validate(&log).is_err()); + options.max_read_pin_bytes = 4096; + assert!(options.validate(&log).is_ok()); + let envelope = maximum_pin * options.max_pending_reads as u128; + if let Ok(envelope) = usize::try_from(envelope) { + options.max_read_pin_bytes = envelope; + assert!(options.validate(&log).is_ok()); + if let Some(too_large) = envelope.checked_add(1) { + options.max_read_pin_bytes = too_large; + assert!(options.validate(&log).is_err()); + } + } + } +} diff --git a/crates/cellar/durability/raft/park_transport.rs b/crates/cellar/durability/raft/park_transport.rs new file mode 100644 index 000000000..1e695d6d9 --- /dev/null +++ b/crates/cellar/durability/raft/park_transport.rs @@ -0,0 +1,664 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Park acknowledgements use the physical group registry and control lane. +//! A response is bound to one request and its TLS session. + +use super::snapshots::{read_bulk_head, Wire}; +use super::*; +use crate::raft::cell_groups::{ + GroupParkCut, ParkAcknowledgement, ParkCertificate, SelectedParkCertificate, + MAX_PARK_NEGOTIATION_MS, +}; +use cellar_logic::{cell_group::GroupRoute as CellRoute, consensus::EngineInstanceId}; +use sha2::{Digest, Sha256}; + +pub(super) const REQUEST_MAGIC: &[u8; 4] = b"CLPQ"; +const RESPONSE_MAGIC: &[u8; 4] = b"CLPR"; +pub(super) const EXPORT_LABEL: &[u8] = b"EXPORTER-cellar-raft-park-1"; +const REQUEST_PURPOSE: &[u8] = b"cellar-park-ack-request-1"; +const RESPONSE_PURPOSE: &[u8] = b"cellar-park-ack-response-1"; +const RETRY_MS: u64 = 50; +const MAX_PARK_BODY: usize = 16 * 1024; + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Request { + route: Route, + cut: GroupParkCut, + timeout_ms: u64, + #[serde(default, skip_serializing_if = "Option::is_none")] + certificate_digest: Option<[u8; 32]>, +} + +impl Request { + fn deadline(&self, connected_at: u64, now: u64) -> Result { + if !(1..=MAX_PARK_NEGOTIATION_MS).contains(&self.timeout_ms) || now < connected_at { + return Err(TransportError::Protocol); + } + connected_at + .checked_add(self.timeout_ms) + .filter(|deadline| now < *deadline) + .ok_or(TransportError::Timeout) + } + + fn certificate(&self, body: &[u8]) -> Result, TransportError> { + match self.certificate_digest { + None if body.is_empty() => Ok(None), + Some(digest) + if !body.is_empty() && digest == <[u8; 32]>::from(Sha256::digest(body)) => + { + let certificate: ParkCertificate = + serde_json::from_slice(body).map_err(|_| TransportError::Protocol)?; + if certificate.cut() != &self.cut { + return Err(TransportError::Identity); + } + Ok(Some(certificate)) + } + _ => Err(TransportError::Authentication), + } + } +} + +#[derive(Serialize, Deserialize)] +#[serde(rename_all = "snake_case", tag = "status", deny_unknown_fields)] +enum Reply { + Acknowledged { sequence: u64 }, + Parked, + Retry, + Refused, +} + +impl Reply { + fn acknowledgement(self, request: Request) -> Result { + let (leader, voter) = request.route.decode_group()?; + match self { + Self::Acknowledged { sequence } + if sequence != 0 && request.certificate_digest.is_none() => + { + Ok(ParkAcknowledgement { + voter, + leader: leader.member, + cut: request.cut, + sync_sequence: sequence, + }) + } + Self::Acknowledged { .. } | Self::Parked => Err(TransportError::Protocol), + Self::Retry => Err(RuntimeError::NotReady.into()), + Self::Refused => Err(TransportError::Refused), + } + } + + fn parked(self) -> Result<(), TransportError> { + match self { + Self::Parked => Ok(()), + Self::Acknowledged { .. } => Err(TransportError::Protocol), + Self::Retry => Err(RuntimeError::NotReady.into()), + Self::Refused => Err(TransportError::Refused), + } + } +} + +struct Peer { + route: RaftPeer, + pool: Arc, +} + +/// The authenticated delivery turn observed a synced parked record on every +/// voter in one exact physical registration. It is a historical observation. +pub(crate) struct ParkDeliveryReceipt { + leader: ReplicaIdentity, + engine: EngineInstanceId, + cut: GroupParkCut, +} + +impl ParkDeliveryReceipt { + pub(crate) fn leader(&self) -> &ReplicaIdentity { + &self.leader + } + + pub(crate) fn engine(&self) -> EngineInstanceId { + self.engine + } + + pub(crate) fn cut(&self) -> &GroupParkCut { + &self.cut + } +} + +struct Roster { + application: CellRoute, + registration: Arc>, + local: ReplicaIdentity, + remote: Vec, +} + +impl Roster { + fn voters(&self) -> Vec> { + self.registration + .peers + .iter() + .map(|peer| peer.replica.clone()) + .collect() + } + + fn matches_certificate( + &self, + leader: &ReplicaIdentity, + certificate: &ParkCertificate, + ) -> bool { + certificate.matches_roster(&self.application, leader, &self.voters()) + } +} + +impl RaftTransport { + /// Collect every voter in the exact current registration. A partial result + /// cannot reach certificate selection. Cancellation drops the exchange; + /// already-submitted local journal work settles through the Raft worker. + pub(crate) async fn collect_park_certificate( + &self, + history: GroupHistory, + cut: GroupParkCut, + deadline: u64, + ) -> Result { + let state = self.handle.0.clone(); + let now = state.domain.mono_ms().map_err(|_| TransportError::Closed)?; + if deadline <= now || deadline - now > MAX_PARK_NEGOTIATION_MS { + return Err(TransportError::Timeout); + } + self.reconcile_managed_physical(&history)?; + let roster = state.park_roster(history, &cut)?; + let (mut reply, wait) = oneshot::channel(); + let worker = state.clone(); + state.spawn(async move { + let result = crate::asyncrt::select! { + _ = reply.closed() => return, + result = worker.seed_until(deadline, worker.collect_park(roster, cut, deadline)) => result, + }; + let _ = reply.send(result); + })?; + wait.await.map_err(|_| TransportError::Closed)? + } + + /// Park the leader before sending the certificate so its routine + /// heartbeats cannot interrupt followers' installation. An incomplete + /// delivery leaves ordinary peer traffic free to wake the parked voters. + pub(crate) async fn deliver_park_certificate( + &self, + history: GroupHistory, + selected: &SelectedParkCertificate, + deadline: u64, + ) -> Result { + let state = self.handle.0.clone(); + let now = state.domain.mono_ms().map_err(|_| TransportError::Closed)?; + if deadline <= now || deadline - now > MAX_PARK_NEGOTIATION_MS { + return Err(TransportError::Timeout); + } + self.reconcile_managed_physical(&history)?; + let certificate = selected.certificate().clone(); + let roster = state.park_roster(history, certificate.cut())?; + if !roster.matches_certificate(&roster.local, &certificate) { + return Err(TransportError::StalePeer); + } + let (mut reply, wait) = oneshot::channel(); + let worker = state.clone(); + state.spawn(async move { + let result = crate::asyncrt::select! { + _ = reply.closed() => return, + result = worker.seed_until(deadline, worker.deliver_park(roster, certificate, deadline)) => result, + }; + let _ = reply.send(result); + })?; + wait.await.map_err(|_| TransportError::Closed)? + } +} + +impl State { + fn park_roster( + &self, + history: GroupHistory, + cut: &GroupParkCut, + ) -> Result { + let group = self + .runtime + .shared()? + .log() + .cell_group(&history) + .map_err(|_| TransportError::UnknownGroup)?; + let (application, voters) = { + let cache = group.read().map_err(RuntimeError::from)?; + if !cache.quiescent_park_cut(cut) { + return Err(RuntimeError::NotReady.into()); + } + let marker = cache + .group_manifest + .as_ref() + .ok_or(TransportError::UnknownGroup)?; + (marker.route().clone(), cache.conf_state.voters.clone()) + }; + let local = group.identity().clone(); + let registry = self.registry.lock().map_err(|_| TransportError::Closed)?; + self.admit(®istry)?; + let registration = registry + .physical_groups + .get(&history) + .ok_or(TransportError::UnknownGroup)? + .clone(); + if registration.schema_digest != application.schema_digest + || registration.peers.len() != voters.len() + || !registration.peers.iter().any(|peer| peer.replica == local) + || !registration + .peers + .iter() + .all(|peer| voters.contains(&peer.replica.member.get())) + { + return Err(TransportError::StalePeer); + } + let remote = registration + .peers + .iter() + .filter(|peer| peer.replica != local) + .map(|peer| { + let pool = registry + .pools + .get(&peer.replica.node) + .filter(|pool| pool.peer.same_host(peer)) + .ok_or(TransportError::StalePeer)?; + Ok(Peer { + route: peer.clone(), + pool: pool.clone(), + }) + }) + .collect::>()?; + Ok(Roster { + application, + registration, + local, + remote, + }) + } + + async fn collect_park( + &self, + roster: Roster, + cut: GroupParkCut, + deadline: u64, + ) -> Result { + let runtime = self.runtime.shared()?; + let history = roster.local.scope; + let voters = roster.voters(); + let group = runtime + .log() + .cell_group(&history) + .map_err(RuntimeError::from)?; + let prior = group + .read() + .map_err(RuntimeError::from)? + .local_park + .as_ref() + .and_then(|park| park.certificate()) + .cloned(); + let certificate = if let Some(prior) = prior { + // A lost reply can be retried without substituting new sync + // sequences for an already durable certificate. + if prior.cut() != &cut + || !prior.matches_roster(&roster.application, &roster.local, &voters) + { + return Err(TransportError::StalePeer); + } + prior + } else { + let local = self + .retry_park(deadline, || async { + Ok(runtime + .acknowledge_park(history, roster.local.member, cut.clone()) + .await?) + }) + .await?; + let mut acknowledgements = futures_util::future::try_join_all( + roster + .remote + .iter() + .map(|peer| self.request_park_ack(&roster, peer, &cut, deadline)), + ) + .await?; + acknowledgements.push(local); + ParkCertificate::assemble( + &roster.application, + cut, + &roster.local, + &voters, + acknowledgements, + ) + .map_err(RuntimeError::from)? + }; + // Resolve the roster again after the exchanges. A replaced transport + // registration or physical process invalidates the entire selection. + let current = self.park_roster(history, certificate.cut())?; + if !Arc::ptr_eq(¤t.registration, &roster.registration) + || current.local != roster.local + || current.application != roster.application + { + return Err(TransportError::StalePeer); + } + let selected = self + .retry_park(deadline, || async { + Ok(runtime.certify_park(history, certificate.clone()).await?) + }) + .await?; + // The leader may have synced while membership discovery was refreshed. + // A lost selection reply is safe and the negotiation can expire. + if !Arc::ptr_eq( + &self + .park_roster(history, selected.certificate().cut())? + .registration, + &roster.registration, + ) { + return Err(TransportError::StalePeer); + } + Ok(selected) + } + + async fn deliver_park( + &self, + roster: Roster, + certificate: ParkCertificate, + deadline: u64, + ) -> Result { + let body = serde_json::to_vec(&certificate).map_err(|_| TransportError::Protocol)?; + if body.len() > MAX_PARK_BODY { + return Err(TransportError::Limit); + } + let runtime = self.runtime.shared()?; + let history = roster.local.scope; + self.retry_park(deadline, || async { + Ok(runtime.enter_park(history, certificate.clone()).await?) + }) + .await?; + let cut = certificate.cut(); + futures_util::future::try_join_all(roster.remote.iter().map(|peer| async { + let (_, reply) = self + .request_park(&roster, peer, cut, &body, deadline) + .await?; + reply.parked() + })) + .await?; + let group = runtime + .log() + .cell_group(&history) + .map_err(RuntimeError::from)?; + if !group + .read() + .map_err(RuntimeError::from)? + .local_park + .as_ref() + .is_some_and(|park| park.parked() && park.certificate() == Some(&certificate)) + { + return Err(TransportError::StalePeer); + } + if !Arc::ptr_eq( + &self.park_roster(history, cut)?.registration, + &roster.registration, + ) { + return Err(TransportError::StalePeer); + } + Ok(ParkDeliveryReceipt { + leader: roster.local, + engine: runtime + .log() + .storage_instance() + .map_err(RuntimeError::from)?, + cut: cut.clone(), + }) + } + + fn park_charge(&self) -> usize { + 2 * (self.options.max_header_bytes + frame::PREFIX + frame::TAG) + MAX_PARK_BODY + 1024 + } + + async fn request_park_ack( + &self, + roster: &Roster, + peer: &Peer, + cut: &GroupParkCut, + deadline: u64, + ) -> Result { + let (request, reply) = self.request_park(roster, peer, cut, &[], deadline).await?; + reply.acknowledgement(request) + } + + async fn request_park( + &self, + roster: &Roster, + peer: &Peer, + cut: &GroupParkCut, + body: &[u8], + deadline: u64, + ) -> Result<(Request, Reply), TransportError> { + let wire_route = Route::new_group( + &roster.local, + self.auth.source(), + &peer.route.replica, + &peer.route.label, + roster.application.schema_digest, + )?; + self.retry_park(deadline, || async { + if !self.current_physical(&roster.registration, &peer.route) { + return Err(TransportError::StalePeer); + } + let result = self + .bounded( + Some(&peer.pool), + self.park_exchange(peer, &wire_route, cut, body, deadline), + ) + .await?; + if !self.current_physical(&roster.registration, &peer.route) { + return Err(TransportError::StalePeer); + } + match result.1 { + Reply::Retry => Err(RuntimeError::NotReady.into()), + Reply::Refused => Err(TransportError::Refused), + _ => Ok(result), + } + }) + .await + } + + async fn retry_park>>( + &self, + deadline: u64, + mut attempt: impl FnMut() -> F, + ) -> Result { + loop { + let now = self.domain.mono_ms().map_err(|_| TransportError::Closed)?; + if now >= deadline { + return Err(TransportError::Timeout); + } + match attempt().await { + Err( + TransportError::Runtime(RuntimeError::NotReady | RuntimeError::Overloaded) + | TransportError::Overloaded, + ) => { + let now = self.domain.mono_ms().map_err(|_| TransportError::Closed)?; + self.domain + .sleep_until(now.saturating_add(RETRY_MS).min(deadline)) + .map_err(|_| TransportError::Closed)? + .await; + } + result => return result, + } + } + } + + async fn park_exchange( + &self, + peer: &Peer, + route: &Route, + cut: &GroupParkCut, + body: &[u8], + deadline: u64, + ) -> Result<(Request, Reply), TransportError> { + let charge = self.park_charge(); + let _node = self + .outgoing + .control + .acquire(charge) + .ok_or(TransportError::Overloaded)?; + let _peer = peer + .pool + .outgoing + .control + .acquire(charge) + .ok_or(TransportError::Overloaded)?; + let _connection = self.outgoing_connections.acquire(Lane::Control); + let tcp = TcpStream::connect(peer.route.control_address).await?; + tcp.set_nodelay(true)?; + let tcp = self.observed_socket(tcp, metrics::Lane::Control); + let name = rustls::pki_types::ServerName::try_from(peer.route.server_name.clone()) + .map_err(|_| TransportError::Identity)?; + let mut stream = tokio_rustls::TlsConnector::from(self.tls.client.clone()) + .connect(name, tcp) + .await + .map_err(|error| TransportError::Tls(error.to_string()))?; + let exporter = stream + .get_ref() + .1 + .export_keying_material([0; 32], EXPORT_LABEL, None) + .map_err(|error| TransportError::Tls(error.to_string()))?; + let timeout_ms = deadline + .checked_sub(self.domain.mono_ms().map_err(|_| TransportError::Closed)?) + .filter(|remaining| *remaining > 0) + .ok_or(TransportError::Timeout)?; + let request = Request { + route: route.clone(), + cut: cut.clone(), + timeout_ms, + certificate_digest: (!body.is_empty()).then(|| Sha256::digest(body).into()), + }; + let wire = Wire { + auth: &self.auth, + exporter: &exporter, + options: &self.options, + max_body: MAX_PARK_BODY, + }; + let head = wire.encode(REQUEST_MAGIC, REQUEST_PURPOSE, &[], &request, body)?; + write_frame(&mut stream, &head, body).await?; + stream.flush().await?; + let response = read_bulk_head(&mut stream, None, RESPONSE_MAGIC, 0, &self.options).await?; + let reply: Reply = + wire.decode(RESPONSE_MAGIC, RESPONSE_PURPOSE, &response, Some(&head.tag))?; + Ok((request, reply)) + } + + pub(super) async fn receive_park( + &self, + stream: &mut S, + exporter: &[u8; 32], + prefix: [u8; frame::PREFIX], + connected_at: u64, + ) -> Result<(), TransportError> { + let charge = self.park_charge(); + let _node = self + .incoming + .control + .acquire(charge) + .ok_or(TransportError::Overloaded)?; + let head = self + .bounded( + None, + read_bulk_head( + stream, + Some(prefix), + REQUEST_MAGIC, + MAX_PARK_BODY, + &self.options, + ), + ) + .await?; + let wire = Wire { + auth: &self.auth, + exporter, + options: &self.options, + max_body: MAX_PARK_BODY, + }; + let request: Request = wire.decode(REQUEST_MAGIC, REQUEST_PURPOSE, &head, None)?; + let deadline = request.deadline( + connected_at, + self.domain.mono_ms().map_err(|_| TransportError::Closed)?, + )?; + let (leader, voter) = request.route.decode_group()?; + let pool = self.authorize_physical(&request.route)?; + let _peer = pool + .incoming + .control + .acquire(charge) + .ok_or(TransportError::Overloaded)?; + self.seed_until( + deadline, + self.bounded(Some(&pool), async { + let mut body = vec![0; head.payload_len]; + if !body.is_empty() { + stream.read_exact(&mut body).await?; + } + let response = if let Some(certificate) = request.certificate(&body)? { + let roster = self.park_roster(voter.scope, certificate.cut())?; + if roster.local != voter || !roster.matches_certificate(&leader, &certificate) { + return Err(TransportError::StalePeer); + } + let cut = certificate.cut().clone(); + match self + .runtime + .shared()? + .enter_park(voter.scope, certificate) + .await + { + Ok(()) + if Arc::ptr_eq( + &self.park_roster(voter.scope, &cut)?.registration, + &roster.registration, + ) => + { + Reply::Parked + } + Ok(()) => return Err(TransportError::StalePeer), + Err(RuntimeError::NotReady | RuntimeError::Overloaded) => Reply::Retry, + Err(_) => Reply::Refused, + } + } else { + match self + .runtime + .shared()? + .acknowledge_park(voter.scope, leader.member, request.cut.clone()) + .await + { + Ok(receipt) + if receipt.voter == voter + && receipt.leader == leader.member + && receipt.cut == request.cut + && receipt.sync_sequence != 0 => + { + Reply::Acknowledged { + sequence: receipt.sync_sequence, + } + } + Ok(_) => return Err(TransportError::Protocol), + Err(RuntimeError::NotReady | RuntimeError::Overloaded) => Reply::Retry, + Err(_) => Reply::Refused, + } + }; + if !Arc::ptr_eq(&self.authorize_physical(&request.route)?, &pool) { + return Err(TransportError::StalePeer); + } + let response = + wire.encode(RESPONSE_MAGIC, RESPONSE_PURPOSE, &head.tag, &response, &[])?; + write_head(stream, &response).await?; + stream.flush().await?; + Ok(()) + }), + ) + .await + } +} + +#[cfg(test)] +#[path = "park_transport_tests.rs"] +mod tests; diff --git a/crates/cellar/durability/raft/park_transport_tests.rs b/crates/cellar/durability/raft/park_transport_tests.rs new file mode 100644 index 000000000..e0a9e5eef --- /dev/null +++ b/crates/cellar/durability/raft/park_transport_tests.rs @@ -0,0 +1,501 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use cellar_logic::consensus::{DiskLineage, GroupId, IncarnationId, MemberId, ProcessGeneration}; + +fn route() -> anyhow::Result<( + Route, + ReplicaIdentity, + ReplicaIdentity, +)> { + let history = GroupHistory { + cluster: ClusterId::new([1; 16])?, + namespace: NamespaceId::new([2; 16])?, + group: GroupId::new([3; 16])?, + incarnation: IncarnationId::new([4; 16])?, + }; + let replica = |id| -> anyhow::Result<_> { + Ok(ReplicaIdentity { + scope: history, + member: MemberId::new(u64::from(id))?, + node: NodeId::new([id; 16])?, + disk: DiskLineage::new([id + 3; 16])?, + process: ProcessGeneration::new([id + 6; 16])?, + }) + }; + let leader = replica(1)?; + let voter = replica(2)?; + let route = Route::new_group(&leader, "leader", &voter, "voter", [5; 32])?; + Ok((route, leader, voter)) +} + +#[test] +fn park_reply_is_bound_to_tls_request_cut_and_process() -> anyhow::Result<()> { + let auth = PeerAuth::new([8; 32], "leader")?; + let options = TransportOptions::default(); + let wire = Wire { + auth: &auth, + exporter: &[9; 32], + options: &options, + max_body: 0, + }; + let (route, leader, voter) = route()?; + let cut = serde_json::from_str(r#"{"generation":3,"term":2,"membership_revision":1}"#)?; + let request = Request { + route, + cut, + timeout_ms: 100, + certificate_digest: None, + }; + assert_eq!(request.deadline(10, 109)?, 110); + assert!(matches!( + request.deadline(10, 110), + Err(TransportError::Timeout) + )); + let head = wire.encode(REQUEST_MAGIC, REQUEST_PURPOSE, &[], &request, &[])?; + let reply = wire.encode( + RESPONSE_MAGIC, + RESPONSE_PURPOSE, + &head.tag, + &Reply::Acknowledged { sequence: 6 }, + &[], + )?; + + let mut rebooted = voter.clone(); + rebooted.process = ProcessGeneration::new([19; 16])?; + let other_route = Route::new_group(&leader, "leader", &rebooted, "voter", [5; 32])?; + let other = Request { + route: other_route, + cut: request.cut.clone(), + timeout_ms: request.timeout_ms, + certificate_digest: None, + }; + let other_head = wire.encode(REQUEST_MAGIC, REQUEST_PURPOSE, &[], &other, &[])?; + assert!(matches!( + wire.decode::( + RESPONSE_MAGIC, + RESPONSE_PURPOSE, + &reply, + Some(&other_head.tag) + ), + Err(TransportError::Authentication) + )); + let other_cut = Request { + route: request.route.clone(), + cut: serde_json::from_str(r#"{"generation":4,"term":2,"membership_revision":1}"#)?, + timeout_ms: request.timeout_ms, + certificate_digest: None, + }; + let other_head = wire.encode(REQUEST_MAGIC, REQUEST_PURPOSE, &[], &other_cut, &[])?; + assert!(matches!( + wire.decode::( + RESPONSE_MAGIC, + RESPONSE_PURPOSE, + &reply, + Some(&other_head.tag) + ), + Err(TransportError::Authentication) + )); + let install = Request { + route: request.route.clone(), + cut: request.cut.clone(), + timeout_ms: request.timeout_ms, + certificate_digest: Some(Sha256::digest(b"certificate").into()), + }; + let install_head = wire.encode(REQUEST_MAGIC, REQUEST_PURPOSE, &[], &install, b"")?; + assert!(matches!( + wire.decode::( + RESPONSE_MAGIC, + RESPONSE_PURPOSE, + &reply, + Some(&install_head.tag) + ), + Err(TransportError::Authentication) + )); + assert!(matches!( + install.certificate(b"modified certificate"), + Err(TransportError::Authentication) + )); + let other_wire = Wire { + exporter: &[10; 32], + ..wire + }; + assert!(matches!( + other_wire.decode::(RESPONSE_MAGIC, RESPONSE_PURPOSE, &reply, Some(&head.tag)), + Err(TransportError::Authentication) + )); + + let decoded: Reply = wire.decode(RESPONSE_MAGIC, RESPONSE_PURPOSE, &reply, Some(&head.tag))?; + let expected_cut = request.cut.clone(); + let receipt = decoded.acknowledgement(request)?; + assert_eq!(receipt.voter, voter); + assert_eq!(receipt.leader, leader.member); + assert_eq!(receipt.cut, expected_cut); + assert_eq!(receipt.sync_sequence, 6); + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn park_certificate_delivery_and_wake_over_three_tls_voters() -> anyhow::Result<()> { + use crate::{ + asyncrt::ExecutionDomain, + raft::{ + catalog::{Catalog, CatalogOptions, InitialMember, NamespaceConfig}, + cell_groups::{ + CellGroupApplier, CellGroupApplyImages, CellGroupBirthSpec, CellGroupEffect, + CellGroupRuntime, DirectoryLimits, ManifestPageStore, ManifestTree, + ParkPrepareRequest, RangeDirectory, + }, + options::{LogOpenMode, RaftRuntimeOptions, ReplicaLogOptions}, + storage::ReplicaLog, + }, + }; + use anyhow::Context as _; + use cellar_logic::{ + cell_group::{CellRange, MembershipRevision, ParkGeneration, RouteEpoch}, + consensus::ProposalId, + }; + use cellar_ltx::{ + generation::{GenerationStore, ReadServices}, + DirectFileSystem, + }; + + let root = tempfile::tempdir()?; + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let (_, leader, _) = route()?; + let history = leader.scope; + let catalog = Catalog::new( + Arc::new(object_store::memory::InMemory::new()), + CatalogOptions::default(), + )?; + let namespace = domain + .run(catalog.open_namespace(NamespaceConfig::new( + history.cluster, + history.namespace, + cellar_logic::NamespaceMode::Raft, + [5; 32], + ))) + .await??; + let directory = RangeDirectory::new(namespace, DirectoryLimits::default())?; + let voters = (1..=3) + .map(|id| -> anyhow::Result<_> { + Ok(InitialMember { + member: MemberId::new(u64::from(id))?, + node: NodeId::new([id; 16])?, + }) + }) + .collect::>>()?; + let birth_request = directory.prepare_group_birth(CellGroupBirthSpec { + history, + range: CellRange::all(), + epoch: RouteEpoch::new(1)?, + creation: ProposalId::new([18; 16])?, + voters: voters.clone(), + learners: vec![], + })?; + domain + .run(directory.reserve_group_birth(&birth_request)) + .await??; + let initialize = directory + .prepare_initialization(ProposalId::new([19; 16])?, &[birth_request.route().clone()])?; + domain.run(directory.publish(&initialize)).await??; + let birth = domain + .run(directory.materialize_group_birth(&birth_request)) + .await??; + + let mut peers = Vec::new(); + let mut logs = Vec::new(); + let mut runtimes = Vec::new(); + let mut transports = Vec::new(); + let mut pumps = Vec::new(); + for (index, voter) in voters.iter().enumerate() { + let base = root.path().join(format!("voter-{index}")); + let log = domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + base.join("log"), + voter.node, + DiskLineage::new([index as u8 + 11; 16])?, + ProcessGeneration::new([index as u8 + 21; 16])?, + LogOpenMode::Create, + ), + Arc::new(DirectFileSystem), + )) + .await??; + let permit = domain + .run(directory.bootstrap_group_member(&birth, &log)) + .await??; + let group = permit.group().clone(); + let identity = group.identity().clone(); + let (root, applier) = domain + .blocking(move || -> anyhow::Result<_> { + let mut pages = ManifestPageStore::open( + &base.join("pages"), + Arc::new(DirectFileSystem), + 128, + 1024 * 1024, + 16, + )?; + let marker = group.group_manifest()?.context("genesis")?; + let tree = ManifestTree::new(marker.route().clone(), Default::default())?; + let root = tree.prepare_complete( + &group, + &mut pages, + marker.root(), + marker.position(), + marker.membership_revision(), + |_| unreachable!("empty genesis"), + )?; + let images = CellGroupApplyImages::Materialized { + images: GenerationStore::open( + base.join("images"), + Arc::new(DirectFileSystem), + Default::default(), + )?, + services: ReadServices { + now_unix_millis: Arc::new(crate::asyncrt::wall_ms), + fill_random: Arc::new(|bytes| bytes.fill(17)), + }, + }; + let applier = CellGroupApplier::new( + group, + pages, + images, + Default::default(), + Default::default(), + 32, + )?; + Ok((root, applier)) + })? + .await??; + let (runtime, mut events) = CellGroupRuntime::start( + log.clone(), + RaftRuntimeOptions { + election_seed: Some([index as u8 + 31; 32]), + ..Default::default() + }, + ) + .await?; + let tls = RaftTls::new( + vec![include_bytes!("../../transport/tcp/fixtures/ca.der") + .to_vec() + .into()], + vec![include_bytes!("../../transport/tcp/fixtures/server.der") + .to_vec() + .into()], + rustls::pki_types::PrivatePkcs8KeyDer::from( + include_bytes!("../../transport/tcp/fixtures/key.der").to_vec(), + ) + .into(), + )?; + let label = format!("park-voter-{index}"); + let transport = RaftTransport::start_shared( + runtime.clone(), + Arc::new(PeerAuth::new([8; 32], &label)?), + tls, + TcpListener::bind("127.0.0.1:0").await?, + TcpListener::bind("127.0.0.1:0").await?, + TcpListener::bind("127.0.0.1:0").await?, + TransportOptions::default(), + ) + .await?; + runtime.attach(permit, root).await?; + let pump = domain.spawn({ + let runtime = runtime.clone(); + let transport = transport.clone(); + async move { + while let Some(event) = events.recv().await { + match event.effect().map_err(Clone::clone)? { + CellGroupEffect::Message(_) => { + // The transport owns queued delivery independent of the waiter. + let _delivery = transport.submit_cell_group_event(&event)?; + } + CellGroupEffect::Apply(_) => { + let work = event.retain_apply().context("physical apply")?; + drop(event); + let prepare = work.clone(); + let applier = applier.clone(); + let next = crate::asyncrt::blocking(move || applier.prepare(&prepare)) + .await??; + runtime.complete_apply(history, work, next).await?; + } + _ => (), + } + } + Ok::<_, anyhow::Error>(()) + } + })?; + peers.push(RaftPeer { + replica: identity, + label, + server_name: "localhost".into(), + control_address: transport.control_address(), + data_address: transport.data_address(), + query_address: transport.query_address(), + }); + logs.push(log); + runtimes.push(runtime); + transports.push(transport); + pumps.push(pump); + } + for transport in &transports { + transport.install_physical_group(GroupRoute { + scope: history, + schema_digest: birth.route().schema_digest, + revision: 1, + peers: peers.clone(), + })?; + } + let runtime = &runtimes[0]; + let transport = &transports[0]; + let log = &logs[0]; + runtime.campaign(history).await?; + let status = wait_park_test(&domain, || async { + let status = runtime.status(history).await?; + Ok(status.ready.then_some(status)) + }) + .await?; + let abandoned = ParkGeneration::new(status.applied.index.get() + 1)?; + let deadline = log.domain().mono_ms()? + MAX_PARK_NEGOTIATION_MS; + let status = wait_park_test(&domain, || async { + match runtime + .fence_managed_park(history, abandoned, status.term, deadline) + .await + { + Ok(()) => Ok(Some(runtime.status(history).await?)), + Err(RuntimeError::NotReady) => Ok(None), + Err(error) => Err(error.into()), + } + }) + .await?; + assert!(status.applied.index.get() > abandoned.get()); + assert!(log + .cell_group(&history)? + .group_manifest()? + .context("root")? + .park_cut() + .is_none()); + let deadline = log.domain().mono_ms()? + MAX_PARK_NEGOTIATION_MS; + let request = ParkPrepareRequest { + proposal: ProposalId::new([20; 16])?, + term: status.term, + membership: MembershipRevision::new(1)?, + expected_applied: status.applied, + deadline_ms: deadline, + }; + let prepared = wait_park_test(&domain, || async { + match runtime.prepare_park(history, request).await { + Ok(prepared) => Ok(Some(prepared)), + Err(RuntimeError::NotReady) => Ok(None), + Err(error) => Err(error.into()), + } + }) + .await?; + let cut = wait_park_test(&domain, || async { + Ok(log + .cell_group(&history)? + .group_manifest()? + .and_then(|marker| marker.park_cut().cloned()) + .filter(|cut| cut.position() == prepared.proposal.position)) + }) + .await?; + let certificate = transport + .collect_park_certificate(history, cut.clone(), deadline) + .await?; + transport + .deliver_park_certificate(history, &certificate, deadline) + .await?; + for log in &logs { + let group = log.cell_group(&history)?; + let cache = group.read()?; + let parked = cache.local_park.as_ref().context("durable park")?; + assert!(parked.parked()); + assert_eq!(parked.certificate(), Some(certificate.certificate())); + } + assert!(!runtime.status(history).await?.ready); + let deadline = log.domain().mono_ms()? + MAX_PARK_NEGOTIATION_MS; + assert!(matches!( + runtime.park_admission(history, deadline).await, + Err(RuntimeError::NotReady) + )); + assert!(matches!( + runtime + .wake_park( + history, + ParkGeneration::new(cut.generation() - 1)?, + deadline + ) + .await, + Err(RuntimeError::StaleCompletion) + )); + assert!(matches!( + runtime + .prepare_park( + history, + ParkPrepareRequest { + deadline_ms: deadline, + ..request + } + ) + .await, + Err(RuntimeError::NotReady) + )); + let retry = transport + .collect_park_certificate(history, cut.clone(), deadline) + .await?; + assert_eq!(retry.certificate(), certificate.certificate()); + transport + .deliver_park_certificate(history, &retry, deadline) + .await?; + let observed = runtime + .wake_park(history, ParkGeneration::new(cut.generation())?, deadline) + .await?; + assert!(observed.root().position().index > cut.position().index); + drop(observed); + let resumed = wait_park_test(&domain, || async { + let status = runtime.status(history).await?; + Ok((status.ready && status.applied.index > cut.position().index).then_some(status)) + }) + .await?; + assert!(resumed.term > cut.position().term); + assert!(matches!( + runtime + .enter_park(history, certificate.certificate().clone()) + .await, + Err(RuntimeError::NotReady) + )); + + for runtime in &runtimes { + runtime.close().await?; + } + for pump in pumps { + pump.await??; + } + for transport in &transports { + transport.close().await; + } + for log in &logs { + log.close().await?; + } + domain.shutdown().await; + Ok(()) +} + +async fn wait_park_test>>>( + domain: &crate::asyncrt::ExecutionDomain, + mut sample: impl FnMut() -> F, +) -> anyhow::Result { + domain + .run(crate::asyncrt::timeout( + std::time::Duration::from_secs(10), + async { + loop { + if let Some(value) = sample().await? { + return Ok(value); + } + crate::asyncrt::sleep(std::time::Duration::from_millis(10)).await; + } + }, + )) + .await?? +} diff --git a/crates/cellar/durability/raft/performance.rs b/crates/cellar/durability/raft/performance.rs index 8d26260af..d1b5fd7b9 100644 --- a/crates/cellar/durability/raft/performance.rs +++ b/crates/cellar/durability/raft/performance.rs @@ -58,6 +58,7 @@ pub(crate) enum Operation { DataQueue, ControlExchange, DataExchange, + QueryReplyWrite, ControlAdmit, DataAdmit, ArchiveAdmissionCheck, @@ -83,7 +84,7 @@ pub(crate) enum Operation { EnginePurge, } -const OPERATIONS: [Operation; 44] = [ +const OPERATIONS: [Operation; 45] = [ Operation::EngineQueue, Operation::EngineFileSync, Operation::EngineDirectorySync, @@ -105,6 +106,7 @@ const OPERATIONS: [Operation; 44] = [ Operation::DataQueue, Operation::ControlExchange, Operation::DataExchange, + Operation::QueryReplyWrite, Operation::ControlAdmit, Operation::DataAdmit, Operation::ArchiveAdmissionCheck, @@ -154,6 +156,7 @@ impl Operation { Self::DataQueue => "transport_data_queue", Self::ControlExchange => "transport_control_exchange", Self::DataExchange => "transport_data_exchange", + Self::QueryReplyWrite => "transport_query_reply_write", Self::ControlAdmit => "transport_control_admit", Self::DataAdmit => "transport_data_admit", Self::ArchiveAdmissionCheck => "archive_admission_check", diff --git a/crates/cellar/durability/raft/physical_inventory.rs b/crates/cellar/durability/raft/physical_inventory.rs index f8ab10d68..9dc46ee42 100644 --- a/crates/cellar/durability/raft/physical_inventory.rs +++ b/crates/cellar/durability/raft/physical_inventory.rs @@ -2,10 +2,11 @@ //! Compact complete physical occupancy, including unattached/retired histories. //! Immutable identities need no Raft quorum or eager application registration. -use super::cache::GroupLog; +use super::cache::{GroupLog, StoredGroup}; use super::host::RaftHostObservation; use super::storage::LogError; use super::transport::TransportError; +use cellar_logic::cell_group::GroupHistory; use cellar_logic::consensus::{GroupScope, MemberId}; use sha2::{Digest, Sha256}; use std::collections::BTreeMap; @@ -13,6 +14,8 @@ use std::collections::BTreeMap; const MAX_GROUPS: usize = 4096; const PREFIX: usize = 45; const ENTRY_BYTES: usize = 81; +const IN_CONFIGURATION: u8 = 1; +const PARKED_RETIRED_SOURCE: u8 = 2; pub(crate) type PhysicalNodeObservation = (RaftHostObservation, RaftPhysicalInventory); @@ -32,14 +35,20 @@ pub struct RaftPhysicalHistory { // reservation. Construction validates member; no untrusted raw constructor. member: [u8; 8], completed_sequence: [u8; 8], - in_configuration: bool, + flags: u8, } impl RaftPhysicalHistory { /// This local member is present in the durable incoming/outgoing voter or /// learner sets. False is retained occupancy, never physical release proof. pub fn in_configuration(&self) -> bool { - self.in_configuration + self.flags & IN_CONFIGURATION != 0 + } + + /// This exact source slot has a verified retirement tombstone and opened + /// without a Raft owner. Its retained storage remains charged. + pub fn parked_retired_source(&self) -> bool { + self.flags & PARKED_RETIRED_SOURCE != 0 } /// Locally durable Complete phase for this birth/history. A fresh quorum @@ -92,7 +101,7 @@ impl RaftPhysicalInventory { digest.update(history.birth); digest.update(history.member); digest.update(history.completed_sequence); - digest.update([u8::from(history.in_configuration)]); + digest.update([history.flags]); } digest.finalize().into() } @@ -110,6 +119,30 @@ impl RaftPhysicalInventory { .map(|index| &self.histories[index])) } + /// Look up a cell-group history without inventing an application cell. + /// Its digest domain is distinct from the temporary single-cell adapter. + pub fn group_history(&self, history: &GroupHistory) -> Option<&RaftPhysicalHistory> { + let digest = group_digest(history); + self.histories + .binary_search_by_key(&digest, |h| h.scope) + .ok() + .map(|index| &self.histories[index]) + } + + pub(crate) fn mark_parked_retired_sources( + &mut self, + histories: &[GroupHistory], + ) -> Result<(), LogError> { + for history in histories { + let index = self + .histories + .binary_search_by_key(&group_digest(history), |row| row.scope) + .map_err(|_| LogError::WrongIdentity)?; + self.histories[index].flags |= PARKED_RETIRED_SOURCE; + } + Ok(()) + } + pub(crate) fn validate(&self, view: &RaftHostObservation) -> Result<(), TransportError> { super::node_inspection_wire::validate(view)?; if self.histories.len() > MAX_GROUPS @@ -129,8 +162,15 @@ impl RaftPhysicalInventory { .binary_search_by_key(&digest, |h| h.scope) .is_err() })); + let parked = self + .histories + .iter() + .filter(|row| row.parked_retired_source()) + .count(); if self.histories.len() + extra != view.node.groups as usize || view.unattached_engine_histories as usize > self.histories.len() + || parked > view.unattached_engine_histories as usize + || parked != 0 && view.fleet_boot.is_none() { return Err(TransportError::Protocol); } @@ -147,36 +187,60 @@ pub(crate) fn scope_digest(scope: &GroupScope) -> Result<[u8; 32], LogError> { Ok(Sha256::digest(bytes).into()) } +pub(crate) fn group_digest(history: &GroupHistory) -> [u8; 32] { + let mut digest = Sha256::new(); + digest.update(b"cellar-cell-group-physical-history-1\0"); + digest.update(history.cluster.bytes()); + digest.update(history.namespace.bytes()); + digest.update(history.group.bytes()); + digest.update(history.incarnation.bytes()); + digest.finalize().into() +} + +pub(crate) fn capture_cells( + groups: &BTreeMap, + pending: Option<&GroupHistory>, +) -> Result { + // A group-native owner cannot attach to a legacy application engine. + if groups.values().any(|group| !group.is_cells()) { + return Err(LogError::WrongIdentity); + } + capture_histories(groups, pending.map(group_digest)) +} + pub(crate) fn capture( - groups: &BTreeMap, + groups: &BTreeMap, pending: Option<&GroupScope>, +) -> Result { + if groups.len() > MAX_GROUPS { + return Err(LogError::Overloaded); + } + if let Some(scope) = pending { + scope.validate().map_err(|_| LogError::WrongIdentity)?; + if let Some(group) = groups.get(&GroupHistory::from(scope)) { + if &group.as_cell()?.identity().scope != scope { + return Err(LogError::WrongIdentity); + } + } + } + capture_histories(groups, pending.map(scope_digest).transpose()?) +} + +fn capture_histories( + groups: &BTreeMap, + initializing: Option<[u8; 32]>, ) -> Result { if groups.len() > MAX_GROUPS { return Err(LogError::Overloaded); } let mut histories = Vec::with_capacity(groups.len()); - for (scope, group) in groups { - if scope != &group.identity().scope || group.birth_digest == [0; 32] { + for (history, group) in groups { + if history != &group.history() { return Err(LogError::WrongIdentity); } - let cache = group.read()?; - let completed_sequence = cache - .membership - .fleet_status() - .filter(|status| { - status.phase == Some(super::member_state::FleetPermitPhase::Complete) - && status - .permit - .as_ref() - .is_some_and(|permit| permit.sequence() == status.consumed_sequence) - }) - .map_or(0, |status| status.consumed_sequence); - histories.push(RaftPhysicalHistory { - scope: scope_digest(scope)?, - birth: group.birth_digest, - member: group.identity().member.get().to_be_bytes(), - completed_sequence: completed_sequence.to_be_bytes(), - in_configuration: contains_member(&cache.conf_state, group.identity().member), + histories.push(match group { + StoredGroup::Cell(group) => capture_row(group, scope_digest(&group.identity().scope)?)?, + StoredGroup::Cells(group) => capture_row(group, group_digest(history))?, }); } histories.sort_by_key(|h| h.scope); @@ -187,7 +251,36 @@ pub(crate) fn capture( } Ok(RaftPhysicalInventory { histories, - initializing: pending.map(scope_digest).transpose()?, + initializing, + }) +} + +fn capture_row(group: &GroupLog, scope: [u8; 32]) -> Result { + if group.birth_digest == [0; 32] { + return Err(LogError::WrongIdentity); + } + let cache = group.read()?; + let completed_sequence = cache + .membership + .fleet_status() + .filter(|status| { + status.phase == Some(super::member_state::FleetPermitPhase::Complete) + && status + .permit + .as_ref() + .is_some_and(|permit| permit.sequence() == status.consumed_sequence) + }) + .map_or(0, |status| status.consumed_sequence); + Ok(RaftPhysicalHistory { + scope, + birth: group.birth_digest, + member: group.identity().member.get().to_be_bytes(), + completed_sequence: completed_sequence.to_be_bytes(), + flags: if contains_member(&cache.conf_state, group.identity().member) { + IN_CONFIGURATION + } else { + 0 + }, }) } @@ -220,7 +313,7 @@ pub(crate) fn encode( bytes.extend_from_slice(&history.birth); bytes.extend_from_slice(&history.member); bytes.extend_from_slice(&history.completed_sequence); - bytes.push(u8::from(history.in_configuration)); + bytes.push(history.flags); } Ok(bytes) } @@ -263,16 +356,16 @@ pub(crate) fn decode( let mut sequence = [0; 8]; sequence.copy_from_slice(&row[72..80]); MemberId::new(u64::from_be_bytes(member)).map_err(|_| TransportError::Identity)?; + let flags = row[80]; + if flags & !(IN_CONFIGURATION | PARKED_RETIRED_SOURCE) != 0 { + return Err(TransportError::Protocol); + } histories.push(RaftPhysicalHistory { scope, birth, member, completed_sequence: sequence, - in_configuration: match row[80] { - 0 => false, - 1 => true, - _ => return Err(TransportError::Protocol), - }, + flags, }); } let inventory = RaftPhysicalInventory { diff --git a/crates/cellar/durability/raft/physical_inventory_tests.rs b/crates/cellar/durability/raft/physical_inventory_tests.rs index d64c62fea..76db69d83 100644 --- a/crates/cellar/durability/raft/physical_inventory_tests.rs +++ b/crates/cellar/durability/raft/physical_inventory_tests.rs @@ -28,7 +28,7 @@ fn occupied(count: usize) -> (RaftHostObservation, RaftPhysicalInventory) { birth: [8; 32], member: (i as u64).to_be_bytes(), completed_sequence: (i as u64).to_be_bytes(), - in_configuration: true, + flags: IN_CONFIGURATION, } }) .collect(); @@ -128,7 +128,7 @@ fn physical_inventory_scope_matching_binds_the_full_history() { birth: [8; 32], member: 3u64.to_be_bytes(), completed_sequence: 0u64.to_be_bytes(), - in_configuration: false, + flags: 0, }], initializing: None, }; @@ -148,18 +148,26 @@ fn physical_inventory_scope_matching_binds_the_full_history() { } #[test] -fn physical_inventory_configuration_flag_is_strict_and_digest_bound() { - let (node, mut inventory) = occupied(1); +fn physical_inventory_flags_are_strict_and_digest_bound() { + let (mut node, mut inventory) = occupied(1); let active = inventory.digest(); - inventory.histories[0].in_configuration = false; + inventory.histories[0].flags = 0; assert_ne!(active, inventory.digest()); + let bytes = encode(&node, &inventory).unwrap(); + let row = decode(&bytes).unwrap().1.histories()[0]; + assert!(!row.in_configuration() && !row.parked_retired_source()); + + let inactive = inventory.digest(); + inventory.histories[0].flags = PARKED_RETIRED_SOURCE; + assert_ne!(inactive, inventory.digest()); let mut bytes = encode(&node, &inventory).unwrap(); - assert!(!decode(&bytes).unwrap().1.histories()[0].in_configuration()); + assert!(decode(&bytes).unwrap().1.histories()[0].parked_retired_source()); + node.unattached_engine_histories = 0; + assert!(encode(&node, &inventory).is_err()); + let last = bytes.len() - 1; - bytes[last] = 2; + bytes[last] = 4; assert!(decode(&bytes).is_err()); - // The internal format marker remains 1; old shorter rows cannot silently - // default the new role observation to false during a mixed deployment. bytes.pop(); assert!(decode(&bytes).is_err()); } @@ -183,3 +191,38 @@ fn physical_inventory_role_includes_every_joint_configuration_set() { member )); } + +#[test] +fn physical_inventory_group_history_is_canonical_and_domain_separated() { + let legacy = scope(); + let history = GroupHistory::from(&legacy); + let digest = group_digest(&history); + // Independent SHA-256 vector over the fixed domain and four 16-byte IDs. + assert_eq!( + hex::encode(digest), + "08040247cafda32726f8585b91312bb8d84c904fae55e05fa22009b1b4a276f6" + ); + assert_ne!(digest, scope_digest(&legacy).unwrap()); + let inventory = RaftPhysicalInventory { + histories: vec![RaftPhysicalHistory { + scope: digest, + birth: [8; 32], + member: 3u64.to_be_bytes(), + completed_sequence: 0u64.to_be_bytes(), + flags: 0, + }], + initializing: None, + }; + assert_eq!(inventory.group_history(&history).unwrap().member().get(), 3); + assert!(inventory.history(&legacy).unwrap().is_none()); + for field in 0..4 { + let mut changed = history; + match field { + 0 => changed.cluster = ClusterId::new([9; 16]).unwrap(), + 1 => changed.namespace = NamespaceId::new([9; 16]).unwrap(), + 2 => changed.group = GroupId::new([9; 16]).unwrap(), + _ => changed.incarnation = IncarnationId::new([9; 16]).unwrap(), + } + assert!(inventory.group_history(&changed).is_none()); + } +} diff --git a/crates/cellar/durability/raft/query.rs b/crates/cellar/durability/raft/query.rs index 9dade0421..9f24a9a37 100644 --- a/crates/cellar/durability/raft/query.rs +++ b/crates/cellar/durability/raft/query.rs @@ -1,12 +1,9 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. -//! Request-owned read constraints. A fresh ReadIndex result is correlated to -//! one invocation; a cached commit watermark can never replace that barrier. +//! Internal SQL minimum checks and the remaining seed barrier owner. Business +//! reads are admitted only through the shared physical-group controller. use crate::application::consensus::client::Request; -use crate::application::consensus::{ - OperationContract, QueryConsistency, QueryOptions, StaleReadPolicy, -}; use cellar_logic::consensus::{CommandFailure, CommittedVersion, GroupScope, LogIndex, Term}; pub(crate) type ReadId = [u8; 32]; @@ -48,8 +45,6 @@ impl Minimum { #[derive(Clone, Copy, PartialEq, Eq)] pub(crate) enum Phase { - Source, - Probing, StartBarrier, CheckingLeader, Barrier, @@ -57,28 +52,13 @@ pub(crate) enum Phase { Executing, } -pub(crate) enum ReadOperation { - Fleet { - authorization: crate::raft::fleet_catalog::FleetGroupAuthorization, - charge: crate::raft::budget::Permit, - }, - Query { - contract: OperationContract, - input: Vec, - }, - Seed { - target: cellar_logic::consensus::ReplicaIdentity, - kind: crate::raft::transport::seeds::SeedKind, - charge: std::sync::Arc, - }, -} - pub(crate) struct PendingRead { pub request: Request, - pub operation: ReadOperation, - pub minimum: Option, + pub target: cellar_logic::consensus::ReplicaIdentity, + pub kind: crate::raft::transport::seeds::SeedKind, + pub charge: std::sync::Arc, + pub minimum: Option, pub phase: Phase, - pub source_deadline: Option, pub retry_at: u64, pub leader: Option<(Term, Option, bool)>, pub checking_leader: bool, @@ -89,81 +69,19 @@ pub(crate) struct PendingRead { } impl PendingRead { - pub fn fleet( - request: Request, - authorization: crate::raft::fleet_catalog::FleetGroupAuthorization, - charge: crate::raft::budget::Permit, - ) -> Self { - Self::with_options( - request, - ReadOperation::Fleet { - authorization, - charge, - }, - &QueryOptions { - consistency: QueryConsistency::Linearizable, - ..Default::default() - }, - None, - ) - } - - pub fn new( - request: Request, - contract: OperationContract, - input: Vec, - options: &QueryOptions, - minimum: Option, - ) -> Self { - Self::with_options( - request, - ReadOperation::Query { contract, input }, - options, - minimum, - ) - } - pub fn seed( request: Request, target: cellar_logic::consensus::ReplicaIdentity, kind: crate::raft::transport::seeds::SeedKind, charge: std::sync::Arc, ) -> Self { - Self::with_options( - request, - ReadOperation::Seed { - target, - kind, - charge, - }, - &QueryOptions { - consistency: QueryConsistency::Linearizable, - ..Default::default() - }, - None, - ) - } - - fn with_options( - request: Request, - operation: ReadOperation, - options: &QueryOptions, - minimum: Option, - ) -> Self { - let (phase, minimum) = match options.consistency { - QueryConsistency::Linearizable => (Phase::StartBarrier, None), - QueryConsistency::AtLeast(_) => (Phase::Image, minimum.map(Minimum::Version)), - QueryConsistency::Stale if options.stale_policy == StaleReadPolicy::RecentSource => { - (Phase::Source, Some(Minimum::Any)) - } - QueryConsistency::Stale => (Phase::Image, Some(Minimum::Any)), - }; Self { request, - operation, - minimum, - phase, - source_deadline: None, + target, + kind, + charge, + minimum: None, + phase: Phase::StartBarrier, retry_at: 0, leader: None, checking_leader: false, @@ -176,8 +94,5 @@ impl PendingRead { pub fn expired(&self) -> bool { self.request.control.cancelled() - || self - .source_deadline - .is_some_and(|deadline| !self.request.control.before(deadline)) } } diff --git a/crates/cellar/durability/raft/query_cell_discovery.rs b/crates/cellar/durability/raft/query_cell_discovery.rs new file mode 100644 index 000000000..f49b8cffc --- /dev/null +++ b/crates/cellar/durability/raft/query_cell_discovery.rs @@ -0,0 +1,171 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Read-only physical discovery; no local RawNode, membership or write authority. +use super::*; +use crate::raft::storage::LogError; +use cellar_logic::consensus::MemberId; + +impl RaftTransport { + pub(crate) fn cell_query_router_members( + &self, + route: &cellar_logic::cell_group::GroupRoute, + limit: usize, + ) -> Result, TransportError> { + let state = &self.handle.0; + let registry = state.registry.lock().map_err(|_| TransportError::Closed)?; + state.admit(®istry)?; + let view = registry + .cell_query_groups + .get(&route.history) + .ok_or(TransportError::UnknownGroup)?; + if view.schema_digest != route.schema_digest { + return Err(TransportError::Identity); + } + Ok(view + .peers + .iter() + .take(limit) + .map(|p| p.replica.member) + .collect()) + } + + /// Install trusted read destinations for a physical group absent from this + /// node. The caller supplies discovery; each destination independently checks + /// namespace permission, live membership and the requested cell read proof. + /// Shares node route/peer bounds with consensus and all other query routes. + pub fn install_cell_query_group( + &self, + route: GroupRoute, + ) -> Result<(), TransportError> { + let state = &self.handle.0; + let runtime = state.runtime.shared()?; + if !matches!( + runtime.log().cell_group(&route.scope), + Err(LogError::NotFound) + ) { + return Err(TransportError::Identity); + } + if route.revision == 0 + || route.schema_digest == [0; 32] + || route.peers.is_empty() + || route.peers.len() > 128 + { + return Err(TransportError::Identity); + } + let mut members = BTreeSet::new(); + let mut nodes = BTreeSet::new(); + let physical: Vec<_> = route.peers.iter().map(RaftPeerNode::from).collect(); + for (peer, endpoint) in route.peers.iter().zip(&physical) { + endpoint.validate_endpoint()?; + if peer.replica.scope != route.scope + || peer.replica.node == runtime.log().limits().node + || !members.insert(peer.replica.member) + || !nodes.insert(peer.replica.node) + { + return Err(TransportError::Identity); + } + } + let mut registry = state.registry.lock().map_err(|_| TransportError::Closed)?; + state.admit(®istry)?; + if registry.physical_groups.contains_key(&route.scope) { + return Err(TransportError::Identity); + } + for peer in &physical { + registry.check_node_view(peer)?; + } + if let Some(old) = registry.cell_query_groups.get(&route.scope) { + if **old == route { + return Ok(()); + } + if old.revision >= route.revision || old.schema_digest != route.schema_digest { + return Err(TransportError::StalePeer); + } + if route.peers.iter().any(|peer| { + old.peers.iter().any(|previous| { + previous.replica.member == peer.replica.member + && (previous.replica.node != peer.replica.node + || previous.replica.disk != peer.replica.disk) + }) + }) { + return Err(TransportError::Identity); + } + } else if registry.route_count() >= state.options.max_groups { + return Err(TransportError::Overloaded); + } + for peer in &physical { + if registry + .physical_groups + .values() + .chain( + registry + .cell_query_groups + .iter() + .filter(|(scope, _)| **scope != route.scope) + .map(|(_, view)| view), + ) + .flat_map(|view| &view.peers) + .any(|other| other.replica.node == peer.node && !peer.same_host(other)) + || registry + .groups + .values() + .chain(registry.query_groups.values()) + .flat_map(|view| &view.peers) + .any(|other| other.replica.node == peer.node && !peer.same_host(other)) + || registry + .origins + .values() + .chain(registry.inspection_origins.values()) + .filter_map(|grant| grant.peer.as_ref()) + .any(|other| other.node == peer.node && other != peer) + { + return Err(TransportError::StalePeer); + } + } + let starts = state.prepare_pools(&mut registry, &physical)?; + registry + .cell_query_groups + .insert(route.scope, Arc::new(route)); + registry.prune_pools(); + drop(registry); + state.start_pools(starts) + } + + /// Remove read-only discovery, invalidating queued/in-flight route identity + /// and retiring peer pools that no other route or grant retains. + pub fn remove_cell_query_group(&self, history: &GroupHistory) -> Result<(), TransportError> { + let state = &self.handle.0; + state.runtime.shared()?; + let mut registry = state.registry.lock().map_err(|_| TransportError::Closed)?; + state.admit(®istry)?; + registry.cell_query_groups.remove(history); + registry.prune_pools(); + Ok(()) + } +} + +impl State { + pub(super) fn current_cell_query_route( + &self, + route: &Arc>, + peer: &RaftPeer, + ) -> bool { + let Ok(runtime) = self.runtime.shared() else { + return false; + }; + match runtime.log().cell_group(&route.scope) { + Ok(_) => self.current_physical(route, peer), + Err(LogError::NotFound) => self.registry.lock().is_ok_and(|registry| { + self.admit(®istry).is_ok() + && registry + .cell_query_groups + .get(&route.scope) + .is_some_and(|current| Arc::ptr_eq(current, route)) + && registry + .pools + .get(&peer.replica.node) + .is_some_and(|pool| pool.peer.same_host(peer)) + }), + Err(_) => false, + } + } +} diff --git a/crates/cellar/durability/raft/query_cell_transport.rs b/crates/cellar/durability/raft/query_cell_transport.rs new file mode 100644 index 000000000..973d49432 --- /dev/null +++ b/crates/cellar/durability/raft/query_cell_transport.rs @@ -0,0 +1,521 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Shared cell reads reuse the peer query lane; namespace grants remain read-only. +use super::*; +use crate::application::consensus::CommandFailure; +use crate::raft::cell_groups::{ + CellDispatchError, CellGroupController, CellGroupDispatcher, CellGroupNamespaceStorage, + CellRoutingLimits, +}; +use crate::raft::query_identity::QueryRoute; +use crate::raft::query_wire::{ + self as wire, + cell::{CellQueryCall, CellQueryValue}, +}; +use crate::raft::version::{CellVersionObservation, CellVersionToken, VersionKeys}; +use cellar_logic::{ + cell_group::CellHistory, + consensus::{MemberId, NodeIdentity}, +}; +use queries::{QueryConnection, SendQuery}; +use std::time::Duration; + +/// Host-owned registration. The transport retains only a weak reference, so it +/// cannot extend the host's lifetime or create a runtime/transport ownership cycle. +#[derive(Clone)] +pub struct CellQueryGateway { + inner: Arc, +} + +// Accepted requests retain Gateway, never this admission owner. In-flight work +// cannot keep accepting new reads after the final host registration is dropped. +struct Registration { + gateway: Arc, +} + +impl Drop for Registration { + fn drop(&mut self) { + self.gateway.accepting.store(false, Ordering::Release); + } +} + +pub(super) struct Gateway { + accepting: AtomicBool, + controller: CellGroupController, + dispatcher: CellGroupDispatcher, + storage: CellGroupNamespaceStorage, +} + +/// Retains the same node/peer query admission through consumption of the result. +pub struct CellRemoteQueryResponse { + value: CellQueryValue, + replica: cellar_logic::consensus::ReplicaIdentity, + _node: Permit, + _peer: Permit, +} + +impl CellRemoteQueryResponse { + pub fn result(&self) -> &[u8] { + &self.value.result + } + + pub fn observation(&self) -> &CellVersionObservation { + &self.value.observation + } + + pub fn token(&self) -> &CellVersionToken { + &self.value.token + } + + pub fn served_by(&self) -> MemberId { + self.value.served_by + } + + pub fn replica(&self) -> &cellar_logic::consensus::ReplicaIdentity { + &self.replica + } +} + +pub(in crate::raft) struct CellRemoteQueryReply { + pub result: Result, + pub leader_hint: Option, +} + +pub(crate) struct CellQueryContext { + pub keys: VersionKeys, + pub admission: Arc, +} + +pub(super) struct SendCellQuery { + route: Arc>, + peer: RaftPeer, + cell: CellHistory, + call: CellQueryCall, + keys: VersionKeys, + _admission: Arc, + deadline: u64, + cancel: watch::Receiver, + reply: oneshot::Sender>, + node: Permit, + peer_bytes: Permit, +} + +struct Cancel(watch::Sender); + +impl Drop for Cancel { + fn drop(&mut self) { + self.0.send_replace(true); + } +} + +impl RaftTransport { + /// Register one namespace's compiled reads on this shared runtime and SQL + /// pool. Options provide the namespace's immutable-query services and cache; + /// cell images are selected only by fresh runtime authority. This grants no + /// origin permission. Retain the returned owner while serving requests. + /// Dropping its final clone refuses new reads; already admitted reads retain + /// their physical owners until completion or joined shutdown. + pub fn install_cell_query_gateway( + &self, + dispatcher: CellGroupDispatcher, + controller: CellGroupController, + storage: CellGroupNamespaceStorage, + ) -> Result { + let state = &self.handle.0; + let runtime = state.runtime.shared()?; + if !dispatcher.uses_domain(&state.domain) || !controller.belongs_to(runtime) { + return Err(TransportError::Identity); + } + let (namespace, _, _) = dispatcher + .host_binding() + .map_err(|_| TransportError::Identity)?; + storage + .validate_namespace(&namespace) + .map_err(|_| TransportError::Identity)?; + let key = dispatcher.namespace(); + let mut gateways = state + .cell_query_gateways + .lock() + .map_err(|_| TransportError::Closed)?; + let registry = state.registry.lock().map_err(|_| TransportError::Closed)?; + state.admit(®istry)?; + gateways.retain(|_, value| value.strong_count() > 0); + if gateways.contains_key(&key) { + return Err(TransportError::Refused); + } + if gateways.len() >= state.options.max_groups { + return Err(TransportError::Overloaded); + } + let inner = Arc::new(Gateway { + accepting: AtomicBool::new(true), + controller, + dispatcher, + storage, + }); + gateways.insert(key, Arc::downgrade(&inner)); + Ok(CellQueryGateway { + inner: Arc::new(Registration { gateway: inner }), + }) + } + + pub(crate) fn uses_query_domain(&self, domain: &DomainToken) -> bool { + self.handle.0.domain.same_owner(domain) + } + + /// Called only by a consumed catalog attempt. No forwarding or retry occurs + /// here; the caller keeps its original deadline and attempt budget. + pub(in crate::raft) async fn query_cell( + &self, + route: &cellar_logic::cell_group::GroupRoute, + cell: &CellHistory, + member: MemberId, + call: CellQueryCall, + context: CellQueryContext, + deadline: u64, + ) -> Result { + let state = &self.handle.0; + // Discovery never creates a local replica or grants voting authority. + state.runtime.shared()?; + let (view, peer, pool) = { + let registry = state.registry.lock().map_err(|_| TransportError::Closed)?; + state.admit(®istry)?; + let view = registry + .physical_groups + .get(&route.history) + .or_else(|| registry.cell_query_groups.get(&route.history)) + .ok_or(TransportError::UnknownGroup)? + .clone(); + if view.schema_digest != route.schema_digest || !route.history.same_namespace(cell) { + return Err(TransportError::Identity); + } + let peer = view + .peers + .iter() + .find(|p| { + p.replica.member == member + && p.replica.node != state.runtime.log().limits().node + }) + .ok_or(TransportError::Identity)? + .clone(); + let pool = registry + .pools + .get(&peer.replica.node) + .ok_or(TransportError::StalePeer)? + .clone(); + (view, peer, pool) + }; + if state.domain.mono_ms().map_err(|_| TransportError::Closed)? >= deadline { + return Err(TransportError::Timeout); + } + let charge = queries::reservation( + call.request + .input + .len() + .checked_add(8192) + .ok_or(TransportError::Limit)?, + &state.options, + )?; + let node = state + .outgoing + .query + .acquire(charge) + .ok_or(TransportError::Overloaded)?; + let peer_bytes = pool + .outgoing + .query + .acquire(charge) + .ok_or(TransportError::Overloaded)?; + drop(call.encode(state.options.max_query_request_bytes)?); + if !state.current_cell_query_route(&view, &peer) { + return Err(TransportError::StalePeer); + } + let (reply, result) = oneshot::channel(); + let (cancel, cancelled) = watch::channel(false); + let _cancel = Cancel(cancel); + pool.query + .try_send(SendQuery::Shared(Box::new(SendCellQuery { + route: view, + peer, + cell: cell.clone(), + call, + keys: context.keys, + _admission: context.admission, + deadline, + cancel: cancelled, + reply, + node, + peer_bytes, + }))) + .map_err(|e| match e { + mpsc::error::TrySendError::Full(_) => TransportError::Overloaded, + mpsc::error::TrySendError::Closed(_) => TransportError::Closed, + })?; + result.await.map_err(|_| TransportError::Closed)? + } +} + +impl CellQueryGateway { + /// The cluster and namespace bound to this host-owned registration. + pub fn namespace(&self) -> (ClusterId, NamespaceId) { + self.inner.gateway.dispatcher.namespace() + } +} + +impl State { + pub(super) fn authorize_cell_query( + &self, + wire: &QueryRoute, + ) -> Result<(Arc, Option>), TransportError> { + let runtime = self.runtime.shared()?; + let (source, target, _) = wire.decode_shared()?; + let registry = self.registry.lock().map_err(|_| TransportError::Closed)?; + self.admit(®istry)?; + let route = registry + .physical_groups + .get(&target.scope) + .ok_or(TransportError::UnknownGroup)?; + let group = runtime + .log() + .cell_group(&target.scope) + .map_err(|_| TransportError::UnknownGroup)?; + if *group.identity() != target + || wire.target_label() != self.auth.source() + || wire.schema() != route.schema_digest + { + return Err(TransportError::StalePeer); + } + shared::permitted(&group, &[&target])?; + let implicit = route.peers.iter().find(|p| { + p.replica.node_identity() == source + && p.label == wire.source_label() + && shared::permitted(&group, &[&p.replica]).is_ok() + }); + self.authorize_query_origin( + ®istry, + target.scope.cluster, + target.scope.namespace, + source, + wire.source_label(), + implicit, + ) + } + + pub(super) async fn send_cell_query( + &self, + pool: &Pool, + mut request: SendCellQuery, + connection: &mut Option, + ) -> bool { + let result = self + .query_until( + pool, + request.deadline, + request.cancel.clone(), + None, + async { + if !self.current_cell_query_route(&request.route, &request.peer) { + return Err(TransportError::StalePeer); + } + let remaining = request + .deadline + .checked_sub(self.domain.mono_ms().map_err(|_| TransportError::Closed)?) + .filter(|n| *n > 0) + .ok_or(TransportError::Timeout)?; + request.call.timeout_ms = request.call.timeout_ms.min(remaining); + let body = request.call.encode(self.options.max_query_request_bytes)?; + let local = self.runtime.log().limits(); + let route = QueryRoute::new_shared( + NodeIdentity { + node: local.node, + disk: local.disk, + process: local.process, + }, + self.auth.source(), + &request.peer.replica, + &request.peer.label, + &request.cell, + request.route.schema_digest, + )?; + let body = self + .exchange_query(connection, &RaftPeerNode::from(&request.peer), route, body) + .await?; + if !self.current_cell_query_route(&request.route, &request.peer) { + return Err(TransportError::StalePeer); + } + let now = u64::try_from(crate::asyncrt::wall_ms()) + .map_err(|_| TransportError::Authentication)?; + wire::cell::decode_cell_reply( + &body, + &request.cell, + request.route.schema_digest, + &request.keys, + now, + self.options.max_query_response_bytes, + request.peer.replica.member, + ) + }, + ) + .await; + // A fully authenticated application refusal consumed a complete reply; + // retain the peer connection for the next authenticated request. + let failed = result.is_err(); + let _ = request + .reply + .send(result.map(|(value, leader_hint)| CellRemoteQueryReply { + result: value.map(|value| CellRemoteQueryResponse { + value, + replica: request.peer.replica.clone(), + _node: request.node, + _peer: request.peer_bytes, + }), + leader_hint, + })); + failed + } +} + +pub(super) enum IncomingCall { + Shared(CellQueryCall), + Command(crate::raft::query_wire::command::CellCommandCall), + CommandProbe(crate::raft::query_wire::command::CellCommandProbe), + Enrollment(crate::raft::query_wire::enrollment::CellEnrollmentCall), + Receipt(crate::raft::query_wire::receipt::CellReceiptCall), +} + +impl IncomingCall { + pub(super) fn decode( + route: &QueryRoute, + bytes: &[u8], + limit: usize, + ) -> Result { + if route.is_receipt() { + return Ok(Self::Receipt( + crate::raft::query_wire::receipt::CellReceiptCall::decode(bytes, limit)?, + )); + } + if route.is_enrollment() { + return Ok(Self::Enrollment( + crate::raft::query_wire::enrollment::CellEnrollmentCall::decode(bytes, limit)?, + )); + } + if route.is_command_probe() { + return Ok(Self::CommandProbe( + crate::raft::query_wire::command::CellCommandProbe::decode(bytes, limit)?, + )); + } + if route.is_command() { + return Ok(Self::Command( + crate::raft::query_wire::command::CellCommandCall::decode(bytes, limit)?, + )); + } + Ok(Self::Shared(CellQueryCall::decode(bytes, limit)?)) + } + + pub(super) fn timeout_ms(&self) -> u64 { + match self { + Self::Shared(call) => call.timeout_ms, + Self::Command(call) => call.timeout_ms, + Self::CommandProbe(call) => call.timeout_ms, + Self::Enrollment(call) => call.timeout_ms, + Self::Receipt(call) => call.timeout_ms, + } + } + + pub(super) async fn execute( + self, + state: &State, + route: &QueryRoute, + ) -> Result { + match self { + Self::Command(call) => state.execute_cell_command(route, call).await, + Self::CommandProbe(call) => state.execute_cell_command_probe(route, call).await, + Self::Enrollment(call) => state.execute_cell_enrollment(route, call).await, + Self::Receipt(call) => state.execute_cell_receipt(route, call).await, + Self::Shared(call) => { + let (_, target, cell) = route.decode_shared()?; + let gateway = state + .cell_query_gateways + .lock() + .map_err(|_| TransportError::Closed)? + .get(&(cell.cluster, cell.namespace)) + .and_then(|h| h.upgrade()) + .ok_or(TransportError::Refused)?; + if !gateway.accepting.load(Ordering::Acquire) { + return Err(TransportError::Refused); + } + // Queue first: metadata observation cannot renew the query's + // captured deadline or delay ownership transfer to its producer. + let queued = gateway.enqueue(call, target.scope, cell, route.schema()); + let leader_hint = if queued.is_ok() { + state + .runtime + .shared()? + .status(target.scope) + .await + .ok() + .and_then(|status| status.leader) + .and_then(|member| MemberId::new(member).ok()) + } else { + None + }; + let result = match queued { + Ok(call) => call.response().await, + Err(error) => Err(error), + } + .map_err(|e| query_failure(&e)); + wire::cell::encode_cell_result( + &result, + leader_hint, + state.options.max_query_response_bytes, + ) + .map(super::query_replies::Reply::plain) + } + } + } +} + +impl Gateway { + fn enqueue( + &self, + call: CellQueryCall, + group: GroupHistory, + cell: CellHistory, + schema: [u8; 32], + ) -> Result> { + let options = self.storage.for_cell(&cell); + let queued = (|| { + let mut query = self.dispatcher.query( + cell, + call.request, + CellRoutingLimits { + timeout: Duration::from_millis(call.timeout_ms), + max_attempts: 1, + }, + None, + )?; + query.bind_remote_route(crate::raft::cell_groups::RemoteMutationRoute { + group, + schema, + epoch: call.route_epoch, + birth: call.birth_digest, + }); + self.controller.query(query, options) + })(); + queued.map_err(Arc::new) + } +} + +fn query_failure(error: &CellDispatchError) -> CommandFailure { + use crate::raft::{execution::ExecutionError, version::VersionError}; + match error { + CellDispatchError::Execution(ExecutionError::Refused(failure)) => failure.clone(), + CellDispatchError::Runtime(RuntimeError::Overloaded) => CommandFailure::Overloaded, + CellDispatchError::Runtime(RuntimeError::DeadlineExceeded | RuntimeError::NotReady) => { + CommandFailure::VersionUnavailable + } + CellDispatchError::Version(VersionError::WrongHistory) => CommandFailure::WrongIncarnation, + CellDispatchError::Version(_) => CommandFailure::VersionUnavailable, + CellDispatchError::Registration(_) => CommandFailure::UnsupportedCapability, + _ => CommandFailure::Unavailable, + } +} diff --git a/crates/cellar/durability/raft/query_cell_wire.rs b/crates/cellar/durability/raft/query_cell_wire.rs new file mode 100644 index 000000000..22ed03132 --- /dev/null +++ b/crates/cellar/durability/raft/query_cell_wire.rs @@ -0,0 +1,225 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Logical cell query payloads on the existing authenticated query framing. +use super::*; +use crate::raft::cell_groups::{CellQueryConsistency, CellQueryRequest, CellQueryResponse}; +use crate::raft::version::{CellVersionObservation, CellVersionToken}; +use cellar_logic::{cell_group::CellHistory, consensus::ProposalId}; + +pub(in crate::raft) struct CellQueryCall { + pub request: CellQueryRequest, + pub timeout_ms: u64, + pub route_epoch: u64, + pub birth_digest: [u8; 32], +} + +pub(in crate::raft) struct CellQueryValue { + pub result: Vec, + pub observation: CellVersionObservation, + pub token: CellVersionToken, + pub served_by: MemberId, +} + +impl CellQueryCall { + pub fn encode(&self, limit: usize) -> Result, TransportError> { + let request = &self.request; + if self.route_epoch == 0 || self.birth_digest == [0; 32] { + return Err(TransportError::Identity); + } + if request.contract.kind != OperationKind::Query + || request.contract.operation.is_empty() + || request.contract.operation.len() > 128 + || request.contract.operation.chars().any(char::is_control) + || !(1..=3_600_000).contains(&self.timeout_ms) + { + return Err(TransportError::Protocol); + } + let mut body = b"CLGCQQ01".to_vec(); + body.extend_from_slice(&request.invocation.bytes()); + body.extend_from_slice(&self.route_epoch.to_be_bytes()); + body.extend_from_slice(&self.birth_digest); + string(&mut body, &request.contract.operation)?; + body.extend_from_slice(&request.contract.digest); + match &request.consistency { + CellQueryConsistency::Stale => body.push(0), + CellQueryConsistency::RecentSource { max_age_ms } => { + if !(1..=3_600_000).contains(max_age_ms) { + return Err(TransportError::Protocol); + } + body.push(3); + body.extend_from_slice(&max_age_ms.to_be_bytes()); + } + CellQueryConsistency::Linearizable => body.push(1), + CellQueryConsistency::AtLeast(token) => { + body.push(2); + string(&mut body, token.as_str())?; + } + } + body.extend_from_slice(&self.timeout_ms.to_be_bytes()); + let length = u32::try_from(request.input.len()).map_err(|_| TransportError::Limit)?; + if body + .len() + .checked_add(4) + .and_then(|n| n.checked_add(request.input.len())) + .is_none_or(|n| n > limit) + { + return Err(TransportError::Limit); + } + body.extend_from_slice(&length.to_be_bytes()); + body.extend_from_slice(&request.input); + Ok(body) + } + + pub fn decode(bytes: &[u8], limit: usize) -> Result { + if bytes.len() > limit { + return Err(TransportError::Limit); + } + let mut body = Reader(bytes); + if body.take(8)? != b"CLGCQQ01" { + return Err(TransportError::Protocol); + } + let invocation = ProposalId::new(body.array()?).map_err(|_| TransportError::Identity)?; + let route_epoch = u64::from_be_bytes(body.array()?); + let birth_digest = body.array()?; + if route_epoch == 0 || birth_digest == [0; 32] { + return Err(TransportError::Identity); + } + let operation = body.string(128)?; + if operation.is_empty() || operation.chars().any(char::is_control) { + return Err(TransportError::Protocol); + } + let contract = OperationContract { + operation, + kind: OperationKind::Query, + digest: body.array()?, + }; + let consistency = match body.byte()? { + 0 => CellQueryConsistency::Stale, + 1 => CellQueryConsistency::Linearizable, + 2 => CellQueryConsistency::AtLeast( + CellVersionToken::from_encoded(&body.string(4096)?) + .map_err(|_| TransportError::Authentication)?, + ), + 3 => { + let max_age_ms = u64::from_be_bytes(body.array()?); + if !(1..=3_600_000).contains(&max_age_ms) { + return Err(TransportError::Protocol); + } + CellQueryConsistency::RecentSource { max_age_ms } + } + _ => return Err(TransportError::Protocol), + }; + let timeout_ms = u64::from_be_bytes(body.array()?); + if !(1..=3_600_000).contains(&timeout_ms) { + return Err(TransportError::Protocol); + } + let length = u32::from_be_bytes(body.array()?) as usize; + let input = body.take(length)?.to_vec(); + if !body.0.is_empty() { + return Err(TransportError::Protocol); + } + Ok(Self { + request: CellQueryRequest { + invocation, + contract, + input, + consistency, + }, + timeout_ms, + route_epoch, + birth_digest, + }) + } +} + +pub(in crate::raft) fn encode_cell_result( + result: &Result, + leader_hint: Option, + limit: usize, +) -> Result, TransportError> { + let mut body = b"CLGCQS01".to_vec(); + match result { + Ok(value) => { + body.push(0); + body.extend_from_slice(&leader_hint.map_or(0, MemberId::get).to_be_bytes()); + string(&mut body, value.token().as_str())?; + let length = u32::try_from(value.result().len()).map_err(|_| TransportError::Limit)?; + if body + .len() + .checked_add(4) + .and_then(|n| n.checked_add(value.result().len())) + .is_none_or(|n| n > limit) + { + return Err(TransportError::Limit); + } + body.extend_from_slice(&length.to_be_bytes()); + body.extend_from_slice(value.result()); + } + Err(error) => { + body.push(failure_code(error)); + body.extend_from_slice(&leader_hint.map_or(0, MemberId::get).to_be_bytes()); + } + } + if body.len() > limit { + return Err(TransportError::Limit); + } + Ok(body) +} + +pub(in crate::raft) fn decode_cell_reply( + bytes: &[u8], + cell: &CellHistory, + schema: [u8; 32], + keys: &VersionKeys, + now: u64, + limit: usize, + served_by: MemberId, +) -> Result<(Result, Option), TransportError> { + if bytes.len() > limit { + return Err(TransportError::Limit); + } + let mut body = Reader(bytes); + if body.take(8)? != b"CLGCQS01" { + return Err(TransportError::Protocol); + } + let status = body.byte()?; + let member = u64::from_be_bytes(body.array()?); + let leader_hint = if member == 0 { + None + } else { + Some(MemberId::new(member).map_err(|_| TransportError::Identity)?) + }; + let result = if status == 0 { + let token = CellVersionToken::from_encoded(&body.string(4096)?) + .map_err(|_| TransportError::Authentication)?; + let observation = keys + .verify_cell(&token, cell, schema, now) + .map_err(|_| TransportError::Authentication)?; + let length = u32::from_be_bytes(body.array()?) as usize; + Ok(CellQueryValue { + result: body.take(length)?.to_vec(), + observation, + token, + served_by, + }) + } else { + Err(decode_failure(status)?) + }; + if !body.0.is_empty() { + return Err(TransportError::Protocol); + } + Ok((result, leader_hint)) +} + +#[cfg(test)] +pub(in crate::raft) fn decode_cell_result( + bytes: &[u8], + cell: &CellHistory, + schema: [u8; 32], + keys: &VersionKeys, + now: u64, + limit: usize, + served_by: MemberId, +) -> Result, TransportError> { + decode_cell_reply(bytes, cell, schema, keys, now, limit, served_by).map(|(result, _)| result) +} diff --git a/crates/cellar/durability/raft/query_identity.rs b/crates/cellar/durability/raft/query_identity.rs index 4baf8d7fb..cfcaae16b 100644 --- a/crates/cellar/durability/raft/query_identity.rs +++ b/crates/cellar/durability/raft/query_identity.rs @@ -4,9 +4,10 @@ //! The complete target history remains authenticated in each request/reply. use super::transport::TransportError; -use crate::application::consensus::protocol::WireScope; +use cellar_logic::cell_group::{CellHistory, CellId, GroupHistory}; use cellar_logic::consensus::{ - DiskLineage, MemberId, NodeId, NodeIdentity, ProcessGeneration, ReplicaIdentity, + ClusterId, DiskLineage, GroupId, IncarnationId, MemberId, NamespaceId, NodeId, NodeIdentity, + ProcessGeneration, ReplicaIdentity, }; use serde::{Deserialize, Serialize}; @@ -41,45 +42,41 @@ impl Endpoint { } } +// Marker-1 hard cutover: only physical-group plus logical-cell query histories. +#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(tag = "kind", content = "identity", deny_unknown_fields)] +enum History { + Shared { + group: [[u8; 16]; 4], + cell: String, + incarnation: [u8; 16], + }, +} + +#[derive(Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +enum Purpose { + Query, + Command, + CommandProbe, + Enrollment, + Receipt, +} + #[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] #[serde(deny_unknown_fields)] pub(super) struct QueryRoute { - scope: WireScope, + purpose: Purpose, + scope: History, schema: [u8; 32], source: Endpoint, target: Endpoint, } impl QueryRoute { - pub fn new( - source: NodeIdentity, - source_label: &str, - target: &ReplicaIdentity, - target_label: &str, - schema: [u8; 32], - ) -> Result { - let route = Self { - scope: WireScope::from(&target.scope), - schema, - source: Endpoint::new(source, source_label, None), - target: Endpoint::new(target.node_identity(), target_label, Some(target.member)), - }; - route.decode()?; - Ok(route) - } - - /// Request orientation only. A reversed response route is compared with - /// the exact outstanding request; it cannot decode as another request. - pub fn decode(&self) -> Result<(NodeIdentity, ReplicaIdentity), TransportError> { - if self.source.member.is_some() { + fn endpoints(&self, scope: S) -> Result<(NodeIdentity, ReplicaIdentity), TransportError> { + if self.source.member.is_some() || self.schema == [0; 32] { return Err(TransportError::Identity); } - let scope = self - .scope - .clone() - .decode() - .map_err(|_| TransportError::Identity)?; - scope.validate().map_err(|_| TransportError::Identity)?; let member = self .target .member @@ -104,6 +101,116 @@ impl QueryRoute { )) } + pub fn new_shared( + source: NodeIdentity, + source_label: &str, + target: &ReplicaIdentity, + target_label: &str, + cell: &CellHistory, + schema: [u8; 32], + ) -> Result { + if !target.scope.same_namespace(cell) { + return Err(TransportError::Identity); + } + let g = target.scope; + let route = Self { + purpose: Purpose::Query, + scope: History::Shared { + group: [ + g.cluster.bytes(), + g.namespace.bytes(), + g.group.bytes(), + g.incarnation.bytes(), + ], + cell: cell.cell.as_str().into(), + incarnation: cell.incarnation.bytes(), + }, + schema, + source: Endpoint::new(source, source_label, None), + target: Endpoint::new(target.node_identity(), target_label, Some(target.member)), + }; + route.decode_shared()?; + Ok(route) + } + + pub fn decode_shared( + &self, + ) -> Result<(NodeIdentity, ReplicaIdentity, CellHistory), TransportError> { + let History::Shared { + group, + cell, + incarnation, + } = &self.scope; + let scope = GroupHistory { + cluster: ClusterId::new(group[0]).map_err(|_| TransportError::Identity)?, + namespace: NamespaceId::new(group[1]).map_err(|_| TransportError::Identity)?, + group: GroupId::new(group[2]).map_err(|_| TransportError::Identity)?, + incarnation: IncarnationId::new(group[3]).map_err(|_| TransportError::Identity)?, + }; + let cell = CellHistory { + cluster: scope.cluster, + namespace: scope.namespace, + cell: CellId::new(cell).map_err(|_| TransportError::Identity)?, + incarnation: IncarnationId::new(*incarnation).map_err(|_| TransportError::Identity)?, + }; + let (source, target) = self.endpoints(scope)?; + Ok((source, target, cell)) + } + + pub fn for_command(mut self) -> Result { + self.decode_shared()?; + self.purpose = Purpose::Command; + Ok(self) + } + + pub fn is_command(&self) -> bool { + self.purpose == Purpose::Command + } + + pub fn for_command_probe(mut self) -> Result { + self.decode_shared()?; + self.purpose = Purpose::CommandProbe; + Ok(self) + } + + pub fn is_command_probe(&self) -> bool { + self.purpose == Purpose::CommandProbe + } + + pub fn for_enrollment(mut self) -> Result { + self.decode_shared()?; + self.purpose = Purpose::Enrollment; + Ok(self) + } + + pub fn is_enrollment(&self) -> bool { + self.purpose == Purpose::Enrollment + } + + pub fn for_receipt(mut self) -> Result { + self.decode_shared()?; + self.purpose = Purpose::Receipt; + Ok(self) + } + + pub fn is_receipt(&self) -> bool { + self.purpose == Purpose::Receipt + } + + pub fn requires_command_grant(&self) -> bool { + self.is_command() || self.is_command_probe() || self.is_enrollment() || self.is_receipt() + } + + pub fn validate(&self) -> Result<(), TransportError> { + self.decode_shared()?; + Ok(()) + } + + pub fn source(&self) -> Result { + self.validate()?; + self.source.identity() + } + pub fn schema(&self) -> [u8; 32] { self.schema } @@ -118,6 +225,7 @@ impl QueryRoute { pub fn reversed(&self) -> Self { Self { + purpose: self.purpose, scope: self.scope.clone(), schema: self.schema, source: self.target.clone(), diff --git a/crates/cellar/durability/raft/query_reply.rs b/crates/cellar/durability/raft/query_reply.rs new file mode 100644 index 000000000..c341661dc --- /dev/null +++ b/crates/cellar/durability/raft/query_reply.rs @@ -0,0 +1,95 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! An authenticated reply owns its S3 output until the physical writer finishes. +use super::TransportError; +use crate::application::consensus::CommandFailure; +use crate::raft::cell_groups::{CellCommandResponse, CellReceiptResponse, CellSqlEnrollment}; +use crate::raft::query_wire as wire; +use cellar_logic::consensus::Durability; + +enum Owner { + Command { _response: Box }, + Enrollment { _response: Box }, + Receipt { _response: Box }, +} + +pub(super) struct Reply { + bytes: Vec, + _owner: Option, +} + +impl Reply { + pub(super) fn plain(bytes: Vec) -> Self { + Self { + bytes, + _owner: None, + } + } + + pub(super) fn bytes(&self) -> &[u8] { + &self.bytes + } + + pub(super) fn command( + result: Result, + limit: usize, + ) -> Result { + let bytes = wire::command::encode_result(&result, limit)?; + let owner = result + .ok() + .filter(|r| r.durability() == Durability::S3) + .map(|response| Owner::Command { + _response: Box::new(response), + }); + Ok(Self { + bytes, + _owner: owner, + }) + } + + pub(super) fn enrollment( + result: Result, + limit: usize, + ) -> Result { + let encoded = result + .as_ref() + .map(|r| (r.record(), r.durability())) + .map_err(Clone::clone); + let bytes = wire::enrollment::encode_result(&encoded, limit)?; + let owner = result + .ok() + .filter(|r| r.durability() == Durability::S3) + .map(|response| Owner::Enrollment { + _response: Box::new(response), + }); + Ok(Self { + bytes, + _owner: owner, + }) + } + + pub(super) fn receipt( + result: Result, + limit: usize, + ) -> Result { + let bytes = wire::receipt::encode_result(&result, limit)?; + let owner = result + .ok() + .filter(|r| r.durability() == Durability::S3) + .map(|response| Owner::Receipt { + _response: Box::new(response), + }); + Ok(Self { + bytes, + _owner: owner, + }) + } +} + +/// Fields drop in declaration order. On cancellation or task destruction, close +/// the real TLS/socket stream before refunding the proof/coverage/hold. On success +/// the caller takes the stream only after flush, then drops the consumed reply. +pub(super) struct Write { + pub stream: S, + pub reply: Reply, +} diff --git a/crates/cellar/durability/raft/query_route.rs b/crates/cellar/durability/raft/query_route.rs deleted file mode 100644 index 9f0f0a8a6..000000000 --- a/crates/cellar/durability/raft/query_route.rs +++ /dev/null @@ -1,194 +0,0 @@ -// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. - -//! Bounded read attempts. Every destination receives the original contract, -//! input and consistency; attempts consume one shared invocation deadline. - -use super::budget::Permit; -use super::query_wire::QueryCall; -use super::runtime::RaftRuntime; -use super::transport::{RaftTransport, TransportError}; -use crate::application::consensus::client::AttemptControl; -use crate::application::consensus::{ - CommandFailure, CommittedVersion, MemberId, OperationContract, QueryOptions, QueryResult, - QueryRouting, RaftCellHandle, VersionKeys, -}; -use std::{collections::BTreeSet, sync::Arc, time::Duration}; - -pub(super) struct RouteContext { - pub local: RaftCellHandle, - pub member: Option, - pub leader_hint: Option, - pub runtime: RaftRuntime, - pub transport: RaftTransport, - pub keys: VersionKeys, - pub control: Arc, - pub minimum: Option, -} - -pub(super) struct RoutedOutput { - pub result: QueryResult, - pub bytes: Permit, -} - -enum Attempt { - Success(RoutedOutput), - Retry(CommandFailure), - Fatal(CommandFailure), -} - -impl RouteContext { - pub async fn run( - self, - contract: OperationContract, - input: Vec, - mut options: QueryOptions, - ) -> Result { - let QueryRouting::Replicas { - members, - attempt_timeout, - leader_fallback, - } = std::mem::take(&mut options.routing) - else { - return Err(CommandFailure::UnsupportedConsistency); - }; - if members.is_empty() - || members.len() > 8 - || attempt_timeout.as_millis() == 0 - || attempt_timeout > Duration::from_secs(60) - { - return Err(CommandFailure::UnsupportedConsistency); - } - let mut tried = BTreeSet::new(); - let mut last = CommandFailure::VersionUnavailable; - for member in members { - if !tried.insert(member) { - return Err(CommandFailure::UnsupportedConsistency); - } - match self - .attempt(member, &contract, &input, &options, attempt_timeout) - .await - { - Attempt::Success(result) => return Ok(result), - Attempt::Fatal(error) => return Err(error), - Attempt::Retry(error) => last = error, - } - } - if leader_fallback && !self.control.cancelled() { - // This is a routing hint only. The destination still obtains the - // requested read proof; a stale leader hint grants no authority. - let leader = if self.member.is_some() { - let status = self.runtime.status(self.local.scope()); - let status = crate::asyncrt::select! { _ = self.control.cancellation() => return Err(CommandFailure::VersionUnavailable), value = status => value }; - status - .ok() - .and_then(|status| status.leader) - .and_then(|id| MemberId::new(id).ok()) - } else { - self.leader_hint - }; - if let Some(member) = leader.filter(|member| tried.insert(*member)) { - match self - .attempt(member, &contract, &input, &options, attempt_timeout) - .await - { - Attempt::Success(result) => return Ok(result), - Attempt::Fatal(error) => return Err(error), - Attempt::Retry(error) => last = error, - } - } - } - if self.control.cancelled() { - Err(CommandFailure::VersionUnavailable) - } else { - Err(last) - } - } - - async fn attempt( - &self, - member: MemberId, - contract: &OperationContract, - input: &[u8], - options: &QueryOptions, - timeout: Duration, - ) -> Attempt { - if self.control.cancelled() { - return Attempt::Fatal(CommandFailure::VersionUnavailable); - } - let domain = &self.local.handle.domain; - let Ok(now) = domain.mono_ms() else { - return Attempt::Fatal(CommandFailure::Unavailable); - }; - let deadline = self - .control - .deadline() - .min(now.saturating_add(timeout.as_millis() as u64)); - let Some(remaining) = deadline.checked_sub(now).filter(|ms| *ms > 0) else { - return Attempt::Fatal(CommandFailure::VersionUnavailable); - }; - let Ok(timer) = domain.sleep_until(deadline) else { - return Attempt::Fatal(CommandFailure::Unavailable); - }; - let mut options = options.clone(); - options.routing = QueryRouting::Local; - options.timeout = Duration::from_millis(remaining); - let query = async { - let result = if Some(member) == self.member { - self.local - .query(contract.clone(), input.to_vec(), options) - .await - } else { - let delivery = self.transport.query( - self.local.scope(), - member, - QueryCall { - contract: contract.clone(), - input: input.to_vec(), - options, - }, - self.keys.clone(), - deadline, - )?; - delivery.receive().await?.value - }; - Ok::<_, TransportError>(result) - }; - let result = crate::asyncrt::select! { - _ = self.control.cancellation() => return Attempt::Fatal(CommandFailure::VersionUnavailable), - _ = timer => return Attempt::Retry(CommandFailure::VersionUnavailable), - result = query => result, - }; - if self.control.cancelled() { - return Attempt::Fatal(CommandFailure::VersionUnavailable); - } - match result { - Ok(Ok(result)) => { - if result.served_by != member - || result.version.scope != *self.local.scope() - || self - .minimum - .as_ref() - .is_some_and(|minimum| result.version.covers(minimum) != Ok(true)) - { - return Attempt::Fatal(CommandFailure::CorruptState); - } - let Some(bytes) = self.local.handle.budget.acquire(result.result.len()) else { - return Attempt::Retry(CommandFailure::Overloaded); - }; - Attempt::Success(RoutedOutput { result, bytes }) - } - Ok(Err(error @ (CommandFailure::WrongIncarnation | CommandFailure::CorruptState))) => { - Attempt::Fatal(error) - } - Ok(Err(error)) => Attempt::Retry(error), - Err( - TransportError::Authentication - | TransportError::Identity - | TransportError::Protocol - | TransportError::Replay, - ) => Attempt::Fatal(CommandFailure::Unavailable), - Err(TransportError::Overloaded) => Attempt::Retry(CommandFailure::Overloaded), - Err(_) => Attempt::Retry(CommandFailure::VersionUnavailable), - } - } -} diff --git a/crates/cellar/durability/raft/query_transport.rs b/crates/cellar/durability/raft/query_transport.rs index 9884bd667..d85da607c 100644 --- a/crates/cellar/durability/raft/query_transport.rs +++ b/crates/cellar/durability/raft/query_transport.rs @@ -4,51 +4,15 @@ //! occupy the control or LTX connection required to establish its barrier. use super::*; -use crate::application::consensus::{client, CommandFailure, QueryResult, RaftCells, VersionKeys}; +use crate::application::consensus::client; use crate::raft::query_identity::QueryRoute; -use crate::raft::query_wire::{self as wire, QueryCall}; -use cellar_logic::consensus::{MemberId, NodeIdentity}; +use crate::raft::query_wire as wire; +use cellar_logic::consensus::NodeIdentity; use sha2::{Digest, Sha256}; -pub(crate) struct QueryResponse { - pub value: Result, - _node: Permit, - _peer: Permit, -} - -pub(crate) struct QueryDelivery { - result: Option>>, - cancel: watch::Sender, -} - -impl QueryDelivery { - pub async fn receive(mut self) -> Result { - self.result - .take() - .ok_or(TransportError::Closed)? - .await - .map_err(|_| TransportError::Closed)? - } -} - -impl Drop for QueryDelivery { - fn drop(&mut self) { - self.cancel.send_replace(true); - } -} - -pub(super) struct SendQuery { - scope: GroupScope, - route: Arc, - peer: RaftPeer, - call: QueryCall, - sequence: u64, - deadline: u64, - keys: VersionKeys, - cancel: watch::Receiver, - reply: oneshot::Sender>, - node: Permit, - peer_bytes: Permit, +pub(super) enum SendQuery { + Shared(Box), + Write(Box), } pub(super) struct OriginGrant { @@ -70,6 +34,7 @@ impl RaftTransport { let mut registry = state.registry.lock().map_err(|_| TransportError::Closed)?; state.admit(®istry)?; registry.check_node_view(&grant.peer)?; + registry.check_physical_endpoint(&grant.peer)?; if let Some(old) = registry.origins.get(&key) { if old.revision == grant.revision && old.peer.as_ref() == Some(&grant.peer) { return Ok(()); @@ -198,6 +163,7 @@ impl RaftTransport { state.admit(®istry)?; for peer in &physical { registry.check_node_view(peer)?; + registry.check_physical_endpoint(peer)?; } if let Some(old) = registry.query_groups.get(&route.scope) { if **old == route { @@ -206,7 +172,7 @@ impl RaftTransport { if old.revision >= route.revision || old.schema_digest != route.schema_digest { return Err(TransportError::StalePeer); } - } else if registry.query_groups.len() + registry.groups.len() >= state.options.max_groups { + } else if registry.route_count() >= state.options.max_groups { return Err(TransportError::Overloaded); } for peer in &physical { @@ -260,107 +226,12 @@ impl RaftTransport { *gateway = Some(Arc::downgrade(&handle)); Ok(()) } - - pub(in crate::raft) fn query( - &self, - scope: &GroupScope, - member: MemberId, - call: QueryCall, - keys: VersionKeys, - deadline: u64, - ) -> Result { - let state = &self.handle.0; - let registry = state.registry.lock().map_err(|_| TransportError::Closed)?; - state.admit(®istry)?; - if state.domain.mono_ms().map_err(|_| TransportError::Closed)? >= deadline { - return Err(TransportError::Timeout); - } - let route = registry - .groups - .get(scope) - .or_else(|| registry.query_groups.get(scope)) - .ok_or(TransportError::UnknownGroup)? - .clone(); - let local_node = state.runtime.log().limits().node; - let peer = route - .peers - .iter() - .find(|p| p.replica.member == member && p.replica.node != local_node) - .ok_or(TransportError::Identity)? - .clone(); - let pool = registry - .pools - .get(&peer.replica.node) - .ok_or(TransportError::StalePeer)? - .clone(); - drop(registry); - let charge = reservation( - call.input - .len() - .checked_add(4096) - .ok_or(TransportError::Limit)?, - &state.options, - )?; - let node = state - .outgoing - .query - .acquire(charge) - .ok_or(TransportError::Overloaded)?; - let peer_bytes = pool - .outgoing - .query - .acquire(charge) - .ok_or(TransportError::Overloaded)?; - // Validate/size the bounded representation before queueing, but retain - // only captured inputs. The sender encodes again with remaining time. - drop(call.encode(state.options.max_query_request_bytes)?); - let registry = state.registry.lock().map_err(|_| TransportError::Closed)?; - state.admit(®istry)?; - if !registry - .groups - .get(scope) - .or_else(|| registry.query_groups.get(scope)) - .is_some_and(|r| Arc::ptr_eq(r, &route)) - || !registry - .pools - .get(&peer.replica.node) - .is_some_and(|p| Arc::ptr_eq(p, &pool)) - { - return Err(TransportError::StalePeer); - } - let sequence = state - .sequence - .fetch_update(Ordering::AcqRel, Ordering::Acquire, |n| n.checked_add(1)) - .map_err(|_| TransportError::Closed)? - + 1; - let (reply, result) = oneshot::channel(); - let (cancel, cancelled) = watch::channel(false); - pool.query - .try_send(SendQuery { - scope: scope.clone(), - route, - peer, - call, - sequence, - deadline, - keys, - cancel: cancelled, - reply, - node, - peer_bytes, - }) - .map_err(|e| match e { - mpsc::error::TrySendError::Full(_) => TransportError::Overloaded, - mpsc::error::TrySendError::Closed(_) => TransportError::Closed, - })?; - Ok(QueryDelivery { - result: Some(result), - cancel, - }) - } } -fn reservation(request: usize, options: &TransportOptions) -> Result { +pub(super) fn reservation( + request: usize, + options: &TransportOptions, +) -> Result { request .checked_mul(2) .and_then(|n| { @@ -391,58 +262,33 @@ impl State { &self, wire: &QueryRoute, ) -> Result<(Arc, Option>), TransportError> { - let (source, target) = wire.decode()?; - let registry = self.registry.lock().map_err(|_| TransportError::Closed)?; - self.admit(®istry)?; - let route = registry - .groups - .get(&target.scope) - .ok_or(TransportError::UnknownGroup)?; - let group = self - .runtime - .log() - .group(&target.scope) - .map_err(|_| TransportError::UnknownGroup)?; - if *group.identity() != target - || wire.target_label() != self.auth.source() - || wire.schema() != route.schema_digest - { - return Err(TransportError::StalePeer); - } - let member_peer = route.peers.iter().find(|peer| { - peer.replica.node_identity() == source && peer.label == wire.source_label() - }); - let cache = group - .read() - .map_err(|error| TransportError::Runtime(error.into()))?; - if !cache.membership.permits(&target) { - return Err(TransportError::StalePeer); + if wire.requires_command_grant() { + return self.authorize_cell_command(wire); } - let conf = &cache.conf_state; - let present = |member| { - conf.voters - .iter() - .chain(&conf.voters_outgoing) - .chain(&conf.learners) - .chain(&conf.learners_next) - .any(|id| *id == member) - }; - if !present(target.member.get()) { - return Err(TransportError::StalePeer); - } - let implicit = member_peer.filter(|peer| { - present(peer.replica.member.get()) && cache.membership.permits(&peer.replica) - }); + self.authorize_cell_query(wire) + } + + /// Every cell read uses the same read-only origin grant, process binding, + /// revocation and peer-pool permission path. + pub(super) fn authorize_query_origin( + &self, + registry: &Registry, + cluster: ClusterId, + namespace: NamespaceId, + source: NodeIdentity, + source_label: &str, + implicit: Option<&RaftPeer>, + ) -> Result<(Arc, Option>), TransportError> { let grant = registry .origins - .get(&(target.scope.cluster, target.scope.namespace, source.node)) + .get(&(cluster, namespace, source.node)) .filter(|grant| !*grant.stop.borrow()) .filter(|grant| { grant.peer.as_ref().is_some_and(|peer| { peer.node == source.node && peer.disk == source.disk && peer.process == source.process - && peer.label == wire.source_label() + && peer.label == source_label }) }); let permission = if implicit.is_some() { @@ -464,28 +310,6 @@ impl State { Ok((pool, permission)) } - fn current_query_route( - &self, - scope: &GroupScope, - route: &Arc, - peer: &RaftPeer, - ) -> bool { - if self.runtime.log().group(scope).is_ok() { - return self.current_route(scope, route, peer); - } - self.registry.lock().is_ok_and(|registry| { - self.admit(®istry).is_ok() - && registry - .query_groups - .get(scope) - .is_some_and(|view| Arc::ptr_eq(view, route)) - && registry - .pools - .get(&peer.replica.node) - .is_some_and(|pool| pool.peer.same_host(peer)) - }) - } - pub(super) async fn query_until( &self, pool: &Pool, @@ -572,7 +396,7 @@ impl State { .bounded(None, read_query_head(&mut stream, &self.options, false)) .await?; let header = wire::read_request(&head, &self.auth, &exporter, &self.options)?; - let (source, target) = header.route.decode()?; + let source = header.route.source()?; let host = ( source.node, source.disk, @@ -615,65 +439,67 @@ impl State { }) .await?; header.verify_body(&body)?; - let call = QueryCall::decode(&body, self.options.max_query_request_bytes)?; + let call = super::cell_queries::IncomingCall::decode( + &header.route, + &body, + self.options.max_query_request_bytes, + )?; drop(body); self.authorize_query_route(&header.route)?; - let handle = self - .query_gateway - .lock() - .map_err(|_| TransportError::Closed)? - .as_ref() - .and_then(|handle| handle.upgrade()) - .ok_or(TransportError::Refused)?; - let gateway = RaftCells { handle }; - let cell = gateway - .cell(target.scope.namespace, &target.scope.cell) - .map_err(|_| TransportError::Refused)?; - if cell.scope() != &target.scope - || cell.binding.class.schema_digest() != header.route.schema() - { - return Err(TransportError::StalePeer); - } - let timeout = u64::try_from(call.options.timeout.as_millis()) - .map_err(|_| TransportError::Limit)?; let until = self .domain .mono_ms() .map_err(|_| TransportError::Closed)? - .saturating_add(timeout); - let query = cell.query(call.contract, call.input, call.options); + .checked_add(call.timeout_ms()) + .ok_or(TransportError::Limit)?; + let query = call.execute(&self, &header.route); let mut extra = [0; 1]; - // Query connections do not pipeline. EOF or unexpected input while - // waiting cancels the SDK waiter and its bounded SQL work. - let result = self.query_until(&pool, until, cancel.clone(), permission.clone(), async { + // No pipelining: EOF/unexpected input cancels the SDK waiter while + // the SQL owner retains any entered physical work through cleanup. + let body = self.query_until(&pool, until, cancel.clone(), permission.clone(), async { crate::asyncrt::select! { - value = query => Ok(value), + value = query => value, read = stream.read(&mut extra) => { let _ = read?; Err(TransportError::Closed) }, } }).await?; - if result - .as_ref() - .is_ok_and(|value| value.served_by != target.member) - { - return Err(TransportError::Identity); - } self.authorize_query_route(&header.route)?; - let body = wire::encode_result(&result, self.options.max_query_response_bytes)?; let response = wire::response_head( &self.auth, &exporter, &header, &head.tag, - &body, + body.bytes(), &self.options, )?; - self.query_until(&pool, until, cancel, permission, async { - write_head(&mut stream, &response).await?; - stream.write_all(&body).await?; - stream.flush().await?; - Ok(()) - }) - .await?; + // This aggregate closes the stream before dropping the output proof + // on every exit, including destruction of the entire serving task. + let mut writing = super::query_replies::Write { + stream, + reply: body, + }; + let observation = self + .performance + .timer(crate::raft::performance::Operation::QueryReplyWrite); + let result = self + .query_until(&pool, until, cancel, permission, async { + write_head(&mut writing.stream, &response).await?; + writing.stream.write_all(writing.reply.bytes()).await?; + writing.stream.flush().await?; + Ok(()) + }) + .await; + if let Err(error) = result { + drop(writing); + observation.finish(false); + return Err(error); + } + let super::query_replies::Write { + stream: resumed, + reply, + } = writing; + stream = resumed; + drop(reply); + observation.finish(true); self.accepted.fetch_add(1, Ordering::Relaxed); } } @@ -685,133 +511,115 @@ impl State { ) { let mut stop = self.stop.subscribe(); let mut peer_stop = pool.stop.subscribe(); - let mut connection = None; + let mut connection: Option = None; loop { if *stop.borrow() || *peer_stop.borrow() { return; } - let mut request = crate::asyncrt::select! { _ = stop.changed() => return, _ = peer_stop.changed() => return, request = requests.recv() => match request { Some(request) => request, None => return } }; - let result = self - .query_until( - &pool, - request.deadline, - request.cancel.clone(), - None, - async { - if !self.current_query_route(&request.scope, &request.route, &request.peer) - { - return Err(TransportError::StalePeer); - } - if connection.is_none() { - let owner = self.outgoing_connections.query(); - let tcp = TcpStream::connect(request.peer.query_address).await?; - tcp.set_nodelay(true)?; - let tcp = self.observed_socket(tcp, metrics::Lane::Query); - let name = rustls::pki_types::ServerName::try_from( - request.peer.server_name.clone(), - ) - .map_err(|_| TransportError::Identity)?; - let tls = tokio_rustls::TlsConnector::from(self.tls.client.clone()) - .connect(name, tcp) - .await - .map_err(|e| TransportError::Tls(e.to_string()))?; - let exporter = tls - .get_ref() - .1 - .export_keying_material([0; 32], wire::EXPORT_LABEL, None) - .map_err(|e| TransportError::Tls(e.to_string()))?; - connection = Some((tls, exporter, owner)); - } - let now = self.domain.mono_ms().map_err(|_| TransportError::Closed)?; - let remaining = request - .deadline - .checked_sub(now) - .filter(|ms| *ms > 0) - .ok_or(TransportError::Timeout)?; - request.call.options.timeout = request - .call - .options - .timeout - .min(std::time::Duration::from_millis(remaining)); - let body = request.call.encode(self.options.max_query_request_bytes)?; - let local = self.runtime.log().limits(); - let route = QueryRoute::new( - NodeIdentity { - node: local.node, - disk: local.disk, - process: local.process, - }, - self.auth.source(), - &request.peer.replica, - &request.peer.label, - request.route.schema_digest, - )?; - let header = wire::RequestHead::new(route, request.sequence, &body); - let (stream, exporter, _owner) = - connection.as_mut().ok_or(TransportError::Closed)?; - let head = wire::request_head( - &self.auth, - exporter, - &header, - &body, - &self.options, - )?; - write_head(stream, &head).await?; - stream.write_all(&body).await?; - stream.flush().await?; - drop(body); - let response = read_query_head(stream, &self.options, true).await?; - let digest = wire::read_response( - &response, - &self.auth, - exporter, - &header, - &head.tag, - &self.options, - )?; - let mut body = vec![0; response.payload_len]; - stream.read_exact(&mut body).await?; - if digest != <[u8; 32]>::from(Sha256::digest(&body)) { - return Err(TransportError::Authentication); - } - if !self.current_query_route(&request.scope, &request.route, &request.peer) - { - return Err(TransportError::StalePeer); - } - let now = u64::try_from(crate::asyncrt::wall_ms()) - .map_err(|_| TransportError::Authentication)?; - wire::decode_result( - &body, - &request.scope, - request.route.schema_digest, - &request.keys, - now, - self.options.max_query_response_bytes, - request.peer.replica.member, - ) - }, - ) - .await; - if result.is_err() { + // The server bounds its idle header read. Observe peer closure + // while the lane is idle so the next request can connect afresh. + // No response bytes are valid before the next request is sent. + let idle_read = async { + if let Some((stream, _, _)) = connection.as_mut() { + let mut unexpected = [0]; + let _ = stream.read(&mut unexpected).await; + } else { + std::future::pending::<()>().await; + } + }; + let next = crate::asyncrt::select_biased! { + "retire a closed idle connection before sending a queued request"; + _ = stop.changed() => return, + _ = peer_stop.changed() => return, + _ = idle_read => None, + request = requests.recv() => match request { + Some(request) => Some(request), + None => return, + }, + }; + let Some(request) = next else { + connection = None; + continue; + }; + let failed = match request { + SendQuery::Write(request) => { + self.send_cell_write(&pool, *request, &mut connection).await + } + SendQuery::Shared(request) => { + self.send_cell_query(&pool, *request, &mut connection).await + } + }; + if failed { connection = None; self.failed.fetch_add(1, Ordering::Relaxed); } - let SendQuery { - reply, - node, - peer_bytes, - .. - } = request; - let result = result.map(|value| QueryResponse { - value, - _node: node, - _peer: peer_bytes, - }); - let _ = reply.send(result); } } + + pub(super) async fn exchange_query( + &self, + connection: &mut Option, + peer: &RaftPeerNode, + route: QueryRoute, + body: Vec, + ) -> Result, TransportError> { + if connection.is_none() { + let owner = self.outgoing_connections.query(); + let tcp = TcpStream::connect(peer.query_address).await?; + tcp.set_nodelay(true)?; + let tcp = self.observed_socket(tcp, metrics::Lane::Query); + let name = rustls::pki_types::ServerName::try_from(peer.server_name.clone()) + .map_err(|_| TransportError::Identity)?; + let tls = tokio_rustls::TlsConnector::from(self.tls.client.clone()) + .connect(name, tcp) + .await + .map_err(|e| TransportError::Tls(e.to_string()))?; + let exporter = tls + .get_ref() + .1 + .export_keying_material([0; 32], wire::EXPORT_LABEL, None) + .map_err(|e| TransportError::Tls(e.to_string()))?; + *connection = Some((tls, exporter, owner)); + } + let (stream, exporter, _owner) = connection.as_mut().ok_or(TransportError::Closed)?; + // Assign in this peer's single sender, after dequeueing. Allocating at + // ingress can enqueue concurrent callers in reverse sequence order and + // make the receiver correctly refuse an otherwise valid read as replay. + let sequence = self + .sequence + .fetch_update(Ordering::AcqRel, Ordering::Acquire, |n| n.checked_add(1)) + .map_err(|_| TransportError::Closed)? + + 1; + let header = wire::RequestHead::new(route, sequence, &body); + let head = wire::request_head(&self.auth, exporter, &header, &body, &self.options)?; + write_head(stream, &head).await?; + stream.write_all(&body).await?; + stream.flush().await?; + drop(body); + let response = read_query_head(stream, &self.options, true).await?; + let digest = wire::read_response( + &response, + &self.auth, + exporter, + &header, + &head.tag, + &self.options, + )?; + let mut body = vec![0; response.payload_len]; + stream.read_exact(&mut body).await?; + if digest != <[u8; 32]>::from(Sha256::digest(&body)) { + return Err(TransportError::Authentication); + } + Ok(body) + } } +pub(super) type QueryConnection = ( + tokio_rustls::client::TlsStream>, + [u8; 32], + resources::ConnectionOwner, +); + async fn read_query_head( stream: &mut S, options: &TransportOptions, diff --git a/crates/cellar/durability/raft/query_wire.rs b/crates/cellar/durability/raft/query_wire.rs index 2e47a84d3..181048d95 100644 --- a/crates/cellar/durability/raft/query_wire.rs +++ b/crates/cellar/durability/raft/query_wire.rs @@ -8,204 +8,66 @@ use super::transport::TransportError; use super::transport_frame::{FrameHead, PREFIX}; use super::transport_options::TransportOptions; use crate::application::consensus::{ - CommandFailure, OperationContract, OperationKind, QueryConsistency, QueryOptions, QueryResult, - QueryRouting, StaleReadPolicy, VersionKeys, VersionToken, + CommandFailure, OperationContract, OperationKind, VersionKeys, }; use crate::peer_auth::PeerAuth; -use cellar_logic::consensus::{GroupScope, MemberId}; +use cellar_logic::consensus::{Durability, MemberId}; use serde::{Deserialize, Serialize}; use sha2::{Digest, Sha256}; -use std::time::Duration; -pub(super) const EXPORT_LABEL: &[u8] = b"EXPORTER-cellar-raft-query-1"; -const REQUEST: &[u8] = b"cellar-raft-query-request-1"; -const RESPONSE: &[u8] = b"cellar-raft-query-response-1"; -const MAX_TOKEN: usize = 1878; - -pub(super) struct QueryCall { - pub contract: OperationContract, - pub input: Vec, - pub options: QueryOptions, -} +#[path = "query_cell_wire.rs"] +pub(super) mod cell; +#[path = "command_cell_wire.rs"] +pub(super) mod command; +#[path = "enrollment_cell_wire.rs"] +pub(super) mod enrollment; +#[path = "receipt_cell_wire.rs"] +pub(super) mod receipt; -impl QueryCall { - pub fn encode(&self, limit: usize) -> Result, TransportError> { - if self.contract.kind != OperationKind::Query - || self.contract.operation.is_empty() - || self.contract.operation.len() > 128 - || self.contract.operation.chars().any(char::is_control) - || self.options.timeout.as_millis() == 0 - || self.options.timeout > Duration::from_secs(3600) - || !matches!(self.options.routing, QueryRouting::Local) - { - return Err(TransportError::Protocol); - } - let mut body = b"CLRQRQ01".to_vec(); - string(&mut body, &self.contract.operation)?; - body.extend_from_slice(&self.contract.digest); - match &self.options.consistency { - QueryConsistency::Stale => body.push(0), - QueryConsistency::AtLeast(token) => { - body.push(1); - string(&mut body, token.as_str())?; - } - QueryConsistency::Linearizable => body.push(2), - } - body.push(match self.options.stale_policy { - StaleReadPolicy::RecentSource => 0, - StaleReadPolicy::DisconnectedHistorical => 1, - }); - body.extend_from_slice(&(self.options.timeout.as_millis() as u64).to_be_bytes()); - let length = u32::try_from(self.input.len()).map_err(|_| TransportError::Limit)?; - if body - .len() - .checked_add(4) - .and_then(|n| n.checked_add(self.input.len())) - .is_none_or(|n| n > limit) - { - return Err(TransportError::Limit); - } - body.extend_from_slice(&length.to_be_bytes()); - body.extend_from_slice(&self.input); - Ok(body) +fn durability_code(value: Durability) -> u8 { + match value { + Durability::Quorum => 1, + Durability::S3 => 2, } +} - pub fn decode(bytes: &[u8], limit: usize) -> Result { - if bytes.len() > limit { - return Err(TransportError::Limit); - } - let mut body = Reader(bytes); - if body.take(8)? != b"CLRQRQ01" { - return Err(TransportError::Protocol); - } - let operation = body.string(128)?; - if operation.is_empty() || operation.chars().any(char::is_control) { - return Err(TransportError::Protocol); - } - let contract = OperationContract { - operation, - kind: OperationKind::Query, - digest: body.array()?, - }; - let consistency = match body.byte()? { - 0 => QueryConsistency::Stale, - 1 => QueryConsistency::AtLeast( - VersionToken::from_encoded(&body.string(MAX_TOKEN)?) - .map_err(|_| TransportError::Protocol)?, - ), - 2 => QueryConsistency::Linearizable, - _ => return Err(TransportError::Protocol), - }; - let stale_policy = match body.byte()? { - 0 => StaleReadPolicy::RecentSource, - 1 => StaleReadPolicy::DisconnectedHistorical, - _ => return Err(TransportError::Protocol), - }; - let millis = u64::from_be_bytes(body.array()?); - if millis == 0 || millis > 3_600_000 { - return Err(TransportError::Protocol); - } - let length = u32::from_be_bytes(body.array()?) as usize; - let input = body.take(length)?.to_vec(); - if !body.0.is_empty() { - return Err(TransportError::Protocol); - } - Ok(Self { - contract, - input, - options: QueryOptions { - consistency, - stale_policy, - timeout: Duration::from_millis(millis), - routing: QueryRouting::Local, - }, - }) +fn read_durability(body: &mut Reader<'_>) -> Result { + match body.byte()? { + 1 => Ok(Durability::Quorum), + 2 => Ok(Durability::S3), + _ => Err(TransportError::Protocol), } } -pub(super) fn encode_result( - result: &Result, - limit: usize, -) -> Result, TransportError> { - let mut body = b"CLRQRS01".to_vec(); - match result { - Ok(result) => { - body.push(0); - string(&mut body, result.version_token.as_str())?; - let length = u32::try_from(result.result.len()).map_err(|_| TransportError::Limit)?; - if body - .len() - .checked_add(4) - .and_then(|n| n.checked_add(result.result.len())) - .is_none_or(|n| n > limit) - { - return Err(TransportError::Limit); - } - body.extend_from_slice(&length.to_be_bytes()); - body.extend_from_slice(&result.result); - } - Err(failure) => body.push(match failure { - CommandFailure::VersionUnavailable => 1, - CommandFailure::WrongIncarnation => 2, - CommandFailure::UnsupportedCapability => 3, - CommandFailure::UnsupportedConsistency => 4, - CommandFailure::Overloaded => 5, - CommandFailure::CorruptState => 6, - CommandFailure::StorageFull => 8, - _ => 7, - }), - } - if body.len() > limit { - return Err(TransportError::Limit); +pub(super) const EXPORT_LABEL: &[u8] = b"EXPORTER-cellar-raft-query-1"; +const REQUEST: &[u8] = b"cellar-raft-query-request-1"; +const RESPONSE: &[u8] = b"cellar-raft-query-response-1"; + +fn failure_code(failure: &CommandFailure) -> u8 { + match failure { + CommandFailure::VersionUnavailable => 1, + CommandFailure::WrongIncarnation => 2, + CommandFailure::UnsupportedCapability => 3, + CommandFailure::UnsupportedConsistency => 4, + CommandFailure::Overloaded => 5, + CommandFailure::CorruptState => 6, + CommandFailure::StorageFull => 8, + _ => 7, } - Ok(body) } -pub(super) fn decode_result( - bytes: &[u8], - scope: &GroupScope, - schema: [u8; 32], - keys: &VersionKeys, - now_ms: u64, - limit: usize, - served_by: MemberId, -) -> Result, TransportError> { - if bytes.len() > limit { - return Err(TransportError::Limit); - } - let mut body = Reader(bytes); - if body.take(8)? != b"CLRQRS01" { - return Err(TransportError::Protocol); - } - let result = match body.byte()? { - 0 => { - let version_token = VersionToken::from_encoded(&body.string(MAX_TOKEN)?) - .map_err(|_| TransportError::Authentication)?; - let version = keys - .verify(&version_token, scope, schema, now_ms) - .map_err(|_| TransportError::Authentication)?; - let length = u32::from_be_bytes(body.array()?) as usize; - Ok(QueryResult { - result: body.take(length)?.to_vec(), - version, - version_token, - served_by, - }) - } - 1 => Err(CommandFailure::VersionUnavailable), - 2 => Err(CommandFailure::WrongIncarnation), - 3 => Err(CommandFailure::UnsupportedCapability), - 4 => Err(CommandFailure::UnsupportedConsistency), - 5 => Err(CommandFailure::Overloaded), - 6 => Err(CommandFailure::CorruptState), - 7 => Err(CommandFailure::Unavailable), - 8 => Err(CommandFailure::StorageFull), +fn decode_failure(status: u8) -> Result { + Ok(match status { + 1 => CommandFailure::VersionUnavailable, + 2 => CommandFailure::WrongIncarnation, + 3 => CommandFailure::UnsupportedCapability, + 4 => CommandFailure::UnsupportedConsistency, + 5 => CommandFailure::Overloaded, + 6 => CommandFailure::CorruptState, + 7 => CommandFailure::Unavailable, + 8 => CommandFailure::StorageFull, _ => return Err(TransportError::Protocol), - }; - if !body.0.is_empty() { - return Err(TransportError::Protocol); - } - Ok(result) + }) } #[derive(Clone, Serialize, Deserialize)] @@ -261,7 +123,7 @@ pub(super) fn request_head( options: &TransportOptions, ) -> Result { request.sequence()?; - request.route.decode()?; + request.route.validate()?; request.verify_body(body)?; make_head( auth, @@ -284,7 +146,7 @@ pub(super) fn read_request( let request: RequestHead = serde_json::from_slice(&head.header).map_err(|_| TransportError::Protocol)?; request.sequence()?; - request.route.decode()?; + request.route.validate()?; if serde_json::to_vec(&request).map_err(|_| TransportError::Protocol)? != head.header { return Err(TransportError::Protocol); } diff --git a/crates/cellar/durability/raft/query_wire_tests.rs b/crates/cellar/durability/raft/query_wire_tests.rs index 0a97ea159..d6b1319c0 100644 --- a/crates/cellar/durability/raft/query_wire_tests.rs +++ b/crates/cellar/durability/raft/query_wire_tests.rs @@ -1,15 +1,17 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. use super::*; -use crate::application::consensus::{CommittedVersion, VersionSigningKey}; +use crate::application::consensus::VersionSigningKey; +use crate::raft::cell_groups::CellQueryConsistency; +use cellar_logic::cell_group::{CellHistory, CellId, GroupHistory}; use cellar_logic::consensus::*; +use std::time::Duration; -fn identity(member: u64) -> ReplicaIdentity { +fn identity(member: u64) -> ReplicaIdentity { ReplicaIdentity { - scope: GroupScope { + scope: GroupHistory { cluster: ClusterId::new([1; 16]).unwrap(), namespace: NamespaceId::new([2; 16]).unwrap(), - cell: "Counter:one".into(), group: GroupId::new([3; 16]).unwrap(), incarnation: IncarnationId::new([4; 16]).unwrap(), }, @@ -20,12 +22,22 @@ fn identity(member: u64) -> ReplicaIdentity { } } +fn cell_history() -> CellHistory { + CellHistory { + cluster: identity(1).scope.cluster, + namespace: identity(1).scope.namespace, + cell: CellId::new("Counter:one").unwrap(), + incarnation: IncarnationId::new([5; 16]).unwrap(), + } +} + fn route() -> Route { - Route::new( + Route::new_shared( identity(1).node_identity(), "node-1", &identity(2), "node-2", + &cell_history(), [9; 32], ) .unwrap() @@ -40,62 +52,95 @@ fn keys() -> VersionKeys { .unwrap() } -fn call(consistency: QueryConsistency) -> QueryCall { - QueryCall { - contract: OperationContract { - operation: "Get".into(), - kind: OperationKind::Query, - digest: [11; 32], - }, - input: vec![1, 2, 3], - options: QueryOptions { - consistency, - timeout: Duration::from_millis(500), - ..Default::default() - }, - } +fn result_bytes() -> Vec { + let token = keys() + .issue_cell(&cell_observation(), [9; 32], 1000) + .unwrap(); + let mut bytes = b"CLGCQS01\0".to_vec(); + bytes.extend_from_slice(&0u64.to_be_bytes()); + string(&mut bytes, token.as_str()).unwrap(); + bytes.extend_from_slice(&3u32.to_be_bytes()); + bytes.extend_from_slice(&[8, 7, 6]); + bytes } -fn result() -> QueryResult { - let version = CommittedVersion { - scope: identity(1).scope, - position: LogPosition { - index: LogIndex::new(7), - term: Term::new(2), - }, - }; - QueryResult { - version_token: keys().issue(&version, [9; 32], 1000).unwrap(), - result: vec![8, 7, 6], - version, - served_by: MemberId::new(2).unwrap(), +// Canonical superseded values are test inputs, never production readers. +fn legacy_scope() -> GroupScope { + let group = identity(1).scope; + GroupScope { + cluster: group.cluster, + namespace: group.namespace, + cell: "Counter:one".into(), + group: group.group, + incarnation: group.incarnation, } } +fn legacy_route() -> serde_json::Value { + let mut value = serde_json::to_value(route()).unwrap(); + value["scope"] = serde_json::json!({ + "kind": "Cell", + "identity": crate::application::consensus::protocol::WireScope::from(&legacy_scope()), + }); + value +} + +fn legacy_request() -> Vec { + let mut body = b"CLRQRQ01".to_vec(); + string(&mut body, "Get").unwrap(); + body.extend_from_slice(&[11; 32]); + body.extend_from_slice(&[0, 0]); // Stale, RecentSource. + body.extend_from_slice(&500u64.to_be_bytes()); + body.extend_from_slice(&3u32.to_be_bytes()); + body.extend_from_slice(&[1, 2, 3]); + body +} + +fn legacy_reply() -> Vec { + let token = keys() + .issue( + &CommittedVersion { + scope: legacy_scope(), + position: LogPosition { + index: LogIndex::new(7), + term: Term::new(2), + }, + }, + [9; 32], + 1000, + ) + .unwrap(); + let mut body = b"CLRQRS01\0".to_vec(); + string(&mut body, token.as_str()).unwrap(); + body.extend_from_slice(&3u32.to_be_bytes()); + body.extend_from_slice(&[8, 7, 6]); + body +} + #[test] fn query_wire_preserves_explicit_constraints() { + let token = keys() + .issue_cell(&cell_observation(), [9; 32], 1000) + .unwrap(); for consistency in [ - QueryConsistency::Stale, - QueryConsistency::AtLeast(result().version_token), - QueryConsistency::Linearizable, + CellQueryConsistency::Stale, + CellQueryConsistency::RecentSource { max_age_ms: 5000 }, + CellQueryConsistency::AtLeast(token), + CellQueryConsistency::Linearizable, ] { - for stale_policy in [ - StaleReadPolicy::RecentSource, - StaleReadPolicy::DisconnectedHistorical, - ] { - let mut call = call(consistency.clone()); - call.options.stale_policy = stale_policy; - let bytes = call.encode(4096).unwrap(); - let decoded = QueryCall::decode(&bytes, 4096).unwrap(); - assert_eq!(decoded.contract, call.contract); - assert_eq!(decoded.input, call.input); - assert_eq!(decoded.options.consistency, consistency); - assert_eq!(decoded.options.timeout, call.options.timeout); - assert_eq!(decoded.options.stale_policy, stale_policy); - let mut trailing = bytes; - trailing.push(0); - assert!(QueryCall::decode(&trailing, 4096).is_err()); - } + let call = cell_call(consistency); + let bytes = call.encode(4096).unwrap(); + let decoded = cell::CellQueryCall::decode(&bytes, 4096).unwrap(); + assert_eq!(decoded.encode(4096).unwrap(), bytes); + assert_eq!(decoded.request.contract, call.request.contract); + assert_eq!(decoded.request.input, call.request.input); + assert_eq!(decoded.request.invocation, call.request.invocation); + assert_eq!(decoded.timeout_ms, call.timeout_ms); + assert_eq!(decoded.route_epoch, call.route_epoch); + assert_eq!(decoded.birth_digest, call.birth_digest); + let mut trailing = bytes; + trailing.push(0); + assert!(cell::CellQueryCall::decode(&trailing, 4096).is_err()); } } @@ -103,7 +148,9 @@ fn query_wire_preserves_explicit_constraints() { fn query_wire_binds_channel_request_route_and_payload() { let auth = PeerAuth::new([17; 32], "node-1").unwrap(); let options = TransportOptions::default(); - let body = call(QueryConsistency::Linearizable).encode(4096).unwrap(); + let body = cell_call(CellQueryConsistency::Linearizable) + .encode(4096) + .unwrap(); let request = RequestHead::new(route(), 7, &body); let head = request_head(&auth, &[3; 32], &request, &body, &options).unwrap(); assert!(read_request(&head, &auth, &[3; 32], &options).is_ok()); @@ -113,8 +160,7 @@ fn query_wire_binds_channel_request_route_and_payload() { modified[index] ^= 1; assert!(request.verify_body(&modified).is_err()); } - let value = result(); - let reply = encode_result(&Ok(value.clone()), 4096).unwrap(); + let reply = result_bytes(); let response = response_head(&auth, &[3; 32], &request, &head.tag, &reply, &options).unwrap(); assert_eq!( read_response(&response, &auth, &[3; 32], &request, &head.tag, &options).unwrap(), @@ -122,21 +168,21 @@ fn query_wire_binds_channel_request_route_and_payload() { ); assert!(read_response(&response, &auth, &[4; 32], &request, &head.tag, &options).is_err()); assert!(read_response(&response, &auth, &[3; 32], &request, &[5; 32], &options).is_err()); - let other_request = RequestHead::new(route(), 8, &body); assert!(read_response( &response, &auth, &[3; 32], - &other_request, + &RequestHead::new(route(), 8, &body), &head.tag, &options ) .is_err()); - let other_route = Route::new( + let other_route = Route::new_shared( identity(1).node_identity(), "node-1", &identity(3), "node-3", + &cell_history(), [9; 32], ) .unwrap(); @@ -149,42 +195,28 @@ fn query_wire_binds_channel_request_route_and_payload() { &options ) .is_err()); - assert_eq!( - decode_result( - &reply, - &value.version.scope, - [9; 32], - &keys(), - 1000, - 4096, - MemberId::new(2).unwrap() - ) + let member = MemberId::new(2).unwrap(); + let cell = cell_history(); + let value = cell::decode_cell_result(&reply, &cell, [9; 32], &keys(), 1000, 4096, member) .unwrap() - .unwrap(), - value + .unwrap(); + assert_eq!(value.observation, cell_observation()); + assert_eq!( + value.token, + keys() + .issue_cell(&cell_observation(), [9; 32], 1000) + .unwrap() ); - assert!(decode_result( - &reply, - &value.version.scope, - [10; 32], - &keys(), - 1000, - 4096, - MemberId::new(2).unwrap() - ) - .is_err()); - let mut wrong = value.version.scope.clone(); + assert_eq!(value.result, [8, 7, 6]); + assert_eq!(value.served_by, member); + assert!( + cell::decode_cell_result(&reply, &cell, [10; 32], &keys(), 1000, 4096, member).is_err() + ); + let mut wrong = cell; wrong.incarnation = IncarnationId::new([99; 16]).unwrap(); - assert!(decode_result( - &reply, - &wrong, - [9; 32], - &keys(), - 1000, - 4096, - MemberId::new(2).unwrap() - ) - .is_err()); + assert!( + cell::decode_cell_result(&reply, &wrong, [9; 32], &keys(), 1000, 4096, member).is_err() + ); } #[test] @@ -195,14 +227,23 @@ fn query_origin_binds_node_process_without_voting_identity() { process: ProcessGeneration::new([93; 16]).unwrap(), }; let target = identity(2); - let route = Route::new(origin, "ingress", &target, "node-2", [9; 32]).unwrap(); - assert_eq!(route.decode().unwrap(), (origin, target.clone())); - assert!(route.reversed().decode().is_err()); + let route = Route::new_shared( + origin, + "ingress", + &target, + "node-2", + &cell_history(), + [9; 32], + ) + .unwrap(); + assert_eq!( + route.decode_shared().unwrap(), + (origin, target.clone(), cell_history()) + ); + assert!(route.reversed().decode_shared().is_err()); let encoded = serde_json::to_value(&route).unwrap(); assert!(encoded["source"]["member"].is_null()); assert_eq!(encoded["target"]["member"], "2"); - // Even an authenticated query origin cannot reuse its route as the - // member-bearing route required for a consensus message. assert!( serde_json::from_value::(encoded.clone()).is_err() ); @@ -216,31 +257,33 @@ fn query_origin_binds_node_process_without_voting_identity() { modified["target"]["member"] = member; assert!(serde_json::from_value::(modified) .unwrap() - .decode() + .decode_shared() .is_err()); } let mut voting_origin = encoded.clone(); voting_origin["source"]["member"] = serde_json::json!("1"); assert!(serde_json::from_value::(voting_origin) .unwrap() - .decode() + .decode_shared() .is_err()); for endpoint in ["source", "target"] { for field in ["node", "disk", "process"] { let mut modified = encoded.clone(); - modified[endpoint][field] = serde_json::to_value([0_u8; 16]).unwrap(); + modified[endpoint][field] = serde_json::to_value([0u8; 16]).unwrap(); assert!(serde_json::from_value::(modified) .unwrap() - .decode() + .decode_shared() .is_err()); } } let auth = PeerAuth::new([17; 32], "ingress").unwrap(); let options = TransportOptions::default(); - let body = call(QueryConsistency::Linearizable).encode(4096).unwrap(); + let body = cell_call(CellQueryConsistency::Linearizable) + .encode(4096) + .unwrap(); let request = RequestHead::new(route, 7, &body); let head = request_head(&auth, &[3; 32], &request, &body, &options).unwrap(); - let reply = encode_result(&Ok(result()), 4096).unwrap(); + let reply = result_bytes(); let response = response_head(&auth, &[3; 32], &request, &head.tag, &reply, &options).unwrap(); let mut changed_origins = Vec::new(); for field in 0..3 { @@ -255,7 +298,7 @@ fn query_origin_binds_node_process_without_voting_identity() { changed_origins.push((origin, "other-ingress")); for (changed, label) in changed_origins { let other = RequestHead::new( - Route::new(changed, label, &target, "node-2", [9; 32]).unwrap(), + Route::new_shared(changed, label, &target, "node-2", &cell_history(), [9; 32]).unwrap(), 7, &body, ); @@ -267,7 +310,7 @@ fn query_origin_binds_node_process_without_voting_identity() { fn query_wire_refuses_size_replay_and_consensus_domain_confusion() { let auth = PeerAuth::new([17; 32], "node-1").unwrap(); let options = TransportOptions::default(); - let body = call(QueryConsistency::Stale).encode(4096).unwrap(); + let body = cell_call(CellQueryConsistency::Stale).encode(4096).unwrap(); assert!(request_head( &auth, &[3; 32], @@ -304,14 +347,14 @@ fn query_wire_refuses_size_replay_and_consensus_domain_confusion() { prefix = head.prefix; prefix[12..].copy_from_slice(&u64::MAX.to_be_bytes()); assert!(parse_prefix(&prefix, &options, false).is_err()); - let mut call = call(QueryConsistency::Stale); - call.options.timeout = Duration::ZERO; + let mut call = cell_call(CellQueryConsistency::Stale); + call.timeout_ms = 0; assert!(call.encode(4096).is_err()); - call.options.timeout = Duration::from_millis(500); - call.contract.kind = OperationKind::Command; + call.timeout_ms = 500; + call.request.contract.kind = OperationKind::Command; assert!(call.encode(4096).is_err()); - call.contract.kind = OperationKind::Query; - call.input = vec![0; 4096]; + call.request.contract.kind = OperationKind::Query; + call.request.input = vec![0; 4096]; assert!(call.encode(4096).is_err()); } @@ -327,27 +370,1055 @@ fn query_wire_failures_remain_typed() { CommandFailure::StorageFull, CommandFailure::Unavailable, ] { - let bytes = encode_result(&Err(failure.clone()), 4096).unwrap(); + let bytes = cell::encode_cell_result(&Err(failure.clone()), None, 4096).unwrap(); + assert!( + matches!(cell::decode_cell_result(&bytes, &cell_history(), [9; 32], &keys(), 1000, 4096, MemberId::new(2).unwrap()).unwrap(), Err(actual) if actual == failure) + ); + assert!(cell::decode_cell_result( + &bytes, + &cell_history(), + [9; 32], + &keys(), + 1000, + bytes.len() - 1, + MemberId::new(2).unwrap() + ) + .is_err()); + } +} + +#[test] +fn query_hard_cutover_refuses_legacy_route_request_and_reply() { + assert!(serde_json::from_value::(legacy_route()).is_err()); + assert!(cell::CellQueryCall::decode(&legacy_request(), 4096).is_err()); + assert!(cell::decode_cell_result( + &legacy_reply(), + &cell_history(), + [9; 32], + &keys(), + 1000, + 4096, + MemberId::new(2).unwrap() + ) + .is_err()); +} + +fn cell_observation() -> crate::raft::version::CellVersionObservation { + use cellar_logic::cell_group::*; + crate::raft::version::CellVersionObservation { + version: CellVersion { + history: cell_history(), + sequence: MutationSequence::new(7).unwrap(), + image: CellImage { + epoch: 1, + txid: 10, + checksum: 17, + commitment: CellImageCommitment::Materialized([8; 32]), + }, + }, + witness: CommitWitness { + group: identity(1).scope, + position: LogPosition { + index: LogIndex::new(100), + term: Term::new(9), + }, + proposal: ProposalId::new([6; 16]).unwrap(), + }, + } +} + +fn cell_call(consistency: crate::raft::cell_groups::CellQueryConsistency) -> cell::CellQueryCall { + cell::CellQueryCall { + request: crate::raft::cell_groups::CellQueryRequest { + invocation: ProposalId::new([91; 16]).unwrap(), + contract: OperationContract { + operation: "Get".into(), + kind: OperationKind::Query, + digest: [11; 32], + }, + input: vec![1, 2, 3], + consistency, + }, + timeout_ms: 700, + route_epoch: 3, + birth_digest: [92; 32], + } +} + +#[test] +fn shared_query_wire_preserves_cell_constraints_and_refuses_other_payloads() { + use crate::raft::cell_groups::CellQueryConsistency; + let token = keys() + .issue_cell(&cell_observation(), [9; 32], 1000) + .unwrap(); + for consistency in [ + CellQueryConsistency::Stale, + CellQueryConsistency::Linearizable, + CellQueryConsistency::AtLeast(token), + ] { + let call = cell_call(consistency); + let bytes = call.encode(4096).unwrap(); + let decoded = cell::CellQueryCall::decode(&bytes, 4096).unwrap(); + assert_eq!(decoded.encode(4096).unwrap(), bytes); + assert_eq!(decoded.request.invocation, call.request.invocation); + assert_eq!(decoded.request.contract, call.request.contract); + assert_eq!(decoded.request.input, call.request.input); + assert_eq!(decoded.timeout_ms, 700); + assert_eq!(decoded.route_epoch, 3); + assert_eq!(decoded.birth_digest, [92; 32]); + for end in 0..bytes.len() { + assert!(cell::CellQueryCall::decode(&bytes[..end], 4096).is_err()); + } + let mut trailing = bytes.clone(); + trailing.push(0); + assert!(cell::CellQueryCall::decode(&trailing, 4096).is_err()); + assert!(cell::CellQueryCall::decode(&bytes, bytes.len() - 1).is_err()); + } + assert!(cell::CellQueryCall::decode(&legacy_request(), 4096).is_err()); + let mut call = cell_call(CellQueryConsistency::Stale); + call.request.contract.kind = OperationKind::Command; + assert!(call.encode(4096).is_err()); + call.request.contract.kind = OperationKind::Query; + call.timeout_ms = 0; + assert!(call.encode(4096).is_err()); + call.timeout_ms = 700; + call.route_epoch = 0; + assert!(call.encode(4096).is_err()); + call.route_epoch = 3; + call.birth_digest = [0; 32]; + assert!(call.encode(4096).is_err()); +} + +#[test] +fn shared_query_route_binds_physical_group_logical_cell_and_process() { + use crate::raft::cell_groups::CellQueryConsistency; + use cellar_logic::cell_group::CellId; + let old = identity(2); + let target = ReplicaIdentity { + scope: old.scope, + node: old.node, + disk: old.disk, + process: old.process, + member: old.member, + }; + let cell = cell_observation().version.history; + let source = identity(1).node_identity(); + let route = Route::new_shared(source, "node-1", &target, "node-2", &cell, [9; 32]).unwrap(); + assert_eq!( + route.decode_shared().unwrap(), + (source, target.clone(), cell.clone()) + ); + assert!(route.reversed().decode_shared().is_err()); + assert!(serde_json::from_value::(legacy_route()).is_err()); + let value = serde_json::to_value(&route).unwrap(); + assert!(value["source"]["member"].is_null()); + assert!(serde_json::from_value::(value.clone()).is_err()); + for path in [ + ["source", "node"], + ["source", "disk"], + ["source", "process"], + ["target", "node"], + ["target", "disk"], + ["target", "process"], + ] { + let mut changed = value.clone(); + changed[path[0]][path[1]] = serde_json::to_value([0u8; 16]).unwrap(); + assert!(serde_json::from_value::(changed) + .unwrap() + .decode_shared() + .is_err()); + } + let mut changed = value.clone(); + changed["source"]["member"] = serde_json::json!("1"); + assert!(serde_json::from_value::(changed) + .unwrap() + .decode_shared() + .is_err()); + let mut changed = value.clone(); + changed["scope"]["identity"]["cell"] = serde_json::json!("bad\ncell"); + assert!(serde_json::from_value::(changed) + .unwrap() + .decode_shared() + .is_err()); + let auth = PeerAuth::new([17; 32], "node-1").unwrap(); + let options = TransportOptions::default(); + let body = cell_call(CellQueryConsistency::Linearizable) + .encode(4096) + .unwrap(); + let request = RequestHead::new(route.clone(), 17, &body); + let head = request_head(&auth, &[3; 32], &request, &body, &options).unwrap(); + assert!(read_request(&head, &auth, &[3; 32], &options).is_ok()); + assert!(read_request(&head, &auth, &[4; 32], &options).is_err()); + let reply = cell::encode_cell_result(&Err(CommandFailure::Overloaded), None, 4096).unwrap(); + let response = response_head(&auth, &[3; 32], &request, &head.tag, &reply, &options).unwrap(); + let mut other_cell = cell.clone(); + other_cell.cell = CellId::new("neighbor").unwrap(); + let mut other_group = target.clone(); + other_group.scope.group = GroupId::new([99; 16]).unwrap(); + let mut other_process = target.clone(); + other_process.process = ProcessGeneration::new([99; 16]).unwrap(); + for (target, cell) in [ + (&target, &other_cell), + (&other_group, &cell), + (&other_process, &cell), + ] { + let wrong = RequestHead::new( + Route::new_shared(source, "node-1", target, "node-2", cell, [9; 32]).unwrap(), + 17, + &body, + ); + assert!(read_response(&response, &auth, &[3; 32], &wrong, &head.tag, &options).is_err()); + } + for i in 0..body.len() { + let mut altered = body.clone(); + altered[i] ^= 1; + assert!(request.verify_body(&altered).is_err()); + } + let mut wrong_namespace = cell; + wrong_namespace.namespace = NamespaceId::new([99; 16]).unwrap(); + assert!(Route::new_shared( + source, + "node-1", + &target, + "node-2", + &wrong_namespace, + [9; 32] + ) + .is_err()); +} + +#[test] +fn shared_query_result_authenticates_cell_progress_not_group_log_index() { + let observation = cell_observation(); + let token = keys().issue_cell(&observation, [9; 32], 1000).unwrap(); + let mut bytes = b"CLGCQS01\0".to_vec(); + bytes.extend_from_slice(&0u64.to_be_bytes()); + string(&mut bytes, token.as_str()).unwrap(); + bytes.extend_from_slice(&3u32.to_be_bytes()); + bytes.extend_from_slice(b"yes"); + let member = MemberId::new(2).unwrap(); + let decoded = cell::decode_cell_result( + &bytes, + &observation.version.history, + [9; 32], + &keys(), + 1000, + 4096, + member, + ) + .unwrap() + .unwrap(); + assert_eq!(decoded.observation, observation); + assert_eq!(decoded.observation.version.sequence.get(), 7); + assert_eq!(decoded.observation.witness.position.index.get(), 100); + assert_eq!(decoded.result, b"yes"); + assert_eq!(decoded.served_by, member); + let mut wrong = observation.version.history.clone(); + wrong.incarnation = IncarnationId::new([99; 16]).unwrap(); + assert!( + cell::decode_cell_result(&bytes, &wrong, [9; 32], &keys(), 1000, 4096, member).is_err() + ); + assert!(cell::decode_cell_result( + &bytes, + &observation.version.history, + [8; 32], + &keys(), + 1000, + 4096, + member + ) + .is_err()); + assert!(cell::decode_cell_result( + &bytes, + &observation.version.history, + [9; 32], + &keys(), + 3000, + 4096, + member + ) + .is_err()); + assert!(cell::decode_cell_result( + &legacy_reply(), + &observation.version.history, + [9; 32], + &keys(), + 1000, + 4096, + member, + ) + .is_err()); + for end in 0..bytes.len() { + assert!(cell::decode_cell_result( + &bytes[..end], + &observation.version.history, + [9; 32], + &keys(), + 1000, + 4096, + member + ) + .is_err()); + } + for error in [ + CommandFailure::WrongIncarnation, + CommandFailure::Overloaded, + CommandFailure::VersionUnavailable, + CommandFailure::Unavailable, + CommandFailure::StorageFull, + CommandFailure::CorruptState, + CommandFailure::UnsupportedCapability, + CommandFailure::UnsupportedConsistency, + ] { + let bytes = cell::encode_cell_result(&Err(error.clone()), None, 4096).unwrap(); + assert!( + matches!(cell::decode_cell_result(&bytes, &observation.version.history, [9; 32], &keys(), 1000, 4096, member).unwrap(), Err(actual) if actual == error) + ); + } +} + +fn command_call() -> command::CellCommandCall { + command::CellCommandCall { + request: crate::raft::cell_groups::CellCommandRequest { + id: CommandId::new(ClientId::new([71; 16]).unwrap(), 3).unwrap(), + contract: OperationContract { + operation: "Add".into(), + digest: [72; 32], + kind: OperationKind::Command, + }, + input: vec![1, 2, 3], + durability: Durability::Quorum, + retry_until_ms: 9000, + }, + inputs: crate::storage::CommandInputs { + now_unix_millis: 1000, + random: [73; 32], + }, + timeout_ms: 1000, + route_epoch: 3, + birth_digest: [74; 32], + } +} + +#[test] +fn shared_command_wire_preserves_inputs_and_refuses_query_payloads() { + let call = command_call(); + let bytes = call.encode(4096).unwrap(); + let decoded = command::CellCommandCall::decode(&bytes, 4096).unwrap(); + assert_eq!(decoded.encode(4096).unwrap(), bytes); + assert_eq!(decoded.inputs.random, call.inputs.random); + assert_eq!(decoded.inputs.now_unix_millis, call.inputs.now_unix_millis); + assert_eq!(decoded.request.id, call.request.id); + assert!(cell::CellQueryCall::decode(&bytes, 4096).is_err()); + assert!(command::CellCommandCall::decode( + &cell_call(crate::raft::cell_groups::CellQueryConsistency::Stale) + .encode(4096) + .unwrap(), + 4096 + ) + .is_err()); + for end in 0..bytes.len() { + assert!(command::CellCommandCall::decode(&bytes[..end], 4096).is_err()); + } + let mut trailing = bytes.clone(); + trailing.push(0); + assert!(command::CellCommandCall::decode(&trailing, 4096).is_err()); + assert!(command::CellCommandCall::decode(&bytes, bytes.len() - 1).is_err()); + let mut call = call; + call.request.durability = Durability::S3; + let bytes = call.encode(4096).unwrap(); + assert_eq!( + command::CellCommandCall::decode(&bytes, 4096) + .unwrap() + .request + .durability, + Durability::S3 + ); +} + +#[test] +fn shared_command_purpose_is_authenticated_and_cannot_reuse_read_reply() { + let old = identity(2); + let target = ReplicaIdentity { + scope: old.scope, + member: old.member, + node: old.node, + disk: old.disk, + process: old.process, + }; + let cell = cell_observation().version.history; + let read = Route::new_shared( + identity(1).node_identity(), + "node-1", + &target, + "node-2", + &cell, + [9; 32], + ) + .unwrap(); + let write = read.clone().for_command().unwrap(); + assert!(write.is_command()); + assert!(!read.is_command()); + let auth = PeerAuth::new([17; 32], "node-1").unwrap(); + let options = TransportOptions::default(); + let body = command_call().encode(4096).unwrap(); + let request = RequestHead::new(write.clone(), 3, &body); + let head = request_head(&auth, &[3; 32], &request, &body, &options).unwrap(); + let mut value = serde_json::to_value(&request).unwrap(); + value["route"]["purpose"] = serde_json::json!("Query"); + let changed = FrameHead { + prefix: head.prefix, + header: serde_json::to_vec(&value).unwrap(), + tag: head.tag, + payload_len: head.payload_len, + }; + assert!(read_request(&changed, &auth, &[3; 32], &options).is_err()); + let response_body = + command::encode_result(&Err(CommandFailure::CommandConflict), 4096).unwrap(); + let response = response_head( + &auth, + &[3; 32], + &request, + &head.tag, + &response_body, + &options, + ) + .unwrap(); + let read_request = RequestHead::new(read, 3, &body); + assert!(read_response( + &response, + &auth, + &[3; 32], + &read_request, + &head.tag, + &options + ) + .is_err()); + let mut missing = serde_json::to_value(write).unwrap(); + missing.as_object_mut().unwrap().remove("purpose"); + assert!(serde_json::from_value::(missing).is_err()); +} + +#[test] +fn shared_command_results_bind_identity_history_and_typed_refusals() { + let call = command_call(); + let observation = cell_observation(); + let cell = &observation.version.history; + let token = keys().issue_cell(&observation, [9; 32], 1000).unwrap(); + let mut body = b"CLGCCS01".to_vec(); + body.push(0); + body.push(1); + body.extend_from_slice(&call.request.id.client.bytes()); + body.extend_from_slice(&call.request.id.sequence().to_be_bytes()); + body.extend_from_slice(&9000u64.to_be_bytes()); + string(&mut body, token.as_str()).unwrap(); + body.extend_from_slice(&2u32.to_be_bytes()); + body.extend_from_slice(b"ok"); + let decoded = command::decode_result( + &body, + (call.request.id, cell, Durability::Quorum), + [9; 32], + &keys(), + 1001, + 4096, + MemberId::new(2).unwrap(), + ) + .unwrap() + .unwrap(); + assert_eq!(decoded.value.observation, observation); + assert_eq!(decoded.value.result, b"ok"); + let wrong = CommandId::new(call.request.id.client, 4).unwrap(); + assert!(command::decode_result( + &body, + (wrong, cell, Durability::Quorum), + [9; 32], + &keys(), + 1001, + 4096, + MemberId::new(2).unwrap() + ) + .is_err()); + let mut other = cell.clone(); + other.incarnation = IncarnationId::new([78; 16]).unwrap(); + assert!(command::decode_result( + &body, + (call.request.id, &other, Durability::Quorum), + [9; 32], + &keys(), + 1001, + 4096, + MemberId::new(2).unwrap() + ) + .is_err()); + for failure in [ + CommandFailure::CommandConflict, + CommandFailure::CommandExpired, + CommandFailure::SequenceGap { next: 9 }, + CommandFailure::NotLeader { + leader: Some(MemberId::new(2).unwrap()), + }, + ] { + let body = command::encode_result(&Err(failure.clone()), 4096).unwrap(); + let decoded = command::decode_result( + &body, + (call.request.id, cell, Durability::Quorum), + [9; 32], + &keys(), + 1001, + 4096, + MemberId::new(2).unwrap(), + ) + .unwrap(); + assert_eq!(decoded.err(), Some(failure)); + } +} + +#[test] +fn shared_s3_results_refuse_durability_downgrades() { + let call = command_call(); + let observation = cell_observation(); + let cell = &observation.version.history; + let token = keys().issue_cell(&observation, [9; 32], 1000).unwrap(); + let creation = enrollment_call().request.creation; + let record = crate::raft::cell_groups::EnrollmentRecord::new( + cell.clone(), + creation, + [9; 32], + observation.version.image, + ) + .unwrap(); + for achieved in [Durability::Quorum, Durability::S3] { + let mut command = b"CLGCCS01".to_vec(); + command.extend_from_slice(&[0, durability_code(achieved)]); + command.extend_from_slice(&call.request.id.client.bytes()); + command.extend_from_slice(&call.request.id.sequence().to_be_bytes()); + command.extend_from_slice(&9000_u64.to_be_bytes()); + string(&mut command, token.as_str()).unwrap(); + command.extend_from_slice(&0_u32.to_be_bytes()); + let mut receipt = b"CLGCRS01".to_vec(); + receipt.extend_from_slice(&[13, durability_code(achieved)]); + receipt.extend_from_slice(&call.request.id.client.bytes()); + receipt.extend_from_slice(&call.request.id.sequence().to_be_bytes()); + string(&mut receipt, token.as_str()).unwrap(); + let enrollment = enrollment::encode_result(&Ok((&record, achieved)), 4096).unwrap(); + for required in [Durability::Quorum, Durability::S3] { + let a = command::decode_result( + &command, + (call.request.id, cell, required), + [9; 32], + &keys(), + 1001, + 4096, + MemberId::new(2).unwrap(), + ); + let b = receipt::decode_result( + &receipt, + (call.request.id, cell, required), + [9; 32], + &keys(), + 1001, + 4096, + MemberId::new(2).unwrap(), + ); + let c = + enrollment::decode_result(&enrollment, (creation, required), cell, [9; 32], 4096); + if achieved < required { + assert!(matches!(a, Err(TransportError::Authentication))); + assert!(matches!(b, Err(TransportError::Authentication))); + assert!(matches!(c, Err(TransportError::Authentication))); + } else { + assert_eq!(a.unwrap().unwrap().durability, achieved); + assert_eq!(b.unwrap().unwrap().value.durability, achieved); + let value = c.unwrap().unwrap(); + assert_eq!(value.record, record); + assert_eq!(value.durability, achieved); + } + } + // Unknown durability codes must not become either successful contract. + for invalid in [0, 3, 255] { + command[9] = invalid; + receipt[9] = invalid; + let mut enrollment = enrollment.clone(); + enrollment[9] = invalid; + assert!(command::decode_result( + &command, + (call.request.id, cell, Durability::Quorum), + [9; 32], + &keys(), + 1001, + 4096, + MemberId::new(2).unwrap() + ) + .is_err()); + assert!(receipt::decode_result( + &receipt, + (call.request.id, cell, Durability::Quorum), + [9; 32], + &keys(), + 1001, + 4096, + MemberId::new(2).unwrap() + ) + .is_err()); + assert!(enrollment::decode_result( + &enrollment, + (creation, Durability::Quorum), + cell, + [9; 32], + 4096 + ) + .is_err()); + } + } +} + +fn enrollment_call() -> enrollment::CellEnrollmentCall { + enrollment::CellEnrollmentCall { + request: crate::raft::cell_groups::CellEnrollmentRequest { + creation: cellar_logic::consensus::ProposalId::new([81; 16]).unwrap(), + durability: Durability::Quorum, + }, + inputs: crate::storage::CommandInputs { + now_unix_millis: 1234, + random: [82; 32], + }, + preview: cellar_ltx::generation::ImageId::new([83; 16]).unwrap(), + timeout_ms: 1000, + route_epoch: 3, + birth_digest: [84; 32], + } +} + +#[test] +fn shared_enrollment_wire_preserves_captured_identity_and_inputs() { + let call = enrollment_call(); + let bytes = call.encode(4096).unwrap(); + let decoded = enrollment::CellEnrollmentCall::decode(&bytes, 4096).unwrap(); + assert_eq!(decoded.encode(4096).unwrap(), bytes); + assert_eq!(decoded.request.creation, call.request.creation); + assert_eq!(decoded.preview, call.preview); + assert_eq!(decoded.inputs.random, call.inputs.random); + assert_eq!(decoded.inputs.now_unix_millis, call.inputs.now_unix_millis); + for end in 0..bytes.len() { + assert!(enrollment::CellEnrollmentCall::decode(&bytes[..end], 4096).is_err()); + } + let mut extra = bytes.clone(); + extra.push(0); + assert!(enrollment::CellEnrollmentCall::decode(&extra, 4096).is_err()); + assert!(enrollment::CellEnrollmentCall::decode(&bytes, bytes.len() - 1).is_err()); + assert!( + enrollment::CellEnrollmentCall::decode(&command_call().encode(4096).unwrap(), 4096) + .is_err() + ); + assert!(command::CellCommandCall::decode(&bytes, 4096).is_err()); + assert!(cell::CellQueryCall::decode(&bytes, 4096).is_err()); + let mut call = call; + call.request.durability = Durability::S3; + let bytes = call.encode(4096).unwrap(); + assert_eq!( + enrollment::CellEnrollmentCall::decode(&bytes, 4096) + .unwrap() + .request + .durability, + Durability::S3 + ); + call.request.durability = Durability::Quorum; + call.timeout_ms = 0; + assert!(call.encode(4096).is_err()); +} + +#[test] +fn shared_enrollment_purpose_cannot_be_reinterpreted_as_command_or_read() { + let old = identity(2); + let target = ReplicaIdentity { + scope: old.scope, + member: old.member, + node: old.node, + disk: old.disk, + process: old.process, + }; + let read = Route::new_shared( + identity(1).node_identity(), + "node-1", + &target, + "node-2", + &cell_observation().version.history, + [9; 32], + ) + .unwrap(); + let enrollment = read.clone().for_enrollment().unwrap(); + assert!(enrollment.requires_command_grant()); + assert!(enrollment.is_enrollment()); + assert!(!enrollment.is_command()); + let auth = PeerAuth::new([17; 32], "node-1").unwrap(); + let options = TransportOptions::default(); + let body = enrollment_call().encode(4096).unwrap(); + let request = RequestHead::new(enrollment, 4, &body); + let head = request_head(&auth, &[3; 32], &request, &body, &options).unwrap(); + for purpose in ["Query", "Command"] { + let mut value = serde_json::to_value(&request).unwrap(); + value["route"]["purpose"] = serde_json::json!(purpose); + let changed = FrameHead { + prefix: head.prefix, + header: serde_json::to_vec(&value).unwrap(), + tag: head.tag, + payload_len: head.payload_len, + }; + assert!(read_request(&changed, &auth, &[3; 32], &options).is_err()); + } + let bytes = enrollment::encode_result(&Err(CommandFailure::CommandConflict), 4096).unwrap(); + let response = response_head(&auth, &[3; 32], &request, &head.tag, &bytes, &options).unwrap(); + for route in [read.clone(), read.for_command().unwrap()] { + let other = RequestHead::new(route, 4, &body); + assert!(read_response(&response, &auth, &[3; 32], &other, &head.tag, &options).is_err()); + } +} + +#[test] +fn shared_enrollment_reply_binds_creation_cell_schema_and_initial_image() { + let call = enrollment_call(); + let cell = cell_observation().version.history; + let image = cell_observation().version.image; + let record = crate::raft::cell_groups::EnrollmentRecord::new( + cell.clone(), + call.request.creation, + [9; 32], + image, + ) + .unwrap(); + let bytes = enrollment::encode_result(&Ok((&record, Durability::Quorum)), 4096).unwrap(); + assert_eq!( + enrollment::decode_result( + &bytes, + (call.request.creation, Durability::Quorum), + &cell, + [9; 32], + 4096 + ) + .unwrap() + .unwrap() + .record, + record + ); + let mut other = cell.clone(); + other.incarnation = cellar_logic::consensus::IncarnationId::new([85; 16]).unwrap(); + assert!(enrollment::decode_result( + &bytes, + (call.request.creation, Durability::Quorum), + &other, + [9; 32], + 4096 + ) + .is_err()); + assert!(enrollment::decode_result( + &bytes, + (call.request.creation, Durability::Quorum), + &cell, + [8; 32], + 4096 + ) + .is_err()); + assert!(enrollment::decode_result( + &bytes, + ( + cellar_logic::consensus::ProposalId::new([86; 16]).unwrap(), + Durability::Quorum + ), + &cell, + [9; 32], + 4096 + ) + .is_err()); + for end in 0..bytes.len() { + assert!(enrollment::decode_result( + &bytes[..end], + (call.request.creation, Durability::Quorum), + &cell, + [9; 32], + 4096 + ) + .is_err()); + } + let mut extra = bytes.clone(); + extra.push(0); + assert!(enrollment::decode_result( + &extra, + (call.request.creation, Durability::Quorum), + &cell, + [9; 32], + 4096 + ) + .is_err()); + for failure in [ + CommandFailure::CommandConflict, + CommandFailure::NotLeader { + leader: Some(MemberId::new(2).unwrap()), + }, + CommandFailure::Overloaded, + ] { + let bytes = enrollment::encode_result(&Err(failure.clone()), 4096).unwrap(); assert_eq!( - decode_result( + enrollment::decode_result( &bytes, - &identity(1).scope, + (call.request.creation, Durability::Quorum), + &cell, + [9; 32], + 4096 + ) + .unwrap() + .unwrap_err(), + failure + ); + } +} + +fn receipt_call() -> receipt::CellReceiptCall { + let command = command_call(); + receipt::CellReceiptCall { + request: crate::raft::cell_groups::CellReceiptRequest { + id: command.request.id, + contract: command.request.contract, + input: command.request.input, + durability: command.request.durability, + }, + timeout_ms: command.timeout_ms, + route_epoch: command.route_epoch, + birth_digest: command.birth_digest, + } +} + +#[test] +fn shared_receipt_wire_refuses_command_payloads_and_preserves_identity() { + let call = receipt_call(); + let bytes = call.encode(4096).unwrap(); + let decoded = receipt::CellReceiptCall::decode(&bytes, 4096).unwrap(); + assert_eq!(decoded.encode(4096).unwrap(), bytes); + assert_eq!(decoded.request.id, call.request.id); + for end in 0..bytes.len() { + assert!(receipt::CellReceiptCall::decode(&bytes[..end], 4096).is_err()); + } + let mut extra = bytes.clone(); + extra.push(0); + assert!(receipt::CellReceiptCall::decode(&extra, 4096).is_err()); + assert!(receipt::CellReceiptCall::decode(&bytes, bytes.len() - 1).is_err()); + assert!(command::CellCommandCall::decode(&bytes, 4096).is_err()); + assert!(enrollment::CellEnrollmentCall::decode(&bytes, 4096).is_err()); + assert!(cell::CellQueryCall::decode(&bytes, 4096).is_err()); + assert!(receipt::CellReceiptCall::decode(&command_call().encode(4096).unwrap(), 4096).is_err()); + let mut call = call; + call.request.durability = Durability::S3; + let bytes = call.encode(4096).unwrap(); + assert_eq!( + receipt::CellReceiptCall::decode(&bytes, 4096) + .unwrap() + .request + .durability, + Durability::S3 + ); +} + +#[test] +fn shared_receipt_reply_distinguishes_unknown_from_empty_committed_result() { + let call = receipt_call(); + let observation = cell_observation(); + let cell = &observation.version.history; + let token = keys().issue_cell(&observation, [9; 32], 1000).unwrap(); + let member = MemberId::new(2).unwrap(); + for retained in [false, true] { + let mut body = b"CLGCRS01".to_vec(); + body.push(if retained { 0 } else { 13 }); + body.push(1); + body.extend_from_slice(&call.request.id.client.bytes()); + body.extend_from_slice(&call.request.id.sequence().to_be_bytes()); + string(&mut body, token.as_str()).unwrap(); + if retained { + body.extend_from_slice(&9000_u64.to_be_bytes()); + body.extend_from_slice(&0_u32.to_be_bytes()); + } + let value = receipt::decode_result( + &body, + (call.request.id, cell, Durability::Quorum), + [9; 32], + &keys(), + 1000, + 4096, + member, + ) + .unwrap() + .unwrap(); + assert_eq!(value.retained, retained); + assert!(value.value.value.result.is_empty()); + assert_eq!(value.value.value.observation, observation); + assert_eq!(value.value.retry_until_ms, if retained { 9000 } else { 0 }); + for end in 0..body.len() { + assert!(receipt::decode_result( + &body[..end], + (call.request.id, cell, Durability::Quorum), [9; 32], &keys(), 1000, 4096, - MemberId::new(2).unwrap() + member ) - .unwrap(), - Err(failure) + .is_err()); + } + let other_id = CommandId::new(call.request.id.client, 99).unwrap(); + assert!(receipt::decode_result( + &body, + (other_id, cell, Durability::Quorum), + [9; 32], + &keys(), + 1000, + 4096, + member + ) + .is_err()); + assert!(receipt::decode_result( + &body, + (call.request.id, cell, Durability::Quorum), + [8; 32], + &keys(), + 1000, + 4096, + member + ) + .is_err()); + let mut other = cell.clone(); + other.incarnation = IncarnationId::new([99; 16]).unwrap(); + assert!(receipt::decode_result( + &body, + (call.request.id, &other, Durability::Quorum), + [9; 32], + &keys(), + 1000, + 4096, + member + ) + .is_err()); + assert!(command::decode_result( + &body, + (call.request.id, cell, Durability::Quorum), + [9; 32], + &keys(), + 1000, + 4096, + member + ) + .is_err()); + body.push(0); + assert!(receipt::decode_result( + &body, + (call.request.id, cell, Durability::Quorum), + [9; 32], + &keys(), + 1000, + 4096, + member + ) + .is_err()); + } + for error in [ + CommandFailure::CommandExpired, + CommandFailure::CommandConflict, + CommandFailure::SequenceGap { next: 4 }, + ] { + let body = receipt::encode_result(&Err(error.clone()), 4096).unwrap(); + assert_eq!( + receipt::decode_result( + &body, + (call.request.id, cell, Durability::Quorum), + [9; 32], + &keys(), + 1000, + 4096, + member + ) + .unwrap() + .err(), + Some(error) ); - assert!(decode_result( + } +} + +#[test] +fn shared_receipt_purpose_cannot_authorize_mutation() { + let old = identity(2); + let target = ReplicaIdentity { + scope: old.scope, + member: old.member, + node: old.node, + disk: old.disk, + process: old.process, + }; + let base = Route::new_shared( + identity(1).node_identity(), + "node-1", + &target, + "node-2", + &cell_observation().version.history, + [9; 32], + ) + .unwrap(); + let lookup = base.clone().for_receipt().unwrap(); + assert!(lookup.requires_command_grant()); + assert!(!lookup.is_command()); + assert!(!lookup.is_enrollment()); + let auth = PeerAuth::new([17; 32], "node-1").unwrap(); + let options = TransportOptions::default(); + let body = receipt_call().encode(4096).unwrap(); + let request = RequestHead::new(lookup, 8, &body); + let head = request_head(&auth, &[3; 32], &request, &body, &options).unwrap(); + for purpose in ["Query", "Command", "Enrollment"] { + let mut value = serde_json::to_value(&request).unwrap(); + value["route"]["purpose"] = serde_json::json!(purpose); + let changed = FrameHead { + prefix: head.prefix, + header: serde_json::to_vec(&value).unwrap(), + tag: head.tag, + payload_len: head.payload_len, + }; + assert!(read_request(&changed, &auth, &[3; 32], &options).is_err()); + } +} + +#[test] +fn shared_query_leader_hint_is_advice_inside_authenticated_payload() { + for hint in [None, Some(MemberId::new(3).unwrap())] { + let bytes = + cell::encode_cell_result(&Err(CommandFailure::VersionUnavailable), hint, 4096).unwrap(); + let (result, decoded) = cell::decode_cell_reply( &bytes, - &identity(1).scope, + &cell_history(), [9; 32], &keys(), 1000, - bytes.len() - 1, + 4096, + MemberId::new(2).unwrap(), + ) + .unwrap(); + assert!(matches!(result, Err(CommandFailure::VersionUnavailable))); + assert_eq!(decoded, hint); + for length in 0..bytes.len() { + assert!(cell::decode_cell_reply( + &bytes[..length], + &cell_history(), + [9; 32], + &keys(), + 1000, + 4096, + MemberId::new(2).unwrap() + ) + .is_err()); + } + let mut extra = bytes.clone(); + extra.push(0); + assert!(cell::decode_cell_reply( + &extra, + &cell_history(), + [9; 32], + &keys(), + 1000, + 4096, MemberId::new(2).unwrap() ) .is_err()); diff --git a/crates/cellar/durability/raft/receipt_cell_wire.rs b/crates/cellar/durability/raft/receipt_cell_wire.rs new file mode 100644 index 000000000..49ecf9e6d --- /dev/null +++ b/crates/cellar/durability/raft/receipt_cell_wire.rs @@ -0,0 +1,228 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Receipt inspection has a distinct authenticated purpose and never stages a command. +use super::*; +use crate::raft::cell_groups::{CellReceiptRequest, CellReceiptResponse}; +use cellar_logic::{ + cell_group::CellHistory, + consensus::{CommandId, Durability}, +}; + +pub(in crate::raft) struct CellReceiptCall { + pub request: CellReceiptRequest, + pub timeout_ms: u64, + pub route_epoch: u64, + pub birth_digest: [u8; 32], +} + +pub(in crate::raft) struct CellReceiptValue { + pub retained: bool, + pub value: command::CellCommandValue, +} + +impl CellReceiptCall { + pub fn encode(&self, limit: usize) -> Result, TransportError> { + let r = &self.request; + if self.route_epoch == 0 || self.birth_digest == [0; 32] { + return Err(TransportError::Identity); + } + if r.contract.kind != OperationKind::Command + || r.contract.operation.is_empty() + || r.contract.operation.len() > 128 + || r.contract.operation.chars().any(char::is_control) + || !(1..=3_600_000).contains(&self.timeout_ms) + { + return Err(TransportError::Protocol); + } + let mut body = b"CLGCRQ01".to_vec(); + body.extend_from_slice(&r.id.client.bytes()); + body.extend_from_slice(&r.id.sequence().to_be_bytes()); + body.extend_from_slice(&self.route_epoch.to_be_bytes()); + body.extend_from_slice(&self.birth_digest); + string(&mut body, &r.contract.operation)?; + body.extend_from_slice(&r.contract.digest); + body.push(durability_code(r.durability)); + body.extend_from_slice(&self.timeout_ms.to_be_bytes()); + let length = u32::try_from(r.input.len()).map_err(|_| TransportError::Limit)?; + if body + .len() + .checked_add(4) + .and_then(|n| n.checked_add(r.input.len())) + .is_none_or(|n| n > limit) + { + return Err(TransportError::Limit); + } + body.extend_from_slice(&length.to_be_bytes()); + body.extend_from_slice(&r.input); + Ok(body) + } + + pub fn decode(bytes: &[u8], limit: usize) -> Result { + if bytes.len() > limit { + return Err(TransportError::Limit); + } + let mut body = Reader(bytes); + if body.take(8)? != b"CLGCRQ01" { + return Err(TransportError::Protocol); + } + let id = command::command_id(&mut body)?; + let route_epoch = u64::from_be_bytes(body.array()?); + let birth_digest = body.array()?; + if route_epoch == 0 || birth_digest == [0; 32] { + return Err(TransportError::Identity); + } + let operation = body.string(128)?; + if operation.is_empty() || operation.chars().any(char::is_control) { + return Err(TransportError::Protocol); + } + let contract = OperationContract { + operation, + kind: OperationKind::Command, + digest: body.array()?, + }; + let durability = read_durability(&mut body)?; + let timeout_ms = u64::from_be_bytes(body.array()?); + if !(1..=3_600_000).contains(&timeout_ms) { + return Err(TransportError::Protocol); + } + let length = u32::from_be_bytes(body.array()?) as usize; + let input = body.take(length)?.to_vec(); + if !body.0.is_empty() { + return Err(TransportError::Protocol); + } + Ok(Self { + request: CellReceiptRequest { + id, + contract, + input, + durability, + }, + timeout_ms, + route_epoch, + birth_digest, + }) + } +} + +pub(in crate::raft) fn encode_result( + result: &Result, + limit: usize, +) -> Result, TransportError> { + let mut body = b"CLGCRS01".to_vec(); + match result { + Ok(response) => { + body.push(if response.receipt().is_some() { 0 } else { 13 }); + body.push(durability_code(response.durability())); + body.extend_from_slice(&response.command().client.bytes()); + body.extend_from_slice(&response.command().sequence().to_be_bytes()); + string(&mut body, response.token().as_str())?; + if let Some(receipt) = response.receipt() { + body.extend_from_slice(&receipt.retry_until_ms().to_be_bytes()); + let len = + u32::try_from(receipt.result().len()).map_err(|_| TransportError::Limit)?; + if body + .len() + .checked_add(4) + .and_then(|n| n.checked_add(receipt.result().len())) + .is_none_or(|n| n > limit) + { + return Err(TransportError::Limit); + } + body.extend_from_slice(&len.to_be_bytes()); + body.extend_from_slice(receipt.result()); + } + } + Err(CommandFailure::CommandConflict) => body.push(9), + Err(CommandFailure::CommandExpired) => body.push(10), + Err(CommandFailure::SequenceGap { next }) => { + body.push(11); + body.extend_from_slice(&next.to_be_bytes()); + } + Err(CommandFailure::NotLeader { leader }) => { + body.push(12); + body.extend_from_slice(&leader.map_or(0, MemberId::get).to_be_bytes()); + } + Err(error) => body.push(failure_code(error)), + } + if body.len() > limit { + return Err(TransportError::Limit); + } + Ok(body) +} + +pub(in crate::raft) fn decode_result( + bytes: &[u8], + expected: (CommandId, &CellHistory, Durability), + schema: [u8; 32], + keys: &VersionKeys, + now: u64, + limit: usize, + served_by: MemberId, +) -> Result, TransportError> { + if bytes.len() > limit { + return Err(TransportError::Limit); + } + let mut body = Reader(bytes); + if body.take(8)? != b"CLGCRS01" { + return Err(TransportError::Protocol); + } + let result = match body.byte()? { + status @ (0 | 13) => { + let durability = read_durability(&mut body)?; + if durability < expected.2 { + return Err(TransportError::Authentication); + } + let command = command::command_id(&mut body)?; + if command != expected.0 { + return Err(TransportError::Authentication); + } + let token = crate::raft::version::CellVersionToken::from_encoded(&body.string(4096)?) + .map_err(|_| TransportError::Authentication)?; + let observation = keys + .verify_cell(&token, expected.1, schema, now) + .map_err(|_| TransportError::Authentication)?; + let retained = status == 0; + let (retry_until_ms, result) = if retained { + let retry = u64::from_be_bytes(body.array()?); + let len = u32::from_be_bytes(body.array()?) as usize; + (retry, body.take(len)?.to_vec()) + } else { + (0, vec![]) + }; + Ok(CellReceiptValue { + retained, + value: command::CellCommandValue { + command, + durability, + retry_until_ms, + value: cell::CellQueryValue { + result, + observation, + token, + served_by, + }, + }, + }) + } + 9 => Err(CommandFailure::CommandConflict), + 10 => Err(CommandFailure::CommandExpired), + 11 => Err(CommandFailure::SequenceGap { + next: u64::from_be_bytes(body.array()?), + }), + 12 => { + let id = u64::from_be_bytes(body.array()?); + Err(CommandFailure::NotLeader { + leader: if id == 0 { + None + } else { + Some(MemberId::new(id).map_err(|_| TransportError::Identity)?) + }, + }) + } + status => Err(decode_failure(status)?), + }; + if !body.0.is_empty() { + return Err(TransportError::Protocol); + } + Ok(result) +} diff --git a/crates/cellar/durability/raft/receipts.rs b/crates/cellar/durability/raft/receipts.rs index cacb68e00..713b3a287 100644 --- a/crates/cellar/durability/raft/receipts.rs +++ b/crates/cellar/durability/raft/receipts.rs @@ -90,6 +90,98 @@ pub(super) fn preflight_identity( request: &CommandRequest, limits: &ExecutionLimits, ) -> Result, ExecutionError> { + match inspect_identity(connection, request, limits)? { + Identity::Retained(receipt) => Ok(Some(receipt)), + Identity::Vacant { new_client } => { + if new_client { + let count: usize = connection + .query_row("SELECT count(*) FROM __sys_raft_clients", [], |r| r.get(0)) + .map_err(storage)?; + if count >= limits.max_client_identities { + return Err(ExecutionError::Refused(CommandFailure::Overloaded)); + } + } + Ok(None) + } + } +} + +/// Inspect immutable committed state without admitting a new client or receipt. +/// Capacity cannot obscure absence; identity, expiry, sequence and result-size +/// checks still apply. The caller supplies the consensus proof for using absence. +pub(super) fn lookup_identity( + connection: &Connection, + request: &CommandRequest, + limits: &ExecutionLimits, +) -> Result, ExecutionError> { + match inspect_identity(connection, request, limits)? { + Identity::Retained(receipt) => Ok(Some(receipt)), + Identity::Vacant { .. } => Ok(None), + } +} + +/// An initial cell image may precede receipt schema installation. Losing schema +/// after a mutation, or retaining only half of it, is corruption, not absence. +/// A missing receipt never proves that a submitted command cannot still commit. +pub(super) fn lookup_cell_identity( + connection: &Connection, + request: &CommandRequest, + limits: &ExecutionLimits, + cell_sequence: u64, +) -> Result, ExecutionError> { + lookup_cell_identity_admitted(connection, request, limits, cell_sequence, |_| Ok(())) +} + +/// Reserve a retained result after inspecting its length, before fetching bytes. +/// Admission failure is not absence and consumes no additional operation slot. +pub(super) fn lookup_cell_identity_admitted( + connection: &Connection, + request: &CommandRequest, + limits: &ExecutionLimits, + cell_sequence: u64, + mut reserve_result: impl FnMut(usize) -> Result<(), ExecutionError>, +) -> Result, ExecutionError> { + let tables: u64 = connection.query_row( + "SELECT count(*) FROM sqlite_schema WHERE type='table' AND name IN ('__sys_raft_clients','__sys_raft_receipts')", + [], |row| row.get(0), + ).map_err(storage)?; + match tables { + 0 if cell_sequence == 1 => { + if request.id.sequence() == 1 { + Ok(None) + } else { + Err(ExecutionError::Refused(CommandFailure::SequenceGap { + next: 1, + })) + } + } + 2 => match inspect_identity_admitted(connection, request, limits, &mut reserve_result)? { + Identity::Retained(receipt) => Ok(Some(receipt)), + Identity::Vacant { .. } => Ok(None), + }, + _ => Err(corrupt()), + } +} + +enum Identity { + Retained(StoredReceipt), + Vacant { new_client: bool }, +} + +fn inspect_identity( + connection: &Connection, + request: &CommandRequest, + limits: &ExecutionLimits, +) -> Result { + inspect_identity_admitted(connection, request, limits, &mut |_| Ok(())) +} + +fn inspect_identity_admitted( + connection: &Connection, + request: &CommandRequest, + limits: &ExecutionLimits, + reserve_result: &mut dyn FnMut(usize) -> Result<(), ExecutionError>, +) -> Result { let state = client(connection, request.id.client)?; let (last, floor) = state.unwrap_or((0, 0)); let sequence = request.id.sequence(); @@ -113,6 +205,7 @@ pub(super) fn preflight_identity( if length.is_none_or(|n| n < 0 || n as u64 > limits.max_result_bytes as u64) { return Err(corrupt()); } + reserve_result(length.ok_or_else(corrupt)? as usize)?; let row=connection.query_row( "SELECT operation,contract_digest,request_digest,result,minimum_durability,retry_until FROM __sys_raft_receipts WHERE client_id=?1 AND sequence=?2", @@ -133,7 +226,7 @@ pub(super) fn preflight_identity( 1 => Durability::S3, _ => return Err(corrupt()), }; - return Ok(Some(StoredReceipt { + return Ok(Identity::Retained(StoredReceipt { id: request.id, operation: row.0, contract_digest: contract, @@ -149,15 +242,9 @@ pub(super) fn preflight_identity( next, })); } - if state.is_none() { - let count: usize = connection - .query_row("SELECT count(*) FROM __sys_raft_clients", [], |r| r.get(0)) - .map_err(storage)?; - if count >= limits.max_client_identities { - return Err(ExecutionError::Refused(CommandFailure::Overloaded)); - } - } - Ok(None) + Ok(Identity::Vacant { + new_client: state.is_none(), + }) } pub(super) fn record( diff --git a/crates/cellar/durability/raft/receipts_tests.rs b/crates/cellar/durability/raft/receipts_tests.rs index 3e56e4f22..f4e0fa570 100644 --- a/crates/cellar/durability/raft/receipts_tests.rs +++ b/crates/cellar/durability/raft/receipts_tests.rs @@ -62,3 +62,131 @@ fn maximum_retry_deadline_is_never_expired() -> anyhow::Result<()> { assert_eq!(client(&connection, command.id.client)?, Some((1, 0))); Ok(()) } + +#[test] +fn committed_lookup_does_not_admit_new_client_at_capacity() -> anyhow::Result<()> { + let connection = Connection::open_in_memory()?; + install(&connection)?; + let limits = ExecutionLimits { + max_client_identities: 1, + max_retained_receipts: 1, + ..Default::default() + }; + let retained = request(1, 1000); + record(&connection, &retained, vec![7])?; + let mut absent = request(1, 1000); + absent.id = CommandId::new(ClientId::new([9; 16])?, 1)?; + assert!(lookup_identity(&connection, &absent, &limits)?.is_none()); + assert!(matches!( + preflight_identity(&connection, &absent, &limits), + Err(ExecutionError::Refused(CommandFailure::Overloaded)) + )); + assert_eq!( + lookup_identity(&connection, &retained, &limits)? + .unwrap() + .result(), + &[7] + ); + // Existing client admission remains independent from the new-client bound; + // the separate retained-receipt ceiling still refuses a new command. + assert!(preflight_identity(&connection, &request(2, 1000), &limits)?.is_none()); + assert!(matches!( + preflight(&connection, &request(2, 1000), &limits), + Err(ExecutionError::Refused(CommandFailure::Overloaded)) + )); + assert_eq!(retained_count(&connection)?, 1); + assert!(client(&connection, absent.id.client)?.is_none()); + Ok(()) +} + +#[test] +fn committed_lookup_preserves_identity_floor_and_allocation_checks() -> anyhow::Result<()> { + let connection = Connection::open_in_memory()?; + install(&connection)?; + let limits = ExecutionLimits::default(); + let original = request(1, 1000); + record(&connection, &original, vec![7, 8])?; + let mut conflict = request(1, 1000); + conflict.request_digest = [99; 32]; + assert!(matches!( + lookup_identity(&connection, &conflict, &limits), + Err(ExecutionError::Refused(CommandFailure::CommandConflict)) + )); + assert!(matches!( + lookup_identity(&connection, &request(3, 1000), &limits), + Err(ExecutionError::Refused(CommandFailure::SequenceGap { + next: 2 + })) + )); + let bounded = ExecutionLimits { + max_result_bytes: 1, + ..limits + }; + assert!(matches!( + lookup_identity(&connection, &original, &bounded), + Err(ExecutionError::Refused(CommandFailure::CorruptState)) + )); + prune(&connection, original.id.client, 1, 1001)?; + assert!(matches!( + lookup_identity(&connection, &original, &limits), + Err(ExecutionError::Refused(CommandFailure::CommandExpired)) + )); + assert!(lookup_identity(&connection, &request(2, 2000), &limits)?.is_none()); + assert_eq!(client(&connection, original.id.client)?, Some((1, 1))); + Ok(()) +} + +#[test] +fn cell_lookup_distinguishes_initial_absence_from_lost_schema() -> anyhow::Result<()> { + let connection = Connection::open_in_memory()?; + let command = request(1, 1000); + let limits = ExecutionLimits::default(); + assert!(lookup_cell_identity(&connection, &command, &limits, 1)?.is_none()); + assert!(matches!( + lookup_cell_identity(&connection, &request(2, 1000), &limits, 1), + Err(ExecutionError::Refused(CommandFailure::SequenceGap { + next: 1 + })) + )); + for sequence in [0, 2] { + assert!(matches!( + lookup_cell_identity(&connection, &command, &limits, sequence), + Err(ExecutionError::Refused(CommandFailure::CorruptState)) + )); + } + install(&connection)?; + // Inspection does not consume write capacity or invoke receipt maintenance. + let saturated = ExecutionLimits { + max_retained_receipts: 1, + max_client_identities: 1, + ..limits + }; + record(&connection, &command, vec![7])?; + assert_eq!( + lookup_cell_identity(&connection, &command, &saturated, 2)? + .unwrap() + .result(), + &[7] + ); + let mut other = command.clone(); + other.id = CommandId::new(ClientId::new([9; 16])?, 1)?; + assert!(lookup_cell_identity(&connection, &other, &saturated, 2)?.is_none()); + prune(&connection, command.id.client, 1, 1001)?; + assert!(matches!( + lookup_cell_identity(&connection, &command, &limits, 3), + Err(ExecutionError::Refused(CommandFailure::CommandExpired)) + )); + connection.execute_batch("DROP TABLE __sys_raft_receipts")?; + for sequence in [1, 3] { + assert!(matches!( + lookup_cell_identity(&connection, &command, &limits, sequence), + Err(ExecutionError::Refused(CommandFailure::CorruptState)) + )); + } + connection.execute_batch("DROP TABLE __sys_raft_clients")?; + assert!(matches!( + lookup_cell_identity(&connection, &command, &limits, 3), + Err(ExecutionError::Refused(CommandFailure::CorruptState)) + )); + Ok(()) +} diff --git a/crates/cellar/durability/raft/records.rs b/crates/cellar/durability/raft/records.rs index d7155e21d..a5da1cd1b 100644 --- a/crates/cellar/durability/raft/records.rs +++ b/crates/cellar/durability/raft/records.rs @@ -3,10 +3,14 @@ //! Marker-1 local metadata, atomically stored with its Raft log changes. use super::cache::{CachedGroup, CommittedImage, DurableApply}; -use super::storage::{GroupBootstrap, LogError}; -use crate::application::consensus::protocol::WireScope; +use super::cell_groups::GroupManifest; +use super::storage::{CellGroupBootstrap, GroupBootstrap, LogError}; use ::raft::prelude::{ConfState, HardState, Snapshot}; -use cellar_logic::consensus::{DiskLineage, EngineInstanceId, LogIndex, LogPosition, NodeId, Term}; +use cellar_logic::cell_group::GroupHistory; +use cellar_logic::consensus::{ + ClusterId, DiskLineage, EngineInstanceId, GroupId, GroupScope, IncarnationId, LogIndex, + LogPosition, NamespaceId, NodeId, Term, +}; use protobuf::Message; use serde::{Deserialize, Serialize}; @@ -57,7 +61,9 @@ pub(crate) struct RootRecord { pub(crate) struct MappingRecord { pub format: u32, pub key: u64, - pub scope: WireScope, + domain: MappingDomain, + history: PhysicalHistory, + application: MappingApplication, pub member: u64, pub birth_digest: [u8; 32], pub initial_conf: Vec, @@ -68,15 +74,147 @@ impl MappingRecord { Ok(Self { format: 1, key, - scope: WireScope::from(&bootstrap.scope), + domain: MappingDomain::Group, + history: PhysicalHistory::from(GroupHistory::from(&bootstrap.scope)), + application: MappingApplication::SingleCell(bootstrap.scope.cell.clone()), member: bootstrap.member.get(), birth_digest: bootstrap.birth_digest, initial_conf: conf.write_to_bytes().map_err(corrupt)?, }) } + pub fn application_scope(&self) -> Result { + let cell = match &self.application { + MappingApplication::SingleCell(cell) => cell.clone(), + MappingApplication::CellGroup(_) => return Err(LogError::WrongIdentity), + }; + let history = self.history()?; + let scope = GroupScope { + cluster: history.cluster, + namespace: history.namespace, + group: history.group, + incarnation: history.incarnation, + cell, + }; + scope.validate().map_err(corrupt)?; + Ok(scope) + } + + pub fn new_cells( + key: u64, + bootstrap: &CellGroupBootstrap, + conf: &ConfState, + ) -> Result { + let marker = GroupManifest::genesis(bootstrap.route.clone()).map_err(corrupt)?; + Ok(Self { + format: 1, + key, + domain: MappingDomain::Group, + history: PhysicalHistory::from(bootstrap.route.history), + application: MappingApplication::CellGroup(Box::new(marker)), + member: bootstrap.member.get(), + birth_digest: bootstrap.route.birth_digest, + initial_conf: conf.write_to_bytes().map_err(corrupt)?, + }) + } + + pub fn history(&self) -> Result { + self.history.decode() + } + + /// An imported genesis permanently binds the complete root and origin. + pub(crate) fn new_imported_cells( + key: u64, + bootstrap: &CellGroupBootstrap, + conf: &ConfState, + genesis: &GroupManifest, + ) -> Result { + genesis.check().map_err(corrupt)?; + if !(genesis.is_recovery_genesis() || genesis.is_transfer_genesis()) + || genesis.route() != &bootstrap.route + { + return Err(LogError::WrongIdentity); + } + let mut mapping = Self::new_cells(key, bootstrap, conf)?; + mapping.application = MappingApplication::CellGroup(Box::new(genesis.clone())); + Ok(mapping) + } + + pub(crate) fn recovery_genesis(&self) -> Option<&GroupManifest> { + match &self.application { + MappingApplication::CellGroup(marker) if marker.recovery_request().is_some() => { + Some(marker) + } + _ => None, + } + } + + pub fn is_cells(&self) -> bool { + matches!(&self.application, MappingApplication::CellGroup(_)) + } + + pub fn validate_cache(&self, cache: &CachedGroup) -> Result<(), LogError> { + if self.format != 1 { + return Err(corrupt("unsupported group marker")); + } + let history = self.history()?; + if cache.local_park.as_ref().is_some_and(|park| { + !park.matches_member(self.member) + || (park.is_open() && !cache.conf_state.voters.contains(&self.member)) + }) { + return Err(LogError::WrongIdentity); + } + if cache.recovery_genesis.as_deref() != self.recovery_genesis() { + return Err(LogError::WrongIdentity); + } + if let Some(marker) = &cache.group_manifest { + if !marker.matches_group(&history, self.birth_digest) { + return Err(LogError::WrongIdentity); + } + } + match &self.application { + MappingApplication::SingleCell(_) => { + cache.membership.validate_scope(&self.application_scope()?) + } + MappingApplication::CellGroup(genesis) => { + let expected = GroupManifest::genesis(genesis.route().clone()).map_err(corrupt)?; + if !genesis.matches_group(&history, self.birth_digest) + || cache + .group_manifest + .as_ref() + .is_none_or(|root| root.route() != genesis.route()) + || cache.applied.image.is_some() + { + return Err(LogError::WrongIdentity); + } + cache.membership.validate_scope(&history)?; + let current = cache + .group_manifest + .as_ref() + .ok_or(LogError::WrongIdentity)?; + if !current.preserves_transfer(genesis) { + return Err(LogError::WrongIdentity); + } + if genesis.recovery_request().is_some() { + if !genesis.is_recovery_genesis() + || !current.preserves_recovery(genesis) + || current.recovery_commitment().map_err(corrupt)? + != genesis.recovery_commitment().map_err(corrupt)? + || (current.recovery_activation().is_none() + && current.root() != genesis.root()) + { + return Err(LogError::WrongIdentity); + } + } else if !genesis.is_transfer_genesis() && genesis.as_ref() != &expected { + return Err(LogError::WrongIdentity); + } + Ok(()) + } + } + } + pub fn catalog_key(&self) -> Result, LogError> { - let scope = self.scope.clone().decode().map_err(corrupt)?; + let scope = self.history.decode()?; let mut key = b"g:".to_vec(); key.extend_from_slice(&scope.group.bytes()); key.extend_from_slice(&scope.incarnation.bytes()); @@ -84,6 +222,52 @@ impl MappingRecord { } } +#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(tag = "domain", content = "state", deny_unknown_fields)] +enum MappingApplication { + #[serde(rename = "single-cell-1")] + SingleCell(String), + #[serde(rename = "cell-group-1")] + CellGroup(Box), +} + +#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] +enum MappingDomain { + #[serde(rename = "cellar-group-mapping-1")] + Group, +} + +#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct PhysicalHistory { + cluster: [u8; 16], + namespace: [u8; 16], + group: [u8; 16], + incarnation: [u8; 16], +} + +impl From for PhysicalHistory { + fn from(history: GroupHistory) -> Self { + Self { + cluster: history.cluster.bytes(), + namespace: history.namespace.bytes(), + group: history.group.bytes(), + incarnation: history.incarnation.bytes(), + } + } +} + +impl PhysicalHistory { + fn decode(&self) -> Result { + Ok(GroupHistory { + cluster: ClusterId::new(self.cluster).map_err(corrupt)?, + namespace: NamespaceId::new(self.namespace).map_err(corrupt)?, + group: GroupId::new(self.group).map_err(corrupt)?, + incarnation: IncarnationId::new(self.incarnation).map_err(corrupt)?, + }) + } +} + #[derive(Serialize, Deserialize)] #[serde(deny_unknown_fields)] pub(crate) struct GroupRecord { @@ -95,11 +279,51 @@ pub(crate) struct GroupRecord { pub last_index: u64, pub applied_index: u64, pub applied_term: u64, - pub image: Option, + pub application: ApplicationRecord, +} + +// Explicit domains refuse the old bare-image record shape. This staged image +// variant is removed with legacy runtime consumers at the public hard cutover. +#[derive(Serialize, Deserialize)] +#[serde(tag = "domain", content = "state", deny_unknown_fields)] +pub(crate) enum ApplicationRecord { + #[serde(rename = "cellar-image-apply-1")] + Image(Option), + #[serde(rename = "cellar-group-apply-1")] + Group(Box), +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(crate) struct GroupApplication { + manifest: super::cell_groups::GroupManifest, + // Required and local. A peer snapshot never changes this retention policy. + archive_required: bool, + // Required, local to this mapping/engine, and never snapshot payload. + seed: LocalGroupSeed, + #[serde(default, skip_serializing_if = "Option::is_none")] + local_park: Option, +} + +#[derive(Serialize, Deserialize)] +#[serde(tag = "state", rename_all = "snake_case", deny_unknown_fields)] +enum LocalGroupSeed { + Absent, + Installed { index: u64 }, +} + +impl LocalGroupSeed { + fn index(&self) -> Option { + match self { + Self::Absent => None, + Self::Installed { index } => Some(*index), + } + } } impl GroupRecord { pub fn from_cache(mapping: MappingRecord, cache: &CachedGroup) -> Result { + mapping.validate_cache(cache)?; Ok(Self { mapping, hard_state: cache.hard_state.write_to_bytes().map_err(corrupt)?, @@ -109,7 +333,17 @@ impl GroupRecord { last_index: cache.last_index(), applied_index: cache.applied.position.index.get(), applied_term: cache.applied.position.term.get(), - image: cache.applied.image.clone(), + application: match &cache.group_manifest { + Some(marker) => ApplicationRecord::Group(Box::new(GroupApplication { + manifest: marker.clone(), + archive_required: cache.archive_required, + local_park: cache.local_park.clone(), + seed: cache.group_seed.map_or(LocalGroupSeed::Absent, |index| { + LocalGroupSeed::Installed { index } + }), + })), + None => ApplicationRecord::Image(cache.applied.image.clone()), + }, }) } @@ -117,20 +351,62 @@ impl GroupRecord { if self.mapping.format != 1 { return Err(LogError::Corrupt("unsupported group marker".into())); } - Ok(CachedGroup { + let (image, group_manifest, group_seed) = match &self.application { + ApplicationRecord::Image(image) => (image.clone(), None, None), + ApplicationRecord::Group(application) => { + let marker = &application.manifest; + let scope = self.mapping.history()?; + if !marker.matches_group(&scope, self.mapping.birth_digest) { + return Err(LogError::WrongIdentity); + } + (None, Some(marker.clone()), application.seed.index()) + } + }; + let snapshot = Snapshot::parse_from_bytes(&self.snapshot).map_err(corrupt)?; + let checkpoint_manifest = group_manifest + .as_ref() + .map(|current| { + super::cell_groups::checkpoint::persisted_marker( + &snapshot, + current, + &self.membership, + ) + }) + .transpose()? + .flatten(); + let cache = CachedGroup { + archive_required: match &self.application { + ApplicationRecord::Group(application) => application.archive_required, + ApplicationRecord::Image(_) => false, + }, hard_state: HardState::parse_from_bytes(&self.hard_state).map_err(corrupt)?, conf_state: ConfState::parse_from_bytes(&self.conf_state).map_err(corrupt)?, membership: self.membership.clone(), - snapshot: Snapshot::parse_from_bytes(&self.snapshot).map_err(corrupt)?, + snapshot, entries: Default::default(), + group_manifest, + recovery_genesis: self + .mapping + .recovery_genesis() + .cloned() + .map(std::sync::Arc::new), + group_seed, + local_park: match &self.application { + ApplicationRecord::Group(application) => application.local_park.clone(), + ApplicationRecord::Image(_) => None, + }, + manifest_pins: None, + checkpoint_manifest, applied: DurableApply { position: LogPosition { index: LogIndex::new(self.applied_index), term: Term::new(self.applied_term), }, - image: self.image.clone(), + image, }, - }) + }; + self.mapping.validate_cache(&cache)?; + Ok(cache) } } diff --git a/crates/cellar/durability/raft/replica_leadership.rs b/crates/cellar/durability/raft/replica_leadership.rs index c3adf17c6..c8e672ea6 100644 --- a/crates/cellar/durability/raft/replica_leadership.rs +++ b/crates/cellar/durability/raft/replica_leadership.rs @@ -44,76 +44,16 @@ impl Replica { ) -> Result<(), RuntimeError> { let progress = self.replication_status()?; let cache = self.log.read()?; - let voters = &cache.conf_state.voters; - if check.through.get() == 0 - || self.node.raft.pending_conf_index > self.node.raft.raft_log.applied - || self.node.raft.lead_transferee.is_some() - || progress.term != check.expected_term - || progress.membership_revision != check.expected_revision - || progress.committed < check.through - || voters.len() != 3 - || !cache.conf_state.voters_outgoing.is_empty() - || !cache.conf_state.learners.is_empty() - || !cache.conf_state.learners_next.is_empty() - || cache.conf_state.auto_leave - || check.survivors[0] == check.survivors[1] - || check - .survivors - .iter() - .any(|member| !voters.contains(&member.get())) - || check.survivors.iter().any(|member| { - progress - .members - .iter() - .find(|p| p.member == *member) - .is_none_or(|p| { - !p.voter - || p.matched < check.through - || (*member != self.identity().member && !p.recent_active) - }) - }) - { - return Err(RuntimeError::NotReady); - } - if check.all_voters - && voters.iter().any(|id| { - progress - .members - .iter() - .find(|p| p.member.get() == *id) - .is_none_or(|p| { - !p.voter - || p.matched < check.through - || (*id != self.identity().member.get() && !p.recent_active) - }) - }) - { - return Err(RuntimeError::NotReady); - } - let local = self.identity().member; - match &check.transfer { - Some((target, selector)) => { - if check.survivors.contains(&local) - || !check.survivors.contains(target) - || selector.boot.node != self.identity().node.bytes() - || selector.boot.disk != self.identity().disk.bytes() - || selector.boot.process != self.identity().process.bytes() - { - return Err(RuntimeError::NotReady); - } - } - None if !check.survivors.contains(&local) => return Err(RuntimeError::NotReady), - None => (), - } + let transfer = super::super::restart_leadership::check_progress( + &progress, + &cache.conf_state, + &check, + self.node.raft.pending_conf_index > self.node.raft.raft_log.applied, + self.node.raft.lead_transferee.is_some(), + )?; drop(cache); - if let Some((target, _)) = check.transfer { - // The existing owner also requires this destination to be caught up - // to the latest commit, and refuses repair/joint/active transfers. - self.transfer_leadership(LeadershipTransfer { - target, - expected_term: check.expected_term, - expected_revision: check.expected_revision, - })?; + if let Some(transfer) = transfer { + self.transfer_leadership(transfer)?; } Ok(()) } diff --git a/crates/cellar/durability/raft/restart_leadership.rs b/crates/cellar/durability/raft/restart_leadership.rs new file mode 100644 index 000000000..5e73e5156 --- /dev/null +++ b/crates/cellar/durability/raft/restart_leadership.rs @@ -0,0 +1,107 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! One restart-readiness contract, evaluated by the serialized physical owner. +//! Replication observations are inputs; only the owner can admit a transfer. +use super::completion::{ + LeadershipTransfer, ReplicationStatus, RestartLeadershipCheck, RuntimeError, +}; +use super::storage::ReplicaLog; +use ::raft::prelude::ConfState; + +#[cfg(test)] +#[path = "restart_leadership_tests.rs"] +mod tests; + +pub(crate) fn check_gate( + log: &ReplicaLog, + now: u64, + check: &RestartLeadershipCheck, +) -> Result<(), RuntimeError> { + if now >= check.deadline { + return Err(RuntimeError::Closed); + } + if let Some((_, selector)) = &check.transfer { + if log.lifecycle_guard()?.as_ref().and_then(|g| g.quiescing()) != Some(selector) { + return Err(RuntimeError::NotReady); + } + } + Ok(()) +} + +pub(crate) fn check_progress( + progress: &ReplicationStatus, + configuration: &ConfState, + check: &RestartLeadershipCheck, + pending_configuration: bool, + transferring: bool, +) -> Result, RuntimeError> { + let voters = &configuration.voters; + if check.through.get() == 0 + || pending_configuration + || transferring + || progress.term != check.expected_term + || progress.membership_revision != check.expected_revision + || progress.committed < check.through + || voters.len() != 3 + || !configuration.voters_outgoing.is_empty() + || !configuration.learners.is_empty() + || !configuration.learners_next.is_empty() + || configuration.auto_leave + || check.survivors[0] == check.survivors[1] + || check + .survivors + .iter() + .any(|member| !voters.contains(&member.get())) + || check.survivors.iter().any(|member| { + progress + .members + .iter() + .find(|p| p.member == *member) + .is_none_or(|p| { + !p.voter + || p.matched < check.through + || (*member != progress.replica.member && !p.recent_active) + }) + }) + { + return Err(RuntimeError::NotReady); + } + if check.all_voters + && voters.iter().any(|id| { + progress + .members + .iter() + .find(|p| p.member.get() == *id) + .is_none_or(|p| { + !p.voter + || p.matched < check.through + || (*id != progress.replica.member.get() && !p.recent_active) + }) + }) + { + return Err(RuntimeError::NotReady); + } + let local = progress.replica.member; + match &check.transfer { + Some((target, selector)) => { + if check.survivors.contains(&local) + || !check.survivors.contains(target) + || selector.boot.node != progress.replica.node.bytes() + || selector.boot.disk != progress.replica.disk.bytes() + || selector.boot.process != progress.replica.process.bytes() + { + return Err(RuntimeError::NotReady); + } + } + None if !check.survivors.contains(&local) => return Err(RuntimeError::NotReady), + None => (), + } + Ok(check + .transfer + .as_ref() + .map(|(target, _)| LeadershipTransfer { + target: *target, + expected_term: check.expected_term, + expected_revision: check.expected_revision, + })) +} diff --git a/crates/cellar/durability/raft/restart_leadership_tests.rs b/crates/cellar/durability/raft/restart_leadership_tests.rs new file mode 100644 index 000000000..503545dae --- /dev/null +++ b/crates/cellar/durability/raft/restart_leadership_tests.rs @@ -0,0 +1,240 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Metadata checks grant no physical group, quorum or lifecycle authority. +use super::*; +use crate::raft::completion::MemberProgress; +use crate::raft::fleet_stop::{BootBinding, LifecyclePurpose, LifecycleUpdate, StopSelector}; +use cellar_logic::cell_group::GroupHistory; +use cellar_logic::consensus::*; + +fn progress() -> ReplicationStatus { + ReplicationStatus { + replica: ReplicaIdentity { + scope: GroupHistory { + cluster: ClusterId::new([4; 16]).unwrap(), + namespace: NamespaceId::new([5; 16]).unwrap(), + group: GroupId::new([6; 16]).unwrap(), + incarnation: IncarnationId::new([7; 16]).unwrap(), + }, + node: NodeId::new([1; 16]).unwrap(), + disk: DiskLineage::new([2; 16]).unwrap(), + process: ProcessGeneration::new([3; 16]).unwrap(), + member: MemberId::new(1).unwrap(), + }, + term: Term::new(8), + membership_revision: LogIndex::new(9), + committed: LogIndex::new(10), + members: (1..=3) + .map(|id| MemberProgress { + member: MemberId::new(id).unwrap(), + matched: LogIndex::new(10), + voter: true, + recent_active: true, + }) + .collect(), + } +} + +fn configuration() -> ConfState { + ConfState { + voters: vec![1, 2, 3], + ..Default::default() + } +} + +fn check() -> RestartLeadershipCheck { + RestartLeadershipCheck { + deadline: 1000, + expected_term: Term::new(8), + expected_revision: LogIndex::new(9), + through: LogIndex::new(10), + survivors: [MemberId::new(1).unwrap(), MemberId::new(2).unwrap()], + all_voters: true, + transfer: None, + } +} + +#[test] +fn group_restart_observation_requires_third_voter_matched_and_active() { + let mut progress = progress(); + let conf = configuration(); + let mut check = check(); + assert!(matches!( + check_progress(&progress, &conf, &check, false, false), + Ok(None) + )); + progress.members[2].matched = LogIndex::new(9); + assert!(matches!( + check_progress(&progress, &conf, &check, false, false), + Err(RuntimeError::NotReady) + )); + check.all_voters = false; + assert!(matches!( + check_progress(&progress, &conf, &check, false, false), + Ok(None) + )); + check.all_voters = true; + progress.members[2].matched = LogIndex::new(10); + progress.members[2].recent_active = false; + assert!(matches!( + check_progress(&progress, &conf, &check, false, false), + Err(RuntimeError::NotReady) + )); + progress.members[2].recent_active = true; + assert!(matches!( + check_progress(&progress, &conf, &check, false, false), + Ok(None) + )); +} + +#[test] +fn group_restart_readiness_rejects_changed_term_revision_and_configuration() { + for scenario in 0..8 { + let mut progress = progress(); + let mut conf = configuration(); + let mut check = check(); + let mut pending = false; + let mut transferring = false; + match scenario { + 0 => progress.term = Term::new(9), + 1 => progress.membership_revision = LogIndex::new(10), + 2 => conf.voters_outgoing = vec![1, 2, 3], + 3 => conf.learners = vec![4], + 4 => pending = true, + 5 => transferring = true, + 6 => check.survivors[1] = check.survivors[0], + _ => progress.committed = LogIndex::new(9), + } + assert!( + matches!( + check_progress(&progress, &conf, &check, pending, transferring), + Err(RuntimeError::NotReady) + ), + "accepted stale readiness scenario {scenario}" + ); + } +} + +fn selector(boot: BootBinding) -> StopSelector { + StopSelector { + purpose: LifecyclePurpose::Maintenance, + boot, + epoch: 2, + operation: [11; 16], + finalization: [12; 16], + request: [13; 16], + } +} + +#[test] +fn group_restart_transfer_binds_source_process_and_survivor_roles() { + let progress = progress(); + let conf = configuration(); + let mut check = check(); + let boot = BootBinding { + catalog: [14; 32], + fleet: [15; 16], + node: progress.replica.node.bytes(), + disk: progress.replica.disk.bytes(), + engine: [16; 16], + boot: [17; 16], + process: progress.replica.process.bytes(), + epoch: 1, + join_key: [18; 32], + }; + check.survivors = [MemberId::new(2).unwrap(), MemberId::new(3).unwrap()]; + check.transfer = Some((MemberId::new(2).unwrap(), selector(boot))); + let transfer = check_progress(&progress, &conf, &check, false, false) + .unwrap() + .unwrap(); + assert_eq!(transfer.target, MemberId::new(2).unwrap()); + assert_eq!(transfer.expected_term, progress.term); + check.transfer.as_mut().unwrap().1.boot.process = [19; 16]; + assert!(matches!( + check_progress(&progress, &conf, &check, false, false), + Err(RuntimeError::NotReady) + )); + check.transfer.as_mut().unwrap().1.boot.process = progress.replica.process.bytes(); + check.survivors[0] = progress.replica.member; + assert!(matches!( + check_progress(&progress, &conf, &check, false, false), + Err(RuntimeError::NotReady) + )); +} + +#[tokio::test] +async fn group_restart_gate_requires_exact_synced_quiescence_and_rejects_fence( +) -> anyhow::Result<()> { + use crate::asyncrt::ExecutionDomain; + use crate::raft::options::{LogOpenMode, ReplicaLogOptions}; + let root = tempfile::tempdir()?; + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let log = domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + root.path().join("log"), + NodeId::new([1; 16])?, + DiskLineage::new([2; 16])?, + ProcessGeneration::new([3; 16])?, + LogOpenMode::Create, + ), + std::sync::Arc::new(cellar_ltx::DirectFileSystem), + )) + .await??; + let deadline = log.domain().mono_ms()? + 5000; + let boot = BootBinding { + catalog: [14; 32], + fleet: [15; 16], + node: log.limits().node.bytes(), + disk: log.limits().disk.bytes(), + process: log.limits().process.bytes(), + engine: log.storage_instance()?.bytes(), + boot: [17; 16], + epoch: 1, + join_key: crate::peer_probe::PeerProbeSigner::from_secret([18; 32]).public_key_bytes(), + }; + log.update_lifecycle(LifecycleUpdate::Boot { + boot: boot.clone(), + deadline, + }) + .await?; + let selector = selector(boot.clone()); + let mut check = check(); + check.deadline = deadline; + check.transfer = Some((MemberId::new(2)?, selector.clone())); + assert!(matches!( + check_gate(&log, log.domain().mono_ms()?, &check), + Err(RuntimeError::NotReady) + )); + log.update_lifecycle(LifecycleUpdate::Quiesce { + selector: selector.clone(), + deadline, + }) + .await?; + check_gate(&log, log.domain().mono_ms()?, &check)?; + check.transfer.as_mut().unwrap().1.request = [20; 16]; + assert!(matches!( + check_gate(&log, log.domain().mono_ms()?, &check), + Err(RuntimeError::NotReady) + )); + check.transfer.as_mut().unwrap().1 = selector.clone(); + log.update_lifecycle(LifecycleUpdate::Fence { + boot, + epoch: selector.epoch, + operation: selector.operation, + request: [21; 16], + deadline, + }) + .await?; + assert!(matches!( + check_gate(&log, log.domain().mono_ms()?, &check), + Err(RuntimeError::NotReady) + )); + assert!(matches!( + check_gate(&log, deadline, &check), + Err(RuntimeError::Closed) + )); + log.close().await?; + domain.shutdown().await; + Ok(()) +} diff --git a/crates/cellar/durability/raft/rooted/query.rs b/crates/cellar/durability/raft/rooted/query.rs index fe0584c1a..e71a5081b 100644 --- a/crates/cellar/durability/raft/rooted/query.rs +++ b/crates/cellar/durability/raft/rooted/query.rs @@ -69,7 +69,7 @@ pub(crate) fn query_registered_rooted_controlled( position: published.position(), }; let result = class - .invoke_query(contract, &version, &database, input) + .invoke_group_query(contract, &version, &database, input) .map_err(ExecutionError::Storage)?; if result.len() > options.execution.max_result_bytes { return Err(ExecutionError::Refused( diff --git a/crates/cellar/durability/raft/runtime.rs b/crates/cellar/durability/raft/runtime.rs index ab173a63b..a8edaee06 100644 --- a/crates/cellar/durability/raft/runtime.rs +++ b/crates/cellar/durability/raft/runtime.rs @@ -143,13 +143,20 @@ impl Drop for Requests { } } -struct Terminal { +pub(crate) struct Terminal { result: Mutex>>, changed: Notify, } impl Terminal { - fn complete(&self, result: Result<(), RuntimeError>) { + pub(crate) fn new() -> Self { + Self { + result: Mutex::new(None), + changed: Notify::new(), + } + } + + pub(crate) fn complete(&self, result: Result<(), RuntimeError>) { let mut terminal = self.result.lock().unwrap_or_else(|e| e.into_inner()); if terminal.is_none() { *terminal = Some(result); @@ -158,7 +165,7 @@ impl Terminal { self.changed.notify_waiters(); } - async fn wait(&self) -> Result<(), RuntimeError> { + pub(crate) async fn wait(&self) -> Result<(), RuntimeError> { loop { let changed = self.changed.notified(); tokio::pin!(changed); @@ -246,10 +253,7 @@ impl RaftRuntime { let (ready, wait) = oneshot::channel(); let closed = Arc::new(AtomicBool::new(false)); let closing = Arc::new(AtomicBool::new(false)); - let terminal = Arc::new(Terminal { - result: Mutex::new(None), - changed: Notify::new(), - }); + let terminal = Arc::new(Terminal::new()); let stop = Arc::new(Notify::new()); let requests = Requests { receiver: requests, @@ -1154,29 +1158,13 @@ impl Worker { // catalog cancellation CAS was lost. Read-only observation // on a surviving leader needs no local quiescence. let gate = match &action { - GroupAction::RestartLeadership(check) - if self - .domain - .mono_ms() - .map_or(true, |now| now >= check.deadline) => - { - Err(RuntimeError::Closed) - } - GroupAction::RestartLeadership(check) => match &check.transfer { - Some((_, selector)) => self - .log - .lifecycle_guard() - .map_err(RuntimeError::from) - .and_then(|guard| { - if guard.as_ref().and_then(|g| g.quiescing()) == Some(selector) - { - Ok(()) - } else { - Err(RuntimeError::NotReady) - } - }), - None => Ok(()), - }, + GroupAction::RestartLeadership(check) => self + .domain + .mono_ms() + .map_err(|_| RuntimeError::Closed) + .and_then(|now| { + super::restart_leadership::check_gate(&self.log, now, check) + }), _ => Ok(()), }; let result = if new_read && pending_reads >= self.options.max_pending_reads { diff --git a/crates/cellar/durability/raft/seed_transport.rs b/crates/cellar/durability/raft/seed_transport.rs index 4721a84a7..b6a7c4cf0 100644 --- a/crates/cellar/durability/raft/seed_transport.rs +++ b/crates/cellar/durability/raft/seed_transport.rs @@ -69,7 +69,6 @@ impl SourceSeed { } use super::snapshots::{read_bulk_head, Wire}; -use crate::application::consensus::RaftCells; use sha2::{Digest, Sha256}; use std::sync::Arc; @@ -576,117 +575,6 @@ impl State { .map_err(|_| TransportError::Closed)?; crate::asyncrt::select! {_=timer=>Err(TransportError::Timeout),result=self.bounded(None,future)=>result} } - - pub(super) async fn receive_seed( - &self, - stream: &mut S, - exporter: &[u8; 32], - prefix: [u8; frame::PREFIX], - ) -> Result<(), TransportError> { - let metadata = Arc::new( - self.seed_metadata - .acquire(META_BYTES) - .ok_or(TransportError::Overloaded)?, - ); - let head = self - .bounded( - None, - read_bulk_head(stream, Some(prefix), REQUEST_MAGIC, 0, &self.options), - ) - .await?; - let wire = Wire { - auth: &self.auth, - exporter, - options: &self.options, - max_body: SNAPSHOT_META_BYTES, - }; - let request: SeedRequest = wire.decode(REQUEST_MAGIC, REQUEST_PURPOSE, &head, None)?; - if frame::decimal(&request.nonce)? == 0 - || request.timeout_ms == 0 - || request.timeout_ms > 3_600_000 - { - return Err(TransportError::Protocol); - } - self.authorize_route(&request.route)?; - let (joining, source) = request.route.decode()?; - let group = self - .runtime - .log() - .group(&source.scope) - .map_err(|_| TransportError::UnknownGroup)?; - if !group - .read() - .map_err(|e| TransportError::Runtime(e.into()))? - .conf_state - .learners - .contains(&joining.member.get()) - { - return Err(TransportError::Refused); - } - let handle = self - .query_gateway - .lock() - .map_err(|_| TransportError::Closed)? - .as_ref() - .and_then(|h| h.upgrade()) - .ok_or(TransportError::Refused)?; - let gateway = RaftCells { handle }; - let cell = gateway - .cell(source.scope.namespace, &source.scope.cell) - .map_err(|_| TransportError::Refused)?; - if cell.scope() != &source.scope - || cell.binding.class.schema_digest() != request.route.schema() - { - return Err(TransportError::Identity); - } - let deadline = self - .domain - .mono_ms() - .map_err(|_| TransportError::Closed)? - .saturating_add(request.timeout_ms); - let seed = cell.prepare_seed( - joining, - request.kind, - metadata.clone(), - std::time::Duration::from_millis(request.timeout_ms), - ); - let mut extra = [0; 1]; - let result=self.seed_until(deadline,async { - crate::asyncrt::select! {result=seed=>Ok(result),read=stream.read(&mut extra)=>{let _=read?;Err(TransportError::Closed)}} - }).await?; - self.authorize_route(&request.route)?; - let (status, barrier, lease, body) = match result { - Ok(seed) if seed.kind() == request.kind => { - (0, seed.barrier.get(), seed.lease(), seed.body()?) - } - Ok(_) => return Err(TransportError::Protocol), - Err(crate::application::consensus::CommandFailure::Overloaded) => (1, 0, 0, vec![]), - Err(_) => (2, 0, 0, vec![]), - }; - let reply = SeedReply { - route: request.route.reversed(), - request: head.tag, - nonce: request.nonce, - status, - kind: request.kind, - barrier: barrier.to_string(), - lease: lease.to_string(), - digest: Sha256::digest(&body).into(), - }; - let response = wire.encode(RESPONSE_MAGIC, RESPONSE_PURPOSE, &head.tag, &reply, &body)?; - let result = self - .seed_io(deadline, async { - write_head(stream, &response).await?; - stream.write_all(&body).await?; - stream.flush().await?; - Ok(()) - }) - .await; - if result.is_err() && lease != 0 { - self.finish_seed_source(lease); - } - result - } } #[path = "seed_archive.rs"] diff --git a/crates/cellar/durability/raft/snapshot_seed.rs b/crates/cellar/durability/raft/snapshot_seed.rs index 07b20f941..47a7333ad 100644 --- a/crates/cellar/durability/raft/snapshot_seed.rs +++ b/crates/cellar/durability/raft/snapshot_seed.rs @@ -288,11 +288,18 @@ impl PreparedSeed { ..Default::default() }; Ok(crate::raft::cache::CachedGroup { + archive_required: false, hard_state, conf_state: snapshot.get_metadata().get_conf_state().clone(), membership: self.descriptor.membership.clone(), snapshot, entries: std::collections::BTreeMap::new(), + group_manifest: None, + recovery_genesis: None, + manifest_pins: None, + checkpoint_manifest: None, + group_seed: None, + local_park: None, applied: crate::raft::cache::DurableApply { position: LogPosition { index: LogIndex::new(self.descriptor.index), diff --git a/crates/cellar/durability/raft/snapshot_transport.rs b/crates/cellar/durability/raft/snapshot_transport.rs index 79a6a3272..5d6401bef 100644 --- a/crates/cellar/durability/raft/snapshot_transport.rs +++ b/crates/cellar/durability/raft/snapshot_transport.rs @@ -568,7 +568,7 @@ impl State { Ok((route, peer, pool)) } - async fn snapshot_io( + pub(super) async fn snapshot_io( &self, pool: &Pool, deadline: u64, diff --git a/crates/cellar/durability/raft/storage.rs b/crates/cellar/durability/raft/storage.rs index de2316034..e73a5c888 100644 --- a/crates/cellar/durability/raft/storage.rs +++ b/crates/cellar/durability/raft/storage.rs @@ -2,7 +2,13 @@ //! One domain-owned engine worker. Every successful mutation has crossed fsync. -use super::cache::{genesis, validate_conf, CachedGroup, GroupLog, GroupWrite, SnapshotChange}; +use super::cache::{ + genesis, validate_conf, CachedGroup, GroupLog, GroupWrite, SnapshotChange, StoredGroup, +}; +use super::cell_groups::{ + reclaim::ReclaimCells, CellGroupApplier, CellGroupReclaimError, CellGroupReclaimLimits, + CellGroupReclaimReport, PreparedGroupManifest, +}; use super::filesystem::EngineFileSystem; use super::fleet_stop::{Guard, LifecycleRecord, LifecycleUpdate}; use super::options::{LogOpenMode, ReplicaLogOptions}; @@ -13,7 +19,10 @@ use super::records::{ self, GroupRecord, MappingRecord, NodeRecord, RootRecord, GROUP_KEY, ROOT_KEY, }; use ::raft::prelude::{ConfState, Entry}; -use cellar_logic::consensus::{GroupScope, MemberId, OperationId, ReplicaIdentity}; +use cellar_logic::cell_group::{GroupHistory, GroupRoute}; +use cellar_logic::consensus::{ + EngineInstanceId, GroupScope, MemberId, OperationId, ReplicaIdentity, +}; use cellar_ltx::{FileSystem, HostFileLock}; use protobuf::Message; use raft_engine::{Command, Engine, LogBatch, MessageExt, ReadableSize, RecoveryMode}; @@ -24,6 +33,10 @@ use std::sync::{mpsc, Arc, Mutex, RwLock}; use std::time::Duration; use tokio::sync::{oneshot, Notify}; +#[cfg(test)] +#[path = "storage_cell_genesis_tests.rs"] +mod cell_genesis_tests; + #[cfg(test)] #[path = "storage_genesis_tests.rs"] mod genesis_tests; @@ -36,6 +49,7 @@ const LIFECYCLE_KEY: &[u8] = b"cellar-node-lifecycle"; #[derive(Clone)] struct EngineCaches { + log_retention: Arc, usage: Arc, lifecycle: Arc>, } @@ -96,30 +110,71 @@ impl GroupBootstrap { self.scope .validate() .map_err(|error| LogError::Invalid(error.to_string()))?; - let mut conf = ConfState::default(); - conf.set_voters(self.voters.iter().map(|id| id.get()).collect()); - conf.set_learners(self.learners.iter().map(|id| id.get()).collect()); - validate_conf(&conf).map_err(|error| LogError::Invalid(error.to_string()))?; - super::member_state::GroupMembership::initial(&conf) - .validate(&conf, 1) + initial_configuration(self.member, &self.voters, &self.learners) + } +} + +/// Physical genesis for a whole cell range. Construction supplies no membership +/// claim or serving authority; those require the catalog bootstrap protocol. +#[derive(Clone, Debug)] +pub struct CellGroupBootstrap { + pub route: GroupRoute, + pub member: MemberId, + pub voters: Vec, + pub learners: Vec, +} + +impl CellGroupBootstrap { + fn configuration(&self) -> Result { + super::cell_groups::GroupManifest::genesis(self.route.clone()) .map_err(|error| LogError::Invalid(error.to_string()))?; - if !conf.voters.contains(&self.member.get()) && !conf.learners.contains(&self.member.get()) - { - return Err(LogError::WrongIdentity); - } - Ok(conf) + initial_configuration(self.member, &self.voters, &self.learners) + } +} + +fn initial_configuration( + member: MemberId, + voters: &[MemberId], + learners: &[MemberId], +) -> Result { + let mut conf = ConfState::default(); + conf.set_voters(voters.iter().map(|id| id.get()).collect()); + conf.set_learners(learners.iter().map(|id| id.get()).collect()); + validate_conf(&conf).map_err(|error| LogError::Invalid(error.to_string()))?; + super::member_state::GroupMembership::initial(&conf) + .validate(&conf, 1) + .map_err(|error| LogError::Invalid(error.to_string()))?; + if !conf.voters.contains(&member.get()) && !conf.learners.contains(&member.get()) { + return Err(LogError::WrongIdentity); } + Ok(conf) } /// Identity-bound observation of one successfully synced batch. #[derive(Clone, Debug)] -pub struct SyncReceipt { +pub struct SyncReceipt { pub operation: OperationId, pub sequence: u64, - pub groups: Vec, + pub groups: Vec>, +} + +type Catalog = Arc>>; + +fn release_closed_pins(groups: &Catalog) -> Result<(), LogError> { + let (groups, mut result) = match groups.read() { + Ok(groups) => (groups, Ok(())), + Err(error) => ( + error.into_inner(), + Err(LogError::Corrupt("closed log catalog poisoned".into())), + ), + }; + for group in groups.values() { + let released = group.release_closed_pins(); + result = result.and(released); + } + result } -type Catalog = Arc>>; type Reply = oneshot::Sender>; #[derive(Clone)] @@ -128,6 +183,7 @@ pub struct ReplicaLog { } struct LogHandle { + log_retention: Arc, snapshot_scratch: cellar_ltx::stream_snapshot::ScratchPool, performance: Arc, instance: std::sync::OnceLock, @@ -141,6 +197,7 @@ struct LogHandle { options: ReplicaLogOptions, closing: Arc, runtime_claimed: AtomicBool, + sql_claimed: AtomicBool, abort: Arc, closed: Arc, terminal: Arc, @@ -204,6 +261,38 @@ struct Request { _permit: Permit, } +struct ManagedCells { + binding: super::member_state::FleetGroupBinding, + allow_genesis: bool, + boot: super::fleet_stop::BootBinding, +} + +enum ImportedGenesis { + Recovery(Box), + Transfer(Box), +} + +impl ImportedGenesis { + fn binding(&self) -> (&CellGroupBootstrap, &super::cell_groups::GroupManifest) { + match self { + Self::Recovery(seed) => (seed.bootstrap(), seed.genesis()), + Self::Transfer(seed) => (seed.bootstrap(), seed.genesis()), + } + } + + fn check( + &self, + owner: &Arc<()>, + identity: &ReplicaIdentity, + engine: EngineInstanceId, + ) -> Result<(), LogError> { + match self { + Self::Recovery(seed) => seed.check(owner, identity, engine), + Self::Transfer(seed) => seed.check(owner, identity, engine), + } + } +} + enum Work { NodeFence(Reply), Lifecycle(Box, Reply), @@ -212,14 +301,55 @@ enum Work { Option>, Reply, ), + CreateCells( + CellGroupBootstrap, + Option>, + Option>, + Reply>, + ), + ImportCells(ImportedGenesis, Reply>), Seeded(Box, Reply), Persist(OperationId, Vec, Reply), + PersistCells( + OperationId, + Vec>, + Reply>, + ), + ReclaimCells( + Box, + MaintenanceLease, + Reply>, + ), Purge(Reply<()>), } /// Exclusive consensus ownership; normal release follows a final storage fence. pub(crate) struct RuntimeLease(Arc); +/// Held by the SQL owner, physical workers and retained response admission. +pub(crate) struct SqlPoolLease(Arc); + +// Maintenance performs no consensus writes of its own. It may refund an +// unqueued reservation, but an entered operation retains both claims to exit. +struct MaintenanceLease { + runtime: Option, + _sql: Arc, +} + +impl Drop for MaintenanceLease { + fn drop(&mut self) { + if let Some(runtime) = self.runtime.take() { + runtime.release(); + } + } +} + +impl Drop for SqlPoolLease { + fn drop(&mut self) { + self.0.sql_claimed.store(false, Ordering::Release); + } +} + impl RuntimeLease { /// Dropping an unfenced worker deliberately leaves the claim poisoned. pub(crate) fn release(self) { @@ -228,6 +358,15 @@ impl RuntimeLease { } impl ReplicaLog { + /// Advisory node-wide logical retained-entry debt and reservation use. + /// This is not physical file accounting or an admission capability. + pub fn cell_group_log_retention_usage( + &self, + ) -> Result { + self.admit()?; + self.inner.log_retention.usage() + } + pub(crate) fn performance(&self) -> Arc { self.inner.performance.clone() } @@ -278,6 +417,15 @@ impl ReplicaLog { Ok(RuntimeLease(self.inner.clone())) } + pub(crate) fn claim_sql_pool(&self) -> Result, LogError> { + self.admit()?; + self.inner + .sql_claimed + .compare_exchange(false, true, Ordering::AcqRel, Ordering::Acquire) + .map_err(|_| LogError::AlreadyExists)?; + Ok(Arc::new(SqlPoolLease(self.inner.clone()))) + } + pub(crate) fn domain(&self) -> crate::asyncrt::DomainToken { self.inner.domain.clone() } @@ -328,6 +476,10 @@ impl ReplicaLog { let (ready, wait) = oneshot::channel(); let groups = Arc::new(RwLock::new(BTreeMap::new())); let closed = Arc::new(AtomicBool::new(false)); + let log_retention = super::cell_groups::log_retention::NodeLogRetention::new( + options.cell_group_retention, + closed.clone(), + ); let abort = Arc::new(AtomicBool::new(false)); let terminal = Arc::new(Terminal { result: Mutex::new(None), @@ -339,6 +491,7 @@ impl ReplicaLog { let lifecycle_cache = Arc::new(Mutex::new(LifecycleRecord::empty())); let handle = Self { inner: Arc::new(LogHandle { + log_retention: log_retention.clone(), snapshot_scratch, performance: performance.clone(), instance: std::sync::OnceLock::new(), @@ -352,6 +505,7 @@ impl ReplicaLog { options: options.clone(), closing: closing.clone(), runtime_claimed: AtomicBool::new(false), + sql_claimed: AtomicBool::new(false), abort: abort.clone(), closed: closed.clone(), terminal: terminal.clone(), @@ -362,11 +516,12 @@ impl ReplicaLog { // if the open waiter or public handle is dropped. let _worker = domain .blocking(move || { - struct ClosedOnDrop(Arc, Arc); + struct ClosedOnDrop(Arc, Arc, Catalog); impl Drop for ClosedOnDrop { fn drop(&mut self) { self.0.store(true, Ordering::Release); + let _ = release_closed_pins(&self.2); // Worker/resources unwind before this outer guard. A // panic may close admission early, but cannot fake join. let mut terminal = self @@ -381,7 +536,7 @@ impl ReplicaLog { self.1.changed.notify_waiters(); } } - let _guard = ClosedOnDrop(closed.clone(), terminal.clone()); + let guard = ClosedOnDrop(closed.clone(), terminal.clone(), groups.clone()); let result = Worker::open( options, files, @@ -389,6 +544,7 @@ impl ReplicaLog { owner, closed.clone(), EngineCaches { + log_retention, usage: cache_usage, lifecycle: lifecycle_cache, }, @@ -413,6 +569,9 @@ impl ReplicaLog { // Engine drop joins its metrics thread; directory locks are owned // by this same worker and cannot outlive its completion. drop(worker); + // Closed public metadata handles may outlive this worker. They + // must not keep image/page-directory locks through cache pins. + let result = result.and(release_closed_pins(&guard.2)); *terminal .result .lock() @@ -442,24 +601,204 @@ impl ReplicaLog { ) -> Result { self.admit()?; let groups = self.inner.groups.read().map_err(|_| LogError::Closed)?; + let additional = match pending { + Some(scope) => { + scope.validate().map_err(|_| LogError::WrongIdentity)?; + match groups.get(&GroupHistory::from(scope)) { + Some(group) => { + if &group.as_cell()?.identity.scope != scope { + return Err(LogError::WrongIdentity); + } + false + } + None => true, + } + } + None => false, + }; groups .len() - .checked_add(usize::from( - pending.is_some_and(|scope| !groups.contains_key(scope)), - )) + .checked_add(usize::from(additional)) .ok_or(LogError::Overloaded) } pub fn groups(&self) -> Result, LogError> { self.admit()?; - Ok(self - .inner + self.inner + .groups + .read() + .map_err(|_| LogError::Closed)? + .values() + .map(|group| group.as_cell().cloned()) + .collect() + } + + pub fn cell_groups(&self) -> Result>, LogError> { + self.admit()?; + self.inner .groups .read() .map_err(|_| LogError::Closed)? .values() + .map(|group| group.as_cells().cloned()) + .collect() + } + + /// Reconcile every physical group in this engine, then collect a bounded + /// batch of unselected image/page files. Supply one exact applier per group, + /// including unattached/retired groups, and every additional durable archive + /// or transition root. Image stores must belong exclusively to this node + /// engine. Live reader/transfer pins remain protected independently. + /// + /// Runtime and SQL ownership must have joined shutdown before maintenance; + /// the engine reserves both claims until verification/deletion finishes. + /// A busy applier refuses instead of waiting. Cancellation retains work and admission; + /// log.close() joins physical completion. This supplies no serving, archive + /// coverage, log truncation or voting authority. + pub async fn reclaim_cell_storage( + &self, + appliers: Vec, + retained: Vec, + limits: CellGroupReclaimLimits, + ) -> Result { + let work = ReclaimCells::new(appliers, retained, limits)?; + let bytes = work.reservation_bytes()?; + let runtime = self.claim_runtime().map_err(|e| match e { + LogError::AlreadyExists => CellGroupReclaimError::Busy, + e => CellGroupReclaimError::Log(e), + })?; + let sql = match self.claim_sql_pool() { + Ok(sql) => sql, + Err(error) => { + runtime.release(); + return Err(match error { + LogError::AlreadyExists => CellGroupReclaimError::Busy, + e => CellGroupReclaimError::Log(e), + }); + } + }; + let lease = MaintenanceLease { + runtime: Some(runtime), + _sql: sql, + }; + self.request(bytes, |reply| { + Work::ReclaimCells(Box::new(work), lease, reply) + }) + .await + .map_err(CellGroupReclaimError::Log)? + } + + pub fn cell_group(&self, history: &GroupHistory) -> Result, LogError> { + self.admit()?; + self.inner + .groups + .read() + .map_err(|_| LogError::Closed)? + .get(history) + .ok_or(LogError::NotFound)? + .as_cells() .cloned() - .collect()) + } + + pub async fn create_cell_group( + &self, + bootstrap: CellGroupBootstrap, + ) -> Result, LogError> { + bootstrap.configuration()?; + self.request(16 * 1024, |reply| { + Work::CreateCells(bootstrap, None, None, reply) + }) + .await + } + + pub(crate) async fn stage_cell_group( + &self, + bootstrap: CellGroupBootstrap, + grant: Arc, + ) -> Result, LogError> { + grant.check_log(self)?; + self.request(96 * 1024, |reply| { + Work::CreateCells(bootstrap, Some(grant), None, reply) + }) + .await + } + + /// The catalog-selected initial membership and fleet guard are persisted + /// in the same first engine sync. An existing unguarded history refuses. + pub(crate) async fn create_managed_cell_group( + &self, + bootstrap: CellGroupBootstrap, + authorization: &super::fleet_catalog::FleetGroupAuthorization, + boot: &super::fleet_stop::BootBinding, + ) -> Result, LogError> { + let conf = bootstrap.configuration()?; + if !authorization.matches_scope(&bootstrap.route.history) + || authorization.history_digests() + != (bootstrap.route.birth_digest, bootstrap.route.schema_digest) + { + return Err(LogError::WrongIdentity); + } + let binding = authorization.binding().clone(); + boot.validate()?; + if boot.fleet != binding.fleet() + || boot.catalog != binding.catalog() + || boot.node != self.limits().node.bytes() + || boot.disk != self.limits().disk.bytes() + || boot.engine != self.storage_instance()?.bytes() + || boot.process != self.limits().process.bytes() + || self + .lifecycle_guard()? + .is_none_or(|guard| !guard.admits_group_start(boot)) + { + return Err(LogError::WrongIdentity); + } + super::member_state::GroupMembership::managed_initial(&conf, binding.clone())?; + self.request(24 * 1024, |reply| { + Work::CreateCells( + bootstrap, + None, + Some(Box::new(ManagedCells { + binding, + allow_genesis: authorization.allows_managed_genesis(), + boot: boot.clone(), + })), + reply, + ) + }) + .await + } + + pub(crate) async fn install_recovery_seed( + &self, + seed: super::cell_groups::RecoverySeed, + ) -> Result, LogError> { + self.install_imported_seed(ImportedGenesis::Recovery(Box::new(seed))) + .await + } + + pub(crate) async fn install_transfer_seed( + &self, + seed: super::cell_groups::TransferSeed, + ) -> Result, LogError> { + self.install_imported_seed(ImportedGenesis::Transfer(Box::new(seed))) + .await + } + + async fn install_imported_seed( + &self, + seed: ImportedGenesis, + ) -> Result, LogError> { + let (bootstrap, _) = seed.binding(); + let identity = ReplicaIdentity { + scope: bootstrap.route.history, + member: bootstrap.member, + node: self.limits().node, + disk: self.limits().disk, + process: self.limits().process, + }; + seed.check(&self.owner(), &identity, self.storage_instance()?)?; + self.request(96 * 1024, |reply| Work::ImportCells(seed, reply)) + .await } /// Read occupied identities and locally durable completed membership under @@ -474,15 +813,30 @@ impl ReplicaLog { super::physical_inventory::capture(&groups, pending) } + /// Complete physical occupancy for a cell-group engine, including retained + /// histories and an optional initializing group. This is not serving or + /// membership authority. Legacy application engines refuse this accessor. + pub fn cell_group_inventory( + &self, + pending: Option<&GroupHistory>, + ) -> Result { + self.admit()?; + let groups = self.inner.groups.read().map_err(|_| LogError::Closed)?; + super::physical_inventory::capture_cells(&groups, pending) + } + pub fn group(&self, scope: &GroupScope) -> Result { self.admit()?; - self.inner - .groups - .read() - .map_err(|_| LogError::Closed)? - .get(scope) - .cloned() - .ok_or(LogError::NotFound) + scope.validate().map_err(|_| LogError::WrongIdentity)?; + let groups = self.inner.groups.read().map_err(|_| LogError::Closed)?; + let group = groups + .get(&GroupHistory::from(scope)) + .ok_or(LogError::NotFound)? + .as_cell()?; + if &group.identity.scope != scope { + return Err(LogError::WrongIdentity); + } + Ok(group.clone()) } pub async fn create_group(&self, bootstrap: GroupBootstrap) -> Result { @@ -518,16 +872,58 @@ impl ReplicaLog { .await } + pub async fn persist_cells( + &self, + operation: OperationId, + writes: Vec>, + ) -> Result, LogError> { + let bytes = self.write_bytes(operation, &writes)?; + self.request(bytes, |reply| Work::PersistCells(operation, writes, reply)) + .await + } + + /// Enable a local irreversible retention requirement before archive I/O. + /// The fixed nonzero operation correlates only this private idempotent + /// metadata write. Its receipt is never a business or durability response; + /// the storage worker assigns the actual durable sync sequence. + pub(crate) async fn require_cell_group_archive( + &self, + history: &GroupHistory, + activation: Arc, + ) -> Result<(), LogError> { + let group = self.cell_group(history)?; + activation.drained(&group.archive_compaction)?; + let required = group.read()?.archive_required; + if !required { + let mut write = GroupWrite::new(&group); + write.require_archive = Some(activation); + self.persist_cells(OperationId::new(1), vec![write]).await?; + } else { + activation.published(&group.archive_compaction)?; + } + Ok(()) + } + pub async fn persist( &self, operation: OperationId, writes: Vec, ) -> Result { + let bytes = self.write_bytes(operation, &writes)?; + self.request(bytes, |reply| Work::Persist(operation, writes, reply)) + .await + } + + fn write_bytes( + &self, + operation: OperationId, + writes: &[GroupWrite], + ) -> Result { if operation.get() == 0 { return Err(LogError::Invalid("zero persistence operation".into())); } let mut bytes = 1024_usize; - for write in &writes { + for write in writes { bytes = bytes .checked_add(write.estimated_bytes()?) .ok_or(LogError::Overloaded)?; @@ -535,8 +931,7 @@ impl ReplicaLog { if bytes > self.inner.options.max_batch_bytes { return Err(LogError::Overloaded); } - self.request(bytes, |reply| Work::Persist(operation, writes, reply)) - .await + Ok(bytes) } /// Reclaim obsolete shared engine files, retaining every group's live log, @@ -608,6 +1003,7 @@ impl MessageExt for EntryCodec { } struct Worker { + log_retention: Arc, filesystem: Arc, snapshot_scratch: cellar_ltx::stream_snapshot::ScratchPool, performance: Arc, @@ -767,6 +1163,7 @@ impl Worker { } *caches.lifecycle.lock().map_err(|_| LogError::Closed)? = lifecycle.clone(); let mut worker = Self { + log_retention: caches.log_retention, filesystem: fs.clone(), snapshot_scratch, performance, @@ -822,7 +1219,14 @@ impl Worker { return Err(error); } let mut known = BTreeSet::from([0]); + let mut application = None; for mapping in mappings { + if application + .replace(mapping.is_cells()) + .is_some_and(|kind| kind != mapping.is_cells()) + { + return Err(LogError::Corrupt("mixed engine application domains".into())); + } if mapping.key == 0 || mapping.key >= self.root.next_key || !known.insert(mapping.key) { return Err(LogError::Corrupt( "reused or unallocated numeric group key".into(), @@ -877,7 +1281,7 @@ impl Worker { .groups .write() .map_err(|_| LogError::Closed)? - .insert(handle.identity.scope.clone(), handle) + .insert(handle.history(), handle) .is_some() { return Err(LogError::Corrupt("duplicate logical group mapping".into())); @@ -895,7 +1299,7 @@ impl Worker { Ok(()) } - fn handle(&self, mapping: &MappingRecord, cache: CachedGroup) -> Result { + fn handle(&self, mapping: &MappingRecord, cache: CachedGroup) -> Result { let initial = ConfState::parse_from_bytes(&mapping.initial_conf).map_err(records::corrupt)?; validate_conf(&initial)?; @@ -910,10 +1314,45 @@ impl Worker { { return Err(LogError::Corrupt("member binding birth mismatch".into())); } - let scope = mapping.scope.clone().decode().map_err(records::corrupt)?; - scope.validate().map_err(records::corrupt)?; - cache.membership.validate_scope(&scope)?; + mapping.validate_cache(&cache)?; + if mapping.is_cells() { + Ok(StoredGroup::Cells(self.scoped_handle( + mapping, + cache, + mapping.history()?, + )?)) + } else { + Ok(StoredGroup::Cell(self.scoped_handle( + mapping, + cache, + mapping.application_scope()?, + )?)) + } + } + + fn scoped_handle( + &self, + mapping: &MappingRecord, + mut cache: CachedGroup, + scope: S, + ) -> Result, LogError> { + cache.recovery_genesis = mapping.recovery_genesis().cloned().map(Arc::new); + let log_retention = if mapping.is_cells() { + Some( + self.log_retention + .register(mapping.key, cache.entries.values())?, + ) + } else { + None + }; Ok(GroupLog { + archive_compaction: Arc::new( + super::cell_groups::archive_compaction::ArchiveCompaction::new( + cache.archive_required, + log_retention, + ), + ), + consensus_claimed: Arc::new(AtomicBool::new(false)), scratch_directory: self.options.directory.clone(), snapshot_scratch: self.snapshot_scratch.clone(), filesystem: self.filesystem.clone(), @@ -990,13 +1429,15 @@ impl Worker { .groups .read() .map_err(|_| LogError::Closed)? - .get(&bootstrap.scope) + .get(&GroupHistory::from(&bootstrap.scope)) { + let group = group.as_cell()?; let mapping = self .mappings .get(&group.key) .ok_or(LogError::LostStableState)?; - if mapping.member == bootstrap.member.get() + if group.identity.scope == bootstrap.scope + && mapping.member == bootstrap.member.get() && mapping.birth_digest == bootstrap.birth_digest && mapping.initial_conf == conf.write_to_bytes().map_err(records::corrupt)? { @@ -1039,13 +1480,17 @@ impl Worker { if !seed.authorized(&self.owner, &identity) { return Err(LogError::WrongIdentity); } - if self + if let Some(group) = self .groups .read() .map_err(|_| LogError::Closed)? - .contains_key(&identity.scope) + .get(&GroupHistory::from(&identity.scope)) { - return Err(LogError::AlreadyExists); + return Err(if group.as_cell()?.identity.scope == identity.scope { + LogError::AlreadyExists + } else { + LogError::WrongIdentity + }); } let bootstrap = seed.bootstrap()?; let conf = seed.initial().clone(); @@ -1060,22 +1505,202 @@ impl Worker { cache: CachedGroup, seed: Option<&super::snapshot::PreparedSeed>, ) -> Result { - if self.mappings.len() >= self.options.max_groups { - return Err(LogError::Overloaded); + let mapping = MappingRecord::new(self.root.next_key, &bootstrap, &conf)?; + self.insert_mapping(mapping, cache, seed, None)? + .as_cell() + .cloned() + } + + fn create_cells( + &mut self, + bootstrap: CellGroupBootstrap, + seed: Option<&super::transport::CellGroupSeedGrant>, + managed: Option, + ) -> Result, LogError> { + if seed.is_some() && managed.is_some() { + return Err(LogError::WrongIdentity); } - // A group identifier cannot be reused for a different logical history. - if self + if managed.as_ref().is_some_and(|managed| { + self.lifecycle + .guard + .as_ref() + .is_none_or(|guard| !guard.admits_group_start(&managed.boot)) + }) { + return Err(LogError::WrongIdentity); + } + let conf = if let Some(seed) = seed { + seed.check( + &self.owner, + bootstrap.route.history, + &ReplicaIdentity { + scope: bootstrap.route.history, + node: self.options.node, + disk: self.options.disk, + process: self.options.process, + member: bootstrap.member, + }, + )?; + seed.check_lifecycle(self.lifecycle.guard.as_ref())?; + seed.initial_configuration(&bootstrap)? + } else { + bootstrap.configuration()? + }; + if let Some(group) = self .groups .read() .map_err(|_| LogError::Closed)? - .keys() - .any(|scope| scope.group == bootstrap.scope.group) + .get(&bootstrap.route.history) { - return Err(LogError::WrongIdentity); + let group = group.as_cells()?; + let expected = MappingRecord::new_cells(group.key, &bootstrap, &conf)?; + if self.mappings.get(&group.key) != Some(&expected) { + return Err(LogError::WrongIdentity); + } + let cache = group.read()?; + let binding = managed + .as_ref() + .map(|m| &m.binding) + .or_else(|| seed.and_then(|grant| grant.managed_binding())); + let staged = seed + .map(|seed| seed.initial_membership(&conf)) + .transpose()?; + match (binding, cache.membership.fleet_status()) { + (Some(binding), Some(status)) if binding == &status.binding => { + if seed.is_none() + && (!super::member_state::members(&cache.conf_state) + .contains(&bootstrap.member.get()) + || !cache.membership.permits_group(group.identity())) + { + return Err(LogError::WrongIdentity); + } + } + (Some(_), None) + if staged.as_ref() == Some(&cache.membership) + && cache.group_seed.is_none() + && cache.applied.position.index.get() == 1 + && cache.group_manifest.as_ref().is_some_and(|root| { + root.root().cells() == 0 && root.position().index.get() == 1 + }) => {} + (None, None) => (), + _ => return Err(LogError::WrongIdentity), + } + return Ok(group.clone()); + } + let mapping = MappingRecord::new_cells(self.root.next_key, &bootstrap, &conf)?; + let mut cache = genesis(conf.clone()); + if let Some(managed) = managed { + if !managed.allow_genesis { + return Err(LogError::WrongIdentity); + } + let binding = managed.binding; + if !binding.matches_scope(&bootstrap.route.history) { + return Err(LogError::WrongIdentity); + } + cache.membership = + super::member_state::GroupMembership::managed_initial(&conf, binding)?; + } + if let Some(seed) = seed { + // Preserve the source's real genesis/enrollment origin. The learner + // remains outside this birth configuration until authenticated install. + cache.membership = seed.initial_membership(&conf)?; + } + cache.group_manifest = Some( + super::cell_groups::GroupManifest::genesis(bootstrap.route) + .map_err(|error| LogError::Invalid(error.to_string()))?, + ); + self.insert_mapping(mapping, cache, None, None)? + .as_cells() + .cloned() + } + + fn import_cells( + &mut self, + seed: &ImportedGenesis, + domain: &crate::asyncrt::DomainToken, + ) -> Result, LogError> { + let (bootstrap, marker) = seed.binding(); + seed.check( + &self.owner, + &ReplicaIdentity { + scope: bootstrap.route.history, + member: bootstrap.member, + node: self.options.node, + disk: self.options.disk, + process: self.options.process, + }, + EngineInstanceId::new(self.root.identity.storage_instance).map_err(records::corrupt)?, + )?; + let conf = bootstrap.configuration()?; + let guard = self.lifecycle.guard.as_ref(); + let now = domain.mono_ms().map_err(|_| LogError::Closed)?; + let managed = match seed { + ImportedGenesis::Recovery(seed) => seed.managed_binding(guard, now)?, + ImportedGenesis::Transfer(seed) => seed.managed_binding(guard, now)?, + } + .map(|binding| { + super::member_state::GroupMembership::managed_initial(&conf, binding.clone()) + }) + .transpose()?; + if let Some(group) = self + .groups + .read() + .map_err(|_| LogError::Closed)? + .get(&bootstrap.route.history) + { + let group = group.as_cells()?; + let expected = MappingRecord::new_imported_cells(group.key, bootstrap, &conf, marker)?; + let cache = group.read()?; + if self.mappings.get(&group.key) != Some(&expected) + || cache.group_manifest.as_ref() != Some(marker) + || managed + .as_ref() + .is_some_and(|membership| membership != &cache.membership) + { + return Err(LogError::WrongIdentity); + } + return Ok(group.clone()); + } + let mapping = + MappingRecord::new_imported_cells(self.root.next_key, bootstrap, &conf, marker)?; + let mut cache = genesis(conf); + if let Some(membership) = managed { + cache.membership = membership; + } + cache.group_manifest = Some(marker.clone()); + if marker.is_recovery_genesis() { + cache.recovery_genesis = Some(Arc::new(marker.clone())); + } + self.insert_mapping(mapping, cache, None, Some(seed))? + .as_cells() + .cloned() + } + + fn insert_mapping( + &mut self, + mapping: MappingRecord, + mut cache: CachedGroup, + seed: Option<&super::snapshot::PreparedSeed>, + imported: Option<&ImportedGenesis>, + ) -> Result { + if self.mappings.len() >= self.options.max_groups { + return Err(LogError::Overloaded); + } + let history = mapping.history()?; + { + let groups = self.groups.read().map_err(|_| LogError::Closed)?; + if groups.keys().any(|h| h.group == history.group) + || groups + .values() + .any(|group| group.is_cells() != mapping.is_cells()) + { + return Err(LogError::WrongIdentity); + } } let key = self.root.next_key; + if mapping.key != key { + return Err(LogError::WrongIdentity); + } let next_key = key.checked_add(1).ok_or(LogError::Overloaded)?; - let mapping = MappingRecord::new(key, &bootstrap, &conf)?; cache.validate(self.options.max_entries_per_group)?; let bytes = cache.bytes(); if self @@ -1085,6 +1710,18 @@ impl Worker { { return Err(LogError::Overloaded); } + if seed.is_some() && imported.is_some() { + return Err(LogError::WrongIdentity); + } + let prepared = match imported { + Some(ImportedGenesis::Recovery(seed)) => Some(seed.prepare(&self.closed)?), + Some(ImportedGenesis::Transfer(seed)) => Some(seed.prepare(&self.closed)?), + None => None, + }; + let mut publication = prepared.map(|(root, publication)| { + cache.manifest_pins = Some(root.images.clone()); + publication + }); let mut batch = LogBatch::default(); batch .put(0, mapping.catalog_key()?, records::encode(&mapping)?) @@ -1102,6 +1739,9 @@ impl Worker { } let previous_key = self.root.next_key; self.root.next_key = next_key; + if let Some(publication) = &mut publication { + publication.arm(); + } if let Err(error) = self.fence(&mut batch) { self.root.next_key = previous_key; return Err(error); @@ -1109,32 +1749,95 @@ impl Worker { if let Some(seed) = seed { seed.retain().map_err(|e| LogError::Io(e.to_string()))?; } + if let Some(publication) = publication { + publication + .commit() + .map_err(|error| LogError::Io(error.to_string()))?; + } let handle = self.handle(&mapping, cache)?; self.groups .write() .map_err(|_| LogError::Closed)? - .insert(bootstrap.scope, handle.clone()); + .insert(history, handle.clone()); self.mappings.insert(key, mapping); self.cache_bytes += bytes; self.cache_usage.store(self.cache_bytes, Ordering::Release); Ok(handle) } - fn persist( + fn persist( &mut self, operation: OperationId, - writes: Vec, - ) -> Result { + writes: Vec>, + ) -> Result, LogError> { let mut unique = BTreeSet::new(); let mut candidates = Vec::with_capacity(writes.len()); let mut next_bytes = self.cache_bytes; let mut batch = LogBatch::default(); + let mut archive_admissions = Vec::new(); + let mut log_admissions = Vec::new(); for write in &writes { if !Arc::ptr_eq(&write.group.owner, &self.owner) || !unique.insert(write.group.key) { return Err(LogError::WrongIdentity); } + let mapping = self + .mappings + .get(&write.group.key) + .ok_or(LogError::WrongIdentity)?; let old = write.group.read()?; - let next = old.changed(write, self.options.max_entries_per_group)?; + if write.checkpoint_admission.is_some() + && (write.require_archive.is_some() + || !matches!( + &write.snapshot, + Some(SnapshotChange::CellGroupCheckpoint(_)) + )) + { + return Err(LogError::WrongIdentity); + } + if let Some(activation) = &write.require_archive { + activation.drained(&write.group.archive_compaction)?; + } + if let Some(SnapshotChange::CellGroupCheckpoint(prepared)) = &write.snapshot { + let admission = match &write.checkpoint_admission { + Some(admission) => { + admission.validate(&write.group.archive_compaction, prepared.marker())?; + admission.clone() + } + None => write + .group + .archive_compaction + .admit(prepared.marker())? + .ok_or(LogError::Overloaded)?, + }; + archive_admissions.push(admission); + } + let next = old.changed( + write, + self.options.max_entries_per_group, + mapping.history()?, + )?; + let mut leases = Vec::new(); + if let Some(retention) = write.group.log_retention() { + let indices: BTreeSet<_> = write.entries.iter().map(|e| e.index).collect(); + if !write + .log_reservations + .keys() + .all(|index| indices.contains(index)) + { + return Err(LogError::WrongIdentity); + } + for entry in &write.entries { + if let Some(lease) = write.log_reservations.get(&entry.index) { + lease.validate(retention, entry)?; + leases.push(lease.clone()); + } else if old.entries.get(&entry.index) != Some(entry) { + leases.push(retention.reserve_entry(entry)?); + } + } + } else if !write.log_reservations.is_empty() { + return Err(LogError::WrongIdentity); + } + log_admissions.push(leases); next_bytes = next_bytes .checked_sub(old.bytes()) .and_then(|size| size.checked_add(next.bytes())) @@ -1143,15 +1846,17 @@ impl Worker { return Err(LogError::Overloaded); } drop(old); - let mapping = self - .mappings - .get(&write.group.key) - .ok_or(LogError::WrongIdentity)?; if let Some(change) = &write.snapshot { match change { - SnapshotChange::Install { .. } => { + SnapshotChange::Install { .. } | SnapshotChange::CellGroup(_) => { batch.add_command(write.group.key, Command::Clean) } + SnapshotChange::CellGroupCheckpoint(prepared) => batch.add_command( + write.group.key, + Command::Compact { + index: prepared.snapshot().get_metadata().index + 1, + }, + ), SnapshotChange::Checkpoint(snapshot) => batch.add_command( write.group.key, Command::Compact { @@ -1172,6 +1877,43 @@ impl Worker { .map_err(engine_error)?; candidates.push(next); } + // Retain both closures before any engine I/O. Failed preflight drops + // unarmed reservations; once armed, an unknown outcome keeps both roots + // and quarantines collection until recovery reconstructs the selection. + let mut publications = Vec::new(); + for write in &writes { + if let Some(prepared) = write.prepared_group_manifest() { + let previous = write.group.group_manifest()?; + publications.push( + prepared + .begin_publication( + &write.group, + previous.as_ref(), + matches!( + &write.snapshot, + Some( + SnapshotChange::CellGroup(_) + | SnapshotChange::CellGroupCheckpoint(_) + ) + ), + ) + .map_err(|e| LogError::Invalid(e.to_string()))?, + ); + } + } + for write in &writes { + if let Some(grant) = &write.seed_grant { + grant.check( + &self.owner, + self.mappings[&write.group.key].history()?, + write.group.identity(), + )?; + grant.check_lifecycle(self.lifecycle.guard.as_ref())?; + } + } + for publication in &mut publications { + publication.arm(); + } let sequence = self.fence(&mut batch)?; // No cache write lock is held during engine I/O. Readers may continue // to observe the previous durable prefix while a sync is stalled. @@ -1191,6 +1933,30 @@ impl Worker { .cache_revision .store(sequence, Ordering::Release); } + for publication in publications { + publication + .commit() + .map_err(|e| LogError::Corrupt(e.to_string()))?; + } + for (write, leases) in writes.iter().zip(&log_admissions) { + if let Some(retention) = write.group.log_retention() { + let cache = write.group.read()?; + retention + .publish(cache.entries.values(), leases) + .map_err(|error| { + LogError::Corrupt(format!("retained log publication: {error}")) + })?; + } + } + for write in &writes { + if let Some(activation) = &write.require_archive { + activation + .published(&write.group.archive_compaction) + .map_err(|error| { + LogError::Corrupt(format!("archive policy publication: {error}")) + })?; + } + } self.cache_bytes = next_bytes; self.cache_usage.store(self.cache_bytes, Ordering::Release); self.performance @@ -1302,6 +2068,52 @@ impl Worker { let _ = reply.send(result); fatal } + Work::CreateCells(bootstrap, seed, binding, reply) => { + let result = self.create_cells(bootstrap, seed.as_deref(), binding.map(|b| *b)); + let fatal = result.as_ref().err().filter(|error| error.fatal()).cloned(); + if fatal.is_some() { + self.closed.store(true, Ordering::Release); + } + let _ = reply.send(result); + fatal + } + Work::ImportCells(seed, reply) => { + let result = self.import_cells(&seed, domain); + let fatal = result.as_ref().err().filter(|error| error.fatal()).cloned(); + if fatal.is_some() { + self.closed.store(true, Ordering::Release); + } + let _ = reply.send(result); + fatal + } + Work::PersistCells(operation, writes, reply) => { + let timer = self.performance.timer(Measured::EnginePersist); + let result = self.persist(operation, writes); + timer.finish(result.is_ok()); + let fatal = result.as_ref().err().filter(|error| error.fatal()).cloned(); + if fatal.is_some() { + self.closed.store(true, Ordering::Release); + } + let _ = reply.send(result); + fatal + } + Work::ReclaimCells(work, lease, reply) => { + let result = (|| { + let groups = self + .groups + .read() + .map_err(|_| LogError::Closed)? + .values() + .map(|g| g.as_cells().cloned()) + .collect::, _>>()?; + Ok(work.run(&self.owner, groups)) + })(); + // Physical stores quarantine their own uncertain deletes; + // a refused maintenance request cannot relabel log state. + drop(lease); + let _ = reply.send(result); + None + } Work::Seeded(seed, reply) => { let result = self.seeded(*seed); let fatal = result.as_ref().err().filter(|error| error.fatal()).cloned(); diff --git a/crates/cellar/durability/raft/storage_cell_genesis_tests.rs b/crates/cellar/durability/raft/storage_cell_genesis_tests.rs new file mode 100644 index 000000000..51eb9c774 --- /dev/null +++ b/crates/cellar/durability/raft/storage_cell_genesis_tests.rs @@ -0,0 +1,802 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use crate::asyncrt::ExecutionDomain; +use ::raft::Storage as _; +use cellar_logic::{ + cell_group::{CellRange, RouteEpoch}, + consensus::*, +}; + +fn bootstrap(id: u8) -> CellGroupBootstrap { + CellGroupBootstrap { + route: GroupRoute { + history: GroupHistory { + cluster: ClusterId::new([1; 16]).unwrap(), + namespace: NamespaceId::new([2; 16]).unwrap(), + group: GroupId::new([id; 16]).unwrap(), + incarnation: IncarnationId::new([4; 16]).unwrap(), + }, + range: CellRange::all(), + epoch: RouteEpoch::new(1).unwrap(), + birth_digest: [9; 32], + schema_digest: [8; 32], + }, + member: MemberId::new(1).unwrap(), + voters: (1..=3).map(|n| MemberId::new(n).unwrap()).collect(), + learners: vec![], + } +} + +fn options(root: &std::path::Path) -> ReplicaLogOptions { + ReplicaLogOptions::new( + root.join("log"), + NodeId::new([1; 16]).unwrap(), + DiskLineage::new([2; 16]).unwrap(), + ProcessGeneration::new([3; 16]).unwrap(), + LogOpenMode::Create, + ) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn cell_group_genesis_restores_physical_handles_and_exact_birth() -> anyhow::Result<()> { + let root = tempfile::tempdir()?; + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let mut opts = options(root.path()); + let initial = bootstrap(5); + let log = domain + .run(ReplicaLog::open( + opts.clone(), + Arc::new(cellar_ltx::DirectFileSystem), + )) + .await??; + let group = log.create_cell_group(initial.clone()).await?; + let marker = group.group_manifest()?.unwrap(); + assert_eq!(marker.route(), &initial.route); + assert_eq!( + marker.position(), + LogPosition { + index: LogIndex::new(1), + term: Term::new(1) + } + ); + assert_eq!(marker.root().cells(), 0); + assert_eq!(group.identity().scope, initial.route.history); + assert!(group.applied().is_err()); + assert_eq!( + log.create_cell_group(initial.clone()).await?.numeric_key(), + group.numeric_key() + ); + for field in 0..4 { + let mut wrong = initial.clone(); + match field { + 0 => wrong.route.schema_digest = [7; 32], + 1 => wrong.route.epoch = RouteEpoch::new(2)?, + 2 => wrong.member = MemberId::new(2)?, + _ => wrong.voters.pop().map(|_| ()).unwrap(), + } + assert!(matches!( + log.create_cell_group(wrong).await, + Err(LogError::WrongIdentity) + )); + } + log.create_cell_group(bootstrap(7)).await?; + assert_eq!(log.cell_groups()?.len(), 2); + let inventory = log.cell_group_inventory(None)?; + assert_eq!(inventory.histories().len(), 2); + assert_eq!(inventory, log.physical_inventory(None)?); + let row = inventory.group_history(&initial.route.history).unwrap(); + assert_eq!(row.member(), initial.member); + assert_eq!(row.birth_digest(), initial.route.birth_digest); + assert!(row.in_configuration()); + assert_eq!(row.completed_sequence(), 0); + assert!(inventory + .group_history(&bootstrap(8).route.history) + .is_none()); + // Both a pending existing slot and a genuinely new slot must survive the + // bounded node wire representation, with the correct occupied count. + for (pending, count) in [(initial.route.history, 2), (bootstrap(8).route.history, 3)] { + let captured = log.cell_group_inventory(Some(&pending))?; + assert_eq!(captured.histories(), inventory.histories()); + let mut view = crate::raft::node_inspection_wire::tests::view(); + view.node.groups = count; + view.node.reserved_bytes = u64::from(count) * view.node.profile.bytes_per_group; + view.unattached_engine_histories = 2; + let encoded = crate::raft::physical_inventory::encode(&view, &captured)?; + let (_, decoded) = crate::raft::physical_inventory::decode(&encoded)?; + assert_eq!(*decoded, captured); + view.node.groups -= 1; + view.node.reserved_bytes -= view.node.profile.bytes_per_group; + assert!(crate::raft::physical_inventory::encode(&view, &captured).is_err()); + } + assert!(matches!(log.groups(), Err(LogError::WrongIdentity))); + let old = GroupBootstrap { + scope: GroupScope { + cluster: initial.route.history.cluster, + namespace: initial.route.history.namespace, + group: GroupId::new([10; 16])?, + incarnation: initial.route.history.incarnation, + cell: "actual-old-cell".into(), + }, + member: initial.member, + birth_digest: initial.route.birth_digest, + voters: initial.voters.clone(), + learners: vec![], + }; + assert!(matches!( + log.create_group(old).await, + Err(LogError::WrongIdentity) + )); + let mut write = GroupWrite::new(&group); + write.entries.push(Entry { + index: 2, + term: 2, + data: b"control-prefix".to_vec().into(), + ..Default::default() + }); + let mut hard = group.initial_state()?.hard_state; + hard.term = 2; + hard.commit = 2; + hard.vote = 1; + write.hard_state = Some(hard.clone()); + let receipt = log.persist_cells(OperationId::new(1), vec![write]).await?; + assert_eq!(receipt.groups, vec![group.identity().clone()]); + let instance = log.storage_instance()?; + log.close().await?; + opts.mode = LogOpenMode::Resume; + opts.process = ProcessGeneration::new([6; 16])?; + let log = domain + .run(ReplicaLog::open( + opts.clone(), + Arc::new(cellar_ltx::DirectFileSystem), + )) + .await??; + assert_eq!(log.storage_instance()?, instance); + let recovered = log.create_cell_group(initial.clone()).await?; + assert_eq!(recovered.identity().scope, initial.route.history); + assert_eq!(recovered.identity().process, opts.process); + assert_eq!(recovered.group_manifest()?, Some(marker)); + assert_eq!(recovered.initial_state()?.hard_state, hard); + assert_eq!( + recovered.entries(2, 3, None, ::raft::GetEntriesContext::empty(false))?[0] + .data + .as_ref(), + b"control-prefix" + ); + assert_eq!(log.cell_groups()?.len(), 2); + assert_eq!(log.cell_group_inventory(None)?, inventory); + log.close().await?; + domain.shutdown().await; + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn cell_group_genesis_stays_invisible_until_actual_sync() -> anyhow::Result<()> { + use crate::raft::storage_fault as fault; + let root = tempfile::tempdir()?; + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let gate = fault::Gate::new(None); + let log = domain + .run(ReplicaLog::open( + options(root.path()), + Arc::new(fault::FaultFileSystem { gate: gate.clone() }), + )) + .await??; + let initial = bootstrap(5); + let release = gate.arm(fault::HOLD_SYNC); + let writer = log.clone(); + let proposal = initial.clone(); + let create = domain.spawn(async move { writer.create_cell_group(proposal).await })?; + let held = domain + .run(crate::asyncrt::timeout( + Duration::from_secs(5), + gate.entered.notified(), + )) + .await?; + let hidden = matches!( + log.cell_group(&initial.route.history), + Err(LogError::NotFound) + ); + let count = log.group_count(); + let pending = log.cell_group_inventory(Some(&initial.route.history)); + drop(release); + let group = create.await??; + held?; + assert!(hidden); + assert_eq!(count?, 0); + let pending = pending?; + assert!(pending.histories().is_empty()); + assert_eq!( + pending.initializing(), + Some( + log.cell_group_inventory(None)? + .group_history(&initial.route.history) + .unwrap() + .scope_digest() + ) + ); + assert_eq!(group.group_manifest()?.unwrap().route(), &initial.route); + assert_eq!(log.group_count()?, 1); + log.close().await?; + domain.shutdown().await; + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn archive_requirement_is_local_irreversible_and_recovered() -> anyhow::Result<()> { + let root = tempfile::tempdir()?; + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let mut opts = options(root.path()); + let initial = bootstrap(5); + let log = domain + .run(ReplicaLog::open( + opts.clone(), + Arc::new(cellar_ltx::DirectFileSystem), + )) + .await??; + let group = log.create_cell_group(initial.clone()).await?; + let marker = group.group_manifest()?.unwrap(); + assert!(group.archive_compaction.permits(&marker)?); + let admitted = group.archive_compaction.admit(&marker)?.unwrap(); + let activation = group.archive_compaction.begin_activation()?; + assert!(matches!( + log.require_cell_group_archive(&initial.route.history, activation.clone()) + .await, + Err(LogError::Overloaded) + )); + assert!(!group.read()?.archive_required); + assert!( + group.archive_compaction.admit(&marker)?.is_none(), + "a pending archive activation must stop later checkpoint admissions" + ); + let physical = admitted.clone(); + drop(admitted); + assert!(matches!( + log.require_cell_group_archive(&initial.route.history, activation.clone()) + .await, + Err(LogError::Overloaded) + )); + drop(physical); + log.require_cell_group_archive(&initial.route.history, activation) + .await?; + assert!(group.read()?.archive_required); + assert!(!group.clone().archive_compaction.permits(&marker)?); + let record = { + let cache = group.read()?; + let mapping = MappingRecord::new_cells(group.numeric_key(), &initial, &cache.conf_state)?; + GroupRecord::from_cache(mapping, &cache)? + }; + let mut encoded = serde_json::to_value(record)?; + encoded["application"]["state"] + .as_object_mut() + .unwrap() + .remove("archive_required"); + assert!(records::decode::(&serde_json::to_vec(&encoded)?).is_err()); + log.persist_cells(OperationId::new(17), vec![GroupWrite::new(&group)]) + .await?; + assert!(group.read()?.archive_required); + log.close().await?; + opts.mode = LogOpenMode::Resume; + opts.process = ProcessGeneration::new([6; 16])?; + let resumed = domain + .run(ReplicaLog::open( + opts, + Arc::new(cellar_ltx::DirectFileSystem), + )) + .await??; + let restored = resumed.cell_group(&initial.route.history)?; + assert!(restored.read()?.archive_required); + assert!(!restored.archive_compaction.permits(&marker)?); + resumed.close().await?; + domain.shutdown().await; + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn archive_activation_drains_an_admitted_checkpoint_at_storage() -> anyhow::Result<()> { + use crate::raft::cell_groups::{ + CellGroupCheckpointLimits, CellGroupCheckpointPool, ManifestLimits, ManifestPageStore, + ManifestTree, + }; + let root = tempfile::tempdir()?; + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let log = domain + .run(ReplicaLog::open( + options(root.path()), + Arc::new(cellar_ltx::DirectFileSystem), + )) + .await??; + let initial = bootstrap(5); + let group = log.create_cell_group(initial.clone()).await?; + let genesis = group.group_manifest()?.unwrap(); + let earlier = group.archive_compaction.admit(&genesis)?.unwrap(); + let source = group.clone(); + let directory = root.path().join("pages"); + let (pages, next) = domain + .blocking(move || -> anyhow::Result<_> { + let mut pages = ManifestPageStore::open( + &directory, + Arc::new(cellar_ltx::DirectFileSystem), + 1024, + 16 * 1024 * 1024, + 16, + )?; + let tree = ManifestTree::new(genesis.route().clone(), ManifestLimits::default())?; + let prepared = tree.prepare_complete( + &source, + &mut pages, + genesis.root(), + genesis.position(), + genesis.membership_revision(), + |_| unreachable!("empty group"), + )?; + let next = tree.prepare_unchanged( + &pages, + &prepared, + LogPosition { + index: LogIndex::new(2), + term: Term::new(1), + }, + )?; + Ok((pages, next)) + })? + .await??; + let mut write = GroupWrite::new(&group); + write.entries.push(Entry { + index: 2, + term: 1, + ..Default::default() + }); + let mut hard = group.initial_state()?.hard_state; + hard.commit = 2; + write.hard_state = Some(hard); + log.persist_cells(OperationId::new(20), vec![write]).await?; + let mut write = GroupWrite::new(&group); + write.group_manifest = Some(next.clone()); + log.persist_cells(OperationId::new(21), vec![write]).await?; + let pool = CellGroupCheckpointPool::new(CellGroupCheckpointLimits { + max_sources: 1, + max_retained_bytes: 32 * 1024 * 1024, + max_cells: 64, + max_transfer_bytes: 32 * 1024 * 1024, + max_chunk_bytes: 4093, + })?; + let capture = pool.clone(); + let source = group.clone(); + let cut = next.clone(); + let checkpoint = domain + .blocking(move || capture.capture(&source, &cut)?.prepare_publication())? + .await??; + let admitted = group + .archive_compaction + .admit(checkpoint.marker())? + .unwrap(); + let activation = group.archive_compaction.begin_activation()?; + assert!(!group.archive_compaction.permits(checkpoint.marker())?); + assert!(group + .archive_compaction + .admit(checkpoint.marker())? + .is_none()); + + let mut wrong_cut = GroupWrite::new(&group); + wrong_cut.snapshot = Some(SnapshotChange::CellGroupCheckpoint(checkpoint.clone())); + wrong_cut.checkpoint_admission = Some(earlier.clone()); + assert!(matches!( + log.persist_cells(OperationId::new(22), vec![wrong_cut]) + .await, + Err(LogError::WrongIdentity) + )); + drop(earlier); + let foreign = log.create_cell_group(bootstrap(7)).await?; + let mut wrong_group = GroupWrite::new(&foreign); + wrong_group.snapshot = Some(SnapshotChange::CellGroupCheckpoint(checkpoint.clone())); + wrong_group.checkpoint_admission = Some(admitted.clone()); + assert!(matches!( + log.persist_cells(OperationId::new(23), vec![wrong_group]) + .await, + Err(LogError::WrongIdentity) + )); + let mut missing_cut = GroupWrite::new(&group); + missing_cut.checkpoint_admission = Some(admitted.clone()); + assert!(matches!( + log.persist_cells(OperationId::new(24), vec![missing_cut]) + .await, + Err(LogError::WrongIdentity) + )); + let mut unadmitted = GroupWrite::new(&group); + unadmitted.snapshot = Some(SnapshotChange::CellGroupCheckpoint(checkpoint.clone())); + assert!(matches!( + log.persist_cells(OperationId::new(25), vec![unadmitted]) + .await, + Err(LogError::Overloaded) + )); + let mut write = GroupWrite::new(&group); + write.snapshot = Some(SnapshotChange::CellGroupCheckpoint(checkpoint.clone())); + write.checkpoint_admission = Some(admitted); + // This unpolled physical request now owns the final checkpoint lease. + // Activation must wait, then storage must accept its inherited exact cut. + let saving = log.persist_cells(OperationId::new(26), vec![write]); + assert!(matches!( + log.require_cell_group_archive(&initial.route.history, activation.clone()) + .await, + Err(LogError::Overloaded) + )); + saving.await?; + assert_eq!(group.checkpoint_manifest()?.as_ref(), Some(next.marker())); + assert_eq!(group.first_index()?, 3); + log.require_cell_group_archive(&initial.route.history, activation) + .await?; + assert!(group.read()?.archive_required); + assert!(!group.archive_compaction.permits(next.marker())?); + assert_eq!(pool.usage().sources, 1); + drop(checkpoint); + assert_eq!(pool.usage().sources, 0); + log.close().await?; + drop(pages); + domain.shutdown().await; + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn archive_activation_is_retained_by_entered_storage() -> anyhow::Result<()> { + use crate::raft::storage_fault as fault; + let root = tempfile::tempdir()?; + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let gate = fault::Gate::new(None); + let log = domain + .run(ReplicaLog::open( + options(root.path()), + Arc::new(fault::FaultFileSystem { gate: gate.clone() }), + )) + .await??; + let initial = bootstrap(5); + let group = log.create_cell_group(initial.clone()).await?; + let marker = group.group_manifest()?.unwrap(); + let abandoned = group.archive_compaction.begin_activation()?; + assert!(!group.archive_compaction.permits(&marker)?); + drop(abandoned); + assert!(group.archive_compaction.permits(&marker)?); + assert!(!group.read()?.archive_required); + + let activation = group.archive_compaction.begin_activation()?; + let release = gate.arm(fault::HOLD_SYNC); + let mut writing = + Box::pin(log.require_cell_group_archive(&initial.route.history, activation.clone())); + let pending = domain + .run(crate::asyncrt::timeout( + Duration::from_millis(50), + writing.as_mut(), + )) + .await?; + assert!(pending.is_err(), "actual metadata sync remains held"); + domain + .run(crate::asyncrt::timeout( + Duration::from_secs(5), + gate.entered.notified(), + )) + .await??; + drop(writing); + drop(activation); + assert!(!group.archive_compaction.permits(&marker)?); + assert!(!group.read()?.archive_required); + drop(release); + log.reserve_node_revision().await?; + assert!(group.read()?.archive_required); + assert!(!group.archive_compaction.permits(&marker)?); + log.close().await?; + domain.shutdown().await; + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn retained_log_admission_accounts_selected_entries_and_reopen() -> anyhow::Result<()> { + use crate::raft::cell_groups::{CellGroupLogCapacity, CellGroupLogRetentionLimits}; + let root = tempfile::tempdir()?; + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let mut settings = options(root.path()); + settings.cell_group_retention = CellGroupLogRetentionLimits { + node: CellGroupLogCapacity { + bytes: 8192, + entries: 5, + }, + group: CellGroupLogCapacity { + bytes: 4096, + entries: 3, + }, + progress_node: CellGroupLogCapacity { + bytes: 2048, + entries: 2, + }, + progress_group: CellGroupLogCapacity { + bytes: 1024, + entries: 1, + }, + }; + let log = domain + .run(ReplicaLog::open( + settings.clone(), + Arc::new(cellar_ltx::DirectFileSystem), + )) + .await??; + let group = log.create_cell_group(bootstrap(5)).await?; + let sibling = log.create_cell_group(bootstrap(7)).await?; + let data = |index| Entry { + index, + term: 1, + data: vec![9; 256].into(), + ..Default::default() + }; + let mut first = GroupWrite::new(&group); + first.entries = vec![data(2), data(3)]; + log.persist_cells(OperationId::new(50), vec![first]).await?; + let selected = group.log_retention_usage()?; + assert_eq!(selected.retained_entries, 2); + assert_eq!(selected.reserved_entries, 0); + let mut full = GroupWrite::new(&group); + full.entries.push(data(4)); + assert!(matches!( + log.persist_cells(OperationId::new(51), vec![full]).await, + Err(LogError::Overloaded) + )); + assert_eq!(group.last_index()?, 3); + assert_eq!(group.log_retention_usage()?, selected); + // Empty Raft control entries have their own bounded space. Another data + // entry cannot consume it, and a healthy sibling still has node capacity. + let mut control = GroupWrite::new(&group); + control.entries.push(Entry { + index: 4, + term: 1, + ..Default::default() + }); + log.persist_cells(OperationId::new(52), vec![control]) + .await?; + let mut other = GroupWrite::new(&sibling); + other.entries.push(data(2)); + log.persist_cells(OperationId::new(53), vec![other]).await?; + let before = log.cell_group_log_retention_usage()?; + assert_eq!(before.retained_entries, 4); + assert_eq!(before.reserved_entries, 0); + log.close().await?; + drop(group); + drop(sibling); + drop(log); + settings.mode = LogOpenMode::Resume; + settings.process = ProcessGeneration::new([9; 16])?; + let log = domain + .run(ReplicaLog::open( + settings, + Arc::new(cellar_ltx::DirectFileSystem), + )) + .await??; + assert_eq!(log.cell_group_log_retention_usage()?, before); + log.close().await?; + domain.shutdown().await; + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn retained_log_lease_survives_waiter_loss_and_transfers_once() -> anyhow::Result<()> { + use crate::raft::storage_fault as fault; + let root = tempfile::tempdir()?; + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let gate = fault::Gate::new(None); + let log = domain + .run(ReplicaLog::open( + options(root.path()), + Arc::new(fault::FaultFileSystem { gate: gate.clone() }), + )) + .await??; + let group = log.create_cell_group(bootstrap(5)).await?; + let sibling = log.create_cell_group(bootstrap(7)).await?; + let entry = Entry { + index: 2, + term: 1, + data: vec![1; 256].into(), + ..Default::default() + }; + let lease = group.log_retention().unwrap().reserve_entry(&entry)?; + let reserved = group.log_retention_usage()?; + assert_eq!( + (reserved.retained_entries, reserved.reserved_entries), + (0, 1) + ); + let mut wrong = GroupWrite::new(&sibling); + wrong.entries.push(entry.clone()); + wrong.log_reservations.insert(2, lease.clone()); + assert!(matches!( + log.persist_cells(OperationId::new(60), vec![wrong]).await, + Err(LogError::WrongIdentity) + )); + let mut changed = GroupWrite::new(&group); + changed.entries.push(Entry { + term: 2, + ..entry.clone() + }); + // Keep the proposed cache state valid so the negative reaches the exact + // lease binding check, rather than failing the independent term invariant. + let mut hard = group.initial_state()?.hard_state; + hard.term = 2; + changed.hard_state = Some(hard); + changed.log_reservations.insert(2, lease.clone()); + let changed_result = log.persist_cells(OperationId::new(61), vec![changed]).await; + assert!( + matches!(&changed_result, Err(LogError::WrongIdentity)), + "wrong-term lease refusal: {changed_result:?}" + ); + assert_eq!(group.initial_state()?.hard_state.term, 1); + assert_eq!(group.last_index()?, 1); + assert_eq!(group.log_retention_usage()?, reserved); + let release = gate.arm(fault::HOLD_SYNC); + let mut write = GroupWrite::new(&group); + write.entries.push(entry.clone()); + write.log_reservations.insert(2, lease.clone()); + let mut saving = Box::pin(log.persist_cells(OperationId::new(62), vec![write])); + assert!(domain + .run(crate::asyncrt::timeout( + Duration::from_millis(50), + saving.as_mut() + )) + .await? + .is_err()); + domain + .run(crate::asyncrt::timeout( + Duration::from_secs(5), + gate.entered.notified(), + )) + .await??; + let physical_lease = Arc::downgrade(&lease); + drop(saving); + drop(lease); + assert!( + physical_lease.upgrade().is_some(), + "entered storage owns the final strong lease" + ); + assert_eq!(group.log_retention_usage()?, reserved); + drop(release); + log.reserve_node_revision().await?; + let selected = group.log_retention_usage()?; + assert_eq!( + (selected.retained_entries, selected.reserved_entries), + (1, 0) + ); + assert_eq!(selected.retained_bytes, reserved.reserved_bytes); + assert!( + physical_lease.upgrade().is_none(), + "FIFO join releases every physical lease clone" + ); + assert_eq!(group.log_retention_usage()?, selected); + // A separately retained client lease is consumed exactly once. Keeping it + // cannot double-charge selected debt or authorize reuse of the capability. + let next = Entry { + index: 3, + ..entry.clone() + }; + let lease = group.log_retention().unwrap().reserve_entry(&next)?; + let mut write = GroupWrite::new(&group); + write.entries.push(next.clone()); + write.log_reservations.insert(3, lease.clone()); + log.persist_cells(OperationId::new(63), vec![write]).await?; + let complete = group.log_retention_usage()?; + assert_eq!( + (complete.retained_entries, complete.reserved_entries), + (2, 0) + ); + assert!(matches!( + lease.validate(group.log_retention().unwrap(), &next), + Err(LogError::WrongIdentity) + )); + drop(lease); + assert_eq!(group.log_retention_usage()?, complete); + log.close().await?; + domain.shutdown().await; + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn retained_log_byte_limits_and_selected_suffix_replacement() -> anyhow::Result<()> { + use crate::raft::cell_groups::{CellGroupLogCapacity, CellGroupLogRetentionLimits}; + let root = tempfile::tempdir()?; + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let mut settings = options(root.path()); + settings.cell_group_retention = CellGroupLogRetentionLimits { + node: CellGroupLogCapacity { + bytes: 8192, + entries: 100, + }, + group: CellGroupLogCapacity { + bytes: 4096, + entries: 20, + }, + progress_node: CellGroupLogCapacity { + bytes: 2048, + entries: 20, + }, + progress_group: CellGroupLogCapacity { + bytes: 1024, + entries: 2, + }, + }; + let log = domain + .run(ReplicaLog::open( + settings, + Arc::new(cellar_ltx::DirectFileSystem), + )) + .await??; + let a = log.create_cell_group(bootstrap(5)).await?; + let b = log.create_cell_group(bootstrap(7)).await?; + let c = log.create_cell_group(bootstrap(9)).await?; + let large = Entry { + index: 2, + term: 1, + data: vec![1; 1800].into(), + ..Default::default() + }; + let mut write = GroupWrite::new(&a); + write.entries.push(large.clone()); + log.persist_cells(OperationId::new(70), vec![write]).await?; + let original = a.log_retention_usage()?; + assert_eq!( + (original.retained_entries, original.reserved_entries), + (1, 0) + ); + let mut write = GroupWrite::new(&a); + write.entries.push(Entry { + index: 3, + ..large.clone() + }); + assert!( + matches!( + log.persist_cells(OperationId::new(71), vec![write]).await, + Err(LogError::Overloaded) + ), + "group bytes refuse while entry counts have capacity" + ); + assert_eq!(a.last_index()?, 2); + assert_eq!(a.log_retention_usage()?, original); + let mut write = GroupWrite::new(&b); + write.entries.push(large.clone()); + log.persist_cells(OperationId::new(72), vec![write]).await?; + let before = log.cell_group_log_retention_usage()?; + assert_eq!((before.retained_entries, before.reserved_entries), (2, 0)); + let mut write = GroupWrite::new(&c); + write.entries.push(large.clone()); + assert!( + matches!( + log.persist_cells(OperationId::new(73), vec![write]).await, + Err(LogError::Overloaded) + ), + "node bytes refuse a fitting group while entry counts have capacity" + ); + assert_eq!(c.last_index()?, 1); + assert_eq!(log.cell_group_log_retention_usage()?, before); + // A smaller replacement of an uncommitted suffix releases debt only after + // its new entry and term are actually selected by storage. Both versions + // fit the admitted overlap; no archive observation provides the refund. + let mut write = GroupWrite::new(&a); + write.entries.push(Entry { + index: 2, + term: 2, + data: vec![2; 64].into(), + ..Default::default() + }); + let mut hard = a.initial_state()?.hard_state; + hard.term = 2; + write.hard_state = Some(hard); + log.persist_cells(OperationId::new(74), vec![write]).await?; + let smaller = a.log_retention_usage()?; + assert_eq!((smaller.retained_entries, smaller.reserved_entries), (1, 0)); + assert!(smaller.retained_bytes < original.retained_bytes); + let mut write = GroupWrite::new(&c); + write.entries.push(large); + log.persist_cells(OperationId::new(75), vec![write]).await?; + let complete = log.cell_group_log_retention_usage()?; + assert_eq!( + (complete.retained_entries, complete.reserved_entries), + (3, 0) + ); + assert_eq!(a.last_index()?, 2); + assert_eq!(a.term(2)?, 2); + log.close().await?; + domain.shutdown().await; + Ok(()) +} diff --git a/crates/cellar/durability/raft/storage_genesis_tests.rs b/crates/cellar/durability/raft/storage_genesis_tests.rs index 911345f36..021d322fd 100644 --- a/crates/cellar/durability/raft/storage_genesis_tests.rs +++ b/crates/cellar/durability/raft/storage_genesis_tests.rs @@ -1,3 +1,5 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + use super::*; use crate::asyncrt::ExecutionDomain; use crate::raft::member_state::FleetGroupBinding; @@ -45,7 +47,11 @@ async fn managed_genesis_reopen_never_retrofits_bare_or_rebound_slots() -> anyho )) .await??; let initial = bootstrap(5); - let binding = FleetGroupBinding::new(IncarnationId::new([6; 16])?, &initial.scope, [90; 32])?; + let binding = FleetGroupBinding::new( + IncarnationId::new([6; 16])?, + &GroupHistory::from(&initial.scope), + [90; 32], + )?; let guarded = log .create_managed_group(initial.clone(), binding.clone()) .await?; @@ -53,7 +59,11 @@ async fn managed_genesis_reopen_never_retrofits_bare_or_rebound_slots() -> anyho assert_eq!(before.fleet_status().unwrap().changed_at.get(), 1); let bare = bootstrap(7); log.create_group(bare.clone()).await?; - let bare_binding = FleetGroupBinding::new(IncarnationId::new([6; 16])?, &bare.scope, [90; 32])?; + let bare_binding = FleetGroupBinding::new( + IncarnationId::new([6; 16])?, + &GroupHistory::from(&bare.scope), + [90; 32], + )?; assert!(matches!( log.create_managed_group(bare.clone(), bare_binding).await, Err(LogError::WrongIdentity) @@ -63,13 +73,20 @@ async fn managed_genesis_reopen_never_retrofits_bare_or_rebound_slots() -> anyho .membership()? .fleet_status() .is_none()); - let wrong = FleetGroupBinding::new(IncarnationId::new([8; 16])?, &initial.scope, [90; 32])?; + let wrong = FleetGroupBinding::new( + IncarnationId::new([8; 16])?, + &GroupHistory::from(&initial.scope), + [90; 32], + )?; assert!(matches!( log.create_managed_group(initial.clone(), wrong).await, Err(LogError::WrongIdentity) )); - let other_catalog = - FleetGroupBinding::new(IncarnationId::new([6; 16])?, &initial.scope, [91; 32])?; + let other_catalog = FleetGroupBinding::new( + IncarnationId::new([6; 16])?, + &GroupHistory::from(&initial.scope), + [91; 32], + )?; assert!(matches!( log.create_managed_group(initial.clone(), other_catalog) .await, @@ -106,7 +123,11 @@ async fn managed_genesis_publication_waits_for_actual_engine_sync() -> anyhow::R )) .await??; let initial = bootstrap(5); - let binding = FleetGroupBinding::new(IncarnationId::new([6; 16])?, &initial.scope, [90; 32])?; + let binding = FleetGroupBinding::new( + IncarnationId::new([6; 16])?, + &GroupHistory::from(&initial.scope), + [90; 32], + )?; let release = gate.arm(fault::HOLD_SYNC); let owned = log.clone(); let proposal = initial.clone(); @@ -133,3 +154,174 @@ async fn managed_genesis_publication_waits_for_actual_engine_sync() -> anyhow::R domain.shutdown().await; Ok(()) } + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn physical_catalog_rejects_cell_aliases_across_restart() -> anyhow::Result<()> { + let root = tempfile::tempdir()?; + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let mut options = options(root.path()); + let initial = bootstrap(5); + let mut alias = initial.clone(); + alias.scope.cell = "Counter:alias".into(); + let log = domain + .run(ReplicaLog::open( + options.clone(), + Arc::new(cellar_ltx::DirectFileSystem), + )) + .await??; + let key = log.create_group(initial.clone()).await?.numeric_key(); + for pass in 0..2 { + let log = if pass == 0 { + log.clone() + } else { + options.mode = LogOpenMode::Resume; + options.process = ProcessGeneration::new([9; 16])?; + domain + .run(ReplicaLog::open( + options.clone(), + Arc::new(cellar_ltx::DirectFileSystem), + )) + .await?? + }; + assert_eq!(log.group(&initial.scope)?.numeric_key(), key); + assert_eq!(log.create_group(initial.clone()).await?.numeric_key(), key); + assert!(matches!( + log.group(&alias.scope), + Err(LogError::WrongIdentity) + )); + assert!(matches!( + log.create_group(alias.clone()).await, + Err(LogError::WrongIdentity) + )); + assert_eq!(log.group_count_with_pending(Some(&initial.scope))?, 1); + assert!(matches!( + log.group_count_with_pending(Some(&alias.scope)), + Err(LogError::WrongIdentity) + )); + assert!(matches!( + log.physical_inventory(Some(&alias.scope)), + Err(LogError::WrongIdentity) + )); + assert!(matches!( + log.cell_group_inventory(None), + Err(LogError::WrongIdentity) + )); + let inventory = log.physical_inventory(Some(&initial.scope))?; + assert_eq!(inventory.histories().len(), 1); + assert_eq!( + inventory.history(&initial.scope)?.unwrap().birth_digest(), + initial.birth_digest + ); + assert_eq!(log.group_count_with_pending(Some(&bootstrap(7).scope))?, 2); + assert_eq!(log.groups()?.len(), 1); + log.close().await?; + } + domain.shutdown().await; + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn physical_mapping_domain_refuses_old_and_corrupt_engine_records() -> anyhow::Result<()> { + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + for corruption in [ + "old-scope", + "old-adapter", + "unknown-domain", + "zero-history", + "empty-cell", + ] { + let root = tempfile::tempdir()?; + let mut options = options(root.path()); + let initial = bootstrap(5); + let log = domain + .run(ReplicaLog::open( + options.clone(), + Arc::new(cellar_ltx::DirectFileSystem), + )) + .await??; + let key = log.create_group(initial.clone()).await?.numeric_key(); + let performance = log.performance(); + log.close().await?; + let dir = options.directory.join("log").to_string_lossy().into_owned(); + domain + .blocking(move || -> anyhow::Result<()> { + let engine = Engine::open_with_file_system( + raft_engine::Config { + dir, + ..Default::default() + }, + Arc::new(EngineFileSystem( + Arc::new(cellar_ltx::DirectFileSystem), + performance, + )), + )?; + let mut record: serde_json::Value = + serde_json::from_slice(&engine.get(key, GROUP_KEY).unwrap())?; + let mapping = &mut record["mapping"]; + assert_eq!(mapping["format"], 1); + assert_eq!(mapping["domain"], "cellar-group-mapping-1"); + assert_eq!(mapping["application"]["domain"], "single-cell-1"); + assert_eq!(mapping["application"]["state"], initial.scope.cell); + assert_eq!(mapping["history"].as_object().unwrap().len(), 4); + assert!(mapping["history"].get("cell").is_none()); + let mut catalog_key = b"g:".to_vec(); + catalog_key.extend_from_slice(&initial.scope.group.bytes()); + catalog_key.extend_from_slice(&initial.scope.incarnation.bytes()); + assert_eq!( + serde_json::from_slice::( + &engine.get(0, &catalog_key).unwrap() + )?, + *mapping + ); + match corruption { + "old-scope" => { + let fields = mapping.as_object_mut().unwrap(); + fields.remove("domain"); + fields.remove("history"); + fields.remove("application"); + fields.insert( + "scope".into(), + serde_json::to_value( + crate::application::consensus::protocol::WireScope::from( + &initial.scope, + ), + )?, + ); + } + "old-adapter" => { + mapping.as_object_mut().unwrap().remove("application"); + mapping["application_cell"] = serde_json::json!(initial.scope.cell); + } + "unknown-domain" => { + mapping["domain"] = serde_json::json!("cellar-image-mapping-1") + } + "zero-history" => { + mapping["history"]["group"] = serde_json::json!(vec![0u8; 16]) + } + "empty-cell" => mapping["application"]["state"] = serde_json::json!(""), + _ => unreachable!(), + } + let mut batch = LogBatch::default(); + batch.put(0, catalog_key, serde_json::to_vec(mapping)?)?; + batch.put(key, GROUP_KEY.to_vec(), serde_json::to_vec(&record)?)?; + engine.write(&mut batch, true)?; + Ok(()) + })? + .await??; + options.mode = LogOpenMode::Resume; + assert!( + matches!( + domain + .run(ReplicaLog::open( + options, + Arc::new(cellar_ltx::DirectFileSystem) + )) + .await?, + Err(LogError::Corrupt(_)) + ), + "accepted {corruption}" + ); + } + domain.shutdown().await; + Ok(()) +} diff --git a/crates/cellar/durability/raft/storage_lifecycle_tests.rs b/crates/cellar/durability/raft/storage_lifecycle_tests.rs index 390392a25..697ba1aea 100644 --- a/crates/cellar/durability/raft/storage_lifecycle_tests.rs +++ b/crates/cellar/durability/raft/storage_lifecycle_tests.rs @@ -67,6 +67,13 @@ async fn fleet_lifecycle_maintenance_stop_requires_exact_gate_and_retains_accept }) .await?; let mut gate = stop(&binding, 2); + assert!(log.lifecycle_guard()?.unwrap().admits_group_start(&binding)); + let mut other_boot = binding.clone(); + other_boot.boot = [99; 16]; + assert!(!log + .lifecycle_guard()? + .unwrap() + .admits_group_start(&other_boot)); gate.purpose = LifecyclePurpose::Maintenance; let mut selector = gate.clone(); selector.finalization = [13; 16]; @@ -93,6 +100,7 @@ async fn fleet_lifecycle_maintenance_stop_requires_exact_gate_and_retains_accept deadline: deadline(&log)?, }) .await?; + assert!(!log.lifecycle_guard()?.unwrap().admits_group_start(&binding)); let mut foreign = gate.clone(); foreign.request = [15; 16]; assert!(matches!( @@ -113,6 +121,7 @@ async fn fleet_lifecycle_maintenance_stop_requires_exact_gate_and_retains_accept .await?; } assert_eq!(log.lifecycle_guard()?.unwrap().accepted(), Some(&selector)); + assert!(!log.lifecycle_guard()?.unwrap().admits_group_start(&binding)); assert!(matches!( log.update_lifecycle(LifecycleUpdate::Fence { boot: binding, @@ -790,6 +799,7 @@ async fn fleet_lifecycle_resume_requires_exact_gate_and_survives_reopen() -> any log.lifecycle_guard()?.unwrap().resumed(), Some((&resume, &gate)) ); + assert!(log.lifecycle_guard()?.unwrap().admits_group_start(&binding)); assert!(matches!( log.update_lifecycle(LifecycleUpdate::Fence { boot: binding.clone(), diff --git a/crates/cellar/durability/raft/transport.rs b/crates/cellar/durability/raft/transport.rs index 8c3285cc5..bcbf958d1 100644 --- a/crates/cellar/durability/raft/transport.rs +++ b/crates/cellar/durability/raft/transport.rs @@ -8,14 +8,32 @@ use super::completion::RuntimeError; use super::mailbox; use super::performance::{Operation, Timer}; use super::runtime::RaftRuntime; +use cellar_logic::cell_group::GroupHistory; +#[path = "transport_runtime.rs"] +mod runtime_adapter; +use runtime_adapter::TransportRuntime; +#[path = "group_snapshot_transport/mod.rs"] +mod group_snapshots; +#[path = "transport_shared.rs"] +mod shared; +#[cfg(all(test, not(cellar_internal_tests)))] +#[path = "transport_shared_fixture.rs"] +mod shared_fixture; use super::transport_frame::{self as frame, Disposition, FrameHead, RequestHeader, Route}; use super::transport_options::{NamespaceQueryOrigin, RaftPeerNode, RaftTls, TransportOptions}; use crate::asyncrt::DomainToken; use crate::peer_auth::PeerAuth; use ::raft::prelude::{Message, MessageType}; use cellar_logic::consensus::{ClusterId, GroupScope, NamespaceId, NodeId, ReplicaIdentity}; +pub(crate) use group_snapshots::seed::Grant as CellGroupSeedGrant; +pub(crate) use group_snapshots::seed::{ManagedArchiveSeed, ManagedPeerSeed}; +pub use group_snapshots::{ + CellGroupCheckpointRegistration, CellGroupSeedError, CellGroupSeedSession, +}; use protobuf::Message as _; use serde::{Deserialize, Serialize}; +pub(crate) use shared::PhysicalRegistration; +use shared::SendRoute; use std::collections::{BTreeMap, BTreeSet}; use std::future::Future; use std::io::IoSlice; @@ -28,6 +46,20 @@ use tokio::io::{AsyncRead, AsyncReadExt, AsyncWrite, AsyncWriteExt}; use tokio::net::{TcpListener, TcpStream}; use tokio::sync::{mpsc, oneshot, watch, Notify, Semaphore}; +#[path = "command_cell_transport.rs"] +mod cell_commands; +#[path = "query_cell_transport.rs"] +mod cell_queries; +#[path = "command_origins.rs"] +mod command_origins; +#[path = "query_reply.rs"] +mod query_replies; +pub use cell_commands::{ + CellCommandGateway, CellRemoteCommandResponse, CellRemoteEnrollmentResponse, + CellRemoteReceiptResponse, +}; +#[path = "query_cell_discovery.rs"] +mod cell_query_discovery; #[path = "transport_control_send.rs"] mod control_sender; #[path = "transport_directory.rs"] @@ -46,8 +78,13 @@ pub(crate) mod inspections; mod metrics; #[path = "node_inspection_transport.rs"] pub(crate) mod node_inspections; +#[path = "park_transport.rs"] +mod parks; +pub(crate) use parks::ParkDeliveryReceipt; #[path = "query_transport.rs"] mod queries; +pub(crate) use cell_queries::CellQueryContext; +pub use cell_queries::{CellQueryGateway, CellRemoteQueryResponse}; #[path = "transport_resources.rs"] mod resources; #[path = "transport_send.rs"] @@ -71,6 +108,8 @@ pub enum TransportError { Tls(String), Runtime(RuntimeError), Authentication, + Query(cellar_logic::consensus::CommandFailure), + Command(cellar_logic::consensus::CommandFailure), Protocol, Identity, Replay, @@ -104,8 +143,8 @@ impl From for TransportError { } #[derive(Clone, Debug, PartialEq, Eq)] -pub struct RaftPeer { - pub replica: ReplicaIdentity, +pub struct RaftPeer { + pub replica: ReplicaIdentity, pub label: String, pub server_name: String, pub control_address: SocketAddr, @@ -113,7 +152,7 @@ pub struct RaftPeer { pub query_address: SocketAddr, } -impl RaftPeer { +impl RaftPeer { fn same_host(&self, other: &Self) -> bool { self.replica.node == other.replica.node && self.replica.disk == other.replica.disk @@ -126,8 +165,8 @@ impl RaftPeer { } } -impl From<&RaftPeer> for RaftPeerNode { - fn from(peer: &RaftPeer) -> Self { +impl From<&RaftPeer> for RaftPeerNode { + fn from(peer: &RaftPeer) -> Self { Self { node: peer.replica.node, disk: peer.replica.disk, @@ -164,7 +203,7 @@ impl RaftPeerNode { } } - fn same_host(&self, peer: &RaftPeer) -> bool { + fn same_host(&self, peer: &RaftPeer) -> bool { self.node == peer.replica.node && self.disk == peer.replica.disk && self.process == peer.replica.process @@ -179,11 +218,11 @@ impl RaftPeerNode { /// An already authorized routing view, installed by the membership controller. /// Installing it never creates storage or changes Raft's configuration. #[derive(Clone, Debug, PartialEq, Eq)] -pub struct GroupRoute { - pub scope: GroupScope, +pub struct GroupRoute { + pub scope: S, pub schema_digest: [u8; 32], pub revision: u64, - pub peers: Vec, + pub peers: Vec>, } /// Message admission only. No durability, commit or applied position is implied. @@ -239,9 +278,10 @@ impl Banks { } struct SendRequest { - scope: GroupScope, - route: Arc, - peer: RaftPeer, + // The runtime configuration fence survives a dropped Delivery waiter. + _source: Option>, + route: SendRoute, + member: cellar_logic::consensus::MemberId, header: RequestHeader, body: Vec, snapshot: bool, @@ -272,7 +312,16 @@ struct Registry { closed: bool, tasks: usize, groups: BTreeMap>, + physical_groups: BTreeMap>>, + // One bounded lifecycle token per physical route. Advisory process/member + // refreshes keep it; explicit replacement/removal starts another lifetime. + physical_owners: BTreeMap>, + group_checkpoint_pool: Option, + group_archive_seeds: Option, + group_checkpoints: BTreeMap>, query_groups: BTreeMap>, + cell_query_groups: BTreeMap>>, + command_origins: BTreeMap<(ClusterId, NamespaceId, NodeId), Arc>, origins: BTreeMap<(ClusterId, NamespaceId, NodeId), Arc>, inspection_origins: BTreeMap<(ClusterId, NamespaceId, NodeId), Arc>, inspection_peers: BTreeMap>, @@ -283,15 +332,29 @@ struct Registry { } impl Registry { + fn route_count(&self) -> usize { + self.groups.len() + + self.physical_groups.len() + + self.query_groups.len() + + self.cell_query_groups.len() + } + fn prune_pools(&mut self) { let used: BTreeSet<_> = self .groups .values() .chain(self.query_groups.values()) .flat_map(|route| route.peers.iter().map(|peer| peer.replica.node)) + .chain( + self.physical_groups + .values() + .chain(self.cell_query_groups.values()) + .flat_map(|route| route.peers.iter().map(|peer| peer.replica.node)), + ) .chain( self.origins .values() + .chain(self.command_origins.values()) .filter_map(|grant| grant.peer.as_ref().map(|peer| peer.node)), ) .collect(); @@ -309,14 +372,20 @@ impl Registry { struct State { performance: Arc, domain: DomainToken, - runtime: RaftRuntime, + runtime: TransportRuntime, auth: Arc, tls: RaftTls, options: TransportOptions, local_control: SocketAddr, local_data: SocketAddr, local_query: SocketAddr, + cell_command_gateways: + Mutex>>, + cell_query_gateways: + Mutex>>, query_gateway: Mutex>>, + fleet_gateway: + Mutex>>, registry: Mutex, outgoing: Banks, incoming: Banks, @@ -326,7 +395,7 @@ struct State { accepted: AtomicU64, refused: AtomicU64, failed: AtomicU64, - claim: Mutex>, + claim: Mutex>>, snapshot_source: Mutex>>, prepared_snapshot: super::snapshot::prepared::Cache, snapshot_source_budget: Arc, @@ -403,6 +472,48 @@ impl RaftTransport { data: TcpListener, query: TcpListener, options: TransportOptions, + ) -> Result { + Self::start_runtime( + TransportRuntime::Legacy(runtime), + auth, + tls, + control, + data, + query, + options, + ) + .await + } + + pub async fn start_shared( + runtime: super::cell_groups::CellGroupRuntime, + auth: Arc, + tls: RaftTls, + control: TcpListener, + data: TcpListener, + query: TcpListener, + options: TransportOptions, + ) -> Result { + Self::start_runtime( + TransportRuntime::Shared(runtime), + auth, + tls, + control, + data, + query, + options, + ) + .await + } + + async fn start_runtime( + runtime: TransportRuntime, + auth: Arc, + tls: RaftTls, + control: TcpListener, + data: TcpListener, + query: TcpListener, + options: TransportOptions, ) -> Result { options.validate()?; let local_control = control.local_addr()?; @@ -433,11 +544,21 @@ impl RaftTransport { local_data, local_query, query_gateway: Mutex::new(None), + fleet_gateway: Mutex::new(None), + cell_command_gateways: Mutex::new(BTreeMap::new()), + cell_query_gateways: Mutex::new(BTreeMap::new()), registry: Mutex::new(Registry { closed: false, tasks: 0, groups: BTreeMap::new(), + physical_groups: BTreeMap::new(), + physical_owners: BTreeMap::new(), + group_checkpoint_pool: None, + group_archive_seeds: None, + group_checkpoints: BTreeMap::new(), query_groups: BTreeMap::new(), + cell_query_groups: BTreeMap::new(), + command_origins: BTreeMap::new(), origins: BTreeMap::new(), inspection_origins: BTreeMap::new(), inspection_peers: BTreeMap::new(), @@ -570,6 +691,7 @@ impl RaftTransport { state.admit(®istry)?; for peer in &route.peers { registry.check_node_view(&RaftPeerNode::from(peer))?; + registry.check_physical_endpoint(&RaftPeerNode::from(peer))?; } if registry.query_groups.contains_key(&route.scope) { return Err(TransportError::Identity); @@ -594,7 +716,7 @@ impl RaftTransport { } } } - } else if registry.groups.len() + registry.query_groups.len() >= state.options.max_groups { + } else if registry.route_count() >= state.options.max_groups { return Err(TransportError::Overloaded); } // Existing peers' physical addresses/process must agree across groups. @@ -745,9 +867,13 @@ impl RaftTransport { header.set_sequence(sequence); let (result, wait) = oneshot::channel(); let request = SendRequest { - scope: scope.clone(), - route, - peer, + _source: None, + route: SendRoute::Legacy { + scope: scope.clone(), + route, + peer: peer.clone(), + }, + member: peer.replica.member, header, body, snapshot: message.get_msg_type() == MessageType::MsgSnapshot, @@ -962,6 +1088,15 @@ impl State { if let Ok(mut gateway) = self.query_gateway.lock() { gateway.take(); } + if let Ok(mut gateway) = self.fleet_gateway.lock() { + gateway.take(); + } + if let Ok(mut gateways) = self.cell_command_gateways.lock() { + gateways.clear(); + } + if let Ok(mut gateways) = self.cell_query_gateways.lock() { + gateways.clear(); + } if finished { if let Ok(mut claim) = self.claim.lock() { claim.take(); @@ -1026,7 +1161,7 @@ impl State { query_items: qi + rqi, tasks: registry.tasks, peers: registry.pools.len(), - groups: registry.groups.len(), + groups: registry.groups.len() + registry.physical_groups.len(), accepted: self.accepted.load(Ordering::Acquire), refused: self.refused.load(Ordering::Acquire), failed: self.failed.load(Ordering::Acquire), @@ -1034,7 +1169,10 @@ impl State { } fn current(&self, request: &SendRequest) -> bool { - self.current_route(&request.scope, &request.route, &request.peer) + match &request.route { + SendRoute::Legacy { scope, route, peer } => self.current_route(scope, route, peer), + SendRoute::Physical { route, peer } => self.current_physical(route, peer), + } } fn current_route(&self, scope: &GroupScope, route: &Arc, peer: &RaftPeer) -> bool { @@ -1072,6 +1210,9 @@ impl State { } fn authorize_route(&self, wire: &Route) -> Result, TransportError> { + if wire.is_group() { + return self.authorize_physical(wire); + } let (source, target) = wire.decode()?; let registry = self.registry.lock().map_err(|_| TransportError::Closed)?; self.admit(®istry)?; @@ -1212,7 +1353,7 @@ impl State { Ok(()) }) .await?; - if &prefix[..4] == inspections::REQUEST_MAGIC { + if &prefix[..4] == node_inspections::REQUEST_MAGIC { if lane != Lane::Control || sequence != 0 || !self.inspection_enabled.load(Ordering::Acquire) @@ -1222,13 +1363,13 @@ impl State { let exporter = stream .get_ref() .1 - .export_keying_material([0; 32], inspections::EXPORT_LABEL, None) + .export_keying_material([0; 32], node_inspections::EXPORT_LABEL, None) .map_err(|error| TransportError::Tls(error.to_string()))?; return self - .receive_inspection(&mut stream, &exporter, prefix) + .receive_node_inspection(&mut stream, &exporter, prefix) .await; } - if &prefix[..4] == node_inspections::REQUEST_MAGIC { + if &prefix[..4] == fleet_actions::REQUEST_MAGIC { if lane != Lane::Control || sequence != 0 || !self.inspection_enabled.load(Ordering::Acquire) @@ -1238,26 +1379,23 @@ impl State { let exporter = stream .get_ref() .1 - .export_keying_material([0; 32], node_inspections::EXPORT_LABEL, None) + .export_keying_material([0; 32], fleet_actions::EXPORT_LABEL, None) .map_err(|error| TransportError::Tls(error.to_string()))?; return self - .receive_node_inspection(&mut stream, &exporter, prefix) + .receive_fleet_action(&mut stream, &exporter, prefix, connected_at) .await; } - if &prefix[..4] == fleet_actions::REQUEST_MAGIC { - if lane != Lane::Control - || sequence != 0 - || !self.inspection_enabled.load(Ordering::Acquire) - { + if &prefix[..4] == parks::REQUEST_MAGIC { + if lane != Lane::Control || sequence != 0 { return Err(TransportError::Protocol); } let exporter = stream .get_ref() .1 - .export_keying_material([0; 32], fleet_actions::EXPORT_LABEL, None) + .export_keying_material([0; 32], parks::EXPORT_LABEL, None) .map_err(|error| TransportError::Tls(error.to_string()))?; return self - .receive_fleet_action(&mut stream, &exporter, prefix, connected_at) + .receive_park(&mut stream, &exporter, prefix, connected_at) .await; } if &prefix[..4] == fleet_stops::REQUEST_MAGIC { @@ -1292,16 +1430,26 @@ impl State { .receive_fleet_inspection(&mut stream, &exporter, prefix) .await; } - if &prefix[..4] == seeds::REQUEST_MAGIC { + if &prefix[..4] == group_snapshots::seed::REQUEST_MAGIC { if lane != Lane::Data || sequence != 0 { return Err(TransportError::Protocol); } let exporter = stream .get_ref() .1 - .export_keying_material([0; 32], seeds::EXPORT_LABEL, None) + .export_keying_material([0; 32], group_snapshots::seed::EXPORT_LABEL, None) .map_err(|error| TransportError::Tls(error.to_string()))?; - return self.receive_seed(&mut stream, &exporter, prefix).await; + return self + .receive_group_seed(&mut stream, &exporter, prefix) + .await; + } + if &prefix[..4] == group_snapshots::START_MAGIC { + if lane != Lane::Data || sequence != 0 { + return Err(TransportError::Protocol); + } + return self + .receive_group_checkpoint(&mut stream, &exporter, prefix) + .await; } if &prefix[..4] == snapshots::REQUEST_MAGIC { if lane != Lane::Data || sequence != 0 { @@ -1327,13 +1475,7 @@ impl State { if header.lane != lane || header.sequence()? <= sequence { return Err(TransportError::Replay); } - let (source, _) = header.route.decode()?; - let host = ( - source.node, - source.disk, - source.process, - header.route.source_label().to_owned(), - ); + let host = header.route.source_host()?; if source_host.as_ref().is_some_and(|h| *h != host) { return Err(TransportError::StalePeer); } @@ -1377,7 +1519,6 @@ impl State { let message = header.verify_body(&body, &self.options)?; // Routing/configuration may have changed while reading the body. self.authorize(&header)?; - let scope = source.scope.clone(); let admission = self.performance.timer(match lane { Lane::Control => Operation::ControlAdmit, Lane::Data => Operation::DataAdmit, @@ -1385,7 +1526,7 @@ impl State { let result = self .bounded(Some(&pool), async { self.runtime - .admit_step(&scope, message) + .admit_route(&header.route, message) .await .map_err(TransportError::from) }) diff --git a/crates/cellar/durability/raft/transport_directory.rs b/crates/cellar/durability/raft/transport_directory.rs index ba220c088..7695482e5 100644 --- a/crates/cellar/durability/raft/transport_directory.rs +++ b/crates/cellar/durability/raft/transport_directory.rs @@ -28,6 +28,26 @@ impl Registry { } impl RaftTransport { + pub(crate) fn local_label(&self) -> &str { + self.handle.0.auth.source() + } + + /// Accepted physical metadata only. This grants neither group membership + /// nor application authority; route installation rechecks the current cut. + pub(crate) fn observed_peer( + &self, + node: NodeId, + ) -> Result, TransportError> { + let state = &self.handle.0; + let registry = state.registry.lock().map_err(|_| TransportError::Closed)?; + state.admit(®istry)?; + let Some(view) = registry.node_views.get(&node) else { + return Ok(None); + }; + registry.check_node_view(view.peer())?; + Ok(Some(view.peer().clone())) + } + /// Refresh one explicitly configured physical node across all routes and /// permissions. This changes no group/member identity or configuration /// revision. A superseded local process is refused; its host must close and @@ -102,10 +122,22 @@ impl RaftTransport { return Err(TransportError::Identity); } } + for old in registry + .physical_groups + .values() + .chain(registry.cell_query_groups.values()) + .flat_map(|route| &route.peers) + .filter(|old| old.replica.node == peer.node) + { + if !same_registration(&RaftPeerNode::from(old), peer) { + return Err(TransportError::Identity); + } + } for old in registry .origins .values() .chain(registry.inspection_origins.values()) + .chain(registry.command_origins.values()) .filter_map(|grant| grant.peer.as_ref()) .filter(|old| old.node == peer.node) { @@ -129,10 +161,16 @@ impl RaftTransport { .values() .chain(registry.query_groups.values()) .any(|route| route.peers.iter().any(|old| old.replica.node == peer.node)) + || registry + .physical_groups + .values() + .chain(registry.cell_query_groups.values()) + .any(|route| route.peers.iter().any(|old| old.replica.node == peer.node)) || registry .origins .values() .chain(registry.inspection_origins.values()) + .chain(registry.command_origins.values()) .any(|grant| grant.peer.as_ref().is_some_and(|old| old.node == peer.node))); let physical = if registry.pools.contains_key(&peer.node) || routed { std::slice::from_ref(peer) @@ -142,16 +180,20 @@ impl RaftTransport { // Reservations include old workers until physical completion. A full // budget refuses the refresh without partially changing routing. let starts = state.prepare_pools(&mut registry, physical)?; + refresh_routes(&mut registry.physical_groups, peer); + refresh_routes(&mut registry.cell_query_groups, peer); refresh_routes(&mut registry.groups, peer); refresh_routes(&mut registry.query_groups, peer); let Registry { origins, inspection_origins, + command_origins, .. } = &mut *registry; for grant in origins .values_mut() .chain(inspection_origins.values_mut()) + .chain(command_origins.values_mut()) .filter(|grant| { grant .peer @@ -188,7 +230,10 @@ impl RaftTransport { } } -fn refresh_routes(routes: &mut BTreeMap>, peer: &RaftPeerNode) { +fn refresh_routes( + routes: &mut BTreeMap>>, + peer: &RaftPeerNode, +) { for route in routes.values_mut() { if !route .peers diff --git a/crates/cellar/durability/raft/transport_fleet.rs b/crates/cellar/durability/raft/transport_fleet.rs index 70721e849..d9ceb2121 100644 --- a/crates/cellar/durability/raft/transport_fleet.rs +++ b/crates/cellar/durability/raft/transport_fleet.rs @@ -20,6 +20,38 @@ impl Registry { } impl RaftTransport { + /// Resolve an endpoint only from an installed rooted trust cut and its + /// authenticated current node view. Discovery supplies no membership grant. + pub(crate) fn managed_group_peer( + &self, + scope: &GroupHistory, + node: NodeId, + minimum_revision: u64, + ) -> Result { + let state = &self.handle.0; + let registry = state.registry.lock().map_err(|_| TransportError::Closed)?; + state.admit(®istry)?; + let trust = registry + .fleet_trust + .as_ref() + .ok_or(TransportError::Refused)?; + if trust.cluster() != scope.cluster || trust.revision() < minimum_revision { + return Err(TransportError::StalePeer); + } + let view = registry + .node_views + .get(&node) + .ok_or(TransportError::StalePeer)?; + let enrolled = trust.node(node).ok_or(TransportError::Refused)?; + if view.cluster() != scope.cluster + || enrolled.storage_instance() != Some(view.storage_instance()) + || !enrolled.matches_peer(view.peer()) + { + return Err(TransportError::StalePeer); + } + Ok(view.peer().clone()) + } + /// Routing may omit a permanently quarantined member. Its Raft vote and /// physical placement still belong to the durable configuration until a /// quorum commits removal. Missing discovery alone is never sufficient. @@ -40,13 +72,17 @@ impl RaftTransport { /// Check a rooted admission against current trust and actual durable /// configuration. Exclusion may continue the exact reserved destination /// boot; it cannot substitute another process or admit fresh placement. - pub(crate) fn check_fleet_placement( + pub(crate) fn check_fleet_placement( &self, - scope: &GroupScope, + scope: &S, members: &[NodeId], target: &JoiningMember, authorization: &FleetPermitAuthorization, - ) -> Result<(), TransportError> { + ) -> Result<(), TransportError> + where + for<'a> GroupHistory: From<&'a S>, + { + let scope = GroupHistory::from(scope); let state = &self.handle.0; let registry = state.registry.lock().map_err(|_| TransportError::Closed)?; state.admit(®istry)?; @@ -77,7 +113,7 @@ impl RaftTransport { if !destination.permits_transport() || (!destination.eligibility().receive_replicas && !continuing) || destination.boot_binding().ok().as_ref() != Some(original_boot) - || !authorization.matches_scope(scope) + || !authorization.matches_scope::(&scope) || destination.storage_instance().is_none() { return Err(TransportError::StalePeer); @@ -163,9 +199,14 @@ impl RaftTransport { let Registry { origins, inspection_origins, + command_origins, .. } = &mut *registry; - for grant in origins.values_mut().chain(inspection_origins.values_mut()) { + for grant in origins + .values_mut() + .chain(inspection_origins.values_mut()) + .chain(command_origins.values_mut()) + { if grant.peer.as_ref().is_some_and(|peer| !allowed(peer)) { grant.stop.send_replace(true); // A boot supersession cancels this physical owner, but the diff --git a/crates/cellar/durability/raft/transport_frame.rs b/crates/cellar/durability/raft/transport_frame.rs index e9ddbd3ba..343f86000 100644 --- a/crates/cellar/durability/raft/transport_frame.rs +++ b/crates/cellar/durability/raft/transport_frame.rs @@ -8,8 +8,10 @@ use super::transport_options::TransportOptions; use crate::application::consensus::protocol::WireScope; use crate::peer_auth::PeerAuth; use ::raft::prelude::{Message, MessageType}; +use cellar_logic::cell_group::GroupHistory; use cellar_logic::consensus::{ - DiskLineage, GroupScope, MemberId, NodeId, ProcessGeneration, ReplicaIdentity, + ClusterId, DiskLineage, GroupId, IncarnationId, MemberId, NamespaceId, NodeId, + ProcessGeneration, ReplicaIdentity, }; use protobuf::Message as _; use serde::{Deserialize, Serialize}; @@ -31,10 +33,19 @@ struct WirePeer { label: String, } +// Both temporary cell adapters and physical groups use this marker-1 typed +// envelope. There is no reader for the old untagged scope layout. +#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(tag = "kind", content = "identity", deny_unknown_fields)] +enum History { + Cell(WireScope), + Group([[u8; 16]; 4]), +} + #[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] #[serde(deny_unknown_fields)] pub(crate) struct Route { - scope: WireScope, + scope: History, schema: [u8; 32], source: WirePeer, target: WirePeer, @@ -56,7 +67,7 @@ impl Route { return Err(TransportError::Identity); } Ok(Self { - scope: WireScope::from(&source.scope), + scope: History::Cell(WireScope::from(&source.scope)), schema, source: WirePeer::new(source, source_label), target: WirePeer::new(target, target_label), @@ -64,8 +75,10 @@ impl Route { } pub fn decode(&self) -> Result<(ReplicaIdentity, ReplicaIdentity), TransportError> { - let scope = self - .scope + let History::Cell(scope) = &self.scope else { + return Err(TransportError::Identity); + }; + let scope = scope .clone() .decode() .map_err(|_| TransportError::Identity)?; @@ -76,6 +89,71 @@ impl Route { )) } + pub fn new_group( + source: &ReplicaIdentity, + source_label: &str, + target: &ReplicaIdentity, + target_label: &str, + schema: [u8; 32], + ) -> Result { + if source.scope != target.scope { + return Err(TransportError::Identity); + } + let history = source.scope; + Ok(Self { + scope: History::Group([ + history.cluster.bytes(), + history.namespace.bytes(), + history.group.bytes(), + history.incarnation.bytes(), + ]), + schema, + source: WirePeer::new(source, source_label), + target: WirePeer::new(target, target_label), + }) + } + + pub fn is_group(&self) -> bool { + matches!(self.scope, History::Group(_)) + } + + pub fn decode_group( + &self, + ) -> Result<(ReplicaIdentity, ReplicaIdentity), TransportError> + { + let History::Group(ids) = self.scope else { + return Err(TransportError::Identity); + }; + let scope = GroupHistory { + cluster: ClusterId::new(ids[0]).map_err(|_| TransportError::Identity)?, + namespace: NamespaceId::new(ids[1]).map_err(|_| TransportError::Identity)?, + group: GroupId::new(ids[2]).map_err(|_| TransportError::Identity)?, + incarnation: IncarnationId::new(ids[3]).map_err(|_| TransportError::Identity)?, + }; + Ok((self.source.decode(scope)?, self.target.decode(scope)?)) + } + + fn validate(&self) -> Result<(), TransportError> { + if self.is_group() { + self.decode_group()?; + } else { + self.decode()?; + } + Ok(()) + } + + pub fn source_host( + &self, + ) -> Result<(NodeId, DiskLineage, ProcessGeneration, String), TransportError> { + let peer = self.source.decode(())?; + Ok(( + peer.node, + peer.disk, + peer.process, + self.source.label.clone(), + )) + } + pub fn schema(&self) -> [u8; 32] { self.schema } @@ -99,7 +177,7 @@ impl Route { } impl WirePeer { - fn new(replica: &ReplicaIdentity, label: &str) -> Self { + fn new(replica: &ReplicaIdentity, label: &str) -> Self { Self { node: replica.node.bytes(), disk: replica.disk.bytes(), @@ -109,7 +187,7 @@ impl WirePeer { } } - fn decode(&self, scope: GroupScope) -> Result { + fn decode(&self, scope: S) -> Result, TransportError> { crate::machine::validate_node_name(&self.label).map_err(|_| TransportError::Identity)?; Ok(ReplicaIdentity { scope, @@ -159,7 +237,7 @@ impl RequestHeader { return Err(TransportError::Protocol); } self.sequence()?; - self.route.decode()?; + self.route.validate()?; Ok(()) } @@ -179,7 +257,9 @@ impl RequestHeader { // number of decoded Entry objects. scan_message(bytes, options.max_entries)?; let message = Message::parse_from_bytes(bytes).map_err(|_| TransportError::Protocol)?; - let (source, target) = self.route.decode()?; + self.route.validate()?; + let source = self.route.source.decode(())?; + let target = self.route.target.decode(())?; if message.from != source.member.get() || message.to != target.member.get() || classify(&message)? != self.lane @@ -395,7 +475,7 @@ pub(crate) fn classify(message: &Message) -> Result { fn protocol_digest() -> [u8; 32] { Sha256::digest( - b"cellar-raft-peer-1:tls13-exporter:hmac-sha256:protobuf2:bounded-strict:extent-stream", + b"cellar-raft-peer-1:typed-history:tls13-exporter:hmac-sha256:protobuf2:bounded-strict:extent-stream", ) .into() } diff --git a/crates/cellar/durability/raft/transport_frame_tests.rs b/crates/cellar/durability/raft/transport_frame_tests.rs index 587c9fb2f..9d8bcd5bb 100644 --- a/crates/cellar/durability/raft/transport_frame_tests.rs +++ b/crates/cellar/durability/raft/transport_frame_tests.rs @@ -205,3 +205,58 @@ fn strict_header_identity_sequence_and_protocol() { bytes.insert_str(1, "\"sequence\":\"2\","); assert!(serde_json::from_str::(&bytes).is_err()); } + +#[test] +fn physical_history_is_typed_authenticated_and_refuses_old_wire() { + let physical = |n| { + let peer = replica(n); + ReplicaIdentity { + scope: GroupHistory::from(&peer.scope), + member: peer.member, + node: peer.node, + disk: peer.disk, + process: peer.process, + } + }; + let route = Route::new_group(&physical(1), "node-1", &physical(2), "node-2", [5; 32]).unwrap(); + assert!(route.decode().is_err()); + assert_eq!(route.decode_group().unwrap(), (physical(1), physical(2))); + let (mut request, body) = request(); + request.route = route.clone(); + let auth = PeerAuth::new([8; 32], "node-1").unwrap(); + let options = TransportOptions::default(); + let exporter = [7; 32]; + let head = request_head(&auth, &exporter, &request, body.len(), &options).unwrap(); + let decoded = head.request(&auth, &exporter).unwrap(); + assert!(decoded.verify_body(&body, &options).is_ok()); + let response = response_head( + &auth, + &exporter, + &request, + &head.tag, + Disposition::Accepted, + &options, + ) + .unwrap(); + let mut changed = request.clone(); + let mut target = physical(2); + target.scope.incarnation = IncarnationId::new([77; 16]).unwrap(); + let mut source = physical(1); + source.scope = target.scope; + changed.route = Route::new_group(&source, "node-1", &target, "node-2", [5; 32]).unwrap(); + assert_eq!( + response.response(&auth, &exporter, &changed, &head.tag), + Err(TransportError::Authentication) + ); + let mut invalid = route.clone(); + invalid.scope = History::Group([[0; 16]; 4]); + assert!(invalid.validate().is_err()); + let mut old = serde_json::to_value(route).unwrap(); + old["scope"] = serde_json::to_value(WireScope::from(&replica(1).scope)).unwrap(); + assert!(serde_json::from_value::(old).is_err()); + request.protocol = Sha256::digest( + b"cellar-raft-peer-1:tls13-exporter:hmac-sha256:protobuf2:bounded-strict:extent-stream", + ) + .into(); + assert_eq!(request.validate(), Err(TransportError::Protocol)); +} diff --git a/crates/cellar/durability/raft/transport_options.rs b/crates/cellar/durability/raft/transport_options.rs index 29e1f5f77..7f35ec5b3 100644 --- a/crates/cellar/durability/raft/transport_options.rs +++ b/crates/cellar/durability/raft/transport_options.rs @@ -31,6 +31,16 @@ pub struct NamespaceQueryOrigin { pub peer: RaftPeerNode, } +/// Explicit command, enrollment and receipt-inspection permission for an exact namespace/process. +/// Read grants, discovery and Raft membership never create this permission. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct NamespaceCommandOrigin { + pub cluster: ClusterId, + pub namespace: NamespaceId, + pub revision: u64, + pub peer: RaftPeerNode, +} + /// Explicit namespace inspection permission for a physical node process. /// Query grants and directory observations cannot create this permission. #[derive(Clone, Debug, PartialEq, Eq)] @@ -46,6 +56,7 @@ pub struct TransportOptions { pub max_peers: usize, pub max_groups: usize, pub max_query_origins: usize, + pub max_command_origins: usize, pub max_connections_per_lane: usize, pub max_header_bytes: usize, pub max_message_bytes: usize, @@ -77,6 +88,7 @@ impl Default for TransportOptions { max_peers: 128, max_groups: 4096, max_query_origins: 4096, + max_command_origins: 4096, max_connections_per_lane: 128, max_header_bytes: 8192, max_message_bytes: 20 * 1024 * 1024, @@ -129,6 +141,8 @@ impl TransportOptions { || self.max_peers > 4096 || self.max_groups == 0 || self.max_groups > 32_768 + || self.max_command_origins == 0 + || self.max_command_origins > 65_536 || self.max_query_origins == 0 || self.max_query_origins > 65_536 || self.max_connections_per_lane == 0 diff --git a/crates/cellar/durability/raft/transport_pipeline_tests.rs b/crates/cellar/durability/raft/transport_pipeline_tests.rs index d5a145613..798cde7c8 100644 --- a/crates/cellar/durability/raft/transport_pipeline_tests.rs +++ b/crates/cellar/durability/raft/transport_pipeline_tests.rs @@ -597,9 +597,13 @@ async fn control_window_drives_both_directions_with_one_byte_buffers() -> anyhow ); let charge = body.len() + state.options.max_header_bytes + frame::PREFIX + frame::TAG; requests.push(SendRequest { - scope, - route, - peer, + _source: None, + route: SendRoute::Legacy { + scope, + route, + peer: peer.clone(), + }, + member: peer.replica.member, header, body, snapshot: false, @@ -615,7 +619,9 @@ async fn control_window_drives_both_directions_with_one_byte_buffers() -> anyhow let (mut client, mut server) = tokio::io::duplex(1); let exporter = [7; 32]; let (sender, mut inbox) = mailbox::channel(WINDOW); - let pool = state.registry.lock().unwrap().pools[&requests[0].peer.replica.node].clone(); + let pool = state.registry.lock().unwrap().pools + [&requests[0].header.route.decode()?.1.node] + .clone(); for request in requests.drain(..) { sender .try_send(request) diff --git a/crates/cellar/durability/raft/transport_query_tests.rs b/crates/cellar/durability/raft/transport_query_tests.rs new file mode 100644 index 000000000..0f67ff3bd --- /dev/null +++ b/crates/cellar/durability/raft/transport_query_tests.rs @@ -0,0 +1,248 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Actual synced shared engines and transport grants. These authentication +//! observations grant no serving, quorum or logical-cell admission authority. +use super::*; +use crate::raft::storage::CellGroupBootstrap; +use cellar_logic::cell_group::{CellHistory, CellId, CellRange, GroupHistory, RouteEpoch}; + +use super::super::shared_fixture::Fixture; + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn query_grants_are_scoped_revocable_and_never_voting_authority() -> anyhow::Result<()> { + use crate::raft::query_identity::QueryRoute; + let first = GroupHistory { + cluster: ClusterId::new([1; 16])?, + namespace: NamespaceId::new([2; 16])?, + group: GroupId::new([3; 16])?, + incarnation: IncarnationId::new([4; 16])?, + }; + let mut second = first; + second.namespace = NamespaceId::new([5; 16])?; + second.group = GroupId::new([6; 16])?; + let groups = [&first, &second] + .into_iter() + .map(|scope| { + Ok(CellGroupBootstrap { + route: cellar_logic::cell_group::GroupRoute { + history: *scope, + range: CellRange::all(), + epoch: RouteEpoch::new(1)?, + birth_digest: [10; 32], + schema_digest: [9; 32], + }, + member: MemberId::new(1)?, + voters: (1..=3).map(MemberId::new).collect::, _>>()?, + learners: vec![], + }) + }) + .collect::>>()?; + let fixture = Fixture::start( + TransportOptions { + max_query_origins: 2, + ..Default::default() + }, + groups, + ) + .await?; + for scope in [&first, &second] { + let peers = (1..=3) + .map(|member| { + Ok(RaftPeer { + replica: ReplicaIdentity { + scope: *scope, + node: NodeId::new([member as u8; 16])?, + member: MemberId::new(member)?, + disk: DiskLineage::new([member as u8 + 1; 16])?, + process: ProcessGeneration::new([member as u8 + 2; 16])?, + }, + label: format!("node-{member}"), + server_name: "localhost".into(), + control_address: if member == 1 { + fixture.transport.control_address() + } else { + format!("127.0.0.1:{}", 40000 + member * 3).parse()? + }, + data_address: if member == 1 { + fixture.transport.data_address() + } else { + format!("127.0.0.1:{}", 40001 + member * 3).parse()? + }, + query_address: if member == 1 { + fixture.transport.query_address() + } else { + format!("127.0.0.1:{}", 40002 + member * 3).parse()? + }, + }) + }) + .collect::>>()?; + fixture.transport.install_physical_group(GroupRoute { + scope: *scope, + schema_digest: [9; 32], + revision: 1, + peers, + })?; + } + fixture + .domain + .run(async { + let peer = RaftPeerNode { + node: NodeId::new([4; 16])?, + disk: DiskLineage::new([9; 16])?, + process: ProcessGeneration::new([10; 16])?, + label: "node-4".into(), + server_name: "localhost".into(), + control_address: "127.0.0.1:41001".parse()?, + data_address: "127.0.0.1:41002".parse()?, + query_address: "127.0.0.1:41003".parse()?, + }; + let origin = NodeIdentity { + node: peer.node, + disk: peer.disk, + process: peer.process, + }; + let route = |scope: &GroupHistory, source| -> anyhow::Result<_> { + Ok(QueryRoute::new_shared( + source, + "node-4", + fixture.log.cell_group(scope).unwrap().identity(), + "node-1", + &CellHistory { + cluster: scope.cluster, + namespace: scope.namespace, + cell: CellId::new("Counter:one")?, + incarnation: IncarnationId::new([20; 16])?, + }, + [9; 32], + )?) + }; + let first_route = route(&first, origin)?; + let second_route = route(&second, origin)?; + let state = &fixture.transport.handle.0; + assert!(matches!( + state.authorize_query_route(&first_route), + Err(TransportError::Refused) + )); + let mut grant = NamespaceQueryOrigin { + cluster: first.cluster, + namespace: first.namespace, + revision: 1, + peer: peer.clone(), + }; + fixture.transport.grant_query_origin(grant.clone())?; + let (pool, permission) = state.authorize_query_route(&first_route)?; + let permission = permission.expect("nonmember requires an explicit grant"); + assert!(matches!( + state.authorize_query_route(&second_route), + Err(TransportError::Refused) + )); + let mut wrong_process = origin; + wrong_process.process = ProcessGeneration::new([11; 16])?; + assert!(matches!( + state.authorize_query_route(&route(&first, wrong_process)?), + Err(TransportError::Refused) + )); + let mut other = grant.clone(); + other.namespace = second.namespace; + fixture.transport.grant_query_origin(other.clone())?; + let (_, other_permission) = state.authorize_query_route(&second_route)?; + let fake_member = ReplicaIdentity { + scope: first, + node: origin.node, + disk: origin.disk, + process: origin.process, + member: MemberId::new(99)?, + }; + assert!(state + .authorize_physical(&Route::new_group( + &fake_member, + "node-4", + fixture.log.cell_group(&first)?.identity(), + "node-1", + [9; 32] + )?) + .is_err()); + let (_cancel, cancel) = watch::channel(false); + let deadline = state.domain.mono_ms()?.saturating_add(1000); + let waiting = state.query_until( + &pool, + deadline, + cancel.clone(), + Some(permission.clone()), + std::future::pending::>(), + ); + tokio::pin!(waiting); + assert!(futures_util::poll!(&mut waiting).is_pending()); + fixture + .transport + .revoke_query_origin(first.cluster, first.namespace, peer.node, 2)?; + assert!(matches!( + crate::asyncrt::timeout(std::time::Duration::from_secs(1), waiting).await?, + Err(TransportError::Refused) + )); + assert!(*permission.borrow()); + assert!( + !*pool.stop.borrow(), + "another namespace still owns this physical pool" + ); + assert!(!*other_permission.unwrap().borrow()); + assert!(matches!( + fixture.transport.grant_query_origin(grant.clone()), + Err(TransportError::StalePeer) + )); + let mut third = grant.clone(); + third.namespace = NamespaceId::new([7; 16])?; + assert!(matches!( + fixture.transport.grant_query_origin(third), + Err(TransportError::Overloaded) + )); + grant.revision = 3; + fixture.transport.grant_query_origin(grant.clone())?; + assert!(!*state + .authorize_query_route(&first_route)? + .1 + .unwrap() + .borrow()); + assert!( + matches!( + state + .query_until( + &pool, + deadline, + cancel, + Some(permission), + std::future::ready(Ok::<(), TransportError>(())) + ) + .await, + Err(TransportError::Refused) + ), + "regrant cannot revive an old in-flight authorization" + ); + let mut replacement = grant.clone(); + replacement.revision = 4; + replacement.peer.process = wrong_process.process; + assert!( + matches!( + fixture.transport.grant_query_origin(replacement.clone()), + Err(TransportError::StalePeer) + ), + "other namespace still binds the old physical process" + ); + fixture.transport.revoke_query_origin( + second.cluster, + second.namespace, + peer.node, + 2, + )?; + fixture.transport.grant_query_origin(replacement)?; + assert!(*pool.stop.borrow()); + assert!(state.authorize_query_route(&first_route).is_err()); + assert!(state + .authorize_query_route(&route(&first, wrong_process)?) + .is_ok()); + assert_eq!(fixture.log.cell_groups()?.len(), 2); + Ok::<_, anyhow::Error>(()) + }) + .await??; + fixture.close().await +} diff --git a/crates/cellar/durability/raft/transport_resources.rs b/crates/cellar/durability/raft/transport_resources.rs index 9bcef7b48..05c039f86 100644 --- a/crates/cellar/durability/raft/transport_resources.rs +++ b/crates/cellar/durability/raft/transport_resources.rs @@ -100,8 +100,8 @@ impl ResourceObserver { ( registry.tasks, registry.pools.len(), - registry.groups.len(), - registry.query_groups.len(), + registry.groups.len() + registry.physical_groups.len(), + registry.query_groups.len() + registry.cell_query_groups.len(), connections, limits, ) diff --git a/crates/cellar/durability/raft/transport_runtime.rs b/crates/cellar/durability/raft/transport_runtime.rs new file mode 100644 index 000000000..d493007e9 --- /dev/null +++ b/crates/cellar/durability/raft/transport_runtime.rs @@ -0,0 +1,101 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Temporary adapter while the public host cuts over. A shared transport starts +//! directly on its physical runtime; it never constructs a legacy runtime. +use super::*; +use crate::raft::{cell_groups::CellGroupRuntime, storage::ReplicaLog}; + +pub(super) enum TransportRuntime { + Legacy(RaftRuntime), + Shared(CellGroupRuntime), +} + +impl TransportRuntime { + pub(super) fn log(&self) -> &ReplicaLog { + match self { + Self::Legacy(r) => r.log(), + Self::Shared(r) => r.log(), + } + } + + pub(super) fn admit(&self) -> Result<(), RuntimeError> { + match self { + Self::Legacy(r) => r.admit(), + Self::Shared(r) => r.admit(), + } + } + + pub(super) fn claim_transport(&self) -> Result, RuntimeError> { + match self { + Self::Legacy(r) => Ok(Box::new(r.claim_transport()?)), + Self::Shared(r) => Ok(Box::new(r.claim_transport()?)), + } + } + + pub(super) fn shared(&self) -> Result<&CellGroupRuntime, TransportError> { + match self { + Self::Shared(r) => Ok(r), + Self::Legacy(_) => Err(TransportError::Identity), + } + } + + pub(super) async fn admit_route( + &self, + route: &Route, + message: Message, + ) -> Result<(), RuntimeError> { + match self { + Self::Shared(runtime) => { + if message.get_msg_type() == MessageType::MsgSnapshot { + return Err(RuntimeError::Protocol( + "complete-group snapshot transport unavailable".into(), + )); + } + let (source, _) = route + .decode_group() + .map_err(|_| RuntimeError::Protocol("physical route".into()))?; + runtime.admit_step(source.scope, message).await + } + Self::Legacy(runtime) => { + let (source, _) = route + .decode() + .map_err(|_| RuntimeError::Protocol("legacy route".into()))?; + runtime.admit_step(&source.scope, message).await + } + } + } + + pub(super) async fn delivery_hint( + &self, + route: &SendRoute, + member: u64, + delivered: bool, + snapshot: bool, + ) -> Result<(), RuntimeError> { + match (self, route) { + (Self::Legacy(runtime), SendRoute::Legacy { scope, .. }) => { + if !delivered { + runtime.admit_unreachable(scope, member).await?; + } + if snapshot { + runtime + .admit_snapshot_status(scope, member, delivered) + .await?; + } + Ok(()) + } + (Self::Shared(runtime), SendRoute::Physical { route, .. }) => { + if snapshot { + return runtime + .report_snapshot(route.scope, member, delivered) + .await; + } + if !delivered { + runtime.report_unreachable(route.scope, member).await?; + } + Ok(()) + } + _ => Err(RuntimeError::Protocol("transport/runtime identity".into())), + } + } +} diff --git a/crates/cellar/durability/raft/transport_send.rs b/crates/cellar/durability/raft/transport_send.rs index 1de168d61..f3bf5f737 100644 --- a/crates/cellar/durability/raft/transport_send.rs +++ b/crates/cellar/durability/raft/transport_send.rs @@ -182,6 +182,11 @@ impl State { mut request: SendRequest, result: Result<(), TransportError>, ) { + // The frame's physical exchange has settled. Feedback is advisory and + // may wait for an idle Raft owner; retaining its send fence through that + // wait would prevent the configuration which makes the owner idle. + // Keep transport metadata/queue charges until this task actually exits. + drop(request._source.take()); if result.is_err() { self.failed.fetch_add(1, Ordering::Relaxed); if self.current(&request) { @@ -190,17 +195,13 @@ impl State { let _ = self .bounded(Some(pool), async { self.runtime - .admit_unreachable(&request.scope, request.peer.replica.member.get()) + .delivery_hint( + &request.route, + request.member.get(), + false, + request.snapshot, + ) .await?; - if request.snapshot { - self.runtime - .admit_snapshot_status( - &request.scope, - request.peer.replica.member.get(), - false, - ) - .await?; - } Ok(()) }) .await; @@ -212,11 +213,7 @@ impl State { let _ = self .bounded(Some(pool), async { self.runtime - .admit_snapshot_status( - &request.scope, - request.peer.replica.member.get(), - true, - ) + .delivery_hint(&request.route, request.member.get(), true, true) .await?; Ok(()) }) diff --git a/crates/cellar/durability/raft/transport_shared.rs b/crates/cellar/durability/raft/transport_shared.rs new file mode 100644 index 000000000..4ee025be1 --- /dev/null +++ b/crates/cellar/durability/raft/transport_shared.rs @@ -0,0 +1,659 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Physical group routing on the existing node transport's queues and sockets. +use super::*; +use crate::raft::cache::GroupLog; +use crate::raft::cell_groups::CellGroupRuntime; + +type PhysicalRoute = GroupRoute; +type PhysicalPeer = RaftPeer; + +pub(super) enum SendRoute { + Legacy { + scope: GroupScope, + route: Arc, + peer: RaftPeer, + }, + Physical { + route: Arc, + peer: PhysicalPeer, + }, +} + +pub(super) fn permitted( + group: &GroupLog, + peers: &[&ReplicaIdentity], +) -> Result<(), TransportError> { + let cache = group + .read() + .map_err(|e| TransportError::Runtime(e.into()))?; + let conf = &cache.conf_state; + for peer in peers { + if !cache.membership.permits_group(peer) + || !conf + .voters + .iter() + .chain(&conf.voters_outgoing) + .chain(&conf.learners) + .chain(&conf.learners_next) + .any(|member| *member == peer.member.get()) + { + return Err(TransportError::StalePeer); + } + } + Ok(()) +} + +impl RaftTransport { + pub(crate) fn uses_shared_runtime(&self, runtime: &CellGroupRuntime) -> bool { + self.handle + .0 + .runtime + .shared() + .is_ok_and(|owner| owner.same_owner(runtime)) + } + + /// Install already-authorized physical membership. This creates no group, + /// grants no serving authority and changes no committed configuration. + pub fn install_physical_group(&self, route: PhysicalRoute) -> Result<(), TransportError> { + self.install_physical(route, false, None).map(|_| ()) + } + + pub(crate) fn register_physical_group( + &self, + route: PhysicalRoute, + ) -> Result { + let (route, owner) = self.install_physical(route, true, None)?; + Ok(PhysicalRegistration { + state: Arc::downgrade(&self.handle.0), + route, + owner, + checkpoint: None, + }) + } + + pub(super) fn install_physical( + &self, + route: PhysicalRoute, + exclusive: bool, + refresh: Option>, + ) -> Result<(Arc, Arc<()>), TransportError> { + let state = &self.handle.0; + let runtime = state.runtime.shared()?; + let group = runtime + .log() + .cell_group(&route.scope) + .map_err(|_| TransportError::UnknownGroup)?; + if route.revision == 0 || route.peers.is_empty() || route.peers.len() > 128 { + return Err(TransportError::Invalid("physical group route".into())); + } + let cache = group + .read() + .map_err(|e| TransportError::Runtime(e.into()))?; + if cache + .group_manifest + .as_ref() + .is_none_or(|root| root.route().schema_digest != route.schema_digest) + { + return Err(TransportError::Identity); + } + drop(cache); + let local = group.identity(); + let mut members = BTreeSet::new(); + let mut nodes = BTreeSet::new(); + for peer in &route.peers { + if peer.replica.scope != route.scope + || !members.insert(peer.replica.member) + || !nodes.insert(peer.replica.node) + { + return Err(TransportError::Identity); + } + RaftPeerNode::from(peer).validate_endpoint()?; + permitted(&group, &[&peer.replica])?; + } + let own = route + .peers + .iter() + .find(|p| p.replica == *local) + .ok_or(TransportError::Identity)?; + if own.label != state.auth.source() + || own.control_address != state.local_control + || own.data_address != state.local_data + || own.query_address != state.local_query + { + return Err(TransportError::Identity); + } + let mut registry = state.registry.lock().map_err(|_| TransportError::Closed)?; + state.admit(®istry)?; + if let Some(expected) = &refresh { + if !registry + .physical_owners + .get(&route.scope) + .is_some_and(|owner| Arc::ptr_eq(owner, expected)) + { + return Err(TransportError::StalePeer); + } + } + for peer in &route.peers { + registry.check_node_view(&RaftPeerNode::from(peer))?; + } + if registry.cell_query_groups.contains_key(&route.scope) { + return Err(TransportError::Identity); + } + if let Some(old) = registry.physical_groups.get(&route.scope) { + if exclusive { + return Err(TransportError::Identity); + } + if **old == route { + let owner = registry + .physical_owners + .get(&route.scope) + .ok_or(TransportError::Closed)?; + return Ok((old.clone(), owner.clone())); + } + if old.revision >= route.revision || old.schema_digest != route.schema_digest { + return Err(TransportError::StalePeer); + } + for peer in &route.peers { + if old.peers.iter().any(|p| { + p.replica.member == peer.replica.member + && (p.replica.node != peer.replica.node + || p.replica.disk != peer.replica.disk) + }) { + return Err(TransportError::Identity); + } + } + } else if registry.route_count() >= state.options.max_groups { + return Err(TransportError::Overloaded); + } + let physical: Vec<_> = route + .peers + .iter() + .filter(|p| p.replica.node != local.node) + .map(RaftPeerNode::from) + .collect(); + for peer in &physical { + if registry + .physical_groups + .values() + .chain(registry.cell_query_groups.values()) + .filter(|view| view.scope != route.scope) + .flat_map(|view| &view.peers) + .any(|other| other.replica.node == peer.node && !peer.same_host(other)) + || registry + .groups + .values() + .chain(registry.query_groups.values()) + .flat_map(|view| &view.peers) + .any(|other| other.replica.node == peer.node && !peer.same_host(other)) + || registry + .origins + .values() + .chain(registry.inspection_origins.values()) + .filter_map(|g| g.peer.as_ref()) + .any(|other| other.node == peer.node && other != peer) + { + return Err(TransportError::StalePeer); + } + } + let starts = state.prepare_pools(&mut registry, &physical)?; + let route = Arc::new(route); + if refresh.is_none() { + group_snapshots::revoke(&mut registry, &route.scope); + } + let owner = refresh.unwrap_or_else(|| Arc::new(())); + registry.physical_owners.insert(route.scope, owner.clone()); + registry.physical_groups.insert(route.scope, route.clone()); + registry.prune_pools(); + drop(registry); + if let Err(error) = state.start_pools(starts) { + if exclusive { + drop(PhysicalRegistration { + state: Arc::downgrade(state), + route, + owner, + checkpoint: None, + }); + } + return Err(error); + } + Ok((route, owner)) + } + + /// Refresh only advisory managed peer routes. Committed membership supplies + /// member/disk bindings; a current rooted node view supplies the endpoint. + /// Missing discovery keeps known voters usable and never removes Raft debt. + pub(crate) fn reconcile_managed_physical( + &self, + history: &GroupHistory, + ) -> Result<(), TransportError> { + let state = &self.handle.0; + let runtime = state.runtime.shared()?; + let (old, owner, has_isolated_peer) = { + let registry = state.registry.lock().map_err(|_| TransportError::Closed)?; + state.admit(®istry)?; + let old = registry + .physical_groups + .get(history) + .cloned() + .ok_or(TransportError::UnknownGroup)?; + let owner = registry + .physical_owners + .get(history) + .cloned() + .ok_or(TransportError::UnknownGroup)?; + let has_isolated_peer = registry.fleet_trust.as_ref().is_some_and(|trust| { + old.peers.iter().any(|peer| { + trust + .node(peer.replica.node) + .is_some_and(|node| node.is_isolated()) + }) + }); + (old, owner, has_isolated_peer) + }; + let group = runtime + .log() + .cell_group(history) + .map_err(|_| TransportError::UnknownGroup)?; + // Never hold a group read guard while taking the registry lock. A queued + // writer can otherwise deadlock against registry->group authorization. + let (binding, members) = { + let cache = group + .read() + .map_err(|e| TransportError::Runtime(e.into()))?; + let Some(fleet) = cache.membership.fleet_status() else { + return Ok(()); + }; + let current = crate::raft::member_state::members(&cache.conf_state); + if !has_isolated_peer + && current.len() == old.peers.len() + && old.peers.iter().all(|peer| { + current.contains(&peer.replica.member.get()) + && cache.membership.permits_group(&peer.replica) + }) + { + return Ok(()); + } + if !current.contains(&group.identity().member.get()) { + return Err(TransportError::StalePeer); + } + let members = current + .into_iter() + .map(|id| { + let member = cellar_logic::consensus::MemberId::new(id) + .map_err(|_| TransportError::Identity)?; + let joined = cache + .membership + .joined(member) + .map_err(|e| TransportError::Runtime(e.into()))?; + Ok((member, joined.map(|(binding, _)| binding))) + }) + .collect::, TransportError>>()?; + (fleet.binding, members) + }; + let peers = { + let registry = state.registry.lock().map_err(|_| TransportError::Closed)?; + state.admit(®istry)?; + if !registry + .physical_owners + .get(history) + .is_some_and(|current| Arc::ptr_eq(current, &owner)) + { + return Err(TransportError::StalePeer); + } + // An advisory view by itself cannot expand a managed route. Until + // trust arrives, leave existing routing permissions unchanged. + let Some(trust) = ®istry.fleet_trust else { + return Ok(()); + }; + if trust.fleet() != binding.fleet() + || trust.catalog() != binding.catalog() + || trust.cluster() != history.cluster + { + return Err(TransportError::Refused); + } + let mut peers = Vec::with_capacity(members.len()); + for (member, joined) in members { + let prior = old.peers.iter().find(|peer| peer.replica.member == member); + let peer = if let Some(joined) = joined { + let Some(view) = registry.node_views.get(&joined.node) else { + continue; + }; + let node = view.peer(); + if node.disk != joined.disk || registry.check_node_view(node).is_err() { + continue; + } + RaftPeer { + replica: ReplicaIdentity { + scope: *history, + member, + node: node.node, + disk: node.disk, + process: node.process, + }, + label: node.label.clone(), + server_name: node.server_name.clone(), + control_address: node.control_address, + data_address: node.data_address, + query_address: node.query_address, + } + } else { + // Initial member identities come from the authorized birth + // route, never from guessing an ID from node discovery. + let Some(prior) = prior else { + continue; + }; + prior.clone() + }; + // Only rooted permanent quarantine can omit an initial voter + // whose retained endpoint would otherwise reject the whole + // refreshed route, including a newly admitted learner. Keep + // its durable membership and seed-owner/resource debt intact. + if peer.replica.node != group.identity().node + && trust + .node(peer.replica.node) + .is_some_and(|node| node.is_isolated()) + { + continue; + } + peers.push(peer); + } + peers + }; + if peers == old.peers { + return Ok(()); + } + let revision = old + .revision + .checked_add(1) + .ok_or(TransportError::StalePeer)?; + self.install_physical( + PhysicalRoute { + scope: *history, + schema_digest: old.schema_digest, + revision, + peers, + }, + false, + Some(owner), + )?; + Ok(()) + } + + pub fn remove_physical_group(&self, history: &GroupHistory) -> Result<(), TransportError> { + let state = &self.handle.0; + state.runtime.shared()?; + let mut registry = state.registry.lock().map_err(|_| TransportError::Closed)?; + state.admit(®istry)?; + group_snapshots::revoke(&mut registry, history); + registry.physical_groups.remove(history); + registry.physical_owners.remove(history); + registry.prune_pools(); + Ok(()) + } + + /// Queue a physical-group frame on existing bounded per-node lanes. Delivery + /// acknowledges receiver admission only. Snapshots use the bounded checkpoint stream. + pub fn submit_physical( + &self, + history: &GroupHistory, + message: Message, + ) -> Result { + self.submit_physical_inner(history, message, true, None) + } + + /// Retain the runtime's exact prior-protocol send admission through physical + /// transport completion, independently of the Delivery response waiter. + pub fn submit_cell_group_event( + &self, + event: &crate::raft::cell_groups::CellGroupRuntimeEvent, + ) -> Result { + let crate::raft::cell_groups::CellGroupEffect::Message(message) = event + .effect() + .map_err(|e| TransportError::Runtime(e.clone()))? + else { + return Err(TransportError::Protocol); + }; + self.submit_physical_inner( + &event.group(), + message.clone(), + true, + Some(event.delivery_admission()), + ) + } + + fn submit_physical_inner( + &self, + history: &GroupHistory, + message: Message, + discover: bool, + source: Option>, + ) -> Result { + let state = &self.handle.0; + let runtime = state.runtime.shared()?; + if message.get_msg_type() == MessageType::MsgSnapshot { + self.reconcile_managed_physical(history)?; + return self.submit_group_checkpoint(history, message, source); + } + let lane = frame::classify(&message)?; + let size = message.compute_size() as usize; + if size > frame::payload_limit(lane, &state.options) { + return Err(TransportError::Limit); + } + let registry = state.registry.lock().map_err(|_| TransportError::Closed)?; + state.admit(®istry)?; + let route = registry + .physical_groups + .get(history) + .ok_or(TransportError::UnknownGroup)? + .clone(); + let group = runtime + .log() + .cell_group(history) + .map_err(|_| TransportError::UnknownGroup)?; + let local = group.identity().clone(); + if message.from != local.member.get() { + return Err(TransportError::Identity); + } + let peer = match route + .peers + .iter() + .find(|p| p.replica.member.get() == message.to) + { + Some(peer) => peer.clone(), + None if discover => { + // Only a missing route needs discovery on ordinary traffic. + // Retry once, before allocating/charging a serialized frame. + drop(registry); + self.reconcile_managed_physical(history)?; + return self.submit_physical_inner(history, message, false, source); + } + None => return Err(TransportError::StalePeer), + }; + if peer.replica.node == local.node { + return Err(TransportError::Identity); + } + permitted(&group, &[&local, &peer.replica])?; + let pool = registry + .pools + .get(&peer.replica.node) + .ok_or(TransportError::StalePeer)? + .clone(); + drop(registry); + let charge = size + state.options.max_header_bytes + frame::PREFIX + frame::TAG; + let node = state + .outgoing + .lane(lane) + .acquire(charge) + .ok_or(TransportError::Overloaded)?; + let peer_permit = pool + .outgoing + .lane(lane) + .acquire(charge) + .ok_or(TransportError::Overloaded)?; + let body = message + .write_to_bytes() + .map_err(|_| TransportError::Protocol)?; + frame::scan_message(&body, state.options.max_entries)?; + let wire = Route::new_group( + &local, + state.auth.source(), + &peer.replica, + &peer.label, + route.schema_digest, + )?; + let mut header = RequestHeader::new(wire, 1, lane, &body); + let registry = state.registry.lock().map_err(|_| TransportError::Closed)?; + state.admit(®istry)?; + if !registry + .physical_groups + .get(history) + .is_some_and(|r| Arc::ptr_eq(r, &route)) + || !registry + .pools + .get(&peer.replica.node) + .is_some_and(|p| Arc::ptr_eq(p, &pool)) + { + return Err(TransportError::StalePeer); + } + let sequence = state + .sequence + .fetch_update(Ordering::AcqRel, Ordering::Acquire, |n| n.checked_add(1)) + .map_err(|_| TransportError::Closed)? + + 1; + header.set_sequence(sequence); + let (result, wait) = oneshot::channel(); + let request = SendRequest { + _source: source, + member: peer.replica.member, + route: SendRoute::Physical { route, peer }, + header, + body, + snapshot: false, + result: Some(result), + queued: Some(state.performance.timer(match lane { + Lane::Control => Operation::ControlQueue, + Lane::Data => Operation::DataQueue, + })), + _node: node, + _peer: peer_permit, + }; + pool.tx[lane_index(lane)].try_send(request).map_err(|e| { + let error = match &e { + mpsc::error::TrySendError::Full(_) => TransportError::Overloaded, + mpsc::error::TrySendError::Closed(_) => TransportError::Closed, + }; + if let Some(timer) = e.into_inner().queued.take() { + timer.finish(false); + } + error + })?; + Ok(Delivery { result: wait }) + } +} + +impl State { + pub(super) fn current_physical(&self, route: &Arc, peer: &PhysicalPeer) -> bool { + self.registry.lock().is_ok_and(|r| { + self.admit(&r).is_ok() + && r.physical_groups + .get(&route.scope) + .is_some_and(|current| Arc::ptr_eq(current, route)) + && r.pools + .get(&peer.replica.node) + .is_some_and(|pool| pool.peer.same_host(peer)) + && self.runtime.shared().is_ok_and(|runtime| { + runtime.log().cell_group(&route.scope).is_ok_and(|group| { + permitted(&group, &[group.identity(), &peer.replica]).is_ok() + }) + }) + }) + } + + pub(super) fn authorize_physical(&self, wire: &Route) -> Result, TransportError> { + let runtime = self.runtime.shared()?; + let (source, target) = wire.decode_group()?; + let registry = self.registry.lock().map_err(|_| TransportError::Closed)?; + self.admit(®istry)?; + let route = registry + .physical_groups + .get(&target.scope) + .ok_or(TransportError::UnknownGroup)?; + let group = runtime + .log() + .cell_group(&target.scope) + .map_err(|_| TransportError::UnknownGroup)?; + if *group.identity() != target + || wire.target_label() != self.auth.source() + || wire.schema() != route.schema_digest + { + return Err(TransportError::StalePeer); + } + let peer = route + .peers + .iter() + .find(|p| p.replica == source && p.label == wire.source_label()) + .ok_or(TransportError::StalePeer)?; + permitted(&group, &[&source, &target])?; + registry + .pools + .get(&source.node) + .filter(|pool| pool.peer.same_host(peer)) + .cloned() + .ok_or(TransportError::StalePeer) + } +} + +impl Registry { + pub(super) fn check_physical_endpoint( + &self, + peer: &RaftPeerNode, + ) -> Result<(), TransportError> { + if self + .command_origins + .values() + .filter_map(|grant| grant.peer.as_ref()) + .any(|old| old.node == peer.node && old != peer) + { + return Err(TransportError::StalePeer); + } + if self + .physical_groups + .values() + .chain(self.cell_query_groups.values()) + .flat_map(|route| &route.peers) + .any(|other| other.replica.node == peer.node && !peer.same_host(other)) + { + return Err(TransportError::StalePeer); + } + Ok(()) + } +} + +/// Removing an old registration must never remove a replacement route. +pub(crate) struct PhysicalRegistration { + state: std::sync::Weak, + route: Arc, + owner: Arc<()>, + pub(crate) checkpoint: Option, +} + +impl Drop for PhysicalRegistration { + fn drop(&mut self) { + let Some(state) = self.state.upgrade() else { + return; + }; + if let Ok(mut registry) = state.registry.lock() { + if registry + .physical_owners + .get(&self.route.scope) + .is_some_and(|owner| Arc::ptr_eq(owner, &self.owner)) + { + group_snapshots::revoke(&mut registry, &self.route.scope); + registry.physical_groups.remove(&self.route.scope); + registry.physical_owners.remove(&self.route.scope); + registry.prune_pools(); + } + }; + } +} diff --git a/crates/cellar/durability/raft/transport_shared_fixture.rs b/crates/cellar/durability/raft/transport_shared_fixture.rs new file mode 100644 index 000000000..bd8506f5f --- /dev/null +++ b/crates/cellar/durability/raft/transport_shared_fixture.rs @@ -0,0 +1,85 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Synced shared engines and real TLS listeners for transport ownership tests. +//! Groups are parked: these fixtures grant no quorum or serving authority. +use super::*; +use crate::raft::{ + cell_groups::{CellGroupRuntime, CellGroupRuntimeEvents}, + options::{LogOpenMode, RaftRuntimeOptions, ReplicaLogOptions}, + storage::{CellGroupBootstrap, ReplicaLog}, +}; +use cellar_logic::consensus::{DiskLineage, ProcessGeneration}; + +pub(super) struct Fixture { + pub(super) domain: crate::asyncrt::ExecutionDomain, + pub(super) log: ReplicaLog, + pub(super) runtime: CellGroupRuntime, + pub(super) transport: RaftTransport, + _events: CellGroupRuntimeEvents, + _directory: tempfile::TempDir, +} + +impl Fixture { + pub(super) async fn start( + options: TransportOptions, + groups: Vec, + ) -> anyhow::Result { + let directory = tempfile::tempdir()?; + let domain = crate::asyncrt::ExecutionDomain::new(tokio::runtime::Handle::current()); + let log = domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + directory.path().join("log"), + NodeId::new([1; 16])?, + DiskLineage::new([2; 16])?, + ProcessGeneration::new([3; 16])?, + LogOpenMode::Create, + ), + Arc::new(cellar_ltx::DirectFileSystem), + )) + .await??; + for group in groups { + log.create_cell_group(group).await?; + } + let (runtime, events) = + CellGroupRuntime::start(log.clone(), RaftRuntimeOptions::default()).await?; + let tls = RaftTls::new( + vec![include_bytes!("../../transport/tcp/fixtures/ca.der") + .to_vec() + .into()], + vec![include_bytes!("../../transport/tcp/fixtures/server.der") + .to_vec() + .into()], + rustls::pki_types::PrivatePkcs8KeyDer::from( + include_bytes!("../../transport/tcp/fixtures/key.der").to_vec(), + ) + .into(), + )?; + let transport = RaftTransport::start_shared( + runtime.clone(), + Arc::new(PeerAuth::new([8; 32], "node-1")?), + tls, + TcpListener::bind("127.0.0.1:0").await?, + TcpListener::bind("127.0.0.1:0").await?, + TcpListener::bind("127.0.0.1:0").await?, + options, + ) + .await?; + Ok(Self { + domain, + log, + runtime, + transport, + _events: events, + _directory: directory, + }) + } + + pub(super) async fn close(self) -> anyhow::Result<()> { + self.transport.close().await; + self.runtime.close().await?; + self.log.close().await?; + self.domain.shutdown().await; + Ok(()) + } +} diff --git a/crates/cellar/durability/raft/transport_tests.rs b/crates/cellar/durability/raft/transport_tests.rs index 3f0e20c25..3e1bf9d09 100644 --- a/crates/cellar/durability/raft/transport_tests.rs +++ b/crates/cellar/durability/raft/transport_tests.rs @@ -7,6 +7,8 @@ mod directory; mod pipeline; #[path = "transport_pipeline_regression_tests.rs"] mod pipeline_regressions; +#[path = "transport_query_tests.rs"] +mod queries; use crate::raft::options::{LogOpenMode, RaftRuntimeOptions, ReplicaLogOptions}; use crate::raft::storage::ReplicaLog; use cellar_logic::consensus::{DiskLineage, ProcessGeneration}; @@ -316,229 +318,88 @@ async fn poisoned_registry_still_waits_for_resource_owners() -> anyhow::Result<( fixture.close().await } -#[tokio::test(flavor = "multi_thread", worker_threads = 4)] -async fn query_grants_are_scoped_revocable_and_never_voting_authority() -> anyhow::Result<()> { - use crate::raft::{query_identity::QueryRoute, storage::GroupBootstrap}; - let first = GroupScope { - cluster: ClusterId::new([1; 16])?, - namespace: NamespaceId::new([2; 16])?, - cell: "Counter:one".into(), - group: GroupId::new([3; 16])?, - incarnation: IncarnationId::new([4; 16])?, - }; - let mut second = first.clone(); - second.namespace = NamespaceId::new([5; 16])?; - second.group = GroupId::new([6; 16])?; - let groups = [&first, &second] - .into_iter() - .map(|scope| { - Ok(GroupBootstrap { - scope: scope.clone(), - member: MemberId::new(1)?, - birth_digest: [10; 32], - voters: (1..=3).map(MemberId::new).collect::, _>>()?, - learners: vec![], - }) - }) - .collect::>>()?; +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn command_grants_bound_tombstones_without_changing_read_permission() -> anyhow::Result<()> { + use crate::raft::transport_options::NamespaceCommandOrigin; let fixture = Fixture::start( TransportOptions { - max_query_origins: 2, + max_command_origins: 2, ..Default::default() }, - groups, + vec![], ) .await?; - for scope in [&first, &second] { - let peers = (1..=3) - .map(|member| { - Ok(RaftPeer { - replica: ReplicaIdentity { - scope: scope.clone(), - node: NodeId::new([member as u8; 16])?, - member: MemberId::new(member)?, - disk: DiskLineage::new([member as u8 + 1; 16])?, - process: ProcessGeneration::new([member as u8 + 2; 16])?, - }, - label: format!("node-{member}"), - server_name: "localhost".into(), - control_address: if member == 1 { - fixture.transport.control_address() - } else { - format!("127.0.0.1:{}", 40000 + member * 3).parse()? - }, - data_address: if member == 1 { - fixture.transport.data_address() - } else { - format!("127.0.0.1:{}", 40001 + member * 3).parse()? - }, - query_address: if member == 1 { - fixture.transport.query_address() - } else { - format!("127.0.0.1:{}", 40002 + member * 3).parse()? - }, - }) - }) - .collect::>>()?; - fixture.transport.install_group(GroupRoute { - scope: scope.clone(), - schema_digest: [9; 32], - revision: 1, - peers, - })?; - } fixture .domain .run(async { let peer = RaftPeerNode { node: NodeId::new([4; 16])?, - disk: DiskLineage::new([9; 16])?, - process: ProcessGeneration::new([10; 16])?, + disk: DiskLineage::new([5; 16])?, + process: ProcessGeneration::new([6; 16])?, label: "node-4".into(), server_name: "localhost".into(), - control_address: "127.0.0.1:41001".parse()?, - data_address: "127.0.0.1:41002".parse()?, - query_address: "127.0.0.1:41003".parse()?, - }; - let origin = NodeIdentity { - node: peer.node, - disk: peer.disk, - process: peer.process, + control_address: "127.0.0.1:41000".parse()?, + data_address: "127.0.0.1:41001".parse()?, + query_address: "127.0.0.1:41002".parse()?, }; - let route = |scope: &GroupScope, source| { - QueryRoute::new( - source, - "node-4", - fixture.log.group(scope).unwrap().identity(), - "node-1", - [9; 32], - ) - }; - let first_route = route(&first, origin)?; - let second_route = route(&second, origin)?; - let state = &fixture.transport.handle.0; - assert!(matches!( - state.authorize_query_route(&first_route), - Err(TransportError::Refused) - )); - let mut grant = NamespaceQueryOrigin { - cluster: first.cluster, - namespace: first.namespace, + let grant = NamespaceCommandOrigin { + cluster: ClusterId::new([1; 16])?, + namespace: NamespaceId::new([2; 16])?, revision: 1, peer: peer.clone(), }; - fixture.transport.grant_query_origin(grant.clone())?; - let (pool, permission) = state.authorize_query_route(&first_route)?; - let permission = permission.expect("nonmember requires an explicit grant"); - assert!(matches!( - state.authorize_query_route(&second_route), - Err(TransportError::Refused) - )); - let mut wrong_process = origin; - wrong_process.process = ProcessGeneration::new([11; 16])?; - assert!(matches!( - state.authorize_query_route(&route(&first, wrong_process)?), - Err(TransportError::Refused) - )); - let mut other = grant.clone(); - other.namespace = second.namespace; - fixture.transport.grant_query_origin(other.clone())?; - let (_, other_permission) = state.authorize_query_route(&second_route)?; - let fake_member = ReplicaIdentity { - scope: first.clone(), - node: origin.node, - disk: origin.disk, - process: origin.process, - member: MemberId::new(99)?, + let key = (grant.cluster, grant.namespace, peer.node); + let transport = &fixture.transport; + transport.grant_command_origin(grant.clone())?; + transport.grant_command_origin(grant.clone())?; + let old = { + let registry = transport.handle.0.registry.lock().unwrap(); + assert!(registry.origins.is_empty()); + assert_eq!(registry.command_origins.len(), 1); + registry.command_origins[&key].clone() }; - assert!(state - .authorize_route(&Route::new( - &fake_member, - "node-4", - fixture.log.group(&first)?.identity(), - "node-1", - [9; 32] - )?) - .is_err()); - let (_cancel, cancel) = watch::channel(false); - let deadline = state.domain.mono_ms()?.saturating_add(1000); - let waiting = state.query_until( - &pool, - deadline, - cancel.clone(), - Some(permission.clone()), - std::future::pending::>(), - ); - tokio::pin!(waiting); - assert!(futures_util::poll!(&mut waiting).is_pending()); - fixture - .transport - .revoke_query_origin(first.cluster, first.namespace, peer.node, 2)?; - assert!(matches!( - crate::asyncrt::timeout(std::time::Duration::from_secs(1), waiting).await?, - Err(TransportError::Refused) - )); - assert!(*permission.borrow()); - assert!( - !*pool.stop.borrow(), - "another namespace still owns this physical pool" + assert_eq!(transport.usage().peers, 1); + let mut foreign = grant.clone(); + foreign.namespace = NamespaceId::new([3; 16])?; + transport.grant_command_origin(foreign.clone())?; + let mut full = grant.clone(); + full.namespace = NamespaceId::new([7; 16])?; + assert_eq!( + transport.grant_command_origin(full.clone()), + Err(TransportError::Overloaded) ); - assert!(!*other_permission.unwrap().borrow()); - assert!(matches!( - fixture.transport.grant_query_origin(grant.clone()), + transport.revoke_command_origin(grant.cluster, grant.namespace, peer.node, 2)?; + assert!(*old.stop.borrow()); + assert_eq!( + transport.grant_command_origin(grant.clone()), Err(TransportError::StalePeer) - )); - let mut third = grant.clone(); - third.namespace = NamespaceId::new([7; 16])?; - assert!(matches!( - fixture.transport.grant_query_origin(third), - Err(TransportError::Overloaded) - )); - grant.revision = 3; - fixture.transport.grant_query_origin(grant.clone())?; - assert!(!*state - .authorize_query_route(&first_route)? - .1 - .unwrap() - .borrow()); - assert!( - matches!( - state - .query_until( - &pool, - deadline, - cancel, - Some(permission), - std::future::ready(Ok::<(), TransportError>(())) - ) - .await, - Err(TransportError::Refused) - ), - "regrant cannot revive an old in-flight authorization" ); - let mut replacement = grant.clone(); - replacement.revision = 4; - replacement.peer.process = wrong_process.process; - assert!( - matches!( - fixture.transport.grant_query_origin(replacement.clone()), - Err(TransportError::StalePeer) - ), - "other namespace still binds the old physical process" + assert_eq!( + transport.revoke_command_origin(full.cluster, full.namespace, peer.node, 2), + Err(TransportError::Overloaded) ); - fixture.transport.revoke_query_origin( - second.cluster, - second.namespace, - peer.node, - 2, - )?; - fixture.transport.grant_query_origin(replacement)?; - assert!(*pool.stop.borrow()); - assert!(state.authorize_query_route(&first_route).is_err()); - assert!(state - .authorize_query_route(&route(&first, wrong_process)?) - .is_ok()); - assert_eq!(fixture.log.groups()?.len(), 2); + // A read grant neither removes the write tombstone nor grants writes. + transport.grant_query_origin(NamespaceQueryOrigin { + cluster: grant.cluster, + namespace: grant.namespace, + revision: 1, + peer: peer.clone(), + })?; + { + let registry = transport.handle.0.registry.lock().unwrap(); + assert_eq!(registry.command_origins.len(), 2); + assert!(registry.command_origins[&key].peer.is_none()); + assert!(registry.origins[&key].peer.is_some()); + } + let mut restored = grant.clone(); + restored.revision = 3; + transport.grant_command_origin(restored)?; + transport.revoke_command_origin(grant.cluster, grant.namespace, peer.node, 4)?; + transport.revoke_command_origin(foreign.cluster, foreign.namespace, peer.node, 2)?; + assert_eq!(transport.usage().peers, 1); // Independent read grant retains its pool. + transport.revoke_query_origin(grant.cluster, grant.namespace, peer.node, 2)?; + assert_eq!(transport.usage().peers, 0); + assert!(fixture.log.groups()?.is_empty()); Ok::<_, anyhow::Error>(()) }) .await??; diff --git a/crates/cellar/durability/raft/version.rs b/crates/cellar/durability/raft/version.rs index 22e680fc6..49312c9d3 100644 --- a/crates/cellar/durability/raft/version.rs +++ b/crates/cellar/durability/raft/version.rs @@ -4,6 +4,10 @@ //! and contain no leader identity: a normal election preserves their meaning. use base64::{engine::general_purpose::URL_SAFE_NO_PAD, Engine as _}; +use cellar_logic::cell_group::{ + CellHistory, CellId, CellImage, CellImageCommitment, CellVersion, CommitWitness, GroupHistory, + MutationSequence, +}; use cellar_logic::consensus::*; use ed25519_dalek::{Signature, Signer, SigningKey, VerifyingKey}; use sha2::{Digest, Sha256}; @@ -15,6 +19,55 @@ const MAX_BYTES: usize = 1408; const MAX_ENCODED: usize = 1878; const MAX_KEYS: usize = 64; const MAX_LIFETIME_MS: u64 = 7 * 24 * 60 * 60 * 1000; +const CELL_MAGIC: &[u8; 8] = b"CLRCVS01"; +const CELL_DOMAIN: &[u8] = b"cellar-cell-group-version-1"; + +// Framing/key/time, cell identity, sequence/image, schema, witness and signature. +const CELL_FIXED_BYTES: usize = 8 + 32 + 16 + 48 + 2 + 8 + 24 + 37 + 32 + 64 + 16 + 16 + 64; +const CELL_MAX_BYTES: usize = CELL_FIXED_BYTES + cellar_logic::cell_group::MAX_CELL_ID_BYTES; +const CELL_MAX_ENCODED: usize = CELL_MAX_BYTES.div_ceil(3) * 4; + +/// Cell-scoped token for the group runtime. Old group-scoped tokens cannot be +/// interpreted in this domain. The original witness is provenance, not routing. +#[derive(Clone, PartialEq, Eq)] +pub struct CellVersionToken(String); + +impl CellVersionToken { + pub fn from_encoded(encoded: &str) -> Result { + if encoded.len() > CELL_MAX_ENCODED { + return Err(VersionError::Limit); + } + let bytes = URL_SAFE_NO_PAD + .decode(encoded) + .map_err(|_| VersionError::Invalid)?; + if bytes.len() < CELL_FIXED_BYTES + 1 + || bytes.len() > CELL_MAX_BYTES + || !bytes.starts_with(CELL_MAGIC) + || URL_SAFE_NO_PAD.encode(&bytes) != encoded + { + return Err(VersionError::Invalid); + } + Ok(Self(encoded.into())) + } + + pub fn as_str(&self) -> &str { + &self.0 + } +} + +impl fmt::Debug for CellVersionToken { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_struct("CellVersionToken") + .field("bytes", &self.0.len()) + .finish() + } +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct CellVersionObservation { + pub version: CellVersion, + pub witness: CommitWitness, +} #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub enum VersionError { @@ -170,6 +223,187 @@ impl fmt::Debug for VersionKeys { } impl VersionKeys { + pub fn issue_cell( + &self, + observation: &CellVersionObservation, + schema: [u8; 32], + now_ms: u64, + ) -> Result { + let version = &observation.version; + let witness = &observation.witness; + witness + .validate(&version.history) + .map_err(|_| VersionError::Invalid)?; + crate::application::consensus::LtxImage::try_from(version.image) + .map_err(|_| VersionError::Invalid)?; + self.validate_retention(&[], now_ms)?; + let mut bytes = Vec::with_capacity(CELL_FIXED_BYTES + version.history.cell.as_str().len()); + bytes.extend_from_slice(CELL_MAGIC); + bytes.extend_from_slice(&self.0.active.verifier.id()); + bytes.extend_from_slice(&now_ms.to_be_bytes()); + bytes.extend_from_slice(&(now_ms + self.0.lifetime_ms).to_be_bytes()); + for id in [ + version.history.cluster.bytes(), + version.history.namespace.bytes(), + version.history.incarnation.bytes(), + ] { + bytes.extend_from_slice(&id); + } + let cell = version.history.cell.as_str().as_bytes(); + bytes.extend_from_slice(&(cell.len() as u16).to_be_bytes()); + bytes.extend_from_slice(cell); + bytes.extend_from_slice(&version.sequence.get().to_be_bytes()); + for value in [ + version.image.epoch, + version.image.txid, + version.image.checksum, + ] { + bytes.extend_from_slice(&value.to_be_bytes()); + } + let (tag, digest, length) = match version.image.commitment { + CellImageCommitment::Materialized(digest) => (1, digest, 0), + CellImageCommitment::Extents { + root, + manifest_bytes, + } => (2, root, manifest_bytes), + }; + bytes.push(tag); + bytes.extend_from_slice(&digest); + bytes.extend_from_slice(&length.to_be_bytes()); + bytes.extend_from_slice(&schema); + for id in [ + witness.group.cluster.bytes(), + witness.group.namespace.bytes(), + witness.group.group.bytes(), + witness.group.incarnation.bytes(), + ] { + bytes.extend_from_slice(&id); + } + bytes.extend_from_slice(&witness.position.index.get().to_be_bytes()); + bytes.extend_from_slice(&witness.position.term.get().to_be_bytes()); + bytes.extend_from_slice(&witness.proposal.bytes()); + let signature = self.0.active.secret.sign(&[CELL_DOMAIN, &bytes].concat()); + bytes.extend_from_slice(&signature.to_bytes()); + CellVersionToken::from_encoded(&URL_SAFE_NO_PAD.encode(bytes)) + } + + /// Authenticate against logical history. The caller must separately check + /// installed transfer provenance and current quorum/read authority. + pub fn verify_cell( + &self, + token: &CellVersionToken, + history: &CellHistory, + schema: [u8; 32], + now_ms: u64, + ) -> Result { + let bytes = URL_SAFE_NO_PAD + .decode(token.as_str()) + .map_err(|_| VersionError::Invalid)?; + let (payload, signature) = + bytes.split_at(bytes.len().checked_sub(64).ok_or(VersionError::Invalid)?); + let mut input = Input(payload); + if input.take(8)? != CELL_MAGIC { + return Err(VersionError::Invalid); + } + let key = self + .0 + .verification + .get(&input.array()?) + .ok_or(VersionError::Invalid)?; + VerifyingKey::from_bytes(&key.public) + .map_err(|_| VersionError::Invalid)? + .verify_strict( + &[CELL_DOMAIN, payload].concat(), + &Signature::from_slice(signature).map_err(|_| VersionError::Invalid)?, + ) + .map_err(|_| VersionError::Invalid)?; + let issued = input.u64()?; + let expires = input.u64()?; + if issued > now_ms + || now_ms >= expires + || expires > key.verify_until_ms + || expires <= issued + || expires - issued > MAX_LIFETIME_MS + { + return Err(VersionError::Expired); + } + let cluster = ClusterId::new(input.array()?).map_err(|_| VersionError::Invalid)?; + let namespace = NamespaceId::new(input.array()?).map_err(|_| VersionError::Invalid)?; + let incarnation = IncarnationId::new(input.array()?).map_err(|_| VersionError::Invalid)?; + let n = u16::from_be_bytes(input.array()?) as usize; + if n > cellar_logic::cell_group::MAX_CELL_ID_BYTES { + return Err(VersionError::Limit); + } + let cell = + CellId::new(std::str::from_utf8(input.take(n)?).map_err(|_| VersionError::Invalid)?) + .map_err(|_| VersionError::Invalid)?; + let decoded = CellHistory { + cluster, + namespace, + cell, + incarnation, + }; + if &decoded != history { + return Err(VersionError::WrongHistory); + } + let sequence = MutationSequence::new(input.u64()?).map_err(|_| VersionError::Invalid)?; + let epoch = input.u64()?; + let txid = input.u64()?; + let checksum = input.u64()?; + let tag = input.array::<1>()?[0]; + let root = input.array()?; + let manifest_bytes = u32::from_be_bytes(input.array()?); + let commitment = match (tag, manifest_bytes) { + (1, 0) => CellImageCommitment::Materialized(root), + (2, _) => CellImageCommitment::Extents { + root, + manifest_bytes, + }, + _ => return Err(VersionError::Invalid), + }; + let image = CellImage { + epoch, + txid, + checksum, + commitment, + }; + crate::application::consensus::LtxImage::try_from(image) + .map_err(|_| VersionError::Invalid)?; + if input.array::<32>()? != schema { + return Err(VersionError::WrongSchema); + } + let group = GroupHistory { + cluster: ClusterId::new(input.array()?).map_err(|_| VersionError::Invalid)?, + namespace: NamespaceId::new(input.array()?).map_err(|_| VersionError::Invalid)?, + group: GroupId::new(input.array()?).map_err(|_| VersionError::Invalid)?, + incarnation: IncarnationId::new(input.array()?).map_err(|_| VersionError::Invalid)?, + }; + let position = LogPosition { + index: LogIndex::new(input.u64()?), + term: Term::new(input.u64()?), + }; + let proposal = ProposalId::new(input.array()?).map_err(|_| VersionError::Invalid)?; + let witness = CommitWitness { + group, + position, + proposal, + }; + witness + .validate(&decoded) + .map_err(|_| VersionError::Invalid)?; + if !input.0.is_empty() { + return Err(VersionError::Invalid); + } + Ok(CellVersionObservation { + version: CellVersion { + history: decoded, + sequence, + image, + }, + witness, + }) + } + pub fn new( active: VersionSigningKey, retained: Vec, diff --git a/crates/cellar/durability/raft/working.rs b/crates/cellar/durability/raft/working.rs index d90aadad1..959cd46c2 100644 --- a/crates/cellar/durability/raft/working.rs +++ b/crates/cellar/durability/raft/working.rs @@ -5,7 +5,7 @@ use super::apply::{image_position, PublishedImage}; use super::execution::PrivateOptions; use super::receipts; -use crate::application::consensus::{CellUpdate, LtxUpdate}; +use crate::application::consensus::{CellUpdate, LtxImage, LtxUpdate}; use crate::storage::CommandDatabase; use anyhow::{bail, ensure, Context as _, Result}; use cellar_logic::consensus::{GroupScope, ProposalId}; @@ -15,6 +15,46 @@ use cellar_ltx::{ltx, Db, FileSystem, TXID}; use std::collections::BTreeMap; use std::path::{Path, PathBuf}; +/// A verified physical cut, independent of routing, consensus and cell identity. +/// The execution owner must bind it to the exact cell base before proposal. +pub(crate) struct CapturedUpdate { + pub base: LtxImage, + pub result: LtxImage, + pub files: Vec, +} + +pub(crate) struct CapturedUpdateRef<'a> { + pub base: LtxImage, + pub result: LtxImage, + pub files: &'a [LtxUpdate], +} + +impl CapturedUpdate { + pub fn as_ref(&self) -> CapturedUpdateRef<'_> { + CapturedUpdateRef { + base: self.base, + result: self.result, + files: &self.files, + } + } + + pub fn bind( + self, + scope: GroupScope, + application_digest: [u8; 32], + proposal: ProposalId, + ) -> CellUpdate { + CellUpdate { + scope, + application_digest, + proposal, + base: self.base, + result: self.result, + files: self.files, + } + } +} + pub(super) struct Working { pub database: CommandDatabase, capture: Db, @@ -23,17 +63,20 @@ pub(super) struct Working { impl Working { pub fn open(options: &PrivateOptions, base: Option<&PublishedImage>) -> Result { + Self::open_pin(options, base.map(PublishedImage::pin).as_ref()) + } + + pub fn open_pin(options: &PrivateOptions, base: Option<&GenerationPin>) -> Result { let filesystem = options.host.filesystem(); let path = options.directory.join("working.sqlite"); clean(filesystem.as_ref(), &options.directory, &path)?; if let Some(base) = base { - let pin = base.pin(); - let state = pin.state(); + let state = base.state(); ensure!( state.length <= options.limits.max_image_bytes, "command base exceeds image bound" ); - pin.private_copy(&filesystem, &path)?; + base.private_copy(&filesystem, &path)?; // Only the independently sealed base is a recovery authority. // This copy is discarded on every reopen, including after a crash. } @@ -49,7 +92,7 @@ impl Working { )?; receipts::install(database.framework())?; if let Some(base) = base { - let state = base.pin().state(); + let state = base.state(); // The zero-page baseline is local capture bookkeeping. Starting at // the WAL header includes every opening/control/schema write. capture @@ -86,8 +129,18 @@ impl Working { proposal: ProposalId, base: Option<&PublishedImage>, ) -> Result { + Ok(self + .capture_pin(options, base.map(PublishedImage::pin).as_ref())? + .bind(scope.clone(), options.application_digest, proposal)) + } + + pub fn capture_pin( + &mut self, + options: &PrivateOptions, + base: Option<&GenerationPin>, + ) -> Result { let base_state = base - .map(|b| b.pin().state()) + .map(GenerationPin::state) .unwrap_or(ImageState::empty(1)?); let (result_position, files) = capture_update( &mut self.capture, @@ -110,10 +163,7 @@ impl Working { length, digest, }; - let update = CellUpdate { - scope: scope.clone(), - application_digest: options.application_digest, - proposal, + let update = CapturedUpdate { base: image_position(base_state), result: image_position(result), files, @@ -248,28 +298,27 @@ pub(super) fn capture_update( Ok((result_position, files.into_values().collect())) } -pub(super) fn verify_preview( +pub(super) fn verify_preview_pin( options: &PrivateOptions, - base: Option<&PublishedImage>, + base_pin: Option<&GenerationPin>, id: ImageId, - update: &CellUpdate, + before: LtxImage, + after: LtxImage, + files: &[LtxUpdate], ) -> Result { - let header = ltx::Header::parse(&update.files.last().context("empty command cut")?.bytes)?; + let header = ltx::Header::parse(&files.last().context("empty command cut")?.bytes)?; let result = ImageState { - epoch: update.result.epoch, - txid: TXID(update.result.txid), - checksum: update.result.checksum, - digest: update.result.commitment.sha256()?, + epoch: after.epoch, + txid: TXID(after.txid), + checksum: after.checksum, + digest: after.commitment.sha256()?, page_size: header.page_size, length: header.commit as u64 * header.page_size as u64, }; - let base_pin = base.map(PublishedImage::pin); let base_state = base_pin - .as_ref() .map(GenerationPin::state) - .unwrap_or(ImageState::empty(update.base.epoch)?); - let segments = update - .files + .unwrap_or(ImageState::empty(before.epoch)?); + let segments = files .iter() .map(|file| { Ok(LtxSegment { @@ -283,7 +332,7 @@ pub(super) fn verify_preview( .collect::>>()?; Ok(options.images.prepare( id, - base_pin.as_ref(), + base_pin, &ImageUpdate { base: base_state, result, diff --git a/crates/cellar/fleet/management/mod.rs b/crates/cellar/fleet/management/mod.rs index 4cd98497c..9fc74b332 100644 --- a/crates/cellar/fleet/management/mod.rs +++ b/crates/cellar/fleet/management/mod.rs @@ -30,14 +30,32 @@ pub use raft::{ RaftFleetCapacityAttestation, RaftFleetConnection, RaftFleetControl, RaftFleetEvacuationAcceptance, RaftFleetEvacuationDebt, RaftFleetEvacuationPhase, RaftFleetEvacuationProgress, RaftFleetEvacuationStatus, RaftFleetFailure, RaftFleetFailureCode, - RaftFleetMaintenanceAcceptance, RaftFleetMaintenanceAction, RaftFleetMaintenanceCancel, - RaftFleetMaintenancePhase, RaftFleetMaintenancePrepare, RaftFleetMaintenanceReturnContext, - RaftFleetMaintenanceStatus, RaftFleetMovement, RaftFleetMovements, RaftFleetNamespace, - RaftFleetNamespaceScope, RaftFleetNodeAcceptance, RaftFleetNodeAction, RaftFleetNodeAdmission, - RaftFleetNodeCancel, RaftFleetNodeCapacity, RaftFleetNodeDescriptor, RaftFleetNodeOperation, - RaftFleetNodeRemove, RaftFleetNodeState, RaftFleetNodeStatus, RaftFleetNodeStop, - RaftFleetOperation, RaftFleetOutcome, RaftFleetRegisterNode, RaftFleetStatus, - RaftFleetVerificationBlocker, RaftFleetVerificationReason, RaftFleetVerificationStage, + RaftFleetFirstGroupAcceptance, RaftFleetFirstGroupMutation, RaftFleetFirstGroupPlan, + RaftFleetFirstGroupPublish, RaftFleetFirstGroupStage, RaftFleetFirstGroupStandby, + RaftFleetFirstGroupVoter, RaftFleetMaintenanceAcceptance, RaftFleetMaintenanceAction, + RaftFleetMaintenanceCancel, RaftFleetMaintenancePhase, RaftFleetMaintenancePrepare, + RaftFleetMaintenanceReturnContext, RaftFleetMaintenanceStatus, RaftFleetMovement, + RaftFleetMovements, RaftFleetNamespace, RaftFleetNamespaceScope, RaftFleetNodeAcceptance, + RaftFleetNodeAction, RaftFleetNodeAdmission, RaftFleetNodeCancel, RaftFleetNodeCapacity, + RaftFleetNodeDescriptor, RaftFleetNodeOperation, RaftFleetNodeRemove, RaftFleetNodeState, + RaftFleetNodeStatus, RaftFleetNodeStop, RaftFleetOperation, RaftFleetOutcome, + RaftFleetParkPhase, RaftFleetParkRequest, RaftFleetParkStatus, RaftFleetPreparedRecoveryRange, + RaftFleetRegisterNode, RaftFleetStatus, RaftFleetTransitionActivation, + RaftFleetTransitionActivationPosition, RaftFleetTransitionActivationRequest, + RaftFleetTransitionAdmission, RaftFleetTransitionAdmissionRequest, + RaftFleetTransitionArchiveVerification, RaftFleetTransitionBasis, RaftFleetTransitionChild, + RaftFleetTransitionChildSetup, RaftFleetTransitionChildSetupRequest, + RaftFleetTransitionChildSetupResult, RaftFleetTransitionChildStep, RaftFleetTransitionGroup, + RaftFleetTransitionPartition, RaftFleetTransitionPreparation, + RaftFleetTransitionPrepareRequest, RaftFleetTransitionPublication, + RaftFleetTransitionPublishRequest, RaftFleetTransitionRecoveryBasis, RaftFleetTransitionResume, + RaftFleetTransitionResumeAcceptance, RaftFleetTransitionResumeStep, + RaftFleetTransitionSealAuthorization, RaftFleetTransitionSealRequest, + RaftFleetTransitionSourceArchive, RaftFleetTransitionSourceCut, + RaftFleetTransitionSourceRetirement, RaftFleetTransitionSourceSeal, + RaftFleetTransitionSourceSealRequest, RaftFleetTransitionStatus, RaftFleetVerificationBlocker, + RaftFleetVerificationReason, RaftFleetVerificationStage, RaftFleetVerifiedTransitionRecovery, + RaftFleetWakeStatus, }; pub use raft_stop::{RaftStopHandoff, RaftStopOwners, RaftStopProof}; mod routing; diff --git a/crates/cellar/fleet/management/raft/connection.rs b/crates/cellar/fleet/management/raft/connection.rs index e7ea6d349..895a6866e 100644 --- a/crates/cellar/fleet/management/raft/connection.rs +++ b/crates/cellar/fleet/management/raft/connection.rs @@ -32,6 +32,8 @@ pub struct RaftFleetNamespace { /// Must match the existing namespace Root; opening never changes its mode. #[serde(default = "materialized")] pub image_storage: crate::raft::catalog::RaftImageStorage, + /// Explicit bounded physical-group directory discovery and operation limits. + pub group_directory: crate::raft::cell_groups::DirectoryLimits, } fn materialized() -> crate::raft::catalog::RaftImageStorage { @@ -121,7 +123,9 @@ impl RaftFleetControl { _ = timer => return Err(FleetCatalogError::Unconfirmed.into()), result = adapter.existing_namespace(namespace_config) => result.map_err(namespace_failure)?, }; - namespaces.push(opened); + namespaces.push( + RangeDirectory::new(opened, entry.group_directory).map_err(namespace_failure)?, + ); } catalog .check_inventory(&root, &namespaces, turn.remaining()?) diff --git a/crates/cellar/fleet/management/raft/first_group.rs b/crates/cellar/fleet/management/raft/first_group.rs new file mode 100644 index 000000000..284df4f89 --- /dev/null +++ b/crates/cellar/fleet/management/raft/first_group.rs @@ -0,0 +1,328 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Authenticated first-group requests use the configured managed directory. +use super::*; +use crate::raft::{ + catalog::InitialMember, cell_groups::CellGroupBirthSpec, + fleet_catalog::RaftManagedFirstGroupRequest, +}; +use cellar_logic::{ + cell_group::{CellRange, GroupHistory, RouteEpoch}, + consensus::{GroupId, IncarnationId, MemberId}, +}; +use serde::{Deserialize, Serialize}; +use std::collections::BTreeSet; + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct RaftFleetFirstGroupStandby { + pub catalog_id: String, + pub namespace_id: String, + pub standby: String, +} + +impl RaftFleetFirstGroupStandby { + pub fn validate(&self) -> Result<()> { + records::catalog_id(&self.catalog_id)?; + records::id(&self.namespace_id)?; + records::catalog_id(&self.standby) + } +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct RaftFleetFirstGroupVoter { + pub node_id: String, + pub member_id: u64, +} + +/// Keep this complete plan for reserve and publish. Voters are ordered by +/// ascending member ID. No address or application class comes from the caller. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct RaftFleetFirstGroupPlan { + pub catalog_id: String, + pub namespace_id: String, + pub standby: String, + pub request_id: String, + pub operation_id: String, + pub expected_revision: u64, + pub publication_id: String, + pub group_id: String, + pub incarnation_id: String, + pub creation_id: String, + pub voters: [RaftFleetFirstGroupVoter; 3], +} + +impl RaftFleetFirstGroupPlan { + pub fn validate(&self, namespace: &str) -> Result<()> { + records::catalog_id(&self.catalog_id)?; + records::catalog_id(&self.standby)?; + records::id(namespace)?; + if namespace != self.namespace_id { + return Err(RaftFleetFailure::invalid("first group namespace binding")); + } + for id in [ + &self.request_id, + &self.operation_id, + &self.publication_id, + &self.group_id, + &self.incarnation_id, + &self.creation_id, + ] { + records::id(id)?; + } + records::revision(self.expected_revision)?; + if self + .voters + .windows(2) + .any(|pair| pair[0].member_id >= pair[1].member_id) + || self + .voters + .iter() + .map(|voter| &voter.node_id) + .collect::>() + .len() + != 3 + { + return Err(RaftFleetFailure::invalid( + "first group voters must be distinct and ordered", + )); + } + for voter in &self.voters { + records::id(&voter.node_id)?; + MemberId::new(voter.member_id) + .map_err(|_| RaftFleetFailure::invalid("first group member ID"))?; + } + Ok(()) + } + + fn setup( + &self, + principal: &str, + directory: &RangeDirectory, + ) -> Result { + let config = directory.namespace_config(); + let mut standby = [0; 32]; + hex::decode_to_slice(&self.standby, &mut standby) + .map_err(|_| RaftFleetFailure::invalid("first group standby digest"))?; + let voters = self + .voters + .iter() + .map(|voter| { + Ok(InitialMember { + node: NodeId::new(records::id(&voter.node_id)?) + .map_err(|_| RaftFleetFailure::invalid("first group node ID"))?, + member: MemberId::new(voter.member_id) + .map_err(|_| RaftFleetFailure::invalid("first group member ID"))?, + }) + }) + .collect::>()?; + Ok(RaftManagedFirstGroupRequest { + reservation: self.identity(principal, &self.request_id, self.expected_revision)?, + standby, + publication: proposal(&self.publication_id)?, + group: CellGroupBirthSpec { + history: GroupHistory { + cluster: config.cluster, + namespace: config.namespace, + group: GroupId::new(records::id(&self.group_id)?) + .map_err(|_| RaftFleetFailure::invalid("first group ID"))?, + incarnation: IncarnationId::new(records::id(&self.incarnation_id)?) + .map_err(|_| RaftFleetFailure::invalid("first group incarnation"))?, + }, + range: CellRange::all(), + epoch: RouteEpoch::new(1) + .map_err(|_| RaftFleetFailure::invalid("first group epoch"))?, + creation: proposal(&self.creation_id)?, + voters, + learners: vec![], + }, + }) + } + + fn identity( + &self, + principal: &str, + request: &str, + revision: u64, + ) -> Result { + Ok(FleetRequestIdentity { + id: records::id(request)?, + operation: records::id(&self.operation_id)?, + principal: principal.into(), + expected_revision: revision, + }) + } +} + +fn proposal(id: &str) -> Result { + ProposalId::new(records::id(id)?) + .map_err(|_| RaftFleetFailure::invalid("first group proposal ID")) +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct RaftFleetFirstGroupMutation { + pub plan: RaftFleetFirstGroupPlan, + pub request_id: String, + pub expected_revision: u64, +} + +pub type RaftFleetFirstGroupPublish = RaftFleetFirstGroupMutation; + +impl RaftFleetFirstGroupMutation { + pub fn validate(&self, namespace: &str) -> Result<()> { + self.plan.validate(namespace)?; + records::id(&self.request_id)?; + records::revision(self.expected_revision)?; + if self.request_id == self.plan.request_id + || self.expected_revision <= self.plan.expected_revision + { + return Err(RaftFleetFailure::invalid( + "first group confirmation identity", + )); + } + Ok(()) + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum RaftFleetFirstGroupStage { + Reserved, + Published, + ArchiveInitialized, +} + +/// Metadata admission for one exact request. A voter still needs its own +/// activation and a fresh quorum before the application can serve. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct RaftFleetFirstGroupAcceptance { + pub plan: RaftFleetFirstGroupPlan, + pub request_id: String, + pub expected_revision: u64, + pub stage: RaftFleetFirstGroupStage, +} + +impl RaftFleetFirstGroupAcceptance { + pub(crate) fn from_plan( + plan: &RaftFleetFirstGroupPlan, + request_id: &str, + expected_revision: u64, + stage: RaftFleetFirstGroupStage, + ) -> Self { + Self { + plan: plan.clone(), + request_id: request_id.into(), + expected_revision, + stage, + } + } +} + +impl RaftFleetControl { + pub async fn first_group_standby( + &self, + namespace: &str, + timeout: Duration, + ) -> Result { + let directory = self.managed_namespace(namespace)?; + let standby = self + .catalog + .observe_managed_standby_at(directory, timeout) + .await?; + Ok(RaftFleetFirstGroupStandby { + catalog_id: self.catalog_id.clone(), + namespace_id: namespace.into(), + standby: hex::encode(standby), + }) + } + + pub async fn reserve_first_group( + &self, + principal: &str, + namespace: &str, + plan: &RaftFleetFirstGroupPlan, + timeout: Duration, + ) -> Result { + plan.validate(namespace)?; + self.check_catalog(&plan.catalog_id)?; + let directory = self.managed_namespace(namespace)?; + let setup = plan.setup(principal, directory)?; + self.catalog + .reserve_managed_first_group_at(directory, setup, timeout) + .await?; + Ok(RaftFleetFirstGroupAcceptance::from_plan( + plan, + &plan.request_id, + plan.expected_revision, + RaftFleetFirstGroupStage::Reserved, + )) + } + + pub async fn publish_first_group( + &self, + principal: &str, + namespace: &str, + request: &RaftFleetFirstGroupPublish, + timeout: Duration, + ) -> Result { + request.validate(namespace)?; + let plan = &request.plan; + self.check_catalog(&plan.catalog_id)?; + let directory = self.managed_namespace(namespace)?; + let setup = plan.setup(principal, directory)?; + let confirmation = + plan.identity(principal, &request.request_id, request.expected_revision)?; + let turn = Turn::new(&self.domain, timeout)?; + let prepared = self + .catalog + .reopen_managed_first_group_at(directory, setup, turn.remaining()?) + .await?; + self.catalog + .publish_managed_first_group(&prepared, &confirmation, turn.remaining()?) + .await?; + Ok(RaftFleetFirstGroupAcceptance::from_plan( + plan, + &request.request_id, + request.expected_revision, + RaftFleetFirstGroupStage::Published, + )) + } + + pub async fn initialize_first_group_archive( + &self, + principal: &str, + namespace: &str, + request: &RaftFleetFirstGroupMutation, + timeout: Duration, + ) -> Result { + request.validate(namespace)?; + let plan = &request.plan; + self.check_catalog(&plan.catalog_id)?; + let directory = self.managed_namespace(namespace)?; + let setup = plan.setup(principal, directory)?; + let identity = plan.identity(principal, &request.request_id, request.expected_revision)?; + let turn = Turn::new(&self.domain, timeout)?; + let prepared = self + .catalog + .reopen_managed_first_group_at(directory, setup, turn.remaining()?) + .await?; + self.catalog + .initialize_managed_first_group_archive( + &prepared, + &identity, + &self.archive_pool, + turn.remaining()?, + ) + .await?; + Ok(RaftFleetFirstGroupAcceptance::from_plan( + plan, + &request.request_id, + request.expected_revision, + RaftFleetFirstGroupStage::ArchiveInitialized, + )) + } +} diff --git a/crates/cellar/fleet/management/raft/hibernation.rs b/crates/cellar/fleet/management/raft/hibernation.rs new file mode 100644 index 000000000..9d52b7854 --- /dev/null +++ b/crates/cellar/fleet/management/raft/hibernation.rs @@ -0,0 +1,320 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Reconcile the fleet's park reservation with physical voter evidence. + +use super::*; +use crate::{ + application::consensus::RaftGroupHandle, + raft::{ + cell_groups::CellGroupControllerError, + completion::RuntimeError, + fleet_catalog::{ManagedParkOperation, ManagedParkPhase, ManagedParkSettlement}, + transport::TransportError, + }, +}; +use cellar_logic::cell_group::GroupHistory; +use serde::{Deserialize, Serialize}; + +/// Bind an operator request to one group incarnation and fleet revision. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct RaftFleetParkRequest { + pub catalog_id: String, + pub namespace_id: String, + pub group: RaftFleetTransitionGroup, + pub operation_id: String, + pub request_id: String, + pub settlement_request_id: String, + pub expected_revision: u64, +} + +impl RaftFleetParkRequest { + pub fn validate(&self, namespace: &str, operation: &str) -> Result<()> { + if self.namespace_id != namespace || self.operation_id != operation { + return Err(RaftFleetFailure::invalid("park selector mismatch")); + } + self.ids().map(|_| ()) + } + + fn ids(&self) -> Result<(ProposalId, [u8; 16], [u8; 16])> { + records::catalog_id(&self.catalog_id)?; + records::revision(self.expected_revision)?; + for id in [ + &self.namespace_id, + &self.group.group_id, + &self.group.incarnation_id, + ] { + records::id(id)?; + } + let id = records::id(&self.request_id)?; + let settlement_id = records::id(&self.settlement_request_id)?; + if id == settlement_id { + return Err(RaftFleetFailure::invalid( + "Park admission and settlement require distinct request IDs.", + )); + } + let operation = ProposalId::new(records::id(&self.operation_id)?) + .map_err(|_| RaftFleetFailure::invalid("park operation ID"))?; + Ok((operation, id, settlement_id)) + } +} + +/// Historical outcome of the reservation, independent of current readiness. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum RaftFleetParkPhase { + Negotiating, + Delivered, + Superseded, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct RaftFleetParkStatus { + pub catalog_id: String, + pub namespace_id: String, + pub group: RaftFleetTransitionGroup, + pub operation_id: String, + pub generation: u64, + pub phase: RaftFleetParkPhase, + pub observed_revision: u64, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct RaftFleetWakeStatus { + pub park: RaftFleetParkStatus, + pub applied_index: u64, + pub applied_term: u64, +} + +struct Selection { + history: GroupHistory, + operation: ProposalId, + identity: FleetRequestIdentity, + settlement_id: [u8; 16], +} + +impl RaftFleetControl { + /// Deliver the park certificate to every registered voter and settle the + /// reservation. Retry with the same request after an unknown result. + pub async fn park_group( + &self, + principal: &str, + request: &RaftFleetParkRequest, + timeout: Duration, + ) -> Result { + let turn = Turn::new(&self.domain, timeout)?; + let selection = self.park_selection(principal, request)?; + let existing = self.observe_park(&selection, &turn).await?; + let voter = match &existing { + Some(park) if park.phase != ManagedParkPhase::Negotiating => { + return park_status(request, park); + } + _ => self.park_voter(selection.history)?, + }; + let permit = if let Some(park) = existing { + park.permit + } else { + let snapshot = self.catalog.read(turn.remaining()?).await?; + if snapshot.revision() != selection.identity.expected_revision { + return Err(FleetCatalogError::Conflict.into()); + } + let observation = voter + .observe_park_admission(turn.remaining()?) + .await + .map_err(park_failure)?; + self.catalog + .admit_managed_park( + &snapshot, + &selection.identity, + &observation, + turn.remaining()?, + ) + .await? + // The observation pins the leader's active state until dropped. + }; + let receipt = voter + .execute_managed_park(&permit, turn.remaining()?) + .await + .map_err(park_failure)?; + let snapshot = self.catalog.read(turn.remaining()?).await?; + let settled = self + .catalog + .settle_managed_park( + &snapshot, + &selection.settlement(snapshot.revision()), + &permit, + ManagedParkSettlement::Delivered(&receipt), + turn.remaining()?, + ) + .await?; + park_status( + request, + &ManagedParkOperation { + permit, + phase: ManagedParkPhase::Delivered, + revision: settled.snapshot.revision(), + }, + ) + } + + /// Read the rooted operation without sending any messages to voters. + pub async fn park_group_status( + &self, + principal: &str, + request: &RaftFleetParkRequest, + timeout: Duration, + ) -> Result> { + let turn = Turn::new(&self.domain, timeout)?; + let selection = self.park_selection(principal, request)?; + self.observe_park(&selection, &turn) + .await? + .as_ref() + .map(|park| park_status(request, park)) + .transpose() + } + + /// Wake the exact generation and wait for a fresh quorum read. A pending + /// reservation settles only once the current leader proves a later cut. + pub async fn wake_group( + &self, + principal: &str, + request: &RaftFleetParkRequest, + timeout: Duration, + ) -> Result { + let turn = Turn::new(&self.domain, timeout)?; + let selection = self.park_selection(principal, request)?; + let mut park = + self.observe_park(&selection, &turn) + .await? + .ok_or_else(|| RaftFleetFailure { + code: RaftFleetFailureCode::NotFound, + outcome: RaftFleetOutcome::NotSubmitted, + message: "The park operation is absent from the fleet catalog.".into(), + })?; + let unsettled = park.phase == ManagedParkPhase::Negotiating; + let observation = self + .park_voter(selection.history)? + .wake_managed_park(&park.permit, unsettled, turn.remaining()?) + .await + .map_err(park_failure)?; + let applied = observation.root().position(); + if unsettled { + let snapshot = self.catalog.read(turn.remaining()?).await?; + let settled = self + .catalog + .settle_managed_park( + &snapshot, + &selection.settlement(snapshot.revision()), + &park.permit, + ManagedParkSettlement::Advanced(&observation), + turn.remaining()?, + ) + .await?; + park.phase = ManagedParkPhase::Superseded; + park.revision = settled.snapshot.revision(); + } + Ok(RaftFleetWakeStatus { + park: park_status(request, &park)?, + applied_index: applied.index.get(), + applied_term: applied.term.get(), + }) + } + + fn park_selection(&self, principal: &str, request: &RaftFleetParkRequest) -> Result { + let (operation, id, settlement_id) = request.ids()?; + self.check_catalog(&request.catalog_id)?; + let directory = self.managed_namespace(&request.namespace_id)?; + Ok(Selection { + history: request.group.history(directory)?, + operation, + identity: FleetRequestIdentity { + id, + operation: operation.bytes(), + principal: principal.into(), + expected_revision: request.expected_revision, + }, + settlement_id, + }) + } + + async fn observe_park( + &self, + selection: &Selection, + turn: &Turn<'_>, + ) -> Result> { + let observed = self + .catalog + .observe_managed_park(selection.operation, turn.remaining()?) + .await?; + if observed.as_ref().is_some_and(|park| { + !park + .permit + .matches_request(selection.history, &selection.identity) + }) { + return Err(FleetCatalogError::PayloadConflict.into()); + } + Ok(observed) + } + + fn park_voter(&self, history: GroupHistory) -> Result { + self.node_observer + .as_ref() + .ok_or_else(RaftFleetFailure::unsupported)? + .group(history) + .map_err(|error| park_failure(error.into())) + } +} + +impl Selection { + fn settlement(&self, revision: u64) -> FleetRequestIdentity { + FleetRequestIdentity { + id: self.settlement_id, + expected_revision: revision, + ..self.identity.clone() + } + } +} + +fn park_status( + request: &RaftFleetParkRequest, + park: &ManagedParkOperation, +) -> Result { + let phase = match park.phase { + ManagedParkPhase::Negotiating => RaftFleetParkPhase::Negotiating, + ManagedParkPhase::Delivered => RaftFleetParkPhase::Delivered, + ManagedParkPhase::Superseded => RaftFleetParkPhase::Superseded, + }; + Ok(RaftFleetParkStatus { + catalog_id: request.catalog_id.clone(), + namespace_id: request.namespace_id.clone(), + group: request.group.clone(), + operation_id: request.operation_id.clone(), + generation: park.permit.generation()?.get(), + phase, + observed_revision: park.revision, + }) +} + +fn park_failure(error: CellGroupControllerError) -> RaftFleetFailure { + use RaftFleetFailureCode as Code; + let code = match error { + CellGroupControllerError::Runtime(error) + | CellGroupControllerError::Transport(TransportError::Runtime(error)) => match error { + RuntimeError::Overloaded => Code::Busy, + RuntimeError::DeadlineExceeded => Code::WaitTimeout, + RuntimeError::StaleCompletion => Code::Conflict, + RuntimeError::NotFound => Code::NotFound, + _ => Code::Unavailable, + }, + CellGroupControllerError::Transport(TransportError::Timeout) => Code::WaitTimeout, + CellGroupControllerError::Transport(TransportError::Overloaded) => Code::Busy, + _ => Code::Unavailable, + }; + RaftFleetFailure { + code, + outcome: RaftFleetOutcome::Unknown, + message: "The group operation could not be confirmed. Check its status and retry the exact request on the current leader.".into(), + } +} diff --git a/crates/cellar/fleet/management/raft/lifecycle/mod.rs b/crates/cellar/fleet/management/raft/lifecycle/mod.rs index dee52d410..b76827ca0 100644 --- a/crates/cellar/fleet/management/raft/lifecycle/mod.rs +++ b/crates/cellar/fleet/management/raft/lifecycle/mod.rs @@ -5,13 +5,12 @@ mod records; pub(super) mod verification; mod waiting; use super::*; -use crate::raft::archive::{ArchiveLimits, ArchiveVerifier}; +use crate::raft::fleet_catalog::FleetGroupArchiveVerifier; use crate::raft::fleet_catalog::{ FleetFinalizationAdmission, FleetRemovalAdmission, FleetRemovalAttempt, FleetRemovalRequest, FleetSnapshot, }; use cellar_logic::consensus::ProcessGeneration; -use cellar_ltx::LtxHost; pub use records::*; use verification::{Advance, Key, Session, Verification}; pub use waiting::{ @@ -138,15 +137,10 @@ fn request_from(identity: &FleetRequestIdentity, catalog: &str) -> RaftFleetNode impl RaftFleetControl { /// Explicit replay services for one bounded lifecycle proof session. The - /// caller owns the private GenerationStore and its configured disk/memory - /// limits. This neither creates a node nor starts a background controller. - pub fn with_removal_verifier( - mut self, - host: LtxHost, - verifier: ArchiveVerifier, - limits: ArchiveLimits, - ) -> Result { - self.removal_verification = Some(Arc::new(Verification::new(host, verifier, limits)?)); + /// caller owns private group replay inventories and shared checkpoint, + /// archive, tail and scratch pools. This starts no background controller. + pub fn with_removal_verifier(mut self, verifier: FleetGroupArchiveVerifier) -> Result { + self.removal_verification = Some(Arc::new(Verification::new(verifier))); Ok(self) } @@ -284,7 +278,9 @@ impl RaftFleetControl { }); } let session = lease.session.as_mut().ok_or(FleetCatalogError::Closed)?; - let advance = runtime.advance(self, session, observer, &turn).await?; + let advance = runtime + .advance(self, session, observer, &root, &turn) + .await?; if advance != Advance::Exhausted { if advance == Advance::Waiting { let current = self.catalog.read(turn.remaining()?).await?; @@ -520,7 +516,9 @@ impl RaftFleetControl { }); } let session = lease.session.as_mut().ok_or(FleetCatalogError::Closed)?; - let advance = runtime.advance(self, session, observer, &turn).await?; + let advance = runtime + .advance(self, session, observer, &root, &turn) + .await?; if advance != Advance::Exhausted { if advance == Advance::Waiting { let current = self.catalog.read(turn.remaining()?).await?; @@ -726,7 +724,6 @@ impl RaftFleetControl { namespace: hex::encode(m.group.namespace.bytes()), group: hex::encode(m.group.group.bytes()), incarnation: hex::encode(m.group.incarnation.bytes()), - cell: m.group.cell, target: hex::encode(m.target.bytes()), peak_engine_slots: m.peak.engine_slots, peak_durable_bytes: m.peak.durable_bytes, diff --git a/crates/cellar/fleet/management/raft/lifecycle/records.rs b/crates/cellar/fleet/management/raft/lifecycle/records.rs index 470a29dbe..d1ab80779 100644 --- a/crates/cellar/fleet/management/raft/lifecycle/records.rs +++ b/crates/cellar/fleet/management/raft/lifecycle/records.rs @@ -240,7 +240,6 @@ pub struct RaftFleetEvacuationDebt { pub namespace: String, pub group: String, pub incarnation: String, - pub cell: String, pub target: String, pub peak_engine_slots: u32, pub peak_durable_bytes: u64, diff --git a/crates/cellar/fleet/management/raft/lifecycle/verification.rs b/crates/cellar/fleet/management/raft/lifecycle/verification.rs index 697b05eb1..9db7d49ae 100644 --- a/crates/cellar/fleet/management/raft/lifecycle/verification.rs +++ b/crates/cellar/fleet/management/raft/lifecycle/verification.rs @@ -2,21 +2,18 @@ //! At most one in-memory complete-inventory walk. Restart repeats its proofs. use super::*; -use crate::raft::archive::{ArchiveError, ArchiveLimits, ArchiveVerifier}; use crate::raft::fleet_catalog::{ - FleetEvacuationArchive, FleetEvacuationVerification, FleetHistoryState, + FleetEvacuationArchive, FleetEvacuationVerification, FleetGroupArchiveVerifier, + FleetHistoryState, }; -use cellar_logic::consensus::{MemberId, NodeId}; -use cellar_ltx::LtxHost; +use cellar_logic::consensus::NodeId; use std::sync::{ atomic::{AtomicBool, Ordering}, Mutex, }; pub(in super::super) struct Verification { - host: LtxHost, - verifier: ArchiveVerifier, - limits: ArchiveLimits, + archive: FleetGroupArchiveVerifier, session: Mutex>, busy: AtomicBool, } @@ -93,21 +90,12 @@ impl Verification { Ok(Some(value)) } - pub(super) fn new( - host: LtxHost, - verifier: ArchiveVerifier, - limits: ArchiveLimits, - ) -> Result { - limits - .validate() - .map_err(|_| RaftFleetFailure::invalid("removal archive bounds"))?; - Ok(Self { - host, - verifier, - limits, + pub(super) fn new(archive: FleetGroupArchiveVerifier) -> Self { + Self { + archive, session: Mutex::new(None), busy: AtomicBool::new(false), - }) + } } pub(super) fn lease(&self, key: &Key) -> Result> { @@ -140,6 +128,7 @@ impl Verification { control: &RaftFleetControl, session: &mut Session, observer: &RaftCells, + cut: &FleetSnapshot, turn: &Turn<'_>, ) -> Result { session.waiting = None; @@ -150,6 +139,9 @@ impl Verification { FleetHistoryState::Cancelled { .. } => { session.walk.verify_cancelled(turn.remaining()?).await? } + FleetHistoryState::Retired { .. } => { + session.walk.verify_retired(turn.remaining()?).await? + } FleetHistoryState::Live => { let scope = history .binding() @@ -158,87 +150,56 @@ impl Verification { let namespace = control .namespaces .iter() - .find(|n| n.config().namespace == scope.namespace) - .ok_or(FleetCatalogError::InventoryIncomplete)?; - let birth = turn - .run(namespace.birth(&scope.cell)) - .await? - .map_err(FleetCatalogError::from)? + .find(|n| n.namespace_config().namespace == scope.namespace) .ok_or(FleetCatalogError::InventoryIncomplete)?; - if birth.scope() != &scope || birth.digest() != history.birth_digest() { - return Err(FleetCatalogError::Conflict.into()); - } - // Bind archive I/O to the management owner's execution domain - // even for SDK calls made outside an HTTP/domain task. - let archive = turn - .run(async { namespace.cell_archive(&birth, self.host.clone(), self.limits) }) - .await? - .map_err(FleetCatalogError::from)?; - let metadata = turn - .run(archive.recovery_metadata()) - .await? - .map_err(archive_error)? - .ok_or(FleetCatalogError::Unconfirmed)?; - let configuration = metadata.configuration().map_err(archive_error)?; - if configuration.voters.len() != 3 - || !configuration.voters_outgoing.is_empty() - || !configuration.learners.is_empty() - || !configuration.learners_next.is_empty() - { - return Err(FleetCatalogError::Unconfirmed.into()); - } - let membership = metadata.membership(); - let mut nodes = Vec::with_capacity(3); - for member in configuration.voters { - let member = MemberId::new(member) - .map_err(|_| FleetCatalogError::Corrupt("archive voter"))?; - let node = match membership - .joined(member) - .map_err(|_| FleetCatalogError::Corrupt("archive binding"))? - { - Some((binding, _)) => binding.node, - None => birth - .voters() - .iter() - .chain(birth.learners()) - .find(|v| v.member == member) - .map(|v| v.node) - .ok_or(FleetCatalogError::InventoryIncomplete)?, - }; - nodes.push(node); - } - let nodes: [NodeId; 3] = nodes + // Rooted current membership only selects candidates; the + // walk independently authenticates all three live survivors. + let (voters, _) = turn + .run(control.catalog.settled_group_voters(cut, &history)) + .await??; + let nodes: [NodeId; 3] = voters + .into_keys() + .collect::>() .try_into() .map_err(|_| FleetCatalogError::InventoryIncomplete)?; - let reference = metadata.reference(); + let fixed = match session + .walk + .freeze_live(observer, nodes, turn.remaining()?) + .await + { + Ok(fixed) => fixed, + Err(error) => return waiting(session, &history, error), + }; + let registry = turn + .run(self.archive.open(&control.domain, namespace, &history)) + .await??; + let hold = session + .walk + .acquire_archive_hold(®istry, turn.remaining()?) + .await?; + let receiver = turn + .run(self.archive.receiver( + &control.domain, + fixed.root().route(), + namespace.namespace_config().image_storage, + )) + .await??; let result = session .walk .verify_live( observer, nodes, FleetEvacuationArchive { - archive: &archive, - reference: &reference, - verifier: &self.verifier, + registry: ®istry, + hold: &hold, + tails: self.archive.tails(), + receiver, }, turn.remaining()?, ) .await; if let Err(error) = result { - let Some(blocker) = RaftFleetVerificationBlocker::observation( - &error, - RaftFleetVerificationStage::HistoryQuorum, - Some( - history - .binding() - .key() - .map_err(|_| FleetCatalogError::Corrupt("waiting history key"))?, - ), - ) else { - return Err(error.into()); - }; - session.waiting = Some(blocker); - return Ok(Advance::Waiting); + return waiting(session, &history, error); } } _ => return Err(FleetCatalogError::Unconfirmed.into()), @@ -251,14 +212,23 @@ impl Verification { } } -fn archive_error(error: ArchiveError) -> FleetCatalogError { - match error { - ArchiveError::Invalid(reason) => FleetCatalogError::Invalid(reason), - ArchiveError::Corrupt(reason) => FleetCatalogError::Corrupt(reason), - ArchiveError::Conflict => FleetCatalogError::Conflict, - ArchiveError::Limit => FleetCatalogError::Limit, - ArchiveError::Unconfirmed | ArchiveError::Timeout => FleetCatalogError::Unconfirmed, - ArchiveError::Stopped => FleetCatalogError::Closed, - error => FleetCatalogError::Storage(error.into()), - } +fn waiting( + session: &mut Session, + history: &crate::raft::fleet_catalog::FleetHistory, + error: FleetCatalogError, +) -> Result { + let Some(blocker) = RaftFleetVerificationBlocker::observation( + &error, + RaftFleetVerificationStage::HistoryQuorum, + Some( + history + .binding() + .key() + .map_err(|_| FleetCatalogError::Corrupt("waiting history key"))?, + ), + ) else { + return Err(error.into()); + }; + session.waiting = Some(blocker); + Ok(Advance::Waiting) } diff --git a/crates/cellar/fleet/management/raft/mod.rs b/crates/cellar/fleet/management/raft/mod.rs index f67894981..414adf72f 100644 --- a/crates/cellar/fleet/management/raft/mod.rs +++ b/crates/cellar/fleet/management/raft/mod.rs @@ -5,21 +5,37 @@ //! proofs remain host-owned; capacity-granted operators explicitly attest limits. mod connection; mod failure; +mod first_group; +mod hibernation; mod lifecycle; mod maintenance; pub use maintenance::*; mod nodes; mod records; +mod transition; +mod transition_admission; +mod transition_child; +mod transition_seal; +mod transition_verify; use crate::application::consensus::RaftCells; use crate::asyncrt::{DomainToken, ExecutionDomain}; -use crate::raft::catalog::NamespaceCatalog; +use crate::raft::catalog::RangeDirectory; +use crate::raft::cell_groups::{CellGroupArchiveLimits, CellGroupArchivePool}; use crate::raft::fleet_catalog::{ FleetCatalog, FleetCatalogError, FleetCursor, FleetRequestIdentity, FleetTopologyIntentRequest, FleetTopologyOperation, FleetTopologyPhase, FleetTopologyPolicy, }; -use cellar_logic::consensus::{NodeId, ProposalId}; +use cellar_logic::consensus::{NamespaceId, NodeId, ProposalId}; pub use connection::{RaftFleetConnection, RaftFleetNamespace, RaftFleetNamespaceScope}; pub use failure::{RaftFleetFailure, RaftFleetFailureCode, RaftFleetOutcome}; +pub use first_group::{ + RaftFleetFirstGroupAcceptance, RaftFleetFirstGroupMutation, RaftFleetFirstGroupPlan, + RaftFleetFirstGroupPublish, RaftFleetFirstGroupStage, RaftFleetFirstGroupStandby, + RaftFleetFirstGroupVoter, +}; +pub use hibernation::{ + RaftFleetParkPhase, RaftFleetParkRequest, RaftFleetParkStatus, RaftFleetWakeStatus, +}; pub use lifecycle::{ RaftFleetEvacuationAcceptance, RaftFleetEvacuationDebt, RaftFleetEvacuationPhase, RaftFleetEvacuationProgress, RaftFleetEvacuationStatus, RaftFleetNodeCancel, @@ -36,6 +52,34 @@ pub use records::{ RaftFleetOperation, RaftFleetStatus, }; use std::{sync::Arc, time::Duration}; +pub use transition::{ + RaftFleetTransitionActivation, RaftFleetTransitionActivationPosition, + RaftFleetTransitionActivationRequest, RaftFleetTransitionPreparation, + RaftFleetTransitionPrepareRequest, RaftFleetTransitionPublication, + RaftFleetTransitionPublishRequest, RaftFleetTransitionRecoveryBasis, RaftFleetTransitionResume, + RaftFleetTransitionResumeAcceptance, RaftFleetTransitionResumeStep, + RaftFleetTransitionSealAuthorization, RaftFleetTransitionSealRequest, + RaftFleetTransitionStatus, +}; +pub use transition_admission::{ + RaftFleetTransitionAdmission, RaftFleetTransitionAdmissionRequest, RaftFleetTransitionBasis, + RaftFleetTransitionChild, RaftFleetTransitionGroup, RaftFleetTransitionPartition, +}; + +pub use transition_child::{ + RaftFleetTransitionChildSetup, RaftFleetTransitionChildSetupRequest, + RaftFleetTransitionChildSetupResult, RaftFleetTransitionChildStep, +}; +pub use transition_seal::{ + RaftFleetTransitionSourceArchive, RaftFleetTransitionSourceCut, + RaftFleetTransitionSourceRetirement, RaftFleetTransitionSourceSeal, + RaftFleetTransitionSourceSealRequest, +}; + +pub use transition_verify::{ + RaftFleetPreparedRecoveryRange, RaftFleetTransitionArchiveVerification, + RaftFleetVerifiedTransitionRecovery, +}; type Result = std::result::Result; @@ -46,9 +90,11 @@ type Result = std::result::Result; pub struct RaftFleetControl { catalog: FleetCatalog, catalog_id: String, - namespaces: Arc<[NamespaceCatalog]>, + namespaces: Arc<[RangeDirectory]>, domain: DomainToken, node_observer: Option, + archive_pool: CellGroupArchivePool, + transition_preparation: Option, removal_verification: Option>, } @@ -56,7 +102,7 @@ impl RaftFleetControl { pub fn new( domain: &ExecutionDomain, catalog: FleetCatalog, - namespaces: Vec, + namespaces: Vec, ) -> Result { if namespaces.len() > 64 { return Err(RaftFleetFailure::invalid("too many managed namespaces")); @@ -72,10 +118,33 @@ impl RaftFleetControl { namespaces: namespaces.into(), domain, node_observer: None, + archive_pool: CellGroupArchivePool::new(CellGroupArchiveLimits::default()) + .map_err(FleetCatalogError::from)?, + transition_preparation: None, removal_verification: None, }) } + fn managed_namespace(&self, namespace: &str) -> Result<&RangeDirectory> { + let id = NamespaceId::new(records::id(namespace)?) + .map_err(|_| RaftFleetFailure::invalid("managed namespace ID"))?; + let mut found = self + .namespaces + .iter() + .filter(|directory| directory.namespace_config().namespace == id); + let directory = found.next().ok_or_else(|| RaftFleetFailure { + code: RaftFleetFailureCode::NotFound, + outcome: RaftFleetOutcome::NotSubmitted, + message: "The namespace is outside configured fleet management.".into(), + })?; + if found.next().is_some() { + return Err(RaftFleetFailure::invalid( + "duplicate configured managed namespace", + )); + } + Ok(directory) + } + pub fn catalog_id(&self) -> &str { &self.catalog_id } @@ -233,7 +302,6 @@ impl RaftFleetControl { namespace: hex::encode(m.group.namespace.bytes()), group: hex::encode(m.group.group.bytes()), incarnation: hex::encode(m.group.incarnation.bytes()), - cell: m.group.cell, source: hex::encode(m.source.bytes()), target: hex::encode(m.target.bytes()), admitted_revision: m.admitted_revision, diff --git a/crates/cellar/fleet/management/raft/records.rs b/crates/cellar/fleet/management/raft/records.rs index 82b739cc9..df567853c 100644 --- a/crates/cellar/fleet/management/raft/records.rs +++ b/crates/cellar/fleet/management/raft/records.rs @@ -168,7 +168,6 @@ pub struct RaftFleetMovement { pub namespace: String, pub group: String, pub incarnation: String, - pub cell: String, pub source: String, pub target: String, pub admitted_revision: u64, diff --git a/crates/cellar/fleet/management/raft/transition.rs b/crates/cellar/fleet/management/raft/transition.rs new file mode 100644 index 000000000..2f2391746 --- /dev/null +++ b/crates/cellar/fleet/management/raft/transition.rs @@ -0,0 +1,827 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Operator status and exact continuation of managed range transitions. + +use super::*; +use crate::raft::{ + cell_groups::{CellGroupControllerError, CellGroupTransferActivation, TransitionPhase}, + completion::RuntimeError, + fleet_catalog::ReopenedManagedCellGroupTransition, +}; +use cellar_logic::{ + cell_group::GroupHistory, + consensus::{GroupId, IncarnationId, LogPosition}, +}; +use serde::{Deserialize, Serialize}; + +/// The plan digest and rooted progress for one managed split or merge. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct RaftFleetTransitionStatus { + pub catalog_id: String, + pub namespace_id: String, + pub operation_id: String, + pub observed_fleet_revision: u64, + pub plan_digest: String, + /// Absent until the accepted Fleet intent has a directory journal. + pub phase: Option, + /// Exact active-source preparation selected by the journal. + pub preparation_digest: Option, + /// Exact independently verified transfer archive selected by the journal. + pub transfer_archive_digest: Option, + pub child_archives_verified: bool, + pub parent_holds_released: bool, + /// The exact directory receipt when recovery superseded this operation. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub recovery_completion_digest: Option, + /// Sampled at the reported Fleet revision. Directory cleanup precedes + /// Fleet completion and may already be visible while this is still true. + pub fleet_slot_held: bool, +} + +/// Rooted topology basis for choosing archives after an interrupted operation. +/// These identities do not describe a complete infrastructure fence inventory. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct RaftFleetTransitionRecoveryBasis { + pub catalog_id: String, + pub namespace_id: String, + pub operation_id: String, + pub fleet_revision: u64, + pub fleet_digest: String, + pub directory_digest: String, + pub plan_digest: String, + pub phase: TransitionPhase, + pub publication_digest: Option, + pub transfer_archive_digest: Option, + pub child_archives_digest: Option, + pub parents: Vec, + pub children: Vec, +} + +impl RaftFleetTransitionRecoveryBasis { + pub(super) fn from_observation( + catalog: &str, + namespace: &str, + operation: ProposalId, + observation: &crate::raft::fleet_catalog::ManagedTransitionRecoveryObservation, + ) -> Self { + let selected = &observation.selection; + let groups = |routes: &[cellar_logic::cell_group::GroupRoute]| { + routes + .iter() + .map(|route| RaftFleetTransitionGroup { + group_id: hex::encode(route.history.group.bytes()), + incarnation_id: hex::encode(route.history.incarnation.bytes()), + }) + .collect() + }; + Self { + catalog_id: catalog.into(), + namespace_id: namespace.into(), + operation_id: hex::encode(operation.bytes()), + fleet_revision: observation.fleet_revision, + fleet_digest: hex::encode(observation.fleet_digest), + directory_digest: hex::encode(selected.directory_digest), + plan_digest: hex::encode(selected.plan.digest()), + phase: selected.journal.phase(), + publication_digest: selected.publication.map(hex::encode), + transfer_archive_digest: selected.journal.transfer_archive().map(hex::encode), + child_archives_digest: selected.journal.child_archives().map(hex::encode), + parents: groups(&selected.parents), + children: groups(&selected.children), + } + } + + /// Published children can have acknowledged effects after the transfer cut. + /// Their archive closure must account for those effects independently. + pub fn archive_sources(&self) -> &[RaftFleetTransitionGroup] { + if self.publication_digest.is_some() { + &self.children + } else { + &self.parents + } + } +} + +impl RaftFleetControl { + /// Resolve the exact interrupted transition while it owns Fleet and the + /// directory. This read does not retain or verify archives, fence writers, + /// or authorize any state change. + pub async fn transition_recovery_basis( + &self, + namespace: &str, + operation: &str, + timeout: Duration, + ) -> Result { + let directory = self.managed_namespace(namespace)?; + let operation = ProposalId::new(records::id(operation)?) + .map_err(|_| RaftFleetFailure::invalid("operation ID"))?; + let observation = self + .catalog + .observe_managed_transition_recovery(directory, operation, timeout) + .await?; + Ok(RaftFleetTransitionRecoveryBasis::from_observation( + self.catalog_id(), + namespace, + operation, + &observation, + )) + } +} + +impl RaftFleetControl { + /// Inspect the exact admitted operation without advancing it. + pub async fn transition_status( + &self, + namespace: &str, + operation: &str, + timeout: Duration, + ) -> Result { + let directory = self.managed_namespace(namespace)?; + let operation = ProposalId::new(records::id(operation)?) + .map_err(|_| RaftFleetFailure::invalid("operation ID"))?; + let observation = self + .catalog + .observe_managed_transition(directory, operation, timeout) + .await? + .ok_or_else(missing_transition)?; + let journal = observation.transition.journal.as_ref(); + Ok(RaftFleetTransitionStatus { + catalog_id: self.catalog_id.clone(), + namespace_id: namespace.into(), + operation_id: hex::encode(operation.bytes()), + observed_fleet_revision: observation.fleet_revision, + plan_digest: hex::encode(observation.transition.plan.digest()), + phase: journal.map(|journal| journal.phase()), + preparation_digest: journal + .and_then(|journal| journal.preparation()) + .map(hex::encode), + transfer_archive_digest: journal + .and_then(|journal| journal.transfer_archive()) + .map(hex::encode), + child_archives_verified: journal + .is_some_and(|journal| journal.child_archives().is_some()), + parent_holds_released: journal.is_some_and(|journal| journal.parent_holds_released()), + recovery_completion_digest: journal + .and_then(|journal| journal.recovery_settlement()) + .map(hex::encode), + fleet_slot_held: observation.fleet_slot_held, + }) + } +} + +/// One explicitly selected terminal step. Keep it with the request ID across +/// an uncertain reply so a retry has the same meaning after progress. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum RaftFleetTransitionResumeStep { + ConfirmChildren, + Complete, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct RaftFleetTransitionResume { + pub catalog_id: String, + pub namespace_id: String, + pub operation_id: String, + pub request_id: String, + pub expected_revision: u64, + pub plan_digest: String, + pub step: RaftFleetTransitionResumeStep, +} + +impl RaftFleetTransitionResume { + pub fn validate(&self, namespace: &str, operation: &str) -> Result<()> { + records::catalog_id(&self.catalog_id)?; + records::catalog_id(&self.plan_digest)?; + records::id(namespace)?; + records::id(operation)?; + records::id(&self.request_id)?; + records::revision(self.expected_revision)?; + if self.namespace_id != namespace || self.operation_id != operation { + return Err(RaftFleetFailure::invalid("transition selector mismatch")); + } + Ok(()) + } +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct RaftFleetTransitionResumeAcceptance { + pub request: RaftFleetTransitionResume, + /// Revision that applied this request, including on exact replay. + pub applied_revision: u64, + pub observed_fleet_revision: u64, +} + +impl RaftFleetControl { + /// Resume one terminal step of the exact admitted split or merge. + pub async fn resume_transition( + &self, + principal: &str, + namespace: &str, + operation: &str, + request: &RaftFleetTransitionResume, + timeout: Duration, + ) -> Result { + request.validate(namespace, operation)?; + self.check_catalog(&request.catalog_id)?; + let directory = self.managed_namespace(namespace)?; + let operation = ProposalId::new(records::id(operation)?) + .map_err(|_| RaftFleetFailure::invalid("operation ID"))?; + let turn = Turn::new(&self.domain, timeout)?; + let transition = self + .authorize_transition_operator( + directory, + operation, + principal, + &request.plan_digest, + turn.remaining()?, + ) + .await?; + let identity = FleetRequestIdentity { + id: records::id(&request.request_id)?, + operation: operation.bytes(), + principal: principal.into(), + expected_revision: request.expected_revision, + }; + let snapshot = self.catalog.read(turn.remaining()?).await?; + let commit = match request.step { + RaftFleetTransitionResumeStep::ConfirmChildren => { + self.catalog + .confirm_managed_transfer_children( + &snapshot, + &identity, + &transition, + directory, + &self.archive_pool, + turn.remaining()?, + ) + .await? + } + RaftFleetTransitionResumeStep::Complete => { + let parent_release = self + .node_observer + .as_ref() + .map(|observer| (&self.archive_pool, observer)); + self.catalog + .complete_managed_transfer_with_parent_release( + &snapshot, + &identity, + &transition, + directory, + parent_release, + turn.remaining()?, + ) + .await? + } + }; + Ok(RaftFleetTransitionResumeAcceptance { + request: request.clone(), + applied_revision: commit.applied_revision, + observed_fleet_revision: commit.snapshot.revision(), + }) + } +} + +/// Select pre-copy for the exact admitted plan. Archive inputs and scratch +/// resources come from the trusted operator server configuration. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct RaftFleetTransitionPrepareRequest { + pub catalog_id: String, + pub namespace_id: String, + pub operation_id: String, + pub request_id: String, + pub plan_digest: String, +} + +impl RaftFleetTransitionPrepareRequest { + pub fn validate(&self, namespace: &str, operation: &str) -> Result<()> { + records::catalog_id(&self.catalog_id)?; + records::catalog_id(&self.plan_digest)?; + for id in [namespace, operation, &self.request_id] { + records::id(id)?; + } + if self.namespace_id != namespace || self.operation_id != operation { + return Err(RaftFleetFailure::invalid("transition selector mismatch")); + } + Ok(()) + } +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct RaftFleetTransitionPreparation { + pub request: RaftFleetTransitionPrepareRequest, + pub preparation_digest: String, + /// Exact retries can observe a later phase with the same preparation. + pub phase: TransitionPhase, +} + +#[derive(Clone)] +pub(super) struct Preparation { + pub(super) verifier: crate::raft::fleet_catalog::FleetGroupArchiveVerifier, + pub(super) recovery: crate::raft::cell_groups::CellGroupRecoveryPool, +} + +impl RaftFleetControl { + /// Configure private physical replay resources and bounded restore + /// admission for operator split and merge preparation. + pub fn with_transition_preparer( + mut self, + verifier: crate::raft::fleet_catalog::FleetGroupArchiveVerifier, + recovery: crate::raft::cell_groups::CellGroupRecoveryPool, + ) -> Self { + self.transition_preparation = Some(Preparation { verifier, recovery }); + self + } + + /// Prepare the admitted plan from operation-owned archive cuts. Only + /// Planned work can enter physical replay. Later exact retries return the + /// rooted preparation digest, with no extra allocation or authority. + pub async fn prepare_transition( + &self, + principal: &str, + namespace: &str, + operation: &str, + request: &RaftFleetTransitionPrepareRequest, + timeout: Duration, + ) -> Result { + request.validate(namespace, operation)?; + self.check_catalog(&request.catalog_id)?; + let directory = self.managed_namespace(namespace)?; + let operation = ProposalId::new(records::id(operation)?) + .map_err(|_| RaftFleetFailure::invalid("operation ID"))?; + let turn = Turn::new(&self.domain, timeout)?; + let transition = self + .authorize_transition_operator( + directory, + operation, + principal, + &request.plan_digest, + turn.remaining()?, + ) + .await?; + let journal = transition.journal.ok_or(FleetCatalogError::Conflict)?; + let journal = match journal.phase() { + TransitionPhase::Planned => { + let services = self.transition_preparation.as_ref().ok_or_else(|| { + RaftFleetFailure { + code: RaftFleetFailureCode::Unsupported, + outcome: RaftFleetOutcome::NotSubmitted, + message: "Transition preparation requires a trusted archive verifier on the management server.".into(), + } + })?; + self.catalog + .allocate_managed_preparation_holds( + directory, + operation, + &self.archive_pool, + turn.remaining()?, + ) + .await?; + let prepared = self + .catalog + .resume_managed_preparation( + directory, + operation, + &services.verifier, + services.recovery.clone(), + turn.remaining()?, + ) + .await?; + prepared.journal().clone() + } + TransitionPhase::Cancelled => return Err(FleetCatalogError::Conflict.into()), + _ => journal, + }; + let preparation_digest = journal.preparation().ok_or(FleetCatalogError::Conflict)?; + Ok(RaftFleetTransitionPreparation { + request: request.clone(), + preparation_digest: hex::encode(preparation_digest), + phase: journal.phase(), + }) + } +} + +/// Select the irreversible seal decision for the exact prepared transition. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct RaftFleetTransitionSealRequest { + pub catalog_id: String, + pub namespace_id: String, + pub operation_id: String, + pub request_id: String, + pub plan_digest: String, + pub preparation_digest: String, +} + +impl RaftFleetTransitionSealRequest { + pub fn validate(&self, namespace: &str, operation: &str) -> Result<()> { + for digest in [ + &self.catalog_id, + &self.plan_digest, + &self.preparation_digest, + ] { + records::catalog_id(digest)?; + } + for id in [namespace, operation, &self.request_id] { + records::id(id)?; + } + if self.namespace_id != namespace || self.operation_id != operation { + return Err(RaftFleetFailure::invalid("transition selector mismatch")); + } + Ok(()) + } +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct RaftFleetTransitionSealAuthorization { + pub request: RaftFleetTransitionSealRequest, + /// The observed phase can advance after the seal decision. + pub phase: TransitionPhase, +} + +impl RaftFleetControl { + /// Authorize source sealing for the exact durable pre-copy. A retry can + /// observe the same authorization after the transition has advanced. + pub async fn authorize_transition_seal( + &self, + principal: &str, + namespace: &str, + operation: &str, + request: &RaftFleetTransitionSealRequest, + timeout: Duration, + ) -> Result { + request.validate(namespace, operation)?; + self.check_catalog(&request.catalog_id)?; + let directory = self.managed_namespace(namespace)?; + let operation = ProposalId::new(records::id(operation)?) + .map_err(|_| RaftFleetFailure::invalid("operation ID"))?; + let turn = Turn::new(&self.domain, timeout)?; + let transition = self + .authorize_transition_operator( + directory, + operation, + principal, + &request.plan_digest, + turn.remaining()?, + ) + .await?; + let journal = transition + .journal + .as_ref() + .ok_or(FleetCatalogError::Conflict)?; + if journal.preparation().map(hex::encode).as_deref() + != Some(request.preparation_digest.as_str()) + { + return Err(FleetCatalogError::Conflict.into()); + } + let phase = match journal.phase() { + TransitionPhase::Prepared => self + .catalog + .authorize_managed_transition_seal( + &transition.request, + directory, + &transition.plan, + turn.remaining()?, + ) + .await? + .phase(), + TransitionPhase::Planned | TransitionPhase::Cancelled => { + return Err(FleetCatalogError::Conflict.into()); + } + phase => phase, + }; + Ok(RaftFleetTransitionSealAuthorization { + request: request.clone(), + phase, + }) + } +} + +/// Select the child ranges for an exact verified transfer archive. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct RaftFleetTransitionPublishRequest { + pub catalog_id: String, + pub namespace_id: String, + pub operation_id: String, + pub request_id: String, + pub plan_digest: String, + pub archive_digest: String, +} + +impl RaftFleetTransitionPublishRequest { + pub fn validate(&self, namespace: &str, operation: &str) -> Result<()> { + for digest in [&self.catalog_id, &self.plan_digest, &self.archive_digest] { + records::catalog_id(digest)?; + } + for id in [namespace, operation, &self.request_id] { + records::id(id)?; + } + if self.namespace_id != namespace || self.operation_id != operation { + return Err(RaftFleetFailure::invalid("transition selector mismatch")); + } + Ok(()) + } +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct RaftFleetTransitionPublication { + pub request: RaftFleetTransitionPublishRequest, + /// Original namespace directory revision, including on historical retry. + pub directory_revision: u64, + pub tree_digest: String, + pub publication_digest: String, +} + +impl RaftFleetControl { + /// Atomically publish the admitted child partition, or return its exact + /// immutable publication certificate after an uncertain response. + pub async fn publish_transition( + &self, + principal: &str, + namespace: &str, + operation: &str, + request: &RaftFleetTransitionPublishRequest, + timeout: Duration, + ) -> Result { + request.validate(namespace, operation)?; + self.check_catalog(&request.catalog_id)?; + let directory = self.managed_namespace(namespace)?; + let operation = ProposalId::new(records::id(operation)?) + .map_err(|_| RaftFleetFailure::invalid("operation ID"))?; + let turn = Turn::new(&self.domain, timeout)?; + let transition = self + .authorize_transition_operator( + directory, + operation, + principal, + &request.plan_digest, + turn.remaining()?, + ) + .await?; + let journal = transition + .journal + .as_ref() + .ok_or(FleetCatalogError::Conflict)?; + let archive = journal + .transfer_archive() + .ok_or(FleetCatalogError::Conflict)?; + if hex::encode(archive) != request.archive_digest { + return Err(FleetCatalogError::Conflict.into()); + } + let publication = match journal.phase() { + TransitionPhase::Archived => { + self.catalog + .publish_managed_transfer( + &transition, + directory, + &self.archive_pool, + turn.remaining()?, + ) + .await? + } + TransitionPhase::Published | TransitionPhase::Active | TransitionPhase::Retired => self + .domain + .run(Box::pin(crate::asyncrt::timeout( + turn.remaining()?, + Box::pin(directory.publication(&transition.plan)), + ))) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(|_| FleetCatalogError::Unconfirmed)? + .map_err(|error| match error { + crate::raft::catalog::CatalogError::Unconfirmed => { + FleetCatalogError::Unconfirmed + } + error => FleetCatalogError::from(error), + })? + .ok_or(FleetCatalogError::Corrupt("missing transfer publication"))?, + _ => return Err(FleetCatalogError::Conflict.into()), + }; + if publication.operation() != operation || publication.transfer_archive() != Some(archive) { + return Err(FleetCatalogError::Corrupt("transfer publication binding").into()); + } + let tree = publication.tree_digest().ok_or(FleetCatalogError::Corrupt( + "empty transfer publication tree", + ))?; + Ok(RaftFleetTransitionPublication { + request: request.clone(), + directory_revision: publication.revision(), + tree_digest: hex::encode(tree), + publication_digest: hex::encode(publication.journal_digest()), + }) + } +} + +/// Activate one exact child on the operator endpoint's managed voter. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct RaftFleetTransitionActivationRequest { + pub catalog_id: String, + pub namespace_id: String, + pub operation_id: String, + pub request_id: String, + pub plan_digest: String, + pub group_id: String, + pub incarnation_id: String, +} + +impl RaftFleetTransitionActivationRequest { + pub fn validate(&self, namespace: &str, operation: &str) -> Result<()> { + records::catalog_id(&self.catalog_id)?; + records::catalog_id(&self.plan_digest)?; + for id in [ + namespace, + operation, + &self.request_id, + &self.group_id, + &self.incarnation_id, + ] { + records::id(id)?; + } + if self.namespace_id != namespace || self.operation_id != operation { + return Err(RaftFleetFailure::invalid("transition selector mismatch")); + } + Ok(()) + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct RaftFleetTransitionActivationPosition { + pub index: u64, + pub term: u64, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct RaftFleetTransitionActivation { + pub request: RaftFleetTransitionActivationRequest, + pub phase: TransitionPhase, + /// Absent while this call has only a provisional proposal. + pub applied: Option, +} + +impl RaftFleetControl { + /// Submit or reconcile an exact child activation. A configured voter can + /// commit the operation; an already staged proof can be reconciled by any + /// controller that can reach the Fleet catalog. + pub async fn activate_transition_child( + &self, + principal: &str, + namespace: &str, + operation: &str, + request: &RaftFleetTransitionActivationRequest, + timeout: Duration, + ) -> Result { + request.validate(namespace, operation)?; + self.check_catalog(&request.catalog_id)?; + let directory = self.managed_namespace(namespace)?; + let operation = ProposalId::new(records::id(operation)?) + .map_err(|_| RaftFleetFailure::invalid("operation ID"))?; + let group = GroupId::new(records::id(&request.group_id)?) + .map_err(|_| RaftFleetFailure::invalid("child group ID"))?; + let incarnation = IncarnationId::new(records::id(&request.incarnation_id)?) + .map_err(|_| RaftFleetFailure::invalid("child incarnation ID"))?; + let config = directory.namespace_config(); + let history = GroupHistory { + cluster: config.cluster, + namespace: config.namespace, + group, + incarnation, + }; + let turn = Turn::new(&self.domain, timeout)?; + let transition = self + .authorize_transition_operator( + directory, + operation, + principal, + &request.plan_digest, + turn.remaining()?, + ) + .await?; + let result = |phase, applied: Option| RaftFleetTransitionActivation { + request: request.clone(), + phase, + applied: applied.map(|position| RaftFleetTransitionActivationPosition { + index: position.index.get(), + term: position.term.get(), + }), + }; + if let Some((position, journal)) = self + .catalog + .resume_managed_transfer_activation( + &transition, + directory, + history, + None, + turn.remaining()?, + ) + .await? + { + return Ok(result(journal.phase(), Some(position))); + } + let observer = self + .node_observer + .as_ref() + .ok_or_else(|| RaftFleetFailure { + code: RaftFleetFailureCode::Unsupported, + outcome: RaftFleetOutcome::NotSubmitted, + message: "Use a managed endpoint with a voter for the selected child.".into(), + })?; + let voter = observer + .group(history) + .map_err(|error| activation_failure(CellGroupControllerError::Runtime(error)))?; + match voter + .activate_managed_transfer_child(&transition, turn.remaining()?) + .await + .map_err(activation_failure)? + { + CellGroupTransferActivation::Accepted(_) => { + Ok(result(TransitionPhase::Published, None)) + } + CellGroupTransferActivation::Applied(ack) => { + let (position, journal) = self + .catalog + .resume_managed_transfer_activation( + &transition, + directory, + history, + Some(*ack), + turn.remaining()?, + ) + .await? + .ok_or(FleetCatalogError::Unconfirmed)?; + Ok(result(journal.phase(), Some(position))) + } + } + } + + pub(super) async fn authorize_transition_operator( + &self, + directory: &RangeDirectory, + operation: ProposalId, + principal: &str, + plan_digest: &str, + timeout: Duration, + ) -> Result { + let transition = self + .catalog + .reopen_managed_transition(directory, operation, timeout) + .await? + .ok_or_else(missing_transition)?; + if transition.request.principal != principal { + return Err(RaftFleetFailure { + code: RaftFleetFailureCode::Forbidden, + outcome: RaftFleetOutcome::NotSubmitted, + message: "The admitted transition belongs to another principal.".into(), + }); + } + if plan_digest != hex::encode(transition.plan.digest()) { + return Err(FleetCatalogError::Conflict.into()); + } + Ok(transition) + } +} + +fn activation_failure(error: CellGroupControllerError) -> RaftFleetFailure { + let (code, message) = match error { + CellGroupControllerError::Runtime(RuntimeError::NotFound) => ( + RaftFleetFailureCode::Blocked, + "The endpoint has no local voter for the selected child.", + ), + CellGroupControllerError::Runtime(RuntimeError::NotLeader | RuntimeError::NotReady) => ( + RaftFleetFailureCode::Unavailable, + "Use the current ready voter leader or retry the exact request.", + ), + CellGroupControllerError::Runtime(RuntimeError::Overloaded) => ( + RaftFleetFailureCode::Busy, + "The voter is busy. Retry the exact request.", + ), + _ => ( + RaftFleetFailureCode::Unavailable, + "The child activation could not be confirmed. Reconcile the exact request.", + ), + }; + RaftFleetFailure { + code, + outcome: RaftFleetOutcome::Unknown, + message: message.into(), + } +} + +fn missing_transition() -> RaftFleetFailure { + RaftFleetFailure { + code: RaftFleetFailureCode::NotFound, + outcome: RaftFleetOutcome::Unknown, + message: "The operation is absent at the observed Fleet revision.".into(), + } +} diff --git a/crates/cellar/fleet/management/raft/transition_admission.rs b/crates/cellar/fleet/management/raft/transition_admission.rs new file mode 100644 index 000000000..b6c6b0204 --- /dev/null +++ b/crates/cellar/fleet/management/raft/transition_admission.rs @@ -0,0 +1,541 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Reserve fresh children and admit one exact managed split or merge. + +use super::*; +use crate::raft::{ + catalog::{CatalogError, InitialMember}, + cell_groups::{CellGroupBirthSpec, TransitionPhase}, + fleet_catalog::TransitionOperatorStep, +}; +use cellar_logic::{ + cell_group::{CellId, GroupHistory, GroupRoute, RouteEpoch}, + cell_group_placement::{PartitionBasis, PartitionDecision}, + consensus::{GroupId, IncarnationId, MemberId}, +}; +use serde::{Deserialize, Serialize}; +use sha2::{Digest, Sha256}; +use std::collections::BTreeSet; + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct RaftFleetTransitionGroup { + pub group_id: String, + pub incarnation_id: String, +} + +impl RaftFleetTransitionGroup { + pub(super) fn history(&self, directory: &RangeDirectory) -> Result { + let config = directory.namespace_config(); + Ok(GroupHistory { + cluster: config.cluster, + namespace: config.namespace, + group: GroupId::new(records::id(&self.group_id)?) + .map_err(|_| RaftFleetFailure::invalid("transition group ID"))?, + incarnation: IncarnationId::new(records::id(&self.incarnation_id)?) + .map_err(|_| RaftFleetFailure::invalid("transition incarnation ID"))?, + }) + } +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case", deny_unknown_fields)] +pub enum RaftFleetTransitionPartition { + Split { + parent: RaftFleetTransitionGroup, + pivot_cell: String, + }, + Merge { + left: RaftFleetTransitionGroup, + right: RaftFleetTransitionGroup, + }, +} + +impl RaftFleetTransitionPartition { + fn parents(&self) -> Vec<&RaftFleetTransitionGroup> { + match self { + Self::Split { parent, .. } => vec![parent], + Self::Merge { left, right } => vec![left, right], + } + } +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct RaftFleetTransitionChild { + pub group: RaftFleetTransitionGroup, + pub request_id: String, + pub creation_id: String, + pub voters: [RaftFleetFirstGroupVoter; 3], +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct RaftFleetTransitionAdmissionRequest { + pub catalog_id: String, + pub namespace_id: String, + pub operation_id: String, + pub request_id: String, + pub expected_fleet_revision: u64, + pub expected_directory_digest: String, + pub class: String, + pub partition: RaftFleetTransitionPartition, + /// In key order for a split. Each child has its own exact reservation ID. + pub children: Vec, +} + +impl RaftFleetTransitionAdmissionRequest { + pub fn validate(&self, namespace: &str, operation: &str) -> Result<()> { + records::catalog_id(&self.catalog_id)?; + records::catalog_id(&self.expected_directory_digest)?; + records::revision(self.expected_fleet_revision)?; + records::id(namespace)?; + records::id(operation)?; + let children = match &self.partition { + RaftFleetTransitionPartition::Split { pivot_cell, .. } => { + CellId::new(pivot_cell) + .map_err(|_| RaftFleetFailure::invalid("split pivot cell"))?; + 2 + } + RaftFleetTransitionPartition::Merge { .. } => 1, + }; + if self.namespace_id != namespace + || self.operation_id != operation + || self.children.len() != children + || self.class.is_empty() + || self.class.len() > 128 + { + return Err(RaftFleetFailure::invalid("transition admission selector")); + } + let mut groups = BTreeSet::new(); + for group in self + .partition + .parents() + .into_iter() + .chain(self.children.iter().map(|child| &child.group)) + { + records::id(&group.incarnation_id)?; + if !groups.insert(records::id(&group.group_id)?) { + return Err(RaftFleetFailure::invalid( + "transition group IDs must be distinct", + )); + } + } + let mut requests = BTreeSet::from([records::id(&self.request_id)?]); + for child in &self.children { + records::id(&child.creation_id)?; + if !requests.insert(records::id(&child.request_id)?) { + return Err(RaftFleetFailure::invalid( + "transition request IDs must be distinct", + )); + } + let nodes = child + .voters + .iter() + .map(|voter| records::id(&voter.node_id)) + .collect::>>()?; + if nodes.len() != 3 + || child.voters[0].member_id == 0 + || child + .voters + .windows(2) + .any(|pair| pair[0].member_id >= pair[1].member_id) + { + return Err(RaftFleetFailure::invalid( + "transition voters must be distinct and ordered", + )); + } + } + Ok(()) + } + + fn child_proposals( + &self, + directory: &RangeDirectory, + parents: &[GroupRoute], + ) -> Result> { + let expected = self + .partition + .parents() + .into_iter() + .map(|parent| parent.history(directory)) + .collect::>>()?; + if !parents.iter().map(|route| route.history).eq(expected) { + return Err(FleetCatalogError::Conflict.into()); + } + let ranges = match &self.partition { + RaftFleetTransitionPartition::Split { pivot_cell, .. } => { + let cell = CellId::new(pivot_cell) + .map_err(|_| RaftFleetFailure::invalid("split pivot cell"))?; + let pivot = crate::application::consensus::cell_routing_key( + parents[0].history.namespace, + &cell, + ); + let (left, right) = parents[0].range.split(pivot).map_err(|_| { + RaftFleetFailure::invalid("split pivot must be inside the parent range") + })?; + vec![left, right] + } + RaftFleetTransitionPartition::Merge { .. } => { + vec![parents[0].range.merge(&parents[1].range).map_err(|_| { + RaftFleetFailure::invalid("merge parents must be adjacent and ordered") + })?] + } + }; + let epoch = parents + .iter() + .map(|route| route.epoch.get()) + .max() + .and_then(|epoch| epoch.checked_add(1)) + .and_then(|epoch| RouteEpoch::new(epoch).ok()) + .ok_or_else(|| RaftFleetFailure::invalid("transition route epoch"))?; + self.children + .iter() + .zip(ranges) + .map(|(child, range)| { + let voters = child + .voters + .iter() + .map(|voter| { + Ok(InitialMember { + node: NodeId::new(records::id(&voter.node_id)?) + .map_err(|_| RaftFleetFailure::invalid("transition voter node"))?, + member: MemberId::new(voter.member_id).map_err(|_| { + RaftFleetFailure::invalid("transition voter member") + })?, + }) + }) + .collect::>()?; + Ok(CellGroupBirthSpec { + history: child.group.history(directory)?, + range, + epoch, + creation: ProposalId::new(records::id(&child.creation_id)?) + .map_err(|_| RaftFleetFailure::invalid("transition child creation"))?, + voters, + learners: vec![], + }) + }) + .collect() + } +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct RaftFleetTransitionBasis { + pub catalog_id: String, + pub namespace_id: String, + pub fleet_revision: u64, + pub directory_revision: u64, + pub directory_digest: String, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct RaftFleetTransitionAdmission { + pub request_id: String, + pub operation_id: String, + pub plan_digest: String, + pub phase: TransitionPhase, +} + +impl RaftFleetControl { + pub async fn transition_basis( + &self, + namespace: &str, + timeout: Duration, + ) -> Result { + let directory = self.managed_namespace(namespace)?; + let turn = Turn::new(&self.domain, timeout)?; + let snapshot = self.catalog.read(turn.remaining()?).await?; + let view = self + .domain + .run(crate::asyncrt::timeout( + turn.remaining()?, + directory.observe(), + )) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(|_| FleetCatalogError::Unconfirmed)? + .map_err(directory_failure)?; + Ok(RaftFleetTransitionBasis { + catalog_id: self.catalog_id.clone(), + namespace_id: namespace.into(), + fleet_revision: snapshot.revision(), + directory_revision: view.revision(), + directory_digest: hex::encode(view.digest()), + }) + } + + /// Stage the exact plan, reconcile prior child reservations, then admit + /// the replacement with the existing disruption fence. + pub async fn admit_transition( + &self, + principal: &str, + namespace: &str, + operation: &str, + request: &RaftFleetTransitionAdmissionRequest, + timeout: Duration, + ) -> Result { + Box::pin(self.admit_transition_inner( + principal, + namespace, + operation, + request, + &[], + timeout, + )) + .await + } + + /// Admit the exact split or merge selected by a placement recommendation. + /// Fresh work requires live source quorum checks. A rooted plan can resume + /// its original reservations even after the observed leadership changes. + pub async fn admit_recommended_partition( + &self, + principal: &str, + request: &RaftFleetTransitionAdmissionRequest, + recommendation: PartitionDecision<'_>, + timeout: Duration, + ) -> Result { + let sources = match (&request.partition, recommendation) { + ( + RaftFleetTransitionPartition::Split { pivot_cell, .. }, + PartitionDecision::Split { basis, pivot, .. }, + ) if pivot.cell().as_str() == pivot_cell => vec![basis], + ( + RaftFleetTransitionPartition::Merge { .. }, + PartitionDecision::Merge { left, right }, + ) if left.placement_class == right.placement_class => vec![left, right], + _ => { + return Err(RaftFleetFailure::invalid( + "partition recommendation mismatch", + )) + } + }; + Box::pin(self.admit_transition_inner( + principal, + &request.namespace_id, + &request.operation_id, + request, + &sources, + timeout, + )) + .await + } + + async fn admit_transition_inner( + &self, + principal: &str, + namespace: &str, + operation: &str, + request: &RaftFleetTransitionAdmissionRequest, + sources: &[&PartitionBasis], + timeout: Duration, + ) -> Result { + request.validate(namespace, operation)?; + self.check_catalog(&request.catalog_id)?; + let directory = self.managed_namespace(namespace)?; + let operation = ProposalId::new(records::id(operation)?) + .map_err(|_| RaftFleetFailure::invalid("transition operation ID"))?; + let bytes = serde_json::to_vec(&(principal, request)) + .map_err(|_| RaftFleetFailure::invalid("transition admission payload"))?; + let digest: [u8; 32] = Sha256::digest(bytes).into(); + let step = TransitionOperatorStep { + operation, + principal: principal.into(), + request_digest: digest, + step: records::id(&request.request_id)?, + }; + let turn = Turn::new(&self.domain, timeout)?; + let mut snapshot = self.catalog.read(turn.remaining()?).await?; + let (plan, proposals) = if let Some(plan) = self + .catalog + .reopen_transition_operator_plan(directory, operation, digest, turn.remaining()?) + .await? + { + let (parents, children) = directory + .transition_routes(&plan) + .map_err(directory_failure)?; + let proposals = request.child_proposals(directory, &parents)?; + if self + .catalog + .prepare_transition_child_routes( + &snapshot, + directory, + &request.class, + proposals.clone(), + turn.remaining()?, + ) + .await? + != children + { + return Err(FleetCatalogError::Corrupt("operator transition child binding").into()); + } + (plan, proposals) + } else { + if snapshot.revision() != request.expected_fleet_revision { + return Err(FleetCatalogError::Conflict.into()); + } + let (view, parents) = self + .domain + .run(Box::pin(crate::asyncrt::timeout( + turn.remaining()?, + Box::pin(async { + let view = directory.observe().await.map_err(directory_failure)?; + if hex::encode(view.digest()) != request.expected_directory_digest { + return Err(FleetCatalogError::Conflict.into()); + } + let mut parents = Vec::new(); + for parent in request.partition.parents() { + let birth = directory + .materialize_selected_history(parent.history(directory)?) + .await + .map_err(directory_failure)?; + parents.push(birth.route().clone()); + } + Ok::<_, RaftFleetFailure>((view, parents)) + }), + ))) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(|_| FleetCatalogError::Unconfirmed)??; + if !sources.is_empty() { + if sources.len() != parents.len() + || sources.iter().zip(&parents).any(|(source, route)| { + source.route != *route || source.directory_digest != view.digest() + }) + { + return Err(FleetCatalogError::Conflict.into()); + } + let observer = self + .node_observer + .as_ref() + .ok_or_else(|| RaftFleetFailure { + code: RaftFleetFailureCode::Unsupported, + outcome: RaftFleetOutcome::NotSubmitted, + message: "Recommended admission requires a managed quorum observer.".into(), + })?; + for source in sources { + Box::pin(self.catalog.require_partition_source( + &snapshot, + directory, + observer, + source, + turn.remaining()?, + )) + .await?; + } + } + let proposals = request.child_proposals(directory, &parents)?; + let children = self + .catalog + .prepare_transition_child_routes( + &snapshot, + directory, + &request.class, + proposals.clone(), + turn.remaining()?, + ) + .await?; + let plan = directory + .prepare_replacement(&view, operation, &parents, &children) + .map_err(directory_failure)?; + self.catalog + .stage_transition_operator_plan( + &snapshot, + directory, + &step.identity(snapshot.revision()), + digest, + &plan, + turn.remaining()?, + ) + .await?; + (plan, proposals) + }; + + let identity = if let Some(admitted) = self + .catalog + .reopen_managed_transition(directory, operation, turn.remaining()?) + .await? + { + if admitted.request != step.identity(admitted.request.expected_revision) + || admitted.plan.digest() != plan.digest() + { + return Err(FleetCatalogError::PayloadConflict.into()); + } + admitted.request + } else { + let (_, children) = directory + .transition_routes(&plan) + .map_err(directory_failure)?; + self.domain + .run(crate::asyncrt::timeout( + turn.remaining()?, + directory.validate_managed_transition_input(&plan), + )) + .await + .map_err(|_| FleetCatalogError::Closed)? + .map_err(|_| FleetCatalogError::Unconfirmed)? + .map_err(directory_failure)?; + for ((child, proposal), route) in request.children.iter().zip(proposals).zip(children) { + let child_step = TransitionOperatorStep { + operation, + principal: principal.into(), + request_digest: digest, + step: records::id(&child.request_id)?, + }; + if self + .catalog + .reopen_transition_operator_child( + &snapshot, + directory, + &route, + &child_step, + turn.remaining()?, + ) + .await? + { + continue; + } + let identity = child_step.identity(snapshot.revision()); + let reserved = self + .catalog + .reserve_birth( + &snapshot, + &identity, + directory, + proposal, + &request.class, + turn.remaining()?, + ) + .await?; + if reserved.authorization.route() != &route { + return Err( + FleetCatalogError::Corrupt("operator child reservation route").into(), + ); + } + snapshot = reserved.commit.snapshot; + } + step.identity(snapshot.revision()) + }; + let journal = self + .catalog + .plan_managed_transition(&identity, directory, &plan, turn.remaining()?) + .await?; + Ok(RaftFleetTransitionAdmission { + request_id: request.request_id.clone(), + operation_id: request.operation_id.clone(), + plan_digest: hex::encode(plan.digest()), + phase: journal.phase(), + }) + } +} + +fn directory_failure(error: CatalogError) -> FleetCatalogError { + match error { + CatalogError::Unconfirmed => FleetCatalogError::Unconfirmed, + CatalogError::Conflict(_) | CatalogError::Fenced => FleetCatalogError::Conflict, + error => FleetCatalogError::from(error), + } +} diff --git a/crates/cellar/fleet/management/raft/transition_child.rs b/crates/cellar/fleet/management/raft/transition_child.rs new file mode 100644 index 000000000..c31b1c583 --- /dev/null +++ b/crates/cellar/fleet/management/raft/transition_child.rs @@ -0,0 +1,281 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Operator setup for a pending managed transfer child on one reserved voter. + +use super::*; +use crate::raft::{ + catalog::CatalogError, + cell_groups::{CellGroupControllerError, TransitionPhase}, + completion::RuntimeError, +}; +use serde::{Deserialize, Serialize}; + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum RaftFleetTransitionChildStep { + Install, + Attach, + Quorum, + InitializeArchive, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct RaftFleetTransitionChildSetupRequest { + pub catalog_id: String, + pub namespace_id: String, + pub operation_id: String, + pub request_id: String, + pub plan_digest: String, + pub child: RaftFleetTransitionGroup, + pub step: RaftFleetTransitionChildStep, +} + +impl RaftFleetTransitionChildSetupRequest { + pub fn validate(&self, namespace: &str, operation: &str) -> Result<()> { + records::catalog_id(&self.catalog_id)?; + records::catalog_id(&self.plan_digest)?; + for id in [ + namespace, + operation, + &self.request_id, + &self.child.group_id, + &self.child.incarnation_id, + ] { + records::id(id)?; + } + if self.namespace_id != namespace || self.operation_id != operation { + return Err(RaftFleetFailure::invalid("transition selector mismatch")); + } + Ok(()) + } +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(tag = "stage", rename_all = "snake_case", deny_unknown_fields)] +pub enum RaftFleetTransitionChildSetupResult { + Installed { + member: u64, + node_id: String, + disk_lineage: String, + storage_instance: String, + }, + Attached, + ArchiveInitialized, + Quorum { + index: u64, + term: u64, + phase: TransitionPhase, + }, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct RaftFleetTransitionChildSetup { + pub request: RaftFleetTransitionChildSetupRequest, + /// The exact local setup result or a selected historical quorum witness. + pub result: RaftFleetTransitionChildSetupResult, +} + +impl RaftFleetControl { + /// Set up the exact pending child on a reserved local voter. Install + /// independently replays the selected sealed archives. Attach requires + /// the local receipt and a durable install majority. Quorum selects a + /// fresh-term witness on the child leader or reopens its selected proof. + /// InitializeArchive binds the activated child's first archive identity. + pub async fn setup_transition_child( + &self, + principal: &str, + namespace: &str, + operation: &str, + request: &RaftFleetTransitionChildSetupRequest, + timeout: Duration, + ) -> Result { + request.validate(namespace, operation)?; + self.check_catalog(&request.catalog_id)?; + let directory = self.managed_namespace(namespace)?; + let operation = ProposalId::new(records::id(operation)?) + .map_err(|_| RaftFleetFailure::invalid("operation ID"))?; + let history = request.child.history(directory)?; + let turn = Turn::new(&self.domain, timeout)?; + let transition = self + .authorize_transition_operator( + directory, + operation, + principal, + &request.plan_digest, + turn.remaining()?, + ) + .await?; + let phase = transition.journal.as_ref().map(|journal| journal.phase()); + match (request.step, phase) { + (RaftFleetTransitionChildStep::Install, Some(TransitionPhase::SealAuthorized)) + | ( + RaftFleetTransitionChildStep::InitializeArchive, + Some(TransitionPhase::Active | TransitionPhase::Retired), + ) + | ( + RaftFleetTransitionChildStep::Install + | RaftFleetTransitionChildStep::Attach + | RaftFleetTransitionChildStep::Quorum, + Some( + TransitionPhase::Sealed + | TransitionPhase::Installed + | TransitionPhase::Archived, + ), + ) => {} + _ => return Err(FleetCatalogError::Conflict.into()), + } + let (_, children) = directory + .transition_routes(&transition.plan) + .map_err(FleetCatalogError::from)?; + let child = children + .into_iter() + .find(|route| route.history == history) + .ok_or(FleetCatalogError::Conflict)?; + let transition = Arc::new(transition); + let result = match request.step { + RaftFleetTransitionChildStep::Install => { + let observer = self.child_setup_voter()?; + let services = self.transition_preparation.as_ref().ok_or_else(|| { + RaftFleetFailure { + code: RaftFleetFailureCode::Unsupported, + outcome: RaftFleetOutcome::NotSubmitted, + message: "Child installation requires a trusted archive verifier on the management server.".into(), + } + })?; + let sources = self + .catalog + .replay_managed_sealed_sources( + directory, + operation, + &services.verifier, + turn.remaining()?, + ) + .await?; + let installed = observer + .install_managed_transfer_child( + transition, + sources, + history, + services.recovery.clone(), + turn.remaining()?, + ) + .await + .map_err(setup_failure)?; + RaftFleetTransitionChildSetupResult::Installed { + member: installed.member.get(), + node_id: hex::encode(installed.node.bytes()), + disk_lineage: hex::encode(installed.disk.bytes()), + storage_instance: hex::encode(installed.engine.bytes()), + } + } + RaftFleetTransitionChildStep::Attach => { + self.child_setup_voter()? + .attach_managed_transfer_child(transition, child, turn.remaining()?) + .await + .map_err(setup_failure)?; + RaftFleetTransitionChildSetupResult::Attached + } + RaftFleetTransitionChildStep::InitializeArchive => { + self.catalog + .initialize_managed_transfer_child_archive( + &transition, + directory, + history, + &self.archive_pool, + turn.remaining()?, + ) + .await?; + RaftFleetTransitionChildSetupResult::ArchiveInitialized + } + RaftFleetTransitionChildStep::Quorum => { + let selected = self + .catalog + .resume_managed_transfer_child_quorum( + &transition, + directory, + history, + None, + turn.remaining()?, + ) + .await?; + let (position, journal) = if let Some(selected) = selected { + selected + } else { + let proof = self + .child_setup_voter()? + .group(history) + .map_err(|error| setup_failure(CellGroupControllerError::Runtime(error)))? + .managed_pending_transfer_quorum(turn.remaining()?) + .await + .map_err(setup_failure)?; + self.catalog + .resume_managed_transfer_child_quorum( + &transition, + directory, + history, + Some(proof), + turn.remaining()?, + ) + .await? + .ok_or(FleetCatalogError::Unconfirmed)? + }; + RaftFleetTransitionChildSetupResult::Quorum { + index: position.index.get(), + term: position.term.get(), + phase: journal.phase(), + } + } + }; + Ok(RaftFleetTransitionChildSetup { + request: request.clone(), + result, + }) + } + + fn child_setup_voter(&self) -> Result<&RaftCells> { + self.node_observer.as_ref().ok_or_else(|| RaftFleetFailure { + code: RaftFleetFailureCode::Unsupported, + outcome: RaftFleetOutcome::NotSubmitted, + message: "Use a managed endpoint on a reserved voter for the selected child.".into(), + }) + } +} + +fn setup_failure(error: CellGroupControllerError) -> RaftFleetFailure { + let (code, message) = match error { + CellGroupControllerError::Catalog(error) => match error.as_ref() { + CatalogError::Conflict(_) | CatalogError::Fenced | CatalogError::NotInitialMember => { + return FleetCatalogError::Conflict.into(); + } + CatalogError::FleetAuthority(error) + if matches!(error.as_ref(), FleetCatalogError::Conflict) => + { + return FleetCatalogError::Conflict.into(); + } + CatalogError::FleetAuthority(error) + if matches!(error.as_ref(), FleetCatalogError::Overloaded) => + { + return FleetCatalogError::Overloaded.into(); + } + _ => ( + RaftFleetFailureCode::Unavailable, + "The destination setup could not be confirmed. Reconcile the exact request.", + ), + }, + CellGroupControllerError::Runtime(RuntimeError::Overloaded) => ( + RaftFleetFailureCode::Busy, + "The destination voter is busy. Retry the exact request.", + ), + _ => ( + RaftFleetFailureCode::Unavailable, + "The destination setup could not be confirmed. Reconcile the exact request.", + ), + }; + RaftFleetFailure { + code, + outcome: RaftFleetOutcome::Unknown, + message: message.into(), + } +} diff --git a/crates/cellar/fleet/management/raft/transition_seal.rs b/crates/cellar/fleet/management/raft/transition_seal.rs new file mode 100644 index 000000000..f11e7a515 --- /dev/null +++ b/crates/cellar/fleet/management/raft/transition_seal.rs @@ -0,0 +1,356 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Operator dispatch to the current managed parent voter. + +use super::*; +use crate::raft::{ + catalog::RangeDirectory, + cell_groups::{ + CellGroupControllerError, CellGroupSealOutcome, CellGroupSourceRetirement, TransitionPhase, + }, + completion::RuntimeError, + fleet_catalog::ReopenedManagedCellGroupTransition, +}; +use cellar_logic::cell_group::GroupHistory; +use serde::{Deserialize, Serialize}; + +/// Submit or observe the authorized seal for one exact parent history. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct RaftFleetTransitionSourceSealRequest { + pub catalog_id: String, + pub namespace_id: String, + pub operation_id: String, + pub request_id: String, + pub plan_digest: String, + pub source: RaftFleetTransitionGroup, +} + +impl RaftFleetTransitionSourceSealRequest { + pub fn validate(&self, namespace: &str, operation: &str) -> Result<()> { + records::catalog_id(&self.catalog_id)?; + records::catalog_id(&self.plan_digest)?; + for id in [ + namespace, + operation, + &self.request_id, + &self.source.group_id, + &self.source.incarnation_id, + ] { + records::id(id)?; + } + if self.namespace_id != namespace || self.operation_id != operation { + return Err(RaftFleetFailure::invalid("transition selector mismatch")); + } + Ok(()) + } +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct RaftFleetTransitionSourceCut { + pub index: u64, + pub term: u64, + pub authorization_digest: String, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct RaftFleetTransitionSourceSeal { + pub request: RaftFleetTransitionSourceSealRequest, + /// A missing cut means the voter has admitted a proposal. The exact + /// request must be retried to obtain its committed, durably applied cut. + pub applied: Option, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct RaftFleetTransitionSourceArchive { + pub request: RaftFleetTransitionSourceSealRequest, + /// The applied parent cut independently verified from its retained archive. + pub sealed: RaftFleetTransitionSourceCut, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct RaftFleetTransitionSourceRetirement { + pub request: RaftFleetTransitionSourceSealRequest, + pub phase: TransitionPhase, + /// Durable parent position, even while other parents remain unsettled. + pub applied: Option, +} + +enum SourceAction { + Seal, + Retire, +} + +impl RaftFleetControl { + /// Submit or reconcile a seal on a locally hosted, authorized parent. + /// The voter checks its current boot and applied membership before it can + /// propose. Reconcile a lost result using the same request before publication. + pub async fn seal_transition_source( + &self, + principal: &str, + namespace: &str, + operation: &str, + request: &RaftFleetTransitionSourceSealRequest, + timeout: Duration, + ) -> Result { + let turn = Turn::new(&self.domain, timeout)?; + let (_, history, transition) = self + .authorize_source_action( + principal, + namespace, + operation, + request, + &turn, + SourceAction::Seal, + ) + .await?; + let observer = self + .node_observer + .as_ref() + .ok_or_else(|| RaftFleetFailure { + code: RaftFleetFailureCode::Unsupported, + outcome: RaftFleetOutcome::NotSubmitted, + message: "Use a managed endpoint with a voter for the selected parent.".into(), + })?; + let voter = observer + .group(history) + .map_err(|error| source_failure(CellGroupControllerError::Runtime(error)))?; + let applied = match voter + .seal_managed_source(&transition, turn.remaining()?) + .await + .map_err(source_failure)? + { + CellGroupSealOutcome::Accepted(_) => None, + CellGroupSealOutcome::Applied(seal) => Some(source_cut(&seal)), + }; + Ok(RaftFleetTransitionSourceSeal { + request: request.clone(), + applied, + }) + } + + /// Verify and retain the selected parent's sealed archive. This can run + /// on an archive-enabled management endpoint without a local voter. + pub async fn capture_transition_source( + &self, + principal: &str, + namespace: &str, + operation: &str, + request: &RaftFleetTransitionSourceSealRequest, + timeout: Duration, + ) -> Result { + let turn = Turn::new(&self.domain, timeout)?; + let (directory, history, transition) = self + .authorize_source_action( + principal, + namespace, + operation, + request, + &turn, + SourceAction::Seal, + ) + .await?; + let services = self + .transition_preparation + .as_ref() + .ok_or_else(|| RaftFleetFailure { + code: RaftFleetFailureCode::Unsupported, + outcome: RaftFleetOutcome::NotSubmitted, + message: + "Source capture requires a trusted archive verifier on the management server." + .into(), + })?; + let hold = self + .catalog + .capture_managed_sealed_source( + directory, + transition.plan.operation(), + history, + &services.verifier, + turn.remaining()?, + ) + .await?; + let seal = hold + .marker() + .source_seal() + .ok_or(FleetCatalogError::Corrupt("verified source has no seal"))?; + Ok(RaftFleetTransitionSourceArchive { + request: request.clone(), + sealed: source_cut(seal), + }) + } + + /// Submit the parent's tombstone after child archive coverage is selected, + /// or reopen the exact acknowledgement without a local voter. + pub async fn retire_transition_source( + &self, + principal: &str, + namespace: &str, + operation: &str, + request: &RaftFleetTransitionSourceSealRequest, + timeout: Duration, + ) -> Result { + let turn = Turn::new(&self.domain, timeout)?; + let (directory, history, transition) = self + .authorize_source_action( + principal, + namespace, + operation, + request, + &turn, + SourceAction::Retire, + ) + .await?; + let result = |phase, applied: Option| { + RaftFleetTransitionSourceRetirement { + request: request.clone(), + phase, + applied: applied.map(|position| RaftFleetTransitionActivationPosition { + index: position.index.get(), + term: position.term.get(), + }), + } + }; + if let Some((position, journal)) = self + .catalog + .resume_managed_source_retirement( + &transition, + directory, + &self.archive_pool, + history, + None, + turn.remaining()?, + ) + .await? + { + return Ok(result(journal.phase(), Some(position))); + } + let observer = self + .node_observer + .as_ref() + .ok_or_else(|| RaftFleetFailure { + code: RaftFleetFailureCode::Unsupported, + outcome: RaftFleetOutcome::NotSubmitted, + message: "Use a managed endpoint with a voter for the selected parent.".into(), + })?; + let voter = observer + .group(history) + .map_err(|error| source_failure(CellGroupControllerError::Runtime(error)))?; + match voter + .retire_managed_source(&transition, turn.remaining()?) + .await + .map_err(source_failure)? + { + CellGroupSourceRetirement::Accepted(_) => Ok(result(TransitionPhase::Active, None)), + CellGroupSourceRetirement::Applied(ack) => { + let (position, journal) = self + .catalog + .resume_managed_source_retirement( + &transition, + directory, + &self.archive_pool, + history, + Some(*ack), + turn.remaining()?, + ) + .await? + .ok_or(FleetCatalogError::Unconfirmed)?; + Ok(result(journal.phase(), Some(position))) + } + } + } + + async fn authorize_source_action<'a>( + &'a self, + principal: &str, + namespace: &str, + operation: &str, + request: &RaftFleetTransitionSourceSealRequest, + turn: &Turn<'_>, + action: SourceAction, + ) -> Result<( + &'a RangeDirectory, + GroupHistory, + ReopenedManagedCellGroupTransition, + )> { + request.validate(namespace, operation)?; + self.check_catalog(&request.catalog_id)?; + let directory = self.managed_namespace(namespace)?; + let operation = ProposalId::new(records::id(operation)?) + .map_err(|_| RaftFleetFailure::invalid("operation ID"))?; + let history = request.source.history(directory)?; + let transition = self + .authorize_transition_operator( + directory, + operation, + principal, + &request.plan_digest, + turn.remaining()?, + ) + .await?; + let (parents, _) = directory + .transition_routes(&transition.plan) + .map_err(FleetCatalogError::from)?; + if !parents.iter().any(|route| route.history == history) + || !transition + .journal + .as_ref() + .is_some_and(|journal| match action { + SourceAction::Seal => matches!( + journal.phase(), + TransitionPhase::SealAuthorized + | TransitionPhase::Sealed + | TransitionPhase::Installed + | TransitionPhase::Archived + ), + SourceAction::Retire => { + matches!( + journal.phase(), + TransitionPhase::Active | TransitionPhase::Retired + ) && journal.child_archives().is_some() + } + }) + { + return Err(FleetCatalogError::Conflict.into()); + } + Ok((directory, history, transition)) + } +} + +fn source_cut(seal: &crate::raft::cell_groups::GroupSourceSeal) -> RaftFleetTransitionSourceCut { + RaftFleetTransitionSourceCut { + index: seal.position().index.get(), + term: seal.position().term.get(), + authorization_digest: hex::encode(seal.authorization()), + } +} + +fn source_failure(error: CellGroupControllerError) -> RaftFleetFailure { + let (code, message) = match error { + CellGroupControllerError::Runtime(RuntimeError::NotFound) => ( + RaftFleetFailureCode::Blocked, + "The endpoint has no local voter for the selected parent.", + ), + CellGroupControllerError::Runtime(RuntimeError::NotLeader | RuntimeError::NotReady) => ( + RaftFleetFailureCode::Unavailable, + "Use the current ready parent leader or retry the exact request.", + ), + CellGroupControllerError::Runtime(RuntimeError::Overloaded) => ( + RaftFleetFailureCode::Busy, + "The parent voter is busy. Retry the exact request.", + ), + _ => ( + RaftFleetFailureCode::Unavailable, + "The parent operation could not be confirmed. Reconcile the exact request.", + ), + }; + RaftFleetFailure { + code, + outcome: RaftFleetOutcome::Unknown, + message: message.into(), + } +} diff --git a/crates/cellar/fleet/management/raft/transition_verify.rs b/crates/cellar/fleet/management/raft/transition_verify.rs new file mode 100644 index 000000000..7b1fd12a2 --- /dev/null +++ b/crates/cellar/fleet/management/raft/transition_verify.rs @@ -0,0 +1,721 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Independent archive verification during a managed transfer. + +use super::*; +use crate::raft::{ + catalog::{CellGroupBirthSpec, DirectoryPublication, DisasterFenceAuthority}, + cell_groups::{ + CellGroupArchiveCoverage, CellGroupArchiveHold, CellGroupArchiveStore, + CellGroupTransferTarget, PreparedGroupManifest, PreparedRecoveryData, TransitionPhase, + }, + fleet_catalog::{ + FleetFencedTransitionRecovery, FleetRecoveryRangeReservation, FleetRecoverySeedQuorums, + FleetRecoveryVoterLineage, FleetTransitionRecoveryBarrier, + FleetTransitionRecoveryFenceRequest, FleetTransitionRecoveryIntent, + FleetTransitionRecoveryReplacement, ManagedRecoveryIntentBasis, ManagedRecoveryIntentInput, + PreparedManagedRecoveryFence, + }, +}; +use serde::{Deserialize, Serialize}; + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct RaftFleetTransitionArchiveVerification { + pub request: RaftFleetTransitionPrepareRequest, + pub archive_digest: String, + pub phase: TransitionPhase, +} + +/// Private replay state bound to a single observed Fleet and directory root. +/// Retaining this value keeps the local replay pins alive. The remote holds +/// still belong to the operator that acquired them. +pub struct RaftFleetVerifiedTransitionRecovery { + basis: RaftFleetTransitionRecoveryBasis, + intent_basis: ManagedRecoveryIntentBasis, + principal: String, + archives: Vec, + lineages: Vec, +} + +impl RaftFleetVerifiedTransitionRecovery { + pub fn basis(&self) -> &RaftFleetTransitionRecoveryBasis { + &self.basis + } + + pub fn archives(&self) -> &[CellGroupArchiveCoverage] { + &self.archives + } + + /// The fence must cover these Fleet and archive lineages, along with the + /// platform's complete inventory of possible old writers. + pub fn required_voter_lineages(&self) -> &[FleetRecoveryVoterLineage] { + &self.lineages + } +} + +/// A restored range bound to one durable replacement plan and recovery +/// genesis commitment. Its data and source pins survive until this value drops. +#[derive(Clone)] +pub struct RaftFleetPreparedRecoveryRange { + plan: [u8; 32], + data: PreparedRecoveryData, + engine: Option, +} + +impl RaftFleetPreparedRecoveryRange { + pub(crate) fn for_node( + plan: [u8; 32], + data: PreparedRecoveryData, + engine: cellar_logic::consensus::EngineInstanceId, + ) -> Self { + Self { + plan, + data, + engine: Some(engine), + } + } + + pub(crate) fn engine(&self) -> Option { + self.engine + } + + pub(crate) fn data(&self) -> &PreparedRecoveryData { + &self.data + } + + pub fn plan_digest(&self) -> [u8; 32] { + self.plan + } + + pub fn store(&self) -> &CellGroupArchiveStore { + &self.data.store + } + + pub fn manifest(&self) -> &PreparedGroupManifest { + &self.data.manifest + } + + pub fn commitment(&self) -> [u8; 32] { + self.data.commitment + } +} + +impl RaftFleetControl { + /// Replay the caller's retained hold for every selected range, in + /// `basis.archive_sources()` order. Every hold must still be the selected + /// archive head. This read does not lock topology or fence old voters. + /// An eventual fencing operation must revalidate these exact inputs. + pub async fn verify_transition_recovery_archives( + &self, + principal: &str, + basis: &RaftFleetTransitionRecoveryBasis, + holds: Vec, + timeout: Duration, + ) -> Result { + self.check_catalog(&basis.catalog_id)?; + let directory = self.managed_namespace(&basis.namespace_id)?; + let operation = ProposalId::new(records::id(&basis.operation_id)?) + .map_err(|_| RaftFleetFailure::invalid("operation ID"))?; + let turn = Turn::new(&self.domain, timeout)?; + let transition = self + .authorize_transition_operator( + directory, + operation, + principal, + &basis.plan_digest, + turn.remaining()?, + ) + .await?; + let observation = self + .catalog + .observe_managed_transition_recovery(directory, operation, turn.remaining()?) + .await?; + let current = RaftFleetTransitionRecoveryBasis::from_observation( + self.catalog_id(), + &basis.namespace_id, + operation, + &observation, + ); + if current != *basis { + return Err(FleetCatalogError::Conflict.into()); + } + let services = self.transition_preparation.as_ref().ok_or_else(|| RaftFleetFailure { + code: RaftFleetFailureCode::Unsupported, + outcome: RaftFleetOutcome::NotSubmitted, + message: "Recovery archive verification requires a trusted verifier on the management server.".into(), + })?; + let verified = self + .catalog + .verify_managed_transition_recovery_archives( + directory, + &observation, + &transition.request, + holds, + &services.verifier, + turn.remaining()?, + ) + .await?; + Ok(RaftFleetVerifiedTransitionRecovery { + basis: current, + intent_basis: verified.intent_basis, + principal: transition.request.principal, + archives: verified.archives, + lineages: verified.lineages, + }) + } + + /// Persist one exact recovery selection. Use a new operation ID and keep + /// its archive holds retained for the lifetime of the recovery. + /// An identical retry reopens the original immutable record. + pub async fn stage_transition_recovery_intent( + &self, + principal: &str, + operation: &str, + verified: &RaftFleetVerifiedTransitionRecovery, + timeout: Duration, + ) -> Result { + self.check_catalog(&verified.basis.catalog_id)?; + if principal != verified.principal { + return Err(RaftFleetFailure { + code: RaftFleetFailureCode::Forbidden, + outcome: RaftFleetOutcome::NotSubmitted, + message: "The verified recovery belongs to another principal.".into(), + }); + } + let directory = self.managed_namespace(&verified.basis.namespace_id)?; + let operation = ProposalId::new(records::id(operation)?) + .map_err(|_| RaftFleetFailure::invalid("recovery operation ID"))?; + Ok(self + .catalog + .stage_managed_transition_recovery_intent( + directory, + ManagedRecoveryIntentInput { + operation, + principal, + basis: &verified.intent_basis, + holds: verified + .archives + .iter() + .map(|proof| proof.hold().clone()) + .collect(), + lineages: &verified.lineages, + }, + &self.archive_pool, + timeout, + ) + .await?) + } + + /// Reopen an immutable recovery selection by operation ID after losing + /// the verifier process or the response to the staging call. + pub async fn reopen_transition_recovery_intent( + &self, + principal: &str, + namespace: &str, + operation: &str, + timeout: Duration, + ) -> Result> { + let directory = self.managed_namespace(namespace)?; + let operation = ProposalId::new(records::id(operation)?) + .map_err(|_| RaftFleetFailure::invalid("recovery operation ID"))?; + Ok(self + .catalog + .reopen_managed_transition_recovery_intent(directory, operation, principal, timeout) + .await?) + } + + /// Select the exact recovery intent against its original directory root. + /// The barrier refuses old startup and transfer mutation for its component. + /// A deployment controller must still stop and permanently fence old writers. + #[doc(hidden)] + pub async fn select_transition_recovery_barrier( + &self, + principal: &str, + namespace: &str, + operation: &str, + expected_intent: [u8; 32], + timeout: Duration, + ) -> Result { + let directory = self.managed_namespace(namespace)?; + let operation = ProposalId::new(records::id(operation)?) + .map_err(|_| RaftFleetFailure::invalid("recovery operation ID"))?; + Ok(self + .catalog + .select_managed_transition_recovery_barrier( + directory, + operation, + principal, + expected_intent, + &self.archive_pool, + timeout, + ) + .await?) + } + + /// Permanently fence the old execution scope after replaying the exact + /// selected archive cuts. Retries reopen and verify the durable receipt. + #[doc(hidden)] + pub async fn fence_transition_recovery( + &self, + principal: &str, + namespace: &str, + operation: &str, + barrier: FleetTransitionRecoveryBarrier, + authority: &dyn DisasterFenceAuthority, + timeout: Duration, + ) -> Result { + let directory = self.managed_namespace(namespace)?; + let operation = ProposalId::new(records::id(operation)?) + .map_err(|_| RaftFleetFailure::invalid("recovery operation ID"))?; + let turn = Turn::new(&self.domain, timeout)?; + let services = self + .transition_preparation + .as_ref() + .ok_or_else(|| RaftFleetFailure { + code: RaftFleetFailureCode::Unsupported, + outcome: RaftFleetOutcome::NotSubmitted, + message: "Recovery fencing requires a trusted verifier on the management server." + .into(), + })?; + let prepared = self + .catalog + .prepare_managed_recovery_fence( + directory, + operation, + principal, + barrier, + &services.verifier, + turn.remaining()?, + ) + .await?; + let request = prepared.request(); + let evidence = match prepared.evidence() { + Some(evidence) => evidence.to_vec(), + None => turn + .run(authority.fence(request)) + .await? + .map_err(FleetCatalogError::Storage)?, + }; + PreparedManagedRecoveryFence::check_evidence(&evidence)?; + turn.run(authority.verify(request, &evidence)) + .await? + .map_err(FleetCatalogError::Storage)?; + Ok(self + .catalog + .confirm_managed_recovery_fence(directory, prepared, evidence, turn.remaining()?) + .await?) + } + + /// Freeze fresh placement and range identities against the permanent + /// fence. Repeat with the same births or reopen the immutable record. + #[doc(hidden)] + pub async fn stage_transition_recovery_replacement( + &self, + namespace: &str, + fenced: &FleetFencedTransitionRecovery, + replacements: Vec, + timeout: Duration, + ) -> Result { + let directory = self.managed_namespace(namespace)?; + Ok(self + .catalog + .stage_managed_recovery_replacement( + directory, + fenced, + replacements, + &self.archive_pool, + timeout, + ) + .await?) + } + + #[doc(hidden)] + pub async fn reopen_transition_recovery_replacement( + &self, + namespace: &str, + fenced: &FleetFencedTransitionRecovery, + timeout: Duration, + ) -> Result> { + let directory = self.managed_namespace(namespace)?; + Ok(self + .catalog + .reopen_managed_recovery_replacement(directory, fenced, &self.archive_pool, timeout) + .await?) + } + + /// Reserve one exact replacement under the active recovery barrier. A + /// retry reconciles the rooted range and the originally selected voter boots. + #[doc(hidden)] + pub async fn reserve_transition_recovery_range( + &self, + namespace: &str, + fenced: &FleetFencedTransitionRecovery, + plan: &FleetTransitionRecoveryReplacement, + history: cellar_logic::cell_group::GroupHistory, + timeout: Duration, + ) -> Result { + let directory = self.managed_namespace(namespace)?; + Ok(self + .catalog + .reserve_managed_recovery_range( + directory, + fenced, + plan, + history, + &self.archive_pool, + timeout, + ) + .await?) + } + + /// Verify a durable seed majority for every replacement. An exact retry + /// reopens the selected receipts even if more voters have completed. + #[doc(hidden)] + pub async fn select_transition_recovery_seed_quorums( + &self, + namespace: &str, + fenced: &FleetFencedTransitionRecovery, + plan: &FleetTransitionRecoveryReplacement, + timeout: Duration, + ) -> Result { + let directory = self.managed_namespace(namespace)?; + Ok(self + .catalog + .select_managed_recovery_seed_quorums( + directory, + fenced, + plan, + &self.archive_pool, + timeout, + ) + .await?) + } + + /// Atomically select the seeded replacements while retaining the startup + /// fence. Repeat with the exact selected receipt set after a lost response. + #[doc(hidden)] + pub async fn publish_transition_recovery( + &self, + namespace: &str, + fenced: &FleetFencedTransitionRecovery, + plan: &FleetTransitionRecoveryReplacement, + selected: &FleetRecoverySeedQuorums, + timeout: Duration, + ) -> Result { + let directory = self.managed_namespace(namespace)?; + Ok(self + .catalog + .publish_managed_recovery_replacement( + directory, + fenced, + plan, + selected, + &self.archive_pool, + timeout, + ) + .await?) + } + + /// Reconcile a publication after controller restart using the original + /// recovery operation and selected replacement and seed digests. + #[doc(hidden)] + pub async fn reopen_transition_recovery_publication( + &self, + namespace: &str, + operation: &str, + principal: &str, + expected: ([u8; 32], [u8; 32]), + timeout: Duration, + ) -> Result> { + let directory = self.managed_namespace(namespace)?; + let operation = ProposalId::new(records::id(operation)?) + .map_err(|_| RaftFleetFailure::invalid("recovery operation ID"))?; + Ok(self + .catalog + .reopen_managed_recovery_publication(directory, operation, principal, expected, timeout) + .await?) + } + + /// Reopen the committed activation receipt from the selected recovery. + /// This read needs no running replacement voter and grants no serving authority. + #[doc(hidden)] + pub async fn reopen_transition_recovery_activation( + &self, + namespace: &str, + request: &crate::raft::fleet_catalog::FleetRecoveryAttachRequest, + history: cellar_logic::cell_group::GroupHistory, + timeout: Duration, + ) -> Result> { + let directory = self.managed_namespace(namespace)?; + Ok(self + .catalog + .reopen_managed_recovery_activation(directory, request, history, timeout) + .await?) + } + + /// Promote the next bounded batch of replacement histories after every + /// range has committed activation. Retries reopen the rooted cursor. + #[doc(hidden)] + pub async fn advance_transition_recovery_settlement( + &self, + namespace: &str, + request: &crate::raft::fleet_catalog::FleetRecoveryAttachRequest, + timeout: Duration, + ) -> Result { + let directory = self.managed_namespace(namespace)?; + Ok(self + .catalog + .advance_managed_recovery_settlement(directory, request, timeout) + .await?) + } + + /// Select the exact completed Fleet promotion in the directory and + /// supersede the original transition. Returns the rooted receipt digest. + #[doc(hidden)] + pub async fn record_transition_recovery_directory_completion( + &self, + namespace: &str, + request: &crate::raft::fleet_catalog::FleetRecoveryAttachRequest, + timeout: Duration, + ) -> Result<[u8; 32]> { + let directory = self.managed_namespace(namespace)?; + Ok(self + .catalog + .record_managed_recovery_directory_completion(directory, request, None, timeout) + .await?) + } + + /// Release the directory fence after the exact completion has been rooted. + /// Retained voters can then start under current Fleet and quorum authority. + #[doc(hidden)] + pub async fn release_transition_recovery_serving( + &self, + namespace: &str, + request: &crate::raft::fleet_catalog::FleetRecoveryAttachRequest, + completion: [u8; 32], + timeout: Duration, + ) -> Result<[u8; 32]> { + let directory = self.managed_namespace(namespace)?; + Ok(self + .catalog + .record_managed_recovery_directory_completion( + directory, + request, + Some((completion, &self.archive_pool)), + timeout, + ) + .await?) + } + + /// Retire a bounded batch of old histories after serving release. A + /// permanent fence is reverified before each mutation. The terminal CAS + /// also releases the original Fleet slot. + #[doc(hidden)] + pub async fn advance_transition_recovery_retirement( + &self, + namespace: &str, + request: &crate::raft::fleet_catalog::FleetRecoveryAttachRequest, + completion: [u8; 32], + authority: &dyn DisasterFenceAuthority, + timeout: Duration, + ) -> Result { + let directory = self.managed_namespace(namespace)?; + Ok(self + .catalog + .advance_managed_recovery_retirement(directory, request, completion, authority, timeout) + .await?) + } + + /// Copy one exact selected cut into dedicated fresh-history stores. This + /// does not install Raft state or authorize placement or serving. + #[doc(hidden)] + pub async fn prepare_transition_recovery_range( + &self, + namespace: &str, + fenced: &FleetFencedTransitionRecovery, + plan: &FleetTransitionRecoveryReplacement, + target: CellGroupTransferTarget, + timeout: Duration, + ) -> Result { + let directory = self.managed_namespace(namespace)?; + let turn = Turn::new(&self.domain, timeout)?; + let services = self + .transition_preparation + .as_ref() + .ok_or_else(|| RaftFleetFailure { + code: RaftFleetFailureCode::Unsupported, + outcome: RaftFleetOutcome::NotSubmitted, + message: "Recovery data preparation requires a trusted recovery pool.".into(), + })?; + if !plan.matches_fence(fenced) { + return Err(FleetCatalogError::PayloadConflict.into()); + } + let index = plan + .births() + .iter() + .position(|birth| birth.route() == target.store.route()) + .ok_or(FleetCatalogError::Conflict)?; + let source = fenced + .archives() + .get(index) + .ok_or(FleetCatalogError::Conflict)?; + let operation = plan.mutation().operation(); + let observe = || async { + let selected = self + .catalog + .reopen_managed_recovery_replacement( + directory, + fenced, + &self.archive_pool, + turn.remaining()?, + ) + .await? + .ok_or(FleetCatalogError::Unconfirmed)?; + if selected.digest() != plan.digest() { + return Err(FleetCatalogError::PayloadConflict.into()); + } + Ok::<_, RaftFleetFailure>(()) + }; + observe().await?; + let data = turn + .run(Box::pin(services.recovery.prepare_cut( + &self.domain, + source.clone(), + target, + plan.digest(), + operation, + ))) + .await? + .map_err(FleetCatalogError::from)?; + observe().await?; + Ok(RaftFleetPreparedRecoveryRange { + plan: plan.digest(), + data, + engine: None, + }) + } + + /// Rebuild the installed children from the retained sealed parent cuts, + /// then select the verified transfer archive. A retry can reopen the + /// selected archive after the directory has advanced. + pub async fn verify_transition_archive( + &self, + principal: &str, + namespace: &str, + operation: &str, + request: &RaftFleetTransitionPrepareRequest, + timeout: Duration, + ) -> Result { + request.validate(namespace, operation)?; + self.check_catalog(&request.catalog_id)?; + let directory = self.managed_namespace(namespace)?; + let operation = ProposalId::new(records::id(operation)?) + .map_err(|_| RaftFleetFailure::invalid("operation ID"))?; + let turn = Turn::new(&self.domain, timeout)?; + let transition = self + .authorize_transition_operator( + directory, + operation, + principal, + &request.plan_digest, + turn.remaining()?, + ) + .await?; + let journal = transition.journal.ok_or(FleetCatalogError::Conflict)?; + let journal = + if journal.phase() == TransitionPhase::Installed { + let services = self.transition_preparation.as_ref().ok_or_else(|| { + RaftFleetFailure { + code: RaftFleetFailureCode::Unsupported, + outcome: RaftFleetOutcome::NotSubmitted, + message: + "Archive verification requires a trusted verifier on the management server." + .into(), + } + })?; + self.catalog + .resume_managed_transfer_archive( + directory, + operation, + &services.verifier, + services.recovery.clone(), + turn.remaining()?, + ) + .await? + } else { + journal + }; + let digest = journal + .transfer_archive() + .ok_or(FleetCatalogError::Conflict)?; + Ok(RaftFleetTransitionArchiveVerification { + request: request.clone(), + archive_digest: hex::encode(digest), + phase: journal.phase(), + }) + } + + /// Select independently replayed archive coverage for the active children. + /// Repeating the request can return the historical selected digest. + pub async fn verify_transition_child_archives( + &self, + principal: &str, + namespace: &str, + operation: &str, + request: &RaftFleetTransitionPrepareRequest, + timeout: Duration, + ) -> Result { + request.validate(namespace, operation)?; + self.check_catalog(&request.catalog_id)?; + let directory = self.managed_namespace(namespace)?; + let operation = ProposalId::new(records::id(operation)?) + .map_err(|_| RaftFleetFailure::invalid("operation ID"))?; + let turn = Turn::new(&self.domain, timeout)?; + let transition = self + .authorize_transition_operator( + directory, + operation, + principal, + &request.plan_digest, + turn.remaining()?, + ) + .await?; + let journal = transition + .journal + .as_ref() + .ok_or(FleetCatalogError::Conflict)?; + let (digest, phase) = if let Some(digest) = journal.child_archives() { + (digest, journal.phase()) + } else { + if journal.phase() != TransitionPhase::Active { + return Err(FleetCatalogError::Conflict.into()); + } + let services = self.transition_preparation.as_ref().ok_or_else(|| RaftFleetFailure { + code: RaftFleetFailureCode::Unsupported, + outcome: RaftFleetOutcome::NotSubmitted, + message: "Child archive verification requires a trusted verifier on the management server.".into(), + })?; + let verified = self + .catalog + .verify_managed_transfer_child_archives( + &transition, + directory, + &services.verifier, + turn.remaining()?, + ) + .await?; + ( + verified + .child_archives() + .ok_or(FleetCatalogError::Unconfirmed)?, + verified.phase(), + ) + }; + Ok(RaftFleetTransitionArchiveVerification { + request: request.clone(), + archive_digest: hex::encode(digest), + phase, + }) + } +} diff --git a/crates/cellar/fleet/management/raft_stop.rs b/crates/cellar/fleet/management/raft_stop.rs index a3d2d5cc7..8d0d75ff2 100644 --- a/crates/cellar/fleet/management/raft_stop.rs +++ b/crates/cellar/fleet/management/raft_stop.rs @@ -1,7 +1,7 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. //! Bounded stop observations, without per-group identities or deletion authority. -use crate::application::consensus::RaftNodeShutdownReport; +use crate::application::consensus::{MutationOutcomes, RaftNodeShutdownReport}; use anyhow::{ensure, Result}; use serde::{Deserialize, Serialize}; @@ -34,7 +34,10 @@ pub struct RaftStopOwners { pub omitted_group_details: usize, pub groups_with_archive_lag: usize, pub groups_with_unknown_archive: usize, - pub unfinished_commands: usize, + pub mutation_outcomes: MutationOutcomes, + /// Calls, including queries, retained when public admission was sealed. + /// This is not a count of unresolved business mutations. + pub retained_calls_at_shutdown: usize, } /// Accepted transfer requests and observed old-leader releases are separate. @@ -84,7 +87,8 @@ impl From<&RaftNodeShutdownReport> for RaftStopProof { omitted_group_details: owners.omitted_groups, groups_with_archive_lag: owners.groups_with_archive_lag, groups_with_unknown_archive: owners.groups_with_unknown_archive, - unfinished_commands: owners.unfinished.len(), + mutation_outcomes: owners.mutation_outcomes, + retained_calls_at_shutdown: owners.retained_calls_at_shutdown, }, handoff, } @@ -105,11 +109,8 @@ impl RaftStopProof { ); ensure!( owners.omitted_group_details <= owners.groups - && owners.groups_with_unknown_archive <= owners.omitted_group_details - && owners.unfinished_commands - <= owners - .groups - .saturating_sub(owners.groups_with_unknown_archive) + && (owners.mutation_outcomes != MutationOutcomes::NoUnresolvedLocalProducers + || owners.retained_calls_at_shutdown == 0) && owners .groups_with_archive_lag .checked_add(owners.groups_with_unknown_archive) @@ -126,10 +127,7 @@ impl RaftStopProof { } = handoff { ensure!( - *groups_examined - <= owners - .groups - .saturating_sub(owners.groups_with_unknown_archive) + *groups_examined <= owners.groups && *unavailable_groups <= *groups_examined && *accepted <= *groups_examined && *accepted <= *attempts @@ -156,7 +154,7 @@ impl RaftStopProof { deadline_elapsed: false, .. }, - } => owners.unfinished_commands == 0, + } => owners.mutation_outcomes == MutationOutcomes::NoUnresolvedLocalProducers, _ => false, } } diff --git a/crates/cellar/fleet/management/raft_stop_tests.rs b/crates/cellar/fleet/management/raft_stop_tests.rs index b96d58e4d..5ad3bd304 100644 --- a/crates/cellar/fleet/management/raft_stop_tests.rs +++ b/crates/cellar/fleet/management/raft_stop_tests.rs @@ -14,7 +14,8 @@ fn closed() -> RaftStopProof { omitted_group_details: 999_999, groups_with_archive_lag: 800_000, groups_with_unknown_archive: 100_000, - unfinished_commands: 0, + mutation_outcomes: MutationOutcomes::NoUnresolvedLocalProducers, + retained_calls_at_shutdown: 0, }, handoff: RaftStopHandoff::Observed { groups_examined: 10, @@ -61,7 +62,8 @@ fn raft_stop_refuses_incomplete_work_and_unknown_handoff() { } let mut proof = closed(); if let RaftStopProof::Closed { owners, .. } = &mut proof { - owners.unfinished_commands = 1; + owners.mutation_outcomes = MutationOutcomes::Unobserved; + owners.retained_calls_at_shutdown = 8; } assert!(!proof.complete()); let mut proof = closed(); @@ -86,6 +88,12 @@ fn raft_stop_refuses_incomplete_work_and_unknown_handoff() { #[test] fn raft_stop_refuses_fabricated_identity_counts_and_fields() { let valid = serde_json::to_value(closed()).unwrap(); + let mut invalid = valid.clone(); + invalid["owners"]["retained_calls_at_shutdown"] = serde_json::json!(8); + assert!(serde_json::from_value::(invalid) + .unwrap() + .validate() + .is_err()); for key in ["node", "disk", "process", "engine"] { let mut invalid = valid.clone(); invalid["owners"][key] = serde_json::json!(vec![0; 16]); @@ -116,3 +124,36 @@ fn raft_stop_refuses_fabricated_identity_counts_and_fields() { assert!(serde_json::from_value::(invalid).is_err()); assert!(serde_json::from_str::("{}").is_err()); } + +#[test] +fn shared_group_stop_never_infers_outcomes_from_call_count() { + for count in [0, 1, 8, 1_000_000] { + let mut proof = closed(); + if let RaftStopProof::Closed { owners, .. } = &mut proof { + owners.groups = 1; + owners.omitted_group_details = 0; + owners.groups_with_archive_lag = 0; + owners.groups_with_unknown_archive = 1; + owners.retained_calls_at_shutdown = count; + owners.mutation_outcomes = MutationOutcomes::Unobserved; + } + if let RaftStopProof::Closed { handoff, .. } = &mut proof { + *handoff = RaftStopHandoff::Observed { + groups_examined: 1, + attempts: 0, + accepted: 0, + leadership_released: 0, + unavailable_groups: 1, + deadline_elapsed: false, + }; + } + assert!(proof.validate().is_ok()); + assert!(!proof.complete()); + let mut value = serde_json::to_value(proof).unwrap(); + value["owners"] + .as_object_mut() + .unwrap() + .remove("mutation_outcomes"); + assert!(serde_json::from_value::(value).is_err()); + } +} diff --git a/crates/cellar/node/driver.rs b/crates/cellar/node/driver.rs index ce44de35d..f1a411d64 100644 --- a/crates/cellar/node/driver.rs +++ b/crates/cellar/node/driver.rs @@ -16,6 +16,8 @@ pub(super) struct RunPolicy { } pub(super) struct Channels { + pub shutdown_tx: mpsc::UnboundedSender, + pub shutdown_rx: mpsc::UnboundedReceiver, pub do_call_rx: mpsc::UnboundedReceiver, pub gate_rx: mpsc::UnboundedReceiver, pub facet_rx: mpsc::UnboundedReceiver, @@ -30,6 +32,7 @@ pub(super) struct Channels { impl Channels { /// Install dispatch once in the owning domain, after predecessor recovery. + /// The daemon's Raft stop reporter can start before the driver first polls. pub(super) fn install(websocket_rx: mpsc::UnboundedReceiver) -> Self { let (tx, do_call_rx) = mpsc::unbounded_channel(); crate::host_channels::set_do_call_tx(tx); @@ -49,7 +52,13 @@ impl Channels { crate::host_channels::set_asset_call_tx(tx); let (tx, outbound_ws_rx) = mpsc::unbounded_channel(); crate::host_channels::set_outbound_ws_tx(tx); + let (shutdown_tx, shutdown_rx) = mpsc::unbounded_channel(); + crate::asyncrt::services() + .raft_lifecycle() + .configure_shutdown(shutdown_tx.clone()); Self { + shutdown_tx, + shutdown_rx, do_call_rx, gate_rx, facet_rx, @@ -229,6 +238,8 @@ where .and_then(|runtime| runtime.replication()); let local_cache_max_bytes = policy.local_cache_max_bytes; let Channels { + shutdown_tx, + mut shutdown_rx, mut do_call_rx, mut gate_rx, mut facet_rx, @@ -241,10 +252,6 @@ where mut outbound_ws_rx, } = channels; let mut stop = std::pin::pin!(stop); - let (shutdown_tx, mut shutdown_rx) = mpsc::unbounded_channel(); - crate::asyncrt::services() - .raft_lifecycle() - .configure_shutdown(shutdown_tx.clone()); #[cfg(all(test, cellar_internal_tests))] if shutdown_accept_failure_test_active { let _ = shutdown_tx.send(ShutdownMode::Handoff); diff --git a/crates/cellar/node/embedded.rs b/crates/cellar/node/embedded.rs index 44199663d..11c3c4ba6 100644 --- a/crates/cellar/node/embedded.rs +++ b/crates/cellar/node/embedded.rs @@ -723,7 +723,6 @@ impl Node { .domain_token() .ok_or(crate::asyncrt::DomainClosed)?, data_dir: &self.info.data_dir, - node: &self.info.node, auth: self.raft_auth.clone(), } .start(config, services) @@ -748,7 +747,6 @@ impl Node { .domain_token() .ok_or(crate::asyncrt::DomainClosed)?, data_dir: &self.info.data_dir, - node: &self.info.node, auth: self.raft_auth.clone(), } .infrastructure(config, listeners, tls, None) diff --git a/crates/cellar/node/lifecycle.rs b/crates/cellar/node/lifecycle.rs index 385a9b685..755faa07d 100644 --- a/crates/cellar/node/lifecycle.rs +++ b/crates/cellar/node/lifecycle.rs @@ -205,7 +205,7 @@ mod tests { let outcome = NodeOutcome::Stopped(ShutdownReport { raft: Some(Box::new( crate::application::consensus::RaftNodeShutdownReport { - result: Err(crate::application::consensus::CommandFailure::Unavailable), + result: Err(crate::application::consensus::CommandFailure::Unavailable.into()), startup_incomplete: true, drain: None, }, diff --git a/crates/cellar/node/mod.rs b/crates/cellar/node/mod.rs index 1a2834ec3..b765e834a 100644 --- a/crates/cellar/node/mod.rs +++ b/crates/cellar/node/mod.rs @@ -84,6 +84,7 @@ pub use embedded::*; #[cfg(not(cellar_internal_tests))] pub use ingress::{IngressBody, IngressHandle, IngressResponse}; mod native; +mod native_namespace; mod native_raft; use native::Backend; pub use native::{CompiledApplications, NativeDaemon}; @@ -105,15 +106,16 @@ pub struct NodeRaftListeners { } /// Infrastructure handles owned by the embedded node's execution domain. The -/// host dispatcher consumes events, routes messages and supplies verified image -/// completions. Transport admission itself is never a write acknowledgement. -/// Recovered groups remain parked until catalog/bootstrap admission attaches them. +/// group controller consumes events, routes messages and supplies verified +/// whole-group manifest completions. Transport admission itself is never a write +/// acknowledgement. Recovered groups remain parked until their exact catalog +/// member permit and complete selected root authorize attachment. #[doc(hidden)] pub struct NodeRaftService { - pub log: crate::application::consensus::verification::ReplicaLog, - pub runtime: crate::application::consensus::verification::RaftRuntime, - pub events: crate::application::consensus::verification::RuntimeEvents, - pub transport: crate::application::consensus::verification::RaftTransport, + pub log: crate::raft::storage::ReplicaLog, + pub runtime: crate::raft::cell_groups::CellGroupRuntime, + pub events: crate::raft::cell_groups::CellGroupRuntimeEvents, + pub transport: crate::raft::transport::RaftTransport, } /// Let an admitted HTTP request finish, then close the transport even when a @@ -5159,7 +5161,7 @@ async fn async_main( } } let durability_owner = durability_owner.select(); - let channels = driver::Channels::install(websocket_rx); + let mut channels = driver::Channels::install(websocket_rx); // KV large values live in the fleet bucket. The cloneable handle follows // the Blob binding's startup pattern, so independent cells issue bucket I/O // independently instead of queueing behind one task. Blob I/O takes no @@ -5185,6 +5187,7 @@ async fn async_main( crate::asyncrt::select! { result = raft.start(&app, data_dir.clone(), node.clone(), fleet_bucket.is_some()) => result, _ = supplied_stop => Err(anyhow::anyhow!("Raft startup cancelled by host")), + _ = channels.shutdown_rx.recv() => Err(anyhow::anyhow!("Raft startup interrupted by fleet stop")), _ = sigterm.recv() => Err(anyhow::anyhow!("Raft startup interrupted by SIGTERM")), _ = sigint.recv() => Err(anyhow::anyhow!("Raft startup interrupted by SIGINT")), } diff --git a/crates/cellar/node/native.rs b/crates/cellar/node/native.rs index a190a235d..9b5c97140 100644 --- a/crates/cellar/node/native.rs +++ b/crates/cellar/node/native.rs @@ -1,7 +1,10 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. use super::*; -use crate::application::Application; +use crate::application::{ + consensus::{ClusterId, GroupHistory, NamespaceId, RaftCells}, + Application, +}; use crate::generation::GenerationId; use std::collections::{BTreeSet, HashMap}; @@ -13,6 +16,7 @@ pub(super) fn print_help() -> anyhow::Result<()> { USAGE:\n\ APPLICATION [NODE OPTIONS]\n\ APPLICATION cell|sql|kv|queue|outbox|blob [COMMAND OPTIONS]\n\ + APPLICATION --provision-sql-namespace PLAN.json --bucket NAME [OPTIONS]\n\n\ APPLICATION diagnose --bucket NAME [OPTIONS]\n\n\ NODE OPTIONS:\n\ --bucket [s3://|gs://|az://]NAME[/PREFIX] Fleet storage\n\ @@ -117,7 +121,7 @@ impl NativeDaemon { /// authentication. Raft-enabled daemons pin their deployment until process restart; /// live deployment reload does not yet rebind Raft definitions safely. pub fn raft_cells( - mut self, + self, script: impl Into, prepare: F, bind: impl FnOnce(crate::application::consensus::RaftCells) -> anyhow::Result<()> @@ -127,6 +131,61 @@ impl NativeDaemon { where F: FnOnce(super::NativeRaftNode) -> Fut + Send + 'static, Fut: std::future::Future> + Send + 'static, + { + self.raft_cells_with_startup(script, prepare, move |cells| { + std::future::ready(bind(cells)) + }) + } + + /// Serve a newly provisioned managed SQL namespace from an initial voter. + /// The daemon discovers the first full-range group, checks its rooted archive, + /// and establishes a fresh quorum before handing the selected history to + /// `bind`. The same path can reopen a retained initial voter while that + /// first group and membership remain selected. + /// + /// Use the operator workflow to admit capacity and provision the group. + /// `timeout` bounds archive observation, local attachment and quorum together. + /// A failed or interrupted barrier closes the installed host before exit. + pub fn managed_sql_namespace( + self, + script: impl Into, + namespace: (ClusterId, NamespaceId), + timeout: std::time::Duration, + prepare: F, + bind: Bind, + ) -> anyhow::Result + where + F: FnOnce(super::NativeRaftNode) -> Fut + Send + 'static, + Fut: std::future::Future> + Send + 'static, + Bind: FnOnce(RaftCells, GroupHistory) -> Bound + Send + 'static, + Bound: std::future::Future> + Send + 'static, + { + self.raft_cells_with_startup(script, prepare, move |cells| async move { + let (cluster, namespace) = namespace; + let history = cells + .start_managed_namespace_first_group(cluster, namespace, timeout) + .await?; + bind(cells, history).await + }) + } + + /// Run asynchronous application readiness after starting the node-owned + /// Raft transport and before opening public ingress. For the first managed + /// group, call [`RaftCells::start_managed_first_group`](crate::application::consensus::RaftCells::start_managed_first_group) + /// with the planned history. It can wait while the operator provisions the + /// group. Failure or cancellation closes the installed Raft owner and joins + /// its work. + pub fn raft_cells_with_startup( + mut self, + script: impl Into, + prepare: F, + bind: Bind, + ) -> anyhow::Result + where + F: FnOnce(super::NativeRaftNode) -> Fut + Send + 'static, + Fut: std::future::Future> + Send + 'static, + Bind: FnOnce(crate::application::consensus::RaftCells) -> Bound + Send + 'static, + Bound: std::future::Future> + Send + 'static, { anyhow::ensure!( self.raft.is_none(), @@ -140,7 +199,7 @@ impl NativeDaemon { self.raft = Some(super::native_raft::Startup { script, prepare: Box::new(move |node| Box::pin(prepare(node))), - bind: Box::new(bind), + bind: Box::new(move |cells| Box::pin(bind(cells))), }); Ok(self) } @@ -177,6 +236,12 @@ impl NativeDaemon { observer: Option, ) -> anyhow::Result<()> { let arguments: Vec<_> = std::env::args_os().skip(1).collect(); + if arguments + .first() + .is_some_and(|arg| arg == super::native_namespace::FLAG) + { + return super::native_namespace::run(self.applications, arguments.into_iter().skip(1)); + } if arguments .first() .is_some_and(|arg| arg == "--check-native-package") @@ -313,6 +378,31 @@ impl CompiledApplications { Ok(self) } + pub(super) fn namespace_class( + &self, + script: &str, + version: &str, + class: &str, + ) -> anyhow::Result> { + let factory = self + .factories + .get(&(script.to_owned(), version.to_owned())) + .ok_or_else(|| { + anyhow::anyhow!("no compiled application registered for {script} at {version}") + })?; + let application = factory()?; + anyhow::ensure!( + !application.supports_cell(class) && !application.supports_workflow(class), + crate::application::consensus::RegistrationError::UnsupportedFactory(class.into()) + ); + let definition = application.resolve_raft_class(class)?; + anyhow::ensure!( + definition.name() == class, + "compiled Raft class differs from the plan" + ); + Ok(definition) + } + fn check_package(&self, package: &crate::native_deploy::Package) -> anyhow::Result<()> { let executable = crate::native_deploy::current_executable_identity()?; // Validate every registration before invoking any application factory. diff --git a/crates/cellar/node/native_namespace.rs b/crates/cellar/node/native_namespace.rs new file mode 100644 index 000000000..d0702eeb8 --- /dev/null +++ b/crates/cellar/node/native_namespace.rs @@ -0,0 +1,182 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! One-shot namespace provisioning through a registered native Raft class. +#![allow(clippy::disallowed_methods)] + +use super::native::CompiledApplications; +use crate::cli_options::{FleetFlags, Storage}; +use crate::cli_output::{Format, Output}; +use crate::sdk::{consensus::*, ExecutionDomain}; +use anyhow::{ensure, Context as _, Result}; +use serde::{Deserialize, Serialize}; +use std::{ffi::OsString, io::Read, path::Path, time::Duration}; + +pub(super) const FLAG: &str = "--provision-sql-namespace"; +const PLAN_BYTES: u64 = 16 * 1024; +const TIMEOUT: Duration = Duration::from_secs(25); + +#[derive(Deserialize)] +#[serde(deny_unknown_fields)] +struct Plan { + catalog_id: String, + cluster_id: String, + incarnation_id: String, + coordination_prefix: String, + catalog_prefix: String, + namespace_id: String, + script: String, + version: String, + class: String, + principal: String, + request_id: String, + operation_id: String, + expected_revision: u64, + #[serde(default)] + image_storage: ImageStorage, +} + +#[derive(Clone, Copy, Default, Deserialize, Serialize)] +#[serde(rename_all = "snake_case")] +enum ImageStorage { + #[default] + Materialized, + Extents, +} + +impl From for RaftImageStorage { + fn from(storage: ImageStorage) -> Self { + match storage { + ImageStorage::Materialized => Self::Materialized, + ImageStorage::Extents => Self::Extents, + } + } +} + +fn identity(value: &str, label: &str) -> Result<[u8; N]> { + let mut bytes = [0; N]; + hex::decode_to_slice(value, &mut bytes).with_context(|| format!("invalid {label}"))?; + ensure!( + hex::encode(bytes) == value && bytes != [0; N], + "invalid {label}" + ); + Ok(bytes) +} + +fn inputs(mut arguments: impl Iterator) -> Result<(Plan, Storage)> { + let path = arguments + .next() + .context("--provision-sql-namespace requires a plan path")?; + let mut arguments = arguments.map(|arg| { + arg.into_string() + .map_err(|_| anyhow::anyhow!("storage flags require UTF-8")) + }); + let mut fleet = FleetFlags::default(); + while let Some(argument) = arguments.next().transpose()? { + ensure!( + fleet.consume(&argument, &mut |flag| { + arguments + .next() + .transpose()? + .with_context(|| format!("{flag} requires a value")) + })?, + "unexpected namespace provisioning option {argument}" + ); + } + let mut bytes = Vec::new(); + std::fs::File::open(Path::new(&path))? + .take(PLAN_BYTES + 1) + .read_to_end(&mut bytes)?; + ensure!( + bytes.len() as u64 <= PLAN_BYTES, + "namespace provision plan exceeds 16 KiB" + ); + let plan = serde_json::from_slice(&bytes).context("invalid namespace provision plan")?; + Ok((plan, fleet.resolve(FLAG)?)) +} + +pub(super) fn run( + applications: CompiledApplications, + arguments: impl Iterator, +) -> Result<()> { + let (plan, storage) = inputs(arguments)?; + rustls::crypto::ring::default_provider() + .install_default() + .ok(); + let runtime = tokio::runtime::Builder::new_multi_thread() + .enable_all() + .build()?; + let value = runtime.block_on(async { + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let result = domain + .run(provision(&domain, applications, plan, storage)) + .await; + domain.shutdown().await; + result? + })?; + Output::new(Format::Json).line(format_args!("{}", serde_json::to_string(&value)?)) +} + +async fn provision( + domain: &ExecutionDomain, + applications: CompiledApplications, + plan: Plan, + storage: Storage, +) -> Result { + let cluster = ClusterId::new(identity(&plan.cluster_id, "cluster ID")?)?; + let incarnation = IncarnationId::new(identity(&plan.incarnation_id, "fleet incarnation ID")?)?; + let namespace = NamespaceId::new(identity(&plan.namespace_id, "namespace ID")?)?; + let catalog_id: [u8; 32] = identity(&plan.catalog_id, "catalog ID")?; + let request = FleetRequestIdentity { + id: identity(&plan.request_id, "request ID")?, + operation: identity(&plan.operation_id, "operation ID")?, + principal: plan.principal, + expected_revision: plan.expected_revision, + }; + let class = applications.namespace_class(&plan.script, &plan.version, &plan.class)?; + let bucket = storage.open().await?; + let fleet = FleetCatalog::new( + domain, + bucket.clone(), + cluster, + incarnation, + &plan.coordination_prefix, + )?; + ensure!( + fleet.catalog_id()? == catalog_id, + "fleet catalog identity differs from the plan" + ); + let catalog = Catalog::new( + bucket.raft_store(), + CatalogOptions { + prefix: plan.catalog_prefix, + ..Default::default() + }, + )?; + let mut config = NamespaceConfig::new( + cluster, + namespace, + NamespaceMode::Raft, + class.schema_digest(), + ); + config.image_storage = plan.image_storage.into(); + let registered = fleet + .provision_managed_namespace( + &request, + &catalog, + config, + class.as_ref(), + DirectoryLimits::default(), + TIMEOUT, + ) + .await?; + Ok(serde_json::json!({ + "catalog_id": plan.catalog_id, + "namespace_id": plan.namespace_id, + "request_id": plan.request_id, + "operation_id": plan.operation_id, + "expected_revision": request.expected_revision, + "class": registered.class, + "image_storage": plan.image_storage, + "schema_digest": hex::encode(registered.namespace.schema_digest), + })) +} diff --git a/crates/cellar/node/native_raft.rs b/crates/cellar/node/native_raft.rs index cf2377b63..9bf47b0b9 100644 --- a/crates/cellar/node/native_raft.rs +++ b/crates/cellar/node/native_raft.rs @@ -28,7 +28,7 @@ pub struct NativeRaftCells { type Prepare = Box BoxFuture<'static, anyhow::Result> + Send>; -type Bind = Box anyhow::Result<()> + Send>; +type Bind = Box BoxFuture<'static, anyhow::Result<()>> + Send>; pub(super) struct Startup { pub script: String, @@ -83,7 +83,7 @@ impl Startup { .iter() .any(|origin| origin.peer.node != local_node); anyhow::ensure!( - !remote_routes || shared_peer_auth, + !(remote_routes || prepared.config.host.managed.is_some()) || shared_peer_auth, "multi-node Raft CLI startup requires a fleet bucket for shared peer authentication" ); let services = crate::asyncrt::services(); @@ -92,12 +92,12 @@ impl Startup { .domain_token() .ok_or(crate::asyncrt::DomainClosed)?, data_dir: &data_dir, - node: &node, auth: app.peer_auth.clone(), } .start( prepared.config, RaftCellsServices { + recovery: None, listeners: prepared.listeners, tls: prepared.tls, store: prepared.store, @@ -106,6 +106,6 @@ impl Startup { }, ) .await?; - (self.bind)(service) + (self.bind)(service).await } } diff --git a/crates/cellar/node/raft_startup.rs b/crates/cellar/node/raft_startup.rs index 0d5992449..ee2551cc9 100644 --- a/crates/cellar/node/raft_startup.rs +++ b/crates/cellar/node/raft_startup.rs @@ -9,7 +9,6 @@ use std::{path::Path, sync::Arc}; pub(super) struct Startup<'a> { pub domain: DomainToken, pub data_dir: &'a Path, - pub node: &'a str, pub auth: Arc, } @@ -49,177 +48,33 @@ impl Startup<'_> { crate::application::consensus::RaftCells, Option, )> { - use crate::application::consensus::{RaftCellPlacement, RegistrationError}; anyhow::ensure!( config.infrastructure.log.directory == self.data_dir.join("raft"), "Raft log must use this node's raft directory" ); - config.host.validate()?; - anyhow::ensure!( - config.host.extents.is_some() - || config.host.managed.as_ref().is_none_or(|managed| { - managed.namespaces.iter().all(|entry| { - entry.namespace.image_storage - != crate::application::consensus::RaftImageStorage::Extents - }) - }), - "managed extent namespace requires node extent storage" - ); - config.infrastructure.transport.validate()?; - config - .infrastructure - .transport - .snapshot_limits(config.host.execution.max_image_bytes)?; - anyhow::ensure!( - (!config.cells.is_empty() || config.host.managed.is_some()) - && config.cells.len() <= config.host.max_groups, - "invalid registered Raft cell count" - ); - let mut definitions = Vec::new(); - for cell in &config.cells { - anyhow::ensure!( - cell.birth.voters.len() <= 32 && cell.birth.learners.len() <= 32, - "registered birth membership exceeds the catalog bound" - ); - if let Some(managed) = &config.host.managed { - anyhow::ensure!( - managed - .namespaces - .iter() - .any(|n| n.namespace == cell.namespace && n.class == cell.class), - "cell is outside managed host capabilities" - ); - } - anyhow::ensure!( - cell.namespace.image_storage - != crate::application::consensus::RaftImageStorage::Extents - || matches!(cell.placement, RaftCellPlacement::QueryRouter { .. }) - || config.host.extents.is_some(), - "extent namespace requires node extent storage" - ); + if services.recovery.is_some() { anyhow::ensure!( - !services.application.supports_cell(&cell.class) - && !services.application.supports_workflow(&cell.class), - RegistrationError::UnsupportedFactory(cell.class.clone()) + config.host.managed.is_none() + && config.infrastructure.log.mode == crate::raft::options::LogOpenMode::Resume, + "recovered startup requires an existing seeded log and unmanaged recovery authority" ); - let class = services.application.resolve_raft_class(&cell.class)?; - class.validate_host(&config.host)?; - anyhow::ensure!( - cell.namespace.mode == cellar_logic::NamespaceMode::Raft - && cell.namespace.schema_digest == class.schema_digest(), - "Raft namespace mode/schema disagrees with registration" - ); - anyhow::ensure!( - cellar_logic::cell::valid_cell_scope(&cell.birth.scope.cell) - && cell - .birth - .scope - .cell - .split_once(':') - .is_some_and(|(name, _)| name == class.name()), - "Raft cell scope must name its registered class" - ); - definitions.push(class); - } - let mut declared_definitions = Vec::new(); - if let Some(managed) = &config.host.managed { - anyhow::ensure!( - managed.authorization.node() == config.infrastructure.log.node - && managed.authorization.disk() == config.infrastructure.log.disk, - "managed startup identity mismatch" - ); - for entry in &managed.namespaces { - anyhow::ensure!( - !services.application.supports_cell(&entry.class) - && !services.application.supports_workflow(&entry.class), - RegistrationError::UnsupportedFactory(entry.class.clone()) - ); - let class = services.application.resolve_raft_class(&entry.class)?; - class.validate_host(&config.host)?; - anyhow::ensure!( - class.schema_digest() == entry.namespace.schema_digest, - "managed compiled schema mismatch" - ); - declared_definitions.push(class); - } } - let (performance, registrations, declared_namespaces) = self + let (performance, prepared) = self .domain .run(async { let performance = crate::raft::performance::Registry::new(self.domain.clone()); let store = crate::raft::object_metrics::wrap(services.store, performance.clone()); - let catalog = crate::raft::catalog::Catalog::new(store, config.catalog)?; - let mut declared_namespaces = Vec::new(); - if let Some(managed) = &mut config.host.managed { - managed.authorization = managed - .authorization - .refresh(std::time::Duration::from_secs(5)) - .await?; - if managed.authorization.state() - == crate::raft::fleet_catalog::FleetNodeState::RestartPending - { - crate::asyncrt::services() - .raft_lifecycle() - .hold_startup_maintenance(); - } - for (entry, class) in managed.namespaces.iter().zip(declared_definitions) { - let namespace = catalog.existing_namespace(entry.namespace.clone()).await?; - namespace - .require_version_keys( - &services.versions, - u64::try_from(crate::asyncrt::wall_ms())?, - ) - .await?; - declared_namespaces - .push(crate::raft::host::NamespaceRegistration { namespace, class }); - } - } - let managed = config.host.managed.is_some(); - let mut registrations = Vec::new(); - for (cell, class) in config.cells.into_iter().zip(definitions) { - let namespace = match cell.placement { - RaftCellPlacement::Replica if !managed => { - catalog.open_namespace(cell.namespace).await? - } - _ => catalog.existing_namespace(cell.namespace).await?, - }; - let now_ms = u64::try_from(crate::asyncrt::wall_ms())?; - namespace - .require_version_keys(&services.versions, now_ms) - .await?; - let birth = - match cell.placement { - RaftCellPlacement::Replica if !managed => { - namespace.resolve_birth(cell.birth).await? - } - _ => { - let mut expected = cell.birth; - expected.voters.sort_by_key(|member| member.member); - expected.learners.sort_by_key(|member| member.member); - let birth = - namespace.birth(&expected.scope.cell).await?.ok_or_else( - || anyhow::anyhow!("existing member/route birth missing"), - )?; - anyhow::ensure!( - birth.scope() == &expected.scope - && birth.voters() == expected.voters.as_slice() - && birth.learners() == expected.learners.as_slice(), - "member/route disagrees with existing birth" - ); - birth - } - }; - registrations.push(crate::raft::host::Registration { - placement: cell.placement, - namespace, - birth, - class, - }); - } - Ok::<_, anyhow::Error>((performance, registrations, declared_namespaces)) + let prepared = crate::raft::group_host::prepare( + &mut config, + services.application.as_ref(), + store, + &services.versions, + ) + .await?; + Ok::<_, anyhow::Error>((performance, prepared)) }) .await??; - let service = self + let infrastructure = self .infrastructure( config.infrastructure, services.listeners, @@ -227,49 +82,41 @@ impl Startup<'_> { Some(performance), ) .await?; - let log = service.log.clone(); - let runtime = service.runtime.clone(); - let transport = service.transport.clone(); - let stop = config - .host - .managed - .as_ref() - .map(|managed| { - Ok::<_, anyhow::Error>(crate::raft::fleet_stop::Services { - authorization: managed.authorization.clone(), - log: log.clone(), - signer: crate::asyncrt::services().peer_probe_signer()?, - }) - }) - .transpose()?; + let log = infrastructure.log.clone(); + let runtime = infrastructure.runtime.clone(); + let transport = infrastructure.transport.clone(); let result = self .domain - .run(crate::raft::host::start( - service, - crate::raft::host::Registrations { - cells: registrations, - namespaces: declared_namespaces, - peers: config.peers, - origins: config.query_origins, - }, - self.data_dir.join("raft-cells"), - self.node.to_owned(), - config.host, - services.versions, - )) + .run(async { + let service = crate::raft::group_host::start( + infrastructure, + prepared, + self.data_dir.join("raft-cells"), + config.host, + config.shared, + services.versions, + services.recovery, + ) + .await?; + match service.fleet_stop_services() { + Ok(stop) => Ok((service, stop)), + Err(error) => { + let _ = service.close().await; + Err(anyhow::Error::new(error)) + } + } + }) .await; let result = match result { Ok(result) => result, Err(error) => Err(error.into()), }; if result.is_err() { - // Domain entry/cancellation can fail before the host's owned - // startup guard runs. Join the infrastructure here as well. let _ = runtime.close().await; transport.close().await; let _ = log.close().await; } - result.map(|service| (service, stop)) + result } pub(super) async fn infrastructure( @@ -279,7 +126,9 @@ impl Startup<'_> { tls: crate::application::consensus::verification::RaftTls, performance: Option>, ) -> anyhow::Result { - use crate::application::consensus::verification::{RaftRuntime, RaftTransport, ReplicaLog}; + use crate::raft::{ + cell_groups::CellGroupRuntime, storage::ReplicaLog, transport::RaftTransport, + }; anyhow::ensure!( config.log.directory == self.data_dir.join("raft"), "Raft log must use this node's raft directory" @@ -289,14 +138,14 @@ impl Startup<'_> { let log = ReplicaLog::open_observed(config.log, crate::asyncrt::fs(), performance) .await?; let (runtime, events) = - match RaftRuntime::start_parked(log.clone(), config.consensus).await { + match CellGroupRuntime::start(log.clone(), config.consensus).await { Ok(runtime) => runtime, Err(error) => { let _ = log.close().await; return Err(anyhow::Error::new(error)); } }; - let transport = match RaftTransport::start( + let transport = match RaftTransport::start_shared( runtime.clone(), self.auth.clone(), tls, diff --git a/crates/cellar/sdk.rs b/crates/cellar/sdk.rs index 31b5ef5a0..43939a706 100644 --- a/crates/cellar/sdk.rs +++ b/crates/cellar/sdk.rs @@ -186,7 +186,9 @@ //! ``` pub use crate::actor::{Actor, ActorIdentity, ActorOptions, ActorServices}; pub use crate::application::consensus::{ - CommandContext, OperationContract, OperationKind, QueryContext, RaftClass, RegistrationError, + CellGroupPartitionCell, CellGroupPartitionContext, CellGroupPartitionObservation, + CellGroupPartitionWriter, CellGroupWorkload, CommandContext, OperationContract, OperationKind, + QueryContext, QueryVersion, RaftClass, RegistrationError, }; pub use crate::application::filesystem::FilesystemLimits; pub use crate::application::limits::{ @@ -201,17 +203,42 @@ pub mod fleet_operations { RaftFleetCapacityAttestation, RaftFleetConnection, RaftFleetControl, RaftFleetEvacuationAcceptance, RaftFleetEvacuationDebt, RaftFleetEvacuationPhase, RaftFleetEvacuationProgress, RaftFleetEvacuationStatus, RaftFleetFailure, - RaftFleetFailureCode, RaftFleetMaintenanceAcceptance, RaftFleetMaintenanceAction, - RaftFleetMaintenanceCancel, RaftFleetMaintenancePhase, RaftFleetMaintenancePrepare, - RaftFleetMaintenanceReturnContext, RaftFleetMaintenanceStatus, RaftFleetMovement, - RaftFleetMovements, RaftFleetNamespace, RaftFleetNamespaceScope, RaftFleetNodeAcceptance, - RaftFleetNodeAction, RaftFleetNodeAdmission, RaftFleetNodeCancel, RaftFleetNodeCapacity, - RaftFleetNodeDescriptor, RaftFleetNodeOperation, RaftFleetNodeRemove, RaftFleetNodeState, - RaftFleetNodeStatus, RaftFleetNodeStop, RaftFleetOperation, RaftFleetOutcome, - RaftFleetRegisterNode, RaftFleetStatus, RaftFleetVerificationBlocker, - RaftFleetVerificationReason, RaftFleetVerificationStage, + RaftFleetFailureCode, RaftFleetFirstGroupAcceptance, RaftFleetFirstGroupMutation, + RaftFleetFirstGroupPlan, RaftFleetFirstGroupPublish, RaftFleetFirstGroupStage, + RaftFleetFirstGroupStandby, RaftFleetFirstGroupVoter, RaftFleetMaintenanceAcceptance, + RaftFleetMaintenanceAction, RaftFleetMaintenanceCancel, RaftFleetMaintenancePhase, + RaftFleetMaintenancePrepare, RaftFleetMaintenanceReturnContext, RaftFleetMaintenanceStatus, + RaftFleetMovement, RaftFleetMovements, RaftFleetNamespace, RaftFleetNamespaceScope, + RaftFleetNodeAcceptance, RaftFleetNodeAction, RaftFleetNodeAdmission, RaftFleetNodeCancel, + RaftFleetNodeCapacity, RaftFleetNodeDescriptor, RaftFleetNodeOperation, + RaftFleetNodeRemove, RaftFleetNodeState, RaftFleetNodeStatus, RaftFleetNodeStop, + RaftFleetOperation, RaftFleetOutcome, RaftFleetParkPhase, RaftFleetParkRequest, + RaftFleetParkStatus, RaftFleetPreparedRecoveryRange, RaftFleetRegisterNode, + RaftFleetStatus, RaftFleetTransitionActivation, RaftFleetTransitionActivationPosition, + RaftFleetTransitionActivationRequest, RaftFleetTransitionAdmission, + RaftFleetTransitionAdmissionRequest, RaftFleetTransitionArchiveVerification, + RaftFleetTransitionBasis, RaftFleetTransitionChild, RaftFleetTransitionChildSetup, + RaftFleetTransitionChildSetupRequest, RaftFleetTransitionChildSetupResult, + RaftFleetTransitionChildStep, RaftFleetTransitionGroup, RaftFleetTransitionPartition, + RaftFleetTransitionPreparation, RaftFleetTransitionPrepareRequest, + RaftFleetTransitionPublication, RaftFleetTransitionPublishRequest, + RaftFleetTransitionRecoveryBasis, RaftFleetTransitionResume, + RaftFleetTransitionResumeAcceptance, RaftFleetTransitionResumeStep, + RaftFleetTransitionSealAuthorization, RaftFleetTransitionSealRequest, + RaftFleetTransitionSourceArchive, RaftFleetTransitionSourceCut, + RaftFleetTransitionSourceRetirement, RaftFleetTransitionSourceSeal, + RaftFleetTransitionSourceSealRequest, RaftFleetTransitionStatus, + RaftFleetVerificationBlocker, RaftFleetVerificationReason, RaftFleetVerificationStage, + RaftFleetVerifiedTransitionRecovery, RaftFleetWakeStatus, + }; + pub use crate::raft::catalog::{ManagedRecoverySeedQuorum, ManagedRecoveryVoterSeed}; + pub use crate::raft::fleet_catalog::{ + FleetFencedTransitionRecovery, FleetRecoveryAttachRequest, FleetRecoveryRangeReservation, + FleetRecoveryRetirement, FleetRecoverySeedQuorums, FleetRecoverySettlement, + FleetRecoveryVoterLineage, FleetTopologyPolicy, FleetTransitionRecoveryBarrier, + FleetTransitionRecoveryFenceRequest, FleetTransitionRecoveryIntent, + FleetTransitionRecoveryReplacement, }; - pub use crate::raft::fleet_catalog::FleetTopologyPolicy; } pub use crate::assets::{AssetCache, AssetCacheOptions, AssetResolver}; #[cfg(not(cellar_internal_tests))] diff --git a/crates/cellar/tests/native_cell_groups_archive.rs b/crates/cellar/tests/native_cell_groups_archive.rs new file mode 100644 index 000000000..a05841333 --- /dev/null +++ b/crates/cellar/tests/native_cell_groups_archive.rs @@ -0,0 +1,1192 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! CG-06 complete recovery, S3 dependencies and physical log reclamation. +//! Fresh-incarnation disaster serving remains a separate required gate. +#![allow(clippy::disallowed_methods)] +// The shared engine fixture also defines fault/membership helpers selected by +// sibling integration targets and are not all selected by this archive gate. +#![allow(dead_code, unused_imports)] + +#[path = "support/cell_group_archive_store.rs"] +mod archive_store; +#[path = "support/cell_group_archive_binding.rs"] +mod binding_fixture; +#[path = "support/cell_group_checkpoint.rs"] +#[allow(dead_code)] +mod checkpoint_fixture; +#[path = "support/cell_group_disaster_fence.rs"] +mod disaster_fixture; +#[path = "support/raft_storage_fault.rs"] +#[allow(dead_code)] +mod fault; +#[path = "support/raft_ltx_fixture.rs"] +#[allow(dead_code)] +mod images; +#[path = "support/cell_group_archive_isolation.rs"] +mod isolation_fixture; +#[path = "support/cell_group_archive_publication.rs"] +mod publication_fixture; +#[path = "support/cell_group_rustfs.rs"] +mod rustfs; +#[path = "support/cell_group_archive_tail.rs"] +mod tail_fixture; + +#[path = "support/cell_group_runtime.rs"] +#[allow(dead_code, unused_imports)] +mod runtime_fixture; + +include!("support/cell_group_engine_fixture.rs"); + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn fenced_group_seed_new_incarnations() -> Result<()> { + for rooted in [false, true] { + Box::pin(disaster_fixture::registered_receipts( + rooted, + Arc::new(object_store::memory::InMemory::new()), + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS"] +async fn rustfs_fenced_group_seed_new_incarnations() -> Result<()> { + for rooted in [false, true] { + Box::pin(disaster_fixture::registered_receipts( + rooted, + rustfs::store("recovery-receipts", rooted)?, + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn group_recovery_publication_requires_seeded_quorum_activation() -> Result<()> { + for rooted in [false, true] { + Box::pin(disaster_fixture::activate_quorum( + rooted, + Arc::new(object_store::memory::InMemory::new()), + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS"] +async fn rustfs_group_recovery_publication_requires_seeded_quorum_activation() -> Result<()> { + for rooted in [false, true] { + Box::pin(disaster_fixture::activate_quorum( + rooted, + rustfs::store("recovery-activation", rooted)?, + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn group_recovery_genesis_survives_lost_reply_and_restart() -> Result<()> { + for rooted in [false, true] { + Box::pin(disaster_fixture::seed_engine( + rooted, + Arc::new(object_store::memory::InMemory::new()), + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS"] +async fn rustfs_group_recovery_genesis_survives_lost_reply_and_restart() -> Result<()> { + for rooted in [false, true] { + Box::pin(disaster_fixture::seed_engine( + rooted, + rustfs::store("recovery-genesis", rooted)?, + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn group_recovery_import_rebinds_complete_cut() -> Result<()> { + for rooted in [false, true] { + Box::pin(disaster_fixture::import( + rooted, + Arc::new(object_store::memory::InMemory::new()), + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn group_recovery_cancelled_import_retains_physical_admission() -> Result<()> { + for rooted in [false, true] { + Box::pin(disaster_fixture::cancel_import(rooted)).await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS"] +async fn rustfs_group_recovery_import_rebinds_complete_cut() -> Result<()> { + for rooted in [false, true] { + Box::pin(disaster_fixture::import( + rooted, + rustfs::store("recovery-import", rooted)?, + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn group_recovery_fence_resumes_exact_intent() -> Result<()> { + for rooted in [false, true] { + Box::pin(disaster_fixture::resume( + rooted, + Arc::new(object_store::memory::InMemory::new()), + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn group_recovery_topology_race_refuses_stale_cut() -> Result<()> { + for rooted in [false, true] { + Box::pin(disaster_fixture::topology_wins( + rooted, + Arc::new(object_store::memory::InMemory::new()), + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS"] +async fn rustfs_group_recovery_fence_intent_and_topology() -> Result<()> { + for rooted in [false, true] { + Box::pin(disaster_fixture::resume( + rooted, + rustfs::store("recovery-fence-resume", rooted)?, + )) + .await?; + Box::pin(disaster_fixture::topology_wins( + rooted, + rustfs::store("recovery-fence-race", rooted)?, + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn quorum_continues_bounded_s3_outage() -> Result<()> { + for rooted in [false, true] { + runtime_fixture::full_backend_outage( + rooted, + Arc::new(object_store::memory::InMemory::new()), + None, + ) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn group_archive_requires_all_dependencies() -> Result<()> { + for rooted in [false, true] { + runtime_fixture::archive_dependencies( + rooted, + Arc::new(object_store::memory::InMemory::new()), + ) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn group_reclaim_restart_keeps_history() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::physical_log_reclamation( + rooted, + Arc::new(object_store::memory::InMemory::new()), + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn archive_cut_upload_survives_quorum_progress() -> Result<()> { + for rooted in [false, true] { + publication_fixture::moving_upload_cut(rooted).await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn archive_catalog_binding_refuses_replacement_and_reset() -> Result<()> { + binding_fixture::identity().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn archive_tails_cover_enrollment_and_controls() -> Result<()> { + for rooted in [false, true] { + tail_fixture::enrollment_controls(rooted).await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn archive_tail_missing_and_forged_dependencies_refuse() -> Result<()> { + for rooted in [false, true] { + tail_fixture::dependencies(rooted).await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn archive_tail_warm_ltx_is_independently_replayed() -> Result<()> { + for rooted in [false, true] { + tail_fixture::warm_replay(rooted).await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn archive_tail_cancelled_replay_retains_physical_owners() -> Result<()> { + tail_fixture::physical_cancellation().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn archive_tail_extension_survives_local_prefix_compaction() -> Result<()> { + for rooted in [false, true] { + tail_fixture::extend_compacted(rooted).await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn archive_data_recovers_after_all_voter_disks_erased() -> Result<()> { + for rooted in [false, true] { + isolation_fixture::erased_voters(rooted).await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn archive_verification_isolated_from_serving_cut() -> Result<()> { + for rooted in [false, true] { + isolation_fixture::moving_cut(rooted).await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn archive_head_requires_complete_committed_source() -> Result<()> { + for rooted in [false, true] { + publication_fixture::complete_source_cut(rooted).await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn archive_gc_retains_holds_and_pending_parents() -> Result<()> { + for rooted in [false, true] { + publication_fixture::holds_and_pending_parents(rooted).await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn archive_publication_reconciles_exact_uncertain_intent() -> Result<()> { + for rooted in [false, true] { + publication_fixture::uncertain_publication(rooted).await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn archive_delayed_delete_cannot_erase_new_generation() -> Result<()> { + for rooted in [false, true] { + publication_fixture::delayed_delete(rooted).await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn archive_registry_refuses_reset_and_expired_intents() -> Result<()> { + for rooted in [false, true] { + publication_fixture::registry_identity_and_intents(rooted).await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn archive_upload_owners_bound_source_witness_lifetimes() -> Result<()> { + for rooted in [false, true] { + publication_fixture::source_owner_lifetimes(rooted).await?; + } + Ok(()) +} + +use cellar_ltx::{ + generation::ReadServices, + internal::{CacheLimits, EXTENT_BYTES}, + stream_snapshot::{ScratchLimits, ScratchPool}, +}; +use futures_util::TryStreamExt; +use object_store::{path::Path, ObjectStore}; +use std::sync::atomic::Ordering; + +struct ArchiveDestination { + store: CellGroupArchiveStore, + images: GenerationStore, + apply_images: CellGroupApplyImages, +} + +async fn destination(cluster: &Cluster) -> Result { + destination_with_filesystem(cluster, Arc::new(DirectFileSystem)).await +} + +async fn destination_with_filesystem( + cluster: &Cluster, + filesystem: Arc, +) -> Result { + let rooted = cluster + .voters + .first() + .is_some_and(|voter| voter.extents.is_some()); + destination_mode(cluster, rooted, filesystem).await +} + +async fn destination_mode( + cluster: &Cluster, + rooted: bool, + filesystem: Arc, +) -> Result { + let base = cluster._root.path().join("archive-data"); + let route = cluster.birth.route().clone(); + destination_at(cluster, rooted, filesystem, base, route).await +} + +async fn destination_at( + cluster: &Cluster, + rooted: bool, + filesystem: Arc, + base: std::path::PathBuf, + route: GroupRoute, +) -> Result { + cluster + .domain + .blocking(move || { + let pages = ManifestPageStore::open( + &base.join("pages"), + filesystem.clone(), + 1024, + 16 * 1024 * 1024, + 16, + )?; + let generations = + GenerationStore::open(base.join("images"), filesystem.clone(), Default::default())?; + let services = ReadServices { + now_unix_millis: Arc::new(|| 10), + fill_random: Arc::new(|bytes| bytes.fill(17)), + }; + let images = if rooted { + CellGroupApplyImages::Extents { + images: cellar_ltx::internal::NodeExtentStore::open( + base.join("extents"), + filesystem, + Default::default(), + )?, + services, + limits: Default::default(), + cache_directory: base.join("cache"), + cache_pool: ScratchPool::new(ScratchLimits { + max_bytes: 32 * 1024 * 1024, + max_files: 1, + })?, + cache: CacheLimits { + max_cached_bytes: EXTENT_BYTES, + max_cached_extents: 1, + }, + } + } else { + CellGroupApplyImages::Materialized { + images: generations.clone(), + services, + } + }; + let store = CellGroupArchiveStore::new( + route, + pages, + images.clone(), + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + )?; + Ok(ArchiveDestination { + store, + images: generations, + apply_images: images, + }) + })? + .await? +} + +fn receiver( + pool: &CellGroupCheckpointPool, + destination: &ArchiveDestination, + cluster: &Cluster, +) -> Result { + Ok(pool.receive_archive( + destination.store.clone(), + ScratchPool::new(ScratchLimits { + max_bytes: 32 * 1024 * 1024, + max_files: 1, + })?, + cluster._root.path().into(), + )?) +} + +fn source(pool: &CellGroupCheckpointPool, cluster: &Cluster) -> Result { + Ok(pool.capture( + &cluster.voters[0] + .log + .cell_group(&cluster.birth.route().history)?, + cluster.voters[0].owner.as_ref().unwrap().manifest(), + )?) +} + +async fn new_archive( + cluster: &Cluster, + store: Arc, + prefix: Path, + limits: CellGroupArchiveLimits, +) -> Result { + let label = prefix.as_ref().to_owned(); + let archive = CellGroupArchivePool::new(limits)?.for_group( + store, + prefix, + cluster.birth.route().history, + )?; + start_upload(cluster, &archive, &label).await +} + +async fn persist_archive_intent( + cluster: &Cluster, + intent: &T, +) -> Result { + let bytes = serde_json::to_vec(intent)?; + let name = hex::encode(Sha256::digest(&bytes)); + let directory = cluster._root.path().join("archive-intents"); + let loaded = cluster + .domain + .blocking(move || -> Result> { + use std::io::Write; + std::fs::create_dir_all(&directory)?; + let path = directory.join(name); + let mut file = std::fs::OpenOptions::new() + .create_new(true) + .write(true) + .open(&path)?; + file.write_all(&bytes)?; + file.sync_all()?; + std::fs::File::open(&directory)?.sync_all()?; + Ok(std::fs::read(path)?) + })? + .await??; + Ok(serde_json::from_slice(&loaded)?) +} + +async fn start_upload( + cluster: &Cluster, + archive: &CellGroupArchive, + label: &str, +) -> Result { + let binding = cluster._root.path().join(format!( + "archive-binding-{}.json", + hex::encode(Sha256::digest(label.as_bytes())) + )); + let existing = cluster + .domain + .blocking({ + let binding = binding.clone(); + move || std::fs::read(binding) + })? + .await?; + let registry = match existing { + Ok(bytes) => { + let creation: CellGroupArchiveCreation = serde_json::from_slice(&bytes)?; + archive.open_registry(&cluster.domain, &creation).await? + } + Err(error) if error.kind() == std::io::ErrorKind::NotFound => { + let prepared = archive + .prepare_registry_creation(&cluster.domain, cluster.birth.route().clone()) + .await?; + let creation = persist_archive_intent(cluster, prepared.intent()).await?; + assert_eq!(prepared.intent(), &creation); + let bytes = serde_json::to_vec(&creation)?; + cluster + .domain + .blocking(move || -> Result<()> { + use std::io::Write; + let mut file = std::fs::OpenOptions::new() + .create_new(true) + .write(true) + .open(&binding)?; + file.write_all(&bytes)?; + file.sync_all()?; + std::fs::File::open(binding.parent().context("binding directory")?)? + .sync_all()?; + Ok(()) + })? + .await??; + prepared.commit(&cluster.domain).await? + } + Err(error) => return Err(error.into()), + }; + let intent = registry.prepare_upload(&cluster.domain).await?; + let intent = persist_archive_intent(cluster, &intent).await?; + let ticket = registry.reserve(&cluster.domain, &intent).await?; + let owners = CellGroupArchiveUploadPool::new(Default::default())?; + let tails = CellGroupArchiveTailPool::new(Default::default())?; + Ok(registry + .open_upload(&cluster.domain, &owners, &ticket, &tails) + .await?) +} + +fn registry_id(path: &Path) -> Result<[u8; 32]> { + let relative = path + .as_ref() + .split_once("/registries/") + .context("registry path")? + .1; + let id = relative.split('/').next().context("registry ID")?; + let mut registry = [0; 32]; + hex::decode_to_slice(id, &mut registry)?; + Ok(registry) +} + +fn generation_prefix(path: &Path) -> Result<(&str, u64)> { + let (prefix, relative) = path + .as_ref() + .split_once("/generations/") + .context("generation prefix")?; + let generation = relative.split('/').next().context("generation")?.parse()?; + Ok((prefix, generation)) +} + +async fn index( + store: &dyn ObjectStore, + reference: &CellGroupArchiveReference, +) -> Result<(Path, serde_json::Value)> { + let suffix = format!( + "/indexes/{}-{}", + reference.length(), + hex::encode(reference.digest()) + ); + let objects: Vec<_> = store.list(None).try_collect().await?; + let path = objects + .into_iter() + .find(|m| { + m.location.as_ref().ends_with(&suffix) + && generation_prefix(&m.location) + .is_ok_and(|(_, generation)| generation == reference.generation()) + && registry_id(&m.location).is_ok_and(|id| id == reference.registry()) + }) + .context("checkpoint index object")? + .location; + let bytes = store.get(&path).await?.bytes().await?; + Ok((path, serde_json::from_slice(&bytes)?)) +} + +fn cell_objects(index_path: &Path, index: &serde_json::Value, cell: &str) -> Vec { + let (prefix, _) = generation_prefix(index_path).unwrap(); + index["chunks"] + .as_array() + .unwrap() + .iter() + .filter(|c| c["part"]["cell"] == cell) + .map(|chunk| { + let digest: Vec = + serde_json::from_value(chunk["object"]["digest"].clone()).unwrap(); + Path::from(format!( + "{prefix}/generations/{}/chunks/{}-{}", + chunk["object"]["generation"], + chunk["object"]["length"], + hex::encode(digest) + )) + }) + .collect() +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_archive_roundtrip_and_reuse() -> Result<()> { + for rooted in [false, true] { + let (mut cluster, mut expected) = + checkpoint_fixture::populate_images(rooted, true, false, 1, 8192).await?; + let store = archive_store::Store::new(); + let archive = new_archive( + &cluster, + store.clone(), + Path::from("qualification"), + Default::default(), + ) + .await?; + let sources = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + store.lose_put_response.store(true, Ordering::Release); + let original = archive + .upload_checkpoint(&cluster.domain, source(&sources, &cluster)?) + .await?; + let count = store.puts.load(Ordering::Acquire); + let repeat = archive + .upload_checkpoint(&cluster.domain, source(&sources, &cluster)?) + .await?; + assert_eq!(repeat, original); + assert_eq!( + store.puts.load(Ordering::Acquire), + count, + "unchanged cut must not PUT again" + ); + let (before_path, before_index) = index(store.as_ref(), &original).await?; + checkpoint_fixture::advance_partition( + &mut cluster, + "alpha", + &expected["alpha"], + rooted, + Some(2), + ) + .await?; + expected.insert( + "alpha".into(), + images::image(Some(&expected["alpha"]), "UPDATE value SET n=n+1")?, + ); + let latest = archive + .upload_checkpoint(&cluster.domain, source(&sources, &cluster)?) + .await?; + let (after_path, after_index) = index(store.as_ref(), &latest).await?; + for cell in ["beta", "cell-000"] { + let objects = cell_objects(&before_path, &before_index, cell); + assert!(!objects.is_empty()); + assert_eq!(objects, cell_objects(&after_path, &after_index, cell)); + } + assert_ne!(original, latest); + let marker = cluster.voters[0] + .owner + .as_ref() + .unwrap() + .manifest() + .marker() + .clone(); + let applier = destination(&cluster).await?; + let receive_pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + // Prove the independent receiver uses remote data after its source engine + // has closed; no borrowed source pin can satisfy this recovery. + drop(cluster.voters[0].owner.take()); + cluster.voters[0].log.close().await?; + let verified = archive + .verify_checkpoint( + &cluster.domain, + &latest, + receiver(&receive_pool, &applier, &cluster)?, + ) + .await?; + assert_eq!(verified.reference(), &latest); + assert_eq!(verified.checkpoint().marker(), &marker); + assert_eq!(archive.usage().operations, 0); + assert_eq!(sources.usage().sources, 0); + assert_eq!(receive_pool.usage().sources, 1); + for (cell, wanted) in expected { + let (_, pin) = verified + .checkpoint() + .manifest() + .pin_cell(&CellId::new(cell)?) + .context("recovered cell")?; + let bytes = cluster + .domain + .blocking(move || -> Result> { + Ok(match pin { + CellImagePin::Empty => Vec::new(), + CellImagePin::Materialized(pin) => pin + .open_reader()? + .read_exact_at(0, pin.state().length as usize)?, + CellImagePin::Extents(pin) => { + let mut bytes = Vec::new(); + for n in 0..pin.manifest().geometry().extent_count() { + bytes + .extend(pin.view().read_extent( + pin.manifest().extent(n).context("extent")?, + )?); + } + bytes + } + }) + })? + .await??; + assert_eq!(bytes, wanted); + } + drop(verified); + assert_eq!(receive_pool.usage().sources, 0); + cluster.close().await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_archive_limits_and_cancelled_receiver() -> Result<()> { + for rooted in [false, true] { + let (cluster, _) = + checkpoint_fixture::populate_images(rooted, true, false, 1, 8192).await?; + let store = archive_store::Store::new(); + let history = cluster.birth.route().history; + let sources = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + // Chunk and encoded-index bounds are independent, and neither may leave + // a selected index or a blocked physical producer behind on failure. + for limits in [ + CellGroupArchiveLimits { + max_chunks: 1, + ..Default::default() + }, + CellGroupArchiveLimits { + max_index_bytes: 1024, + ..Default::default() + }, + ] { + let limited = + new_archive(&cluster, store.clone(), Path::from("limited"), limits).await?; + assert!(matches!( + limited + .upload_checkpoint(&cluster.domain, source(&sources, &cluster)?) + .await, + Err(CellGroupArchiveError::Limit) + )); + assert_eq!(limited.usage().operations, 0); + assert_eq!(sources.usage().sources, 0); + let objects: Vec<_> = store + .list(Some(&Path::from("limited"))) + .try_collect() + .await?; + assert!(!objects + .iter() + .any(|m| m.location.as_ref().contains("/indexes/"))); + } + let node_pool = CellGroupArchivePool::new(CellGroupArchiveLimits { + max_operations: 1, + ..Default::default() + })?; + let archive = node_pool.for_group(store.clone(), Path::from("qualification"), history)?; + let other = node_pool.for_group( + store.clone(), + Path::from("qualification"), + GroupHistory { + group: GroupId::new([81; 16])?, + ..history + }, + )?; + let archive = start_upload(&cluster, &archive, "qualification").await?; + let reference = archive + .upload_checkpoint(&cluster.domain, source(&sources, &cluster)?) + .await?; + let (path, encoded) = index(store.as_ref(), &reference).await?; + let dependency = cell_objects(&path, &encoded, "beta") + .into_iter() + .last() + .context("held beta")?; + let applier = destination(&cluster).await?; + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let gate = store.hold(dependency); + let mut pending = Box::pin(archive.verify_checkpoint( + &cluster.domain, + &reference, + receiver(&pool, &applier, &cluster)?, + )); + assert!( + tokio::time::timeout(Duration::from_millis(50), &mut pending) + .await + .is_err() + ); + tokio::time::timeout(Duration::from_secs(5), gate.entered.notified()).await?; + assert_eq!(pool.usage().sources, 1); + assert_eq!(archive.usage().operations, 1); + let charged = archive.usage().retained_bytes; + assert!(charged > 0 && charged <= CellGroupArchiveLimits::default().max_retained_bytes); + // Node admission is shared by clones and both directions. Refusal must + // drop the rejected source without borrowing the active job's charge. + assert!(matches!( + archive + .clone() + .upload_checkpoint(&cluster.domain, source(&sources, &cluster)?) + .await, + Err(CellGroupArchiveError::Busy) + )); + assert_eq!(archive.usage().retained_bytes, charged); + assert_eq!(node_pool.usage().operations, 1); + let other_receivers = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + assert!(matches!( + other + .verify_checkpoint( + &cluster.domain, + &reference, + receiver(&other_receivers, &applier, &cluster)? + ) + .await, + Err(CellGroupArchiveError::Busy) + )); + assert_eq!(other_receivers.usage().sources, 0); + assert_eq!(other.usage().retained_bytes, charged); + assert_eq!(sources.usage().sources, 0); + drop(pending); + tokio::time::timeout(Duration::from_secs(5), async { + while archive.usage().operations != 0 || pool.usage().sources != 0 { + tokio::time::sleep(Duration::from_millis(1)).await; + } + }) + .await?; + store.clear(); + gate.release.notify_one(); + let verified = archive + .verify_checkpoint( + &cluster.domain, + &reference, + receiver(&pool, &applier, &cluster)?, + ) + .await?; + drop(verified); + // A blocked GET is bounded by the configured complete-object deadline. + let gate = store.hold(path); + let timed = new_archive( + &cluster, + store.clone(), + Path::from("qualification"), + CellGroupArchiveLimits { + io_timeout_ms: 20, + ..Default::default() + }, + ) + .await?; + assert!(matches!( + timed + .verify_checkpoint( + &cluster.domain, + &reference, + receiver(&pool, &applier, &cluster)? + ) + .await, + Err(CellGroupArchiveError::Timeout) + )); + assert_eq!(timed.usage().operations, 0); + assert_eq!(pool.usage().sources, 0); + store.clear(); + gate.release.notify_one(); + cluster.close().await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_archive_missing_or_corrupt_cell_refuses() -> Result<()> { + for rooted in [false, true] { + let (mut cluster, expected) = + checkpoint_fixture::populate_images(rooted, true, false, 1, 8192).await?; + checkpoint_fixture::advance_partition( + &mut cluster, + "alpha", + &expected["alpha"], + rooted, + Some(2), + ) + .await?; + let store = archive_store::Store::new(); + let archive = new_archive( + &cluster, + store.clone(), + Path::from("qualification"), + Default::default(), + ) + .await?; + let sources = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let reference = archive + .upload_checkpoint(&cluster.domain, source(&sources, &cluster)?) + .await?; + let (path, encoded) = index(store.as_ref(), &reference).await?; + let dependency = cell_objects(&path, &encoded, "beta") + .into_iter() + .last() + .context("beta dependency")?; + let bytes = store.get(&dependency).await?.bytes().await?; + let applier = destination(&cluster).await?; + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + store.delete(&dependency).await?; + assert!(matches!( + archive + .verify_checkpoint( + &cluster.domain, + &reference, + receiver(&pool, &applier, &cluster)? + ) + .await, + Err(CellGroupArchiveError::Invalid("missing archive object")) + )); + assert_eq!(pool.usage().sources, 0); + let mut corrupt = bytes.to_vec(); + corrupt[0] ^= 0x80; + store.inner.put(&dependency, corrupt.into()).await?; + assert!(matches!( + archive + .verify_checkpoint( + &cluster.domain, + &reference, + receiver(&pool, &applier, &cluster)? + ) + .await, + Err(CellGroupArchiveError::Invalid("object digest/length")) + )); + assert_eq!(pool.usage().sources, 0); + // Existing corrupt immutable payloads cannot be silently overwritten. + assert!(archive + .upload_checkpoint(&cluster.domain, source(&sources, &cluster)?) + .await + .is_err()); + store.inner.put(&dependency, bytes.into()).await?; + let verified = archive + .verify_checkpoint( + &cluster.domain, + &reference, + receiver(&pool, &applier, &cluster)?, + ) + .await?; + assert_eq!(verified.checkpoint().marker().root().cells(), 4); + drop(verified); + assert_eq!(archive.usage().retained_bytes, 0); + cluster.close().await?; + } + Ok(()) +} + +async fn forged_index( + store: &dyn ObjectStore, + original_path: &Path, + value: &serde_json::Value, +) -> Result { + let bytes = serde_json::to_vec(value)?; + let digest: [u8; 32] = Sha256::digest(&bytes).into(); + let reference = serde_json::from_value(serde_json::json!({ + "registry": registry_id(original_path)?, "generation": generation_prefix(original_path)?.1, "digest": digest, "length": bytes.len(), + }))?; + let prefix = original_path + .as_ref() + .split("/indexes/") + .next() + .context("index prefix")?; + let path = Path::from(format!( + "{prefix}/indexes/{}-{}", + bytes.len(), + hex::encode(digest) + )); + store.put(&path, bytes.into()).await?; + Ok(reference) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_archive_warm_cache_cannot_hide_wrong_extent_payload() -> Result<()> { + let (cluster, _) = checkpoint_fixture::populate_images(true, true, false, 1, 8192).await?; + let store = archive_store::Store::new(); + let archive = new_archive( + &cluster, + store.clone(), + Path::from("warm-source"), + Default::default(), + ) + .await?; + let sources = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let reference = archive + .upload_checkpoint(&cluster.domain, source(&sources, &cluster)?) + .await?; + let (path, mut encoded) = index(store.as_ref(), &reference).await?; + let applier = destination(&cluster).await?; + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let verified = archive + .verify_checkpoint( + &cluster.domain, + &reference, + receiver(&pool, &applier, &cluster)?, + ) + .await?; + drop(verified); + let CellGroupApplyImages::Extents { images, .. } = &applier.apply_images else { + anyhow::bail!("extent destination"); + }; + assert!(images.usage()?.objects > 0); + let chunk = encoded["chunks"] + .as_array_mut() + .context("chunks")? + .iter_mut() + .find(|chunk| chunk["part"]["kind"] == "Extent" && chunk["offset"] == 0) + .context("first streamed extent chunk")?; + let (prefix, _) = generation_prefix(&path)?; + let digest: Vec = serde_json::from_value(chunk["object"]["digest"].clone())?; + let old = Path::from(format!( + "{prefix}/generations/{}/chunks/{}-{}", + chunk["object"]["generation"], + chunk["object"]["length"], + hex::encode(digest) + )); + let mut corrupt = store.get(&old).await?.bytes().await?.to_vec(); + corrupt[100] ^= 0x80; + let digest: [u8; 32] = Sha256::digest(&corrupt).into(); + let new = Path::from(format!( + "{prefix}/generations/{}/chunks/{}-{}", + chunk["object"]["generation"], + corrupt.len(), + hex::encode(digest) + )); + store.put(&new, corrupt.into()).await?; + chunk["object"]["digest"] = serde_json::to_value(digest)?; + // The forged locator and object have correct hashes/lengths, while the + // announced extent identity still belongs to the original committed image. + let forged = forged_index(store.as_ref(), &path, &encoded).await?; + let cold = destination_at( + &cluster, + true, + Arc::new(DirectFileSystem), + cluster._root.path().join("cold-source-check"), + cluster.birth.route().clone(), + ) + .await?; + assert!(archive + .verify_checkpoint(&cluster.domain, &forged, receiver(&pool, &cold, &cluster)?) + .await + .is_err()); + assert_eq!(pool.usage().sources, 0); + assert!( + archive + .verify_checkpoint( + &cluster.domain, + &forged, + receiver(&pool, &applier, &cluster)? + ) + .await + .is_err(), + "warm local data hid an invalid remote extent payload" + ); + assert_eq!(pool.usage().sources, 0); + let recovered = archive + .verify_checkpoint( + &cluster.domain, + &reference, + receiver(&pool, &applier, &cluster)?, + ) + .await?; + assert_eq!(recovered.checkpoint().marker().root().cells(), 4); + drop(recovered); + cluster.close().await?; + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_archive_untrusted_index_and_cut_binding() -> Result<()> { + for rooted in [false, true] { + let (cluster, _) = + checkpoint_fixture::populate_images(rooted, true, false, 1, 8192).await?; + let store = archive_store::Store::new(); + let archive = new_archive( + &cluster, + store.clone(), + Path::from("qualification"), + Default::default(), + ) + .await?; + let sources = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let reference = archive + .upload_checkpoint(&cluster.domain, source(&sources, &cluster)?) + .await?; + let (path, original) = index(store.as_ref(), &reference).await?; + let group = cluster.voters[2] + .log + .cell_group(&cluster.birth.route().history)?; + let base = group.group_manifest()?.context("destination base")?; + let applier = destination(&cluster).await?; + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + for attack in 0..4 { + let mut value = original.clone(); + match attack { + 0 => value["domain"] = "cellar-single-cell-archive-1".into(), + 1 => value["chunks"][1]["offset"] = 1.into(), + 2 => { + let chunks = value["chunks"].as_array_mut().unwrap(); + chunks.retain(|chunk| chunk["part"]["cell"] != "beta"); + value["bytes"] = chunks + .iter() + .map(|c| c["object"]["length"].as_u64().unwrap()) + .sum::() + .into(); + } + 3 => value["marker"] = serde_json::to_value(&base)?, + _ => unreachable!(), + } + let forged = forged_index(store.as_ref(), &path, &value).await?; + let result = archive + .verify_checkpoint( + &cluster.domain, + &forged, + receiver(&pool, &applier, &cluster)?, + ) + .await; + match attack { + 0 => assert!(matches!( + result, + Err(CellGroupArchiveError::Invalid("checkpoint domain/history")) + )), + 1 => assert!(matches!(result, Err(CellGroupArchiveError::Invalid(_)))), + 2 => assert!(matches!(result, Err(CellGroupArchiveError::Checkpoint(_)))), + 3 => assert!(matches!( + result, + Err(CellGroupArchiveError::Invalid("archive checkpoint binding")) + )), + _ => unreachable!(), + } + assert_eq!(archive.usage().operations, 0); + assert_eq!(pool.usage().sources, 0); + assert_eq!(group.group_manifest()?.as_ref(), Some(&base)); + assert_eq!(group.checkpoint_manifest()?, None); + } + for length in [0, u64::MAX] { + let invalid = serde_json::from_value( + serde_json::json!({"registry": reference.registry(), "generation": reference.generation(), "digest": reference.digest(), "length": length}), + )?; + assert!(matches!( + archive + .verify_checkpoint( + &cluster.domain, + &invalid, + receiver(&pool, &applier, &cluster)? + ) + .await, + Err(CellGroupArchiveError::Limit) + )); + } + let verified = archive + .verify_checkpoint( + &cluster.domain, + &reference, + receiver(&pool, &applier, &cluster)?, + ) + .await?; + assert_ne!(verified.checkpoint().marker(), &base); + assert_eq!(group.group_manifest()?.as_ref(), Some(&base)); + assert_eq!(group.checkpoint_manifest()?, None); + drop(verified); + cluster.close().await?; + } + Ok(()) +} diff --git a/crates/cellar/tests/native_cell_groups_archive_controller.rs b/crates/cellar/tests/native_cell_groups_archive_controller.rs new file mode 100644 index 000000000..0d0a9d997 --- /dev/null +++ b/crates/cellar/tests/native_cell_groups_archive_controller.rs @@ -0,0 +1,526 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Controller-owned archive publication; no fixture event pumping after startup. +#![allow(clippy::disallowed_methods)] + +#[path = "support/cell_group_archive_store.rs"] +#[allow(dead_code)] +mod archive_store; +#[path = "support/cell_group_checkpoint.rs"] +#[allow(dead_code)] +mod checkpoint_fixture; +#[path = "support/raft_storage_fault.rs"] +#[allow(dead_code)] +mod fault; +#[path = "support/raft_ltx_fixture.rs"] +#[allow(dead_code)] +mod images; +#[path = "support/cell_group_runtime.rs"] +#[allow(dead_code, unused_imports)] +mod runtime_fixture; +#[path = "support/cell_group_rustfs.rs"] +mod rustfs; + +include!("support/cell_group_engine_fixture.rs"); + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn short_archived_groups_checkpoint_and_recover_independently() -> Result<()> { + Box::pin(runtime_fixture::archived_groups_reclaim_independently( + Arc::new(object_store::memory::InMemory::new()), + )) + .await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[ignore = "requires isolated local RustFS"] +async fn rustfs_short_archived_groups_checkpoint_and_recover_independently() -> Result<()> { + Box::pin(runtime_fixture::archived_groups_reclaim_independently( + rustfs::store("concurrent-archive-reclamation", false)?, + )) + .await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn archived_progress_wakes_commands_without_maintenance_poll() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::archive_progress_wakes_commands(rooted)).await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS; tools/multi-raft/cell_groups.py"] +async fn rustfs_quorum_continues_bounded_s3_outage() -> Result<()> { + for rooted in [false, true] { + runtime_fixture::full_backend_outage( + rooted, + rustfs::outage_store("full-backend-outage", rooted)?, + Some(rustfs::control_directory()?), + ) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS; tools/multi-raft/cell_groups.py"] +async fn rustfs_group_archive_requires_all_dependencies() -> Result<()> { + for rooted in [false, true] { + runtime_fixture::archive_dependencies( + rooted, + rustfs::store("archive-dependencies", rooted)?, + ) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS; tools/multi-raft/cell_groups.py"] +async fn rustfs_group_reclaim_restart_keeps_history() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::physical_log_reclamation( + rooted, + rustfs::store("physical-log-reclaim", rooted)?, + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn archive_coverage_gates_local_compaction() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::archive_checkpoint_frontier( + rooted, + Arc::new(object_store::memory::InMemory::new()), + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS; tools/multi-raft/cell_groups.py"] +async fn rustfs_archive_coverage_gates_local_compaction() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::archive_checkpoint_frontier( + rooted, + rustfs::store("archive-compaction", rooted)?, + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn s3_remote_outputs_preserve_durability_and_creation() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::s3_remote_outputs( + rooted, + Arc::new(object_store::memory::InMemory::new()), + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn s3_remote_deadlines_preserve_ambiguity_and_callbacks() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::s3_remote_deadlines( + rooted, + Arc::new(object_store::memory::InMemory::new()), + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn s3_remote_slow_delivery_retains_proofs_and_bounds() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::s3_remote_slow_delivery( + rooted, + Arc::new(object_store::memory::InMemory::new()), + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS; tools/multi-raft/cell_groups.py"] +async fn rustfs_s3_remote_outputs_preserve_durability_and_creation() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::s3_remote_outputs( + rooted, + rustfs::store("s3-remote-outputs", rooted)?, + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS; tools/multi-raft/cell_groups.py"] +async fn rustfs_s3_remote_deadlines_preserve_ambiguity_and_callbacks() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::s3_remote_deadlines( + rooted, + rustfs::store("s3-remote-deadlines", rooted)?, + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS; tools/multi-raft/cell_groups.py"] +async fn rustfs_s3_remote_slow_delivery_retains_proofs_and_bounds() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::s3_remote_slow_delivery( + rooted, + rustfs::store("s3-remote-delivery", rooted)?, + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn s3_namespace_minimum_cannot_be_downgraded() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::s3_namespace_minimum( + rooted, + Arc::new(object_store::memory::InMemory::new()), + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS; tools/multi-raft/cell_groups.py"] +async fn rustfs_s3_namespace_minimum_cannot_be_downgraded() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::s3_namespace_minimum( + rooted, + rustfs::store("s3-minimum", rooted)?, + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn s3_command_requires_the_sibling_archive_dependencies() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::s3_missing_sibling( + rooted, + Arc::new(object_store::memory::InMemory::new()), + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS; tools/multi-raft/cell_groups.py"] +async fn rustfs_s3_command_requires_the_sibling_archive_dependencies() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::s3_missing_sibling( + rooted, + rustfs::store("s3-sibling", rooted)?, + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn s3_commands_require_exact_archived_results() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::s3_commands( + rooted, + Arc::new(object_store::memory::InMemory::new()), + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn s3_command_deadlines_do_not_quarantine_committed_sql() -> Result<()> { + for rooted in [false, true] { + for cancel in [false, true] { + Box::pin(runtime_fixture::s3_deadline( + rooted, + Arc::new(object_store::memory::InMemory::new()), + cancel, + )) + .await?; + } + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS; tools/multi-raft/cell_groups.py"] +async fn rustfs_s3_commands_require_exact_archived_results() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::s3_commands( + rooted, + rustfs::store("s3-commands", rooted)?, + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS; tools/multi-raft/cell_groups.py"] +async fn rustfs_s3_command_deadlines_do_not_quarantine_committed_sql() -> Result<()> { + for rooted in [false, true] { + for cancel in [false, true] { + Box::pin(runtime_fixture::s3_deadline( + rooted, + rustfs::store(if cancel { "s3-cancel" } else { "s3-deadline" }, rooted)?, + cancel, + )) + .await?; + } + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn controller_archive_publishes_stable_cuts_while_quorum_advances() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::archive_controller_publish(rooted)).await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn controller_archive_reconciles_exact_intent_and_joins_close() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::archive_controller_reconcile(rooted)).await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS; tools/multi-raft/cell_groups.py"] +async fn rustfs_controller_archive_publishes_stable_cuts() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::archive_controller_publish_store( + rooted, + rustfs::store("publish", rooted)?, + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS; tools/multi-raft/cell_groups.py"] +async fn rustfs_controller_archive_reconciles_exact_intent_and_joins_close() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::archive_controller_reconcile_store( + rooted, + rustfs::store("reconcile", rooted)?, + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS; tools/multi-raft/cell_groups.py"] +async fn rustfs_unproved_s3_responses_refuse() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::unproved_s3_responses( + rooted, + rustfs::store("responses", rooted)?, + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn s3_enrollment_requires_the_original_archived_creation() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::s3_enrollment( + rooted, + Arc::new(object_store::memory::InMemory::new()), + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn s3_enrollment_deadlines_preserve_resolved_ownership() -> Result<()> { + for rooted in [false, true] { + for cancel in [false, true] { + Box::pin(runtime_fixture::s3_enrollment_deadline( + rooted, + Arc::new(object_store::memory::InMemory::new()), + cancel, + )) + .await?; + } + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS; tools/multi-raft/cell_groups.py"] +async fn rustfs_s3_enrollment_requires_the_original_archived_creation() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::s3_enrollment( + rooted, + rustfs::store("s3-enrollment", rooted)?, + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS; tools/multi-raft/cell_groups.py"] +async fn rustfs_s3_enrollment_deadlines_preserve_resolved_ownership() -> Result<()> { + for rooted in [false, true] { + for cancel in [false, true] { + Box::pin(runtime_fixture::s3_enrollment_deadline( + rooted, + rustfs::store( + if cancel { + "s3-enrollment-cancel" + } else { + "s3-enrollment-deadline" + }, + rooted, + )?, + cancel, + )) + .await?; + } + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn s3_receipt_lookup_tracks_archived_observations() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::s3_receipt_observations( + rooted, + Arc::new(object_store::memory::InMemory::new()), + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS; tools/multi-raft/cell_groups.py"] +async fn rustfs_s3_receipt_lookup_tracks_archived_observations() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::s3_receipt_observations( + rooted, + rustfs::store("s3-receipt-observations", rooted)?, + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn s3_receipt_lookup_honors_namespace_and_stored_minimums() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::s3_receipt_minimums( + rooted, + Arc::new(object_store::memory::InMemory::new()), + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS; tools/multi-raft/cell_groups.py"] +async fn rustfs_s3_receipt_lookup_honors_namespace_and_stored_minimums() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::s3_receipt_minimums( + rooted, + rustfs::store("s3-receipt-minimums", rooted)?, + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn s3_receipt_lookup_preserves_typed_failures_and_result_admission() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::s3_receipt_failures( + rooted, + Arc::new(object_store::memory::InMemory::new()), + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS; tools/multi-raft/cell_groups.py"] +async fn rustfs_s3_receipt_lookup_preserves_typed_failures_and_result_admission() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::s3_receipt_failures( + rooted, + rustfs::store("s3-receipt-failures", rooted)?, + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn s3_receipt_lookup_deadlines_refund_read_admission() -> Result<()> { + for rooted in [false, true] { + for cancel in [false, true] { + Box::pin(runtime_fixture::s3_receipt_deadline( + rooted, + Arc::new(object_store::memory::InMemory::new()), + cancel, + )) + .await?; + } + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS; tools/multi-raft/cell_groups.py"] +async fn rustfs_s3_receipt_lookup_deadlines_refund_read_admission() -> Result<()> { + for rooted in [false, true] { + for cancel in [false, true] { + Box::pin(runtime_fixture::s3_receipt_deadline( + rooted, + rustfs::store( + if cancel { + "s3-receipt-cancel" + } else { + "s3-receipt-deadline" + }, + rooted, + )?, + cancel, + )) + .await?; + } + } + Ok(()) +} diff --git a/crates/cellar/tests/native_cell_groups_catalog.rs b/crates/cellar/tests/native_cell_groups_catalog.rs new file mode 100644 index 000000000..52c3878a3 --- /dev/null +++ b/crates/cellar/tests/native_cell_groups_catalog.rs @@ -0,0 +1,391 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Catalog admission contracts. Pure decisions do not qualify a live group or +//! catalog root publication; those have separate native integration gates. + +use cellar::application::consensus::{ + verification::cell_groups::*, ClusterId, GroupId, ImageCommitment, IncarnationId, LtxImage, + NamespaceId, ProposalId, +}; +use sha2::{Digest, Sha256}; + +#[path = "support/cell_group_directory.rs"] +mod directory; +#[path = "support/raft_catalog_fault.rs"] +#[allow(dead_code)] +mod fault; + +#[tokio::test] +async fn seal_authorization_cancel_race() -> anyhow::Result<()> { + directory::seal_authorization_cancel_race().await +} + +#[tokio::test] +async fn unknown_transition_authorization_is_irreversible() -> anyhow::Result<()> { + directory::unknown_transition_authorization().await +} + +#[tokio::test] +async fn standby_directory_is_explicit_durable_and_non_serving() -> anyhow::Result<()> { + directory::standby_initialization().await +} + +#[tokio::test] +async fn standby_first_range_cas_reconciles_and_never_reopens() -> anyhow::Result<()> { + directory::standby_first_ranges().await +} + +#[tokio::test] +async fn standby_import_freezes_a_real_empty_inventory() -> anyhow::Result<()> { + directory::standby_import().await +} + +#[tokio::test] +async fn group_import_freeze_retains_pending_and_fences_topology() -> anyhow::Result<()> { + directory::group_import_freeze().await +} + +#[tokio::test] +async fn directory_inventory_pages_bind_the_exact_topology_cut() -> anyhow::Result<()> { + directory::inventory_pages().await +} + +#[tokio::test] +async fn range_directory_has_single_owner() -> anyhow::Result<()> { + directory::single_owner().await +} + +#[tokio::test] +async fn bounded_catalog_discovery() -> anyhow::Result<()> { + directory::bounded_discovery().await +} + +#[tokio::test] +async fn directory_reconciles_unknown_and_refuses_missing_state() -> anyhow::Result<()> { + directory::recovery().await +} + +#[tokio::test] +async fn directory_genesis_retry_after_later_mutation() -> anyhow::Result<()> { + directory::genesis_race().await +} + +#[tokio::test] +async fn directory_churn_preserves_exact_partition_and_old_views() -> anyhow::Result<()> { + directory::churn().await +} + +#[tokio::test] +async fn group_birth_is_lazy_and_reconciles_unknown_by_exact_identity() -> anyhow::Result<()> { + directory::birth_recovery().await +} + +#[tokio::test] +async fn group_birth_reservations_bind_global_identity_and_canonical_placement( +) -> anyhow::Result<()> { + directory::birth_reservations().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn birth_unknown_and_claimed_blank_refuse() -> anyhow::Result<()> { + directory::member_bootstrap().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn group_member_permit_refuses_retired_birth_and_foreign_node() -> anyhow::Result<()> { + directory::member_fencing().await +} + +fn route() -> GroupRoute { + GroupRoute { + history: GroupHistory { + cluster: ClusterId::new([1; 16]).unwrap(), + namespace: NamespaceId::new([2; 16]).unwrap(), + group: GroupId::new([3; 16]).unwrap(), + incarnation: IncarnationId::new([4; 16]).unwrap(), + }, + range: CellRange::all(), + epoch: RouteEpoch::new(1).unwrap(), + birth_digest: [5; 32], + schema_digest: [6; 32], + } +} + +fn entry(route: &GroupRoute) -> CellGroupEntry<'static> { + CellGroupEntry { + binding: GroupEntryBinding { + group: route.history, + route_epoch: route.epoch, + birth_digest: route.birth_digest, + schema_digest: route.schema_digest, + proposal: ProposalId::new([7; 16]).unwrap(), + }, + command: GroupCommand::Enroll { + cell: CellHistory { + cluster: route.history.cluster, + namespace: route.history.namespace, + cell: CellId::new("counter").unwrap(), + incarnation: IncarnationId::new([8; 16]).unwrap(), + }, + image: LtxImage { + epoch: 1, + txid: 0, + checksum: 0, + commitment: ImageCommitment::Sha256(Sha256::digest([]).into()), + }, + files: vec![], + }, + } +} + +fn encoded(entry: &CellGroupEntry<'_>, route: &GroupRoute) -> Vec { + entry.encode(route, GroupEntryLimits::default()).unwrap() +} + +fn admitted(route: &GroupRoute, request: &[u8]) -> EnrollmentRecord { + match decide_enrollment( + route, + EnrollmentPhase::Active, + request, + None, + GroupEntryLimits::default(), + ) + .unwrap() + { + EnrollmentDecision::Propose(record) => record, + EnrollmentDecision::Existing(_) => panic!("new cell cannot be an existing creation"), + } +} + +#[test] +fn enrollment_decision_checks_identity() { + let route = route(); + let original = entry(&route); + let request = encoded(&original, &route); + let selected = admitted(&route, &request); + assert_eq!(selected.creation(), original.binding.proposal); + assert_eq!(selected.cell().cell.as_str(), "counter"); + assert_eq!(selected.schema(), route.schema_digest); + assert_eq!(selected.initial().txid, 0); + + // A lost reply is reconciled using the durable original creation record, + // even after subsequent mutations have replaced the initial cell image. + let saved = selected.encode().unwrap(); + let restarted = EnrollmentRecord::decode(&saved).unwrap(); + let retry = |request: &[u8], current: &GroupRoute| { + decide_enrollment( + current, + EnrollmentPhase::Active, + request, + Some(&restarted), + GroupEntryLimits::default(), + ) + }; + assert_eq!( + retry(&request, &route), + Ok(EnrollmentDecision::Existing(selected.clone())) + ); + + // These requests may both be prepared while no committed record exists. + // Rechecking against the selected record cannot replace its incarnation or + // repurpose the same operation ID for a different initial image. + for field in 0..4 { + let mut competing = original.clone(); + let GroupCommand::Enroll { cell, image, .. } = &mut competing.command else { + unreachable!() + }; + match field { + 0 => competing.binding.proposal = ProposalId::new([9; 16]).unwrap(), + 1 => cell.incarnation = IncarnationId::new([10; 16]).unwrap(), + 2 => cell.cell = CellId::new("other").unwrap(), + _ => image.epoch += 1, + } + let competitor = encoded(&competing, &route); + assert_ne!(admitted(&route, &competitor), selected); + assert_eq!(retry(&competitor, &route), Err(EnrollmentError::Conflict)); + } + + for field in 0..7 { + let mut current = route.clone(); + match field { + 0 => current.history.cluster = ClusterId::new([11; 16]).unwrap(), + 1 => current.history.namespace = NamespaceId::new([11; 16]).unwrap(), + 2 => current.history.group = GroupId::new([11; 16]).unwrap(), + 3 => current.history.incarnation = IncarnationId::new([11; 16]).unwrap(), + 4 => current.epoch = current.epoch.checked_next().unwrap(), + 5 => current.schema_digest = [11; 32], + _ => current.birth_digest = [11; 32], + } + assert!(matches!( + retry(&request, ¤t), + Err(EnrollmentError::Protocol(_)) + )); + } + + // A target at a half-open upper bound belongs to the neighboring range. + let mut outside = route.clone(); + outside.range = CellRange::new( + CellBound::NegativeInfinity, + CellBound::Key(cell_routing_key( + route.history.namespace, + &selected.cell().cell, + )), + ) + .unwrap(); + assert!(matches!( + retry(&request, &outside), + Err(EnrollmentError::Protocol(_)) + )); + + // Normal transfer rebinds the request to the current physical route while + // the immutable logical creation record remains unchanged. + let mut destination = route.clone(); + destination.history.group = GroupId::new([12; 16]).unwrap(); + destination.history.incarnation = IncarnationId::new([13; 16]).unwrap(); + destination.epoch = route.epoch.checked_next().unwrap(); + destination.birth_digest = [14; 32]; + let mut moved = original.clone(); + moved.binding = entry(&destination).binding; + assert_eq!( + retry(&encoded(&moved, &destination), &destination), + Ok(EnrollmentDecision::Existing(selected)) + ); + assert!(retry(&request, &destination).is_err()); +} + +#[test] +fn sealed_range_rejects_enrollment() { + let route = route(); + let mut entry = entry(&route); + let prepared_before_seal = encoded(&entry, &route); + let selected = admitted(&route, &prepared_before_seal); + for (phase, error) in [ + (EnrollmentPhase::Sealed, EnrollmentError::Sealed), + (EnrollmentPhase::Retired, EnrollmentError::Retired), + ] { + for existing in [None, Some(&selected)] { + assert_eq!( + decide_enrollment( + &route, + phase, + &prepared_before_seal, + existing, + GroupEntryLimits::default(), + ), + Err(error) + ); + } + } + entry.command = GroupCommand::Retire { + transition: TransitionId::new([17; 16]).unwrap(), + publication: [18; 32], + }; + assert_eq!( + decide_enrollment( + &route, + EnrollmentPhase::Active, + &encoded(&entry, &route), + None, + GroupEntryLimits::default(), + ), + Err(EnrollmentError::NotEnrollment) + ); +} + +#[test] +fn enrollment_record_is_bounded_canonical_data() { + let route = route(); + let mut entry = entry(&route); + let GroupCommand::Enroll { cell, .. } = &mut entry.command else { + unreachable!() + }; + cell.cell = CellId::new("\\".repeat(1024)).unwrap(); + let record = admitted(&route, &encoded(&entry, &route)); + let bytes = record.encode().unwrap(); + assert_eq!(EnrollmentRecord::decode(&bytes).unwrap(), record); + assert_eq!( + EnrollmentRecord::decode(&vec![0; 4097]), + Err(EnrollmentError::Limit) + ); + for end in 0..bytes.len() { + assert!(EnrollmentRecord::decode(&bytes[..end]).is_err()); + } + let mut spaced = bytes.clone(); + spaced.push(b' '); + assert!(EnrollmentRecord::decode(&spaced).is_err()); + let original: serde_json::Value = serde_json::from_slice(&bytes).unwrap(); + let replace_field = |field: &str, value: serde_json::Value| { + // Preserve canonical field order so refusal must come from the changed + // value, not from Value's map ordering when serializing the whole object. + String::from_utf8(bytes.clone()) + .unwrap() + .replacen( + &format!("\"{field}\":{}", original[field]), + &format!("\"{field}\":{value}"), + 1, + ) + .into_bytes() + }; + let another = replace_field("creation", serde_json::json!(vec![9; 16])); + assert_eq!( + EnrollmentRecord::decode(&another).unwrap().creation(), + ProposalId::new([9; 16]).unwrap() + ); + for (field, value) in [ + ("domain", serde_json::json!("cellar-cell-enrollment-2")), + ("cluster", serde_json::json!(vec![0; 16])), + ("creation", serde_json::json!(vec![0; 16])), + ("initial_epoch", serde_json::json!(0)), + ("initial_checksum", serde_json::json!(1)), + ] { + assert!( + EnrollmentRecord::decode(&replace_field(field, value)).is_err(), + "{field}" + ); + } + let mut unknown = bytes[..bytes.len() - 1].to_vec(); + unknown.extend_from_slice(b",\"extra\":true}"); + assert!(EnrollmentRecord::decode(&unknown).is_err()); + let mut damaged = encoded(&entry, &route); + *damaged.last_mut().unwrap() ^= 1; + assert!(matches!( + decide_enrollment( + &route, + EnrollmentPhase::Active, + &damaged, + None, + GroupEntryLimits::default() + ), + Err(EnrollmentError::Protocol(_)) + )); +} + +#[tokio::test] +async fn routing_redirect_requires_catalog_confirmation() -> anyhow::Result<()> { + directory::routing_authentication().await +} + +#[tokio::test] +async fn routing_cached_requests_preserve_identity_without_catalog_reads() -> anyhow::Result<()> { + directory::routing_cache_and_budget().await +} + +#[tokio::test] +async fn routing_attempt_deadline_survives_redirect_and_cancellation() -> anyhow::Result<()> { + directory::routing_deadline_and_cancellation().await +} + +#[tokio::test] +async fn dispatcher_route_cache_shares_ranges_across_operations() -> anyhow::Result<()> { + directory::cached_operation_discovery().await +} + +#[tokio::test] +async fn dispatcher_route_cache_bounds_and_refreshes_hints() -> anyhow::Result<()> { + directory::bounded_route_hints().await +} + +#[tokio::test] +async fn dispatcher_route_cache_refuses_superseded_inflight_refill() -> anyhow::Result<()> { + directory::stale_cache_refill().await +} diff --git a/crates/cellar/tests/native_cell_groups_contracts.rs b/crates/cellar/tests/native_cell_groups_contracts.rs new file mode 100644 index 000000000..af87dd1da --- /dev/null +++ b/crates/cellar/tests/native_cell_groups_contracts.rs @@ -0,0 +1,864 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Cell-group protocol boundary tests. These do not qualify a running group. + +use anyhow::Result; +use cellar::application::consensus::verification::cell_groups::*; +use cellar::application::consensus::verification::{Catalog, CatalogOptions}; +use cellar::application::consensus::{ + ClusterId, GroupId, IncarnationId, LogIndex, LogPosition, ProposalId, Term, VersionError, + VersionKeys, VersionSigningKey, VersionToken, +}; +use cellar::application::consensus::{NamespaceConfig, NamespaceId, NamespaceMode}; +use std::{ + sync::{atomic::Ordering, Arc}, + time::Duration, +}; + +#[path = "support/raft_catalog_fault.rs"] +#[allow(dead_code)] +mod store; + +fn observation() -> CellVersionObservation { + let history = CellHistory { + cluster: ClusterId::new([1; 16]).unwrap(), + namespace: NamespaceId::new([2; 16]).unwrap(), + cell: CellId::new("alpha").unwrap(), + incarnation: IncarnationId::new([3; 16]).unwrap(), + }; + let group = GroupHistory { + cluster: history.cluster, + namespace: history.namespace, + group: GroupId::new([4; 16]).unwrap(), + incarnation: IncarnationId::new([5; 16]).unwrap(), + }; + CellVersionObservation { + version: CellVersion { + history, + sequence: MutationSequence::new(7).unwrap(), + image: CellImage { + epoch: 1, + txid: 10, + checksum: 17, + commitment: CellImageCommitment::Materialized([8; 32]), + }, + }, + witness: CommitWitness { + group, + position: LogPosition { + index: LogIndex::new(100), + term: Term::new(9), + }, + proposal: ProposalId::new([6; 16]).unwrap(), + }, + } +} + +fn keys() -> VersionKeys { + VersionKeys::new( + VersionSigningKey::new([7; 32], 100_000).unwrap(), + vec![], + Duration::from_secs(10), + ) + .unwrap() +} + +#[tokio::test] +async fn version_key_rotation_survives_restart_and_ambiguous_catalog_write() -> Result<()> { + let observation = observation(); + let history = &observation.version.history; + let schema = [7; 32]; + let keys = |secret, until, retained| { + VersionKeys::new( + VersionSigningKey::new([secret; 32], until)?, + retained, + Duration::from_millis(1000), + ) + }; + let store = Arc::new(store::Store::default()); + let catalog = Catalog::new(store.clone(), CatalogOptions::default())?; + let namespace = catalog + .open_namespace(NamespaceConfig::new( + history.cluster, + history.namespace, + NamespaceMode::Raft, + schema, + )) + .await?; + let old = keys(11, 10000, vec![])?; + store.fail_after.store(true, Ordering::Release); + namespace.require_version_keys(&old, 1000).await?; + let token = old.issue_cell(&observation, schema, 1000)?; + let next_key = VersionSigningKey::new([12; 32], 20000)?; + // Verifier-first rolling deployment: advertise both keys while the old + // signer is active, then switch to the new signer with identical trust. + let distributed = keys(11, 10000, vec![next_key.verification_key()])?; + store.fail_confirmation.store(true, Ordering::Release); + assert!(namespace + .require_version_keys(&distributed, 1100) + .await + .is_err()); + namespace.require_version_keys(&distributed, 1100).await?; + let next = VersionKeys::new( + next_key, + old.verification_keys(), + Duration::from_millis(1000), + )?; + namespace.require_version_keys(&next, 1100).await?; + assert_eq!( + next.verify_cell(&token, history, schema, 1100)?, + observation + ); + assert!(namespace.require_version_keys(&old, 1100).await.is_err()); + let dropped = keys(12, 20000, vec![])?; + assert!(namespace + .require_version_keys(&dropped, 1100) + .await + .is_err()); + // Reopening the catalog must preserve the same refusal; it cannot rely on + // an in-memory history of the previous configuration. + let reopened = Catalog::new(store, CatalogOptions::default())? + .open_namespace(namespace.config().clone()) + .await?; + assert!(reopened.require_version_keys(&dropped, 1100).await.is_err()); + reopened.require_version_keys(&dropped, 10000).await?; + assert!(next.verify_cell(&token, history, schema, 10000).is_err()); + Ok(()) +} + +#[test] +fn version_survives_transfer_not_disaster() { + let keys = keys(); + let original = observation(); + let schema = [42; 32]; + let token = keys.issue_cell(&original, schema, 100).unwrap(); + assert!(VersionToken::from_encoded(token.as_str()).is_err()); + let decoded = keys + .verify_cell(&token, &original.version.history, schema, 101) + .unwrap(); + assert_eq!(decoded, original); + let mut moved = original.clone(); + moved.witness.group.group = GroupId::new([10; 16]).unwrap(); + moved.witness.group.incarnation = IncarnationId::new([11; 16]).unwrap(); + moved.witness.position = LogPosition { + index: LogIndex::new(2), + term: Term::new(1), + }; + moved.version.sequence = moved.version.sequence.checked_next().unwrap(); + moved.version.image.txid += 1; + let successor = keys.issue_cell(&moved, schema, 102).unwrap(); + let successor = keys + .verify_cell(&successor, &original.version.history, schema, 103) + .unwrap(); + assert!(successor.version.covers(&decoded.version).unwrap()); + assert!(!decoded.version.covers(&successor.version).unwrap()); + for field in 0..4 { + let mut other = original.version.history.clone(); + match field { + 0 => other.cluster = ClusterId::new([12; 16]).unwrap(), + 1 => other.namespace = NamespaceId::new([12; 16]).unwrap(), + 2 => other.cell = CellId::new("beta").unwrap(), + _ => other.incarnation = IncarnationId::new([12; 16]).unwrap(), + } + assert_eq!( + keys.verify_cell(&token, &other, schema, 103), + Err(VersionError::WrongHistory) + ); + } + assert_eq!( + keys.verify_cell(&token, &original.version.history, [43; 32], 103), + Err(VersionError::WrongSchema) + ); + assert_eq!( + keys.verify_cell(&token, &original.version.history, schema, 10_100), + Err(VersionError::Expired) + ); + assert_eq!( + keys.verify_cell(&token, &original.version.history, schema, 99), + Err(VersionError::Expired) + ); +} + +#[test] +fn cell_token_bounds_tampering_and_commitment_domains() { + use base64::{engine::general_purpose::URL_SAFE_NO_PAD, Engine as _}; + let keys = keys(); + let mut original = observation(); + original.version.history.cell = CellId::new("a").unwrap(); + let shortest = keys.issue_cell(&original, [42; 32], 100).unwrap(); + assert_eq!( + URL_SAFE_NO_PAD.decode(shortest.as_str()).unwrap().len(), + 368 + ); + assert_eq!( + keys.verify_cell(&shortest, &original.version.history, [42; 32], 101) + .unwrap(), + original + ); + original.version.history.cell = CellId::new("é".repeat(512)).unwrap(); + original.version.image.commitment = CellImageCommitment::Extents { + root: [9; 32], + manifest_bytes: 112, + }; + let token = keys.issue_cell(&original, [42; 32], 100).unwrap(); + assert_eq!( + keys.verify_cell(&token, &original.version.history, [42; 32], 101) + .unwrap(), + original + ); + assert_eq!( + CellVersionToken::from_encoded(token.as_str()).unwrap(), + token + ); + assert!(CellVersionToken::from_encoded(&format!("{}=", token.as_str())).is_err()); + let other = VersionKeys::new( + VersionSigningKey::new([8; 32], 100_000).unwrap(), + vec![], + Duration::from_secs(10), + ) + .unwrap(); + assert!(other + .verify_cell(&token, &original.version.history, [42; 32], 101) + .is_err()); + let bytes = URL_SAFE_NO_PAD.decode(token.as_str()).unwrap(); + for offset in 0..bytes.len() { + let mut changed = bytes.clone(); + changed[offset] ^= 1; + if let Ok(tampered) = CellVersionToken::from_encoded(&URL_SAFE_NO_PAD.encode(changed)) { + assert!(keys + .verify_cell(&tampered, &original.version.history, [42; 32], 101) + .is_err()); + } + } + for end in 0..bytes.len() { + if let Ok(truncated) = + CellVersionToken::from_encoded(&URL_SAFE_NO_PAD.encode(&bytes[..end])) + { + assert!(keys + .verify_cell(&truncated, &original.version.history, [42; 32], 101) + .is_err()); + } + } + assert_eq!( + CellVersionToken::from_encoded(&"a".repeat(2049)), + Err(VersionError::Limit) + ); + original.version.image.commitment = CellImageCommitment::Extents { + root: [9; 32], + manifest_bytes: 113, + }; + assert_eq!( + keys.issue_cell(&original, [42; 32], 100), + Err(VersionError::Invalid) + ); + original.version.image.commitment = CellImageCommitment::Materialized([9; 32]); + original.witness.group.namespace = NamespaceId::new([15; 16]).unwrap(); + assert_eq!( + keys.issue_cell(&original, [42; 32], 100), + Err(VersionError::Invalid) + ); +} + +#[test] +fn version_key_deadline_and_payload_limits() -> Result<()> { + let ring = VersionKeys::new( + VersionSigningKey::new([11; 32], 2000)?, + vec![], + Duration::from_millis(1000), + )?; + let original = observation(); + assert!(ring.issue_cell(&original, [7; 32], 1001).is_err()); + assert!(ring.issue_cell(&original, [7; 32], u64::MAX).is_err()); + let mut invalid = original.clone(); + invalid.witness.position.index = LogIndex::new(0); + assert!(ring.issue_cell(&invalid, [7; 32], 1000).is_err()); + let mut longest = original; + longest.version.history.cell = CellId::new("x".repeat(1024))?; + let token = ring.issue_cell(&longest, [7; 32], 1000)?; + assert_eq!( + ring.verify_cell(&token, &longest.version.history, [7; 32], 1000)?, + longest + ); + assert!(CellId::new("x".repeat(1025)).is_err()); + let old = VersionSigningKey::new([11; 32], 10000)?.verification_key(); + assert!(VersionKeys::new( + VersionSigningKey::new([11; 32], 20000)?, + vec![old], + Duration::from_millis(1000), + ) + .is_err()); + assert!(VersionSigningKey::new([0; 32], 20000).is_err()); + Ok(()) +} + +#[test] +fn identity_and_hash_boundaries() { + let namespace = NamespaceId::new([2; 16]).unwrap(); + let alpha = CellId::new("alpha").unwrap(); + let key = cell_routing_key(namespace, &alpha); + // Independent SHA-256 golden vector for the specified preimage, including NUL. + assert_eq!( + hex::encode(key.digest()), + "9125bede939b7512fcd2b4567a7c60ee066a3baf5847a265088e48449e408f1b" + ); + assert_ne!( + key, + cell_routing_key(NamespaceId::new([3; 16]).unwrap(), &alpha) + ); + let tied = CellKey::from_digest(key.digest(), CellId::new("beta").unwrap()); + assert!(key < tied); + let (left, right) = CellRange::all().split(tied.clone()).unwrap(); + assert!(left.contains(&key) && !right.contains(&key)); + assert!(!left.contains(&tied) && right.contains(&tied)); + assert_eq!(left.merge(&right).unwrap(), CellRange::all()); + assert!(right.merge(&left).is_err()); + assert!(CellRange::new(CellBound::PositiveInfinity, CellBound::NegativeInfinity).is_err()); + assert!(CellId::new("").is_err()); + assert!(CellId::new("a\nb").is_err()); + assert!(CellId::new("a".repeat(1025)).is_err()); + assert!(MutationSequence::new(u64::MAX) + .unwrap() + .checked_next() + .is_err()); +} + +#[allow(dead_code)] +#[path = "support/raft_ltx_fixture.rs"] +mod image_fixture; + +fn group_route() -> GroupRoute { + GroupRoute { + history: observation().witness.group, + range: CellRange::all(), + epoch: RouteEpoch::new(1).unwrap(), + birth_digest: [22; 32], + schema_digest: [42; 32], + } +} + +fn binding(route: &GroupRoute) -> GroupEntryBinding { + GroupEntryBinding { + group: route.history, + route_epoch: route.epoch, + birth_digest: route.birth_digest, + schema_digest: route.schema_digest, + proposal: ProposalId::new([6; 16]).unwrap(), + } +} + +fn ltx_cut() -> image_fixture::Cut { + let before = image_fixture::image( + None, + "CREATE TABLE value(n INTEGER); INSERT INTO value VALUES (1);", + ) + .unwrap(); + let after = image_fixture::image(Some(&before), "UPDATE value SET n=2;").unwrap(); + image_fixture::Cut::new(&before, &after, 1, true, true) +} + +fn image_value(image: cellar_ltx::apply::ImageState) -> cellar::application::consensus::LtxImage { + cellar::application::consensus::LtxImage { + epoch: image.epoch, + txid: image.txid.0, + checksum: image.checksum, + commitment: image.digest.into(), + } +} + +fn mutation_entry(cut: &image_fixture::Cut) -> CellGroupEntry<'_> { + CellGroupEntry { + binding: binding(&group_route()), + command: GroupCommand::Mutation(GroupMutation { + cell: observation().version.history, + sequence: MutationSequence::new(2).unwrap(), + base: image_value(cut.base), + result: image_value(cut.result), + files: vec![GroupLtx { + min_txid: cut.base.txid.0 + 1, + max_txid: cut.result.txid.0, + page_count: cut.pages.len() as u64, + decoded_bytes: cut.pages.len() as u64 * 512, + bytes: &cut.bytes, + }], + }), + } +} + +fn rehash(bytes: &mut [u8]) { + use sha2::{Digest, Sha256}; + let end = bytes.len() - 32; + let hash = Sha256::digest(&bytes[..end]); + bytes[end..].copy_from_slice(&hash); +} + +#[test] +fn envelope_rejects_wrong_cell_and_domain() { + use cellar::application::consensus::{ + CellUpdate, EnvelopeLimits, GroupScope, LtxUpdate, ProtocolError, + }; + let cut = ltx_cut(); + assert!(cellar_ltx::ltx::decode_file(&cut.bytes).is_ok()); + let route = group_route(); + let limits = GroupEntryLimits::default(); + let entry = mutation_entry(&cut); + let bytes = entry.encode(&route, limits).unwrap(); + let decoded = CellGroupEntry::decode(&bytes, &route, limits).unwrap(); + assert_eq!(decoded, entry); + let GroupCommand::Mutation(update) = &decoded.command else { + panic!("mutation changed kind") + }; + let current = CellVersion { + history: observation().version.history, + sequence: MutationSequence::new(1).unwrap(), + image: image_value(cut.base).into(), + }; + assert!(update.check_base(¤t).is_ok()); + for field in 0..6 { + let mut other = current.clone(); + match field { + 0 => other.history.cell = CellId::new("beta").unwrap(), + 1 => other.history.incarnation = IncarnationId::new([90; 16]).unwrap(), + 2 => other.history.namespace = NamespaceId::new([90; 16]).unwrap(), + 3 => other.sequence = MutationSequence::new(2).unwrap(), + 4 => other.image.checksum ^= 1, + _ => other.image.commitment = CellImageCommitment::Materialized([90; 32]), + } + assert_eq!(update.check_base(&other), Err(ProtocolError::Chain)); + } + for field in 0..7 { + let mut other = route.clone(); + match field { + 0 => other.history.cluster = ClusterId::new([90; 16]).unwrap(), + 1 => other.history.namespace = NamespaceId::new([90; 16]).unwrap(), + 2 => other.history.group = GroupId::new([90; 16]).unwrap(), + 3 => other.history.incarnation = IncarnationId::new([90; 16]).unwrap(), + 4 => other.epoch = RouteEpoch::new(2).unwrap(), + 5 => other.schema_digest = [90; 32], + _ => other.birth_digest = [90; 32], + } + assert_eq!( + CellGroupEntry::decode(&bytes, &other, limits), + Err(ProtocolError::Identity) + ); + } + let mut outside = route.clone(); + outside.range = CellRange::all() + .split(cell_routing_key( + route.history.namespace, + ¤t.history.cell, + )) + .unwrap() + .0; + assert_eq!( + CellGroupEntry::decode(&bytes, &outside, limits), + Err(ProtocolError::Identity) + ); + let old = CellUpdate { + scope: GroupScope { + cluster: route.history.cluster, + namespace: route.history.namespace, + group: route.history.group, + incarnation: route.history.incarnation, + cell: current.history.cell.as_str().into(), + }, + application_digest: route.schema_digest, + proposal: entry.binding.proposal, + base: image_value(cut.base), + result: image_value(cut.result), + files: vec![LtxUpdate { + min_txid: 2, + max_txid: 2, + page_count: cut.pages.len() as u64, + decoded_bytes: cut.pages.len() as u64 * 512, + bytes: cut.bytes.clone(), + }], + } + .encode(EnvelopeLimits::default()) + .unwrap(); + assert_eq!( + CellGroupEntry::decode(&old, &route, limits), + Err(ProtocolError::Protocol) + ); + assert!(CellUpdate::decode(&bytes, EnvelopeLimits::default()).is_err()); + let mut unknown = bytes.clone(); + unknown[24] ^= 1; // New-format framing with an unrecognized semantic domain. + rehash(&mut unknown); + assert_eq!( + CellGroupEntry::decode(&unknown, &route, limits), + Err(ProtocolError::Protocol) + ); +} + +#[test] +fn codec_limits_before_allocation() { + use cellar::application::consensus::ProtocolError; + let cut = ltx_cut(); + let route = group_route(); + let limits = GroupEntryLimits::default(); + let bytes = mutation_entry(&cut).encode(&route, limits).unwrap(); + let header_len = u32::from_be_bytes(bytes[12..16].try_into().unwrap()) as usize; + let decoded = CellGroupEntry::decode(&bytes, &route, limits).unwrap(); + let GroupCommand::Mutation(update) = decoded.command else { + panic!("not a mutation") + }; + assert_eq!( + update.files[0].bytes.as_ptr(), + bytes[24 + header_len..].as_ptr() + ); + for end in 0..bytes.len() { + assert!( + CellGroupEntry::decode(&bytes[..end], &route, limits).is_err(), + "truncation {end}" + ); + } + for field in 0..5 { + let mut bounded = limits; + match field { + 0 => bounded.envelope.max_header_bytes = 1, + 1 => bounded.envelope.max_payload_bytes = 1, + 2 => bounded.envelope.max_decoded_bytes = 1, + 3 => bounded.envelope.max_pages = 1, + _ => bounded.envelope.max_files = 0, + } + assert_eq!( + CellGroupEntry::decode(&bytes, &route, bounded), + Err(ProtocolError::Limit) + ); + } + let file_count = 24 + 185 + 16 + 2 + "alpha".len() + 8 + 2 * 61; + let malformed_fields = [ + (12, u32::MAX.to_be_bytes().to_vec()), + (16, u64::MAX.to_be_bytes().to_vec()), + (file_count, u32::MAX.to_be_bytes().to_vec()), + (file_count + 4, u64::MAX.to_be_bytes().to_vec()), + (file_count + 4 + 24, u64::MAX.to_be_bytes().to_vec()), + (file_count + 4 + 32, u64::MAX.to_be_bytes().to_vec()), + ]; + for (offset, value) in malformed_fields { + let mut oversized = bytes.clone(); + oversized[offset..offset + value.len()].copy_from_slice(&value); + rehash(&mut oversized); + assert_eq!( + CellGroupEntry::decode(&oversized, &route, limits), + Err(ProtocolError::Limit), + "field {offset}" + ); + } + let mut unknown = bytes.clone(); + unknown[24 + 184] = 255; + rehash(&mut unknown); + assert_eq!( + CellGroupEntry::decode(&unknown, &route, limits), + Err(ProtocolError::Protocol) + ); + let mut marker = bytes.clone(); + marker[8..12].copy_from_slice(&2_u32.to_be_bytes()); + rehash(&mut marker); + assert_eq!( + CellGroupEntry::decode(&marker, &route, limits), + Err(ProtocolError::Protocol) + ); + // Fixed-position fields occur exactly once. An appended duplicate group + // identity cannot be reinterpreted as an override or optional metadata. + let mut duplicate = bytes.clone(); + duplicate.splice( + 24 + header_len..24 + header_len, + bytes[56..120].iter().copied(), + ); + duplicate[12..16].copy_from_slice(&((header_len + 64) as u32).to_be_bytes()); + rehash(&mut duplicate); + assert_eq!( + CellGroupEntry::decode(&duplicate, &route, limits), + Err(ProtocolError::Framing) + ); + let mut sequence = bytes.clone(); + let at = 24 + 185 + 16 + 2 + "alpha".len(); + sequence[at..at + 8].copy_from_slice(&1_u64.to_be_bytes()); + rehash(&mut sequence); + assert_eq!( + CellGroupEntry::decode(&sequence, &route, limits), + Err(ProtocolError::Chain) + ); + let mut corrupt = bytes.clone(); + corrupt[24 + header_len] ^= 1; + assert_eq!( + CellGroupEntry::decode(&corrupt, &route, limits), + Err(ProtocolError::Digest) + ); +} + +#[test] +fn group_administration_roundtrips_and_golden() { + use cellar::application::consensus::{DiskLineage, MemberId, NodeId, ProtocolError}; + let route = group_route(); + let limits = GroupEntryLimits::default(); + let transition = TransitionId::new([77; 16]).unwrap(); + let initial = image_fixture::image(None, "CREATE TABLE seed(n INTEGER);").unwrap(); + let cut = image_fixture::Cut::new(&[], &initial, 0, true, true); + let (left, right) = CellRange::all() + .split(cell_routing_key( + route.history.namespace, + &CellId::new("alpha").unwrap(), + )) + .unwrap(); + let children: Vec<_> = [left, right] + .into_iter() + .enumerate() + .map(|(i, range)| { + let mut child = route.clone(); + child.history.group = GroupId::new([40 + i as u8; 16]).unwrap(); + child.history.incarnation = IncarnationId::new([50 + i as u8; 16]).unwrap(); + child.birth_digest = [60 + i as u8; 32]; + child.range = range; + child + }) + .collect(); + let seal = GroupCommand::Seal { + transition, + authorization: [66; 32], + children: children.clone(), + }; + let membership = GroupCommand::Membership { + previous: MembershipRevision::new(1).unwrap(), + next: MembershipRevision::new(2).unwrap(), + transition: GroupMembershipTransition::JointExplicit, + changes: vec![GroupMemberChange { + member: MemberId::new(1).unwrap(), + node: NodeId::new([7; 16]).unwrap(), + disk: DiskLineage::new([8; 16]).unwrap(), + role: GroupMemberRole::Voter, + }], + }; + let commands = vec![ + GroupCommand::Enroll { + cell: observation().version.history, + image: image_value(cut.result), + files: vec![GroupLtx { + min_txid: 1, + max_txid: 1, + page_count: cut.pages.len() as u64, + decoded_bytes: cut.pages.len() as u64 * 512, + bytes: &cut.bytes, + }], + }, + membership.clone(), + GroupCommand::Membership { + previous: MembershipRevision::new(2).unwrap(), + next: MembershipRevision::new(3).unwrap(), + transition: GroupMembershipTransition::LeaveJoint, + changes: vec![], + }, + GroupCommand::Checkpoint { + position: observation().witness.position, + manifest: [25; 32], + membership: MembershipRevision::new(1).unwrap(), + }, + seal.clone(), + GroupCommand::Install { + transition, + sources: children + .iter() + .map(|child| GroupSeal { + group: child.history, + position: observation().witness.position, + manifest: [26; 32], + }) + .collect(), + manifest: [27; 32], + }, + GroupCommand::Activate { + transition, + publication: [88; 32], + manifest: [27; 32], + }, + GroupCommand::PreparePark { + generation: ParkGeneration::new(1).unwrap(), + term: Term::new(9), + membership: MembershipRevision::new(1).unwrap(), + }, + GroupCommand::Retire { + transition, + publication: [88; 32], + }, + ]; + for command in commands { + let entry = CellGroupEntry { + binding: binding(&route), + command, + }; + let bytes = entry.encode(&route, limits).unwrap(); + assert_eq!( + CellGroupEntry::decode(&bytes, &route, limits).unwrap(), + entry + ); + assert_eq!(&bytes[8..12], &1_u32.to_be_bytes()); + if matches!(entry.command, GroupCommand::Retire { .. }) { + assert_eq!(bytes.len(), 289); + assert_eq!( + hex::encode(&bytes[bytes.len() - 32..]), + "77aa30713c91abfafa483e9709b1680271a04873894bbd60bd9374114ed24000" + ); + } + } + let mut repeated = membership; + for transition in [ + GroupMembershipTransition::Auto, + GroupMembershipTransition::JointImplicit, + GroupMembershipTransition::JointExplicit, + ] { + for role in [ + GroupMemberRole::Voter, + GroupMemberRole::Learner, + GroupMemberRole::Remove, + ] { + let mut command = repeated.clone(); + if let GroupCommand::Membership { + transition: mode, + changes, + .. + } = &mut command + { + *mode = transition; + changes[0].role = role; + } + let entry = CellGroupEntry { + binding: binding(&route), + command, + }; + let bytes = entry.encode(&route, limits).unwrap(); + assert_eq!( + CellGroupEntry::decode(&bytes, &route, limits).unwrap(), + entry + ); + } + } + let mut two_members = repeated.clone(); + if let GroupCommand::Membership { changes, .. } = &mut two_members { + changes.push(GroupMemberChange { + member: MemberId::new(2).unwrap(), + node: NodeId::new([9; 16]).unwrap(), + disk: DiskLineage::new([10; 16]).unwrap(), + role: GroupMemberRole::Learner, + }); + } + let mut duplicated_member = CellGroupEntry { + binding: binding(&route), + command: two_members, + } + .encode(&route, limits) + .unwrap(); + let second_member = 24 + 185 + 8 + 8 + 1 + 4 + 41; + duplicated_member[second_member..second_member + 8].copy_from_slice(&1_u64.to_be_bytes()); + rehash(&mut duplicated_member); + assert_eq!( + CellGroupEntry::decode(&duplicated_member, &route, limits), + Err(ProtocolError::Identity) + ); + if let GroupCommand::Membership { changes, .. } = &mut repeated { + changes.push(changes[0]); + } + assert_eq!( + CellGroupEntry { + binding: binding(&route), + command: repeated + } + .encode(&route, limits), + Err(ProtocolError::Identity) + ); + let mut duplicate_child = seal.clone(); + if let GroupCommand::Seal { children, .. } = &mut duplicate_child { + children[1].history.group = children[0].history.group; + } + assert!(CellGroupEntry { + binding: binding(&route), + command: duplicate_child + } + .encode(&route, limits) + .is_err()); + let mut wrong_hash = seal.clone(); + if let GroupCommand::Seal { children, .. } = &mut wrong_hash { + children[0].range = CellRange::new( + CellBound::NegativeInfinity, + CellBound::Key(CellKey::from_digest([0; 32], CellId::new("alpha").unwrap())), + ) + .unwrap(); + } + assert!(CellGroupEntry { + binding: binding(&route), + command: wrong_hash + } + .encode(&route, limits) + .is_err()); + let mut bounded = limits; + bounded.max_transition_groups = 1; + let bytes = CellGroupEntry { + binding: binding(&route), + command: seal, + } + .encode(&route, limits) + .unwrap(); + assert_eq!( + CellGroupEntry::decode(&bytes, &route, bounded), + Err(ProtocolError::Limit) + ); + assert!(ParkGeneration::new(0).is_err()); + assert!(ParkGeneration::new(u64::MAX) + .unwrap() + .checked_next() + .is_err()); + assert!(TransitionId::new([0; 16]).is_err()); +} + +#[test] +fn group_enrollment_contains_replayable_initial_image() { + use cellar::application::consensus::{ImageCommitment, ProtocolError}; + use sha2::{Digest, Sha256}; + let initial = image_fixture::image( + None, + "CREATE TABLE seed(n INTEGER); INSERT INTO seed VALUES (17);", + ) + .unwrap(); + let seed = image_fixture::Cut::new(&[], &initial, 0, true, true); + let route = group_route(); + let limits = GroupEntryLimits::default(); + let entry = CellGroupEntry { + binding: binding(&route), + command: GroupCommand::Enroll { + cell: observation().version.history, + image: image_value(seed.result), + files: vec![GroupLtx { + min_txid: 1, + max_txid: 1, + page_count: seed.pages.len() as u64, + decoded_bytes: seed.pages.len() as u64 * 512, + bytes: &seed.bytes, + }], + }, + }; + let bytes = entry.encode(&route, limits).unwrap(); + let decoded = CellGroupEntry::decode(&bytes, &route, limits).unwrap(); + let GroupCommand::Enroll { image, files, .. } = decoded.command else { + panic!("enrollment changed kind") + }; + // A remote replica can reconstruct this initial SQLite image from the log + // bytes alone, without the proposing host's disk or an object-store read. + let recovered = cellar_ltx::ltx::decode_database_image(files[0].bytes).unwrap(); + assert_eq!(recovered, initial); + assert_eq!( + image.commitment, + ImageCommitment::Sha256(Sha256::digest(&recovered).into()) + ); + let mut no_bytes = entry.clone(); + if let GroupCommand::Enroll { files, .. } = &mut no_bytes.command { + files.clear(); + } + assert_eq!(no_bytes.encode(&route, limits), Err(ProtocolError::Limit)); + let mut missing_prefix = entry; + if let GroupCommand::Enroll { files, .. } = &mut missing_prefix.command { + files[0].min_txid = 2; + } + assert_eq!( + missing_prefix.encode(&route, limits), + Err(ProtocolError::Chain) + ); +} diff --git a/crates/cellar/tests/native_cell_groups_effects.rs b/crates/cellar/tests/native_cell_groups_effects.rs new file mode 100644 index 000000000..c722d44cd --- /dev/null +++ b/crates/cellar/tests/native_cell_groups_effects.rs @@ -0,0 +1,98 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Shared-group effect scheduling and disposable execution resource lifetimes. +#![allow(clippy::disallowed_methods)] +// The shared engine fixture has helpers selected by sibling native targets. +#![allow(dead_code, unused_imports)] + +#[path = "support/cell_group_archive_store.rs"] +mod archive_store; +#[path = "support/cell_group_checkpoint.rs"] +mod checkpoint_fixture; +#[path = "support/raft_storage_fault.rs"] +mod fault; +#[path = "support/raft_ltx_fixture.rs"] +mod images; +#[path = "support/cell_group_runtime.rs"] +mod runtime_fixture; +#[path = "support/cell_group_rustfs.rs"] +mod rustfs; + +include!("support/cell_group_engine_fixture.rs"); + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn effect_identity_stable_across_group_binding() -> Result<()> { + Box::pin(runtime_fixture::effect_identity_across_bindings()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn idle_cache_reopens_exact_cell_base() -> Result<()> { + Box::pin(runtime_fixture::idle_cache_reopens_exact_base()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn materialized_cleanup_failure_preserves_outcome_and_group_capacity() -> Result<()> { + Box::pin(runtime_fixture::cleanup_failure_retains_outcome()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn due_effect_without_client_request() -> Result<()> { + Box::pin(runtime_fixture::alarms_without_clients()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn hot_due_cell_rotates_without_hydrating_cold_cells() -> Result<()> { + Box::pin(runtime_fixture::hot_and_cold_alarms()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn pending_command_or_dispatch_pins_cache() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::routed_commands(rooted)).await?; + } + Box::pin(runtime_fixture::effect_leader_change()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn cache_and_effect_shutdown_joins() -> Result<()> { + Box::pin(runtime_fixture::shutdown_joins_alarm()).await?; + Box::pin(runtime_fixture::effect_delivery_shutdown()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn receiver_lost_reply_keeps_exact_delivery_identity() -> Result<()> { + Box::pin(runtime_fixture::effect_lost_reply()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn s3_claim_receiver_and_completion_use_verified_coverage() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::effect_s3_lost_reply( + rooted, + Arc::new(object_store::memory::InMemory::new()), + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn committed_s3_output_survives_later_group_apply() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::output_during_apply(rooted)).await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS"] +async fn rustfs_claim_receiver_and_completion_use_verified_coverage() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::effect_s3_lost_reply( + rooted, + rustfs::store("cell-group-effects", rooted)?, + )) + .await?; + } + Ok(()) +} diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs new file mode 100644 index 000000000..20b755de9 --- /dev/null +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -0,0 +1,1500 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Real raft-rs owners, three independent engines and cell-addressed SQLite/LTX. +//! Selected fixed-pool and consensus boundaries; not SDK or snapshot qualification. +#![allow(clippy::disallowed_methods)] + +#[path = "support/cell_group_archive_store.rs"] +#[allow(dead_code)] +mod archive_store; +#[path = "support/raft_storage_fault.rs"] +#[allow(dead_code)] +mod fault; + +#[path = "support/cell_group_runtime.rs"] +// The shared fixture also owns cases selected by native_cell_groups_effects. +#[allow(dead_code, unused_imports)] +mod runtime_fixture; + +#[path = "support/cell_group_checkpoint.rs"] +mod checkpoint_fixture; + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn background_sql_admits_one_pending_command() -> Result<()> { + Box::pin(runtime_fixture::background_admission(false)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn command_expired_before_sql_dequeue_does_not_execute() -> Result<()> { + Box::pin(runtime_fixture::background_admission(true)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn repair_budget_under_load() -> Result<()> { + Box::pin(runtime_fixture::repair_budget_under_load()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn many_groups_mixed_roles() -> Result<()> { + Box::pin(runtime_fixture::mixed_roles()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn hot_group_fairness() -> Result<()> { + Box::pin(runtime_fixture::hot_group_fairness()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn controller_effects_outbox_retries_lost_receiver_receipt() -> Result<()> { + Box::pin(runtime_fixture::effect_lost_reply()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn controller_effects_outbox_leader_change_preserves_dispatch() -> Result<()> { + Box::pin(runtime_fixture::effect_leader_change()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn controller_effects_outbox_shutdown_joins_delivery() -> Result<()> { + Box::pin(runtime_fixture::effect_delivery_shutdown()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn controller_effects_shutdown_joins_held_alarm_callback() -> Result<()> { + Box::pin(runtime_fixture::shutdown_joins_alarm()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn controller_effects_alarms_run_without_client_traffic() -> Result<()> { + Box::pin(runtime_fixture::alarms_without_clients()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn runtime_archive_capture_retains_cut_without_blocking_quorum() -> Result<()> { + Box::pin(runtime_fixture::capture_archive_cut()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn seed_retry_materialized_reclaims_interrupted_disk() -> Result<()> { + Box::pin(runtime_fixture::seed_retry(false)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn seed_retry_extents_reclaim_interrupted_disk() -> Result<()> { + Box::pin(runtime_fixture::seed_retry(true)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn node_reclaim_materialized_preserves_complete_shared_inventory() -> Result<()> { + Box::pin(runtime_fixture::shared_inventory(false)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn node_reclaim_extents_preserve_complete_shared_inventory() -> Result<()> { + Box::pin(runtime_fixture::shared_inventory(true)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn seed_transfer_materialized_revocation_retains_physical_owner() -> Result<()> { + Box::pin(runtime_fixture::seed_transfer_revocation(false)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn seed_transfer_extents_revocation_retains_physical_owner() -> Result<()> { + Box::pin(runtime_fixture::seed_transfer_revocation(true)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_pressure_materialized_preserves_bounds_and_reader() -> Result<()> { + runtime_fixture::transfer_pressure(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_pressure_extents_preserves_bounds_and_reader() -> Result<()> { + runtime_fixture::transfer_pressure(true).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn membership_crash_materialized_recovers_and_completes_joint_consensus() -> Result<()> { + Box::pin(runtime_fixture::membership_crash(false)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn membership_crash_extents_recover_and_complete_joint_consensus() -> Result<()> { + Box::pin(runtime_fixture::membership_crash(true)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "owned child process controlled by membership crash tests"] +async fn membership_process_fixture() -> Result<()> { + Box::pin(runtime_fixture::membership_process_fixture()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn seed_crash_materialized_recovers_atomic_install() -> Result<()> { + Box::pin(runtime_fixture::seed_crash(false)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn seed_crash_extents_recovers_atomic_install() -> Result<()> { + Box::pin(runtime_fixture::seed_crash(true)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "owned child process controlled by seed crash tests"] +async fn seed_process_fixture() -> Result<()> { + Box::pin(runtime_fixture::seed_process_fixture()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn seed_persistence_materialized_revocation_and_sync_outcomes() -> Result<()> { + Box::pin(runtime_fixture::seed_persistence(false)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn seed_persistence_extents_revocation_and_sync_outcomes() -> Result<()> { + Box::pin(runtime_fixture::seed_persistence(true)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn seed_receiver_materialized_refuses_forged_transfers_and_revoked_process() -> Result<()> { + Box::pin(runtime_fixture::seed_receiver_faults(false)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn seed_receiver_extents_refuse_forged_transfers_and_revoked_process() -> Result<()> { + Box::pin(runtime_fixture::seed_receiver_faults(true)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn seed_install_materialized_replaces_deleted_voter_and_resumes() -> Result<()> { + Box::pin(runtime_fixture::seed_install(false)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn seed_install_extents_replaces_deleted_voter_and_resumes() -> Result<()> { + Box::pin(runtime_fixture::seed_install(true)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn seed_group_materialized_requires_quorum_and_bound_membership() -> Result<()> { + runtime_fixture::seed_source(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn seed_group_extents_require_quorum_and_bound_membership() -> Result<()> { + runtime_fixture::seed_source(true).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn seed_group_refuses_isolated_leader_and_expired_stream() -> Result<()> { + runtime_fixture::seed_minority().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn membership_group_materialized_joint_changes_preserve_cells() -> Result<()> { + runtime_fixture::membership_lifecycle(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn membership_group_extent_joint_changes_preserve_cells() -> Result<()> { + runtime_fixture::membership_lifecycle(true).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn membership_group_configuration_waits_for_sync_and_joins_close() -> Result<()> { + runtime_fixture::membership_sync().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn membership_delivery_fence_holds_prior_messages_and_allows_later_progress() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::membership_delivery_fence(rooted)).await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn membership_delivery_fence_outlives_dropped_physical_waiter() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::membership_physical_delivery_fence(rooted)).await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn membership_quorum_waits_for_durable_floor_before_its_barrier() -> Result<()> { + runtime_fixture::membership_quorum_floor().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn membership_follower_quorum_preserves_group_root_without_leader_authority() -> Result<()> { + runtime_fixture::membership_quorum_followers().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_controller_compacts_materialized_groups_during_writes() -> Result<()> { + runtime_fixture::automatic_checkpoints(false, false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_controller_compacts_extent_groups_during_writes() -> Result<()> { + runtime_fixture::automatic_checkpoints(true, false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_controller_busy_pool_preserves_apply_and_other_group_progress() -> Result<()> { + runtime_fixture::automatic_checkpoints(false, true).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_controller_materialized_progresses_with_full_apply_capacity() -> Result<()> { + runtime_fixture::checkpoint_during_apply(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_controller_extents_progress_with_full_apply_capacity() -> Result<()> { + runtime_fixture::checkpoint_during_apply(true).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_materialized_waits_for_full_completion_admission() -> Result<()> { + runtime_fixture::checkpoint_completion_pressure(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_extents_waits_for_full_completion_admission() -> Result<()> { + runtime_fixture::checkpoint_completion_pressure(true).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_materialized_concurrent_writers_share_capture_capacity() -> Result<()> { + runtime_fixture::concurrent_checkpoints(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_extents_concurrent_writers_share_capture_capacity() -> Result<()> { + runtime_fixture::concurrent_checkpoints(true).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_controller_materialized_holds_sync_and_joins_cancelled_close() -> Result<()> { + runtime_fixture::controller_checkpoint_shutdown(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_controller_extents_hold_sync_and_join_cancelled_close() -> Result<()> { + runtime_fixture::controller_checkpoint_shutdown(true).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_tcp_refuses_invalid_streams_and_preserves_other_group_progress() -> Result<()> { + runtime_fixture::tcp_protocol().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_tcp_accepted_transfer_survives_dropped_delivery() -> Result<()> { + runtime_fixture::tcp_cancelled_delivery().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_tcp_failed_transfer_allows_raft_retry() -> Result<()> { + runtime_fixture::tcp_retry().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_tcp_materialized_catches_up_and_reopens() -> Result<()> { + runtime_fixture::tcp_publication(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_tcp_extents_catch_up_and_reopen() -> Result<()> { + runtime_fixture::tcp_publication(true).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_local_publication_preserves_suffix_and_catches_up() -> Result<()> { + runtime_fixture::local_publication(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_local_extents_preserve_suffix_and_catch_up() -> Result<()> { + runtime_fixture::local_publication(true).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_local_runtime_joins_publication_shutdown() -> Result<()> { + runtime_fixture::local_shutdown().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_owner_rejects_overflow_cuts_before_raft() -> Result<()> { + runtime_fixture::checkpoint_bounds().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_owner_replays_matching_log_without_install() -> Result<()> { + runtime_fixture::checkpoint_matching_log().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_owner_restores_only_after_raft_sync() -> Result<()> { + runtime_fixture::checkpoint_owner(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_owner_restores_extents_only_after_raft_sync() -> Result<()> { + runtime_fixture::checkpoint_owner(true).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_runtime_joins_accepted_snapshot_shutdown() -> Result<()> { + runtime_fixture::checkpoint_shutdown().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_receive_installs_and_recovers_materialized_closure() -> Result<()> { + runtime_fixture::install_checkpoint(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_receive_installs_and_recovers_extent_closure() -> Result<()> { + runtime_fixture::install_checkpoint(true).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_receive_rejects_partial_corrupt_and_over_budget_streams() -> Result<()> { + runtime_fixture::refuse_checkpoint().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_capture_streams_one_cut_during_writes() -> Result<()> { + checkpoint_fixture::coherent().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_capture_rejects_stale_foreign_and_unpublished_roots() -> Result<()> { + checkpoint_fixture::refusals().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_capture_retains_bounds_until_physical_release() -> Result<()> { + checkpoint_fixture::ownership().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_stream_detects_corruption_and_closed_engine() -> Result<()> { + checkpoint_fixture::corruption().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn command_construction_reserves_admission_and_original_deadline() -> Result<()> { + Box::pin(runtime_fixture::command_admission_starts_before_poll()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn controller_owned_commands_survive_sql_disconnect() -> Result<()> { + runtime_fixture::owned_commands(0, 0).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn controller_owned_commands_seal_late_preparations() -> Result<()> { + runtime_fixture::owned_commands(0, 1).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn controller_owned_commands_survive_accepted_disconnect() -> Result<()> { + runtime_fixture::owned_commands(1, 0).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn controller_owned_commands_resolve_after_response_deadline() -> Result<()> { + runtime_fixture::owned_commands(1, 1).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn controller_owned_commands_join_cancelled_shutdown() -> Result<()> { + runtime_fixture::owned_commands(0, 2).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn controller_owned_commands_release_completed_callback_errors() -> Result<()> { + runtime_fixture::owned_command_failure(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn controller_owned_commands_join_a_lost_sql_reply() -> Result<()> { + runtime_fixture::owned_command_failure(true).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn group_controller_drives_tls_apply_and_bounded_retry() -> Result<()> { + runtime_fixture::controller_replication(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn rooted_group_controller_drives_tls_apply_and_bounded_retry() -> Result<()> { + runtime_fixture::controller_replication(true).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn group_controller_checks_exact_owners() -> Result<()> { + runtime_fixture::controller_owners().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn group_controller_cancelled_close_joins_sql() -> Result<()> { + runtime_fixture::controller_close().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn group_controller_apply_failure_and_shutdown_preserve_selected_root() -> Result<()> { + runtime_fixture::controller_apply_faults(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn rooted_group_controller_apply_failure_and_shutdown_preserve_selected_root() -> Result<()> { + runtime_fixture::controller_apply_faults(true).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn sql_submission_rejection_seals_delayed_retries() -> Result<()> { + runtime_fixture::submission_rejections(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn rooted_sql_submission_rejection_seals_delayed_retries() -> Result<()> { + runtime_fixture::submission_rejections(true).await +} + +#[path = "support/cell_group_admission.rs"] +mod admission_fixture; + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn pending_enrollments_respect_group_capacity() -> Result<()> { + admission_fixture::capacity().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn proposal_acceptance_retries_do_not_append_twice() -> Result<()> { + admission_fixture::mutation_retries().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn inherited_enrollments_count_after_applied_term_barrier() -> Result<()> { + admission_fixture::restart_capacity().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn node_runtime_bounds_proposal_acceptance_across_groups() -> Result<()> { + runtime_fixture::shared_proposal_budget().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn node_runtime_replicates_independent_cells() -> Result<()> { + runtime_fixture::replicated_cells().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn node_runtime_cancelled_close_joins_physical_sync() -> Result<()> { + runtime_fixture::cancelled_close_joins_sync().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn node_runtime_batches_distinct_group_histories() -> Result<()> { + runtime_fixture::shared_batches().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn node_runtime_reserved_completion_releases_apply_admission() -> Result<()> { + runtime_fixture::bounded_apply_completion().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn node_runtime_attachment_reserves_before_owner() -> Result<()> { + runtime_fixture::attachment_reserves_before_owner().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn cell_receipt_not_group_progress() -> Result<()> { + runtime_fixture::cell_receipts(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn rooted_cell_receipt_not_group_progress() -> Result<()> { + runtime_fixture::cell_receipts(true).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_group_fair_admission() -> Result<()> { + runtime_fixture::sql_pool_admission(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn rooted_shared_group_fair_admission() -> Result<()> { + runtime_fixture::sql_pool_admission(true).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn cell_sql_reconciles_after_leadership_change() -> Result<()> { + runtime_fixture::reconcile_sql_producers(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn rooted_cell_sql_reconciles_after_leadership_change() -> Result<()> { + runtime_fixture::reconcile_sql_producers(true).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn cell_sql_reconciles_expired_and_conflicting_receipts() -> Result<()> { + runtime_fixture::reconcile_sql_refusals(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn rooted_cell_sql_reconciles_expired_and_conflicting_receipts() -> Result<()> { + runtime_fixture::reconcile_sql_refusals(true).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn concurrent_enrollment_one_cell() -> Result<()> { + runtime_fixture::initialize_cells(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn rooted_concurrent_enrollment_one_cell() -> Result<()> { + runtime_fixture::initialize_cells(true).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn rooted_enrollment_and_commands_reuse_owned_workspace_barriers() -> Result<()> { + Box::pin(runtime_fixture::owned_workspace_barriers()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn enrollment_recovery_after_leadership_change() -> Result<()> { + runtime_fixture::recover_initialization(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn rooted_enrollment_recovery_after_leadership_change() -> Result<()> { + runtime_fixture::recover_initialization(true).await +} + +#[path = "support/raft_ltx_fixture.rs"] +#[allow(dead_code)] +mod images; + +include!("support/cell_group_engine_fixture.rs"); + +use raft::prelude::Entry; + +impl Cluster { + async fn restart(&mut self, n: usize) -> Result<()> { + let mut old = self.voters.remove(n); + drop(old.held.take()); + drop(old.owner.take()); + old.log.close().await?; + drop(old); + let number = n as u8 + 1; + let base = self._root.path().join(format!("node-{number}")); + let gate = fault::Gate::new(None); + let log = self + .domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + base.join("log"), + NodeId::new([number; 16])?, + DiskLineage::new([number + 10; 16])?, + ProcessGeneration::new([21; 16])?, + LogOpenMode::Resume, + ), + Arc::new(fault::FaultFileSystem { gate: gate.clone() }), + )) + .await??; + let permit = self + .domain + .run(self.directory.bootstrap_group_member(&self.birth, &log)) + .await??; + let tree = ManifestTree::new(self.birth.route().clone(), ManifestLimits::default())?; + let mut pages = ManifestPageStore::open( + &base.join("pages"), + Arc::new(DirectFileSystem), + 1024, + 16 * 1024 * 1024, + 16, + )?; + let images = GenerationStore::open( + base.join("images"), + Arc::new(DirectFileSystem), + Default::default(), + )?; + let marker = permit.group().group_manifest()?.context("selected root")?; + let prepared = tree.prepare_complete( + permit.group(), + &mut pages, + marker.root(), + marker.position(), + marker.membership_revision(), + |leaf| { + let CellImageCommitment::Materialized(digest) = leaf.version.image.commitment + else { + return Err(ManifestError::Invalid("fixture materialized image")); + }; + let state = ImageState { + epoch: leaf.version.image.epoch, + txid: TXID(leaf.version.image.txid), + checksum: leaf.version.image.checksum, + digest, + length: leaf.geometry.length, + page_size: leaf.geometry.page_size, + }; + let id = image_id(&leaf.version.history, leaf.version.sequence.get()) + .map_err(|_| ManifestError::Invalid("fixture image id"))?; + images + .adopt(id, state) + .map(CellImagePin::Materialized) + .map_err(|_| ManifestError::Invalid("fixture recovery image")) + }, + )?; + let owner = self + .domain + .run(CellGroupOwner::start( + permit, + prepared, + self.options.clone(), + [number; 32], + )) + .await??; + self.voters.insert( + n, + Voter { + log, + owner: Some(owner), + tree, + pages: Some(pages), + images, + extents: None, + gate, + paused: false, + held: None, + }, + ); + Ok(()) + } +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn three_cells_one_rawnode_per_voter() -> Result<()> { + let mut cluster = Cluster::new().await?; + cluster.elect().await?; + let route = cluster.birth.route().clone(); + let mut expected = Vec::new(); + for (n, name) in ["alpha", "beta", "gamma"].into_iter().enumerate() { + let bytes = images::image( + None, + &format!( + "CREATE TABLE value(n); INSERT INTO value VALUES ({});", + n + 1 + ), + )?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let cell = history(&route, name)?; + cluster.voters[0].owner.as_mut().unwrap().propose(request( + &route, + cell.clone(), + 1, + &cut, + )?)?; + cluster.drive(false).await?; + expected.push((cell, bytes)); + } + for step in 0..9 { + let i = step % 3; + let (cell, before) = &mut expected[i]; + let sequence = step / 3 + 2; + let after = images::image(Some(before), "UPDATE value SET n = n + 10;")?; + let cut = images::Cut::new(before, &after, sequence as u64 - 1, true, false); + cluster.voters[0].owner.as_mut().unwrap().propose(request( + &route, + cell.clone(), + sequence as u64, + &cut, + )?)?; + cluster.drive(false).await?; + *before = after; + } + assert_eq!(cluster.voters.len(), 3); + for n in 0..3 { + let owner = cluster.voters[n].owner.as_ref().unwrap(); + assert_eq!(owner.manifest().marker().root().cells(), 3); + for (cell, bytes) in &expected { + let (leaf, CellImagePin::Materialized(pin)) = + owner.manifest().pin_cell(&cell.cell).context("cell")? + else { + anyhow::bail!("image") + }; + assert_eq!(leaf.version.history, *cell); + assert_eq!(leaf.version.sequence.get(), 4); + assert_eq!(&images::read(&pin)?, bytes); + } + let duplicate = cluster + .domain + .run(CellGroupOwner::start( + cluster.permit(n).await?, + owner.manifest().clone(), + RaftRuntimeOptions::default(), + [50; 32], + )) + .await?; + assert!(matches!(duplicate, Err(RuntimeError::AlreadyAttached))); + assert_eq!( + owner.manifest().marker(), + cluster.voters[0] + .owner + .as_ref() + .unwrap() + .manifest() + .marker() + ); + } + cluster.voters[0] + .owner + .as_mut() + .unwrap() + .read_index(b"barrier".to_vec())?; + cluster.drive(false).await?; + assert!(cluster.reads.iter().any(|(n, term, index, context)| *n == 0 + && context == b"barrier" + && *term + == cluster.voters[0] + .owner + .as_ref() + .unwrap() + .status() + .unwrap() + .term + && *index + <= cluster.voters[0] + .owner + .as_ref() + .unwrap() + .manifest() + .marker() + .position() + .index)); + cluster.close().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn physical_owner_rejects_exhausted_peer_entry_counters() -> Result<()> { + let mut cluster = Cluster::new().await?; + let before = cluster.voters[0].owner.as_ref().unwrap().status()?; + let message = Message { + msg_type: MessageType::MsgAppend, + from: 2, + to: 1, + term: 2, + log_term: 1, + index: u64::MAX - 3, + entries: vec![Entry { + term: 1, + index: u64::MAX - 2, + ..Default::default() + }] + .into(), + ..Default::default() + }; + assert!(matches!( + cluster.voters[0].owner.as_mut().unwrap().step(message), + Err(RuntimeError::Invalid(_)) + )); + let after = cluster.voters[0].owner.as_ref().unwrap().status()?; + assert_eq!(before.term, after.term); + assert_eq!(before.applied, after.applied); + cluster.elect().await?; + cluster.close().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn physical_owner_requires_read_quorum() -> Result<()> { + let mut cluster = Cluster::new().await?; + cluster.elect().await?; + cluster.voters[0] + .owner + .as_mut() + .unwrap() + .read_index(b"minority".to_vec())?; + cluster.drive(true).await?; + assert!(cluster.reads.is_empty()); + for _ in 0..50 { + cluster.voters[0].owner.as_mut().unwrap().tick()?; + cluster.drive(true).await?; + } + assert!(!cluster.voters[0].owner.as_ref().unwrap().status()?.ready); + assert!(cluster.voters[0] + .owner + .as_mut() + .unwrap() + .read_index(b"stale".to_vec()) + .is_err()); + cluster.close().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn apply_prefix_never_skips_slow_cell() -> Result<()> { + let mut cluster = Cluster::new().await?; + cluster.elect().await?; + let route = cluster.birth.route().clone(); + let mut expected = Vec::new(); + for name in ["slow", "later"] { + let cell = history(&route, name)?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES (1);")?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + cluster.voters[0].owner.as_mut().unwrap().propose(request( + &route, + cell.clone(), + 1, + &cut, + )?)?; + cluster.drive(false).await?; + expected.push((cell, bytes)); + } + let before = cluster.voters[0] + .owner + .as_ref() + .unwrap() + .manifest() + .marker() + .clone(); + cluster.voters[1].paused = true; + cluster.voters[2].paused = true; + for (cell, bytes) in &mut expected { + let after = images::image(Some(bytes), "UPDATE value SET n = 2;")?; + let cut = images::Cut::new(bytes, &after, 1, true, false); + cluster.voters[0].owner.as_mut().unwrap().propose(request( + &route, + cell.clone(), + 2, + &cut, + )?)?; + cluster.drive(false).await?; + *bytes = after; + } + for voter in &cluster.voters[1..] { + let status = voter.owner.as_ref().unwrap().status()?; + assert_eq!(status.applied, before.position()); + assert_eq!(status.committed.get(), before.position().index.get() + 2); + assert_eq!(voter.owner.as_ref().unwrap().manifest().marker(), &before); + assert_eq!( + voter.held.as_ref().unwrap().ticket.position().index.get(), + before.position().index.get() + 1 + ); + } + let foreign = cluster.voters[2].held.as_ref().unwrap().ticket.clone(); + let voter = &mut cluster.voters[1]; + let apply = voter.held.take().context("held slow apply")?; + let next = prepare(voter, &apply)?; + assert!(matches!( + voter + .owner + .as_mut() + .unwrap() + .complete_apply(foreign, next.clone()), + Err(RuntimeError::StaleCompletion) + )); + // Correct image and apply position cannot authorize a fabricated proposal + // witness. Completion must match the exact committed cell-addressed entry. + let (leaf, image) = next.pin_cell(&expected[0].0.cell).context("changed cell")?; + let mut forged_leaf = leaf.clone(); + forged_leaf.witness.proposal = ProposalId::new([254; 16])?; + assert_ne!(forged_leaf.witness, leaf.witness); + let old = apply + .base + .pin_cell(&expected[0].0.cell) + .context("old cell")? + .0; + let forged = voter.tree.prepare_update( + voter.pages.as_mut().context("fixture page owner")?, + &apply.base, + forged_leaf, + Some(&old.version), + image, + )?; + assert!(matches!( + voter + .owner + .as_mut() + .unwrap() + .complete_apply(apply.ticket.clone(), forged,), + Err(RuntimeError::Invalid(_)) + )); + assert_eq!(voter.owner.as_ref().unwrap().manifest().marker(), &before); + let unchanged = voter.tree.prepare_unchanged( + voter.pages.as_ref().context("fixture page owner")?, + &apply.base, + apply.ticket.position(), + )?; + assert!(matches!( + voter + .owner + .as_mut() + .unwrap() + .complete_apply(apply.ticket.clone(), unchanged), + Err(RuntimeError::Invalid(_)) + )); + voter.paused = false; + voter + .owner + .as_mut() + .unwrap() + .complete_apply(apply.ticket, next)?; + cluster.drive(false).await?; + assert_eq!( + cluster.voters[1] + .owner + .as_ref() + .unwrap() + .manifest() + .marker(), + cluster.voters[0] + .owner + .as_ref() + .unwrap() + .manifest() + .marker() + ); + + // Restart the other follower with both entries committed but neither applied. + // The restored marker stays behind until the real owner replays its log tail. + cluster.restart(2).await?; + assert_eq!( + cluster.voters[2] + .owner + .as_ref() + .unwrap() + .manifest() + .marker(), + &before + ); + cluster.drive(false).await?; + for voter in &cluster.voters { + let owner = voter.owner.as_ref().unwrap(); + assert_eq!( + owner.manifest().marker(), + cluster.voters[0] + .owner + .as_ref() + .unwrap() + .manifest() + .marker() + ); + for (cell, bytes) in &expected { + let (leaf, CellImagePin::Materialized(pin)) = owner + .manifest() + .pin_cell(&cell.cell) + .context("restored cell")? + else { + anyhow::bail!("image") + }; + assert_eq!(leaf.version.sequence.get(), 2); + assert_eq!(&images::read(&pin)?, bytes); + } + } + cluster.close().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn physical_owner_backpressures_slow_apply_without_losing_commits() -> Result<()> { + let mut cluster = Cluster::with_options(RaftRuntimeOptions { + max_proposal_bytes: 8192, + max_message_bytes: 12_288, + max_group_apply_bytes: 16_384, + max_apply_bytes: 32_768, + ..Default::default() + }) + .await?; + cluster.elect().await?; + let route = cluster.birth.route().clone(); + let cell = history(&route, "busy")?; + let mut bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES (1);")?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + cluster.voters[0] + .owner + .as_mut() + .unwrap() + .propose(request(&route, cell.clone(), 1, &cut)?)?; + cluster.drive(false).await?; + let before = cluster.voters[2] + .owner + .as_ref() + .unwrap() + .manifest() + .marker() + .clone(); + cluster.voters[2].paused = true; + for sequence in 2..=40 { + let after = images::image(Some(&bytes), "UPDATE value SET n = n + 1;")?; + let cut = images::Cut::new(&bytes, &after, sequence - 1, true, false); + cluster.voters[0].owner.as_mut().unwrap().propose(request( + &route, + cell.clone(), + sequence, + &cut, + )?)?; + cluster.drive(false).await?; + bytes = after; + } + assert!( + !cluster.network.is_empty(), + "the slow voter must exert real transport backpressure" + ); + assert_eq!( + cluster.voters[2] + .owner + .as_ref() + .unwrap() + .manifest() + .marker(), + &before + ); + let voter = &mut cluster.voters[2]; + let apply = voter.held.take().context("held apply")?; + let next = prepare(voter, &apply)?; + voter.paused = false; + voter + .owner + .as_mut() + .unwrap() + .complete_apply(apply.ticket, next)?; + cluster.drive(false).await?; + assert!(cluster.network.is_empty()); + for voter in &cluster.voters { + let (leaf, CellImagePin::Materialized(pin)) = voter + .owner + .as_ref() + .unwrap() + .manifest() + .pin_cell(&cell.cell) + .context("caught-up cell")? + else { + anyhow::bail!("image") + }; + assert_eq!(leaf.version.sequence.get(), 40); + assert_eq!(images::read(&pin)?, bytes); + } + cluster.close().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn physical_owner_retains_lease_through_cancelled_sync() -> Result<()> { + let mut cluster = Cluster::new().await?; + cluster.elect().await?; + let permit = cluster.permit(0).await?; + let root = cluster.voters[0].owner.as_ref().unwrap().manifest().clone(); + let route = cluster.birth.route().clone(); + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES (1);")?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let entry = request(&route, history(&route, "held")?, 1, &cut)?; + let owner = cluster.voters[0].owner.as_mut().unwrap(); + owner.propose(entry)?; + let CellGroupEffect::Persist(work) = owner.poll()?.context("append persistence")? else { + anyhow::bail!("append must persist before sending") + }; + assert!(owner.poll()?.is_none(), "no append messages before sync"); + for _ in 0..100 { + assert!(matches!(owner.tick(), Err(RuntimeError::NotReady))); + } + let gate = cluster.voters[0].gate.clone(); + let release = gate.arm(fault::HOLD_SYNC); + let mut write = Box::pin(work.run()); + match tokio::time::timeout( + Duration::from_secs(5), + futures_util::future::select(&mut write, Box::pin(gate.entered.notified())), + ) + .await? + { + futures_util::future::Either::Left(_) => anyhow::bail!("sync did not enter held gate"), + futures_util::future::Either::Right(_) => (), + } + drop(write); + drop(cluster.voters[0].owner.take()); + let duplicate = cluster + .domain + .run(CellGroupOwner::start( + permit, + root.clone(), + RaftRuntimeOptions::default(), + [61; 32], + )) + .await?; + assert!(matches!(duplicate, Err(RuntimeError::AlreadyAttached))); + drop(release); + cluster.voters[0].log.purge().await?; + let permit = cluster.permit(0).await?; + let owner = cluster + .domain + .run(CellGroupOwner::start( + permit, + root, + RaftRuntimeOptions::default(), + [62; 32], + )) + .await??; + cluster.voters[0].owner = Some(owner); + cluster.close().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn strong_read_at_group_barrier() -> Result<()> { + runtime_fixture::coherent_reads(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn rooted_strong_read_at_group_barrier() -> Result<()> { + runtime_fixture::coherent_reads(true).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn cell_query_pins_deadline_and_cancelled_cleanup() -> Result<()> { + runtime_fixture::query_lifecycle(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn rooted_cell_query_pins_deadline_and_cancelled_cleanup() -> Result<()> { + runtime_fixture::query_lifecycle(true).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_workers_prepare_committed_cell_images() -> Result<()> { + runtime_fixture::apply_images(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn rooted_shared_workers_prepare_committed_cell_images() -> Result<()> { + runtime_fixture::apply_images(true).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_apply_cancelled_close_joins_preparation() -> Result<()> { + runtime_fixture::apply_cancellation(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn rooted_shared_apply_cancelled_close_joins_preparation() -> Result<()> { + runtime_fixture::apply_cancellation(true).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_apply_sync_failure_preserves_selected_root() -> Result<()> { + runtime_fixture::apply_failure(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn rooted_shared_apply_sync_failure_preserves_selected_root() -> Result<()> { + runtime_fixture::apply_failure(true).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_apply_recovers_selected_images_before_replay() -> Result<()> { + runtime_fixture::recover_apply(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn rooted_shared_apply_recovers_selected_images_before_replay() -> Result<()> { + runtime_fixture::recover_apply(true).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn registered_commands_use_cell_identity_and_receipts() -> Result<()> { + runtime_fixture::registered_commands(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn rooted_registered_commands_use_cell_identity_and_receipts() -> Result<()> { + runtime_fixture::registered_commands(true).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn registered_queries_keep_cell_observation_and_ownership() -> Result<()> { + Box::pin(runtime_fixture::registered_queries(false)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn rooted_registered_queries_keep_cell_observation_and_ownership() -> Result<()> { + Box::pin(runtime_fixture::registered_queries(true)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn catalog_routed_queries_preserve_invocation_and_authority() -> Result<()> { + Box::pin(runtime_fixture::routed_queries(false)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn rooted_catalog_routed_queries_preserve_invocation_and_authority() -> Result<()> { + Box::pin(runtime_fixture::routed_queries(true)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn catalog_routed_commands_preserve_producer_and_receipts() -> Result<()> { + Box::pin(runtime_fixture::routed_commands(false)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn rooted_catalog_routed_commands_preserve_producer_and_receipts() -> Result<()> { + Box::pin(runtime_fixture::routed_commands(true)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn physical_group_transport_multiplexes_real_raft() -> Result<()> { + Box::pin(runtime_fixture::physical_transport()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn catalog_commands_replicate_over_physical_tls() -> Result<()> { + Box::pin(runtime_fixture::commands_over_tcp(false)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn rooted_catalog_commands_replicate_over_physical_tls() -> Result<()> { + Box::pin(runtime_fixture::commands_over_tcp(true)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn controller_enrollment_survives_initializer_disconnect() -> Result<()> { + Box::pin(runtime_fixture::owned_enrollment(0, 0)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn controller_enrollment_seals_late_preparation() -> Result<()> { + Box::pin(runtime_fixture::owned_enrollment(0, 1)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn controller_enrollment_survives_accepted_disconnect() -> Result<()> { + Box::pin(runtime_fixture::owned_enrollment(1, 0)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn controller_enrollment_resolves_after_deadline() -> Result<()> { + Box::pin(runtime_fixture::owned_enrollment(1, 1)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn controller_enrollment_joins_cancelled_shutdown() -> Result<()> { + Box::pin(runtime_fixture::owned_enrollment(0, 2)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn controller_enrollment_releases_completed_initializer_errors() -> Result<()> { + Box::pin(runtime_fixture::owned_enrollment_failure(false)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn controller_enrollment_joins_lost_worker_reply() -> Result<()> { + Box::pin(runtime_fixture::owned_enrollment_failure(true)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn controller_enrollment_requires_registered_initializer() -> Result<()> { + Box::pin(runtime_fixture::enrollment_requires_registration()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_remote_queries_bind_cell_versions_and_gateway_owners() -> Result<()> { + Box::pin(runtime_fixture::remote_queries()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_remote_queries_retain_cancelled_sql_until_join() -> Result<()> { + Box::pin(runtime_fixture::remote_query_lifecycle(false)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_remote_queries_retain_expired_sql_until_join() -> Result<()> { + Box::pin(runtime_fixture::remote_query_lifecycle(true)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_nonmember_queries_require_grants_without_local_replicas() -> Result<()> { + Box::pin(runtime_fixture::nonmember_queries()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_nonmember_query_revocation_retains_physical_sql() -> Result<()> { + Box::pin(runtime_fixture::nonmember_query_revocation(false)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_nonmember_query_route_replacement_refuses_stale_reply() -> Result<()> { + Box::pin(runtime_fixture::nonmember_query_revocation(true)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_cached_dispatch_preserves_authority_during_catalog_outage() -> Result<()> { + Box::pin(runtime_fixture::cached_dispatch_execution()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_remote_commands_require_write_grants_and_exact_receipts() -> Result<()> { + Box::pin(runtime_fixture::remote_commands()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_remote_commands_survive_disconnected_sql_waiters() -> Result<()> { + Box::pin(runtime_fixture::remote_command_lifecycle(false)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_remote_command_revocation_preserves_admitted_ownership() -> Result<()> { + Box::pin(runtime_fixture::remote_command_lifecycle(true)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_remote_enrollment_requires_write_grants_and_exact_creation() -> Result<()> { + Box::pin(runtime_fixture::remote_enrollment()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_remote_enrollment_survives_disconnected_initializer() -> Result<()> { + Box::pin(runtime_fixture::remote_enrollment_lifecycle(0)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_remote_enrollment_revocation_retains_admitted_creation() -> Result<()> { + Box::pin(runtime_fixture::remote_enrollment_lifecycle(1)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_remote_enrollment_resolves_after_publication_deadline() -> Result<()> { + Box::pin(runtime_fixture::remote_enrollment_lifecycle(2)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_receipt_lookup_requires_grants_and_never_executes_commands() -> Result<()> { + Box::pin(runtime_fixture::receipt_lookup_ingress()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_receipt_lookup_keeps_unresolved_producers_at_capacity() -> Result<()> { + Box::pin(runtime_fixture::receipt_lookup_pending()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_receipt_lookup_cancellation_preserves_active_writer() -> Result<()> { + Box::pin(runtime_fixture::receipt_lookup_cancelled()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn dynamic_registration_preserves_live_groups_and_disconnected_waiters() -> Result<()> { + runtime_fixture::registration_live().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn dynamic_registration_refuses_foreign_owners_and_retries_failed_catalog() -> Result<()> { + runtime_fixture::registration_refusals().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn dynamic_registration_joins_attachment_before_cancelled_shutdown() -> Result<()> { + runtime_fixture::registration_close().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn dynamic_registration_recovers_selected_roots_into_an_empty_controller() -> Result<()> { + runtime_fixture::registration_recovery().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn group_quarantine_retains_producer_and_serves_healthy_group() -> Result<()> { + runtime_fixture::group_quarantine_retains_producer_and_serves_healthy_group().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_engine_sync_failure_still_stops_all_groups() -> Result<()> { + runtime_fixture::shared_engine_sync_failure_still_stops_all_groups().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn provisioning_coalesces_and_serves() -> Result<()> { + runtime_fixture::provisioning_coalesces_and_serves().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn provisioning_retries_retained_owner() -> Result<()> { + runtime_fixture::provisioning_retries_retained_owner().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn provisioning_joins_cancelled_close() -> Result<()> { + runtime_fixture::provisioning_joins_cancelled_close().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn provisioning_recovers_selected_roots() -> Result<()> { + runtime_fixture::provisioning_recovers_selected_roots().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn provisioning_refusals_preserve_healthy_group() -> Result<()> { + runtime_fixture::provisioning_refusals_preserve_healthy_group().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn retained_log_materialized_refuses_new_callbacks_and_reclaims() -> Result<()> { + Box::pin(runtime_fixture::retained_log_admission(false)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn retained_log_extents_refuse_new_callbacks_and_reclaim() -> Result<()> { + Box::pin(runtime_fixture::retained_log_admission(true)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_follower_linearizable_reads_bind_live_quorum() -> Result<()> { + Box::pin(runtime_fixture::follower_linearizable(false)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn rooted_follower_linearizable_reads_bind_live_quorum() -> Result<()> { + Box::pin(runtime_fixture::follower_linearizable(true)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_recent_source_reads_keep_physical_deadline() -> Result<()> { + Box::pin(runtime_fixture::recent_source_reads(false)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn rooted_recent_source_reads_keep_physical_deadline() -> Result<()> { + Box::pin(runtime_fixture::recent_source_reads(true)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_remote_follower_reads_preserve_constraints() -> Result<()> { + Box::pin(runtime_fixture::remote_follower_reads(false)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn rooted_remote_follower_reads_preserve_constraints() -> Result<()> { + Box::pin(runtime_fixture::remote_follower_reads(true)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 8)] +async fn shared_controller_replica_routing_keeps_one_invocation() -> Result<()> { + Box::pin(runtime_fixture::controller_routed_queries(false)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 8)] +async fn rooted_controller_replica_routing_keeps_one_invocation() -> Result<()> { + Box::pin(runtime_fixture::controller_routed_queries(true)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 8)] +async fn shared_controller_queries_without_local_group_require_grants() -> Result<()> { + Box::pin(runtime_fixture::nonmember_controller_queries(false)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 8)] +async fn rooted_controller_queries_without_local_group_require_grants() -> Result<()> { + Box::pin(runtime_fixture::nonmember_controller_queries(true)).await +} diff --git a/crates/cellar/tests/native_cell_groups_host.rs b/crates/cellar/tests/native_cell_groups_host.rs new file mode 100644 index 000000000..388a74d3b --- /dev/null +++ b/crates/cellar/tests/native_cell_groups_host.rs @@ -0,0 +1,73 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Shared namespace host boundary; canonical SDK/fleet cutover remains separate. +#![allow(clippy::disallowed_methods)] +// Common engine fixture helpers are selected by several independent targets. +#![allow(dead_code, unused_imports)] +#[path = "support/cell_group_archive_store.rs"] +mod archive_store; +#[path = "support/cell_group_checkpoint.rs"] +mod checkpoint_fixture; +#[path = "support/raft_storage_fault.rs"] +mod fault; +#[path = "support/raft_ltx_fixture.rs"] +mod images; +#[path = "support/cell_group_runtime.rs"] +mod runtime_fixture; +include!("support/cell_group_engine_fixture.rs"); +#[path = "support/cell_group_rustfs.rs"] +mod rustfs; + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn sealed_source_archives_require_exact_live_holds() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::sealed_source_archives( + rooted, + Arc::new(object_store::memory::InMemory::new()), + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires explicit owned RustFS endpoint, bucket and isolated prefix"] +async fn rustfs_sealed_source_archives() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::sealed_source_archives( + rooted, + rustfs::store("sealed-source-archives", rooted)?, + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn source_seal_blocks_delayed_work() -> Result<()> { + Box::pin(runtime_fixture::source_seal_blocks_delayed_work()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn namespace_host_many_cells_share_one_physical_group() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::host_many_cells(rooted)).await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn namespace_host_query_cancellation_retains_physical_join() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::host_query_shutdown(rooted)).await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn namespace_host_bounds_identity_and_enrollment_deadline() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::host_registration_and_deadline(rooted)).await?; + } + Ok(()) +} diff --git a/crates/cellar/tests/native_cell_groups_managed.rs b/crates/cellar/tests/native_cell_groups_managed.rs new file mode 100644 index 000000000..b13c93d4a --- /dev/null +++ b/crates/cellar/tests/native_cell_groups_managed.rs @@ -0,0 +1,234 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Managed boot, cold capacity and retained-group qualification over embedded Nodes. +//! Deployment bounds use fixture attestations; the full lifecycle gate is separate. +#![allow(clippy::disallowed_methods)] +use anyhow::{ensure, Context as _, Result}; +use cellar::sdk::{consensus::*, *}; +use managed_fixture::{fault, limits, request, start_node, Fixture, TIMEOUT}; +use std::{ + path::Path, + sync::{atomic::Ordering, Arc}, + time::Duration, +}; + +#[path = "support/cell_group_managed_fixture.rs"] +pub mod managed_fixture; + +#[path = "support/cell_group_managed_bootstrap.rs"] +mod bootstrap; +#[path = "support/cell_group_managed_registry.rs"] +mod registry; +#[path = "support/cell_group_rustfs.rs"] +#[allow(dead_code)] +mod rustfs; + +async fn stopped(node: Node, service: &RaftCells) -> Result<()> { + ensure!(matches!( + node.shutdown(ShutdownMode::Preserve).await?, + NodeOutcome::Stopped(_) + )); + let report = service.close().await?; + ensure!( + report.groups_total == 0 + && report.mutation_outcomes == MutationOutcomes::NoUnresolvedLocalProducers + ); + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn managed_boot_retained_restart_and_exact_stop_owner() -> Result<()> { + for rooted in [false, true] { + let f = Fixture::new(rooted).await?; + let root = tempfile::tempdir()?; + let mut previous: Option = None; + for generation in [24, 25] { + let node = start_node(root.path(), f.application.clone(), &f.spec.label).await?; + let (config, services) = f + .config( + &node, + if previous.is_none() { + LogOpenMode::Create + } else { + LogOpenMode::Resume + }, + generation, + ) + .await?; + let host = node.start_raft_cells(config, services).await?; + let view = host.observe_node(TIMEOUT).await?; + let boot = view.fleet_boot.context("synced boot")?; + ensure!(view.node.groups == 0 && view.declared_namespaces == 1); + ensure!(boot.process == [generation; 16] && boot.node == f.spec.node.bytes()); + if let Some(old) = &previous { + ensure!( + boot.engine == old.engine + && boot.epoch > old.epoch + && boot.boot != old.boot + && boot.join_key != old.join_key + ); + } + let mut foreign = boot.clone(); + foreign.process = [99; 16]; + let request: FleetStopRequest = serde_json::from_value( + serde_json::json!({"selector":{"purpose":"maintenance","boot":foreign,"epoch":boot.epoch+1,"operation":vec![41;16],"finalization":vec![42;16],"request":vec![43;16]},"action":"stop"}), + )?; + ensure!( + matches!( + node.request_fleet_stop(request, TIMEOUT).await, + Err(FleetCatalogError::Conflict) + ), + "Node must install the exact managed stop reporter" + ); + stopped(node, &host).await?; + if previous.is_none() { + let blank = tempfile::tempdir()?; + let candidate = + start_node(blank.path(), f.application.clone(), &f.spec.label).await?; + let (config, services) = f.config(&candidate, LogOpenMode::Create, 26).await?; + let before = f.catalog.read(TIMEOUT).await?.revision(); + ensure!( + candidate.start_raft_cells(config, services).await.is_err(), + "recreated managed engine accepted the pinned disk identity" + ); + ensure!( + f.catalog.read(TIMEOUT).await?.revision() == before, + "invalid engine consumed a rooted boot" + ); + candidate.shutdown(ShutdownMode::Preserve).await?; + } + previous = Some(boot); + } + f.domain.shutdown().await; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn managed_declaration_mismatch_refuses_before_boot_claim() -> Result<()> { + let f = Fixture::new(false).await?; + let root = tempfile::tempdir()?; + let node = start_node(root.path(), f.application.clone(), &f.spec.label).await?; + let (mut config, services) = f.config(&node, LogOpenMode::Create, 24).await?; + config.namespaces[0].namespace.schema_digest = [82; 32]; + let before = f.catalog.read(TIMEOUT).await?.revision(); + ensure!(node.start_raft_cells(config, services).await.is_err()); + ensure!(f.catalog.read(TIMEOUT).await?.revision() == before); + node.shutdown(ShutdownMode::Preserve).await?; + f.domain.shutdown().await; + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn managed_directory_shutdown_joins_entered_root_refresh() -> Result<()> { + let f = Fixture::new(false).await?; + let root = tempfile::tempdir()?; + let node = start_node(root.path(), f.application.clone(), &f.spec.label).await?; + let (config, services) = f.config(&node, LogOpenMode::Create, 24).await?; + let host = node.start_raft_cells(config, services).await?; + // No application traffic drives discovery. A rooted node-table change must + // be installed by the independently timed directory reader. + let mut peer = f.spec.clone(); + peer.node = NodeId::new([5; 16])?; + peer.disk = DiskLineage::new([15; 16])?; + peer.physical_host = IncarnationId::new([91; 16])?; + peer.label = "managed-other".into(); + let cut = f.catalog.read(TIMEOUT).await?; + let cut = f + .catalog + .register_node(&cut, &request(3, cut.revision()), &peer, TIMEOUT) + .await? + .snapshot; + tokio::time::timeout(TIMEOUT, async { + while host + .observe_node(TIMEOUT) + .await? + .fleet_trust_revision + .is_none_or(|revision| revision < cut.revision()) + { + tokio::task::yield_now().await; + } + Ok::<_, anyhow::Error>(()) + }) + .await??; + f.store.fleet_read_hang.store(true, Ordering::Release); + f.store.hold_fleet_reads.store(true, Ordering::Release); + tokio::time::timeout(TIMEOUT, async { + while f.store.active_fleet_reads.load(Ordering::Acquire) == 0 { + tokio::task::yield_now().await; + } + }) + .await?; + let mut closing = Box::pin(node.shutdown(ShutdownMode::Preserve)); + ensure!( + tokio::time::timeout(Duration::from_millis(30), &mut closing) + .await + .is_err(), + "shutdown dropped entered directory work" + ); + ensure!(f.store.active_fleet_reads.load(Ordering::Acquire) > 0); + f.store.hold_fleet_reads.store(false, Ordering::Release); + f.store.fleet_read_hang.store(false, Ordering::Release); + f.store.fleet_read_resume.notify_waiters(); + tokio::time::timeout(TIMEOUT, closing).await??; + ensure!(f.store.active_fleet_reads.load(Ordering::Acquire) == 0); + ensure!(host.close().await?.groups_total == 0); + f.domain.shutdown().await; + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn managed_directory_preserves_explicit_advertised_endpoints() -> Result<()> { + let f = Fixture::new(false).await?; + let root = tempfile::tempdir()?; + let node = start_node(root.path(), f.application.clone(), &f.spec.label).await?; + let (mut config, mut services) = f.config(&node, LogOpenMode::Create, 24).await?; + let listeners = NodeRaftListeners { + control: tokio::net::TcpListener::bind("0.0.0.0:0").await?, + data: tokio::net::TcpListener::bind("0.0.0.0:0").await?, + query: tokio::net::TcpListener::bind("0.0.0.0:0").await?, + }; + let advertised = |bound: std::net::SocketAddr| { + std::net::SocketAddr::new(std::net::Ipv4Addr::LOCALHOST.into(), bound.port()) + }; + let peer = &mut config.peers[0]; + peer.control_address = advertised(listeners.control.local_addr()?); + peer.data_address = advertised(listeners.data.local_addr()?); + peer.query_address = advertised(listeners.query.local_addr()?); + ensure!(peer.control_address != listeners.control.local_addr()?); + services.listeners = listeners; + let host = node.start_raft_cells(config, services).await?; + ensure!(host.observe_node(TIMEOUT).await?.fleet_boot.is_some()); + stopped(node, &host).await?; + f.domain.shutdown().await; + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn managed_owner_mismatch_refuses_before_boot_claim() -> Result<()> { + for wrong_node in [true, false] { + let f = Fixture::new(false).await?; + let root = tempfile::tempdir()?; + let node = start_node(root.path(), f.application.clone(), &f.spec.label).await?; + let (mut config, services) = f.config(&node, LogOpenMode::Create, 24).await?; + if wrong_node { + let other = NodeId::new([5; 16])?; + let disk = DiskLineage::new([15; 16])?; + config.infrastructure.log.node = other; + config.infrastructure.log.disk = disk; + config.peers[0].node = other; + config.peers[0].disk = disk; + } else { + config.peers[0].server_name = "other.example".into(); + } + let before = f.catalog.read(TIMEOUT).await?.revision(); + ensure!(node.start_raft_cells(config, services).await.is_err()); + ensure!( + f.catalog.read(TIMEOUT).await?.revision() == before, + "invalid managed owner consumed a rooted boot (wrong_node={wrong_node})" + ); + node.shutdown(ShutdownMode::Preserve).await?; + f.domain.shutdown().await; + } + Ok(()) +} diff --git a/crates/cellar/tests/native_cell_groups_public.rs b/crates/cellar/tests/native_cell_groups_public.rs new file mode 100644 index 000000000..6874f60af --- /dev/null +++ b/crates/cellar/tests/native_cell_groups_public.rs @@ -0,0 +1,160 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Public SDK/process acceptance. Other CG-07 gates remain separate requirements. +#![allow(clippy::disallowed_methods)] + +use anyhow::{ensure, Context as _, Result}; +use sha2::{Digest, Sha256}; +use std::{path::Path, process::Command, sync::Mutex}; + +#[path = "support/cell_group_managed_fixture.rs"] +pub mod managed_fixture; + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn ingress_cannot_cross_cell_authority() -> Result<()> { + Box::pin(managed_fixture::requests::run_native()).await +} + +#[path = "support/native_daemon_counter.rs"] +pub mod native_daemon_counter; + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn feature_off_fleet_contract_unchanged() -> Result<()> { + use native_daemon_counter::DurabilityMode; + for mode in [DurabilityMode::Bucket, DurabilityMode::Fleet] { + Box::pin(native_daemon_counter::run(mode)).await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn managed_group_lifecycle() -> Result<()> { + Box::pin(managed_fixture::maintenance::run_public(|_| { + Ok(std::sync::Arc::new(managed_fixture::fault::Store::default())) + })) + .await +} + +#[path = "support/cell_group_rustfs.rs"] +pub mod rustfs; + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[ignore = "requires explicit owned RustFS endpoint, bucket and isolated prefix"] +async fn rustfs_managed_group_lifecycle() -> Result<()> { + Box::pin(managed_fixture::maintenance::run_public(|rooted| { + Ok(std::sync::Arc::new(managed_fixture::fault::Store::around( + rustfs::store("public-managed-lifecycle", rooted)?, + ))) + })) + .await +} + +#[test] +fn harness_rejects_stale_binary() -> Result<()> { + let root = Path::new(env!("CARGO_MANIFEST_DIR")).join("../.."); + let output = Command::new("python3") + .args(["-m", "unittest", "test_raft_run.Freshness"]) + .current_dir(root.join("tools/perf")) + .output()?; + ensure!( + output.status.success(), + "image refusal cases failed: {}", + String::from_utf8_lossy(&output.stderr) + ); + Ok(()) +} + +fn digest(path: &Path) -> Result> { + Ok(Sha256::digest(std::fs::read(path)?).to_vec()) +} + +// Both public cases use the same six development listeners. +static COUNTER_PORTS: Mutex<()> = Mutex::new(()); + +#[test] +fn sdk_many_cells_share_group() -> Result<()> { + counter_process(false) +} + +#[test] +fn occupancy_one_uses_shared_path() -> Result<()> { + counter_process(true) +} + +fn counter_process(single_cell: bool) -> Result<()> { + let _ports = COUNTER_PORTS + .lock() + .map_err(|_| anyhow::anyhow!("counter port lock poisoned"))?; + let expected_cells = if single_cell { 1 } else { 2 }; + let binary = Path::new(env!("CARGO_BIN_EXE_cellar-raft-counter")); + let expected = digest(binary)?; + let repository = Path::new(env!("CARGO_MANIFEST_DIR")).join("../.."); + let scratch = Path::new(env!("CARGO_TARGET_TMPDIR")); + std::fs::create_dir_all(scratch)?; + // Retain success and failure disks/logs for process and receipt inspection. + let root = tempfile::Builder::new() + .prefix("cell-group-public-sdk-") + .tempdir_in(scratch)? + .keep(); + + let data = root.join("replicas"); + let mut command = Command::new("python3"); + command + .arg(repository.join("examples/raft-counter/smoke.py")) + .arg(binary) + .arg(&data) + .current_dir(&repository); + if single_cell { + command.arg("--single-cell"); + } + let output = command + .output() + .context("start original public native/Axum process drill")?; + std::fs::write(root.join("stdout.log"), &output.stdout)?; + std::fs::write(root.join("stderr.log"), &output.stderr)?; + ensure!( + output.status.success(), + "public SDK process drill failed; retained evidence at {}", + root.display() + ); + ensure!(digest(binary)? == expected, "public SDK executable changed"); + let inventory: serde_json::Value = + serde_json::from_slice(&std::fs::read(data.join("inventory.json"))?)?; + let replicas = inventory.as_array().context("replica inventory array")?; + ensure!(replicas.len() == 3, "incomplete replica inventory"); + for (i, replica) in replicas.iter().enumerate() { + ensure!(replica["groups"] == 1, "expected one observed shared group"); + ensure!( + replica["cells"] == expected_cells, + "unexpected number of logical cell histories" + ); + ensure!( + replica["observed_cells"] == expected_cells, + "owner occupancy disagrees with resources" + ); + ensure!( + replica["group"] == replicas[0]["group"] + && replica["group_incarnation"] == replicas[0]["group_incarnation"], + "replicas observe different physical histories" + ); + ensure!(replica["member"] == i as u64 + 1, "wrong physical member"); + let progress = ["checkpoint", "applied", "committed", "persisted"] + .map(|key| { + replica[key] + .as_u64() + .with_context(|| format!("missing physical {key}")) + }) + .into_iter() + .collect::>>()?; + ensure!( + progress.windows(2).all(|pair| pair[0] <= pair[1]), + "incoherent physical progress" + ); + ensure!( + replica["retained_entries"].as_u64() == Some(progress[3] - progress[0]), + "retained entries differ from the durable log" + ); + } + eprintln!("Public SDK/process evidence retained at {}", root.display()); + Ok(()) +} diff --git a/crates/cellar/tests/native_cell_groups_reads.rs b/crates/cellar/tests/native_cell_groups_reads.rs new file mode 100644 index 000000000..973c57f2a --- /dev/null +++ b/crates/cellar/tests/native_cell_groups_reads.rs @@ -0,0 +1,1461 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! SQL consistency and authority at the public shared-group SDK boundary. +#![allow(clippy::disallowed_methods)] +#[path = "support/raft_shared_read_cluster.rs"] +mod cluster; +#[path = "support/cell_group_disaster_public_node.rs"] +mod public_node; +#[path = "support/raft_read_gate.rs"] +mod read_gate; + +use anyhow::{Context as _, Result}; +use cellar::application::consensus::verification::cell_groups::{ + CellRoutingError, CellRoutingFailure, +}; +use cellar::application::consensus::verification::TransportError; +use cellar::sdk::{consensus::*, Application, Output, Registry}; +use std::{ + sync::{ + atomic::{AtomicU64, Ordering}, + Arc, + }, + time::{Duration, SystemTime, UNIX_EPOCH}, +}; + +const TIMEOUT: Duration = Duration::from_secs(5); +type Query = dyn for<'a> Fn(&QueryContext<'a>, &[u8]) -> Result> + Send + Sync; + +#[derive(Clone)] +struct Definition { + schema: [u8; 32], + serves_queries: bool, + increment: OperationContract, + get: OperationContract, + application: Arc, +} + +impl Definition { + fn new(query: Option>) -> Result { + let class = RaftClass::new("Counter", 1, [21; 32])? + .initializer([24; 32], |_, sql| { + sql.execute_batch( + "CREATE TABLE counter(value INTEGER); INSERT INTO counter VALUES(0)", + )?; + Ok(()) + })? + .command("Increment", [22; 32], |context, input| { + let delta = i64::from_be_bytes(input.try_into()?); + context + .sql() + .execute("UPDATE counter SET value=value+?1", [delta])?; + let value: i64 = + context + .sql() + .query_row("SELECT value FROM counter", [], |row| row.get(0))?; + Ok(value.to_be_bytes().to_vec()) + })?; + let serves_queries = query.is_some(); + let class = if let Some(query) = query { + class.query("Get", [23; 32], move |context, input| query(context, input))? + } else { + class + }; + Ok(Self { + schema: class.schema_digest(), + serves_queries, + increment: class.contract("Increment").context("increment contract")?, + get: OperationContract { + operation: "Get".into(), + kind: OperationKind::Query, + digest: [23; 32], + }, + application: Arc::new( + Registry::new(|_, _| Box::pin(async { Ok(Output::Done) })).raft_class(class)?, + ), + }) + } +} + +fn get(context: &QueryContext<'_>, _: &[u8]) -> Result> { + let value: i64 = context.query_row("SELECT value FROM counter", [], |row| row.get(0))?; + Ok(value.to_be_bytes().to_vec()) +} + +fn query_id() -> ProposalId { + static NEXT: AtomicU64 = AtomicU64::new(1); + let sequence = NEXT.fetch_add(1, Ordering::Relaxed); + ProposalId::new(u128::from(sequence).to_be_bytes()).expect("nonzero query ID") +} + +fn read_deadline(error: &CellDispatchError) -> bool { + error.refusal() == Some(&CommandFailure::VersionUnavailable) + || matches!( + error, + CellDispatchError::Runtime(RuntimeError::DeadlineExceeded) + | CellDispatchError::Transport(TransportError::Timeout) + | CellDispatchError::Routing(CellRoutingError { + reason: CellRoutingFailure::Deadline, + .. + }) + ) +} + +fn assert_unavailable(result: Result>) { + let error = result.expect_err("read must refuse without its required consistency proof"); + assert!(read_deadline(&error), "unexpected read failure: {error:?}"); +} + +fn at_least(receipt: &CommandReceipt) -> QueryOptions { + QueryOptions { + consistency: QueryConsistency::AtLeast(receipt.version_token.clone()), + routing: QueryRouting::Local, + ..Default::default() + } +} + +async fn increment( + cell: &RaftCellHandle, + definition: &Definition, + sequence: u64, + delta: i64, +) -> Result { + let now = u64::try_from(SystemTime::now().duration_since(UNIX_EPOCH)?.as_millis())?; + let options = CommandOptions::new( + CommandId::new(ClientId::new([7; 16])?, sequence)?, + now + 60_000, + ); + Ok(cell + .command(&definition.increment, delta.to_be_bytes().to_vec(), options) + .await?) +} + +struct Fixture { + cluster: cluster::Cluster, + handles: Vec, + definition: Definition, + initial: CommandReceipt, +} + +impl Fixture { + async fn start(query: Arc) -> Result { + Self::start_with_limits(query, public_node::limits(), RaftHostOptions::default()).await + } + + async fn start_with_limits( + query: Arc, + shared: CellGroupNodeLimits, + host: RaftHostOptions, + ) -> Result { + let definition = Definition::new(Some(query))?; + Self::start_definitions(std::array::from_fn(|_| definition.clone()), shared, host).await + } + + async fn start_definitions( + definitions: [Definition; 3], + shared: CellGroupNodeLimits, + host: RaftHostOptions, + ) -> Result { + let definition = definitions[0].clone(); + let cluster = cluster::Cluster::start(&definitions, shared, host).await?; + let cell = CellHistory { + cluster: cluster.history.cluster, + namespace: cluster.history.namespace, + cell: CellId::new("Counter:one")?, + incarnation: IncarnationId::new([7; 16])?, + }; + let handles = cluster + .services + .iter() + .map(|service| service.cell(cell.clone())) + .collect::, _>>()?; + let primary = cluster.leader(None).await?; + handles[primary] + .enroll( + CellEnrollmentRequest { + creation: ProposalId::new([8; 16])?, + durability: Durability::Quorum, + }, + TIMEOUT, + ) + .await?; + let enrollment = cluster.services[primary] + .group(cluster.history)? + .status() + .await? + .applied + .index; + // Every voter starts from the committed enrollment. Voters with query + // code also serve the initial SQL image through their local SDK. + for (index, handle) in handles.iter().enumerate() { + if !definitions[index].serves_queries { + cluster.services[index] + .group(cluster.history)? + .quorum_membership_after(enrollment, TIMEOUT) + .await?; + continue; + } + let ready = handle + .query( + query_id(), + &definition.get, + vec![], + QueryOptions { + routing: QueryRouting::Local, + ..Default::default() + }, + ) + .await + .with_context(|| format!("verify enrolled cell on voter {index}"))?; + assert_eq!(ready.served_by, MemberId::new(index as u64 + 1)?); + assert_eq!(ready.result, 0_i64.to_be_bytes()); + } + let primary = cluster.leader(None).await?; + let initial = increment(&handles[primary], &definition, 1, 1) + .await + .context("commit the baseline write")?; + for (index, handle) in handles.iter().enumerate() { + if !definitions[index].serves_queries { + cluster.services[index] + .group(cluster.history)? + .quorum_membership_after(initial.witness.position.index, TIMEOUT) + .await?; + continue; + } + let result = handle + .query(query_id(), &definition.get, vec![], at_least(&initial)) + .await?; + assert_eq!(result.result, 1_i64.to_be_bytes()); + } + Ok(Self { + cluster, + handles, + definition, + initial, + }) + } + + async fn write(&self, leader: usize, sequence: u64, delta: i64) -> Result { + increment(&self.handles[leader], &self.definition, sequence, delta).await + } + + fn group(&self, replica: usize) -> Result { + Ok(self.cluster.services[replica].group(self.cluster.history)?) + } + + fn sql_idle(&self, replica: usize) -> bool { + self.cluster.services[replica] + .resource_usage() + .sql + .is_some_and(|usage| usage.queued_or_running == 0) + } + + async fn join_reads(&self, replicas: &[usize], timeout: Duration) -> Result<()> { + tokio::time::timeout(timeout, async { + while replicas.iter().any(|replica| { + let usage = self.cluster.services[*replica].resource_usage(); + usage.host.controller.retained_commands != 0 + || usage.runtime.read_pins.items != 0 + || !usage.sql.is_some_and(|sql| sql.queued_or_running == 0) + }) { + tokio::time::sleep(Duration::from_millis(5)).await; + } + }) + .await + .context("join read work") + } + + async fn close(self) -> Result<()> { + self.cluster.close().await + } +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 8)] +async fn read_your_writes_follower() -> Result<()> { + let fixture = Fixture::start(Arc::new(get)).await?; + let primary = fixture.cluster.leader(None).await?; + let follower = (primary + 1) % 3; + fixture.cluster.isolate(follower).await?; + let written = fixture.write(primary, 2, 4).await?; + assert_eq!(written.result, 5_i64.to_be_bytes()); + let pending = fixture.handles[follower].query( + query_id(), + &fixture.definition.get, + vec![], + at_least(&written), + ); + tokio::pin!(pending); + assert!( + tokio::time::timeout(Duration::from_millis(100), &mut pending) + .await + .is_err() + ); + let historical = fixture.handles[follower] + .query( + query_id(), + &fixture.definition.get, + vec![], + QueryOptions { + consistency: QueryConsistency::Stale, + routing: QueryRouting::Local, + stale_policy: StaleReadPolicy::DisconnectedHistorical, + ..Default::default() + }, + ) + .await?; + assert_eq!(historical.result, 1_i64.to_be_bytes()); + fixture.cluster.heal(); + let read = pending.await?; + assert_eq!(read.result, 5_i64.to_be_bytes()); + assert!(read.version.covers(&written.version)?); + fixture.close().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 8)] +async fn remote_query_wait_cannot_block_its_ltx_catchup() -> Result<()> { + let fixture = Fixture::start(Arc::new(get)).await?; + let primary = fixture.cluster.leader(None).await?; + let source = (primary + 1) % 3; + let target = (primary + 2) % 3; + fixture.cluster.isolate_consensus(target).await?; + let newer = fixture.write(primary, 2, 4).await?; + let options = QueryOptions { + routing: QueryRouting::Replicas { + members: vec![MemberId::new(target as u64 + 1)?], + attempt_timeout: TIMEOUT, + leader_fallback: false, + }, + timeout: TIMEOUT, + ..at_least(&newer) + }; + let query = fixture.handles[source].query(query_id(), &fixture.definition.get, vec![], options); + tokio::time::timeout(Duration::from_secs(2), async { + while fixture.cluster.services[target] + .resource_usage() + .runtime + .read_pins + .items + == 0 + { + tokio::time::sleep(Duration::from_millis(5)).await; + } + }) + .await?; + fixture.cluster.heal(); + let read = query.await?; + assert_eq!(read.result, 5_i64.to_be_bytes()); + assert_eq!(read.served_by, MemberId::new(target as u64 + 1)?); + assert!(read.version.covers(&newer.version)?); + fixture.close().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 8)] +async fn lost_read_index_response_retries_without_leader_change() -> Result<()> { + let fixture = Fixture::start(Arc::new(get)).await?; + let primary = fixture.cluster.leader(None).await?; + let follower = (primary + 1) % 3; + fixture.handles[follower] + .query( + query_id(), + &fixture.definition.get, + vec![], + at_least(&fixture.initial), + ) + .await?; + let before = fixture.group(follower)?.status().await?; + fixture.cluster.block_control(primary, follower).await?; + let pending = fixture.handles[follower].query( + query_id(), + &fixture.definition.get, + vec![], + QueryOptions { + consistency: QueryConsistency::Linearizable, + routing: QueryRouting::Local, + timeout: Duration::from_secs(2), + ..Default::default() + }, + ); + let membership = fixture + .group(follower)? + .quorum_membership(Duration::from_secs(2)); + // Both requests can reach the leader, and the third voter supplies its safe + // ReadIndex majority. Only the reply direction is unavailable. Heal before + // the follower's minimum election timeout, keeping the term/leader fixed. + let pending = async { tokio::join!(pending, membership) }; + tokio::pin!(pending); + assert!( + tokio::time::timeout(Duration::from_millis(200), &mut pending) + .await + .is_err() + ); + assert_eq!( + fixture.cluster.services[follower] + .resource_usage() + .runtime + .read_pins + .items, + 2, + "both ReadIndex requests must be in flight before healing" + ); + fixture.cluster.heal(); + let (result, membership) = pending.await; + let membership = membership.context("retry forwarded membership ReadIndex")?; + assert_eq!(membership.barrier().term, before.term); + assert_eq!( + membership.replica().member, + MemberId::new(follower as u64 + 1)? + ); + assert_eq!(membership.configuration().voters.len(), 3); + drop(membership); + let after = fixture.group(follower)?.status().await?; + assert_eq!(after.term, before.term); + assert_eq!(after.leader, before.leader); + fixture.close().await?; + assert_eq!( + result + .context("a lost ReadIndex response must be retried under the original invocation")? + .result, + 1i64.to_be_bytes() + ); + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 8)] +async fn linearizable_read_barrier() -> Result<()> { + let fixture = Fixture::start(Arc::new(get)).await?; + let old = fixture.cluster.leader(None).await?; + let written = fixture.write(old, 2, 4).await?; + let options = QueryOptions { + consistency: QueryConsistency::Linearizable, + ..Default::default() + }; + for (i, handle) in fixture.handles.iter().enumerate() { + let read = handle + .query(query_id(), &fixture.definition.get, vec![], options.clone()) + .await + .with_context(|| format!("initial linearizable read on replica {i}"))?; + assert_eq!(read.result, 5_i64.to_be_bytes()); + assert!(read.version.covers(&written.version)?); + } + fixture.cluster.isolate(old).await?; + let next = fixture.cluster.leader(Some(old)).await?; + // The new term commits a physical barrier without mutating this cell. + let read = fixture.handles[next] + .query(query_id(), &fixture.definition.get, vec![], options.clone()) + .await + .context("new leader protocol-only cut")?; + assert_eq!(read.result, 5_i64.to_be_bytes()); + assert_eq!(read.version, written.version); + assert_eq!(read.witness, written.witness); + let barrier = fixture.group(next)?.status().await?.applied; + assert!(barrier.index > written.witness.position.index); + assert!(barrier.term > written.witness.position.term); + let newer = fixture.write(next, 3, 7).await?; + assert_eq!(newer.result, 12_i64.to_be_bytes()); + for (i, handle) in fixture + .handles + .iter() + .enumerate() + .filter(|(i, _)| *i != old) + { + let read = handle + .query(query_id(), &fixture.definition.get, vec![], options.clone()) + .await + .with_context(|| format!("post-election linearizable read on replica {i}"))?; + assert_eq!(read.result, 12_i64.to_be_bytes(), "replica {i}"); + assert!(read.version.covers(&newer.version)?); + } + fixture.cluster.heal(); + let former_leader = fixture.group(old)?; + tokio::time::timeout(TIMEOUT, async { + loop { + let status = former_leader.status().await?; + if status.term == barrier.term + && status.leader == Some(next as u64 + 1) + && status.applied.index >= newer.witness.position.index + { + return Ok::<_, RuntimeError>(()); + } + tokio::time::sleep(Duration::from_millis(5)).await; + } + }) + .await + .context("former leader did not catch up after healing")??; + let read = fixture.handles[old] + .query(query_id(), &fixture.definition.get, vec![], options) + .await + .context("healed former leader linearizable read")?; + assert_eq!(read.result, 12_i64.to_be_bytes()); + fixture.close().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 8)] +async fn minority_strong_read_refused() -> Result<()> { + let fixture = Fixture::start(Arc::new(get)).await?; + let primary = fixture.cluster.leader(None).await?; + let follower = (primary + 1) % 3; + fixture.cluster.isolate(follower).await?; + let newer = fixture.write(primary, 2, 4).await?; + let strong = QueryOptions { + consistency: QueryConsistency::Linearizable, + timeout: Duration::from_millis(250), + ..Default::default() + }; + assert_unavailable( + fixture.handles[follower] + .query(query_id(), &fixture.definition.get, vec![], strong) + .await, + ); + let mut minimum = at_least(&newer); + minimum.timeout = Duration::from_millis(250); + assert_unavailable( + fixture.handles[follower] + .query(query_id(), &fixture.definition.get, vec![], minimum) + .await, + ); + // A minimum-version read can legitimately use an already-covered prefix + // without quorum; it must not be confused with a fresh linearizable read. + assert_eq!( + fixture.handles[follower] + .query( + query_id(), + &fixture.definition.get, + vec![], + at_least(&fixture.initial) + ) + .await? + .result, + 1_i64.to_be_bytes() + ); + assert_unavailable( + fixture.handles[follower] + .query( + query_id(), + &fixture.definition.get, + vec![], + QueryOptions::default(), + ) + .await, + ); + assert_eq!( + fixture.handles[follower] + .query( + query_id(), + &fixture.definition.get, + vec![], + QueryOptions { + consistency: QueryConsistency::Stale, + routing: QueryRouting::Local, + stale_policy: StaleReadPolicy::DisconnectedHistorical, + ..Default::default() + } + ) + .await? + .result, + 1_i64.to_be_bytes() + ); + fixture.cluster.heal(); + assert_eq!( + fixture.handles[follower] + .query( + query_id(), + &fixture.definition.get, + vec![], + at_least(&newer) + ) + .await? + .result, + 5_i64.to_be_bytes() + ); + fixture.close().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 8)] +async fn native_readiness_distinguishes_historical_and_recent_source() -> Result<()> { + let fixture = Fixture::start(Arc::new(get)).await?; + let result = async { + let primary = fixture.cluster.leader(None).await?; + let follower = (primary + 1) % fixture.handles.len(); + let handle = &fixture.handles[follower]; + let member = MemberId::new(follower as u64 + 1)?; + let recent = QueryOptions { + consistency: QueryConsistency::Stale, + routing: QueryRouting::Local, + ..Default::default() + }; + let before = handle + .query(query_id(), &fixture.definition.get, vec![], recent.clone()) + .await?; + assert_eq!(before.served_by, member); + + fixture.cluster.isolate_consensus(follower).await?; + tokio::time::sleep(cluster::STALE_SOURCE_TIMEOUT).await; + let historical = handle + .query( + query_id(), + &fixture.definition.get, + vec![], + QueryOptions { + stale_policy: StaleReadPolicy::DisconnectedHistorical, + ..recent.clone() + }, + ) + .await?; + assert_eq!(historical.served_by, member); + assert_eq!(historical.result, 1_i64.to_be_bytes()); + assert_eq!(historical.witness, fixture.initial.witness); + assert_eq!(historical.version, fixture.initial.version); + for consistency in [QueryConsistency::Stale, QueryConsistency::Linearizable] { + assert_unavailable( + handle + .query( + query_id(), + &fixture.definition.get, + vec![], + QueryOptions { + consistency, + timeout: Duration::from_millis(250), + ..recent.clone() + }, + ) + .await, + ); + } + fixture.cluster.heal(); + for consistency in [QueryConsistency::Stale, QueryConsistency::Linearizable] { + let read = handle + .query( + query_id(), + &fixture.definition.get, + vec![], + QueryOptions { + consistency, + ..recent.clone() + }, + ) + .await?; + assert_eq!(read.served_by, member); + assert_eq!(read.result, historical.result); + assert_eq!(read.witness, historical.witness); + assert_eq!(read.version, historical.version); + } + Ok::<_, anyhow::Error>(()) + } + .await; + let joined = fixture.close().await; + result?; + joined +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 8)] +async fn native_readiness_survives_protocol_only_leadership_transfer() -> Result<()> { + use cellar::application::consensus::RaftReplicaRole; + + let fixture = Fixture::start(Arc::new(get)).await?; + let result = async { + let primary = fixture.cluster.leader(None).await?; + let target = (primary + 1) % fixture.handles.len(); + let member = MemberId::new(target as u64 + 1)?; + let leader = fixture.group(primary)?; + let before = leader.observe(TIMEOUT).await?; + fixture + .group(target)? + .quorum_membership_after(before.status.applied.index, TIMEOUT) + .await?; + leader + .transfer_leadership( + LeadershipTransfer { + target: member, + expected_term: before.status.term, + expected_revision: before.membership_revision, + }, + TIMEOUT, + ) + .await?; + + // The transfer changes the physical log without modifying the cell. + for (replica, handle) in fixture.handles.iter().enumerate() { + let role = if replica == target { + RaftReplicaRole::Leader + } else { + RaftReplicaRole::Voter + }; + let group = fixture.group(replica)?; + let observation = tokio::time::timeout(TIMEOUT, async { + loop { + let sample = group.observe(TIMEOUT).await?; + if sample.role == role + && sample.status.leader == Some(member.get()) + && sample.status.term > before.status.term + && sample.status.applied.index > before.status.applied.index + && sample.status.ready == (replica == target) + { + return Ok::<_, RuntimeError>(sample); + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .with_context(|| format!("voter {replica} did not apply the new term"))??; + assert_eq!(observation.cells, 1); + for (consistency, stale_policy) in [ + ( + QueryConsistency::Linearizable, + StaleReadPolicy::RecentSource, + ), + (QueryConsistency::Stale, StaleReadPolicy::RecentSource), + ( + QueryConsistency::Stale, + StaleReadPolicy::DisconnectedHistorical, + ), + ] { + let read = handle + .query( + query_id(), + &fixture.definition.get, + vec![], + QueryOptions { + consistency, + stale_policy, + routing: QueryRouting::Local, + ..Default::default() + }, + ) + .await?; + assert_eq!(read.served_by, MemberId::new(replica as u64 + 1)?); + assert_eq!(read.result, 1_i64.to_be_bytes()); + assert_eq!(read.witness, fixture.initial.witness); + assert_eq!(read.version, fixture.initial.version); + } + } + Ok::<_, anyhow::Error>(()) + } + .await; + let joined = fixture.close().await; + result?; + joined +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 8)] +async fn query_write_capability_denied() -> Result<()> { + let query = Arc::new(|context: &QueryContext<'_>, input: &[u8]| { + if input == b"attack" { + for sql in [ + "UPDATE counter SET value=999 RETURNING value", + "CREATE TABLE injected(value INTEGER)", + "PRAGMA journal_mode=WAL", + "ATTACH ':memory:' AS injected", + "DELETE FROM counter RETURNING value", + ] { + let error = context + .query_row(sql, [], |row| row.get::<_, i64>(0)) + .unwrap_err(); + let sqlite = error + .downcast_ref::() + .ok_or_else(|| anyhow::anyhow!("missing SQLite refusal: {error}"))?; + assert!( + matches!(sqlite, rusqlite::Error::SqliteFailure(error, _) if error.code == rusqlite::ErrorCode::AuthorizationForStatementDenied) + ); + } + } + get(context, input) + }); + let fixture = Fixture::start(query).await?; + for handle in &fixture.handles { + let result = handle + .query( + query_id(), + &fixture.definition.get, + b"attack".to_vec(), + at_least(&fixture.initial), + ) + .await?; + assert_eq!(result.result, 1_i64.to_be_bytes()); + } + let primary = fixture.cluster.leader(None).await?; + assert_eq!( + fixture.write(primary, 2, 4).await?.result, + 5_i64.to_be_bytes() + ); + fixture.close().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 8)] +async fn unpolled_query_deadline_interrupts_sql_and_joins_work() -> Result<()> { + let (entered, mut receive) = tokio::sync::mpsc::unbounded_channel(); + let query = Arc::new(move |context: &QueryContext<'_>, input: &[u8]| { + if input == b"slow" { + entered.send(())?; + let _: i64 = context.query_row("WITH RECURSIVE n(x) AS (VALUES(1) UNION ALL SELECT x+1 FROM n WHERE x<1000000000) SELECT sum(x) FROM n", [], |row| row.get(0))?; + } + get(context, input) + }); + let fixture = Fixture::start(query).await?; + let primary = fixture.cluster.leader(None).await?; + let pending = fixture.handles[primary].query( + query_id(), + &fixture.definition.get, + b"slow".to_vec(), + QueryOptions { + timeout: Duration::from_millis(150), + ..at_least(&fixture.initial) + }, + ); + tokio::time::timeout(Duration::from_secs(5), receive.recv()) + .await? + .ok_or_else(|| anyhow::anyhow!("query missing"))?; + tokio::time::timeout(Duration::from_secs(3), async { + while !fixture.cluster.services[primary] + .resource_usage() + .sql + .is_some_and(|usage| usage.queued_or_running == 0) + { + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await?; + assert_unavailable(pending.await); + assert_eq!( + fixture.handles[primary] + .query( + query_id(), + &fixture.definition.get, + vec![], + at_least(&fixture.initial) + ) + .await? + .result, + 1_i64.to_be_bytes() + ); + fixture.close().await +} + +fn routed(receipt: &CommandReceipt, member: MemberId, fallback: bool) -> QueryOptions { + QueryOptions { + routing: QueryRouting::Replicas { + members: vec![member], + attempt_timeout: Duration::from_millis(180), + leader_fallback: fallback, + }, + timeout: Duration::from_secs(2), + ..at_least(receipt) + } +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 8)] +async fn fallback_preserves_contract() -> Result<()> { + let fixture = Fixture::start(Arc::new(get)).await?; + let primary = fixture.cluster.leader(None).await?; + let source = (primary + 1) % 3; + let target = (primary + 2) % 3; + let target_member = MemberId::new(target as u64 + 1)?; + let primary_member = MemberId::new(primary as u64 + 1)?; + fixture.cluster.isolate_consensus(target).await?; + let newer = fixture.write(primary, 2, 4).await?; + assert_eq!(newer.result, 5_i64.to_be_bytes()); + + assert_unavailable( + fixture.handles[source] + .query( + query_id(), + &fixture.definition.get, + vec![], + routed(&newer, target_member, false), + ) + .await, + ); + for consistency in [ + QueryConsistency::AtLeast(newer.version_token.clone()), + QueryConsistency::Linearizable, + ] { + let read = fixture.handles[source] + .query( + query_id(), + &fixture.definition.get, + vec![], + QueryOptions { + consistency, + ..routed(&newer, target_member, true) + }, + ) + .await?; + assert_eq!(read.served_by, primary_member); + assert_eq!(read.result, 5_i64.to_be_bytes()); + assert!(read.version.covers(&newer.version)?); + } + + let timeout = Duration::from_millis(100); + let short = |attempt_timeout| QueryOptions { + timeout, + routing: QueryRouting::Replicas { + members: vec![target_member], + attempt_timeout, + leader_fallback: true, + }, + ..at_least(&newer) + }; + let error = fixture.handles[source] + .query( + query_id(), + &fixture.definition.get, + vec![], + short(Duration::from_secs(1)), + ) + .await + .expect_err("attempt longer than the invocation must refuse admission"); + assert!(matches!( + error.as_ref(), + CellDispatchError::Invalid("query replica routing bounds") + )); + assert_unavailable( + tokio::time::timeout( + Duration::from_millis(500), + fixture.handles[source].query( + query_id(), + &fixture.definition.get, + vec![], + short(timeout), + ), + ) + .await?, + ); + fixture + .join_reads(&[source, target], Duration::from_millis(500)) + .await?; + fixture.cluster.heal(); + fixture.close().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 8)] +async fn delayed_tcp_query_reply_cannot_settle_a_later_read() -> Result<()> { + let (gate, mut entered) = read_gate::Gate::new(); + let release = read_gate::Release(gate.clone()); + let fixture = Fixture::start(Arc::new(move |context, input| { + if input == b"hold" { + gate.wait()?; + } + get(context, input) + })) + .await?; + let primary = fixture.cluster.leader(None).await?; + let source = (primary + 1) % 3; + let target = (primary + 2) % 3; + let target_member = MemberId::new(target as u64 + 1)?; + let timeout = Duration::from_secs(1); + let options = QueryOptions { + timeout, + routing: QueryRouting::Replicas { + members: vec![target_member], + attempt_timeout: timeout, + leader_fallback: false, + }, + ..at_least(&fixture.initial) + }; + let warm = fixture.handles[source] + .query(query_id(), &fixture.definition.get, vec![], options.clone()) + .await?; + assert_eq!(warm.served_by, target_member); + + let delayed = fixture.handles[source].query( + query_id(), + &fixture.definition.get, + b"hold".to_vec(), + options.clone(), + ); + tokio::time::timeout(Duration::from_secs(2), entered.recv()) + .await? + .context("remote query did not enter")?; + let held = fixture.cluster.hold_query_responses(source, target)?; + release.0.release(); + tokio::time::timeout(Duration::from_millis(750), async { + while fixture.cluster.held_query_response_bytes(source, target)? == 0 + || !fixture.sql_idle(target) + { + tokio::time::sleep(Duration::from_millis(5)).await; + } + Ok::<_, anyhow::Error>(()) + }) + .await??; + let newer = fixture.write(primary, 2, 4).await?; + assert_eq!(newer.result, 5_i64.to_be_bytes()); + assert_unavailable(tokio::time::timeout(Duration::from_secs(2), delayed).await?); + + drop(held); + let read = fixture.handles[source] + .query( + query_id(), + &fixture.definition.get, + vec![], + QueryOptions { + consistency: QueryConsistency::AtLeast(newer.version_token.clone()), + ..options + }, + ) + .await?; + assert_eq!(read.result, 5_i64.to_be_bytes()); + assert_eq!(read.served_by, target_member); + assert!(read.version.covers(&newer.version)?); + fixture + .join_reads(&[0, 1, 2], Duration::from_secs(2)) + .await?; + fixture.close().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 8)] +async fn follower_drains_committed_batch_after_held_reader() -> Result<()> { + let (gate, mut entered) = read_gate::Gate::new(); + let release = read_gate::Release(gate.clone()); + let fixture = Fixture::start(Arc::new(move |context, input| { + if input == b"hold" { + gate.wait()?; + } + get(context, input) + })) + .await?; + let result = async { + let primary = fixture.cluster.leader(None).await?; + let follower = (primary + 1) % 3; + let pending = fixture.handles[follower].query( + query_id(), + &fixture.definition.get, + b"hold".to_vec(), + QueryOptions { + timeout: Duration::from_secs(30), + ..at_least(&fixture.initial) + }, + ); + tokio::time::timeout(TIMEOUT, entered.recv()) + .await? + .context("follower reader did not enter")?; + let mut newest = fixture.initial.clone(); + // The pinned follower read must coexist with quorum writes through the + // other voters. Its original SQLite image must remain coherent. + for sequence in 2..=33 { + newest = fixture + .write(primary, sequence, 1) + .await + .with_context(|| format!("write backlog sequence {sequence}"))?; + assert_eq!(newest.result, (sequence as i64).to_be_bytes()); + } + release.0.release(); + assert_eq!(pending.await?.result, 1_i64.to_be_bytes()); + for (index, handle) in fixture.handles.iter().enumerate() { + let result = handle + .query( + query_id(), + &fixture.definition.get, + vec![], + QueryOptions { + timeout: Duration::from_secs(10), + ..at_least(&newest) + }, + ) + .await; + if let Err(error) = &result { + let mut statuses = Vec::new(); + for replica in 0..fixture.handles.len() { + statuses.push(fixture.group(replica)?.status().await); + } + anyhow::bail!( + "replica {index} did not drain writes: {error:?}; groups={statuses:?}" + ); + } + let read = result?; + assert_eq!(read.result, 33_i64.to_be_bytes()); + assert!(read.version.covers(&newest.version)?); + } + let final_write = fixture.write(primary, 34, 1).await?; + let read = fixture.handles[follower] + .query( + query_id(), + &fixture.definition.get, + vec![], + at_least(&final_write), + ) + .await?; + assert_eq!(read.result, 34_i64.to_be_bytes()); + Ok::<_, anyhow::Error>(()) + } + .await; + release.0.release(); + let cleanup = fixture.close().await; + result.and(cleanup) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 8)] +async fn stale_is_committed_prefix() -> Result<()> { + let (gate, mut entered) = read_gate::Gate::new(); + let release = read_gate::Release(gate.clone()); + let fixture = Fixture::start(Arc::new(move |context, input| { + let mut value = get(context, input)?; + if input == b"hold" { + let QueryVersion::Cell(observed) = context.version() else { + anyhow::bail!("missing logical cell observation"); + }; + let index = observed.witness.position.index.get(); + gate.wait()?; + assert_eq!(get(context, input)?, value); + value.extend_from_slice(&index.to_be_bytes()); + } + Ok(value) + })) + .await?; + let primary = fixture.cluster.leader(None).await?; + let follower = (primary + 1) % 3; + let pending = fixture.handles[follower].query( + query_id(), + &fixture.definition.get, + b"hold".to_vec(), + QueryOptions { + consistency: QueryConsistency::Stale, + routing: QueryRouting::Local, + stale_policy: StaleReadPolicy::DisconnectedHistorical, + ..Default::default() + }, + ); + tokio::time::timeout(TIMEOUT, entered.recv()) + .await? + .context("query did not pin")?; + let newer = fixture.write(primary, 2, 4).await?; + assert_eq!(newer.result, 5_i64.to_be_bytes()); + release.0.release(); + + let read = pending.await?; + assert_eq!(&read.result[..8], &1_i64.to_be_bytes()); + assert_eq!( + u64::from_be_bytes(read.result[8..].try_into()?), + read.witness.position.index.get() + ); + assert_eq!(read.witness, fixture.initial.witness); + assert_eq!(read.version, fixture.initial.version); + assert!(read.witness.position.index < newer.witness.position.index); + let latest = fixture.handles[follower] + .query( + query_id(), + &fixture.definition.get, + vec![], + at_least(&newer), + ) + .await?; + assert_eq!(latest.result, 5_i64.to_be_bytes()); + assert!(latest.version.covers(&newer.version)?); + fixture.close().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 8)] +async fn completed_unpolled_responses_keep_their_byte_budget() -> Result<()> { + const PAYLOAD: usize = 60_000; + const REPLIES: usize = 16; + let mut shared = public_node::limits(); + shared.controller.max_commands = 32; + shared.controller.max_command_bytes = 1024 * 1024; + shared.dispatch.max_invocations = 32; + shared.dispatch.max_input_bytes = 4096; + shared.sql.max_pending_cells = 32; + shared.sql.group_pending_cells = 32; + shared.sql.group_bytes = 1024 * 1024; + // A small command envelope fits the same per-group SQL byte budget. + let mut host = RaftHostOptions::default(); + host.execution.envelope.max_payload_bytes = 128 * 1024; + host.archive.limits.envelope = host.execution.envelope; + let fixture = Fixture::start_with_limits( + Arc::new(|context, input| { + if input == b"large" { + Ok(vec![17; PAYLOAD]) + } else { + get(context, input) + } + }), + shared, + host, + ) + .await?; + let primary = fixture.cluster.leader(None).await?; + let query = || { + fixture.handles[primary].query( + query_id(), + &fixture.definition.get, + b"large".to_vec(), + at_least(&fixture.initial), + ) + }; + let mut responses = Vec::new(); + for count in 1..=REPLIES { + responses.push(query()); + tokio::time::timeout(TIMEOUT, async { + loop { + let usage = fixture.cluster.services[primary].resource_usage(); + if usage.sql.is_some_and(|sql| { + sql.queued_or_running == 0 && sql.retained_bytes == count * (PAYLOAD + 4096) + }) { + break; + } + tokio::time::sleep(Duration::from_millis(5)).await; + } + }) + .await + .with_context(|| format!("complete and retain unpolled response {count}"))?; + } + let error = query() + .await + .expect_err("held replies exhaust the SQL byte budget"); + assert_eq!(error.refusal(), Some(&CommandFailure::Overloaded)); + + assert_eq!(responses.remove(0).await?.result, vec![17; PAYLOAD]); + assert_eq!(query().await?.result, vec![17; PAYLOAD]); + drop(responses); + fixture.join_reads(&[primary], TIMEOUT).await?; + assert_eq!( + fixture.cluster.services[primary] + .resource_usage() + .sql + .context("SQL budget observation")? + .retained_bytes, + 0 + ); + fixture.close().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 8)] +async fn saturated_query_peer_refuses_and_expires_queued_work() -> Result<()> { + use std::sync::atomic::AtomicUsize; + let (gate, mut entered) = read_gate::Gate::new(); + let release = read_gate::Release(gate.clone()); + let unexpected = Arc::new(AtomicUsize::new(0)); + let executed = unexpected.clone(); + let mut shared = public_node::limits(); + shared.controller.max_commands = 96; + shared.controller.max_command_bytes = 4 * 1024 * 1024; + shared.dispatch.max_invocations = 96; + shared.dispatch.retained_bytes = 4 * 1024 * 1024; + let fixture = Fixture::start_with_limits( + Arc::new(move |context, input| { + if input == b"queued" { + executed.fetch_add(1, Ordering::SeqCst); + } + if input == b"hold" { + gate.wait()?; + } + get(context, input) + }), + shared, + RaftHostOptions::default(), + ) + .await?; + let primary = fixture.cluster.leader(None).await?; + let source = (primary + 1) % 3; + let target = (primary + 2) % 3; + let target_member = MemberId::new(target as u64 + 1)?; + let options = |timeout| QueryOptions { + timeout, + routing: QueryRouting::Replicas { + members: vec![target_member], + attempt_timeout: timeout, + leader_fallback: false, + }, + ..at_least(&fixture.initial) + }; + // Hold the real peer connection in its destination callback. Later calls + // must consume finite peer bytes while queued behind this invocation. + let first = fixture.handles[source].query( + query_id(), + &fixture.definition.get, + b"hold".to_vec(), + options(Duration::from_secs(5)), + ); + tokio::time::timeout(Duration::from_secs(2), entered.recv()) + .await? + .ok_or_else(|| anyhow::anyhow!("remote callback did not enter"))?; + let source_usage = fixture.cluster.services[source] + .resource_usage() + .transport + .context("source transport observer")?; + let target_usage = fixture.cluster.services[target] + .resource_usage() + .transport + .context("target transport observer")?; + assert!(source_usage.outgoing.query.items > 0 && source_usage.outgoing.query.bytes > 0); + assert!(target_usage.incoming.query.items > 0 && target_usage.incoming.query.bytes > 0); + assert!(source_usage.outgoing_connections.query > 0); + assert!(target_usage.incoming_connections.query > 0); + assert!( + target_usage.incoming_connections.query <= target_usage.incoming_connection_limits.query + ); + assert!(source_usage.outgoing.query.bytes <= source_usage.outgoing.query.max_bytes); + assert!(target_usage.incoming.query.bytes <= target_usage.incoming.query.max_bytes); + let queued: Vec<_> = (0..64) + .map(|_| { + fixture.handles[source].query( + query_id(), + &fixture.definition.get, + b"queued".to_vec(), + options(Duration::from_secs(1)), + ) + }) + .collect(); + let mut overloaded = 0; + let mut expired = 0; + // Poll all responses while queued work waits. An already refused + // response still expires when the caller waits past its deadline. + let replies = tokio::time::timeout( + Duration::from_secs(2), + futures_util::future::join_all(queued), + ) + .await?; + for response in replies { + match response { + Err(error) + if matches!( + error.as_ref(), + CellDispatchError::Transport(TransportError::Overloaded) + ) => + { + overloaded += 1 + } + Err(error) if read_deadline(&error) => expired += 1, + result => anyhow::bail!("unexpected queued response: {result:?}"), + } + } + assert!(overloaded > 0, "peer saturation was not observed"); + assert!( + expired > 0, + "no admitted request expired behind the held RPC" + ); + assert_eq!(unexpected.load(Ordering::SeqCst), 0); + // Consensus and LTX use separate connections/capacity. They must publish a + // new command while the first query and expired peer queue are still held. + let newer = fixture.write(primary, 2, 4).await?; + assert_eq!(newer.result, 5_i64.to_be_bytes()); + release.0.release(); + let first = first.await?; + assert_eq!(first.result, 1_i64.to_be_bytes()); + assert_eq!(first.served_by, MemberId::new(target as u64 + 1)?); + assert!(first.witness.position.index < newer.witness.position.index); + let mut fresh = options(Duration::from_secs(2)); + fresh.consistency = QueryConsistency::AtLeast(newer.version_token.clone()); + let read = fixture.handles[source] + .query(query_id(), &fixture.definition.get, vec![], fresh) + .await?; + assert_eq!(read.result, 5_i64.to_be_bytes()); + assert!(read.version.covers(&newer.version)?); + assert_eq!( + unexpected.load(Ordering::SeqCst), + 0, + "expired queued request executed after the peer resumed" + ); + fixture + .join_reads(&[0, 1, 2], Duration::from_secs(2)) + .await?; + let performance: Vec<_> = fixture + .cluster + .services + .iter() + .map(RaftCells::performance) + .collect(); + assert!(performance + .iter() + .all(|sample| sample.available && !sample.saturated)); + for lane in ["control", "data", "query"] { + for direction in ["read", "write"] { + let name = format!("transport_{lane}_{direction}_bytes"); + let bytes: u64 = performance + .iter() + .map(|sample| { + sample + .counters + .iter() + .find(|counter| counter.name == name) + .unwrap() + .value + }) + .sum(); + assert!(bytes > 0, "real TLS traffic missing from {name}"); + } + } + for lane in ["control", "data"] { + for phase in ["queue", "exchange", "admit"] { + let name = format!("transport_{lane}_{phase}"); + let successes: u64 = performance + .iter() + .map(|sample| { + sample + .operations + .iter() + .find(|operation| operation.name == name) + .unwrap() + .succeeded + }) + .sum(); + assert!(successes > 0, "real TLS traffic missing from {name}"); + } + } + fixture.close().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 8)] +async fn voter_without_query_code_can_route_to_a_compatible_replica() -> Result<()> { + let queryless = Definition::new(None)?; + let registered = Definition::new(Some(Arc::new(get)))?; + assert_eq!(queryless.schema, registered.schema); + let fixture = Fixture::start_definitions( + [queryless, registered.clone(), registered], + public_node::limits(), + RaftHostOptions::default(), + ) + .await?; + let queryless = &fixture.handles[0]; + assert!(matches!( + queryless.contract("Get"), + Err(RegistrationError::UnknownOperation(_)) + )); + let local = queryless + .query( + query_id(), + &fixture.definition.get, + vec![], + at_least(&fixture.initial), + ) + .await + .expect_err("queryless voter must refuse local execution"); + assert!(matches!( + local.as_ref(), + CellDispatchError::Registration(RegistrationError::UnknownOperation(_)) + )); + + let remote = QueryOptions { + routing: QueryRouting::Replicas { + members: vec![MemberId::new(1)?, MemberId::new(2)?], + attempt_timeout: Duration::from_millis(500), + leader_fallback: false, + }, + ..at_least(&fixture.initial) + }; + let read = queryless + .query(query_id(), &fixture.definition.get, vec![], remote.clone()) + .await?; + assert_eq!(read.result, 1_i64.to_be_bytes()); + assert_eq!(read.served_by, MemberId::new(2)?); + assert!(read.version.covers(&fixture.initial.version)?); + + let wrong = OperationContract { + digest: [99; 32], + ..fixture.definition.get.clone() + }; + let refused = queryless + .query(query_id(), &wrong, vec![], remote.clone()) + .await + .expect_err("remote replica must check the exact query contract"); + assert_eq!( + refused.refusal(), + Some(&CommandFailure::UnsupportedCapability) + ); + let refused = fixture.handles[1] + .query(query_id(), &wrong, vec![], remote) + .await + .expect_err("origin must refuse a wrong digest for a known query"); + assert!(matches!( + refused.as_ref(), + CellDispatchError::Registration(RegistrationError::ContractMismatch) + )); + fixture.close().await +} diff --git a/crates/cellar/tests/native_cell_groups_snapshot.rs b/crates/cellar/tests/native_cell_groups_snapshot.rs new file mode 100644 index 000000000..5b581a73a --- /dev/null +++ b/crates/cellar/tests/native_cell_groups_snapshot.rs @@ -0,0 +1,118 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! CG-05 acceptance over the same real engines, SQLite images and TLS controllers +//! as native_cell_groups_execution. Each named case retains direct fixture oracles. +#![allow(clippy::disallowed_methods)] + +#[path = "support/cell_group_archive_store.rs"] +#[allow(dead_code)] +mod archive_store; +#[path = "support/raft_storage_fault.rs"] +#[allow(dead_code)] +mod fault; +#[path = "support/raft_ltx_fixture.rs"] +#[allow(dead_code)] +mod images; +// These shared fixture modules expose other execution scenarios to the sibling +// integration target. Only the CG-05 scenarios are selected in this target. +#[path = "support/cell_group_checkpoint.rs"] +#[allow(dead_code)] +mod checkpoint_fixture; +#[path = "support/cell_group_runtime.rs"] +#[allow(dead_code, unused_imports)] +mod runtime_fixture; + +include!("support/cell_group_engine_fixture.rs"); + +// Construct large fixture futures outside the composite test's poll frame. +// Boxing at the call site alone still leaves their construction temporaries on +// that frame in debug builds, consuming stack while a nested fixture is polled. +#[inline(never)] +fn fixture>>( + make: impl FnOnce() -> F, +) -> std::pin::Pin> { + Box::pin(make()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn snapshot_contains_every_cell_at_cut() -> Result<()> { + // Hold an actual source while two other cell mutations commit, then verify + // every original image and manifest page at one cut in both representations. + checkpoint_fixture::coherent().await?; + checkpoint_fixture::refusals().await?; + checkpoint_fixture::corruption().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn partial_snapshot_never_serves() -> Result<()> { + runtime_fixture::refuse_checkpoint().await?; + for rooted in [false, true] { + runtime_fixture::checkpoint_owner(rooted).await?; + Box::pin(runtime_fixture::seed_receiver_faults(rooted)).await?; + Box::pin(runtime_fixture::seed_transfer_revocation(rooted)).await?; + Box::pin(runtime_fixture::seed_persistence(rooted)).await?; + // These parents launch this same executable's ignored child and verify + // actual retained disks after owned SIGKILL at all six install cuts. + Box::pin(runtime_fixture::seed_crash(rooted)).await?; + fixture(|| runtime_fixture::seed_retry(rooted)).await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn whole_group_learner_and_joint_change() -> Result<()> { + for rooted in [false, true] { + // Erase one voter, install every cell into a fresh learner, require + // catch-up before promotion, reconnect and refuse isolated commits. + fixture(|| runtime_fixture::seed_source(rooted)).await?; + fixture(|| runtime_fixture::seed_install(rooted)).await?; + fixture(|| runtime_fixture::membership_lifecycle(rooted)).await?; + fixture(|| runtime_fixture::membership_crash(rooted)).await?; + } + fixture(runtime_fixture::membership_sync).await?; + fixture(runtime_fixture::seed_minority).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn snapshot_empty_cell_and_extent_restart() -> Result<()> { + for rooted in [false, true] { + // Raft generates the snapshot over TLS after prefix compaction. The + // selected/retained closures include empty enrollment and later writes, + // and cold reopen verifies the exact checkpoint plus surviving suffix. + fixture(|| runtime_fixture::tcp_publication(rooted)).await?; + fixture(|| runtime_fixture::install_checkpoint(rooted)).await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn snapshot_memory_and_pin_bounds() -> Result<()> { + for rooted in [false, true] { + runtime_fixture::transfer_pressure(rooted).await?; + fixture(|| runtime_fixture::shared_inventory(rooted)).await?; + } + checkpoint_fixture::ownership().await?; + runtime_fixture::local_shutdown().await?; + runtime_fixture::checkpoint_shutdown().await?; + fixture(|| runtime_fixture::checkpoint_during_apply(false)).await?; + fixture(|| runtime_fixture::checkpoint_during_apply(true)).await?; + fixture(|| runtime_fixture::checkpoint_completion_pressure(false)).await?; + fixture(|| runtime_fixture::checkpoint_completion_pressure(true)).await?; + fixture(|| runtime_fixture::concurrent_checkpoints(false)).await?; + fixture(|| runtime_fixture::concurrent_checkpoints(true)).await?; + fixture(|| runtime_fixture::controller_checkpoint_shutdown(false)).await?; + fixture(|| runtime_fixture::controller_checkpoint_shutdown(true)).await?; + runtime_fixture::automatic_checkpoints(false, true).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "owned child process controlled by partial_snapshot_never_serves"] +async fn seed_process_fixture() -> Result<()> { + Box::pin(runtime_fixture::seed_process_fixture()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "owned child process controlled by whole_group_learner_and_joint_change"] +async fn membership_process_fixture() -> Result<()> { + Box::pin(runtime_fixture::membership_process_fixture()).await +} diff --git a/crates/cellar/tests/native_cell_groups_storage.rs b/crates/cellar/tests/native_cell_groups_storage.rs new file mode 100644 index 000000000..48cc9b178 --- /dev/null +++ b/crates/cellar/tests/native_cell_groups_storage.rs @@ -0,0 +1,1541 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Manifest and engine publication contracts on independently sealed SQLite images. +//! Full crash-cut and both-image-mode qualification remain separate gates. +#![allow(clippy::disallowed_methods)] + +#[path = "support/raft_ltx_fixture.rs"] +#[allow(dead_code)] +mod images; + +#[path = "support/raft_ltx_fault.rs"] +#[allow(dead_code)] +mod faults; + +#[path = "support/raft_storage_fault.rs"] +#[allow(dead_code)] +mod engine_fault; + +#[path = "support/cell_group_batch_crash.rs"] +mod batch_crash; +#[path = "support/cell_group_crash_fixture.rs"] +#[allow(dead_code)] +mod crash_fixture; +#[path = "support/cell_group_recovery.rs"] +mod recovery; + +use anyhow::Result; +use cellar::application::consensus::{ + verification::{cell_groups::*, Catalog, CatalogOptions, CommittedImage}, + *, +}; +use cellar::sdk::ExecutionDomain; +use cellar_ltx::{ + generation::{GenerationLimits, GenerationPin, GenerationStore}, + DirectFileSystem, +}; +use std::{collections::BTreeMap, sync::Arc}; + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn both_image_representations_share_contract() -> Result<()> { + recovery::both_images().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn missing_cell_image_refuses_group() -> Result<()> { + recovery::missing_images().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn manifest_marker_crash_cuts() -> Result<()> { + recovery::crash_matrix().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[ignore = "process-kill fixture, launched only by manifest_marker_crash_cuts"] +async fn manifest_process_fixture() -> Result<()> { + recovery::process_fixture().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn cross_group_batch_crash() -> Result<()> { + batch_crash::run().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[ignore = "owned process-kill fixture for cross_group_batch_crash"] +async fn batch_crash_child() -> Result<()> { + batch_crash::child().await +} + +fn route() -> GroupRoute { + GroupRoute { + history: GroupHistory { + cluster: ClusterId::new([1; 16]).unwrap(), + namespace: NamespaceId::new([2; 16]).unwrap(), + group: GroupId::new([3; 16]).unwrap(), + incarnation: IncarnationId::new([4; 16]).unwrap(), + }, + range: CellRange::all(), + epoch: RouteEpoch::new(1).unwrap(), + birth_digest: [5; 32], + schema_digest: [6; 32], + } +} + +fn position(index: u64) -> LogPosition { + LogPosition { + index: LogIndex::new(index), + term: Term::new(1), + } +} + +fn leaf(id: u8, pin: &GenerationPin, sequence: u64, index: u64) -> CellManifestLeaf { + let state = pin.state(); + let image = CommittedImage { + name: format!("{}.sqlite", hex::encode(pin.id().bytes())), + commitment: state.digest.into(), + length: state.length, + ltx_epoch: state.epoch, + txid: state.txid.0, + checksum: state.checksum, + page_size: state.page_size, + }; + let version = CellVersion { + history: CellHistory { + cluster: route().history.cluster, + namespace: route().history.namespace, + cell: CellId::new(format!("cell-{id}")).unwrap(), + incarnation: IncarnationId::new([7; 16]).unwrap(), + }, + sequence: MutationSequence::new(sequence).unwrap(), + image: LtxImage { + epoch: image.ltx_epoch, + txid: image.txid, + checksum: image.checksum, + commitment: image.commitment, + } + .into(), + }; + CellManifestLeaf { + enrollment: EnrollmentRecord::new( + version.history.clone(), + ProposalId::new([id; 16]).unwrap(), + route().schema_digest, + version.image, + ) + .unwrap(), + version, + geometry: CellImageGeometry { + length: image.length, + page_size: image.page_size, + }, + last_mutation: position(index), + witness: CommitWitness { + group: route().history, + position: position(index), + proposal: ProposalId::new([id; 16]).unwrap(), + }, + effects: EffectSummary::default(), + } +} + +#[test] +fn manifest_updates_only_changed_path() -> Result<()> { + let temp = tempfile::tempdir()?; + let fs = Arc::new(DirectFileSystem); + let generations = GenerationStore::open( + temp.path().join("images"), + fs.clone(), + GenerationLimits { + max_generations: 128, + ..Default::default() + }, + )?; + let mut pages = ManifestPageStore::open( + &temp.path().join("pages"), + fs.clone(), + 4096, + 64 * 1024 * 1024, + 64, + )?; + let tree = ManifestTree::new(route(), ManifestLimits::default())?; + let mut root = ManifestRoot::default(); + let mut cells = BTreeMap::new(); + for id in 1..=64 { + let bytes = images::image( + None, + &format!("CREATE TABLE value(n); INSERT INTO value VALUES ({id});"), + )?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let pin = generations.prepare(images::id(id), None, &cut.update())?; + assert_eq!(images::read(&pin)?, bytes); + let value = leaf(id, &pin, 1, u64::from(id)); + root = tree.update(&mut pages, root, value.clone(), None)?; + cells.insert(value.version.history.cell.clone(), (value, pin)); + } + assert_eq!(root.cells(), 64); + assert!(root.depth() <= 9, "AVL height must remain logarithmic"); + let old_root = root; + let target = CellId::new("cell-32").unwrap(); + let (old, pin) = cells.get(&target).unwrap(); + let before = images::read(pin)?; + let after = images::image(Some(&before), "UPDATE value SET n = 99;")?; + let cut = images::Cut::new(&before, &after, 1, true, false); + let changed = generations.prepare(images::id(100), Some(pin), &cut.update())?; + let mut value = leaf(32, &changed, 2, 65); + value.enrollment = old.enrollment.clone(); + let written = pages.usage().written_pages; + root = tree.update(&mut pages, root, value.clone(), Some(&old.version))?; + assert!(pages.usage().written_pages - written <= u64::from(old_root.depth())); + assert_eq!(tree.get(&pages, old_root, &target)?.unwrap(), *old); + assert_eq!(tree.get(&pages, root, &target)?.unwrap(), value); + for (cell, (expected, _)) in &cells { + if cell != &target { + assert_eq!(tree.get(&pages, root, cell)?.unwrap(), *expected); + } + } + tree.verify(&pages, root, position(65), |actual| { + let pin = if actual.version.history.cell == target { + &changed + } else { + &cells[&actual.version.history.cell].1 + }; + if actual.version.image.commitment != CellImageCommitment::Materialized(pin.state().digest) + { + return Err(ManifestError::Invalid("fixture image")); + } + // The retained GenerationPin is an independently verified/synced image. + Ok(()) + })?; + drop(pages); + let reopened = + ManifestPageStore::open(&temp.path().join("pages"), fs, 4096, 64 * 1024 * 1024, 64)?; + assert_eq!(tree.get(&reopened, root, &target)?.unwrap(), value); + tree.verify(&reopened, root, position(65), |_| Ok(()))?; + Ok(()) +} + +#[test] +fn manifest_rejects_conflicts_bounds_and_missing_pages() -> Result<()> { + let temp = tempfile::tempdir()?; + let fs = Arc::new(DirectFileSystem); + let generations = GenerationStore::open( + temp.path().join("images"), + fs.clone(), + GenerationLimits::default(), + )?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES (1);")?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let pin = generations.prepare(images::id(1), None, &cut.update())?; + let value = leaf(1, &pin, 1, 1); + let mut pages = ManifestPageStore::open(&temp.path().join("pages"), fs, 64, 1024 * 1024, 64)?; + let tree = ManifestTree::new( + route(), + ManifestLimits { + max_cells: 1, + max_depth: 1, + }, + )?; + let root = tree.update(&mut pages, ManifestRoot::default(), value.clone(), None)?; + let before = pages.usage(); + for replace_initial in [false, true] { + let mut changed = value.clone(); + changed.version.sequence = MutationSequence::new(2)?; + changed.version.image.txid = 2; + changed.last_mutation = position(2); + let mut initial = value.enrollment.initial(); + let creation = if replace_initial { + initial.commitment = CellImageCommitment::Materialized([99; 32]); + value.enrollment.creation() + } else { + ProposalId::new([99; 16])? + }; + changed.enrollment = EnrollmentRecord::new( + value.version.history.clone(), + creation, + route().schema_digest, + initial, + )?; + assert!(matches!( + tree.update(&mut pages, root, changed, Some(&value.version)), + Err(ManifestError::Conflict) + )); + } + assert!(tree.update(&mut pages, root, value.clone(), None).is_err()); + assert!(tree + .update(&mut pages, root, value.clone(), Some(&value.version)) + .is_err()); + assert!(tree + .update(&mut pages, root, leaf(2, &pin, 1, 2), None) + .is_err()); + assert_eq!(pages.usage(), before); + assert!(tree.verify(&pages, root, position(0), |_| Ok(())).is_err()); + let mut other = route(); + other.schema_digest = [9; 32]; + assert!(ManifestTree::new(other, ManifestLimits::default())? + .get(&pages, root, &value.version.history.cell) + .is_err()); + let path = temp + .path() + .join("pages") + .join(format!("{}.page", hex::encode(root.digest().unwrap()))); + let original = std::fs::read(&path)?; + std::fs::write(&path, vec![0; 16 * 1024 + 1])?; + assert!(tree.get(&pages, root, &value.version.history.cell).is_err()); + std::fs::write(&path, original)?; + std::fs::remove_file(&path)?; + assert!(tree.verify(&pages, root, position(1), |_| Ok(())).is_err()); + Ok(()) +} + +#[test] +fn manifest_ignores_local_image_names_and_quarantines_sync_failure() -> Result<()> { + let temp = tempfile::tempdir()?; + let fs = Arc::new(DirectFileSystem); + let generations = GenerationStore::open( + temp.path().join("images"), + fs.clone(), + GenerationLimits::default(), + )?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES (3);")?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let first = generations.prepare(images::id(1), None, &cut.update())?; + let second = generations.prepare(images::id(2), None, &cut.update())?; + let tree = ManifestTree::new(route(), ManifestLimits::default())?; + let mut a = ManifestPageStore::open(&temp.path().join("a"), fs.clone(), 64, 1024 * 1024, 64)?; + let mut b = ManifestPageStore::open(&temp.path().join("b"), fs, 64, 1024 * 1024, 64)?; + let a_root = tree.update(&mut a, ManifestRoot::default(), leaf(1, &first, 1, 1), None)?; + let b_root = tree.update( + &mut b, + ManifestRoot::default(), + leaf(1, &second, 1, 1), + None, + )?; + assert_ne!(first.id(), second.id()); + assert_eq!( + a_root, b_root, + "local generation names must not enter logical roots" + ); + let directory = temp.path().join("faulted"); + let faults = faults::Faults::new(directory.clone(), None); + let filesystem = Arc::new(faults::FaultFileSystem { + faults: faults.clone(), + }); + let mut pages = ManifestPageStore::open(&directory, filesystem.clone(), 64, 1024 * 1024, 64)?; + faults.mode.store( + faults::FAIL_DIRECTORY_SYNC, + std::sync::atomic::Ordering::Release, + ); + assert!(tree + .update( + &mut pages, + ManifestRoot::default(), + leaf(1, &first, 1, 1), + None + ) + .is_err()); + assert!(matches!( + tree.update( + &mut pages, + ManifestRoot::default(), + leaf(1, &first, 1, 1), + None + ), + Err(ManifestError::Quarantined) + )); + drop(pages); + let mut recovered = ManifestPageStore::open(&directory, filesystem, 64, 1024 * 1024, 64)?; + // Retried bytes must be independently checked and synced, even if the + // interrupted attempt left a complete content-addressed filename. + let recovered_root = tree.update( + &mut recovered, + ManifestRoot::default(), + leaf(1, &first, 1, 1), + None, + )?; + assert_eq!(a_root, recovered_root); + assert!(!faults.synced_paths.lock().unwrap().is_empty()); + Ok(()) +} + +fn engine_bootstrap() -> cellar::application::consensus::verification::GroupBootstrap { + cellar::application::consensus::verification::GroupBootstrap { + scope: GroupScope { + cluster: route().history.cluster, + namespace: route().history.namespace, + group: route().history.group, + incarnation: route().history.incarnation, + cell: "storage-fixture-group".into(), + }, + member: MemberId::new(1).unwrap(), + birth_digest: route().birth_digest, + voters: vec![MemberId::new(1).unwrap()], + learners: vec![], + } +} + +fn engine_options( + path: &std::path::Path, + mode: cellar::application::consensus::verification::LogOpenMode, + process: u8, +) -> cellar::application::consensus::verification::ReplicaLogOptions { + cellar::application::consensus::verification::ReplicaLogOptions::new( + path.to_owned(), + NodeId::new([8; 16]).unwrap(), + DiskLineage::new([9; 16]).unwrap(), + ProcessGeneration::new([process; 16]).unwrap(), + mode, + ) +} + +fn root_write( + group: &cellar::application::consensus::verification::GroupLog, + prepared: PreparedGroupManifest, +) -> Result> { + use raft::Storage as _; + let index = prepared.marker().position().index.get(); + let mut write = cellar::application::consensus::verification::GroupWrite::new(group); + // Storage fixture payload: application interpretation is qualified by the + // group apply owner separately; the real engine still checks term/prefix. + write.entries = vec![raft::prelude::Entry { + index, + term: 1, + data: b"manifest-cut".to_vec().into(), + ..Default::default() + }]; + let mut hard = group.initial_state()?.hard_state; + hard.commit = index; + write.hard_state = Some(hard); + write.group_manifest = Some(prepared); + Ok(write) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn warm_manifest_preparations_revalidate_pages_and_reuse_completed_barriers() -> Result<()> { + use cellar::application::consensus::verification::{LogOpenMode, ReplicaLog}; + use cellar::sdk::ExecutionDomain; + use std::sync::atomic::Ordering; + let temp = tempfile::tempdir()?; + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let log = domain + .run(ReplicaLog::open( + engine_options(&temp.path().join("node"), LogOpenMode::Create, 1), + Arc::new(DirectFileSystem), + )) + .await??; + let group = log.create_group(engine_bootstrap()).await?; + let generations = GenerationStore::open( + temp.path().join("images"), + Arc::new(DirectFileSystem), + GenerationLimits::default(), + )?; + let bytes = images::image( + None, + "CREATE TABLE value(n); INSERT INTO value VALUES (11);", + )?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let pin = generations.prepare(images::id(1), None, &cut.update())?; + let directory = temp.path().join("pages"); + let faults = faults::Faults::new(directory.clone(), None); + faults.all_directory_files.store(true, Ordering::Release); + let filesystem = Arc::new(faults::FaultFileSystem { + faults: faults.clone(), + }); + let open = || ManifestPageStore::open(&directory, filesystem.clone(), 64, 1024 * 1024, 16); + let mut pages = open()?; + let tree = ManifestTree::new(route(), ManifestLimits::default())?; + let value = leaf(1, &pin, 1, 2); + let root = tree.update(&mut pages, ManifestRoot::default(), value.clone(), None)?; + let before = faults.image_seals.lock().unwrap().len(); + let reads = faults.reads.load(Ordering::Acquire); + let first = tree.prepare_complete(&group, &mut pages, root, position(2), 1, |_| { + Ok(CellImagePin::Materialized(pin.clone())) + })?; + assert!( + faults.reads.load(Ordering::Acquire) > reads, + "warm preparation skipped physical reads" + ); + assert_eq!( + faults.image_seals.lock().unwrap().len(), + before, + "already durable page was flushed again" + ); + let directories = faults.synced_paths.lock().unwrap().len(); + assert_eq!( + tree.update(&mut pages, ManifestRoot::default(), value.clone(), None)?, + root + ); + assert_eq!(faults.image_seals.lock().unwrap().len(), before); + assert_eq!(faults.synced_paths.lock().unwrap().len(), directories); + + let path = faults.image_seals.lock().unwrap()[0].clone(); + let original = std::fs::read(&path)?; + let mut corrupt = original.clone(); + corrupt[0] ^= 0x80; + std::fs::write(&path, corrupt)?; + assert!( + tree.prepare_complete(&group, &mut pages, root, position(2), 1, |_| { + Ok(CellImagePin::Materialized(pin.clone())) + }) + .is_err(), + "held pins hid damaged page bytes" + ); + std::fs::write(path, original)?; + faults + .mode + .store(faults::FAIL_IMAGE_SYNC, Ordering::Release); + assert!( + tree.prepare_complete(&group, &mut pages, root, position(2), 1, |_| { + Ok(CellImagePin::Materialized(pin.clone())) + }) + .is_err(), + "failed read retained a page persistence observation" + ); + assert!(matches!( + tree.get(&pages, root, &value.version.history.cell), + Err(ManifestError::Quarantined) + )); + drop(first); + drop(pages); + + let mut pages = open()?; + tree.get(&pages, root, &value.version.history.cell)?; + faults + .mode + .store(faults::FAIL_DIRECTORY_SYNC, Ordering::Release); + assert!( + tree.prepare_complete(&group, &mut pages, root, position(2), 1, |_| { + Ok(CellImagePin::Materialized(pin.clone())) + }) + .is_err(), + "reopened filename or metadata read supplied a directory barrier" + ); + assert!(matches!( + tree.get(&pages, root, &value.version.history.cell), + Err(ManifestError::Quarantined) + )); + drop(pages); + let mut pages = open()?; + let recovered = tree.prepare_complete(&group, &mut pages, root, position(2), 1, |_| { + Ok(CellImagePin::Materialized(pin.clone())) + })?; + let before = faults.image_seals.lock().unwrap().len(); + let again = tree.prepare_complete(&group, &mut pages, root, position(2), 1, |_| { + Ok(CellImagePin::Materialized(pin.clone())) + })?; + assert_eq!(faults.image_seals.lock().unwrap().len(), before); + drop(again); + drop(recovered); + drop(pages); + log.close().await?; + domain.shutdown().await; + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn group_manifest_engine_sync_publication() -> Result<()> { + use cellar::application::consensus::verification::{ + DurableApply, GroupWrite, LogOpenMode, ReplicaLog, + }; + use cellar::sdk::ExecutionDomain; + use cellar_logic::consensus::OperationId; + let temp = tempfile::tempdir()?; + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let gate = engine_fault::Gate::new(None); + let filesystem = Arc::new(engine_fault::FaultFileSystem { gate: gate.clone() }); + let node = temp.path().join("node"); + let log = domain + .run(ReplicaLog::open( + engine_options(&node, LogOpenMode::Create, 1), + filesystem.clone(), + )) + .await??; + let group = log.create_group(engine_bootstrap()).await?; + let fs = Arc::new(DirectFileSystem); + let generations = GenerationStore::open( + temp.path().join("images"), + fs.clone(), + GenerationLimits::default(), + )?; + let bytes = images::image( + None, + "CREATE TABLE value(n); INSERT INTO value VALUES (11);", + )?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let pin = generations.prepare(images::id(1), None, &cut.update())?; + let tree = ManifestTree::new(route(), ManifestLimits::default())?; + let mut pages = ManifestPageStore::open( + &temp.path().join("pages"), + fs.clone(), + 256, + 4 * 1024 * 1024, + 64, + )?; + let root = tree.update( + &mut pages, + ManifestRoot::default(), + leaf(1, &pin, 1, 2), + None, + )?; + assert!(tree + .prepare_complete(&group, &mut pages, root, position(2), 1, |_| Ok( + CellImagePin::Empty + )) + .is_err()); + let prepared = tree.prepare_complete(&group, &mut pages, root, position(2), 1, |_| { + Ok(CellImagePin::Materialized(pin.clone())) + })?; + let expected = prepared.marker().clone(); + let stale_preparation = prepared.clone(); + let release = gate.arm(engine_fault::HOLD_SYNC); + let write = root_write(&group, prepared)?; + let selected = log.clone(); + let mut pending = + domain.spawn(async move { selected.persist(OperationId::new(1), vec![write]).await })?; + tokio::time::timeout(std::time::Duration::from_secs(5), gate.entered.notified()).await?; + assert!(group.group_manifest()?.is_none()); + assert_eq!(group.applied()?.position, position(1)); + assert!( + tokio::time::timeout(std::time::Duration::from_millis(30), &mut pending) + .await + .is_err() + ); + drop(release); + pending.await??; + assert_eq!(group.group_manifest()?.as_ref(), Some(&expected)); + assert!( + group.applied().is_err(), + "legacy empty-image consumers must refuse" + ); + let mut legacy = GroupWrite::new(&group); + legacy.applied = Some(DurableApply { + position: position(3), + image: None, + }); + assert!(log + .persist(OperationId::new(2), vec![legacy]) + .await + .is_err()); + // A complete closure does not permit skipping an unapplied log index. + let gap = tree.prepare_complete(&group, &mut pages, root, position(4), 1, |_| { + Ok(CellImagePin::Materialized(pin.clone())) + })?; + assert!(log + .persist(OperationId::new(3), vec![root_write(&group, gap)?]) + .await + .is_err()); + assert_eq!(group.group_manifest()?.as_ref(), Some(&expected)); + log.close().await?; + drop(group); + drop(log); + drop(pin); + drop(generations); + drop(pages); + let resumed = domain + .run(ReplicaLog::open( + engine_options(&node, LogOpenMode::Resume, 2), + filesystem, + )) + .await??; + let group = resumed.group(&engine_bootstrap().scope)?; + let observed = group.group_manifest()?.unwrap(); + assert_eq!(observed, expected); + let mut stale = GroupWrite::new(&group); + stale.group_manifest = Some(stale_preparation); + assert!( + resumed + .persist(OperationId::new(4), vec![stale]) + .await + .is_err(), + "prepared closure belongs to the old engine instance" + ); + let generations = GenerationStore::open( + temp.path().join("images"), + fs.clone(), + GenerationLimits::default(), + )?; + let mut pages = + ManifestPageStore::open(&temp.path().join("pages"), fs, 256, 4 * 1024 * 1024, 64)?; + let restored = tree.prepare_complete( + &group, + &mut pages, + observed.root(), + observed.position(), + observed.membership_revision(), + |_| { + generations + .adopt(images::id(1), cut.result) + .map(CellImagePin::Materialized) + .map_err(|_| ManifestError::Invalid("recovery image")) + }, + )?; + assert_eq!(restored.marker(), &expected); + assert_eq!(restored.pinned_images(), 1); + resumed.close().await?; + domain.shutdown().await; + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn group_manifest_recovery_requires_every_image() -> Result<()> { + use cellar::application::consensus::verification::{LogOpenMode, ReplicaLog}; + use cellar::sdk::ExecutionDomain; + use cellar_logic::consensus::OperationId; + let temp = tempfile::tempdir()?; + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let fs = Arc::new(DirectFileSystem); + let node = temp.path().join("node"); + let tree = ManifestTree::new(route(), ManifestLimits::default())?; + let mut states = Vec::new(); + { + let log = domain + .run(ReplicaLog::open( + engine_options(&node, LogOpenMode::Create, 1), + fs.clone(), + )) + .await??; + let group = log.create_group(engine_bootstrap()).await?; + let generations = GenerationStore::open( + temp.path().join("images"), + fs.clone(), + GenerationLimits::default(), + )?; + let mut pages = ManifestPageStore::open( + &temp.path().join("pages"), + fs.clone(), + 256, + 4 * 1024 * 1024, + 64, + )?; + let mut pins = Vec::new(); + let mut root = ManifestRoot::default(); + for id in 1..=2 { + let bytes = images::image( + None, + &format!("CREATE TABLE value(n); INSERT INTO value VALUES ({id});"), + )?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let pin = generations.prepare(images::id(id), None, &cut.update())?; + states.push(cut.result); + pins.push(pin.clone()); + root = tree.update(&mut pages, root, leaf(id, &pin, 1, u64::from(id) + 1), None)?; + let prepared = tree.prepare_complete( + &group, + &mut pages, + root, + position(u64::from(id) + 1), + 1, + |cell| { + let selected = if cell.version.history.cell.as_str() == "cell-1" { + 0 + } else { + 1 + }; + Ok(CellImagePin::Materialized(pins[selected].clone())) + }, + )?; + log.persist( + OperationId::new(u64::from(id)), + vec![root_write(&group, prepared)?], + ) + .await?; + } + log.close().await?; + } + // Only this owned fixture's image is removed. The actual engine marker and + // every manifest page survive; recovering the other cell cannot cover it. + std::fs::remove_file( + temp.path() + .join("images") + .join(format!("{}.sqlite", hex::encode(images::id(1).bytes()))), + )?; + let log = domain + .run(ReplicaLog::open( + engine_options(&node, LogOpenMode::Resume, 2), + fs.clone(), + )) + .await??; + let group = log.group(&engine_bootstrap().scope)?; + let marker = group.group_manifest()?.unwrap(); + assert_eq!(marker.root().cells(), 2); + let generations = GenerationStore::open( + temp.path().join("images"), + fs.clone(), + GenerationLimits::default(), + )?; + let mut pages = + ManifestPageStore::open(&temp.path().join("pages"), fs, 256, 4 * 1024 * 1024, 64)?; + assert!(tree + .prepare_complete( + &group, + &mut pages, + marker.root(), + marker.position(), + marker.membership_revision(), + |cell| { + let id = if cell.version.history.cell.as_str() == "cell-1" { + 1 + } else { + 2 + }; + generations + .adopt(images::id(id), states[usize::from(id - 1)]) + .map(CellImagePin::Materialized) + .map_err(|_| ManifestError::Invalid("missing selected cell image")) + } + ) + .is_err()); + assert!(group.applied().is_err()); + log.close().await?; + domain.shutdown().await; + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn incremental_manifest_preserves_images_and_rejects_forks() -> Result<()> { + use cellar::application::consensus::verification::{LogOpenMode, ReplicaLog}; + use cellar::sdk::ExecutionDomain; + use cellar_logic::consensus::OperationId; + let temp = tempfile::tempdir()?; + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let fs = Arc::new(DirectFileSystem); + let log = domain + .run(ReplicaLog::open( + engine_options(&temp.path().join("node"), LogOpenMode::Create, 1), + fs.clone(), + )) + .await??; + let group = log.create_group(engine_bootstrap()).await?; + let generations = GenerationStore::open( + temp.path().join("images"), + fs.clone(), + GenerationLimits { + max_generations: 128, + ..Default::default() + }, + )?; + let page_path = temp.path().join("pages"); + let trace = faults::Faults::new(page_path.clone(), None); + let page_fs = Arc::new(faults::FaultFileSystem { + faults: trace.clone(), + }); + let mut pages = + ManifestPageStore::open(&page_path, page_fs.clone(), 4096, 64 * 1024 * 1024, 64)?; + let tree = ManifestTree::new(route(), ManifestLimits::default())?; + let mut pins = BTreeMap::new(); + let mut root = ManifestRoot::default(); + // All cells are part of this fixture's first complete checkpoint at index 2. + for id in 1..=64 { + let bytes = images::image( + None, + &format!("CREATE TABLE value(n); INSERT INTO value VALUES ({id});"), + )?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let pin = generations.prepare(images::id(id), None, &cut.update())?; + let value = leaf(id, &pin, 1, 2); + root = tree.update(&mut pages, root, value.clone(), None)?; + pins.insert(value.version.history.cell, pin); + } + let original = tree.prepare_complete(&group, &mut pages, root, position(2), 1, |cell| { + Ok(CellImagePin::Materialized( + pins[&cell.version.history.cell].clone(), + )) + })?; + log.persist( + OperationId::new(1), + vec![root_write(&group, original.clone())?], + ) + .await?; + drop(pins); + let target = CellId::new("cell-32")?; + let (old_leaf, CellImagePin::Materialized(reader)) = original.pin_cell(&target).unwrap() else { + panic!("materialized fixture"); + }; + let before = images::read(&reader)?; + let after = images::image(Some(&before), "UPDATE value SET n = 999;")?; + let cut = images::Cut::new(&before, &after, 1, true, false); + let changed = generations.prepare(images::id(100), Some(&reader), &cut.update())?; + let mut next_leaf = leaf(32, &changed, 2, 3); + next_leaf.enrollment = old_leaf.enrollment.clone(); + let written = pages.usage().written_pages; + let reads = trace.reads.load(std::sync::atomic::Ordering::Acquire); + trace.synced_paths.lock().unwrap().clear(); + assert!(tree + .prepare_update( + &mut pages, + &original, + next_leaf.clone(), + Some(&old_leaf.version), + CellImagePin::Empty + ) + .is_err()); + assert_eq!(pages.usage().written_pages, written); + let next = tree.prepare_update( + &mut pages, + &original, + next_leaf.clone(), + Some(&old_leaf.version), + CellImagePin::Materialized(changed.clone()), + )?; + assert_eq!(next.pinned_images(), 64); + assert!(pages.usage().written_pages - written <= u64::from(root.depth())); + let read_count = trace.reads.load(std::sync::atomic::Ordering::Acquire) - reads; + assert!(read_count <= 4 * usize::from(root.depth()) + 2); + assert!( + trace.synced_paths.lock().unwrap().len() <= usize::from(root.depth()) * 2, + "mutation must not sync the complete closure" + ); + assert_eq!(original.pin_cell(&target).unwrap().0, old_leaf); + assert_eq!(next.pin_cell(&target).unwrap().0, &next_leaf); + tree.verify(&pages, next.marker().root(), position(3), |cell| { + assert_eq!(next.pin_cell(&cell.version.history.cell).unwrap().0, cell); + if cell.version.history.cell != target { + assert_eq!( + original.pin_cell(&cell.version.history.cell).unwrap().0, + cell + ); + } + Ok(()) + })?; + // Stage a different branch at 3 and its successor at 4. Matching the next + // index alone must never allow it to replace the selected branch at 3. + let mut branch = original.clone(); + for id in 65..=66 { + let bytes = images::image( + None, + &format!("CREATE TABLE value(n); INSERT INTO value VALUES ({id});"), + )?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let pin = generations.prepare(images::id(id), None, &cut.update())?; + branch = tree.prepare_update( + &mut pages, + &branch, + leaf(id, &pin, 1, u64::from(id) - 62), + None, + CellImagePin::Materialized(pin), + )?; + } + log.persist(OperationId::new(2), vec![root_write(&group, next.clone())?]) + .await?; + assert!(log + .persist(OperationId::new(3), vec![root_write(&group, branch)?]) + .await + .is_err()); + assert_eq!(group.group_manifest()?.as_ref(), Some(next.marker())); + let mut retry = cellar::application::consensus::verification::GroupWrite::new(&group); + retry.group_manifest = Some(next.clone()); + log.persist(OperationId::new(4), vec![retry]).await?; + // Explicitly retire fixture inventory after authoritative roots are pinned. + // Only the two abandoned branch images may disappear at this point. + generations.retain_only(&[])?; + assert_eq!(generations.reclaim(128)?.files, 2); + drop(original); + assert_eq!( + generations.reclaim(128)?.files, + 0, + "old reader still pins cell-32" + ); + assert_eq!(images::read(&reader)?, before); + drop(reader); + assert_eq!( + generations.reclaim(128)?.files, + 1, + "only the superseded cell image is now reclaimable" + ); + drop(changed); + assert_eq!(generations.reclaim(128)?.files, 0); + // Live page pins keep the directory lock even after the store handle drops. + // A different owner also cannot accept this preparation capability. + drop(pages); + assert!( + ManifestPageStore::open(&page_path, page_fs.clone(), 4096, 64 * 1024 * 1024, 64).is_err() + ); + let mut pages = ManifestPageStore::open( + &temp.path().join("other-pages"), + page_fs.clone(), + 4096, + 64 * 1024 * 1024, + 64, + )?; + let mut candidate = next_leaf.clone(); + candidate.last_mutation = position(4); + assert!(matches!( + tree.prepare_update( + &mut pages, + &next, + candidate, + Some(&next_leaf.version), + next.pin_cell(&target).unwrap().1 + ), + Err(ManifestError::Invalid("root/page owner")) + )); + drop(next); + assert_eq!( + generations.reclaim(128)?.files, + 0, + "engine owns the selected root's images" + ); + log.close().await?; + drop(group); + drop(log); + assert_eq!(generations.reclaim(128)?.files, 64); + let _reopened = ManifestPageStore::open(&page_path, page_fs, 4096, 64 * 1024 * 1024, 64)?; + domain.shutdown().await; + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn incremental_enrollment_preserves_rotated_subtrees() -> Result<()> { + use cellar::application::consensus::verification::{LogOpenMode, ReplicaLog}; + use cellar::sdk::ExecutionDomain; + let temp = tempfile::tempdir()?; + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let fs = Arc::new(DirectFileSystem); + let log = domain + .run(ReplicaLog::open( + engine_options(&temp.path().join("node"), LogOpenMode::Create, 1), + fs.clone(), + )) + .await??; + let group = log.create_group(engine_bootstrap()).await?; + let generations = GenerationStore::open( + temp.path().join("images"), + fs.clone(), + GenerationLimits::default(), + )?; + // Identical initial bytes may be physically shared; logical cell identity + // and independently ordered enrollment still remain distinct. + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES (1);")?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let pin = generations.prepare(images::id(1), None, &cut.update())?; + let mut pages = + ManifestPageStore::open(&temp.path().join("pages"), fs, 4096, 64 * 1024 * 1024, 64)?; + let tree = ManifestTree::new(route(), ManifestLimits::default())?; + let mut current = tree.prepare_complete( + &group, + &mut pages, + ManifestRoot::default(), + position(1), + 1, + |_| panic!("empty closure"), + )?; + let empty = current.clone(); + let mut historical = None; + for id in 1..=96 { + let before = pages.usage().written_pages; + let depth = current.marker().root().depth(); + current = tree.prepare_update( + &mut pages, + ¤t, + leaf(id, &pin, 1, u64::from(id) + 1), + None, + CellImagePin::Materialized(pin.clone()), + )?; + assert_eq!(current.pinned_images(), usize::from(id)); + assert!(pages.usage().written_pages - before <= 3 * u64::from(depth) + 3); + tree.verify( + &pages, + current.marker().root(), + current.marker().position(), + |cell| { + assert_eq!( + current.pin_cell(&cell.version.history.cell).unwrap().0, + cell + ); + Ok(()) + }, + )?; + if id == 32 { + historical = Some(current.clone()); + } + } + assert_eq!(empty.pinned_images(), 0); + let historical = historical.unwrap(); + assert_eq!(historical.pinned_images(), 32); + assert!(historical.pin_cell(&CellId::new("cell-33")?).is_none()); + tree.verify( + &pages, + historical.marker().root(), + historical.marker().position(), + |cell| { + assert_eq!( + historical.pin_cell(&cell.version.history.cell).unwrap().0, + cell + ); + Ok(()) + }, + )?; + let full = tree.prepare_complete( + &group, + &mut pages, + current.marker().root(), + current.marker().position(), + 1, + |_| Ok(CellImagePin::Materialized(pin.clone())), + )?; + assert_eq!(full.marker(), current.marker()); + assert_eq!(full.pinned_images(), current.pinned_images()); + log.close().await?; + domain.shutdown().await; + Ok(()) +} + +fn collect_pages(pages: &mut ManifestPageStore) -> Result { + let mut deleted = 0; + for _ in 0..257 { + let report = pages.reclaim(16)?; + assert!(report.scanned <= 16); + deleted += report.pages; + if report.scanned < 16 { + return Ok(deleted); + } + } + anyhow::bail!("bounded fixture inventory did not finish scanning") +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn manifest_pin_reclamation() -> Result<()> { + use cellar::application::consensus::verification::{LogOpenMode, ReplicaLog}; + use cellar::sdk::ExecutionDomain; + use cellar_logic::consensus::OperationId; + let temp = tempfile::tempdir()?; + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let fs = Arc::new(DirectFileSystem); + let gate = engine_fault::Gate::new(None); + let log = domain + .run(ReplicaLog::open( + engine_options(&temp.path().join("node"), LogOpenMode::Create, 1), + Arc::new(engine_fault::FaultFileSystem { gate: gate.clone() }), + )) + .await??; + let group = log.create_group(engine_bootstrap()).await?; + let generations = GenerationStore::open( + temp.path().join("images"), + fs.clone(), + GenerationLimits::default(), + )?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES (1);")?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let pin = generations.prepare(images::id(1), None, &cut.update())?; + let mut pages = + ManifestPageStore::open(&temp.path().join("pages"), fs, 256, 4 * 1024 * 1024, 16)?; + let tree = ManifestTree::new(route(), ManifestLimits::default())?; + let root = tree.update( + &mut pages, + ManifestRoot::default(), + leaf(1, &pin, 1, 2), + None, + )?; + let old = tree.prepare_complete(&group, &mut pages, root, position(2), 1, |_| { + Ok(CellImagePin::Materialized(pin.clone())) + })?; + log.persist(OperationId::new(1), vec![root_write(&group, old.clone())?]) + .await?; + assert!( + pages.reclaim(16).is_err(), + "inventory must first be reconciled" + ); + let reader = old.clone(); + let transition = old.clone(); + // The stored snapshot reference survives release of the caller's handle. + assert!(matches!( + pages.reconcile_retained(&vec![old.clone(); 15]), + Err(ManifestError::Limit) + )); + assert!( + pages.reclaim(16).is_err(), + "failed reconciliation must not open collection" + ); + pages.reconcile_retained(std::slice::from_ref(&old))?; + let orphan = tree.update(&mut pages, root, leaf(3, &pin, 1, 3), None)?; + let next = tree.prepare_update( + &mut pages, + &old, + leaf(2, &pin, 1, 3), + None, + CellImagePin::Materialized(pin.clone()), + )?; + let release = gate.arm(engine_fault::HOLD_SYNC); + let writer = log.clone(); + let write = root_write(&group, next.clone())?; + let pending = + domain.spawn(async move { writer.persist(OperationId::new(2), vec![write]).await })?; + tokio::time::timeout(std::time::Duration::from_secs(5), gate.entered.notified()).await?; + assert!(matches!(pages.reclaim(16), Err(ManifestError::Conflict))); + assert!( + pages.reconcile_retained(&[]).is_err(), + "cannot retire roots during publication" + ); + drop(release); + pending.await??; + assert!( + collect_pages(&mut pages)? > 0, + "unmarked paths are reclaimable" + ); + assert!(tree + .verify(&pages, orphan, position(3), |_| Ok(())) + .is_err()); + tree.verify(&pages, root, position(2), |_| Ok(()))?; + drop(old); + drop(reader); + pages.reconcile_retained(&[])?; + collect_pages(&mut pages)?; + tree.verify(&pages, root, position(2), |_| Ok(()))?; + pages.reconcile_retained(std::slice::from_ref(&transition))?; + drop(transition); + collect_pages(&mut pages)?; + tree.verify(&pages, root, position(2), |_| Ok(()))?; + pages.reconcile_retained(&[])?; + assert!(collect_pages(&mut pages)? > 0); + assert!(tree.verify(&pages, root, position(2), |_| Ok(())).is_err()); + tree.verify(&pages, next.marker().root(), position(3), |_| Ok(()))?; + assert_eq!(pages.usage().pages, 2); + drop(next); + assert_eq!( + collect_pages(&mut pages)?, + 0, + "engine-selected root survives caller release" + ); + log.close().await?; + assert!(matches!(pages.reclaim(16), Err(ManifestError::Quarantined))); + domain.shutdown().await; + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn uncertain_manifest_publication_retains_both_roots() -> Result<()> { + use cellar::application::consensus::verification::{LogOpenMode, ReplicaLog}; + use cellar::sdk::ExecutionDomain; + use cellar_logic::consensus::OperationId; + let temp = tempfile::tempdir()?; + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let fs = Arc::new(DirectFileSystem); + let gate = engine_fault::Gate::new(None); + let node = temp.path().join("node"); + let log = domain + .run(ReplicaLog::open( + engine_options(&node, LogOpenMode::Create, 1), + Arc::new(engine_fault::FaultFileSystem { gate: gate.clone() }), + )) + .await??; + let group = log.create_group(engine_bootstrap()).await?; + let image_path = temp.path().join("images"); + let page_path = temp.path().join("pages"); + let generations = + GenerationStore::open(image_path.clone(), fs.clone(), GenerationLimits::default())?; + let before = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES (1);")?; + let cut = images::Cut::new(&[], &before, 0, true, false); + let pin = generations.prepare(images::id(1), None, &cut.update())?; + let mut pages = ManifestPageStore::open(&page_path, fs.clone(), 256, 4 * 1024 * 1024, 16)?; + let tree = ManifestTree::new(route(), ManifestLimits::default())?; + let old_leaf = leaf(1, &pin, 1, 2); + let root = tree.update(&mut pages, ManifestRoot::default(), old_leaf.clone(), None)?; + let current = tree.prepare_complete(&group, &mut pages, root, position(2), 1, |_| { + Ok(CellImagePin::Materialized(pin.clone())) + })?; + log.persist( + OperationId::new(1), + vec![root_write(&group, current.clone())?], + ) + .await?; + pages.reconcile_retained(&[])?; + let after = images::image(Some(&before), "UPDATE value SET n = 2;")?; + let changed = images::Cut::new(&before, &after, 1, true, false); + let next_pin = generations.prepare(images::id(2), Some(&pin), &changed.update())?; + let next = tree.prepare_update( + &mut pages, + ¤t, + CellManifestLeaf { + enrollment: old_leaf.enrollment.clone(), + ..leaf(1, &next_pin, 2, 3) + }, + Some(&old_leaf.version), + CellImagePin::Materialized(next_pin.clone()), + )?; + let expected = next.marker().clone(); + drop(pin); + drop(next_pin); + drop(current); + generations.retain_only(&[])?; + let release = gate.arm(engine_fault::FAIL_AFTER_SYNC); + assert!(log + .persist(OperationId::new(2), vec![root_write(&group, next)?]) + .await + .is_err()); + drop(release); + assert!(log.close().await.is_err()); + assert!(matches!(pages.reclaim(16), Err(ManifestError::Quarantined))); + assert!(pages.reconcile_retained(&[]).is_err()); + assert_eq!( + generations.reclaim(16)?.files, + 0, + "both images survive loss of request and engine authority" + ); + drop(group); + drop(log); + // The page owner, rather than the failed engine cache, still retains both. + assert_eq!(generations.reclaim(16)?.files, 0); + drop(pages); + drop(generations); + let log = domain + .run(ReplicaLog::open( + engine_options(&node, LogOpenMode::Resume, 2), + fs.clone(), + )) + .await??; + let group = log.group(&engine_bootstrap().scope)?; + assert_eq!( + group.group_manifest()?.as_ref(), + Some(&expected), + "injected failure was after actual fsync" + ); + let generations = GenerationStore::open(image_path, fs.clone(), GenerationLimits::default())?; + let mut pages = ManifestPageStore::open(&page_path, fs, 256, 4 * 1024 * 1024, 16)?; + assert!(pages.reclaim(16).is_err()); + let restored = tree.prepare_complete( + &group, + &mut pages, + expected.root(), + expected.position(), + expected.membership_revision(), + |_| { + generations + .adopt(images::id(2), changed.result) + .map(CellImagePin::Materialized) + .map_err(|_| ManifestError::Invalid("recovered image")) + }, + )?; + pages.reconcile_retained(&[])?; + assert_eq!(collect_pages(&mut pages)?, 1); + generations.retain_only(&[])?; + assert_eq!(generations.reclaim(16)?.files, 1); + let (_, CellImagePin::Materialized(restored_image)) = + restored.pin_cell(&CellId::new("cell-1")?).unwrap() + else { + panic!("materialized fixture"); + }; + assert_eq!(images::read(&restored_image)?, after); + drop(restored_image); + drop(restored); + assert_eq!( + generations.reclaim(16)?.files, + 0, + "recovery registers selected root retention without a new write" + ); + log.close().await?; + domain.shutdown().await; + Ok(()) +} + +#[test] +fn manifest_collection_barrier_failure_keeps_quota() -> Result<()> { + for mode in [faults::FAIL_REMOVE, faults::FAIL_DIRECTORY_SYNC] { + let temp = tempfile::tempdir()?; + let path = temp.path().join("pages"); + let trace = faults::Faults::new(path.clone(), None); + let fs = Arc::new(faults::FaultFileSystem { + faults: trace.clone(), + }); + let generations = GenerationStore::open( + temp.path().join("images"), + Arc::new(DirectFileSystem), + GenerationLimits::default(), + )?; + let bytes = images::image(None, "CREATE TABLE value(n);")?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let pin = generations.prepare(images::id(1), None, &cut.update())?; + let tree = ManifestTree::new(route(), ManifestLimits::default())?; + let mut pages = ManifestPageStore::open(&path, fs.clone(), 64, 1024 * 1024, 8)?; + // No engine root was published: every page in this fixture is an orphan. + tree.update( + &mut pages, + ManifestRoot::default(), + leaf(1, &pin, 1, 2), + None, + )?; + pages.reconcile_retained(&[])?; + let before = pages.usage(); + trace.mode.store(mode, std::sync::atomic::Ordering::Release); + assert!(pages.reclaim(16).is_err()); + assert_eq!( + pages.usage(), + before, + "uncertain deletion must not refund quota" + ); + assert!(matches!(pages.reclaim(16), Err(ManifestError::Quarantined))); + drop(pages); + let mut pages = ManifestPageStore::open(&path, fs, 64, 1024 * 1024, 8)?; + assert!(pages.reclaim(16).is_err()); + pages.reconcile_retained(&[])?; + collect_pages(&mut pages)?; + assert_eq!(pages.usage().pages, 0); + assert_eq!(pages.usage().bytes, 0); + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn manifest_batch_retention_refusal_cancels_unwritten_reservations() -> Result<()> { + use cellar::application::consensus::verification::{LogOpenMode, ReplicaLog}; + use cellar::sdk::ExecutionDomain; + use cellar_logic::consensus::OperationId; + use raft::Storage as _; + let temp = tempfile::tempdir()?; + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let fs = Arc::new(DirectFileSystem); + let log = domain + .run(ReplicaLog::open( + engine_options(&temp.path().join("node"), LogOpenMode::Create, 1), + fs.clone(), + )) + .await??; + let first_group = log.create_group(engine_bootstrap()).await?; + let mut second_birth = engine_bootstrap(); + second_birth.scope.group = GroupId::new([33; 16])?; + let second_group = log.create_group(second_birth).await?; + let mut second_route = route(); + second_route.history.group = second_group.identity().scope.group; + let first_tree = ManifestTree::new(route(), ManifestLimits::default())?; + let second_tree = ManifestTree::new(second_route, ManifestLimits::default())?; + let generations = GenerationStore::open( + temp.path().join("images"), + fs.clone(), + GenerationLimits::default(), + )?; + let bytes = images::image(None, "CREATE TABLE value(n);")?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let pin = generations.prepare(images::id(1), None, &cut.update())?; + // Capacity admits one group's current/pending pair, never two groups. + let mut pages = ManifestPageStore::open(&temp.path().join("pages"), fs, 64, 1024 * 1024, 2)?; + let first_root = first_tree.update( + &mut pages, + ManifestRoot::default(), + leaf(1, &pin, 1, 2), + None, + )?; + let second_root = second_tree.update( + &mut pages, + ManifestRoot::default(), + leaf(2, &pin, 1, 2), + None, + )?; + let first = first_tree.prepare_complete( + &first_group, + &mut pages, + first_root, + position(2), + 1, + |_| Ok(CellImagePin::Materialized(pin.clone())), + )?; + let second = second_tree.prepare_complete( + &second_group, + &mut pages, + second_root, + position(2), + 1, + |_| Ok(CellImagePin::Materialized(pin.clone())), + )?; + assert!(log + .persist( + OperationId::new(1), + vec![ + root_write(&first_group, first.clone())?, + root_write(&second_group, second.clone())? + ] + ) + .await + .is_err()); + for group in [&first_group, &second_group] { + assert!(group.group_manifest()?.is_none()); + assert_eq!( + group.last_index()?, + 1, + "no part of the rejected batch reached storage" + ); + } + pages.reconcile_retained(&[])?; + assert_eq!( + collect_pages(&mut pages)?, + 0, + "prepared apply pins protect both staged roots" + ); + log.persist(OperationId::new(2), vec![root_write(&first_group, first)?]) + .await?; + drop(second); + assert_eq!(collect_pages(&mut pages)?, 1); + assert_eq!(pages.usage().pages, 1); + log.close().await?; + domain.shutdown().await; + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn unchanged_group_root_refuses_quarantined_page_store() -> Result<()> { + use cellar::application::consensus::verification::{ + CellGroupBootstrap, LogOpenMode, ReplicaLog, + }; + use cellar::sdk::ExecutionDomain; + let temp = tempfile::tempdir()?; + let fs = Arc::new(DirectFileSystem); + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let log = domain + .run(ReplicaLog::open( + engine_options(&temp.path().join("node"), LogOpenMode::Create, 1), + fs.clone(), + )) + .await??; + let group = log + .create_cell_group(CellGroupBootstrap { + route: route(), + member: MemberId::new(1)?, + voters: vec![MemberId::new(1)?], + learners: vec![], + }) + .await?; + let page_path = temp.path().join("pages"); + let fault = faults::Faults::new(page_path.clone(), None); + let page_fs = Arc::new(faults::FaultFileSystem { + faults: fault.clone(), + }); + let mut pages = ManifestPageStore::open(&page_path, page_fs, 64, 1024 * 1024, 16)?; + let tree = ManifestTree::new(route(), ManifestLimits::default())?; + let current = tree.prepare_complete( + &group, + &mut pages, + ManifestRoot::default(), + position(1), + 1, + |_| panic!("genesis is empty"), + )?; + assert!(tree + .prepare_unchanged(&pages, ¤t, position(2)) + .is_ok()); + let generations = + GenerationStore::open(temp.path().join("images"), fs, GenerationLimits::default())?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES (1);")?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let pin = generations.prepare(images::id(1), None, &cut.update())?; + fault.mode.store( + faults::FAIL_DIRECTORY_SYNC, + std::sync::atomic::Ordering::Release, + ); + assert!(tree + .prepare_update( + &mut pages, + ¤t, + leaf(1, &pin, 1, 2), + None, + CellImagePin::Materialized(pin) + ) + .is_err()); + assert_eq!(fault.mode.load(std::sync::atomic::Ordering::Acquire), 0); + assert!(matches!( + tree.prepare_unchanged(&pages, ¤t, position(2)), + Err(ManifestError::Quarantined) + )); + assert_eq!(group.group_manifest()?.unwrap().position(), position(1)); + log.close().await?; + domain.shutdown().await; + Ok(()) +} diff --git a/crates/cellar/tests/native_daemon.rs b/crates/cellar/tests/native_daemon.rs index 672ae20e2..38814c7a5 100644 --- a/crates/cellar/tests/native_daemon.rs +++ b/crates/cellar/tests/native_daemon.rs @@ -1,551 +1,12 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. //! Process-level native ingress, routing, bucket restore and reload coverage. - -// Process orchestration runs outside the cell execution boundary and uses -// host wall-clock deadlines while child nodes own their execution domains. #![allow(clippy::disallowed_methods)] -use anyhow::{bail, Context as _, Result}; -use cellar::bucket::Bucket; -use reqwest::{Client, Method}; -use serde_json::{json, Value}; -use std::fs::File; -use std::path::{Path, PathBuf}; -use std::process::{Child, Command, Stdio}; -use std::time::{Duration, Instant}; - -struct Node { - process: Child, - log: PathBuf, - diagnostics: PathBuf, - ingress: String, - internal: String, -} - -impl Drop for Node { - fn drop(&mut self) { - if self.process.try_wait().ok().flatten().is_none() { - let _ = self.process.kill(); - } - let _ = self.process.wait(); - } -} - -impl Node { - async fn start(root: &Path, name: &str, bucket: Option<&Path>) -> Result { - Self::start_program( - root, - name, - bucket, - env!("CARGO_BIN_EXE_cellar-native-counter"), - None, - ) - .await - } - - async fn start_program( - root: &Path, - name: &str, - bucket: Option<&Path>, - program: &str, - source: Option<&Path>, - ) -> Result { - let log = root.join(format!("{name}.log")); - let stdout = File::create(&log)?; - // Listener announcements are stdout data. Merging stderr into the - // same file can splice a concurrent diagnostic into their lines. - let diagnostics = root.join(format!("{name}-stderr.log")); - let stderr = File::create(&diagnostics)?; - let mut command = Command::new(program); - command - .env_clear() - .args([ - "--listen", - "127.0.0.1:0", - "--internal-listen", - "127.0.0.1:0", - ]) - .env("CELLAR_NODE", name) - .env("CELLAR_TEST_DATA_DIR", root.join(format!("{name}-data"))) - .env("CELLAR_TOKIO_THREADS", "2") - .env("CELLAR_PLACEMENT_WEIGHT", "1") - .env("CELLAR_REBALANCE_INTERVAL_MS", "0") - .env("CELLAR_DEPLOY_POLL_S", "3600") - .env("CELLAR_DURABILITY", "bucket") - .env("CELLAR_SHUTDOWN_TOTAL_MS", "8000") - .env("RUST_LOG", "warn") - .stdout(Stdio::from(stdout)) - .stderr(Stdio::from(stderr)); - if let Some(bucket) = bucket { - command - .env("CELLAR_NATIVE_FLEET", "1") - .env("CELLAR_BUCKET", "cellar-dev") - .env("CELLAR_INTERNAL_DEV_STORE", bucket); - } - if let Some(source) = source { - command.env("CELLAR_TEST_SCRIPT_PATH", source); - } - let mut node = Self { - process: command.spawn()?, - log, - diagnostics, - ingress: String::new(), - internal: String::new(), - }; - let deadline = Instant::now() + Duration::from_secs(30); - loop { - let text = std::fs::read_to_string(&node.log)?; - let address = |prefix: &str| { - text.lines().find_map(|line| { - line.strip_prefix(prefix) - .and_then(|rest| rest.split_whitespace().next()) - .map(|address| format!("http://{address}")) - }) - }; - if let (Some(ingress), Some(internal)) = ( - address("cellar listening on "), - address("cellar internal listening on "), - ) { - node.ingress = ingress; - node.internal = internal; - return Ok(node); - } - if let Some(status) = node.process.try_wait()? { - bail!( - "native node {name} exited during boot: {status}\n{}", - node.logs()? - ); - } - if Instant::now() >= deadline { - bail!("native node {name} did not boot\n{}", node.logs()?); - } - tokio::time::sleep(Duration::from_millis(50)).await; - } - } - - fn logs(&self) -> Result { - Ok(format!( - "stdout:\n{}\nstderr:\n{}", - std::fs::read_to_string(&self.log)?, - std::fs::read_to_string(&self.diagnostics)? - )) - } - - async fn ready(&mut self, client: &Client) -> Result<()> { - let deadline = Instant::now() + Duration::from_secs(30); - loop { - if client - .get(format!("{}/.well-known/cellar/health", self.ingress)) - .send() - .await - .is_ok_and(|response| response.status().is_success()) - { - return Ok(()); - } - if let Some(status) = self.process.try_wait()? { - bail!("node exited before readiness: {status}\n{}", self.logs()?); - } - if Instant::now() >= deadline { - bail!("native readiness timed out\n{}", self.logs()?); - } - tokio::time::sleep(Duration::from_millis(50)).await; - } - } - - async fn stop(&mut self, client: &Client) -> Result<()> { - client - .post(format!("{}/shutdown", self.internal)) - .send() - .await? - .error_for_status()?; - let deadline = Instant::now() + Duration::from_secs(15); - loop { - if let Some(status) = self.process.try_wait()? { - anyhow::ensure!( - status.success(), - "native shutdown failed: {status}\n{}", - self.logs()? - ); - return Ok(()); - } - if Instant::now() >= deadline { - bail!("native node did not complete shutdown"); - } - tokio::time::sleep(Duration::from_millis(50)).await; - } - } -} - -async fn counter(client: &Client, node: &Node, name: &str, write: bool) -> Result<(u64, String)> { - let response = client - .request( - if write { Method::POST } else { Method::GET }, - format!("{}/counters/{name}", node.ingress), - ) - .send() - .await?; - let status = response.status(); - let version = response - .headers() - .get("x-native-version") - .context("missing compiled version")? - .to_str()? - .to_string(); - let body = response.text().await?; - anyhow::ensure!( - status.is_success(), - "native counter failed: {status} {body}" - ); - Ok(( - serde_json::from_str::(&body)?["count"] - .as_u64() - .context("missing count")?, - version, - )) -} - -async fn deployment( - bucket: &Bucket, - version: &str, - executable: &std::path::Path, -) -> Result { - deployment_with_assets(bucket, version, version, executable).await -} - -async fn deployment_with_assets( - bucket: &Bucket, - version: &str, - asset_revision: &str, - executable: &Path, -) -> Result { - use cellar::native_deploy::{current_target, executable_identity, ApplicationConfig, Package}; - let artifact = executable_identity(executable, current_target())?; - let root = executable.parent().context("executable has no directory")?; - let assets = root.join(format!("assets-{asset_revision}")); - std::fs::create_dir_all(assets.join("counters"))?; - std::fs::write( - assets.join("index.html"), - format!("native assets {asset_revision}"), - )?; - std::fs::write(assets.join("counters/tenant-acme"), "static counter shadow")?; - std::fs::write(assets.join("counters/static.txt"), "static route exclusion")?; - std::fs::write( - assets.join(format!("{asset_revision}-only.txt")), - format!("only {asset_revision}"), - )?; - std::fs::write( - assets.join("_headers"), - format!("/*\n X-Asset-Version: {asset_revision}\n"), - )?; - std::fs::write(assets.join("_redirects"), "/old / 302\n")?; - let config: ApplicationConfig = serde_json::from_value(json!({ - "script_name": "native-counter", "application_version": version, - "compatibility_date":"2026-01-01", - "cells": ["Counter"], - "bindings": [{"type": "durable_object_namespace", "name": "COUNTERS", "class_name": "Counter"}], - "assets": {"directory":format!("assets-{asset_revision}"), "binding":"ASSETS", - "not_found_handling":"single-page-application", - "run_worker_first":["/counters/*","!/counters/static.txt","/assets-binding/*"]}, - }))?; - let staged = root.join(format!("package-{version}-{asset_revision}")); - let primary = config.stage_manifest(vec![artifact.clone()], root, &staged)?; - std::fs::copy(executable, staged.join(&artifact.sha256))?; - let package = Package { - primary, - cohosted: Vec::new(), - }; - package.publish_staged(bucket, &staged).await?; - Ok(package.primary.version) -} +#[path = "support/native_daemon_counter.rs"] +pub mod counter; #[tokio::test(flavor = "multi_thread", worker_threads = 2)] -async fn compiled_daemon_routes_persists_reloads_and_hands_off() -> Result<()> { - cellar::asyncrt::set_host_handle(tokio::runtime::Handle::current()); - let root = tempfile::tempdir()?; - // Freeze the compiled artifact: another Cargo feature build can replace - // target/debug binaries while a process test is still exercising reload. - let executable = root.path().join("compiled-host"); - std::fs::copy(env!("CARGO_BIN_EXE_cellar-native-counter"), &executable)?; - let client = Client::builder() - .no_proxy() - .timeout(Duration::from_secs(15)) - .build()?; - // Embedded applications use the same listener and Actor without a bucket. - let mut local = Node::start(root.path(), "native-local", None).await?; - local.ready(&client).await?; - assert_eq!( - counter(&client, &local, "alice", true).await?, - (1, "v1".into()) - ); - assert_eq!(counter(&client, &local, "bob", false).await?.0, 0); - assert_eq!(counter(&client, &local, "alice", false).await?.0, 1); - assert_eq!( - client - .post(format!("{}/reload", local.internal)) - .send() - .await? - .status(), - 503 - ); - local.stop(&client).await?; - - // A real local development store backs leases and LTX for two processes. - // This proves production protocol integration, not cloud failure coverage. - let database = root.path().join("bucket.sqlite"); - let bucket = Bucket::open_dev(&database)?; - deployment(&bucket, "v1", &executable).await?; - let pointer = cellar::fleet::read_current_pointer(&bucket).await?; - let manifest: cellar::protocol::Manifest = serde_json::from_slice( - &bucket - .get(&format!("{}/manifest.json", pointer.prefix)) - .await? - .unwrap() - .0, - )?; - let package = cellar::native_deploy::Package { - primary: manifest, - cohosted: Vec::new(), - }; - let installed = root.path().join("native-node"); - package - .install(&bucket, cellar::native_deploy::current_target(), &installed) - .await?; - let installed = installed.to_str().unwrap(); - let mut first = Node::start_program( - root.path(), - "native-first", - Some(&database), - installed, - None, - ) - .await?; - let mut second = Node::start_program( - root.path(), - "native-second", - Some(&database), - installed, - None, - ) - .await?; - first.ready(&client).await?; - second.ready(&client).await?; - for node in [&first, &second] { - let response = client - .get(format!("{}/", node.ingress)) - .send() - .await? - .error_for_status()?; - assert_eq!(response.headers()["x-asset-version"], "v1"); - assert_eq!(response.text().await?, "native assets v1"); - let response = client - .get(format!("{}/assets-binding/", node.ingress)) - .send() - .await? - .error_for_status()?; - assert_eq!(response.headers()["x-native-version"], "v1"); - assert_eq!(response.text().await?, "native assets v1"); - let response = client - .get(format!("{}/counters/static.txt", node.ingress)) - .send() - .await? - .error_for_status()?; - assert!(response.headers().get("x-native-version").is_none()); - assert_eq!(response.text().await?, "static route exclusion"); - let response = client - .get(format!("{}/missing-navigation", node.ingress)) - .header("sec-fetch-mode", "navigate") - .send() - .await? - .error_for_status()?; - assert_eq!(response.text().await?, "native assets v1"); - let redirect_client = Client::builder() - .no_proxy() - .redirect(reqwest::redirect::Policy::none()) - .build()?; - let response = redirect_client - .get(format!("{}/old", node.ingress)) - .send() - .await?; - assert_eq!(response.status(), 302); - assert_eq!(response.headers()["location"], "/"); - } - assert_eq!(counter(&client, &first, "tenant-acme", true).await?.0, 1); - assert_eq!(counter(&client, &second, "tenant-acme", true).await?.0, 2); - assert_eq!(counter(&client, &first, "tenant-acme", false).await?.0, 2); - - // Updating assets changes the deployment without compiling new code or - // requiring another application registration. Business state stays intact. - let asset_revision = deployment_with_assets(&bucket, "v1", "v1-assets", &executable).await?; - for node in [&first, &second] { - let outcome: Value = client - .post(format!("{}/reload", node.internal)) - .send() - .await? - .error_for_status()? - .json() - .await?; - assert_eq!(outcome["version"], asset_revision); - let response = client - .get(format!("{}/assets-binding/", node.ingress)) - .send() - .await? - .error_for_status()?; - assert_eq!(response.headers()["x-native-version"], "v1"); - assert_eq!(response.headers()["x-asset-version"], "v1-assets"); - assert_eq!(response.text().await?, "native assets v1-assets"); - assert_eq!( - counter(&client, node, "tenant-acme", false).await?, - (2, "v1".into()) - ); - } - - let v2_deployment = deployment(&bucket, "v2", &executable).await?; - // Asset version-skew fallback serves a new content path even while the - // native application still executes its boot generation. - let deadline = Instant::now() + Duration::from_secs(15); - loop { - let response = client - .get(format!("{}/v2-only.txt", first.ingress)) - .send() - .await?; - if response.status().is_success() && response.text().await? == "only v2" { - break; - } - anyhow::ensure!( - Instant::now() < deadline, - "native asset fallback did not refresh" - ); - tokio::time::sleep(Duration::from_millis(50)).await; - } - assert_eq!( - counter(&client, &first, "tenant-acme", false).await?.1, - "v1" - ); - for node in [&first, &second] { - let outcome: Value = client - .post(format!("{}/reload", node.internal)) - .send() - .await? - .error_for_status()? - .json() - .await?; - assert_eq!(outcome["outcome"], "adopted"); - assert_eq!(outcome["version"], v2_deployment); - let response = client - .get(format!("{}/assets-binding/", node.ingress)) - .send() - .await? - .error_for_status()?; - assert_eq!(response.headers()["x-native-version"], "v2"); - assert_eq!(response.headers()["x-asset-version"], "v2"); - assert_eq!(response.text().await?, "native assets v2"); - assert_eq!( - counter(&client, node, "tenant-acme", false).await?, - (2, "v2".into()) - ); - } - let instance = client - .get(format!("{}/counters/tenant-acme", first.ingress)) - .send() - .await? - .error_for_status()? - .headers()["x-native-instance"] - .to_str()? - .to_string(); - client - .post(format!("{}/reload", first.internal)) - .send() - .await? - .error_for_status()?; - let fresh = client - .get(format!("{}/counters/tenant-acme", first.ingress)) - .send() - .await? - .error_for_status()? - .headers()["x-native-instance"] - .to_str()? - .to_string(); - assert_ne!( - instance, fresh, - "forced reload must reconstruct compiled application state" - ); - // A declared but incompatible executable must not adopt, even when its - // application code version is registered in the current process. - let mut artifact = cellar::native_deploy::executable_identity( - &executable, - cellar::native_deploy::current_target(), - )?; - artifact.target = "different-target".into(); - let config: cellar::native_deploy::ApplicationConfig = serde_json::from_value(json!({ - "script_name": "native-counter", "application_version": "v2", "cells": ["Counter"], - "bindings": [{"type": "durable_object_namespace", "name": "COUNTERS", "class_name": "Counter"}], - }))?; - cellar::native_deploy::Package { - primary: config.manifest(vec![artifact.clone()])?, - cohosted: Vec::new(), - } - .publish( - &bucket, - &std::collections::BTreeMap::from([(artifact.sha256, executable.clone())]), - ) - .await?; - for node in [&first, &second] { - let response = client - .post(format!("{}/reload", node.internal)) - .send() - .await?; - assert_eq!(response.status(), 422); - let failure: Value = response.json().await?; - assert!(failure["error"] - .as_str() - .unwrap() - .contains("running executable is not declared")); - assert_eq!( - counter(&client, node, "tenant-acme", false).await?, - (2, "v2".into()) - ); - } - // Unknown compiled versions fail before adoption and leave v2 serving. - deployment(&bucket, "v3", &executable).await?; - for node in [&first, &second] { - let response = client - .post(format!("{}/reload", node.internal)) - .send() - .await?; - assert_eq!(response.status(), 422); - let failure: Value = response.json().await?; - assert!(failure["error"] - .as_str() - .unwrap() - .contains("no compiled application registered")); - assert_eq!( - counter(&client, node, "tenant-acme", false).await?, - (2, "v2".into()) - ); - } - // Move ownership through the real graceful drain. The surviving process - // restores the committed SQLite state from the retained replication path. - first.stop(&client).await?; - assert_eq!( - counter(&client, &second, "tenant-acme", true).await?, - (3, "v2".into()) - ); - second.stop(&client).await?; - deployment(&bucket, "v2", &executable).await?; - let mut restarted = Node::start_program( - root.path(), - "native-restarted", - Some(&database), - installed, - None, - ) - .await?; - restarted.ready(&client).await?; - assert_eq!( - counter(&client, &restarted, "tenant-acme", false).await?, - (3, "v2".into()) - ); - restarted.stop(&client).await?; - Ok(()) +async fn compiled_daemon_routes_persists_reloads_and_hands_off() -> anyhow::Result<()> { + Box::pin(counter::run(counter::DurabilityMode::Bucket)).await } diff --git a/crates/cellar/tests/native_raft_daemon.rs b/crates/cellar/tests/native_raft_daemon.rs index 80b871569..378bc751a 100644 --- a/crates/cellar/tests/native_raft_daemon.rs +++ b/crates/cellar/tests/native_raft_daemon.rs @@ -2,16 +2,30 @@ //! Production process startup, retained-history resume and refusal boundaries. #![allow(clippy::disallowed_methods)] +#[path = "support/raft_daemon_archive.rs"] +mod archive; +#[path = "support/cell_group_archive_verifier.rs"] +mod archive_verifier; +#[path = "../../../examples/raft-counter/configuration.rs"] +#[allow(dead_code)] +mod configuration; #[path = "../../../examples/raft-counter/counter.rs"] #[allow(dead_code)] mod counter; -#[path = "support/html.rs"] +#[path = "support/raft_daemon_docker.rs"] +mod docker; +#[path = "support/raft_daemon_managed.rs"] +mod managed; +#[path = "support/cell_group_disaster_public_node.rs"] #[allow(dead_code)] -mod graph; +mod public_node; #[path = "support/raft_daemon_recovery.rs"] mod recovery; +#[path = "support/raft_retained_disk.rs"] +mod retained_disk; use anyhow::{ensure, Context as _, Result}; use cellar::sdk::{consensus::*, *}; +use public_node::graph; use serde_json::Value; use std::{ fs::File, @@ -27,127 +41,198 @@ use std::{ const SECRET: &str = "fixture-secret-raft-cli"; fn child(root: PathBuf, mode: String) -> Result<()> { + if mode == "provision-sql" { + return Host::local(graph::graph("raft-cli", "", vec![])) + .register("raft-cli", "test", || { + let class = counter::definition()?; + let api = counter::CounterApi::new( + SECRET.into(), + ClusterId::new([1; 16])?, + NamespaceId::new([2; 16])?, + &class, + )?; + counter::application(api, class) + })? + .run(); + } + if mode.starts_with("cluster-") { + return managed::child(root, &mode); + } let class = counter::definition()?; let namespace = NamespaceId::new([2; 16])?; - let api = counter::CounterApi::new(SECRET.into(), namespace, &class)?; + let api = counter::CounterApi::new(SECRET.into(), ClusterId::new([1; 16])?, namespace, &class)?; let serving = api.clone(); let prepared_root = root.clone(); - let verify_join = mode == "bind-fail"; + let cluster = ClusterId::new([1; 16])?; + let history = configuration::group(cluster, namespace)?; + let local_member = MemberId::new(1)?; + let bind_fail = mode == "bind-fail"; + let cancel_bind = mode == "cancel-bind"; + let verify_join = bind_fail || cancel_bind; + let binding_root = root.clone(); let verify_directory = root.join("disk/raft"); let factories = AtomicUsize::new(0); let result = Host::local(graph::graph("raft-cli", "", vec![])) .register("raft-cli", "test", move || { // Exactly one application instance must serve HTTP and register Raft. - ensure!(factories.fetch_add(1, Ordering::SeqCst) == 0, "Raft startup constructed a second application instance"); + ensure!( + factories.fetch_add(1, Ordering::SeqCst) == 0, + "Raft startup constructed a second application instance" + ); std::fs::write(root.join("factory-called"), b"1")?; counter::application(serving.clone(), counter::definition()?) })? - .raft_cells(if mode == "missing-app" { "absent" } else { "raft-cli" }, move |node| async move { - std::fs::write(prepared_root.join("prepared"), node.data_dir.as_os_str().as_encoded_bytes())?; - if mode == "cancel-prepare" { - std::future::pending::<()>().await; - } - let control = tokio::net::TcpListener::bind("127.0.0.1:0").await?; - let data = tokio::net::TcpListener::bind("127.0.0.1:0").await?; - let query = tokio::net::TcpListener::bind("127.0.0.1:0").await?; - let resume = mode.starts_with("cluster-resume-"); - let ordinal = mode.strip_prefix("cluster-resume-").or_else(|| mode.strip_prefix("cluster-")).map(str::parse::).transpose()?; - let index = ordinal.unwrap_or(0); - ensure!(index < 3, "invalid daemon fixture member"); - let id = NodeId::new([index + 3; 16])?; - let disk = DiskLineage::new([index + 4; 16])?; - let mut process = [0; 16]; - process[..4].copy_from_slice(&std::process::id().to_be_bytes()); - let process = ProcessGeneration::new(process)?; - let peer = RaftPeerNode { - node: id, disk, process, label: node.node, - server_name: "localhost".into(), - control_address: control.local_addr()?, - data_address: data.local_addr()?, - query_address: query.local_addr()?, - }; - let mut peers = if ordinal.is_some() { - std::fs::write(prepared_root.join(format!("peer-{index}.tmp")), serde_json::to_vec(&serde_json::json!({ - "pid": std::process::id(), "label": peer.label, - "control": peer.control_address, "data": peer.data_address, "query": peer.query_address, - }))?)?; - std::fs::rename(prepared_root.join(format!("peer-{index}.tmp")), prepared_root.join(format!("peer-{index}.json")))?; - let deadline = Instant::now() + Duration::from_secs(15); - while !(0..3).all(|i| prepared_root.join(format!("peer-{i}.json")).exists()) { - ensure!(Instant::now() < deadline, "peer startup rendezvous expired"); - cellar::asyncrt::sleep(Duration::from_millis(25)).await; + .raft_cells_with_startup( + if mode == "missing-app" { + "absent" + } else { + "raft-cli" + }, + move |node| async move { + std::fs::write( + prepared_root.join("prepared"), + node.data_dir.as_os_str().as_encoded_bytes(), + )?; + if mode == "cancel-prepare" { + std::future::pending::<()>().await; } - (0..3u8).map(|i| -> Result<_> { - let file: Value = serde_json::from_slice(&std::fs::read(prepared_root.join(format!("peer-{i}.json")))?)?; - let mut generation = [0; 16]; - generation[..4].copy_from_slice(&u32::try_from(file["pid"].as_u64().context("pid missing")?)?.to_be_bytes()); - Ok(RaftPeerNode { - node: NodeId::new([i + 3; 16])?, disk: DiskLineage::new([i + 4; 16])?, process: ProcessGeneration::new(generation)?, - label: file["label"].as_str().context("label missing")?.into(), server_name: "localhost".into(), - control_address: serde_json::from_value(file["control"].clone())?, - data_address: serde_json::from_value(file["data"].clone())?, - query_address: serde_json::from_value(file["query"].clone())?, - }) - }).collect::>>()? - } else { vec![peer] }; - if mode == "missing-peer-auth" { - let mut remote = peers[0].clone(); - remote.node = NodeId::new([42; 16])?; - remote.label = "unreachable-fixture-peer".into(); - peers.push(remote); - } - let cluster = ClusterId::new([1; 16])?; - let mut config = NamespaceConfig::new(cluster, namespace, NamespaceMode::Raft, - if mode == "bad-schema" { [99; 32] } else { class.schema_digest() }); - config.voters = if ordinal.is_some() { VoterPolicy::ThreeVoters } else { VoterPolicy::SingleVoterDevelopment }; - let cells = ["one", "two"].into_iter().enumerate().map(|(i, name)| -> Result<_> { - Ok(RaftCellRegistration { - class: "Counter".into(), placement: RaftCellPlacement::Replica, - namespace: config.clone(), - birth: BirthProposal { - scope: GroupScope { cluster, namespace, cell: format!("Counter:{name}"), - group: GroupId::new([i as u8 + 5; 16])?, incarnation: IncarnationId::new([7; 16])? }, - voters: peers.iter().enumerate().map(|(i, peer)| Ok(InitialMember { node: peer.node, member: MemberId::new(i as u64 + 1)? })).collect::>>()?, learners: vec![], + let control = tokio::net::TcpListener::bind("127.0.0.1:0").await?; + let data = tokio::net::TcpListener::bind("127.0.0.1:0").await?; + let query = tokio::net::TcpListener::bind("127.0.0.1:0").await?; + let id = NodeId::new([3; 16])?; + let disk = DiskLineage::new([4; 16])?; + let mut process = [0; 16]; + process[..4].copy_from_slice(&std::process::id().to_be_bytes()); + let process = ProcessGeneration::new(process)?; + let peer = RaftPeerNode { + node: id, + disk, + process, + label: node.node, + server_name: "localhost".into(), + control_address: control.local_addr()?, + data_address: data.local_addr()?, + query_address: query.local_addr()?, + }; + let mut peers = vec![peer]; + if mode == "missing-peer-auth" { + let mut remote = peers[0].clone(); + remote.node = NodeId::new([42; 16])?; + remote.label = "unreachable-fixture-peer".into(); + peers.push(remote); + } + let mut namespace = NamespaceConfig::new( + cluster, + namespace, + NamespaceMode::Raft, + if mode == "bad-schema" { + [99; 32] + } else { + class.schema_digest() }, - }) - }).collect::>>()?; - let log_mode = if resume { - LogOpenMode::Resume - } else if ordinal.is_some() || mode == "create" || mode == "bad-schema" || mode == "bind-fail" { - LogOpenMode::Create - } else { LogOpenMode::Resume }; - Ok(NativeRaftCells { - config: RaftCellsConfig { - infrastructure: NodeRaftConfig { - log: ReplicaLogOptions::new(node.data_dir.join("raft"), id, disk, process, log_mode), - consensus: RaftRuntimeOptions { tick_interval_ms: Some(25), ..Default::default() }, - transport: TransportOptions::default(), + ); + namespace.voters = VoterPolicy::SingleVoterDevelopment; + let log_mode = + if mode == "create" || mode == "bad-schema" || bind_fail || cancel_bind { + LogOpenMode::Create + } else { + LogOpenMode::Resume + }; + let initialize = if log_mode == LogOpenMode::Create { + Some(RaftNamespaceInitialization { + operation: ProposalId::new([11; 16])?, + groups: vec![CellGroupBirthSpec { + history, + range: CellRange::all(), + epoch: RouteEpoch::new(1)?, + creation: ProposalId::new([12; 16])?, + voters: peers + .iter() + .enumerate() + .map(|(i, peer)| { + Ok(InitialMember { + node: peer.node, + member: MemberId::new(i as u64 + 1)?, + }) + }) + .collect::>>()?, + learners: vec![], + }], + }) + } else { + None + }; + let mut shared = configuration::limits(); + shared.archive = None; + let mut log = + ReplicaLogOptions::new(node.data_dir.join("raft"), id, disk, process, log_mode); + log.max_groups = shared.controller.max_groups; + Ok(NativeRaftCells { + config: RaftCellsConfig { + infrastructure: NodeRaftConfig { + log, + consensus: RaftRuntimeOptions { + tick_interval_ms: Some(25), + max_groups: shared.controller.max_groups, + ..Default::default() + }, + transport: TransportOptions::default(), + }, + catalog: CatalogOptions { + prefix: "raft-cli".into(), + ..Default::default() + }, + namespaces: vec![RaftNamespaceConfig { + namespace, + class: "Counter".into(), + initialize, + }], + groups: vec![RaftGroupAssignment { + history, + placement: RaftGroupPlacement::Replica, + }], + peers, + query_origins: vec![], + command_origins: vec![], + host: RaftHostOptions { + max_groups: shared.controller.max_groups, + max_input_bytes: shared.dispatch.max_input_bytes, + ..Default::default() + }, + shared, }, - catalog: CatalogOptions { prefix: "raft-cli".into(), ..Default::default() }, - cells, peers, query_origins: vec![], host: RaftHostOptions { - node_directory: ordinal.map(|_| RaftNodeDirectoryOptions::default()), - ..Default::default() + listeners: NodeRaftListeners { + control, + data, + query, }, - }, - listeners: NodeRaftListeners { control, data, query }, - tls: RaftTls::new( - vec![include_bytes!("../transport/tcp/fixtures/ca.der").to_vec().into()], - vec![include_bytes!("../transport/tcp/fixtures/server.der").to_vec().into()], - rustls::pki_types::PrivatePkcs8KeyDer::from(include_bytes!("../transport/tcp/fixtures/key.der").to_vec()).into(), - )?, - store: cellar::bucket::Bucket::open_dev(&prepared_root.join("catalog.sqlite"))?.raft_store(), - versions: VersionKeys::new(VersionSigningKey::new([29; 32], u64::MAX)?, vec![], Duration::from_secs(86400))?, - }) - }, move |service| { - if std::env::var("CELLAR_RAFT_CLI_CHILD").as_deref() == Ok("bind-fail") { - anyhow::bail!("fixture binding failed"); - } - *api.cells.lock().map_err(|_| anyhow::anyhow!("binding poisoned"))? = Some(service); - Ok(()) - })? + tls: fixture_tls()?, + store: cellar::bucket::Bucket::open_dev(&prepared_root.join("catalog.sqlite"))? + .raft_store(), + versions: fixture_versions()?, + }) + }, + move |service| async move { + ready_counter(&service, history, local_member).await?; + if cancel_bind { + std::fs::write(binding_root.join("binding"), b"entered")?; + std::future::pending::<()>().await; + } + if bind_fail { + anyhow::bail!("fixture binding failed"); + } + *api.cells + .lock() + .map_err(|_| anyhow::anyhow!("binding poisoned"))? = Some(service); + Ok(()) + }, + )? .run_until(std::future::pending()); if verify_join { - ensure!(result.is_err(), "failed binding admitted a node"); + ensure!( + result.is_err(), + "failed or cancelled binding admitted a node" + ); // Stay in this process after the daemon returned. A surviving storage // worker/owner would still hold the engine lock and refuse this open. tokio::runtime::Builder::new_current_thread() @@ -175,20 +260,133 @@ fn child(root: PathBuf, mode: String) -> Result<()> { domain.shutdown().await; opened })?; - cellar::note!("failed Raft binding physically joined before daemon returned"); + if bind_fail { + cellar::note!("failed Raft binding physically joined before daemon returned"); + } else { + cellar::note!("cancelled Raft binding physically joined before daemon returned"); + } } result } +fn fixture_tls() -> Result { + Ok(RaftTls::new( + vec![include_bytes!("../transport/tcp/fixtures/ca.der") + .to_vec() + .into()], + vec![include_bytes!("../transport/tcp/fixtures/server.der") + .to_vec() + .into()], + rustls::pki_types::PrivatePkcs8KeyDer::from( + include_bytes!("../transport/tcp/fixtures/key.der").to_vec(), + ) + .into(), + )?) +} + +fn fixture_versions() -> Result { + Ok(VersionKeys::new( + VersionSigningKey::new([29; 32], u64::MAX)?, + vec![], + Duration::from_secs(86400), + )?) +} + +async fn ready_counter(service: &RaftCells, history: GroupHistory, local: MemberId) -> Result<()> { + let group = service.group(history)?; + let deadline = Instant::now() + Duration::from_secs(15); + loop { + let usage = service.resource_usage(); + ensure!( + usage.host.controller.groups == 1, + "daemon must host one shared group" + ); + let remaining = deadline.saturating_duration_since(Instant::now()); + let status = group.status().await?; + ensure!( + !remaining.is_zero(), + "replicated enrollment did not finish for member {local:?}: {status:?}, cells={:?}", + usage.retained_manifest_cells + ); + if usage.retained_manifest_cells == Some(2) { + group.quorum_membership(remaining).await?; + return Ok(()); + } + if status.ready && status.leader == Some(local.get()) { + for (index, name) in ["one", "two"].into_iter().enumerate() { + let handle = + service.cell(counter::history(history.cluster, history.namespace, name)?)?; + let request = CellEnrollmentRequest { + creation: ProposalId::new([31 + index as u8; 16])?, + durability: Durability::Quorum, + }; + let remaining = deadline.saturating_duration_since(Instant::now()); + match handle.enroll(request, remaining).await { + Ok(_) => (), + Err(error) + if !error.ambiguous + && matches!( + error.cause.as_ref(), + CellDispatchError::Runtime( + RuntimeError::NotLeader | RuntimeError::NotReady + ) + ) => + { + break + } + Err(error) => return Err(error.into()), + } + } + } + cellar::asyncrt::sleep(Duration::from_millis(20)).await; + } +} + +#[derive(Clone)] +struct ManagedIngress { + native: String, + internal: String, + axum: String, +} + +#[derive(Clone, Copy)] +enum Ingress { + Native, + Axum, +} + +impl Ingress { + const ALL: [Self; 2] = [Self::Native, Self::Axum]; + + fn label(self) -> &'static str { + match self { + Self::Native => "native", + Self::Axum => "axum", + } + } + + fn endpoint(self, address: &ManagedIngress) -> &str { + match self { + Self::Native => &address.native, + Self::Axum => &address.axum, + } + } +} + struct Process { child: Child, + voter: Option, log: PathBuf, diagnostics: PathBuf, } impl Drop for Process { fn drop(&mut self) { - if self.child.try_wait().ok().flatten().is_none() { + if let Some(voter) = &self.voter { + if voter.kill_if_running().is_err() { + let _ = self.child.kill(); + } + } else if self.child.try_wait().ok().flatten().is_none() { let _ = self.child.kill(); } let _ = self.child.wait(); @@ -206,7 +404,7 @@ impl Process { let diagnostics = root.join(format!("{mode}-stderr.log")); // A new process generation keeps the configured physical node label. // The authenticated directory refreshes its new addresses and boot. - let label = mode.strip_prefix("cluster-resume-").map_or_else( + let label = managed::retained_index(mode).map_or_else( || format!("raft-cli-{mode}"), |index| format!("raft-cli-cluster-{index}"), ); @@ -227,23 +425,49 @@ impl Process { "--internal-listen", "127.0.0.1:0", ]) - .args(data_args) - .stdout(Stdio::from(File::create(&log)?)) - .stderr(Stdio::from(File::create(&diagnostics)?)); + .args(data_args); if mode.starts_with("cluster-") { cmd.env("CELLAR_TEST_BUCKET", "cellar-dev") .env("CELLAR_INTERNAL_DEV_STORE", root.join("fleet.sqlite")); + if std::env::var_os(managed::RUSTFS_ENDPOINT).is_some() { + for key in managed::RUSTFS_ENV { + cmd.env( + key, + std::env::var_os(key).with_context(|| format!("missing {key}"))?, + ); + } + } } if override_dir { cmd.env("CELLAR_TEST_DATA_DIR", root.join("override")); } + let stdout = File::create(&log)?; + let stderr = File::create(&diagnostics)?; + let (child, voter) = if mode.starts_with("cluster-") && docker::enabled() { + let voter = docker::Voter::create(root, mode, &cmd)?; + (voter.start(stdout, stderr)?, Some(voter)) + } else { + ( + cmd.stdout(Stdio::from(stdout)) + .stderr(Stdio::from(stderr)) + .spawn()?, + None, + ) + }; Ok(Self { - child: cmd.spawn()?, + child, + voter, log, diagnostics, }) } + fn pid(&self) -> Result { + self.voter + .as_ref() + .map_or_else(|| Ok(u64::from(self.child.id())), docker::Voter::pid) + } + fn logs(&self) -> Result { Ok(format!( "{}\n{}", @@ -276,17 +500,13 @@ impl Process { let deadline = Instant::now() + Duration::from_secs(30); loop { let logs = self.logs()?; - let addr = |prefix| { - logs.lines().find_map(|line| { - line.strip_prefix(prefix) - .and_then(|s| s.split_whitespace().next()) - .map(|s| format!("http://{s}")) - }) - }; if let (Some(public), Some(internal)) = ( - addr("cellar listening on "), - addr("cellar internal listening on "), + listener_address(&logs, "cellar listening on "), + listener_address(&logs, "cellar internal listening on "), ) { + if let Some(voter) = &self.voter { + voter.running()?; + } return Ok((public, internal)); } ensure!( @@ -296,6 +516,34 @@ impl Process { tokio::time::sleep(Duration::from_millis(25)).await; } } + + async fn managed_addresses(&mut self) -> Result { + let (native, internal) = self.addresses().await?; + let deadline = Instant::now() + Duration::from_secs(30); + loop { + let logs = self.logs()?; + if let Some(axum) = listener_address(&logs, "cellar axum listening on ") { + return Ok(ManagedIngress { + native, + internal, + axum, + }); + } + ensure!( + self.child.try_wait()?.is_none() && Instant::now() < deadline, + "Axum startup failed\n{logs}" + ); + tokio::time::sleep(Duration::from_millis(25)).await; + } + } +} + +fn listener_address(logs: &str, prefix: &str) -> Option { + logs.lines().find_map(|line| { + line.strip_prefix(prefix) + .and_then(|value| value.split_whitespace().next()) + .map(|address| format!("http://{address}")) + }) } async fn qualify() -> Result<()> { @@ -341,7 +589,7 @@ async fn qualify() -> Result<()> { } for (mode, expected) in [ ("missing-app", "absent from the active deployment"), - ("bad-schema", "mode/schema disagrees"), + ("bad-schema", "compiled namespace schema mismatch"), ( "missing-peer-auth", "requires a fleet bucket for shared peer authentication", @@ -472,57 +720,78 @@ async fn qualify() -> Result<()> { ensure!( report["raft"]["owners_closed"] == true && report["raft"]["startup_incomplete"] == false - && report["raft"]["groups"] == 2 + && report["raft"]["groups"] == 1 && report["raft"]["retain_replica_disks"] == true, "wrong close proof: {report}" ); } - let root = tempfile::tempdir()?; - let disk = root.path().join("disk"); - let mut process = Process::spawn( - root.path(), - "cancel-prepare", - &[std::ffi::OsStr::new("--data-dir"), disk.as_os_str()], - false, - )?; - let deadline = Instant::now() + Duration::from_secs(20); - while !root.path().join("prepared").exists() { + for (mode, marker) in [("cancel-prepare", "prepared"), ("cancel-bind", "binding")] { + let root = tempfile::tempdir()?; + let disk = root.path().join("disk"); + let mut process = Process::spawn( + root.path(), + mode, + &[std::ffi::OsStr::new("--data-dir"), disk.as_os_str()], + false, + )?; + let deadline = Instant::now() + Duration::from_secs(20); + while !root.path().join(marker).exists() { + ensure!( + process.child.try_wait()?.is_none() && Instant::now() < deadline, + "{mode} did not start\n{}", + process.logs()? + ); + tokio::time::sleep(Duration::from_millis(25)).await; + } + ensure!(Command::new("/bin/kill") + .args(["-TERM", &process.child.id().to_string()]) + .status()? + .success()); + let logs = process.exited(false).await?; ensure!( - process.child.try_wait()?.is_none() && Instant::now() < deadline, - "prepare did not start\n{}", - process.logs()? + logs.contains("startup interrupted by SIGTERM") + && !logs.contains("cellar listening on "), + "{logs}" ); - tokio::time::sleep(Duration::from_millis(25)).await; + if mode == "cancel-bind" { + ensure!( + logs.contains("cancelled Raft binding physically joined before daemon returned"), + "{logs}" + ); + } } - ensure!(Command::new("/bin/kill") - .args(["-TERM", &process.child.id().to_string()]) - .status()? - .success()); - let logs = process.exited(false).await?; - ensure!( - logs.contains("startup interrupted by SIGTERM") && !logs.contains("cellar listening on "), - "{logs}" - ); // Three separate production CLI processes use a shared catalog and peer - // authentication key, two independent groups and real TLS replication. - let root = tempfile::tempdir()?; + // authentication key, one shared group with two independent cells and real TLS replication. + let mut root = tempfile::tempdir()?; + root.disable_cleanup(docker::enabled()); // The real dev supervisor initializes its local store before starting // children. Concurrent first-open WAL/schema setup is not peer readiness. cellar::bucket::Bucket::open_dev(&root.path().join("fleet.sqlite"))?; - cellar::bucket::Bucket::open_dev(&root.path().join("catalog.sqlite"))?; + managed::initialize(root.path()).await?; let mut cluster = Vec::new(); - for i in 0..3 { - let disk = root.path().join(format!("disk-{i}")); - cluster.push(Process::spawn( + for i in 0u8..3 { + let disk = docker::disk(root.path(), usize::from(i)); + let mut process = Process::spawn( root.path(), &format!("cluster-{i}"), &[std::ffi::OsStr::new("--data-dir"), disk.as_os_str()], false, - )?); + )?; + // Serialize the independent boot CAS operations by observing the + // rooted engine. The operator checks each live standby over peer RPC. + managed::wait_for_cold_boot(root.path(), i, &mut process).await?; + if i < 2 { + ensure!( + !process.logs()?.contains("cellar listening on ") + && !process.logs()?.contains("cellar axum listening on "), + "standby opened public ingress" + ); + } + cluster.push(process); } let mut addresses = Vec::new(); for index in 0..cluster.len() { - match cluster[index].addresses().await { + match cluster[index].managed_addresses().await { Ok(address) => addresses.push(address), Err(error) => { let logs = cluster @@ -536,9 +805,14 @@ async fn qualify() -> Result<()> { } } } - let mut ledger = recovery::Ledger::new(root.path())?; + let ledger_backend = if std::env::var_os(managed::RUSTFS_ENDPOINT).is_some() { + "rustfs_catalog_and_archives" + } else { + "shared_dev_sqlite" + }; + let mut ledger = recovery::Ledger::new(root.path(), ledger_backend)?; let mut acknowledged = Vec::new(); - for name in ["one", "two"] { + for (name, ingress) in [("one", Ingress::Native), ("two", Ingress::Axum)] { let command = format!("{}:1", hex::encode([90; 16])); let retry = (std::time::SystemTime::now() .duration_since(std::time::UNIX_EPOCH)? @@ -546,12 +820,14 @@ async fn qualify() -> Result<()> { + 60_000) .to_string(); let deadline = Instant::now() + Duration::from_secs(20); - let (leader, receipt) = loop { + let (ingress_member, receipt) = loop { let mut committed = None; - for (index, (public, _)) in addresses.iter().enumerate() { + for (index, address) in addresses.iter().enumerate() { + let public = ingress.endpoint(address); ledger.record(serde_json::json!({ "phase": "send", "cell": name, "command": command, "retry_until": retry, "delta": 11, "member": index + 1, + "ingress": ingress.label(), }))?; let response = client .post(format!("{public}/counters/{name}?delta=11")) @@ -581,42 +857,70 @@ async fn qualify() -> Result<()> { ledger.record(serde_json::json!({ "phase": "acknowledged", "cell": name, "command": command, "retry_until": retry, "delta": 11, "receipt": receipt, + "ingress": ingress.label(), }))?; - for (index, (public, _)) in addresses + for (index, address) in addresses .iter() .enumerate() - .filter(|(index, _)| *index != leader) + .filter(|(index, _)| *index != ingress_member) { - let query: Value = client - .get(format!("{public}/counters/{name}")) - .bearer_auth(SECRET) - .header("x-read-consistency", "at-least") - .header( - "x-min-version", - receipt["version_token"] - .as_str() - .context("version token absent")?, - ) - .send() - .await? - .error_for_status()? - .json() - .await?; - ensure!( - query["value"] == 11 - && query["served_by"] == index as u64 + 1 - && query["group"] == receipt["group"], - "follower query failed: {query}" - ); + for ingress in Ingress::ALL { + let public = ingress.endpoint(address); + let query: Value = client + .get(format!("{public}/counters/{name}")) + .bearer_auth(SECRET) + .header("x-read-consistency", "at-least") + .header( + "x-min-version", + receipt["version_token"] + .as_str() + .context("version token absent")?, + ) + .send() + .await? + .error_for_status()? + .json() + .await?; + ensure!( + query["value"] == 11 + && query["served_by"] == index as u64 + 1 + && query["group"] == receipt["group"], + "follower query failed: {query}" + ); + } } acknowledged.push(recovery::Acknowledged { cell: name, - leader, + ingress, + ingress_member, command, retry_until: retry, receipt, }); } + ensure!( + acknowledged[0].receipt["group"] == acknowledged[1].receipt["group"] + && acknowledged[0].receipt["group_incarnation"] + == acknowledged[1].receipt["group_incarnation"] + && acknowledged[0].receipt["cell"] != acknowledged[1].receipt["cell"], + "two independent cell histories must share the physical group" + ); + managed::verify_park_wake( + root.path(), + &client, + &addresses, + &acknowledged[0].receipt, + &mut ledger, + ) + .await?; + archive::verify( + root.path(), + &client, + &addresses[acknowledged[0].ingress_member], + &acknowledged[0].receipt, + &mut ledger, + ) + .await?; recovery::hard_kill_and_resume( root.path(), &client, @@ -626,9 +930,53 @@ async fn qualify() -> Result<()> { &mut ledger, ) .await?; - for (process, (_, internal)) in cluster.iter_mut().zip(&addresses) { + managed::maintenance::qualify( + root.path(), + &client, + &mut cluster, + &mut addresses, + &acknowledged[0], + &mut ledger, + ) + .await?; + let mut members = [1, 2, 3]; + let repair = std::env::var_os(managed::repair::ENABLED).is_some(); + if repair { + let (replacement, ingress) = Box::pin(managed::repair::qualify( + root.path(), + &client, + &mut cluster, + &addresses, + &acknowledged[0].receipt, + &mut ledger, + )) + .await?; + cluster[0] = replacement; + addresses[0] = ingress; + members = managed::repair::REPAIRED; + } else if std::env::var_os(managed::standby::ENABLED).is_some() { + managed::standby::qualify(root.path(), &addresses[0], &mut ledger).await?; + } + let survivor = cluster.len() - 1; + if !repair { + recovery::verify_linearizable_before_loss(&client, &addresses[survivor], &mut ledger) + .await?; + } + let receipt = &acknowledged[0].receipt; + for (index, (process, address)) in cluster.iter_mut().zip(&addresses).enumerate() { + if index == survivor { + recovery::verify_quorum_loss(&client, address, index, &mut ledger).await?; + } + let drain_term = if repair && index == 0 { + let (leader, term, _) = + recovery::observe_members(&client, &addresses, &members, None, 1, receipt).await?; + ensure!(leader == index, "replacement lost leadership before drain"); + Some(term) + } else { + None + }; client - .post(format!("{internal}/shutdown")) + .post(format!("{}/shutdown", address.internal)) .send() .await? .error_for_status()?; @@ -637,8 +985,49 @@ async fn qualify() -> Result<()> { logs.contains("\"owners_closed\":true"), "three-voter owner did not join\n{logs}" ); + if let Some(term) = drain_term { + let report: Value = logs + .lines() + .filter_map(|line| serde_json::from_str::(line).ok()) + .find(|value| value["event"] == "daemon_node_stopped") + .context("replacement drain report missing")?; + let raft = &report["raft"]; + let handoff = &raft["handoff"]; + ensure!( + report["mode"] == "handoff" + && raft["startup_incomplete"] == false + && raft["groups"] == 1 + && raft["retain_replica_disks"] == true + && handoff["groups_examined"] == 1 + && handoff["accepted"] == 1 + && handoff["leadership_released"] == 1 + && handoff["unavailable_groups"] == 0 + && handoff["deadline_elapsed"] == false, + "incomplete replacement drain: {report}" + ); + let next_term = term.checked_add(1).context("Raft term overflow")?; + let (successor, successor_term, voters) = recovery::observe_members( + &client, + &addresses, + &members, + Some(index), + next_term, + receipt, + ) + .await?; + ledger.record(serde_json::json!({ + "phase":"repair_drain_complete", "member":members[index], "term":term, + "successor":members[successor], "successor_term":successor_term, + "handoff":handoff, "voters":voters, + }))?; + recovery::verify_linearizable_before_loss(&client, &addresses[survivor], &mut ledger) + .await?; + } + ledger.record( + serde_json::json!({"phase":"voter_shutdown_joined", "member":members[index]}), + )?; } - cellar::note!("production Raft CLI qualification passed: directory and registration refusals, failed binding joins, two retained groups, linearizable queries, process restart, semantic closure, startup cancellation, and three daemon processes with two groups over TLS and local follower reads"); + cellar::note!("production Raft CLI qualification passed: directory and registration refusals, failed binding joins, two cells in one retained group, linearizable queries, process restart, semantic closure, startup cancellation, and three managed daemon processes with one shared group over TLS with native and Axum reads and writes"); Ok(()) } @@ -651,6 +1040,13 @@ fn main() -> Result<()> { mode, ); } + if std::env::var_os("CELLAR_RAFT_DAEMON_REPAIR_GATE").is_some() { + tracing_subscriber::fmt() + .with_env_filter("cellar_raft_repair=debug") + .with_ansi(false) + .try_init() + .map_err(|error| anyhow::anyhow!("repair trace subscriber: {error}"))?; + } tokio::runtime::Builder::new_multi_thread() .worker_threads(3) .enable_all() diff --git a/crates/cellar/tests/native_raft_directory.rs b/crates/cellar/tests/native_raft_directory.rs index 9fabdc108..4705b474f 100644 --- a/crates/cellar/tests/native_raft_directory.rs +++ b/crates/cellar/tests/native_raft_directory.rs @@ -1,13 +1,13 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. //! Real engine fences and conditional object-store registration. These adapter -//! cases do not qualify controller RPCs or automatic membership repair. +//! tests exercise physical group directory refresh and inspection. #![allow(clippy::disallowed_methods)] #[path = "support/raft_storage_fault.rs"] mod fault; #[path = "support/raft_directory_native.rs"] mod native; -#[path = "support/raft_node.rs"] +#[path = "support/cell_group_disaster_public_node.rs"] #[allow(dead_code)] mod public_node; #[path = "support/raft_catalog_fault.rs"] diff --git a/crates/cellar/tests/native_raft_execution.rs b/crates/cellar/tests/native_raft_execution.rs index 9ef850e5d..704eab5dc 100644 --- a/crates/cellar/tests/native_raft_execution.rs +++ b/crates/cellar/tests/native_raft_execution.rs @@ -656,7 +656,7 @@ async fn registered_command_uses_exact_image_receipt_and_read_only_query() -> Re let class = RaftClass::new("Counter", 1, [22; 32])? .command("Increment", [23; 32], move |context, input| { called.fetch_add(1, Ordering::SeqCst); - assert_eq!(context.identity(), &scope()); + assert_eq!(context.identity(), &cellar_logic::cell_group::CellHistory::try_from(&scope())?); assert_eq!(context.command().sequence(), 1); assert_eq!(context.inputs().now_unix_millis, 10); assert_eq!(context.inputs().random, [17; 32]); @@ -667,8 +667,9 @@ async fn registered_command_uses_exact_image_receipt_and_read_only_query() -> Re Ok(value.to_be_bytes().to_vec()) })? .query("Get", [24; 32], |context, _| { - assert_eq!(context.identity(), &scope()); - assert!(context.version().position.index.get() > 0); + assert_eq!(context.identity(), &cellar_logic::cell_group::CellHistory::try_from(&scope())?); + let cellar::sdk::consensus::QueryVersion::GroupCut(version) = context.version() else { anyhow::bail!("single-cell adapter observation") }; + assert!(version.position.index.get() > 0); for sql in [ "UPDATE counter SET value=999 RETURNING value", "PRAGMA user_version=8", diff --git a/crates/cellar/tests/native_raft_extents.rs b/crates/cellar/tests/native_raft_extents.rs index 822564f26..afc5d96ac 100644 --- a/crates/cellar/tests/native_raft_extents.rs +++ b/crates/cellar/tests/native_raft_extents.rs @@ -2,16 +2,18 @@ //! Public node ownership across cells, closed registrations and joined restart. #![allow(clippy::disallowed_methods)] -#[path = "support/raft_node.rs"] +#[path = "support/cell_group_disaster_public_node.rs"] #[allow(dead_code)] mod public_node; use anyhow::{ensure, Context as _, Result}; -use cellar::sdk::{consensus::*, Node, ShutdownMode}; +use cellar::sdk::{consensus::*, *}; use futures_util::TryStreamExt as _; use object_store::ObjectStore as _; use std::{ collections::BTreeMap, + io::Write as _, + path::Path, sync::{ atomic::{AtomicUsize, Ordering}, Arc, Mutex, @@ -19,52 +21,263 @@ use std::{ time::Duration, }; -fn registrations(schema: [u8; 32]) -> Result> { - let mut namespace = NamespaceConfig::new( - ClusterId::new([1; 16])?, - NamespaceId::new([2; 16])?, - NamespaceMode::Raft, - schema, - ); - namespace.voters = VoterPolicy::SingleVoterDevelopment; - namespace.image_storage = RaftImageStorage::Extents; - (1..=2) - .map(|index| { - Ok(RaftCellRegistration { - namespace: namespace.clone(), - placement: RaftCellPlacement::Replica, - class: "Counter".into(), - birth: BirthProposal { - scope: GroupScope { - cluster: namespace.cluster, - namespace: namespace.namespace, - cell: format!("Counter:{index}"), - group: GroupId::new([index; 16])?, - incarnation: IncarnationId::new([3; 16])?, - }, - voters: vec![InitialMember { - member: MemberId::new(1)?, - node: NodeId::new([1; 16])?, - }], - learners: vec![], +const TIMEOUT: Duration = Duration::from_secs(5); + +struct Group { + namespace: NamespaceConfig, + birth: CellGroupBirthSpec, + cell: CellHistory, + operation: ProposalId, +} + +impl Group { + fn new(schema: [u8; 32], index: u8) -> Result { + let cluster = ClusterId::new([1; 16])?; + let namespace_id = NamespaceId::new([2 + index; 16])?; + let mut namespace = + NamespaceConfig::new(cluster, namespace_id, NamespaceMode::Raft, schema); + namespace.voters = VoterPolicy::SingleVoterDevelopment; + namespace.image_storage = RaftImageStorage::Extents; + Ok(Self { + namespace, + birth: CellGroupBirthSpec { + history: GroupHistory { + cluster, + namespace: namespace_id, + group: GroupId::new([index; 16])?, + incarnation: IncarnationId::new([3; 16])?, }, - }) + range: CellRange::all(), + epoch: RouteEpoch::new(1)?, + creation: ProposalId::new([12 + index; 16])?, + voters: vec![InitialMember { + member: MemberId::new(1)?, + node: NodeId::new([1; 16])?, + }], + learners: vec![], + }, + cell: CellHistory { + cluster, + namespace: namespace_id, + cell: CellId::new(format!("Counter:{index}"))?, + incarnation: IncarnationId::new([4; 16])?, + }, + operation: ProposalId::new([22 + index; 16])?, + }) + } + + fn config(&self, initialize: bool) -> RaftNamespaceConfig { + RaftNamespaceConfig { + namespace: self.namespace.clone(), + class: "Counter".into(), + initialize: initialize.then(|| RaftNamespaceInitialization { + operation: self.operation, + groups: vec![self.birth.clone()], + }), + } + } + + async fn provision( + &self, + root: &Path, + store: Arc, + domain: &ExecutionDomain, + ) -> Result<()> { + let provision = RaftNamespaceProvisioning::open( + domain, + store, + CatalogOptions::default(), + self.config(true), + DirectoryLimits::default(), + CellGroupArchiveLimits::default(), + ) + .await?; + let archive = provision + .prepare_archive(domain, self.birth.history) + .await?; + let mut journal = std::fs::File::create_new(root.join(self.cell.cell.as_str()))?; + journal.write_all(archive.encoded_intent())?; + journal.sync_all()?; + std::fs::File::open(root)?.sync_all()?; + archive.commit(domain).await?; + Ok(()) + } + + async fn ready(&self, service: &RaftCells) -> Result { + let group = service.group(self.birth.history)?; + tokio::time::timeout(TIMEOUT, async { + loop { + if group + .status() + .await + .is_ok_and(|status| status.ready && status.leader == Some(1)) + { + return; + } + tokio::time::sleep(Duration::from_millis(5)).await; + } }) - .collect() + .await?; + Ok(service.cell(self.cell.clone())?) + } +} + +async fn start( + node: &Node, + application: Arc, + store: Arc, + groups: &[Group], + attached: usize, + generation: u8, + mut host: RaftHostOptions, +) -> Result { + let listeners = NodeRaftListeners { + control: tokio::net::TcpListener::bind("127.0.0.1:0").await?, + data: tokio::net::TcpListener::bind("127.0.0.1:0").await?, + query: tokio::net::TcpListener::bind("127.0.0.1:0").await?, + }; + let peer = RaftPeerNode { + node: NodeId::new([1; 16])?, + disk: DiskLineage::new([5; 16])?, + process: ProcessGeneration::new([generation; 16])?, + label: "extent-native".into(), + server_name: "localhost".into(), + control_address: listeners.control.local_addr()?, + data_address: listeners.data.local_addr()?, + query_address: listeners.query.local_addr()?, + }; + let mut log = ReplicaLogOptions::new( + node.info().data_dir.join("raft"), + peer.node, + peer.disk, + peer.process, + if generation == 1 { + LogOpenMode::Create + } else { + LogOpenMode::Resume + }, + ); + log.max_groups = 2; + let mut shared = public_node::limits(); + shared.namespaces.max_namespaces = 2; + if host.extents.is_some() { + shared.controller.checkpoints = Some(CellGroupCheckpointPolicy { + interval_entries: 1, + max_parallel: 1, + }); + shared.checkpoints = Some(CellGroupCheckpointLimits { + max_sources: 2, + max_retained_bytes: 32 * 1024 * 1024, + max_cells: 32, + max_transfer_bytes: 32 * 1024 * 1024, + max_chunk_bytes: 4096, + }); + shared.archive = Some(CellGroupNodeArchiveOptions { + io: Default::default(), + uploads: Default::default(), + tails: Default::default(), + }); + } + host.max_input_bytes = shared.dispatch.max_input_bytes; + node.start_raft_cells( + RaftCellsConfig { + infrastructure: NodeRaftConfig { + log, + consensus: RaftRuntimeOptions { + max_groups: 2, + tick_interval_ms: Some(20), + ..Default::default() + }, + transport: TransportOptions::default(), + }, + catalog: CatalogOptions::default(), + namespaces: groups + .iter() + .map(|group| group.config(host.extents.is_none())) + .collect(), + groups: groups + .iter() + .take(attached) + .map(|group| RaftGroupAssignment { + history: group.birth.history, + placement: RaftGroupPlacement::Replica, + }) + .collect(), + peers: vec![peer], + query_origins: vec![], + command_origins: vec![], + host, + shared, + }, + RaftCellsServices { + listeners, + tls: RaftTls::new( + vec![include_bytes!("../transport/tcp/fixtures/ca.der") + .to_vec() + .into()], + vec![include_bytes!("../transport/tcp/fixtures/server.der") + .to_vec() + .into()], + rustls::pki_types::PrivatePkcs8KeyDer::from( + include_bytes!("../transport/tcp/fixtures/key.der").to_vec(), + ) + .into(), + )?, + store, + application, + versions: VersionKeys::new( + VersionSigningKey::new([9; 32], u64::MAX)?, + vec![], + Duration::from_secs(60), + )?, + recovery: None, + }, + ) + .await +} + +fn counter(callbacks: Arc) -> Result { + Ok(RaftClass::new("Counter", 1, [21; 32])? + .initializer([24; 32], |_, sql| { + sql.execute_batch( + "CREATE TABLE counter(value INTEGER NOT NULL); INSERT INTO counter VALUES(0)", + )?; + Ok(()) + })? + .command("Increment", [22; 32], move |context, input| { + callbacks.record(context.identity()); + let amount = i64::from_be_bytes(input.try_into()?); + context + .sql() + .execute("UPDATE counter SET value=value+?1", [amount])?; + let value: i64 = context + .sql() + .query_row("SELECT value FROM counter", [], |row| row.get(0))?; + Ok(value.to_be_bytes().to_vec()) + })? + .query("Get", [23; 32], |context, _| { + let value: i64 = + context.query_row("SELECT value FROM counter", [], |row| row.get(0))?; + Ok(value.to_be_bytes().to_vec()) + })?) +} + +fn application(class: RaftClass) -> Result> { + Ok(Arc::new( + Registry::new(|_, _| Box::pin(async { Ok(Output::Done) })).raft_class(class)?, + )) } fn host(max_roots: usize) -> RaftHostOptions { RaftHostOptions { max_groups: 2, - checkpoint_entries: 1, + poll_ms: 10, extents: Some(RaftExtentOptions { inventory: RaftExtentLimits { max_roots, max_root_extents: 256, ..Default::default() }, - gc_interval_ms: 10, - gc_batch_files: 2, ..Default::default() }), ..Default::default() @@ -78,20 +291,20 @@ struct Callbacks { } impl Callbacks { - fn record(&self, scope: &GroupScope) { + fn record(&self, scope: &cellar_logic::cell_group::CellHistory) { *self .by_cell .lock() .unwrap() - .entry(scope.cell.clone()) + .entry(scope.cell.as_str().to_owned()) .or_default() += 1; } - fn count(&self, scope: &GroupScope) -> usize { + fn count(&self, scope: &CellHistory) -> usize { self.by_cell .lock() .unwrap() - .get(&scope.cell) + .get(scope.cell.as_str()) .copied() .unwrap_or(0) } @@ -108,7 +321,16 @@ async fn write( amount: i64, callbacks: &Callbacks, ) -> Result { - let mut options = public_node::archived_options(sequence)?; + let retry_until_ms = u64::try_from( + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH)? + .as_millis(), + )? + 60_000; + let mut options = CommandOptions::new( + CommandId::new(ClientId::new([7; 16])?, sequence)?, + retry_until_ms, + ); + options.durability = Durability::S3; let deadline = tokio::time::Instant::now() + options.timeout; loop { options.timeout = deadline.saturating_duration_since(tokio::time::Instant::now()); @@ -116,21 +338,23 @@ async fn write( !options.timeout.is_zero(), "extent command admission deadline" ); - let before = callbacks.count(cell.scope()); + let before = callbacks.count(cell.history()); let result = cell - .command( - contract.clone(), - amount.to_be_bytes().to_vec(), - options.clone(), - ) + .command(contract, amount.to_be_bytes().to_vec(), options.clone()) .await; - let executed = callbacks.count(cell.scope()) - before; + let executed = callbacks.count(cell.history()) - before; ensure!( executed <= 1, "one command attempt executed its callback more than once" ); match result { - Err(CommandFailure::Overloaded) => { + Err(error) + if matches!(error.cause.refusal(), Some(CommandFailure::Overloaded)) + || matches!( + *error.cause, + CellDispatchError::Runtime(RuntimeError::Overloaded) + ) => + { // Capture may exhaust physical roots after the private SQL // callback. This definite pre-proposal refusal discards that // execution; retrying the same identity may run it again. @@ -139,6 +363,7 @@ async fn write( } result => { return result + .map_err(anyhow::Error::from) .with_context(|| format!("extent write sequence={sequence}, amount={amount}")) } } @@ -149,10 +374,10 @@ async fn close(node: Node, service: RaftCells) -> Result<()> { let closed = service.close().await; let shutdown = node.shutdown(ShutdownMode::Preserve).await; ensure!( - service.resource_usage().extents.is_none(), + service.resource_usage().serving_images.extents.is_none(), "retained SDK handle owns extent storage after close" ); - ensure!(closed?.unfinished.is_empty()); + ensure!(closed?.mutation_outcomes == MutationOutcomes::NoUnresolvedLocalProducers); shutdown?; Ok(()) } @@ -160,17 +385,22 @@ async fn close(node: Node, service: RaftCells) -> Result<()> { #[tokio::test(flavor = "multi_thread", worker_threads = 4)] async fn extent_activation_requires_physical_admission_before_catalog_mutation() -> Result<()> { let root = tempfile::tempdir()?; - let (gate, _entered) = public_node::Gate::new(); - gate.release(); - let definition = public_node::counter(Arc::new(AtomicUsize::new(0)), gate)?; - let node = public_node::node(root.path(), definition.application.clone()).await?; + let class = counter(Arc::new(Callbacks::default()))?; + let group = Group::new(class.schema_digest(), 1)?; + let app = application(class)?; + let node = public_node::node_named(root.path(), app.clone(), "extent-native").await?; let store = Arc::new(object_store::memory::InMemory::new()); - let result = public_node::cells( + let result = start( &node, - definition.application, + app, store.clone(), - registrations(definition.schema)?.remove(0), - false, + &[group], + 1, + 1, + RaftHostOptions { + max_groups: 2, + ..Default::default() + }, ) .await; node.shutdown(ShutdownMode::Preserve).await?; @@ -202,48 +432,50 @@ async fn exercise(max_roots: usize) -> Result<()> { .with_test_writer() .try_init(); let root = tempfile::tempdir()?; - // The shared counter signals its second command even after this gate is - // released. Retain the receiver for the lifetime of all three node runs. - let (gate, _entered) = public_node::Gate::new(); - gate.release(); let calls = Arc::new(Callbacks::default()); - let observed = calls.clone(); - let definition = public_node::counter_observed( - move |scope| { - observed.record(scope); - Ok(()) - }, - gate, - )?; - let registrations = registrations(definition.schema)?; + let class = counter(calls.clone())?; + let groups = [ + Group::new(class.schema_digest(), 1)?, + Group::new(class.schema_digest(), 2)?, + ]; + let increment = class.contract("Increment").context("Increment contract")?; + let get = class.contract("Get").context("Get contract")?; + let app = application(class)?; let store = Arc::new(object_store::memory::InMemory::new()); - let node = public_node::node(root.path(), definition.application.clone()).await?; - let service = public_node::cells_many( + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + for group in &groups { + group.provision(root.path(), store.clone(), &domain).await?; + } + domain.shutdown().await; + let node = public_node::node_named(root.path(), app.clone(), "extent-native").await?; + let service = start( &node, - definition.application.clone(), + app.clone(), store.clone(), - registrations.clone(), - false, + &groups, + 2, + 1, host(max_roots), - Default::default(), ) .await?; let result = async { - let a = service.cell( - registrations[0].namespace.namespace, - ®istrations[0].birth.scope.cell, - )?; - let b = service.cell( - registrations[1].namespace.namespace, - ®istrations[1].birth.scope.cell, - )?; - public_node::ready(&a).await?; - public_node::ready(&b).await?; + let a = groups[0].ready(&service).await?; + let b = groups[1].ready(&service).await?; + for (cell, id) in [(&a, 1), (&b, 2)] { + cell.enroll( + CellEnrollmentRequest { + creation: ProposalId::new([32 + id; 16])?, + durability: Durability::S3, + }, + TIMEOUT, + ) + .await?; + } let mut retained = None; for sequence in 1..=16 { let (left, right) = futures_util::future::join( - write(&a, &definition.increment, sequence, 1, &calls), - write(&b, &definition.increment, sequence, 3, &calls), + write(&a, &increment, sequence, 1, &calls), + write(&b, &increment, sequence, 3, &calls), ) .await; ensure!(left?.result == i64::try_from(sequence)?.to_be_bytes()); @@ -253,16 +485,35 @@ async fn exercise(max_roots: usize) -> Result<()> { } let usage = service .resource_usage() + .serving_images .extents .context("missing shared node inventory")?; ensure!(!usage.quarantined && usage.roots <= max_roots && usage.workspaces <= 2); let text = service.resource_usage().prometheus(); - let samples = text - .lines() - .filter(|line| line.starts_with("cellar_raft_extents_")) - .collect::>(); - ensure!(samples.len() == 14 && samples.iter().all(|line| !line.contains('{'))); - ensure!(text.contains("cellar_raft_extents_sample_available 1\n")); + for inventory in ["serving", "archive_verifier"] { + for metric in [ + "bytes", + "capacity_bytes", + "objects", + "roots", + "workspace_bytes", + "quarantined", + ] { + ensure!( + text.lines().any(|line| line.starts_with(&format!( + "cellar_cell_groups_extent_{metric}{{kind=\"{inventory}\"}} " + ))), + "missing {inventory} extent {metric} metric" + ); + } + ensure!(text.contains(&format!( + "cellar_cell_groups_extents_sample_available{{kind=\"{inventory}\"}} 1\n" + ))); + } + for group in &groups { + let observed = service.group(group.birth.history)?.observe(TIMEOUT).await?; + ensure!(observed.archive_required && observed.checkpoint.index.get() > 0); + } retained.context("missing retained receipt") } .await; @@ -270,33 +521,41 @@ async fn exercise(max_roots: usize) -> Result<()> { let retained = result?; closed?; - // Register only A. GC must also preserve B's engine-selected current and - // checkpoint roots, even though no B SQL owner exists in this process. - let node = public_node::node(root.path(), definition.application.clone()).await?; - let service = public_node::cells_many( + // Reattach only A. The node inventory also owns B's retained current + // and checkpoint roots while the B SQL owner is absent. + let node = public_node::node_named(root.path(), app.clone(), "extent-native").await?; + let service = start( &node, - definition.application.clone(), + app.clone(), store.clone(), - vec![registrations[0].clone()], - true, + &groups, + 1, + 2, host(max_roots), - Default::default(), ) .await?; let result = async { - let a = service.cell( - registrations[0].namespace.namespace, - ®istrations[0].birth.scope.cell, - )?; - public_node::ready(&a).await?; + let a = groups[0].ready(&service).await?; + let b = service.group(groups[1].birth.history)?; + ensure!( + b.status().await.is_err(), + "retained group gained a runtime owner" + ); + ensure!(service.resource_usage().retained_manifest_cells == Some(2)); for sequence in 17..=32 { ensure!( - write(&a, &definition.increment, sequence, 1, &calls) - .await? - .result + write(&a, &increment, sequence, 1, &calls).await?.result == i64::try_from(sequence)?.to_be_bytes() ); } + let collected = service + .resource_usage() + .collection + .context("node collector")?; + ensure!( + collected.files_reclaimed > 0 && collected.failures == 0, + "collector did not reclaim files while B was unattached: {collected:?}" + ); Ok::<_, anyhow::Error>(()) } .await; @@ -304,26 +563,14 @@ async fn exercise(max_roots: usize) -> Result<()> { result?; closed?; - let node = public_node::node(root.path(), definition.application.clone()).await?; - let service = public_node::cells_many( - &node, - definition.application.clone(), - store, - registrations.clone(), - true, - host(max_roots), - Default::default(), - ) - .await?; + let node = public_node::node_named(root.path(), app.clone(), "extent-native").await?; + let service = start(&node, app, store, &groups, 2, 3, host(max_roots)).await?; let result = async { - let b = service.cell( - registrations[1].namespace.namespace, - ®istrations[1].birth.scope.cell, - )?; - public_node::ready(&b).await?; + let b = groups[1].ready(&service).await?; let historical = b .query( - definition.get.clone(), + ProposalId::new([40; 16])?, + &get, vec![], QueryOptions { consistency: QueryConsistency::AtLeast(retained.version_token.clone()), @@ -333,18 +580,13 @@ async fn exercise(max_roots: usize) -> Result<()> { .await?; ensure!(historical.result == 48_i64.to_be_bytes()); let before = calls.total(); - let duplicate = write(&b, &definition.increment, 16, 3, &calls).await?; + let duplicate = write(&b, &increment, 16, 3, &calls).await?; ensure!(duplicate.id == retained.id && duplicate.result == retained.result); ensure!( calls.total() == before, "restored receipt reran the callback" ); - ensure!( - write(&b, &definition.increment, 17, 3, &calls) - .await? - .result - == 51_i64.to_be_bytes() - ); + ensure!(write(&b, &increment, 17, 3, &calls).await?.result == 51_i64.to_be_bytes()); let refused = calls.refused.load(Ordering::SeqCst); ensure!( calls.total().checked_sub(refused) == Some(49), diff --git a/crates/cellar/tests/native_raft_fleet_permits.rs b/crates/cellar/tests/native_raft_fleet_permits.rs index d2183b7d1..0ba40b872 100644 --- a/crates/cellar/tests/native_raft_fleet_permits.rs +++ b/crates/cellar/tests/native_raft_fleet_permits.rs @@ -1,1040 +1,497 @@ -//! Managed permits through native SQLite hosts, real TLS links and retained -//! engines. Catalog edits here stand in for the later typed fleet controller. -#[path = "support/raft_catalog_fault.rs"] -#[allow(dead_code)] -mod catalog_fault; -#[path = "support/raft_cluster.rs"] -#[allow(dead_code)] -mod cluster; -#[path = "support/raft_namespace_import.rs"] -mod namespace_import; -#[path = "support/raft_fleet_observation.rs"] -mod observation; -#[path = "support/raft_node.rs"] -#[allow(dead_code)] -mod public_node; +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. +//! Exact shared-group membership permits through rooted managed movement APIs. +#![allow(clippy::disallowed_methods)] use anyhow::{ensure, Context as _, Result}; -use cellar::application::consensus::verification::{ - Catalog, CatalogOptions, FleetCatalog, FleetCatalogError, FleetChange, FleetMutation, - FleetTable, +use cellar::sdk::{consensus::*, *}; +use managed_fixture::{ + archives, inspect_current, movement, request, start_node, ArchiveSetup, FirstGroupSetup, + ManagedCluster, }; -use cellar::bucket::{Bucket, StorageBackend}; -use cellar::sdk::{consensus::*, ExecutionDomain}; -use object_store::{list::*, ObjectStore}; use std::{ sync::{ atomic::{AtomicUsize, Ordering}, Arc, }, - time::Duration, + time::{Duration, Instant}, }; -const TIMEOUT: Duration = Duration::from_secs(10); +#[path = "support/cell_group_managed_fixture.rs"] +#[allow(dead_code, unused_imports)] +mod managed_fixture; -#[derive(Debug)] -struct NoListing; +#[path = "support/raft_fleet_permits_shared.rs"] +mod fixture; +use fixture::*; -#[async_trait::async_trait] -impl PaginatedListStore for NoListing { - async fn list_paginated( - &self, - _: Option<&str>, - _: PaginatedListOptions, - ) -> object_store::Result { - Err(object_store::Error::NotSupported { - source: "permit authority never lists objects".into(), - }) - } -} - -struct Fixture { - cluster: cluster::Cluster, - definitions: Vec, - calls: Vec>, - handles: Vec, - registration: RaftCellRegistration, - primary: usize, - domain: ExecutionDomain, - catalog: FleetCatalog, - binding: FleetGroupBinding, - _entered: tokio::sync::mpsc::UnboundedReceiver<()>, -} - -async fn leader(domain: &ExecutionDomain, cells: &[RaftCellHandle]) -> Result { - Ok(leader_status(domain, cells).await?.0) -} - -// Local readiness can briefly outlive isolation. Route at the highest -// observed term; the guarded operation still has to establish its proof. -async fn leader_status( - domain: &ExecutionDomain, - cells: &[RaftCellHandle], -) -> Result<(usize, CellStatus)> { - Ok(domain - .run(cellar::asyncrt::timeout(TIMEOUT, async { - loop { - let mut highest_term = 0; - let mut ready = None; - for (index, cell) in cells.iter().enumerate() { - if let Ok(status) = cell.status().await { - if status.term.get() > highest_term { - highest_term = status.term.get(); - ready = None; - } - if status.term.get() == highest_term && status.command_ready { - ready = Some((index, status)); - } - } - } - if let Some(ready) = ready { - return ready; - } - cellar::asyncrt::sleep(Duration::from_millis(10)).await; - } - })) - .await??) -} - -impl Fixture { - async fn new() -> Result { - Self::create(true, true).await - } - - async fn create(write_image: bool, enroll: bool) -> Result { - Self::configured( - write_image, - enroll, - RaftHostOptions::default(), - Arc::new(object_store::memory::InMemory::new()), - ) - .await - } - - async fn configured( - write_image: bool, - enroll: bool, - host: RaftHostOptions, - store: Arc, - ) -> Result { - let (gate, entered) = public_node::Gate::new(); - gate.release(); - let calls: Vec<_> = (0..4).map(|_| Arc::new(AtomicUsize::new(0))).collect(); - let definitions = calls - .iter() - .map(|n| public_node::counter(n.clone(), gate.clone())) - .collect::>>()?; - let scope = GroupScope { - cluster: ClusterId::new([1; 16])?, - namespace: NamespaceId::new([2; 16])?, - cell: "Counter:fleet".into(), - group: GroupId::new([55; 16])?, - incarnation: IncarnationId::new([3; 16])?, - }; - let registration = RaftCellRegistration { - placement: RaftCellPlacement::Replica, - namespace: NamespaceConfig::new( - scope.cluster, - scope.namespace, - NamespaceMode::Raft, - definitions[0].schema, - ), - class: "Counter".into(), - birth: BirthProposal { - scope: scope.clone(), - voters: (1..=3) - .map(|n| { - Ok(InitialMember { - member: MemberId::new(n)?, - node: NodeId::new([n as u8; 16])?, - }) - }) - .collect::>>()?, - learners: vec![], - }, - }; - let cluster = cluster::Cluster::start_with_standby_store( - &definitions, - vec![registration.clone()], - host, - store, - ) - .await?; - let handles = cluster - .services - .iter() - .map(|s| s.cell(scope.namespace, &scope.cell)) - .collect::, _>>()?; - let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); - let primary = leader(&domain, &handles).await?; - let catalog = Self::open_catalog(&domain, cluster.store.clone())?; - catalog.initialize(TIMEOUT).await?; - let binding = catalog.group_binding(&scope)?; - let f = Self { - cluster, - definitions, - calls, - handles, - registration, - primary, - domain, - catalog, - binding, - _entered: entered, - }; - let namespace = Catalog::new( - f.cluster.store.clone(), - CatalogOptions { - prefix: "partition/test".into(), - ..Default::default() - }, - )? - .existing_namespace(f.registration.namespace.clone()) - .await?; - let root = f.catalog.read(TIMEOUT).await?; - f.catalog - .declare_namespace( - &root, - &FleetRequestIdentity { - id: [250; 16], - operation: [251; 16], - principal: "fixture-declaration".into(), - expected_revision: root.revision(), - }, - &namespace, - "Counter", - TIMEOUT, - ) +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn fleet_permit_delayed_after_cancel() -> Result<()> { + qualify(false, async |f| { + let reserved = f.reserve(210).await?; + let fenced = f + .group(f.primary)? + .fence_fleet_permit(reserved.permit.clone(), TIMEOUT) .await?; - let birth = namespace - .birth(&scope.cell) - .await? - .context("selected birth")?; - let snapshot = f.catalog.read(TIMEOUT).await?; - f.catalog - .enroll_history( - &snapshot, - &FleetRequestIdentity { - id: [10; 16], - operation: [99; 16], - principal: "permit-test".into(), - expected_revision: snapshot.revision(), - }, - &namespace, - &birth, + let guard = status(&fenced)?; + ensure!(guard.consumed_sequence == 1 && guard.phase == Some(FleetPermitPhase::Fenced)); + drop(fenced); + ensure!(f.strong(f.primary).await? == 5); + f.settle(&reserved, 230, FleetPermitPhase::Fenced).await?; + // Settlement keeps the historical permit. Even a freshly read + // authorization cannot restart the quorum-fenced sequence. + let reconciled = f.catalog_permit().await?; + ensure!(f + .group(f.primary)? + .begin_managed_repair( + reserved.group.clone(), + reconciled, + reserved.plan.clone(), TIMEOUT, ) + .await + .is_err()); + ensure!(f.begin(f.primary, &reserved).await.is_err()); + let fenced = f + .group(f.primary)? + .fence_fleet_permit(reserved.permit.clone(), TIMEOUT) .await?; - // Establish a real SQL image before metadata-only enrollment changes. - if write_image { - f.write(1, 5, true).await?; - } - let root = f.catalog.read(TIMEOUT).await?; - let authorization = f.catalog.authorize_group(&root, &scope, TIMEOUT).await?; - let expected = f.handles[primary].status().await?.membership_revision; - if enroll { - let enrolled = f.handles[primary] - .enroll_fleet(authorization, expected, TIMEOUT) - .await?; - ensure!( - enrolled.binding == f.binding - && enrolled.consumed_sequence == 0 - && enrolled.permit.is_none() - ); - } - Ok(f) - } + ensure!(status(&fenced)? == guard && fenced.membership().high_water() == 3); + ensure!(fenced.membership().repair_status()?.is_none()); + drop(fenced); - async fn change(&self, id: u8, table: FleetTable, value: Option>) -> Result<()> { - let root = self.catalog.read(TIMEOUT).await?; - self.catalog - .commit( - &root, - &FleetMutation { - id: [id; 16], - operation: [99; 16], - expected_revision: root.revision(), - principal: "native-test-controller".into(), - changes: vec![FleetChange { - table, - key: self.binding.key()?, - value, - }], - }, - TIMEOUT, - ) + let next = f.reserve(212).await?; + ensure!(next.expected.sequence() == 2); + verify_phase( + &f.begin(f.primary, &next).await?, + &next, + FleetPermitPhase::Seeding, + )?; + ensure!(f + .group(f.primary)? + .fence_fleet_permit(reserved.permit, TIMEOUT) + .await + .is_err()); + f.advance(&next, RepairStep::AbortLearner, FleetPermitPhase::Aborted) .await?; Ok(()) - } - - async fn plan(&self, id: u8) -> Result { - Ok(RepairPlan { - id: ProposalId::new([id; 16])?, - policy_revision: 1, - expected_revision: self.handles[self.primary] - .status() - .await? - .membership_revision, - obsolete: MemberId::new(((self.primary + 1) % 3) as u64 + 1)?, - replacement: JoiningMember { - member: MemberId::new(4)?, - node: NodeId::new([4; 16])?, - disk: DiskLineage::new([13; 16])?, - }, - }) - } + }) + .await +} - async fn reserve( - &self, - sequence: u64, - id: u8, - plan: &RepairPlan, - ) -> Result { - let permit = FleetMembershipPermit::new( - self.binding.clone(), - sequence, - ProposalId::new([id; 16])?, - ProposalId::new([99; 16])?, - plan, - )?; - self.change(id, FleetTable::Admissions, Some(permit.encode()?)) - .await?; - let root = self.catalog.read(TIMEOUT).await?; - Ok(self - .catalog - .authorize_membership(&root, &self.registration.birth.scope, TIMEOUT) - .await?) - } +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn fleet_permit_repair_step_follows_new_quorum_leader() -> Result<()> { + qualify(false, async |f| { + let reserved = f.reserve(210).await?; + let source = f.primary; + let began = f.begin(source, &reserved).await?; + verify_phase(&began, &reserved, FleetPermitPhase::Seeding)?; + let term = began.status().term; + drop(began); + f.stop(source).await?; + let (leader, successor_term) = f.leader().await?; + ensure!(leader != source && successor_term > term); + f.advance( + &reserved, + RepairStep::AbortLearner, + FleetPermitPhase::Aborted, + ) + .await?; + ensure!(f.strong(leader).await? == 5); + f.ensure_pending(&reserved).await?; + Ok(()) + }) + .await +} - async fn write(&self, id: u8, value: i64, s3: bool) -> Result { - let mut options = public_node::options(u64::from(id))?; - if s3 { - options.durability = Durability::S3; - } - Ok(self.handles[self.primary] - .command( - self.definitions[self.primary].increment.clone(), - value.to_be_bytes().to_vec(), - options, - ) - .await?) - } +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn fleet_permit_crash_recovery() -> Result<()> { + qualify(false, async |f| { + let reserved = f.reserve(210).await?; + let calls = f.counts(); + // The spare's original boot owns the reservation throughout this voter + // restart. Changing it would require independent capacity settlement. + f.restart_voters(40).await?; + let selected = f.group(f.primary)?.quorum_membership(TIMEOUT).await?; + ensure!(status(&selected)?.consumed_sequence == 0 && status(&selected)?.permit.is_none()); + drop(selected); + let reopened_permit = f.catalog_permit().await?; - async fn strong(&self) -> Result { - let read = self.handles[self.primary] - .query( - self.definitions[self.primary].get.clone(), - vec![], - QueryOptions { - consistency: QueryConsistency::Linearizable, - ..Default::default() - }, - ) + let lost = tokio::spawn(f.group(f.primary)?.begin_managed_repair( + reserved.group.clone(), + reopened_permit, + reserved.plan.clone(), + TIMEOUT, + )); + let consumed = f + .wait_phase(f.primary, &reserved, FleetPermitPhase::Seeding) .await?; - Ok(i64::from_be_bytes(read.result.as_slice().try_into()?)) - } - - async fn step( - &self, - authorization: &FleetPermitAuthorization, - step: RepairStep, - ) -> Result { - let mut last_status = None; - let mut last_refusal = None; - let result = self - .domain - .run(cellar::asyncrt::timeout(TIMEOUT, async { - loop { - // A repair survives leader loss. Route the same permit - // and idempotent step to a freshly observed leader; the - // initial fixture leader is not permanent authority. - let (primary, status) = leader_status(&self.domain, &self.handles).await?; - let expected = status.membership_revision; - last_status = Some((primary, status)); - match self.handles[primary] - .advance_managed_repair( - authorization.clone(), - step, - expected, - Duration::from_secs(2), - ) - .await - { - Ok(status) => return Ok(status), - Err( - error @ (CommandFailure::Unavailable - | CommandFailure::Overloaded - | CommandFailure::NotLeader { .. }), - ) => { - last_refusal = Some(error); - cellar::asyncrt::sleep(Duration::from_millis(10)).await - } - Err(error) => return Err(error.into()), - } - } - })) + // The independent quorum observation owns the evidence. Discard the + // original reply even if its producer already finished. + lost.abort(); + let _ = lost.await; + f.restart_voters(50).await?; + let restored = f + .wait_phase(f.primary, &reserved, FleetPermitPhase::Seeding) .await?; - result.map_err(anyhow::Error::from).and_then(|status| status) - .with_context(|| format!( - "managed repair {step:?}: last leader status {last_status:?}, refusal {last_refusal:?}", - )) - } - - fn activation(&self, from_s3: bool) -> Result { - Ok(LearnerActivation { - member: MemberId::new(4)?, - source: MemberId::new(self.primary as u64 + 1)?, - source_node: NodeId::new([self.primary as u8 + 1; 16])?, - from_s3, - timeout: TIMEOUT, - }) - } + ensure!(restored == consumed); + let resumed = f.begin(f.primary, &reserved).await?; + ensure!(status(&resumed)? == consumed); + drop(resumed); + ensure!(f.counts() == calls); - async fn standby(&mut self) -> Result { - let service = self - .cluster - .join_standby( - self.definitions[3].application.clone(), - RaftCellPlacement::Standby, + let terminal = f + .advance( + &reserved, + RepairStep::AbortLearner, + FleetPermitPhase::Aborted, ) .await?; - let cell = service.cell( - self.registration.namespace.namespace, - &self.registration.birth.scope.cell, - )?; - self.handles.push(cell.clone()); - Ok(cell) - } - - async fn observed( - &self, - cell: &RaftCellHandle, - sequence: u64, - phase: FleetPermitPhase, - ) -> Result { - self.domain - .run(cellar::asyncrt::timeout(TIMEOUT, async { - loop { - if let Some(status) = cell.fleet_membership_status().await? { - if status.consumed_sequence == sequence && status.phase == Some(phase) { - return Ok(status); - } - } - cellar::asyncrt::sleep(Duration::from_millis(10)).await; - } - })) - .await? - .context("expected durable fleet phase not observed")? - } - - fn open_catalog(domain: &ExecutionDomain, store: Arc) -> Result { - let bucket = Bucket::from_clients( - store.clone(), - store, - Arc::new(NoListing), - StorageBackend::S3, - "fleet-permits".into(), - "".into(), - )?; - Ok(FleetCatalog::new( - domain, - bucket, - ClusterId::new([1; 16])?, - IncarnationId::new([80; 16])?, - "coordination/", - )?) - } - - async fn restart(mut self) -> Result { + ensure!(f.strong(f.primary).await? == 5); + let refused = f + .settle(&reserved, 230, FleetPermitPhase::Aborted) + .await + .expect_err("new voter boots released the old physical debit"); ensure!( - self.handles.len() == 4, - "all retained engines must exist before Resume" + matches!( + refused.downcast_ref::(), + Some(FleetCatalogError::Unconfirmed) + ), + "retained boot settlement: {refused:#}" ); - self.cluster = self - .cluster - .restart( - &self.definitions, - vec![self.registration.clone()], - vec![ - RaftCellPlacement::Replica, - RaftCellPlacement::Replica, - RaftCellPlacement::Replica, - RaftCellPlacement::Standby, - ], - Box::new(|_| Box::pin(async { Ok(()) })), + f.ensure_pending(&reserved).await?; + f.restart_voters(60).await?; + let replay = f + .advance( + &reserved, + RepairStep::AbortLearner, + FleetPermitPhase::Aborted, ) .await?; - self.handles = self - .cluster - .services - .iter() - .map(|s| { - s.cell( - self.registration.namespace.namespace, - &self.registration.birth.scope.cell, - ) - }) - .collect::, _>>()?; - self.primary = leader(&self.domain, &self.handles).await?; - self.catalog = Self::open_catalog(&self.domain, self.cluster.store.clone())?; - Ok(self) - } - - async fn close(self) -> Result<()> { - self.cluster.close().await?; - self.domain.shutdown().await; + ensure!(replay == terminal); + let delayed = f + .group(f.primary)? + .begin_managed_repair( + reserved.group.clone(), + reserved.permit.clone(), + reserved.plan.clone(), + TIMEOUT, + ) + .await; + ensure!(matches!( + delayed, + Err(CellGroupControlError::Refused( + RuntimeError::StaleCompletion + )) + )); + ensure!(f.counts() == calls); + f.ensure_pending(&reserved).await?; Ok(()) - } + }) + .await } #[tokio::test(flavor = "multi_thread", worker_threads = 4)] -async fn fleet_permit_delayed_after_cancel() -> Result<()> { - let mut f = Fixture::new().await?; - // The reserved replacement receives protocol traffic once the plan starts. - // Start its real dormant process; activation still requires normal seeding. - f.standby().await?; - let plan = f.plan(40).await?; - let cached = f.reserve(1, 20, &plan).await?; - let status = f.handles[f.primary] - .fence_fleet_permit(cached.clone(), TIMEOUT) - .await?; - ensure!(status.consumed_sequence == 1 && status.phase == Some(FleetPermitPhase::Fenced)); - // This quorum read is separate from local status and precedes the simulated - // controller release. FO-06 owns typed terminal proof/root reconciliation. - assert_eq!(f.strong().await?, 5); - f.change(21, FleetTable::Admissions, None).await?; - let root = f.catalog.read(TIMEOUT).await?; - ensure!(matches!( - f.catalog - .authorize_membership(&root, &f.registration.birth.scope, TIMEOUT) - .await, - Err(FleetCatalogError::Conflict) - )); - ensure!(matches!( - f.handles[f.primary] - .begin_managed_repair(cached.clone(), plan, TIMEOUT) - .await, - Err(CommandFailure::CommandConflict) - )); - ensure!(f.handles[f.primary].repair_status().await?.is_none()); - assert_eq!(f.handles[f.primary].status().await?.member_high_water, 3); - assert_eq!( - f.handles[f.primary] - .fence_fleet_permit(cached.clone(), TIMEOUT) - .await?, - status - ); - let next = f.plan(41).await?; - let authorization = f.reserve(2, 22, &next).await?; - let repair = f.handles[f.primary] - .begin_managed_repair(authorization.clone(), next, TIMEOUT) - .await?; - ensure!(repair.phase == RepairPhase::Seeding); - ensure!(f.handles[f.primary] - .fence_fleet_permit(cached, TIMEOUT) - .await - .is_err()); - assert_eq!( - f.step(&authorization, RepairStep::AbortLearner) - .await? - .phase, - RepairPhase::Aborted - ); - f.close().await -} +async fn fleet_permit_joint_cancel() -> Result<()> { + qualify(false, async |f| { + let reserved = f.reserve(210).await?; + verify_phase( + &f.begin(f.primary, &reserved).await?, + &reserved, + FleetPermitPhase::Seeding, + )?; + f.activate(&reserved, f.primary, false).await?; + let joint = f + .advance( + &reserved, + RepairStep::PromoteAndRemove, + FleetPermitPhase::Joint, + ) + .await?; + ensure!( + f.wait_phase(SPARE, &reserved, FleetPermitPhase::Joint) + .await? + == joint + ); + let primary = f.primary; + let group = f.group(primary)?; + ensure!(group + .fence_fleet_permit(reserved.permit.clone(), TIMEOUT) + .await + .is_err()); + let revision = group.observe(TIMEOUT).await?.membership_revision; + ensure!(group + .advance_managed_repair( + reserved.group.clone(), + reserved.permit.clone(), + RepairStep::AbortLearner, + revision, + TIMEOUT, + ) + .await + .is_err()); -#[tokio::test(flavor = "multi_thread", worker_threads = 4)] -async fn fleet_permit_repair_step_follows_new_quorum_leader() -> Result<()> { - let mut f = Fixture::new().await?; - f.standby().await?; - let plan = f.plan(40).await?; - let authorization = f.reserve(1, 20, &plan).await?; - let repair = f.handles[f.primary] - .begin_managed_repair(authorization.clone(), plan, TIMEOUT) - .await?; - ensure!(repair.phase == RepairPhase::Seeding); - let (source, status) = leader_status(&f.domain, &f.handles).await?; - ensure!(source == f.primary); - // Optional transfers correctly refuse an active repair. Exercise actual - // leader loss instead, retaining the same permit and committed plan. - f.cluster.isolate_consensus(source).await?; - let survivors = f - .handles - .iter() - .enumerate() - .filter(|(index, _)| *index != source) - .map(|(_, cell)| cell.clone()) - .collect::>(); - let (_, successor) = leader_status(&f.domain, &survivors).await?; - ensure!(successor.term > status.term); - let advanced = f.step(&authorization, RepairStep::AbortLearner).await; - if advanced.is_err() { - for (index, cell) in f.handles.iter().enumerate() { - eprintln!("stalled repair node {index}: {:?}", cell.status().await); + // Keep a majority of the incoming voters while losing the old majority. + for voter in 0..SPARE { + if voter != primary { + f.stop(voter).await?; + } } - } - f.cluster.heal(); - let closed = f.close().await; - let advanced = advanced.with_context(|| format!("joined repair fixture: {closed:?}"))?; - ensure!(advanced.phase == RepairPhase::Aborted); - closed -} - -#[tokio::test(flavor = "multi_thread", worker_threads = 4)] -async fn fleet_permit_crash_recovery() -> Result<()> { - let mut f = Fixture::new().await?; - // An empty registered host has an actual intact engine to resume; no - // learner or SQL image is initialized by this step. - f.standby().await?; - let plan = f.plan(40).await?; - f.reserve(1, 20, &plan).await?; - let calls: Vec<_> = f.calls.iter().map(|n| n.load(Ordering::SeqCst)).collect(); - f = f.restart().await.context("restart after reservation")?; - assert_eq!( - f.handles[f.primary] - .fleet_membership_status() - .await? - .unwrap() - .consumed_sequence, - 0 - ); - let root = f.catalog.read(TIMEOUT).await?; - let authorization = f - .catalog - .authorize_membership(&root, &f.registration.birth.scope, TIMEOUT) - .await?; - // An unpolled returned future still represents a submitted configuration. - let lost = - f.handles[f.primary].begin_managed_repair(authorization.clone(), plan.clone(), TIMEOUT); - let consumed = f - .observed(&f.handles[f.primary], 1, FleetPermitPhase::Seeding) - .await?; - drop(lost); - f = f - .restart() - .await - .context("restart after committed plan/lost response")?; - let restored = f.handles[f.primary] - .fleet_membership_status() - .await? - .context("fleet guard lost on engine reopen")?; - assert_eq!(restored, consumed); - let resumed = f.handles[f.primary] - .begin_managed_repair(authorization.clone(), plan.clone(), TIMEOUT) - .await?; - ensure!(resumed.phase == RepairPhase::Seeding && resumed.changed_at == consumed.changed_at); - assert_eq!( - f.calls - .iter() - .map(|n| n.load(Ordering::SeqCst)) - .collect::>(), - calls - ); - let terminal = f.step(&authorization, RepairStep::AbortLearner).await?; - assert_eq!(terminal.phase, RepairPhase::Aborted); - assert_eq!(f.strong().await?, 5); - f.change(21, FleetTable::Admissions, None).await?; - f = f - .restart() - .await - .context("restart after terminal receipt/release")?; - assert_eq!( - f.handles[f.primary] - .begin_managed_repair(authorization, plan, TIMEOUT) - .await?, - terminal - ); - let root = f.catalog.read(TIMEOUT).await?; - ensure!(matches!( - f.catalog - .authorize_membership(&root, &f.registration.birth.scope, TIMEOUT) - .await, - Err(FleetCatalogError::Conflict) - )); - assert_eq!( - f.calls - .iter() - .map(|n| n.load(Ordering::SeqCst)) - .collect::>(), - calls - ); - f.close().await -} - -#[tokio::test(flavor = "multi_thread", worker_threads = 4)] -async fn fleet_permit_joint_cancel() -> Result<()> { - let mut f = Fixture::new().await?; - let plan = f.plan(40).await?; - let authorization = f.reserve(1, 20, &plan).await?; - f.handles[f.primary] - .begin_managed_repair(authorization.clone(), plan.clone(), TIMEOUT) - .await?; - let target = f.standby().await?; - target - .activate_managed_learner(authorization.clone(), f.activation(false)?) - .await?; - assert_eq!( - f.step(&authorization, RepairStep::PromoteAndRemove) - .await? - .phase, - RepairPhase::Joint - ); - let joint_guard = f.observed(&target, 1, FleetPermitPhase::Joint).await?; - ensure!(f.handles[f.primary] - .fence_fleet_permit(authorization.clone(), TIMEOUT) - .await - .is_err()); - let expected = f.handles[f.primary].status().await?.membership_revision; - ensure!(f.handles[f.primary] - .advance_managed_repair( - authorization.clone(), - RepairStep::AbortLearner, - expected, - TIMEOUT - ) - .await - .is_err()); - // Preserve only the leader and incoming target: new majority exists, old - // majority does not. Neither a cancellation nor finish may invent it. - for (index, service) in f.cluster.services[..3].iter().enumerate() { - if index != f.primary { - ensure!(service.close().await?.unfinished.is_empty()); + ensure!(group + .advance_managed_repair( + reserved.group.clone(), + reserved.permit.clone(), + RepairStep::FinishJoint, + revision, + SHORT, + ) + .await + .is_err()); + ensure!(group.observe(TIMEOUT).await?.membership_revision == revision); + f.ensure_pending(&reserved).await?; + for index in [primary, SPARE] { + ensure!(f + .group(index)? + .observe_fleet_membership(reserved.group.clone(), SHORT) + .await + .is_err()); } - } - ensure!(f.handles[f.primary] - .advance_managed_repair( - authorization.clone(), - RepairStep::FinishJoint, - expected, - Duration::from_millis(500) - ) - .await - .is_err()); - assert_eq!( - f.handles[f.primary] - .fleet_membership_status() - .await? - .unwrap() - .phase, - Some(FleetPermitPhase::Joint) - ); - let root = f.catalog.read(TIMEOUT).await?; - ensure!(f - .catalog - .get(&root, FleetTable::Admissions, &f.binding.key()?, TIMEOUT) - .await? - .is_some()); - let ns = f.registration.namespace.namespace; - let name = f.registration.birth.scope.cell.clone(); - let get = f.definitions[3].get.clone(); - let group_authorization = f - .catalog - .authorize_group(&root, &f.registration.birth.scope, TIMEOUT) - .await?; - for cell in [&f.handles[f.primary], &target] { + + f.stop(SPARE).await?; + f.reopen(SPARE, 45).await?; + let retained = f.group(SPARE)?; + ensure!(retained.observe(TIMEOUT).await?.membership_revision == revision); + let refused = f.read(SPARE, QueryConsistency::Linearizable, SHORT).await; + let refused = refused.expect_err("retained joint read bypassed the missing old majority"); ensure!( - cell.observe_fleet_membership(group_authorization.clone(), Duration::from_millis(500),) - .await - .is_err(), - "a new majority alone cannot authorize a joint observation" + matches!( + refused + .downcast_ref::>() + .map(Arc::as_ref), + Some(CellDispatchError::Runtime( + RuntimeError::DeadlineExceeded | RuntimeError::NotReady + )) + ), + "retained joint read outcome: {refused:?}" ); - } - let reopen_authorization = group_authorization.clone(); - f.cluster = f - .cluster - .restart( - &f.definitions, - vec![f.registration.clone()], - vec![ - RaftCellPlacement::Replica, - RaftCellPlacement::Replica, - RaftCellPlacement::Replica, - RaftCellPlacement::Standby, - ], - Box::new(move |service| { - Box::pin(async move { - let cell = service.cell(ns, &name)?; - assert_eq!(cell.fleet_membership_status().await?, Some(joint_guard)); - let read = cell - .query( - get, - vec![], - QueryOptions { - consistency: QueryConsistency::Linearizable, - timeout: Duration::from_millis(200), - ..Default::default() - }, - ) - .await; - ensure!( - matches!(read, Err(CommandFailure::VersionUnavailable)), - "retained joint target bypassed the missing old majority: {read:?}" - ); - ensure!( - cell.observe_fleet_membership( - reopen_authorization, - Duration::from_millis(200), - ) - .await - .is_err(), - "reopened joint metadata bypassed the missing old majority" - ); - Ok(()) - }) - }), - ) - .await?; - f.handles = f - .cluster - .services - .iter() - .map(|s| { - s.cell( - f.registration.namespace.namespace, - &f.registration.birth.scope.cell, + ensure!(retained + .observe_fleet_membership(reserved.group.clone(), SHORT) + .await + .is_err()); + + for voter in 0..SPARE { + if voter != primary { + f.reopen(voter, 50 + voter as u8).await?; + } + } + let terminal = f + .advance( + &reserved, + RepairStep::FinishJoint, + FleetPermitPhase::Complete, ) - }) - .collect::, _>>()?; - f.primary = leader(&f.domain, &f.handles).await?; - assert_eq!( - f.step(&authorization, RepairStep::FinishJoint).await?.phase, - RepairPhase::Complete - ); - let terminal = f - .observed(&f.handles[f.primary], 1, FleetPermitPhase::Complete) - .await?; - ensure!( - terminal.permit - == Some(FleetMembershipPermit::new( - f.binding.clone(), - 1, - ProposalId::new([20; 16])?, - ProposalId::new([99; 16])?, - &plan - )?) - ); - let fresh = f.handles[f.primary] - .observe_fleet_membership(group_authorization, TIMEOUT) - .await?; - assert_eq!(fresh.status(), &terminal); - ensure!(fresh.configuration().outgoing_voters().is_empty()); - ensure!(fresh.applied().index >= fresh.barrier()); - assert_eq!(f.strong().await?, 5); - f.change(21, FleetTable::Admissions, None).await?; - f.close().await + .await?; + let fresh = f + .group(f.primary)? + .observe_fleet_membership(reserved.group.clone(), TIMEOUT) + .await?; + ensure!(status(&fresh)? == terminal); + ensure!(fresh.configuration().voters_outgoing.is_empty()); + ensure!(fresh.status().applied.index >= fresh.barrier().index); + ensure!(f.strong(f.primary).await? == 5); + // The retained learner changed boot. Completion of consensus does + // not release the original destination's physical capacity debit. + f.ensure_pending(&reserved).await?; + Ok(()) + }) + .await } #[tokio::test(flavor = "multi_thread", worker_threads = 4)] async fn fleet_permit_snapshot_roundtrip() -> Result<()> { for from_s3 in [false, true] { - let mut f = Fixture::new().await?; - let plan = f.plan(40).await?; - let authorization = f.reserve(1, 20, &plan).await?; - f.handles[f.primary] - .begin_managed_repair(authorization.clone(), plan, TIMEOUT) - .await?; - let receipt = f.write(2, 1, true).await?; - let before = f.handles[f.primary].fleet_membership_status().await?; - let target = f.standby().await?; - target - .activate_managed_learner(authorization.clone(), f.activation(from_s3)?) - .await?; - let read = target - .query( - f.definitions[3].get.clone(), - vec![], - QueryOptions { - consistency: QueryConsistency::AtLeast(receipt.version_token), - ..Default::default() - }, + qualify(from_s3, async |f| { + let reserved = f.reserve(210).await?; + let began = f.begin(f.primary, &reserved).await?; + let before = status(&began)?; + drop(began); + let receipt = f.write(2, 1, Durability::S3).await?; + let target = f.activate(&reserved, f.primary, from_s3).await?; + ensure!(target == before); + let read = f + .read( + SPARE, + QueryConsistency::AtLeast(receipt.version_token), + TIMEOUT, + ) + .await?; + ensure!(read.result == 6_i64.to_be_bytes() && read.served_by == MemberId::new(4)?); + ensure!(f.counts()[SPARE] == 0, "seed reran application commands"); + f.advance( + &reserved, + RepairStep::PromoteAndRemove, + FleetPermitPhase::Joint, ) .await?; - assert_eq!(read.result, 6_i64.to_be_bytes()); - assert_eq!(target.fleet_membership_status().await?, before); - assert_eq!( - f.calls[3].load(Ordering::SeqCst), - 0, - "snapshot recovery reran application commands" - ); - ensure!(target - .activate_learner(f.activation(from_s3)?) - .await - .is_err()); - assert_eq!( - f.step(&authorization, RepairStep::PromoteAndRemove) - .await? - .phase, - RepairPhase::Joint - ); - assert_eq!( - f.step(&authorization, RepairStep::FinishJoint).await?.phase, - RepairPhase::Complete - ); - f.observed(&target, 1, FleetPermitPhase::Complete).await?; - target - .activate_managed_learner(authorization.clone(), f.activation(from_s3)?) - .await?; - ensure!(f.handles[f.primary] - .fence_fleet_permit(authorization, TIMEOUT) - .await - .is_err()); - f.close().await?; + let complete = f + .advance( + &reserved, + RepairStep::FinishJoint, + FleetPermitPhase::Complete, + ) + .await?; + ensure!( + f.wait_phase(SPARE, &reserved, FleetPermitPhase::Complete) + .await? + == complete + ); + ensure!(f.activate(&reserved, f.primary, from_s3).await? == complete); + ensure!(f + .group(f.primary)? + .fence_fleet_permit(reserved.permit, TIMEOUT) + .await + .is_err()); + Ok(()) + }) + .await?; } Ok(()) } #[tokio::test(flavor = "multi_thread", worker_threads = 4)] async fn fleet_permit_quorumless_fence() -> Result<()> { - let f = Fixture::new().await?; - let plan = f.plan(40).await?; - let authorization = f.reserve(1, 20, &plan).await?; - for (index, service) in f.cluster.services.iter().enumerate() { - if index != f.primary { - ensure!(service.close().await?.unfinished.is_empty()); + qualify(false, async |f| { + let reserved = f.reserve(210).await?; + let group = f.group(f.primary)?; + let before = group.observe(TIMEOUT).await?.membership_revision; + for voter in 0..SPARE { + if voter != f.primary { + f.stop(voter).await?; + } } - } - ensure!(f.handles[f.primary] - .fence_fleet_permit(authorization.clone(), Duration::from_millis(500)) - .await - .is_err()); - let status = f.handles[f.primary] - .fleet_membership_status() - .await? - .unwrap(); - assert_eq!(status.consumed_sequence, 0); - assert!(status.permit.is_none()); - let root = f.catalog.read(TIMEOUT).await?; - ensure!(f - .catalog - .get(&root, FleetTable::Admissions, &f.binding.key()?, TIMEOUT) - .await? - .is_some()); - ensure!(f.handles[f.primary] - .begin_managed_repair(authorization, plan, Duration::from_millis(500)) - .await - .is_err()); - f.close().await + ensure!(group + .fence_fleet_permit(reserved.permit.clone(), SHORT) + .await + .is_err()); + ensure!(group + .begin_managed_repair( + reserved.group.clone(), + reserved.permit.clone(), + reserved.plan.clone(), + SHORT, + ) + .await + .is_err()); + ensure!(group.observe(TIMEOUT).await?.membership_revision == before); + f.ensure_pending(&reserved).await?; + Ok(()) + }) + .await } #[tokio::test(flavor = "multi_thread", worker_threads = 4)] async fn fleet_permit_no_manual_bypass() -> Result<()> { - let mut f = Fixture::new().await?; - let plan = f.plan(40).await?; - let authorization = f.reserve(1, 20, &plan).await?; - ensure!(matches!( - f.handles[f.primary] - .begin_repair(plan.clone(), TIMEOUT) - .await, - Err(CommandFailure::UnsupportedCapability) - )); - ensure!(matches!( - f.handles[f.primary] - .add_learner( - plan.replacement.clone(), - LearnerOptions { - expected_revision: plan.expected_revision, - timeout: TIMEOUT - } - ) - .await, - Err(CommandFailure::UnsupportedCapability) - )); - let mut changed = plan.clone(); - changed.policy_revision += 1; - ensure!(f.handles[f.primary] - .begin_managed_repair(authorization.clone(), changed, TIMEOUT) - .await - .is_err()); - f.handles[f.primary] - .begin_managed_repair(authorization.clone(), plan.clone(), TIMEOUT) - .await?; - let expected = f.handles[f.primary].status().await?.membership_revision; - ensure!(matches!( - f.handles[f.primary] - .advance_repair(plan.id, RepairStep::AbortLearner, expected, TIMEOUT) - .await, - Err(CommandFailure::UnsupportedCapability) - )); - ensure!( - matches!( - f.handles[f.primary] - .add_learner( - plan.replacement, - LearnerOptions { - expected_revision: expected, - timeout: TIMEOUT - } + qualify(false, async |f| { + let reserved = f.reserve(210).await?; + let source = f.group(f.primary)?; + let mut changed = reserved.plan.clone(); + changed.policy_revision += 1; + ensure!(matches!( + source + .begin_managed_repair( + reserved.group.clone(), + reserved.permit.clone(), + changed, + TIMEOUT, ) .await, - Err(CommandFailure::UnsupportedCapability) - ), - "idempotent learner ingress bypassed the guard" - ); - let target = f.standby().await?; - ensure!(target.activate_learner(f.activation(false)?).await.is_err()); - ensure!(matches!( - target.status().await, - Err(CommandFailure::UnsupportedCapability) - )); - target - .activate_managed_learner(authorization.clone(), f.activation(false)?) + Err(CellGroupControlError::Refused(_)) + )); + ensure!(source + .advance_managed_repair( + reserved.group.clone(), + reserved.permit.clone(), + RepairStep::AbortLearner, + reserved.plan.expected_revision, + TIMEOUT, + ) + .await + .is_err()); + ensure!( + f.activate(&reserved, f.primary, false).await.is_err(), + "unconsumed permit seeded a learner" + ); + ensure!(f.group(SPARE).is_err()); + verify_phase( + &f.begin(f.primary, &reserved).await?, + &reserved, + FleetPermitPhase::Seeding, + )?; + let mut wrong = f.activation(f.primary, false)?; + wrong.member = MemberId::new(5)?; + ensure!(f.cluster.hosts[SPARE] + .activate_managed_learner_group(reserved.group.clone(), reserved.permit.clone(), wrong) + .await + .is_err()); + ensure!(f.group(SPARE).is_err()); + let seeded = f.activate(&reserved, f.primary, false).await?; + ensure!(seeded.phase == Some(FleetPermitPhase::Seeding)); + f.advance( + &reserved, + RepairStep::AbortLearner, + FleetPermitPhase::Aborted, + ) .await?; - ensure!(target.activate_learner(f.activation(false)?).await.is_err()); - assert_eq!( - f.step(&authorization, RepairStep::AbortLearner) - .await? - .phase, - RepairPhase::Aborted - ); - f.close().await + Ok(()) + }) + .await } #[tokio::test(flavor = "multi_thread", worker_threads = 4)] async fn fleet_permit_legacy_startup_cannot_seed_managed_history() -> Result<()> { for from_s3 in [false, true] { - let mut f = Fixture::new().await?; - let plan = f.plan(40).await?; - let authorization = f.reserve(1, 20, &plan).await?; - f.handles[f.primary] - .begin_managed_repair(authorization.clone(), plan, TIMEOUT) + qualify(from_s3, async |f| { + let reserved = f.reserve(210).await?; + verify_phase( + &f.begin(f.primary, &reserved).await?, + &reserved, + FleetPermitPhase::Seeding, + )?; + f.write(2, 1, Durability::S3).await?; + f.stop(SPARE).await?; + let config = &f.cluster.fixtures[SPARE]; + let node = start_node( + f.cluster.roots[SPARE].path(), + config.application.clone(), + &config.spec.label, + ) .await?; - f.write(2, 1, true).await?; - let activation = f.activation(from_s3)?; - let placement = if from_s3 { - RaftCellPlacement::JoiningFromS3 { - member: activation.member, - source: activation.source, - source_node: activation.source_node, - } - } else { - RaftCellPlacement::Joining { - member: activation.member, - source: activation.source, - source_node: activation.source_node, - } - }; - ensure!(f - .cluster - .join_standby(f.definitions[3].application.clone(), placement) - .await - .is_err()); - use futures_util::TryStreamExt as _; - let objects = f.cluster.store.list(None).try_collect::>().await?; - ensure!( - !objects.iter().any(|object| object - .location - .as_ref() - .ends_with("/claims/0000000000000004")), - "unguarded startup claimed a managed learner" - ); - assert_eq!(f.calls[3].load(Ordering::SeqCst), 0); - assert_eq!( - f.step(&authorization, RepairStep::AbortLearner) - .await? - .phase, - RepairPhase::Aborted - ); - f.close().await?; + let (mut options, services) = config.config(&node, LogOpenMode::Resume, 45).await?; + options.host.managed = None; + options.groups.push(RaftGroupAssignment { + history: f.cluster.history, + placement: RaftGroupPlacement::Replica, + }); + archives::configure(&mut options); + let before = f.catalog().read(TIMEOUT).await?.revision(); + let refused = node.start_raft_cells(options, services).await; + node.shutdown(ShutdownMode::Preserve).await?; + ensure!( + refused.is_err(), + "unmanaged startup attached a managed replica" + ); + ensure!(f.catalog().read(TIMEOUT).await?.revision() == before); + f.ensure_spare_unclaimed().await?; + ensure!(f.counts()[SPARE] == 0); + f.ensure_pending(&reserved).await?; + f.advance( + &reserved, + RepairStep::AbortLearner, + FleetPermitPhase::Aborted, + ) + .await?; + Ok(()) + }) + .await?; } Ok(()) } diff --git a/crates/cellar/tests/native_raft_host.rs b/crates/cellar/tests/native_raft_host.rs index e4a2f19e3..fb7d18af1 100644 --- a/crates/cellar/tests/native_raft_host.rs +++ b/crates/cellar/tests/native_raft_host.rs @@ -855,7 +855,12 @@ async fn native_axum_command_parity() -> Result<()> { let schema = class.schema_digest(); let namespace = NamespaceId::new([2; 16])?; let secret = "counter-test-authorization"; - let api = counter_example::CounterApi::new(secret.into(), namespace, &class)?; + let api = counter_example::CounterApi::new( + secret.into(), + ClusterId::new([1; 16])?, + namespace, + &class, + )?; let application = counter_example::application(api.clone(), class)?; let node = public_node::node(root.path(), application.clone()).await?; let mut namespace_config = config(2, NamespaceMode::Raft, true); @@ -949,6 +954,7 @@ async fn native_axum_command_parity() -> Result<()> { let response = client .get(url) .bearer_auth(secret) + .header("x-read-consistency", "stale") .header("x-read-policy", "disconnected-historical") .send() .await?; @@ -1100,7 +1106,12 @@ async fn native_axum_nonmember_query_router_parity() -> Result<()> { let namespace = NamespaceId::new([2; 16])?; let secret = "counter-router-authorization"; let class = counter_example::definition()?; - let api = counter_example::CounterApi::new(secret.into(), namespace, &class)?; + let api = counter_example::CounterApi::new( + secret.into(), + ClusterId::new([1; 16])?, + namespace, + &class, + )?; let source = public_node::Counter { schema: class.schema_digest(), increment: class.contract("Increment").unwrap(), @@ -1336,6 +1347,7 @@ async fn public_multi_group_replication_and_leader_failover() -> Result<()> { host: RaftHostOptions::default(), }, RaftCellsServices { + recovery: None, listeners, tls: public_node::tls()?, store: store.clone(), diff --git a/crates/cellar/tests/native_raft_multigroup.rs b/crates/cellar/tests/native_raft_multigroup.rs deleted file mode 100644 index 574b4400c..000000000 --- a/crates/cellar/tests/native_raft_multigroup.rs +++ /dev/null @@ -1,312 +0,0 @@ -// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. - -//! Multi-group consensus scheduling and real-engine process-crash boundaries. -//! The scheduler fixture's application markers are stubs. The repair-load -//! fixture and container runner exercise actual native SQLite/LTX publication. -#![allow(clippy::disallowed_methods)] -#[path = "support/raft_archive_reclaim.rs"] -mod archive_reclaim; -#[path = "support/raft_archive_wakeup.rs"] -mod archive_wakeup; -#[path = "support/raft_background_admission.rs"] -mod background_admission; -#[allow(dead_code)] -#[path = "support/raft_catalog_fault.rs"] -mod bundle_faults; -#[allow(dead_code)] -#[path = "support/raft_ltx_fixture.rs"] -mod bundle_images; -#[path = "support/raft_bundle_native.rs"] -mod bundles; -#[path = "support/raft_storage_fault.rs"] -mod fault; -#[allow(dead_code)] // Shared fixture also serves snapshot, peer-fault and lifecycle targets. -#[path = "support/raft_runtime_fixture.rs"] -mod fixture; -#[allow(dead_code)] // Shared native fixture also serves membership and snapshot drills. -#[path = "support/raft_cluster.rs"] -mod native_cluster; -#[allow(dead_code)] -#[path = "support/raft_node.rs"] -mod public_node; -#[path = "support/raft_repair_load.rs"] -mod repair_load; - -use anyhow::{ensure, Result}; -use cellar::application::consensus::verification::*; -use cellar::sdk::ExecutionDomain; -use cellar_logic::consensus::OperationId; -use cellar_ltx::DirectFileSystem; -use fixture::*; -use raft::{prelude::Entry, GetEntriesContext, Storage}; -use std::{path::Path, sync::Arc, time::Duration}; - -#[tokio::test(flavor = "multi_thread", worker_threads = 4)] -async fn archived_progress_wakes_commands_without_maintenance_poll() -> Result<()> { - archive_wakeup::run().await -} - -#[tokio::test(flavor = "multi_thread", worker_threads = 4)] -async fn background_sql_admits_one_pending_command() -> Result<()> { - background_admission::run(false).await -} - -#[tokio::test(flavor = "multi_thread", worker_threads = 4)] -async fn cancelled_command_waiting_for_background_sql_does_not_execute() -> Result<()> { - background_admission::run(true).await -} - -#[tokio::test(flavor = "multi_thread", worker_threads = 4)] -async fn repair_budget_under_load() -> Result<()> { - repair_load::run().await -} - -#[tokio::test(flavor = "multi_thread", worker_threads = 4)] -async fn archive_bundle_group_index() -> Result<()> { - bundles::run().await -} - -#[tokio::test(flavor = "multi_thread", worker_threads = 2)] -async fn many_groups_mixed_roles() -> Result<()> { - let ids: Vec = (10..22).collect(); - let mut cluster = Cluster::new(3, &ids).await?; - for (index, &id) in ids.iter().enumerate() { - cluster.elect(id, index % 3).await?; - } - cluster.settled(&ids).await?; - for (index, &id) in ids.iter().enumerate() { - let leader = index % 3; - let receipt = cluster.hosts[leader] - .runtime - .propose(&scope(id), vec![id], vec![id; 73]) - .await?; - for member in 0..3 { - cluster - .applied_to(id, member, receipt.position.index.get()) - .await?; - let state = cluster.hosts[member].runtime.status(&scope(id)).await?; - assert_eq!(state.is_leader, member == leader); - let log = cluster.hosts[member].log.group(&scope(id))?; - let entries = log.entries( - receipt.position.index.get(), - receipt.position.index.get() + 1, - None, - GetEntriesContext::empty(false), - )?; - assert_eq!(&entries[0].data[..], &[id; 73]); - } - } - cluster.close().await -} - -#[tokio::test(flavor = "multi_thread", worker_threads = 2)] -async fn hot_group_fairness() -> Result<()> { - let mut cluster = Cluster::new(3, &[10, 11, 12]).await?; - cluster.elect(10, 0).await?; - cluster.elect(11, 1).await?; - cluster.settled(&[10, 11]).await?; - // Fill one leader's per-group queue behind a retained apply. Other groups - // must still complete writes and a previously unelected group must elect. - cluster.hold_apply.insert((0, scope(10).group)); - cluster.hosts[0] - .runtime - .propose(&scope(10), vec![1], vec![10; 4096]) - .await?; - tokio::time::timeout(Duration::from_secs(5), async { - while cluster.held_apply.is_empty() { - cluster.pump().await?; - } - Ok::<_, anyhow::Error>(()) - }) - .await??; - let mut flood = tokio::task::JoinSet::new(); - for index in 0..64u8 { - let runtime = cluster.hosts[0].runtime.clone(); - flood.spawn(async move { - runtime - .propose(&scope(10), vec![index], vec![index; 4096]) - .await - }); - } - let receipt = cluster.hosts[1] - .runtime - .propose(&scope(11), vec![2], vec![11; 1024]) - .await?; - for member in 0..3 { - cluster - .applied_to(11, member, receipt.position.index.get()) - .await?; - } - cluster.elect(12, 2).await?; - let elected = cluster.hosts[2].runtime.status(&scope(12)).await?; - assert!(elected.is_leader && elected.protocol_ready); - assert!( - !cluster.held_apply.is_empty(), - "hot apply remained held throughout independent progress" - ); - // Cancellation drops request waiters; runtime closure physically joins the - // retained work. No accepted proposal is claimed to be a committed result. - flood.abort_all(); - while let Some(result) = flood.join_next().await { - match result { - Ok(result) => { - assert!(result.is_ok() || matches!(result, Err(RuntimeError::Overloaded))) - } - Err(error) => ensure!(error.is_cancelled(), "hot caller panicked: {error}"), - } - } - cluster.close().await -} - -async fn open_batch( - path: &Path, - mode: LogOpenMode, - process: u8, - fs: Arc, -) -> Result<(ExecutionDomain, ReplicaLog)> { - let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); - let log = domain - .run(ReplicaLog::open( - storage_options(path, 1, process, mode), - fs, - )) - .await??; - Ok((domain, log)) -} - -async fn persist_batch(log: &ReplicaLog, index: u64) -> Result<()> { - let mut writes = Vec::new(); - for id in [10, 11, 12] { - let group = log.group(&scope(id))?; - let mut write = GroupWrite::new(&group); - let mut hard = group.initial_state()?.hard_state; - hard.commit = index; - write.hard_state = Some(hard); - write.entries.push(Entry { - index, - term: 1, - data: vec![id; index as usize * 31].into(), - ..Default::default() - }); - writes.push(write); - } - let receipt = log.persist(OperationId::new(index), writes).await?; - assert_eq!(receipt.groups.len(), 3); - Ok(()) -} - -struct Child(std::process::Child); - -impl Drop for Child { - fn drop(&mut self) { - let _ = self.0.kill(); - let _ = self.0.wait(); - } -} - -#[tokio::test(flavor = "multi_thread", worker_threads = 2)] -async fn cross_group_batch_crash() -> Result<()> { - for cut in [fault::CRASH_BEFORE_SYNC, fault::CRASH_AFTER_SYNC, 0] { - let root = tempfile::tempdir()?; - let path = root.path().join("node"); - let signal = root.path().join("cut.signal"); - let (domain, log) = - open_batch(&path, LogOpenMode::Create, 1, Arc::new(DirectFileSystem)).await?; - for id in [10, 11, 12] { - seed_group(&log, id, 1, 3).await?; - } - persist_batch(&log, 2).await?; - log.close().await?; - domain.shutdown().await; - let output = root.path().join("child.log"); - let file = std::fs::File::create(&output)?; - let mut child = Child( - std::process::Command::new(std::env::current_exe()?) - .args(["--ignored", "--exact", "batch_crash_child", "--nocapture"]) - .env("CELLAR_BATCH_ROOT", &path) - .env("CELLAR_BATCH_SIGNAL", &signal) - .env("CELLAR_BATCH_CUT", cut.to_string()) - .stdout(file.try_clone()?) - .stderr(file) - .spawn()?, - ); - tokio::time::timeout(Duration::from_secs(20), async { - while !signal.exists() { - ensure!( - child.0.try_wait()?.is_none(), - "child exited before cut: {}", - std::fs::read_to_string(&output)? - ); - tokio::time::sleep(Duration::from_millis(5)).await; - } - Ok::<_, anyhow::Error>(()) - }) - .await??; - child.0.kill()?; - ensure!(!child.0.wait()?.success()); - let (domain, log) = - open_batch(&path, LogOpenMode::Resume, 3, Arc::new(DirectFileSystem)).await?; - for id in [10, 11, 12] { - let group = log.group(&scope(id))?; - let committed = group.initial_state()?.hard_state.commit; - if cut == fault::CRASH_BEFORE_SYNC { - assert!([2, 3].contains(&committed)); - } else { - assert_eq!(committed, 3); - } - assert_eq!(group.last_index()?, committed); - for index in 2..=committed { - let entries = - group.entries(index, index + 1, None, GetEntriesContext::empty(false))?; - assert_eq!(&entries[0].data[..], &vec![id; index as usize * 31]); - } - } - log.close().await?; - domain.shutdown().await; - } - Ok(()) -} - -#[tokio::test(flavor = "multi_thread", worker_threads = 2)] -#[ignore = "process-kill helper launched by cross_group_batch_crash"] -async fn batch_crash_child() -> Result<()> { - let path = std::env::var("CELLAR_BATCH_ROOT")?; - let signal = std::path::PathBuf::from(std::env::var("CELLAR_BATCH_SIGNAL")?); - let cut = std::env::var("CELLAR_BATCH_CUT")?.parse()?; - let gate = fault::Gate::new(Some(signal.clone())); - let (_domain, log) = open_batch( - Path::new(&path), - LogOpenMode::Resume, - 2, - Arc::new(fault::FaultFileSystem { gate: gate.clone() }), - ) - .await?; - let _release = gate.arm(cut); - persist_batch(&log, 3).await?; - if cut == 0 { - std::fs::write(&signal, b"acknowledged")?; - std::fs::File::open(signal)?.sync_all()?; - } - std::future::pending::<()>().await; - Ok(()) -} - -#[test] -fn harness_rejects_stale_binary() -> Result<()> { - let root = Path::new(env!("CARGO_MANIFEST_DIR")).join("../.."); - let output = std::process::Command::new("python3") - .args(["-m", "unittest", "test_raft_run.Freshness"]) - .current_dir(root.join("tools/perf")) - .output()?; - ensure!( - output.status.success(), - "image refusal cases failed: {}", - String::from_utf8_lossy(&output.stderr) - ); - Ok(()) -} - -#[tokio::test(flavor = "multi_thread", worker_threads = 4)] -async fn short_archived_groups_checkpoint_and_recover_without_reexecuting() -> Result<()> { - archive_reclaim::run().await -} diff --git a/crates/cellar/tests/native_raft_reads.rs b/crates/cellar/tests/native_raft_reads.rs index 9b122b5a1..796231b53 100644 --- a/crates/cellar/tests/native_raft_reads.rs +++ b/crates/cellar/tests/native_raft_reads.rs @@ -12,16 +12,10 @@ mod ordinary_context; mod public_node; #[path = "support/raft_readiness_native.rs"] mod readiness; -#[path = "support/raft_catalog_fault.rs"] -mod store; use anyhow::{Context as _, Result}; -use base64::{engine::general_purpose::URL_SAFE_NO_PAD, Engine as _}; use cellar::sdk::consensus::{verification::*, *}; -use std::{ - sync::{atomic::Ordering, Arc}, - time::Duration, -}; +use std::{sync::Arc, time::Duration}; type Query = dyn for<'a> Fn(&QueryContext<'a>, &[u8]) -> Result> + Send + Sync; @@ -224,450 +218,6 @@ async fn leader(handles: &[RaftCellHandle], excluded: Option) -> Result Result<()> { - let fixture = Fixture::start(Arc::new(get)).await?; - let primary = leader(&fixture.handles, None).await?; - let follower = (primary + 1) % 3; - fixture.cluster.isolate(follower, false).await?; - let written = fixture.write(primary, 2, 4).await?; - assert_eq!(written.result, 5_i64.to_be_bytes()); - let pending = - fixture.handles[follower].query(fixture.definition.get.clone(), vec![], at_least(&written)); - tokio::pin!(pending); - assert!( - tokio::time::timeout(Duration::from_millis(100), &mut pending) - .await - .is_err() - ); - let historical = fixture.handles[follower] - .query( - fixture.definition.get.clone(), - vec![], - QueryOptions { - stale_policy: StaleReadPolicy::DisconnectedHistorical, - ..Default::default() - }, - ) - .await?; - assert_eq!(historical.result, 1_i64.to_be_bytes()); - fixture.cluster.heal(); - let read = pending.await?; - assert_eq!(read.result, 5_i64.to_be_bytes()); - assert!(read.version.covers(&written.version)?); - fixture.close().await -} - -fn routed(receipt: &CommandReceipt, member: usize, fallback: bool) -> QueryOptions { - QueryOptions { - routing: QueryRouting::Replicas { - members: vec![MemberId::new(member as u64 + 1).unwrap()], - attempt_timeout: Duration::from_millis(180), - leader_fallback: fallback, - }, - timeout: Duration::from_secs(2), - ..at_least(receipt) - } -} - -#[tokio::test(flavor = "multi_thread", worker_threads = 8)] -async fn fallback_preserves_contract() -> Result<()> { - let fixture = Fixture::start(Arc::new(get)) - .await - .context("fallback fixture startup")?; - let primary = leader(&fixture.handles, None) - .await - .context("fallback initial leader")?; - let source = (primary + 1) % 3; - let target = (primary + 2) % 3; - fixture.cluster.isolate_consensus(target).await?; - let newer = fixture - .write(primary, 2, 4) - .await - .context("fallback committed command")?; - assert_eq!(newer.result, 5_i64.to_be_bytes()); - // A real RPC can reach the isolated replica's older image. It must wait - // for the exact token rather than answer its stale value of one. - assert!(matches!( - fixture.handles[source] - .query( - fixture.definition.get.clone(), - vec![], - routed(&newer, target, false) - ) - .await, - Err(CommandFailure::VersionUnavailable) - )); - let read = fixture.handles[source] - .query( - fixture.definition.get.clone(), - vec![], - routed(&newer, target, true), - ) - .await - .context("minimum-version leader fallback")?; - assert_eq!(read.result, 5_i64.to_be_bytes()); - assert_eq!(read.served_by, MemberId::new(primary as u64 + 1)?); - assert!(read.version.covers(&newer.version)?); - let mut strong = routed(&newer, target, true); - strong.consistency = QueryConsistency::Linearizable; - let read = fixture.handles[source] - .query(fixture.definition.get.clone(), vec![], strong) - .await - .context("linearizable leader fallback")?; - assert_eq!(read.result, 5_i64.to_be_bytes()); - assert_eq!(read.served_by, MemberId::new(primary as u64 + 1)?); - // The candidate's longer attempt budget and fallback cannot restart the - // original invocation deadline. Both nodes must release cancelled work. - let mut short = routed(&newer, target, true); - short.timeout = Duration::from_millis(100); - short.routing = QueryRouting::Replicas { - members: vec![MemberId::new(target as u64 + 1)?], - attempt_timeout: Duration::from_secs(1), - leader_fallback: true, - }; - assert!(matches!( - tokio::time::timeout( - Duration::from_millis(500), - fixture.handles[source].query(fixture.definition.get.clone(), vec![], short) - ) - .await?, - Err(CommandFailure::VersionUnavailable) - )); - tokio::time::timeout(Duration::from_millis(500), async { - while [source, target].iter().any(|i| { - fixture.cluster.nodes[*i] - .services() - .application_work() - .pending_events - != 0 - }) { - tokio::time::sleep(Duration::from_millis(5)).await; - } - }) - .await?; - fixture.cluster.heal(); - fixture.close().await -} - -#[tokio::test(flavor = "multi_thread", worker_threads = 8)] -async fn remote_query_wait_cannot_block_its_ltx_catchup() -> Result<()> { - let fixture = Fixture::start(Arc::new(get)).await?; - let primary = leader(&fixture.handles, None).await?; - let source = (primary + 1) % 3; - let target = (primary + 2) % 3; - fixture.cluster.isolate_consensus(target).await?; - let newer = fixture.write(primary, 2, 4).await?; - let mut options = routed(&newer, target, false); - options.timeout = Duration::from_secs(5); - options.routing = QueryRouting::Replicas { - members: vec![MemberId::new(target as u64 + 1)?], - attempt_timeout: Duration::from_secs(5), - leader_fallback: false, - }; - let query = fixture.handles[source].query(fixture.definition.get.clone(), vec![], options); - tokio::time::timeout(Duration::from_secs(2), async { - while fixture.cluster.nodes[target] - .services() - .application_work() - .pending_events - == 0 - { - tokio::time::sleep(Duration::from_millis(5)).await; - } - }) - .await?; - fixture.cluster.heal(); - let read = query.await?; - assert_eq!(read.result, 5_i64.to_be_bytes()); - assert_eq!(read.served_by, MemberId::new(target as u64 + 1)?); - assert!(read.version.covers(&newer.version)?); - fixture.close().await -} - -#[tokio::test(flavor = "multi_thread", worker_threads = 8)] -async fn saturated_query_peer_refuses_and_expires_queued_work() -> Result<()> { - use std::sync::atomic::AtomicUsize; - let (gate, mut entered) = public_node::Gate::new(); - let release = public_node::Release(gate.clone()); - let unexpected = Arc::new(AtomicUsize::new(0)); - let executed = unexpected.clone(); - let fixture = Fixture::start(Arc::new(move |context, input| { - if input == b"queued" { - executed.fetch_add(1, Ordering::SeqCst); - } - if input == b"hold" { - gate.wait()?; - } - get(context, input) - })) - .await?; - let primary = leader(&fixture.handles, None).await?; - let source = (primary + 1) % 3; - let target = (primary + 2) % 3; - let options = |timeout| QueryOptions { - timeout, - routing: QueryRouting::Replicas { - members: vec![MemberId::new(target as u64 + 1).unwrap()], - attempt_timeout: timeout, - leader_fallback: false, - }, - ..at_least(&fixture.initial) - }; - // Hold the real peer connection in its destination callback. Later calls - // must consume finite peer bytes while queued behind this invocation. - let first = fixture.handles[source].query( - fixture.definition.get.clone(), - b"hold".to_vec(), - options(Duration::from_secs(5)), - ); - tokio::time::timeout(Duration::from_secs(2), entered.recv()) - .await? - .ok_or_else(|| anyhow::anyhow!("remote callback did not enter"))?; - let source_usage = fixture.cluster.services[source] - .resource_usage() - .transport - .context("source transport observer")?; - let target_usage = fixture.cluster.services[target] - .resource_usage() - .transport - .context("target transport observer")?; - assert!(source_usage.outgoing.query.items > 0 && source_usage.outgoing.query.bytes > 0); - assert!(target_usage.incoming.query.items > 0 && target_usage.incoming.query.bytes > 0); - assert!(source_usage.outgoing_connections.query > 0); - assert!(target_usage.incoming_connections.query > 0); - assert!( - target_usage.incoming_connections.query <= target_usage.incoming_connection_limits.query - ); - assert!(source_usage.outgoing.query.bytes <= source_usage.outgoing.query.max_bytes); - assert!(target_usage.incoming.query.bytes <= target_usage.incoming.query.max_bytes); - let queued: Vec<_> = (0..64) - .map(|_| { - fixture.handles[source].query( - fixture.definition.get.clone(), - b"queued".to_vec(), - options(Duration::from_secs(1)), - ) - }) - .collect(); - let mut overloaded = 0; - let mut expired = 0; - tokio::time::timeout(Duration::from_secs(2), async { - for response in queued { - match response.await { - Err(CommandFailure::Overloaded) => overloaded += 1, - Err(CommandFailure::VersionUnavailable) => expired += 1, - result => anyhow::bail!("unexpected queued response: {result:?}"), - } - } - Ok::<_, anyhow::Error>(()) - }) - .await??; - assert!(overloaded > 0, "peer saturation was not observed"); - assert!( - expired > 0, - "no admitted request expired behind the held RPC" - ); - assert_eq!(unexpected.load(Ordering::SeqCst), 0); - // Consensus and LTX use separate connections/capacity. They must publish a - // new command while the first query and expired peer queue are still held. - let newer = fixture.write(primary, 2, 4).await?; - assert_eq!(newer.result, 5_i64.to_be_bytes()); - release.0.release(); - let first = first.await?; - assert_eq!(first.result, 1_i64.to_be_bytes()); - assert_eq!(first.served_by, MemberId::new(target as u64 + 1)?); - assert!(first.version.position.index < newer.version.position.index); - let mut fresh = options(Duration::from_secs(2)); - fresh.consistency = QueryConsistency::AtLeast(newer.version_token.clone()); - let read = fixture.handles[source] - .query(fixture.definition.get.clone(), vec![], fresh) - .await?; - assert_eq!(read.result, 5_i64.to_be_bytes()); - assert!(read.version.covers(&newer.version)?); - assert_eq!( - unexpected.load(Ordering::SeqCst), - 0, - "expired queued request executed after the peer resumed" - ); - tokio::time::timeout(Duration::from_secs(2), async { - while fixture - .cluster - .nodes - .iter() - .any(|node| node.services().application_work().pending_events != 0) - { - tokio::time::sleep(Duration::from_millis(5)).await; - } - }) - .await?; - let performance: Vec<_> = fixture - .cluster - .services - .iter() - .map(RaftCells::performance) - .collect(); - assert!(performance - .iter() - .all(|sample| sample.available && !sample.saturated)); - for lane in ["control", "data", "query"] { - for direction in ["read", "write"] { - let name = format!("transport_{lane}_{direction}_bytes"); - let bytes: u64 = performance - .iter() - .map(|sample| { - sample - .counters - .iter() - .find(|counter| counter.name == name) - .unwrap() - .value - }) - .sum(); - assert!(bytes > 0, "real TLS traffic missing from {name}"); - } - } - for lane in ["control", "data"] { - for phase in ["queue", "exchange", "admit"] { - let name = format!("transport_{lane}_{phase}"); - let successes: u64 = performance - .iter() - .map(|sample| { - sample - .operations - .iter() - .find(|operation| operation.name == name) - .unwrap() - .succeeded - }) - .sum(); - assert!(successes > 0, "real TLS traffic missing from {name}"); - } - } - fixture.close().await -} - -#[tokio::test(flavor = "multi_thread", worker_threads = 8)] -async fn delayed_tcp_query_reply_cannot_settle_a_later_read() -> Result<()> { - let (gate, mut entered) = public_node::Gate::new(); - let release = public_node::Release(gate.clone()); - let fixture = Fixture::start(Arc::new(move |context, input| { - if input == b"hold" { - gate.wait()?; - } - get(context, input) - })) - .await?; - let primary = leader(&fixture.handles, None).await?; - let source = (primary + 1) % 3; - let target = (primary + 2) % 3; - let options = QueryOptions { - timeout: Duration::from_secs(1), - routing: QueryRouting::Replicas { - members: vec![MemberId::new(target as u64 + 1)?], - attempt_timeout: Duration::from_secs(1), - leader_fallback: false, - }, - ..at_least(&fixture.initial) - }; - // Warm the pooled TLS connection before holding server-to-client bytes, - // so this history delays a query response rather than its handshake. - let warm = fixture.handles[source] - .query(fixture.definition.get.clone(), vec![], options.clone()) - .await?; - assert_eq!(warm.served_by, MemberId::new(target as u64 + 1)?); - let delayed = fixture.handles[source].query( - fixture.definition.get.clone(), - b"hold".to_vec(), - options.clone(), - ); - tokio::time::timeout(Duration::from_secs(2), entered.recv()) - .await? - .ok_or_else(|| anyhow::anyhow!("remote query missing"))?; - let held = fixture.cluster.hold_query_responses(source, target)?; - release.0.release(); - tokio::time::timeout(Duration::from_millis(750), async { - while fixture.cluster.held_query_response_bytes(source, target) == 0 - || fixture.cluster.nodes[target] - .services() - .application_work() - .pending_events - != 0 - { - tokio::time::sleep(Duration::from_millis(5)).await; - } - }) - .await?; - let newer = fixture.write(primary, 2, 4).await?; - assert_eq!(newer.result, 5_i64.to_be_bytes()); - assert!(matches!( - tokio::time::timeout(Duration::from_secs(2), delayed).await?, - Err(CommandFailure::VersionUnavailable) - )); - // Dropping the timed-out connection discards its delayed ciphertext. A - // later attempt must perform its own exchange and cover the newer token. - drop(held); - let read = fixture.handles[source] - .query( - fixture.definition.get.clone(), - vec![], - QueryOptions { - consistency: QueryConsistency::AtLeast(newer.version_token.clone()), - ..options - }, - ) - .await?; - assert_eq!(read.result, 5_i64.to_be_bytes()); - assert_eq!(read.served_by, MemberId::new(target as u64 + 1)?); - assert!(read.version.covers(&newer.version)?); - tokio::time::timeout(Duration::from_secs(2), async { - while fixture - .cluster - .nodes - .iter() - .any(|node| node.services().application_work().pending_events != 0) - { - tokio::time::sleep(Duration::from_millis(5)).await; - } - }) - .await?; - fixture.close().await -} - -#[tokio::test(flavor = "multi_thread", worker_threads = 8)] -async fn voter_without_query_code_can_route_to_a_compatible_replica() -> Result<()> { - let definitions = vec![ - counter_query(None)?, - counter(Arc::new(get))?, - counter(Arc::new(get))?, - ]; - assert_eq!(definitions[0].schema, definitions[1].schema); - let fixture = Fixture::start_definitions(definitions, RaftHostOptions::default()).await?; - readiness::without_query_code(&fixture.handles[0]).await?; - assert!(matches!( - fixture.handles[0] - .query( - fixture.definition.get.clone(), - vec![], - at_least(&fixture.initial) - ) - .await, - Err(CommandFailure::UnsupportedCapability) - )); - let mut options = at_least(&fixture.initial); - options.routing = QueryRouting::Replicas { - members: vec![MemberId::new(1)?, MemberId::new(2)?], - attempt_timeout: Duration::from_millis(500), - leader_fallback: false, - }; - let read = fixture.handles[0] - .query(fixture.definition.get.clone(), vec![], options) - .await?; - assert_eq!(read.result, 1_i64.to_be_bytes()); - assert_eq!(read.served_by, MemberId::new(2)?); - fixture.close().await -} - #[tokio::test(flavor = "multi_thread", worker_threads = 8)] async fn nonmember_router_reads_only_granted_namespaces_without_local_state() -> Result<()> { let mut definitions = (0..3) @@ -862,384 +412,6 @@ async fn nonmember_router_reads_only_granted_namespaces_without_local_state() -> cluster.close().await } -#[tokio::test(flavor = "multi_thread", worker_threads = 8)] -async fn lost_read_index_response_retries_without_leader_change() -> Result<()> { - let fixture = Fixture::start(Arc::new(get)).await?; - let primary = leader(&fixture.handles, None).await?; - let follower = (primary + 1) % 3; - fixture.handles[follower] - .query( - fixture.definition.get.clone(), - vec![], - at_least(&fixture.initial), - ) - .await?; - let before = fixture.handles[follower].status().await?; - fixture - .cluster - .block_control(primary, follower, true) - .await?; - let pending = fixture.handles[follower].query( - fixture.definition.get.clone(), - vec![], - QueryOptions { - consistency: QueryConsistency::Linearizable, - timeout: Duration::from_secs(2), - ..Default::default() - }, - ); - // The request can reach the leader, and the third voter supplies its safe - // ReadIndex majority. Only the reply direction is unavailable. Heal before - // the follower's minimum election timeout, keeping the term/leader fixed. - tokio::time::sleep(Duration::from_millis(200)).await; - fixture - .cluster - .block_control(primary, follower, false) - .await?; - let result = pending.await; - let after = fixture.handles[follower].status().await?; - assert_eq!(after.term, before.term); - assert_eq!(after.leader, before.leader); - fixture.close().await?; - assert_eq!( - result - .context("a lost ReadIndex response must be retried under the original invocation")? - .result, - 1i64.to_be_bytes() - ); - Ok(()) -} - -#[tokio::test(flavor = "multi_thread", worker_threads = 8)] -async fn linearizable_read_barrier() -> Result<()> { - let fixture = Fixture::start(Arc::new(get)).await?; - let old = leader(&fixture.handles, None).await?; - let written = fixture.write(old, 2, 4).await?; - let options = QueryOptions { - consistency: QueryConsistency::Linearizable, - ..Default::default() - }; - for (i, handle) in fixture.handles.iter().enumerate() { - let read = handle - .query(fixture.definition.get.clone(), vec![], options.clone()) - .await - .with_context(|| format!("initial linearizable read on replica {i}"))?; - assert_eq!(read.result, 5_i64.to_be_bytes()); - assert!(read.version.covers(&written.version)?); - } - fixture.cluster.isolate(old, false).await?; - let next = leader(&fixture.handles, Some(old)).await?; - // No data command follows the election. Its protocol-only barrier must be - // covered by the same pinned image and returned as the served position. - let read = fixture.handles[next] - .query(fixture.definition.get.clone(), vec![], options.clone()) - .await - .context("new leader protocol-only cut")?; - assert_eq!(read.result, 5_i64.to_be_bytes()); - assert!(read.version.position.index > written.version.position.index); - assert!(read.version.position.term > written.version.position.term); - let newer = fixture.write(next, 3, 7).await?; - assert_eq!(newer.result, 12_i64.to_be_bytes()); - for (i, handle) in fixture - .handles - .iter() - .enumerate() - .filter(|(i, _)| *i != old) - { - let read = handle - .query(fixture.definition.get.clone(), vec![], options.clone()) - .await - .with_context(|| format!("post-election linearizable read on replica {i}"))?; - assert_eq!(read.result, 12_i64.to_be_bytes(), "replica {i}"); - assert!(read.version.covers(&newer.version)?); - } - fixture.cluster.heal(); - let read = fixture.handles[old] - .query(fixture.definition.get.clone(), vec![], options) - .await - .context("healed former leader linearizable read")?; - assert_eq!(read.result, 12_i64.to_be_bytes()); - fixture.close().await -} - -#[tokio::test(flavor = "multi_thread", worker_threads = 8)] -async fn minority_strong_read_refused() -> Result<()> { - let fixture = Fixture::start(Arc::new(get)).await?; - let primary = leader(&fixture.handles, None).await?; - let follower = (primary + 1) % 3; - fixture.cluster.isolate(follower, false).await?; - let newer = fixture.write(primary, 2, 4).await?; - let strong = QueryOptions { - consistency: QueryConsistency::Linearizable, - timeout: Duration::from_millis(250), - ..Default::default() - }; - assert!(matches!( - fixture.handles[follower] - .query(fixture.definition.get.clone(), vec![], strong) - .await, - Err(CommandFailure::VersionUnavailable) - )); - let mut minimum = at_least(&newer); - minimum.timeout = Duration::from_millis(250); - assert!(matches!( - fixture.handles[follower] - .query(fixture.definition.get.clone(), vec![], minimum) - .await, - Err(CommandFailure::VersionUnavailable) - )); - // A minimum-version read can legitimately use an already-covered prefix - // without quorum; it must not be confused with a fresh linearizable read. - assert_eq!( - fixture.handles[follower] - .query( - fixture.definition.get.clone(), - vec![], - at_least(&fixture.initial) - ) - .await? - .result, - 1_i64.to_be_bytes() - ); - assert!(matches!( - fixture.handles[follower] - .query( - fixture.definition.get.clone(), - vec![], - QueryOptions::default() - ) - .await, - Err(CommandFailure::VersionUnavailable) - )); - assert_eq!( - fixture.handles[follower] - .query( - fixture.definition.get.clone(), - vec![], - QueryOptions { - stale_policy: StaleReadPolicy::DisconnectedHistorical, - ..Default::default() - } - ) - .await? - .result, - 1_i64.to_be_bytes() - ); - fixture.cluster.heal(); - assert_eq!( - fixture.handles[follower] - .query(fixture.definition.get.clone(), vec![], at_least(&newer)) - .await? - .result, - 5_i64.to_be_bytes() - ); - fixture.close().await -} - -#[tokio::test(flavor = "multi_thread", worker_threads = 8)] -async fn follower_drains_committed_batch_after_held_reader() -> Result<()> { - let (gate, mut entered) = public_node::Gate::new(); - let release = public_node::Release(gate.clone()); - let fixture = Fixture::start(Arc::new(move |context, input| { - if input == b"hold" { - gate.wait()?; - } - get(context, input) - })) - .await?; - let result = async { - let primary = leader(&fixture.handles, None).await?; - let follower = (primary + 1) % 3; - let pending = fixture.handles[follower].query( - fixture.definition.get.clone(), b"hold".to_vec(), QueryOptions { - timeout: Duration::from_secs(30), - ..at_least(&fixture.initial) - }, - ); - tokio::time::timeout(Duration::from_secs(5), entered.recv()).await? - .context("follower reader did not enter")?; - let mut newest = fixture.initial.clone(); - // The other two durable voters acknowledge while this follower's SQL - // owner is held. Release a real backlog through prepare/marker/publish. - for sequence in 2..=33 { - newest = fixture.write(primary, sequence, 1).await - .with_context(|| format!("write backlog sequence {sequence}"))?; - assert_eq!(newest.result, (sequence as i64).to_be_bytes()); - } - release.0.release(); - assert_eq!(pending.await.context("held reader completion")?.result, 1_i64.to_be_bytes()); - for (index, handle) in fixture.handles.iter().enumerate() { - let read = handle.query(fixture.definition.get.clone(), vec![], QueryOptions { - timeout: Duration::from_secs(10), - ..at_least(&newest) - }).await; - if let Err(error) = &read { - let mut statuses = Vec::new(); - for handle in &fixture.handles { - statuses.push(handle.status().await.map(|status| ( - status.committed, - status.applied.index, - status.version.map(|version| version.position.index), - status.command_ready, - ))); - } - anyhow::bail!("replica {index} did not drain backlog: {error:?}; committed/applied/image/ready={statuses:?}"); - } - let read = read?; - assert_eq!(read.result, 33_i64.to_be_bytes()); - } - let final_write = fixture.write(primary, 34, 1).await?; - let read = fixture.handles[follower].query( - fixture.definition.get.clone(), vec![], at_least(&final_write), - ).await?; - assert_eq!(read.result, 34_i64.to_be_bytes()); - Ok::<_, anyhow::Error>(()) - }.await; - release.0.release(); - let cleanup = fixture.close().await; - result.and(cleanup) -} - -#[tokio::test(flavor = "multi_thread", worker_threads = 8)] -async fn stale_is_committed_prefix() -> Result<()> { - let (gate, mut entered) = public_node::Gate::new(); - let _release = public_node::Release(gate.clone()); - let query = Arc::new(move |context: &QueryContext<'_>, input: &[u8]| { - let mut value = get(context, input)?; - if input == b"hold" { - let index = context.version().position.index.get(); - gate.wait()?; - assert_eq!(get(context, input)?, value); - value.extend_from_slice(&index.to_be_bytes()); - } - Ok(value) - }); - let fixture = Fixture::start(query).await?; - let primary = leader(&fixture.handles, None).await?; - let follower = (primary + 1) % 3; - let pending = fixture.handles[follower].query( - fixture.definition.get.clone(), - b"hold".to_vec(), - QueryOptions { - stale_policy: StaleReadPolicy::DisconnectedHistorical, - ..Default::default() - }, - ); - tokio::time::timeout(Duration::from_secs(5), entered.recv()) - .await? - .ok_or_else(|| anyhow::anyhow!("query did not pin"))?; - let newer = fixture.write(primary, 2, 4).await?; - assert_eq!(newer.result, 5_i64.to_be_bytes()); - _release.0.release(); - let read = pending.await?; - assert_eq!(&read.result[..8], &1_i64.to_be_bytes()); - assert_eq!( - u64::from_be_bytes(read.result[8..].try_into()?), - read.version.position.index.get() - ); - assert!(read.version.position.index < newer.version.position.index); - let latest = fixture.handles[follower] - .query(fixture.definition.get.clone(), vec![], at_least(&newer)) - .await?; - assert_eq!(latest.result, 5_i64.to_be_bytes()); - fixture.close().await -} - -#[tokio::test(flavor = "multi_thread", worker_threads = 8)] -async fn query_write_capability_denied() -> Result<()> { - let query = Arc::new(|context: &QueryContext<'_>, input: &[u8]| { - if input == b"attack" { - for sql in [ - "UPDATE counter SET value=999 RETURNING value", - "CREATE TABLE injected(value INTEGER)", - "PRAGMA journal_mode=WAL", - "ATTACH ':memory:' AS injected", - "DELETE FROM counter RETURNING value", - ] { - let error = context - .query_row(sql, [], |row| row.get::<_, i64>(0)) - .unwrap_err(); - let sqlite = error - .downcast_ref::() - .ok_or_else(|| anyhow::anyhow!("missing SQLite refusal: {error}"))?; - assert!( - matches!(sqlite, rusqlite::Error::SqliteFailure(error, _) if error.code == rusqlite::ErrorCode::AuthorizationForStatementDenied) - ); - } - } - get(context, input) - }); - let fixture = Fixture::start(query).await?; - for handle in &fixture.handles { - let result = handle - .query( - fixture.definition.get.clone(), - b"attack".to_vec(), - at_least(&fixture.initial), - ) - .await?; - assert_eq!(result.result, 1_i64.to_be_bytes()); - } - let primary = leader(&fixture.handles, None).await?; - assert_eq!( - fixture.write(primary, 2, 4).await?.result, - 5_i64.to_be_bytes() - ); - fixture.close().await -} - -#[tokio::test(flavor = "multi_thread", worker_threads = 8)] -async fn unpolled_query_deadline_interrupts_sql_and_joins_work() -> Result<()> { - let (entered, mut receive) = tokio::sync::mpsc::unbounded_channel(); - let query = Arc::new(move |context: &QueryContext<'_>, input: &[u8]| { - if input == b"slow" { - entered.send(())?; - let _: i64 = context.query_row("WITH RECURSIVE n(x) AS (VALUES(1) UNION ALL SELECT x+1 FROM n WHERE x<1000000000) SELECT sum(x) FROM n", [], |row| row.get(0))?; - } - get(context, input) - }); - let fixture = Fixture::start(query).await?; - let primary = leader(&fixture.handles, None).await?; - let pending = fixture.handles[primary].query( - fixture.definition.get.clone(), - b"slow".to_vec(), - QueryOptions { - timeout: Duration::from_millis(150), - ..at_least(&fixture.initial) - }, - ); - tokio::time::timeout(Duration::from_secs(5), receive.recv()) - .await? - .ok_or_else(|| anyhow::anyhow!("query missing"))?; - tokio::time::timeout(Duration::from_secs(3), async { - while fixture.cluster.nodes[primary] - .services() - .application_work() - .pending_events - != 0 - { - tokio::time::sleep(Duration::from_millis(10)).await; - } - }) - .await?; - assert!(matches!( - pending.await, - Err(CommandFailure::VersionUnavailable) - )); - assert_eq!( - fixture.handles[primary] - .query( - fixture.definition.get.clone(), - vec![], - at_least(&fixture.initial) - ) - .await? - .result, - 1_i64.to_be_bytes() - ); - fixture.close().await -} - fn scope() -> GroupScope { GroupScope { cluster: ClusterId::new([1; 16]).unwrap(), @@ -1249,216 +421,3 @@ fn scope() -> GroupScope { incarnation: IncarnationId::new([4; 16]).unwrap(), } } - -fn version() -> CommittedVersion { - CommittedVersion { - scope: scope(), - position: LogPosition { - index: LogIndex::new(7), - term: Term::new(2), - }, - } -} - -fn keys(secret: u8, until: u64, retained: Vec) -> Result { - Ok(VersionKeys::new( - VersionSigningKey::new([secret; 32], until)?, - retained, - Duration::from_millis(1000), - )?) -} - -#[test] -fn query_token_auth_scope() -> Result<()> { - let ring = keys(11, 10000, vec![])?; - let expected = version(); - let token = ring.issue(&expected, [7; 32], 1000)?; - assert_eq!( - ring.verify(&token, &expected.scope, [7; 32], 1000)?, - expected - ); - assert_eq!(VersionToken::from_encoded(token.as_str())?, token); - for mismatch in 0..5 { - let mut other = expected.scope.clone(); - match mismatch { - 0 => other.cluster = ClusterId::new([99; 16])?, - 1 => other.namespace = NamespaceId::new([99; 16])?, - 2 => other.cell = "Counter:other".into(), - 3 => other.group = GroupId::new([99; 16])?, - _ => other.incarnation = IncarnationId::new([99; 16])?, - } - assert_eq!( - ring.verify(&token, &other, [7; 32], 1000), - Err(VersionError::WrongHistory) - ); - } - assert_eq!( - ring.verify(&token, &scope(), [8; 32], 1000), - Err(VersionError::WrongSchema) - ); - assert_eq!( - ring.verify(&token, &scope(), [7; 32], 999), - Err(VersionError::Expired) - ); - assert_eq!( - ring.verify(&token, &scope(), [7; 32], 2000), - Err(VersionError::Expired) - ); - assert!(keys(12, 10000, vec![])? - .verify(&token, &scope(), [7; 32], 1000) - .is_err()); - let raw = URL_SAFE_NO_PAD.decode(token.as_str())?; - // Alter every byte of an otherwise valid observation: framing, identity, - // position, timestamp, key ID and signature must all be protected. - for offset in 0..raw.len() { - let mut forged = raw.clone(); - forged[offset] ^= 1; - if let Ok(forged) = VersionToken::from_encoded(&URL_SAFE_NO_PAD.encode(forged)) { - assert!( - ring.verify(&forged, &scope(), [7; 32], 1000).is_err(), - "accepted modified byte {offset}" - ); - } - } - assert!(VersionToken::from_encoded(&"a".repeat(100000)).is_err()); - assert!(VersionToken::from_encoded(&format!("{}=", token.as_str())).is_err()); - Ok(()) -} - -#[tokio::test(flavor = "multi_thread", worker_threads = 8)] -async fn completed_unpolled_responses_keep_their_byte_budget() -> Result<()> { - let fixture = Fixture::start_with_host( - Arc::new(|context, input| { - if input == b"large" { - Ok(vec![17; 60000]) - } else { - get(context, input) - } - }), - RaftHostOptions { - max_pending_bytes: 1024 * 1024, - max_input_bytes: 4096, - ..Default::default() - }, - ) - .await?; - let primary = leader(&fixture.handles, None).await?; - let mut responses = Vec::new(); - // These requests complete on the node, but their futures remain unpolled. - // Sixteen retained payloads fit; admitting a seventeenth payload must not - // reuse byte capacity released merely by producer completion. - for _ in 0..16 { - responses.push(fixture.handles[primary].query( - fixture.definition.get.clone(), - b"large".to_vec(), - at_least(&fixture.initial), - )); - tokio::time::timeout(Duration::from_secs(5), async { - while fixture.cluster.nodes[primary] - .services() - .application_work() - .pending_events - != 0 - { - tokio::time::sleep(Duration::from_millis(5)).await; - } - }) - .await?; - } - assert!(matches!( - fixture.handles[primary] - .query( - fixture.definition.get.clone(), - b"large".to_vec(), - at_least(&fixture.initial) - ) - .await, - Err(CommandFailure::Overloaded) - )); - // Consume one charged result and free enough space for exactly one more. - let read = responses.remove(0).await?; - assert_eq!(read.result, vec![17; 60000]); - assert_eq!( - fixture.handles[primary] - .query( - fixture.definition.get.clone(), - b"large".to_vec(), - at_least(&fixture.initial) - ) - .await? - .result, - vec![17; 60000] - ); - drop(responses); - fixture.close().await -} - -#[tokio::test] -async fn version_key_rotation_survives_restart_and_ambiguous_catalog_write() -> Result<()> { - let store = Arc::new(store::Store::default()); - let catalog = Catalog::new(store.clone(), CatalogOptions::default())?; - let namespace = catalog - .open_namespace(NamespaceConfig::new( - scope().cluster, - scope().namespace, - NamespaceMode::Raft, - [7; 32], - )) - .await?; - let old = keys(11, 10000, vec![])?; - store.fail_after.store(true, Ordering::Release); - namespace.require_version_keys(&old, 1000).await?; - let token = old.issue(&version(), [7; 32], 1000)?; - let next_key = VersionSigningKey::new([12; 32], 20000)?; - // Verifier-first rolling deployment: advertise both keys while the old - // signer is active, then switch to the new signer with identical trust. - let distributed = keys(11, 10000, vec![next_key.verification_key()])?; - store.fail_confirmation.store(true, Ordering::Release); - assert!(namespace - .require_version_keys(&distributed, 1100) - .await - .is_err()); - namespace.require_version_keys(&distributed, 1100).await?; - let next = VersionKeys::new( - next_key, - old.verification_keys(), - Duration::from_millis(1000), - )?; - namespace.require_version_keys(&next, 1100).await?; - assert_eq!(next.verify(&token, &scope(), [7; 32], 1100)?, version()); - assert!(namespace.require_version_keys(&old, 1100).await.is_err()); - let dropped = keys(12, 20000, vec![])?; - assert!(namespace - .require_version_keys(&dropped, 1100) - .await - .is_err()); - // Reopening the catalog must preserve the same refusal; it cannot rely on - // an in-memory history of the previous configuration. - let reopened = Catalog::new(store, CatalogOptions::default())? - .open_namespace(namespace.config().clone()) - .await?; - assert!(reopened.require_version_keys(&dropped, 1100).await.is_err()); - reopened.require_version_keys(&dropped, 10000).await?; - assert!(next.verify(&token, &scope(), [7; 32], 10000).is_err()); - Ok(()) -} - -#[test] -fn version_key_deadline_and_payload_limits() -> Result<()> { - let ring = keys(11, 2000, vec![])?; - assert!(ring.issue(&version(), [7; 32], 1001).is_err()); - assert!(ring.issue(&version(), [7; 32], u64::MAX).is_err()); - let mut invalid = version(); - invalid.position.index = LogIndex::new(0); - assert!(ring.issue(&invalid, [7; 32], 1000).is_err()); - let mut longest = version(); - longest.scope.cell = "x".repeat(1024); - let token = ring.issue(&longest, [7; 32], 1000)?; - assert_eq!(ring.verify(&token, &longest.scope, [7; 32], 1000)?, longest); - longest.scope.cell.push('x'); - assert!(ring.issue(&longest, [7; 32], 1000).is_err()); - let old = VersionSigningKey::new([11; 32], 10000)?.verification_key(); - assert!(keys(11, 20000, vec![old]).is_err()); - assert!(VersionSigningKey::new([0; 32], 20000).is_err()); - Ok(()) -} diff --git a/crates/cellar/tests/native_raft_snapshot.rs b/crates/cellar/tests/native_raft_snapshot.rs index 46e611055..266f98cab 100644 --- a/crates/cellar/tests/native_raft_snapshot.rs +++ b/crates/cellar/tests/native_raft_snapshot.rs @@ -1537,6 +1537,7 @@ async fn native_seed_drill_store( }, }, RaftCellsServices { + recovery: None, listeners: cellar::sdk::NodeRaftListeners { control: tokio::net::TcpListener::bind("127.0.0.1:0").await?, data: tokio::net::TcpListener::bind("127.0.0.1:0").await?, @@ -1826,6 +1827,7 @@ async fn missing_quorum_restore_refused() -> Result<()> { host: Default::default(), }, RaftCellsServices { + recovery: None, listeners: cellar::sdk::NodeRaftListeners { control: tokio::net::TcpListener::bind("127.0.0.1:0").await?, data: tokio::net::TcpListener::bind("127.0.0.1:0").await?, diff --git a/crates/cellar/tests/support/cell_group_admission.rs b/crates/cellar/tests/support/cell_group_admission.rs new file mode 100644 index 000000000..27da4aa9e --- /dev/null +++ b/crates/cellar/tests/support/cell_group_admission.rs @@ -0,0 +1,224 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; + +pub(super) async fn capacity() -> Result<()> { + let mut cluster = Cluster::with_options(RaftRuntimeOptions { + max_cells_per_group: 2, + ..Default::default() + }) + .await?; + cluster.elect().await?; + let route = cluster.birth.route().clone(); + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES (1);")?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let a = history(&route, "capacity-a")?; + let b = history(&route, "capacity-b")?; + let c = history(&route, "capacity-c")?; + let request_a = request(&route, a.clone(), 1, &cut)?; + let owner = cluster.voters[0].owner.as_mut().unwrap(); + let term = owner.status()?.term; + let first = owner.propose_at_term(term, request_a.clone())?; + let repeated = owner.propose_at_term(term, request_a.clone())?; + assert_eq!(first.position, repeated.position); + assert_eq!(first.digest, repeated.digest); + let mut rival = a.clone(); + rival.incarnation = IncarnationId::new([99; 16])?; + assert!(matches!( + owner.propose_at_term(term, request(&route, rival, 1, &cut)?), + Err(RuntimeError::Invalid(_)) + )); + let second = owner.propose_at_term(term, request(&route, b.clone(), 1, &cut)?)?; + assert_eq!(second.position.index.get(), first.position.index.get() + 1); + assert_eq!(owner.manifest().marker().root().cells(), 0); + assert!(matches!( + owner.propose_at_term(term, request(&route, c.clone(), 1, &cut)?), + Err(RuntimeError::Overloaded) + )); + cluster.drive(false).await?; + for voter in &cluster.voters { + assert_eq!( + voter + .owner + .as_ref() + .unwrap() + .manifest() + .marker() + .root() + .cells(), + 2 + ); + } + let owner = cluster.voters[0].owner.as_mut().unwrap(); + assert!(matches!( + owner.propose_at_term(term, request_a), + Err(RuntimeError::StaleCompletion) + )); + assert!(matches!( + owner.propose_at_term(term, request(&route, c, 1, &cut)?), + Err(RuntimeError::Overloaded) + )); + let updated = images::image(Some(&bytes), "UPDATE value SET n=n+1;")?; + let mutation = images::Cut::new(&bytes, &updated, 1, true, false); + let accepted = owner.propose_at_term(term, request(&route, a.clone(), 2, &mutation)?)?; + assert_eq!( + accepted.position.index.get(), + second.position.index.get() + 1 + ); + cluster.drive(false).await?; + for voter in &cluster.voters { + let (leaf, _) = voter + .owner + .as_ref() + .unwrap() + .manifest() + .pin_cell(&a.cell) + .context("updated cell")?; + assert_eq!(leaf.version.sequence.get(), 2); + } + cluster.close().await +} + +pub(super) async fn mutation_retries() -> Result<()> { + let mut cluster = Cluster::with_options(RaftRuntimeOptions { + max_requests: 1, + ..Default::default() + }) + .await?; + cluster.elect().await?; + let route = cluster.birth.route().clone(); + let a = history(&route, "retry-a")?; + let b = history(&route, "retry-b")?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES (1);")?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + for cell in [&a, &b] { + cluster.voters[0].owner.as_mut().unwrap().propose(request( + &route, + cell.clone(), + 1, + &cut, + )?)?; + cluster.drive(false).await?; + } + let updated = images::image(Some(&bytes), "UPDATE value SET n=n+1;")?; + let cut = images::Cut::new(&bytes, &updated, 1, true, false); + let mutation_a = request(&route, a.clone(), 2, &cut)?; + let mutation_b = request(&route, b.clone(), 2, &cut)?; + let owner = cluster.voters[0].owner.as_mut().unwrap(); + let term = owner.status()?.term; + let accepted = owner.propose_at_term(term, mutation_a.clone())?; + // Model a lost acceptance response: the same bytes recover its original + // position without another append, even when acceptance capacity is full. + let retry = owner.propose_at_term(term, mutation_a.clone())?; + assert_eq!(accepted.position, retry.position); + assert_eq!(accepted.digest, retry.digest); + assert!(matches!( + owner.propose_at_term(term, mutation_b.clone()), + Err(RuntimeError::Overloaded) + )); + let different = images::image(Some(&bytes), "UPDATE value SET n=n+99;")?; + let cut = images::Cut::new(&bytes, &different, 1, true, false); + assert!(matches!( + owner.propose_at_term(term, request(&route, a.clone(), 2, &cut)?), + Err(RuntimeError::Invalid(_)) + )); + cluster.drive(false).await?; + let owner = cluster.voters[0].owner.as_mut().unwrap(); + // The applied image excludes the old base after its acceptance record is + // released. The caller must inspect its receipt; this is not a failed write. + assert!(matches!( + owner.propose_at_term(term, mutation_a), + Err(RuntimeError::StaleCompletion) + )); + let accepted_b = owner.propose_at_term(term, mutation_b)?; + assert_eq!( + accepted_b.position.index.get(), + accepted.position.index.get() + 1 + ); + cluster.drive(false).await?; + for voter in &cluster.voters { + for cell in [&a, &b] { + let (leaf, CellImagePin::Materialized(pin)) = voter + .owner + .as_ref() + .unwrap() + .manifest() + .pin_cell(&cell.cell) + .context("cell")? + else { + anyhow::bail!("materialized cell") + }; + assert_eq!(leaf.version.sequence.get(), 2); + assert_eq!(images::read(&pin)?, updated); + } + } + cluster.close().await +} + +pub(super) async fn restart_capacity() -> Result<()> { + let mut cluster = Cluster::with_options(RaftRuntimeOptions { + max_cells_per_group: 1, + ..Default::default() + }) + .await?; + cluster.elect().await?; + let route = cluster.birth.route().clone(); + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES (1);")?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let a = history(&route, "inherited")?; + let b = history(&route, "extra")?; + let owner = cluster.voters[0].owner.as_mut().unwrap(); + let term = owner.status()?.term; + let old = owner.propose_at_term(term, request(&route, a.clone(), 1, &cut)?)?; + cluster.drive(true).await?; + assert_eq!( + cluster.voters[0] + .owner + .as_ref() + .unwrap() + .manifest() + .marker() + .root() + .cells(), + 0 + ); + // Recover every voter from disk. Live followers retain the old leader's + // lease and may correctly reject an immediate pre-vote from a lone restart. + for n in 0..3 { + cluster.restart(n).await?; + } + cluster.voters[0].paused = true; + cluster.voters[0].owner.as_mut().unwrap().campaign()?; + cluster.drive(false).await?; + assert!(cluster.voters[0].held.is_some()); + let owner = cluster.voters[0].owner.as_mut().unwrap(); + assert!(owner.status()?.term > term); + assert!(!owner.status()?.ready); + assert!(matches!( + owner.propose(request(&route, b.clone(), 1, &cut)?), + Err(RuntimeError::NotReady) + )); + assert!(matches!( + owner.propose_at_term(term, request(&route, a.clone(), 1, &cut)?), + Err(RuntimeError::StaleCompletion) + )); + let voter = &mut cluster.voters[0]; + voter.paused = false; + let work = voter.held.take().context("held inherited apply")?; + assert_eq!(work.ticket.position(), old.position); + let next = prepare(voter, &work)?; + voter + .owner + .as_mut() + .unwrap() + .complete_apply(work.ticket, next)?; + cluster.drive(false).await?; + let owner = cluster.voters[0].owner.as_mut().unwrap(); + assert!(owner.status()?.ready); + assert_eq!(owner.manifest().marker().root().cells(), 1); + assert!(matches!( + owner.propose(request(&route, b, 1, &cut)?), + Err(RuntimeError::Overloaded) + )); + cluster.close().await +} diff --git a/crates/cellar/tests/support/cell_group_archive_binding.rs b/crates/cellar/tests/support/cell_group_archive_binding.rs new file mode 100644 index 000000000..fc4102edf --- /dev/null +++ b/crates/cellar/tests/support/cell_group_archive_binding.rs @@ -0,0 +1,157 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Fresh catalog owners must recover the immutable archive identity remotely. +use super::*; + +pub(super) async fn reopen_directory( + cluster: &Cluster, +) -> Result<(RangeDirectory, CellGroupBirth)> { + let route = cluster.birth.route(); + let catalog = Catalog::new(cluster.store.clone(), Default::default())?; + let namespace = cluster + .domain + .run(catalog.existing_namespace(NamespaceConfig::new( + route.history.cluster, + route.history.namespace, + cellar_logic::NamespaceMode::Raft, + route.schema_digest, + ))) + .await??; + let directory = RangeDirectory::new(namespace, Default::default())?; + let birth = cluster + .domain + .run(directory.materialize_selected_group(route)) + .await??; + Ok((directory, birth)) +} + +pub(super) async fn identity() -> Result<()> { + let cluster = Cluster::new().await?; + let pool = CellGroupArchivePool::new(Default::default())?; + let before = cluster + .store + .list(None) + .try_collect::>() + .await? + .len(); + assert!(matches!( + cluster + .directory + .open_group_archive(&cluster.domain, &pool, &cluster.birth) + .await, + Err(CellGroupArchiveBindingError::Catalog( + CatalogError::Corrupt("missing group archive binding") + )) + )); + assert_eq!( + cluster + .store + .list(None) + .try_collect::>() + .await? + .len(), + before + ); + let first = cluster + .directory + .prepare_group_archive_binding(&cluster.domain, &pool, &cluster.birth) + .await?; + let replacement = cluster + .directory + .prepare_group_archive_binding(&cluster.domain, &pool, &cluster.birth) + .await?; + assert_ne!(first.encode(), replacement.encode()); + let saved = persist_archive_intent(&cluster, &first.encode().to_vec()).await?; + assert_eq!(saved, first.encode()); + let registry = first.commit(&cluster.domain).await?; + let intent = registry.prepare_upload(&cluster.domain).await?; + let saved = persist_archive_intent(&cluster, &intent).await?; + let ticket = registry.reserve(&cluster.domain, &saved).await?; + drop(registry); + assert!(matches!( + replacement.commit(&cluster.domain).await, + Err(CellGroupArchiveBindingError::Catalog( + CatalogError::Conflict("group archive binding") + )) + )); + let (directory, birth) = reopen_directory(&cluster).await?; + // Even the original typed birth observation belongs to another catalog + // owner. Materialize and verify it again rather than transferring authority. + assert!(matches!( + directory + .open_group_archive(&cluster.domain, &pool, &cluster.birth) + .await, + Err(CellGroupArchiveBindingError::Catalog( + CatalogError::Conflict("foreign group birth observation") + )) + )); + let reopened = directory + .open_group_archive(&cluster.domain, &pool, &birth) + .await?; + // Persisted exact allocations reconcile through the remotely bound identity. + assert_eq!(reopened.reserve(&cluster.domain, &saved).await?, ticket); + reopened.abort(&cluster.domain, &ticket).await?; + let objects = cluster.store.list(None).try_collect::>().await?; + let binding = objects + .iter() + .find(|m| m.location.as_ref().ends_with("/binding")) + .context("remote binding")? + .location + .clone(); + let counter = objects + .iter() + .find(|m| m.location.as_ref().ends_with("/ownership/registry")) + .context("remote counters")? + .location + .clone(); + let identity = objects + .iter() + .find(|m| m.location.as_ref().ends_with("/ownership/identity")) + .context("remote identity")? + .location + .clone(); + let expected_binding = cluster.store.get(&binding).await?.bytes().await?; + let expected_identity = cluster.store.get(&identity).await?.bytes().await?; + drop(reopened); + cluster.store.delete(&counter).await?; + let before = cluster + .store + .list(None) + .try_collect::>() + .await? + .len(); + assert!(directory + .open_group_archive(&cluster.domain, &pool, &birth) + .await + .is_err()); + assert_eq!( + cluster + .store + .list(None) + .try_collect::>() + .await? + .len(), + before + ); + let replacement = directory + .prepare_group_archive_binding(&cluster.domain, &pool, &birth) + .await?; + assert!(replacement.commit(&cluster.domain).await.is_err()); + assert_eq!( + cluster.store.get(&binding).await?.bytes().await?, + expected_binding + ); + assert_eq!( + cluster.store.get(&identity).await?.bytes().await?, + expected_identity + ); + assert!(cluster.store.get(&counter).await.is_err()); + // A lost binding cannot be inferred from whichever mutable identity remains. + cluster.store.delete(&binding).await?; + assert!(directory + .open_group_archive(&cluster.domain, &pool, &birth) + .await + .is_err()); + assert!(cluster.store.get(&binding).await.is_err()); + cluster.close().await +} diff --git a/crates/cellar/tests/support/cell_group_archive_dependencies.rs b/crates/cellar/tests/support/cell_group_archive_dependencies.rs new file mode 100644 index 000000000..ad7cd42de --- /dev/null +++ b/crates/cellar/tests/support/cell_group_archive_dependencies.rs @@ -0,0 +1,539 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Complete-group coverage must include B even when only A is awaiting S3. +use super::*; + +struct Base { + h: Harness, + registry: CellGroupArchiveRegistry, + hold: CellGroupArchiveHold, + dependency: ObjectPath, + bytes: Vec, +} + +pub(super) fn image_value(pin: CellImagePin, expected: u64) -> Result<()> { + let bytes = match pin { + CellImagePin::Empty => anyhow::bail!("expected a physical SQLite image"), + CellImagePin::Materialized(pin) => { + let mut reader = pin.open_reader()?; + let mut bytes = Vec::new(); + while (bytes.len() as u64) < reader.length() { + let count = (reader.length() - bytes.len() as u64).min(64 * 1024) as usize; + bytes.extend(reader.read_exact_at(bytes.len() as u64, count)?); + } + bytes + } + CellImagePin::Extents(pin) => { + let mut bytes = Vec::new(); + for ordinal in 0..pin.manifest().geometry().extent_count() { + bytes.extend( + pin.view() + .read_extent(pin.manifest().extent(ordinal).context("SQLite extent")?)?, + ); + } + bytes + } + }; + let root = tempfile::tempdir()?; + let path = root.path().join("value.sqlite"); + std::fs::write(&path, bytes)?; + let db = + rusqlite::Connection::open_with_flags(path, rusqlite::OpenFlags::SQLITE_OPEN_READ_ONLY)?; + assert_eq!( + db.query_row("SELECT n FROM value", [], |r| r.get::<_, u64>(0))?, + expected + ); + Ok(()) +} + +async fn archive_values(h: &Harness, coverage: &CellGroupArchiveCoverage) -> Result<()> { + let root = coverage.recovery().manifest().clone(); + h.fixture + .cluster + .domain + .blocking(move || -> Result<()> { + assert_eq!(root.pinned_images(), 2); + for (name, expected) in [("registered-a", 0), ("registered-b", 1)] { + let (leaf, pin) = root + .pin_cell(&CellId::new(name)?) + .context("archived cell")?; + assert_eq!(leaf.version.sequence.get(), expected + 1); + image_value(pin, expected)?; + } + Ok(()) + })? + .await? +} + +async fn local_values(h: &Harness, index: LogIndex) -> Result<()> { + controller::applied(&h.fixture, h.route.history, index).await?; + let b = history(&h.route, "registered-b")?; + for runtime in &h.fixture.runtimes { + for (cell, value) in [(&h.cell, 2), (&b, 1)] { + let selected = runtime + .reconciliation_authority(h.route.history, cell.clone()) + .await?; + assert_eq!(selected.cell().version.sequence.get(), value + 1); + let image = selected.image().clone(); + h.fixture + .cluster + .domain + .blocking(move || image_value(image, value))? + .await??; + } + } + assert_eq!(h.calls.load(Ordering::SeqCst), 3); + assert!(h.controllers[0].group_failure(h.route.history)?.is_none()); + Ok(()) +} + +async fn setup(rooted: bool, store: Arc) -> Result { + let h = Harness::new(rooted, store, 2).await?; + // Give B different actual SQLite bytes, so the fault cannot hit a shared + // empty/header chunk or any part of A's image. + let b = history(&h.route, "registered-b")?; + let invocation = h + .dispatcher + .command( + b.clone(), + CellCommandRequest { + id: CommandId::new(ClientId::new([189; 16])?, 1)?, + contract: h.class.contract("Add").context("Add")?, + input: b"b-base".to_vec(), + durability: Durability::Quorum, + retry_until_ms: h.retry_until, + }, + CellRoutingLimits { + timeout: Duration::from_secs(15), + max_attempts: 4, + }, + None, + ) + .await?; + let response = h.controllers[0] + .command(invocation, h.fixture.cell_options(&b).await?)? + .response() + .await?; + assert_eq!(response.durability(), Durability::Quorum); + assert_eq!( + response + .local_receipt() + .expect("local command receipt") + .receipt + .result(), + b"b-base" + ); + let index = response + .local_receipt() + .expect("local command receipt") + .position + .index; + drop(response); + archive_controller::published(&h.controllers[0], h.route.history, index).await?; + let coverage = + archive_controller::verified_current(&h.fixture, &h.archive, rooted, "dependency-base") + .await?; + archive_values(&h, &coverage).await?; + let marker = coverage.marker().clone(); + // The controller retains its durable hold. Release this completed receiver + // before asking the same one-slot production pool to archive a new cut. + drop(coverage); + + // A separate recovery client owns its own bounded two-operation I/O pool, + // one checkpoint receiver and one tail operation. Its persistent hold is + // independent of the controller's head retirement. + let registry = h + .archive + .directory + .open_group_archive( + &h.fixture.cluster.domain, + &CellGroupArchivePool::new(Default::default())?, + &h.archive.birth, + ) + .await?; + let intent = registry.prepare_acquire(&h.fixture.cluster.domain).await?; + let encoded = serde_json::to_vec(&intent)?; + let path = h + .fixture + .cluster + ._root + .path() + .join("dependency-recovery-intent"); + h.fixture + .cluster + .domain + .blocking(move || -> Result<()> { + use std::io::Write; + let mut file = std::fs::OpenOptions::new() + .create_new(true) + .write(true) + .open(&path)?; + file.write_all(&encoded)?; + file.sync_all()?; + std::fs::File::open(path.parent().context("intent parent")?)?.sync_all()?; + Ok(()) + })? + .await??; + let hold = registry.acquire(&h.fixture.cluster.domain, &intent).await?; + assert_eq!(hold.marker(), &marker); + + let objects = h.remote.list(None).try_collect::>().await?; + let mut dependency = None; + for object in objects + .iter() + .filter(|o| o.location.as_ref().contains("/indexes/")) + { + let value: serde_json::Value = + serde_json::from_slice(&h.remote.get(&object.location).await?.bytes().await?)?; + let Some(chunks) = value.get("chunks").and_then(|v| v.as_array()) else { + continue; + }; + if serde_json::from_value::(value["marker"].clone())? != marker { + continue; + } + let prefix = object + .location + .as_ref() + .split_once("/generations/") + .context("generation prefix")? + .0; + let path = |chunk: &serde_json::Value| -> Result { + let reference = &chunk["object"]; + let digest: Vec = serde_json::from_value(reference["digest"].clone())?; + Ok(ObjectPath::from(format!( + "{prefix}/generations/{}/chunks/{}-{}", + reference["generation"], + reference["length"], + hex::encode(digest) + ))) + }; + let a = chunks + .iter() + .filter(|c| c["part"]["cell"] == "registered-a") + .map(&path) + .collect::>>()?; + for chunk in chunks.iter().filter(|c| { + c["part"]["cell"] == "registered-b" + && c["part"]["kind"] == if rooted { "Extent" } else { "Materialized" } + }) { + let candidate = path(chunk)?; + if !a.contains(&candidate) { + dependency = Some(candidate); + break; + } + } + if dependency.is_some() { + break; + } + } + let dependency = dependency.context("B image bytes outside A's entire image closure")?; + let bytes = h + .remote + .inner + .get(&dependency) + .await? + .bytes() + .await? + .to_vec(); + assert!(!bytes.is_empty()); + assert!(dependency + .as_ref() + .ends_with(&hex::encode(Sha256::digest(&bytes)))); + assert_eq!(h.calls.load(Ordering::SeqCst), 1); + Ok(Base { + h, + registry, + hold, + dependency, + bytes, + }) +} + +async fn selected_stays_before_a(base: &Base) -> Result<()> { + // Read the actual registry selection, not the advisory controller status. + // The uncommitted acquire intent below is only an observation. + let intent = base + .registry + .prepare_acquire(&base.h.fixture.cluster.domain) + .await?; + let value = serde_json::to_value(intent)?; + let selected: GroupManifest = + serde_json::from_value(value["action"]["Acquire"]["root"]["marker"].clone())?; + assert_eq!(&selected, base.hold.marker()); + Ok(()) +} + +async fn receiver( + base: &Base, + rooted: bool, + name: &str, + checkpoints: &CellGroupCheckpointPool, +) -> Result { + let destination = + archive_controller::verifier(&base.h.fixture, &base.h.route, rooted, name).await?; + Ok(checkpoints.receive_archive( + destination, + ScratchPool::new(ScratchLimits { + max_bytes: 32 * 1024 * 1024, + max_files: 1, + })?, + base.h.fixture.cluster._root.path().into(), + )?) +} + +async fn exact_local_retry(h: &Harness) -> Result<()> { + let mut request = cell_attempt(207)?; + request.request.id = CommandId::new(ClientId::new([188; 16])?, 1)?; + request.request.operation = "Add".into(); + request.request.contract_digest = [82; 32]; + request.request.request_digest = command_request_digest(b"later-a"); + request.request.durability = Durability::S3; + request.request.retry_until_ms = h.retry_until; + let authority = h.fixture.runtimes[0] + .execution_authority(h.route.history, h.cell.clone()) + .await?; + let CellSqlOutcome::Receipt(receipt) = h.fixture.appliers[&(0, h.route.history)] + .0 + .stage_registered( + h.fixture.cell_options(&h.cell).await?, + authority, + request, + h.class.clone(), + b"later-a".to_vec(), + false, + ) + .await? + else { + anyhow::bail!("known command must resolve without callback") + }; + assert_eq!(receipt.receipt().receipt.result(), b"later-a"); + assert_eq!( + receipt.receipt().receipt.minimum_durability(), + Durability::S3 + ); + assert_eq!(h.calls.load(Ordering::SeqCst), 3); + // This internal receipt carries no archive proof and is never emitted as an + // S3 response. The stored minimum remains S3, including during repair. + Ok(()) +} + +async fn delayed(rooted: bool, store: Arc) -> Result<()> { + let base = Box::pin(setup(rooted, store)).await?; + let gate = base.h.remote.hold(base.dependency.clone()); + let call = base + .h + .command(1, b"later-a", Durability::S3, Duration::from_secs(15)) + .await?; + tokio::time::timeout(Duration::from_secs(5), gate.entered.notified()).await?; + base.h.waiting().await?; + let mut output = Box::pin(call.response()); + assert!( + tokio::time::timeout(Duration::from_millis(30), output.as_mut()) + .await + .is_err() + ); + + let next = base + .h + .command( + 2, + b"quorum-successor", + Durability::Quorum, + Duration::from_secs(15), + ) + .await? + .response() + .await?; + assert_eq!(next.durability(), Durability::Quorum); + assert_eq!( + next.local_receipt() + .expect("local command receipt") + .receipt + .result(), + b"quorum-successor" + ); + let index = next + .local_receipt() + .expect("local command receipt") + .position + .index; + assert!(index > base.hold.marker().position().index); + drop(next); + local_values(&base.h, index).await?; + selected_stays_before_a(&base).await?; + + let checkpoints = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let tails = CellGroupArchiveTailPool::new(Default::default())?; + let target = receiver(&base, rooted, "dependency-delayed", &checkpoints).await?; + let mut coverage = Box::pin(base.registry.verify_hold( + &base.h.fixture.cluster.domain, + &base.hold, + &tails, + target, + )); + assert!( + tokio::time::timeout(Duration::from_millis(30), coverage.as_mut()) + .await + .is_err() + ); + assert_eq!(checkpoints.usage().sources, 1); + assert!( + tokio::time::timeout(Duration::from_millis(30), output.as_mut()) + .await + .is_err() + ); + + base.h.remote.clear(); + gate.release.notify_waiters(); + let recovered = coverage.await?; + archive_values(&base.h, &recovered).await?; + drop(recovered); + assert_eq!(checkpoints.usage().sources, 0); + assert_eq!(tails.usage().operations, 0); + let response = output.await?; + assert_eq!(response.durability(), Durability::S3); + assert_eq!( + response + .local_receipt() + .expect("local command receipt") + .receipt + .result(), + b"later-a" + ); + drop(response); + archive_controller::published(&base.h.controllers[0], base.h.route.history, index).await?; + exact_local_retry(&base.h).await?; + base.registry + .release(&base.h.fixture.cluster.domain, &base.hold) + .await?; + base.h.close().await +} + +async fn corrupt(rooted: bool, store: Arc) -> Result<()> { + let base = Box::pin(setup(rooted, store)).await?; + let mut damaged = base.bytes.clone(); + damaged[0] ^= 1; + assert_ne!(Sha256::digest(&base.bytes), Sha256::digest(&damaged)); + base.h + .remote + .inner + .put(&base.dependency, damaged.clone().into()) + .await?; + assert_eq!( + base.h + .remote + .inner + .get(&base.dependency) + .await? + .bytes() + .await? + .as_ref(), + damaged + ); + let checkpoints = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let tails = CellGroupArchiveTailPool::new(Default::default())?; + let target = receiver(&base, rooted, "dependency-corrupt", &checkpoints).await?; + let error = base + .registry + .verify_hold(&base.h.fixture.cluster.domain, &base.hold, &tails, target) + .await + .err() + .context("corrupt B must refuse complete coverage")?; + assert!(matches!( + error, + CellGroupArchiveError::Invalid("object digest/length") + )); + assert_eq!(checkpoints.usage().sources, 0); + assert_eq!(tails.usage().operations, 0); + + let error = base + .h + .command(1, b"later-a", Durability::S3, Duration::from_secs(15)) + .await? + .response() + .await + .err() + .context("corrupt B must refuse A S3 acknowledgement")?; + assert!(error.ambiguous); + assert!(matches!( + error.cause.as_ref(), + CellDispatchError::Archive(_) + )); + let status = base.h.controllers[0] + .archive_status(base.h.route.history)? + .context("archive status")?; + assert!(status.stopped); + assert!(status + .error + .context("corruption failure")? + .to_string() + .contains("object digest/length")); + selected_stays_before_a(&base).await?; + let next = base + .h + .command( + 2, + b"quorum-successor", + Durability::Quorum, + Duration::from_secs(15), + ) + .await? + .response() + .await?; + assert_eq!(next.durability(), Durability::Quorum); + let index = next + .local_receipt() + .expect("local command receipt") + .position + .index; + assert!(index > base.hold.marker().position().index); + drop(next); + local_values(&base.h, index).await?; + + // Repair only the fault-injected object in this isolated fixture. Recovery + // uses a fresh physical receiver, while the failed archive owner stays stopped. + base.h + .remote + .inner + .put(&base.dependency, base.bytes.clone().into()) + .await?; + let target = receiver(&base, rooted, "dependency-repaired", &checkpoints).await?; + let recovered = base + .registry + .verify_hold(&base.h.fixture.cluster.domain, &base.hold, &tails, target) + .await?; + archive_values(&base.h, &recovered).await?; + drop(recovered); + exact_local_retry(&base.h).await?; + assert!( + base.h.controllers[0] + .archive_status(base.h.route.history)? + .context("archive status")? + .stopped + ); + base.registry + .release(&base.h.fixture.cluster.domain, &base.hold) + .await?; + base.h.close().await +} + +#[inline(never)] +pub(crate) fn archive_dependencies( + rooted: bool, + store: Arc, +) -> std::pin::Pin>>> { + Box::pin(async move { + // Each scenario creates real fixed-identity voter disks. Isolate their + // catalogs and permanent claims instead of trying to reuse that birth. + let delayed_store = Arc::new(object_store::prefix::PrefixStore::new( + store.clone(), + ObjectPath::from("delayed"), + )); + let corrupt_store = Arc::new(object_store::prefix::PrefixStore::new( + store, + ObjectPath::from("corrupt"), + )); + Box::pin(delayed(rooted, delayed_store)).await?; + Box::pin(corrupt(rooted, corrupt_store)).await + }) +} diff --git a/crates/cellar/tests/support/cell_group_archive_fairness.rs b/crates/cellar/tests/support/cell_group_archive_fairness.rs new file mode 100644 index 000000000..07a95bbe5 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_archive_fairness.rs @@ -0,0 +1,305 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Independently archived histories contend for one controller and compaction lane. +use super::*; +use raft::Storage as _; + +async fn reclaimed( + fixture: &RuntimeCluster, + controller: &CellGroupController, + group: GroupHistory, + cut: LogIndex, +) -> Result<()> { + let log = fixture.cluster.voters[0].log.cell_group(&group)?; + controller::wait_for(async || Ok(log.first_index()? > cut.get())) + .await + .with_context(|| format!("checkpoint did not compact {group:?} through {cut:?}"))?; + let checkpoint = log + .checkpoint_manifest()? + .context("compacted archive checkpoint")?; + let archived = controller + .archive_status(group)? + .context("archive status")? + .published + .context("verified archive")?; + assert!(checkpoint.position().index >= cut); + assert!(checkpoint.position().index <= archived.position().index); + assert_eq!(checkpoint.route().history, group); + Ok(()) +} + +fn retained( + fixture: &RuntimeCluster, + controller: &CellGroupController, + group: GroupHistory, + cut: LogIndex, +) -> Result<()> { + let log = fixture.cluster.voters[0].log.cell_group(&group)?; + assert!(log.first_index()? <= cut.get()); + assert!(log.last_index()? >= cut.get()); + let status = controller + .archive_status(group)? + .context("archive status")?; + assert!(!status.stopped); + assert!(status + .published + .is_none_or(|marker| marker.position().index < cut)); + Ok(()) +} + +pub(crate) async fn archived_groups_reclaim_independently( + store: Arc, +) -> Result<()> { + let (mut fixture, primary) = Box::pin(setup_store(false, store.clone())).await?; + let (base, remote) = registration(&fixture, &primary, false).await?; + let mut archives = vec![(base.clone(), remote)]; + for number in [31, 41] { + let (directory, birth) = fixture + .add_group_in_store(number, store.clone(), |_| Arc::new(DirectFileSystem)) + .await?; + controller::attach_group_appliers(&mut fixture, primary.history, birth.route().history) + .await?; + let (mut config, remote) = registration_for(&fixture, &directory, &birth, false).await?; + config.io = base.io.clone(); + config.owners = base.owners.clone(); + config.tails = base.tails.clone(); + config.checkpoints = base.checkpoints.clone(); + config.scratch = base.scratch.clone(); + archives.push((config, remote)); + } + fixture.connect_tcp().await?; + for transport in &fixture.transports { + transport.enable_group_checkpoints(CellGroupCheckpointPool::new( + checkpoint_fixture::limits(), + )?)?; + } + let controllers = controller::start_with_limits( + &mut fixture, + CellGroupControllerLimits { + max_groups: 3, + checkpoints: Some(CellGroupCheckpointPolicy { + interval_entries: 1, + max_parallel: 1, + }), + ..controller::limits() + }, + ) + .await?; + let owner = &controllers[0]; + for (config, _) in &archives { + let group = config.birth.route().history; + if group != primary.history { + fixture.runtimes[0].campaign(group).await?; + controller::wait_for(async || Ok(fixture.runtimes[0].status(group).await?.ready)) + .await?; + } + owner.archive(config.clone())?.response().await?; + } + + let held = archives[0].1.hold_publication(); + archives[0] + .1 + .lose_immutable_put_response + .store(true, Ordering::Release); + let mut first = vec![add(&fixture, &primary, false, 1).await?]; + tokio::time::timeout(Duration::from_secs(10), held.entered.notified()) + .await + .context("first group entered the held archive publication")?; + for (config, _) in &archives[1..] { + first.push(add(&fixture, config.birth.route(), false, 1).await?); + } + for ((config, _), &cut) in archives.iter().zip(&first) { + retained(&fixture, owner, config.birth.route().history, cut)?; + } + held.release.notify_one(); + for ((config, _), cut) in archives.iter().zip(first) { + let route = config.birth.route(); + published(owner, route.history, cut) + .await + .with_context(|| format!("first archive cut {cut:?} for {:?}", route.history))?; + // Checkpoint scheduling retains the successor apply ticket while it + // checkpoints the already archived base. + add(&fixture, route, false, 2).await?; + reclaimed(&fixture, owner, route.history, cut).await?; + } + assert!(!archives[0] + .1 + .lose_immutable_put_response + .load(Ordering::Acquire)); + + // An unavailable history retains its local tail without starving the two + // healthy histories that share this node's archive and checkpoint owners. + let blocked = &archives[1].1; + blocked.outage.store(true, Ordering::Release); + let mut third = Vec::new(); + for (config, _) in &archives { + third.push(add(&fixture, config.birth.route(), false, 3).await?); + } + let mut latest = Vec::new(); + for index in [0, 2, 1] { + if index == 1 { + retained( + &fixture, + owner, + archives[1].0.birth.route().history, + third[1], + )?; + blocked.outage.store(false, Ordering::Release); + } + let route = archives[index].0.birth.route(); + let cut = third[index]; + published(owner, route.history, cut) + .await + .with_context(|| format!("archive catch-up to {cut:?} for {:?}", route.history))?; + let successor = add(&fixture, route, false, 4).await?; + reclaimed(&fixture, owner, route.history, cut).await?; + latest.push((index, successor)); + } + + for &(index, cut) in &latest { + let history = archives[index].0.birth.route().history; + published(owner, history, cut) + .await + .with_context(|| format!("final archive cut {cut:?} for {history:?}"))?; + } + for controller in controllers { + assert_eq!(controller.usage().quarantined_groups, 0); + controller.close().await?; + } + assert_eq!(base.tails.usage().operations, 0); + + let mut selected = BTreeMap::new(); + for (index, cut) in latest { + let config = &archives[index].0; + let history = config.birth.route().history; + let coverage = verified_current( + &fixture, + config, + false, + &format!("archive-audit-{}", hex::encode(history.group.bytes())), + ) + .await + .with_context(|| format!("restore selected cut {cut:?} for {history:?}"))?; + selected.insert( + index, + ( + coverage.hold().clone(), + coverage.recovery().reference().clone(), + ), + ); + assert_eq!(coverage.marker().route().history, history); + assert_eq!(coverage.marker().position().index, cut); + let root = coverage.recovery().manifest(); + assert_eq!( + root.pinned_images(), + if history == primary.history { 6 } else { 4 } + ); + for ordinal in 1..=4 { + let (_, pin) = root + .pin_cell(&CellId::new(format!("later-{ordinal}"))?) + .context("restored cell")?; + let CellImagePin::Materialized(pin) = pin else { + anyhow::bail!("expected restored SQLite generation"); + }; + let bytes = fixture + .cluster + .domain + .blocking(move || { + pin.open_reader()? + .read_exact_at(0, pin.state().length as usize) + })? + .await??; + let expected = images::image( + None, + &format!("CREATE TABLE value(n); INSERT INTO value VALUES({ordinal})"), + )?; + assert_eq!(bytes, expected); + } + } + let (primary_hold, tail) = &selected[&0]; + verify_archive_isolation( + &fixture, + &archives[0].0, + primary_hold, + tail, + &archives[1].0, + &selected[&1].0, + ) + .await?; + fixture.close().await +} + +async fn verify_archive_isolation( + fixture: &RuntimeCluster, + primary: &CellGroupArchiveRegistration, + primary_hold: &CellGroupArchiveHold, + tail: &CellGroupArchiveTailReference, + sibling: &CellGroupArchiveRegistration, + sibling_hold: &CellGroupArchiveHold, +) -> Result<()> { + let domain = &fixture.cluster.domain; + let registry = sibling + .directory + .open_group_archive(domain, &sibling.io, &sibling.birth) + .await?; + let destination = verifier(fixture, sibling.birth.route(), false, "foreign-hold").await?; + let foreign = registry + .verify_hold( + domain, + primary_hold, + &sibling.tails, + sibling.checkpoints.receive_archive( + destination, + sibling.scratch.clone(), + sibling.scratch_directory.clone(), + )?, + ) + .await; + assert!(matches!(foreign, Err(CellGroupArchiveError::Conflict))); + + let suffix = format!( + "/registries/{}/generations/{}/tails/{}-{}", + hex::encode(tail.registry()), + tail.generation(), + tail.length(), + hex::encode(tail.digest()) + ); + let store = &fixture.cluster.store; + let objects = store.list(None).try_collect::>().await?; + let mut selected = objects + .iter() + .filter(|object| object.location.as_ref().ends_with(&suffix)); + let path = &selected.next().context("selected immutable tail")?.location; + assert!(selected.next().is_none()); + let original = store.get(path).await?.bytes().await?; + let mut corrupted = original.to_vec(); + *corrupted.first_mut().context("nonempty archive tail")? ^= 1; + store.put(path, corrupted.into()).await?; + + let checked = async { + let failure = verified_current(fixture, primary, false, "corrupt-tail").await; + anyhow::ensure!( + matches!( + failure + .as_ref() + .err() + .and_then(|error| error.downcast_ref()), + Some(CellGroupArchiveError::Invalid("object digest/length")) + ), + "corrupt tail did not fail digest validation: {:?}", + failure.err() + ); + let restored = verified_current(fixture, sibling, false, "healthy-sibling").await?; + anyhow::ensure!( + restored.hold() == sibling_hold, + "sibling selected cut changed" + ); + Ok::<_, anyhow::Error>(()) + } + .await; + store.put(path, original.into()).await?; + checked?; + let repaired = verified_current(fixture, primary, false, "repaired-tail").await?; + assert!(repaired.hold() == primary_hold); + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_archive_isolation.rs b/crates/cellar/tests/support/cell_group_archive_isolation.rs new file mode 100644 index 000000000..9b64d7e69 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_archive_isolation.rs @@ -0,0 +1,254 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Data recovery deliberately has neither an engine nor serving authority. +use super::*; +use cellar_logic::consensus::OperationId; + +pub(super) async fn erased_voters(rooted: bool) -> Result<()> { + let (mut cluster, mut expected) = + checkpoint_fixture::populate_images(rooted, true, false, 1, 8192).await?; + let archive_pool = CellGroupArchivePool::new(Default::default())?; + let binding = cluster + .directory + .prepare_group_archive_binding(&cluster.domain, &archive_pool, &cluster.birth) + .await?; + let saved = persist_archive_intent(&cluster, &binding.encode().to_vec()).await?; + assert_eq!(saved, binding.encode()); + let registry = binding.commit(&cluster.domain).await?; + let owners = CellGroupArchiveUploadPool::new(Default::default())?; + let tails = CellGroupArchiveTailPool::new(Default::default())?; + let intent = registry.prepare_upload(&cluster.domain).await?; + let saved = persist_archive_intent(&cluster, &intent).await?; + let ticket = registry.reserve(&cluster.domain, &saved).await?; + let archive = registry + .open_upload(&cluster.domain, &owners, &ticket, &tails) + .await?; + let sources = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let base = archive + .upload_checkpoint(&cluster.domain, source(&sources, &cluster)?) + .await?; + checkpoint_fixture::advance_partition( + &mut cluster, + "alpha", + &expected["alpha"], + rooted, + Some(2), + ) + .await?; + expected.insert( + "alpha".into(), + images::image(Some(&expected["alpha"]), "UPDATE value SET n=n+1")?, + ); + let group = cluster.voters[0] + .log + .cell_group(&cluster.birth.route().history)?; + let current = cluster.voters[0].owner.as_ref().unwrap().manifest().clone(); + let marker = current.marker().clone(); + let membership = group.membership()?; + let reference = archive + .upload_tail(&cluster.domain, &tails, &base, &group, ¤t) + .await? + .into_reference(); + // No source physical pin, open GroupLog or voter survives the disk erasure. + drop(current); + drop(group); + drop(archive); + drop(registry); + drop(owners); + for voter in &mut cluster.voters { + drop(voter.owner.take()); + drop(voter.held.take()); + voter.log.close().await?; + } + drop(std::mem::take(&mut cluster.voters)); + cluster.network.clear(); + let directory = cluster._root.path().to_owned(); + cluster + .domain + .blocking(move || -> Result<()> { + for n in 1..=3 { + let path = directory.join(format!("node-{n}")); + assert!(path.join("log").exists()); + assert!(path.join("images").exists()); + std::fs::remove_dir_all(&path)?; + assert!(!path.exists()); + let extents = directory.join(format!("extents-{}", n - 1)); + if rooted { + assert!(extents.exists()); + std::fs::remove_dir_all(&extents)?; + assert!(!extents.exists()); + } + } + assert!(!directory.join("archive-data").exists()); + std::fs::remove_dir_all(directory.join("archive-intents"))?; + Ok(()) + })? + .await??; + // Reconstruct the expected registry identity with a fresh catalog owner, + // using only remote metadata after all local intent journals were erased. + let (directory, birth) = binding_fixture::reopen_directory(&cluster).await?; + let registry = directory + .open_group_archive(&cluster.domain, &archive_pool, &birth) + .await?; + let archive = registry.archive(); + // Open only physical data stores, after erasure. No bootstrap permit, local + // member/disk identity, Raft metadata or replacement genesis is fabricated. + let destination = destination_mode(&cluster, rooted, Arc::new(DirectFileSystem)).await?; + let receivers = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let recovered = archive + .verify_tail( + &cluster.domain, + &tails, + &reference, + receiver(&receivers, &destination, &cluster)?, + ) + .await?; + assert_eq!(recovered.manifest().marker(), &marker); + assert_eq!(recovered.membership(), &membership); + tail_fixture::check_images(&cluster, recovered.manifest(), &expected).await?; + assert!(cluster.voters.is_empty()); + // Pins protect all currently returned materialized images; releasing the + // proof must also release private image retention and both admission pools. + if !rooted { + let images = destination.images.clone(); + assert_eq!( + cluster + .domain + .blocking(move || images.reclaim(4096))? + .await?? + .files, + 0 + ); + } + drop(recovered); + assert_eq!(receivers.usage().sources, 0); + assert_eq!(tails.usage().operations, 0); + if !rooted { + let images = destination.images.clone(); + assert!( + cluster + .domain + .blocking(move || images.reclaim(4096))? + .await?? + .files + > 0 + ); + // Re-fetching the whole closure also works after private cache eviction. + let recovered = archive + .verify_tail( + &cluster.domain, + &tails, + &reference, + receiver(&receivers, &destination, &cluster)?, + ) + .await?; + tail_fixture::check_images(&cluster, recovered.manifest(), &expected).await?; + } + let pending = receiver(&receivers, &destination, &cluster)?; + destination.store.close(); + assert!(archive + .verify_checkpoint(&cluster.domain, &base, pending) + .await + .is_err()); + assert!(receiver(&receivers, &destination, &cluster).is_err()); + assert_eq!(receivers.usage().sources, 0); + cluster.close().await?; + Ok(()) +} + +pub(super) async fn moving_cut(rooted: bool) -> Result<()> { + let (mut cluster, expected) = + checkpoint_fixture::populate_images(rooted, true, false, 1, 8192).await?; + let remote = archive_store::Store::new(); + let archive = new_archive( + &cluster, + remote.clone(), + Path::from("moving-cut"), + Default::default(), + ) + .await?; + let sources = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let original = source(&sources, &cluster)?; + let marker = original.marker().clone(); + let reference = archive.upload_checkpoint(&cluster.domain, original).await?; + let (path, index) = index(remote.as_ref(), &reference).await?; + let gate = remote.hold(cell_objects(&path, &index, "beta")[0].clone()); + let destination = destination(&cluster).await?; + let receivers = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let archive_domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let mut pending = Box::pin(archive.verify_checkpoint( + &archive_domain, + &reference, + receiver(&receivers, &destination, &cluster)?, + )); + let entered = Box::pin(tokio::time::timeout( + Duration::from_secs(5), + gate.entered.notified(), + )); + match futures_util::future::select(entered, pending.as_mut()).await { + futures_util::future::Either::Left((result, _)) => result?, + futures_util::future::Either::Right(_) => { + anyhow::bail!("archive ended before held dependency") + } + } + checkpoint_fixture::advance_partition( + &mut cluster, + "alpha", + &expected["alpha"], + rooted, + Some(2), + ) + .await?; + let group = cluster.voters[0] + .log + .cell_group(&cluster.birth.route().history)?; + let advanced = group.group_manifest()?.context("advanced serving cut")?; + assert!(advanced.position().index > marker.position().index); + remote.clear(); + gate.release.notify_one(); + let recovered = pending.await?; + assert_eq!(recovered.checkpoint().marker(), &marker); + tail_fixture::check_images(&cluster, recovered.checkpoint().manifest(), &expected).await?; + assert_eq!(group.group_manifest()?.as_ref(), Some(&advanced)); + assert!(matches!( + sources.capture(&group, recovered.checkpoint().manifest()), + Err(CellGroupCheckpointError::Invalid("source root/owner")) + )); + drop(recovered); + // Verifying a newer checkpoint does not prevent selecting an older archive + // cut later, and neither result may mutate or publish a live engine root. + let latest = archive + .upload_checkpoint(&cluster.domain, source(&sources, &cluster)?) + .await?; + let newer = archive + .verify_checkpoint( + &cluster.domain, + &latest, + receiver(&receivers, &destination, &cluster)?, + ) + .await?; + assert_eq!(newer.checkpoint().marker(), &advanced); + drop(cluster.voters[0].owner.take()); + let mut write = GroupWrite::new(&group); + write.group_manifest = Some(newer.checkpoint().manifest().clone()); + assert!( + matches!(cluster.voters[0].log.persist_cells(OperationId::new(777), vec![write]).await, + Err(LogError::Invalid(message)) if message == "invalid cell-group apply binding or prefix") + ); + assert_eq!(group.group_manifest()?.as_ref(), Some(&advanced)); + drop(newer); + let older = archive + .verify_checkpoint( + &cluster.domain, + &reference, + receiver(&receivers, &destination, &cluster)?, + ) + .await?; + assert_eq!(older.checkpoint().marker(), &marker); + tail_fixture::check_images(&cluster, older.checkpoint().manifest(), &expected).await?; + drop(older); + assert_eq!(receivers.usage().sources, 0); + archive_domain.shutdown().await; + cluster.close().await?; + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_archive_publication.rs b/crates/cellar/tests/support/cell_group_archive_publication.rs new file mode 100644 index 000000000..3472070a5 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_archive_publication.rs @@ -0,0 +1,779 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Actual conditional object-store operations, three-voter engines and physical +//! SQLite/LTX verification. Publication is historical coverage, never serving. +use super::*; +use std::collections::BTreeMap; + +pub(super) async fn moving_upload_cut(rooted: bool) -> Result<()> { + let mut s = Scenario::new(rooted).await?; + let checkpoint = s.checkpoint(&s.upload).await?; + let (path, index) = index(s.remote.as_ref(), &checkpoint).await?; + let sources = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let captured = source(&sources, &s.cluster)?; + let original = captured.marker().clone(); + let gate = s + .remote + .hold(cell_objects(&path, &index, "beta")[0].clone()); + let mut pending = Box::pin(s.upload.upload_cut(&s.archive_domain, &s.tails, captured)); + let entered = Box::pin(tokio::time::timeout( + Duration::from_secs(5), + gate.entered.notified(), + )); + match futures_util::future::select(entered, pending.as_mut()).await { + futures_util::future::Either::Left((result, _)) => result?, + futures_util::future::Either::Right(_) => anyhow::bail!("cut upload skipped held B"), + } + assert_eq!(sources.usage().sources, 1); + checkpoint_fixture::advance_partition( + &mut s.cluster, + "alpha", + &s.expected["alpha"], + rooted, + Some(2), + ) + .await?; + assert!( + s.cluster.voters[0] + .owner + .as_ref() + .unwrap() + .manifest() + .marker() + .position() + .index + > original.position().index + ); + s.remote.clear(); + gate.release.notify_one(); + let uploaded = pending.await?; + assert_eq!(sources.usage().sources, 0); + // The same single-source pool must admit the independent receiver. Upload + // cannot retain its source slot while waiting for physical verification. + let prepared = s + .upload + .prepare_publication( + &s.archive_domain, + &s.tails, + &uploaded, + receiver(&sources, &s.destination, &s.cluster)?, + ) + .await?; + let intent = persist_archive_intent(&s.cluster, prepared.intent()).await?; + assert_eq!(&intent, prepared.intent()); + let coverage = prepared.commit(&s.archive_domain).await?; + assert_eq!(coverage.marker(), &original); + tail_fixture::check_images(&s.cluster, coverage.recovery().manifest(), &s.expected).await?; + let hold = coverage.hold().clone(); + drop(coverage); + drop(uploaded); + assert_eq!(sources.usage().sources, 0); + s.registry.release(&s.cluster.domain, &hold).await?; + s.finish().await +} + +struct Scenario { + cluster: Cluster, + expected: BTreeMap>, + remote: Arc, + registry: CellGroupArchiveRegistry, + upload: CellGroupArchiveUpload, + owners: CellGroupArchiveUploadPool, + tails: CellGroupArchiveTailPool, + destination: ArchiveDestination, + receivers: CellGroupCheckpointPool, + archive_domain: ExecutionDomain, + gc_domain: ExecutionDomain, + gc_registry: CellGroupArchiveRegistry, + gc_owners: CellGroupArchiveUploadPool, + gc_tails: CellGroupArchiveTailPool, +} + +impl Scenario { + async fn new(rooted: bool) -> Result { + Self::with_owner_limits(rooted, Default::default()).await + } + + async fn with_owner_limits(rooted: bool, limits: CellGroupArchiveUploadLimits) -> Result { + let (cluster, expected) = + checkpoint_fixture::populate_images(rooted, true, false, 1, 8192).await?; + let remote = archive_store::Store::new(); + let archive = CellGroupArchivePool::new(Default::default())?.for_group( + remote.clone(), + Path::from("protected-head"), + cluster.birth.route().history, + )?; + let creation = archive + .prepare_registry_creation(&cluster.domain, cluster.birth.route().clone()) + .await?; + let saved = persist_archive_intent(&cluster, creation.intent()).await?; + assert_eq!(creation.intent(), &saved); + let registry = creation.commit(&cluster.domain).await?; + let owners = CellGroupArchiveUploadPool::new(limits)?; + let tails = CellGroupArchiveTailPool::new(Default::default())?; + let upload = next_upload(&cluster, ®istry, &owners, &tails).await?; + let destination = destination(&cluster).await?; + let receivers = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let archive_domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + // The collector is a separate node/domain with its own admission pools. + // Both nodes coordinate exclusively through real conditional registry I/O. + let gc_domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let gc_archive = CellGroupArchivePool::new(Default::default())?.for_group( + remote.clone(), + Path::from("protected-head"), + cluster.birth.route().history, + )?; + let gc_registry = gc_archive.open_registry(&gc_domain, &saved).await?; + Ok(Self { + cluster, + expected, + remote, + registry, + upload, + owners, + tails, + destination, + receivers, + archive_domain, + gc_domain, + gc_registry, + gc_owners: CellGroupArchiveUploadPool::new(Default::default())?, + gc_tails: CellGroupArchiveTailPool::new(Default::default())?, + }) + } + + async fn checkpoint( + &self, + upload: &CellGroupArchiveUpload, + ) -> Result { + let sources = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + Ok(upload + .upload_checkpoint(&self.archive_domain, source(&sources, &self.cluster)?) + .await?) + } + + async fn tail( + &self, + upload: &CellGroupArchiveUpload, + checkpoint: &CellGroupArchiveReference, + ) -> Result { + let group = self.cluster.voters[0] + .log + .cell_group(&self.cluster.birth.route().history)?; + let root = self.cluster.voters[0].owner.as_ref().unwrap().manifest(); + Ok(upload + .upload_tail(&self.archive_domain, &self.tails, checkpoint, &group, root) + .await?) + } + + async fn publish( + &self, + upload: &CellGroupArchiveUpload, + uploaded: &CellGroupArchiveUploadedTail, + ) -> Result { + let prepared = upload + .prepare_publication( + &self.archive_domain, + &self.tails, + uploaded, + receiver(&self.receivers, &self.destination, &self.cluster)?, + ) + .await?; + let saved = persist_archive_intent(&self.cluster, prepared.intent()).await?; + assert_eq!(prepared.intent(), &saved); + Ok(prepared.commit(&self.archive_domain).await?) + } + + async fn collect(&self) -> Result { + Ok(self + .gc_registry + .collect( + &self.gc_domain, + &self.gc_owners, + &self.gc_tails, + Default::default(), + ) + .await?) + } + + async fn finish(self) -> Result<()> { + self.archive_domain.shutdown().await; + self.gc_domain.shutdown().await; + self.cluster.close().await + } +} + +async fn next_upload( + cluster: &Cluster, + registry: &CellGroupArchiveRegistry, + owners: &CellGroupArchiveUploadPool, + tails: &CellGroupArchiveTailPool, +) -> Result { + let intent = registry.prepare_upload(&cluster.domain).await?; + let saved = persist_archive_intent(cluster, &intent).await?; + let ticket = registry.reserve(&cluster.domain, &saved).await?; + Ok(registry + .open_upload(&cluster.domain, owners, &ticket, tails) + .await?) +} + +async fn control_path(remote: &dyn ObjectStore) -> Result { + Ok(remote + .list(None) + .try_collect::>() + .await? + .into_iter() + .find(|m| m.location.as_ref().ends_with("/ownership/registry")) + .context("registry path")? + .location) +} + +fn object_path(prefix: &str, kind: &str, object: &serde_json::Value) -> Result { + let digest: Vec = serde_json::from_value(object["digest"].clone())?; + Ok(Path::from(format!( + "{prefix}/registries/{}/generations/{}/{kind}/{}-{}", + hex::encode(serde_json::from_value::>( + object["registry"].clone() + )?), + object["generation"], + object["length"], + hex::encode(digest) + ))) +} + +async fn held_root_path(remote: &dyn ObjectStore, hold: &CellGroupArchiveHold) -> Result { + let path = control_path(remote).await?; + let prefix = path + .as_ref() + .strip_suffix("/ownership/registry") + .context("registry namespace")?; + object_path( + prefix, + "roots", + &serde_json::to_value(hold)?["root"]["object"], + ) +} + +pub(super) async fn complete_source_cut(rooted: bool) -> Result<()> { + let mut s = Scenario::new(rooted).await?; + let checkpoint = s.checkpoint(&s.upload).await?; + let (_, original) = index(s.remote.as_ref(), &checkpoint).await?; + checkpoint_fixture::advance_partition( + &mut s.cluster, + "alpha", + &s.expected["alpha"], + rooted, + Some(2), + ) + .await?; + s.expected.insert( + "alpha".into(), + images::image(Some(&s.expected["alpha"]), "UPDATE value SET n=n+1")?, + ); + let uploaded = s.tail(&s.upload, &checkpoint).await?; + let (cp_path, _) = index(s.remote.as_ref(), &checkpoint).await?; + let dependency = cell_objects(&cp_path, &original, "beta")[0].clone(); + // A reservation protects the whole in-progress generation before any head + // exists. It cannot justify publishing a missing dependency. + assert_eq!(s.collect().await?.deleted_objects, 0); + let gate = s.remote.hold(dependency.clone()); + let mut pending = Box::pin(s.upload.prepare_publication( + &s.archive_domain, + &s.tails, + &uploaded, + receiver(&s.receivers, &s.destination, &s.cluster)?, + )); + let entered = Box::pin(tokio::time::timeout( + Duration::from_secs(5), + gate.entered.notified(), + )); + match futures_util::future::select(entered, pending.as_mut()).await { + futures_util::future::Either::Left((result, _)) => result?, + futures_util::future::Either::Right(_) => { + anyhow::bail!("publication ended before delayed B") + } + } + assert!(matches!( + s.registry.prepare_acquire(&s.cluster.domain).await, + Err(CellGroupArchiveError::Unconfirmed) + )); + // Source authority was captured with the tail, before the source advanced. + // Another cell can commit without invalidating that immutable committed cut. + checkpoint_fixture::advance_partition( + &mut s.cluster, + "beta", + &s.expected["beta"], + rooted, + Some(2), + ) + .await?; + let later = s.cluster.voters[0] + .owner + .as_ref() + .unwrap() + .manifest() + .marker() + .clone(); + s.remote.clear(); + gate.release.notify_one(); + let prepared = pending.await?; + let intent = persist_archive_intent(&s.cluster, prepared.intent()).await?; + assert_eq!(prepared.intent(), &intent); + let coverage = prepared.commit(&s.archive_domain).await?; + assert!(coverage.marker().position().index < later.position().index); + tail_fixture::check_images(&s.cluster, coverage.recovery().manifest(), &s.expected).await?; + let hold = coverage.hold().clone(); + drop(coverage); + // A later A cut still includes unchanged B. Corrupt B on the actual store, + // and no new head or remote reference may be created. + let next = next_upload(&s.cluster, &s.registry, &s.owners, &s.tails).await?; + let extended = next + .extend_tail( + &s.archive_domain, + &s.tails, + uploaded.reference(), + &s.cluster.voters[0] + .log + .cell_group(&s.cluster.birth.route().history)?, + s.cluster.voters[0].owner.as_ref().unwrap().manifest(), + ) + .await?; + let before = s + .remote + .get(&control_path(s.remote.as_ref()).await?) + .await? + .bytes() + .await?; + let bytes = s.remote.inner.get(&dependency).await?.bytes().await?; + let mut corrupt = bytes.to_vec(); + corrupt[0] ^= 0x80; + s.remote.inner.put(&dependency, corrupt.into()).await?; + assert!(next + .prepare_publication( + &s.archive_domain, + &s.tails, + &extended, + receiver(&s.receivers, &s.destination, &s.cluster)? + ) + .await + .is_err()); + let after = s + .remote + .get(&control_path(s.remote.as_ref()).await?) + .await? + .bytes() + .await?; + assert_eq!(before, after); + let deletes = s.remote.deletes.load(Ordering::Acquire); + assert!(s.collect().await.is_err()); + assert_eq!( + s.remote.deletes.load(Ordering::Acquire), + deletes, + "a partial mark must issue no DELETE" + ); + s.remote.inner.put(&dependency, bytes.into()).await?; + let restored = s + .registry + .verify_hold( + &s.archive_domain, + &hold, + &s.tails, + receiver(&s.receivers, &s.destination, &s.cluster)?, + ) + .await?; + tail_fixture::check_images(&s.cluster, restored.recovery().manifest(), &s.expected).await?; + drop(restored); + s.registry.abort(&s.archive_domain, &next.ticket()).await?; + s.finish().await +} + +pub(super) async fn holds_and_pending_parents(rooted: bool) -> Result<()> { + let mut s = Scenario::new(rooted).await?; + let first = s.checkpoint(&s.upload).await?; + let first_tail = s.tail(&s.upload, &first).await?; + let first_coverage = s.publish(&s.upload, &first_tail).await?; + let first_hold = first_coverage.hold().clone(); + let first_root = held_root_path(s.remote.as_ref(), &first_hold).await?; + drop(first_coverage); + let intent = s.registry.prepare_acquire(&s.archive_domain).await?; + let intent = persist_archive_intent(&s.cluster, &intent).await?; + let reader = s.registry.acquire(&s.archive_domain, &intent).await?; + // Keep a child reservation on the old head while a different publisher wins. + let pending = next_upload(&s.cluster, &s.registry, &s.owners, &s.tails).await?; + let newer = next_upload(&s.cluster, &s.registry, &s.owners, &s.tails).await?; + checkpoint_fixture::advance_partition( + &mut s.cluster, + "alpha", + &s.expected["alpha"], + rooted, + Some(2), + ) + .await?; + s.expected.insert( + "alpha".into(), + images::image(Some(&s.expected["alpha"]), "UPDATE value SET n=n+1")?, + ); + let second = s.checkpoint(&newer).await?; + let second_tail = s.tail(&newer, &second).await?; + let second_coverage = s.publish(&newer, &second_tail).await?; + let second_hold = second_coverage.hold().clone(); + drop(second_coverage); + s.registry.release(&s.archive_domain, &first_hold).await?; + s.collect().await?; + assert!( + s.remote.get(&first_root).await.is_ok(), + "an independent reader hold protects the old root" + ); + s.registry.release(&s.archive_domain, &reader).await?; + s.collect().await?; + assert!( + s.remote.get(&first_root).await.is_ok(), + "pending child retains its exact parent" + ); + s.registry + .abort(&s.archive_domain, &pending.ticket()) + .await?; + // All old bytes still exist, but the current head's complete checkpoint + // no longer owns the old checkpoint index. A new publisher cannot resurrect + // that arbitrary old reference, even after successful physical replay. + let unprotected = next_upload(&s.cluster, &s.registry, &s.owners, &s.tails).await?; + let candidate = unprotected + .extend_tail( + &s.archive_domain, + &s.tails, + first_tail.reference(), + &s.cluster.voters[0] + .log + .cell_group(&s.cluster.birth.route().history)?, + s.cluster.voters[0].owner.as_ref().unwrap().manifest(), + ) + .await?; + let control = control_path(s.remote.as_ref()).await?; + let before = s.remote.inner.get(&control).await?.bytes().await?; + assert!(matches!( + unprotected + .prepare_publication( + &s.archive_domain, + &s.tails, + &candidate, + receiver(&s.receivers, &s.destination, &s.cluster)? + ) + .await, + Err(CellGroupArchiveError::Invalid( + "unprotected inherited archive object" + )) + )); + assert_eq!(s.remote.inner.get(&control).await?.bytes().await?, before); + s.registry + .abort(&s.archive_domain, &unprotected.ticket()) + .await?; + drop(candidate); + drop(unprotected); + let report = s.collect().await?; + assert!(report.deleted_objects > 0); + assert!(matches!( + s.remote.get(&first_root).await, + Err(object_store::Error::NotFound { .. }) + )); + let recovered = s + .registry + .verify_hold( + &s.archive_domain, + &second_hold, + &s.tails, + receiver(&s.receivers, &s.destination, &s.cluster)?, + ) + .await?; + tail_fixture::check_images(&s.cluster, recovered.recovery().manifest(), &s.expected).await?; + drop(recovered); + // Retired tokens and allocations do not resurrect an owner on retry. + assert!(s + .registry + .acquire(&s.archive_domain, &intent) + .await + .is_err()); + assert!(s + .registry + .open_upload(&s.archive_domain, &s.owners, &pending.ticket(), &s.tails) + .await + .is_err()); + s.finish().await +} + +pub(super) async fn uncertain_publication(rooted: bool) -> Result<()> { + let mut s = Scenario::new(rooted).await?; + let checkpoint = s.checkpoint(&s.upload).await?; + let uploaded = s.tail(&s.upload, &checkpoint).await?; + let prepared = s + .upload + .prepare_publication( + &s.archive_domain, + &s.tails, + &uploaded, + receiver(&s.receivers, &s.destination, &s.cluster)?, + ) + .await?; + let saved = persist_archive_intent(&s.cluster, prepared.intent()).await?; + let path = control_path(s.remote.as_ref()).await?; + let gate = s + .remote + .hold_after(path, s.remote.puts.load(Ordering::Acquire) + 1); + s.remote.lose_put_response.store(true, Ordering::Release); + let mut pending = Box::pin(prepared.commit(&s.archive_domain)); + let entered = Box::pin(tokio::time::timeout( + Duration::from_secs(5), + gate.entered.notified(), + )); + match futures_util::future::select(entered, pending.as_mut()).await { + futures_util::future::Either::Left((result, _)) => result?, + futures_util::future::Either::Right(_) => anyhow::bail!("commit ended before readback cut"), + } + drop(pending); // The conditional write landed, but the producer has no reply. + s.remote.clear(); + gate.release.notify_one(); + assert_eq!(s.tails.usage().operations, 0); + assert_eq!(s.receivers.usage().sources, 0); + for voter in &mut s.cluster.voters { + drop(voter.owner.take()); + drop(voter.held.take()); + voter.log.close().await?; + } + let puts = s.remote.puts.load(Ordering::Acquire); + let coverage = s + .registry + .reconcile_publication( + &s.archive_domain, + &saved, + &s.tails, + receiver(&s.receivers, &s.destination, &s.cluster)?, + ) + .await?; + tail_fixture::check_images(&s.cluster, coverage.recovery().manifest(), &s.expected).await?; + assert_eq!( + s.remote.puts.load(Ordering::Acquire), + puts, + "reconciliation must not reallocate or republish" + ); + assert!( + s.registry.reserve(&s.archive_domain, &saved).await.is_err(), + "serialized publication intents are not publishing authority" + ); + let hold = coverage.hold().clone(); + drop(coverage); + s.registry.release(&s.archive_domain, &hold).await?; + assert!(s + .registry + .reconcile_publication( + &s.archive_domain, + &saved, + &s.tails, + receiver(&s.receivers, &s.destination, &s.cluster)? + ) + .await + .is_err()); + s.finish().await +} + +pub(super) async fn delayed_delete(rooted: bool) -> Result<()> { + let s = Scenario::new(rooted).await?; + let first = s.checkpoint(&s.upload).await?; + let (old_index, old_metadata) = index(s.remote.as_ref(), &first).await?; + s.registry + .abort(&s.archive_domain, &s.upload.ticket()) + .await?; + let gate = s.remote.hold_delete(old_index.clone()); + let mut collection = Box::pin(s.gc_registry.collect( + &s.gc_domain, + &s.gc_owners, + &s.gc_tails, + Default::default(), + )); + let entered = Box::pin(tokio::time::timeout( + Duration::from_secs(5), + gate.entered.notified(), + )); + match futures_util::future::select(entered, collection.as_mut()).await { + futures_util::future::Either::Left((result, _)) => result?, + futures_util::future::Either::Right(_) => { + anyhow::bail!("collector ended before held DELETE") + } + } + let next = next_upload(&s.cluster, &s.registry, &s.owners, &s.tails).await?; + let second = s.checkpoint(&next).await?; + let (new_index, new_metadata) = index(s.remote.as_ref(), &second).await?; + assert_ne!(first.generation(), second.generation()); + assert_ne!(old_index, new_index); + assert_eq!( + old_metadata["chunks"][0]["object"]["digest"], + new_metadata["chunks"][0]["object"]["digest"] + ); + assert_ne!( + old_metadata["chunks"][0]["object"]["generation"], + new_metadata["chunks"][0]["object"]["generation"] + ); + let uploaded = s.tail(&next, &second).await?; + let coverage = s.publish(&next, &uploaded).await?; + let hold = coverage.hold().clone(); + drop(coverage); + gate.release.notify_one(); + assert!(collection.await?.deleted_objects > 0); + assert!(s.remote.get(&old_index).await.is_err()); + assert!(s.remote.get(&new_index).await.is_ok()); + let recovered = s + .registry + .verify_hold( + &s.archive_domain, + &hold, + &s.tails, + receiver(&s.receivers, &s.destination, &s.cluster)?, + ) + .await?; + tail_fixture::check_images(&s.cluster, recovered.recovery().manifest(), &s.expected).await?; + drop(recovered); + s.finish().await +} + +pub(super) async fn registry_identity_and_intents(rooted: bool) -> Result<()> { + let mut s = Scenario::new(rooted).await?; + let first = s.registry.prepare_upload(&s.archive_domain).await?; + let stale = s.registry.prepare_upload(&s.archive_domain).await?; + assert_ne!( + first, stale, + "distinct callers must not share an allocation" + ); + let saved = persist_archive_intent(&s.cluster, &first).await?; + let ticket = s.registry.reserve(&s.archive_domain, &saved).await?; + let puts = s.remote.puts.load(Ordering::Acquire); + assert_eq!(s.registry.reserve(&s.archive_domain, &saved).await?, ticket); + assert!(s.registry.reserve(&s.archive_domain, &stale).await.is_err()); + assert_eq!(s.remote.puts.load(Ordering::Acquire), puts); + s.registry.abort(&s.archive_domain, &ticket).await?; + let puts = s.remote.puts.load(Ordering::Acquire); + assert!(s.registry.reserve(&s.archive_domain, &saved).await.is_err()); + assert_eq!( + s.remote.puts.load(Ordering::Acquire), + puts, + "an aborted allocation never reappears" + ); + let next = s.registry.prepare_upload(&s.archive_domain).await?; + let saved = persist_archive_intent(&s.cluster, &next).await?; + let newer = s.registry.reserve(&s.archive_domain, &saved).await?; + assert!(newer.generation() > ticket.generation()); + + // Opening requires the owner's persisted identity. Counter loss refuses and + // cannot be repaired by creating a new identity in the same namespace. + let path = control_path(s.remote.as_ref()).await?; + let identity_path = Path::from( + path.as_ref() + .replace("/ownership/registry", "/ownership/identity"), + ); + let identity = s.remote.inner.get(&identity_path).await?.bytes().await?; + s.remote.inner.delete(&path).await?; + assert!(s + .registry + .archive() + .open_registry(&s.archive_domain, s.registry.creation()) + .await + .is_err()); + let fresh = s + .registry + .archive() + .prepare_registry_creation(&s.archive_domain, s.cluster.birth.route().clone()) + .await?; + let saved = persist_archive_intent(&s.cluster, fresh.intent()).await?; + assert_ne!(&saved, s.registry.creation()); + assert!(fresh.commit(&s.archive_domain).await.is_err()); + assert!(s.remote.inner.get(&path).await.is_err()); + assert_eq!( + s.remote.inner.get(&identity_path).await?.bytes().await?, + identity + ); + // The unavailable S3 metadata does not stop the independent local quorum. + checkpoint_fixture::advance_partition( + &mut s.cluster, + "alpha", + &s.expected["alpha"], + rooted, + Some(2), + ) + .await?; + + // Losing the identity-create response leaves no permission to initialize or + // reset a registry on restart. A serialized intent is opening metadata only. + let uncertain = CellGroupArchivePool::new(Default::default())?.for_group( + s.remote.clone(), + Path::from("uncertain-initialization"), + s.cluster.birth.route().history, + )?; + let creation = uncertain + .prepare_registry_creation(&s.archive_domain, s.cluster.birth.route().clone()) + .await?; + let saved = persist_archive_intent(&s.cluster, creation.intent()).await?; + s.remote.lose_put_response.store(true, Ordering::Release); + assert!(matches!( + creation.commit(&s.archive_domain).await, + Err(CellGroupArchiveError::Unconfirmed) + )); + assert!(uncertain + .open_registry(&s.archive_domain, &saved) + .await + .is_err()); + let controls: Vec<_> = s + .remote + .list(Some(&Path::from("uncertain-initialization"))) + .try_collect() + .await?; + assert_eq!(controls.len(), 1); + assert!(controls[0] + .location + .as_ref() + .ends_with("/ownership/identity")); + s.finish().await +} + +pub(super) async fn source_owner_lifetimes(rooted: bool) -> Result<()> { + let mut s = Scenario::with_owner_limits( + rooted, + CellGroupArchiveUploadLimits { + max_owners: 2, + ..Default::default() + }, + ) + .await?; + assert_eq!(s.owners.usage().operations, 1); + let checkpoint = s.checkpoint(&s.upload).await?; + let first = s.tail(&s.upload, &checkpoint).await?; + assert_eq!(s.owners.usage().operations, 2); + let retained = first.clone(); + drop(first); + assert_eq!( + s.owners.usage().operations, + 2, + "clones retain the source witness owner" + ); + assert!(matches!(s.tail(&s.upload, &checkpoint).await, + Err(error) if error.downcast_ref::().is_some_and(|e| matches!(e, CellGroupArchiveError::Busy)))); + assert_eq!(s.tails.usage().operations, 0); + checkpoint_fixture::advance_partition( + &mut s.cluster, + "alpha", + &s.expected["alpha"], + rooted, + Some(2), + ) + .await?; + s.expected.insert( + "alpha".into(), + images::image(Some(&s.expected["alpha"]), "UPDATE value SET n=n+1")?, + ); + let _persistent_locator = retained.into_reference(); + assert_eq!(s.owners.usage().operations, 1); + let latest = s.tail(&s.upload, &checkpoint).await?; + assert_eq!(s.owners.usage().operations, 2); + let coverage = s.publish(&s.upload, &latest).await?; + tail_fixture::check_images(&s.cluster, coverage.recovery().manifest(), &s.expected).await?; + drop(coverage); + drop(latest); + assert_eq!(s.owners.usage().operations, 1); + s.finish().await +} diff --git a/crates/cellar/tests/support/cell_group_archive_store.rs b/crates/cellar/tests/support/cell_group_archive_store.rs new file mode 100644 index 000000000..b3009fa95 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_archive_store.rs @@ -0,0 +1,264 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Instrument the selected object-store adapter, without replacing its +//! conditional-create semantics, bytes or physical checkpoint verification. +use futures_util::stream::BoxStream; +use object_store::{path::Path, *}; +use std::sync::{ + atomic::{AtomicBool, AtomicUsize, Ordering}, + Arc, Mutex, +}; +use tokio::sync::Notify; + +#[derive(Debug, Default)] +pub struct Gate { + pub entered: Notify, + pub release: Notify, +} + +#[derive(Debug)] +pub struct Store { + pub inner: Arc, + pub puts: AtomicUsize, + pub gets: AtomicUsize, + pub lose_put_response: AtomicBool, + pub lose_immutable_put_response: AtomicBool, + pub lose_directory_head_and_reads: AtomicBool, + pub deletes: AtomicUsize, + pub lose_delete_response: AtomicBool, + held: Mutex)>>, + held_put: Mutex)>>, + held_delete: Mutex)>>, + publication: Mutex>>, + pub publication_cuts: Mutex>, + pub outage: AtomicBool, + pub lose_publication_and_reads: AtomicBool, +} + +impl Store { + pub fn new() -> Arc { + Self::around(Arc::new(memory::InMemory::new())) + } + + pub fn around(inner: Arc) -> Arc { + Arc::new(Self { + inner, + puts: AtomicUsize::new(0), + gets: AtomicUsize::new(0), + lose_put_response: AtomicBool::new(false), + lose_immutable_put_response: AtomicBool::new(false), + lose_directory_head_and_reads: AtomicBool::new(false), + deletes: AtomicUsize::new(0), + lose_delete_response: AtomicBool::new(false), + held: Mutex::new(None), + held_put: Mutex::new(None), + held_delete: Mutex::new(None), + publication: Mutex::new(None), + publication_cuts: Mutex::new(Vec::new()), + outage: AtomicBool::new(false), + lose_publication_and_reads: AtomicBool::new(false), + }) + } + + pub fn hold_publication(&self) -> Arc { + let gate = Arc::new(Gate::default()); + *self.publication.lock().unwrap() = Some(gate.clone()); + gate + } + + fn available(&self) -> Result<()> { + if self.outage.load(Ordering::Acquire) { + return Err(Error::Generic { + store: "archive outage", + source: "injected".into(), + }); + } + Ok(()) + } + + pub fn hold(&self, path: Path) -> Arc { + self.hold_after(path, 0) + } + + pub fn hold_after(&self, path: Path, puts: usize) -> Arc { + let gate = Arc::new(Gate::default()); + *self.held.lock().unwrap() = Some((path, puts, gate.clone())); + gate + } + + pub fn hold_delete(&self, path: Path) -> Arc { + let gate = Arc::new(Gate::default()); + *self.held_delete.lock().unwrap() = Some((path, gate.clone())); + gate + } + + pub fn hold_put(&self, path: Path) -> Arc { + let gate = Arc::new(Gate::default()); + *self.held_put.lock().unwrap() = Some((path, gate.clone())); + gate + } + + pub fn clear_put(&self) { + self.held_put.lock().unwrap().take(); + } + + pub fn clear(&self) { + self.held.lock().unwrap().take(); + } +} + +impl std::fmt::Display for Store { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.write_str("cell-group archive instrumented store") + } +} + +#[async_trait::async_trait] +impl ObjectStore for Store { + async fn put_opts( + &self, + path: &Path, + payload: PutPayload, + opts: PutOptions, + ) -> Result { + self.available()?; + self.puts.fetch_add(1, Ordering::AcqRel); + let gate = self + .held_put + .lock() + .unwrap() + .as_ref() + .filter(|(held, _)| held == path) + .map(|(_, gate)| gate.clone()); + if let Some(gate) = gate { + gate.entered.notify_one(); + gate.release.notified().await; + } + let publication = if path.as_ref().ends_with("/ownership/registry") { + let bytes: Vec = payload + .iter() + .flat_map(|bytes| bytes.iter().copied()) + .collect(); + let value: serde_json::Value = serde_json::from_slice(&bytes).unwrap(); + value["selected"]["marker"]["index"].as_u64() + } else { + None + }; + if publication.is_some() { + let gate = self.publication.lock().unwrap().take(); + if let Some(gate) = gate { + gate.entered.notify_one(); + gate.release.notified().await; + } + } + let immutable = + matches!(opts.mode, PutMode::Create) && path.as_ref().contains("/generations/"); + let result = self.inner.put_opts(path, payload, opts).await?; + if immutable + && self + .lose_immutable_put_response + .swap(false, Ordering::AcqRel) + { + return Err(Error::Generic { + store: "lost immutable archive PUT response", + source: "injected after accepted create".into(), + }); + } + if let Some(index) = publication { + self.publication_cuts.lock().unwrap().push(index); + } + if publication.is_some() + && self + .lose_publication_and_reads + .swap(false, Ordering::AcqRel) + { + self.outage.store(true, Ordering::Release); + return Err(Error::Generic { + store: "lost archive publication", + source: "injected".into(), + }); + } + if path.as_ref().ends_with("/cell-groups/head") + && self + .lose_directory_head_and_reads + .swap(false, Ordering::AcqRel) + { + self.outage.store(true, Ordering::Release); + return Err(Error::Generic { + store: "lost directory head response", + source: "injected after real CAS".into(), + }); + } + if self.lose_put_response.swap(false, Ordering::AcqRel) { + return Err(Error::Generic { + store: "lost PUT response", + source: "injected".into(), + }); + } + Ok(result) + } + + async fn put_multipart_opts( + &self, + path: &Path, + opts: PutMultipartOptions, + ) -> Result> { + self.inner.put_multipart_opts(path, opts).await + } + + async fn get_opts(&self, path: &Path, opts: GetOptions) -> Result { + self.gets.fetch_add(1, Ordering::AcqRel); + self.available()?; + let gate = self + .held + .lock() + .unwrap() + .as_ref() + .filter(|(held, puts, _)| held == path && self.puts.load(Ordering::Acquire) >= *puts) + .map(|(_, _, gate)| gate.clone()); + if let Some(gate) = gate { + gate.entered.notify_one(); + gate.release.notified().await; + } + self.inner.get_opts(path, opts).await + } + + async fn delete(&self, path: &Path) -> Result<()> { + self.deletes.fetch_add(1, Ordering::AcqRel); + let gate = self + .held_delete + .lock() + .unwrap() + .as_ref() + .filter(|(held, _)| held == path) + .map(|(_, gate)| gate.clone()); + if let Some(gate) = gate { + gate.entered.notify_one(); + gate.release.notified().await; + } + self.inner.delete(path).await?; + if self.lose_delete_response.swap(false, Ordering::AcqRel) { + return Err(Error::Generic { + store: "lost DELETE response", + source: "injected".into(), + }); + } + Ok(()) + } + + fn list(&self, prefix: Option<&Path>) -> BoxStream<'static, Result> { + self.inner.list(prefix) + } + + async fn list_with_delimiter(&self, prefix: Option<&Path>) -> Result { + self.inner.list_with_delimiter(prefix).await + } + + async fn copy(&self, from: &Path, to: &Path) -> Result<()> { + self.inner.copy(from, to).await + } + + async fn copy_if_not_exists(&self, from: &Path, to: &Path) -> Result<()> { + self.inner.copy_if_not_exists(from, to).await + } +} diff --git a/crates/cellar/tests/support/cell_group_archive_tail.rs b/crates/cellar/tests/support/cell_group_archive_tail.rs new file mode 100644 index 000000000..0d535aac8 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_archive_tail.rs @@ -0,0 +1,770 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use raft::prelude::{ConfChangeSingle, ConfChangeTransition, ConfChangeType, ConfChangeV2}; + +pub(super) async fn tail_index( + store: &dyn ObjectStore, + reference: &CellGroupArchiveTailReference, +) -> Result<(Path, serde_json::Value)> { + let suffix = format!( + "/tails/{}-{}", + reference.length(), + hex::encode(reference.digest()) + ); + let objects: Vec<_> = store.list(None).try_collect().await?; + let path = objects + .into_iter() + .find(|m| { + m.location.as_ref().ends_with(&suffix) + && generation_prefix(&m.location) + .is_ok_and(|(_, generation)| generation == reference.generation()) + && registry_id(&m.location).is_ok_and(|id| id == reference.registry()) + }) + .context("tail index")? + .location; + let body = store.get(&path).await?.bytes().await?; + Ok((path, serde_json::from_slice(&body)?)) +} + +fn entry_path(path: &Path, entry: &serde_json::Value) -> Result { + let digest: Vec = serde_json::from_value(entry["object"]["digest"].clone())?; + let (prefix, _) = generation_prefix(path)?; + Ok(Path::from(format!( + "{prefix}/generations/{}/entries/{}-{}", + entry["object"]["generation"], + entry["object"]["length"], + hex::encode(digest) + ))) +} + +async fn write_tail( + store: &dyn ObjectStore, + original: &Path, + tail: &serde_json::Value, +) -> Result { + let bytes = serde_json::to_vec(tail)?; + let digest: [u8; 32] = Sha256::digest(&bytes).into(); + let prefix = original + .as_ref() + .split("/tails/") + .next() + .context("tail prefix")?; + store + .put( + &Path::from(format!( + "{prefix}/tails/{}-{}", + bytes.len(), + hex::encode(digest) + )), + bytes.clone().into(), + ) + .await?; + Ok(serde_json::from_value( + serde_json::json!({"registry": registry_id(original)?, "generation": generation_prefix(original)?.1, "digest": digest, "length": bytes.len()}), + )?) +} + +pub(super) async fn check_images( + cluster: &Cluster, + root: &PreparedGroupManifest, + expected: &std::collections::BTreeMap>, +) -> Result<()> { + assert_eq!(root.pinned_images(), expected.len()); + for (name, wanted) in expected { + let (_, pin) = root + .pin_cell(&CellId::new(name)?) + .context("archived cell")?; + let bytes = cluster + .domain + .blocking(move || -> Result> { + Ok(match pin { + CellImagePin::Empty => Vec::new(), + CellImagePin::Materialized(pin) => pin + .open_reader()? + .read_exact_at(0, pin.state().length as usize)?, + CellImagePin::Extents(pin) => { + let mut bytes = Vec::new(); + for n in 0..pin.manifest().geometry().extent_count() { + bytes.extend( + pin.view() + .read_extent(pin.manifest().extent(n).context("extent")?)?, + ); + } + bytes + } + }) + })? + .await??; + assert_eq!(&bytes, wanted); + } + Ok(()) +} + +pub(super) async fn enrollment_controls(rooted: bool) -> Result<()> { + let mut cluster = Cluster::new().await?; + cluster.elect().await?; + cluster.voters[2].paused = true; + let store = archive_store::Store::new(); + let archive = new_archive( + &cluster, + store.clone(), + Path::from("tail-qualification"), + Default::default(), + ) + .await?; + let sources = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let base = archive + .upload_checkpoint(&cluster.domain, source(&sources, &cluster)?) + .await?; + let mut expected = + checkpoint_fixture::enroll_images(&mut cluster, rooted, false, 1, 8192).await?; + for name in ["alpha", "beta"] { + checkpoint_fixture::advance_partition(&mut cluster, name, &expected[name], rooted, Some(2)) + .await?; + expected.insert( + name.into(), + images::image(Some(&expected[name]), "UPDATE value SET n=n+1")?, + ); + } + let group = cluster.voters[0] + .log + .cell_group(&cluster.birth.route().history)?; + // A genuine explicit joint demotion and leave preserve initial member 3 as + // a learner. Both changes commit on the two healthy voters, with no fake cell. + let mut joint = ConfChangeV2::default(); + joint.set_transition(ConfChangeTransition::Explicit); + joint.changes.push(ConfChangeSingle { + node_id: 3, + change_type: ConfChangeType::AddLearnerNode, + ..Default::default() + }); + let change = + CellGroupMembershipChange::new(cluster.birth.route(), &group.membership()?, joint, &[])?; + cluster.voters[0] + .owner + .as_mut() + .unwrap() + .configure(change)?; + cluster + .drive(false) + .await + .context("explicit joint configuration apply")?; + let leave = CellGroupMembershipChange::new( + cluster.birth.route(), + &group.membership()?, + ConfChangeV2::default(), + &[], + )?; + cluster.voters[0].owner.as_mut().unwrap().configure(leave)?; + cluster.drive(false).await?; + // Real leadership turnover appends the term's genuine Raft no-op after all + // data/control entries. The original leader applies it as a follower. + cluster.voters[0] + .owner + .as_mut() + .unwrap() + .transfer_leader(2)?; + cluster.drive(false).await?; + assert!(cluster.voters[1].owner.as_ref().unwrap().status()?.ready); + let current = cluster.voters[0].owner.as_ref().unwrap().manifest().clone(); + let expected_membership = group.membership()?; + let tails = CellGroupArchiveTailPool::new(Default::default())?; + let reference = archive + .upload_tail(&cluster.domain, &tails, &base, &group, ¤t) + .await?; + let (_, encoded) = tail_index(store.as_ref(), &reference).await?; + assert_eq!(encoded["entries"].as_array().unwrap().len(), 9); + assert_eq!(tails.usage().operations, 0); + let applier = destination(&cluster).await?; + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let receiver_group = cluster.voters[2] + .log + .cell_group(&cluster.birth.route().history)?; + let selected = receiver_group.group_manifest()?; + for voter in &mut cluster.voters[..2] { + drop(voter.owner.take()); + voter.log.close().await?; + } + let recovered = archive + .verify_tail( + &cluster.domain, + &tails, + &reference, + receiver(&pool, &applier, &cluster)?, + ) + .await?; + assert_eq!(recovered.manifest().marker(), current.marker()); + assert_eq!(recovered.reference(), reference.reference()); + assert!(recovered.configuration().voters_outgoing.is_empty()); + assert_eq!(recovered.configuration().learners, vec![3]); + assert_eq!(recovered.membership(), &expected_membership); + assert_eq!(recovered.manifest().marker().root().cells(), 4); + for name in ["alpha", "beta"] { + assert_eq!( + recovered + .manifest() + .pin_cell(&CellId::new(name)?) + .unwrap() + .0 + .version + .sequence + .get(), + 2 + ); + } + assert_eq!(receiver_group.group_manifest()?, selected); + assert_eq!(receiver_group.checkpoint_manifest()?, None); + assert_eq!( + tails.usage().operations, + 1, + "returned closure retains replay admission" + ); + assert_eq!(pool.usage().sources, 1); + check_images(&cluster, recovered.manifest(), &expected).await?; + drop(recovered); + assert_eq!(tails.usage().operations, 0); + assert_eq!(pool.usage().sources, 0); + cluster.close().await?; + Ok(()) +} + +pub(super) async fn dependencies(rooted: bool) -> Result<()> { + let (mut cluster, mut expected) = + checkpoint_fixture::populate_images(rooted, true, false, 1, 8192).await?; + let store = archive_store::Store::new(); + let archive = new_archive( + &cluster, + store.clone(), + Path::from("tail-qualification"), + Default::default(), + ) + .await?; + let sources = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let base = archive + .upload_checkpoint(&cluster.domain, source(&sources, &cluster)?) + .await?; + checkpoint_fixture::advance_partition( + &mut cluster, + "alpha", + &expected["alpha"], + rooted, + Some(2), + ) + .await?; + expected.insert( + "alpha".into(), + images::image(Some(&expected["alpha"]), "UPDATE value SET n=n+1")?, + ); + let group = cluster.voters[0] + .log + .cell_group(&cluster.birth.route().history)?; + let current = cluster.voters[0].owner.as_ref().unwrap().manifest().clone(); + let tails = CellGroupArchiveTailPool::new(Default::default())?; + let reference = archive + .upload_tail(&cluster.domain, &tails, &base, &group, ¤t) + .await?; + let (path, encoded) = tail_index(store.as_ref(), &reference).await?; + let entry = entry_path(&path, &encoded["entries"][0])?; + let entry_bytes = store.get(&entry).await?.bytes().await?; + let (base_path, base_index) = index(store.as_ref(), &base).await?; + let dependency = cell_objects(&base_path, &base_index, "beta") + .into_iter() + .last() + .context("base beta")?; + let dependency_bytes = store.get(&dependency).await?.bytes().await?; + let applier = destination(&cluster).await?; + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + for missing in [&dependency, &entry] { + store.inner.delete(missing).await?; + assert!(matches!( + archive + .verify_tail( + &cluster.domain, + &tails, + &reference, + receiver(&pool, &applier, &cluster)? + ) + .await, + Err(CellGroupArchiveError::Invalid("missing archive object")) + )); + let bytes = if missing == &dependency { + dependency_bytes.clone() + } else { + entry_bytes.clone() + }; + store.inner.put(missing, bytes.into()).await?; + assert_eq!(tails.usage().operations, 0); + assert_eq!(pool.usage().sources, 0); + } + let mut corrupt = entry_bytes.to_vec(); + corrupt[0] ^= 1; + store.inner.put(&entry, corrupt.into()).await?; + assert!(matches!( + archive + .verify_tail( + &cluster.domain, + &tails, + &reference, + receiver(&pool, &applier, &cluster)? + ) + .await, + Err(CellGroupArchiveError::Invalid("object digest/length")) + )); + store.inner.put(&entry, entry_bytes.into()).await?; + let recovered = archive + .verify_tail( + &cluster.domain, + &tails, + &reference, + receiver(&pool, &applier, &cluster)?, + ) + .await?; + check_images(&cluster, recovered.manifest(), &expected).await?; + assert_eq!(recovered.manifest().marker(), current.marker()); + drop(recovered); + for attack in 0..3 { + let mut forged = encoded.clone(); + match attack { + 0 => forged["entries"] = serde_json::json!([]), + 1 => { + forged["entries"][0]["index"] = + serde_json::json!(current.marker().position().index.get() + 1) + } + 2 => forged["cut"]["marker"]["root"] = base_index["marker"]["root"].clone(), + _ => unreachable!(), + } + let forged = write_tail(store.as_ref(), &path, &forged).await?; + let result = archive + .verify_tail( + &cluster.domain, + &tails, + &forged, + receiver(&pool, &applier, &cluster)?, + ) + .await; + match attack { + 0 => assert!(matches!( + result, + Err(CellGroupArchiveError::Invalid("tail domain/cuts/limits")) + )), + 1 => assert!(matches!( + result, + Err(CellGroupArchiveError::Invalid("tail continuity")) + )), + 2 => assert!(matches!( + result, + Err(CellGroupArchiveError::Invalid("replayed complete cut")) + )), + _ => unreachable!(), + } + assert_eq!(tails.usage().operations, 0); + assert_eq!(pool.usage().sources, 0); + } + cluster.close().await?; + Ok(()) +} + +pub(super) async fn warm_replay(rooted: bool) -> Result<()> { + let (mut cluster, before) = + checkpoint_fixture::populate_images(rooted, true, false, 1, 8192).await?; + let store = archive_store::Store::new(); + let archive = new_archive( + &cluster, + store.clone(), + Path::from("tail-warm"), + Default::default(), + ) + .await?; + let sources = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let base = archive + .upload_checkpoint(&cluster.domain, source(&sources, &cluster)?) + .await?; + checkpoint_fixture::advance_partition(&mut cluster, "alpha", &before["alpha"], rooted, Some(2)) + .await?; + let group = cluster.voters[0] + .log + .cell_group(&cluster.birth.route().history)?; + let current = cluster.voters[0].owner.as_ref().unwrap().manifest().clone(); + let tails = CellGroupArchiveTailPool::new(Default::default())?; + let reference = archive + .upload_tail(&cluster.domain, &tails, &base, &group, ¤t) + .await?; + let (path, original) = tail_index(store.as_ref(), &reference).await?; + let entry_location = entry_path(&path, &original["entries"][0])?; + let entry = store.get(&entry_location).await?.bytes().await?; + let applier = destination(&cluster).await?; + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let recovered = archive + .verify_tail( + &cluster.domain, + &tails, + &reference, + receiver(&pool, &applier, &cluster)?, + ) + .await?; + let (_, final_pin) = recovered + .manifest() + .pin_cell(&CellId::new("alpha")?) + .context("warm result")?; + let (_, base_pin) = recovered + .checkpoint() + .checkpoint() + .manifest() + .pin_cell(&CellId::new("alpha")?) + .context("warm base")?; + drop(recovered); + let wrong_image = images::image(Some(&before["alpha"]), "UPDATE value SET n=n+17")?; + let wrong = images::Cut::new(&before["alpha"], &wrong_image, 1, true, false); + if let (CellImagePin::Materialized(base_pin), CellImagePin::Materialized(final_pin)) = + (&base_pin, &final_pin) + { + let store = applier.images.clone(); + let base_pin = base_pin.clone(); + let final_pin = final_pin.clone(); + let wrong = wrong.bytes.clone(); + let before = before["alpha"].clone(); + cluster + .domain + .blocking(move || -> Result<()> { + let mut cut = images::Cut::new(&before, &wrong_image, 1, true, false); + cut.bytes = wrong; + let mut update = cut.update(); + update.base = base_pin.state(); + update.result = final_pin.state(); + assert!( + matches!( + store.prepare(final_pin.id(), Some(&base_pin), &update), + Err(cellar_ltx::generation::GenerationError::Exists) + ), + "ordinary cached preparation does not replay the supplied LTX" + ); + assert!(store + .pin_prepared(final_pin.id(), final_pin.state()) + .is_ok()); + Ok(()) + })? + .await??; + } + // Re-encode a valid group command carrying a different valid SQLite delta, + // while retaining the original declared result. Hash the forged entry/index + // correctly so only actual LTX replay can reject it, including on warm pins. + let fixed = b"cellar-cell-group-archive-entry-1\0".len(); + let context = u32::from_be_bytes(entry[fixed + 18..fixed + 22].try_into()?) as usize; + let data = fixed + 26 + context; + let mut decoded = CellGroupEntry::decode( + &entry[data..], + cluster.birth.route(), + GroupEntryLimits::default(), + )?; + let GroupCommand::Mutation(mutation) = &mut decoded.command else { + anyhow::bail!("expected mutation tail") + }; + let bad = wrong.update(); + mutation.files = vec![GroupLtx { + min_txid: bad.segments[0].min_txid.0, + max_txid: bad.segments[0].max_txid.0, + page_count: bad.segments[0].page_count as u64, + decoded_bytes: bad.segments[0].decoded_bytes, + bytes: &wrong.bytes, + }]; + let command = decoded.encode(cluster.birth.route(), GroupEntryLimits::default())?; + let mut body = entry[..data].to_vec(); + body[fixed + 22..fixed + 26].copy_from_slice(&(command.len() as u32).to_be_bytes()); + body.extend(command); + let digest: [u8; 32] = Sha256::digest(&body).into(); + let mut forged = original.clone(); + forged["entries"][0]["object"] = serde_json::json!({"registry": original["entries"][0]["object"]["registry"], "generation": original["entries"][0]["object"]["generation"], "digest": digest, "length": body.len()}); + forged["bytes"] = serde_json::json!(body.len()); + let object = entry_path(&path, &forged["entries"][0])?; + store.inner.put(&object, body.into()).await?; + let forged = write_tail(store.as_ref(), &path, &forged).await?; + assert!(matches!( + archive + .verify_tail( + &cluster.domain, + &tails, + &forged, + receiver(&pool, &applier, &cluster)? + ) + .await, + Err(CellGroupArchiveError::Replay(_)) + )); + assert_eq!(tails.usage().operations, 0); + // A cached final destination with damaged bytes must also refuse. Repair it + // only in this owned fault fixture; verification must independently resync it. + let physical = match &final_pin { + CellImagePin::Materialized(pin) => cluster + ._root + .path() + .join("archive-data/images") + .join(format!("{}.sqlite", hex::encode(pin.id().bytes()))), + CellImagePin::Extents(pin) => { + let id = pin.manifest().extent(0).context("result extent")?; + cluster + ._root + .path() + .join("archive-data/extents") + .join(format!("e-{:08x}-{}", id.length(), hex::encode(id.bytes()))) + } + CellImagePin::Empty => anyhow::bail!("expected populated result"), + }; + let bytes = std::fs::read(&physical)?; + let mut corrupt = bytes.clone(); + corrupt[100] ^= 0x80; + std::fs::write(&physical, corrupt)?; + assert!(archive + .verify_tail( + &cluster.domain, + &tails, + &reference, + receiver(&pool, &applier, &cluster)? + ) + .await + .is_err()); + assert_eq!(tails.usage().operations, 0); + std::fs::write(physical, bytes)?; + let recovered = archive + .verify_tail( + &cluster.domain, + &tails, + &reference, + receiver(&pool, &applier, &cluster)?, + ) + .await?; + assert_eq!(recovered.manifest().marker(), current.marker()); + drop(recovered); + cluster.close().await?; + Ok(()) +} + +pub(super) async fn physical_cancellation() -> Result<()> { + let gate = fault::Gate::new(None); + let mut cluster = Cluster::with_storage( + Default::default(), + tempfile::tempdir()?, + Arc::new(fault::FaultFileSystem { gate: gate.clone() }), + ) + .await?; + cluster.elect().await?; + cluster.voters[2].paused = true; + let expected = checkpoint_fixture::enroll_images(&mut cluster, false, false, 1, 8192).await?; + let store = archive_store::Store::new(); + let archive = new_archive( + &cluster, + store, + Path::from("tail-cancellation"), + Default::default(), + ) + .await?; + let sources = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let base = archive + .upload_checkpoint(&cluster.domain, source(&sources, &cluster)?) + .await?; + checkpoint_fixture::advance_partition( + &mut cluster, + "alpha", + &expected["alpha"], + false, + Some(2), + ) + .await?; + let group = cluster.voters[0] + .log + .cell_group(&cluster.birth.route().history)?; + let current = cluster.voters[0].owner.as_ref().unwrap().manifest().clone(); + let tails = CellGroupArchiveTailPool::new(Default::default())?; + let reference = archive + .upload_tail(&cluster.domain, &tails, &base, &group, ¤t) + .await?; + let applier = destination_with_filesystem( + &cluster, + Arc::new(fault::FaultFileSystem { gate: gate.clone() }), + ) + .await?; + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let recovered = archive + .verify_tail( + &cluster.domain, + &tails, + &reference, + receiver(&pool, &applier, &cluster)?, + ) + .await?; + let (_, pin) = recovered + .manifest() + .pin_cell(&CellId::new("alpha")?) + .context("physical replay result")?; + let CellImagePin::Materialized(pin) = pin else { + anyhow::bail!("expected materialized fixture") + }; + let path = cluster + ._root + .path() + .join("archive-data/images") + .join(format!("{}.sqlite", hex::encode(pin.id().bytes()))); + drop(recovered); + // Only the replayed result is selected; checkpoint/base reads cannot enter + // this gate. The warm path first recomputes incoming LTX into private scratch. + gate.image_read_paths.lock().unwrap().push(path); + let release = gate.arm(fault::HOLD_IMAGE_READ); + let mut pending = Box::pin(archive.verify_tail( + &cluster.domain, + &tails, + &reference, + receiver(&pool, &applier, &cluster)?, + )); + let entered = Box::pin(tokio::time::timeout( + Duration::from_secs(5), + gate.entered.notified(), + )); + match futures_util::future::select(entered, pending.as_mut()).await { + futures_util::future::Either::Left((result, _)) => result?, + futures_util::future::Either::Right(_) => { + anyhow::bail!("replay ended before selected physical read") + } + } + assert_eq!(archive.usage().operations, 1); + assert_eq!(tails.usage().operations, 1); + assert_eq!(pool.usage().sources, 1); + drop(pending); // Stop the waiter while its physical job remains entered. + assert_eq!(archive.usage().operations, 1); + assert_eq!(tails.usage().operations, 1); + assert_eq!(pool.usage().sources, 1); + checkpoint_fixture::advance_partition(&mut cluster, "beta", &expected["beta"], false, Some(2)) + .await?; + assert!( + cluster.voters[0] + .owner + .as_ref() + .unwrap() + .manifest() + .marker() + .position() + .index + > current.marker().position().index + ); + for voter in &mut cluster.voters { + drop(voter.owner.take()); + drop(voter.held.take()); + voter.log.close().await?; + } + let mut shutdown = Box::pin(cluster.domain.shutdown()); + assert!( + tokio::time::timeout(Duration::from_millis(30), &mut shutdown) + .await + .is_err(), + "domain shutdown must join the held physical replay" + ); + assert_eq!(tails.usage().operations, 1); + drop(release); + tokio::time::timeout(Duration::from_secs(5), &mut shutdown).await?; + drop(shutdown); + assert_eq!(archive.usage().retained_bytes, 0); + assert_eq!(tails.usage().retained_bytes, 0); + assert_eq!(pool.usage().sources, 0); + cluster.close().await?; + Ok(()) +} + +pub(super) async fn extend_compacted(rooted: bool) -> Result<()> { + let (mut cluster, mut expected) = + checkpoint_fixture::populate_images(rooted, true, false, 1, 8192).await?; + let store = archive_store::Store::new(); + let archive = new_archive( + &cluster, + store.clone(), + Path::from("tail-extension"), + Default::default(), + ) + .await?; + let sources = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let base = archive + .upload_checkpoint(&cluster.domain, source(&sources, &cluster)?) + .await?; + checkpoint_fixture::advance_partition( + &mut cluster, + "alpha", + &expected["alpha"], + rooted, + Some(2), + ) + .await?; + expected.insert( + "alpha".into(), + images::image(Some(&expected["alpha"]), "UPDATE value SET n=n+1")?, + ); + let group = cluster.voters[0] + .log + .cell_group(&cluster.birth.route().history)?; + let first = cluster.voters[0].owner.as_ref().unwrap().manifest().clone(); + let tails = CellGroupArchiveTailPool::new(Default::default())?; + let prefix = archive + .upload_tail(&cluster.domain, &tails, &base, &group, &first) + .await?; + let (prefix_path, prefix_index) = tail_index(store.as_ref(), &prefix).await?; + let checkpoint = source(&sources, &cluster)?.prepare_publication()?; + cluster.voters[0] + .owner + .as_mut() + .unwrap() + .publish_checkpoint(checkpoint)?; + cluster.drive_partition(false, Some(2)).await?; + assert_eq!(group.checkpoint_manifest()?.as_ref(), Some(first.marker())); + assert_eq!( + raft::Storage::first_index(&group)?, + first.marker().position().index.get() + 1 + ); + checkpoint_fixture::advance_partition(&mut cluster, "beta", &expected["beta"], rooted, Some(2)) + .await?; + expected.insert( + "beta".into(), + images::image(Some(&expected["beta"]), "UPDATE value SET n=n+1")?, + ); + let latest = cluster.voters[0].owner.as_ref().unwrap().manifest().clone(); + assert!( + matches!( + archive + .upload_tail(&cluster.domain, &tails, &base, &group, &latest) + .await, + Err(CellGroupArchiveError::Invalid("tail source committed cut")) + ), + "the original local prefix is genuinely unavailable" + ); + let count = store.puts.load(Ordering::Acquire); + let reference = archive + .extend_tail(&cluster.domain, &tails, &prefix, &group, &latest) + .await?; + assert_eq!( + store.puts.load(Ordering::Acquire), + count + 2, + "only the new entry and complete index require PUT" + ); + let (_, index) = tail_index(store.as_ref(), &reference).await?; + assert_eq!(index["entries"].as_array().unwrap().len(), 2); + assert_eq!(index["entries"][0], prefix_index["entries"][0]); + assert_eq!(index["checkpoint"], prefix_index["checkpoint"]); + // The flattened extension names the original checkpoint and all immutable + // entry objects directly; the old tail index is not a recovery dependency. + store.inner.delete(&prefix_path).await?; + let applier = destination(&cluster).await?; + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + for voter in &mut cluster.voters[..2] { + drop(voter.owner.take()); + voter.log.close().await?; + } + let recovered = archive + .verify_tail( + &cluster.domain, + &tails, + &reference, + receiver(&pool, &applier, &cluster)?, + ) + .await?; + assert_eq!(recovered.manifest().marker(), latest.marker()); + check_images(&cluster, recovered.manifest(), &expected).await?; + drop(recovered); + assert_eq!(tails.usage().retained_bytes, 0); + cluster.close().await?; + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_archive_verifier.rs b/crates/cellar/tests/support/cell_group_archive_verifier.rs new file mode 100644 index 000000000..762d9e7ed --- /dev/null +++ b/crates/cellar/tests/support/cell_group_archive_verifier.rs @@ -0,0 +1,93 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Private replay inventory: serving image pins cannot satisfy terminal proof. +use anyhow::Result; +use cellar::application::consensus::verification::cell_groups::{ + CellGroupApplyImages, CellGroupArchivePool, CellGroupArchiveTailLimits, + CellGroupArchiveTailPool, CellGroupCheckpointLimits, CellGroupCheckpointPool, +}; +use cellar::sdk::{ + consensus::{CellGroupNodeLimits, FleetGroupArchiveReplayOptions, FleetGroupArchiveVerifier}, + ExecutionDomain, +}; +use cellar_ltx::{ + generation::{GenerationStore, ReadServices}, + internal::{CacheLimits, NodeExtentStore, EXTENT_BYTES}, + stream_snapshot::{ScratchLimits, ScratchPool}, + DirectFileSystem, +}; +use rand::RngCore as _; +use std::{path::Path, sync::Arc}; + +pub async fn open( + domain: &ExecutionDomain, + root: &Path, + rooted: bool, + tails: CellGroupArchiveTailLimits, + limits: CellGroupNodeLimits, + checkpoints: CellGroupCheckpointLimits, +) -> Result { + let directory = root.to_owned(); + let base = directory.clone(); + let images = domain + .blocking(move || { + let filesystem = Arc::new(DirectFileSystem); + let services = ReadServices { + now_unix_millis: Arc::new(cellar::asyncrt::wall_ms), + fill_random: Arc::new(|bytes| { + cellar::asyncrt::rng("managed-recovery-verifier").fill_bytes(bytes) + }), + }; + if rooted { + Ok::<_, anyhow::Error>(CellGroupApplyImages::Extents { + images: NodeExtentStore::open( + base.join("extents"), + filesystem, + Default::default(), + )?, + services, + limits: Default::default(), + cache_directory: base.join("cache"), + cache_pool: ScratchPool::new(ScratchLimits { + max_bytes: 32 * 1024 * 1024, + max_files: tails.max_operations, + })?, + cache: CacheLimits { + max_cached_bytes: EXTENT_BYTES, + max_cached_extents: 1, + }, + }) + } else { + Ok(CellGroupApplyImages::Materialized { + images: GenerationStore::open( + base.join("images"), + filesystem, + Default::default(), + )?, + services, + }) + } + })? + .await??; + Ok(FleetGroupArchiveVerifier::new( + CellGroupArchivePool::new(Default::default())?, + CellGroupArchiveTailPool::new(tails)?, + CellGroupCheckpointPool::new(checkpoints)?, + ScratchPool::new(ScratchLimits { + max_bytes: 32 * 1024 * 1024, + max_files: tails.max_operations, + })?, + FleetGroupArchiveReplayOptions { + directory, + filesystem: Arc::new(DirectFileSystem), + materialized: (!rooted).then(|| images.clone()), + extents: rooted.then_some(images), + manifest: limits.manifest, + entries: limits.entries, + max_pages: limits.manifest_pages, + max_page_bytes: limits.manifest_bytes, + max_retained_roots: limits.retained_roots, + max_effect_rows: limits.max_effect_rows, + }, + )?) +} diff --git a/crates/cellar/tests/support/cell_group_backend_control.rs b/crates/cellar/tests/support/cell_group_backend_control.rs new file mode 100644 index 000000000..72823a45b --- /dev/null +++ b/crates/cellar/tests/support/cell_group_backend_control.rs @@ -0,0 +1,131 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! An explicit test-only handshake with the runner that owns the RustFS container. +use super::*; +use serde::{Deserialize, Serialize}; +use std::{ + io::{Read, Write}, + path::PathBuf, +}; + +#[derive(Serialize)] +struct Request { + format: u8, + sequence: u8, + mode: &'static str, + action: &'static str, +} + +#[derive(Deserialize)] +#[serde(deny_unknown_fields)] +struct Response { + format: u8, + sequence: u8, + mode: String, + action: String, + running: bool, +} + +pub(super) struct Control { + store: Arc, + directory: Option, + rooted: bool, +} + +impl Control { + pub(super) fn new( + store: Arc, + directory: Option, + rooted: bool, + ) -> Self { + Self { + store, + directory, + rooted, + } + } + + pub(super) async fn set_running(&self, domain: &ExecutionDomain, running: bool) -> Result<()> { + let Some(directory) = self.directory.clone() else { + self.store.outage.store(!running, Ordering::Release); + return Ok(()); + }; + let sequence = u8::from(self.rooted) * 2 + u8::from(running); + let mode = if self.rooted { + "extents" + } else { + "materialized" + }; + let action = if running { "start" } else { "stop" }; + let encoded = serde_json::to_vec(&Request { + format: 1, + sequence, + mode, + action, + })?; + let response = directory.join(format!("{sequence:02}-response.json")); + domain + .blocking(move || -> Result<()> { + anyhow::ensure!( + directory.is_absolute() && directory.is_dir(), + "explicit control directory" + ); + let pending = directory.join(format!("{sequence:02}-request.pending")); + let request = directory.join(format!("{sequence:02}-request.json")); + let mut file = std::fs::OpenOptions::new() + .create_new(true) + .write(true) + .open(&pending)?; + file.write_all(&encoded)?; + file.sync_all()?; + // Atomic create, never replacement of another operation's request. + std::fs::hard_link(&pending, request)?; + std::fs::remove_file(pending)?; + std::fs::File::open(directory)?.sync_all()?; + Ok(()) + })? + .await??; + let deadline = domain + .run(async { cellar::asyncrt::mono_ms().saturating_add(30_000) }) + .await?; + loop { + let path = response.clone(); + let reply = domain + .blocking(move || -> Result> { + let file = match std::fs::File::open(path) { + Ok(file) => file, + Err(error) if error.kind() == std::io::ErrorKind::NotFound => { + return Ok(None) + } + Err(error) => return Err(error.into()), + }; + let mut bytes = Vec::new(); + file.take(4097).read_to_end(&mut bytes)?; + anyhow::ensure!(bytes.len() <= 4096, "bounded control reply"); + Ok(Some(serde_json::from_slice(&bytes)?)) + })? + .await??; + if let Some(reply) = reply { + anyhow::ensure!( + reply.format == 1 + && reply.sequence == sequence + && reply.mode == mode + && reply.action == action + && reply.running == running, + "exact backend transition reply" + ); + return Ok(()); + } + domain + .run(async { + anyhow::ensure!( + cellar::asyncrt::mono_ms() < deadline, + "backend control deadline" + ); + cellar::asyncrt::sleep(Duration::from_millis(5)).await; + Ok::<_, anyhow::Error>(()) + }) + .await??; + } + } +} diff --git a/crates/cellar/tests/support/cell_group_background_admission.rs b/crates/cellar/tests/support/cell_group_background_admission.rs new file mode 100644 index 000000000..4f90f1bd8 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_background_admission.rs @@ -0,0 +1,211 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Background apply and an application command share one physical SQL worker. +use super::*; + +pub(crate) async fn run(expire: bool) -> Result<()> { + let calls = Arc::new(AtomicUsize::new(0)); + let counted = calls.clone(); + let class = Arc::new( + RaftClass::new("Counter", 1, [81; 32])? + .command("Add", [82; 32], move |ctx, _| { + counted.fetch_add(1, Ordering::SeqCst); + ctx.sql().execute("UPDATE value SET n=n+7", [])?; + Ok(7_i64.to_be_bytes().to_vec()) + })? + .query("Get", [83; 32], |ctx, _| { + let value: i64 = ctx.query_row("SELECT n FROM value", [], |row| row.get(0))?; + Ok(value.to_be_bytes().to_vec()) + })?, + ); + let (mut fixture, route, cell, _) = Box::pin(registered::setup(false, &class)).await?; + let faults = physical::Faults::new( + fixture.cluster._root.path().join("extra-31-node-0/pages"), + None, + ); + let background = fixture + .add_group_with_filesystem(31, |_| { + Arc::new(physical::FaultFileSystem { + faults: faults.clone(), + }) + }) + .await?; + controller::attach_group_appliers(&mut fixture, route.history, background).await?; + let controllers = controller::start_with_limits( + &mut fixture, + CellGroupControllerLimits { + max_commands: 1, + ..controller::limits() + }, + ) + .await?; + let owner = &controllers[0]; + fixture.runtimes[0].campaign(background).await?; + controller::wait_for(async || Ok(fixture.runtimes[0].status(background).await?.ready)).await?; + let bg_route = fixture.cluster.voters[0] + .log + .cell_group(&background)? + .group_manifest()? + .context("background route")? + .route() + .clone(); + + let (entered, started) = tokio::sync::oneshot::channel(); + let (release, hold) = std::sync::mpsc::sync_channel(1); + *faults.before_directory_sync.lock().unwrap() = Some(Box::new(move || { + let _ = entered.send(()); + let _ = hold.recv_timeout(Duration::from_secs(15)); + })); + let image = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(42);")?; + let bg_cell = history(&bg_route, "background-apply")?; + let accepted = fixture.runtimes[0] + .propose( + background, + apply::enrollment_bytes(&bg_route, &bg_cell, &image, false)?, + ) + .await?; + tokio::time::timeout(Duration::from_secs(5), started) + .await + .context("background apply entered physical page sync")??; + let pool = fixture.appliers[&(0, route.history)].0.clone(); + let held = pool.usage()?; + assert_eq!(held.limits.workers, 1); + assert_eq!(held.worker_capacity, 1); + assert_eq!(held.apply_retained_operations, 1); + assert_eq!(held.queued_or_running, 0); + assert_eq!(owner.usage().retained_commands, 0); + + let dispatcher = CellGroupDispatcher::new( + &fixture.cluster.domain, + fixture.cluster.directory.clone(), + class.clone(), + remote_queries::keys(89)?, + CellDispatchLimits { + max_cached_routes: 1, + route_cache_bytes: 8192, + max_invocations: 4, + retained_bytes: 128 * 1024, + max_input_bytes: 1024, + }, + )?; + let retry_until_ms = fixture + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms().max(0) as u64 + 60_000 }) + .await?; + let request = CellCommandRequest { + id: CommandId::new(ClientId::new([188; 16])?, 1)?, + contract: class.contract("Add").context("command contract")?, + input: vec![], + durability: Durability::Quorum, + retry_until_ms, + }; + let options = fixture.cell_options(&cell).await?; + let routing = |timeout| CellRoutingLimits { + timeout, + max_attempts: 1, + }; + let call = owner.dispatch_command( + &dispatcher, + cell.clone(), + request.clone(), + routing(Duration::from_secs(if expire { 1 } else { 10 })), + options.clone(), + )?; + controller::wait_for(async || { + Ok(pool.usage()?.queued_or_running == 1 && owner.usage().retained_commands == 1) + }) + .await?; + assert_eq!(calls.load(Ordering::Acquire), 0); + let mut overflow = request.clone(); + overflow.id = CommandId::new(ClientId::new([188; 16])?, 2)?; + let refused = owner + .dispatch_command( + &dispatcher, + cell.clone(), + overflow, + routing(Duration::from_secs(10)), + options.clone(), + ) + .err() + .context("second pending command exceeded admission")?; + assert!(!refused.ambiguous); + assert!(matches!( + refused.cause.as_ref(), + CellDispatchError::Runtime(RuntimeError::Overloaded) + )); + let pending = if expire { + let error = call + .response() + .await + .err() + .context("queued command deadline")?; + assert!(matches!( + error.cause.as_ref(), + CellDispatchError::Runtime(RuntimeError::DeadlineExceeded) + )); + assert_eq!(pool.usage()?.queued_or_running, 1); + None + } else { + Some(call) + }; + release.send(())?; + controller::applied(&fixture, background, accepted.position.index).await?; + let committed = match pending { + Some(call) => call.response().await?, + None => { + controller::wait_for(async || { + Ok(pool.usage()?.queued_or_running == 0 && owner.usage().retained_commands == 0) + }) + .await?; + assert_eq!(calls.load(Ordering::Acquire), 0); + let state = fixture.runtimes[0] + .reconciliation_authority(route.history, cell.clone()) + .await?; + assert_eq!(state.cell().version.sequence.get(), 1); + owner + .dispatch_command( + &dispatcher, + cell.clone(), + request, + routing(Duration::from_secs(10)), + options, + )? + .response() + .await? + } + }; + assert_eq!(calls.load(Ordering::Acquire), 1); + assert_eq!(committed.durability(), Durability::Quorum); + let receipt = committed.local_receipt().context("committed SQL receipt")?; + assert_eq!(receipt.receipt.result(), 7_i64.to_be_bytes()); + controller::applied(&fixture, route.history, receipt.position.index).await?; + let selected = fixture.runtimes[1] + .read_authority( + route.history, + cell.clone(), + CellReadConsistency::AtLeast(receipt.version.clone()), + reads::deadline(&fixture, 5000).await?, + ) + .await?; + let read = fixture.appliers[&(1, route.history)] + .0 + .query_registered( + fixture.cell_options_at(&cell, 1).await?, + selected, + class.clone(), + class.contract("Get").context("query contract")?, + vec![], + ) + .await?; + assert_eq!(read.result(), 7_i64.to_be_bytes()); + assert_eq!(read.observation(), &receipt.observation()); + drop(committed); + drop(read); + controller::wait_for(async || Ok(owner.usage().retained_commands == 0)).await?; + assert_eq!(pool.usage()?.active_cells, 0); + for controller in controllers { + controller.close().await?; + } + fixture.close().await +} diff --git a/crates/cellar/tests/support/cell_group_batch_crash.rs b/crates/cellar/tests/support/cell_group_batch_crash.rs new file mode 100644 index 000000000..1d70db032 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_batch_crash.rs @@ -0,0 +1,190 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Crash the shared physical journal while it publishes three group roots. + +use super::*; +use anyhow::{bail, Context}; +use cellar::application::consensus::verification::{ + CellGroupBootstrap, GroupWrite, LogOpenMode, ReplicaLog, +}; +use cellar_logic::consensus::OperationId; +use cellar_ltx::FileSystem; +use raft::{GetEntriesContext, Storage}; +use std::path::Path; + +const GROUPS: [u8; 3] = [10, 11, 12]; +const BASE: u64 = 2; +const ADVANCED: u64 = 3; + +fn group_route(id: u8) -> GroupRoute { + let mut route = route(); + route.history.group = GroupId::new([id; 16]).unwrap(); + route.history.incarnation = IncarnationId::new([id + 10; 16]).unwrap(); + route.birth_digest = [id + 20; 32]; + route +} + +fn payload(id: u8, index: u64) -> Vec { + vec![id; index as usize * 31] +} + +async fn open( + root: &Path, + mode: LogOpenMode, + process: u8, + filesystem: Arc, +) -> Result<(ExecutionDomain, ReplicaLog)> { + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let log = domain + .run(ReplicaLog::open( + engine_options(&root.join("node"), mode, process), + filesystem, + )) + .await??; + Ok((domain, log)) +} + +fn prepare(root: &Path, log: &ReplicaLog, index: u64) -> Result>> { + let mut writes = Vec::new(); + for id in GROUPS { + let route = group_route(id); + let group = log.cell_group(&route.history)?; + let previous = group.group_manifest()?.context("previous group manifest")?; + assert_eq!(previous.route(), &route); + assert_eq!(previous.position(), position(index - 1)); + let mut pages = ManifestPageStore::open( + &root.join(format!("pages-{id}")), + Arc::new(DirectFileSystem), + 32, + 1024 * 1024, + 16, + )?; + let tree = ManifestTree::new(route, ManifestLimits::default())?; + // Restore the selected root's retention ownership in this process + // before preparing its successor, as the runtime does on attachment. + let recovered = tree.prepare_complete( + &group, + &mut pages, + previous.root(), + previous.position(), + previous.membership_revision(), + |_| unreachable!("empty storage fixture"), + )?; + assert_eq!(recovered.marker(), &previous); + let prepared = tree.prepare_complete( + &group, + &mut pages, + previous.root(), + position(index), + previous.membership_revision(), + |_| unreachable!("empty storage fixture"), + )?; + let mut write = root_write(&group, prepared)?; + write.entries[0].data = payload(id, index).into(); + writes.push(write); + } + Ok(writes) +} + +async fn persist( + log: &ReplicaLog, + index: u64, + writes: Vec>, +) -> Result<()> { + let expected: Vec<_> = writes + .iter() + .map(|write| write.group.identity().clone()) + .collect(); + let operation = OperationId::new(index); + let receipt = log.persist_cells(operation, writes).await?; + assert_eq!(receipt.operation, operation); + assert_eq!(receipt.groups, expected); + assert!(receipt.sequence > 0); + Ok(()) +} + +pub(super) async fn run() -> Result<()> { + for cut in ["before-sync", "after-sync", "acknowledged"] { + let temp = tempfile::tempdir()?; + let root = temp.path(); + let (domain, log) = open(root, LogOpenMode::Create, 1, Arc::new(DirectFileSystem)).await?; + let local = MemberId::new(1)?; + for id in GROUPS { + log.create_cell_group(CellGroupBootstrap { + route: group_route(id), + member: local, + voters: vec![local], + learners: vec![], + }) + .await?; + } + persist(&log, BASE, prepare(root, &log, BASE)?).await?; + log.close().await?; + domain.shutdown().await; + + crash_fixture::kill_at(root, "batch_crash_child", false, cut).await?; + let (domain, log) = open(root, LogOpenMode::Resume, 3, Arc::new(DirectFileSystem)).await?; + assert_eq!(log.cell_groups()?.len(), GROUPS.len()); + let first = log.cell_group(&group_route(GROUPS[0]).history)?; + let committed = first.initial_state()?.hard_state.commit; + if cut == "before-sync" { + assert!([BASE, ADVANCED].contains(&committed)); + } else { + assert_eq!(committed, ADVANCED); + } + for id in GROUPS { + let route = group_route(id); + let group = log.cell_group(&route.history)?; + assert_eq!( + group.initial_state()?.hard_state.commit, + committed, + "partial batch at {cut}" + ); + assert_eq!(group.last_index()?, committed); + let marker = group.group_manifest()?.context("reopened group manifest")?; + assert_eq!(marker.route(), &route); + assert_eq!( + marker.position(), + position(committed), + "partial applied batch at {cut}" + ); + assert_eq!(marker.membership_revision(), 1); + assert_eq!(marker.root().cells(), 0); + let entries = + group.entries(BASE, committed + 1, None, GetEntriesContext::empty(false))?; + assert_eq!(entries.len() as u64, committed - BASE + 1); + for (index, entry) in (BASE..=committed).zip(entries) { + assert_eq!((entry.index, entry.term), (index, 1)); + assert_eq!(&entry.data[..], &payload(id, index)); + } + } + log.close().await?; + domain.shutdown().await; + } + Ok(()) +} + +pub(super) async fn child() -> Result<()> { + let (root, _, cut) = crash_fixture::parameters()?; + let gate = engine_fault::Gate::new(Some(root.join("signal"))); + let (_domain, log) = open( + &root, + LogOpenMode::Resume, + 2, + Arc::new(engine_fault::FaultFileSystem { gate: gate.clone() }), + ) + .await?; + let writes = prepare(&root, &log, ADVANCED)?; + let gate_mode = match cut.as_str() { + "before-sync" => engine_fault::CRASH_BEFORE_SYNC, + "after-sync" => engine_fault::CRASH_AFTER_SYNC, + "acknowledged" => { + persist(&log, ADVANCED, writes).await?; + return crash_fixture::stop_at(&root).await; + } + _ => bail!("unknown batch crash cut"), + }; + let _release = gate.arm(gate_mode); + persist(&log, ADVANCED, writes).await?; + bail!("batch persistence returned without entering its sync gate") +} diff --git a/crates/cellar/tests/support/cell_group_checkpoint.rs b/crates/cellar/tests/support/cell_group_checkpoint.rs new file mode 100644 index 000000000..ca9633e50 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_checkpoint.rs @@ -0,0 +1,586 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Source-side checkpoint qualification over the existing real three-voter +//! fixture. This does not qualify a receiver, learner promotion or snapshot RPC. + +use super::*; +use cellar_ltx::internal::{ + ExtentId, ExtentManifest, ExtentRootReference, Geometry, ImageBinding, NodeExtentStore, + EXTENT_BYTES, +}; +use std::collections::BTreeMap; + +pub(super) fn limits() -> CellGroupCheckpointLimits { + CellGroupCheckpointLimits { + max_sources: 1, + max_retained_bytes: 32 * 1024 * 1024, + max_cells: 64, + max_transfer_bytes: 32 * 1024 * 1024, + max_chunk_bytes: 4093, + } +} + +async fn populated(rooted: bool) -> Result<(Cluster, BTreeMap>)> { + populated_with_lagging(rooted, false).await +} + +pub(super) async fn populated_with_lagging( + rooted: bool, + lagging: bool, +) -> Result<(Cluster, BTreeMap>)> { + populate(rooted, lagging, false).await +} + +pub(super) async fn populated_partition( + rooted: bool, +) -> Result<(Cluster, BTreeMap>)> { + populate(rooted, false, true).await +} + +async fn populate( + rooted: bool, + lagging: bool, + partition: bool, +) -> Result<(Cluster, BTreeMap>)> { + populate_images(rooted, lagging, partition, 0, 4_202_496).await +} + +pub(super) async fn populate_images( + rooted: bool, + lagging: bool, + partition: bool, + additional: usize, + padding: usize, +) -> Result<(Cluster, BTreeMap>)> { + let mut cluster = Cluster::new().await?; + cluster.elect().await?; + cluster.voters[2].paused = lagging; + let expected = enroll_images(&mut cluster, rooted, partition, additional, padding).await?; + Ok((cluster, expected)) +} + +/// Enroll into an existing elected group so archive tests can capture its empty +/// checkpoint first. Existing source/snapshot fixtures call this same path. +pub(super) async fn enroll_images( + cluster: &mut Cluster, + rooted: bool, + partition: bool, + additional: usize, + padding: usize, +) -> Result>> { + let route = cluster.birth.route().clone(); + if rooted { + for n in 0..3 { + cluster.voters[n].extents = Some(NodeExtentStore::open( + cluster._root.path().join(format!("extents-{n}")), + Arc::new(DirectFileSystem), + Default::default(), + )?); + } + } + let mut expected = BTreeMap::new(); + let names = ["alpha", "beta", "empty"] + .into_iter() + .map(str::to_owned) + .chain((0..additional).map(|n| format!("cell-{n:03}"))); + for (ordinal, name) in names.enumerate() { + let cell = history(&route, &name)?; + let bytes = if name == "empty" { + Vec::new() + } else { + // Keep the original two-image fixture unchanged. Pressure cohorts + // use unique contents so other cells cannot accidentally retain an + // old reader's extent after that cell advances. + let value = if additional == 0 { 1 } else { ordinal + 1 }; + images::image(None, &format!("CREATE TABLE value(n); INSERT INTO value VALUES({value}); CREATE TABLE padding(bytes); INSERT INTO padding VALUES(zeroblob({padding}));"))? + }; + let encoded = if bytes.is_empty() { + CellGroupEntry { + binding: GroupEntryBinding { + group: route.history, + route_epoch: route.epoch, + birth_digest: route.birth_digest, + schema_digest: route.schema_digest, + proposal: ProposalId::new([99; 16])?, + }, + command: GroupCommand::Enroll { + cell: cell.clone(), + image: LtxImage { + epoch: 1, + txid: 0, + checksum: 0, + commitment: ImageCommitment::Sha256(Sha256::digest([]).into()), + }, + files: vec![], + }, + } + .encode(&route, GroupEntryLimits::default())? + } else { + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let encoded = request(&route, cell.clone(), 1, &cut)?; + let mut entry = CellGroupEntry::decode(&encoded, &route, GroupEntryLimits::default())?; + if rooted { + let commitment = extent_commitment(&route, &cell, cut.result, &bytes)?; + let GroupCommand::Enroll { image, .. } = &mut entry.command else { + unreachable!() + }; + image.commitment = commitment; + } + entry.encode(&route, GroupEntryLimits::default())? + }; + cluster.voters[0].owner.as_mut().unwrap().propose(encoded)?; + cluster + .drive_partition(false, partition.then_some(2)) + .await?; + expected.insert(name.to_owned(), bytes); + } + Ok(expected) +} + +pub(super) fn extent_commitment( + route: &GroupRoute, + cell: &CellHistory, + state: ImageState, + bytes: &[u8], +) -> Result { + let manifest = ExtentManifest::new( + ImageBinding { + history: cell_image_history_digest(cell), + schema: route.schema_digest, + epoch: state.epoch, + txid: state.txid.0, + checksum: state.checksum, + }, + Geometry::new(state.page_size, state.length)?, + &bytes + .chunks(EXTENT_BYTES as usize) + .map(ExtentId::of) + .collect::, _>>()?, + )?; + Ok(ImageCommitment::Extents(ExtentRootReference::new( + manifest.root(), + manifest.encode()?.len() as u32, + )?)) +} + +async fn advance(cluster: &mut Cluster, name: &str, before: &[u8], rooted: bool) -> Result<()> { + advance_partition(cluster, name, before, rooted, None).await +} + +pub(super) async fn advance_partition( + cluster: &mut Cluster, + name: &str, + before: &[u8], + rooted: bool, + excluded: Option, +) -> Result<()> { + let route = cluster.birth.route().clone(); + let cell = history(&route, name)?; + let after = images::image(Some(before), "UPDATE value SET n=n+1")?; + let cut = images::Cut::new(before, &after, 1, true, false); + let encoded = request(&route, cell.clone(), 2, &cut)?; + let mut entry = CellGroupEntry::decode(&encoded, &route, GroupEntryLimits::default())?; + if rooted { + let GroupCommand::Mutation(mutation) = &mut entry.command else { + unreachable!() + }; + mutation.base.commitment = extent_commitment(&route, &cell, cut.base, before)?; + mutation.result.commitment = extent_commitment(&route, &cell, cut.result, &after)?; + } + cluster.voters[0] + .owner + .as_mut() + .unwrap() + .propose(entry.encode(&route, GroupEntryLimits::default())?)?; + cluster.drive_partition(false, excluded).await?; + Ok(()) +} + +#[derive(Default)] +pub(super) struct Received { + header: Vec, + pages: BTreeMap<[u8; 32], Vec>, + images: BTreeMap>, + manifests: BTreeMap>, + extents: BTreeMap<(String, u32), (ExtentId, Vec)>, + largest_chunk: usize, +} + +impl Received { + pub(super) fn append( + &mut self, + chunk: CellGroupCheckpointChunk<'_>, + ) -> std::result::Result<(), CellGroupCheckpointError> { + self.largest_chunk = self.largest_chunk.max(chunk.bytes.len()); + let output = match chunk.part { + CellGroupCheckpointPart::Header => &mut self.header, + CellGroupCheckpointPart::Page(key) => self.pages.entry(key).or_default(), + CellGroupCheckpointPart::Materialized(cell) => { + self.images.entry(cell.as_str().to_owned()).or_default() + } + CellGroupCheckpointPart::ExtentManifest(cell) => { + self.manifests.entry(cell.as_str().to_owned()).or_default() + } + CellGroupCheckpointPart::Extent { cell, ordinal, id } => { + let output = self + .extents + .entry((cell.as_str().to_owned(), ordinal)) + .or_insert_with(|| (id, vec![])); + assert_eq!(output.0, id); + &mut output.1 + } + }; + assert_eq!(chunk.offset, output.len() as u64); + assert!(chunk.offset + chunk.bytes.len() as u64 <= chunk.length); + output.extend_from_slice(chunk.bytes); + Ok(()) + } + + pub(super) fn verify( + mut self, + marker: &GroupManifest, + leaves: &BTreeMap, + expected: &BTreeMap>, + rooted: bool, + ) -> Result<()> { + assert!(self.largest_chunk <= limits().max_chunk_bytes); + let header: serde_json::Value = serde_json::from_slice(&self.header)?; + assert_eq!(header["domain"], "cellar-cell-group-checkpoint-1"); + assert_eq!(header["transition"], "active"); + assert_eq!( + header["membership"]["revision"], + marker.membership_revision() + ); + assert_eq!(header["root"], serde_json::to_value(marker)?); + let conf = header["conf"] + .as_array() + .context("configuration")? + .iter() + .map(|n| n.as_u64().unwrap() as u8) + .collect::>(); + let conf: raft::prelude::ConfState = protobuf::Message::parse_from_bytes(&conf)?; + assert_eq!(conf.voters, vec![1, 2, 3]); + assert_eq!(self.pages.len(), expected.len()); + assert!(self.pages.contains_key(&marker.root().digest().unwrap())); + for (digest, bytes) in &self.pages { + let mut hash = Sha256::new(); + hash.update(b"cellar-cell-group-manifest-page-1\0"); + hash.update(bytes); + assert_eq!(*digest, <[u8; 32]>::from(hash.finalize())); + } + for (name, wanted) in expected { + let leaf = &leaves[name]; + assert_eq!(leaf.version.sequence.get(), 1); + if wanted.is_empty() { + assert_eq!(leaf.version.image.txid, 0); + assert!(!self.images.contains_key(name)); + assert!(!self.manifests.contains_key(name)); + continue; + } + if rooted { + let CellImageCommitment::Extents { + root, + manifest_bytes, + } = leaf.version.image.commitment + else { + unreachable!() + }; + let bytes = self.manifests.remove(name).context("manifest")?; + assert_eq!(bytes.len(), manifest_bytes as usize); + let manifest = ExtentManifest::decode( + &bytes, + cellar_ltx::internal::ImageRoot::from_bytes(root), + )?; + let mut assembled = Vec::new(); + for n in 0..manifest.geometry().extent_count() { + let (id, bytes) = self.extents.remove(&(name.clone(), n)).context("extent")?; + assert_eq!(Some(id), manifest.extent(n)); + id.verify(&bytes)?; + assembled.extend(bytes); + } + assert_eq!(&assembled, wanted); + } else { + assert_eq!(self.images.remove(name).as_ref(), Some(wanted)); + } + } + assert!(self.images.is_empty() && self.extents.is_empty() && self.manifests.is_empty()); + Ok(()) + } +} + +pub(super) async fn coherent() -> Result<()> { + for rooted in [false, true] { + let (mut cluster, expected) = populated(rooted).await?; + let group = cluster.voters[0] + .log + .cell_group(&cluster.birth.route().history)?; + let root = cluster.voters[0].owner.as_ref().unwrap().manifest().clone(); + let old_page = cluster._root.path().join("node-1/pages").join(format!( + "{}.page", + hex::encode(root.marker().root().digest().unwrap()) + )); + let pool = CellGroupCheckpointPool::new(limits())?; + let mut source = pool.capture(&group, &root)?; + let marker = root.marker().clone(); + let leaves = expected + .keys() + .map(|name| { + let cell = CellId::new(name.clone()).unwrap(); + (name.clone(), root.pin_cell(&cell).unwrap().0.clone()) + }) + .collect::>(); + drop(root); // Only the stream retains the old physical closure after apply. + let (entered, entered_rx) = tokio::sync::oneshot::channel(); + let (release, release_rx) = std::sync::mpsc::channel(); + let worker = cluster.domain.blocking(move || -> Result<_> { + let mut entered = Some(entered); + let mut received = Received::default(); + source.stream(|chunk| { + if matches!(chunk.part, CellGroupCheckpointPart::Page(_)) { + if let Some(entered) = entered.take() { + let _ = entered.send(()); + release_rx.recv().map_err(|_| { + CellGroupCheckpointError::Sink(std::io::Error::other("released")) + })?; + } + } + received.append(chunk) + })?; + Ok((source, received)) + })?; + tokio::time::timeout(Duration::from_secs(10), entered_rx).await??; + // Real quorum publication continues while the transfer owner is held. + advance(&mut cluster, "alpha", &expected["alpha"], rooted).await?; + advance(&mut cluster, "beta", &expected["beta"], rooted).await?; + for voter in &mut cluster.voters { + let current = voter.owner.as_ref().unwrap().manifest(); + assert!(current.marker().position().index > marker.position().index); + assert_eq!( + current + .pin_cell(&CellId::new("alpha")?) + .unwrap() + .0 + .version + .sequence + .get(), + 2 + ); + voter.pages.as_mut().unwrap().reconcile_retained(&[])?; + voter.pages.as_mut().unwrap().reclaim(1024)?; + voter.images.retain_only(&[])?; + voter.images.reclaim(1024)?; + if let Some(store) = &voter.extents { + store.retain_only(&[])?; + store.reclaim(1024)?; + } + } + assert!(old_page.exists(), "live source must pin its manifest page"); + release.send(())?; + let (source, received) = worker.await??; + received.verify(&marker, &leaves, &expected, rooted)?; + assert_eq!(source.marker(), &marker); + assert_eq!(pool.usage().sources, 1); + drop(source); + assert_eq!(pool.usage().sources, 0); + cluster.voters[0].pages.as_mut().unwrap().reclaim(1024)?; + cluster.voters[0].pages.as_mut().unwrap().reclaim(1024)?; + assert!( + !old_page.exists(), + "released obsolete source page can be reclaimed" + ); + cluster.close().await?; + } + Ok(()) +} + +pub(super) async fn refusals() -> Result<()> { + let mut cluster = Cluster::new().await?; + let group = cluster.voters[0] + .log + .cell_group(&cluster.birth.route().history)?; + let original = cluster.voters[0].owner.as_ref().unwrap().manifest().clone(); + let pool = CellGroupCheckpointPool::new(limits())?; + // A recovered empty genesis is a complete closure, with no cache image pins yet. + let mut empty = pool.capture(&group, &original)?; + let mut parts = 0; + empty.stream(|chunk| { + assert_eq!(chunk.part, CellGroupCheckpointPart::Header); + parts += 1; + Ok(()) + })?; + assert!(parts > 0); + drop(empty); + let foreign = cluster.voters[1].owner.as_ref().unwrap().manifest(); + assert!(matches!( + pool.capture(&group, foreign), + Err(CellGroupCheckpointError::Invalid(_)) + )); + let prospective = cluster.voters[0].tree.prepare_unchanged( + cluster.voters[0].pages.as_ref().unwrap(), + &original, + LogPosition { + index: LogIndex::new(2), + term: Term::new(1), + }, + )?; + assert!(matches!( + pool.capture(&group, &prospective), + Err(CellGroupCheckpointError::Invalid(_)) + )); + assert_eq!(pool.usage().retained_bytes, 0); + cluster.elect().await?; + assert!(matches!( + pool.capture(&group, &original), + Err(CellGroupCheckpointError::Invalid(_)) + )); + let current = cluster.voters[0].owner.as_ref().unwrap().manifest(); + let source = pool.capture(&group, current)?; + let bytes = pool.usage().retained_bytes; + // Two slots, but only enough bytes for one source: exercise byte admission + // independently of the item ceiling, with no refund on a failed acquire. + let small = CellGroupCheckpointPool::new(CellGroupCheckpointLimits { + max_sources: 2, + max_retained_bytes: bytes, + ..limits() + })?; + let first = small.capture(&group, current)?; + assert!(matches!( + small.capture(&group, current), + Err(CellGroupCheckpointError::Busy) + )); + assert_eq!(small.usage().retained_bytes, bytes); + drop(first); + assert_eq!(small.usage().retained_bytes, 0); + drop(source); + drop(prospective); + drop(original); + cluster.close().await?; + Ok(()) +} + +pub(super) async fn ownership() -> Result<()> { + let (cluster, _) = populated(false).await?; + let group = cluster.voters[0] + .log + .cell_group(&cluster.birth.route().history)?; + let root = cluster.voters[0].owner.as_ref().unwrap().manifest(); + let pool = CellGroupCheckpointPool::new(limits())?; + let mut source = pool.capture(&group, root)?; + let usage = pool.usage(); + let mut limited = limits(); + limited.max_cells = 2; + assert!(matches!( + CellGroupCheckpointPool::new(limited)?.capture(&group, root), + Err(CellGroupCheckpointError::Invalid(_)) + )); + limited = limits(); + limited.max_transfer_bytes = 64 * 1024; + assert!(matches!( + CellGroupCheckpointPool::new(limited)?.capture(&group, root), + Err(CellGroupCheckpointError::Limit) + )); + assert!(matches!( + pool.capture(&group, root), + Err(CellGroupCheckpointError::Busy) + )); + assert_eq!(pool.usage(), usage); + let (entered, rx) = tokio::sync::oneshot::channel(); + let (release, release_rx) = std::sync::mpsc::channel(); + let worker = cluster.domain.blocking(move || { + let mut entered = Some(entered); + let result = source.stream(|_| { + if let Some(entered) = entered.take() { + let _ = entered.send(()); + let _ = release_rx.recv(); + } + Err(CellGroupCheckpointError::Sink(std::io::Error::other( + "cancelled sink", + ))) + }); + assert!(matches!(result, Err(CellGroupCheckpointError::Sink(_)))); + assert!(matches!( + source.stream(|_| Ok(())), + Err(CellGroupCheckpointError::Invalid(_)) + )); + })?; + tokio::time::timeout(Duration::from_secs(10), rx).await??; + drop(worker); // Losing the result waiter must not refund the physical source. + assert_eq!(pool.usage(), usage); + assert!(matches!( + pool.capture(&group, root), + Err(CellGroupCheckpointError::Busy) + )); + release.send(())?; + tokio::time::timeout(Duration::from_secs(10), async { + while pool.usage().sources != 0 { + tokio::time::sleep(Duration::from_millis(5)).await; + } + }) + .await?; + assert_eq!(pool.usage().retained_bytes, 0); + cluster.close().await?; + Ok(()) +} + +pub(super) async fn corruption() -> Result<()> { + for rooted in [false, true] { + let (mut cluster, _) = populated(rooted).await?; + let group = cluster.voters[0] + .log + .cell_group(&cluster.birth.route().history)?; + let root = cluster.voters[0].owner.as_ref().unwrap().manifest().clone(); + let pool = CellGroupCheckpointPool::new(limits())?; + let mut source = pool.capture(&group, &root)?; + let (_, pin) = root.pin_cell(&CellId::new("alpha")?).unwrap(); + let path = match pin { + CellImagePin::Materialized(pin) => cluster + ._root + .path() + .join("node-1/images") + .join(format!("{}.sqlite", hex::encode(pin.id().bytes()))), + CellImagePin::Extents(pin) => { + let id = pin.manifest().extent(0).unwrap(); + cluster._root.path().join("extents-0").join(format!( + "e-{:08x}-{}", + id.length(), + hex::encode(id.bytes()) + )) + } + CellImagePin::Empty => unreachable!(), + }; + let mut bytes = std::fs::read(&path)?; + bytes[100] ^= 0x80; + std::fs::write(path, bytes)?; + let result = cluster + .domain + .blocking(move || { + let result = source.stream(|_| Ok(())); + (source, result) + })? + .await?; + assert!(matches!( + result.1, + Err(CellGroupCheckpointError::Invalid( + "materialized image digest" + )) | Err(CellGroupCheckpointError::Extent(_)) + )); + assert_eq!(pool.usage().sources, 1); + drop(result); + let mut closed = pool.capture(&group, &root)?; + drop(cluster.voters[0].owner.take()); + cluster.voters[0].log.close().await?; + let mut emissions = 0; + assert!(matches!( + closed.stream(|_| { + emissions += 1; + Ok(()) + }), + Err(CellGroupCheckpointError::Log(_)) + )); + assert_eq!(emissions, 0); + drop(closed); + drop(root); + cluster.close().await?; + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_checkpoint_fairness.rs b/crates/cellar/tests/support/cell_group_checkpoint_fairness.rs new file mode 100644 index 000000000..b3b768b76 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_checkpoint_fairness.rs @@ -0,0 +1,165 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Concurrent real enrollments compete for one shared checkpoint slot. Inspect +//! each group's durable progress at fixed cuts before either writer finishes. +use super::*; + +struct Trace { + cut: tokio::sync::Barrier, + resume: tokio::sync::Barrier, +} + +async fn write( + runtime: CellGroupRuntime, + route: GroupRoute, + rooted: bool, + trace: Arc, + controller: CellGroupController, +) -> Result { + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(83);")?; + let mut last = None; + for round in 0..32 { + let cell = history(&route, &format!("concurrent-checkpoint-{round}"))?; + let accepted = runtime + .propose( + route.history, + apply::enrollment_bytes(&route, &cell, &bytes, rooted)?, + ) + .await?; + tokio::time::timeout(Duration::from_secs(15), async { + while runtime.status(route.history).await?.applied.index < accepted.position.index { + tokio::time::sleep(Duration::from_millis(2)).await; + } + Ok::<_, anyhow::Error>(()) + }) + .await??; + assert!(controller.usage().active_checkpoints <= 1); + assert_eq!(controller.usage().quarantined_groups, 0); + if round == 8 || round == 24 { + trace.cut.wait().await; + trace.resume.wait().await; + } + last = Some(accepted); + } + last.context("nonempty write trace") +} + +pub(crate) async fn concurrent_checkpoints(rooted: bool) -> Result<()> { + let class = RaftClass::new("Counter", 1, [81; 32])?; + let (mut f, route, _, _) = Box::pin(registered::setup(rooted, &class)).await?; + let second = controller::add_second(&mut f, route.history).await?; + f.connect_tcp().await?; + let pools: Vec<_> = (0..3) + .map(|_| CellGroupCheckpointPool::new(checkpoint_fixture::limits())) + .collect::>()?; + for (transport, pool) in f.transports.iter().zip(&pools) { + transport.enable_group_checkpoints(pool.clone())?; + } + let held = pools[0].receive( + f.appliers[&(0, route.history)].1.clone(), + checkpoint_receive::scratch()?, + f.cluster._root.path().into(), + )?; + let charged = pools[0].usage(); + let mut bounds = controller::limits(); + bounds.checkpoints = Some(CellGroupCheckpointPolicy { + interval_entries: 2, + max_parallel: 1, + }); + let controllers = controller::start_with_limits(&mut f, bounds).await?; + f.runtimes[0].campaign(second).await?; + controller::wait_for(async || Ok(f.runtimes[0].status(second).await?.ready)).await?; + let second_route = f.cluster.voters[0] + .log + .cell_group(&second)? + .group_manifest()? + .context("second route")? + .route() + .clone(); + let trace = Arc::new(Trace { + cut: tokio::sync::Barrier::new(3), + resume: tokio::sync::Barrier::new(3), + }); + let mut writers = Vec::new(); + for (route, rooted) in [(route.clone(), rooted), (second_route, false)] { + let runtime = f.runtimes[0].clone(); + let control = controllers[0].clone(); + let trace = trace.clone(); + let group = route.history; + writers.push(( + group, + f.cluster + .domain + .spawn(async move { write(runtime, route, rooted, trace, control).await })?, + )); + } + tokio::time::timeout(Duration::from_secs(15), trace.cut.wait()).await?; + assert_eq!(pools[0].usage(), charged); + assert!(controllers[0].usage().checkpoint_retries > 0); + assert_eq!(controllers[0].usage().checkpoints_published, 0); + let mut release_cuts = BTreeMap::new(); + for group in [route.history, second] { + let log = f.cluster.voters[0].log.cell_group(&group)?; + assert!(log.checkpoint_manifest()?.is_none()); + release_cuts.insert( + group, + log.group_manifest()?.context("held-slot cut")?.position(), + ); + } + drop(held); + trace.resume.wait().await; + // Each writer has sixteen more committed/applied entries before stopping + // at this cut. Both must obtain a newer checkpoint while its peer is active. + tokio::time::timeout(Duration::from_secs(15), trace.cut.wait()).await?; + for group in [route.history, second] { + let log = f.cluster.voters[0].log.cell_group(&group)?; + let current = log.group_manifest()?.context("concurrent applied cut")?; + let checkpoint = log + .checkpoint_manifest()? + .context("concurrent checkpoint")?; + assert!(checkpoint.position().index > release_cuts[&group].index); + assert!(checkpoint.position().index < current.position().index); + assert_eq!( + current.root().cells(), + if group == route.history { 27 } else { 25 } + ); + } + trace.resume.wait().await; + for (group, writer) in writers { + let last = writer.await??; + controller::applied(&f, group, last.position.index).await?; + } + for (node, controller) in controllers.iter().enumerate() { + assert_eq!(controller.usage().quarantined_groups, 0); + assert!(!controller.usage().closing); + for group in [route.history, second] { + let log = f.cluster.voters[node].log.cell_group(&group)?; + assert!(log.checkpoint_manifest()?.is_some()); + let current = log.group_manifest()?.context("final applied cut")?; + assert_eq!( + current.root().cells(), + if group == route.history { 34 } else { 32 } + ); + let (sql, applier) = &f.appliers[&(node, group)]; + let applier = applier.clone(); + let recovered = f + .cluster + .domain + .blocking(move || applier.recover_selected())? + .await??; + assert_eq!(recovered.marker(), ¤t); + for round in 0..32 { + let (leaf, _) = recovered + .pin_cell(&CellId::new(format!("concurrent-checkpoint-{round}"))?) + .context("independent concurrent cell")?; + assert_eq!(leaf.version.sequence.get(), 1); + } + assert!(sql.usage()?.completion_reservations <= 1); + } + controller.close().await?; + assert_eq!(controller.usage().active_checkpoints, 0); + assert_eq!(pools[node].usage().sources, 0); + assert_eq!(pools[node].usage().retained_bytes, 0); + } + f.close().await +} diff --git a/crates/cellar/tests/support/cell_group_checkpoint_local.rs b/crates/cellar/tests/support/cell_group_checkpoint_local.rs new file mode 100644 index 000000000..bcb39f105 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_checkpoint_local.rs @@ -0,0 +1,562 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Local publication, retained suffixes and real Raft-generated catch-up over +//! native streaming and authenticated TLS, followed by cold source recovery. +use super::*; +use raft::{GetEntriesContext, Storage}; + +impl RuntimeCluster { + pub(super) async fn transfer_checkpoint( + &mut self, + history: GroupHistory, + message: Message, + ) -> Result<()> { + let from = message.from as usize - 1; + let to = message.to as usize - 1; + let root = self + .snapshot_sources + .get(&(from, history)) + .context("snapshot source not enabled")?; + let group = self.cluster.voters[from].log.cell_group(&history)?; + let mut limits = checkpoint_fixture::limits(); + limits.max_sources = 2; + limits.max_retained_bytes *= 2; + let pool = CellGroupCheckpointPool::new(limits)?; + let mut source = pool.capture_snapshot(&group, root, message.get_snapshot())?; + let (_, applier) = self + .appliers + .get(&(to, history)) + .context("snapshot destination not enabled")?; + let mut receiver = pool.receive( + applier.clone(), + checkpoint_receive::scratch()?, + self.cluster._root.path().into(), + )?; + let checkpoint = self + .cluster + .domain + .blocking(move || { + source.stream(|chunk| receiver.append(chunk))?; + receiver.finish() + })? + .await??; + match self.runtimes[to] + .step_checkpoint(history, message.clone(), checkpoint) + .await + { + Ok(CellGroupSnapshotAdmission::Restoring) => self.snapshot_transfers += 1, + Ok(CellGroupSnapshotAdmission::Ignored) => (), + Err(RuntimeError::NotReady | RuntimeError::Overloaded) => { + self.network.push_back((history, message)) + } + Err(error) => return Err(error.into()), + } + assert!( + pool.usage().sources <= 1, + "only the accepted receiver may retain admission" + ); + Ok(()) + } +} + +pub(crate) async fn local_publication(rooted: bool) -> Result<()> { + publication(rooted, Transfer::Native).await +} + +pub(crate) async fn tcp_publication(rooted: bool) -> Result<()> { + publication(rooted, Transfer::Tcp).await +} + +pub(crate) async fn tcp_protocol() -> Result<()> { + publication(false, Transfer::Probes).await +} + +pub(crate) async fn tcp_cancelled_delivery() -> Result<()> { + publication(false, Transfer::Cancelled).await +} + +pub(crate) async fn tcp_retry() -> Result<()> { + publication(false, Transfer::Retry).await +} + +enum Transfer { + Native, + Tcp, + Probes, + Cancelled, + Retry, +} + +async fn publication(rooted: bool, transfer: Transfer) -> Result<()> { + let tcp = !matches!(transfer, Transfer::Native); + let probes = matches!(transfer, Transfer::Probes); + let (mut cluster, _) = checkpoint_fixture::populated_partition(rooted).await?; + let route = cluster.birth.route().clone(); + let group = cluster.voters[0].log.cell_group(&route.history)?; + let root = cluster.voters[0].owner.as_ref().unwrap().manifest().clone(); + let cut = root.marker().clone(); + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let checkpoint = pool.capture(&group, &root)?.prepare_publication()?; + let snapshot = checkpoint.snapshot().clone(); + assert!(pool.capture_snapshot(&group, &root, &snapshot).is_err()); + let cell = history(&route, "suffix")?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(41);")?; + let proposal = cluster.voters[0] + .owner + .as_mut() + .unwrap() + .propose(sql::apply::enrollment_bytes(&route, &cell, &bytes, rooted)?)?; + // Persist a real suffix but withhold all peer replies, so the cut remains + // applied while the leader's next proposal remains uncommitted. + loop { + match cluster.voters[0].owner.as_mut().unwrap().poll()? { + Some(CellGroupEffect::Persist(work)) => { + cluster.voters[0] + .owner + .as_mut() + .unwrap() + .complete_persist(work.run().await?)?; + } + Some(CellGroupEffect::Message(message)) => cluster.network.push_back(message), + None => break, + _ => anyhow::bail!("isolated suffix must not apply"), + } + } + let hard = group.initial_state()?.hard_state; + assert_eq!(hard.commit, cut.position().index.get()); + assert_eq!(group.last_index()?, proposal.position.index.get()); + let suffix = group.entries( + hard.commit + 1, + group.last_index()? + 1, + None, + GetEntriesContext::empty(false), + )?; + // A local publication cannot bundle even an identical hard state, and a + // captured closure from another engine cannot publish on this replica. + let mut mixed = GroupWrite::new(&group); + mixed.snapshot = Some(SnapshotChange::CellGroupCheckpoint(checkpoint.clone())); + mixed.hard_state = Some(hard.clone()); + assert!(cluster.voters[0] + .log + .persist_cells(cellar_logic::consensus::OperationId::new(973), vec![mixed]) + .await + .is_err()); + let foreign_group = cluster.voters[1].log.cell_group(&route.history)?; + let mut foreign = GroupWrite::new(&foreign_group); + foreign.snapshot = Some(SnapshotChange::CellGroupCheckpoint(checkpoint.clone())); + assert!(cluster.voters[1] + .log + .persist_cells( + cellar_logic::consensus::OperationId::new(974), + vec![foreign] + ) + .await + .is_err()); + assert_eq!(group.checkpoint_manifest()?, None); + drop(foreign_group); + let stale = checkpoint.clone(); + cluster.voters[0] + .owner + .as_mut() + .unwrap() + .publish_checkpoint(checkpoint)?; + let Some(CellGroupEffect::Persist(work)) = cluster.voters[0].owner.as_mut().unwrap().poll()? + else { + anyhow::bail!("local checkpoint persistence"); + }; + let release = cluster.voters[0].gate.arm(fault::HOLD_SYNC); + let mut pending = cluster.domain.spawn(work.run())?; + tokio::time::timeout( + Duration::from_secs(5), + cluster.voters[0].gate.entered.notified(), + ) + .await?; + assert_eq!(group.checkpoint_manifest()?, None); + assert_eq!(pool.usage().sources, 1); + assert!(cluster.voters[0].owner.as_mut().unwrap().poll()?.is_none()); + assert!( + tokio::time::timeout(Duration::from_millis(30), &mut pending) + .await + .is_err() + ); + drop(release); + let completion = pending.await??; + assert_eq!(group.snapshot(0, 3)?, snapshot); + assert_eq!(group.first_index()?, cut.position().index.get() + 1); + assert_eq!(group.initial_state()?.hard_state, hard); + assert_eq!(group.group_manifest()?.as_ref(), Some(&cut)); + assert_eq!( + group.entries( + hard.commit + 1, + group.last_index()? + 1, + None, + GetEntriesContext::empty(false) + )?, + suffix + ); + cluster.voters[0] + .owner + .as_mut() + .unwrap() + .complete_persist(completion)?; + let Some(CellGroupEffect::CheckpointPublished { manifest }) = + cluster.voters[0].owner.as_mut().unwrap().poll()? + else { + anyhow::bail!("local publication completion"); + }; + assert_eq!(manifest.as_ref(), &cut); + cluster.drive_partition(false, Some(2)).await?; + assert_eq!( + cluster.voters[0].owner.as_ref().unwrap().status()?.applied, + proposal.position + ); + assert!(matches!( + cluster.voters[0] + .owner + .as_mut() + .unwrap() + .publish_checkpoint(stale), + Err(RuntimeError::Invalid(_)) + )); + assert_eq!(pool.usage().sources, 0); + assert!(pool.capture(&group, &root).is_err()); + let retained = pool.capture_snapshot( + &group, + cluster.voters[0].owner.as_ref().unwrap().manifest(), + &snapshot, + )?; + assert_eq!(retained.marker(), &cut); + drop(retained); + let anchor = cluster.voters[0].owner.as_ref().unwrap().manifest().clone(); + let destination = cluster.voters[2].log.cell_group(&route.history)?; + let applier = CellGroupApplier::new( + destination, + cluster.voters[2] + .pages + .take() + .context("destination pages")?, + checkpoint_receive::config( + cluster.voters[2].images.clone(), + cluster.voters[2].extents.clone(), + cluster._root.path(), + )?, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + )?; + let source_applier = if tcp { + Some(CellGroupApplier::new( + group.clone(), + cluster.voters[0].pages.take().context("source pages")?, + checkpoint_receive::config( + cluster.voters[0].images.clone(), + cluster.voters[0].extents.clone(), + cluster._root.path(), + )?, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + )?) + } else { + None + }; + let mut fixture = RuntimeCluster::from_cluster(cluster, 256, 256, 1000).await?; + let sql = CellGroupSqlPool::start( + &fixture.cluster.voters[2].log, + CellSqlLimits { + workers: 1, + queue_per_worker: 2, + max_pending_cells: 2, + group_pending_cells: 2, + node_bytes: 512 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + }, + )?; + fixture.appliers.insert((2, route.history), (sql, applier)); + fixture.snapshot_sources.insert((0, route.history), anchor); + let mut registrations = Vec::new(); + let mut transport_pools = Vec::new(); + if let Some(source_applier) = source_applier { + fixture.appliers.insert( + (0, route.history), + ( + CellGroupSqlPool::start( + &fixture.cluster.voters[0].log, + CellSqlLimits { + workers: 1, + queue_per_worker: 2, + max_pending_cells: 2, + group_pending_cells: 2, + node_bytes: 512 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + }, + )?, + source_applier, + ), + ); + let other = if probes { + Some(fixture.add_group(91).await?) + } else { + None + }; + fixture.connect_tcp().await?; + for n in [0, 2] { + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let transport = &fixture.transports[n]; + transport.enable_group_checkpoints(pool.clone())?; + assert!(matches!( + transport.enable_group_checkpoints(pool.clone()), + Err(TransportError::Identity) + )); + let applier = fixture.appliers[&(n, route.history)].1.clone(); + registrations.push(transport.register_group_checkpoint(applier.clone())?); + assert!(matches!( + transport.register_group_checkpoint(applier), + Err(TransportError::Identity) + )); + transport_pools.push(pool); + } + assert!(matches!( + fixture.transports[1] + .register_group_checkpoint(fixture.appliers[&(0, route.history)].1.clone()), + Err(TransportError::Identity) + )); + if let Some(other) = other { + network_probes::probe( + &mut fixture, + route.history, + &snapshot, + &transport_pools[1], + other, + &mut registrations, + ) + .await?; + } + } + fixture.retry_snapshot = matches!(transfer, Transfer::Retry); + if fixture.retry_snapshot { + drop(registrations.pop().context("receiver registration")?); + } + fixture.drop_snapshot_delivery = matches!(transfer, Transfer::Cancelled); + fixture.runtimes[0].campaign(route.history).await?; + fixture.drive(None).await?; + assert!( + fixture.snapshot_transfers >= 1, + "Raft must generate a snapshot after compaction" + ); + if fixture.retry_snapshot { + assert!(fixture.failed_snapshot); + assert!(fixture.transports[0].usage().failed >= 1); + } + let restored = fixture.cluster.voters[2].log.cell_group(&route.history)?; + assert_eq!(restored.checkpoint_manifest()?.as_ref(), Some(&cut)); + let source = group.group_manifest()?.context("source root")?; + assert_eq!(restored.group_manifest()?.as_ref(), Some(&source)); + assert_eq!(restored.group_manifest()?.unwrap().root().cells(), 4); + let tail = group.entries( + group.first_index()?, + group.last_index()? + 1, + None, + GetEntriesContext::empty(false), + )?; + assert_eq!(tail.first(), suffix.first()); + let disk = std::mem::replace(&mut fixture.cluster._root, tempfile::tempdir()?); + drop(restored); + drop(root); + drop(group); + drop(registrations); + fixture.close().await?; + for pool in transport_pools { + assert_eq!(pool.usage().sources, 0); + } + // Reopen the actual leader engine and physical closures. Cache-only suffix + // preservation cannot pass this check if publication cleaned durable entries. + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let base = disk.path().join("node-1"); + let log = domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + base.join("log"), + NodeId::new([1; 16])?, + DiskLineage::new([11; 16])?, + ProcessGeneration::new([67; 16])?, + LogOpenMode::Resume, + ), + Arc::new(DirectFileSystem), + )) + .await??; + let group = log.cell_group(&route.history)?; + assert_eq!(group.snapshot(0, 3)?, snapshot); + assert_eq!(group.group_manifest()?.as_ref(), Some(&source)); + assert_eq!( + group.entries( + group.first_index()?, + group.last_index()? + 1, + None, + GetEntriesContext::empty(false) + )?, + tail + ); + let pages = ManifestPageStore::open( + &base.join("pages"), + Arc::new(DirectFileSystem), + 1024, + 16 * 1024 * 1024, + 16, + )?; + let config = checkpoint_receive::config( + GenerationStore::open( + base.join("images"), + Arc::new(DirectFileSystem), + Default::default(), + )?, + if rooted { + Some(cellar_ltx::internal::NodeExtentStore::open( + disk.path().join("extents-0"), + Arc::new(DirectFileSystem), + Default::default(), + )?) + } else { + None + }, + disk.path(), + )?; + let recovering = group.clone(); + let (applier, recovered) = domain + .blocking(move || recover_source(recovering, pages, config, rooted))? + .await??; + assert_eq!(recovered.marker(), &source); + assert_eq!(recovered.pinned_images(), 4); + let (empty, empty_image) = recovered + .pin_cell(&CellId::new("empty")?) + .context("retained empty enrollment")?; + assert_eq!(empty.version.sequence.get(), 1); + assert_eq!(empty.version.image.txid, 0); + assert!(matches!(empty_image, CellImagePin::Empty)); + for name in ["alpha", "beta", "suffix"] { + let (leaf, image) = recovered + .pin_cell(&CellId::new(name)?) + .context("retained populated cell")?; + assert_eq!(leaf.version.sequence.get(), 1); + assert!(!matches!(image, CellImagePin::Empty)); + } + + let mut retained = pool.capture_snapshot(&group, &recovered, &snapshot)?; + assert_eq!(retained.marker(), &cut); + domain + .blocking(move || retained.stream(|_| Ok(())))? + .await??; + assert_eq!(pool.usage().sources, 0); + drop(recovered); + drop(applier); + drop(group); + log.close().await?; + drop(log); + domain.shutdown().await; + Ok(()) +} + +pub(crate) async fn local_shutdown() -> Result<()> { + for rooted in [false, true] { + let (cluster, _) = checkpoint_fixture::populated_with_lagging(rooted, false).await?; + let history = cluster.birth.route().history; + let group = cluster.voters[0].log.cell_group(&history)?; + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let checkpoint = pool + .capture(&group, cluster.voters[0].owner.as_ref().unwrap().manifest())? + .prepare_publication()?; + let cut = checkpoint.marker().clone(); + let gate = cluster.voters[0].gate.clone(); + let fixture = RuntimeCluster::from_cluster(cluster, 256, 256, 1000).await?; + let release = gate.arm(fault::HOLD_SYNC); + fixture.runtimes[0] + .publish_checkpoint(history, checkpoint) + .await?; + tokio::time::timeout(Duration::from_secs(5), gate.entered.notified()).await?; + let runtime = fixture.runtimes[0].clone(); + let mut close = fixture + .cluster + .domain + .spawn(async move { runtime.close().await })?; + assert!(tokio::time::timeout(Duration::from_millis(30), &mut close) + .await + .is_err()); + assert_eq!(pool.usage().sources, 1); + assert_eq!(group.checkpoint_manifest()?, None); + drop(release); + close.await??; + assert_eq!(pool.usage().sources, 0); + assert_eq!(group.checkpoint_manifest()?.as_ref(), Some(&cut)); + drop(group); + fixture.close().await?; + } + Ok(()) +} + +// The source fixture deliberately uses local generation IDs distinct from the +// production SQL adapter. Restore its materialized pins with that same explicit +// resolver; logical roots exclude local filenames. Extents use the production +// recovery adapter. Both paths verify current and retained checkpoint closures. +fn recover_source( + group: GroupLog, + mut pages: ManifestPageStore, + config: CellGroupApplyImages, + rooted: bool, +) -> Result<(Option, PreparedGroupManifest)> { + if rooted { + let (applier, root) = CellGroupApplier::recover( + group, + pages, + config, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + )?; + return Ok((Some(applier), root)); + } + let CellGroupApplyImages::Materialized { images, .. } = config else { + anyhow::bail!("materialized fixture resolver"); + }; + let current = group.group_manifest()?.context("current marker")?; + let checkpoint = group.checkpoint_manifest()?.context("checkpoint marker")?; + let tree = ManifestTree::new(current.route().clone(), ManifestLimits::default())?; + let mut recovered = None; + for marker in [¤t, &checkpoint] { + let root = tree.prepare_complete( + &group, + &mut pages, + marker.root(), + marker.position(), + marker.membership_revision(), + |leaf| { + if leaf.version.image.txid == 0 { + return Ok(CellImagePin::Empty); + } + let CellImageCommitment::Materialized(digest) = leaf.version.image.commitment + else { + return Err(ManifestError::Invalid("fixture materialized commitment")); + }; + let id = image_id(&leaf.version.history, leaf.version.sequence.get()) + .map_err(|error| ManifestError::Io(std::io::Error::other(error)))?; + images + .adopt( + id, + ImageState { + epoch: leaf.version.image.epoch, + txid: TXID(leaf.version.image.txid), + checksum: leaf.version.image.checksum, + digest, + length: leaf.geometry.length, + page_size: leaf.geometry.page_size, + }, + ) + .map(CellImagePin::Materialized) + .map_err(|error| ManifestError::Io(std::io::Error::other(error))) + }, + )?; + if marker == ¤t { + recovered = Some(root); + } + } + Ok((None, recovered.context("current closure")?)) +} diff --git a/crates/cellar/tests/support/cell_group_checkpoint_pressure.rs b/crates/cellar/tests/support/cell_group_checkpoint_pressure.rs new file mode 100644 index 000000000..b728772c1 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_checkpoint_pressure.rs @@ -0,0 +1,297 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Bounded physical transfer and read views; quorum admission is tested separately. +use super::*; +use cellar_ltx::stream_snapshot::{ScratchLimits, ScratchPool}; + +fn digest(pin: &CellImagePin) -> Result<[u8; 32]> { + let mut hash = Sha256::new(); + match pin { + CellImagePin::Empty => {} + CellImagePin::Materialized(pin) => { + let mut reader = pin.open_reader()?; + let mut offset = 0; + while offset < reader.length() { + let size = (reader.length() - offset).min(64 * 1024) as usize; + hash.update(reader.read_exact_at(offset, size)?); + offset += size as u64; + } + } + CellImagePin::Extents(pin) => { + for n in 0..pin.manifest().geometry().extent_count() { + hash.update( + pin.view() + .read_extent(pin.manifest().extent(n).context("extent")?)?, + ); + } + } + } + Ok(hash.finalize().into()) +} + +pub(crate) async fn transfer_pressure(rooted: bool) -> Result<()> { + // 31 populated cells plus empty enrollment: aggregate payload exceeds both + // source workspace and one receiver image spool, without collecting a stream. + let (mut cluster, expected) = + checkpoint_fixture::populate_images(rooted, true, false, 29, 524_288).await?; + let source_group = cluster.voters[0] + .log + .cell_group(&cluster.birth.route().history)?; + let root = cluster.voters[0].owner.as_ref().unwrap().manifest().clone(); + assert_eq!(root.pinned_images(), 32); + let marker = root.marker().clone(); + let (_, reader) = root + .pin_cell(&CellId::new("alpha")?) + .context("reader pin")?; + let old_image = match &reader { + CellImagePin::Materialized(pin) => cluster + ._root + .path() + .join("node-1/images") + .join(format!("{}.sqlite", hex::encode(pin.id().bytes()))), + CellImagePin::Extents(pin) => { + let id = pin.manifest().extent(0).context("reader extent")?; + cluster._root.path().join("extents-0").join(format!( + "e-{:08x}-{}", + id.length(), + hex::encode(id.bytes()) + )) + } + CellImagePin::Empty => anyhow::bail!("populated reader"), + }; + let old_page = cluster._root.path().join("node-1/pages").join(format!( + "{}.page", + hex::encode(marker.root().digest().unwrap()) + )); + let source_pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let mut source = source_pool.capture(&source_group, &root)?; + let source_usage = source_pool.usage(); + let payload: usize = expected.values().map(Vec::len).sum(); + assert!(payload > source_usage.retained_bytes); + assert!(matches!( + source_pool.capture(&source_group, &root), + Err(CellGroupCheckpointError::Busy) + )); + drop(root); + + let applier = checkpoint_receive::applier(&mut cluster)?; + let destination = cluster.voters[2] + .log + .cell_group(&cluster.birth.route().history)?; + let before = destination.group_manifest()?.context("receiver base")?; + let limits = CellGroupCheckpointLimits { + max_sources: 2, + ..checkpoint_fixture::limits() + }; + let pool = CellGroupCheckpointPool::new(limits)?; + let image_bytes = expected["alpha"].len() as u64; + let scratch = ScratchPool::new(ScratchLimits { + max_files: 1, + max_bytes: image_bytes, + })?; + let directory = cluster._root.path().to_owned(); + let mut receiver = pool.receive(applier.clone(), scratch.clone(), directory.clone())?; + let receive_usage = pool.usage(); + assert!(receive_usage.retained_bytes <= limits.max_retained_bytes); + // Verify byte admission independently of item admission, including rollback + // when a second full receiver cannot reserve its metadata/workspace. + let bytes_pool = CellGroupCheckpointPool::new(CellGroupCheckpointLimits { + max_sources: 2, + max_retained_bytes: receive_usage.retained_bytes, + ..limits + })?; + let reserved = bytes_pool.receive(applier.clone(), scratch.clone(), directory.clone())?; + assert!(matches!( + bytes_pool.receive(applier.clone(), scratch.clone(), directory.clone()), + Err(CellGroupCheckpointError::Busy) + )); + assert_eq!(bytes_pool.usage(), receive_usage); + drop(reserved); + assert_eq!(bytes_pool.usage().retained_bytes, 0); + + let (entered, entered_rx) = tokio::sync::oneshot::channel(); + let (release, release_rx) = std::sync::mpsc::channel(); + let observed_pool = pool.clone(); + let observed_scratch = scratch.clone(); + let worker = cluster.domain.blocking(move || -> Result<_> { + let mut entered = Some(entered); + let mut transferred = 0; + let mut peak_scratch = 0; + source.stream(|chunk| { + let image = matches!( + chunk.part, + CellGroupCheckpointPart::Materialized(_) | CellGroupCheckpointPart::Extent { .. } + ); + assert!(chunk.bytes.len() <= limits.max_chunk_bytes); + transferred += chunk.bytes.len(); + receiver.append(chunk)?; + let usage = observed_scratch.usage().unwrap(); + assert!(usage.reserved_files <= 1 && usage.reserved_bytes <= image_bytes); + peak_scratch = peak_scratch.max(usage.reserved_bytes); + assert_eq!(observed_pool.usage(), receive_usage); + if image { + if let Some(entered) = entered.take() { + let _ = entered.send(()); + release_rx.recv().map_err(|_| { + CellGroupCheckpointError::Sink(std::io::Error::other("released")) + })?; + } + } + Ok(()) + })?; + Ok((receiver.finish()?, transferred, peak_scratch)) + })?; + tokio::time::timeout(Duration::from_secs(10), entered_rx).await??; + assert_eq!(pool.usage(), receive_usage); + assert_eq!(source_pool.usage(), source_usage); + let spare = pool.receive(applier.clone(), scratch.clone(), directory.clone())?; + assert!(matches!( + pool.receive(applier.clone(), scratch.clone(), directory.clone()), + Err(CellGroupCheckpointError::Busy) + )); + drop(spare); + assert_eq!(pool.usage(), receive_usage); + if !rooted { + // A different node can produce another valid stream, but this destination + // has only one shared image spool. Refusal must not refund the first one. + let peer_pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let mut peer_source = peer_pool.capture( + &cluster.voters[1] + .log + .cell_group(&cluster.birth.route().history)?, + cluster.voters[1].owner.as_ref().unwrap().manifest(), + )?; + let mut competing = pool.receive(applier.clone(), scratch.clone(), directory.clone())?; + cluster + .domain + .blocking(move || { + assert!(matches!( + peer_source.stream(|chunk| competing.append(chunk)), + Err(CellGroupCheckpointError::Image( + cellar_ltx::generation::GenerationError::Snapshot( + cellar_ltx::stream_snapshot::Error::ScratchFull + ) + )) + )); + })? + .await?; + assert_eq!(pool.usage(), receive_usage); + assert_eq!(peer_pool.usage().sources, 0); + assert_eq!(scratch.usage()?.reserved_files, 1); + assert_eq!(scratch.usage()?.reserved_bytes, image_bytes); + assert_eq!(destination.group_manifest()?.as_ref(), Some(&before)); + } + assert_eq!( + scratch.usage()?.reserved_bytes, + if rooted { 0 } else { image_bytes } + ); + assert_eq!(destination.group_manifest()?.as_ref(), Some(&before)); + // Receiver is deliberately detached. The other two real voters still commit + // mutations while its physical stream is blocked midway through an image. + for name in ["alpha", "beta"] { + checkpoint_fixture::advance_partition(&mut cluster, name, &expected[name], rooted, Some(2)) + .await?; + } + for voter in &mut cluster.voters[..2] { + assert!( + voter + .owner + .as_ref() + .unwrap() + .manifest() + .marker() + .position() + .index + > marker.position().index + ); + for name in ["alpha", "beta"] { + assert_eq!( + voter + .owner + .as_ref() + .unwrap() + .manifest() + .pin_cell(&CellId::new(name)?) + .context("advanced cell")? + .0 + .version + .sequence + .get(), + 2 + ); + } + voter.pages.as_mut().unwrap().reconcile_retained(&[])?; + voter.pages.as_mut().unwrap().reclaim(1024)?; + voter.images.retain_only(&[])?; + voter.images.reclaim(1024)?; + if let Some(store) = &voter.extents { + store.retain_only(&[])?; + store.reclaim(1024)?; + } + } + assert!(old_page.exists()); + release.send(())?; + let (prepared, transferred, peak_scratch) = worker.await??; + assert!(transferred >= payload); + assert_eq!(peak_scratch, if rooted { 0 } else { image_bytes }); + assert_eq!(source_pool.usage().sources, 0); + assert_eq!(source_pool.usage().retained_bytes, 0); + assert_eq!(scratch.usage()?.reserved_bytes, 0); + assert_eq!(scratch.usage()?.reserved_files, 0); + assert_eq!( + pool.usage(), + receive_usage, + "prepared closure still owns admission" + ); + assert_eq!(prepared.marker(), &marker); + assert_eq!(destination.group_manifest()?.as_ref(), Some(&before)); + let source_voter = &mut cluster.voters[0]; + source_voter.pages.as_mut().unwrap().reclaim(1024)?; + source_voter.pages.as_mut().unwrap().reclaim(1024)?; + assert!( + !old_page.exists(), + "released transfer no longer pins its old tree" + ); + source_voter.images.reclaim(1024)?; + if let Some(store) = &source_voter.extents { + store.reclaim(1024)?; + } + assert!( + old_image.exists(), + "reader alone retains its obsolete image" + ); + cluster + .domain + .blocking(move || -> Result<()> { + // The old physical reader survives even after the transfer and its tree + // pin are gone. Check all destination images independently, in bounded reads. + assert_eq!( + digest(&reader)?, + <[u8; 32]>::from(Sha256::digest(&expected["alpha"])) + ); + for (name, bytes) in expected { + let (leaf, pin) = prepared + .manifest() + .pin_cell(&CellId::new(name)?) + .context("received cell")?; + assert_eq!(leaf.version.sequence.get(), 1); + assert_eq!(digest(&pin)?, <[u8; 32]>::from(Sha256::digest(bytes))); + } + Ok(()) + })? + .await??; + assert_eq!(pool.usage().sources, 0); + assert_eq!(pool.usage().retained_bytes, 0); + let source_voter = &mut cluster.voters[0]; + source_voter.images.reclaim(1024)?; + if let Some(store) = &source_voter.extents { + store.reclaim(1024)?; + } + assert!( + !old_image.exists(), + "released reader permits obsolete image reclamation" + ); + drop(applier); + cluster.close().await?; + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_checkpoint_receive.rs b/crates/cellar/tests/support/cell_group_checkpoint_receive.rs new file mode 100644 index 000000000..fc7b65438 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_checkpoint_receive.rs @@ -0,0 +1,496 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Receiver/storage qualification with a lagging member of the real quorum. +//! Snapshot transport and Raft Ready acceptance are deliberately separate gates. +use super::*; +use cellar_logic::consensus::OperationId; +use cellar_ltx::internal::{CacheLimits, NodeExtentStore, EXTENT_BYTES}; +use cellar_ltx::stream_snapshot::{ScratchLimits, ScratchPool}; +use raft::Storage; +use std::path::Path; + +pub(super) fn scratch() -> Result { + Ok(ScratchPool::new(ScratchLimits { + max_bytes: 32 * 1024 * 1024, + max_files: 1, + })?) +} + +pub(super) fn config( + images: GenerationStore, + extents: Option, + directory: &Path, +) -> Result { + let services = cellar_ltx::generation::ReadServices { + now_unix_millis: Arc::new(|| 10), + fill_random: Arc::new(|bytes| bytes.fill(17)), + }; + Ok(match extents { + Some(images) => CellGroupApplyImages::Extents { + images, + services, + limits: Default::default(), + cache_directory: directory.join("cache"), + cache_pool: scratch()?, + cache: CacheLimits { + max_cached_bytes: EXTENT_BYTES, + max_cached_extents: 1, + }, + }, + None => CellGroupApplyImages::Materialized { images, services }, + }) +} + +pub(super) fn applier(cluster: &mut Cluster) -> Result { + let route = cluster.birth.route(); + let voter = &mut cluster.voters[2]; + drop(voter.owner.take()); + drop(voter.held.take()); + voter.paused = false; + Ok(CellGroupApplier::new( + voter.log.cell_group(&route.history)?, + voter.pages.take().context("pages")?, + config( + voter.images.clone(), + voter.extents.clone(), + cluster._root.path(), + )?, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + )?) +} + +pub(crate) async fn install_checkpoint(rooted: bool) -> Result<()> { + let (mut cluster, expected) = checkpoint_fixture::populated_with_lagging(rooted, true).await?; + let route = cluster.birth.route().clone(); + let group = cluster.voters[2].log.cell_group(&route.history)?; + let before = group.group_manifest()?.context("old marker")?; + let source_pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let mut source = source_pool.capture( + &cluster.voters[0].log.cell_group(&route.history)?, + cluster.voters[0].owner.as_ref().unwrap().manifest(), + )?; + let cut = source.marker().clone(); + assert!(cut.position().index > before.position().index); + let applier = applier(&mut cluster)?; + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let scratch = scratch()?; + let mut receiver = pool.receive( + applier.clone(), + scratch.clone(), + cluster._root.path().into(), + )?; + let prepared = cluster + .domain + .blocking(move || { + source.stream(|chunk| receiver.append(chunk))?; + receiver.finish() + })? + .await??; + assert_eq!(scratch.usage()?.reserved_bytes, 0); + assert_eq!(prepared.marker(), &cut); + assert_eq!(group.group_manifest()?.as_ref(), Some(&before)); + assert_eq!(group.checkpoint_manifest()?, None); + assert_eq!(pool.usage().sources, 1); + // A receiver proof cannot grant serving admission before atomic publication. + assert!(cluster + .domain + .run(CellGroupOwner::start( + cluster.permit(2).await?, + prepared.manifest().clone(), + cluster.options.clone(), + [91; 32] + )) + .await? + .is_err()); + let mut write = GroupWrite::new(&group); + write.snapshot = Some(SnapshotChange::CellGroup(prepared.clone())); + let mut hard = group.initial_state()?.hard_state; + hard.commit = cut.position().index.get(); + write.hard_state = Some(hard); + // Reject mixed contracts without releasing the receiver's physical owners. + let mut mixed = write.clone(); + mixed.group_manifest = Some(prepared.manifest().clone()); + assert!(cluster.voters[2] + .log + .persist_cells(OperationId::new(910), vec![mixed]) + .await + .is_err()); + assert_eq!(group.group_manifest()?.as_ref(), Some(&before)); + let installed = prepared.manifest().clone(); + drop(prepared); + let gate = cluster.voters[2].gate.clone(); + let release = gate.arm(fault::HOLD_SYNC); + let writer = cluster.voters[2].log.clone(); + let mut pending = cluster.domain.spawn(async move { + writer + .persist_cells(OperationId::new(911), vec![write]) + .await + })?; + tokio::time::timeout(Duration::from_secs(5), gate.entered.notified()).await?; + assert_eq!(group.group_manifest()?.as_ref(), Some(&before)); + assert_eq!(group.checkpoint_manifest()?, None); + assert_eq!( + pool.usage().sources, + 1, + "queued physical work owns receive admission" + ); + assert!( + tokio::time::timeout(Duration::from_millis(30), &mut pending) + .await + .is_err() + ); + cluster.voters[2].images.retain_only(&[])?; + cluster.voters[2].images.reclaim(1024)?; + if let Some(extents) = &cluster.voters[2].extents { + extents.retain_only(&[])?; + extents.reclaim(1024)?; + } + drop(release); + pending.await??; + assert_eq!(pool.usage().sources, 0); + assert_eq!(group.group_manifest()?.as_ref(), Some(&cut)); + assert_eq!(group.checkpoint_manifest()?.as_ref(), Some(&cut)); + // Stream the receiver's independently sealed closure and compare every byte + // against the SQL fixtures, including the canonical empty cell. + let mut received = checkpoint_fixture::Received::default(); + let mut local = source_pool.capture(&group, &installed)?; + local.stream(|chunk| received.append(chunk))?; + let leaves = expected + .keys() + .map(|name| { + Ok(( + name.clone(), + installed + .pin_cell(&CellId::new(name)?) + .context("received leaf")? + .0 + .clone(), + )) + }) + .collect::>>()?; + received.verify(&cut, &leaves, &expected, rooted)?; + drop(local); + cluster.voters[2].owner = Some( + cluster + .domain + .run(CellGroupOwner::start( + cluster.permit(2).await?, + installed.clone(), + cluster.options.clone(), + [92; 32], + )) + .await??, + ); + assert!(!cluster.voters[2].owner.as_ref().unwrap().status()?.ready); + drop(installed); + assert_eq!(pool.usage().sources, 0); + drop(group); + // Resume real quorum replication, now through the production apply pool on + // the receiver. A newer root must not erase the installed Raft checkpoint. + cluster.network.clear(); + let mut fixture = RuntimeCluster::from_cluster(cluster, 256, 256, 1000).await?; + let sql = CellGroupSqlPool::start( + &fixture.cluster.voters[2].log, + CellSqlLimits { + workers: 1, + queue_per_worker: 2, + max_pending_cells: 2, + group_pending_cells: 2, + node_bytes: 512 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + }, + )?; + fixture.appliers.insert((2, route.history), (sql, applier)); + fixture.runtimes[0].campaign(route.history).await?; + fixture.drive(None).await?; + let later = history(&route, "after-checkpoint")?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(42);")?; + let accepted = fixture.runtimes[0] + .propose( + route.history, + sql::apply::enrollment_bytes(&route, &later, &bytes, rooted)?, + ) + .await?; + fixture.drive(Some(accepted.position.index)).await?; + let group = fixture.cluster.voters[2].log.cell_group(&route.history)?; + let current = group.group_manifest()?.context("later marker")?; + assert!(current.position().index > cut.position().index); + assert_ne!(current.root(), cut.root()); + assert_eq!(group.checkpoint_manifest()?.as_ref(), Some(&cut)); + drop(group); + let directory = fixture.cluster.directory.clone(); + let birth = fixture.cluster.birth.clone(); + let disk = std::mem::replace(&mut fixture.cluster._root, tempfile::tempdir()?); + fixture.close().await?; + // Every original engine, runtime, image/page pin and receiver is now gone. + // Remove the old checkpoint root page, while the later root stays on disk. + let checkpoint_path = disk.path().join("node-3/pages").join(format!( + "{}.page", + hex::encode(cut.root().digest().unwrap()) + )); + // Independently traverse the page references to prove this fault removes + // only the old checkpoint, not any dependency of the current applied tree. + let mut pending_pages = vec![current.root().digest().unwrap()]; + while let Some(digest) = pending_pages.pop() { + assert_ne!(digest, cut.root().digest().unwrap()); + let bytes = std::fs::read( + disk.path() + .join("node-3/pages") + .join(format!("{}.page", hex::encode(digest))), + )?; + let page: serde_json::Value = serde_json::from_slice(&bytes[8..])?; + for child in ["left", "right"] { + if !page[child].is_null() { + pending_pages.push(serde_json::from_value::<[u8; 32]>( + page[child]["digest"].clone(), + )?); + } + } + } + let checkpoint_bytes = std::fs::read(&checkpoint_path)?; + std::fs::remove_file(&checkpoint_path)?; + for missing in [true, false] { + if !missing { + std::fs::write(&checkpoint_path, &checkpoint_bytes)?; + } + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let base = disk.path().join("node-3"); + let log = domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + base.join("log"), + NodeId::new([3; 16])?, + DiskLineage::new([13; 16])?, + ProcessGeneration::new([if missing { 41 } else { 42 }; 16])?, + LogOpenMode::Resume, + ), + Arc::new(fault::FaultFileSystem { + gate: fault::Gate::new(None), + }), + )) + .await??; + let group = log.cell_group(&route.history)?; + assert_eq!(group.group_manifest()?.as_ref(), Some(¤t)); + assert_eq!(group.checkpoint_manifest()?.as_ref(), Some(&cut)); + let pages = ManifestPageStore::open( + &base.join("pages"), + Arc::new(DirectFileSystem), + 1024, + 16 * 1024 * 1024, + 16, + )?; + let config = config( + GenerationStore::open( + base.join("images"), + Arc::new(DirectFileSystem), + Default::default(), + )?, + if rooted { + Some(NodeExtentStore::open( + disk.path().join("extents-2"), + Arc::new(DirectFileSystem), + Default::default(), + )?) + } else { + None + }, + disk.path(), + )?; + let recovering = group.clone(); + let recovered = domain + .blocking(move || { + CellGroupApplier::recover( + recovering, + pages, + config, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + ) + })? + .await?; + if missing { + assert!( + recovered.is_err(), + "missing installed checkpoint was admitted" + ); + } else { + let (applier, root) = recovered?; + assert_eq!(root.marker(), ¤t); + assert_eq!(root.pinned_images(), 4); + assert!(root.pin_cell(&later.cell).is_some()); + let permit = domain + .run(directory.bootstrap_group_member(&birth, &log)) + .await??; + let owner = domain + .run(CellGroupOwner::start( + permit, + root, + RaftRuntimeOptions::default(), + [93; 32], + )) + .await??; + assert!( + !owner.status()?.ready, + "storage recovery cannot renew quorum authority" + ); + drop(owner); + drop(applier); + } + drop(group); + log.close().await?; + drop(log); + domain.shutdown().await; + } + Ok(()) +} + +pub(crate) async fn refuse_checkpoint() -> Result<()> { + for rooted in [false, true] { + let (mut cluster, _) = checkpoint_fixture::populated_with_lagging(rooted, true).await?; + let route = cluster.birth.route().clone(); + let group = cluster.voters[2].log.cell_group(&route.history)?; + let before = group.group_manifest()?.context("old marker")?; + let applier = applier(&mut cluster)?; + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let source_pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + for failure in 0..6 { + let scratch = if failure == 3 { + ScratchPool::new(ScratchLimits { + max_bytes: 1, + max_files: 1, + })? + } else { + scratch()? + }; + let mut receiver = pool.receive( + applier.clone(), + scratch.clone(), + cluster._root.path().into(), + )?; + assert!(matches!( + pool.receive( + applier.clone(), + scratch.clone(), + cluster._root.path().into() + ), + Err(CellGroupCheckpointError::Busy) + )); + let mut source = source_pool.capture( + &cluster.voters[0].log.cell_group(&route.history)?, + cluster.voters[0].owner.as_ref().unwrap().manifest(), + )?; + let mut changed = false; + let result = source.stream(|chunk| { + if failure >= 4 && matches!(chunk.part, CellGroupCheckpointPart::Header) { + changed = true; + if failure == 5 { + return receiver.append(CellGroupCheckpointChunk { + length: 65537, + ..chunk + }); + } + assert_eq!(chunk.offset, 0); + assert_eq!(chunk.length, chunk.bytes.len() as u64); + let mut header: serde_json::Value = + serde_json::from_slice(chunk.bytes).unwrap(); + header["root"]["incarnation"] = serde_json::to_value([99u8; 16]).unwrap(); + let foreign = serde_json::to_vec(&header).unwrap(); + return receiver.append(CellGroupCheckpointChunk { + length: foreign.len() as u64, + bytes: &foreign, + ..chunk + }); + } + if !changed + && (matches!(chunk.part, CellGroupCheckpointPart::Materialized(_)) + || matches!(chunk.part, CellGroupCheckpointPart::Extent { .. })) + { + changed = true; + if failure == 0 { + return Err(CellGroupCheckpointError::Invalid("test truncation")); + } + if failure == 1 { + return receiver.append(CellGroupCheckpointChunk { + offset: chunk.offset + 1, + ..chunk + }); + } + if failure == 2 { + let mut corrupt = chunk.bytes.to_vec(); + corrupt[0] ^= 0xff; + return receiver.append(CellGroupCheckpointChunk { + bytes: &corrupt, + ..chunk + }); + } + } + receiver.append(chunk) + }); + if failure != 3 || !rooted { + assert!(changed); + assert!(result.is_err()); + assert!(receiver.finish().is_err()); + } else { + // Extent transfer consumes bounded memory, no flat-image spool. + result?; + drop(receiver.finish()?); + } + assert_eq!(group.group_manifest()?.as_ref(), Some(&before)); + assert_eq!(group.checkpoint_manifest()?, None); + assert_eq!(scratch.usage()?.reserved_files, 0); + assert_eq!(pool.usage().sources, 0); + } + if !rooted { + // Keep the existing image seal cached while corrupting only this + // disposable, unselected destination. Fresh incoming bytes must not + // make the stale cached seal an install proof. + let mut source = source_pool.capture( + &cluster.voters[0].log.cell_group(&route.history)?, + cluster.voters[0].owner.as_ref().unwrap().manifest(), + )?; + let mut receiver = + pool.receive(applier.clone(), scratch()?, cluster._root.path().into())?; + source.stream(|chunk| receiver.append(chunk))?; + drop(source); + let prepared = receiver.finish()?; + let (_, CellImagePin::Materialized(pin)) = prepared + .manifest() + .pin_cell(&CellId::new("alpha")?) + .context("alpha image")? + else { + anyhow::bail!("materialized image required"); + }; + drop(prepared); + let path = cluster + ._root + .path() + .join("node-3/images") + .join(format!("{}.sqlite", hex::encode(pin.id().bytes()))); + let mut file = std::fs::OpenOptions::new().write(true).open(path)?; + std::io::Write::write_all(&mut file, b"corrupt")?; + file.sync_all()?; + drop(file); + let mut source = source_pool.capture( + &cluster.voters[0].log.cell_group(&route.history)?, + cluster.voters[0].owner.as_ref().unwrap().manifest(), + )?; + let mut receiver = + pool.receive(applier.clone(), scratch()?, cluster._root.path().into())?; + assert!( + source.stream(|chunk| receiver.append(chunk)).is_err(), + "cached destination corruption must refuse transfer" + ); + assert!(receiver.finish().is_err()); + assert_eq!(group.group_manifest()?.as_ref(), Some(&before)); + assert_eq!(pool.usage().sources, 0); + drop(pin); + } + drop(applier); + drop(group); + cluster.close().await?; + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_completion_pressure.rs b/crates/cellar/tests/support/cell_group_completion_pressure.rs new file mode 100644 index 000000000..72ceab5d2 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_completion_pressure.rs @@ -0,0 +1,193 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Real completed images fill the runtime's reserved admission before polling +//! its worker. A separately captured checkpoint must wait without losing pins. +use super::*; +use futures_util::FutureExt; +use raft::Storage; + +pub(crate) async fn checkpoint_completion_pressure(rooted: bool) -> Result<()> { + // A current-thread owner makes the admission burst deterministic even when + // this fixture is composed inside a multi-thread integration test. Both + // execution domains and the nested runtime join before this call returns. + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let result = domain + .blocking(move || -> Result<()> { + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build()?; + runtime.block_on(Box::pin(run(rooted))) + })? + .await; + domain.shutdown().await; + result? +} + +async fn run(rooted: bool) -> Result<()> { + let class = RaftClass::new("Counter", 1, [81; 32])?; + let (mut f, route, _, _) = Box::pin(registered::setup(rooted, &class)).await?; + let slots = f.runtimes[0].usage().completions.max_items; + assert_eq!(slots, 8, "qualify the actual fixed completion capacity"); + let mut groups = vec![route.history]; + for n in 0..slots { + groups.push(f.add_group(100 + n as u8).await?); + } + f.connect_tcp().await?; + for &group in &groups[1..] { + f.runtimes[0].campaign(group).await?; + } + f.settle(&groups).await?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(71);")?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let mut targets = BTreeMap::new(); + let mut cells = Vec::new(); + for &group in &groups[1..] { + let marker = f.cluster.voters[0] + .log + .cell_group(&group)? + .group_manifest()? + .context("completed-image group")?; + let cell = history(marker.route(), "completed-before-checkpoint")?; + let accepted = f.runtimes[0] + .propose(group, request(marker.route(), cell.clone(), 1, &cut)?) + .await?; + targets.insert(group, accepted.position); + cells.push((group, cell)); + } + let successor_cell = history(&route, "after-capacity-checkpoint")?; + let successor = f.runtimes[0] + .propose( + route.history, + apply::enrollment_bytes(&route, &successor_cell, &bytes, rooted)?, + ) + .await?; + let mut held = BTreeMap::new(); + tokio::time::timeout(Duration::from_secs(15), async { + while held.len() != groups.len() { + f.pump(&groups, &mut held).await?; + tokio::task::yield_now().await; + } + Ok::<_, anyhow::Error>(()) + }) + .await??; + let work = held + .remove(&route.history) + .context("checkpoint successor")?; + let before = work.base.marker().clone(); + let group = f.cluster.voters[0].log.cell_group(&route.history)?; + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let source = pool.clone(); + let log = group.clone(); + let root = work.base.clone(); + let checkpoint = f + .cluster + .domain + .blocking(move || source.capture(&log, &root)?.prepare_publication())? + .await??; + let charged = pool.usage(); + assert_eq!(charged.sources, 1); + assert!(charged.retained_bytes > 0); + let mut prepared = Vec::new(); + for &group in &groups[1..] { + let work = held.remove(&group).context("completed-image ticket")?; + let mut voters = f.extra.remove(&group).context("physical image owners")?; + let (voters, work, next) = f + .cluster + .domain + .blocking(move || -> Result<_> { + let next = prepare(&mut voters[0], &work)?; + Ok((voters, work, next)) + })? + .await??; + f.extra.insert(group, voters); + prepared.push((group, work, next)); + } + let runtime = f.runtimes[0].clone(); + let mut pending = prepared + .into_iter() + .map(|(group, work, next)| runtime.complete_apply(group, work, next).boxed()) + .collect::>(); + // No owner turn occurs between these polls. All eight real completions are + // published to the mailbox, retaining their exact fsynced image/ticket. + for call in &mut pending { + assert!(call.as_mut().now_or_never().is_none()); + } + let usage = runtime.usage().completions; + assert_eq!(usage.items, usage.max_items); + assert!(usage.bytes > 0 && usage.bytes <= usage.max_bytes); + pending.push( + runtime + .checkpoint_before_apply(route.history, work.ticket.clone(), checkpoint) + .boxed(), + ); + assert!(pending + .last_mut() + .unwrap() + .as_mut() + .now_or_never() + .is_none()); + assert_eq!(runtime.usage().completions, usage); + assert_eq!(pool.usage(), charged); + assert!(group.checkpoint_manifest()?.is_none()); + + let gate = f.cluster.voters[0].gate.clone(); + let release = gate.arm(fault::HOLD_SYNC); + let mut admitted = futures_util::future::try_join_all(pending).boxed(); + assert!(admitted.as_mut().now_or_never().is_none()); + tokio::time::timeout(Duration::from_secs(5), gate.entered.notified()).await?; + // A real persistence fence is now held. Polling the ninth request lets it + // acquire refunded completion admission while its source remains physical. + assert!(admitted.as_mut().now_or_never().is_none()); + runtime.status(route.history).await?; + tokio::time::timeout(Duration::from_secs(5), admitted).await??; + assert_eq!(runtime.usage().completions.items, 0); + assert_eq!(runtime.usage().completions.bytes, 0); + assert_eq!(pool.usage(), charged); + assert!(group.checkpoint_manifest()?.is_none()); + assert_eq!(group.group_manifest()?.as_ref(), Some(&before)); + assert_eq!( + runtime.status(route.history).await?.applied, + before.position() + ); + drop(release); + controller::wait_for(async || Ok(group.checkpoint_manifest()?.as_ref() == Some(&before))) + .await?; + controller::wait_for(async || Ok(pool.usage().sources == 0)).await?; + assert_eq!(group.group_manifest()?.as_ref(), Some(&before)); + assert!(group.last_index()? >= successor.position.index.get()); + let (sql, applier) = &f.appliers[&(0, route.history)]; + sql.prepare_apply(applier, work) + .await? + .complete(&runtime) + .await?; + f.drive(Some(successor.position.index)).await?; + tokio::time::timeout(Duration::from_secs(15), async { + while targets + .iter() + .any(|(group, position)| f.applied[group].iter().any(|index| *index < position.index)) + { + f.pump(&[], &mut BTreeMap::new()).await?; + tokio::task::yield_now().await; + } + Ok::<_, anyhow::Error>(()) + }) + .await??; + for runtime in &f.runtimes { + for (group, cell) in &cells { + let authority = runtime + .reconciliation_authority(*group, cell.clone()) + .await?; + assert_eq!(authority.cell().version.sequence.get(), 1); + assert_eq!(authority.cell().witness.position, targets[group]); + } + let authority = runtime + .reconciliation_authority(route.history, successor_cell.clone()) + .await?; + assert_eq!(authority.cell().version.sequence.get(), 1); + assert_eq!(authority.cell().witness.position, successor.position); + assert_eq!(runtime.usage().completions.items, 0); + assert_eq!(runtime.usage().completions.bytes, 0); + } + assert_eq!(pool.usage().retained_bytes, 0); + f.close().await +} diff --git a/crates/cellar/tests/support/cell_group_controller.rs b/crates/cellar/tests/support/cell_group_controller.rs new file mode 100644 index 000000000..7d9419ca6 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_controller.rs @@ -0,0 +1,458 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! No fixture event pumping after the production controllers start. +use super::apply_faults::physical; +use super::*; +use std::sync::{ + atomic::{AtomicUsize, Ordering}, + Mutex, +}; + +pub(super) fn limits() -> CellGroupControllerLimits { + CellGroupControllerLimits { + max_groups: 2, + max_parallel_apply: 2, + retry_interval_ms: 5, + max_commands: 2, + max_command_bytes: 64 * 1024, + checkpoints: None, + } +} + +async fn start(fixture: &mut RuntimeCluster) -> Result> { + start_with_limits(fixture, limits()).await +} + +pub(super) async fn start_with_limits( + fixture: &mut RuntimeCluster, + bounds: CellGroupControllerLimits, +) -> Result> { + if fixture.transports.is_empty() { + fixture.connect_tcp().await?; + } + let mut controllers = Vec::new(); + for (node, events) in fixture.events.drain(..).enumerate() { + let selected: Vec<_> = fixture + .appliers + .iter() + .filter(|((n, _), _)| *n == node) + .map(|(_, (_, applier))| applier.clone()) + .collect(); + controllers.push( + CellGroupController::start( + fixture.runtimes[node].clone(), + events, + fixture.transports[node].clone(), + fixture.appliers[&(node, fixture.cluster.birth.route().history)] + .0 + .clone(), + selected, + bounds, + ) + .await?, + ); + } + assert!(fixture.events.is_empty()); + Ok(controllers) +} + +pub(super) async fn wait_for(mut predicate: impl AsyncFnMut() -> Result) -> Result<()> { + tokio::time::timeout(Duration::from_secs(15), async { + while !predicate().await? { + tokio::time::sleep(Duration::from_millis(2)).await; + } + Ok::<_, anyhow::Error>(()) + }) + .await? +} + +pub(super) async fn applied( + fixture: &RuntimeCluster, + group: GroupHistory, + index: LogIndex, +) -> Result<()> { + wait_for(async || { + for runtime in &fixture.runtimes { + if runtime.status(group).await?.applied.index < index { + return Ok(false); + } + } + Ok(true) + }) + .await +} + +pub(super) async fn add_second( + fixture: &mut RuntimeCluster, + primary: GroupHistory, +) -> Result { + let second = fixture.add_group(31).await?; + attach_group_appliers(fixture, primary, second).await?; + Ok(second) +} + +pub(super) async fn attach_group_appliers( + fixture: &mut RuntimeCluster, + primary: GroupHistory, + second: GroupHistory, +) -> Result<()> { + for node in 0..3 { + let pool = fixture.appliers[&(node, primary)].0.clone(); + let voter = &mut fixture.extra.get_mut(&second).context("extra")?[node]; + let applier = CellGroupApplier::new( + voter.log.cell_group(&second)?, + voter.pages.take().context("pages")?, + CellGroupApplyImages::Materialized { + images: voter.images.clone(), + services: cellar_ltx::generation::ReadServices { + now_unix_millis: Arc::new(|| 10), + fill_random: Arc::new(|bytes| bytes.fill(17)), + }, + }, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + )?; + fixture.appliers.insert((node, second), (pool, applier)); + } + Ok(()) +} + +pub(crate) async fn controller_replication(rooted: bool) -> Result<()> { + let calls = Arc::new(AtomicUsize::new(0)); + let count = calls.clone(); + let (entered, started) = tokio::sync::oneshot::channel(); + let entered = Mutex::new(Some(entered)); + let (release, hold) = std::sync::mpsc::sync_channel(1); + let hold = Mutex::new(hold); + let class = Arc::new(RaftClass::new("Counter", 1, [81; 32])?.command( + "Add", + [82; 32], + move |context, _| { + count.fetch_add(1, Ordering::SeqCst); + let _ = entered + .lock() + .unwrap() + .take() + .context("one callback")? + .send(()); + hold.lock().unwrap().recv_timeout(Duration::from_secs(15))?; + context.sql().execute("UPDATE value SET n=n+1", [])?; + Ok(b"committed".to_vec()) + }, + )?); + let (mut fixture, route, a, _) = Box::pin(registered::setup(rooted, &class)).await?; + let second = add_second(&mut fixture, route.history).await?; + let controllers = start(&mut fixture).await?; + fixture.runtimes[0].campaign(second).await?; + wait_for(async || Ok(fixture.runtimes[0].status(second).await?.ready)).await?; + let pool = fixture.appliers[&(0, route.history)].0.clone(); + let authority = fixture.runtimes[0] + .execution_authority(route.history, a.clone()) + .await?; + let options = fixture.cell_options(&a).await?; + let mut attempt = cell_attempt(179)?; + attempt.request.operation = "Add".into(); + attempt.request.contract_digest = [82; 32]; + attempt.request.request_digest = command_request_digest(b""); + let caller = pool.clone(); + let preparation = fixture.cluster.domain.spawn(async move { + caller + .stage_registered(options, authority, attempt, class, vec![], true) + .await + })?; + tokio::time::timeout(Duration::from_secs(5), started).await??; + assert_eq!(pool.usage()?.active_cells, 1); + // The sole SQL worker is physically occupied. Two committed groups must + // share its one reserved apply slot without losing either retained ticket. + let mut accepted = Vec::new(); + for (group, rooted) in [(route.history, rooted), (second, false)] { + let marker = fixture.cluster.voters[0] + .log + .cell_group(&group)? + .group_manifest()? + .context("group")?; + let cell = history(marker.route(), "controller-enrolled")?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(23);")?; + accepted.push(( + group, + fixture.runtimes[0] + .propose( + group, + apply::enrollment_bytes(marker.route(), &cell, &bytes, rooted)?, + ) + .await?, + )); + } + wait_for(async || Ok(controllers[0].usage().apply_retries > 0)).await?; + assert!(!controllers[0].usage().closing); + assert!(controllers[0].usage().active_apply <= 2); + assert!(controllers[0].usage().pending_apply <= 2); + assert_eq!(pool.usage()?.completion_reservations, 1); + release.send(())?; + let CellSqlOutcome::Prepared(prepared) = preparation.await?? else { + anyhow::bail!("preparation"); + }; + for (group, receipt) in accepted { + applied(&fixture, group, receipt.position.index).await?; + } + pool.mark_submitted(a.clone(), prepared.proposal()).await?; + let accepted = accepted_sql(&fixture.runtimes[0], &prepared).await?; + applied(&fixture, route.history, accepted.position.index).await?; + let authority = fixture.runtimes[0] + .execution_authority(route.history, a.clone()) + .await?; + let receipt = pool.finish(authority).await?; + assert_eq!(receipt.receipt().receipt.result(), b"committed"); + assert_eq!(receipt.receipt().version.history, a); + assert_eq!(calls.load(Ordering::SeqCst), 1); + drop(receipt); + drop(prepared); + for (n, controller) in controllers.iter().enumerate() { + wait_for(async || Ok(controller.usage().applied >= 3)).await?; + assert_eq!(controller.usage().groups, 2); + assert_eq!(fixture.transports[n].usage().peers, 2); + controller.close().await?; + assert_eq!(controller.usage().active_apply, 0); + assert_eq!(controller.usage().pending_apply, 0); + assert_eq!( + fixture.appliers[&(n, route.history)] + .0 + .usage()? + .running_workers, + 0 + ); + } + drop(controllers); + fixture.close().await +} + +pub(crate) async fn controller_owners() -> Result<()> { + for fault in 0..6 { + let class = RaftClass::new("Counter", 1, [81; 32])?; + let (mut fixture, route, _, _) = Box::pin(registered::setup(false, &class)).await?; + fixture.connect_tcp().await?; + // All three engines share an execution domain: domain equality is not + // sufficient to authorize any of these borrowed owners. + let events = fixture.events.remove(if fault == 0 { 1 } else { 0 }); + let transport = fixture.transports[usize::from(fault == 1)].clone(); + let pool = fixture.appliers[&(usize::from(fault == 2), route.history)] + .0 + .clone(); + let applier = fixture.appliers[&(usize::from(fault == 3), route.history)] + .1 + .clone(); + let appliers = if fault == 4 { + vec![applier.clone(), applier] + } else { + vec![applier] + }; + let mut bounds = limits(); + if fault == 5 { + bounds.max_parallel_apply = 0; + } + assert!(matches!( + CellGroupController::start( + fixture.runtimes[0].clone(), + events, + transport, + pool, + appliers, + bounds + ) + .await, + Err(CellGroupControllerError::Invalid(_)) + )); + fixture.close().await?; + } + Ok(()) +} + +pub(crate) async fn controller_close() -> Result<()> { + let class = RaftClass::new("Counter", 1, [81; 32])?; + let (mut fixture, route, a, _) = Box::pin(registered::setup(false, &class)).await?; + let controllers = start(&mut fixture).await?; + let pool = fixture.appliers[&(0, route.history)].0.clone(); + let (entered, started) = tokio::sync::oneshot::channel(); + let (release, wait) = std::sync::mpsc::sync_channel(1); + let authority = fixture.runtimes[0] + .execution_authority(route.history, a.clone()) + .await?; + let options = fixture.cell_options(&a).await?; + let caller = pool.clone(); + let producer = fixture.cluster.domain.spawn(async move { + caller + .stage( + options, + authority, + cell_attempt(179).unwrap(), + true, + move |tx| { + let _ = entered.send(()); + wait.recv_timeout(Duration::from_secs(10))?; + tx.execute("UPDATE value SET n=n+1", [])?; + Ok(vec![]) + }, + ) + .await + })?; + tokio::time::timeout(Duration::from_secs(5), started).await??; + producer.abort(); + assert!(producer.await.is_err()); + assert!( + tokio::time::timeout(Duration::from_millis(30), controllers[0].close()) + .await + .is_err() + ); + assert!(controllers[0].usage().closing); + assert_eq!(pool.usage()?.running_workers, 1); + assert!(apply::pool(&fixture, 0).is_err()); + assert!(CellGroupRuntime::start( + fixture.cluster.voters[0].log.clone(), + RaftRuntimeOptions::default() + ) + .await + .is_err()); + release.send(())?; + tokio::time::timeout(Duration::from_secs(5), controllers[0].wait()).await??; + assert_eq!(pool.usage()?.running_workers, 0); + assert_eq!(pool.usage()?.active_cells, 0); + for controller in &controllers { + controller.close().await?; + } + drop(controllers); + drop(pool); + fixture.appliers.clear(); + let replacement = apply::pool(&fixture, 0)?; + replacement.close().await?; + drop(replacement); + fixture.close().await +} + +pub(crate) async fn controller_apply_faults(rooted: bool) -> Result<()> { + for mode in 0..3 { + let root = tempfile::tempdir()?; + let faults = physical::Faults::new(root.path().join("node-1/pages"), None); + let filesystem: Arc = Arc::new(physical::FaultFileSystem { + faults: faults.clone(), + }); + let cluster = + Cluster::with_storage(RaftRuntimeOptions::default(), root, filesystem.clone()).await?; + let fixture = Box::pin(RuntimeCluster::from_cluster(cluster, 256, 256, 1_000_000)).await?; + let (mut fixture, route, a, _) = + Box::pin(enroll_sql_cells(fixture, rooted, filesystem)).await?; + for node in 0..3 { + apply::install(&mut fixture, &a, node).await?; + } + let mut controllers = start(&mut fixture).await?; + let pool = fixture.appliers[&(0, route.history)].0.clone(); + let before = fixture.cluster.voters[0] + .log + .cell_group(&route.history)? + .group_manifest()? + .context("selected")?; + let (entered, started) = tokio::sync::oneshot::channel(); + let (release, held) = std::sync::mpsc::sync_channel(1); + if mode == 0 { + faults + .mode + .store(physical::FAIL_DIRECTORY_SYNC, Ordering::Release); + } else { + *faults.before_directory_sync.lock().unwrap() = Some(Box::new(move || { + let _ = entered.send(()); + held.recv_timeout(Duration::from_secs(15)) + .expect("physical apply release"); + })); + } + let cell = history(&route, "controller-faulted")?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(73);")?; + fixture.runtimes[0] + .propose( + route.history, + apply::enrollment_bytes(&route, &cell, &bytes, rooted)?, + ) + .await?; + if mode == 0 { + // The diagnostic roster changes before the runtime control request + // completes; wait for the actual authority fence, not scheduler timing. + wait_for(async || { + Ok(matches!( + fixture.runtimes[0].status(route.history).await, + Err(RuntimeError::GroupQuarantined) + )) + }) + .await?; + assert!( + matches!(controllers[0].group_failure(route.history)?, Some(CellGroupControllerError::Apply(ref error)) if matches!(error.as_ref(), ExecutionError::Storage(_))) + ); + assert!(!controllers[0].usage().closing); + assert_eq!(controllers[0].usage().quarantined_groups, 1); + assert!(matches!( + fixture.runtimes[0].status(route.history).await, + Err(RuntimeError::GroupQuarantined) + )); + controllers[0].close().await?; + assert_eq!( + faults.mode.load(Ordering::Acquire), + 0, + "real manifest sync fault consumed" + ); + assert!(controllers[0].usage().closing); + } else { + tokio::time::timeout(Duration::from_secs(5), started).await??; + assert_eq!(pool.usage()?.apply_retained_operations, 1); + assert_eq!(controllers[0].usage().active_apply, 1); + if mode == 1 { + assert!( + tokio::time::timeout(Duration::from_millis(30), controllers[0].close()) + .await + .is_err() + ); + } else { + // The worker cannot retain a public handle and prevent its own + // final-handle drop from initiating physical shutdown. + drop(controllers.remove(0)); + wait_for(async || Ok(fixture.transports[0].usage().tasks == 0)).await?; + } + assert!(apply::pool(&fixture, 0).is_err()); + assert!(CellGroupRuntime::start( + fixture.cluster.voters[0].log.clone(), + RaftRuntimeOptions::default() + ) + .await + .is_err()); + assert_eq!(pool.usage()?.running_workers, 1); + assert_eq!(pool.usage()?.apply_retained_operations, 1); + release.send(())?; + if mode == 1 { + tokio::time::timeout(Duration::from_secs(5), controllers[0].wait()).await??; + } else { + wait_for(async || { + Ok(matches!( + fixture.runtimes[0].status(route.history).await, + Err(RuntimeError::Closed) + )) + }) + .await?; + } + } + assert_eq!(pool.usage()?.running_workers, 0); + // A failed or abandoned preparation must not advance engine authority. + assert_eq!( + fixture.cluster.voters[0] + .log + .cell_group(&route.history)? + .group_manifest()? + .as_ref(), + Some(&before) + ); + if mode < 2 { + drop(controllers.remove(0)); + } + for controller in controllers { + controller.close().await?; + } + fixture.close().await?; + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_controller_archive.rs b/crates/cellar/tests/support/cell_group_controller_archive.rs new file mode 100644 index 000000000..8e55aa0e5 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_controller_archive.rs @@ -0,0 +1,617 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Real three-voter controllers, conditional object-store I/O and private replay. +#[path = "cell_group_archive_fairness.rs"] +mod fairness; +pub(crate) use fairness::archived_groups_reclaim_independently; + +use super::*; +use cellar_logic::consensus::OperationId; +use futures_util::TryStreamExt; +use object_store::ObjectStore; +use std::sync::atomic::Ordering; + +pub(super) use crate::archive_store; + +#[path = "cell_group_log_reclaim.rs"] +mod log_reclaim; +pub(crate) use log_reclaim::physical_log_reclamation; + +pub(super) async fn verifier( + fixture: &RuntimeCluster, + route: &GroupRoute, + rooted: bool, + name: &str, +) -> Result { + let route = route.clone(); + let path = fixture.cluster._root.path().join(name); + fixture + .cluster + .domain + .blocking(move || -> Result<_> { + let filesystem = Arc::new(DirectFileSystem); + let pages = ManifestPageStore::open( + &path.join("pages"), + filesystem.clone(), + 1024, + 16 * 1024 * 1024, + 16, + )?; + let services = cellar_ltx::generation::ReadServices { + now_unix_millis: Arc::new(|| 10), + fill_random: Arc::new(|b| b.fill(17)), + }; + let images = if rooted { + CellGroupApplyImages::Extents { + images: NodeExtentStore::open( + path.join("extents"), + filesystem, + Default::default(), + )?, + services, + limits: Default::default(), + cache_directory: path.join("cache"), + cache_pool: ScratchPool::new(ScratchLimits { + max_bytes: 32 * 1024 * 1024, + max_files: 1, + })?, + cache: CacheLimits { + max_cached_bytes: EXTENT_BYTES, + max_cached_extents: 1, + }, + } + } else { + CellGroupApplyImages::Materialized { + images: GenerationStore::open( + path.join("images"), + filesystem, + Default::default(), + )?, + services, + } + }; + Ok(CellGroupArchiveStore::new( + route, + pages, + images, + Default::default(), + Default::default(), + 100, + )?) + })? + .await? +} + +pub(super) async fn registration( + fixture: &RuntimeCluster, + route: &GroupRoute, + rooted: bool, +) -> Result<(CellGroupArchiveRegistration, Arc)> { + registration_minimum(fixture, route, rooted, Durability::Quorum).await +} + +pub(super) async fn registration_minimum( + fixture: &RuntimeCluster, + route: &GroupRoute, + rooted: bool, + minimum: Durability, +) -> Result<(CellGroupArchiveRegistration, Arc)> { + assert_eq!(fixture.cluster.birth.route(), route); + assert_eq!( + fixture + .cluster + .directory + .namespace_config() + .minimum_durability, + minimum + ); + registration_for( + fixture, + &fixture.cluster.directory, + &fixture.cluster.birth, + rooted, + ) + .await +} + +async fn registration_for( + fixture: &RuntimeCluster, + selected: &RangeDirectory, + birth: &CellGroupBirth, + rooted: bool, +) -> Result<(CellGroupArchiveRegistration, Arc)> { + let route = birth.route(); + let name = hex::encode(route.history.group.bytes()); + let io = CellGroupArchivePool::new(Default::default())?; + let binding = selected + .prepare_group_archive_binding(&fixture.cluster.domain, &io, birth) + .await?; + let bytes = binding.encode().to_vec(); + let path = fixture + .cluster + ._root + .path() + .join(format!("archive-binding-intent-{name}")); + fixture + .cluster + .domain + .blocking(move || -> Result<()> { + use std::io::Write; + let mut file = std::fs::OpenOptions::new() + .create_new(true) + .write(true) + .open(&path)?; + file.write_all(&bytes)?; + file.sync_all()?; + std::fs::File::open(path.parent().unwrap())?.sync_all()?; + Ok(()) + })? + .await??; + binding.commit(&fixture.cluster.domain).await?; + let remote = archive_store::Store::around(fixture.cluster.store.clone()); + let catalog = Catalog::new(remote.clone(), Default::default())?; + let config = selected.namespace_config().clone(); + let namespace = fixture + .cluster + .domain + .run(catalog.existing_namespace(config)) + .await??; + let directory = RangeDirectory::new(namespace, Default::default())?; + let birth = fixture + .cluster + .domain + .run(directory.materialize_selected_group(route)) + .await??; + let store = verifier( + fixture, + route, + rooted, + &format!("controller-archive-{name}"), + ) + .await?; + Ok(( + CellGroupArchiveRegistration { + directory, + birth, + io, + owners: CellGroupArchiveUploadPool::new(Default::default())?, + tails: CellGroupArchiveTailPool::new(Default::default())?, + checkpoints: CellGroupCheckpointPool::new(checkpoint_fixture::limits())?, + store, + scratch: ScratchPool::new(ScratchLimits { + max_bytes: 32 * 1024 * 1024, + max_files: 1, + })?, + scratch_directory: fixture.cluster._root.path().into(), + }, + remote, + )) +} + +pub(super) async fn published( + controller: &CellGroupController, + group: GroupHistory, + index: LogIndex, +) -> Result<()> { + controller::wait_for(async || { + let status = controller + .archive_status(group)? + .context("archive status")?; + if let Some(error) = &status.error { + if status.stopped { + anyhow::bail!("archive stopped: {error}"); + } + } + Ok(status + .published + .is_some_and(|marker| marker.position().index >= index)) + }) + .await +} + +async fn add( + fixture: &RuntimeCluster, + route: &GroupRoute, + rooted: bool, + ordinal: u8, +) -> Result { + let image = images::image( + None, + &format!("CREATE TABLE value(n); INSERT INTO value VALUES({ordinal})"), + )?; + let cell = history(route, &format!("later-{ordinal}"))?; + let proposal = fixture.runtimes[0] + .propose( + route.history, + apply::enrollment_bytes(route, &cell, &image, rooted)?, + ) + .await?; + controller::applied(fixture, route.history, proposal.position.index).await?; + Ok(proposal.position.index) +} + +async fn setup(rooted: bool) -> Result<(RuntimeCluster, GroupRoute)> { + setup_store(rooted, Arc::new(object_store::memory::InMemory::new())).await +} + +async fn setup_store( + rooted: bool, + store: Arc, +) -> Result<(RuntimeCluster, GroupRoute)> { + let class = RaftClass::new("Counter", 1, [81; 32])?; + let (fixture, route, _, _) = Box::pin(registered::setup_store(rooted, &class, store)).await?; + Ok((fixture, route)) +} + +pub(super) async fn verified_current( + fixture: &RuntimeCluster, + config: &CellGroupArchiveRegistration, + rooted: bool, + name: &str, +) -> Result { + let route = config.birth.route(); + let registry = config + .directory + .open_group_archive(&fixture.cluster.domain, &config.io, &config.birth) + .await?; + let directory = fixture + .cluster + ._root + .path() + .join("node-1/log/cell-group-archive-journals"); + let history = route.history; + let hold = fixture + .cluster + .domain + .blocking(move || -> Result { + for entry in std::fs::read_dir(directory)? { + let state = entry?.path().join("state"); + let envelope: serde_json::Value = serde_json::from_slice(&std::fs::read(state)?)?; + let record = &envelope["record"]; + let Some(hold) = + serde_json::from_value::>(record["head"].clone())? + else { + continue; + }; + if hold.marker().route().history == history { + assert!(record["retire"].is_null()); + assert_eq!(record["pending"], "None"); + return Ok(hold); + } + } + anyhow::bail!("no selected journal hold for {history:?}") + })? + .await??; + let destination = verifier(fixture, route, rooted, name).await?; + let coverage = registry + .verify_hold( + &fixture.cluster.domain, + &hold, + &config.tails, + config.checkpoints.receive_archive( + destination, + config.scratch.clone(), + config.scratch_directory.clone(), + )?, + ) + .await?; + Ok(coverage) +} + +async fn audit( + fixture: &RuntimeCluster, + config: &CellGroupArchiveRegistration, + rooted: bool, + last: u8, +) -> Result<()> { + let coverage = verified_current(fixture, config, rooted, "controller-archive-audit").await?; + let root = coverage.recovery().manifest(); + assert_eq!(root.pinned_images(), usize::from(last) + 2); + let names = ["registered-a".to_owned(), "registered-b".to_owned()] + .into_iter() + .chain((1..=last).map(|n| format!("later-{n}"))); + for name in names { + let expected = if let Some(n) = name.strip_prefix("later-") { + images::image( + None, + &format!("CREATE TABLE value(n); INSERT INTO value VALUES({n})"), + )? + } else { + images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(0);")? + }; + let (_, pin) = root + .pin_cell(&CellId::new(&name)?) + .context("archived cell")?; + let restored = fixture + .cluster + .domain + .blocking(move || -> Result> { + Ok(match pin { + CellImagePin::Empty => Vec::new(), + CellImagePin::Materialized(pin) => pin + .open_reader()? + .read_exact_at(0, pin.state().length as usize)?, + CellImagePin::Extents(pin) => { + let mut bytes = Vec::new(); + for n in 0..pin.manifest().geometry().extent_count() { + bytes.extend( + pin.view() + .read_extent(pin.manifest().extent(n).context("extent")?)?, + ); + } + bytes + } + }) + })? + .await??; + assert_eq!(restored, expected); + } + Ok(()) +} + +pub(crate) async fn archive_controller_publish(rooted: bool) -> Result<()> { + let (mut fixture, route) = setup(rooted).await?; + publish(&mut fixture, &route, rooted).await?; + fixture.close().await +} + +pub(crate) async fn archive_controller_publish_store( + rooted: bool, + store: Arc, +) -> Result<()> { + let (mut fixture, route) = setup_store(rooted, store).await?; + publish(&mut fixture, &route, rooted).await?; + fixture.close().await +} + +async fn publish(fixture: &mut RuntimeCluster, route: &GroupRoute, rooted: bool) -> Result<()> { + let (config, remote) = registration(fixture, route, rooted).await?; + let initial = fixture.runtimes[0] + .status(route.history) + .await? + .applied + .index; + let gate = remote.hold_publication(); + let controllers = controller::start_with_limits(fixture, controller::limits()).await?; + let call = controllers[0].archive(config.clone())?; + tokio::time::timeout(Duration::from_secs(10), gate.entered.notified()).await?; + // A held archive publication does not stop real quorum/apply progress. + let later = add(fixture, route, rooted, 1).await?; + assert!(later > initial); + assert!(controllers[0] + .archive_status(route.history)? + .unwrap() + .published + .is_none()); + gate.release.notify_one(); + call.response().await?; + published(&controllers[0], route.history, later).await?; + assert!(remote + .publication_cuts + .lock() + .unwrap() + .contains(&initial.get())); + for controller in &controllers { + controller.close().await?; + } + assert!( + controllers[0] + .archive_status(route.history)? + .unwrap() + .stopped + ); + audit(fixture, &config, rooted, 1).await?; + drop(config); + Ok(()) +} + +pub(crate) async fn archive_controller_reconcile(rooted: bool) -> Result<()> { + let (mut fixture, route) = setup(rooted).await?; + reconcile(&mut fixture, &route, rooted).await?; + fixture.close().await +} + +pub(crate) async fn archive_controller_reconcile_store( + rooted: bool, + store: Arc, +) -> Result<()> { + let (mut fixture, route) = setup_store(rooted, store).await?; + reconcile(&mut fixture, &route, rooted).await?; + fixture.close().await +} + +async fn reconcile(fixture: &mut RuntimeCluster, route: &GroupRoute, rooted: bool) -> Result<()> { + let (config, remote) = registration(fixture, route, rooted).await?; + let initial = fixture.runtimes[0] + .status(route.history) + .await? + .applied + .index; + remote + .lose_publication_and_reads + .store(true, Ordering::Release); + let controllers = controller::start_with_limits(fixture, controller::limits()).await?; + // Cancelling a registration response does not cancel its admitted owner. + drop(controllers[0].archive(config.clone())?); + controller::wait_for(async || Ok(remote.outage.load(Ordering::Acquire))).await?; + let mut latest = initial; + for n in 1..=4 { + latest = add(fixture, route, rooted, n).await?; + } + let status = controllers[0].archive_status(route.history)?.unwrap(); + assert!(status.published.is_none()); + assert!(status.error.is_some()); + // Reconciliation must recover the exact earlier publication without a new + // allocation/CAS. Hold the first subsequent registry write to inspect that. + let gate = remote.hold_publication(); + remote.outage.store(false, Ordering::Release); + tokio::time::timeout(Duration::from_secs(10), gate.entered.notified()).await?; + assert_eq!( + controllers[0] + .archive_status(route.history)? + .unwrap() + .published + .unwrap() + .position() + .index, + initial + ); + let mut closing = Box::pin(controllers[0].close()); + assert!( + tokio::time::timeout(Duration::from_millis(50), closing.as_mut()) + .await + .is_err() + ); + gate.release.notify_one(); + closing.await?; + // Closing joins the already admitted archive job, including its new cut. + let selected = controllers[0].archive_status(route.history)?.unwrap(); + assert!(selected.published.unwrap().position().index >= latest); + for controller in controllers.iter().skip(1) { + controller.close().await?; + } + audit(fixture, &config, rooted, 4).await?; + let objects = remote.list(None).try_collect::>().await?; + let key = objects + .iter() + .find(|o| o.location.as_ref().ends_with("/ownership/registry")) + .context("registry")?; + let bytes = remote.get(&key.location).await?.bytes().await?; + let registry: serde_json::Value = serde_json::from_slice(&bytes)?; + assert_eq!(registry["holds"].as_array().context("holds")?.len(), 1); + drop(config); + Ok(()) +} + +/// A configured archive blocks local compaction, not quorum apply or a sibling's +/// checkpoint lane. Only actual complete coverage can reopen that boundary. +pub(crate) async fn archive_checkpoint_frontier( + rooted: bool, + store: Arc, +) -> Result<()> { + use raft::Storage as _; + let (mut fixture, route) = setup_store(rooted, store).await?; + let second = controller::add_second(&mut fixture, route.history).await?; + fixture.connect_tcp().await?; + let pools: Vec<_> = (0..3) + .map(|_| CellGroupCheckpointPool::new(checkpoint_fixture::limits())) + .collect::>()?; + for (transport, pool) in fixture.transports.iter().zip(&pools) { + transport.enable_group_checkpoints(pool.clone())?; + } + let (config, remote) = registration(&fixture, &route, rooted).await?; + let publication = remote.hold_publication(); + let mut limits = controller::limits(); + limits.checkpoints = Some(CellGroupCheckpointPolicy { + interval_entries: 1, + max_parallel: 1, + }); + let controllers = controller::start_with_limits(&mut fixture, limits).await?; + fixture.runtimes[0].campaign(second).await?; + controller::wait_for(async || Ok(fixture.runtimes[0].status(second).await?.ready)).await?; + let group = fixture.cluster.voters[0].log.cell_group(&route.history)?; + let before = group.checkpoint_manifest()?; + let first = group.first_index()?; + let call = controllers[0].archive(config.clone())?; + tokio::time::timeout(Duration::from_secs(10), publication.entered.notified()) + .await + .with_context(|| { + format!( + "archive publication did not enter (rooted={rooted}): {:?}", + controllers[0].archive_status(route.history).map(|status| { + status.map(|status| { + ( + status.attempts, + status.published, + status.error, + status.stopped, + ) + }) + }) + ) + })?; + let later = add(&fixture, &route, rooted, 1).await?; + assert_eq!(group.checkpoint_manifest()?, before); + assert_eq!(group.first_index()?, first); + assert!(controllers[0] + .archive_status(route.history)? + .unwrap() + .published + .is_none()); + let sibling = fixture.cluster.voters[0].log.cell_group(&second)?; + let sibling_route = sibling + .group_manifest()? + .context("sibling")? + .route() + .clone(); + add(&fixture, &sibling_route, false, 1).await?; + let sibling_base = sibling.group_manifest()?.context("sibling base")?; + add(&fixture, &sibling_route, false, 2).await?; + controller::wait_for(async || { + Ok(sibling + .checkpoint_manifest()? + .as_ref() + .is_some_and(|cut| cut.position().index >= sibling_base.position().index)) + }) + .await?; + assert_eq!(group.checkpoint_manifest()?, before); + assert_eq!(controllers[0].usage().quarantined_groups, 0); + assert_eq!(pools[0].usage().sources, 0); + let marker = group.group_manifest()?.context("held archive cut")?; + let source = fixture.runtimes[0] + .capture_checkpoint(route.history, &pools[0]) + .await?; + let prepared = fixture + .cluster + .domain + .blocking(move || source.prepare_publication())? + .await??; + assert_eq!(prepared.marker(), &marker); + controller::wait_for(async || { + match fixture.runtimes[0] + .publish_checkpoint(route.history, prepared.clone()) + .await + { + Err(RuntimeError::Overloaded) => Ok(true), + Err(RuntimeError::NotReady) => Ok(false), + other => anyhow::bail!("unverified checkpoint admitted: {other:?}"), + } + }) + .await?; + let mut write = GroupWrite::new(&group); + write.snapshot = Some(SnapshotChange::CellGroupCheckpoint(prepared.clone())); + assert!(matches!( + fixture.cluster.voters[0] + .log + .persist_cells(OperationId::new(911), vec![write]) + .await, + Err(LogError::Overloaded) + )); + assert_eq!(group.checkpoint_manifest()?, before); + assert_eq!(group.first_index()?, first); + + publication.release.notify_one(); + call.response().await?; + published(&controllers[0], route.history, later).await?; + fixture.runtimes[0] + .publish_checkpoint(route.history, prepared) + .await?; + controller::wait_for(async || Ok(group.checkpoint_manifest()?.as_ref() == Some(&marker))) + .await?; + assert_eq!(group.first_index()?, later.get() + 1); + remote.outage.store(true, Ordering::Release); + add(&fixture, &route, rooted, 2).await?; + let last = add(&fixture, &route, rooted, 3).await?; + assert_eq!(group.checkpoint_manifest()?.as_ref(), Some(&marker)); + assert_eq!(group.first_index()?, later.get() + 1); + assert!(group.last_index()? >= last.get()); + assert_eq!(controllers[0].usage().quarantined_groups, 0); + remote.outage.store(false, Ordering::Release); + published(&controllers[0], route.history, last).await?; + audit(&fixture, &config, rooted, 3).await?; + for controller in &controllers { + controller.close().await?; + } + assert_eq!(config.tails.usage().operations, 0); + fixture.close().await +} diff --git a/crates/cellar/tests/support/cell_group_controller_checkpoints.rs b/crates/cellar/tests/support/cell_group_controller_checkpoints.rs new file mode 100644 index 000000000..4beeb5e60 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_controller_checkpoints.rs @@ -0,0 +1,384 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Production controller owns every event after startup; no fixture pumping. +use super::*; +use raft::Storage; + +struct ReleaseApply(std::sync::mpsc::SyncSender<()>); + +impl Drop for ReleaseApply { + fn drop(&mut self) { + let _ = self.0.send(()); + } +} + +pub(crate) async fn checkpoint_during_apply(rooted: bool) -> Result<()> { + let root = tempfile::tempdir()?; + let directory = if rooted { + root.path().join("rooted-0") + } else { + root.path().join("node-1/images") + }; + let faults = apply_faults::physical::Faults::new(directory, None); + let filesystem = Arc::new(apply_faults::physical::FaultFileSystem { + faults: faults.clone(), + }); + let class = RaftClass::new("Counter", 1, [81; 32])?; + let (mut fixture, route, _, _) = + Box::pin(registered::setup_in(rooted, &class, root, filesystem)).await?; + let second = controller::add_second(&mut fixture, route.history).await?; + fixture.connect_tcp().await?; + let pools: Vec<_> = (0..3) + .map(|_| CellGroupCheckpointPool::new(checkpoint_fixture::limits())) + .collect::>()?; + for (transport, pool) in fixture.transports.iter().zip(&pools) { + transport.enable_group_checkpoints(pool.clone())?; + } + let mut bounds = controller::limits(); + bounds.max_parallel_apply = 1; + bounds.checkpoints = Some(CellGroupCheckpointPolicy { + interval_entries: 2, + max_parallel: 1, + }); + let controllers = controller::start_with_limits(&mut fixture, bounds).await?; + fixture.runtimes[0].campaign(second).await?; + controller::wait_for(async || Ok(fixture.runtimes[0].status(second).await?.ready)).await?; + let group = fixture.cluster.voters[0].log.cell_group(&second)?; + let route_b = group + .group_manifest()? + .context("second birth")? + .route() + .clone(); + let first_b = history(&route_b, "before-busy-apply")?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(43);")?; + let accepted = fixture.runtimes[0] + .propose( + second, + apply::enrollment_bytes(&route_b, &first_b, &bytes, false)?, + ) + .await?; + controller::applied(&fixture, second, accepted.position.index).await?; + let before_b = group.group_manifest()?.context("second durable cut")?; + assert!(group.checkpoint_manifest()?.is_none()); + + // Block actual image persistence on this node, not a consensus/log fence. + // The sole SQL apply slot and its retained physical owners remain occupied. + let (entered, started) = tokio::sync::oneshot::channel(); + let (release, wait) = std::sync::mpsc::sync_channel(1); + let release = ReleaseApply(release); + *faults.before_directory_sync.lock().unwrap() = Some(Box::new(move || { + let _ = entered.send(()); + wait.recv_timeout(Duration::from_secs(30)) + .expect("release actual image persistence"); + })); + let before_a = fixture.cluster.voters[0] + .log + .cell_group(&route.history)? + .group_manifest()? + .context("first durable cut")?; + let held_cell = history(&route, "held-physical-apply")?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(97);")?; + let held = fixture.runtimes[0] + .propose( + route.history, + apply::enrollment_bytes(&route, &held_cell, &bytes, rooted)?, + ) + .await?; + tokio::time::timeout(Duration::from_secs(5), started).await??; + assert_eq!(controllers[0].usage().active_apply, 1); + let sql = &fixture.appliers[&(0, route.history)].0; + assert_eq!(sql.usage()?.apply_retained_operations, 1); + assert_eq!(sql.usage()?.completion_reservations, 1); + assert_eq!(controllers[0].usage().active_checkpoints, 0); + assert_eq!(pools[0].usage().sources, 0); + + let next_b = history(&route_b, "after-busy-apply")?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(67);")?; + let successor = fixture.runtimes[0] + .propose( + second, + apply::enrollment_bytes(&route_b, &next_b, &bytes, false)?, + ) + .await?; + controller::wait_for(async || { + for runtime in &fixture.runtimes[1..] { + if runtime.status(second).await?.applied.index < successor.position.index { + return Ok(false); + } + } + Ok(true) + }) + .await?; + assert!(fixture.runtimes[0].status(second).await?.committed >= successor.position.index); + // The second group's committed successor makes its preceding complete cut + // due. Checkpoint capture/publication does not require an SQL apply slot. + tokio::time::timeout(Duration::from_secs(5), async { + while group.checkpoint_manifest()?.as_ref() != Some(&before_b) { + tokio::time::sleep(Duration::from_millis(2)).await; + } + Ok::<_, anyhow::Error>(()) + }) + .await + .with_context(|| format!( + "unrelated checkpoint must publish while the SQL apply slot is held: controller={:?}, checkpoint_pool={:?}, sql={:?}", + controllers[0].usage(), pools[0].usage(), + sql.usage().map(|usage| (usage.apply_retained_operations, usage.completion_reservations)) + ))??; + assert_eq!(group.group_manifest()?.as_ref(), Some(&before_b)); + assert_eq!(group.first_index()?, before_b.position().index.get() + 1); + assert!(group.last_index()? >= successor.position.index.get()); + assert_eq!( + fixture.runtimes[0].status(second).await?.applied, + before_b.position() + ); + assert_eq!( + fixture.runtimes[0].status(route.history).await?.applied, + before_a.position() + ); + assert_eq!(controllers[0].usage().active_apply, 1); + assert_eq!(sql.usage()?.apply_retained_operations, 1); + assert_eq!(sql.usage()?.completion_reservations, 1); + assert_eq!(controllers[0].usage().quarantined_groups, 0); + drop(release); + controller::applied(&fixture, route.history, held.position.index).await?; + controller::applied(&fixture, second, successor.position.index).await?; + for runtime in &fixture.runtimes { + for (group, cell) in [ + (route.history, &held_cell), + (second, &first_b), + (second, &next_b), + ] { + let authority = runtime + .reconciliation_authority(group, cell.clone()) + .await?; + assert_eq!(authority.cell().version.sequence.get(), 1); + } + } + for (node, controller) in controllers.iter().enumerate() { + controller.close().await?; + assert_eq!(controller.usage().active_apply, 0); + assert_eq!(controller.usage().active_checkpoints, 0); + assert_eq!(pools[node].usage().sources, 0); + } + fixture.close().await +} + +pub(crate) async fn automatic_checkpoints(rooted: bool, pressure: bool) -> Result<()> { + let class = RaftClass::new("Counter", 1, [81; 32])?; + let (mut fixture, route, _, _) = Box::pin(registered::setup(rooted, &class)).await?; + let second = controller::add_second(&mut fixture, route.history).await?; + fixture.connect_tcp().await?; + let pools: Vec<_> = (0..3) + .map(|_| CellGroupCheckpointPool::new(checkpoint_fixture::limits())) + .collect::>()?; + for (transport, pool) in fixture.transports.iter().zip(&pools) { + transport.enable_group_checkpoints(pool.clone())?; + } + let mut held = if pressure { + Some(pools[0].receive( + fixture.appliers[&(0, route.history)].1.clone(), + checkpoint_receive::scratch()?, + fixture.cluster._root.path().into(), + )?) + } else { + None + }; + let mut bounds = controller::limits(); + bounds.checkpoints = Some(CellGroupCheckpointPolicy { + interval_entries: 2, + max_parallel: 1, + }); + let controllers = controller::start_with_limits(&mut fixture, bounds).await?; + fixture.runtimes[0].campaign(second).await?; + controller::wait_for(async || Ok(fixture.runtimes[0].status(second).await?.ready)).await?; + let initial = fixture.runtimes[0] + .status(route.history) + .await? + .applied + .index + .get(); + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(23);")?; + for round in 0..12 { + // Interleaved writes to two groups share one checkpoint slot per node. + // All voters must apply every successor, including under pool pressure. + for (group, format) in [(route.history, rooted), (second, false)] { + let log = fixture.cluster.voters[0].log.cell_group(&group)?; + let marker = log.group_manifest()?.context("selected")?; + let cell = history(marker.route(), &format!("auto-{round}"))?; + let accepted = fixture.runtimes[0] + .propose( + group, + apply::enrollment_bytes(marker.route(), &cell, &bytes, format)?, + ) + .await?; + controller::applied(&fixture, group, accepted.position.index).await?; + } + for controller in &controllers { + assert!(!controller.usage().closing); + assert!(controller.usage().active_checkpoints <= 1); + assert_eq!(controller.usage().quarantined_groups, 0); + } + if pressure && round == 4 { + assert!(controllers[0].usage().checkpoint_retries > 0); + assert_eq!(controllers[0].usage().checkpoints_published, 0); + assert_eq!(pools[0].usage().sources, 1); + drop(held.take()); + tokio::time::sleep(Duration::from_millis(bounds.retry_interval_ms + 1)).await; + } + } + for (node, controller) in controllers.iter().enumerate() { + controller::wait_for(async || Ok(controller.usage().active_checkpoints == 0)).await?; + assert!(controller.usage().checkpoints_published >= 4); + for group in [route.history, second] { + let log = fixture.cluster.voters[node].log.cell_group(&group)?; + let current = log.group_manifest()?.context("applied root")?; + let checkpoint = log.checkpoint_manifest()?.context("automatic checkpoint")?; + assert!(checkpoint.position().index.get() > initial); + assert!(checkpoint.position().index < current.position().index); + assert!(current.position().index.get() - checkpoint.position().index.get() <= 2); + assert_eq!(log.first_index()?, checkpoint.position().index.get() + 1); + assert_eq!(log.last_index()?, current.position().index.get()); + } + controller.close().await?; + assert_eq!(controller.usage().active_checkpoints, 0); + assert_eq!(pools[node].usage().sources, 0); + } + drop(controllers); + fixture.close().await +} + +pub(crate) async fn controller_checkpoint_shutdown(rooted: bool) -> Result<()> { + let class = RaftClass::new("Counter", 1, [81; 32])?; + let (mut fixture, route, _, _) = Box::pin(registered::setup(rooted, &class)).await?; + let second = controller::add_second(&mut fixture, route.history).await?; + fixture.connect_tcp().await?; + let pools: Vec<_> = (0..3) + .map(|_| CellGroupCheckpointPool::new(checkpoint_fixture::limits())) + .collect::>()?; + for (transport, pool) in fixture.transports.iter().zip(&pools) { + transport.enable_group_checkpoints(pool.clone())?; + } + let mut bounds = controller::limits(); + bounds.checkpoints = Some(CellGroupCheckpointPolicy { + interval_entries: 2, + max_parallel: 1, + }); + let controllers = controller::start_with_limits(&mut fixture, bounds).await?; + fixture.runtimes[1].campaign(second).await?; + controller::wait_for(async || Ok(fixture.runtimes[1].status(second).await?.ready)).await?; + let elected = fixture.runtimes[1].status(second).await?.applied.index; + controller::applied(&fixture, second, elected).await?; + let group = fixture.cluster.voters[0].log.cell_group(&route.history)?; + let before = group.group_manifest()?.context("preceding durable cut")?; + assert!(group.checkpoint_manifest()?.is_none()); + let controller = controllers[0].clone(); + let checkpoint_pool = pools[0].clone(); + let gate = fixture.cluster.voters[0].gate.clone(); + // Pass ordinary append/commit fences. Hold a sync only after the controller + // owns both its lifecycle slot and the captured checkpoint reservation. + *gate.sync_when.lock().unwrap() = Some(Arc::new(move || { + controller.usage().active_checkpoints == 1 && checkpoint_pool.usage().sources == 1 + })); + let release = gate.arm(fault::HOLD_SYNC); + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(29);")?; + let cell = history(&route, "checkpoint-successor")?; + let successor = fixture.runtimes[0] + .propose( + route.history, + apply::enrollment_bytes(&route, &cell, &bytes, rooted)?, + ) + .await?; + tokio::time::timeout(Duration::from_secs(5), gate.entered.notified()).await?; + assert_eq!(controllers[0].usage().active_checkpoints, 1); + let charged = pools[0].usage(); + assert_eq!(charged.sources, 1); + assert!(charged.retained_bytes > 0); + assert_eq!(group.group_manifest()?.as_ref(), Some(&before)); + assert!(group.checkpoint_manifest()?.is_none()); + assert_eq!( + fixture.runtimes[0].status(route.history).await?.applied, + before.position() + ); + assert!(matches!( + pools[0].receive( + fixture.appliers[&(0, route.history)].1.clone(), + checkpoint_receive::scratch()?, + fixture.cluster._root.path().into(), + ), + Err(CellGroupCheckpointError::Busy) + )); + assert_eq!(pools[0].usage(), charged); + + // This node's shared engine is stalled. The other two real voters still + // form a quorum and must advance the other group through repeated writes. + for round in 0..4 { + let selected = fixture.cluster.voters[1] + .log + .cell_group(&second)? + .group_manifest()? + .context("healthy group")?; + let cell = history(selected.route(), &format!("while-checkpoint-sync-{round}"))?; + let accepted = fixture.runtimes[1] + .propose( + second, + apply::enrollment_bytes(selected.route(), &cell, &bytes, false)?, + ) + .await?; + controller::wait_for(async || { + for runtime in &fixture.runtimes[1..] { + if runtime.status(second).await?.applied.index < accepted.position.index { + return Ok(false); + } + } + Ok(true) + }) + .await?; + assert_eq!(controllers[0].usage().active_checkpoints, 1); + assert_eq!(pools[0].usage(), charged); + assert_eq!(controllers[0].usage().checkpoints_published, 0); + } + // Cancelling both close and wait leaves the cleanup task, source pins and + // exclusive engine owner alive until the actual sync returns. + assert!( + tokio::time::timeout(Duration::from_millis(30), controllers[0].close()) + .await + .is_err() + ); + assert!( + tokio::time::timeout(Duration::from_millis(30), controllers[0].wait()) + .await + .is_err() + ); + assert!(controllers[0].usage().closing); + assert_eq!(controllers[0].usage().active_checkpoints, 1); + assert_eq!(pools[0].usage(), charged); + assert!(matches!( + CellGroupRuntime::start( + fixture.cluster.voters[0].log.clone(), + RaftRuntimeOptions::default() + ) + .await, + Err(RuntimeError::Storage(LogError::AlreadyExists)) + )); + assert!(apply::pool(&fixture, 0).is_err()); + drop(release); + tokio::time::timeout(Duration::from_secs(5), controllers[0].close()).await??; + assert_eq!(controllers[0].usage().active_checkpoints, 0); + assert_eq!(pools[0].usage().sources, 0); + assert_eq!(pools[0].usage().retained_bytes, 0); + assert_eq!(group.checkpoint_manifest()?.as_ref(), Some(&before)); + assert_eq!(group.group_manifest()?.as_ref(), Some(&before)); + assert!(group.last_index()? >= successor.position.index.get()); + let (replacement, events) = CellGroupRuntime::start( + fixture.cluster.voters[0].log.clone(), + RaftRuntimeOptions::default(), + ) + .await?; + replacement.close().await?; + drop(events); + for controller in controllers { + controller.close().await?; + assert_eq!(controller.usage().active_checkpoints, 0); + } + fixture.close().await +} diff --git a/crates/cellar/tests/support/cell_group_controller_commands.rs b/crates/cellar/tests/support/cell_group_controller_commands.rs new file mode 100644 index 000000000..1f3b97ff1 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_controller_commands.rs @@ -0,0 +1,634 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Response lifetimes through real TLS, Raft Engine, SQLite and physical apply. +use super::apply_faults::physical; +use super::*; +use std::sync::{ + atomic::{AtomicUsize, Ordering}, + Mutex, +}; + +#[path = "cell_group_background_admission.rs"] +mod background; +pub(crate) use background::run as background_admission; + +async fn wait_for(mut predicate: impl AsyncFnMut() -> Result) -> Result<()> { + tokio::time::timeout(Duration::from_secs(15), async { + while !predicate().await? { + tokio::time::sleep(Duration::from_millis(2)).await; + } + Ok::<_, anyhow::Error>(()) + }) + .await? +} + +pub(super) async fn invocation( + dispatcher: &CellGroupDispatcher, + class: &RaftClass, + cell: &CellHistory, + timeout: Duration, + retry_until_ms: u64, +) -> Result { + Ok(dispatcher + .command( + cell.clone(), + CellCommandRequest { + id: CommandId::new(ClientId::new([188; 16])?, 1)?, + contract: class.contract("Add").context("Add")?, + input: b"owned".to_vec(), + durability: Durability::Quorum, + retry_until_ms, + }, + CellRoutingLimits { + timeout, + max_attempts: 4, + }, + None, + ) + .await?) +} + +pub(crate) async fn command_admission_starts_before_poll() -> Result<()> { + for rooted in [false, true] { + let calls = Arc::new(AtomicUsize::new(0)); + let counted = calls.clone(); + let class = Arc::new(RaftClass::new("Counter", 1, [81; 32])?.command( + "Add", + [82; 32], + move |_, _| { + counted.fetch_add(1, Ordering::SeqCst); + Ok(Vec::new()) + }, + )?); + let (mut fixture, route, cell, _) = Box::pin(registered::setup(rooted, &class)).await?; + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let pool = fixture.appliers[&(0, route.history)].0.clone(); + let dispatcher = CellGroupDispatcher::new( + &fixture.cluster.domain, + fixture.cluster.directory.clone(), + class.clone(), + super::remote_queries::keys(89)?, + CellDispatchLimits { + max_cached_routes: 1, + route_cache_bytes: 8192, + max_invocations: 1, + retained_bytes: 64 * 1024, + max_input_bytes: 1024, + }, + )?; + let request = CellCommandRequest { + id: CommandId::new(ClientId::new([188; 16])?, 1)?, + contract: class.contract("Add").context("Add")?, + input: b"unpolled".to_vec(), + durability: Durability::Quorum, + retry_until_ms: u64::MAX, + }; + let limits = CellRoutingLimits { + timeout: Duration::from_millis(40), + max_attempts: 1, + }; + let pending = dispatcher.command(cell.clone(), request.clone(), limits, None); + assert_eq!( + dispatcher.usage().1, + 1, + "unpolled construction retains admission" + ); + let refused = dispatcher + .command(cell.clone(), request.clone(), limits, None) + .await + .err() + .context("unpolled construction exhausts admission")?; + assert!(!refused.ambiguous); + assert!(matches!( + refused.cause.as_ref(), + CellDispatchError::Runtime(RuntimeError::Overloaded) + )); + tokio::time::sleep(Duration::from_millis(60)).await; + let expired = pending + .await + .err() + .context("construction keeps original deadline")?; + assert!(!expired.ambiguous); + assert!(matches!( + expired.cause.as_ref(), + CellDispatchError::Routing(CellRoutingError { + reason: CellRoutingFailure::Deadline, + .. + }) + )); + assert_eq!(dispatcher.usage().1, 0); + let cancelled = dispatcher.command(cell, request, limits, None); + assert_eq!(dispatcher.usage().1, 1); + drop(cancelled); + assert_eq!(dispatcher.usage().1, 0); + assert_eq!(calls.load(Ordering::SeqCst), 0); + assert_eq!(pool.usage()?.active_cells, 0); + for controller in &controllers { + controller.close().await?; + } + drop(controllers); + fixture.close().await?; + } + Ok(()) +} + +// phase=0 holds callback execution; phase=1 holds the committed manifest sync. +// mode=0 disconnects, mode=1 expires the response, mode=2 cancels node close. +pub(crate) async fn owned_commands(phase: u8, mode: u8) -> Result<()> { + for rooted in [false, true] { + let root = tempfile::tempdir()?; + let faults = physical::Faults::new(root.path().join("node-1/pages"), None); + let filesystem = Arc::new(physical::FaultFileSystem { + faults: faults.clone(), + }); + let calls = Arc::new(AtomicUsize::new(0)); + let count = calls.clone(); + let (entered, started) = tokio::sync::oneshot::channel(); + let entered = Arc::new(Mutex::new(Some(entered))); + let (release, held) = std::sync::mpsc::sync_channel(1); + let held = Arc::new(Mutex::new(held)); + let callback_entered = entered.clone(); + let callback_held = held.clone(); + let class = Arc::new(RaftClass::new("Counter", 1, [81; 32])?.command( + "Add", + [82; 32], + move |context, _| { + if count.fetch_add(1, Ordering::SeqCst) == 0 && phase == 0 { + let _ = callback_entered + .lock() + .unwrap() + .take() + .context("callback signal")? + .send(()); + callback_held + .lock() + .unwrap() + .recv_timeout(Duration::from_secs(15))?; + } + context.sql().execute("UPDATE value SET n=n+1", [])?; + Ok(b"owned-result".to_vec()) + }, + )?); + let (mut fixture, route, a, b) = + Box::pin(registered::setup_in(rooted, &class, root, filesystem)).await?; + let bounds = CellGroupControllerLimits { + max_commands: 1, + ..controller::limits() + }; + let controllers = controller::start_with_limits(&mut fixture, bounds).await?; + assert!(fixture.events.is_empty()); // No fixture pumping below this point. + let keys = VersionKeys::new( + VersionSigningKey::new([89; 32], u64::MAX)?, + vec![], + Duration::from_secs(30), + )?; + let dispatcher = CellGroupDispatcher::new( + &fixture.cluster.domain, + fixture.cluster.directory.clone(), + class.clone(), + keys.clone(), + CellDispatchLimits { + max_cached_routes: 32, + route_cache_bytes: 256 * 1024, + max_invocations: 8, + retained_bytes: 1024 * 1024, + max_input_bytes: 1024, + }, + )?; + if phase == 1 { + *faults.before_directory_sync.lock().unwrap() = Some(Box::new(move || { + let _ = entered + .lock() + .unwrap() + .take() + .expect("apply signal") + .send(()); + held.lock() + .unwrap() + .recv_timeout(Duration::from_secs(15)) + .expect("apply release"); + })); + } + let pool = fixture.appliers[&(0, route.history)].0.clone(); + let retry_until = fixture + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms().max(0) as u64 + 60_000 }) + .await?; + let mut used = invocation( + &dispatcher, + &class, + &b, + Duration::from_secs(15), + retry_until, + ) + .await?; + used.begin().await?; + let original_deadline = used.attempt().context("routed invocation")?.deadline_ms(); + let refused = controllers[0] + .command(used, fixture.cell_options(&b).await?) + .err() + .context("fresh invocation required")?; + assert!(matches!( + refused.error.cause.as_ref(), + CellDispatchError::Invalid(_) + )); + assert_eq!( + refused + .invocation + .attempt() + .context("returned original owner")? + .deadline_ms(), + original_deadline + ); + drop(refused); + let mut oversized = fixture.cell_options(&b).await?; + match &mut oversized { + CellSqlOptions::Materialized(o) => { + o.directory = std::path::PathBuf::from("x".repeat(4097)) + } + CellSqlOptions::Extents(o) => { + o.cache_directory = std::path::PathBuf::from("x".repeat(4097)) + } + } + let refused = controllers[0] + .command( + invocation( + &dispatcher, + &class, + &b, + Duration::from_secs(15), + retry_until, + ) + .await?, + oversized, + ) + .err() + .context("option admission")?; + assert!(matches!( + refused.error.cause.as_ref(), + CellDispatchError::Runtime(RuntimeError::Invalid(_)) + )); + drop(refused); + assert_eq!(dispatcher.usage().1, 0); + assert_eq!(calls.load(Ordering::SeqCst), 0); + let timeout = if mode == 1 { + Duration::from_millis(400) + } else { + Duration::from_secs(15) + }; + let call = controllers[0].dispatch_command( + &dispatcher, + a.clone(), + CellCommandRequest { + id: CommandId::new(ClientId::new([188; 16])?, 1)?, + contract: class.contract("Add").context("Add")?, + input: b"owned".to_vec(), + durability: Durability::Quorum, + retry_until_ms: retry_until, + }, + CellRoutingLimits { + timeout, + max_attempts: 4, + }, + fixture.cell_options(&a).await?, + )?; + tokio::time::timeout(Duration::from_secs(5), started).await??; + assert_eq!(calls.load(Ordering::SeqCst), 1); + assert_eq!(pool.usage()?.active_cells, 1); + assert_eq!(controllers[0].usage().retained_commands, 1); + assert_eq!(dispatcher.usage().1, 1); + if phase == 1 { + let status = fixture.runtimes[0].status(route.history).await?; + assert!(status.committed > status.applied.index); + } + let rejected = controllers[0] + .command( + invocation( + &dispatcher, + &class, + &b, + Duration::from_secs(10), + retry_until, + ) + .await?, + fixture.cell_options(&b).await?, + ) + .err() + .context("bounded admission")?; + assert!(!rejected.error.ambiguous); + assert!(matches!( + rejected.error.cause.as_ref(), + CellDispatchError::Runtime(RuntimeError::Overloaded) + )); + assert!(rejected.invocation.attempt().is_none()); + assert_eq!(dispatcher.usage().1, 2); + drop(rejected); + assert_eq!(dispatcher.usage().1, 1); + if mode == 1 { + let error = call.response().await.err().context("response deadline")?; + assert!(error.ambiguous); + assert!(matches!( + error.cause.as_ref(), + CellDispatchError::Runtime(RuntimeError::DeadlineExceeded) + )); + } else { + drop(call); + } + if mode == 2 { + assert!( + tokio::time::timeout(Duration::from_millis(30), controllers[0].close()) + .await + .is_err() + ); + assert!(apply::pool(&fixture, 0).is_err()); + assert!(CellGroupRuntime::start( + fixture.cluster.voters[0].log.clone(), + RaftRuntimeOptions::default() + ) + .await + .is_err()); + } + // Neither a lost waiter nor its deadline can free the entered producer. + assert_eq!(pool.usage()?.active_cells, 1); + assert_eq!(controllers[0].usage().retained_commands, 1); + assert_eq!(dispatcher.usage().1, 1); + release.send(())?; + if mode == 2 { + tokio::time::timeout(Duration::from_secs(5), controllers[0].wait()).await??; + assert_eq!(pool.usage()?.running_workers, 0); + assert_eq!(pool.usage()?.active_cells, 0); + assert_eq!(controllers[0].usage().retained_commands, 0); + assert_eq!(dispatcher.usage().1, 0); + } else { + wait_for(async || { + Ok(controllers[0].usage().retained_commands == 0 + && pool.usage()?.active_cells == 0 + && dispatcher.usage().1 == 0) + }) + .await?; + let sealed = phase == 0 && mode == 1; + let state = fixture.runtimes[0] + .reconciliation_authority(route.history, a.clone()) + .await?; + assert_eq!( + state.cell().version.sequence.get(), + if sealed { 1 } else { 2 } + ); + assert_eq!(calls.load(Ordering::SeqCst), 1); + // Retrying the same identity reads a committed receipt, or executes + // once after the exact untouched preparation was sealed and closed. + let retry = controllers[0].command( + invocation( + &dispatcher, + &class, + &a, + Duration::from_secs(15), + retry_until, + ) + .await?, + fixture.cell_options(&a).await?, + )?; + let response = retry.response().await?; + assert_eq!( + response + .local_receipt() + .expect("local command receipt") + .receipt + .result(), + b"owned-result" + ); + assert_eq!( + response + .local_receipt() + .expect("local command receipt") + .version + .history, + a + ); + assert_eq!( + response + .local_receipt() + .expect("local command receipt") + .version + .sequence + .get(), + 2 + ); + assert_eq!(calls.load(Ordering::SeqCst), if sealed { 2 } else { 1 }); + let now = fixture + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms().max(0) as u64 }) + .await?; + assert_eq!( + keys.verify_cell(response.token(), &a, route.schema_digest, now)?, + response + .local_receipt() + .expect("local command receipt") + .observation() + ); + drop(response); + wait_for(async || { + Ok(controllers[0].usage().retained_commands == 0 && dispatcher.usage().1 == 0) + }) + .await?; + // Successful, unpolled replies also retain node admission. + let unpolled = controllers[0].command( + invocation( + &dispatcher, + &class, + &b, + Duration::from_secs(15), + retry_until, + ) + .await?, + fixture.cell_options(&b).await?, + )?; + wait_for(async || Ok(dispatcher.usage().1 == 0 && pool.usage()?.active_cells == 0)) + .await?; + assert_eq!(controllers[0].usage().retained_commands, 1); + let refused = controllers[0] + .command( + invocation( + &dispatcher, + &class, + &a, + Duration::from_secs(15), + retry_until, + ) + .await?, + fixture.cell_options(&a).await?, + ) + .err() + .context("unpolled result admission")?; + assert!(matches!( + refused.error.cause.as_ref(), + CellDispatchError::Runtime(RuntimeError::Overloaded) + )); + drop(refused); + assert_eq!( + unpolled + .response() + .await? + .local_receipt() + .expect("local command receipt") + .version + .history, + b + ); + wait_for(async || Ok(controllers[0].usage().retained_commands == 0)).await?; + } + for controller in &controllers { + controller.close().await?; + } + drop(controllers); + fixture.close().await?; + } + Ok(()) +} + +pub(crate) async fn owned_command_failure(lost_reply: bool) -> Result<()> { + for rooted in [false, true] { + let root = tempfile::tempdir()?; + let directory = root + .path() + .join(if rooted { "rooted-0" } else { "node-1/images" }); + let faults = physical::Faults::new(directory, None); + let filesystem = Arc::new(physical::FaultFileSystem { + faults: faults.clone(), + }); + let calls = Arc::new(AtomicUsize::new(0)); + let count = calls.clone(); + let class = Arc::new(RaftClass::new("Counter", 1, [81; 32])?.command( + "Add", + [82; 32], + move |context, _| { + context.sql().execute("UPDATE value SET n=n+1", [])?; + if count.fetch_add(1, Ordering::SeqCst) == 0 { + if lost_reply { + // Application panics are contained by the transaction. + // Lose the worker reply during subsequent physical capture. + *faults.before_directory_sync.lock().unwrap() = Some(Box::new(|| { + panic!("injected lost SQL stage reply during image capture"); + })); + } else { + anyhow::bail!("injected callback refusal"); + } + } + Ok(b"retry-result".to_vec()) + }, + )?); + let (mut fixture, route, a, _) = + Box::pin(registered::setup_in(rooted, &class, root, filesystem)).await?; + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let pool = fixture.appliers[&(0, route.history)].0.clone(); + let dispatcher = CellGroupDispatcher::new( + &fixture.cluster.domain, + fixture.cluster.directory.clone(), + class.clone(), + VersionKeys::new( + VersionSigningKey::new([89; 32], u64::MAX)?, + vec![], + Duration::from_secs(30), + )?, + CellDispatchLimits { + max_cached_routes: 32, + route_cache_bytes: 256 * 1024, + max_invocations: 8, + retained_bytes: 1024 * 1024, + max_input_bytes: 1024, + }, + )?; + let retry_until = fixture + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms().max(0) as u64 + 60_000 }) + .await?; + let call = controllers[0].command( + invocation( + &dispatcher, + &class, + &a, + Duration::from_secs(15), + retry_until, + ) + .await?, + fixture.cell_options(&a).await?, + )?; + let error = call.response().await.err().context("failed stage")?; + assert!(error.ambiguous); + assert_eq!(calls.load(Ordering::SeqCst), 1); + if lost_reply { + assert!(matches!( + error.cause.as_ref(), + CellDispatchError::Execution(ExecutionError::Refused(CommandFailure::Unavailable)) + )); + assert!(matches!( + tokio::time::timeout(Duration::from_secs(5), controllers[0].wait()).await?, + Err(CellGroupControllerError::Command(_)) + )); + assert_eq!(pool.usage()?.running_workers, 0); + assert_eq!(controllers[0].usage().retained_commands, 0); + assert_eq!(dispatcher.usage().1, 0); + assert!(pool.close().await.is_err()); + fixture.appliers.remove(&(0, route.history)); + } else { + assert!(matches!( + error.cause.as_ref(), + CellDispatchError::Execution(ExecutionError::Storage(_)) + )); + wait_for(async || { + Ok(controllers[0].usage().retained_commands == 0 && dispatcher.usage().1 == 0) + }) + .await?; + assert!(!controllers[0].usage().closing); + assert_eq!(pool.usage()?.active_cells, 0); + let state = fixture.runtimes[0] + .reconciliation_authority(route.history, a.clone()) + .await?; + assert_eq!(state.cell().version.sequence.get(), 1); + let retried = controllers[0] + .command( + invocation( + &dispatcher, + &class, + &a, + Duration::from_secs(15), + retry_until, + ) + .await?, + fixture.cell_options(&a).await?, + )? + .response() + .await?; + assert_eq!( + retried + .local_receipt() + .expect("local command receipt") + .receipt + .result(), + b"retry-result" + ); + assert_eq!( + retried + .local_receipt() + .expect("local command receipt") + .version + .sequence + .get(), + 2 + ); + assert_eq!(calls.load(Ordering::SeqCst), 2); + drop(retried); + controllers[0].close().await?; + } + for controller in &controllers[1..] { + controller.close().await?; + } + drop(controllers); + drop(pool); + fixture.close().await?; + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_controller_enrollment.rs b/crates/cellar/tests/support/cell_group_controller_enrollment.rs new file mode 100644 index 000000000..f2a45970a --- /dev/null +++ b/crates/cellar/tests/support/cell_group_controller_enrollment.rs @@ -0,0 +1,480 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Registered creation on the shared controller, SQL pool and physical TLS peers. +use super::apply_faults::physical; +use super::*; +use std::sync::{ + atomic::{AtomicUsize, Ordering}, + Mutex, +}; + +async fn wait_for(mut predicate: impl AsyncFnMut() -> Result) -> Result<()> { + tokio::time::timeout(Duration::from_secs(15), async { + while !predicate().await? { + tokio::time::sleep(Duration::from_millis(2)).await; + } + Ok::<_, anyhow::Error>(()) + }) + .await? +} + +async fn enroll( + dispatcher: &CellGroupDispatcher, + cell: &CellHistory, + tag: u8, + timeout: Duration, +) -> Result { + Ok(dispatcher + .enroll( + cell.clone(), + CellEnrollmentRequest { + creation: ProposalId::new([tag; 16])?, + durability: Durability::Quorum, + }, + CellRoutingLimits { + timeout, + max_attempts: 4, + }, + None, + ) + .await?) +} + +fn dispatcher(fixture: &RuntimeCluster, class: Arc) -> Result { + Ok(CellGroupDispatcher::new( + &fixture.cluster.domain, + fixture.cluster.directory.clone(), + class, + VersionKeys::new( + VersionSigningKey::new([89; 32], u64::MAX)?, + vec![], + Duration::from_secs(30), + )?, + CellDispatchLimits { + max_cached_routes: 32, + route_cache_bytes: 256 * 1024, + max_invocations: 8, + retained_bytes: 1024 * 1024, + max_input_bytes: 1024, + }, + )?) +} + +// phase 0 holds initialization; phase 1 holds committed manifest publication. +// mode 0 disconnects; 1 expires the response; 2 cancels physical shutdown. +pub(crate) async fn owned_enrollment(phase: u8, mode: u8) -> Result<()> { + for rooted in [false, true] { + let root = tempfile::tempdir()?; + let faults = physical::Faults::new(root.path().join("node-1/pages"), None); + let filesystem = Arc::new(physical::FaultFileSystem { + faults: faults.clone(), + }); + let calls = Arc::new(AtomicUsize::new(0)); + let count = calls.clone(); + let seen = Arc::new(Mutex::new(Vec::new())); + let observed = seen.clone(); + let (entered, started) = tokio::sync::oneshot::channel(); + let entered = Arc::new(Mutex::new(Some(entered))); + let (release, held) = std::sync::mpsc::sync_channel(1); + let held = Arc::new(Mutex::new(held)); + let init_entered = entered.clone(); + let init_held = held.clone(); + let base = || RaftClass::new("Counter", 1, [81; 32]); + let missing = base()?; + let class = Arc::new( + base()? + .initializer([90; 32], move |cell, tx| { + let first = count.fetch_add(1, Ordering::SeqCst) == 0; + observed.lock().unwrap().push((cell.clone(), tx.inputs())); + if first && phase == 0 { + let _ = init_entered + .lock() + .unwrap() + .take() + .context("one initializer")? + .send(()); + init_held + .lock() + .unwrap() + .recv_timeout(Duration::from_secs(15))?; + } + tx.execute_batch("CREATE TABLE value(n); INSERT INTO value VALUES(7);")?; + Ok(()) + })? + .command("Add", [82; 32], |context, _| { + context.sql().execute("UPDATE value SET n=n+1", [])?; + Ok(b"updated".to_vec()) + })?, + ); + assert_ne!(missing.schema_digest(), class.schema_digest()); + assert!(base()?.initializer([0; 32], |_, _| Ok(())).is_err()); + assert!(base()? + .initializer([90; 32], |_, _| Ok(()))? + .initializer([90; 32], |_, _| Ok(())) + .is_err()); + let different = + base()? + .initializer([91; 32], |_, _| Ok(()))? + .command("Add", [82; 32], |_, _| Ok(vec![]))?; + assert_ne!(different.schema_digest(), class.schema_digest()); + let (mut fixture, route, existing, _) = + Box::pin(registered::setup_in(rooted, &class, root, filesystem)).await?; + let controllers = controller::start_with_limits( + &mut fixture, + CellGroupControllerLimits { + max_commands: 1, + ..controller::limits() + }, + ) + .await?; + let dispatcher = dispatcher(&fixture, class.clone())?; + let cell = history(&route, "new-enrollment")?; + let pool = fixture.appliers[&(0, route.history)].0.clone(); + let rejected = dispatcher + .enroll( + cell.clone(), + CellEnrollmentRequest { + creation: ProposalId::new([170; 16])?, + durability: Durability::S3, + }, + CellRoutingLimits { + timeout: Duration::from_secs(5), + max_attempts: 4, + }, + None, + ) + .await + .err() + .context("S3 refusal")?; + assert!(!rejected.ambiguous); + assert_eq!(calls.load(Ordering::SeqCst), 0); + if phase == 1 { + *faults.before_directory_sync.lock().unwrap() = Some(Box::new(move || { + let _ = entered + .lock() + .unwrap() + .take() + .expect("apply entered") + .send(()); + held.lock() + .unwrap() + .recv_timeout(Duration::from_secs(15)) + .expect("apply released"); + })); + } + let timeout = if mode == 1 { + Duration::from_millis(400) + } else { + Duration::from_secs(15) + }; + let call = controllers[0].enroll( + enroll(&dispatcher, &cell, 170, timeout).await?, + fixture.cell_options(&cell).await?, + )?; + tokio::time::timeout(Duration::from_secs(5), started).await??; + assert_eq!(calls.load(Ordering::SeqCst), 1); + assert_eq!(seen.lock().unwrap()[0].0, cell); + assert_eq!(pool.usage()?.active_cells, 1); + assert_eq!(dispatcher.usage().1, 1); + assert_eq!(controllers[0].usage().retained_commands, 1); + // Enrollment consumes the same controller budget as ordinary commands. + let retry_until = fixture + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms().max(0) as u64 + 60_000 }) + .await?; + let command = dispatcher + .command( + existing.clone(), + CellCommandRequest { + id: CommandId::new(ClientId::new([171; 16])?, 1)?, + contract: class.contract("Add").context("Add")?, + input: vec![], + durability: Durability::Quorum, + retry_until_ms: retry_until, + }, + CellRoutingLimits { + timeout: Duration::from_secs(15), + max_attempts: 4, + }, + None, + ) + .await?; + let refused = controllers[0] + .command(command, fixture.cell_options(&existing).await?) + .err() + .context("shared admission")?; + assert!(matches!( + refused.error.cause.as_ref(), + CellDispatchError::Runtime(RuntimeError::Overloaded) + )); + drop(refused); + if mode == 1 { + let error = call.response().await.err().context("response deadline")?; + assert!(error.ambiguous); + assert!(matches!( + error.cause.as_ref(), + CellDispatchError::Runtime(RuntimeError::DeadlineExceeded) + )); + } else { + drop(call); + } + if mode == 2 { + assert!( + tokio::time::timeout(Duration::from_millis(30), controllers[0].close()) + .await + .is_err() + ); + assert!(apply::pool(&fixture, 0).is_err()); + } + assert_eq!(controllers[0].usage().retained_commands, 1); + assert_eq!(pool.usage()?.active_cells, 1); + release.send(())?; + if mode == 2 { + tokio::time::timeout(Duration::from_secs(5), controllers[0].wait()).await??; + assert_eq!(pool.usage()?.running_workers, 0); + assert_eq!(pool.usage()?.active_cells, 0); + assert_eq!(controllers[0].usage().retained_commands, 0); + assert_eq!(dispatcher.usage().1, 0); + } else { + wait_for(async || { + Ok( + controllers[0].usage().retained_commands == 0 + && pool.usage()?.active_cells == 0, + ) + }) + .await?; + let state = fixture.runtimes[0] + .enrollment_state(route.history, cell.clone()) + .await?; + if phase == 0 && mode == 1 { + assert!(state.record().is_none()); + } else { + assert_eq!( + state.record().context("committed creation")?.creation(), + ProposalId::new([170; 16])? + ); + } + let original = controllers[0] + .enroll( + enroll(&dispatcher, &cell, 170, Duration::from_secs(15)).await?, + fixture.cell_options(&cell).await?, + )? + .response() + .await?; + let record = original.record().clone(); + drop(original); + assert_eq!( + calls.load(Ordering::SeqCst), + if phase == 0 && mode == 1 { 2 } else { 1 } + ); + wait_for(async || Ok(controllers[0].usage().retained_commands == 0)).await?; + let command = dispatcher + .command( + cell.clone(), + CellCommandRequest { + id: CommandId::new(ClientId::new([171; 16])?, 1)?, + contract: class.contract("Add").context("Add")?, + input: vec![], + durability: Durability::Quorum, + retry_until_ms: retry_until, + }, + CellRoutingLimits { + timeout: Duration::from_secs(15), + max_attempts: 4, + }, + None, + ) + .await?; + let updated = controllers[0] + .command(command, fixture.cell_options(&cell).await?)? + .response() + .await?; + assert_eq!( + updated + .local_receipt() + .expect("local command receipt") + .version + .sequence + .get(), + 2 + ); + drop(updated); + wait_for(async || Ok(controllers[0].usage().retained_commands == 0)).await?; + let duplicate = controllers[0].enroll( + enroll(&dispatcher, &cell, 170, Duration::from_secs(15)).await?, + fixture.cell_options(&cell).await?, + )?; + wait_for(async || Ok(pool.usage()?.active_cells == 0 && dispatcher.usage().1 == 0)) + .await?; + assert_eq!(controllers[0].usage().retained_commands, 1); // Unconsumed creation response remains charged. + let duplicate = duplicate.response().await?; + assert_eq!(duplicate.record(), &record); + drop(duplicate); + wait_for(async || Ok(controllers[0].usage().retained_commands == 0)).await?; + for incarnation in [false, true] { + let mut conflict = cell.clone(); + if incarnation { + conflict.incarnation = IncarnationId::new([172; 16])?; + } + let rejected = controllers[0] + .enroll( + enroll(&dispatcher, &conflict, 173, Duration::from_secs(15)).await?, + fixture.cell_options(&conflict).await?, + )? + .response() + .await + .err() + .context("creation conflict")?; + assert!(matches!( + rejected.cause.as_ref(), + CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::CommandConflict + )) + )); + wait_for(async || Ok(controllers[0].usage().retained_commands == 0)).await?; + } + assert_eq!( + calls.load(Ordering::SeqCst), + if phase == 0 && mode == 1 { 2 } else { 1 } + ); + assert!(!controllers[0].usage().closing); + controllers[0].close().await?; + } + for controller in &controllers[1..] { + controller.close().await?; + } + drop(controllers); + drop(pool); + fixture.close().await?; + } + Ok(()) +} + +pub(crate) async fn owned_enrollment_failure(lost_reply: bool) -> Result<()> { + for rooted in [false, true] { + let root = tempfile::tempdir()?; + let faults = physical::Faults::new( + root.path() + .join(if rooted { "rooted-0" } else { "node-1/images" }), + None, + ); + let filesystem = Arc::new(physical::FaultFileSystem { + faults: faults.clone(), + }); + let calls = Arc::new(AtomicUsize::new(0)); + let count = calls.clone(); + let class = Arc::new(RaftClass::new("Counter", 1, [81; 32])?.initializer( + [90; 32], + move |_, tx| { + tx.execute_batch("CREATE TABLE value(n); INSERT INTO value VALUES(7);")?; + if count.fetch_add(1, Ordering::SeqCst) == 0 { + if lost_reply { + *faults.before_directory_sync.lock().unwrap() = Some(Box::new(|| { + panic!("injected initialization capture reply loss") + })); + } else { + anyhow::bail!("injected initializer failure"); + } + } + Ok(()) + }, + )?); + let (mut fixture, route, _, _) = + Box::pin(registered::setup_in(rooted, &class, root, filesystem)).await?; + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let dispatcher = dispatcher(&fixture, class)?; + let cell = history(&route, "failed-enrollment")?; + let pool = fixture.appliers[&(0, route.history)].0.clone(); + let error = controllers[0] + .enroll( + enroll(&dispatcher, &cell, 174, Duration::from_secs(15)).await?, + fixture.cell_options(&cell).await?, + )? + .response() + .await + .err() + .context("failed initializer")?; + assert!(error.ambiguous); + assert_eq!(calls.load(Ordering::SeqCst), 1); + if lost_reply { + assert!(matches!( + error.cause.as_ref(), + CellDispatchError::Execution(ExecutionError::Refused(CommandFailure::Unavailable)) + )); + assert!(matches!( + tokio::time::timeout(Duration::from_secs(5), controllers[0].wait()).await?, + Err(CellGroupControllerError::Enrollment(_)) + )); + assert_eq!(pool.usage()?.running_workers, 0); + assert_eq!(controllers[0].usage().retained_commands, 0); + assert_eq!(dispatcher.usage().1, 0); + assert!(pool.close().await.is_err()); + fixture.appliers.remove(&(0, route.history)); + } else { + assert!(matches!( + error.cause.as_ref(), + CellDispatchError::Execution(ExecutionError::Storage(_)) + )); + wait_for(async || { + Ok(controllers[0].usage().retained_commands == 0 && dispatcher.usage().1 == 0) + }) + .await?; + assert!(!controllers[0].usage().closing); + assert_eq!(pool.usage()?.active_cells, 0); + assert!(fixture.runtimes[0] + .enrollment_state(route.history, cell.clone()) + .await? + .record() + .is_none()); + let result = controllers[0] + .enroll( + enroll(&dispatcher, &cell, 174, Duration::from_secs(15)).await?, + fixture.cell_options(&cell).await?, + )? + .response() + .await?; + assert_eq!(result.record().creation(), ProposalId::new([174; 16])?); + drop(result); + assert_eq!(calls.load(Ordering::SeqCst), 2); + controllers[0].close().await?; + } + for controller in &controllers[1..] { + controller.close().await?; + } + drop(controllers); + drop(pool); + fixture.close().await?; + } + Ok(()) +} + +pub(crate) async fn enrollment_requires_registration() -> Result<()> { + let class = Arc::new(RaftClass::new("Counter", 1, [81; 32])?); + let (fixture, route, _, _) = Box::pin(registered::setup(false, &class)).await?; + let dispatcher = dispatcher(&fixture, class)?; + let error = dispatcher + .enroll( + history(&route, "unregistered")?, + CellEnrollmentRequest { + creation: ProposalId::new([175; 16])?, + durability: Durability::Quorum, + }, + CellRoutingLimits { + timeout: Duration::from_secs(5), + max_attempts: 4, + }, + None, + ) + .await + .err() + .context("initializer required")?; + assert!(!error.ambiguous); + assert!(matches!( + error.cause.as_ref(), + CellDispatchError::Registration(RegistrationError::Invalid("missing initializer")) + )); + assert_eq!(dispatcher.usage().1, 0); + drop(dispatcher); + fixture.close().await +} diff --git a/crates/cellar/tests/support/cell_group_crash_fixture.rs b/crates/cellar/tests/support/cell_group_crash_fixture.rs new file mode 100644 index 000000000..1cd6ec99e --- /dev/null +++ b/crates/cellar/tests/support/cell_group_crash_fixture.rs @@ -0,0 +1,160 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Owned crash children and bounded synthetic catalog snapshots for native tests. +//! These objects are fixture inputs, never a production recovery protocol. +use super::*; +use anyhow::Context as _; +use futures_util::TryStreamExt; +use object_store::{path::Path as ObjectPath, ObjectStore}; +use std::{io::Write, path::Path, time::Duration}; + +struct Child(std::process::Child); + +impl Drop for Child { + fn drop(&mut self) { + let _ = self.0.kill(); + let _ = self.0.wait(); + } +} + +pub(super) async fn kill_at(root: &Path, fixture: &str, rooted: bool, cut: &str) -> Result<()> { + let output = root.join("child.log"); + let file = std::fs::File::create(&output)?; + let mut child = Child( + std::process::Command::new(std::env::current_exe()?) + .args([ + "--ignored", + "--exact", + fixture, + "--nocapture", + "--test-threads=1", + ]) + .env("CELLAR_GROUP_CRASH_ROOT", root) + .env( + "CELLAR_GROUP_CRASH_MODE", + if rooted { "extents" } else { "materialized" }, + ) + .env("CELLAR_GROUP_CRASH_CUT", cut) + .stdout(file.try_clone()?) + .stderr(file) + .spawn()?, + ); + let wait = tokio::time::timeout(Duration::from_secs(30), async { + loop { + let reached = match std::fs::read(root.join("signal")) { + Ok(bytes) => bytes == b"reached", + Err(error) if error.kind() == std::io::ErrorKind::NotFound => false, + Err(error) => return Err(error.into()), + }; + if reached { + break; + } + if let Some(status) = child.0.try_wait()? { + anyhow::bail!("child exited before {cut}: {status}"); + } + tokio::time::sleep(Duration::from_millis(5)).await; + } + Ok::<_, anyhow::Error>(()) + }) + .await; + let reached = wait.map_err(anyhow::Error::from).and_then(|r| r); + reached.with_context(|| { + format!( + "{rooted}/{cut}: {}", + std::fs::read_to_string(&output).unwrap_or_default() + ) + })?; + assert_eq!(std::fs::read(root.join("signal"))?, b"reached"); + child.0.kill()?; + let status = child.0.wait()?; + #[cfg(unix)] + { + use std::os::unix::process::ExitStatusExt; + assert_eq!(status.signal(), Some(9)); + } + assert!(!status.success()); + Ok(()) +} + +pub(super) fn save(path: &Path, bytes: &[u8]) -> Result<()> { + let mut file = std::fs::File::create(path)?; + file.write_all(bytes)?; + file.sync_all()?; + Ok(()) +} + +pub(super) async fn stop_at(root: &Path) -> Result<()> { + save(&root.join("signal"), b"reached")?; + std::future::pending::<()>().await; + unreachable!("parent kills the held child") +} + +pub(super) fn parameters() -> Result<(std::path::PathBuf, bool, String)> { + let root = std::path::PathBuf::from(std::env::var("CELLAR_GROUP_CRASH_ROOT")?); + let rooted = match std::env::var("CELLAR_GROUP_CRASH_MODE")?.as_str() { + "materialized" => false, + "extents" => true, + _ => anyhow::bail!("invalid crash image mode"), + }; + let cut = std::env::var("CELLAR_GROUP_CRASH_CUT")?; + anyhow::ensure!( + root.is_dir() && !root.join("signal").exists(), + "fresh owned parent directory required" + ); + Ok((root, rooted, cut)) +} + +pub(super) async fn capture_catalog(store: &dyn ObjectStore) -> Result)>> { + let mut objects = Vec::new(); + let listed = store.list(None).try_collect::>().await?; + anyhow::ensure!(listed.len() <= 128, "bounded fixture catalog"); + let mut bytes = 0; + for item in listed { + let body = store.get(&item.location).await?.bytes().await?.to_vec(); + bytes += body.len(); + anyhow::ensure!(bytes < 1024 * 1024, "bounded fixture catalog bytes"); + objects.push((item.location.to_string(), body)); + } + Ok(objects) +} + +pub(super) async fn restore_catalog( + objects: Vec<(String, Vec)>, + route: &GroupRoute, + domain: &ExecutionDomain, + image_storage: RaftImageStorage, +) -> Result<(RangeDirectory, CellGroupBirth)> { + let (directory, birth, _) = + restore_catalog_store(objects, route, domain, image_storage).await?; + Ok((directory, birth)) +} + +pub(super) async fn restore_catalog_store( + objects: Vec<(String, Vec)>, + route: &GroupRoute, + domain: &ExecutionDomain, + image_storage: RaftImageStorage, +) -> Result<( + RangeDirectory, + CellGroupBirth, + Arc, +)> { + let store = Arc::new(object_store::memory::InMemory::new()); + for (name, bytes) in objects { + store.put(&ObjectPath::from(name), bytes.into()).await?; + } + let catalog = Catalog::new(store.clone(), CatalogOptions::default())?; + let mut config = NamespaceConfig::new( + route.history.cluster, + route.history.namespace, + cellar_logic::NamespaceMode::Raft, + route.schema_digest, + ); + config.image_storage = image_storage; + let namespace = domain.run(catalog.existing_namespace(config)).await??; + let directory = RangeDirectory::new(namespace, DirectoryLimits::default())?; + let birth = domain + .run(directory.materialize_selected_group(route)) + .await??; + Ok((directory, birth, store)) +} diff --git a/crates/cellar/tests/support/cell_group_directory.rs b/crates/cellar/tests/support/cell_group_directory.rs new file mode 100644 index 000000000..daa03376e --- /dev/null +++ b/crates/cellar/tests/support/cell_group_directory.rs @@ -0,0 +1,1565 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Real conditional object-store operations with dropped responses. No fake +//! publication successes, voter claims or Raft authority are synthesized here. + +use super::*; +use anyhow::{Context, Result}; +use cellar::application::consensus::verification::{ + Catalog, CatalogError, CatalogOptions, InitialMember, NamespaceConfig, +}; +use cellar::sdk::ExecutionDomain; +use cellar_logic::consensus::{MemberId, NodeId}; +use object_store::{path::Path, ObjectStore}; +use std::{ + sync::{atomic::Ordering, Arc}, + time::Duration, +}; + +#[path = "cell_group_directory_transition.rs"] +mod transition; +pub(super) use transition::{seal_authorization_cancel_race, unknown_transition_authorization}; + +struct Fixture { + domain: ExecutionDomain, + store: Arc, + directory: RangeDirectory, + config: NamespaceConfig, + limits: DirectoryLimits, +} + +pub(super) async fn standby_initialization() -> Result<()> { + let f = Fixture::new(DirectoryLimits::default()).await?; + anyhow::ensure!(f.run(f.directory.observe()).await.is_err()); + anyhow::ensure!(f + .directory + .prepare_initialization(operation(72000), &[]) + .is_err()); + let intent = f + .directory + .prepare_standby_initialization(operation(72001))?; + let saved = intent.encode().to_vec(); + f.store.fail_head_after.store(true, Ordering::Release); + let accepted = f.run(f.directory.publish(&intent)).await?; + anyhow::ensure!(accepted.revision() == 1 && accepted.tree_digest().is_none()); + let reopened = f.reopen().await?; + let retry = reopened.decode_mutation(&saved)?; + anyhow::ensure!(f.run(reopened.publish(&retry)).await? == accepted); + let view = f.run(reopened.observe()).await?; + let page = f.run(reopened.group_page(&view, None, 16)).await?; + anyhow::ensure!(view.groups() == 0 && page.routes.is_empty() && page.next.is_none()); + anyhow::ensure!(matches!( + f.run(reopened.lookup(&view, &CellId::new("not-enrolled")?)) + .await, + Err(CatalogError::Conflict(_)) + )); + let head = f.key("head"); + let bytes = f.store.inner.get(&head).await?.bytes().await?; + f.store.inner.delete(&head).await?; + anyhow::ensure!(f.run(reopened.observe()).await.is_err()); + anyhow::ensure!(f.run(reopened.publish(&retry)).await.is_err()); + f.store.inner.put(&head, bytes.into()).await?; + anyhow::ensure!(f.run(reopened.observe()).await?.digest() == view.digest()); + anyhow::ensure!(f.store.listing_calls.load(Ordering::Acquire) == 0); + f.domain.shutdown().await; + Ok(()) +} + +pub(super) async fn standby_import() -> Result<()> { + let f = Fixture::new(DirectoryLimits::default()).await?; + let intent = f + .directory + .prepare_standby_initialization(operation(72200))?; + f.run(f.directory.publish(&intent)).await?; + let standby = f.run(f.directory.observe()).await?; + let route = + f.directory + .prepare_group_birth(birth_spec(&assigned(72201, CellRange::all(), 1)))?; + let first = f.directory.prepare_initialization_at( + &standby, + operation(72202), + std::slice::from_ref(route.route()), + )?; + let cut = f + .run(f.directory.freeze_group_births(operation(72203))) + .await?; + anyhow::ensure!(cut.is_empty() && cut.len() == 0); + let frozen = f.run(f.directory.observe()).await?; + anyhow::ensure!(frozen.revision() == 2 && frozen.groups() == 0); + anyhow::ensure!(f + .run(f.directory.group_page(&frozen, None, 16)) + .await? + .routes + .is_empty()); + anyhow::ensure!(matches!( + f.run(f.directory.publish(&first)).await, + Err(CatalogError::BirthsFrozen) + )); + let reopened = f.reopen().await?; + anyhow::ensure!(f + .run(reopened.freeze_group_births(operation(72203))) + .await? + .is_empty()); + anyhow::ensure!(f.run(reopened.observe()).await?.digest() == frozen.digest()); + f.domain.shutdown().await; + Ok(()) +} + +pub(super) async fn standby_first_ranges() -> Result<()> { + let f = Fixture::new(DirectoryLimits::default()).await?; + let empty = f + .directory + .prepare_standby_initialization(operation(72100))?; + let receipt = f.run(f.directory.publish(&empty)).await?; + let standby = f.run(f.directory.observe()).await?; + let mut routes = ranges(3); + f.reserve(&mut routes).await?; + let first = f + .directory + .prepare_initialization_at(&standby, operation(72101), &routes)?; + let loser = f + .directory + .prepare_initialization_at(&standby, operation(72102), &routes)?; + // Lose confirmation after the real root CAS. Reopen and retry the original + // canonical intent; neither a timeout nor another operation can rebase it. + let saved = first.encode().to_vec(); + f.store + .fail_head_confirmation + .store(true, Ordering::Release); + anyhow::ensure!(matches!( + f.run(f.directory.publish(&first)).await, + Err(CatalogError::Unconfirmed) + )); + f.store.fail_read.store(false, Ordering::Release); + let reopened = f.reopen().await?; + let accepted = f + .run(reopened.publish(&reopened.decode_mutation(&saved)?)) + .await?; + anyhow::ensure!(accepted.revision() == 2 && accepted.tree_digest().is_some()); + anyhow::ensure!(matches!( + f.run(f.directory.publish(&loser)).await, + Err(CatalogError::Conflict(_)) + )); + let active = f.run(reopened.observe()).await?; + anyhow::ensure!(active.groups() == 3 && active.digest() != standby.digest()); + anyhow::ensure!(reopened + .prepare_initialization_at(&active, operation(72103), &routes) + .is_err()); + // A standby receipt stays historical after ranges exist. Replaying its + // operation cannot remove or replace the selected partition. + anyhow::ensure!( + f.run(reopened.publish(&reopened.decode_mutation(empty.encode())?)) + .await? + == receipt + ); + anyhow::ensure!(f.run(reopened.observe()).await?.digest() == active.digest()); + let page = f.run(reopened.group_page(&active, None, 16)).await?; + anyhow::ensure!(page.routes == routes && page.next.is_none()); + anyhow::ensure!(f.store.listing_calls.load(Ordering::Acquire) == 0); + f.domain.shutdown().await; + Ok(()) +} + +pub(super) async fn group_import_freeze() -> Result<()> { + let f = Fixture::new(DirectoryLimits::default()).await?; + let mut selected = ranges(3); + f.reserve(&mut selected).await?; + let genesis = f + .directory + .prepare_initialization(operation(71000), &selected)?; + f.run(f.directory.publish(&genesis)).await?; + let before = f.run(f.directory.observe()).await?; + let mut children = split(&selected[1], 71001); + f.reserve(&mut children).await?; + let stale = + f.directory + .prepare_replacement(&before, operation(71002), &selected[1..2], &children)?; + let pending = + f.directory + .prepare_group_birth(birth_spec(&assigned(71003, CellRange::all(), 2)))?; + f.run(f.directory.reserve_group_birth(&pending)).await?; + let delayed = + f.directory + .prepare_group_birth(birth_spec(&assigned(71004, CellRange::all(), 2)))?; + f.store + .birth_inventory_cas_hang + .store(true, Ordering::Release); + f.store + .hold_next_birth_inventory_cas + .store(true, Ordering::Release); + let writer = f.run(f.directory.reserve_group_birth(&delayed)); + let freezer = async { + let reached = tokio::time::timeout(Duration::from_secs(5), async { + while !f.store.birth_inventory_cas_waiting.load(Ordering::Acquire) { + tokio::task::yield_now().await; + } + }) + .await; + // Lose the reply after the actual namespace freeze CAS. The original + // operation must reconcile; no successful CAS is synthesized. + f.store + .fail_birth_inventory_after + .store(true, Ordering::Release); + let frozen = if reached.is_ok() { + f.run(f.directory.freeze_group_births(operation(71005))) + .await + } else { + Err(CatalogError::Unconfirmed) + }; + f.store + .birth_inventory_cas_hang + .store(false, Ordering::Release); + f.store.birth_inventory_cas_resume.notify_waiters(); + frozen + }; + let (delayed_result, frozen) = tokio::join!(writer, freezer); + anyhow::ensure!(matches!(delayed_result, Err(CatalogError::BirthsFrozen))); + let frozen = frozen?; + anyhow::ensure!(frozen.len() == 6 && !frozen.is_empty()); + anyhow::ensure!(frozen.operation() == operation(71005)); + anyhow::ensure!(matches!( + f.run(f.directory.publish(&stale)).await, + Err(CatalogError::BirthsFrozen) + )); + let new = f + .directory + .prepare_group_birth(birth_spec(&assigned(71006, CellRange::all(), 2)))?; + anyhow::ensure!(matches!( + f.run(f.directory.reserve_group_birth(&new)).await, + Err(CatalogError::BirthsFrozen) + )); + anyhow::ensure!(matches!( + f.run(f.directory.freeze_group_births(operation(71007))) + .await, + Err(CatalogError::Conflict(_)) + )); + let after = f.run(f.directory.observe()).await?; + anyhow::ensure!(after.digest() != before.digest() && after.groups() == 3); + for route in &selected { + // Lazy selected births remain live; no cell is chosen to stand in for + // the physical group. Their voters still need real bootstrap/quorum. + let birth = f.run(f.directory.materialize_selected_group(route)).await?; + anyhow::ensure!(birth.route() == route); + } + let reopened = f.reopen().await?; + let retried = f + .run(reopened.freeze_group_births(operation(71005))) + .await?; + anyhow::ensure!(retried.len() == frozen.len() && retried.operation() == frozen.operation()); + anyhow::ensure!(f.store.listing_calls.load(Ordering::Acquire) == 0); + Ok(()) +} + +impl Fixture { + async fn new(limits: DirectoryLimits) -> Result { + Self::with_schema(limits, route().schema_digest).await + } + + async fn with_schema(limits: DirectoryLimits, schema: [u8; 32]) -> Result { + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let store = Arc::new(fault::Store::default()); + store.reject_listing.store(true, Ordering::Release); + let route = route(); + let config = NamespaceConfig::new( + route.history.cluster, + route.history.namespace, + cellar_logic::NamespaceMode::Raft, + schema, + ); + let catalog = Catalog::new(store.clone(), CatalogOptions::default())?; + let namespace = domain.run(catalog.open_namespace(config.clone())).await??; + let directory = RangeDirectory::new(namespace, limits)?; + Ok(Self { + domain, + store, + directory, + config, + limits, + }) + } + + async fn run(&self, work: F) -> F::Output { + self.domain + .run(work) + .await + .expect("fixture execution domain closed before joined shutdown") + } + + async fn reopen(&self) -> Result { + let catalog = Catalog::new(self.store.clone(), CatalogOptions::default())?; + Ok(RangeDirectory::new( + self.run(catalog.existing_namespace(self.config.clone())) + .await?, + self.limits, + )?) + } + + async fn reserve(&self, routes: &mut [GroupRoute]) -> Result<(), CatalogError> { + for route in routes { + let request = self.directory.prepare_group_birth(birth_spec(route))?; + self.run(self.directory.reserve_group_birth(&request)) + .await?; + *route = request.route().clone(); + } + Ok(()) + } + + fn key(&self, name: &str) -> Path { + Path::from(format!( + "namespaces/{}/cell-groups/{name}", + hex::encode(self.config.namespace.bytes()) + )) + } +} + +fn birth_spec(route: &GroupRoute) -> CellGroupBirthSpec { + CellGroupBirthSpec { + history: route.history, + range: route.range.clone(), + epoch: route.epoch, + creation: ProposalId::new(route.history.group.bytes()).unwrap(), + voters: (1..=3) + .map(|n| InitialMember { + member: MemberId::new(n).unwrap(), + node: NodeId::new([n as u8; 16]).unwrap(), + }) + .collect(), + learners: vec![], + } +} + +fn operation(n: u64) -> ProposalId { + let mut bytes = [0; 16]; + bytes[..8].copy_from_slice(&n.to_be_bytes()); + ProposalId::new(bytes).unwrap() +} + +fn assigned(id: u64, range: CellRange, epoch: u64) -> GroupRoute { + let mut r = route(); + r.history.group = GroupId::new(operation(id).bytes()).unwrap(); + r.history.incarnation = IncarnationId::new(operation(id + 10000).bytes()).unwrap(); + r.birth_digest[..8].copy_from_slice(&id.to_be_bytes()); + r.range = range; + r.epoch = RouteEpoch::new(epoch).unwrap(); + r +} + +fn ranges(count: usize) -> Vec { + let ns = route().history.namespace; + let mut keys: Vec<_> = (1..count) + .map(|i| cell_routing_key(ns, &CellId::new(format!("boundary-{i}")).unwrap())) + .collect(); + keys.sort(); + let mut lower = CellBound::NegativeInfinity; + let mut result = vec![]; + for (i, upper) in keys + .into_iter() + .map(CellBound::Key) + .chain([CellBound::PositiveInfinity]) + .enumerate() + { + result.push(assigned( + i as u64 + 1, + CellRange::new(lower, upper.clone()).unwrap(), + 1, + )); + lower = upper; + } + result +} + +fn split(parent: &GroupRoute, id: u64) -> Vec { + let key = (0..100_000) + .map(|i| { + cell_routing_key( + parent.history.namespace, + &CellId::new(format!("split-{id}-{i}")).unwrap(), + ) + }) + .find(|key| { + parent.range.contains(key) && CellBound::Key(key.clone()) != *parent.range.lower() + }) + .unwrap(); + let (left, right) = parent.range.split(key).unwrap(); + vec![ + assigned(id, left, parent.epoch.get() + 1), + assigned(id + 1, right, parent.epoch.get() + 1), + ] +} + +pub async fn inventory_pages() -> Result<()> { + let f = Fixture::new(DirectoryLimits { + max_cached_pages: 2, + max_cache_bytes: 2 * 64 * 1024, + ..DirectoryLimits::default() + }) + .await?; + let d = &f.directory; + let mut initial = ranges(7); + f.reserve(&mut initial).await?; + let init = d.prepare_initialization(operation(991), &initial)?; + f.run(d.publish(&init)).await?; + let view = f.run(d.observe()).await?; + for limit in [0, 129] { + assert!(f.run(d.group_page(&view, None, limit)).await.is_err()); + } + let page = f.run(d.group_page(&view, None, 2)).await?; + assert_eq!(page.routes, initial[..2]); + let first_cursor = page.next.context("seven groups require a second page")?; + + let mut replacement = split(&initial[0], 992); + f.reserve(&mut replacement).await?; + let mutation = d.prepare_replacement(&view, operation(993), &initial[..1], &replacement)?; + f.run(d.publish(&mutation)).await?; + let current = f.run(d.observe()).await?; + assert_eq!(current.groups(), 8); + assert!(f + .run(d.group_page(¤t, Some(&first_cursor), 2)) + .await + .is_err()); + let reopened = f.reopen().await?; + assert!(f.run(reopened.group_page(&view, None, 2)).await.is_err()); + + let mut routes = page.routes; + let mut cursor = Some(first_cursor); + while let Some(after) = cursor { + let page = f.run(d.group_page(&view, Some(&after), 2)).await?; + assert!(!page.routes.is_empty() && page.routes.len() <= 2); + routes.extend(page.routes); + cursor = page.next; + } + assert_eq!(routes, initial); + let page = f.run(d.group_page(¤t, None, 128)).await?; + let expected: Vec<_> = replacement + .into_iter() + .chain(initial.into_iter().skip(1)) + .collect(); + assert_eq!(page.routes, expected); + assert!(page.next.is_none()); + let usage = d.cache_usage()?; + assert!(usage.0 <= f.limits.max_cached_pages && usage.1 <= f.limits.max_cache_bytes); + assert_eq!(f.store.listing_calls.load(Ordering::Acquire), 0); + f.domain.shutdown().await; + Ok(()) +} + +pub async fn single_owner() -> Result<()> { + let f = Fixture::new(DirectoryLimits::default()).await?; + let d = &f.directory; + let mut initial = ranges(2); + f.reserve(&mut initial).await?; + let init = d.prepare_initialization(operation(1), &initial)?; + for bad in [ + vec![], + vec![initial[0].clone()], + vec![initial[1].clone(), initial[0].clone()], + vec![initial[0].clone(), initial[0].clone()], + ] { + assert!(d.prepare_initialization(operation(2), &bad).is_err()); + } + let mut duplicate = initial.clone(); + duplicate[1].history.group = duplicate[0].history.group; + assert!(d.prepare_initialization(operation(2), &duplicate).is_err()); + let mut wrong_epoch = initial.clone(); + wrong_epoch[0].epoch = RouteEpoch::new(2)?; + assert!(d + .prepare_initialization(operation(2), &wrong_epoch) + .is_err()); + f.run(d.publish(&init)).await?; + let view = f.run(d.observe()).await?; + let mut left = split(&initial[0], 100); + let mut right = split(&initial[1], 200); + f.reserve(&mut left).await?; + f.reserve(&mut right).await?; + let a = d.prepare_replacement(&view, operation(2), &initial[..1], &left)?; + let b = d.prepare_replacement(&view, operation(3), &initial[1..], &right)?; + let mut stale_parent = initial[0].clone(); + stale_parent.birth_digest = [22; 32]; + let stale = d.prepare_replacement(&view, operation(4), &[stale_parent], &left)?; + assert!(f.run(d.publish(&stale)).await.is_err()); + f.store.range_before.armed.store(true, Ordering::Release); + let entered = f.store.range_before.entered.notified(); + let (loser, winner) = f + .run(cellar::asyncrt::timeout(Duration::from_secs(15), async { + tokio::join!(d.publish(&a), async { + entered.await; + let result = d.publish(&b).await; + f.store.range_before.resume.notify_one(); + result + }) + })) + .await?; + assert!(matches!(loser, Err(CatalogError::Conflict(_)))); + assert_eq!(winner?.revision(), 2); + let after = f.run(d.observe()).await?; + assert_eq!(after.groups(), 3); + assert_eq!(view.groups(), 2); + assert!(matches!( + f.run(d.publish(&a)).await, + Err(CatalogError::Conflict(_)) + )); + // Previously assigned group IDs stay reserved even after range replacement. + let mut reused = assigned(201, initial[0].range.clone(), 2); + reused.history.incarnation = IncarnationId::new([42; 16])?; + let reused = d.prepare_replacement(&after, operation(5), &initial[..1], &[reused])?; + assert!(matches!( + f.run(d.publish(&reused)).await, + Err(CatalogError::Conflict(_)) + )); + for i in 0..100 { + let cell = CellId::new(format!("check-{i}"))?; + let key = cell_routing_key(f.config.namespace, &cell); + let old = initial.iter().find(|r| r.range.contains(&key)).unwrap(); + let expected = if old == &initial[0] { + old + } else { + right.iter().find(|r| r.range.contains(&key)).unwrap() + }; + assert_eq!(&f.run(d.lookup(&after, &cell)).await?, expected); + assert_eq!(&f.run(d.lookup(&view, &cell)).await?, old); + } + assert_eq!(f.store.listing_calls.load(Ordering::Acquire), 0); + f.domain.shutdown().await; + Ok(()) +} + +pub async fn bounded_discovery() -> Result<()> { + let limits = DirectoryLimits { + max_cached_pages: 4, + max_cache_bytes: 4 * 64 * 1024, + operation_timeout_ms: 30_000, + ..Default::default() + }; + let f = Fixture::new(limits).await?; + let d = &f.directory; + let mut initial = ranges(128); + f.reserve(&mut initial).await?; + let init = d.prepare_initialization(operation(1), &initial)?; + f.run(d.publish(&init)).await?; + let view = f.run(d.observe()).await?; + for i in 0..300 { + let cell = CellId::new(format!("lookup-{i}"))?; + let (found, reads) = fault::count_reads(f.run(d.lookup(&view, &cell))).await; + assert!( + reads.values().sum::() <= 8, + "lookup must read one AVL path: {reads:?}" + ); + assert!(reads.keys().all(|k| k.contains("/cell-groups/ranges/"))); + let key = cell_routing_key(f.config.namespace, &cell); + assert_eq!( + &found?, + initial.iter().find(|r| r.range.contains(&key)).unwrap() + ); + let (pages, bytes) = d.cache_usage()?; + assert!(pages <= 4 && bytes <= limits.max_cache_bytes); + } + let parent = &initial[63]; + let mut children = split(parent, 1000); + f.reserve(&mut children).await?; + let mutation = + d.prepare_replacement(&view, operation(2), std::slice::from_ref(parent), &children)?; + let before = f.store.range_page_puts.load(Ordering::Acquire); + f.run(d.publish(&mutation)).await?; + let pages = f.store.range_page_puts.load(Ordering::Acquire) - before; + assert!( + pages <= 64, + "one split rewrote {pages} pages for 128 groups" + ); + let after = f.run(d.observe()).await?; + assert_eq!(after.groups(), 129); + let mut merged = assigned(2000, parent.range.clone(), 3); + f.reserve(std::slice::from_mut(&mut merged)).await?; + let merge = d.prepare_replacement( + &after, + operation(3), + &children, + std::slice::from_ref(&merged), + )?; + f.run(d.publish(&merge)).await?; + let final_view = f.run(d.observe()).await?; + assert_eq!(final_view.groups(), 128); + let catalog = Catalog::new(f.store.clone(), CatalogOptions::default())?; + let namespace = f.run(catalog.existing_namespace(f.config.clone())).await?; + let warm = RangeDirectory::new(namespace, DirectoryLimits::default())?; + let warm_view = f.run(warm.observe()).await?; + let cell = CellId::new("warm-cell")?; + f.run(warm.lookup(&warm_view, &cell)).await?; + let (result, reads) = fault::count_reads(f.run(warm.lookup(&warm_view, &cell))).await; + result?; + assert!( + reads.is_empty(), + "cached route lookup must not refresh S3 metadata" + ); + assert_eq!(f.store.listing_calls.load(Ordering::Acquire), 0); + f.domain.shutdown().await; + Ok(()) +} + +pub async fn recovery() -> Result<()> { + let f = Fixture::new(DirectoryLimits::default()).await?; + let d = &f.directory; + let mut initial = ranges(1); + f.reserve(&mut initial).await?; + let init = d.prepare_initialization(operation(1), &initial)?; + f.run(d.publish(&init)).await?; + let view = f.run(d.observe()).await?; + let mut children = split(&initial[0], 100); + f.reserve(&mut children).await?; + let first = d.prepare_replacement(&view, operation(2), &initial, &children)?; + f.store.fail_head_after.store(true, Ordering::Release); + f.store + .fail_head_confirmation + .store(true, Ordering::Release); + assert!(matches!( + f.run(d.publish(&first)).await, + Err(CatalogError::Unconfirmed) + )); + let restarted = f.reopen().await?; + let retry = restarted.decode_mutation(first.encode())?; + let selected = f.run(restarted.observe()).await?; + assert_eq!(selected.revision(), 2); + let mut second_children = split(&children[0], 200); + f.reserve(&mut second_children).await?; + let second = + restarted.prepare_replacement(&selected, operation(3), &children[..1], &second_children)?; + let latest_publication = f.run(restarted.publish(&second)).await?; + let journal_key = f.key(&format!( + "journal/{}", + hex::encode(latest_publication.journal_digest()) + )); + assert!( + fault::missing_read(journal_key.to_string(), f.run(restarted.observe())) + .await + .is_err(), + "a head without its selected journal cannot be observed as valid" + ); + let receipt = f.run(restarted.publish(&retry)).await?; + assert_eq!( + receipt.revision(), + 2, + "retry finds its own publication after a later CAS" + ); + assert_eq!(f.run(restarted.publication(&retry)).await?, Some(receipt)); + let latest = f.run(restarted.observe()).await?; + let reused = restarted.prepare_replacement( + &latest, + operation(2), + &children[1..], + &split(&children[1], 300), + )?; + assert!(matches!( + f.run(restarted.publish(&reused)).await, + Err(CatalogError::Conflict(_)) + )); + assert!( + restarted.lookup(&view, &CellId::new("x")?).await.is_err(), + "old process view cannot attach to a new owner" + ); + // Cold discovery must verify the selected immutable page, not just its name. + let cold = f.reopen().await?; + let cold_view = f.run(cold.observe()).await?; + let cell = CellId::new("corruption-probe")?; + let (result, reads) = fault::count_reads(f.run(cold.lookup(&cold_view, &cell))).await; + result?; + let path = reads + .keys() + .find(|k| k.contains("/ranges/")) + .context("range read")? + .clone(); + let original = f + .store + .inner + .get(&Path::from(path.clone())) + .await? + .bytes() + .await?; + let mut corrupt = original.to_vec(); + *corrupt.last_mut().unwrap() ^= 1; + f.store + .inner + .put(&Path::from(path.clone()), corrupt.into()) + .await?; + let fresh = f.reopen().await?; + let fresh_view = f.run(fresh.observe()).await?; + assert!(f.run(fresh.lookup(&fresh_view, &cell)).await.is_err()); + f.store + .inner + .put(&Path::from(path.clone()), original.into()) + .await?; + assert!( + fault::missing_read(path, f.run(fresh.lookup(&fresh_view, &cell))) + .await + .is_err() + ); + f.store.inner.delete(&f.key("head")).await?; + assert!(f.run(restarted.observe()).await.is_err()); + assert!( + f.run(restarted.publish(&init)).await.is_err(), + "missing occupied metadata cannot reinitialize" + ); + assert!(matches!( + f.store.inner.get(&f.key("head")).await, + Err(object_store::Error::NotFound { .. }) + )); + f.domain.shutdown().await; + Ok(()) +} + +pub async fn genesis_race() -> Result<()> { + let f = Fixture::new(DirectoryLimits::default()).await?; + let d = &f.directory; + let mut initial = ranges(1); + f.reserve(&mut initial).await?; + let init = d.prepare_initialization(operation(1), &initial)?; + f.store + .range_genesis_after + .armed + .store(true, Ordering::Release); + let entered = f.store.range_genesis_after.entered.notified(); + let (first, second) = f + .run(cellar::asyncrt::timeout(Duration::from_secs(15), async { + tokio::join!(d.publish(&init), async { + entered.await; + let result = async { + let receipt = d.publish(&init).await?; + let view = d.observe().await?; + let mut children = split(&initial[0], 100); + f.reserve(&mut children).await?; + let replacement = + d.prepare_replacement(&view, operation(2), &initial, &children)?; + d.publish(&replacement).await?; + Ok::<_, CatalogError>(receipt) + } + .await; + f.store.range_genesis_after.resume.notify_one(); + result + }) + })) + .await?; + assert_eq!(first?, second?); + assert_eq!(f.run(d.observe()).await?.revision(), 2); + f.domain.shutdown().await; + Ok(()) +} + +pub async fn churn() -> Result<()> { + let f = Fixture::new(DirectoryLimits { + max_cached_pages: 3, + max_cache_bytes: 3 * 64 * 1024, + operation_timeout_ms: 30_000, + ..Default::default() + }) + .await?; + let d = &f.directory; + let mut initial = ranges(16); + f.reserve(&mut initial).await?; + let init = d.prepare_initialization(operation(1), &initial)?; + f.run(d.publish(&init)).await?; + let original_view = f.run(d.observe()).await?; + // The oracle is an independently maintained flat partition. Alternate + // splitting and merging different interior ranges so deletions exercise + // successor replacement and rotations, rather than undoing the last split. + let mut expected = initial.clone(); + for step in 0..80_usize { + let view = f.run(d.observe()).await?; + let position = (step * 7 + 3) % (expected.len() - 1); + let id = 1000 + step as u64 * 10; + let (parents, mut children) = if step % 2 == 0 { + let parent = expected[position].clone(); + let children = split(&parent, id); + (vec![parent], children) + } else { + let parents = expected[position..position + 2].to_vec(); + let range = parents[0].range.merge(&parents[1].range)?; + let epoch = parents.iter().map(|r| r.epoch.get()).max().unwrap() + 1; + (parents, vec![assigned(id, range, epoch)]) + }; + f.reserve(&mut children).await?; + let mutation = + d.prepare_replacement(&view, operation(step as u64 + 2), &parents, &children)?; + let receipt = f.run(d.publish(&mutation)).await?; + expected.splice(position..position + parents.len(), children); + let after = f.run(d.observe()).await?; + assert_eq!(after.revision(), step as u64 + 2); + assert_eq!(after.groups() as usize, expected.len()); + assert_eq!(f.run(d.publication(&mutation)).await?, Some(receipt)); + // Probe every boundary itself (half-open ownership) plus independent + // cell keys that are not selected from the tree under test. + let probes = expected + .iter() + .filter_map(|r| match r.range.lower() { + CellBound::Key(key) => Some(key.cell().clone()), + _ => None, + }) + .chain((0..32).map(|i| CellId::new(format!("churn-{i}")).unwrap())); + for cell in probes { + let key = cell_routing_key(f.config.namespace, &cell); + let owners: Vec<_> = expected.iter().filter(|r| r.range.contains(&key)).collect(); + assert_eq!(owners.len(), 1); + assert_eq!(&f.run(d.lookup(&after, &cell)).await?, owners[0]); + let old = initial.iter().find(|r| r.range.contains(&key)).unwrap(); + assert_eq!(&f.run(d.lookup(&original_view, &cell)).await?, old); + } + assert!(d.cache_usage()?.0 <= 3); + } + let restarted = f.reopen().await?; + assert_eq!(f.run(restarted.observe()).await?.revision(), 81); + assert_eq!(f.run(restarted.publish(&init)).await?.revision(), 1); + assert_eq!(f.store.listing_calls.load(Ordering::Acquire), 0); + f.domain.shutdown().await; + Ok(()) +} + +pub async fn birth_recovery() -> Result<()> { + let f = Fixture::new(DirectoryLimits::default()).await?; + let d = &f.directory; + let request = d.prepare_group_birth(birth_spec(&ranges(1)[0]))?; + let route = request.route(); + let birth_key = f.key(&format!( + "births/{}/{}", + hex::encode(route.history.group.bytes()), + hex::encode(route.history.incarnation.bytes()) + )); + let init = d.prepare_initialization(operation(99), std::slice::from_ref(route))?; + assert!(matches!( + f.run(d.publish(&init)).await, + Err(CatalogError::Corrupt(_)) + )); + assert!(matches!( + f.store.inner.get(&f.key("head")).await, + Err(object_store::Error::NotFound { .. }) + )); + f.run(d.reserve_group_birth(&request)).await?; + assert!( + f.run(d.materialize_group_birth(&request)).await.is_err(), + "an unselected reservation is not a birth" + ); + f.run(d.publish(&init)).await?; + assert!( + matches!( + f.store.inner.get(&birth_key).await, + Err(object_store::Error::NotFound { .. }) + ), + "range publication must not eagerly materialize groups" + ); + let unknown = fault::missing_read( + birth_key.to_string(), + f.run(d.materialize_group_birth(&request)), + ) + .await; + assert!(matches!(unknown, Err(CatalogError::Unconfirmed))); + assert_eq!( + f.store.inner.get(&birth_key).await?.bytes().await?.as_ref(), + request.encode(), + "unknown confirmation follows actual immutable creation" + ); + let restarted = f.reopen().await?; + let retry = restarted.decode_group_birth(request.encode())?; + assert_eq!(retry.route(), request.route()); + let birth = f.run(restarted.materialize_group_birth(&retry)).await?; + assert_eq!(birth.route(), route); + let selected = f + .run(restarted.materialize_selected_history(route.history)) + .await?; + assert_eq!(selected.route(), route); + let mut wrong = route.history; + wrong.incarnation = IncarnationId::new([77; 16])?; + assert!(matches!( + f.run(restarted.materialize_selected_history(wrong)).await, + Err(CatalogError::Fenced) + )); + wrong = route.history; + wrong.namespace = NamespaceId::new([77; 16])?; + assert!(matches!( + f.run(restarted.materialize_selected_history(wrong)).await, + Err(CatalogError::Conflict(_)) + )); + + assert_eq!(birth.creation(), request.creation()); + assert_eq!(birth.voters(), request.voters()); + assert!(birth.learners().is_empty()); + f.run(restarted.require_current_group_birth(&birth)).await?; + assert!(matches!( + f.run(d.require_current_group_birth(&birth)).await, + Err(CatalogError::Conflict(_)) + )); + assert!(matches!( + fault::missing_read( + birth_key.to_string(), + f.run(restarted.require_current_group_birth(&birth)) + ) + .await, + Err(CatalogError::Corrupt(_)) + )); + let mut corrupt = request.encode().to_vec(); + *corrupt.last_mut().unwrap() ^= 1; + f.store.inner.put(&birth_key, corrupt.into()).await?; + assert!(matches!( + f.run(restarted.require_current_group_birth(&birth)).await, + Err(CatalogError::Corrupt(_)) + )); + f.store + .inner + .put(&birth_key, request.encode().to_vec().into()) + .await?; + let view = f.run(restarted.observe()).await?; + let mut children = split(route, 700); + f.reserve(&mut children).await?; + let replace = restarted.prepare_replacement( + &view, + operation(100), + std::slice::from_ref(route), + &children, + )?; + f.run(restarted.publish(&replace)).await?; + assert!(matches!( + f.run(restarted.require_current_group_birth(&birth)).await, + Err(CatalogError::Fenced) + )); + assert!(matches!( + f.run(restarted.materialize_group_birth(&retry)).await, + Err(CatalogError::Fenced) + )); + assert!(matches!( + f.run(restarted.materialize_selected_history(route.history)) + .await, + Err(CatalogError::Fenced) + )); + assert_eq!( + f.store.inner.get(&birth_key).await?.bytes().await?.as_ref(), + request.encode() + ); + assert_eq!(f.store.listing_calls.load(Ordering::Acquire), 0); + f.domain.shutdown().await; + Ok(()) +} + +pub async fn birth_reservations() -> Result<()> { + let f = Fixture::new(DirectoryLimits::default()).await?; + let d = &f.directory; + let route = ranges(1).remove(0); + let original = d.prepare_group_birth(birth_spec(&route))?; + assert_eq!(&original.encode()[..9], b"CLRCAT01\xba"); + let mut wrong_domain = original.encode().to_vec(); + wrong_domain[8] = 86; + assert!(d.decode_group_birth(&wrong_domain).is_err()); + let mut unordered = birth_spec(&route); + unordered.voters.reverse(); + assert_eq!( + d.prepare_group_birth(unordered)?.encode(), + original.encode() + ); + let key = Path::from(format!( + "groups/{}", + hex::encode(route.history.group.bytes()) + )); + assert!(matches!( + fault::missing_read(key.to_string(), f.run(d.reserve_group_birth(&original))).await, + Err(CatalogError::Unconfirmed) + )); + let reservation = f.store.inner.get(&key).await?.bytes().await?; + let payload = serde_json::to_vec(&serde_json::json!({ + "birth": original.encode(), + "recovery": null, + }))?; + let end = 13 + payload.len(); + assert_eq!(reservation.len(), end + 32); + assert_eq!(&reservation[..9], b"CLRCAT01\xc2"); + assert_eq!(&reservation[9..13], &(payload.len() as u32).to_be_bytes()); + assert_eq!(&reservation[13..end], payload.as_slice()); + assert_eq!( + &reservation[end..], + &Sha256::digest(&reservation[..end])[..] + ); + // A bare birth is no longer a global reservation. The required origin + // wrapper must not be synthesized while reading an older record. + f.store + .inner + .put(&key, original.encode().to_vec().into()) + .await?; + assert!(matches!( + f.run(d.reserve_group_birth(&original)).await, + Err(CatalogError::Conflict(_)) + )); + f.store.inner.put(&key, reservation.clone().into()).await?; + f.run(d.reserve_group_birth(&original)).await?; + for which in 0..4 { + let mut spec = birth_spec(&route); + match which { + 0 => spec.creation = operation(555), + 1 => spec.history.incarnation = IncarnationId::new([55; 16])?, + 2 => spec.voters[0].node = NodeId::new([55; 16])?, + _ => spec.epoch = RouteEpoch::new(2)?, + } + let changed = d.prepare_group_birth(spec)?; + assert_ne!(changed.encode(), original.encode()); + assert!(matches!( + f.run(d.reserve_group_birth(&changed)).await, + Err(CatalogError::Conflict(_)) + )); + } + let mut duplicate = birth_spec(&route); + duplicate.voters[1].node = duplicate.voters[0].node; + assert!(d.prepare_group_birth(duplicate).is_err()); + let mut wrong_count = birth_spec(&route); + wrong_count.voters.pop(); + assert!(d.prepare_group_birth(wrong_count).is_err()); + let mut oversized = birth_spec(&route); + oversized.learners = vec![oversized.voters[0]; 33]; + assert!(matches!( + d.prepare_group_birth(oversized), + Err(CatalogError::Limit) + )); + for end in 0..original.encode().len() { + assert!(d.decode_group_birth(&original.encode()[..end]).is_err()); + } + assert!(d.decode_group_birth(&vec![0; 64 * 1024 + 1]).is_err()); + let mut config = f.config.clone(); + config.namespace = NamespaceId::new([42; 16])?; + let catalog = Catalog::new(f.store.clone(), CatalogOptions::default())?; + let foreign = RangeDirectory::new( + f.run(catalog.open_namespace(config.clone())).await?, + DirectoryLimits::default(), + )?; + assert!(matches!( + foreign.decode_group_birth(original.encode()), + Err(CatalogError::Conflict(_)) + )); + let mut spec = birth_spec(&route); + spec.history.namespace = config.namespace; + let competing = foreign.prepare_group_birth(spec)?; + assert!(matches!( + f.run(foreign.reserve_group_birth(&competing)).await, + Err(CatalogError::Conflict(_)) + )); + f.run(d.reserve_group_birth(&original)).await?; + // The original reservation survives every conflicting attempt unchanged. + assert_eq!( + f.store.inner.get(&key).await?.bytes().await?.as_ref(), + reservation.as_ref() + ); + // Configuring a managed policy does not authorize the unmanaged APIs. No + // fleet permit is fabricated, and refusal must precede any store access. + config.namespace = NamespaceId::new([43; 16])?; + config.managed_fleet = Some(serde_json::from_value(serde_json::json!({ + "fleet": vec![44; 16], "catalog": vec![45; 32], + }))?); + let managed = RangeDirectory::new( + f.run(catalog.open_namespace(config.clone())).await?, + DirectoryLimits::default(), + )?; + let mut spec = birth_spec(&route); + spec.history.namespace = config.namespace; + assert!(matches!( + managed.prepare_group_birth(spec), + Err(CatalogError::ManagedAdmissionRequired) + )); + let (result, reads) = fault::count_reads(f.run(managed.reserve_group_birth(&original))).await; + assert!(matches!( + result, + Err(CatalogError::ManagedAdmissionRequired) + )); + assert!(reads.is_empty()); + assert!(matches!( + f.run(managed.materialize_group_birth(&original)).await, + Err(CatalogError::ManagedAdmissionRequired) + )); + assert_eq!(f.store.listing_calls.load(Ordering::Acquire), 0); + f.domain.shutdown().await; + Ok(()) +} + +fn member_options( + path: &std::path::Path, + node: u8, +) -> cellar::application::consensus::verification::ReplicaLogOptions { + use cellar::application::consensus::verification::{LogOpenMode, ReplicaLogOptions}; + use cellar_logic::consensus::{DiskLineage, ProcessGeneration}; + ReplicaLogOptions::new( + path.to_owned(), + NodeId::new([node; 16]).unwrap(), + DiskLineage::new([node + 10; 16]).unwrap(), + ProcessGeneration::new([1; 16]).unwrap(), + LogOpenMode::Create, + ) +} + +async fn selected_birth(f: &Fixture) -> Result { + let d = &f.directory; + let request = d.prepare_group_birth(birth_spec(&ranges(1)[0]))?; + f.run(d.reserve_group_birth(&request)).await?; + let initial = + d.prepare_initialization(operation(999), std::slice::from_ref(request.route()))?; + f.run(d.publish(&initial)).await?; + Ok(f.run(d.materialize_group_birth(&request)).await?) +} + +pub async fn member_bootstrap() -> Result<()> { + use cellar::application::consensus::verification::{LogOpenMode, ReplicaLog}; + use cellar_logic::consensus::ProcessGeneration; + let f = Fixture::new(DirectoryLimits::default()).await?; + let birth = selected_birth(&f).await?; + let temp = tempfile::tempdir()?; + let mut options = member_options(&temp.path().join("member"), 1); + let log = f + .run(ReplicaLog::open( + options.clone(), + Arc::new(cellar_ltx::DirectFileSystem), + )) + .await?; + let history = birth.route().history; + let claim_key = f.key(&format!( + "claims/{}/{}/0000000000000001", + hex::encode(history.group.bytes()), + hex::encode(history.incarnation.bytes()) + )); + let result = fault::missing_read( + claim_key.to_string(), + f.run(f.directory.bootstrap_group_member(&birth, &log)), + ) + .await; + assert!(matches!(result, Err(CatalogError::Unconfirmed))); + let group = log.cell_group(&history)?; + assert_eq!(group.group_manifest()?.unwrap().route(), birth.route()); + assert_eq!(group.group_manifest()?.unwrap().root().cells(), 0); + let key = group.numeric_key(); + let claim = f.store.inner.get(&claim_key).await?.bytes().await?; + assert_eq!(&claim[..9], b"CLRCAT01\xbb"); + let original_engine = log.storage_instance()?; + log.close().await?; + options.mode = LogOpenMode::Resume; + options.process = ProcessGeneration::new([9; 16])?; + let log = f + .run(ReplicaLog::open( + options.clone(), + Arc::new(cellar_ltx::DirectFileSystem), + )) + .await?; + let reopened = f.reopen().await?; + // An observation from a different catalog owner never grants admission. + assert!(matches!( + f.run(reopened.bootstrap_group_member(&birth, &log)).await, + Err(CatalogError::Conflict(_)) + )); + let request = reopened.prepare_group_birth(birth_spec(birth.route()))?; + let current = f.run(reopened.materialize_group_birth(&request)).await?; + let permit = f + .run(reopened.bootstrap_group_member(¤t, &log)) + .await?; + f.run(permit.revalidate()).await?; + assert_eq!(permit.group().numeric_key(), key); + assert_eq!(permit.group().identity().scope, history); + assert_eq!(permit.group().identity().process, options.process); + assert_eq!(log.storage_instance()?, original_engine); + assert_eq!(f.store.inner.get(&claim_key).await?.bytes().await?, claim); + // Corrupt claims are not overwritten or interpreted as absent. + f.store + .inner + .put(&claim_key, b"not-a-claim".to_vec().into()) + .await?; + assert!(matches!( + f.run(permit.revalidate()).await, + Err(CatalogError::Conflict(_)) + )); + f.store.inner.put(&claim_key, claim.clone().into()).await?; + drop(permit); + log.close().await?; + // Remove only this owned fixture's group state, preserving the actual + // engine instance and permanent node pin. This exercises the member claim + // refusal independently of the earlier node-storage pin check. + let dir = options.directory.join("log").to_string_lossy().into_owned(); + f.domain + .blocking(move || -> Result<()> { + let engine = raft_engine::Engine::open(raft_engine::Config { + dir, + ..Default::default() + })?; + let mut catalog_key = b"g:".to_vec(); + catalog_key.extend_from_slice(&history.group.bytes()); + catalog_key.extend_from_slice(&history.incarnation.bytes()); + let mut batch = raft_engine::LogBatch::default(); + batch.delete(0, catalog_key); + batch.add_command(key, raft_engine::Command::Clean); + engine.write(&mut batch, true)?; + Ok(()) + })? + .await??; + let log = f + .run(ReplicaLog::open( + options, + Arc::new(cellar_ltx::DirectFileSystem), + )) + .await?; + assert_eq!(log.storage_instance()?, original_engine); + assert!(log.cell_groups()?.is_empty()); + assert!(matches!( + f.run(reopened.bootstrap_group_member(¤t, &log)).await, + Err(CatalogError::LostStableState) + )); + assert!(log.cell_groups()?.is_empty()); + log.close().await?; + let blank = f + .run(ReplicaLog::open( + member_options(&temp.path().join("new-path"), 1), + Arc::new(cellar_ltx::DirectFileSystem), + )) + .await?; + assert!(matches!( + f.run(reopened.bootstrap_group_member(¤t, &blank)) + .await, + Err(CatalogError::LostStableState) + )); + assert!(blank.cell_groups()?.is_empty()); + blank.close().await?; + assert_eq!(f.store.inner.get(&claim_key).await?.bytes().await?, claim); + f.domain.shutdown().await; + Ok(()) +} + +pub async fn member_fencing() -> Result<()> { + use cellar::application::consensus::verification::ReplicaLog; + let f = Fixture::new(DirectoryLimits::default()).await?; + let birth = selected_birth(&f).await?; + let temp = tempfile::tempdir()?; + let foreign = f + .run(ReplicaLog::open( + member_options(&temp.path().join("foreign"), 9), + Arc::new(cellar_ltx::DirectFileSystem), + )) + .await?; + assert!(matches!( + f.run(f.directory.bootstrap_group_member(&birth, &foreign)) + .await, + Err(CatalogError::NotInitialMember) + )); + assert!(foreign.cell_groups()?.is_empty()); + foreign.close().await?; + let log = f + .run(ReplicaLog::open( + member_options(&temp.path().join("member"), 1), + Arc::new(cellar_ltx::DirectFileSystem), + )) + .await?; + let permit = f + .run(f.directory.bootstrap_group_member(&birth, &log)) + .await?; + let view = f.run(f.directory.observe()).await?; + let mut children = split(birth.route(), 700); + f.reserve(&mut children).await?; + let replace = f.directory.prepare_replacement( + &view, + operation(1000), + std::slice::from_ref(birth.route()), + &children, + )?; + f.run(f.directory.publish(&replace)).await?; + assert!(matches!( + f.run(permit.revalidate()).await, + Err(CatalogError::Fenced) + )); + assert_eq!(log.cell_groups()?.len(), 1); + log.close().await?; + f.domain.shutdown().await; + Ok(()) +} + +fn route_request() -> CellRouteRequest { + use cellar_logic::consensus::{ClientId, CommandId}; + CellRouteRequest { + cell: CellHistory { + cluster: route().history.cluster, + namespace: route().history.namespace, + cell: CellId::new("counter-routing").unwrap(), + incarnation: IncarnationId::new([81; 16]).unwrap(), + }, + operation: CellRouteOperation::Command( + CommandId::new(ClientId::new([82; 16]).unwrap(), 7).unwrap(), + ), + fingerprint: Sha256::digest(b"canonical command with input and durability").into(), + } +} + +fn routing_limits(max_attempts: u8) -> CellRoutingLimits { + CellRoutingLimits { + timeout: Duration::from_secs(10), + max_attempts, + } +} + +async fn publish_route_split( + f: &Fixture, + parent: &GroupRoute, + cell: &CellId, +) -> Result { + let view = f.run(f.directory.observe()).await?; + let mut children = split(parent, 700); + for child in &mut children { + child.schema_digest = parent.schema_digest; + } + f.reserve(&mut children).await?; + let replace = f.directory.prepare_replacement( + &view, + operation(1000), + std::slice::from_ref(parent), + &children, + )?; + f.run(f.directory.publish(&replace)).await?; + let key = cell_routing_key(parent.history.namespace, cell); + Ok(children + .into_iter() + .find(|r| r.range.contains(&key)) + .unwrap()) +} + +pub async fn routing_authentication() -> Result<()> { + let f = Fixture::new(DirectoryLimits::default()).await?; + let birth = selected_birth(&f).await?; + let request = route_request(); + let mut session = + f.directory + .routing_session(&f.domain, request.clone(), routing_limits(4), None)?; + let first = session.begin().await?; + assert_eq!(first.route(), birth.route()); + session.mark_submitted(&first)?; + let next = publish_route_split(&f, birth.route(), &request.cell.cell).await?; + let second = session.redirect(&first, &next).await?; + assert_eq!(second.route(), &next); + assert_eq!(second.request(), &request); + assert_eq!(second.deadline_ms(), first.deadline_ms()); + assert_eq!(second.ordinal(), 2); + let stale = session.redirect(&first, &next).await.err().unwrap(); + assert!(matches!(stale.reason, CellRoutingFailure::StaleAttempt)); + assert!(stale.ambiguous_command.is_some()); + let third = session.retry(&second).await?; + assert_eq!(third.request(), first.request()); + assert_eq!(third.deadline_ms(), first.deadline_ms()); + + // Valid-looking but unpublished history must fail catalog confirmation. + let mut forged = next.clone(); + forged.history.group = GroupId::new([99; 16])?; + let mut refused = f.directory.routing_session( + &f.domain, + request.clone(), + routing_limits(4), + Some(first.cached_route()), + )?; + let attempt = refused.begin().await?; + refused.mark_submitted(&attempt)?; + let (result, reads) = fault::count_reads(refused.redirect(&attempt, &forged)).await; + let failure = result.err().unwrap(); + assert!(matches!(failure.reason, CellRoutingFailure::InvalidHint)); + assert!(!reads.is_empty()); + let CellRouteOperation::Command(id) = request.operation else { + unreachable!() + }; + assert_eq!(failure.ambiguous_command, Some(id)); + assert!(matches!( + refused.retry(&attempt).await.err().unwrap().reason, + CellRoutingFailure::StaleAttempt + )); + + // Syntactically valid namespace/schema rebinding is rejected before I/O. + for field in 0..4 { + let mut wrong = next.clone(); + match field { + 0 => wrong.history.namespace = NamespaceId::new([91; 16])?, + 1 => wrong.history.cluster = ClusterId::new([92; 16])?, + 2 => wrong.schema_digest = [93; 32], + _ => wrong.epoch = birth.route().epoch, + } + let mut session = f.directory.routing_session( + &f.domain, + request.clone(), + routing_limits(4), + Some(first.cached_route()), + )?; + let attempt = session.begin().await?; + let (result, reads) = fault::count_reads(session.redirect(&attempt, &wrong)).await; + assert!(matches!( + result.err().unwrap().reason, + CellRoutingFailure::InvalidHint + )); + assert!(reads.is_empty()); + } + assert_eq!(f.store.listing_calls.load(Ordering::Acquire), 0); + f.domain.shutdown().await; + Ok(()) +} + +pub async fn routing_cache_and_budget() -> Result<()> { + let f = Fixture::new(DirectoryLimits::default()).await?; + let birth = selected_birth(&f).await?; + let request = route_request(); + let mut origin = + f.directory + .routing_session(&f.domain, request.clone(), routing_limits(2), None)?; + let first = origin.begin().await?; + let cached = first.cached_route(); + let next = publish_route_split(&f, birth.route(), &request.cell.cell).await?; + origin.mark_submitted(&first)?; + let second = origin.redirect(&first, &next).await?; + let error = origin.retry(&second).await.err().unwrap(); + assert!(matches!( + error.reason, + CellRoutingFailure::AttemptsExhausted + )); + assert!(error.ambiguous_command.is_some()); + + // Cached attempts can use a historical route during a catalog outage. + // This grants no serving authority: the destination must check its epoch. + f.store.fail_read.store(true, Ordering::Release); + let mut known = f.directory.routing_session( + &f.domain, + request.clone(), + routing_limits(2), + Some(cached.clone()), + )?; + let (attempt, reads) = fault::count_reads(known.begin()).await; + let attempt = attempt?; + assert!(reads.is_empty()); + assert!(f.store.fail_read.load(Ordering::Acquire)); + assert_eq!(attempt.route(), birth.route()); + assert_eq!(attempt.request(), &request); + let (retry, reads) = fault::count_reads(known.retry(&attempt)).await; + assert!(reads.is_empty()); + assert_eq!(retry?.request(), &request); + let foreign_ticket = known.mark_submitted(&first).unwrap_err(); + assert!(matches!( + foreign_ticket.reason, + CellRoutingFailure::StaleAttempt + )); + f.store.fail_read.store(false, Ordering::Release); + let foreign = f.reopen().await?; + assert!(matches!( + foreign + .routing_session( + &f.domain, + request.clone(), + routing_limits(2), + Some(cached.clone()) + ) + .err() + .unwrap() + .reason, + CellRoutingFailure::InvalidRequest + )); + for field in 0..3 { + let mut wrong = request.clone(); + match field { + 0 => wrong.cell.cell = CellId::new("another-cell")?, + 1 => wrong.cell.incarnation = IncarnationId::new([93; 16])?, + _ => wrong.fingerprint = [0; 32], + } + assert!(matches!( + f.directory + .routing_session(&f.domain, wrong, routing_limits(2), Some(cached.clone())) + .err() + .unwrap() + .reason, + CellRoutingFailure::InvalidRequest + )); + } + for maximum in [0, 17, 255] { + assert!(f + .directory + .routing_session(&f.domain, request.clone(), routing_limits(maximum), None) + .is_err()); + } + f.domain.shutdown().await; + assert!(matches!( + origin.retry(&second).await.err().unwrap().reason, + CellRoutingFailure::StaleAttempt + )); + Ok(()) +} + +#[allow(clippy::disallowed_methods)] +pub async fn routing_deadline_and_cancellation() -> Result<()> { + let f = Fixture::new(DirectoryLimits::default()).await?; + let birth = selected_birth(&f).await?; + let request = route_request(); + let mut initial = + f.directory + .routing_session(&f.domain, request.clone(), routing_limits(4), None)?; + let observed = initial.begin().await?; + let next = publish_route_split(&f, birth.route(), &request.cell.cell).await?; + let limits = CellRoutingLimits { + timeout: Duration::from_secs(5), + max_attempts: 4, + }; + let mut session = f.directory.routing_session( + &f.domain, + request.clone(), + limits, + Some(observed.cached_route()), + )?; + let attempt = session.begin().await?; + session.mark_submitted(&attempt)?; + // Spend most of the original budget before redirecting. The two-second + // external guard must beat a fresh five-second timeout, but leaves a full + // second of scheduling tolerance for the original one-second remainder. + tokio::time::sleep(Duration::from_secs(4)).await; + let gate = Arc::new(fault::RegistryGate::default()); + gate.armed.store(true, Ordering::Release); + let result = tokio::time::timeout( + Duration::from_secs(2), + fault::gate_read( + f.key("pin").to_string(), + gate.clone(), + session.redirect(&attempt, &next), + ), + ) + .await?; + assert!(!gate.armed.load(Ordering::Acquire)); + let error = result.err().unwrap(); + assert!(matches!(error.reason, CellRoutingFailure::Deadline)); + assert!(error.ambiguous_command.is_some()); + assert!(matches!( + session.retry(&attempt).await.err().unwrap().reason, + CellRoutingFailure::StaleAttempt + )); + + // Dropping discovery cannot reset its attempt budget or accept its ticket. + let mut cancelled = + f.directory + .routing_session(&f.domain, request.clone(), routing_limits(3), None)?; + let gate = Arc::new(fault::RegistryGate::default()); + gate.armed.store(true, Ordering::Release); + { + let call = fault::gate_read(f.key("pin").to_string(), gate.clone(), cancelled.begin()); + tokio::pin!(call); + match tokio::time::timeout( + Duration::from_secs(2), + futures_util::future::select(&mut call, Box::pin(gate.entered.notified())), + ) + .await? + { + futures_util::future::Either::Left((result, _)) => { + panic!("stalled discovery completed: {}", result.is_ok()) + } + futures_util::future::Either::Right(((), _pending)) => {} + } + } + assert!(matches!( + cancelled.begin().await.err().unwrap().reason, + CellRoutingFailure::StaleAttempt + )); + let mut closed = f.directory.routing_session( + &f.domain, + request, + routing_limits(2), + Some(observed.cached_route()), + )?; + f.domain.shutdown().await; + assert!(matches!( + closed.begin().await.err().unwrap().reason, + CellRoutingFailure::Closed + )); + Ok(()) +} + +#[path = "cell_group_routing_cache.rs"] +mod routing_cache; +pub(crate) use routing_cache::{ + bounded_route_hints, cached_operation_discovery, stale_cache_refill, +}; diff --git a/crates/cellar/tests/support/cell_group_directory_transition.rs b/crates/cellar/tests/support/cell_group_directory_transition.rs new file mode 100644 index 000000000..fb010124b --- /dev/null +++ b/crates/cellar/tests/support/cell_group_directory_transition.rs @@ -0,0 +1,144 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; + +async fn split_plan(f: &Fixture) -> Result<(DirectoryMutation, Vec, GroupRoute)> { + let mut parent = assigned(1000, CellRange::all(), 1); + f.reserve(std::slice::from_mut(&mut parent)).await?; + let d = &f.directory; + let initialize = d.prepare_initialization(operation(100), std::slice::from_ref(&parent))?; + f.run(d.publish(&initialize)).await?; + let initial = f.run(d.observe()).await?; + let mut children = split(&parent, 2000); + f.reserve(&mut children).await?; + let plan = d.prepare_replacement( + &initial, + operation(101), + std::slice::from_ref(&parent), + &children, + )?; + Ok((plan, children, parent)) +} + +pub(crate) async fn seal_authorization_cancel_race() -> Result<()> { + let f = Fixture::new(DirectoryLimits::default()).await?; + let d = &f.directory; + let (plan, children, parent) = split_plan(&f).await?; + let planned = f.run(d.plan_transition(&plan)).await?; + anyhow::ensure!(planned.phase() == TransitionPhase::Planned); + anyhow::ensure!(matches!( + f.run(d.authorize_transition_seal(&plan)).await, + Err(CatalogError::Conflict(_)) + )); + let view = f.run(d.observe()).await?; + anyhow::ensure!(view.groups() == 1 && view.revision() == 1); + let foreign = d.prepare_replacement( + &view, + operation(102), + std::slice::from_ref(&parent), + &children, + )?; + anyhow::ensure!(matches!( + f.run(d.publish(&foreign)).await, + Err(CatalogError::Fenced) + )); + anyhow::ensure!(matches!( + f.run(d.freeze_group_births(operation(103))).await, + Err(CatalogError::Fenced) + )); + let prepared = f.run(d.prepare_transition(&plan, [72; 32])).await?; + anyhow::ensure!( + prepared.phase() == TransitionPhase::Prepared && prepared.preparation() == Some([72; 32]) + ); + let reopened = f.reopen().await?; + let (request, retained) = f + .run(reopened.active_transition()) + .await? + .context("active transition")?; + anyhow::ensure!(request.encode() == plan.encode() && retained == prepared); + // Hold authorization after it read Prepared and built its next head. + // Cancellation commits against that same head while the first CAS waits. + f.store.range_before.armed.store(true, Ordering::Release); + let entered = f.store.range_before.entered.notified(); + let (authorize, cancel) = f + .run(cellar::asyncrt::timeout(Duration::from_secs(15), async { + tokio::join!(reopened.authorize_transition_seal(&request), async { + entered.await; + let result = d.cancel_transition(&plan).await; + f.store.range_before.resume.notify_one(); + result + }) + })) + .await?; + anyhow::ensure!(matches!(authorize, Err(CatalogError::Conflict(_)))); + let winner = cancel?; + anyhow::ensure!(winner.phase() == TransitionPhase::Cancelled); + anyhow::ensure!(f.run(reopened.active_transition()).await?.is_none()); + anyhow::ensure!(f.run(d.plan_transition(&plan)).await? == winner); + let current = f.run(d.observe()).await?; + let reused = d.prepare_replacement( + ¤t, + operation(101), + std::slice::from_ref(&parent), + &children, + )?; + anyhow::ensure!(matches!( + f.run(d.publish(&reused)).await, + Err(CatalogError::Conflict(_)) + )); + anyhow::ensure!(matches!( + f.run(d.plan_transition(&reused)).await, + Err(CatalogError::Conflict(_)) + )); + anyhow::ensure!(f.run(reopened.observe()).await?.groups() == 1); + f.domain.shutdown().await; + Ok(()) +} + +pub(crate) async fn unknown_transition_authorization() -> Result<()> { + let f = Fixture::new(DirectoryLimits::default()).await?; + let d = &f.directory; + let (plan, _, parent) = split_plan(&f).await?; + f.run(d.plan_transition(&plan)).await?; + f.run(d.prepare_transition(&plan, [81; 32])).await?; + f.store + .fail_head_confirmation + .store(true, Ordering::Release); + anyhow::ensure!(matches!( + f.run(d.authorize_transition_seal(&plan)).await, + Err(CatalogError::Unconfirmed) + )); + f.store.fail_read.store(false, Ordering::Release); + let restarted = f.reopen().await?; + let (original, journal) = f + .run(restarted.active_transition()) + .await? + .context("rooted authorization")?; + anyhow::ensure!(original.encode() == plan.encode()); + anyhow::ensure!( + journal.phase() == TransitionPhase::SealAuthorized + && journal.preparation() == Some([81; 32]) + ); + anyhow::ensure!( + f.run(restarted.authorize_transition_seal(&original)) + .await? + == journal + ); + anyhow::ensure!(matches!( + f.run(restarted.cancel_transition(&original)).await, + Err(CatalogError::Conflict(_)) + )); + anyhow::ensure!(matches!( + f.run(restarted.prepare_transition(&original, [82; 32])) + .await, + Err(CatalogError::Conflict(_)) + )); + let view = f.run(restarted.observe()).await?; + anyhow::ensure!( + f.run(restarted.lookup(&view, &CellId::new("unchanged-cell")?)) + .await? + == parent + ); + f.domain.shutdown().await; + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_disaster_activation.rs b/crates/cellar/tests/support/cell_group_disaster_activation.rs new file mode 100644 index 000000000..702b4fe6e --- /dev/null +++ b/crates/cellar/tests/support/cell_group_disaster_activation.rs @@ -0,0 +1,766 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Real fresh voters, scheduler, physical apply and topology CAS after all old +//! voter disks are gone. The external fence is still a contract fixture. +use super::*; + +#[path = "cell_group_disaster_receipts.rs"] +mod receipts; +pub(crate) use receipts::registered_receipts; +pub(super) use receipts::ReceiptWorkload; + +struct FreshVoter { + log: ReplicaLog, + applier: CellGroupApplier, + member: u64, + base: std::path::PathBuf, + options: ReplicaLogOptions, + runtime: CellGroupRuntime, + events: Option, + images: CellGroupApplyImages, + sql: CellGroupSqlPool, +} + +async fn pump( + voters: &mut [FreshVoter], + history: GroupHistory, + messages: &mut VecDeque, +) -> Result<()> { + for (ordinal, voter) in voters.iter_mut().enumerate() { + for _ in 0..64 { + let Ok(event) = voter + .events + .as_mut() + .context("owned recovery events")? + .try_recv() + else { + break; + }; + match event.effect().map_err(|e| anyhow::anyhow!(e.to_string()))? { + CellGroupEffect::Message(message) => messages.push_back(message.clone()), + CellGroupEffect::Apply(_) => { + let work = event.retain_apply().context("recovered apply")?; + Box::pin(voter.sql.prepare_apply(&voter.applier, work)) + .await? + .complete(&voter.runtime) + .await?; + } + CellGroupEffect::Applied { .. } | CellGroupEffect::ReadIndex { .. } => (), + _ => anyhow::bail!("unexpected recovered event on {ordinal}"), + } + } + } + for _ in 0..messages.len() { + let message = messages.pop_front().context("queued fresh message")?; + let target = voters + .iter() + .position(|v| v.member == message.to) + .context("fresh target member")?; + match voters[target].runtime.step(history, message.clone()).await { + Ok(()) => (), + Err(RuntimeError::NotReady) => messages.push_back(message), + Err(error) => return Err(error.into()), + } + } + tokio::task::yield_now().await; + Ok(()) +} + +async fn applied(voters: &mut [FreshVoter], history: GroupHistory, index: LogIndex) -> Result<()> { + let mut messages = VecDeque::new(); + tokio::time::timeout(Duration::from_secs(15), async { + loop { + pump(voters, history, &mut messages).await?; + let mut done = true; + for voter in voters.iter() { + done &= voter.runtime.status(history).await?.applied.index >= index; + } + if done { + return Ok::<_, anyhow::Error>(()); + } + } + }) + .await? +} + +// Each cold control phase owns its future. A single inline state machine for +// the entire disk-loss drill otherwise consumes the default test-thread stack +// before nested archive verification can run. +type SeededVoters = ( + Vec, + Vec, + Vec<(std::path::PathBuf, ReplicaLogOptions, CellGroupApplyImages)>, +); + +struct FreshQuorum { + voters: Vec, + route: GroupRoute, + cell: CellHistory, + authority: Arc, + publication: DirectoryPublication, +} + +pub(crate) async fn activate_quorum(rooted: bool, store: Arc) -> Result<()> { + let mut s = Box::new(Scenario::new(rooted, store).await?); + let seeds = seed_voters(&mut s, rooted).await?; + let mut quorum = publish_voters(&mut s, seeds).await?; + activate_voters(&s, &mut quorum).await?; + mutate_voters(&mut s, &mut quorum, rooted).await?; + restart_voters(&s, &mut quorum, rooted).await?; + for voter in quorum.voters { + voter.runtime.close().await?; + voter.sql.close().await?; + voter.log.close().await?; + } + s.cluster.close().await +} + +fn seed_voters( + s: &mut Scenario, + rooted: bool, +) -> futures_util::future::BoxFuture<'_, Result> { + Box::pin(async move { + seed::erase_voters(s, rooted).await?; + let proof = s.fence(&s.cluster.directory, &s.request).await?; + let route = s.request.replacement().route().clone(); + let pool = CellGroupRecoveryPool::new(Default::default())?; + let scratch = ScratchPool::new(ScratchLimits { + max_bytes: 32 * 1024 * 1024, + max_files: 1, + })?; + let mut seeds = Vec::new(); + let mut logs = Vec::new(); + let mut reopen = Vec::new(); + for (ordinal, member) in s.request.replacement().voters().iter().enumerate() { + assert!(s + .cluster + .domain + .run( + s.cluster + .directory + .publish_group_recovery(&s.request, &s.authority) + ) + .await? + .is_err()); + let base = s.cluster._root.path().join(format!("fresh-{ordinal}")); + let destination = destination_at( + &s.cluster, + rooted, + Arc::new(DirectFileSystem), + base.join("data"), + route.clone(), + ) + .await?; + let prepared = pool + .prepare( + &s.cluster.domain, + proof.clone(), + destination.store.clone(), + scratch.clone(), + s.cluster._root.path().into(), + ) + .await?; + let options = ReplicaLogOptions::new( + base.join("log"), + member.node, + DiskLineage::new([210 + ordinal as u8; 16])?, + ProcessGeneration::new([220; 16])?, + LogOpenMode::Create, + ); + let log = s + .cluster + .domain + .run(ReplicaLog::open( + options.clone(), + Arc::new(DirectFileSystem), + )) + .await??; + reopen.push((base, options, destination.apply_images.clone())); + let seed = s + .cluster + .directory + .seed_group_recovery(&s.cluster.domain, prepared, &log, &s.authority) + .await?; + assert!(s + .cluster + .domain + .run(s.cluster.directory.bootstrap_recovered_group_member( + &s.request, + &log, + Arc::new(Authority::default()) + )) + .await? + .is_err()); + if let Some(first) = seeds.first() { + let first: &SeededCellGroupRecovery = first; + assert_eq!(seed.manifest().marker(), first.manifest().marker()); + assert_eq!(seed.binding(), first.binding()); + } + seeds.push(seed); + logs.push(log); + } + assert_eq!(pool.usage().operations, 0); + drop(proof); + assert_eq!(s.receivers.usage().sources, 0); + Ok((seeds, logs, reopen)) + }) +} + +fn publish_voters( + s: &mut Scenario, + seeded: SeededVoters, +) -> futures_util::future::BoxFuture<'_, Result> { + Box::pin(async move { + let (seeds, logs, reopen) = seeded; + let route = s.request.replacement().route().clone(); + s.authority.reject.store(true, Ordering::Release); + assert!(s + .cluster + .domain + .run( + s.cluster + .directory + .publish_group_recovery(&s.request, &s.authority) + ) + .await? + .is_err()); + s.authority.reject.store(false, Ordering::Release); + // Lose the actual head CAS response and all immediate reads. The replacement + // is selected; retry must recover that exact publication without rebasing. + s.remote + .lose_directory_head_and_reads + .store(true, Ordering::Release); + assert!(matches!( + s.cluster + .domain + .run( + s.cluster + .directory + .publish_group_recovery(&s.request, &s.authority) + ) + .await?, + Err(CatalogError::Unconfirmed) + )); + s.remote.outage.store(false, Ordering::Release); + let publication = s + .cluster + .domain + .run( + s.cluster + .directory + .publish_group_recovery(&s.request, &s.authority), + ) + .await??; + assert_eq!( + publication.journal_digest(), + s.cluster + .domain + .run( + s.cluster + .directory + .publish_group_recovery(&s.request, &s.authority) + ) + .await?? + .journal_digest() + ); + let birth = s + .cluster + .domain + .run( + s.cluster + .directory + .materialize_group_birth(s.request.replacement()), + ) + .await??; + assert!(matches!( + s.cluster + .domain + .run(s.cluster.directory.bootstrap_group_member(&birth, &logs[0])) + .await?, + Err(CatalogError::Fenced) + )); + // The same head still fences ordinary topology while activation is pending. + assert!(matches!( + s.cluster + .domain + .run(s.cluster.directory.publish(&s.competing)) + .await?, + Err(CatalogError::Fenced) + )); + let cell = seeds[0] + .manifest() + .pin_cell(&CellId::new("alpha")?) + .context("restored alpha")? + .0 + .version + .history + .clone(); + let authority = Arc::new(std::mem::take(&mut s.authority)); + let binding_key = s + .remote + .list(None) + .try_collect::>() + .await? + .into_iter() + .find(|meta| meta.location.as_ref().contains("/recovery-bindings/")) + .context("recovery binding key")? + .location; + let binding_bytes = s.remote.get(&binding_key).await?.bytes().await?; + s.remote.delete(&binding_key).await?; + assert!(s + .cluster + .domain + .run(s.cluster.directory.bootstrap_recovered_group_member( + &s.request, + &logs[0], + authority.clone() + )) + .await? + .is_err()); + assert!(matches!( + s.cluster + .domain + .run(s.cluster.directory.bootstrap_group_member(&birth, &logs[0])) + .await?, + Err(CatalogError::Fenced) + )); + s.remote.put(&binding_key, binding_bytes.into()).await?; + let mut voters = Vec::new(); + for ((seed, log), (base, options, images)) in seeds.into_iter().zip(logs).zip(reopen) { + let permit = s + .cluster + .domain + .run(s.cluster.directory.bootstrap_recovered_group_member( + &s.request, + &log, + authority.clone(), + )) + .await??; + let (runtime, events) = CellGroupRuntime::start( + log.clone(), + RaftRuntimeOptions { + tick_interval_ms: None, + election_seed: Some([231; 32]), + ..Default::default() + }, + ) + .await?; + runtime.attach(permit, seed.manifest().clone()).await?; + let sql = CellGroupSqlPool::start( + &log, + CellSqlLimits { + workers: 1, + queue_per_worker: 4, + max_pending_cells: 8, + group_pending_cells: 8, + node_bytes: 128 * 1024 * 1024, + group_bytes: 64 * 1024 * 1024, + }, + )?; + voters.push(FreshVoter { + member: seed.group().identity().member.get(), + applier: seed.applier().clone(), + base, + options, + log, + runtime, + events: Some(events), + images, + sql, + }); + } + Ok(FreshQuorum { + voters, + route, + cell, + authority, + publication, + }) + }) +} + +fn activate_voters<'a>( + s: &'a Scenario, + quorum: &'a mut FreshQuorum, +) -> futures_util::future::BoxFuture<'a, Result<()>> { + Box::pin(async move { + let FreshQuorum { + voters, + route, + cell, + authority, + publication, + } = quorum; + let history = route.history; + let grant = s + .cluster + .domain + .run(s.cluster.directory.prepare_group_recovery_activation( + &s.request, + &voters[0].log, + authority.as_ref(), + )) + .await??; + assert!(matches!( + voters[0].runtime.activate_recovery(grant).await, + Err(RuntimeError::NotLeader) + )); + voters[0].runtime.campaign(history).await?; + applied(voters, history, LogIndex::new(2)).await?; + let status = voters[0].runtime.status(history).await?; + assert_eq!(status.leader, Some(11)); + assert!(status.term.get() > 1 && !status.ready); + assert!(matches!( + voters[0] + .runtime + .execution_authority(history, cell.clone()) + .await, + Err(RuntimeError::NotReady) + )); + let deadline = s + .cluster + .domain + .run(async { cellar::asyncrt::mono_ms() + 5000 }) + .await?; + let pending_read = voters[0] + .runtime + .read_authority(history, cell.clone(), CellReadConsistency::Stale, deadline) + .await; + assert!( + matches!(&pending_read, Err(RuntimeError::NotReady)), + "pending recovery read refusal: {:?}", + pending_read.as_ref().err() + ); + let grant = s + .cluster + .domain + .run(s.cluster.directory.prepare_group_recovery_activation( + &s.request, + &voters[0].log, + authority.as_ref(), + )) + .await??; + assert!(matches!( + voters[1].runtime.activate_recovery(grant).await, + Err(RuntimeError::StaleCompletion) + )); + let grant = s + .cluster + .domain + .run(s.cluster.directory.prepare_group_recovery_activation( + &s.request, + &voters[0].log, + authority.as_ref(), + )) + .await??; + let CellGroupRecoveryActivation::Accepted(proposal) = + voters[0].runtime.activate_recovery(grant).await? + else { + anyhow::bail!("activation skipped replication") + }; + let grant = s + .cluster + .domain + .run(s.cluster.directory.prepare_group_recovery_activation( + &s.request, + &voters[0].log, + authority.as_ref(), + )) + .await??; + let CellGroupRecoveryActivation::Accepted(retry) = + voters[0].runtime.activate_recovery(grant).await? + else { + anyhow::bail!("activation committed without network") + }; + assert_eq!(proposal.position, retry.position); + assert_eq!(proposal.digest, retry.digest); + assert!(!voters[0].runtime.status(history).await?.ready); + applied(voters, history, proposal.position.index).await?; + assert!(voters[0].runtime.status(history).await?.ready); + voters[0] + .runtime + .execution_authority(history, cell.clone()) + .await?; + let grant = s + .cluster + .domain + .run(s.cluster.directory.prepare_group_recovery_activation( + &s.request, + &voters[0].log, + authority.as_ref(), + )) + .await??; + let CellGroupRecoveryActivation::Applied(acknowledgement) = + voters[0].runtime.activate_recovery(grant).await? + else { + anyhow::bail!("activation not selected") + }; + s.remote + .lose_directory_head_and_reads + .store(true, Ordering::Release); + assert!(matches!( + s.cluster + .domain + .run(s.cluster.directory.complete_group_recovery( + &s.request, + &acknowledgement, + authority.as_ref() + )) + .await?, + Err(CatalogError::Unconfirmed) + )); + s.remote.outage.store(false, Ordering::Release); + assert_eq!( + s.cluster + .domain + .run(s.cluster.directory.complete_group_recovery( + &s.request, + &acknowledgement, + authority.as_ref() + )) + .await?? + .journal_digest(), + publication.journal_digest() + ); + assert_eq!( + s.cluster + .domain + .run(s.cluster.directory.complete_group_recovery( + &s.request, + &acknowledgement, + authority.as_ref() + )) + .await?? + .journal_digest(), + publication.journal_digest() + ); + drop(acknowledgement); + Ok(()) + }) +} + +fn mutate_voters<'a>( + s: &'a mut Scenario, + quorum: &'a mut FreshQuorum, + rooted: bool, +) -> futures_util::future::BoxFuture<'a, Result<()>> { + Box::pin(async move { + let FreshQuorum { + voters, + route, + cell, + authority, + .. + } = quorum; + let history = route.history; + // Exercise a real cell-addressed LTX mutation on the restored group, through + // the bounded production physical applier, in both image representations. + let before = &s.expected["alpha"]; + let after = images::image(Some(before), "UPDATE value SET n=n+1")?; + let cut = images::Cut::new(before, &after, 1, true, false); + let bytes = request(route, cell.clone(), 2, &cut)?; + let mut entry = CellGroupEntry::decode(&bytes, route, GroupEntryLimits::default())?; + if rooted { + let GroupCommand::Mutation(mutation) = &mut entry.command else { + anyhow::bail!("mutation") + }; + mutation.base.commitment = + checkpoint_fixture::extent_commitment(route, cell, cut.base, before)?; + mutation.result.commitment = + checkpoint_fixture::extent_commitment(route, cell, cut.result, &after)?; + } + let proposal = voters[0] + .runtime + .propose(history, entry.encode(route, GroupEntryLimits::default())?) + .await?; + applied(voters, history, proposal.position.index).await?; + s.expected.insert("alpha".into(), after); + for voter in voters.iter() { + let applier = voter.applier.clone(); + let selected = s + .cluster + .domain + .blocking(move || applier.recover_selected())? + .await??; + assert!(selected.marker().is_active()); + assert_eq!( + selected + .pin_cell(&cell.cell) + .context("mutated cell")? + .0 + .version + .sequence + .get(), + 2 + ); + tail_fixture::check_images(&s.cluster, &selected, &s.expected).await?; + s.cluster + .domain + .run(s.cluster.directory.bootstrap_recovered_group_member( + &s.request, + &voter.log, + authority.clone(), + )) + .await??; + } + Ok(()) + }) +} + +fn restart_voters<'a>( + s: &'a Scenario, + quorum: &'a mut FreshQuorum, + rooted: bool, +) -> futures_util::future::BoxFuture<'a, Result<()>> { + Box::pin(async move { + let FreshQuorum { + voters, + route, + cell, + authority, + .. + } = quorum; + let history = route.history; + let mut reopen = Vec::new(); + for voter in voters.drain(..) { + voter.runtime.close().await?; + voter.sql.close().await?; + voter.log.close().await?; + let mut options = voter.options.clone(); + options.mode = LogOpenMode::Resume; + options.process = ProcessGeneration::new([240; 16])?; + reopen.push((voter.base.clone(), options)); + drop(voter); + } + for (base, options) in reopen { + let log = s + .cluster + .domain + .run(ReplicaLog::open( + options.clone(), + Arc::new(DirectFileSystem), + )) + .await??; + let group = log.cell_group(&history)?; + let target = base.join("data"); + let (applier, root, images) = s + .cluster + .domain + .blocking(move || reopen_applier(group, target, rooted))? + .await??; + assert!(root.marker().is_active()); + tail_fixture::check_images(&s.cluster, &root, &s.expected).await?; + let permit = s + .cluster + .domain + .run(s.cluster.directory.bootstrap_recovered_group_member( + &s.request, + &log, + authority.clone(), + )) + .await??; + let member = permit.group().identity().member.get(); + let (runtime, events) = CellGroupRuntime::start( + log.clone(), + RaftRuntimeOptions { + tick_interval_ms: None, + election_seed: Some([241; 32]), + ..Default::default() + }, + ) + .await?; + runtime.attach(permit, root).await?; + assert!(!runtime.status(history).await?.ready); + let sql = CellGroupSqlPool::start( + &log, + CellSqlLimits { + workers: 1, + queue_per_worker: 4, + max_pending_cells: 8, + group_pending_cells: 8, + node_bytes: 128 * 1024 * 1024, + group_bytes: 64 * 1024 * 1024, + }, + )?; + voters.push(FreshVoter { + log, + applier, + member, + runtime, + events: Some(events), + images, + sql, + base, + options, + }); + } + let old_term = voters[0].runtime.status(history).await?.term; + let before = voters[0].runtime.status(history).await?.applied.index; + voters[0].runtime.campaign(history).await?; + applied(voters, history, LogIndex::new(before.get() + 1)).await?; + assert!(voters[0].runtime.status(history).await?.ready); + assert!(voters[0].runtime.status(history).await?.term > old_term); + voters[0] + .runtime + .execution_authority(history, cell.clone()) + .await?; + Ok(()) + }) +} + +fn reopen_applier( + group: cellar::application::consensus::verification::GroupLog, + base: std::path::PathBuf, + rooted: bool, +) -> Result<( + CellGroupApplier, + PreparedGroupManifest, + CellGroupApplyImages, +)> { + let filesystem: Arc = Arc::new(DirectFileSystem); + let pages = ManifestPageStore::open( + &base.join("pages"), + filesystem.clone(), + 1024, + 16 * 1024 * 1024, + 16, + )?; + let services = ReadServices { + now_unix_millis: Arc::new(|| 10), + fill_random: Arc::new(|bytes| bytes.fill(17)), + }; + let images = if rooted { + CellGroupApplyImages::Extents { + images: cellar_ltx::internal::NodeExtentStore::open( + base.join("extents"), + filesystem, + Default::default(), + )?, + services, + limits: Default::default(), + cache_directory: base.join("cache"), + cache_pool: ScratchPool::new(ScratchLimits { + max_bytes: 32 * 1024 * 1024, + max_files: 1, + })?, + cache: CacheLimits { + max_cached_bytes: EXTENT_BYTES, + max_cached_extents: 1, + }, + } + } else { + CellGroupApplyImages::Materialized { + images: GenerationStore::open(base.join("images"), filesystem, Default::default())?, + services, + } + }; + let (applier, root) = CellGroupApplier::recover( + group, + pages, + images.clone(), + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + )?; + Ok((applier, root, images)) +} diff --git a/crates/cellar/tests/support/cell_group_disaster_fence.rs b/crates/cellar/tests/support/cell_group_disaster_fence.rs new file mode 100644 index 000000000..536967e01 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_disaster_fence.rs @@ -0,0 +1,718 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Real catalog CAS/archive I/O and voter disks, with a contract-test authority. +//! This is not Docker host fencing or fresh-incarnation serving qualification. +use super::*; +use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; + +#[path = "cell_group_disaster_activation.rs"] +mod activation; +#[path = "cell_group_disaster_seed.rs"] +mod seed; +pub(super) use activation::{activate_quorum, registered_receipts}; +pub(super) use seed::seed_engine; + +#[derive(Default)] +struct Authority { + fences: AtomicUsize, + verifications: AtomicUsize, + permanently_stopped: AtomicBool, + lose_response: AtomicBool, + reject: AtomicBool, + held_verification: std::sync::Mutex>>, +} + +impl DisasterFenceAuthority for Authority { + fn fence<'a>( + &'a self, + request: &'a CellGroupDisasterFenceRequest, + ) -> futures_util::future::BoxFuture<'a, Result>> { + Box::pin(async move { + self.fences.fetch_add(1, Ordering::AcqRel); + self.permanently_stopped.store(true, Ordering::Release); + if self.lose_response.swap(false, Ordering::AcqRel) { + anyhow::bail!("lost authority response after fence"); + } + Ok(request.digest().to_vec()) + }) + } + + fn verify<'a>( + &'a self, + request: &'a CellGroupDisasterFenceRequest, + evidence: &'a [u8], + ) -> futures_util::future::BoxFuture<'a, Result<()>> { + Box::pin(async move { + self.verifications.fetch_add(1, Ordering::AcqRel); + anyhow::ensure!( + self.permanently_stopped.load(Ordering::Acquire), + "no external fence" + ); + anyhow::ensure!( + !self.reject.load(Ordering::Acquire), + "independent verification rejected" + ); + anyhow::ensure!(evidence == request.digest(), "foreign fence evidence"); + let gate = self.held_verification.lock().unwrap().take(); + if let Some(gate) = gate { + gate.entered.notify_one(); + gate.release.notified().await; + } + Ok(()) + }) + } +} + +struct Scenario { + cluster: Cluster, + expected: std::collections::BTreeMap>, + remote: Arc, + archives: CellGroupArchivePool, + tails: CellGroupArchiveTailPool, + receivers: CellGroupCheckpointPool, + destination: ArchiveDestination, + request: CellGroupRecoveryRequest, + alternate: CellGroupRecoveryRequest, + competing: DirectoryMutation, + checkpoint: CellGroupArchiveReference, + authority: Authority, + original_head: [u8; 32], +} + +impl Scenario { + async fn new(rooted: bool, store: Arc) -> Result { + Box::pin(Self::with_workload(rooted, store, None)).await + } + + async fn with_workload( + rooted: bool, + store: Arc, + mut workload: Option<&mut activation::ReceiptWorkload>, + ) -> Result { + let remote = archive_store::Store::around(store); + let mut cluster = Cluster::with_store_birth( + Default::default(), + tempfile::tempdir()?, + Arc::new(DirectFileSystem), + workload + .as_ref() + .map_or([6; 32], |w| w.class.schema_digest()), + remote.clone(), + ClusterBirthOptions { + group: GroupId::new([3; 16])?, + image_storage: if rooted { + RaftImageStorage::Extents + } else { + RaftImageStorage::Materialized + }, + minimum: Durability::Quorum, + log: Default::default(), + }, + ) + .await?; + cluster.elect().await?; + let mut expected = + checkpoint_fixture::enroll_images(&mut cluster, rooted, false, 0, 8192).await?; + let archives = CellGroupArchivePool::new(Default::default())?; + let binding = cluster + .directory + .prepare_group_archive_binding(&cluster.domain, &archives, &cluster.birth) + .await?; + let registry = binding.commit(&cluster.domain).await?; + let owners = CellGroupArchiveUploadPool::new(Default::default())?; + let tails = CellGroupArchiveTailPool::new(Default::default())?; + let intent = registry.prepare_upload(&cluster.domain).await?; + let ticket = registry.reserve(&cluster.domain, &intent).await?; + let upload = registry + .open_upload(&cluster.domain, &owners, &ticket, &tails) + .await?; + let sources = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let checkpoint = upload + .upload_checkpoint(&cluster.domain, source(&sources, &cluster)?) + .await?; + // Put registered receipts and client-floor maintenance in the archived + // LTX tail, after the independent base checkpoint was uploaded. + if let Some(workload) = workload.as_mut() { + Box::pin(workload.populate(&mut cluster, &mut expected)).await?; + } + let group = cluster.voters[0] + .log + .cell_group(&cluster.birth.route().history)?; + let uploaded = upload + .upload_tail( + &cluster.domain, + &tails, + &checkpoint, + &group, + cluster.voters[0].owner.as_ref().unwrap().manifest(), + ) + .await?; + drop(group); + let destination = destination(&cluster).await?; + let receivers = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let coverage = upload + .prepare_publication( + &cluster.domain, + &tails, + &uploaded, + receiver(&receivers, &destination, &cluster)?, + ) + .await? + .commit(&cluster.domain) + .await?; + let hold = coverage.hold().clone(); + drop(coverage); + let route = cluster.birth.route(); + let target = cluster.directory.prepare_group_birth(CellGroupBirthSpec { + history: GroupHistory { + group: GroupId::new([70; 16])?, + incarnation: IncarnationId::new([71; 16])?, + ..route.history + }, + range: route.range.clone(), + epoch: RouteEpoch::new(2)?, + creation: ProposalId::new([72; 16])?, + voters: (11..=13) + .map(|n| InitialMember { + member: MemberId::new(n).unwrap(), + node: NodeId::new([n as u8 + 100; 16]).unwrap(), + }) + .collect(), + learners: vec![], + })?; + let view = cluster.domain.run(cluster.directory.observe()).await??; + let request = cluster.directory.prepare_group_recovery( + &view, + ProposalId::new([73; 16])?, + &cluster.birth, + &target, + hold.clone(), + )?; + let alternate = cluster.directory.prepare_group_recovery( + &view, + ProposalId::new([74; 16])?, + &cluster.birth, + &target, + hold, + )?; + let competing_birth = cluster.directory.prepare_group_birth(CellGroupBirthSpec { + history: GroupHistory { + group: GroupId::new([80; 16])?, + incarnation: IncarnationId::new([81; 16])?, + ..route.history + }, + range: route.range.clone(), + epoch: RouteEpoch::new(2)?, + creation: ProposalId::new([82; 16])?, + voters: target.voters().to_vec(), + learners: vec![], + })?; + let competing = cluster.directory.prepare_replacement( + &view, + ProposalId::new([75; 16])?, + std::slice::from_ref(route), + std::slice::from_ref(competing_birth.route()), + )?; + cluster + .domain + .run(cluster.directory.reserve_group_birth(&competing_birth)) + .await??; + let saved: Vec = persist_archive_intent(&cluster, &request.encode().to_vec()).await?; + assert_eq!(saved, request.encode()); + let decoded = cluster.directory.decode_group_recovery(&saved)?; + assert_eq!(decoded.digest(), request.digest()); + assert_eq!(decoded.hold(), request.hold()); + Ok(Self { + cluster, + expected, + remote, + archives, + tails, + receivers, + destination, + request, + alternate, + competing, + checkpoint, + authority: Default::default(), + original_head: view.digest(), + }) + } + + fn fence<'a>( + &'a self, + directory: &'a RangeDirectory, + request: &'a CellGroupRecoveryRequest, + ) -> futures_util::future::BoxFuture<'a, Result> + { + // Repeated by-value recovery futures made the debug caller's polling + // frame exceed the default test-thread stack before its first poll. + // Keep the complete admitted operation owned behind this one boundary. + Box::pin(async move { + directory + .fence_group_recovery( + &self.cluster.domain, + &self.archives, + &self.tails, + request, + receiver(&self.receivers, &self.destination, &self.cluster).unwrap(), + &self.authority, + ) + .await + }) + } + + async fn reopen(&self) -> Result { + let catalog = Catalog::new(self.remote.clone(), Default::default())?; + let namespace = self + .cluster + .domain + .run(catalog.existing_namespace(self.cluster.directory.namespace_config().clone())) + .await??; + Ok(RangeDirectory::new(namespace, Default::default())?) + } + + async fn assert_closed(&self) -> Result<()> { + let cluster = &self.cluster; + assert!(matches!( + cluster + .domain + .run( + cluster + .directory + .require_current_group_birth(&cluster.birth) + ) + .await?, + Err(CatalogError::Fenced) + )); + assert!(cluster + .domain + .run( + cluster + .directory + .materialize_selected_group(cluster.birth.route()) + ) + .await? + .is_err()); + assert!(cluster + .domain + .run( + cluster + .directory + .bootstrap_group_member(&cluster.birth, &cluster.voters[0].log) + ) + .await? + .is_err()); + assert!(cluster + .domain + .run( + cluster + .directory + .materialize_group_birth(self.request.replacement()) + ) + .await? + .is_err()); + assert!(cluster + .domain + .run(cluster.directory.publish(&self.competing)) + .await? + .is_err()); + let before = self.authority.fences.load(Ordering::Acquire); + assert!(self + .fence(&cluster.directory, &self.alternate) + .await + .is_err()); + assert_eq!(self.authority.fences.load(Ordering::Acquire), before); + let catalog = Catalog::new(self.remote.clone(), Default::default())?; + let namespace = cluster + .domain + .run(catalog.existing_namespace(self.cluster.directory.namespace_config().clone())) + .await??; + assert!(matches!( + cluster + .domain + .run(namespace.freeze_births(ProposalId::new([80; 16])?, Duration::from_secs(5))) + .await?, + Err(CatalogError::Conflict( + "disaster namespace cannot enter managed ownership" + )) + )); + Ok(()) + } + + async fn finish(self) -> Result<()> { + assert_eq!(self.receivers.usage().sources, 0); + assert_eq!(self.tails.usage().operations, 0); + self.destination.store.close(); + self.cluster.close().await + } +} + +pub(super) async fn cancel_import(rooted: bool) -> Result<()> { + let s = Box::new(Scenario::new(rooted, Arc::new(object_store::memory::InMemory::new())).await?); + let gate = fault::Gate::new(None); + let target = destination_at( + &s.cluster, + rooted, + Arc::new(fault::FaultFileSystem { gate: gate.clone() }), + s.cluster._root.path().join("recovery-cancel"), + s.request.replacement().route().clone(), + ) + .await?; + let pool = CellGroupRecoveryPool::new(Default::default())?; + let scratch = ScratchPool::new(ScratchLimits { + max_bytes: 32 * 1024 * 1024, + max_files: 1, + })?; + let proof = s.fence(&s.cluster.directory, &s.request).await?; + let release = gate.arm(fault::HOLD_DEPENDENCY_SYNC); + let mut work = Box::pin(pool.prepare( + &s.cluster.domain, + proof, + target.store, + scratch.clone(), + s.cluster._root.path().into(), + )); + let entered = Box::pin(tokio::time::timeout( + Duration::from_secs(15), + gate.entered.notified(), + )); + match futures_util::future::select(entered, work.as_mut()).await { + futures_util::future::Either::Left((result, _)) => result?, + futures_util::future::Either::Right(_) => { + anyhow::bail!("import skipped physical dependency sync") + } + } + drop(work); + assert_eq!(pool.usage().operations, 1); + assert_eq!(s.receivers.usage().sources, 1); + drop(release); + let s = *s; + let receivers = s.receivers.clone(); + s.cluster.close().await?; + assert_eq!(pool.usage().operations, 0); + assert_eq!(receivers.usage().sources, 0); + assert_eq!(scratch.usage()?.reserved_bytes, 0); + Ok(()) +} + +pub(super) async fn import(rooted: bool, store: Arc) -> Result<()> { + let s = Box::new(Scenario::new(rooted, store).await?); + let route = s.request.replacement().route().clone(); + let target = destination_at( + &s.cluster, + rooted, + Arc::new(DirectFileSystem), + s.cluster._root.path().join("recovery-import"), + route.clone(), + ) + .await?; + let pool = CellGroupRecoveryPool::new(Default::default())?; + let scratch = ScratchPool::new(ScratchLimits { + max_bytes: 32 * 1024 * 1024, + max_files: 1, + })?; + let proof = s.fence(&s.cluster.directory, &s.request).await?; + let retained_proof = proof.clone(); + let bounded = CellGroupRecoveryPool::new(CellGroupRecoveryLimits { + max_image_bytes: 512, + max_restore_bytes: 512, + ..Default::default() + })?; + assert!(matches!( + bounded + .prepare( + &s.cluster.domain, + proof.clone(), + target.store.clone(), + scratch.clone(), + s.cluster._root.path().into() + ) + .await, + Err(CellGroupArchiveError::Limit) + )); + assert_eq!(bounded.usage().operations, 0); + let source = proof.coverage().recovery().manifest().clone(); + let preparation = pool + .prepare( + &s.cluster.domain, + proof, + target.store.clone(), + scratch.clone(), + s.cluster._root.path().into(), + ) + .await?; + let marker = preparation.manifest().marker(); + let expected_marker = marker.clone(); + let expected_commitment = preparation.commitment(); + assert_eq!(marker.route(), &route); + assert_eq!( + marker.position(), + LogPosition { + index: LogIndex::new(1), + term: Term::new(1) + } + ); + assert_eq!(marker.membership_revision(), 1); + assert_eq!(marker.recovery_request(), Some(s.request.digest())); + assert!(!marker.is_active()); + assert_ne!(preparation.commitment(), [0; 32]); + assert_eq!(preparation.request().encode(), s.request.encode()); + assert_eq!(pool.usage().operations, 1); + assert_eq!(scratch.usage()?.reserved_bytes, 0); + tail_fixture::check_images(&s.cluster, preparation.manifest(), &s.expected).await?; + let keys = VersionKeys::new( + VersionSigningKey::new([91; 32], 100_000)?, + vec![], + Duration::from_secs(10), + )?; + for name in s.expected.keys() { + let cell = CellId::new(name)?; + let (old, old_pin) = source.pin_cell(&cell).context("old leaf")?; + let (new, new_pin) = preparation.manifest().pin_cell(&cell).context("new leaf")?; + assert_eq!(new.version.history.incarnation, route.history.incarnation); + assert_ne!(new.version.history, old.version.history); + assert_eq!(new.version.sequence.get(), 1); + assert_eq!(new.enrollment.cell(), &new.version.history); + assert_eq!(new.enrollment.creation(), s.request.operation()); + assert_eq!(new.enrollment.initial(), new.version.image); + assert_eq!(new.effects, old.effects); + assert_eq!(new.geometry, old.geometry); + assert_eq!(new.witness.group, route.history); + assert_eq!(new.witness.position, marker.position()); + assert_eq!(new.witness.proposal, s.request.operation()); + assert_eq!(new.last_mutation, marker.position()); + match (old_pin, new_pin) { + (CellImagePin::Materialized(old), CellImagePin::Materialized(new)) => { + assert_ne!(old.id(), new.id()); + assert_eq!(old.state(), new.state()); + } + (CellImagePin::Extents(old), CellImagePin::Extents(new)) => { + assert_ne!(old.reference(), new.reference()); + assert_ne!( + old.manifest().binding().history, + new.manifest().binding().history + ); + for n in 0..old.manifest().geometry().extent_count() { + assert_eq!(old.manifest().extent(n), new.manifest().extent(n)); + } + } + (CellImagePin::Empty, CellImagePin::Empty) => {} + _ => anyhow::bail!("recovery changed image representation"), + } + let old_token = keys.issue_cell(&old.observation(), route.schema_digest, 10)?; + assert_eq!( + keys.verify_cell(&old_token, &new.version.history, route.schema_digest, 11), + Err(VersionError::WrongHistory) + ); + } + // Holding the result owns its admission. Refusal precedes physical work; + // the replacement still has no local Raft mapping or startup permission. + assert!(matches!( + pool.prepare( + &s.cluster.domain, + retained_proof, + target.store.clone(), + scratch.clone(), + s.cluster._root.path().into() + ) + .await, + Err(CellGroupArchiveError::Busy) + )); + for voter in &s.cluster.voters { + assert!(voter.log.cell_group(&route.history).is_err()); + } + let retained = preparation.manifest().clone(); + drop(preparation); + assert_eq!(pool.usage().operations, 1); + drop(retained); + assert_eq!(pool.usage().operations, 0); + // Exact retry produces the same imported genesis/commitment. A wrong route + // cannot turn the fenced old data owner into a replacement data owner. + let proof = s.fence(&s.cluster.directory, &s.request).await?; + assert!(pool + .prepare( + &s.cluster.domain, + proof, + s.destination.store.clone(), + scratch.clone(), + s.cluster._root.path().into() + ) + .await + .is_err()); + assert_eq!(pool.usage().operations, 0); + let proof = s.fence(&s.cluster.directory, &s.request).await?; + let retried = pool + .prepare( + &s.cluster.domain, + proof, + target.store, + scratch, + s.cluster._root.path().into(), + ) + .await?; + tail_fixture::check_images(&s.cluster, retried.manifest(), &s.expected).await?; + assert!(!retried.manifest().marker().is_active()); + assert_eq!(retried.manifest().marker(), &expected_marker); + assert_eq!(retried.commitment(), expected_commitment); + drop(retried); + drop(source); + s.assert_closed().await?; + assert_eq!(pool.usage().operations, 0); + assert_eq!(s.receivers.usage().sources, 0); + let s = *s; + s.cluster.close().await +} + +pub(super) async fn resume(rooted: bool, store: Arc) -> Result<()> { + let s = Box::pin(Scenario::new(rooted, store)).await?; + let (path, index) = index(s.remote.as_ref(), &s.checkpoint).await?; + let dependency = cell_objects(&path, &index, "beta")[0].clone(); + let bytes = s.remote.get(&dependency).await?.bytes().await?; + let mut corrupt = bytes.to_vec(); + corrupt[0] ^= 1; + s.remote.put(&dependency, corrupt.into()).await?; + assert!(s.fence(&s.cluster.directory, &s.request).await.is_err()); + assert_eq!(s.authority.fences.load(Ordering::Acquire), 0); + assert!(!s.authority.permanently_stopped.load(Ordering::Acquire)); + assert_eq!( + s.cluster + .domain + .run(s.cluster.directory.observe()) + .await?? + .digest(), + s.original_head + ); + s.cluster + .domain + .run( + s.cluster + .directory + .require_current_group_birth(&s.cluster.birth), + ) + .await??; + s.remote.put(&dependency, bytes.into()).await?; + + // Both the catalog CAS and the external stop can commit before their + // responses are lost. Read back the exact lock, then resume that same fence. + s.authority.lose_response.store(true, Ordering::Release); + let head = s + .remote + .list(None) + .try_collect::>() + .await? + .into_iter() + .find(|meta| meta.location.as_ref().ends_with("/cell-groups/head")) + .context("directory head")? + .location; + let gate = s.remote.hold_put(head); + let mut work = s.fence(&s.cluster.directory, &s.request); + let entered = Box::pin(tokio::time::timeout( + Duration::from_secs(15), + gate.entered.notified(), + )); + match futures_util::future::select(entered, work.as_mut()).await { + futures_util::future::Either::Left((result, _)) => result?, + futures_util::future::Either::Right(_) => anyhow::bail!("recovery skipped topology lock"), + } + s.remote.lose_put_response.store(true, Ordering::Release); + s.remote.clear_put(); + gate.release.notify_one(); + assert!(matches!( + work.await, + Err(CellGroupRecoveryError::Catalog(CatalogError::Fence(_))) + )); + assert!(!s.remote.lose_put_response.load(Ordering::Acquire)); + assert!(s.authority.permanently_stopped.load(Ordering::Acquire)); + assert_eq!(s.authority.fences.load(Ordering::Acquire), 1); + s.assert_closed().await?; + + s.authority.reject.store(true, Ordering::Release); + assert!(s.fence(&s.cluster.directory, &s.request).await.is_err()); + assert_eq!(s.authority.fences.load(Ordering::Acquire), 2); + assert_eq!(s.authority.verifications.load(Ordering::Acquire), 1); + s.assert_closed().await?; + s.authority.reject.store(false, Ordering::Release); + let proof = s.fence(&s.cluster.directory, &s.request).await?; + assert_eq!(proof.request().encode(), s.request.encode()); + assert_eq!(proof.coverage().marker(), s.request.hold().marker()); + tail_fixture::check_images( + &s.cluster, + proof.coverage().recovery().manifest(), + &s.expected, + ) + .await?; + for voter in &s.cluster.voters { + assert_eq!( + voter.owner.as_ref().unwrap().manifest().marker(), + proof.coverage().marker() + ); + assert!(voter + .log + .cell_group(&s.request.replacement().route().history) + .is_err()); + } + drop(proof); + assert_eq!(s.authority.fences.load(Ordering::Acquire), 3); + assert_eq!(s.authority.verifications.load(Ordering::Acquire), 3); + s.assert_closed().await?; + + // Fresh catalog owner reconstructs intent from durable bytes and rechecks + // complete physical data and the winning receipt, without calling fence. + let directory = s.reopen().await?; + let request = directory.decode_group_recovery(s.request.encode())?; + s.authority.reject.store(true, Ordering::Release); + assert!(s.fence(&directory, &request).await.is_err()); + assert_eq!(s.authority.fences.load(Ordering::Acquire), 3); + s.authority.reject.store(false, Ordering::Release); + let resumed = s.fence(&directory, &request).await?; + tail_fixture::check_images( + &s.cluster, + resumed.coverage().recovery().manifest(), + &s.expected, + ) + .await?; + assert_eq!(resumed.request().encode(), s.request.encode()); + drop(resumed); + assert_eq!(s.authority.fences.load(Ordering::Acquire), 3); + assert_eq!(s.authority.verifications.load(Ordering::Acquire), 5); + s.finish().await +} + +pub(super) async fn topology_wins( + rooted: bool, + store: Arc, +) -> Result<()> { + let s = Box::pin(Scenario::new(rooted, store)).await?; + let head = s + .remote + .list(None) + .try_collect::>() + .await? + .into_iter() + .find(|meta| meta.location.as_ref().ends_with("/cell-groups/head")) + .context("directory head")? + .location; + let gate = s.remote.hold_put(head); + let mut work = Box::pin(s.fence(&s.cluster.directory, &s.request)); + let entered = Box::pin(tokio::time::timeout( + Duration::from_secs(15), + gate.entered.notified(), + )); + match futures_util::future::select(entered, work.as_mut()).await { + futures_util::future::Either::Left((result, _)) => result?, + futures_util::future::Either::Right(_) => anyhow::bail!("recovery skipped topology CAS"), + } + s.remote.clear_put(); + s.cluster + .domain + .run(s.cluster.directory.publish(&s.competing)) + .await??; + gate.release.notify_one(); + assert!(work.await.is_err()); + assert_eq!(s.authority.fences.load(Ordering::Acquire), 0); + assert!(!s.authority.permanently_stopped.load(Ordering::Acquire)); + assert!(s.fence(&s.cluster.directory, &s.request).await.is_err()); + assert_eq!(s.authority.fences.load(Ordering::Acquire), 0); + s.finish().await +} diff --git a/crates/cellar/tests/support/cell_group_disaster_public.rs b/crates/cellar/tests/support/cell_group_disaster_public.rs new file mode 100644 index 000000000..ae483947e --- /dev/null +++ b/crates/cellar/tests/support/cell_group_disaster_public.rs @@ -0,0 +1,663 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Public Node startup owns recovered activation and its shutdown lifetime. +//! These are real files and quorum RPCs; fencing is a contract fixture. +use super::*; +use cellar::sdk::{ + consensus::{ + CellGroupNodeArchiveOptions, CommandOptions, QueryConsistency, QueryOptions, RaftCells, + RaftCellsConfig, RaftCellsServices, RaftGroupAssignment, RaftGroupPlacement, + RaftNamespaceConfig, RaftPeerNode, RaftTls, + }, + Application, Cell, CellIdentity, Context as AppContext, HandlerFuture, Invocation, Node, + NodeRaftConfig, NodeRaftListeners, Output, ShutdownMode, +}; + +#[path = "cell_group_disaster_public_node.rs"] +mod node; + +struct App(Arc); + +impl Application for App { + fn invoke(self: Arc, _: AppContext, _: Invocation) -> HandlerFuture { + Box::pin(async { Ok(Output::Done) }) + } + + fn supports_cell(&self, _: &str) -> bool { + false + } + + fn create_cell(&self, _: AppContext, _: &CellIdentity) -> Result> { + anyhow::bail!("only explicit Raft definitions") + } + + fn resolve_raft_class(&self, name: &str) -> Result, RegistrationError> { + if name == "RecoveryCounter" { + Ok(self.0.clone()) + } else { + Err(RegistrationError::UnknownClass(name.into())) + } + } +} + +fn tls() -> Result { + Ok(RaftTls::new( + vec![include_bytes!("../../transport/tcp/fixtures/ca.der") + .to_vec() + .into()], + vec![include_bytes!("../../transport/tcp/fixtures/server.der") + .to_vec() + .into()], + rustls::pki_types::PrivatePkcs8KeyDer::from( + include_bytes!("../../transport/tcp/fixtures/key.der").to_vec(), + ) + .into(), + )?) +} + +async fn listeners() -> Result { + Ok(NodeRaftListeners { + control: tokio::net::TcpListener::bind("127.0.0.1:0").await?, + data: tokio::net::TcpListener::bind("127.0.0.1:0").await?, + query: tokio::net::TcpListener::bind("127.0.0.1:0").await?, + }) +} + +fn peer( + s: &Scenario, + ordinal: usize, + endpoints: &NodeRaftListeners, + generation: u8, +) -> Result { + Ok(RaftPeerNode { + node: s.request.replacement().voters()[ordinal].node, + disk: DiskLineage::new([210 + ordinal as u8; 16])?, + process: ProcessGeneration::new([generation; 16])?, + label: format!("fresh-{ordinal}"), + server_name: "localhost".into(), + control_address: endpoints.control.local_addr()?, + data_address: endpoints.data.local_addr()?, + query_address: endpoints.query.local_addr()?, + }) +} + +async fn start( + s: &Scenario, + node: &Node, + ordinal: usize, + endpoints: NodeRaftListeners, + peers: Vec, + app: Arc, + rooted: bool, + authority: Option>, +) -> Result { + let local = &peers[ordinal]; + let mut log = ReplicaLogOptions::new( + node.info().data_dir.join("raft"), + local.node, + local.disk, + local.process, + LogOpenMode::Resume, + ); + log.max_groups = 2; + let mut host = cellar::sdk::consensus::RaftHostOptions { + max_groups: 2, + max_input_bytes: 1024, + ..Default::default() + }; + host.execution.envelope.max_payload_bytes = 1024 * 1024; + host.archive.limits.envelope = host.execution.envelope; + host.node_profile = Some(cellar::sdk::consensus::RaftNodeProfile { + pool: "recovery-fixture".into(), + region: "local".into(), + zone: format!("voter-{ordinal}"), + group_capacity: u32::try_from(host.max_groups)?, + byte_capacity: host.images.max_bytes * host.max_groups as u64, + bytes_per_group: host.images.max_bytes, + leader_capacity: u32::try_from(host.max_groups)?, + draining: false, + }); + if rooted { + host.extents = Some(Default::default()); + } + let mut shared = node::limits(); + shared.entries.envelope = host.execution.envelope; + shared.checkpoints = Some(checkpoint_fixture::limits()); + shared.archive = Some(CellGroupNodeArchiveOptions { + io: Default::default(), + uploads: Default::default(), + tails: Default::default(), + }); + node.start_raft_cells( + RaftCellsConfig { + infrastructure: NodeRaftConfig { + log, + consensus: RaftRuntimeOptions { + max_groups: 2, + tick_interval_ms: Some(10), + election_seed: Some([17 + ordinal as u8; 32]), + ..Default::default() + }, + transport: Default::default(), + }, + catalog: CatalogOptions::default(), + namespaces: vec![RaftNamespaceConfig { + namespace: s.cluster.directory.namespace_config().clone(), + class: "RecoveryCounter".into(), + initialize: None, + }], + groups: vec![RaftGroupAssignment { + history: s.request.replacement().route().history, + placement: RaftGroupPlacement::Replica, + }], + peers, + query_origins: vec![], + command_origins: vec![], + host, + shared, + }, + RaftCellsServices { + recovery: authority + .map(|a| a as Arc>), + listeners: endpoints, + tls: tls()?, + store: s.remote.clone(), + application: app, + versions: keys()?, + }, + ) + .await +} + +async fn stopped(nodes: Vec, hosts: Vec) -> Result<()> { + for node in nodes { + node.shutdown(ShutdownMode::Preserve).await?; + } + for host in hosts { + let report = host.close().await?; + assert_eq!( + report.mutation_outcomes, + cellar::sdk::consensus::MutationOutcomes::NoUnresolvedLocalProducers + ); + } + Ok(()) +} + +async fn leader(s: &Scenario, hosts: &[RaftCells]) -> Result { + tokio::time::timeout(Duration::from_secs(15), async { + loop { + for (ordinal, host) in hosts.iter().enumerate() { + let observed = host + .group(s.request.replacement().route().history)? + .observe(Duration::from_secs(5)) + .await?; + if observed.status.ready && observed.role == RaftReplicaRole::Leader { + return Ok(ordinal); + } + } + tokio::time::sleep(Duration::from_millis(2)).await; + } + }) + .await? +} + +async fn pending_shutdown( + s: &Scenario, + root: &std::path::Path, + app: Arc, + rooted: bool, + authority: Arc, +) -> Result<()> { + let node = node::node_named(root, app.clone(), "fresh-0").await?; + let mut endpoints = vec![listeners().await?, listeners().await?, listeners().await?]; + let peers = endpoints + .iter() + .enumerate() + .map(|(i, e)| peer(s, i, e, 221)) + .collect::>>()?; + let host = start( + s, + &node, + 0, + endpoints.remove(0), + peers, + app, + rooted, + Some(authority.clone()), + ) + .await?; + let gate = Arc::new(archive_store::Gate::default()); + *authority.held_verification.lock().unwrap() = Some(gate.clone()); + tokio::time::timeout(Duration::from_secs(5), gate.entered.notified()).await?; + let mut closing = Box::pin(node.shutdown(ShutdownMode::Preserve)); + assert!( + tokio::time::timeout(Duration::from_millis(30), &mut closing) + .await + .is_err(), + "shutdown dropped entered recovery verification" + ); + gate.release.notify_one(); + tokio::time::timeout(Duration::from_secs(5), closing).await??; + assert_eq!( + host.close().await?.mutation_outcomes, + cellar::sdk::consensus::MutationOutcomes::NoUnresolvedLocalProducers + ); + assert!( + s.cluster + .domain + .run(s.cluster.directory.group_recovery_pending(&s.request)) + .await?? + ); + Ok(()) +} + +async fn serving( + s: &Scenario, + hosts: &[RaftCells], + w: &ReceiptWorkload, + generation: u64, +) -> Result<()> { + let leader = leader(s, hosts).await?; + let route = s.request.replacement().route(); + let mut cell = w.old["alpha"].version.history.clone(); + cell.incarnation = route.history.incarnation; + let handle = hosts[leader].cell(cell.clone())?; + let read = handle + .query( + ProposalId::new([230 + generation as u8; 16])?, + &handle.contract("Get")?, + vec![], + QueryOptions { + consistency: QueryConsistency::Linearizable, + ..Default::default() + }, + ) + .await?; + assert_eq!(read.result, (3 + generation as i64).to_be_bytes()); + assert_eq!(read.version.history, cell); + assert_eq!(read.witness.group, route.history); + let old = &w.old["alpha"]; + let token = keys()?.issue_cell(old, route.schema_digest, w.now)?; + let now = s + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms().max(0) as u64 }) + .await?; + assert_eq!( + keys()?.verify_cell(&token, &old.version.history, route.schema_digest, now)?, + *old + ); + assert_eq!( + keys()?.verify_cell(&token, &cell, route.schema_digest, now), + Err(VersionError::WrongHistory) + ); + assert!(handle + .query( + ProposalId::new([235 + generation as u8; 16])?, + &handle.contract("Get")?, + vec![], + QueryOptions { + consistency: QueryConsistency::AtLeast(token), + ..Default::default() + } + ) + .await + .is_err()); + let before = w.calls.load(Ordering::Acquire); + // The durable S3 receipt retains its original bytes and stronger minimum. + let duplicate = handle + .command( + &handle.contract("Add")?, + vec![2], + CommandOptions::new( + CommandId::new(ClientId::new([188; 16])?, 2)?, + w.now + 300_000, + ), + ) + .await?; + assert_eq!(duplicate.result, w.results[&("alpha".into(), 2)]); + assert_eq!(duplicate.durability, Durability::S3); + assert_eq!(duplicate.version.history, cell); + assert_eq!(w.calls.load(Ordering::Acquire), before); + let expired = handle + .command( + &handle.contract("Add")?, + vec![1], + CommandOptions::new( + CommandId::new(ClientId::new([188; 16])?, 1)?, + w.now + 300_000, + ), + ) + .await; + assert!(expired.is_err()); + let conflict = handle + .command( + &handle.contract("Add")?, + vec![99], + CommandOptions::new( + CommandId::new(ClientId::new([188; 16])?, 2)?, + w.now + 300_000, + ), + ) + .await; + assert!(conflict.is_err()); + assert_eq!(w.calls.load(Ordering::Acquire), before); + let id = CommandId::new(ClientId::new([189; 16])?, generation)?; + let added = handle + .command( + &handle.contract("Add")?, + vec![7], + CommandOptions::new(id, w.now + 300_000), + ) + .await?; + assert_eq!(added.version.history, cell); + assert_eq!(added.witness.group, route.history); + let mut expected = cell.cell.as_str().as_bytes().to_vec(); + expected.extend_from_slice(&cell.incarnation.bytes()); + expected.push(7); + assert_eq!(added.result, expected); + assert_eq!(w.calls.load(Ordering::Acquire), before + 1); + let replay = handle + .command( + &handle.contract("Add")?, + vec![7], + CommandOptions::new(id, w.now + 300_000), + ) + .await?; + assert_eq!(replay.id, added.id); + assert_eq!(replay.result, added.result); + assert_eq!(replay.version, added.version); + assert_eq!(replay.witness, added.witness); + assert_eq!(replay.durability, added.durability); + assert_eq!(w.calls.load(Ordering::Acquire), before + 1); + let read = handle + .query( + ProposalId::new([240 + generation as u8; 16])?, + &handle.contract("Get")?, + vec![], + QueryOptions { + consistency: QueryConsistency::Linearizable, + ..Default::default() + }, + ) + .await?; + assert_eq!(read.result, (4 + generation as i64).to_be_bytes()); + // Other cells in the same physical group keep independent state and floors. + let mut beta = w.old["beta"].version.history.clone(); + beta.incarnation = cell.incarnation; + let handle = hosts[leader].cell(beta.clone())?; + let read = handle + .query( + ProposalId::new([245 + generation as u8; 16])?, + &handle.contract("Get")?, + vec![], + QueryOptions { + consistency: QueryConsistency::Linearizable, + ..Default::default() + }, + ) + .await?; + assert_eq!(read.result, 2i64.to_be_bytes()); + assert_eq!(read.version.history, beta); + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn sdk_recovery_owns_activation_after_erasing_voters() -> Result<()> { + for rooted in [false, true] { + Box::pin(drill( + rooted, + Arc::new(object_store::memory::InMemory::new()), + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[ignore = "requires isolated local RustFS"] +async fn rustfs_sdk_recovery_owns_activation_after_erasing_voters() -> Result<()> { + for rooted in [false, true] { + Box::pin(drill(rooted, rustfs::store("sdk-recovery", rooted)?)).await?; + } + Ok(()) +} + +async fn drill(rooted: bool, store: Arc) -> Result<()> { + let mut w = ReceiptWorkload::new()?; + let class = Arc::try_unwrap(w.class).map_err(|_| anyhow::anyhow!("unshared workload"))?; + w.class = Arc::new(class.query("Get", [86; 32], |context, _| { + let n: i64 = context.query_row("SELECT n FROM value", [], |row| row.get(0))?; + Ok(n.to_be_bytes().to_vec()) + })?); + let mut s = Box::new(Box::pin(Scenario::with_workload(rooted, store, Some(&mut w))).await?); + let (seeds, logs, reopen) = super::super::seed_voters(&mut s, rooted).await?; + drop(seeds); + for log in &logs { + log.close().await?; + } + drop(logs); + drop(reopen); + s.cluster + .domain + .run( + s.cluster + .directory + .publish_group_recovery(&s.request, &s.authority), + ) + .await??; + let birth = s + .cluster + .domain + .run( + s.cluster + .directory + .materialize_group_birth(s.request.replacement()), + ) + .await??; + let binding = s + .cluster + .directory + .prepare_group_archive_binding(&s.cluster.domain, &s.archives, &birth) + .await?; + let saved = binding.encode().to_vec(); + let intent = s.cluster._root.path().join("sdk-archive-initialization"); + s.cluster + .domain + .blocking(move || -> Result<()> { + use std::io::Write; + let mut file = std::fs::OpenOptions::new() + .create_new(true) + .write(true) + .open(&intent)?; + file.write_all(&saved)?; + file.sync_all()?; + std::fs::File::open(intent.parent().context("intent parent")?)?.sync_all()?; + Ok(()) + })? + .await??; + binding.commit(&s.cluster.domain).await?; + let authority = Arc::new(std::mem::take(&mut s.authority)); + let app: Arc = Arc::new(App(w.class.clone())); + let roots = (0..3) + .map(|n| s.cluster._root.path().join(format!("sdk-fresh-{n}"))) + .collect::>(); + // Move complete offline stores without rewriting any pin or genesis. These + // are the exact fresh stores created after all source voter files were erased. + for (ordinal, root) in roots.iter().enumerate() { + let node = node::node_named(root, app.clone(), &format!("fresh-{ordinal}")).await?; + let target = node.info().data_dir.clone(); + node.shutdown(ShutdownMode::Preserve).await?; + let source = s.cluster._root.path().join(format!("fresh-{ordinal}")); + let route = s.request.replacement().route().history; + s.cluster + .domain + .blocking(move || -> Result<()> { + let physical = target.join("raft-cells"); + std::fs::create_dir_all(&physical)?; + std::fs::rename(source.join("log"), target.join("raft"))?; + let pages = physical + .join("groups") + .join(hex::encode(route.cluster.bytes())) + .join(hex::encode(route.namespace.bytes())) + .join(hex::encode(route.group.bytes())) + .join(hex::encode(route.incarnation.bytes())) + .join("pages"); + std::fs::create_dir_all(pages.parent().context("page parent")?)?; + std::fs::rename(source.join("data/pages"), pages)?; + if rooted { + std::fs::rename( + source.join("data/extents"), + physical.join("extent-inventory"), + )?; + } else { + std::fs::rename(source.join("data/images"), physical.join("images"))?; + } + Ok(()) + })? + .await??; + } + // Missing authority and a rejected independent fence both refuse the same + // intact seeded disk; a catalog publication alone cannot grant serving. + let before_refusals = s + .cluster + .domain + .run(s.cluster.directory.observe()) + .await?? + .digest(); + for supplied in [None, Some(authority.clone())] { + let before_verifications = authority.verifications.load(Ordering::Acquire); + let rejected_authority = supplied.is_some(); + authority + .reject + .store(supplied.is_some(), Ordering::Release); + let node = node::node_named(&roots[0], app.clone(), "fresh-0").await?; + let mut all_endpoints = vec![listeners().await?, listeners().await?, listeners().await?]; + let peers = all_endpoints + .iter() + .enumerate() + .map(|(i, e)| peer(&s, i, e, 221)) + .collect::>>()?; + let endpoints = all_endpoints.remove(0); + let refusal = start( + &s, + &node, + 0, + endpoints, + peers, + app.clone(), + rooted, + supplied, + ) + .await; + let error = refusal + .err() + .context("missing or rejected authority must refuse startup")?; + eprintln!("recovered-startup refusal: {error:#}"); + node.shutdown(ShutdownMode::Preserve).await?; + assert_eq!( + authority.verifications.load(Ordering::Acquire) > before_verifications, + rejected_authority + ); + assert_eq!( + s.cluster + .domain + .run(s.cluster.directory.observe()) + .await?? + .digest(), + before_refusals + ); + } + authority.reject.store(false, Ordering::Release); + Box::pin(pending_shutdown( + &s, + &roots[0], + app.clone(), + rooted, + authority.clone(), + )) + .await?; + assert_eq!(w.calls.load(Ordering::Acquire), 4); + for generation in 1..=2 { + let mut nodes = Vec::new(); + let mut endpoints = Vec::new(); + for (ordinal, root) in roots.iter().enumerate() { + nodes.push(node::node_named(root, app.clone(), &format!("fresh-{ordinal}")).await?); + endpoints.push(listeners().await?); + } + let peers = endpoints + .iter() + .enumerate() + .map(|(ordinal, e)| peer(&s, ordinal, e, 221 + generation as u8)) + .collect::>>()?; + let mut hosts = Vec::new(); + for (ordinal, e) in endpoints.into_iter().enumerate() { + hosts.push( + start( + &s, + &nodes[ordinal], + ordinal, + e, + peers.clone(), + app.clone(), + rooted, + Some(authority.clone()), + ) + .await?, + ); + if ordinal == 0 && generation == 1 { + assert!( + s.cluster + .domain + .run(s.cluster.directory.group_recovery_pending(&s.request)) + .await?? + ); + let mut cell = w.old["alpha"].version.history.clone(); + cell.incarnation = s.request.replacement().route().history.incarnation; + let handle = hosts[0].cell(cell)?; + let refusal = handle + .command( + &handle.contract("Add")?, + vec![7], + CommandOptions::new( + CommandId::new(ClientId::new([190; 16])?, 1)?, + w.now + 300_000, + ), + ) + .await; + assert!(refusal.is_err()); + assert_eq!(w.calls.load(Ordering::Acquire), 4); + } + } + tokio::time::timeout(Duration::from_secs(15), async { + loop { + if !s + .cluster + .domain + .run(s.cluster.directory.group_recovery_pending(&s.request)) + .await?? + { + return Ok::<_, anyhow::Error>(()); + } + tokio::time::sleep(Duration::from_millis(2)).await; + } + }) + .await??; + for host in &hosts { + let group = host + .group(s.request.replacement().route().history)? + .observe(Duration::from_secs(5)) + .await?; + assert_eq!(group.replica.scope, s.request.replacement().route().history); + assert_eq!(group.cells, 3); + assert!(group.archive_required); + assert_eq!( + host.observe_node(Duration::from_secs(5)).await?.node.groups, + 1 + ); + } + Box::pin(serving(&s, &hosts, &w, generation)).await?; + stopped(nodes, hosts).await?; + } + s.cluster.close().await +} diff --git a/crates/cellar/tests/support/cell_group_disaster_public_node.rs b/crates/cellar/tests/support/cell_group_disaster_public_node.rs new file mode 100644 index 000000000..8363e1bab --- /dev/null +++ b/crates/cellar/tests/support/cell_group_disaster_public_node.rs @@ -0,0 +1,168 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Public recovered-node fixture with the ordinary Node lifecycle and limits. +use anyhow::Result; +use cellar::sdk::{consensus::*, *}; +use std::{path::Path, sync::Arc, time::Duration}; +#[path = "html.rs"] +pub(super) mod graph; + +pub(super) async fn node_named( + directory: &Path, + application: Arc, + label: &str, +) -> Result { + node_with_peer_key(directory, application, label, [11; 32]).await +} + +pub(super) async fn node_with_peer_key( + directory: &Path, + application: Arc, + label: &str, + peer_key: [u8; 32], +) -> Result { + let classes = vec![]; + let filesystem = Arc::new(cellar_ltx::DirectFileSystem); + let shutdown = ShutdownPolicy::default(); + let bucket: Option = None; + + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await?; + let internal_listener = tokio::net::TcpListener::bind("127.0.0.1:0").await?; + let probe_signer = Arc::new(PeerProbeSigner::generate()); + let mut config = NodeConfig::new( + ActorIdentity { + node: label.into(), + advertise: internal_listener.local_addr()?.to_string(), + region: "test".into(), + }, + ActorOptions { + core: ActorPolicy { + max_resident: 16, + max_cached_routes: 0, + max_activations: 8, + max_evictions: 4, + max_releases: 4, + max_outbound_websockets: 8, + ownership_on_evict: OwnershipOnEvict::Sticky, + require_node_lease: true, + peer_protocol: cellar::peer_auth::PROTOCOL_VERSION, + operation_deadline_ms: Some(3000), + owner_log_recovery_backoff_ms: 50, + owner_log_recovery_attempts: 3, + alarm_resident_ms: 100, + idle_evict_ms: None, + pressure: Default::default(), + }, + lease_ttl_ms: 1000, + placement_weight: 1, + generation: probe_signer.public_key_hex(), + resume_generation: None, + }, + directory.into(), + ); + config.shutdown = shutdown; + let peer_http = reqwest::Client::builder().no_proxy().build()?; + let peer_auth = Arc::new(PeerAuth::new(peer_key, label)?); + let fleet = bucket + .map(|bucket| -> Result { + Ok(NodeFleet { + replica_storage: ReplicaStorage::new( + bucket.store.clone(), + ReplicaStorageOptions { + prefix: bucket.prefix.clone(), + ..Default::default() + }, + )?, + lease_bucket: bucket.clone(), + bucket, + replication: ReplicationOptions { + flush_interval: Duration::ZERO, + compaction: None, + ..Default::default() + }, + log_transport: Arc::new(SignedPeerTransport::from_clients( + peer_http.clone(), + peer_http.clone(), + peer_auth.clone(), + )), + log: LogOptions::default(), + fleet_durability: true, + }) + }) + .transpose()?; + let services = NodeServices { + wall_clock: None, + http_client: None, + host_executor: tokio::runtime::Handle::current(), + core_executor: tokio::runtime::Handle::current(), + listener, + internal_listener, + peer_http, + peer_auth, + probe_signer, + fleet, + containers: None, + metrics: None, + }; + let compiled = + CompiledApplications::default().register(label, "test", move || Ok(application.clone()))?; + Node::start_with_filesystem( + config, + services, + graph::graph(label, "", classes), + compiled, + filesystem, + ) + .await +} + +pub(super) fn limits() -> CellGroupNodeLimits { + CellGroupNodeLimits { + namespaces: CellGroupHostLimits { + max_namespaces: 1, + namespace_bytes: 16 * 1024 * 1024, + }, + controller: CellGroupControllerLimits { + max_groups: 2, + max_parallel_apply: 2, + retry_interval_ms: 5, + max_commands: 2, + max_command_bytes: 64 * 1024, + checkpoints: None, + }, + sql: CellSqlLimits { + workers: 2, + queue_per_worker: 4, + max_pending_cells: 8, + group_pending_cells: 8, + node_bytes: 512 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + }, + provisioning: CellGroupProvisionLimits { + max_preparations: 1, + max_preparation_bytes: 512 * 1024 * 1024, + max_parallel_preparations: 1, + max_waiters: 2, + peer_revision: 1, + }, + directory: DirectoryLimits::default(), + dispatch: CellDispatchLimits { + max_cached_routes: 32, + route_cache_bytes: 256 * 1024, + max_invocations: 8, + retained_bytes: 1024 * 1024, + max_input_bytes: 1024, + }, + manifest: ManifestLimits { + max_cells: 32, + max_depth: 16, + }, + entries: Default::default(), + manifest_pages: 1024, + manifest_bytes: 16 * 1024 * 1024, + retained_roots: 16, + max_effect_rows: 100, + checkpoints: None, + archive: None, + } +} diff --git a/crates/cellar/tests/support/cell_group_disaster_receipts.rs b/crates/cellar/tests/support/cell_group_disaster_receipts.rs new file mode 100644 index 000000000..77e3d5c22 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_disaster_receipts.rs @@ -0,0 +1,753 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Registered receipts and non-reusable client floors across a cold group seed. +use super::*; +#[path = "cell_group_disaster_public.rs"] +mod public; +use cellar::storage::CommandInputs; +use std::collections::BTreeMap; + +pub(in super::super) struct ReceiptWorkload { + pub(in super::super) class: Arc, + calls: Arc, + now: u64, + old: BTreeMap, + results: BTreeMap<(String, u64), Vec>, +} + +impl ReceiptWorkload { + fn new() -> Result { + let calls = Arc::new(AtomicUsize::new(0)); + let count = calls.clone(); + let class = Arc::new(RaftClass::new("RecoveryCounter", 1, [84; 32])?.command( + "Add", + [85; 32], + move |context, input| { + count.fetch_add(1, Ordering::AcqRel); + context.sql().execute("UPDATE value SET n=n+1", [])?; + // Include the incarnation in the result: re-execution under the + // fresh identity could not produce the archived result. + let mut result = context.identity().cell.as_str().as_bytes().to_vec(); + result.extend_from_slice(&context.identity().incarnation.bytes()); + result.extend_from_slice(input); + Ok(result) + }, + )?); + Ok(Self { + class, + calls, + now: 0, + old: BTreeMap::new(), + results: BTreeMap::new(), + }) + } + + fn request(&self, sequence: u64, input: &[u8]) -> Result { + Ok(CellCommandRequest { + id: CommandId::new(ClientId::new([188; 16])?, sequence)?, + contract: self.class.contract("Add").context("Add")?, + input: input.to_vec(), + durability: Durability::Quorum, + retry_until_ms: self.now + 300_000, + }) + } + + pub(in super::super) async fn populate( + &mut self, + cluster: &mut Cluster, + expected: &mut BTreeMap>, + ) -> Result<()> { + self.now = cluster + .domain + .run(async { cellar::asyncrt::wall_ms().max(0) as u64 }) + .await?; + for (name, sequence) in [("alpha", 1), ("alpha", 2), ("alpha", 3), ("beta", 1)] { + let cell = history(cluster.birth.route(), name)?; + let authority = cluster.voters[0] + .owner + .as_ref() + .context("source owner")? + .execution_authority(&cell)?; + let voter = &cluster.voters[0]; + let services = ReadServices { + now_unix_millis: Arc::new(|| 10), + fill_random: Arc::new(|bytes| bytes.fill(17)), + }; + let images = match &voter.extents { + Some(images) => CellGroupApplyImages::Extents { + images: images.clone(), + services, + limits: Default::default(), + cache_directory: cluster._root.path().join("node-1/source-query-cache"), + cache_pool: ScratchPool::new(ScratchLimits { + max_bytes: 2 * EXTENT_BYTES, + max_files: 2, + })?, + cache: CacheLimits { + max_cached_bytes: EXTENT_BYTES, + max_cached_extents: 1, + }, + }, + None => CellGroupApplyImages::Materialized { + images: voter.images.clone(), + services, + }, + }; + let mut options = sql_options( + images, + cluster + ._root + .path() + .join(format!("node-1/source-sql-{name}")), + self.class.schema_digest(), + )?; + match &mut options { + CellSqlOptions::Materialized(o) => o.limits.max_retained_receipts = 2, + CellSqlOptions::Extents(o) => o.limits.max_retained_receipts = 2, + } + let input = vec![sequence as u8]; + let mut request = self.request(sequence, &input)?; + request.durability = if name == "alpha" && sequence >= 2 { + Durability::S3 + } else { + Durability::Quorum + }; + if name == "alpha" && sequence == 1 { + request.retry_until_ms = self.now - 1000; + } + let attempt = CommandAttempt { + request: CommandRequest { + id: request.id, + operation: "Add".into(), + contract_digest: [85; 32], + request_digest: command_request_digest(&input), + durability: request.durability, + retry_until_ms: request.retry_until_ms, + }, + proposal: ProposalId::new([100 + self.calls.load(Ordering::Acquire) as u8; 16])?, + preview: ImageId::new([100 + self.calls.load(Ordering::Acquire) as u8; 16])?, + inputs: CommandInputs { + now_unix_millis: if sequence <= 2 && name == "alpha" { + self.now - 2000 + } else { + self.now + }, + random: [17; 32], + }, + }; + let class = self.class.clone(); + let (executor, prepared) = cluster + .domain + .blocking(move || -> Result<_> { + let mut executor = open_private(options, authority)?; + let CellStageOutcome::Prepared(prepared) = + executor.stage_registered(attempt, &class, &input, true)? + else { + anyhow::bail!("source command must execute once") + }; + executor.mark_submitted(prepared.proposal())?; + Ok((executor, prepared)) + })? + .await??; + let CellSubmissionOutcome::Accepted(accepted) = cluster.voters[0] + .owner + .as_mut() + .context("source owner")? + .submit(prepared)? + else { + anyhow::bail!("source command refused") + }; + cluster.drive(false).await?; + let authority = cluster.voters[0] + .owner + .as_ref() + .context("source owner")? + .execution_authority(&cell)?; + assert!(authority.cell().last_mutation.index >= accepted.position.index); + let receipt = cluster + .domain + .blocking(move || -> Result<_> { + let mut executor = executor; + let receipt = executor.finish(authority)?; + executor.close()?; + Ok(receipt) + })? + .await??; + self.results + .insert((name.into(), sequence), receipt.receipt.result().to_vec()); + self.old.insert(name.into(), receipt.observation()); + expected.insert( + name.into(), + image_bytes( + cluster, + cluster.voters[0] + .owner + .as_ref() + .context("source owner")? + .manifest(), + name, + ) + .await?, + ); + } + assert_eq!(self.calls.load(Ordering::Acquire), 4); + Ok(()) + } +} + +fn sql_options( + images: CellGroupApplyImages, + directory: std::path::PathBuf, + schema: [u8; 32], +) -> Result { + let host = cellar_ltx::LtxHost::new( + || 10, + |_| Ok(Duration::ZERO), + |path| async move { std::fs::read(path) }, + |job| async move { + job(); + Ok(()) + }, + ) + .with_filesystem(Arc::new(DirectFileSystem)); + let mut limits = ExecutionLimits::default(); + // This small workload has a 1 MiB capture contract. The default 16 MiB + // payload contract alone reserves four buffers, exceeding the fixture's + // unchanged 64 MiB group SQL budget before receipt lookup can run. + limits.envelope.max_payload_bytes = 1024 * 1024; + Ok(match images { + CellGroupApplyImages::Materialized { images, services } => { + CellSqlOptions::Materialized(PrivateOptions { + directory, + host, + sqlite_vfs: None, + images, + query_services: services, + application_digest: schema, + limits, + }) + } + CellGroupApplyImages::Extents { + images, + services, + cache, + .. + } => CellSqlOptions::Extents(CellSqlRootOptions { + host, + sqlite_vfs: None, + images, + query_services: services, + application_digest: schema, + limits, + cache_directory: directory.with_extension("read-cache"), + // The writer and immutable receipt reader may coexist. Neither may + // borrow the applier's independently admitted single-file cache. + cache_pool: ScratchPool::new(ScratchLimits { + max_bytes: 2 * EXTENT_BYTES, + max_files: 2, + })?, + cache, + }), + }) +} + +fn open_private( + options: CellSqlOptions, + authority: CellExecutionAuthority, +) -> Result { + Ok(match options { + CellSqlOptions::Materialized(o) => CellPrivateExecutor::open(o, authority)?, + CellSqlOptions::Extents(o) => { + let workspace = o.images.create_workspace( + cell_image_history_digest(&authority.cell().version.history), + RootPrivateOptions::workspace_bounds(o.limits)?, + )?; + CellPrivateExecutor::open_rooted( + RootPrivateOptions { + workspace, + host: o.host, + sqlite_vfs: o.sqlite_vfs, + images: o.images, + query_services: o.query_services, + application_digest: o.application_digest, + limits: o.limits, + cache_directory: o.cache_directory, + cache_pool: o.cache_pool, + cache: o.cache, + }, + authority, + )? + } + }) +} + +async fn image_bytes( + cluster: &Cluster, + root: &PreparedGroupManifest, + name: &str, +) -> Result> { + let (_, pin) = root + .pin_cell(&CellId::new(name)?) + .context("selected image")?; + pin_bytes(cluster, pin).await +} + +async fn pin_bytes(cluster: &Cluster, pin: CellImagePin) -> Result> { + cluster + .domain + .blocking(move || -> Result<_> { + Ok(match pin { + CellImagePin::Empty => vec![], + CellImagePin::Materialized(pin) => pin + .open_reader()? + .read_exact_at(0, pin.state().length as usize)?, + CellImagePin::Extents(pin) => { + let mut bytes = Vec::new(); + for ordinal in 0..pin.manifest().geometry().extent_count() { + bytes.extend( + pin.view() + .read_extent(pin.manifest().extent(ordinal).context("extent")?)?, + ); + } + bytes + } + }) + })? + .await? +} + +fn keys() -> Result { + Ok(VersionKeys::new( + VersionSigningKey::new([89; 32], u64::MAX)?, + vec![], + Duration::from_secs(600), + )?) +} + +async fn controllers_applied( + voters: &[FreshVoter], + history: GroupHistory, + index: LogIndex, +) -> Result<()> { + tokio::time::timeout(Duration::from_secs(15), async { + loop { + let mut done = true; + for voter in voters { + done &= voter.runtime.status(history).await?.applied.index >= index; + } + if done { + return Ok::<_, anyhow::Error>(()); + } + tokio::time::sleep(Duration::from_millis(2)).await; + } + }) + .await? +} + +struct Serving { + controllers: Vec, + transports: Vec, + dispatcher: CellGroupDispatcher, +} + +impl Serving { + async fn start( + s: &Scenario, + quorum: &mut FreshQuorum, + workload: &ReceiptWorkload, + rooted: bool, + ordinal: u8, + ) -> Result { + let mut transports = Vec::new(); + for (node, voter) in quorum.voters.iter().enumerate() { + transports.push(runtime_fixture::tcp::start(&voter.runtime, node).await?); + } + let route = cellar::application::consensus::verification::GroupRoute { + scope: quorum.route.history, + schema_digest: quorum.route.schema_digest, + revision: 1, + peers: quorum + .voters + .iter() + .zip(&transports) + .enumerate() + .map(|(node, (voter, transport))| -> Result<_> { + Ok(RaftPeer { + replica: voter + .log + .cell_group(&quorum.route.history)? + .identity() + .clone(), + label: format!("node-{}", node + 1), + server_name: "localhost".into(), + control_address: transport.control_address(), + data_address: transport.data_address(), + query_address: transport.query_address(), + }) + }) + .collect::>()?, + }; + for transport in &transports { + transport.install_physical_group(route.clone())?; + } + let mut controllers = Vec::new(); + for (voter, transport) in quorum.voters.iter_mut().zip(&transports) { + controllers.push( + CellGroupController::start( + voter.runtime.clone(), + voter.events.take().context("exclusive recovered events")?, + transport.clone(), + voter.sql.clone(), + vec![voter.applier.clone()], + CellGroupControllerLimits { + max_groups: 1, + max_parallel_apply: 1, + retry_interval_ms: 5, + max_commands: 2, + max_command_bytes: 64 * 1024, + checkpoints: None, + }, + ) + .await?, + ); + } + let io = CellGroupArchivePool::new(Default::default())?; + let birth = s + .cluster + .domain + .run( + s.cluster + .directory + .materialize_selected_group(&quorum.route), + ) + .await??; + if ordinal == 0 { + s.cluster + .directory + .prepare_group_archive_binding(&s.cluster.domain, &io, &birth) + .await? + .commit(&s.cluster.domain) + .await?; + } else { + s.cluster + .directory + .open_group_archive(&s.cluster.domain, &io, &birth) + .await?; + } + let verifier = destination_at( + &s.cluster, + rooted, + Arc::new(DirectFileSystem), + s.cluster + ._root + .path() + .join(format!("fresh-verifier-{ordinal}")), + quorum.route.clone(), + ) + .await?; + controllers[0] + .archive(CellGroupArchiveRegistration { + directory: s.cluster.directory.clone(), + birth, + io, + owners: CellGroupArchiveUploadPool::new(Default::default())?, + tails: CellGroupArchiveTailPool::new(Default::default())?, + checkpoints: CellGroupCheckpointPool::new(checkpoint_fixture::limits())?, + store: verifier.store, + scratch: ScratchPool::new(ScratchLimits { + max_bytes: 32 * 1024 * 1024, + max_files: 1, + })?, + scratch_directory: s.cluster._root.path().into(), + })? + .response() + .await?; + let dispatcher = CellGroupDispatcher::new( + &s.cluster.domain, + s.cluster.directory.clone(), + workload.class.clone(), + keys()?, + CellDispatchLimits { + max_cached_routes: 4, + route_cache_bytes: 256 * 1024, + max_invocations: 4, + retained_bytes: 1024 * 1024, + max_input_bytes: 1024, + }, + )? + .with_s3(&controllers[0])?; + Ok(Self { + controllers, + transports, + dispatcher, + }) + } + + async fn command( + &self, + s: &Scenario, + quorum: &FreshQuorum, + w: &ReceiptWorkload, + name: &str, + sequence: u64, + input: &[u8], + ) -> Result { + let cell = fresh_cell(s, quorum, name)?; + let call = self + .dispatcher + .command( + cell.clone(), + w.request(sequence, input)?, + CellRoutingLimits { + timeout: Duration::from_secs(15), + max_attempts: 4, + }, + None, + ) + .await?; + Ok(self.controllers[0] + .command( + call, + sql_options( + quorum.voters[0].images.clone(), + quorum.voters[0].base.join(format!("sql-{name}")), + quorum.route.schema_digest, + )?, + )? + .response() + .await?) + } + + async fn close(self) -> Result<()> { + for controller in self.controllers { + controller.close().await?; + } + for transport in self.transports { + transport.close().await; + } + Ok(()) + } +} + +fn fresh_cell(_s: &Scenario, quorum: &FreshQuorum, name: &str) -> Result { + Ok(CellHistory { + cell: CellId::new(name)?, + ..quorum.cell.clone() + }) +} + +pub(crate) async fn registered_receipts(rooted: bool, store: Arc) -> Result<()> { + let mut workload = ReceiptWorkload::new()?; + let mut s = + Box::new(Box::pin(Scenario::with_workload(rooted, store, Some(&mut workload))).await?); + let seeds = seed_voters(&mut s, rooted).await?; + let mut quorum = publish_voters(&mut s, seeds).await?; + activate_voters(&s, &mut quorum).await?; + serving_phase(&mut s, &mut quorum, &workload, rooted, 0).await?; + restart_voters(&s, &mut quorum, rooted).await?; + serving_phase(&mut s, &mut quorum, &workload, rooted, 1).await?; + for voter in quorum.voters { + voter.runtime.close().await?; + voter.sql.close().await?; + voter.log.close().await?; + } + s.cluster.close().await +} + +fn serving_phase<'a>( + s: &'a mut Scenario, + quorum: &'a mut FreshQuorum, + w: &'a ReceiptWorkload, + rooted: bool, + ordinal: u8, +) -> futures_util::future::BoxFuture<'a, Result<()>> { + Box::pin(async move { + let serving = Serving::start(s, quorum, w, rooted, ordinal).await?; + let now = s + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms().max(0) as u64 }) + .await?; + for name in ["alpha", "beta"] { + let cell = fresh_cell(s, quorum, name)?; + let old = &w.old[name]; + let old_token = keys()?.issue_cell(old, quorum.route.schema_digest, w.now)?; + // Prove rejection is due to the history, while the old token remains + // valid for its original identity at the same real wall time. + assert_eq!( + keys()?.verify_cell( + &old_token, + &old.version.history, + quorum.route.schema_digest, + now + )?, + *old + ); + assert_eq!( + keys()?.verify_cell(&old_token, &cell, quorum.route.schema_digest, now), + Err(VersionError::WrongHistory) + ); + assert_ne!(old.version.history, cell); + assert_ne!(old.witness.group, quorum.route.history); + let sequence = if name == "alpha" { 2 } else { 1 }; + let response = serving + .command(s, quorum, w, name, sequence, &[sequence as u8]) + .await?; + assert_eq!( + response + .local_receipt() + .expect("local command receipt") + .receipt + .result(), + &w.results[&(name.into(), sequence)] + ); + assert_eq!( + response + .local_receipt() + .expect("local command receipt") + .version + .history, + cell + ); + assert_eq!( + response + .local_receipt() + .expect("local command receipt") + .witness + .group, + quorum.route.history + ); + assert_eq!( + response + .local_receipt() + .expect("local command receipt") + .version + .sequence + .get(), + if ordinal == 1 && name == "alpha" { + 2 + } else { + 1 + } + ); + assert_eq!( + response + .local_receipt() + .expect("local command receipt") + .receipt + .minimum_durability(), + if name == "alpha" { + Durability::S3 + } else { + Durability::Quorum + } + ); + assert_eq!( + response.durability(), + if name == "alpha" { + Durability::S3 + } else { + Durability::Quorum + } + ); + let response_time = s + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms().max(0) as u64 }) + .await?; + assert_eq!( + keys()?.verify_cell( + response.token(), + &cell, + quorum.route.schema_digest, + response_time + )?, + response + .local_receipt() + .expect("local command receipt") + .observation() + ); + } + assert_eq!( + w.calls.load(Ordering::Acquire), + if ordinal == 0 { 4 } else { 5 } + ); + let floor = serving + .command(s, quorum, w, "alpha", 1, &[1]) + .await + .err() + .context("restored expired floor")?; + assert!(matches!( + floor + .downcast_ref::>() + .map(|e| e.cause.as_ref()), + Some(CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::CommandExpired + ))) + )); + let conflict = serving + .command(s, quorum, w, "alpha", 2, b"foreign input") + .await + .err() + .context("restored receipt identity")?; + assert!(matches!( + conflict + .downcast_ref::>() + .map(|e| e.cause.as_ref()), + Some(CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::CommandConflict + ))) + )); + if ordinal == 0 { + let response = serving.command(s, quorum, w, "alpha", 4, &[4]).await?; + assert_eq!( + response + .local_receipt() + .expect("local command receipt") + .version + .sequence + .get(), + 2 + ); + assert_eq!( + response + .local_receipt() + .expect("local command receipt") + .version + .history, + quorum.cell + ); + let index = response + .local_receipt() + .expect("local command receipt") + .position + .index; + drop(response); + controllers_applied(&quorum.voters, quorum.route.history, index).await?; + let root = quorum.voters[0].applier.history(); + // Read from the actual fresh engine-selected pin; retained restart + // below must preserve this new image as well as both old receipts. + let authority = quorum.voters[0] + .runtime + .reconciliation_authority(root, quorum.cell.clone()) + .await?; + s.expected.insert( + "alpha".into(), + pin_bytes(&s.cluster, authority.image().clone()).await?, + ); + } else { + let response = serving.command(s, quorum, w, "alpha", 4, &[4]).await?; + assert_eq!( + response + .local_receipt() + .expect("local command receipt") + .version + .sequence + .get(), + 2 + ); + } + assert_eq!(w.calls.load(Ordering::Acquire), 5); + serving.close().await + }) +} diff --git a/crates/cellar/tests/support/cell_group_disaster_seed.rs b/crates/cellar/tests/support/cell_group_disaster_seed.rs new file mode 100644 index 000000000..4c59284d0 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_disaster_seed.rs @@ -0,0 +1,294 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Actual fresh engine seeding after removing every old voter directory. The +//! authority remains a contract fixture; no Docker launch fence is qualified. +use super::*; + +pub(super) async fn erase_voters(s: &mut Scenario, rooted: bool) -> Result<()> { + for mut voter in std::mem::take(&mut s.cluster.voters) { + drop(voter.owner.take()); + drop(voter.held.take()); + voter.log.close().await?; + } + s.cluster.network.clear(); + let directory = s.cluster._root.path().to_owned(); + s.cluster + .domain + .blocking(move || -> Result<()> { + for n in 1..=3 { + let node = directory.join(format!("node-{n}")); + assert!(node.join("log").exists()); + std::fs::remove_dir_all(&node)?; + assert!(!node.exists()); + if rooted { + let extents = directory.join(format!("extents-{}", n - 1)); + assert!(extents.exists()); + std::fs::remove_dir_all(extents)?; + } + } + Ok(()) + })? + .await??; + // Replace and erase the earlier publication verifier cache as well. The + // fence/recovery now has neither a voter disk nor that warm recovery image. + let cold = destination_at( + &s.cluster, + rooted, + Arc::new(DirectFileSystem), + s.cluster._root.path().join("verify-after-voter-loss"), + s.cluster.birth.route().clone(), + ) + .await?; + drop(std::mem::replace(&mut s.destination, cold)); + let old_cache = s.cluster._root.path().join("archive-data"); + s.cluster + .domain + .blocking(move || std::fs::remove_dir_all(old_cache))? + .await??; + Ok(()) +} + +pub(crate) async fn seed_engine( + rooted: bool, + store: Arc, +) -> Result<()> { + let mut s = Box::new(Scenario::new(rooted, store).await?); + let old_disk = s.cluster.voters[0] + .log + .cell_group(&s.cluster.birth.route().history)? + .identity() + .disk; + erase_voters(&mut s, rooted).await?; + let proof = s.fence(&s.cluster.directory, &s.request).await?; + let route = s.request.replacement().route().clone(); + let node = s.request.replacement().voters()[0].node; + let member = s.request.replacement().voters()[0].member; + let physical = s.cluster._root.path().join("recovered-images"); + let destination = destination_at( + &s.cluster, + rooted, + Arc::new(DirectFileSystem), + physical.clone(), + route.clone(), + ) + .await?; + let pool = CellGroupRecoveryPool::new(Default::default())?; + let scratch = ScratchPool::new(ScratchLimits { + max_bytes: 32 * 1024 * 1024, + max_files: 1, + })?; + let prepared = pool + .prepare( + &s.cluster.domain, + proof, + destination.store.clone(), + scratch.clone(), + s.cluster._root.path().into(), + ) + .await?; + let marker = prepared.manifest().marker().clone(); + let wrong = s + .cluster + .domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + s.cluster._root.path().join("wrong-recovery-disk/log"), + node, + old_disk, + ProcessGeneration::new([199; 16])?, + LogOpenMode::Create, + ), + Arc::new(DirectFileSystem), + )) + .await??; + assert!(matches!( + s.cluster + .directory + .seed_group_recovery(&s.cluster.domain, prepared.clone(), &wrong, &s.authority) + .await, + Err(CellGroupRecoveryError::Catalog(CatalogError::Conflict( + "reused recovery storage" + ))) + )); + wrong.close().await?; + drop(wrong); + let gate = fault::Gate::new(None); + let mut options = ReplicaLogOptions::new( + s.cluster._root.path().join("recovered-node/log"), + node, + DiskLineage::new([200; 16])?, + ProcessGeneration::new([201; 16])?, + LogOpenMode::Create, + ); + options.purge_interval_ms = None; + let log = s + .cluster + .domain + .run(ReplicaLog::open( + options.clone(), + Arc::new(fault::FaultFileSystem { gate: gate.clone() }), + )) + .await??; + let release = gate.arm(fault::HOLD_SYNC); + let mut work = Box::pin(s.cluster.directory.seed_group_recovery( + &s.cluster.domain, + prepared.clone(), + &log, + &s.authority, + )); + let entered = Box::pin(tokio::time::timeout( + Duration::from_secs(15), + gate.entered.notified(), + )); + match futures_util::future::select(entered, work.as_mut()).await { + futures_util::future::Either::Left((result, _)) => result?, + futures_util::future::Either::Right(_) => anyhow::bail!("genesis skipped engine sync"), + } + assert!(matches!( + log.cell_group(&route.history), + Err(LogError::NotFound) + )); + drop(work); + drop(release); + let group = tokio::time::timeout(Duration::from_secs(15), async { + loop { + match log.cell_group(&route.history) { + Ok(group) => break Ok::<_, LogError>(group), + Err(LogError::NotFound) => tokio::task::yield_now().await, + Err(error) => break Err(error), + } + } + }) + .await??; + assert_eq!(group.group_manifest()?.as_ref(), Some(&marker)); + assert!(!group.group_manifest()?.unwrap().is_active()); + assert_eq!(group.identity().member, member); + let key = group.numeric_key(); + assert!(!s + .remote + .list(None) + .try_collect::>() + .await? + .iter() + .any(|meta| meta.location.as_ref().contains(&format!( + "recovery/{}/seeds/", + hex::encode(s.request.digest()) + )) && meta.location.as_ref().ends_with("/complete"))); + let seeded = s + .cluster + .directory + .seed_group_recovery(&s.cluster.domain, prepared.clone(), &log, &s.authority) + .await?; + assert_eq!(seeded.group().numeric_key(), key); + assert_eq!(seeded.manifest().marker(), &marker); + assert_ne!(seeded.binding(), [0; 32]); + tail_fixture::check_images(&s.cluster, seeded.manifest(), &s.expected).await?; + let ordinary = CellGroupBootstrap { + route: route.clone(), + member, + voters: s + .request + .replacement() + .voters() + .iter() + .map(|m| m.member) + .collect(), + learners: vec![], + }; + assert!(matches!( + log.create_cell_group(ordinary.clone()).await, + Err(LogError::WrongIdentity) + )); + drop(prepared); + assert_eq!(pool.usage().operations, 0); + // The engine owns the complete physical closure after import admission ends. + let applier = seeded.applier().clone(); + let selected = s + .cluster + .domain + .blocking(move || applier.recover_selected())? + .await??; + tail_fixture::check_images(&s.cluster, &selected, &s.expected).await?; + drop(selected); + drop(group); + drop(seeded); + drop(destination); + log.close().await?; + drop(log); + options.mode = LogOpenMode::Resume; + options.process = ProcessGeneration::new([202; 16])?; + let log = s + .cluster + .domain + .run(ReplicaLog::open( + options.clone(), + Arc::new(DirectFileSystem), + )) + .await??; + let target = destination_at( + &s.cluster, + rooted, + Arc::new(DirectFileSystem), + physical, + route.clone(), + ) + .await?; + let proof = s.fence(&s.cluster.directory, &s.request).await?; + let prepared = pool + .prepare( + &s.cluster.domain, + proof, + target.store.clone(), + scratch, + s.cluster._root.path().into(), + ) + .await?; + let seeded = s + .cluster + .directory + .seed_group_recovery(&s.cluster.domain, prepared.clone(), &log, &s.authority) + .await?; + assert_eq!(seeded.group().numeric_key(), key); + assert_eq!(seeded.group().identity().process, options.process); + assert_eq!(seeded.manifest().marker(), &marker); + tail_fixture::check_images(&s.cluster, seeded.manifest(), &s.expected).await?; + assert!(matches!( + log.create_cell_group(ordinary).await, + Err(LogError::WrongIdentity) + )); + drop(seeded); + log.close().await?; + drop(log); + let path = options.directory.clone(); + s.cluster + .domain + .blocking(move || std::fs::remove_dir_all(path))? + .await??; + options.mode = LogOpenMode::Create; + options.process = ProcessGeneration::new([203; 16])?; + let blank = s + .cluster + .domain + .run(ReplicaLog::open(options, Arc::new(DirectFileSystem))) + .await??; + assert!(matches!( + s.cluster + .directory + .seed_group_recovery(&s.cluster.domain, prepared.clone(), &blank, &s.authority) + .await, + Err(CellGroupRecoveryError::Catalog( + CatalogError::LostStableState + )) + )); + assert!(matches!( + blank.cell_group(&route.history), + Err(LogError::NotFound) + )); + blank.close().await?; + drop(prepared); + drop(target); + assert_eq!(pool.usage().operations, 0); + assert_eq!(s.receivers.usage().sources, 0); + let s = *s; + s.cluster.close().await +} diff --git a/crates/cellar/tests/support/cell_group_dispatch_cache_execution.rs b/crates/cellar/tests/support/cell_group_dispatch_cache_execution.rs new file mode 100644 index 000000000..8fc0e1676 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_dispatch_cache_execution.rs @@ -0,0 +1,181 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Known local execution still needs real quorum; hints cannot authorize RPC. +use super::remote_queries::{dispatcher, gateways, invocation, keys}; +use super::*; +use object_store::{path::Path, ObjectStore}; + +pub(crate) async fn cached_dispatch_execution() -> Result<()> { + for rooted in [false, true] { + let class = Arc::new( + RaftClass::new("Counter", 1, [81; 32])? + .initializer([90; 32], |_, tx| { + tx.execute_batch("CREATE TABLE value(n); INSERT INTO value VALUES(7);")?; + Ok(()) + })? + .command("Add", [82; 32], |context, _| { + context.sql().execute("UPDATE value SET n=n+1", [])?; + Ok(b"added".to_vec()) + })? + .query("Get", [83; 32], |context, _| { + Ok(context + .query_row("SELECT n FROM value", [], |r| r.get::<_, i64>(0))? + .to_le_bytes() + .to_vec()) + })?, + ); + let (mut fixture, route, a, b) = Box::pin(registered::setup(rooted, &class)).await?; + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let gateways = gateways(&fixture, &controllers, class.clone()).await?; + let dispatch = dispatcher(&fixture, class.clone(), keys(190)?)?; + let mut warm = invocation( + &dispatch, + &class, + &a, + CellQueryConsistency::Linearizable, + Duration::from_secs(5), + b"", + ) + .await?; + let pool = &fixture.appliers[&(0, route.history)].0; + assert_eq!( + warm.execute(&fixture.runtimes[0], pool, fixture.cell_options(&a).await?) + .await? + .result(), + &0_i64.to_le_bytes() + ); + drop(warm); + assert_eq!(dispatch.route_cache_usage()?.entries, 1); + let head = Path::from(format!( + "namespaces/{}/cell-groups/head", + hex::encode(a.namespace.bytes()) + )); + let saved = fixture.cluster.store.get(&head).await?.bytes().await?; + fixture.cluster.store.delete(&head).await?; + assert!(dispatch.refresh_routes().await.is_err()); + // The range covers B and an as-yet unenrolled C; no per-cell cache entry + // or fresh metadata read is needed for these captured invocations. + let mut other = invocation( + &dispatch, + &class, + &b, + CellQueryConsistency::Stale, + Duration::from_secs(5), + b"", + ) + .await?; + assert_eq!( + other + .execute(&fixture.runtimes[0], pool, fixture.cell_options(&b).await?) + .await? + .result(), + &0_i64.to_le_bytes() + ); + drop(other); + let retry_until_ms = fixture + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms() as u64 + 60_000 }) + .await?; + let command = dispatch + .command( + a.clone(), + CellCommandRequest { + id: CommandId::new(ClientId::new([191; 16])?, 1)?, + contract: class.contract("Add").context("Add")?, + input: vec![], + durability: Durability::Quorum, + retry_until_ms, + }, + CellRoutingLimits { + timeout: Duration::from_secs(5), + max_attempts: 2, + }, + None, + ) + .await?; + let result = controllers[0] + .command(command, fixture.cell_options(&a).await?)? + .response() + .await?; + assert_eq!( + result + .local_receipt() + .expect("local command receipt") + .receipt + .result(), + b"added" + ); + drop(result); + let c = history(&route, "cache-enrolled")?; + let enrollment = dispatch + .enroll( + c.clone(), + CellEnrollmentRequest { + creation: ProposalId::new([192; 16])?, + durability: Durability::Quorum, + }, + CellRoutingLimits { + timeout: Duration::from_secs(5), + max_attempts: 2, + }, + None, + ) + .await?; + let enrolled = controllers[0] + .enroll(enrollment, fixture.cell_options(&c).await?)? + .response() + .await?; + drop(enrolled); + for (cell, expected) in [(&a, 1_i64), (&c, 7_i64)] { + let mut read = invocation( + &dispatch, + &class, + cell, + CellQueryConsistency::Linearizable, + Duration::from_secs(5), + b"", + ) + .await?; + assert_eq!( + read.execute( + &fixture.runtimes[0], + pool, + fixture.cell_options(cell).await? + ) + .await? + .result(), + &expected.to_le_bytes() + ); + } + // The RPC destination deliberately keeps fresh catalog admission until + // transfer-seal authority is integrated. Sender hints cannot bypass it. + let mut remote = invocation( + &dispatch, + &class, + &a, + CellQueryConsistency::Stale, + Duration::from_secs(5), + b"", + ) + .await?; + assert!(matches!( + remote + .execute_remote(&fixture.transports[1], MemberId::new(1)?) + .await, + Err(CellDispatchError::Transport(TransportError::Query( + CommandFailure::Unavailable + ))) + )); + drop(remote); + assert_eq!(dispatch.route_cache_usage()?.entries, 1); + assert_eq!(dispatch.usage(), (0, 0)); + fixture.cluster.store.put(&head, saved.into()).await?; + for controller in controllers { + controller.close().await?; + } + drop(gateways); + fixture.close().await?; + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_effects.rs b/crates/cellar/tests/support/cell_group_effects.rs new file mode 100644 index 000000000..a65f02a2d --- /dev/null +++ b/crates/cellar/tests/support/cell_group_effects.rs @@ -0,0 +1,335 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! The production node controller owns all timers, transport and apply work. +//! No fixture event pumping or client traffic drives an armed alarm to commit. + +use super::*; +use cellar::application::consensus::{RaftAlarmLimits, RaftEffectOptions}; +use std::sync::atomic::{AtomicUsize, Ordering}; + +pub(crate) async fn alarms_without_clients() -> Result<()> { + alarms(false, false).await +} + +pub(crate) async fn shutdown_joins_alarm() -> Result<()> { + alarms(true, false).await +} + +pub(crate) async fn hot_and_cold_alarms() -> Result<()> { + alarms(false, true).await +} + +async fn alarms(hold_callback: bool, rearm_hot: bool) -> Result<()> { + let _ = tracing_subscriber::fmt() + .with_env_filter("cellar::raft::cell_groups::controller::effects=debug") + .with_test_writer() + .try_init(); + for rooted in [false, true] { + let calls = Arc::new(AtomicUsize::new(0)); + let observed = calls.clone(); + let hot_calls = Arc::new(AtomicUsize::new(0)); + let hot = hot_calls.clone(); + let hot_at_other = Arc::new(AtomicUsize::new(usize::MAX)); + let fairness = hot_at_other.clone(); + let (entered, started) = tokio::sync::oneshot::channel(); + let entered = Arc::new(std::sync::Mutex::new(Some(entered))); + let (release, held) = std::sync::mpsc::sync_channel(1); + let held = Arc::new(std::sync::Mutex::new(held)); + let class = Arc::new( + RaftClass::new("Counter", 1, [if rearm_hot { 225 } else { 201 }; 32])? + .alarms(RaftAlarmLimits { + max_alarms: 2, + max_payload_bytes: 128, + })? + .alarm("Wake", [202; 32], move |context, input| { + assert_eq!(input, b"wake"); + observed.fetch_add(1, Ordering::SeqCst); + if hold_callback { + let _ = entered + .lock() + .unwrap() + .take() + .context("one callback")? + .send(()); + held.lock().unwrap().recv_timeout(Duration::from_secs(15))?; + } + if rearm_hot { + if context.identity().cell.as_str() == "registered-a" { + let count = hot.fetch_add(1, Ordering::SeqCst) + 1; + if count < 4 { + context.set_alarm( + "Wake", + context.inputs().now_unix_millis, + b"wake", + )?; + } + } else { + fairness.store(hot.load(Ordering::SeqCst), Ordering::SeqCst); + } + } + context.sql().execute("UPDATE value SET n=n+1", [])?; + Ok(()) + })? + .command("Arm", [203; 32], |context, _| { + context.set_alarm("Wake", context.inputs().now_unix_millis + 50, b"wake")?; + Ok(b"armed".to_vec()) + })?, + ); + let (mut fixture, route, a, b) = Box::pin(registered::setup(rooted, &class)).await?; + let mut cold = Vec::new(); + let empty = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(0);")?; + for ordinal in 0..32 { + let cell = history(&route, &format!("cold-{ordinal}"))?; + let accepted = fixture.runtimes[0] + .propose( + route.history, + apply::enrollment_bytes(&route, &cell, &empty, rooted)?, + ) + .await?; + fixture.drive(Some(accepted.position.index)).await?; + cold.push(cell); + } + let keys = VersionKeys::new( + VersionSigningKey::new([204; 32], u64::MAX)?, + vec![], + Duration::from_secs(30), + )?; + let dispatcher = CellGroupDispatcher::new( + &fixture.cluster.domain, + fixture.cluster.directory.clone(), + class.clone(), + keys, + CellDispatchLimits { + max_cached_routes: 32, + route_cache_bytes: 256 * 1024, + max_invocations: 8, + retained_bytes: 1024 * 1024, + max_input_bytes: 32 * 1024, + }, + )?; + let retry_until_ms = fixture + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms().max(0) as u64 + 60_000 }) + .await?; + for cell in [&a, &b].into_iter().take(if hold_callback { 1 } else { 2 }) { + let mut invocation = dispatcher + .command( + cell.clone(), + CellCommandRequest { + id: CommandId::new(ClientId::new([205; 16])?, 1)?, + contract: class.contract("Arm").context("Arm")?, + input: Vec::new(), + durability: Durability::Quorum, + retry_until_ms, + }, + CellRoutingLimits { + timeout: Duration::from_secs(5), + max_attempts: 4, + }, + None, + ) + .await?; + invocation.begin().await?; + assert!(matches!( + invocation + .prepare( + &fixture.runtimes[0], + &fixture.appliers[&(0, route.history)].0, + fixture.cell_options(cell).await? + ) + .await?, + CellCommandPreparation::Prepared + )); + let accepted = command_dispatch::accepted(&mut invocation).await?; + fixture.drive(Some(accepted.position.index)).await?; + let receipt = command_dispatch::committed(&mut invocation).await?; + assert_eq!( + receipt + .local_receipt() + .expect("local command receipt") + .receipt + .result(), + b"armed" + ); + } + // Seed already committed due work before starting the node controller. + // Its initial marker hints, not another client request or fixture pump, + // must discover both cells; one-slot admission stays unchanged. + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + assert!(fixture.events.is_empty()); + let cold_opens = Arc::new(AtomicUsize::new(0)); + for (node, controller) in controllers.iter().enumerate() { + let mut resources = BTreeMap::new(); + for cell in [&a, &b] { + resources.insert(cell.clone(), fixture.cell_options_at(cell, node).await?); + } + let unexpected = cold_opens.clone(); + controller.configure_effects(vec![CellGroupEffectRegistration::new( + dispatcher.clone(), + RaftEffectOptions { + metadata_bytes: 64 * 1024, + ..Default::default() + }, + move |cell| { + resources.get(cell).cloned().ok_or_else(|| { + unexpected.fetch_add(1, Ordering::SeqCst); + ExecutionError::Invalid("cold cell must not open for discovery") + }) + }, + )?])?; + } + if hold_callback { + tokio::time::timeout(Duration::from_secs(5), started) + .await + .with_context(|| { + format!( + "alarm callback never entered; usage={:?}; calls={}", + controllers + .iter() + .map(|controller| controller.usage()) + .collect::>(), + calls.load(Ordering::SeqCst) + ) + })??; + assert_eq!(calls.load(Ordering::SeqCst), 1); + let pool = &fixture.appliers[&(0, route.history)].0; + assert_eq!(pool.usage()?.active_cells, 1); + assert_eq!(controllers[0].usage().retained_commands, 1); + assert!( + tokio::time::timeout(Duration::from_millis(30), controllers[0].close()) + .await + .is_err() + ); + assert_eq!(pool.usage()?.active_cells, 1); + release.send(())?; + controllers[0].close().await?; + // Closing waited for the actual SQL callback. It did not publish + // the unsubmitted private write or erase the durable alarm debt. + for runtime in fixture.runtimes.iter().skip(1) { + let selected = runtime + .reconciliation_authority(route.history, a.clone()) + .await?; + assert_eq!(selected.cell().version.sequence.get(), 2); + assert!(selected.cell().effects.next_alarm_unix_millis.is_some()); + } + } else { + // Only observations follow. The controller's durable index and shared + // SQL/consensus workers must discover and execute both cells unaided. + let completed = controller::wait_for(async || { + for runtime in &fixture.runtimes { + for cell in [&a, &b] { + let selected = runtime + .reconciliation_authority(route.history, cell.clone()) + .await?; + let sequence = if rearm_hot && cell == &a { 6 } else { 3 }; + if selected.cell().version.sequence.get() != sequence + || selected.cell().effects.next_due_unix_millis().is_some() + { + return Ok(false); + } + } + } + // Root publication precedes physical producer/response disposal. + // Inspect through the same one-slot pool only after ownership + // has drained; keep the original overall progress deadline. + for controller in &controllers { + let usage = controller.usage(); + if usage.active_effects != 0 || usage.retained_commands != 0 { + return Ok(false); + } + } + for (pool, _) in fixture.appliers.values() { + let usage = pool.usage()?; + if usage.active_cells != 0 || usage.retained_operations != 0 { + return Ok(false); + } + } + Ok(true) + }) + .await; + if let Err(error) = completed { + let status = fixture.runtimes[0].status(route.history).await; + let alpha = fixture.runtimes[0] + .reconciliation_authority(route.history, a.clone()) + .await + .map(|authority| (authority.cell().version.clone(), authority.cell().effects)); + let beta = fixture.runtimes[0] + .reconciliation_authority(route.history, b.clone()) + .await + .map(|authority| (authority.cell().version.clone(), authority.cell().effects)); + return Err(error.context(format!("alarm progress: status={status:?}; alpha={alpha:?}; beta={beta:?}; usage={:?}; callbacks={}", controllers.iter().map(|controller| controller.usage()).collect::>(), calls.load(Ordering::SeqCst)))); + } + assert_eq!(calls.load(Ordering::SeqCst), if rearm_hot { 5 } else { 2 }); + if rearm_hot { + assert_eq!(hot_calls.load(Ordering::SeqCst), 4); + assert!( + hot_at_other.load(Ordering::SeqCst) <= 1, + "a re-armed cell must rotate behind the other due cell" + ); + } + for (node, runtime) in fixture.runtimes.iter().enumerate() { + let pool = &fixture.appliers[&(node, route.history)].0; + for cell in [&a, &b] { + let deadline = fixture + .cluster + .domain + .run(async { cellar::asyncrt::mono_ms() + 5000 }) + .await?; + let authority = runtime + .read_authority( + route.history, + cell.clone(), + CellReadConsistency::Stale, + deadline, + ) + .await?; + let result = pool + .query( + fixture.cell_options_at(cell, node).await?, + authority, + |database, _| { + let n: i64 = database.connection().query_row( + "SELECT n FROM value", + [], + |row| row.get(0), + )?; + Ok(n.to_be_bytes().to_vec()) + }, + ) + .await?; + let expected = if rearm_hot && cell == &a { + 4_i64 + } else { + 1_i64 + }; + assert_eq!(result.result(), expected.to_be_bytes()); + } + } + } + assert!(controllers[0].usage().effect_turns > 0); + assert_eq!(cold_opens.load(Ordering::SeqCst), 0); + for runtime in fixture.runtimes.iter().skip(usize::from(hold_callback)) { + for cell in &cold { + let selected = runtime + .reconciliation_authority(route.history, cell.clone()) + .await?; + assert_eq!(selected.cell().version.sequence.get(), 1); + assert_eq!(selected.cell().effects.next_due_unix_millis(), None); + } + } + for controller in &controllers { + controller.close().await?; + assert_eq!(controller.usage().active_effects, 0); + assert_eq!(controller.usage().retained_commands, 0); + } + for (pool, _) in fixture.appliers.values() { + assert_eq!(pool.usage()?.active_cells, 0); + assert_eq!(pool.usage()?.retained_bytes, 0); + assert_eq!(pool.usage()?.queued_or_running, 0); + } + drop(controllers); + fixture.close().await?; + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_engine_fixture.rs b/crates/cellar/tests/support/cell_group_engine_fixture.rs new file mode 100644 index 000000000..8b20b34bf --- /dev/null +++ b/crates/cellar/tests/support/cell_group_engine_fixture.rs @@ -0,0 +1,730 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +// Shared at each integration target root to preserve the support modules' paths. +use anyhow::{Context, Result}; +use cellar::application::consensus::verification::cell_groups::GroupRoute; +use cellar::application::consensus::{ + verification::{cell_groups::*, *}, + *, +}; +use cellar::sdk::ExecutionDomain; +use cellar_ltx::{ + apply::{ImageState, ImageUpdate, LtxSegment}, + generation::{GenerationPin, GenerationStore, ImageId}, + DirectFileSystem, TXID, +}; +use raft::prelude::{Message, MessageType}; +use sha2::{Digest, Sha256}; +use std::{collections::VecDeque, sync::Arc, time::Duration}; + +struct Voter { + log: ReplicaLog, + owner: Option, + pages: Option, + images: GenerationStore, + extents: Option, + tree: ManifestTree, + gate: Arc, + paused: bool, + held: Option, +} + +struct Cluster { + _root: tempfile::TempDir, + domain: ExecutionDomain, + directory: RangeDirectory, + store: Arc, + birth: CellGroupBirth, + voters: Vec, + network: VecDeque, + reads: Vec<(usize, Term, LogIndex, Vec)>, + options: RaftRuntimeOptions, +} + +#[derive(Clone, Copy)] +struct ClusterLogLimits { + retention: CellGroupLogRetentionLimits, + target_file_bytes: u64, + purge_threshold_bytes: u64, + purge_interval_ms: Option, +} + +struct ClusterBirthOptions { + group: GroupId, + image_storage: RaftImageStorage, + minimum: Durability, + log: ClusterLogLimits, +} + +impl Default for ClusterLogLimits { + fn default() -> Self { + Self { + retention: Default::default(), + target_file_bytes: 16 * 1024 * 1024, + purge_threshold_bytes: 64 * 1024 * 1024, + purge_interval_ms: Some(1000), + } + } +} + +impl Cluster { + async fn new() -> Result { + Self::with_options(RaftRuntimeOptions::default()).await + } + + async fn with_options(options: RaftRuntimeOptions) -> Result { + Box::pin(Self::with_storage( + options, + tempfile::tempdir()?, + Arc::new(DirectFileSystem), + )) + .await + } + + async fn with_storage( + options: RaftRuntimeOptions, + root: tempfile::TempDir, + filesystem: Arc, + ) -> Result { + Box::pin(Self::with_storage_schema( + options, root, filesystem, [6; 32], + )) + .await + } + + async fn with_storage_schema( + options: RaftRuntimeOptions, + root: tempfile::TempDir, + filesystem: Arc, + schema: [u8; 32], + ) -> Result { + Self::with_store( + options, + root, + filesystem, + schema, + Arc::new(object_store::memory::InMemory::new()), + ) + .await + } + + async fn with_store( + options: RaftRuntimeOptions, + root: tempfile::TempDir, + filesystem: Arc, + schema: [u8; 32], + store: Arc, + ) -> Result { + Self::with_store_minimum(options, root, filesystem, schema, store, Durability::Quorum).await + } + + async fn with_store_minimum( + options: RaftRuntimeOptions, + root: tempfile::TempDir, + filesystem: Arc, + schema: [u8; 32], + store: Arc, + minimum: Durability, + ) -> Result { + Self::with_store_retention( + options, + root, + filesystem, + schema, + store, + minimum, + Default::default(), + ) + .await + } + + async fn with_store_retention( + options: RaftRuntimeOptions, + root: tempfile::TempDir, + filesystem: Arc, + schema: [u8; 32], + store: Arc, + minimum: Durability, + retention: CellGroupLogRetentionLimits, + ) -> Result { + Box::pin(Self::with_store_log_limits( + options, + root, + filesystem, + schema, + store, + minimum, + ClusterLogLimits { + retention, + ..Default::default() + }, + )) + .await + } + + async fn with_store_log_limits( + options: RaftRuntimeOptions, + root: tempfile::TempDir, + filesystem: Arc, + schema: [u8; 32], + store: Arc, + minimum: Durability, + log_limits: ClusterLogLimits, + ) -> Result { + Self::with_store_birth( + options, + root, + filesystem, + schema, + store, + ClusterBirthOptions { + group: GroupId::new([3; 16])?, + image_storage: RaftImageStorage::Materialized, + minimum, + log: log_limits, + }, + ) + .await + } + + async fn with_store_birth( + options: RaftRuntimeOptions, + root: tempfile::TempDir, + filesystem: Arc, + schema: [u8; 32], + store: Arc, + birth_options: ClusterBirthOptions, + ) -> Result { + let minimum = birth_options.minimum; + let log_limits = birth_options.log; + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let history = GroupHistory { + cluster: ClusterId::new([1; 16])?, + namespace: NamespaceId::new([2; 16])?, + group: birth_options.group, + incarnation: IncarnationId::new([4; 16])?, + }; + let catalog = Catalog::new(store.clone(), CatalogOptions::default())?; + let mut config = NamespaceConfig::new( + history.cluster, + history.namespace, + cellar_logic::NamespaceMode::Raft, + schema, + ); + config.minimum_durability = minimum; + config.image_storage = birth_options.image_storage; + let namespace = domain.run(catalog.open_namespace(config)).await??; + let directory = RangeDirectory::new(namespace, DirectoryLimits::default())?; + let request = directory.prepare_group_birth(CellGroupBirthSpec { + history, + range: CellRange::all(), + epoch: RouteEpoch::new(1)?, + creation: ProposalId::new([5; 16])?, + voters: (1..=3) + .map(|n| InitialMember { + member: MemberId::new(n).unwrap(), + node: NodeId::new([n as u8; 16]).unwrap(), + }) + .collect(), + learners: vec![], + })?; + domain + .run(directory.reserve_group_birth(&request)) + .await??; + let init = directory + .prepare_initialization(ProposalId::new([7; 16])?, &[request.route().clone()])?; + domain.run(directory.publish(&init)).await??; + let birth = domain + .run(directory.materialize_group_birth(&request)) + .await??; + let mut voters = Vec::new(); + for n in 1..=3 { + let base = root.path().join(format!("node-{n}")); + let gate = fault::Gate::new(None); + let log = domain + .run(ReplicaLog::open( + { + let mut log = ReplicaLogOptions::new( + base.join("log"), + NodeId::new([n; 16])?, + DiskLineage::new([n + 10; 16])?, + ProcessGeneration::new([20; 16])?, + LogOpenMode::Create, + ); + log.cell_group_retention = log_limits.retention; + log.target_file_bytes = log_limits.target_file_bytes; + log.purge_threshold_bytes = log_limits.purge_threshold_bytes; + log.purge_interval_ms = log_limits.purge_interval_ms; + log + }, + Arc::new(fault::FaultFileSystem { gate: gate.clone() }), + )) + .await??; + let permit = domain + .run(directory.bootstrap_group_member(&birth, &log)) + .await??; + let tree = ManifestTree::new(birth.route().clone(), ManifestLimits::default())?; + let mut pages = ManifestPageStore::open( + &base.join("pages"), + filesystem.clone(), + 1024, + 16 * 1024 * 1024, + 16, + )?; + let marker = permit.group().group_manifest()?.context("genesis root")?; + let prepared = tree.prepare_complete( + permit.group(), + &mut pages, + marker.root(), + marker.position(), + marker.membership_revision(), + |_| panic!("empty genesis"), + )?; + let owner = domain + .run(CellGroupOwner::start( + permit, + prepared, + options.clone(), + [n; 32], + )) + .await??; + let images = + GenerationStore::open(base.join("images"), filesystem.clone(), Default::default())?; + voters.push(Voter { + log, + owner: Some(owner), + pages: Some(pages), + images, + extents: None, + tree, + gate, + paused: false, + held: None, + }); + } + Ok(Self { + _root: root, + domain, + directory, + store, + birth, + voters, + network: VecDeque::new(), + reads: vec![], + options, + }) + } + + async fn permit(&self, voter: usize) -> Result { + Ok(self + .domain + .run( + self.directory + .bootstrap_group_member(&self.birth, &self.voters[voter].log), + ) + .await??) + } + + async fn drive(&mut self, isolated: bool) -> Result<()> { + self.drive_partition(isolated, None).await + } + + async fn drive_partition(&mut self, isolated: bool, excluded: Option) -> Result<()> { + for _ in 0..2000 { + let mut progress = false; + for (n, voter) in self.voters.iter_mut().enumerate() { + if excluded == Some(n) { + continue; + } + let event = voter.owner.as_mut().unwrap().poll()?; + let Some(event) = event else { continue }; + progress = true; + match event { + CellGroupEffect::Persist(work) => { + let done = work.run().await?; + voter.owner.as_mut().unwrap().complete_persist(done)?; + } + CellGroupEffect::Message(message) => { + if !isolated { + self.network.push_back(message); + } + } + CellGroupEffect::Apply(apply) => { + if voter.paused { + assert!(voter.held.replace(*apply).is_none()); + } else { + let prepared = prepare(voter, &apply)?; + voter + .owner + .as_mut() + .unwrap() + .complete_apply(apply.ticket, prepared)?; + } + } + CellGroupEffect::Applied { ticket, manifest } => { + assert_eq!(ticket.position(), manifest.position()); + assert_eq!( + voter.log.cell_groups()?[0].group_manifest()?.as_ref(), + Some(manifest.as_ref()) + ); + } + CellGroupEffect::CheckpointInstalled { manifest } + | CellGroupEffect::CheckpointPublished { manifest } => { + assert_eq!( + voter.log.cell_groups()?[0].group_manifest()?.as_ref(), + Some(manifest.as_ref()) + ); + } + CellGroupEffect::ReadIndex { + term, + index, + context, + } => self.reads.push((n, term, index, context)), + } + } + for _ in 0..self.network.len() { + let message = self.network.pop_front().unwrap(); + let target = message.to as usize - 1; + if excluded == Some(target) || excluded == Some(message.from as usize - 1) { + continue; + } + match self.voters[target] + .owner + .as_mut() + .unwrap() + .step(message.clone()) + { + Ok(()) => progress = true, + Err(RuntimeError::NotReady) => self.network.push_back(message), + Err(error) => return Err(error.into()), + } + } + // A bounded receiver may refuse messages until a held apply settles. + // Keep those messages for the next pump instead of busy-spinning. + if !progress { + return Ok(()); + } + } + anyhow::bail!("group did not quiesce") + } + + async fn elect(&mut self) -> Result<()> { + self.voters[0].owner.as_mut().unwrap().campaign()?; + self.drive(false).await?; + assert!(self.voters[0].owner.as_ref().unwrap().status()?.ready); + Ok(()) + } + + async fn close(self) -> Result<()> { + for mut voter in self.voters { + drop(voter.owner.take()); + voter.log.close().await?; + } + self.domain.shutdown().await; + Ok(()) + } +} + +fn image_id(history: &CellHistory, sequence: u64) -> Result { + let mut hash = Sha256::new(); + hash.update(history.cell.as_str()); + hash.update(sequence.to_be_bytes()); + Ok(ImageId::new(hash.finalize()[..16].try_into()?)?) +} + +fn apply_image( + store: &GenerationStore, + history: &CellHistory, + sequence: u64, + base: Option<&GenerationPin>, + image: LtxImage, + files: &[GroupLtx<'_>], +) -> Result { + let header = cellar_ltx::ltx::Header::parse(files.last().context("LTX")?.bytes)?; + let ImageCommitment::Sha256(digest) = image.commitment else { + anyhow::bail!("materialized commitment required") + }; + let result = ImageState { + epoch: image.epoch, + txid: TXID(image.txid), + checksum: image.checksum, + digest, + page_size: header.page_size, + length: u64::from(header.page_size) * u64::from(header.commit), + }; + let update = ImageUpdate { + base: match base { + Some(pin) => pin.state(), + None => ImageState::empty(image.epoch)?, + }, + result, + segments: files + .iter() + .map(|f| LtxSegment { + bytes: f.bytes, + min_txid: TXID(f.min_txid), + max_txid: TXID(f.max_txid), + page_count: f.page_count as usize, + decoded_bytes: f.decoded_bytes, + }) + .collect(), + }; + Ok(store.prepare(image_id(history, sequence)?, base, &update)?) +} + +fn prepare_cell_image( + stores: ( + &GenerationStore, + Option<&cellar_ltx::internal::NodeExtentStore>, + ), + schema: [u8; 32], + cell: &CellHistory, + sequence: u64, + base: Option<&CellImagePin>, + image: LtxImage, + files: &[GroupLtx<'_>], +) -> Result { + use cellar_ltx::internal::{ + ExtentManifest, ExtentRootReference, Geometry, ImageBinding, ImageRootUpdate, EXTENT_BYTES, + }; + let (images, extents) = stores; + if image.txid == 0 { + anyhow::ensure!(base.is_none() && files.is_empty()); + return Ok(EnrollmentImage { + pin: CellImagePin::Empty, + geometry: CellImageGeometry { + length: 0, + page_size: 512, + }, + effects: EffectSummary::default(), + }); + } + let (pin, geometry) = match image.commitment { + ImageCommitment::Sha256(_) => { + let base = match base { + Some(CellImagePin::Materialized(pin)) => Some(pin), + None => None, + _ => anyhow::bail!("materialized base"), + }; + let pin = apply_image(images, cell, sequence, base, image, files)?; + let geometry = CellImageGeometry { + length: pin.state().length, + page_size: pin.state().page_size, + }; + (CellImagePin::Materialized(pin), geometry) + } + ImageCommitment::Extents(result) => { + let store = extents.context("extent store")?; + let base = match base { + Some(CellImagePin::Extents(pin)) => Some(pin.as_ref()), + None => None, + _ => anyhow::bail!("extent base"), + }; + let binding = ImageBinding { + history: cell_image_history_digest(cell), + schema, + epoch: image.epoch, + txid: image.txid, + checksum: image.checksum, + }; + let empty = ExtentManifest::new( + ImageBinding { + txid: 0, + checksum: 0, + ..binding + }, + Geometry::new(0, 0)?, + &[], + )?; + let before = base + .map(|pin| pin.reference()) + .unwrap_or(ExtentRootReference::new( + empty.root(), + empty.encode()?.len().try_into()?, + )?); + let header = cellar_ltx::ltx::Header::parse(files.last().context("extent LTX")?.bytes)?; + let geometry = Geometry::new( + header.page_size, + u64::from(header.page_size) * u64::from(header.commit), + )?; + let update = ImageRootUpdate { + base: before, + result, + binding, + geometry, + segments: files + .iter() + .map(|file| LtxSegment { + bytes: file.bytes, + min_txid: TXID(file.min_txid), + max_txid: TXID(file.max_txid), + page_count: file.page_count as usize, + decoded_bytes: file.decoded_bytes, + }) + .collect(), + }; + let prepared = store.prepare_image_root( + base, + &update, + &images.apply_limits(), + &mut vec![0; EXTENT_BYTES as usize], + )?; + ( + CellImagePin::Extents(Arc::new(prepared.image)), + CellImageGeometry { + length: geometry.length(), + page_size: geometry.page_size(), + }, + ) + } + }; + Ok(EnrollmentImage { + pin, + geometry, + effects: EffectSummary::default(), + }) +} + +fn prepare(voter: &mut Voter, apply: &CellGroupApply) -> Result { + if apply.entry.get_entry_type() == raft::prelude::EntryType::EntryConfChangeV2 { + let group = voter.log.cell_group(&apply.base.marker().route().history)?; + return Ok(voter.tree.prepare_membership( + voter.pages.as_ref().context("fixture page owner")?, + &apply.base, + &apply.entry, + &raft::Storage::initial_state(&group)?.conf_state, + &group.membership()?, + )?); + } + if apply.entry.data.is_empty() { + return Ok(voter.tree.prepare_unchanged( + voter.pages.as_ref().context("fixture page owner")?, + &apply.base, + apply.ticket.position(), + )?); + } + let decoded = CellGroupEntry::decode( + &apply.entry.data, + apply.base.marker().route(), + GroupEntryLimits::default(), + )?; + match decoded.command { + GroupCommand::Enroll { cell, image, files } => { + let (next, _) = voter.tree.prepare_enrollment( + voter.pages.as_mut().context("fixture page owner")?, + &apply.base, + EnrollmentApply { + position: apply.ticket.position(), + phase: EnrollmentPhase::Active, + entry: &apply.entry.data, + limits: GroupEntryLimits::default(), + }, + |_| { + prepare_cell_image( + (&voter.images, voter.extents.as_ref()), + apply.base.marker().route().schema_digest, + &cell, + 1, + None, + image, + &files, + ) + .map_err(|_| ManifestError::Invalid("fixture initial LTX")) + }, + )?; + Ok(next) + } + GroupCommand::Mutation(mutation) => { + let (old, base) = apply.base.pin_cell(&mutation.cell.cell).context("cell")?; + let prepared = prepare_cell_image( + (&voter.images, voter.extents.as_ref()), + apply.base.marker().route().schema_digest, + &mutation.cell, + mutation.sequence.get(), + Some(&base), + mutation.result, + &mutation.files, + )?; + let leaf = CellManifestLeaf { + enrollment: old.enrollment.clone(), + version: CellVersion { + history: mutation.cell, + sequence: mutation.sequence, + image: mutation.result.into(), + }, + geometry: prepared.geometry, + last_mutation: apply.ticket.position(), + witness: CommitWitness { + group: decoded.binding.group, + position: apply.ticket.position(), + proposal: decoded.binding.proposal, + }, + effects: old.effects, + }; + Ok(voter.tree.prepare_update( + voter.pages.as_mut().context("fixture page owner")?, + &apply.base, + leaf, + Some(&old.version), + prepared.pin, + )?) + } + _ => anyhow::bail!("unexpected administration"), + } +} + +fn history(route: &GroupRoute, name: &str) -> Result { + Ok(CellHistory { + cluster: route.history.cluster, + namespace: route.history.namespace, + cell: CellId::new(name)?, + incarnation: IncarnationId::new([8; 16])?, + }) +} + +fn request( + route: &GroupRoute, + cell: CellHistory, + sequence: u64, + cut: &images::Cut, +) -> Result> { + let image = |state: ImageState| LtxImage { + epoch: state.epoch, + txid: state.txid.0, + checksum: state.checksum, + commitment: state.digest.into(), + }; + let files = vec![GroupLtx { + min_txid: cut.base.txid.0 + 1, + max_txid: cut.result.txid.0, + page_count: cut.pages.len() as u64, + decoded_bytes: cut.pages.len() as u64 * 512, + bytes: &cut.bytes, + }]; + let command = if sequence == 1 { + GroupCommand::Enroll { + cell, + image: image(cut.result), + files, + } + } else { + GroupCommand::Mutation(GroupMutation { + cell, + sequence: MutationSequence::new(sequence)?, + base: image(cut.base), + result: image(cut.result), + files, + }) + }; + Ok(CellGroupEntry { + binding: GroupEntryBinding { + group: route.history, + route_epoch: route.epoch, + birth_digest: route.birth_digest, + schema_digest: route.schema_digest, + proposal: ProposalId::new([sequence as u8; 16])?, + }, + command, + } + .encode(route, GroupEntryLimits::default())?) +} diff --git a/crates/cellar/tests/support/cell_group_host.rs b/crates/cellar/tests/support/cell_group_host.rs new file mode 100644 index 000000000..eec5ab2b5 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_host.rs @@ -0,0 +1,537 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Namespace host over three engines/TLS peers, with production enrollment only. +use super::*; +use std::sync::{ + atomic::{AtomicUsize, Ordering}, + Mutex, +}; + +async fn idle(host: &CellGroupHost) -> Result<()> { + controller::wait_for(async || Ok(host.usage().controller.retained_commands == 0)).await +} + +fn routing() -> CellRoutingLimits { + CellRoutingLimits { + timeout: Duration::from_secs(15), + max_attempts: 4, + } +} + +async fn setup( + rooted: bool, + class: &Arc, +) -> Result<(RuntimeCluster, Vec, GroupRoute)> { + let filesystem = Arc::new(DirectFileSystem); + let cluster = Box::pin(Cluster::with_store_birth( + RaftRuntimeOptions::default(), + tempfile::tempdir()?, + filesystem.clone(), + class.schema_digest(), + Arc::new(object_store::memory::InMemory::new()), + ClusterBirthOptions { + group: GroupId::new([3; 16])?, + minimum: Durability::Quorum, + image_storage: if rooted { + RaftImageStorage::Extents + } else { + RaftImageStorage::Materialized + }, + log: ClusterLogLimits::default(), + }, + )) + .await?; + let mut f = Box::pin(RuntimeCluster::from_cluster(cluster, 256, 256, 32)).await?; + if rooted { + for (node, voter) in f.cluster.voters.iter_mut().enumerate() { + voter.extents = Some(NodeExtentStore::open( + f.cluster._root.path().join(format!("rooted-{node}")), + filesystem.clone(), + Default::default(), + )?); + } + } + let route = f.cluster.birth.route().clone(); + let template = history(&route, "resource-template")?; + for node in 0..3 { + apply::install(&mut f, &template, node).await?; + } + let controllers = controller::start_with_limits( + &mut f, + CellGroupControllerLimits { + max_groups: 1, + max_parallel_apply: 1, + max_commands: 1, + ..controller::limits() + }, + ) + .await?; + f.runtimes[0].campaign(route.history).await?; + controller::wait_for(async || Ok(f.runtimes[0].status(route.history).await?.ready)).await?; + Ok((f, controllers, route)) +} + +async fn registration( + f: &RuntimeCluster, + class: &Arc, + node: usize, +) -> Result { + let template = history(f.cluster.birth.route(), "namespace-options")?; + let mut options = f.cell_options_at(&template, node).await?; + let base = f.cluster._root.path().join(format!("namespace-sql-{node}")); + match &mut options { + CellSqlOptions::Materialized(o) => o.directory = base, + CellSqlOptions::Extents(o) => o.cache_directory = base, + } + Ok(CellGroupHostRegistration { + dispatcher: remote_queries::dispatcher(f, class.clone(), remote_queries::keys(190)?)?, + storage: CellGroupNamespaceStorage::new(options)?, + }) +} + +async fn bind( + f: &RuntimeCluster, + controllers: &[CellGroupController], + class: &Arc, + node: usize, +) -> Result { + Ok(CellGroupHost::bind( + &f.runtimes[node], + &f.appliers[&(node, f.cluster.birth.route().history)].0, + &f.transports[node], + controllers[node].clone(), + vec![registration(f, class, node).await?], + CellGroupHostLimits { + max_namespaces: 1, + namespace_bytes: 8 * 1024 * 1024, + }, + )?) +} + +fn command(class: &RaftClass, value: u8, retry_until_ms: u64) -> Result { + Ok(CellCommandRequest { + id: CommandId::new(ClientId::new([173; 16])?, 1)?, + contract: class.contract("Add").context("Add")?, + input: vec![value], + durability: Durability::Quorum, + retry_until_ms, + }) +} + +fn query(class: &RaftClass, consistency: CellQueryConsistency) -> Result { + Ok(CellQueryRequest { + invocation: ProposalId::new([174; 16])?, + contract: class.contract("Get").context("Get")?, + input: vec![], + consistency, + }) +} + +pub(crate) async fn many_cells(rooted: bool) -> Result<()> { + let initializers = Arc::new(AtomicUsize::new(0)); + let commands = Arc::new(AtomicUsize::new(0)); + let inits = initializers.clone(); + let calls = commands.clone(); + let class = Arc::new( + RaftClass::new("Counter", 1, [81; 32])? + .initializer([83; 32], move |_, tx| { + inits.fetch_add(1, Ordering::SeqCst); + tx.execute_batch("CREATE TABLE value(n); INSERT INTO value VALUES(0);")?; + Ok(()) + })? + .command("Add", [82; 32], move |ctx, input| { + calls.fetch_add(1, Ordering::SeqCst); + let n = *input.first().context("amount")?; + ctx.sql() + .execute("UPDATE value SET n=n+?", [i64::from(n)])?; + Ok(input.to_vec()) + })? + .query("Get", [84; 32], |ctx, _| { + Ok(ctx + .query_row("SELECT n FROM value", [], |r| r.get::<_, i64>(0))? + .to_be_bytes() + .to_vec()) + })?, + ); + let (f, controllers, route) = Box::pin(setup(rooted, &class)).await?; + let retry_until = f + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms().max(0) as u64 + 60_000 }) + .await?; + let mut hosts = Vec::new(); + for node in 0..3 { + hosts.push(bind(&f, &controllers, &class, node).await?); + } + let bytes_before = hosts[0].usage().namespace_bytes; + let mut observations = Vec::new(); + let mut tokens = Vec::new(); + for i in 0..8u8 { + let cell = history(&route, &format!("host-{i}"))?; + let handle = hosts[0].cell(cell.clone())?; + let enrollment = handle + .enroll( + CellEnrollmentRequest { + creation: ProposalId::new([i + 1; 16])?, + durability: Durability::Quorum, + }, + routing(), + ) + .await?; + assert_eq!(enrollment.record().cell(), &cell); + drop(enrollment); + idle(&hosts[0]).await?; + let call = handle.command(command(&class, i + 1, retry_until)?, routing())?; + // The response remains unpolled while the node enters the callback. + controller::wait_for(async || Ok(commands.load(Ordering::SeqCst) == usize::from(i) + 1)) + .await?; + let response = call.response().await?; + assert_eq!( + response + .local_receipt() + .expect("local command receipt") + .receipt + .result(), + &[i + 1] + ); + assert_eq!( + response + .local_receipt() + .expect("local command receipt") + .version + .history, + cell + ); + assert_eq!( + response + .local_receipt() + .expect("local command receipt") + .version + .sequence + .get(), + 2 + ); + observations.push( + response + .local_receipt() + .expect("local command receipt") + .observation(), + ); + tokens.push(response.token().clone()); + drop(response); + idle(&hosts[0]).await?; + } + assert_eq!(initializers.load(Ordering::SeqCst), 8); + assert_eq!(commands.load(Ordering::SeqCst), 8); + for (i, observation) in observations.iter().enumerate() { + for (node, host) in hosts.iter().enumerate() { + let handle = host.cell(observation.version.history.clone())?; + assert_eq!(handle.history(), &observation.version.history); + let response = handle + .query( + query(&class, CellQueryConsistency::AtLeast(tokens[i].clone()))?, + routing(), + )? + .response() + .await?; + assert_eq!(response.result(), (i as i64 + 1).to_be_bytes()); + assert_eq!(response.observation(), observation); + drop(response); + idle(host).await?; + assert_eq!(host.usage().namespaces, 1); + assert_eq!(host.usage().controller.groups, 1); + assert_eq!(f.cluster.voters[node].log.cell_groups()?.len(), 1); + } + } + let first = hosts[0].cell(observations[0].version.history.clone())?; + let retry = first + .command(command(&class, 1, retry_until)?, routing())? + .response() + .await?; + assert_eq!( + retry + .local_receipt() + .expect("local command receipt") + .observation(), + observations[0] + ); + drop(retry); + idle(&hosts[0]).await?; + assert_eq!(commands.load(Ordering::SeqCst), 8); + let receipt = first + .receipt( + CellReceiptRequest { + id: command(&class, 1, retry_until)?.id, + contract: class.contract("Add").context("Add")?, + input: vec![1], + durability: Durability::Quorum, + }, + routing(), + ) + .await?; + assert_eq!( + receipt.receipt().context("retained receipt")?.result(), + &[1] + ); + drop(receipt); + let wrong_token = first + .query( + query(&class, CellQueryConsistency::AtLeast(tokens[1].clone()))?, + routing(), + )? + .response() + .await; + assert!(wrong_token.is_err()); + let mut tenant = first.history().clone(); + tenant.namespace = NamespaceId::new([99; 16])?; + assert!(hosts[0].cell(tenant).is_err()); + let mut wrong = first.history().clone(); + wrong.incarnation = IncarnationId::new([99; 16])?; + let refused = hosts[0] + .cell(wrong)? + .query(query(&class, CellQueryConsistency::Stale)?, routing())? + .response() + .await; + assert!(refused.is_err()); + for _ in 0..100 { + drop(hosts[0].cell(first.history().clone())?); + } + assert_eq!(hosts[0].usage().namespace_bytes, bytes_before); + for host in &hosts { + host.close().await?; + assert!(host.is_closed()); + } + assert!(first + .command(command(&class, 1, retry_until)?, routing()) + .is_err()); + assert!(first + .query(query(&class, CellQueryConsistency::Stale)?, routing()) + .is_err()); + drop((first, hosts, controllers)); + f.close().await?; + Ok(()) +} + +pub(crate) async fn query_shutdown(rooted: bool) -> Result<()> { + for expired in [false, true] { + held_query_shutdown(rooted, expired).await?; + } + Ok(()) +} + +async fn held_query_shutdown(rooted: bool, expired: bool) -> Result<()> { + let (entered, started) = tokio::sync::oneshot::channel(); + let entered = Mutex::new(Some(entered)); + let (release, held) = std::sync::mpsc::sync_channel(1); + let held = Mutex::new(held); + let class = Arc::new( + RaftClass::new("Counter", 1, [81; 32])? + .initializer([83; 32], |_, tx| { + tx.execute_batch("CREATE TABLE value(n); INSERT INTO value VALUES(7);")?; + Ok(()) + })? + .command("Add", [82; 32], |_, input| Ok(input.to_vec()))? + .query("Get", [84; 32], move |ctx, _| { + let _ = entered + .lock() + .unwrap() + .take() + .context("one held query")? + .send(()); + held.lock().unwrap().recv_timeout(Duration::from_secs(15))?; + Ok(ctx + .query_row("SELECT n FROM value", [], |r| r.get::<_, i64>(0))? + .to_be_bytes() + .to_vec()) + })?, + ); + let (f, controllers, route) = Box::pin(setup(rooted, &class)).await?; + let retry_until = f + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms().max(0) as u64 + 60_000 }) + .await?; + let host = bind(&f, &controllers, &class, 0).await?; + let handle = host.cell(history(&route, "held-read")?)?; + drop( + handle + .enroll( + CellEnrollmentRequest { + creation: ProposalId::new([29; 16])?, + durability: Durability::Quorum, + }, + routing(), + ) + .await?, + ); + idle(&host).await?; + let pool = f.appliers[&(0, route.history)].0.clone(); + let mut limits = routing(); + if expired { + limits.timeout = Duration::from_millis(600); + } + let call = handle.query(query(&class, CellQueryConsistency::Linearizable)?, limits)?; + tokio::time::timeout(Duration::from_secs(5), started).await??; + assert_eq!(host.usage().controller.retained_commands, 1); + if expired { + let error = call + .response() + .await + .err() + .context("original read deadline")?; + assert!(matches!( + error.as_ref(), + CellDispatchError::Runtime(RuntimeError::DeadlineExceeded) + )); + } else { + drop(call); + } + assert_eq!(host.usage().controller.retained_commands, 1); + assert!( + f.cluster + .domain + .services() + .application_work() + .pending_events + > 0 + ); + let refused = handle + .command(command(&class, 1, retry_until)?, routing()) + .err() + .context("query retains global request admission")?; + assert!(matches!( + refused.cause.as_ref(), + CellDispatchError::Runtime(RuntimeError::Overloaded) + )); + assert!( + tokio::time::timeout(Duration::from_millis(30), host.close()) + .await + .is_err() + ); + assert!(host.is_closed()); + assert!(CellGroupRuntime::start( + f.cluster.voters[0].log.clone(), + RaftRuntimeOptions::default() + ) + .await + .is_err()); + release.send(())?; + tokio::time::timeout(Duration::from_secs(5), host.close()).await??; + assert_eq!(pool.usage()?.running_workers, 0); + assert_eq!(host.usage().controller.retained_commands, 0); + assert_eq!( + f.cluster + .domain + .services() + .application_work() + .pending_events, + 0 + ); + for c in &controllers { + c.close().await?; + } + drop((host, handle, controllers, pool)); + f.close().await?; + Ok(()) +} + +pub(crate) async fn registration_and_deadline(rooted: bool) -> Result<()> { + let initializers = Arc::new(AtomicUsize::new(0)); + let counted = initializers.clone(); + let class = Arc::new( + RaftClass::new("Counter", 1, [81; 32])? + .initializer([83; 32], move |_, tx| { + counted.fetch_add(1, Ordering::SeqCst); + tx.execute_batch("CREATE TABLE value(n);")?; + Ok(()) + })? + .command("Add", [82; 32], |_, input| Ok(input.to_vec()))?, + ); + let (f, controllers, route) = Box::pin(setup(rooted, &class)).await?; + let pool = &f.appliers[&(0, route.history)].0; + let duplicate = CellGroupHost::bind( + &f.runtimes[0], + pool, + &f.transports[0], + controllers[0].clone(), + vec![ + registration(&f, &class, 0).await?, + registration(&f, &class, 0).await?, + ], + CellGroupHostLimits { + max_namespaces: 2, + namespace_bytes: 16 * 1024 * 1024, + }, + ); + assert!(matches!( + duplicate, + Err(CellDispatchError::Invalid("duplicate host namespace")) + )); + let capacity = CellGroupHost::bind( + &f.runtimes[0], + pool, + &f.transports[0], + controllers[0].clone(), + vec![registration(&f, &class, 0).await?], + CellGroupHostLimits { + max_namespaces: 1, + namespace_bytes: 8192, + }, + ); + assert!(matches!( + capacity, + Err(CellDispatchError::Runtime(RuntimeError::Overloaded)) + )); + let foreign = CellGroupHost::bind( + &f.runtimes[0], + pool, + &f.transports[0], + controllers[1].clone(), + vec![registration(&f, &class, 0).await?], + CellGroupHostLimits { + max_namespaces: 1, + namespace_bytes: 8 * 1024 * 1024, + }, + ); + assert!(matches!( + foreign, + Err(CellDispatchError::Invalid("shared host resource owner")) + )); + assert!(!controllers[0].usage().closing); + let host = bind(&f, &controllers, &class, 0).await?; + let handle = host.cell(history(&route, "expired-enrollment")?)?; + let request = CellEnrollmentRequest { + creation: ProposalId::new([39; 16])?, + durability: Durability::Quorum, + }; + let pending = handle.enroll( + request, + CellRoutingLimits { + timeout: Duration::from_millis(40), + max_attempts: 1, + }, + ); + tokio::time::sleep(Duration::from_millis(60)).await; + let error = pending + .await + .err() + .context("original enrollment deadline")?; + assert!(!error.ambiguous); + assert!(matches!( + error.cause.as_ref(), + CellDispatchError::Routing(CellRoutingError { + reason: CellRoutingFailure::Deadline, + .. + }) + )); + assert_eq!(initializers.load(Ordering::SeqCst), 0); + assert_eq!(pool.usage()?.active_cells, 0); + drop(handle.enroll(request, routing()).await?); + assert_eq!(initializers.load(Ordering::SeqCst), 1); + for c in &controllers { + c.close().await?; + } + drop((host, handle, controllers)); + f.close().await?; + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_idle_cache.rs b/crates/cellar/tests/support/cell_group_idle_cache.rs new file mode 100644 index 000000000..2bc696d1d --- /dev/null +++ b/crates/cellar/tests/support/cell_group_idle_cache.rs @@ -0,0 +1,568 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Disposable SQL state is never a committed base or receipt authority. +use super::*; +use std::sync::atomic::{AtomicUsize, Ordering}; + +pub(crate) async fn idle_cache_reopens_exact_base() -> Result<()> { + for rooted in [false, true] { + let calls = Arc::new(AtomicUsize::new(0)); + let observed = calls.clone(); + let class = Arc::new(RaftClass::new("Counter", 1, [221; 32])?.command( + "Add", + [222; 32], + move |context, input| { + observed.fetch_add(1, Ordering::SeqCst); + let delta = i64::from_be_bytes(input.try_into()?); + let before: i64 = context + .sql() + .query_row("SELECT n FROM value", [], |row| row.get(0))?; + assert_eq!(before, if delta == 7 { 0 } else { 7 }); + context.sql().execute("UPDATE value SET n=n+?1", [delta])?; + Ok((before + delta).to_be_bytes().to_vec()) + }, + )?); + let (mut fixture, route, a, b) = Box::pin(registered::setup(rooted, &class)).await?; + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let dispatcher = CellGroupDispatcher::new( + &fixture.cluster.domain, + fixture.cluster.directory.clone(), + class.clone(), + VersionKeys::new( + VersionSigningKey::new([223; 32], u64::MAX)?, + vec![], + Duration::from_secs(30), + )?, + CellDispatchLimits { + max_cached_routes: 32, + route_cache_bytes: 256 * 1024, + max_invocations: 8, + retained_bytes: 1024 * 1024, + max_input_bytes: 1024, + }, + )?; + let retry_until = fixture + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms().max(0) as u64 + 60_000 }) + .await?; + let term = fixture.runtimes[0].status(route.history).await?.term; + let first = add( + &fixture, + &controllers[0], + &dispatcher, + &class, + &a, + (1, 7), + retry_until, + ) + .await?; + assert_eq!( + first + .local_receipt() + .expect("local command receipt") + .receipt + .result(), + 7_i64.to_be_bytes() + ); + assert_eq!( + first + .local_receipt() + .expect("local command receipt") + .version + .sequence + .get(), + 2 + ); + drop(first); + idle(&fixture, &controllers, route.history, &a, 2).await?; + let deadline = fixture + .cluster + .domain + .run(async { cellar::asyncrt::mono_ms() + 5000 }) + .await?; + let old = fixture.runtimes[0] + .read_authority( + route.history, + a.clone(), + CellReadConsistency::Stale, + deadline, + ) + .await?; + let version = old.cell().version.clone(); + for node in 0..3 { + let options = fixture.cell_options_at(&a, node).await?; + fixture + .cluster + .domain + .blocking(move || -> Result<()> { + match options { + CellSqlOptions::Materialized(options) => { + if node == 0 { + assert!(options.directory.is_dir()); + } + if options.directory.is_dir() { + let mut entries = std::fs::read_dir(&options.directory)?; + let only = entries.next().context("private lock inode")??; + assert_eq!(only.file_name(), "working.lock"); + assert!( + entries.next().is_none(), + "settled private scratch was retained" + ); + // Reacquiring this exact private lock proves the + // settled producer closed before cache eviction. + drop( + options + .host + .filesystem() + .lock_file(&options.directory.join("working.lock"))?, + ); + std::fs::remove_dir_all(&options.directory)?; + std::fs::create_dir_all(&options.directory)?; + std::fs::write( + options.directory.join("working.sqlite"), + b"discarded cache is not SQLite", + )?; + } + } + CellSqlOptions::Extents(options) => { + let usage = options.images.usage()?; + assert_eq!(usage.workspaces, 0); + assert_eq!(usage.workspace_bytes, 0); + if options.cache_directory.is_dir() { + std::fs::remove_dir_all(options.cache_directory)?; + } + } + } + Ok(()) + })? + .await??; + } + // A keeps its group and leadership. The next command must open from + // the verified committed image, including its receipt/client tables. + let next = add( + &fixture, + &controllers[0], + &dispatcher, + &class, + &a, + (2, 11), + retry_until, + ) + .await?; + assert_eq!( + next.local_receipt() + .expect("local command receipt") + .receipt + .result(), + 18_i64.to_be_bytes() + ); + assert_eq!( + next.local_receipt() + .expect("local command receipt") + .version + .sequence + .get(), + 3 + ); + drop(next); + idle(&fixture, &controllers, route.history, &a, 3).await?; + let duplicate = add( + &fixture, + &controllers[0], + &dispatcher, + &class, + &a, + (1, 7), + retry_until, + ) + .await?; + assert_eq!( + duplicate + .local_receipt() + .expect("local command receipt") + .receipt + .result(), + 7_i64.to_be_bytes() + ); + assert_eq!( + duplicate + .local_receipt() + .expect("local command receipt") + .version + .sequence + .get(), + 3 + ); + drop(duplicate); + idle(&fixture, &controllers, route.history, &a, 3).await?; + assert_eq!(calls.load(Ordering::SeqCst), 2); + assert_eq!(fixture.runtimes[0].status(route.history).await?.term, term); + let result = fixture.appliers[&(0, route.history)] + .0 + .query( + fixture.cell_options(&a).await?, + old, + move |database, selected| { + assert_eq!(selected, &version); + let value: i64 = + database + .connection() + .query_row("SELECT n FROM value", [], |row| row.get(0))?; + Ok(value.to_be_bytes().to_vec()) + }, + ) + .await?; + assert_eq!(result.result(), 7_i64.to_be_bytes()); + drop(result); + for (node, runtime) in fixture.runtimes.iter().enumerate() { + for (cell, expected, sequence) in [(&a, 18_i64, 3_u64), (&b, 0_i64, 1_u64)] { + let deadline = fixture + .cluster + .domain + .run(async { cellar::asyncrt::mono_ms() + 5000 }) + .await?; + let authority = runtime + .read_authority( + route.history, + cell.clone(), + CellReadConsistency::Stale, + deadline, + ) + .await?; + let result = fixture.appliers[&(node, route.history)] + .0 + .query( + fixture.cell_options_at(cell, node).await?, + authority, + move |database, selected| { + assert_eq!(selected.sequence.get(), sequence); + let value: i64 = database.connection().query_row( + "SELECT n FROM value", + [], + |row| row.get(0), + )?; + Ok(value.to_be_bytes().to_vec()) + }, + ) + .await?; + assert_eq!(result.result(), expected.to_be_bytes()); + drop(result); + } + } + for controller in &controllers { + controller.close().await?; + } + for (pool, _) in fixture.appliers.values() { + let usage = pool.usage()?; + assert_eq!(usage.running_workers, 0); + assert_eq!(usage.active_cells, 0); + assert_eq!(usage.retained_bytes, 0); + } + drop(controllers); + fixture.close().await?; + } + Ok(()) +} + +async fn idle( + fixture: &RuntimeCluster, + controllers: &[CellGroupController], + group: GroupHistory, + cell: &CellHistory, + sequence: u64, +) -> Result<()> { + controller::wait_for(async || { + for runtime in &fixture.runtimes { + if runtime + .reconciliation_authority(group, cell.clone()) + .await? + .cell() + .version + .sequence + .get() + != sequence + { + return Ok(false); + } + } + for controller in controllers { + if controller.usage().retained_commands != 0 + || controller.usage().active_apply != 0 + || controller.usage().pending_apply != 0 + { + return Ok(false); + } + } + for (pool, _) in fixture.appliers.values() { + let usage = pool.usage()?; + if usage.active_cells != 0 + || usage.retained_operations != 0 + || usage.retained_bytes != 0 + { + return Ok(false); + } + } + Ok(true) + }) + .await +} + +async fn add( + fixture: &RuntimeCluster, + controller: &CellGroupController, + dispatcher: &CellGroupDispatcher, + class: &RaftClass, + cell: &CellHistory, + (sequence, delta): (u64, i64), + retry_until_ms: u64, +) -> Result { + let invocation = dispatcher + .command( + cell.clone(), + CellCommandRequest { + id: CommandId::new(ClientId::new([224; 16])?, sequence)?, + contract: class.contract("Add").context("Add")?, + input: delta.to_be_bytes().to_vec(), + durability: Durability::Quorum, + retry_until_ms, + }, + CellRoutingLimits { + timeout: Duration::from_secs(5), + max_attempts: 4, + }, + None, + ) + .await?; + Ok(controller + .command(invocation, fixture.cell_options(cell).await?)? + .response() + .await?) +} + +/// The cleanup fails after a committed receipt exists, while the private lock +/// is still owned. A real leader handoff must preserve its ambiguous outcome. +pub(crate) async fn cleanup_failure_retains_outcome() -> Result<()> { + let calls = Arc::new(AtomicUsize::new(0)); + let observed = calls.clone(); + let gate = fault::Gate::new(None); + let class = Arc::new(RaftClass::new("Counter", 1, [227; 32])?.command( + "Add", + [228; 32], + move |context, _| { + observed.fetch_add(1, Ordering::SeqCst); + context.sql().execute("UPDATE value SET n=n+1", [])?; + Ok(b"committed".to_vec()) + }, + )?); + let (mut fixture, route, cell, _) = Box::pin(registered::setup(false, &class)).await?; + let controllers = controller::start_with_limits( + &mut fixture, + CellGroupControllerLimits { + max_groups: 1, + max_parallel_apply: 1, + ..controller::limits() + }, + ) + .await?; + let dispatcher = CellGroupDispatcher::new( + &fixture.cluster.domain, + fixture.cluster.directory.clone(), + class.clone(), + VersionKeys::new( + VersionSigningKey::new([229; 32], u64::MAX)?, + vec![], + Duration::from_secs(30), + )?, + CellDispatchLimits { + max_cached_routes: 32, + route_cache_bytes: 256 * 1024, + max_invocations: 8, + retained_bytes: 1024 * 1024, + max_input_bytes: 1024, + }, + )?; + let retry_until_ms = fixture + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms().max(0) as u64 + 60_000 }) + .await?; + let request = CellCommandRequest { + id: CommandId::new(ClientId::new([230; 16])?, 1)?, + contract: class.contract("Add").context("Add")?, + input: Vec::new(), + durability: Durability::Quorum, + retry_until_ms, + }; + let routing = CellRoutingLimits { + timeout: Duration::from_secs(5), + max_attempts: 4, + }; + let invocation = dispatcher + .command(cell.clone(), request.clone(), routing, None) + .await?; + let CellSqlOptions::Materialized(mut options) = fixture.cell_options(&cell).await? else { + anyhow::bail!("materialized private state required"); + }; + let directory = options.directory.clone(); + options.host = options + .host + .with_filesystem(Arc::new(fault::FaultFileSystem { gate: gate.clone() })); + let held = gate.arm(fault::HOLD_FAIL_PRIVATE_CLEANUP); + let call = controllers[0].command(invocation, CellSqlOptions::Materialized(options))?; + tokio::time::timeout(Duration::from_secs(5), gate.entered.notified()).await?; + let pool = &fixture.appliers[&(0, route.history)].0; + assert_eq!(pool.usage()?.active_cells, 1); + assert_eq!(controllers[0].usage().retained_commands, 1); + let old_term = fixture.runtimes[0].status(route.history).await?.term; + let mut transferred = false; + controller::wait_for(async || { + if !transferred { + match fixture.runtimes[0].transfer_leader(route.history, 2).await { + Ok(()) => transferred = true, + Err(RuntimeError::NotReady) => return Ok(false), + Err(error) => return Err(error.into()), + } + } + let status = fixture.runtimes[1].status(route.history).await?; + Ok(status.ready && status.term > old_term) + }) + .await?; + drop(held); + let error = call + .response() + .await + .err() + .context("cleanup failure cannot acknowledge success")?; + assert!(error.ambiguous); + assert!(matches!( + error.cause.as_ref(), + CellDispatchError::Execution(ExecutionError::Storage(_)) + )); + assert!(error + .to_string() + .contains("injected private scratch cleanup failure")); + controller::wait_for(async || Ok(controllers[0].usage().quarantined_groups == 1)).await?; + assert_eq!(controllers[0].usage().groups, 0); + assert_eq!(controllers[0].usage().retained_commands, 1); + assert_eq!(pool.usage()?.active_cells, 0); + assert!(!controllers[0].usage().closing); + let mut replacement = registration::prepare(&fixture, 31, &cell).await?; + let refused = controllers[0] + .register(replacement.inputs.remove(0)) + .err() + .context("quarantined group retains the only registration slot")?; + assert!(matches!( + refused.error, + CellGroupControllerError::Runtime(RuntimeError::Overloaded) + )); + drop(refused); + drop(replacement); + assert_eq!(controllers[0].usage().pending_registrations, 0); + assert_eq!(controllers[0].usage().quarantined_groups, 1); + fixture + .cluster + .domain + .blocking(move || -> Result<()> { + assert!(directory.join("working.sqlite").is_file()); + Ok(()) + })? + .await??; + // The next leader reads the original replicated receipt. Neither the lost + // cleanup response nor a discarded private image can rerun the mutation. + let invocation = dispatcher + .command(cell.clone(), request, routing, None) + .await?; + let receipt = controllers[1] + .command(invocation, fixture.cell_options_at(&cell, 1).await?)? + .response() + .await?; + assert_eq!( + receipt + .local_receipt() + .expect("local command receipt") + .receipt + .result(), + b"committed" + ); + assert_eq!( + receipt + .local_receipt() + .expect("local command receipt") + .version + .sequence + .get(), + 2 + ); + drop(receipt); + assert_eq!(calls.load(Ordering::SeqCst), 1); + assert_eq!(controllers[0].usage().retained_commands, 1); + controller::wait_for(async || { + for runtime in fixture.runtimes.iter().skip(1) { + if runtime + .reconciliation_authority(route.history, cell.clone()) + .await? + .cell() + .version + .sequence + .get() + != 2 + { + return Ok(false); + } + } + for (node, controller) in controllers.iter().enumerate().skip(1) { + let usage = fixture.appliers[&(node, route.history)].0.usage()?; + if controller.usage().retained_commands != 0 + || usage.active_cells != 0 + || usage.retained_operations != 0 + { + return Ok(false); + } + } + Ok(true) + }) + .await?; + for (node, runtime) in fixture.runtimes.iter().enumerate().skip(1) { + let deadline = fixture + .cluster + .domain + .run(async { cellar::asyncrt::mono_ms() + 5000 }) + .await?; + let authority = runtime + .read_authority( + route.history, + cell.clone(), + CellReadConsistency::Stale, + deadline, + ) + .await?; + let result = fixture.appliers[&(node, route.history)] + .0 + .query( + fixture.cell_options_at(&cell, node).await?, + authority, + |database, version| { + assert_eq!(version.sequence.get(), 2); + let value: i64 = + database + .connection() + .query_row("SELECT n FROM value", [], |row| row.get(0))?; + Ok(value.to_be_bytes().to_vec()) + }, + ) + .await?; + assert_eq!(result.result(), 1_i64.to_be_bytes()); + drop(result); + } + for controller in &controllers { + controller.close().await?; + } + assert_eq!(controllers[0].usage().retained_commands, 0); + for (pool, _) in fixture.appliers.values() { + assert_eq!(pool.usage()?.active_cells, 0); + assert_eq!(pool.usage()?.running_workers, 0); + } + drop(controllers); + fixture.close().await +} diff --git a/crates/cellar/tests/support/cell_group_log_reclaim.rs b/crates/cellar/tests/support/cell_group_log_reclaim.rs new file mode 100644 index 000000000..0f7acb684 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_log_reclaim.rs @@ -0,0 +1,730 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Physical Raft file reclamation, independent archive verification and retained +//! restart. Filesystem observations are assertions, not cache-debt estimates. +use super::*; +use raft::{GetEntriesContext, Storage as _}; +use std::path::PathBuf; +use std::sync::atomic::{AtomicUsize, Ordering}; + +const PREFIX: u64 = 8; +const TOTAL: u64 = 12; + +fn log_limits() -> ClusterLogLimits { + ClusterLogLimits { + target_file_bytes: 4096, + purge_threshold_bytes: 8192, + purge_interval_ms: None, + ..Default::default() + } +} + +fn payload(sequence: u64, cell: usize) -> Vec { + let mut bytes = sequence.to_be_bytes().to_vec(); + for block in 0..512_u64 { + let mut hash = Sha256::new(); + hash.update(sequence.to_be_bytes()); + hash.update((cell as u64).to_be_bytes()); + hash.update(block.to_be_bytes()); + bytes.extend_from_slice(&hash.finalize()); + } + bytes +} + +fn attempt(sequence: u64, cell: usize, input: &[u8]) -> Result { + let mut request = cell_attempt(100 + (sequence as u8) * 2 + cell as u8)?; + request.request.id = CommandId::new(ClientId::new([81; 16])?, sequence)?; + request.request.operation = "Write".into(); + request.request.contract_digest = [83; 32]; + request.request.request_digest = command_request_digest(input); + Ok(request) +} + +fn read_image(pin: &CellImagePin) -> Result> { + Ok(match pin { + CellImagePin::Empty => Vec::new(), + CellImagePin::Materialized(pin) => { + let mut reader = pin.open_reader()?; + let mut bytes = Vec::new(); + while (bytes.len() as u64) < reader.length() { + let count = (reader.length() - bytes.len() as u64).min(64 * 1024) as usize; + bytes.extend(reader.read_exact_at(bytes.len() as u64, count)?); + } + bytes + } + CellImagePin::Extents(pin) => { + let mut bytes = Vec::new(); + for n in 0..pin.manifest().geometry().extent_count() { + bytes.extend( + pin.view() + .read_extent(pin.manifest().extent(n).context("extent")?)?, + ); + } + bytes + } + }) +} + +fn assert_image(root: &PreparedGroupManifest, name: &CellId, count: u64) -> Result<()> { + let (leaf, pin) = root.pin_cell(name).context("selected SQL cell")?; + assert_eq!(leaf.version.sequence.get(), count + 1); + assert_eq!(leaf.effects.next_alarm_unix_millis, Some(5000)); + let temp = tempfile::tempdir()?; + let file = temp.path().join("oracle.sqlite"); + std::fs::write(&file, read_image(&pin)?)?; + let db = + rusqlite::Connection::open_with_flags(file, rusqlite::OpenFlags::SQLITE_OPEN_READ_ONLY)?; + assert_eq!( + db.query_row("SELECT n FROM value", [], |r| r.get::<_, u64>(0))?, + count + ); + assert_eq!( + db.query_row("SELECT COUNT(*) FROM payload", [], |r| r.get::<_, u64>(0))?, + count + ); + Ok(()) +} + +async fn command( + fixture: &RuntimeCluster, + class: Arc, + cell: &CellHistory, + ordinal: usize, + sequence: u64, +) -> Result { + let group = fixture.cluster.birth.route().history; + let pool = &fixture.appliers[&(0, group)].0; + let input = payload(sequence, ordinal); + let expected = Sha256::digest(&input).to_vec(); + let authority = fixture.runtimes[0] + .execution_authority(group, cell.clone()) + .await?; + let CellSqlOutcome::Prepared(prepared) = pool + .stage_registered( + fixture.cell_options(cell).await?, + authority, + attempt(sequence, ordinal, &input)?, + class, + input, + true, + ) + .await? + else { + anyhow::bail!("new registered command must prepare") + }; + pool.mark_submitted(cell.clone(), prepared.proposal()) + .await?; + let accepted = accepted_sql(&fixture.runtimes[0], &prepared).await?; + controller::applied(fixture, group, accepted.position.index).await?; + let authority = fixture.runtimes[0] + .execution_authority(group, cell.clone()) + .await?; + let receipt = pool.finish(authority).await?; + assert_eq!(receipt.receipt().receipt.result(), expected); + assert_eq!(receipt.receipt().version.history, *cell); + assert_eq!(receipt.receipt().version.sequence.get(), sequence + 1); + Ok(accepted.position.index) +} + +#[derive(Clone)] +struct SelectedLog { + marker: GroupManifest, + checkpoint: Option, + entries: Vec, +} + +impl SelectedLog { + fn capture(log: &ReplicaLog, history: GroupHistory) -> Result { + let group = log.cell_group(&history)?; + Ok(Self { + marker: group.group_manifest()?.context("selected manifest")?, + checkpoint: group.checkpoint_manifest()?, + entries: group.entries( + group.first_index()?, + group.last_index()? + 1, + None, + GetEntriesContext::empty(false), + )?, + }) + } + + fn check(&self, log: &ReplicaLog) -> Result<()> { + let actual = Self::capture(log, self.marker.route().history)?; + assert_eq!(actual.marker, self.marker); + assert_eq!(actual.checkpoint, self.checkpoint); + assert_eq!(actual.entries, self.entries); + Ok(()) + } +} + +async fn files(domain: &ExecutionDomain, path: PathBuf) -> Result> { + domain + .blocking(move || { + let mut files = BTreeMap::new(); + for item in std::fs::read_dir(path)? { + let item = item?; + let name = item.file_name().to_string_lossy().into_owned(); + if name.ends_with(".raftlog") || name.ends_with(".rewrite") { + files.insert(name, item.metadata()?.len()); + } + } + Ok(files) + })? + .await? +} + +async fn checkpoint( + fixture: &RuntimeCluster, + pool: &CellGroupCheckpointPool, + marker: &GroupManifest, +) -> Result<()> { + let history = marker.route().history; + let source = fixture.runtimes[0] + .capture_checkpoint(history, pool) + .await?; + let prepared = fixture + .cluster + .domain + .blocking(move || source.prepare_publication())? + .await??; + assert_eq!(prepared.marker(), marker); + fixture.runtimes[0] + .publish_checkpoint(history, prepared) + .await?; + controller::wait_for(async || { + Ok(fixture.cluster.voters[0] + .log + .cell_group(&history)? + .checkpoint_manifest()? + .as_ref() + == Some(marker) + && pool.usage().sources == 0) + }) + .await +} + +async fn recover_primary( + disk: tempfile::TempDir, + directory: RangeDirectory, + birth: CellGroupBirth, + store: Arc, + rooted: bool, + saved: &[Vec], +) -> Result { + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let mut voters = Vec::new(); + let mut appliers = Vec::new(); + for (node, selected) in saved.iter().enumerate() { + let number = node as u8 + 1; + let base = disk.path().join(format!("node-{number}")); + let mut options = ReplicaLogOptions::new( + base.join("log"), + NodeId::new([number; 16])?, + DiskLineage::new([number + 10; 16])?, + ProcessGeneration::new([91; 16])?, + LogOpenMode::Resume, + ); + options.target_file_bytes = log_limits().target_file_bytes; + options.purge_threshold_bytes = log_limits().purge_threshold_bytes; + options.purge_interval_ms = None; + let log = domain + .run(ReplicaLog::open(options, Arc::new(DirectFileSystem))) + .await??; + for expected in selected { + expected.check(&log)?; + } + let images = GenerationStore::open( + base.join("images"), + Arc::new(DirectFileSystem), + Default::default(), + )?; + let extents = if rooted { + Some(NodeExtentStore::open( + disk.path().join(format!("rooted-{node}")), + Arc::new(DirectFileSystem), + Default::default(), + )?) + } else { + None + }; + let config = checkpoint_receive::config(images.clone(), extents.clone(), &base)?; + let group = log.cell_group(&birth.route().history)?; + let (applier, root) = domain + .blocking(move || -> Result<_> { + let pages = ManifestPageStore::open( + &base.join("pages"), + Arc::new(DirectFileSystem), + 1024, + 16 * 1024 * 1024, + 16, + )?; + Ok(CellGroupApplier::recover( + group, + pages, + config, + Default::default(), + Default::default(), + 100, + )?) + })? + .await??; + let sibling = selected[1].clone(); + let sibling_group = log.cell_group(&sibling.marker.route().history)?; + let sibling_base = disk.path().join(format!("extra-31-node-{node}")); + domain + .blocking(move || -> Result<()> { + let pages = ManifestPageStore::open( + &sibling_base.join("pages"), + Arc::new(DirectFileSystem), + 1024, + 16 * 1024 * 1024, + 16, + )?; + let images = GenerationStore::open( + sibling_base.join("images"), + Arc::new(DirectFileSystem), + Default::default(), + )?; + let (applier, root) = CellGroupApplier::recover( + sibling_group, + pages, + checkpoint_receive::config(images, None, &sibling_base)?, + Default::default(), + Default::default(), + 100, + )?; + assert_eq!(root.marker(), &sibling.marker); + let (_, pin) = root + .pin_cell(&CellId::new("live-sibling")?) + .context("sibling image")?; + assert_eq!( + read_image(&pin)?, + images::image( + None, + "CREATE TABLE value(n); INSERT INTO value VALUES(197);" + )? + ); + drop(root); + drop(applier); + Ok(()) + })? + .await??; + let permit = domain + .run(directory.bootstrap_group_member(&birth, &log)) + .await??; + let owner = domain + .run(CellGroupOwner::start( + permit, + root, + RaftRuntimeOptions::default(), + [number + 70; 32], + )) + .await??; + let pool = CellGroupSqlPool::start(&log, sql_limits())?; + appliers.push((pool, applier)); + voters.push(Voter { + log, + owner: Some(owner), + pages: None, + images, + extents, + tree: ManifestTree::new(birth.route().clone(), Default::default())?, + gate: fault::Gate::new(None), + paused: false, + held: None, + }); + } + let history = birth.route().history; + let cluster = Cluster { + _root: disk, + domain, + directory: directory.clone(), + store, + birth, + voters, + network: VecDeque::new(), + reads: Vec::new(), + options: RaftRuntimeOptions::default(), + }; + let mut fixture = Box::pin(RuntimeCluster::from_cluster(cluster, 256, 256, 1_000_000)).await?; + for (node, applier) in appliers.into_iter().enumerate() { + fixture.appliers.insert((node, history), applier); + } + Ok(fixture) +} + +fn sql_limits() -> CellSqlLimits { + CellSqlLimits { + workers: 1, + queue_per_worker: 2, + max_pending_cells: 2, + group_pending_cells: 2, + node_bytes: 512 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + } +} + +// Construct the large scenario outside its caller's async poll frame, as the +// snapshot acceptance harness does for its composite physical scenarios. +#[inline(never)] +pub(crate) fn physical_log_reclamation( + rooted: bool, + store: Arc, +) -> std::pin::Pin>>> { + Box::pin(run(rooted, store)) +} + +async fn run(rooted: bool, store: Arc) -> Result<()> { + let calls = Arc::new(AtomicUsize::new(0)); + let observed = calls.clone(); + let class = Arc::new( + RaftClass::new("Counter", 1, [81; 32])? + .alarm("Wake", [82; 32], |_, _| Ok(()))? + .command("Write", [83; 32], move |context, input| { + observed.fetch_add(1, Ordering::SeqCst); + let sequence = u64::from_be_bytes(input[..8].try_into()?); + context.sql().execute_batch( + "CREATE TABLE IF NOT EXISTS payload(seq INTEGER PRIMARY KEY, body BLOB);", + )?; + context.sql().execute("UPDATE value SET n=n+1", [])?; + context.sql().execute( + "INSERT INTO payload VALUES(?1,?2)", + rusqlite::params![sequence, &input[8..]], + )?; + context.set_alarm("Wake", 5000, b"reclaimed-but-retained")?; + Ok(Sha256::digest(input).to_vec()) + })?, + ); + let cluster = Box::pin(Cluster::with_store_birth( + RaftRuntimeOptions::default(), + tempfile::tempdir()?, + Arc::new(DirectFileSystem), + class.schema_digest(), + store, + ClusterBirthOptions { + group: GroupId::new([3; 16])?, + image_storage: if rooted { + RaftImageStorage::Extents + } else { + RaftImageStorage::Materialized + }, + minimum: Durability::Quorum, + log: log_limits(), + }, + )) + .await?; + let (mut fixture, route, a, b) = Box::pin(registered::setup_cluster( + rooted, + cluster, + Arc::new(DirectFileSystem), + )) + .await?; + let second = controller::add_second(&mut fixture, route.history).await?; + let (archive, remote) = archive_controller::registration(&fixture, &route, rooted).await?; + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + fixture.runtimes[0].campaign(second).await?; + controller::wait_for(async || Ok(fixture.runtimes[0].status(second).await?.ready)).await?; + let sibling_route = fixture.cluster.voters[0] + .log + .cell_group(&second)? + .group_manifest()? + .context("sibling root")? + .route() + .clone(); + let sibling = history(&sibling_route, "live-sibling")?; + let sibling_image = images::image( + None, + "CREATE TABLE value(n); INSERT INTO value VALUES(197);", + )?; + let accepted = fixture.runtimes[0] + .propose( + second, + apply::enrollment_bytes(&sibling_route, &sibling, &sibling_image, false)?, + ) + .await?; + controller::applied(&fixture, second, accepted.position.index).await?; + controllers[0].archive(archive.clone())?.response().await?; + let mut reader = None; + for sequence in 1..=PREFIX { + for (ordinal, cell) in [&a, &b].into_iter().enumerate() { + command(&fixture, class.clone(), cell, ordinal, sequence).await?; + } + if sequence == 1 { + reader = Some(fixture.appliers[&(0, route.history)].1.recover_selected()?); + } + } + let reader = reader.context("old physical reader")?; + let (_, pin) = reader.pin_cell(&a.cell).context("old image")?; + let old_bytes = read_image(&pin)?; + drop(pin); + let group = fixture.cluster.voters[0].log.cell_group(&route.history)?; + let cut = group.group_manifest()?.context("covered cut")?; + archive_controller::published(&controllers[0], route.history, cut.position().index).await?; + let coverage = + archive_controller::verified_current(&fixture, &archive, rooted, "reclaim-audit").await?; + assert_eq!(coverage.marker(), &cut); + assert_image(coverage.recovery().manifest(), &a.cell, PREFIX)?; + assert_image(coverage.recovery().manifest(), &b.cell, PREFIX)?; + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let log_path = fixture.cluster._root.path().join("node-1/log/log"); + let before = files(&fixture.cluster.domain, log_path.clone()).await?; + assert!(before.len() > 8, "actual file rotation is required"); + checkpoint(&fixture, &pool, &cut).await?; + assert_eq!(group.first_index()?, cut.position().index.get() + 1); + let current = fixture.appliers[&(0, route.history)].1.recover_selected()?; + let mut retained_source = pool.capture_snapshot(&group, ¤t, &group.snapshot(0, 3)?)?; + drop(current); + remote.outage.store(true, Ordering::Release); + for sequence in PREFIX + 1..=TOTAL { + for (ordinal, cell) in [&a, &b].into_iter().enumerate() { + command(&fixture, class.clone(), cell, ordinal, sequence).await?; + } + } + let saved: Vec<_> = fixture + .cluster + .voters + .iter() + .map(|voter| { + Ok(vec![ + SelectedLog::capture(&voter.log, route.history)?, + SelectedLog::capture(&voter.log, second)?, + ]) + }) + .collect::>()?; + assert_eq!(saved[0][0].entries.len(), ((TOTAL - PREFIX) * 2) as usize); + assert!(!saved[0][1].entries.is_empty()); + let mut after = before.clone(); + for _ in 0..64 { + fixture.cluster.voters[0].log.purge().await?; + for expected in &saved[0] { + expected.check(&fixture.cluster.voters[0].log)?; + } + let (_, pin) = reader.pin_cell(&a.cell).context("retained old reader")?; + assert_eq!(read_image(&pin)?, old_bytes); + after = files(&fixture.cluster.domain, log_path.clone()).await?; + if before.keys().any(|name| !after.contains_key(name)) + && after.values().sum::() < before.values().sum::() + { + break; + } + } + let removed: Vec<_> = before + .keys() + .filter(|name| !after.contains_key(*name)) + .cloned() + .collect(); + assert!( + !removed.is_empty(), + "purge must remove actual pre-checkpoint files" + ); + assert!( + after.values().sum::() < before.values().sum::(), + "actual log-file length must decrease" + ); + let receiver_pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let destination = + archive_controller::verifier(&fixture, &route, rooted, "post-purge-snapshot").await?; + let mut receiver = receiver_pool.receive_archive( + destination, + checkpoint_receive::scratch()?, + fixture.cluster._root.path().into(), + )?; + let received = fixture + .cluster + .domain + .blocking(move || { + retained_source.stream(|chunk| receiver.append(chunk))?; + receiver.finish() + })? + .await??; + assert_eq!(received.marker(), &cut); + assert_eq!(pool.usage().sources, 0); + assert_eq!(receiver_pool.usage().sources, 1); + assert_image(received.manifest(), &a.cell, PREFIX)?; + assert_image(received.manifest(), &b.cell, PREFIX)?; + drop(received); + assert_eq!(receiver_pool.usage().sources, 0); + assert_eq!(pool.usage().sources, 0); + // A retained checkpoint is data for lagging/recovering destinations, never + // permission to roll back an already-ahead live voter. Exercise its original + // cut guard with a separate one-shot source, without enlarging either pool. + assert!(saved[2][0].marker.position().index > cut.position().index); + let current = fixture.appliers[&(0, route.history)].1.recover_selected()?; + let mut source = pool.capture_snapshot(&group, ¤t, &group.snapshot(0, 3)?)?; + drop(current); + let mut live = receiver_pool.receive( + fixture.appliers[&(2, route.history)].1.clone(), + checkpoint_receive::scratch()?, + fixture.cluster._root.path().into(), + )?; + fixture + .cluster + .domain + .blocking(move || -> Result<()> { + assert!(matches!( + source.stream(|chunk| live.append(chunk)), + Err(CellGroupCheckpointError::Invalid( + "receiver history/range/cut" + )) + )); + Ok(()) + })? + .await??; + assert_eq!(pool.usage().sources, 0); + assert_eq!(receiver_pool.usage().sources, 0); + // Independent verification owns the archive pool's sole receiver slot. + // Its checks and post-purge use are complete; the journal still owns the + // remote hold. Release this physical result before waiting for a new cut. + assert_eq!(archive.checkpoints.usage().sources, 1); + drop(coverage); + remote.outage.store(false, Ordering::Release); + archive_controller::published( + &controllers[0], + route.history, + saved[0][0].marker.position().index, + ) + .await + .with_context(|| { + format!( + "post-purge archive catch-up (checkpoint slots {})", + archive.checkpoints.usage().sources + ) + })?; + for controller in controllers { + controller.close().await?; + } + // Quiesce all node-zero SQL owners, then reconcile the complete two-group + // image inventory. Old readers and the selected checkpoint still pin their + // closures while obsolete image/page files can be physically collected. + let inventory: Vec<_> = fixture + .appliers + .iter() + .filter(|((node, _), _)| *node == 0) + .map(|(_, (_, applier))| applier.clone()) + .collect(); + fixture.appliers.retain(|(node, _), _| *node != 0); + let mut image_files = 0; + let mut page_files = 0; + for _ in 0..32 { + let report = fixture.cluster.voters[0] + .log + .reclaim_cell_storage( + inventory.clone(), + vec![], + CellGroupReclaimLimits { + max_selected_cells: 64, + max_files: 1, + max_page_scan: 64, + }, + ) + .await?; + assert_eq!(report.groups, 2); + assert!(report.image_files + report.page_files <= 1); + image_files += report.image_files; + page_files += report.page_files; + let (_, pin) = reader + .pin_cell(&a.cell) + .context("reader during collection")?; + assert_eq!(read_image(&pin)?, old_bytes); + } + assert!(image_files + page_files > 0, "actual orphan collection"); + let birth = fixture.cluster.birth.clone(); + let directory = fixture.cluster.directory.clone(); + let store = fixture.cluster.store.clone(); + let disk = std::mem::replace(&mut fixture.cluster._root, tempfile::tempdir()?); + drop(reader); + drop(group); + drop(archive); + drop(inventory); + fixture.close().await?; + let mut fixture = Box::pin(recover_primary( + disk, directory, birth, store, rooted, &saved, + )) + .await?; + for node in 0..3 { + let root = fixture.appliers[&(node, route.history)] + .1 + .recover_selected()?; + assert_image(&root, &a.cell, TOTAL)?; + assert_image(&root, &b.cell, TOTAL)?; + } + fixture.runtimes[0].campaign(route.history).await?; + fixture.drive(None).await?; + for sequence in 1..=TOTAL { + for (ordinal, cell) in [&a, &b].into_iter().enumerate() { + let input = payload(sequence, ordinal); + let authority = fixture.runtimes[0] + .execution_authority(route.history, cell.clone()) + .await?; + let CellSqlOutcome::Receipt(receipt) = fixture.appliers[&(0, route.history)] + .0 + .stage_registered( + fixture.cell_options(cell).await?, + authority, + attempt(sequence, ordinal, &input)?, + class.clone(), + input.clone(), + false, + ) + .await? + else { + anyhow::bail!("retained receipt") + }; + assert_eq!( + receipt.receipt().receipt.result(), + Sha256::digest(input).to_vec() + ); + } + } + assert_eq!(calls.load(Ordering::SeqCst), TOTAL as usize * 2); + let source = fixture.runtimes[0] + .capture_checkpoint(route.history, &pool) + .await?; + let prepared = fixture + .cluster + .domain + .blocking(move || source.prepare_publication())? + .await??; + let group = fixture.cluster.voters[0].log.cell_group(&route.history)?; + let mut write = GroupWrite::new(&group); + write.snapshot = Some(SnapshotChange::CellGroupCheckpoint(prepared)); + assert!( + matches!( + fixture.cluster.voters[0] + .log + .persist_cells(cellar_logic::consensus::OperationId::new(912), vec![write]) + .await, + Err(LogError::Overloaded) + ), + "reopen must not trust the lost archive frontier" + ); + drop(group); + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + for (ordinal, cell) in [&a, &b].into_iter().enumerate() { + command(&fixture, class.clone(), cell, ordinal, TOTAL + 1).await?; + } + assert_eq!(calls.load(Ordering::SeqCst), (TOTAL + 1) as usize * 2); + for node in 0..3 { + let root = fixture.appliers[&(node, route.history)] + .1 + .recover_selected()?; + assert_image(&root, &a.cell, TOTAL + 1)?; + assert_image(&root, &b.cell, TOTAL + 1)?; + } + for controller in controllers { + controller.close().await?; + } + cellar::note!( + "cell_group_log_reclamation_observations {}", + serde_json::json!({ + "mode": if rooted { "extents" } else { "materialized" }, + "files_before": before.len(), "files_after": after.len(), "removed": removed, + "bytes_before": before.values().sum::(), "bytes_after": after.values().sum::(), + "checkpoint": cut.position().index.get(), "retained_suffix_entries": saved[0][0].entries.len(), + "receipts_retried_without_callback": TOTAL * 2, "voters_reopened": 3, "groups_per_engine": 2, + "obsolete_image_files_removed": image_files, "obsolete_page_files_removed": page_files, + "new_commands_after_restart": 2 + }) + ); + fixture.close().await +} diff --git a/crates/cellar/tests/support/cell_group_log_retention.rs b/crates/cellar/tests/support/cell_group_log_retention.rs new file mode 100644 index 000000000..b35fdc45f --- /dev/null +++ b/crates/cellar/tests/support/cell_group_log_retention.rs @@ -0,0 +1,343 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Actual three-voter SQL and checkpoint boundaries at a small entry ceiling. +//! Full-backend outage authority and physical file amplification are separate. +use super::*; +use raft::{ + prelude::{ConfChangeSingle, ConfChangeTransition, ConfChangeType, ConfChangeV2}, + Storage as _, +}; +use std::sync::atomic::{AtomicUsize, Ordering}; + +pub(crate) async fn retained_log_admission(rooted: bool) -> Result<()> { + let mut retention = CellGroupLogRetentionLimits::default(); + retention.group.entries = 5; + retention.progress_group.entries = 1; + let cluster = Cluster::with_store_retention( + RaftRuntimeOptions::default(), + tempfile::tempdir()?, + Arc::new(DirectFileSystem), + [6; 32], + Arc::new(object_store::memory::InMemory::new()), + Durability::Quorum, + retention, + ) + .await?; + let fixture = RuntimeCluster::from_cluster(cluster, 256, 256, 1_000_000).await?; + let (mut fixture, route, a, b) = + enroll_sql_cells(fixture, rooted, Arc::new(DirectFileSystem)).await?; + let group = fixture.cluster.voters[0].log.cell_group(&route.history)?; + assert_eq!(group.log_retention_usage()?.retained_entries, 4); + let mut executor = fixture.private_cell(&a).await?; + let (mut executor, prepared) = fixture + .cluster + .domain + .blocking(move || -> Result<_> { + let CellStageOutcome::Prepared(prepared) = + executor.stage(cell_attempt(91)?, true, |tx| { + tx.execute("UPDATE value SET n=n+1", [])?; + Ok(b"retention-result".to_vec()) + })? + else { + anyhow::bail!("prepared A") + }; + executor.mark_submitted(prepared.proposal())?; + Ok((executor, prepared)) + })? + .await??; + let reserved = group.log_retention_usage()?; + assert_eq!( + (reserved.retained_entries, reserved.reserved_entries), + (4, 1) + ); + let calls = Arc::new(AtomicUsize::new(0)); + for committed in [false, true] { + if committed { + let accepted = accepted_private(&fixture.runtimes[0], &prepared).await?; + fixture.drive(Some(accepted.position.index)).await?; + assert_eq!(group.log_retention_usage()?.reserved_entries, 0); + assert_eq!(group.log_retention_usage()?.retained_entries, 5); + } + let mut other = fixture.private_cell(&b).await?; + let observed = calls.clone(); + fixture + .cluster + .domain + .blocking(move || -> Result<()> { + assert!(matches!( + other.stage(cell_attempt(92)?, true, move |_| { + observed.fetch_add(1, Ordering::SeqCst); + Ok(vec![]) + }), + Err(ExecutionError::Refused(CommandFailure::Overloaded)) + )); + other.close()?; + Ok(()) + })? + .await??; + assert_eq!(calls.load(Ordering::SeqCst), 0); + } + let authority = fixture.runtimes[0] + .execution_authority(route.history, a.clone()) + .await?; + fixture + .cluster + .domain + .blocking(move || -> Result<()> { + assert_eq!( + executor.finish(authority)?.receipt.result(), + b"retention-result" + ); + executor.close()?; + Ok(()) + })? + .await??; + // A known receipt still resolves when no new ordinary entry can fit. + let mut duplicate = fixture.private_cell(&a).await?; + fixture + .cluster + .domain + .blocking(move || -> Result<()> { + let CellStageOutcome::Duplicate(receipt) = + duplicate.stage(cell_attempt(93)?, true, |_| { + panic!("duplicate callback must not run at the retention ceiling") + })? + else { + anyhow::bail!("known receipt") + }; + assert_eq!(receipt.receipt.result(), b"retention-result"); + duplicate.close()?; + Ok(()) + })? + .await??; + let d = history(&route, "retention-new")?; + let pool = CellGroupSqlPool::start( + &fixture.cluster.voters[0].log, + CellSqlLimits { + workers: 1, + queue_per_worker: 2, + max_pending_cells: 2, + group_pending_cells: 2, + node_bytes: 512 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + }, + )?; + let authority = fixture.runtimes[0] + .enrollment_authority(route.history, d.clone()) + .await?; + let observed = calls.clone(); + let result = pool + .initialize( + fixture.cell_options(&d).await?, + authority, + CellEnrollmentAttempt { + creation: ProposalId::new([94; 16])?, + preview: ImageId::new([94; 16])?, + inputs: cellar::storage::CommandInputs { + now_unix_millis: 10, + random: [17; 32], + }, + }, + move |_| { + observed.fetch_add(1, Ordering::SeqCst); + Ok(()) + }, + ) + .await; + assert!(matches!( + result, + Err(ExecutionError::Refused(CommandFailure::Overloaded)) + )); + assert_eq!(calls.load(Ordering::SeqCst), 0); + pool.close().await?; + drop(pool); + // The saturated group must not consume the sibling's own entry allowance. + let sibling = fixture.add_group(47).await?; + fixture.runtimes[0].campaign(sibling).await?; + fixture.settle(&[route.history, sibling]).await?; + let other_route = fixture.extra[&sibling][0] + .log + .cell_group(&sibling)? + .group_manifest()? + .context("sibling root")? + .route() + .clone(); + let image = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(41);")?; + let cut = images::Cut::new(&[], &image, 0, true, false); + let bytes = request(&other_route, history(&other_route, "healthy")?, 1, &cut)?; + fixture.runtimes[0].propose(sibling, bytes).await?; + fixture.settle(&[route.history, sibling]).await?; + assert_eq!(group.log_retention_usage()?.retained_entries, 5); + // Only actual selected local checkpoints retire debt. Use the original + // one-source pool and wait for each physical publication before recapture. + let checkpoints = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + for node in 0..3 { + let source = fixture.runtimes[node] + .capture_checkpoint(route.history, &checkpoints) + .await?; + let prepared = fixture + .cluster + .domain + .blocking(move || source.prepare_publication())? + .await??; + let cut = prepared.marker().clone(); + fixture.runtimes[node] + .publish_checkpoint(route.history, prepared) + .await?; + tokio::time::timeout(Duration::from_secs(20), async { + loop { + fixture.pump(&[], &mut BTreeMap::new()).await?; + let group = fixture.cluster.voters[node] + .log + .cell_group(&route.history)?; + let usage = group.log_retention_usage()?; + if usage.retained_entries == 0 + && group.checkpoint_manifest()?.as_ref() == Some(&cut) + && checkpoints.usage().sources == 0 + { + break; + } + tokio::task::yield_now().await; + } + Ok::<_, anyhow::Error>(()) + }) + .await + .with_context(|| format!("voter {node} selects checkpoint and releases its source"))??; + } + assert_eq!(checkpoints.usage().sources, 0); + let mut executor = fixture.private_cell(&b).await?; + let (mut executor, next) = fixture + .cluster + .domain + .blocking(move || -> Result<_> { + let CellStageOutcome::Prepared(next) = + executor.stage(cell_attempt(95)?, true, |tx| { + tx.execute("UPDATE value SET n=n+7", [])?; + Ok(b"after-checkpoint".to_vec()) + })? + else { + anyhow::bail!("new callback after actual reclaim") + }; + executor.mark_submitted(next.proposal())?; + Ok((executor, next)) + })? + .await??; + let accepted = accepted_private(&fixture.runtimes[0], &next).await?; + fixture.drive(Some(accepted.position.index)).await?; + let authority = fixture.runtimes[0] + .execution_authority(route.history, b.clone()) + .await?; + fixture + .cluster + .domain + .blocking(move || -> Result<()> { + assert_eq!( + executor.finish(authority)?.receipt.result(), + b"after-checkpoint" + ); + executor.close()?; + Ok(()) + })? + .await??; + assert_eq!(group.log_retention_usage()?.reserved_entries, 0); + // Fill three of the four ordinary slots with real cell commands. The + // revision-bound joint configuration consumes the fourth; its automatic + // leave must use promised progress capacity while ordinary admission is full. + for sequence in 2..=3 { + let mut executor = fixture.private_cell(&b).await?; + let (mut executor, command) = fixture + .cluster + .domain + .blocking(move || -> Result<_> { + let mut attempt = cell_attempt(94 + sequence as u8)?; + attempt.request.id = CommandId::new(ClientId::new([81; 16])?, sequence)?; + let CellStageOutcome::Prepared(command) = executor.stage(attempt, true, |tx| { + tx.execute("UPDATE value SET n=n+1", [])?; + Ok(b"joint-progress".to_vec()) + })? + else { + anyhow::bail!("ordinary command before joint transition") + }; + executor.mark_submitted(command.proposal())?; + Ok((executor, command)) + })? + .await??; + let accepted = accepted_private(&fixture.runtimes[0], &command).await?; + fixture.drive(Some(accepted.position.index)).await?; + let authority = fixture.runtimes[0] + .execution_authority(route.history, b.clone()) + .await?; + fixture + .cluster + .domain + .blocking(move || -> Result<()> { + assert_eq!( + executor.finish(authority)?.receipt.result(), + b"joint-progress" + ); + executor.close()?; + Ok(()) + })? + .await??; + } + assert_eq!(group.log_retention_usage()?.retained_entries, 3); + let configuration = ConfChangeV2 { + transition: ConfChangeTransition::Auto, + changes: vec![ + ConfChangeSingle { + node_id: 3, + change_type: ConfChangeType::AddLearnerNode, + ..Default::default() + }, + ConfChangeSingle { + node_id: 2, + change_type: ConfChangeType::AddNode, + ..Default::default() + }, + ] + .into(), + ..Default::default() + }; + let request = CellGroupMembershipChange::new(&route, &group.membership()?, configuration, &[])?; + let accepted = fixture.runtimes[0] + .configure(route.history, request) + .await?; + tokio::time::timeout(Duration::from_secs(20), async { + loop { + fixture.pump(&[], &mut BTreeMap::new()).await?; + let conf = group.initial_state()?.conf_state; + let status = fixture.runtimes[0].status(route.history).await?; + if conf.voters_outgoing.is_empty() + && conf.learners == vec![3] + && status.applied.index > accepted.position.index + && fixture.applied[&route.history] + .iter() + .all(|index| *index >= status.applied.index) + { + break; + } + tokio::task::yield_now().await; + } + Ok::<_, anyhow::Error>(()) + }) + .await + .context("automatic joint leave progresses at the ordinary entry ceiling")??; + let usage = group.log_retention_usage()?; + assert_eq!((usage.retained_entries, usage.reserved_entries), (5, 0)); + for voter in &fixture.cluster.voters { + let group = voter.log.cell_group(&route.history)?; + assert_eq!(group.initial_state()?.conf_state.learners, vec![3]); + assert_eq!( + group + .group_manifest()? + .context("complete post-joint root")? + .root() + .cells(), + 3 + ); + assert_eq!(group.log_retention_usage()?.reserved_entries, 0); + } + drop(prepared); + drop(next); + fixture.close().await +} diff --git a/crates/cellar/tests/support/cell_group_managed_archives.rs b/crates/cellar/tests/support/cell_group_managed_archives.rs new file mode 100644 index 000000000..cacae8487 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_managed_archives.rs @@ -0,0 +1,183 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Post-startup archive ownership is exercised through actual managed Nodes. +use super::*; +pub(super) use crate::managed_fixture::archives::configure; +use crate::managed_fixture::archives::{initialize, open_provisioning}; + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn managed_dynamic_group_archives_and_joins_on_close() -> Result<()> { + for rooted in [false, true] { + Box::pin(archived_group(rooted)).await?; + } + Ok(()) +} + +async fn archived_group(rooted: bool) -> Result<()> { + let mut cluster = Box::pin(ManagedCluster::start_with_archives( + rooted, + &[], + ArchiveSetup::Initialized, + )) + .await?; + let work = async { + let seeded = seed_cells(&cluster.hosts, cluster.history).await?; + let index = leader(&cluster.hosts, cluster.history).await?; + let handle = cluster.hosts[index].cell(seeded.cells[0].clone())?; + let mut options = + CommandOptions::new(CommandId::new(ClientId::new([201; 16])?, 1)?, seeded.retry); + options.durability = Durability::S3; + let receipt = handle + .command( + &handle.contract("Add")?, + 3i64.to_be_bytes().to_vec(), + options, + ) + .await?; + ensure!(receipt.durability == Durability::S3 && receipt.result == 10i64.to_be_bytes()); + Ok::<_, anyhow::Error>(()) + } + .await; + if let Err(error) = work { + close_group_nodes(std::mem::take(&mut cluster.nodes), &cluster.hosts).await?; + cluster.fixtures[0].domain.shutdown().await; + return Err(error.context("S3 acknowledgement for a post-startup managed group")); + } + let store = cluster.fixtures[0].store.clone(); + store + .group_archive_read + .armed + .store(true, Ordering::Release); + // A fresh ordinary mutation makes a real archive job enter the remote read. + let index = leader(&cluster.hosts, cluster.history).await?; + let handle = cluster.hosts[index].cell(CellHistory { + cluster: cluster.history.cluster, + namespace: cluster.history.namespace, + cell: CellId::new("Counter:cold-0")?, + incarnation: IncarnationId::new([91; 16])?, + })?; + let retry = u64::try_from( + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH)? + .as_millis(), + )? + 60_000; + let quorum = handle + .command( + &handle.contract("Add")?, + 1i64.to_be_bytes().to_vec(), + CommandOptions::new(CommandId::new(ClientId::new([202; 16])?, 1)?, retry), + ) + .await?; + ensure!(quorum.durability == Durability::Quorum && quorum.result == 11i64.to_be_bytes()); + let entered = tokio::time::timeout(TIMEOUT, store.group_archive_read.entered.notified()).await; + let nodes = std::mem::take(&mut cluster.nodes); + let hosts = cluster.hosts.clone(); + let mut closing = tokio::spawn(async move { + futures_util::future::try_join_all(nodes.into_iter().zip(hosts).map( + |(node, host)| async move { + node.shutdown(ShutdownMode::Preserve).await?; + Ok::<_, anyhow::Error>(host.close().await?) + }, + )) + .await + }); + let premature = tokio::time::timeout(Duration::from_millis(100), &mut closing).await; + let returned_early = premature.is_ok(); + // Releasing the backend lets the admitted job settle and physical owners + // join. Expiring the close waiter above must not release those owners. + store + .group_archive_read + .armed + .store(false, Ordering::Release); + store.group_archive_read.resume.notify_one(); + let reports = if let Ok(result) = premature { + result?? + } else { + tokio::time::timeout(TIMEOUT, closing).await??? + }; + cluster.fixtures[0].domain.shutdown().await; + ensure!(entered.is_ok(), "no admitted archive read was observed"); + ensure!( + !returned_early, + "close returned while archive work was held" + ); + ensure!(reports.iter().all(|r| r.groups_total == 1)); + ensure!(reports + .iter() + .any(|r| r.groups.iter().any(|g| g.archived.is_some()))); + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn managed_dynamic_group_requires_explicit_archive_initialization() -> Result<()> { + for rooted in [false, true] { + let mut cluster = Box::pin(ManagedCluster::start_with_archives( + rooted, + &[], + ArchiveSetup::Missing, + )) + .await?; + let f = &cluster.fixtures[0]; + let work = async { + let birth = f + .domain + .run( + cluster + .directory + .materialize_selected_history(cluster.history), + ) + .await??; + let pool = CellGroupArchivePool::new(Default::default())?; + ensure!(cluster + .directory + .open_group_archive(f.domain.as_ref(), &pool, &birth) + .await + .is_err()); + let adapter = Catalog::new( + f.store.clone(), + CatalogOptions { + prefix: "managed/".into(), + ..Default::default() + }, + )?; + let setup = open_provisioning(f, &adapter).await?; + let candidate = setup.prepare_archive(cluster.history, TIMEOUT).await?; + ensure!(matches!( + setup + .verify_archive(cluster.history, candidate.encoded_intent(), TIMEOUT) + .await, + Err(FleetCatalogError::Corrupt("missing group archive binding")) + )); + initialize(f, &adapter, cluster.history, cluster.roots[0].path()).await?; + ensure!(matches!( + setup + .verify_archive(cluster.history, candidate.encoded_intent(), TIMEOUT) + .await, + Err(FleetCatalogError::Conflict) + )); + let cut = f.catalog.read(TIMEOUT).await?; + for host in &cluster.hosts { + let authorization = f + .catalog + .authorize_group(&cut, &cluster.history, TIMEOUT) + .await?; + host.activate_managed_initial_group(authorization, TIMEOUT) + .await?; + } + seed_cells(&cluster.hosts, cluster.history).await?; + Ok::<_, anyhow::Error>(()) + } + .await; + // Even a failed assertion must join all entered physical owners. + for (node, host) in std::mem::take(&mut cluster.nodes) + .into_iter() + .zip(&cluster.hosts) + { + node.shutdown(ShutdownMode::Preserve).await?; + host.close().await?; + } + f.domain.shutdown().await; + work?; + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_managed_bootstrap.rs b/crates/cellar/tests/support/cell_group_managed_bootstrap.rs new file mode 100644 index 000000000..7eb4a58f4 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_managed_bootstrap.rs @@ -0,0 +1,399 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Cold managed bootstrap must obtain inventory before reserving its first group. +use super::*; +use crate::managed_fixture::{ + admit_capacity, checkpoint_limits, close_group_nodes, counter_class, inspect_current, leader, + peer_fixture, seed_cells, ArchiveSetup, FirstGroupSetup, ManagedCluster, SeededCells, +}; +use cellar::application::consensus::verification::cell_groups::{ + CellGroupArchivePool, RangeDirectory, +}; + +#[path = "cell_group_managed_first_group_operator.rs"] +mod first_group_operator; + +#[path = "cell_group_managed_request_cases.rs"] +mod requests; + +#[path = "cell_group_managed_transition.rs"] +mod transition; + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn managed_cold_inventory_bootstrap() -> Result<()> { + let f = Fixture::new(false).await?; + let namespace = f + .domain + .run( + Catalog::new( + f.store.clone(), + CatalogOptions { + prefix: "managed/".into(), + ..Default::default() + }, + )? + .existing_namespace(f.declaration.namespace.clone()), + ) + .await??; + let directory = RangeDirectory::new(namespace, DirectoryLimits::default())?; + let initialization = directory.prepare_standby_initialization(ProposalId::new([100; 16])?)?; + let publication = f + .domain + .run(directory.publication(&initialization)) + .await?? + .context("SDK provisioning must publish standby before declaring the namespace")?; + ensure!(publication.tree_digest().is_none()); + let root = tempfile::tempdir()?; + let node = start_node(root.path(), f.application.clone(), &f.spec.label).await?; + let (config, services) = f.config(&node, LogOpenMode::Create, 24).await?; + let host = node.start_raft_cells(config, services).await?; + let cut = f.catalog.read(TIMEOUT).await?; + let inventory = f + .catalog + .check_inventory(&cut, std::slice::from_ref(&directory), TIMEOUT) + .await; + stopped(node, &host).await?; + f.domain.shutdown().await; + let inventory = inventory.context("cold namespace inventory before capacity/birth")?; + ensure!(inventory.namespaces() == 1 && inventory.histories() == 0); + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn managed_namespace_provisioning_replays_lost_confirmation() -> Result<()> { + let f = Fixture::new(false).await?; + let result = async { + let class = counter_class()?; + let mut config = f.declaration.namespace.clone(); + config.namespace = NamespaceId::new([3; 16])?; + config.schema_digest = class.schema_digest(); + let adapter = Catalog::new( + f.store.clone(), + CatalogOptions { + prefix: "managed/".into(), + ..Default::default() + }, + )?; + let before = f.catalog.read(TIMEOUT).await?.revision(); + let identity = request(94, before); + let provision = |namespace| { + f.catalog.provision_managed_sql_namespace( + &identity, + &adapter, + namespace, + &class, + DirectoryLimits::default(), + TIMEOUT, + ) + }; + f.store + .fail_fleet_confirmation + .store(true, Ordering::Release); + ensure!(matches!( + provision(config.namespace).await, + Err(FleetCatalogError::Storage(_)) + )); + ensure!(!f.store.fail_fleet_confirmation.load(Ordering::Acquire)); + let after = f.catalog.read(TIMEOUT).await?.revision(); + ensure!(after == before + 1); + let mut changed = config.clone(); + changed.namespace = NamespaceId::new([7; 16])?; + ensure!(matches!( + provision(changed.namespace).await, + Err(FleetCatalogError::PayloadConflict) + )); + ensure!(f + .domain + .run(adapter.existing_namespace(changed)) + .await? + .is_err()); + let replay = provision(config.namespace).await?; + ensure!(replay.class == class.name() && replay.namespace == config); + ensure!( + replay.namespace.mode == NamespaceMode::Raft + && replay.namespace.minimum_durability == Durability::Quorum + && replay.namespace.voters == VoterPolicy::ThreeVoters + && replay.namespace.image_storage == RaftImageStorage::Materialized + ); + ensure!(f.catalog.read(TIMEOUT).await?.revision() == after); + let opened = f + .domain + .run(adapter.existing_namespace(replay.namespace)) + .await??; + let directory = RangeDirectory::new(opened, DirectoryLimits::default())?; + let view = f.domain.run(directory.observe()).await??; + ensure!(view.revision() == 1 && view.groups() == 0); + Ok::<_, anyhow::Error>(()) + } + .await; + f.domain.shutdown().await; + result +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn managed_namespace_provisioning_reuses_standby_after_root_conflict() -> Result<()> { + let f = Fixture::new(false).await?; + let result = async { + let class = counter_class()?; + let mut config = f.declaration.namespace.clone(); + config.namespace = NamespaceId::new([3; 16])?; + config.schema_digest = class.schema_digest(); + let adapter = Catalog::new( + f.store.clone(), + CatalogOptions { + prefix: "managed/".into(), + ..Default::default() + }, + )?; + let provisioner = Fixture::open_catalog(&f.domain, f.store.clone(), config.cluster)?; + let revision = f.catalog.read(TIMEOUT).await?.revision(); + let original = request(94, revision); + let gate = &f.store.range_genesis_after; + gate.armed.store(true, Ordering::Release); + let (submitted, concurrent) = tokio::time::timeout(TIMEOUT, async { + tokio::join!( + provisioner.provision_managed_namespace( + &original, + &adapter, + config.clone(), + &class, + DirectoryLimits::default(), + TIMEOUT, + ), + async { + gate.entered.notified().await; + let result = peer_fixture(&f, 8).await; + gate.resume.notify_one(); + result + } + ) + }) + .await?; + concurrent?; + ensure!(matches!(submitted, Err(FleetCatalogError::Conflict))); + let advanced = f.catalog.read(TIMEOUT).await?.revision(); + ensure!(advanced == revision + 1); + let reopened = f + .domain + .run(adapter.existing_namespace(config.clone())) + .await??; + let directory = RangeDirectory::new(reopened, DirectoryLimits::default())?; + let standby = f.domain.run(directory.observe()).await??; + ensure!(standby.revision() == 1 && standby.groups() == 0); + let retry = request(95, advanced); + ensure!(retry.operation == original.operation && retry.id != original.id); + let declaration = provisioner + .provision_managed_namespace( + &retry, + &adapter, + config.clone(), + &class, + DirectoryLimits::default(), + TIMEOUT, + ) + .await?; + ensure!(declaration.namespace == config && declaration.class == class.name()); + ensure!(f.catalog.read(TIMEOUT).await?.revision() == advanced + 1); + ensure!(f.domain.run(directory.observe()).await??.digest() == standby.digest()); + Ok::<_, anyhow::Error>(()) + } + .await; + f.domain.shutdown().await; + result +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn managed_cold_capacity_group_and_retained_restart() -> Result<()> { + for rooted in [false, true] { + Box::pin(cold_group(rooted)).await?; + } + Ok(()) +} + +async fn cold_group(rooted: bool) -> Result<()> { + let first = Fixture::with_class(rooted, counter_class()?).await?; + let ManagedCluster { + fixtures, + roots, + nodes, + hosts, + directory, + history, + .. + } = Box::pin(ManagedCluster::start_with_applications( + first, + &[], + ArchiveSetup::Initialized, + false, + None, + FirstGroupSetup::ExerciseReplay, + )) + .await?; + let f = &fixtures[0]; + let proposal = || -> Result { + Ok(CellGroupBirthSpec { + history, + range: CellRange::all(), + epoch: RouteEpoch::new(1)?, + creation: ProposalId::new([150; 16])?, + voters: fixtures + .iter() + .take(3) + .enumerate() + .map(|(i, f)| { + Ok(InitialMember { + member: MemberId::new(i as u64 + 1)?, + node: f.spec.node, + }) + }) + .collect::>()?, + learners: vec![], + }) + }; + let SeededCells { + cells, + receipts, + retry, + } = seed_cells(&hosts, history).await?; + close_group_nodes(nodes, &hosts).await?; + drop(hosts); + let mut nodes = Vec::new(); + let mut hosts = Vec::new(); + for (i, fixture) in fixtures.iter().enumerate() { + let node = start_node( + roots[i].path(), + fixture.application.clone(), + &fixture.spec.label, + ) + .await?; + let (config, services) = fixture + .config(&node, LogOpenMode::Resume, 34 + i as u8) + .await?; + hosts.push(node.start_raft_cells(config, services).await?); + nodes.push(node); + } + let index = leader(&hosts, history).await?; + for (i, (cell, original)) in cells.iter().zip(&receipts).enumerate() { + let handle = hosts[index].cell(cell.clone())?; + let delta = [7i64, 11][i]; + let duplicate = handle + .command( + &handle.contract("Add")?, + delta.to_be_bytes().to_vec(), + CommandOptions::new(original.id, retry), + ) + .await?; + ensure!(duplicate.id == original.id && duplicate.result == original.result); + let observed = handle + .query( + ProposalId::new([190 + i as u8; 16])?, + &handle.contract("Get")?, + vec![], + QueryOptions { + consistency: QueryConsistency::AtLeast(original.version_token.clone()), + timeout: TIMEOUT, + ..Default::default() + }, + ) + .await?; + ensure!( + observed.result == original.result, + "retained duplicate ran twice" + ); + let next = CommandId::new(original.id.client, 2)?; + let updated = handle + .command( + &handle.contract("Add")?, + 1i64.to_be_bytes().to_vec(), + CommandOptions::new(next, retry), + ) + .await?; + ensure!(updated.result == (delta + 1).to_be_bytes()); + } + let cut = f.catalog.read(TIMEOUT).await?; + ensure!( + f.catalog + .check_inventory(&cut, std::slice::from_ref(&directory), TIMEOUT) + .await? + .histories() + == 1 + ); + for fixture in &fixtures { + ensure!( + matches!( + f.catalog + .authorize_capacity(&cut, fixture.spec.node, TIMEOUT) + .await, + Err(FleetCatalogError::Conflict) + ), + "old boot capacity survived restart" + ); + } + for host in &hosts { + ensure!( + host.observe_node(TIMEOUT).await?.node.groups == 1 + && host.resource_usage().retained_manifest_cells == Some(2) + ); + } + let next_birth = || -> Result { + Ok(CellGroupBirthSpec { + history: GroupHistory { + group: GroupId::new([4; 16])?, + incarnation: IncarnationId::new([8; 16])?, + ..history + }, + creation: ProposalId::new([151; 16])?, + ..proposal()? + }) + }; + let attempted = f + .catalog + .reserve_birth( + &cut, + &request(79, cut.revision()), + &directory, + next_birth()?, + "Counter", + TIMEOUT, + ) + .await; + let blocked = matches!(attempted, Err(FleetCatalogError::Conflict)); + let after = f.catalog.read(TIMEOUT).await?.revision(); + if !blocked || after != cut.revision() { + close_group_nodes(nodes, &hosts).await?; + f.domain.shutdown().await; + anyhow::bail!("new boot admitted a birth using stale capacity: blocked={blocked}, before={}, after={after}", cut.revision()); + } + for (i, fixture) in fixtures.iter().enumerate() { + admit_capacity(fixture, &hosts[i], &directory, 70 + 2 * i as u8).await?; + } + let cut = f.catalog.read(TIMEOUT).await?; + f.catalog + .reserve_birth( + &cut, + &request(80, cut.revision()), + &directory, + next_birth()?, + "Counter", + TIMEOUT, + ) + .await?; + let cut = f.catalog.read(TIMEOUT).await?; + ensure!( + f.catalog + .check_inventory(&cut, std::slice::from_ref(&directory), TIMEOUT) + .await? + .histories() + == 2 + ); + leader(&hosts, history).await?; + close_group_nodes(nodes, &hosts).await?; + f.domain.shutdown().await; + Ok(()) +} + +#[path = "cell_group_managed_lifecycle.rs"] +mod lifecycle; + +#[path = "cell_group_managed_archives.rs"] +mod archives; diff --git a/crates/cellar/tests/support/cell_group_managed_cold_router.rs b/crates/cellar/tests/support/cell_group_managed_cold_router.rs new file mode 100644 index 000000000..cf3a9addd --- /dev/null +++ b/crates/cellar/tests/support/cell_group_managed_cold_router.rs @@ -0,0 +1,388 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! A retained ingress returns with no local group after all birth voters retire. +use super::*; + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn managed_cold_router_returns_after_all_birth_voters_retire() -> Result<()> { + for rooted in [false, true] { + Box::pin(cold_return(rooted, Arc::new(fault::Store::default()))).await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[ignore = "requires explicit owned RustFS endpoint, bucket and isolated prefix"] +async fn rustfs_managed_cold_router_returns_after_all_birth_voters_retire() -> Result<()> { + for rooted in [false, true] { + let store = Arc::new(fault::Store::around(rustfs::store("cold-router", rooted)?)); + Box::pin(cold_return(rooted, store)).await?; + } + Ok(()) +} + +async fn cold_return(rooted: bool, store: Arc) -> Result<()> { + let disk = tempfile::tempdir()?; + let mut cluster = Box::pin(ManagedCluster::start_with_store( + rooted, + &[7, 8, 9], + ArchiveSetup::Initialized, + store, + )) + .await?; + let work = Box::pin(exercise(&mut cluster, disk.path())).await; + let joined = close(std::mem::take(&mut cluster.nodes), &cluster.hosts).await; + cluster.fixtures[0].domain.shutdown().await; + if work.is_err() || joined.is_err() { + for disk in std::iter::once(disk).chain(std::mem::take(&mut cluster.roots)) { + eprintln!("retained cold router disk: {}", disk.keep().display()); + } + } + work?; + joined +} + +async fn exercise(cluster: &mut ManagedCluster, disk: &Path) -> Result<()> { + let seed = seed_cells(&cluster.hosts, cluster.history).await?; + let mut live = [0, 1, 2]; + let router = peer_fixture(&cluster.fixtures[0], 10).await?; + let (node, ingress, peer) = + open_router(cluster.history, &router, disk, LogOpenMode::Create, 63).await?; + let initial = async { + grant(cluster, &cluster.hosts[..3], &live, &router, &peer).await?; + let handle = ingress.cell(seed.cells[0].clone())?; + let read = read_at(&handle, MemberId::new(1)?, &seed.receipts[0], 213).await?; + ensure!(read.served_by_node.node == cluster.fixtures[0].spec.node); + Ok::<_, anyhow::Error>(()) + } + .await; + close_router(node, ingress).await?; + initial?; + + for (spare, ids) in [(3, [220, 230]), (4, [221, 231]), (5, [222, 232])] { + Box::pin(replace_birth_voter(cluster, &mut live, spare, ids)).await?; + } + live.sort_unstable(); + ensure!(live == [3, 4, 5]); + + // No old voter process survives the cold ingress restart. Keep its disk + // and its fleet accounting records so the test never grants debt release. + let survivors = cluster.nodes.split_off(3); + let retired = std::mem::replace(&mut cluster.nodes, survivors); + let retired_hosts = cluster.hosts.drain(..3).collect::>(); + close(retired, &retired_hosts).await?; + + let (node, ingress, peer) = + open_router(cluster.history, &router, disk, LogOpenMode::Resume, 64).await?; + let returned = async { + let survivors = [3, 4, 5]; + grant(cluster, &cluster.hosts, &survivors, &router, &peer).await?; + let handle = ingress.cell(seed.cells[0].clone())?; + let member = MemberId::new(4)?; + let original = &seed.receipts[0]; + let read = read_at(&handle, member, original, 214).await?; + ensure!(read.served_by_node.node == cluster.fixtures[3].spec.node); + let add = handle.contract("Add")?; + let replay = handle + .command( + &add, + 7i64.to_be_bytes().to_vec(), + CommandOptions::new(original.id, seed.retry), + ) + .await?; + ensure!(replay.id == original.id && replay.result == original.result); + let next = CommandId::new(original.id.client, 2)?; + let written = handle + .command( + &add, + 1i64.to_be_bytes().to_vec(), + CommandOptions::new(next, seed.retry), + ) + .await?; + ensure!( + written.id == next + && written.result == 8i64.to_be_bytes() + && written.durability == Durability::Quorum + ); + let verified = read_at(&handle, member, &written, 215).await?; + ensure!(verified.served_by_node.node == cluster.fixtures[3].spec.node); + let other = ingress.cell(seed.cells[1].clone())?; + let unchanged = read_at(&other, member, &seed.receipts[1], 216).await?; + ensure!(unchanged.result == 11i64.to_be_bytes()); + ensure!(ingress.resource_usage().host.controller.groups == 0); + Ok::<_, anyhow::Error>(()) + } + .await; + close_router(node, ingress).await?; + returned +} + +async fn replace_birth_voter( + cluster: &ManagedCluster, + live: &mut [usize], + spare: usize, + ids: [u8; 2], +) -> Result<()> { + let mut source = live_leader(cluster, live).await?; + let obsolete = *live + .iter() + .filter(|i| **i < 3) + .find(|i| **i != source) + .or_else(|| live.iter().find(|i| **i < 3)) + .context("retained birth voter")?; + if source == obsolete { + let target = *live + .iter() + .find(|i| **i != obsolete) + .context("joined survivor")?; + let group = cluster.hosts[source].group(cluster.history)?; + let progress = group.replication_status(TIMEOUT).await?; + let target_member = MemberId::new(target as u64 + 1)?; + group + .transfer_leadership( + LeadershipTransfer { + target: target_member, + expected_term: progress.term, + expected_revision: progress.membership_revision, + }, + TIMEOUT, + ) + .await?; + tokio::time::timeout(TIMEOUT, async { + loop { + let status = cluster.hosts[target] + .group(cluster.history)? + .status() + .await?; + if status.ready && status.leader == Some(target_member.get()) { + return Ok::<_, anyhow::Error>(()); + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .context("handoff from the final birth voter")??; + source = target; + } + + discover(cluster).await?; + let (_, execution) = admit_replacement( + cluster, + ReplacementTarget { + source, + obsolete, + spare, + }, + ids, + false, + ) + .await?; + let catalog = &cluster.fixtures[0].catalog; + let mut settled = false; + for turn in 0..8 { + discover(cluster).await?; + let cut = catalog.read(TIMEOUT).await?; + let result = catalog + .reconcile_movement(&cut, &execution, &cluster.hosts[source], TIMEOUT) + .await; + match result { + Ok(progress) if progress.settled => { + ensure!(progress.phase == FleetMovementPhase::Complete); + settled = true; + break; + } + Ok(_) => { + let current = catalog.read(TIMEOUT).await?; + let pending = catalog.pending_reservations(¤t, TIMEOUT).await?; + ensure!(pending.len() == 1 && pending[0].reservation == execution.reservation); + } + Err(FleetCatalogError::Unconfirmed) => { + let current = catalog.read(TIMEOUT).await?; + let pending = catalog.pending_reservations(¤t, TIMEOUT).await?; + ensure!( + pending.is_empty() + || (pending.len() == 1 && pending[0].reservation == execution.reservation) + ); + // An absent debit can follow a lost terminal reply. The next + // exact reconciliation still has to return the terminal receipt. + } + Err(error) => { + return Err(error).with_context(|| format!("cold router move {spare}, turn {turn}")) + } + } + } + ensure!( + settled, + "cold router move for voter {} did not settle", + spare + 1 + ); + let cut = catalog.read(TIMEOUT).await?; + ensure!(catalog + .pending_reservations(&cut, TIMEOUT) + .await? + .is_empty()); + ensure!(catalog.pending_movements(&cut, TIMEOUT).await?.is_empty()); + discover(cluster).await?; + let inventory = cluster.hosts[source] + .inspect_peer_physical_inventory( + cluster.fixtures[obsolete].spec.node, + cut.revision(), + TIMEOUT, + ) + .await? + .inventory + .context("retired birth voter inventory")?; + let history = inventory + .group_history(&cluster.history) + .context("retained birth voter root")?; + ensure!(!history.in_configuration() && history.member().get() == obsolete as u64 + 1); + let retired = cluster.hosts[obsolete].group(cluster.history)?; + let read = tokio::time::timeout( + Duration::from_millis(250), + retired.quorum_membership(TIMEOUT), + ) + .await + .context("retired voter held a quorum read until its deadline")?; + ensure!(matches!(read, Err(RuntimeError::NotFound))); + *live + .iter_mut() + .find(|i| **i == obsolete) + .context("replace live birth voter")? = spare; + let leader = live_leader(cluster, live).await?; + let quorum = cluster.hosts[leader] + .group(cluster.history)? + .quorum_membership(TIMEOUT) + .await?; + let mut actual = quorum.configuration().voters.clone(); + let mut expected = live.iter().map(|i| *i as u64 + 1).collect::>(); + actual.sort_unstable(); + expected.sort_unstable(); + ensure!(actual == expected && quorum.configuration().voters_outgoing.is_empty()); + ensure!(quorum.configuration().learners.is_empty()); + Ok(()) +} + +async fn live_leader(cluster: &ManagedCluster, live: &[usize]) -> Result { + let hosts = live + .iter() + .map(|i| cluster.hosts[*i].clone()) + .collect::>(); + let members = live.iter().map(|i| *i as u64 + 1).collect::>(); + let index = current_leader(&hosts, &members, cluster.history).await?; + Ok(live[index]) +} + +async fn open_router( + history: GroupHistory, + fixture: &Fixture, + disk: &Path, + mode: LogOpenMode, + process: u8, +) -> Result<(Node, RaftCells, RaftPeerNode)> { + let node = start_node(disk, fixture.application.clone(), &fixture.spec.label).await?; + let started = async { + let (mut config, services) = fixture.config(&node, mode, process).await?; + config.groups.push(RaftGroupAssignment { + history, + placement: RaftGroupPlacement::QueryRouter, + }); + let peer = config.peers[0].clone(); + let ingress = node.start_raft_cells(config, services).await?; + ensure!(ingress.observe_node(TIMEOUT).await?.node.groups == 0); + Ok::<_, anyhow::Error>((ingress, peer)) + } + .await; + match started { + Ok((ingress, peer)) => Ok((node, ingress, peer)), + Err(error) => { + node.shutdown(ShutdownMode::Preserve).await?; + Err(error) + } + } +} + +async fn close_router(node: Node, ingress: RaftCells) -> Result<()> { + ensure!(matches!( + node.shutdown(ShutdownMode::Preserve).await?, + NodeOutcome::Stopped(_) + )); + let report = ingress.close().await?; + ensure!( + report.groups_total == 0 + && report.mutation_outcomes == MutationOutcomes::NoUnresolvedLocalProducers + ); + Ok(()) +} + +async fn grant( + cluster: &ManagedCluster, + hosts: &[RaftCells], + members: &[usize], + router: &Fixture, + peer: &RaftPeerNode, +) -> Result<()> { + let cut = router.catalog.read(TIMEOUT).await?; + ensure!(hosts.len() == members.len()); + for (host, member) in hosts.iter().zip(members) { + ensure!(host.identity().node == cluster.fixtures[*member].spec.node); + inspect_current(host, router.spec.node, cut.revision()).await?; + host.grant_query_origin(NamespaceQueryOrigin { + cluster: cluster.history.cluster, + namespace: cluster.history.namespace, + revision: 1, + peer: peer.clone(), + })?; + host.grant_command_origin(NamespaceCommandOrigin { + cluster: cluster.history.cluster, + namespace: cluster.history.namespace, + revision: 1, + peer: peer.clone(), + })?; + } + Ok(()) +} + +async fn read_at( + handle: &RaftCellHandle, + member: MemberId, + receipt: &CommandReceipt, + invocation: u8, +) -> Result { + let contract = handle.contract("Get")?; + let deadline = tokio::time::Instant::now() + TIMEOUT; + let mut last = None; + let result = tokio::time::timeout_at(deadline, async { + loop { + let budget = deadline + .saturating_duration_since(tokio::time::Instant::now()) + .min(Duration::from_millis(500)); + match handle + .query( + ProposalId::new([invocation; 16])?, + &contract, + vec![], + QueryOptions { + consistency: QueryConsistency::AtLeast(receipt.version_token.clone()), + timeout: budget, + routing: QueryRouting::Replicas { + members: vec![member], + attempt_timeout: budget, + leader_fallback: false, + }, + ..Default::default() + }, + ) + .await + { + Ok(read) => { + ensure!(read.served_by == member && read.result == receipt.result); + return Ok::<_, anyhow::Error>(read); + } + Err(error) => last = Some(error), + } + tokio::time::sleep(Duration::from_millis(20)).await; + } + }) + .await; + result.with_context(|| format!("cold ingress did not route to voter {member:?}: {last:?}"))? +} diff --git a/crates/cellar/tests/support/cell_group_managed_controls.rs b/crates/cellar/tests/support/cell_group_managed_controls.rs new file mode 100644 index 000000000..02d6abff5 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_managed_controls.rs @@ -0,0 +1,144 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Public SDK observation and guarded control of a shared physical group. +use super::*; + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn managed_group_observation_and_guarded_handoff() -> Result<()> { + for rooted in [false, true] { + Box::pin(guarded_handoff(rooted)).await?; + } + Ok(()) +} + +async fn guarded_handoff(rooted: bool) -> Result<()> { + let mut cluster = Box::pin(ManagedCluster::start(rooted, &[])).await?; + let mut retained = None; + let work = async { + let seed = seed_cells(&cluster.hosts, cluster.history).await?; + let initial = leader(&cluster.hosts, cluster.history).await?; + let target = (initial + 1) % 3; + for (i, host) in cluster.hosts.iter().enumerate() { + let group = host.group(cluster.history)?; + let observed: CellGroupObservation = group.observe(TIMEOUT).await?; + ensure!(observed.replica.scope == cluster.history); + ensure!(observed.replica.node == cluster.fixtures[i].spec.node); + ensure!(observed.replica.member.get() == i as u64 + 1); + ensure!(observed.cells == 2); + ensure!(observed.checkpoint.index <= observed.status.applied.index); + ensure!(observed.status.applied.index <= observed.status.committed); + ensure!(observed.status.committed <= observed.persisted); + ensure!(observed.membership_revision <= observed.status.applied.index); + ensure!( + observed.persisted.get() - observed.checkpoint.index.get() + == observed.retained_entries as u64 + ); + ensure!( + observed.role + == if i == initial { + RaftReplicaRole::Leader + } else { + RaftReplicaRole::Voter + } + ); + // Merely reading a publication must not turn it into coverage. + let _: Option = group.archive_status()?; + } + let group = cluster.hosts[initial].group(cluster.history)?; + let progress = group.replication_status(TIMEOUT).await?; + let transfer = LeadershipTransfer { + target: MemberId::new(target as u64 + 1)?, + expected_term: progress.term, + expected_revision: progress.membership_revision, + }; + for timeout in [Duration::ZERO, Duration::from_secs(61)] { + ensure!(matches!( + group.observe(timeout).await, + Err(RuntimeError::Invalid(_)) + )); + ensure!(matches!( + group.transfer_leadership(transfer, timeout).await, + Err(RuntimeError::Invalid(_)) + )); + } + let unpolled = group.transfer_leadership(transfer, Duration::from_millis(1)); + let unpolled_observation = group.observe(Duration::from_millis(1)); + tokio::time::sleep(Duration::from_millis(20)).await; + ensure!(matches!( + unpolled.await, + Err(RuntimeError::DeadlineExceeded) + )); + ensure!(matches!( + unpolled_observation.await, + Err(RuntimeError::DeadlineExceeded) + )); + for rejected in [ + LeadershipTransfer { + expected_term: Term::new(progress.term.get() + 1), + ..transfer + }, + LeadershipTransfer { + expected_revision: LogIndex::new(progress.membership_revision.get() + 1), + ..transfer + }, + LeadershipTransfer { + target: MemberId::new(4)?, + ..transfer + }, + ] { + ensure!(matches!( + group.transfer_leadership(rejected, TIMEOUT).await, + Err(RuntimeError::Invalid(_)) + )); + let observed = group.observe(TIMEOUT).await?; + ensure!(observed.status.leader == Some(initial as u64 + 1)); + ensure!(observed.status.term == progress.term); + ensure!(observed.membership_revision == progress.membership_revision); + } + tokio::time::timeout(TIMEOUT, async { + loop { + match group.transfer_leadership(transfer, TIMEOUT).await { + Ok(()) => break Ok::<_, anyhow::Error>(()), + Err(RuntimeError::NotReady) => { + tokio::time::sleep(Duration::from_millis(10)).await + } + Err(error) => break Err(error.into()), + } + } + }) + .await??; + tokio::time::timeout(TIMEOUT, async { + loop { + let observed = cluster.hosts[target] + .group(cluster.history)? + .observe(TIMEOUT) + .await?; + if observed.role == RaftReplicaRole::Leader && observed.status.ready { + break Ok::<_, anyhow::Error>(()); + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await??; + verify_cells(&cluster.hosts[target], &seed, 2).await?; + retained = Some((group.clone(), group.observe(TIMEOUT), transfer)); + Ok::<_, anyhow::Error>(()) + } + .await; + let stopped = close(std::mem::take(&mut cluster.nodes), &cluster.hosts).await; + if let Some((group, unpolled, transfer)) = retained { + ensure!(matches!(unpolled.await, Err(RuntimeError::Closed))); + ensure!(matches!( + group.observe(TIMEOUT).await, + Err(RuntimeError::Closed) + )); + ensure!(matches!(group.archive_status(), Err(RuntimeError::Closed))); + ensure!(matches!( + group.transfer_leadership(transfer, TIMEOUT).await, + Err(RuntimeError::Closed) + )); + } + cluster.fixtures[0].domain.shutdown().await; + stopped?; + work +} diff --git a/crates/cellar/tests/support/cell_group_managed_discovery.rs b/crates/cellar/tests/support/cell_group_managed_discovery.rs new file mode 100644 index 000000000..dc56ba6a5 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_managed_discovery.rs @@ -0,0 +1,109 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! One lost metadata reply must not consume a whole movement turn. +use super::*; + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn managed_movement_retries_one_lost_leader_observation() -> Result<()> { + for rooted in [false, true] { + Box::pin(lost_observation(rooted, Arc::new(fault::Store::default()))).await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[ignore = "requires explicit owned RustFS endpoint, bucket and isolated prefix"] +async fn rustfs_managed_movement_retries_one_lost_leader_observation() -> Result<()> { + for rooted in [false, true] { + Box::pin(lost_observation( + rooted, + Arc::new(fault::Store::around(rustfs::store( + "movement-observation", + rooted, + )?)), + )) + .await?; + } + Ok(()) +} + +async fn lost_observation(rooted: bool, store: Arc) -> Result<()> { + let mut cluster = Box::pin(ManagedCluster::start_with_catalog_clients( + rooted, + &[7], + ArchiveSetup::Initialized, + store, + true, + )) + .await?; + let work = async { + // The controller uses catalog client0. Member2's foreground observer + // owns a distinct client over the same backend, so its one lost read + // cannot block initial CAS checks, trust refresh or other peers. + maintenance::handoff_to(&cluster, 3, 1).await?; + let (_, source, _, _, execution) = Box::pin(reserve_replacement(&cluster, true)).await?; + ensure!(source == 1); + let catalog = &cluster.fixtures[0].catalog; + let snapshot = catalog.read(TIMEOUT).await?; + let lost = Arc::new(fault::RegistryGate::default()); + lost.armed.store(true, Ordering::Release); + let result = fault::gate_store_read( + cluster.fixtures[source].store.as_ref(), + "/fleet/head".into(), + lost.clone(), + catalog.reconcile_movement(&snapshot, &execution, &cluster.hosts[source], TIMEOUT), + ) + .await; + ensure!( + !lost.armed.load(Ordering::Acquire), + "leader observation fault was not entered" + ); + match result { + Ok(progress) => { + ensure!(progress.phase == FleetMovementPhase::Reserved && !progress.settled) + } + Err(FleetCatalogError::Unconfirmed) => (), + Err(error) => return Err(error.into()), + } + // Unknown action delivery is not rollback proof. A fresh quorum must + // show that this single reconciliation actually reached durable Begin; + // another mutation request is never issued to rescue the attempt. + let observed = cluster.hosts[source] + .group(cluster.history)? + .quorum_membership(TIMEOUT) + .await?; + ensure!( + observed + .membership() + .fleet_status() + .is_some_and(|s| s.phase == Some(FleetPermitPhase::Seeding)), + "one lost leader observation prevented movement Begin" + ); + ensure!( + catalog + .pending_reservations(&catalog.read(TIMEOUT).await?, TIMEOUT) + .await? + .len() + == 1, + "beginning a movement released its physical debit" + ); + ensure!( + cluster.hosts[3].resource_usage().host.controller.groups == 0, + "discovery initialized a learner without activation" + ); + Ok::<_, anyhow::Error>(()) + } + .await; + let joined = close(std::mem::take(&mut cluster.nodes), &cluster.hosts).await; + cluster.fixtures[0].domain.shutdown().await; + if work.is_err() || joined.is_err() { + for root in std::mem::take(&mut cluster.roots) { + eprintln!( + "retained failed movement observation disk: {}", + root.keep().display() + ); + } + } + work?; + joined +} diff --git a/crates/cellar/tests/support/cell_group_managed_first_group_operator.rs b/crates/cellar/tests/support/cell_group_managed_first_group_operator.rs new file mode 100644 index 000000000..a71931758 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_managed_first_group_operator.rs @@ -0,0 +1,61 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Managed operator startup refuses readiness after losing quorum. + +use super::*; + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn managed_first_group_cli_admin_and_cold_quorum() -> Result<()> { + let first = Fixture::with_class(false, counter_class()?).await?; + let mut cluster = Box::pin(ManagedCluster::start_with_applications( + first, + &[], + ArchiveSetup::Initialized, + true, + None, + FirstGroupSetup::Operator, + )) + .await?; + let work = async { + seed_cells(&cluster.hosts, cluster.history).await?; + let peers = cluster.nodes.split_off(1); + close_group_nodes(peers, &cluster.hosts[1..]).await?; + let survivor = &cluster.hosts[0]; + tokio::time::timeout(TIMEOUT, async { + let group = survivor.group(cluster.history)?; + loop { + let status = group.status().await?; + if !status.ready && status.leader.is_none() { + return Ok::<_, anyhow::Error>(()); + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .context("observe first group quorum loss")??; + let readiness = survivor + .start_managed_namespace_first_group( + cluster.history.cluster, + cluster.history.namespace, + TIMEOUT, + ) + .await; + ensure!( + matches!( + readiness, + Err(CellGroupStartupError::Unconfirmed( + CellGroupControllerError::Runtime(RuntimeError::DeadlineExceeded) + )) + ), + "retained attachment must refuse readiness without quorum: {readiness:?}" + ); + ensure!(survivor.resource_usage().host.controller.groups == 1); + Ok::<_, anyhow::Error>(()) + } + .await; + let joined = close_group_nodes(cluster.nodes, &cluster.hosts).await; + cluster.fixtures[0].domain.shutdown().await; + work?; + joined?; + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_managed_fixture.rs b/crates/cellar/tests/support/cell_group_managed_fixture.rs new file mode 100644 index 000000000..5d1a64ab5 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_managed_fixture.rs @@ -0,0 +1,43 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Shared managed fixtures for the lifecycle and public request targets. + +use anyhow::{ensure, Context as _, Result}; +use cellar::application::consensus::verification::cell_groups::RangeDirectory; +use cellar::bucket::{Bucket, StorageBackend}; +use cellar::sdk::{consensus::*, *}; +use std::{ + path::Path, + sync::{atomic::Ordering, Arc}, + time::Duration, +}; + +#[path = "cell_group_archive_verifier.rs"] +mod archive_verifier; +#[path = "cell_group_managed_fixture_archive.rs"] +pub mod archives; +#[path = "cell_group_managed_fixture_cluster.rs"] +mod cluster; +#[path = "raft_catalog_fault.rs"] +pub mod fault; +#[path = "html.rs"] +pub mod graph; +#[path = "cell_group_managed_fixture_maintenance.rs"] +pub mod maintenance; +#[path = "cell_group_managed_fixture_movement.rs"] +pub mod movement; +#[path = "cell_group_managed_fixture_node.rs"] +mod node; +#[path = "cell_group_managed_fixture_operator.rs"] +pub(super) mod operator; +#[path = "cell_group_managed_requests.rs"] +pub mod requests; +#[path = "cell_group_managed_fixture_seed_failure.rs"] +pub mod seed_failures; +#[path = "cell_group_managed_fixture_verifier.rs"] +pub mod verifier; + +pub use cluster::*; +pub use node::{limits, request, start_node, Fixture, NoListing}; + +pub const TIMEOUT: Duration = Duration::from_secs(5); diff --git a/crates/cellar/tests/support/cell_group_managed_fixture_archive.rs b/crates/cellar/tests/support/cell_group_managed_fixture_archive.rs new file mode 100644 index 000000000..e62cb73c0 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_managed_fixture_archive.rs @@ -0,0 +1,55 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Explicit archive initialization for the managed fixture. + +use super::*; + +pub fn configure(config: &mut RaftCellsConfig) { + config.shared.checkpoints = Some(checkpoint_limits()); + config.shared.archive = Some(CellGroupNodeArchiveOptions { + io: Default::default(), + uploads: Default::default(), + tails: Default::default(), + }); +} + +pub async fn open_provisioning( + fixture: &Fixture, + adapter: &Catalog, +) -> Result { + Ok(fixture + .catalog + .open_managed_archive_provisioning( + adapter, + &fixture.declaration, + DirectoryLimits::default(), + CellGroupArchiveLimits::default(), + TIMEOUT, + ) + .await?) +} + +pub async fn initialize( + fixture: &Fixture, + adapter: &Catalog, + history: GroupHistory, + disk: &Path, +) -> Result<()> { + let prepared = open_provisioning(fixture, adapter) + .await? + .prepare_archive(history, TIMEOUT) + .await?; + // Sync the exact operator intent before entering the one-use operation. + let intent = disk.join("archive-initialization"); + use std::io::Write as _; + let mut file = std::fs::File::create_new(&intent)?; + file.write_all(prepared.encoded_intent())?; + file.sync_all()?; + std::fs::File::open(disk)?.sync_all()?; + prepared.commit(fixture.domain.as_ref()).await?; + open_provisioning(fixture, adapter) + .await? + .verify_archive(history, &std::fs::read(intent)?, TIMEOUT) + .await?; + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_managed_fixture_cluster.rs b/crates/cellar/tests/support/cell_group_managed_fixture_cluster.rs new file mode 100644 index 000000000..cffc8ad68 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_managed_fixture_cluster.rs @@ -0,0 +1,676 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Managed RF3 bootstrap, capacity and selected group fixture. + +use super::*; + +pub fn counter_class() -> Result { + counter_class_with_callback(|| {}) +} + +pub fn counter_class_with_callback( + on_command: impl Fn() + Send + Sync + 'static, +) -> Result { + Ok(RaftClass::new("Counter", 1, [81; 32])? + .initializer([82; 32], |_, sql| { + sql.execute_batch( + "CREATE TABLE counter(value INTEGER NOT NULL); INSERT INTO counter VALUES(0)", + )?; + Ok(()) + })? + .command("Add", [83; 32], move |context, input| { + on_command(); + let delta = i64::from_be_bytes(input.try_into()?); + let current: i64 = context + .sql() + .query_row("SELECT value FROM counter", [], |r| r.get(0))?; + let value = current.checked_add(delta).context("counter overflow")?; + context + .sql() + .execute("UPDATE counter SET value=?1", [value])?; + Ok(value.to_be_bytes().to_vec()) + })? + .query("Get", [84; 32], |context, _| { + let value: i64 = context.query_row("SELECT value FROM counter", [], |r| r.get(0))?; + Ok(value.to_be_bytes().to_vec()) + })?) +} + +pub async fn peer_fixture(f: &Fixture, id: u8) -> Result { + let mut spec = f.spec.clone(); + spec.node = NodeId::new([id; 16])?; + spec.disk = DiskLineage::new([id + 10; 16])?; + spec.physical_host = IncarnationId::new([id + 90; 16])?; + spec.label = format!("shared-managed-{id}"); + let cut = f.catalog.read(TIMEOUT).await?; + f.catalog + .register_node(&cut, &request(id, cut.revision()), &spec, TIMEOUT) + .await?; + Ok(Fixture { + domain: f.domain.clone(), + store: Arc::new(fault::Store::around(f.store.clone())), + catalog: f.catalog.clone(), + declaration: f.declaration.clone(), + application: f.application.clone(), + spec, + }) +} + +pub async fn inspect_current(host: &RaftCells, node: NodeId, revision: u64) -> Result<()> { + // The directory's actual timer must install the new rooted trust/process + // view. Retrying this read does not mint capacity or alter the fleet cut. + tokio::time::timeout(TIMEOUT, async { + loop { + match host.inspect_peer_node(node, revision, TIMEOUT).await { + Ok(view) => { + ensure!(view + .view + .fleet_trust_revision + .is_some_and(|r| r >= revision)); + return Ok::<_, anyhow::Error>(()); + } + Err(_) => tokio::time::sleep(Duration::from_millis(10)).await, + } + } + }) + .await??; + Ok(()) +} + +pub async fn admit_capacity( + f: &Fixture, + host: &RaftCells, + directory: &RangeDirectory, + id: u8, +) -> Result<()> { + let local = host.observe_node(TIMEOUT).await?; + let evidence = FleetCapacityAttestation { + boot: local.fleet_boot.context("capacity boot")?, + profile_revision: 1, + profile_digest: fleet_capacity_profile_digest(&local.node.profile)?, + cpu_millis: 1000, + memory_bytes: 3 * 1024 * 1024 * 1024, + durable_bytes: local.node.profile.byte_capacity.max(1024 * 1024 * 1024), + engine_slots: 2, + memory_swap_disabled: true, + used_memory_bytes: 64 * 1024 * 1024, + used_durable_bytes: 1024 * 1024, + evidence_digest: [71; 32], + evidence_reference: "fixture-only/deployment-attestation".into(), + }; + for (offset, action, state) in [ + (0, FleetCapacityAction::Qualify, FleetNodeState::Ready), + (1, FleetCapacityAction::Activate, FleetNodeState::Active), + ] { + let cut = f.catalog.read(TIMEOUT).await?; + inspect_current(host, f.spec.node, cut.revision()).await?; + let accepted = f + .catalog + .admit_node_capacity( + &cut, + &FleetCapacityRequest { + identity: request(id + offset, cut.revision()), + attestation: evidence.clone(), + action, + }, + host, + std::slice::from_ref(directory), + TIMEOUT, + ) + .await?; + ensure!(accepted.state() == state); + let current = f.catalog.read(TIMEOUT).await?; + let capacity = f + .catalog + .authorize_capacity(¤t, f.spec.node, TIMEOUT) + .await?; + ensure!(capacity.receive_replicas() == (state == FleetNodeState::Active)); + } + Ok(()) +} + +pub async fn leader(hosts: &[RaftCells], history: GroupHistory) -> Result { + Ok(tokio::time::timeout(TIMEOUT, async { + loop { + for (i, host) in hosts.iter().enumerate() { + let status = host.group(history)?.status().await?; + if status.ready && status.leader == Some(i as u64 + 1) { + return Ok::<_, anyhow::Error>(i); + } + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await??) +} + +pub async fn close_group_nodes(nodes: Vec, hosts: &[RaftCells]) -> Result<()> { + for (node, host) in nodes.into_iter().zip(hosts) { + ensure!(matches!( + node.shutdown(ShutdownMode::Preserve).await?, + NodeOutcome::Stopped(_) + )); + let report = host.close().await?; + ensure!( + report.groups_total == 1 + && report.mutation_outcomes == MutationOutcomes::NoUnresolvedLocalProducers + ); + } + Ok(()) +} + +pub fn checkpoint_limits() -> CellGroupCheckpointLimits { + CellGroupCheckpointLimits { + max_sources: 2, + max_retained_bytes: 32 * 1024 * 1024, + max_cells: 32, + max_transfer_bytes: 32 * 1024 * 1024, + max_chunk_bytes: 4093, + } +} + +#[derive(Clone, Copy, PartialEq, Eq)] +pub enum ArchiveSetup { + Disabled, + Initialized, + Missing, +} + +pub enum FirstGroupSetup { + Normal, + ExerciseReplay, + Operator, +} + +pub struct ManagedCluster { + pub fixtures: Vec, + pub roots: Vec, + pub nodes: Vec, + pub hosts: Vec, + pub peers: Vec, + pub directory: RangeDirectory, + pub history: GroupHistory, +} + +impl ManagedCluster { + pub async fn start(rooted: bool, spares: &[u8]) -> Result { + Box::pin(Self::start_with_archives( + rooted, + spares, + ArchiveSetup::Initialized, + )) + .await + } + + pub async fn start_with_archives( + rooted: bool, + spares: &[u8], + archives: ArchiveSetup, + ) -> Result { + Box::pin(Self::start_with_store( + rooted, + spares, + archives, + Arc::new(fault::Store::default()), + )) + .await + } + + pub async fn start_with_store( + rooted: bool, + spares: &[u8], + archives: ArchiveSetup, + store: Arc, + ) -> Result { + Box::pin(Self::start_with_catalog_clients( + rooted, spares, archives, store, false, + )) + .await + } + + pub async fn start_with_catalog_clients( + rooted: bool, + spares: &[u8], + archives: ArchiveSetup, + store: Arc, + independent: bool, + ) -> Result { + let first = Fixture::with_store(rooted, counter_class()?, store).await?; + Box::pin(Self::start_with_applications( + first, + spares, + archives, + independent, + None, + FirstGroupSetup::Normal, + )) + .await + } + + pub async fn start_with_applications( + first: Fixture, + spares: &[u8], + archives: ArchiveSetup, + independent: bool, + applications: Option>>, + first_group: FirstGroupSetup, + ) -> Result { + let second = peer_fixture(&first, 5).await?; + let third = peer_fixture(&first, 6).await?; + let mut fixtures = vec![first, second, third]; + for id in spares { + fixtures.push(peer_fixture(&fixtures[0], *id).await?); + } + if let Some(applications) = applications { + ensure!(applications.len() == fixtures.len()); + for (fixture, application) in fixtures.iter_mut().zip(applications) { + fixture.application = application; + } + } + if independent { + for fixture in fixtures.iter_mut().skip(1) { + let bucket = Bucket::from_clients( + fixture.store.clone(), + fixture.store.clone(), + Arc::new(NoListing), + StorageBackend::S3, + "managed-component".into(), + "tenant/".into(), + )?; + fixture.catalog = FleetCatalog::new( + &fixture.domain, + bucket, + fixture.declaration.namespace.cluster, + IncarnationId::new([80; 16])?, + "hosts/", + )?; + } + } + let f = &fixtures[0]; + let adapter = Catalog::new( + f.store.clone(), + CatalogOptions { + prefix: "managed/".into(), + ..Default::default() + }, + )?; + let namespace = f + .domain + .run(adapter.existing_namespace(f.declaration.namespace.clone())) + .await??; + let directory = RangeDirectory::new(namespace, DirectoryLimits::default())?; + let standby = f + .catalog + .observe_managed_standby( + &adapter, + &f.declaration, + DirectoryLimits::default(), + TIMEOUT, + ) + .await?; + let observed = f.domain.run(directory.observe()).await??; + ensure!(observed.revision() == 1 && observed.groups() == 0 && observed.digest() == standby); + let history = GroupHistory { + cluster: f.declaration.namespace.cluster, + namespace: f.declaration.namespace.namespace, + group: GroupId::new([3; 16])?, + incarnation: IncarnationId::new([7; 16])?, + }; + let proposal = || -> Result { + Ok(CellGroupBirthSpec { + history, + range: CellRange::all(), + epoch: RouteEpoch::new(1)?, + creation: ProposalId::new([150; 16])?, + voters: fixtures + .iter() + .take(3) + .enumerate() + .map(|(i, f)| { + Ok(InitialMember { + member: MemberId::new(i as u64 + 1)?, + node: f.spec.node, + }) + }) + .collect::>()?, + learners: vec![], + }) + }; + let roots = (0..fixtures.len()) + .map(|_| tempfile::tempdir()) + .collect::>>()?; + let mut nodes = Vec::new(); + let mut hosts = Vec::new(); + let mut peers = Vec::new(); + for (i, fixture) in fixtures.iter().enumerate() { + let node = start_node( + roots[i].path(), + fixture.application.clone(), + &fixture.spec.label, + ) + .await?; + let (mut config, services) = fixture + .config(&node, LogOpenMode::Create, 24 + i as u8) + .await?; + if !spares.is_empty() || archives != ArchiveSetup::Disabled { + config.shared.checkpoints = Some(checkpoint_limits()); + } + if archives != ArchiveSetup::Disabled { + archives::configure(&mut config); + } + peers.push(config.peers[0].clone()); + let host = node.start_raft_cells(config, services).await?; + ensure!(host.observe_node(TIMEOUT).await?.node.groups == 0); + nodes.push(node); + hosts.push(host); + } + let reserve = |reservation| -> Result { + Ok(RaftManagedFirstGroupRequest { + reservation, + standby, + publication: ProposalId::new([121; 16])?, + group: proposal()?, + }) + }; + let cut = f.catalog.read(TIMEOUT).await?; + ensure!(matches!( + f.catalog + .reserve_managed_first_group( + &adapter, + &f.declaration, + reserve(request(59, cut.revision()))?, + DirectoryLimits::default(), + TIMEOUT, + ) + .await, + Err(FleetCatalogError::Conflict) + )); + ensure!(f.catalog.read(TIMEOUT).await?.revision() == cut.revision()); + for (i, fixture) in fixtures.iter().enumerate() { + admit_capacity(fixture, &hosts[i], &directory, 40 + 2 * i as u8).await?; + } + let cut = f.catalog.read(TIMEOUT).await?; + let reserve_identity = request(60, cut.revision()); + let public_operator = matches!(first_group, FirstGroupSetup::Operator); + let authorization = if public_operator { + Box::pin(operator::provision( + f, + &directory, + &hosts[0], + reserve(reserve_identity)?, + )) + .await? + } else { + let reserved = f + .catalog + .reserve_managed_first_group( + &adapter, + &f.declaration, + reserve(reserve_identity.clone())?, + DirectoryLimits::default(), + TIMEOUT, + ) + .await?; + let cut = f.catalog.read(TIMEOUT).await?; + ensure!( + f.catalog + .check_inventory(&cut, std::slice::from_ref(&directory), TIMEOUT) + .await? + .histories() + == 1, + "unselected reservation must retain its physical references" + ); + let confirmation = request(61, cut.revision()); + let reserved = if matches!(first_group, FirstGroupSetup::ExerciseReplay) { + verify_first_group_replay( + f, + &adapter, + &directory, + &reserved, + &confirmation, + reserve(reserve_identity)?, + history, + ) + .await? + } else { + reserved + }; + f.catalog + .publish_managed_first_group(&reserved, &confirmation, TIMEOUT) + .await? + }; + if archives == ArchiveSetup::Initialized && !public_operator { + archives::initialize(f, &adapter, history, roots[0].path()).await?; + } + let cut = f.catalog.read(TIMEOUT).await?; + for host in hosts.iter().take(3) { + for fixture in &fixtures { + inspect_current(host, fixture.spec.node, cut.revision()).await?; + } + if public_operator { + continue; + } + let result = host + .activate_managed_initial_group(authorization.clone(), TIMEOUT) + .await; + if archives == ArchiveSetup::Missing { + ensure!( + matches!( + result, + Err(CellGroupStartupError::Unconfirmed( + CellGroupControllerError::Archive(_) + )) + ), + "missing archive must refuse before bootstrap: {result:?}" + ); + ensure!(host.observe_node(TIMEOUT).await?.node.groups == 0); + ensure!(f.catalog.read(TIMEOUT).await?.revision() == cut.revision()); + } else { + result?; + } + } + if public_operator { + let selected = futures_util::future::try_join_all(hosts.iter().take(3).map(|voter| { + voter.start_managed_namespace_first_group( + history.cluster, + history.namespace, + TIMEOUT, + ) + })) + .await?; + ensure!( + selected.iter().all(|selected| *selected == history), + "unexpected managed namespace group" + ); + } + Ok(Self { + fixtures, + roots, + nodes, + hosts, + peers, + directory, + history, + }) + } +} + +pub struct SeededCells { + pub cells: Vec, + pub receipts: Vec, + pub retry: u64, +} + +async fn verify_first_group_replay( + f: &Fixture, + adapter: &Catalog, + directory: &RangeDirectory, + prepared: &RaftManagedFirstGroup, + confirmation: &FleetRequestIdentity, + request_plan: RaftManagedFirstGroupRequest, + history: GroupHistory, +) -> Result { + let cut = f.catalog.read(TIMEOUT).await?; + let reservation = f + .catalog + .recover_birth_reservation(&cut, directory, &history, TIMEOUT) + .await?; + let mut changed = request_plan.clone(); + changed.publication = ProposalId::new([125; 16])?; + ensure!(matches!( + f.catalog + .reopen_managed_first_group( + adapter, + &f.declaration, + changed.clone(), + DirectoryLimits::default(), + TIMEOUT + ) + .await, + Err(FleetCatalogError::PayloadConflict) + )); + ensure!(matches!( + f.catalog + .reserve_managed_first_group( + adapter, + &f.declaration, + changed, + DirectoryLimits::default(), + TIMEOUT + ) + .await, + Err(FleetCatalogError::PayloadConflict) + )); + ensure!(f.catalog.read(TIMEOUT).await?.revision() == cut.revision()); + let gate = &f.store.first_group_birth_after; + gate.armed.store(true, Ordering::Release); + let conflicting = request(62, cut.revision() + 1); + let (publication, cancellation) = tokio::time::timeout(TIMEOUT, async { + tokio::join!( + f.catalog.publish_managed_first_group(prepared, &conflicting, TIMEOUT), + async { + gate.entered.notified().await; + let result = async { + let view = f.domain.run(directory.observe()).await??; + ensure!(view.groups() == 0 && view.digest() == request_plan.standby); + let winner = f.domain.run(directory.cancel_fleet_group_birth( + &reservation, ProposalId::new([124; 16])?, + )).await??; + ensure!(matches!(winner, cellar::application::consensus::verification::cell_groups::CellGroupBirthDisposition::Live(birth) if birth.route() == reservation.route())); + Ok::<_, anyhow::Error>(()) + }.await; + gate.resume.notify_one(); + result + } + ) + }).await?; + cancellation?; + ensure!(matches!(publication, Err(FleetCatalogError::Conflict))); + ensure!(f.domain.run(directory.observe()).await??.groups() == 1); + ensure!(f.catalog.read(TIMEOUT).await?.revision() == cut.revision()); + ensure!(matches!( + f.catalog.authorize_group(&cut, &history, TIMEOUT).await, + Err(FleetCatalogError::Conflict) + )); + f.store + .fail_fleet_confirmation + .store(true, Ordering::Release); + ensure!(matches!( + f.catalog + .publish_managed_first_group(prepared, confirmation, TIMEOUT) + .await, + Err(FleetCatalogError::Storage(_)) + )); + ensure!(!f.store.fail_fleet_confirmation.load(Ordering::Acquire)); + ensure!(f.catalog.read(TIMEOUT).await?.revision() == cut.revision() + 1); + // A confirmed range with a lost immutable birth must refuse rather than + // reconstructing it from the operator's retained setup. + use object_store::{path::Path as ObjectPath, ObjectStore}; + let key = ObjectPath::from(format!( + "managed/namespaces/{}/cell-groups/births/{}/{}", + hex::encode(history.namespace.bytes()), + hex::encode(history.group.bytes()), + hex::encode(history.incarnation.bytes()), + )); + let saved = f.store.inner.get(&key).await?.bytes().await?; + f.store.inner.delete(&key).await?; + let replay = f + .catalog + .publish_managed_first_group(prepared, confirmation, TIMEOUT) + .await; + let missing = matches!( + f.store.inner.get(&key).await, + Err(object_store::Error::NotFound { .. }) + ); + f.store.inner.put(&key, saved.into()).await?; + ensure!(matches!(replay, Err(FleetCatalogError::Corrupt(_))) && missing); + ensure!(f.catalog.read(TIMEOUT).await?.revision() == cut.revision() + 1); + Ok(f.catalog + .reopen_managed_first_group( + adapter, + &f.declaration, + request_plan, + DirectoryLimits::default(), + TIMEOUT, + ) + .await?) +} + +pub async fn seed_cells(hosts: &[RaftCells], history: GroupHistory) -> Result { + let index = leader(&hosts[..3], history).await?; + let retry = u64::try_from( + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH)? + .as_millis(), + )? + .checked_add(60_000) + .context("retry deadline")?; + let mut cells = Vec::new(); + let mut receipts = Vec::new(); + for (i, delta) in [7i64, 11].into_iter().enumerate() { + let cell = CellHistory { + cluster: history.cluster, + namespace: history.namespace, + cell: CellId::new(format!("Counter:cold-{i}"))?, + incarnation: IncarnationId::new([91; 16])?, + }; + let handle = hosts[index].cell(cell.clone())?; + handle + .enroll( + CellEnrollmentRequest { + creation: ProposalId::new([180 + i as u8; 16])?, + durability: Durability::Quorum, + }, + TIMEOUT, + ) + .await?; + let id = CommandId::new(ClientId::new([200; 16])?, 1)?; + let receipt = handle + .command( + &handle.contract("Add")?, + delta.to_be_bytes().to_vec(), + CommandOptions::new(id, retry), + ) + .await?; + ensure!(receipt.result == delta.to_be_bytes()); + cells.push(cell); + receipts.push(receipt); + } + tokio::time::timeout(TIMEOUT, async { + loop { + if hosts.iter().take(3).all(|host| { + host.resource_usage().host.controller.groups == 1 + && host.resource_usage().retained_manifest_cells == Some(2) + }) { + break; + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await?; + Ok(SeededCells { + cells, + receipts, + retry, + }) +} diff --git a/crates/cellar/tests/support/cell_group_managed_fixture_maintenance.rs b/crates/cellar/tests/support/cell_group_managed_fixture_maintenance.rs new file mode 100644 index 000000000..e1ddd7b61 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_managed_fixture_maintenance.rs @@ -0,0 +1,804 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Rooted stop, return and resume after a whole-group repair. + +use super::{movement::*, *}; +use cellar::application::consensus::verification::{ + FleetRestartGateRequest, FleetRestartHistoryProgress, FleetRestartOfflineRequest, + FleetRestartReadyRequest, FleetRestartRequest, FleetRestartResumeCompleteRequest, + FleetRestartResumePhase, FleetRestartResumeProgress, FleetRestartResumeRequest, + FleetRestartReturnRequest, FleetRestartRuntimeProgress, FleetRestartStopRequest, +}; + +fn intent(id: u8, revision: u64) -> FleetRequestIdentity { + let mut identity = request(id, revision); + identity.operation = [230; 16]; + identity +} + +pub async fn maintained_with_loss( + rooted: bool, + joined: bool, + store: Arc, + lose_observation: bool, +) -> Result<()> { + let cluster = Box::pin(ManagedCluster::start_with_catalog_clients( + rooted, + &[7], + ArchiveSetup::Initialized, + store, + lose_observation, + )) + .await?; + Box::pin(exercise(cluster, joined, lose_observation)).await +} + +pub async fn run_public(store: impl Fn(bool) -> Result>) -> Result<()> { + for rooted in [false, true] { + let first = Fixture::with_store(rooted, counter_class()?, store(rooted)?).await?; + let cluster = Box::pin(ManagedCluster::start_with_applications( + first, + &[7], + ArchiveSetup::Initialized, + true, + None, + FirstGroupSetup::Operator, + )) + .await?; + Box::pin(exercise(cluster, true, false)).await?; + } + Ok(()) +} + +async fn exercise(mut cluster: ManagedCluster, joined: bool, lose_observation: bool) -> Result<()> { + let work = async { + let (seed, obsolete) = replace(&cluster) + .await + .context("maintenance joined member repair")?; + let target = if joined { + 3 + } else { + leader(&cluster.hosts[..3], cluster.history).await? + }; + cycle(&mut cluster, &seed, obsolete, target, lose_observation).await + } + .await; + if work.is_err() { + // Preserve actual local status before cleanup joins the hosts. These + // bounded read-only samples never replace or clear the original error. + for (index, host) in cluster.hosts.iter().enumerate() { + let usage = host.resource_usage(); + eprintln!( + "maintenance failure: host {index} control={:?} runtime={:?} host={:?}", + usage.control, usage.runtime, usage.host + ); + if let Ok(group) = host.group(cluster.history) { + let sample = tokio::time::timeout(TIMEOUT, group.status()).await; + eprintln!("maintenance failure: host {index} status={sample:?}"); + match group.archive_status() { + Ok(Some(archive)) => eprintln!( + "maintenance failure: host {index} archive attempts={} published={:?} stopped={} error={:?}", + archive.attempts, + archive.published.as_ref().map(|manifest| manifest.position()), + archive.stopped, + archive.error, + ), + Ok(None) => eprintln!("maintenance failure: host {index} has no archive owner"), + Err(error) => eprintln!("maintenance failure: host {index} archive status={error:?}"), + } + } + } + } + let joined = close(std::mem::take(&mut cluster.nodes), &cluster.hosts).await; + cluster.fixtures[0].domain.shutdown().await; + if work.is_err() || joined.is_err() { + for root in std::mem::take(&mut cluster.roots) { + eprintln!( + "retained failed managed maintenance disk: {}", + root.keep().display() + ); + } + } + joined?; + work +} + +async fn current_survivor( + cluster: &ManagedCluster, + obsolete: usize, + target: usize, +) -> Result { + let originals = (0..4) + .filter(|i| *i != obsolete && *i != target) + .collect::>(); + let hosts = originals + .iter() + .map(|i| cluster.hosts[*i - usize::from(*i > target)].clone()) + .collect::>(); + let members = originals.iter().map(|i| *i as u64 + 1).collect::>(); + let index = current_leader(&hosts, &members, cluster.history).await?; + Ok(hosts[index].clone()) +} + +async fn target_leader(cluster: &ManagedCluster, target: usize) -> Result<()> { + tokio::time::timeout(TIMEOUT, async { + loop { + let status = cluster.hosts[target] + .group(cluster.history)? + .status() + .await?; + if status.ready && status.leader == Some(target as u64 + 1) { + return Ok::<_, anyhow::Error>(()); + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .context("observe actual maintenance target leadership")??; + Ok(()) +} + +pub async fn handoff_to(cluster: &ManagedCluster, obsolete: usize, target: usize) -> Result<()> { + let member = MemberId::new(target as u64 + 1)?; + let end = tokio::time::Instant::now() + TIMEOUT; + tokio::time::timeout(TIMEOUT, async { + loop { + let target_status = cluster.hosts[target] + .group(cluster.history)? + .status() + .await?; + if target_status.ready && target_status.leader == Some(member.get()) { + return Ok::<_, anyhow::Error>(()); + } + // A previous survivor sample is historical. Re-observe the actual + // leader and current term/revision before each guarded admission. + for (index, host) in cluster.hosts.iter().enumerate() { + if index == obsolete || index == target { + continue; + } + let remaining = end + .checked_duration_since(tokio::time::Instant::now()) + .context("maintenance handoff deadline")?; + let group = host.group(cluster.history)?; + let progress = match group.replication_status(remaining).await { + Ok(progress) => progress, + Err( + RuntimeError::NotLeader | RuntimeError::NotReady | RuntimeError::Overloaded, + ) => continue, + Err(error) => return Err(error.into()), + }; + let remaining = end + .checked_duration_since(tokio::time::Instant::now()) + .context("maintenance handoff deadline")?; + match group + .transfer_leadership( + LeadershipTransfer { + target: member, + expected_term: progress.term, + expected_revision: progress.membership_revision, + }, + remaining, + ) + .await + { + Ok(()) + | Err( + RuntimeError::NotReady + | RuntimeError::NotLeader + | RuntimeError::Overloaded + | RuntimeError::DeadlineExceeded, + ) => (), + Err(error) => return Err(error.into()), + } + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .context("observe actual maintenance target leadership")??; + Ok(()) +} + +async fn paused_write_refuses(host: &RaftCells, seed: &SeededCells) -> Result<()> { + let cell = host.cell(seed.cells[0].clone())?; + let id = CommandId::new(ClientId::new([202; 16])?, 1)?; + let result = cell + .command( + &cell.contract("Add")?, + 9i64.to_be_bytes().to_vec(), + CommandOptions::new(id, seed.retry), + ) + .await; + let Err(error) = result else { + anyhow::bail!("paused maintenance owner admitted a business write"); + }; + ensure!( + !error.ambiguous && error.cause.refusal() == Some(&CommandFailure::Unavailable), + "paused write was not a definite admission refusal: {error:?}" + ); + Ok(()) +} + +async fn cycle( + cluster: &mut ManagedCluster, + seed: &SeededCells, + obsolete: usize, + target_index: usize, + lose_observation: bool, +) -> Result<()> { + let member = MemberId::new(target_index as u64 + 1)?; + let catalog = cluster.fixtures[0].catalog.clone(); + discover(cluster).await?; + handoff_to(cluster, obsolete, target_index).await?; + eprintln!( + "maintenance: target member {} leader established", + member.get() + ); + let target = cluster.hosts[target_index].observe_node(TIMEOUT).await?; + let boot = target.fleet_boot.context("maintenance target boot")?; + let node = NodeId::new(boot.node)?; + let process = ProcessGeneration::new(boot.process)?; + let observer_index = (0..4) + .find(|i| *i != obsolete && *i != target_index && (!lose_observation || *i > 0)) + .context("surviving observer")?; + let observer = cluster.hosts[observer_index].clone(); + let cut = catalog.read(TIMEOUT).await?; + let begin = FleetRestartRequest { + identity: intent(220, cut.revision()), + node, + expected_process: process, + grace_seconds: 120, + }; + catalog + .begin_restart_maintenance( + &cut, + &begin, + std::slice::from_ref(&cluster.directory), + TIMEOUT, + ) + .await + .context("begin joined maintenance")?; + let cut = catalog.read(TIMEOUT).await?; + ensure!( + catalog.authorize_host(&cut, node, TIMEOUT).await?.state() + == FleetNodeState::RestartPending + ); + ensure!(catalog + .authorize_capacity(&cut, node, TIMEOUT) + .await + .is_err()); + let gate = FleetRestartGateRequest { + identity: intent(221, cut.revision()), + node, + expected_process: process, + gate: ProposalId::new([222; 16])?, + }; + catalog + .admit_restart_gate(&cut, &gate, TIMEOUT) + .await + .context("root joined maintenance gate")?; + discover(cluster).await?; + let cut = catalog.read(TIMEOUT).await?; + let mut walk = catalog + .prepare_restart_verification( + &cut, + ProposalId::new(begin.identity.operation)?, + std::slice::from_ref(&cluster.directory), + &observer, + TIMEOUT, + ) + .await + .context("prepare joined maintenance verification")?; + let authority = catalog + .authorize_group(&cut, &cluster.history, TIMEOUT) + .await?; + for (i, fixture) in cluster + .fixtures + .iter() + .enumerate() + .filter(|(i, _)| *i != obsolete) + { + let sample = observer + .observe_peer_fleet_membership( + fixture.spec.node, + authority.clone(), + cut.revision(), + TIMEOUT, + ) + .await?; + eprintln!( + "maintenance: voter {} actual node leader load {:?}", + i + 1, + sample.progress().leader_load + ); + let load = sample + .progress() + .leader_load + .as_ref() + .context("missing complete shared leader load")?; + ensure!(load.capacity == 2 && load.leaders == u32::from(i == target_index)); + } + eprintln!("maintenance: complete target inventory, start survivor verification"); + let mut handoffs = 0; + let mut verified = 0; + let mut waiting = false; + let lost = Arc::new(fault::RegistryGate::default()); + let mut armed = false; + let mut refused = false; + let renew = async { + tokio::time::timeout(TIMEOUT, async { + loop { + match walk.reconcile_next(TIMEOUT).await { + Ok(Some(FleetRestartRuntimeProgress::HandoffRequested { + group, + from, + to, + .. + })) => { + ensure!(group == cluster.history && from == member && to != from); + handoffs += 1; + eprintln!("maintenance: survivor handoff requested"); + if lose_observation && !armed { + // The next foreground local-member observation loses + // its catalog reply. It is never resumed by the test. + lost.armed.store(true, Ordering::Release); + armed = true; + } + } + Ok(Some(FleetRestartRuntimeProgress::Verified( + FleetRestartHistoryProgress::Voting { + group, + member, + survivors, + }, + ))) => { + ensure!( + group == cluster.history + && member == MemberId::new(target_index as u64 + 1)? + && survivors[0] != survivors[1] + && !survivors.contains(&node) + ); + verified += 1; + } + Ok(None) => break, + Err( + FleetCatalogError::Unconfirmed + | FleetCatalogError::Overloaded + | FleetCatalogError::InventoryIncomplete, + ) + | Err(FleetCatalogError::Observation( + InspectionError::Unavailable | InspectionError::Overloaded, + )) => { + refused = true; + if !waiting { + eprintln!("maintenance: survivor proof not yet available"); + waiting = true; + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + other => anyhow::bail!("joined maintenance verification: {other:?}"), + } + } + Ok::<_, anyhow::Error>(()) + }) + .await + .context("renew joined maintenance survivor proof")??; + Ok::<_, anyhow::Error>(()) + }; + if lose_observation { + fault::gate_store_read( + cluster.fixtures[observer_index].store.as_ref(), + "/fleet/head".into(), + lost.clone(), + renew, + ) + .await?; + ensure!( + armed && refused && !lost.armed.load(Ordering::Acquire), + "lost member observation was not entered and retried" + ); + } else { + renew.await?; + } + ensure!(handoffs == 1); + if lose_observation { + // This regression ends at its owning renewal boundary. Full + // stop/return/resume remains required by the original lifecycle case. + ensure!(verified == 1); + return Ok(()); + } + let cut = catalog.read(TIMEOUT).await?; + let ready = FleetRestartReadyRequest { + identity: intent(223, cut.revision()), + node, + expected_process: process, + }; + eprintln!("maintenance: survivors verified; publish readiness"); + let readiness = walk + .publish_ready(&ready, TIMEOUT) + .await + .context("publish joined readiness")?; + ensure!( + readiness.candidate().histories() == 1 + && readiness.candidate().physical_histories() == 1 + && readiness.candidate().voting_histories() == 1 + && readiness.candidate().matched_histories() == 1 + ); + let cut = catalog.read(TIMEOUT).await?; + paused_write_refuses(&cluster.hosts[target_index], seed) + .await + .context("quiesced business admission")?; + // Preserve the nested-RPC regression: this stop originates at the actual + // survivor leader. Its fresh matched proof inspects the stopping node while + // the stopping node owns the stop-renewal action back to that leader. + target_leader(cluster, observer_index).await?; + let stop = FleetRestartStopRequest { + identity: intent(224, cut.revision()), + ready: ready.identity, + node, + expected_process: process, + stop: ProposalId::new([225; 16])?, + }; + let stop_admission = catalog.admit_restart_stop(&cut, &stop, TIMEOUT).await?; + let cut = catalog.read(TIMEOUT).await?; + let premature = FleetRestartOfflineRequest { + identity: intent(234, cut.revision()), + stop: stop.clone(), + }; + ensure!(matches!( + catalog + .confirm_restart_offline(&cut, &premature, TIMEOUT) + .await, + Err(FleetCatalogError::Unconfirmed) + )); + ensure!(catalog.read(TIMEOUT).await?.revision() == cut.revision()); + discover(cluster).await?; + let cut = catalog.read(TIMEOUT).await?; + // Overload before submission may be retried. Once proof work is queued, + // Unconfirmed is reconciled by actual join and signed closure below, not + // by flooding the same bounded metadata bank with more stop submissions. + let result = tokio::time::timeout(TIMEOUT, async { + loop { + match catalog + .stop_restart_maintenance(&cut, &stop, &observer, TIMEOUT) + .await + { + Err(FleetCatalogError::Overloaded) => { + tokio::time::sleep(Duration::from_millis(10)).await; + } + result => break result, + } + } + }) + .await + .context("submit exact owned stop within original deadline")?; + ensure!( + result.is_ok() + || matches!( + result, + Err(FleetCatalogError::Unconfirmed | FleetCatalogError::Closed) + ), + "owned maintenance stop refused: {:?}", + result.err() + ); + eprintln!("maintenance: owned stop submitted; join target"); + // Leave the node in the cluster's cleanup owner until its join succeeds. + let outcome = tokio::time::timeout(TIMEOUT, cluster.nodes[target_index].finished()) + .await + .context("join actual stopped maintenance node")??; + let _old_node = cluster.nodes.remove(target_index); + let old_host = cluster.hosts.remove(target_index); + eprintln!("maintenance: target closed and joined"); + let NodeOutcome::Stopped(report) = outcome else { + anyhow::bail!("joined owner did not stop: {outcome:?}"); + }; + let closed = report.raft.context("joined maintenance stop report")?; + ensure!( + !closed.startup_incomplete + && closed.result?.mutation_outcomes == MutationOutcomes::NoUnresolvedLocalProducers + ); + ensure!(old_host.close().await?.groups_total == 1); + let closure = catalog.restart_stop_closure(&stop, TIMEOUT).await?; + ensure!( + closure.node() == boot.node + && closure.process() == boot.process + && closure.operation() == begin.identity.operation + && closure.request() == stop.stop.bytes() + ); + let cut = catalog.read(TIMEOUT).await?; + let offline = FleetRestartOfflineRequest { + identity: intent(226, cut.revision()), + stop: stop.clone(), + }; + catalog + .confirm_restart_offline(&cut, &offline, TIMEOUT) + .await + .context("confirm joined owner offline")?; + eprintln!("maintenance: offline root confirmed"); + let survivor = current_survivor(cluster, obsolete, target_index) + .await + .context("find actual post-stop survivor leader")?; + verify_cells(&survivor, seed, 3) + .await + .context("survivor SQL while target offline")?; + eprintln!("maintenance: survivor writes while target offline"); + let cut = catalog.read(TIMEOUT).await?; + let returning = FleetRestartReturnRequest { + identity: intent(227, cut.revision()), + offline: offline.identity, + node, + expected_process: process, + next_process: ProcessGeneration::new([54; 16])?, + }; + catalog + .admit_restart_return(&cut, &returning, TIMEOUT) + .await + .context("admit joined return")?; + let f = &cluster.fixtures[target_index]; + let resumed_node = start_node( + cluster.roots[target_index].path(), + f.application.clone(), + &f.spec.label, + ) + .await?; + let (mut config, services) = f.config(&resumed_node, LogOpenMode::Resume, 54).await?; + archives::configure(&mut config); + config.groups.push(RaftGroupAssignment { + history: cluster.history, + placement: RaftGroupPlacement::Replica, + }); + let resumed_host = match resumed_node.start_raft_cells(config, services).await { + Ok(host) => host, + Err(error) => { + let joined = resumed_node.shutdown(ShutdownMode::Preserve).await; + return Err(error).context(format!( + "canonical retained maintenance return; startup cleanup={joined:?}" + )); + } + }; + cluster.nodes.insert(target_index, resumed_node); + cluster.hosts.insert(target_index, resumed_host); + eprintln!("maintenance: retained return attached; discover new boot"); + discover(cluster) + .await + .context("discover returned paused node and current peers")?; + eprintln!("maintenance: returned paused node discovered"); + let view = cluster.hosts[target_index].observe_node(TIMEOUT).await?; + ensure!( + view.node.profile.draining + && view.node.storage_instance.bytes() == boot.engine + && view.node.groups == 1 + && cluster.hosts[target_index] + .resource_usage() + .retained_manifest_cells + == Some(2) + ); + let mut planned_profile = view.node.profile.clone(); + planned_profile.draining = false; + let attestation = FleetCapacityAttestation { + boot: view.fleet_boot.context("returned joined boot")?, + profile_revision: 1, + profile_digest: fleet_capacity_profile_digest(&planned_profile)?, + cpu_millis: 1000, + memory_bytes: 3 * 1024 * 1024 * 1024, + durable_bytes: planned_profile.byte_capacity.max(1024 * 1024 * 1024), + engine_slots: 2, + memory_swap_disabled: true, + used_memory_bytes: 64 * 1024 * 1024, + used_durable_bytes: 1024 * 1024, + evidence_digest: [71; 32], + evidence_reference: "fixture-only/deployment-attestation".into(), + }; + ensure!( + attestation.boot.process == returning.next_process.bytes() + && attestation.boot.engine == boot.engine + ); + let cut = catalog.read(TIMEOUT).await?; + ensure!(catalog + .authorize_capacity(&cut, node, TIMEOUT) + .await + .is_err()); + paused_write_refuses(&cluster.hosts[target_index], seed) + .await + .context("returned paused business admission")?; + ensure!(matches!( + observer + .request_peer_fleet_stop(stop_admission.stop_request()?, cut.revision(), TIMEOUT) + .await, + Err(FleetCatalogError::Conflict) + )); + ensure!( + !cluster.nodes[target_index].is_closed() + && catalog.read(TIMEOUT).await?.revision() == cut.revision() + ); + let resume = FleetRestartResumeRequest { + identity: intent(228, cut.revision()), + returning: returning.identity, + attestation: attestation.clone(), + resume: ProposalId::new([229; 16])?, + }; + let mut walk = catalog + .prepare_restart_resume( + &cut, + &resume, + std::slice::from_ref(&cluster.directory), + &observer, + TIMEOUT, + ) + .await + .context("prepare joined resume")?; + ensure!(walk.phase() == FleetRestartResumePhase::Paused); + eprintln!("maintenance: verify returned paused voting history"); + tokio::time::timeout(TIMEOUT, async { + loop { + match walk.reconcile_next(TIMEOUT).await { + Ok(Some(FleetRestartResumeProgress::Verified { + group, + current_member, + retained_member, + replaced, + })) => ensure!( + group == cluster.history + && current_member == Some(member) + && retained_member.is_none() + && !replaced + ), + Ok(None) => break, + Err( + FleetCatalogError::Unconfirmed + | FleetCatalogError::Overloaded + | FleetCatalogError::InventoryIncomplete, + ) + | Err(FleetCatalogError::Observation( + InspectionError::Unavailable | InspectionError::Overloaded, + )) => tokio::time::sleep(Duration::from_millis(10)).await, + other => anyhow::bail!("joined resume verification: {other:?}"), + } + } + Ok::<_, anyhow::Error>(()) + }) + .await + .context("renew returned paused voting proof")??; + ensure!(walk.verified_histories() == 1); + let consent = walk + .admit_resume(&resume, TIMEOUT) + .await + .context("admit joined resume")?; + verify_cells(&survivor, seed, 4).await?; + discover(cluster).await?; + let before_resume = catalog.read(TIMEOUT).await?; + let mut wrong_child = serde_json::to_value(consent.resume_request()?)?; + wrong_child["selector"]["request"] = serde_json::json!(vec![235; 16]); + ensure!(matches!( + observer + .request_peer_fleet_stop( + serde_json::from_value(wrong_child)?, + before_resume.revision(), + TIMEOUT, + ) + .await, + Err(FleetCatalogError::PayloadConflict) + )); + ensure!(catalog.read(TIMEOUT).await?.revision() == before_resume.revision()); + ensure!( + cluster.hosts[target_index] + .observe_node(TIMEOUT) + .await? + .node + .profile + .draining + ); + let resumed = tokio::time::timeout(TIMEOUT, async { + loop { + match catalog + .resume_restart_maintenance(&consent, &observer, TIMEOUT) + .await + { + Err(FleetCatalogError::Overloaded) => { + tokio::time::sleep(Duration::from_millis(10)).await; + } + result => break result, + } + } + }) + .await + .context("submit exact owned resume within original deadline")?; + ensure!( + resumed.is_ok() || matches!(resumed, Err(FleetCatalogError::Unconfirmed)), + "owned resume refused: {:?}", + resumed.err() + ); + // This local hint waits without retaining another peer RPC. It grants no + // authority: the fresh Resumed walk and terminal root checks below remain + // required before capacity or successful return is claimed. + tokio::time::timeout(TIMEOUT, async { + loop { + if !cluster.hosts[target_index] + .observe_node(TIMEOUT) + .await? + .node + .profile + .draining + { + break Ok::<_, anyhow::Error>(()); + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .context("observe actual owned resume barrier")??; + let cut = catalog.read(TIMEOUT).await?; + ensure!( + catalog.authorize_host(&cut, node, TIMEOUT).await?.state() + == FleetNodeState::RestartPending + && catalog + .authorize_capacity(&cut, node, TIMEOUT) + .await + .is_err() + ); + ensure!( + !cluster.hosts[target_index] + .observe_node(TIMEOUT) + .await? + .node + .profile + .draining + ); + let mut completed = catalog + .prepare_restart_resume( + &cut, + &resume, + std::slice::from_ref(&cluster.directory), + &observer, + TIMEOUT, + ) + .await?; + ensure!(completed.phase() == FleetRestartResumePhase::Resumed); + let progress = completed + .reconcile_next(TIMEOUT) + .await? + .context("resumed joined history")?; + ensure!( + matches!(progress, FleetRestartResumeProgress::Verified { group, current_member: Some(member), retained_member: None, replaced: false } if group == cluster.history && member == MemberId::new(target_index as u64 + 1)?) + ); + ensure!(completed.reconcile_next(TIMEOUT).await?.is_none()); + let cut = catalog.read(TIMEOUT).await?; + let terminal = FleetRestartResumeCompleteRequest { + identity: intent(231, cut.revision()), + resume: resume.identity, + }; + let done = completed + .complete_resume(&terminal, TIMEOUT) + .await + .context("settle joined resume")?; + ensure!(done.intact() && done.retained_copies() == 0 && done.replaced_histories() == 0); + let cut = catalog.read(TIMEOUT).await?; + ensure!(catalog.authorize_host(&cut, node, TIMEOUT).await?.state() == FleetNodeState::Active); + let capacity = catalog.authorize_capacity(&cut, node, TIMEOUT).await?; + ensure!(capacity.receive_replicas() && capacity.attestation() == &attestation); + discover(cluster).await?; + handoff_to(cluster, obsolete, target_index).await?; + verify_cells(&cluster.hosts[target_index], seed, 5) + .await + .context("actual resumed owner serves both cells")?; + let next = FleetRestartRequest { + identity: { + let mut i = request(232, cut.revision()); + i.operation = [233; 16]; + i + }, + node, + expected_process: returning.next_process, + grace_seconds: 120, + }; + catalog + .begin_restart_maintenance( + &cut, + &next, + std::slice::from_ref(&cluster.directory), + TIMEOUT, + ) + .await + .context("terminal resume releases disruption slot")?; + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_managed_fixture_movement.rs b/crates/cellar/tests/support/cell_group_managed_fixture_movement.rs new file mode 100644 index 000000000..7c20df5d6 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_managed_fixture_movement.rs @@ -0,0 +1,462 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Whole-group movement, exact debt reconciliation and owned drain. + +use super::*; + +pub async fn discover(cluster: &ManagedCluster) -> Result<()> { + discover_survivors(cluster, None).await +} + +pub async fn discover_survivors(cluster: &ManagedCluster, stopped: Option) -> Result<()> { + let cut = cluster.fixtures[0].catalog.read(TIMEOUT).await?; + for (i, host) in cluster.hosts.iter().enumerate() { + if stopped == Some(i) { + ensure!(host.is_closed()); + continue; + } + for (j, fixture) in cluster.fixtures.iter().enumerate() { + if stopped == Some(j) { + continue; + } + inspect_current(host, fixture.spec.node, cut.revision()).await?; + } + } + Ok(()) +} + +pub async fn close(nodes: Vec, hosts: &[RaftCells]) -> Result<()> { + let mut failures = Vec::new(); + for (node, host) in nodes.into_iter().zip(hosts) { + match node.shutdown(ShutdownMode::Preserve).await { + Ok(NodeOutcome::Stopped(_)) => (), + result => failures.push(format!("node shutdown: {result:?}")), + } + match host.close().await { + Ok(report) + if report.mutation_outcomes == MutationOutcomes::NoUnresolvedLocalProducers => + { + () + } + result => failures.push(format!("group host close: {result:?}")), + } + } + ensure!( + failures.is_empty(), + "joined managed shutdown failed: {failures:?}" + ); + Ok(()) +} + +pub async fn replace(cluster: &ManagedCluster) -> Result<(SeededCells, usize)> { + Box::pin(replace_with_source(cluster, false)).await +} + +pub async fn replace_with_source( + cluster: &ManagedCluster, + from_s3: bool, +) -> Result<(SeededCells, usize)> { + Box::pin(replace_with_retry(cluster, from_s3, false)).await +} + +pub async fn reserve_replacement( + cluster: &ManagedCluster, + from_s3: bool, +) -> Result<( + SeededCells, + usize, + usize, + MemberId, + FleetMovementExecutionRequest, +)> { + let seed = seed_cells(&cluster.hosts, cluster.history).await?; + discover(cluster).await?; + let index = leader(&cluster.hosts[..3], cluster.history).await?; + let obsolete = (index + 1) % 3; + let target = ReplacementTarget { + source: index, + obsolete, + spare: 3, + }; + let (obsolete_member, execution) = + admit_replacement(cluster, target, [210, 211], from_s3).await?; + Ok((seed, index, obsolete, obsolete_member, execution)) +} + +pub struct ReplacementTarget { + pub source: usize, + pub obsolete: usize, + pub spare: usize, +} + +pub async fn admit_replacement( + cluster: &ManagedCluster, + target: ReplacementTarget, + ids: [u8; 2], + from_s3: bool, +) -> Result<(MemberId, FleetMovementExecutionRequest)> { + let f = &cluster.fixtures[0]; + let observer = &cluster.hosts[target.source]; + let membership = observer + .group(cluster.history)? + .quorum_membership(TIMEOUT) + .await?; + let local = cluster.hosts[target.spare].observe_node(TIMEOUT).await?; + let cut = f.catalog.read(TIMEOUT).await?; + let movement = FleetMovementRequest { + resources: FleetResourceRequest { + identity: request(ids[0], cut.revision()), + group: cluster.history, + attestation: FleetResourceAttestation { + boot: local.fleet_boot.context("spare boot")?, + profile_revision: 1, + used: FleetResourceBounds { + engine_slots: 0, + durable_bytes: 1024 * 1024, + memory_bytes: 64 * 1024 * 1024, + }, + evidence_digest: [71; 32], + evidence_reference: "fixture-only/occupancy-attestation".into(), + }, + peak: FleetResourceBounds { + engine_slots: 1, + durable_bytes: local.node.profile.bytes_per_group, + memory_bytes: 8 * 1024 * 1024, + }, + }, + source: cluster.fixtures[target.source].spec.node, + plan: RepairPlan { + id: ProposalId::new([ids[1]; 16])?, + policy_revision: 1, + expected_revision: membership.membership().revision(), + obsolete: MemberId::new(target.obsolete as u64 + 1)?, + replacement: JoiningMember { + member: MemberId::new(target.spare as u64 + 1)?, + node: cluster.fixtures[target.spare].spec.node, + disk: cluster.fixtures[target.spare].spec.disk, + }, + }, + }; + drop(membership); + let accepted = f + .catalog + .reserve_movement( + &cut, + &movement, + observer, + std::slice::from_ref(&cluster.directory), + TIMEOUT, + ) + .await + .context("reserve managed group replacement")?; + let admitted = f.catalog.read(TIMEOUT).await?; + ensure!(admitted.revision() == cut.revision() + 1); + ensure!(accepted.resources().applied_revision() == admitted.revision()); + let rows = f.catalog.pending_reservations(&admitted, TIMEOUT).await?; + ensure!( + rows.len() == 1 + && rows[0].reservation == movement.resources.identity.id + && rows[0].boot == movement.resources.attestation.boot + && rows[0].peak == movement.resources.peak + && rows[0].executable_movement + ); + let execution = FleetMovementExecutionRequest { + reservation: movement.resources.identity.id, + attestation: movement.resources.attestation.clone(), + from_s3, + }; + Ok((movement.plan.obsolete, execution)) +} + +pub async fn replace_with_retry( + cluster: &ManagedCluster, + from_s3: bool, + refuse_then_failover: bool, +) -> Result<(SeededCells, usize)> { + let (seed, index, obsolete, obsolete_member, execution) = + Box::pin(reserve_replacement(cluster, from_s3)).await?; + let f = &cluster.fixtures[0]; + let mut observer = &cluster.hosts[index]; + let stopped_source = if refuse_then_failover { + ensure!(from_s3); + let next = Box::pin(seed_failures::refuse_then_failover( + cluster, &execution, index, + )) + .await?; + observer = &cluster.hosts[next]; + Some(index) + } else { + None + }; + let mut settled = false; + let mut phases = Vec::new(); + let mut lose_settlement = false; + let mut lost_settlement = false; + for turn in 0..8 { + discover_survivors(cluster, stopped_source).await?; + let cut = f.catalog.read(TIMEOUT).await?; + if stopped_source.is_none() + && phases.last() == Some(&FleetMovementPhase::Joint) + && !lose_settlement + { + // Lose the actual terminal root-CAS reply and its confirmation. + // Background directory reads cannot consume this task-bound fault. + f.store + .fail_fleet_confirmation + .store(true, Ordering::Release); + lose_settlement = true; + } + let progress = match f + .catalog + .reconcile_movement(&cut, &execution, observer, TIMEOUT) + .await + { + Ok(progress) => progress, + Err(FleetCatalogError::Unconfirmed) => { + let current = f.catalog.read(TIMEOUT).await?; + let pending = f.catalog.pending_reservations(¤t, TIMEOUT).await?; + if lose_settlement && !f.store.fail_fleet_confirmation.load(Ordering::Acquire) { + ensure!(current.revision() == cut.revision() + 1 && pending.is_empty()); + lost_settlement = true; + eprintln!("managed replacement turn {turn}: terminal CAS reply lost; receipt reconciliation required"); + } else { + // The learner or configuration may already be durable. + // A nonterminal outcome cannot release its resource debit. + ensure!(pending.len() == 1 && pending[0].reservation == execution.reservation); + eprintln!("managed replacement turn {turn}: Unconfirmed; debit retained"); + } + continue; + } + Err(FleetCatalogError::Observation( + cellar::application::consensus::InspectionError::Unreachable, + )) if stopped_source.is_some() => { + ensure!(f.catalog.read(TIMEOUT).await?.revision() == cut.revision()); + Box::pin(seed_failures::verify_blocked_settlement( + cluster, &seed, &execution, observer, obsolete, + )) + .await?; + return Ok((seed, obsolete)); + } + Err(error) => { + return Err(error).with_context(|| { + format!("managed replacement turn {turn}, observed {phases:?}") + }) + } + }; + eprintln!( + "managed replacement rooted={:?} phase={:?} settled={}", + f.declaration.namespace.image_storage, progress.phase, progress.settled + ); + phases.push(progress.phase); + if progress.settled { + settled = true; + break; + } + ensure!( + f.catalog + .pending_reservations(&f.catalog.read(TIMEOUT).await?, TIMEOUT) + .await? + .len() + == 1, + "unsettled work released its debit" + ); + } + ensure!( + settled && lost_settlement, + "replacement did not reconcile its lost settlement: {phases:?}" + ); + ensure!( + phases.contains(&FleetMovementPhase::Seeding) + && phases.contains(&FleetMovementPhase::Joint) + && phases.last() == Some(&FleetMovementPhase::Complete) + ); + let cut = f.catalog.read(TIMEOUT).await?; + ensure!(f + .catalog + .pending_reservations(&cut, TIMEOUT) + .await? + .is_empty()); + ensure!(f.catalog.pending_movements(&cut, TIMEOUT).await?.is_empty()); + // Replaying a lost terminal reply observes the permanent receipt. + let replay = f + .catalog + .reconcile_movement(&cut, &execution, observer, TIMEOUT) + .await?; + ensure!(replay.settled && replay.phase == FleetMovementPhase::Complete); + let membership = observer + .group(cluster.history)? + .quorum_membership(TIMEOUT) + .await?; + let mut voters = membership.configuration().voters.clone(); + voters.sort_unstable(); + let expected = (1..=4) + .filter(|id| *id != obsolete as u64 + 1) + .collect::>(); + ensure!( + voters == expected + && membership.configuration().voters_outgoing.is_empty() + && membership.configuration().learners.is_empty() + ); + drop(membership); + discover_survivors(cluster, stopped_source).await?; + // Settlement retires the move, not the old disk's physical debt. + let removed = observer + .inspect_peer_physical_inventory( + cluster.fixtures[obsolete].spec.node, + cut.revision(), + TIMEOUT, + ) + .await?; + let inventory = removed + .inventory + .context("removed member physical inventory")?; + ensure!(inventory.histories().len() == 1); + let retained = inventory + .group_history(&cluster.history) + .context("retained removed history")?; + ensure!( + !retained.in_configuration() + && retained.completed_sequence() == 1 + && retained.member() == obsolete_member + ); + ensure!( + cluster.hosts[obsolete] + .resource_usage() + .retained_manifest_cells + == Some(2) + ); + ensure!(cluster.hosts[3].resource_usage().retained_manifest_cells == Some(2)); + let archive_reads = cluster.fixtures[3] + .store + .group_payload_reads + .load(Ordering::Acquire); + if from_s3 { + ensure!( + archive_reads > 0, + "S3 replacement never fetched archive payloads" + ); + } + verify_cells(observer, &seed, 2).await?; + Ok((seed, obsolete)) +} + +pub async fn drain_leader( + cluster: &mut ManagedCluster, + seed: &SeededCells, + obsolete: usize, +) -> Result<()> { + let members = (1..=4).collect::>(); + let index = current_leader(&cluster.hosts, &members, cluster.history).await?; + ensure!(index != obsolete); + let node = cluster.nodes.remove(index); + let host = cluster.hosts.remove(index); + let outcome = node.shutdown(ShutdownMode::Handoff).await?; + let closure = host.close().await?; + ensure!( + closure.groups_total == 1 + && closure.mutation_outcomes == MutationOutcomes::NoUnresolvedLocalProducers + ); + let NodeOutcome::Stopped(report) = outcome else { + anyhow::bail!("leader did not stop: {outcome:?}"); + }; + let raft = report.raft.context("leader shutdown Raft proof")?; + ensure!(!raft.startup_incomplete && raft.result.is_ok()); + let drain = raft.drain.context("actual group drain proof")??; + ensure!( + drain.groups_examined == 1 + && drain.handoffs_accepted == 1 + && drain.leadership_released == 1 + && drain.unavailable_groups == 0 + && !drain.deadline_elapsed, + "incomplete group handoff: {drain:?}" + ); + let survivors = (0..4) + .filter(|i| *i != index && *i != obsolete) + .collect::>(); + let hosts = survivors + .iter() + .map(|original| { + cluster.hosts[if *original > index { + original - 1 + } else { + *original + }] + .clone() + }) + .collect::>(); + let members = survivors.iter().map(|i| *i as u64 + 1).collect::>(); + let leader = current_leader(&hosts, &members, cluster.history).await?; + verify_cells(&hosts[leader], seed, 3).await?; + Ok(()) +} + +pub async fn current_leader( + hosts: &[RaftCells], + members: &[u64], + history: GroupHistory, +) -> Result { + Ok(tokio::time::timeout(TIMEOUT, async { + loop { + for (i, (host, member)) in hosts.iter().zip(members).enumerate() { + let status = host.group(history)?.status().await?; + if status.ready && status.leader == Some(*member) { + return Ok::<_, anyhow::Error>(i); + } + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await??) +} + +pub async fn verify_cells(host: &RaftCells, seed: &SeededCells, sequence: u64) -> Result<()> { + for (i, (cell, original)) in seed.cells.iter().zip(&seed.receipts).enumerate() { + let handle = host.cell(cell.clone())?; + let delta = [7i64, 11][i]; + let mut duplicate_options = CommandOptions::new(original.id, seed.retry); + duplicate_options.durability = Durability::S3; + let duplicate = handle + .command( + &handle.contract("Add")?, + delta.to_be_bytes().to_vec(), + duplicate_options, + ) + .await?; + ensure!( + duplicate.result == original.result + && duplicate.id == original.id + && duplicate.durability == Durability::S3 + ); + let query = handle + .query( + ProposalId::new([190 + i as u8; 16])?, + &handle.contract("Get")?, + vec![], + QueryOptions { + consistency: QueryConsistency::AtLeast(original.version_token.clone()), + timeout: TIMEOUT, + ..Default::default() + }, + ) + .await?; + ensure!( + query.result == (delta + sequence as i64 - 2).to_be_bytes(), + "duplicate changed the installed image" + ); + let mut write_options = + CommandOptions::new(CommandId::new(original.id.client, sequence)?, seed.retry); + write_options.durability = Durability::S3; + let result = handle + .command( + &handle.contract("Add")?, + 1i64.to_be_bytes().to_vec(), + write_options, + ) + .await?; + ensure!( + result.result == (delta + sequence as i64 - 1).to_be_bytes() + && result.durability == Durability::S3 + ); + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_managed_fixture_node.rs b/crates/cellar/tests/support/cell_group_managed_fixture_node.rs new file mode 100644 index 000000000..b1c2efffe --- /dev/null +++ b/crates/cellar/tests/support/cell_group_managed_fixture_node.rs @@ -0,0 +1,421 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Managed namespace capability and actual node startup fixture. + +use super::*; +use object_store::list::*; + +pub struct NoListing; + +#[async_trait::async_trait] +impl PaginatedListStore for NoListing { + async fn list_paginated( + &self, + _: Option<&str>, + _: PaginatedListOptions, + ) -> object_store::Result { + Err(object_store::Error::NotSupported { + source: "managed discovery never lists".into(), + }) + } +} + +#[derive(Clone)] +pub struct Fixture { + pub domain: Arc, + pub store: Arc, + pub catalog: FleetCatalog, + pub declaration: RaftNamespaceRegistration, + pub application: Arc, + pub spec: FleetNodeSpec, +} + +impl Fixture { + pub fn version_keys() -> Result { + Ok(VersionKeys::new( + VersionSigningKey::new([31; 32], u64::MAX)?, + vec![], + Duration::from_secs(86400), + )?) + } + + pub async fn new(rooted: bool) -> Result { + Box::pin(Self::with_class( + rooted, + RaftClass::new("Counter", 1, [81; 32])?, + )) + .await + } + + pub async fn with_class(rooted: bool, class: RaftClass) -> Result { + Box::pin(Self::with_store( + rooted, + class, + Arc::new(fault::Store::default()), + )) + .await + } + + pub async fn with_store( + rooted: bool, + class: RaftClass, + store: Arc, + ) -> Result { + let domain = Arc::new(ExecutionDomain::new(tokio::runtime::Handle::current())); + let catalog = Self::open_catalog(&domain, store.clone(), ClusterId::new([1; 16])?)?; + catalog.initialize(TIMEOUT).await?; + let namespace = NamespaceId::new([2; 16])?; + let namespace_catalog = Catalog::new( + store.clone(), + CatalogOptions { + prefix: "managed/".into(), + ..Default::default() + }, + )?; + let cut = catalog.read(TIMEOUT).await?; + let identity = request(1, cut.revision()); + let directory_limits = DirectoryLimits::default(); + let declaration = if rooted { + let mut config = NamespaceConfig::new( + ClusterId::new([1; 16])?, + namespace, + NamespaceMode::Raft, + class.schema_digest(), + ); + config.image_storage = RaftImageStorage::Extents; + catalog + .provision_managed_namespace( + &identity, + &namespace_catalog, + config, + &class, + directory_limits, + TIMEOUT, + ) + .await? + } else { + catalog + .provision_managed_sql_namespace( + &identity, + &namespace_catalog, + namespace, + &class, + directory_limits, + TIMEOUT, + ) + .await? + }; + let application: Arc = + Arc::new(Registry::new(|_, _| Box::pin(async { Ok(Output::Done) })).raft_class(class)?); + let spec = FleetNodeSpec { + node: NodeId::new([4; 16])?, + disk: DiskLineage::new([14; 16])?, + physical_host: IncarnationId::new([90; 16])?, + label: "shared-managed".into(), + server_name: "localhost".into(), + capability_digest: RaftManagedHostOptions::capability_digest(std::slice::from_ref( + &declaration, + ))?, + storage_instance: None, + }; + let cut = catalog.read(TIMEOUT).await?; + catalog + .register_node(&cut, &request(2, cut.revision()), &spec, TIMEOUT) + .await?; + Ok(Self { + domain, + store, + catalog, + declaration, + application, + spec, + }) + } + + pub fn open_catalog( + domain: &ExecutionDomain, + store: Arc, + cluster: ClusterId, + ) -> Result { + let bucket = Bucket::from_clients( + store.clone(), + store, + Arc::new(NoListing), + StorageBackend::S3, + "managed-component".into(), + "tenant/".into(), + )?; + Ok(FleetCatalog::new( + domain, + bucket, + cluster, + IncarnationId::new([80; 16])?, + "hosts/", + )?) + } + + pub async fn config( + &self, + node: &Node, + mode: LogOpenMode, + generation: u8, + ) -> Result<(RaftCellsConfig, RaftCellsServices)> { + let cut = self.catalog.read(TIMEOUT).await?; + let authorization = self + .catalog + .authorize_host(&cut, self.spec.node, TIMEOUT) + .await?; + let listeners = NodeRaftListeners { + control: tokio::net::TcpListener::bind("127.0.0.1:0").await?, + data: tokio::net::TcpListener::bind("127.0.0.1:0").await?, + query: tokio::net::TcpListener::bind("127.0.0.1:0").await?, + }; + let process = ProcessGeneration::new([generation; 16])?; + let peer = RaftPeerNode { + node: self.spec.node, + disk: self.spec.disk, + process, + label: self.spec.label.clone(), + server_name: "localhost".into(), + control_address: listeners.control.local_addr()?, + data_address: listeners.data.local_addr()?, + query_address: listeners.query.local_addr()?, + }; + let mut host = RaftHostOptions { + max_groups: 2, + max_input_bytes: 1024, + ..Default::default() + }; + host.node_profile = Some(RaftNodeProfile { + pool: "test".into(), + region: "local".into(), + zone: "node".into(), + group_capacity: 2, + byte_capacity: host.images.max_bytes * 2, + bytes_per_group: host.images.max_bytes, + leader_capacity: 2, + draining: false, + }); + host.node_directory = Some(RaftNodeDirectoryOptions::default()); + host.managed = Some(RaftManagedHostOptions { + authorization, + namespaces: vec![self.declaration.clone()], + }); + if self.declaration.namespace.image_storage == RaftImageStorage::Extents { + host.extents = Some(Default::default()); + } + let mut log = ReplicaLogOptions::new( + node.info().data_dir.join("raft"), + peer.node, + peer.disk, + process, + mode, + ); + log.max_groups = 2; + Ok(( + RaftCellsConfig { + infrastructure: NodeRaftConfig { + log, + consensus: RaftRuntimeOptions { + max_groups: 2, + ..Default::default() + }, + transport: TransportOptions::default(), + }, + catalog: CatalogOptions { + prefix: "managed/".into(), + ..Default::default() + }, + namespaces: vec![RaftNamespaceConfig { + namespace: self.declaration.namespace.clone(), + class: "Counter".into(), + initialize: None, + }], + groups: vec![], + peers: vec![peer], + query_origins: vec![], + command_origins: vec![], + host, + shared: limits(), + }, + RaftCellsServices { + recovery: None, + listeners, + tls: RaftTls::new( + vec![include_bytes!("../../transport/tcp/fixtures/ca.der") + .to_vec() + .into()], + vec![include_bytes!("../../transport/tcp/fixtures/server.der") + .to_vec() + .into()], + rustls::pki_types::PrivatePkcs8KeyDer::from( + include_bytes!("../../transport/tcp/fixtures/key.der").to_vec(), + ) + .into(), + )?, + store: self.store.clone(), + application: self.application.clone(), + versions: Self::version_keys()?, + }, + )) + } +} + +pub fn request(id: u8, revision: u64) -> FleetRequestIdentity { + FleetRequestIdentity { + id: [id; 16], + operation: [100; 16], + principal: "managed-test".into(), + expected_revision: revision, + } +} + +pub fn limits() -> CellGroupNodeLimits { + CellGroupNodeLimits { + namespaces: CellGroupHostLimits { + max_namespaces: 1, + namespace_bytes: 16 * 1024 * 1024, + }, + controller: CellGroupControllerLimits { + max_groups: 2, + max_parallel_apply: 2, + retry_interval_ms: 5, + max_commands: 2, + max_command_bytes: 64 * 1024, + checkpoints: None, + }, + sql: CellSqlLimits { + workers: 2, + queue_per_worker: 4, + max_pending_cells: 8, + group_pending_cells: 8, + node_bytes: 512 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + }, + provisioning: CellGroupProvisionLimits { + max_preparations: 1, + max_preparation_bytes: 512 * 1024 * 1024, + max_parallel_preparations: 1, + max_waiters: 2, + peer_revision: 1, + }, + directory: DirectoryLimits::default(), + dispatch: CellDispatchLimits { + max_cached_routes: 32, + route_cache_bytes: 256 * 1024, + max_invocations: 8, + retained_bytes: 1024 * 1024, + max_input_bytes: 1024, + }, + manifest: ManifestLimits { + max_cells: 32, + max_depth: 16, + }, + entries: Default::default(), + manifest_pages: 1024, + manifest_bytes: 16 * 1024 * 1024, + retained_roots: 16, + max_effect_rows: 100, + checkpoints: None, + archive: None, + } +} + +pub async fn start_node( + directory: &Path, + application: Arc, + label: &str, +) -> Result { + let classes = vec![]; + let filesystem = Arc::new(cellar_ltx::DirectFileSystem); + let shutdown = ShutdownPolicy::default(); + let bucket: Option = None; + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await?; + let internal_listener = tokio::net::TcpListener::bind("127.0.0.1:0").await?; + let probe_signer = Arc::new(PeerProbeSigner::generate()); + let mut config = NodeConfig::new( + ActorIdentity { + node: label.into(), + advertise: internal_listener.local_addr()?.to_string(), + region: "test".into(), + }, + ActorOptions { + core: ActorPolicy { + max_resident: 16, + max_cached_routes: 0, + max_activations: 8, + max_evictions: 4, + max_releases: 4, + max_outbound_websockets: 8, + ownership_on_evict: OwnershipOnEvict::Sticky, + require_node_lease: true, + peer_protocol: cellar::peer_auth::PROTOCOL_VERSION, + operation_deadline_ms: Some(3000), + owner_log_recovery_backoff_ms: 50, + owner_log_recovery_attempts: 3, + alarm_resident_ms: 100, + idle_evict_ms: None, + pressure: Default::default(), + }, + lease_ttl_ms: 1000, + placement_weight: 1, + generation: probe_signer.public_key_hex(), + resume_generation: None, + }, + directory.into(), + ); + config.shutdown = shutdown; + let peer_http = reqwest::Client::builder().no_proxy().build()?; + let peer_auth = Arc::new(PeerAuth::new([11; 32], label)?); + let fleet = bucket + .map(|bucket| -> Result { + Ok(NodeFleet { + replica_storage: ReplicaStorage::new( + bucket.store.clone(), + ReplicaStorageOptions { + prefix: bucket.prefix.clone(), + ..Default::default() + }, + )?, + lease_bucket: bucket.clone(), + bucket, + replication: ReplicationOptions { + flush_interval: Duration::ZERO, + compaction: None, + ..Default::default() + }, + log_transport: Arc::new(SignedPeerTransport::from_clients( + peer_http.clone(), + peer_http.clone(), + peer_auth.clone(), + )), + log: LogOptions::default(), + fleet_durability: true, + }) + }) + .transpose()?; + let services = NodeServices { + wall_clock: None, + http_client: None, + host_executor: tokio::runtime::Handle::current(), + core_executor: tokio::runtime::Handle::current(), + listener, + internal_listener, + peer_http, + peer_auth, + probe_signer, + fleet, + containers: None, + metrics: None, + }; + let compiled = + CompiledApplications::default().register(label, "test", move || Ok(application.clone()))?; + Node::start_with_filesystem( + config, + services, + graph::graph(label, "", classes), + compiled, + filesystem, + ) + .await +} diff --git a/crates/cellar/tests/support/cell_group_managed_fixture_operator.rs b/crates/cellar/tests/support/cell_group_managed_fixture_operator.rs new file mode 100644 index 000000000..339f79d30 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_managed_fixture_operator.rs @@ -0,0 +1,724 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Real operator CLI and admin admission for the managed fixture. + +use super::*; +use cellar::application::consensus::{CellGroupRecoveryPool, FleetGroupArchiveVerifier}; +use cellar::sdk::fleet_operations::*; +use futures_util::TryStreamExt as _; +use object_store::ObjectStore as _; +#[path = "raft_first_group_operator_cli.rs"] +mod cli; +use cli::{first_group_plan, OperatorCli, FLEET, OWNER, READER}; + +pub(super) async fn provision( + f: &Fixture, + directory: &RangeDirectory, + voter: &RaftCells, + setup: RaftManagedFirstGroupRequest, +) -> Result { + let root = tempfile::tempdir()?; + let cli = open_cli( + f, + directory, + root.path(), + &setup.reservation.principal, + None, + ) + .await?; + let result = exercise(f, &cli, root.path(), voter, setup).await; + cli.close().await?; + result +} + +async fn open_cli( + fixture: &Fixture, + directory: &RangeDirectory, + root: &Path, + principal: &str, + observer: Option<&RaftCells>, +) -> Result { + let (bucket, mut control) = control(fixture, directory)?; + if let Some(observer) = observer { + control = control.with_node_observer(observer.clone()); + } + OperatorCli::start(&fixture.domain, bucket, control, root, principal).await +} + +fn control(fixture: &Fixture, directory: &RangeDirectory) -> Result<(Bucket, RaftFleetControl)> { + let bucket = Bucket::from_clients( + fixture.store.clone(), + fixture.store.clone(), + Arc::new(NoListing), + StorageBackend::S3, + "managed-component".into(), + "tenant/".into(), + )?; + let catalog = Fixture::open_catalog( + &fixture.domain, + fixture.store.clone(), + directory.namespace_config().cluster, + )?; + Ok(( + bucket, + RaftFleetControl::new(&fixture.domain, catalog, vec![directory.clone()])?, + )) +} + +pub(crate) async fn prepare_transition( + fixture: &Fixture, + directory: &RangeDirectory, + principal: &str, + request: &RaftFleetTransitionPrepareRequest, + preparer: Option<(FleetGroupArchiveVerifier, CellGroupRecoveryPool)>, +) -> Result { + let root = tempfile::tempdir()?; + let (bucket, mut control) = control(fixture, directory)?; + if let Some((verifier, recovery)) = preparer { + control = control.with_transition_preparer(verifier, recovery); + } + let cli = OperatorCli::start(&fixture.domain, bucket, control, root.path(), principal).await?; + let result = async { + let plan = root.path().join("transition-prepare.json"); + std::fs::write(&plan, serde_json::to_vec(request)?)?; + let command = [ + "transition", + "prepare", + &request.namespace_id, + "--plan", + plan.to_str() + .context("UTF-8 transition preparation request path")?, + ]; + ensure!(cli.call(&command, READER).await?.code == 4); + Ok::<_, anyhow::Error>(serde_json::from_value( + cli.call(&command, OWNER).await?.success()?, + )?) + } + .await; + cli.close().await?; + result +} + +pub(crate) async fn capture_transition_source( + fixture: &Fixture, + directory: &RangeDirectory, + principal: &str, + request: &RaftFleetTransitionSourceSealRequest, + preparer: (FleetGroupArchiveVerifier, CellGroupRecoveryPool), +) -> Result { + let root = tempfile::tempdir()?; + let (bucket, mut control) = control(fixture, directory)?; + control = control.with_transition_preparer(preparer.0, preparer.1); + let cli = OperatorCli::start(&fixture.domain, bucket, control, root.path(), principal).await?; + let result = async { + let plan = root.path().join("transition-capture-source.json"); + std::fs::write(&plan, serde_json::to_vec(request)?)?; + let command = [ + "transition", + "capture-source", + &request.namespace_id, + "--plan", + plan.to_str().context("UTF-8 source capture request path")?, + ]; + ensure!(cli.call(&command, READER).await?.code == 4); + Ok::<_, anyhow::Error>(serde_json::from_value( + cli.call_with_timeout(&command, OWNER, Duration::from_secs(32)) + .await? + .success()?, + )?) + } + .await; + cli.close().await?; + result +} + +pub(crate) async fn reopen_transition_retirement( + fixture: &Fixture, + directory: &RangeDirectory, + principal: &str, + request: &RaftFleetTransitionSourceSealRequest, +) -> Result { + let root = tempfile::tempdir()?; + let (bucket, control) = control(fixture, directory)?; + let cli = OperatorCli::start(&fixture.domain, bucket, control, root.path(), principal).await?; + let result = async { + let plan = root.path().join("transition-retire-source.json"); + std::fs::write(&plan, serde_json::to_vec(request)?)?; + let command = [ + "transition", + "retire-source", + &request.namespace_id, + "--plan", + plan.to_str().context("UTF-8 retirement request path")?, + ]; + ensure!(cli.call(&command, READER).await?.code == 4); + Ok::<_, anyhow::Error>(serde_json::from_value( + cli.call(&command, OWNER).await?.success()?, + )?) + } + .await; + cli.close().await?; + result +} + +pub(crate) async fn verify_transition_archive( + fixture: &Fixture, + directory: &RangeDirectory, + principal: &str, + request: &RaftFleetTransitionPrepareRequest, + preparer: (FleetGroupArchiveVerifier, CellGroupRecoveryPool), + command_name: &str, +) -> Result { + let root = tempfile::tempdir()?; + let (bucket, mut control) = control(fixture, directory)?; + control = control.with_transition_preparer(preparer.0, preparer.1); + let cli = OperatorCli::start(&fixture.domain, bucket, control, root.path(), principal).await?; + let result = async { + let plan = root.path().join("transition-verify-archive.json"); + std::fs::write(&plan, serde_json::to_vec(request)?)?; + let command = [ + "transition", + command_name, + &request.namespace_id, + "--plan", + plan.to_str() + .context("UTF-8 archive verification request path")?, + ]; + ensure!(cli.call(&command, READER).await?.code == 4); + Ok::<_, anyhow::Error>(serde_json::from_value( + cli.call_with_timeout(&command, OWNER, Duration::from_secs(32)) + .await? + .success()?, + )?) + } + .await; + cli.close().await?; + result +} + +pub(crate) async fn setup_transition_child( + fixture: &Fixture, + directory: &RangeDirectory, + principal: &str, + observer: &RaftCells, + request: &RaftFleetTransitionChildSetupRequest, + preparer: Option<(FleetGroupArchiveVerifier, CellGroupRecoveryPool)>, +) -> Result { + let root = tempfile::tempdir()?; + let (bucket, mut control) = control(fixture, directory)?; + control = control.with_node_observer(observer.clone()); + if let Some((verifier, recovery)) = preparer { + control = control.with_transition_preparer(verifier, recovery); + } + let cli = OperatorCli::start(&fixture.domain, bucket, control, root.path(), principal).await?; + let result = async { + let plan = root.path().join("transition-setup-child.json"); + std::fs::write(&plan, serde_json::to_vec(request)?)?; + let command = [ + "transition", + "setup-child", + &request.namespace_id, + "--plan", + plan.to_str().context("UTF-8 child setup request path")?, + ]; + ensure!(cli.call(&command, READER).await?.code == 4); + Ok::<_, anyhow::Error>(serde_json::from_value( + cli.call_with_timeout(&command, OWNER, Duration::from_secs(32)) + .await? + .success()?, + )?) + } + .await; + cli.close().await?; + result +} + +pub(crate) async fn transition_status( + fixture: &Fixture, + directory: &RangeDirectory, + principal: &str, + expected: &RaftFleetTransitionStatus, +) -> Result<()> { + let root = tempfile::tempdir()?; + let cli = open_cli(fixture, directory, root.path(), principal, None).await?; + let result = cli + .call( + &[ + "transition", + "status", + &expected.namespace_id, + "--operation-id", + &expected.operation_id, + ], + READER, + ) + .await; + cli.close().await?; + let observed: RaftFleetTransitionStatus = serde_json::from_value(result?.success()?)?; + ensure!(observed == *expected); + Ok(()) +} + +pub(crate) async fn resume_transition( + fixture: &Fixture, + directory: &RangeDirectory, + principal: &str, + observer: &RaftCells, + request: &RaftFleetTransitionResume, +) -> Result { + let root = tempfile::tempdir()?; + let cli = open_cli(fixture, directory, root.path(), principal, Some(observer)).await?; + let plan = root.path().join("transition-resume.json"); + std::fs::write(&plan, serde_json::to_vec(request)?)?; + let path = plan.to_str().context("UTF-8 transition plan path")?; + let result = cli + .call( + &[ + "transition", + "resume", + &request.namespace_id, + "--plan", + path, + ], + OWNER, + ) + .await; + cli.close().await?; + Ok(serde_json::from_value(result?.success()?)?) +} + +pub(crate) async fn admit_transition( + fixture: &Fixture, + directory: &RangeDirectory, + principal: &str, + request: &RaftFleetTransitionAdmissionRequest, +) -> Result { + let root = tempfile::tempdir()?; + let cli = open_cli(fixture, directory, root.path(), principal, None).await?; + let plan = root.path().join("transition-admission.json"); + std::fs::write(&plan, serde_json::to_vec(request)?)?; + let path = plan + .to_str() + .context("UTF-8 transition admission plan path")?; + let command = ["transition", "admit", &request.namespace_id, "--plan", path]; + let result = async { + let basis: RaftFleetTransitionBasis = serde_json::from_value( + cli.call(&["transition", "basis", &request.namespace_id], READER) + .await? + .success()?, + )?; + ensure!( + basis.directory_digest == request.expected_directory_digest + && basis.fleet_revision >= request.expected_fleet_revision + ); + ensure!(cli.call(&command, READER).await?.code == 4); + Ok::<_, anyhow::Error>(serde_json::from_value( + cli.call(&command, OWNER).await?.success()?, + )?) + } + .await; + cli.close().await?; + result +} + +pub(crate) async fn authorize_transition_seal( + fixture: &Fixture, + directory: &RangeDirectory, + principal: &str, + request: &RaftFleetTransitionSealRequest, +) -> Result { + let root = tempfile::tempdir()?; + let cli = open_cli(fixture, directory, root.path(), principal, None).await?; + let plan = root.path().join("transition-seal-authorization.json"); + std::fs::write(&plan, serde_json::to_vec(request)?)?; + let command = [ + "transition", + "authorize-seal", + &request.namespace_id, + "--plan", + plan.to_str() + .context("UTF-8 transition authorization plan path")?, + ]; + let result = async { + ensure!(cli.call(&command, READER).await?.code == 4); + Ok::<_, anyhow::Error>(serde_json::from_value( + cli.call(&command, OWNER).await?.success()?, + )?) + } + .await; + cli.close().await?; + result +} + +pub(crate) async fn seal_transition_source( + fixture: &Fixture, + directory: &RangeDirectory, + principal: &str, + observer: &RaftCells, + request: &RaftFleetTransitionSourceSealRequest, +) -> Result { + let root = tempfile::tempdir()?; + let cli = open_cli(fixture, directory, root.path(), principal, Some(observer)).await?; + let result = async { + let plan = root.path().join("transition-seal-source.json"); + std::fs::write(&plan, serde_json::to_vec(request)?)?; + let command = [ + "transition", + "seal-source", + &request.namespace_id, + "--plan", + plan.to_str().context("UTF-8 source seal request path")?, + ]; + ensure!(cli.call(&command, READER).await?.code == 4); + tokio::time::timeout(TIMEOUT, async { + loop { + let reply = cli.call(&command, OWNER).await?; + if reply.code == 5 { + let failure: RaftFleetFailure = + serde_json::from_value(reply.value["error"].clone())?; + ensure!( + matches!( + failure.code, + RaftFleetFailureCode::Busy | RaftFleetFailureCode::Unavailable + ), + "unexpected source seal failure: {failure:?}" + ); + } else { + let sealed: RaftFleetTransitionSourceSeal = + serde_json::from_value(reply.success()?)?; + ensure!(&sealed.request == request); + if let Some(cut) = sealed.applied { + let replay: RaftFleetTransitionSourceSeal = + serde_json::from_value(cli.call(&command, OWNER).await?.success()?)?; + ensure!( + &replay.request == request && replay.applied.as_ref() == Some(&cut) + ); + return Ok(cut); + } + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .context("operator parent seal did not apply")? + } + .await; + cli.close().await?; + result +} + +pub(crate) async fn publish_transition( + fixture: &Fixture, + directory: &RangeDirectory, + principal: &str, + request: &RaftFleetTransitionPublishRequest, +) -> Result { + let root = tempfile::tempdir()?; + let cli = open_cli(fixture, directory, root.path(), principal, None).await?; + let plan = root.path().join("transition-publication.json"); + std::fs::write(&plan, serde_json::to_vec(request)?)?; + let path = plan + .to_str() + .context("UTF-8 transition publication plan path")?; + let command = [ + "transition", + "publish", + &request.namespace_id, + "--plan", + path, + ]; + let result = async { + ensure!(cli.call(&command, READER).await?.code == 4); + let value = cli.call(&command, OWNER).await?.success()?; + Ok::<_, anyhow::Error>(serde_json::from_value(value)?) + } + .await; + cli.close().await?; + result +} + +pub(crate) async fn activate_transition_child( + fixture: &Fixture, + directory: &RangeDirectory, + principal: &str, + observer: &RaftCells, + request: &RaftFleetTransitionActivationRequest, +) -> Result { + let root = tempfile::tempdir()?; + let cli = open_cli(fixture, directory, root.path(), principal, Some(observer)).await?; + let plan = root.path().join("transition-activation.json"); + std::fs::write(&plan, serde_json::to_vec(request)?)?; + let path = plan + .to_str() + .context("UTF-8 transition activation plan path")?; + let command = [ + "transition", + "activate", + &request.namespace_id, + "--plan", + path, + ]; + let result = async { + ensure!(cli.call(&command, READER).await?.code == 4); + tokio::time::timeout(TIMEOUT, async { + loop { + let reply = cli.call(&command, OWNER).await?; + if reply.code != 5 { + let applied: RaftFleetTransitionActivation = + serde_json::from_value(reply.success()?)?; + if applied.applied.is_some() { + return Ok::<_, anyhow::Error>(applied); + } + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .context("operator did not confirm the child activation")? + } + .await; + cli.close().await?; + result +} + +async fn exercise( + f: &Fixture, + cli: &OperatorCli, + root: &Path, + voter: &RaftCells, + setup: RaftManagedFirstGroupRequest, +) -> Result { + let history = setup.group.history; + let plan = first_group_plan(&cli.catalog, setup)?; + let namespace = &plan.namespace_id; + let url = &cli.url; + let confirmation_id = hex::encode([61; 16]); + let confirmation_revision = plan.expected_revision + 1; + let observed: RaftFleetFirstGroupStandby = serde_json::from_value( + cli.call(&["first-group", "standby", namespace], READER) + .await? + .success()?, + )?; + ensure!(observed.standby == plan.standby); + let plan_path = root.join("plan.json"); + std::fs::write(&plan_path, serde_json::to_vec(&plan)?)?; + let path = plan_path.to_str().context("UTF-8 plan path")?; + let reserve = ["first-group", "reserve", namespace, "--plan", path]; + let revision = confirmation_revision.to_string(); + let publish = [ + "first-group", + "publish", + namespace, + "--plan", + path, + "--request-id", + &confirmation_id, + "--expected-revision", + &revision, + ]; + + let archive_id = hex::encode([62; 16]); + let archive_revision = (confirmation_revision + 1).to_string(); + let archive = [ + "first-group", + "archive", + namespace, + "--plan", + path, + "--request-id", + &archive_id, + "--expected-revision", + &archive_revision, + ]; + let apply_path = root.join("setup.json"); + let mut setup = serde_json::json!({ + "reservation": plan, + "publication": { + "request_id": confirmation_id, + "expected_revision": confirmation_revision, + }, + "archive": { + "request_id": confirmation_id, + "expected_revision": confirmation_revision + 1, + }, + }); + let apply = [ + "first-group", + "apply", + namespace, + "--plan", + apply_path.to_str().context("UTF-8 setup path")?, + ]; + std::fs::write(&apply_path, serde_json::to_vec(&setup)?)?; + ensure!(cli.call(&apply, OWNER).await?.code == 2); + ensure!(cli.revision().await? == plan.expected_revision); + setup["archive"]["request_id"] = serde_json::Value::String(archive_id.clone()); + std::fs::write(&apply_path, serde_json::to_vec(&setup)?)?; + ensure!(cli.call(&archive, READER).await?.code == 4); + ensure!(cli.call(&archive, OWNER).await?.code == 3); + let denied = cli.call(&reserve, READER).await?; + ensure!(denied.code == 4); + let early = cli.call(&publish, OWNER).await?; + ensure!(early.code == 3); + ensure!(f.catalog.read(TIMEOUT).await?.revision() == plan.expected_revision); + ensure!(cli.revision().await? == plan.expected_revision); + + let client = reqwest::Client::builder() + .no_proxy() + .timeout(TIMEOUT) + .build()?; + let foreign_namespace = hex::encode([77; 16]); + for (fleet, namespace, status) in [ + ( + "foreign-fleet", + namespace.as_str(), + reqwest::StatusCode::FORBIDDEN, + ), + ( + FLEET, + foreign_namespace.as_str(), + reqwest::StatusCode::BAD_REQUEST, + ), + ] { + let response = client + .post(format!( + "{url}/api/v1/fleets/{fleet}/raft/namespaces/{namespace}/first-group/reserve" + )) + .bearer_auth(OWNER) + .json(&plan) + .send() + .await?; + ensure!( + response.status() == status, + "foreign fleet or namespace route" + ); + } + ensure!(f.catalog.read(TIMEOUT).await?.revision() == plan.expected_revision); + + let reservation = cli.call(&reserve, OWNER).await?.success()?; + ensure!(reservation["stage"] == "reserved"); + ensure!(cli.call(&reserve, OWNER).await?.success()? == reservation); + ensure!(f.catalog.read(TIMEOUT).await?.revision() == confirmation_revision); + + let mut forged_plan = plan.clone(); + forged_plan.group_id = hex::encode([125; 16]); + let forged = client + .post(format!( + "{url}/api/v1/fleets/{FLEET}/raft/namespaces/{namespace}/first-group/publish" + )) + .bearer_auth(OWNER) + .json(&RaftFleetFirstGroupPublish { + plan: forged_plan, + request_id: hex::encode([126; 16]), + expected_revision: confirmation_revision, + }) + .send() + .await?; + ensure!( + forged.status() == reqwest::StatusCode::CONFLICT, + "forged group confirmation: {}", + forged.status() + ); + let failure = forged.json::().await?; + ensure!(failure["error"]["code"] == "conflict"); + ensure!(cli.call(&reserve, OWNER).await?.success()? == reservation); + + let mut changed = plan.clone(); + changed.publication_id = hex::encode([126; 16]); + let changed_path = root.join("changed.json"); + std::fs::write(&changed_path, serde_json::to_vec(&changed)?)?; + let altered = cli + .call( + &[ + "first-group", + "reserve", + namespace, + "--plan", + changed_path.to_str().context("UTF-8 plan path")?, + ], + OWNER, + ) + .await?; + ensure!(altered.code == 3 && altered.value["error"]["code"] == "payload_conflict"); + ensure!(f.catalog.read(TIMEOUT).await?.revision() == confirmation_revision); + + f.store + .fail_fleet_confirmation + .store(true, Ordering::Release); + let lost = cli.call(&apply, OWNER).await?; + ensure!(lost.code == 5 && lost.value["error"]["outcome"] == "unknown"); + ensure!(lost.value["error"]["message"] + .as_str() + .is_some_and(|message| message.starts_with("First group publish stage."))); + ensure!(!f.store.fail_fleet_confirmation.load(Ordering::Acquire)); + let selected = cli.call(&publish, OWNER).await?.success()?; + ensure!(selected["stage"] == "published"); + let cut = f.catalog.read(TIMEOUT).await?; + ensure!(cut.revision() == confirmation_revision + 1); + + // Lose the fleet CAS reply and its first confirmation read after the + // archive intent commits. The active initializer must reconcile it. + f.store + .fail_fleet_confirmation + .store(true, Ordering::Release); + let archive_receipt = cli.call(&archive, OWNER).await?.success()?; + ensure!(!f.store.fail_fleet_confirmation.load(Ordering::Acquire)); + ensure!(!f.store.fail_fleet_read.load(Ordering::Acquire)); + ensure!(archive_receipt["stage"] == "archive_initialized"); + let writes = f.store.registry_writes.load(Ordering::Acquire); + ensure!(cli.call(&archive, OWNER).await?.success()? == archive_receipt); + let completed_revision = cli.revision().await?; + let replay = cli.call(&apply, OWNER).await?.success()?; + ensure!( + replay["reservation"] == reservation + && replay["publication"] == selected + && replay["archive"] == archive_receipt + ); + ensure!(cli.revision().await? == completed_revision); + ensure!(f.store.registry_writes.load(Ordering::Acquire) == writes); + let changed_id = hex::encode([63; 16]); + let mut competing = archive; + competing[6] = &changed_id; + let conflict = cli.call(&competing, OWNER).await?; + ensure!(conflict.code == 3 && conflict.value["error"]["code"] == "payload_conflict"); + + // The fixture owns its store prefix and no voter has attached yet. Remove + // its one registry while retaining the fleet intent and archive identity. + let registry = f + .store + .list(None) + .try_filter(|meta| { + let is_registry = meta.location.as_ref().contains("/cell-group-archives/") + && meta.location.as_ref().ends_with("/ownership/registry"); + std::future::ready(is_registry) + }) + .try_collect::>() + .await?; + let [registry] = registry.as_slice() else { + anyhow::bail!("one archive registry must exist before activation"); + }; + let key = ®istry.location; + let bytes = f.store.get(key).await?.bytes().await?; + f.store.delete(key).await?; + let replay = cli.call(&archive, OWNER).await?; + ensure!(replay.code == 5 && replay.value["error"]["outcome"] == "unknown"); + ensure!(matches!( + f.store.head(key).await, + Err(object_store::Error::NotFound { .. }) + )); + let readiness = voter + .start_managed_first_group(history, Duration::from_millis(250)) + .await; + ensure!(matches!(readiness, Err(CellGroupStartupError::Refused(_)))); + ensure!(voter.observe_node(TIMEOUT).await?.node.groups == 0); + ensure!(matches!( + f.store.head(key).await, + Err(object_store::Error::NotFound { .. }) + )); + f.store.put(key, bytes.into()).await?; + ensure!(cli.call(&archive, OWNER).await?.success()? == archive_receipt); + + let cut = f.catalog.read(TIMEOUT).await?; + ensure!(cut.revision() == confirmation_revision + 2); + Ok(f.catalog.authorize_group(&cut, &history, TIMEOUT).await?) +} diff --git a/crates/cellar/tests/support/cell_group_managed_fixture_seed_failure.rs b/crates/cellar/tests/support/cell_group_managed_fixture_seed_failure.rs new file mode 100644 index 000000000..743bd8445 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_managed_fixture_seed_failure.rs @@ -0,0 +1,548 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! S3 seed failure, successor and physical archive-debt oracles. + +use super::{movement::*, *}; +use cellar::application::consensus::verification::TransportError; + +pub async fn refuse_then_failover( + cluster: &ManagedCluster, + execution: &FleetMovementExecutionRequest, + source: usize, +) -> Result { + let catalog = &cluster.fixtures[0].catalog; + let target = &cluster.hosts[3]; + discover(cluster).await?; + let cut = catalog.read(TIMEOUT).await?; + let began = catalog + .reconcile_movement(&cut, execution, &cluster.hosts[source], TIMEOUT) + .await; + match began { + Ok(progress) => { + ensure!(progress.phase == FleetMovementPhase::Reserved && !progress.settled) + } + Err(FleetCatalogError::Unconfirmed) => { + // A missing response cannot prove that Begin did not commit. The + // following fresh quorum observation must establish its actual phase. + } + Err(error) => return Err(error).context("begin S3 refusal fixture"), + } + let current = cluster.hosts[source] + .group(cluster.history)? + .quorum_membership(TIMEOUT) + .await + .context("observe committed Seeding before S3 denial")?; + ensure!( + current + .membership() + .fleet_status() + .is_some_and(|status| status.phase + == Some(cellar::application::consensus::FleetPermitPhase::Seeding)), + "S3 refusal fixture did not reach committed Seeding" + ); + drop(current); + discover(cluster).await?; + let cut = catalog.read(TIMEOUT).await?; + let group = catalog + .authorize_group(&cut, &cluster.history, TIMEOUT) + .await?; + let permit = catalog + .authorize_membership(&cut, &cluster.history, TIMEOUT) + .await?; + let activation = LearnerActivation { + member: MemberId::new(4)?, + source: MemberId::new(source as u64 + 1)?, + source_node: cluster.fixtures[source].spec.node, + from_s3: true, + timeout: TIMEOUT, + }; + let store = &cluster.fixtures[3].store; + *store.registry_denial_on_payload_refusal.lock().unwrap() = + Some(cluster.fixtures[source].store.deny_registry_writes.clone()); + store + .deny_group_payload_reads + .store(true, Ordering::Release); + let refused = async { + for attempt in 0..8 { + let result = target + .activate_managed_learner_group(group.clone(), permit.clone(), activation.clone()) + .await; + match result { + Err(CellGroupStartupError::Unconfirmed(CellGroupControllerError::Archive(_))) => { + return Ok(()) + } + Err(CellGroupStartupError::Unconfirmed(CellGroupControllerError::Transport( + TransportError::Io(_) | TransportError::Timeout, + ))) + | Err(CellGroupStartupError::Unconfirmed(CellGroupControllerError::Runtime( + RuntimeError::DeadlineExceeded | RuntimeError::NotReady, + ))) => { + eprintln!( + "S3 denial attempt {attempt}: ambiguous response; exact admission retained" + ); + ensure!(target.resource_usage().host.controller.groups == 0); + ensure!( + catalog + .pending_reservations(&catalog.read(TIMEOUT).await?, TIMEOUT) + .await? + .len() + == 1 + ); + } + result => anyhow::bail!( + "S3 refusal changed classification or used peer bytes: {result:?}" + ), + } + } + anyhow::bail!("S3 denial never reached the physical receiver") + } + .await; + store + .deny_group_payload_reads + .store(false, Ordering::Release); + refused?; + ensure!(store.denied_group_payload_reads.load(Ordering::Acquire) > 0); + wait_for_seed_debt(&cluster.hosts[source], true) + .await + .context("failed seed retirement did not retain observable debt")?; + ensure!( + cluster.fixtures[source] + .store + .denied_registry_writes + .load(Ordering::Acquire) + > 0 + ); + let journals = seed_journals(&cluster.nodes[source])?; + let pending = journals + .iter() + .map(|p| seed_pending(p)) + .collect::>>()?; + ensure!(pending.into_iter().any(|pending| pending)); + cluster.fixtures[source] + .store + .deny_registry_writes + .store(false, Ordering::Release); + // No seed request follows the outage recovery. The node must discover and + // retire its abandoned generation through its own bounded cleanup owner. + wait_for_seed_debt(&cluster.hosts[source], false) + .await + .context("idle node did not retry abandoned seed retirement")?; + for journal in &journals { + ensure!( + !seed_pending(journal)?, + "seed intent was not durably retired" + ); + } + let denied = store.denied_group_payload_reads.load(Ordering::Acquire); + store + .deny_group_payload_reads + .store(true, Ordering::Release); + let second = target + .activate_managed_learner_group(group.clone(), permit.clone(), activation.clone()) + .await; + store + .deny_group_payload_reads + .store(false, Ordering::Release); + ensure!( + matches!( + second, + Err(CellGroupStartupError::Unconfirmed( + CellGroupControllerError::Archive(_) + )) + ), + "second physical S3 refusal changed: {second:?}" + ); + ensure!(store.denied_group_payload_reads.load(Ordering::Acquire) > denied); + wait_for_seed_debt(&cluster.hosts[source], true).await?; + ensure!(journals + .iter() + .map(|p| seed_pending(p)) + .collect::>>()? + .into_iter() + .any(|pending| pending)); + // Node inventory includes the retained staged engine mapping. Admission is + // proved independently by the controller and its absent runtime owner. + ensure!(target.observe_node(TIMEOUT).await?.node.groups == 1); + ensure!(target.resource_usage().host.controller.groups == 0); + ensure!( + target + .resource_usage() + .host + .controller + .pending_registrations + == 1 + ); + ensure!(matches!( + target.group(cluster.history)?.status().await, + Err(RuntimeError::NotFound) + )); + ensure!( + target + .resource_usage() + .host + .controller + .retained_preparations + == 1 + ); + ensure!( + catalog + .pending_reservations(&catalog.read(TIMEOUT).await?, TIMEOUT) + .await? + .len() + == 1 + ); + let fallback = target + .activate_managed_learner_group( + group, + permit, + LearnerActivation { + from_s3: false, + ..activation + }, + ) + .await; + ensure!( + matches!( + fallback, + Err(CellGroupStartupError::Unconfirmed( + CellGroupControllerError::Invalid("conflicting provision binding") + )) + ), + "a retained S3 learner changed transfer modes: {fallback:?}" + ); + ensure!(target.resource_usage().host.controller.groups == 0); + eprintln!("S3 payload refusal witnessed before attachment; peer fallback refused; preparation and debit retained"); + let group = cluster.hosts[source].group(cluster.history)?; + let status = group.replication_status(TIMEOUT).await?; + let next = (source + 2) % 3; + let transfer = LeadershipTransfer { + target: MemberId::new(next as u64 + 1)?, + expected_term: status.term, + expected_revision: status.membership_revision, + }; + ensure!( + matches!( + group.transfer_leadership(transfer, TIMEOUT).await, + Err(RuntimeError::NotReady) + ), + "active repair handoff guard changed" + ); + let report = cluster.hosts[source].close().await?; + ensure!(report.mutation_outcomes == MutationOutcomes::NoUnresolvedLocalProducers); + // The original source has joined with its journal still pending. Recover + // writes now so the surviving quorum can repair; source 0's fault wrapper + // also surrounds the shared backend. No stopped source can retry cleanup. + cluster.fixtures[source] + .store + .deny_registry_writes + .store(false, Ordering::Release); + let survivors = (0..3).filter(|i| *i != source).collect::>(); + let hosts = survivors + .iter() + .map(|i| cluster.hosts[*i].clone()) + .collect::>(); + let members = survivors.iter().map(|i| *i as u64 + 1).collect::>(); + let ready = current_leader(&hosts, &members, cluster.history).await; + if ready.is_err() { + // The readiness deadline has already failed. Bound observations before + // joined teardown so intermittent election/persistence failures retain + // their actual owner state without extending or rescuing the attempt. + for (i, host) in survivors.iter().zip(&hosts) { + let status = match host.group(cluster.history) { + Ok(group) => format!( + "{:?}", + tokio::time::timeout(Duration::from_millis(200), group.status()).await + ), + Err(error) => format!("{error:?}"), + }; + let usage = host.resource_usage(); + eprintln!("S3 failover readiness failed: stopped={source} live={i} status={status} controller={:?} runtime={:?} sql={:?} transport={:?}", + usage.host.controller, usage.runtime, usage.sql, usage.transport); + } + } + let ready = ready.context("fresh ready successor after S3 source-service close")?; + discover_survivors(cluster, Some(source)).await?; + let selected = survivors[ready]; + let cut = catalog.read(TIMEOUT).await?; + let group = catalog + .authorize_group(&cut, &cluster.history, TIMEOUT) + .await?; + let permit = catalog + .authorize_membership(&cut, &cluster.history, TIMEOUT) + .await?; + let retried = target + .activate_managed_learner_group( + group, + permit, + LearnerActivation { + member: MemberId::new(4)?, + source: MemberId::new(selected as u64 + 1)?, + source_node: cluster.fixtures[selected].spec.node, + from_s3: true, + timeout: TIMEOUT, + }, + ) + .await; + ensure!( + !matches!( + retried, + Err(CellGroupStartupError::Unconfirmed( + CellGroupControllerError::Invalid("conflicting provision binding") + )) + ), + "same boot/member/permit/mode refused a new source hint: {retried:?}" + ); + // The other retained initial voter remains in both configurations even if + // the temporary elected leader is the member this repair will remove. + Ok((source + 2) % 3) +} + +async fn wait_for_seed_debt(host: &RaftCells, pending: bool) -> Result<()> { + tokio::time::timeout(TIMEOUT, async { + loop { + let usage = host.resource_usage().host.controller; + if usage.active_archive_seeds == 0 && (usage.archive_seed_cleanup_errors > 0) == pending + { + return; + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await?; + Ok(()) +} + +fn seed_journals(node: &Node) -> Result> { + let root = node.info().data_dir.join("raft/cell-group-seed-archives"); + let journals = std::fs::read_dir(root)? + .map(|entry| entry.map(|entry| entry.path().join("state"))) + .collect::>>()?; + ensure!(!journals.is_empty(), "no physical seed journal"); + Ok(journals) +} + +fn seed_pending(path: &std::path::Path) -> Result { + let record: serde_json::Value = serde_json::from_slice(&std::fs::read(path)?)?; + let pending = record + .get("record") + .and_then(|r| r.get("pending")) + .context("seed journal pending record missing")?; + Ok(pending != "None") +} + +pub struct SeedDebt { + source: usize, + journals: Vec, + identities: Vec>, +} + +pub fn retained_seed_debt(cluster: &ManagedCluster) -> Result { + let sources = cluster.hosts[..3] + .iter() + .enumerate() + .filter(|(_, host)| host.is_closed()) + .map(|(i, _)| i) + .collect::>(); + ensure!( + sources.len() == 1, + "seed retirement outage did not select one source" + ); + let source = sources[0]; + ensure!(cluster.hosts[source].is_closed()); + let journals = seed_journals(&cluster.nodes[source])?; + let mut identities = Vec::new(); + for journal in &journals { + ensure!(seed_pending(journal)?, "source close discarded seed debt"); + identities.push(std::fs::read(journal.with_file_name("identity"))?); + } + Ok(SeedDebt { + source, + journals, + identities, + }) +} + +pub async fn restart_seed_cleanup(cluster: &ManagedCluster, debt: SeedDebt) -> Result<()> { + // Every old host has joined. Reopen only the source: cleanup must not need + // a live quorum, a learner request or new activation authority. + ensure!(cluster.hosts.iter().all(RaftCells::is_closed)); + let f = &cluster.fixtures[debt.source]; + f.store.deny_registry_writes.store(true, Ordering::Release); + let node = start_node( + cluster.roots[debt.source].path(), + f.application.clone(), + &f.spec.label, + ) + .await?; + let started = async { + let (mut config, services) = f + .config(&node, LogOpenMode::Resume, 94 + debt.source as u8) + .await?; + archives::configure(&mut config); + config.groups.push(RaftGroupAssignment { + history: cluster.history, + placement: RaftGroupPlacement::Replica, + }); + node.start_raft_cells(config, services) + .await + .map_err(anyhow::Error::from) + } + .await; + let host = match started { + Ok(host) => host, + Err(error) => { + let stopped = node.shutdown(ShutdownMode::Preserve).await; + return Err(error.context(format!( + "seed cleanup retained startup; shutdown={stopped:?}" + ))); + } + }; + let work = async { + // Inventory is available while the recovered engine is open. Closing + // seals its I/O admission, independently of cleanup's physical joins. + ensure!(host.resource_usage().retained_manifest_cells == Some(2)); + wait_for_seed_debt(&host, true) + .await + .context("restart did not discover existing seed debt")?; + for (journal, identity) in debt.journals.iter().zip(&debt.identities) { + ensure!(seed_pending(journal)?); + ensure!(&std::fs::read(journal.with_file_name("identity"))? == identity); + } + let cut = f.catalog.read(TIMEOUT).await?; + let gate = &f.store.group_archive_read; + gate.armed.store(true, Ordering::Release); + let release = ResumeSeedRead(gate); + tokio::time::timeout(TIMEOUT, gate.entered.notified()) + .await + .context("restart cleanup did not enter the actual archive read")?; + let closing_host = host.clone(); + let mut closing = tokio::spawn(async move { closing_host.close().await }); + ensure!( + tokio::time::timeout(Duration::from_millis(30), &mut closing) + .await + .is_err(), + "close released an entered seed cleanup read" + ); + ensure!(host.is_closed(), "close waiter did not seal admission"); + closing.abort(); + ensure!(matches!(closing.await, Err(error) if error.is_cancelled())); + let usage = host.resource_usage().host.controller; + ensure!(usage.active_archive_seed_cleanup == 1 && usage.s3_bytes >= 8192); + f.store.deny_registry_writes.store(false, Ordering::Release); + drop(release); + wait_for_seed_debt(&host, false) + .await + .context("restart did not retire seed debt without quorum")?; + for journal in &debt.journals { + ensure!(!seed_pending(journal)?); + } + ensure!( + f.catalog.read(TIMEOUT).await?.revision() == cut.revision(), + "seed cleanup changed fleet authority" + ); + Ok::<_, anyhow::Error>(()) + } + .await; + let stopped = close(vec![node], std::slice::from_ref(&host)).await; + match (work, stopped) { + (Ok(()), Ok(())) => Ok(()), + (Err(error), Ok(())) | (Ok(()), Err(error)) => Err(error), + (Err(error), Err(stopped)) => Err(anyhow::anyhow!( + "seed restart: {error:#}; joined shutdown: {stopped:#}" + )), + } +} + +struct ResumeSeedRead<'a>(&'a fault::RegistryGate); + +impl Drop for ResumeSeedRead<'_> { + fn drop(&mut self) { + self.0.armed.store(false, Ordering::Release); + self.0.resume.notify_one(); + } +} + +/// A surviving original voter is offline. Its retained physical/producer debt +/// cannot be released merely because repair finished on the remaining quorum. +pub async fn verify_blocked_settlement( + cluster: &ManagedCluster, + seed: &SeededCells, + execution: &FleetMovementExecutionRequest, + observer: &RaftCells, + obsolete: usize, +) -> Result<()> { + let observed = observer + .group(cluster.history)? + .quorum_membership(TIMEOUT) + .await?; + let configuration = observed.configuration(); + let mut voters = configuration.voters.clone(); + voters.sort_unstable(); + ensure!( + voters + == (1..=4) + .filter(|id| *id != obsolete as u64 + 1) + .collect::>() + ); + ensure!(configuration.voters_outgoing.is_empty() && configuration.learners.is_empty()); + let status = observed + .membership() + .fleet_status() + .context("repaired fleet status")?; + ensure!(status.phase == Some(cellar::application::consensus::FleetPermitPhase::Complete)); + drop(observed); + let catalog = &cluster.fixtures[0].catalog; + let cut = catalog.read(TIMEOUT).await?; + let pending = catalog.pending_reservations(&cut, TIMEOUT).await?; + ensure!(pending.len() == 1 && pending[0].reservation == execution.reservation); + ensure!(catalog.pending_movements(&cut, TIMEOUT).await?.len() == 1); + ensure!(!cluster.fixtures[0] + .store + .fail_fleet_confirmation + .load(Ordering::Acquire)); + let target = &cluster.hosts[3]; + ensure!(target.resource_usage().host.controller.groups == 1); + ensure!(target.resource_usage().retained_manifest_cells == Some(2)); + for (i, (cell, receipt)) in seed.cells.iter().zip(&seed.receipts).enumerate() { + let handle = target.cell(cell.clone())?; + let restored = handle + .query( + ProposalId::new([222 + i as u8; 16])?, + &handle.contract("Get")?, + vec![], + QueryOptions { + consistency: QueryConsistency::AtLeast(receipt.version_token.clone()), + routing: QueryRouting::Local, + timeout: TIMEOUT, + ..Default::default() + }, + ) + .await?; + ensure!( + restored.result == [7i64, 11][i].to_be_bytes() && restored.version == receipt.version + ); + ensure!( + restored.served_by == MemberId::new(4)? && restored.served_by_node == target.identity() + ); + } + let members = voters + .into_iter() + .filter(|id| !cluster.hosts[*id as usize - 1].is_closed()) + .collect::>(); + ensure!(members.len() == 2); + let live = members + .iter() + .map(|id| cluster.hosts[*id as usize - 1].clone()) + .collect::>(); + let ready = current_leader(&live, &members, cluster.history).await?; + verify_cells(&live[ready], seed, 2).await?; + ensure!( + catalog + .pending_reservations(&catalog.read(TIMEOUT).await?, TIMEOUT) + .await? + .len() + == 1 + ); + eprintln!("Fresh quorum completed S3-backed repair; local replacement images/versions and S3 writes verified; offline survivor inventory blocks catalog settlement and retains debit"); + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_managed_fixture_verifier.rs b/crates/cellar/tests/support/cell_group_managed_fixture_verifier.rs new file mode 100644 index 000000000..92b69737b --- /dev/null +++ b/crates/cellar/tests/support/cell_group_managed_fixture_verifier.rs @@ -0,0 +1,22 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Private replay inventory for the embedded managed fixture. +use super::*; +use cellar::application::consensus::verification::cell_groups::CellGroupArchiveTailLimits; + +pub async fn open( + fixture: &Fixture, + root: &Path, + rooted: bool, + tails: CellGroupArchiveTailLimits, +) -> Result { + super::archive_verifier::open( + &fixture.domain, + root, + rooted, + tails, + limits(), + checkpoint_limits(), + ) + .await +} diff --git a/crates/cellar/tests/support/cell_group_managed_isolation.rs b/crates/cellar/tests/support/cell_group_managed_isolation.rs new file mode 100644 index 000000000..837bc2661 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_managed_isolation.rs @@ -0,0 +1,414 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Quarantined obsolete voters can be replaced without a live inventory probe. +//! The fixture joins the complete Node before attesting isolation; it does not +//! qualify a deployment's permanent process/launch fencing mechanism. +use super::*; +use cellar::application::consensus::verification::FleetTable; + +fn intent(id: u8, revision: u64) -> FleetRequestIdentity { + let mut identity = request(id, revision); + identity.operation = [231; 16]; + identity +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn managed_isolated_voter_s3_replacement_retains_disk_and_fence() -> Result<()> { + for rooted in [false, true] { + Box::pin(isolated(rooted, Arc::new(fault::Store::default()))).await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[ignore = "requires explicit owned RustFS endpoint, bucket and isolated prefix"] +async fn rustfs_managed_isolated_voter_s3_replacement_retains_disk_and_fence() -> Result<()> { + for rooted in [false, true] { + let store = Arc::new(fault::Store::around(rustfs::store( + "managed-isolated-voter", + rooted, + )?)); + Box::pin(isolated(rooted, store)).await?; + } + Ok(()) +} + +#[path = "raft_retained_disk.rs"] +mod retained_disk; +pub(super) use retained_disk::disk_digest; + +async fn isolated(rooted: bool, store: Arc) -> Result<()> { + let mut cluster = Box::pin(ManagedCluster::start_with_store( + rooted, + &[7], + ArchiveSetup::Initialized, + store, + )) + .await?; + let mut stopped_index = None; + let work = async { + let seed = seed_cells(&cluster.hosts, cluster.history).await?; + discover(&cluster).await?; + let index = leader(&cluster.hosts[..3], cluster.history).await?; + let obsolete = (index + 1) % 3; + let failed = &cluster.fixtures[obsolete]; + let host = &cluster.hosts[obsolete]; + let boot = host + .observe_node(TIMEOUT) + .await? + .fleet_boot + .context("obsolete boot")?; + ensure!(host.resource_usage().retained_manifest_cells == Some(2)); + // Retain an authorization and listeners minted before quarantine. The + // startup path must recheck them, rather than treating them as a lease. + let (mut stale, services) = failed + .config(&cluster.nodes[obsolete], LogOpenMode::Resume, 90) + .await?; + archives::configure(&mut stale); + stale.groups.push(RaftGroupAssignment { + history: cluster.history, + placement: RaftGroupPlacement::Replica, + }); + let catalog = &cluster.fixtures[0].catalog; + let cut = catalog.read(TIMEOUT).await?; + let mut isolation = FleetNodeIsolationRequest { + identity: intent(221, cut.revision()), + attestation: FleetIsolationAttestation { + boot: boot.clone(), + evidence_digest: [92; 32], + evidence_reference: "fixture/joined-node-and-restart-disabled".into(), + }, + }; + ensure!(catalog + .attest_node_isolation(&cut, &isolation, TIMEOUT) + .await + .is_err()); + ensure!(catalog.read(TIMEOUT).await?.revision() == cut.revision()); + let node = cluster.nodes.remove(obsolete); + stopped_index = Some(obsolete); + ensure!(matches!( + node.shutdown(ShutdownMode::Preserve).await?, + NodeOutcome::Stopped(_) + )); + let closure = host.close().await?; + ensure!( + host.is_closed() + && closure.groups_total == 1 + && closure.mutation_outcomes == MutationOutcomes::NoUnresolvedLocalProducers + ); + let disk = cluster.roots[obsolete].path().join("raft"); + let retained = disk_digest(&disk)?; + let cut = catalog.read(TIMEOUT).await?; + catalog + .begin_evacuation( + &cut, + &intent(220, cut.revision()), + failed.spec.node, + std::slice::from_ref(&cluster.directory), + TIMEOUT, + ) + .await?; + let cut = catalog.read(TIMEOUT).await?; + isolation.identity.expected_revision = cut.revision(); + let mut wrong = isolation.clone(); + wrong.attestation.boot.process = [93; 16]; + ensure!(catalog + .attest_node_isolation(&cut, &wrong, TIMEOUT) + .await + .is_err()); + ensure!(catalog.read(TIMEOUT).await?.revision() == cut.revision()); + let accepted = catalog + .attest_node_isolation(&cut, &isolation, TIMEOUT) + .await?; + ensure!(accepted.boot() == &boot); + let cut = catalog.read(TIMEOUT).await?; + ensure!(catalog + .authorize_host(&cut, failed.spec.node, TIMEOUT) + .await + .is_err()); + let retry = start_node( + cluster.roots[obsolete].path(), + failed.application.clone(), + &failed.spec.label, + ) + .await?; + ensure!( + disk_digest(&disk)? == retained, + "quarantine changed a joined engine before startup" + ); + let attempt = retry.start_raft_cells(stale, services).await; + retry.shutdown(ShutdownMode::Preserve).await?; + let refusal = match attempt { + Ok(host) => { + host.close().await?; + anyhow::bail!("saved pre-isolation startup authority reopened the voter"); + } + Err(error) => error, + }; + ensure!( + refusal.chain().any(|error| matches!( + error.downcast_ref::(), + Some(FleetCatalogError::Conflict) + )), + "unexpected startup refusal: {refusal:#}" + ); + ensure!( + catalog.read(TIMEOUT).await?.revision() == cut.revision(), + "refused startup mutated the fleet root" + ); + let after = disk_digest(&disk)?; + let changed = retained + .keys() + .chain(after.keys()) + .filter(|path| retained.get(*path) != after.get(*path)) + .collect::>(); + ensure!( + after == retained, + "refused startup mutated the quarantined disk: {changed:?}" + ); + discover_survivors(&cluster, Some(obsolete)).await?; + let observer = &cluster.hosts[index]; + let membership = observer + .group(cluster.history)? + .quorum_membership(TIMEOUT) + .await?; + let target = cluster.hosts[3].observe_node(TIMEOUT).await?; + let cut = catalog.read(TIMEOUT).await?; + let movement = FleetMovementRequest { + resources: FleetResourceRequest { + identity: intent(210, cut.revision()), + group: cluster.history, + attestation: FleetResourceAttestation { + boot: target.fleet_boot.context("spare boot")?, + profile_revision: 1, + used: FleetResourceBounds { + engine_slots: 0, + durable_bytes: 1024 * 1024, + memory_bytes: 64 * 1024 * 1024, + }, + evidence_digest: [71; 32], + evidence_reference: "fixture-only/occupancy-attestation".into(), + }, + peak: FleetResourceBounds { + engine_slots: 1, + durable_bytes: target.node.profile.bytes_per_group, + memory_bytes: 8 * 1024 * 1024, + }, + }, + source: cluster.fixtures[index].spec.node, + plan: RepairPlan { + id: ProposalId::new([211; 16])?, + policy_revision: 1, + expected_revision: membership.membership().revision(), + obsolete: MemberId::new(obsolete as u64 + 1)?, + replacement: JoiningMember { + member: MemberId::new(4)?, + node: cluster.fixtures[3].spec.node, + disk: cluster.fixtures[3].spec.disk, + }, + }, + }; + drop(membership); + catalog + .reserve_movement( + &cut, + &movement, + observer, + std::slice::from_ref(&cluster.directory), + TIMEOUT, + ) + .await + .context("reserve isolated shared-group replacement")?; + let execution = FleetMovementExecutionRequest { + reservation: movement.resources.identity.id, + attestation: movement.resources.attestation.clone(), + from_s3: true, + }; + let expected = (1..=4) + .filter(|id| *id != obsolete as u64 + 1) + .collect::>(); + let mut completion = None; + let mut receipt = None; + let mut lost_reply = false; + for turn in 0..8 { + discover_survivors(&cluster, Some(obsolete)).await?; + let cut = catalog.read(TIMEOUT).await?; + let membership = observer + .group(cluster.history)? + .quorum_membership(TIMEOUT) + .await?; + let configuration = membership.configuration(); + let mut voters = configuration.voters.clone(); + voters.sort_unstable(); + let complete = voters == expected + && configuration.voters_outgoing.is_empty() + && configuration.learners.is_empty(); + drop(membership); + if complete { + if completion.is_none() { + completion = Some(FleetMovementCompletionRequest { + identity: intent(212, cut.revision()), + reservation: execution.reservation, + attestation: execution.attestation.clone(), + }); + cluster.fixtures[0] + .store + .fail_fleet_confirmation + .store(true, Ordering::Release); + } + match catalog + .complete_movement(&cut, completion.as_ref().unwrap(), observer, TIMEOUT) + .await + { + Ok(done) => { + receipt = Some(done); + break; + } + Err(error) => { + let root = catalog.read(TIMEOUT).await?; + if root.revision() == cut.revision() + 1 + && catalog + .pending_reservations(&root, TIMEOUT) + .await? + .is_empty() + { + lost_reply = true; + } else if matches!(error, FleetCatalogError::Unconfirmed) + && root.revision() == cut.revision() + && catalog.pending_reservations(&root, TIMEOUT).await?.len() == 1 + { + eprintln!( + "isolated settlement turn {turn}: Unconfirmed; debit retained" + ); + } else { + return Err(error) + .with_context(|| format!("isolated settlement turn {turn}")); + } + } + } + } else { + match catalog + .reconcile_movement(&cut, &execution, observer, TIMEOUT) + .await + { + Ok(progress) => ensure!(!progress.settled), + Err(FleetCatalogError::Unconfirmed) => { + eprintln!("isolated replacement turn {turn}: Unconfirmed; debit retained"); + } + Err(error) => { + return Err(error) + .with_context(|| format!("isolated replacement turn {turn}")) + } + } + ensure!( + catalog + .pending_reservations(&catalog.read(TIMEOUT).await?, TIMEOUT) + .await? + .len() + == 1 + ); + } + } + let receipt = receipt.context("isolated replacement did not settle within eight turns")?; + ensure!(lost_reply && receipt.isolated_sources() == [failed.spec.node.bytes()]); + let cut = catalog.read(TIMEOUT).await?; + let replay = catalog + .complete_movement(&cut, completion.as_ref().unwrap(), observer, TIMEOUT) + .await?; + ensure!( + replay.applied_revision() == receipt.applied_revision() + && replay.isolated_sources() == receipt.isolated_sources() + ); + ensure!(catalog.pending_movements(&cut, TIMEOUT).await?.is_empty()); + ensure!(catalog + .pending_reservations(&cut, TIMEOUT) + .await? + .is_empty()); + ensure!(catalog + .authorize_host(&cut, failed.spec.node, TIMEOUT) + .await + .is_err()); + // Settling a move grants no old-volume cleanup or terminal node removal. + ensure!( + disk_digest(&disk)? == retained, + "movement settlement reclaimed an isolated voter disk" + ); + ensure!( + catalog + .get( + &cut, + FleetTable::Operations, + &hex::encode(isolation.identity.operation), + TIMEOUT + ) + .await? + .is_some(), + "isolation settlement dropped the evacuation operation" + ); + let floor = catalog + .get( + &cut, + FleetTable::Operations, + &format!( + "resource-floor/{}", + hex::encode(cluster.fixtures[3].spec.node.bytes()) + ), + TIMEOUT, + ) + .await? + .context("replacement physical floor")?; + let floor: serde_json::Value = serde_json::from_slice(&floor[13..floor.len() - 32])?; + ensure!(floor["physical"]["engine_slots"] == 1); + ensure!(cluster.hosts[3].resource_usage().retained_manifest_cells == Some(2)); + ensure!( + cluster.fixtures[3] + .store + .group_payload_reads + .load(Ordering::Acquire) + > 0, + "replacement did not independently fetch S3 payloads" + ); + for (i, (cell, original)) in seed.cells.iter().zip(&seed.receipts).enumerate() { + let handle = cluster.hosts[3].cell(cell.clone())?; + let restored = handle + .query( + ProposalId::new([222 + i as u8; 16])?, + &handle.contract("Get")?, + vec![], + QueryOptions { + consistency: QueryConsistency::AtLeast(original.version_token.clone()), + routing: QueryRouting::Local, + timeout: TIMEOUT, + ..Default::default() + }, + ) + .await?; + ensure!( + restored.result == [7i64, 11][i].to_be_bytes() + && restored.version == original.version + ); + ensure!( + restored.served_by == MemberId::new(4)? + && restored.served_by_node == cluster.hosts[3].identity() + ); + } + verify_cells(observer, &seed, 2).await?; + Ok::<_, anyhow::Error>(()) + } + .await; + if let Some(index) = stopped_index { + cluster.hosts.remove(index); + } + let joined = close(std::mem::take(&mut cluster.nodes), &cluster.hosts).await; + cluster.fixtures[0].domain.shutdown().await; + if work.is_err() || joined.is_err() { + for root in std::mem::take(&mut cluster.roots) { + eprintln!( + "retained failed managed isolation disk: {}", + root.keep().display() + ); + } + } + joined?; + work +} diff --git a/crates/cellar/tests/support/cell_group_managed_lifecycle.rs b/crates/cellar/tests/support/cell_group_managed_lifecycle.rs new file mode 100644 index 000000000..1db0cff51 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_managed_lifecycle.rs @@ -0,0 +1,224 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Resource-backed whole-group replacement through the public managed APIs. +use super::*; +use crate::managed_fixture::movement::*; + +#[path = "cell_group_managed_discovery.rs"] +mod discovery; + +#[path = "cell_group_managed_router.rs"] +mod router; + +#[path = "cell_group_managed_cold_router.rs"] +mod cold_router; + +#[path = "cell_group_managed_controls.rs"] +mod controls; + +#[path = "cell_group_managed_s3.rs"] +mod s3; + +#[path = "cell_group_managed_isolation.rs"] +mod isolation; + +#[path = "cell_group_managed_recovery.rs"] +mod recovery; + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn managed_group_replacement_and_joined_restart() -> Result<()> { + for rooted in [false, true] { + Box::pin(replacement(rooted)).await?; + } + Ok(()) +} + +async fn replacement(rooted: bool) -> Result<()> { + Box::pin(replacement_with_source(rooted, false)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn managed_s3_group_replacement_and_joined_restart() -> Result<()> { + for rooted in [false, true] { + Box::pin(replacement_with_source(rooted, true)).await?; + } + Ok(()) +} + +async fn replacement_with_source(rooted: bool, from_s3: bool) -> Result<()> { + Box::pin(replacement_with_store( + rooted, + from_s3, + Arc::new(fault::Store::default()), + )) + .await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[ignore = "requires explicit owned RustFS endpoint, bucket and isolated prefix"] +async fn rustfs_managed_s3_group_replacement_and_joined_restart() -> Result<()> { + for rooted in [false, true] { + let store = Arc::new(fault::Store::around(rustfs::store( + "managed-s3-replacement", + rooted, + )?)); + Box::pin(replacement_with_store(rooted, true, store)).await?; + } + Ok(()) +} + +async fn replacement_with_store( + rooted: bool, + from_s3: bool, + store: Arc, +) -> Result<()> { + let mut cluster = Box::pin(ManagedCluster::start_with_store( + rooted, + &[7], + ArchiveSetup::Initialized, + store, + )) + .await?; + if from_s3 { + // Seed from retained startup, with the transport pool enabled and no + // periodic checkpoint policy. Fresh attachment uses a different path. + close(std::mem::take(&mut cluster.nodes), &cluster.hosts).await?; + cluster.hosts.clear(); + for (index, fixture) in cluster.fixtures.iter().enumerate() { + let node = start_node( + cluster.roots[index].path(), + fixture.application.clone(), + &fixture.spec.label, + ) + .await?; + let (mut config, services) = fixture + .config(&node, LogOpenMode::Resume, 34 + index as u8) + .await?; + archives::configure(&mut config); + ensure!( + config.shared.checkpoints.is_some() + && config.shared.controller.checkpoints.is_none() + ); + cluster.peers[index] = config.peers[0].clone(); + cluster + .hosts + .push(node.start_raft_cells(config, services).await?); + cluster.nodes.push(node); + } + for (index, fixture) in cluster.fixtures.iter().enumerate() { + admit_capacity( + fixture, + &cluster.hosts[index], + &cluster.directory, + 180 + 2 * index as u8, + ) + .await + .context("requalify the retained boot for repair")?; + } + } + let work = async { + let (seed, obsolete) = replace_with_source(&cluster, from_s3).await?; + drain_leader(&mut cluster, &seed, obsolete).await?; + Ok((seed, obsolete)) + } + .await; + // Even failed entered work must have a joined owner before this test exits. + let stopped = close(std::mem::take(&mut cluster.nodes), &cluster.hosts).await; + let (seed, obsolete) = match work { + Ok(value) => { + stopped?; + value + } + Err(error) => { + stopped?; + cluster.fixtures[0].domain.shutdown().await; + return Err(error); + } + }; + cluster.hosts.clear(); + // The replacement's retained disk is opened through the canonical joined + // member path, together with the two surviving initial members. + let mut members = Vec::new(); + for (i, fixture) in cluster.fixtures.iter().enumerate() { + if i == obsolete { + continue; + } + let node = start_node( + cluster.roots[i].path(), + fixture.application.clone(), + &fixture.spec.label, + ) + .await?; + let (mut config, services) = fixture + .config(&node, LogOpenMode::Resume, 44 + i as u8) + .await?; + archives::configure(&mut config); + cluster + .hosts + .push(node.start_raft_cells(config, services).await?); + cluster.nodes.push(node); + members.push(i as u64 + 1); + } + let resumed = async { + let index = current_leader(&cluster.hosts, &members, cluster.history).await?; + verify_cells(&cluster.hosts[index], &seed, 4).await?; + let catalog = &cluster.fixtures[0].catalog; + let cut = catalog.read(TIMEOUT).await?; + let group = catalog + .authorize_group(&cut, &cluster.history, TIMEOUT) + .await?; + let old_permit = catalog + .authorize_membership(&cut, &cluster.history, TIMEOUT) + .await?; + let target = members + .iter() + .position(|member| *member == 4) + .context("joined target")?; + let source = if index == target { + (target + 1) % members.len() + } else { + index + }; + let source_node = cluster.hosts[source] + .observe_node(TIMEOUT) + .await? + .node + .identity + .node; + let result = cluster.hosts[target] + .activate_managed_learner_group( + group, + old_permit, + LearnerActivation { + member: MemberId::new(4)?, + source: MemberId::new(members[source])?, + source_node, + from_s3: false, + timeout: TIMEOUT, + }, + ) + .await; + ensure!( + matches!( + result, + Err(CellGroupStartupError::Refused( + CellGroupControllerError::Invalid("managed learner admission") + )) + ), + "old-boot learner grant was not refused before submission: {result:?}" + ); + ensure!(catalog.read(TIMEOUT).await?.revision() == cut.revision()); + for host in &cluster.hosts { + ensure!(host.observe_node(TIMEOUT).await?.node.groups == 1); + ensure!(host.resource_usage().retained_manifest_cells == Some(2)); + } + Ok(()) + } + .await; + close(std::mem::take(&mut cluster.nodes), &cluster.hosts).await?; + cluster.fixtures[0].domain.shutdown().await; + resumed +} + +#[path = "cell_group_managed_maintenance.rs"] +mod maintenance; diff --git a/crates/cellar/tests/support/cell_group_managed_maintenance.rs b/crates/cellar/tests/support/cell_group_managed_maintenance.rs new file mode 100644 index 000000000..1eb9a5850 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_managed_maintenance.rs @@ -0,0 +1,46 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! A repaired member returns only through rooted maintenance and fresh resume. +use super::*; +pub(super) use crate::managed_fixture::maintenance::handoff_to; +use crate::managed_fixture::maintenance::maintained_with_loss; +#[path = "cell_group_managed_maintenance_loss.rs"] +mod loss; + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn managed_group_maintenance_stop_return_and_resume() -> Result<()> { + for rooted in [false, true] { + for joined in [false, true] { + Box::pin(maintained(rooted, joined)).await?; + } + } + Ok(()) +} + +async fn maintained(rooted: bool, joined: bool) -> Result<()> { + Box::pin(maintained_with_store( + rooted, + joined, + Arc::new(fault::Store::default()), + )) + .await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[ignore = "requires explicit owned RustFS endpoint, bucket and isolated prefix"] +async fn rustfs_managed_archive_maintenance_stop_return_and_resume() -> Result<()> { + for rooted in [false, true] { + for joined in [false, true] { + let store = Arc::new(fault::Store::around(rustfs::store( + &format!("managed-archive-maintenance-joined-{joined}"), + rooted, + )?)); + Box::pin(maintained_with_store(rooted, joined, store)).await?; + } + } + Ok(()) +} + +async fn maintained_with_store(rooted: bool, joined: bool, store: Arc) -> Result<()> { + Box::pin(maintained_with_loss(rooted, joined, store, false)).await +} diff --git a/crates/cellar/tests/support/cell_group_managed_maintenance_loss.rs b/crates/cellar/tests/support/cell_group_managed_maintenance_loss.rs new file mode 100644 index 000000000..6d31fa606 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_managed_maintenance_loss.rs @@ -0,0 +1,31 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! A lost read-only observation cannot consume the complete renewal budget. +use super::*; + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn managed_maintenance_recovers_one_lost_membership_observation() -> Result<()> { + for rooted in [false, true] { + Box::pin(maintained_with_loss( + rooted, + true, + Arc::new(fault::Store::default()), + true, + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[ignore = "requires explicit owned RustFS endpoint, bucket and isolated prefix"] +async fn rustfs_managed_maintenance_recovers_one_lost_membership_observation() -> Result<()> { + for rooted in [false, true] { + let store = Arc::new(fault::Store::around(rustfs::store( + "maintenance-lost-observation", + rooted, + )?)); + Box::pin(maintained_with_loss(rooted, true, store, true)).await?; + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_managed_recovery.rs b/crates/cellar/tests/support/cell_group_managed_recovery.rs new file mode 100644 index 000000000..f81f9c1f9 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_managed_recovery.rs @@ -0,0 +1,560 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! The Node-owned failure service drives shared-group repair and terminal S3 +//! verification. Platform/occupancy assertions are fixtures, not Docker fences. +use super::*; +use cellar::application::consensus::verification::cell_groups::RangeDirectory; +use std::{ + future::Future, + pin::Pin, + sync::{ + atomic::{AtomicBool, AtomicUsize}, + Mutex, + }, +}; + +type CatalogResult = std::result::Result; + +struct Callback(Arc); + +impl Drop for Callback { + fn drop(&mut self) { + self.0.fetch_sub(1, Ordering::AcqRel); + } +} + +struct Platform { + catalog: FleetCatalog, + directory: RangeDirectory, + boot: FleetBootBinding, + stopped: RaftCells, + hold: AtomicBool, + active: Arc, + entered: tokio::sync::Notify, + contexts: Mutex>, +} + +impl FleetFailureRecoveryPlatform for Platform { + fn namespaces<'a>( + &'a self, + _: Duration, + ) -> Pin>> + Send + 'a>> { + Box::pin(async move { Ok(vec![self.directory.clone()]) }) + } + + fn isolate<'a>( + &'a self, + context: &'a FleetFailureRecoveryContext, + timeout: Duration, + ) -> Pin> + Send + 'a>> { + Box::pin(async move { + if context.boot() != &self.boot || !self.stopped.is_closed() { + return Err(FleetCatalogError::Invalid( + "fixture isolation boot not joined", + )); + } + let cut = self.catalog.read(timeout).await?; + let admission = self + .catalog + .evacuation_admission(&cut, context.operation(), timeout) + .await?; + if admission.node().bytes() != self.boot.node { + return Err(FleetCatalogError::Invalid( + "fixture isolation before exclusion", + )); + } + { + let mut contexts = self.contexts.lock().unwrap(); + if contexts.last().is_some_and(|old| old != context) { + return Err(FleetCatalogError::Conflict); + } + contexts.push(context.clone()); + } + self.active.fetch_add(1, Ordering::AcqRel); + let _callback = Callback(self.active.clone()); + self.entered.notify_one(); + while self.hold.load(Ordering::Acquire) { + cellar::asyncrt::sleep(Duration::from_millis(10)).await; + } + Ok(FleetIsolationAttestation { + boot: self.boot.clone(), + evidence_digest: [92; 32], + evidence_reference: "fixture/complete-node-joined-and-restart-disabled".into(), + }) + }) + } +} + +struct Inputs { + hosts: Vec, + directory: RangeDirectory, + history: GroupHistory, + failed: FleetBootBinding, + calls: AtomicUsize, +} + +fn occupancy(view: RaftHostObservation) -> CatalogResult { + Ok(FleetResourceAttestation { + boot: view.fleet_boot.ok_or(FleetCatalogError::Unconfirmed)?, + profile_revision: 1, + used: FleetResourceBounds { + engine_slots: view.node.groups, + durable_bytes: 1024 * 1024, + memory_bytes: 64 * 1024 * 1024, + }, + evidence_digest: [73; 32], + evidence_reference: "fixture-only/automatic-group-repair-occupancy".into(), + }) +} + +impl FleetIsolatedRepairEvidence for Inputs { + fn refresh<'a>( + &'a self, + context: &'a FleetIsolatedRepairContext, + timeout: Duration, + ) -> Pin> + Send + 'a>> { + Box::pin(async move { + if context.group != self.history || context.failed_boot != self.failed { + return Err(FleetCatalogError::Conflict); + } + self.calls.fetch_add(1, Ordering::AcqRel); + let mut nodes = Vec::new(); + let mut bytes = 0; + for host in &self.hosts { + let view = host + .observe_node(timeout) + .await + .map_err(|_| FleetCatalogError::Unconfirmed)?; + bytes = bytes.max(view.node.profile.bytes_per_group); + nodes.push(FleetTopologyNodeResources { + attestation: occupancy(view)?, + reserve: FleetResourceBounds { + engine_slots: 0, + durable_bytes: 0, + memory_bytes: 8 * 1024 * 1024, + }, + }); + } + let bounds = FleetResourceBounds { + engine_slots: 1, + durable_bytes: bytes, + memory_bytes: 8 * 1024 * 1024, + }; + Ok(FleetIsolatedRepairInputs { + history: FleetTopologyHistoryResources { + group: self.history, + resident: bounds, + peak: bounds, + }, + nodes, + namespaces: vec![self.directory.clone()], + }) + }) + } +} + +impl FleetMovementEvidence for Inputs { + fn refresh<'a>( + &'a self, + movement: &'a FleetPendingMovement, + timeout: Duration, + ) -> Pin> + Send + 'a>> + { + Box::pin(async move { + let host = self + .hosts + .iter() + .find(|host| host.identity().node == movement.plan.replacement.node) + .ok_or(FleetCatalogError::InventoryIncomplete)?; + Ok(FleetMovementExecutionRequest { + reservation: movement.reservation, + attestation: occupancy( + host.observe_node(timeout) + .await + .map_err(|_| FleetCatalogError::Unconfirmed)?, + )?, + from_s3: true, + }) + }) + } +} + +fn options() -> FleetMovementServiceOptions { + FleetMovementServiceOptions { + interval: Duration::from_millis(50), + retry_max_interval: Duration::from_millis(500), + turn_timeout: TIMEOUT, + } +} + +async fn start( + catalog: &FleetCatalog, + observer: &RaftCells, + inputs: &Arc, + platform: &Arc, + verifier: &FleetGroupArchiveVerifier, +) -> Result { + Ok(catalog + .start_failure_recovery_service( + observer.clone(), + inputs.clone(), + inputs.clone(), + platform.clone(), + options(), + FleetFailureRecoveryOptions { + detection: FleetFailureDetectionOptions::default(), + archive: verifier.clone(), + }, + ) + .await?) +} + +async fn wait( + service: &FleetMovementService, + ready: impl Fn(&FleetMovementServiceStatus) -> bool, +) -> Result<()> { + // Keep the existing automatic fleet fixture's sixteen five-second round + // envelope. Individual calls, failure grace and service turns stay bounded. + tokio::time::timeout(TIMEOUT * 16, async { + loop { + let status = service.status(); + if ready(&status) { + return Ok::<_, anyhow::Error>(()); + } + ensure!(status.running, "failure recovery stopped: {status:?}"); + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .with_context(|| format!("automatic group recovery deadline: {:?}", service.status()))??; + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn managed_failure_recovery_restarts_and_verifies_s3_before_removal() -> Result<()> { + let _ = tracing_subscriber::fmt() + .with_env_filter("cellar::raft::fleet_catalog::resources::execution::service::worker=debug") + .with_test_writer() + .try_init(); + for rooted in [false, true] { + Box::pin(recover(rooted, Arc::new(fault::Store::default()))).await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[ignore = "requires explicit owned RustFS endpoint, bucket and isolated prefix"] +async fn rustfs_managed_failure_recovery_restarts_and_verifies_s3_before_removal() -> Result<()> { + for rooted in [false, true] { + let store = Arc::new(fault::Store::around(rustfs::store( + "managed-failure-recovery", + rooted, + )?)); + Box::pin(recover(rooted, store)).await?; + } + Ok(()) +} + +async fn recover(rooted: bool, store: Arc) -> Result<()> { + let mut cluster = Box::pin(ManagedCluster::start_with_store( + rooted, + &[7], + ArchiveSetup::Initialized, + store, + )) + .await?; + let mut stopped_index = None; + let mut service = None; + let replay_root = tempfile::tempdir()?; + let work = async { + let seed = seed_cells(&cluster.hosts, cluster.history).await?; + discover(&cluster).await?; + let index = leader(&cluster.hosts[..3], cluster.history).await?; + let obsolete = (index + 1) % 3; + let observer = &cluster.hosts[index]; + // Establish an explicit complete S3 cut before failing a voter; do not + // execute new business mutations to drive the subsequent repair. + for (i, original) in seed.receipts.iter().enumerate() { + let handle = observer.cell(seed.cells[i].clone())?; + let mut options = CommandOptions::new(original.id, seed.retry); + options.durability = Durability::S3; + let receipt = handle + .command( + &handle.contract("Add")?, + [7i64, 11][i].to_be_bytes().to_vec(), + options, + ) + .await?; + ensure!( + receipt.result == original.result + && receipt.version == original.version + && receipt.durability == Durability::S3 + ); + } + let boot = cluster.hosts[obsolete] + .observe_node(TIMEOUT) + .await? + .fleet_boot + .context("failed boot")?; + let node = cluster.nodes.remove(obsolete); + stopped_index = Some(obsolete); + ensure!(matches!( + node.shutdown(ShutdownMode::Preserve).await?, + NodeOutcome::Stopped(_) + )); + let closure = cluster.hosts[obsolete].close().await?; + ensure!( + closure.groups_total == 1 + && closure.mutation_outcomes == MutationOutcomes::NoUnresolvedLocalProducers + ); + let disk = cluster.roots[obsolete].path().join("raft"); + let retained = isolation::disk_digest(&disk)?; + let f = &cluster.fixtures[0]; + let independent = Arc::new(fault::Store::around(f.store.clone())); + let namespace = f + .domain + .run( + Catalog::new( + independent.clone(), + CatalogOptions { + prefix: "managed/".into(), + ..Default::default() + }, + )? + .existing_namespace(f.declaration.namespace.clone()), + ) + .await??; + let verifier_directory = RangeDirectory::new(namespace, DirectoryLimits::default())?; + let verifier = crate::managed_fixture::verifier::open( + f, + replay_root.path(), + rooted, + Default::default(), + ) + .await?; + independent + .deny_group_payload_reads + .store(true, Ordering::Release); + let platform = Arc::new(Platform { + catalog: f.catalog.clone(), + directory: verifier_directory, + boot: boot.clone(), + stopped: cluster.hosts[obsolete].clone(), + hold: AtomicBool::new(true), + active: Arc::new(AtomicUsize::new(0)), + entered: tokio::sync::Notify::new(), + contexts: Mutex::new(Vec::new()), + }); + let inputs = Arc::new(Inputs { + hosts: cluster + .hosts + .iter() + .enumerate() + .filter(|(i, _)| *i != obsolete) + .map(|(_, host)| host.clone()) + .collect(), + directory: cluster.directory.clone(), + history: cluster.history, + failed: boot.clone(), + calls: AtomicUsize::new(0), + }); + let registered = peer_fixture(f, 8).await?; + let registered_root = tempfile::tempdir()?; + let registered_node = start_node( + registered_root.path(), + registered.application.clone(), + ®istered.spec.label, + ) + .await?; + let (config, services) = registered + .config(®istered_node, LogOpenMode::Create, 42) + .await?; + let inspection = registered_node.start_raft_cells(config, services).await?; + let revision = f.catalog.read(TIMEOUT).await?.revision(); + let baseline = inspection.resource_usage().control; + let refused = start(&f.catalog, &inspection, &inputs, &platform, &verifier).await; + let refused = match refused { + Ok(worker) => { + worker.shutdown().await; + false + } + Err(error) => matches!( + error.downcast_ref::(), + Some(FleetCatalogError::Conflict) + ), + }; + let clean = inspection.resource_usage().control == baseline; + registered_node.shutdown(ShutdownMode::Preserve).await?; + ensure!(refused && clean && f.catalog.read(TIMEOUT).await?.revision() == revision); + ensure!( + platform.contexts.lock().unwrap().is_empty(), + "registered boot entered isolation" + ); + let baseline = observer.resource_usage().control; + service = Some(start(&f.catalog, observer, &inputs, &platform, &verifier).await?); + tokio::time::timeout(TIMEOUT * 4, platform.entered.notified()) + .await + .context("failure detector did not enter rooted platform isolation")?; + let context = platform + .contexts + .lock() + .unwrap() + .last() + .cloned() + .context("rooted failure context")?; + ensure!(context.boot() == &boot && platform.active.load(Ordering::Acquire) == 1); + let charged = observer.resource_usage().control; + ensure!( + charged.items == baseline.items + 1 + && charged.bytes > baseline.bytes + && charged.bytes <= charged.max_bytes + && charged.max_items == baseline.max_items + && charged.max_bytes == baseline.max_bytes + ); + let duplicate = start(&f.catalog, observer, &inputs, &platform, &verifier).await; + ensure!(matches!( + duplicate + .err() + .and_then(|e| e.downcast::().ok()), + Some(FleetCatalogError::Overloaded) + )); + ensure!(observer.resource_usage().control == charged); + ensure!( + inputs.calls.load(Ordering::Acquire) == 0, + "repair ran before platform isolation completed" + ); + service.take().unwrap().shutdown().await; + ensure!(observer.resource_usage().control == baseline); + ensure!( + platform.active.load(Ordering::Acquire) == 0, + "shutdown did not join the isolation callback" + ); + let cut = f.catalog.read(TIMEOUT).await?; + ensure!( + f.catalog + .evacuation_admission(&cut, context.operation(), TIMEOUT) + .await? + .node() + .bytes() + == boot.node + ); + ensure!(f.catalog.pending_movements(&cut, TIMEOUT).await?.is_empty()); + platform.hold.store(false, Ordering::Release); + service = Some(start(&f.catalog, observer, &inputs, &platform, &verifier).await?); + wait(service.as_ref().unwrap(), |status| { + status + .failure_recovery + .as_ref() + .is_some_and(|r| r.phase == FleetFailureRecoveryPhase::VerifyingFinalization) + && independent + .denied_group_payload_reads + .load(Ordering::Acquire) + > 0 + }) + .await?; + let cut = f.catalog.read(TIMEOUT).await?; + ensure!(f + .catalog + .finalization_receipt(&cut, context.operation(), TIMEOUT) + .await? + .is_none()); + ensure!(f + .catalog + .isolated_removal_receipt(&cut, context.operation(), TIMEOUT) + .await? + .is_none()); + ensure!( + f.catalog + .pending_reservations(&cut, TIMEOUT) + .await? + .is_empty(), + "completed repair retained the active debit" + ); + ensure!(cluster.hosts[3].resource_usage().retained_manifest_cells == Some(2)); + ensure!(isolation::disk_digest(&disk)? == retained); + service.take().unwrap().shutdown().await; + ensure!(observer.resource_usage().control == baseline); + let payload_reads = independent.group_payload_reads.load(Ordering::Acquire); + let isolation_calls = platform.contexts.lock().unwrap().len(); + ensure!(isolation_calls >= 2 && inputs.calls.load(Ordering::Acquire) > 0); + independent + .deny_group_payload_reads + .store(false, Ordering::Release); + service = Some(start(&f.catalog, observer, &inputs, &platform, &verifier).await?); + wait(service.as_ref().unwrap(), |status| { + status.failure_recovery.as_ref().is_some_and(|r| { + r.phase == FleetFailureRecoveryPhase::Removed && r.removed_revision.is_some() + }) + }) + .await?; + ensure!( + platform.contexts.lock().unwrap().len() == isolation_calls, + "restart repeated a committed platform isolation" + ); + let status = service + .as_ref() + .unwrap() + .status() + .failure_recovery + .context("terminal failure recovery status")?; + ensure!(status.context == context); + service.take().unwrap().shutdown().await; + ensure!(observer.resource_usage().control == baseline); + let cut = f.catalog.read(TIMEOUT).await?; + let removed = f + .catalog + .isolated_removal_receipt(&cut, context.operation(), TIMEOUT) + .await? + .context("journal-confirmed isolated removal")?; + ensure!( + removed.node() == boot.node + && Some(removed.applied_revision()) == status.removed_revision + ); + ensure!(removed.isolation_digest() != [0; 32] && removed.candidate_digest() != [0; 32]); + ensure!(f + .catalog + .authorize_host(&cut, cluster.fixtures[obsolete].spec.node, TIMEOUT) + .await + .is_err()); + ensure!( + f.catalog + .pending_reservations(&cut, TIMEOUT) + .await? + .is_empty() + && f.catalog.pending_movements(&cut, TIMEOUT).await?.is_empty() + ); + ensure!( + isolation::disk_digest(&disk)? == retained, + "terminal removal reclaimed the old engine" + ); + ensure!( + independent.group_payload_reads.load(Ordering::Acquire) > payload_reads, + "terminal proof borrowed serving state instead of S3 replay" + ); + verify_cells(observer, &seed, 2).await?; + Ok::<_, anyhow::Error>(()) + } + .await; + if let Some(service) = service { + service.shutdown().await; + } + if let Some(index) = stopped_index { + cluster.hosts.remove(index); + } + let joined = close(std::mem::take(&mut cluster.nodes), &cluster.hosts).await; + cluster.fixtures[0].domain.shutdown().await; + if work.is_err() || joined.is_err() { + for root in std::mem::take(&mut cluster.roots) { + eprintln!( + "retained failed automatic recovery disk: {}", + root.keep().display() + ); + } + eprintln!( + "retained independent recovery verifier: {}", + replay_root.keep().display() + ); + } + joined?; + work +} diff --git a/crates/cellar/tests/support/cell_group_managed_registry.rs b/crates/cellar/tests/support/cell_group_managed_registry.rs new file mode 100644 index 000000000..98d83dd43 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_managed_registry.rs @@ -0,0 +1,162 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Declaration registry replay and corruption oracles on the managed fixture. +use super::*; +use cellar::application::consensus::verification::{ + FleetChange, FleetMutation, FleetTable, NamespaceCatalog, +}; +use object_store::{path::Path as ObjectPath, ObjectStore}; + +async fn open_namespace(f: &Fixture, id: u8, prefix: &str) -> Result { + let mut config = f.declaration.namespace.clone(); + config.namespace = NamespaceId::new([id; 16])?; + let catalog = Catalog::new( + f.store.clone(), + CatalogOptions { + prefix: prefix.into(), + ..Default::default() + }, + )?; + Ok(f.domain.run(catalog.open_namespace(config)).await??) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn fleet_inventory_registry_retries_and_scope() -> Result<()> { + let f = Fixture::new(false).await?; + let result = async { + let other = open_namespace(&f, 88, "managed/").await?; + let root = f.catalog.read(TIMEOUT).await?; + let request = request(90, root.revision()); + f.store.fail_fleet_after.store(true, Ordering::Release); + f.store + .fail_fleet_confirmation + .store(true, Ordering::Release); + ensure!(f + .catalog + .declare_namespace(&root, &request, &other, "Counter", TIMEOUT) + .await + .is_err()); + f.store.fail_fleet_after.store(false, Ordering::Release); + f.store + .fail_fleet_confirmation + .store(false, Ordering::Release); + let third = open_namespace(&f, 89, "managed/").await?; + let later = f.catalog.read(TIMEOUT).await?; + f.catalog + .declare_namespace( + &later, + &super::request(91, later.revision()), + &third, + "Counter", + TIMEOUT, + ) + .await?; + let reopened = + Fixture::open_catalog(&f.domain, f.store.clone(), f.declaration.namespace.cluster)?; + let now = reopened.read(TIMEOUT).await?; + let reply = reopened + .declare_namespace(&now, &request, &other, "Counter", TIMEOUT) + .await?; + ensure!( + reply.applied_revision < now.revision() && reply.snapshot.revision() == now.revision() + ); + ensure!(matches!( + reopened + .declare_namespace(&now, &request, &other, "Different", TIMEOUT) + .await, + Err(FleetCatalogError::PayloadConflict) + )); + let conflict = super::request(92, now.revision()); + ensure!(matches!( + reopened + .declare_namespace(&now, &conflict, &other, "Different", TIMEOUT) + .await, + Err(FleetCatalogError::Conflict) + )); + let wrong = open_namespace(&f, 88, "foreign-data/").await?; + ensure!(matches!( + reopened + .declare_namespace(&now, &conflict, &wrong, "Counter", TIMEOUT) + .await, + Err(FleetCatalogError::Conflict) + )); + Ok::<_, anyhow::Error>(()) + } + .await; + f.domain.shutdown().await; + result +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn fleet_inventory_registry_no_reset() -> Result<()> { + let f = Fixture::new(false).await?; + let result = async { + let root = f.catalog.read(TIMEOUT).await?; + let original = f + .catalog + .get(&root, FleetTable::Policy, "compiled-namespaces", TIMEOUT) + .await? + .context("registry")?; + let request = request(90, root.revision()); + let deletion = FleetMutation { + id: request.id, + operation: request.operation, + principal: request.principal, + expected_revision: request.expected_revision, + changes: vec![FleetChange { + table: FleetTable::Policy, + key: "compiled-namespaces".into(), + value: None, + }], + }; + ensure!(matches!( + f.catalog.commit(&root, &deletion, TIMEOUT).await, + Err(FleetCatalogError::Invalid(_)) + )); + ensure!(f.catalog.read(TIMEOUT).await?.revision() == root.revision()); + let value: serde_json::Value = serde_json::from_slice(&original[13..original.len() - 32])?; + let index: Vec = serde_json::from_value(value["index"].clone())?; + let page = ObjectPath::from(format!( + "tenant/hosts/raft/{}/fleet/objects/{}", + hex::encode(f.declaration.namespace.cluster.bytes()), + hex::encode(index) + )); + let saved = f.store.inner.get(&page).await?.bytes().await?; + f.store.inner.delete(&page).await?; + let other = open_namespace(&f, 88, "managed/").await?; + let request = super::request(91, root.revision()); + ensure!(matches!( + f.catalog + .declare_namespace(&root, &request, &other, "Counter", TIMEOUT) + .await, + Err(FleetCatalogError::Corrupt(_)) + )); + ensure!(f.catalog.read(TIMEOUT).await?.revision() == root.revision()); + f.store.inner.put(&page, saved.into()).await?; + f.catalog + .declare_namespace(&root, &request, &other, "Counter", TIMEOUT) + .await?; + let root = f.catalog.read(TIMEOUT).await?; + let request = super::request(92, root.revision()); + let rollback = FleetMutation { + id: request.id, + operation: request.operation, + principal: request.principal, + expected_revision: request.expected_revision, + changes: vec![FleetChange { + table: FleetTable::Policy, + key: "compiled-namespaces".into(), + value: Some(original), + }], + }; + ensure!(matches!( + f.catalog.commit(&root, &rollback, TIMEOUT).await, + Err(FleetCatalogError::Invalid(_)) + )); + ensure!(f.catalog.read(TIMEOUT).await?.revision() == root.revision()); + Ok::<_, anyhow::Error>(()) + } + .await; + f.domain.shutdown().await; + result +} diff --git a/crates/cellar/tests/support/cell_group_managed_request_cases.rs b/crates/cellar/tests/support/cell_group_managed_request_cases.rs new file mode 100644 index 000000000..bce739e35 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_managed_request_cases.rs @@ -0,0 +1,24 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Managed test entry points shared with the public acceptance target. + +use super::*; +use crate::managed_fixture::requests::{request_path, run_native}; + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn managed_native_axum_read_write_and_retained_restart() -> Result<()> { + Box::pin(run_native()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[ignore = "requires explicit owned RustFS endpoint, bucket and isolated prefix"] +async fn rustfs_managed_native_axum_read_write_and_retained_restart() -> Result<()> { + for rooted in [false, true] { + Box::pin(request_path( + rooted, + Arc::new(fault::Store::around(rustfs::store("managed-http", rooted)?)), + )) + .await?; + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_managed_request_http.rs b/crates/cellar/tests/support/cell_group_managed_request_http.rs new file mode 100644 index 000000000..fabcae3b5 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_managed_request_http.rs @@ -0,0 +1,203 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Actual native and Axum ingress for the shared counter application. +use super::*; + +const SECRET: &str = "managed-counter-fixture-bearer"; + +pub(super) fn secret() -> String { + SECRET.into() +} + +pub(super) struct Reply { + pub status: u16, + body: String, + label: String, +} + +impl Reply { + pub fn refused(self, status: u16, body: &str) -> Result<()> { + ensure!( + self.status == status && self.body == body, + "{}: expected HTTP {status} ({body}), got {} ({})", + self.label, + self.status, + self.body + ); + Ok(()) + } + + pub fn success(self) -> Result { + ensure!( + self.status == 200, + "{}: HTTP {}: {}", + self.label, + self.status, + self.body + ); + Ok(serde_json::from_str(&self.body)?) + } +} + +pub(super) struct Web { + native: Vec, + axum: Vec, + stop: tokio::sync::watch::Sender, + tasks: Vec>>, +} + +impl Web { + pub async fn start(nodes: &[Option], apis: &[Arc]) -> Result { + let (stop, _) = tokio::sync::watch::channel(false); + let mut web = Self { + native: vec![], + axum: vec![], + stop, + tasks: vec![], + }; + let started = async { + for (node, api) in nodes.iter().zip(apis) { + web.native.push(format!( + "http://{}", + node.as_ref() + .context("live HTTP node")? + .info() + .public_address + )); + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await?; + web.axum.push(format!("http://{}", listener.local_addr()?)); + let mut stopping = web.stop.subscribe(); + let router = counter::router(api.clone()); + web.tasks.push(tokio::spawn(async move { + axum::serve(listener, router) + .with_graceful_shutdown(async move { + let _ = stopping.changed().await; + }) + .await + })); + } + Ok::<_, anyhow::Error>(()) + } + .await; + if let Err(error) = started { + let joined = web.close().await; + return Err(match joined { + Ok(()) => error, + Err(join) => error.context(format!("partial HTTP startup join: {join:?}")), + }); + } + Ok(web) + } + + pub(super) fn url(&self, node: usize, axum: bool, name: &str) -> String { + let base = if axum { + &self.axum[node] + } else { + &self.native[node] + }; + format!("{base}/counters/{name}") + } + + pub(super) async fn send(request: reqwest::RequestBuilder, label: String) -> Result { + let response = request.bearer_auth(SECRET).send().await?; + let status = response.status().as_u16(); + let body = response.text().await?; + Ok(Reply { + status, + body, + label, + }) + } + + pub async fn command( + &self, + client: &reqwest::Client, + (node, axum): (usize, bool), + name: &str, + delta: i64, + sequence: u64, + retry: u64, + ) -> Result { + Self::send( + client + .post(format!("{}?delta={delta}", self.url(node, axum, name))) + .header("x-command-id", format!("{}:{sequence}", "c7".repeat(16))) + .header("x-retry-until-ms", retry), + format!("command node={node} axum={axum} cell={name} sequence={sequence}"), + ) + .await + } + + pub async fn query( + &self, + client: &reqwest::Client, + (node, axum): (usize, bool), + name: &str, + mode: &str, + receipt: &serde_json::Value, + remote: Option, + ) -> Result { + let mut request = client + .get(self.url(node, axum, name)) + .header("x-read-consistency", mode); + if mode == "at-least" { + request = request.header( + "x-min-version", + receipt["version_token"] + .as_str() + .context("opaque receipt token")?, + ); + } + if let Some(remote) = remote { + request = request + .header("x-query-members", remote as u64 + 1) + .header("x-query-attempt-ms", "4000"); + } + Self::send( + request, + format!("query node={node} axum={axum} cell={name} mode={mode} remote={remote:?}"), + ) + .await + } + + pub async fn close(self) -> Result<()> { + let _ = self.stop.send(true); + let results = futures_util::future::join_all(self.tasks).await; + for result in results { + result??; + } + Ok(()) + } +} + +pub(super) fn value(reply: &serde_json::Value, name: &str, expected: i64) -> Result<()> { + ensure!(reply["cell"].as_str() == Some(format!("Counter:{name}").as_str())); + ensure!( + reply["value"].as_i64() == Some(expected), + "wrong business result: {reply}" + ); + Ok(()) +} + +pub(super) fn same_receipt( + original: &serde_json::Value, + duplicate: &serde_json::Value, +) -> Result<()> { + for field in ["value", "cell", "incarnation"] { + ensure!( + original[field] == duplicate[field], + "duplicate changed {field}" + ); + } + // The selected image may cover a historical receipt. Its version is an + // observation floor, not the client's immutable command sequence. + ensure!( + duplicate["sequence"] + .as_u64() + .context("duplicate image version")? + >= original["sequence"] + .as_u64() + .context("original image version")? + ); + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_managed_request_nodes.rs b/crates/cellar/tests/support/cell_group_managed_request_nodes.rs new file mode 100644 index 000000000..b9cfece9c --- /dev/null +++ b/crates/cellar/tests/support/cell_group_managed_request_nodes.rs @@ -0,0 +1,337 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Managed node owners and physical inventory for the public request drill. +use super::*; +use cellar::application::consensus::{ + verification::cell_groups::cell_routing_key, NamespaceQueryOrigin, +}; + +pub(super) const INGRESS: usize = 3; + +pub(super) struct Cluster { + pub managed: ManagedCluster, + nodes: Vec>, + apis: Vec>, + web: Option, +} + +impl Cluster { + pub async fn start(rooted: bool, store: Arc) -> Result { + let class = counter::definition()?; + let first = Fixture::with_store(rooted, class, store).await?; + let mut apis = Vec::new(); + let mut applications = Vec::new(); + for _ in 0..3 { + let class = counter::definition()?; + let api = counter::CounterApi::new( + http::secret(), + first.declaration.namespace.cluster, + first.declaration.namespace.namespace, + &class, + )?; + applications.push(counter::application(api.clone(), class)?); + apis.push(api); + } + let mut managed = Box::pin(ManagedCluster::start_with_applications( + first, + &[], + ArchiveSetup::Initialized, + true, + Some(applications), + FirstGroupSetup::Operator, + )) + .await?; + let nodes = std::mem::take(&mut managed.nodes) + .into_iter() + .map(Some) + .collect(); + let mut cluster = Self { + managed, + nodes, + apis, + web: None, + }; + let started = async { + Box::pin(cluster.start_ingress()).await?; + cluster.start_web().await + } + .await; + if let Err(error) = started { + let joined = cluster.close().await; + cluster.managed.fixtures[0].domain.shutdown().await; + joined?; + return Err(error); + } + Ok(cluster) + } + + async fn start_ingress(&mut self) -> Result<()> { + let mut fixture = peer_fixture(&self.managed.fixtures[0], 7).await?; + let class = counter::definition()?; + let group = self.managed.history; + let api = + counter::CounterApi::with_history_resolver(http::secret(), &class, move |name| { + // Submit canonical cell IDs with an out-of-scope cluster or + // namespace so the SDK must enforce its own namespace binding. + match name { + "foreign-cluster" => { + counter::history(ClusterId::new([211; 16])?, group.namespace, "one") + } + "foreign-namespace" => { + counter::history(group.cluster, NamespaceId::new([211; 16])?, "one") + } + _ => counter::history(group.cluster, group.namespace, name), + } + })?; + fixture.application = counter::application(api.clone(), class)?; + let root = tempfile::tempdir()?; + let node = start_node( + root.path(), + fixture.application.clone(), + &fixture.spec.label, + ) + .await?; + let started = async { + let (mut config, services) = fixture.config(&node, LogOpenMode::Create, 27).await?; + config.groups.push(RaftGroupAssignment { + history: self.managed.history, + placement: RaftGroupPlacement::QueryRouter, + }); + let peer = config.peers[0].clone(); + let host = node.start_raft_cells(config, services).await?; + Ok::<_, anyhow::Error>((peer, host)) + } + .await; + let (peer, host) = match started { + Ok(started) => started, + Err(error) => { + eprintln!( + "retained failed ingress startup disk at {}", + root.keep().display() + ); + node.shutdown(ShutdownMode::Preserve).await?; + return Err(error); + } + }; + self.managed.fixtures.push(fixture); + self.managed.roots.push(root); + self.managed.peers.push(peer); + self.managed.hosts.push(host); + self.nodes.push(Some(node)); + self.apis.push(api); + Ok(()) + } + + async fn start_web(&mut self) -> Result<()> { + self.refresh().await?; + self.web = Some(http::Web::start(&self.nodes, &self.apis).await?); + Ok(()) + } + + async fn refresh(&self) -> Result<()> { + let cut = self.managed.fixtures[0].catalog.read(TIMEOUT).await?; + for host in &self.managed.hosts { + for fixture in &self.managed.fixtures { + inspect_current(host, fixture.spec.node, cut.revision()).await?; + } + } + Ok(()) + } + + pub fn bind(&self) -> Result<()> { + for (api, host) in self.apis.iter().zip(&self.managed.hosts) { + *api.cells + .lock() + .map_err(|_| anyhow::anyhow!("counter binding poisoned"))? = Some(host.clone()); + } + Ok(()) + } + + pub fn authorize_commands(&self) -> Result<()> { + for (index, host) in self.managed.hosts.iter().enumerate() { + for peer in &self.managed.peers { + if peer.node == self.managed.fixtures[index].spec.node { + continue; + } + host.grant_command_origin(NamespaceCommandOrigin { + cluster: self.managed.history.cluster, + namespace: self.managed.history.namespace, + revision: 1, + peer: peer.clone(), + })?; + } + } + Ok(()) + } + + pub fn authorize_ingress_reads(&self) -> Result<()> { + for host in &self.managed.hosts[..INGRESS] { + host.grant_query_origin(NamespaceQueryOrigin { + cluster: self.managed.history.cluster, + namespace: self.managed.history.namespace, + revision: 1, + peer: self.managed.peers[INGRESS].clone(), + })?; + } + Ok(()) + } + + pub fn web(&self) -> Result<&http::Web> { + self.web.as_ref().context("joined HTTP owner") + } + + pub async fn leader(&self) -> Result { + tokio::time::timeout(TIMEOUT, async { + loop { + for (i, host) in self.managed.hosts[..INGRESS].iter().enumerate() { + if self.nodes[i].is_none() { + continue; + } + let status = host.group(self.managed.history)?.status().await?; + if status.ready && status.leader == Some(i as u64 + 1) { + return Ok::<_, anyhow::Error>(i); + } + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .context("actual managed leader")? + } + + pub async fn inventory(&self) -> Result<()> { + let ingress = &self.managed.hosts[INGRESS]; + ensure!(ingress.observe_node(TIMEOUT).await?.node.groups == 0); + ensure!(ingress.resource_usage().host.controller.groups == 0); + ensure!(matches!( + ingress.group(self.managed.history), + Err(RuntimeError::NotFound) + )); + tokio::time::timeout(TIMEOUT, async { + loop { + let mut complete = true; + for host in &self.managed.hosts[..INGRESS] { + let local = host.observe_node(TIMEOUT).await?; + ensure!(local.fleet_boot.is_some()); + let observed = host.group(self.managed.history)?.observe(TIMEOUT).await?; + ensure!(observed.replica.scope == self.managed.history); + complete &= local.node.groups == 1 + && observed.cells == 2 + && host.resource_usage().retained_manifest_cells == Some(2); + } + if complete { + return Ok::<_, anyhow::Error>(()); + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .context("one physical group and two independently enrolled cells")??; + + let history = self.managed.history; + let voter = &self.managed.hosts[self.leader().await?]; + let sample = voter.group(history)?.observe_partition(TIMEOUT).await?; + let one = counter::history(history.cluster, history.namespace, "one")?; + let two = counter::history(history.cluster, history.namespace, "two")?; + let mut keys = [ + cell_routing_key(history.namespace, &one.cell), + cell_routing_key(history.namespace, &two.cell), + ]; + keys.sort_unstable(); + let [left, right] = keys; + ensure!(sample.route.history == history && sample.cells == 2); + ensure!(sample.balanced_pivot.as_ref() == Some(&right)); + let largest = sample + .largest_cell + .context("largest image in a nonempty group")?; + ensure!(sample.logical_bytes >= largest.logical_bytes); + ensure!(largest.key == left || largest.key == right); + ensure!(largest.has_predecessor == (largest.key == right)); + let expected_next = (largest.key == left).then_some(&right); + ensure!(largest.next.as_ref() == expected_next); + Ok(()) + } + + pub async fn stop(&mut self, index: usize) -> Result<()> { + if let Some(node) = self.nodes[index].take() { + ensure!(matches!( + node.shutdown(ShutdownMode::Preserve).await?, + NodeOutcome::Stopped(_) + )); + } + let report = self.managed.hosts[index].close().await?; + ensure!( + report.groups_total == usize::from(index != INGRESS) + && report.groups.iter().all(|g| !g.failed) + ); + ensure!(report.mutation_outcomes == MutationOutcomes::NoUnresolvedLocalProducers); + Ok(()) + } + + pub async fn close(&mut self) -> Result<()> { + let mut failures = Vec::new(); + if let Some(web) = self.web.take() { + if let Err(error) = web.close().await { + failures.push(format!("HTTP join: {error:?}")); + } + } + for i in 0..self.nodes.len() { + if let Err(error) = self.stop(i).await { + failures.push(format!("node {i}: {error:?}")); + } + } + for api in &self.apis { + match api.cells.lock() { + Ok(mut cells) => { + cells.take(); + } + Err(_) => failures.push("counter binding poisoned".into()), + } + } + ensure!( + failures.is_empty(), + "joined managed HTTP shutdown: {failures:?}" + ); + Ok(()) + } + + pub async fn restart(&mut self) -> Result<()> { + self.close().await?; + self.managed.hosts.clear(); + self.nodes.clear(); + self.managed.peers.clear(); + for (i, fixture) in self.managed.fixtures.iter().enumerate() { + let node = start_node( + self.managed.roots[i].path(), + fixture.application.clone(), + &fixture.spec.label, + ) + .await?; + let (mut config, services) = fixture + .config(&node, LogOpenMode::Resume, 104 + i as u8) + .await?; + if i == INGRESS { + config.groups.push(RaftGroupAssignment { + history: self.managed.history, + placement: RaftGroupPlacement::QueryRouter, + }); + } else { + archives::configure(&mut config); + } + self.managed.peers.push(config.peers[0].clone()); + let host = node.start_raft_cells(config, services).await?; + self.managed.hosts.push(host); + self.nodes.push(Some(node)); + } + futures_util::future::try_join_all( + self.managed.hosts[..INGRESS] + .iter() + .map(|voter| voter.start_managed_first_group(self.managed.history, TIMEOUT)), + ) + .await?; + self.bind()?; + self.start_web().await?; + self.authorize_commands()?; + self.authorize_ingress_reads() + } +} diff --git a/crates/cellar/tests/support/cell_group_managed_request_parity.rs b/crates/cellar/tests/support/cell_group_managed_request_parity.rs new file mode 100644 index 000000000..d456fa9d0 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_managed_request_parity.rs @@ -0,0 +1,219 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Shared-group SDK receipt and option parity through both HTTP adapters. +use super::*; +use base64::{engine::general_purpose::URL_SAFE_NO_PAD, Engine as _}; + +pub(super) async fn check( + web: &http::Web, + client: &reqwest::Client, + voter: usize, + cell: &RaftCellHandle, + retry: u64, +) -> Result<()> { + // Keep parity independent of the request fixture's c7 client. + let client_bytes = [197; 16]; + let options = CommandOptions::new(CommandId::new(ClientId::new(client_bytes)?, 1)?, retry); + let increment = cell.contract("Increment")?; + let original = cell + .command(&increment, 0i64.to_be_bytes().to_vec(), options.clone()) + .await?; + ensure!(original.result == 0i64.to_be_bytes() && original.durability == Durability::Quorum); + let conflict = cell + .command(&increment, 1i64.to_be_bytes().to_vec(), options) + .await + .err() + .context("SDK accepted a conflicting payload for the original command")?; + ensure!(conflict.command == original.id && conflict.ambiguous); + ensure!(matches!( + conflict.cause.refusal(), + Some(CommandFailure::CommandConflict) + )); + let conflict_body = format!( + "{:?}; cause={:?}", + CommandFailure::Ambiguous { + command: conflict.command + }, + CommandFailure::CommandConflict, + ); + let token = original.version_token.as_str(); + let mut forged = URL_SAFE_NO_PAD.decode(token)?; + *forged.last_mut().context("version signature")? ^= 1; + let forged = URL_SAFE_NO_PAD.encode(forged); + CellVersionToken::from_encoded(&forged)?; + let member = voter as u64 + 1; + let command_id = format!("{}:1", hex::encode(client_bytes)); + let wrong_digest = hex::encode([99; 32]); + let large_token = "a".repeat(1900); + let invalid: &[&[(&str, &str)]] = &[ + &[("x-read-consistency", "at-least")], + &[("x-min-version", token)], + &[ + ("x-read-consistency", "linearizable"), + ("x-min-version", token), + ], + &[("x-read-consistency", "unknown")], + &[("x-read-policy", "unknown")], + &[ + ("x-read-consistency", "linearizable"), + ("x-read-policy", "recent-source"), + ], + &[("x-query-timeout-ms", "0")], + &[("x-query-timeout-ms", "5001")], + &[("x-query-members", "1,1")], + &[("x-query-members", "0")], + &[("x-query-members", "01")], + &[("x-query-members", "1,2,3,4,5,6,7,8,9")], + &[("x-query-attempt-ms", "10")], + &[("x-leader-fallback", "true")], + &[("x-query-members", "1"), ("x-query-attempt-ms", "0")], + &[ + ("x-query-members", "1"), + ("x-query-attempt-ms", "1001"), + ("x-query-timeout-ms", "1000"), + ], + &[("x-query-members", "1"), ("x-leader-fallback", "yes")], + &[ + ("x-read-consistency", "stale"), + ("x-read-consistency", "linearizable"), + ], + &[ + ("x-read-consistency", "at-least"), + ("x-min-version", &large_token), + ], + &[("x-write-durability", "quorum")], + ]; + for axum in [false, true] { + let url = web.url(voter, axum, "one"); + let command = |delta| { + client + .post(format!("{url}?delta={delta}")) + .header("x-command-id", &command_id) + .header("x-retry-until-ms", retry) + }; + let receipt = http::Web::send(command(0), format!("SDK receipt replay axum={axum}")) + .await? + .success()?; + http::value(&receipt, "one", 0)?; + ensure!(receipt["served_by"].is_null() && receipt["durability"] == "quorum"); + ensure!(receipt["group"] == hex::encode(original.witness.group.group.bytes())); + ensure!( + receipt["incarnation"] == hex::encode(original.version.history.incarnation.bytes()) + ); + ensure!(receipt["index"].as_u64() == Some(original.witness.position.index.get())); + ensure!(receipt["term"].as_u64() == Some(original.witness.position.term.get())); + CellVersionToken::from_encoded( + receipt["version_token"] + .as_str() + .context("HTTP receipt token")?, + )?; + + for mode in ["stale", "at-least", "linearizable"] { + for routed in [false, true] { + let mut request = client.get(&url).header("x-read-consistency", mode); + if mode == "at-least" { + request = request.header("x-min-version", token); + } + if routed { + request = request + .header("x-query-members", member) + .header("x-query-attempt-ms", "1000") + .header("x-query-timeout-ms", "2000") + .header("x-leader-fallback", "true"); + } + let read = http::Web::send(request, format!("{mode} routed={routed} axum={axum}")) + .await? + .success()?; + http::value(&read, "one", 0)?; + ensure!( + read["group"] == receipt["group"] + && read["incarnation"] == receipt["incarnation"] + ); + ensure!(read["served_by"].as_u64() == Some(member)); + ensure!( + read["index"].as_u64().context("HTTP query index")? + >= original.witness.position.index.get() + ); + CellVersionToken::from_encoded( + read["version_token"].as_str().context("HTTP query token")?, + )?; + } + } + let stale = http::Web::send( + client + .get(&url) + .header("x-read-consistency", "stale") + .header("x-read-policy", "disconnected-historical"), + format!("historical stale query axum={axum}"), + ) + .await? + .success()?; + http::value(&stale, "one", 0)?; + for headers in invalid { + let mut request = client.get(&url); + for (key, value) in *headers { + request = request.header(*key, *value); + } + http::Web::send( + request, + format!("invalid read options axum={axum}: {headers:?}"), + ) + .await? + .refused(400, "invalid request or unavailable cell")?; + } + http::Web::send( + client + .get(&url) + .header("x-read-consistency", "at-least") + .header("x-min-version", &forged), + format!("forged token axum={axum}"), + ) + .await? + .refused(503, "VersionUnavailable")?; + http::Web::send( + client + .get(&url) + .header("x-read-consistency", "linearizable") + .header("x-contract-digest", &wrong_digest), + format!("wrong query contract axum={axum}"), + ) + .await? + .refused(422, "UnsupportedCapability")?; + http::Web::send( + command(0).header("x-contract-digest", &wrong_digest), + format!("wrong command contract axum={axum}"), + ) + .await? + .refused(422, "UnsupportedCapability")?; + http::Web::send( + command(1), + format!("conflicting SDK command replay axum={axum}"), + ) + .await? + .refused(409, &conflict_body)?; + http::Web::send( + command(0).header("x-read-consistency", "linearizable"), + format!("read options on command axum={axum}"), + ) + .await? + .refused(400, "invalid request or unavailable cell")?; + ensure!(client.get(&url).send().await?.status() == 401); + ensure!(command(0).send().await?.status() == 401); + } + let sdk = cell + .query( + ProposalId::new([199; 16])?, + &cell.contract("Get")?, + vec![], + QueryOptions { + consistency: QueryConsistency::AtLeast(original.version_token), + ..Default::default() + }, + ) + .await?; + ensure!( + sdk.result == 0i64.to_be_bytes(), + "HTTP retries changed the SQL image" + ); + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_managed_requests.rs b/crates/cellar/tests/support/cell_group_managed_requests.rs new file mode 100644 index 000000000..f78b5ee13 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_managed_requests.rs @@ -0,0 +1,547 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Public HTTP requests over cold managed birth, failover and retained restart. +use super::*; +use nodes::INGRESS; + +#[path = "../../../../examples/raft-counter/counter.rs"] +mod counter; +#[path = "cell_group_managed_request_http.rs"] +mod http; +#[path = "cell_group_managed_request_nodes.rs"] +mod nodes; +#[path = "cell_group_managed_request_parity.rs"] +mod parity; + +pub async fn run_native() -> Result<()> { + for rooted in [false, true] { + Box::pin(request_path(rooted, Arc::new(fault::Store::default()))).await?; + } + Ok(()) +} + +pub async fn request_path(rooted: bool, store: Arc) -> Result<()> { + let mut cluster = Box::pin(nodes::Cluster::start(rooted, store)).await?; + let work = Box::pin(exercise(&mut cluster)).await; + let joined = cluster.close().await; + if work.is_err() || joined.is_err() { + for root in std::mem::take(&mut cluster.managed.roots) { + eprintln!("retained managed HTTP disk: {}", root.keep().display()); + } + } + cluster.managed.fixtures[0].domain.shutdown().await; + work?; + joined +} + +async fn exercise(cluster: &mut nodes::Cluster) -> Result<()> { + let retry = u64::try_from( + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH)? + .as_millis(), + )? + .checked_add(60_000) + .context("HTTP retry horizon")?; + let client = reqwest::Client::builder() + .no_proxy() + .timeout(TIMEOUT + Duration::from_secs(1)) + .build()?; + let first = cluster.leader().await?; + let one_history = counter::history( + cluster.managed.history.cluster, + cluster.managed.history.namespace, + "one", + )?; + cluster.managed.hosts[first] + .cell(one_history.clone())? + .enroll( + CellEnrollmentRequest { + creation: ProposalId::new([181; 16])?, + durability: Durability::Quorum, + }, + TIMEOUT, + ) + .await?; + cluster.bind()?; + let two_history = counter::history( + cluster.managed.history.cluster, + cluster.managed.history.namespace, + "two", + )?; + let two_handle = cluster.managed.hosts[INGRESS].cell(two_history.clone())?; + let creation = CellEnrollmentRequest { + creation: ProposalId::new([182; 16])?, + durability: Durability::Quorum, + }; + let denied = two_handle + .enroll(creation, TIMEOUT) + .await + .err() + .context("enrollment without command grant")?; + ensure!(denied.creation == creation.creation && !denied.ambiguous); + let origin = (first + 1) % 3; + ensure!( + cluster + .web()? + .command(&client, (origin, false), "one", 7, 1, retry) + .await? + .status + == 503, + "membership alone granted a command origin" + ); + for axum in [false, true] { + ensure!( + cluster + .web()? + .query( + &client, + (INGRESS, axum), + "one", + "linearizable", + &serde_json::Value::Null, + None, + ) + .await? + .status + == 503, + "an ingress process without a read grant served a query" + ); + ensure!( + cluster + .web()? + .command(&client, (INGRESS, axum), "two", 1, 2, retry) + .await? + .status + == 503, + "an ingress process without a command grant submitted a write" + ); + } + let empty = cluster + .web()? + .query( + &client, + (first, true), + "one", + "linearizable", + &serde_json::Value::Null, + None, + ) + .await? + .success()?; + http::value(&empty, "one", 0)?; + let mut foreign = NamespaceCommandOrigin { + cluster: cluster.managed.history.cluster, + namespace: NamespaceId::new([211; 16])?, + revision: 1, + peer: cluster.managed.peers[origin].clone(), + }; + ensure!(cluster.managed.hosts[first] + .grant_command_origin(foreign.clone()) + .is_err()); + foreign.namespace = cluster.managed.history.namespace; + foreign.peer.process = ProcessGeneration::new([250; 16])?; + ensure!(cluster.managed.hosts[first] + .grant_command_origin(foreign) + .is_err()); + cluster.authorize_commands()?; + ensure!( + cluster + .web()? + .query( + &client, + (INGRESS, true), + "one", + "linearizable", + &serde_json::Value::Null, + None, + ) + .await? + .status + == 503, + "a command grant authorized application reads" + ); + cluster.authorize_ingress_reads()?; + let enrolled = two_handle.enroll(creation, TIMEOUT).await?; + ensure!(enrolled.local_enrollment().is_none()); + ensure!(enrolled.record().cell() == &two_history); + ensure!(enrolled.record().creation() == creation.creation); + ensure!(enrolled.durability() == Durability::Quorum); + let original = enrolled.record().clone(); + drop(enrolled); + let archived = two_handle + .enroll( + CellEnrollmentRequest { + durability: Durability::S3, + ..creation + }, + TIMEOUT, + ) + .await?; + ensure!(archived.local_enrollment().is_none() && archived.durability() == Durability::S3); + ensure!(archived.record() == &original); + drop(archived); + cluster.inventory().await?; + Box::pin(parity::check( + cluster.web()?, + &client, + first, + &cluster.managed.hosts[first].cell(one_history)?, + retry, + )) + .await?; + let group = cluster.managed.hosts[first].group(cluster.managed.history)?; + let before_writes = group.observe_partition(TIMEOUT).await?; + let one = cluster + .web()? + .command(&client, ((first + 1) % 3, false), "one", 7, 1, retry) + .await? + .success()?; + let two = cluster + .web()? + .command(&client, ((first + 2) % 3, true), "two", 11, 1, retry) + .await? + .success()?; + http::value(&one, "one", 7)?; + http::value(&two, "two", 11)?; + ensure!(one["group"] == two["group"]); + ensure!(one["cell"] != two["cell"]); + let after_writes = group.observe_partition(TIMEOUT).await?; + ensure!(after_writes.workload.admitted_writes == before_writes.workload.admitted_writes + 2); + ensure!(after_writes.workload.applied_writes == before_writes.workload.applied_writes + 2); + let context = CellGroupPartitionContext { + directory_digest: [1; 32], + topology_changed_at_ms: before_writes.workload.sampled_at_ms, + durability: Durability::Quorum, + placement_class: [2; 32], + leader_cpu_bps: 0, + failure_domain_load_bps: 0, + active_and_stable: true, + }; + let rates = after_writes + .sample_since(&before_writes, context) + .context("same-owner workload sample")?; + let elapsed = after_writes.workload.sampled_at_ms - before_writes.workload.sampled_at_ms; + ensure!(rates.load.cells == 2 && rates.load.writes_per_second == 2_000 / elapsed); + ensure!(rates.applies_per_second == rates.load.writes_per_second); + ensure!(after_writes.sample_since(&after_writes, context).is_none()); + for index in 0..3 { + for axum in [false, true] { + let duplicate = cluster + .web()? + .command(&client, (index, axum), "one", 7, 1, retry) + .await? + .success()?; + http::same_receipt(&one, &duplicate)?; + for (name, receipt) in [("one", &one), ("two", &two)] { + for mode in ["at-least", "linearizable"] { + let remote = (index + 1) % 3; + let read = cluster + .web()? + .query(&client, (index, axum), name, mode, receipt, Some(remote)) + .await? + .success()?; + ensure!(read["value"] == receipt["value"]); + ensure!(read["served_by"].as_u64() == Some(remote as u64 + 1)); + } + } + ensure!( + cluster + .web()? + .command(&client, (index, axum), "one", 9, 1, retry) + .await? + .status + == 409, + "conflicting retry changed its identity or input" + ); + ensure!( + cluster + .web()? + .query(&client, (index, axum), "two", "at-least", &one, None) + .await? + .status + == 409, + "another cell's token granted a minimum version" + ); + } + } + let after_replays = group.observe_partition(TIMEOUT).await?; + drop(group); + ensure!(after_replays.workload.admitted_writes == after_writes.workload.admitted_writes); + ensure!(after_replays.workload.applied_writes == after_writes.workload.applied_writes); + Box::pin(check_routed_authority(cluster.web()?, &client, retry, &one)).await?; + let routed = cluster + .web()? + .command(&client, (INGRESS, false), "two", 1, 2, retry) + .await? + .success()?; + http::value(&routed, "two", 12)?; + ensure!(routed["group"] == one["group"] && routed["cell"] != one["cell"]); + for axum in [false, true] { + for mode in ["at-least", "linearizable"] { + let read = cluster + .web()? + .query(&client, (INGRESS, axum), "two", mode, &routed, None) + .await? + .success()?; + http::value(&read, "two", 12)?; + ensure!(matches!(read["served_by"].as_u64(), Some(1..=3))); + } + ensure!( + cluster + .web()? + .query(&client, (INGRESS, axum), "one", "at-least", &routed, None) + .await? + .status + == 409, + "the routed request used another cell's version token" + ); + } + + cluster.managed.hosts[first].revoke_command_origin( + cluster.managed.history.cluster, + cluster.managed.history.namespace, + cluster.managed.peers[origin].node, + 2, + )?; + ensure!( + cluster + .web()? + .command(&client, (origin, true), "one", 3, 2, retry) + .await? + .status + == 503, + "revoked process still originated a command" + ); + let unchanged = cluster + .web()? + .query(&client, (first, false), "one", "linearizable", &one, None) + .await? + .success()?; + http::value(&unchanged, "one", 7)?; + let mut restored = NamespaceCommandOrigin { + cluster: cluster.managed.history.cluster, + namespace: cluster.managed.history.namespace, + revision: 1, + peer: cluster.managed.peers[origin].clone(), + }; + ensure!(cluster.managed.hosts[first] + .grant_command_origin(restored.clone()) + .is_err()); + restored.revision = 3; + cluster.managed.hosts[first].grant_command_origin(restored)?; + let updated = cluster + .web()? + .command(&client, (origin, true), "one", 3, 2, retry) + .await? + .success()?; + http::value(&updated, "one", 10)?; + + let history = counter::history( + cluster.managed.history.cluster, + cluster.managed.history.namespace, + "one", + )?; + let ingress = cluster.managed.hosts[INGRESS].cell(history.clone())?; + let archive_store = &cluster.managed.fixtures[0].store; + let denied = archive_store + .denied_group_archive_writes + .load(Ordering::Acquire); + let mut options = CommandOptions::new(CommandId::new(ClientId::new([204; 16])?, 1)?, retry); + options.durability = Durability::S3; + + // Leave catalog and Raft traffic live while publication of new archive + // objects is refused. The same nonmember path must retain both minima. + archive_store + .deny_group_archive_writes + .store(true, Ordering::Release); + let during_outage = async { + let contract = ingress.contract("Increment")?; + let quorum = ingress + .command( + &contract, + 0i64.to_be_bytes().to_vec(), + CommandOptions::new(CommandId::new(ClientId::new([203; 16])?, 1)?, retry), + ) + .await?; + ensure!(quorum.durability == Durability::Quorum && quorum.result == 10i64.to_be_bytes()); + let error = ingress + .command(&contract, 0i64.to_be_bytes().to_vec(), options.clone()) + .await + .err() + .context("S3 command acknowledged without coverage")?; + ensure!(error.command == options.id && error.ambiguous); + ensure!( + archive_store + .denied_group_archive_writes + .load(Ordering::Acquire) + > denied + ); + Ok::<_, anyhow::Error>(()) + } + .await; + archive_store + .deny_group_archive_writes + .store(false, Ordering::Release); + during_outage?; + + let remote = cluster.managed.hosts[(first + 1) % 3].cell(history)?; + let archived = remote + .command( + &remote.contract("Increment")?, + 0i64.to_be_bytes().to_vec(), + options, + ) + .await?; + ensure!(archived.durability == Durability::S3 && archived.result == 10i64.to_be_bytes()); + ensure!(archived.witness.group == cluster.managed.history); + + // A joined leader stop is a local failover gate, not a process-crash drill. + cluster.stop(first).await?; + let successor = cluster.leader().await?; + ensure!(successor != first); + let duplicate = cluster + .web()? + .command(&client, (successor, false), "one", 7, 1, retry) + .await? + .success()?; + http::same_receipt(&one, &duplicate)?; + let read = cluster + .web()? + .query( + &client, + (successor, true), + "one", + "linearizable", + &updated, + None, + ) + .await? + .success()?; + http::value(&read, "one", 10)?; + let routed_after_stop = cluster + .web()? + .command(&client, (INGRESS, true), "two", 1, 2, retry) + .await? + .success()?; + http::same_receipt(&routed, &routed_after_stop)?; + + Box::pin(cluster.restart()).await?; + let resumed = cluster.leader().await?; + cluster.inventory().await?; + let restarted = cluster.managed.hosts[resumed] + .group(cluster.managed.history)? + .observe_partition(TIMEOUT) + .await?; + ensure!(restarted.sample_since(&after_replays, context).is_none()); + let duplicate = cluster + .web()? + .command(&client, (resumed, true), "one", 7, 1, retry) + .await? + .success()?; + http::same_receipt(&one, &duplicate)?; + let old_floor = cluster + .web()? + .query( + &client, + ((resumed + 1) % 3, false), + "one", + "at-least", + &one, + None, + ) + .await? + .success()?; + http::value(&old_floor, "one", 10)?; + let routed_after_restart = cluster + .web()? + .command(&client, (INGRESS, false), "two", 1, 2, retry) + .await? + .success()?; + http::same_receipt(&routed, &routed_after_restart)?; + let next = cluster + .web()? + .command(&client, ((resumed + 2) % 3, false), "one", 2, 3, retry) + .await? + .success()?; + http::value(&next, "one", 12)?; + + for index in [2, 1] { + cluster.stop(index).await?; + } + tokio::time::timeout(TIMEOUT, async { + loop { + let status = cluster.managed.hosts[0] + .group(cluster.managed.history)? + .status() + .await?; + if !status.ready && status.leader.is_none() { + return Ok::<_, anyhow::Error>(()); + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .context("observe loss of the actual majority")??; + let web = cluster.web()?; + for (address, axum) in [(0, false), (0, true), (INGRESS, false), (INGRESS, true)] { + let default_read = http::Web::send( + client.get(web.url(address, axum, "one")), + format!("default read without quorum node={address} axum={axum}"), + ) + .await?; + ensure!( + default_read.status == 503, + "default read without quorum returned HTTP {}", + default_read.status + ); + ensure!( + cluster + .web()? + .command(&client, (address, axum), "one", 5, 4, retry) + .await? + .status + == 503, + "write succeeded without quorum" + ); + } + Ok(()) +} + +async fn check_routed_authority( + web: &http::Web, + client: &reqwest::Client, + retry: u64, + one: &serde_json::Value, +) -> Result<()> { + for axum in [false, true] { + let ingress = (INGRESS, axum); + for name in ["unlisted", "foreign-cluster", "foreign-namespace"] { + web.command(client, ingress, name, 9, 2, retry) + .await? + .refused(400, "invalid request or unavailable cell")?; + web.query(client, ingress, name, "linearizable", one, None) + .await? + .refused(400, "invalid request or unavailable cell")?; + } + web.query(client, ingress, "two", "at-least", one, None) + .await? + .refused(409, "WrongIncarnation")?; + let replay = web + .command(client, ingress, "one", 7, 1, retry) + .await? + .success()?; + http::same_receipt(one, &replay)?; + ensure!( + replay["group"] == one["group"] + && replay["group_incarnation"] == one["group_incarnation"], + "retry selected a different group history" + ); + let read = web + .query(client, ingress, "one", "linearizable", one, None) + .await? + .success()?; + http::value(&read, "one", 7)?; + ensure!(read["sequence"] == one["sequence"]); + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_managed_router.rs b/crates/cellar/tests/support/cell_group_managed_router.rs new file mode 100644 index 000000000..127c59034 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_managed_router.rs @@ -0,0 +1,225 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! A managed ingress learns replacement membership through the directory. +use super::*; + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn managed_router_learns_replacement_voter() -> Result<()> { + for rooted in [false, true] { + Box::pin(routed_replacement( + rooted, + Arc::new(fault::Store::default()), + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[ignore = "requires explicit owned RustFS endpoint, bucket and isolated prefix"] +async fn rustfs_managed_router_learns_replacement_voter() -> Result<()> { + for rooted in [false, true] { + let store = Arc::new(fault::Store::around(rustfs::store( + "router-replacement", + rooted, + )?)); + Box::pin(routed_replacement(rooted, store)).await?; + } + Ok(()) +} + +async fn routed_replacement(rooted: bool, store: Arc) -> Result<()> { + let root = tempfile::tempdir()?; + let mut cluster = Box::pin(ManagedCluster::start_with_store( + rooted, + &[7], + ArchiveSetup::Initialized, + store, + )) + .await?; + let started = async { + let fixture = peer_fixture(&cluster.fixtures[0], 8).await?; + let node = start_node( + root.path(), + fixture.application.clone(), + &fixture.spec.label, + ) + .await?; + let attachment = async { + let (mut config, services) = fixture.config(&node, LogOpenMode::Create, 63).await?; + config.groups.push(RaftGroupAssignment { + history: cluster.history, + placement: RaftGroupPlacement::QueryRouter, + }); + let peer = config.peers[0].clone(); + let ingress = node.start_raft_cells(config, services).await?; + Ok::<_, anyhow::Error>((peer, ingress)) + } + .await; + match attachment { + Ok((peer, ingress)) => Ok((fixture, node, peer, ingress)), + Err(error) => { + node.shutdown(ShutdownMode::Preserve).await?; + Err(error) + } + } + } + .await; + let (fixture, node, peer, ingress) = match started { + Ok(started) => started, + Err(error) => { + let joined = close(std::mem::take(&mut cluster.nodes), &cluster.hosts).await; + cluster.fixtures[0].domain.shutdown().await; + for disk in std::iter::once(root).chain(std::mem::take(&mut cluster.roots)) { + eprintln!( + "retained failed router startup disk: {}", + disk.keep().display() + ); + } + joined?; + return Err(error); + } + }; + let work = async { + ensure!(ingress.observe_node(TIMEOUT).await?.node.groups == 0); + let (seed, obsolete) = Box::pin(replace_with_retry(&cluster, false, false)).await?; + maintenance::handoff_to(&cluster, obsolete, 3).await?; + let cut = fixture.catalog.read(TIMEOUT).await?; + for (index, host) in cluster.hosts.iter().enumerate() { + if index == obsolete { + continue; + } + inspect_current(host, fixture.spec.node, cut.revision()).await?; + host.grant_query_origin(NamespaceQueryOrigin { + cluster: cluster.history.cluster, + namespace: cluster.history.namespace, + revision: 1, + peer: peer.clone(), + })?; + host.grant_command_origin(NamespaceCommandOrigin { + cluster: cluster.history.cluster, + namespace: cluster.history.namespace, + revision: 1, + peer: peer.clone(), + })?; + } + let handle = ingress.cell(seed.cells[0].clone())?; + let contract = handle.contract("Get")?; + let member = MemberId::new(4)?; + let old_member = MemberId::new(obsolete as u64 + 1)?; + let deadline = tokio::time::Instant::now() + TIMEOUT; + let query = |member, consistency, invocation, timeout: Duration| { + handle.query( + invocation, + &contract, + vec![], + QueryOptions { + consistency, + timeout, + routing: QueryRouting::Replicas { + members: vec![member], + attempt_timeout: timeout, + leader_fallback: false, + }, + ..Default::default() + }, + ) + }; + let read = tokio::time::timeout_at(deadline, async { + loop { + let budget = || { + deadline + .saturating_duration_since(tokio::time::Instant::now()) + .min(Duration::from_millis(500)) + }; + let old = query( + old_member, + QueryConsistency::Linearizable, + ProposalId::new([211; 16])?, + budget(), + ) + .await; + if matches!( + old.as_ref().err().map(Arc::as_ref), + Some(CellDispatchError::Transport( + cellar::application::consensus::verification::TransportError::Identity + )) + ) { + let result = query( + member, + QueryConsistency::AtLeast(seed.receipts[0].version_token.clone()), + ProposalId::new([210; 16])?, + budget(), + ) + .await; + if let Ok(read) = result { + return Ok::<_, anyhow::Error>(read); + } + } + tokio::time::sleep(Duration::from_millis(20)).await; + } + }) + .await + .context("ingress did not replace the retired voter route")??; + ensure!( + read.served_by == member && read.served_by_node.node == cluster.fixtures[3].spec.node + ); + // Replacement verification commits sequence two (+1) before handoff. + ensure!(read.result == 8i64.to_be_bytes()); + let original = &seed.receipts[0]; + let add = handle.contract("Add")?; + let replay = handle + .command( + &add, + 7i64.to_be_bytes().to_vec(), + CommandOptions::new(original.id, seed.retry), + ) + .await?; + ensure!(replay.id == original.id && replay.result == original.result); + let next = CommandId::new(original.id.client, 3)?; + let written = handle + .command( + &add, + 1i64.to_be_bytes().to_vec(), + CommandOptions::new(next, seed.retry), + ) + .await?; + ensure!(written.id == next && written.result == 9i64.to_be_bytes()); + ensure!(written.durability == Durability::Quorum); + let observed = query( + member, + QueryConsistency::AtLeast(written.version_token), + ProposalId::new([212; 16])?, + TIMEOUT, + ) + .await?; + ensure!(observed.served_by == member && observed.result == written.result); + ensure!(ingress.resource_usage().host.controller.groups == 0); + Ok::<_, anyhow::Error>(()) + } + .await; + let ingress_join = async { + ensure!(matches!( + node.shutdown(ShutdownMode::Preserve).await?, + NodeOutcome::Stopped(_) + )); + let report = ingress.close().await?; + ensure!(report.groups_total == 0); + ensure!(report.mutation_outcomes == MutationOutcomes::NoUnresolvedLocalProducers); + Ok::<_, anyhow::Error>(()) + } + .await; + let joined = close(std::mem::take(&mut cluster.nodes), &cluster.hosts).await; + cluster.fixtures[0].domain.shutdown().await; + if work.is_err() || ingress_join.is_err() || joined.is_err() { + for disk in std::iter::once(root).chain(std::mem::take(&mut cluster.roots)) { + eprintln!( + "retained router replacement disk: {}", + disk.keep().display() + ); + } + } + work?; + ingress_join?; + joined +} diff --git a/crates/cellar/tests/support/cell_group_managed_s3.rs b/crates/cellar/tests/support/cell_group_managed_s3.rs new file mode 100644 index 000000000..9b6381e1d --- /dev/null +++ b/crates/cellar/tests/support/cell_group_managed_s3.rs @@ -0,0 +1,59 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Refusing S3 bytes retains the exact nonserving learner, without peer fallback. +use super::*; +use crate::managed_fixture::seed_failures::{restart_seed_cleanup, retained_seed_debt}; + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn managed_s3_seed_refusal_preserves_admission_and_retries_after_failover() -> Result<()> { + for rooted in [false, true] { + Box::pin(refused_with_store( + rooted, + Arc::new(fault::Store::default()), + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[ignore = "requires explicit owned RustFS endpoint, bucket and isolated prefix"] +async fn rustfs_managed_s3_seed_refusal_and_source_failover() -> Result<()> { + for rooted in [false, true] { + let store = Arc::new(fault::Store::around(rustfs::store( + "managed-s3-failover", + rooted, + )?)); + Box::pin(refused_with_store(rooted, store)).await?; + } + Ok(()) +} + +async fn refused_with_store(rooted: bool, store: Arc) -> Result<()> { + let mut cluster = Box::pin(ManagedCluster::start_with_store( + rooted, + &[7], + ArchiveSetup::Initialized, + store, + )) + .await?; + let work = Box::pin(replace_with_retry(&cluster, true, true)) + .await + .and_then(|_| retained_seed_debt(&cluster)); + let stopped = close(std::mem::take(&mut cluster.nodes), &cluster.hosts).await; + let result = match (work, stopped) { + (Ok(debt), Ok(())) => Box::pin(restart_seed_cleanup(&cluster, debt)).await, + (Err(error), Ok(())) => Err(error), + (Ok(_), Err(error)) => Err(error), + (Err(error), Err(stopped)) => Err(anyhow::anyhow!( + "S3 learner failure: {error:#}; joined shutdown failure: {stopped:#}" + )), + }; + cluster.fixtures[0].domain.shutdown().await; + if result.is_err() { + for root in std::mem::take(&mut cluster.roots) { + eprintln!("retained failed managed S3 disk: {}", root.keep().display()); + } + } + result +} diff --git a/crates/cellar/tests/support/cell_group_managed_transition.rs b/crates/cellar/tests/support/cell_group_managed_transition.rs new file mode 100644 index 000000000..a76a46b32 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_managed_transition.rs @@ -0,0 +1,670 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! The fleet slot owns the exact pre-seal directory operation. +use super::*; +use cellar::application::consensus::verification::{ + cell_groups::{cell_routing_key, TransitionPhase}, + FleetChange, FleetMutation, FleetTable, +}; + +use cellar::sdk::fleet_operations::{ + RaftFleetControl, RaftFleetFailureCode, RaftFleetTransitionGroup, + RaftFleetTransitionSealRequest, RaftFleetTransitionSourceSealRequest, +}; + +#[path = "cell_group_managed_transition_preparation.rs"] +mod preparation; + +#[path = "cell_group_managed_transition_retirement.rs"] +mod retirement; + +#[path = "cell_group_managed_transition_merge.rs"] +mod merge; + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn managed_split_plan_fences_membership_and_replays_cancellation() -> Result<()> { + Box::pin(run_plan( + false, + Arc::new(fault::Store::default()), + Decision::Cancel, + )) + .await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[ignore = "requires explicit owned RustFS endpoint, bucket and isolated prefix"] +async fn rustfs_managed_split_plan() -> Result<()> { + for rooted in [false, true] { + let store = Arc::new(fault::Store::around(rustfs::store( + "managed-transition", + rooted, + )?)); + Box::pin(run_plan(rooted, store, Decision::Cancel)).await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn managed_split_authorizes_only_the_prepared_plan() -> Result<()> { + Box::pin(run_plan( + false, + Arc::new(fault::Store::default()), + Decision::Authorize, + )) + .await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[ignore = "requires explicit owned RustFS endpoint, bucket and isolated prefix"] +async fn rustfs_managed_split_authorization() -> Result<()> { + for rooted in [false, true] { + let store = Arc::new(fault::Store::around(rustfs::store( + "managed-transition-authorize", + rooted, + )?)); + Box::pin(run_plan(rooted, store, Decision::Authorize)) + .await + .with_context(|| format!("managed split authorization, extent images={rooted}"))?; + } + Ok(()) +} + +enum Decision { + Authorize, + Cancel, + SplitMerge { recovery_barrier: bool }, +} + +async fn run_plan(rooted: bool, store: Arc, decision: Decision) -> Result<()> { + let first = Fixture::with_store(rooted, counter_class()?, store).await?; + let mut cluster = Box::pin(ManagedCluster::start_with_applications( + first, + &[7, 8, 9], + ArchiveSetup::Initialized, + false, + None, + FirstGroupSetup::Normal, + )) + .await?; + let merge = matches!(decision, Decision::SplitMerge { .. }); + let expected_children = match decision { + Decision::Authorize | Decision::SplitMerge { .. } => 2, + Decision::Cancel => 0, + }; + let result = exercise(&mut cluster, rooted, decision).await; + if merge { + let shutdown = merge::close( + cluster.nodes, + &cluster.hosts, + result.as_ref().ok().copied().flatten(), + ) + .await; + cluster.fixtures[0].domain.shutdown().await; + result?; + return shutdown; + } + let activated = result.is_ok() && expected_children > 0; + let pending_quorums = if activated { expected_children } else { 0 }; + let spares = cluster.nodes.split_off(3); + let joined = close_group_nodes(cluster.nodes, &cluster.hosts).await; + // Start all destination drains together. Serial shutdown can elect a + // successor while the next voter is already draining its local apply. + let joined_spares = + futures_util::future::join_all(spares.into_iter().zip(&cluster.hosts[3..]).map( + |(node, host)| { + stop_transfer_destination(node, host, expected_children, pending_quorums, activated) + }, + )) + .await; + cluster.fixtures[0].domain.shutdown().await; + result?; + joined?; + for result in joined_spares { + result?; + } + Ok(()) +} + +async fn stop_transfer_destination( + node: Node, + host: &RaftCells, + expected: usize, + pending_quorums: usize, + active: bool, +) -> Result<()> { + ensure!(matches!( + node.shutdown(ShutdownMode::Preserve).await?, + NodeOutcome::Stopped(_) + )); + let report = host.close().await?; + ensure!( + report.groups_total == expected + && report.groups.len() == expected + && report.omitted_groups == 0 + && report.mutation_outcomes == MutationOutcomes::NoUnresolvedLocalProducers, + "transfer destination shutdown inventory: {report:?}" + ); + ensure!( + report.groups.iter().all(|group| group.active == active + && !group.failed + && group.applied.index == group.committed), + "expected active={active}, shutdown inventory: {report:?}" + ); + ensure!( + report + .groups + .iter() + .filter(|group| group.applied.index.get() > 1) + .count() + == pending_quorums + ); + Ok(()) +} + +// Allocate the fixture's composite future outside the caller's poll frame. +// Real S3 adds HTTP stack below retained startup. +fn exercise( + cluster: &mut ManagedCluster, + rooted: bool, + decision: Decision, +) -> impl std::future::Future>> + '_ { + Box::pin(exercise_inner(cluster, rooted, decision)) +} + +async fn exercise_inner( + cluster: &mut ManagedCluster, + rooted: bool, + decision: Decision, +) -> Result> { + let fixture = &cluster.fixtures[0]; + let fleet = &fixture.catalog; + let directory = &cluster.directory; + let domain = &fixture.domain; + let scope = cluster.history; + let merge = matches!(decision, Decision::SplitMerge { .. }); + let pivot = if merge { + let left = cell_routing_key(scope.namespace, &CellId::new("Counter:cold-0")?); + let right = cell_routing_key(scope.namespace, &CellId::new("Counter:cold-1")?); + left.max(right) + } else { + cell_routing_key(scope.namespace, &CellId::new("managed-split-pivot")?) + }; + let operation = ProposalId::new([231; 16])?; + let identity = |id, revision| FleetRequestIdentity { + operation: operation.bytes(), + ..request(id, revision) + }; + let (children, plan, original) = if merge { + let admitted = Box::pin(merge::admit_operator_split( + cluster, + pivot.cell().as_str(), + operation, + )) + .await?; + ( + admitted.children, + admitted.selected.plan, + admitted.selected.request, + ) + } else { + let parent = domain + .run(directory.materialize_selected_history(scope)) + .await??; + let view = domain.run(directory.observe()).await??; + let (left, right) = parent.route().range.split(pivot)?; + let voters = cluster.fixtures[3..] + .iter() + .enumerate() + .map(|(index, fixture)| { + Ok(InitialMember { + member: MemberId::new(index as u64 + 1)?, + node: fixture.spec.node, + }) + }) + .collect::>>()?; + let mut children = Vec::new(); + for (id, range) in [(232, left), (233, right)] { + let snapshot = fleet.read(TIMEOUT).await?; + let reserved = fleet + .reserve_birth( + &snapshot, + &identity(id, snapshot.revision()), + directory, + CellGroupBirthSpec { + history: GroupHistory { + group: GroupId::new([id; 16])?, + incarnation: IncarnationId::new([id; 16])?, + ..scope + }, + range, + epoch: RouteEpoch::new(2)?, + creation: ProposalId::new([id; 16])?, + voters: voters.clone(), + learners: vec![], + }, + "Counter", + TIMEOUT, + ) + .await?; + children.push(reserved.authorization.route().clone()); + } + let plan = directory.prepare_split(&view, operation, parent.route(), &children)?; + ensure!(fleet + .reopen_managed_transition(directory, operation, TIMEOUT) + .await? + .is_none()); + let before = fleet.read(TIMEOUT).await?; + let original = identity(234, before.revision()); + fixture + .store + .fail_fleet_confirmation + .store(true, Ordering::Release); + let lost = fleet + .plan_managed_transition(&original, directory, &plan, TIMEOUT) + .await; + ensure!( + matches!(lost, Err(FleetCatalogError::Storage(_))), + "lost plan confirmation: {lost:?}" + ); + ensure!(!fixture + .store + .fail_fleet_confirmation + .load(Ordering::Acquire)); + ensure!(fleet.read(TIMEOUT).await?.revision() == before.revision() + 1); + ensure!(domain.run(directory.active_transition()).await??.is_none()); + (children, plan, original) + }; + let reopened = Fixture::open_catalog(domain, fixture.store.clone(), scope.cluster)?; + let adapter = Catalog::new( + fixture.store.clone(), + CatalogOptions { + prefix: "managed/".into(), + ..Default::default() + }, + )?; + let namespace = domain + .run(adapter.existing_namespace(fixture.declaration.namespace.clone())) + .await??; + let directory = RangeDirectory::new(namespace, DirectoryLimits::default())?; + let directory = &directory; + let recovered = reopened + .reopen_managed_transition(directory, operation, TIMEOUT) + .await? + .context("reopen the exact admitted split with a fresh controller")?; + ensure!( + recovered.journal.as_ref().map(|journal| journal.phase()) + == merge.then_some(TransitionPhase::Planned) + ); + ensure!(recovered.request == original && recovered.plan.encode() == plan.encode()); + drop(plan); + let plan = &recovered.plan; + let original = &recovered.request; + let parent = domain + .run(directory.materialize_selected_history(scope)) + .await??; + let first = reopened + .plan_managed_transition(original, directory, plan, TIMEOUT) + .await?; + ensure!(first.phase() == TransitionPhase::Planned); + ensure!( + fleet + .plan_managed_transition(original, directory, plan, TIMEOUT) + .await? + == first + ); + ensure!(matches!( + domain + .run(directory.authorize_transition_seal(plan)) + .await?, + Err(CatalogError::ManagedAdmissionRequired) + )); + + ensure!(matches!( + reopened + .authorize_managed_transition_seal(original, directory, plan, TIMEOUT) + .await, + Err(FleetCatalogError::Conflict) + )); + + let control = RaftFleetControl::new(domain, reopened.clone(), vec![directory.clone()])?; + let mut seal_request = RaftFleetTransitionSealRequest { + catalog_id: hex::encode(reopened.catalog_id()?), + namespace_id: hex::encode(scope.namespace.bytes()), + operation_id: hex::encode(operation.bytes()), + request_id: hex::encode([237; 16]), + plan_digest: hex::encode(plan.digest()), + preparation_digest: hex::encode([88; 32]), + }; + let unprepared = control + .authorize_transition_seal( + &original.principal, + &seal_request.namespace_id, + &seal_request.operation_id, + &seal_request, + TIMEOUT, + ) + .await; + ensure!(matches!(unprepared, Err(error) if error.code == RaftFleetFailureCode::Conflict)); + + let (registry, held, seeded) = Box::pin(preparation::exercise( + cluster, &reopened, directory, &recovered, &parent, &children, rooted, + )) + .await?; + + let admitted = fleet.read(TIMEOUT).await?; + let other = identity(235, admitted.revision()); + let mutation = FleetMutation { + id: other.id, + operation: other.operation, + expected_revision: other.expected_revision, + principal: other.principal, + changes: vec![FleetChange { + table: FleetTable::Admissions, + key: "test-membership".into(), + value: None, + }], + }; + ensure!(matches!( + fleet.commit(&admitted, &mutation, TIMEOUT).await, + Err(FleetCatalogError::DisruptionInProgress { operation: actual }) if actual == operation.bytes() + )); + let cancel = identity(236, admitted.revision()); + if matches!(decision, Decision::Authorize | Decision::SplitMerge { .. }) { + ensure!(matches!( + reopened + .authorize_managed_transition_seal(&cancel, directory, plan, TIMEOUT) + .await, + Err(FleetCatalogError::PayloadConflict) + )); + let prepared = control + .transition_status( + &seal_request.namespace_id, + &seal_request.operation_id, + TIMEOUT, + ) + .await?; + ensure!(prepared.phase == Some(TransitionPhase::Prepared)); + let source_request = RaftFleetTransitionSourceSealRequest { + catalog_id: seal_request.catalog_id.clone(), + namespace_id: seal_request.namespace_id.clone(), + operation_id: seal_request.operation_id.clone(), + request_id: hex::encode([177; 16]), + plan_digest: seal_request.plan_digest.clone(), + source: RaftFleetTransitionGroup { + group_id: hex::encode(scope.group.bytes()), + incarnation_id: hex::encode(scope.incarnation.bytes()), + }, + }; + let before_authorization = control + .seal_transition_source( + &original.principal, + &source_request.namespace_id, + &source_request.operation_id, + &source_request, + TIMEOUT, + ) + .await; + ensure!( + matches!(before_authorization, Err(error) if error.code == RaftFleetFailureCode::Conflict) + ); + let mismatch = control + .authorize_transition_seal( + &original.principal, + &seal_request.namespace_id, + &seal_request.operation_id, + &seal_request, + TIMEOUT, + ) + .await; + ensure!(matches!(mismatch, Err(error) if error.code == RaftFleetFailureCode::Conflict)); + seal_request.preparation_digest = + prepared.preparation_digest.context("prepared archive")?; + let foreign = control + .authorize_transition_seal( + "different-operator", + &seal_request.namespace_id, + &seal_request.operation_id, + &seal_request, + TIMEOUT, + ) + .await; + ensure!(matches!(foreign, Err(error) if error.code == RaftFleetFailureCode::Forbidden)); + let reply = crate::managed_fixture::operator::authorize_transition_seal( + fixture, + directory, + &original.principal, + &seal_request, + ) + .await?; + ensure!(reply.request == seal_request && reply.phase == TransitionPhase::SealAuthorized); + let replay = control + .authorize_transition_seal( + &original.principal, + &seal_request.namespace_id, + &seal_request.operation_id, + &seal_request, + TIMEOUT, + ) + .await?; + ensure!(replay == reply); + let authorized = reopened + .authorize_managed_transition_seal(original, directory, plan, TIMEOUT) + .await?; + ensure!(authorized.phase() == TransitionPhase::SealAuthorized); + ensure!( + authorized.preparation().map(hex::encode).as_deref() + == Some(seal_request.preparation_digest.as_str()) + ); + ensure!( + fleet + .authorize_managed_transition_seal(original, directory, plan, TIMEOUT) + .await? + == authorized + ); + ensure!(matches!( + fleet + .cancel_managed_transition(&admitted, &cancel, directory, plan, TIMEOUT) + .await, + Err(FleetCatalogError::Conflict) + )); + ensure!(fleet.read(TIMEOUT).await?.revision() == admitted.revision()); + ensure!( + domain + .run(directory.active_transition()) + .await?? + .context("authorized plan still owns the topology")? + .1 + == authorized + ); + ensure!( + domain + .run(directory.managed_preparation_holds(plan)) + .await?? + == vec![held.clone()] + ); + let without_observer = control + .seal_transition_source( + &original.principal, + &source_request.namespace_id, + &source_request.operation_id, + &source_request, + TIMEOUT, + ) + .await; + ensure!( + matches!(without_observer, Err(error) if error.code == RaftFleetFailureCode::Unsupported) + ); + let voter = leader(&cluster.hosts[..3], scope).await?; + let voter = &cluster.hosts[voter]; + let cut = Box::pin(crate::managed_fixture::operator::seal_transition_source( + fixture, + directory, + &original.principal, + voter, + &source_request, + )) + .await?; + let source = voter.group(scope)?; + let CellGroupSealOutcome::Applied(seal) = + source.seal_managed_source(&recovered, TIMEOUT).await? + else { + anyhow::bail!("operator seal lacks its durable source receipt"); + }; + ensure!( + cut.index == seal.position().index.get() + && cut.term == seal.position().term.get() + && cut.authorization_digest == hex::encode(seal.authorization()) + && !source.status().await?.ready + ); + Box::pin(preparation::verify_sealed_cut( + cluster, + &reopened, + directory, + &recovered, + &parent, + held.clone(), + &seal, + )) + .await?; + let cells = seeded.cells.clone(); + Box::pin(preparation::verify_transferred_sql( + cluster, directory, seeded, + )) + .await?; + let tombstone = Box::pin(retirement::reopen_and_retire( + cluster, directory, &recovered, + )) + .await?; + Box::pin(preparation::verify_child_archive_coverage( + cluster, + directory, + &recovered, + tombstone, + cells.clone(), + )) + .await?; + Box::pin(retirement::park_and_release_parent( + cluster, directory, &recovered, + )) + .await?; + Box::pin(preparation::verify_released_source_holds( + cluster, directory, plan, ®istry, held, + )) + .await?; + return match decision { + Decision::SplitMerge { recovery_barrier } => { + merge::run(cluster, directory, cells, rooted, recovery_barrier).await + } + _ => Ok(None), + }; + } + fixture + .store + .fail_fleet_confirmation + .store(true, Ordering::Release); + let lost = fleet + .cancel_managed_transition(&admitted, &cancel, directory, plan, TIMEOUT) + .await; + ensure!(matches!(lost, Err(FleetCatalogError::Storage(_)))); + ensure!(!fixture + .store + .fail_fleet_confirmation + .load(Ordering::Acquire)); + ensure!(fleet.read(TIMEOUT).await?.revision() == admitted.revision() + 1); + let cancelled = fleet + .cancel_managed_transition(&admitted, &cancel, directory, plan, TIMEOUT) + .await?; + ensure!( + fleet + .cancel_managed_transition(&admitted, &cancel, directory, plan, TIMEOUT) + .await? + .applied_revision + == cancelled.applied_revision + ); + ensure!( + fleet + .plan_managed_transition(original, directory, plan, TIMEOUT) + .await? + .phase() + == TransitionPhase::Cancelled + ); + ensure!(domain.run(directory.active_transition()).await??.is_none()); + ensure!(matches!( + reopened + .authorize_managed_transition_seal(original, directory, plan, TIMEOUT) + .await, + Err(FleetCatalogError::Conflict) + )); + let cancelled_plan = reopened + .reopen_managed_transition(directory, operation, TIMEOUT) + .await? + .context("cancelled operation must retain its exact historical receipt")?; + ensure!(cancelled_plan.request == *original && cancelled_plan.plan.encode() == plan.encode()); + ensure!( + cancelled_plan + .journal + .context("missing cancellation decision")? + .phase() + == TransitionPhase::Cancelled + ); + ensure!(matches!( + domain + .run(directory.managed_preparation_holds(plan)) + .await?, + Err(CatalogError::Fenced) + )); + ensure!( + reopened + .staged_managed_preparation_holds(directory, operation, TIMEOUT) + .await? + == Some(vec![held.clone()]) + ); + let archives = CellGroupArchivePool::new(Default::default())?; + for _ in 0..2 { + reopened + .release_cancelled_preparation_holds(directory, operation, &archives, TIMEOUT) + .await?; + } + preparation::verify_expired_holds(domain, directory, ®istry, vec![held]).await?; + let view = domain.run(directory.observe()).await??; + ensure!(view.groups() == 1); + Ok(None) +} + +async fn commit_source_seal( + voters: &[RaftCells], + transition: &ReopenedManagedCellGroupTransition, + history: GroupHistory, +) -> Result { + let voter = leader(voters, history).await?; + let follower = voters[(voter + 1) % voters.len()].group(history)?; + ensure!(matches!( + follower.seal_managed_source(transition, TIMEOUT).await, + Err(CellGroupControllerError::Runtime(RuntimeError::NotLeader)) + )); + let source = voters[voter].group(history)?; + let deadline = tokio::time::Instant::now() + TIMEOUT; + let applied = tokio::time::timeout_at(deadline, async { + loop { + let remaining = deadline.saturating_duration_since(tokio::time::Instant::now()); + match source.seal_managed_source(transition, remaining).await { + Ok(CellGroupSealOutcome::Applied(seal)) => return Ok(seal), + Ok(CellGroupSealOutcome::Accepted(_)) + | Err(CellGroupControllerError::Runtime( + RuntimeError::NotReady | RuntimeError::Overloaded, + )) => tokio::time::sleep(Duration::from_millis(10)).await, + Err(error) => return Err(error), + } + } + }) + .await + .context("managed parent did not apply its exact source seal")??; + ensure!(applied.position().index.get() > 0 && !source.status().await?.ready); + let CellGroupSealOutcome::Applied(replayed) = + source.seal_managed_source(transition, TIMEOUT).await? + else { + anyhow::bail!("managed seal replay attempted another proposal"); + }; + ensure!(replayed == applied); + Ok(applied) +} diff --git a/crates/cellar/tests/support/cell_group_managed_transition_fence.rs b/crates/cellar/tests/support/cell_group_managed_transition_fence.rs new file mode 100644 index 000000000..81d434660 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_managed_transition_fence.rs @@ -0,0 +1,230 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Fence and erase only the disks owned by the managed host test. +use super::*; +use futures_util::future::BoxFuture; +use std::{collections::BTreeSet, path::PathBuf, sync::atomic::AtomicUsize}; + +type PhysicalVoter = (NodeId, DiskLineage, IncarnationId); + +struct Sources { + owners: BTreeSet, + nodes: Vec, + hosts: Vec, + disks: Vec, +} + +impl Sources { + async fn erase(self, domain: Arc) -> Result> { + let Self { + nodes, + hosts, + disks, + .. + } = self; + // Stop and join every process before removing any source disk. + Box::pin(super::close(nodes, &hosts, None)).await?; + ensure!(hosts.iter().all(RaftCells::is_closed)); + drop(hosts); + domain + .blocking(move || { + for disk in &disks { + ensure!(disk.join("raft").is_dir(), "owned voter disk is missing"); + ensure!(!std::fs::symlink_metadata(disk)?.is_symlink()); + } + for disk in &disks { + std::fs::remove_dir_all(disk).context("erase owned old voter disk")?; + ensure!(!disk.try_exists()?); + } + Ok(disks) + })? + .await? + } +} + +#[derive(Default)] +struct FenceState { + sources: Option, + digest: Option<[u8; 32]>, + erasing: Option>>>, + erased: Vec, +} + +#[derive(Default)] +pub(super) struct TransitionFence { + domain: Option>, + state: tokio::sync::Mutex, + pub(super) fences: AtomicUsize, + pub(super) verifications: AtomicUsize, +} + +impl TransitionFence { + pub(super) fn own( + domain: Arc, + fixtures: &[Fixture], + roots: &mut Vec, + nodes: &mut Vec, + hosts: &mut Vec, + ) -> Result { + let voters = fixtures.len(); + ensure!( + voters > 0 && roots.len() == voters && nodes.len() == voters && hosts.len() == voters + ); + let mut owners = BTreeSet::new(); + let mut disks = BTreeSet::new(); + for (((fixture, root), node), host) in + fixtures.iter().zip(&*roots).zip(&*nodes).zip(&*hosts) + { + let spec = &fixture.spec; + let identity = host.identity(); + ensure!( + identity.node == spec.node && identity.disk == spec.disk, + "foreign test voter for {}", + spec.label + ); + // NodeDirectory canonicalizes its path when taking ownership. + let owned_root = root.path().canonicalize()?; + ensure!( + node.info().data_dir == owned_root, + "foreign test disk owner for {}", + spec.label + ); + ensure!(owners.insert((spec.node, spec.disk, spec.physical_host))); + ensure!(disks.insert(root.path())); + } + // Keep paths if the fixture fails or a fence call is cancelled. + // Only the joined erasure job may delete them. + let disks = std::mem::take(roots) + .into_iter() + .map(tempfile::TempDir::keep) + .collect(); + Ok(Self { + domain: Some(domain), + state: tokio::sync::Mutex::new(FenceState { + sources: Some(Sources { + owners, + nodes: std::mem::take(nodes), + hosts: std::mem::take(hosts), + disks, + }), + ..Default::default() + }), + ..Default::default() + }) + } + + pub(super) async fn erased_disks(&self) -> usize { + self.state.lock().await.erased.len() + } +} + +pub(super) async fn check_old_boot_refused(fixture: &Fixture) -> Result<()> { + let root = tempfile::tempdir()?; + let node = start_node( + root.path(), + fixture.application.clone(), + &fixture.spec.label, + ) + .await?; + let config = fixture.config(&node, LogOpenMode::Create, 211).await; + // Poll startup and its error capture outside the split/merge scenario stack. + let boot = fixture.domain.spawn(Box::pin(async move { + let attempted = match config { + Ok((config, services)) => Ok(Box::pin(node.start_raft_cells(config, services)).await), + Err(error) => Err(error), + }; + ensure!(matches!( + node.shutdown(ShutdownMode::Preserve).await?, + NodeOutcome::Stopped(_) + )); + match attempted.context("prepare the old voter restart probe")? { + Err(error) => { + ensure!( + error.chain().any(|cause| matches!( + cause.downcast_ref::(), + Some(cellar::application::consensus::CatalogError::LostStableState) + )), + "expected the permanent engine pin to refuse an old voter on empty storage: {error:#}" + ); + Ok(()) + } + Ok(host) => { + host.close().await?; + anyhow::bail!("a new engine booted with an erased voter's identity"); + } + } + }))?; + boot.await? +} + +impl DisasterFenceAuthority for TransitionFence { + fn fence<'a>( + &'a self, + request: &'a FleetTransitionRecoveryFenceRequest, + ) -> BoxFuture<'a, Result>> { + Box::pin(async move { + self.fences.fetch_add(1, Ordering::AcqRel); + let mut state = self.state.lock().await; + let digest = request.digest(); + if let Some(selected) = state.digest { + ensure!(selected == digest, "different fence scope"); + } + if let Some(sources) = &state.sources { + let required = request + .intent() + .required_voter_lineages() + .iter() + .map(|lineage| (lineage.node, lineage.disk, lineage.physical_host)) + .collect::>(); + ensure!( + sources.owners == required, + "incomplete physical writer inventory" + ); + let domain = self.domain.as_ref().context("fence execution domain")?; + let sources = state.sources.take().context("owned voters already taken")?; + let job = domain.spawn(Box::pin(sources.erase(domain.clone())))?; + state.digest = Some(digest); + // Keep the task handle in the authority across a lost or + // cancelled caller. A retry joins the same physical operation. + state.erasing = Some(Box::pin(async move { job.await? })); + } + if let Some(erasing) = &mut state.erasing { + let erased = erasing.await; + state.erasing = None; + state.erased = erased?; + } + ensure!(!state.erased.is_empty(), "source erasure is unconfirmed"); + Ok(digest.to_vec()) + }) + } + + fn verify<'a>( + &'a self, + request: &'a FleetTransitionRecoveryFenceRequest, + evidence: &'a [u8], + ) -> BoxFuture<'a, Result<()>> { + Box::pin(async move { + self.verifications.fetch_add(1, Ordering::AcqRel); + let digest = request.digest(); + ensure!(evidence == digest, "unbound fence evidence"); + let disks = { + let state = self.state.lock().await; + ensure!( + state.digest == Some(digest) && !state.erased.is_empty(), + "fence not effective" + ); + state.erased.clone() + }; + self.domain + .as_ref() + .context("fence execution domain")? + .blocking(move || { + for disk in disks { + ensure!(!disk.try_exists()?, "erased voter disk reappeared"); + } + Ok(()) + })? + .await? + }) + } +} diff --git a/crates/cellar/tests/support/cell_group_managed_transition_merge.rs b/crates/cellar/tests/support/cell_group_managed_transition_merge.rs new file mode 100644 index 000000000..a3853e307 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_managed_transition_merge.rs @@ -0,0 +1,3435 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Merge the live children of a closed split and retire both sources. +use super::*; +use cellar::application::consensus::verification::{ + cell_groups::{ + CellGroupArchiveHold, CellGroupArchiveTailLimits, CellGroupArchiveTailPool, CellGroupBirth, + CellGroupCheckpointPool, CellGroupTransitionSource, CellImagePin, CellRoutingError, + CellRoutingFailure, GroupRoute, + }, + FleetChange, FleetHistoryState, FleetMutation, FleetSnapshot, FleetTable, +}; +use cellar::application::consensus::{DisasterFenceAuthority, FleetNodeSpec}; +use cellar::sdk::fleet_operations::{ + FleetFencedTransitionRecovery, FleetRecoveryAttachRequest, FleetTransitionRecoveryFenceRequest, + RaftFleetControl, RaftFleetFailureCode, RaftFleetFirstGroupVoter, RaftFleetOutcome, + RaftFleetTransitionAdmission, RaftFleetTransitionAdmissionRequest, RaftFleetTransitionChild, + RaftFleetTransitionChildSetupRequest, RaftFleetTransitionChildSetupResult, + RaftFleetTransitionChildStep, RaftFleetTransitionGroup, RaftFleetTransitionPartition, + RaftFleetTransitionPrepareRequest, RaftFleetTransitionPublishRequest, + RaftFleetTransitionResume, RaftFleetTransitionResumeStep, RaftFleetTransitionSealRequest, + RaftFleetTransitionSourceRetirement, RaftFleetTransitionSourceSealRequest, +}; + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn managed_operator_split_reserves_children_in_key_order() -> Result<()> { + let cluster = Box::pin(ManagedCluster::start(false, &[7, 8, 9])).await?; + let result = Box::pin(check_operator_split_cancellation(&cluster)).await; + let joined = close(cluster.nodes, &cluster.hosts, None).await; + cluster.fixtures[0].domain.shutdown().await; + result?; + joined +} + +pub(super) struct OperatorSplit { + pub(super) selected: ReopenedManagedCellGroupTransition, + pub(super) children: Vec, + request: RaftFleetTransitionAdmissionRequest, + receipt: RaftFleetTransitionAdmission, +} + +pub(super) async fn admit_operator_split( + cluster: &ManagedCluster, + pivot_cell: &str, + operation: ProposalId, +) -> Result { + let fixture = &cluster.fixtures[0]; + let directory = &cluster.directory; + let control = RaftFleetControl::new( + &fixture.domain, + fixture.catalog.clone(), + vec![directory.clone()], + )?; + let namespace = hex::encode(cluster.history.namespace.bytes()); + let basis = control.transition_basis(&namespace, TIMEOUT).await?; + let view = fixture.domain.run(directory.observe()).await??; + ensure!(hex::encode(view.digest()) == basis.directory_digest); + let principal = request(234, basis.fleet_revision).principal; + let parent = fixture + .domain + .run(directory.materialize_selected_history(cluster.history)) + .await??; + let pivot = cell_routing_key(cluster.history.namespace, &CellId::new(pivot_cell)?); + let (left, right) = parent.route().range.split(pivot.clone())?; + let histories = [232, 233].map(|id| -> Result<_> { + Ok(GroupHistory { + group: GroupId::new([id; 16])?, + incarnation: IncarnationId::new([id; 16])?, + ..cluster.history + }) + }); + let [left_history, right_history] = histories; + let histories = [left_history?, right_history?]; + let admission = RaftFleetTransitionAdmissionRequest { + catalog_id: basis.catalog_id, + namespace_id: namespace.clone(), + operation_id: hex::encode(operation.bytes()), + request_id: hex::encode([234; 16]), + expected_fleet_revision: basis.fleet_revision, + expected_directory_digest: basis.directory_digest, + class: "Counter".into(), + partition: RaftFleetTransitionPartition::Split { + parent: transition_group(cluster.history), + pivot_cell: pivot_cell.into(), + }, + children: histories + .iter() + .enumerate() + .map(|(index, history)| RaftFleetTransitionChild { + group: transition_group(*history), + request_id: hex::encode([232 + index as u8; 16]), + creation_id: hex::encode([232 + index as u8; 16]), + voters: std::array::from_fn(|member| RaftFleetFirstGroupVoter { + node_id: hex::encode(cluster.fixtures[3 + member].spec.node.bytes()), + member_id: member as u64 + 1, + }), + }) + .collect(), + }; + let invalid = control + .admit_transition("", &namespace, &admission.operation_id, &admission, TIMEOUT) + .await; + ensure!(matches!(invalid, Err(failure) if failure.code == RaftFleetFailureCode::InvalidInput)); + ensure!(fixture.catalog.read(TIMEOUT).await?.revision() == basis.fleet_revision); + let voters = &cluster.hosts[..3]; + let leader = &voters[leader(voters, cluster.history).await?]; + let sample = leader + .group(cluster.history)? + .observe_partition(TIMEOUT) + .await?; + let selected = PartitionBasis { + route: sample.route, + directory_digest: hex::decode(&admission.expected_directory_digest)? + .try_into() + .map_err(|_| anyhow::anyhow!("directory digest length"))?, + membership: sample.membership, + term: sample.status.term, + durability: directory.namespace_config().minimum_durability, + placement_class: [1; 32], + }; + let control = control.with_node_observer(leader.clone()); + let stale = PartitionBasis { + term: Term::new(selected.term.get() + 1), + ..selected.clone() + }; + let rejected = Box::pin(control.admit_recommended_partition( + &principal, + &admission, + PartitionDecision::Split { + basis: &stale, + pivot: &pivot, + reason: SplitReason::GroupLoad, + }, + TIMEOUT, + )) + .await; + ensure!(matches!(rejected, Err(failure) if failure.code == RaftFleetFailureCode::Conflict)); + ensure!(fixture.catalog.read(TIMEOUT).await?.revision() == basis.fleet_revision); + fixture + .store + .fail_fleet_confirmation + .store(true, Ordering::Release); + let lost = Box::pin(control.admit_recommended_partition( + &principal, + &admission, + PartitionDecision::Split { + basis: &selected, + pivot: &pivot, + reason: SplitReason::GroupLoad, + }, + TIMEOUT, + )) + .await; + ensure!(matches!(lost, Err(failure) if failure.code == RaftFleetFailureCode::Unavailable)); + let reserved = fixture.catalog.read(TIMEOUT).await?; + ensure!(reserved.revision() == basis.fleet_revision + 1); + ensure!( + *fixture + .catalog + .history(&reserved, &histories[0], TIMEOUT) + .await? + .state() + == FleetHistoryState::Reserved + ); + ensure!(matches!( + fixture + .catalog + .history(&reserved, &histories[1], TIMEOUT) + .await, + Err(FleetCatalogError::Conflict) + )); + // This unrelated mutation keeps its own Fleet request ID. The serving + // case later submits a real cancellation against the split operation. + let receipt = request(254, reserved.revision()); + fixture + .catalog + .commit( + &reserved, + &FleetMutation { + id: receipt.id, + operation: receipt.operation, + principal: receipt.principal, + expected_revision: receipt.expected_revision, + changes: vec![FleetChange { + table: FleetTable::Admissions, + key: "qualification-advance-between-children".into(), + value: None, + }], + }, + TIMEOUT, + ) + .await?; + // The CLI opens a fresh control against the changed Fleet revision. + let admitted = crate::managed_fixture::operator::admit_transition( + fixture, directory, &principal, &admission, + ) + .await?; + let cut = fixture.catalog.read(TIMEOUT).await?; + ensure!( + admitted.phase == TransitionPhase::Planned && cut.revision() == basis.fleet_revision + 4 + ); + let mut children = Vec::with_capacity(histories.len()); + for (history, range) in histories.iter().zip([left, right]) { + ensure!( + *fixture + .catalog + .history(&cut, history, TIMEOUT) + .await? + .state() + == FleetHistoryState::Reserved + ); + let birth = fixture + .catalog + .recover_birth_reservation(&cut, directory, history, TIMEOUT) + .await?; + ensure!(birth.route().range == range && birth.route().epoch == RouteEpoch::new(2)?); + children.push(birth.route().clone()); + } + let planned_view = fixture.domain.run(directory.observe()).await??; + ensure!(planned_view.revision() == view.revision() && planned_view.groups() == 1); + let selected = fixture + .catalog + .reopen_managed_transition(directory, operation, TIMEOUT) + .await? + .context("admitted split plan")?; + let expected = directory.prepare_split(&view, operation, parent.route(), &children)?; + ensure!( + selected.plan.encode() == expected.encode(), + "admission changed split ranges" + ); + ensure!( + selected.request.principal == principal + && selected.request.operation == operation.bytes() + && selected.request.expected_revision.checked_add(1) == Some(cut.revision()) + && selected.journal.as_ref().map(|journal| journal.phase()) + == Some(TransitionPhase::Planned), + "admission did not select the exact Fleet step" + ); + ensure!( + admitted.operation_id == admission.operation_id + && admitted.request_id == admission.request_id + && admitted.plan_digest == hex::encode(selected.plan.digest()), + "CLI admission receipt differs from the rooted plan" + ); + let replay = control + .admit_transition( + &principal, + &namespace, + &admission.operation_id, + &admission, + TIMEOUT, + ) + .await?; + ensure!( + replay == admitted && fixture.catalog.read(TIMEOUT).await?.revision() == cut.revision() + ); + Ok(OperatorSplit { + selected, + children, + request: admission, + receipt: admitted, + }) +} + +async fn check_operator_split_cancellation(cluster: &ManagedCluster) -> Result<()> { + let operation = ProposalId::new([231; 16])?; + let split = Box::pin(admit_operator_split( + cluster, + "Counter:operator-split", + operation, + )) + .await?; + let fixture = &cluster.fixtures[0]; + let directory = &cluster.directory; + let cut = fixture.catalog.read(TIMEOUT).await?; + let control = RaftFleetControl::new( + &fixture.domain, + fixture.catalog.clone(), + vec![directory.clone()], + )?; + let cancellation = FleetRequestIdentity { + operation: operation.bytes(), + ..request(235, cut.revision()) + }; + fixture + .catalog + .cancel_managed_transition( + &cut, + &cancellation, + directory, + &split.selected.plan, + TIMEOUT, + ) + .await?; + let cancelled = control + .admit_transition( + &split.selected.request.principal, + &split.request.namespace_id, + &split.request.operation_id, + &split.request, + TIMEOUT, + ) + .await?; + ensure!( + cancelled.phase == TransitionPhase::Cancelled + && cancelled.plan_digest == split.receipt.plan_digest + ); + Ok(()) +} + +fn transition_group(history: GroupHistory) -> RaftFleetTransitionGroup { + RaftFleetTransitionGroup { + group_id: hex::encode(history.group.bytes()), + incarnation_id: hex::encode(history.incarnation.bytes()), + } +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn managed_split_then_merge_replays_late_writes() -> Result<()> { + Box::pin(run_plan( + false, + Arc::new(fault::Store::default()), + Decision::SplitMerge { + recovery_barrier: false, + }, + )) + .await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[ignore = "requires explicit owned RustFS endpoint, bucket and isolated prefix"] +async fn rustfs_managed_split_merge() -> Result<()> { + for rooted in [false, true] { + let store = Arc::new(fault::Store::around(rustfs::store( + "managed-split-merge", + rooted, + )?)); + Box::pin(run_plan( + rooted, + store, + Decision::SplitMerge { + recovery_barrier: false, + }, + )) + .await + .with_context(|| format!("managed split and merge, extent images={rooted}"))?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn managed_recovery_barrier_fences_transfer_and_reopens() -> Result<()> { + Box::pin(run_plan( + false, + Arc::new(fault::Store::default()), + Decision::SplitMerge { + recovery_barrier: true, + }, + )) + .await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[ignore = "requires explicit owned RustFS endpoint, bucket and isolated prefix"] +async fn rustfs_managed_recovery_barrier_fences_transfer_and_reopens() -> Result<()> { + for rooted in [false, true] { + let store = Arc::new(fault::Store::around(rustfs::store( + "managed-recovery-barrier", + rooted, + )?)); + Box::pin(run_plan( + rooted, + store, + Decision::SplitMerge { + recovery_barrier: true, + }, + )) + .await + .with_context(|| format!("managed recovery barrier, extent images={rooted}"))?; + } + Ok(()) +} + +#[path = "cell_group_managed_transition_fence.rs"] +mod fence; +use fence::TransitionFence; + +struct ArchiveInputs { + domain: Arc, + pool: CellGroupArchivePool, + tails: CellGroupArchiveTailPool, + scratch: tempfile::TempDir, + rooted: bool, +} + +impl ArchiveInputs { + async fn sources( + &self, + stage: &str, + cuts: &[(CellGroupBirth, CellGroupArchiveHold)], + ) -> Result> { + let mut sources = Vec::with_capacity(cuts.len()); + for (index, (birth, hold)) in cuts.iter().enumerate() { + let base = self.scratch.path().join(stage).join(index.to_string()); + let route = birth.route().clone(); + let rooted = self.rooted; + let target = self + .domain + .blocking(move || preparation::store(base, route, rooted))? + .await??; + sources.push(CellGroupTransitionSource { + birth: birth.clone(), + hold: hold.clone(), + receiver: CellGroupCheckpointPool::new(checkpoint_limits())?.receive_archive( + target.store, + target.scratch, + target.directory, + )?, + }); + } + Ok(sources) + } +} + +pub(super) fn run<'a>( + cluster: &'a mut ManagedCluster, + directory: &'a RangeDirectory, + cells: Vec, + rooted: bool, + recovery_barrier: bool, +) -> impl std::future::Future>> + 'a { + Box::pin(run_inner( + cluster, + directory, + cells, + rooted, + recovery_barrier, + )) +} + +async fn run_inner( + cluster: &mut ManagedCluster, + directory: &RangeDirectory, + cells: Vec, + rooted: bool, + recovery_barrier: bool, +) -> Result> { + let fixture = &cluster.fixtures[0]; + let fleet = fixture.catalog.clone(); + let control = RaftFleetControl::new(&fixture.domain, fleet.clone(), vec![directory.clone()])?; + let archives = ArchiveInputs { + domain: fixture.domain.clone(), + pool: CellGroupArchivePool::new(Default::default())?, + // Both independently verified source cuts stay pinned through staging. + tails: CellGroupArchiveTailPool::new(CellGroupArchiveTailLimits { + max_operations: 2, + max_retained_bytes: 2 * CellGroupArchiveTailLimits::default().max_retained_bytes, + ..Default::default() + })?, + scratch: tempfile::tempdir()?, + rooted, + }; + let domain = &archives.domain; + let view = domain.run(directory.observe()).await??; + let page = domain.run(directory.group_page(&view, None, 3)).await??; + ensure!(page.next.is_none()); + let parents: [GroupRoute; 2] = page + .routes + .try_into() + .map_err(|_| anyhow::anyhow!("merge requires exactly two split children"))?; + let mut routed = Vec::with_capacity(parents.len()); + for parent in &parents { + let cell = cells + .iter() + .enumerate() + .find(|(_, cell)| { + parent + .range + .contains(&cell_routing_key(parent.history.namespace, &cell.cell)) + }) + .context("each split child must hold one cell")?; + routed.push((cell.1.clone(), [12i64, 13][cell.0])); + } + for (index, (fixture, host)) in cluster.fixtures[..3] + .iter() + .zip(&cluster.hosts[..3]) + .enumerate() + { + admit_capacity(fixture, host, directory, 130 + 2 * index as u8).await?; + } + let operation = ProposalId::new([244; 16])?; + let history = GroupHistory { + group: GroupId::new([243; 16])?, + incarnation: IncarnationId::new([243; 16])?, + ..cluster.history + }; + let (admitted, cut) = Box::pin(admit_merge( + cluster, directory, &control, &parents, history, operation, + )) + .await?; + let planned = fleet + .reopen_managed_transition(directory, operation, TIMEOUT) + .await? + .context("operator admitted merge")?; + ensure!( + planned + .journal + .as_ref() + .is_some_and(|journal| journal.phase() == TransitionPhase::Planned) + && admitted.plan_digest == hex::encode(planned.plan.digest()) + ); + let child = fleet + .recover_birth_reservation(&cut, directory, &history, TIMEOUT) + .await? + .route() + .clone(); + let mut reused = child.clone(); + reused.history.group = parents[0].history.group; + ensure!(directory + .prepare_merge(&view, operation, &parents, &reused) + .is_err()); + let mut incompatible = parents.clone(); + incompatible[1].schema_digest = [7; 32]; + ensure!(directory + .prepare_merge(&view, operation, &incompatible, &child) + .is_err()); + let mut gap = parents.clone(); + let gap_key = cell_routing_key(child.history.namespace, &CellId::new("Counter:gap-21")?); + gap[1].range = gap[1] + .range + .split(gap_key) + .context("gap fixture boundary")? + .1; + ensure!(directory + .prepare_merge(&view, operation, &gap, &child) + .is_err()); + let plan = planned.plan; + let request = planned.request; + + let mut registries = Vec::with_capacity(2); + for route in &parents { + let birth = domain + .run(directory.materialize_selected_history(route.history)) + .await??; + let registry = directory + .open_group_archive(domain.as_ref(), &archives.pool, &birth) + .await?; + registries.push(registry); + } + let (preparer, reopened_directory) = reopen_merge_controller(fixture).await?; + let workspace = tempfile::tempdir()?; + let tail_limits = CellGroupArchiveTailLimits { + max_operations: 2, + max_retained_bytes: 2 * CellGroupArchiveTailLimits::default().max_retained_bytes, + ..Default::default() + }; + let verifier = crate::managed_fixture::verifier::open( + fixture, + &workspace.path().join("operator"), + rooted, + tail_limits, + ) + .await?; + let prepare_request = RaftFleetTransitionPrepareRequest { + catalog_id: hex::encode(fleet.catalog_id()?), + namespace_id: hex::encode(cluster.history.namespace.bytes()), + operation_id: hex::encode(operation.bytes()), + request_id: hex::encode([240; 16]), + plan_digest: hex::encode(plan.digest()), + }; + let preparation = crate::managed_fixture::operator::prepare_transition( + fixture, + &reopened_directory, + &request.principal, + &prepare_request, + Some((verifier, preparation::recovery_pool()?)), + ) + .await?; + ensure!(preparation.phase == TransitionPhase::Prepared); + let holds = preparer + .staged_managed_preparation_holds(&reopened_directory, operation, TIMEOUT) + .await? + .context("selected merge preparation holds")?; + ensure!(holds.iter().map(|hold| hold.marker().route()).eq(&parents)); + let verifier = crate::managed_fixture::verifier::open( + fixture, + &workspace.path().join("replay"), + rooted, + tail_limits, + ) + .await?; + let prepared = preparer + .resume_managed_preparation( + &reopened_directory, + operation, + &verifier, + preparation::recovery_pool()?, + TIMEOUT, + ) + .await + .context("reopen both merge preparation archives")?; + ensure!(prepared.journal().phase() == TransitionPhase::Prepared); + ensure!(prepared.children().len() == 1 && prepared.children()[0].root().cells() == 2); + ensure!( + prepared.journal().preparation().map(hex::encode).as_deref() + == Some(preparation.preparation_digest.as_str()) + ); + drop(prepared); + drop(verifier); + drop(workspace); + let authorization = crate::managed_fixture::operator::authorize_transition_seal( + fixture, + &reopened_directory, + &request.principal, + &RaftFleetTransitionSealRequest { + catalog_id: prepare_request.catalog_id, + namespace_id: prepare_request.namespace_id, + operation_id: prepare_request.operation_id, + request_id: hex::encode([239; 16]), + plan_digest: prepare_request.plan_digest, + preparation_digest: preparation.preparation_digest, + }, + ) + .await?; + ensure!(authorization.phase == TransitionPhase::SealAuthorized); + let transition = Arc::new( + fleet + .reopen_managed_transition(directory, operation, TIMEOUT) + .await? + .context("reopened authorized merge")?, + ); + let retry = u64::try_from( + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH)? + .as_millis(), + )? + .checked_add(60_000) + .context("merge retry deadline")?; + let limits = CellGroupArchiveTailLimits { + max_operations: parents.len(), + max_retained_bytes: parents.len() + * CellGroupArchiveTailLimits::default().max_retained_bytes, + ..Default::default() + }; + let sealed_workspace = tempfile::tempdir()?; + let verifier = crate::managed_fixture::verifier::open( + &cluster.fixtures[0], + sealed_workspace.path(), + rooted, + limits, + ) + .await?; + let mut controller = (fleet.clone(), directory.clone()); + let mut receipts = Vec::with_capacity(2); + for (index, (parent, (cell, expected))) in parents.iter().zip(&routed).enumerate() { + let fixture = &cluster.fixtures[0]; + let voters = &cluster.hosts[3..]; + let voter = leader(voters, parent.history).await?; + let handle = voters[voter].cell(cell.clone())?; + let receipt = handle + .command( + &handle.contract("Add")?, + 1i64.to_be_bytes().to_vec(), + CommandOptions::new(CommandId::new(ClientId::new([202; 16])?, 1)?, retry), + ) + .await?; + ensure!( + receipt.result == (expected + 1).to_be_bytes() + && receipt.witness.group == parent.history + ); + receipts.push(receipt); + let seal = Box::pin(commit_source_seal(voters, &transition, parent.history)).await?; + preparation::wait_transfer_applied(voters, &[(parent.history, seal.position().index)]) + .await?; + wait_archive(voters, parent.history, &seal).await?; + if index == 0 { + let denied_before = fixture + .store + .denied_group_payload_reads + .load(Ordering::Acquire); + fixture + .store + .deny_group_payload_reads + .store(true, Ordering::Release); + let denied = controller + .0 + .capture_managed_sealed_source( + &controller.1, + operation, + parent.history, + &verifier, + TIMEOUT, + ) + .await; + fixture + .store + .deny_group_payload_reads + .store(false, Ordering::Release); + ensure!( + denied.is_err() + && fixture + .store + .denied_group_payload_reads + .load(Ordering::Acquire) + > denied_before + ); + let progress = controller + .0 + .staged_managed_sealed_sources(&controller.1, operation, TIMEOUT) + .await?; + ensure!(progress.iter().all(|(_, hold)| hold.is_none())); + controller = reopen_merge_controller(fixture).await?; + } + let capture = RaftFleetTransitionSourceSealRequest { + catalog_id: control.catalog_id().into(), + namespace_id: hex::encode(parent.history.namespace.bytes()), + operation_id: hex::encode(operation.bytes()), + request_id: hex::encode([237 + index as u8; 16]), + plan_digest: hex::encode(plan.digest()), + source: transition_group(parent.history), + }; + let captured = crate::managed_fixture::operator::capture_transition_source( + fixture, + &controller.1, + &request.principal, + &capture, + (verifier.clone(), preparation::recovery_pool()?), + ) + .await?; + ensure!( + captured.request == capture + && captured.sealed.index == seal.position().index.get() + && captured.sealed.term == seal.position().term.get() + && captured.sealed.authorization_digest == hex::encode(seal.authorization()) + ); + let hold = controller + .0 + .staged_managed_sealed_sources(&controller.1, operation, TIMEOUT) + .await? + .into_iter() + .nth(index) + .and_then(|(_, hold)| hold) + .context("captured sealed parent slot")?; + ensure!(hold.marker().source_seal() == Some(&seal)); + ensure!( + controller + .0 + .capture_managed_sealed_source( + &controller.1, + operation, + parent.history, + &verifier, + TIMEOUT, + ) + .await? + == hold + ); + let birth = domain + .run(controller.1.materialize_selected_history(parent.history)) + .await??; + let cut = (birth, hold); + if index == 0 { + let active = Some(parents[1].history); + Box::pin(restart_voter(cluster, 3, 120, active, 2)) + .await + .context("restart a voter with one sealed merge parent")?; + let fifth = Box::pin(stop_voter(cluster, 5, active)).await?; + let fourth = Box::pin(stop_voter(cluster, 4, active)).await?; + let survivor = &cluster.hosts[3]; + Box::pin(verify_minority( + survivor, + &transition, + &seal, + &parents, + &routed, + retry, + )) + .await?; + ensure!(matches!( + fleet + .verify_managed_sealed_sources( + &transition, + directory, + &archives.pool, + &archives.tails, + archives + .sources("partial", std::slice::from_ref(&cut)) + .await?, + TIMEOUT, + ) + .await, + Err(FleetCatalogError::Invalid(_) | FleetCatalogError::Conflict) + )); + controller = reopen_merge_controller(&cluster.fixtures[0]).await?; + let observed = controller + .0 + .staged_managed_sealed_sources(&controller.1, operation, TIMEOUT) + .await?; + ensure!( + observed.len() == 2 + && observed[0].0 == parents[0] + && observed[0].1.as_ref() == Some(&cut.1) + && observed[1].0 == parents[1] + && observed[1].1.is_none() + ); + drop(observed); + drop(cut); + ensure!(matches!( + controller + .0 + .replay_managed_sealed_sources(&controller.1, operation, &verifier, TIMEOUT,) + .await, + Err(FleetCatalogError::Unconfirmed) + )); + let selected = controller + .0 + .reopen_managed_transition(&controller.1, operation, TIMEOUT) + .await? + .context("reopened partial merge")?; + let alternate = preparation::acquire_hold(domain, ®istries[index]).await?; + let birth = domain + .run(controller.1.materialize_selected_history(parent.history)) + .await??; + let mut replacement = archives + .sources("competing-hold", &[(birth, alternate.clone())]) + .await?; + ensure!(matches!( + controller + .0 + .stage_managed_sealed_source( + &selected, + &controller.1, + &archives.pool, + &archives.tails, + replacement + .pop() + .context("competing sealed parent receiver")?, + TIMEOUT, + ) + .await, + Err(FleetCatalogError::Conflict) + )); + registries[index] + .release(domain.as_ref(), &alternate) + .await?; + let early = fleet + .publish_managed_transfer(&transition, directory, &archives.pool, TIMEOUT) + .await; + ensure!( + matches!(early, Err(FleetCatalogError::Unconfirmed)), + "merge publication without both parents and a child quorum: {early:?}" + ); + let unchanged = domain.run(directory.observe()).await??; + ensure!(unchanged.revision() == view.revision() && unchanged.groups() == 2); + let namespace = hex::encode(child.history.namespace.bytes()); + let operation_id = hex::encode(operation.bytes()); + let status = control + .transition_status(&namespace, &operation_id, TIMEOUT) + .await?; + ensure!( + status.namespace_id == namespace + && status.operation_id == operation_id + && status.phase == Some(TransitionPhase::SealAuthorized) + && status.plan_digest == hex::encode(plan.digest()) + && status.fleet_slot_held + && !status.child_archives_verified + && !status.parent_holds_released + ); + let recovery = + Box::pin(control.transition_recovery_basis(&namespace, &operation_id, TIMEOUT)) + .await?; + let parent_groups: Vec<_> = parents + .iter() + .map(|parent| transition_group(parent.history)) + .collect(); + ensure!( + recovery.phase == TransitionPhase::SealAuthorized + && recovery.plan_digest == status.plan_digest + && recovery.directory_digest == hex::encode(unchanged.digest()) + && recovery.publication_digest.is_none() + && recovery.archive_sources() == parent_groups + && recovery.children == [transition_group(child.history)] + ); + let missing = hex::encode([249; 16]); + ensure!( + matches!(control.transition_status(&namespace, &missing, TIMEOUT).await, + Err(failure) if failure.code == RaftFleetFailureCode::NotFound) + ); + let candidate = &cluster.hosts[0]; + let attached = candidate + .attach_managed_transfer_child(transition.clone(), child.clone(), TIMEOUT) + .await; + ensure!( + matches!(attached, Err(CellGroupControllerError::Catalog(ref cause)) + if matches!(cause.as_ref(), CatalogError::Unconfirmed)), + "uninstalled merge child must refuse attachment: {attached:?}" + ); + ensure!(matches!( + candidate.group(child.history), + Err(RuntimeError::NotFound) + )); + + for (index, generation, previous) in [(4, 125, fourth), (5, 126, fifth)] { + Box::pin(start_voter(cluster, index, generation, previous, 2)).await?; + } + let revision = fleet.read(TIMEOUT).await?.revision(); + for host in &cluster.hosts[4..] { + inspect_current(&cluster.hosts[3], host.identity().node, revision).await?; + } + leader(&cluster.hosts[3..], parents[1].history) + .await + .context("unsealed merge parent did not regain quorum")?; + verify_durable_seal(&cluster.hosts[3], &transition, parent.history, &seal).await?; + } + } + let fixture = &cluster.fixtures[0]; + let (resumed, reopened) = controller; + let holds = resumed + .staged_managed_sealed_sources(&reopened, operation, TIMEOUT) + .await?; + ensure!(holds.len() == parents.len() && holds.iter().all(|(_, hold)| hold.is_some())); + drop(holds); + let recovery = preparation::recovery_pool()?; + ensure!(matches!( + resumed + .resume_managed_transfer_archive( + &reopened, + operation, + &verifier, + recovery.clone(), + TIMEOUT, + ) + .await, + Err(FleetCatalogError::Conflict) + )); + let store = &fixture.store; + let denied_before = store.denied_group_payload_reads.load(Ordering::Acquire); + store + .deny_group_payload_reads + .store(true, Ordering::Release); + let denied = resumed + .replay_managed_sealed_sources(&reopened, operation, &verifier, TIMEOUT) + .await; + store + .deny_group_payload_reads + .store(false, Ordering::Release); + ensure!( + denied.is_err() && store.denied_group_payload_reads.load(Ordering::Acquire) > denied_before + ); + let pending = resumed + .reopen_managed_transition(&reopened, operation, TIMEOUT) + .await? + .context("reopened after denied source payload")?; + ensure!(pending + .journal + .is_some_and(|journal| journal.phase() == TransitionPhase::SealAuthorized + && journal.sealed_sources().is_none())); + let sources = resumed + .replay_managed_sealed_sources(&reopened, operation, &verifier, TIMEOUT) + .await + .context("select both staged cuts on a replacement controller")?; + ensure!(sources.journal().phase() == TransitionPhase::Sealed); + let sealed_digest = sources + .journal() + .sealed_sources() + .context("sealed source digest")?; + let transition = Arc::new( + resumed + .reopen_managed_transition(&reopened, operation, TIMEOUT) + .await? + .context("reopened sealed merge")?, + ); + ensure!( + transition + .journal + .as_ref() + .and_then(|journal| journal.sealed_sources()) + == Some(sealed_digest) + ); + let operator_install = RaftFleetTransitionChildSetupRequest { + catalog_id: control.catalog_id().into(), + namespace_id: hex::encode(child.history.namespace.bytes()), + operation_id: hex::encode(operation.bytes()), + request_id: hex::encode([178; 16]), + plan_digest: hex::encode(plan.digest()), + child: transition_group(child.history), + step: RaftFleetTransitionChildStep::Install, + }; + let operator_workspace = tempfile::tempdir()?; + let operator_verifier = + crate::managed_fixture::verifier::open(fixture, operator_workspace.path(), rooted, limits) + .await?; + let operator_voter = &cluster.hosts[0]; + let installed = Box::pin(crate::managed_fixture::operator::setup_transition_child( + fixture, + &reopened, + &request.principal, + operator_voter, + &operator_install, + Some((operator_verifier, recovery.clone())), + )) + .await?; + ensure!(installed.request == operator_install); + ensure!(matches!( + installed.result, + RaftFleetTransitionChildSetupResult::Installed { member: 1, ref node_id, .. } + if node_id == &hex::encode(operator_voter.identity().node.bytes()) + )); + ensure!(matches!( + operator_voter.group(child.history)?.status().await, + Err(RuntimeError::NotFound) + )); + for host in &cluster.hosts[1..3] { + host.install_managed_transfer_child( + transition.clone(), + sources.clone(), + child.history, + recovery.clone(), + TIMEOUT, + ) + .await + .context("install merged child")?; + } + drop(sources); + drop(verifier); + drop(sealed_workspace); + let voters = &cluster.hosts[..3]; + let operator_attach = RaftFleetTransitionChildSetupRequest { + request_id: hex::encode([179; 16]), + step: RaftFleetTransitionChildStep::Attach, + ..operator_install + }; + let attached = Box::pin(crate::managed_fixture::operator::setup_transition_child( + fixture, + &reopened, + &request.principal, + operator_voter, + &operator_attach, + None, + )) + .await?; + ensure!( + attached.request == operator_attach + && attached.result == RaftFleetTransitionChildSetupResult::Attached + ); + for host in &voters[1..] { + host.attach_managed_transfer_child(transition.clone(), child.clone(), TIMEOUT) + .await + .context("attach merged child")?; + } + let (child_leader, _) = preparation::wait_pending_child_witness(voters, child.history).await?; + let quorum_request = RaftFleetTransitionChildSetupRequest { + request_id: hex::encode([181; 16]), + step: RaftFleetTransitionChildStep::Quorum, + ..operator_attach + }; + let quorum = Box::pin(crate::managed_fixture::operator::setup_transition_child( + fixture, + &reopened, + &request.principal, + &voters[child_leader], + &quorum_request, + None, + )) + .await?; + ensure!(quorum.request == quorum_request); + ensure!(matches!( + quorum.result, + RaftFleetTransitionChildSetupResult::Quorum { + phase: TransitionPhase::Installed, + .. + } + )); + let archive_workspace = tempfile::tempdir()?; + let verifier = + crate::managed_fixture::verifier::open(fixture, archive_workspace.path(), rooted, limits) + .await?; + let verify_request = RaftFleetTransitionPrepareRequest { + catalog_id: quorum_request.catalog_id.clone(), + namespace_id: quorum_request.namespace_id.clone(), + operation_id: quorum_request.operation_id.clone(), + request_id: hex::encode([182; 16]), + plan_digest: quorum_request.plan_digest.clone(), + }; + let verified = Box::pin(crate::managed_fixture::operator::verify_transition_archive( + fixture, + &reopened, + &request.principal, + &verify_request, + (verifier, recovery), + "verify-archive", + )) + .await?; + ensure!(verified.request == verify_request && verified.phase == TransitionPhase::Archived); + let archived = resumed + .reopen_managed_transition(&reopened, operation, TIMEOUT) + .await? + .context("reopen verified merge")? + .journal + .context("archived merge journal")?; + ensure!( + archived.phase() == TransitionPhase::Archived + && archived.sealed_sources() == Some(sealed_digest) + && archived.transfer_archive().map(hex::encode).as_deref() + == Some(verified.archive_digest.as_str()) + ); + let namespace = hex::encode(child.history.namespace.bytes()); + let operation_id = hex::encode(operation.bytes()); + let principal = &transition.request.principal; + let expected_lineages: Vec<_> = parents + .iter() + .flat_map(|parent| { + cluster.fixtures[3..] + .iter() + .enumerate() + .map(|(index, fixture)| (parent.history, index, fixture.spec.clone())) + }) + .chain( + cluster.fixtures[..3] + .iter() + .enumerate() + .map(|(index, fixture)| (child.history, index, fixture.spec.clone())), + ) + .collect(); + let recovery_workspace = tempfile::tempdir()?; + let recovery_verifier = + crate::managed_fixture::verifier::open(fixture, recovery_workspace.path(), rooted, limits) + .await?; + let recovery_control = control + .clone() + .with_transition_preparer(recovery_verifier.clone(), preparation::recovery_pool()?); + let parent_basis = + Box::pin(control.transition_recovery_basis(&namespace, &operation_id, TIMEOUT)).await?; + let mut parent_holds = Vec::new(); + for registry in ®istries { + parent_holds.push(preparation::acquire_hold(domain, registry).await?); + } + ensure!(matches!( + Box::pin(recovery_control.verify_transition_recovery_archives( + principal, &parent_basis, vec![parent_holds[0].clone()], TIMEOUT, + )) + .await, + Err(failure) if failure.code == RaftFleetFailureCode::InvalidInput + )); + let recovery_operation = hex::encode([207; 16]); + let parent_intent = { + let restored = Box::pin(recovery_control.verify_transition_recovery_archives( + principal, + &parent_basis, + parent_holds.clone(), + TIMEOUT, + )) + .await + .context("verify selected parent archives before publication")?; + assert_recovery_lineages(&restored, &expected_lineages)?; + ensure!( + restored.basis() == &parent_basis + && restored + .archives() + .iter() + .map(|proof| proof.hold()) + .eq(&parent_holds) + ); + fixture + .store + .fail_recovery_record_confirmation + .store(true, Ordering::Release); + let unknown = Box::pin(recovery_control.stage_transition_recovery_intent( + principal, + &recovery_operation, + &restored, + TIMEOUT, + )) + .await; + ensure!( + matches!(unknown, Err(failure) if failure.code == RaftFleetFailureCode::Unavailable) + ); + ensure!(!fixture + .store + .fail_recovery_record_confirmation + .load(Ordering::Acquire)); + let staged = Box::pin(recovery_control.stage_transition_recovery_intent( + principal, + &recovery_operation, + &restored, + TIMEOUT, + )) + .await?; + ensure!(staged.operation() == [207; 16] && staged.transition() == operation.bytes()); + ensure!(staged.archive_holds() == parent_holds); + ensure!(staged.required_voter_lineages() == restored.required_voter_lineages()); + staged + }; + let status = control + .transition_status(&namespace, &operation_id, TIMEOUT) + .await?; + ensure!(status.phase == Some(TransitionPhase::Archived)); + let archive = archived + .transfer_archive() + .context("merged archive closure")?; + let publish = RaftFleetTransitionPublishRequest { + catalog_id: status.catalog_id, + namespace_id: namespace.clone(), + operation_id: operation_id.clone(), + request_id: hex::encode([218; 16]), + plan_digest: status.plan_digest, + archive_digest: status + .transfer_archive_digest + .context("archived merge cut")?, + }; + ensure!(publish.archive_digest == hex::encode(archive)); + if recovery_barrier { + let birth = domain + .run(directory.materialize_selected_history(parents[0].history)) + .await??; + let wrong = Box::pin(control.select_transition_recovery_barrier( + principal, + &namespace, + &recovery_operation, + [99; 32], + TIMEOUT, + )) + .await; + ensure!( + matches!(wrong, Err(failure) if failure.code == RaftFleetFailureCode::PayloadConflict) + ); + let work = control.select_transition_recovery_barrier( + principal, + &namespace, + &recovery_operation, + parent_intent.digest(), + TIMEOUT, + ); + let (unknown, delivered) = Box::pin(fault::lose_directory_confirmation(work)).await; + ensure!( + delivered + && matches!(unknown, Err(failure) if failure.code == RaftFleetFailureCode::Unavailable) + ); + let (fresh_catalog, fresh_directory) = reopen_merge_controller(fixture).await?; + let fresh = RaftFleetControl::new(domain, fresh_catalog, vec![fresh_directory])?; + let confirmed = Box::pin(fresh.select_transition_recovery_barrier( + principal, + &namespace, + &recovery_operation, + parent_intent.digest(), + TIMEOUT, + )) + .await?; + ensure!(confirmed.intent_digest() == parent_intent.digest()); + let locked = domain.run(directory.observe()).await??; + ensure!(locked.digest() == confirmed.directory_digest()); + let authority = TransitionFence::own( + domain.clone(), + &cluster.fixtures, + &mut cluster.roots, + &mut cluster.nodes, + &mut cluster.hosts, + )?; + let foreign = Box::pin(recovery_control.fence_transition_recovery( + "foreign-operator", + &namespace, + &recovery_operation, + confirmed, + &authority, + TIMEOUT, + )) + .await; + ensure!(matches!(foreign, Err(failure) if failure.code == RaftFleetFailureCode::Conflict)); + ensure!(authority.fences.load(Ordering::Acquire) == 0); + fixture + .store + .fail_recovery_record_confirmation + .store(true, Ordering::Release); + let unknown = Box::pin(recovery_control.fence_transition_recovery( + principal, + &namespace, + &recovery_operation, + confirmed, + &authority, + TIMEOUT, + )) + .await; + ensure!( + matches!(unknown, Err(failure) if failure.code == RaftFleetFailureCode::Unavailable) + ); + ensure!(!fixture + .store + .fail_recovery_record_confirmation + .load(Ordering::Acquire)); + ensure!(authority.fences.load(Ordering::Acquire) == 1); + ensure!(authority.erased_disks().await == cluster.fixtures.len()); + drop(recovery_control); + drop(recovery_verifier); + domain + .blocking(move || recovery_workspace.close())? + .await??; + let cold_workspace = tempfile::tempdir()?; + let cold_verifier = + crate::managed_fixture::verifier::open(fixture, cold_workspace.path(), rooted, limits) + .await?; + let fresh = fresh.with_transition_preparer(cold_verifier, preparation::recovery_pool()?); + let fenced = Box::pin(fresh.fence_transition_recovery( + principal, + &namespace, + &recovery_operation, + confirmed, + &authority, + TIMEOUT, + )) + .await?; + let fenced = Arc::new(fenced); + ensure!(fenced.request().barrier() == confirmed); + ensure!(fenced.request().intent().digest() == parent_intent.digest()); + ensure!( + fenced.request().intent().required_voter_lineages() + == parent_intent.required_voter_lineages() + ); + ensure!(fenced.receipt_digest() != [0; 32]); + ensure!(fenced + .archives() + .iter() + .map(|cut| cut.hold()) + .eq(parent_holds.iter())); + ensure!(authority.fences.load(Ordering::Acquire) == 1); + ensure!(authority.verifications.load(Ordering::Acquire) == 2); + let cut = fleet.read(TIMEOUT).await?; + Box::pin(fence::check_old_boot_refused(fixture)).await?; + ensure!(fleet.read(TIMEOUT).await?.revision() == cut.revision()); + ensure!(matches!( + domain + .run(directory.require_current_group_birth(&birth)) + .await?, + Err(cellar::application::consensus::CatalogError::Fenced) + )); + let refused = Box::pin(control.publish_transition( + principal, + &namespace, + &operation_id, + &publish, + TIMEOUT, + )) + .await; + ensure!(matches!(refused, Err(failure) if failure.code == RaftFleetFailureCode::Conflict)); + let observed = control + .transition_status(&namespace, &operation_id, TIMEOUT) + .await?; + ensure!(observed.phase == Some(TransitionPhase::Archived) && observed.fleet_slot_held); + let page = domain.run(directory.group_page(&locked, None, 2)).await??; + ensure!(page.routes == parents && page.next.is_none()); + let fixture = fixture.clone(); + // Replacement is a separate lifecycle phase with its own polling + // stack. Keep the cold verifier workspace alive until it joins. + let replacement = domain.spawn(Box::pin(async move { + Box::pin(check_recovery_replacements( + &fixture, + &control, + &fresh, + &namespace, + &fenced, + &RecoveryScenario { + routed: &routed, + receipts: &receipts, + authority: &authority, + }, + rooted, + )) + .await + }))?; + replacement.await??; + return Ok(None); + } + for (registry, hold) in registries.iter().zip(&parent_holds) { + registry.release(domain.as_ref(), hold).await?; + } + let work = control.publish_transition(principal, &namespace, &operation_id, &publish, TIMEOUT); + let (attempt, delivered) = Box::pin(fault::lose_directory_confirmation(work)).await; + ensure!( + delivered, + "the publication CAS did not reach the injected confirmation loss" + ); + if let Err(failure) = attempt { + ensure!( + failure.code == RaftFleetFailureCode::Unavailable + && failure.outcome == RaftFleetOutcome::Unknown + ); + } + let after = domain.run(directory.observe()).await??; + let recovery = + Box::pin(control.transition_recovery_basis(&namespace, &operation_id, TIMEOUT)).await?; + ensure!( + recovery.phase == TransitionPhase::Published + && recovery.directory_digest == hex::encode(after.digest()) + && recovery.transfer_archive_digest.as_deref() == Some(publish.archive_digest.as_str()) + && recovery.child_archives_digest.is_none() + && recovery.archive_sources() == [transition_group(child.history)] + && recovery.parents.len() == 2 + ); + let published = control + .publish_transition(principal, &namespace, &operation_id, &publish, TIMEOUT) + .await + .context("reopen merged publication after losing its first confirmation")?; + ensure!( + published.directory_revision == view.revision() + 1 + && recovery.publication_digest.as_deref() + == Some(published.publication_digest.as_str()) + ); + let page = domain.run(directory.group_page(&after, None, 2)).await??; + ensure!(page.next.is_none() && page.routes == vec![child.clone()]); + let ack = preparation::wait_managed_activation(voters, &transition, child.history).await?; + ensure!( + fleet + .record_managed_transfer_activations(&transition, directory, vec![ack], TIMEOUT,) + .await? + .phase() + == TransitionPhase::Active + ); + Box::pin(verify_merged_sql( + voters, + child.history, + &routed, + receipts, + retry, + )) + .await?; + Box::pin(initialize_merge_child_archive( + cluster, + directory, + &transition, + child.history, + &control, + )) + .await?; + let old_scope = Box::pin(recovery_control.verify_transition_recovery_archives( + principal, + &parent_basis, + parent_holds, + TIMEOUT, + )) + .await; + ensure!(matches!(old_scope, Err(failure) if failure.code == RaftFleetFailureCode::Conflict)); + let child_birth = domain + .run(directory.materialize_selected_history(child.history)) + .await??; + let child_registry = directory + .open_group_archive(domain.as_ref(), &archives.pool, &child_birth) + .await?; + let voters = &cluster.hosts[..3]; + let archive_leader = &voters[leader(voters, child.history).await?]; + preparation::wait_active_archive(archive_leader, child.history).await?; + let older = preparation::acquire_hold(domain, &child_registry).await?; + let (cell, expected) = &routed[0]; + let handle = archive_leader.cell(cell.clone())?; + let s3_receipt = handle + .command( + &handle.contract("Add")?, + 0i64.to_be_bytes().to_vec(), + CommandOptions { + durability: cellar_logic::consensus::Durability::S3, + timeout: TIMEOUT, + ..CommandOptions::new(CommandId::new(ClientId::new([211; 16])?, 1)?, retry) + }, + ) + .await?; + ensure!( + s3_receipt.result == (expected + 2).to_be_bytes() + && s3_receipt.durability == cellar_logic::consensus::Durability::S3 + ); + let applied = preparation::wait_active_archive(archive_leader, child.history).await?; + let active_basis = + Box::pin(control.transition_recovery_basis(&namespace, &operation_id, TIMEOUT)).await?; + ensure!(matches!( + Box::pin(recovery_control.verify_transition_recovery_archives( + principal, &active_basis, vec![older.clone()], TIMEOUT, + )) + .await, + Err(failure) if failure.code == RaftFleetFailureCode::Conflict + )); + let mut current = preparation::acquire_hold(domain, &child_registry).await?; + ensure!( + current.marker().position().index >= applied + && current.marker().position().index > older.marker().position().index + ); + { + // During selection, a producer can replace the archive descriptor + // for the same Raft cut. Keep the selected logical cut and Fleet + // basis fixed while reacquiring an exact hold before staging intent. + let mut replacements = 0; + let restored = loop { + let result = Box::pin(recovery_control.verify_transition_recovery_archives( + principal, + &active_basis, + vec![current.clone()], + TIMEOUT, + )) + .await; + match result { + Ok(restored) => break restored, + Err(failure) + if failure.code == RaftFleetFailureCode::Conflict && replacements < 3 => + { + let next = preparation::acquire_hold(domain, &child_registry).await?; + ensure!( + next.marker() == current.marker(), + "the selected child Raft cut changed during verification" + ); + child_registry.release(domain.as_ref(), ¤t).await?; + current = next; + replacements += 1; + } + Err(error) => { + return Err(anyhow::Error::from(error) + .context("verify the selected child archive after S3 acknowledgement")); + } + } + }; + assert_recovery_lineages(&restored, &expected_lineages)?; + ensure!(matches!( + Box::pin(recovery_control.stage_transition_recovery_intent( + principal, &recovery_operation, &restored, TIMEOUT, + )).await, + Err(failure) if failure.code == RaftFleetFailureCode::PayloadConflict + )); + let (fresh_catalog, reopened_directory) = + reopen_merge_controller(&cluster.fixtures[0]).await?; + let fresh = RaftFleetControl::new(domain, fresh_catalog, vec![reopened_directory])?; + let reopened = fresh + .reopen_transition_recovery_intent(principal, &namespace, &recovery_operation, TIMEOUT) + .await? + .context("reopen recovery intent after transfer publication")?; + ensure!( + reopened.digest() == parent_intent.digest() + && reopened.archive_holds() == parent_intent.archive_holds() + && reopened.required_voter_lineages() == parent_intent.required_voter_lineages() + ); + let [archive] = restored.archives() else { + anyhow::bail!("expected exactly one selected child archive"); + }; + ensure!(archive.hold() == ¤t); + let manifest = archive.recovery().manifest(); + ensure!(manifest.pinned_images() == routed.len()); + for (cell, _) in &routed { + let (leaf, _) = manifest + .pin_cell(&cell.cell) + .context("restored merged cell")?; + ensure!(leaf.version.history == *cell); + if leaf.version.history == s3_receipt.version.history { + ensure!(leaf.version == s3_receipt.version); + } + } + } + for hold in [&older, ¤t] { + child_registry.release(domain.as_ref(), hold).await?; + } + finish_merge( + cluster, + directory, + &transition, + &parents, + child.history, + &archives, + &control, + ) + .await?; + let historical = control + .publish_transition(principal, &namespace, &operation_id, &publish, TIMEOUT) + .await?; + ensure!(historical == published); + ensure!(matches!( + Box::pin(control.transition_recovery_basis(&namespace, &operation_id, TIMEOUT)).await, + Err(failure) if failure.code == RaftFleetFailureCode::Conflict + )); + let voters = &cluster.hosts[..3]; + let host = &voters[leader(voters, child.history).await?]; + for (index, (cell, expected)) in routed.into_iter().enumerate() { + let handle = host.cell(cell)?; + let query = handle + .query( + ProposalId::new([218 + index as u8; 16])?, + &handle.contract("Get")?, + vec![], + QueryOptions::default(), + ) + .await?; + ensure!(query.result == (expected + 2).to_be_bytes()); + } + Ok(Some(child.history)) +} + +struct RecoveryScenario<'a> { + routed: &'a [(CellHistory, i64)], + receipts: &'a [CommandReceipt], + authority: &'a TransitionFence, +} + +async fn check_recovery_replacements( + fixture: &Fixture, + control: &RaftFleetControl, + fresh: &RaftFleetControl, + namespace: &str, + fenced: &Arc, + sql: &RecoveryScenario<'_>, + rooted: bool, +) -> Result<()> { + let routed = sql.routed; + let epoch = fenced + .archives() + .iter() + .map(|cut| cut.marker().route().epoch.get()) + .max() + .context("recovery epoch")?; + let replacements = fenced + .archives() + .iter() + .enumerate() + .map(|(index, cut)| { + let route = cut.marker().route(); + let id = 170 + index as u8; + Ok(CellGroupBirthSpec { + history: GroupHistory { + group: GroupId::new([id; 16])?, + incarnation: IncarnationId::new([id + 10; 16])?, + ..route.history + }, + range: route.range.clone(), + epoch: RouteEpoch::new(epoch + 1)?, + creation: ProposalId::new([id + 20; 16])?, + voters: (0..3) + .map(|offset| { + Ok(InitialMember { + node: NodeId::new([70 + offset; 16])?, + member: MemberId::new(21 + u64::from(offset))?, + }) + }) + .collect::>()?, + learners: vec![], + }) + }) + .collect::>>()?; + let mut unsafe_roster = replacements.clone(); + unsafe_roster[0].voters[0].node = fenced.request().intent().required_voter_lineages()[0].node; + let refused = Box::pin(control.stage_transition_recovery_replacement( + namespace, + fenced, + unsafe_roster, + TIMEOUT, + )) + .await; + ensure!(matches!(refused, Err(failure) if failure.code == RaftFleetFailureCode::Conflict)); + fixture + .store + .fail_recovery_record_confirmation + .store(true, Ordering::Release); + let unknown = Box::pin(control.stage_transition_recovery_replacement( + namespace, + fenced, + replacements.clone(), + TIMEOUT, + )) + .await; + ensure!(matches!(unknown, Err(failure) if failure.code == RaftFleetFailureCode::Unavailable)); + let plan = Box::pin(fresh.reopen_transition_recovery_replacement(namespace, fenced, TIMEOUT)) + .await? + .context("reopen selected recovery replacements")?; + let plan = Arc::new(plan); + let repeated = Box::pin(control.stage_transition_recovery_replacement( + namespace, + fenced, + replacements.clone(), + TIMEOUT, + )) + .await?; + ensure!(plan.digest() == repeated.digest()); + ensure!(plan.mutation().operation().bytes() == fenced.request().intent().operation()); + let mut changed = replacements; + changed[0].creation = ProposalId::new([215; 16])?; + let conflict = Box::pin( + control.stage_transition_recovery_replacement(namespace, fenced, changed, TIMEOUT), + ) + .await; + ensure!( + matches!(conflict, Err(failure) if failure.code == RaftFleetFailureCode::PayloadConflict) + ); + + let first = plan.births()[0].route().history; + let absent = + Box::pin(fresh.reserve_transition_recovery_range(namespace, fenced, &plan, first, TIMEOUT)) + .await; + ensure!(matches!(absent, Err(failure) if failure.code == RaftFleetFailureCode::Conflict)); + let workspace = tempfile::tempdir()?; + let mut checked = 0; + for (index, (birth, cut)) in plan.births().iter().zip(fenced.archives()).enumerate() { + let base = workspace.path().join(index.to_string()); + let route = birth.route().clone(); + let target = fixture + .domain + .blocking(move || preparation::store(base, route, rooted))? + .await??; + let restored = Box::pin( + fresh.prepare_transition_recovery_range(namespace, fenced, &plan, target, TIMEOUT), + ) + .await?; + ensure!(restored.plan_digest() == plan.digest() && restored.commitment() != [0; 32]); + let marker = restored.manifest().marker(); + ensure!( + marker.route() == birth.route() + && marker.position().index.get() == 1 + && !marker.is_active() + ); + for (cell, _) in routed { + if let Some((old, before)) = cut.recovery().manifest().pin_cell(&cell.cell) { + let (new, after) = restored + .manifest() + .pin_cell(&cell.cell) + .context("restored cell")?; + ensure!(new.version.history.incarnation == birth.route().history.incarnation); + ensure!(old.version.history.incarnation != new.version.history.incarnation); + ensure!( + new.version.sequence.get() == 1 && new.witness.group == birth.route().history + ); + ensure!(new.version.image.checksum == old.version.image.checksum); + fixture + .domain + .blocking(move || check_restored_image(before, after))? + .await??; + checked += 1; + } + } + } + ensure!(checked == routed.len()); + Box::pin(check_recovery_voter_reservations( + fixture, control, fresh, namespace, fenced, &plan, sql, + )) + .await?; + + Ok(()) +} + +async fn check_recovery_voter_reservations( + fixture: &Fixture, + control: &RaftFleetControl, + fresh: &RaftFleetControl, + namespace: &str, + fenced: &Arc, + plan: &Arc, + sql: &RecoveryScenario<'_>, +) -> Result<()> { + let routed = sql.routed; + let (_, directory) = reopen_merge_controller(fixture).await?; + let mut standbys = Vec::new(); + let checked = Box::pin(async { + let mut boots = Vec::new(); + for (id, capacity_request) in [(70, 190), (71, 192), (72, 194)] { + let peer = peer_fixture(fixture, id).await?; + let root = tempfile::tempdir()?; + let voter = start_recovery_voter(peer, root, LogOpenMode::Create, id + 100).await?; + standbys.push(voter); + let (_, _, host, peer) = standbys.last().context("started recovery voter")?; + Box::pin(admit_capacity(peer, host, &directory, capacity_request)).await?; + let view = host.observe_node(TIMEOUT).await?; + ensure!(view.node.groups == 0); + boots.push(view.fleet_boot.context("new recovery voter boot")?); + } + + let first = plan.births()[0].route().history; + fixture + .store + .fail_fleet_confirmation + .store(true, Ordering::Release); + let unknown = Box::pin( + control.reserve_transition_recovery_range(namespace, fenced, plan, first, TIMEOUT), + ) + .await; + ensure!( + matches!(unknown, Err(failure) if failure.code == RaftFleetFailureCode::Unavailable) + ); + ensure!(!fixture + .store + .fail_fleet_confirmation + .load(Ordering::Acquire)); + let mut previous = 0; + let mut reserved = Vec::new(); + for birth in plan.births() { + let proof = Box::pin(fresh.reserve_transition_recovery_range( + namespace, + fenced, + plan, + birth.route().history, + TIMEOUT, + )) + .await?; + ensure!(proof.route() == birth.route() && proof.plan_digest() == plan.digest()); + ensure!(proof.voter_boots() == boots.as_slice()); + ensure!(proof.applied_revision() > previous); + previous = proof.applied_revision(); + reserved.push(Arc::new(proof)); + } + let replayed = Box::pin( + control.reserve_transition_recovery_range(namespace, fenced, plan, first, TIMEOUT), + ) + .await?; + ensure!( + replayed.applied_revision() < previous && replayed.voter_boots() == boots.as_slice() + ); + let revision = fixture.catalog.read(TIMEOUT).await?.revision(); + let recovery = preparation::recovery_pool()?; + let mut quorum_digest = None; + for (voter_index, (_, _, host, _)) in standbys.iter().enumerate() { + let node = NodeId::new(boots[voter_index].node)?; + inspect_current(host, node, revision).await?; + for (range_index, reservation) in reserved.iter().enumerate() { + let prepared = Box::pin(host.prepare_managed_recovery_seed( + fenced, + plan, + reservation, + &recovery, + TIMEOUT, + )) + .await?; + if voter_index == 0 && range_index == 0 { + fixture + .store + .fail_recovery_record_confirmation + .store(true, Ordering::Release); + let lost = Box::pin(host.seed_managed_recovery_range( + fenced.clone(), + plan.clone(), + reservation.clone(), + prepared.clone(), + TIMEOUT, + )) + .await; + ensure!(lost.is_err()); + ensure!(!fixture + .store + .fail_recovery_record_confirmation + .load(Ordering::Acquire)); + ensure!(host.observe_node(TIMEOUT).await?.node.groups == 1); + } + let receipt = Box::pin(host.seed_managed_recovery_range( + fenced.clone(), + plan.clone(), + reservation.clone(), + prepared.clone(), + TIMEOUT, + )) + .await?; + ensure!( + receipt.route == *reservation.route() && receipt.plan_digest == plan.digest() + ); + ensure!(receipt.genesis_commitment == prepared.commitment()); + ensure!( + receipt.boot == boots[voter_index] + && receipt.member.get() == 21 + voter_index as u64 + ); + if voter_index == 1 && range_index == 0 { + let incomplete = Box::pin(control.select_transition_recovery_seed_quorums( + namespace, fenced, plan, TIMEOUT, + )).await; + ensure!(matches!(incomplete, Err(failure) if failure.code == RaftFleetFailureCode::Unavailable)); + } + } + if voter_index == 1 { + fixture.store.fail_recovery_record_confirmation.store(true, Ordering::Release); + let unknown = Box::pin(control.select_transition_recovery_seed_quorums( + namespace, fenced, plan, TIMEOUT, + )).await; + ensure!(matches!(unknown, Err(failure) if failure.code == RaftFleetFailureCode::Unavailable)); + ensure!(!fixture.store.fail_recovery_record_confirmation.load(Ordering::Acquire)); + let selected = Box::pin(fresh.select_transition_recovery_seed_quorums( + namespace, fenced, plan, TIMEOUT, + )).await?; + ensure!(selected.plan_digest() == plan.digest() && selected.ranges().len() == reserved.len()); + ensure!(selected.ranges().iter().zip(plan.births()).all(|(range, birth)| { + range.birth_digest() == birth.route().birth_digest + && range.members() == [21, 22] + && range.genesis_commitment() != [0; 32] + })); + quorum_digest = Some(selected.digest()); + } + let view = host.observe_node(TIMEOUT).await?; + ensure!(view.node.groups == reserved.len() as u32 && view.registered_histories == 0); + } + let reopened = Box::pin(control.select_transition_recovery_seed_quorums( + namespace, fenced, plan, TIMEOUT, + )).await?; + ensure!(Some(reopened.digest()) == quorum_digest); + ensure!(reopened.ranges().iter().all(|range| range.members() == [21, 22])); + let request = FleetRecoveryAttachRequest { + operation: ProposalId::new(fenced.request().intent().operation())?, + principal: "managed-test".into(), + plan_digest: plan.digest(), + seed_digest: reopened.digest(), + }; + Box::pin(restart_recovery_voter(&mut standbys, 1, 197, reserved.len(), (false, false))) + .await.context("restart a selected recovery voter before publication")?; + let (_, _, host, peer) = &standbys[1]; + let view = host.observe_node(TIMEOUT).await?; + let new_boot = view.fleet_boot.context("restarted recovery voter boot")?; + ensure!( + new_boot.disk == boots[1].disk && new_boot.engine == boots[1].engine + && new_boot.process != boots[1].process && new_boot.boot != boots[1].boot + ); + let unqualified = Box::pin(fresh.select_transition_recovery_seed_quorums( + namespace, fenced, plan, TIMEOUT, + )).await; + ensure!(matches!(unqualified, Err(failure) if failure.code == RaftFleetFailureCode::Unavailable)); + Box::pin(admit_capacity(peer, host, &directory, 197)).await?; + let confirmed = Box::pin(fresh.select_transition_recovery_seed_quorums( + namespace, fenced, plan, TIMEOUT, + )).await?; + ensure!(confirmed.digest() == reopened.digest()); + let first_host = &standbys[0].2; + let premature = Box::pin(first_host.attach_managed_recovery_voter( + request.clone(), first, TIMEOUT, + )).await; + ensure!(premature.is_err() && first_host.observe_node(TIMEOUT).await?.registered_histories == 0); + let locked = fixture.domain.run(directory.observe()).await??; + let work = control.publish_transition_recovery(namespace, fenced, plan, &reopened, TIMEOUT); + let (unknown, delivered) = Box::pin(fault::lose_directory_confirmation(work)).await; + ensure!(delivered && matches!(unknown, Err(failure) if failure.code == RaftFleetFailureCode::Unavailable)); + let operation = hex::encode(request.operation.bytes()); + let expected = (request.plan_digest, request.seed_digest); + let publication = Box::pin(fresh.reopen_transition_recovery_publication( + namespace, &operation, &request.principal, expected, TIMEOUT, + )).await?.context("reopen recovery publication by durable operation")?; + let replayed = Box::pin(control.publish_transition_recovery( + namespace, fenced, plan, &reopened, TIMEOUT, + )).await?; + ensure!(replayed == publication); + let after = fixture.domain.run(directory.observe()).await??; + ensure!(after.revision() == locked.revision() + 1 && publication.revision() == after.revision()); + let page = fixture.domain.run(directory.group_page(&after, None, 2)).await??; + ensure!(page.next.is_none() && page.routes.iter().eq(plan.births().iter().map(|birth| birth.route()))); + for birth in plan.births() { + let denied = fixture.domain.run(directory.materialize_selected_history(birth.route().history)).await?; + ensure!(matches!(denied, Err(cellar::application::consensus::CatalogError::Fenced))); + } + let revision = fixture.catalog.read(TIMEOUT).await?.revision(); + let hosts: Vec<_> = standbys.iter().map(|(_, _, host, _)| host).collect(); + for host in &hosts { + for peer in &hosts { + inspect_current(host, peer.identity().node, revision).await?; + } + } + let wrong = FleetRecoveryAttachRequest { seed_digest: [1; 32], ..request.clone() }; + let denied = Box::pin(first_host.attach_managed_recovery_voter(wrong, first, TIMEOUT)).await; + ensure!(denied.is_err() && first_host.observe_node(TIMEOUT).await?.registered_histories == 0); + let mut activations = Vec::new(); + for (birth, archive) in plan.births().iter().zip(fenced.archives()) { + let history = birth.route().history; + Box::pin(check_pending_recovery_quorum(&hosts, &request, history)).await?; + let absent = Box::pin(fresh.reopen_transition_recovery_activation( + namespace, &request, history, TIMEOUT, + )).await?; + ensure!(absent.is_none()); + let activated = Box::pin(check_managed_recovery_activation( + &hosts, &request, history, &fixture.store, + )).await?; + let reopened = Box::pin(fresh.reopen_transition_recovery_activation( + namespace, &request, history, TIMEOUT, + )).await?; + ensure!(reopened == Some(activated)); + activations.push((history, activated)); + if activations.len() < plan.births().len() { + let before = fixture.catalog.read(TIMEOUT).await?; + let refused = Box::pin(fresh.advance_transition_recovery_settlement( + namespace, &request, TIMEOUT, + )).await; + ensure!(matches!(refused, Err(failure) if failure.code == RaftFleetFailureCode::Unavailable)); + ensure!(fixture.catalog.read(TIMEOUT).await?.revision() == before.revision()); + } + let (cell, _) = routed.iter().find(|(cell, _)| { + archive.recovery().manifest().pin_cell(&cell.cell).is_some() + }).context("recovered range cell")?; + let cell = CellHistory { incarnation: history.incarnation, ..cell.clone() }; + let handle = first_host.cell(cell)?; + let read = handle.query( + request.operation, + &handle.contract("Get")?, + vec![], + QueryOptions { + consistency: QueryConsistency::Stale, + stale_policy: StaleReadPolicy::DisconnectedHistorical, + routing: QueryRouting::Local, + ..Default::default() + }, + ).await; + ensure!( + read.as_ref().is_err_and(|error| pending_recovery_route_fenced(error)), + "pending recovery did not refuse the stale read at routing: {read:?}" + ); + let retry = u64::try_from( + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH)? + .as_millis(), + )? + .checked_add(60_000) + .context("pending recovery command retry deadline")?; + let id = CommandId::new(ClientId::new([213; 16])?, 1)?; + let write = handle.command( + &handle.contract("Add")?, + 1i64.to_be_bytes().to_vec(), + CommandOptions::new(id, retry), + ).await; + ensure!( + matches!(write, Err(ref failure) if failure.command == id && !failure.ambiguous + && pending_recovery_route_fenced(&failure.cause)), + "pending recovery did not refuse the write before dispatch: {write:?}" + ); + } + let before_directory = fixture.domain.run(directory.observe()).await??; + let premature = Box::pin(fresh.record_transition_recovery_directory_completion( + namespace, &request, TIMEOUT, + )).await; + ensure!(matches!(premature, Err(failure) if failure.code == RaftFleetFailureCode::Unavailable)); + let unchanged = fixture.domain.run(directory.observe()).await??; + ensure!(unchanged.digest() == before_directory.digest()); + let before = fixture.catalog.read(TIMEOUT).await?; + fixture.store.fail_fleet_confirmation.store(true, Ordering::Release); + let unknown = Box::pin(fresh.advance_transition_recovery_settlement( + namespace, &request, TIMEOUT, + )).await; + ensure!(matches!(unknown, Err(failure) if failure.code == RaftFleetFailureCode::Unavailable)); + ensure!(!fixture.store.fail_fleet_confirmation.load(Ordering::Acquire)); + let settled = Box::pin(fresh.advance_transition_recovery_settlement( + namespace, &request, TIMEOUT, + )).await?; + let cut = fixture.catalog.read(TIMEOUT).await?; + Box::pin(fixture.catalog.check_inventory(&cut, std::slice::from_ref(&directory), TIMEOUT)) + .await.context("promoted recovery birth inventory")?; + ensure!(settled.promoted_ranges == plan.births().len() + && settled.total_ranges == plan.births().len() + && settled.publication_digest == publication.journal_digest() + && settled.applied_revision == before.revision() + 1 + && cut.revision() == settled.applied_revision); + for birth in plan.births() { + let history = birth.route().history; + let live = fixture.catalog.history(&cut, &history, TIMEOUT).await?; + ensure!(live.state() == &FleetHistoryState::Live); + let fenced = fixture.domain.run(directory.materialize_selected_history(history)).await?; + ensure!(matches!(fenced, Err(cellar::application::consensus::CatalogError::Fenced))); + } + let work = fresh.record_transition_recovery_directory_completion(namespace, &request, TIMEOUT); + let (lost, delivered) = Box::pin(fault::lose_directory_confirmation(work)).await; + ensure!(delivered && matches!(lost, Err(failure) if failure.code == RaftFleetFailureCode::Unavailable)); + let completion = Box::pin(control.record_transition_recovery_directory_completion( + namespace, &request, TIMEOUT, + )).await?; + ensure!(completion != [0; 32]); + let reopened = Box::pin(fresh.record_transition_recovery_directory_completion( + namespace, &request, TIMEOUT, + )).await?; + ensure!(reopened == completion); + ensure!(fixture.domain.run(directory.active_transition()).await??.is_none()); + let old_operation = hex::encode(fenced.request().intent().transition()); + let superseded = Box::pin(fresh.transition_status(namespace, &old_operation, TIMEOUT)).await?; + ensure!(superseded.recovery_completion_digest == Some(hex::encode(completion)) + && superseded.fleet_slot_held && !superseded.parent_holds_released); + let after_completion = fixture.domain.run(directory.observe()).await??; + ensure!(after_completion.revision() == before_directory.revision() + && after_completion.digest() != before_directory.digest()); + for birth in plan.births() { + let fenced = fixture.domain.run(directory.materialize_selected_history(birth.route().history)).await?; + ensure!(matches!(fenced, Err(cellar::application::consensus::CatalogError::Fenced))); + } + Box::pin(restart_recovery_voter(&mut standbys, 2, 199, reserved.len(), (true, false))) + .await.context("restart an activated recovery voter after directory completion")?; + let (_, _, restarted, peer) = &standbys[2]; + Box::pin(admit_capacity(peer, restarted, &directory, 199)).await?; + let revision = fixture.catalog.read(TIMEOUT).await?.revision(); + for (_, _, other, _) in &standbys { + inspect_current(restarted, other.identity().node, revision).await?; + inspect_current(other, restarted.identity().node, revision).await?; + } + for &(history, position) in &activations { + Box::pin(restarted.attach_managed_recovery_voter(request.clone(), history, TIMEOUT)).await?; + let group = restarted.group(history)?; + let status = group.status().await?; + ensure!(!status.ready && status.applied.index >= position.index); + let reopened = Box::pin(group.activate_managed_recovery(request.clone(), TIMEOUT)).await?; + ensure!(reopened == Some(position)); + } + + let refused = Box::pin(fresh.release_transition_recovery_serving( + namespace, &request, [1; 32], TIMEOUT, + )).await; + ensure!(refused.is_err()); + let still_fenced = fixture.domain.run(directory.observe()).await??; + ensure!(still_fenced.digest() == after_completion.digest()); + let premature = Box::pin(fresh.advance_transition_recovery_retirement( + namespace, &request, completion, sql.authority, TIMEOUT, + )).await; + ensure!(premature.is_err()); + for (controller, suffix) in [(fresh, "/ownership/registry"), (control, "/ownership/identity")] { + let release = controller.release_transition_recovery_serving(namespace, &request, completion, TIMEOUT); + let (denied, delivered) = Box::pin(fault::deny_archive_creation(suffix, release)).await; + ensure!(delivered && denied.is_err(), "archive initialization did not stop at {suffix}"); + let view = fixture.domain.run(directory.observe()).await??; + ensure!(view.digest() == after_completion.digest()); + } + let release = fresh.release_transition_recovery_serving(namespace, &request, completion, TIMEOUT); + let (unknown, delivered) = Box::pin(fault::lose_directory_confirmation(release)).await; + ensure!(delivered && matches!(unknown, Err(failure) if failure.code == RaftFleetFailureCode::Unavailable)); + let released = Box::pin(control.release_transition_recovery_serving( + namespace, &request, completion, TIMEOUT, + )).await?; + ensure!(released == completion); + let current = fixture.domain.run(directory.observe()).await??; + ensure!(current.revision() == after_completion.revision() + && current.digest() != after_completion.digest()); + let published = Box::pin(fresh.reopen_transition_recovery_publication( + namespace, &operation, &request.principal, expected, TIMEOUT, + )).await?; + ensure!(published.as_ref() == Some(&publication)); + for archive in fenced.archives() { + let history = archive.marker().route().history; + let old = fixture.domain.run(directory.materialize_selected_history(history)).await?; + ensure!(matches!(old, Err(cellar::application::consensus::CatalogError::Fenced))); + } + for &(history, position) in &activations { + let selected = fixture.domain.run(directory.materialize_selected_history(history)).await??; + ensure!(selected.route().history == history); + let reopened = Box::pin(fresh.reopen_transition_recovery_activation( + namespace, &request, history, TIMEOUT, + )).await?; + ensure!(reopened == Some(position)); + // Existing owners require a new attachment under the serving decision. + ensure!(!standbys[0].2.group(history)?.status().await?.ready); + } + + for (index, generation) in [201, 203, 205].into_iter().enumerate() { + Box::pin(restart_recovery_voter(&mut standbys, index, generation, reserved.len(), (true, true))) + .await.context("restart a released recovery voter")?; + let (_, _, restarted, peer) = &standbys[index]; + Box::pin(admit_capacity(peer, restarted, &directory, generation)).await?; + let revision = fixture.catalog.read(TIMEOUT).await?.revision(); + for (_, _, other, _) in &standbys { + inspect_current(restarted, other.identity().node, revision).await?; + inspect_current(other, restarted.identity().node, revision).await?; + } + } + Box::pin(check_recovered_sql(&standbys, plan, fenced, sql, &activations)).await?; + let before = fixture.catalog.read(TIMEOUT).await?; + let wrong_authority = TransitionFence::default(); + let rejected = Box::pin(fresh.advance_transition_recovery_retirement( + namespace, &request, completion, &wrong_authority, TIMEOUT, + )).await; + ensure!(rejected.is_err() + && wrong_authority.verifications.load(Ordering::Acquire) == 1 + && fixture.catalog.read(TIMEOUT).await?.revision() == before.revision(), + "retirement must reverify the external fence before a Fleet mutation: {rejected:?}"); + let verifications = sql.authority.verifications.load(Ordering::Acquire); + fixture.store.fail_fleet_confirmation.store(true, Ordering::Release); + let unknown = Box::pin(fresh.advance_transition_recovery_retirement( + namespace, &request, completion, sql.authority, TIMEOUT, + )).await; + let fault_delivered = !fixture.store.fail_fleet_confirmation.swap(false, Ordering::AcqRel); + let reverified = sql.authority.verifications.load(Ordering::Acquire) - verifications; + ensure!( + fault_delivered + && matches!(&unknown, Err(failure) if failure.code == RaftFleetFailureCode::Unavailable), + "retirement lost CAS returned {unknown:?}, fault delivered={fault_delivered}, fence verifications={reverified}" + ); + let retired = Box::pin(control.advance_transition_recovery_retirement( + namespace, &request, completion, sql.authority, TIMEOUT, + )).await?; + ensure!(retired.retired_histories == 3 && retired.total_histories == 3 + && retired.fleet_slot_released && retired.applied_revision == before.revision() + 1); + ensure!(sql.authority.verifications.load(Ordering::Acquire) == verifications + 1); + let cut = fixture.catalog.read(TIMEOUT).await?; + Box::pin(fixture.catalog.check_inventory(&cut, std::slice::from_ref(&directory), TIMEOUT)) + .await.context("recovery retirement inventory")?; + let old_histories: std::collections::BTreeSet<_> = fenced.request().intent() + .required_voter_lineages().iter().map(|lineage| lineage.group).collect(); + for history in old_histories { + let old = fixture.catalog.history(&cut, &history, TIMEOUT).await?; + ensure!(matches!(old.state(), FleetHistoryState::Retired { operation, .. } + if *operation == fenced.request().intent().transition())); + } + let status = Box::pin(fresh.transition_status(namespace, &old_operation, TIMEOUT)).await?; + ensure!(!status.fleet_slot_held && !status.parent_holds_released + && status.recovery_completion_digest == Some(hex::encode(completion))); + let release_replay = Box::pin(fresh.release_transition_recovery_serving( + namespace, &request, completion, TIMEOUT, + )).await?; + ensure!(release_replay == completion); + + Ok::<(), anyhow::Error>(()) + }) + .await; + let mut shutdown = Ok(()); + for (_root, node, host, peer) in standbys { + let stopped = node.shutdown(ShutdownMode::Preserve).await; + let closed = host.close().await; + let joined = (|| { + ensure!( + matches!(&stopped, Ok(NodeOutcome::Stopped(_))), + "node outcome {stopped:?}, Raft close {closed:?}" + ); + let report = closed?; + ensure!(report.groups_total == plan.births().len()); + ensure!( + report.groups.iter().all(|group| group.active + && !group.failed + && group.applied.index.get() > 2 + && group.committed == group.applied.index), + "managed recovery shutdown must preserve committed activation: {report:?}" + ); + Ok::<(), anyhow::Error>(()) + })() + .with_context(|| format!("join recovery voter {}", peer.spec.label)); + if let (Err(_), Err(error)) = (&checked, &joined) { + eprintln!("recovery voter cleanup after failure: {error:#}"); + } + shutdown = shutdown.and(joined); + } + checked?; + shutdown +} + +type RecoveryVoter = (tempfile::TempDir, Node, RaftCells, Fixture); + +async fn restart_recovery_voter( + voters: &mut Vec, + index: usize, + generation: u8, + ranges: usize, + expected: (bool, bool), +) -> Result<()> { + let (activated, serving) = expected; + let (root, node, host, peer) = voters.remove(index); + let previous = host.identity(); + ensure!(matches!( + node.shutdown(ShutdownMode::Preserve).await?, + NodeOutcome::Stopped(_) + )); + let report = host.close().await?; + ensure!( + report.groups_total == ranges + && report.omitted_groups == 0 + && report.groups.iter().all(|group| group.active == activated + && !group.failed + && group.committed == group.applied.index + && if activated { + group.applied.index.get() > 2 + } else { + group.applied.index.get() == 1 + }), + "closed recovery voter inventory: {report:?}" + ); + let voter = start_recovery_voter(peer, root, LogOpenMode::Resume, generation).await?; + voters.insert(index, voter); + let host = &voters[index].2; + ensure!(host.identity().disk == previous.disk && host.identity().process != previous.process); + let view = host.observe_node(TIMEOUT).await?; + ensure!(view.node.groups == ranges as u32); + ensure!(view.registered_histories == if serving { ranges as u32 } else { 0 }); + Ok(()) +} + +async fn check_recovered_sql( + voters: &[RecoveryVoter], + plan: &cellar::sdk::fleet_operations::FleetTransitionRecoveryReplacement, + fenced: &FleetFencedTransitionRecovery, + sql: &RecoveryScenario<'_>, + activations: &[(GroupHistory, LogPosition)], +) -> Result<()> { + use cellar::application::consensus::{CellReceiptRequest, VersionError}; + ensure!(sql.routed.len() == sql.receipts.len()); + let keys = Fixture::version_keys()?; + let hosts: Vec<_> = voters.iter().map(|(_, _, host, _)| host.clone()).collect(); + let retry = u64::try_from( + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH)? + .as_millis(), + )? + .checked_add(60_000) + .context("recovered SQL retry deadline")?; + let mut positions = Vec::new(); + for ((birth, archive), &(history, activation)) in + plan.births().iter().zip(fenced.archives()).zip(activations) + { + ensure!(birth.route().history == history); + let host = tokio::time::timeout(TIMEOUT, async { + loop { + for (index, host) in hosts.iter().enumerate() { + let status = host.group(history)?.status().await?; + if status.ready + && status.leader == Some(21 + index as u64) + && status.term > activation.term + && status.applied.term == status.term + { + return Ok::<_, anyhow::Error>(host); + } + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .context("released recovery did not establish a fresh serving quorum")??; + for ((old_cell, value), original) in sql + .routed + .iter() + .zip(sql.receipts) + .filter(|((cell, _), _)| archive.recovery().manifest().pin_cell(&cell.cell).is_some()) + { + let cell = CellHistory { + incarnation: history.incarnation, + ..old_cell.clone() + }; + let handle = host.cell(cell)?; + let query = handle + .query( + ProposalId::new([214; 16])?, + &handle.contract("Get")?, + vec![], + QueryOptions { + consistency: QueryConsistency::Linearizable, + ..Default::default() + }, + ) + .await?; + ensure!(query.result == (value + 1).to_be_bytes()); + let old_token = handle + .query( + ProposalId::new([215; 16])?, + &handle.contract("Get")?, + vec![], + QueryOptions { + consistency: QueryConsistency::AtLeast(original.version_token.clone()), + routing: QueryRouting::Local, + ..Default::default() + }, + ) + .await; + ensure!(matches!(old_token, Err(error) if matches!(error.as_ref(), + CellDispatchError::Version(VersionError::WrongHistory)))); + let add = handle.contract("Add")?; + let restored = handle + .receipt( + CellReceiptRequest { + id: original.id, + contract: add.clone(), + input: 1i64.to_be_bytes().to_vec(), + durability: Durability::Quorum, + }, + TIMEOUT, + ) + .await?; + let stored = restored + .receipt() + .context("original archived receipt missing")?; + ensure!( + stored.command() == original.id + && stored.result() == original.result.as_slice() + && restored.observation().version == query.version + && restored.observation().witness.group == history + ); + let id = CommandId::new(ClientId::new([212; 16])?, 1)?; + let options = CommandOptions::new(id, retry); + let receipt = handle + .command(&add, 1i64.to_be_bytes().to_vec(), options) + .await?; + ensure!( + receipt.result == (value + 2).to_be_bytes() && receipt.witness.group == history + ); + let duplicate = handle + .command( + &add, + 1i64.to_be_bytes().to_vec(), + CommandOptions::new(id, retry), + ) + .await?; + ensure!( + duplicate.id == receipt.id + && duplicate.result == receipt.result + && duplicate.version == receipt.version + && duplicate.witness == receipt.witness + && duplicate.durability == receipt.durability, + "recovered cell {:?} changed its retry receipt", + old_cell.cell + ); + // Issuance time is part of each signed token. Authenticate both + // responses against the same exact committed observation. + let now = u64::try_from( + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH)? + .as_millis(), + )?; + for response in [&receipt, &duplicate] { + let observation = keys.verify_cell( + &response.version_token, + &receipt.version.history, + birth.route().schema_digest, + now, + )?; + ensure!( + observation.version == receipt.version + && observation.witness == receipt.witness + ); + } + let archived = handle + .command( + &add, + 1i64.to_be_bytes().to_vec(), + CommandOptions { + durability: Durability::S3, + ..CommandOptions::new(CommandId::new(id.client, 2)?, retry) + }, + ) + .await?; + ensure!( + archived.result == (value + 3).to_be_bytes() + && archived.witness.group == history + && archived.durability == Durability::S3 + ); + positions.push((history, archived.witness.position.index)); + } + } + ensure!(positions.len() == sql.routed.len()); + preparation::wait_transfer_applied(&hosts, &positions).await +} + +async fn start_recovery_voter( + fixture: Fixture, + root: tempfile::TempDir, + mode: LogOpenMode, + generation: u8, +) -> Result { + let domain = fixture.domain.clone(); + // Run boot on its own polling stack. The split/merge scenario keeps + // substantial verification state live while RustFS boot binds a voter. + let boot = domain.spawn(Box::pin(async move { + let node = start_node( + root.path(), + fixture.application.clone(), + &fixture.spec.label, + ) + .await?; + let result = Box::pin(async { + let (mut config, services) = fixture.config(&node, mode, generation).await?; + crate::managed_fixture::archives::configure(&mut config); + Box::pin(node.start_raft_cells(config, services)).await + }) + .await; + match result { + Ok(host) => Ok((root, node, host, fixture)), + Err(error) => { + let stopped = node.shutdown(ShutdownMode::Preserve).await; + Err(error.context(format!("start recovery voter, shutdown={stopped:?}"))) + } + } + }))?; + boot.await? +} + +fn pending_recovery_route_fenced(error: &CellDispatchError) -> bool { + matches!( + error, + CellDispatchError::Routing(CellRoutingError { + reason: CellRoutingFailure::Catalog( + cellar::application::consensus::CatalogError::Fenced + ), + ambiguous_command: None, + }) + ) +} + +async fn check_pending_recovery_quorum( + hosts: &[&RaftCells], + request: &FleetRecoveryAttachRequest, + history: GroupHistory, +) -> Result<()> { + for (index, host) in hosts.iter().enumerate() { + Box::pin(host.attach_managed_recovery_voter(request.clone(), history, TIMEOUT)) + .await + .with_context(|| { + format!( + "attach recovery group {history:?} on {:?}", + host.identity().node + ) + })?; + if index == 0 { + let status = host.group(history)?.status().await?; + ensure!(!status.ready && status.committed.get() == 1); + let refused = Box::pin( + host.group(history)? + .activate_managed_recovery(request.clone(), TIMEOUT), + ) + .await; + ensure!( + matches!( + refused, + Err(CellGroupControllerError::Runtime( + RuntimeError::NotLeader | RuntimeError::NotReady + )) + ), + "minority admitted recovery activation: {refused:?}" + ); + } + } + tokio::time::timeout(TIMEOUT, async { + loop { + let mut fresh = true; + for host in hosts { + let status = host.group(history)?.status().await?; + ensure!(!status.ready); + fresh &= status.leader.is_some() + && status.term.get() > 1 + && status.applied.term == status.term + && status.applied.index.get() > 1 + && status.committed == status.applied.index; + } + if fresh { + break Ok::<_, anyhow::Error>(()); + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .context("pending recovery did not form a fresh Raft quorum")??; + Box::pin(hosts[0].attach_managed_recovery_voter(request.clone(), history, TIMEOUT)) + .await + .with_context(|| { + format!( + "replay recovery attachment for {history:?} on {:?}", + hosts[0].identity().node + ) + })?; + Ok(()) +} + +async fn check_managed_recovery_activation( + hosts: &[&RaftCells], + request: &FleetRecoveryAttachRequest, + history: GroupHistory, + store: &fault::Store, +) -> Result { + store + .fail_recovery_record_confirmation + .store(true, Ordering::Release); + let position = tokio::time::timeout(TIMEOUT, async { + let mut lost = false; + loop { + for host in hosts { + let group = host.group(history)?; + let observed = + Box::pin(group.activate_managed_recovery(request.clone(), TIMEOUT)).await; + match observed { + Ok(Some(position)) => { + ensure!(lost && position.term.get() > 1 && position.index.get() > 2); + return Ok::<_, anyhow::Error>(position); + } + Ok(None) + | Err(CellGroupControllerError::Runtime( + RuntimeError::NotLeader | RuntimeError::NotReady, + )) => (), + Err(CellGroupControllerError::Catalog(error)) + if matches!( + error.as_ref(), + cellar::application::consensus::CatalogError::Storage(_) + ) && !store + .fail_recovery_record_confirmation + .load(Ordering::Acquire) => + { + lost = true; + } + Err(error) => return Err(error.into()), + } + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .context("managed recovery activation did not commit")??; + + tokio::time::timeout(TIMEOUT, async { + loop { + let mut applied = true; + for host in hosts { + let status = host.group(history)?.status().await?; + ensure!( + !status.ready, + "committed recovery activated serving before settlement" + ); + applied &= status.applied.index >= position.index + && status.committed == status.applied.index; + } + if applied { + return Ok::<_, anyhow::Error>(()); + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .context("managed recovery activation did not reach all voters")??; + let replayed = Box::pin( + hosts[2] + .group(history)? + .activate_managed_recovery(request.clone(), TIMEOUT), + ) + .await?; + ensure!(replayed == Some(position)); + Ok(position) +} + +fn check_restored_image(before: CellImagePin, after: CellImagePin) -> Result<()> { + match (before, after) { + (CellImagePin::Materialized(before), CellImagePin::Materialized(after)) => { + ensure!(before.state() == after.state()); + let (mut before, mut after) = (before.open_reader()?, after.open_reader()?); + let mut offset = 0; + while offset < before.length() { + let length = (before.length() - offset).min(64 * 1024) as usize; + ensure!( + before.read_exact_at(offset, length)? == after.read_exact_at(offset, length)? + ); + offset += length as u64; + } + } + (CellImagePin::Extents(before), CellImagePin::Extents(after)) => { + let count = before.manifest().geometry().extent_count(); + ensure!(count == after.manifest().geometry().extent_count()); + for index in 0..count { + let id = before.manifest().extent(index).context("source extent")?; + ensure!(after.manifest().extent(index) == Some(id)); + ensure!(before.view().read_extent(id)? == after.view().read_extent(id)?); + } + } + _ => anyhow::bail!("expected matching nonempty image representations"), + } + Ok(()) +} + +async fn admit_merge( + cluster: &ManagedCluster, + directory: &RangeDirectory, + control: &RaftFleetControl, + parents: &[GroupRoute; 2], + child: GroupHistory, + operation: ProposalId, +) -> Result<( + cellar::sdk::fleet_operations::RaftFleetTransitionAdmission, + FleetSnapshot, +)> { + let fixture = &cluster.fixtures[0]; + let fleet = &fixture.catalog; + let basis = control + .transition_basis(&hex::encode(child.namespace.bytes()), TIMEOUT) + .await?; + let principal = request(245, basis.fleet_revision).principal; + let admission = RaftFleetTransitionAdmissionRequest { + catalog_id: basis.catalog_id, + namespace_id: basis.namespace_id, + operation_id: hex::encode(operation.bytes()), + request_id: hex::encode([245; 16]), + expected_fleet_revision: basis.fleet_revision, + expected_directory_digest: basis.directory_digest, + class: "Counter".into(), + partition: RaftFleetTransitionPartition::Merge { + left: transition_group(parents[0].history), + right: transition_group(parents[1].history), + }, + children: vec![RaftFleetTransitionChild { + group: transition_group(child), + request_id: hex::encode([243; 16]), + creation_id: hex::encode([243; 16]), + voters: std::array::from_fn(|index| RaftFleetFirstGroupVoter { + node_id: hex::encode(cluster.fixtures[index].spec.node.bytes()), + member_id: index as u64 + 1, + }), + }], + }; + let namespace = &admission.namespace_id; + let id = &admission.operation_id; + let stale = RaftFleetTransitionAdmissionRequest { + expected_directory_digest: hex::encode([67; 32]), + ..admission.clone() + }; + let refused = control + .admit_transition(&principal, namespace, id, &stale, TIMEOUT) + .await; + ensure!(matches!(refused, Err(failure) if failure.code == RaftFleetFailureCode::Conflict)); + ensure!(fleet.read(TIMEOUT).await?.revision() == basis.fleet_revision); + + fixture + .store + .fail_fleet_confirmation + .store(true, Ordering::Release); + let lost = control + .admit_transition(&principal, namespace, id, &admission, TIMEOUT) + .await; + ensure!(matches!(lost, Err(failure) if failure.code == RaftFleetFailureCode::Unavailable)); + let reserved = fleet.read(TIMEOUT).await?; + ensure!(reserved.revision() == basis.fleet_revision + 1); + ensure!( + *fleet.history(&reserved, &child, TIMEOUT).await?.state() == FleetHistoryState::Reserved + ); + ensure!(fleet + .reopen_managed_transition(directory, operation, TIMEOUT) + .await? + .is_none()); + let mut conflicting = admission.clone(); + conflicting.children[0].creation_id = hex::encode([242; 16]); + let refused = control + .admit_transition(&principal, namespace, id, &conflicting, TIMEOUT) + .await; + ensure!( + matches!(refused, Err(failure) if failure.code == RaftFleetFailureCode::PayloadConflict) + ); + + let admitted = crate::managed_fixture::operator::admit_transition( + fixture, directory, &principal, &admission, + ) + .await?; + ensure!(admitted.phase == TransitionPhase::Planned); + let exact = control + .admit_transition(&principal, namespace, id, &admission, TIMEOUT) + .await?; + ensure!(admitted == exact); + let cut = fleet.read(TIMEOUT).await?; + ensure!(cut.revision() == basis.fleet_revision + 2); + Ok((admitted, cut)) +} + +async fn reopen_merge_controller(fixture: &Fixture) -> Result<(FleetCatalog, RangeDirectory)> { + let fleet = Fixture::open_catalog( + &fixture.domain, + fixture.store.clone(), + fixture.declaration.namespace.cluster, + )?; + let catalog = Catalog::new( + fixture.store.clone(), + CatalogOptions { + prefix: "managed/".into(), + ..Default::default() + }, + )?; + let namespace = fixture + .domain + .run(catalog.existing_namespace(fixture.declaration.namespace.clone())) + .await??; + Ok(( + fleet, + RangeDirectory::new(namespace, DirectoryLimits::default())?, + )) +} + +async fn verify_minority( + survivor: &RaftCells, + transition: &ReopenedManagedCellGroupTransition, + seal: &GroupSourceSeal, + parents: &[GroupRoute; 2], + cells: &[(CellHistory, i64)], + retry: u64, +) -> Result<()> { + let unsealed = parents[1].history; + let group = survivor.group(unsealed)?; + let committed = tokio::time::timeout(TIMEOUT, async { + loop { + let status = group.status().await?; + if !status.ready && status.leader.is_none() { + return Ok::<_, anyhow::Error>(status.committed); + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .context("unsealed merge parent did not lose quorum")??; + let handle = survivor.cell(cells[1].0.clone())?; + let get = handle.contract("Get")?; + let historical = handle + .query( + ProposalId::new([220; 16])?, + &get, + vec![], + QueryOptions { + consistency: QueryConsistency::Stale, + stale_policy: StaleReadPolicy::DisconnectedHistorical, + routing: QueryRouting::Local, + ..Default::default() + }, + ) + .await?; + ensure!( + historical.result == cells[1].1.to_be_bytes() + && historical.witness.group == unsealed + && historical.served_by_node == survivor.identity() + ); + let fresh = handle + .query( + ProposalId::new([221; 16])?, + &get, + vec![], + QueryOptions { + routing: QueryRouting::Local, + ..Default::default() + }, + ) + .await; + ensure!( + matches!( + fresh.as_ref().err().map(Arc::as_ref), + Some(CellDispatchError::Runtime( + RuntimeError::NotLeader | RuntimeError::NotReady | RuntimeError::DeadlineExceeded + )) + ), + "minority cannot serve a fresh read: {fresh:?}" + ); + let refused = group.seal_managed_source(transition, TIMEOUT).await.err(); + ensure!( + matches!( + refused, + Some(CellGroupControllerError::Runtime(RuntimeError::NotLeader)) + ), + "minority cannot seal the other parent: {refused:?}" + ); + ensure!(group.status().await?.committed == committed); + + let sealed = parents[0].history; + verify_durable_seal(survivor, transition, sealed, seal).await?; + let handle = survivor.cell(cells[0].0.clone())?; + let id = CommandId::new(ClientId::new([222; 16])?, 1)?; + let refused = handle + .command( + &handle.contract("Add")?, + 64i64.to_be_bytes().to_vec(), + CommandOptions::new(id, retry), + ) + .await; + ensure!( + matches!(refused, Err(ref failure) if failure.command == id && !failure.ambiguous), + "sealed parent admitted a new command: {refused:?}" + ); + verify_durable_seal(survivor, transition, sealed, seal).await +} + +async fn verify_durable_seal( + host: &RaftCells, + transition: &ReopenedManagedCellGroupTransition, + history: GroupHistory, + seal: &GroupSourceSeal, +) -> Result<()> { + let source = host.group(history)?; + let observed = source.seal_managed_source(transition, TIMEOUT).await?; + ensure!(matches!(observed, CellGroupSealOutcome::Applied(ref applied) if applied == seal)); + ensure!(!source.status().await?.ready); + Ok(()) +} + +async fn retire_merge_parent( + control: &RaftFleetControl, + voters: &[RaftCells], + principal: &str, + request: &RaftFleetTransitionSourceSealRequest, +) -> Result { + tokio::time::timeout(TIMEOUT, async { + loop { + for voter in voters { + let local = control.clone().with_node_observer(voter.clone()); + match local + .retire_transition_source( + principal, + &request.namespace_id, + &request.operation_id, + request, + TIMEOUT, + ) + .await + { + Ok(retired) if retired.applied.is_some() => return Ok(retired), + Ok(_) => (), + Err(error) + if matches!( + error.code, + RaftFleetFailureCode::Unavailable | RaftFleetFailureCode::Busy + ) => + { + () + } + Err(error) => return Err(error.into()), + } + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .context("operator parent retirement")? +} + +async fn initialize_merge_child_archive( + cluster: &mut ManagedCluster, + directory: &RangeDirectory, + transition: &ReopenedManagedCellGroupTransition, + child: GroupHistory, + control: &RaftFleetControl, +) -> Result<()> { + let archive_request = RaftFleetTransitionChildSetupRequest { + catalog_id: control.catalog_id().into(), + namespace_id: hex::encode(child.namespace.bytes()), + operation_id: hex::encode(transition.request.operation), + request_id: hex::encode([185; 16]), + plan_digest: hex::encode(transition.plan.digest()), + child: RaftFleetTransitionGroup { + group_id: hex::encode(child.group.bytes()), + incarnation_id: hex::encode(child.incarnation.bytes()), + }, + step: RaftFleetTransitionChildStep::InitializeArchive, + }; + for _ in 0..2 { + let initialized = Box::pin(crate::managed_fixture::operator::setup_transition_child( + &cluster.fixtures[0], + directory, + &transition.request.principal, + &cluster.hosts[0], + &archive_request, + None, + )) + .await?; + ensure!( + initialized.request == archive_request + && initialized.result == RaftFleetTransitionChildSetupResult::ArchiveInitialized + ); + } + for (index, generation) in [121, 126, 127].into_iter().enumerate() { + Box::pin(restart_voter(cluster, index, generation, Some(child), 1)).await?; + } + Ok(()) +} + +fn finish_merge<'a>( + cluster: &'a mut ManagedCluster, + directory: &'a RangeDirectory, + transition: &'a ReopenedManagedCellGroupTransition, + parents: &'a [GroupRoute; 2], + child: GroupHistory, + archives: &'a ArchiveInputs, + control: &'a RaftFleetControl, +) -> impl std::future::Future> + 'a { + Box::pin(finish_merge_inner( + cluster, directory, transition, parents, child, archives, control, + )) +} + +async fn finish_merge_inner( + cluster: &mut ManagedCluster, + directory: &RangeDirectory, + transition: &ReopenedManagedCellGroupTransition, + parents: &[GroupRoute; 2], + child: GroupHistory, + archives: &ArchiveInputs, + control: &RaftFleetControl, +) -> Result<()> { + let fleet = cluster.fixtures[0].catalog.clone(); + let domain = &archives.domain; + let applied = preparation::wait_active_archive(&cluster.hosts[0], child).await?; + let coverage_request = RaftFleetTransitionPrepareRequest { + catalog_id: control.catalog_id().into(), + namespace_id: hex::encode(child.namespace.bytes()), + operation_id: hex::encode(transition.request.operation), + request_id: hex::encode([186; 16]), + plan_digest: hex::encode(transition.plan.digest()), + }; + let mut digest = None; + for _ in 0..2 { + let verifier_root = tempfile::tempdir()?; + let verifier = crate::managed_fixture::verifier::open( + &cluster.fixtures[0], + verifier_root.path(), + archives.rooted, + Default::default(), + ) + .await?; + let covered = Box::pin(crate::managed_fixture::operator::verify_transition_archive( + &cluster.fixtures[0], + directory, + &transition.request.principal, + &coverage_request, + (verifier, preparation::recovery_pool()?), + "verify-child-archives", + )) + .await?; + ensure!(covered.request == coverage_request && covered.phase == TransitionPhase::Active); + ensure!(digest + .as_ref() + .is_none_or(|old| old == &covered.archive_digest)); + digest = Some(covered.archive_digest); + } + let holds = fleet + .reopen_managed_transfer_child_holds(transition, directory, TIMEOUT) + .await? + .context("selected child archive")?; + ensure!(holds.len() == 1 && holds[0].marker().position().index >= applied); + let principal = &transition.request.principal; + for (index, parent) in parents.iter().enumerate() { + let retirement = RaftFleetTransitionSourceSealRequest { + catalog_id: control.catalog_id().into(), + namespace_id: hex::encode(child.namespace.bytes()), + operation_id: hex::encode(transition.request.operation), + request_id: hex::encode([187 + index as u8; 16]), + plan_digest: hex::encode(transition.plan.digest()), + source: RaftFleetTransitionGroup { + group_id: hex::encode(parent.history.group.bytes()), + incarnation_id: hex::encode(parent.history.incarnation.bytes()), + }, + }; + let completed = Box::pin(retire_merge_parent( + control, + &cluster.hosts[3..], + principal, + &retirement, + )) + .await?; + let expected_phase = if index == 0 { + TransitionPhase::Active + } else { + TransitionPhase::Retired + }; + ensure!(completed.request == retirement && completed.phase == expected_phase); + let applied = completed + .applied + .as_ref() + .context("durable parent tombstone")?; + let replay = Box::pin( + crate::managed_fixture::operator::reopen_transition_retirement( + &cluster.fixtures[0], + directory, + principal, + &retirement, + ), + ) + .await?; + ensure!(replay == completed); + tokio::time::timeout(TIMEOUT, async { + loop { + let mut applied_on_all = true; + for voter in &cluster.hosts[3..] { + let status = voter.group(parent.history)?.status().await?; + applied_on_all &= status.applied.index.get() >= applied.index; + } + if applied_on_all { + return Ok::<_, anyhow::Error>(()); + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .context("retirement applied on retained parent voters")??; + } + let retired = fleet + .reopen_managed_transition(directory, transition.plan.operation(), TIMEOUT) + .await? + .and_then(|reopened| reopened.journal) + .context("rooted merge retirement")?; + ensure!(retired.phase() == TransitionPhase::Retired && !retired.parent_holds_released()); + let cut = fleet.read(TIMEOUT).await?; + let confirmation = RaftFleetTransitionResume { + catalog_id: control.catalog_id().into(), + namespace_id: hex::encode(child.namespace.bytes()), + operation_id: hex::encode(transition.request.operation), + request_id: hex::encode([246; 16]), + expected_revision: cut.revision(), + plan_digest: hex::encode(transition.plan.digest()), + step: RaftFleetTransitionResumeStep::ConfirmChildren, + }; + let namespace = &confirmation.namespace_id; + let operation = &confirmation.operation_id; + let foreign = control + .resume_transition( + "another-operator", + namespace, + operation, + &confirmation, + TIMEOUT, + ) + .await; + ensure!( + matches!(foreign, Err(failure) if failure.code == RaftFleetFailureCode::Forbidden + && failure.outcome == RaftFleetOutcome::NotSubmitted) + ); + let confirmed = control + .resume_transition(principal, namespace, operation, &confirmation, TIMEOUT) + .await?; + let cut = fleet.read(TIMEOUT).await?; + ensure!( + confirmed.applied_revision == cut.revision() + && *fleet.history(&cut, &child, TIMEOUT).await?.state() == FleetHistoryState::Live + ); + let control = control.clone().with_node_observer(cluster.hosts[0].clone()); + + Box::pin(restart_voter(cluster, 3, 122, None, 0)).await?; + let cut = fleet.read(TIMEOUT).await?; + let observer = &cluster.hosts[0]; + for source in &cluster.hosts[3..] { + inspect_current(observer, source.identity().node, cut.revision()).await?; + } + let premature = RaftFleetTransitionResume { + request_id: hex::encode([248; 16]), + expected_revision: cut.revision(), + step: RaftFleetTransitionResumeStep::Complete, + ..confirmation.clone() + }; + let attached = control + .resume_transition(principal, namespace, operation, &premature, TIMEOUT) + .await; + ensure!( + matches!(attached, Err(failure) if failure.code == RaftFleetFailureCode::Unavailable + && failure.outcome == RaftFleetOutcome::Unknown) + ); + let still_held = control + .transition_status(namespace, operation, TIMEOUT) + .await?; + ensure!(still_held.fleet_slot_held && !still_held.parent_holds_released); + for index in 4..6 { + Box::pin(restart_voter(cluster, index, 119 + index as u8, None, 0)).await?; + } + let cut = fleet.read(TIMEOUT).await?; + let observer = &cluster.hosts[0]; + for source in &cluster.hosts[3..] { + let node = source.identity().node; + inspect_current(observer, node, cut.revision()).await?; + let parked = observer + .inspect_peer_physical_inventory(node, cut.revision(), TIMEOUT) + .await?; + ensure!( + parked.view.unattached_engine_histories == 2 + && parked.view.node.reserved_bytes == 2 * parked.view.node.profile.bytes_per_group + ); + let inventory = parked.inventory.context("two parked merge parents")?; + for parent in parents { + ensure!(inventory + .group_history(&parent.history) + .context("parked merge parent")? + .parked_retired_source()); + } + } + let stale = control + .resume_transition(principal, namespace, operation, &premature, TIMEOUT) + .await; + ensure!(matches!(stale, Err(failure) if failure.code == RaftFleetFailureCode::Conflict)); + let completion = RaftFleetTransitionResume { + request_id: hex::encode([247; 16]), + expected_revision: cut.revision(), + step: RaftFleetTransitionResumeStep::Complete, + ..confirmation.clone() + }; + let collision = RaftFleetTransitionResume { + request_id: confirmation.request_id.clone(), + ..completion.clone() + }; + let reused = control + .resume_transition(principal, namespace, operation, &collision, TIMEOUT) + .await; + ensure!( + matches!(reused, Err(failure) if failure.code == RaftFleetFailureCode::PayloadConflict) + ); + let still_held = control + .transition_status(namespace, operation, TIMEOUT) + .await?; + ensure!(still_held.fleet_slot_held && !still_held.parent_holds_released); + + let store = &cluster.fixtures[0].store; + store.fail_fleet_before.store(true, Ordering::Release); + let interrupted = control + .resume_transition(principal, namespace, operation, &completion, TIMEOUT) + .await; + ensure!( + matches!(interrupted, Err(failure) if failure.code == RaftFleetFailureCode::Unavailable + && failure.outcome == RaftFleetOutcome::Unknown) + ); + ensure!(!store.fail_fleet_before.load(Ordering::Acquire)); + ensure!(domain.run(directory.active_transition()).await??.is_none()); + let cleaning = control + .transition_status(namespace, operation, TIMEOUT) + .await?; + ensure!( + cleaning.phase == Some(TransitionPhase::Retired) + && cleaning.child_archives_verified + && cleaning.parent_holds_released + && cleaning.fleet_slot_held + && cleaning.observed_fleet_revision == cut.revision() + ); + + let resumed = crate::managed_fixture::operator::resume_transition( + &cluster.fixtures[0], + directory, + principal, + observer, + &completion, + ) + .await?; + let replayed = control + .resume_transition(principal, namespace, operation, &completion, TIMEOUT) + .await?; + ensure!(resumed == replayed && resumed.applied_revision == cut.revision() + 1); + let settled = fleet.read(TIMEOUT).await?; + let terminal = control + .transition_status(namespace, operation, TIMEOUT) + .await?; + ensure!( + terminal.observed_fleet_revision == settled.revision() + && terminal.phase == cleaning.phase + && terminal.plan_digest == cleaning.plan_digest + && terminal.parent_holds_released + && !terminal.fleet_slot_held + ); + crate::managed_fixture::operator::transition_status( + &cluster.fixtures[0], + directory, + principal, + &terminal, + ) + .await?; + let retirement = retired + .retired_parents() + .context("merge parent retirement digest")?; + for parent in parents { + ensure!( + *fleet + .history(&settled, &parent.history, TIMEOUT) + .await? + .state() + == FleetHistoryState::Retired { + operation: transition.request.operation, + retirement + } + ); + } + let inventory = fleet + .check_inventory(&settled, std::slice::from_ref(directory), TIMEOUT) + .await?; + ensure!(inventory.histories() == 4 && inventory.namespaces() == 1); + Ok(()) +} + +async fn wait_archive( + voters: &[RaftCells], + history: GroupHistory, + seal: &GroupSourceSeal, +) -> Result<()> { + tokio::time::timeout(TIMEOUT, async { + loop { + for host in voters { + if host + .group(history)? + .archive_status()? + .and_then(|status| status.published) + .is_some_and(|marker| marker.source_seal() == Some(seal)) + { + return Ok::<_, anyhow::Error>(()); + } + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .context("merge parent did not publish its sealed cut")? +} + +async fn restart_voter( + cluster: &mut ManagedCluster, + index: usize, + generation: u8, + active: Option, + registered: u32, +) -> Result<()> { + let previous = stop_voter(cluster, index, active).await?; + Box::pin(start_voter( + cluster, index, generation, previous, registered, + )) + .await +} + +async fn stop_voter( + cluster: &mut ManagedCluster, + index: usize, + active: Option, +) -> Result { + let node = cluster.nodes.remove(index); + let old = cluster.hosts.remove(index); + let identity = old.identity(); + ensure!(matches!( + node.shutdown(ShutdownMode::Preserve).await?, + NodeOutcome::Stopped(_) + )); + let report = old.close().await?; + ensure!( + report.groups_total == 2 + && report.omitted_groups == 0 + && report.mutation_outcomes == MutationOutcomes::NoUnresolvedLocalProducers + && report + .groups + .iter() + .all(|group| group.active == (active == Some(group.history)) + && !group.failed + && group.applied.index == group.committed), + "merge voter shutdown inventory: {report:?}" + ); + Ok(identity) +} + +async fn start_voter( + cluster: &mut ManagedCluster, + index: usize, + generation: u8, + previous: NodeIdentity, + registered: u32, +) -> Result<()> { + let fixture = &cluster.fixtures[index]; + let node = start_node( + cluster.roots[index].path(), + fixture.application.clone(), + &fixture.spec.label, + ) + .await?; + let startup = async { + let (mut config, services) = fixture + .config(&node, LogOpenMode::Resume, generation) + .await?; + crate::managed_fixture::archives::configure(&mut config); + node.start_raft_cells(config, services).await + } + .await; + let host = match startup { + Ok(host) => host, + Err(error) => { + let stopped = node.shutdown(ShutdownMode::Preserve).await; + return Err(error.context(format!("restart merge voter {index}, shutdown={stopped:?}"))); + } + }; + cluster.nodes.insert(index, node); + cluster.hosts.insert(index, host); + let host = &cluster.hosts[index]; + ensure!(host.identity().disk == previous.disk && host.identity().process != previous.process); + let view = host.observe_node(TIMEOUT).await?; + ensure!(view.node.groups == 2 && view.registered_histories == registered); + Ok(()) +} + +fn assert_recovery_lineages( + restored: &cellar::sdk::fleet_operations::RaftFleetVerifiedTransitionRecovery, + expected: &[(GroupHistory, usize, FleetNodeSpec)], +) -> Result<()> { + let required = restored.required_voter_lineages(); + ensure!(required.len() == expected.len()); + for (group, index, node) in expected { + ensure!(required.iter().any(|lineage| { + lineage.group == *group + && lineage.member.get() == *index as u64 + 1 + && lineage.node == node.node + && lineage.disk == node.disk + && lineage.physical_host == node.physical_host + })); + } + Ok(()) +} + +async fn verify_merged_sql( + voters: &[RaftCells], + history: GroupHistory, + cells: &[(CellHistory, i64)], + receipts: Vec, + retry: u64, +) -> Result<()> { + let host = &voters[leader(voters, history).await?]; + let mut positions = Vec::with_capacity(cells.len()); + for (index, ((cell, expected), receipt)) in cells.iter().zip(receipts).enumerate() { + let handle = host.cell(cell.clone())?; + let query = handle + .query( + ProposalId::new([214 + index as u8; 16])?, + &handle.contract("Get")?, + vec![], + QueryOptions::default(), + ) + .await?; + ensure!(query.result == (expected + 1).to_be_bytes()); + let add = handle.contract("Add")?; + let retry_receipt = handle + .command( + &add, + 1i64.to_be_bytes().to_vec(), + CommandOptions::new(receipt.id, retry), + ) + .await?; + ensure!( + retry_receipt.id == receipt.id + && retry_receipt.result == receipt.result + && retry_receipt.version == receipt.version + ); + let write = handle + .command( + &add, + 1i64.to_be_bytes().to_vec(), + CommandOptions::new(CommandId::new(receipt.id.client, 2)?, retry), + ) + .await?; + ensure!(write.result == (expected + 2).to_be_bytes() && write.witness.group == history); + positions.push((history, write.witness.position.index)); + } + preparation::wait_transfer_applied(voters, &positions).await +} + +pub(super) async fn close( + nodes: Vec, + hosts: &[RaftCells], + merged: Option, +) -> Result<()> { + let reports = futures_util::future::join_all(nodes.into_iter().zip(hosts).map( + |(node, host)| async move { + ensure!(matches!( + node.shutdown(ShutdownMode::Preserve).await?, + NodeOutcome::Stopped(_) + )); + Ok::<_, anyhow::Error>(host.close().await?) + }, + )) + .await; + for (index, report) in reports.into_iter().enumerate() { + let report = report?; + if let Some(merged) = merged { + ensure!( + report.groups_total == 2 + && report.groups.len() == 2 + && report.omitted_groups == 0 + && report.mutation_outcomes == MutationOutcomes::NoUnresolvedLocalProducers + && report.groups.iter().all(|group| !group.failed + && group.active == (index < 3 && group.history == merged) + && group.applied.index == group.committed), + "merged shutdown inventory: {report:?}" + ); + } + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_managed_transition_preparation.rs b/crates/cellar/tests/support/cell_group_managed_transition_preparation.rs new file mode 100644 index 000000000..69062ba1d --- /dev/null +++ b/crates/cellar/tests/support/cell_group_managed_transition_preparation.rs @@ -0,0 +1,1925 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Rebuild a durable pre-copy from the same real managed archive hold. +use super::*; +use cellar::application::consensus::verification::cell_groups::{ + CellGroupApplyImages, CellGroupArchiveError, CellGroupArchiveHold, CellGroupArchiveRegistry, + CellGroupArchiveStore, CellGroupArchiveTailLimits, CellGroupArchiveTailPool, CellGroupBirth, + CellGroupCheckpointPool, CellGroupRecoveryLimits, CellGroupRecoveryPool, + CellGroupTransferTarget, CellGroupTransitionPreparationInputs, CellGroupTransitionSource, + DirectoryMutation, GroupRoute, ManifestPageStore, TransitionJournal, +}; +use cellar::application::consensus::verification::FleetHistoryState; +use cellar::sdk::fleet_operations::{ + RaftFleetControl, RaftFleetFailureCode, RaftFleetOutcome, RaftFleetTransitionActivationRequest, + RaftFleetTransitionChildSetupRequest, RaftFleetTransitionChildSetupResult, + RaftFleetTransitionChildStep, RaftFleetTransitionGroup, RaftFleetTransitionPrepareRequest, + RaftFleetTransitionPublishRequest, +}; +use cellar_ltx::{ + generation::{GenerationStore, ReadServices}, + internal::{CacheLimits, NodeExtentStore, EXTENT_BYTES}, + stream_snapshot::{ScratchLimits, ScratchPool}, + DirectFileSystem, +}; +use rand::RngCore as _; + +pub(super) async fn exercise( + cluster: &ManagedCluster, + controller: &FleetCatalog, + directory: &RangeDirectory, + transition: &ReopenedManagedCellGroupTransition, + parent: &CellGroupBirth, + children: &[GroupRoute], + rooted: bool, +) -> Result<(CellGroupArchiveRegistry, CellGroupArchiveHold, SeededCells)> { + let seeded = seed_cells(&cluster.hosts, cluster.history).await?; + let fixture = &cluster.fixtures[0]; + let domain = &fixture.domain; + let leader = leader(&cluster.hosts[..3], cluster.history).await?; + let cell = &seeded.cells[0]; + let handle = cluster.hosts[leader].cell(cell.clone())?; + let client = seeded.receipts[0].id.client; + let mut options = CommandOptions::new(CommandId::new(client, 2)?, seeded.retry); + options.durability = Durability::S3; + let archived = handle + .command( + &handle.contract("Add")?, + 2i64.to_be_bytes().to_vec(), + options, + ) + .await?; + ensure!(archived.durability == Durability::S3 && archived.result == 9i64.to_be_bytes()); + let archives = CellGroupArchivePool::new(Default::default())?; + let registry = directory + .open_group_archive(domain.as_ref(), &archives, parent) + .await?; + let operation = transition.plan.operation(); + let before = controller + .reopen_managed_transition(directory, operation, TIMEOUT) + .await + .context("read planned transition before hold acquire")? + .context("planned transition before acquire")?; + let denied_before = fixture.store.denied_registry_writes.load(Ordering::Acquire); + fixture + .store + .deny_registry_writes + .store(true, Ordering::Release); + let denied = controller + .allocate_managed_preparation_holds(directory, operation, &archives, TIMEOUT) + .await; + fixture + .store + .deny_registry_writes + .store(false, Ordering::Release); + ensure!( + denied.is_err() + && fixture.store.denied_registry_writes.load(Ordering::Acquire) > denied_before + ); + let reopened = Fixture::open_catalog(domain, fixture.store.clone(), cluster.history.cluster)?; + let after = reopened + .reopen_managed_transition(directory, operation, TIMEOUT) + .await + .context("read hold acquisition after denied registry update")? + .context("admitted acquire after denied registry update")?; + ensure!(after.journal != before.journal); + ensure!(reopened + .staged_managed_preparation_holds(directory, operation, TIMEOUT) + .await + .context("read staging after denied registry update")? + .is_none()); + + let staged = allocate_preparation_holds(&reopened, directory, operation, &archives).await?; + ensure!(staged.phase() == TransitionPhase::Planned && staged.preparation().is_none()); + let [held]: [CellGroupArchiveHold; 1] = reopened + .staged_managed_preparation_holds(directory, operation, TIMEOUT) + .await? + .context("missing operator-owned staged hold")? + .try_into() + .map_err(|_| anyhow::anyhow!("split must own one preparation hold"))?; + ensure!( + reopened + .allocate_managed_preparation_holds(directory, operation, &archives, TIMEOUT) + .await + .context("reopen the exact staged hold allocation")? + == staged + ); + ensure!(matches!( + reopened + .release_cancelled_preparation_holds(directory, operation, &archives, TIMEOUT) + .await, + Err(FleetCatalogError::Conflict) + )); + let alternate = acquire_hold(domain, ®istry).await?; + ensure!(alternate != held); + ensure!(matches!( + controller + .stage_managed_preparation_holds( + transition, + directory, + &archives, + std::slice::from_ref(&alternate), + TIMEOUT, + ) + .await, + Err(FleetCatalogError::Conflict) + )); + registry.release(domain.as_ref(), &alternate).await?; + + let scratch = tempfile::tempdir()?; + ensure!( + reopened + .staged_managed_preparation_holds(directory, operation, TIMEOUT) + .await? + == Some(vec![held.clone()]) + ); + let verifier = crate::managed_fixture::verifier::open( + fixture, + &scratch.path().join("denied"), + rooted, + CellGroupArchiveTailLimits::default(), + ) + .await?; + let denied_before = fixture + .store + .denied_group_payload_reads + .load(Ordering::Acquire); + fixture + .store + .deny_group_payload_reads + .store(true, Ordering::Release); + let denied = reopened + .resume_managed_preparation(directory, operation, &verifier, recovery_pool()?, TIMEOUT) + .await; + fixture + .store + .deny_group_payload_reads + .store(false, Ordering::Release); + ensure!( + denied.is_err() + && fixture + .store + .denied_group_payload_reads + .load(Ordering::Acquire) + > denied_before + ); + let after_denial = reopened + .reopen_managed_transition(directory, operation, TIMEOUT) + .await? + .context("admitted transition after pre-copy outage")?; + ensure!(after_denial.journal.as_ref() == Some(&staged)); + drop(verifier); + + let control = RaftFleetControl::new(domain, reopened.clone(), vec![directory.clone()])?; + let request = RaftFleetTransitionPrepareRequest { + catalog_id: control.catalog_id().into(), + namespace_id: hex::encode(cluster.history.namespace.bytes()), + operation_id: hex::encode(operation.bytes()), + request_id: hex::encode([176; 16]), + plan_digest: hex::encode(transition.plan.digest()), + }; + let refused = control + .prepare_transition( + &transition.request.principal, + &request.namespace_id, + &request.operation_id, + &request, + TIMEOUT, + ) + .await + .err() + .context("preparation without a configured verifier")?; + ensure!(refused.code == RaftFleetFailureCode::Unsupported); + let verifier = crate::managed_fixture::verifier::open( + fixture, + &scratch.path().join("operator"), + rooted, + CellGroupArchiveTailLimits::default(), + ) + .await?; + let prepared = crate::managed_fixture::operator::prepare_transition( + fixture, + directory, + &transition.request.principal, + &request, + Some((verifier, recovery_pool()?)), + ) + .await?; + ensure!(prepared.phase == TransitionPhase::Prepared); + // The selected cut survives a later quorum write to the active parent. + let next = handle + .command( + &handle.contract("Add")?, + 1i64.to_be_bytes().to_vec(), + CommandOptions::new(CommandId::new(client, 3)?, seeded.retry), + ) + .await?; + ensure!(next.result == 10i64.to_be_bytes() && next.version != archived.version); + let replay = crate::managed_fixture::operator::prepare_transition( + fixture, + directory, + &transition.request.principal, + &request, + None, + ) + .await?; + ensure!(replay == prepared); + + let verifier = crate::managed_fixture::verifier::open( + fixture, + &scratch.path().join("reopened"), + rooted, + CellGroupArchiveTailLimits::default(), + ) + .await?; + let reopened = Fixture::open_catalog(domain, fixture.store.clone(), cluster.history.cluster)?; + let replay = reopened + .resume_managed_preparation(directory, operation, &verifier, recovery_pool()?, TIMEOUT) + .await + .context("reopen the selected split preparation")?; + ensure!(replay + .children() + .iter() + .map(|child| child.route()) + .eq(children)); + ensure!(replay.journal().phase() == TransitionPhase::Prepared); + ensure!(replay + .children() + .iter() + .all(|child| child.genesis().is_none())); + let digest = replay + .journal() + .preparation() + .context("missing pre-copy digest")?; + ensure!(prepared.preparation_digest == hex::encode(digest)); + ensure!( + replay + .children() + .iter() + .map(|child| child.root().cells()) + .sum::() + == 2 + ); + let mut staged = replay + .children() + .iter() + .filter_map(|child| child.pin_cell(&cell.cell)); + let (leaf, _) = staged.next().context("missing archived cell")?; + ensure!(leaf.version == archived.version && leaf.witness == archived.witness); + ensure!(staged.next().is_none()); + ensure!( + domain + .run(directory.managed_preparation_holds(&transition.plan)) + .await?? + == vec![held.clone()] + ); + Ok((registry, held, seeded)) +} + +pub(super) async fn allocate_preparation_holds( + fleet: &FleetCatalog, + directory: &RangeDirectory, + operation: ProposalId, + archives: &CellGroupArchivePool, +) -> Result { + let journal = tokio::time::timeout(TIMEOUT, async { + loop { + match fleet + .allocate_managed_preparation_holds(directory, operation, archives, TIMEOUT) + .await + { + Ok(journal) => return Ok(journal), + Err(FleetCatalogError::Conflict | FleetCatalogError::Unconfirmed) => { + tokio::task::yield_now().await; + } + Err(error) => return Err(error), + } + } + }) + .await??; + Ok(journal) +} + +pub(super) fn recovery_pool() -> Result { + Ok(CellGroupRecoveryPool::new(CellGroupRecoveryLimits { + max_cells: 4, + max_image_bytes: 1024 * 1024, + max_restore_bytes: 4 * 1024 * 1024, + ..Default::default() + })?) +} + +pub(super) fn store( + base: std::path::PathBuf, + route: GroupRoute, + rooted: bool, +) -> Result { + let filesystem = Arc::new(DirectFileSystem); + let scratch = ScratchPool::new(ScratchLimits { + max_bytes: 32 * 1024 * 1024, + max_files: 1, + })?; + let services = ReadServices { + now_unix_millis: Arc::new(cellar::asyncrt::wall_ms), + fill_random: Arc::new(|bytes| cellar::asyncrt::rng("transition-precopy").fill_bytes(bytes)), + }; + let images = if rooted { + CellGroupApplyImages::Extents { + images: NodeExtentStore::open( + base.join("extents"), + filesystem.clone(), + Default::default(), + )?, + services, + limits: Default::default(), + cache_directory: base.join("cache"), + cache_pool: scratch.clone(), + cache: CacheLimits { + max_cached_bytes: EXTENT_BYTES, + max_cached_extents: 1, + }, + } + } else { + CellGroupApplyImages::Materialized { + images: GenerationStore::open( + base.join("images"), + filesystem.clone(), + Default::default(), + )?, + services, + } + }; + let limits = limits(); + Ok(CellGroupTransferTarget { + store: CellGroupArchiveStore::new( + route, + ManifestPageStore::open( + &base.join("pages"), + filesystem, + limits.manifest_pages, + limits.manifest_bytes, + limits.retained_roots, + )?, + images, + limits.manifest, + limits.entries, + limits.max_effect_rows, + )?, + scratch, + directory: base, + }) +} + +pub(super) async fn acquire_hold( + domain: &ExecutionDomain, + registry: &CellGroupArchiveRegistry, +) -> Result { + Ok(tokio::time::timeout(TIMEOUT, async { + let mut intent = registry.prepare_acquire(domain).await?; + loop { + match registry.acquire(domain, &intent).await { + Ok(hold) => return Ok(hold), + // A newer registry revision has fenced this exact intent. + Err(CellGroupArchiveError::Expired) => { + intent = registry.prepare_acquire(domain).await?; + } + // Reconcile the same allocation until a hold or definite + // expiry is observed. The archive publisher is still live. + Err(CellGroupArchiveError::Conflict | CellGroupArchiveError::Unconfirmed) => { + tokio::task::yield_now().await; + } + Err(error) => return Err(error), + } + } + }) + .await??) +} + +pub(super) async fn verify_sealed_cut( + cluster: &mut ManagedCluster, + fleet: &FleetCatalog, + directory: &RangeDirectory, + transition: &ReopenedManagedCellGroupTransition, + parent: &CellGroupBirth, + old_hold: CellGroupArchiveHold, + seal: &GroupSourceSeal, +) -> Result<()> { + let domain = cluster.fixtures[0].domain.clone(); + let archives = CellGroupArchivePool::new(Default::default())?; + let tails = CellGroupArchiveTailPool::new(Default::default())?; + let registry = directory + .open_group_archive(domain.as_ref(), &archives, parent) + .await?; + let scratch = tempfile::tempdir()?; + let rooted = directory.namespace_config().image_storage == RaftImageStorage::Extents; + let source = |label: &str, hold: CellGroupArchiveHold| { + let base = scratch.path().join(label); + let route = parent.route().clone(); + let domain = domain.clone(); + async move { + let receiver_store = domain + .blocking(move || store(base, route, rooted))? + .await??; + let receiver = CellGroupCheckpointPool::new(checkpoint_limits())?.receive_archive( + receiver_store.store, + receiver_store.scratch, + receiver_store.directory, + )?; + Ok::<_, anyhow::Error>(CellGroupTransitionSource { + birth: parent.clone(), + hold, + receiver, + }) + } + }; + ensure!(matches!( + fleet + .verify_managed_sealed_sources( + transition, + directory, + &archives, + &tails, + vec![source("pre-copy", old_hold).await?], + TIMEOUT, + ) + .await, + Err(FleetCatalogError::Conflict) + )); + tokio::time::timeout(TIMEOUT, async { + loop { + for host in &cluster.hosts[..3] { + let group = host.group(parent.route().history)?; + if group + .archive_status()? + .and_then(|status| status.published) + .is_some_and(|marker| marker.source_seal() == Some(seal)) + { + return Ok::<_, anyhow::Error>(()); + } + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .context("managed archiver did not publish the sealed cut")??; + let held = acquire_hold(&domain, ®istry).await?; + ensure!(held.marker().source_seal() == Some(seal)); + + let first = fleet + .verify_managed_sealed_sources( + transition, + directory, + &archives, + &tails, + vec![source("verify", held.clone()).await?], + TIMEOUT, + ) + .await?; + ensure!(first.journal().phase() == TransitionPhase::Sealed); + ensure!(first.sources().len() == 1 && first.sources()[0].marker().source_seal() == Some(seal)); + let journal = first.journal().clone(); + let marker = first.sources()[0].marker().clone(); + ensure!(tails.usage().operations == 1); + drop(first); + ensure!(tails.usage().operations == 0); + let reopened = domain + .run(directory.sealed_source_holds(&transition.plan)) + .await??; + ensure!(reopened == vec![held.clone()]); + let replay = fleet + .verify_managed_sealed_sources( + transition, + directory, + &archives, + &tails, + vec![ + source( + "reopen", + reopened + .into_iter() + .next() + .context("selected source hold")?, + ) + .await?, + ], + TIMEOUT, + ) + .await?; + ensure!(replay.journal() == &journal); + ensure!(replay.sources()[0].marker() == &marker); + + let root = scratch.path().join("final"); + let routes = replay.children().to_vec(); + let targets = domain + .blocking(move || { + routes + .into_iter() + .enumerate() + .map(|(index, route)| store(root.join(index.to_string()), route, rooted)) + .collect::>>() + })? + .await??; + let prepared = recovery_pool()? + .prepare_transfer_partition(domain.as_ref(), replay.clone(), targets) + .await?; + ensure!(prepared.children().len() == 2); + ensure!(prepared + .children() + .iter() + .all(|child| child.genesis().is_some())); + ensure!( + prepared + .children() + .iter() + .map(|child| child.root().cells()) + .sum::() + == 2 + ); + let first = prepared + .children() + .iter() + .find(|child| child.root().cells() > 0) + .context("retained restart requires a nonempty child")? + .route() + .history; + drop(prepared); + let transition = Arc::new( + fleet + .reopen_managed_transition(directory, transition.plan.operation(), TIMEOUT) + .await? + .context("reopen managed sealed transition for install")?, + ); + let recovery = recovery_pool()?; + ensure!(matches!( + cluster.hosts[0] + .install_managed_transfer_child( + transition.clone(), + replay.clone(), + first, + recovery.clone(), + TIMEOUT, + ) + .await, + Err(CellGroupControllerError::Catalog(_)) + )); + ensure!(matches!( + cluster.hosts[0].group(first), + Err(RuntimeError::NotFound) + )); + let mut retained = None; + for route in replay.children() { + let mut foreign_route = route.clone(); + foreign_route.epoch = RouteEpoch::new(route.epoch.get() + 1)?; + ensure!(matches!( + fleet + .observe_managed_transfer_install_quorum( + &transition, + directory, + &foreign_route, + TIMEOUT, + ) + .await, + Err(FleetCatalogError::Conflict) + )); + ensure!(fleet + .observe_managed_transfer_install_quorum(&transition, directory, route, TIMEOUT) + .await? + .is_none()); + let mut installed = Vec::new(); + for (index, host) in cluster.hosts[3..].iter().enumerate() { + let receipt = host + .install_managed_transfer_child( + transition.clone(), + replay.clone(), + route.history, + recovery.clone(), + TIMEOUT, + ) + .await + .with_context(|| format!("install pending voter {}", index + 1))?; + ensure!(receipt.member.get() == index as u64 + 1); + ensure!(receipt.node == host.identity().node && receipt.disk == host.identity().disk); + ensure!( + recovery.usage().operations == 0, + "a completed install retained the recovery slot" + ); + installed.push(receipt.clone()); + let observed = fleet + .observe_managed_transfer_install_quorum(&transition, directory, route, TIMEOUT) + .await?; + if index == 0 { + ensure!(observed.is_none()); + ensure!(matches!( + host.attach_managed_transfer_child(transition.clone(), route.clone(), TIMEOUT) + .await, + Err(CellGroupControllerError::Catalog(_)) + )); + } else { + let observed = observed.context("managed install majority")?; + ensure!(observed.genesis.route() == route && observed.voters == installed); + } + let repeated = host + .install_managed_transfer_child( + transition.clone(), + replay.clone(), + route.history, + recovery.clone(), + TIMEOUT, + ) + .await?; + ensure!(repeated == receipt); + if route.history == first && index == 0 { + retained = Some(receipt); + } + ensure!(matches!( + host.group(route.history)?.status().await, + Err(RuntimeError::NotFound) + )); + } + } + Box::pin(restart_destination( + cluster, + directory, + transition.clone(), + replay.clone(), + first, + recovery, + retained.context("first destination receipt")?, + )) + .await?; + let voters = &cluster.hosts[DESTINATION..]; + let mut proofs = Vec::with_capacity(replay.children().len()); + for route in replay.children() { + if route.history != first { + for voter in voters { + voter + .attach_managed_transfer_child(transition.clone(), route.clone(), TIMEOUT) + .await + .context("attach second pending child")?; + } + } + proofs.push(wait_pending_child(voters, route.history).await?); + } + verify_managed_installed_decision( + cluster, + fleet, + directory, + &transition, + parent, + replay.children(), + proofs, + ) + .await?; + let routes = replay.children().to_vec(); + drop(replay); + ensure!(tails.usage().operations == 0); + let root = scratch.path().join("archive"); + let target_routes = routes.clone(); + let targets = domain + .blocking(move || { + target_routes + .into_iter() + .enumerate() + .map(|(index, route)| store(root.join(index.to_string()), route, rooted)) + .collect::>>() + })? + .await??; + let inputs = CellGroupTransitionPreparationInputs { + archives, + tails, + restore: recovery_pool()?, + sources: vec![source("archive-source", held).await?], + targets, + }; + Box::pin(verify_managed_archive_decision( + cluster, + fleet, + directory, + &transition, + parent, + &routes, + inputs, + )) + .await +} + +async fn verify_managed_archive_decision( + cluster: &ManagedCluster, + fleet: &FleetCatalog, + directory: &RangeDirectory, + transition: &ReopenedManagedCellGroupTransition, + parent: &CellGroupBirth, + children: &[GroupRoute], + inputs: CellGroupTransitionPreparationInputs, +) -> Result<()> { + let fixture = &cluster.fixtures[0]; + let domain = &fixture.domain; + let archives = inputs.archives.clone(); + let hold = inputs.sources[0].hold.clone(); + let before = domain.run(directory.observe()).await??; + ensure!(fleet + .reopen_managed_transfer_archive(transition, directory, TIMEOUT) + .await? + .is_none()); + + ensure!(matches!( + fleet + .publish_managed_transfer(transition, directory, &archives, TIMEOUT) + .await, + Err(FleetCatalogError::Conflict) + )); + fixture + .store + .fail_head_confirmation + .store(true, Ordering::Release); + ensure!(matches!( + fleet + .verify_managed_transfer_archive(transition, directory, inputs, TIMEOUT) + .await, + Err(FleetCatalogError::Unconfirmed) + )); + let reopened = fleet + .reopen_managed_transition(directory, transition.plan.operation(), TIMEOUT) + .await? + .context("archived plan")?; + let archived = reopened.journal.as_ref().context("archived journal")?; + ensure!(archived.phase() == TransitionPhase::Archived); + let closure = fleet + .reopen_managed_transfer_archive(&reopened, directory, TIMEOUT) + .await? + .context("selected managed archive closure")?; + ensure!(archived.transfer_archive() == Some(closure.digest)); + ensure!(closure.source_holds == vec![hold]); + ensure!(closure.children.len() == children.len()); + ensure!(closure + .children + .iter() + .zip(children) + .all(|(marker, route)| { marker.route() == route && !marker.is_active() })); + ensure!( + closure + .children + .iter() + .map(|marker| marker.root().cells()) + .sum::() + == 2 + ); + let after = domain.run(directory.observe()).await??; + ensure!(after.revision() == before.revision() && after.groups() == before.groups()); + ensure!( + domain + .run(directory.materialize_selected_history(parent.route().history)) + .await?? + .route() + == parent.route() + ); + Box::pin(verify_managed_publication( + cluster, + fleet, + directory, + &reopened, + &archives, + children, + closure.digest, + )) + .await +} + +async fn verify_managed_publication( + cluster: &ManagedCluster, + fleet: &FleetCatalog, + directory: &RangeDirectory, + transition: &ReopenedManagedCellGroupTransition, + archives: &CellGroupArchivePool, + children: &[GroupRoute], + closure: [u8; 32], +) -> Result<()> { + let fixture = &cluster.fixtures[0]; + let domain = &fixture.domain; + let voters = &cluster.hosts[DESTINATION..]; + let before = domain.run(directory.observe()).await??; + ensure!(matches!( + voters[0] + .group(children[0].history)? + .activate_managed_transfer_child(transition, TIMEOUT) + .await, + Err(CellGroupControllerError::Catalog(_)) + )); + + let controller = Fixture::open_catalog(domain, fixture.store.clone(), cluster.history.cluster)?; + let control = RaftFleetControl::new(domain, controller, vec![directory.clone()])?; + let namespace = hex::encode(cluster.history.namespace.bytes()); + let operation = hex::encode(transition.plan.operation().bytes()); + let principal = &transition.request.principal; + let status = control + .transition_status(&namespace, &operation, TIMEOUT) + .await?; + ensure!(status.phase == Some(TransitionPhase::Archived)); + let archive_digest = status + .transfer_archive_digest + .context("archived operator cut")?; + ensure!(archive_digest == hex::encode(closure)); + let publish = RaftFleetTransitionPublishRequest { + catalog_id: status.catalog_id, + namespace_id: namespace.clone(), + operation_id: operation.clone(), + request_id: hex::encode([218; 16]), + plan_digest: status.plan_digest, + archive_digest, + }; + let wrong_archive = RaftFleetTransitionPublishRequest { + archive_digest: hex::encode([33; 32]), + ..publish.clone() + }; + let refused = control + .publish_transition(principal, &namespace, &operation, &wrong_archive, TIMEOUT) + .await; + ensure!(matches!(refused, Err(failure) if failure.code == RaftFleetFailureCode::Conflict)); + ensure!(domain.run(directory.observe()).await??.revision() == before.revision()); + + fixture + .store + .fail_head_confirmation + .store(true, Ordering::Release); + let published = crate::managed_fixture::operator::publish_transition( + fixture, directory, principal, &publish, + ) + .await?; + ensure!(!fixture.store.fail_head_confirmation.load(Ordering::Acquire)); + let publication = domain + .run(directory.publication(&transition.plan)) + .await?? + .context("operator publication certificate")?; + ensure!( + publication.transfer_archive() == Some(closure) + && published.directory_revision == before.revision() + 1 + && published.directory_revision == publication.revision() + && published.publication_digest == hex::encode(publication.journal_digest()) + && Some(published.tree_digest.as_str()) + == publication.tree_digest().map(hex::encode).as_deref() + ); + ensure!( + control + .publish_transition(principal, &namespace, &operation, &publish, TIMEOUT) + .await? + == published + ); + let after = domain.run(directory.observe()).await??; + let selected = domain.run(directory.group_page(&after, None, 3)).await??; + ensure!(after.revision() == publication.revision()); + ensure!(selected.next.is_none() && selected.routes == children); + let reopened = fleet + .reopen_managed_transition(directory, transition.plan.operation(), TIMEOUT) + .await? + .context("published managed transfer")?; + ensure!( + reopened + .journal + .as_ref() + .context("publication journal")? + .phase() + == TransitionPhase::Published + ); + ensure!( + fleet + .publish_managed_transfer(&reopened, directory, archives, TIMEOUT) + .await? + == publication + ); + ensure!(matches!( + cluster.hosts[0] + .group(cluster.history)? + .retire_managed_source(&reopened, TIMEOUT) + .await, + Err(CellGroupControllerError::Catalog(_)) + )); + ensure!(matches!( + fleet + .prepare_managed_transfer_child_archive( + &reopened, + directory, + children[0].history, + archives, + TIMEOUT + ) + .await, + Err(FleetCatalogError::Conflict) + )); + for route in children { + for voter in voters { + ensure!(!voter.group(route.history)?.status().await?.ready); + } + } + + let request = |index: usize| RaftFleetTransitionActivationRequest { + catalog_id: control.catalog_id().to_owned(), + namespace_id: namespace.clone(), + operation_id: operation.clone(), + request_id: hex::encode([219 + index as u8; 16]), + plan_digest: hex::encode(reopened.plan.digest()), + group_id: hex::encode(children[index].history.group.bytes()), + incarnation_id: hex::encode(children[index].history.incarnation.bytes()), + }; + let first = request(0); + let leader = tokio::time::timeout(TIMEOUT, async { + loop { + for (index, voter) in voters.iter().enumerate() { + if voter.group(children[0].history)?.status().await?.leader + == Some(index as u64 + 1) + { + return Ok::<_, anyhow::Error>(voter); + } + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .context("operator needs a pending child leader")??; + let applied = crate::managed_fixture::operator::activate_transition_child( + fixture, directory, principal, leader, &first, + ) + .await?; + ensure!(applied.phase == TransitionPhase::Published && applied.applied.is_some()); + ensure!( + control + .activate_transition_child(principal, &namespace, &operation, &first, TIMEOUT) + .await? + == applied + ); + for voter in voters { + ensure!(!voter.group(children[1].history)?.status().await?.ready); + } + ensure!(matches!( + cluster.hosts[0] + .group(cluster.history)? + .retire_managed_source(&reopened, TIMEOUT) + .await, + Err(CellGroupControllerError::Catalog(_)) + )); + + let mut acknowledgements = Vec::with_capacity(children.len()); + for route in children { + acknowledgements.push(wait_managed_activation(voters, &reopened, route.history).await?); + } + ensure!(matches!( + fleet + .record_managed_transfer_activations( + &reopened, + directory, + vec![acknowledgements[0].clone()], + TIMEOUT + ) + .await, + Err(FleetCatalogError::Invalid(_)) + )); + let mut reversed = acknowledgements.clone(); + reversed.reverse(); + ensure!(matches!( + fleet + .record_managed_transfer_activations(&reopened, directory, reversed, TIMEOUT) + .await, + Err(FleetCatalogError::Conflict) + )); + let second = request(1); + let second_leader = crate::managed_fixture::leader(voters, children[1].history).await?; + let voter_control = control + .clone() + .with_node_observer(voters[second_leader].clone()); + fixture + .store + .fail_head_confirmation + .store(true, Ordering::Release); + let lost = voter_control + .activate_transition_child(principal, &namespace, &operation, &second, TIMEOUT) + .await; + ensure!(matches!(lost, Err(failure) if failure.code == RaftFleetFailureCode::Unavailable)); + ensure!(!fixture.store.fail_head_confirmation.load(Ordering::Acquire)); + let confirmed = control + .activate_transition_child(principal, &namespace, &operation, &second, TIMEOUT) + .await?; + let position = acknowledgements[1].position(); + ensure!( + confirmed.phase == TransitionPhase::Active + && confirmed + .applied + .is_some_and(|applied| applied.index == position.index.get() + && applied.term == position.term.get()) + ); + let active = fleet + .reopen_managed_transition(directory, transition.plan.operation(), TIMEOUT) + .await? + .context("active managed transfer")?; + let journal = active.journal.as_ref().context("activation journal")?; + ensure!(journal.phase() == TransitionPhase::Active && journal.activated_children().is_some()); + ensure!( + fleet + .record_managed_transfer_activations( + &active, + directory, + acknowledgements.clone(), + TIMEOUT + ) + .await? + == *journal + ); + + let positions = children + .iter() + .zip(&acknowledgements) + .map(|(route, ack)| (route.history, ack.position().index)) + .collect::>(); + wait_transfer_applied(voters, &positions).await +} + +pub(super) async fn wait_transfer_applied( + voters: &[RaftCells], + positions: &[(GroupHistory, LogIndex)], +) -> Result<()> { + tokio::time::timeout(TIMEOUT, async { + loop { + let mut applied = true; + for (history, index) in positions { + for voter in voters { + applied &= voter.group(*history)?.status().await?.applied.index >= *index; + } + } + if applied { + return Ok::<_, anyhow::Error>(()); + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .context("transfer voter apply did not reach the committed position")? +} + +pub(super) async fn wait_managed_activation( + voters: &[RaftCells], + transition: &ReopenedManagedCellGroupTransition, + history: GroupHistory, +) -> Result { + use CellGroupTransferActivation::{Accepted, Applied}; + let end = std::time::Instant::now() + TIMEOUT; + loop { + for voter in voters { + let remaining = end.saturating_duration_since(std::time::Instant::now()); + ensure!( + !remaining.is_zero(), + "managed transfer child did not activate" + ); + match voter + .group(history)? + .activate_managed_transfer_child(transition, remaining) + .await + { + Ok(Applied(acknowledgement)) => return Ok(*acknowledgement), + Ok(Accepted(_)) + | Err(CellGroupControllerError::Runtime( + RuntimeError::NotLeader | RuntimeError::NotReady, + )) => (), + Err(error) => return Err(error.into()), + } + } + tokio::time::sleep(Duration::from_millis(10)).await; + } +} + +pub(super) async fn verify_transferred_sql( + cluster: &mut ManagedCluster, + directory: &RangeDirectory, + seeded: SeededCells, +) -> Result<()> { + let domain = cluster.fixtures[0].domain.clone(); + let view = domain.run(directory.observe()).await??; + let hosts = &cluster.hosts[DESTINATION..]; + let mut positions = Vec::with_capacity(seeded.cells.len()); + for (index, (cell, original)) in seeded + .cells + .iter() + .cloned() + .zip(seeded.receipts) + .enumerate() + { + let route = domain.run(directory.lookup(&view, &cell.cell)).await??; + let leader = leader(hosts, route.history).await?; + let handle = hosts[leader].cell(cell)?; + let get = handle.contract("Get")?; + let query = handle + .query( + ProposalId::new([190 + index as u8; 16])?, + &get, + vec![], + QueryOptions::default(), + ) + .await?; + let expected = [10i64, 11][index]; + ensure!(query.result == expected.to_be_bytes()); + let add = handle.contract("Add")?; + let replay = handle + .command( + &add, + original.result.clone(), + CommandOptions::new(original.id, seeded.retry), + ) + .await?; + ensure!(replay.id == original.id && replay.result == original.result); + let next = handle + .command( + &add, + 1i64.to_be_bytes().to_vec(), + CommandOptions::new( + CommandId::new(original.id.client, [4, 2][index])?, + seeded.retry, + ), + ) + .await?; + ensure!(next.result == (expected + 1).to_be_bytes() && next.witness.group == route.history); + positions.push((route.history, next.witness.position.index)); + } + wait_transfer_applied(hosts, &positions).await?; + reopen_transfer_destination(cluster, 111, 2, true).await?; + let restarted = &cluster.hosts[DESTINATION]; + let view = domain.run(directory.observe()).await??; + for (index, cell) in seeded.cells.into_iter().enumerate() { + let route = domain.run(directory.lookup(&view, &cell.cell)).await??; + let group = restarted.group(route.history)?; + ensure!(group.archive_status()?.is_none()); + let handle = restarted.cell(cell)?; + let query = handle + .query( + ProposalId::new([194 + index as u8; 16])?, + &handle.contract("Get")?, + vec![], + QueryOptions { + routing: QueryRouting::Local, + ..Default::default() + }, + ) + .await?; + ensure!(query.result == [11i64, 12][index].to_be_bytes()); + ensure!(query.served_by_node == restarted.identity()); + } + Ok(()) +} + +pub(super) async fn initialize_transfer_archive( + fleet: &FleetCatalog, + directory: &RangeDirectory, + transition: &ReopenedManagedCellGroupTransition, + history: GroupHistory, + archives: &CellGroupArchivePool, + path: &std::path::Path, + domain: &ExecutionDomain, +) -> Result<()> { + use std::io::Write as _; + + let init = fleet + .prepare_managed_transfer_child_archive(transition, directory, history, archives, TIMEOUT) + .await?; + let mut intent = std::fs::File::create_new(path)?; + let encoded = init.encoded_intent().to_vec(); + intent.write_all(&encoded)?; + intent.sync_all()?; + std::fs::File::open(path.parent().context("child archive intent directory")?)?.sync_all()?; + init.commit(domain).await?; + fleet + .verify_managed_transfer_child_archive( + transition, directory, history, archives, &encoded, TIMEOUT, + ) + .await?; + ensure!(matches!( + fleet + .prepare_managed_transfer_child_archive( + transition, directory, history, archives, TIMEOUT + ) + .await, + Err(FleetCatalogError::Conflict) + )); + Ok(()) +} + +pub(super) async fn wait_active_archive( + host: &RaftCells, + history: GroupHistory, +) -> Result { + let group = host.group(history)?; + let applied = group.status().await?.applied.index; + tokio::time::timeout(TIMEOUT, async { + loop { + if group + .archive_status()? + .and_then(|status| status.published) + .is_some_and(|marker| marker.position().index >= applied && marker.is_active()) + { + return Ok::<_, anyhow::Error>(()); + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .context("child archiver did not publish its active cut")??; + Ok(applied) +} + +pub(super) async fn verify_child_archive_coverage( + cluster: &mut ManagedCluster, + directory: &RangeDirectory, + transition: &ReopenedManagedCellGroupTransition, + tombstone: CellGroupSourceRetirementAck, + cells: Vec, +) -> Result<()> { + let domain = cluster.fixtures[0].domain.clone(); + let fleet = cluster.fixtures[0].catalog.clone(); + let archives = CellGroupArchivePool::new(Default::default())?; + ensure!(matches!( + fleet + .settle_managed_transfer_retirement( + transition, + directory, + &archives, + vec![tombstone.clone()], + TIMEOUT + ) + .await, + Err(FleetCatalogError::Conflict) + )); + let scratch = tempfile::tempdir()?; + let rooted = directory.namespace_config().image_storage == RaftImageStorage::Extents; + let view = domain.run(directory.observe()).await??; + let children = domain + .run(directory.group_page(&view, None, 3)) + .await?? + .routes; + ensure!(children.len() == 2); + for (index, route) in children.iter().enumerate() { + initialize_transfer_archive( + &fleet, + directory, + transition, + route.history, + &archives, + &scratch.path().join(format!("child-{index}.intent")), + &domain, + ) + .await?; + } + + reopen_transfer_destination(cluster, 113, 2, true).await?; + let host = &cluster.hosts[DESTINATION]; + let mut applied = Vec::with_capacity(children.len()); + for route in children { + applied.push(( + route.history, + wait_active_archive(host, route.history).await?, + )); + } + let verifier_root = tempfile::tempdir_in(scratch.path())?; + let verifier = crate::managed_fixture::verifier::open( + &cluster.fixtures[0], + verifier_root.path(), + rooted, + Default::default(), + ) + .await?; + ensure!(fleet + .reopen_managed_transfer_child_holds(transition, directory, TIMEOUT) + .await? + .is_none()); + let work = + fleet.verify_managed_transfer_child_archives(transition, directory, &verifier, TIMEOUT); + let (recorded, delivered) = Box::pin(fault::lose_directory_confirmation(work)).await; + ensure!( + delivered && matches!(recorded, Err(FleetCatalogError::Unconfirmed)), + "child archive CAS result: {recorded:?}" + ); + let reopened = fleet + .reopen_managed_transition(directory, transition.plan.operation(), TIMEOUT) + .await? + .context("managed child archive decision")?; + let journal = reopened.journal.as_ref().context("child archive journal")?; + ensure!( + journal.phase() == TransitionPhase::Active + && journal.child_archives().is_some() + && !journal.parent_holds_released() + ); + let holds = fleet + .reopen_managed_transfer_child_holds(&reopened, directory, TIMEOUT) + .await? + .context("selected child archive holds")?; + ensure!(holds.len() == applied.len()); + for (hold, (history, index)) in holds.iter().zip(applied) { + ensure!( + hold.marker().route().history == history && hold.marker().position().index >= index + ); + } + let reserved = fleet.read(TIMEOUT).await?; + let early = FleetRequestIdentity { + operation: reopened.request.operation, + ..request(237, reserved.revision()) + }; + ensure!(matches!( + fleet + .confirm_managed_transfer_children( + &reserved, &early, &reopened, directory, &archives, TIMEOUT + ) + .await, + Err(FleetCatalogError::Conflict) + )); + ensure!(matches!( + fleet + .settle_managed_transfer_retirement( + &reopened, + directory, + &archives, + Vec::new(), + TIMEOUT + ) + .await, + Err(FleetCatalogError::Invalid(_)) + )); + let work = fleet.settle_managed_transfer_retirement( + &reopened, + directory, + &archives, + vec![tombstone.clone()], + TIMEOUT, + ); + let (lost, delivered) = Box::pin(fault::lose_directory_confirmation(work)).await; + ensure!( + delivered && matches!(lost, Err(FleetCatalogError::Unconfirmed)), + "retirement CAS result: {lost:?}" + ); + let retired = fleet + .settle_managed_transfer_retirement( + &reopened, + directory, + &archives, + vec![tombstone], + TIMEOUT, + ) + .await?; + ensure!( + retired.phase() == TransitionPhase::Retired + && retired.retired_parents().is_some() + && retired.child_archives() == journal.child_archives() + && !retired.parent_holds_released() + ); + ensure!( + domain + .run(directory.active_transition()) + .await?? + .context("retirement retains topology ownership")? + .1 + == retired + ); + ensure!(matches!( + cluster.hosts[DESTINATION] + .group(holds[0].marker().route().history)? + .activate_managed_transfer_child(&reopened, TIMEOUT) + .await, + Err(CellGroupControllerError::Catalog(_)) + )); + reopen_transfer_destination(cluster, 114, 2, true) + .await + .context("reopen activated child voters during retirement cleanup")?; + Box::pin(verify_live_transfer_children( + cluster, directory, &reopened, &archives, &holds, cells, + )) + .await +} + +async fn verify_live_transfer_children( + cluster: &mut ManagedCluster, + directory: &RangeDirectory, + transition: &ReopenedManagedCellGroupTransition, + archives: &CellGroupArchivePool, + holds: &[CellGroupArchiveHold], + cells: Vec, +) -> Result<()> { + let domain = cluster.fixtures[0].domain.clone(); + let fleet = cluster.fixtures[0].catalog.clone(); + let reserved = fleet.read(TIMEOUT).await?; + let confirmation = FleetRequestIdentity { + operation: transition.request.operation, + ..request(238, reserved.revision()) + }; + for hold in holds { + let history = hold.marker().route().history; + ensure!( + *fleet.history(&reserved, &history, TIMEOUT).await?.state() + == FleetHistoryState::Reserved + ); + } + cluster.fixtures[0] + .store + .fail_fleet_confirmation + .store(true, Ordering::Release); + let lost = fleet + .confirm_managed_transfer_children( + &reserved, + &confirmation, + transition, + directory, + archives, + TIMEOUT, + ) + .await; + ensure!( + matches!(lost, Err(FleetCatalogError::Storage(_))), + "promotion CAS result: {:?}", + lost.as_ref().err() + ); + let live = fleet.read(TIMEOUT).await?; + ensure!(live.revision() == reserved.revision() + 1); + for hold in holds { + let history = hold.marker().route().history; + ensure!(*fleet.history(&live, &history, TIMEOUT).await?.state() == FleetHistoryState::Live); + } + ensure!( + fleet + .confirm_managed_transfer_children( + &live, + &confirmation, + transition, + directory, + archives, + TIMEOUT, + ) + .await? + .applied_revision + == live.revision() + ); + ensure!(domain.run(directory.active_transition()).await??.is_some()); + reopen_transfer_destination(cluster, 115, 2, true) + .await + .context("reopen Live transfer children using permanent proofs")?; + let view = domain.run(directory.observe()).await??; + let voters = &cluster.hosts[DESTINATION..]; + let retry = u64::try_from( + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH)? + .as_millis(), + )? + .checked_add(60_000) + .context("retry deadline")?; + for (index, cell) in cells.into_iter().enumerate() { + let route = domain.run(directory.lookup(&view, &cell.cell)).await??; + let reopened = &voters[0]; + ensure!(reopened.group(route.history)?.archive_status()?.is_some()); + let handle = reopened.cell(cell.clone())?; + let query = handle + .query( + ProposalId::new([196 + index as u8; 16])?, + &handle.contract("Get")?, + vec![], + QueryOptions { + routing: QueryRouting::Local, + ..Default::default() + }, + ) + .await?; + ensure!(query.result == [11i64, 12][index].to_be_bytes()); + ensure!(query.served_by_node == reopened.identity()); + let leader = &voters[leader(voters, route.history).await?]; + let handle = leader.cell(cell)?; + let receipt = handle + .command( + &handle.contract("Add")?, + 1i64.to_be_bytes().to_vec(), + CommandOptions::new(CommandId::new(ClientId::new([201; 16])?, 1)?, retry), + ) + .await?; + ensure!(receipt.result == [12i64, 13][index].to_be_bytes()); + ensure!(receipt.witness.group == route.history); + } + Ok(()) +} + +pub(super) async fn verify_released_source_holds( + cluster: &ManagedCluster, + directory: &RangeDirectory, + plan: &DirectoryMutation, + registry: &CellGroupArchiveRegistry, + preparation: CellGroupArchiveHold, +) -> Result<()> { + let domain = &cluster.fixtures[0].domain; + let mut holds = domain.run(directory.sealed_source_holds(plan)).await??; + ensure!(holds.len() == 1); + holds.push(preparation); + verify_expired_holds(domain, directory, registry, holds).await +} + +pub(super) async fn verify_expired_holds( + domain: &ExecutionDomain, + directory: &RangeDirectory, + registry: &CellGroupArchiveRegistry, + holds: Vec, +) -> Result<()> { + let scratch = tempfile::tempdir()?; + let rooted = directory.namespace_config().image_storage == RaftImageStorage::Extents; + let tails = CellGroupArchiveTailPool::new(Default::default())?; + for (index, hold) in holds.into_iter().enumerate() { + let base = scratch.path().join(format!("released-source-{index}")); + let route = hold.marker().route().clone(); + let stored = domain + .blocking(move || store(base, route, rooted))? + .await??; + let receiver = CellGroupCheckpointPool::new(checkpoint_limits())?.receive_archive( + stored.store, + stored.scratch, + stored.directory, + )?; + ensure!(matches!( + registry.verify_hold(domain, &hold, &tails, receiver).await, + Err(CellGroupArchiveError::Expired) + )); + } + Ok(()) +} + +async fn verify_managed_installed_decision( + cluster: &ManagedCluster, + fleet: &FleetCatalog, + directory: &RangeDirectory, + transition: &ReopenedManagedCellGroupTransition, + parent: &CellGroupBirth, + children: &[GroupRoute], + proofs: Vec, +) -> Result<()> { + let fixture = &cluster.fixtures[0]; + let domain = &fixture.domain; + let before = domain.run(directory.observe()).await??; + ensure!(matches!( + fleet + .record_managed_transfer_child_quorums( + transition, + directory, + vec![proofs[0].clone()], + TIMEOUT + ) + .await, + Err(FleetCatalogError::Invalid(_)) + )); + let mut reversed = proofs.clone(); + reversed.reverse(); + ensure!(matches!( + fleet + .record_managed_transfer_child_quorums(transition, directory, reversed, TIMEOUT) + .await, + Err(FleetCatalogError::Conflict) + )); + let operation = transition.plan.operation(); + let prior = fleet + .reopen_managed_transition(directory, operation, TIMEOUT) + .await? + .context("sealed plan")?; + ensure!(prior.journal.context("sealed journal")?.phase() == TransitionPhase::Sealed); + let voters = &cluster.hosts[DESTINATION..]; + let make_request = |index: usize, catalog: &str| RaftFleetTransitionChildSetupRequest { + catalog_id: catalog.into(), + namespace_id: hex::encode(children[index].history.namespace.bytes()), + operation_id: hex::encode(operation.bytes()), + request_id: hex::encode([180 + index as u8; 16]), + plan_digest: hex::encode(transition.plan.digest()), + child: RaftFleetTransitionGroup { + group_id: hex::encode(children[index].history.group.bytes()), + incarnation_id: hex::encode(children[index].history.incarnation.bytes()), + }, + step: RaftFleetTransitionChildStep::Quorum, + }; + let mut first = None; + for (index, route) in children.iter().enumerate() { + let (leader, _) = wait_pending_child_witness(voters, route.history).await?; + let controller = + Fixture::open_catalog(domain, fixture.store.clone(), cluster.history.cluster)?; + let control = RaftFleetControl::new(domain, controller, vec![directory.clone()])? + .with_node_observer(voters[leader].clone()); + let request = make_request(index, control.catalog_id()); + let work = control.setup_transition_child( + &transition.request.principal, + &request.namespace_id, + &request.operation_id, + &request, + TIMEOUT, + ); + if index == 0 { + let observed = work.await?; + ensure!(matches!( + observed.result, + RaftFleetTransitionChildSetupResult::Quorum { + phase: TransitionPhase::Sealed, + .. + } + )); + first = Some(observed); + } else { + let (result, delivered) = fault::lose_directory_confirmation(work).await; + ensure!( + delivered + && matches!(result, Err(ref failure) + if failure.code == RaftFleetFailureCode::Unavailable && failure.outcome == RaftFleetOutcome::Unknown), + "lost install confirmation was not isolated to its directory CAS: {result:?}" + ); + } + } + let controller = Fixture::open_catalog(domain, fixture.store.clone(), cluster.history.cluster)?; + let control = RaftFleetControl::new(domain, controller.clone(), vec![directory.clone()])?; + let first = first.context("first selected child witness")?; + let replay = control + .setup_transition_child( + &transition.request.principal, + &first.request.namespace_id, + &first.request.operation_id, + &first.request, + TIMEOUT, + ) + .await?; + let RaftFleetTransitionChildSetupResult::Quorum { index, term, .. } = first.result else { + anyhow::bail!("missing first child quorum witness"); + }; + ensure!( + replay.result + == RaftFleetTransitionChildSetupResult::Quorum { + index, + term, + phase: TransitionPhase::Installed, + } + ); + let second = make_request(1, control.catalog_id()); + let replay = control + .setup_transition_child( + &transition.request.principal, + &second.namespace_id, + &second.operation_id, + &second, + TIMEOUT, + ) + .await?; + ensure!(matches!( + replay.result, + RaftFleetTransitionChildSetupResult::Quorum { + phase: TransitionPhase::Installed, + .. + } + )); + let reopened = controller + .reopen_managed_transition(directory, operation, TIMEOUT) + .await? + .context("installed plan")?; + let installed = reopened.journal.as_ref().context("installed journal")?; + ensure!( + installed.phase() == TransitionPhase::Installed && installed.installed_children().is_some() + ); + let after = domain.run(directory.observe()).await??; + ensure!(after.revision() == before.revision() && after.groups() == before.groups()); + ensure!( + domain + .run(directory.materialize_selected_history(parent.route().history)) + .await?? + .route() + == parent.route() + ); + Ok(()) +} + +async fn restart_destination( + cluster: &mut ManagedCluster, + directory: &RangeDirectory, + transition: Arc, + sources: VerifiedCellGroupTransitionSources, + child: GroupHistory, + recovery: CellGroupRecoveryPool, + expected: InstalledTransferVoter, +) -> Result<()> { + reopen_transfer_destination(cluster, 109, 0, false).await?; + let fixture = &cluster.fixtures[DESTINATION]; + let host = &cluster.hosts[DESTINATION]; + for route in sources.children() { + ensure!(matches!( + host.group(route.history)?.status().await, + Err(RuntimeError::NotFound) + )); + } + ensure!(matches!( + host.install_managed_transfer_child( + transition.clone(), + sources.clone(), + child, + recovery.clone(), + TIMEOUT, + ) + .await, + Err(CellGroupControllerError::Catalog(_)) + )); + let route = sources + .children() + .iter() + .find(|route| route.history == child) + .cloned() + .context("retained child route")?; + let historical = fixture + .catalog + .observe_managed_transfer_install_quorum(&transition, directory, &route, TIMEOUT) + .await? + .context("durable receipts survive an ordinary boot")?; + ensure!(historical.voters.len() == 3 && historical.voters.contains(&expected)); + ensure!(matches!( + host.attach_managed_transfer_child(transition.clone(), route.clone(), TIMEOUT) + .await, + Err(CellGroupControllerError::Catalog(_)) + )); + admit_capacity(fixture, host, directory, 225).await?; + let reopened = host + .install_managed_transfer_child( + transition.clone(), + sources.clone(), + child, + recovery.clone(), + TIMEOUT, + ) + .await?; + ensure!(reopened == expected); + ensure!(matches!( + host.group(child)?.status().await, + Err(RuntimeError::NotFound) + )); + let voters = &cluster.hosts[DESTINATION..]; + // Boot and capacity are rooted before peer discovery installs the new + // endpoint. The transport fences the old process during that interval. + let revision = fixture.catalog.read(TIMEOUT).await?.revision(); + for voter in &voters[1..] { + inspect_current(voter, host.identity().node, revision) + .await + .context("discover restarted pending voter before peer attachment")?; + } + voters[0] + .attach_managed_transfer_child(transition.clone(), route.clone(), TIMEOUT) + .await + .context("attach restarted pending voter before its peers")?; + ensure!(matches!( + voters[0] + .group(child)? + .managed_pending_transfer_quorum(TIMEOUT) + .await, + Err(CellGroupControllerError::Runtime( + RuntimeError::NotLeader | RuntimeError::NotReady + )) + )); + for voter in &voters[1..] { + voter + .attach_managed_transfer_child(transition.clone(), route.clone(), TIMEOUT) + .await + .context("attach peer of restarted pending voter")?; + } + wait_pending_child(voters, child).await?; + for voter in voters { + voter + .attach_managed_transfer_child(transition.clone(), route.clone(), TIMEOUT) + .await + .context("reconcile an attached pending voter")?; + } + reopen_transfer_destination(cluster, 110, 1, false).await?; + let fixture = &cluster.fixtures[DESTINATION]; + let host = &cluster.hosts[DESTINATION]; + ensure!(matches!( + host.group(child)?.status().await, + Err(RuntimeError::NotFound) + )); + ensure!(matches!( + host.attach_managed_transfer_child(transition.clone(), route.clone(), TIMEOUT) + .await, + Err(CellGroupControllerError::Catalog(_)) + )); + admit_capacity(fixture, host, directory, 227).await?; + ensure!( + host.install_managed_transfer_child(transition.clone(), sources, child, recovery, TIMEOUT) + .await? + == expected + ); + host.attach_managed_transfer_child(transition, route, TIMEOUT) + .await + .context("reattach retained pending voter after the fresh term")?; + wait_pending_child(&cluster.hosts[DESTINATION..], child).await?; + Ok(()) +} + +const DESTINATION: usize = 3; + +async fn reopen_transfer_destination( + cluster: &mut ManagedCluster, + generation: u8, + pending_quorums: usize, + activated: bool, +) -> Result<()> { + let old = cluster.nodes.remove(DESTINATION); + let old_host = cluster.hosts.remove(DESTINATION); + let previous = old_host.identity(); + stop_transfer_destination(old, &old_host, 2, pending_quorums, activated).await?; + let fixture = &cluster.fixtures[DESTINATION]; + let node = start_node( + cluster.roots[DESTINATION].path(), + fixture.application.clone(), + &fixture.spec.label, + ) + .await?; + let startup = async { + let (mut config, services) = fixture + .config(&node, LogOpenMode::Resume, generation) + .await?; + crate::managed_fixture::archives::configure(&mut config); + node.start_raft_cells(config, services).await + } + .await; + let host = match startup { + Ok(host) => host, + Err(error) => { + let stopped = node.shutdown(ShutdownMode::Preserve).await; + return Err(error.context(format!("retained transfer startup, shutdown={stopped:?}"))); + } + }; + cluster.nodes.insert(DESTINATION, node); + cluster.hosts.insert(DESTINATION, host); + let host = &cluster.hosts[DESTINATION]; + ensure!(host.identity().process != previous.process); + ensure!(host.observe_node(TIMEOUT).await?.node.groups == 2); + Ok(()) +} + +pub(super) async fn wait_pending_child( + voters: &[RaftCells], + child: GroupHistory, +) -> Result { + wait_pending_child_witness(voters, child) + .await + .map(|(_, proof)| proof) +} + +pub(super) async fn wait_pending_child_witness( + voters: &[RaftCells], + child: GroupHistory, +) -> Result<(usize, TransferChildQuorum)> { + let mut last = vec![None; voters.len()]; + let result = tokio::time::timeout(TIMEOUT, async { + loop { + let mut applied = true; + let mut witness = None; + for (index, voter) in voters.iter().enumerate() { + let group = voter.group(child)?; + let status = group.status().await?; + ensure!(!status.ready, "a pending child served before publication"); + applied &= status.applied.index.get() > 1; + last[index] = Some((status, false)); + match group.managed_pending_transfer_quorum(TIMEOUT).await { + Ok(proof) => { + if let Some((_, observed)) = &mut last[index] { + *observed = true; + } + witness = Some((index, proof)); + } + Err(CellGroupControllerError::Runtime( + RuntimeError::NotLeader | RuntimeError::NotReady, + )) => (), + Err(error) => return Err(error.into()), + } + } + if let (true, Some(witness)) = (applied, witness) { + return Ok::<_, anyhow::Error>(witness); + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await; + match result { + Ok(witness) => witness, + Err(elapsed) => { + let owners = futures_util::future::join_all(voters.iter().map(|voter| async { + let physical = match voter.group(child) { + Ok(group) => group.observe(Duration::from_secs(1)).await.map(|sample| { + ( + sample.role, + sample.persisted, + sample.checkpoint, + sample.retained_entries, + ) + }), + Err(error) => Err(error), + }; + let usage = voter.resource_usage(); + let transport = usage.transport.map(|usage| { + ( + usage.peers, + usage.member_routes, + usage.accepted, + usage.refused, + usage.failed, + ) + }); + (physical, usage.host.controller, transport) + })) + .await; + Err(anyhow::Error::new(elapsed).context(format!( + "pending transfer child {child:?} must commit a fresh term, last voter status and proof: {last:?}, physical log/host/transport: {owners:?}", + ))) + } + } +} diff --git a/crates/cellar/tests/support/cell_group_managed_transition_retirement.rs b/crates/cellar/tests/support/cell_group_managed_transition_retirement.rs new file mode 100644 index 000000000..9fec610f8 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_managed_transition_retirement.rs @@ -0,0 +1,410 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! A retained sealed parent reopens for retirement after the route switch. +use super::*; +use cellar::application::consensus::verification::{ + cell_groups::TransitionJournal, FleetHistoryState, +}; +use std::time::Instant; + +pub(super) async fn reopen_and_retire( + cluster: &mut ManagedCluster, + directory: &RangeDirectory, + transition: &ReopenedManagedCellGroupTransition, +) -> Result { + let source = cluster.history; + let fixture = &cluster.fixtures[0]; + let publication = fixture + .domain + .run(directory.publication(&transition.plan)) + .await?? + .context("selected transfer publication")?; + let selected = fixture + .catalog + .reopen_managed_transition(directory, transition.plan.operation(), TIMEOUT) + .await? + .context("selected managed transfer")?; + ensure!( + selected + .journal + .as_ref() + .context("transfer journal")? + .phase() + == TransitionPhase::Active + ); + + restart_source(cluster, 0, 112).await?; + let fixture = &cluster.fixtures[0]; + let reopened = &cluster.hosts[0]; + ensure!(reopened.observe_node(TIMEOUT).await?.node.groups == 1); + ensure!(!reopened.group(source)?.status().await?.ready); + ensure!(reopened.group(source)?.archive_status()?.is_none()); + ensure!(fixture + .domain + .run(directory.materialize_selected_history(source)) + .await? + .is_err()); + + let voters = &cluster.hosts[..3]; + let tombstone = commit_on_all(voters, source, &selected).await?; + ensure!(tombstone.tombstone().publication() == publication.journal_digest()); + ensure!(!reopened.group(source)?.status().await?.ready); + let retained = fixture + .catalog + .reopen_managed_transition(directory, transition.plan.operation(), TIMEOUT) + .await? + .context("managed retirement remains owned")?; + let journal = retained.journal.context("active transition journal")?; + ensure!(journal.phase() == TransitionPhase::Active && !journal.parent_holds_released()); + Ok(tombstone) +} + +pub(super) async fn park_and_release_parent( + cluster: &mut ManagedCluster, + directory: &RangeDirectory, + transition: &ReopenedManagedCellGroupTransition, +) -> Result<()> { + let archives = CellGroupArchivePool::new(Default::default())?; + let fleet = cluster.fixtures[0].catalog.clone(); + let domain = cluster.fixtures[0].domain.clone(); + Box::pin(reopen_parked_source(cluster, 0)).await?; + let observer = &cluster.hosts[3]; + let revision = fleet.read(TIMEOUT).await?.revision(); + for source in &cluster.hosts[..3] { + inspect_current(observer, source.identity().node, revision).await?; + } + ensure!(matches!( + fleet + .release_managed_transfer_parent_holds( + transition, directory, &archives, observer, TIMEOUT, + ) + .await, + Err(FleetCatalogError::Unconfirmed) + )); + ensure!(!domain + .run(directory.active_transition()) + .await?? + .context("an attached parent keeps topology owned")? + .1 + .parent_holds_released()); + let cut = fleet.read(TIMEOUT).await?; + let premature = FleetRequestIdentity { + operation: transition.request.operation, + ..request(239, cut.revision()) + }; + ensure!(matches!( + fleet + .complete_managed_transfer(&cut, &premature, transition, directory, TIMEOUT) + .await, + Err(FleetCatalogError::Conflict) + )); + for index in 1..3 { + Box::pin(reopen_parked_source(cluster, index)).await?; + } + let observer = &cluster.hosts[3]; + let revision = fleet.read(TIMEOUT).await?.revision(); + for source in &cluster.hosts[..3] { + inspect_current(observer, source.identity().node, revision).await?; + } + let store = &cluster.fixtures[0].store; + store.deny_registry_writes.store(true, Ordering::Release); + let denied = fleet + .release_managed_transfer_parent_holds(transition, directory, &archives, observer, TIMEOUT) + .await; + store.deny_registry_writes.store(false, Ordering::Release); + ensure!(matches!(denied, Err(FleetCatalogError::Unconfirmed))); + ensure!(!domain + .run(directory.active_transition()) + .await?? + .context("archive outage keeps topology owned")? + .1 + .parent_holds_released()); + store.fail_head_confirmation.store(true, Ordering::Release); + let completed = fleet + .release_managed_transfer_parent_holds(transition, directory, &archives, observer, TIMEOUT) + .await?; + ensure!(!store.fail_head_confirmation.load(Ordering::Acquire)); + ensure!(completed.phase() == TransitionPhase::Retired && completed.parent_holds_released()); + ensure!(domain.run(directory.active_transition()).await??.is_none()); + ensure!( + fleet + .release_managed_transfer_parent_holds( + transition, directory, &archives, observer, TIMEOUT, + ) + .await? + == completed + ); + let cut = fleet.read(TIMEOUT).await?; + ensure!(cut.revision() == revision); + Box::pin(complete_fleet_transfer( + cluster, directory, transition, completed, + )) + .await?; + Ok(()) +} + +async fn complete_fleet_transfer( + cluster: &mut ManagedCluster, + directory: &RangeDirectory, + transition: &ReopenedManagedCellGroupTransition, + released: TransitionJournal, +) -> Result<()> { + let fleet = cluster.fixtures[0].catalog.clone(); + let domain = cluster.fixtures[0].domain.clone(); + let cut = fleet.read(TIMEOUT).await?; + let confirmation = FleetRequestIdentity { + operation: transition.request.operation, + ..request(240, cut.revision()) + }; + let parent = cluster.history; + let live = fleet.history(&cut, &parent, TIMEOUT).await?; + ensure!(*live.state() == FleetHistoryState::Live); + let store = &cluster.fixtures[0].store; + store.fail_fleet_confirmation.store(true, Ordering::Release); + ensure!(matches!( + fleet + .complete_managed_transfer(&cut, &confirmation, transition, directory, TIMEOUT) + .await, + Err(FleetCatalogError::Storage(_)) + )); + ensure!(!store.fail_fleet_confirmation.load(Ordering::Acquire)); + let settled = fleet.read(TIMEOUT).await?; + let retired = FleetHistoryState::Retired { + operation: transition.request.operation, + retirement: released + .retired_parents() + .context("released retirement digest")?, + }; + ensure!(settled.revision() == cut.revision() + 1); + ensure!(*fleet.history(&settled, &parent, TIMEOUT).await?.state() == retired); + ensure!(matches!( + fleet.authorize_group(&settled, &parent, TIMEOUT).await, + Err(FleetCatalogError::Conflict) + )); + ensure!( + fleet + .complete_managed_transfer(&settled, &confirmation, transition, directory, TIMEOUT) + .await? + .applied_revision + == settled.revision() + ); + ensure!( + fleet + .plan_managed_transition(&transition.request, directory, &transition.plan, TIMEOUT,) + .await? + == released + ); + let view = domain.run(directory.observe()).await??; + let children = domain.run(directory.group_page(&view, None, 3)).await??; + ensure!(children.next.is_none() && children.routes.len() == 2); + for child in children.routes { + fleet + .authorize_group(&settled, &child.history, TIMEOUT) + .await?; + } + let inventory = fleet + .check_inventory(&settled, std::slice::from_ref(directory), TIMEOUT) + .await?; + ensure!(inventory.histories() == 3 && inventory.namespaces() == 1); + + let revival = request(241, settled.revision()); + let revival = FleetMutation { + id: revival.id, + operation: revival.operation, + expected_revision: revival.expected_revision, + principal: revival.principal, + changes: vec![FleetChange { + table: FleetTable::Histories, + key: live + .binding() + .key() + .map_err(|_| anyhow::anyhow!("retired history key"))?, + value: Some(live.encode()?), + }], + }; + ensure!(matches!( + fleet.commit(&settled, &revival, TIMEOUT).await, + Err(FleetCatalogError::Conflict) + )); + let admitted = request(242, settled.revision()); + let unrelated = FleetMutation { + id: admitted.id, + operation: admitted.operation, + expected_revision: admitted.expected_revision, + principal: admitted.principal, + changes: vec![FleetChange { + table: FleetTable::Admissions, + key: "test-membership".into(), + value: None, + }], + }; + ensure!( + fleet + .commit(&settled, &unrelated, TIMEOUT) + .await? + .applied_revision + == settled.revision() + 1 + ); + restart_source(cluster, 0, 119).await?; + let revision = fleet.read(TIMEOUT).await?.revision(); + let observer = &cluster.hosts[3]; + let node = cluster.hosts[0].identity().node; + inspect_current(observer, node, revision).await?; + let retained = observer + .inspect_peer_physical_inventory(node, revision, TIMEOUT) + .await?; + ensure!( + retained.view.node.groups == 1 + && retained.view.node.reserved_bytes == retained.view.node.profile.bytes_per_group + ); + ensure!(retained + .inventory + .context("closed parent inventory")? + .group_history(&parent) + .context("closed parent slot")? + .parked_retired_source()); + Ok(()) +} + +async fn reopen_parked_source(cluster: &mut ManagedCluster, index: usize) -> Result<()> { + let source = cluster.history; + let host = &cluster.hosts[index]; + let identity = host.identity(); + let attached = host + .inspect_peer_physical_inventory(identity.node, 1, TIMEOUT) + .await?; + ensure!(attached.view.registered_histories == 1); + ensure!(!attached + .inventory + .context("attached source inventory")? + .group_history(&source) + .context("attached source")? + .parked_retired_source()); + + restart_source(cluster, index, 116 + index as u8).await?; + let observer = &cluster.hosts[3]; + let revision = cluster.fixtures[0].catalog.read(TIMEOUT).await?.revision(); + inspect_current(observer, identity.node, revision).await?; + let parked = observer + .inspect_peer_physical_inventory(identity.node, revision, TIMEOUT) + .await?; + let boot = parked + .view + .fleet_boot + .as_ref() + .context("parked source boot")?; + let old_boot = attached + .view + .fleet_boot + .as_ref() + .context("attached source boot")?; + ensure!( + boot.engine == old_boot.engine + && boot.disk == old_boot.disk + && boot.process != old_boot.process + ); + ensure!( + parked.view.node.groups == 1 + && parked.view.registered_histories == 0 + && parked.view.unattached_engine_histories == 1 + && parked.view.node.reserved_bytes == parked.view.node.profile.bytes_per_group + ); + let inventory = parked.inventory.context("parked source inventory")?; + let row = inventory + .group_history(&source) + .context("retained retired source")?; + ensure!(row.parked_retired_source() && row.in_configuration()); + ensure!(matches!( + cluster.hosts[index].group(source)?.status().await, + Err(RuntimeError::NotFound) + )); + Ok(()) +} + +async fn restart_source(cluster: &mut ManagedCluster, index: usize, generation: u8) -> Result<()> { + let fixture = &cluster.fixtures[index]; + let old = cluster.nodes.remove(index); + let previous = cluster.hosts.remove(index); + let identity = previous.identity(); + ensure!(matches!( + old.shutdown(ShutdownMode::Preserve).await?, + NodeOutcome::Stopped(_) + )); + let report = previous.close().await?; + ensure!( + report.groups_total == 1 + && report.groups.len() == 1 + && report.omitted_groups == 0 + && report + .groups + .iter() + .all(|group| !group.active && !group.failed) + && report.mutation_outcomes == MutationOutcomes::NoUnresolvedLocalProducers, + "sealed source shutdown inventory: {report:?}" + ); + let node = start_node( + cluster.roots[index].path(), + fixture.application.clone(), + &fixture.spec.label, + ) + .await?; + let startup = async { + let (mut config, services) = fixture + .config(&node, LogOpenMode::Resume, generation) + .await?; + crate::managed_fixture::archives::configure(&mut config); + node.start_raft_cells(config, services).await + } + .await; + let host = match startup { + Ok(host) => host, + Err(error) => { + let stopped = node.shutdown(ShutdownMode::Preserve).await; + return Err(error.context(format!("retained source startup, shutdown={stopped:?}"))); + } + }; + ensure!(host.identity().process != identity.process); + cluster.nodes.insert(index, node); + cluster.hosts.insert(index, host); + Ok(()) +} + +pub(super) async fn commit_on_all( + voters: &[RaftCells], + source: GroupHistory, + transition: &ReopenedManagedCellGroupTransition, +) -> Result { + let tombstone = commit(voters, source, transition).await?; + for voter in voters { + let local = commit(std::slice::from_ref(voter), source, transition).await?; + ensure!(local.tombstone() == tombstone.tombstone()); + } + Ok(tombstone) +} + +async fn commit( + voters: &[RaftCells], + source: GroupHistory, + transition: &ReopenedManagedCellGroupTransition, +) -> Result { + let deadline = Instant::now() + TIMEOUT; + loop { + for voter in voters { + let remaining = deadline.saturating_duration_since(Instant::now()); + ensure!(!remaining.is_zero(), "sealed source did not retire"); + match voter + .group(source)? + .retire_managed_source(transition, remaining) + .await + { + Ok(CellGroupSourceRetirement::Applied(ack)) => return Ok(*ack), + Ok(CellGroupSourceRetirement::Accepted(_)) + | Err(CellGroupControllerError::Runtime( + RuntimeError::NotLeader | RuntimeError::NotReady, + )) => (), + Err(error) => return Err(error.into()), + } + } + tokio::time::sleep(Duration::from_millis(10)).await; + } +} diff --git a/crates/cellar/tests/support/cell_group_membership.rs b/crates/cellar/tests/support/cell_group_membership.rs new file mode 100644 index 000000000..6780bce64 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_membership.rs @@ -0,0 +1,924 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Whole-group configuration changes over actual TCP Raft and production apply. +use super::*; +use cellar::application::consensus::verification::JoiningMember; +use cellar_logic::consensus::OperationId; +use raft::{ + prelude::{ConfChangeSingle, ConfChangeTransition, ConfChangeType, ConfChangeV2}, + Storage, +}; + +pub(super) fn change( + member: u64, + kind: ConfChangeType, + transition: ConfChangeTransition, +) -> ConfChangeV2 { + ConfChangeV2 { + changes: vec![ConfChangeSingle { + node_id: member, + change_type: kind, + ..Default::default() + }] + .into(), + transition, + ..Default::default() + } +} + +pub(super) async fn configure_at( + fixture: &RuntimeCluster, + leader: usize, + route: &GroupRoute, + command: ConfChangeV2, + joins: &[JoiningMember], +) -> Result { + let group = fixture.cluster.voters[leader] + .log + .cell_group(&route.history)?; + let label = format!("{command:?}"); + let request = CellGroupMembershipChange::new(route, &group.membership()?, command, joins)?; + let accepted = tokio::time::timeout(Duration::from_secs(10), async { + loop { + match fixture.runtimes[leader] + .configure(route.history, request.clone()) + .await + { + Ok(accepted) => return Ok::<_, anyhow::Error>(accepted), + // Retry only a definite pre-admission refusal, preserving the + // same observed revision. A lost response is never retried. + Err(RuntimeError::NotReady) => tokio::task::yield_now().await, + Err(error) => return Err(error.into()), + } + } + }) + .await??; + controller::applied(fixture, route.history, accepted.position.index) + .await + .with_context(|| format!("configuration apply: {label}"))?; + Ok(accepted) +} + +pub(super) async fn configure( + fixture: &RuntimeCluster, + route: &GroupRoute, + command: ConfChangeV2, + joins: &[JoiningMember], +) -> Result { + configure_at(fixture, 0, route, command, joins).await +} + +/// Hold an actual runtime message across a committed membership change. The +/// configuration may not invalidate the old physical send's membership binding. +pub(crate) async fn membership_delivery_fence(rooted: bool) -> Result<()> { + membership_delivery_fence_case(rooted, false).await +} + +pub(crate) async fn membership_physical_delivery_fence(rooted: bool) -> Result<()> { + membership_delivery_fence_case(rooted, true).await +} + +async fn membership_delivery_fence_case(rooted: bool, physical: bool) -> Result<()> { + let class = RaftClass::new("Counter", 1, [81; 32])?; + let (mut fixture, route, _, _) = Box::pin(registered::setup(rooted, &class)).await?; + let second = controller::add_second(&mut fixture, route.history).await?; + for _ in 0..16 { + fixture.pump(&[], &mut BTreeMap::new()).await?; + tokio::task::yield_now().await; + } + let (transports, sink) = if physical { + // Declare the sink before any pool is installed. A physical node has + // one endpoint across groups; replacing one group's existing endpoint + // would correctly be refused as a stale peer before admitting a send. + for (node, runtime) in fixture.runtimes.iter().enumerate() { + fixture.transports.push(tcp::start(runtime, node).await?); + } + let sink = tokio::net::TcpListener::bind("127.0.0.1:0").await?; + for history in fixture.applied.keys() { + let mut route = fixture.tcp_route(*history)?; + route.peers[2].control_address = sink.local_addr()?; + fixture.transports[0].install_physical_group(route)?; + } + (std::mem::take(&mut fixture.transports), Some(sink)) + } else { + (Vec::new(), None) + }; + for _ in 0..2 { + fixture.runtimes[0].tick().await?; + } + let prior = tokio::time::timeout(Duration::from_secs(5), async { + loop { + let event = fixture.events[0].recv().await.context("leader event")?; + match event.effect().map_err(|e| anyhow::anyhow!(e.to_string()))? { + CellGroupEffect::Message(message) if message.to == 3 => { + return Ok::<_, anyhow::Error>(event) + } + CellGroupEffect::Message(message) => { + fixture.network.push_back((route.history, message.clone())) + } + _ => anyhow::bail!("expected settled protocol message"), + } + } + }) + .await??; + let (prior, socket) = if let Some(sink) = sink { + let delivery = transports[0].submit_cell_group_event(&prior)?; + let (socket, _) = tokio::time::timeout(Duration::from_secs(5), sink.accept()).await??; + // The physical handshake is entered. Dropping both response owners must + // not refund the runtime admission still held by the actual SendRequest. + drop(delivery); + drop(prior); + (None, Some(socket)) + } else { + (Some(prior), None) + }; + let group = fixture.cluster.voters[0].log.cell_group(&route.history)?; + let request = CellGroupMembershipChange::new( + &route, + &group.membership()?, + change( + 3, + ConfChangeType::AddLearnerNode, + ConfChangeTransition::Explicit, + ), + &[], + )?; + let accepted = fixture.runtimes[0] + .configure(route.history, request) + .await?; + let mut held = BTreeMap::new(); + tokio::time::timeout(Duration::from_secs(5), async { + while group.initial_state()?.hard_state.commit < accepted.position.index.get() { + fixture.pump(&[route.history], &mut held).await?; + tokio::task::yield_now().await; + } + Ok::<_, anyhow::Error>(()) + }) + .await??; + // A membership fence is group-local even though sends, persistence and + // output admission are node-wide. Elect/apply another group while held. + fixture.runtimes[0].campaign(second).await?; + tokio::time::timeout(Duration::from_secs(5), async { + loop { + fixture.pump(&[route.history], &mut held).await?; + let status = fixture.runtimes[0].status(second).await?; + if status.ready + && fixture.applied[&second] + .iter() + .all(|i| *i >= status.applied.index) + { + break; + } + tokio::task::yield_now().await; + } + Ok::<_, anyhow::Error>(()) + }) + .await??; + // Inject later protocol work while the finite prior-send boundary is held. + for _ in 0..8 { + fixture.runtimes[0].tick().await?; + fixture.pump(&[route.history], &mut held).await?; + } + assert!( + !held.contains_key(&route.history), + "configuration Apply overtook an owned prior message" + ); + assert!(group.membership()?.revision() < accepted.position.index); + drop(prior); + drop(socket); + let notices = tokio::time::timeout(Duration::from_secs(5), async { + let mut notices = Vec::new(); + while notices.len() < 2 { + let event = fixture.events[0] + .recv() + .await + .context("commit announcement")?; + if event.group() != route.history { + match event.effect().map_err(|e| anyhow::anyhow!(e.to_string()))? { + CellGroupEffect::Message(message) => { + fixture.network.push_back((event.group(), message.clone())) + } + CellGroupEffect::Applied { manifest, .. } => { + fixture + .applied + .get_mut(&event.group()) + .context("other group")?[0] = manifest.position().index + } + _ => anyhow::bail!( + "unexpected other-group work while awaiting configuration announcement" + ), + } + continue; + } + let CellGroupEffect::Message(message) = + event.effect().map_err(|e| anyhow::anyhow!(e.to_string()))? + else { + anyhow::bail!("configuration skipped terminal message fence"); + }; + assert_eq!( + message.get_msg_type(), + MessageType::MsgAppend, + "fenced group emitted a nonterminal message after releasing its prior send" + ); + assert!(message.entries.is_empty()); + assert_eq!( + (message.index, message.log_term, message.commit), + ( + accepted.position.index.get(), + accepted.position.term.get(), + accepted.position.index.get() + ) + ); + notices.push(event); + } + let targets = notices + .iter() + .map(|event| match event.effect().unwrap() { + CellGroupEffect::Message(message) => message.to, + _ => unreachable!(), + }) + .collect::>(); + assert_eq!(targets, std::collections::BTreeSet::from([2, 3])); + Ok::<_, anyhow::Error>(notices) + }) + .await??; + for _ in 0..2 { + fixture.runtimes[0].tick().await?; + fixture.pump(&[route.history], &mut held).await?; + } + assert!( + !held.contains_key(&route.history), + "configuration skipped its terminal announcement deliveries" + ); + drop(notices); + fixture.drive(Some(accepted.position.index)).await?; + assert_eq!(group.membership()?.revision(), accepted.position.index); + fixture.transports = transports; + fixture.close().await +} + +pub(crate) async fn membership_lifecycle(rooted: bool) -> Result<()> { + let class = RaftClass::new("Counter", 1, [81; 32])?; + let (mut fixture, route, a, _) = Box::pin(registered::setup(rooted, &class)).await?; + let second = controller::add_second(&mut fixture, route.history).await?; + fixture.connect_tcp().await?; + for transport in &fixture.transports { + transport.enable_group_checkpoints(CellGroupCheckpointPool::new( + checkpoint_fixture::limits(), + )?)?; + } + let mut limits = controller::limits(); + limits.checkpoints = Some(CellGroupCheckpointPolicy { + interval_entries: 2, + max_parallel: 1, + }); + let controllers = controller::start_with_limits(&mut fixture, limits).await?; + fixture.runtimes[0].campaign(second).await?; + controller::wait_for(async || Ok(fixture.runtimes[0].status(second).await?.ready)).await?; + let group = fixture.cluster.voters[0].log.cell_group(&route.history)?; + // Administrative observations address the physical group, with no chosen + // cell. Every invocation obtains its own current-term ReadIndex barrier. + let deadline = fixture + .cluster + .domain + .run(async { cellar::asyncrt::mono_ms() + 10_000 }) + .await?; + let observation = fixture.runtimes[0] + .quorum_membership(route.history, deadline) + .await?; + assert_eq!(observation.replica().scope, route.history); + assert_eq!(observation.root().route(), &route); + assert_eq!(observation.configuration().voters, vec![1, 2, 3]); + assert!(observation.barrier().index.get() > 0); + assert!(observation.root().position().index >= observation.barrier().index); + assert!(observation.membership().revision() <= observation.barrier().index); + assert!(fixture.runtimes[0].usage().read_pins.items > 0); + drop(observation); + let expired = fixture + .cluster + .domain + .run(async { cellar::asyncrt::mono_ms() }) + .await?; + assert!(matches!( + fixture.runtimes[0] + .quorum_membership(route.history, expired) + .await, + Err(RuntimeError::DeadlineExceeded) + )); + let original = fixture.runtimes[0] + .execution_authority(route.history, a.clone()) + .await?; + let root = group.group_manifest()?.context("root")?.root(); + let demote = change( + 3, + ConfChangeType::AddLearnerNode, + ConfChangeTransition::Explicit, + ); + let stale = CellGroupMembershipChange::new(&route, &group.membership()?, demote.clone(), &[])?; + let foreign = fixture.extra[&second][0] + .log + .cell_group(&second)? + .group_manifest()? + .context("second root")?; + let wrong_group = + CellGroupMembershipChange::new(foreign.route(), &group.membership()?, demote.clone(), &[])?; + let last = group.last_index()?; + assert!(fixture.runtimes[0] + .configure(route.history, wrong_group) + .await + .is_err()); + assert_eq!(group.last_index()?, last); + let entered = configure(&fixture, &route, demote, &[]).await?; + let deadline = fixture + .cluster + .domain + .run(async { cellar::asyncrt::mono_ms() + 10_000 }) + .await?; + let observation = fixture.runtimes[0] + .quorum_membership(route.history, deadline) + .await?; + assert_eq!(observation.membership().revision(), entered.position.index); + assert!(observation.barrier().index >= entered.position.index); + assert_eq!(observation.configuration().voters, vec![1, 2]); + assert_eq!(observation.configuration().voters_outgoing, vec![1, 2, 3]); + assert_eq!(observation.configuration().learners_next, vec![3]); + assert_eq!(observation.root().root(), root); + drop(observation); + for voter in &fixture.cluster.voters { + let log = voter.log.cell_group(&route.history)?; + let conf = log.initial_state()?.conf_state; + assert_eq!(conf.voters, vec![1, 2]); + assert_eq!(conf.voters_outgoing, vec![1, 2, 3]); + assert_eq!(conf.learners_next, vec![3]); + assert_eq!(log.membership()?.revision(), entered.position.index); + assert_eq!(log.group_manifest()?.context("root")?.root(), root); + } + let last = group.last_index()?; + assert!(fixture.runtimes[0] + .configure(route.history, stale) + .await + .is_err()); + assert_eq!(group.last_index()?, last); + let last = group.last_index()?; + let term = fixture.runtimes[0].status(route.history).await?.term.get(); + let mut malformed = group + .entries( + entered.position.index.get(), + entered.position.index.get() + 1, + None::, + raft::GetEntriesContext::empty(false), + )? + .remove(0); + let mut bytes = malformed.data.to_vec(); + bytes.extend_from_slice(&[8, 0]); // Duplicate transition with conflicting value. + malformed.data = bytes.into(); + malformed.index = last + 1; + malformed.term = term + 1; + let message = raft::prelude::Message { + msg_type: raft::prelude::MessageType::MsgAppend, + from: 2, + to: 1, + term: term + 1, + index: last, + log_term: group.term(last)?, + entries: vec![malformed].into(), + ..Default::default() + }; + assert!(fixture.runtimes[0] + .step(route.history, message) + .await + .is_err()); + assert_eq!( + fixture.runtimes[0].status(route.history).await?.term.get(), + term + ); + assert_eq!(group.last_index()?, last); + configure(&fixture, &route, ConfChangeV2::default(), &[]).await?; + assert_eq!(group.initial_state()?.conf_state.learners, vec![3]); + + let join = JoiningMember { + member: MemberId::new(4)?, + node: NodeId::new([4; 16])?, + disk: DiskLineage::new([14; 16])?, + }; + configure( + &fixture, + &route, + change( + 4, + ConfChangeType::AddLearnerNode, + ConfChangeTransition::Auto, + ), + std::slice::from_ref(&join), + ) + .await?; + assert_eq!( + group + .membership()? + .joined(join.member)? + .context("bound learner")? + .0, + join + ); + // Checkpoint sends have the same retryable route-discovery refusal as + // ordinary append traffic; no stream or source owner is admitted. + let snapshot = raft::prelude::Message { + msg_type: raft::prelude::MessageType::MsgSnapshot, + from: 1, + to: 4, + ..Default::default() + }; + assert!(matches!( + fixture.transports[0].submit_physical(&route.history, snapshot), + Err(TransportError::StalePeer) + )); + controller::wait_for(async || Ok(controllers[0].usage().send_refusals > 0)).await?; + assert!( + !controllers[0].usage().closing, + "discovery lag must not close the node" + ); + let promotion = CellGroupMembershipChange::new( + &route, + &group.membership()?, + change(4, ConfChangeType::AddNode, ConfChangeTransition::Auto), + &[], + )?; + let last = group.last_index()?; + assert!(matches!( + fixture.runtimes[0] + .configure(route.history, promotion) + .await, + Err(RuntimeError::NotReady) + )); + assert_eq!(group.last_index()?, last); + configure( + &fixture, + &route, + change(3, ConfChangeType::AddNode, ConfChangeTransition::Auto), + &[], + ) + .await?; + configure( + &fixture, + &route, + change(4, ConfChangeType::RemoveNode, ConfChangeTransition::Auto), + &[], + ) + .await?; + let reused = CellGroupMembershipChange::new( + &route, + &group.membership()?, + change( + 4, + ConfChangeType::AddLearnerNode, + ConfChangeTransition::Auto, + ), + &[join], + )?; + assert!(fixture.runtimes[0] + .configure(route.history, reused) + .await + .is_err()); + assert_eq!(group.membership()?.high_water(), 4); + + // Implicit joint consensus must generate and durably apply its own empty + // leave entry. The harness never manufactures that automatic command. + let implicit = configure( + &fixture, + &route, + change( + 3, + ConfChangeType::AddLearnerNode, + ConfChangeTransition::Implicit, + ), + &[], + ) + .await?; + controller::wait_for(async || { + for voter in &fixture.cluster.voters { + let log = voter.log.cell_group(&route.history)?; + if log.membership()?.revision() <= implicit.position.index + || !log.initial_state()?.conf_state.voters_outgoing.is_empty() + { + return Ok(false); + } + } + Ok(true) + }) + .await + .context("automatic joint leave did not finish")?; + configure( + &fixture, + &route, + change(3, ConfChangeType::AddNode, ConfChangeTransition::Auto), + &[], + ) + .await?; + assert_eq!( + fixture.runtimes[0] + .reconciliation_authority(route.history, a.clone()) + .await? + .cell(), + original.cell() + ); + + let progress = fixture.runtimes[0] + .replication_status(route.history) + .await?; + assert_eq!(progress.replica.scope, route.history); + assert_eq!(progress.replica.member, MemberId::new(1)?); + for (term, revision) in [ + ( + Term::new(progress.term.get() + 1), + progress.membership_revision, + ), + ( + progress.term, + LogIndex::new(progress.membership_revision.get() + 1), + ), + ] { + assert!(matches!( + fixture.runtimes[0] + .transfer_leadership( + route.history, + cellar::application::consensus::LeadershipTransfer { + target: MemberId::new(2)?, + expected_term: term, + expected_revision: revision + } + ) + .await, + Err(RuntimeError::Invalid(_)) + )); + } + + // Move leadership using the real TimeoutNow protocol. The fixture does + // not advance election ticks or inject votes to manufacture this handoff. + assert!(fixture.runtimes[0] + .transfer_leader(route.history, 4) + .await + .is_err()); + controller::wait_for(async || { + match fixture.runtimes[0].transfer_leader(route.history, 2).await { + Ok(()) => Ok(true), + Err(RuntimeError::NotReady) => Ok(false), + Err(error) => Err(error.into()), + } + }) + .await?; + assert!(original.check().is_err()); + controller::wait_for(async || Ok(fixture.runtimes[1].status(route.history).await?.ready)) + .await?; + let producer = fixture.runtimes[1] + .execution_authority(route.history, a.clone()) + .await?; + let cell = history(&route, "after-new-leader")?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(73);")?; + let accepted = fixture.runtimes[1] + .propose( + route.history, + apply::enrollment_bytes(&route, &cell, &bytes, rooted)?, + ) + .await?; + controller::applied(&fixture, route.history, accepted.position.index).await?; + // Demote the new current leader. Captured producer authority must be + // revoked while another group on the original node continues serving. + configure_at( + &fixture, + 1, + &route, + change( + 2, + ConfChangeType::AddLearnerNode, + ConfChangeTransition::Explicit, + ), + &[], + ) + .await?; + configure_at(&fixture, 1, &route, ConfChangeV2::default(), &[]).await?; + assert!(producer.check().is_err()); + assert!(!fixture.runtimes[1].status(route.history).await?.ready); + assert!(fixture.runtimes[0].status(second).await?.ready); + let second_root = fixture.cluster.voters[0] + .log + .cell_group(&second)? + .group_manifest()? + .context("second")?; + let cell = history(second_root.route(), "unrelated")?; + let accepted = fixture.runtimes[0] + .propose( + second, + apply::enrollment_bytes(second_root.route(), &cell, &bytes, false)?, + ) + .await?; + controller::applied(&fixture, second, accepted.position.index).await?; + let conf = group.initial_state()?.conf_state; + let membership = group.membership()?; + assert_eq!(conf.voters, vec![1, 3]); + assert_eq!(conf.learners, vec![2]); + assert_eq!(membership.high_water(), 4); + assert!(group.checkpoint_manifest()?.is_some()); + for controller in &controllers { + assert_eq!(controller.usage().quarantined_groups, 0); + controller.close().await?; + } + drop(controllers); + fixture.cluster.voters[0].log.close().await?; + let reopened = fixture + .cluster + .domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + fixture.cluster._root.path().join("node-1/log"), + NodeId::new([1; 16])?, + DiskLineage::new([11; 16])?, + ProcessGeneration::new([21; 16])?, + LogOpenMode::Resume, + ), + Arc::new(DirectFileSystem), + )) + .await??; + let restored = reopened.cell_group(&route.history)?; + assert_eq!(restored.initial_state()?.conf_state, conf); + assert_eq!(restored.membership()?, membership); + assert_eq!( + restored + .group_manifest()? + .context("restored root")? + .membership_revision(), + membership.revision().get() + ); + // Metadata reopening alone is not permission to serve or a disk-replacement proof. + reopened.close().await?; + fixture.close().await +} + +pub(crate) async fn membership_sync() -> Result<()> { + for close in [false, true] { + let class = RaftClass::new("Counter", 1, [81; 32])?; + let (mut fixture, route, _, _) = Box::pin(registered::setup(false, &class)).await?; + let group = fixture.cluster.voters[0].log.cell_group(&route.history)?; + let before = group.group_manifest()?.context("before")?; + let membership = group.membership()?; + let conf = group.initial_state()?.conf_state; + let request = CellGroupMembershipChange::new( + &route, + &membership, + change( + 3, + ConfChangeType::AddLearnerNode, + ConfChangeTransition::Explicit, + ), + &[], + )?; + let accepted = fixture.runtimes[0] + .configure(route.history, request) + .await?; + let mut held = BTreeMap::new(); + tokio::time::timeout(Duration::from_secs(10), async { + while !held.contains_key(&route.history) { + fixture.pump(&[route.history], &mut held).await?; + tokio::task::yield_now().await; + } + Ok::<_, anyhow::Error>(()) + }) + .await??; + let work = held.remove(&route.history).context("configuration apply")?; + let mut incomplete = GroupWrite::new(&group); + incomplete.conf_state = Some(conf.clone()); + incomplete.membership = Some(membership.clone()); + assert!(fixture.cluster.voters[0] + .log + .persist_cells(OperationId::new(991), vec![incomplete]) + .await + .is_err()); + let (sql, applier) = fixture.appliers[&(0, route.history)].clone(); + let prepared = sql.prepare_apply(&applier, work).await?; + let gate = fixture.cluster.voters[0].gate.clone(); + let release = gate.arm(fault::HOLD_SYNC); + prepared.complete(&fixture.runtimes[0]).await?; + tokio::time::timeout(Duration::from_secs(5), gate.entered.notified()).await?; + assert_eq!(group.group_manifest()?.as_ref(), Some(&before)); + assert_eq!(group.initial_state()?.conf_state, conf); + assert_eq!(group.membership()?, membership); + assert_eq!( + fixture.runtimes[0].status(route.history).await?.applied, + before.position() + ); + if close { + assert!( + tokio::time::timeout(Duration::from_millis(30), fixture.runtimes[0].close()) + .await + .is_err() + ); + } + drop(release); + if close { + fixture.runtimes[0].close().await?; + } else { + fixture.drive(Some(accepted.position.index)).await?; + } + assert_eq!(group.membership()?.revision(), accepted.position.index); + assert_eq!( + group + .group_manifest()? + .context("after")? + .membership_revision(), + accepted.position.index.get() + ); + assert_eq!(group.initial_state()?.conf_state.learners_next, vec![3]); + fixture.close().await?; + } + Ok(()) +} + +pub(crate) async fn membership_quorum_floor() -> Result<()> { + for rooted in [false, true] { + let class = RaftClass::new("Counter", 1, [81; 32])?; + let (mut fixture, route, _, _) = Box::pin(registered::setup(rooted, &class)).await?; + fixture.connect_tcp().await?; + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let runtime = fixture.runtimes[0].clone(); + controller::wait_for(async || Ok(runtime.status(route.history).await?.ready)).await?; + let group = fixture.cluster.voters[0].log.cell_group(&route.history)?; + let before = group.group_manifest()?.context("before")?; + let request = CellGroupMembershipChange::new( + &route, + &group.membership()?, + change( + 3, + ConfChangeType::AddLearnerNode, + ConfChangeTransition::Explicit, + ), + &[], + )?; + let gate = fixture.cluster.voters[0].gate.clone(); + let release = gate.arm(fault::HOLD_SYNC); + let accepted = runtime.configure(route.history, request).await?; + tokio::time::timeout(Duration::from_secs(5), gate.entered.notified()).await?; + assert_eq!(group.group_manifest()?.as_ref(), Some(&before)); + // A real held engine sync keeps control work in the serialized queue. + // Expiry must win at owner execution even though the configuration also + // changes: without the deadline check this would report Invalid instead. + let transfer_deadline = fixture + .cluster + .domain + .run(async { cellar::asyncrt::mono_ms() + 40 }) + .await?; + let transfer = runtime.transfer_leadership_before( + route.history, + cellar::application::consensus::LeadershipTransfer { + target: MemberId::new(2)?, + expected_term: runtime.status(route.history).await?.term, + expected_revision: LogIndex::new(before.membership_revision()), + }, + transfer_deadline, + ); + tokio::pin!(transfer); + assert!( + tokio::time::timeout(Duration::from_millis(60), &mut transfer) + .await + .is_err() + ); + let deadline = fixture + .cluster + .domain + .run(async { cellar::asyncrt::mono_ms() + 10_000 }) + .await?; + let read = + runtime.quorum_membership_after(route.history, accepted.position.index, deadline); + tokio::pin!(read); + assert!(tokio::time::timeout(Duration::from_millis(30), &mut read) + .await + .is_err()); + assert!(runtime.usage().read_pins.items > 0); + drop(release); + assert!(matches!( + tokio::time::timeout(Duration::from_secs(5), &mut transfer).await?, + Err(RuntimeError::DeadlineExceeded) + )); + let observed = tokio::time::timeout(Duration::from_secs(10), &mut read).await??; + assert_eq!(observed.membership().revision(), accepted.position.index); + assert!(observed.barrier().index >= accepted.position.index); + assert!(observed.root().position().index >= observed.barrier().index); + assert_eq!(observed.configuration().learners_next, vec![3]); + assert_eq!(observed.status().leader, Some(1)); + drop(observed); + controller::wait_for(async || Ok(runtime.usage().read_pins.items == 0)).await?; + let unreachable = LogIndex::new(accepted.position.index.get() + 10_000); + let deadline = fixture + .cluster + .domain + .run(async { cellar::asyncrt::mono_ms() + 30 }) + .await?; + assert!(matches!( + runtime + .quorum_membership_after(route.history, unreachable, deadline) + .await, + Err(RuntimeError::DeadlineExceeded) + )); + controller::wait_for(async || Ok(runtime.usage().read_pins.items == 0)).await?; + for controller in controllers { + controller.close().await?; + } + fixture.close().await?; + } + Ok(()) +} + +pub(crate) async fn membership_quorum_followers() -> Result<()> { + use raft::prelude::{Entry, Message, MessageType}; + for rooted in [false, true] { + let class = RaftClass::new("Counter", 1, [81; 32])?; + let (mut fixture, route, cell, _) = Box::pin(registered::setup(rooted, &class)).await?; + fixture.connect_tcp().await?; + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + controller::wait_for(async || Ok(fixture.runtimes[0].status(route.history).await?.ready)) + .await?; + let accepted = configure_at( + &fixture, + 0, + &route, + change( + 3, + ConfChangeType::AddLearnerNode, + ConfChangeTransition::Explicit, + ), + &[], + ) + .await?; + let expected = fixture.cluster.voters[0] + .log + .cell_group(&route.history)? + .group_manifest()? + .context("selected group")?; + assert_eq!(expected.root().cells(), 2); + + let mut context = b"cellar-group-membership-read-1\0".to_vec(); + context.extend_from_slice(&2u64.to_be_bytes()); + context.extend_from_slice(&[97; 16]); + context.extend_from_slice(&1u64.to_be_bytes()); + let status = fixture.runtimes[1].status(route.history).await?; + let forged = Message { + msg_type: MessageType::MsgReadIndexResp, + from: 3, + to: 2, + term: status.term.get(), + index: expected.position().index.get(), + entries: vec![Entry { + data: context.into(), + ..Default::default() + }] + .into(), + ..Default::default() + }; + assert!(matches!( + fixture.runtimes[1].step(route.history, forged).await, + Err(RuntimeError::Invalid(_)) + )); + + for node in 0..3 { + let runtime = &fixture.runtimes[node]; + let deadline = fixture + .cluster + .domain + .run(async { cellar::asyncrt::mono_ms() + 10_000 }) + .await?; + let observed = runtime + .quorum_membership_after(route.history, accepted.position.index, deadline) + .await?; + assert_eq!(observed.replica().member.get(), node as u64 + 1); + assert_eq!( + observed.engine(), + fixture.cluster.voters[node].log.storage_instance()? + ); + assert_eq!(observed.root(), &expected); + assert!(observed.barrier().index >= accepted.position.index); + assert_eq!(observed.barrier().term, observed.status().term); + assert_eq!(observed.status().leader, Some(1)); + assert_eq!(observed.configuration().learners_next, vec![3]); + if node != 0 { + assert!(!observed.status().ready); + // A metadata barrier cannot become local leader/execution or + // membership-transfer authority on either follower. + assert!(matches!( + runtime.replication_status(route.history).await, + Err(RuntimeError::NotLeader) + )); + assert!(matches!( + runtime + .execution_authority(route.history, cell.clone()) + .await, + Err(RuntimeError::NotLeader) + )); + } + drop(observed); + controller::wait_for(async || Ok(runtime.usage().read_pins.items == 0)).await?; + } + for controller in controllers { + controller.close().await?; + } + fixture.close().await?; + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_membership_crash.rs b/crates/cellar/tests/support/cell_group_membership_crash.rs new file mode 100644 index 000000000..a54fbf85e --- /dev/null +++ b/crates/cellar/tests/support/cell_group_membership_crash.rs @@ -0,0 +1,382 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Retained voter disks resume joint consensus through production TCP controllers. +use super::*; +use raft::{ + prelude::{ConfChangeTransition, ConfChangeType, ConfChangeV2, ConfState}, + Storage, +}; +use serde::{Deserialize, Serialize}; +use std::path::Path; + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Oracle { + rooted: bool, + image_storage: RaftImageStorage, + joint: bool, + source_directory: String, + expected: GroupManifest, + objects: Vec<(String, Vec)>, +} + +fn configuration(conf: &ConfState, joint: bool) { + assert_eq!(conf.voters, vec![1, 2]); + if joint { + assert_eq!(conf.voters_outgoing, vec![1, 2, 3]); + assert_eq!(conf.learners_next, vec![3]); + assert!(conf.learners.is_empty()); + } else { + assert!(conf.voters_outgoing.is_empty()); + assert!(conf.learners_next.is_empty()); + assert_eq!(conf.learners, vec![3]); + } + assert!(!conf.auto_leave); +} + +pub(crate) async fn membership_crash(rooted: bool) -> Result<()> { + for cut in ["joint", "left"] { + let directory = tempfile::tempdir()?; + crash_fixture::kill_at(directory.path(), "membership_process_fixture", rooted, cut).await?; + Box::pin(recover(directory.path(), rooted, cut == "joint")) + .await + .with_context(|| format!("membership restart rooted={rooted} cut={cut}"))?; + } + Ok(()) +} + +pub(crate) async fn membership_process_fixture() -> Result<()> { + let (root, rooted, cut) = crash_fixture::parameters()?; + anyhow::ensure!( + cut == "joint" || cut == "left", + "invalid membership crash cut" + ); + let class = RaftClass::new("Counter", 1, [81; 32])?; + let (mut fixture, route, _, _) = Box::pin(registered::setup_in( + rooted, + &class, + tempfile::tempdir_in(&root)?, + Arc::new(DirectFileSystem), + )) + .await?; + let _controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + membership::configure( + &fixture, + &route, + membership::change( + 3, + ConfChangeType::AddLearnerNode, + ConfChangeTransition::Explicit, + ), + &[], + ) + .await?; + if cut == "left" { + membership::configure(&fixture, &route, ConfChangeV2::default(), &[]).await?; + } + let expected = fixture.cluster.voters[0] + .log + .cell_group(&route.history)? + .group_manifest()? + .context("source root")?; + for voter in &fixture.cluster.voters { + let group = voter.log.cell_group(&route.history)?; + configuration(&group.initial_state()?.conf_state, cut == "joint"); + assert_eq!(group.group_manifest()?.as_ref(), Some(&expected)); + } + let source_directory = fixture + .cluster + ._root + .path() + .file_name() + .context("fixture directory")? + .to_str() + .context("fixture directory name")? + .to_owned(); + let objects = crash_fixture::capture_catalog(fixture.cluster.store.as_ref()).await?; + crash_fixture::save( + &root.join("oracle.json"), + &serde_json::to_vec(&Oracle { + rooted, + image_storage: fixture.cluster.directory.namespace_config().image_storage, + joint: cut == "joint", + source_directory, + expected, + objects, + })?, + )?; + crash_fixture::stop_at(&root).await +} + +struct Node { + log: ReplicaLog, + runtime: CellGroupRuntime, + transport: RaftTransport, + controller: CellGroupController, +} + +async fn applied(nodes: &[Node], history: GroupHistory, index: LogIndex) -> Result<()> { + controller::wait_for(async || { + for node in nodes { + if node.runtime.status(history).await?.applied.index < index { + return Ok(false); + } + } + Ok(true) + }) + .await +} + +async fn enroll( + nodes: &[Node], + leader: usize, + route: &GroupRoute, + rooted: bool, + name: &str, +) -> Result { + let cell = history(route, name)?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(91);")?; + let accepted = nodes[leader] + .runtime + .propose( + route.history, + apply::enrollment_bytes(route, &cell, &bytes, rooted)?, + ) + .await?; + applied(nodes, route.history, accepted.position.index).await?; + Ok(accepted.position.index) +} + +async fn recover(root: &Path, rooted: bool, joint: bool) -> Result<()> { + let oracle: Oracle = serde_json::from_slice(&std::fs::read(root.join("oracle.json"))?)?; + assert_eq!(oracle.rooted, rooted); + assert_eq!(oracle.joint, joint); + let name = Path::new(&oracle.source_directory); + anyhow::ensure!( + name.components().count() == 1 + && matches!( + name.components().next(), + Some(std::path::Component::Normal(_)) + ), + "owned source directory name" + ); + let disk = root.join(name); + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let route = oracle.expected.route().clone(); + let (directory, birth) = + crash_fixture::restore_catalog(oracle.objects, &route, &domain, oracle.image_storage) + .await?; + let mut nodes = Vec::new(); + let mut peers = Vec::new(); + for n in 1..=3u8 { + let base = disk.join(format!("node-{n}")); + let log = domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + base.join("log"), + NodeId::new([n; 16])?, + DiskLineage::new([n + 10; 16])?, + ProcessGeneration::new([n + 90; 16])?, + LogOpenMode::Resume, + ), + Arc::new(DirectFileSystem), + )) + .await??; + let group = log.cell_group(&route.history)?; + assert_eq!(group.group_manifest()?.as_ref(), Some(&oracle.expected)); + configuration(&group.initial_state()?.conf_state, joint); + assert_eq!( + group.membership()?.revision().get(), + oracle.expected.membership_revision() + ); + let permit = domain + .run(directory.bootstrap_group_member(&birth, &log)) + .await??; + let filesystem = Arc::new(DirectFileSystem); + let pages = ManifestPageStore::open( + &base.join("pages"), + filesystem.clone(), + 1024, + 16 * 1024 * 1024, + 16, + )?; + let images = + GenerationStore::open(base.join("images"), filesystem.clone(), Default::default())?; + let extents = if rooted { + Some(NodeExtentStore::open( + disk.join(format!("rooted-{}", n - 1)), + filesystem, + Default::default(), + )?) + } else { + None + }; + let config = super::super::checkpoint_receive::config(images, extents, &base)?; + let recovering = group.clone(); + let (applier, recovered) = domain + .blocking(move || { + CellGroupApplier::recover( + recovering, + pages, + config, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + ) + })? + .await??; + assert_eq!(recovered.marker(), &oracle.expected); + assert_eq!(recovered.pinned_images(), 2); + let (runtime, events) = CellGroupRuntime::start( + log.clone(), + RaftRuntimeOptions { + tick_interval_ms: None, + ..Default::default() + }, + ) + .await?; + runtime.attach(permit, recovered).await?; + assert!( + !runtime.status(route.history).await?.ready, + "retained joint membership is not fresh quorum authority" + ); + let transport = tcp::start(&runtime, usize::from(n - 1)).await?; + peers.push(RaftPeer { + replica: group.identity().clone(), + label: format!("node-{n}"), + server_name: "localhost".into(), + control_address: transport.control_address(), + data_address: transport.data_address(), + query_address: transport.query_address(), + }); + let sql = CellGroupSqlPool::start( + &log, + CellSqlLimits { + workers: 1, + queue_per_worker: 1, + max_pending_cells: 2, + group_pending_cells: 2, + node_bytes: 1024 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + }, + )?; + let controller = CellGroupController::start( + runtime.clone(), + events, + transport.clone(), + sql, + vec![applier], + controller::limits(), + ) + .await?; + nodes.push(Node { + log, + runtime, + transport, + controller, + }); + } + let physical = cellar::application::consensus::verification::GroupRoute { + scope: route.history, + schema_digest: route.schema_digest, + revision: 1, + peers, + }; + for node in &nodes { + node.transport.install_physical_group(physical.clone())?; + } + nodes[0].runtime.campaign(route.history).await?; + let mut leader = None; + controller::wait_for(async || { + for (i, node) in nodes.iter().enumerate() { + if node.runtime.status(route.history).await?.ready { + leader = Some(i); + return Ok(true); + } + node.runtime.tick().await?; + } + Ok(false) + }) + .await + .context("retained voters regain current-term quorum")?; + let leader = leader.context("actual ready leader")?; + assert!( + leader < 2, + "demoted/outgoing-only member cannot lead the incoming configuration" + ); + enroll(&nodes, leader, &route, rooted, "after-joint-restart").await?; + for node in &nodes { + configuration( + &node + .log + .cell_group(&route.history)? + .initial_state()? + .conf_state, + joint, + ); + } + if joint { + let group = nodes[leader].log.cell_group(&route.history)?; + let request = CellGroupMembershipChange::new( + &route, + &group.membership()?, + ConfChangeV2::default(), + &[], + )?; + let left = nodes[leader] + .runtime + .configure(route.history, request) + .await?; + applied(&nodes, route.history, left.position.index).await?; + } + for node in &nodes { + configuration( + &node + .log + .cell_group(&route.history)? + .initial_state()? + .conf_state, + false, + ); + } + let final_index = enroll(&nodes, leader, &route, rooted, "after-joint-completion").await?; + for node in &nodes { + let manifest = node + .log + .cell_group(&route.history)? + .group_manifest()? + .context("final root")?; + assert_eq!(manifest.root().cells(), 4); + assert!(manifest.position().index >= final_index); + } + for name in [ + "registered-a", + "registered-b", + "after-joint-restart", + "after-joint-completion", + ] { + let cell = history(&route, name)?; + let deadline = domain + .run(async { cellar::asyncrt::mono_ms() + 5000 }) + .await?; + let selected = nodes[leader] + .runtime + .read_authority( + route.history, + cell.clone(), + CellReadConsistency::Linearizable, + deadline, + ) + .await?; + assert_eq!(selected.cell().version.history, cell); + assert_eq!(selected.cell().version.sequence.get(), 1); + } + assert!(!nodes[2].runtime.status(route.history).await?.ready); + eprintln!("membership crash rooted={rooted} joint_at_kill={joint} killed=true recovered_replicas=3 final_cells=4 final_index={}", final_index.get()); + for node in nodes { + node.controller.close().await?; + node.log.close().await?; + } + domain.shutdown().await; + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_multigroup.rs b/crates/cellar/tests/support/cell_group_multigroup.rs new file mode 100644 index 000000000..4fb08e12f --- /dev/null +++ b/crates/cellar/tests/support/cell_group_multigroup.rs @@ -0,0 +1,202 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Exercise simultaneous roles and independent apply on a shared node runtime. + +use super::*; +use ::raft::{GetEntriesContext, Storage}; + +pub(crate) async fn mixed_roles() -> Result<()> { + let mut fixture = RuntimeCluster::new().await?; + let first = fixture.cluster.birth.route().clone(); + let mut groups = vec![first]; + for id in 30..41 { + let scope = fixture.add_group(id).await?; + let marker = fixture.cluster.voters[0] + .log + .cell_group(&scope)? + .group_manifest()? + .context("new group manifest")?; + groups.push(marker.route().clone()); + } + for (index, route) in groups.iter().enumerate() { + fixture.runtimes[index % 3].campaign(route.history).await?; + } + let mut held = BTreeMap::new(); + tokio::time::timeout(Duration::from_secs(20), async { + loop { + fixture.pump(&[], &mut held).await?; + let mut ready = true; + for (index, route) in groups.iter().enumerate() { + ready &= fixture.runtimes[index % 3] + .status(route.history) + .await? + .ready; + } + if ready { + return Ok::<_, anyhow::Error>(()); + } + tokio::task::yield_now().await; + } + }) + .await??; + + for (index, route) in groups.iter().enumerate() { + let leader = index % 3; + let scope = route.history; + let bytes = images::image( + None, + &format!("CREATE TABLE value(n); INSERT INTO value VALUES ({index});"), + )?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + for name in ["alpha", "beta"] { + let data = request(route, history(route, name)?, 1, &cut)?; + let accepted = fixture.runtimes[leader] + .propose(scope, data.clone()) + .await?; + applied(&mut fixture, scope, accepted.position.index, &[], &mut held).await?; + for (member, voter) in fixture.cluster.voters.iter().enumerate() { + let status = fixture.runtimes[member].status(scope).await?; + assert_eq!(status.leader, Some(leader as u64 + 1)); + assert_eq!(status.ready, member == leader); + let group = voter.log.cell_group(&scope)?; + let index = accepted.position.index.get(); + let entries = + group.entries(index, index + 1, None, GetEntriesContext::empty(false))?; + assert_eq!(&entries[0].data[..], data.as_slice()); + } + } + for voter in &fixture.cluster.voters { + let marker = voter + .log + .cell_group(&scope)? + .group_manifest()? + .context("applied manifest")?; + assert_eq!(marker.root().cells(), 2); + } + } + fixture.close().await +} + +pub(crate) async fn hot_group_fairness() -> Result<()> { + let mut fixture = RuntimeCluster::new().await?; + let hot = fixture.cluster.birth.route().clone(); + let neighbor = fixture.add_group(51).await?; + let cold = fixture.add_group(61).await?; + let neighbor_route = fixture.cluster.voters[1] + .log + .cell_group(&neighbor)? + .group_manifest()? + .context("neighbor manifest")? + .route() + .clone(); + fixture.runtimes[0].campaign(hot.history).await?; + fixture.runtimes[1].campaign(neighbor).await?; + let mut held = BTreeMap::new(); + tokio::time::timeout(Duration::from_secs(20), async { + loop { + fixture.pump(&[], &mut held).await?; + if fixture.runtimes[0].status(hot.history).await?.ready + && fixture.runtimes[1].status(neighbor).await?.ready + { + return Ok::<_, anyhow::Error>(()); + } + tokio::task::yield_now().await; + } + }) + .await??; + + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES (1);")?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let data = request(&hot, history(&hot, "hot-first")?, 1, &cut)?; + let hot_target = fixture.runtimes[0].propose(hot.history, data).await?; + tokio::time::timeout(Duration::from_secs(20), async { + while !held.contains_key(&hot.history) { + fixture.pump(&[hot.history], &mut held).await?; + tokio::task::yield_now().await; + } + Ok::<_, anyhow::Error>(()) + }) + .await??; + + let mut flood = tokio::task::JoinSet::new(); + for index in 0..64u8 { + let runtime = fixture.runtimes[0].clone(); + let scope = hot.history; + let encoded = request(&hot, history(&hot, &format!("hot-{index}"))?, 1, &cut)?; + let mut entry = CellGroupEntry::decode(&encoded, &hot, GroupEntryLimits::default())?; + entry.binding.proposal = ProposalId::new([index + 2; 16])?; + let data = entry.encode(&hot, GroupEntryLimits::default())?; + flood.spawn(async move { runtime.propose(scope, data).await }); + } + let data = request( + &neighbor_route, + history(&neighbor_route, "neighbor")?, + 1, + &cut, + )?; + let accepted = fixture.runtimes[1].propose(neighbor, data).await?; + applied( + &mut fixture, + neighbor, + accepted.position.index, + &[hot.history], + &mut held, + ) + .await?; + fixture.runtimes[2].campaign(cold).await?; + tokio::time::timeout(Duration::from_secs(20), async { + while !fixture.runtimes[2].status(cold).await?.ready { + fixture.pump(&[hot.history], &mut held).await?; + tokio::task::yield_now().await; + } + Ok::<_, anyhow::Error>(()) + }) + .await??; + assert!(held.contains_key(&hot.history)); + assert!( + fixture.runtimes[0].status(hot.history).await?.applied.index < hot_target.position.index + ); + + flood.abort_all(); + while let Some(outcome) = flood.join_next().await { + match outcome { + Ok(Ok(_) | Err(RuntimeError::Overloaded)) => (), + Err(error) if error.is_cancelled() => (), + other => anyhow::bail!("hot submission failed: {other:?}"), + } + } + let work = held.remove(&hot.history).context("held hot apply")?; + let next = prepare(&mut fixture.cluster.voters[0], &work)?; + fixture.runtimes[0] + .complete_apply(hot.history, work, next) + .await?; + applied( + &mut fixture, + hot.history, + hot_target.position.index, + &[], + &mut held, + ) + .await?; + fixture.close().await +} + +async fn applied( + fixture: &mut RuntimeCluster, + scope: GroupHistory, + index: LogIndex, + hold: &[GroupHistory], + held: &mut BTreeMap, +) -> Result<()> { + tokio::time::timeout(Duration::from_secs(20), async { + while fixture.applied[&scope] + .iter() + .any(|applied| *applied < index) + { + fixture.pump(hold, held).await?; + tokio::task::yield_now().await; + } + Ok::<_, anyhow::Error>(()) + }) + .await? +} diff --git a/crates/cellar/tests/support/cell_group_network_probes.rs b/crates/cellar/tests/support/cell_group_network_probes.rs new file mode 100644 index 000000000..5f77721d6 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_network_probes.rs @@ -0,0 +1,387 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Independent wire client exercises authenticated framing on real TLS sockets. +use super::*; +use hmac::{Hmac, Mac}; +use protobuf::Message as _; +use serde_json::{json, Value}; +use tokio::io::{AsyncReadExt, AsyncWriteExt}; +use tokio::net::TcpStream; +use tokio_rustls::client::TlsStream; + +pub(super) struct Connection> { + stream: S, + pub(super) exporter: [u8; 32], +} + +impl Connection { + async fn open(address: std::net::SocketAddr) -> Result { + Self::open_with_exporter(address, b"EXPORTER-cellar-raft-1").await + } + + pub(super) async fn open_with_exporter( + address: std::net::SocketAddr, + label: &[u8], + ) -> Result { + Self::open_with_receive_buffer(address, label, None).await + } + + pub(super) async fn open_with_receive_buffer( + address: std::net::SocketAddr, + label: &[u8], + receive_buffer: Option, + ) -> Result { + let mut roots = rustls::RootCertStore::empty(); + roots.add( + include_bytes!("../../transport/tcp/fixtures/ca.der") + .to_vec() + .into(), + )?; + let mut config = rustls::ClientConfig::builder_with_provider(Arc::new( + rustls::crypto::ring::default_provider(), + )) + .with_protocol_versions(&[&rustls::version::TLS13])? + .with_root_certificates(roots) + .with_no_client_auth(); + config.alpn_protocols = vec![b"cellar-raft-1".to_vec()]; + let tcp = if let Some(bytes) = receive_buffer { + let socket = tokio::net::TcpSocket::new_v4()?; + socket.set_recv_buffer_size(bytes)?; + socket.connect(address).await? + } else { + TcpStream::connect(address).await? + }; + let stream = tokio_rustls::TlsConnector::from(Arc::new(config)) + .connect(rustls::pki_types::ServerName::try_from("localhost")?, tcp) + .await?; + let exporter = stream + .get_ref() + .1 + .export_keying_material([0; 32], label, None)?; + Ok(Self { stream, exporter }) + } +} + +impl Connection { + pub(super) fn from_stream(stream: S, exporter: [u8; 32]) -> Self { + Self { stream, exporter } + } + + pub(super) async fn raw(&mut self, bytes: &[u8]) -> Result<()> { + self.stream.write_all(bytes).await?; + self.stream.flush().await?; + Ok(()) + } + + pub(super) async fn frame( + &mut self, + magic: &[u8; 4], + purpose: &[u8], + binding: &[u8], + header: Value, + body: &[u8], + ) -> Result<[u8; 32]> { + let header = serde_json::to_vec(&header)?; + self.frame_bytes(magic, purpose, Some(binding), &header, body) + .await + } + + pub(super) async fn frame_bytes( + &mut self, + magic: &[u8; 4], + purpose: &[u8], + binding: Option<&[u8]>, + header: &[u8], + body: &[u8], + ) -> Result<[u8; 32]> { + let mut prefix = [0; 20]; + prefix[..4].copy_from_slice(magic); + prefix[4..8].copy_from_slice(&1u32.to_be_bytes()); + prefix[8..12].copy_from_slice(&(header.len() as u32).to_be_bytes()); + prefix[12..].copy_from_slice(&(body.len() as u64).to_be_bytes()); + let mut mac = Hmac::::new_from_slice(&[17; 32])?; + for part in std::iter::once(purpose) + .chain(std::iter::once(self.exporter.as_slice())) + .chain(binding) + .chain([prefix.as_slice(), header]) + { + mac.update(&(part.len() as u64).to_be_bytes()); + mac.update(part); + } + let tag: [u8; 32] = mac.finalize().into_bytes().into(); + self.stream.write_all(&prefix).await?; + self.stream.write_all(header).await?; + self.stream.write_all(&tag).await?; + self.stream.write_all(body).await?; + self.stream.flush().await?; + Ok(tag) + } + + /// Independently check the server's framing and exporter-bound MAC before + /// using a reply in a later probe. Bounds apply before allocating either body. + pub(super) async fn receive( + &mut self, + magic: &[u8; 4], + purpose: &[u8], + binding: &[u8], + ) -> Result<(Value, Vec, [u8; 32])> { + tokio::time::timeout(Duration::from_secs(5), async { + let (header, length, tag) = self.receive_head(magic, purpose, binding).await?; + let body = self.receive_body(length).await?; + Ok((header, body, tag)) + }) + .await? + } + + pub(super) async fn receive_head( + &mut self, + magic: &[u8; 4], + purpose: &[u8], + binding: &[u8], + ) -> Result<(Value, usize, [u8; 32])> { + tokio::time::timeout(Duration::from_secs(5), async { + let mut prefix = [0; 20]; + self.stream.read_exact(&mut prefix).await?; + assert_eq!(&prefix[..4], magic); + assert_eq!(&prefix[4..8], &1u32.to_be_bytes()); + let header_len = u32::from_be_bytes(prefix[8..12].try_into()?) as usize; + let body_len = u64::from_be_bytes(prefix[12..].try_into()?); + assert!(header_len <= 64 * 1024 && body_len <= 64 * 1024); + let mut header = vec![0; header_len]; + self.stream.read_exact(&mut header).await?; + let mut tag = [0; 32]; + self.stream.read_exact(&mut tag).await?; + let mut mac = Hmac::::new_from_slice(&[17; 32])?; + for part in [purpose, &self.exporter, binding, &prefix, &header] { + mac.update(&(part.len() as u64).to_be_bytes()); + mac.update(part); + } + mac.verify_slice(&tag)?; + Ok((serde_json::from_slice(&header)?, body_len as usize, tag)) + }) + .await? + } + + pub(super) async fn receive_body(&mut self, length: usize) -> Result> { + assert!(length <= 64 * 1024); + let mut body = vec![0; length]; + tokio::time::timeout(Duration::from_secs(5), self.stream.read_exact(&mut body)).await??; + Ok(body) + } + + pub(super) async fn refused(mut self) -> Result<()> { + let mut byte = [0]; + let result = + tokio::time::timeout(Duration::from_secs(5), self.stream.read(&mut byte)).await?; + assert!( + !matches!(result, Ok(n) if n > 0), + "refused stream must not acknowledge admission" + ); + Ok(()) + } +} + +pub(super) fn peer(peer: &RaftPeer) -> Value { + json!({"node":peer.replica.node.bytes(), "disk":peer.replica.disk.bytes(), "process":peer.replica.process.bytes(), "member":peer.replica.member.get().to_string(), "label":peer.label}) +} + +pub(super) async fn probe( + fixture: &mut RuntimeCluster, + history: GroupHistory, + snapshot: &raft::prelude::Snapshot, + pool: &CellGroupCheckpointPool, + other: GroupHistory, + registrations: &mut [CellGroupCheckpointRegistration], +) -> Result<()> { + let route = fixture.tcp_route(history)?; + let group = fixture.cluster.voters[2].log.cell_group(&history)?; + let before = group.group_manifest()?; + let mut message = Message { + from: 1, + to: 3, + term: snapshot.get_metadata().term, + ..Default::default() + }; + message.set_msg_type(MessageType::MsgSnapshot); + message.set_snapshot(snapshot.clone()); + let body = message.write_to_bytes()?; + let start = json!({"route": { + "scope": {"kind":"Group", "identity":[history.cluster.bytes(), history.namespace.bytes(), history.group.bytes(), history.incarnation.bytes()]}, + "schema":route.schema_digest, "source":peer(&route.peers[0]), "target":peer(&route.peers[2]) + }, "digest": <[u8; 32]>::from(Sha256::digest(&body))}); + let address = fixture.transports[2].data_address(); + for case in 0..8 { + let mut connection = Connection::open(address).await?; + let mut header = start.clone(); + match case { + 0 => { + connection.exporter[0] ^= 1; + } + 1 => { + header["route"]["source"]["process"] = json!(vec![244; 16]); + } + 2 => { + header["digest"] = json!(vec![0; 32]); + } + _ => (), + } + let binding = connection + .frame( + b"CLGS", + b"cellar-group-snapshot-start-1", + &[], + header, + &body, + ) + .await?; + if case >= 3 { + let sequence = if case == 3 { 1 } else { 0 }; + let binding = if case == 4 { [241; 32] } else { binding }; + // Case 5 sends an empty header part, case 6 a premature End. + let part = if case == 6 { + Value::Null + } else { + json!({"kind":"Header"}) + }; + // A nonempty Header with an invalid offset reaches the physical + // verifier. Keep the socket open without End: worker failure must + // close admission without waiting for another network packet. + let bytes: &[u8] = if case == 7 { &[0] } else { &[] }; + let offset = usize::from(case == 7); + connection.frame(b"CLGD", b"cellar-group-snapshot-data-1", &binding, + json!({"sequence":sequence, "part":part, "offset":offset, "length":bytes.len(), "digest":<[u8; 32]>::from(Sha256::digest(bytes))}), bytes).await?; + } + connection + .refused() + .await + .with_context(|| format!("checkpoint fault {case}"))?; + tokio::time::timeout(Duration::from_secs(5), async { + while pool.usage().sources != 0 || fixture.transports[2].usage().data_items != 0 { + tokio::task::yield_now().await; + } + }) + .await?; + assert_eq!( + group.group_manifest()?, + before, + "fault {case} changed selected root" + ); + assert_eq!(group.checkpoint_manifest()?, None); + } + // Authenticated start with a declared oversized data body refuses before + // allocation or waiting for that body. + let mut connection = Connection::open(address).await?; + connection + .frame( + b"CLGS", + b"cellar-group-snapshot-start-1", + &[], + start.clone(), + &body, + ) + .await?; + let mut oversized = [0; 20]; + oversized[..4].copy_from_slice(b"CLGD"); + oversized[4..8].copy_from_slice(&1u32.to_be_bytes()); + oversized[8..12].copy_from_slice(&1u32.to_be_bytes()); + oversized[12..].copy_from_slice(&65537u64.to_be_bytes()); + connection.stream.write_all(&oversized).await?; + connection.refused().await?; + tokio::time::timeout(Duration::from_secs(5), async { + while pool.usage().sources != 0 || fixture.transports[2].usage().data_items != 0 { + tokio::task::yield_now().await; + } + }) + .await?; + // A disconnected stream cannot use channel closure as successful End. + let mut connection = Connection::open(address).await?; + let binding = connection + .frame( + b"CLGS", + b"cellar-group-snapshot-start-1", + &[], + start.clone(), + &body, + ) + .await?; + tokio::time::timeout(Duration::from_secs(5), async { + while pool.usage().sources == 0 { + tokio::task::yield_now().await; + } + }) + .await?; + // The checkpoint owns receive admission while another group on the same + // runtimes/transport/engines elects, commits and applies a real cell write. + fixture.runtimes[0].campaign(other).await?; + fixture.settle(&[other]).await?; + let other_route = fixture.cluster.voters[0] + .log + .cell_group(&other)? + .group_manifest()? + .context("other root")? + .route() + .clone(); + let bytes = images::image( + None, + "CREATE TABLE progress(n); INSERT INTO progress VALUES(17);", + )?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let accepted = fixture.runtimes[0] + .propose( + other, + request( + &other_route, + super::history(&other_route, "unrelated")?, + 1, + &cut, + )?, + ) + .await?; + tokio::time::timeout(Duration::from_secs(10), async { + while fixture.applied[&other] + .iter() + .any(|index| *index < accepted.position.index) + { + fixture.pump(&[], &mut BTreeMap::new()).await?; + tokio::task::yield_now().await; + } + Ok::<_, anyhow::Error>(()) + }) + .await??; + assert_eq!(pool.usage().sources, 1); + // Revocation while a stream is admitted must refuse its authenticated End. + fixture.transports[2].remove_physical_group(&history)?; + connection.frame(b"CLGD", b"cellar-group-snapshot-data-1", &binding, + json!({"sequence":0, "part":null, "offset":0, "length":0, "digest":<[u8; 32]>::from(Sha256::digest([]))}), &[]).await?; + connection.refused().await?; + tokio::time::timeout(Duration::from_secs(5), async { + while pool.usage().sources != 0 || fixture.transports[2].usage().data_items != 0 { + tokio::task::yield_now().await; + } + }) + .await?; + fixture.transports[2].install_physical_group(route)?; + // Replacing the old guard after registering must not remove the new owner. + registrations[1] = fixture.transports[2] + .register_group_checkpoint(fixture.appliers[&(2, history)].1.clone())?; + let mut connection = Connection::open(address).await?; + connection + .frame(b"CLGS", b"cellar-group-snapshot-start-1", &[], start, &body) + .await?; + tokio::time::timeout(Duration::from_secs(5), async { + while pool.usage().sources == 0 { + tokio::task::yield_now().await; + } + }) + .await?; + drop(connection); + tokio::time::timeout(Duration::from_secs(5), async { + while pool.usage().sources != 0 || fixture.transports[2].usage().data_items != 0 { + tokio::task::yield_now().await; + } + }) + .await?; + assert_eq!(group.group_manifest()?, before); + assert_eq!(group.checkpoint_manifest()?, None); + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_node_reclaim.rs b/crates/cellar/tests/support/cell_group_node_reclaim.rs new file mode 100644 index 000000000..d7ccf9964 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_node_reclaim.rs @@ -0,0 +1,394 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Complete engine inventory over two groups sharing the same physical images. +use super::*; +use cellar_ltx::internal::{ExtentId, ExtentManifest, Geometry, ImageBinding, EXTENT_BYTES}; +use std::path::PathBuf; + +pub(super) fn limits() -> CellGroupReclaimLimits { + CellGroupReclaimLimits { + max_selected_cells: 64, + max_files: 1, + max_page_scan: 64, + } +} + +fn appliers(f: &RuntimeCluster) -> Vec { + f.appliers + .iter() + .filter(|((node, _), _)| *node == 0) + .map(|(_, (_, a))| a.clone()) + .collect() +} + +async fn images_for( + f: &RuntimeCluster, + cell: &CellHistory, + node: usize, +) -> Result { + Ok(match f.cell_options_at(cell, node).await? { + CellSqlOptions::Materialized(o) => CellGroupApplyImages::Materialized { + images: o.images, + services: o.query_services, + }, + CellSqlOptions::Extents(o) => CellGroupApplyImages::Extents { + images: o.images, + services: o.query_services, + limits: Default::default(), + cache_directory: o.cache_directory, + cache_pool: o.cache_pool, + cache: o.cache, + }, + }) +} + +async fn enroll(f: &RuntimeCluster, group: GroupHistory, name: &str, rooted: bool) -> Result<()> { + let selected = f.cluster.voters[0] + .log + .cell_group(&group)? + .group_manifest()? + .context("group")?; + let cell = history(selected.route(), name)?; + let bytes = images::image( + None, + "CREATE TABLE value(n); INSERT INTO value VALUES(113);", + )?; + let accepted = f.runtimes[0] + .propose( + group, + apply::enrollment_bytes(selected.route(), &cell, &bytes, rooted)?, + ) + .await?; + controller::applied(f, group, accepted.position.index).await +} + +fn physical_image( + images: &CellGroupApplyImages, + base: PathBuf, + cell: CellHistory, + number: u8, +) -> Result<(CellImagePin, PathBuf)> { + let bytes = images::image( + None, + &format!("CREATE TABLE value(n); INSERT INTO value VALUES({number});"), + )?; + let state = images::Cut::new(&[], &bytes, 0, true, false).result; + match images { + CellGroupApplyImages::Materialized { images, .. } => { + let source = base.join(format!("private-{number}.sqlite")); + std::fs::write(&source, &bytes)?; + let id = ImageId::new([number; 16])?; + let pin = images.prepare_image_file(id, &source, state)?; + images.retire(id)?; + Ok(( + CellImagePin::Materialized(pin), + base.join("node-1/images") + .join(format!("{}.sqlite", hex::encode(id.bytes()))), + )) + } + CellGroupApplyImages::Extents { images, .. } => { + let ids = bytes + .chunks(EXTENT_BYTES as usize) + .map(ExtentId::of) + .collect::, _>>()?; + let manifest = ExtentManifest::new( + ImageBinding { + history: cell_image_history_digest(&cell), + schema: [81; 32], + epoch: state.epoch, + txid: state.txid.0, + checksum: state.checksum, + }, + Geometry::new(state.page_size, state.length)?, + &ids, + )?; + let view = images.stage_transient_root(manifest.clone())?; + for (id, chunk) in ids.iter().zip(bytes.chunks(EXTENT_BYTES as usize)) { + images.put_extent_bytes(*id, chunk)?; + } + let pin = images.seal_image_root(manifest, &mut vec![0; EXTENT_BYTES as usize])?; + drop(view); + let id = ids[0]; + Ok(( + CellImagePin::Extents(Arc::new(pin)), + base.join("rooted-0").join(format!( + "e-{:08x}-{}", + id.length(), + hex::encode(id.bytes()) + )), + )) + } + } +} + +fn read(pin: &CellImagePin) -> Result<[u8; 32]> { + let mut hash = Sha256::new(); + match pin { + CellImagePin::Materialized(pin) => { + let mut r = pin.open_reader()?; + let mut offset = 0; + while offset < r.length() { + let count = (r.length() - offset).min(65536) as usize; + hash.update(r.read_exact_at(offset, count)?); + offset += count as u64; + } + } + CellImagePin::Extents(pin) => { + for n in 0..pin.manifest().geometry().extent_count() { + hash.update( + pin.view() + .read_extent(pin.manifest().extent(n).context("extent")?)?, + ); + } + } + CellImagePin::Empty => (), + } + Ok(hash.finalize().into()) +} + +pub(crate) async fn shared_inventory(rooted: bool) -> Result<()> { + let class = RaftClass::new("Counter", 1, [81; 32])?; + let (mut f, route, a, _) = Box::pin(registered::setup(rooted, &class)).await?; + let second = f.add_group(31).await?; + let selected_images = images_for(&f, &a, 0).await?; + for node in 0..3 { + let images = images_for(&f, &a, node).await?; + let pool = f.appliers[&(node, route.history)].0.clone(); + let voter = &mut f.extra.get_mut(&second).context("second")?[node]; + let applier = CellGroupApplier::new( + voter.log.cell_group(&second)?, + voter.pages.take().context("page store")?, + images, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + )?; + f.appliers.insert((node, second), (pool, applier)); + } + let mut controllers = controller::start_with_limits(&mut f, controller::limits()).await?; + f.runtimes[0].campaign(second).await?; + controller::wait_for(async || Ok(f.runtimes[0].status(second).await?.ready)).await?; + enroll(&f, second, "other-group", rooted).await?; + let mut selected = Vec::new(); + for group in [route.history, second] { + let applier = f.appliers[&(0, group)].1.clone(); + let log = f.cluster.voters[0].log.cell_group(&group)?; + let (root, checkpoint) = f + .cluster + .domain + .blocking(move || -> Result<_> { + let root = applier.recover_selected()?; + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let checkpoint = pool.capture(&log, &root)?.prepare_publication()?; + Ok((root, checkpoint)) + })? + .await??; + let marker = root.marker().clone(); + f.runtimes[0].publish_checkpoint(group, checkpoint).await?; + controller::wait_for(async || { + Ok(f.cluster.voters[0] + .log + .cell_group(&group)? + .checkpoint_manifest()? + .as_ref() + == Some(&marker)) + }) + .await?; + selected.push(root.marker().clone()); + drop(root); + } + enroll(&f, route.history, "after-checkpoint", rooted).await?; + enroll(&f, second, "other-after-checkpoint", rooted).await?; + let images = selected_images.clone(); + let base = f.cluster._root.path().to_path_buf(); + let cell = history(&route, "private-reader")?; + let (reader, reader_path, orphan_path) = f + .cluster + .domain + .blocking(move || -> Result<_> { + let (reader, path) = physical_image(&images, base.clone(), cell.clone(), 211)?; + let (orphan, orphan_path) = physical_image(&images, base, cell, 212)?; + drop(orphan); + Ok((reader, path, orphan_path)) + })? + .await??; + let expected = read(&reader)?; + let selected_appliers = appliers(&f); + let log = &f.cluster.voters[0].log; + assert!(matches!( + log.reclaim_cell_storage(selected_appliers.clone(), vec![], limits()) + .await, + Err(CellGroupReclaimError::Busy) + )); + controllers.remove(0).close().await?; + f.appliers.retain(|(node, _), _| *node != 0); + assert!(matches!( + log.reclaim_cell_storage(vec![selected_appliers.clone()[0].clone()], vec![], limits()) + .await, + Err(CellGroupReclaimError::Invalid(_)) + )); + assert!(orphan_path.exists()); + assert!(matches!( + log.reclaim_cell_storage( + vec![selected_appliers.clone()[0].clone(); 2], + vec![], + limits() + ) + .await, + Err(CellGroupReclaimError::Invalid(_)) + )); + let foreign = f.appliers[&(1, route.history)].1.clone(); + assert!(matches!( + log.reclaim_cell_storage( + vec![foreign, selected_appliers.clone()[1].clone()], + vec![], + limits() + ) + .await, + Err(CellGroupReclaimError::Invalid(_)) + )); + let mut small = limits(); + small.max_selected_cells = 1; + assert!(matches!( + log.reclaim_cell_storage(selected_appliers.clone(), vec![], small) + .await, + Err(CellGroupReclaimError::Limit) + )); + assert!(orphan_path.exists()); + let mut deleted = 0; + for _ in 0..32 { + let report = log + .reclaim_cell_storage(selected_appliers.clone(), vec![], limits()) + .await?; + assert_eq!(report.groups, 2); + assert_eq!(report.selected_roots, 4); + assert!(report.page_files + report.image_files <= 1); + assert!(report.pages_scanned <= 64); + deleted += report.image_files; + assert_eq!(read(&reader)?, expected); + if !orphan_path.exists() { + break; + } + } + assert!(deleted > 0 && !orphan_path.exists()); + assert!(reader_path.exists()); + for marker in &selected { + let applier = selected_appliers + .iter() + .find(|a| a.history() == marker.route().history) + .context("group applier")? + .clone(); + let current = f + .cluster + .domain + .blocking(move || applier.recover_selected())? + .await??; + for name in if marker.route().history == route.history { + vec!["registered-a", "registered-b"] + } else { + vec!["other-group"] + } { + let (_, pin) = current + .pin_cell(&CellId::new(name)?) + .context("retained cell")?; + read(&pin)?; + } + } + drop(reader); + for _ in 0..32 { + log.reclaim_cell_storage(selected_appliers.clone(), vec![], limits()) + .await?; + if !reader_path.exists() { + break; + } + } + assert!(!reader_path.exists()); + for (index, group) in [route.history, second].into_iter().enumerate() { + assert_eq!( + log.cell_group(&group)?.checkpoint_manifest()?.as_ref(), + Some(&selected[index]) + ); + let applier = selected_appliers + .iter() + .find(|a| a.history() == group) + .context("group applier")? + .clone(); + let current = f + .cluster + .domain + .blocking(move || applier.recover_selected())? + .await??; + assert!(current.marker().position().index > selected[index].position().index); + } + // Queue maintenance behind a real engine fence, then lose its waiter. + // Neither cancellation nor a cancelled close may refund physical ownership. + let image_store = selected_images.clone(); + let base = f.cluster._root.path().to_path_buf(); + let cell = history(&route, "cancelled-orphan")?; + let orphan = f + .cluster + .domain + .blocking(move || -> Result<_> { + let (pin, path) = physical_image(&image_store, base, cell, 213)?; + drop(pin); + Ok(path) + })? + .await??; + let gate = f.cluster.voters[0].gate.clone(); + let release = gate.arm(fault::HOLD_SYNC); + let writer = (*log).clone(); + let ahead = f + .cluster + .domain + .spawn(async move { writer.reserve_node_revision().await })?; + tokio::time::timeout(Duration::from_secs(5), gate.entered.notified()).await?; + let writer = (*log).clone(); + let owned = selected_appliers.clone(); + let pending = f.cluster.domain.spawn(async move { + writer + .reclaim_cell_storage( + owned, + vec![], + CellGroupReclaimLimits { + max_files: 4096, + max_page_scan: 4096, + ..limits() + }, + ) + .await + })?; + controller::wait_for(async || { + let sample = log.performance_snapshot(); + assert!(sample.available && !sample.saturated); + let queue = sample + .operations + .iter() + .find(|o| o.name == "engine_queue") + .context("engine queue")?; + Ok(queue.started - queue.succeeded - queue.failed - queue.abandoned == 1) + }) + .await?; + pending.abort(); + assert!(pending.await.is_err()); + assert!(matches!( + CellGroupRuntime::start((*log).clone(), RaftRuntimeOptions::default()).await, + Err(RuntimeError::Storage(LogError::AlreadyExists)) + )); + assert!(apply::pool(&f, 0).is_err()); + assert!(tokio::time::timeout(Duration::from_millis(30), log.close()) + .await + .is_err()); + assert!(orphan.exists()); + drop(release); + ahead.await??; + tokio::time::timeout(Duration::from_secs(5), log.close()).await??; + assert!( + !orphan.exists(), + "queued collection must finish after waiter cancellation" + ); + for c in controllers { + c.close().await?; + } + drop((selected, selected_appliers)); + f.close().await +} diff --git a/crates/cellar/tests/support/cell_group_outbox_effects.rs b/crates/cellar/tests/support/cell_group_outbox_effects.rs new file mode 100644 index 000000000..735ad6c7f --- /dev/null +++ b/crates/cellar/tests/support/cell_group_outbox_effects.rs @@ -0,0 +1,663 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Real receiver commands and signatures; only the source-side reply is lost. +use super::*; +use cellar::application::consensus::{ + RaftDeliveryRequest, RaftEffectOptions, RaftEffectSigningKey, RaftEffectSink, + RaftEffectSourceGrant, RaftEffectTarget, RaftInbox, RaftOutbox, RaftOutboxLimits, +}; +use futures_util::future::BoxFuture; +use std::future::Future; +use std::sync::{ + atomic::{AtomicBool, AtomicUsize, Ordering}, + Mutex, Weak, +}; + +#[derive(Clone, Copy, PartialEq, Eq)] +enum Mode { + LostReply, + LeaderChange, + Shutdown, +} + +struct State { + mode: Mode, + arrivals: AtomicUsize, + first: Mutex)>>, + first_proof: Mutex>>, + changed: AtomicBool, + entered: tokio::sync::Notify, + release: tokio::sync::Notify, + dropped: AtomicBool, + polls: AtomicUsize, +} + +struct Endpoint { + controllers: Vec, + dispatcher: CellGroupDispatcher, + options: Vec, + leader: AtomicUsize, + runtimes: Vec, + group: GroupHistory, + source: CellHistory, +} + +struct Routing { + endpoint: Mutex>, + state: Arc, +} + +struct PendingDelivery(Arc); + +impl Drop for PendingDelivery { + fn drop(&mut self) { + self.0.dropped.store(true, Ordering::SeqCst); + } +} + +impl RaftEffectSink for Routing { + fn deliver( + &self, + request: RaftDeliveryRequest, + ) -> BoxFuture<'static, Result, CommandFailure>> { + let endpoint = self.endpoint.lock().unwrap().upgrade(); + let state = self.state.clone(); + Box::pin(async move { + let endpoint = endpoint.ok_or(CommandFailure::Unavailable)?; + let attempt = state.arrivals.fetch_add(1, Ordering::SeqCst); + { + let mut first = state.first.lock().unwrap(); + if let Some((id, bytes)) = first.as_ref() { + if *id != request.command_id() || bytes != request.encoded() { + state.changed.store(true, Ordering::SeqCst); + return Err(CommandFailure::CommandConflict); + } + } else { + *first = Some((request.command_id(), request.encoded().to_vec())); + } + } + let node = endpoint.leader.load(Ordering::SeqCst); + if request.target().durability() == Durability::S3 { + let selected = endpoint.runtimes[node] + .reconciliation_authority(endpoint.group, endpoint.source.clone()) + .await + .map_err(|_| CommandFailure::Unavailable)?; + let published = endpoint.controllers[node] + .archive_status(endpoint.group) + .map_err(|_| CommandFailure::Unavailable)? + .and_then(|status| status.published) + .ok_or(CommandFailure::Unavailable)?; + assert_eq!(selected.cell().effects.pending_outbox, 1); + assert!( + published.position().index >= selected.cell().last_mutation.index, + "source claim must have published S3 coverage before send" + ); + } + let dispatcher = endpoint + .dispatcher + .clone() + .with_s3(&endpoint.controllers[node]) + .map_err(|_| CommandFailure::Unavailable)?; + let invocation = dispatcher + .command( + request.target().scope().clone(), + CellCommandRequest { + id: request.command_id(), + contract: request.target().contract().clone(), + input: request.encoded().to_vec(), + durability: request.target().durability(), + retry_until_ms: cellar::asyncrt::wall_ms().max(0) as u64 + 60_000, + }, + CellRoutingLimits { + timeout: Duration::from_secs(5), + max_attempts: 4, + }, + None, + ) + .await + .map_err(|_| CommandFailure::Unavailable)?; + let receipt = endpoint.controllers[node] + .command(invocation, endpoint.options[node].clone()) + .map_err(|_| CommandFailure::Unavailable)? + .response() + .await + .map_err(|_| CommandFailure::Unavailable)?; + assert_eq!(receipt.durability(), request.target().durability()); + let proof = receipt + .local_receipt() + .expect("local command receipt") + .receipt + .result() + .to_vec(); + // The adapter has consumed the typed receiver result into bounded + // reply bytes. A simulated lost network reply must not retain the + // receiver's SQL operation slot while another cell runs. + drop(receipt); + request + .verify_receipt(&proof) + .map_err(|_| CommandFailure::CommandConflict)?; + if attempt == 0 { + *state.first_proof.lock().unwrap() = Some(proof.clone()); + state.entered.notify_one(); + if state.mode == Mode::LostReply { + return Err(CommandFailure::Unavailable); + } + let _pending = PendingDelivery(state.clone()); + let released = state.release.notified(); + tokio::pin!(released); + std::future::poll_fn(|cx| { + state.polls.fetch_add(1, Ordering::SeqCst); + released.as_mut().poll(cx) + }) + .await; + } + Ok(proof) + }) + } +} + +pub(crate) async fn lost_reply() -> Result<()> { + run(Mode::LostReply).await +} + +pub(crate) async fn leader_change() -> Result<()> { + run(Mode::LeaderChange).await +} + +pub(crate) async fn shutdown() -> Result<()> { + run(Mode::Shutdown).await +} + +#[derive(Debug, PartialEq, Eq)] +struct DeliveryObservation { + command: CommandId, + bytes: Vec, + proof: Vec, +} + +pub(crate) async fn identity_across_bindings() -> Result<()> { + // Independently boot both physical bindings through real Raft and SQL. + // Each loses one receiver reply, then proves a single receiver mutation. + // A live transfer's seal/publication protocol belongs to CG-09. + for rooted in [false, true] { + let before = Box::pin(run_image( + Mode::LostReply, + rooted, + Arc::new(object_store::memory::InMemory::new()), + Durability::Quorum, + GroupId::new([3; 16])?, + )) + .await?; + let after = Box::pin(run_image( + Mode::LostReply, + rooted, + Arc::new(object_store::memory::InMemory::new()), + Durability::Quorum, + GroupId::new([31; 16])?, + )) + .await?; + assert_eq!( + before, after, + "group binding changed logical delivery identity or proof" + ); + } + Ok(()) +} + +async fn run(mode: Mode) -> Result<()> { + for rooted in [false, true] { + Box::pin(run_image( + mode, + rooted, + Arc::new(object_store::memory::InMemory::new()), + Durability::Quorum, + GroupId::new([3; 16])?, + )) + .await?; + } + Ok(()) +} + +pub(crate) async fn s3_lost_reply( + rooted: bool, + store: Arc, +) -> Result<()> { + Box::pin(run_image( + Mode::LostReply, + rooted, + store, + Durability::S3, + GroupId::new([3; 16])?, + )) + .await?; + Ok(()) +} + +async fn run_image( + mode: Mode, + rooted: bool, + store: Arc, + minimum: Durability, + group: GroupId, +) -> Result { + let state = Arc::new(State { + mode, + arrivals: AtomicUsize::new(0), + first: Mutex::new(None), + first_proof: Mutex::new(None), + changed: AtomicBool::new(false), + entered: tokio::sync::Notify::new(), + release: tokio::sync::Notify::new(), + dropped: AtomicBool::new(false), + polls: AtomicUsize::new(0), + }); + let routing = Arc::new(Routing { + endpoint: Mutex::new(Weak::new()), + state: state.clone(), + }); + let sender = Arc::new(RaftEffectSigningKey::new([211; 32])?); + let receiver = Arc::new(RaftEffectSigningKey::new([212; 32])?); + let received = Arc::new(AtomicUsize::new(0)); + let count = received.clone(); + let base = RaftClass::new("Counter", 1, [213; 32])?; + let definition = base.definition_digest(); + let target_cell = CellHistory { + cluster: ClusterId::new([1; 16])?, + namespace: NamespaceId::new([2; 16])?, + cell: CellId::new("Counter:registered-b")?, + incarnation: IncarnationId::new([8; 16])?, + }; + let receive = OperationContract { + operation: "Receive".into(), + kind: OperationKind::Command, + digest: [214; 32], + }; + let target = RaftEffectTarget::new( + target_cell.clone(), + definition, + receive.clone(), + receiver.verification_key(), + minimum, + )?; + let inbox = RaftInbox::new( + vec![RaftEffectSourceGrant::new( + target_cell.cluster, + target_cell.namespace, + "Counter", + "Events", + [215; 32], + sender.verification_key(), + )?], + receiver, + minimum, + 4, + )?; + let outbox = RaftOutbox::new( + sender, + RaftOutboxLimits { + retry_delay_ms: 100, + ..Default::default() + }, + )? + .binding("Events", [215; 32], target, routing.clone())?; + let class = Arc::new( + base.inbox("Receive", receive.digest, inbox, move |context, input| { + count.fetch_add(1, Ordering::SeqCst); + let amount = i64::from_be_bytes(input.try_into()?); + context.sql().execute("UPDATE value SET n=n+?1", [amount])?; + Ok(amount.to_be_bytes().to_vec()) + })? + .outbox(outbox)? + .command("Emit", [216; 32], |context, input| { + Ok(context.emit("Events", input)?.to_be_bytes().to_vec()) + })? + .command("Touch", [217; 32], |context, _| { + context.sql().execute("UPDATE value SET n=n+100", [])?; + Ok(Vec::new()) + })?, + ); + assert_eq!(class.definition_digest(), definition); + let filesystem = Arc::new(DirectFileSystem); + let cluster = Cluster::with_store_birth( + RaftRuntimeOptions::default(), + tempfile::tempdir()?, + filesystem.clone(), + class.schema_digest(), + store, + ClusterBirthOptions { + image_storage: if rooted { + RaftImageStorage::Extents + } else { + RaftImageStorage::Materialized + }, + group, + minimum, + log: ClusterLogLimits::default(), + }, + ) + .await?; + let (mut fixture, route, source, destination) = Box::pin(registered::setup_cluster_named( + rooted, + cluster, + filesystem, + ["Counter:registered-a", "Counter:registered-b"], + )) + .await?; + assert_eq!(destination, target_cell); + let keys = VersionKeys::new( + VersionSigningKey::new([218; 32], u64::MAX)?, + vec![], + Duration::from_secs(30), + )?; + let dispatcher = CellGroupDispatcher::new( + &fixture.cluster.domain, + fixture.cluster.directory.clone(), + class.clone(), + keys, + CellDispatchLimits { + max_cached_routes: 32, + route_cache_bytes: 256 * 1024, + max_invocations: 8, + retained_bytes: 1024 * 1024, + max_input_bytes: 64 * 1024, + }, + )?; + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let archive = if minimum == Durability::S3 { + let (archive, _) = + archive_controller::registration_minimum(&fixture, &route, rooted, minimum).await?; + controllers[0].archive(archive.clone())?.response().await?; + archive_controller::published( + &controllers[0], + route.history, + fixture.runtimes[0] + .status(route.history) + .await? + .applied + .index, + ) + .await?; + Some(archive) + } else { + None + }; + let dispatcher = if archive.is_some() { + dispatcher.with_s3(&controllers[0])? + } else { + dispatcher + }; + let mut destination_options = Vec::new(); + for (node, controller) in controllers.iter().enumerate() { + let mut resources = BTreeMap::new(); + for cell in [&source, &destination] { + resources.insert(cell.clone(), fixture.cell_options_at(cell, node).await?); + } + destination_options.push(resources[&destination].clone()); + controller.configure_effects(vec![CellGroupEffectRegistration::new( + dispatcher.clone(), + RaftEffectOptions::default(), + move |cell| { + resources + .get(cell) + .cloned() + .ok_or(ExecutionError::Invalid("effect resources")) + }, + )?])?; + } + let endpoint = Arc::new(Endpoint { + controllers: controllers.clone(), + dispatcher: dispatcher.clone(), + options: destination_options, + leader: AtomicUsize::new(0), + runtimes: fixture.runtimes.clone(), + group: route.history, + source: source.clone(), + }); + *routing.endpoint.lock().unwrap() = Arc::downgrade(&endpoint); + let retry_until = fixture + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms().max(0) as u64 + 60_000 }) + .await?; + let invocation = dispatcher + .command( + source.clone(), + CellCommandRequest { + id: CommandId::new(ClientId::new([219; 16])?, 1)?, + contract: class.contract("Emit").context("Emit")?, + input: 7_i64.to_be_bytes().to_vec(), + durability: minimum, + retry_until_ms: retry_until, + }, + CellRoutingLimits { + timeout: Duration::from_secs(5), + max_attempts: 4, + }, + None, + ) + .await?; + let receipt = controllers[0] + .command(invocation, fixture.cell_options(&source).await?)? + .response() + .await?; + assert_eq!( + receipt + .local_receipt() + .expect("local command receipt") + .receipt + .result(), + 1_u64.to_be_bytes() + ); + assert_eq!(receipt.durability(), minimum); + drop(receipt); + tokio::time::timeout(Duration::from_secs(5), state.entered.notified()).await?; + if mode == Mode::LeaderChange { + let previous_polls = state.polls.load(Ordering::SeqCst); + let touch = dispatcher + .command( + source.clone(), + CellCommandRequest { + id: CommandId::new(ClientId::new([220; 16])?, 1)?, + contract: class.contract("Touch").context("Touch")?, + input: Vec::new(), + durability: minimum, + retry_until_ms: retry_until, + }, + CellRoutingLimits { + timeout: Duration::from_secs(5), + max_attempts: 4, + }, + None, + ) + .await?; + drop( + controllers[0] + .command(touch, fixture.cell_options(&source).await?)? + .response() + .await?, + ); + controller::wait_for(async || { + Ok(state.polls.load(Ordering::SeqCst) >= previous_polls + 2 + || state.dropped.load(Ordering::SeqCst)) + }) + .await?; + assert!( + !state.dropped.load(Ordering::SeqCst), + "a later cell mutation must not revoke a committed dispatch" + ); + let old_term = fixture.runtimes[0].status(route.history).await?.term; + endpoint.leader.store(1, Ordering::SeqCst); + let mut transferred = false; + controller::wait_for(async || { + // The healthy leader's lease suppresses ordinary pre-votes. + // Use the actual TimeoutNow handoff, including its caught-up + // target check, within this same progress deadline. + if !transferred { + match fixture.runtimes[0].transfer_leader(route.history, 2).await { + Ok(()) => transferred = true, + Err(RuntimeError::NotReady) => return Ok(false), + Err(error) => return Err(error.into()), + } + } + let status = fixture.runtimes[1].status(route.history).await?; + Ok(status.ready && status.term > old_term && state.dropped.load(Ordering::SeqCst)) + }) + .await?; + } else if mode == Mode::Shutdown { + controllers[0].close().await?; + assert!(state.dropped.load(Ordering::SeqCst)); + assert_eq!(received.load(Ordering::SeqCst), 1); + controller::wait_for(async || { + for runtime in fixture.runtimes.iter().skip(1) { + let source_view = runtime + .reconciliation_authority(route.history, source.clone()) + .await?; + let destination_view = runtime + .reconciliation_authority(route.history, destination.clone()) + .await?; + if source_view.cell().version.sequence.get() != 3 + || destination_view.cell().version.sequence.get() != 2 + { + return Ok(false); + } + } + Ok(true) + }) + .await?; + } + if mode != Mode::Shutdown { + controller::wait_for(async || { + for runtime in &fixture.runtimes { + let selected = runtime + .reconciliation_authority(route.history, source.clone()) + .await?; + if selected.cell().effects.pending_outbox != 0 { + return Ok(false); + } + } + for controller in &controllers { + let usage = controller.usage(); + if usage.active_effects != 0 || usage.retained_commands != 0 { + return Ok(false); + } + } + for (pool, _) in fixture.appliers.values() { + let usage = pool.usage()?; + if usage.active_cells != 0 || usage.retained_operations != 0 { + return Ok(false); + } + } + Ok(true) + }) + .await?; + assert_eq!(state.arrivals.load(Ordering::SeqCst), 2); + } + assert!(!state.changed.load(Ordering::SeqCst)); + assert_eq!(received.load(Ordering::SeqCst), 1); + for (node, runtime) in fixture + .runtimes + .iter() + .enumerate() + .skip(usize::from(mode == Mode::Shutdown)) + { + let source_view = runtime + .reconciliation_authority(route.history, source.clone()) + .await?; + assert_eq!( + source_view.cell().version.sequence.get(), + match mode { + Mode::LostReply => 5, + Mode::LeaderChange => 6, + Mode::Shutdown => 3, + } + ); + assert_eq!( + source_view.cell().effects.pending_outbox, + u64::from(mode == Mode::Shutdown) + ); + let deadline = fixture + .cluster + .domain + .run(async { cellar::asyncrt::mono_ms() + 5000 }) + .await?; + let authority = runtime + .read_authority( + route.history, + destination.clone(), + CellReadConsistency::Stale, + deadline, + ) + .await?; + let result = fixture.appliers[&(node, route.history)] + .0 + .query( + fixture.cell_options_at(&destination, node).await?, + authority, + |database, version| { + assert_eq!(version.sequence.get(), 2); + let value: i64 = + database + .connection() + .query_row("SELECT n FROM value", [], |row| row.get(0))?; + Ok(value.to_be_bytes().to_vec()) + }, + ) + .await?; + assert_eq!(result.result(), 7_i64.to_be_bytes()); + } + if let Some(archive) = &archive { + let current = fixture.runtimes[0] + .status(route.history) + .await? + .applied + .index; + archive_controller::published(&controllers[0], route.history, current).await?; + let coverage = archive_controller::verified_current( + &fixture, + archive, + rooted, + "effect-recovery-audit", + ) + .await?; + assert_eq!(coverage.recovery().manifest().pinned_images(), 2); + for cell in [&source, &destination] { + let selected = fixture.runtimes[0] + .reconciliation_authority(route.history, cell.clone()) + .await?; + let (restored, _) = coverage + .recovery() + .manifest() + .pin_cell(&cell.cell) + .context("S3 effect cell")?; + assert_eq!(restored, selected.cell()); + } + } + for controller in &controllers { + controller.close().await?; + assert_eq!(controller.usage().active_effects, 0); + } + for (pool, _) in fixture.appliers.values() { + assert_eq!(pool.usage()?.active_cells, 0); + assert_eq!(pool.usage()?.retained_bytes, 0); + } + drop(endpoint); + drop(controllers); + fixture.close().await?; + let (command, bytes) = state + .first + .lock() + .unwrap() + .take() + .context("first delivery")?; + let proof = state + .first_proof + .lock() + .unwrap() + .take() + .context("receiver proof")?; + Ok(DeliveryObservation { + command, + bytes, + proof, + }) +} diff --git a/crates/cellar/tests/support/cell_group_provision.rs b/crates/cellar/tests/support/cell_group_provision.rs new file mode 100644 index 000000000..6b9c43fa6 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_provision.rs @@ -0,0 +1,720 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Explicit assigned-range provisioning over real engines, files and TLS. +use super::apply_faults::physical; +use super::registration::catalog_fault; +use super::remote_queries::{keys, wait_for}; +use super::*; +use std::sync::atomic::{AtomicBool, Ordering}; + +async fn assignment( + f: &RuntimeCluster, + number: u8, +) -> Result<(RangeDirectory, GroupRoute, Arc)> { + let store = Arc::new(catalog_fault::Store::default()); + let old = f.cluster.birth.route(); + let scope = GroupHistory { + cluster: old.history.cluster, + namespace: NamespaceId::new([number; 16])?, + group: GroupId::new([number; 16])?, + incarnation: IncarnationId::new([number; 16])?, + }; + let catalog = Catalog::new(store.clone(), CatalogOptions::default())?; + let mut namespace_config = NamespaceConfig::new( + scope.cluster, + scope.namespace, + cellar_logic::NamespaceMode::Raft, + old.schema_digest, + ); + namespace_config.image_storage = if f.cluster.voters[0].extents.is_some() { + RaftImageStorage::Extents + } else { + RaftImageStorage::Materialized + }; + let namespace = f + .cluster + .domain + .run(catalog.open_namespace(namespace_config)) + .await??; + let directory = RangeDirectory::new(namespace, DirectoryLimits::default())?; + let birth = directory.prepare_group_birth(CellGroupBirthSpec { + history: scope, + range: CellRange::all(), + epoch: RouteEpoch::new(1)?, + creation: ProposalId::new([number; 16])?, + voters: (1..=3) + .map(|n| InitialMember { + member: MemberId::new(n).unwrap(), + node: NodeId::new([n as u8; 16]).unwrap(), + }) + .collect(), + learners: vec![], + })?; + f.cluster + .domain + .run(directory.reserve_group_birth(&birth)) + .await??; + let initialize = directory + .prepare_initialization(ProposalId::new([number + 1; 16])?, &[birth.route().clone()])?; + f.cluster + .domain + .run(directory.publish(&initialize)) + .await??; + // No materialized birth, bootstrap permit, page store or selected root has + // been prepared by the fixture. Production provisioning owns all of them. + for voter in &f.cluster.voters { + assert!(matches!( + voter.log.cell_group(&scope), + Err(LogError::NotFound) + )); + } + Ok((directory, birth.route().clone(), store)) +} + +fn class() -> Result> { + Ok(Arc::new( + RaftClass::new("Counter", 1, [81; 32])? + .initializer([83; 32], |_, tx| { + tx.execute_batch("CREATE TABLE value(n); INSERT INTO value VALUES(7);")?; + Ok(()) + })? + .command("Add", [82; 32], |ctx, input| { + ctx.sql().execute("UPDATE value SET n=n+1", [])?; + Ok(input.to_vec()) + })?, + )) +} + +async fn settings( + f: &RuntimeCluster, + a: &CellHistory, + node: usize, + directories: &[RangeDirectory], +) -> Result { + let images = match f.cell_options_at(a, node).await? { + CellSqlOptions::Materialized(o) => CellGroupApplyImages::Materialized { + images: o.images, + services: o.query_services, + }, + CellSqlOptions::Extents(o) => CellGroupApplyImages::Extents { + images: o.images, + services: o.query_services, + limits: Default::default(), + cache_directory: o.cache_directory, + cache_pool: o.cache_pool, + cache: o.cache, + }, + }; + let peers = f + .tcp_route(f.cluster.birth.route().history)? + .peers + .iter() + .map(RaftPeerNode::from) + .collect(); + Ok(CellGroupProvisioning { + directory: f.cluster._root.path().join(format!("provision-{node}")), + filesystem: Arc::new(DirectFileSystem), + namespaces: directories + .iter() + .cloned() + .map(|directory| CellGroupProvisionNamespace { + directory, + images: images.clone(), + }) + .collect(), + manifest: ManifestLimits { + max_cells: 32, + max_depth: 16, + }, + entries: GroupEntryLimits::default(), + max_effect_rows: 100, + max_pages: 1024, + max_page_bytes: 16 * 1024 * 1024, + max_retained_roots: 16, + peers, + peer_revision: 1, + max_preparations: 1, + max_preparation_bytes: 512 * 1024 * 1024, + max_parallel_preparations: 1, + max_waiters: 2, + }) +} + +fn pages_path(base: &std::path::Path, h: GroupHistory) -> std::path::PathBuf { + // Independent expectation for the canonical startup layout. The restart + // drill below must reopen these selected pages, not a second empty store. + base.join("groups") + .join(hex::encode(h.cluster.bytes())) + .join(hex::encode(h.namespace.bytes())) + .join(hex::encode(h.group.bytes())) + .join(hex::encode(h.incarnation.bytes())) + .join("pages") +} + +pub(crate) async fn provisioning_coalesces_and_serves() -> Result<()> { + for rooted in [false, true] { + let class = class()?; + let (mut f, _, a, _) = Box::pin(registered::setup(rooted, &class)).await?; + let mut limits = controller::limits(); + limits.max_commands = 1; + let controllers = controller::start_with_limits(&mut f, limits).await?; + let (directory, route, store) = assignment(&f, 31).await?; + let (foreign, _, _) = assignment(&f, 31).await?; + let (other, other_route, _) = assignment(&f, 41).await?; + let mut invalid = settings(&f, &a, 0, &[directory.clone(), other.clone()]).await?; + let directory_before = invalid.directory.clone(); + match &mut invalid.namespaces[0].images { + CellGroupApplyImages::Extents { + cache_directory, .. + } => *cache_directory = "relative-cache".into(), + CellGroupApplyImages::Materialized { .. } => { + invalid.directory = "relative-pages".into() + } + } + assert!(matches!( + controllers[0].configure_provisioning(invalid), + Err(CellGroupControllerError::Invalid(_)) + )); + assert!( + !directory_before.exists(), + "invalid configuration performs no filesystem work" + ); + let duplicate = settings(&f, &a, 0, &[directory.clone(), directory.clone()]).await?; + assert!(matches!( + controllers[0].configure_provisioning(duplicate), + Err(CellGroupControllerError::Invalid(_)) + )); + let mut too_small = settings(&f, &a, 0, &[directory.clone(), other.clone()]).await?; + too_small.max_preparation_bytes = 1; + assert!(matches!( + controllers[0].configure_provisioning(too_small), + Err(CellGroupControllerError::Invalid(_)) + )); + assert!( + !directory_before.exists(), + "invalid namespace profiles perform no filesystem work" + ); + for (node, c) in controllers.iter().enumerate() { + c.configure_provisioning( + settings(&f, &a, node, &[directory.clone(), other.clone()]).await?, + )?; + } + assert!(controllers[0] + .configure_provisioning(settings(&f, &a, 0, &[directory.clone(), other.clone()]).await?) + .is_err()); + let error = controllers[0] + .provision(foreign.clone(), route.clone())? + .response() + .await + .unwrap_err(); + assert!(matches!( + error.as_ref(), + CellGroupControllerError::Invalid("unconfigured namespace owner") + )); + assert!(matches!( + f.cluster.voters[0].log.cell_group(&route.history), + Err(LogError::NotFound) + )); + assert_eq!(controllers[0].usage().preparation_bytes, 0); + assert!(controllers[0].usage().provision_configuration_bytes > 0); + store.hold_group_head_reads.store(true, Ordering::Release); + let first = controllers[0].provision(directory.clone(), route.clone())?; + wait_for(async || Ok(store.active_group_head_reads.load(Ordering::Acquire) == 1)).await?; + let second = controllers[0].provision(directory.clone(), route.clone())?; + wait_for(async || Ok(controllers[0].usage().provision_waiters == 2)).await?; + assert!(matches!( + controllers[0].provision(directory.clone(), route.clone()), + Err(CellGroupControllerError::Runtime(RuntimeError::Overloaded)) + )); + assert_eq!(controllers[0].usage().pending_registrations, 1); + assert_eq!(controllers[0].usage().retained_preparations, 1); + assert_eq!(store.active_group_head_reads.load(Ordering::Acquire), 1); + let mut second = Box::pin(second.response()); + assert!( + tokio::time::timeout(Duration::from_millis(30), second.as_mut()) + .await + .is_err() + ); + drop(first); + // Dropping waiters cannot release a still-running physical preparation. + assert_eq!(controllers[0].usage().retained_preparations, 1); + store.hold_group_head_reads.store(false, Ordering::Release); + store.group_head_reads_resume.notify_waiters(); + second.await?; + wait_for(async || { + Ok(controllers[0].usage().groups == 2 && controllers[0].usage().provision_waiters == 0) + }) + .await?; + assert_eq!(controllers[0].usage().preparation_bytes, 0); + let rejected = controllers[0] + .provision(other, other_route)? + .response() + .await + .unwrap_err(); + assert!(matches!( + rejected.as_ref(), + CellGroupControllerError::Runtime(RuntimeError::Overloaded) + )); + assert!(controllers[0] + .provision(foreign, route.clone())? + .response() + .await + .is_err()); + let mut wrong = route.clone(); + wrong.schema_digest = [99; 32]; + assert!(controllers[0] + .provision(directory.clone(), wrong)? + .response() + .await + .is_err()); + for c in &controllers[1..] { + c.provision(directory.clone(), route.clone())? + .response() + .await?; + } + controllers[0] + .provision(directory.clone(), route.clone())? + .response() + .await?; + f.runtimes[0].campaign(route.history).await?; + wait_for(async || Ok(f.runtimes[0].status(route.history).await?.ready)).await?; + let dispatch = CellGroupDispatcher::new( + &f.cluster.domain, + directory.clone(), + class.clone(), + keys(190)?, + CellDispatchLimits { + max_cached_routes: 32, + route_cache_bytes: 256 * 1024, + max_invocations: 8, + retained_bytes: 1024 * 1024, + max_input_bytes: 1024, + }, + )?; + let cell = history(&route, "provisioned-cell")?; + registration::enroll(&f, &controllers[0], &dispatch, &cell).await?; + wait_for(async || Ok(controllers[0].usage().retained_commands == 0)).await?; + let retry = f + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms() as u64 + 60_000 }) + .await?; + registration::mutate(&f, &controllers[0], &dispatch, &class, &cell, retry).await?; + wait_for(async || Ok(controllers[0].usage().retained_commands == 0)).await?; + registration::mutate(&f, &controllers[0], &dispatch, &class, &cell, retry).await?; + wait_for(async || Ok(controllers[0].usage().retained_commands == 0)).await?; + for runtime in &f.runtimes { + wait_for(async || { + Ok(runtime + .reconciliation_authority(route.history, cell.clone()) + .await + .is_ok_and(|a| a.cell().version.sequence.get() == 2)) + }) + .await?; + } + for c in &controllers { + assert_eq!(c.usage().groups, 2); + c.close().await?; + assert_eq!(c.usage().provision_waiters, 0); + } + drop(dispatch); + drop(controllers); + f.close().await?; + } + Ok(()) +} + +pub(crate) async fn provisioning_retries_retained_owner() -> Result<()> { + for rooted in [false, true] { + let class = class()?; + let (mut f, _, a, _) = Box::pin(registered::setup(rooted, &class)).await?; + let controllers = controller::start_with_limits(&mut f, controller::limits()).await?; + let (directory, route, store) = assignment(&f, 31).await?; + let mut options = settings(&f, &a, 0, &[directory.clone()]).await?; + let path = pages_path(&options.directory, route.history); + let faults = physical::Faults::new(path.clone(), None); + options.filesystem = Arc::new(physical::FaultFileSystem { + faults: faults.clone(), + }); + controllers[0].configure_provisioning(options)?; + store.fail_read.store(true, Ordering::Release); + assert!(controllers[0] + .provision(directory.clone(), route.clone())? + .response() + .await + .is_err()); + assert_eq!(controllers[0].usage().pending_registrations, 0); + assert_eq!(controllers[0].usage().preparation_bytes, 0); + let fault_store = store.clone(); + *faults.before_directory_sync.lock().unwrap() = Some(Box::new(move || { + fault_store.fail_read.store(true, Ordering::Release); + })); + // Catalog refusal occurs only after physical recovery. The selected page + // owner remains held and must not be reopened on the next attempt. + assert!(controllers[0] + .provision(directory.clone(), route.clone())? + .response() + .await + .is_err()); + assert_eq!(controllers[0].usage().retained_preparations, 1); + assert_eq!(controllers[0].usage().pending_registrations, 1); + assert!(cellar_ltx::FileSystem::lock_file(&DirectFileSystem, &path.join("LOCK")).is_err()); + let syncs = faults.synced_paths.lock().unwrap().len(); + controllers[0] + .provision(directory.clone(), route.clone())? + .response() + .await?; + assert_eq!( + faults.synced_paths.lock().unwrap().len(), + syncs, + "retry must reuse recovered page owner" + ); + assert_eq!(controllers[0].usage().retained_preparations, 0); + for c in controllers { + c.close().await?; + } + f.close().await?; + } + Ok(()) +} + +struct Release(Option>); + +impl Drop for Release { + fn drop(&mut self) { + if let Some(sender) = self.0.take() { + let _ = sender.send(()); + } + } +} + +pub(crate) async fn provisioning_joins_cancelled_close() -> Result<()> { + for rooted in [false, true] { + let class = class()?; + let (mut f, _, a, _) = Box::pin(registered::setup(rooted, &class)).await?; + let controllers = controller::start_with_limits(&mut f, controller::limits()).await?; + let (directory, route, _) = assignment(&f, 31).await?; + let mut options = settings(&f, &a, 0, &[directory.clone()]).await?; + let faults = physical::Faults::new(pages_path(&options.directory, route.history), None); + let entered = Arc::new(AtomicBool::new(false)); + let reached = entered.clone(); + let (send, receive) = std::sync::mpsc::sync_channel(1); + let release = Release(Some(send)); + *faults.before_directory_sync.lock().unwrap() = Some(Box::new(move || { + reached.store(true, Ordering::Release); + receive + .recv_timeout(Duration::from_secs(15)) + .expect("physical join release"); + })); + options.filesystem = Arc::new(physical::FaultFileSystem { faults }); + controllers[0].configure_provisioning(options)?; + let response = controllers[0].provision(directory, route)?; + wait_for(async || Ok(entered.load(Ordering::Acquire))).await?; + assert!( + tokio::time::timeout(Duration::from_millis(30), controllers[0].close()) + .await + .is_err() + ); + assert!( + tokio::time::timeout(Duration::from_millis(30), controllers[0].wait()) + .await + .is_err() + ); + assert_eq!(controllers[0].usage().retained_preparations, 1); + assert!(matches!( + CellGroupRuntime::start( + f.cluster.voters[0].log.clone(), + RaftRuntimeOptions::default() + ) + .await, + Err(RuntimeError::Storage(LogError::AlreadyExists)) + )); + drop(response); + drop(release); + controllers[0].close().await?; + let (replacement, events) = CellGroupRuntime::start( + f.cluster.voters[0].log.clone(), + RaftRuntimeOptions::default(), + ) + .await?; + replacement.close().await?; + drop(events); + for c in controllers { + c.close().await?; + assert_eq!(c.usage().retained_preparations, 0); + } + f.close().await?; + } + Ok(()) +} + +pub(crate) async fn provisioning_recovers_selected_roots() -> Result<()> { + for rooted in [false, true] { + let class = class()?; + let (mut f, _, a, _) = Box::pin(registered::setup(rooted, &class)).await?; + let controllers = controller::start_with_limits(&mut f, controller::limits()).await?; + let (directory, route, _) = assignment(&f, 31).await?; + for (n, c) in controllers.iter().enumerate() { + c.configure_provisioning(settings(&f, &a, n, &[directory.clone()]).await?)?; + c.provision(directory.clone(), route.clone())? + .response() + .await?; + } + f.runtimes[0].campaign(route.history).await?; + wait_for(async || Ok(f.runtimes[0].status(route.history).await?.ready)).await?; + let dispatch = CellGroupDispatcher::new( + &f.cluster.domain, + directory.clone(), + class.clone(), + keys(190)?, + CellDispatchLimits { + max_cached_routes: 32, + route_cache_bytes: 256 * 1024, + max_invocations: 8, + retained_bytes: 1024 * 1024, + max_input_bytes: 1024, + }, + )?; + let cell = history(&route, "recovered-provision")?; + registration::enroll(&f, &controllers[0], &dispatch, &cell).await?; + let retry = f + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms() as u64 + 60_000 }) + .await?; + registration::mutate(&f, &controllers[0], &dispatch, &class, &cell, retry).await?; + for runtime in &f.runtimes { + wait_for(async || { + Ok(runtime + .reconciliation_authority(route.history, cell.clone()) + .await + .is_ok_and(|a| a.cell().version.sequence.get() == 2)) + }) + .await?; + } + for c in controllers { + c.close().await?; + } + reopen_and_verify(&mut f, directory, &route, &cell, &dispatch, &class, retry).await?; + drop(dispatch); + f.close().await?; + } + Ok(()) +} + +// Image-store handles remain in this process; the engine/page owners and runtime +// are fresh. This is deliberately not independent-process/power-loss evidence. +async fn reopen_and_verify( + f: &mut RuntimeCluster, + directory: RangeDirectory, + route: &GroupRoute, + cell: &CellHistory, + dispatch: &CellGroupDispatcher, + class: &RaftClass, + retry: u64, +) -> Result<()> { + f.appliers.clear(); + f.transports.clear(); + f.runtimes.clear(); + for n in 0..3 { + f.cluster.voters[n].log.close().await?; + let number = n as u8 + 1; + let log = f + .cluster + .domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + f.cluster._root.path().join(format!("node-{number}/log")), + NodeId::new([number; 16])?, + DiskLineage::new([number + 10; 16])?, + ProcessGeneration::new([21; 16])?, + LogOpenMode::Resume, + ), + Arc::new(fault::FaultFileSystem { + gate: f.cluster.voters[n].gate.clone(), + }), + )) + .await??; + f.cluster.voters[n].log = log; + let (runtime, events) = CellGroupRuntime::start( + f.cluster.voters[n].log.clone(), + RaftRuntimeOptions { + tick_interval_ms: None, + election_seed: Some([71; 32]), + ..Default::default() + }, + ) + .await?; + f.transports + .push(super::super::tcp::start(&runtime, n).await?); + f.runtimes.push(runtime); + f.events.push(events); + } + let mut controllers = vec![]; + let mut pools = vec![]; + let events = std::mem::take(&mut f.events); + for (n, events) in events.into_iter().enumerate() { + let pool = CellGroupSqlPool::start( + &f.cluster.voters[n].log, + CellSqlLimits { + workers: 1, + queue_per_worker: 1, + max_pending_cells: 1, + group_pending_cells: 1, + node_bytes: 1024 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + }, + )?; + let c = CellGroupController::start( + f.runtimes[n].clone(), + events, + f.transports[n].clone(), + pool.clone(), + vec![], + controller::limits(), + ) + .await?; + assert_eq!(c.usage().groups, 0); + let options = settings(f, cell, n, &[directory.clone()]).await?; + let marker = f.cluster.voters[n] + .log + .cell_group(&route.history)? + .group_manifest()? + .context("selected marker")?; + let page = pages_path(&options.directory, route.history).join(format!( + "{}.page", + hex::encode(marker.root().digest().context("nonempty root")?) + )); + let saved = if n == 0 { + Some(std::fs::read(&page)?) + } else { + None + }; + c.configure_provisioning(options)?; + if let Some(bytes) = saved { + std::fs::remove_file(&page)?; + assert!(c + .provision(directory.clone(), route.clone())? + .response() + .await + .is_err()); + assert_eq!(c.usage().groups, 0); + assert_eq!( + f.cluster.voters[n] + .log + .cell_group(&route.history)? + .group_manifest()? + .as_ref(), + Some(&marker) + ); + std::fs::write(&page, bytes)?; + } + c.provision(directory.clone(), route.clone())? + .response() + .await?; + controllers.push(c); + pools.push(pool); + } + assert!(matches!( + f.runtimes[0] + .execution_authority(route.history, cell.clone()) + .await, + Err(RuntimeError::NotLeader) + )); + f.runtimes[0].campaign(route.history).await?; + wait_for(async || Ok(f.runtimes[0].status(route.history).await?.ready)).await?; + registration::mutate(f, &controllers[0], dispatch, class, cell, retry).await?; + let deadline = f + .cluster + .domain + .run(async { cellar::asyncrt::mono_ms() + 5_000 }) + .await?; + let authority = f.runtimes[0] + .read_authority( + route.history, + cell.clone(), + CellReadConsistency::Linearizable, + deadline, + ) + .await?; + let output = pools[0] + .query(f.cell_options(cell).await?, authority, |db, version| { + assert_eq!(version.sequence.get(), 2); + let n: i64 = db.query_row("SELECT n FROM value", [], |row| row.get(0))?; + Ok(n.to_le_bytes().to_vec()) + }) + .await?; + assert_eq!(output.result(), &8_i64.to_le_bytes()); + drop(output); + for c in controllers { + c.close().await?; + } + Ok(()) +} + +pub(crate) async fn provisioning_refusals_preserve_healthy_group() -> Result<()> { + for rooted in [false, true] { + for missing_member in [false, true] { + let class = class()?; + let (mut f, _, a, _) = Box::pin(registered::setup(rooted, &class)).await?; + let controllers = controller::start_with_limits(&mut f, controller::limits()).await?; + let (directory, route, _) = assignment(&f, 31).await?; + let mut options = settings(&f, &a, 0, &[directory.clone()]).await?; + let incorrect_endpoint = std::net::TcpListener::bind("127.0.0.1:0")?; + if missing_member { + options.peers.pop(); + } else { + options.peers[0].control_address = incorrect_endpoint.local_addr()?; + } + controllers[0].configure_provisioning(options)?; + let error = controllers[0] + .provision(directory, route.clone())? + .response() + .await + .unwrap_err(); + if missing_member { + assert!(matches!( + error.as_ref(), + CellGroupControllerError::Invalid("unconfigured birth member") + )); + assert!(matches!( + f.cluster.voters[0].log.cell_group(&route.history), + Err(LogError::NotFound) + )); + assert_eq!(controllers[0].usage().retained_preparations, 0); + } else { + assert!(matches!( + error.as_ref(), + CellGroupControllerError::Transport(TransportError::Identity) + )); + assert_eq!(controllers[0].usage().retained_preparations, 1); + } + assert!(!controllers[0].usage().closing); + let dispatch = CellGroupDispatcher::new( + &f.cluster.domain, + f.cluster.directory.clone(), + class.clone(), + keys(190)?, + CellDispatchLimits { + max_cached_routes: 32, + route_cache_bytes: 256 * 1024, + max_invocations: 8, + retained_bytes: 1024 * 1024, + max_input_bytes: 1024, + }, + )?; + let retry = f + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms() as u64 + 60_000 }) + .await?; + registration::mutate(&f, &controllers[0], &dispatch, &class, &a, retry).await?; + for c in controllers { + c.close().await?; + } + drop(dispatch); + f.close().await?; + } + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_quarantine.rs b/crates/cellar/tests/support/cell_group_quarantine.rs new file mode 100644 index 000000000..3256dcdb8 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_quarantine.rs @@ -0,0 +1,320 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Faults cross real manifest/engine barriers; healthy groups share the same node. +use super::apply_faults::physical; +use super::remote_queries::{keys, wait_for}; +use super::*; +use std::sync::atomic::{AtomicUsize, Ordering}; + +pub(crate) async fn group_quarantine_retains_producer_and_serves_healthy_group() -> Result<()> { + for rooted in [false, true] { + let root = tempfile::tempdir()?; + let faults = physical::Faults::new(root.path().join("node-1/pages"), None); + let filesystem = Arc::new(physical::FaultFileSystem { + faults: faults.clone(), + }); + let calls = Arc::new(AtomicUsize::new(0)); + let count = calls.clone(); + let class = Arc::new( + RaftClass::new("Counter", 1, [81; 32])? + .initializer([83; 32], |_, tx| { + tx.execute_batch("CREATE TABLE value(n); INSERT INTO value VALUES(7);")?; + Ok(()) + })? + .command("Add", [82; 32], move |ctx, input| { + count.fetch_add(1, Ordering::SeqCst); + ctx.sql().execute("UPDATE value SET n=n+1", [])?; + Ok(input.to_vec()) + })?, + ); + let (mut f, route, a, _) = + Box::pin(registered::setup_in(rooted, &class, root, filesystem)).await?; + // One retained producer must consume its group quota, not every node + // slot. Keep the same one physical SQL worker and bounded queue. + for n in 0..3 { + let (old, applier) = f.appliers.remove(&(n, route.history)).context("old pool")?; + old.close().await?; + drop(old); + let pool = CellGroupSqlPool::start( + &f.cluster.voters[n].log, + CellSqlLimits { + workers: 1, + queue_per_worker: 1, + max_pending_cells: 4, + group_pending_cells: 1, + node_bytes: 1024 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + }, + )?; + f.appliers.insert((n, route.history), (pool, applier)); + } + let controllers = controller::start_with_limits(&mut f, controller::limits()).await?; + let mut second = registration::prepare(&f, 31, &a).await?; + let other = second.birth.route().history; + for (n, input) in std::mem::take(&mut second.inputs).into_iter().enumerate() { + controllers[n].register(input)?.response().await?; + } + let healthy_dispatch = registration::dispatcher(&f, &second, class.clone())?; + let healthy_cell = history(second.birth.route(), "healthy-after-quarantine")?; + f.runtimes[0].campaign(other).await?; + wait_for(async || Ok(f.runtimes[0].status(other).await?.ready)).await?; + let before = f.cluster.voters[0] + .log + .cell_group(&route.history)? + .group_manifest()? + .context("before")?; + let captured = f.runtimes[0] + .execution_authority(route.history, a.clone()) + .await?; + let dispatcher = CellGroupDispatcher::new( + &f.cluster.domain, + f.cluster.directory.clone(), + class.clone(), + keys(89)?, + CellDispatchLimits { + max_cached_routes: 32, + route_cache_bytes: 256 * 1024, + max_invocations: 8, + retained_bytes: 1024 * 1024, + max_input_bytes: 1024, + }, + )?; + let retry = f + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms().max(0) as u64 + 60_000 }) + .await?; + faults + .mode + .store(physical::FAIL_DIRECTORY_SYNC, Ordering::Release); + let call = controllers[0].command( + controller_commands::invocation( + &dispatcher, + &class, + &a, + Duration::from_secs(15), + retry, + ) + .await?, + f.cell_options(&a).await?, + )?; + let error = call.response().await.err().context("quarantined command")?; + assert!(error.ambiguous); + assert!(matches!( + error.cause.as_ref(), + CellDispatchError::Runtime(RuntimeError::GroupQuarantined) + )); + wait_for(async || { + Ok(controllers[0].usage().quarantined_groups == 1 + && controllers[0].usage().retained_commands == 1) + }) + .await?; + assert_eq!(faults.mode.load(Ordering::Acquire), 0); + assert!(matches!( + controllers[0].group_failure(route.history)?, + Some(CellGroupControllerError::Apply(_)) + )); + assert!(!controllers[0].usage().closing); + assert_eq!(controllers[0].usage().groups, 1); + assert!( + captured.check().is_err(), + "captured leader authority revoked" + ); + assert!(matches!( + f.runtimes[0].campaign(route.history).await, + Err(RuntimeError::GroupQuarantined) + )); + assert!(matches!( + f.runtimes[0] + .read_authority( + route.history, + a.clone(), + CellReadConsistency::Stale, + f.cluster + .domain + .run(async { cellar::asyncrt::mono_ms() + 5000 }) + .await? + ) + .await, + Err(RuntimeError::GroupQuarantined) + )); + let pool = f.appliers[&(0, route.history)].0.clone(); + assert_eq!(pool.usage()?.active_cells, 1, "ambiguous producer retained"); + assert!(CellGroupRuntime::start( + f.cluster.voters[0].log.clone(), + RaftRuntimeOptions::default() + ) + .await + .is_err()); + // A refused retry must not replay the callback or settle the original producer. + let refused = controllers[0] + .command( + controller_commands::invocation( + &dispatcher, + &class, + &a, + Duration::from_secs(15), + retry, + ) + .await?, + f.cell_options(&a).await?, + )? + .response() + .await + .err() + .context("quarantine refusal")?; + assert!(!refused.ambiguous); + assert!(matches!( + refused.cause.as_ref(), + CellDispatchError::Runtime(RuntimeError::GroupQuarantined) + )); + assert_eq!(calls.load(Ordering::SeqCst), 1); + registration::enroll(&f, &controllers[0], &healthy_dispatch, &healthy_cell).await?; + registration::mutate( + &f, + &controllers[0], + &healthy_dispatch, + &class, + &healthy_cell, + retry, + ) + .await?; + registration::mutate( + &f, + &controllers[0], + &healthy_dispatch, + &class, + &healthy_cell, + retry, + ) + .await?; + assert_eq!( + calls.load(Ordering::SeqCst), + 2, + "healthy retry returns its receipt without replay" + ); + let deadline = f + .cluster + .domain + .run(async { cellar::asyncrt::mono_ms() + 5000 }) + .await?; + let authority = f.runtimes[0] + .read_authority( + other, + healthy_cell.clone(), + CellReadConsistency::Linearizable, + deadline, + ) + .await?; + let result = pool + .query( + f.cell_options(&healthy_cell).await?, + authority, + |db, version| { + assert_eq!(version.sequence.get(), 2); + let value: i64 = db.query_row("SELECT n FROM value", [], |row| row.get(0))?; + Ok(value.to_le_bytes().to_vec()) + }, + ) + .await?; + assert_eq!(result.result(), &8_i64.to_le_bytes()); + drop(result); + let mut third = registration::prepare(&f, 35, &a).await?; + let refused = controllers[0] + .register(third.inputs.remove(0)) + .err() + .context("quarantine retains group capacity")?; + assert!(matches!( + refused.error, + CellGroupControllerError::Runtime(RuntimeError::Overloaded) + )); + drop(refused); + drop(third); + assert_eq!( + f.cluster.voters[0] + .log + .cell_group(&route.history)? + .group_manifest()? + .as_ref(), + Some(&before) + ); + assert_eq!(controllers[0].usage().retained_commands, 1); + assert_eq!(pool.usage()?.active_cells, 1); + assert_eq!(f.transports[0].usage().peers, 2); + for controller in &controllers { + controller.close().await?; + } + assert_eq!(controllers[0].usage().retained_commands, 0); + assert_eq!(pool.usage()?.active_cells, 0); + assert_eq!(dispatcher.usage().1, 0); + drop(captured); + drop(controllers); + f.close().await?; + } + Ok(()) +} + +pub(crate) async fn shared_engine_sync_failure_still_stops_all_groups() -> Result<()> { + for rooted in [false, true] { + let class = RaftClass::new("Counter", 1, [81; 32])?; + let (mut f, route, _, _) = Box::pin(registered::setup(rooted, &class)).await?; + let other = controller::add_second(&mut f, route.history).await?; + let controllers = controller::start_with_limits(&mut f, controller::limits()).await?; + f.runtimes[0].campaign(other).await?; + wait_for(async || Ok(f.runtimes[0].status(other).await?.ready)).await?; + let gate = f.cluster.voters[0].gate.clone(); + let release = gate.arm(fault::FAIL_SYNC); + let cell = history(&route, "engine-failure")?; + let image = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(91);")?; + let _ = f.runtimes[0] + .propose( + route.history, + apply::enrollment_bytes(&route, &cell, &image, rooted)?, + ) + .await; + assert!( + tokio::time::timeout(Duration::from_secs(10), controllers[0].wait()) + .await? + .is_err() + ); + assert_eq!( + gate.mode.load(Ordering::Acquire), + fault::OFF, + "real log sync fault consumed" + ); + assert!(controllers[0].usage().closing); + for group in [route.history, other] { + assert!(matches!( + f.runtimes[0].status(group).await, + Err(RuntimeError::Closed) + )); + assert!(f.runtimes[0].campaign(group).await.is_err()); + } + assert!(CellGroupRuntime::start( + f.cluster.voters[0].log.clone(), + RaftRuntimeOptions::default() + ) + .await + .is_err()); + drop(release); + for controller in &controllers[1..] { + controller.close().await?; + } + for (pool, _) in f.appliers.values() { + pool.close().await?; + } + f.appliers.clear(); + assert!(f.runtimes[0].close().await.is_err()); + for runtime in &f.runtimes[1..] { + runtime.close().await?; + } + f.runtimes.clear(); + drop(controllers); + assert!(f.cluster.voters[0].log.close().await.is_err()); + for voter in &f.cluster.voters[1..] { + voter.log.close().await?; + } + f.cluster.domain.shutdown().await; + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_query_origins.rs b/crates/cellar/tests/support/cell_group_query_origins.rs new file mode 100644 index 000000000..6a604c6f2 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_query_origins.rs @@ -0,0 +1,694 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! A fourth TLS process identity routes reads without any local group or voter. +use super::remote_queries::{dispatcher, gateways, invocation, keys, wait_for}; +use super::*; +use cellar::application::consensus::verification::cell_groups::CellQueryRouting; +use cellar::peer_auth::PeerAuth; +use std::sync::{ + atomic::{AtomicUsize, Ordering}, + Mutex, +}; +use tokio::net::TcpListener; + +pub(crate) async fn nonmember_controller_queries(rooted: bool) -> Result<()> { + let calls = Arc::new(AtomicUsize::new(0)); + let count = calls.clone(); + let class = Arc::new(RaftClass::new("Counter", 1, [81; 32])?.query( + "Get", + [82; 32], + move |ctx, input| { + assert_eq!(input, b"router-input"); + count.fetch_add(1, Ordering::SeqCst); + Ok(ctx + .query_row("SELECT n FROM value", [], |r| r.get::<_, i64>(0))? + .to_be_bytes() + .to_vec()) + }, + )?); + let (mut fixture, route, a, b) = Box::pin(registered::setup(rooted, &class)).await?; + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let owners = gateways(&fixture, &controllers, class.clone()).await?; + let Router { + log, + runtime, + events, + transport, + peer, + } = Router::start(&fixture).await?; + // The first configured endpoint accepts TCP but never completes TLS. An + // automatic query must leave enough of its deadline for the second voter. + let silent = TcpListener::bind("127.0.0.1:0").await?; + let mut discovery = fixture.tcp_route(route.history)?; + discovery.peers[0].query_address = silent.local_addr()?; + transport.install_cell_query_group(discovery)?; + let pool = CellGroupSqlPool::start( + &log, + fixture.appliers[&(0, route.history)].0.usage()?.limits, + )?; + let controller = CellGroupController::start( + runtime.clone(), + events, + transport.clone(), + pool.clone(), + vec![], + controller::limits(), + ) + .await?; + let dispatch = dispatcher(&fixture, class.clone(), keys(190)?)?; + let query = |cell: &CellHistory, consistency, routing| -> Result { + Ok(dispatch.query_routed( + cell.clone(), + CellQueryRequest { + invocation: ProposalId::new([219; 16])?, + contract: class.contract("Get").context("Get")?, + input: b"router-input".to_vec(), + consistency, + }, + CellRoutingLimits { + timeout: Duration::from_secs(2), + max_attempts: 2, + }, + None, + routing, + )?) + }; + let preferred = || -> Result { + Ok(CellQueryRouting::Replicas { + members: vec![MemberId::new(2)?, MemberId::new(3)?], + attempt_timeout: Duration::from_millis(180), + leader_fallback: true, + }) + }; + let automatic = cellar::application::consensus::QueryOptions::default().routing; + assert_eq!(controller.usage().groups, 0); + assert!(controller + .query( + query(&a, CellQueryConsistency::Linearizable, preferred()?)?, + fixture.cell_options(&a).await? + )? + .response() + .await + .is_err()); + assert!(controller + .query( + query(&a, CellQueryConsistency::Linearizable, automatic.clone())?, + fixture.cell_options(&a).await?, + )? + .response() + .await + .is_err()); + assert_eq!(calls.load(Ordering::SeqCst), 0); + for destination in &fixture.transports { + destination.grant_query_origin(NamespaceQueryOrigin { + cluster: a.cluster, + namespace: a.namespace, + peer: peer.clone(), + revision: 1, + })?; + } + let response = controller + .query( + query(&a, CellQueryConsistency::Linearizable, preferred()?)?, + fixture.cell_options(&a).await?, + )? + .response() + .await?; + assert_eq!(response.result(), 0i64.to_be_bytes()); + assert_eq!(response.observation().version.history, a); + assert_eq!( + response.replica(), + fixture.cluster.voters[1] + .log + .cell_group(&route.history)? + .identity() + ); + assert_eq!(transport.usage().query_items, 1); + let token = response.token().clone(); + drop(response); + wait_for(async || Ok(transport.usage().query_items == 0 && dispatch.usage().0 == 0)).await?; + assert!(matches!( + controller + .query( + query( + &b, + CellQueryConsistency::AtLeast(token.clone()), + preferred()? + )?, + fixture.cell_options(&b).await? + )? + .response() + .await + .err() + .context("wrong cell")? + .as_ref(), + CellDispatchError::Version(cellar::application::consensus::VersionError::WrongHistory) + )); + assert_eq!(calls.load(Ordering::SeqCst), 1); + let response = controller + .query( + query( + &a, + CellQueryConsistency::AtLeast(token.clone()), + automatic.clone(), + )?, + fixture.cell_options(&a).await?, + )? + .response() + .await?; + assert_eq!(response.result(), 0i64.to_be_bytes()); + assert_eq!(response.observation().version.history, a); + assert_eq!( + response.replica(), + fixture.cluster.voters[1] + .log + .cell_group(&route.history)? + .identity() + ); + drop(response); + assert!(matches!( + controller + .query( + query(&b, CellQueryConsistency::AtLeast(token), automatic)?, + fixture.cell_options(&b).await?, + )? + .response() + .await + .err() + .context("automatic wrong cell")? + .as_ref(), + CellDispatchError::Version(cellar::application::consensus::VersionError::WrongHistory) + )); + assert!(matches!( + controller + .query( + query( + &a, + CellQueryConsistency::Linearizable, + CellQueryRouting::Local + )?, + fixture.cell_options(&a).await?, + )? + .response() + .await + .err() + .context("explicit local route has no replica")? + .as_ref(), + CellDispatchError::Runtime(RuntimeError::NotFound) + )); + assert_eq!(calls.load(Ordering::SeqCst), 2); + wait_for(async || Ok(transport.usage().query_items == 0 && dispatch.usage().0 == 0)).await?; + assert_eq!(controller.usage().groups, 0); + assert!(log.cell_groups()?.is_empty()); + assert_eq!(transport.usage().groups, 0); + assert_eq!(pool.usage()?.retained_operations, 0); + assert!(matches!( + runtime.status(route.history).await, + Err(RuntimeError::NotFound) + )); + controller.close().await?; + log.close().await?; + drop(owners); + for controller in controllers { + controller.close().await?; + } + fixture.close().await?; + Ok(()) +} + +pub(super) struct Router { + pub(super) log: ReplicaLog, + pub(super) runtime: CellGroupRuntime, + pub(super) events: CellGroupRuntimeEvents, + pub(super) transport: RaftTransport, + pub(super) peer: RaftPeerNode, +} + +impl Router { + pub(super) async fn start(fixture: &RuntimeCluster) -> Result { + let node = NodeId::new([4; 16])?; + let disk = DiskLineage::new([14; 16])?; + let process = ProcessGeneration::new([20; 16])?; + let log = fixture + .cluster + .domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + fixture.cluster._root.path().join("router-log"), + node, + disk, + process, + LogOpenMode::Create, + ), + Arc::new(DirectFileSystem), + )) + .await??; + let (runtime, events) = CellGroupRuntime::start( + log.clone(), + RaftRuntimeOptions { + tick_interval_ms: None, + ..Default::default() + }, + ) + .await?; + let transport = RaftTransport::start_shared( + runtime.clone(), + Arc::new(PeerAuth::new([17; 32], "node-4".to_owned())?), + super::super::tcp::tls()?, + TcpListener::bind("127.0.0.1:0").await?, + TcpListener::bind("127.0.0.1:0").await?, + TcpListener::bind("127.0.0.1:0").await?, + TransportOptions { + max_groups: 2, + ..Default::default() + }, + ) + .await?; + let peer = RaftPeerNode { + node, + disk, + process, + label: "node-4".into(), + server_name: "localhost".into(), + control_address: transport.control_address(), + data_address: transport.data_address(), + query_address: transport.query_address(), + }; + Ok(Self { + log, + runtime, + events, + transport, + peer, + }) + } + + pub(super) fn grant(&self, fixture: &RuntimeCluster, revision: u64) -> Result<()> { + let history = fixture.cluster.birth.route().history; + for transport in &fixture.transports { + transport.grant_query_origin(NamespaceQueryOrigin { + cluster: history.cluster, + namespace: history.namespace, + peer: self.peer.clone(), + revision, + })?; + } + Ok(()) + } + + pub(super) async fn close(self) -> Result<()> { + self.transport.close().await; + self.runtime.close().await?; + drop(self.events); + self.log.close().await?; + Ok(()) + } +} + +pub(crate) async fn nonmember_queries() -> Result<()> { + for rooted in [false, true] { + let calls = Arc::new(AtomicUsize::new(0)); + let count = calls.clone(); + let class = Arc::new(RaftClass::new("Counter", 1, [81; 32])?.query( + "Get", + [82; 32], + move |ctx, _| { + count.fetch_add(1, Ordering::SeqCst); + assert_eq!( + ctx.query_row("SELECT n FROM value", [], |r| r.get::<_, i64>(0))?, + 0 + ); + Ok(ctx.identity().cell.as_str().as_bytes().to_vec()) + }, + )?); + let (mut fixture, route, a, b) = Box::pin(registered::setup(rooted, &class)).await?; + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let owners = gateways(&fixture, &controllers, class.clone()).await?; + let router = Router::start(&fixture).await?; + let discovery = fixture.tcp_route(route.history)?; + router + .transport + .install_cell_query_group(discovery.clone())?; + router + .transport + .install_cell_query_group(discovery.clone())?; + assert!(router.log.cell_groups()?.is_empty()); + assert_eq!(router.transport.usage().groups, 0); + assert_eq!(router.transport.usage().peers, 3); + assert!(router.runtime.status(route.history).await.is_err()); + let mut heartbeat = Message { + from: 1, + to: 2, + ..Default::default() + }; + heartbeat.set_msg_type(MessageType::MsgHeartbeat); + assert!(matches!( + router.transport.submit_physical(&route.history, heartbeat), + Err(TransportError::UnknownGroup) + )); + assert!(matches!( + fixture.transports[0].install_cell_query_group(discovery.clone()), + Err(TransportError::Identity) + )); + assert!(matches!( + router.transport.install_physical_group(discovery.clone()), + Err(TransportError::UnknownGroup) + )); + let dispatch = dispatcher(&fixture, class.clone(), keys(190)?)?; + let mut denied = invocation( + &dispatch, + &class, + &a, + CellQueryConsistency::Stale, + Duration::from_secs(5), + b"", + ) + .await?; + assert!(denied + .execute_remote(&router.transport, MemberId::new(1)?) + .await + .is_err()); + drop(denied); + assert_eq!(calls.load(Ordering::SeqCst), 0); + // Discovery and a grant in another namespace cannot authorize this read. + fixture.transports[0].grant_query_origin(NamespaceQueryOrigin { + cluster: a.cluster, + namespace: NamespaceId::new([199; 16])?, + revision: 1, + peer: router.peer.clone(), + })?; + let mut denied = invocation( + &dispatch, + &class, + &a, + CellQueryConsistency::Stale, + Duration::from_secs(5), + b"", + ) + .await?; + assert!(denied + .execute_remote(&router.transport, MemberId::new(1)?) + .await + .is_err()); + drop(denied); + assert_eq!(calls.load(Ordering::SeqCst), 0); + let mut wrong_process = router.peer.clone(); + wrong_process.process = ProcessGeneration::new([195; 16])?; + // Use a different destination: its namespace grant must bind the actual + // authenticated source process, not merely the router's stable node ID. + fixture.transports[1].grant_query_origin(NamespaceQueryOrigin { + cluster: a.cluster, + namespace: a.namespace, + revision: 1, + peer: wrong_process, + })?; + let mut denied = invocation( + &dispatch, + &class, + &a, + CellQueryConsistency::Stale, + Duration::from_secs(5), + b"", + ) + .await?; + assert!(denied + .execute_remote(&router.transport, MemberId::new(2)?) + .await + .is_err()); + drop(denied); + assert_eq!(calls.load(Ordering::SeqCst), 0); + let mut denied_strong = invocation( + &dispatch, + &class, + &a, + CellQueryConsistency::Linearizable, + Duration::from_secs(5), + b"", + ) + .await?; + assert!(denied_strong + .execute_remote(&router.transport, MemberId::new(2)?) + .await + .is_err()); + drop(denied_strong); + assert_eq!( + calls.load(Ordering::SeqCst), + 0, + "strong reads also require the exact namespace/process grant" + ); + router.grant(&fixture, 2)?; + let mut query = invocation( + &dispatch, + &class, + &a, + CellQueryConsistency::Linearizable, + Duration::from_secs(5), + b"", + ) + .await?; + let first_attempt = query.attempt().context("first strong attempt")?.clone(); + // A granted follower obtains its own correlated quorum read. This + // router still has no local replica and cannot supply that authority. + let follower = query + .execute_remote(&router.transport, MemberId::new(2)?) + .await?; + assert_eq!(follower.result(), a.cell.as_str().as_bytes()); + assert_eq!(follower.observation().version.history, a); + assert_eq!(follower.served_by(), MemberId::new(2)?); + let follower_observation = follower.observation().clone(); + assert_eq!(router.transport.usage().query_items, 1); + drop(follower); + assert_eq!(router.transport.usage().query_items, 0); + assert_eq!(calls.load(Ordering::SeqCst), 1); + query.retry().await?; + assert_eq!( + query.attempt().context("leader retry")?.deadline_ms(), + first_attempt.deadline_ms() + ); + assert_eq!(query.attempt().context("leader retry")?.ordinal(), 2); + let reply = query + .execute_remote(&router.transport, MemberId::new(1)?) + .await?; + assert_eq!(reply.result(), a.cell.as_str().as_bytes()); + assert_eq!(reply.observation().version.history, a); + assert_eq!(reply.observation(), &follower_observation); + assert_eq!(reply.served_by(), MemberId::new(1)?); + assert_eq!(calls.load(Ordering::SeqCst), 2); + let token = reply.token().clone(); + assert_eq!(router.transport.usage().query_items, 1); + drop(reply); + drop(query); + for (cell, consistency) in [ + (&a, CellQueryConsistency::AtLeast(token)), + (&b, CellQueryConsistency::Stale), + ] { + let mut query = invocation( + &dispatch, + &class, + cell, + consistency, + Duration::from_secs(5), + b"", + ) + .await?; + assert_eq!( + query + .execute_remote(&router.transport, MemberId::new(3)?) + .await? + .result(), + cell.cell.as_str().as_bytes() + ); + } + assert_eq!(calls.load(Ordering::SeqCst), 4); + // All discovery routes share the same three physical peer pools and + // bounded route admission. They never allocate local group storage. + let mut second = discovery.clone(); + second.scope.group = GroupId::new([198; 16])?; + for peer in &mut second.peers { + peer.replica.scope = second.scope; + } + router.transport.install_cell_query_group(second.clone())?; + assert_eq!(router.transport.usage().peers, 3); + let mut third = second.clone(); + third.scope.group = GroupId::new([197; 16])?; + for peer in &mut third.peers { + peer.replica.scope = third.scope; + } + assert!(matches!( + router.transport.install_cell_query_group(third), + Err(TransportError::Overloaded) + )); + let mut changed = discovery.clone(); + changed.revision = 2; + changed.peers[0].replica.process = ProcessGeneration::new([196; 16])?; + assert!(matches!( + router.transport.install_cell_query_group(changed), + Err(TransportError::StalePeer) + )); + let mut changed = discovery.clone(); + changed.revision = 2; + router.transport.install_cell_query_group(changed)?; + assert!(matches!( + router.transport.install_cell_query_group(discovery), + Err(TransportError::StalePeer) + )); + router.transport.remove_cell_query_group(&second.scope)?; + assert_eq!(router.transport.usage().peers, 3); + router.transport.remove_cell_query_group(&route.history)?; + assert_eq!(router.transport.usage().peers, 0); + let mut removed = invocation( + &dispatch, + &class, + &a, + CellQueryConsistency::Stale, + Duration::from_secs(5), + b"", + ) + .await?; + assert!(matches!( + removed + .execute_remote(&router.transport, MemberId::new(1)?) + .await, + Err(CellDispatchError::Transport(TransportError::UnknownGroup)) + )); + drop(removed); + assert_eq!(dispatch.usage(), (0, 0)); + assert!(router.log.cell_groups()?.is_empty()); + router.close().await?; + for controller in controllers { + controller.close().await?; + } + drop(owners); + fixture.close().await?; + } + Ok(()) +} + +pub(crate) async fn nonmember_query_revocation(replace_route: bool) -> Result<()> { + for rooted in [false, true] { + let (entered, started) = tokio::sync::oneshot::channel(); + let entered = Mutex::new(Some(entered)); + let (release, held) = std::sync::mpsc::sync_channel(1); + let held = Mutex::new(held); + let calls = Arc::new(AtomicUsize::new(0)); + let count = calls.clone(); + let class = Arc::new(RaftClass::new("Counter", 1, [81; 32])?.query( + "Get", + [82; 32], + move |ctx, _| { + if count.fetch_add(1, Ordering::SeqCst) == 0 { + entered + .lock() + .unwrap() + .take() + .context("once")? + .send(()) + .map_err(|_| anyhow::anyhow!("observer"))?; + held.lock().unwrap().recv_timeout(Duration::from_secs(15))?; + } + Ok(ctx.identity().cell.as_str().as_bytes().to_vec()) + }, + )?); + let (mut fixture, route, a, _) = Box::pin(registered::setup(rooted, &class)).await?; + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let owners = gateways(&fixture, &controllers, class.clone()).await?; + let router = Router::start(&fixture).await?; + let mut discovery = fixture.tcp_route(route.history)?; + router + .transport + .install_cell_query_group(discovery.clone())?; + router.grant(&fixture, 1)?; + let dispatch = dispatcher(&fixture, class.clone(), keys(190)?)?; + let mut query = invocation( + &dispatch, + &class, + &a, + CellQueryConsistency::Linearizable, + Duration::from_secs(10), + b"", + ) + .await?; + let source = router.transport.clone(); + let waiter = fixture.cluster.domain.spawn(async move { + query + .execute_remote(&source, MemberId::new(1).unwrap()) + .await + })?; + tokio::time::timeout(Duration::from_secs(5), started).await??; + let pool = &fixture.appliers[&(0, route.history)].0; + assert_eq!(pool.usage()?.retained_operations, 1); + if replace_route { + discovery.revision += 1; + router.transport.install_cell_query_group(discovery)?; + // Even a complete authenticated reply cannot escape with a captured + // discovery view superseded while the remote SQL callback ran. + release.send(())?; + assert!(matches!( + waiter.await?, + Err(CellDispatchError::Transport(TransportError::StalePeer)) + )); + } else { + fixture.transports[0].revoke_query_origin( + a.cluster, + a.namespace, + router.peer.node, + 2, + )?; + assert!(tokio::time::timeout(Duration::from_secs(5), waiter) + .await?? + .is_err()); + wait_for(async || { + Ok(router.transport.usage().query_items == 0 + && fixture.transports[0].usage().query_items == 0) + }) + .await?; + assert_eq!(pool.usage()?.retained_operations, 1); + assert!(router.grant(&fixture, 1).is_err()); + let mut denied = invocation( + &dispatch, + &class, + &a, + CellQueryConsistency::Stale, + Duration::from_secs(5), + b"", + ) + .await?; + assert!(denied + .execute_remote(&router.transport, MemberId::new(1)?) + .await + .is_err()); + drop(denied); + assert_eq!(calls.load(Ordering::SeqCst), 1); + release.send(())?; + router.grant(&fixture, 3)?; + } + wait_for(async || Ok(pool.usage()?.retained_operations == 0)).await?; + let mut fresh = invocation( + &dispatch, + &class, + &a, + CellQueryConsistency::Stale, + Duration::from_secs(5), + b"", + ) + .await?; + assert_eq!( + fresh + .execute_remote(&router.transport, MemberId::new(1)?) + .await? + .result(), + a.cell.as_str().as_bytes() + ); + drop(fresh); + assert_eq!(calls.load(Ordering::SeqCst), 2); + assert_eq!(dispatch.usage(), (0, 0)); + assert!(router.log.cell_groups()?.is_empty()); + router.close().await?; + for controller in controllers { + controller.close().await?; + } + drop(owners); + fixture.close().await?; + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_query_routing.rs b/crates/cellar/tests/support/cell_group_query_routing.rs new file mode 100644 index 000000000..e1b696fe5 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_query_routing.rs @@ -0,0 +1,282 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Controller-owned attempts keep one invocation while expired physical SQL joins. +use super::remote_queries::{dispatcher, gateways, keys, wait_for}; +use super::*; +use cellar::application::consensus::verification::cell_groups::CellQueryRouting; +use cellar::application::consensus::VersionError; +use std::sync::{ + atomic::{AtomicBool, AtomicUsize, Ordering}, + Mutex, +}; + +pub(crate) async fn controller_routed_queries(rooted: bool) -> Result<()> { + for mode in 0..4 { + let (entered, started) = tokio::sync::oneshot::channel(); + let entered = Mutex::new(Some(entered)); + let (release, held) = std::sync::mpsc::sync_channel(1); + let held = Mutex::new(held); + let blocked = AtomicBool::new(false); + let calls = Arc::new(AtomicUsize::new(0)); + let count = calls.clone(); + let class = Arc::new(RaftClass::new("Counter", 1, [81; 32])?.query( + "Get", + [82; 32], + move |ctx, input| { + count.fetch_add(1, Ordering::SeqCst); + assert!(input == b"warm" || input == b"captured-input"); + if input == b"captured-input" && !blocked.swap(true, Ordering::SeqCst) { + entered + .lock() + .unwrap() + .take() + .context("first entry")? + .send(()) + .map_err(|_| anyhow::anyhow!("entry observer"))?; + held.lock().unwrap().recv_timeout(Duration::from_secs(15))?; + } + Ok(ctx + .query_row("SELECT n FROM value", [], |r| r.get::<_, i64>(0))? + .to_be_bytes() + .to_vec()) + }, + )?); + let (mut fixture, route, a, b) = Box::pin(registered::setup(rooted, &class)).await?; + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let owners = gateways(&fixture, &controllers, class.clone()).await?; + let dispatch = dispatcher(&fixture, class.clone(), keys(190)?)?; + let options = fixture.cell_options_at(&a, 1).await?; + let request = |tag, input: &[u8], consistency| -> Result { + Ok(CellQueryRequest { + invocation: ProposalId::new([tag; 16])?, + contract: class.contract("Get").context("Get")?, + input: input.to_vec(), + consistency, + }) + }; + let bounds = CellRoutingLimits { + timeout: Duration::from_secs(2), + max_attempts: if mode == 2 { 1 } else { 4 }, + }; + for policy in [ + CellQueryRouting::Replicas { + members: vec![], + attempt_timeout: Duration::from_millis(180), + leader_fallback: false, + }, + CellQueryRouting::Replicas { + members: vec![MemberId::new(2)?, MemberId::new(2)?], + attempt_timeout: Duration::from_millis(180), + leader_fallback: false, + }, + CellQueryRouting::Replicas { + members: vec![MemberId::new(2)?], + attempt_timeout: Duration::ZERO, + leader_fallback: false, + }, + CellQueryRouting::Replicas { + members: vec![MemberId::new(2)?], + attempt_timeout: Duration::from_secs(3), + leader_fallback: false, + }, + ] { + assert!(matches!( + dispatch.query_routed( + a.clone(), + request(211, b"warm", CellQueryConsistency::Stale)?, + bounds, + None, + policy + ), + Err(CellDispatchError::Invalid("query replica routing bounds")) + )); + } + assert_eq!(dispatch.usage(), (0, 0)); + assert_eq!(calls.load(Ordering::SeqCst), 0); + let warm = dispatch.query_routed( + a.clone(), + request(212, b"warm", CellQueryConsistency::Linearizable)?, + bounds, + None, + CellQueryRouting::Replicas { + members: vec![MemberId::new(3)?], + attempt_timeout: Duration::from_millis(180), + leader_fallback: false, + }, + )?; + let warm = controllers[1] + .query(warm, options.clone())? + .response() + .await?; + let token = warm.token().clone(); + let observed = warm.observation().clone(); + assert_eq!(warm.replica().member, MemberId::new(3)?); + assert_eq!( + warm.replica().node_identity(), + fixture.cluster.voters[2] + .log + .cell_group(&route.history)? + .identity() + .node_identity() + ); + assert_eq!(fixture.transports[1].usage().query_items, 1); + drop(warm); + wait_for(async || { + Ok(fixture.transports[1].usage().query_items == 0 + && dispatch.usage() == (0, 0) + && controllers[1].usage().retained_commands == 0) + }) + .await?; + + let now = fixture + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms().max(0) as u64 }) + .await?; + let wrong_schema = keys(190)?.issue_cell(&observed, [99; 32], now)?; + let invalid = dispatch.query_routed( + a.clone(), + request(215, b"warm", CellQueryConsistency::AtLeast(wrong_schema))?, + bounds, + None, + CellQueryRouting::Replicas { + members: vec![MemberId::new(3)?, MemberId::new(1)?], + attempt_timeout: Duration::from_millis(180), + leader_fallback: true, + }, + )?; + assert!(matches!( + controllers[1] + .query(invalid, options.clone())? + .response() + .await + .err() + .context("wrong schema")? + .as_ref(), + CellDispatchError::Version(VersionError::WrongSchema) + )); + assert_eq!(calls.load(Ordering::SeqCst), 1); + // Delivery may wake this task before the sending producer has returned. + // Observe its joined cleanup; the reply itself is not release authority. + wait_for(async || { + Ok(dispatch.usage() == (0, 0) && controllers[1].usage().retained_commands == 0) + }) + .await?; + assert_eq!(dispatch.usage(), (0, 0)); + assert_eq!(fixture.transports[1].usage().query_items, 0); + + let (members, fallback, held_node) = match mode { + 0 | 2 => (vec![MemberId::new(2)?, MemberId::new(3)?], false, 1), + 1 => (vec![MemberId::new(3)?], true, 2), + _ => (vec![MemberId::new(1)?], true, 0), + }; + let routed = dispatch.query_routed( + a.clone(), + request( + 213, + b"captured-input", + CellQueryConsistency::AtLeast(token.clone()), + )?, + bounds, + None, + CellQueryRouting::Replicas { + members, + attempt_timeout: Duration::from_millis(180), + leader_fallback: fallback, + }, + )?; + let response = controllers[1].query(routed, options.clone())?; + tokio::time::timeout(Duration::from_secs(5), started).await??; + let response = response.response().await; + if mode < 2 { + let response = response?; + assert_eq!(response.result(), 0i64.to_be_bytes()); + assert_eq!(response.observation(), &observed); + let member = if mode == 0 { 3 } else { 1 }; + assert_eq!(response.replica().member, MemberId::new(member)?); + assert_eq!( + response.replica().node_identity(), + fixture.cluster.voters[member as usize - 1] + .log + .cell_group(&route.history)? + .identity() + .node_identity() + ); + assert_eq!(calls.load(Ordering::SeqCst), 3); + drop(response); + } else { + let error = response.err().context("bounded routing must refuse")?; + if mode == 2 { + assert!( + matches!( + error.as_ref(), + CellDispatchError::Routing(CellRoutingError { + reason: CellRoutingFailure::AttemptsExhausted, + .. + }) + ), + "mode={mode} error={error:?}" + ); + } else { + assert!(matches!( + error.as_ref(), + CellDispatchError::Runtime(RuntimeError::DeadlineExceeded) + | CellDispatchError::Transport( + TransportError::Timeout + | TransportError::Query(CommandFailure::VersionUnavailable) + ) + )); + } + assert_eq!(calls.load(Ordering::SeqCst), 2); + } + let held_pool = fixture.appliers[&(held_node, route.history)].0.clone(); + assert_eq!(held_pool.usage()?.retained_operations, 1); + assert_eq!(controllers[held_node].usage().retained_commands, 1); + if held_node == 1 { + assert_eq!( + dispatch.usage().1, + 1, + "expired local callback retains invocation admission" + ); + assert!(dispatch.usage().0 >= 8192 + 2 * b"captured-input".len()); + } + release.send(())?; + wait_for(async || { + Ok(held_pool.usage()?.retained_operations == 0 + && controllers[held_node].usage().retained_commands == 0 + && dispatch.usage() == (0, 0) + && fixture.transports[1].usage().query_items == 0) + }) + .await?; + // A correctly signed token for the other cell cannot be retried into A. + let before = calls.load(Ordering::SeqCst); + let wrong = dispatch.query_routed( + b.clone(), + request(214, b"warm", CellQueryConsistency::AtLeast(token))?, + bounds, + None, + CellQueryRouting::Replicas { + members: vec![MemberId::new(2)?, MemberId::new(3)?], + attempt_timeout: Duration::from_millis(180), + leader_fallback: true, + }, + )?; + assert!(matches!( + controllers[1] + .query(wrong, fixture.cell_options_at(&b, 1).await?)? + .response() + .await + .err() + .context("foreign token")? + .as_ref(), + CellDispatchError::Version(VersionError::WrongHistory) + )); + assert_eq!(calls.load(Ordering::SeqCst), before); + drop(owners); + for controller in controllers { + controller.close().await?; + } + fixture.close().await?; + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_receipt_lookup.rs b/crates/cellar/tests/support/cell_group_receipt_lookup.rs new file mode 100644 index 000000000..92e8fda70 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_receipt_lookup.rs @@ -0,0 +1,466 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Receipt-only inspection across real TLS and immutable SQLite images. +use super::query_origins::Router; +use super::remote_queries::{dispatcher, keys, wait_for}; +use super::*; +use cellar::application::consensus::NamespaceCommandOrigin; +use std::sync::{ + atomic::{AtomicUsize, Ordering}, + Mutex, +}; + +fn request(class: &RaftClass, input: &[u8]) -> Result { + Ok(CellReceiptRequest { + id: CommandId::new(ClientId::new([221; 16])?, 1)?, + contract: class.contract("Add").context("Add")?, + input: input.to_vec(), + durability: Durability::Quorum, + }) +} + +fn limits() -> CellRoutingLimits { + CellRoutingLimits { + timeout: Duration::from_secs(15), + max_attempts: 8, + } +} + +async fn lookup( + dispatch: &CellGroupDispatcher, + class: &RaftClass, + cell: &CellHistory, + input: &[u8], +) -> Result { + let mut lookup = + dispatch.lookup_receipt(cell.clone(), request(class, input)?, limits(), None)?; + lookup.begin().await?; + Ok(lookup) +} + +async fn command( + dispatch: &CellGroupDispatcher, + class: &RaftClass, + cell: &CellHistory, + now: u64, +) -> Result { + let r = request(class, b"once")?; + Ok(dispatch + .command( + cell.clone(), + CellCommandRequest { + id: r.id, + contract: r.contract, + input: r.input, + durability: r.durability, + retry_until_ms: now + 60_000, + }, + limits(), + None, + ) + .await?) +} + +fn grant(fixture: &RuntimeCluster, router: &Router) -> Result<()> { + let g = fixture.cluster.birth.route().history; + fixture.transports[0].grant_command_origin(NamespaceCommandOrigin { + cluster: g.cluster, + namespace: g.namespace, + revision: 1, + peer: router.peer.clone(), + })?; + Ok(()) +} + +pub(crate) async fn receipt_lookup_ingress() -> Result<()> { + for rooted in [false, true] { + let calls = Arc::new(AtomicUsize::new(0)); + let count = calls.clone(); + let class = Arc::new(RaftClass::new("Counter", 1, [81; 32])?.command( + "Add", + [82; 32], + move |ctx, input| { + count.fetch_add(1, Ordering::SeqCst); + ctx.sql().execute("UPDATE value SET n=n+1", [])?; + Ok(input.to_vec()) + }, + )?); + let (mut fixture, route, a, _) = Box::pin(registered::setup(rooted, &class)).await?; + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let dispatch = dispatcher(&fixture, class.clone(), keys(190)?)?; + let gateway = fixture.transports[0].install_cell_command_gateway( + dispatch.clone(), + controllers[0].clone(), + cellar::application::consensus::verification::cell_groups::CellGroupNamespaceStorage::new( + fixture.cell_options(&a).await?, + )?, + )?; + let router = Router::start(&fixture).await?; + router + .transport + .install_cell_query_group(fixture.tcp_route(route.history)?)?; + let mut read = lookup(&dispatch, &class, &a, b"once").await?; + let deadline = read.attempt().unwrap().deadline_ms(); + assert!(read + .execute_remote(&router.transport, MemberId::new(1)?) + .await + .is_err()); + router.grant(&fixture, 1)?; + read.retry().await?; + assert!(read + .execute_remote(&router.transport, MemberId::new(1)?) + .await + .is_err()); + assert_eq!(calls.load(Ordering::SeqCst), 0); + grant(&fixture, &router)?; + read.retry().await?; + assert_eq!(read.attempt().unwrap().deadline_ms(), deadline); + let unknown = read + .execute_remote(&router.transport, MemberId::new(1)?) + .await?; + assert!(unknown.result().is_none()); + assert!(unknown.retry_until_ms().is_none()); + assert_eq!(unknown.observation().version.sequence.get(), 1); + assert_eq!(unknown.command(), request(&class, b"once")?.id); + assert_eq!(calls.load(Ordering::SeqCst), 0); + assert_eq!(router.transport.usage().query_items, 1); + assert!(read + .execute_remote(&router.transport, MemberId::new(1)?) + .await + .is_err()); + drop(unknown); + let now = fixture + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms() as u64 }) + .await?; + let committed = controllers[0] + .command( + command(&dispatch, &class, &a, now).await?, + fixture.cell_options(&a).await?, + )? + .response() + .await?; + assert_eq!( + committed + .local_receipt() + .expect("local command receipt") + .receipt + .result(), + b"once" + ); + drop(committed); + read.retry().await?; + let found = read + .execute_remote(&router.transport, MemberId::new(1)?) + .await?; + assert_eq!(found.result(), Some(b"once".as_slice())); + assert_eq!(found.retry_until_ms(), Some(now + 60_000)); + assert_eq!(found.observation().version.sequence.get(), 2); + assert_eq!(found.served_by(), MemberId::new(1)?); + let observed = fixture + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms() as u64 }) + .await?; + assert_eq!( + keys(190)?.verify_cell(found.token(), &a, class.schema_digest(), observed)?, + *found.observation() + ); + drop(found); + let mut conflict = lookup(&dispatch, &class, &a, b"different").await?; + assert!(matches!( + conflict + .execute_remote(&router.transport, MemberId::new(1)?) + .await + .err(), + Some(CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::CommandConflict + ))) + )); + drop(conflict); + let mut gap = request(&class, b"once")?; + gap.id = CommandId::new(gap.id.client, 3)?; + let mut gap = dispatch.lookup_receipt(a.clone(), gap, limits(), None)?; + gap.begin().await?; + assert!(matches!( + gap.execute_remote(&router.transport, MemberId::new(1)?) + .await + .err(), + Some(CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::SequenceGap { next: 2 } + ))) + )); + drop(gap); + assert_eq!(calls.load(Ordering::SeqCst), 1); + drop(gateway); + read.retry().await?; + assert!(read + .execute_remote(&router.transport, MemberId::new(1)?) + .await + .is_err()); + drop(read); + assert!(router.log.cell_groups()?.is_empty()); + assert_eq!(dispatch.usage(), (0, 0)); + router.close().await?; + for c in controllers { + c.close().await?; + } + fixture.close().await?; + } + Ok(()) +} + +async fn allow_two_sql_reservations( + fixture: &mut RuntimeCluster, + group: GroupHistory, +) -> Result<()> { + let (old_pool, applier) = fixture.appliers.remove(&(0, group)).context("old pool")?; + old_pool.close().await?; + drop(old_pool); + let pool = CellGroupSqlPool::start( + &fixture.cluster.voters[0].log, + CellSqlLimits { + workers: 1, + queue_per_worker: 2, + max_pending_cells: 2, + group_pending_cells: 2, + node_bytes: 1024 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + }, + )?; + fixture.appliers.insert((0, group), (pool, applier)); + Ok(()) +} + +pub(crate) async fn receipt_lookup_pending() -> Result<()> { + for rooted in [false, true] { + let calls = Arc::new(AtomicUsize::new(0)); + let count = calls.clone(); + let class = Arc::new(RaftClass::new("Counter", 1, [81; 32])?.command( + "Add", + [82; 32], + move |ctx, input| { + count.fetch_add(1, Ordering::SeqCst); + ctx.sql().execute("UPDATE value SET n=n+1", [])?; + Ok(input.to_vec()) + }, + )?); + let (mut fixture, route, a, _) = Box::pin(registered::setup(rooted, &class)).await?; + allow_two_sql_reservations(&mut fixture, route.history).await?; + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let dispatch = dispatcher(&fixture, class.clone(), keys(190)?)?; + let pool = fixture.appliers[&(0, route.history)].0.clone(); + let now = fixture + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms() as u64 }) + .await?; + let mut mutation = command(&dispatch, &class, &a, now).await?; + mutation.begin().await?; + assert!(matches!( + mutation + .prepare(&fixture.runtimes[0], &pool, fixture.cell_options(&a).await?) + .await?, + CellCommandPreparation::Prepared + )); + // Inspection can coexist with a pending producer, but it shares bounded + // SQL admission and cannot expose or retire the uncommitted receipt. + assert_eq!(pool.usage()?.active_cells, 1); + assert_eq!(calls.load(Ordering::SeqCst), 1); + let mut read = lookup(&dispatch, &class, &a, b"once").await?; + let empty = read + .execute(&fixture.runtimes[0], &pool, fixture.cell_options(&a).await?) + .await?; + assert!(empty.receipt().is_none()); + assert_eq!(empty.observation().version.sequence.get(), 1); + assert_eq!(pool.usage()?.active_cells, 1); + let mut saturated = lookup(&dispatch, &class, &a, b"once").await?; + assert!(matches!( + saturated + .execute(&fixture.runtimes[0], &pool, fixture.cell_options(&a).await?) + .await, + Err(CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::Overloaded + ))) + )); + assert_eq!(pool.usage()?.active_cells, 1); + drop(saturated); + drop(empty); + let r = request(&class, b"once")?; + for invalid in [0, 1, 2] { + let authority = fixture.runtimes[0] + .read_authority( + route.history, + a.clone(), + if invalid == 2 { + CellReadConsistency::Stale + } else { + CellReadConsistency::Linearizable + }, + read.attempt().unwrap().deadline_ms(), + ) + .await?; + let error = pool + .lookup_receipt_registered( + fixture.cell_options(&a).await?, + authority, + class.clone(), + CommandRequest { + id: r.id, + operation: r.contract.operation.clone(), + contract_digest: if invalid == 0 { + [0; 32] + } else { + r.contract.digest + }, + request_digest: if invalid == 1 { + [0; 32] + } else { + command_request_digest(&r.input) + }, + durability: Durability::Quorum, + retry_until_ms: 0, + }, + r.input.clone(), + ) + .await + .err() + .context("invalid lookup refuses")?; + match invalid { + 0 => assert!(matches!(error, ExecutionError::Registration(_))), + 1 => assert!(matches!( + error, + ExecutionError::Refused(CommandFailure::CommandConflict) + )), + _ => assert!(matches!( + error, + ExecutionError::Invalid("receipt lookup requires quorum read") + )), + } + } + assert_eq!(pool.usage()?.active_cells, 1); + assert!(matches!( + mutation.submit().await?, + CellCommandSubmission::Accepted(_) + )); + wait_for(async || { + match fixture.runtimes[0] + .execution_authority(route.history, a.clone()) + .await + { + Ok(authority) => Ok(authority.cell().version.sequence.get() == 2), + Err(RuntimeError::NotReady) => Ok(false), + Err(error) => Err(error.into()), + } + }) + .await?; + read.retry().await?; + let found = read + .execute(&fixture.runtimes[0], &pool, fixture.cell_options(&a).await?) + .await?; + assert_eq!(found.receipt().context("receipt")?.result(), b"once"); + assert_eq!(pool.usage()?.active_cells, 1); // Lookup cannot finish another owner. + drop(found); + assert!(matches!( + mutation.settle().await?, + CellCommandSettlement::Committed(_) + )); + assert_eq!(pool.usage()?.active_cells, 0); + assert_eq!(calls.load(Ordering::SeqCst), 1); + drop(read); + drop(mutation); + for c in &controllers { + c.close().await?; + } + // Closed controller handles cannot keep the SQL engine claim alive. + drop(fixture.appliers.remove(&(0, route.history))); + drop(pool); + let replacement = apply::pool(&fixture, 0)?; + replacement.close().await?; + drop(replacement); + drop(controllers); + fixture.close().await?; + } + Ok(()) +} + +pub(crate) async fn receipt_lookup_cancelled() -> Result<()> { + for rooted in [false, true] { + let calls = Arc::new(AtomicUsize::new(0)); + let count = calls.clone(); + let (entered, entered_rx) = tokio::sync::oneshot::channel(); + let entered = Mutex::new(Some(entered)); + let (release, held) = std::sync::mpsc::sync_channel(1); + let held = Mutex::new(held); + let class = Arc::new(RaftClass::new("Counter", 1, [81; 32])?.command( + "Add", + [82; 32], + move |ctx, input| { + count.fetch_add(1, Ordering::SeqCst); + let _ = entered.lock().unwrap().take().context("entered")?.send(()); + held.lock().unwrap().recv_timeout(Duration::from_secs(15))?; + ctx.sql().execute("UPDATE value SET n=n+1", [])?; + Ok(input.to_vec()) + }, + )?); + let (mut fixture, route, a, _) = Box::pin(registered::setup(rooted, &class)).await?; + allow_two_sql_reservations(&mut fixture, route.history).await?; + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let dispatch = dispatcher(&fixture, class.clone(), keys(190)?)?; + let gateway = fixture.transports[0].install_cell_command_gateway( + dispatch.clone(), + controllers[0].clone(), + cellar::application::consensus::verification::cell_groups::CellGroupNamespaceStorage::new( + fixture.cell_options(&a).await?, + )?, + )?; + let router = Router::start(&fixture).await?; + router + .transport + .install_cell_query_group(fixture.tcp_route(route.history)?)?; + grant(&fixture, &router)?; + let now = fixture + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms() as u64 }) + .await?; + let work = controllers[0].command( + command(&dispatch, &class, &a, now).await?, + fixture.cell_options(&a).await?, + )?; + tokio::time::timeout(Duration::from_secs(5), entered_rx).await??; + let mut read = lookup(&dispatch, &class, &a, b"once").await?; + let transport = router.transport.clone(); + let task = tokio::spawn(async move { + read.execute_remote(&transport, MemberId::new(1).unwrap()) + .await + }); + let pool = fixture.appliers[&(0, route.history)].0.clone(); + wait_for(async || Ok(pool.usage()?.queued_or_running >= 2)).await?; + task.abort(); + assert!(task.await.err().context("cancelled lookup")?.is_cancelled()); + wait_for(async || Ok(router.transport.usage().query_items == 0)).await?; + assert_eq!(controllers[0].usage().retained_commands, 1); + assert_eq!(pool.usage()?.active_cells, 1); + release.send(())?; + let committed = work.response().await?; + drop(committed); + wait_for(async || Ok(pool.usage()?.queued_or_running == 0)).await?; + let mut retry = lookup(&dispatch, &class, &a, b"once").await?; + let found = retry + .execute_remote(&router.transport, MemberId::new(1)?) + .await?; + assert_eq!(found.result(), Some(b"once".as_slice())); + assert_eq!(calls.load(Ordering::SeqCst), 1); + drop(found); + drop(retry); + drop(gateway); + router.close().await?; + for c in controllers { + c.close().await?; + } + fixture.close().await?; + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_recovery.rs b/crates/cellar/tests/support/cell_group_recovery.rs new file mode 100644 index 000000000..3c92cf92d --- /dev/null +++ b/crates/cellar/tests/support/cell_group_recovery.rs @@ -0,0 +1,1006 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Recovery qualification on actual SQLite/LTX, Raft Engine and both physical +//! image stores. These fixtures provide storage cuts, not a running Raft quorum. +#![allow(clippy::disallowed_methods)] + +use super::*; +use anyhow::{bail, ensure, Context}; +use cellar::application::consensus::verification::{GroupLog, LogOpenMode, ReplicaLog}; +use cellar::sdk::ExecutionDomain; +use cellar_logic::consensus::OperationId; +use cellar_ltx::{ + apply::{ApplyLimits, ImageState}, + generation::ImageId, + internal::{ + ExtentId, ExtentManifest, ExtentRootReference, Geometry, ImageBinding, ImageRoot, + ImageRootUpdate, NodeExtentStore, EXTENT_BYTES, + }, + FileSystem, TXID, +}; +use std::{ + path::{Path, PathBuf}, + sync::atomic::Ordering, + time::Duration, +}; + +fn engine_bootstrap() -> cellar::application::consensus::verification::CellGroupBootstrap { + cellar::application::consensus::verification::CellGroupBootstrap { + route: route(), + member: MemberId::new(1).unwrap(), + voters: vec![MemberId::new(1).unwrap()], + learners: vec![], + } +} + +#[derive(Clone, Copy, Debug)] +enum Mode { + Materialized, + Extents, +} + +impl Mode { + fn name(self) -> &'static str { + match self { + Self::Materialized => "materialized", + Self::Extents => "extents", + } + } + + fn parse(value: &str) -> Result { + match value { + "materialized" => Ok(Self::Materialized), + "extents" => Ok(Self::Extents), + _ => bail!("unknown image mode"), + } + } +} + +enum Images { + Materialized(GenerationStore), + Extents(NodeExtentStore), +} + +fn history(cell: u8) -> CellHistory { + CellHistory { + cluster: route().history.cluster, + namespace: route().history.namespace, + cell: CellId::new(format!("cell-{cell}")).unwrap(), + incarnation: IncarnationId::new([7; 16]).unwrap(), + } +} + +fn image_id(leaf: &CellManifestLeaf) -> Result { + let cell: u8 = leaf + .version + .history + .cell + .as_str() + .strip_prefix("cell-") + .context("fixture cell")? + .parse()?; + let sequence = u8::try_from(leaf.version.sequence.get())?; + let mut bytes = [0; 16]; + bytes[0] = cell; + bytes[1] = sequence; + bytes[2] = 123; + Ok(ImageId::new(bytes)?) +} + +fn extent_reference(manifest: &ExtentManifest) -> Result { + Ok(ExtentRootReference::new( + manifest.root(), + manifest.encode()?.len() as u32, + )?) +} + +fn leaf_reference(leaf: &CellManifestLeaf) -> Result { + let CellImageCommitment::Extents { + root, + manifest_bytes, + } = leaf.version.image.commitment + else { + bail!("extent leaf required"); + }; + Ok(ExtentRootReference::new( + ImageRoot::from_bytes(root), + manifest_bytes, + )?) +} + +impl Images { + fn open(path: &Path, mode: Mode, fs: Arc) -> Result { + Ok(match mode { + Mode::Materialized => Self::Materialized(GenerationStore::open( + path.to_owned(), + fs, + GenerationLimits::default(), + )?), + Mode::Extents => Self::Extents(NodeExtentStore::open( + path.to_owned(), + fs, + Default::default(), + )?), + }) + } + + fn prepare( + &self, + cell: u8, + base: Option<(&CellManifestLeaf, &CellImagePin)>, + cut: &images::Cut, + after: &[u8], + index: u64, + ) -> Result<(CellManifestLeaf, CellImagePin)> { + let state = cut.result; + let mut leaf = CellManifestLeaf { + enrollment: EnrollmentRecord::new( + history(cell), + ProposalId::new([cell; 16])?, + route().schema_digest, + CellImage { + epoch: state.epoch, + txid: state.txid.0, + checksum: state.checksum, + commitment: CellImageCommitment::Materialized(state.digest), + }, + )?, + version: CellVersion { + history: history(cell), + sequence: MutationSequence::new(state.txid.0)?, + image: CellImage { + epoch: state.epoch, + txid: state.txid.0, + checksum: state.checksum, + commitment: CellImageCommitment::Materialized(state.digest), + }, + }, + geometry: CellImageGeometry { + length: state.length, + page_size: state.page_size, + }, + last_mutation: position(index), + witness: CommitWitness { + group: route().history, + position: position(index), + proposal: ProposalId::new([if state.txid.0 == 1 { cell } else { cell + 32 }; 16])?, + }, + effects: EffectSummary::default(), + }; + let previous = base.map(|(leaf, _)| leaf.enrollment.clone()); + let base = base.map(|(_, pin)| pin); + let pin = match self { + Self::Materialized(store) => { + let base = match base { + None => None, + Some(CellImagePin::Materialized(pin)) => Some(pin), + _ => bail!("wrong physical base"), + }; + CellImagePin::Materialized(store.prepare(image_id(&leaf)?, base, &cut.update())?) + } + Self::Extents(store) => { + let binding = ImageBinding { + history: cell_image_history_digest(&leaf.version.history), + schema: route().schema_digest, + epoch: state.epoch, + txid: state.txid.0, + checksum: state.checksum, + }; + let geometry = Geometry::new(state.page_size, state.length)?; + let ids = after + .chunks(EXTENT_BYTES as usize) + .map(ExtentId::of) + .collect::, _>>()?; + let manifest = ExtentManifest::new(binding, geometry, &ids)?; + let reference = extent_reference(&manifest)?; + let base = match base { + None => None, + Some(CellImagePin::Extents(pin)) => Some(pin.as_ref()), + _ => bail!("wrong physical base"), + }; + let empty = ExtentManifest::new( + ImageBinding { + txid: 0, + checksum: 0, + ..binding + }, + Geometry::new(0, 0)?, + &[], + )?; + let update = ImageRootUpdate { + base: base + .map(|p| p.reference()) + .unwrap_or(extent_reference(&empty)?), + result: reference, + binding, + geometry, + segments: cut.update().segments, + }; + let prepared = store.prepare_image_root( + base, + &update, + &ApplyLimits::default(), + &mut vec![0; EXTENT_BYTES as usize], + )?; + leaf.version.image.commitment = CellImageCommitment::Extents { + root: reference.root().bytes(), + manifest_bytes: reference.encoded_length(), + }; + CellImagePin::Extents(Arc::new(prepared.image)) + } + }; + leaf.enrollment = match previous { + Some(record) => record, + None => EnrollmentRecord::new( + history(cell), + ProposalId::new([cell; 16])?, + route().schema_digest, + leaf.version.image, + )?, + }; + Ok((leaf, pin)) + } + + fn recover(&self, leaf: &CellManifestLeaf) -> Result { + Ok(match self { + Self::Materialized(store) => { + let CellImageCommitment::Materialized(digest) = leaf.version.image.commitment + else { + bail!("materialized leaf required"); + }; + CellImagePin::Materialized(store.adopt( + image_id(leaf)?, + ImageState { + epoch: leaf.version.image.epoch, + txid: TXID(leaf.version.image.txid), + checksum: leaf.version.image.checksum, + length: leaf.geometry.length, + page_size: leaf.geometry.page_size, + digest, + }, + )?) + } + Self::Extents(store) => { + let view = store.open_root(leaf_reference(leaf)?)?; + let manifest = view.manifest().clone(); + // Inventory/root metadata is never a seal after restart. Verify + // every dependency, including extents after the SQLite header. + for i in 0..manifest.geometry().extent_count() { + store.verify_extent(manifest.extent(i).context("extent closure")?)?; + } + CellImagePin::Extents(Arc::new( + store.seal_image_root(manifest, &mut vec![0; EXTENT_BYTES as usize])?, + )) + } + }) + } + + fn collect(&self, leaves: &[CellManifestLeaf]) -> Result<()> { + match self { + Self::Materialized(store) => { + store.retain_only(&leaves.iter().map(image_id).collect::>>()?)?; + store.reclaim(4096)?; + } + Self::Extents(store) => { + store.retain_only( + &leaves + .iter() + .map(leaf_reference) + .collect::>>()?, + )?; + store.reclaim(4096)?; + } + } + Ok(()) + } +} + +fn read_image(pin: &CellImagePin) -> Result> { + match pin { + CellImagePin::Materialized(pin) => images::read(pin), + CellImagePin::Extents(pin) => { + let manifest = pin.manifest(); + let mut bytes = Vec::new(); + for i in 0..manifest.geometry().extent_count() { + bytes.extend( + pin.view() + .read_extent(manifest.extent(i).context("extent")?)?, + ); + } + ensure!(bytes.len() as u64 == manifest.geometry().length()); + Ok(bytes) + } + CellImagePin::Empty => Ok(Vec::new()), + } +} + +fn initial(cell: u8, large: bool) -> Result> { + let padding = if large { + "CREATE TABLE padding(bytes); INSERT INTO padding VALUES(zeroblob(4202496));" + } else { + "" + }; + images::image( + None, + &format!( + "CREATE TABLE value(n INTEGER); INSERT INTO value VALUES ({}); + CREATE TABLE receipts(client TEXT PRIMARY KEY, floor INTEGER, result BLOB); + INSERT INTO receipts VALUES ('client-{cell}',1,X'{}'); {padding}", + cell * 10 + 1, + hex::encode(format!("r{cell}-1")) + ), + ) +} + +fn mutate(before: &[u8]) -> Result> { + images::image( + Some(before), + "UPDATE value SET n = n + 1; UPDATE receipts SET floor = 2, result = X'72312d32';", + ) +} + +fn assert_receipt(pin: &CellImagePin, cell: u8, sequence: u64) -> Result<()> { + let temp = tempfile::tempdir()?; + let path = temp.path().join("oracle.sqlite"); + std::fs::write(&path, read_image(pin)?)?; + let db = + rusqlite::Connection::open_with_flags(path, rusqlite::OpenFlags::SQLITE_OPEN_READ_ONLY)?; + let value: u64 = db.query_row("SELECT n FROM value", [], |row| row.get(0))?; + let row: (String, u64, Vec) = + db.query_row("SELECT client, floor, result FROM receipts", [], |row| { + Ok((row.get(0)?, row.get(1)?, row.get(2)?)) + })?; + assert_eq!(value, u64::from(cell) * 10 + sequence); + assert_eq!( + row, + ( + format!("client-{cell}"), + sequence, + format!("r{cell}-{sequence}").into_bytes() + ) + ); + assert_eq!( + db.query_row("SELECT COUNT(*) FROM receipts", [], |r| r.get::<_, u64>(0))?, + 1 + ); + Ok(()) +} + +fn prepare_root( + tree: &ManifestTree, + group: &GroupLog, + pages: &mut ManifestPageStore, + marker: &GroupManifest, + images: &Images, +) -> Result { + Ok(tree.prepare_complete( + group, + pages, + marker.root(), + marker.position(), + marker.membership_revision(), + |leaf| { + images + .recover(leaf) + .map_err(|_| ManifestError::Invalid("incomplete physical image")) + }, + )?) +} + +async fn seed(path: &Path, mode: Mode, large: bool) -> Result { + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let fs = Arc::new(DirectFileSystem); + let log = domain + .run(ReplicaLog::open( + engine_options(&path.join("node"), LogOpenMode::Create, 1), + fs.clone(), + )) + .await??; + let group = log.create_cell_group(engine_bootstrap()).await?; + let images = Images::open(&path.join("images"), mode, fs.clone())?; + let mut pages = ManifestPageStore::open(&path.join("pages"), fs, 256, 4 * 1024 * 1024, 16)?; + let tree = ManifestTree::new(route(), ManifestLimits::default())?; + let mut prepared = tree.prepare_complete( + &group, + &mut pages, + ManifestRoot::default(), + position(1), + 1, + |_| unreachable!(), + )?; + for cell in 1..=2 { + let bytes = initial(cell, large && cell == 1)?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let (leaf, pin) = images.prepare(cell, None, &cut, &bytes, u64::from(cell) + 1)?; + let entry = enrollment_entry(&leaf.enrollment, &cut)?; + let (next, decision) = tree.prepare_enrollment( + &mut pages, + &prepared, + EnrollmentApply { + position: leaf.last_mutation, + phase: EnrollmentPhase::Active, + entry: &entry, + limits: GroupEntryLimits::default(), + }, + |record| { + assert_eq!(record, &leaf.enrollment); + Ok(EnrollmentImage { + geometry: leaf.geometry, + effects: leaf.effects, + pin, + }) + }, + )?; + assert!(matches!(decision, EnrollmentDecision::Propose(_))); + prepared = next; + let mut write = root_write(&group, prepared.clone())?; + write.entries[0].data = entry.into(); + log.persist_cells(OperationId::new(u64::from(cell)), vec![write]) + .await?; + } + let marker = prepared.marker().clone(); + log.close().await?; + domain.shutdown().await; + Ok(marker) +} + +pub async fn both_images() -> Result<()> { + for mode in [Mode::Materialized, Mode::Extents] { + let temp = tempfile::tempdir()?; + seed(temp.path(), mode, true).await?; + { + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let fs = Arc::new(DirectFileSystem); + let log = domain + .run(ReplicaLog::open( + engine_options(&temp.path().join("node"), LogOpenMode::Resume, 2), + fs.clone(), + )) + .await??; + let group = log.cell_group(&engine_bootstrap().route.history)?; + let images = Images::open(&temp.path().join("images"), mode, fs.clone())?; + let mut pages = + ManifestPageStore::open(&temp.path().join("pages"), fs, 256, 4 * 1024 * 1024, 16)?; + let tree = ManifestTree::new(route(), ManifestLimits::default())?; + let current = prepare_root( + &tree, + &group, + &mut pages, + &group.group_manifest()?.context("selected root")?, + &images, + )?; + let (old, old_pin) = current.pin_cell(&history(1).cell).unwrap(); + let sibling = current.pin_cell(&history(2).cell).unwrap().0.clone(); + let before = read_image(&old_pin)?; + let after = mutate(&before)?; + let cut = images::Cut::new(&before, &after, 1, true, false); + let (changed, next_pin) = images.prepare(1, Some((old, &old_pin)), &cut, &after, 4)?; + if let (CellImagePin::Extents(old), CellImagePin::Extents(next)) = (&old_pin, &next_pin) + { + assert!(old.manifest().geometry().extent_count() > 1); + assert_eq!( + old.manifest().extent(1), + next.manifest().extent(1), + "unchanged tail extent is shared" + ); + } + let next = + tree.prepare_update(&mut pages, ¤t, changed, Some(&old.version), next_pin)?; + log.persist_cells(OperationId::new(3), vec![root_write(&group, next.clone())?]) + .await?; + assert_eq!(next.pin_cell(&history(2).cell).unwrap().0, &sibling); + assert_receipt(&old_pin, 1, 1)?; + assert_receipt(&next.pin_cell(&history(1).cell).unwrap().1, 1, 2)?; + assert_receipt(&next.pin_cell(&history(2).cell).unwrap().1, 2, 1)?; + log.close().await?; + domain.shutdown().await; + } + recover_and_check(temp.path(), mode, Some(4)).await?; + replay_enrollment(temp.path(), mode).await?; + recover_and_check(temp.path(), mode, Some(5)).await?; + } + Ok(()) +} + +async fn recover_and_check(path: &Path, mode: Mode, expected: Option) -> Result<()> { + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let fs = Arc::new(DirectFileSystem); + let log = domain + .run(ReplicaLog::open( + engine_options(&path.join("node"), LogOpenMode::Resume, 3), + fs.clone(), + )) + .await??; + let group = log.cell_group(&engine_bootstrap().route.history)?; + let marker = group.group_manifest()?.context("selected root")?; + let index = marker.position().index.get(); + if let Some(expected) = expected { + assert_eq!(index, expected); + } + if expected != Some(5) { + assert!([3, 4].contains(&index)); + } + assert_eq!(marker.root().cells(), 2); + let images = Images::open(&path.join("images"), mode, fs.clone())?; + let mut pages = ManifestPageStore::open(&path.join("pages"), fs, 256, 4 * 1024 * 1024, 16)?; + let tree = ManifestTree::new(route(), ManifestLimits::default())?; + let restored = prepare_root(&tree, &group, &mut pages, &marker, &images)?; + let mut leaves = Vec::new(); + for cell in 1..=2 { + let (leaf, pin) = restored.pin_cell(&history(cell).cell).unwrap(); + let sequence = if cell == 1 && index >= 4 { 2 } else { 1 }; + assert_eq!(leaf.version.history, history(cell)); + assert_eq!(leaf.enrollment.cell(), &history(cell)); + assert_eq!(leaf.enrollment.creation(), ProposalId::new([cell; 16])?); + assert_eq!(leaf.enrollment.initial().txid, 1); + assert_eq!(leaf.version.sequence.get(), sequence); + assert_eq!(leaf.witness.group, route().history); + assert_eq!( + leaf.witness.position, + position(if sequence == 2 { + 4 + } else { + u64::from(cell) + 1 + }) + ); + assert_eq!( + leaf.witness.proposal, + ProposalId::new([if sequence == 1 { cell } else { cell + 32 }; 16])? + ); + let keys = cellar::application::consensus::VersionKeys::new( + cellar::application::consensus::VersionSigningKey::new([72; 32], 100_000)?, + vec![], + Duration::from_secs(10), + )?; + let observation = leaf.observation(); + let token = keys.issue_cell(&observation, route().schema_digest, 100)?; + assert_eq!( + keys.verify_cell(&token, &history(cell), route().schema_digest, 101)?, + observation + ); + assert_receipt(&pin, cell, sequence)?; + leaves.push(leaf.clone()); + } + pages.reconcile_retained(&[])?; + collect_pages(&mut pages)?; + images.collect(&leaves)?; + // Collection must not turn either selected image into an empty/missing cell. + let verified = prepare_root(&tree, &group, &mut pages, &marker, &images)?; + assert_eq!(verified.marker(), &marker); + assert_eq!(pages.usage().pages, 2); + log.close().await?; + domain.shutdown().await; + Ok(()) +} + +pub async fn missing_images() -> Result<()> { + for mode in [Mode::Materialized, Mode::Extents] { + let kinds: &[&str] = match mode { + Mode::Materialized => &[ + "page-missing", + "page-corrupt", + "image-missing", + "image-corrupt", + ], + Mode::Extents => &[ + "page-missing", + "page-corrupt", + "root-missing", + "root-corrupt", + "tail-missing", + "tail-corrupt", + ], + }; + for kind in kinds { + let temp = tempfile::tempdir()?; + let marker = seed(temp.path(), mode, true).await?; + let victim = if kind.starts_with("page-") { + temp.path().join("pages").join(format!( + "{}.page", + hex::encode(marker.root().digest().unwrap()) + )) + } else { + let fs = Arc::new(DirectFileSystem); + let pages = ManifestPageStore::open( + &temp.path().join("pages"), + fs.clone(), + 256, + 4 * 1024 * 1024, + 16, + )?; + let tree = ManifestTree::new(route(), ManifestLimits::default())?; + let cell = if kind.starts_with("tail-") { 1 } else { 2 }; + let leaf = tree + .get(&pages, marker.root(), &history(cell).cell)? + .unwrap(); + match mode { + Mode::Materialized => temp + .path() + .join("images") + .join(format!("{}.sqlite", hex::encode(image_id(&leaf)?.bytes()))), + Mode::Extents => { + let reference = leaf_reference(&leaf)?; + if kind.starts_with("root-") { + temp.path() + .join("images") + .join(format!("r-{}", hex::encode(reference.root().bytes()))) + } else { + let store = NodeExtentStore::open( + temp.path().join("images"), + fs, + Default::default(), + )?; + let view = store.open_root(reference)?; + assert!(view.manifest().geometry().extent_count() > 1); + let extent = view.manifest().extent(1).unwrap(); + temp.path().join("images").join(format!( + "e-{:08x}-{}", + extent.length(), + hex::encode(extent.bytes()) + )) + } + } + } + }; + if kind.ends_with("missing") { + std::fs::remove_file(&victim)?; + } else { + let mut bytes = std::fs::read(&victim)?; + let last = bytes.len() - 1; + bytes[last] ^= 0x80; + std::fs::write(&victim, bytes)?; + } + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let fs = Arc::new(DirectFileSystem); + let log = domain + .run(ReplicaLog::open( + engine_options(&temp.path().join("node"), LogOpenMode::Resume, 2), + fs.clone(), + )) + .await??; + let group = log.cell_group(&engine_bootstrap().route.history)?; + assert_eq!(group.group_manifest()?.as_ref(), Some(&marker)); + let images = Images::open(&temp.path().join("images"), mode, fs.clone())?; + let mut pages = + ManifestPageStore::open(&temp.path().join("pages"), fs, 256, 4 * 1024 * 1024, 16)?; + let tree = ManifestTree::new(route(), ManifestLimits::default())?; + assert!( + prepare_root(&tree, &group, &mut pages, &marker, &images).is_err(), + "{mode:?}/{kind}" + ); + assert!(group.applied().is_err()); + assert!( + pages.reclaim(16).is_err(), + "failed recovery cannot authorize reclamation" + ); + log.close().await?; + domain.shutdown().await; + } + } + Ok(()) +} + +struct Child(std::process::Child); + +impl Drop for Child { + fn drop(&mut self) { + let _ = self.0.kill(); + let _ = self.0.wait(); + } +} + +pub async fn crash_matrix() -> Result<()> { + for mode in [Mode::Materialized, Mode::Extents] { + for cut in [ + "before-image", + "image-write", + "image-before-sync", + "image-file-sync", + "image-directory-sync", + "after-image", + "manifest-write", + "manifest-before-sync", + "manifest-file-sync", + "manifest-directory-sync", + "after-manifest", + "engine-before-sync", + "engine-after-sync", + "after-publication", + "reclaim-after-remove", + ] { + let temp = tempfile::tempdir()?; + seed(temp.path(), mode, false).await?; + let signal = temp.path().join("signal"); + let output = temp.path().join("child.log"); + let file = std::fs::File::create(&output)?; + let mut child = Child( + std::process::Command::new(std::env::current_exe()?) + .args([ + "--ignored", + "--exact", + "manifest_process_fixture", + "--nocapture", + "--test-threads=1", + ]) + .env("CELLAR_GROUP_TEST_ROOT", temp.path()) + .env("CELLAR_GROUP_TEST_MODE", mode.name()) + .env("CELLAR_GROUP_TEST_CUT", cut) + .env("CELLAR_GROUP_TEST_SIGNAL", &signal) + .stdout(file.try_clone()?) + .stderr(file) + .spawn()?, + ); + tokio::time::timeout(Duration::from_secs(30), async { + while !signal.exists() { + if let Some(status) = child.0.try_wait()? { + bail!( + "{mode:?}/{cut} exited {status}: {}", + std::fs::read_to_string(&output)? + ); + } + tokio::time::sleep(Duration::from_millis(5)).await; + } + Ok::<_, anyhow::Error>(()) + }) + .await??; + child.0.kill()?; + let status = child.0.wait()?; + #[cfg(unix)] + { + use std::os::unix::process::ExitStatusExt; + assert_eq!(status.signal(), Some(9)); + } + assert!(!status.success()); + let expected = match cut { + "engine-before-sync" => None, + "engine-after-sync" | "after-publication" | "reclaim-after-remove" => Some(4), + _ => Some(3), + }; + recover_and_check(temp.path(), mode, expected) + .await + .with_context(|| format!("{mode:?}/{cut}"))?; + assert_eq!( + std::fs::read_to_string(temp.path().join("executions"))?, + "1", + "recovery must not rerun the callback" + ); + } + } + Ok(()) +} + +pub async fn process_fixture() -> Result<()> { + let path = PathBuf::from(std::env::var("CELLAR_GROUP_TEST_ROOT")?); + let mode = Mode::parse(&std::env::var("CELLAR_GROUP_TEST_MODE")?)?; + let cut_name = std::env::var("CELLAR_GROUP_TEST_CUT")?; + let signal = PathBuf::from(std::env::var("CELLAR_GROUP_TEST_SIGNAL")?); + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let gate = engine_fault::Gate::new(Some(signal.clone())); + let log = domain + .run(ReplicaLog::open( + engine_options(&path.join("node"), LogOpenMode::Resume, 2), + Arc::new(engine_fault::FaultFileSystem { gate: gate.clone() }), + )) + .await??; + let group = log.cell_group(&engine_bootstrap().route.history)?; + let image_fault = faults::Faults::new(path.join("images"), Some(signal.clone())); + let page_fault = faults::Faults::new(path.join("pages"), Some(signal)); + image_fault + .all_directory_files + .store(true, Ordering::Release); + page_fault + .all_directory_files + .store(true, Ordering::Release); + let images = Images::open( + &path.join("images"), + mode, + Arc::new(faults::FaultFileSystem { + faults: image_fault.clone(), + }), + )?; + let mut pages = ManifestPageStore::open( + &path.join("pages"), + Arc::new(faults::FaultFileSystem { + faults: page_fault.clone(), + }), + 256, + 4 * 1024 * 1024, + 16, + )?; + let tree = ManifestTree::new(route(), ManifestLimits::default())?; + let current = prepare_root( + &tree, + &group, + &mut pages, + &group.group_manifest()?.unwrap(), + &images, + )?; + let (old, old_pin) = current.pin_cell(&history(1).cell).unwrap(); + let before = read_image(&old_pin)?; + let after = mutate(&before)?; + // This independent callback ledger is durable before any crash gate arms. + std::fs::write(path.join("executions"), b"1")?; + std::fs::File::open(path.join("executions"))?.sync_all()?; + std::fs::File::open(&path)?.sync_all()?; + if cut_name == "before-image" { + image_fault.pause()?; + } + let select = |prefix: &str| match cut_name.strip_prefix(prefix) { + Some("write") => faults::AFTER_IMAGE_WRITE, + Some("before-sync") => faults::BEFORE_IMAGE_SYNC, + Some("file-sync") => faults::AFTER_IMAGE_SYNC, + Some("directory-sync") => faults::AFTER_DIRECTORY_SYNC, + _ => 0, + }; + image_fault.mode.store(select("image-"), Ordering::Release); + let cut = images::Cut::new(&before, &after, 1, true, false); + let (changed, pin) = images.prepare(1, Some((old, &old_pin)), &cut, &after, 4)?; + if cut_name == "after-image" { + image_fault.pause()?; + } + page_fault + .mode + .store(select("manifest-"), Ordering::Release); + let next = tree.prepare_update(&mut pages, ¤t, changed, Some(&old.version), pin)?; + if cut_name == "after-manifest" { + page_fault.pause()?; + } + let _release = gate.arm(match cut_name.as_str() { + "engine-before-sync" => engine_fault::CRASH_BEFORE_SYNC, + "engine-after-sync" => engine_fault::CRASH_AFTER_SYNC, + _ => engine_fault::OFF, + }); + log.persist_cells(OperationId::new(3), vec![root_write(&group, next)?]) + .await?; + if cut_name == "after-publication" { + page_fault.pause()?; + } + drop(current); + drop(old_pin); + pages.reconcile_retained(&[])?; + if cut_name == "reclaim-after-remove" { + page_fault + .mode + .store(faults::AFTER_REMOVE, Ordering::Release); + collect_pages(&mut pages)?; + } + bail!("requested cut was never reached: {cut_name}") +} + +fn enrollment_entry(record: &EnrollmentRecord, cut: &images::Cut) -> Result> { + let route = route(); + Ok(CellGroupEntry { + binding: GroupEntryBinding { + group: route.history, + route_epoch: route.epoch, + birth_digest: route.birth_digest, + schema_digest: route.schema_digest, + proposal: record.creation(), + }, + command: GroupCommand::Enroll { + cell: record.cell().clone(), + image: LtxImage::try_from(record.initial())?, + files: vec![GroupLtx { + min_txid: 1, + max_txid: cut.result.txid.0, + page_count: cut.pages.len() as u64, + decoded_bytes: cut.pages.len() as u64 * 512, + bytes: &cut.bytes, + }], + }, + } + .encode(&route, GroupEntryLimits::default())?) +} + +async fn replay_enrollment(path: &Path, mode: Mode) -> Result<()> { + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let fs = Arc::new(DirectFileSystem); + let log = domain + .run(ReplicaLog::open( + engine_options(&path.join("node"), LogOpenMode::Resume, 4), + fs.clone(), + )) + .await??; + let group = log.cell_group(&route().history)?; + let images = Images::open(&path.join("images"), mode, fs.clone())?; + let mut pages = ManifestPageStore::open(&path.join("pages"), fs, 256, 4 * 1024 * 1024, 16)?; + let tree = ManifestTree::new(route(), ManifestLimits::default())?; + let current = prepare_root( + &tree, + &group, + &mut pages, + &group.group_manifest()?.context("selected root")?, + &images, + )?; + let (old, old_pin) = current.pin_cell(&history(1).cell).unwrap(); + assert_eq!(old.version.sequence.get(), 2); + assert_eq!(old.enrollment.initial().txid, 1); + assert_ne!(old.enrollment.initial(), old.version.image); + let initial_leaf = CellManifestLeaf { + version: CellVersion { + sequence: MutationSequence::new(1)?, + image: old.enrollment.initial(), + ..old.version.clone() + }, + ..old.clone() + }; + assert!( + images.recover(&initial_leaf).is_err(), + "the original image has been reclaimed; creation identity must survive independently" + ); + let bytes = initial(1, true)?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let entry = enrollment_entry(&old.enrollment, &cut)?; + let before = pages.usage().written_pages; + let apply = |phase, index| EnrollmentApply { + position: position(index), + phase, + entry: &entry, + limits: GroupEntryLimits::default(), + }; + for phase in [EnrollmentPhase::Sealed, EnrollmentPhase::Retired] { + assert!(tree + .prepare_enrollment(&mut pages, ¤t, apply(phase, 5), |_| panic!( + "sealed or retired retry cannot initialize" + )) + .is_err()); + } + for index in [4, 6] { + assert!(tree + .prepare_enrollment( + &mut pages, + ¤t, + apply(EnrollmentPhase::Active, index), + |_| panic!("invalid prefix cannot initialize") + ) + .is_err()); + } + let competing = EnrollmentRecord::new( + old.version.history.clone(), + ProposalId::new([99; 16])?, + route().schema_digest, + old.enrollment.initial(), + )?; + let competing_entry = enrollment_entry(&competing, &cut)?; + let error = tree.prepare_enrollment( + &mut pages, + ¤t, + EnrollmentApply { + entry: &competing_entry, + ..apply(EnrollmentPhase::Active, 5) + }, + |_| panic!("competing creation cannot initialize"), + ); + assert!(matches!( + error, + Err(EnrollmentApplyError::Enrollment(EnrollmentError::Conflict)) + )); + let (next, decision) = tree.prepare_enrollment( + &mut pages, + ¤t, + apply(EnrollmentPhase::Active, 5), + |_| panic!("duplicate must not rerun initialization"), + )?; + assert_eq!( + decision, + EnrollmentDecision::Existing(old.enrollment.clone()) + ); + assert_eq!(next.marker().root(), current.marker().root()); + assert_eq!(next.marker().position(), position(5)); + assert_eq!(pages.usage().written_pages, before); + for cell in 1..=2 { + assert_eq!( + next.pin_cell(&history(cell).cell).unwrap().0, + current.pin_cell(&history(cell).cell).unwrap().0 + ); + } + let mut write = root_write(&group, next.clone())?; + write.entries[0].data = entry.into(); + log.persist_cells(OperationId::new(7), vec![write]).await?; + assert_eq!(group.group_manifest()?.unwrap(), *next.marker()); + assert_receipt(&next.pin_cell(&history(1).cell).unwrap().1, 1, 2)?; + assert_receipt(&old_pin, 1, 2)?; + log.close().await?; + domain.shutdown().await; + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_registration.rs b/crates/cellar/tests/support/cell_group_registration.rs new file mode 100644 index 000000000..388338a6f --- /dev/null +++ b/crates/cellar/tests/support/cell_group_registration.rs @@ -0,0 +1,728 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Runtime registration uses real engines, catalog reads, TLS and shared SQL. +use super::remote_queries::{keys, wait_for}; +use super::*; +use std::sync::atomic::Ordering; +#[path = "raft_catalog_fault.rs"] +#[allow(dead_code)] +pub(super) mod catalog_fault; + +pub(super) struct Prepared { + directory: RangeDirectory, + pub(super) birth: CellGroupBirth, + store: Arc, + pub(super) inputs: Vec, + number: u8, +} + +fn class() -> Result> { + Ok(Arc::new( + RaftClass::new("Counter", 1, [81; 32])? + .initializer([83; 32], |_, tx| { + tx.execute_batch("CREATE TABLE value(n); INSERT INTO value VALUES(7);")?; + Ok(()) + })? + .command("Add", [82; 32], |ctx, input| { + ctx.sql().execute("UPDATE value SET n=n+1", [])?; + Ok(input.to_vec()) + })?, + )) +} + +pub(super) async fn prepare(f: &RuntimeCluster, number: u8, a: &CellHistory) -> Result { + let store = Arc::new(catalog_fault::Store::default()); + let old = f.cluster.birth.route(); + let scope = GroupHistory { + cluster: old.history.cluster, + namespace: NamespaceId::new([number; 16])?, + group: GroupId::new([number; 16])?, + incarnation: IncarnationId::new([number; 16])?, + }; + let catalog = Catalog::new(store.clone(), CatalogOptions::default())?; + let namespace = f + .cluster + .domain + .run(catalog.open_namespace(NamespaceConfig::new( + scope.cluster, + scope.namespace, + cellar_logic::NamespaceMode::Raft, + old.schema_digest, + ))) + .await??; + let directory = RangeDirectory::new(namespace, DirectoryLimits::default())?; + let birth = directory.prepare_group_birth(CellGroupBirthSpec { + history: scope, + range: CellRange::all(), + epoch: RouteEpoch::new(1)?, + creation: ProposalId::new([number; 16])?, + voters: (1..=3) + .map(|n| InitialMember { + member: MemberId::new(n).unwrap(), + node: NodeId::new([n as u8; 16]).unwrap(), + }) + .collect(), + learners: vec![], + })?; + f.cluster + .domain + .run(directory.reserve_group_birth(&birth)) + .await??; + let initialize = directory + .prepare_initialization(ProposalId::new([number + 1; 16])?, &[birth.route().clone()])?; + f.cluster + .domain + .run(directory.publish(&initialize)) + .await??; + let birth = f + .cluster + .domain + .run(directory.materialize_group_birth(&birth)) + .await??; + let mut prepared = Prepared { + directory, + birth, + store, + inputs: vec![], + number, + }; + prepared.inputs = recover(f, &prepared, a).await?; + Ok(prepared) +} + +async fn recover( + f: &RuntimeCluster, + prepared: &Prepared, + a: &CellHistory, +) -> Result> { + let mut parts = vec![]; + for n in 0..3 { + let permit = f + .cluster + .domain + .run( + prepared + .directory + .bootstrap_group_member(&prepared.birth, &f.cluster.voters[n].log), + ) + .await??; + let pages = ManifestPageStore::open( + &f.cluster + ._root + .path() + .join(format!("registered-{}-{n}", prepared.number)), + Arc::new(DirectFileSystem), + 1024, + 16 * 1024 * 1024, + 16, + )?; + let images = match f.cell_options_at(a, n).await? { + CellSqlOptions::Materialized(o) => CellGroupApplyImages::Materialized { + images: o.images, + services: o.query_services, + }, + CellSqlOptions::Extents(o) => CellGroupApplyImages::Extents { + images: o.images, + services: o.query_services, + limits: Default::default(), + cache_directory: o.cache_directory, + cache_pool: o.cache_pool, + cache: o.cache, + }, + }; + let (applier, current) = CellGroupApplier::recover( + permit.group().clone(), + pages, + images, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + )?; + parts.push((permit, current, applier)); + } + let route = f.tcp_route(prepared.birth.route().history)?; + Ok(parts + .into_iter() + .map(|(permit, current, applier)| CellGroupRegistration { + permit, + current, + applier, + transport: route.clone(), + }) + .collect()) +} + +async fn duplicate( + f: &RuntimeCluster, + prepared: &Prepared, + input: &CellGroupRegistration, + n: usize, +) -> Result { + Ok(CellGroupRegistration { + permit: f + .cluster + .domain + .run( + prepared + .directory + .bootstrap_group_member(&prepared.birth, &f.cluster.voters[n].log), + ) + .await??, + current: input.current.clone(), + applier: input.applier.clone(), + transport: input.transport.clone(), + }) +} + +pub(super) fn dispatcher( + f: &RuntimeCluster, + prepared: &Prepared, + class: Arc, +) -> Result { + Ok(CellGroupDispatcher::new( + &f.cluster.domain, + prepared.directory.clone(), + class, + keys(190)?, + CellDispatchLimits { + max_cached_routes: 32, + route_cache_bytes: 256 * 1024, + max_invocations: 8, + retained_bytes: 1024 * 1024, + max_input_bytes: 1024, + }, + )?) +} + +pub(super) async fn enroll( + f: &RuntimeCluster, + controller: &CellGroupController, + dispatch: &CellGroupDispatcher, + cell: &CellHistory, +) -> Result<()> { + let invocation = dispatch + .enroll( + cell.clone(), + CellEnrollmentRequest { + creation: ProposalId::new([219; 16])?, + durability: Durability::Quorum, + }, + CellRoutingLimits { + timeout: Duration::from_secs(10), + max_attempts: 2, + }, + None, + ) + .await?; + let enrolled = controller + .enroll(invocation, f.cell_options(cell).await?)? + .response() + .await?; + assert_eq!(enrolled.record().cell(), cell); + drop(enrolled); + Ok(()) +} + +async fn replicate( + f: &RuntimeCluster, + route: &GroupRoute, + cell: &CellHistory, + rooted: bool, +) -> Result<()> { + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(23);")?; + let accepted = f.runtimes[0] + .propose( + route.history, + apply::enrollment_bytes(route, cell, &bytes, rooted)?, + ) + .await?; + wait_for(async || { + for runtime in &f.runtimes { + if runtime.status(route.history).await?.applied.index < accepted.position.index { + return Ok(false); + } + } + Ok(true) + }) + .await +} + +pub(crate) async fn registration_live() -> Result<()> { + for rooted in [false, true] { + let class = class()?; + let (mut f, route, a, _) = Box::pin(registered::setup(rooted, &class)).await?; + let controllers = controller::start_with_limits(&mut f, controller::limits()).await?; + let mut prepared = prepare(&f, 31, &a).await?; + let duplicate = duplicate(&f, &prepared, &prepared.inputs[0], 0).await?; + let mut excess = prepare(&f, 41, &a).await?; + prepared + .store + .hold_group_head_reads + .store(true, Ordering::Release); + let calls = prepared + .inputs + .drain(..) + .enumerate() + .map(|(n, input)| controllers[n].register(input)) + .collect::, _>>()?; + wait_for(async || { + Ok(prepared + .store + .active_group_head_reads + .load(Ordering::Acquire) + == 3) + }) + .await?; + assert_eq!(controllers[0].usage().groups, 1); + assert_eq!(controllers[0].usage().pending_registrations, 1); + let rejected = controllers[0] + .register(duplicate) + .err() + .context("duplicate")?; + assert!(matches!( + rejected.error, + CellGroupControllerError::Runtime(RuntimeError::AlreadyAttached) + )); + drop(rejected); + let rejected = controllers[0] + .register(excess.inputs.remove(0)) + .err() + .context("pending capacity")?; + assert!(matches!( + rejected.error, + CellGroupControllerError::Runtime(RuntimeError::Overloaded) + )); + replicate( + &f, + &route, + &history(&route, "live-during-register")?, + rooted, + ) + .await?; + // Lost waiters cannot abandon the attachment or free its reserved slot. + drop(calls); + prepared + .store + .hold_group_head_reads + .store(false, Ordering::Release); + prepared.store.group_head_reads_resume.notify_waiters(); + wait_for(async || { + Ok(controllers.iter().all(|c| { + c.usage().groups == 2 + && c.usage().pending_registrations == 0 + && c.usage().retained_commands == 0 + })) + }) + .await?; + assert!(matches!( + controllers[0] + .register(*rejected.registration) + .err() + .context("active capacity")? + .error, + CellGroupControllerError::Runtime(RuntimeError::Overloaded) + )); + let group = prepared.birth.route().history; + f.runtimes[0].campaign(group).await?; + wait_for(async || Ok(f.runtimes[0].status(group).await?.ready)).await?; + let cell = history(prepared.birth.route(), "dynamic-enrollment")?; + let dispatch = dispatcher(&f, &prepared, class.clone())?; + enroll(&f, &controllers[0], &dispatch, &cell).await?; + enroll(&f, &controllers[0], &dispatch, &cell).await?; + for runtime in &f.runtimes { + wait_for(async || { + Ok(runtime + .reconciliation_authority(group, cell.clone()) + .await + .is_ok()) + }) + .await?; + assert_eq!( + runtime + .reconciliation_authority(group, cell.clone()) + .await? + .cell() + .version + .sequence + .get(), + 1 + ); + } + for transport in &f.transports { + assert_eq!(transport.usage().peers, 2); + } + for c in controllers { + c.close().await?; + } + f.close().await?; + } + Ok(()) +} + +pub(crate) async fn registration_refusals() -> Result<()> { + for rooted in [false, true] { + let class = class()?; + let (mut f, route, a, _) = Box::pin(registered::setup(rooted, &class)).await?; + let controllers = controller::start_with_limits(&mut f, controller::limits()).await?; + let mut prepared = prepare(&f, 31, &a).await?; + let mut wrong = duplicate(&f, &prepared, &prepared.inputs[0], 0).await?; + wrong.applier = prepared.inputs[1].applier.clone(); + assert!(controllers[0].register(wrong).is_err()); + let mut wrong = duplicate(&f, &prepared, &prepared.inputs[0], 0).await?; + wrong.current = prepared.inputs[1].current.clone(); + assert!(controllers[0].register(wrong).is_err()); + let mut wrong = duplicate(&f, &prepared, &prepared.inputs[0], 0).await?; + let pages = ManifestPageStore::open( + &f.cluster._root.path().join("foreign-page-owner"), + Arc::new(DirectFileSystem), + 1024, + 16 * 1024 * 1024, + 16, + )?; + let config = match f.cell_options(&a).await? { + CellSqlOptions::Materialized(o) => CellGroupApplyImages::Materialized { + images: o.images, + services: o.query_services, + }, + CellSqlOptions::Extents(o) => CellGroupApplyImages::Extents { + images: o.images, + services: o.query_services, + limits: Default::default(), + cache_directory: o.cache_directory, + cache_pool: o.cache_pool, + cache: o.cache, + }, + }; + wrong.applier = CellGroupApplier::new( + wrong.permit.group().clone(), + pages, + config, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + )?; + assert!(matches!( + controllers[0] + .register(wrong) + .err() + .context("foreign pages")? + .error, + CellGroupControllerError::Apply(_) + )); + // An already installed external route cannot be adopted or removed. + f.transports[0].install_physical_group(prepared.inputs[0].transport.clone())?; + let input = duplicate(&f, &prepared, &prepared.inputs[0], 0).await?; + assert!( + matches!(controllers[0].register(input)?.response().await, Err(e) if matches!(e.as_ref(),CellGroupControllerError::Transport(TransportError::Identity))) + ); + f.transports[0].remove_physical_group(&prepared.birth.route().history)?; + // Bootstrap revalidation fails after real local genesis; healthy groups + // remain usable and the failed registration's slot/route is released. + let input = duplicate(&f, &prepared, &prepared.inputs[0], 0).await?; + prepared + .store + .fail_directory_reads + .store(true, Ordering::Release); + assert!(controllers[0].register(input)?.response().await.is_err()); + prepared + .store + .fail_directory_reads + .store(false, Ordering::Release); + assert!(!controllers[0].usage().closing); + assert_eq!(controllers[0].usage().groups, 1); + assert_eq!(controllers[0].usage().pending_registrations, 0); + replicate( + &f, + &route, + &history(&route, "after-registration-failure")?, + rooted, + ) + .await?; + for (n, input) in prepared.inputs.drain(..).enumerate() { + controllers[n].register(input)?.response().await?; + } + f.runtimes[0] + .campaign(prepared.birth.route().history) + .await?; + wait_for(async || { + Ok(f.runtimes[0] + .status(prepared.birth.route().history) + .await? + .ready) + }) + .await?; + let dispatch = dispatcher(&f, &prepared, class)?; + enroll( + &f, + &controllers[0], + &dispatch, + &history(prepared.birth.route(), "after-retry")?, + ) + .await?; + for c in controllers { + c.close().await?; + } + f.close().await?; + } + Ok(()) +} + +pub(crate) async fn registration_close() -> Result<()> { + for rooted in [false, true] { + let class = class()?; + let (mut f, _, a, _) = Box::pin(registered::setup(rooted, &class)).await?; + let controllers = controller::start_with_limits(&mut f, controller::limits()).await?; + let mut prepared = prepare(&f, 31, &a).await?; + prepared + .store + .hold_group_head_reads + .store(true, Ordering::Release); + let call = controllers[0].register(prepared.inputs.remove(0))?; + wait_for(async || { + Ok(prepared + .store + .active_group_head_reads + .load(Ordering::Acquire) + == 1) + }) + .await?; + drop(call); + assert!( + tokio::time::timeout(Duration::from_millis(30), controllers[0].close()) + .await + .is_err() + ); + assert!(controllers[0].usage().closing); + assert_eq!(controllers[0].usage().pending_registrations, 1); + assert!(CellGroupRuntime::start( + f.cluster.voters[0].log.clone(), + RaftRuntimeOptions::default() + ) + .await + .is_err()); + prepared + .store + .hold_group_head_reads + .store(false, Ordering::Release); + prepared.store.group_head_reads_resume.notify_waiters(); + tokio::time::timeout(Duration::from_secs(5), controllers[0].wait()).await??; + assert_eq!(controllers[0].usage().pending_registrations, 0); + let (replacement, events) = CellGroupRuntime::start( + f.cluster.voters[0].log.clone(), + RaftRuntimeOptions::default(), + ) + .await?; + replacement.close().await?; + drop(events); + for c in controllers { + c.close().await?; + } + f.close().await?; + } + Ok(()) +} + +pub(super) async fn mutate( + f: &RuntimeCluster, + controller: &CellGroupController, + dispatch: &CellGroupDispatcher, + class: &RaftClass, + cell: &CellHistory, + retry: u64, +) -> Result<()> { + let invocation = dispatch + .command( + cell.clone(), + CellCommandRequest { + id: CommandId::new(ClientId::new([223; 16])?, 1)?, + contract: class.contract("Add").context("Add")?, + input: b"persisted".to_vec(), + durability: Durability::Quorum, + retry_until_ms: retry, + }, + CellRoutingLimits { + timeout: Duration::from_secs(10), + max_attempts: 2, + }, + None, + ) + .await?; + let response = controller + .command(invocation, f.cell_options(cell).await?)? + .response() + .await?; + assert_eq!( + response + .local_receipt() + .expect("local command receipt") + .receipt + .result(), + b"persisted" + ); + assert_eq!( + response + .local_receipt() + .expect("local command receipt") + .version + .sequence + .get(), + 2 + ); + drop(response); + Ok(()) +} + +pub(crate) async fn registration_recovery() -> Result<()> { + for rooted in [false, true] { + let class = class()?; + let (mut f, _, a, _) = Box::pin(registered::setup(rooted, &class)).await?; + let controllers = controller::start_with_limits(&mut f, controller::limits()).await?; + let mut prepared = prepare(&f, 31, &a).await?; + for (n, input) in prepared.inputs.drain(..).enumerate() { + controllers[n].register(input)?.response().await?; + } + let group = prepared.birth.route().history; + f.runtimes[0].campaign(group).await?; + wait_for(async || Ok(f.runtimes[0].status(group).await?.ready)).await?; + let cell = history(prepared.birth.route(), "recovered-registration")?; + let dispatch = dispatcher(&f, &prepared, class.clone())?; + enroll(&f, &controllers[0], &dispatch, &cell).await?; + let retry = f + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms() as u64 + 60_000 }) + .await?; + mutate(&f, &controllers[0], &dispatch, &class, &cell, retry).await?; + wait_for(async || { + for runtime in &f.runtimes { + if runtime + .reconciliation_authority(group, cell.clone()) + .await? + .cell() + .version + .sequence + .get() + != 2 + { + return Ok(false); + } + } + Ok(true) + }) + .await?; + for c in &controllers { + c.close().await?; + } + drop(controllers); + f.appliers.clear(); + f.transports.clear(); + f.runtimes.clear(); + for n in 0..3 { + f.cluster.voters[n].log.close().await?; + let number = n as u8 + 1; + let log = f + .cluster + .domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + f.cluster._root.path().join(format!("node-{number}/log")), + NodeId::new([number; 16])?, + DiskLineage::new([number + 10; 16])?, + ProcessGeneration::new([21; 16])?, + LogOpenMode::Resume, + ), + Arc::new(fault::FaultFileSystem { + gate: f.cluster.voters[n].gate.clone(), + }), + )) + .await??; + f.cluster.voters[n].log = log; + } + + // Reopen only the engine-selected complete closure, then register with + // empty controllers; registration alone still grants no quorum authority. + for n in 0..3 { + let (runtime, events) = CellGroupRuntime::start( + f.cluster.voters[n].log.clone(), + RaftRuntimeOptions { + tick_interval_ms: None, + election_seed: Some([71; 32]), + ..Default::default() + }, + ) + .await?; + f.transports + .push(super::super::tcp::start(&runtime, n).await?); + f.runtimes.push(runtime); + f.events.push(events); + } + prepared.inputs = recover(&f, &prepared, &a).await?; + let mut controllers = vec![]; + for (n, events) in f.events.drain(..).enumerate() { + let pool = CellGroupSqlPool::start( + &f.cluster.voters[n].log, + CellSqlLimits { + workers: 1, + queue_per_worker: 1, + max_pending_cells: 1, + group_pending_cells: 1, + node_bytes: 1024 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + }, + )?; + f.appliers.insert( + (n, group), + (pool.clone(), prepared.inputs[n].applier.clone()), + ); + let c = CellGroupController::start( + f.runtimes[n].clone(), + events, + f.transports[n].clone(), + pool, + vec![], + controller::limits(), + ) + .await?; + assert_eq!(c.usage().groups, 0); + controllers.push(c); + } + for (n, input) in prepared.inputs.drain(..).enumerate() { + controllers[n].register(input)?.response().await?; + } + assert!(matches!( + f.runtimes[0].execution_authority(group, cell.clone()).await, + Err(RuntimeError::NotLeader) + )); + f.runtimes[0].campaign(group).await?; + wait_for(async || Ok(f.runtimes[0].status(group).await?.ready)).await?; + mutate(&f, &controllers[0], &dispatch, &class, &cell, retry).await?; + let deadline = f + .cluster + .domain + .run(async { cellar::asyncrt::mono_ms() + 5_000 }) + .await?; + let authority = f.runtimes[0] + .read_authority( + group, + cell.clone(), + CellReadConsistency::Linearizable, + deadline, + ) + .await?; + let output = f.appliers[&(0, group)] + .0 + .query(f.cell_options(&cell).await?, authority, |db, version| { + assert_eq!(version.sequence.get(), 2); + let n: i64 = db.query_row("SELECT n FROM value", [], |row| row.get(0))?; + Ok(n.to_le_bytes().to_vec()) + }) + .await?; + assert_eq!(output.result(), &8_i64.to_le_bytes()); + drop(output); + for c in controllers { + c.close().await?; + } + f.close().await?; + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_remote_commands.rs b/crates/cellar/tests/support/cell_group_remote_commands.rs new file mode 100644 index 000000000..391f66db0 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_remote_commands.rs @@ -0,0 +1,393 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Real TLS ingress never inherits write authority from read grants or membership. +use super::query_origins::Router; +use super::remote_queries::{dispatcher, keys, wait_for}; +use super::*; +use cellar::application::consensus::NamespaceCommandOrigin; +use std::sync::{ + atomic::{AtomicUsize, Ordering}, + Mutex, +}; + +async fn invocation( + dispatch: &CellGroupDispatcher, + class: &RaftClass, + cell: &CellHistory, + retry_until: u64, + input: &[u8], +) -> Result { + let mut command = dispatch + .command( + cell.clone(), + CellCommandRequest { + id: CommandId::new(ClientId::new([211; 16])?, 1)?, + contract: class.contract("Add").context("Add")?, + input: input.to_vec(), + durability: Durability::Quorum, + retry_until_ms: retry_until, + }, + CellRoutingLimits { + timeout: Duration::from_secs(10), + max_attempts: 8, + }, + None, + ) + .await?; + command.begin().await?; + Ok(command) +} + +fn grant(fixture: &RuntimeCluster, router: &Router, revision: u64) -> Result<()> { + let history = fixture.cluster.birth.route().history; + fixture.transports[0].grant_command_origin(NamespaceCommandOrigin { + cluster: history.cluster, + namespace: history.namespace, + revision, + peer: router.peer.clone(), + })?; + Ok(()) +} + +pub(crate) async fn remote_commands() -> Result<()> { + for rooted in [false, true] { + let calls = Arc::new(AtomicUsize::new(0)); + let count = calls.clone(); + let class = Arc::new(RaftClass::new("Counter", 1, [81; 32])?.command( + "Add", + [82; 32], + move |ctx, input| { + count.fetch_add(1, Ordering::SeqCst); + ctx.sql().execute("UPDATE value SET n=n+1", [])?; + let mut result = ctx.inputs().random.to_vec(); + result.extend_from_slice(&ctx.inputs().now_unix_millis.to_be_bytes()); + result.extend_from_slice(input); + Ok(result) + }, + )?); + let (mut fixture, route, a, _) = Box::pin(registered::setup(rooted, &class)).await?; + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let dispatch = dispatcher(&fixture, class.clone(), keys(190)?)?; + let options = fixture.cell_options(&a).await?; + assert!(matches!( + fixture.transports[0].install_cell_command_gateway( + dispatch.clone(), + controllers[1].clone(), + cellar::application::consensus::verification::cell_groups::CellGroupNamespaceStorage::new( + options.clone() + )?, + ), + Err(TransportError::Identity) + )); + let gateway = fixture.transports[0].install_cell_command_gateway( + dispatch.clone(), + controllers[0].clone(), + cellar::application::consensus::verification::cell_groups::CellGroupNamespaceStorage::new(options)?, + )?; + let router = Router::start(&fixture).await?; + router + .transport + .install_cell_query_group(fixture.tcp_route(route.history)?)?; + let retry_until = fixture + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms() as u64 + 60_000 }) + .await?; + let mut command = invocation(&dispatch, &class, &a, retry_until, b"once").await?; + let deadline = command.attempt().unwrap().deadline_ms(); + assert!(command + .execute_remote(&router.transport, MemberId::new(1)?) + .await + .is_err()); + assert_eq!(calls.load(Ordering::SeqCst), 0); + let mut wrong_process = router.peer.clone(); + wrong_process.process = ProcessGeneration::new([213; 16])?; + fixture.transports[0].grant_command_origin(NamespaceCommandOrigin { + cluster: a.cluster, + namespace: a.namespace, + revision: 1, + peer: wrong_process, + })?; + command.retry().await?; + assert!(command + .execute_remote(&router.transport, MemberId::new(1)?) + .await + .is_err()); + assert_eq!(calls.load(Ordering::SeqCst), 0); + fixture.transports[0].revoke_command_origin(a.cluster, a.namespace, router.peer.node, 2)?; + router.grant(&fixture, 1)?; + command.retry().await?; + assert!(command + .execute_remote(&router.transport, MemberId::new(1)?) + .await + .is_err()); + assert_eq!(calls.load(Ordering::SeqCst), 0); + // Voter membership also does not confer write permission. + let mut voter = invocation(&dispatch, &class, &a, retry_until, b"once").await?; + assert!(voter + .execute_remote(&fixture.transports[1], MemberId::new(1)?) + .await + .is_err()); + drop(voter); + fixture.transports[0].grant_command_origin(NamespaceCommandOrigin { + cluster: a.cluster, + namespace: NamespaceId::new([212; 16])?, + revision: 1, + peer: router.peer.clone(), + })?; + command.retry().await?; + assert!(command + .execute_remote(&router.transport, MemberId::new(1)?) + .await + .is_err()); + assert_eq!(calls.load(Ordering::SeqCst), 0); + grant(&fixture, &router, 3)?; + command.retry().await?; + assert_eq!(command.attempt().unwrap().deadline_ms(), deadline); + fixture.transports[1].grant_command_origin(NamespaceCommandOrigin { + cluster: a.cluster, + namespace: a.namespace, + revision: 1, + peer: router.peer.clone(), + })?; + let follower_gateway = fixture.transports[1].install_cell_command_gateway( + dispatch.clone(), + controllers[1].clone(), + cellar::application::consensus::verification::cell_groups::CellGroupNamespaceStorage::new( + fixture.cell_options_at(&a, 1).await?, + )?, + )?; + let mut follower = invocation(&dispatch, &class, &a, retry_until, b"once").await?; + let refusal = follower + .execute_remote(&router.transport, MemberId::new(2)?) + .await + .err() + .context("follower refuses")?; + assert!( + matches!( + refusal.cause.as_ref(), + CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::NotLeader { .. } + )) + ), + "{refusal:?}" + ); + assert_eq!(calls.load(Ordering::SeqCst), 0); + drop(follower); + let result = command + .execute_remote(&router.transport, MemberId::new(1)?) + .await?; + let bytes = result.result().to_vec(); + assert_eq!( + result.command(), + CommandId::new(ClientId::new([211; 16])?, 1)? + ); + assert_eq!(result.retry_until_ms(), retry_until); + assert_eq!(result.observation().version.history, a); + assert_eq!(result.observation().version.sequence.get(), 2); + assert_eq!(result.durability(), Durability::Quorum); + assert_eq!(calls.load(Ordering::SeqCst), 1); + assert_eq!(router.transport.usage().query_items, 1); + assert!(router.transport.usage().query_bytes > bytes.len()); + drop(result); + command.retry().await?; + assert_eq!( + command + .execute_remote(&router.transport, MemberId::new(1)?) + .await? + .result(), + bytes + ); + assert_eq!(calls.load(Ordering::SeqCst), 1); + let mut conflict = invocation(&dispatch, &class, &a, retry_until, b"other").await?; + let error = conflict + .execute_remote(&router.transport, MemberId::new(1)?) + .await + .err() + .context("conflicting receipt")?; + assert!(error.ambiguous); + assert!(matches!( + *error.cause, + CellDispatchError::Execution(ExecutionError::Refused(CommandFailure::CommandConflict)) + )); + drop(conflict); + assert_eq!(calls.load(Ordering::SeqCst), 1); + + // Queue through the SDK's controller entry point on a node that owns + // no group. The first peer is a live follower, so discovery must reach + // the actual leader before sending a single mutation. + let mut discovery = fixture.tcp_route(route.history)?; + discovery.revision += 1; + discovery.peers.swap(0, 1); + router.transport.install_cell_query_group(discovery)?; + let sql = CellGroupSqlPool::start( + &router.log, + fixture.appliers[&(0, route.history)].0.usage()?.limits, + )?; + let ingress = CellGroupController::start( + router.runtime, + router.events, + router.transport.clone(), + sql, + vec![], + controller::limits(), + ) + .await?; + let options = fixture.cell_options(&a).await?; + let request = CellCommandRequest { + id: CommandId::new(ClientId::new([211; 16])?, 2)?, + contract: class.contract("Add").context("Add")?, + input: b"routed-once".to_vec(), + durability: Durability::Quorum, + retry_until_ms: retry_until, + }; + let routing = CellRoutingLimits { + timeout: Duration::from_secs(5), + max_attempts: 3, + }; + let response = ingress + .dispatch_command( + &dispatch, + a.clone(), + request.clone(), + routing, + options.clone(), + )? + .response() + .await?; + assert!(response.local_receipt().is_none()); + assert_eq!(response.observation().version.history, a); + assert_eq!(response.observation().version.sequence.get(), 3); + assert!(response.result().ends_with(b"routed-once")); + assert_eq!(calls.load(Ordering::SeqCst), 2); + let saved = response.result().to_vec(); + drop(response); + // The receiver bounds its idle header read. The sender must discard + // the expired socket before it sends the next probe or command. + tokio::time::sleep(Duration::from_millis( + TransportOptions::default().io_timeout_ms + 250, + )) + .await; + assert_eq!( + ingress + .dispatch_command(&dispatch, a.clone(), request, routing, options)? + .response() + .await + .context("exact retry after query connection idle expiry")? + .result(), + saved + ); + assert_eq!(calls.load(Ordering::SeqCst), 2); + assert_eq!(ingress.usage().groups, 0); + drop(follower_gateway); + drop(gateway); + command.retry().await?; + assert!(command + .execute_remote(&router.transport, MemberId::new(1)?) + .await + .is_err()); + drop(command); + assert!(router.log.cell_groups()?.is_empty()); + assert_eq!(dispatch.usage(), (0, 0)); + ingress.close().await?; + router.log.close().await?; + for controller in controllers { + controller.close().await?; + } + fixture.close().await?; + } + Ok(()) +} + +pub(crate) async fn remote_command_lifecycle(revoke: bool) -> Result<()> { + for rooted in [false, true] { + let calls = Arc::new(AtomicUsize::new(0)); + let count = calls.clone(); + let (entered, entered_rx) = tokio::sync::oneshot::channel(); + let entered = Arc::new(Mutex::new(Some(entered))); + let (release, held) = std::sync::mpsc::sync_channel(1); + let held = Arc::new(Mutex::new(held)); + let class = Arc::new(RaftClass::new("Counter", 1, [81; 32])?.command( + "Add", + [82; 32], + move |ctx, _| { + if count.fetch_add(1, Ordering::SeqCst) == 0 { + let _ = entered.lock().unwrap().take().context("signal")?.send(()); + held.lock().unwrap().recv_timeout(Duration::from_secs(15))?; + } + ctx.sql().execute("UPDATE value SET n=n+1", [])?; + Ok(b"committed".to_vec()) + }, + )?); + let (mut fixture, route, a, _) = Box::pin(registered::setup(rooted, &class)).await?; + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let dispatch = dispatcher(&fixture, class.clone(), keys(190)?)?; + let gateway = fixture.transports[0].install_cell_command_gateway( + dispatch.clone(), + controllers[0].clone(), + cellar::application::consensus::verification::cell_groups::CellGroupNamespaceStorage::new( + fixture.cell_options(&a).await?, + )?, + )?; + let router = Router::start(&fixture).await?; + router + .transport + .install_cell_query_group(fixture.tcp_route(route.history)?)?; + // No read-origin grant is installed: write permission is independent. + grant(&fixture, &router, 1)?; + let retry_until = fixture + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms() as u64 + 60_000 }) + .await?; + let mut command = invocation(&dispatch, &class, &a, retry_until, b"once").await?; + let transport = router.transport.clone(); + let task = tokio::spawn(async move { + command + .execute_remote(&transport, MemberId::new(1).unwrap()) + .await + }); + tokio::time::timeout(Duration::from_secs(5), entered_rx).await??; + assert_eq!(controllers[0].usage().retained_commands, 1); + if revoke { + fixture.transports[0].revoke_command_origin( + a.cluster, + a.namespace, + router.peer.node, + 2, + )?; + let error = tokio::time::timeout(Duration::from_secs(5), task) + .await?? + .err() + .context("revoked reply")?; + assert!(error.ambiguous); + assert!(grant(&fixture, &router, 1).is_err()); + } else { + task.abort(); + assert!(task.await.err().context("cancelled task")?.is_cancelled()); + } + wait_for(async || Ok(router.transport.usage().query_items == 0)).await?; + assert_eq!(controllers[0].usage().retained_commands, 1); + release.send(())?; + wait_for(async || Ok(controllers[0].usage().retained_commands == 0)).await?; + if revoke { + grant(&fixture, &router, 3)?; + } + let mut retry = invocation(&dispatch, &class, &a, retry_until, b"once").await?; + let result = retry + .execute_remote(&router.transport, MemberId::new(1)?) + .await?; + assert_eq!(result.result(), b"committed"); + assert_eq!(result.observation().version.sequence.get(), 2); + assert_eq!(calls.load(Ordering::SeqCst), 1); + drop(result); + drop(retry); + drop(gateway); + router.close().await?; + for controller in controllers { + controller.close().await?; + } + fixture.close().await?; + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_remote_enrollment.rs b/crates/cellar/tests/support/cell_group_remote_enrollment.rs new file mode 100644 index 000000000..7b3b6c742 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_remote_enrollment.rs @@ -0,0 +1,352 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Enrollment traverses real TLS and the shared controller without router replicas. +use super::query_origins::Router; +use super::remote_queries::{dispatcher, keys, wait_for}; +use super::*; +use cellar::application::consensus::NamespaceCommandOrigin; +use std::sync::{ + atomic::{AtomicUsize, Ordering}, + Mutex, +}; + +async fn invocation( + dispatch: &CellGroupDispatcher, + cell: &CellHistory, + tag: u8, + timeout: Duration, +) -> Result { + let mut call = dispatch + .enroll( + cell.clone(), + CellEnrollmentRequest { + creation: ProposalId::new([tag; 16])?, + durability: Durability::Quorum, + }, + CellRoutingLimits { + timeout, + max_attempts: 8, + }, + None, + ) + .await?; + call.begin().await?; + Ok(call) +} + +fn grant(fixture: &RuntimeCluster, router: &Router, index: usize, revision: u64) -> Result<()> { + let history = fixture.cluster.birth.route().history; + fixture.transports[index].grant_command_origin(NamespaceCommandOrigin { + cluster: history.cluster, + namespace: history.namespace, + revision, + peer: router.peer.clone(), + })?; + Ok(()) +} + +pub(crate) async fn remote_enrollment() -> Result<()> { + for rooted in [false, true] { + let calls = Arc::new(AtomicUsize::new(0)); + let count = calls.clone(); + let class = Arc::new( + RaftClass::new("Counter", 1, [81; 32])? + .initializer([90; 32], move |_, tx| { + count.fetch_add(1, Ordering::SeqCst); + tx.execute_batch("CREATE TABLE value(n); INSERT INTO value VALUES(7);")?; + Ok(()) + })? + .command("Add", [82; 32], |ctx, _| { + ctx.sql().execute("UPDATE value SET n=n+1", [])?; + Ok(b"updated".to_vec()) + })?, + ); + let (mut fixture, route, _, _) = Box::pin(registered::setup(rooted, &class)).await?; + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let dispatch = dispatcher(&fixture, class.clone(), keys(190)?)?; + let cell = history(&route, "remote-enrollment")?; + let gateway = fixture.transports[0].install_cell_command_gateway( + dispatch.clone(), + controllers[0].clone(), + cellar::application::consensus::verification::cell_groups::CellGroupNamespaceStorage::new( + fixture.cell_options(&cell).await?, + )?, + )?; + let router = Router::start(&fixture).await?; + router + .transport + .install_cell_query_group(fixture.tcp_route(route.history)?)?; + let mut call = invocation(&dispatch, &cell, 215, Duration::from_secs(15)).await?; + let deadline = call.attempt().unwrap().deadline_ms(); + assert!(call + .execute_remote(&router.transport, MemberId::new(1)?) + .await + .is_err()); + router.grant(&fixture, 1)?; + call.retry().await?; + assert!(call + .execute_remote(&router.transport, MemberId::new(1)?) + .await + .is_err()); + assert_eq!(calls.load(Ordering::SeqCst), 0); + let mut voter = invocation(&dispatch, &cell, 215, Duration::from_secs(15)).await?; + assert!(voter + .execute_remote(&fixture.transports[1], MemberId::new(1)?) + .await + .is_err()); + drop(voter); + grant(&fixture, &router, 0, 1)?; + grant(&fixture, &router, 1, 1)?; + let follower_gateway = fixture.transports[1].install_cell_command_gateway( + dispatch.clone(), + controllers[1].clone(), + cellar::application::consensus::verification::cell_groups::CellGroupNamespaceStorage::new( + fixture.cell_options_at(&cell, 1).await?, + )?, + )?; + let mut follower = invocation(&dispatch, &cell, 215, Duration::from_secs(15)).await?; + let error = follower + .execute_remote(&router.transport, MemberId::new(2)?) + .await + .err() + .context("follower refuses")?; + assert!( + matches!( + error.cause.as_ref(), + CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::NotLeader { .. } + )) + ), + "{error:?}" + ); + assert_eq!(calls.load(Ordering::SeqCst), 0); + drop(follower); + drop(follower_gateway); + call.retry().await?; + assert_eq!(call.attempt().unwrap().deadline_ms(), deadline); + let response = call + .execute_remote(&router.transport, MemberId::new(1)?) + .await?; + let original = response.record().clone(); + assert_eq!(original.cell(), &cell); + assert_eq!(original.creation(), ProposalId::new([215; 16])?); + assert_eq!(original.schema(), class.schema_digest()); + assert_eq!(response.served_by(), MemberId::new(1)?); + assert_eq!(response.durability(), Durability::Quorum); + assert_eq!(router.transport.usage().query_items, 1); + assert!(router.transport.usage().query_bytes >= 8192); + assert!(call + .execute_remote(&router.transport, MemberId::new(1)?) + .await + .is_err()); + drop(response); + // Mutate through the same remote write gateway before retrying creation. + let retry_until = fixture + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms() as u64 + 60_000 }) + .await?; + let mut command = dispatch + .command( + cell.clone(), + CellCommandRequest { + id: CommandId::new(ClientId::new([216; 16])?, 1)?, + contract: class.contract("Add").context("Add")?, + input: vec![], + durability: Durability::Quorum, + retry_until_ms: retry_until, + }, + CellRoutingLimits { + timeout: Duration::from_secs(15), + max_attempts: 4, + }, + None, + ) + .await?; + command.begin().await?; + let updated = command + .execute_remote(&router.transport, MemberId::new(1)?) + .await?; + assert_eq!(updated.observation().version.sequence.get(), 2); + drop(updated); + drop(command); + call.retry().await?; + assert_eq!( + call.execute_remote(&router.transport, MemberId::new(1)?) + .await? + .record(), + &original + ); + assert_eq!(calls.load(Ordering::SeqCst), 1); + for new_incarnation in [false, true] { + let mut other = cell.clone(); + if new_incarnation { + other.incarnation = IncarnationId::new([217; 16])?; + } + let mut conflict = invocation(&dispatch, &other, 218, Duration::from_secs(15)).await?; + let error = conflict + .execute_remote(&router.transport, MemberId::new(1)?) + .await + .err() + .context("creation conflict")?; + assert!(error.ambiguous); + assert!( + matches!( + error.cause.as_ref(), + CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::CommandConflict + )) + ), + "{error:?}" + ); + } + assert_eq!(calls.load(Ordering::SeqCst), 1); + // A routed/possibly transmitted invocation is never fresh local work. + assert!(controllers[0] + .enroll(call, fixture.cell_options(&cell).await?) + .is_err()); + let mut late = invocation(&dispatch, &cell, 215, Duration::from_secs(15)).await?; + drop(gateway); + assert!(late + .execute_remote(&router.transport, MemberId::new(1)?) + .await + .is_err()); + drop(late); + assert!(router.log.cell_groups()?.is_empty()); + assert_eq!(dispatch.usage(), (0, 0)); + router.close().await?; + for controller in controllers { + controller.close().await?; + } + fixture.close().await?; + } + Ok(()) +} + +// Disconnect/revocation hold entered initialization. Timeout holds committed +// manifest publication, so a response deadline cannot seal a committed creation. +pub(crate) async fn remote_enrollment_lifecycle(mode: u8) -> Result<()> { + for rooted in [false, true] { + let root = tempfile::tempdir()?; + let faults = + super::apply_faults::physical::Faults::new(root.path().join("node-1/pages"), None); + let filesystem = Arc::new(super::apply_faults::physical::FaultFileSystem { + faults: faults.clone(), + }); + let calls = Arc::new(AtomicUsize::new(0)); + let count = calls.clone(); + let (entered, entered_rx) = tokio::sync::oneshot::channel(); + let entered = Arc::new(Mutex::new(Some(entered))); + let (release, held) = std::sync::mpsc::sync_channel(1); + let held = Arc::new(Mutex::new(held)); + let init_entered = entered.clone(); + let init_held = held.clone(); + let class = Arc::new(RaftClass::new("Counter", 1, [81; 32])?.initializer( + [90; 32], + move |_, tx| { + if count.fetch_add(1, Ordering::SeqCst) == 0 && mode != 2 { + let _ = init_entered + .lock() + .unwrap() + .take() + .context("signal")? + .send(()); + init_held + .lock() + .unwrap() + .recv_timeout(Duration::from_secs(15))?; + } + tx.execute_batch("CREATE TABLE value(n); INSERT INTO value VALUES(7);")?; + Ok(()) + }, + )?); + let (mut fixture, route, _, _) = + Box::pin(registered::setup_in(rooted, &class, root, filesystem)).await?; + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let dispatch = dispatcher(&fixture, class, keys(190)?)?; + let cell = history(&route, "remote-enrollment-lifecycle")?; + let gateway = fixture.transports[0].install_cell_command_gateway( + dispatch.clone(), + controllers[0].clone(), + cellar::application::consensus::verification::cell_groups::CellGroupNamespaceStorage::new( + fixture.cell_options(&cell).await?, + )?, + )?; + let router = Router::start(&fixture).await?; + router + .transport + .install_cell_query_group(fixture.tcp_route(route.history)?)?; + grant(&fixture, &router, 0, 1)?; + if mode == 2 { + *faults.before_directory_sync.lock().unwrap() = Some(Box::new(move || { + let _ = entered + .lock() + .unwrap() + .take() + .expect("apply entered") + .send(()); + held.lock() + .unwrap() + .recv_timeout(Duration::from_secs(15)) + .expect("release apply"); + })); + } + let timeout = if mode == 2 { + Duration::from_millis(800) + } else { + Duration::from_secs(15) + }; + let mut call = invocation(&dispatch, &cell, 219, timeout).await?; + let transport = router.transport.clone(); + let task = tokio::spawn(async move { + call.execute_remote(&transport, MemberId::new(1).unwrap()) + .await + }); + tokio::time::timeout(Duration::from_secs(5), entered_rx).await??; + assert_eq!(controllers[0].usage().retained_commands, 1); + if mode == 0 { + task.abort(); + assert!(task.await.err().context("cancelled task")?.is_cancelled()); + } else { + if mode == 1 { + fixture.transports[0].revoke_command_origin( + cell.cluster, + cell.namespace, + router.peer.node, + 2, + )?; + } + let error = tokio::time::timeout(Duration::from_secs(5), task) + .await?? + .err() + .context("lost response")?; + assert!(error.ambiguous); + } + wait_for(async || Ok(router.transport.usage().query_items == 0)).await?; + assert_eq!(controllers[0].usage().retained_commands, 1); + release.send(())?; + wait_for(async || Ok(controllers[0].usage().retained_commands == 0)).await?; + if mode == 1 { + grant(&fixture, &router, 0, 3)?; + } + let state = fixture.runtimes[0] + .enrollment_state(route.history, cell.clone()) + .await?; + let original = state.record().context("committed enrollment")?.clone(); + let mut retry = invocation(&dispatch, &cell, 219, Duration::from_secs(15)).await?; + let result = retry + .execute_remote(&router.transport, MemberId::new(1)?) + .await?; + assert_eq!(result.record(), &original); + assert_eq!(calls.load(Ordering::SeqCst), 1); + drop(result); + drop(retry); + drop(gateway); + router.close().await?; + for controller in controllers { + controller.close().await?; + } + fixture.close().await?; + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_remote_queries.rs b/crates/cellar/tests/support/cell_group_remote_queries.rs new file mode 100644 index 000000000..8aac775e4 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_remote_queries.rs @@ -0,0 +1,585 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Shared catalog dispatch over the production query TLS lane and SQL owners. +use super::*; +use std::sync::{ + atomic::{AtomicUsize, Ordering}, + Mutex, +}; + +pub(super) fn dispatcher( + fixture: &RuntimeCluster, + class: Arc, + keys: VersionKeys, +) -> Result { + Ok(CellGroupDispatcher::new( + &fixture.cluster.domain, + fixture.cluster.directory.clone(), + class, + keys, + CellDispatchLimits { + max_cached_routes: 32, + route_cache_bytes: 256 * 1024, + max_invocations: 8, + retained_bytes: 1024 * 1024, + max_input_bytes: 1024, + }, + )?) +} + +pub(super) fn keys(tag: u8) -> Result { + Ok(VersionKeys::new( + VersionSigningKey::new([tag; 32], u64::MAX)?, + vec![], + Duration::from_secs(30), + )?) +} + +pub(super) async fn invocation( + dispatcher: &CellGroupDispatcher, + class: &RaftClass, + cell: &CellHistory, + consistency: CellQueryConsistency, + timeout: Duration, + input: &[u8], +) -> Result { + let mut query = dispatcher.query( + cell.clone(), + CellQueryRequest { + invocation: ProposalId::new([197; 16])?, + contract: class.contract("Get").context("Get")?, + input: input.to_vec(), + consistency, + }, + CellRoutingLimits { + timeout, + max_attempts: 2, + }, + None, + )?; + query.begin().await?; + Ok(query) +} + +pub(super) async fn gateways( + fixture: &RuntimeCluster, + controllers: &[CellGroupController], + class: Arc, +) -> Result> { + let mut gateways = Vec::new(); + let cell = history(fixture.cluster.birth.route(), "query-options")?; + for n in 0..3 { + gateways.push(fixture.transports[n].install_cell_query_gateway( + dispatcher(fixture, class.clone(), keys(190)?)?, + controllers[n].clone(), + cellar::application::consensus::verification::cell_groups::CellGroupNamespaceStorage::new( + fixture.cell_options_at(&cell, n).await?, + )?, + )?); + } + Ok(gateways) +} + +pub(super) async fn wait_for(mut predicate: impl AsyncFnMut() -> Result) -> Result<()> { + tokio::time::timeout(Duration::from_secs(10), async { + while !predicate().await? { + tokio::time::sleep(Duration::from_millis(2)).await; + } + Ok::<_, anyhow::Error>(()) + }) + .await? +} + +pub(crate) async fn remote_queries() -> Result<()> { + for rooted in [false, true] { + let calls = Arc::new(AtomicUsize::new(0)); + let count = calls.clone(); + let class = Arc::new(RaftClass::new("Counter", 1, [81; 32])?.query( + "Get", + [82; 32], + move |context, _| { + count.fetch_add(1, Ordering::SeqCst); + let QueryVersion::Cell(observation) = context.version() else { + anyhow::bail!("physical group cut"); + }; + assert_eq!(context.identity(), &observation.version.history); + assert_eq!( + context.query_row("SELECT n FROM value", [], |r| r.get::<_, i64>(0))?, + 0 + ); + Ok(context.identity().cell.as_str().as_bytes().to_vec()) + }, + )?); + let (mut fixture, route, a, b) = Box::pin(registered::setup(rooted, &class)).await?; + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let mut gateways = gateways(&fixture, &controllers, class.clone()).await?; + let dispatch = dispatcher(&fixture, class.clone(), keys(190)?)?; + assert_eq!(gateways[0].namespace(), (a.cluster, a.namespace)); + let options = fixture.cell_options(&a).await?; + assert!(matches!( + fixture.transports[0].install_cell_query_gateway( + dispatch.clone(), + controllers[0].clone(), + cellar::application::consensus::verification::cell_groups::CellGroupNamespaceStorage::new( + options.clone() + )?, + ), + Err(TransportError::Refused) + )); + assert!(matches!( + fixture.transports[0].install_cell_query_gateway( + dispatch.clone(), + controllers[1].clone(), + cellar::application::consensus::verification::cell_groups::CellGroupNamespaceStorage::new( + options.clone() + )?, + ), + Err(TransportError::Identity) + )); + let source = &fixture.transports[1]; + let leader = MemberId::new(1)?; + let mut first = invocation( + &dispatch, + &class, + &a, + CellQueryConsistency::Linearizable, + Duration::from_secs(5), + b"", + ) + .await?; + let attempt = first.attempt().context("attempt")?.clone(); + let response = first.execute_remote(source, leader).await?; + assert_eq!(response.result(), a.cell.as_str().as_bytes()); + assert_eq!(response.observation().version.history, a); + assert_eq!(response.observation().version.sequence.get(), 1); + assert_eq!(response.served_by(), leader); + let token = response.token().clone(); + let expected = response.observation().clone(); + assert_eq!(source.usage().query_items, 1); + assert!(source.usage().query_bytes > response.result().len()); + assert!(matches!( + first.execute_remote(source, leader).await, + Err(CellDispatchError::Invalid("consumed route attempt")) + )); + drop(response); + assert_eq!(source.usage().query_items, 0); + first.retry().await?; + assert_eq!( + first.attempt().context("retry")?.deadline_ms(), + attempt.deadline_ms() + ); + assert_eq!(first.attempt().context("retry")?.ordinal(), 2); + assert_eq!( + *first.execute_remote(source, leader).await?.observation(), + expected + ); + assert!(matches!( + first.retry().await, + Err(CellDispatchError::Routing(CellRoutingError { + reason: CellRoutingFailure::AttemptsExhausted, + .. + })) + )); + drop(first); + for (cell, member, consistency) in [ + (&a, leader, CellQueryConsistency::AtLeast(token.clone())), + (&b, MemberId::new(3)?, CellQueryConsistency::Stale), + ] { + let mut query = invocation( + &dispatch, + &class, + cell, + consistency, + Duration::from_secs(5), + b"", + ) + .await?; + let response = query.execute_remote(source, member).await?; + assert_eq!(response.result(), cell.cell.as_str().as_bytes()); + assert_eq!(response.observation().version.history, *cell); + assert_eq!(response.observation().version.sequence.get(), 1); + } + // Concurrent ingress shares one FIFO peer lane. Wire sequences must be + // assigned by its sender, not racing callers before channel admission. + for _ in 0..8 { + let barrier = Arc::new(tokio::sync::Barrier::new(2)); + let mut waiters = Vec::new(); + for cell in [&a, &b] { + let cell = cell.clone(); + let source = source.clone(); + let dispatch = dispatch.clone(); + let class = class.clone(); + let barrier = barrier.clone(); + waiters.push(fixture.cluster.domain.spawn(async move { + let mut query = invocation( + &dispatch, + &class, + &cell, + CellQueryConsistency::Stale, + Duration::from_secs(5), + b"", + ) + .await?; + barrier.wait().await; + let response = query.execute_remote(&source, leader).await?; + assert_eq!(response.result(), cell.cell.as_str().as_bytes()); + assert_eq!(response.observation().version.history, cell); + Ok::<_, anyhow::Error>(()) + })?); + } + for waiter in waiters { + waiter.await??; + } + } + assert_eq!(source.usage().query_items, 0); + let before = calls.load(Ordering::SeqCst); + let mut wrong = invocation( + &dispatch, + &class, + &b, + CellQueryConsistency::AtLeast(token), + Duration::from_secs(5), + b"", + ) + .await?; + assert!(matches!( + wrong.execute_remote(source, leader).await, + Err(CellDispatchError::Transport(TransportError::Query( + CommandFailure::WrongIncarnation + ))) + )); + drop(wrong); + let mut wrong_cell = a.clone(); + wrong_cell.incarnation = IncarnationId::new([199; 16])?; + let mut wrong = invocation( + &dispatch, + &class, + &wrong_cell, + CellQueryConsistency::Stale, + Duration::from_secs(5), + b"", + ) + .await?; + assert!(wrong.execute_remote(source, leader).await.is_err()); + drop(wrong); + assert_eq!(calls.load(Ordering::SeqCst), before); + // A signature under a different namespace key cannot become a read result. + let other_keys = dispatcher(&fixture, class.clone(), keys(191)?)?; + let mut wrong = invocation( + &other_keys, + &class, + &a, + CellQueryConsistency::Stale, + Duration::from_secs(5), + b"", + ) + .await?; + assert!(matches!( + wrong.execute_remote(source, leader).await, + Err(CellDispatchError::Transport(TransportError::Authentication)) + )); + drop(wrong); + // Dropping the host registration refuses new reads; transport does not + // secretly retain a public handle or keep an abandoned gateway alive. + drop(gateways.remove(0)); + let before = calls.load(Ordering::SeqCst); + let mut absent = invocation( + &dispatch, + &class, + &a, + CellQueryConsistency::Stale, + Duration::from_secs(5), + b"", + ) + .await?; + assert!(absent.execute_remote(source, leader).await.is_err()); + assert_eq!(calls.load(Ordering::SeqCst), before); + let replacement = fixture.transports[0].install_cell_query_gateway( + dispatch.clone(), + controllers[0].clone(), + cellar::application::consensus::verification::cell_groups::CellGroupNamespaceStorage::new(options)?, + )?; + absent.retry().await?; + assert_eq!( + absent.execute_remote(source, leader).await?.result(), + a.cell.as_str().as_bytes() + ); + drop(absent); + assert_eq!(dispatch.usage(), (0, 0)); + // Change only the catalog, not physical state: the old destination must + // refuse the captured route. This is not a live-movement qualification. + let mut stale = invocation( + &dispatch, + &class, + &a, + CellQueryConsistency::Stale, + Duration::from_secs(5), + b"", + ) + .await?; + let original_deadline = stale.attempt().context("original")?.deadline_ms(); + let directory = fixture.cluster.directory.clone(); + let next = directory.prepare_group_birth(CellGroupBirthSpec { + history: GroupHistory { + group: GroupId::new([203; 16])?, + ..route.history + }, + range: route.range.clone(), + epoch: RouteEpoch::new(2)?, + creation: ProposalId::new([204; 16])?, + voters: fixture.cluster.birth.voters().to_vec(), + learners: vec![], + })?; + fixture + .cluster + .domain + .run(directory.reserve_group_birth(&next)) + .await??; + let view = fixture.cluster.domain.run(directory.observe()).await??; + let publish = directory.prepare_replacement( + &view, + ProposalId::new([205; 16])?, + std::slice::from_ref(&route), + &[next.route().clone()], + )?; + fixture + .cluster + .domain + .run(directory.publish(&publish)) + .await??; + let before = calls.load(Ordering::SeqCst); + assert!(matches!( + stale.execute_remote(source, leader).await, + Err(CellDispatchError::Transport(TransportError::Query( + CommandFailure::Unavailable + ))) + )); + assert_eq!(calls.load(Ordering::SeqCst), before); + stale.redirect(next.route()).await?; + assert_eq!( + stale.attempt().context("redirect")?.deadline_ms(), + original_deadline + ); + assert_eq!(stale.attempt().context("redirect")?.ordinal(), 2); + assert!(matches!( + stale.execute_remote(source, leader).await, + Err(CellDispatchError::Transport(TransportError::UnknownGroup)) + )); + assert_eq!(calls.load(Ordering::SeqCst), before); + drop(stale); + assert_eq!(dispatch.usage(), (0, 0)); + for transport in &fixture.transports { + assert_eq!(transport.usage().groups, 1); + assert_eq!(transport.usage().peers, 2); + } + for controller in controllers { + controller.close().await?; + } + drop(replacement); + drop(gateways); + fixture.close().await?; + } + Ok(()) +} + +pub(crate) async fn remote_query_lifecycle(expire: bool) -> Result<()> { + for rooted in [false, true] { + let (entered, started) = tokio::sync::oneshot::channel(); + let entered = Mutex::new(Some(entered)); + let (release, held) = std::sync::mpsc::sync_channel(1); + let held = Mutex::new(held); + let calls = Arc::new(AtomicUsize::new(0)); + let count = calls.clone(); + let class = Arc::new(RaftClass::new("Counter", 1, [81; 32])?.query( + "Get", + [82; 32], + move |context, _| { + count.fetch_add(1, Ordering::SeqCst); + entered + .lock() + .unwrap() + .take() + .context("once")? + .send(()) + .map_err(|_| anyhow::anyhow!("entered"))?; + held.lock().unwrap().recv_timeout(Duration::from_secs(15))?; + Ok(context + .query_row("SELECT n FROM value", [], |r| r.get::<_, i64>(0))? + .to_be_bytes() + .to_vec()) + }, + )?); + let (mut fixture, route, a, _) = Box::pin(registered::setup(rooted, &class)).await?; + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let mut gateways = gateways(&fixture, &controllers, class.clone()).await?; + let dispatch = dispatcher(&fixture, class.clone(), keys(190)?)?; + let timeout = if expire { + Duration::from_millis(600) + } else { + Duration::from_secs(10) + }; + let mut query = invocation( + &dispatch, + &class, + &a, + CellQueryConsistency::Linearizable, + timeout, + b"", + ) + .await?; + let source = fixture.transports[1].clone(); + let waiter = fixture.cluster.domain.spawn(async move { + query + .execute_remote(&source, MemberId::new(1).unwrap()) + .await + })?; + tokio::time::timeout(Duration::from_secs(5), started).await??; + let pool = fixture.appliers[&(0, route.history)].0.clone(); + assert_eq!(pool.usage()?.retained_operations, 1); + assert_eq!(calls.load(Ordering::SeqCst), 1); + // The in-flight read retains Gateway data, but cannot retain its public + // admission owner and perpetuate serving after the host drops it. + drop(gateways.remove(0)); + let mut late = invocation( + &dispatch, + &class, + &a, + CellQueryConsistency::Stale, + Duration::from_secs(5), + b"", + ) + .await?; + assert!(late + .execute_remote(&fixture.transports[2], MemberId::new(1)?) + .await + .is_err()); + drop(late); + assert_eq!(calls.load(Ordering::SeqCst), 1); + assert_eq!(pool.usage()?.retained_operations, 1); + // The blocked query does not occupy control connections or Raft work. + let deadline = fixture + .cluster + .domain + .run(async { cellar::asyncrt::mono_ms() + 1000 }) + .await?; + let authority = fixture.runtimes[0] + .read_authority( + route.history, + a.clone(), + CellReadConsistency::Linearizable, + deadline, + ) + .await?; + drop(authority); + if expire { + assert!(waiter.await?.is_err()); + } else { + waiter.abort(); + assert!(waiter.await.is_err()); + } + wait_for(async || { + Ok(fixture.transports[1].usage().query_items == 0 + && fixture.transports[0].usage().query_items == 0) + }) + .await?; + assert_eq!(pool.usage()?.retained_operations, 1); + assert_eq!(dispatch.usage(), (0, 0)); + // Cancelling the close waiter cannot release the entered SQL worker. + assert!( + tokio::time::timeout(Duration::from_millis(30), controllers[0].close()) + .await + .is_err() + ); + assert_eq!(pool.usage()?.running_workers, 1); + release.send(())?; + tokio::time::timeout(Duration::from_secs(5), controllers[0].close()).await??; + assert_eq!(pool.usage()?.retained_operations, 0); + assert_eq!(pool.usage()?.running_workers, 0); + for controller in &controllers[1..] { + controller.close().await?; + } + drop(gateways); + fixture.close().await?; + } + Ok(()) +} + +/// The selected follower, not the ingress node, executes all three read modes +/// over the actual authenticated query and consensus TLS lanes. +pub(crate) async fn remote_follower_reads(rooted: bool) -> Result<()> { + let calls = Arc::new(AtomicUsize::new(0)); + let count = calls.clone(); + let class = Arc::new(RaftClass::new("Counter", 1, [81; 32])?.query( + "Get", + [82; 32], + move |context, input| { + count.fetch_add(1, Ordering::SeqCst); + let QueryVersion::Cell(observation) = context.version() else { + anyhow::bail!("physical group observation"); + }; + assert_eq!(context.identity(), &observation.version.history); + let value: i64 = context.query_row("SELECT n FROM value", [], |row| row.get(0))?; + let mut output = value.to_le_bytes().to_vec(); + output.extend_from_slice(input); + Ok(output) + }, + )?); + let (mut fixture, route, a, _) = Box::pin(registered::setup(rooted, &class)).await?; + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let gateways = gateways(&fixture, &controllers, class.clone()).await?; + let dispatch = dispatcher(&fixture, class.clone(), keys(190)?)?; + assert_eq!( + fixture.runtimes[2].status(route.history).await?.leader, + Some(1) + ); + assert!(!fixture.runtimes[2].status(route.history).await?.ready); + let mut minimum: Option = None; + let mut observation = None; + for index in 0..3_u8 { + let consistency = match index { + 0 => CellQueryConsistency::Linearizable, + 1 => CellQueryConsistency::RecentSource { max_age_ms: 5000 }, + _ => CellQueryConsistency::AtLeast(minimum.clone().context("first token")?), + }; + let mut query = dispatch.query( + a.clone(), + CellQueryRequest { + invocation: ProposalId::new([210 + index; 16])?, + contract: class.contract("Get").context("Get")?, + input: vec![index], + consistency, + }, + CellRoutingLimits { + timeout: Duration::from_secs(5), + max_attempts: 1, + }, + None, + )?; + query.begin().await?; + let response = query + .execute_remote(&fixture.transports[1], MemberId::new(3)?) + .await?; + let mut expected = 0_i64.to_le_bytes().to_vec(); + expected.push(index); + assert_eq!(response.result(), expected); + assert_eq!(response.served_by(), MemberId::new(3)?); + assert_eq!(response.observation().version.history, a); + assert_eq!(response.observation().version.sequence.get(), 1); + if let Some(expected) = &observation { + assert_eq!(response.observation(), expected); + } + observation = Some(response.observation().clone()); + minimum = Some(response.token().clone()); + assert_eq!(fixture.transports[1].usage().query_items, 1); + drop(response); + assert_eq!(fixture.transports[1].usage().query_items, 0); + drop(query); + assert_eq!(dispatch.usage(), (0, 0)); + } + assert_eq!(calls.load(Ordering::SeqCst), 3); + for controller in controllers { + controller.close().await?; + } + drop(gateways); + fixture.close().await +} diff --git a/crates/cellar/tests/support/cell_group_repair_load.rs b/crates/cellar/tests/support/cell_group_repair_load.rs new file mode 100644 index 000000000..0c263c33f --- /dev/null +++ b/crates/cellar/tests/support/cell_group_repair_load.rs @@ -0,0 +1,356 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Competing TLS checkpoint restores on one voter while a third group serves SQL. +use super::*; +use std::sync::atomic::{AtomicUsize, Ordering}; + +const RECEIVER: usize = 2; +const RECEIVER_MEMBER: u64 = 3; +const REPAIR_CELLS: usize = 12; + +#[derive(Default)] +struct Network { + partitioned: Vec, + repairs: BTreeMap, +} + +impl Network { + async fn pump(&mut self, fixture: &mut RuntimeCluster) -> Result<()> { + fixture + .pump_with_network(&[], &mut BTreeMap::new(), |group, message| { + if self.partitioned.contains(&group) + && (message.from == RECEIVER_MEMBER || message.to == RECEIVER_MEMBER) + { + return false; + } + if message.get_msg_type() == MessageType::MsgSnapshot { + assert_eq!(message.to, RECEIVER_MEMBER); + self.repairs.insert(group, message.clone()); + return false; + } + true + }) + .await + } + + async fn applied( + &mut self, + fixture: &mut RuntimeCluster, + group: GroupHistory, + index: LogIndex, + members: &[usize], + ) -> Result<()> { + tokio::time::timeout(Duration::from_secs(15), async { + while members + .iter() + .any(|member| fixture.applied[&group][*member] < index) + { + self.pump(fixture).await?; + tokio::task::yield_now().await; + } + Ok::<_, anyhow::Error>(()) + }) + .await??; + Ok(()) + } +} + +async fn repair_cut( + fixture: &mut RuntimeCluster, + network: &mut Network, + group: GroupHistory, + leader: usize, + pool: &CellGroupCheckpointPool, +) -> Result { + let log = fixture.cluster.voters[leader].log.cell_group(&group)?; + let route = log + .group_manifest()? + .context("repair route")? + .route() + .clone(); + let mut target = LogIndex::new(0); + for round in 0..REPAIR_CELLS { + let bytes = images::image( + None, + &format!("CREATE TABLE value(n); INSERT INTO value VALUES({round});"), + )?; + let cell = history(&route, &format!("restore-{round}"))?; + let proposal = fixture.runtimes[leader] + .propose( + group, + apply::enrollment_bytes(&route, &cell, &bytes, false)?, + ) + .await?; + target = proposal.position.index; + network.applied(fixture, group, target, &[0, 1]).await?; + } + let source = fixture.runtimes[leader] + .capture_checkpoint(group, pool) + .await?; + let checkpoint = fixture + .cluster + .domain + .blocking(move || source.prepare_publication())? + .await??; + let selected = checkpoint.marker().clone(); + assert_eq!(selected.position().index, target); + assert_eq!(selected.root().cells(), REPAIR_CELLS as u32); + fixture.runtimes[leader] + .publish_checkpoint(group, checkpoint) + .await?; + tokio::time::timeout(Duration::from_secs(15), async { + while log.checkpoint_manifest()?.as_ref() != Some(&selected) { + network.pump(fixture).await?; + tokio::task::yield_now().await; + } + Ok::<_, anyhow::Error>(()) + }) + .await??; + Ok(selected) +} + +async fn write_read( + fixture: &mut RuntimeCluster, + network: &mut Network, + route: &GroupRoute, + cell: &CellHistory, + class: &Arc, + sequence: u8, +) -> Result<()> { + let group = route.history; + let pool = fixture.appliers[&(0, group)].0.clone(); + let mut attempt = cell_attempt(130 + sequence)?; + attempt.request.id = CommandId::new(ClientId::new([81; 16])?, u64::from(sequence))?; + attempt.request.operation = "Add".into(); + attempt.request.contract_digest = [82; 32]; + attempt.request.request_digest = command_request_digest(&[]); + let authority = fixture.runtimes[0] + .execution_authority(group, cell.clone()) + .await?; + let CellSqlOutcome::Prepared(prepared) = pool + .stage_registered( + fixture.cell_options(cell).await?, + authority, + attempt, + class.clone(), + vec![], + true, + ) + .await? + else { + anyhow::bail!("expected a new command preparation"); + }; + pool.mark_submitted(cell.clone(), prepared.proposal()) + .await?; + let accepted = accepted_sql(&fixture.runtimes[0], &prepared).await?; + network + .applied(fixture, group, accepted.position.index, &[0, 1, RECEIVER]) + .await?; + let authority = fixture.runtimes[0] + .execution_authority(group, cell.clone()) + .await?; + let receipt = pool.finish(authority).await?; + let expected = (i64::from(sequence) * 7).to_be_bytes(); + assert_eq!(receipt.receipt().receipt.result(), expected); + let selected = fixture.runtimes[RECEIVER] + .read_authority( + group, + cell.clone(), + CellReadConsistency::AtLeast(receipt.receipt().version.clone()), + reads::deadline(fixture, 5000).await?, + ) + .await?; + let output = fixture.appliers[&(RECEIVER, group)] + .0 + .query_registered( + fixture.cell_options_at(cell, RECEIVER).await?, + selected, + class.clone(), + class.contract("Get").context("read contract")?, + vec![], + ) + .await?; + assert_eq!(output.observation(), &receipt.receipt().observation()); + assert_eq!(output.result(), expected); + Ok(()) +} + +pub(crate) async fn repair_budget_under_load() -> Result<()> { + let calls = Arc::new(AtomicUsize::new(0)); + let observed = calls.clone(); + let class = Arc::new( + RaftClass::new("Counter", 1, [81; 32])? + .command("Add", [82; 32], move |ctx, _| { + observed.fetch_add(1, Ordering::SeqCst); + ctx.sql().execute("UPDATE value SET n=n+7", [])?; + let n: i64 = ctx + .sql() + .query_row("SELECT n FROM value", [], |row| row.get(0))?; + Ok(n.to_be_bytes().to_vec()) + })? + .query("Get", [83; 32], |ctx, _| { + let n: i64 = ctx.query_row("SELECT n FROM value", [], |row| row.get(0))?; + Ok(n.to_be_bytes().to_vec()) + })?, + ); + let (mut fixture, route, traffic, _) = Box::pin(registered::setup(false, &class)).await?; + let mut groups = Vec::new(); + for number in [71, 72] { + let history = fixture + .add_group_with_filesystem(number, |voter| { + Arc::new(fault::FaultFileSystem { + gate: voter.gate.clone(), + }) + }) + .await?; + controller::attach_group_appliers(&mut fixture, route.history, history).await?; + groups.push(history); + } + fixture + .connect_tcp_options( + None, + TransportOptions { + io_timeout_ms: 30_000, + snapshot_timeout_ms: 60_000, + ..Default::default() + }, + ) + .await?; + let mut pools = Vec::new(); + let mut registrations = Vec::new(); + for node in 0..3 { + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + fixture.transports[node].enable_group_checkpoints(pool.clone())?; + pools.push(pool); + for group in &groups { + registrations.push( + fixture.transports[node] + .register_group_checkpoint(fixture.appliers[&(node, *group)].1.clone())?, + ); + } + } + + let mut network = Network { + partitioned: groups.clone(), + ..Default::default() + }; + let publication = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let mut cuts = Vec::new(); + for (leader, group) in groups.iter().copied().enumerate() { + fixture.runtimes[leader].campaign(group).await?; + network + .applied(&mut fixture, group, LogIndex::new(2), &[0, 1]) + .await?; + cuts.push(repair_cut(&mut fixture, &mut network, group, leader, &publication).await?); + assert_eq!(fixture.applied[&group][RECEIVER], LogIndex::new(1)); + } + network.partitioned.clear(); + tokio::time::timeout(Duration::from_secs(15), async { + while !groups + .iter() + .all(|group| network.repairs.contains_key(group)) + { + for leader in 0..2 { + fixture.runtimes[leader].tick().await?; + } + network.pump(&mut fixture).await?; + tokio::time::sleep(Duration::from_millis(10)).await; + } + Ok::<_, anyhow::Error>(()) + }) + .await??; + let first = network + .repairs + .remove(&groups[0]) + .context("first Raft repair")?; + let second = network + .repairs + .remove(&groups[1]) + .context("second Raft repair")?; + assert_eq!(first.from, 1); + assert_eq!(second.from, 2); + + let gate = fixture.cluster.voters[RECEIVER].gate.clone(); + let hold = gate.arm(fault::HOLD_DEPENDENCY_PREFIX); + let first_delivery = fixture.transports[0].submit_physical(&groups[0], first)?; + tokio::time::timeout(Duration::from_secs(10), gate.entered.notified()) + .await + .context("first TLS restore reached the receiver image spool")?; + let charged = pools[RECEIVER].usage(); + assert_eq!(charged.sources, 1); + assert!(charged.retained_bytes > 0); + assert!(fixture.transports[0].usage().data_items > 0); + let competitor = fixture.transports[1].submit_physical(&groups[1], second.clone())?; + assert!( + tokio::time::timeout(Duration::from_secs(5), competitor.accepted()) + .await? + .is_err() + ); + + tokio::time::timeout(Duration::from_secs(20), async { + for sequence in 1..=24 { + assert_eq!(pools[RECEIVER].usage(), charged); + write_read( + &mut fixture, + &mut network, + &route, + &traffic, + &class, + sequence, + ) + .await?; + } + Ok::<_, anyhow::Error>(()) + }) + .await??; + for group in &groups { + assert!(fixture.cluster.voters[RECEIVER] + .log + .cell_group(group)? + .checkpoint_manifest()? + .is_none()); + } + assert_eq!(pools[RECEIVER].usage(), charged); + assert_eq!(calls.load(Ordering::SeqCst), 24); + drop(hold); + tokio::time::timeout(Duration::from_secs(10), first_delivery.accepted()).await??; + network + .applied( + &mut fixture, + groups[0], + cuts[0].position().index, + &[RECEIVER], + ) + .await?; + controller::wait_for(async || { + Ok(pools[RECEIVER].usage().sources == 0 && pools[1].usage().sources == 0) + }) + .await?; + let second_delivery = fixture.transports[1].submit_physical(&groups[1], second)?; + tokio::time::timeout(Duration::from_secs(10), second_delivery.accepted()).await??; + network + .applied( + &mut fixture, + groups[1], + cuts[1].position().index, + &[RECEIVER], + ) + .await?; + for (group, cut) in groups.into_iter().zip(cuts) { + let log = fixture.cluster.voters[RECEIVER].log.cell_group(&group)?; + assert_eq!(log.checkpoint_manifest()?.as_ref(), Some(&cut)); + assert_eq!(log.group_manifest()?.as_ref(), Some(&cut)); + let applier = fixture.appliers[&(RECEIVER, group)].1.clone(); + let recovered = fixture + .cluster + .domain + .blocking(move || applier.recover_selected())? + .await??; + assert_eq!(recovered.marker(), &cut); + assert_eq!(recovered.pinned_images(), REPAIR_CELLS); + } + controller::wait_for(async || Ok(pools.iter().all(|pool| pool.usage().retained_bytes == 0))) + .await?; + drop(registrations); + fixture.close().await +} diff --git a/crates/cellar/tests/support/cell_group_routing_cache.rs b/crates/cellar/tests/support/cell_group_routing_cache.rs new file mode 100644 index 000000000..6028f734f --- /dev/null +++ b/crates/cellar/tests/support/cell_group_routing_cache.rs @@ -0,0 +1,235 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Real catalog calls qualify bounded hints, not permission to serve a cell. +use super::*; +use cellar::application::consensus::{RaftClass, VersionKeys, VersionSigningKey}; +use cellar_logic::consensus::{ClientId, CommandId, Durability}; + +async fn setup( + count: usize, + capacity: usize, + bytes: usize, +) -> Result<( + Fixture, + Arc, + CellGroupDispatcher, + Vec, +)> { + let class = Arc::new( + RaftClass::new("Cache", 1, [161; 32])? + .query("Get", [162; 32], |_, _| Ok(vec![]))? + .command("Add", [163; 32], |_, _| Ok(vec![]))?, + ); + let f = Fixture::with_schema( + DirectoryLimits { + max_cached_pages: 0, + max_cache_bytes: 0, + ..Default::default() + }, + class.schema_digest(), + ) + .await?; + let mut routes = ranges(count); + for route in &mut routes { + route.schema_digest = class.schema_digest(); + } + f.reserve(&mut routes).await?; + let mutation = f + .directory + .prepare_initialization(operation(990), &routes)?; + f.run(f.directory.publish(&mutation)).await?; + let dispatcher = CellGroupDispatcher::new( + &f.domain, + f.directory.clone(), + class.clone(), + VersionKeys::new( + VersionSigningKey::new([164; 32], u64::MAX)?, + vec![], + Duration::from_secs(30), + )?, + CellDispatchLimits { + max_cached_routes: capacity, + route_cache_bytes: bytes, + max_invocations: 8, + retained_bytes: 1024 * 1024, + max_input_bytes: 1024, + }, + )?; + Ok((f, class, dispatcher, routes)) +} + +fn cell(route: &GroupRoute, tag: usize) -> CellHistory { + let cell = (0..100_000) + .map(|i| CellId::new(format!("cache-{tag}-{i}")).unwrap()) + .find(|cell| { + route + .range + .contains(&cell_routing_key(route.history.namespace, cell)) + }) + .unwrap(); + CellHistory { + cluster: route.history.cluster, + namespace: route.history.namespace, + cell, + incarnation: IncarnationId::new([165; 16]).unwrap(), + } +} + +fn query( + dispatcher: &CellGroupDispatcher, + class: &RaftClass, + cell: CellHistory, +) -> Result { + Ok(dispatcher.query( + cell, + CellQueryRequest { + invocation: operation(991), + contract: class.contract("Get").context("Get")?, + input: vec![], + consistency: CellQueryConsistency::Stale, + }, + routing_limits(4), + None, + )?) +} + +pub(crate) async fn cached_operation_discovery() -> Result<()> { + let (f, class, dispatcher, routes) = setup(1, 2, 16 * 1024).await?; + let first = cell(&routes[0], 0); + let mut warm = query(&dispatcher, &class, first.clone())?; + let (result, reads) = fault::count_reads(warm.begin()).await; + result?; + assert!(!reads.is_empty()); + drop(warm); + f.store.fail_read.store(true, Ordering::Release); + for n in 1..65 { + let mut query = query(&dispatcher.clone(), &class, cell(&routes[0], n))?; + let (result, reads) = fault::count_reads(query.begin()).await; + result?; + assert!(reads.is_empty()); + assert_eq!(query.attempt().context("cached")?.route(), &routes[0]); + } + let mut command = dispatcher + .command( + first, + CellCommandRequest { + id: CommandId::new(ClientId::new([166; 16])?, 1)?, + contract: class.contract("Add").context("Add")?, + input: vec![], + durability: Durability::Quorum, + retry_until_ms: u64::MAX, + }, + routing_limits(2), + None, + ) + .await?; + let (result, reads) = fault::count_reads(command.begin()).await; + result?; + assert!(reads.is_empty()); + assert_eq!(command.attempt().context("command")?.route(), &routes[0]); + drop(command); + let usage = dispatcher.route_cache_usage()?; + assert_eq!(usage.entries, 1); + assert!(usage.reserved_bytes <= 16 * 1024); + assert!(dispatcher.refresh_routes().await.is_err()); + assert_eq!(dispatcher.route_cache_usage()?, usage); + assert_eq!(dispatcher.usage(), (0, 0)); + f.domain.shutdown().await; + Ok(()) +} + +pub(crate) async fn bounded_route_hints() -> Result<()> { + let (f, class, dispatcher, routes) = setup(4, 64, 16 * 1024).await?; + assert_eq!(dispatcher.route_cache_usage()?.capacity, 2); + for (index, hit) in [(0, false), (1, false), (0, true), (2, false), (1, false)] { + let mut query = query(&dispatcher, &class, cell(&routes[index], index))?; + let (result, reads) = fault::count_reads(query.begin()).await; + result?; + assert_eq!(reads.is_empty(), hit); + assert!(dispatcher.route_cache_usage()?.entries <= 2); + assert!(dispatcher.route_cache_usage()?.reserved_bytes <= 16 * 1024); + } + let selected = cell(&routes[0], 0); + let before = dispatcher.route_cache_usage()?; + let view = f.run(f.directory.observe()).await?; + let mut planned = split(&routes[0], 730); + for child in &mut planned { + child.schema_digest = routes[0].schema_digest; + } + f.reserve(&mut planned).await?; + let transition = f.directory.prepare_replacement( + &view, + operation(1002), + std::slice::from_ref(&routes[0]), + &planned, + )?; + f.run(f.directory.plan_transition(&transition)).await?; + assert_ne!(f.run(f.directory.observe()).await?.digest(), view.digest()); + assert_eq!(dispatcher.refresh_routes().await?, view.revision()); + assert_eq!(dispatcher.route_cache_usage()?, before); + f.run(f.directory.prepare_transition(&transition, [11; 32])) + .await?; + assert_eq!(dispatcher.refresh_routes().await?, view.revision()); + f.run(f.directory.cancel_transition(&transition)).await?; + assert_eq!(dispatcher.refresh_routes().await?, view.revision()); + assert_eq!(dispatcher.route_cache_usage()?, before); + let next = publish_route_split(&f, &routes[0], &selected.cell).await?; + assert_eq!(dispatcher.refresh_routes().await?, 2); + assert_eq!(dispatcher.route_cache_usage()?.entries, 0); + let mut moved = query(&dispatcher, &class, selected)?; + let (result, reads) = fault::count_reads(moved.begin()).await; + result?; + assert!(!reads.is_empty()); + assert_eq!(moved.attempt().context("successor")?.route(), &next); + assert_eq!(dispatcher.route_cache_usage()?.revision, Some(2)); + drop(moved); + f.domain.shutdown().await; + Ok(()) +} + +pub(crate) async fn stale_cache_refill() -> Result<()> { + let (f, class, dispatcher, routes) = setup(1, 2, 16 * 1024).await?; + let selected = cell(&routes[0], 0); + let view = f.run(f.directory.observe()).await?; + let (result, reads) = + fault::count_reads(f.run(f.directory.lookup(&view, &selected.cell))).await; + result?; + let page = reads + .keys() + .find(|path| path.contains("/ranges/")) + .context("range page")? + .clone(); + let gate = Arc::new(fault::RegistryGate::default()); + gate.armed.store(true, Ordering::Release); + let wait_gate = gate.clone(); + let mut old = query(&dispatcher, &class, selected.clone())?; + let waiter = f + .domain + .spawn(async move { fault::gate_read(page, wait_gate, old.begin()).await })?; + f.run(cellar::asyncrt::timeout( + Duration::from_secs(5), + gate.entered.notified(), + )) + .await?; + let successor = publish_route_split(&f, &routes[0], &selected.cell).await?; + assert_eq!(dispatcher.refresh_routes().await?, 2); + gate.resume.notify_one(); + let error = waiter.await?.err().context("old lookup must refuse")?; + assert!(matches!( + error, + CellDispatchError::Routing(CellRoutingError { + reason: CellRoutingFailure::Catalog(CatalogError::Conflict( + "superseded routing observation" + )), + .. + }) + )); + assert_eq!(dispatcher.route_cache_usage()?.entries, 0); + let mut fresh = query(&dispatcher, &class, selected)?; + fresh.begin().await?; + assert_eq!(fresh.attempt().context("current")?.route(), &successor); + drop(fresh); + assert_eq!(dispatcher.usage(), (0, 0)); + f.domain.shutdown().await; + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs new file mode 100644 index 000000000..b887ee6ec --- /dev/null +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -0,0 +1,1018 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Real node workers share the existing SQLite/image fixture and engine owners. + +use super::*; +use std::collections::BTreeMap; + +#[path = "cell_group_multigroup.rs"] +mod multigroup; +pub(super) use multigroup::{hot_group_fairness, mixed_roles}; + +#[path = "cell_group_checkpoint_local.rs"] +mod checkpoint_local; +#[path = "cell_group_network_probes.rs"] +mod network_probes; +#[path = "cell_group_seed_probes.rs"] +mod seed_probes; +pub(super) use checkpoint_local::{ + local_publication, local_shutdown, tcp_cancelled_delivery, tcp_protocol, tcp_publication, + tcp_retry, +}; + +#[path = "cell_group_snapshot_owner.rs"] +mod snapshot_owner; +pub(super) use snapshot_owner::{ + checkpoint_bounds, checkpoint_matching_log, checkpoint_owner, checkpoint_shutdown, +}; + +#[path = "cell_group_checkpoint_pressure.rs"] +mod checkpoint_pressure; +pub(super) use checkpoint_pressure::transfer_pressure; + +#[path = "cell_group_checkpoint_receive.rs"] +mod checkpoint_receive; +pub(super) use checkpoint_receive::{install_checkpoint, refuse_checkpoint}; + +#[path = "cell_group_tcp.rs"] +pub(super) mod tcp; +pub(super) use tcp::physical_transport; + +#[path = "cell_group_source_seal.rs"] +mod source_seal; +pub(super) use source_seal::source_seal_blocks_delayed_work; +#[path = "cell_group_source_archives.rs"] +mod source_archives; +pub(super) use source_archives::sealed_source_archives; + +struct RuntimeCluster { + snapshot_sources: BTreeMap<(usize, GroupHistory), PreparedGroupManifest>, + snapshot_transfers: usize, + drop_snapshot_delivery: bool, + retry_snapshot: bool, + failed_snapshot: bool, + retry_checkpoint: Option, + transports: Vec, + appliers: BTreeMap<(usize, GroupHistory), (CellGroupSqlPool, CellGroupApplier)>, + cluster: Cluster, + runtimes: Vec, + events: Vec, + applied: BTreeMap>, + extra: BTreeMap>, + network: VecDeque<(GroupHistory, Message)>, +} + +impl RuntimeCluster { + async fn new() -> Result { + Self::with_limits(256, 256, 1_000_000).await + } + + async fn with_limits( + max_events: usize, + max_requests: usize, + max_cells_per_group: u32, + ) -> Result { + Box::pin(Self::from_cluster( + Cluster::new().await?, + max_events, + max_requests, + max_cells_per_group, + )) + .await + } + + async fn from_cluster( + mut cluster: Cluster, + max_events: usize, + max_requests: usize, + max_cells_per_group: u32, + ) -> Result { + let mut runtimes = Vec::new(); + let mut events = Vec::new(); + let mut applied = Vec::new(); + for n in 0..3 { + let owner = cluster.voters[n].owner.take().context("old direct owner")?; + let root = owner.manifest().clone(); + drop(owner); + applied.push(root.marker().position().index); + let (runtime, receiver) = CellGroupRuntime::start( + cluster.voters[n].log.clone(), + RaftRuntimeOptions { + tick_interval_ms: None, + election_seed: Some([71; 32]), + max_events, + max_requests, + max_cells_per_group, + ..Default::default() + }, + ) + .await?; + runtime.attach(cluster.permit(n).await?, root).await?; + runtimes.push(runtime); + events.push(receiver); + } + let applied = BTreeMap::from([(cluster.birth.route().history, applied)]); + Ok(Self { + snapshot_sources: BTreeMap::new(), + snapshot_transfers: 0, + drop_snapshot_delivery: false, + retry_snapshot: false, + failed_snapshot: false, + retry_checkpoint: None, + transports: Vec::new(), + appliers: BTreeMap::new(), + cluster, + runtimes, + events, + applied, + extra: BTreeMap::new(), + network: VecDeque::new(), + }) + } + + async fn pump( + &mut self, + hold: &[GroupHistory], + held: &mut BTreeMap, + ) -> Result<()> { + self.pump_with_network(hold, held, |_, _| true).await + } + + async fn pump_with_network( + &mut self, + hold: &[GroupHistory], + held: &mut BTreeMap, + mut forward: impl FnMut(GroupHistory, &Message) -> bool, + ) -> Result<()> { + let primary = self.cluster.birth.route().history; + for n in 0..3 { + for _ in 0..64 { + let Ok(event) = self.events[n].try_recv() else { + break; + }; + let scope = event.group(); + match event.effect().map_err(|e| anyhow::anyhow!(e.to_string()))? { + CellGroupEffect::Message(message) => { + self.network.push_back((scope, message.clone())) + } + CellGroupEffect::Apply(_) => { + let work = event.retain_apply().context("retained apply")?; + if n == 0 && hold.contains(&scope) { + assert!(held.insert(scope, work).is_none()); + } else { + if let Some((pool, applier)) = self.appliers.get(&(n, scope)) { + // Keep the physical apply future out of every + // caller's inline transport/recovery state. + Box::pin(async { + pool.prepare_apply(applier, work) + .await? + .complete(&self.runtimes[n]) + .await?; + Ok::<_, anyhow::Error>(()) + }) + .await?; + continue; + } + let voter = if scope == primary { + &mut self.cluster.voters[n] + } else { + &mut self.extra.get_mut(&scope).context("extra voter")?[n] + }; + let next = prepare(voter, &work)?; + self.runtimes[n].complete_apply(scope, work, next).await?; + } + } + CellGroupEffect::Applied { manifest, .. } + | CellGroupEffect::CheckpointInstalled { manifest } + | CellGroupEffect::CheckpointPublished { manifest } => { + self.applied.get_mut(&scope).context("applied")?[n] = + manifest.position().index + } + CellGroupEffect::ReadIndex { .. } => (), + CellGroupEffect::Persist(_) => anyhow::bail!("node worker leaked persistence"), + } + } + } + let messages = std::mem::take(&mut self.network) + .into_iter() + .filter(|(history, message)| forward(*history, message)); + if !self.transports.is_empty() { + for (history, message) in messages { + let snapshot = message.get_msg_type() == MessageType::MsgSnapshot; + let transport = &self.transports[message.from as usize - 1]; + let delivery = match transport.submit_physical(&history, message.clone()) { + Ok(delivery) => delivery, + Err(TransportError::Overloaded) if self.retry_snapshot && snapshot => { + self.network.push_back((history, message)); + continue; + } + Err(error) => { + return Err(error).with_context(|| { + format!( + "submit physical {:?} from {} to {} for {history:?}", + message.get_msg_type(), + message.from, + message.to + ) + }); + } + }; + if snapshot && self.drop_snapshot_delivery { + drop(delivery); + } else { + let result = delivery.accepted().await; + if snapshot && self.retry_snapshot && !self.failed_snapshot { + assert!( + result.is_err(), + "revoked receiver must refuse the first actual Raft snapshot" + ); + self.failed_snapshot = true; + let to = message.to as usize - 1; + self.retry_checkpoint = Some( + self.transports[to] + .register_group_checkpoint(self.appliers[&(to, history)].1.clone()) + .with_context(|| { + format!( + "restore checkpoint registration at member {}", + message.to + ) + })?, + ); + // Do not requeue this failed snapshot. Raft must create + // another after its delivery failure is consumed. + continue; + } + result.with_context(|| { + format!( + "deliver physical {:?} from {} to {} for {history:?}", + message.get_msg_type(), + message.from, + message.to + ) + })?; + } + if snapshot { + self.snapshot_transfers += 1; + } + } + if self.retry_snapshot && self.snapshot_transfers == 0 { + self.runtimes[0].tick().await?; + } + return Ok(()); + } + let mut ordinary = VecDeque::new(); + for (scope, message) in messages { + if message.get_msg_type() == MessageType::MsgSnapshot { + self.transfer_checkpoint(scope, message).await?; + } else { + ordinary.push_back((scope, message)); + } + } + let runtimes = &self.runtimes; + for (scope, message, result) in futures_util::future::join_all(ordinary.into_iter().map( + |(scope, message)| async move { + let result = runtimes[message.to as usize - 1] + .admit_step(scope, message.clone()) + .await; + (scope, message, result) + }, + )) + .await + { + match result { + Ok(()) => (), + // Preserve the peer frame until bounded transport admission is + // available. Test deadlines still fail a permanently stuck peer. + Err(RuntimeError::Overloaded | RuntimeError::NotReady) => { + self.network.push_back((scope, message)) + } + Err(error) => return Err(error.into()), + } + } + Ok(()) + } + + async fn settle(&mut self, histories: &[GroupHistory]) -> Result<()> { + tokio::time::timeout(Duration::from_secs(20), async { + loop { + self.pump(&[], &mut BTreeMap::new()).await?; + let mut done = true; + for &scope in histories { + let status = self.runtimes[0].status(scope).await?; + done &= status.ready + && self.applied[&scope] + .iter() + .all(|index| *index >= status.applied.index); + } + if done { + return Ok::<_, anyhow::Error>(()); + } + tokio::task::yield_now().await; + } + }) + .await? + } + + async fn drive(&mut self, target: Option) -> Result<()> { + let history = self.cluster.birth.route().history; + tokio::time::timeout(Duration::from_secs(20), async { + loop { + self.pump(&[], &mut BTreeMap::new()).await?; + let status = match self.runtimes[0].status(history).await { + Ok(status) => status, + Err(RuntimeError::Overloaded) => { + // With one ordinary slot, an admitted peer frame can + // still own it. Keep pumping under the existing deadline. + tokio::task::yield_now().await; + continue; + } + Err(error) => return Err(error.into()), + }; + let target = target.unwrap_or(status.applied.index); + if status.ready && self.applied[&history].iter().all(|index| *index >= target) { + return Ok::<_, anyhow::Error>(()); + } + tokio::task::yield_now().await; + } + }) + .await? + } + + async fn close(self) -> Result<()> { + for transport in &self.transports { + transport.close().await; + } + drop(self.transports); + for (pool, _) in self.appliers.values() { + pool.close().await?; + } + drop(self.appliers); + for runtime in &self.runtimes { + runtime.close().await?; + } + drop(self.events); + drop(self.runtimes); + drop(self.extra); + drop(self.snapshot_sources); + drop(self.retry_checkpoint); + self.cluster.close().await + } +} + +pub(super) async fn replicated_cells() -> Result<()> { + let mut fixture = RuntimeCluster::new().await?; + let route = fixture.cluster.birth.route().clone(); + fixture.runtimes[0].campaign(route.history).await?; + fixture.drive(None).await?; + for name in ["worker-a", "worker-b", "worker-c"] { + let cell = history(&route, name)?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES (1);")?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let accepted = fixture.runtimes[0] + .propose(route.history, request(&route, cell, 1, &cut)?) + .await?; + fixture.drive(Some(accepted.position.index)).await?; + } + for voter in &fixture.cluster.voters { + let group = voter.log.cell_group(&route.history)?; + assert_eq!( + group + .group_manifest()? + .context("published group")? + .root() + .cells(), + 3 + ); + } + fixture.close().await +} + +pub(super) async fn capture_archive_cut() -> Result<()> { + let (cluster, expected) = + checkpoint_fixture::populate_images(false, false, false, 0, 8192).await?; + let mut fixture = RuntimeCluster::from_cluster(cluster, 256, 256, 64).await?; + let route = fixture.cluster.birth.route().clone(); + fixture.runtimes[0].campaign(route.history).await?; + fixture.drive(None).await?; + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let mut source = fixture.runtimes[0] + .capture_checkpoint(route.history, &pool) + .await?; + let cut = source.marker().clone(); + assert!(matches!( + fixture.runtimes[0] + .capture_checkpoint(route.history, &pool) + .await, + Err(CellGroupCheckpointError::Busy) + )); + let image = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(4)")?; + let delta = images::Cut::new(&[], &image, 0, true, false); + let accepted = fixture.runtimes[0] + .propose( + route.history, + request(&route, history(&route, "later")?, 1, &delta)?, + ) + .await?; + fixture.drive(Some(accepted.position.index)).await?; + assert!( + fixture.runtimes[0] + .status(route.history) + .await? + .applied + .index + > cut.position().index + ); + let restored = fixture + .cluster + .domain + .blocking(move || -> Result<_> { + let mut restored = BTreeMap::>::new(); + source.stream(|chunk| { + if let CellGroupCheckpointPart::Materialized(cell) = chunk.part { + restored + .entry(cell.as_str().into()) + .or_default() + .extend_from_slice(chunk.bytes); + } + Ok(()) + })?; + Ok(restored) + })? + .await??; + for (cell, bytes) in expected { + if !bytes.is_empty() { + assert_eq!(restored[&cell], bytes); + } + } + assert!(!restored.contains_key("later")); + assert_eq!(pool.usage().sources, 0); + let next = fixture.runtimes[0] + .capture_checkpoint(route.history, &pool) + .await?; + assert!(next.marker().position().index > cut.position().index); + drop(next); + let runtime = fixture.runtimes[0].clone(); + fixture.close().await?; + assert!(matches!( + runtime.capture_checkpoint(route.history, &pool).await, + Err(CellGroupCheckpointError::Authority(RuntimeError::Closed)) + )); + Ok(()) +} + +pub(super) async fn cancelled_close_joins_sync() -> Result<()> { + let mut fixture = RuntimeCluster::new().await?; + let route = fixture.cluster.birth.route().clone(); + fixture.runtimes[0].campaign(route.history).await?; + fixture.drive(None).await?; + let gate = fixture.cluster.voters[0].gate.clone(); + let release = gate.arm(fault::HOLD_SYNC); + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES (1);")?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + fixture.runtimes[0] + .propose( + route.history, + request(&route, history(&route, "closing")?, 1, &cut)?, + ) + .await?; + tokio::time::timeout(Duration::from_secs(5), gate.entered.notified()).await?; + let runtime = fixture.runtimes[0].clone(); + let closing = runtime.close(); + assert!(tokio::time::timeout(Duration::from_millis(30), closing) + .await + .is_err()); + assert!(CellGroupRuntime::start( + fixture.cluster.voters[0].log.clone(), + RaftRuntimeOptions::default() + ) + .await + .is_err()); + drop(release); + tokio::time::timeout(Duration::from_secs(5), runtime.close()).await??; + let (replacement, events) = CellGroupRuntime::start( + fixture.cluster.voters[0].log.clone(), + RaftRuntimeOptions::default(), + ) + .await?; + replacement.close().await?; + drop(events); + fixture.close().await +} + +impl RuntimeCluster { + async fn add_group(&mut self, number: u8) -> Result { + self.add_group_with_filesystem(number, |_| Arc::new(DirectFileSystem)) + .await + } + + async fn add_group_with_filesystem( + &mut self, + number: u8, + filesystem: impl Fn(&Voter) -> Arc, + ) -> Result { + let (_, birth) = self + .add_group_in_store( + number, + Arc::new(object_store::memory::InMemory::new()), + filesystem, + ) + .await?; + Ok(birth.route().history) + } + + async fn add_group_in_store( + &mut self, + number: u8, + store: Arc, + filesystem: impl Fn(&Voter) -> Arc, + ) -> Result<(RangeDirectory, CellGroupBirth)> { + let scope = GroupHistory { + cluster: self.cluster.birth.route().history.cluster, + namespace: NamespaceId::new([number; 16])?, + group: GroupId::new([number; 16])?, + incarnation: IncarnationId::new([number; 16])?, + }; + let domain = &self.cluster.domain; + let catalog = Catalog::new(store, CatalogOptions::default())?; + let namespace = domain + .run(catalog.open_namespace(NamespaceConfig::new( + scope.cluster, + scope.namespace, + cellar_logic::NamespaceMode::Raft, + [6; 32], + ))) + .await??; + let directory = RangeDirectory::new(namespace, DirectoryLimits::default())?; + let request = directory.prepare_group_birth(CellGroupBirthSpec { + history: scope, + range: CellRange::all(), + epoch: RouteEpoch::new(1)?, + creation: ProposalId::new([number; 16])?, + voters: (1..=3) + .map(|n| InitialMember { + member: MemberId::new(n).unwrap(), + node: NodeId::new([n as u8; 16]).unwrap(), + }) + .collect(), + learners: vec![], + })?; + domain + .run(directory.reserve_group_birth(&request)) + .await??; + let initialization = directory.prepare_initialization( + ProposalId::new([number + 1; 16])?, + &[request.route().clone()], + )?; + domain.run(directory.publish(&initialization)).await??; + let birth = domain + .run(directory.materialize_group_birth(&request)) + .await??; + let mut voters = Vec::new(); + let mut applied = Vec::new(); + for n in 0..3 { + let log = self.cluster.voters[n].log.clone(); + let permit = domain + .run(directory.bootstrap_group_member(&birth, &log)) + .await??; + let base = self + .cluster + ._root + .path() + .join(format!("extra-{number}-node-{n}")); + let filesystem = filesystem(&self.cluster.voters[n]); + let tree = ManifestTree::new(birth.route().clone(), ManifestLimits::default())?; + let mut pages = ManifestPageStore::open( + &base.join("pages"), + filesystem.clone(), + 1024, + 16 * 1024 * 1024, + 16, + )?; + let images = + GenerationStore::open(base.join("images"), filesystem, Default::default())?; + let marker = permit.group().group_manifest()?.context("genesis")?; + applied.push(marker.position().index); + let root = tree.prepare_complete( + permit.group(), + &mut pages, + marker.root(), + marker.position(), + marker.membership_revision(), + |_| panic!("empty genesis"), + )?; + self.runtimes[n].attach(permit, root).await?; + voters.push(Voter { + log, + owner: None, + pages: Some(pages), + images, + extents: None, + tree, + gate: self.cluster.voters[n].gate.clone(), + paused: false, + held: None, + }); + } + self.applied.insert(scope, applied); + self.extra.insert(scope, voters); + Ok((directory, birth)) + } +} + +pub(super) async fn shared_batches() -> Result<()> { + let mut fixture = RuntimeCluster::new().await?; + let second = fixture.add_group(31).await?; + let third = fixture.add_group(41).await?; + let histories = [fixture.cluster.birth.route().history, second, third]; + let before = fixture.cluster.voters[0].log.performance_snapshot(); + for result in futures_util::future::join_all( + histories + .iter() + .map(|&history| fixture.runtimes[0].campaign(history)), + ) + .await + { + result?; + } + fixture.settle(&histories).await?; + let after = fixture.cluster.voters[0].log.performance_snapshot(); + assert!(before.available && after.available && !before.saturated && !after.saturated); + let groups = |s: &RaftPerformanceSnapshot| { + s.counters + .iter() + .find(|c| c.name == "engine_persist_groups") + .unwrap() + .value + }; + let batches = |s: &RaftPerformanceSnapshot| { + s.operations + .iter() + .find(|o| o.name == "engine_persist") + .unwrap() + .succeeded + }; + let group_delta = groups(&after) - groups(&before); + let batch_delta = batches(&after) - batches(&before); + assert!(batch_delta > 0 && group_delta > batch_delta, "actual engine batches must include multiple groups: {group_delta} groups / {batch_delta} batches"); + fixture.close().await +} + +pub(super) async fn shared_proposal_budget() -> Result<()> { + let mut fixture = RuntimeCluster::with_limits(256, 8, 1_000_000).await?; + let first = fixture.cluster.birth.route().history; + let second = fixture.add_group(61).await?; + for scope in [first, second] { + fixture.runtimes[0].campaign(scope).await?; + fixture.settle(&[scope]).await?; + } + let route = fixture.cluster.birth.route().clone(); + let other_route = fixture.extra[&second][0] + .log + .cell_group(&second)? + .group_manifest()? + .context("other group")? + .route() + .clone(); + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES (1);")?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let mut first_request = None; + let mut target = LogIndex::new(0); + for n in 0..8 { + let data = request(&route, history(&route, &format!("budget-{n}"))?, 1, &cut)?; + let accepted = fixture.runtimes[0].propose(first, data.clone()).await?; + if n == 0 { + first_request = Some((data, accepted.clone())); + } + target = accepted.position.index; + } + let other = request(&other_route, history(&other_route, "other-group")?, 1, &cut)?; + assert!(matches!( + fixture.runtimes[0].propose(second, other.clone()).await, + Err(RuntimeError::Overloaded) + )); + let (data, accepted) = first_request.context("first acceptance")?; + let retry = fixture.runtimes[0].propose(first, data).await?; + assert_eq!(accepted.position, retry.position); + assert_eq!(accepted.digest, retry.digest); + tokio::time::timeout(Duration::from_secs(20), async { + while fixture.applied[&first].iter().any(|index| *index < target) { + fixture.pump(&[], &mut BTreeMap::new()).await?; + tokio::task::yield_now().await; + } + Ok::<_, anyhow::Error>(()) + }) + .await??; + let accepted = fixture.runtimes[0].propose(second, other).await?; + tokio::time::timeout(Duration::from_secs(20), async { + while fixture.applied[&second] + .iter() + .any(|index| *index < accepted.position.index) + { + fixture.pump(&[], &mut BTreeMap::new()).await?; + tokio::task::yield_now().await; + } + Ok::<_, anyhow::Error>(()) + }) + .await??; + fixture.close().await +} + +pub(super) async fn bounded_apply_completion() -> Result<()> { + let mut fixture = RuntimeCluster::with_limits(2, 256, 1_000_000).await?; + let first = fixture.cluster.birth.route().history; + let second = fixture.add_group(51).await?; + let histories = [first, second]; + for result in futures_util::future::join_all( + histories + .iter() + .map(|&scope| fixture.runtimes[0].campaign(scope)), + ) + .await + { + result?; + } + fixture + .settle(&histories) + .await + .context("settling bounded-output groups")?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES (7);")?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let mut targets = BTreeMap::new(); + for scope in histories { + let route = if scope == first { + fixture.cluster.birth.route().clone() + } else { + fixture.extra[&scope][0] + .log + .cell_group(&scope)? + .group_manifest()? + .context("route")? + .route() + .clone() + }; + let accepted = fixture.runtimes[0] + .propose(scope, request(&route, history(&route, "quota")?, 1, &cut)?) + .await?; + targets.insert(scope, accepted.position.index); + } + let mut held = BTreeMap::new(); + tokio::time::timeout(Duration::from_secs(20), async { + while held.len() != 2 { + fixture.pump(&histories, &mut held).await?; + tokio::task::yield_now().await; + } + Ok::<_, anyhow::Error>(()) + }) + .await + .with_context(|| { + format!( + "bounded apply progress: held={}, applied={:?}", + held.len(), + fixture.applied + ) + })??; + // Both output slots are retained by physical work. Completing one must + // release its reservation before admitting its distinct Applied event. + let work = held.remove(&first).context("first physical job")?; + let next = prepare(&mut fixture.cluster.voters[0], &work)?; + fixture.runtimes[0] + .complete_apply(first, work, next) + .await?; + tokio::time::timeout(Duration::from_secs(20), async { + while fixture.applied[&first][0] < targets[&first] { + fixture.pump(&[second], &mut held).await?; + tokio::task::yield_now().await; + } + Ok::<_, anyhow::Error>(()) + }) + .await + .with_context(|| { + format!( + "bounded apply progress: held={}, applied={:?}", + held.len(), + fixture.applied + ) + })??; + assert_eq!(held.len(), 1, "another group's physical job remains held"); + let work = held.remove(&second).context("second physical job")?; + let next = prepare( + &mut fixture.extra.get_mut(&second).context("second voter")?[0], + &work, + )?; + fixture.runtimes[0] + .complete_apply(second, work, next) + .await?; + fixture + .settle(&histories) + .await + .context("settling bounded-output groups")?; + for scope in histories { + assert!(fixture.applied[&scope] + .iter() + .all(|index| *index >= targets[&scope])); + } + fixture.close().await +} + +// A pending peer/protocol request retains ordinary admission through a physical +// sync. Attachment must refuse before attempting to claim an existing owner. +pub(super) async fn attachment_reserves_before_owner() -> Result<()> { + let mut cluster = Cluster::new().await?; + let scope = cluster.birth.route().history; + let owner = cluster.voters[0].owner.take().context("direct owner")?; + let root = owner.manifest().clone(); + drop(owner); + let (runtime, events) = CellGroupRuntime::start( + cluster.voters[0].log.clone(), + RaftRuntimeOptions { + max_requests: 1, + tick_interval_ms: None, + election_seed: Some([71; 32]), + ..Default::default() + }, + ) + .await?; + runtime + .attach(cluster.permit(0).await?, root.clone()) + .await?; + let duplicate = cluster.permit(0).await?; + let gate = cluster.voters[0].gate.clone(); + let release = gate.arm(fault::HOLD_SYNC); + let mut vote = Message::default(); + vote.set_msg_type(MessageType::MsgRequestVote); + vote.from = 2; + vote.to = 1; + vote.term = root.marker().position().term.get() + 1; + vote.index = root.marker().position().index.get(); + vote.log_term = root.marker().position().term.get(); + runtime.step(scope, vote.clone()).await?; + tokio::time::timeout(Duration::from_secs(5), gate.entered.notified()).await?; + let mut pending = Box::pin(runtime.step(scope, vote)); + assert!( + tokio::time::timeout(Duration::from_millis(30), pending.as_mut()) + .await + .is_err() + ); + assert!(matches!( + runtime.attach(duplicate, root).await, + Err(RuntimeError::Overloaded) + )); + drop(release); + tokio::time::timeout(Duration::from_secs(5), pending).await??; + runtime.close().await?; + drop(events); + drop(runtime); + cluster.close().await +} + +#[path = "cell_group_sql.rs"] +mod sql; +#[allow(unused_imports)] +pub(super) use sql::unproved_s3_responses; +pub(super) use sql::{ + alarms_without_clients, cleanup_failure_retains_outcome, effect_delivery_shutdown, + effect_identity_across_bindings, effect_leader_change, effect_lost_reply, effect_s3_lost_reply, + hot_and_cold_alarms, idle_cache_reopens_exact_base, shutdown_joins_alarm, +}; +#[allow(unused_imports)] +pub(super) use sql::{ + archive_checkpoint_frontier, archive_controller_publish, archive_controller_reconcile, + archived_groups_reclaim_independently, +}; +#[allow(unused_imports)] +pub(super) use sql::{archive_controller_publish_store, archive_controller_reconcile_store}; +#[allow(unused_imports)] +pub(super) use sql::{ + archive_dependencies, archive_progress_wakes_commands, full_backend_outage, + output_during_apply, s3_commands, s3_deadline, s3_enrollment, s3_enrollment_deadline, + s3_missing_sibling, s3_namespace_minimum, s3_receipt_deadline, s3_receipt_failures, + s3_receipt_minimums, s3_receipt_observations, s3_remote_deadlines, s3_remote_outputs, + s3_remote_slow_delivery, +}; +pub(super) use sql::{ + background_admission, command_admission_starts_before_poll, owned_command_failure, + owned_commands, +}; +pub(super) use sql::{ + controller_apply_faults, controller_close, controller_owners, controller_replication, +}; + +pub(super) async fn cell_receipts(rooted: bool) -> Result<()> { + sql::cell_receipts(rooted).await +} + +pub(super) async fn sql_pool_admission(rooted: bool) -> Result<()> { + sql::pool_admission(rooted).await +} + +pub(super) async fn reconcile_sql_producers(rooted: bool) -> Result<()> { + sql::reconcile_producers(rooted).await +} + +pub(super) async fn reconcile_sql_refusals(rooted: bool) -> Result<()> { + sql::reconcile_refusals(rooted).await +} + +pub(super) async fn initialize_cells(rooted: bool) -> Result<()> { + sql::initialize_cells(rooted).await +} + +pub(super) async fn owned_workspace_barriers() -> Result<()> { + sql::owned_workspace_barriers().await +} + +pub(super) async fn recover_initialization(rooted: bool) -> Result<()> { + sql::recover_initialization(rooted).await +} + +pub(super) async fn submission_rejections(rooted: bool) -> Result<()> { + sql::submission_rejections(rooted).await +} + +pub(super) use sql::{coherent_reads, follower_linearizable, query_lifecycle, recent_source_reads}; + +pub(super) use sql::apply_images; + +pub(super) use sql::{apply_cancellation, apply_failure}; + +pub(super) use sql::recover_apply; + +pub(super) use sql::registered_commands; + +pub(super) use sql::registered_queries; + +pub(super) use sql::routed_queries; + +pub(super) use sql::routed_commands; + +pub(super) async fn commands_over_tcp(rooted: bool) -> Result<()> { + Box::pin(sql::commands_over_tcp(rooted)).await +} + +pub(super) use sql::{ + enrollment_requires_registration, owned_enrollment, owned_enrollment_failure, +}; + +pub(super) use sql::{remote_follower_reads, remote_queries, remote_query_lifecycle}; + +pub(super) use sql::{nonmember_controller_queries, nonmember_queries, nonmember_query_revocation}; + +pub(super) use sql::cached_dispatch_execution; + +pub(super) use sql::{remote_command_lifecycle, remote_commands}; + +pub(super) use sql::{remote_enrollment, remote_enrollment_lifecycle}; + +pub(super) use sql::{receipt_lookup_cancelled, receipt_lookup_ingress, receipt_lookup_pending}; + +pub(super) use sql::{ + registration_close, registration_live, registration_recovery, registration_refusals, +}; + +pub(super) use sql::{ + group_quarantine_retains_producer_and_serves_healthy_group, + shared_engine_sync_failure_still_stops_all_groups, +}; + +pub(super) use sql::{ + provisioning_coalesces_and_serves, provisioning_joins_cancelled_close, + provisioning_recovers_selected_roots, provisioning_refusals_preserve_healthy_group, + provisioning_retries_retained_owner, +}; + +pub(super) use sql::checkpoint_completion_pressure; +pub(super) use sql::concurrent_checkpoints; +pub(super) use sql::repair_budget_under_load; +pub(super) use sql::{ + automatic_checkpoints, checkpoint_during_apply, controller_checkpoint_shutdown, +}; + +pub(super) use sql::{ + membership_delivery_fence, membership_lifecycle, membership_physical_delivery_fence, + membership_quorum_floor, membership_quorum_followers, membership_sync, +}; + +pub(super) use sql::{seed_minority, seed_source}; + +pub(super) use sql::seed_install; + +pub(super) use sql::seed_receiver_faults; + +pub(super) use sql::{seed_persistence, seed_transfer_revocation}; + +pub(super) use sql::{seed_crash, seed_process_fixture}; + +pub(super) use sql::{membership_crash, membership_process_fixture}; + +pub(super) use sql::{seed_retry, shared_inventory}; + +pub(super) async fn retained_log_admission(rooted: bool) -> Result<()> { + Box::pin(sql::retained_log_admission(rooted)).await +} + +#[allow(unused_imports)] +pub(super) use sql::physical_log_reclamation; + +pub(super) use sql::{host_many_cells, host_query_shutdown, host_registration_and_deadline}; + +pub(super) use sql::controller_routed_queries; diff --git a/crates/cellar/tests/support/cell_group_rustfs.rs b/crates/cellar/tests/support/cell_group_rustfs.rs new file mode 100644 index 000000000..62353be71 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_rustfs.rs @@ -0,0 +1,63 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Explicit local RustFS configuration; never fall back to memory or ambient IAM. +use anyhow::{ensure, Result}; +use object_store::{aws::AmazonS3Builder, path::Path, prefix::PrefixStore, ObjectStore}; +use std::{env, sync::Arc}; + +pub fn store(case: &str, rooted: bool) -> Result> { + configured_store(case, rooted, None) +} + +pub fn outage_store(case: &str, rooted: bool) -> Result> { + // A stopped backend must expose its actual connection failure promptly. + // Other RustFS cases keep their existing adapter retry policy. + configured_store( + case, + rooted, + Some(object_store::RetryConfig { + max_retries: 0, + ..Default::default() + }), + ) +} + +pub fn control_directory() -> Result { + let path = std::path::PathBuf::from(env::var("CELLAR_CELL_GROUPS_RUSTFS_CONTROL")?); + ensure!( + path.is_absolute() && path.is_dir(), + "explicit owned backend control" + ); + Ok(path) +} + +fn configured_store( + case: &str, + rooted: bool, + retry: Option, +) -> Result> { + let endpoint = env::var("CELLAR_CELL_GROUPS_RUSTFS_ENDPOINT")?; + ensure!( + endpoint.starts_with("http://127.0.0.1:"), + "local RustFS endpoint" + ); + let prefix = env::var("CELLAR_CELL_GROUPS_RUSTFS_PREFIX")?; + ensure!(!prefix.is_empty(), "isolated RustFS prefix"); + let prefix = Path::parse(format!( + "{prefix}/{case}/{}", + if rooted { "extents" } else { "materialized" } + ))?; + let mut builder = AmazonS3Builder::new() + .with_bucket_name(env::var("CELLAR_CELL_GROUPS_RUSTFS_BUCKET")?) + .with_endpoint(endpoint) + .with_access_key_id(env::var("AWS_ACCESS_KEY_ID")?) + .with_secret_access_key(env::var("AWS_SECRET_ACCESS_KEY")?) + .with_region("us-east-1") + .with_allow_http(true) + .with_conditional_put(object_store::aws::S3ConditionalPut::ETagMatch); + if let Some(retry) = retry { + builder = builder.with_retry(retry); + } + let store = builder.build()?; + Ok(Arc::new(PrefixStore::new(store, prefix))) +} diff --git a/crates/cellar/tests/support/cell_group_s3_commands.rs b/crates/cellar/tests/support/cell_group_s3_commands.rs new file mode 100644 index 000000000..dbd6e2b5f --- /dev/null +++ b/crates/cellar/tests/support/cell_group_s3_commands.rs @@ -0,0 +1,723 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! SQL output proofs on the actual controller and archive lane. +#[path = "cell_group_archive_dependencies.rs"] +mod dependencies; +pub(crate) use dependencies::archive_dependencies; + +#[path = "cell_group_s3_output.rs"] +mod output; +pub(crate) use output::output_during_apply; + +#[path = "cell_group_s3_outage.rs"] +mod outage; +pub(crate) use outage::full_backend_outage; + +#[path = "cell_group_s3_enrollment.rs"] +mod enrollment; +#[path = "cell_group_s3_receipts.rs"] +mod receipts; +#[path = "cell_group_s3_remote.rs"] +mod remote; +use super::*; +pub(crate) use enrollment::{s3_enrollment, s3_enrollment_deadline}; +use futures_util::TryStreamExt; +use object_store::{path::Path as ObjectPath, ObjectStore}; +pub(crate) use receipts::{ + s3_receipt_deadline, s3_receipt_failures, s3_receipt_minimums, s3_receipt_observations, +}; +pub(crate) use remote::{s3_remote_deadlines, s3_remote_outputs, s3_remote_slow_delivery}; +use std::sync::atomic::{AtomicUsize, Ordering}; + +struct Harness { + fixture: RuntimeCluster, + route: GroupRoute, + cell: CellHistory, + controllers: Vec, + dispatcher: CellGroupDispatcher, + class: Arc, + archive: CellGroupArchiveRegistration, + remote: Arc, + calls: Arc, + initializations: Arc, + retry_until: u64, +} + +enum Resources { + SlowMaintenance, + Retention(CellGroupLogRetentionLimits), + Storage { + root: tempfile::TempDir, + filesystem: Arc, + }, +} + +impl Harness { + async fn new( + rooted: bool, + store: Arc, + max_commands: usize, + ) -> Result { + Self::with_minimum(rooted, store, max_commands, Durability::Quorum).await + } + + async fn with_minimum( + rooted: bool, + store: Arc, + max_commands: usize, + minimum: Durability, + ) -> Result { + Self::configured(rooted, store, max_commands, minimum, None).await + } + + async fn configured( + rooted: bool, + store: Arc, + max_commands: usize, + minimum: Durability, + result_size: Option, + ) -> Result { + Self::configured_transport(rooted, store, max_commands, minimum, result_size, None).await + } + + async fn configured_transport( + rooted: bool, + store: Arc, + max_commands: usize, + minimum: Durability, + result_size: Option, + send_buffer: Option, + ) -> Result { + Box::pin(Self::configured_retention( + rooted, + store, + max_commands, + minimum, + result_size, + send_buffer, + None, + )) + .await + } + + async fn configured_retention( + rooted: bool, + store: Arc, + max_commands: usize, + minimum: Durability, + result_size: Option, + send_buffer: Option, + retention: Option, + ) -> Result { + Self::configured_resources( + rooted, + store, + max_commands, + minimum, + result_size, + send_buffer, + retention.map(Resources::Retention), + ) + .await + } + + async fn configured_resources( + rooted: bool, + store: Arc, + max_commands: usize, + minimum: Durability, + result_size: Option, + send_buffer: Option, + resources: Option, + ) -> Result { + let calls = Arc::new(AtomicUsize::new(0)); + let count = calls.clone(); + let initializations = Arc::new(AtomicUsize::new(0)); + let init_count = initializations.clone(); + let class = Arc::new( + RaftClass::new("Counter", 1, [81; 32])? + .initializer([83; 32], move |_, transaction| { + init_count.fetch_add(1, Ordering::SeqCst); + transaction + .execute_batch("CREATE TABLE value(n); INSERT INTO value VALUES(7);")?; + Ok(()) + })? + .command( + "Add", + if result_size.is_some() { + [84; 32] + } else { + [82; 32] + }, + move |context, input| { + count.fetch_add(1, Ordering::SeqCst); + context.sql().execute("UPDATE value SET n=n+1", [])?; + Ok(result_size.map_or_else(|| input.to_vec(), |length| vec![7; length])) + }, + )?, + ); + let mut limits = controller::limits(); + limits.max_commands = max_commands; + if matches!(resources, Some(Resources::SlowMaintenance)) { + limits.retry_interval_ms = 60_000; + } + let (mut fixture, route, cell, _) = if let Some(Resources::Retention(retention)) = resources + { + let filesystem = Arc::new(DirectFileSystem); + let cluster = Box::pin(Cluster::with_store_birth( + RaftRuntimeOptions::default(), + tempfile::tempdir()?, + filesystem.clone(), + class.schema_digest(), + store, + ClusterBirthOptions { + group: GroupId::new([3; 16])?, + image_storage: if rooted { + RaftImageStorage::Extents + } else { + RaftImageStorage::Materialized + }, + minimum, + log: ClusterLogLimits { + retention, + ..Default::default() + }, + }, + )) + .await?; + Box::pin(registered::setup_cluster(rooted, cluster, filesystem)).await? + } else if let Some(Resources::Storage { root, filesystem }) = resources { + let cluster = Cluster::with_store_birth( + RaftRuntimeOptions::default(), + root, + filesystem.clone(), + class.schema_digest(), + store, + ClusterBirthOptions { + image_storage: if rooted { + RaftImageStorage::Extents + } else { + RaftImageStorage::Materialized + }, + group: GroupId::new([3; 16])?, + minimum, + log: ClusterLogLimits::default(), + }, + ) + .await?; + Box::pin(registered::setup_cluster(rooted, cluster, filesystem)).await? + } else { + Box::pin(registered::setup_store_minimum( + rooted, &class, store, minimum, + )) + .await? + }; + let (archive, remote) = + archive_controller::registration_minimum(&fixture, &route, rooted, minimum).await?; + if send_buffer.is_some() { + fixture.connect_tcp_with_query_buffer(send_buffer).await?; + } + let controllers = controller::start_with_limits(&mut fixture, limits).await?; + controllers[0].archive(archive.clone())?.response().await?; + archive_controller::published( + &controllers[0], + route.history, + fixture.runtimes[0] + .status(route.history) + .await? + .applied + .index, + ) + .await?; + let dispatcher = CellGroupDispatcher::new( + &fixture.cluster.domain, + fixture.cluster.directory.clone(), + class.clone(), + VersionKeys::new( + VersionSigningKey::new([89; 32], u64::MAX)?, + vec![], + Duration::from_secs(30), + )?, + CellDispatchLimits { + max_cached_routes: 32, + route_cache_bytes: 256 * 1024, + max_invocations: 8, + retained_bytes: 1024 * 1024, + max_input_bytes: 1024, + }, + )? + .with_s3(&controllers[0])?; + let retry_until = fixture + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms() as u64 + 60_000 }) + .await?; + Ok(Self { + fixture, + route, + cell, + controllers, + dispatcher, + class, + archive, + remote, + calls, + initializations, + retry_until, + }) + } + + async fn command( + &self, + sequence: u64, + input: &[u8], + durability: Durability, + timeout: Duration, + ) -> Result { + self.command_using( + sequence, + input, + durability, + timeout, + self.fixture.cell_options(&self.cell).await?, + ) + .await + } + + async fn command_using( + &self, + sequence: u64, + input: &[u8], + durability: Durability, + timeout: Duration, + options: CellSqlOptions, + ) -> Result { + let invocation = self + .dispatcher + .command( + self.cell.clone(), + CellCommandRequest { + id: CommandId::new(ClientId::new([188; 16])?, sequence)?, + contract: self.class.contract("Add").context("Add")?, + input: input.to_vec(), + durability, + retry_until_ms: self.retry_until, + }, + CellRoutingLimits { + timeout, + max_attempts: 4, + }, + None, + ) + .await?; + Ok(self.controllers[0].command(invocation, options)?) + } + + async fn waiting(&self) -> Result<()> { + controller::wait_for(async || { + Ok(self.controllers[0].usage().retained_s3 == 1 + && self.fixture.appliers[&(0, self.route.history)] + .0 + .usage()? + .active_cells + == 0) + }) + .await + } + + async fn close(self) -> Result<()> { + for controller in &self.controllers { + controller.close().await?; + } + drop(self.dispatcher); + drop(self.archive); + self.fixture.close().await + } +} + +pub(crate) async fn archive_progress_wakes_commands(rooted: bool) -> Result<()> { + let h = Harness::configured_resources( + rooted, + Arc::new(object_store::memory::InMemory::new()), + 1, + Durability::Quorum, + None, + None, + Some(Resources::SlowMaintenance), + ) + .await?; + // Each response must observe its own verified publication. The entire + // sequence fits inside one maintenance interval without polling archive + // status or issuing a manual archive request between commands. + tokio::time::timeout(Duration::from_secs(20), async { + let mut previous = LogIndex::new(0); + for sequence in 1..=8u64 { + let payload = sequence.to_be_bytes(); + let response = h + .command(sequence, &payload, Durability::S3, Duration::from_secs(20)) + .await? + .response() + .await?; + assert_eq!(response.durability(), Durability::S3); + let local = response.local_receipt().context("local S3 receipt")?; + assert_eq!(local.receipt.result(), payload); + assert!(local.position.index > previous); + previous = local.position.index; + } + let published = h.controllers[0] + .archive_status(h.route.history)? + .context("archive registration")? + .published + .context("published archive marker")?; + assert!(published.position().index >= previous); + assert_eq!(h.calls.load(Ordering::Acquire), 8); + Ok::<_, anyhow::Error>(()) + }) + .await + .context("S3 completion waited for periodic maintenance")??; + h.close().await +} + +pub(crate) async fn s3_namespace_minimum(rooted: bool, store: Arc) -> Result<()> { + let h = Harness::with_minimum(rooted, store, 1, Durability::S3).await?; + let gate = h.remote.hold_publication(); + let call = h + .command(1, b"minimum", Durability::Quorum, Duration::from_secs(15)) + .await?; + tokio::time::timeout(Duration::from_secs(10), gate.entered.notified()).await?; + let mut waiting = Box::pin(call.response()); + assert!( + tokio::time::timeout(Duration::from_millis(30), waiting.as_mut()) + .await + .is_err() + ); + gate.release.notify_one(); + let response = waiting.await?; + assert_eq!(response.durability(), Durability::S3); + assert_eq!( + response + .local_receipt() + .expect("local command receipt") + .receipt + .minimum_durability(), + Durability::S3 + ); + assert_eq!( + response + .local_receipt() + .expect("local command receipt") + .receipt + .result(), + b"minimum" + ); + assert_eq!(h.calls.load(Ordering::SeqCst), 1); + drop(response); + h.close().await +} + +pub(crate) async fn s3_commands( + rooted: bool, + store: Arc, +) -> Result<()> { + let h = Harness::new(rooted, store, 2).await?; + // A controller in the same execution domain still cannot lend another + // runtime's archive authority to this producer. + let foreign = h.dispatcher.clone().with_s3(&h.controllers[1])?; + let invocation = foreign + .command( + h.cell.clone(), + CellCommandRequest { + id: CommandId::new(ClientId::new([188; 16])?, 1)?, + contract: h.class.contract("Add").context("Add")?, + input: b"first".to_vec(), + durability: Durability::S3, + retry_until_ms: h.retry_until, + }, + CellRoutingLimits { + timeout: Duration::from_secs(15), + max_attempts: 4, + }, + None, + ) + .await?; + let error = h.controllers[0] + .command(invocation, h.fixture.cell_options(&h.cell).await?)? + .response() + .await + .err() + .context("foreign S3 runtime refused")?; + assert!(!error.ambiguous); + assert!(matches!( + error.cause.as_ref(), + CellDispatchError::Runtime(RuntimeError::Invalid(_)) + )); + assert_eq!(h.calls.load(Ordering::SeqCst), 0); + h.remote.outage.store(true, Ordering::Release); + let call = h + .command(1, b"first", Durability::S3, Duration::from_secs(15)) + .await?; + h.waiting().await?; + assert_eq!( + h.fixture.appliers[&(0, h.route.history)] + .0 + .usage()? + .retained_operations, + 0 + ); + // The same cell commits its successor during the first S3 wait. The proof + // must read the original receipt from that later independently replayed image. + let next = h + .command(2, b"second", Durability::Quorum, Duration::from_secs(15)) + .await? + .response() + .await?; + assert_eq!(next.durability(), Durability::Quorum); + assert_eq!( + next.local_receipt() + .expect("local command receipt") + .receipt + .result(), + b"second" + ); + drop(next); + let gate = h.remote.hold_publication(); + h.remote.outage.store(false, Ordering::Release); + tokio::time::timeout(Duration::from_secs(10), gate.entered.notified()).await?; + let mut waiting = Box::pin(call.response()); + assert!( + tokio::time::timeout(Duration::from_millis(30), waiting.as_mut()) + .await + .is_err() + ); + gate.release.notify_one(); + let response = waiting.await?; + assert_eq!(response.durability(), Durability::S3); + assert_eq!( + response + .local_receipt() + .expect("local command receipt") + .receipt + .result(), + b"first" + ); + assert_eq!( + response + .local_receipt() + .expect("local command receipt") + .version + .sequence + .get(), + 2 + ); + assert_eq!(h.archive.tails.usage().operations, 1); + assert_eq!(h.calls.load(Ordering::SeqCst), 2); + // A Quorum duplicate still honors the stored S3 minimum. A retained response + // keeps its physical proof and remote hold until the final consumer drops it. + let mut duplicate = Box::pin( + h.command(1, b"first", Durability::Quorum, Duration::from_secs(15)) + .await? + .response(), + ); + controller::wait_for(async || Ok(h.controllers[0].usage().retained_s3 == 2)).await?; + assert!( + tokio::time::timeout(Duration::from_millis(30), duplicate.as_mut()) + .await + .is_err() + ); + drop(response); + let response = duplicate.await?; + assert_eq!(response.durability(), Durability::S3); + assert_eq!( + response + .local_receipt() + .expect("local command receipt") + .receipt + .result(), + b"first" + ); + assert_eq!( + response + .local_receipt() + .expect("local command receipt") + .version + .sequence + .get(), + 3 + ); + assert_eq!(h.calls.load(Ordering::SeqCst), 2); + drop(response); + controller::wait_for(async || Ok(h.controllers[0].usage().retained_s3 == 0)).await?; + assert!(h.controllers[0].group_failure(h.route.history)?.is_none()); + h.close().await +} + +pub(crate) async fn s3_deadline( + rooted: bool, + store: Arc, + cancel: bool, +) -> Result<()> { + let h = Harness::new(rooted, store, 1).await?; + h.remote.outage.store(true, Ordering::Release); + let call = h + .command(1, b"first", Durability::S3, Duration::from_millis(800)) + .await?; + h.waiting().await?; + if cancel { + drop(call); + } else { + let error = call.response().await.err().context("S3 deadline")?; + assert!(error.ambiguous); + assert!(matches!( + error.cause.as_ref(), + CellDispatchError::Runtime(RuntimeError::DeadlineExceeded) + )); + } + controller::wait_for(async || { + Ok(h.controllers[0].usage().retained_commands == 0 + && h.controllers[0].usage().retained_s3 == 0) + }) + .await?; + assert!(h.controllers[0].group_failure(h.route.history)?.is_none()); + assert_eq!( + h.fixture.appliers[&(0, h.route.history)] + .0 + .usage()? + .active_cells, + 0 + ); + let next = h + .command(2, b"second", Durability::Quorum, Duration::from_secs(15)) + .await? + .response() + .await?; + assert_eq!( + next.local_receipt() + .expect("local command receipt") + .receipt + .result(), + b"second" + ); + drop(next); + h.remote.outage.store(false, Ordering::Release); + let response = h + .command(1, b"first", Durability::S3, Duration::from_secs(15)) + .await? + .response() + .await?; + assert_eq!(response.durability(), Durability::S3); + assert_eq!( + response + .local_receipt() + .expect("local command receipt") + .receipt + .result(), + b"first" + ); + assert_eq!(h.calls.load(Ordering::SeqCst), 2); + drop(response); + h.close().await +} + +pub(crate) async fn s3_missing_sibling(rooted: bool, store: Arc) -> Result<()> { + let h = Harness::new(rooted, store, 1).await?; + let result = h + .command(1, b"first", Durability::Quorum, Duration::from_secs(15)) + .await? + .response() + .await?; + let index = result + .local_receipt() + .expect("local command receipt") + .position + .index; + drop(result); + archive_controller::published(&h.controllers[0], h.route.history, index).await?; + let marker = h.controllers[0] + .archive_status(h.route.history)? + .context("published archive")? + .published + .context("archive marker")?; + let objects = h.remote.list(None).try_collect::>().await?; + let mut missing = None; + for object in objects + .iter() + .filter(|object| object.location.as_ref().contains("/indexes/")) + { + let value: serde_json::Value = + serde_json::from_slice(&h.remote.get(&object.location).await?.bytes().await?)?; + let Some(chunks) = value.get("chunks").and_then(|v| v.as_array()) else { + continue; + }; + let selected: GroupManifest = serde_json::from_value(value["marker"].clone())?; + if selected != marker { + continue; + } + let prefix = object + .location + .as_ref() + .split_once("/generations/") + .context("generation prefix")? + .0; + let path = |chunk: &serde_json::Value| -> Result { + let object = &chunk["object"]; + let digest: Vec = serde_json::from_value(object["digest"].clone())?; + Ok(ObjectPath::from(format!( + "{prefix}/generations/{}/chunks/{}-{}", + object["generation"], + object["length"], + hex::encode(digest) + ))) + }; + let a = chunks + .iter() + .filter(|chunk| chunk["part"]["cell"] == "registered-a") + .map(&path) + .collect::>>()?; + for chunk in chunks + .iter() + .filter(|chunk| chunk["part"]["cell"] == "registered-b") + { + let candidate = path(chunk)?; + if !a.contains(&candidate) { + missing = Some(candidate); + break; + } + } + if missing.is_some() { + break; + } + } + let missing = missing.context("B dependency outside A closure")?; + h.remote.inner.delete(&missing).await?; + // A still has its committed receipt. A cached published marker must not + // acknowledge it while any other cell's selected dependency is missing. + let error = h + .command(1, b"first", Durability::S3, Duration::from_secs(15)) + .await? + .response() + .await + .err() + .context("incomplete group must refuse S3")?; + assert!(error.ambiguous); + assert!(matches!( + error.cause.as_ref(), + CellDispatchError::Archive(_) + )); + assert!(h.controllers[0].group_failure(h.route.history)?.is_none()); + let result = h + .command(2, b"second", Durability::Quorum, Duration::from_secs(15)) + .await? + .response() + .await?; + assert_eq!( + result + .local_receipt() + .expect("local command receipt") + .receipt + .result(), + b"second" + ); + assert_eq!(h.calls.load(Ordering::SeqCst), 2); + drop(result); + h.close().await +} diff --git a/crates/cellar/tests/support/cell_group_s3_enrollment.rs b/crates/cellar/tests/support/cell_group_s3_enrollment.rs new file mode 100644 index 000000000..16b34dfd9 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_s3_enrollment.rs @@ -0,0 +1,235 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Original creation records survive delayed archival and later cell mutations. +use super::*; + +impl Harness { + async fn enroll( + &self, + dispatcher: &CellGroupDispatcher, + creation: u8, + durability: Durability, + timeout: Duration, + ) -> Result { + let invocation = dispatcher + .enroll( + self.cell.clone(), + CellEnrollmentRequest { + creation: ProposalId::new([creation; 16])?, + durability, + }, + CellRoutingLimits { + timeout, + max_attempts: 4, + }, + None, + ) + .await?; + Ok(self.controllers[0].enroll(invocation, self.fixture.cell_options(&self.cell).await?)?) + } + + async fn creation(&self) -> Result { + Ok(self.fixture.runtimes[0] + .enrollment_state(self.route.history, self.cell.clone()) + .await? + .record() + .context("committed creation")? + .clone()) + } +} + +pub(crate) async fn s3_enrollment(rooted: bool, store: Arc) -> Result<()> { + for minimum in [Durability::Quorum, Durability::S3] { + let backend = Arc::new(object_store::prefix::PrefixStore::new( + store.clone(), + ObjectPath::from(if minimum == Durability::S3 { + "namespace-s3" + } else { + "namespace-quorum" + }), + )); + let mut h = Harness::with_minimum(rooted, backend, 2, minimum).await?; + h.cell.cell = CellId::new("new-s3-enrollment")?; + let requested = if minimum == Durability::S3 { + Durability::Quorum + } else { + Durability::S3 + }; + let foreign = h.dispatcher.clone().with_s3(&h.controllers[1])?; + let error = h + .enroll(&foreign, 210, requested, Duration::from_secs(15)) + .await? + .response() + .await + .err() + .context("foreign enrollment runtime")?; + assert!(!error.ambiguous); + assert!(matches!( + error.cause.as_ref(), + CellDispatchError::Runtime(RuntimeError::Invalid(_)) + )); + assert_eq!(h.initializations.load(Ordering::SeqCst), 0); + h.remote.outage.store(true, Ordering::Release); + let call = h + .enroll(&h.dispatcher, 210, requested, Duration::from_secs(15)) + .await?; + h.waiting().await?; + let original = h.creation().await?; + assert_eq!(original.creation(), ProposalId::new([210; 16])?); + assert_eq!(h.initializations.load(Ordering::SeqCst), 1); + assert_eq!( + h.fixture.appliers[&(0, h.route.history)] + .0 + .usage()? + .retained_operations, + 0 + ); + if minimum == Durability::Quorum { + let changed = h + .command( + 1, + b"after-initialization", + Durability::Quorum, + Duration::from_secs(15), + ) + .await? + .response() + .await?; + assert_eq!( + changed + .local_receipt() + .expect("local command receipt") + .version + .sequence + .get(), + 2 + ); + drop(changed); + } + let gate = h.remote.hold_publication(); + h.remote.outage.store(false, Ordering::Release); + tokio::time::timeout(Duration::from_secs(10), gate.entered.notified()).await?; + let mut response = Box::pin(call.response()); + assert!( + tokio::time::timeout(Duration::from_millis(30), response.as_mut()) + .await + .is_err() + ); + gate.release.notify_one(); + let response = response.await?; + assert_eq!(response.durability(), Durability::S3); + assert_eq!(response.record(), &original); + assert_eq!(h.archive.tails.usage().operations, 1); + assert_eq!(h.initializations.load(Ordering::SeqCst), 1); + let duplicate = h + .enroll(&h.dispatcher, 210, Durability::S3, Duration::from_secs(15)) + .await?; + controller::wait_for(async || Ok(h.controllers[0].usage().retained_s3 == 2)).await?; + let mut duplicate = Box::pin(duplicate.response()); + assert!( + tokio::time::timeout(Duration::from_millis(30), duplicate.as_mut()) + .await + .is_err() + ); + drop(response); + let response = duplicate.await?; + assert_eq!(response.record(), &original); + assert_eq!(response.durability(), Durability::S3); + assert_eq!(h.initializations.load(Ordering::SeqCst), 1); + drop(response); + controller::wait_for(async || Ok(h.controllers[0].usage().retained_s3 == 0)).await?; + let error = h + .enroll(&h.dispatcher, 211, Durability::S3, Duration::from_secs(15)) + .await? + .response() + .await + .err() + .context("conflicting creation")?; + assert!(matches!( + error.cause.as_ref(), + CellDispatchError::Execution(ExecutionError::Refused(CommandFailure::CommandConflict)) + )); + assert_eq!(h.initializations.load(Ordering::SeqCst), 1); + assert!(h.controllers[0].group_failure(h.route.history)?.is_none()); + h.close().await?; + } + Ok(()) +} + +pub(crate) async fn s3_enrollment_deadline( + rooted: bool, + store: Arc, + cancel: bool, +) -> Result<()> { + let mut h = Harness::new(rooted, store, 1).await?; + h.cell.cell = CellId::new("expired-s3-enrollment")?; + h.remote.outage.store(true, Ordering::Release); + let call = h + .enroll( + &h.dispatcher, + 212, + Durability::S3, + Duration::from_millis(800), + ) + .await?; + h.waiting().await?; + let original = h.creation().await?; + if cancel { + drop(call); + } else { + let error = call + .response() + .await + .err() + .context("S3 enrollment deadline")?; + assert!(error.ambiguous); + assert!(matches!( + error.cause.as_ref(), + CellDispatchError::Runtime(RuntimeError::DeadlineExceeded) + )); + } + controller::wait_for(async || { + Ok(h.controllers[0].usage().retained_commands == 0 + && h.controllers[0].usage().retained_s3 == 0) + }) + .await?; + assert!(h.controllers[0].group_failure(h.route.history)?.is_none()); + assert_eq!( + h.fixture.appliers[&(0, h.route.history)] + .0 + .usage()? + .retained_operations, + 0 + ); + let changed = h + .command( + 1, + b"after-deadline", + Durability::Quorum, + Duration::from_secs(15), + ) + .await? + .response() + .await?; + assert_eq!( + changed + .local_receipt() + .expect("local command receipt") + .version + .sequence + .get(), + 2 + ); + drop(changed); + h.remote.outage.store(false, Ordering::Release); + let response = h + .enroll(&h.dispatcher, 212, Durability::S3, Duration::from_secs(15)) + .await? + .response() + .await?; + assert_eq!(response.record(), &original); + assert_eq!(response.durability(), Durability::S3); + assert_eq!(h.initializations.load(Ordering::SeqCst), 1); + drop(response); + h.close().await +} diff --git a/crates/cellar/tests/support/cell_group_s3_outage.rs b/crates/cellar/tests/support/cell_group_s3_outage.rs new file mode 100644 index 000000000..612784dac --- /dev/null +++ b/crates/cellar/tests/support/cell_group_s3_outage.rs @@ -0,0 +1,558 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! A complete backend outage bounds Quorum admission without inventing S3 proof. +use super::*; +use raft::Storage as _; +use std::path::PathBuf; + +#[path = "cell_group_backend_control.rs"] +mod backend_control; + +const ENTRY_LIMIT: usize = 9; +const OFFLINE_COMMANDS: u64 = 6; +const REFUSED_COMMAND: u64 = OFFLINE_COMMANDS + 1; + +async fn gateway( + h: &Harness, + store: Arc, +) -> Result { + let catalog = Catalog::new(store, Default::default())?; + let config = h.fixture.cluster.directory.namespace_config().clone(); + let namespace = h + .fixture + .cluster + .domain + .run(catalog.existing_namespace(config)) + .await??; + Ok(CellGroupDispatcher::new( + &h.fixture.cluster.domain, + RangeDirectory::new(namespace, Default::default())?, + h.class.clone(), + VersionKeys::new( + VersionSigningKey::new([89; 32], u64::MAX)?, + vec![], + Duration::from_secs(30), + )?, + CellDispatchLimits { + max_cached_routes: 32, + route_cache_bytes: 256 * 1024, + max_invocations: 8, + retained_bytes: 1024 * 1024, + max_input_bytes: 1024, + }, + )? + .with_s3(&h.controllers[0])?) +} + +async fn state(h: &Harness, index: LogIndex, value: u64) -> Result<()> { + controller::applied(&h.fixture, h.route.history, index).await?; + let b = history(&h.route, "registered-b")?; + for runtime in &h.fixture.runtimes { + for (cell, expected) in [(&h.cell, value), (&b, 0)] { + let selected = runtime + .reconciliation_authority(h.route.history, cell.clone()) + .await?; + assert_eq!(selected.cell().version.sequence.get(), expected + 1); + let pin = selected.image().clone(); + h.fixture + .cluster + .domain + .blocking(move || dependencies::image_value(pin, expected))? + .await??; + } + assert_eq!(h.controllers[0].usage().quarantined_groups, 0); + } + assert_eq!(h.calls.load(Ordering::SeqCst) as u64, value); + Ok(()) +} + +async fn initialize_at_capacity(h: &Harness) -> Result<()> { + let cell = history(&h.route, "outage-new")?; + let authority = h.fixture.runtimes[0] + .enrollment_authority(h.route.history, cell.clone()) + .await?; + let observed = h.initializations.clone(); + let result = h.fixture.appliers[&(0, h.route.history)] + .0 + .initialize( + h.fixture.cell_options(&cell).await?, + authority, + CellEnrollmentAttempt { + creation: ProposalId::new([203; 16])?, + preview: ImageId::new([203; 16])?, + inputs: cellar::storage::CommandInputs { + now_unix_millis: 10, + random: [17; 32], + }, + }, + move |_| { + observed.fetch_add(1, Ordering::SeqCst); + Ok(()) + }, + ) + .await; + assert!(matches!( + result, + Err(ExecutionError::Refused(CommandFailure::Overloaded)) + )); + assert_eq!(h.initializations.load(Ordering::SeqCst), 0); + Ok(()) +} + +async fn restored_probe( + h: &Harness, + store: &Arc, + path: &ObjectPath, + expected: &[u8], + deadline: u64, +) -> Result<()> { + let mut last_transient = None; + let result = h + .fixture + .cluster + .domain + .run(async { + cellar::asyncrt::timeout_at(deadline, async { + loop { + let read = async { store.get(path).await?.bytes().await }.await; + match read { + Ok(bytes) => return Ok(bytes), + // Container health can precede S3 data readiness. Retry only + // the idempotent read; absent/authentication failures remain + // failures and no operation receives a new deadline. + Err(error @ object_store::Error::Generic { .. }) => { + last_transient = Some(error) + } + Err(error) => return Err(error), + } + cellar::asyncrt::sleep(Duration::from_millis(5)).await; + } + }) + .await + }) + .await?; + let bytes = match result { + Ok(read) => read?, + Err(_) => anyhow::bail!( + "S3 data readiness exhausted the original invocation deadline: {last_transient:?}" + ), + }; + assert_eq!(bytes.as_ref(), expected); + Ok(()) +} + +async fn run( + rooted: bool, + store: Arc, + control_directory: Option, +) -> Result<()> { + // This adapter is shared by catalog and archive. In real RustFS mode it + // never injects an outage: the owning runner actually stops the container. + let global = archive_controller::archive_store::Store::around(store); + let control = backend_control::Control::new(global.clone(), control_directory, rooted); + let mut retention = CellGroupLogRetentionLimits::default(); + retention.group.entries = ENTRY_LIMIT; + retention.progress_group.entries = 1; + let h = Box::pin(Harness::configured_retention( + rooted, + global.clone(), + 2, + Durability::Quorum, + None, + None, + Some(retention), + )) + .await?; + let group = h.fixture.cluster.voters[0] + .log + .cell_group(&h.route.history)?; + // This registered fixture enrolls two cells. Genesis is snapshot index one; + // the retained inventory is one election no-op and two enrollments. + for voter in &h.fixture.cluster.voters { + let selected = voter.log.cell_group(&h.route.history)?; + assert_eq!(selected.first_index()?, 2); + let entries = selected.entries( + selected.first_index()?, + selected.last_index()? + 1, + None, + raft::GetEntriesContext::empty(false), + )?; + assert_eq!( + entries.iter().map(|entry| entry.index).collect::>(), + vec![2, 3, 4] + ); + assert!(entries[0].data.is_empty()); + assert!(entries[1..].iter().all(|entry| !entry.data.is_empty())); + assert_eq!( + selected.log_retention_usage()?.retained_entries, + entries.len() + ); + } + let warm = h + .command(1, b"warm", Durability::Quorum, Duration::from_secs(15)) + .await? + .response() + .await?; + let warm_index = warm + .local_receipt() + .expect("local command receipt") + .position + .index; + assert_eq!(warm.durability(), Durability::Quorum); + drop(warm); + archive_controller::published(&h.controllers[0], h.route.history, warm_index).await?; + let gateway_store = archive_controller::archive_store::Store::around(global.clone()); + let remote = remote::Remote::start(&h, gateway(&h, gateway_store.clone()).await?).await?; + // Warm the receiver too. It must still perform fresh discovery after stop. + let found = remote::command_using( + &h, + &remote, + remote::command_request(&h, 1, b"warm", Durability::Quorum)?, + CellRoutingLimits { + timeout: Duration::from_secs(15), + max_attempts: 4, + }, + ) + .await?; + assert_eq!(found.result(), b"warm"); + drop(found); + assert_eq!(h.calls.load(Ordering::SeqCst), 1); + let fresh_gets = gateway_store.gets.load(Ordering::SeqCst); + let probe = ObjectPath::from("backend-probe"); + let probe_bytes = vec![29_u8; 128]; + global + .put_opts( + &probe, + probe_bytes.clone().into(), + object_store::PutOptions::from(object_store::PutMode::Create), + ) + .await?; + let before_checkpoint = group.checkpoint_manifest()?; + let before_first = group.first_index()?; + control + .set_running(&h.fixture.cluster.domain, false) + .await?; + let failed_probe = tokio::time::timeout(Duration::from_secs(2), global.get(&probe)).await?; + assert!( + failed_probe.is_err(), + "stopped backend must refuse actual S3 I/O" + ); + + let remote_request = CellCommandRequest { + id: CommandId::new(ClientId::new([187; 16])?, 1)?, + contract: h.class.contract("Add").context("Add")?, + input: b"remote-outage".to_vec(), + durability: Durability::Quorum, + retry_until_ms: h.retry_until, + }; + let mut remote_invocation = h + .dispatcher + .command( + h.cell.clone(), + remote_request, + CellRoutingLimits { + timeout: Duration::from_secs(15), + max_attempts: 4, + }, + None, + ) + .await?; + remote_invocation.begin().await?; + let refused = remote_invocation + .execute_remote(remote.transport(), MemberId::new(1)?) + .await; + assert!( + refused.is_err(), + "remote reception requires fresh catalog discovery" + ); + assert!(gateway_store.gets.load(Ordering::SeqCst) > fresh_gets); + assert_eq!(h.calls.load(Ordering::SeqCst), 1); + + let pending = h + .command(2, b"archive-wait", Durability::S3, Duration::from_secs(2)) + .await?; + let mut pending = Box::pin(pending.response()); + match futures_util::future::select(pending.as_mut(), Box::pin(h.waiting())).await { + futures_util::future::Either::Left((result, _)) => match result { + Ok(_) => anyhow::bail!("outage returned an unproved S3 response"), + Err(error) => { + anyhow::bail!("outage command ended before retaining S3 settlement: {error}") + } + }, + futures_util::future::Either::Right((result, _)) => result?, + } + assert!( + tokio::time::timeout(Duration::from_millis(30), pending.as_mut()) + .await + .is_err() + ); + let mut latest = warm_index; + for sequence in 3..=OFFLINE_COMMANDS { + let response = h + .command( + sequence, + b"quorum-offline", + Durability::Quorum, + Duration::from_secs(15), + ) + .await? + .response() + .await?; + assert_eq!(response.durability(), Durability::Quorum); + assert_eq!( + response + .local_receipt() + .expect("local command receipt") + .receipt + .result(), + b"quorum-offline" + ); + latest = response + .local_receipt() + .expect("local command receipt") + .position + .index; + drop(response); + } + state(&h, latest, OFFLINE_COMMANDS).await?; + let usage = group.log_retention_usage()?; + assert_eq!( + (usage.retained_entries, usage.reserved_entries), + (ENTRY_LIMIT, 0) + ); + let refused = h + .command( + REFUSED_COMMAND, + b"after-ceiling", + Durability::Quorum, + Duration::from_secs(15), + ) + .await? + .response() + .await + .err() + .context("retention ceiling must refuse new command")?; + assert!(matches!( + refused.cause.as_ref(), + CellDispatchError::Execution(ExecutionError::Refused(CommandFailure::Overloaded)) + )); + assert_eq!(h.calls.load(Ordering::SeqCst) as u64, OFFLINE_COMMANDS); + initialize_at_capacity(&h).await?; + let known = h + .command(1, b"warm", Durability::Quorum, Duration::from_secs(15)) + .await? + .response() + .await?; + assert_eq!(known.durability(), Durability::Quorum); + assert_eq!( + known + .local_receipt() + .expect("local command receipt") + .receipt + .result(), + b"warm" + ); + drop(known); + let error = pending + .await + .err() + .context("S3 output cannot settle during full outage")?; + assert!(error.ambiguous); + assert!(matches!( + error.cause.as_ref(), + CellDispatchError::Runtime(RuntimeError::DeadlineExceeded) + )); + assert_eq!(h.calls.load(Ordering::SeqCst) as u64, OFFLINE_COMMANDS); + assert!( + !h.controllers[0] + .archive_status(h.route.history)? + .context("archive status")? + .stopped + ); + controller::wait_for(async || { + Ok(h.controllers[0].usage().retained_s3 == 0 + && h.controllers[0].usage().retained_commands == 0) + }) + .await?; + let downgraded = h + .command( + 2, + b"archive-wait", + Durability::Quorum, + Duration::from_millis(800), + ) + .await? + .response() + .await + .err() + .context("stored S3 minimum survives caller downgrade")?; + assert!(downgraded.ambiguous); + assert!(matches!( + downgraded.cause.as_ref(), + CellDispatchError::Runtime(RuntimeError::DeadlineExceeded) + )); + assert_eq!(h.calls.load(Ordering::SeqCst) as u64, OFFLINE_COMMANDS); + + let checkpoints = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let source = h.fixture.runtimes[0] + .capture_checkpoint(h.route.history, &checkpoints) + .await?; + let prepared = h + .fixture + .cluster + .domain + .blocking(move || source.prepare_publication())? + .await??; + let cut = prepared.marker().clone(); + assert_eq!(cut.position().index, latest); + controller::wait_for(async || { + match h.fixture.runtimes[0] + .publish_checkpoint(h.route.history, prepared.clone()) + .await + { + Err(RuntimeError::Overloaded) => Ok(true), + Err(RuntimeError::NotReady) => Ok(false), + other => anyhow::bail!("uncovered checkpoint must refuse: {other:?}"), + } + }) + .await?; + assert_eq!(group.checkpoint_manifest()?, before_checkpoint); + assert_eq!(group.first_index()?, before_first); + assert_eq!(group.log_retention_usage()?.retained_entries, ENTRY_LIMIT); + + control.set_running(&h.fixture.cluster.domain, true).await?; + restored_probe( + &h, + &global, + &probe, + &probe_bytes, + remote_invocation + .attempt() + .context("retained remote attempt")? + .deadline_ms(), + ) + .await?; + archive_controller::published(&h.controllers[0], h.route.history, latest).await?; + let coverage = + archive_controller::verified_current(&h.fixture, &h.archive, rooted, "outage-recovery") + .await?; + assert!(coverage.marker().position().index >= latest); + let image = coverage + .recovery() + .manifest() + .pin_cell(&h.cell.cell) + .context("archived A")? + .1; + h.fixture + .cluster + .domain + .blocking(move || dependencies::image_value(image, OFFLINE_COMMANDS))? + .await??; + let image = coverage + .recovery() + .manifest() + .pin_cell(&CellId::new("registered-b")?) + .context("archived B")? + .1; + h.fixture + .cluster + .domain + .blocking(move || dependencies::image_value(image, 0))? + .await??; + // Release the completed receiver before another S3 output uses that same + // unchanged one-source archive pool. + drop(coverage); + let recovered = h + .command( + 2, + b"archive-wait", + Durability::Quorum, + Duration::from_secs(15), + ) + .await? + .response() + .await?; + assert_eq!(recovered.durability(), Durability::S3); + assert_eq!( + recovered + .local_receipt() + .expect("local command receipt") + .receipt + .result(), + b"archive-wait" + ); + drop(recovered); + assert_eq!(h.calls.load(Ordering::SeqCst) as u64, OFFLINE_COMMANDS); + + h.fixture.runtimes[0] + .publish_checkpoint(h.route.history, prepared) + .await?; + for node in 0..3 { + if node != 0 { + controller::wait_for(async || Ok(checkpoints.usage().sources == 0)).await?; + let source = h.fixture.runtimes[node] + .capture_checkpoint(h.route.history, &checkpoints) + .await?; + let prepared = h + .fixture + .cluster + .domain + .blocking(move || source.prepare_publication())? + .await??; + assert_eq!(prepared.marker(), &cut); + h.fixture.runtimes[node] + .publish_checkpoint(h.route.history, prepared) + .await?; + } + let log = h.fixture.cluster.voters[node] + .log + .cell_group(&h.route.history)?; + controller::wait_for(async || { + Ok(log.checkpoint_manifest()?.as_ref() == Some(&cut) + && log.log_retention_usage()?.retained_entries == 0 + && checkpoints.usage().sources == 0) + }) + .await?; + } + let resumed = h + .command( + REFUSED_COMMAND, + b"after-ceiling", + Durability::Quorum, + Duration::from_secs(15), + ) + .await? + .response() + .await?; + assert_eq!(resumed.durability(), Durability::Quorum); + drop(resumed); + // Preserve the same invocation, captured inputs, original deadline and + // attempt budget after the failed remote response. + remote_invocation.retry().await?; + let resumed = remote_invocation + .execute_remote(remote.transport(), MemberId::new(1)?) + .await?; + assert_eq!(resumed.durability(), Durability::Quorum); + assert_eq!(resumed.result(), b"remote-outage"); + drop(resumed); + drop(remote_invocation); + latest = h.fixture.runtimes[0] + .status(h.route.history) + .await? + .applied + .index; + state(&h, latest, OFFLINE_COMMANDS + 2).await?; + archive_controller::published(&h.controllers[0], h.route.history, latest).await?; + remote.close().await?; + drop(group); + h.close().await +} + +#[inline(never)] +pub(crate) fn full_backend_outage( + rooted: bool, + store: Arc, + control_directory: Option, +) -> std::pin::Pin>>> { + Box::pin(run(rooted, store, control_directory)) +} diff --git a/crates/cellar/tests/support/cell_group_s3_output.rs b/crates/cellar/tests/support/cell_group_s3_output.rs new file mode 100644 index 000000000..3299f7771 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_s3_output.rs @@ -0,0 +1,196 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! A resolved SQL producer cannot be reconciled again during S3 output fencing. +use super::*; + +pub(crate) async fn output_during_apply(rooted: bool) -> Result<()> { + for duplicate in [false, true] { + Box::pin(run(rooted, duplicate)).await?; + } + Ok(()) +} + +async fn run(rooted: bool, duplicate: bool) -> Result<()> { + let root = tempfile::tempdir()?; + let faults = apply_faults::physical::Faults::new(root.path().join("node-1/pages"), None); + let filesystem = Arc::new(apply_faults::physical::FaultFileSystem { + faults: faults.clone(), + }); + let h = Harness::configured_resources( + rooted, + Arc::new(object_store::memory::InMemory::new()), + 2, + Durability::Quorum, + None, + None, + Some(Resources::Storage { root, filesystem }), + ) + .await?; + let publication = h.remote.hold_publication(); + + struct Publication(Arc); + + impl Drop for Publication { + fn drop(&mut self) { + self.0.release.notify_one(); + } + } + let publication_release = Publication(publication.clone()); + if duplicate { + // The duplicate enters S3 settlement from preparation's receipt path, + // rather than reconciling a newly accepted SQL producer. + let first = h + .command( + 1, + b"retained-output", + Durability::Quorum, + Duration::from_secs(5), + ) + .await? + .response() + .await?; + assert_eq!( + first + .local_receipt() + .expect("local command receipt") + .receipt + .result(), + b"retained-output" + ); + drop(first); + } + let call = h + .command( + 1, + b"retained-output", + Durability::S3, + Duration::from_secs(5), + ) + .await?; + tokio::time::timeout(Duration::from_secs(5), publication.entered.notified()).await?; + h.waiting().await?; + + // The first producer has retired. Hold a later cell's real manifest sync, + // without enlarging the one-item SQL admission or changing the old image. + let other = history(&h.route, "registered-b")?; + let authority = h.fixture.runtimes[0] + .execution_authority(h.route.history, other.clone()) + .await?; + let options = h.fixture.cell_options(&other).await?; + let pool = h.fixture.appliers[&(0, h.route.history)].0.clone(); + let CellSqlOutcome::Prepared(private) = pool + .stage(options, authority, cell_attempt(239)?, true, |tx| { + tx.execute("UPDATE value SET n=n+1", [])?; + Ok(b"blocking-result".to_vec()) + }) + .await? + else { + anyhow::bail!("private preparation") + }; + pool.mark_submitted(other.clone(), private.proposal()) + .await?; + let original = h.fixture.runtimes[0] + .reconciliation_authority(h.route.history, h.cell.clone()) + .await? + .cell() + .last_mutation; + let (entered, started) = tokio::sync::oneshot::channel(); + let (release, blocked) = std::sync::mpsc::sync_channel(1); + + struct Unblock(std::sync::mpsc::SyncSender<()>); + + impl Drop for Unblock { + fn drop(&mut self) { + let _ = self.0.try_send(()); + } + } + let unblock = Unblock(release); + *faults.before_directory_sync.lock().unwrap() = Some(Box::new(move || { + let _ = entered.send(()); + blocked + .recv_timeout(Duration::from_secs(5)) + .expect("release real manifest sync"); + })); + let accepted = accepted_sql(&h.fixture.runtimes[0], &private).await?; + tokio::time::timeout(Duration::from_secs(5), started).await??; + controller::wait_for(async || { + let status = h.fixture.runtimes[0].status(h.route.history).await?; + Ok(status.committed >= accepted.position.index + && status.applied.index < accepted.position.index) + }) + .await?; + drop(publication_release); + archive_controller::published(&h.controllers[0], h.route.history, original.index).await?; + let mut response = Box::pin(call.response()); + assert!( + tokio::time::timeout(Duration::from_millis(30), response.as_mut()) + .await + .is_err(), + "committed output must wait for fresh authority without revisiting its retired SQL producer" + ); + assert_eq!(h.controllers[0].usage().quarantined_groups, 0); + assert_eq!(h.calls.load(Ordering::SeqCst), 1); + drop(unblock); + controller::applied(&h.fixture, h.route.history, accepted.position.index).await?; + let authority = h.fixture.runtimes[0] + .reconciliation_authority(h.route.history, other.clone()) + .await?; + let CellSqlReconciliation::Committed(blocker_receipt) = + pool.reconcile(private.proposal(), authority).await? + else { + anyhow::bail!("blocking command receipt") + }; + assert_eq!( + blocker_receipt.receipt().receipt.result(), + b"blocking-result" + ); + drop(blocker_receipt); + drop(private); + let receipt = response.await?; + assert_eq!(receipt.durability(), Durability::S3); + assert_eq!( + receipt + .local_receipt() + .expect("local command receipt") + .receipt + .result(), + b"retained-output" + ); + assert_eq!( + receipt + .local_receipt() + .expect("local command receipt") + .version + .sequence + .get(), + 2 + ); + drop(receipt); + let retry = h + .command( + 1, + b"retained-output", + Durability::S3, + Duration::from_secs(5), + ) + .await? + .response() + .await?; + assert_eq!( + retry + .local_receipt() + .expect("local command receipt") + .receipt + .result(), + b"retained-output" + ); + assert_eq!(h.calls.load(Ordering::SeqCst), 1); + drop(retry); + for runtime in &h.fixture.runtimes { + let cell = runtime + .reconciliation_authority(h.route.history, other.clone()) + .await?; + assert_eq!(cell.cell().version.sequence.get(), 2); + } + h.close().await +} diff --git a/crates/cellar/tests/support/cell_group_s3_receipts.rs b/crates/cellar/tests/support/cell_group_s3_receipts.rs new file mode 100644 index 000000000..88892836a --- /dev/null +++ b/crates/cellar/tests/support/cell_group_s3_receipts.rs @@ -0,0 +1,518 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Read-only S3 results at a later cut, with bounded payload allocation. +use super::*; + +impl Harness { + pub(super) fn receipt_request( + &self, + sequence: u64, + input: &[u8], + durability: Durability, + ) -> Result { + Ok(CellReceiptRequest { + id: CommandId::new(ClientId::new([188; 16])?, sequence)?, + contract: self.class.contract("Add").context("Add")?, + input: input.to_vec(), + durability, + }) + } + + async fn lookup( + &self, + request: CellReceiptRequest, + timeout: Duration, + ) -> Result { + let mut invocation = self.dispatcher.lookup_receipt( + self.cell.clone(), + request, + CellRoutingLimits { + timeout, + max_attempts: 4, + }, + None, + )?; + invocation.begin().await?; + Ok(invocation) + } + + async fn now(&self) -> Result { + Ok(self + .fixture + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms().max(0) as u64 }) + .await?) + } + + async fn token(&self, response: &CellReceiptResponse) -> Result<()> { + let keys = VersionKeys::new( + VersionSigningKey::new([89; 32], u64::MAX)?, + vec![], + Duration::from_secs(30), + )?; + assert_eq!( + keys.verify_cell( + response.token(), + &self.cell, + self.class.schema_digest(), + self.now().await? + )?, + *response.observation() + ); + Ok(()) + } + + async fn published_command(&self, sequence: u64, input: &[u8]) -> Result<()> { + self.published_with_ceiling(sequence, input, None).await + } + + async fn published_with_ceiling( + &self, + sequence: u64, + input: &[u8], + ceiling: Option, + ) -> Result<()> { + let mut options = self.fixture.cell_options(&self.cell).await?; + if let Some(ceiling) = ceiling { + match &mut options { + CellSqlOptions::Materialized(o) => o.limits.max_retained_receipts = ceiling, + CellSqlOptions::Extents(o) => o.limits.max_retained_receipts = ceiling, + } + } + let response = self + .command_using( + sequence, + input, + Durability::Quorum, + Duration::from_secs(15), + options, + ) + .await? + .response() + .await?; + let index = response + .local_receipt() + .expect("local command receipt") + .position + .index; + drop(response); + archive_controller::published(&self.controllers[0], self.route.history, index).await + } +} + +pub(crate) async fn s3_receipt_observations( + rooted: bool, + store: Arc, +) -> Result<()> { + let h = Harness::new(rooted, store, 1).await?; + let pool = &h.fixture.appliers[&(0, h.route.history)].0; + let runtime = &h.fixture.runtimes[0]; + let requested = h.receipt_request(1, b"late", Durability::S3)?; + let mut unknown = h.lookup(requested.clone(), Duration::from_secs(15)).await?; + let response = unknown + .execute(runtime, pool, h.fixture.cell_options(&h.cell).await?) + .await?; + assert_eq!(response.durability(), Durability::S3); + assert!(response.receipt().is_none()); + assert_eq!(response.observation().version.sequence.get(), 1); + h.token(&response).await?; + drop(response); + drop(unknown); + let mut lookup = h.lookup(requested.clone(), Duration::from_secs(15)).await?; + let selection = lookup + .select(runtime, pool, h.fixture.cell_options(&h.cell).await?) + .await?; + assert_eq!(h.controllers[0].usage().retained_s3, 1); + assert_eq!(h.controllers[0].usage().s3_bytes, 8192); + assert_eq!(pool.usage()?.retained_operations, 0); + // The original SQL read observed None at version 1. The same one-slot node + // can now commit and archive the command before proof submission. + h.published_command(1, b"late").await?; + let response = selection.response().await?; + assert_eq!(response.durability(), Durability::S3); + assert_eq!( + response + .receipt() + .context("later archived receipt")? + .result(), + b"late" + ); + assert_eq!(response.observation().version.sequence.get(), 2); + assert_eq!(h.archive.tails.usage().operations, 1); + assert_eq!(h.controllers[0].usage().s3_bytes, 8192 + 2 * b"late".len()); + h.token(&response).await?; + drop(response); + drop(lookup); + // A retained result is exact across newer images, and its token identifies + // the actually inspected archived image rather than the older SQL selection. + let mut known = h.lookup(requested, Duration::from_secs(15)).await?; + let selection = known + .select(runtime, pool, h.fixture.cell_options(&h.cell).await?) + .await?; + h.published_command(2, b"").await?; + let response = selection.response().await?; + assert_eq!( + response.receipt().context("retained result")?.result(), + b"late" + ); + assert_eq!(response.observation().version.sequence.get(), 3); + h.token(&response).await?; + drop(response); + drop(known); + let mut empty = h + .lookup( + h.receipt_request(2, b"", Durability::S3)?, + Duration::from_secs(15), + ) + .await?; + let response = empty + .execute(runtime, pool, h.fixture.cell_options(&h.cell).await?) + .await?; + assert_eq!( + response.receipt().map(|row| row.result()), + Some(b"".as_slice()) + ); + drop(response); + drop(empty); + let mut missing = h.receipt_request(1, b"missing", Durability::S3)?; + missing.id = CommandId::new(ClientId::new([189; 16])?, 1)?; + let mut missing = h.lookup(missing, Duration::from_secs(15)).await?; + let response = missing + .execute(runtime, pool, h.fixture.cell_options(&h.cell).await?) + .await?; + assert!(response.receipt().is_none()); + assert_eq!(response.durability(), Durability::S3); + assert_eq!(response.observation().version.sequence.get(), 3); + drop(response); + drop(missing); + // A provider in the same domain cannot lend a different runtime's proof. + let foreign = h.dispatcher.clone().with_s3(&h.controllers[1])?; + let mut foreign = foreign.lookup_receipt( + h.cell.clone(), + h.receipt_request(1, b"late", Durability::S3)?, + CellRoutingLimits { + timeout: Duration::from_secs(15), + max_attempts: 4, + }, + None, + )?; + foreign.begin().await?; + assert!(matches!( + foreign + .select(runtime, pool, h.fixture.cell_options(&h.cell).await?) + .await + .err(), + Some(CellDispatchError::Runtime(RuntimeError::Invalid(_))) + )); + drop(foreign); + assert_eq!(h.calls.load(Ordering::SeqCst), 2); + assert_eq!(h.initializations.load(Ordering::SeqCst), 0); + assert_eq!(h.controllers[0].usage().retained_s3, 0); + assert_eq!(pool.usage()?.retained_operations, 0); + h.close().await +} + +pub(crate) async fn s3_receipt_minimums(rooted: bool, store: Arc) -> Result<()> { + for minimum in [Durability::Quorum, Durability::S3] { + let backend = Arc::new(object_store::prefix::PrefixStore::new( + store.clone(), + ObjectPath::from(if minimum == Durability::S3 { + "namespace-s3" + } else { + "namespace-quorum" + }), + )); + let h = Harness::with_minimum(rooted, backend, 1, minimum).await?; + let pool = &h.fixture.appliers[&(0, h.route.history)].0; + let runtime = &h.fixture.runtimes[0]; + let result = h + .command(1, b"once", Durability::S3, Duration::from_secs(15)) + .await? + .response() + .await?; + drop(result); + let request = h.receipt_request(1, b"once", Durability::Quorum)?; + let mut lookup = h.lookup(request, Duration::from_secs(15)).await?; + let selection = lookup + .select(runtime, pool, h.fixture.cell_options(&h.cell).await?) + .await?; + assert_eq!(h.controllers[0].usage().retained_s3, 1); + assert_eq!(pool.usage()?.retained_operations, 0); + drop(selection); + assert_eq!(h.controllers[0].usage().retained_s3, 0); + drop(lookup); + let mut lookup = h + .lookup( + h.receipt_request(1, b"once", Durability::Quorum)?, + Duration::from_secs(15), + ) + .await?; + let response = lookup + .execute(runtime, pool, h.fixture.cell_options(&h.cell).await?) + .await?; + assert_eq!(response.durability(), Durability::S3); + assert_eq!( + response + .receipt() + .context("stored S3 minimum")? + .minimum_durability(), + Durability::S3 + ); + assert_eq!(response.receipt().unwrap().result(), b"once"); + h.token(&response).await?; + drop(response); + drop(lookup); + if minimum == Durability::Quorum { + let gateway = h.fixture.transports[0].install_cell_command_gateway( + h.dispatcher.clone(), + h.controllers[0].clone(), + cellar::application::consensus::verification::cell_groups::CellGroupNamespaceStorage::new( + h.fixture.cell_options(&h.cell).await?, + )?, + )?; + let router = super::super::query_origins::Router::start(&h.fixture).await?; + router + .transport + .install_cell_query_group(h.fixture.tcp_route(h.route.history)?)?; + router.grant(&h.fixture, 1)?; + h.fixture.transports[0].grant_command_origin( + cellar::application::consensus::NamespaceCommandOrigin { + cluster: h.route.history.cluster, + namespace: h.route.history.namespace, + revision: 1, + peer: router.peer.clone(), + }, + )?; + // A Quorum retry of an S3 command must achieve and report S3 + // through the authenticated wire, without invoking its callback. + let mut remote = h + .lookup( + h.receipt_request(1, b"once", Durability::Quorum)?, + Duration::from_secs(15), + ) + .await?; + let response = remote + .execute_remote(&router.transport, MemberId::new(1)?) + .await?; + assert_eq!(response.durability(), Durability::S3); + assert_eq!(response.result(), Some(b"once".as_slice())); + drop(response); + drop(remote); + controller::wait_for(async || Ok(h.controllers[0].usage().retained_s3 == 0)).await?; + let mut duplicate = h + .dispatcher + .command( + h.cell.clone(), + CellCommandRequest { + id: CommandId::new(ClientId::new([188; 16])?, 1)?, + contract: h.class.contract("Add").context("Add")?, + input: b"once".to_vec(), + durability: Durability::Quorum, + retry_until_ms: h.retry_until, + }, + CellRoutingLimits { + timeout: Duration::from_secs(15), + max_attempts: 4, + }, + None, + ) + .await?; + duplicate.begin().await?; + let response = duplicate + .execute_remote(&router.transport, MemberId::new(1)?) + .await?; + assert_eq!(response.durability(), Durability::S3); + assert_eq!(response.result(), b"once"); + drop(response); + drop(duplicate); + controller::wait_for(async || Ok(h.controllers[0].usage().retained_s3 == 0)).await?; + assert!(h.controllers[0].group_failure(h.route.history)?.is_none()); + drop(gateway); + router.close().await?; + } + let mut missing = h.receipt_request(1, b"missing", Durability::Quorum)?; + missing.id = CommandId::new(ClientId::new([189; 16])?, 1)?; + let mut missing = h.lookup(missing, Duration::from_secs(15)).await?; + let response = missing + .execute(runtime, pool, h.fixture.cell_options(&h.cell).await?) + .await?; + assert!(response.receipt().is_none()); + assert_eq!(response.durability(), minimum); + drop(response); + drop(missing); + assert_eq!(h.calls.load(Ordering::SeqCst), 1); + h.close().await?; + } + Ok(()) +} + +pub(crate) async fn s3_receipt_failures(rooted: bool, store: Arc) -> Result<()> { + for profile in ["conflict", "expiry", "allocation"] { + let backend = Arc::new(object_store::prefix::PrefixStore::new( + store.clone(), + ObjectPath::from(profile), + )); + let mut h = Harness::configured( + rooted, + backend, + 1, + Durability::Quorum, + (profile == "allocation").then_some(40 * 1024), + ) + .await?; + let mut expiry = None; + if profile == "expiry" { + h.retry_until = h.now().await? + 2000; + expiry = Some(h.retry_until); + h.published_command(1, b"wanted").await?; + h.retry_until = h.now().await? + 60_000; + h.published_command(2, b"protected-latest").await?; + } + let pool = &h.fixture.appliers[&(0, h.route.history)].0; + let runtime = &h.fixture.runtimes[0]; + let mut lookup = h + .lookup( + h.receipt_request(1, b"wanted", Durability::S3)?, + Duration::from_secs(15), + ) + .await?; + let selection = lookup + .select(runtime, pool, h.fixture.cell_options(&h.cell).await?) + .await?; + assert_eq!(pool.usage()?.retained_operations, 0); + if profile == "expiry" { + let expiry = expiry.context("original retry horizon")?; + controller::wait_for(async || Ok(h.now().await? > expiry)).await?; + // Only the retry horizon changes; the selected original operation, + // inputs and deadline stay sealed in the lookup phase. + h.retry_until = h.now().await? + 60_000; + // Two retained rows fill this new scenario's ceiling. Maintenance + // prunes only the expired prefix; the latest receipt stays protected. + h.published_with_ceiling(3, b"next", Some(2)).await?; + } else { + h.published_command( + 1, + if profile == "conflict" { + b"other" + } else { + b"wanted" + }, + ) + .await?; + } + let error = selection + .response() + .await + .err() + .context("typed archived refusal")?; + assert!( + matches!(error, CellDispatchError::Execution(ExecutionError::Refused(failure)) if match profile { + "conflict" => failure == CommandFailure::CommandConflict, + "expiry" => failure == CommandFailure::CommandExpired, + _ => failure == CommandFailure::Overloaded, + }) + ); + drop(lookup); + assert_eq!(h.controllers[0].usage().retained_s3, 0); + assert_eq!(h.controllers[0].usage().s3_bytes, 0); + assert_eq!(pool.usage()?.retained_operations, 0); + assert!(h.controllers[0].group_failure(h.route.history)?.is_none()); + if profile == "allocation" { + // The legitimate 40 KiB result remains readable by Quorum. It must + // be refused before payload allocation when the 64 KiB S3 budget + // cannot retain the original and extra read bytes. No quota grows. + let mut read = h + .lookup( + h.receipt_request(1, b"wanted", Durability::Quorum)?, + Duration::from_secs(15), + ) + .await?; + let result = read + .execute(runtime, pool, h.fixture.cell_options(&h.cell).await?) + .await?; + assert_eq!( + result + .receipt() + .context("intact large result")? + .result() + .len(), + 40 * 1024 + ); + assert_eq!(result.durability(), Durability::Quorum); + drop(result); + drop(read); + } + assert_eq!( + h.calls.load(Ordering::SeqCst), + if profile == "expiry" { 3 } else { 1 } + ); + h.close().await?; + } + Ok(()) +} + +pub(crate) async fn s3_receipt_deadline( + rooted: bool, + store: Arc, + cancel: bool, +) -> Result<()> { + let h = Harness::new(rooted, store, 1).await?; + let pool = &h.fixture.appliers[&(0, h.route.history)].0; + let runtime = &h.fixture.runtimes[0]; + let request = h.receipt_request(1, b"after", Durability::S3)?; + let mut lookup = h + .lookup(request.clone(), Duration::from_millis(800)) + .await?; + let selection = lookup + .select(runtime, pool, h.fixture.cell_options(&h.cell).await?) + .await?; + h.remote.outage.store(true, Ordering::Release); + let attempts = h.controllers[0] + .archive_status(h.route.history)? + .context("archive registration")? + .attempts; + let mut pending = Box::pin(selection.response()); + assert!(futures_util::poll!(pending.as_mut()).is_pending()); + controller::wait_for(async || { + Ok(h.controllers[0] + .archive_status(h.route.history)? + .context("archive status")? + .attempts + > attempts + && h.controllers[0].usage().retained_s3 == 1) + }) + .await?; + assert_eq!(pool.usage()?.retained_operations, 0); + if !cancel { + assert!(matches!( + pending.as_mut().await.err(), + Some(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)) + )); + } + drop(pending); + drop(lookup); + controller::wait_for(async || Ok(h.controllers[0].usage().retained_s3 == 0)).await?; + assert!(h.controllers[0].group_failure(h.route.history)?.is_none()); + let response = h + .command(1, b"after", Durability::Quorum, Duration::from_secs(15)) + .await? + .response() + .await?; + drop(response); + h.remote.outage.store(false, Ordering::Release); + let mut retry = h.lookup(request, Duration::from_secs(15)).await?; + let response = retry + .execute(runtime, pool, h.fixture.cell_options(&h.cell).await?) + .await?; + assert_eq!(response.durability(), Durability::S3); + assert_eq!( + response.receipt().context("original command")?.result(), + b"after" + ); + assert_eq!(h.calls.load(Ordering::SeqCst), 1); + drop(response); + drop(retry); + assert_eq!(pool.usage()?.retained_operations, 0); + h.close().await +} diff --git a/crates/cellar/tests/support/cell_group_s3_remote.rs b/crates/cellar/tests/support/cell_group_s3_remote.rs new file mode 100644 index 000000000..f61a03126 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_s3_remote.rs @@ -0,0 +1,426 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Real TLS output proofs, original deadlines and physical response ownership. +use super::*; +use cellar::application::consensus::verification::cell_groups::{ + CellCommandGateway, CellRemoteCommandResponse, CellRemoteEnrollmentResponse, + CellRemoteReceiptResponse, +}; +#[path = "cell_group_s3_remote_wire.rs"] +mod wire; +pub(crate) use wire::s3_remote_slow_delivery; + +pub(super) struct Remote { + gateway: CellCommandGateway, + router: super::super::query_origins::Router, +} + +impl Remote { + pub(super) fn transport(&self) -> &RaftTransport { + &self.router.transport + } + + pub(super) async fn start(h: &Harness, dispatcher: CellGroupDispatcher) -> Result { + // A gateway derives history paths below a namespace base. Nesting that + // base inside an existing cell's disposable SQLite directory leaves a + // foreign child which must make a later local reopen refuse. + let mut options = h.fixture.cell_options(&h.cell).await?; + let base = h + .fixture + .cluster + ._root + .path() + .join("remote-command-namespace"); + match &mut options { + CellSqlOptions::Materialized(options) => options.directory = base, + CellSqlOptions::Extents(options) => options.cache_directory = base, + } + let gateway = h.fixture.transports[0].install_cell_command_gateway( + dispatcher, + h.controllers[0].clone(), + cellar::application::consensus::verification::cell_groups::CellGroupNamespaceStorage::new( + options, + )?, + )?; + let router = super::super::query_origins::Router::start(&h.fixture).await?; + router + .transport + .install_cell_query_group(h.fixture.tcp_route(h.route.history)?)?; + router.grant(&h.fixture, 1)?; + h.fixture.transports[0].grant_command_origin( + cellar::application::consensus::NamespaceCommandOrigin { + cluster: h.route.history.cluster, + namespace: h.route.history.namespace, + revision: 1, + peer: router.peer.clone(), + }, + )?; + Ok(Self { gateway, router }) + } + + pub(super) async fn close(self) -> Result<()> { + drop(self.gateway); + self.router.close().await + } +} + +pub(super) fn command_request( + h: &Harness, + sequence: u64, + input: &[u8], + durability: Durability, +) -> Result { + Ok(CellCommandRequest { + id: CommandId::new(ClientId::new([188; 16])?, sequence)?, + contract: h.class.contract("Add").context("Add")?, + input: input.to_vec(), + durability, + retry_until_ms: h.retry_until, + }) +} + +async fn command( + h: &Harness, + remote: &Remote, + sequence: u64, + input: &[u8], + durability: Durability, +) -> Result { + command_using( + h, + remote, + command_request(h, sequence, input, durability)?, + bounds(), + ) + .await +} + +pub(super) async fn command_using( + h: &Harness, + remote: &Remote, + request: CellCommandRequest, + limits: CellRoutingLimits, +) -> Result { + let mut invocation = h + .dispatcher + .command(h.cell.clone(), request, limits, None) + .await?; + invocation.begin().await?; + Ok(invocation + .execute_remote(&remote.router.transport, MemberId::new(1)?) + .await?) +} + +async fn receipt( + h: &Harness, + remote: &Remote, + known: bool, + durability: Durability, +) -> Result { + let mut request = h.receipt_request(1, b"remote", durability)?; + if !known { + request.id = CommandId::new(ClientId::new([189; 16])?, 1)?; + } + let mut invocation = h + .dispatcher + .lookup_receipt(h.cell.clone(), request, bounds(), None)?; + invocation.begin().await?; + Ok(invocation + .execute_remote(&remote.router.transport, MemberId::new(1)?) + .await?) +} + +async fn enrollment( + h: &Harness, + remote: &Remote, + durability: Durability, +) -> Result { + let mut invocation = h + .dispatcher + .enroll( + h.cell.clone(), + CellEnrollmentRequest { + creation: ProposalId::new([210; 16])?, + durability, + }, + bounds(), + None, + ) + .await?; + invocation.begin().await?; + Ok(invocation + .execute_remote(&remote.router.transport, MemberId::new(1)?) + .await?) +} + +fn bounds() -> CellRoutingLimits { + CellRoutingLimits { + timeout: Duration::from_secs(15), + max_attempts: 4, + } +} + +async fn refunded(h: &Harness) -> Result<()> { + controller::wait_for(async || { + Ok(h.controllers[0].usage().retained_s3 == 0 + && h.controllers[0].usage().retained_commands == 0 + && h.fixture.transports[0].usage().query_items == 0) + }) + .await +} + +pub(crate) async fn s3_remote_outputs(rooted: bool, store: Arc) -> Result<()> { + for minimum in [Durability::Quorum, Durability::S3] { + let store = Arc::new(object_store::prefix::PrefixStore::new( + store.clone(), + ObjectPath::from(if minimum == Durability::S3 { + "minimum-s3" + } else { + "minimum-quorum" + }), + )); + let mut h = Harness::with_minimum(rooted, store, 1, minimum).await?; + let remote = Remote::start(&h, h.dispatcher.clone()).await?; + let requested = if minimum == Durability::S3 { + Durability::Quorum + } else { + Durability::S3 + }; + let response = command(&h, &remote, 1, b"remote", requested).await?; + assert_eq!(response.durability(), Durability::S3); + assert_eq!(response.result(), b"remote"); + assert_eq!(response.observation().version.sequence.get(), 2); + let expected = response.observation().clone(); + drop(response); + refunded(&h).await?; + let found = receipt(&h, &remote, true, Durability::Quorum).await?; + assert_eq!(found.durability(), Durability::S3); + assert_eq!(found.result(), Some(b"remote".as_slice())); + assert_eq!(*found.observation(), expected); + drop(found); + refunded(&h).await?; + let empty = command(&h, &remote, 2, b"", requested).await?; + assert_eq!(empty.result(), b""); + assert_eq!(empty.durability(), Durability::S3); + drop(empty); + refunded(&h).await?; + let unknown = receipt(&h, &remote, false, requested).await?; + assert_eq!(unknown.durability(), Durability::S3); + assert_eq!(unknown.result(), None); + assert_eq!(unknown.observation().version.sequence.get(), 3); + drop(unknown); + refunded(&h).await?; + assert_eq!(h.calls.load(Ordering::SeqCst), 2); + h.cell.cell = CellId::new("remote-created")?; + let created = enrollment(&h, &remote, requested).await?; + assert_eq!(created.durability(), Durability::S3); + assert_eq!(created.record().cell(), &h.cell); + let original = created.record().clone(); + drop(created); + refunded(&h).await?; + let changed = command(&h, &remote, 1, b"changed", requested).await?; + assert_eq!(changed.durability(), Durability::S3); + drop(changed); + refunded(&h).await?; + let retry = enrollment(&h, &remote, requested).await?; + assert_eq!(retry.record(), &original); + assert_eq!(retry.durability(), Durability::S3); + drop(retry); + refunded(&h).await?; + assert_eq!(h.initializations.load(Ordering::SeqCst), 1); + assert_eq!(h.calls.load(Ordering::SeqCst), 3); + assert!(h.controllers[0].group_failure(h.route.history)?.is_none()); + remote.close().await?; + h.close().await?; + } + // Reachable S3 and a bound sender cannot lend an unbound gateway a proof. + let h = Harness::new( + rooted, + Arc::new(object_store::prefix::PrefixStore::new(store, "unbound")), + 1, + ) + .await?; + let unbound = CellGroupDispatcher::new( + &h.fixture.cluster.domain, + h.fixture.cluster.directory.clone(), + h.class.clone(), + VersionKeys::new( + VersionSigningKey::new([89; 32], u64::MAX)?, + vec![], + Duration::from_secs(30), + )?, + CellDispatchLimits { + max_cached_routes: 32, + route_cache_bytes: 256 * 1024, + max_invocations: 8, + retained_bytes: 1024 * 1024, + max_input_bytes: 1024, + }, + )?; + let remote = Remote::start(&h, unbound).await?; + let mut attempt = h + .dispatcher + .command( + h.cell.clone(), + command_request(&h, 1, b"remote", Durability::S3)?, + bounds(), + None, + ) + .await?; + attempt.begin().await?; + let error = attempt + .execute_remote(&remote.router.transport, MemberId::new(1)?) + .await + .err() + .context("unbound command")?; + assert!(error.ambiguous); + assert!(matches!( + *error.cause, + CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::UnsupportedCapability + )) + )); + drop(attempt); + assert!(receipt(&h, &remote, false, Durability::S3).await.is_err()); + assert!(enrollment(&h, &remote, Durability::S3).await.is_err()); + assert_eq!(h.calls.load(Ordering::SeqCst), 0); + assert_eq!(h.initializations.load(Ordering::SeqCst), 0); + refunded(&h).await?; + remote.close().await?; + h.close().await +} + +async fn deadline_attempt(h: &Harness, remote: &Remote, kind: &str) -> Result<()> { + let bounds = CellRoutingLimits { + timeout: Duration::from_millis(800), + max_attempts: 4, + }; + match kind { + "command" => { + let mut i = h + .dispatcher + .command( + h.cell.clone(), + command_request(h, 1, b"deadline", Durability::S3)?, + bounds, + None, + ) + .await?; + i.begin().await?; + let error = i + .execute_remote(&remote.router.transport, MemberId::new(1)?) + .await + .err() + .context("S3 command deadline")?; + assert!(error.ambiguous); + } + "enrollment" => { + let mut i = h + .dispatcher + .enroll( + h.cell.clone(), + CellEnrollmentRequest { + creation: ProposalId::new([210; 16])?, + durability: Durability::S3, + }, + bounds, + None, + ) + .await?; + i.begin().await?; + let error = i + .execute_remote(&remote.router.transport, MemberId::new(1)?) + .await + .err() + .context("S3 enrollment deadline")?; + assert!(error.ambiguous); + } + "receipt" => { + let mut request = h.receipt_request(1, b"remote", Durability::S3)?; + request.id = CommandId::new(ClientId::new([189; 16])?, 1)?; + let mut i = h + .dispatcher + .lookup_receipt(h.cell.clone(), request, bounds, None)?; + i.begin().await?; + assert!(i + .execute_remote(&remote.router.transport, MemberId::new(1)?) + .await + .is_err()); + } + _ => anyhow::bail!("deadline kind"), + } + Ok(()) +} + +pub(crate) async fn s3_remote_deadlines(rooted: bool, store: Arc) -> Result<()> { + for kind in ["command", "enrollment", "receipt"] { + for cancel in [false, true] { + let store = Arc::new(object_store::prefix::PrefixStore::new( + store.clone(), + ObjectPath::from(format!("{kind}-{cancel}")), + )); + let mut h = Harness::new(rooted, store, 1).await?; + if kind == "enrollment" { + h.cell.cell = CellId::new("deadline-creation")?; + } + let remote = Remote::start(&h, h.dispatcher.clone()).await?; + h.remote.outage.store(true, Ordering::Release); + let mut pending = Box::pin(deadline_attempt(&h, &remote, kind)); + match futures_util::future::select(pending.as_mut(), Box::pin(h.waiting())).await { + futures_util::future::Either::Left((result, _)) => { + anyhow::bail!("{kind} completed before S3 wait: {result:?}") + } + futures_util::future::Either::Right((result, _)) => result?, + } + if cancel { + drop(pending); + } else { + pending.await?; + } + refunded(&h).await?; + assert!(h.controllers[0].group_failure(h.route.history)?.is_none()); + let next = if kind == "command" { 2 } else { 1 }; + let successor = h + .command( + next, + b"successor", + Durability::Quorum, + Duration::from_secs(15), + ) + .await? + .response() + .await?; + drop(successor); + h.remote.outage.store(false, Ordering::Release); + match kind { + "command" => { + let r = command(&h, &remote, 1, b"deadline", Durability::S3).await?; + assert_eq!(r.result(), b"deadline"); + assert_eq!(r.durability(), Durability::S3); + drop(r); + assert_eq!(h.calls.load(Ordering::SeqCst), 2); + } + "enrollment" => { + let r = enrollment(&h, &remote, Durability::S3).await?; + assert_eq!(r.durability(), Durability::S3); + drop(r); + assert_eq!(h.initializations.load(Ordering::SeqCst), 1); + assert_eq!(h.calls.load(Ordering::SeqCst), 1); + } + "receipt" => { + let r = receipt(&h, &remote, false, Durability::S3).await?; + assert_eq!(r.result(), None); + assert_eq!(r.durability(), Durability::S3); + drop(r); + assert_eq!(h.initializations.load(Ordering::SeqCst), 0); + assert_eq!(h.calls.load(Ordering::SeqCst), 1); + } + _ => unreachable!(), + } + refunded(&h).await?; + remote.close().await?; + h.close().await?; + } + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_s3_remote_wire.rs b/crates/cellar/tests/support/cell_group_s3_remote_wire.rs new file mode 100644 index 000000000..7f4537e91 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_s3_remote_wire.rs @@ -0,0 +1,387 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! A deliberately slow independent TLS reader holds an actual response write. +use super::super::super::super::network_probes::Connection; +use super::*; +use serde::Serialize; + +#[derive(Serialize)] +struct Endpoint<'a> { + node: [u8; 16], + disk: [u8; 16], + process: [u8; 16], + label: &'a str, + member: Option, +} + +#[derive(Serialize)] +#[serde(tag = "kind", content = "identity")] +enum Scope<'a> { + Shared { + group: [[u8; 16]; 4], + cell: &'a str, + incarnation: [u8; 16], + }, +} + +#[derive(Serialize)] +struct Route<'a> { + purpose: &'a str, + scope: Scope<'a>, + schema: [u8; 32], + source: Endpoint<'a>, + target: Endpoint<'a>, +} + +#[derive(Serialize)] +struct Header<'a> { + route: Route<'a>, + sequence: &'a str, + digest: [u8; 32], +} + +fn header(h: &Harness, remote: &Remote, kind: &str, body: &[u8], sequence: u64) -> Result> { + let g = h.route.history; + let target = h.fixture.cluster.voters[0] + .log + .cell_group(&g)? + .identity() + .clone(); + let source = &remote.router.peer; + let route = Route { + purpose: if kind == "command" { + "Command" + } else { + "Receipt" + }, + scope: Scope::Shared { + group: [ + g.cluster.bytes(), + g.namespace.bytes(), + g.group.bytes(), + g.incarnation.bytes(), + ], + cell: h.cell.cell.as_str(), + incarnation: h.cell.incarnation.bytes(), + }, + schema: h.class.schema_digest(), + source: Endpoint { + node: source.node.bytes(), + disk: source.disk.bytes(), + process: source.process.bytes(), + label: &source.label, + member: None, + }, + target: Endpoint { + node: target.node.bytes(), + disk: target.disk.bytes(), + process: target.process.bytes(), + label: "node-1", + member: Some(target.member.get().to_string()), + }, + }; + Ok(serde_json::to_vec(&Header { + route, + sequence: &sequence.to_string(), + digest: Sha256::digest(body).into(), + })?) +} + +fn request(h: &Harness, kind: &str) -> Result> { + let request = command_request(h, 1, b"held", Durability::S3)?; + let mut body = if kind == "command" { + b"CLGCCQ01".to_vec() + } else { + b"CLGCRQ01".to_vec() + }; + body.extend_from_slice(&request.id.client.bytes()); + body.extend_from_slice(&request.id.sequence().to_be_bytes()); + body.extend_from_slice(&h.route.epoch.get().to_be_bytes()); + body.extend_from_slice(&h.route.birth_digest); + body.extend_from_slice(&(request.contract.operation.len() as u16).to_be_bytes()); + body.extend_from_slice(request.contract.operation.as_bytes()); + body.extend_from_slice(&request.contract.digest); + body.push(2); + if kind == "command" { + body.extend_from_slice(&h.retry_until.to_be_bytes()); + body.extend_from_slice(&(h.retry_until - 60_000).to_be_bytes()); + body.extend_from_slice(&[23; 32]); + } + body.extend_from_slice(&15_000_u64.to_be_bytes()); + body.extend_from_slice(&(request.input.len() as u32).to_be_bytes()); + body.extend_from_slice(&request.input); + Ok(body) +} + +fn written(h: &Harness) -> Result { + let snapshot = h.fixture.cluster.voters[0].log.performance_snapshot(); + anyhow::ensure!( + snapshot.available && !snapshot.saturated, + "socket counter available" + ); + snapshot + .counters + .iter() + .find(|c| c.name == "transport_query_write_bytes") + .map(|c| c.value) + .context("query socket byte counter") +} + +fn writes(h: &Harness) -> Result<[u64; 4]> { + let snapshot = h.fixture.cluster.voters[0].log.performance_snapshot(); + anyhow::ensure!( + snapshot.available && !snapshot.saturated, + "write observations available" + ); + let sample = snapshot + .operations + .iter() + .find(|sample| sample.name == "transport_query_reply_write") + .context("physical reply write observation")?; + Ok([ + sample.started, + sample.succeeded, + sample.failed, + sample.abandoned, + ]) +} + +async fn fill( + h: &Harness, + remote: &Remote, + connection: &mut Connection, + kind: &str, +) -> Result> { + let body = request(h, kind)?; + let mut bindings = Vec::new(); + // One exact retry at a time: the next request is sent only after the + // preceding physical flush and resource refund. No callbacks are replayed. + // Bound the undrained window independently of all original node quotas. + for sequence in 1..=128 { + let before = writes(h)?; + let request_header = header(h, remote, kind, &body, sequence)?; + bindings.push( + connection + .frame_bytes( + b"CLQP", + b"cellar-raft-query-request-1", + None, + &request_header, + &body, + ) + .await?, + ); + controller::wait_for(async || { + let state = writes(h)?; + anyhow::ensure!( + state[2..] == before[2..], + "physical write failed before pressure" + ); + Ok(state[0] == before[0] + 1) + }) + .await?; + // A TLS reader would prefetch and empty the kernel window. Observe + // only the server's fixed physical-write phase without client reads. + for _ in 0..3 { + if writes(h)?[1] != before[1] { + break; + } + tokio::time::sleep(Duration::from_millis(100)).await; + } + let state = writes(h)?; + anyhow::ensure!( + state[2..] == before[2..], + "physical write failed while filling window" + ); + if state[1] == before[1] { + anyhow::ensure!( + h.controllers[0].usage().retained_s3 == 1 + && h.fixture.transports[0].usage().query_items == 1, + "incomplete physical write retains its owners" + ); + return Ok(bindings); + } + assert_eq!(state[1], before[1] + 1); + refunded(h).await?; + } + anyhow::bail!("bounded undrained window did not create physical backpressure") +} + +async fn complete( + connection: &mut Connection, + binding: &[u8; 32], + kind: &str, + sequence: usize, +) -> Result<()> { + let (reply, length, _) = connection + .receive_head(b"CLQA", b"cellar-raft-query-response-1", binding) + .await?; + assert_eq!(reply["request"], serde_json::json!(binding)); + assert_eq!(reply["sequence"], sequence.to_string()); + assert!(length > 28 * 1024); + let bytes = connection.receive_body(length).await?; + assert_eq!( + &bytes[..8], + if kind == "command" { + b"CLGCCS01" + } else { + b"CLGCRS01" + } + ); + assert_eq!(&bytes[8..10], &[0, 2]); + assert_eq!( + reply["digest"], + serde_json::json!(<[u8; 32]>::from(Sha256::digest(&bytes))) + ); + assert_eq!(&bytes[bytes.len() - 28 * 1024..], vec![7; 28 * 1024]); + Ok(()) +} + +async fn completed_prefix( + connection: &mut Connection, + bindings: &[[u8; 32]], + kind: &str, +) -> Result<()> { + for (i, binding) in bindings.iter().enumerate().take(bindings.len() - 1) { + complete(connection, binding, kind, i + 1).await?; + } + Ok(()) +} + +pub(crate) async fn s3_remote_slow_delivery( + rooted: bool, + store: Arc, +) -> Result<()> { + for kind in ["command", "receipt"] { + for fault in ["drain", "disconnect", "revoke", "timeout"] { + let store = Arc::new(object_store::prefix::PrefixStore::new( + store.clone(), + ObjectPath::from(format!("{kind}-{fault}")), + )); + // Exactly fills the original 64 KiB S3 envelope (8 KiB + 2*28 KiB). + // Only the real socket buffers are reduced; no node quota is raised. + let h = Harness::configured_transport( + rooted, + store, + 1, + Durability::Quorum, + Some(28 * 1024), + Some(4096), + ) + .await?; + let remote = Remote::start(&h, h.dispatcher.clone()).await?; + if kind == "receipt" { + let r = h + .command(1, b"held", Durability::Quorum, Duration::from_secs(15)) + .await? + .response() + .await?; + drop(r); + } + let mut connection = Connection::open_with_receive_buffer( + h.fixture.transports[0].query_address(), + b"EXPORTER-cellar-raft-query-1", + Some(1024), + ) + .await?; + let before = written(&h)?; + let bindings = fill(&h, &remote, &mut connection, kind).await?; + let binding = bindings.last().context("held reply")?; + assert_eq!(h.controllers[0].usage().retained_s3, 1, "{kind}-{fault}"); + assert_eq!(h.controllers[0].usage().s3_bytes, 64 * 1024); + assert_eq!(h.archive.tails.usage().operations, 1); + assert_eq!( + h.fixture.appliers[&(0, h.route.history)] + .0 + .usage()? + .retained_operations, + 0 + ); + let node = &h.controllers[0]; + controller::wait_for(async || Ok(node.usage().retained_commands == 0)).await?; + let successor = h + .command(2, b"successor", Durability::Quorum, Duration::from_secs(15)) + .await? + .response() + .await?; + drop(successor); + assert_eq!(h.controllers[0].usage().retained_s3, 1); + let mut overloaded = h.dispatcher.lookup_receipt( + h.cell.clone(), + h.receipt_request(1, b"held", Durability::S3)?, + bounds(), + None, + )?; + overloaded.begin().await?; + let refused = overloaded + .execute( + &h.fixture.runtimes[0], + &h.fixture.appliers[&(0, h.route.history)].0, + h.fixture.cell_options(&h.cell).await?, + ) + .await + .err() + .context("held wire proof bounds S3 admission")?; + assert!(matches!( + refused, + CellDispatchError::Runtime(RuntimeError::Overloaded) + | CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::Overloaded + )) + )); + drop(overloaded); + assert!(written(&h)? > before, "real socket output entered"); + match fault { + "drain" => { + completed_prefix(&mut connection, &bindings, kind).await?; + complete(&mut connection, binding, kind, bindings.len()).await?; + drop(connection); + } + "disconnect" => drop(connection), + "revoke" => { + h.fixture.transports[0].revoke_command_origin( + h.route.history.cluster, + h.route.history.namespace, + remote.router.peer.node, + 2, + )?; + refunded(&h).await?; + completed_prefix(&mut connection, &bindings, kind).await?; + assert!(complete(&mut connection, binding, kind, bindings.len()) + .await + .is_err()); + drop(connection); + h.fixture.transports[0].grant_command_origin( + cellar::application::consensus::NamespaceCommandOrigin { + cluster: h.route.history.cluster, + namespace: h.route.history.namespace, + revision: 3, + peer: remote.router.peer.clone(), + }, + )?; + } + "timeout" => { + refunded(&h).await?; + completed_prefix(&mut connection, &bindings, kind).await?; + assert!(complete(&mut connection, binding, kind, bindings.len()) + .await + .is_err()); + drop(connection); + } + _ => unreachable!(), + } + refunded(&h).await?; + assert_eq!(h.calls.load(Ordering::SeqCst), 2); + assert!(h.controllers[0].group_failure(h.route.history)?.is_none()); + let retry = command(&h, &remote, 1, b"held", Durability::S3).await?; + assert_eq!(retry.result(), vec![7; 28 * 1024]); + assert_eq!(retry.durability(), Durability::S3); + drop(retry); + refunded(&h).await?; + assert_eq!(h.calls.load(Ordering::SeqCst), 2); + remote.close().await?; + h.close().await?; + } + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_s3_response.rs b/crates/cellar/tests/support/cell_group_s3_response.rs new file mode 100644 index 000000000..14d0a08f4 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_s3_response.rs @@ -0,0 +1,132 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! A reachable S3 backend cannot substitute for an exact output durability proof. +use super::*; +use std::sync::atomic::{AtomicUsize, Ordering}; + +pub(crate) async fn unproved_s3_responses( + rooted: bool, + store: Arc, +) -> Result<()> { + let callbacks = Arc::new(AtomicUsize::new(0)); + let initializations = callbacks.clone(); + let commands = callbacks.clone(); + let class = Arc::new( + RaftClass::new("Counter", 1, [81; 32])? + .initializer([82; 32], move |_, _| { + initializations.fetch_add(1, Ordering::SeqCst); + Ok(()) + })? + .command("Add", [83; 32], move |_, _| { + commands.fetch_add(1, Ordering::SeqCst); + Ok(b"unproved".to_vec()) + })?, + ); + let (fixture, _, cell, _) = + Box::pin(registered::setup_store(rooted, &class, store.clone())).await?; + let mut minimum = NamespaceConfig::new( + cell.cluster, + cell.namespace, + cellar_logic::NamespaceMode::Raft, + class.schema_digest(), + ); + minimum.minimum_durability = Durability::S3; + let catalog = Catalog::new( + Arc::new(object_store::prefix::PrefixStore::new(store, "s3-minimum")), + Default::default(), + )?; + let namespace = fixture + .cluster + .domain + .run(catalog.open_namespace(minimum)) + .await??; + let required = RangeDirectory::new(namespace, Default::default())?; + for (directory, durability) in [ + (fixture.cluster.directory.clone(), Durability::S3), + (required, Durability::Quorum), + ] { + let dispatcher = CellGroupDispatcher::new( + &fixture.cluster.domain, + directory, + class.clone(), + VersionKeys::new( + VersionSigningKey::new([89; 32], u64::MAX)?, + vec![], + Duration::from_secs(30), + )?, + CellDispatchLimits { + max_cached_routes: 32, + route_cache_bytes: 256 * 1024, + max_invocations: 8, + retained_bytes: 1024 * 1024, + max_input_bytes: 1024, + }, + )?; + let bounds = CellRoutingLimits { + timeout: Duration::from_secs(5), + max_attempts: 2, + }; + let id = CommandId::new(ClientId::new([188; 16])?, 1)?; + let contract = class.contract("Add").context("Add")?; + let error = dispatcher + .command( + cell.clone(), + CellCommandRequest { + id, + contract: contract.clone(), + input: vec![], + durability, + retry_until_ms: u64::MAX, + }, + bounds, + None, + ) + .await + .err() + .context("unproved S3 command refused")?; + assert!(!error.ambiguous); + refusal(&error.cause); + let error = dispatcher + .lookup_receipt( + cell.clone(), + CellReceiptRequest { + id, + contract, + input: vec![], + durability, + }, + bounds, + None, + ) + .err() + .context("unproved S3 receipt refused")?; + refusal(&error); + let error = dispatcher + .enroll( + cell.clone(), + CellEnrollmentRequest { + creation: ProposalId::new([188; 16])?, + durability, + }, + bounds, + None, + ) + .await + .err() + .context("unproved S3 enrollment refused")?; + assert!(!error.ambiguous); + refusal(&error.cause); + assert_eq!(dispatcher.usage(), (0, 0)); + } + assert_eq!(callbacks.load(Ordering::SeqCst), 0); + fixture.close().await +} + +fn refusal(error: &CellDispatchError) { + assert!(matches!( + error, + CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::UnsupportedCapability + )) + )); +} diff --git a/crates/cellar/tests/support/cell_group_seed.rs b/crates/cellar/tests/support/cell_group_seed.rs new file mode 100644 index 000000000..91f3be15a --- /dev/null +++ b/crates/cellar/tests/support/cell_group_seed.rs @@ -0,0 +1,383 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Live-quorum seed source boundaries; no fresh destination is admitted here. +use super::*; +use cellar::application::consensus::verification::JoiningMember; +use cellar_logic::consensus::ReplicaIdentity; +use raft::prelude::{ConfChangeTransition, ConfChangeType}; + +fn target(history: GroupHistory) -> Result> { + Ok(ReplicaIdentity { + scope: history, + node: NodeId::new([4; 16])?, + disk: DiskLineage::new([14; 16])?, + member: MemberId::new(4)?, + process: ProcessGeneration::new([24; 16])?, + }) +} + +async fn join(fixture: &RuntimeCluster, route: &GroupRoute) -> Result<()> { + let target = target(route.history)?; + membership::configure( + fixture, + route, + membership::change( + 4, + ConfChangeType::AddLearnerNode, + ConfChangeTransition::Auto, + ), + &[JoiningMember { + member: target.member, + node: target.node, + disk: target.disk, + }], + ) + .await?; + Ok(()) +} + +async fn authorize( + fixture: &RuntimeCluster, + history: GroupHistory, +) -> Result { + Ok(fixture.runtimes[0] + .seed_authority(target(history)?, reads::deadline(fixture, 10_000).await?) + .await?) +} + +async fn capture( + fixture: &RuntimeCluster, + history: GroupHistory, + pool: &CellGroupCheckpointPool, +) -> Result { + let authority = authorize(fixture, history).await?; + let pool = pool.clone(); + Ok(fixture + .cluster + .domain + .blocking(move || authority.capture(&pool))? + .await??) +} + +pub(crate) async fn seed_source(rooted: bool) -> Result<()> { + let class = RaftClass::new("Counter", 1, [81; 32])?; + let (mut fixture, route, _, _) = Box::pin(registered::setup(rooted, &class)).await?; + let second = controller::add_second(&mut fixture, route.history).await?; + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + fixture.runtimes[0].campaign(second).await?; + controller::wait_for(async || Ok(fixture.runtimes[0].status(second).await?.ready)).await?; + let group = fixture.cluster.voters[0].log.cell_group(&route.history)?; + let original = group.group_manifest()?.context("root")?; + assert!(fixture.runtimes[0] + .seed_authority( + target(route.history)?, + reads::deadline(&fixture, 1000).await? + ) + .await + .is_err()); + join(&fixture, &route).await?; + for wrong_disk in [false, true] { + let mut invalid = target(route.history)?; + if wrong_disk { + invalid.disk = DiskLineage::new([99; 16])?; + } else { + invalid.node = NodeId::new([99; 16])?; + } + assert!(fixture.runtimes[0] + .seed_authority(invalid, reads::deadline(&fixture, 1000).await?) + .await + .is_err()); + } + let mut voter = target(route.history)?; + voter.member = MemberId::new(1)?; + voter.node = NodeId::new([1; 16])?; + voter.disk = DiskLineage::new([11; 16])?; + assert!(fixture.runtimes[0] + .seed_authority(voter, reads::deadline(&fixture, 1000).await?) + .await + .is_err()); + assert!(matches!( + fixture.runtimes[1] + .seed_authority( + target(route.history)?, + reads::deadline(&fixture, 1000).await? + ) + .await, + Err(RuntimeError::NotReady | RuntimeError::NotLeader) + )); + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let mut source = capture(&fixture, route.history, &pool).await?; + assert_eq!(source.target(), &target(route.history)?); + assert_eq!(source.marker().root(), original.root()); + assert!(source.marker().position().index >= source.barrier().index); + assert!(source.barrier().index >= group.membership()?.revision()); + let expected = source.snapshot()?; + let receiver_pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let applier = fixture.appliers[&(2, route.history)].1.clone(); + let scratch = super::super::checkpoint_receive::scratch()?; + let directory = fixture.cluster._root.path().to_path_buf(); + let prepared = fixture + .cluster + .domain + .blocking(move || -> Result<_> { + let mut receiver = receiver_pool.receive(applier, scratch, directory)?; + source.stream(|chunk| receiver.append(chunk))?; + assert!(source.stream(|_| Ok(())).is_err(), "one-shot stream"); + Ok(receiver.finish()?) + })? + .await??; + assert_eq!(prepared.snapshot(), &expected); + assert_eq!(prepared.marker().root(), original.root()); + drop(prepared); + assert_eq!(pool.usage().sources, 0); + + // A captured complete cut may stream while later cell enrollment advances + // the group. An authority not yet captured must instead refuse that old cut. + let old = authorize(&fixture, route.history).await?; + let mut captured = capture(&fixture, route.history, &pool).await?; + let before = captured.marker().clone(); + let cell = history(&route, "after-seed-capture")?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(73);")?; + let accepted = fixture.runtimes[0] + .propose( + route.history, + apply::enrollment_bytes(&route, &cell, &bytes, rooted)?, + ) + .await?; + controller::applied(&fixture, route.history, accepted.position.index).await?; + let stale_pool = pool.clone(); + assert!(fixture + .cluster + .domain + .blocking(move || old.capture(&stale_pool))? + .await? + .is_err()); + fixture + .cluster + .domain + .blocking(move || -> Result<()> { + captured.stream(|chunk| { + if let CellGroupCheckpointPart::Materialized(cell) + | CellGroupCheckpointPart::ExtentManifest(cell) = chunk.part + { + assert_ne!(cell.as_str(), "after-seed-capture"); + } + Ok(()) + })?; + assert_eq!(captured.marker(), &before); + Ok(()) + })? + .await??; + + // A fresh ReadIndex also works without a cell; seeding cannot be implemented + // by choosing an arbitrary enrolled cell or replaying its cached observation. + let empty_route = fixture.cluster.voters[0] + .log + .cell_group(&second)? + .group_manifest()? + .context("empty group")? + .route() + .clone(); + join(&fixture, &empty_route).await?; + let mut empty = capture(&fixture, second, &pool).await?; + assert_eq!(empty.marker().root().cells(), 0); + fixture + .cluster + .domain + .blocking(move || -> Result<()> { + let mut header = 0; + empty.stream(|chunk| { + assert_eq!(chunk.part, CellGroupCheckpointPart::Header); + header += chunk.bytes.len(); + Ok(()) + })?; + assert!(header > 0); + Ok(()) + })? + .await??; + + // Source authority shares the existing per-group read limit; another group's + // progress and fresh barrier remain available when this group fills it. + let mut held = Vec::new(); + for _ in 0..32 { + held.push(authorize(&fixture, route.history).await?); + } + assert!(matches!( + fixture.runtimes[0] + .seed_authority( + target(route.history)?, + reads::deadline(&fixture, 1000).await? + ) + .await, + Err(RuntimeError::Overloaded) + )); + drop(authorize(&fixture, second).await?); + drop(held); + let revoked = authorize(&fixture, route.history).await?; + let mut streaming = capture(&fixture, route.history, &pool).await?; + let (entered, started) = tokio::sync::oneshot::channel(); + let (release, hold) = std::sync::mpsc::sync_channel(1); + let worker = fixture.cluster.domain.blocking(move || { + let mut entered = Some(entered); + streaming.stream(|_| { + let _ = entered + .take() + .expect("only the first chunk is allowed") + .send(()); + hold.recv_timeout(Duration::from_secs(10)) + .map_err(|e| CellGroupCheckpointError::Sink(std::io::Error::other(e)))?; + Ok(()) + }) + })?; + tokio::time::timeout(Duration::from_secs(5), started).await??; + assert_eq!(pool.usage().sources, 1); + membership::configure( + &fixture, + &route, + membership::change(4, ConfChangeType::RemoveNode, ConfChangeTransition::Auto), + &[], + ) + .await?; + assert!(revoked.check().is_err()); + assert_eq!( + pool.usage().sources, + 1, + "revocation cannot release a blocked physical owner" + ); + release.send(())?; + let refused = worker.await?; + assert_eq!(pool.usage().sources, 0); + assert!(matches!( + revoked.capture(&pool), + Err(CellGroupCheckpointError::Authority(_)) + )); + assert!(matches!( + refused, + Err(CellGroupCheckpointError::Authority(_)) + )); + assert!(fixture.runtimes[0] + .seed_authority( + target(route.history)?, + reads::deadline(&fixture, 1000).await? + ) + .await + .is_err()); + + let revoked = capture(&fixture, second, &pool).await?; + controller::wait_for( + async || match fixture.runtimes[0].transfer_leader(second, 2).await { + Ok(()) => Ok(true), + Err(RuntimeError::NotReady) => Ok(false), + Err(e) => Err(e.into()), + }, + ) + .await?; + assert!(revoked.check().is_err()); + controller::wait_for(async || Ok(fixture.runtimes[1].status(second).await?.ready)).await?; + let new = fixture.runtimes[1] + .seed_authority(target(second)?, reads::deadline(&fixture, 1000).await?) + .await?; + assert!(new.barrier().term > revoked.barrier().term); + drop(new); + drop(revoked); + for controller in &controllers { + controller.close().await?; + } + drop(controllers); + assert_eq!(pool.usage().sources, 0); + fixture.close().await +} + +pub(crate) async fn seed_minority() -> Result<()> { + let class = RaftClass::new("Counter", 1, [81; 32])?; + let (mut fixture, route, _, _) = Box::pin(registered::setup(false, &class)).await?; + let second = controller::add_second(&mut fixture, route.history).await?; + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + fixture.runtimes[0].campaign(second).await?; + controller::wait_for(async || Ok(fixture.runtimes[0].status(second).await?.ready)).await?; + join(&fixture, &route).await?; + let second_route = fixture.cluster.voters[0] + .log + .cell_group(&second)? + .group_manifest()? + .context("second")? + .route() + .clone(); + join(&fixture, &second_route).await?; + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + // A timeout during the final sink call must refuse End, even if the group + // remains leader and there are no further chunks at which to notice expiry. + let authority = fixture.runtimes[0] + .seed_authority(target(second)?, reads::deadline(&fixture, 500).await?) + .await?; + let capture_pool = pool.clone(); + let expired = fixture + .cluster + .domain + .blocking(move || -> Result<_> { + let mut source = authority.capture(&capture_pool)?; + assert_eq!(source.marker().root().cells(), 0); + let result = source.stream(|chunk| { + assert_eq!(chunk.part, CellGroupCheckpointPart::Header); + assert_eq!(chunk.offset, 0); + assert_eq!( + chunk.length, + chunk.bytes.len() as u64, + "only the final header chunk" + ); + std::thread::sleep(Duration::from_millis(600)); + Ok(()) + }); + Ok(result) + })? + .await??; + assert!(matches!( + expired, + Err(CellGroupCheckpointError::Authority( + RuntimeError::DeadlineExceeded + )) + )); + // Drain an earlier successful barrier, then isolate this group on all routes. + // The runtime still reports ready because fixture clocks do not auto-tick; + // a fresh seed must prove quorum instead of trusting that cached status. + drop(authorize(&fixture, route.history).await?); + for transport in &fixture.transports { + transport.remove_physical_group(&route.history)?; + } + assert!(fixture.runtimes[0].status(route.history).await?.ready); + for _ in 0..3 { + assert!(matches!( + fixture.runtimes[0] + .seed_authority( + target(route.history)?, + reads::deadline(&fixture, 100).await? + ) + .await, + Err(RuntimeError::DeadlineExceeded) + )); + } + let authority = authorize(&fixture, second).await?; + let accepted = fixture.runtimes[0] + .propose( + second, + apply::enrollment_bytes( + &second_route, + &history(&second_route, "healthy")?, + &images::image(None, "CREATE TABLE value(n);")?, + false, + )?, + ) + .await?; + controller::applied(&fixture, second, accepted.position.index).await?; + for controller in &controllers { + controller.close().await?; + } + assert!( + authority.check().is_err(), + "shutdown revokes a selected source" + ); + drop(authority); + drop(controllers); + assert_eq!(pool.usage().sources, 0); + fixture.close().await +} diff --git a/crates/cellar/tests/support/cell_group_seed_crash.rs b/crates/cellar/tests/support/cell_group_seed_crash.rs new file mode 100644 index 000000000..cb15f19cf --- /dev/null +++ b/crates/cellar/tests/support/cell_group_seed_crash.rs @@ -0,0 +1,372 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Owned process kills around an authenticated, whole-group learner install. +//! Frozen synthetic catalog bytes are fixture input, not an S3 recovery protocol. +use super::crash_fixture::{save, stop_at}; +use super::*; +use cellar::application::consensus::verification::JoiningMember; +use raft::prelude::{ConfChangeTransition, ConfChangeType}; +use raft::Storage; +use serde::{Deserialize, Serialize}; +use std::path::Path; + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Oracle { + rooted: bool, + image_storage: RaftImageStorage, + source_root: std::path::PathBuf, + cut: String, + staged: GroupManifest, + expected: GroupManifest, + objects: Vec<(String, Vec)>, +} + +pub(crate) async fn seed_crash(rooted: bool) -> Result<()> { + for cut in [ + "staged", + "image-prefix", + "image-synced", + "before-sync", + "after-sync", + "published", + ] { + let directory = tempfile::tempdir()?; + let root = directory.path(); + crash_fixture::kill_at(root, "seed_process_fixture", rooted, cut).await?; + Box::pin(recover(root, rooted, cut)) + .await + .with_context(|| format!("cold recovery {rooted}/{cut}"))?; + } + Ok(()) +} + +pub(crate) async fn seed_process_fixture() -> Result<()> { + let (root, rooted, cut) = crash_fixture::parameters()?; + anyhow::ensure!( + matches!( + cut.as_str(), + "staged" + | "image-prefix" + | "image-synced" + | "retry-prefix" + | "retry-synced" + | "before-sync" + | "after-sync" + | "published" + ), + "invalid crash cut" + ); + let class = RaftClass::new("Counter", 1, [81; 32])?; + let (mut fixture, route, _, _) = Box::pin(registered::setup_in( + rooted, + &class, + tempfile::tempdir_in(&root)?, + Arc::new(DirectFileSystem), + )) + .await?; + let _controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let local = seed_install::target(route.history)?; + membership::configure( + &fixture, + &route, + membership::change( + 4, + ConfChangeType::AddLearnerNode, + ConfChangeTransition::Auto, + ), + &[JoiningMember { + member: local.member, + node: local.node, + disk: local.disk, + }], + ) + .await?; + let base = root.join("learner"); + let gate = fault::Gate::new(Some(root.join("signal"))); + let log = fixture + .cluster + .domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + base.join("log"), + local.node, + local.disk, + local.process, + LogOpenMode::Create, + ), + Arc::new(fault::FaultFileSystem { gate: gate.clone() }), + )) + .await??; + let (runtime, _events) = CellGroupRuntime::start( + log.clone(), + RaftRuntimeOptions { + tick_interval_ms: None, + ..Default::default() + }, + ) + .await?; + let transport = tcp::start(&runtime, 3).await?; + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + transport.enable_group_checkpoints(pool.clone())?; + // The complete transfer uses the production source and destination over TLS. + // Learner enrollment is committed before its authenticated route is installed. + let mut physical = fixture.tcp_route(route.history)?; + physical.revision += 1; + physical.peers.push(RaftPeer { + replica: local.clone(), + label: "node-4".into(), + server_name: "localhost".into(), + control_address: transport.control_address(), + data_address: transport.data_address(), + query_address: transport.query_address(), + }); + let source_pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + fixture.transports[0].enable_group_checkpoints(source_pool.clone())?; + for peer in &fixture.transports { + peer.install_physical_group(physical.clone())?; + } + let _registration = fixture.transports[0] + .register_group_checkpoint(fixture.appliers[&(0, route.history)].1.clone())?; + let session = Box::pin(transport.prepare_group_seed( + local, + physical.peers[0].clone(), + fixture.cluster.directory.clone(), + fixture.cluster.birth.clone(), + reads::deadline(&fixture, 10_000).await?, + )) + .await?; + let group = session.group().clone(); + let staged = group.group_manifest()?.context("staged root")?; + let expected = fixture.cluster.voters[0] + .log + .cell_group(&route.history)? + .group_manifest()? + .context("source root")?; + // The catalog is external to the learner disk. Freeze its synthetic objects + // before the fault, including the permanent destination claim. The parent + // never manufactures a replacement claim from the recovered disk. + let objects = crash_fixture::capture_catalog(fixture.cluster.store.as_ref()).await?; + save( + &root.join("oracle.json"), + &serde_json::to_vec(&Oracle { + rooted, + image_storage: fixture.cluster.directory.namespace_config().image_storage, + source_root: fixture.cluster._root.path().to_path_buf(), + cut: cut.clone(), + staged, + expected: expected.clone(), + objects, + })?, + )?; + if cut == "staged" { + return stop_at(&root).await; + } + let image_gate = fault::Gate::new(None); + let (pages, images) = seed_install::stores_with_filesystem( + &base, + rooted, + Arc::new(fault::FaultFileSystem { + gate: image_gate.clone(), + }), + )?; + let (applier, empty) = fixture + .cluster + .domain + .blocking(move || { + CellGroupApplier::recover( + group, + pages, + images, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + ) + })? + .await??; + assert_eq!(empty.marker().root().cells(), 0); + drop(empty); + let _release = match cut.as_str() { + "before-sync" => Some(gate.arm(fault::CRASH_BEFORE_SYNC)), + "after-sync" => Some(gate.arm(fault::CRASH_AFTER_SYNC)), + _ => None, + }; + if matches!( + cut.as_str(), + "image-prefix" | "image-synced" | "retry-prefix" | "retry-synced" + ) { + let _release_image = image_gate.arm(if cut.ends_with("prefix") { + fault::HOLD_DEPENDENCY_PREFIX + } else { + fault::HOLD_DEPENDENCY_SYNC + }); + let _installer = fixture + .cluster + .domain + .spawn(async move { session.install(applier).await })?; + tokio::time::timeout(Duration::from_secs(5), image_gate.entered.notified()).await?; + let dependency = image_gate + .dependency_cut + .lock() + .unwrap() + .clone() + .context("physical dependency cut")?; + assert!(dependency.1 > 0); + let group = log.cell_group(&route.history)?; + assert_eq!( + group + .group_manifest()? + .context("staged at image cut")? + .root() + .cells(), + 0 + ); + assert!(group.checkpoint_manifest()?.is_none()); + assert_eq!(pool.usage().sources, 1); + assert!(fixture + .cluster + .domain + .run(fixture.cluster.directory.bootstrap_joined_group_member( + &fixture.cluster.birth, + MemberId::new(4)?, + &log + )) + .await? + .is_err()); + save( + &root.join("dependency.json"), + &serde_json::to_vec(&dependency)?, + )?; + return stop_at(&root).await; + } + let (_permit, installed) = session.install(applier).await?; + assert_eq!( + cut, "published", + "a held physical fence cannot complete before kill" + ); + assert_eq!(installed.marker(), &expected); + stop_at(&root).await +} + +async fn recover(root: &Path, rooted: bool, cut: &str) -> Result<()> { + let oracle: Oracle = serde_json::from_slice(&std::fs::read(root.join("oracle.json"))?)?; + assert_eq!(oracle.rooted, rooted); + assert_eq!(oracle.cut, cut); + if matches!(cut, "image-prefix" | "image-synced") { + let (path, length): (std::path::PathBuf, u64) = + serde_json::from_slice(&std::fs::read(root.join("dependency.json"))?)?; + assert!(length > 0); + if rooted { + assert_eq!(path.parent(), Some(root.join("learner/extents").as_path())); + assert_eq!(std::fs::metadata(&path)?.len(), length); + let name = path + .file_name() + .context("extent file")? + .to_str() + .context("extent name")?; + let complete = u64::from_str_radix(&name[2..10], 16)?; + if cut == "image-prefix" { + assert!(length < complete); + } else { + assert_eq!(length, complete); + } + } else { + assert_eq!(path, std::path::PathBuf::from("snapshot.scratch")); + } + } + let route = oracle.expected.route().clone(); + let local = seed_install::target(route.history)?; + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let (directory, birth) = + crash_fixture::restore_catalog(oracle.objects, &route, &domain, oracle.image_storage) + .await?; + let base = root.join("learner"); + let log = domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + base.join("log"), + local.node, + local.disk, + ProcessGeneration::new([94; 16])?, + LogOpenMode::Resume, + ), + Arc::new(DirectFileSystem), + )) + .await??; + let group = log.cell_group(&route.history)?; + let selected = group.group_manifest()?.context("recovered selected root")?; + match cut { + "staged" | "image-prefix" | "image-synced" => assert_eq!(selected, oracle.staged), + "before-sync" => assert!(selected == oracle.staged || selected == oracle.expected), + "after-sync" | "published" => assert_eq!(selected, oracle.expected), + _ => unreachable!(), + } + let (pages, images) = seed_install::stores(&base, rooted)?; + let recovering = group.clone(); + let (applier, recovered) = domain + .blocking(move || { + CellGroupApplier::recover( + recovering, + pages, + images, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + ) + })? + .await??; + assert_eq!(recovered.marker(), &selected); + let permit = domain + .run(directory.bootstrap_joined_group_member(&birth, local.member, &log)) + .await?; + if selected == oracle.expected { + assert_eq!( + group.checkpoint_manifest()?.as_ref(), + Some(&oracle.expected) + ); + assert_eq!(recovered.pinned_images(), 2); + for name in ["registered-a", "registered-b"] { + let cell = history(&route, name)?; + let (leaf, _) = recovered.pin_cell(&cell.cell).context("recovered cell")?; + assert_eq!(leaf.version.history, cell); + } + assert_eq!(group.initial_state()?.conf_state.learners, vec![4]); + let (runtime, events) = CellGroupRuntime::start( + log.clone(), + RaftRuntimeOptions { + tick_interval_ms: None, + ..Default::default() + }, + ) + .await?; + runtime.attach(permit?, recovered.clone()).await?; + assert!(!runtime.status(route.history).await?.ready); + let bytes = images::image(None, "CREATE TABLE value(n);")?; + let proposal = + apply::enrollment_bytes(&route, &history(&route, "without-quorum")?, &bytes, rooted)?; + assert!(matches!( + runtime.propose(route.history, proposal).await, + Err(RuntimeError::NotReady) + )); + assert_eq!(group.group_manifest()?.as_ref(), Some(&oracle.expected)); + runtime.close().await?; + drop(events); + } else { + assert!(permit.is_err(), "empty stage has no local seed receipt"); + assert!(group.checkpoint_manifest()?.is_none()); + assert_eq!(recovered.pinned_images(), 0); + } + eprintln!( + "seed crash rooted={rooted} cut={cut} killed=true recovered_cells={} recovered_index={}", + selected.root().cells(), + selected.position().index.get() + ); + drop((applier, recovered, group)); + log.close().await?; + domain.shutdown().await; + Ok(()) +} + +#[path = "cell_group_seed_retry.rs"] +mod retry; +pub(crate) use retry::seed_retry; diff --git a/crates/cellar/tests/support/cell_group_seed_install.rs b/crates/cellar/tests/support/cell_group_seed_install.rs new file mode 100644 index 000000000..8f826eb9d --- /dev/null +++ b/crates/cellar/tests/support/cell_group_seed_install.rs @@ -0,0 +1,863 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Fresh learner staging, physical install, catch-up and deleted-disk replacement. +use super::*; +use cellar::application::consensus::verification::JoiningMember; +use cellar_logic::consensus::{OperationId, ReplicaIdentity}; +use cellar_ltx::internal::NodeExtentStore; +use raft::{ + prelude::{ConfChangeSingle, ConfChangeTransition, ConfChangeType, ConfChangeV2}, + Storage, +}; +use std::path::Path; + +pub(super) fn target(history: GroupHistory) -> Result> { + Ok(ReplicaIdentity { + scope: history, + node: NodeId::new([4; 16])?, + disk: DiskLineage::new([14; 16])?, + member: MemberId::new(4)?, + process: ProcessGeneration::new([24; 16])?, + }) +} + +pub(super) fn stores( + base: &Path, + rooted: bool, +) -> Result<(ManifestPageStore, CellGroupApplyImages)> { + stores_with_filesystem(base, rooted, Arc::new(DirectFileSystem)) +} + +pub(super) fn stores_with_filesystem( + base: &Path, + rooted: bool, + filesystem: Arc, +) -> Result<(ManifestPageStore, CellGroupApplyImages)> { + let pages = ManifestPageStore::open( + &base.join("pages"), + filesystem.clone(), + 1024, + 16 * 1024 * 1024, + 16, + )?; + let images = + GenerationStore::open(base.join("images"), filesystem.clone(), Default::default())?; + let extents = if rooted { + Some(NodeExtentStore::open( + base.join("extents"), + filesystem, + Default::default(), + )?) + } else { + None + }; + Ok(( + pages, + super::super::checkpoint_receive::config(images, extents, base)?, + )) +} + +async fn survivors(fixture: &RuntimeCluster, history: GroupHistory, index: LogIndex) -> Result<()> { + controller::wait_for(async || { + for runtime in &fixture.runtimes[..2] { + if runtime.status(history).await?.applied.index < index { + return Ok(false); + } + } + Ok(true) + }) + .await +} + +async fn configure( + fixture: &RuntimeCluster, + route: &GroupRoute, + command: ConfChangeV2, +) -> Result { + let group = fixture.cluster.voters[0].log.cell_group(&route.history)?; + let change = CellGroupMembershipChange::new(route, &group.membership()?, command, &[])?; + let accepted = tokio::time::timeout(Duration::from_secs(10), async { + loop { + match fixture.runtimes[0] + .configure(route.history, change.clone()) + .await + { + Ok(accepted) => return Ok::<_, anyhow::Error>(accepted), + Err(RuntimeError::NotReady) => { + fixture.runtimes[0].tick().await?; + tokio::time::sleep(Duration::from_millis(5)).await; + } + Err(error) => return Err(error.into()), + } + } + }) + .await??; + survivors(fixture, route.history, accepted.position.index).await?; + Ok(accepted.position.index) +} + +async fn prepare( + fixture: &RuntimeCluster, + transport: &RaftTransport, + source: RaftPeer, +) -> Result { + let deadline = reads::deadline(fixture, 10_000).await?; + for attempt in 0..16 { + let result = Box::pin(transport.prepare_group_seed( + target(source.replica.scope)?, + source.clone(), + fixture.cluster.directory.clone(), + fixture.cluster.birth.clone(), + deadline, + )) + .await; + match result { + Ok(session) => return Ok(session), + // A dropped exchange may still own the sole physical source slot. + // This staging operation reconciles the SAME identities and engine; + // retry only a transient transport refusal, within the original deadline. + Err(CellGroupSeedError::Transport( + TransportError::Io(_) | TransportError::Overloaded, + )) if attempt < 15 => tokio::time::sleep(Duration::from_millis(10)).await, + Err(error) => return Err(error.into()), + } + } + unreachable!("bounded retries always return") +} + +pub(crate) async fn seed_install(rooted: bool) -> Result<()> { + let class = RaftClass::new("Counter", 1, [81; 32])?; + let (fixture, route, a, b) = Box::pin(registered::setup(rooted, &class)).await?; + Box::pin(replace(fixture, route, a, b, rooted)).await +} + +async fn replace( + mut fixture: RuntimeCluster, + route: GroupRoute, + a: CellHistory, + b: CellHistory, + rooted: bool, +) -> Result<()> { + let second = controller::add_second(&mut fixture, route.history).await?; + fixture.connect_tcp().await?; + let source_pools = fixture + .transports + .iter() + .map(|transport| -> Result<_> { + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + transport.enable_group_checkpoints(pool.clone())?; + Ok(pool) + }) + .collect::>>()?; + let mut controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + fixture.runtimes[0].campaign(second).await?; + controller::wait_for(async || Ok(fixture.runtimes[0].status(second).await?.ready)).await?; + let local = target(route.history)?; + membership::configure( + &fixture, + &route, + membership::change( + 4, + ConfChangeType::AddLearnerNode, + ConfChangeTransition::Auto, + ), + &[JoiningMember { + member: local.member, + node: local.node, + disk: local.disk, + }], + ) + .await?; + + let mut physical = fixture.tcp_route(route.history)?; + // Remove a real voter's entire fixture disk before starting the transfer. + controllers.remove(2).close().await?; + fixture.appliers.retain(|(node, _), _| *node != 2); + fixture.cluster.voters[2].log.close().await?; + std::fs::remove_dir_all(fixture.cluster._root.path().join("node-3"))?; + if rooted { + std::fs::remove_dir_all(fixture.cluster._root.path().join("rooted-2"))?; + } + let erased = fixture + .cluster + .domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + fixture.cluster._root.path().join("node-3/log"), + NodeId::new([3; 16])?, + DiskLineage::new([13; 16])?, + ProcessGeneration::new([33; 16])?, + LogOpenMode::Create, + ), + Arc::new(DirectFileSystem), + )) + .await??; + assert!( + matches!( + fixture + .cluster + .domain + .run( + fixture + .cluster + .directory + .bootstrap_group_member(&fixture.cluster.birth, &erased) + ) + .await?, + Err(CatalogError::LostStableState) + ), + "old member identity must not recreate a lost disk" + ); + erased.close().await?; + + let base = fixture.cluster._root.path().join("node-4"); + let gate = fault::Gate::new(None); + let log = fixture + .cluster + .domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + base.join("log"), + local.node, + local.disk, + local.process, + LogOpenMode::Create, + ), + Arc::new(fault::FaultFileSystem { gate: gate.clone() }), + )) + .await??; + let (runtime, events) = CellGroupRuntime::start( + log.clone(), + RaftRuntimeOptions { + tick_interval_ms: None, + ..Default::default() + }, + ) + .await?; + let transport = tcp::start(&runtime, 3).await?; + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + transport.enable_group_checkpoints(pool.clone())?; + physical.revision = 2; + physical.peers.push(RaftPeer { + replica: local.clone(), + label: "node-4".into(), + server_name: "localhost".into(), + control_address: transport.control_address(), + data_address: transport.data_address(), + query_address: transport.query_address(), + }); + let mut registrations = Vec::new(); + for node in 0..2 { + fixture.transports[node].install_physical_group(physical.clone())?; + registrations.push( + fixture.transports[node] + .register_group_checkpoint(fixture.appliers[&(node, route.history)].1.clone())?, + ); + } + super::super::seed_probes::source_protocol( + &fixture, + &physical, + &source_pools[0], + &mut registrations[0], + ) + .await + .context("seed protocol probes")?; + // A stale ready flag cannot replace a fresh source quorum. Keep the seed + // connection discoverable while disconnecting the remaining voting peer. + let mut isolated = physical.clone(); + isolated.revision = 3; + isolated.peers.retain(|peer| peer.replica.member.get() != 2); + fixture.transports[1].remove_physical_group(&route.history)?; + fixture.transports[0].install_physical_group(isolated)?; + registrations[0] = fixture.transports[0] + .register_group_checkpoint(fixture.appliers[&(0, route.history)].1.clone())?; + assert!(fixture.runtimes[0].status(route.history).await?.ready); + assert!(Box::pin(transport.prepare_group_seed( + local.clone(), + physical.peers[0].clone(), + fixture.cluster.directory.clone(), + fixture.cluster.birth.clone(), + reads::deadline(&fixture, 100).await? + )) + .await + .is_err()); + assert!( + matches!(log.cell_group(&route.history), Err(LogError::NotFound)), + "unproven source must not create a stage" + ); + physical.revision = 4; + for (node, registration) in registrations.iter_mut().enumerate() { + fixture.transports[node].install_physical_group(physical.clone())?; + *registration = fixture.transports[node] + .register_group_checkpoint(fixture.appliers[&(node, route.history)].1.clone())?; + } + let baseline_tasks = transport.usage().tasks; + let source = physical.peers[0].clone(); + let session = Box::pin(prepare(&fixture, &transport, source.clone())) + .await + .context("authenticate and stage fresh learner")?; + assert!( + fixture + .cluster + .domain + .run(fixture.cluster.directory.bootstrap_joined_group_member( + &fixture.cluster.birth, + local.member, + &log + )) + .await? + .is_err(), + "empty stage must never grant startup" + ); + assert!( + fixture + .cluster + .domain + .run( + fixture + .cluster + .directory + .bootstrap_group_member(&fixture.cluster.birth, &log) + ) + .await? + .is_err(), + "learner is not a birth member" + ); + let group = session.group().clone(); + let (pages, images) = stores(&base, rooted)?; + let recovering = group.clone(); + let (applier, staged_root) = fixture + .cluster + .domain + .blocking(move || { + CellGroupApplier::recover( + recovering, + pages, + images, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + ) + })? + .await??; + assert_eq!(staged_root.marker().root().cells(), 0); + drop(staged_root); + drop(session); + controller::wait_for(async || Ok(transport.usage().tasks == baseline_tasks)) + .await + .context("cancel staged exchange")?; + let staged = group.group_manifest()?; + let revoked = Box::pin(prepare(&fixture, &transport, source.clone())).await?; + fixture.transports[0].remove_physical_group(&route.history)?; + assert!( + revoked.install(applier.clone()).await.is_err(), + "revoked source cannot install its previously signed cut" + ); + controller::wait_for(async || { + Ok(transport.usage().tasks == baseline_tasks && source_pools[0].usage().sources == 0) + }) + .await?; + assert_eq!(group.group_manifest()?, staged); + assert_eq!(group.checkpoint_manifest()?, None); + assert_eq!(pool.usage().sources, 0); + fixture.transports[0].install_physical_group(physical.clone())?; + registrations[0] = fixture.transports[0] + .register_group_checkpoint(fixture.appliers[&(0, route.history)].1.clone())?; + let expired = Box::pin(transport.prepare_group_seed( + local.clone(), + source.clone(), + fixture.cluster.directory.clone(), + fixture.cluster.birth.clone(), + reads::deadline(&fixture, 500).await?, + )) + .await?; + // Wait for the bounded exchange to expire while holding the user session. + // No sleep or fabricated process clock establishes the refusal. + controller::wait_for(async || { + Ok(transport.usage().tasks == baseline_tasks && source_pools[0].usage().sources == 0) + }) + .await?; + assert!( + expired.install(applier.clone()).await.is_err(), + "expired session cannot acquire a fresh install deadline" + ); + assert_eq!(group.group_manifest()?, staged); + assert_eq!(group.checkpoint_manifest()?, None); + assert!(fixture + .cluster + .domain + .run(fixture.cluster.directory.bootstrap_joined_group_member( + &fixture.cluster.birth, + local.member, + &log + )) + .await? + .is_err()); + // Even a fully verified generic snapshot install cannot mint the local + // authenticated seed receipt. Metadata and physical bytes alone are insufficient. + let authority = fixture.runtimes[0] + .seed_authority(local.clone(), reads::deadline(&fixture, 10_000).await?) + .await?; + let capture_pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let receiver_pool = pool.clone(); + let receiver_applier = applier.clone(); + let directory = base.clone(); + let prepared = fixture + .cluster + .domain + .blocking(move || -> Result<_> { + let mut source = authority.capture(&capture_pool)?; + let mut receiver = receiver_pool.receive( + receiver_applier, + super::super::checkpoint_receive::scratch()?, + directory, + )?; + source.stream(|chunk| receiver.append(chunk))?; + Ok(receiver.finish()?) + })? + .await??; + let mut write = GroupWrite::new(&group); + write.hard_state = Some(raft::prelude::HardState { + term: prepared.snapshot().get_metadata().term, + commit: prepared.snapshot().get_metadata().index, + ..Default::default() + }); + write.snapshot = Some(SnapshotChange::CellGroup(prepared)); + log.persist_cells(OperationId::new(1801), vec![write]) + .await?; + assert!( + fixture + .cluster + .domain + .run(fixture.cluster.directory.bootstrap_joined_group_member( + &fixture.cluster.birth, + local.member, + &log + )) + .await? + .is_err(), + "generic snapshot must not mint local seed receipt" + ); + let session = Box::pin(prepare(&fixture, &transport, source)) + .await + .context("retry same staged engine")?; + let expected = fixture.cluster.voters[0] + .log + .cell_group(&route.history)? + .group_manifest()? + .context("source cut")?; + + let release = gate.arm(fault::HOLD_SYNC); + let installer = applier.clone(); + let pending = fixture + .cluster + .domain + .spawn(async move { session.install(installer).await })?; + tokio::time::timeout(Duration::from_secs(5), gate.entered.notified()).await?; + assert!( + fixture + .cluster + .domain + .run(fixture.cluster.directory.bootstrap_joined_group_member( + &fixture.cluster.birth, + local.member, + &log + )) + .await? + .is_err(), + "receipt cannot precede the storage fence" + ); + assert_eq!( + pool.usage().sources, + 1, + "physical receiver retained through sync" + ); + // While the replacement receiver is physically held at its install fence, + // both the surviving group and an unrelated group must keep committing. + // These later mutations must reach the learner through the Raft suffix. + let later = history(&route, "after-seed-cut")?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(73);")?; + let accepted = fixture.runtimes[0] + .propose( + route.history, + apply::enrollment_bytes(&route, &later, &bytes, rooted)?, + ) + .await?; + survivors(&fixture, route.history, accepted.position.index).await?; + let other = fixture.cluster.voters[0] + .log + .cell_group(&second)? + .group_manifest()? + .context("second group")?; + let unrelated = fixture.runtimes[0] + .propose( + second, + apply::enrollment_bytes( + other.route(), + &history(other.route(), "while-seeding")?, + &bytes, + false, + )?, + ) + .await?; + survivors(&fixture, second, unrelated.position.index).await?; + let (permit, root) = if rooted { + drop(release); + pending.await?? + } else { + pending.abort(); + assert!(pending.await.is_err()); + assert_eq!( + pool.usage().sources, + 1, + "caller cancellation cannot release physical owner" + ); + drop(release); + controller::wait_for(async || Ok(transport.usage().tasks == baseline_tasks)).await?; + let permit = fixture + .cluster + .domain + .run(fixture.cluster.directory.bootstrap_joined_group_member( + &fixture.cluster.birth, + local.member, + &log, + )) + .await??; + let recovering = applier.clone(); + let root = fixture + .cluster + .domain + .blocking(move || recovering.recover_selected())? + .await??; + (permit, root) + }; + assert_eq!(root.marker(), &expected); + assert!(root.pin_cell(&a.cell).is_some()); + assert!(root.pin_cell(&b.cell).is_some()); + assert!(root.pin_cell(&later.cell).is_none()); + assert_eq!(group.initial_state()?.conf_state.learners, vec![4]); + transport.install_physical_group(physical.clone())?; + runtime.attach(permit, root).await?; + assert!( + !runtime.status(route.history).await?.ready, + "learner cannot serve writes" + ); + let sql = CellGroupSqlPool::start( + &log, + CellSqlLimits { + workers: 1, + queue_per_worker: 1, + max_pending_cells: 1, + group_pending_cells: 1, + node_bytes: 1024 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + }, + )?; + let controller = CellGroupController::start( + runtime.clone(), + events, + transport.clone(), + sql.clone(), + vec![applier.clone()], + controller::limits(), + ) + .await?; + controller::wait_for(async || { + fixture.runtimes[0].tick().await?; + Ok(runtime.status(route.history).await?.applied.index >= accepted.position.index) + }) + .await + .context("fresh learner catches up suffix")?; + assert_eq!( + group.group_manifest()?.context("caught up")?.root(), + fixture.cluster.voters[0] + .log + .cell_group(&route.history)? + .group_manifest()? + .context("leader")? + .root() + ); + + let joint = ConfChangeV2 { + transition: ConfChangeTransition::Explicit, + changes: vec![ + ConfChangeSingle { + node_id: 4, + change_type: ConfChangeType::AddNode, + ..Default::default() + }, + ConfChangeSingle { + node_id: 3, + change_type: ConfChangeType::RemoveNode, + ..Default::default() + }, + ] + .into(), + ..Default::default() + }; + let entered = configure(&fixture, &route, joint) + .await + .context("promote caught-up replacement")?; + controller::wait_for( + async || Ok(runtime.status(route.history).await?.applied.index >= entered), + ) + .await?; + let mut joint_conf = group.initial_state()?.conf_state; + joint_conf.voters.sort_unstable(); + joint_conf.voters_outgoing.sort_unstable(); + assert_eq!(joint_conf.voters, vec![1, 2, 4]); + assert_eq!(joint_conf.voters_outgoing, vec![1, 2, 3]); + let left = configure(&fixture, &route, ConfChangeV2::default()).await?; + controller::wait_for(async || Ok(runtime.status(route.history).await?.applied.index >= left)) + .await?; + assert!(group.initial_state()?.conf_state.voters_outgoing.is_empty()); + assert!(group.initial_state()?.conf_state.learners.is_empty()); + controller::wait_for(async || { + match fixture.runtimes[0].transfer_leader(route.history, 4).await { + Ok(()) => Ok(true), + Err(RuntimeError::NotReady) => Ok(false), + Err(error) => Err(error.into()), + } + }) + .await?; + controller::wait_for(async || Ok(runtime.status(route.history).await?.ready)).await?; + let final_cell = history(&route, "replacement-leader")?; + let committed = runtime + .propose( + route.history, + apply::enrollment_bytes(&route, &final_cell, &bytes, rooted)?, + ) + .await?; + survivors(&fixture, route.history, committed.position.index).await?; + controller::wait_for(async || { + Ok(runtime.status(route.history).await?.applied.index >= committed.position.index) + }) + .await?; + let current = group.group_manifest()?.context("replacement root")?; + let checkpoint = group + .checkpoint_manifest()? + .context("installed checkpoint")?; + controller.close().await?; + drop((controller, sql, applier)); + log.close().await?; + drop((transport, runtime, group, log)); + let resumed = fixture + .cluster + .domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + base.join("log"), + local.node, + local.disk, + ProcessGeneration::new([25; 16])?, + LogOpenMode::Resume, + ), + Arc::new(DirectFileSystem), + )) + .await??; + let permit = fixture + .cluster + .domain + .run(fixture.cluster.directory.bootstrap_joined_group_member( + &fixture.cluster.birth, + local.member, + &resumed, + )) + .await??; + let resumed_group = resumed.cell_group(&route.history)?; + assert_eq!( + resumed_group.checkpoint_manifest()?.as_ref(), + Some(&checkpoint) + ); + let (pages, images) = stores(&base, rooted)?; + let (applier, root) = fixture + .cluster + .domain + .blocking(move || { + CellGroupApplier::recover( + resumed_group, + pages, + images, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + ) + })? + .await??; + assert_eq!(root.marker(), ¤t); + for cell in [&a, &b, &later, &final_cell] { + assert!(root.pin_cell(&cell.cell).is_some()); + } + Box::pin(reconnect( + &fixture, physical, resumed, permit, root, applier, rooted, + )) + .await + .context("replacement process reconnect")?; + for controller in controllers { + controller.close().await?; + } + drop(registrations); + assert_eq!(pool.usage().sources, 0); + fixture.close().await +} + +/// Start a new process from the installed disk and use only production transport +/// and controllers to reacquire quorum, then prove a minority cannot commit. +async fn reconnect( + fixture: &RuntimeCluster, + mut physical: cellar::application::consensus::verification::GroupRoute, + log: ReplicaLog, + permit: CellGroupBootstrapPermit, + root: PreparedGroupManifest, + applier: CellGroupApplier, + rooted: bool, +) -> Result<()> { + let route = root.marker().route().clone(); + let group = log.cell_group(&route.history)?; + let (runtime, events) = CellGroupRuntime::start( + log.clone(), + RaftRuntimeOptions { + tick_interval_ms: None, + ..Default::default() + }, + ) + .await?; + runtime.attach(permit, root).await?; + assert!( + !runtime.status(route.history).await?.ready, + "disk receipt cannot renew quorum" + ); + let transport = tcp::start(&runtime, 3).await?; + transport + .enable_group_checkpoints(CellGroupCheckpointPool::new(checkpoint_fixture::limits())?)?; + physical.revision += 1; + physical.peers.retain(|peer| peer.replica.member.get() != 3); + let peer = physical + .peers + .iter_mut() + .find(|peer| peer.replica.member.get() == 4) + .context("replacement route")?; + peer.replica = group.identity().clone(); + peer.control_address = transport.control_address(); + peer.data_address = transport.data_address(); + peer.query_address = transport.query_address(); + for source in &fixture.transports[..2] { + source + .install_physical_group(physical.clone()) + .context("refresh survivor route after process restart")?; + } + transport + .install_physical_group(physical) + .context("install restarted voter route")?; + let sql = CellGroupSqlPool::start( + &log, + CellSqlLimits { + workers: 1, + queue_per_worker: 1, + max_pending_cells: 1, + group_pending_cells: 1, + node_bytes: 1024 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + }, + )?; + let controller = CellGroupController::start( + runtime.clone(), + events, + transport.clone(), + sql, + vec![applier], + controller::limits(), + ) + .await?; + runtime.campaign(route.history).await?; + controller::wait_for(async || { + // Every fixture voter has automatic ticking disabled. Survivors must + // also advance past the old process's leader lease before campaigning + // can succeed. If one wins first, use the real caught-up handoff path. + for survivor in &fixture.runtimes[..2] { + survivor.tick().await?; + if survivor.status(route.history).await?.ready { + match survivor.transfer_leader(route.history, 4).await { + Ok(()) | Err(RuntimeError::NotReady) => (), + Err(error) => return Err(error.into()), + } + } + } + runtime.tick().await?; + Ok(runtime.status(route.history).await?.ready) + }) + .await + .context("restarted voter regains quorum")?; + let cell = history(&route, "after-process-restart")?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(91);")?; + let accepted = runtime + .propose( + route.history, + apply::enrollment_bytes(&route, &cell, &bytes, rooted)?, + ) + .await?; + survivors(fixture, route.history, accepted.position.index).await?; + controller::wait_for(async || { + Ok(runtime.status(route.history).await?.applied.index >= accepted.position.index) + }) + .await?; + drop( + runtime + .read_authority( + route.history, + cell.clone(), + CellReadConsistency::Linearizable, + reads::deadline(fixture, 5000).await?, + ) + .await?, + ); + let before = group.group_manifest()?.context("quorum root")?; + for source in &fixture.transports[..2] { + source.remove_physical_group(&route.history)?; + } + transport.remove_physical_group(&route.history)?; + assert!( + runtime.status(route.history).await?.ready, + "exercise stale ready status" + ); + let pending_cell = history(&route, "isolated-cannot-commit")?; + let pending = runtime + .propose( + route.history, + apply::enrollment_bytes(&route, &pending_cell, &bytes, rooted)?, + ) + .await?; + // Admission is not a commit. A fresh correlated quorum read must fail even + // though the old leader can still enqueue a proposal during check-quorum lag. + assert!(matches!( + runtime + .read_authority( + route.history, + cell, + CellReadConsistency::Linearizable, + reads::deadline(fixture, 150).await? + ) + .await, + Err(RuntimeError::DeadlineExceeded) + )); + assert!(runtime.status(route.history).await?.applied.index < pending.position.index); + assert_eq!(group.group_manifest()?.as_ref(), Some(&before)); + controller::wait_for(async || { + runtime.tick().await?; + Ok(!runtime.status(route.history).await?.ready) + }) + .await?; + assert!(matches!( + runtime + .propose( + route.history, + apply::enrollment_bytes(&route, &pending_cell, &bytes, rooted)? + ) + .await, + Err(RuntimeError::NotLeader | RuntimeError::NotReady) + )); + assert_eq!(group.group_manifest()?.as_ref(), Some(&before)); + controller.close().await?; + log.close().await?; + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_seed_persistence.rs b/crates/cellar/tests/support/cell_group_seed_persistence.rs new file mode 100644 index 000000000..1d44f092e --- /dev/null +++ b/crates/cellar/tests/support/cell_group_seed_persistence.rs @@ -0,0 +1,464 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Authenticated seed grants cross the real engine queue and filesystem fence. +use super::*; +use cellar::application::consensus::verification::JoiningMember; +use cellar_logic::consensus::OperationId; +use raft::prelude::{ConfChangeTransition, ConfChangeType}; +use raft::Storage; +use tokio::net::TcpListener; + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum Cut { + StreamingRevocation, + QueuedRevocation, + EnteredRevocation, + EnteredExpiry, + SyncError, + AfterSyncError, +} + +pub(crate) async fn seed_transfer_revocation(rooted: bool) -> Result<()> { + Box::pin(run(rooted, Cut::StreamingRevocation)).await +} + +pub(crate) async fn seed_persistence(rooted: bool) -> Result<()> { + for cut in [ + Cut::QueuedRevocation, + Cut::EnteredRevocation, + Cut::EnteredExpiry, + Cut::SyncError, + Cut::AfterSyncError, + ] { + Box::pin(run(rooted, cut)) + .await + .with_context(|| format!("seed persistence {cut:?}"))?; + } + Ok(()) +} + +fn queued(log: &ReplicaLog) -> Result { + let sample = log.performance_snapshot(); + assert!(sample.available && !sample.saturated); + let queue = sample + .operations + .iter() + .find(|op| op.name == "engine_queue") + .context("queue metric")?; + Ok(queue.started - queue.succeeded - queue.failed - queue.abandoned) +} + +async fn run(rooted: bool, cut: Cut) -> Result<()> { + let class = RaftClass::new("Counter", 1, [81; 32])?; + let (mut fixture, route, a, b) = Box::pin(registered::setup(rooted, &class)).await?; + let mut controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let local = seed_install::target(route.history)?; + membership::configure( + &fixture, + &route, + membership::change( + 4, + ConfChangeType::AddLearnerNode, + ConfChangeTransition::Auto, + ), + &[JoiningMember { + member: local.member, + node: local.node, + disk: local.disk, + }], + ) + .await?; + let captured = seed_receiver::capture(&fixture, route.history).await?; + let expected = fixture.cluster.voters[0] + .log + .cell_group(&route.history)? + .group_manifest()? + .context("source root")?; + let listener = Arc::new(TcpListener::bind("127.0.0.1:0").await?); + let mut peer = fixture.tcp_route(route.history)?.peers[0].clone(); + peer.data_address = listener.local_addr()?; + let base = fixture.cluster._root.path().join("seed-persistence"); + let gate = fault::Gate::new(None); + let log = fixture + .cluster + .domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + base.join("log"), + local.node, + local.disk, + local.process, + LogOpenMode::Create, + ), + Arc::new(fault::FaultFileSystem { gate: gate.clone() }), + )) + .await??; + let (runtime, events) = CellGroupRuntime::start( + log.clone(), + RaftRuntimeOptions { + tick_interval_ms: None, + ..Default::default() + }, + ) + .await?; + let transport = tcp::start(&runtime, 3).await?; + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + transport.enable_group_checkpoints(pool.clone())?; + let baseline = transport.usage().tasks; + let service = fixture.cluster.domain.spawn(seed_receiver::serve( + listener, + captured, + seed_receiver::Fault::Valid, + ))?; + // The expiry cut must complete inside the existing peer's five-second + // disconnect guard. Other cuts keep their original ten-second budget. + let deadline = reads::deadline( + &fixture, + if cut == Cut::EnteredExpiry { + 2000 + } else { + 10_000 + }, + ) + .await?; + let session = Box::pin(transport.prepare_group_seed( + local.clone(), + peer.clone(), + fixture.cluster.directory.clone(), + fixture.cluster.birth.clone(), + deadline, + )) + .await?; + let group = session.group().clone(); + let staged = group.group_manifest()?.context("staged root")?; + let (pages, images) = seed_install::stores_with_filesystem( + &base, + rooted, + Arc::new(fault::FaultFileSystem { gate: gate.clone() }), + ) + .context("open destination stores")?; + let recovering = group.clone(); + let (applier, root) = fixture + .cluster + .domain + .blocking(move || { + CellGroupApplier::recover( + recovering, + pages, + images, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + ) + })? + .await??; + assert_eq!(root.marker(), &staged); + drop(root); + + let release = gate.arm(match cut { + Cut::StreamingRevocation => fault::HOLD_DEPENDENCY_PREFIX, + Cut::QueuedRevocation | Cut::EnteredRevocation | Cut::EnteredExpiry => fault::HOLD_SYNC, + Cut::SyncError => fault::FAIL_SYNC, + Cut::AfterSyncError => fault::FAIL_AFTER_SYNC, + }); + // Hold an unrelated node revision fence ahead of the seed. No other local + // group/controller is attached, so the next queued request is this install. + let ahead = if cut == Cut::QueuedRevocation { + let writer = log.clone(); + let ahead = fixture + .cluster + .domain + .spawn(async move { writer.reserve_node_revision().await })?; + tokio::time::timeout(Duration::from_secs(5), gate.entered.notified()).await?; + assert_eq!(queued(&log)?, 0); + Some(ahead) + } else { + None + }; + let installer = applier.clone(); + let mut pending = fixture + .cluster + .domain + .spawn(async move { session.install(installer).await })?; + let mut restarted_source = None; + let mut early_result = None; + if matches!( + cut, + Cut::StreamingRevocation + | Cut::QueuedRevocation + | Cut::EnteredRevocation + | Cut::EnteredExpiry + ) { + if cut == Cut::QueuedRevocation { + controller::wait_for(async || Ok(queued(&log)? == 1)).await?; + } else { + tokio::time::timeout(Duration::from_secs(5), gate.entered.notified()).await?; + } + assert_eq!(group.group_manifest()?.as_ref(), Some(&staged)); + assert!(group.checkpoint_manifest()?.is_none()); + assert_eq!(pool.usage().sources, 1, "physical closure remains owned"); + assert!( + fixture + .cluster + .domain + .run(fixture.cluster.directory.bootstrap_joined_group_member( + &fixture.cluster.birth, + local.member, + &log, + )) + .await? + .is_err(), + "no receipt before the engine fence" + ); + if cut == Cut::EnteredExpiry { + // Keep the actual fence held through the original operation deadline. + // Its eventual durable result must not grant a late bootstrap reply. + fixture + .cluster + .domain + .run(async { cellar::asyncrt::sleep_until(deadline).await }) + .await?; + } else { + restarted_source = Some( + seed_receiver::revoke_source(&fixture, &mut controllers, &transport, &peer, &route) + .await?, + ); + } + if cut == Cut::StreamingRevocation { + assert!( + gate.dependency_cut + .lock() + .unwrap() + .as_ref() + .context("held dependency")? + .1 + > 0 + ); + early_result = Some(tokio::time::timeout(Duration::from_secs(5), &mut pending).await??); + assert!( + transport.usage().data_items > 0, + "failed network waiter cannot refund the physical receive owner" + ); + assert!(pool.usage().retained_bytes > 0); + } else { + assert!( + tokio::time::timeout(Duration::from_millis(30), &mut pending) + .await + .is_err(), + "revocation cannot abandon entered/queued physical work" + ); + } + assert_eq!(pool.usage().sources, 1); + drop(release); + } else { + // The one-shot sync error consumes the armed fault before completion. + // Retain its release guard until the result has been observed. + } + let result = match early_result { + Some(result) => result, + None => pending.await?, + }; + if let Some(ahead) = ahead { + ahead.await??; + } + match cut { + Cut::StreamingRevocation => assert!( + matches!( + result, + Err(CellGroupSeedError::Transport( + TransportError::StalePeer | TransportError::Closed + )) + ), + "streaming revoked grant: {:?}", + result.err() + ), + Cut::QueuedRevocation => assert!( + matches!( + result, + Err(CellGroupSeedError::Log(LogError::WrongIdentity)) + ), + "queued revoked grant: {:?}", + result.err() + ), + Cut::EnteredRevocation => { + let (_permit, installed) = result?; + assert_eq!(installed.marker(), &expected); + assert!(installed.pin_cell(&a.cell).is_some() && installed.pin_cell(&b.cell).is_some()); + } + Cut::EnteredExpiry => { + assert!( + matches!( + result, + Err(CellGroupSeedError::Log(LogError::WrongIdentity)) + | Err(CellGroupSeedError::Transport(TransportError::Timeout)) + ), + "completed sync cannot extend the original deadline: {:?}", + result.err() + ); + assert_eq!(group.group_manifest()?.as_ref(), Some(&expected)); + assert_eq!(group.checkpoint_manifest()?.as_ref(), Some(&expected)); + } + Cut::SyncError | Cut::AfterSyncError => assert!( + matches!(result, Err(CellGroupSeedError::Log(LogError::Io(_)))), + "sync must report uncertainty: {:?}", + result.err() + ), + } + service.await??; + controller::wait_for(async || { + Ok(transport.usage().tasks == baseline && pool.usage().sources == 0) + }) + .await?; + assert_eq!(transport.usage().data_items, 0); + assert_eq!(pool.usage().retained_bytes, 0); + if matches!(cut, Cut::StreamingRevocation | Cut::QueuedRevocation) { + assert_eq!( + group.group_manifest()?.as_ref(), + Some(&staged), + "failed work cannot publish cache state" + ); + assert!(group.checkpoint_manifest()?.is_none()); + } + if matches!(cut, Cut::SyncError | Cut::AfterSyncError) { + assert!( + matches!(group.group_manifest(), Err(LogError::Closed)), + "uncertain engine cannot expose a selected root" + ); + assert!(matches!(group.checkpoint_manifest(), Err(LogError::Closed))); + assert!( + fixture + .cluster + .domain + .run(fixture.cluster.directory.bootstrap_joined_group_member( + &fixture.cluster.birth, + local.member, + &log, + )) + .await? + .is_err(), + "uncertain engine cannot grant bootstrap" + ); + assert!( + matches!( + log.persist_cells(OperationId::new(5101), vec![GroupWrite::new(&group)]) + .await, + Err(LogError::Closed) + ), + "uncertain engine refuses later writes" + ); + } + transport.close().await; + let runtime_closed = runtime.close().await; + if matches!(cut, Cut::SyncError | Cut::AfterSyncError) { + assert!( + matches!(runtime_closed, Err(RuntimeError::Storage(LogError::Closed))), + "failed engine must propagate through joined runtime close: {runtime_closed:?}" + ); + } else { + runtime_closed?; + } + drop((events, applier, group, runtime, transport)); + let closed = log.close().await; + if matches!(cut, Cut::SyncError | Cut::AfterSyncError) { + assert!(matches!(closed, Err(LogError::Io(_)))); + } else { + closed?; + } + drop(log); + + // Cold reopen resolves the on-disk outcome independently of the failed + // caller and old cache. A pre-sync error can leave either complete root; + // an error after successful sync must retain the complete authenticated seed. + let log = fixture + .cluster + .domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + base.join("log"), + local.node, + local.disk, + ProcessGeneration::new([94; 16])?, + LogOpenMode::Resume, + ), + Arc::new(DirectFileSystem), + )) + .await??; + let group = log.cell_group(&route.history)?; + let selected = group.group_manifest()?.context("reopened root")?; + match cut { + Cut::StreamingRevocation | Cut::QueuedRevocation => assert_eq!(selected, staged), + Cut::EnteredRevocation | Cut::EnteredExpiry | Cut::AfterSyncError => { + assert_eq!(selected, expected) + } + Cut::SyncError => assert!(selected == staged || selected == expected), + } + let (pages, images) = seed_install::stores(&base, rooted) + .context("reopen destination stores after joined close")?; + let recovering = group.clone(); + let (applier, recovered) = fixture + .cluster + .domain + .blocking(move || { + CellGroupApplier::recover( + recovering, + pages, + images, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + ) + })? + .await??; + assert_eq!(recovered.marker(), &selected); + let permit = fixture + .cluster + .domain + .run(fixture.cluster.directory.bootstrap_joined_group_member( + &fixture.cluster.birth, + local.member, + &log, + )) + .await?; + if selected == expected { + assert_eq!(group.checkpoint_manifest()?.as_ref(), Some(&expected)); + assert!(recovered.pin_cell(&a.cell).is_some() && recovered.pin_cell(&b.cell).is_some()); + assert_eq!(group.initial_state()?.conf_state.learners, vec![4]); + let (runtime, events) = CellGroupRuntime::start( + log.clone(), + RaftRuntimeOptions { + tick_interval_ms: None, + ..Default::default() + }, + ) + .await?; + runtime.attach(permit?, recovered.clone()).await?; + assert!( + !runtime.status(route.history).await?.ready, + "seed receipt is not quorum authority" + ); + let bytes = images::image(None, "CREATE TABLE value(n);")?; + let proposal = + apply::enrollment_bytes(&route, &history(&route, "without-quorum")?, &bytes, rooted)?; + assert!(matches!( + runtime.propose(route.history, proposal).await, + Err(RuntimeError::NotReady) + )); + assert_eq!(group.group_manifest()?.as_ref(), Some(&expected)); + runtime.close().await?; + drop(events); + } else { + assert!(permit.is_err()); + assert!(group.checkpoint_manifest()?.is_none()); + assert_eq!(recovered.marker().root().cells(), 0); + } + eprintln!("verified seed persistence rooted={rooted} cut={cut:?} recovered_cells={} recovered_index={}", selected.root().cells(), selected.position().index.get()); + drop((applier, recovered, group)); + log.close().await?; + if let Some(source) = restarted_source { + source.close().await?; + } + for controller in controllers { + controller.close().await?; + } + fixture.close().await +} diff --git a/crates/cellar/tests/support/cell_group_seed_probes.rs b/crates/cellar/tests/support/cell_group_seed_probes.rs new file mode 100644 index 000000000..af441a16c --- /dev/null +++ b/crates/cellar/tests/support/cell_group_seed_probes.rs @@ -0,0 +1,147 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Independent authenticated seed requests against the real source listener. +use super::*; +use network_probes::{peer, Connection}; +use serde_json::{json, Value}; + +pub(super) async fn source_protocol( + fixture: &RuntimeCluster, + route: &cellar::application::consensus::verification::GroupRoute, + pool: &CellGroupCheckpointPool, + registration: &mut CellGroupCheckpointRegistration, +) -> Result<()> { + let source = &fixture.transports[0]; + let before = fixture.cluster.voters[0] + .log + .cell_group(&route.scope)? + .group_manifest()?; + let target = route.peers.last().context("joining peer")?; + assert_eq!(target.replica.member.get(), 4); + let history = route.scope; + let request = json!({"route": { + "scope": {"kind":"Group", "identity":[history.cluster.bytes(), history.namespace.bytes(), history.group.bytes(), history.incarnation.bytes()]}, + "schema":route.schema_digest, "source":peer(target), "target":peer(&route.peers[0]) + }, "nonce":"1", "timeout_ms":10000, "from_s3":false}); + for case in 0..13 { + let mut connection = + Connection::open_with_exporter(source.data_address(), b"EXPORTER-cellar-group-seed-1") + .await?; + let mut invalid = request.clone(); + match case { + 0 => connection.exporter[0] ^= 1, + 1 => invalid["route"]["source"]["process"] = json!(vec![244; 16]), + 2 => invalid["route"]["source"]["disk"] = json!(vec![244; 16]), + 3 => invalid["route"]["scope"]["identity"][3] = json!(vec![244; 16]), + 4 => invalid["route"]["schema"] = json!(vec![244; 32]), + 5 => invalid["nonce"] = json!("0"), + 6 => invalid["nonce"] = json!("01"), + 7 => invalid["timeout_ms"] = json!(0), + 8 => invalid["timeout_ms"] = json!(3_600_001), + 9 => invalid["extra"] = json!(true), + // A current voter cannot request a fresh learner seed. + 10 => invalid["route"]["source"] = peer(&route.peers[1]), + 11 => { + invalid.as_object_mut().unwrap().remove("from_s3"); + } + 12 => invalid["from_s3"] = json!("archive"), + _ => unreachable!(), + } + connection + .frame(b"CLGJ", b"cellar-group-seed-request-1", &[], invalid, &[]) + .await?; + connection + .refused() + .await + .with_context(|| format!("seed request fault {case}"))?; + released(source, pool).await?; + } + let mut previous = None; + for case in 0..5 { + let mut connection = + Connection::open_with_exporter(source.data_address(), b"EXPORTER-cellar-group-seed-1") + .await?; + let tag = connection + .frame( + b"CLGJ", + b"cellar-group-seed-request-1", + &[], + request.clone(), + &[], + ) + .await?; + let (reply, body, binding) = connection + .receive(b"CLGK", b"cellar-group-seed-grant-1", &tag) + .await?; + assert_eq!(reply["request"], json!(tag)); + assert_eq!(reply["nonce"], request["nonce"]); + assert_eq!(reply["data"], json!({"kind":"peer"})); + assert_eq!(reply["route"]["source"], request["route"]["target"]); + assert_eq!(reply["route"]["target"], request["route"]["source"]); + assert_eq!( + reply["digest"], + json!(<[u8; 32]>::from(Sha256::digest(&body))) + ); + assert_eq!( + pool.usage().sources, + 1, + "complete source remains pinned awaiting Ready" + ); + let mut ready = json!({"sequence":0}); + let bound = match case { + 0 => { + ready["sequence"] = json!(1); + binding + } + 1 => previous.context("previous authenticated reply")?, + 2 => { + ready["extra"] = Value::Bool(true); + binding + } + // The captured source registration is revoked while the peer has + // its signed descriptor. Its Ready must not revive the old owner. + 3 => { + source.remove_physical_group(&history)?; + binding + } + 4 => { + drop(connection); + released(source, pool).await?; + continue; + } + _ => unreachable!(), + }; + connection + .frame(b"CLGL", b"cellar-group-seed-ready-1", &bound, ready, &[]) + .await?; + connection + .refused() + .await + .with_context(|| format!("seed Ready fault {case}"))?; + released(source, pool).await?; + if case == 3 { + source.install_physical_group(route.clone())?; + *registration = + source.register_group_checkpoint(fixture.appliers[&(0, history)].1.clone())?; + } + previous = Some(binding); + } + assert_eq!( + fixture.cluster.voters[0] + .log + .cell_group(&history)? + .group_manifest()?, + before + ); + Ok(()) +} + +async fn released(transport: &RaftTransport, pool: &CellGroupCheckpointPool) -> Result<()> { + tokio::time::timeout(Duration::from_secs(5), async { + while pool.usage().sources != 0 || transport.usage().data_items != 0 { + tokio::task::yield_now().await; + } + }) + .await?; + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_seed_receiver.rs b/crates/cellar/tests/support/cell_group_seed_receiver.rs new file mode 100644 index 000000000..52fec0d2e --- /dev/null +++ b/crates/cellar/tests/support/cell_group_seed_receiver.rs @@ -0,0 +1,607 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! An independent authenticated peer sends adversarial replies to the production receiver. +use super::super::network_probes::Connection; +use super::*; +use cellar::application::consensus::verification::JoiningMember; +use protobuf::Message as _; +use raft::prelude::{ConfChangeTransition, ConfChangeType}; +use serde_json::{json, Value}; +use tokio::net::TcpListener; + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(super) enum Fault { + Exporter, + Correlation, + Nonce, + Process, + Digest, + Barrier, + Decimal, + ExtraReply, + MissingData, + UnexpectedArchive, + Descriptor, + ReplyLimit, + Binding, + Sequence, + ChunkDigest, + ExtraChunk, + Offset, + EarlyEnd, + ChunkLimit, + Disconnect, + Replay, + EndOffset, + EndBody, + Valid, + Revoked, +} + +impl Fault { + fn reply(self) -> bool { + matches!( + self, + Self::Exporter + | Self::Correlation + | Self::Nonce + | Self::Process + | Self::Digest + | Self::Barrier + | Self::Decimal + | Self::ExtraReply + | Self::MissingData + | Self::UnexpectedArchive + | Self::Descriptor + | Self::ReplyLimit + ) + } +} + +fn expected_error(fault: Fault, error: &CellGroupSeedError) -> bool { + use CellGroupSeedError::Transport; + use TransportError::*; + matches!( + (fault, error), + ( + Fault::Exporter + | Fault::Correlation + | Fault::Nonce + | Fault::Process + | Fault::Digest + | Fault::Binding + | Fault::UnexpectedArchive + | Fault::ChunkDigest, + Transport(Authentication) + ) | ( + Fault::Decimal + | Fault::MissingData + | Fault::ExtraReply + | Fault::Descriptor + | Fault::ExtraChunk + | Fault::EndOffset + | Fault::EndBody, + Transport(Protocol) + ) | (Fault::ReplyLimit | Fault::ChunkLimit, Transport(Limit)) + | (Fault::Sequence | Fault::Replay, Transport(Replay)) + | (Fault::Offset | Fault::EarlyEnd, Transport(Invalid(_))) + | (Fault::Disconnect, Transport(Io(_) | Closed)) + | (Fault::Barrier, CellGroupSeedError::Checkpoint(_)) + | (Fault::Revoked, Transport(StalePeer | Closed)) + ) +} + +struct Part { + header: Value, + bytes: Vec, +} + +pub(super) struct Capture { + body: Vec, + barrier: u64, + parts: Vec, +} + +pub(super) async fn capture( + fixture: &RuntimeCluster, + history: GroupHistory, +) -> Result> { + let authority = fixture.runtimes[0] + .seed_authority( + seed_install::target(history)?, + reads::deadline(fixture, 10_000).await?, + ) + .await?; + let barrier = authority.barrier().index.get(); + fixture.cluster.domain.blocking(move || { + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let mut source = authority.capture(&pool)?; + let body = source.snapshot()?.write_to_bytes()?; + let mut parts = Vec::new(); + let mut retained = 0; + // Only this small adversarial test peer buffers a captured cut. Production + // transfer remains streamed; bound this fixture before retaining bytes. + source.stream(|chunk| { + retained += chunk.bytes.len(); + assert!(retained < 2 * 1024 * 1024); + let part = match chunk.part { + CellGroupCheckpointPart::Header => json!({"kind":"Header"}), + CellGroupCheckpointPart::Page(digest) => json!({"kind":"Page", "digest":digest}), + CellGroupCheckpointPart::Materialized(cell) => json!({"kind":"Materialized", "cell":cell.as_str()}), + CellGroupCheckpointPart::ExtentManifest(cell) => json!({"kind":"ExtentManifest", "cell":cell.as_str()}), + CellGroupCheckpointPart::Extent { cell, ordinal, id } => json!({"kind":"Extent", "cell":cell.as_str(), "ordinal":ordinal, "digest":id.bytes()}), + }; + parts.push(Part { header: json!({"sequence":parts.len(), "part":part, + "offset":chunk.offset, "length":chunk.length, "digest":<[u8;32]>::from(Sha256::digest(chunk.bytes))}), bytes:chunk.bytes.to_vec() }); + Ok(()) + })?; + Ok(Arc::new(Capture { body, barrier, parts })) + })?.await? +} + +fn acceptor() -> Result { + let mut config = rustls::ServerConfig::builder_with_provider(Arc::new( + rustls::crypto::ring::default_provider(), + )) + .with_protocol_versions(&[&rustls::version::TLS13])? + .with_no_client_auth() + .with_single_cert( + vec![include_bytes!("../../transport/tcp/fixtures/server.der") + .to_vec() + .into()], + rustls::pki_types::PrivatePkcs8KeyDer::from( + include_bytes!("../../transport/tcp/fixtures/key.der").to_vec(), + ) + .into(), + )?; + config.alpn_protocols = vec![b"cellar-raft-1".to_vec()]; + Ok(tokio_rustls::TlsAcceptor::from(Arc::new(config))) +} + +pub(super) async fn serve( + listener: Arc, + cut: Arc, + fault: Fault, +) -> Result<()> { + tokio::time::timeout(Duration::from_secs(10), async { + let (tcp, _) = listener.accept().await?; + let stream = acceptor()?.accept(tcp).await?; + let exporter = stream.get_ref().1.export_keying_material( + [0; 32], + b"EXPORTER-cellar-group-seed-1", + None, + )?; + let mut wire = Connection::from_stream(stream, exporter); + let (request, _, request_tag) = wire + .receive(b"CLGJ", b"cellar-group-seed-request-1", &[]) + .await?; + let mut route = request["route"].clone(); + route["source"] = request["route"]["target"].clone(); + route["target"] = request["route"]["source"].clone(); + let mut body = cut.body.clone(); + let mut reply = json!({"route":route, "request":request_tag, "nonce":request["nonce"], + "barrier":cut.barrier.to_string(), "digest":<[u8;32]>::from(Sha256::digest(&body)), "data":{"kind":"peer"}}); + match fault { + Fault::Exporter => wire.exporter[0] ^= 1, + Fault::Correlation => reply["request"] = json!(vec![0; 32]), + Fault::Nonce => reply["nonce"] = json!("999999"), + Fault::Process => reply["route"]["source"]["process"] = json!(vec![244; 16]), + Fault::Digest => reply["digest"] = json!(vec![0; 32]), + Fault::Barrier => reply["barrier"] = json!(u64::MAX.to_string()), + Fault::Decimal => reply["barrier"] = json!("01"), + Fault::ExtraReply => reply["extra"] = json!(true), + Fault::MissingData => { reply.as_object_mut().unwrap().remove("data"); }, + Fault::UnexpectedArchive => reply["data"] = json!({"kind":"archive", "descriptor":{ + "ticket":{"registry":vec![1;32],"generation":2,"allocation":vec![2;32]}, + "checkpoint":{"registry":vec![1;32],"generation":2,"digest":vec![3;32],"length":1024} + }}), + Fault::Descriptor => { + body = vec![255]; + reply["digest"] = json!(<[u8; 32]>::from(Sha256::digest(&body))); + } + Fault::ReplyLimit => { + oversized(&mut wire, b"CLGK").await?; + return wire.refused().await; + } + _ => (), + } + let binding = wire + .frame( + b"CLGK", + b"cellar-group-seed-grant-1", + &request_tag, + reply, + &body, + ) + .await?; + if fault.reply() || fault == Fault::Revoked { + return wire.refused().await; + } + let (ready, body, _) = wire + .receive(b"CLGL", b"cellar-group-seed-ready-1", &binding) + .await?; + assert_eq!(ready, json!({"sequence":0})); + assert!(body.is_empty()); + if fault == Fault::ChunkLimit { + oversized(&mut wire, b"CLGM").await?; + return wire.refused().await; + } + if fault == Fault::EarlyEnd { + wire.frame(b"CLGM", b"cellar-group-seed-chunk-1", &binding, end(0), &[]) + .await?; + return wire.refused().await; + } + for (i, part) in cut.parts.iter().enumerate() { + let mut header = part.header.clone(); + let mut bound = binding; + if i == 0 { + match fault { + Fault::Binding => bound[0] ^= 1, + Fault::Sequence => header["sequence"] = json!(1), + Fault::ChunkDigest => header["digest"] = json!(vec![0; 32]), + Fault::ExtraChunk => header["extra"] = json!(true), + Fault::Offset => header["offset"] = json!(1), + _ => (), + } + } + wire.frame( + b"CLGM", + b"cellar-group-seed-chunk-1", + &bound, + header, + &part.bytes, + ) + .await?; + if i == 0 && fault == Fault::Disconnect { + return Ok(()); + } + if i == 0 + && matches!( + fault, + Fault::Binding + | Fault::Sequence + | Fault::ChunkDigest + | Fault::ExtraChunk + | Fault::Offset + | Fault::Replay + ) + { + if fault == Fault::Replay { + wire.frame( + b"CLGM", + b"cellar-group-seed-chunk-1", + &binding, + part.header.clone(), + &part.bytes, + ) + .await?; + } + return wire.refused().await; + } + } + let mut header = end(cut.parts.len()); + let body = if fault == Fault::EndBody { + vec![1] + } else { + vec![] + }; + if fault == Fault::EndOffset { + header["offset"] = json!(1); + } + header["digest"] = json!(<[u8; 32]>::from(Sha256::digest(&body))); + wire.frame( + b"CLGM", + b"cellar-group-seed-chunk-1", + &binding, + header, + &body, + ) + .await?; + wire.refused().await // The protocol has no install acknowledgement on this stream. + }) + .await? +} + +fn end(sequence: usize) -> Value { + json!({"sequence":sequence, "part":null, "offset":0, "length":0, "digest":<[u8;32]>::from(Sha256::digest([]))}) +} + +async fn oversized( + wire: &mut Connection, + magic: &[u8; 4], +) -> Result<()> { + let mut prefix = [0; 20]; + prefix[..4].copy_from_slice(magic); + prefix[4..8].copy_from_slice(&1u32.to_be_bytes()); + prefix[8..12].copy_from_slice(&1u32.to_be_bytes()); + prefix[12..].copy_from_slice(&65537u64.to_be_bytes()); + wire.raw(&prefix).await +} + +pub(crate) async fn seed_receiver_faults(rooted: bool) -> Result<()> { + let class = RaftClass::new("Counter", 1, [81; 32])?; + let (fixture, route, _, _) = Box::pin(registered::setup(rooted, &class)).await?; + Box::pin(run(fixture, route, rooted)).await +} + +async fn run(mut fixture: RuntimeCluster, route: GroupRoute, rooted: bool) -> Result<()> { + let mut controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let local = seed_install::target(route.history)?; + membership::configure( + &fixture, + &route, + membership::change( + 4, + ConfChangeType::AddLearnerNode, + ConfChangeTransition::Auto, + ), + &[JoiningMember { + member: local.member, + node: local.node, + disk: local.disk, + }], + ) + .await?; + let cut = capture(&fixture, route.history).await?; + let listener = Arc::new(TcpListener::bind("127.0.0.1:0").await?); + let mut peer = fixture.tcp_route(route.history)?.peers[0].clone(); + peer.data_address = listener.local_addr()?; + let base = fixture.cluster._root.path().join("seed-receiver"); + let log = fixture + .cluster + .domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + base.join("log"), + local.node, + local.disk, + local.process, + LogOpenMode::Create, + ), + Arc::new(DirectFileSystem), + )) + .await??; + let (runtime, events) = CellGroupRuntime::start( + log.clone(), + RaftRuntimeOptions { + tick_interval_ms: None, + ..Default::default() + }, + ) + .await?; + let transport = tcp::start(&runtime, 3).await?; + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + transport.enable_group_checkpoints(pool.clone())?; + let baseline = transport.usage().tasks; + let mut applier: Option = None; + for fault in [ + Fault::Exporter, + Fault::Correlation, + Fault::Nonce, + Fault::Process, + Fault::Digest, + Fault::Barrier, + Fault::Decimal, + Fault::ExtraReply, + Fault::MissingData, + Fault::UnexpectedArchive, + Fault::Descriptor, + Fault::ReplyLimit, + Fault::Binding, + Fault::Sequence, + Fault::ChunkDigest, + Fault::ExtraChunk, + Fault::Offset, + Fault::EarlyEnd, + Fault::ChunkLimit, + Fault::Disconnect, + Fault::Replay, + Fault::EndOffset, + Fault::EndBody, + Fault::Valid, + ] { + let service = fixture + .cluster + .domain + .spawn(serve(listener.clone(), cut.clone(), fault))?; + let prepared = Box::pin(transport.prepare_group_seed( + local.clone(), + peer.clone(), + fixture.cluster.directory.clone(), + fixture.cluster.birth.clone(), + reads::deadline(&fixture, 5000).await?, + )) + .await; + if fault.reply() { + let error = prepared + .err() + .with_context(|| format!("{fault:?} accepted descriptor"))?; + assert!(expected_error(fault, &error), "{fault:?}: {error:?}"); + assert!( + matches!(log.cell_group(&route.history), Err(LogError::NotFound)), + "{fault:?} created a stage" + ); + } else { + let session = prepared.with_context(|| format!("prepare {fault:?}"))?; + if applier.is_none() { + let group = session.group().clone(); + let (pages, images) = seed_install::stores(&base, rooted)?; + let (owner, root) = fixture + .cluster + .domain + .blocking(move || { + CellGroupApplier::recover( + group, + pages, + images, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + ) + })? + .await??; + assert_eq!(root.marker().root().cells(), 0); + applier = Some(owner); + } + let result = session.install(applier.as_ref().unwrap().clone()).await; + if fault == Fault::Valid { + let (_permit, root) = result?; + assert_eq!(root.marker().root().cells(), 2); + } else { + let error = result + .err() + .with_context(|| format!("{fault:?} installed"))?; + assert!(expected_error(fault, &error), "{fault:?}: {error:?}"); + let group = log.cell_group(&route.history)?; + assert_eq!(group.group_manifest()?.context("stage")?.root().cells(), 0); + assert!(group.checkpoint_manifest()?.is_none()); + assert!(fixture + .cluster + .domain + .run(fixture.cluster.directory.bootstrap_joined_group_member( + &fixture.cluster.birth, + local.member, + &log + )) + .await? + .is_err()); + } + } + service + .await? + .with_context(|| format!("independent peer {fault:?}"))?; + controller::wait_for(async || { + Ok(transport.usage().tasks == baseline && pool.usage().sources == 0) + }) + .await + .with_context(|| format!("release {fault:?}"))?; + assert_eq!(transport.usage().data_items, 0); + } + // Capture a later valid cut, then replace the source process while its signed + // grant is admitted at the destination but before installation begins. + let bytes = images::image(None, "CREATE TABLE value(n);")?; + let proposal = fixture.runtimes[0] + .propose( + route.history, + apply::enrollment_bytes( + &route, + &history(&route, "revoked-later-cut")?, + &bytes, + rooted, + )?, + ) + .await?; + controller::applied(&fixture, route.history, proposal.position.index).await?; + let later = capture(&fixture, route.history).await?; + let before = log.cell_group(&route.history)?.group_manifest()?; + let checkpoint = log.cell_group(&route.history)?.checkpoint_manifest()?; + let service = fixture + .cluster + .domain + .spawn(serve(listener, later, Fault::Revoked))?; + let session = Box::pin(transport.prepare_group_seed( + local.clone(), + peer.clone(), + fixture.cluster.directory.clone(), + fixture.cluster.birth.clone(), + reads::deadline(&fixture, 5000).await?, + )) + .await?; + let restarted = revoke_source(&fixture, &mut controllers, &transport, &peer, &route).await?; + let error = session + .install(applier.clone().unwrap()) + .await + .err() + .context("revoked grant installed")?; + assert!( + expected_error(Fault::Revoked, &error), + "revocation: {error:?}" + ); + service.await??; + controller::wait_for(async || { + Ok(transport.usage().tasks == baseline && pool.usage().sources == 0) + }) + .await?; + assert_eq!(log.cell_group(&route.history)?.group_manifest()?, before); + assert_eq!( + log.cell_group(&route.history)?.checkpoint_manifest()?, + checkpoint + ); + assert!(matches!( + Box::pin(transport.prepare_group_seed( + local, + peer, + fixture.cluster.directory.clone(), + fixture.cluster.birth.clone(), + reads::deadline(&fixture, 1000).await? + )) + .await, + Err(CellGroupSeedError::Transport(TransportError::StalePeer)) + )); + restarted.close().await?; + transport.close().await; + runtime.close().await?; + drop((events, applier)); + log.close().await?; + for controller in controllers { + controller.close().await?; + } + fixture.close().await +} + +pub(super) async fn revoke_source( + fixture: &RuntimeCluster, + controllers: &mut Vec, + transport: &RaftTransport, + peer: &RaftPeer, + route: &GroupRoute, +) -> Result { + let namespace = fixture.cluster.directory.namespace_config(); + anyhow::ensure!( + route.history.cluster == namespace.cluster + && route.history.namespace == namespace.namespace + && route.schema_digest == namespace.schema_digest + && peer.replica.scope == route.history, + "source revocation route is outside the fixture namespace", + ); + controllers.remove(0).close().await?; + let new_process = ProcessGeneration::new([99; 16])?; + let options = ReplicaLogOptions::new( + fixture.cluster._root.path().join("node-1/log"), + peer.replica.node, + peer.replica.disk, + new_process, + LogOpenMode::Resume, + ); + fixture.cluster.voters[0].log.close().await?; + let restarted = fixture + .cluster + .domain + .run(ReplicaLog::open(options, Arc::new(DirectFileSystem))) + .await??; + let trusted = RaftPeerNode::from(peer); + let mut latest = trusted.clone(); + latest.process = new_process; + let view = fixture + .cluster + .domain + .run(async { + let catalog = Catalog::new(fixture.cluster.store.clone(), CatalogOptions::default())?; + let namespace = catalog + .existing_namespace(fixture.cluster.directory.namespace_config().clone()) + .await?; + let registration = namespace + .register_node_storage(&restarted, Duration::from_secs(5)) + .await?; + Ok::<_, anyhow::Error>( + registration + .publication(latest, Duration::from_secs(5)) + .await? + .publish(Duration::from_secs(5)) + .await?, + ) + }) + .await??; + transport.accept_node_view(view, &trusted)?; + Ok(restarted) +} diff --git a/crates/cellar/tests/support/cell_group_seed_retry.rs b/crates/cellar/tests/support/cell_group_seed_retry.rs new file mode 100644 index 000000000..1d9b9bec0 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_seed_retry.rs @@ -0,0 +1,450 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Resume the original voter disks and retry an owned, killed learner's disk. +use super::*; +use cellar_ltx::internal::NodeExtentStore; +use std::collections::VecDeque; + +pub(crate) async fn seed_retry(rooted: bool) -> Result<()> { + for cut in ["retry-prefix", "retry-synced"] { + let directory = tempfile::tempdir()?; + crash_fixture::kill_at(directory.path(), "seed_process_fixture", rooted, cut).await?; + Box::pin(retry(directory.path(), rooted, cut)) + .await + .with_context(|| format!("cold retry {rooted}/{cut}"))?; + } + Ok(()) +} + +async fn restore_source( + root: &Path, + oracle: &Oracle, + domain: ExecutionDomain, + directory: RangeDirectory, + birth: CellGroupBirth, + store: Arc, +) -> Result { + anyhow::ensure!(oracle.source_root.starts_with(root), "owned source disks"); + let mut voters = Vec::new(); + let mut appliers = Vec::new(); + let options = RaftRuntimeOptions::default(); + for n in 0..3 { + let base = oracle.source_root.join(format!("node-{}", n + 1)); + let log = domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + base.join("log"), + NodeId::new([n as u8 + 1; 16])?, + DiskLineage::new([n as u8 + 11; 16])?, + ProcessGeneration::new([95; 16])?, + LogOpenMode::Resume, + ), + Arc::new(DirectFileSystem), + )) + .await??; + let images = GenerationStore::open( + base.join("images"), + Arc::new(DirectFileSystem), + Default::default(), + )?; + let extents = if oracle.rooted { + Some(NodeExtentStore::open( + oracle.source_root.join(format!("rooted-{n}")), + Arc::new(DirectFileSystem), + Default::default(), + )?) + } else { + None + }; + let pages = ManifestPageStore::open( + &base.join("pages"), + Arc::new(DirectFileSystem), + 1024, + 16 * 1024 * 1024, + 16, + )?; + let config = + super::super::super::checkpoint_receive::config(images.clone(), extents.clone(), root)?; + let group = log.cell_group(&oracle.expected.route().history)?; + let (applier, prepared) = domain + .blocking(move || { + CellGroupApplier::recover( + group, + pages, + config, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + ) + })? + .await??; + let permit = domain + .run(directory.bootstrap_group_member(&birth, &log)) + .await??; + let owner = domain + .run(CellGroupOwner::start( + permit, + prepared, + options.clone(), + [71; 32], + )) + .await??; + appliers.push(applier); + voters.push(Voter { + log, + owner: Some(owner), + pages: None, + images, + extents, + tree: ManifestTree::new(birth.route().clone(), ManifestLimits::default())?, + gate: fault::Gate::new(None), + paused: false, + held: None, + }); + } + let cluster = Cluster { + _root: tempfile::tempdir_in(root)?, + domain, + directory, + store, + birth, + voters, + network: VecDeque::new(), + reads: vec![], + options, + }; + let mut f = Box::pin(RuntimeCluster::from_cluster(cluster, 256, 256, 1_000_000)).await?; + for (n, applier) in appliers.into_iter().enumerate() { + let pool = apply::pool(&f, n)?; + f.appliers.insert((n, applier.history()), (pool, applier)); + } + f.connect_tcp().await.context("connect cold source peers")?; + Ok(f) +} + +async fn retry(root: &Path, rooted: bool, cut: &str) -> Result<()> { + let oracle: Oracle = serde_json::from_slice(&std::fs::read(root.join("oracle.json"))?)?; + assert_eq!(oracle.rooted, rooted); + assert_eq!(oracle.cut, cut); + let (partial, length): (std::path::PathBuf, u64) = + serde_json::from_slice(&std::fs::read(root.join("dependency.json"))?)?; + assert!(length > 0); + if rooted { + assert_eq!( + partial.parent(), + Some(root.join("learner/extents").as_path()) + ); + assert_eq!(std::fs::metadata(&partial)?.len(), length); + let text = partial + .file_name() + .context("extent")? + .to_str() + .context("name")?; + let full = u64::from_str_radix(&text[2..10], 16)?; + if cut == "retry-prefix" { + assert!(length < full); + } else { + assert_eq!(length, full); + } + } + let route = oracle.expected.route().clone(); + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let (directory, birth, store) = crash_fixture::restore_catalog_store( + oracle.objects.clone(), + &route, + &domain, + oracle.image_storage, + ) + .await?; + let mut f = Box::pin(restore_source( + root, + &oracle, + domain, + directory.clone(), + birth.clone(), + store, + )) + .await + .context("restore cold source quorum")?; + let controllers = controller::start_with_limits(&mut f, controller::limits()) + .await + .context("start cold source controllers")?; + f.runtimes[0] + .campaign(route.history) + .await + .context("campaign cold source")?; + controller::wait_for(async || Ok(f.runtimes[0].status(route.history).await?.ready)).await?; + assert!(f.runtimes[0].status(route.history).await?.term > oracle.expected.position().term); + let later = history(&route, "after-crash-before-retry")?; + let bytes = images::image( + None, + "CREATE TABLE value(n); INSERT INTO value VALUES(317);", + )?; + let accepted = f.runtimes[0] + .propose( + route.history, + apply::enrollment_bytes(&route, &later, &bytes, rooted)?, + ) + .await?; + controller::applied(&f, route.history, accepted.position.index).await?; + let expected = f.cluster.voters[0] + .log + .cell_group(&route.history)? + .group_manifest()? + .context("fresh source cut")?; + assert_eq!(expected.root().cells(), 3); + let base = root.join("learner"); + let mut local = seed_install::target(route.history)?; + local.process = ProcessGeneration::new([94; 16])?; + let log = f + .cluster + .domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + base.join("log"), + local.node, + local.disk, + local.process, + LogOpenMode::Resume, + ), + Arc::new(DirectFileSystem), + )) + .await??; + let engine = log.storage_instance()?; + let group = log.cell_group(&route.history)?; + assert_eq!(group.group_manifest()?.as_ref(), Some(&oracle.staged)); + let (pages, images) = seed_install::stores(&base, rooted)?; + let recovering = group.clone(); + let (applier, empty) = f + .cluster + .domain + .blocking(move || { + CellGroupApplier::recover( + recovering, + pages, + images, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + ) + })? + .await??; + assert_eq!(empty.pinned_images(), 0); + drop(empty); + assert!(f + .cluster + .domain + .run(directory.bootstrap_joined_group_member(&birth, local.member, &log)) + .await? + .is_err()); + let mut deleted = 0; + for _ in 0..16 { + let report = log + .reclaim_cell_storage( + vec![applier.clone()], + vec![], + CellGroupReclaimLimits { + max_selected_cells: 64, + max_files: 1, + max_page_scan: 64, + }, + ) + .await?; + assert!(report.image_files + report.page_files <= 1); + deleted += report.image_files; + if report.image_files + report.page_files == 0 && (!rooted || !partial.exists()) { + break; + } + } + if rooted { + assert!(deleted > 0); + assert!(!partial.exists()); + } + assert_eq!(group.group_manifest()?.as_ref(), Some(&oracle.staged)); + assert!(f + .cluster + .domain + .run(directory.bootstrap_joined_group_member(&birth, local.member, &log)) + .await? + .is_err()); + let (runtime, events) = CellGroupRuntime::start( + log.clone(), + RaftRuntimeOptions { + tick_interval_ms: None, + ..Default::default() + }, + ) + .await?; + let transport = tcp::start(&runtime, 3).await?; + transport + .enable_group_checkpoints(CellGroupCheckpointPool::new(checkpoint_fixture::limits())?)?; + let source_pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + f.transports[0].enable_group_checkpoints(source_pool)?; + let mut physical = f.tcp_route(route.history)?; + physical.revision += 1; + physical.peers.push(RaftPeer { + replica: local.clone(), + label: "node-4".into(), + server_name: "localhost".into(), + control_address: transport.control_address(), + data_address: transport.data_address(), + query_address: transport.query_address(), + }); + for source in &f.transports { + source + .install_physical_group(physical.clone()) + .context("install fresh learner route at source")?; + } + assert!( + matches!( + transport.install_physical_group(physical.clone()), + Err(TransportError::StalePeer) + ), + "an empty stage cannot authorize ordinary group routing" + ); + let _registration = + f.transports[0].register_group_checkpoint(f.appliers[&(0, route.history)].1.clone())?; + + let session = Box::pin(transport.prepare_group_seed( + local.clone(), + physical.peers[0].clone(), + directory.clone(), + birth.clone(), + reads::deadline(&f, 10_000).await?, + )) + .await + .context("prepare fresh authenticated seed")?; + let (permit, installed) = session + .install(applier.clone()) + .await + .context("install retried seed")?; + assert_eq!(installed.marker(), &expected); + transport + .install_physical_group(physical.clone()) + .context("route authenticated installed learner")?; + for name in ["registered-a", "registered-b", "after-crash-before-retry"] { + assert!(installed.pin_cell(&CellId::new(name)?).is_some()); + } + assert_eq!(log.storage_instance()?, engine); + runtime.attach(permit, installed.clone()).await?; + let sql = CellGroupSqlPool::start( + &log, + CellSqlLimits { + workers: 1, + queue_per_worker: 1, + max_pending_cells: 1, + group_pending_cells: 1, + node_bytes: 1024 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + }, + )?; + let learner = CellGroupController::start( + runtime.clone(), + events, + transport.clone(), + sql.clone(), + vec![applier.clone()], + controller::limits(), + ) + .await?; + let live = history(&route, "live-after-retry")?; + let written = f.runtimes[0] + .propose( + route.history, + apply::enrollment_bytes(&route, &live, &bytes, rooted)?, + ) + .await?; + controller::applied(&f, route.history, written.position.index).await?; + controller::wait_for(async || { + f.runtimes[0].tick().await?; + Ok(runtime.status(route.history).await?.applied.index >= written.position.index) + }) + .await?; + let expected = f.cluster.voters[0] + .log + .cell_group(&route.history)? + .group_manifest()? + .context("live suffix source")?; + assert_eq!(expected.root().cells(), 4); + assert_eq!(group.group_manifest()?.as_ref(), Some(&expected)); + assert_eq!( + group.checkpoint_manifest()?.as_ref(), + Some(installed.marker()) + ); + learner.close().await?; + drop((learner, sql, installed, transport, runtime, applier, group)); + log.close().await?; + drop(log); + // Reopen the same successful retry independently of every previous owner. + let log = f + .cluster + .domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + base.join("log"), + local.node, + local.disk, + ProcessGeneration::new([96; 16])?, + LogOpenMode::Resume, + ), + Arc::new(DirectFileSystem), + )) + .await??; + assert_eq!(log.storage_instance()?, engine); + let group = log.cell_group(&route.history)?; + let (pages, images) = seed_install::stores(&base, rooted)?; + let recovering = group.clone(); + let (applier, recovered) = f + .cluster + .domain + .blocking(move || { + CellGroupApplier::recover( + recovering, + pages, + images, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + ) + })? + .await??; + assert_eq!(recovered.marker(), &expected); + assert_eq!(recovered.pinned_images(), 4); + for name in [ + "registered-a", + "registered-b", + "after-crash-before-retry", + "live-after-retry", + ] { + let (leaf, _) = recovered + .pin_cell(&CellId::new(name)?) + .context("cold recovered cell")?; + assert_eq!(leaf.version.sequence.get(), 1); + } + let permit = f + .cluster + .domain + .run(directory.bootstrap_joined_group_member(&birth, local.member, &log)) + .await??; + let (runtime, events) = CellGroupRuntime::start( + log.clone(), + RaftRuntimeOptions { + tick_interval_ms: None, + ..Default::default() + }, + ) + .await?; + runtime.attach(permit, recovered.clone()).await?; + assert!( + !runtime.status(route.history).await?.ready, + "local seed receipt is not fresh quorum authority" + ); + runtime.close().await?; + drop((runtime, events, applier, recovered, group)); + log.close().await?; + for c in controllers { + c.close().await?; + } + drop(_registration); + f.close().await +} diff --git a/crates/cellar/tests/support/cell_group_snapshot_owner.rs b/crates/cellar/tests/support/cell_group_snapshot_owner.rs new file mode 100644 index 000000000..db1dcf568 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_snapshot_owner.rs @@ -0,0 +1,379 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Actual RawNode restore/Ready acceptance on a partitioned existing voter. +//! Transfer uses the native streaming API, not yet the snapshot RPC transport. +use super::*; + +async fn prepared( + rooted: bool, +) -> Result<( + Cluster, + CellGroupApplier, + PreparedCellGroupCheckpoint, + CellGroupCheckpointPool, + Message, +)> { + prepared_at(rooted, None).await +} + +async fn prepared_at( + rooted: bool, + index: Option, +) -> Result<( + Cluster, + CellGroupApplier, + PreparedCellGroupCheckpoint, + CellGroupCheckpointPool, + Message, +)> { + let (mut cluster, _) = checkpoint_fixture::populated_partition(rooted).await?; + let route = cluster.birth.route().clone(); + let group = cluster.voters[2].log.cell_group(&route.history)?; + let before = group.group_manifest()?.context("old marker")?; + let source_pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let source_owner = cluster.voters[0].owner.as_ref().unwrap(); + let mut source = source_pool.capture( + &cluster.voters[0].log.cell_group(&route.history)?, + source_owner.manifest(), + )?; + let term = source_owner.status()?.term.get(); + assert!(source.marker().position().index > before.position().index); + let config = checkpoint_receive::config( + cluster.voters[2].images.clone(), + cluster.voters[2].extents.clone(), + cluster._root.path(), + )?; + let applier = CellGroupApplier::new( + group, + cluster.voters[2].pages.take().context("pages")?, + config, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + )?; + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let mut receiver = pool.receive( + applier.clone(), + checkpoint_receive::scratch()?, + cluster._root.path().into(), + )?; + let checkpoint = cluster + .domain + .blocking(move || { + source.stream(|chunk| { + if let Some(index) = + index.filter(|_| matches!(chunk.part, CellGroupCheckpointPart::Header)) + { + assert_eq!(chunk.offset, 0); + assert_eq!(chunk.length, chunk.bytes.len() as u64); + let mut header: serde_json::Value = + serde_json::from_slice(chunk.bytes).unwrap(); + header["root"]["index"] = serde_json::json!(index); + let bytes = serde_json::to_vec(&header).unwrap(); + receiver.append(CellGroupCheckpointChunk { + length: bytes.len() as u64, + bytes: &bytes, + ..chunk + }) + } else { + receiver.append(chunk) + } + })?; + receiver.finish() + })? + .await??; + let mut message = Message::default(); + message.set_msg_type(MessageType::MsgSnapshot); + message.from = 1; + message.to = 3; + message.term = term; + message.set_snapshot(checkpoint.snapshot().clone()); + Ok((cluster, applier, checkpoint, pool, message)) +} + +pub(crate) async fn checkpoint_owner(rooted: bool) -> Result<()> { + let (mut cluster, applier, checkpoint, pool, message) = prepared(rooted).await?; + let route = cluster.birth.route().clone(); + let group = cluster.voters[2].log.cell_group(&route.history)?; + let before = group.group_manifest()?.context("old root")?; + let cut = checkpoint.marker().clone(); + let owner = cluster.voters[2].owner.as_mut().unwrap(); + assert!( + owner.step(message.clone()).is_err(), + "unverified peer ingress must stay closed" + ); + for variant in 0..4 { + let mut bad = message.clone(); + match variant { + 0 => bad.mut_snapshot().data = b"foreign descriptor".to_vec().into(), + 1 => bad.to = 2, + 2 => bad.from = 99, + _ => bad.term = 1, + } + assert!(owner.step_checkpoint(bad, checkpoint.clone()).is_err()); + assert_eq!(group.group_manifest()?.as_ref(), Some(&before)); + assert!(owner.poll()?.is_none(), "refusal must not mutate Raft"); + } + assert_eq!( + owner.step_checkpoint(message.clone(), checkpoint.clone())?, + CellGroupSnapshotAdmission::Restoring + ); + drop(checkpoint); + assert!(matches!(owner.campaign(), Err(RuntimeError::NotReady))); + assert!(matches!(owner.step(message), Err(RuntimeError::NotReady))); + let Some(CellGroupEffect::Persist(work)) = owner.poll()? else { + anyhow::bail!("snapshot Ready persistence"); + }; + let gate = cluster.voters[2].gate.clone(); + let release = gate.arm(fault::HOLD_SYNC); + let mut pending = cluster.domain.spawn(work.run())?; + tokio::time::timeout(Duration::from_secs(5), gate.entered.notified()).await?; + assert_eq!(pool.usage().sources, 1); + assert_eq!(group.group_manifest()?.as_ref(), Some(&before)); + assert_eq!( + cluster.voters[2].owner.as_ref().unwrap().status()?.applied, + before.position() + ); + assert!( + cluster.voters[2].owner.as_mut().unwrap().poll()?.is_none(), + "snapshot acknowledgement escaped sync" + ); + assert!( + tokio::time::timeout(Duration::from_millis(30), &mut pending) + .await + .is_err() + ); + drop(release); + let completion = pending.await??; + assert_eq!(group.group_manifest()?.as_ref(), Some(&cut)); + let owner = cluster.voters[2].owner.as_mut().unwrap(); + assert_eq!( + owner.status()?.applied, + before.position(), + "engine receipt alone cannot advance RawNode" + ); + owner.complete_persist(completion)?; + let mut acknowledged = false; + let mut installed = false; + for _ in 0..20 { + match owner.poll()? { + Some(CellGroupEffect::Message(response)) => { + assert_eq!(response.get_msg_type(), MessageType::MsgAppendResponse); + assert_eq!( + (response.from, response.to, response.index), + (3, 1, cut.position().index.get()) + ); + acknowledged = true; + } + Some(CellGroupEffect::CheckpointInstalled { manifest }) => { + assert_eq!(manifest.as_ref(), &cut); + installed = true; + } + None => break, + _ => anyhow::bail!("unexpected snapshot completion"), + } + } + assert!(acknowledged && installed); + assert_eq!(owner.manifest().marker(), &cut); + assert_eq!(owner.manifest().pinned_images(), 3); + assert_eq!(pool.usage().sources, 0); + assert!(!owner.status()?.ready); + drop(group); + let mut fixture = RuntimeCluster::from_cluster(cluster, 256, 256, 1000).await?; + let sql = CellGroupSqlPool::start( + &fixture.cluster.voters[2].log, + CellSqlLimits { + workers: 1, + queue_per_worker: 2, + max_pending_cells: 2, + group_pending_cells: 2, + node_bytes: 512 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + }, + )?; + fixture.appliers.insert((2, route.history), (sql, applier)); + fixture.runtimes[0].campaign(route.history).await?; + fixture.drive(None).await?; + let later = history(&route, "after-raft-snapshot")?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(17);")?; + let accepted = fixture.runtimes[0] + .propose( + route.history, + sql::apply::enrollment_bytes(&route, &later, &bytes, rooted)?, + ) + .await?; + fixture.drive(Some(accepted.position.index)).await?; + let group = fixture.cluster.voters[2].log.cell_group(&route.history)?; + assert_eq!(group.checkpoint_manifest()?.as_ref(), Some(&cut)); + assert_eq!( + group + .group_manifest()? + .context("later root")? + .root() + .cells(), + 4 + ); + drop(group); + fixture.close().await +} + +pub(crate) async fn checkpoint_shutdown() -> Result<()> { + for rooted in [false, true] { + let (cluster, applier, checkpoint, pool, message) = prepared(rooted).await?; + let route = cluster.birth.route().clone(); + let group = cluster.voters[2].log.cell_group(&route.history)?; + let before = group.group_manifest()?.context("old root")?; + let cut = checkpoint.marker().clone(); + let gate = cluster.voters[2].gate.clone(); + let fixture = RuntimeCluster::from_cluster(cluster, 256, 256, 1000).await?; + let release = gate.arm(fault::HOLD_SYNC); + assert_eq!( + fixture.runtimes[2] + .step_checkpoint(route.history, message, checkpoint) + .await?, + CellGroupSnapshotAdmission::Restoring + ); + tokio::time::timeout(Duration::from_secs(5), gate.entered.notified()).await?; + let runtime = fixture.runtimes[2].clone(); + let mut close = fixture + .cluster + .domain + .spawn(async move { runtime.close().await })?; + assert!(tokio::time::timeout(Duration::from_millis(30), &mut close) + .await + .is_err()); + assert_eq!(pool.usage().sources, 1); + assert_eq!(group.group_manifest()?.as_ref(), Some(&before)); + drop(release); + close.await??; + assert_eq!(group.group_manifest()?.as_ref(), Some(&cut)); + assert_eq!(group.checkpoint_manifest()?.as_ref(), Some(&cut)); + assert_eq!(pool.usage().sources, 0); + drop(applier); + drop(group); + fixture.close().await?; + } + Ok(()) +} + +pub(crate) async fn checkpoint_bounds() -> Result<()> { + for index in [u64::MAX, u64::MAX - 1, u64::MAX - 2] { + let (mut cluster, applier, checkpoint, pool, message) = + prepared_at(false, Some(index)).await?; + let owner = cluster.voters[2].owner.as_mut().unwrap(); + let before = owner.manifest().marker().clone(); + assert_eq!(checkpoint.marker().position().index.get(), index); + assert!(matches!( + owner.step_checkpoint(message, checkpoint), + Err(RuntimeError::Invalid(_)) + )); + assert_eq!(owner.manifest().marker(), &before); + assert!(owner.poll()?.is_none()); + assert_eq!(pool.usage().sources, 0); + drop(applier); + cluster.close().await?; + } + Ok(()) +} + +pub(crate) async fn checkpoint_matching_log() -> Result<()> { + use raft::{GetEntriesContext, Storage}; + for rooted in [false, true] { + let (mut cluster, applier, checkpoint, pool, message) = prepared(rooted).await?; + let route = cluster.birth.route().clone(); + let group = cluster.voters[2].log.cell_group(&route.history)?; + let before = group.group_manifest()?.context("old root")?; + let cut = checkpoint.marker().clone(); + // Store the leader's actual entries, but withhold its commit advance. + // Raft must recognize this log and replay it, not install the image. + let mut append = Message::default(); + append.set_msg_type(MessageType::MsgAppend); + append.from = 1; + append.to = 3; + append.term = message.term; + append.index = before.position().index.get(); + append.log_term = before.position().term.get(); + append.commit = before.position().index.get(); + append.entries = cluster.voters[0] + .log + .cell_group(&route.history)? + .entries( + append.index + 1, + cut.position().index.get() + 1, + None, + GetEntriesContext::empty(false), + )? + .into(); + let owner = cluster.voters[2].owner.as_mut().unwrap(); + owner.step(append)?; + loop { + match owner.poll()? { + Some(CellGroupEffect::Persist(work)) => { + owner.complete_persist(work.run().await?)? + } + Some(CellGroupEffect::Message(_)) => (), + None => break, + _ => anyhow::bail!("uncommitted entries must not apply"), + } + } + assert_eq!( + group.initial_state()?.hard_state.commit, + before.position().index.get() + ); + assert_eq!( + owner.step_checkpoint(message, checkpoint)?, + CellGroupSnapshotAdmission::Ignored + ); + assert_eq!(pool.usage().sources, 0); + let Some(CellGroupEffect::Persist(work)) = owner.poll()? else { + anyhow::bail!("commit must persist before acknowledgement"); + }; + assert!(owner.poll()?.is_none()); + owner.complete_persist(work.run().await?)?; + let Some(CellGroupEffect::Message(response)) = owner.poll()? else { + anyhow::bail!("ignored snapshot response"); + }; + assert_eq!(response.index, cut.position().index.get()); + assert_eq!(owner.manifest().marker(), &before); + assert_eq!(group.checkpoint_manifest()?, None); + assert_eq!( + group.initial_state()?.hard_state.commit, + cut.position().index.get() + ); + // Restart RawNode ownership over the actual durable committed log and + // replay through the production image adapter on the shared workers. + drop(group); + let mut fixture = RuntimeCluster::from_cluster(cluster, 256, 256, 1000).await?; + let sql = CellGroupSqlPool::start( + &fixture.cluster.voters[2].log, + CellSqlLimits { + workers: 1, + queue_per_worker: 2, + max_pending_cells: 2, + group_pending_cells: 2, + node_bytes: 512 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + }, + )?; + fixture.appliers.insert((2, route.history), (sql, applier)); + fixture.runtimes[0].campaign(route.history).await?; + fixture.drive(None).await?; + let group = fixture.cluster.voters[2].log.cell_group(&route.history)?; + assert_eq!(group.checkpoint_manifest()?, None); + assert_eq!( + group + .group_manifest()? + .context("replayed root")? + .root() + .cells(), + 3 + ); + assert_eq!( + group.group_manifest()?.context("replayed root")?.root(), + cut.root() + ); + drop(group); + fixture.close().await?; + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_source_archives.rs b/crates/cellar/tests/support/cell_group_source_archives.rs new file mode 100644 index 000000000..b9bafce01 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_source_archives.rs @@ -0,0 +1,1813 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use cellar_logic::consensus::LogPosition; +use cellar_ltx::{ + internal::NodeExtentStore, + stream_snapshot::{ScratchLimits, ScratchPool}, +}; +use sha2::Digest; +use std::sync::atomic::Ordering; + +#[path = "raft_catalog_fault.rs"] +#[allow(dead_code)] +mod catalog_fault; + +struct Archives { + pool: CellGroupArchivePool, + tails: CellGroupArchiveTailPool, + uploads: CellGroupArchiveUploadPool, + checkpoints: CellGroupCheckpointPool, + registry: CellGroupArchiveRegistry, + rooted: bool, +} + +impl Archives { + async fn new(cluster: &Cluster, rooted: bool) -> Result { + let pool = CellGroupArchivePool::new(Default::default())?; + let registry = cluster + .directory + .prepare_group_archive_binding(&cluster.domain, &pool, &cluster.birth) + .await? + .commit(&cluster.domain) + .await?; + Ok(Self { + pool, + registry, + rooted, + tails: CellGroupArchiveTailPool::new(Default::default())?, + uploads: CellGroupArchiveUploadPool::new(Default::default())?, + checkpoints: CellGroupCheckpointPool::new(checkpoint_fixture::limits())?, + }) + } + + fn receiver( + &self, + cluster: &Cluster, + label: &str, + ) -> Result { + let (store, base) = self.store(cluster, cluster.birth.route().clone(), label)?; + Ok(self.checkpoints.receive_archive(store, scratch()?, base)?) + } + + fn store( + &self, + cluster: &Cluster, + route: GroupRoute, + label: &str, + ) -> Result<(CellGroupArchiveStore, std::path::PathBuf)> { + let base = cluster._root.path().join(label); + let filesystem = Arc::new(DirectFileSystem); + let pages = ManifestPageStore::open( + &base.join("pages"), + filesystem.clone(), + 1024, + 16 * 1024 * 1024, + 16, + )?; + let images = + GenerationStore::open(base.join("images"), filesystem.clone(), Default::default())?; + let extents = self + .rooted + .then(|| NodeExtentStore::open(base.join("extents"), filesystem, Default::default())) + .transpose()?; + let store = CellGroupArchiveStore::new( + route, + pages, + checkpoint_receive::config(images, extents, &base)?, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + )?; + Ok((store, base)) + } + + async fn publish(&self, fixture: &RuntimeCluster, label: &str) -> Result { + self.publish_group( + fixture, + &self.registry, + fixture.cluster.birth.route(), + label, + ) + .await + } + + async fn publish_group( + &self, + fixture: &RuntimeCluster, + registry: &CellGroupArchiveRegistry, + route: &GroupRoute, + label: &str, + ) -> Result { + let cluster = &fixture.cluster; + let intent = registry.prepare_upload(&cluster.domain).await?; + let ticket = registry.reserve(&cluster.domain, &intent).await?; + let upload = registry + .open_upload(&cluster.domain, &self.uploads, &ticket, &self.tails) + .await?; + let cut = fixture.runtimes[0] + .capture_checkpoint(route.history, &self.checkpoints) + .await?; + let uploaded = upload.upload_cut(&cluster.domain, &self.tails, cut).await?; + let (store, base) = self.store(cluster, route.clone(), label)?; + let receiver = self.checkpoints.receive_archive(store, scratch()?, base)?; + let verified = upload + .prepare_publication(&cluster.domain, &self.tails, &uploaded, receiver) + .await? + .commit(&cluster.domain) + .await?; + Ok(verified.hold().clone()) + } + + fn source( + &self, + cluster: &Cluster, + hold: CellGroupArchiveHold, + label: &str, + ) -> Result { + Ok(CellGroupTransitionSource { + birth: cluster.birth.clone(), + hold, + receiver: self.receiver(cluster, label)?, + }) + } +} + +fn scratch() -> Result { + Ok(ScratchPool::new(ScratchLimits { + max_bytes: 32 * 1024 * 1024, + max_files: 1, + })?) +} + +pub(crate) async fn sealed_source_archives( + rooted: bool, + store: Arc, +) -> Result<()> { + let store = Arc::new(catalog_fault::Store::around(store)); + let mut cluster = Box::pin(Cluster::with_store_birth( + RaftRuntimeOptions::default(), + tempfile::tempdir()?, + Arc::new(DirectFileSystem), + [6; 32], + store.clone(), + ClusterBirthOptions { + group: GroupId::new([3; 16])?, + minimum: Durability::Quorum, + image_storage: if rooted { + RaftImageStorage::Extents + } else { + RaftImageStorage::Materialized + }, + log: Default::default(), + }, + )) + .await?; + cluster.elect().await?; + checkpoint_fixture::enroll_images(&mut cluster, rooted, false, 0, 4096).await?; + let archives = Archives::new(&cluster, rooted).await?; + let mut fixture = RuntimeCluster::from_cluster(cluster, 256, 256, 64).await?; + source_seal::attach_source_appliers(&mut fixture)?; + let scope = fixture.cluster.birth.route().history; + fixture.runtimes[0].campaign(scope).await?; + fixture.drive(None).await?; + let unsealed = archives.publish(&fixture, "verify-unsealed").await?; + let plan = source_seal::split_plan(&fixture.cluster).await?; + let directory = fixture.cluster.directory.clone(); + let domain = &fixture.cluster.domain; + domain.run(directory.plan_transition(&plan)).await??; + domain + .run(directory.prepare_transition(&plan, [15; 32])) + .await??; + domain + .run(directory.authorize_transition_seal(&plan)) + .await??; + let permit = domain + .run(directory.prepare_source_seal(&plan, scope, &fixture.cluster.voters[0].log)) + .await??; + let CellGroupSealOutcome::Accepted(seal) = fixture.runtimes[0].seal_source(permit).await? + else { + anyhow::bail!("first seal proposal"); + }; + tokio::time::timeout(Duration::from_secs(20), async { + while !fixture.applied[&scope] + .iter() + .all(|index| *index >= seal.position.index) + { + fixture.pump(&[], &mut BTreeMap::new()).await?; + tokio::task::yield_now().await; + } + Ok::<_, anyhow::Error>(()) + }) + .await + .context("replicate archived source seal")??; + let installed = Box::pin(verify_sealed_sources( + &fixture, + &archives, + store.clone(), + &plan, + unsealed, + seal.position, + )) + .await?; + let (committed, proofs) = + Box::pin(verify_pending_child_raft(&mut fixture, &plan, installed)).await?; + Box::pin(verify_installed_decision( + &fixture, + &plan, + store.clone(), + &committed, + proofs, + )) + .await?; + Box::pin(verify_transfer_archive( + &fixture, + &archives, + &plan, + store.clone(), + &committed, + )) + .await?; + let active = Box::pin(verify_transfer_activation( + &mut fixture, + &plan, + &committed, + store.clone(), + )) + .await?; + let (retired, acknowledgement) = + Box::pin(verify_parent_retirement(&mut fixture, &plan)).await?; + anyhow::ensure!(matches!( + directory + .settle_transfer_retirement( + &fixture.cluster.domain, + &archives.pool, + &plan, + vec![acknowledgement.clone()], + ) + .await, + Err(CellGroupArchiveBindingError::Catalog(CatalogError::Fenced)) + )); + Box::pin(verify_child_archive_coverage( + &fixture, + &archives, + &plan, + &active, + store.clone(), + )) + .await?; + Box::pin(verify_retirement_settlement( + &fixture, + &archives, + &plan, + acknowledgement, + store, + )) + .await?; + reopen_installed_children(&fixture, &plan, &active, &retired).await?; + fixture.close().await +} + +async fn verify_sealed_sources( + fixture: &RuntimeCluster, + archives: &Archives, + store: Arc, + plan: &DirectoryMutation, + unsealed: CellGroupArchiveHold, + seal: LogPosition, +) -> Result { + let directory = &fixture.cluster.directory; + let scope = fixture.cluster.birth.route().history; + let domain = &fixture.cluster.domain; + let sealed = archives.publish(fixture, "verify-sealed").await?; + anyhow::ensure!( + sealed + .marker() + .source_seal() + .context("archived seal")? + .position() + == seal + ); + anyhow::ensure!(matches!( + directory + .verify_and_record_sealed_sources(domain, &archives.pool, &archives.tails, plan, vec![]) + .await, + Err(CellGroupArchiveBindingError::Catalog( + CatalogError::Invalid(_) + )) + )); + anyhow::ensure!(matches!( + directory + .verify_and_record_sealed_sources( + domain, + &archives.pool, + &archives.tails, + plan, + vec![archives.source(&fixture.cluster, unsealed, "reject-preseal")?] + ) + .await, + Err(CellGroupArchiveBindingError::Catalog(CatalogError::Fenced)) + )); + let intent = archives.registry.prepare_acquire(domain).await?; + let released = archives.registry.acquire(domain, &intent).await?; + archives.registry.release(domain, &released).await?; + anyhow::ensure!(matches!( + directory + .verify_and_record_sealed_sources( + domain, + &archives.pool, + &archives.tails, + plan, + vec![archives.source(&fixture.cluster, released, "reject-released")?] + ) + .await, + Err(CellGroupArchiveBindingError::Archive(_)) + )); + let (_, journal) = domain + .run(directory.active_transition()) + .await?? + .context("authorized source")?; + anyhow::ensure!( + journal.phase() == TransitionPhase::SealAuthorized && journal.sealed_sources().is_none() + ); + + store.fail_head_confirmation.store(true, Ordering::Release); + anyhow::ensure!(matches!( + directory + .verify_and_record_sealed_sources( + domain, + &archives.pool, + &archives.tails, + plan, + vec![archives.source(&fixture.cluster, sealed.clone(), "verify-before-cas")?] + ) + .await, + Err(CellGroupArchiveBindingError::Catalog( + CatalogError::Unconfirmed + )) + )); + let catalog = Catalog::new(store.clone(), CatalogOptions::default())?; + let namespace = domain + .run(catalog.existing_namespace(directory.namespace_config().clone())) + .await??; + let reopened = RangeDirectory::new(namespace, DirectoryLimits::default())?; + let (original, journal) = domain + .run(reopened.active_transition()) + .await?? + .context("retained source set")?; + anyhow::ensure!( + original.encode() == plan.encode() && journal.phase() == TransitionPhase::Sealed + ); + let mut holds = domain + .run(reopened.sealed_source_holds(&original)) + .await??; + anyhow::ensure!(holds == vec![sealed.clone()]); + let birth = domain + .run(reopened.materialize_selected_history(scope)) + .await??; + // The exact verified source retains the only checkpoint receiver slot. + // Exercise the alternate reference before pinning the selected source. + let intent = archives.registry.prepare_acquire(domain).await?; + let alternate = archives.registry.acquire(domain, &intent).await?; + anyhow::ensure!(alternate != sealed); + let mut source = archives.source(&fixture.cluster, alternate.clone(), "reject-alternate")?; + source.birth = birth.clone(); + anyhow::ensure!(matches!( + reopened + .verify_and_record_sealed_sources( + domain, + &archives.pool, + &archives.tails, + &original, + vec![source] + ) + .await, + Err(CellGroupArchiveBindingError::Catalog( + CatalogError::Conflict(_) + )) + )); + archives.registry.release(domain, &alternate).await?; + let permit = domain + .run(reopened.prepare_source_seal(&original, scope, &fixture.cluster.voters[0].log)) + .await??; + anyhow::ensure!(matches!( + fixture.runtimes[0].seal_source(permit).await?, + CellGroupSealOutcome::Applied(_) + )); + let mut source = archives.source( + &fixture.cluster, + holds.pop().context("retained hold")?, + "verify-reopened", + )?; + source.birth = birth; + let verified = reopened + .verify_and_record_sealed_sources( + domain, + &archives.pool, + &archives.tails, + &original, + vec![source], + ) + .await?; + anyhow::ensure!(verified.journal() == &journal && verified.sources().len() == 1); + anyhow::ensure!(verified.sources()[0].marker() == sealed.marker()); + anyhow::ensure!(verified.sources()[0].recovery().manifest().pinned_images() == 3); + let installed = Box::pin(verify_partition( + &fixture.cluster, + archives, + &original, + verified, + )) + .await?; + Ok(installed) +} + +async fn verify_installed_decision( + fixture: &RuntimeCluster, + plan: &DirectoryMutation, + store: Arc, + markers: &[GroupManifest], + proofs: Vec, +) -> Result<()> { + let domain = &fixture.cluster.domain; + let directory = &fixture.cluster.directory; + anyhow::ensure!(domain + .run(directory.installed_transfer_children(plan)) + .await?? + .is_none()); + let before = domain.run(directory.observe()).await??; + anyhow::ensure!(matches!( + domain + .run(directory.record_transfer_child_quorums(plan, vec![proofs[0].clone()])) + .await?, + Err(CatalogError::Invalid(_)) + )); + let mut reversed = proofs.clone(); + reversed.reverse(); + anyhow::ensure!(matches!( + domain + .run(directory.record_transfer_child_quorums(plan, reversed)) + .await?, + Err(CatalogError::Fenced) + )); + store.fail_head_confirmation.store(true, Ordering::Release); + anyhow::ensure!(matches!( + domain + .run(directory.record_transfer_child_quorums(plan, proofs.clone())) + .await?, + Err(CatalogError::Unconfirmed) + )); + let catalog = Catalog::new(store, CatalogOptions::default())?; + let namespace = domain + .run(catalog.existing_namespace(directory.namespace_config().clone())) + .await??; + let reopened = RangeDirectory::new(namespace, DirectoryLimits::default())?; + let (original, decision) = domain + .run(reopened.active_transition()) + .await?? + .context("retained installed decision")?; + anyhow::ensure!(original.encode() == plan.encode()); + anyhow::ensure!(decision.phase() == TransitionPhase::Installed); + anyhow::ensure!(decision.installed_children().is_some()); + anyhow::ensure!( + domain + .run(reopened.installed_transfer_children(&original)) + .await?? + .as_deref() + == Some(markers) + ); + anyhow::ensure!( + domain + .run(reopened.record_transfer_child_quorums(&original, proofs)) + .await?? + == decision + ); + let after = domain.run(reopened.observe()).await??; + anyhow::ensure!(after.revision() == before.revision() && after.groups() == before.groups()); + let cell = CellId::new("alpha")?; + anyhow::ensure!( + domain.run(reopened.lookup(&after, &cell)).await?? == *fixture.cluster.birth.route() + ); + Ok(()) +} + +async fn verify_transfer_archive( + fixture: &RuntimeCluster, + archives: &Archives, + plan: &DirectoryMutation, + store: Arc, + markers: &[GroupManifest], +) -> Result<()> { + let cluster = &fixture.cluster; + let domain = &cluster.domain; + let directory = &cluster.directory; + let holds = domain.run(directory.sealed_source_holds(plan)).await??; + let hold = holds.first().context("sealed source hold")?; + anyhow::ensure!(domain + .run(directory.transfer_archive(plan)) + .await?? + .is_none()); + + anyhow::ensure!(matches!( + directory + .publish_transfer(domain, &archives.pool, plan) + .await, + Err(CellGroupArchiveBindingError::Catalog(CatalogError::Fenced)) + )); + anyhow::ensure!(matches!( + domain + .run(directory.prepare_transfer_activation( + plan, + markers[0].route().history, + &cluster.voters[0].log, + )) + .await?, + Err(CatalogError::Fenced) + )); + let inputs = archive_inputs(archives, cluster, hold, markers, "closure-outage")?; + store + .deny_group_payload_reads + .store(true, Ordering::Release); + let denied = directory + .verify_and_record_transfer_archive(domain, plan, inputs) + .await; + store + .deny_group_payload_reads + .store(false, Ordering::Release); + anyhow::ensure!(matches!( + denied, + Err(CellGroupArchiveBindingError::Archive(_)) + )); + anyhow::ensure!(store.denied_group_payload_reads.load(Ordering::Acquire) > 0); + let (_, installed) = domain + .run(directory.active_transition()) + .await?? + .context("installed decision after archive refusal")?; + anyhow::ensure!(installed.phase() == TransitionPhase::Installed); + anyhow::ensure!(installed.transfer_archive().is_none()); + + let inputs = archive_inputs(archives, cluster, hold, markers, "closure-verify")?; + store.fail_head_confirmation.store(true, Ordering::Release); + anyhow::ensure!(matches!( + directory + .verify_and_record_transfer_archive(domain, plan, inputs) + .await, + Err(CellGroupArchiveBindingError::Catalog( + CatalogError::Unconfirmed + )) + )); + + let catalog = Catalog::new(store.clone(), CatalogOptions::default())?; + let namespace = domain + .run(catalog.existing_namespace(directory.namespace_config().clone())) + .await??; + let reopened = RangeDirectory::new(namespace, DirectoryLimits::default())?; + let (original, archived) = domain + .run(reopened.active_transition()) + .await?? + .context("archived decision")?; + anyhow::ensure!(original.encode() == plan.encode()); + anyhow::ensure!(archived.phase() == TransitionPhase::Archived); + let closure = domain + .run(reopened.transfer_archive(&original)) + .await?? + .context("retained transfer archive")?; + anyhow::ensure!(archived.transfer_archive() == Some(closure.digest)); + anyhow::ensure!(closure.source_holds == holds && closure.children == markers); + + let mut inputs = archive_inputs(archives, cluster, hold, markers, "closure-replay")?; + inputs.sources[0].birth = domain + .run(reopened.materialize_selected_history(cluster.birth.route().history)) + .await??; + anyhow::ensure!( + reopened + .verify_and_record_transfer_archive(domain, &original, inputs) + .await? + == archived + ); + anyhow::ensure!(matches!( + domain.run(reopened.publish(&original)).await?, + Err(CatalogError::Fenced) + )); + let view = domain.run(reopened.observe()).await??; + anyhow::ensure!( + domain + .run(reopened.lookup(&view, &CellId::new("alpha")?)) + .await?? + == *cluster.birth.route() + ); + Box::pin(verify_transfer_publication( + fixture, archives, &reopened, &original, view, store, &closure, + )) + .await +} + +async fn verify_transfer_publication( + fixture: &RuntimeCluster, + archives: &Archives, + directory: &RangeDirectory, + plan: &DirectoryMutation, + before: DirectoryView, + store: Arc, + closure: &CellGroupTransferArchive, +) -> Result<()> { + let cluster = &fixture.cluster; + let domain = &cluster.domain; + + // The conditional update commits but its immediate read-back fails. + store.fail_head_confirmation.store(true, Ordering::Release); + let publication = directory + .publish_transfer(domain, &archives.pool, plan) + .await?; + anyhow::ensure!(!store.fail_head_confirmation.load(Ordering::Acquire)); + anyhow::ensure!(publication.transfer_archive() == Some(closure.digest)); + anyhow::ensure!(publication.revision() == before.revision() + 1); + + let old = domain.run(directory.group_page(&before, None, 3)).await??; + anyhow::ensure!(old.next.is_none() && old.routes.as_slice() == [cluster.birth.route().clone()]); + let after = domain.run(directory.observe()).await??; + let new = domain.run(directory.group_page(&after, None, 3)).await??; + anyhow::ensure!(after.revision() == publication.revision() && new.next.is_none()); + let expected = closure.children.iter().map(GroupManifest::route); + anyhow::ensure!(new.routes.iter().eq(expected)); + anyhow::ensure!(new.routes.len() == 2); + let cell = CellId::new("alpha")?; + let child = domain.run(directory.lookup(&after, &cell)).await??; + anyhow::ensure!(new.routes.contains(&child)); + + let catalog = Catalog::new(store, CatalogOptions::default())?; + let namespace = domain + .run(catalog.existing_namespace(directory.namespace_config().clone())) + .await??; + let reopened = RangeDirectory::new(namespace, DirectoryLimits::default())?; + let (original, decision) = domain + .run(reopened.active_transition()) + .await?? + .context("published transition")?; + anyhow::ensure!(original.encode() == plan.encode()); + anyhow::ensure!(decision.phase() == TransitionPhase::Published); + anyhow::ensure!( + reopened + .publish_transfer(domain, &archives.pool, &original) + .await? + == publication + ); + let retained = domain + .run(reopened.transfer_archive(&original)) + .await?? + .context("published transfer archive")?; + anyhow::ensure!(retained.digest == closure.digest); + anyhow::ensure!( + retained.source_holds == closure.source_holds && retained.children == closure.children + ); + + anyhow::ensure!(matches!( + domain + .run(reopened.prepare_source_retirement( + &original, + cluster.birth.route().history, + &cluster.voters[0].log, + )) + .await?, + Err(CatalogError::Fenced) + )); + for route in &new.routes { + let birth = domain + .run(reopened.materialize_selected_history(route.history)) + .await??; + let (bootstrap, reads) = domain + .run(catalog_fault::count_reads( + reopened.bootstrap_group_member(&birth, &cluster.voters[0].log), + )) + .await?; + anyhow::ensure!(matches!(bootstrap, Err(CatalogError::Fenced))); + anyhow::ensure!(!reads.keys().any(|key| key.contains("/transfer-installs/"))); + anyhow::ensure!(matches!( + fixture.runtimes[0].propose(route.history, vec![1]).await, + Err(RuntimeError::NotReady) + )); + anyhow::ensure!(matches!( + fixture.runtimes[0].read_index(route.history, vec![1]).await, + Err(RuntimeError::NotReady) + )); + } + Ok(()) +} + +async fn verify_transfer_activation( + fixture: &mut RuntimeCluster, + plan: &DirectoryMutation, + children: &[GroupManifest], + store: Arc, +) -> Result> { + let directory = fixture.cluster.directory.clone(); + let domain = &fixture.cluster.domain; + let log = fixture.cluster.voters[0].log.clone(); + let publication = domain + .run(directory.publication(plan)) + .await?? + .context("activation publication")?; + let mut positions = Vec::with_capacity(children.len()); + for child in children { + let scope = child.route().history; + let wrong_engine = domain + .run(directory.prepare_transfer_activation(plan, scope, &fixture.cluster.voters[1].log)) + .await??; + anyhow::ensure!(matches!( + fixture.runtimes[0].activate_transfer(wrong_engine).await, + Err(RuntimeError::StaleCompletion) + )); + let permit = domain + .run(directory.prepare_transfer_activation(plan, scope, &log)) + .await??; + let CellGroupTransferActivation::Accepted(activation) = + fixture.runtimes[0].activate_transfer(permit).await? + else { + anyhow::bail!("expected the first transfer activation proposal"); + }; + let duplicate = domain + .run(directory.prepare_transfer_activation(plan, scope, &log)) + .await??; + let CellGroupTransferActivation::Accepted(retry) = + fixture.runtimes[0].activate_transfer(duplicate).await? + else { + anyhow::bail!("activation cannot commit before messages are delivered"); + }; + anyhow::ensure!(retry.position == activation.position && retry.digest == activation.digest); + anyhow::ensure!(matches!( + fixture.runtimes[0].propose(scope, vec![1]).await, + Err(RuntimeError::NotReady) + )); + positions.push((scope, activation.position.index)); + } + await_applied(fixture, &positions) + .await + .context("commit transfer activations")?; + + let domain = &fixture.cluster.domain; + let mut acknowledgements = Vec::with_capacity(children.len()); + let mut expected_activations = Vec::with_capacity(children.len()); + for (child, (scope, position)) in children.iter().zip(positions) { + let retry = domain + .run(directory.prepare_transfer_activation(plan, scope, &log)) + .await??; + let CellGroupTransferActivation::Applied(applied) = + fixture.runtimes[0].activate_transfer(retry).await? + else { + anyhow::bail!("transfer activation did not return durable acknowledgement"); + }; + anyhow::ensure!(applied.position().index == position); + expected_activations.push((scope, applied.position())); + acknowledgements.push(*applied); + for voter in &fixture.cluster.voters { + let marker = voter + .log + .cell_group(&scope)? + .group_manifest()? + .context("active child")?; + anyhow::ensure!(marker.is_active() && marker.root() == child.root()); + let retained = serde_json::to_value(&marker)?; + anyhow::ensure!(serde_json::from_value::(retained.clone())? == marker); + anyhow::ensure!( + retained["transfer"]["activation"]["publication"] + == serde_json::to_value(publication.journal_digest())? + ); + } + } + + Box::pin(verify_active_decision( + &fixture.cluster, + &directory, + plan, + acknowledgements, + &expected_activations, + store, + )) + .await?; + let view = domain.run(directory.observe()).await??; + let original_cell = CellId::new("alpha")?; + let original_route = domain + .run(directory.lookup(&view, &original_cell)) + .await??; + let existing = history(&original_route, "alpha")?; + let runtime = fixture.runtimes[0].clone(); + let deadline = domain + .run(async { cellar::asyncrt::mono_ms() }) + .await? + .checked_add(5_000) + .context("activation read deadline")?; + let read = runtime.read_authority( + original_route.history, + existing.clone(), + CellReadConsistency::Linearizable, + deadline, + ); + tokio::pin!(read); + let authority = tokio::time::timeout(Duration::from_secs(5), async { + loop { + if let std::task::Poll::Ready(result) = futures_util::poll!(&mut read) { + return Ok::<_, anyhow::Error>(result?); + } + fixture.pump(&[], &mut BTreeMap::new()).await?; + tokio::task::yield_now().await; + } + }) + .await + .context("linearizable read after activation")??; + anyhow::ensure!(authority.cell().version.history == existing && authority.barrier().is_some()); + drop(authority); + + let domain = &fixture.cluster.domain; + let name = CellId::new("after-transfer")?; + let route = domain.run(directory.lookup(&view, &name)).await??; + let cell = history(&route, name.as_str())?; + let encoded = CellGroupEntry { + binding: GroupEntryBinding { + group: route.history, + route_epoch: route.epoch, + birth_digest: route.birth_digest, + schema_digest: route.schema_digest, + proposal: ProposalId::new([91; 16])?, + }, + command: GroupCommand::Enroll { + cell: cell.clone(), + image: LtxImage { + epoch: 1, + txid: 0, + checksum: 0, + commitment: ImageCommitment::Sha256(sha2::Sha256::digest([]).into()), + }, + files: Vec::new(), + }, + } + .encode(&route, GroupEntryLimits::default())?; + let before = log + .cell_group(&route.history)? + .group_manifest()? + .context("child before write")?; + let write = runtime.propose(route.history, encoded).await?; + await_applied(fixture, &[(route.history, write.position.index)]) + .await + .context("commit post-activation enrollment")?; + let after = log + .cell_group(&route.history)? + .group_manifest()? + .context("child after write")?; + anyhow::ensure!(after.is_active() && after.root().cells() == before.root().cells() + 1); + children + .iter() + .map(|child| { + log.cell_group(&child.route().history)? + .group_manifest()? + .context("retained active child") + }) + .collect() +} + +async fn verify_active_decision( + cluster: &Cluster, + directory: &RangeDirectory, + plan: &DirectoryMutation, + acknowledgements: Vec, + positions: &[(GroupHistory, LogPosition)], + store: Arc, +) -> Result<()> { + let domain = &cluster.domain; + anyhow::ensure!(domain + .run(directory.transfer_activations(plan)) + .await?? + .is_none()); + anyhow::ensure!(matches!( + domain + .run(directory.record_transfer_activations(plan, vec![acknowledgements[0].clone()],)) + .await?, + Err(CatalogError::Invalid(_)) + )); + let mut reversed = acknowledgements.clone(); + reversed.reverse(); + anyhow::ensure!(matches!( + domain + .run(directory.record_transfer_activations(plan, reversed)) + .await?, + Err(CatalogError::Fenced) + )); + let (_, before) = domain + .run(directory.active_transition()) + .await?? + .context("published transition before activation settlement")?; + anyhow::ensure!(before.phase() == TransitionPhase::Published); + store.fail_head_confirmation.store(true, Ordering::Release); + anyhow::ensure!(matches!( + domain + .run(directory.record_transfer_activations(plan, acknowledgements.clone())) + .await?, + Err(CatalogError::Unconfirmed) + )); + + let catalog = Catalog::new(store, CatalogOptions::default())?; + let namespace = domain + .run(catalog.existing_namespace(directory.namespace_config().clone())) + .await??; + let reopened = RangeDirectory::new(namespace, DirectoryLimits::default())?; + let (original, active) = domain + .run(reopened.active_transition()) + .await?? + .context("active transition after lost reply")?; + anyhow::ensure!(original.encode() == plan.encode()); + anyhow::ensure!( + active.phase() == TransitionPhase::Active && active.activated_children().is_some() + ); + anyhow::ensure!( + domain + .run(reopened.transfer_activations(&original)) + .await?? + .as_deref() + == Some(positions) + ); + anyhow::ensure!( + domain + .run(reopened.record_transfer_activations(&original, acknowledgements)) + .await?? + == active + ); + anyhow::ensure!( + domain.run(reopened.publication(&original)).await?? + == domain.run(directory.publication(plan)).await?? + ); + Ok(()) +} + +async fn await_applied( + fixture: &mut RuntimeCluster, + positions: &[(GroupHistory, LogIndex)], +) -> Result<()> { + tokio::time::timeout(Duration::from_secs(20), async { + loop { + fixture.pump(&[], &mut BTreeMap::new()).await?; + if positions.iter().all(|(group, position)| { + fixture.applied[group] + .iter() + .all(|applied| applied >= position) + }) { + return Ok::<_, anyhow::Error>(()); + } + tokio::task::yield_now().await; + } + }) + .await + .context("replicate child entries")? +} + +async fn verify_parent_retirement( + fixture: &mut RuntimeCluster, + plan: &DirectoryMutation, +) -> Result<(GroupManifest, CellGroupSourceRetirementAck)> { + let directory = fixture.cluster.directory.clone(); + let scope = fixture.cluster.birth.route().history; + let log = fixture.cluster.voters[0].log.clone(); + let domain = &fixture.cluster.domain; + let before = log + .cell_group(&scope)? + .group_manifest()? + .context("sealed parent")?; + let published = domain + .run(directory.publication(plan)) + .await?? + .context("retirement publication")?; + let wrong = domain + .run(directory.prepare_source_retirement(plan, scope, &fixture.cluster.voters[1].log)) + .await??; + anyhow::ensure!(matches!( + fixture.runtimes[0].retire_source(wrong).await, + Err(RuntimeError::StaleCompletion) + )); + let permit = domain + .run(directory.prepare_source_retirement(plan, scope, &log)) + .await??; + let CellGroupSourceRetirement::Accepted(retirement) = + fixture.runtimes[0].retire_source(permit).await? + else { + anyhow::bail!("expected first retirement proposal"); + }; + let retry = domain + .run(directory.prepare_source_retirement(plan, scope, &log)) + .await??; + let CellGroupSourceRetirement::Accepted(duplicate) = + fixture.runtimes[0].retire_source(retry).await? + else { + anyhow::bail!("retirement cannot commit before Raft delivery"); + }; + anyhow::ensure!( + duplicate.position == retirement.position && duplicate.digest == retirement.digest + ); + await_applied(fixture, &[(scope, retirement.position.index)]) + .await + .context("commit the sealed parent's tombstone")?; + + let domain = &fixture.cluster.domain; + let retry = domain + .run(directory.prepare_source_retirement(plan, scope, &log)) + .await??; + let CellGroupSourceRetirement::Applied(acknowledgement) = + fixture.runtimes[0].retire_source(retry).await? + else { + anyhow::bail!("retirement retry lost its applied tombstone"); + }; + let tombstone = acknowledgement.tombstone(); + anyhow::ensure!(tombstone.position() == retirement.position); + anyhow::ensure!(tombstone.publication() == published.journal_digest()); + for (voter, runtime) in fixture.cluster.voters.iter().zip(&fixture.runtimes) { + let marker = voter + .log + .cell_group(&scope)? + .group_manifest()? + .context("retired parent")?; + anyhow::ensure!(!marker.is_active() && marker.root() == before.root()); + let seal = marker.source_seal().context("retired source seal")?; + anyhow::ensure!( + seal.position() == before.source_seal().context("original seal")?.position() + ); + anyhow::ensure!(seal.retirement() == Some(tombstone)); + anyhow::ensure!(matches!( + runtime.propose(scope, vec![1]).await, + Err(RuntimeError::NotReady) + )); + anyhow::ensure!(matches!( + runtime.read_index(scope, vec![1]).await, + Err(RuntimeError::NotReady) + )); + } + + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let snapshot = fixture.runtimes[0].capture_checkpoint(scope, &pool).await?; + let selected = snapshot.marker().clone(); + anyhow::ensure!(selected.source_seal().and_then(|seal| seal.retirement()) == Some(tombstone)); + fixture.runtimes[0] + .publish_checkpoint(scope, snapshot.prepare_publication()?) + .await?; + let group = log.cell_group(&scope)?; + tokio::time::timeout(Duration::from_secs(20), async { + while group.checkpoint_manifest()?.as_ref() != Some(&selected) { + fixture.pump(&[], &mut BTreeMap::new()).await?; + tokio::task::yield_now().await; + } + Ok::<_, anyhow::Error>(()) + }) + .await + .context("persist the parent tombstone checkpoint")??; + let domain = &fixture.cluster.domain; + let (_, journal) = domain + .run(directory.active_transition()) + .await?? + .context("topology still owns parent archive holds")?; + anyhow::ensure!(journal.phase() == TransitionPhase::Active); + let hold = domain.run(directory.sealed_source_holds(plan)).await??; + anyhow::ensure!(hold.len() == 1); + let held = hold[0].marker(); + anyhow::ensure!(held.route() == fixture.cluster.birth.route()); + anyhow::ensure!(held.root() == before.root() && held.source_seal() == before.source_seal()); + Ok((selected, *acknowledgement)) +} + +async fn verify_child_archive_coverage( + fixture: &RuntimeCluster, + archives: &Archives, + plan: &DirectoryMutation, + active: &[GroupManifest], + store: Arc, +) -> Result<()> { + let cluster = &fixture.cluster; + let domain = &cluster.domain; + let directory = &cluster.directory; + let parent_holds = domain.run(directory.sealed_source_holds(plan)).await??; + let mut holds = Vec::with_capacity(active.len()); + for (index, marker) in active.iter().enumerate() { + let birth = domain + .run(directory.materialize_selected_history(marker.route().history)) + .await??; + let registry = directory + .prepare_group_archive_binding(domain, &archives.pool, &birth) + .await? + .commit(domain) + .await?; + let hold = archives + .publish_group( + fixture, + ®istry, + marker.route(), + &format!("child-coverage-upload-{index}"), + ) + .await?; + anyhow::ensure!(hold.marker() == marker); + holds.push(hold); + } + anyhow::ensure!(domain + .run(directory.retirement_child_holds(plan)) + .await?? + .is_none()); + let incomplete = child_archive_inputs( + cluster, + archives, + directory, + &holds[..1], + "coverage-incomplete", + ) + .await?; + anyhow::ensure!(matches!( + directory + .verify_and_record_child_archives( + domain, + &archives.pool, + &archives.tails, + plan, + incomplete + ) + .await, + Err(CellGroupArchiveBindingError::Catalog( + CatalogError::Invalid(_) + )) + )); + let mut reversed = + child_archive_inputs(cluster, archives, directory, &holds, "coverage-reversed").await?; + reversed.reverse(); + anyhow::ensure!(matches!( + directory + .verify_and_record_child_archives( + domain, + &archives.pool, + &archives.tails, + plan, + reversed + ) + .await, + Err(CellGroupArchiveBindingError::Catalog(CatalogError::Fenced)) + )); + + let outage = + child_archive_inputs(cluster, archives, directory, &holds, "coverage-outage").await?; + let denied_before = store.denied_group_payload_reads.load(Ordering::Acquire); + store + .deny_group_payload_reads + .store(true, Ordering::Release); + let refused = directory + .verify_and_record_child_archives(domain, &archives.pool, &archives.tails, plan, outage) + .await; + store + .deny_group_payload_reads + .store(false, Ordering::Release); + anyhow::ensure!(matches!( + refused, + Err(CellGroupArchiveBindingError::Archive(_)) + )); + anyhow::ensure!(store.denied_group_payload_reads.load(Ordering::Acquire) > denied_before); + anyhow::ensure!(domain + .run(directory.retirement_child_holds(plan)) + .await?? + .is_none()); + + let verified = + child_archive_inputs(cluster, archives, directory, &holds, "coverage-verify").await?; + store.fail_head_confirmation.store(true, Ordering::Release); + let lost = directory + .verify_and_record_child_archives(domain, &archives.pool, &archives.tails, plan, verified) + .await; + anyhow::ensure!( + matches!( + lost, + Err(CellGroupArchiveBindingError::Catalog( + CatalogError::Unconfirmed + )) + ), + "expected unconfirmed child archive coverage, got {lost:?}" + ); + let catalog = Catalog::new(store, CatalogOptions::default())?; + let namespace = domain + .run(catalog.existing_namespace(directory.namespace_config().clone())) + .await??; + let reopened = RangeDirectory::new(namespace, DirectoryLimits::default())?; + let (original, decision) = domain + .run(reopened.active_transition()) + .await?? + .context("active transition with child archive proof")?; + anyhow::ensure!(original.encode() == plan.encode()); + anyhow::ensure!( + decision.phase() == TransitionPhase::Active && decision.child_archives().is_some() + ); + anyhow::ensure!( + domain + .run(reopened.retirement_child_holds(&original)) + .await?? + .as_deref() + == Some(holds.as_slice()) + ); + let replay = + child_archive_inputs(cluster, archives, &reopened, &holds, "coverage-replay").await?; + anyhow::ensure!( + reopened + .verify_and_record_child_archives( + domain, + &archives.pool, + &archives.tails, + &original, + replay, + ) + .await? + == decision + ); + + anyhow::ensure!( + domain + .run(reopened.sealed_source_holds(&original)) + .await?? + == parent_holds + ); + let parent = parent_holds + .first() + .context("original parent archive hold")?; + let held = archives + .registry + .verify_hold( + domain, + parent, + &archives.tails, + archives.receiver(cluster, "parent-hold-after-coverage")?, + ) + .await?; + anyhow::ensure!(held.marker() == parent.marker()); + Ok(()) +} + +async fn verify_retirement_settlement( + fixture: &RuntimeCluster, + archives: &Archives, + plan: &DirectoryMutation, + acknowledgement: CellGroupSourceRetirementAck, + store: Arc, +) -> Result<()> { + let cluster = &fixture.cluster; + let domain = &cluster.domain; + let directory = &cluster.directory; + let publication = domain.run(directory.publication(plan)).await??; + let parent_holds = domain.run(directory.sealed_source_holds(plan)).await??; + let child_holds = domain + .run(directory.retirement_child_holds(plan)) + .await?? + .context("selected child holds")?; + anyhow::ensure!(matches!( + directory + .release_retired_parent_holds(domain, &archives.pool, plan) + .await, + Err(CellGroupArchiveBindingError::Catalog(CatalogError::Fenced)) + )); + store.fail_head_confirmation.store(true, Ordering::Release); + anyhow::ensure!(matches!( + directory + .settle_transfer_retirement(domain, &archives.pool, plan, vec![acknowledgement.clone()]) + .await, + Err(CellGroupArchiveBindingError::Catalog( + CatalogError::Unconfirmed + )) + )); + + let catalog = Catalog::new(store.clone(), CatalogOptions::default())?; + let namespace = domain + .run(catalog.existing_namespace(directory.namespace_config().clone())) + .await??; + let reopened = RangeDirectory::new(namespace, DirectoryLimits::default())?; + let (original, retired) = domain + .run(reopened.active_transition()) + .await?? + .context("retired transition still owns topology")?; + anyhow::ensure!(original.encode() == plan.encode()); + anyhow::ensure!( + retired.phase() == TransitionPhase::Retired && retired.retired_parents().is_some() + ); + anyhow::ensure!(!retired.parent_holds_released()); + let parent = parent_holds.first().context("retired parent hold")?; + let retained = archives + .registry + .verify_hold( + domain, + parent, + &archives.tails, + archives.receiver(cluster, "retired-before-release")?, + ) + .await?; + anyhow::ensure!(retained.marker() == parent.marker()); + drop(retained); + + store.deny_registry_writes.store(true, Ordering::Release); + let denied = reopened + .release_retired_parent_holds(domain, &archives.pool, &original) + .await; + store.deny_registry_writes.store(false, Ordering::Release); + anyhow::ensure!(matches!( + denied, + Err(CellGroupArchiveBindingError::Archive(_)) + )); + let (_, pending) = domain + .run(reopened.active_transition()) + .await?? + .context("release refusal keeps topology owned")?; + anyhow::ensure!(pending == retired); + + store.fail_head_confirmation.store(true, Ordering::Release); + let completed = reopened + .release_retired_parent_holds(domain, &archives.pool, &original) + .await?; + anyhow::ensure!(!store.fail_head_confirmation.load(Ordering::Acquire)); + anyhow::ensure!( + completed.phase() == TransitionPhase::Retired && completed.parent_holds_released() + ); + anyhow::ensure!(domain.run(reopened.active_transition()).await??.is_none()); + anyhow::ensure!( + reopened + .release_retired_parent_holds(domain, &archives.pool, &original) + .await? + == completed + ); + anyhow::ensure!( + reopened + .settle_transfer_retirement(domain, &archives.pool, &original, vec![acknowledgement],) + .await? + == completed + ); + anyhow::ensure!(domain.run(reopened.publication(&original)).await?? == publication); + anyhow::ensure!( + domain + .run(reopened.retirement_child_holds(&original)) + .await?? + .as_deref() + == Some(child_holds.as_slice()) + ); + anyhow::ensure!(matches!( + archives + .registry + .verify_hold( + domain, + parent, + &archives.tails, + archives.receiver(cluster, "retired-after-release")?, + ) + .await, + Err(CellGroupArchiveError::Expired) + )); + Ok(()) +} + +async fn child_archive_inputs( + cluster: &Cluster, + archives: &Archives, + directory: &RangeDirectory, + holds: &[CellGroupArchiveHold], + label: &str, +) -> Result> { + let mut children = Vec::with_capacity(holds.len()); + for (index, hold) in holds.iter().enumerate() { + let route = hold.marker().route(); + let birth = cluster + .domain + .run(directory.materialize_selected_history(route.history)) + .await??; + let (store, base) = + archives.store(cluster, route.clone(), &format!("{label}/child-{index}"))?; + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let receiver = pool.receive_archive(store, scratch()?, base)?; + children.push(CellGroupTransitionSource { + birth, + hold: hold.clone(), + receiver, + }); + } + Ok(children) +} + +fn archive_inputs( + archives: &Archives, + cluster: &Cluster, + hold: &CellGroupArchiveHold, + markers: &[GroupManifest], + label: &str, +) -> Result { + Ok(CellGroupTransitionPreparationInputs { + archives: archives.pool.clone(), + tails: archives.tails.clone(), + restore: transfer_pool()?, + sources: vec![archives.source(cluster, hold.clone(), &format!("{label}/source"))?], + targets: transfer_targets( + archives, + cluster, + markers.iter().map(GroupManifest::route), + label, + )?, + }) +} + +async fn reopen_installed_children( + fixture: &RuntimeCluster, + plan: &DirectoryMutation, + installed: &[GroupManifest], + retired: &GroupManifest, +) -> Result<()> { + let cluster = &fixture.cluster; + fixture.runtimes[0].close().await?; + cluster.voters[0].log.close().await?; + let log = cluster + .domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + cluster._root.path().join("node-1/log"), + NodeId::new([1; 16])?, + DiskLineage::new([11; 16])?, + ProcessGeneration::new([91; 16])?, + LogOpenMode::Resume, + ), + Arc::new(DirectFileSystem), + )) + .await??; + let result = async { + let parent = log.cell_group(&retired.route().history)?; + anyhow::ensure!(parent.group_manifest()?.as_ref() == Some(retired)); + anyhow::ensure!(parent.checkpoint_manifest()?.as_ref() == Some(retired)); + for marker in installed { + let permit = cluster + .domain + .run(cluster.directory.bootstrap_pending_transfer_voter( + plan, + marker.route().history, + &log, + )) + .await??; + let group = permit.group(); + anyhow::ensure!( + group.group_manifest()?.as_ref() == Some(marker) + && group.identity().process == ProcessGeneration::new([91; 16])? + && marker.is_active() + ); + } + Ok::<_, anyhow::Error>(()) + } + .await; + let joined = log.close().await; + result?; + Ok(joined?) +} + +struct InstalledChildPartitions { + genesis: Vec, + voters: Vec<(CellGroupRecoveryPool, PreparedCellGroupTransfer)>, +} + +async fn verify_pending_child_raft( + fixture: &mut RuntimeCluster, + plan: &DirectoryMutation, + installed: InstalledChildPartitions, +) -> Result<(Vec, Vec)> { + let cluster = &fixture.cluster; + let directory = &cluster.directory; + let domain = &cluster.domain; + for (index, (pool, partition)) in installed.voters.into_iter().enumerate() { + let log = &cluster.voters[index].log; + let (apply_pool, _) = fixture + .appliers + .get(&(index, cluster.birth.route().history)) + .context("voter apply pool")?; + let apply_pool = apply_pool.clone(); + for child in partition.children() { + let history = child.route().history; + let permit = domain + .run(directory.bootstrap_pending_transfer_voter(plan, history, log)) + .await??; + // The exact same physical content on another voter grants no right + // to use this voter's page and image owner. + let other = &cluster.voters[(index + 1) % cluster.voters.len()].log; + let foreign = domain + .run(directory.bootstrap_pending_transfer_voter(plan, history, other)) + .await??; + anyhow::ensure!(matches!( + child.installed_applier(&foreign), + Err(RuntimeError::StaleCompletion) + )); + let applier = child.installed_applier(&permit)?; + let recover = applier.clone(); + let root = domain + .blocking(move || recover.recover_selected())? + .await??; + anyhow::ensure!(Some(root.marker()) == child.genesis().map(|genesis| genesis.marker())); + fixture.runtimes[index].attach(permit, root).await?; + fixture + .appliers + .insert((index, history), (apply_pool.clone(), applier)); + fixture + .applied + .entry(history) + .or_insert_with(|| vec![LogIndex::new(1); 3]); + } + let retained = partition.children()[0] + .genesis() + .context("child genesis")? + .clone(); + drop(partition); + anyhow::ensure!(pool.usage().operations == 1); + drop(retained); + anyhow::ensure!(pool.usage().operations == 0); + } + + for genesis in &installed.genesis { + let history = genesis.route().history; + anyhow::ensure!(matches!( + fixture.runtimes[0].pending_transfer_quorum(history).await, + Err(RuntimeError::NotLeader) + )); + fixture.runtimes[0].campaign(history).await?; + anyhow::ensure!(matches!( + fixture.runtimes[0].pending_transfer_quorum(history).await, + Err(RuntimeError::NotLeader | RuntimeError::NotReady) + )); + } + tokio::time::timeout(Duration::from_secs(20), async { + loop { + fixture.pump(&[], &mut BTreeMap::new()).await?; + if installed.genesis.iter().all(|marker| { + fixture.applied[&marker.route().history] + .iter() + .all(|index| index.get() >= 2) + }) { + break; + } + tokio::task::yield_now().await; + } + Ok::<_, anyhow::Error>(()) + }) + .await + .context("commit pending child Raft barriers")??; + + let mut committed = Vec::with_capacity(installed.genesis.len()); + let mut proofs = Vec::with_capacity(installed.genesis.len()); + for genesis in installed.genesis { + let history = genesis.route().history; + let status = fixture.runtimes[0].status(history).await?; + anyhow::ensure!( + !status.ready + && status.leader == Some(1) + && status.term.get() > 1 + && status.applied.term == status.term + && status.committed >= status.applied.index + ); + for runtime in &fixture.runtimes { + anyhow::ensure!(matches!( + runtime.propose(history, vec![1]).await, + Err(RuntimeError::NotReady) + )); + anyhow::ensure!(matches!( + runtime.read_index(history, vec![1]).await, + Err(RuntimeError::NotReady) + )); + } + let marker = fixture.cluster.voters[0] + .log + .cell_group(&history)? + .group_manifest()? + .context("applied child root")?; + anyhow::ensure!( + marker.position() == status.applied + && marker.root() == genesis.root() + && marker.transfer_origin() == genesis.transfer_origin() + && !marker.is_active() + ); + proofs.push(fixture.runtimes[0].pending_transfer_quorum(history).await?); + anyhow::ensure!(matches!( + fixture.runtimes[1].pending_transfer_quorum(history).await, + Err(RuntimeError::NotLeader) + )); + committed.push(marker); + } + Ok((committed, proofs)) +} + +async fn verify_partition( + cluster: &Cluster, + archives: &Archives, + plan: &DirectoryMutation, + verified: VerifiedCellGroupTransitionSources, +) -> Result { + let sources = verified + .journal() + .sealed_sources() + .context("sealed source set missing")?; + let directory = &cluster.directory; + let mut installed = Vec::new(); + let mut partitions = Vec::new(); + for (voter_index, voter) in cluster.voters.iter().enumerate() { + let pool = transfer_pool()?; + let targets = transfer_targets( + archives, + cluster, + verified.children().iter(), + &format!("node-{}", voter_index + 1), + )?; + let partition = pool + .prepare_transfer_partition(&cluster.domain, verified.clone(), targets) + .await?; + anyhow::ensure!(pool.usage().operations == 1); + if voter_index == 0 { + verify_partition_data(archives, plan, &partition, sources)?; + } + for (child_index, child) in partition.children().iter().enumerate() { + let marker = child.genesis().context("child genesis")?.marker(); + let history = child.route().history; + let first = cluster + .domain + .run(directory.install_transfer_voter(plan, child, &voter.log)) + .await??; + anyhow::ensure!(first.group_manifest()?.as_ref() == Some(marker)); + if voter_index == 0 { + verify_install_retry(cluster, plan, child, &first).await?; + installed.push(marker.clone()); + } else { + anyhow::ensure!(&installed[child_index] == marker); + } + let quorum = cluster + .domain + .run(directory.observe_transfer_install_quorum(plan, history)) + .await??; + if voter_index == 0 { + anyhow::ensure!(quorum.is_none()); + anyhow::ensure!(matches!( + cluster + .domain + .run(directory.bootstrap_pending_transfer_voter(plan, history, &voter.log)) + .await?, + Err(CatalogError::Unconfirmed) + )); + } else { + let proof = quorum.context("missing physical install quorum")?; + anyhow::ensure!( + proof.genesis == *marker + && proof.voters.len() == voter_index + 1 + && proof + .voters + .iter() + .zip(&cluster.voters) + .all(|(receipt, expected)| { + let group = expected.log.cell_group(&history); + group.is_ok_and(|group| { + group.identity().member == receipt.member + && group.identity().node == receipt.node + && group.identity().disk == receipt.disk + }) && expected + .log + .storage_instance() + .is_ok_and(|engine| engine == receipt.engine) + }) + ); + } + } + partitions.push((pool, partition)); + } + Ok(InstalledChildPartitions { + genesis: installed, + voters: partitions, + }) +} + +fn transfer_pool() -> Result { + Ok(CellGroupRecoveryPool::new(CellGroupRecoveryLimits { + max_cells: 3, + max_image_bytes: 1024 * 1024, + max_restore_bytes: 4 * 1024 * 1024, + ..Default::default() + })?) +} + +fn transfer_targets<'a>( + archives: &Archives, + cluster: &Cluster, + routes: impl Iterator, + label: &str, +) -> Result> { + routes + .enumerate() + .map(|(index, route)| { + let (store, directory) = + archives.store(cluster, route.clone(), &format!("{label}/child-{index}"))?; + Ok(CellGroupTransferTarget { + store, + directory, + scratch: scratch()?, + }) + }) + .collect() +} + +fn verify_partition_data( + archives: &Archives, + plan: &DirectoryMutation, + partition: &PreparedCellGroupTransfer, + sources: [u8; 32], +) -> Result<()> { + anyhow::ensure!(partition.children().len() == 2); + anyhow::ensure!( + partition + .children() + .iter() + .map(|child| child.root().cells()) + .sum::() + == 3 + ); + for child in partition.children() { + let genesis = child.genesis().context("missing final child genesis")?; + let marker = genesis.marker(); + let origin = marker.transfer_origin().context("child origin missing")?; + anyhow::ensure!( + !marker.is_active() + && marker.recovery_request().is_none() + && marker.route() == child.route() + && marker.root() == child.root() + && marker.position().index.get() == 1 + && marker.position().term.get() == 1 + && marker.membership_revision() == 1 + && origin.operation() == plan.operation().bytes() + && origin.plan() == plan.digest() + && origin.sealed_sources() == sources + ); + let mut encoded = serde_json::to_value(marker)?; + anyhow::ensure!(serde_json::from_value::(encoded.clone())? == *marker); + encoded["transfer"]["sealed_sources"] = serde_json::to_value([0u8; 32])?; + anyhow::ensure!(serde_json::from_value::(encoded).is_err()); + } + let source = partition.sources().sources()[0].recovery().manifest(); + for name in ["alpha", "beta", "empty"] { + let cell = CellId::new(name)?; + let (original, _) = source.pin_cell(&cell).context("sealed cell missing")?; + let mut assigned = partition.children().iter().filter_map(|child| { + child + .pin_cell(&cell) + .map(|(leaf, pin)| (child.route(), leaf, pin)) + }); + let (route, staged, pin) = assigned.next().context("staged cell missing")?; + anyhow::ensure!(assigned.next().is_none()); + anyhow::ensure!(route + .range + .contains(&cell_routing_key(route.history.namespace, &cell))); + let mut expected = original.clone(); + expected.last_mutation = LogPosition { + index: LogIndex::new(1), + term: Term::new(1), + }; + anyhow::ensure!(staged == &expected); + match pin { + CellImagePin::Materialized(pin) if !archives.rooted && name != "empty" => { + pin.verify_and_sync()? + } + CellImagePin::Extents(pin) if archives.rooted && name != "empty" => { + anyhow::ensure!( + pin.reference().root().bytes() + == match staged.version.image.commitment { + CellImageCommitment::Extents { root, .. } => root, + _ => anyhow::bail!("staged extent image commitment"), + } + ); + } + CellImagePin::Empty if name == "empty" => (), + _ => anyhow::bail!("wrong staged image representation"), + } + } + Ok(()) +} + +async fn verify_install_retry( + cluster: &Cluster, + plan: &DirectoryMutation, + child: &PreparedCellGroupTransferChild, + first: &GroupLog, +) -> Result<()> { + let directory = &cluster.directory; + let log = &cluster.voters[0].log; + let retry = cluster + .domain + .run(directory.install_transfer_voter(plan, child, log)) + .await??; + anyhow::ensure!(retry.numeric_key() == first.numeric_key()); + let ordinary = cluster + .domain + .run( + log.create_cell_group(CellGroupBootstrap { + route: child.route().clone(), + member: MemberId::new(1)?, + voters: [1, 2, 3] + .into_iter() + .map(MemberId::new) + .collect::, _>>()?, + learners: vec![], + }), + ) + .await?; + anyhow::ensure!(matches!(ordinary, Err(LogError::WrongIdentity))); + anyhow::ensure!(cluster + .domain + .run(directory.materialize_selected_history(child.route().history)) + .await? + .is_err()); + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_source_seal.rs b/crates/cellar/tests/support/cell_group_source_seal.rs new file mode 100644 index 000000000..6ac2747b9 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_source_seal.rs @@ -0,0 +1,445 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Exercise the exact rooted seal through the multiplexed three-voter owner. +use super::*; + +pub(crate) async fn source_seal_blocks_delayed_work() -> Result<()> { + let mut fixture = RuntimeCluster::new().await?; + let route = fixture.cluster.birth.route().clone(); + let scope = route.history; + attach_source_appliers(&mut fixture)?; + let directory = fixture.cluster.directory.clone(); + let log = fixture.cluster.voters[0].log.clone(); + let domain = &fixture.cluster.domain; + let plan = split_plan(&fixture.cluster).await?; + domain.run(directory.plan_transition(&plan)).await??; + domain + .run(directory.prepare_transition(&plan, [14; 32])) + .await??; + anyhow::ensure!(matches!( + domain + .run(directory.prepare_source_seal(&plan, scope, &log)) + .await?, + Err(CatalogError::Fenced) + )); + domain + .run(directory.authorize_transition_seal(&plan)) + .await??; + fixture.runtimes[0].campaign(scope).await?; + fixture + .drive(None) + .await + .context("initial source election")?; + let cell = history(&route, "before-seal")?; + let delayed = history(&route, "delayed-enrollment")?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(1);")?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let accepted = fixture.runtimes[0] + .propose(scope, request(&route, cell.clone(), 1, &cut)?) + .await?; + let permit = fixture + .cluster + .domain + .run(directory.prepare_source_seal(&plan, scope, &log)) + .await??; + anyhow::ensure!(matches!( + fixture.runtimes[0].seal_source(permit).await, + Err(RuntimeError::NotReady) + )); + fixture + .drive(Some(accepted.position.index)) + .await + .context("drain accepted enrollment")?; + let before = log + .cell_group(&scope)? + .group_manifest()? + .context("committed source image")?; + let permit = fixture + .cluster + .domain + .run(directory.prepare_source_seal(&plan, scope, &log)) + .await??; + let CellGroupSealOutcome::Accepted(seal) = fixture.runtimes[0].seal_source(permit).await? + else { + anyhow::bail!("seal must first enter the log"); + }; + let permit = fixture + .cluster + .domain + .run(directory.prepare_source_seal(&plan, scope, &log)) + .await??; + let CellGroupSealOutcome::Accepted(retry) = fixture.runtimes[0].seal_source(permit).await? + else { + anyhow::bail!("seal committed without a voter quorum"); + }; + anyhow::ensure!(retry.position == seal.position && retry.digest == seal.digest); + anyhow::ensure!(matches!( + fixture.runtimes[0] + .read_index(scope, b"late-read".to_vec()) + .await, + Err(RuntimeError::NotReady) + )); + anyhow::ensure!(matches!( + fixture.runtimes[0] + .propose(scope, request(&route, delayed.clone(), 2, &cut)?) + .await, + Err(RuntimeError::NotReady) + )); + tokio::time::timeout(Duration::from_secs(20), async { + while !fixture.applied[&scope] + .iter() + .all(|index| *index >= seal.position.index) + { + fixture.pump(&[], &mut BTreeMap::new()).await?; + tokio::task::yield_now().await; + } + Ok::<_, anyhow::Error>(()) + }) + .await + .context("replicate and apply the source seal")??; + let permit = fixture + .cluster + .domain + .run(directory.prepare_source_seal(&plan, scope, &log)) + .await??; + let CellGroupSealOutcome::Applied(applied) = fixture.runtimes[0].seal_source(permit).await? + else { + anyhow::bail!("committed seal was proposed a second time"); + }; + anyhow::ensure!(applied.position() == seal.position); + let deadline = fixture + .cluster + .domain + .run(async { cellar::asyncrt::mono_ms() }) + .await? + .checked_add(5_000) + .context("read deadline")?; + for (voter, runtime) in fixture.cluster.voters.iter().zip(&fixture.runtimes) { + let marker = voter + .log + .cell_group(&scope)? + .group_manifest()? + .context("sealed source marker")?; + anyhow::ensure!(marker.source_seal() == Some(&applied) && marker.root() == before.root()); + anyhow::ensure!(marker.root().cells() == 1 && !marker.is_active()); + let restored: GroupManifest = serde_json::from_slice(&serde_json::to_vec(&marker)?)?; + anyhow::ensure!(restored == marker); + anyhow::ensure!(matches!( + runtime + .read_authority( + scope, + cell.clone(), + CellReadConsistency::Linearizable, + deadline + ) + .await, + Err(RuntimeError::NotReady) + )); + anyhow::ensure!(matches!( + runtime.read_index(scope, b"sealed-read".to_vec()).await, + Err(RuntimeError::NotReady) + )); + } + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let snapshot = fixture.runtimes[0].capture_checkpoint(scope, &pool).await?; + anyhow::ensure!(snapshot.marker().source_seal() == Some(&applied)); + let checkpoint = snapshot.marker().clone(); + fixture.runtimes[0] + .publish_checkpoint(scope, snapshot.prepare_publication()?) + .await?; + let group = log.cell_group(&scope)?; + tokio::time::timeout(Duration::from_secs(20), async { + while group.checkpoint_manifest()?.as_ref() != Some(&checkpoint) { + fixture.pump(&[], &mut BTreeMap::new()).await?; + tokio::task::yield_now().await; + } + Ok::<_, anyhow::Error>(()) + }) + .await + .context("publish the exact sealed checkpoint")??; + drop(group); + drop(log); + let disk = std::mem::replace(&mut fixture.cluster._root, tempfile::tempdir()?); + let birth = fixture.cluster.birth.clone(); + let store = fixture.cluster.store.clone(); + let options = fixture.cluster.options.clone(); + fixture.close().await?; + let mut resumed = Box::pin(reopen_voters( + disk, + directory, + store, + birth, + options, + &checkpoint, + )) + .await?; + resumed.runtimes[0].campaign(scope).await?; + let convergence = tokio::time::timeout(Duration::from_secs(20), async { + loop { + resumed.pump(&[], &mut BTreeMap::new()).await?; + let mut converged = true; + for runtime in &resumed.runtimes { + let status = runtime.status(scope).await?; + converged &= status.leader == Some(1) + && status.term > applied.position().term + && status.applied.term == status.term + && status.applied.index == status.committed; + } + if converged { + break Ok::<_, anyhow::Error>(()); + } + // Deliver and persist the explicit campaign's messages before + // advancing election time. A tight tick loop races disk sync. + tokio::task::yield_now().await; + } + }) + .await; + if convergence.is_err() { + for (index, runtime) in resumed.runtimes.iter().enumerate() { + eprintln!( + "retained voter {}: {:?}", + index + 1, + runtime.status(scope).await + ); + } + } + convergence.context("establish a fresh committed term on the reopened voters")??; + let deadline = resumed + .cluster + .domain + .run(async { cellar::asyncrt::mono_ms() }) + .await? + .checked_add(5_000) + .context("retained read deadline")?; + for (voter, runtime) in resumed.cluster.voters.iter().zip(&resumed.runtimes) { + let status = runtime.status(scope).await?; + anyhow::ensure!(!status.ready); + let marker = voter + .log + .cell_group(&scope)? + .group_manifest()? + .context("retained source cut")?; + anyhow::ensure!(marker.source_seal() == Some(&applied) && marker.root() == before.root()); + anyhow::ensure!(matches!( + runtime + .read_authority( + scope, + cell.clone(), + CellReadConsistency::Linearizable, + deadline + ) + .await, + Err(RuntimeError::NotReady) + )); + anyhow::ensure!(matches!( + runtime + .propose(scope, request(&route, delayed.clone(), 2, &cut)?) + .await, + Err(RuntimeError::NotReady) + )); + } + let permit = resumed + .cluster + .domain + .run(resumed.cluster.directory.prepare_source_seal( + &plan, + scope, + &resumed.cluster.voters[0].log, + )) + .await??; + let CellGroupSealOutcome::Applied(retry) = resumed.runtimes[0].seal_source(permit).await? + else { + anyhow::bail!("retained seal replay appended an entry"); + }; + anyhow::ensure!(retry == applied); + resumed.close().await +} + +pub(super) fn attach_source_appliers(fixture: &mut RuntimeCluster) -> Result<()> { + let scope = fixture.cluster.birth.route().history; + for (n, voter) in fixture.cluster.voters.iter_mut().enumerate() { + let pool = apply_pool(&voter.log)?; + let applier = CellGroupApplier::new( + voter.log.cell_group(&scope)?, + voter.pages.take().context("source page owner")?, + checkpoint_receive::config( + voter.images.clone(), + voter.extents.clone(), + fixture.cluster._root.path(), + )?, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + )?; + fixture.appliers.insert((n, scope), (pool, applier)); + } + Ok(()) +} + +pub(super) async fn split_plan(cluster: &Cluster) -> Result { + let directory = &cluster.directory; + let domain = &cluster.domain; + let route = cluster.birth.route().clone(); + let scope = route.history; + let view = domain.run(directory.observe()).await??; + let (left, right) = route + .range + .split(cell_routing_key(scope.namespace, &CellId::new("pivot")?))?; + let mut children = Vec::new(); + for (id, range) in [(11, left), (12, right)] { + let child = directory.prepare_group_birth(CellGroupBirthSpec { + history: GroupHistory { + group: GroupId::new([id; 16])?, + incarnation: IncarnationId::new([id + 10; 16])?, + ..scope + }, + range, + epoch: RouteEpoch::new(2)?, + creation: ProposalId::new([id; 16])?, + voters: (1..=3) + .map(|n| { + Ok(InitialMember { + member: MemberId::new(n)?, + node: NodeId::new([n as u8; 16])?, + }) + }) + .collect::>()?, + learners: vec![], + })?; + domain.run(directory.reserve_group_birth(&child)).await??; + children.push(child.route().clone()); + } + Ok(directory.prepare_replacement( + &view, + ProposalId::new([13; 16])?, + std::slice::from_ref(&route), + &children, + )?) +} + +fn apply_pool(log: &ReplicaLog) -> Result { + Ok(CellGroupSqlPool::start( + log, + CellSqlLimits { + workers: 1, + queue_per_worker: 1, + max_pending_cells: 1, + group_pending_cells: 1, + node_bytes: 1024 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + }, + )?) +} + +async fn reopen_voters( + disk: tempfile::TempDir, + directory: RangeDirectory, + store: Arc, + birth: CellGroupBirth, + options: RaftRuntimeOptions, + checkpoint: &GroupManifest, +) -> Result { + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let scope = birth.route().history; + let mut voters = Vec::with_capacity(3); + let mut runtimes = Vec::with_capacity(3); + let mut events = Vec::with_capacity(3); + let mut appliers = BTreeMap::new(); + for n in 1..=3u8 { + let base = disk.path().join(format!("node-{n}")); + let filesystem = Arc::new(DirectFileSystem); + let log = domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + base.join("log"), + NodeId::new([n; 16])?, + DiskLineage::new([n + 10; 16])?, + ProcessGeneration::new([n + 90; 16])?, + LogOpenMode::Resume, + ), + filesystem.clone(), + )) + .await??; + let group = log.cell_group(&scope)?; + anyhow::ensure!(group.group_manifest()?.as_ref() == Some(checkpoint)); + if n == 1 { + anyhow::ensure!(group.checkpoint_manifest()?.as_ref() == Some(checkpoint)); + } + let permit = domain + .run(directory.bootstrap_group_member(&birth, &log)) + .await??; + let pages = ManifestPageStore::open( + &base.join("pages"), + filesystem.clone(), + 1024, + 16 * 1024 * 1024, + 16, + )?; + let images = GenerationStore::open(base.join("images"), filesystem, Default::default())?; + let config = checkpoint_receive::config(images.clone(), None, &base)?; + let (applier, root) = domain + .blocking(move || { + CellGroupApplier::recover( + group, + pages, + config, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + ) + })? + .await??; + anyhow::ensure!(root.marker() == checkpoint && root.pinned_images() == 1); + let (runtime, receiver) = CellGroupRuntime::start( + log.clone(), + RaftRuntimeOptions { + tick_interval_ms: None, + election_seed: Some([71; 32]), + ..options.clone() + }, + ) + .await?; + runtime.attach(permit, root).await?; + runtimes.push(runtime); + events.push(receiver); + appliers.insert((usize::from(n - 1), scope), (apply_pool(&log)?, applier)); + voters.push(Voter { + log, + owner: None, + pages: None, + images, + extents: None, + tree: ManifestTree::new(birth.route().clone(), ManifestLimits::default())?, + gate: fault::Gate::new(None), + paused: false, + held: None, + }); + } + let cluster = Cluster { + _root: disk, + domain, + directory, + store, + birth, + voters, + network: VecDeque::new(), + reads: vec![], + options, + }; + Ok(RuntimeCluster { + snapshot_sources: BTreeMap::new(), + snapshot_transfers: 0, + drop_snapshot_delivery: false, + retry_snapshot: false, + failed_snapshot: false, + retry_checkpoint: None, + transports: Vec::new(), + appliers, + cluster, + runtimes, + events, + applied: BTreeMap::from([(scope, vec![checkpoint.position().index; 3])]), + extra: BTreeMap::new(), + network: VecDeque::new(), + }) +} diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs new file mode 100644 index 000000000..f2119f99d --- /dev/null +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -0,0 +1,722 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! The actual cell SQL producer on domain workers with three persisted voters. + +use super::*; +use cellar_ltx::internal::{ + CacheLimits, ExtentId, ExtentManifest, ExtentRootReference, Geometry, ImageBinding, ImageRoot, + NodeExtentStore, EXTENT_BYTES, +}; +use cellar_ltx::stream_snapshot::{ScratchLimits, ScratchPool}; + +async fn accepted_sql( + runtime: &CellGroupRuntime, + prepared: &CellSqlPrepared, +) -> Result { + match prepared.submit(runtime).await? { + CellSqlSubmission::Accepted(accepted) => Ok(accepted), + CellSqlSubmission::Rejected(_) => anyhow::bail!("unexpected capacity rejection"), + } +} + +async fn accepted_private( + runtime: &CellGroupRuntime, + prepared: &CellPreparedCommand, +) -> Result { + match runtime.submit(prepared.clone()).await? { + CellSubmissionOutcome::Accepted(accepted) => Ok(accepted), + CellSubmissionOutcome::Rejected(_) => anyhow::bail!("unexpected private rejection"), + } +} + +fn cell_attempt(tag: u8) -> Result { + Ok(CommandAttempt { + request: CommandRequest { + id: CommandId::new(ClientId::new([81; 16])?, 1)?, + operation: "cell.increment".into(), + contract_digest: [6; 32], + request_digest: [9; 32], + durability: Durability::Quorum, + retry_until_ms: 1000, + }, + proposal: ProposalId::new([tag; 16])?, + preview: ImageId::new([tag; 16])?, + inputs: cellar::storage::CommandInputs { + now_unix_millis: 10, + random: [17; 32], + }, + }) +} + +impl RuntimeCluster { + async fn private_cell(&self, cell: &CellHistory) -> Result { + let authority = self.runtimes[0] + .execution_authority(self.cluster.birth.route().history, cell.clone()) + .await?; + let options = self.cell_options(cell).await?; + self.cluster + .domain + .blocking(move || match options { + CellSqlOptions::Materialized(o) => CellPrivateExecutor::open(o, authority), + CellSqlOptions::Extents(o) => { + let workspace = o + .images + .create_workspace( + cell_image_history_digest(&authority.cell().version.history), + RootPrivateOptions::workspace_bounds(o.limits)?, + ) + .map_err(|e| ExecutionError::Storage(e.into()))?; + CellPrivateExecutor::open_rooted( + RootPrivateOptions { + workspace, + host: o.host, + sqlite_vfs: o.sqlite_vfs, + images: o.images, + query_services: o.query_services, + application_digest: o.application_digest, + limits: o.limits, + cache_directory: o.cache_directory, + cache_pool: o.cache_pool, + cache: o.cache, + }, + authority, + ) + } + })? + .await? + .map_err(Into::into) + } + + async fn cell_options(&self, cell: &CellHistory) -> Result { + self.cell_options_at(cell, 0).await + } + + async fn cell_options_at(&self, cell: &CellHistory, node: usize) -> Result { + let options = PrivateOptions { + directory: self + .cluster + ._root + .path() + .join(format!("private-{node}-{}", cell.cell.as_str())), + host: cellar_ltx::LtxHost::new( + || 10, + |_| Ok(Duration::ZERO), + |path| async move { std::fs::read(path) }, + |job| async move { + job(); + Ok(()) + }, + ) + .with_filesystem(Arc::new(DirectFileSystem)), + sqlite_vfs: None, + images: self.cluster.voters[node].images.clone(), + query_services: cellar_ltx::generation::ReadServices { + now_unix_millis: Arc::new(|| 10), + fill_random: Arc::new(|bytes| bytes.fill(17)), + }, + application_digest: self.cluster.birth.route().schema_digest, + limits: ExecutionLimits::default(), + }; + let extents = self.cluster.voters[node].extents.clone(); + self.cluster + .domain + .blocking(move || -> Result<_> { + if let Some(store) = extents { + let rooted = CellSqlRootOptions { + host: options.host, + sqlite_vfs: None, + images: store, + query_services: options.query_services, + application_digest: options.application_digest, + limits: options.limits, + cache_directory: options.directory.with_extension("cache"), + cache_pool: ScratchPool::new(ScratchLimits { + max_bytes: 2 * EXTENT_BYTES, + max_files: 2, + })?, + cache: CacheLimits { + max_cached_bytes: EXTENT_BYTES, + max_cached_extents: 1, + }, + }; + Ok(CellSqlOptions::Extents(rooted)) + } else { + Ok(CellSqlOptions::Materialized(options)) + } + })? + .await? + } +} + +async fn enrolled_sql_cells( + rooted: bool, +) -> Result<(RuntimeCluster, GroupRoute, CellHistory, CellHistory)> { + Box::pin(enroll_sql_cells( + RuntimeCluster::new().await?, + rooted, + Arc::new(DirectFileSystem), + )) + .await +} + +async fn enroll_sql_cells( + mut fixture: RuntimeCluster, + rooted: bool, + filesystem: Arc, +) -> Result<(RuntimeCluster, GroupRoute, CellHistory, CellHistory)> { + let route = fixture.cluster.birth.route().clone(); + if rooted { + for (n, voter) in fixture.cluster.voters.iter_mut().enumerate() { + voter.extents = Some(NodeExtentStore::open( + fixture.cluster._root.path().join(format!("rooted-{n}")), + filesystem.clone(), + Default::default(), + )?); + } + } + + fixture.runtimes[0].campaign(route.history).await?; + fixture.drive(None).await?; + let a = history(&route, "sql-a")?; + let b = history(&route, "sql-b")?; + let c = history(&route, "sql-c")?; + for cell in [&a, &b, &c] { + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES (0);")?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let bytes = if rooted { + let manifest = ExtentManifest::new( + ImageBinding { + history: cell_image_history_digest(cell), + schema: route.schema_digest, + epoch: cut.result.epoch, + txid: cut.result.txid.0, + checksum: cut.result.checksum, + }, + Geometry::new(cut.result.page_size, cut.result.length)?, + &bytes + .chunks(EXTENT_BYTES as usize) + .map(ExtentId::of) + .collect::, _>>()?, + )?; + let encoded = request(&route, cell.clone(), 1, &cut)?; + let mut entry = CellGroupEntry::decode(&encoded, &route, GroupEntryLimits::default())?; + let GroupCommand::Enroll { image, .. } = &mut entry.command else { + anyhow::bail!("enrollment") + }; + image.commitment = ImageCommitment::Extents(ExtentRootReference::new( + manifest.root(), + manifest.encode()?.len().try_into()?, + )?); + entry.encode(&route, GroupEntryLimits::default())? + } else { + request(&route, cell.clone(), 1, &cut)? + }; + let proposed = fixture.runtimes[0].propose(route.history, bytes).await?; + fixture.drive(Some(proposed.position.index)).await?; + } + Ok((fixture, route, a, b)) +} + +pub(super) async fn cell_receipts(rooted: bool) -> Result<()> { + let (mut fixture, route, a, b) = enrolled_sql_cells(rooted).await?; + assert!(matches!( + fixture.runtimes[1] + .execution_authority(route.history, a.clone()) + .await, + Err(RuntimeError::NotLeader) + )); + let mut wrong = a.clone(); + wrong.incarnation = IncarnationId::new([99; 16])?; + assert!(fixture.runtimes[0] + .execution_authority(route.history, wrong) + .await + .is_err()); + let mut ea = fixture.private_cell(&a).await?; + let mut eb = fixture.private_cell(&b).await?; + let (ea_out, pa) = fixture + .cluster + .domain + .blocking(move || -> Result<_> { + let CellStageOutcome::Prepared(prepared) = ea.stage(cell_attempt(82)?, true, |tx| { + tx.execute("UPDATE value SET n=n+1", [])?; + Ok(b"a-result".to_vec()) + })? + else { + anyhow::bail!("new A receipt") + }; + ea.mark_submitted(prepared.proposal())?; + Ok((ea, prepared)) + })? + .await??; + ea = ea_out; + let (eb_out, pb) = fixture + .cluster + .domain + .blocking(move || -> Result<_> { + let CellStageOutcome::Prepared(prepared) = eb.stage(cell_attempt(83)?, true, |tx| { + tx.execute("UPDATE value SET n=n+7", [])?; + Ok(b"b-result".to_vec()) + })? + else { + anyhow::bail!("new B receipt") + }; + eb.mark_submitted(prepared.proposal())?; + Ok((eb, prepared)) + })? + .await??; + eb = eb_out; + // Both use the same client/sequence, but B's committed image cannot settle A. + let proposed = accepted_private(&fixture.runtimes[0], &pb).await?; + fixture.drive(Some(proposed.position.index)).await?; + let cb = fixture.runtimes[0] + .execution_authority(route.history, b.clone()) + .await?; + let (ea_out, eb_out) = fixture + .cluster + .domain + .blocking(move || -> Result<_> { + assert!(ea.finish(cb).is_err()); + assert!(matches!( + ea.stage(cell_attempt(84)?, true, |_| panic!("unresolved A callback")), + Err(ExecutionError::Busy) + )); + Ok((ea, eb)) + })? + .await??; + ea = ea_out; + eb = eb_out; + // A prepared before B committed remains valid because its own base is unchanged. + let proposed = accepted_private(&fixture.runtimes[0], &pa).await?; + fixture.drive(Some(proposed.position.index)).await?; + let ca = fixture.runtimes[0] + .execution_authority(route.history, a.clone()) + .await?; + let cb = fixture.runtimes[0] + .execution_authority(route.history, b.clone()) + .await?; + fixture + .cluster + .domain + .blocking(move || -> Result<()> { + let ra = ea.finish(ca)?; + let rb = eb.finish(cb)?; + assert_eq!(ra.receipt.result(), b"a-result"); + assert_eq!(rb.receipt.result(), b"b-result"); + assert_eq!(ra.version.sequence.get(), 2); + assert_eq!(rb.version.sequence.get(), 2); + assert_ne!(ra.version.history, rb.version.history); + ea.close()?; + eb.close()?; + Ok(()) + })? + .await??; + for (cell, expected, value) in [ + (&a, b"a-result".as_slice(), 1), + (&b, b"b-result".as_slice(), 7), + ] { + let mut executor = fixture.private_cell(cell).await?; + let expected = expected.to_vec(); + fixture + .cluster + .domain + .blocking(move || -> Result<()> { + let CellStageOutcome::Duplicate(receipt) = + executor.stage(cell_attempt(85)?, false, |_| { + panic!("receipt retry must not execute callback") + })? + else { + anyhow::bail!("missing duplicate") + }; + assert_eq!(receipt.receipt.result(), expected); + executor.close()?; + Ok(()) + })? + .await??; + for voter in &fixture.cluster.voters { + let marker = voter + .log + .cell_group(&route.history)? + .group_manifest()? + .context("selected root")?; + let leaf = voter + .tree + .get( + voter.pages.as_ref().context("fixture page owner")?, + marker.root(), + &cell.cell, + )? + .context("selected cell")?; + let query = match leaf.version.image.commitment { + CellImageCommitment::Materialized(digest) => { + let pin = voter.images.adopt( + image_id(cell, 2)?, + ImageState { + epoch: leaf.version.image.epoch, + txid: TXID(leaf.version.image.txid), + checksum: leaf.version.image.checksum, + digest, + length: leaf.geometry.length, + page_size: leaf.geometry.page_size, + }, + )?; + cellar_ltx::generation::open_immutable( + pin, + cellar_ltx::generation::ReadServices { + now_unix_millis: Arc::new(|| 10), + fill_random: Arc::new(|b| b.fill(17)), + }, + )? + } + CellImageCommitment::Extents { + root, + manifest_bytes, + } => { + let store = voter.extents.as_ref().context("rooted store")?; + let view = store.open_root(ExtentRootReference::new( + ImageRoot::from_bytes(root), + manifest_bytes, + )?)?; + let cache = fixture + .cluster + ._root + .path() + .join(format!("verify-{}", cell.cell.as_str())); + std::fs::create_dir_all(&cache)?; + cellar_ltx::internal::open_extent_image( + view.image( + cache, + ScratchPool::new(ScratchLimits { + max_bytes: 2 * EXTENT_BYTES, + max_files: 2, + })?, + CacheLimits { + max_cached_bytes: EXTENT_BYTES, + max_cached_extents: 1, + }, + )?, + cellar_ltx::generation::ReadServices { + now_unix_millis: Arc::new(|| 10), + fill_random: Arc::new(|b| b.fill(17)), + }, + )? + } + }; + let observed: i64 = query + .connection() + .query_row("SELECT n FROM value", [], |r| r.get(0))?; + assert_eq!(observed, value); + } + } + let stale = fixture.runtimes[0] + .execution_authority(route.history, a.clone()) + .await?; + let mut executor = fixture.private_cell(&a).await?; + let (entered, started) = tokio::sync::oneshot::channel(); + let (release, wait) = std::sync::mpsc::sync_channel(1); + let physical = fixture.cluster.domain.blocking(move || -> Result<()> { + let mut attempt = cell_attempt(86)?; + attempt.request.id = CommandId::new(ClientId::new([81; 16])?, 2)?; + let result = executor.stage(attempt, true, |tx| { + tx.execute("UPDATE value SET n=n+100", [])?; + let _ = entered.send(()); + wait.recv()?; + Ok(b"must-not-escape".to_vec()) + }); + assert!(matches!( + result, + Err(ExecutionError::Refused( + cellar_logic::consensus::CommandFailure::Unavailable + )) + )); + executor.close()?; + Ok(()) + })?; + tokio::time::timeout(Duration::from_secs(5), started).await??; + fixture.runtimes[0].close().await?; + assert!(stale.check().is_err()); + release.send(())?; + physical.await??; + let marker = fixture.cluster.voters[0] + .log + .cell_group(&route.history)? + .group_manifest()? + .context("selected root")?; + let leaf = fixture.cluster.voters[0] + .tree + .get( + fixture.cluster.voters[0] + .pages + .as_ref() + .context("fixture page owner")?, + marker.root(), + &a.cell, + )? + .context("selected cell")?; + assert_eq!(leaf.version, stale.cell().version); + drop(stale); + fixture.close().await +} + +#[path = "cell_group_sql_pool.rs"] +mod pool; + +pub(super) async fn pool_admission(rooted: bool) -> Result<()> { + pool::admission(rooted).await +} + +#[path = "cell_group_sql_reconcile.rs"] +mod reconciliation; + +#[path = "cell_group_sql_enrollment.rs"] +mod enrollment; + +pub(super) async fn initialize_cells(rooted: bool) -> Result<()> { + enrollment::concurrent(rooted).await +} + +pub(super) async fn owned_workspace_barriers() -> Result<()> { + enrollment::owned_workspace_barriers().await +} + +pub(super) async fn recover_initialization(rooted: bool) -> Result<()> { + enrollment::recovery(rooted).await +} + +pub(super) async fn reconcile_producers(rooted: bool) -> Result<()> { + reconciliation::reconcile(rooted).await +} + +pub(super) async fn reconcile_refusals(rooted: bool) -> Result<()> { + reconciliation::refusals(rooted).await +} + +#[path = "cell_group_sql_submission.rs"] +mod submission; + +pub(super) async fn submission_rejections(rooted: bool) -> Result<()> { + submission::rejections(rooted).await +} + +#[path = "cell_group_sql_reads.rs"] +mod reads; + +pub(crate) use reads::{ + coherent as coherent_reads, follower_linearizable, lifecycle as query_lifecycle, + recent_source_reads, +}; + +#[path = "cell_group_sql_apply.rs"] +pub(super) mod apply; +pub(crate) use apply::apply_images; + +#[path = "cell_group_sql_apply_faults.rs"] +mod apply_faults; +pub(crate) use apply_faults::{cancellation as apply_cancellation, failure as apply_failure}; + +#[path = "cell_group_sql_apply_recovery.rs"] +mod apply_recovery; +pub(crate) use apply_recovery::recover as recover_apply; + +#[path = "cell_group_sql_registered.rs"] +mod registered; +pub(crate) use registered::commands as registered_commands; + +#[path = "cell_group_sql_registered_queries.rs"] +mod registered_queries; +pub(crate) use registered_queries::queries as registered_queries; + +#[path = "cell_group_sql_dispatch.rs"] +mod dispatch; +pub(crate) use dispatch::routed as routed_queries; + +#[path = "cell_group_sql_command_dispatch.rs"] +mod command_dispatch; +pub(crate) use command_dispatch::commands as routed_commands; + +#[path = "cell_group_controller_archive.rs"] +#[allow(dead_code, unused_imports)] +mod archive_controller; +#[path = "cell_group_s3_commands.rs"] +#[allow(dead_code)] +mod s3_commands; +#[path = "cell_group_s3_response.rs"] +// Selected by the archive-controller target; sibling targets share this SQL fixture. +#[allow(dead_code)] +mod s3_response; +#[allow(unused_imports)] +pub(crate) use s3_commands::{ + archive_dependencies, archive_progress_wakes_commands, full_backend_outage, + output_during_apply, s3_commands, s3_deadline, s3_enrollment, s3_enrollment_deadline, + s3_missing_sibling, s3_namespace_minimum, s3_receipt_deadline, s3_receipt_failures, + s3_receipt_minimums, s3_receipt_observations, s3_remote_deadlines, s3_remote_outputs, + s3_remote_slow_delivery, +}; +#[allow(unused_imports)] +pub(crate) use s3_response::unproved_s3_responses; +#[path = "cell_group_controller.rs"] +mod controller; +#[path = "cell_group_idle_cache.rs"] +mod idle_cache; +pub(crate) use idle_cache::{cleanup_failure_retains_outcome, idle_cache_reopens_exact_base}; +#[path = "cell_group_effects.rs"] +mod effects; +#[path = "cell_group_outbox_effects.rs"] +mod outbox_effects; +#[allow(unused_imports)] +pub(crate) use archive_controller::{ + archive_checkpoint_frontier, archive_controller_publish, archive_controller_reconcile, + archived_groups_reclaim_independently, +}; +#[allow(unused_imports)] +pub(crate) use archive_controller::{ + archive_controller_publish_store, archive_controller_reconcile_store, +}; +pub(crate) use effects::{alarms_without_clients, hot_and_cold_alarms, shutdown_joins_alarm}; +pub(crate) use outbox_effects::{ + identity_across_bindings as effect_identity_across_bindings, + leader_change as effect_leader_change, lost_reply as effect_lost_reply, + s3_lost_reply as effect_s3_lost_reply, shutdown as effect_delivery_shutdown, +}; +#[path = "cell_group_controller_commands.rs"] +mod controller_commands; +pub(crate) use controller::{ + controller_apply_faults, controller_close, controller_owners, controller_replication, +}; +pub(crate) use controller_commands::{ + background_admission, command_admission_starts_before_poll, owned_command_failure, + owned_commands, +}; + +pub(super) async fn commands_over_tcp(rooted: bool) -> Result<()> { + Box::pin(command_dispatch::commands_with_transport(rooted, true)).await +} + +#[path = "cell_group_controller_enrollment.rs"] +mod controller_enrollment; +pub(crate) use controller_enrollment::{ + enrollment_requires_registration, owned_enrollment, owned_enrollment_failure, +}; + +#[path = "cell_group_remote_queries.rs"] +mod remote_queries; +pub(crate) use remote_queries::{remote_follower_reads, remote_queries, remote_query_lifecycle}; + +#[path = "cell_group_query_origins.rs"] +mod query_origins; +pub(crate) use query_origins::{ + nonmember_controller_queries, nonmember_queries, nonmember_query_revocation, +}; + +#[path = "cell_group_dispatch_cache_execution.rs"] +mod dispatch_cache_execution; +pub(crate) use dispatch_cache_execution::cached_dispatch_execution; + +#[path = "cell_group_remote_commands.rs"] +mod remote_commands; +pub(crate) use remote_commands::{remote_command_lifecycle, remote_commands}; + +#[path = "cell_group_remote_enrollment.rs"] +mod remote_enrollment; +pub(crate) use remote_enrollment::{remote_enrollment, remote_enrollment_lifecycle}; + +#[path = "cell_group_receipt_lookup.rs"] +mod receipt_lookup; +pub(crate) use receipt_lookup::{ + receipt_lookup_cancelled, receipt_lookup_ingress, receipt_lookup_pending, +}; + +#[path = "cell_group_registration.rs"] +mod registration; +pub(crate) use registration::{ + registration_close, registration_live, registration_recovery, registration_refusals, +}; + +#[path = "cell_group_quarantine.rs"] +mod quarantine; +pub(crate) use quarantine::{ + group_quarantine_retains_producer_and_serves_healthy_group, + shared_engine_sync_failure_still_stops_all_groups, +}; + +#[path = "cell_group_provision.rs"] +mod provision; +pub(crate) use provision::{ + provisioning_coalesces_and_serves, provisioning_joins_cancelled_close, + provisioning_recovers_selected_roots, provisioning_refusals_preserve_healthy_group, + provisioning_retries_retained_owner, +}; + +#[path = "cell_group_controller_checkpoints.rs"] +mod controller_checkpoints; +pub(crate) use controller_checkpoints::{ + automatic_checkpoints, checkpoint_during_apply, controller_checkpoint_shutdown, +}; + +#[path = "cell_group_completion_pressure.rs"] +mod completion_pressure; +pub(crate) use completion_pressure::checkpoint_completion_pressure; + +#[path = "cell_group_repair_load.rs"] +mod repair_load; +pub(crate) use repair_load::repair_budget_under_load; + +#[path = "cell_group_checkpoint_fairness.rs"] +mod checkpoint_fairness; +pub(crate) use checkpoint_fairness::concurrent_checkpoints; + +#[path = "cell_group_membership.rs"] +mod membership; +pub(crate) use membership::{ + membership_delivery_fence, membership_lifecycle, membership_physical_delivery_fence, + membership_quorum_floor, membership_quorum_followers, membership_sync, +}; + +#[path = "cell_group_seed.rs"] +mod seed; +pub(crate) use seed::{seed_minority, seed_source}; + +#[path = "cell_group_seed_install.rs"] +mod seed_install; +pub(crate) use seed_install::seed_install; + +#[path = "cell_group_seed_receiver.rs"] +mod seed_receiver; +pub(crate) use seed_receiver::seed_receiver_faults; + +#[path = "cell_group_seed_persistence.rs"] +mod seed_persistence; +pub(crate) use seed_persistence::{seed_persistence, seed_transfer_revocation}; + +#[path = "cell_group_seed_crash.rs"] +mod seed_crash; +pub(crate) use seed_crash::{seed_crash, seed_process_fixture}; + +#[path = "cell_group_crash_fixture.rs"] +mod crash_fixture; +#[path = "cell_group_membership_crash.rs"] +mod membership_crash; +pub(crate) use membership_crash::{membership_crash, membership_process_fixture}; + +#[path = "cell_group_node_reclaim.rs"] +mod node_reclaim; +pub(crate) use node_reclaim::shared_inventory; + +pub(crate) use seed_crash::seed_retry; + +#[path = "cell_group_log_retention.rs"] +mod log_retention; +pub(super) use log_retention::retained_log_admission; + +#[allow(unused_imports)] +pub(crate) use archive_controller::physical_log_reclamation; + +#[path = "cell_group_host.rs"] +mod namespace_host; +pub(crate) use namespace_host::{ + many_cells as host_many_cells, query_shutdown as host_query_shutdown, + registration_and_deadline as host_registration_and_deadline, +}; + +#[path = "cell_group_query_routing.rs"] +mod query_routing; +pub(crate) use query_routing::controller_routed_queries; diff --git a/crates/cellar/tests/support/cell_group_sql_apply.rs b/crates/cellar/tests/support/cell_group_sql_apply.rs new file mode 100644 index 000000000..8d03981ee --- /dev/null +++ b/crates/cellar/tests/support/cell_group_sql_apply.rs @@ -0,0 +1,230 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! The production image adapter, exercised with actual committed three-voter work. +use super::*; + +pub(super) fn pool(fixture: &RuntimeCluster, node: usize) -> Result { + Ok(CellGroupSqlPool::start( + &fixture.cluster.voters[node].log, + CellSqlLimits { + workers: 1, + queue_per_worker: 1, + max_pending_cells: 1, + group_pending_cells: 1, + node_bytes: 1024 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + }, + )?) +} + +pub(super) async fn install( + fixture: &mut RuntimeCluster, + cell: &CellHistory, + node: usize, +) -> Result<()> { + let route = fixture.cluster.birth.route().clone(); + let config = match fixture.cell_options_at(cell, node).await? { + CellSqlOptions::Materialized(o) => CellGroupApplyImages::Materialized { + images: o.images, + services: o.query_services, + }, + CellSqlOptions::Extents(o) => CellGroupApplyImages::Extents { + images: o.images, + services: o.query_services, + limits: Default::default(), + cache_directory: o.cache_directory, + cache_pool: o.cache_pool, + cache: o.cache, + }, + }; + let pool = pool(fixture, node)?; + let voter = &mut fixture.cluster.voters[node]; + let applier = CellGroupApplier::new( + voter.log.cell_group(&route.history)?, + voter.pages.take().context("page owner")?, + config, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + )?; + fixture + .appliers + .insert((node, route.history), (pool, applier)); + Ok(()) +} + +pub(in crate::runtime_fixture) fn enrollment_bytes( + route: &GroupRoute, + cell: &CellHistory, + bytes: &[u8], + rooted: bool, +) -> Result> { + let cut = images::Cut::new(&[], bytes, 0, true, false); + let encoded = request(route, cell.clone(), 1, &cut)?; + let mut entry = CellGroupEntry::decode(&encoded, route, GroupEntryLimits::default())?; + if rooted { + let manifest = ExtentManifest::new( + ImageBinding { + history: cell_image_history_digest(cell), + schema: route.schema_digest, + epoch: cut.result.epoch, + txid: cut.result.txid.0, + checksum: cut.result.checksum, + }, + Geometry::new(cut.result.page_size, cut.result.length)?, + &bytes + .chunks(EXTENT_BYTES as usize) + .map(ExtentId::of) + .collect::, _>>()?, + )?; + let GroupCommand::Enroll { image, .. } = &mut entry.command else { + anyhow::bail!("enroll") + }; + image.commitment = ImageCommitment::Extents(ExtentRootReference::new( + manifest.root(), + manifest.encode()?.len().try_into()?, + )?); + } + Ok(entry.encode(route, GroupEntryLimits::default())?) +} + +pub(crate) async fn apply_images(rooted: bool) -> Result<()> { + let (mut fixture, route, a, b) = enrolled_sql_cells(rooted).await?; + for node in 0..3 { + install(&mut fixture, &a, node).await?; + } + let (pool, applier) = fixture.appliers[&(0, route.history)].clone(); + let foreign = fixture.appliers[&(1, route.history)].0.clone(); + + // Enroll through the production adapter on all voters, including scheduling + // summaries derived from real SQLite state rather than fixture defaults. + let cell = history(&route, "effects")?; + // Keep the lane/sequence projection used by real outbox discovery. The + // later A row is immediately due but cannot pass its sleeping lane head. + let bytes = images::image( + None, + "CREATE TABLE value(n); INSERT INTO value VALUES(7); + CREATE TABLE __sys_raft_alarms(due); INSERT INTO __sys_raft_alarms VALUES(90),(30); + CREATE TABLE __sys_raft_outbox(lane,sequence,next_at); + INSERT INTO __sys_raft_outbox VALUES + ('A',x'0000000000000001',80), + ('A',x'0000000000000002',0), + ('B',x'0000000000000001',40);", + )?; + let encoded = enrollment_bytes(&route, &cell, &bytes, rooted)?; + let accepted = fixture.runtimes[0].propose(route.history, encoded).await?; + fixture.drive(Some(accepted.position.index)).await?; + for runtime in &fixture.runtimes { + let state = runtime + .reconciliation_authority(route.history, cell.clone()) + .await?; + assert_eq!(state.cell().effects.next_alarm_unix_millis, Some(30)); + assert_eq!(state.cell().effects.next_outbox_unix_millis, Some(40)); + assert_eq!(state.cell().effects.pending_outbox, 3); + } + + let options = fixture.cell_options(&a).await?; + let authority = fixture.runtimes[0] + .execution_authority(route.history, a.clone()) + .await?; + let CellSqlOutcome::Prepared(sql) = pool + .stage(options, authority, cell_attempt(231)?, true, |tx| { + tx.execute("UPDATE value SET n=n+11", [])?; + Ok(b"applied".to_vec()) + }) + .await? + else { + anyhow::bail!("SQL preparation") + }; + // Business admission is full. Its retained producer cannot prevent apply. + assert_eq!(pool.usage()?.active_cells, 1); + let other = fixture.runtimes[0] + .execution_authority(route.history, b.clone()) + .await?; + assert!(matches!( + pool.stage( + fixture.cell_options(&b).await?, + other, + cell_attempt(232)?, + true, + |_| panic!("saturated business admission") + ) + .await, + Err(ExecutionError::Refused( + cellar_logic::consensus::CommandFailure::Overloaded + )) + )); + pool.mark_submitted(a.clone(), sql.proposal()).await?; + let accepted = accepted_sql(&fixture.runtimes[0], &sql).await?; + let mut held = BTreeMap::new(); + tokio::time::timeout(Duration::from_secs(5), async { + loop { + fixture.pump(&[route.history], &mut held).await?; + if held.contains_key(&route.history) { + return Ok::<_, anyhow::Error>(()); + } + tokio::task::yield_now().await; + } + }) + .await??; + let work = held.remove(&route.history).context("committed work")?; + let before = work.base.marker().clone(); + let mut forged = work.clone(); + forged.entry.data = vec![1, 2, 3].into(); + assert!(pool.prepare_apply(&applier, forged).await.is_err()); + assert!(foreign.prepare_apply(&applier, work.clone()).await.is_err()); + assert_eq!(pool.usage()?.apply_retained_operations, 0); + assert_eq!(foreign.usage()?.apply_retained_operations, 0); + + let prepared = pool.prepare_apply(&applier, work.clone()).await?; + assert_eq!(prepared.manifest().position(), accepted.position); + let duplicate = pool.prepare_apply(&applier, work.clone()).await?; + assert_eq!(prepared.manifest(), duplicate.manifest()); + assert_eq!(pool.usage()?.apply_retained_operations, 2); + drop(duplicate); + let retained = prepared.clone(); + assert_eq!(pool.usage()?.apply_retained_operations, 1); + assert!(pool.usage()?.apply_retained_bytes > 0); + assert_eq!( + fixture.cluster.voters[0] + .log + .cell_group(&route.history)? + .group_manifest()? + .as_ref(), + Some(&before) + ); + + // Completion only queues publication. Retention outlives the caller and a + // blocked physical log sync; the old root remains selected until sync ends. + let gate = fixture.cluster.voters[0].gate.clone(); + let release = gate.arm(fault::HOLD_SYNC); + prepared.complete(&fixture.runtimes[0]).await?; + tokio::time::timeout(Duration::from_secs(5), gate.entered.notified()).await?; + drop(retained); + assert_eq!(pool.usage()?.apply_retained_operations, 1); + assert_eq!( + fixture.cluster.voters[0] + .log + .cell_group(&route.history)? + .group_manifest()? + .as_ref(), + Some(&before) + ); + drop(release); + fixture.drive(Some(accepted.position.index)).await?; + assert_eq!(pool.usage()?.apply_retained_operations, 0); + assert!(pool.prepare_apply(&applier, work).await.is_err()); + for runtime in &fixture.runtimes { + let state = runtime + .reconciliation_authority(route.history, a.clone()) + .await?; + assert_eq!(state.cell().version.sequence.get(), 2); + assert_eq!(state.cell().witness.position, accepted.position); + assert_eq!(state.cell().witness.proposal, sql.proposal()); + } + drop(sql); + drop(pool); + drop(foreign); + drop(applier); + fixture.close().await +} diff --git a/crates/cellar/tests/support/cell_group_sql_apply_faults.rs b/crates/cellar/tests/support/cell_group_sql_apply_faults.rs new file mode 100644 index 000000000..688285069 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_sql_apply_faults.rs @@ -0,0 +1,183 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Physical preparation ownership and refusal at real image/manifest barriers. +use super::*; +use std::sync::atomic::Ordering; + +#[path = "raft_ltx_fault.rs"] +#[allow(dead_code)] +pub(super) mod physical; + +async fn pending( + rooted: bool, + manifest: bool, +) -> Result<( + RuntimeCluster, + Arc, + CellGroupApply, + CellHistory, +)> { + let root = tempfile::tempdir()?; + let directory = if manifest { + root.path().join("node-1/pages") + } else if rooted { + root.path().join("rooted-0") + } else { + root.path().join("node-1/images") + }; + let faults = physical::Faults::new(directory, None); + let filesystem: Arc = Arc::new(physical::FaultFileSystem { + faults: faults.clone(), + }); + let cluster = + Cluster::with_storage(RaftRuntimeOptions::default(), root, filesystem.clone()).await?; + let fixture = RuntimeCluster::from_cluster(cluster, 256, 256, 1_000_000).await?; + let (mut fixture, route, a, _) = enroll_sql_cells(fixture, rooted, filesystem).await?; + apply::install(&mut fixture, &a, 0).await?; + let cell = history(&route, "faulted-apply")?; + // Raw replicated enrollment ensures no leader SQL preview has already + // prepared this result, including in the content-addressed extent store. + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(11);")?; + fixture.runtimes[0] + .propose( + route.history, + apply::enrollment_bytes(&route, &cell, &bytes, rooted)?, + ) + .await?; + let mut held = BTreeMap::new(); + tokio::time::timeout(Duration::from_secs(5), async { + loop { + fixture.pump(&[route.history], &mut held).await?; + if held.contains_key(&route.history) { + return Ok::<_, anyhow::Error>(()); + } + tokio::task::yield_now().await; + } + }) + .await??; + Ok(( + fixture, + faults, + held.remove(&route.history).context("committed work")?, + cell, + )) +} + +pub(crate) async fn cancellation(rooted: bool) -> Result<()> { + for manifest in [false, true] { + let (mut fixture, faults, work, cell) = Box::pin(pending(rooted, manifest)).await?; + let history = fixture.cluster.birth.route().history; + let before = work.base.marker().clone(); + let position = work.ticket.position(); + let (pool, applier) = fixture.appliers.remove(&(0, history)).context("applier")?; + let (entered, started) = tokio::sync::oneshot::channel(); + let (release, wait) = std::sync::mpsc::sync_channel(1); + *faults.before_directory_sync.lock().unwrap() = Some(Box::new(move || { + let _ = entered.send(()); + wait.recv_timeout(Duration::from_secs(10)) + .expect("release physical apply barrier"); + })); + let caller = pool.clone(); + let worker = applier.clone(); + let input = work.clone(); + let waiter = fixture + .cluster + .domain + .spawn(async move { caller.prepare_apply(&worker, input).await })?; + tokio::time::timeout(Duration::from_secs(5), started).await??; + waiter.abort(); + assert!(waiter.await.is_err()); + assert_eq!(pool.usage()?.apply_retained_operations, 1); + assert!(pool.usage()?.apply_retained_bytes > 0); + assert_eq!(pool.usage()?.completion_reservations, 1); + assert_eq!( + fixture.cluster.voters[0] + .log + .cell_group(&history)? + .group_manifest()? + .as_ref(), + Some(&before) + ); + // Cancel a close waiter too. Cleanup still owns the entered worker and + // the exclusive SQL-pool lease until the physical barrier is released. + assert!( + tokio::time::timeout(Duration::from_millis(30), pool.close()) + .await + .is_err() + ); + assert!(apply::pool(&fixture, 0).is_err()); + assert_eq!(pool.usage()?.running_workers, 1); + assert_eq!(pool.usage()?.apply_retained_operations, 1); + release.send(())?; + tokio::time::timeout(Duration::from_secs(5), pool.close()).await??; + let usage = pool.usage()?; + assert_eq!(usage.running_workers, 0); + assert_eq!(usage.apply_retained_operations, 0); + assert_eq!(usage.apply_retained_bytes, 0); + assert_eq!(usage.completion_reservations, 0); + assert_eq!( + fixture.cluster.voters[0] + .log + .cell_group(&history)? + .group_manifest()? + .as_ref(), + Some(&before) + ); + drop(pool); + // Shutdown did not publish a result or settle this committed entry. + // A fresh pool can repeat exact preparation, then the owner publishes it. + let replacement = apply::pool(&fixture, 0)?; + replacement + .prepare_apply(&applier, work) + .await? + .complete(&fixture.runtimes[0]) + .await?; + fixture.drive(Some(position.index)).await?; + for runtime in &fixture.runtimes { + let state = runtime + .reconciliation_authority(history, cell.clone()) + .await?; + assert_eq!(state.cell().version.sequence.get(), 1); + assert_eq!(state.cell().witness.position, position); + } + replacement.close().await?; + drop(replacement); + drop(applier); + fixture.close().await?; + } + Ok(()) +} + +pub(crate) async fn failure(rooted: bool) -> Result<()> { + for manifest in [false, true] { + let (fixture, faults, work, _) = Box::pin(pending(rooted, manifest)).await?; + let history = fixture.cluster.birth.route().history; + let before = work.base.marker().clone(); + let (pool, applier) = &fixture.appliers[&(0, history)]; + faults + .mode + .store(physical::FAIL_DIRECTORY_SYNC, Ordering::Release); + assert!(pool.prepare_apply(applier, work).await.is_err()); + assert_eq!( + faults.mode.load(Ordering::Acquire), + 0, + "fault must reach actual preparation I/O" + ); + assert_eq!(pool.usage()?.apply_retained_operations, 0); + assert_eq!(pool.usage()?.apply_retained_bytes, 0); + assert_eq!(pool.usage()?.completion_reservations, 0); + assert_eq!( + fixture.cluster.voters[0] + .log + .cell_group(&history)? + .group_manifest()? + .as_ref(), + Some(&before) + ); + let status = fixture.runtimes[0].status(history).await?; + assert_eq!(status.applied, before.position()); + assert!(status.committed > before.position().index); + fixture.close().await?; + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_sql_apply_recovery.rs b/crates/cellar/tests/support/cell_group_sql_apply_recovery.rs new file mode 100644 index 000000000..64948ae41 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_sql_apply_recovery.rs @@ -0,0 +1,286 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Cold selected-root verification and replay of committed, unpublished work. +use super::*; + +pub(crate) async fn recover(rooted: bool) -> Result<()> { + let mut fixture = RuntimeCluster::new().await?; + let route = fixture.cluster.birth.route().clone(); + if rooted { + for (n, voter) in fixture.cluster.voters.iter_mut().enumerate() { + voter.extents = Some(NodeExtentStore::open( + fixture.cluster._root.path().join(format!("rooted-{n}")), + Arc::new(DirectFileSystem), + Default::default(), + )?); + } + } + fixture.runtimes[0].campaign(route.history).await?; + fixture.drive(None).await?; + let a = history(&route, "recover-a")?; + let b = history(&route, "recover-b")?; + apply::install(&mut fixture, &a, 0).await?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(29);")?; + let accepted_a = fixture.runtimes[0] + .propose( + route.history, + apply::enrollment_bytes(&route, &a, &bytes, rooted)?, + ) + .await?; + fixture.drive(Some(accepted_a.position.index)).await?; + let selected = fixture.cluster.voters[0] + .log + .cell_group(&route.history)? + .group_manifest()? + .context("selected A")?; + let accepted_b = fixture.runtimes[0] + .propose( + route.history, + apply::enrollment_bytes(&route, &b, &bytes, rooted)?, + ) + .await?; + let mut held = BTreeMap::new(); + tokio::time::timeout(Duration::from_secs(5), async { + loop { + fixture.pump(&[route.history], &mut held).await?; + if held.contains_key(&route.history) { + return Ok::<_, anyhow::Error>(()); + } + tokio::task::yield_now().await; + } + }) + .await??; + let work = held.remove(&route.history).context("unpublished B")?; + let (pool, applier) = &fixture.appliers[&(0, route.history)]; + let prepared = pool.prepare_apply(applier, work).await?; + assert_eq!(prepared.manifest().position(), accepted_b.position); + drop(prepared); + assert_eq!( + fixture.cluster.voters[0] + .log + .cell_group(&route.history)? + .group_manifest()? + .as_ref(), + Some(&selected) + ); + let directory = fixture.cluster.directory.clone(); + let birth = fixture.cluster.birth.clone(); + let disk = std::mem::replace(&mut fixture.cluster._root, tempfile::tempdir()?); + let closed_log = fixture.cluster.voters[0].log.clone(); + let closed_group = closed_log.cell_group(&route.history)?; + fixture.close().await?; + assert!(closed_log.cell_groups().is_err()); + assert!(closed_group.group_manifest().is_err()); + + // Retained closed metadata must not retain physical image/page owners. + // Keep both handles alive throughout fresh recovery on these same paths. + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let base = disk.path().join("node-1"); + let log = domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + base.join("log"), + NodeId::new([1; 16])?, + DiskLineage::new([11; 16])?, + ProcessGeneration::new([31; 16])?, + LogOpenMode::Resume, + ), + Arc::new(fault::FaultFileSystem { + gate: fault::Gate::new(None), + }), + )) + .await??; + let permit = domain + .run(directory.bootstrap_group_member(&birth, &log)) + .await??; + let group = log.cell_group(&route.history)?; + let image_directory = if rooted { + disk.path().join("rooted-0") + } else { + base.join("images") + }; + let config = open_images(&image_directory, rooted)?; + assert!(closed_log.cell_groups().is_err()); + assert!(closed_group.group_manifest().is_err()); + let pages = ManifestPageStore::open( + &base.join("pages"), + Arc::new(DirectFileSystem), + 1024, + 16 * 1024 * 1024, + 16, + )?; + let recovering = group.clone(); + let (applier, recovered) = domain + .blocking(move || { + CellGroupApplier::recover( + recovering, + pages, + config, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + ) + })? + .await??; + assert_eq!(recovered.marker(), &selected); + assert_eq!(recovered.pinned_images(), 1); + assert_eq!( + recovered.pin_cell(&a.cell).context("A")?.0.witness.position, + accepted_a.position + ); + assert!( + recovered.pin_cell(&b.cell).is_none(), + "unselected files became authoritative" + ); + let (runtime, mut events) = CellGroupRuntime::start( + log.clone(), + RaftRuntimeOptions { + tick_interval_ms: None, + election_seed: Some([71; 32]), + ..Default::default() + }, + ) + .await?; + runtime.attach(permit, recovered).await?; + let pool = CellGroupSqlPool::start( + &log, + CellSqlLimits { + workers: 1, + queue_per_worker: 2, + max_pending_cells: 2, + group_pending_cells: 2, + node_bytes: 512 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + }, + )?; + // Replay durable committed B with only this node present. It is apply, not + // renewed serving authority; messages are deliberately not delivered. + tokio::time::timeout(Duration::from_secs(5), async { + loop { + let event = events.recv().await.context("runtime event")?; + match event.effect().map_err(|e| anyhow::anyhow!(e.to_string()))? { + CellGroupEffect::Apply(_) => { + let work = event.retain_apply().context("replay")?; + pool.prepare_apply(&applier, work) + .await? + .complete(&runtime) + .await?; + } + CellGroupEffect::Applied { manifest, .. } + if manifest.position() == accepted_b.position => + { + break + } + CellGroupEffect::Message(_) | CellGroupEffect::Applied { .. } => (), + _ => anyhow::bail!("unexpected recovery event"), + } + } + Ok::<_, anyhow::Error>(()) + }) + .await??; + assert_eq!( + group.group_manifest()?.context("B marker")?.root().cells(), + 2 + ); + assert!(!runtime.status(route.history).await?.ready); + assert!(runtime + .execution_authority(route.history, b.clone()) + .await + .is_err()); + pool.close().await?; + runtime.close().await?; + drop(events); + drop(runtime); + drop(pool); + drop(applier); + let selected = group.group_manifest()?.context("replayed root")?; + // The open engine cache pins its published page/image closure too. A real + // restart closes that owner before reopening the same directories. + drop(group); + log.close().await?; + drop(log); + domain.shutdown().await; + + // Missing physical closure must fail even though the engine marker remains + // intact. Only this test's disposable directory is moved. + std::fs::rename(&image_directory, image_directory.with_extension("saved"))?; + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let log = domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + base.join("log"), + NodeId::new([1; 16])?, + DiskLineage::new([11; 16])?, + ProcessGeneration::new([32; 16])?, + LogOpenMode::Resume, + ), + Arc::new(fault::FaultFileSystem { + gate: fault::Gate::new(None), + }), + )) + .await??; + let group = log.cell_group(&route.history)?; + let config = open_images(&image_directory, rooted)?; + let pages = ManifestPageStore::open( + &base.join("pages"), + Arc::new(DirectFileSystem), + 1024, + 16 * 1024 * 1024, + 16, + )?; + let recovering = group.clone(); + assert!(domain + .blocking(move || CellGroupApplier::recover( + recovering, + pages, + config, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100 + ))? + .await? + .is_err()); + assert_eq!(group.group_manifest()?.as_ref(), Some(&selected)); + drop(group); + log.close().await?; + drop(log); + domain.shutdown().await; + drop(disk); + Ok(()) +} + +fn open_images(directory: &std::path::Path, rooted: bool) -> Result { + let services = cellar_ltx::generation::ReadServices { + now_unix_millis: Arc::new(|| 10), + fill_random: Arc::new(|bytes| bytes.fill(17)), + }; + Ok(if rooted { + CellGroupApplyImages::Extents { + images: NodeExtentStore::open( + directory.into(), + Arc::new(DirectFileSystem), + Default::default(), + )?, + limits: Default::default(), + services, + cache_directory: directory.with_extension("cache"), + cache_pool: ScratchPool::new(ScratchLimits { + max_bytes: 2 * EXTENT_BYTES, + max_files: 2, + })?, + cache: CacheLimits { + max_cached_bytes: EXTENT_BYTES, + max_cached_extents: 1, + }, + } + } else { + CellGroupApplyImages::Materialized { + images: GenerationStore::open( + directory.into(), + Arc::new(DirectFileSystem), + Default::default(), + )?, + services, + } + }) +} diff --git a/crates/cellar/tests/support/cell_group_sql_command_dispatch.rs b/crates/cellar/tests/support/cell_group_sql_command_dispatch.rs new file mode 100644 index 000000000..6058ef5de --- /dev/null +++ b/crates/cellar/tests/support/cell_group_sql_command_dispatch.rs @@ -0,0 +1,369 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use std::sync::{ + atomic::{AtomicUsize, Ordering}, + Mutex, +}; + +fn limits() -> CellRoutingLimits { + CellRoutingLimits { + timeout: Duration::from_secs(15), + max_attempts: 4, + } +} + +fn request( + class: &RaftClass, + sequence: u64, + retry: u64, + input: &[u8], +) -> Result { + Ok(CellCommandRequest { + id: CommandId::new(ClientId::new([211; 16])?, sequence)?, + contract: class.contract("Add").context("Add")?, + input: input.to_vec(), + durability: Durability::Quorum, + retry_until_ms: retry, + }) +} + +pub(super) async fn accepted(invocation: &mut CellCommandInvocation) -> Result { + match invocation.submit().await? { + CellCommandSubmission::Accepted(accepted) => Ok(accepted), + CellCommandSubmission::Rejected => anyhow::bail!("unexpected sealed rejection"), + } +} + +pub(super) async fn committed( + invocation: &mut CellCommandInvocation, +) -> Result { + match invocation.settle().await? { + CellCommandSettlement::Committed(receipt) => Ok(receipt), + _ => anyhow::bail!("expected committed receipt"), + } +} + +pub(crate) async fn commands(rooted: bool) -> Result<()> { + Box::pin(commands_with_transport(rooted, false)).await +} + +pub(crate) async fn commands_with_transport(rooted: bool, tcp: bool) -> Result<()> { + let calls = Arc::new(AtomicUsize::new(0)); + let captured = Arc::new(Mutex::new(Vec::new())); + let count = calls.clone(); + let inputs = captured.clone(); + let (entered, started) = tokio::sync::oneshot::channel(); + let entered = Mutex::new(Some(entered)); + let (release, wait) = std::sync::mpsc::sync_channel(1); + let wait = Mutex::new(wait); + let class = Arc::new( + RaftClass::new("Counter", 1, [91; 32])? + .command("Add", [92; 32], move |context, input| { + count.fetch_add(1, Ordering::SeqCst); + inputs.lock().unwrap().push(( + context.identity().cell.clone(), + context.command(), + context.inputs().now_unix_millis, + context.inputs().random, + )); + if input == b"hold" && entered.lock().unwrap().is_some() { + entered + .lock() + .unwrap() + .take() + .unwrap() + .send(()) + .map_err(|_| anyhow::anyhow!("entered"))?; + wait.lock().unwrap().recv_timeout(Duration::from_secs(10))?; + } + context.sql().execute("UPDATE value SET n=n+1", [])?; + Ok(context.identity().cell.as_str().as_bytes().to_vec()) + })? + .query("Get", [93; 32], |context, _| { + let n: i64 = context.query_row("SELECT n FROM value", [], |row| row.get(0))?; + Ok(n.to_le_bytes().to_vec()) + })?, + ); + let (mut fixture, route, a, b) = Box::pin(registered::setup(rooted, &class)).await?; + if tcp { + fixture.connect_tcp().await?; + } + let keys = VersionKeys::new( + VersionSigningKey::new([94; 32], u64::MAX)?, + vec![], + Duration::from_secs(30), + )?; + let bounds = CellDispatchLimits { + max_cached_routes: 32, + route_cache_bytes: 256 * 1024, + max_invocations: 3, + retained_bytes: 1024 * 1024, + max_input_bytes: 1024, + }; + let dispatcher = CellGroupDispatcher::new( + &fixture.cluster.domain, + fixture.cluster.directory.clone(), + class.clone(), + keys.clone(), + bounds, + )?; + let now = fixture + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms() as u64 }) + .await?; + let pool = fixture.appliers[&(0, route.history)].0.clone(); + let runtime = fixture.runtimes[0].clone(); + let options = fixture.cell_options(&a).await?; + let mut command = dispatcher + .command( + a.clone(), + request(&class, 1, now + 60_000, b"hold")?, + limits(), + None, + ) + .await?; + command.begin().await?; + let first = command.attempt().context("first route")?.clone(); + let mut preparation = Box::pin(command.prepare(&runtime, &pool, options.clone())); + cellar::asyncrt::select! { + result = &mut preparation => anyhow::bail!("held callback finished: {}", result.is_ok()), + result = started => result?, + } + drop(preparation); // Cancelled response, not physical SQL ownership. + assert_eq!(pool.usage()?.active_cells, 1); + release.send(())?; + command.retry().await?; + assert_eq!( + command.attempt().context("retry")?.request(), + first.request() + ); + assert_eq!( + command.attempt().context("retry")?.deadline_ms(), + first.deadline_ms() + ); + let other_pool = fixture.appliers[&(1, route.history)].0.clone(); + assert!(matches!( + command + .prepare( + &fixture.runtimes[1], + &other_pool, + fixture.cell_options_at(&a, 1).await? + ) + .await?, + CellCommandPreparation::Prepared + )); + assert_eq!(calls.load(Ordering::SeqCst), 1); + assert_eq!(other_pool.usage()?.active_cells, 0); + let first_acceptance = accepted(&mut command).await?; + let again = accepted(&mut command).await?; // Lost acceptance response. + assert_eq!(first_acceptance.position, again.position); + assert_eq!(first_acceptance.digest, again.digest); + let absent = command.settle().await.err().context("not yet applied")?; + assert!(absent.ambiguous); + assert!(matches!( + absent.cause.as_ref(), + CellDispatchError::Execution(ExecutionError::Uncertain) + )); + assert_eq!(pool.usage()?.active_cells, 1); + fixture.drive(Some(again.position.index)).await?; + let receipt = committed(&mut command).await?; + assert_eq!( + receipt + .local_receipt() + .expect("local command receipt") + .receipt + .result(), + a.cell.as_str().as_bytes() + ); + assert_eq!( + receipt + .local_receipt() + .expect("local command receipt") + .version + .sequence + .get(), + 2 + ); + assert_eq!(receipt.durability(), Durability::Quorum); + let now = fixture + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms() as u64 }) + .await?; + assert_eq!( + keys.verify_cell(receipt.token(), &a, route.schema_digest, now)?, + receipt + .local_receipt() + .expect("local command receipt") + .observation() + ); + drop(receipt); + drop(command); + + let mut other = dispatcher + .command( + b.clone(), + request(&class, 1, now + 60_000, b"ordinary")?, + limits(), + None, + ) + .await?; + other.begin().await?; + assert!(matches!( + other + .prepare(&runtime, &pool, fixture.cell_options(&b).await?) + .await?, + CellCommandPreparation::Prepared + )); + let b_acceptance = accepted(&mut other).await?; + fixture.drive(Some(b_acceptance.position.index)).await?; + drop(committed(&mut other).await?); + drop(other); + let mut duplicate = dispatcher + .command( + a.clone(), + request(&class, 1, now + 60_000, b"hold")?, + limits(), + Some(first.cached_route()), + ) + .await?; + duplicate.begin().await?; + let CellCommandPreparation::Committed(receipt) = + duplicate.prepare(&runtime, &pool, options.clone()).await? + else { + anyhow::bail!("duplicate executed") + }; + assert_eq!( + receipt + .local_receipt() + .expect("local command receipt") + .receipt + .result(), + a.cell.as_str().as_bytes() + ); + assert_eq!(calls.load(Ordering::SeqCst), 2); + drop(receipt); + drop(duplicate); + + // An actual higher-term applied barrier can exclude a prepared old leader + // command. Retrying preserves host-captured time/randomness and command ID. + let mut retry = dispatcher + .command( + a.clone(), + request(&class, 2, now + 60_000, b"ordinary")?, + limits(), + Some(first.cached_route()), + ) + .await?; + retry.begin().await?; + let retry_first = retry.attempt().context("retry origin")?.clone(); + assert!(matches!( + retry.prepare(&runtime, &pool, options.clone()).await?, + CellCommandPreparation::Prepared + )); + reconciliation::transfer_cell(&mut fixture, 0, 1, a.clone()).await?; + let error = retry.submit().await.err().context("old leader refused")?; + assert!(error.ambiguous); + assert!(matches!( + retry.settle().await?, + CellCommandSettlement::NotCommitted + )); + assert_eq!(pool.usage()?.active_cells, 0); + retry.retry().await?; + assert_eq!( + retry.attempt().context("retry new leader")?.request(), + retry_first.request() + ); + assert!(matches!( + retry + .prepare( + &fixture.runtimes[1], + &other_pool, + fixture.cell_options_at(&a, 1).await? + ) + .await?, + CellCommandPreparation::Prepared + )); + let retry_acceptance = accepted(&mut retry).await?; + reconciliation::apply_through(&mut fixture, retry_acceptance.position.index).await?; + let receipt = committed(&mut retry).await?; + assert_eq!( + receipt + .local_receipt() + .expect("local command receipt") + .version + .sequence + .get(), + 3 + ); + drop(receipt); + drop(retry); + let captured = captured.lock().unwrap().clone(); + assert_eq!(captured.len(), 4); + assert_eq!(captured[2], captured[3]); + + let mut s3 = request(&class, 3, now + 60_000, b"ordinary")?; + s3.durability = Durability::S3; + let error = dispatcher + .command(a.clone(), s3, limits(), None) + .await + .err() + .context("S3 unavailable")?; + assert!(!error.ambiguous); + assert!(matches!( + error.cause.as_ref(), + CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::UnsupportedCapability + )) + )); + let catalog = Catalog::new( + Arc::new(object_store::memory::InMemory::new()), + CatalogOptions::default(), + )?; + let mut config = NamespaceConfig::new( + a.cluster, + a.namespace, + cellar_logic::NamespaceMode::Raft, + class.schema_digest(), + ); + config.minimum_durability = Durability::S3; + let namespace = fixture + .cluster + .domain + .run(catalog.open_namespace(config)) + .await??; + let required = CellGroupDispatcher::new( + &fixture.cluster.domain, + RangeDirectory::new(namespace, DirectoryLimits::default())?, + class.clone(), + keys, + bounds, + )?; + assert!(matches!( + required + .command( + a.clone(), + request(&class, 3, now + 60_000, b"ordinary")?, + limits(), + None + ) + .await, + Err(CellCommandError { + ambiguous: false, + cause, + .. + }) if matches!(cause.as_ref(), CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::UnsupportedCapability + ))) + )); + assert_eq!(calls.load(Ordering::SeqCst), 4); + assert_eq!(dispatcher.usage(), (0, 0)); + drop(required); + drop(dispatcher); + drop(pool); + drop(other_pool); + drop(runtime); + fixture.close().await +} diff --git a/crates/cellar/tests/support/cell_group_sql_dispatch.rs b/crates/cellar/tests/support/cell_group_sql_dispatch.rs new file mode 100644 index 000000000..551804549 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_sql_dispatch.rs @@ -0,0 +1,409 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Catalog-selected invocations use real Raft reads and registered SQL workers. +use super::*; +use std::sync::{ + atomic::{AtomicUsize, Ordering}, + Mutex, +}; + +fn limits() -> CellRoutingLimits { + CellRoutingLimits { + timeout: Duration::from_secs(5), + max_attempts: 2, + } +} + +fn request(class: &RaftClass, consistency: CellQueryConsistency) -> Result { + Ok(CellQueryRequest { + invocation: ProposalId::new([201; 16])?, + contract: class.contract("Get").context("Get")?, + input: vec![7], + consistency, + }) +} + +async fn execute( + fixture: &mut RuntimeCluster, + query: &mut CellQueryInvocation, + cell: &CellHistory, + node: usize, +) -> Result { + let runtime = fixture.runtimes[node].clone(); + let pool = fixture.appliers[&(node, fixture.cluster.birth.route().history)] + .0 + .clone(); + let options = fixture.cell_options_at(cell, node).await?; + let work = query.execute(&runtime, &pool, options); + tokio::pin!(work); + loop { + cellar::asyncrt::select! { + result = &mut work => return result.map_err(Into::into), + result = async { + fixture.pump(&[], &mut BTreeMap::new()).await?; + tokio::task::yield_now().await; + Ok::<_, anyhow::Error>(()) + } => result?, + } + } +} + +pub(crate) async fn routed(rooted: bool) -> Result<()> { + let calls = Arc::new(AtomicUsize::new(0)); + let count = calls.clone(); + let (entered, started) = tokio::sync::oneshot::channel(); + let entered = Mutex::new(Some(entered)); + let (release, waiting) = std::sync::mpsc::sync_channel(1); + let waiting = Mutex::new(waiting); + let class = Arc::new(RaftClass::new("Counter", 1, [81; 32])?.query( + "Get", + [82; 32], + move |context, input| { + count.fetch_add(1, Ordering::SeqCst); + let QueryVersion::Cell(observation) = context.version() else { + anyhow::bail!("group cut") + }; + assert_eq!(context.identity(), &observation.version.history); + if input.starts_with(b"hold") { + entered + .lock() + .unwrap() + .take() + .context("one held query")? + .send(()) + .map_err(|_| anyhow::anyhow!("entered"))?; + waiting + .lock() + .unwrap() + .recv_timeout(Duration::from_secs(10))?; + } + let n: i64 = context.query_row("SELECT n FROM value", [], |row| row.get(0))?; + Ok(n.to_le_bytes().to_vec()) + }, + )?); + let (mut fixture, route, a, b) = Box::pin(registered::setup(rooted, &class)).await?; + let keys = VersionKeys::new( + VersionSigningKey::new([83; 32], u64::MAX)?, + vec![], + Duration::from_secs(10), + )?; + let bounds = CellDispatchLimits { + max_cached_routes: 32, + route_cache_bytes: 256 * 1024, + max_invocations: 1, + retained_bytes: 1024 * 1024, + max_input_bytes: 256 * 1024, + }; + let dispatcher = CellGroupDispatcher::new( + &fixture.cluster.domain, + fixture.cluster.directory.clone(), + class.clone(), + keys.clone(), + bounds, + )?; + let mut query = dispatcher.query( + a.clone(), + request(&class, CellQueryConsistency::Linearizable)?, + limits(), + None, + )?; + assert!(matches!( + dispatcher.clone().query( + b.clone(), + request(&class, CellQueryConsistency::Stale)?, + limits(), + None + ), + Err(CellDispatchError::Runtime(RuntimeError::Overloaded)) + )); + query.begin().await?; + let first = query.attempt().context("attempt")?.clone(); + assert_eq!(first.route(), &route); + let response = execute(&mut fixture, &mut query, &a, 0).await?; + assert_eq!(response.result(), &0_i64.to_le_bytes()); + let now = fixture + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms() as u64 }) + .await?; + assert_eq!( + keys.verify_cell(response.token(), &a, route.schema_digest, now)?, + *response.observation() + ); + let token = response.token().clone(); + let expected = response.observation().clone(); + drop(response); // Lost response: retry the same immutable query and budget. + let pool = fixture.appliers[&(0, route.history)].0.clone(); + assert!(matches!( + query + .execute(&fixture.runtimes[0], &pool, fixture.cell_options(&a).await?) + .await, + Err(CellDispatchError::Invalid("consumed route attempt")) + )); + query.retry().await?; + let second = query.attempt().context("second")?; + assert_eq!(second.request(), first.request()); + assert_eq!(second.deadline_ms(), first.deadline_ms()); + assert_eq!(second.ordinal(), 2); + let repeated = execute(&mut fixture, &mut query, &a, 0).await?; + assert_eq!(repeated.observation(), &expected); + drop(repeated); + assert!(matches!( + query.retry().await, + Err(CellDispatchError::Routing(CellRoutingError { + reason: CellRoutingFailure::AttemptsExhausted, + .. + })) + )); + drop(query); + assert_eq!(dispatcher.usage(), (0, 0)); + + let mut at_least = dispatcher.query( + a.clone(), + request(&class, CellQueryConsistency::AtLeast(token.clone()))?, + limits(), + Some(first.cached_route()), + )?; + at_least.begin().await?; + let output = execute(&mut fixture, &mut at_least, &a, 1).await?; + assert_eq!(output.observation(), &expected); + drop(output); + drop(at_least); + let before = calls.load(Ordering::SeqCst); + let mut wrong = dispatcher.query( + b.clone(), + request(&class, CellQueryConsistency::AtLeast(token))?, + limits(), + None, + )?; + wrong.begin().await?; + assert!(matches!( + wrong + .execute(&fixture.runtimes[0], &pool, fixture.cell_options(&b).await?) + .await, + Err(CellDispatchError::Version( + cellar::application::consensus::VersionError::WrongHistory + )) + )); + drop(wrong); + assert_eq!(calls.load(Ordering::SeqCst), before); + + let mut wrong_engine = dispatcher.query( + a.clone(), + request(&class, CellQueryConsistency::Stale)?, + limits(), + Some(first.cached_route()), + )?; + wrong_engine.begin().await?; + let other_pool = fixture.appliers[&(1, route.history)].0.clone(); + assert!(matches!( + wrong_engine + .execute( + &fixture.runtimes[0], + &other_pool, + fixture.cell_options(&a).await? + ) + .await, + Err(CellDispatchError::Execution(ExecutionError::Invalid( + "cell query engine/schema" + ))) + )); + drop(wrong_engine); + drop(other_pool); + let mut invalid_hint = dispatcher.query( + a.clone(), + request(&class, CellQueryConsistency::Stale)?, + limits(), + Some(first.cached_route()), + )?; + invalid_hint.begin().await?; + let mut hint = route.clone(); + hint.epoch = RouteEpoch::new(2)?; + hint.schema_digest = [86; 32]; + assert!(matches!( + invalid_hint.redirect(&hint).await, + Err(CellDispatchError::Routing(CellRoutingError { + reason: CellRoutingFailure::InvalidHint, + .. + })) + )); + assert!(matches!( + invalid_hint + .execute(&fixture.runtimes[0], &pool, fixture.cell_options(&a).await?) + .await, + Err(CellDispatchError::Routing(CellRoutingError { + reason: CellRoutingFailure::StaleAttempt, + .. + })) + )); + drop(invalid_hint); + assert_eq!(calls.load(Ordering::SeqCst), before); + + let foreign_domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let foreign = CellGroupDispatcher::new( + &foreign_domain, + fixture.cluster.directory.clone(), + class.clone(), + keys.clone(), + bounds, + )?; + let mut wrong = foreign.query( + a.clone(), + request(&class, CellQueryConsistency::Stale)?, + limits(), + None, + )?; + wrong.begin().await?; + assert!(matches!( + wrong + .execute(&fixture.runtimes[0], &pool, fixture.cell_options(&a).await?) + .await, + Err(CellDispatchError::Invalid("foreign execution domain")) + )); + drop(wrong); + drop(foreign); + foreign_domain.shutdown().await; + let bad_class = Arc::new(RaftClass::new("Counter", 1, [84; 32])?); + assert!(matches!( + CellGroupDispatcher::new( + &fixture.cluster.domain, + fixture.cluster.directory.clone(), + bad_class, + keys, + bounds + ), + Err(CellDispatchError::Registration( + RegistrationError::ContractMismatch + )) + )); + + let mut expired = dispatcher.query( + a.clone(), + request(&class, CellQueryConsistency::Stale)?, + CellRoutingLimits { + timeout: Duration::from_millis(10), + max_attempts: 2, + }, + Some(first.cached_route()), + )?; + expired.begin().await?; + let original_deadline = expired.attempt().context("deadline")?.deadline_ms(); + tokio::time::sleep(Duration::from_millis(20)).await; + assert!(matches!( + expired.retry().await, + Err(CellDispatchError::Routing(CellRoutingError { + reason: CellRoutingFailure::Deadline, + .. + })) + )); + assert_eq!( + expired.attempt().context("same deadline")?.deadline_ms(), + original_deadline + ); + drop(expired); + assert_eq!(calls.load(Ordering::SeqCst), before); + + // Confirm a replacement against the real directory, but do not invent its + // physical owner or claim this metadata-only fixture performs a live move. + let directory = fixture.cluster.directory.clone(); + let replacement = directory.prepare_group_birth(CellGroupBirthSpec { + history: GroupHistory { + group: GroupId::new([203; 16])?, + ..route.history + }, + range: route.range.clone(), + epoch: RouteEpoch::new(2)?, + creation: ProposalId::new([204; 16])?, + voters: fixture.cluster.birth.voters().to_vec(), + learners: vec![], + })?; + fixture + .cluster + .domain + .run(directory.reserve_group_birth(&replacement)) + .await??; + let mut relocated = dispatcher.query( + a.clone(), + request(&class, CellQueryConsistency::Stale)?, + limits(), + Some(first.cached_route()), + )?; + relocated.begin().await?; + let original = relocated.attempt().context("original")?.clone(); + let view = fixture.cluster.domain.run(directory.observe()).await??; + let publish = directory.prepare_replacement( + &view, + ProposalId::new([205; 16])?, + std::slice::from_ref(&route), + &[replacement.route().clone()], + )?; + fixture + .cluster + .domain + .run(directory.publish(&publish)) + .await??; + relocated.redirect(replacement.route()).await?; + assert_eq!( + relocated.attempt().context("redirect")?.deadline_ms(), + original.deadline_ms() + ); + assert_eq!( + relocated.attempt().context("redirect")?.request(), + original.request() + ); + assert_eq!( + relocated.attempt().context("redirect")?.route(), + replacement.route() + ); + assert!(matches!( + relocated + .execute(&fixture.runtimes[0], &pool, fixture.cell_options(&a).await?) + .await, + Err(CellDispatchError::Runtime(RuntimeError::NotFound)) + )); + drop(relocated); + assert_eq!(calls.load(Ordering::SeqCst), before); + + // A known route needs no new catalog lookup; actual destination authority is + // still mandatory. Here the old physical owner has not been transitioned. + let mut held_request = request(&class, CellQueryConsistency::Stale)?; + held_request.input = vec![0; 256 * 1024]; + held_request.input[..4].copy_from_slice(b"hold"); + let mut held = dispatcher.query( + a.clone(), + held_request, + limits(), + Some(first.cached_route()), + )?; + held.begin().await?; + let retained = dispatcher.usage(); + assert!(retained.0 >= 512 * 1024); + assert_eq!(retained.1, 1); + let runtime = fixture.runtimes[0].clone(); + let sql = pool.clone(); + let options = fixture.cell_options(&a).await?; + let waiter = fixture + .cluster + .domain + .spawn(async move { held.execute(&runtime, &sql, options).await })?; + tokio::time::timeout(Duration::from_secs(5), started).await??; + waiter.abort(); + assert!(waiter.await.is_err()); + // A dropped routing waiter cannot refund the invocation while its entered + // SQL callback still owns captured input and physical read authority. + assert_eq!(dispatcher.usage(), retained); + assert_eq!(pool.usage()?.retained_operations, 1); + assert!(pool.usage()?.retained_bytes >= 256 * 1024); + assert!( + tokio::time::timeout(Duration::from_millis(30), pool.close()) + .await + .is_err() + ); + release.send(())?; + tokio::time::timeout(Duration::from_secs(5), pool.close()).await??; + assert_eq!(pool.usage()?.retained_bytes, 0); + assert_eq!(dispatcher.usage(), (0, 0)); + drop(pool); + drop(dispatcher); + fixture.close().await +} diff --git a/crates/cellar/tests/support/cell_group_sql_enrollment.rs b/crates/cellar/tests/support/cell_group_sql_enrollment.rs new file mode 100644 index 000000000..6735569b1 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_sql_enrollment.rs @@ -0,0 +1,510 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::reconciliation::{apply_through, commit_on, transfer}; +use super::*; +use std::sync::atomic::{AtomicUsize, Ordering}; + +fn attempt(tag: u8) -> Result { + Ok(CellEnrollmentAttempt { + creation: ProposalId::new([tag; 16])?, + preview: ImageId::new([tag; 16])?, + inputs: cellar::storage::CommandInputs { + now_unix_millis: 10, + random: [17; 32], + }, + }) +} + +fn limits(workers: usize) -> CellSqlLimits { + CellSqlLimits { + workers, + queue_per_worker: 4, + max_pending_cells: 4, + group_pending_cells: 4, + node_bytes: 512 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + } +} + +async fn cluster(rooted: bool) -> Result<(RuntimeCluster, GroupRoute)> { + let mut fixture = RuntimeCluster::new().await?; + if rooted { + for (n, voter) in fixture.cluster.voters.iter_mut().enumerate() { + voter.extents = Some(NodeExtentStore::open( + fixture.cluster._root.path().join(format!("enrollment-{n}")), + Arc::new(DirectFileSystem), + Default::default(), + )?); + } + } + let route = fixture.cluster.birth.route().clone(); + fixture.runtimes[0].campaign(route.history).await?; + fixture.drive(None).await?; + Ok((fixture, route)) +} + +pub(super) async fn prepare_initial( + fixture: &RuntimeCluster, + pool: &CellGroupSqlPool, + node: usize, + cell: &CellHistory, + tag: u8, + value: i64, +) -> Result { + let authority = fixture.runtimes[node] + .enrollment_authority(fixture.cluster.birth.route().history, cell.clone()) + .await?; + let outcome = pool + .initialize( + fixture.cell_options_at(cell, node).await?, + authority, + attempt(tag)?, + move |tx| { + tx.execute_batch("CREATE TABLE value(n);")?; + tx.execute("INSERT INTO value VALUES (?1)", [value])?; + Ok(()) + }, + ) + .await?; + let CellSqlOutcome::Prepared(prepared) = outcome else { + anyhow::bail!("prepared enrollment") + }; + Ok(prepared) +} + +async fn submit( + fixture: &mut RuntimeCluster, + pool: &CellGroupSqlPool, + node: usize, + cell: &CellHistory, + prepared: &CellSqlPrepared, +) -> Result<()> { + pool.mark_submitted(cell.clone(), prepared.proposal()) + .await?; + let accepted = accepted_sql(&fixture.runtimes[node], prepared).await?; + apply_through(fixture, accepted.position.index).await +} + +pub(super) async fn resolved( + fixture: &RuntimeCluster, + pool: &CellGroupSqlPool, + node: usize, + cell: &CellHistory, + creation: ProposalId, +) -> Result { + let state = fixture.runtimes[node] + .enrollment_state(fixture.cluster.birth.route().history, cell.clone()) + .await?; + let CellSqlEnrollmentResolution::Committed(record) = + pool.resolve_enrollment(creation, state).await? + else { + anyhow::bail!("committed enrollment") + }; + Ok(*record) +} + +pub(super) async fn concurrent(rooted: bool) -> Result<()> { + let (mut fixture, route) = cluster(rooted).await?; + let a = history(&route, "sql-a")?; + let b = history(&route, "sql-b")?; + let c = history(&route, "sql-c")?; + let key = |cell: &CellHistory| { + u64::from_le_bytes(cell_image_history_digest(cell)[..8].try_into().unwrap()) + }; + let workers = (2..=8) + .find(|n| key(&a) % n != key(&b) % n) + .context("independent workers")? as usize; + let pool = CellGroupSqlPool::start(&fixture.cluster.voters[0].log, limits(workers))?; + assert!(fixture.runtimes[1] + .enrollment_authority(route.history, a.clone()) + .await + .is_err()); + let callbacks = Arc::new(AtomicUsize::new(0)); + let count = callbacks.clone(); + let (entered, started) = tokio::sync::oneshot::channel(); + let (release, wait) = std::sync::mpsc::sync_channel(1); + let authority = fixture.runtimes[0] + .enrollment_authority(route.history, a.clone()) + .await?; + let options = fixture.cell_options(&a).await?; + let caller = pool.clone(); + let waiter = fixture.cluster.domain.spawn(async move { + caller + .initialize(options, authority, attempt(151)?, move |tx| { + count.fetch_add(1, Ordering::SeqCst); + tx.execute_batch("CREATE TABLE value(n); INSERT INTO value VALUES (1);")?; + let _ = entered.send(()); + wait.recv_timeout(Duration::from_secs(10))?; + Ok(()) + }) + .await + .map_err(anyhow::Error::new) + })?; + tokio::time::timeout(Duration::from_secs(5), started).await??; + waiter.abort(); + assert!(waiter.await.is_err()); + let mut other = a.clone(); + other.incarnation = IncarnationId::new([152; 16])?; + for cell in [&a, &other] { + let authority = fixture.runtimes[0] + .enrollment_authority(route.history, cell.clone()) + .await?; + assert!(matches!( + pool.initialize( + fixture.cell_options(cell).await?, + authority, + attempt(152)?, + |_| panic!("concurrent creation callback") + ) + .await, + Err(ExecutionError::Busy) + )); + } + let pb = prepare_initial(&fixture, &pool, 0, &b, 153, 7).await?; + submit(&mut fixture, &pool, 0, &b, &pb).await?; + let rb = resolved(&fixture, &pool, 0, &b, pb.proposal()).await?; + assert_eq!(rb.record().cell(), &b); + drop(rb); + drop(pb); + assert_eq!(pool.usage()?.active_cells, 1); + release.send(())?; + let pa = tokio::time::timeout( + Duration::from_secs(5), + pool.resume(a.clone(), ProposalId::new([151; 16])?), + ) + .await??; + let state = fixture.runtimes[0] + .enrollment_state(route.history, a.clone()) + .await?; + assert!(matches!( + pool.resolve_enrollment(pa.proposal(), state).await, + Err(ExecutionError::Uncertain) + )); + submit(&mut fixture, &pool, 0, &a, &pa).await?; + let ra = resolved(&fixture, &pool, 0, &a, pa.proposal()).await?; + let original = ra.record().clone(); + assert_eq!(callbacks.load(Ordering::SeqCst), 1); + drop(ra); + drop(pa); + let pc = prepare_initial(&fixture, &pool, 0, &c, 154, 9).await?; + submit(&mut fixture, &pool, 0, &c, &pc).await?; + drop(resolved(&fixture, &pool, 0, &c, pc.proposal()).await?); + drop(pc); + for voter in &fixture.cluster.voters { + assert_eq!( + voter.log.cell_groups()?.len(), + 1, + "enrollment must not create a physical Raft group" + ); + let marker = voter + .log + .cell_group(&route.history)? + .group_manifest()? + .context("root")?; + for cell in [&a, &b, &c] { + let leaf = voter + .tree + .get( + voter.pages.as_ref().context("fixture page owner")?, + marker.root(), + &cell.cell, + )? + .context("enrolled cell")?; + assert_eq!(leaf.version.sequence.get(), 1); + } + } + // A real mutation must not erase or overwrite the immutable creation record. + let authority = fixture.runtimes[0] + .execution_authority(route.history, a.clone()) + .await?; + let CellSqlOutcome::Prepared(command) = pool + .stage( + fixture.cell_options(&a).await?, + authority, + cell_attempt(155)?, + true, + |tx| { + let value: i64 = tx.query_row("SELECT n FROM value", [], |row| row.get(0))?; + assert_eq!(value, 1); + tx.execute("UPDATE value SET n=n+10", [])?; + Ok(b"after-init".to_vec()) + }, + ) + .await? + else { + anyhow::bail!("mutation") + }; + let receipt = commit_on(&mut fixture, &pool, 0, &a, command).await?; + let version = receipt.receipt().version.clone(); + drop(receipt); + let authority = fixture.runtimes[0] + .enrollment_authority(route.history, a.clone()) + .await?; + let CellSqlOutcome::Enrollment(record) = pool + .initialize( + fixture.cell_options(&a).await?, + authority, + attempt(151)?, + |_| panic!("committed creation must not initialize again"), + ) + .await? + else { + anyhow::bail!("creation retry") + }; + assert_eq!(record.record(), &original); + drop(record); + assert_eq!( + fixture.runtimes[0] + .execution_authority(route.history, a.clone()) + .await? + .cell() + .version, + version + ); + for (cell, tag) in [(&a, 156), (&other, 151)] { + let authority = fixture.runtimes[0] + .enrollment_authority(route.history, cell.clone()) + .await?; + assert!(matches!( + pool.initialize( + fixture.cell_options(cell).await?, + authority, + attempt(tag)?, + |_| panic!("conflicting creation") + ) + .await, + Err(ExecutionError::Refused(CommandFailure::CommandConflict)) + )); + } + assert_eq!(pool.usage()?.active_cells, 0); + assert_eq!(pool.usage()?.retained_bytes, 0); + // A cancelled initializer and cancelled close waiter cannot release the + // physical callback or its logical-name reservation before the worker exits. + let d = history(&route, "cancel-init")?; + let authority = fixture.runtimes[0] + .enrollment_authority(route.history, d.clone()) + .await?; + let options = fixture.cell_options(&d).await?; + let (entered, started) = tokio::sync::oneshot::channel(); + let (release, wait) = std::sync::mpsc::sync_channel(1); + let caller = pool.clone(); + let waiter = fixture.cluster.domain.spawn(async move { + caller + .initialize(options, authority, attempt(157)?, move |tx| { + tx.execute_batch("CREATE TABLE value(n); INSERT INTO value VALUES (99);")?; + let _ = entered.send(()); + wait.recv_timeout(Duration::from_secs(10))?; + Ok(()) + }) + .await + .map_err(anyhow::Error::new) + })?; + tokio::time::timeout(Duration::from_secs(5), started).await??; + waiter.abort(); + assert!(waiter.await.is_err()); + let caller = pool.clone(); + let mut closing = fixture + .cluster + .domain + .spawn(async move { caller.close().await })?; + assert!( + tokio::time::timeout(Duration::from_millis(30), &mut closing) + .await + .is_err() + ); + closing.abort(); + assert!(closing.await.is_err()); + assert_eq!(pool.usage()?.active_cells, 1); + assert!(pool.usage()?.running_workers > 0); + assert!( + tokio::time::timeout(Duration::from_millis(30), pool.close()) + .await + .is_err() + ); + release.send(())?; + pool.close().await?; + assert_eq!(pool.usage()?.active_cells, 0); + assert_eq!(pool.usage()?.retained_bytes, 0); + assert_eq!(pool.usage()?.running_workers, 0); + if let Some(store) = &fixture.cluster.voters[0].extents { + assert_eq!(store.usage()?.workspaces, 0); + } + assert!(fixture.runtimes[0] + .enrollment_state(route.history, d) + .await? + .record() + .is_none()); + drop(pool); + fixture.close().await +} + +pub(super) async fn recovery(rooted: bool) -> Result<()> { + let (mut fixture, route) = cluster(rooted).await?; + let pool = CellGroupSqlPool::start(&fixture.cluster.voters[0].log, limits(2))?; + // The existing election fixture holds the sql-a barrier on the old leader. + let a = history(&route, "sql-a")?; + let initial = prepare_initial(&fixture, &pool, 0, &a, 161, 1).await?; + submit(&mut fixture, &pool, 0, &a, &initial).await?; + drop(resolved(&fixture, &pool, 0, &a, initial.proposal()).await?); + drop(initial); + let d = history(&route, "absent-init")?; + let e = history(&route, "conflict-init")?; + let f = history(&route, "committed-init")?; + let pd = prepare_initial(&fixture, &pool, 0, &d, 162, 2).await?; + let pe = prepare_initial(&fixture, &pool, 0, &e, 163, 3).await?; + let pf = prepare_initial(&fixture, &pool, 0, &f, 164, 4).await?; + pool.mark_submitted(d.clone(), pd.proposal()).await?; + pool.mark_submitted(e.clone(), pe.proposal()).await?; + submit(&mut fixture, &pool, 0, &f, &pf).await?; + transfer(&mut fixture, 0, 1).await?; + let successor = CellGroupSqlPool::start(&fixture.cluster.voters[1].log, limits(1))?; + let mut new_e = e.clone(); + new_e.incarnation = IncarnationId::new([165; 16])?; + let next = prepare_initial(&fixture, &successor, 1, &new_e, 165, 30).await?; + submit(&mut fixture, &successor, 1, &new_e, &next).await?; + drop(resolved(&fixture, &successor, 1, &new_e, next.proposal()).await?); + drop(next); + let foreign = fixture.runtimes[1] + .enrollment_state(route.history, d.clone()) + .await?; + assert!(matches!( + pool.resolve_enrollment(pd.proposal(), foreign).await, + Err(ExecutionError::Invalid(_)) + )); + let state = fixture.runtimes[0] + .enrollment_state(route.history, d.clone()) + .await?; + assert!(matches!( + pool.resolve_enrollment(pd.proposal(), state).await?, + CellSqlEnrollmentResolution::NotCommitted + )); + let state = fixture.runtimes[0] + .enrollment_state(route.history, e.clone()) + .await?; + assert!(matches!( + pool.resolve_enrollment(pe.proposal(), state).await?, + CellSqlEnrollmentResolution::Conflict + )); + let authority = fixture.runtimes[1] + .execution_authority(route.history, f.clone()) + .await?; + let CellSqlOutcome::Prepared(command) = successor + .stage( + fixture.cell_options_at(&f, 1).await?, + authority, + cell_attempt(166)?, + true, + |tx| { + tx.execute("UPDATE value SET n=n+1", [])?; + Ok(b"later".to_vec()) + }, + ) + .await? + else { + anyhow::bail!("new leader mutation") + }; + drop(commit_on(&mut fixture, &successor, 1, &f, command).await?); + let record = resolved(&fixture, &pool, 0, &f, pf.proposal()).await?; + assert_eq!(record.record().creation(), pf.proposal()); + assert_eq!(pool.usage()?.active_cells, 0); + assert!(pool.usage()?.retained_bytes > 0); + // Even after this node regains leadership, a retained old prepared response + // cannot be proposed under its new term. Neither sample-before-queue nor + // merely checking that the receiver is a leader would reject this race. + transfer(&mut fixture, 1, 0).await?; + assert!(matches!( + pe.submit(&fixture.runtimes[0]).await, + Err(RuntimeError::StaleCompletion) + )); + assert_eq!( + fixture.runtimes[0] + .enrollment_state(route.history, new_e.clone()) + .await? + .record() + .context("winner")? + .cell(), + &new_e + ); + drop(record); + drop(pd); + drop(pe); + drop(pf); + assert_eq!(pool.usage()?.retained_bytes, 0); + successor.close().await?; + drop(successor); + pool.close().await?; + drop(pool); + fixture.close().await +} + +pub(super) async fn owned_workspace_barriers() -> Result<()> { + let (mut fixture, route) = cluster(false).await?; + let gate = fault::Gate::new(None); + for (n, voter) in fixture.cluster.voters.iter_mut().enumerate() { + let filesystem: Arc = if n == 0 { + Arc::new(fault::FaultFileSystem { gate: gate.clone() }) + } else { + Arc::new(DirectFileSystem) + }; + voter.extents = Some(NodeExtentStore::open( + fixture + .cluster + ._root + .path() + .join(format!("owned-workspace-{n}")), + filesystem, + Default::default(), + )?); + } + let pool = CellGroupSqlPool::start(&fixture.cluster.voters[0].log, limits(2))?; + let cell = history(&route, "owned-workspace")?; + *gate.sync_path.lock().unwrap() = Some(fixture.cluster._root.path().to_path_buf()); + let release = gate.arm(fault::FAIL_PARENT_SYNC); + let initial = prepare_initial(&fixture, &pool, 0, &cell, 170, 7).await?; + submit(&mut fixture, &pool, 0, &cell, &initial).await?; + let enrolled = resolved(&fixture, &pool, 0, &cell, initial.proposal()).await?; + assert_eq!(enrolled.record().cell(), &cell); + drop(enrolled); + drop(initial); + let authority = fixture.runtimes[0] + .execution_authority(route.history, cell.clone()) + .await?; + let CellSqlOutcome::Prepared(command) = pool + .stage( + fixture.cell_options(&cell).await?, + authority, + cell_attempt(171)?, + true, + |tx| { + tx.execute("UPDATE value SET n=n+1", [])?; + let value: i64 = tx.query_row("SELECT n FROM value", [], |row| row.get(0))?; + assert_eq!(value, 8); + Ok(b"after-initialization".to_vec()) + }, + ) + .await? + else { + anyhow::bail!("ordinary command preparation") + }; + let receipt = commit_on(&mut fixture, &pool, 0, &cell, command).await?; + assert_eq!(receipt.receipt().version.sequence.get(), 2); + drop(receipt); + assert_eq!( + gate.mode.load(Ordering::Acquire), + fault::FAIL_PARENT_SYNC, + "an admitted workspace or disposable cache resynced a completed ancestor" + ); + drop(release); + pool.close().await?; + assert_eq!( + fixture.cluster.voters[0] + .extents + .as_ref() + .context("store")? + .usage()? + .workspaces, + 0 + ); + drop(pool); + fixture.close().await +} diff --git a/crates/cellar/tests/support/cell_group_sql_pool.rs b/crates/cellar/tests/support/cell_group_sql_pool.rs new file mode 100644 index 000000000..66bb609da --- /dev/null +++ b/crates/cellar/tests/support/cell_group_sql_pool.rs @@ -0,0 +1,291 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; + +pub(super) async fn admission(rooted: bool) -> Result<()> { + let (mut fixture, route, a, b) = enrolled_sql_cells(rooted).await?; + let c = history(&route, "sql-c")?; + let key = |cell: &CellHistory| -> u64 { + u64::from_le_bytes(cell_image_history_digest(cell)[..8].try_into().unwrap()) + }; + let workers = (2..=8) + .find(|n| key(&a) % n != key(&b) % n) + .context("distinct workers")? as usize; + let pool = CellGroupSqlPool::start( + &fixture.cluster.voters[0].log, + CellSqlLimits { + workers, + queue_per_worker: 2, + max_pending_cells: 3, + group_pending_cells: 2, + node_bytes: 512 * 1024 * 1024, + group_bytes: 256 * 1024 * 1024, + }, + )?; + assert!(CellGroupSqlPool::start( + &fixture.cluster.voters[0].log, + CellSqlLimits { + workers: 1, + queue_per_worker: 1, + max_pending_cells: 1, + group_pending_cells: 1, + node_bytes: 128 * 1024 * 1024, + group_bytes: 128 * 1024 * 1024, + } + ) + .is_err()); + let oa = fixture.cell_options(&a).await?; + let ob = fixture.cell_options(&b).await?; + let oc = fixture.cell_options(&c).await?; + let foreign = CellGroupSqlPool::start( + &fixture.cluster.voters[1].log, + CellSqlLimits { + workers: 1, + queue_per_worker: 1, + max_pending_cells: 1, + group_pending_cells: 1, + node_bytes: 128 * 1024 * 1024, + group_bytes: 128 * 1024 * 1024, + }, + )?; + let ca = fixture.runtimes[0] + .execution_authority(route.history, a.clone()) + .await?; + assert!(matches!( + foreign + .stage(oa.clone(), ca, cell_attempt(90)?, true, |_| panic!( + "cross-engine authority" + )) + .await, + Err(ExecutionError::Invalid(_)) + )); + assert_eq!(foreign.usage()?.active_cells, 0); + foreign.close().await?; + drop(foreign); + let ca = fixture.runtimes[0] + .execution_authority(route.history, a.clone()) + .await?; + let (entered, started) = tokio::sync::oneshot::channel(); + let (release, wait) = std::sync::mpsc::sync_channel(1); + let caller = pool.clone(); + let options = oa.clone(); + let attempt = cell_attempt(91)?; + let pa = attempt.proposal; + let waiter = fixture.cluster.domain.spawn(async move { + caller + .stage(options, ca, attempt, true, move |tx| { + tx.execute("UPDATE value SET n=n+1", [])?; + let _ = entered.send(()); + wait.recv_timeout(Duration::from_secs(10))?; + Ok(b"a-pool".to_vec()) + }) + .await + })?; + tokio::time::timeout(Duration::from_secs(5), started).await??; + waiter.abort(); + assert!(waiter.await.is_err()); + let retained = pool.usage()?; + assert_eq!(retained.active_cells, 1); + assert!(retained.retained_bytes > 0); + let ca = fixture.runtimes[0] + .execution_authority(route.history, a.clone()) + .await?; + assert!(matches!( + pool.stage(oa.clone(), ca, cell_attempt(92)?, true, |_| panic!( + "cancelled waiter released cell ownership" + )) + .await, + Err(ExecutionError::Busy) + )); + let cb = fixture.runtimes[0] + .execution_authority(route.history, b.clone()) + .await?; + let CellSqlOutcome::Prepared(pb) = tokio::time::timeout( + Duration::from_secs(5), + pool.stage(ob.clone(), cb, cell_attempt(93)?, true, |tx| { + tx.execute("UPDATE value SET n=n+7", [])?; + Ok(b"b-pool".to_vec()) + }), + ) + .await?? + else { + anyhow::bail!("prepared B") + }; + assert_eq!(pool.usage()?.active_cells, 2); + assert!( + pool.usage()?.retained_bytes < 2 * retained.retained_bytes, + "completed capture retained its full transient allocation" + ); + let cc = fixture.runtimes[0] + .execution_authority(route.history, c.clone()) + .await?; + assert!(matches!( + pool.stage(oc, cc, cell_attempt(94)?, true, |_| panic!( + "group quota exceeded" + )) + .await, + Err(ExecutionError::Refused( + cellar_logic::consensus::CommandFailure::Overloaded + )) + )); + // B completes through the same consensus group while A's physical SQL is held. + pool.mark_submitted(b.clone(), pb.proposal()).await?; + pool.mark_submitted(b.clone(), pb.proposal()).await?; + let proposal = accepted_sql(&fixture.runtimes[0], &pb).await?; + fixture.drive(Some(proposal.position.index)).await?; + let cb = fixture.runtimes[0] + .execution_authority(route.history, b.clone()) + .await?; + let rb = pool.finish(cb).await?; + assert_eq!(rb.receipt().receipt.result(), b"b-pool"); + assert_eq!(pool.usage()?.active_cells, 1); + drop(rb); + drop(pb); + assert_eq!(pool.usage()?.retained_bytes, retained.retained_bytes); + release.send(())?; + let pa = tokio::time::timeout(Duration::from_secs(5), pool.resume(a.clone(), pa)).await??; + pool.mark_submitted(a.clone(), pa.proposal()).await?; + let proposal = accepted_sql(&fixture.runtimes[0], &pa).await?; + fixture.drive(Some(proposal.position.index)).await?; + let ca = fixture.runtimes[0] + .execution_authority(route.history, a.clone()) + .await?; + let before = ca.cell().version.clone(); + let ra = pool.finish(ca).await?; + assert_eq!(ra.receipt().receipt.result(), b"a-pool"); + drop(ra); + drop(pa); + assert_eq!(pool.usage()?.active_cells, 0); + assert_eq!(pool.usage()?.retained_bytes, 0); + let cb = fixture.runtimes[0] + .execution_authority(route.history, b.clone()) + .await?; + let duplicate = pool + .stage(ob, cb, cell_attempt(95)?, false, |_| { + panic!("receipt replayed callback") + }) + .await?; + let CellSqlOutcome::Receipt(receipt) = duplicate else { + anyhow::bail!("duplicate receipt") + }; + assert_eq!(receipt.receipt().receipt.result(), b"b-pool"); + drop(receipt); + // Cancelling close's waiter must not detach the physical callback or its lock. + let ca = fixture.runtimes[0] + .execution_authority(route.history, a.clone()) + .await?; + let (entered, started) = tokio::sync::oneshot::channel(); + let (release, wait) = std::sync::mpsc::sync_channel(1); + let mut attempt = cell_attempt(96)?; + attempt.request.id = CommandId::new(ClientId::new([81; 16])?, 2)?; + let caller = pool.clone(); + let waiter = fixture.cluster.domain.spawn(async move { + caller + .stage(oa, ca, attempt, true, move |tx| { + tx.execute("UPDATE value SET n=n+100", [])?; + let _ = entered.send(()); + wait.recv_timeout(Duration::from_secs(10))?; + Ok(b"private-only".to_vec()) + }) + .await + })?; + tokio::time::timeout(Duration::from_secs(5), started).await??; + waiter.abort(); + assert!(waiter.await.is_err()); + let closer = pool.clone(); + let mut closing = fixture + .cluster + .domain + .spawn(async move { closer.close().await })?; + assert!( + tokio::time::timeout(Duration::from_millis(30), &mut closing) + .await + .is_err() + ); + closing.abort(); + assert!(closing.await.is_err()); + assert!(pool.usage()?.running_workers > 0); + assert_eq!(pool.usage()?.active_cells, 1); + assert!(CellGroupSqlPool::start( + &fixture.cluster.voters[0].log, + CellSqlLimits { + workers: 1, + queue_per_worker: 1, + max_pending_cells: 1, + group_pending_cells: 1, + node_bytes: 128 * 1024 * 1024, + group_bytes: 128 * 1024 * 1024, + } + ) + .is_err()); + assert!( + tokio::time::timeout(Duration::from_millis(30), pool.close()) + .await + .is_err() + ); + release.send(())?; + tokio::time::timeout(Duration::from_secs(5), pool.close()).await??; + let usage = pool.usage()?; + assert_eq!(usage.running_workers, 0); + assert_eq!(usage.active_cells, 0); + assert_eq!(usage.retained_bytes, 0); + let after = fixture.runtimes[0] + .execution_authority(route.history, a) + .await?; + assert_eq!(after.cell().version, before); + drop(after); + drop(pool); + let replacement = CellGroupSqlPool::start( + &fixture.cluster.voters[0].log, + CellSqlLimits { + workers: 1, + queue_per_worker: 1, + max_pending_cells: 1, + group_pending_cells: 1, + node_bytes: 128 * 1024 * 1024, + group_bytes: 128 * 1024 * 1024, + }, + )?; + let cb = fixture.runtimes[0] + .execution_authority(route.history, b) + .await?; + let options = fixture.cell_options(&cb.cell().version.history).await?; + let receipt = replacement + .stage(options, cb, cell_attempt(97)?, false, |_| { + panic!("restart receipt replayed callback") + }) + .await?; + assert!(matches!(receipt, CellSqlOutcome::Receipt(_))); + replacement.close().await?; + drop(replacement); + assert!( + CellGroupSqlPool::start( + &fixture.cluster.voters[0].log, + CellSqlLimits { + workers: 1, + queue_per_worker: 1, + max_pending_cells: 1, + group_pending_cells: 1, + node_bytes: 128 * 1024 * 1024, + group_bytes: 128 * 1024 * 1024, + } + ) + .is_err(), + "retained response released its node admission owner" + ); + drop(receipt); + let replacement = CellGroupSqlPool::start( + &fixture.cluster.voters[0].log, + CellSqlLimits { + workers: 1, + queue_per_worker: 1, + max_pending_cells: 1, + group_pending_cells: 1, + node_bytes: 128 * 1024 * 1024, + group_bytes: 128 * 1024 * 1024, + }, + )?; + replacement.close().await?; + drop(replacement); + fixture.close().await +} diff --git a/crates/cellar/tests/support/cell_group_sql_reads.rs b/crates/cellar/tests/support/cell_group_sql_reads.rs new file mode 100644 index 000000000..97e3372e6 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_sql_reads.rs @@ -0,0 +1,578 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::reconciliation::apply_through; +use super::*; +use raft::prelude::Entry; + +pub(super) async fn deadline(fixture: &RuntimeCluster, ms: u64) -> Result { + fixture + .cluster + .domain + .run(async { cellar::asyncrt::mono_ms() }) + .await? + .checked_add(ms) + .context("deadline") +} + +fn pool(fixture: &RuntimeCluster, node: usize) -> Result { + Ok(CellGroupSqlPool::start( + &fixture.cluster.voters[node].log, + CellSqlLimits { + workers: 2, + queue_per_worker: 4, + max_pending_cells: 8, + group_pending_cells: 8, + node_bytes: 512 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + }, + )?) +} + +pub(super) async fn selected( + fixture: &mut RuntimeCluster, + node: usize, + cell: &CellHistory, + consistency: CellReadConsistency, +) -> Result { + let runtime = fixture.runtimes[node].clone(); + let read = runtime.read_authority( + fixture.cluster.birth.route().history, + cell.clone(), + consistency, + deadline(fixture, 5000).await?, + ); + tokio::pin!(read); + loop { + cellar::asyncrt::select! { + result = &mut read => return result.map_err(Into::into), + result = async { + fixture.pump(&[], &mut BTreeMap::new()).await?; + tokio::task::yield_now().await; + Ok::<_, anyhow::Error>(()) + } => result?, + } + } +} + +async fn value( + pool: &CellGroupSqlPool, + options: CellSqlOptions, + authority: CellReadAuthority, +) -> Result { + Ok(pool + .query(options, authority, |db, version| { + let n: i64 = db.query_row("SELECT n FROM value", [], |row| row.get(0))?; + assert!(db + .connection() + .execute("UPDATE value SET n=999", []) + .is_err()); + assert!(version.sequence.get() > 0); + Ok(n.to_le_bytes().to_vec()) + }) + .await?) +} + +pub(super) fn verify_observation(output: &CellSqlQueryResult, schema: [u8; 32]) -> Result<()> { + let keys = VersionKeys::new( + VersionSigningKey::new([73; 32], 100_000)?, + vec![], + Duration::from_secs(10), + )?; + let token = keys.issue_cell(output.observation(), schema, 100)?; + assert_eq!( + &keys.verify_cell(&token, &output.version().history, schema, 101)?, + output.observation(), + ); + Ok(()) +} + +pub(crate) async fn coherent(rooted: bool) -> Result<()> { + let (mut fixture, route, a, b) = enrolled_sql_cells(rooted).await?; + let pool = pool(&fixture, 0)?; + let oa = fixture.cell_options(&a).await?; + let ob = fixture.cell_options(&b).await?; + let original = selected(&mut fixture, 0, &a, CellReadConsistency::Stale).await?; + let original_version = original.cell().version.clone(); + let original_observation = original.cell().observation(); + drop(value(&pool, oa.clone(), original).await?); + + // A ready leader without any new peer responses cannot reuse a cached cut. + assert!(fixture.runtimes[0].status(route.history).await?.ready); + assert!(matches!( + fixture.runtimes[0] + .read_authority( + route.history, + a.clone(), + CellReadConsistency::Linearizable, + deadline(&fixture, 50).await? + ) + .await, + Err(RuntimeError::DeadlineExceeded) + )); + fixture.drive(None).await?; + assert!(fixture.runtimes[0] + .read_index(route.history, b"cellar-cell-read-1\0forged".to_vec()) + .await + .is_err()); + + let authority = fixture.runtimes[0] + .execution_authority(route.history, b.clone()) + .await?; + let CellSqlOutcome::Prepared(prepared) = pool + .stage(ob.clone(), authority, cell_attempt(201)?, true, |tx| { + tx.execute("UPDATE value SET n=n+1", [])?; + Ok(b"b".to_vec()) + }) + .await? + else { + anyhow::bail!("B preparation") + }; + let private_b = selected(&mut fixture, 0, &b, CellReadConsistency::Stale).await?; + assert_eq!( + value(&pool, ob.clone(), private_b).await?.result(), + 0_i64.to_le_bytes() + ); + let old_b = selected(&mut fixture, 0, &b, CellReadConsistency::Stale).await?; + let old_b_version = old_b.cell().version.clone(); + let old_b_observation = old_b.cell().observation(); + let encoded = CellGroupEntry::decode(prepared.bytes(), &route, GroupEntryLimits::default())?; + let GroupCommand::Mutation(mutation) = encoded.command else { + anyhow::bail!("B mutation") + }; + let expected_b = CellVersion { + history: b.clone(), + sequence: mutation.sequence, + image: mutation.result.into(), + }; + pool.mark_submitted(b.clone(), prepared.proposal()).await?; + let accepted = accepted_sql(&fixture.runtimes[0], &prepared).await?; + let expected_witness = CommitWitness { + group: route.history, + position: accepted.position, + proposal: prepared.proposal(), + }; + let mut held = BTreeMap::new(); + tokio::time::timeout(Duration::from_secs(5), async { + loop { + fixture.pump(&[route.history], &mut held).await?; + if held.contains_key(&route.history) { + return Ok::<_, anyhow::Error>(()); + } + tokio::task::yield_now().await; + } + }) + .await??; + // A has not changed, but a strong A read must still wait for B's earlier + // committed entry to enter the contiguous group publication prefix. + let runtime = fixture.runtimes[0].clone(); + let pending = runtime.read_authority( + route.history, + a.clone(), + CellReadConsistency::Linearizable, + deadline(&fixture, 5000).await?, + ); + tokio::pin!(pending); + let at_least = runtime.read_authority( + route.history, + b.clone(), + CellReadConsistency::AtLeast(expected_b.clone()), + deadline(&fixture, 5000).await?, + ); + tokio::pin!(at_least); + for _ in 0..30 { + assert!(tokio::time::timeout(Duration::from_millis(2), &mut pending) + .await + .is_err()); + assert!( + tokio::time::timeout(Duration::from_millis(2), &mut at_least) + .await + .is_err() + ); + fixture.pump(&[route.history], &mut held).await?; + } + let work = held.remove(&route.history).context("held B apply")?; + assert_eq!(work.entry.index, accepted.position.index.get()); + let next = prepare(&mut fixture.cluster.voters[0], &work)?; + fixture.runtimes[0] + .complete_apply(route.history, work, next) + .await?; + apply_through(&mut fixture, accepted.position.index).await?; + let selected = tokio::time::timeout(Duration::from_secs(5), pending).await??; + let at_least = tokio::time::timeout(Duration::from_secs(5), at_least).await??; + assert_eq!(at_least.cell().version, expected_b); + drop(value(&pool, ob.clone(), at_least).await?); + let old_output = value(&pool, ob.clone(), old_b).await?; + assert_eq!(old_output.result(), 0_i64.to_le_bytes()); + assert_eq!(old_output.version(), &old_b_version); + assert_eq!(old_output.observation(), &old_b_observation); + verify_observation(&old_output, route.schema_digest)?; + drop(old_output); + assert_eq!(selected.cell().version, original_version); + assert_eq!(selected.cell().observation(), original_observation); + assert!(selected.applied().index >= accepted.position.index); + assert!(selected.barrier().context("fresh barrier")?.index >= accepted.position.index); + assert_eq!( + value(&pool, oa.clone(), selected).await?.result(), + 0_i64.to_le_bytes() + ); + let receipt = pool + .finish( + fixture.runtimes[0] + .execution_authority(route.history, b.clone()) + .await?, + ) + .await?; + assert_eq!(receipt.receipt().witness, expected_witness); + assert_eq!(receipt.receipt().observation().version, expected_b); + drop(receipt); + drop(prepared); + + let version_b = fixture.runtimes[0] + .execution_authority(route.history, b.clone()) + .await? + .cell() + .version + .clone(); + let ready_b = self::selected( + &mut fixture, + 0, + &b, + CellReadConsistency::AtLeast(version_b.clone()), + ) + .await?; + let output = value(&pool, ob, ready_b).await?; + assert_eq!(output.result(), 1_i64.to_le_bytes()); + assert_eq!(output.version(), &version_b); + assert_eq!(output.observation().witness, expected_witness); + verify_observation(&output, route.schema_digest)?; + drop(output); + assert!(fixture.runtimes[0] + .read_authority( + route.history, + a.clone(), + CellReadConsistency::AtLeast(version_b), + deadline(&fixture, 1000).await? + ) + .await + .is_err()); + let mut future = original_version.clone(); + future.sequence = MutationSequence::new(2)?; + assert!(matches!( + fixture.runtimes[0] + .read_authority( + route.history, + a.clone(), + CellReadConsistency::AtLeast(future), + deadline(&fixture, 30).await? + ) + .await, + Err(RuntimeError::DeadlineExceeded) + )); + let mut wrong = a.clone(); + wrong.incarnation = IncarnationId::new([202; 16])?; + assert!(fixture.runtimes[0] + .read_authority( + route.history, + wrong, + CellReadConsistency::Stale, + deadline(&fixture, 1000).await? + ) + .await + .is_err()); + + let strong = self::selected(&mut fixture, 0, &a, CellReadConsistency::Linearizable).await?; + super::reconciliation::transfer(&mut fixture, 0, 1).await?; + assert!(pool + .query(oa.clone(), strong, |_, _| panic!("old leader query")) + .await + .is_err()); + assert!(fixture.runtimes[0] + .read_authority( + route.history, + a.clone(), + CellReadConsistency::Linearizable, + deadline(&fixture, 1000).await? + ) + .await + .is_err()); + let stale = self::selected( + &mut fixture, + 0, + &a, + CellReadConsistency::AtLeast(original_version), + ) + .await?; + drop(value(&pool, oa, stale).await?); + assert_eq!(pool.usage()?.active_cells, 0); + assert_eq!(pool.usage()?.retained_bytes, 0); + pool.close().await?; + fixture.close().await +} + +pub(crate) async fn lifecycle(rooted: bool) -> Result<()> { + let (mut fixture, route, a, _) = enrolled_sql_cells(rooted).await?; + let pool = pool(&fixture, 0)?; + let oa = fixture.cell_options(&a).await?; + let mut pins = Vec::new(); + for _ in 0..32 { + pins.push(selected(&mut fixture, 0, &a, CellReadConsistency::Stale).await?); + } + assert!(matches!( + fixture.runtimes[0] + .read_authority( + route.history, + a.clone(), + CellReadConsistency::Stale, + deadline(&fixture, 1000).await? + ) + .await, + Err(RuntimeError::Overloaded) + )); + drop(pins); + let authority = selected(&mut fixture, 0, &a, CellReadConsistency::Stale).await?; + let foreign = self::pool(&fixture, 1)?; + assert!(foreign + .query(oa.clone(), authority, |_, _| panic!("foreign query")) + .await + .is_err()); + foreign.close().await?; + + let authority = fixture.runtimes[0] + .read_authority( + route.history, + a.clone(), + CellReadConsistency::Stale, + deadline(&fixture, 100).await?, + ) + .await?; + assert!(pool.query(oa.clone(), authority, |db, _| { + let n: i64 = db.query_row("WITH RECURSIVE n(x) AS (VALUES(0) UNION ALL SELECT x+1 FROM n WHERE x<1000000000) SELECT sum(x) FROM n", [], |row| row.get(0))?; + Ok(n.to_le_bytes().to_vec()) + }).await.is_err()); + let authority = selected(&mut fixture, 0, &a, CellReadConsistency::Stale).await?; + drop(value(&pool, oa.clone(), authority).await?); + let authority = selected(&mut fixture, 0, &a, CellReadConsistency::Stale).await?; + assert!(pool + .query(oa.clone(), authority, |_, _| panic!( + "contained query panic" + )) + .await + .is_err()); + let authority = selected(&mut fixture, 0, &a, CellReadConsistency::Stale).await?; + drop(value(&pool, oa.clone(), authority).await?); + + let authority = selected(&mut fixture, 0, &a, CellReadConsistency::Stale).await?; + let (entered, started) = tokio::sync::oneshot::channel(); + let (release, wait) = std::sync::mpsc::sync_channel(1); + let caller = pool.clone(); + let waiter = fixture.cluster.domain.spawn(async move { + caller + .query(oa, authority, move |db, _| { + let _ = entered.send(()); + wait.recv_timeout(Duration::from_secs(10))?; + let n: i64 = db.query_row("SELECT n FROM value", [], |row| row.get(0))?; + Ok(n.to_le_bytes().to_vec()) + }) + .await + })?; + tokio::time::timeout(Duration::from_secs(5), started).await??; + waiter.abort(); + assert!(waiter.await.is_err()); + assert_eq!(pool.usage()?.active_cells, 0); + assert_eq!(pool.usage()?.retained_operations, 1); + assert!( + tokio::time::timeout(Duration::from_millis(30), pool.close()) + .await + .is_err() + ); + release.send(())?; + tokio::time::timeout(Duration::from_secs(5), pool.close()).await??; + assert_eq!(pool.usage()?.retained_operations, 0); + assert_eq!(pool.usage()?.retained_bytes, 0); + assert_eq!(pool.usage()?.running_workers, 0); + fixture.close().await +} + +/// A follower must obtain its own correlated quorum barrier, apply through it, +/// and retain only that invocation's authority across SQL work. +pub(crate) async fn follower_linearizable(rooted: bool) -> Result<()> { + let (mut fixture, route, a, _) = enrolled_sql_cells(rooted).await?; + let pool = pool(&fixture, 1)?; + let options = fixture.cell_options_at(&a, 1).await?; + let authority = selected(&mut fixture, 1, &a, CellReadConsistency::Linearizable).await?; + let barrier = authority.barrier().context("follower correlated barrier")?; + assert!(authority.applied().index >= barrier.index); + assert_eq!(authority.cell().version.history, a); + let output = value(&pool, options.clone(), authority).await?; + assert_eq!( + output.replica(), + fixture.cluster.voters[1] + .log + .cell_group(&route.history)? + .identity() + ); + drop(output); + + // No response from a live quorum: a previous successful read cannot be reused. + assert!(matches!( + fixture.runtimes[1] + .read_authority( + route.history, + a.clone(), + CellReadConsistency::Linearizable, + deadline(&fixture, 50).await?, + ) + .await, + Err(RuntimeError::DeadlineExceeded) + )); + fixture.drive(None).await?; + + let mut forged = b"cellar-cell-forward-read-1\0".to_vec(); + forged.extend_from_slice(&3_u64.to_be_bytes()); + forged.extend_from_slice(&[41; 16]); + forged.extend_from_slice(&1_u64.to_be_bytes()); + let term = fixture.runtimes[0].status(route.history).await?.term.get(); + let message = Message { + msg_type: MessageType::MsgReadIndex, + from: 2, + to: 1, + term, + entries: vec![Entry { + data: forged.into(), + ..Default::default() + }] + .into(), + ..Default::default() + }; + assert!( + fixture.runtimes[0] + .step(route.history, message) + .await + .is_err(), + "forwarded read context must name its authenticated origin" + ); + assert!( + fixture.runtimes[0] + .read_index( + route.history, + b"cellar-cell-forward-read-1\0forged".to_vec(), + ) + .await + .is_err(), + "public contexts cannot enter the private cell-read namespace" + ); + + let retained = selected(&mut fixture, 1, &a, CellReadConsistency::Linearizable).await?; + super::reconciliation::transfer(&mut fixture, 0, 2).await?; + assert!(pool + .query(options, retained, |_, _| panic!( + "revoked follower authority" + )) + .await + .is_err()); + pool.close().await?; + fixture.close().await +} + +/// Source-age bounds follow the physical query pin, including entered callbacks. +pub(crate) async fn recent_source_reads(rooted: bool) -> Result<()> { + let (mut fixture, route, a, _) = enrolled_sql_cells(rooted).await?; + let pool = pool(&fixture, 1)?; + let options = fixture.cell_options_at(&a, 1).await?; + // A fresh actual leader heartbeat, not an advisory catalog refresh. + for _ in 0..2 { + fixture.runtimes[0].tick().await?; + fixture.drive(None).await?; + } + // This read needs no network round. The generic selected() helper pumps + // peer traffic concurrently and may admit another heartbeat after capture. + let authority = fixture.runtimes[1] + .read_authority( + route.history, + a.clone(), + CellReadConsistency::RecentSource { max_age_ms: 100 }, + deadline(&fixture, 5000).await?, + ) + .await?; + let expiry = authority.deadline(); + let (entered, started) = tokio::sync::oneshot::channel(); + let (release, blocked) = std::sync::mpsc::sync_channel(1); + let reader = pool.clone(); + let held_options = options.clone(); + let pending = fixture.cluster.domain.spawn(async move { + reader + .query(held_options, authority, move |db, _| { + let _ = entered.send(()); + blocked.recv_timeout(Duration::from_secs(5))?; + let value: i64 = db.query_row("SELECT n FROM value", [], |row| row.get(0))?; + Ok(value.to_le_bytes().to_vec()) + }) + .await + })?; + tokio::time::timeout(Duration::from_secs(5), started).await??; + fixture + .cluster + .domain + .run(async { cellar::asyncrt::sleep_until(expiry).await }) + .await?; + assert!( + pending.await?.is_err(), + "old source observation cannot authorize a late reply" + ); + assert_eq!( + pool.usage()?.retained_operations, + 1, + "expired waiter retains entered physical SQL" + ); + release.send(())?; + tokio::time::timeout(Duration::from_secs(5), async { + while pool.usage()?.retained_operations != 0 { + tokio::task::yield_now().await; + } + Ok::<_, anyhow::Error>(()) + }) + .await??; + // A stale-term heartbeat cannot refresh the current source observation. + let term = fixture.runtimes[1].status(route.history).await?.term.get(); + assert!(term > 1); + fixture.runtimes[1] + .step( + route.history, + Message { + msg_type: MessageType::MsgHeartbeat, + from: 1, + to: 2, + term: term - 1, + ..Default::default() + }, + ) + .await?; + assert!(matches!( + fixture.runtimes[1] + .read_authority( + route.history, + a.clone(), + CellReadConsistency::RecentSource { max_age_ms: 100 }, + deadline(&fixture, 30).await? + ) + .await, + Err(RuntimeError::DeadlineExceeded) + )); + // Historical reads are a separate explicit policy, never the fallback above. + let authority = selected(&mut fixture, 1, &a, CellReadConsistency::Stale).await?; + drop(value(&pool, options.clone(), authority).await?); + for _ in 0..2 { + fixture.runtimes[0].tick().await?; + fixture.drive(None).await?; + } + let authority = selected( + &mut fixture, + 1, + &a, + CellReadConsistency::RecentSource { max_age_ms: 1000 }, + ) + .await?; + drop(value(&pool, options, authority).await?); + pool.close().await?; + fixture.close().await +} diff --git a/crates/cellar/tests/support/cell_group_sql_reconcile.rs b/crates/cellar/tests/support/cell_group_sql_reconcile.rs new file mode 100644 index 000000000..f80bf0276 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_sql_reconcile.rs @@ -0,0 +1,561 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; + +pub(super) async fn transfer(fixture: &mut RuntimeCluster, from: usize, to: usize) -> Result<()> { + let cell = super::history(fixture.cluster.birth.route(), "sql-a")?; + Box::pin(transfer_cell(fixture, from, to, cell)).await +} + +pub(super) async fn transfer_cell( + fixture: &mut RuntimeCluster, + from: usize, + to: usize, + cell: CellHistory, +) -> Result<()> { + let history = fixture.cluster.birth.route().history; + let term = fixture.runtimes[from].status(history).await?.term; + let mut message = Message::default(); + // The current leader's transfer directive triggers a real higher-term + // election. MsgTransferLeader itself is a local administration request and + // is deliberately rejected by the authenticated peer-message boundary. + message.set_msg_type(MessageType::MsgTimeoutNow); + message.from = from as u64 + 1; + message.to = to as u64 + 1; + message.term = term.get(); + fixture.runtimes[to].step(history, message).await?; + if from == 0 { + let mut held = BTreeMap::new(); + tokio::time::timeout(Duration::from_secs(20), async { + loop { + fixture.pump(&[history], &mut held).await?; + if fixture.runtimes[to].status(history).await?.ready && held.contains_key(&history) + { + return Ok::<_, anyhow::Error>(()); + } + tokio::task::yield_now().await; + } + }) + .await??; + assert!( + matches!( + fixture.runtimes[0] + .reconciliation_authority(history, cell) + .await, + Err(RuntimeError::NotReady) + ), + "a persisted newer term without its applied barrier cannot retire a producer" + ); + let work = held.remove(&history).context("held new-term barrier")?; + if let Some((pool, applier)) = fixture.appliers.get(&(0, history)) { + Box::pin(async { + pool.prepare_apply(applier, work) + .await? + .complete(&fixture.runtimes[0]) + .await?; + Ok::<_, anyhow::Error>(()) + }) + .await?; + } else { + let next = prepare(&mut fixture.cluster.voters[0], &work)?; + fixture.runtimes[0] + .complete_apply(history, work, next) + .await?; + } + } + tokio::time::timeout(Duration::from_secs(20), async { + loop { + fixture.pump(&[], &mut BTreeMap::new()).await?; + let status = fixture.runtimes[to].status(history).await?; + if status.ready + && status.term > term + && fixture.applied[&history] + .iter() + .all(|index| *index >= status.applied.index) + { + return Ok::<_, anyhow::Error>(()); + } + tokio::task::yield_now().await; + } + }) + .await? +} + +fn execution_limits(options: &mut CellSqlOptions) -> &mut ExecutionLimits { + match options { + CellSqlOptions::Materialized(o) => &mut o.limits, + CellSqlOptions::Extents(o) => &mut o.limits, + } +} + +pub(super) async fn apply_through(fixture: &mut RuntimeCluster, index: LogIndex) -> Result<()> { + let group = fixture.cluster.birth.route().history; + tokio::time::timeout(Duration::from_secs(20), async { + loop { + fixture.pump(&[], &mut BTreeMap::new()).await?; + if fixture.applied[&group] + .iter() + .all(|applied| *applied >= index) + { + return Ok::<_, anyhow::Error>(()); + } + tokio::task::yield_now().await; + } + }) + .await? +} + +pub(super) async fn commit_on( + fixture: &mut RuntimeCluster, + pool: &CellGroupSqlPool, + node: usize, + cell: &CellHistory, + prepared: CellSqlPrepared, +) -> Result { + let group = fixture.cluster.birth.route().history; + pool.mark_submitted(cell.clone(), prepared.proposal()) + .await?; + let accepted = accepted_sql(&fixture.runtimes[node], &prepared).await?; + apply_through(fixture, accepted.position.index).await?; + let authority = fixture.runtimes[node] + .execution_authority(group, cell.clone()) + .await?; + Ok(pool.finish(authority).await?) +} + +pub(super) async fn reconcile(rooted: bool) -> Result<()> { + let (mut fixture, route, a, b) = enrolled_sql_cells(rooted).await?; + let pool = CellGroupSqlPool::start( + &fixture.cluster.voters[0].log, + CellSqlLimits { + workers: 2, + queue_per_worker: 2, + max_pending_cells: 3, + group_pending_cells: 3, + node_bytes: 512 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + }, + )?; + let mut prepared = Vec::new(); + for (cell, tag) in [(&a, 111), (&b, 112)] { + let authority = fixture.runtimes[0] + .execution_authority(route.history, cell.clone()) + .await?; + let mut options = fixture.cell_options(cell).await?; + execution_limits(&mut options).max_client_identities = 1; + let CellSqlOutcome::Prepared(command) = pool + .stage(options, authority, cell_attempt(tag)?, true, |tx| { + tx.execute("UPDATE value SET n=n+1", [])?; + Ok(b"reconciled".to_vec()) + }) + .await? + else { + anyhow::bail!("prepared") + }; + pool.mark_submitted(cell.clone(), command.proposal()) + .await?; + prepared.push(command); + } + // A's submission response is unknown. The same-term applied prefix is not + // enough to release it, even though its receipt is currently absent. + let same_term = fixture.runtimes[0] + .reconciliation_authority(route.history, a.clone()) + .await?; + assert!(matches!( + pool.reconcile(prepared[0].proposal(), same_term).await, + Err(ExecutionError::Uncertain) + )); + assert_eq!(pool.usage()?.active_cells, 2); + let stale_b = fixture.runtimes[0] + .reconciliation_authority(route.history, b.clone()) + .await?; + let submitted = accepted_sql(&fixture.runtimes[0], &prepared[1]).await?; + fixture.drive(Some(submitted.position.index)).await?; + assert!( + stale_b.check().is_err(), + "an old cell pin cannot prove absence after commit" + ); + let stale = pool.reconcile(prepared[1].proposal(), stale_b).await; + assert!( + matches!(&stale, Err(ExecutionError::Uncertain)), + "stale observation must remain retryable: {:?}", + stale.as_ref().err() + ); + assert_eq!(pool.usage()?.active_cells, 2); + + // B committed; deliberately lose its exact-entry completion. Transfer real + // leadership so node 0 must reconcile while it is a follower. + transfer(&mut fixture, 0, 1).await?; + assert!(fixture.runtimes[0] + .execution_authority(route.history, a.clone()) + .await + .is_err()); + // The new leader advances B again while the old node retains B's first + // unresolved producer. Reconciliation must find the original receipt in a + // later image, rather than demand that image equal the old private result. + let successor = CellGroupSqlPool::start( + &fixture.cluster.voters[1].log, + CellSqlLimits { + workers: 1, + queue_per_worker: 2, + max_pending_cells: 2, + group_pending_cells: 2, + node_bytes: 256 * 1024 * 1024, + group_bytes: 256 * 1024 * 1024, + }, + )?; + let authority = fixture.runtimes[1] + .execution_authority(route.history, b.clone()) + .await?; + let mut attempt = cell_attempt(115)?; + attempt.request.id = CommandId::new(ClientId::new([81; 16])?, 2)?; + let CellSqlOutcome::Prepared(next) = successor + .stage( + fixture.cell_options_at(&b, 1).await?, + authority, + attempt, + true, + |tx| { + tx.execute("UPDATE value SET n=n+10", [])?; + Ok(b"later-b".to_vec()) + }, + ) + .await? + else { + anyhow::bail!("successor command") + }; + successor.mark_submitted(b.clone(), next.proposal()).await?; + let accepted = fixture.runtimes[1] + .propose(route.history, next.bytes().to_vec()) + .await?; + apply_through(&mut fixture, accepted.position.index).await?; + let authority = fixture.runtimes[1] + .execution_authority(route.history, b.clone()) + .await?; + let later = successor.finish(authority).await?; + assert_eq!(later.receipt().receipt.result(), b"later-b"); + drop(later); + drop(next); + // Fill A's client identity capacity with a different committed client. + // Its changed image forces receipt lookup, not the unchanged-base shortcut. + let authority = fixture.runtimes[1] + .execution_authority(route.history, a.clone()) + .await?; + let mut attempt = cell_attempt(116)?; + attempt.request.id = CommandId::new(ClientId::new([99; 16])?, 1)?; + let CellSqlOutcome::Prepared(next) = successor + .stage( + fixture.cell_options_at(&a, 1).await?, + authority, + attempt, + true, + |tx| { + tx.execute("UPDATE value SET n=n+5", [])?; + Ok(b"other-client".to_vec()) + }, + ) + .await? + else { + anyhow::bail!("other client command") + }; + drop(commit_on(&mut fixture, &successor, 1, &a, next).await?); + successor.close().await?; + drop(successor); + let foreign = fixture.runtimes[1] + .reconciliation_authority(route.history, a.clone()) + .await?; + assert!(matches!( + pool.reconcile(prepared[0].proposal(), foreign).await, + Err(ExecutionError::Invalid(_)) + )); + let wrong = fixture.runtimes[0] + .reconciliation_authority(route.history, b.clone()) + .await?; + assert!(matches!( + pool.reconcile(prepared[0].proposal(), wrong).await, + Err(ExecutionError::Invalid(_)) + )); + assert_eq!(pool.usage()?.active_cells, 2); + let proof = fixture.runtimes[0] + .reconciliation_authority(route.history, a.clone()) + .await?; + assert!(matches!( + pool.reconcile(prepared[0].proposal(), proof).await?, + CellSqlReconciliation::NotCommitted + )); + assert_eq!(pool.usage()?.active_cells, 1); + let proof = fixture.runtimes[0] + .reconciliation_authority(route.history, b.clone()) + .await?; + let CellSqlReconciliation::Committed(receipt) = + pool.reconcile(prepared[1].proposal(), proof).await? + else { + anyhow::bail!("committed receipt") + }; + assert_eq!(receipt.receipt().receipt.result(), b"reconciled"); + assert_eq!(receipt.receipt().version.history, b); + assert_eq!(receipt.receipt().version.sequence.get(), 3); + assert_eq!(receipt.receipt().position, accepted.position); + assert_eq!(pool.usage()?.active_cells, 0); + assert!( + pool.usage()?.retained_bytes > 0, + "responses retain admission" + ); + // Losing a reconciliation response cannot be reported as definite failure. + let proof = fixture.runtimes[0] + .reconciliation_authority(route.history, b.clone()) + .await?; + assert!(matches!( + pool.reconcile(prepared[1].proposal(), proof).await, + Err(ExecutionError::Uncertain) + )); + assert!(matches!( + pool.resume(b.clone(), prepared[1].proposal()).await, + Err(ExecutionError::Uncertain) + )); + drop(receipt); + drop(prepared); + assert_eq!(pool.usage()?.retained_bytes, 0); + + transfer(&mut fixture, 1, 0).await?; + // B retries through the committed receipt without invoking the callback. + let authority = fixture.runtimes[0] + .execution_authority(route.history, b.clone()) + .await?; + let CellSqlOutcome::Receipt(receipt) = pool + .stage( + fixture.cell_options(&b).await?, + authority, + cell_attempt(113)?, + false, + |_| panic!("duplicate callback"), + ) + .await? + else { + anyhow::bail!("retry receipt") + }; + assert_eq!(receipt.receipt().receipt.result(), b"reconciled"); + drop(receipt); + // Read-only absence did not relax fresh command admission at capacity. + let authority = fixture.runtimes[0] + .execution_authority(route.history, a.clone()) + .await?; + let mut full = fixture.cell_options(&a).await?; + execution_limits(&mut full).max_client_identities = 1; + assert!(matches!( + pool.stage(full, authority, cell_attempt(117)?, true, |_| panic!( + "new client at capacity" + )) + .await, + Err(ExecutionError::Refused(CommandFailure::Overloaded)) + )); + assert_eq!(pool.usage()?.active_cells, 0); + // With sufficient capacity, reopen from the committed image, not the old + // private write. The other client's committed value must remain visible. + let authority = fixture.runtimes[0] + .execution_authority(route.history, a.clone()) + .await?; + let CellSqlOutcome::Prepared(command) = pool + .stage( + fixture.cell_options(&a).await?, + authority, + cell_attempt(114)?, + true, + |tx| { + let value: i64 = tx.query_row("SELECT n FROM value", [], |row| row.get(0))?; + assert_eq!(value, 5, "abandoned private writes must not be reused"); + tx.execute("UPDATE value SET n=n+1", [])?; + Ok(b"retry-a".to_vec()) + }, + ) + .await? + else { + anyhow::bail!("retry A") + }; + pool.mark_submitted(a.clone(), command.proposal()).await?; + let submitted = accepted_sql(&fixture.runtimes[0], &command).await?; + fixture.drive(Some(submitted.position.index)).await?; + // A positive same-term receipt resolves without requiring a term change. + let proof = fixture.runtimes[0] + .reconciliation_authority(route.history, a.clone()) + .await?; + let CellSqlReconciliation::Committed(receipt) = + pool.reconcile(command.proposal(), proof).await? + else { + anyhow::bail!("same-term committed receipt") + }; + assert_eq!(receipt.receipt().receipt.result(), b"retry-a"); + drop(receipt); + drop(command); + assert_eq!(pool.usage()?.retained_bytes, 0); + pool.close().await?; + drop(pool); + fixture.close().await +} + +pub(super) async fn refusals(rooted: bool) -> Result<()> { + let (mut fixture, route, a, b) = enrolled_sql_cells(rooted).await?; + let limits = CellSqlLimits { + workers: 2, + queue_per_worker: 2, + max_pending_cells: 3, + group_pending_cells: 3, + node_bytes: 512 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + }; + let pool = CellGroupSqlPool::start(&fixture.cluster.voters[0].log, limits)?; + let mut pending = Vec::new(); + for (cell, tag) in [(&a, 121), (&b, 122)] { + let authority = fixture.runtimes[0] + .execution_authority(route.history, cell.clone()) + .await?; + let CellSqlOutcome::Prepared(prepared) = pool + .stage( + fixture.cell_options(cell).await?, + authority, + cell_attempt(tag)?, + true, + |tx| { + tx.execute("UPDATE value SET n=n+1", [])?; + Ok(b"original".to_vec()) + }, + ) + .await? + else { + anyhow::bail!("original command") + }; + pool.mark_submitted(cell.clone(), prepared.proposal()) + .await?; + pending.push(prepared); + } + // B commits but its response is lost; A's original proposal never commits. + let accepted = accepted_sql(&fixture.runtimes[0], &pending[1]).await?; + apply_through(&mut fixture, accepted.position.index).await?; + transfer(&mut fixture, 0, 1).await?; + let successor = CellGroupSqlPool::start(&fixture.cluster.voters[1].log, limits)?; + let authority = fixture.runtimes[1] + .execution_authority(route.history, a.clone()) + .await?; + let mut conflict = cell_attempt(123)?; + conflict.request.request_digest = [88; 32]; + let CellSqlOutcome::Prepared(prepared) = successor + .stage( + fixture.cell_options_at(&a, 1).await?, + authority, + conflict, + true, + |tx| { + tx.execute("UPDATE value SET n=n+10", [])?; + Ok(b"different-request".to_vec()) + }, + ) + .await? + else { + anyhow::bail!("conflicting identity winner") + }; + drop(commit_on(&mut fixture, &successor, 1, &a, prepared).await?); + // Preserve the latest receipt while replicated maintenance expires B's + // first receipt. At command 3 the two-row ceiling triggers bounded pruning. + for sequence in 2..=3 { + let authority = fixture.runtimes[1] + .execution_authority(route.history, b.clone()) + .await?; + let mut options = fixture.cell_options_at(&b, 1).await?; + execution_limits(&mut options).max_retained_receipts = 2; + let mut attempt = cell_attempt(123 + sequence as u8)?; + attempt.request.id = CommandId::new(ClientId::new([81; 16])?, sequence)?; + if sequence == 3 { + attempt.inputs.now_unix_millis = 2000; + attempt.request.retry_until_ms = 3000; + } + let CellSqlOutcome::Prepared(prepared) = successor + .stage(options, authority, attempt, true, |tx| { + tx.execute("UPDATE value SET n=n+1", [])?; + Ok(b"later".to_vec()) + }) + .await? + else { + anyhow::bail!("maintenance command") + }; + drop(commit_on(&mut fixture, &successor, 1, &b, prepared).await?); + } + successor.close().await?; + drop(successor); + let proof = fixture.runtimes[0] + .reconciliation_authority(route.history, a.clone()) + .await?; + assert!(matches!( + pool.reconcile(pending[0].proposal(), proof).await?, + CellSqlReconciliation::Refused(CommandFailure::CommandConflict) + )); + let proof = fixture.runtimes[0] + .reconciliation_authority(route.history, b.clone()) + .await?; + assert!(matches!( + pool.reconcile(pending[1].proposal(), proof).await?, + CellSqlReconciliation::Refused(CommandFailure::CommandExpired) + )); + assert_eq!(pool.usage()?.active_cells, 0); + assert!(pool.usage()?.retained_bytes > 0); + drop(pending); + assert_eq!(pool.usage()?.retained_bytes, 0); + transfer(&mut fixture, 1, 0).await?; + for cell in [&a, &b] { + let authority = fixture.runtimes[0] + .execution_authority(route.history, cell.clone()) + .await?; + let error = pool + .stage( + fixture.cell_options(cell).await?, + authority, + cell_attempt(128)?, + true, + |_| panic!("refused receipt cannot replay the callback"), + ) + .await; + if cell == &a { + assert!(matches!( + error, + Err(ExecutionError::Refused(CommandFailure::CommandConflict)) + )); + } else { + assert!(matches!( + error, + Err(ExecutionError::Refused(CommandFailure::CommandExpired)) + )); + } + assert_eq!(pool.usage()?.active_cells, 0); + assert_eq!(pool.usage()?.retained_bytes, 0); + } + let authority = fixture.runtimes[0] + .execution_authority(route.history, b.clone()) + .await?; + let mut next = cell_attempt(129)?; + next.request.id = CommandId::new(ClientId::new([81; 16])?, 4)?; + next.inputs.now_unix_millis = 2000; + next.request.retry_until_ms = 3000; + let CellSqlOutcome::Prepared(prepared) = pool + .stage( + fixture.cell_options(&b).await?, + authority, + next, + true, + |tx| { + let value: i64 = tx.query_row("SELECT n FROM value", [], |row| row.get(0))?; + assert_eq!(value, 3); + tx.execute("UPDATE value SET n=n+1", [])?; + Ok(b"after-expiry".to_vec()) + }, + ) + .await? + else { + anyhow::bail!("next command after refused retry") + }; + let receipt = commit_on(&mut fixture, &pool, 0, &b, prepared).await?; + assert_eq!(receipt.receipt().version.sequence.get(), 5); + assert_eq!(receipt.receipt().receipt.result(), b"after-expiry"); + drop(receipt); + pool.close().await?; + drop(pool); + fixture.close().await +} diff --git a/crates/cellar/tests/support/cell_group_sql_registered.rs b/crates/cellar/tests/support/cell_group_sql_registered.rs new file mode 100644 index 000000000..fbafd7f40 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_sql_registered.rs @@ -0,0 +1,288 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Registered callbacks, logical identity and receipts through real shared Raft. +use super::*; +use std::sync::atomic::{AtomicUsize, Ordering}; + +pub(crate) async fn commands(rooted: bool) -> Result<()> { + let calls = Arc::new(AtomicUsize::new(0)); + let observed = calls.clone(); + let class = Arc::new( + RaftClass::new("Counter", 1, [41; 32])? + .alarm("Wake", [42; 32], |_, _| Ok(()))? + .command("Increment", [43; 32], move |context, input| { + observed.fetch_add(1, Ordering::SeqCst); + assert_eq!(context.command().sequence(), 1); + assert_eq!(context.inputs().random, [17; 32]); + assert_eq!(context.identity().incarnation, IncarnationId::new([8; 16])?); + let delta = i64::from_be_bytes(input.try_into()?); + context.sql().execute("UPDATE value SET n=n+?1", [delta])?; + context.set_alarm("Wake", 50, b"wake")?; + Ok(context.identity().cell.as_str().as_bytes().to_vec()) + })?, + ); + let (mut fixture, route, a, b) = Box::pin(setup(rooted, &class)).await?; + let pool = fixture.appliers[&(0, route.history)].0.clone(); + let input = 7_i64.to_be_bytes().to_vec(); + let attempt = |tag| -> Result { + let mut attempt = cell_attempt(tag)?; + attempt.request.operation = "Increment".into(); + attempt.request.contract_digest = [43; 32]; + attempt.request.request_digest = command_request_digest(&input); + Ok(attempt) + }; + for (n, cell) in [&a, &b].into_iter().enumerate() { + let authority = fixture.runtimes[0] + .execution_authority(route.history, cell.clone()) + .await?; + let CellSqlOutcome::Prepared(prepared) = pool + .stage_registered( + fixture.cell_options(cell).await?, + authority, + attempt(150 + n as u8)?, + class.clone(), + input.clone(), + true, + ) + .await? + else { + anyhow::bail!("prepared registered command") + }; + pool.mark_submitted(cell.clone(), prepared.proposal()) + .await?; + let accepted = accepted_sql(&fixture.runtimes[0], &prepared).await?; + fixture.drive(Some(accepted.position.index)).await?; + let authority = fixture.runtimes[0] + .execution_authority(route.history, cell.clone()) + .await?; + let receipt = pool.finish(authority).await?; + assert_eq!( + receipt.receipt().receipt.result(), + cell.cell.as_str().as_bytes() + ); + assert_eq!(receipt.receipt().version.history, *cell); + assert_eq!(receipt.receipt().version.sequence.get(), 2); + drop(receipt); + drop(prepared); + for runtime in &fixture.runtimes { + let selected = runtime + .reconciliation_authority(route.history, cell.clone()) + .await?; + assert_eq!(selected.cell().effects.next_alarm_unix_millis, Some(50)); + assert_eq!(selected.cell().version.sequence.get(), 2); + } + } + assert_eq!(calls.load(Ordering::SeqCst), 2); + // A's exact receipt remains valid after B advances the group; no callback reruns. + let authority = fixture.runtimes[0] + .execution_authority(route.history, a.clone()) + .await?; + let CellSqlOutcome::Receipt(receipt) = pool + .stage_registered( + fixture.cell_options(&a).await?, + authority, + attempt(155)?, + class.clone(), + input.clone(), + false, + ) + .await? + else { + anyhow::bail!("registered duplicate") + }; + assert_eq!( + receipt.receipt().receipt.result(), + a.cell.as_str().as_bytes() + ); + drop(receipt); + for fault in 0..4 { + let mut request = attempt(156 + fault)?; + let mut supplied = input.clone(); + let mut definition = class.clone(); + match fault { + 0 => request.request.contract_digest = [44; 32], + 1 => supplied[0] ^= 1, + 2 => request.request.operation = "Missing".into(), + _ => { + definition = Arc::new(RaftClass::new("Counter", 1, [45; 32])?.command( + "Increment", + [43; 32], + |_, _| panic!("wrong schema callback"), + )?) + } + } + let authority = fixture.runtimes[0] + .execution_authority(route.history, a.clone()) + .await?; + let result = pool + .stage_registered( + fixture.cell_options(&a).await?, + authority, + request, + definition, + supplied, + false, + ) + .await; + match fault { + 0 | 3 => assert!(matches!( + result, + Err(ExecutionError::Registration( + RegistrationError::ContractMismatch + )) + )), + 1 => assert!(matches!( + result, + Err(ExecutionError::Refused(CommandFailure::CommandConflict)) + )), + 2 => assert!(matches!( + result, + Err(ExecutionError::Registration( + RegistrationError::UnknownOperation(_) + )) + )), + _ => unreachable!(), + } + assert_eq!(pool.usage()?.active_cells, 0); + } + assert_eq!(calls.load(Ordering::SeqCst), 2); + assert_eq!(pool.usage()?.retained_bytes, 0); + drop(pool); + fixture.close().await +} + +pub(super) async fn setup( + rooted: bool, + class: &RaftClass, +) -> Result<(RuntimeCluster, GroupRoute, CellHistory, CellHistory)> { + setup_in( + rooted, + class, + tempfile::tempdir()?, + Arc::new(DirectFileSystem), + ) + .await +} + +pub(super) async fn setup_in( + rooted: bool, + class: &RaftClass, + root: tempfile::TempDir, + filesystem: Arc, +) -> Result<(RuntimeCluster, GroupRoute, CellHistory, CellHistory)> { + let cluster = Cluster::with_store_birth( + RaftRuntimeOptions::default(), + root, + filesystem.clone(), + class.schema_digest(), + Arc::new(object_store::memory::InMemory::new()), + ClusterBirthOptions { + group: GroupId::new([3; 16])?, + image_storage: if rooted { + RaftImageStorage::Extents + } else { + RaftImageStorage::Materialized + }, + minimum: Durability::Quorum, + log: ClusterLogLimits::default(), + }, + ) + .await?; + setup_cluster(rooted, cluster, filesystem).await +} + +pub(super) async fn setup_store( + rooted: bool, + class: &RaftClass, + store: Arc, +) -> Result<(RuntimeCluster, GroupRoute, CellHistory, CellHistory)> { + setup_store_minimum(rooted, class, store, Durability::Quorum).await +} + +pub(super) async fn setup_store_minimum( + rooted: bool, + class: &RaftClass, + store: Arc, + minimum: Durability, +) -> Result<(RuntimeCluster, GroupRoute, CellHistory, CellHistory)> { + let filesystem = Arc::new(DirectFileSystem); + let cluster = Cluster::with_store_birth( + RaftRuntimeOptions::default(), + tempfile::tempdir()?, + filesystem.clone(), + class.schema_digest(), + store, + ClusterBirthOptions { + group: GroupId::new([3; 16])?, + image_storage: if rooted { + RaftImageStorage::Extents + } else { + RaftImageStorage::Materialized + }, + minimum, + log: ClusterLogLimits::default(), + }, + ) + .await?; + setup_cluster(rooted, cluster, filesystem).await +} + +pub(super) async fn setup_cluster( + rooted: bool, + cluster: Cluster, + filesystem: Arc, +) -> Result<(RuntimeCluster, GroupRoute, CellHistory, CellHistory)> { + setup_cluster_named( + rooted, + cluster, + filesystem, + ["registered-a", "registered-b"], + ) + .await +} + +pub(super) async fn setup_cluster_named( + rooted: bool, + cluster: Cluster, + filesystem: Arc, + names: [&str; 2], +) -> Result<(RuntimeCluster, GroupRoute, CellHistory, CellHistory)> { + assert_eq!( + cluster.directory.namespace_config().image_storage, + if rooted { + RaftImageStorage::Extents + } else { + RaftImageStorage::Materialized + }, + "fixture backend must match its immutable namespace capability", + ); + let mut fixture = Box::pin(RuntimeCluster::from_cluster(cluster, 256, 256, 1_000_000)).await?; + let route = fixture.cluster.birth.route().clone(); + if rooted { + for (n, voter) in fixture.cluster.voters.iter_mut().enumerate() { + voter.extents = Some(NodeExtentStore::open( + fixture.cluster._root.path().join(format!("rooted-{n}")), + filesystem.clone(), + Default::default(), + )?); + } + } + let a = history(&route, names[0])?; + let b = history(&route, names[1])?; + for node in 0..3 { + apply::install(&mut fixture, &a, node).await?; + } + fixture.runtimes[0].campaign(route.history).await?; + fixture.drive(None).await?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(0);")?; + for cell in [&a, &b] { + let accepted = fixture.runtimes[0] + .propose( + route.history, + apply::enrollment_bytes(&route, cell, &bytes, rooted)?, + ) + .await?; + fixture.drive(Some(accepted.position.index)).await?; + } + Ok((fixture, route, a, b)) +} diff --git a/crates/cellar/tests/support/cell_group_sql_registered_queries.rs b/crates/cellar/tests/support/cell_group_sql_registered_queries.rs new file mode 100644 index 000000000..30cefe1f6 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_sql_registered_queries.rs @@ -0,0 +1,286 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Registered queries keep actual cell versions, immutable images and ownership. +use super::*; +use std::sync::{ + atomic::{AtomicUsize, Ordering}, + Mutex, +}; + +fn verify( + output: &CellSqlQueryResult, + value: i64, + expected: &CellVersionObservation, + schema: [u8; 32], +) -> Result<()> { + let bytes = output.result(); + assert_eq!(bytes.len(), 24); + assert_eq!(i64::from_le_bytes(bytes[..8].try_into()?), value); + assert_eq!( + u64::from_le_bytes(bytes[8..16].try_into()?), + expected.version.sequence.get() + ); + assert_eq!( + u64::from_le_bytes(bytes[16..].try_into()?), + expected.witness.position.index.get() + ); + assert_eq!(output.observation(), expected); + reads::verify_observation(output, schema) +} + +async fn query_released(pool: &CellGroupSqlPool) -> Result<()> { + tokio::time::timeout(Duration::from_secs(3), async { + loop { + let usage = pool.usage()?; + if usage.queued_or_running == 0 && usage.retained_operations == 0 { + return Ok::<_, anyhow::Error>(()); + } + tokio::task::yield_now().await; + } + }) + .await + .context("query physical ownership did not settle")? +} + +pub(crate) async fn queries(rooted: bool) -> Result<()> { + let calls = Arc::new(AtomicUsize::new(0)); + let observed = calls.clone(); + let (entered, started) = tokio::sync::oneshot::channel(); + let entered = Mutex::new(Some(entered)); + let (release, wait) = std::sync::mpsc::sync_channel(1); + let wait = Mutex::new(wait); + let class = Arc::new(RaftClass::new("Counter", 1, [51; 32])? + .command("Increment", [52; 32], |context, _| { + context.sql().execute("UPDATE value SET n=n+7", [])?; + Ok(vec![7]) + })? + .query("Get", [53; 32], move |context, input| { + observed.fetch_add(1, Ordering::SeqCst); + let QueryVersion::Cell(observation) = context.version() else { anyhow::bail!("shared query received a group cut") }; + assert_eq!(context.identity(), &observation.version.history); + for sql in ["UPDATE value SET n=999 RETURNING n", "PRAGMA user_version=8", "ATTACH DATABASE ':memory:' AS other", "SELECT count(*) FROM __sys_raft_receipts"] { + assert!(context.query_row(sql, [], |row| row.get::<_, i64>(0)).is_err(), "{sql}"); + } + if input == b"panic" { panic!("contained registered query panic"); } + if input == b"spin" { + let _: i64 = context.query_row("WITH RECURSIVE n(x) AS (VALUES(0) UNION ALL SELECT x+1 FROM n WHERE x<1000000000) SELECT sum(x) FROM n", [], |row| row.get(0))?; + } + if input.starts_with(b"hold") { + entered.lock().unwrap().take().context("one held invocation")?.send(()).map_err(|_| anyhow::anyhow!("held receiver"))?; + wait.lock().unwrap().recv_timeout(Duration::from_secs(10))?; + } + let value: i64 = context.query_row("SELECT n FROM value", [], |row| row.get(0))?; + let mut result = value.to_le_bytes().to_vec(); + result.extend_from_slice(&observation.version.sequence.get().to_le_bytes()); + result.extend_from_slice(&observation.witness.position.index.get().to_le_bytes()); + Ok(result) + })?); + let (mut fixture, route, a, b) = Box::pin(registered::setup(rooted, &class)).await?; + let pool = fixture.appliers[&(0, route.history)].0.clone(); + let contract = class.contract("Get").context("Get")?; + let options = fixture.cell_options(&a).await?; + let old = reads::selected(&mut fixture, 0, &a, CellReadConsistency::Stale).await?; + let original = old.cell().observation(); + for (n, cell) in [&a, &b].into_iter().enumerate() { + let mut attempt = cell_attempt(170 + n as u8)?; + attempt.request.operation = "Increment".into(); + attempt.request.contract_digest = [52; 32]; + attempt.request.request_digest = command_request_digest(&[]); + let authority = fixture.runtimes[0] + .execution_authority(route.history, cell.clone()) + .await?; + let CellSqlOutcome::Prepared(prepared) = pool + .stage_registered( + fixture.cell_options(cell).await?, + authority, + attempt, + class.clone(), + vec![], + true, + ) + .await? + else { + anyhow::bail!("command preparation") + }; + pool.mark_submitted(cell.clone(), prepared.proposal()) + .await?; + let accepted = accepted_sql(&fixture.runtimes[0], &prepared).await?; + fixture.drive(Some(accepted.position.index)).await?; + let authority = fixture.runtimes[0] + .execution_authority(route.history, cell.clone()) + .await?; + drop(pool.finish(authority).await?); + drop(prepared); + } + let output = pool + .query_registered( + options.clone(), + old, + class.clone(), + contract.clone(), + vec![], + ) + .await?; + verify(&output, 0, &original, route.schema_digest)?; + drop(output); + let fresh = reads::selected(&mut fixture, 0, &a, CellReadConsistency::Linearizable).await?; + let latest = fresh.cell().observation(); + assert_eq!(latest.version.sequence.get(), 2); + assert!( + latest.witness.position.index + < fixture.runtimes[0] + .status(route.history) + .await? + .applied + .index + ); + let output = pool + .query_registered( + options.clone(), + fresh, + class.clone(), + contract.clone(), + vec![], + ) + .await?; + verify(&output, 7, &latest, route.schema_digest)?; + drop(output); + let before = calls.load(Ordering::SeqCst); + for fault in 0..6 { + let authority = reads::selected(&mut fixture, 0, &a, CellReadConsistency::Stale).await?; + let mut request = contract.clone(); + let mut definition = class.clone(); + let mut supplied = vec![]; + let mut config = options.clone(); + match fault { + 0 => request.digest = [54; 32], + 1 => request.kind = OperationKind::Command, + 2 => request.operation = "Missing".into(), + 3 => { + definition = Arc::new(RaftClass::new("Counter", 1, [55; 32])?.query( + "Get", + [53; 32], + |_, _| panic!("wrong class query"), + )?) + } + 4 => { + match &mut config { + CellSqlOptions::Materialized(o) => o.limits.envelope.max_payload_bytes = 1024, + CellSqlOptions::Extents(o) => o.limits.envelope.max_payload_bytes = 1024, + } + supplied = vec![0; 1025]; + } + _ => request.operation = "x".repeat(129), + } + let result = pool + .query_registered(config, authority, definition, request, supplied) + .await; + match fault { + 0 | 3 => assert!(matches!( + result, + Err(ExecutionError::Registration( + RegistrationError::ContractMismatch + )) + )), + 1 => assert!(matches!( + result, + Err(ExecutionError::Registration( + RegistrationError::KindMismatch + )) + )), + 2 => assert!(matches!( + result, + Err(ExecutionError::Registration( + RegistrationError::UnknownOperation(_) + )) + )), + 4 => assert!(matches!( + result, + Err(ExecutionError::Refused(CommandFailure::Overloaded)) + )), + _ => assert!(matches!( + result, + Err(ExecutionError::Registration(RegistrationError::Invalid( + "class/operation name" + ))) + )), + } + assert_eq!(calls.load(Ordering::SeqCst), before); + assert_eq!(pool.usage()?.retained_operations, 0); + } + let authority = reads::selected(&mut fixture, 0, &a, CellReadConsistency::Stale).await?; + assert!(matches!( + pool.query_registered( + options.clone(), + authority, + class.clone(), + contract.clone(), + b"panic".to_vec() + ) + .await, + Err(ExecutionError::Storage(_)) + )); + query_released(&pool).await?; + let deadline = reads::deadline(&fixture, 100).await?; + let authority = fixture.runtimes[0] + .read_authority( + route.history, + a.clone(), + CellReadConsistency::Stale, + deadline, + ) + .await?; + assert!(tokio::time::timeout( + Duration::from_secs(3), + pool.query_registered( + options.clone(), + authority, + class.clone(), + contract.clone(), + b"spin".to_vec() + ) + ) + .await? + .is_err()); + // Deadline delivery cancels the waiter. Entered SQL and its sole queue + // slot remain owned until the worker observes cancellation and closes the + // image; verify that physical completion before expecting new admission. + query_released(&pool).await?; + let authority = reads::selected(&mut fixture, 0, &a, CellReadConsistency::Stale).await?; + let output = pool + .query_registered( + options.clone(), + authority, + class.clone(), + contract.clone(), + vec![], + ) + .await?; + verify(&output, 7, &latest, route.schema_digest)?; + drop(output); + let authority = reads::selected(&mut fixture, 0, &a, CellReadConsistency::Stale).await?; + let caller = pool.clone(); + let waiter = fixture.cluster.domain.spawn(async move { + let mut input = vec![0; 256 * 1024]; + input[..4].copy_from_slice(b"hold"); + caller + .query_registered(options, authority, class, contract, input) + .await + })?; + tokio::time::timeout(Duration::from_secs(5), started).await??; + waiter.abort(); + assert!(waiter.await.is_err()); + assert_eq!(pool.usage()?.retained_operations, 1); + assert!(pool.usage()?.retained_bytes >= 256 * 1024); + assert!( + tokio::time::timeout(Duration::from_millis(30), pool.close()) + .await + .is_err() + ); + release.send(())?; + tokio::time::timeout(Duration::from_secs(5), pool.close()).await??; + assert_eq!(pool.usage()?.retained_operations, 0); + assert_eq!(pool.usage()?.retained_bytes, 0); + drop(pool); + fixture.close().await +} diff --git a/crates/cellar/tests/support/cell_group_sql_submission.rs b/crates/cellar/tests/support/cell_group_sql_submission.rs new file mode 100644 index 000000000..cac2fb2f9 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_sql_submission.rs @@ -0,0 +1,263 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::reconciliation::apply_through; +use super::*; + +async fn initial( + fixture: &RuntimeCluster, + pool: &CellGroupSqlPool, + cell: &CellHistory, + creation: u8, + preview: u8, +) -> Result { + let authority = fixture.runtimes[0] + .enrollment_authority(fixture.cluster.birth.route().history, cell.clone()) + .await?; + let CellSqlOutcome::Prepared(prepared) = pool + .initialize( + fixture.cell_options(cell).await?, + authority, + CellEnrollmentAttempt { + creation: ProposalId::new([creation; 16])?, + preview: ImageId::new([preview; 16])?, + inputs: cellar::storage::CommandInputs { + now_unix_millis: 10, + random: [17; 32], + }, + }, + |tx| { + tx.execute_batch("CREATE TABLE value(n); INSERT INTO value VALUES (0);")?; + Ok(()) + }, + ) + .await? + else { + anyhow::bail!("new enrollment") + }; + pool.mark_submitted(cell.clone(), prepared.proposal()) + .await?; + Ok(prepared) +} + +async fn mutation( + fixture: &RuntimeCluster, + pool: &CellGroupSqlPool, + cell: &CellHistory, + proposal: u8, + preview: u8, +) -> Result { + let authority = fixture.runtimes[0] + .execution_authority(fixture.cluster.birth.route().history, cell.clone()) + .await?; + let mut attempt = cell_attempt(proposal)?; + attempt.preview = ImageId::new([preview; 16])?; + let CellSqlOutcome::Prepared(prepared) = pool + .stage( + fixture.cell_options(cell).await?, + authority, + attempt, + true, + |tx| { + let value: i64 = tx.query_row("SELECT n FROM value", [], |row| row.get(0))?; + assert_eq!(value, 0, "rejected private effects must not be reused"); + tx.execute("UPDATE value SET n=n+1", [])?; + Ok(b"committed".to_vec()) + }, + ) + .await? + else { + anyhow::bail!("new mutation") + }; + pool.mark_submitted(cell.clone(), prepared.proposal()) + .await?; + Ok(prepared) +} + +pub(super) async fn rejections(rooted: bool) -> Result<()> { + let mut fixture = RuntimeCluster::with_limits(256, 1, 2).await?; + if rooted { + for (n, voter) in fixture.cluster.voters.iter_mut().enumerate() { + voter.extents = Some(NodeExtentStore::open( + fixture.cluster._root.path().join(format!("submission-{n}")), + Arc::new(DirectFileSystem), + Default::default(), + )?); + } + } + let route = fixture.cluster.birth.route().clone(); + fixture.runtimes[0].campaign(route.history).await?; + fixture.drive(None).await?; + let limits = CellSqlLimits { + workers: 2, + queue_per_worker: 4, + max_pending_cells: 4, + group_pending_cells: 4, + node_bytes: 512 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + }; + let pool = CellGroupSqlPool::start(&fixture.cluster.voters[0].log, limits)?; + let a = history(&route, "submission-a")?; + let b = history(&route, "submission-b")?; + let c = history(&route, "submission-c")?; + let pa = initial(&fixture, &pool, &a, 181, 181).await?; + let pb = initial(&fixture, &pool, &b, 182, 182).await?; + + // Drop the acceptance response, then hold real engine sync so a queued + // ReadIndex occupies the only ordinary request slot. Queue overload must + // not produce a false owner rejection for B. + let gate = fixture.cluster.voters[0].gate.clone(); + let release = gate.arm(fault::HOLD_SYNC); + drop(pa.submit(&fixture.runtimes[0]).await?); + tokio::time::timeout(Duration::from_secs(5), gate.entered.notified()).await?; + let mut waiting = Box::pin(fixture.runtimes[0].read_index(route.history, vec![77])); + assert!( + tokio::time::timeout(Duration::from_millis(30), waiting.as_mut()) + .await + .is_err() + ); + assert!(matches!( + pb.submit(&fixture.runtimes[0]).await, + Err(RuntimeError::Overloaded) + )); + assert!(pb.rejection()?.is_none()); + assert!(pa.rejection()?.is_none()); + assert_eq!(pool.usage()?.active_cells, 2); + drop(release); + tokio::time::timeout(Duration::from_secs(5), waiting).await??; + let accepted_a = accepted_sql(&fixture.runtimes[0], &pa).await?; + assert!(pa.rejection()?.is_none()); + let state = fixture.runtimes[0] + .enrollment_state(route.history, b.clone()) + .await?; + assert!(matches!( + pool.resolve_enrollment(pb.proposal(), state).await, + Err(ExecutionError::Uncertain) + )); + + // The actual owner now refuses B against A's retained acceptance slot. + // Discard the response and recover its irreversible seal from the producer. + drop(pb.submit(&fixture.runtimes[0]).await?); + let proof_b = pb.rejection()?.context("sealed B")?; + assert!(pool.reject(a.clone(), proof_b.clone()).await.is_err()); + assert_eq!(pool.usage()?.active_cells, 2); + let foreign = CellGroupSqlPool::start(&fixture.cluster.voters[1].log, limits)?; + assert!(foreign.reject(b.clone(), proof_b.clone()).await.is_err()); + foreign.close().await?; + assert!(matches!( + pb.submit(&fixture.runtimes[1]).await, + Err(RuntimeError::Invalid(_)) + )); + pool.reject(b.clone(), proof_b.clone()).await?; + assert_eq!(pool.usage()?.active_cells, 1); + assert!(pool.usage()?.retained_bytes > 0); + assert!(matches!( + fixture.runtimes[0] + .propose_at_term(route.history, pb.term(), pb.bytes().to_vec()) + .await, + Err(RuntimeError::Invalid(_)) + )); + apply_through(&mut fixture, accepted_a.position.index).await?; + drop(super::enrollment::resolved(&fixture, &pool, 0, &a, pa.proposal()).await?); + + // Retrying after capacity returns must still reject every old clone. A fresh + // producer of the same creation identity gets a distinct submission seal. + let fresh_b = initial(&fixture, &pool, &b, 182, 183).await?; + assert_eq!(pb.proposal(), fresh_b.proposal()); + assert!(pool.reject(b.clone(), proof_b.clone()).await.is_err()); + assert_eq!(pool.usage()?.active_cells, 1); + assert!(matches!( + pb.submit(&fixture.runtimes[0]).await?, + CellSqlSubmission::Rejected(_) + )); + let accepted_b = accepted_sql(&fixture.runtimes[0], &fresh_b).await?; + assert_eq!( + accepted_b.position.index.get(), + accepted_a.position.index.get() + 1 + ); + apply_through(&mut fixture, accepted_b.position.index).await?; + drop(super::enrollment::resolved(&fixture, &pool, 0, &b, fresh_b.proposal()).await?); + let pc = initial(&fixture, &pool, &c, 184, 184).await?; + drop(pc.submit(&fixture.runtimes[0]).await?); + pool.reject(c.clone(), pc.rejection()?.context("full group rejection")?) + .await?; + assert_eq!(pool.usage()?.active_cells, 0); + assert!(fixture.runtimes[0] + .enrollment_state(route.history, c) + .await? + .record() + .is_none()); + + // Completed/rejected producers release cell names, but their retained + // outputs still consume all four operation reservations. Release this + // phase's outputs before preparing the independent mutation phase. + assert_eq!(pool.usage()?.retained_operations, 4); + drop((pa, pb, fresh_b, pc, proof_b)); + assert_eq!(pool.usage()?.retained_operations, 0); + assert_eq!(pool.usage()?.retained_bytes, 0); + + let ma = mutation(&fixture, &pool, &a, 185, 185).await?; + let mb = mutation(&fixture, &pool, &b, 186, 186).await?; + drop(ma.submit(&fixture.runtimes[0]).await?); + let accepted_ma = accepted_sql(&fixture.runtimes[0], &ma).await?; + drop(mb.submit(&fixture.runtimes[0]).await?); + let proof_mb = mb.rejection()?.context("mutation seal")?; + assert!(pool.reject(a.clone(), proof_mb.clone()).await.is_err()); + pool.reject(b.clone(), proof_mb.clone()).await?; + apply_through(&mut fixture, accepted_ma.position.index).await?; + let authority = fixture.runtimes[0] + .execution_authority(route.history, a.clone()) + .await?; + assert_eq!( + pool.finish(authority).await?.receipt().receipt.result(), + b"committed" + ); + let cached = accepted_sql(&fixture.runtimes[0], &ma).await?; + assert_eq!(cached.position, accepted_ma.position); + assert_eq!(cached.digest, accepted_ma.digest); + assert!(ma.rejection()?.is_none()); + let fresh_mb = mutation(&fixture, &pool, &b, 186, 187).await?; + assert_eq!(mb.proposal(), fresh_mb.proposal()); + assert!(pool.reject(b.clone(), proof_mb.clone()).await.is_err()); + assert!(matches!( + mb.submit(&fixture.runtimes[0]).await?, + CellSqlSubmission::Rejected(_) + )); + let accepted_mb = accepted_sql(&fixture.runtimes[0], &fresh_mb).await?; + assert_eq!( + accepted_mb.position.index.get(), + accepted_ma.position.index.get() + 1 + ); + apply_through(&mut fixture, accepted_mb.position.index).await?; + let authority = fixture.runtimes[0] + .execution_authority(route.history, b.clone()) + .await?; + assert_eq!( + pool.finish(authority).await?.receipt().receipt.result(), + b"committed" + ); + assert_eq!(pool.usage()?.active_cells, 0); + for node in 0..3 { + for cell in [&a, &b] { + let proof = fixture.runtimes[node] + .reconciliation_authority(route.history, cell.clone()) + .await?; + assert_eq!(proof.cell().version.sequence.get(), 2); + } + } + drop((ma, mb, fresh_mb, proof_mb)); + pool.close().await?; + assert_eq!(pool.usage()?.retained_bytes, 0); + assert_eq!(pool.usage()?.running_workers, 0); + if rooted { + assert_eq!( + fixture.cluster.voters[0] + .extents + .as_ref() + .context("extents")? + .usage()? + .workspaces, + 0 + ); + } + fixture.close().await +} diff --git a/crates/cellar/tests/support/cell_group_tcp.rs b/crates/cellar/tests/support/cell_group_tcp.rs new file mode 100644 index 000000000..6c803b8e2 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_tcp.rs @@ -0,0 +1,259 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Three physical runtimes use the production transport and real TLS sockets. +use super::*; +use cellar::application::consensus::verification::GroupRoute as TransportRoute; +use cellar::peer_auth::PeerAuth; +use tokio::net::TcpListener; + +pub(super) fn tls() -> Result { + Ok(RaftTls::new( + vec![include_bytes!("../../transport/tcp/fixtures/ca.der") + .to_vec() + .into()], + vec![include_bytes!("../../transport/tcp/fixtures/server.der") + .to_vec() + .into()], + rustls::pki_types::PrivatePkcs8KeyDer::from( + include_bytes!("../../transport/tcp/fixtures/key.der").to_vec(), + ) + .into(), + )?) +} + +pub(crate) async fn start( + runtime: &CellGroupRuntime, + n: usize, +) -> Result { + start_with_transport(runtime, n, None, TransportOptions::default()).await +} + +async fn start_with_transport( + runtime: &CellGroupRuntime, + n: usize, + send_buffer: Option, + options: TransportOptions, +) -> Result { + let query = if let Some(bytes) = send_buffer { + let socket = tokio::net::TcpSocket::new_v4()?; + socket.set_send_buffer_size(bytes)?; + socket.bind( + "127.0.0.1:0" + .parse() + .map_err(|_| TransportError::Identity)?, + )?; + socket.listen(128)? + } else { + TcpListener::bind("127.0.0.1:0").await? + }; + RaftTransport::start_shared( + runtime.clone(), + Arc::new( + PeerAuth::new([17; 32], format!("node-{}", n + 1)) + .map_err(|_| TransportError::Identity)?, + ), + tls().map_err(|e| TransportError::Invalid(e.to_string()))?, + TcpListener::bind("127.0.0.1:0").await?, + TcpListener::bind("127.0.0.1:0").await?, + query, + options, + ) + .await +} + +impl RuntimeCluster { + pub(super) async fn connect_tcp(&mut self) -> Result<()> { + self.connect_tcp_with_query_buffer(None).await + } + + pub(super) async fn connect_tcp_with_query_buffer(&mut self, bytes: Option) -> Result<()> { + self.connect_tcp_options(bytes, TransportOptions::default()) + .await + } + + pub(super) async fn connect_tcp_options( + &mut self, + bytes: Option, + options: TransportOptions, + ) -> Result<()> { + assert!(self.transports.is_empty()); + assert!(self.network.is_empty()); + for (n, runtime) in self.runtimes.iter().enumerate() { + self.transports + .push(start_with_transport(runtime, n, bytes, options.clone()).await?); + } + for history in self.applied.keys() { + let route = self.tcp_route(*history)?; + for transport in &self.transports { + transport.install_physical_group(route.clone())?; + } + } + Ok(()) + } + + pub(super) fn tcp_route(&self, history: GroupHistory) -> Result> { + let marker = self.cluster.voters[0] + .log + .cell_group(&history)? + .group_manifest()? + .context("manifest")?; + Ok(TransportRoute { + scope: history, + schema_digest: marker.route().schema_digest, + revision: 1, + peers: self + .transports + .iter() + .enumerate() + .map(|(n, transport)| -> Result<_> { + Ok(RaftPeer { + replica: self.cluster.voters[n] + .log + .cell_group(&history)? + .identity() + .clone(), + label: format!("node-{}", n + 1), + server_name: "localhost".into(), + control_address: transport.control_address(), + data_address: transport.data_address(), + query_address: transport.query_address(), + }) + }) + .collect::>>()?, + }) + } +} + +pub(crate) async fn physical_transport() -> Result<()> { + let mut fixture = RuntimeCluster::new().await?; + let second = fixture.add_group(31).await?; + let histories = [fixture.cluster.birth.route().history, second]; + fixture.connect_tcp().await?; + assert!(matches!( + start(&fixture.runtimes[0], 0).await, + Err(TransportError::Runtime(RuntimeError::AlreadyAttached)) + )); + for transport in &fixture.transports { + assert_eq!(transport.usage().groups, 2); + assert_eq!(transport.usage().peers, 2); + } + for history in histories { + fixture.runtimes[0].campaign(history).await?; + } + fixture.settle(&histories).await?; + for group in histories { + let route = fixture.cluster.voters[0] + .log + .cell_group(&group)? + .group_manifest()? + .context("group")? + .route() + .clone(); + for name in ["tls-a", "tls-b", "tls-c"] { + let cell = history(&route, name)?; + let bytes = + images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES (1);")?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let accepted = fixture.runtimes[0] + .propose(group, request(&route, cell, 1, &cut)?) + .await?; + tokio::time::timeout(Duration::from_secs(20), async { + while fixture.applied[&group] + .iter() + .any(|index| *index < accepted.position.index) + { + fixture.pump(&[], &mut BTreeMap::new()).await?; + tokio::task::yield_now().await; + } + Ok::<_, anyhow::Error>(()) + }) + .await??; + } + for voter in &fixture.cluster.voters { + assert_eq!( + voter + .log + .cell_group(&group)? + .group_manifest()? + .context("applied")? + .root() + .cells(), + 3 + ); + } + } + let route = fixture.tcp_route(histories[0])?; + let mut heartbeat = Message { + from: 1, + to: 2, + term: fixture.runtimes[0].status(histories[0]).await?.term.get(), + ..Default::default() + }; + heartbeat.set_msg_type(MessageType::MsgHeartbeat); + let mut snapshot = heartbeat.clone(); + snapshot.set_msg_type(MessageType::MsgSnapshot); + assert!(matches!( + fixture.transports[0].submit_physical(&histories[0], snapshot), + Err(TransportError::Refused) + )); + let mut unknown = histories[0]; + unknown.group = GroupId::new([222; 16])?; + assert!(matches!( + fixture.transports[0].submit_physical(&unknown, heartbeat.clone()), + Err(TransportError::UnknownGroup) + )); + let mut wrong = route.clone(); + wrong.schema_digest[0] ^= 1; + assert!(matches!( + fixture.transports[0].install_physical_group(wrong), + Err(TransportError::Identity) + )); + let mut wrong = route.clone(); + wrong.revision += 1; + wrong.peers[0].replica.process = ProcessGeneration::new([222; 16])?; + assert!(matches!( + fixture.transports[0].install_physical_group(wrong), + Err(TransportError::Identity) + )); + let mut conflicting = RaftPeerNode::from(&route.peers[1]); + conflicting.process = ProcessGeneration::new([223; 16])?; + assert!(matches!( + fixture.transports[0].grant_query_origin(NamespaceQueryOrigin { + cluster: histories[0].cluster, + namespace: histories[0].namespace, + revision: 1, + peer: conflicting.clone(), + }), + Err(TransportError::StalePeer) + )); + assert!(matches!( + fixture.transports[0].grant_inspection_origin(NamespaceInspectionOrigin { + cluster: histories[0].cluster, + namespace: histories[0].namespace, + revision: 1, + peer: conflicting, + }), + Err(TransportError::StalePeer) + )); + fixture.transports[1].remove_physical_group(&histories[0])?; + assert_eq!(fixture.transports[1].usage().groups, 1); + assert_eq!(fixture.transports[1].usage().peers, 2); + assert!(fixture.transports[0] + .submit_physical(&histories[0], heartbeat.clone())? + .accepted() + .await + .is_err()); + fixture.transports[1].install_physical_group(route.clone())?; + fixture.transports[0] + .submit_physical(&histories[0], heartbeat)? + .accepted() + .await?; + for transport in &fixture.transports { + assert!(transport.usage().accepted > 0); + } + fixture.transports[0].close().await; + assert_eq!(fixture.transports[0].usage().tasks, 0); + let replacement = start(&fixture.runtimes[0], 0).await?; + replacement.close().await; + fixture.close().await +} diff --git a/crates/cellar/tests/support/native_daemon_counter.rs b/crates/cellar/tests/support/native_daemon_counter.rs new file mode 100644 index 000000000..355f630f2 --- /dev/null +++ b/crates/cellar/tests/support/native_daemon_counter.rs @@ -0,0 +1,581 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Process-level native ingress, routing, bucket restore and reload coverage. + +// Process orchestration runs outside the cell execution boundary and uses +// host wall-clock deadlines while child nodes own their execution domains. +use anyhow::{bail, Context as _, Result}; +use cellar::{bucket::Bucket, sdk::ExecutionDomain}; +use reqwest::{Client, Method}; +use serde_json::{json, Value}; +use std::fs::File; +use std::path::{Path, PathBuf}; +use std::process::{Child, Command, Stdio}; +use std::time::{Duration, Instant}; + +#[derive(Clone, Copy)] +pub enum DurabilityMode { + Bucket, + Fleet, +} + +impl DurabilityMode { + fn as_env(self) -> &'static str { + match self { + Self::Bucket => "bucket", + Self::Fleet => "fleet", + } + } +} + +struct Node { + process: Child, + log: PathBuf, + diagnostics: PathBuf, + ingress: String, + internal: String, +} + +impl Drop for Node { + fn drop(&mut self) { + if self.process.try_wait().ok().flatten().is_none() { + let _ = self.process.kill(); + } + let _ = self.process.wait(); + } +} + +impl Node { + async fn start(root: &Path, name: &str, mode: DurabilityMode) -> Result { + Self::start_program( + root, + name, + None, + env!("CARGO_BIN_EXE_cellar-native-counter"), + None, + mode, + ) + .await + } + + async fn start_program( + root: &Path, + name: &str, + bucket: Option<&Path>, + program: &str, + source: Option<&Path>, + mode: DurabilityMode, + ) -> Result { + let log = root.join(format!("{name}.log")); + let stdout = File::create(&log)?; + // Listener announcements are stdout data. Merging stderr into the + // same file can splice a concurrent diagnostic into their lines. + let diagnostics = root.join(format!("{name}-stderr.log")); + let stderr = File::create(&diagnostics)?; + let mut command = Command::new(program); + command + .env_clear() + .args([ + "--listen", + "127.0.0.1:0", + "--internal-listen", + "127.0.0.1:0", + ]) + .env("CELLAR_NODE", name) + .env("CELLAR_TEST_DATA_DIR", root.join(format!("{name}-data"))) + .env("CELLAR_TOKIO_THREADS", "2") + .env("CELLAR_PLACEMENT_WEIGHT", "1") + .env("CELLAR_REBALANCE_INTERVAL_MS", "0") + .env("CELLAR_DEPLOY_POLL_S", "3600") + .env("CELLAR_DURABILITY", mode.as_env()) + .env("CELLAR_SHUTDOWN_TOTAL_MS", "8000") + .env("RUST_LOG", "warn") + .stdout(Stdio::from(stdout)) + .stderr(Stdio::from(stderr)); + if let Some(bucket) = bucket { + command + .env("CELLAR_NATIVE_FLEET", "1") + .env("CELLAR_BUCKET", "cellar-dev") + .env("CELLAR_INTERNAL_DEV_STORE", bucket); + } + if let Some(source) = source { + command.env("CELLAR_TEST_SCRIPT_PATH", source); + } + let mut node = Self { + process: command.spawn()?, + log, + diagnostics, + ingress: String::new(), + internal: String::new(), + }; + let deadline = Instant::now() + Duration::from_secs(30); + loop { + let text = std::fs::read_to_string(&node.log)?; + let address = |prefix: &str| { + text.lines().find_map(|line| { + line.strip_prefix(prefix) + .and_then(|rest| rest.split_whitespace().next()) + .map(|address| format!("http://{address}")) + }) + }; + if let (Some(ingress), Some(internal)) = ( + address("cellar listening on "), + address("cellar internal listening on "), + ) { + node.ingress = ingress; + node.internal = internal; + return Ok(node); + } + if let Some(status) = node.process.try_wait()? { + bail!( + "native node {name} exited during boot: {status}\n{}", + node.logs()? + ); + } + if Instant::now() >= deadline { + bail!("native node {name} did not boot\n{}", node.logs()?); + } + tokio::time::sleep(Duration::from_millis(50)).await; + } + } + + fn logs(&self) -> Result { + Ok(format!( + "stdout:\n{}\nstderr:\n{}", + std::fs::read_to_string(&self.log)?, + std::fs::read_to_string(&self.diagnostics)? + )) + } + + async fn ready(&mut self, client: &Client) -> Result<()> { + let deadline = Instant::now() + Duration::from_secs(30); + loop { + if client + .get(format!("{}/.well-known/cellar/health", self.ingress)) + .send() + .await + .is_ok_and(|response| response.status().is_success()) + { + return Ok(()); + } + if let Some(status) = self.process.try_wait()? { + bail!("node exited before readiness: {status}\n{}", self.logs()?); + } + if Instant::now() >= deadline { + bail!("native readiness timed out\n{}", self.logs()?); + } + tokio::time::sleep(Duration::from_millis(50)).await; + } + } + + async fn stop(&mut self, client: &Client) -> Result<()> { + client + .post(format!("{}/shutdown", self.internal)) + .send() + .await? + .error_for_status()?; + let deadline = Instant::now() + Duration::from_secs(15); + loop { + if let Some(status) = self.process.try_wait()? { + anyhow::ensure!( + status.success(), + "native shutdown failed: {status}\n{}", + self.logs()? + ); + return Ok(()); + } + if Instant::now() >= deadline { + bail!("native node did not complete shutdown"); + } + tokio::time::sleep(Duration::from_millis(50)).await; + } + } +} + +async fn counter(client: &Client, node: &Node, name: &str, write: bool) -> Result<(u64, String)> { + let response = client + .request( + if write { Method::POST } else { Method::GET }, + format!("{}/counters/{name}", node.ingress), + ) + .send() + .await?; + let status = response.status(); + let version = response + .headers() + .get("x-native-version") + .context("missing compiled version")? + .to_str()? + .to_string(); + let body = response.text().await?; + anyhow::ensure!( + status.is_success(), + "native counter failed: {status} {body}" + ); + Ok(( + serde_json::from_str::(&body)?["count"] + .as_u64() + .context("missing count")?, + version, + )) +} + +async fn deployment( + bucket: &Bucket, + version: &str, + executable: &std::path::Path, +) -> Result { + deployment_with_assets(bucket, version, version, executable).await +} + +async fn deployment_with_assets( + bucket: &Bucket, + version: &str, + asset_revision: &str, + executable: &Path, +) -> Result { + use cellar::native_deploy::{current_target, executable_identity, ApplicationConfig, Package}; + let artifact = executable_identity(executable, current_target())?; + let root = executable.parent().context("executable has no directory")?; + let assets = root.join(format!("assets-{asset_revision}")); + std::fs::create_dir_all(assets.join("counters"))?; + std::fs::write( + assets.join("index.html"), + format!("native assets {asset_revision}"), + )?; + std::fs::write(assets.join("counters/tenant-acme"), "static counter shadow")?; + std::fs::write(assets.join("counters/static.txt"), "static route exclusion")?; + std::fs::write( + assets.join(format!("{asset_revision}-only.txt")), + format!("only {asset_revision}"), + )?; + std::fs::write( + assets.join("_headers"), + format!("/*\n X-Asset-Version: {asset_revision}\n"), + )?; + std::fs::write(assets.join("_redirects"), "/old / 302\n")?; + let config: ApplicationConfig = serde_json::from_value(json!({ + "script_name": "native-counter", "application_version": version, + "compatibility_date":"2026-01-01", + "cells": ["Counter"], + "bindings": [{"type": "durable_object_namespace", "name": "COUNTERS", "class_name": "Counter"}], + "assets": {"directory":format!("assets-{asset_revision}"), "binding":"ASSETS", + "not_found_handling":"single-page-application", + "run_worker_first":["/counters/*","!/counters/static.txt","/assets-binding/*"]}, + }))?; + let staged = root.join(format!("package-{version}-{asset_revision}")); + let primary = config.stage_manifest(vec![artifact.clone()], root, &staged)?; + std::fs::copy(executable, staged.join(&artifact.sha256))?; + let package = Package { + primary, + cohosted: Vec::new(), + }; + package.publish_staged(bucket, &staged).await?; + Ok(package.primary.version) +} + +pub async fn run(mode: DurabilityMode) -> Result<()> { + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let result = domain.run(exercise(mode)).await; + domain.shutdown().await; + result? +} + +async fn exercise(mode: DurabilityMode) -> Result<()> { + let root = tempfile::tempdir()?; + // Freeze the compiled artifact: another Cargo feature build can replace + // target/debug binaries while a process test is still exercising reload. + let executable = root.path().join("compiled-host"); + std::fs::copy(env!("CARGO_BIN_EXE_cellar-native-counter"), &executable)?; + let client = Client::builder() + .no_proxy() + .timeout(Duration::from_secs(15)) + .build()?; + // Embedded applications use the same listener and Actor without a bucket. + let mut local = Node::start(root.path(), "native-local", mode).await?; + local.ready(&client).await?; + assert_eq!( + counter(&client, &local, "alice", true).await?, + (1, "v1".into()) + ); + assert_eq!(counter(&client, &local, "bob", false).await?.0, 0); + assert_eq!(counter(&client, &local, "alice", false).await?.0, 1); + assert_eq!( + client + .post(format!("{}/reload", local.internal)) + .send() + .await? + .status(), + 503 + ); + local.stop(&client).await?; + + // A real local development store backs leases and LTX for two processes. + // This proves production protocol integration, not cloud failure coverage. + let database = root.path().join("bucket.sqlite"); + let bucket = Bucket::open_dev(&database)?; + deployment(&bucket, "v1", &executable).await?; + let pointer = cellar::fleet::read_current_pointer(&bucket).await?; + let manifest: cellar::protocol::Manifest = serde_json::from_slice( + &bucket + .get(&format!("{}/manifest.json", pointer.prefix)) + .await? + .unwrap() + .0, + )?; + let package = cellar::native_deploy::Package { + primary: manifest, + cohosted: Vec::new(), + }; + let installed = root.path().join("native-node"); + package + .install(&bucket, cellar::native_deploy::current_target(), &installed) + .await?; + let installed = installed.to_str().unwrap(); + let mut first = Node::start_program( + root.path(), + "native-first", + Some(&database), + installed, + None, + mode, + ) + .await?; + let mut second = Node::start_program( + root.path(), + "native-second", + Some(&database), + installed, + None, + mode, + ) + .await?; + first.ready(&client).await?; + second.ready(&client).await?; + for node in [&first, &second] { + let response = client + .get(format!("{}/", node.ingress)) + .send() + .await? + .error_for_status()?; + assert_eq!(response.headers()["x-asset-version"], "v1"); + assert_eq!(response.text().await?, "native assets v1"); + let response = client + .get(format!("{}/assets-binding/", node.ingress)) + .send() + .await? + .error_for_status()?; + assert_eq!(response.headers()["x-native-version"], "v1"); + assert_eq!(response.text().await?, "native assets v1"); + let response = client + .get(format!("{}/counters/static.txt", node.ingress)) + .send() + .await? + .error_for_status()?; + assert!(response.headers().get("x-native-version").is_none()); + assert_eq!(response.text().await?, "static route exclusion"); + let response = client + .get(format!("{}/missing-navigation", node.ingress)) + .header("sec-fetch-mode", "navigate") + .send() + .await? + .error_for_status()?; + assert_eq!(response.text().await?, "native assets v1"); + let redirect_client = Client::builder() + .no_proxy() + .redirect(reqwest::redirect::Policy::none()) + .build()?; + let response = redirect_client + .get(format!("{}/old", node.ingress)) + .send() + .await?; + assert_eq!(response.status(), 302); + assert_eq!(response.headers()["location"], "/"); + } + assert_eq!(counter(&client, &first, "tenant-acme", true).await?.0, 1); + assert_eq!(counter(&client, &second, "tenant-acme", true).await?.0, 2); + assert_eq!(counter(&client, &first, "tenant-acme", false).await?.0, 2); + + // Updating assets changes the deployment without compiling new code or + // requiring another application registration. Business state stays intact. + let asset_revision = deployment_with_assets(&bucket, "v1", "v1-assets", &executable).await?; + for node in [&first, &second] { + let outcome: Value = client + .post(format!("{}/reload", node.internal)) + .send() + .await? + .error_for_status()? + .json() + .await?; + assert_eq!(outcome["version"], asset_revision); + let response = client + .get(format!("{}/assets-binding/", node.ingress)) + .send() + .await? + .error_for_status()?; + assert_eq!(response.headers()["x-native-version"], "v1"); + assert_eq!(response.headers()["x-asset-version"], "v1-assets"); + assert_eq!(response.text().await?, "native assets v1-assets"); + assert_eq!( + counter(&client, node, "tenant-acme", false).await?, + (2, "v1".into()) + ); + } + + let v2_deployment = deployment(&bucket, "v2", &executable).await?; + // Asset version-skew fallback serves a new content path even while the + // native application still executes its boot generation. + let deadline = Instant::now() + Duration::from_secs(15); + loop { + let response = client + .get(format!("{}/v2-only.txt", first.ingress)) + .send() + .await?; + if response.status().is_success() && response.text().await? == "only v2" { + break; + } + anyhow::ensure!( + Instant::now() < deadline, + "native asset fallback did not refresh" + ); + tokio::time::sleep(Duration::from_millis(50)).await; + } + assert_eq!( + counter(&client, &first, "tenant-acme", false).await?.1, + "v1" + ); + for node in [&first, &second] { + let outcome: Value = client + .post(format!("{}/reload", node.internal)) + .send() + .await? + .error_for_status()? + .json() + .await?; + assert_eq!(outcome["outcome"], "adopted"); + assert_eq!(outcome["version"], v2_deployment); + let response = client + .get(format!("{}/assets-binding/", node.ingress)) + .send() + .await? + .error_for_status()?; + assert_eq!(response.headers()["x-native-version"], "v2"); + assert_eq!(response.headers()["x-asset-version"], "v2"); + assert_eq!(response.text().await?, "native assets v2"); + assert_eq!( + counter(&client, node, "tenant-acme", false).await?, + (2, "v2".into()) + ); + } + let instance = client + .get(format!("{}/counters/tenant-acme", first.ingress)) + .send() + .await? + .error_for_status()? + .headers()["x-native-instance"] + .to_str()? + .to_string(); + client + .post(format!("{}/reload", first.internal)) + .send() + .await? + .error_for_status()?; + let fresh = client + .get(format!("{}/counters/tenant-acme", first.ingress)) + .send() + .await? + .error_for_status()? + .headers()["x-native-instance"] + .to_str()? + .to_string(); + assert_ne!( + instance, fresh, + "forced reload must reconstruct compiled application state" + ); + // A declared but incompatible executable must not adopt, even when its + // application code version is registered in the current process. + let mut artifact = cellar::native_deploy::executable_identity( + &executable, + cellar::native_deploy::current_target(), + )?; + artifact.target = "different-target".into(); + let config: cellar::native_deploy::ApplicationConfig = serde_json::from_value(json!({ + "script_name": "native-counter", "application_version": "v2", "cells": ["Counter"], + "bindings": [{"type": "durable_object_namespace", "name": "COUNTERS", "class_name": "Counter"}], + }))?; + cellar::native_deploy::Package { + primary: config.manifest(vec![artifact.clone()])?, + cohosted: Vec::new(), + } + .publish( + &bucket, + &std::collections::BTreeMap::from([(artifact.sha256, executable.clone())]), + ) + .await?; + for node in [&first, &second] { + let response = client + .post(format!("{}/reload", node.internal)) + .send() + .await?; + assert_eq!(response.status(), 422); + let failure: Value = response.json().await?; + assert!(failure["error"] + .as_str() + .unwrap() + .contains("running executable is not declared")); + assert_eq!( + counter(&client, node, "tenant-acme", false).await?, + (2, "v2".into()) + ); + } + // Unknown compiled versions fail before adoption and leave v2 serving. + deployment(&bucket, "v3", &executable).await?; + for node in [&first, &second] { + let response = client + .post(format!("{}/reload", node.internal)) + .send() + .await?; + assert_eq!(response.status(), 422); + let failure: Value = response.json().await?; + assert!(failure["error"] + .as_str() + .unwrap() + .contains("no compiled application registered")); + assert_eq!( + counter(&client, node, "tenant-acme", false).await?, + (2, "v2".into()) + ); + } + // Move ownership through the real graceful drain. The surviving process + // restores the committed SQLite state from the retained replication path. + first.stop(&client).await?; + assert_eq!( + counter(&client, &second, "tenant-acme", true).await?, + (3, "v2".into()) + ); + second.stop(&client).await?; + deployment(&bucket, "v2", &executable).await?; + let mut restarted = Node::start_program( + root.path(), + "native-restarted", + Some(&database), + installed, + None, + mode, + ) + .await?; + restarted.ready(&client).await?; + assert_eq!( + counter(&client, &restarted, "tenant-acme", false).await?, + (3, "v2".into()) + ); + restarted.stop(&client).await?; + for node in ["native-first", "native-second", "native-restarted"] { + let data = root.path().join(format!("{node}-data")); + anyhow::ensure!( + data.is_dir() && !data.join("raft").exists(), + "ordinary compiled deployment created a Raft directory for {node}" + ); + } + Ok(()) +} diff --git a/crates/cellar/tests/support/raft_alarm_native.rs b/crates/cellar/tests/support/raft_alarm_native.rs index 2ee6b5e14..437d55eeb 100644 --- a/crates/cellar/tests/support/raft_alarm_native.rs +++ b/crates/cellar/tests/support/raft_alarm_native.rs @@ -207,6 +207,7 @@ impl Fixture { host, }, RaftCellsServices { + recovery: None, listeners, tls: public_node::tls()?, store: store.clone(), diff --git a/crates/cellar/tests/support/raft_archive_reclaim.rs b/crates/cellar/tests/support/raft_archive_reclaim.rs deleted file mode 100644 index 0aa1bcda6..000000000 --- a/crates/cellar/tests/support/raft_archive_reclaim.rs +++ /dev/null @@ -1,248 +0,0 @@ -// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. - -//! Short archived histories compact independently; failed publication retains logs. -use super::{bundle_faults as faults, bundle_images as images, public_node}; -use anyhow::{ensure, Context as _, Result}; -use cellar::application::consensus::verification::Catalog; -use cellar::sdk::{consensus::*, ShutdownMode}; -use cellar_ltx::{ - generation::{GenerationLimits, GenerationStore}, - DirectFileSystem, LtxHost, -}; -use std::{ - sync::{ - atomic::{AtomicUsize, Ordering}, - Arc, - }, - time::Duration, -}; - -struct Release(Arc); - -impl Drop for Release { - fn drop(&mut self) { - self.0.archive_hang.store(false, Ordering::Release); - self.0.archive_resume.notify_waiters(); - self.0.archive_heads_hang.store(false, Ordering::Release); - self.0.archive_heads_resume.notify_waiters(); - } -} - -fn host() -> RaftHostOptions { - RaftHostOptions { - // Neither entry-count nor cache-pressure policy can explain compaction. - checkpoint_entries: u64::MAX, - archive_checkpoint_ms: 20, - ..Default::default() - } -} - -pub async fn run() -> Result<()> { - let (gate, _entered) = public_node::Gate::new(); - gate.release(); - let calls = Arc::new(AtomicUsize::new(0)); - let definition = public_node::counter(calls.clone(), gate)?; - let mut namespace = NamespaceConfig::new( - ClusterId::new([1; 16])?, - NamespaceId::new([2; 16])?, - NamespaceMode::Raft, - definition.schema, - ); - namespace.voters = VoterPolicy::SingleVoterDevelopment; - let registrations = (10..13) - .map(|id| { - Ok(RaftCellRegistration { - placement: RaftCellPlacement::Replica, - namespace: namespace.clone(), - class: "Counter".into(), - birth: BirthProposal { - scope: GroupScope { - cluster: namespace.cluster, - namespace: namespace.namespace, - cell: format!("Counter:reclaim-{id}"), - group: GroupId::new([id; 16])?, - incarnation: IncarnationId::new([3; 16])?, - }, - voters: vec![InitialMember { - member: MemberId::new(1)?, - node: NodeId::new([1; 16])?, - }], - learners: vec![], - }, - }) - }) - .collect::>>()?; - let root = tempfile::tempdir()?; - let store = Arc::new(faults::Store::default()); - let release = Release(store.clone()); - store.archive_heads_hang.store(true, Ordering::Release); - let node = public_node::node(root.path(), definition.application.clone()).await?; - let service = public_node::cells_many( - &node, - definition.application.clone(), - store.clone(), - registrations.clone(), - false, - host(), - TransportOptions::default(), - ) - .await?; - let result = async { - let cells = registrations - .iter() - .map(|r| service.cell(namespace.namespace, &r.birth.scope.cell)) - .collect::, _>>()?; - for cell in &cells { - public_node::ready(cell).await?; - } - for sequence in 1..=2 { - let mut previous = Vec::new(); - let mut receipts = Vec::new(); - for cell in &cells { - previous.push(cell.status().await?.checkpoint); - receipts.push( - cell.command( - definition.increment.clone(), - 1_i64.to_be_bytes().to_vec(), - public_node::options(sequence)?, - ) - .await?, - ); - } - if sequence == 1 { - tokio::time::timeout(Duration::from_secs(5), async { - while store.archive_heads_waiting.load(Ordering::Acquire) < 2 { - tokio::time::sleep(Duration::from_millis(5)).await; - } - }) - .await - .context("uploaded recovery sets did not reach held head confirmation")?; - } - // Round one has uploaded objects and written heads, but holds their - // confirmations. Round two holds object I/O completely. Neither - // may establish the local archive proof required by this policy. - tokio::time::sleep(Duration::from_millis(100)).await; - for ((cell, checkpoint), receipt) in cells.iter().zip(&previous).zip(&receipts) { - let status = cell.status().await?; - ensure!(status.checkpoint == *checkpoint); - ensure!(status.archived < receipt.version.position.index); - } - store.archive_hang.store(false, Ordering::Release); - store.archive_resume.notify_waiters(); - store.archive_heads_hang.store(false, Ordering::Release); - store.archive_heads_resume.notify_waiters(); - for cell in &cells { - let durable = cell - .command( - definition.increment.clone(), - 1_i64.to_be_bytes().to_vec(), - public_node::archived_options(sequence)?, - ) - .await?; - ensure!(durable.result == (sequence as i64).to_be_bytes()); - } - tokio::time::timeout(Duration::from_secs(5), async { - for (cell, receipt) in cells.iter().zip(&receipts) { - loop { - let status = cell.status().await?; - if status.checkpoint.index >= receipt.version.position.index { - ensure!(status.checkpoint.index <= status.archived); - break; - } - tokio::time::sleep(Duration::from_millis(5)).await; - } - } - Ok::<_, anyhow::Error>(()) - }) - .await - .context("short archived histories did not checkpoint fairly")??; - store.archive_hang.store(true, Ordering::Release); - } - ensure!( - calls.load(Ordering::Acquire) == 6, - "durability retries executed callbacks twice" - ); - Ok::<_, anyhow::Error>(()) - } - .await; - drop(release); - let closed = service.close().await; - let stopped = node.shutdown(ShutdownMode::Preserve).await; - result?; - ensure!(closed?.unfinished.is_empty()); - stopped?; - - // Independent object-store-only restore proves compaction kept the complete - // recovery set, including command receipts stored in the SQL image. - let catalog = Catalog::new( - store.clone(), - CatalogOptions { - prefix: "host/test".into(), - ..Default::default() - }, - )?; - let selected = catalog.existing_namespace(namespace.clone()).await?; - for registration in ®istrations { - let birth = selected - .birth(®istration.birth.scope.cell) - .await? - .context("birth")?; - let archive = - selected.cell_archive(&birth, LtxHost::default(), ArchiveLimits::default())?; - let set = archive - .recovery_set() - .await? - .context("confirmed recovery set")?; - let target = tempfile::tempdir()?; - let generations = GenerationStore::open( - target.path().join("images"), - Arc::new(DirectFileSystem), - GenerationLimits::default(), - )?; - let pin = set.restore(&generations, images::id(91))?; - let query = images::query(pin)?; - ensure!( - query - .connection() - .query_row("SELECT value FROM counter", [], |row| row.get::<_, i64>(0))? - == 2 - ); - } - let node = public_node::node(root.path(), definition.application.clone()).await?; - let service = public_node::cells_many( - &node, - definition.application.clone(), - store, - registrations.clone(), - true, - host(), - TransportOptions::default(), - ) - .await?; - let result = async { - for registration in ®istrations { - let cell = service.cell(namespace.namespace, ®istration.birth.scope.cell)?; - public_node::ready(&cell).await?; - let receipt = cell - .command( - definition.increment.clone(), - 1_i64.to_be_bytes().to_vec(), - public_node::archived_options(2)?, - ) - .await?; - ensure!(receipt.result == 2_i64.to_be_bytes()); - } - ensure!( - calls.load(Ordering::Acquire) == 6, - "recovered compacted receipts re-executed callbacks" - ); - Ok::<_, anyhow::Error>(()) - } - .await; - let closed = service.close().await; - let stopped = node.shutdown(ShutdownMode::Preserve).await; - result?; - ensure!(closed?.unfinished.is_empty()); - stopped?; - Ok(()) -} diff --git a/crates/cellar/tests/support/raft_archive_wakeup.rs b/crates/cellar/tests/support/raft_archive_wakeup.rs deleted file mode 100644 index af6f89a69..000000000 --- a/crates/cellar/tests/support/raft_archive_wakeup.rs +++ /dev/null @@ -1,106 +0,0 @@ -// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. - -//! Committed images and confirmed archive cuts are explicit host wakeups. -#![allow(clippy::disallowed_methods)] -use super::public_node; -use anyhow::{ensure, Context as _, Result}; -use cellar::sdk::{consensus::*, ShutdownMode}; -use std::{ - sync::{ - atomic::{AtomicUsize, Ordering}, - Arc, - }, - time::{Duration, Instant}, -}; - -pub async fn run() -> Result<()> { - let (gate, _entered) = public_node::Gate::new(); - gate.release(); - let calls = Arc::new(AtomicUsize::new(0)); - let definition = public_node::counter(calls.clone(), gate)?; - let mut namespace = NamespaceConfig::new( - ClusterId::new([1; 16])?, - NamespaceId::new([2; 16])?, - NamespaceMode::Raft, - definition.schema, - ); - namespace.voters = VoterPolicy::SingleVoterDevelopment; - let registration = RaftCellRegistration { - placement: RaftCellPlacement::Replica, - namespace: namespace.clone(), - class: "Counter".into(), - birth: BirthProposal { - scope: GroupScope { - cluster: namespace.cluster, - namespace: namespace.namespace, - cell: "Counter:wakeup".into(), - group: GroupId::new([10; 16])?, - incarnation: IncarnationId::new([3; 16])?, - }, - voters: vec![InitialMember { - member: MemberId::new(1)?, - node: NodeId::new([1; 16])?, - }], - learners: vec![], - }, - }; - let root = tempfile::tempdir()?; - let node = public_node::node(root.path(), definition.application.clone()).await?; - let service = public_node::cells_with_host( - &node, - definition.application.clone(), - Arc::new(object_store::memory::InMemory::new()), - registration, - false, - RaftHostOptions { - // Only maintenance/retries are slow. Raft election and physical - // completion events retain their normal scheduling. - poll_ms: 1000, - ..Default::default() - }, - ) - .await?; - let result = async { - let cell = service.cell(namespace.namespace, "Counter:wakeup")?; - public_node::ready(&cell).await?; - let started = Instant::now(); - let mut acknowledged = 0; - // Eight sequential durable commands force eight distinct publications. - // Four seconds allows ample real file I/O, but cannot accommodate one - // maintenance interval per command. This is a scheduling regression, - // not a production latency target or a test of a mocked archive proof. - let completion = tokio::time::timeout(Duration::from_secs(4), async { - for sequence in 1..=8 { - let receipt = cell - .command( - definition.increment.clone(), - 1_i64.to_be_bytes().to_vec(), - public_node::archived_options(sequence)?, - ) - .await?; - ensure!(receipt.durability == Durability::S3); - ensure!(receipt.result == (sequence as i64).to_be_bytes()); - let observed = cell.status().await?; - ensure!(observed.archived >= receipt.version.position.index); - acknowledged += 1; - } - Ok::<_, anyhow::Error>(()) - }) - .await; - eprintln!( - "archive wake probe: {acknowledged}/8 acknowledged in {:?}, {} callbacks", - started.elapsed(), - calls.load(Ordering::Acquire) - ); - completion.context("S3 completion still depends on maintenance polling")??; - ensure!(calls.load(Ordering::Acquire) == 8); - Ok::<_, anyhow::Error>(()) - } - .await; - let closed = service.close().await; - let stopped = node.shutdown(ShutdownMode::Preserve).await; - result?; - ensure!(closed?.unfinished.is_empty()); - stopped?; - Ok(()) -} diff --git a/crates/cellar/tests/support/raft_background_admission.rs b/crates/cellar/tests/support/raft_background_admission.rs deleted file mode 100644 index 2ed0de2ce..000000000 --- a/crates/cellar/tests/support/raft_background_admission.rs +++ /dev/null @@ -1,265 +0,0 @@ -// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. - -//! A held physical SQL worker makes archive/command admission races repeatable. -#![allow(clippy::disallowed_methods)] -use super::{bundle_faults as faults, public_node}; -use anyhow::{ensure, Context as _, Result}; -use cellar::sdk::{consensus::*, Output, RaftClass, Registry, ShutdownMode}; -use std::{ - sync::{ - atomic::{AtomicUsize, Ordering}, - Arc, - }, - time::Duration, -}; - -struct ArchiveRelease(Arc); - -impl Drop for ArchiveRelease { - fn drop(&mut self) { - self.0.archive_hang.store(false, Ordering::Release); - self.0.archive_resume.notify_waiters(); - } -} - -fn definition( - calls: Arc, - gate: Arc, -) -> Result { - let class = RaftClass::new("Counter", 1, [21;32])? - .command("Increment", [22;32], move |context,input| { - let delta=i64::from_be_bytes(input.try_into()?); - calls.fetch_add(1,Ordering::SeqCst); - context.sql().execute_batch("CREATE TABLE IF NOT EXISTS counter(value INTEGER); INSERT INTO counter SELECT 0 WHERE NOT EXISTS(SELECT 1 FROM counter)")?; - context.sql().execute("UPDATE counter SET value=value+?1",[delta])?; - if delta == 1000 { gate.wait()?; } - let value:i64=context.sql().query_row("SELECT value FROM counter",[],|row|row.get(0))?; - Ok(value.to_be_bytes().to_vec()) - })? - .query("Get",[23;32],|sql,_| { - let value:i64=sql.query_row("SELECT value FROM counter",[],|row|row.get(0))?; - Ok(value.to_be_bytes().to_vec()) - })?; - Ok(public_node::Counter { - schema: class.schema_digest(), - increment: class.contract("Increment").unwrap(), - get: class.contract("Get").unwrap(), - application: Arc::new( - Registry::new(|_, _| Box::pin(async { Ok(Output::Done) })).raft_class(class)?, - ), - }) -} - -pub async fn run(cancel: bool) -> Result<()> { - let (gate, mut entered) = public_node::Gate::new(); - let release_callback = public_node::Release(gate.clone()); - let calls = Arc::new(AtomicUsize::new(0)); - let definition = definition(calls.clone(), gate.clone())?; - let mut namespace = NamespaceConfig::new( - ClusterId::new([1; 16])?, - NamespaceId::new([2; 16])?, - NamespaceMode::Raft, - definition.schema, - ); - namespace.voters = VoterPolicy::SingleVoterDevelopment; - let registrations = (10..12) - .map(|id| { - Ok(RaftCellRegistration { - placement: RaftCellPlacement::Replica, - namespace: namespace.clone(), - class: "Counter".into(), - birth: BirthProposal { - scope: GroupScope { - cluster: namespace.cluster, - namespace: namespace.namespace, - cell: format!("Counter:background-{id}"), - group: GroupId::new([id; 16])?, - incarnation: IncarnationId::new([3; 16])?, - }, - voters: vec![InitialMember { - member: MemberId::new(1)?, - node: NodeId::new([1; 16])?, - }], - learners: vec![], - }, - }) - }) - .collect::>>()?; - let root = tempfile::tempdir()?; - let node = public_node::node(root.path(), definition.application.clone()).await?; - let store = Arc::new(faults::Store::default()); - store.archive_hang.store(true, Ordering::Release); - let mut release_archive = Some(ArchiveRelease(store.clone())); - let service = public_node::cells_many( - &node, - definition.application.clone(), - store.clone(), - registrations.clone(), - false, - RaftHostOptions { - blocking_workers: 1, - archive: RaftArchiveOptions { - max_batch_groups: 1, - ..Default::default() - }, - ..Default::default() - }, - TransportOptions::default(), - ) - .await?; - let result = async { - let cells = registrations - .iter() - .map(|r| service.cell(namespace.namespace, &r.birth.scope.cell)) - .collect::, _>>()?; - for cell in &cells { - public_node::ready(cell).await?; - } - cells[0] - .command( - definition.increment.clone(), - 1_i64.to_be_bytes().to_vec(), - public_node::options(1)?, - ) - .await?; - tokio::time::timeout(Duration::from_secs(5), async { - while store.archive_attempts.load(Ordering::Acquire) == 0 { - tokio::time::sleep(Duration::from_millis(2)).await; - } - }) - .await - .context("first archive producer entered object I/O")?; - cells[1] - .command( - definition.increment.clone(), - 10_i64.to_be_bytes().to_vec(), - public_node::options(1)?, - ) - .await?; - let before = cells[0] - .command( - definition.increment.clone(), - 2_i64.to_be_bytes().to_vec(), - public_node::options(2)?, - ) - .await?; - // Both new cuts are due, but the first producer still owns object I/O. - // Occupy the sole physical SQL worker using the other cell's callback. - let mut blocking_options = public_node::options(2)?; - blocking_options.timeout = Duration::from_secs(10); - let blocking = cells[1].command( - definition.increment.clone(), - 1000_i64.to_be_bytes().to_vec(), - blocking_options, - ); - tokio::time::timeout(Duration::from_secs(5), entered.recv()) - .await? - .context("blocked SQL callback")?; - ensure!(calls.load(Ordering::SeqCst) == 4); - drop(release_archive.take()); - // The busy cell cannot be captured. The only eligible cut belongs to - // the target, and it queues behind the held physical SQL worker. - tokio::time::timeout(Duration::from_secs(5), async { - loop { - let resources = service.resource_usage(); - if resources.groups.is_some_and(|g| g.sql_busy == 2) - && resources.sql.is_some_and(|s| s.work_reservations >= 2) - { - return; - } - tokio::time::sleep(Duration::from_millis(2)).await; - } - }) - .await - .context("target archive capture queued behind SQL worker")?; - let mut options = public_node::options(3)?; - options.timeout = if cancel { - Duration::from_millis(200) - } else { - Duration::from_secs(5) - }; - let waiting = cells[0].command( - definition.increment.clone(), - 4_i64.to_be_bytes().to_vec(), - options.clone(), - ); - tokio::pin!(waiting); - let early = tokio::time::timeout(Duration::from_millis(30), waiting.as_mut()).await; - ensure!( - early.is_err(), - "background SQL displaced the free command slot: {early:?}" - ); - ensure!( - calls.load(Ordering::SeqCst) == 4, - "waiting command executed while physical worker held" - ); - let overflow = cells[0] - .command( - definition.increment.clone(), - 8_i64.to_be_bytes().to_vec(), - public_node::options(4)?, - ) - .await; - ensure!( - matches!(overflow, Err(CommandFailure::Overloaded)), - "second command bypassed the single pending slot: {overflow:?}" - ); - if cancel { - let cancelled = tokio::time::timeout(Duration::from_secs(2), waiting.as_mut()).await?; - ensure!( - matches!(cancelled, Err(CommandFailure::Unavailable)), - "unsubmitted cancelled command had an unexpected result: {cancelled:?}" - ); - } - gate.release(); - ensure!(blocking.await?.result == 1010_i64.to_be_bytes()); - let receipt = if cancel { - let read = cells[0] - .query( - definition.get.clone(), - vec![], - QueryOptions { - consistency: QueryConsistency::AtLeast(before.version_token), - ..Default::default() - }, - ) - .await?; - ensure!( - read.result == 3_i64.to_be_bytes() && calls.load(Ordering::SeqCst) == 4, - "cancelled queued command executed after worker release" - ); - options.timeout = Duration::from_secs(5); - cells[0] - .command( - definition.increment.clone(), - 4_i64.to_be_bytes().to_vec(), - options, - ) - .await? - } else { - waiting.await? - }; - ensure!(receipt.result == 7_i64.to_be_bytes() && calls.load(Ordering::SeqCst) == 5); - let read = cells[0] - .query( - definition.get.clone(), - vec![], - QueryOptions { - consistency: QueryConsistency::AtLeast(receipt.version_token), - ..Default::default() - }, - ) - .await?; - ensure!(read.result == 7_i64.to_be_bytes()); - Ok(()) - } - .await; - drop(release_callback); - drop(release_archive); - let closed = service.close().await; - let stopped = node.shutdown(ShutdownMode::Preserve).await; - result?; - closed?; - stopped?; - Ok(()) -} diff --git a/crates/cellar/tests/support/raft_bundle_native.rs b/crates/cellar/tests/support/raft_bundle_native.rs deleted file mode 100644 index ff18067d5..000000000 --- a/crates/cellar/tests/support/raft_bundle_native.rs +++ /dev/null @@ -1,296 +0,0 @@ -// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. - -//! Two real SDK histories share an immutable object, never a head or proof. -#![allow(clippy::disallowed_methods)] -use super::{bundle_faults as faults, bundle_images as images, public_node}; -use anyhow::{ensure, Context as _, Result}; -use cellar::application::consensus::verification::{ArchiveRecoverySet, Catalog}; -use cellar::sdk::{consensus::*, ShutdownMode}; -use cellar_ltx::{ - generation::{GenerationLimits, GenerationStore}, - DirectFileSystem, LtxHost, -}; -use futures_util::TryStreamExt; -use object_store::ObjectStore; -use std::{ - sync::{ - atomic::{AtomicUsize, Ordering}, - Arc, - }, - time::Duration, -}; - -struct Release(Arc); - -impl Drop for Release { - fn drop(&mut self) { - self.0.archive_hang.store(false, Ordering::Release); - self.0.archive_resume.notify_waiters(); - self.0.archive_heads_hang.store(false, Ordering::Release); - self.0.archive_heads_resume.notify_waiters(); - } -} - -fn restore(set: &ArchiveRecoverySet, expected: i64) -> Result<()> { - let root = tempfile::tempdir()?; - let images = GenerationStore::open( - root.path().join("restore"), - Arc::new(DirectFileSystem), - GenerationLimits::default(), - )?; - let pin = set.restore(&images, images::id(91))?; - let query = images::query(pin)?; - ensure!( - query - .connection() - .query_row("SELECT value FROM counter", [], |row| row.get::<_, i64>(0))? - == expected - ); - Ok(()) -} - -pub async fn run() -> Result<()> { - let (gate, _entered) = public_node::Gate::new(); - gate.release(); - let calls = Arc::new(AtomicUsize::new(0)); - let definition = public_node::counter(calls.clone(), gate)?; - let mut namespace = NamespaceConfig::new( - ClusterId::new([1; 16])?, - NamespaceId::new([2; 16])?, - NamespaceMode::Raft, - definition.schema, - ); - namespace.voters = VoterPolicy::SingleVoterDevelopment; - let registrations = (10..12) - .map(|id| { - Ok(RaftCellRegistration { - placement: RaftCellPlacement::Replica, - namespace: namespace.clone(), - class: "Counter".into(), - birth: BirthProposal { - scope: GroupScope { - cluster: namespace.cluster, - namespace: namespace.namespace, - cell: format!("Counter:bundle-{id}"), - group: GroupId::new([id; 16])?, - incarnation: IncarnationId::new([3; 16])?, - }, - voters: vec![InitialMember { - member: MemberId::new(1)?, - node: NodeId::new([1; 16])?, - }], - learners: vec![], - }, - }) - }) - .collect::>>()?; - let root = tempfile::tempdir()?; - let node = public_node::node(root.path(), definition.application.clone()).await?; - let store = Arc::new(faults::Store::default()); - let release = Release(store.clone()); - store.archive_hang.store(true, Ordering::Release); - let service = public_node::cells_many( - &node, - definition.application.clone(), - store.clone(), - registrations.clone(), - false, - RaftHostOptions { - archive: RaftArchiveOptions { - max_batch_groups: 2, - ..Default::default() - }, - ..Default::default() - }, - TransportOptions::default(), - ) - .await?; - let result = async { - let cells = registrations - .iter() - .map(|registration| service.cell(namespace.namespace, ®istration.birth.scope.cell)) - .collect::, _>>()?; - for cell in &cells { - public_node::ready(cell).await?; - } - let first = public_node::options(1)?; - cells[0] - .command( - definition.increment.clone(), - 3_i64.to_be_bytes().to_vec(), - first.clone(), - ) - .await?; - // Freeze each producer after capture, then finish the old heads together. - // Concurrent publishers otherwise complete their physical replay at - // different times and need not batch the next cuts. - tokio::time::timeout(Duration::from_secs(5), async { - while store.archive_attempts.load(Ordering::Acquire) == 0 { - tokio::time::sleep(Duration::from_millis(2)).await; - } - }) - .await - .context("captured first archive cut")?; - cells[1] - .command( - definition.increment.clone(), - 7_i64.to_be_bytes().to_vec(), - first, - ) - .await?; - tokio::time::timeout(Duration::from_secs(5), async { - while store.archive_attempts.load(Ordering::Acquire) < 2 { - tokio::time::sleep(Duration::from_millis(2)).await; - } - }) - .await - .context("captured both old archive cuts")?; - let second = public_node::options(2)?; - for (cell, value) in cells.iter().zip([3_i64, 7]) { - cell.command( - definition.increment.clone(), - value.to_be_bytes().to_vec(), - second.clone(), - ) - .await?; - } - store.fail_bundle_after.store(true, Ordering::Release); - store.archive_heads_hang.store(true, Ordering::Release); - store.archive_hang.store(false, Ordering::Release); - store.archive_resume.notify_waiters(); - tokio::time::timeout(Duration::from_secs(5), async { - while store.archive_heads_waiting.load(Ordering::Acquire) < 2 { - tokio::time::sleep(Duration::from_millis(2)).await; - } - }) - .await - .context("both old archive heads reached publication")?; - drop(release); - // Upgrade retries to S3 without changing command identity, input or - // deadline. The lost bundle PUT response must be reconciled by readback. - let mut durable = second; - durable.durability = Durability::S3; - durable.timeout = Duration::from_secs(10); - let mut receipts = Vec::new(); - for (cell, value) in cells.iter().zip([3_i64, 7]) { - let receipt = cell - .command( - definition.increment.clone(), - value.to_be_bytes().to_vec(), - durable.clone(), - ) - .await?; - ensure!( - receipt.result == (2 * value).to_be_bytes() && receipt.durability == Durability::S3 - ); - receipts.push(receipt); - } - ensure!( - !store.fail_bundle_after.load(Ordering::Acquire), - "native producer never wrote a bundle" - ); - ensure!( - calls.load(Ordering::Acquire) == 4, - "S3 retry repeated a business callback" - ); - let performance = service.performance(); - ensure!(performance.available && !performance.saturated); - let counter = |name| { - performance - .counters - .iter() - .find(|counter| counter.name == name) - .unwrap() - .value - }; - ensure!(counter("object_put_requested_bytes") > counter("object_put_confirmed_bytes")); - ensure!(counter("object_get_body_bytes") > 0); - ensure!( - performance - .operations - .iter() - .find(|op| op.name == "object_put_create") - .unwrap() - .failed - > 0, - "lost bundle PUT reply was incorrectly recorded as a confirmed request" - ); - let catalog = Catalog::new( - store.clone(), - CatalogOptions { - prefix: "host/test".into(), - ..Default::default() - }, - )?; - let selected = catalog.existing_namespace(namespace.clone()).await?; - let mut archives = Vec::new(); - for (registration, receipt) in registrations.iter().zip(receipts) { - let birth = selected - .birth(®istration.birth.scope.cell) - .await? - .context("selected birth")?; - let archive = - selected.cell_archive(&birth, LtxHost::default(), ArchiveLimits::default())?; - let metadata = archive - .recovery_metadata() - .await? - .context("published head")?; - ensure!( - metadata.scope() == ®istration.birth.scope - && metadata.position().index >= receipt.version.position.index - ); - archives.push((archive, metadata.reference())); - } - Ok::<_, anyhow::Error>(archives) - } - .await; - store.archive_hang.store(false, Ordering::Release); - store.archive_resume.notify_waiters(); - let close = service.close().await?; - ensure!(close.unfinished.is_empty()); - node.shutdown(ShutdownMode::Preserve).await?; - let archives = result?; - let objects = store.inner.list(None).try_collect::>().await?; - let bundles: Vec<_> = objects - .iter() - .filter(|object| object.location.as_ref().contains("/archives/bundles/")) - .collect(); - ensure!( - bundles.len() == 1, - "expected one shared bundle, observed {}", - bundles.len() - ); - ensure!( - objects - .iter() - .filter(|object| object.location.as_ref().contains("/tails/")) - .count() - == 0, - "the new tail must reside inside the shared bundle" - ); - for ((archive, reference), value) in archives.iter().zip([6, 14]) { - let set = archive.recovery_set_at(reference).await?; - restore(&set, value)?; - } - ensure!( - archives[0].0.recovery_set_at(&archives[1].1).await.is_err(), - "foreign manifest supplied a history" - ); - let path = &bundles[0].location; - let bytes = store.inner.get(path).await?.bytes().await?; - let mut corrupt = bytes.to_vec(); - *corrupt.last_mut().context("nonempty bundle")? ^= 1; - store.inner.put(path, corrupt.into()).await?; - for (archive, reference) in &archives { - ensure!( - archive.recovery_set_at(reference).await.is_err(), - "corrupt bundle supplied successful recovery" - ); - } - store.inner.put(path, bytes.into()).await?; - for ((archive, reference), value) in archives.iter().zip([6, 14]) { - restore(&archive.recovery_set_at(reference).await?, value)?; - } - cellar::note!("native bundle: two exact histories, one shared snapshot/tail object, confirmed lost PUT response, isolated post-close restores, corruption refusal, four callbacks"); - Ok(()) -} diff --git a/crates/cellar/tests/support/raft_cache_pressure.rs b/crates/cellar/tests/support/raft_cache_pressure.rs index ebbb9ec85..7773592c8 100644 --- a/crates/cellar/tests/support/raft_cache_pressure.rs +++ b/crates/cellar/tests/support/raft_cache_pressure.rs @@ -51,6 +51,7 @@ async fn many_small_groups_checkpoint_under_shared_cache_pressure() -> Result<() host: RaftHostOptions::default(), }, RaftCellsServices { + recovery: None, listeners: NodeRaftListeners { control: tokio::net::TcpListener::bind("127.0.0.1:0").await?, data: tokio::net::TcpListener::bind("127.0.0.1:0").await?, diff --git a/crates/cellar/tests/support/raft_catalog_fault.rs b/crates/cellar/tests/support/raft_catalog_fault.rs index fa78a7f86..2c9348bc8 100644 --- a/crates/cellar/tests/support/raft_catalog_fault.rs +++ b/crates/cellar/tests/support/raft_catalog_fault.rs @@ -1,6 +1,6 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. -//! Conditional writes are real InMemory operations; faults lose responses only. +//! Conditional writes use the selected store unchanged; faults surround real operations. use futures_util::{stream::BoxStream, StreamExt, TryStreamExt}; use object_store::{ path::Path, GetOptions, GetResult, GetResultPayload, ListResult, MultipartUpload, ObjectMeta, @@ -8,9 +8,82 @@ use object_store::{ }; use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; +fn is_managed_recovery_record(path: &Path) -> bool { + path.as_ref() + .contains("/fleet/managed-transition-recovery/") + || path + .as_ref() + .contains("/fleet/managed-transition-recovery-fences/") + || path + .as_ref() + .contains("/fleet/managed-transition-recovery-replacements/") + || path + .as_ref() + .contains("/fleet/managed-transition-recovery-seed-quorums/") + || (path.as_ref().contains("/managed-recovery-seeds/") + && matches!(path.filename(), Some("complete" | "activation"))) +} + tokio::task_local! { static READ_COUNTS: std::cell::RefCell>; static MISSING_READ: String; + static GATED_READ: (String, std::sync::Arc); + static GATED_STORE_READ: (std::sync::Arc<()>, String, std::sync::Arc); + static DENIED_ARCHIVE_CREATION: (&'static str, std::cell::Cell); + static DIRECTORY_CONFIRMATION: std::cell::Cell; +} + +#[derive(Clone, Copy, PartialEq, Eq)] +enum DirectoryConfirmation { + WaitingForWrite, + WaitingForRead, + Delivered, +} + +/// Lose only this operation's first directory confirmation after a real CAS. +#[allow(dead_code)] +pub async fn lose_directory_confirmation( + work: impl std::future::Future, +) -> (T, bool) { + DIRECTORY_CONFIRMATION + .scope( + std::cell::Cell::new(DirectoryConfirmation::WaitingForWrite), + async { + let result = work.await; + let delivered = DIRECTORY_CONFIRMATION + .with(|state| state.get() == DirectoryConfirmation::Delivered); + (result, delivered) + }, + ) + .await +} + +/// Interrupt one foreground archive initializer before its conditional create. +#[allow(dead_code)] +pub async fn deny_archive_creation( + suffix: &'static str, + work: impl std::future::Future, +) -> (T, bool) { + DENIED_ARCHIVE_CREATION + .scope((suffix, std::cell::Cell::new(false)), async { + let result = work.await; + let delivered = DENIED_ARCHIVE_CREATION.with(|(_, delivered)| delivered.get()); + (result, delivered) + }) + .await +} + +// Keep a lost observation on one foreground client's store, without delaying +// background trust refresh or the controller's independent catalog client. +pub async fn gate_store_read( + store: &Store, + suffix: String, + gate: std::sync::Arc, + work: impl std::future::Future, +) -> T { + GATED_STORE_READ + .scope((store.gate_identity.clone(), suffix, gate), work) + .await } // Keep a read fault local to one verifier; running nodes still see the store. @@ -19,6 +92,16 @@ pub async fn missing_read(path: String, work: impl std::future::Future( + path: String, + gate: std::sync::Arc, + work: impl std::future::Future, +) -> T { + GATED_READ.scope((path, gate), work).await +} + // Shared fault adapter: only catalog work-count regressions use this scope. #[allow(dead_code)] pub async fn count_reads( @@ -59,9 +142,26 @@ impl RegistryGate { #[derive(Debug, Default)] pub struct Store { + external: Option>, + gate_identity: std::sync::Arc<()>, + pub group_payload_reads: AtomicUsize, + pub denied_group_payload_reads: AtomicUsize, + pub deny_group_payload_reads: AtomicBool, + pub deny_group_archive_writes: AtomicBool, + pub denied_group_archive_writes: AtomicUsize, + pub deny_registry_writes: std::sync::Arc, + pub denied_registry_writes: AtomicUsize, + pub registry_denial_on_payload_refusal: std::sync::Mutex>>, + pub range_before: RegistryGate, + pub range_genesis_after: RegistryGate, + pub first_group_birth_after: RegistryGate, + pub range_page_puts: AtomicUsize, pub fail_closure_after: AtomicBool, + pub fail_fleet_before: AtomicBool, pub fail_fleet_after: AtomicBool, pub fail_fleet_confirmation: AtomicBool, + pub fail_recovery_record_confirmation: AtomicBool, + fail_recovery_record_read: AtomicBool, pub fail_fleet_read: AtomicBool, // Background trust refresh must not steal the mutator's confirmation loss. fleet_confirmation_task: std::sync::Mutex>, @@ -76,6 +176,7 @@ pub struct Store { pub fleet_read_resume: tokio::sync::Notify, pub registry_before: RegistryGate, pub registry_after: RegistryGate, + pub group_archive_read: RegistryGate, pub fail_registry_before: AtomicBool, pub fail_registry_after: AtomicBool, pub registry_writes: AtomicUsize, @@ -167,6 +268,9 @@ pub struct Store { pub view_waiting: AtomicBool, pub view_resume: tokio::sync::Notify, pub fail_directory_reads: AtomicBool, + pub hold_group_head_reads: AtomicBool, + pub active_group_head_reads: AtomicUsize, + pub group_head_reads_resume: tokio::sync::Notify, pub hold_view_reads: AtomicBool, pub view_reads: AtomicUsize, pub active_view_reads: AtomicUsize, @@ -195,6 +299,17 @@ impl Drop for BirthWaiting<'_> { } impl Store { + pub fn around(store: std::sync::Arc) -> Self { + Self { + external: Some(store), + ..Default::default() + } + } + + fn backend(&self) -> &dyn ObjectStore { + self.external.as_deref().unwrap_or(&self.inner) + } + async fn wait_birth(&self) { self.birth_waiting.store(true, Ordering::Release); let _waiting = BirthWaiting(&self.birth_waiting); @@ -245,6 +360,27 @@ impl ObjectStore for Store { payload: PutPayload, options: PutOptions, ) -> object_store::Result { + if matches!(options.mode, PutMode::Create) + && path.as_ref().contains("/cell-group-archives/") + && DENIED_ARCHIVE_CREATION + .try_with(|(suffix, delivered)| { + path.as_ref().ends_with(suffix) && { + delivered.set(true); + true + } + }) + .unwrap_or(false) + { + return Err(failure()); + } + if path.as_ref().contains("/cell-groups/ranges/") { + self.range_page_puts.fetch_add(1, Ordering::AcqRel); + } + let directory_cas = path.as_ref().ends_with("/cell-groups/head") + && matches!(options.mode, PutMode::Update(_)); + if directory_cas { + self.range_before.pause().await; + } if path.as_ref().contains("/root-extents/") { self.root_extent_puts.fetch_add(1, Ordering::AcqRel); } @@ -347,6 +483,20 @@ impl ObjectStore for Store { return Err(failure()); } } + if path.as_ref().contains("/cell-group-archives/") + && self.deny_group_archive_writes.load(Ordering::Acquire) + { + self.denied_group_archive_writes + .fetch_add(1, Ordering::AcqRel); + return Err(failure()); + } + if path.as_ref().contains("/cell-group-archives/") + && path.as_ref().ends_with("/ownership/registry") + && self.deny_registry_writes.load(Ordering::Acquire) + { + self.denied_registry_writes.fetch_add(1, Ordering::AcqRel); + return Err(failure()); + } if self.fail_before.swap(false, Ordering::AcqRel) { return Err(failure()); } @@ -375,12 +525,41 @@ impl ObjectStore for Store { } let fleet_cas = path.as_ref().ends_with("/fleet/head") && matches!(options.mode, PutMode::Update(_)); - let result = self.inner.put_opts(path, payload, options).await?; + if fleet_cas && self.fail_fleet_before.swap(false, Ordering::AcqRel) { + return Err(failure()); + } + let range_genesis = + path.as_ref().ends_with("/cell-groups/head") && matches!(options.mode, PutMode::Create); + let group_birth = path.as_ref().contains("/cell-groups/births/") + && matches!(options.mode, PutMode::Create); + let result = self.backend().put_opts(path, payload, options).await?; + if directory_cas { + let _ = DIRECTORY_CONFIRMATION.try_with(|state| { + if state.get() == DirectoryConfirmation::WaitingForWrite { + state.set(DirectoryConfirmation::WaitingForRead); + } + }); + } + if group_birth { + self.first_group_birth_after.pause().await; + } + if range_genesis { + self.range_genesis_after.pause().await; + } if path.as_ref().contains("/closures/") && self.fail_closure_after.swap(false, Ordering::AcqRel) { return Err(failure()); } + if is_managed_recovery_record(path) + && self + .fail_recovery_record_confirmation + .swap(false, Ordering::AcqRel) + { + self.fail_recovery_record_read + .store(true, Ordering::Release); + return Err(failure()); + } if fleet_cas && self.fail_fleet_confirmation.swap(false, Ordering::AcqRel) { *self.fleet_confirmation_task.lock().unwrap() = tokio::task::try_id(); self.fail_fleet_read.store(true, Ordering::Release); @@ -480,13 +659,50 @@ impl ObjectStore for Store { path: &Path, options: PutMultipartOptions, ) -> object_store::Result> { - self.inner.put_multipart_opts(path, options).await + self.backend().put_multipart_opts(path, options).await } async fn get_opts(&self, path: &Path, options: GetOptions) -> object_store::Result { + let gate = GATED_STORE_READ + .try_with(|(identity, suffix, gate)| { + (std::sync::Arc::ptr_eq(identity, &self.gate_identity) + && path.as_ref().ends_with(suffix)) + .then(|| gate.clone()) + }) + .ok() + .flatten(); + if let Some(gate) = gate { + gate.pause().await; + } + if path.as_ref().contains("/cell-group-archives/") && path.as_ref().contains("/chunks/") { + self.group_payload_reads.fetch_add(1, Ordering::AcqRel); + if self.deny_group_payload_reads.load(Ordering::Acquire) { + self.denied_group_payload_reads + .fetch_add(1, Ordering::AcqRel); + if let Some(denial) = self + .registry_denial_on_payload_refusal + .lock() + .unwrap() + .as_ref() + { + denial.store(true, Ordering::Release); + } + return Err(failure()); + } + } + if path.as_ref().contains("/cell-group-archives/") { + self.group_archive_read.pause().await; + } let _ = READ_COUNTS.try_with(|counts| { *counts.borrow_mut().entry(path.to_string()).or_default() += 1; }); + let gate = GATED_READ + .try_with(|(key, gate)| (key == path.as_ref()).then(|| gate.clone())) + .ok() + .flatten(); + if let Some(gate) = gate { + gate.pause().await; + } if MISSING_READ .try_with(|missing| missing == path.as_ref()) .unwrap_or(false) @@ -499,6 +715,24 @@ impl ObjectStore for Store { )), }); } + if path.as_ref().ends_with("/cell-groups/head") + && DIRECTORY_CONFIRMATION + .try_with(|state| { + if state.get() != DirectoryConfirmation::WaitingForRead { + return false; + } + state.set(DirectoryConfirmation::Delivered); + true + }) + .unwrap_or(false) + { + return Err(failure()); + } + if is_managed_recovery_record(path) + && self.fail_recovery_record_read.swap(false, Ordering::AcqRel) + { + return Err(failure()); + } if path.as_ref().ends_with("/fleet/head") && *self.fleet_confirmation_task.lock().unwrap() == tokio::task::try_id() && self.fail_fleet_read.swap(false, Ordering::AcqRel) @@ -549,6 +783,26 @@ impl ObjectStore for Store { { return Err(failure()); } + if path.filename() == Some("head") { + struct Reading<'a>(&'a AtomicUsize); + + impl Drop for Reading<'_> { + fn drop(&mut self) { + self.0.fetch_sub(1, Ordering::AcqRel); + } + } + self.active_group_head_reads.fetch_add(1, Ordering::AcqRel); + let _reading = Reading(&self.active_group_head_reads); + loop { + let resumed = self.group_head_reads_resume.notified(); + tokio::pin!(resumed); + resumed.as_mut().enable(); + if !self.hold_group_head_reads.load(Ordering::Acquire) { + break; + } + resumed.await; + } + } if path.filename() == Some("view") { self.view_reads.fetch_add(1, Ordering::AcqRel); @@ -736,7 +990,7 @@ impl ObjectStore for Store { if self.fail_read.swap(false, Ordering::AcqRel) { return Err(failure()); } - let mut result = self.inner.get_opts(path, options).await?; + let mut result = self.backend().get_opts(path, options).await?; if root_extent { self.root_extent_hits.fetch_add(1, Ordering::AcqRel); } @@ -790,7 +1044,7 @@ impl ObjectStore for Store { } async fn delete(&self, path: &Path) -> object_store::Result<()> { - self.inner.delete(path).await + self.backend().delete(path).await } fn list(&self, prefix: Option<&Path>) -> BoxStream<'static, object_store::Result> { @@ -798,7 +1052,7 @@ impl ObjectStore for Store { if self.reject_listing.load(Ordering::Acquire) { return futures_util::stream::once(async { Err(failure()) }).boxed(); } - self.inner.list(prefix) + self.backend().list(prefix) } async fn list_with_delimiter(&self, prefix: Option<&Path>) -> object_store::Result { @@ -806,14 +1060,14 @@ impl ObjectStore for Store { if self.reject_listing.load(Ordering::Acquire) { return Err(failure()); } - self.inner.list_with_delimiter(prefix).await + self.backend().list_with_delimiter(prefix).await } async fn copy(&self, from: &Path, to: &Path) -> object_store::Result<()> { - self.inner.copy(from, to).await + self.backend().copy(from, to).await } async fn copy_if_not_exists(&self, from: &Path, to: &Path) -> object_store::Result<()> { - self.inner.copy_if_not_exists(from, to).await + self.backend().copy_if_not_exists(from, to).await } } diff --git a/crates/cellar/tests/support/raft_cluster.rs b/crates/cellar/tests/support/raft_cluster.rs index 9a3885e90..47cb5a235 100644 --- a/crates/cellar/tests/support/raft_cluster.rs +++ b/crates/cellar/tests/support/raft_cluster.rs @@ -16,21 +16,12 @@ use std::{ }, time::Duration, }; -use tokio::{ - io::{AsyncRead, AsyncReadExt, AsyncWrite, AsyncWriteExt}, - net::{TcpListener, TcpStream}, - sync::watch, - task::{JoinHandle, JoinSet}, -}; +use tokio::net::TcpListener; -#[derive(Clone, Copy, Default)] -struct Fault { - blocked: bool, - stopped: bool, - hold_response: bool, - hold_after: Option, - hold_once: bool, -} +#[path = "raft_tcp_fault.rs"] +mod tcp_fault; +pub use tcp_fault::HeldResponses; +use tcp_fault::Proxy; #[derive(Clone, Copy, PartialEq, Eq)] enum Link { @@ -39,203 +30,6 @@ enum Link { Query, } -struct Proxy { - address: SocketAddr, - state: watch::Sender, - active: Arc, - attempts: Arc, - held_response_bytes: Arc, - held_connection: Arc, - task: Option>>, -} - -// This shared fixture is also compiled by targets which only cut connections. -#[allow(dead_code)] -pub struct HeldResponses(watch::Sender); - -impl Drop for HeldResponses { - fn drop(&mut self) { - self.0.send_modify(|fault| { - fault.hold_response = false; - fault.hold_after = None; - fault.hold_once = false; - }); - } -} - -async fn connection_cut(mut faults: watch::Receiver) { - loop { - let fault = *faults.borrow_and_update(); - if fault.blocked || fault.stopped || faults.changed().await.is_err() { - return; - } - } -} - -async fn forward_responses( - mut reader: impl AsyncRead + Unpin, - mut writer: impl AsyncWrite + Unpin, - mut faults: watch::Receiver, - held: Arc, - selected: Arc, - connection: usize, -) -> std::io::Result<()> { - let mut buffer = [0; 16 * 1024]; - let mut forwarded = 0usize; - loop { - let count = reader.read(&mut buffer).await?; - if count == 0 { - return Ok(()); - } - let mut offset = 0; - let mut counted = false; - while offset < count { - let mut fault = *faults.borrow_and_update(); - let chosen = selected.load(Ordering::SeqCst); - if fault.hold_once && chosen != 0 && chosen != connection { - fault.hold_response = false; - fault.hold_after = None; - } - let allowance = if fault.hold_response { - 0 - } else { - fault - .hold_after - .map_or(count - offset, |after| after.saturating_sub(forwarded)) - }; - if allowance == 0 { - if fault.hold_once - && selected - .compare_exchange(0, connection, Ordering::SeqCst, Ordering::SeqCst) - .is_err_and(|chosen| chosen != connection) - { - continue; - } - if !counted { - held.fetch_add(count - offset, Ordering::SeqCst); - counted = true; - } - if faults.changed().await.is_err() { - return Ok(()); - } - continue; - } - let end = count.min(offset.saturating_add(allowance)); - writer.write_all(&buffer[offset..end]).await?; - forwarded += end - offset; - offset = end; - } - } -} - -struct Active(Arc); - -impl Drop for Active { - fn drop(&mut self) { - self.0.fetch_sub(1, Ordering::SeqCst); - } -} - -impl Proxy { - async fn start(destination: SocketAddr) -> Result { - let listener = TcpListener::bind("127.0.0.1:0").await?; - let address = listener.local_addr()?; - let (state, mut changes) = watch::channel(Fault::default()); - let active = Arc::new(AtomicUsize::new(0)); - let attempts = Arc::new(AtomicUsize::new(0)); - let held_response_bytes = Arc::new(AtomicUsize::new(0)); - let held_connection = Arc::new(AtomicUsize::new(0)); - let held = held_response_bytes.clone(); - let selected = held_connection.clone(); - let counted = active.clone(); - let attempted = attempts.clone(); - let task = tokio::spawn(async move { - let mut connections = JoinSet::new(); - loop { - cellar::asyncrt::select! { - changed = changes.changed() => { - if changed.is_err() || changes.borrow().stopped { break; } - } - done = connections.join_next(), if !connections.is_empty() => { if let Some(done) = done { done?; } } - accepted = listener.accept(), if connections.len() < 8 => { - let (mut source, _) = accepted?; - // Match the native transport: fault injection must not - // add Nagle/delayed-ACK stalls to small snapshot chunks. - source.set_nodelay(true)?; - let connection=attempted.fetch_add(1, Ordering::SeqCst)+1; - if changes.borrow().blocked { continue; } - let cut = changes.clone(); - let held = held.clone(); - let selected = selected.clone(); - let active = counted.clone(); active.fetch_add(1, Ordering::SeqCst); - let guard = Active(active); - connections.spawn(async move { - let _guard = guard; - let fault = *cut.borrow(); - if fault.blocked || fault.stopped { return; } - // A cut drops both sockets. A response hold instead - // retains one bounded ciphertext chunk until release - // or either connection direction closes. - cellar::asyncrt::select_biased! { "test fault transitions close sockets before more bytes are forwarded"; - _ = connection_cut(cut.clone()) => (), - _ = async { - let target = TcpStream::connect(destination).await?; - target.set_nodelay(true)?; - let (mut from_source, to_source) = source.split(); - let (from_target, mut to_target) = target.into_split(); - cellar::asyncrt::select! { - _ = tokio::io::copy(&mut from_source, &mut to_target) => (), - _ = forward_responses(from_target, to_source, cut, held, selected, connection) => (), - } - Ok::<_, std::io::Error>(()) - } => (), - } - }); - } - } - } - connections.abort_all(); - while let Some(result) = connections.join_next().await { - if let Err(error) = result { - ensure!(error.is_cancelled(), "proxy connection panicked: {error}"); - } - } - Ok(()) - }); - Ok(Self { - address, - state, - active, - attempts, - held_response_bytes, - held_connection, - task: Some(task), - }) - } - - fn block(&self, blocked: bool) { - self.state.send_if_modified(|fault| { - let changed = fault.blocked != blocked; - fault.blocked = blocked; - changed - }); - } - - async fn close(&mut self) -> Result<()> { - self.state.send_modify(|fault| fault.stopped = true); - if let Some(task) = self.task.take() { - task.await??; - } - Ok(()) - } -} - -impl Drop for Proxy { - fn drop(&mut self) { - self.state.send_modify(|fault| fault.stopped = true); - } -} - /// One globally advertised data endpoint, shared by every source. Directory /// refresh therefore preserves this fault boundary rather than bypassing it. #[allow(dead_code)] // Shared fixture targets do not all use global endpoints. @@ -684,6 +478,7 @@ impl Cluster { .start_raft_cells( config, RaftCellsServices { + recovery: None, listeners, tls: public_node::tls()?, store: self.store.clone(), @@ -946,6 +741,7 @@ impl Cluster { .start_raft_cells( config, RaftCellsServices { + recovery: None, listeners, tls: public_node::tls()?, store: store.clone(), diff --git a/crates/cellar/tests/support/raft_controller_native.rs b/crates/cellar/tests/support/raft_controller_native.rs index 21f2ffc9d..6637b9e47 100644 --- a/crates/cellar/tests/support/raft_controller_native.rs +++ b/crates/cellar/tests/support/raft_controller_native.rs @@ -282,6 +282,7 @@ impl Fixture { host, }, RaftCellsServices { + recovery: None, listeners, tls: public_node::tls()?, store: store.clone(), diff --git a/crates/cellar/tests/support/raft_daemon_archive.rs b/crates/cellar/tests/support/raft_daemon_archive.rs new file mode 100644 index 000000000..54c46e4c4 --- /dev/null +++ b/crates/cellar/tests/support/raft_daemon_archive.rs @@ -0,0 +1,229 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Refuse archive writes at each voter while the real backend remains available. +use super::*; +use futures_util::stream::BoxStream; +use object_store::{ + path::Path as Key, GetOptions, GetResult, ListResult, MultipartUpload, ObjectMeta, ObjectStore, + PutMultipartOptions, PutOptions, PutPayload, PutResult, +}; + +const BLOCK: &str = "block-archive-writes"; + +#[derive(Debug)] +pub(super) struct Gate { + store: Arc, + block: PathBuf, + denied: PathBuf, +} + +impl Gate { + pub(super) fn new(store: Arc, root: &Path, voter: u8) -> Self { + Self { + store, + block: root.join(BLOCK), + denied: denied_path(root, voter), + } + } + + async fn permit_write(&self, key: &Key) -> object_store::Result<()> { + let error = |source: std::io::Error| object_store::Error::Generic { + store: "daemon archive fixture", + source: Box::new(source), + }; + if key.as_ref().contains("/cell-group-archives/") + && tokio::fs::try_exists(&self.block).await.map_err(error)? + { + tokio::fs::write(&self.denied, []).await.map_err(error)?; + return Err(error(std::io::Error::other( + "fixture refused an archive write", + ))); + } + Ok(()) + } +} + +impl std::fmt::Display for Gate { + fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + formatter.write_str("daemon archive fixture") + } +} + +#[async_trait::async_trait] +impl ObjectStore for Gate { + async fn put_opts( + &self, + key: &Key, + payload: PutPayload, + options: PutOptions, + ) -> object_store::Result { + self.permit_write(key).await?; + self.store.put_opts(key, payload, options).await + } + + async fn put_multipart_opts( + &self, + key: &Key, + options: PutMultipartOptions, + ) -> object_store::Result> { + self.permit_write(key).await?; + self.store.put_multipart_opts(key, options).await + } + + async fn get_opts(&self, key: &Key, options: GetOptions) -> object_store::Result { + self.store.get_opts(key, options).await + } + + async fn delete(&self, key: &Key) -> object_store::Result<()> { + self.store.delete(key).await + } + + fn list(&self, prefix: Option<&Key>) -> BoxStream<'static, object_store::Result> { + self.store.list(prefix) + } + + async fn list_with_delimiter(&self, prefix: Option<&Key>) -> object_store::Result { + self.store.list_with_delimiter(prefix).await + } + + async fn copy(&self, from: &Key, to: &Key) -> object_store::Result<()> { + self.permit_write(to).await?; + self.store.copy(from, to).await + } + + async fn copy_if_not_exists(&self, from: &Key, to: &Key) -> object_store::Result<()> { + self.permit_write(to).await?; + self.store.copy_if_not_exists(from, to).await + } +} + +fn denied_path(root: &Path, voter: u8) -> PathBuf { + root.join(format!("archive-write-denied-{voter}")) +} + +pub(super) async fn verify( + root: &Path, + client: &reqwest::Client, + address: &ManagedIngress, + before: &Value, + ledger: &mut recovery::Ledger, +) -> Result<()> { + let id = CommandId::new(ClientId::new([91; 16])?, 1)?; + let command = format!("{}:1", hex::encode([91; 16])); + let retry_until = (std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH)? + .as_millis() + + 60_000) + .to_string(); + let submit = |ingress: Ingress, durability: &str| { + client + .post(format!( + "{}/counters/one?delta=0", + ingress.endpoint(address) + )) + .bearer_auth(SECRET) + .header("x-command-id", &command) + .header("x-retry-until-ms", &retry_until) + .header("x-write-durability", durability) + }; + + let block = root.join(BLOCK); + File::options().write(true).create_new(true).open(&block)?; + let expected = format!("Ambiguous {{ command: {id:?} }}; cause="); + let outage = async { + ledger.record(serde_json::json!({ + "phase": "archive_writes_blocked", "cell": "one", "command": command, + "retry_until": retry_until, "delta": 0, + }))?; + let receipt = submit(Ingress::Native, "quorum") + .send() + .await? + .error_for_status()? + .json::() + .await?; + let committed = receipt["index"].as_u64().context("outage receipt index")?; + ensure!( + receipt["value"] == before["value"] + && receipt["group"] == before["group"] + && receipt["durability"] == "quorum" + && committed > before["index"].as_u64().context("initial receipt index")? + ); + ledger.record(serde_json::json!({ + "phase": "archive_outage_quorum", "ingress": "native", + "command": command, "receipt": receipt, + }))?; + let read = client + .get(format!("{}/counters/one", Ingress::Axum.endpoint(address))) + .bearer_auth(SECRET) + .header("x-read-consistency", "linearizable") + .send() + .await? + .error_for_status()? + .json::() + .await?; + recovery::matching_receipt(&read, &receipt)?; + ledger.record(serde_json::json!({ + "phase": "archive_outage_linearizable", "ingress": "axum", + "command": command, "receipt": read, + }))?; + for ingress in Ingress::ALL { + let response = submit(ingress, "s3").send().await?; + let status = response.status().as_u16(); + let body = response.text().await?; + ledger.record(serde_json::json!({ + "phase": "archive_outage_s3_refused", "ingress": ingress.label(), + "command": command, "status": status, "body": body, + }))?; + ensure!( + status == 503 && body.starts_with(&expected), + "explicit S3 acknowledgement during archive outage: {status} {body}" + ); + } + let denied: Vec<_> = (0..3) + .filter(|&voter| denied_path(root, voter).is_file()) + .map(|voter| voter + 1) + .collect(); + ensure!(!denied.is_empty(), "no voter attempted an archive write"); + ledger.record(serde_json::json!({ + "phase": "archive_outage_denied_writes", "members": denied, + }))?; + Ok::<_, anyhow::Error>(receipt) + } + .await; + std::fs::remove_file(block)?; + let quorum = outage?; + ledger.record(serde_json::json!({"phase": "archive_writes_restored"}))?; + + let deadline = Instant::now() + Duration::from_secs(15); + for ingress in Ingress::ALL { + let receipt = loop { + let remaining = deadline.saturating_duration_since(Instant::now()); + ensure!( + !remaining.is_zero(), + "archive receipt reconciliation deadline expired" + ); + let response = submit(ingress, "s3").timeout(remaining).send().await?; + let status = response.status(); + let body = response.text().await?; + if status.is_success() { + break serde_json::from_str::(&body)?; + } + ledger.record(serde_json::json!({ + "phase": "archive_outage_replay_waiting", "ingress": ingress.label(), + "command": command, "status": status.as_u16(), "body": body, + }))?; + ensure!( + status == 503 && body.starts_with(&expected) && Instant::now() < deadline, + "restored archive did not cover the original receipt: {status} {body}" + ); + tokio::time::sleep(Duration::from_millis(50)).await; + }; + recovery::matching_receipt(&receipt, &quorum)?; + ensure!(receipt["sequence"] == quorum["sequence"] && receipt["durability"] == "s3"); + ledger.record(serde_json::json!({ + "phase": "archive_outage_s3_replayed", "ingress": ingress.label(), + "command": command, "receipt": receipt, + }))?; + } + Ok(()) +} diff --git a/crates/cellar/tests/support/raft_daemon_docker.rs b/crates/cellar/tests/support/raft_daemon_docker.rs new file mode 100644 index 000000000..4fb005e16 --- /dev/null +++ b/crates/cellar/tests/support/raft_daemon_docker.rs @@ -0,0 +1,352 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Constrained Linux daemon voters with disjoint retained disks in the lab VM. +use super::*; + +const RUN: &str = "CELLAR_RAFT_DOCKER_RUN"; +const IMAGE: &str = "CELLAR_RAFT_DOCKER_IMAGE"; +const ENGINE_SOCKET: &str = "/run/docker.sock"; +const LABEL: &str = "io.cellar.daemon.run"; +pub(super) const VOTER: &str = "CELLAR_RAFT_DOCKER_VOTER"; +const INSPECT: &str = r#"{"id":{{json .Id}},"image":{{json .Image}},"run":{{json (index .Config.Labels "io.cellar.daemon.run")}},"running":{{json .State.Running}},"pid":{{json .State.Pid}},"exit_code":{{json .State.ExitCode}},"cpu":{{json .HostConfig.NanoCpus}},"memory":{{json .HostConfig.Memory}},"swap":{{json .HostConfig.MemorySwap}},"restart":{{json .HostConfig.RestartPolicy.Name}},"network":{{json .HostConfig.NetworkMode}},"mounts":{{json .Mounts}}}"#; + +pub(super) fn enabled() -> bool { + std::env::var_os(RUN).is_some() +} + +fn fleet_owner(run: &str) -> String { + format!("daemon-{run}") +} + +pub(super) fn disk(root: &Path, index: usize) -> PathBuf { + if !enabled() { + return root.join(format!("disk-{index}")); + } + // Keep the individual disks outside the shared process rendezvous mount. + let mut name = root + .file_name() + .expect("fixture temporary root") + .to_os_string(); + name.push(format!("-disk-{index}")); + root.with_file_name(name) +} + +fn docker() -> Command { + let mut command = Command::new("docker"); + command + .env_remove("DOCKER_CONTEXT") + .env_remove("DOCKER_HOST") + .args(["--host", "unix:///var/run/docker.sock"]); + command +} + +fn output(command: &mut Command, operation: &str) -> Result { + let result = command + .output() + .with_context(|| format!("Docker {operation}"))?; + ensure!( + result.status.success(), + "Docker {operation} failed: {}", + String::from_utf8_lossy(&result.stderr) + ); + Ok(String::from_utf8(result.stdout)?.trim().to_owned()) +} + +fn lower_hex(value: &str, bytes: usize) -> bool { + value.len() == bytes * 2 + && value + .bytes() + .all(|c| c.is_ascii_digit() || (b'a'..=b'f').contains(&c)) +} + +pub(super) struct Voter { + id: String, + run: String, + image: String, + root: PathBuf, + disk: PathBuf, + mode: String, + repair_worker: bool, +} + +impl Voter { + pub(super) fn create(root: &Path, mode: &str, child: &Command) -> Result { + ensure!( + cfg!(target_os = "linux"), + "Docker voter controller must run inside the Linux lab VM" + ); + let run = std::env::var(RUN)?; + let image = std::env::var(IMAGE)?; + ensure!(lower_hex(&run, 8), "invalid owned Docker run ID"); + ensure!( + image + .strip_prefix("sha256:") + .is_some_and(|id| lower_hex(id, 32)), + "Docker voter requires an immutable image ID" + ); + let (retained, index) = match managed::retained_index(mode) { + Some(index) => (true, index.parse::()?), + None => ( + false, + mode.strip_prefix("cluster-") + .context("managed voter mode")? + .parse()?, + ), + }; + ensure!( + index <= usize::from(managed::standby::INDEX) && root.is_absolute() && root.is_dir(), + "invalid voter inventory" + ); + let repair_enabled = std::env::var_os(managed::repair::ENABLED).is_some(); + let repair_worker = repair_enabled && index == usize::from(managed::repair::WORKER); + let disk = disk(root, index); + if retained { + ensure!(disk.join("raft").is_dir(), "retained voter disk is missing"); + } else { + std::fs::create_dir(&disk)?; + } + let catalog = std::fs::read_to_string(root.join(managed::DOCKER_CATALOG))?; + ensure!(lower_hex(&catalog, 32), "invalid rooted Fleet catalog ID"); + let fleet_node = hex::encode(managed::node(u8::try_from(index)?)?.bytes()); + let owner = output(Command::new("id").arg("-u"), "guest user lookup")?; + let group = output(Command::new("id").arg("-g"), "guest group lookup")?; + let mut command = docker(); + command.args([ + "create", + "--name", + &format!("cellar-daemon-{run}-{mode}"), + "--label", + &format!("{LABEL}={run}"), + "--label", + &format!("io.cellar.fleet.owner={}", fleet_owner(&run)), + "--label", + &format!("io.cellar.fleet.catalog={catalog}"), + "--label", + &format!("io.cellar.fleet.node={fleet_node}"), + "--label", + "io.cellar.fleet.boot-mode=single-use", + "--label", + "io.cellar.fleet.workers=contained", + "--network", + "host", + "--restart", + "no", + "--cpus", + "1", + "--memory", + "3g", + "--memory-swap", + "3g", + "--cap-drop", + "ALL", + "--security-opt", + "no-new-privileges", + "--read-only", + "--tmpfs", + "/tmp:rw,nosuid,nodev,size=134217728", + "--user", + &format!("{owner}:{group}"), + "--log-driver", + "local", + "--log-opt", + "max-size=8m", + "--log-opt", + "max-file=2", + ]); + for path in [root, disk.as_path()] { + let path = path.to_str().context("UTF-8 lab mount")?; + ensure!(!path.contains(','), "lab mount cannot contain a comma"); + command.args(["--mount", &format!("type=bind,src={path},dst={path}")]); + } + for (key, value) in child.get_envs() { + if let Some(value) = value { + command.env(key, value).arg("--env").arg(key); + } + } + command.args([ + "--env", + "CELLAR_RAFT_OPERATOR_CLI=/usr/local/bin/cellar", + "--env", + "CELLAR_RAFT_DOCKER_VOTER=1", + ]); + if repair_enabled { + command.args(["--env", "RUST_LOG=warn,cellar_raft_repair=debug"]); + } + if repair_worker { + let engine_group = output( + Command::new("stat").args(["-c", "%g", ENGINE_SOCKET]), + "guest Docker socket group", + )?; + ensure!( + engine_group.parse::().is_ok(), + "invalid Docker socket group" + ); + command.args([ + "--group-add", + &engine_group, + "--mount", + &format!("type=bind,src={ENGINE_SOCKET},dst={ENGINE_SOCKET},readonly"), + "--env", + "CELLAR_RAFT_DAEMON_REPAIR_GATE=1", + ]); + } + command.arg(&image).args(child.get_args()); + let id = output(&mut command, "create owned voter")?; + ensure!(lower_hex(&id, 32), "Docker voter identity is ambiguous"); + let voter = Self { + id, + run, + image, + root: root.into(), + disk, + mode: mode.into(), + repair_worker, + }; + ensure!( + voter.inspect()?["running"] == false, + "voter was started before qualification" + ); + Ok(voter) + } + + fn inspect(&self) -> Result { + let raw = output( + docker().args([ + "inspect", + "--type", + "container", + "--format", + INSPECT, + &self.id, + ]), + "inspect owned voter", + )?; + let state: Value = serde_json::from_str(&raw)?; + ensure!( + state["id"] == self.id && state["run"] == self.run && state["image"] == self.image, + "foreign voter container" + ); + ensure!( + state["network"] == "host" + && state["restart"] == "no" + && state["cpu"] == 1_000_000_000u64 + && state["memory"] == 3 * 1024u64.pow(3) + && state["swap"] == state["memory"], + "voter cgroup or restart settings differ" + ); + let mounts = state["mounts"].as_array().context("voter mounts absent")?; + ensure!( + mounts.len() == 2 + usize::from(self.repair_worker), + "unexpected voter mount inventory" + ); + if self.repair_worker { + ensure!( + mounts.iter().any(|mount| mount["Type"] == "bind" + && mount["Source"] == ENGINE_SOCKET + && mount["Destination"] == ENGINE_SOCKET + && mount["RW"] == false), + "repair worker Docker socket mount differs" + ); + } + for path in [&self.root, &self.disk] { + let path = path.to_str().context("UTF-8 voter mount")?; + ensure!( + mounts.iter().any(|m| m["Type"] == "bind" + && m["Source"] == path + && m["Destination"] == path + && m["RW"] == true), + "voter storage mount differs" + ); + } + Ok(state) + } + + pub(super) fn start(&self, log: File, diagnostics: File) -> Result { + Ok(docker() + .args(["start", "--attach", &self.id]) + .stdout(log) + .stderr(diagnostics) + .spawn()?) + } + + pub(super) fn fleet_identity(&self) -> Result<(String, cellar::docker::ContainerId)> { + self.inspect()?; + Ok(( + fleet_owner(&self.run), + cellar::docker::ContainerId::new(&self.id)?, + )) + } + + pub(super) fn terminate(&self) -> Result<()> { + ensure!(self.inspect()?["running"] == true, "voter is not running"); + output( + docker().args(["kill", "--signal", "TERM", &self.id]), + "stop waiting standby", + )?; + Ok(()) + } + + pub(super) fn running(&self) -> Result<()> { + let mut state = self.inspect()?; + ensure!(state["running"] == true, "voter exited before readiness"); + let limits = output( + docker().args([ + "exec", + &self.id, + "cat", + "/sys/fs/cgroup/cpu.max", + "/sys/fs/cgroup/memory.max", + "/sys/fs/cgroup/memory.swap.max", + ]), + "read effective voter cgroup limits", + )?; + let values = limits + .split_whitespace() + .map(str::parse::) + .collect::, _>>()?; + let [quota, period, memory, swap] = values.as_slice() else { + anyhow::bail!("invalid voter cgroup limit inventory"); + }; + ensure!( + *quota > 0 && quota == period && *memory == 3 * 1024u64.pow(3) && *swap == 0, + "effective voter cgroup limits differ" + ); + state["cgroup"] = serde_json::json!({"cpu_quota_us":quota, "cpu_period_us":period, "memory_bytes":memory, "swap_bytes":swap}); + std::fs::write( + self.root.join(format!("{}-docker.json", self.mode)), + serde_json::to_vec_pretty(&state)?, + )?; + Ok(()) + } + + pub(super) fn pid(&self) -> Result { + let pid = self.inspect()?["pid"] + .as_u64() + .context("voter guest PID absent")?; + ensure!(pid > 0, "voter process is not running"); + Ok(pid) + } + + pub(super) fn kill_if_running(&self) -> Result<()> { + if self.inspect()?["running"] == true { + output( + docker().args(["kill", "--signal", "KILL", &self.id]), + "kill owned voter", + )?; + } + Ok(()) + } + + pub(super) fn killed(&self) -> Result<()> { + let state = self.inspect()?; + ensure!( + state["running"] == false && state["exit_code"] == 137, + "expected stopped SIGKILL voter" + ); + std::fs::write( + self.root.join(format!("{}-killed-docker.json", self.mode)), + serde_json::to_vec_pretty(&state)?, + )?; + Ok(()) + } +} diff --git a/crates/cellar/tests/support/raft_daemon_maintenance.rs b/crates/cellar/tests/support/raft_daemon_maintenance.rs new file mode 100644 index 000000000..e02c88911 --- /dev/null +++ b/crates/cellar/tests/support/raft_daemon_maintenance.rs @@ -0,0 +1,442 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Public maintenance consent and paused retained startup across daemon processes. +use super::*; + +pub(super) const OPERATION: [u8; 16] = [130; 16]; +const NEXT_PROCESS: [u8; 16] = [135; 16]; +const CAPACITY: &str = "maintenance-return-capacity.json"; +const PAUSED: &str = "maintenance retained voter is paused"; +const HANDOFF: &str = "/fixture/maintenance-handoff"; + +#[derive(serde::Serialize, serde::Deserialize)] +struct Handoff { + member: u64, + term: u64, +} + +pub(super) fn with_handoff(router: axum::Router, group: RaftGroupHandle) -> axum::Router { + router.route( + HANDOFF, + axum::routing::post( + move |headers: axum::http::HeaderMap, axum::Json(request): axum::Json| { + let group = group.clone(); + async move { + use axum::http::{header, StatusCode}; + if headers + .get(header::AUTHORIZATION) + .and_then(|value| value.to_str().ok()) + .and_then(|value| value.strip_prefix("Bearer ")) + != Some(SECRET) + { + return Err(StatusCode::UNAUTHORIZED); + } + let target = + MemberId::new(request.member).map_err(|_| StatusCode::BAD_REQUEST)?; + let start = Instant::now(); + let progress = group + .replication_status(TIMEOUT) + .await + .map_err(|_| StatusCode::SERVICE_UNAVAILABLE)?; + if progress.term.get() != request.term { + return Err(StatusCode::SERVICE_UNAVAILABLE); + } + group + .transfer_leadership( + LeadershipTransfer { + target, + expected_term: progress.term, + expected_revision: progress.membership_revision, + }, + TIMEOUT.saturating_sub(start.elapsed()), + ) + .await + .map_err(|_| StatusCode::SERVICE_UNAVAILABLE)?; + Ok(StatusCode::NO_CONTENT) + } + }, + ), + ) +} + +pub(super) async fn handoff_to( + client: &reqwest::Client, + addresses: &[ManagedIngress], + members: &[u64; 3], + target: usize, + receipt: &Value, +) -> Result { + tokio::time::timeout(TIMEOUT, async { + let mut minimum_term = 1; + loop { + let (leader, term, _) = + recovery::observe_members(client, addresses, members, None, minimum_term, receipt) + .await?; + if leader == target { + return Ok(term); + } + let response = client + .post(format!("{}{HANDOFF}", addresses[leader].axum)) + .bearer_auth(SECRET) + .json(&Handoff { + member: members[target], + term, + }) + .send() + .await?; + if response.status().is_success() { + minimum_term = term + .checked_add(1) + .context("maintenance handoff term overflow")?; + } else { + ensure!( + response.status() == 503, + "maintenance handoff refused: {}", + response.status() + ); + tokio::time::sleep(Duration::from_millis(50)).await; + } + } + }) + .await + .context("returned voter did not become leader")? +} + +pub(super) async fn wait_for_resume(service: &RaftCells, root: &Path, index: u8) -> Result<()> { + let observed = service.observe_node(TIMEOUT).await?; + let boot = observed.fleet_boot.context("returned maintenance boot")?; + ensure!( + boot.node == node(index)?.bytes() + && boot.process == NEXT_PROCESS + && observed.node.groups == 1 + && observed.node.profile.draining + && service.resource_usage().retained_manifest_cells == Some(2), + "retained startup did not keep the exact voting history paused" + ); + let mut profile = observed.node.profile; + profile.draining = false; + let mut attestation: FleetCapacityAttestation = + serde_json::from_slice(&std::fs::read(root.join(format!("capacity-{index}.json")))?)?; + ensure!(attestation.boot.disk == boot.disk && attestation.boot.engine == boot.engine); + attestation.boot = boot; + attestation.profile_revision += 1; + attestation.profile_digest = fleet_capacity_profile_digest(&profile)?; + persist(root, CAPACITY, &serde_json::to_vec(&attestation)?)?; + cellar::note!("{PAUSED}"); + + tokio::time::timeout(Duration::from_secs(30), async { + loop { + if !service.observe_node(TIMEOUT).await?.node.profile.draining { + return Ok::<_, anyhow::Error>(()); + } + cellar::asyncrt::sleep(Duration::from_millis(50)).await; + } + }) + .await + .context("returned voter still paused at startup deadline")??; + Ok(()) +} + +async fn node_status(cli: &OperatorCli, node: &str) -> Result { + Ok(serde_json::from_value( + cli.call(&["nodes", "status", node], OWNER) + .await? + .success()?, + )?) +} + +async fn action( + cli: &OperatorCli, + verb: &str, + node: &str, + request: u8, + options: &[&str], +) -> Result { + let operation = hex::encode(OPERATION); + let request = hex::encode([request; 16]); + let revision = cli.revision().await?.to_string(); + let mut args = vec![ + "nodes", + verb, + node, + "--operation-id", + &operation, + "--request-id", + &request, + "--expected-revision", + &revision, + "--wait", + "--timeout-seconds", + "20", + ]; + args.extend_from_slice(options); + if verb == "prepare-maintenance" { + let refused = cli.call(&args, cli::READER).await?; + ensure!(refused.code == 4 && refused.value["error"]["code"] == "forbidden"); + } + let value = cli + .call_with_timeout(&args, OWNER, Duration::from_secs(22)) + .await? + .success() + .with_context(|| format!("public {verb}"))?; + let status: RaftFleetMaintenanceStatus = serde_json::from_value(value)?; + ensure!(status.node_id == node && status.operation_id == operation); + Ok(status) +} + +pub(crate) async fn qualify( + root: &Path, + client: &reqwest::Client, + processes: &mut [Process], + addresses: &mut [ManagedIngress], + original: &recovery::Acknowledged, + ledger: &mut recovery::Ledger, +) -> Result<()> { + let operator = 0; + let (leader, term, _) = + recovery::observe_leader(client, addresses, None, 1, &original.receipt).await?; + let domain = Arc::new(ExecutionDomain::new(tokio::runtime::Handle::current())); + let catalog = Operator::open(domain.clone(), root) + .and_then(|operator| Ok(hex::encode(operator.fleet.catalog_id()?))); + domain.shutdown().await; + let catalog = catalog?; + let cli = OperatorCli::connect(addresses[operator].axum.clone(), catalog); + // The hard-killed voter can be serving again without capacity for its new + // boot. Choose a qualified target while keeping the controller live. + let targets = if leader == 2 { [2, 1] } else { [1, 2] }; + let mut candidate = None; + for target in targets { + let id = hex::encode(node(u8::try_from(target)?)?.bytes()); + let before = node_status(&cli, &id).await?; + if let Some(capacity) = before.capacity { + candidate = Some((target, id, before.descriptor.storage_instance, capacity)); + break; + } + } + let (target, node, retained_engine, original_capacity) = + candidate.context("maintenance needs an independently qualified target")?; + let process = hex::encode(original_capacity.boot.process); + let ready = action( + &cli, + "prepare-maintenance", + &node, + 131, + &[ + "--expected-process", + &process, + "--grace-seconds", + "120", + "--birth-resolution", + "cancel", + ], + ) + .await?; + ensure!(ready.phase == RaftFleetMaintenancePhase::Ready && ready.readiness_request.is_some()); + let old_pid = processes[target].pid()?; + ledger.record(serde_json::json!({ + "phase":"maintenance_ready", "operation":ready.operation_id, "member":target + 1, + "leader":leader + 1, "term":term, "pid":old_pid, + "engine":hex::encode(original_capacity.boot.engine), + }))?; + let offline = action( + &cli, + "stop-maintenance", + &node, + 132, + &[ + "--expected-process", + &process, + "--stop-id", + &hex::encode([133; 16]), + ], + ) + .await?; + // The worker can record the offline proof before advancing the phase. + ensure!( + offline.offline.is_some(), + "stop wait returned without durable offline proof" + ); + let logs = processes[target].exited(true).await?; + ensure!( + logs.contains("\"owners_closed\":true"), + "maintenance stop lacked joined ownership\n{logs}" + ); + let stopped = node_status(&cli, &node).await?; + ensure!(stopped.capacity.is_none() && stopped.descriptor.storage_instance == retained_engine); + let (successor, successor_term, _) = + recovery::observe_leader(client, addresses, Some(target), term, &original.receipt).await?; + ensure!(successor != target && (leader != target || successor_term > term)); + ledger.record(serde_json::json!({ + "phase":"maintenance_offline", "operation":offline.operation_id, "member":target + 1, + "pid":old_pid, "successor":successor + 1, "term":successor_term, + "observed_phase":offline.phase, + }))?; + let command = format!("{}:1", hex::encode([137; 16])); + let retry = (std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH)? + .as_millis() + + 120_000) + .to_string(); + let written = verify_sql( + client, + &addresses[successor], + Ingress::Native, + &command, + &retry, + "quorum", + ) + .await?; + ledger.record(serde_json::json!({ + "phase":"maintenance_offline_sql", "member":successor + 1, + "ingress":"native", "command":command, "receipt":written, + }))?; + + let returning = action( + &cli, + "return-maintenance", + &node, + 134, + &["--next-process", &hex::encode(NEXT_PROCESS)], + ) + .await?; + ensure!( + returning.returning.context("return consent")?.next_process == hex::encode(NEXT_PROCESS) + ); + let disk = docker::disk(root, target); + processes[target] = Process::spawn( + root, + &format!("cluster-maintenance-{target}"), + &[std::ffi::OsStr::new("--data-dir"), disk.as_os_str()], + false, + )?; + let deadline = Instant::now() + Duration::from_secs(20); + loop { + let logs = processes[target].logs()?; + ensure!( + processes[target].child.try_wait()?.is_none() && Instant::now() < deadline, + "retained return did not start paused\n{logs}" + ); + ensure!( + !logs.contains("cellar listening on ") && !logs.contains("cellar axum listening on "), + "paused voter opened public ingress\n{logs}" + ); + if logs.contains(PAUSED) { + break; + } + tokio::time::sleep(Duration::from_millis(25)).await; + } + let path = root.join(CAPACITY); + let attestation: FleetCapacityAttestation = serde_json::from_slice(&std::fs::read(&path)?)?; + ensure!( + attestation.boot.process == NEXT_PROCESS + && attestation.boot.engine == original_capacity.boot.engine + ); + ensure!(node_status(&cli, &node).await?.capacity.is_none()); + ledger.record(serde_json::json!({ + "phase":"maintenance_return_paused", "operation":returning.operation_id, + "member":target + 1, "engine":hex::encode(attestation.boot.engine), + }))?; + let complete = action( + &cli, + "resume-maintenance", + &node, + 136, + &[ + "--attestation", + path.to_str().context("resume attestation path")?, + "--resume-id", + &hex::encode([138; 16]), + ], + ) + .await?; + ensure!( + complete.phase == RaftFleetMaintenancePhase::Complete + && complete.completed_revision.is_some() + ); + addresses[target] = processes[target].managed_addresses().await?; + let new_pid = processes[target].pid()?; + let after = node_status(&cli, &node).await?; + ensure!( + new_pid != old_pid + && after.state == FleetNodeState::Active + && after.capacity == Some(attestation) + ); + ledger.record(serde_json::json!({ + "phase":"maintenance_complete", "operation":complete.operation_id, + "member":target + 1, "pid":new_pid, "revision":complete.completed_revision, + }))?; + // Membership does not grant namespace forwarding. Exercise the returned + // process as leader, using a term- and revision-guarded transfer. + let resumed_term = handoff_to(client, addresses, &[1, 2, 3], target, &written).await?; + ledger.record(serde_json::json!({ + "phase":"maintenance_returned_leader", "member":target + 1, "term":resumed_term, + }))?; + for durability in ["quorum", "s3"] { + for ingress in Ingress::ALL { + let reply = verify_sql( + client, + &addresses[target], + ingress, + &command, + &retry, + durability, + ) + .await?; + recovery::matching_receipt(&reply, &written)?; + ledger.record(serde_json::json!({ + "phase":"maintenance_resumed_sql", "member":target + 1, + "ingress":ingress.label(), "command":command, "receipt":reply, + }))?; + } + } + Ok(()) +} + +pub(super) async fn verify_sql( + client: &reqwest::Client, + address: &ManagedIngress, + ingress: Ingress, + command: &str, + retry: &str, + durability: &str, +) -> Result { + let public = ingress.endpoint(address); + let response = client + .post(format!("{public}/counters/one?delta=0")) + .bearer_auth(SECRET) + .header("x-command-id", command) + .header("x-retry-until-ms", retry) + .header("x-write-durability", durability) + .send() + .await?; + let status = response.status(); + let body = response.text().await?; + ensure!( + status.is_success(), + "maintenance {} {durability} receipt refused {status}: {body}", + ingress.label() + ); + let receipt: Value = serde_json::from_str(&body)?; + let read: Value = client + .get(format!("{public}/counters/one")) + .bearer_auth(SECRET) + .header("x-read-consistency", "linearizable") + .send() + .await? + .error_for_status()? + .json() + .await?; + ensure!( + receipt["value"] == 23 + && receipt["durability"] == durability + && read["value"] == 23 + && read["group"] == receipt["group"] + && read["incarnation"] == receipt["incarnation"] + && read["index"].as_u64().context("maintenance read index")? + >= receipt["index"] + .as_u64() + .context("maintenance receipt index")?, + "maintenance changed SQL history: {receipt} / {read}" + ); + Ok(receipt) +} diff --git a/crates/cellar/tests/support/raft_daemon_managed.rs b/crates/cellar/tests/support/raft_daemon_managed.rs new file mode 100644 index 000000000..2d24a51bd --- /dev/null +++ b/crates/cellar/tests/support/raft_daemon_managed.rs @@ -0,0 +1,996 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Cold RF3 startup on the daemon's public SDK boundary with fixture capacity. +use super::*; +use cellar::application::consensus::verification::cell_groups::{ + CellGroupArchiveTailLimits, RangeDirectory, +}; +use cellar::bucket::{Bucket, StaticCredentials}; +use cellar::sdk::fleet_operations::*; +use std::io::Write; +#[path = "raft_first_group_operator_cli.rs"] +mod cli; +use cli::{first_group_plan, OperatorCli, OWNER}; +#[path = "raft_daemon_maintenance.rs"] +pub(super) mod maintenance; +#[path = "raft_daemon_repair.rs"] +pub(super) mod repair; +#[path = "raft_daemon_standby.rs"] +pub(super) mod standby; + +const TIMEOUT: Duration = Duration::from_secs(5); +const PRINCIPAL: &str = "daemon-process-fixture"; + +pub(super) const RUSTFS_ENDPOINT: &str = "CELLAR_CELL_GROUPS_RUSTFS_ENDPOINT"; +pub(super) const RUSTFS_ENV: [&str; 6] = [ + RUSTFS_ENDPOINT, + "CELLAR_CELL_GROUPS_RUSTFS_BUCKET", + "CELLAR_CELL_GROUPS_RUSTFS_PREFIX", + "CELLAR_RAFT_PROCESS_IMAGE_MODE", + "AWS_ACCESS_KEY_ID", + "AWS_SECRET_ACCESS_KEY", +]; + +fn open_bucket(root: &Path) -> Result { + if std::env::var_os(RUSTFS_ENDPOINT).is_none() { + return Bucket::open_dev(&root.join("catalog.sqlite")); + } + let endpoint = std::env::var(RUSTFS_ENDPOINT)?; + ensure!( + endpoint.starts_with("http://127.0.0.1:"), + "expected a loopback RustFS endpoint" + ); + let bucket = std::env::var("CELLAR_CELL_GROUPS_RUSTFS_BUCKET")?; + let prefix = std::env::var("CELLAR_CELL_GROUPS_RUSTFS_PREFIX")?; + ensure!( + !bucket.is_empty() && !prefix.is_empty(), + "isolated RustFS bucket and prefix required" + ); + Bucket::open( + &format!("s3://{bucket}/{prefix}"), + Some(&endpoint), + "us-east-1", + Some(StaticCredentials { + access_key_id: std::env::var("AWS_ACCESS_KEY_ID")?, + secret_access_key: std::env::var("AWS_SECRET_ACCESS_KEY")?, + session_token: None, + }), + None, + ) +} + +fn request(id: u8, revision: u64) -> FleetRequestIdentity { + FleetRequestIdentity { + id: [id; 16], + operation: [70; 16], + principal: PRINCIPAL.into(), + expected_revision: revision, + } +} + +pub(super) fn node(index: u8) -> Result { + Ok(NodeId::new([index + 3; 16])?) +} + +pub(super) const DOCKER_CATALOG: &str = "managed-fleet-catalog"; + +fn node_spec(index: u8, capability_digest: [u8; 32]) -> Result { + Ok(FleetNodeSpec { + node: node(index)?, + disk: DiskLineage::new([index + 4; 16])?, + physical_host: IncarnationId::new([if docker::enabled() { 90 } else { index + 90 }; 16])?, + label: format!("raft-cli-cluster-{index}"), + server_name: "localhost".into(), + capability_digest, + storage_instance: None, + }) +} + +fn history() -> Result { + configuration::group(ClusterId::new([1; 16])?, NamespaceId::new([2; 16])?) +} + +struct Operator { + domain: Arc, + bucket: Bucket, + fleet: FleetCatalog, + adapter: Catalog, + namespace: NamespaceConfig, +} + +impl Operator { + fn open(domain: Arc, root: &Path) -> Result { + let history = history()?; + let bucket = open_bucket(root)?; + let fleet = FleetCatalog::new( + &domain, + bucket.clone(), + history.cluster, + IncarnationId::new([81; 16])?, + "daemon-control/", + )?; + let adapter = Catalog::new(bucket.raft_store(), catalog_options())?; + let mut namespace = NamespaceConfig::new( + history.cluster, + history.namespace, + NamespaceMode::Raft, + counter::definition()?.schema_digest(), + ); + if std::env::var_os(RUSTFS_ENDPOINT).is_some() { + namespace.image_storage = + match std::env::var("CELLAR_RAFT_PROCESS_IMAGE_MODE")?.as_str() { + "materialized" => RaftImageStorage::Materialized, + "extents" => RaftImageStorage::Extents, + _ => anyhow::bail!("invalid RustFS process image mode"), + }; + } + Ok(Self { + domain, + bucket, + fleet, + adapter, + namespace, + }) + } + + fn declaration(&self) -> Result { + Ok(RaftNamespaceRegistration { + namespace: self.fleet.bind_namespace(self.namespace.clone())?, + class: "Counter".into(), + }) + } +} + +fn catalog_options() -> CatalogOptions { + CatalogOptions { + prefix: "daemon-managed/".into(), + ..Default::default() + } +} + +async fn provision_call(root: &Path, plan: &Value) -> Result { + let path = root.join("sql-namespace.json"); + std::fs::write(&path, serde_json::to_vec(plan)?)?; + let bucket = std::env::var("CELLAR_CELL_GROUPS_RUSTFS_BUCKET")?; + let prefix = std::env::var("CELLAR_CELL_GROUPS_RUSTFS_PREFIX")?; + let mut command = tokio::process::Command::new(std::env::current_exe()?); + command + .env_clear() + .env("CELLAR_RAFT_CLI_CHILD", "provision-sql") + .env("CELLAR_RAFT_CLI_ROOT", root) + .arg("--provision-sql-namespace") + .arg(path) + .arg("--bucket") + .arg(format!("s3://{bucket}/{prefix}")) + .arg("--endpoint") + .arg(std::env::var(RUSTFS_ENDPOINT)?) + .arg("--region") + .arg("us-east-1") + .kill_on_drop(true); + for name in ["AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY"] { + command.env( + name, + std::env::var_os(name).with_context(|| format!("missing {name}"))?, + ); + } + Ok(tokio::time::timeout(Duration::from_secs(30), command.output()).await??) +} + +async fn provision_from_compiled_cli( + root: &Path, + operator: &Operator, + identity: &FleetRequestIdentity, +) -> Result { + let id = hex::encode(operator.fleet.catalog_id()?); + let namespace = hex::encode(operator.namespace.namespace.bytes()); + let mut plan = serde_json::json!({ + "catalog_id": id, + "cluster_id": hex::encode(operator.namespace.cluster.bytes()), + "incarnation_id": hex::encode([81; 16]), + "coordination_prefix": "daemon-control/", + "catalog_prefix": "daemon-managed/", + "namespace_id": namespace, + "script": "raft-cli", + "version": "test", + "class": "Counter", + "principal": identity.principal, + "request_id": hex::encode(identity.id), + "operation_id": hex::encode(identity.operation), + "expected_revision": identity.expected_revision, + }); + let image_storage = match operator.namespace.image_storage { + RaftImageStorage::Materialized => "materialized", + RaftImageStorage::Extents => { + plan["image_storage"] = "extents".into(); + "extents" + } + }; + for (field, value) in [ + ("catalog_id", hex::encode([77; 32])), + ("class", "Absent".into()), + ("image_storage", "unknown".into()), + ] { + let mut wrong = plan.clone(); + wrong[field] = value.into(); + let refused = provision_call(root, &wrong).await?; + ensure!( + !refused.status.success(), + "invalid SQL namespace plan was admitted" + ); + ensure!(operator.fleet.read(TIMEOUT).await?.revision() == identity.expected_revision); + } + let admitted = provision_call(root, &plan).await?; + ensure!( + admitted.status.success(), + "compiled SQL namespace command failed" + ); + let receipt: Value = serde_json::from_slice(&admitted.stdout)?; + ensure!( + receipt["catalog_id"] == id + && receipt["namespace_id"] == namespace + && receipt["class"] == "Counter" + && receipt["image_storage"] == image_storage + && receipt["schema_digest"] == hex::encode(operator.namespace.schema_digest) + && receipt["request_id"] == plan["request_id"] + && receipt["operation_id"] == plan["operation_id"] + ); + ensure!(operator.fleet.read(TIMEOUT).await?.revision() == identity.expected_revision + 1); + let replay = provision_call(root, &plan).await?; + ensure!(replay.status.success() && replay.stdout == admitted.stdout); + let mut changed = plan; + changed["image_storage"] = match operator.namespace.image_storage { + RaftImageStorage::Materialized => "extents", + RaftImageStorage::Extents => "materialized", + } + .into(); + ensure!(!provision_call(root, &changed).await?.status.success()); + ensure!(operator.fleet.read(TIMEOUT).await?.revision() == identity.expected_revision + 1); + operator.declaration() +} + +pub(super) async fn initialize(root: &Path) -> Result<()> { + let domain = Arc::new(ExecutionDomain::new(tokio::runtime::Handle::current())); + let result = domain + .run(async { + let operator = Operator::open(domain.clone(), root)?; + let snapshot = if docker::enabled() { + operator + .fleet + .initialize_with_replica_separation(FleetReplicaSeparation::Nodes, TIMEOUT) + .await? + } else { + operator.fleet.initialize(TIMEOUT).await? + }; + if docker::enabled() { + persist( + root, + DOCKER_CATALOG, + hex::encode(operator.fleet.catalog_id()?).as_bytes(), + )?; + } + let identity = request(10, snapshot.revision()); + let declaration = if std::env::var_os(RUSTFS_ENDPOINT).is_some() { + provision_from_compiled_cli(root, &operator, &identity).await? + } else { + let class = counter::definition()?; + operator + .fleet + .provision_managed_sql_namespace( + &identity, + &operator.adapter, + operator.namespace.namespace, + &class, + configuration::limits().directory, + TIMEOUT, + ) + .await? + }; + let capability_digest = RaftManagedHostOptions::capability_digest(&[declaration])?; + for index in 0..3 { + let snapshot = operator.fleet.read(TIMEOUT).await?; + operator + .fleet + .register_node( + &snapshot, + &request(11 + index, snapshot.revision()), + &node_spec(index, capability_digest)?, + TIMEOUT, + ) + .await?; + } + Ok(()) + }) + .await; + domain.shutdown().await; + result? +} + +async fn open_archive_verifier( + operator: &Operator, + root: &Path, +) -> Result { + let limits = configuration::limits(); + let checkpoints = limits.checkpoints.context("repair replay checkpoints")?; + let tails = CellGroupArchiveTailLimits::default(); + archive_verifier::open( + &operator.domain, + root, + operator.namespace.image_storage == RaftImageStorage::Extents, + tails, + limits, + checkpoints, + ) + .await +} + +pub(super) fn retained_index(mode: &str) -> Option<&str> { + mode.strip_prefix("cluster-resume-") + .or_else(|| mode.strip_prefix("cluster-maintenance-")) +} + +pub(super) fn child(root: PathBuf, mode: &str) -> Result<()> { + let returning = mode.starts_with("cluster-maintenance-"); + let (retained, index) = match retained_index(mode) { + Some(index) => (true, index.parse::()?), + None => ( + false, + mode.strip_prefix("cluster-") + .context("invalid managed daemon mode")? + .parse::()?, + ), + }; + ensure!(index <= standby::INDEX, "invalid daemon fixture member"); + let spare = index == standby::INDEX; + ensure!(!spare || !retained, "standby boot must use a new disk"); + let history = history()?; + let api = counter::CounterApi::new( + SECRET.into(), + history.cluster, + history.namespace, + &counter::definition()?, + )?; + let serving = api.clone(); + // The test operator stays owned for the whole process, including shutdown + // and all catalog work entered by the daemon. + let runtime = tokio::runtime::Builder::new_multi_thread() + .worker_threads(1) + .enable_all() + .build()?; + let domain = Arc::new(ExecutionDomain::new(runtime.handle().clone())); + let operator = Arc::new(Operator::open(domain.clone(), &root)?); + let boot_operator = operator.clone(); + let axum_runtime = runtime.handle().clone(); + let (axum_stop, mut axum_stopping) = tokio::sync::watch::channel(false); + let (axum_started, axum_task) = tokio::sync::oneshot::channel(); + let host = Host::local(graph::graph("raft-cli", "", vec![])).register( + "raft-cli", + "test", + move || counter::application(serving.clone(), counter::definition()?), + )?; + let store = Arc::new(archive::Gate::new( + operator.bucket.raft_store(), + &root, + index, + )); + let prepare = move |node| boot_operator.prepare(node, index, retained, returning, store); + let fixture_operator = operator.clone(); + let (maintenance_started, maintenance_task) = tokio::sync::oneshot::channel(); + let (repair_started, repair_task) = tokio::sync::oneshot::channel(); + let repair_root = root.clone(); + let repair_stop = axum_stopping.clone(); + let return_root = root.clone(); + let admin_root = root.join(format!("park-admin-{index}")); + let bind = move |service: RaftCells, discovered: GroupHistory| async move { + ensure!(discovered == history, "unexpected managed namespace group"); + // A repair standby has already proved promotion in the current quorum. + // The original voters and catalog cut apply only to first-group birth. + if !retained && !spare { + let cut = operator.fleet.read(TIMEOUT).await?; + for index in 0..3 { + inspect_current(&service, node(index)?, cut.revision()).await?; + } + } + ready_counter(&service, history, MemberId::new(u64::from(index) + 1)?).await?; + let group = service.group(history)?; + let declaration = operator.declaration()?; + let selected = operator + .domain + .run(operator.adapter.existing_namespace(declaration.namespace)) + .await??; + let directory = RangeDirectory::new(selected, configuration::limits().directory)?; + if index == repair::WORKER && std::env::var_os(repair::ENABLED).is_some() { + let task = axum_runtime.spawn(Box::pin(repair::run_worker( + operator.clone(), + service.clone(), + repair_root, + directory.clone(), + repair_stop, + ))); + repair_started.send(task).map_err(|task| { + task.abort(); + anyhow::anyhow!("repair worker join owner dropped during startup") + })?; + } + let control = + RaftFleetControl::new(&operator.domain, operator.fleet.clone(), vec![directory])? + .with_node_observer(service.clone()); + std::fs::create_dir_all(&admin_root)?; + if index == 0 { + let worker = control + .start_maintenance_service(None, Default::default()) + .await?; + maintenance_started + .send(worker) + .map_err(|_| anyhow::anyhow!("maintenance join owner dropped"))?; + } + let admin = OperatorCli::router( + &operator.domain, + operator.bucket.clone(), + control, + &admin_root, + PRINCIPAL, + )?; + *api.cells + .lock() + .map_err(|_| anyhow::anyhow!("binding poisoned"))? = Some(service); + let listener = std::net::TcpListener::bind("127.0.0.1:0")?; + listener.set_nonblocking(true)?; + let address = listener.local_addr()?; + let router = recovery::observe_voter(counter::router(api), group.clone()).merge(admin); + let router = maintenance::with_handoff(router, group); + let task = axum_runtime.spawn(async move { + let listener = tokio::net::TcpListener::from_std(listener)?; + cellar::note!("cellar axum listening on {address}"); + axum::serve(listener, router) + .with_graceful_shutdown(async move { + let _ = axum_stopping.changed().await; + }) + .await + }); + axum_started.send(task).map_err(|task| { + task.abort(); + anyhow::anyhow!("Axum join owner dropped during startup") + })?; + Ok(()) + }; + let host = if spare { + host.raft_cells_with_startup("raft-cli", prepare, move |service| async move { + standby::wait_for_attachment(&service, &return_root).await?; + bind(service, history).await + })? + } else if returning { + host.raft_cells_with_startup("raft-cli", prepare, move |service| async move { + maintenance::wait_for_resume(&service, &return_root, index).await?; + service.start_managed_first_group(history, TIMEOUT).await?; + bind(service, history).await + })? + } else if !retained && index == 0 { + // The fixture's first process also runs the independent operator. + host.raft_cells_with_startup("raft-cli", prepare, move |service| async move { + ensure!(service.observe_node(TIMEOUT).await?.node.groups == 0); + provision_first_group(&fixture_operator, &service, &root, history).await?; + let discovered = service + .start_managed_namespace_first_group(history.cluster, history.namespace, TIMEOUT) + .await?; + bind(service, discovered).await + })? + } else { + host.managed_sql_namespace( + "raft-cli", + (history.cluster, history.namespace), + if retained { + TIMEOUT + } else { + Duration::from_secs(20) + }, + prepare, + bind, + )? + }; + let result = host.run_until(std::future::pending()); + let _ = axum_stop.send(true); + let axum_result = runtime.block_on(async { + let joined = match axum_task.await { + Ok(task) => task + .await + .map_err(anyhow::Error::from) + .and_then(|result| result.map_err(Into::into)), + Err(_) => Ok(()), + }; + if let Ok(worker) = maintenance_task.await { + worker.shutdown().await; + } + let repaired = match repair_task.await { + Ok(worker) => worker + .await + .map_err(anyhow::Error::from) + .and_then(|result| result), + Err(_) => Ok(()), + }; + domain.shutdown().await; + joined.and(repaired) + }); + result.and(axum_result) +} + +impl Operator { + async fn prepare( + self: Arc, + native: NativeRaftNode, + index: u8, + retained: bool, + returning: bool, + store: Arc, + ) -> Result { + let declaration = self.declaration()?; + let id = node(index)?; + let disk = DiskLineage::new([index + 4; 16])?; + let snapshot = self.fleet.read(TIMEOUT).await?; + let authorization = self.fleet.authorize_host(&snapshot, id, TIMEOUT).await?; + let listeners = NodeRaftListeners { + control: tokio::net::TcpListener::bind("127.0.0.1:0").await?, + data: tokio::net::TcpListener::bind("127.0.0.1:0").await?, + query: tokio::net::TcpListener::bind("127.0.0.1:0").await?, + }; + let mut process = [0; 16]; + if std::env::var_os(docker::VOTER).is_some() { + getrandom::fill(&mut process) + .map_err(|error| anyhow::anyhow!("process generation: {error}"))?; + } else { + process[..4].copy_from_slice(&std::process::id().to_be_bytes()); + } + let process = if returning { + let consent = self + .fleet + .restart_operator_return_request( + &snapshot, + ProposalId::new(maintenance::OPERATION)?, + TIMEOUT, + ) + .await? + .context("rooted daemon maintenance return")?; + ensure!( + consent.node == id, + "maintenance consent belongs to another node" + ); + consent.next_process + } else { + ProcessGeneration::new(process)? + }; + let peers = vec![RaftPeerNode { + node: id, + disk, + process, + label: native.node, + server_name: "localhost".into(), + control_address: listeners.control.local_addr()?, + data_address: listeners.data.local_addr()?, + query_address: listeners.query.local_addr()?, + }]; + let shared = configuration::limits(); + let mut log = ReplicaLogOptions::new( + native.data_dir.join("raft"), + id, + disk, + process, + if retained { + LogOpenMode::Resume + } else { + LogOpenMode::Create + }, + ); + log.max_groups = shared.controller.max_groups; + let mut host = RaftHostOptions { + max_groups: shared.controller.max_groups, + max_input_bytes: shared.dispatch.max_input_bytes, + node_directory: Some(RaftNodeDirectoryOptions::default()), + managed: Some(RaftManagedHostOptions { + authorization, + namespaces: vec![declaration.clone()], + }), + ..Default::default() + }; + if declaration.namespace.image_storage == RaftImageStorage::Extents { + host.extents = Some(Default::default()); + } + let byte_capacity = host.images.max_bytes + + if index == standby::INDEX { + repair::STANDBY_DURABLE_HEADROOM + } else { + 0 + }; + host.node_profile = Some(RaftNodeProfile { + pool: "daemon-fixture".into(), + region: "local".into(), + zone: format!("process-{index}"), + group_capacity: u32::try_from(host.max_groups)?, + byte_capacity, + bytes_per_group: host.images.max_bytes, + leader_capacity: u32::try_from(host.max_groups)?, + draining: false, + }); + Ok(NativeRaftCells { + config: RaftCellsConfig { + infrastructure: NodeRaftConfig { + log, + consensus: RaftRuntimeOptions { + tick_interval_ms: Some(25), + max_groups: shared.controller.max_groups, + ..Default::default() + }, + transport: TransportOptions::default(), + }, + catalog: catalog_options(), + namespaces: vec![RaftNamespaceConfig { + namespace: declaration.namespace, + class: declaration.class, + initialize: None, + }], + groups: vec![], + peers, + query_origins: vec![], + command_origins: vec![], + host, + shared, + }, + listeners, + tls: fixture_tls()?, + store, + versions: fixture_versions()?, + }) + } +} + +pub(super) async fn verify_park_wake( + root: &Path, + client: &reqwest::Client, + addresses: &[ManagedIngress], + receipt: &Value, + ledger: &mut recovery::Ledger, +) -> Result<()> { + let (leader, term, _) = recovery::observe_leader(client, addresses, None, 1, receipt).await?; + let history = history()?; + let domain = Arc::new(ExecutionDomain::new(tokio::runtime::Handle::current())); + let operator = Operator::open(domain.clone(), root)?; + let catalog = hex::encode(operator.fleet.catalog_id()?); + drop(operator); + domain.shutdown().await; + let cli = OperatorCli::connect(addresses[leader].axum.clone(), catalog.clone()); + let namespace = hex::encode(history.namespace.bytes()); + let request = RaftFleetParkRequest { + catalog_id: catalog, + namespace_id: namespace.clone(), + group: RaftFleetTransitionGroup { + group_id: hex::encode(history.group.bytes()), + incarnation_id: hex::encode(history.incarnation.bytes()), + }, + operation_id: hex::encode([119; 16]), + request_id: hex::encode([120; 16]), + settlement_request_id: hex::encode([121; 16]), + expected_revision: cli.revision().await?, + }; + let plan = root.join("managed-park.json"); + persist(root, "managed-park.json", &serde_json::to_vec(&request)?)?; + let plan = plan.to_str().context("park plan path")?; + let send = async |verb: &str| { + cli.call_with_timeout( + &["group", verb, &namespace, "--plan", plan], + OWNER, + Duration::from_secs(32), + ) + .await? + .success() + }; + let refused = cli + .call(&["group", "park", &namespace, "--plan", plan], cli::READER) + .await?; + ensure!(refused.code == 4, "reader admitted park: {}", refused.value); + ensure!( + send("park-status").await?.is_null(), + "park exists before admission" + ); + + let parked: RaftFleetParkStatus = serde_json::from_value(send("park").await?)?; + ensure!( + parked.phase == RaftFleetParkPhase::Delivered, + "park delivery did not settle" + ); + let observed: RaftFleetParkStatus = serde_json::from_value(send("park-status").await?)?; + let replayed: RaftFleetParkStatus = serde_json::from_value(send("park").await?)?; + ensure!( + observed == parked && replayed == parked, + "park replay changed the rooted decision" + ); + for address in addresses { + let status: Value = client + .get(format!("{}{}", address.axum, recovery::OBSERVATION)) + .bearer_auth(SECRET) + .send() + .await? + .error_for_status()? + .json() + .await?; + ensure!( + status["ready"] == false && status["term"] == term, + "status or replay woke a parked voter: {status}" + ); + } + ledger.record(serde_json::json!({ + "phase": "managed_park_delivered", "term": term, + "operation": parked.operation_id, "generation": parked.generation, + "members": addresses.len(), + }))?; + let wake: RaftFleetWakeStatus = serde_json::from_value(send("wake").await?)?; + ensure!( + wake.park == parked && wake.applied_term > term && wake.applied_index > parked.generation, + "wake lacks a fresh applied quorum barrier: {wake:?}" + ); + ledger.record(serde_json::json!({ + "phase": "managed_park_wake", "operation": wake.park.operation_id, + "generation": wake.park.generation, "term": wake.applied_term, "index": wake.applied_index, + }))?; + for ingress in Ingress::ALL { + let address = ingress.endpoint(&addresses[leader]); + let read: Value = client + .get(format!("{address}/counters/one")) + .bearer_auth(SECRET) + .header("x-read-consistency", "linearizable") + .send() + .await? + .error_for_status()? + .json() + .await?; + ensure!( + read["value"] == receipt["value"] + && read["group"] == receipt["group"] + && read["group_incarnation"] == receipt["group_incarnation"] + && read["incarnation"] == receipt["incarnation"], + "wake read changed cell state or history: {read}" + ); + ledger.record(serde_json::json!({ + "phase": "managed_park_read", "operation": wake.park.operation_id, + "ingress": ingress.label(), "receipt": read, + }))?; + } + Ok(()) +} + +pub(super) async fn wait_for_cold_boot( + root: &Path, + index: u8, + process: &mut Process, +) -> Result<()> { + let domain = Arc::new(ExecutionDomain::new(tokio::runtime::Handle::current())); + let result = domain + .run(async { + let operator = Operator::open(domain.clone(), root)?; + let node = node(index)?; + let deadline = Instant::now() + Duration::from_secs(15); + loop { + let remaining = deadline + .saturating_duration_since(Instant::now()) + .min(TIMEOUT); + ensure!( + process.child.try_wait()?.is_none() && !remaining.is_zero(), + "managed voter {index} did not root its cold boot\n{}", + process.logs()? + ); + let cut = operator.fleet.read(remaining).await?; + let remaining = deadline + .saturating_duration_since(Instant::now()) + .min(TIMEOUT); + let boot = operator.fleet.authorize_host(&cut, node, remaining).await?; + if boot.storage_instance().is_some() { + return Ok(()); + } + cellar::asyncrt::sleep(Duration::from_millis(25)).await; + } + }) + .await; + domain.shutdown().await; + result? +} + +fn persist(root: &Path, name: &str, bytes: &[u8]) -> Result<()> { + let mut file = File::create_new(root.join(name))?; + file.write_all(bytes)?; + file.sync_all()?; + File::open(root)?.sync_all()?; + Ok(()) +} + +async fn provision_first_group( + operator: &Operator, + service: &RaftCells, + root: &Path, + history: GroupHistory, +) -> Result<()> { + let declaration = operator.declaration()?; + let limits = configuration::limits().directory; + let selected = operator + .domain + .run( + operator + .adapter + .existing_namespace(declaration.namespace.clone()), + ) + .await??; + let directory = RangeDirectory::new(selected, limits)?; + let cut = operator.fleet.read(TIMEOUT).await?; + cellar::asyncrt::timeout(Duration::from_secs(15), async { + for index in 1..3 { + let observed = inspect_current(service, node(index)?, cut.revision()).await?; + ensure!( + observed.node.groups == 0, + "standby already attached a group" + ); + } + Ok::<_, anyhow::Error>(()) + }) + .await??; + let control = RaftFleetControl::new(&operator.domain, operator.fleet.clone(), vec![directory])? + .with_node_observer(service.clone()); + let cli = OperatorCli::start( + &operator.domain, + operator.bucket.clone(), + control, + root, + PRINCIPAL, + ) + .await?; + let result = provision_group_metadata(&cli, service, root, history).await; + cli.close().await?; + result +} + +async fn provision_group_metadata( + cli: &OperatorCli, + service: &RaftCells, + root: &Path, + history: GroupHistory, +) -> Result<()> { + for index in 0..3 { + admit_capacity(cli, service, root, index).await?; + } + let namespace = hex::encode(history.namespace.bytes()); + let observed: RaftFleetFirstGroupStandby = serde_json::from_value( + cli.call(&["first-group", "standby", &namespace], OWNER) + .await? + .success()?, + )?; + let mut standby = [0; 32]; + hex::decode_to_slice(observed.standby, &mut standby)?; + let plan = first_group_plan( + &cli.catalog, + RaftManagedFirstGroupRequest { + reservation: request(30, cli.revision().await?), + standby, + publication: ProposalId::new([32; 16])?, + group: CellGroupBirthSpec { + history, + range: CellRange::all(), + epoch: RouteEpoch::new(1)?, + creation: ProposalId::new([33; 16])?, + voters: (0..3) + .map(|index| { + Ok(InitialMember { + node: node(index)?, + member: MemberId::new(u64::from(index) + 1)?, + }) + }) + .collect::>()?, + learners: vec![], + }, + }, + )?; + let file = "first-group-setup.json"; + let plan_path = root.join(file); + let path = plan_path.to_str().context("UTF-8 first group setup path")?; + let setup = serde_json::json!({ + "reservation": plan, + "publication": { + "request_id": hex::encode([31; 16]), + "expected_revision": plan.expected_revision + 1, + }, + "archive": { + "request_id": hex::encode([34; 16]), + "expected_revision": plan.expected_revision + 2, + }, + }); + persist(root, file, &serde_json::to_vec(&setup)?)?; + let applied = cli + .call(&["first-group", "apply", &namespace, "--plan", path], OWNER) + .await? + .success()?; + ensure!( + applied["reservation"]["stage"] == "reserved" + && applied["publication"]["stage"] == "published" + && applied["archive"]["stage"] == "archive_initialized" + ); + Ok(()) +} + +async fn inspect_current( + service: &RaftCells, + node: NodeId, + revision: u64, +) -> Result { + let deadline = Instant::now() + TIMEOUT; + let mut last_error = None; + loop { + let remaining = deadline.saturating_duration_since(Instant::now()); + ensure!( + !remaining.is_zero(), + "daemon could not inspect {node:?} at fleet revision {revision}: {last_error:?}" + ); + match service.inspect_peer_node(node, revision, remaining).await { + Ok(observed) => return Ok(observed.view), + Err(error) => last_error = Some(error), + } + cellar::asyncrt::sleep(Duration::from_millis(10)).await; + } +} + +async fn admit_capacity( + cli: &OperatorCli, + service: &RaftCells, + root: &Path, + index: u8, +) -> Result<()> { + let node = node(index)?; + let observed = inspect_current(service, node, cli.revision().await?).await?; + ensure!( + observed.node.groups == 0, + "capacity must observe an empty standby" + ); + let attestation = capacity_attestation(observed)?; + let filename = format!("capacity-{index}.json"); + persist(root, &filename, &serde_json::to_vec(&attestation)?)?; + let path = root.join(filename); + let path = path.to_str().context("UTF-8 capacity attestation path")?; + let node_hex = hex::encode(node.bytes()); + for (offset, verb, state) in [ + (0, "qualify", FleetNodeState::Ready), + (1, "activate", FleetNodeState::Active), + ] { + let revision = cli.revision().await?; + inspect_current(service, node, revision).await?; + let admission: RaftFleetNodeAcceptance = serde_json::from_value( + cli.call( + &[ + "nodes", + verb, + &node_hex, + "--attestation", + path, + "--operation-id", + &hex::encode([20 + index * 2 + offset; 16]), + "--expected-revision", + &revision.to_string(), + ], + OWNER, + ) + .await? + .success()?, + )?; + ensure!(admission.admission.admitted_state == state); + } + Ok(()) +} + +fn capacity_attestation(observed: RaftHostObservation) -> Result { + Ok(FleetCapacityAttestation { + boot: observed.fleet_boot.context("capacity boot")?, + profile_revision: 1, + profile_digest: fleet_capacity_profile_digest(&observed.node.profile)?, + cpu_millis: 1000, + memory_bytes: 3 * 1024 * 1024 * 1024, + durable_bytes: observed.node.profile.byte_capacity.max(1024 * 1024 * 1024), + engine_slots: observed.node.profile.group_capacity.max(2), + memory_swap_disabled: true, + used_memory_bytes: 64 * 1024 * 1024, + used_durable_bytes: 1024 * 1024, + evidence_digest: [71; 32], + evidence_reference: "fixture-only/daemon-process".into(), + }) +} diff --git a/crates/cellar/tests/support/raft_daemon_recovery.rs b/crates/cellar/tests/support/raft_daemon_recovery.rs index be808f599..49e97f7c9 100644 --- a/crates/cellar/tests/support/raft_daemon_recovery.rs +++ b/crates/cellar/tests/support/raft_daemon_recovery.rs @@ -1,7 +1,7 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. //! A real hard-killed voter, immutable command retries and retained-disk resume. -//! This small dev-store fixture is not a 10 GB or object-store RTO campaign. +//! Records the process campaign for both the dev store and RustFS. use super::*; use std::{fs::OpenOptions, io::Write}; @@ -12,7 +12,7 @@ pub(super) struct Ledger { } impl Ledger { - pub(super) fn new(root: &Path) -> Result { + pub(super) fn new(root: &Path, backend: &str) -> Result { let path = std::env::var_os("CELLAR_RAFT_PROCESS_EVIDENCE") .map(PathBuf::from) .unwrap_or_else(|| root.join("process-recovery-ledger.jsonl")); @@ -27,7 +27,8 @@ impl Ledger { }; ledger.record(serde_json::json!({ "format": 1, "phase": "started", "scope": "retained_disk_hard_kill", - "store": "shared_dev_sqlite", "qualified_10gb": false, + "store": backend, "operator": "cli_admin", "archive_setup": "cli_admin", + "voter_startup": "owned_fresh_quorum", "qualified_10gb": false, }))?; cellar_ltx::FileSystem::sync_all(&cellar_ltx::DirectFileSystem, parent)?; Ok(ledger) @@ -52,7 +53,8 @@ impl Ledger { pub(super) struct Acknowledged { pub(super) cell: &'static str, - pub(super) leader: usize, + pub(super) ingress: Ingress, + pub(super) ingress_member: usize, pub(super) command: String, pub(super) retry_until: String, pub(super) receipt: Value, @@ -60,6 +62,7 @@ pub(super) struct Acknowledged { struct Request<'a> { cell: &'a str, + ingress: Ingress, command: &'a str, retry_until: &'a str, delta: i64, @@ -69,13 +72,14 @@ impl Request<'_> { async fn send( &self, client: &reqwest::Client, - addresses: &[(String, String)], + addresses: &[ManagedIngress], excluded: Option, ledger: &mut Ledger, ) -> Result { let deadline = Instant::now() + Duration::from_secs(20); loop { - for (index, (public, _)) in addresses.iter().enumerate() { + for (index, address) in addresses.iter().enumerate() { + let public = self.ingress.endpoint(address); if Some(index) == excluded { continue; } @@ -86,6 +90,7 @@ impl Request<'_> { ledger.record(serde_json::json!({ "phase": "send", "cell": self.cell, "command": self.command, "retry_until": self.retry_until, "delta": self.delta, "member": index + 1, + "ingress": self.ingress.label(), }))?; let response = client .post(format!( @@ -115,7 +120,7 @@ impl Request<'_> { }; if status.is_success() { let receipt: Value = serde_json::from_str(&body)?; - ledger.record(serde_json::json!({"phase":"acknowledged", "cell":self.cell, "command":self.command, "receipt":receipt}))?; + ledger.record(serde_json::json!({"phase":"acknowledged", "cell":self.cell, "command":self.command, "receipt":receipt, "ingress":self.ingress.label()}))?; return Ok(receipt); } ledger.record(serde_json::json!({"phase":"refused", "cell":self.cell, "command":self.command, "status":status.as_u16(), "body":body}))?; @@ -129,8 +134,8 @@ impl Request<'_> { } } -fn matching_receipt(actual: &Value, expected: &Value) -> Result<()> { - for field in ["value", "group", "incarnation"] { +pub(super) fn matching_receipt(actual: &Value, expected: &Value) -> Result<()> { + for field in ["value", "group", "group_incarnation", "cell", "incarnation"] { ensure!( actual[field] == expected[field], "replayed receipt differs at {field}: {actual}" @@ -151,33 +156,170 @@ fn matching_receipt(actual: &Value, expected: &Value) -> Result<()> { Ok(()) } +pub(super) const OBSERVATION: &str = "/fixture/raft-observation"; + +#[derive(serde::Deserialize, serde::Serialize)] +pub(super) struct VoterObservation { + member: u64, + term: u64, + leader: Option, + is_leader: bool, + ready: bool, + cells: u32, + group: String, + group_incarnation: String, +} + +pub(super) fn observe_voter(router: axum::Router, group: RaftGroupHandle) -> axum::Router { + router.route( + OBSERVATION, + axum::routing::get(move |headers: axum::http::HeaderMap| { + let group = group.clone(); + async move { + use axum::http::StatusCode; + if headers + .get(axum::http::header::AUTHORIZATION) + .and_then(|value| value.to_str().ok()) + .and_then(|value| value.strip_prefix("Bearer ")) + != Some(SECRET) + { + return Err(StatusCode::UNAUTHORIZED); + } + let observed = group + .observe(Duration::from_secs(5)) + .await + .map_err(|_| StatusCode::SERVICE_UNAVAILABLE)?; + Ok(axum::Json(VoterObservation { + member: observed.replica.member.get(), + term: observed.status.term.get(), + leader: observed.status.leader, + is_leader: observed.role == RaftReplicaRole::Leader, + ready: observed.status.ready, + cells: observed.cells, + group: hex::encode(observed.replica.scope.group.bytes()), + group_incarnation: hex::encode(observed.replica.scope.incarnation.bytes()), + })) + } + }), + ) +} + +pub(super) async fn observe_leader( + client: &reqwest::Client, + addresses: &[ManagedIngress], + killed: Option, + minimum_term: u64, + receipt: &Value, +) -> Result<(usize, u64, Vec)> { + observe_members(client, addresses, &[1, 2, 3], killed, minimum_term, receipt).await +} + +pub(super) async fn observe_members( + client: &reqwest::Client, + addresses: &[ManagedIngress], + members: &[u64; 3], + killed: Option, + minimum_term: u64, + receipt: &Value, +) -> Result<(usize, u64, Vec)> { + tokio::time::timeout(Duration::from_secs(20), async { + loop { + let observations = futures_util::future::try_join_all( + addresses + .iter() + .enumerate() + .filter(|(index, _)| Some(*index) != killed) + .map(|(index, address)| async move { + let response = client + .get(format!("{}{OBSERVATION}", address.axum)) + .bearer_auth(SECRET) + .send() + .await?; + if response.status() == reqwest::StatusCode::SERVICE_UNAVAILABLE { + return Ok(None); + } + let voter = response + .error_for_status()? + .json::() + .await?; + ensure!( + voter.member == members[index] + && voter.cells == 2 + && receipt["group"].as_str() == Some(voter.group.as_str()) + && receipt["group_incarnation"].as_str() + == Some(voter.group_incarnation.as_str()), + "wrong local voter observation" + ); + Ok::<_, anyhow::Error>(Some(voter)) + }), + ) + .await?; + if let Some(voters) = observations.into_iter().collect::>>() { + let agreed = voters.iter().find(|leader| { + leader.is_leader + && leader.ready + && leader.term >= minimum_term + && voters.iter().all(|voter| { + voter.term == leader.term + && voter.leader == Some(leader.member) + && voter.is_leader == (voter.member == leader.member) + }) + }); + if let Some(leader) = agreed { + let index = members + .iter() + .position(|member| *member == leader.member) + .context("observed leader is absent from selected members")?; + return Ok((index, leader.term, voters)); + } + } + tokio::time::sleep(Duration::from_millis(50)).await; + } + }) + .await + .context("managed voter leader observation deadline")? +} + pub(super) async fn hard_kill_and_resume( root: &Path, client: &reqwest::Client, processes: &mut [Process], - addresses: &mut [(String, String)], + addresses: &mut [ManagedIngress], acknowledged: &[Acknowledged], ledger: &mut Ledger, ) -> Result<()> { ensure!( acknowledged.len() == 2, - "both groups need acknowledged commands" + "both cells need acknowledged commands" ); - let killed = acknowledged[0].leader; - let old_pid = processes[killed].child.id(); - ledger.record( - serde_json::json!({"phase":"before_hard_kill", "pid":old_pid, "member":killed + 1}), - )?; - processes[killed].child.kill()?; - let status = processes[killed].child.wait()?; - ensure!(!status.success(), "injected hard kill exited successfully"); - #[cfg(unix)] - { - use std::os::unix::process::ExitStatusExt as _; - ensure!( - status.signal() == Some(9), - "worker was not killed by SIGKILL: {status}" - ); + let receipt = &acknowledged[0].receipt; + let minimum_term = receipt["term"].as_u64().context("acknowledged Raft term")?; + let (killed, term, voters) = + observe_leader(client, addresses, None, minimum_term, receipt).await?; + let old_pid = processes[killed].pid()?; + ledger.record(serde_json::json!({ + "phase":"before_hard_kill", "pid":old_pid, "member":killed + 1, "term":term, + "voters":voters, + }))?; + let process = &mut processes[killed]; + if let Some(voter) = &process.voter { + voter.kill_if_running()?; + } else { + process.child.kill()?; + } + let status = process.child.wait()?; + if let Some(voter) = &process.voter { + voter.killed()?; + } else { + ensure!(!status.success(), "injected hard kill exited successfully"); + #[cfg(unix)] + { + use std::os::unix::process::ExitStatusExt as _; + ensure!( + status.signal() == Some(9), + "worker was not killed by SIGKILL: {status}" + ); + } } ledger.record( serde_json::json!({"phase":"hard_kill_joined", "pid":old_pid, "status":status.to_string()}), @@ -186,6 +328,7 @@ pub(super) async fn hard_kill_and_resume( for old in acknowledged { let duplicate = Request { cell: old.cell, + ingress: old.ingress, command: &old.command, retry_until: &old.retry_until, delta: 11, @@ -204,6 +347,7 @@ pub(super) async fn hard_kill_and_resume( .to_string(); let next = Request { cell: old.cell, + ingress: old.ingress, command: &command, retry_until: &retry, delta: 7, @@ -226,23 +370,28 @@ pub(super) async fn hard_kill_and_resume( )?; current.push((old, next)); } - let disk = root.join(format!("disk-{killed}")); + let newer_term = term.checked_add(1).context("Raft term overflow")?; + let (successor, term, voters) = + observe_leader(client, addresses, Some(killed), newer_term, receipt).await?; + ledger.record(serde_json::json!({ + "phase":"successor_before_restart", "member":successor + 1, "term":term, + "voters":voters, + }))?; + let disk = docker::disk(root, killed); processes[killed] = Process::spawn( root, &format!("cluster-resume-{killed}"), &[std::ffi::OsStr::new("--data-dir"), disk.as_os_str()], false, )?; - ensure!( - processes[killed].child.id() != old_pid, - "resume reused the killed PID" - ); - addresses[killed] = processes[killed].addresses().await?; - ledger.record(serde_json::json!({"phase":"restarted", "old_pid":old_pid, "new_pid":processes[killed].child.id(), "member":killed + 1}))?; + addresses[killed] = processes[killed].managed_addresses().await?; + let new_pid = processes[killed].pid()?; + ensure!(new_pid != old_pid, "resume reused the killed PID"); + ledger.record(serde_json::json!({"phase":"restarted", "old_pid":old_pid, "new_pid":new_pid, "member":killed + 1}))?; for (old, receipt) in current { local_read( client, - &addresses[killed].0, + (old.ingress, &addresses[killed]), killed, old.cell, &receipt, @@ -258,6 +407,7 @@ pub(super) async fn hard_kill_and_resume( .to_string(); let next = Request { cell: old.cell, + ingress: old.ingress, command: &command, retry_until: &retry, delta: 5, @@ -272,6 +422,7 @@ pub(super) async fn hard_kill_and_resume( ); let duplicate = Request { cell: old.cell, + ingress: old.ingress, command: &old.command, retry_until: &old.retry_until, delta: 11, @@ -280,25 +431,33 @@ pub(super) async fn hard_kill_and_resume( &duplicate.send(client, addresses, None, ledger).await?, &old.receipt, )?; - local_read( - client, - &addresses[killed].0, - killed, - old.cell, - &next, - 23, - ledger, - ) - .await?; - ledger.record(serde_json::json!({"phase":"post_resume_write_verified", "cell":old.cell, "receipt":next}))?; + for ingress in Ingress::ALL { + local_read( + client, + (ingress, &addresses[killed]), + killed, + old.cell, + &next, + 23, + ledger, + ) + .await?; + } + ledger.record(serde_json::json!({ + "phase":"post_resume_write_verified", "cell":old.cell, + "receipt":next, "ingress":old.ingress.label(), + }))?; } - ledger.record(serde_json::json!({"phase":"retained_disk_hard_kill_passed", "groups":2, "qualified_10gb":false}))?; + ledger.record(serde_json::json!({ + "phase":"retained_disk_hard_kill_passed", "physical_groups":1, + "logical_cells":acknowledged.len(), "qualified_10gb":false, + }))?; Ok(()) } async fn local_read( client: &reqwest::Client, - public: &str, + (ingress, address): (Ingress, &ManagedIngress), member_index: usize, cell: &str, receipt: &Value, @@ -306,6 +465,7 @@ async fn local_read( ledger: &mut Ledger, ) -> Result<()> { let deadline = Instant::now() + Duration::from_secs(20); + let public = ingress.endpoint(address); loop { let response = client .get(format!("{public}/counters/{cell}")) @@ -327,10 +487,12 @@ async fn local_read( value["value"] == expected_value && value["served_by"] == member_index as u64 + 1 && value["group"] == receipt["group"] + && value["group_incarnation"] == receipt["group_incarnation"] && value["incarnation"] == receipt["incarnation"], "restarted voter served the wrong state: {value}" ); - ledger.record(serde_json::json!({"phase":"resumed_local_read_verified", "cell":cell, "receipt":value}))?; + ledger.record(serde_json::json!({"phase":"resumed_local_read_verified", "cell":cell, "receipt":value, + "ingress":ingress.label()}))?; return Ok(()); } ensure!( @@ -340,3 +502,118 @@ async fn local_read( tokio::time::sleep(Duration::from_millis(50)).await; } } + +pub(super) async fn verify_linearizable_before_loss( + client: &reqwest::Client, + address: &ManagedIngress, + ledger: &mut Ledger, +) -> Result<()> { + for ingress in Ingress::ALL { + let public = ingress.endpoint(address); + let reply = client + .get(format!("{public}/counters/one")) + .bearer_auth(SECRET) + .header("x-read-consistency", "linearizable") + .send() + .await? + .error_for_status()? + .json::() + .await?; + ensure!(reply["value"] == 23, "wrong pre-loss read: {reply}"); + ledger.record(serde_json::json!({ + "phase":"pre_loss_linearizable", "reply":reply, "ingress":ingress.label(), + }))?; + } + Ok(()) +} + +pub(super) async fn verify_quorum_loss( + client: &reqwest::Client, + address: &ManagedIngress, + survivor: usize, + ledger: &mut Ledger, +) -> Result<()> { + let client_id = ClientId::new([90; 16])?; + let id = CommandId::new(client_id, 4)?; + let command = format!("{}:4", hex::encode([90; 16])); + let retry = (std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH)? + .as_millis() + + 60_000) + .to_string(); + let ambiguity = format!("Ambiguous {{ command: {id:?} }}; cause="); + for ingress in Ingress::ALL { + let public = ingress.endpoint(address); + let url = format!("{public}/counters/one"); + let response = client + .get(&url) + .bearer_auth(SECRET) + .header("x-read-consistency", "linearizable") + .send() + .await?; + let status = response.status(); + let cause = response.text().await?; + ledger.record(serde_json::json!({ + "phase":"no_quorum_linearizable", "member":survivor + 1, + "status":status.as_u16(), "cause":cause, "ingress":ingress.label(), + }))?; + ensure!( + status == 503 && quorum_refusal(&cause), + "linearizable read without quorum returned {status}: {cause}" + ); + + ledger.record(serde_json::json!({ + "phase":"no_quorum_write_send", "member":survivor + 1, + "cell":"one", "command":command, "retry_until":retry, "delta":9, + "ingress":ingress.label(), + }))?; + let response = client + .post(format!("{url}?delta=9")) + .bearer_auth(SECRET) + .header("x-command-id", &command) + .header("x-retry-until-ms", &retry) + .header("x-write-durability", "quorum") + .send() + .await?; + let status = response.status(); + let body = response.text().await?; + let cause = body.strip_prefix(&ambiguity).unwrap_or(&body); + ledger.record(serde_json::json!({ + "phase":"no_quorum_write", "member":survivor + 1, + "command":command, "status":status.as_u16(), "cause":cause, "body":body, + "ingress":ingress.label(), + }))?; + ensure!( + status == 503 && quorum_refusal(cause), + "quorum write without a majority returned {status}: {body}" + ); + + // The same surviving voter still offers an explicit historical read. + let reply = client + .get(&url) + .bearer_auth(SECRET) + .header("x-read-consistency", "stale") + .header("x-read-policy", "disconnected-historical") + .send() + .await? + .error_for_status()? + .json::() + .await?; + ensure!( + reply["value"] == 23 && reply["served_by"] == survivor + 1, + "surviving voter returned the wrong historical image: {reply}" + ); + ledger.record(serde_json::json!({ + "phase":"no_quorum_historical", "member":survivor + 1, + "reply":reply, "ingress":ingress.label(), + }))?; + } + Ok(()) +} + +fn quorum_refusal(cause: &str) -> bool { + matches!( + cause, + "Unavailable" | "NotLeader { leader: None }" | "VersionUnavailable" + ) +} diff --git a/crates/cellar/tests/support/raft_daemon_repair.rs b/crates/cellar/tests/support/raft_daemon_repair.rs new file mode 100644 index 000000000..a42c2c191 --- /dev/null +++ b/crates/cellar/tests/support/raft_daemon_repair.rs @@ -0,0 +1,627 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Failed voter replacement owned by a surviving Active voter in the lab. +use super::*; + +pub(crate) const ENABLED: &str = "CELLAR_RAFT_DAEMON_REPAIR_GATE"; +pub(crate) const REPAIRED: [u64; 3] = [4, 2, 3]; +const OBSERVER: u8 = 4; +const FAILED: usize = 0; +pub(crate) const WORKER: u8 = 1; +const REQUEST: &str = "repair-worker-request.json"; +const READY: &str = "repair-worker-ready.json"; +const TERMINAL: &str = "repair-worker-terminal.json"; + +#[derive(serde::Serialize, serde::Deserialize)] +struct WorkerRequest { + owner: String, + failed: FleetBootBinding, +} + +#[derive(serde::Serialize, serde::Deserialize)] +struct WorkerReady { + origin: FleetBootBinding, + revision: u64, +} + +#[derive(serde::Serialize, serde::Deserialize)] +struct Removed { + operation: [u8; 16], + revision: u64, +} + +fn publish(root: &Path, file: &str, value: &T) -> Result<()> { + let staged = format!("{file}.pending"); + persist(root, &staged, &serde_json::to_vec(value)?)?; + std::fs::rename(root.join(staged), root.join(file))?; + std::fs::File::open(root)?.sync_all()?; + Ok(()) +} + +fn read(root: &Path, file: &str) -> Result> { + match std::fs::read(root.join(file)) { + Ok(bytes) => Ok(Some(serde_json::from_slice(&bytes)?)), + Err(error) if error.kind() == std::io::ErrorKind::NotFound => Ok(None), + Err(error) => Err(error.into()), + } +} + +async fn await_worker( + root: &Path, + file: &str, + timeout: Duration, +) -> Result { + tokio::time::timeout(timeout, async { + loop { + if let Some(value) = read(root, file)? { + return Ok(value); + } + if let Some(Err(error)) = read::>(root, TERMINAL)? { + anyhow::bail!("Active repair worker: {error}"); + } + tokio::time::sleep(Duration::from_millis(50)).await; + } + }) + .await + .with_context(|| format!("Active repair worker did not publish {file}"))? +} + +const MIB: u64 = 1024 * 1024; +const DURABLE_OVERHEAD: u64 = 128 * MIB; +const DURABLE_SEED: u64 = 256 * MIB; +const DURABLE_RESERVE: u64 = 64 * MIB; +pub(super) const STANDBY_DURABLE_HEADROOM: u64 = DURABLE_OVERHEAD + DURABLE_SEED + DURABLE_RESERVE; + +pub(super) async fn start_observer( + operator: &Arc, + root: &Path, +) -> Result<(Node, RaftCells)> { + let declaration = operator.declaration()?; + let spec = node_spec( + OBSERVER, + RaftManagedHostOptions::capability_digest(&[declaration])?, + )?; + let snapshot = operator.fleet.read(TIMEOUT).await?; + operator + .fleet + .register_node( + &snapshot, + &request(144, snapshot.revision()), + &spec, + TIMEOUT, + ) + .await?; + let disk = root.join("repair-observer"); + std::fs::create_dir(&disk)?; + let history = history()?; + let class = counter::definition()?; + let application = counter::application( + counter::CounterApi::new(SECRET.into(), history.cluster, history.namespace, &class)?, + class, + )?; + // The standalone observer joins the daemon deployment's authenticated + // transport. The application catalogue uses a separate RustFS bucket. + let deployment = Bucket::open_dev(&root.join("fleet.sqlite"))?; + let peer_key = cellar::peer_auth::load_existing(&deployment).await?; + let node = + public_node::node_with_peer_key(&disk, application.clone(), &spec.label, peer_key).await?; + let started = async { + let prepared = operator + .clone() + .prepare( + NativeRaftNode { + node: spec.label.clone(), + data_dir: node.info().data_dir.clone(), + }, + OBSERVER, + false, + false, + operator.bucket.raft_store(), + ) + .await?; + node.start_raft_cells( + prepared.config, + RaftCellsServices { + listeners: prepared.listeners, + tls: prepared.tls, + store: prepared.store, + application, + versions: prepared.versions, + recovery: None, + }, + ) + .await + } + .await; + match started { + Ok(cells) => Ok((node, cells)), + Err(error) => { + let joined = node.shutdown(ShutdownMode::Preserve).await; + Err(error).context(format!("repair observer startup cleanup: {joined:?}")) + } + } +} + +async fn bind_live_voters( + operator: &Operator, + observer: &RaftCells, + root: &Path, + admin: &ManagedIngress, + processes: &[Process], + directory: RangeDirectory, +) -> Result { + let cli = OperatorCli::connect( + admin.axum.clone(), + hex::encode(operator.fleet.catalog_id()?), + ); + let voter = processes[FAILED] + .voter + .as_ref() + .context("repair requires owned Docker voters")?; + let (owner, _) = voter.fleet_identity()?; + let isolation = FleetDockerIsolation::new( + operator.fleet.clone(), + cellar::docker::Docker::new("/var/run/docker.sock"), + &owner, + vec![directory], + )?; + let mut failed = None; + for (index, process) in processes.iter().enumerate() { + let id = node(u8::try_from(index)?)?; + let snapshot = operator.fleet.read(TIMEOUT).await?; + let observed = inspect_current(observer, id, snapshot.revision()).await?; + let attestation = capacity_attestation(observed)?; + match operator + .fleet + .authorize_capacity(&snapshot, id, TIMEOUT) + .await + { + Ok(capacity) => ensure!(capacity.attestation().boot == attestation.boot), + Err(FleetCatalogError::Conflict) => { + // Retained crash startup roots a new boot before its old + // attestation can be reused. Qualify the exact new boot. + let filename = format!("repair-current-capacity-{index}.json"); + persist(root, &filename, &serde_json::to_vec(&attestation)?)?; + standby::capacity( + &cli, + &hex::encode(id.bytes()), + root.join(filename) + .to_str() + .context("current capacity attestation path")?, + "activate", + 150 + u8::try_from(index)?, + FleetNodeState::Active, + ) + .await?; + } + Err(error) => return Err(error.into()), + } + let (owner_check, container) = process + .voter + .as_ref() + .context("unowned repair voter")? + .fleet_identity()?; + ensure!( + owner_check == owner, + "repair voter belongs to another deployment" + ); + isolation.bind_node(id, &container, TIMEOUT).await?; + if index == FAILED { + failed = Some(attestation.boot); + } + } + Ok(WorkerRequest { + owner, + failed: failed.context("failed voter boot binding")?, + }) +} + +async fn wait_for_recovery( + worker: &FleetMovementService, + ready: impl Fn(&FleetMovementServiceStatus) -> bool, + timeout: Duration, +) -> Result { + tokio::time::timeout(timeout, async { + loop { + let status = worker.status(); + if ready(&status) { + return status; + } + tokio::time::sleep(Duration::from_millis(50)).await; + } + }) + .await + .with_context(|| format!("public failure recovery deadline: {:?}", worker.status())) +} + +pub(crate) async fn qualify( + root: &Path, + client: &reqwest::Client, + processes: &mut [Process], + addresses: &[ManagedIngress], + original: &Value, + ledger: &mut recovery::Ledger, +) -> Result<(Process, ManagedIngress)> { + ensure!( + cfg!(target_os = "linux") + && docker::enabled() + && std::env::var_os(RUSTFS_ENDPOINT).is_some(), + "repair gate requires the dedicated RustFS Docker lab" + ); + let domain = Arc::new(ExecutionDomain::new(tokio::runtime::Handle::current())); + let work = domain + .run(async { + let operator = Arc::new(Operator::open(domain.clone(), root)?); + let mut spare = standby::launch(&operator, root, &addresses[FAILED], ledger).await?; + let (node, observer) = Box::pin(start_observer(&operator, root)).await?; + let result = Box::pin(repair( + &operator, &observer, root, client, processes, addresses, original, &mut spare, + ledger, + )) + .await; + let joined = node.shutdown(ShutdownMode::Preserve).await; + ensure!( + matches!(joined, Ok(NodeOutcome::Stopped(_))), + "repair observer join: {joined:?}" + ); + Ok::<_, anyhow::Error>((spare, result?)) + }) + .await; + domain.shutdown().await; + work? +} + +#[allow(clippy::too_many_arguments)] +async fn repair( + operator: &Operator, + observer: &RaftCells, + root: &Path, + client: &reqwest::Client, + processes: &mut [Process], + addresses: &[ManagedIngress], + original: &Value, + spare: &mut Process, + ledger: &mut recovery::Ledger, +) -> Result { + let view = observer.observe_node(TIMEOUT).await?; + ensure!(view.node.groups == 0, "inspection observer owns a voter"); + let selected = operator + .domain + .run( + operator + .adapter + .existing_namespace(operator.declaration()?.namespace), + ) + .await??; + let directory = RangeDirectory::new(selected, configuration::limits().directory)?; + let request = bind_live_voters( + operator, + observer, + root, + &addresses[FAILED], + processes, + directory, + ) + .await?; + let (leader, _, _) = recovery::observe_leader(client, addresses, None, 1, original).await?; + let retry = (std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH)? + .as_millis() + + 300_000) + .to_string(); + let command = format!("{}:1", hex::encode([155; 16])); + let archived = maintenance::verify_sql( + client, + &addresses[leader], + Ingress::Native, + &command, + &retry, + "s3", + ) + .await?; + publish(root, REQUEST, &request)?; + let ready: WorkerReady = await_worker(root, READY, TIMEOUT * 3).await?; + let cut = operator.fleet.read(TIMEOUT).await?; + let worker_node = node(WORKER)?; + let authorized = operator + .fleet + .authorize_host(&cut, worker_node, TIMEOUT) + .await?; + let capacity = operator + .fleet + .authorize_capacity(&cut, worker_node, TIMEOUT) + .await?; + ensure!( + authorized.state() == FleetNodeState::Active + && capacity.attestation().boot == ready.origin + && cut.revision() >= ready.revision + ); + ledger.record(serde_json::json!({"phase":"repair_worker_ready", + "member":u64::from(WORKER) + 1, "boot":hex::encode(ready.origin.boot), + "revision":ready.revision}))?; + let failed = request.failed; + let standby = node(standby::INDEX)?; + let failed_process = &mut processes[FAILED]; + let old_pid = failed_process.pid()?; + failed_process + .voter + .as_ref() + .context("owned failed voter")? + .kill_if_running()?; + failed_process.exited(false).await?; + failed_process + .voter + .as_ref() + .context("joined failed voter")? + .killed()?; + let old_disk = docker::disk(root, FAILED).join("raft"); + let retained = retained_disk::disk_digest(&old_disk)?; + ledger.record( + serde_json::json!({"phase":"repair_failure_joined", "member":FAILED + 1, + "pid":old_pid,"boot":hex::encode(failed.boot),"receipt":archived,"command":command}), + )?; + let terminal: Result = await_worker(root, TERMINAL, TIMEOUT * 17).await?; + let terminal = terminal.map_err(|error| anyhow::anyhow!("Active repair worker: {error}"))?; + let cut = operator.fleet.read(TIMEOUT).await?; + let removed = operator + .fleet + .isolated_removal_receipt(&cut, ProposalId::new(terminal.operation)?, TIMEOUT) + .await? + .context("rooted isolated removal receipt")?; + ensure!( + removed.node() == failed.node + && removed.applied_revision() == terminal.revision + && removed.isolation_digest() != [0; 32] + && removed.candidate_digest() != [0; 32] + ); + ensure!( + operator + .fleet + .authorize_host(&cut, node(u8::try_from(FAILED)?)?, TIMEOUT) + .await + .is_err(), + "terminally removed voter can still obtain host authority" + ); + ensure!( + operator + .fleet + .pending_reservations(&cut, TIMEOUT) + .await? + .is_empty() + && operator + .fleet + .pending_movements(&cut, TIMEOUT) + .await? + .is_empty() + ); + ensure!( + retained_disk::disk_digest(&old_disk)? == retained, + "failed voter engine was reclaimed" + ); + inspect_current(observer, standby, cut.revision()).await?; + let replacement = observer + .observe_peer_fleet_membership( + standby, + operator + .fleet + .authorize_group(&cut, &history()?, TIMEOUT) + .await?, + cut.revision(), + TIMEOUT, + ) + .await?; + let proof = replacement.membership(); + let config = proof.configuration(); + let voters = config + .voters() + .iter() + .map(|member| member.get()) + .collect::>(); + ensure!( + proof.replica().member.get() == 4 + && voters == [2, 3, 4] + && config.learners().is_empty() + && config.next_learners().is_empty() + && config.outgoing_voters().is_empty() + ); + ledger.record(serde_json::json!({"phase":"repair_removed", "member":FAILED + 1, + "replacement":4,"voters":voters,"revision":removed.applied_revision(),"retained_files":retained.len(), + "isolation_digest":hex::encode(removed.isolation_digest())}))?; + let ingress = spare.managed_addresses().await?; + let replacement_routes = [ingress, addresses[1].clone(), addresses[2].clone()]; + let term = + maintenance::handoff_to(client, &replacement_routes, &REPAIRED, 0, &archived).await?; + for adapter in Ingress::ALL { + let replay = maintenance::verify_sql( + client, + &replacement_routes[0], + adapter, + &command, + &retry, + "s3", + ) + .await?; + recovery::matching_receipt(&replay, &archived)?; + let fresh = format!("{}:2", hex::encode([155; 16])); + let written = maintenance::verify_sql( + client, + &replacement_routes[0], + adapter, + &fresh, + &retry, + "quorum", + ) + .await?; + ensure!( + written["group"] == archived["group"] + && written["group_incarnation"] == archived["group_incarnation"] + && written["sequence"] + .as_u64() + .context("replacement sequence")? + == archived["sequence"] + .as_u64() + .and_then(|sequence| sequence.checked_add(1)) + .context("archived sequence")?, + "replacement lost command history" + ); + ledger.record( + serde_json::json!({"phase":"repair_replacement_sql","member":4,"term":term, + "ingress":adapter.label(),"replayed":replay,"written":written}), + )?; + } + let [ingress, _, _] = replacement_routes; + Ok(ingress) +} + +pub(super) async fn run_worker( + operator: Arc, + service: RaftCells, + root: PathBuf, + directory: RangeDirectory, + mut stop: tokio::sync::watch::Receiver, +) -> Result<()> { + let request = cellar::asyncrt::select! { + _ = stop.changed() => return Ok(()), + request = async { + loop { + if let Some(request) = read(&root, REQUEST)? { + return Ok::(request); + } + tokio::time::sleep(Duration::from_millis(50)).await; + } + } => request?, + }; + let domain = operator.domain.clone(); + let result = domain + .run(Box::pin(recover_on_voter( + &operator, &service, &root, directory, request, stop, + ))) + .await?; + let terminal = match result { + Ok(Some(removed)) => Ok(removed), + Ok(None) => return Ok(()), + Err(error) => Err(format!("{error:#}")), + }; + publish(&root, TERMINAL, &terminal) +} + +async fn recover_on_voter( + operator: &Operator, + observer: &RaftCells, + root: &Path, + directory: RangeDirectory, + request: WorkerRequest, + mut stop: tokio::sync::watch::Receiver, +) -> Result> { + let cut = operator.fleet.read(TIMEOUT).await?; + let origin = node(WORKER)?; + let authorized = operator.fleet.authorize_host(&cut, origin, TIMEOUT).await?; + let capacity = operator + .fleet + .authorize_capacity(&cut, origin, TIMEOUT) + .await?; + let origin = capacity.attestation().boot.clone(); + let view = inspect_current(observer, authorized.node(), cut.revision()).await?; + ensure!( + authorized.state() == FleetNodeState::Active + && view.node.groups == 1 + && view.fleet_boot.as_ref() == Some(&origin), + "repair worker must own the exact live Active voter boot" + ); + let bytes = view.node.profile.bytes_per_group; + let failed = request.failed; + let platform = FleetDockerIsolation::new( + operator.fleet.clone(), + cellar::docker::Docker::new("/run/docker.sock"), + &request.owner, + vec![directory], + )?; + let policy = FleetDockerResourcePolicy { + durable_overhead_bytes: DURABLE_OVERHEAD, + resident: FleetResourceBounds { + engine_slots: 1, + durable_bytes: bytes, + memory_bytes: 128 * MIB, + }, + peak: FleetResourceBounds { + engine_slots: 1, + durable_bytes: bytes + DURABLE_SEED, + memory_bytes: 256 * MIB, + }, + reserve: FleetResourceBounds { + engine_slots: 0, + durable_bytes: DURABLE_RESERVE, + memory_bytes: 128 * MIB, + }, + }; + let resources = Arc::new(FleetDockerResources::new( + platform.clone(), + observer.clone(), + policy.clone(), + )?); + let standby = node(standby::INDEX)?; + // Measure the current receiving target before injecting a fault. The same + // provider resamples against the current Root when reserving the movement. + let cut = operator.fleet.read(TIMEOUT).await?; + let standby_view = inspect_current(observer, standby, cut.revision()).await?; + let usage = resources.observe(standby, TIMEOUT).await?; + let qualified = operator + .fleet + .authorize_capacity(&cut, standby, TIMEOUT) + .await?; + let durable_capacity = standby_view + .node + .profile + .byte_capacity + .min(qualified.attestation().durable_bytes); + let durable_required = usage + .used + .durable_bytes + .checked_add(policy.peak.durable_bytes) + .and_then(|total| total.checked_add(policy.reserve.durable_bytes)) + .context("standby repair durable budget overflow")?; + ensure!( + durable_capacity >= durable_required, + "standby disk budget {durable_capacity} cannot cover repair charge {durable_required}" + ); + let verifier = open_archive_verifier(operator, &root.join("repair-replay")).await?; + let worker = operator + .fleet + .start_failure_recovery_service( + observer.clone(), + resources.clone(), + resources, + Arc::new(platform), + FleetMovementServiceOptions { + interval: Duration::from_millis(50), + retry_max_interval: Duration::from_millis(500), + turn_timeout: TIMEOUT, + }, + FleetFailureRecoveryOptions { + detection: FleetFailureDetectionOptions::default(), + archive: verifier, + }, + ) + .await?; + + let outcome = cellar::asyncrt::select! { + _ = stop.changed() => Ok(None), + result = async { + wait_for_recovery(&worker, |status| status.failure_detection.as_ref().is_some_and(|view| + view.nodes.iter().any(|item| item.boot == failed && item.health == FleetNodeHealth::Healthy)), + TIMEOUT * 3).await?; + publish(root, READY, &WorkerReady { origin, revision: cut.revision() })?; + let terminal = wait_for_recovery(&worker, |status| status.failure_recovery.as_ref().is_some_and(|view| + view.phase == FleetFailureRecoveryPhase::Removed), TIMEOUT * 16).await?; + let recovery = terminal.failure_recovery.context("terminal recovery evidence")?; + ensure!(recovery.context.boot() == &failed + && recovery.verified_histories == 1 && recovery.total_histories == 1); + Ok(Some(Removed { + operation: recovery.context.operation().bytes(), + revision: recovery.removed_revision.context("rooted removal revision")?, + })) + } => result, + }; + worker.shutdown().await; + outcome +} diff --git a/crates/cellar/tests/support/raft_daemon_standby.rs b/crates/cellar/tests/support/raft_daemon_standby.rs new file mode 100644 index 000000000..1374873b6 --- /dev/null +++ b/crates/cellar/tests/support/raft_daemon_standby.rs @@ -0,0 +1,267 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! A separately owned repair destination with public capacity admission. +use super::*; + +pub(crate) const INDEX: u8 = 3; +pub(crate) const ENABLED: &str = "CELLAR_RAFT_DAEMON_STANDBY_GATE"; +const ATTESTATION: &str = "repair-standby-capacity.json"; +const WAITING: &str = "repair standby waiting for its local replica"; + +pub(super) async fn wait_for_attachment(service: &RaftCells, root: &Path) -> Result<()> { + let view = service.observe_node(TIMEOUT).await?; + ensure!( + view.node.groups == 0 && view.node.identity.node == node(INDEX)?, + "repair standby already owns a group" + ); + let attestation = capacity_attestation(view)?; + persist(root, ATTESTATION, &serde_json::to_vec(&attestation)?)?; + cellar::note!("{WAITING}"); + tokio::time::timeout(TIMEOUT * 16, async { + loop { + if service.resource_usage().retained_manifest_cells == Some(2) { + let group = service.group(history()?)?; + let promoted = group.quorum_membership(TIMEOUT).await.is_ok_and(|proof| { + proof.replica().member.get() == u64::from(INDEX) + 1 + && proof + .configuration() + .voters + .contains(&proof.replica().member.get()) + && proof.configuration().voters_outgoing.is_empty() + }); + if !promoted { + cellar::asyncrt::sleep(Duration::from_millis(50)).await; + continue; + } + let view = service.observe_node(TIMEOUT).await?; + ensure!( + view.node.groups == 1 && view.fleet_boot.as_ref() == Some(&attestation.boot), + "repair changed the selected standby boot" + ); + return Ok::<_, anyhow::Error>(()); + } + cellar::asyncrt::sleep(Duration::from_millis(50)).await; + } + }) + .await + .context("repair standby did not attach a retained replica")??; + Ok(()) +} + +pub(super) async fn capacity( + cli: &OperatorCli, + node: &str, + path: &str, + verb: &str, + id: u8, + expected: FleetNodeState, +) -> Result<()> { + let operation = hex::encode([id; 16]); + let revision = cli.revision().await?.to_string(); + let args = [ + "nodes", + verb, + node, + "--attestation", + path, + "--operation-id", + &operation, + "--expected-revision", + &revision, + ]; + let deadline = Instant::now() + TIMEOUT; + loop { + let remaining = deadline.saturating_duration_since(Instant::now()); + ensure!(!remaining.is_zero(), "standby {verb} confirmation deadline"); + let reply = cli.call_with_timeout(&args, OWNER, remaining).await?; + if reply.code == 5 + && matches!( + reply.value["error"]["code"].as_str(), + Some("unavailable" | "busy") + ) + { + // Directory refresh can trail the boot CAS. Preserve the same + // revision, attestation and request while reconciling admission. + tokio::time::sleep(Duration::from_millis(25)).await; + continue; + } + let admitted: RaftFleetNodeAcceptance = serde_json::from_value(reply.success()?)?; + ensure!(admitted.admission.admitted_state == expected); + return Ok(()); + } +} + +async fn bind_container(operator: &Operator, voter: &docker::Voter) -> Result<()> { + let (owner, container) = voter.fleet_identity()?; + let selected = operator + .domain + .run( + operator + .adapter + .existing_namespace(operator.declaration()?.namespace), + ) + .await??; + let directory = RangeDirectory::new(selected, configuration::limits().directory)?; + let isolation = FleetDockerIsolation::new( + operator.fleet.clone(), + cellar::docker::Docker::new("/var/run/docker.sock"), + &owner, + vec![directory], + )?; + isolation + .bind_node(node(INDEX)?, &container, TIMEOUT) + .await?; + Ok(()) +} + +pub(super) async fn launch( + operator: &Operator, + root: &Path, + admin: &ManagedIngress, + ledger: &mut recovery::Ledger, +) -> Result { + let cli = OperatorCli::connect( + admin.axum.clone(), + hex::encode(operator.fleet.catalog_id()?), + ); + let spec = node_spec( + INDEX, + RaftManagedHostOptions::capability_digest(&[operator.declaration()?])?, + )?; + let node = hex::encode(spec.node.bytes()); + let descriptor = RaftFleetNodeDescriptor { + node_id: node.clone(), + disk_id: hex::encode(spec.disk.bytes()), + physical_host: hex::encode(spec.physical_host.bytes()), + label: spec.label, + server_name: spec.server_name, + capability_digest: hex::encode(spec.capability_digest), + storage_instance: None, + }; + let filename = "repair-standby-descriptor.json"; + persist(root, filename, &serde_json::to_vec(&descriptor)?)?; + let descriptor_path = root.join(filename); + let args = [ + "nodes", + "add", + "--descriptor", + descriptor_path + .to_str() + .context("standby descriptor path")?, + "--operation-id", + &hex::encode([141; 16]), + "--expected-revision", + &cli.revision().await?.to_string(), + ]; + let registered: RaftFleetNodeAcceptance = + serde_json::from_value(cli.call(&args, OWNER).await?.success()?)?; + ensure!(registered.admission.admitted_state == FleetNodeState::Registered); + let disk = docker::disk(root, usize::from(INDEX)); + let mut process = Process::spawn( + root, + &format!("cluster-{INDEX}"), + &[std::ffi::OsStr::new("--data-dir"), disk.as_os_str()], + false, + )?; + let deadline = Instant::now() + Duration::from_secs(20); + loop { + let logs = process.logs()?; + ensure!( + process.child.try_wait()?.is_none() && Instant::now() < deadline, + "standby did not publish its startup attestation\n{logs}" + ); + ensure!( + !logs.contains("cellar listening on ") && !logs.contains("cellar axum listening on "), + "standby opened SQL ingress without a replica\n{logs}" + ); + if logs.contains(WAITING) { + break; + } + tokio::time::sleep(Duration::from_millis(25)).await; + } + let path = root.join(ATTESTATION); + let attestation: FleetCapacityAttestation = serde_json::from_slice(&std::fs::read(&path)?)?; + let path = path.to_str().context("standby attestation path")?; + ensure!( + attestation.boot.node == spec.node.bytes() && attestation.boot.disk == spec.disk.bytes() + ); + capacity(&cli, &node, path, "qualify", 142, FleetNodeState::Ready).await?; + if let Some(voter) = &process.voter { + voter.running()?; + bind_container(operator, voter) + .await + .context("bind Ready standby to its immutable Docker container")?; + } + capacity(&cli, &node, path, "activate", 143, FleetNodeState::Active).await?; + let status: RaftFleetNodeStatus = serde_json::from_value( + cli.call(&["nodes", "status", &node], OWNER) + .await? + .success()?, + )?; + ensure!(status.state == FleetNodeState::Active && status.capacity == Some(attestation.clone())); + let logs = process.logs()?; + ensure!( + !logs.contains("cellar listening on ") && !logs.contains("cellar axum listening on "), + "capacity activation opened SQL ingress without a replica\n{logs}" + ); + ledger.record(serde_json::json!({ + "phase":"repair_standby_ready", "node":node, "process":hex::encode(attestation.boot.process), + "engine":hex::encode(attestation.boot.engine), "pid":process.pid()?, + "capacity_revision":status.observed_revision, "docker_bound":process.voter.is_some(), + }))?; + Ok(process) +} + +pub(crate) async fn qualify( + root: &Path, + admin: &ManagedIngress, + ledger: &mut recovery::Ledger, +) -> Result<()> { + let domain = Arc::new(ExecutionDomain::new(tokio::runtime::Handle::current())); + let result = domain + .run(async { + let operator = Arc::new(Operator::open(domain.clone(), root)?); + let mut process = launch(&operator, root, admin, ledger).await?; + let (controller, observer) = Box::pin(repair::start_observer(&operator, root)).await?; + let inspected = async { + let own = observer.observe_node(TIMEOUT).await?; + ensure!(own.node.groups == 0 && own.fleet_boot.is_some()); + let cut = operator.fleet.read(TIMEOUT).await?; + let remote = inspect_current(&observer, node(INDEX)?, cut.revision()).await?; + ensure!(remote.node.groups == 0 && remote.node.identity.node == node(INDEX)?); + ledger.record(serde_json::json!({ + "phase":"repair_observer_ready", "local_groups":own.node.groups, + "standby_groups":remote.node.groups,"revision":cut.revision(), + }))?; + Ok::<_, anyhow::Error>(()) + } + .await; + let joined = controller.shutdown(ShutdownMode::Preserve).await; + ensure!( + matches!(joined, Ok(NodeOutcome::Stopped(_))), + "repair observer join: {joined:?}" + ); + inspected?; + if let Some(voter) = &process.voter { + voter.terminate()?; + } else { + ensure!(Command::new("/bin/kill") + .args(["-TERM", &process.child.id().to_string()]) + .status()? + .success()); + } + let logs = process.exited(false).await?; + ensure!( + logs.contains("startup interrupted by SIGTERM") + && !logs.contains("cellar listening on "), + "standby cancellation did not join startup\n{logs}" + ); + ledger.record( + serde_json::json!({"phase":"repair_standby_joined", "startup_cancelled":true}), + )?; + Ok(()) + }) + .await; + domain.shutdown().await; + result? +} diff --git a/crates/cellar/tests/support/raft_directory_managed.rs b/crates/cellar/tests/support/raft_directory_managed.rs new file mode 100644 index 000000000..d26a8bfdf --- /dev/null +++ b/crates/cellar/tests/support/raft_directory_managed.rs @@ -0,0 +1,203 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Root and attach both directories for authenticated inspection tests. +use super::*; +use cellar::application::consensus::verification::cell_groups::RangeDirectory; +use cellar::bucket::Bucket; + +pub(super) struct Managed { + _root: tempfile::TempDir, + pub(super) domain: Arc, + pub(super) fleet: FleetCatalog, + catalog: Catalog, + pub(super) namespaces: Vec, + directories: Vec, +} + +fn request(id: u8, revision: u64) -> FleetRequestIdentity { + FleetRequestIdentity { + id: [id; 16], + operation: [id; 16], + expected_revision: revision, + principal: "directory-qualification".into(), + } +} + +impl Managed { + pub(super) async fn open( + store: Arc, + namespaces: &[NamespaceConfig], + class: &RaftClass, + ) -> Result { + let root = tempfile::tempdir()?; + let domain = Arc::new(ExecutionDomain::new(tokio::runtime::Handle::current())); + let (fleet, catalog, namespaces, directories) = domain + .run(async { + let fleet = FleetCatalog::new( + &domain, + Bucket::open_dev(&root.path().join("fleet.sqlite"))?, + namespaces[0].cluster, + IncarnationId::new([80; 16])?, + "directory/", + )?; + fleet.initialize(TIMEOUT).await?; + let catalog = Catalog::new(store, catalog_options())?; + let mut declarations = Vec::new(); + let mut directories = Vec::new(); + for (index, config) in namespaces.iter().enumerate() { + let cut = fleet.read(TIMEOUT).await?; + let declaration = fleet + .provision_managed_sql_namespace( + &request(10 + index as u8, cut.revision()), + &catalog, + config.namespace, + class, + limits().directory, + TIMEOUT, + ) + .await?; + let namespace = catalog + .existing_namespace(declaration.namespace.clone()) + .await?; + directories.push(RangeDirectory::new(namespace, limits().directory)?); + declarations.push(declaration); + } + Ok::<_, anyhow::Error>((fleet, catalog, declarations, directories)) + }) + .await??; + Ok(Self { + _root: root, + domain, + fleet, + catalog, + namespaces, + directories, + }) + } + + pub(super) async fn enroll(&self, peers: &[RaftPeerNode]) -> Result<()> { + self.domain + .run(async { + let capability_digest = + RaftManagedHostOptions::capability_digest(&self.namespaces)?; + for (index, peer) in peers.iter().enumerate() { + let cut = self.fleet.read(TIMEOUT).await?; + self.fleet + .register_node( + &cut, + &request(20 + index as u8, cut.revision()), + &FleetNodeSpec { + node: peer.node, + disk: peer.disk, + physical_host: IncarnationId::new([90 + index as u8; 16])?, + label: peer.label.clone(), + server_name: peer.server_name.clone(), + capability_digest, + storage_instance: None, + }, + TIMEOUT, + ) + .await?; + } + Ok::<_, anyhow::Error>(()) + }) + .await? + } + + pub(super) async fn attach(&self, cluster: &Cluster) -> Result<()> { + self.domain + .run(async { + for index in 0..cluster.services.len() { + let service = cluster.service(index)?; + let observed = service.observe_node(TIMEOUT).await?; + ensure!(observed.node.groups == 0); + let attestation = FleetCapacityAttestation { + boot: observed + .fleet_boot + .context("directory qualification boot")?, + profile_revision: 1, + profile_digest: fleet_capacity_profile_digest(&observed.node.profile)?, + cpu_millis: 1000, + memory_bytes: 3 * 1024 * 1024 * 1024, + durable_bytes: observed.node.profile.byte_capacity, + engine_slots: 2, + memory_swap_disabled: true, + used_memory_bytes: 64 * 1024 * 1024, + used_durable_bytes: 0, + evidence_digest: [75; 32], + evidence_reference: "fixture/directory".into(), + }; + for (offset, action) in + [FleetCapacityAction::Qualify, FleetCapacityAction::Activate] + .into_iter() + .enumerate() + { + let cut = self.fleet.read(TIMEOUT).await?; + cluster.inspect(index, index, cut.revision()).await?; + self.fleet + .admit_node_capacity( + &cut, + &FleetCapacityRequest { + identity: request( + 30 + (index * 2 + offset) as u8, + cut.revision(), + ), + attestation: attestation.clone(), + action, + }, + service, + &self.directories, + TIMEOUT, + ) + .await?; + } + } + for (index, birth) in cluster.births.iter().enumerate() { + let namespace = &self.namespaces[index]; + let standby = self + .fleet + .observe_managed_standby( + &self.catalog, + namespace, + limits().directory, + TIMEOUT, + ) + .await?; + let cut = self.fleet.read(TIMEOUT).await?; + let identity = request(50 + index as u8 * 2, cut.revision()); + let reservation = self + .fleet + .reserve_managed_first_group( + &self.catalog, + namespace, + RaftManagedFirstGroupRequest { + reservation: identity.clone(), + standby, + publication: ProposalId::new([64 + index as u8; 16])?, + group: birth.clone(), + }, + limits().directory, + TIMEOUT, + ) + .await?; + let cut = self.fleet.read(TIMEOUT).await?; + let mut confirmation = request(51 + index as u8 * 2, cut.revision()); + confirmation.operation = identity.operation; + let group = self + .fleet + .publish_managed_first_group(&reservation, &confirmation, TIMEOUT) + .await?; + let cut = self.fleet.read(TIMEOUT).await?; + for voter in 0..VOTERS { + cluster.inspect(voter, voter, cut.revision()).await?; + cluster + .service(voter)? + .activate_managed_initial_group(group.clone(), TIMEOUT) + .await?; + } + } + Ok::<_, anyhow::Error>(()) + }) + .await? + } +} diff --git a/crates/cellar/tests/support/raft_directory_native.rs b/crates/cellar/tests/support/raft_directory_native.rs index a614fa00f..a447b8a4f 100644 --- a/crates/cellar/tests/support/raft_directory_native.rs +++ b/crates/cellar/tests/support/raft_directory_native.rs @@ -1,31 +1,78 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. -//! Shared advertised endpoints; process refresh never bypasses a fault proxy. +//! Shared advertised endpoints and process fences across two physical groups. use super::{public_node, store}; use anyhow::{ensure, Context as _, Result}; -use cellar::sdk::{consensus::*, Node, NodeRaftConfig, NodeRaftListeners, ShutdownMode}; +use cellar::sdk::{consensus::*, *}; use std::{ sync::{ atomic::{AtomicUsize, Ordering}, Arc, }, - time::Duration, + time::{Duration, Instant}, }; +#[path = "raft_directory_managed.rs"] +mod managed; +use managed::Managed; #[path = "raft_peer_inspection_native.rs"] mod inspection; +const TIMEOUT: Duration = Duration::from_secs(5); +const PREFIX: &str = "directory/native"; +const GROUPS: usize = 2; +const VOTERS: usize = 3; + struct Cluster { - inspection: bool, + managed: Option, roots: Vec, nodes: Vec>, services: Vec>, peers: Vec, - definitions: Vec, - registrations: Vec, + applications: Vec>, + namespaces: Vec, + births: Vec, store: Arc, calls: Vec>, - _entered: tokio::sync::mpsc::UnboundedReceiver<()>, +} + +fn catalog_options() -> CatalogOptions { + CatalogOptions { + prefix: PREFIX.into(), + ..Default::default() + } +} + +fn limits() -> CellGroupNodeLimits { + let mut limits = public_node::limits(); + limits.namespaces.max_namespaces = GROUPS; + limits +} + +fn counter(calls: Arc) -> Result { + Ok(RaftClass::new("Counter", 1, [21; 32])? + .initializer([24; 32], |_, sql| { + sql.execute_batch( + "CREATE TABLE counter(value INTEGER NOT NULL); INSERT INTO counter VALUES(0)", + )?; + Ok(()) + })? + .command("Increment", [22; 32], move |context, input| { + calls.fetch_add(1, Ordering::SeqCst); + let delta = i64::from_be_bytes(input.try_into()?); + context + .sql() + .execute("UPDATE counter SET value=value+?1", [delta])?; + let value: i64 = context + .sql() + .query_row("SELECT value FROM counter", [], |row| row.get(0))?; + Ok(value.to_be_bytes().to_vec()) + })? + .query("Get", [23; 32], |context, _| { + let value: i64 = + context.query_row("SELECT value FROM counter", [], |row| row.get(0))?; + Ok(value.to_be_bytes().to_vec()) + })?) } async fn listeners() -> Result { @@ -36,80 +83,108 @@ async fn listeners() -> Result { }) } +fn tls() -> Result { + Ok(RaftTls::new( + vec![include_bytes!("../../transport/tcp/fixtures/ca.der") + .to_vec() + .into()], + vec![include_bytes!("../../transport/tcp/fixtures/server.der") + .to_vec() + .into()], + rustls::pki_types::PrivatePkcs8KeyDer::from( + include_bytes!("../../transport/tcp/fixtures/key.der").to_vec(), + ) + .into(), + )?) +} + impl Cluster { async fn start() -> Result { - Self::start_with_inspection(false).await + Self::open(false).await } - async fn start_with_inspection(inspection: bool) -> Result { - let count = if inspection { 4 } else { 3 }; - let (gate, entered) = public_node::Gate::new(); - gate.release(); + async fn open(managed: bool) -> Result { + let count = VOTERS + usize::from(managed); let calls: Vec<_> = (0..count).map(|_| Arc::new(AtomicUsize::new(0))).collect(); let definitions = calls .iter() - .map(|calls| public_node::counter(calls.clone(), gate.clone())) + .map(|calls| counter(calls.clone())) .collect::>>()?; - let namespace = NamespaceConfig::new( - ClusterId::new([1; 16])?, - NamespaceId::new([2; 16])?, - NamespaceMode::Raft, - definitions[0].schema, - ); - let registrations = (0..2) - .map(|group| { - Ok(RaftCellRegistration { - placement: RaftCellPlacement::Replica, - namespace: namespace.clone(), - class: "Counter".into(), - birth: BirthProposal { - scope: GroupScope { - cluster: namespace.cluster, - namespace: namespace.namespace, - cell: format!("Counter:directory-{group}"), - group: GroupId::new([50 + group; 16])?, - incarnation: IncarnationId::new([3; 16])?, - }, - voters: (1..=3) - .map(|n| { - Ok(InitialMember { - member: MemberId::new(n)?, - node: NodeId::new([n as u8; 16])?, - }) - }) - .collect::>>()?, - learners: vec![], + let cluster_id = ClusterId::new([1; 16])?; + let namespaces = (0..GROUPS) + .map(|index| { + Ok(NamespaceConfig::new( + cluster_id, + NamespaceId::new([2 + index as u8; 16])?, + NamespaceMode::Raft, + definitions[0].schema_digest(), + )) + }) + .collect::>>()?; + let births = namespaces + .iter() + .enumerate() + .map(|(index, namespace)| { + Ok(CellGroupBirthSpec { + history: GroupHistory { + cluster: namespace.cluster, + namespace: namespace.namespace, + group: GroupId::new([50 + index as u8; 16])?, + incarnation: IncarnationId::new([3; 16])?, }, + range: CellRange::all(), + epoch: RouteEpoch::new(1)?, + creation: ProposalId::new([70 + index as u8; 16])?, + voters: (1..=VOTERS) + .map(|id| { + Ok(InitialMember { + member: MemberId::new(id as u64)?, + node: NodeId::new([id as u8; 16])?, + }) + }) + .collect::>()?, + learners: vec![], }) }) .collect::>>()?; + let store = Arc::new(store::Store::default()); + let managed = if managed { + Some(Managed::open(store.clone(), &namespaces, &definitions[0]).await?) + } else { + None + }; + let applications = definitions + .into_iter() + .map(|class| { + Ok(Arc::new( + Registry::new(|_, _| Box::pin(async { Ok(Output::Done) })).raft_class(class)?, + ) as Arc) + }) + .collect::>>()?; let mut cluster = Self { - inspection, + managed, roots: vec![], nodes: vec![], services: (0..count).map(|_| None).collect(), peers: vec![], - definitions, - registrations, - store: Arc::new(store::Store::default()), + applications, + namespaces, + births, + store, calls, - _entered: entered, }; - let mut bound = vec![]; - for i in 0..count { + let mut bound = Vec::new(); + for index in 0..count { let root = tempfile::tempdir()?; - let label = format!("directory-native-{}", i + 1); - let node = public_node::node_named( - root.path(), - cluster.definitions[i].application.clone(), - &label, - ) - .await?; + let label = format!("directory-native-{}", index + 1); + let node = + public_node::node_named(root.path(), cluster.applications[index].clone(), &label) + .await?; let listeners = listeners().await?; cluster.peers.push(RaftPeerNode { - node: NodeId::new([i as u8 + 1; 16])?, - disk: DiskLineage::new([i as u8 + 20; 16])?, - process: ProcessGeneration::new([i as u8 + 30; 16])?, + node: NodeId::new([index as u8 + 1; 16])?, + disk: DiskLineage::new([index as u8 + 20; 16])?, + process: ProcessGeneration::new([index as u8 + 30; 16])?, label, server_name: "localhost".into(), control_address: listeners.control.local_addr()?, @@ -120,8 +195,27 @@ impl Cluster { cluster.nodes.push(Some(node)); bound.push(listeners); } - for (i, listeners) in bound.into_iter().enumerate() { - cluster.start_service(i, listeners, false).await?; + if let Some(managed) = &cluster.managed { + managed.enroll(&cluster.peers).await?; + } + for (index, bound) in bound.into_iter().enumerate() { + cluster.start_service(index, bound, false).await?; + } + if let Some(managed) = &cluster.managed { + Box::pin(managed.attach(&cluster)).await?; + } + for index in 0..GROUPS { + let leader = cluster.leader(index).await?; + cluster + .cell(leader, index)? + .enroll( + CellEnrollmentRequest { + creation: ProposalId::new([80 + index as u8; 16])?, + durability: Durability::Quorum, + }, + TIMEOUT, + ) + .await?; } Ok(cluster) } @@ -134,93 +228,123 @@ impl Cluster { ) -> Result<()> { let node = self.nodes[index].as_ref().context("missing node")?; let peer = &self.peers[index]; + let managed = if let Some(fleet) = &self.managed { + Some( + fleet + .domain + .run(async { + let cut = fleet.fleet.read(TIMEOUT).await?; + let authorization = + fleet.fleet.authorize_host(&cut, peer.node, TIMEOUT).await?; + Ok::<_, anyhow::Error>(RaftManagedHostOptions { + authorization, + namespaces: fleet.namespaces.clone(), + }) + }) + .await??, + ) + } else { + None + }; + let namespaces = self + .namespaces + .iter() + .enumerate() + .map(|(position, namespace)| { + let namespace = managed + .as_ref() + .map(|mode| mode.namespaces[position].namespace.clone()) + .unwrap_or_else(|| namespace.clone()); + let initialize = if managed.is_none() && index == 0 && !resume { + Some(RaftNamespaceInitialization { + operation: ProposalId::new([60 + position as u8; 16])?, + groups: vec![self.births[position].clone()], + }) + } else { + None + }; + Ok(RaftNamespaceConfig { + namespace, + class: "Counter".into(), + initialize, + }) + }) + .collect::>()?; + let groups = if index < VOTERS && (managed.is_none() || resume) { + self.births + .iter() + .map(|birth| RaftGroupAssignment { + history: birth.history, + placement: RaftGroupPlacement::Replica, + }) + .collect() + } else { + vec![] + }; + let mut log = ReplicaLogOptions::new( + node.info().data_dir.join("raft"), + peer.node, + peer.disk, + peer.process, + if resume { + LogOpenMode::Resume + } else { + LogOpenMode::Create + }, + ); + log.max_groups = GROUPS; + let storage = RaftHostOptions::default().images.max_bytes; + let shared = limits(); self.services[index] = Some( node.start_raft_cells( RaftCellsConfig { infrastructure: NodeRaftConfig { - log: ReplicaLogOptions::new( - node.info().data_dir.join("raft"), - peer.node, - peer.disk, - peer.process, - if resume { - LogOpenMode::Resume - } else { - LogOpenMode::Create - }, - ), + log, consensus: RaftRuntimeOptions { tick_interval_ms: Some(20), + max_groups: GROUPS, ..Default::default() }, transport: TransportOptions::default(), }, - catalog: CatalogOptions { - prefix: "directory/native".into(), - ..Default::default() + catalog: catalog_options(), + namespaces, + groups, + peers: if managed.is_some() { + vec![peer.clone()] + } else { + self.peers.clone() }, - cells: self - .registrations - .iter() - .cloned() - .map(|mut registration| { - if index >= 3 { - registration.placement = RaftCellPlacement::Standby; - } - registration - }) - .collect(), - peers: self.peers.clone(), - // One read grant deliberately lacks a matching inspection - // grant; successful inspection elsewhere has no read grant. - query_origins: if self.inspection && index == 0 { + query_origins: if managed.is_none() && index == 0 { vec![NamespaceQueryOrigin { - cluster: self.registrations[0].namespace.cluster, - namespace: self.registrations[0].namespace.namespace, + cluster: self.namespaces[0].cluster, + namespace: self.namespaces[0].namespace, revision: 1, peer: self.peers[2].clone(), }] } else { vec![] }, + command_origins: vec![], host: RaftHostOptions { + managed, + max_groups: GROUPS, + max_input_bytes: shared.dispatch.max_input_bytes, inspection_scan: RaftInspectionScanOptions { interval_ms: 20, groups_per_batch: 1, ..Default::default() }, - inspection_origins: if self.inspection { - self.peers - .iter() - .enumerate() - .filter(|(source, _)| { - *source != index && !(index == 0 && *source == 2) - }) - .map(|(_, peer)| NamespaceInspectionOrigin { - cluster: self.registrations[0].namespace.cluster, - namespace: self.registrations[0].namespace.namespace, - revision: 1, - peer: peer.clone(), - }) - .collect() - } else { - vec![] - }, - node_profile: if self.inspection { - let reservation = RaftHostOptions::default().images.max_bytes; - Some(RaftNodeProfile { - pool: "native".into(), - region: "local".into(), - zone: format!("zone-{index}"), - group_capacity: 2, - byte_capacity: reservation * 2, - bytes_per_group: reservation, - leader_capacity: 2, - draining: false, - }) - } else { - None - }, + node_profile: Some(RaftNodeProfile { + pool: "native".into(), + region: "local".into(), + zone: format!("zone-{index}"), + group_capacity: GROUPS as u32, + byte_capacity: storage * GROUPS as u64, + bytes_per_group: storage, + leader_capacity: GROUPS as u32, + draining: false, + }), node_directory: Some(RaftNodeDirectoryOptions { poll_ms: 20, timeout_ms: 5000, @@ -228,13 +352,19 @@ impl Cluster { }), ..Default::default() }, + shared, }, RaftCellsServices { + recovery: None, listeners, - tls: public_node::tls()?, + tls: tls()?, store: self.store.clone(), - application: self.definitions[index].application.clone(), - versions: public_node::version_keys()?, + application: self.applications[index].clone(), + versions: VersionKeys::new( + VersionSigningKey::new([31; 32], u64::MAX)?, + vec![], + Duration::from_secs(86400), + )?, }, ) .await?, @@ -242,24 +372,63 @@ impl Cluster { Ok(()) } + fn service(&self, index: usize) -> Result<&RaftCells> { + self.services[index].as_ref().context("missing service") + } + fn cell(&self, index: usize, group: usize) -> Result { - let registration = &self.registrations[group]; - Ok(self.services[index] - .as_ref() - .context("missing service")? - .cell( - registration.namespace.namespace, - ®istration.birth.scope.cell, - )?) + let history = self.births[group].history; + Ok(self.service(index)?.cell(CellHistory { + cluster: history.cluster, + namespace: history.namespace, + cell: CellId::new(format!("Counter:directory-{group}"))?, + incarnation: IncarnationId::new([4; 16])?, + })?) + } + + async fn inspect( + &self, + source: usize, + target: usize, + revision: u64, + ) -> Result { + let service = self.service(source)?; + let peer = &self.peers[target]; + let deadline = Instant::now() + TIMEOUT; + loop { + let remaining = deadline.saturating_duration_since(Instant::now()); + let result = service + .inspect_peer_physical_inventory( + peer.node, + revision, + remaining.min(Duration::from_millis(500)), + ) + .await; + match result { + Ok(proof) if proof.view.node.identity.process == peer.process => return Ok(proof), + _ if Instant::now() >= deadline => anyhow::bail!( + "physical inspection {source} -> {target} at {revision}: {result:?}" + ), + _ => tokio::time::sleep(Duration::from_millis(10)).await, + } + } } async fn leader(&self, group: usize) -> Result { tokio::time::timeout(Duration::from_secs(10), async { loop { - for i in 0..self.services.len() { - if let Ok(cell) = self.cell(i, group) { - if cell.status().await.is_ok_and(|s| s.command_ready) { - return Ok::<_, anyhow::Error>(i); + for index in 0..VOTERS { + if let Ok(service) = self.service(index) { + if !service.is_closed() + && service + .group(self.births[group].history)? + .status() + .await + .is_ok_and(|status| { + status.ready && status.leader == Some(index as u64 + 1) + }) + { + return Ok::<_, anyhow::Error>(index); } } } @@ -269,26 +438,41 @@ impl Cluster { .await? } - async fn write(&self, group: usize, id: u8, value: i64) -> Result { - let mut options = public_node::options(u64::from(id))?; - let deadline = tokio::time::Instant::now() + options.timeout; + async fn write(&self, group: usize, sequence: u8, value: i64) -> Result { + let retry = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH)? + .as_millis() as u64 + + 60_000; + let mut options = CommandOptions::new( + CommandId::new(ClientId::new([7; 16])?, u64::from(sequence))?, + retry, + ); + let deadline = Instant::now() + options.timeout; loop { let primary = self.leader(group).await?; - options.timeout = deadline.saturating_duration_since(tokio::time::Instant::now()); + options.timeout = deadline.saturating_duration_since(Instant::now()); ensure!( !options.timeout.is_zero(), "directory fixture command deadline" ); - let result = self - .cell(primary, group)? + let cell = self.cell(primary, group)?; + match cell .command( - self.definitions[primary].increment.clone(), + &cell.contract("Increment")?, value.to_be_bytes().to_vec(), options.clone(), ) - .await; - match result { - Err(CommandFailure::Overloaded | CommandFailure::NotLeader { .. }) => { + .await + { + Err(error) + if !error.ambiguous + && matches!( + *error.cause, + CellDispatchError::Runtime( + RuntimeError::Overloaded | RuntimeError::NotLeader + ) + ) => + { tokio::time::sleep(Duration::from_millis(5)).await } result => return Ok(result?), @@ -296,20 +480,43 @@ impl Cluster { } } + async fn read( + &self, + index: usize, + group: usize, + receipt: &CommandReceipt, + ) -> Result { + let cell = self.cell(index, group)?; + Ok(cell + .query( + ProposalId::new([180 + (index * GROUPS + group) as u8; 16])?, + &cell.contract("Get")?, + vec![], + QueryOptions { + consistency: QueryConsistency::AtLeast(receipt.version_token.clone()), + ..Default::default() + }, + ) + .await?) + } + + async fn stop_node(&mut self, index: usize) -> Result<()> { + let service = self.services[index].take().context("missing service")?; + let node = self.nodes[index].take().context("missing node")?; + ensure!(matches!( + node.shutdown(ShutdownMode::Preserve).await?, + NodeOutcome::Stopped(_) + )); + ensure!( + service.close().await?.mutation_outcomes + == MutationOutcomes::NoUnresolvedLocalProducers + ); + Ok(()) + } + async fn restart(&mut self, index: usize) -> Result<()> { let bound = listeners().await?; - ensure!(self.services[index] - .take() - .context("missing service")? - .close() - .await? - .unfinished - .is_empty()); - self.nodes[index] - .take() - .context("missing node")? - .shutdown(ShutdownMode::Preserve) - .await?; + self.stop_node(index).await?; let peer = &mut self.peers[index]; peer.process = ProcessGeneration::new([index as u8 + 70; 16])?; peer.control_address = bound.control.local_addr()?; @@ -318,7 +525,7 @@ impl Cluster { self.nodes[index] = Some( public_node::node_named( self.roots[index].path(), - self.definitions[index].application.clone(), + self.applications[index].clone(), &peer.label, ) .await?, @@ -327,11 +534,11 @@ impl Cluster { } async fn close(mut self) -> Result<()> { - for service in self.services.drain(..).flatten() { - ensure!(service.close().await?.unfinished.is_empty()); + for index in 0..self.services.len() { + self.stop_node(index).await?; } - for node in self.nodes.drain(..).flatten() { - node.shutdown(ShutdownMode::Preserve).await?; + if let Some(managed) = &self.managed { + managed.domain.shutdown().await; } Ok(()) } @@ -347,11 +554,12 @@ async fn native_timer_refreshes_restarted_process_across_cell_groups() -> Result ); } let target = (cluster.leader(0).await? + 1) % 3; - let retired = cluster.cell(target, 0)?; + let retired = cluster.service(target)?.clone(); let previous = cluster.peers[target].clone(); cluster.restart(target).await?; + ensure!(retired.is_closed()); ensure!(matches!( - retired.status().await, + retired.observe_node(TIMEOUT).await, Err(CommandFailure::Unavailable) )); assert_ne!(cluster.peers[target].process, previous.process); @@ -369,17 +577,7 @@ async fn native_timer_refreshes_restarted_process_across_cell_groups() -> Result let receipt = cluster.write(group, 2, 2).await?; assert_eq!(receipt.result, 7_i64.to_be_bytes()); for reader in 0..3 { - let read = cluster - .cell(reader, group)? - .query( - cluster.definitions[reader].get.clone(), - vec![], - QueryOptions { - consistency: QueryConsistency::AtLeast(receipt.version_token.clone()), - ..Default::default() - }, - ) - .await?; + let read = cluster.read(reader, group, &receipt).await?; assert_eq!(read.result, 7_i64.to_be_bytes()); } } @@ -398,6 +596,15 @@ async fn native_timer_refreshes_restarted_process_across_cell_groups() -> Result async fn native_directory_outage_preserves_cached_quorum_routes() -> Result<()> { let cluster = Cluster::start().await?; assert_eq!(cluster.write(0, 1, 1).await?.result, 1_i64.to_be_bytes()); + // Node zero accepts namespace read grants from node two. That grant is + // insufficient for the Fleet physical-inspection protocol. + ensure!(matches!( + cluster + .service(2)? + .inspect_peer_physical_inventory(cluster.peers[0].node, 1, TIMEOUT) + .await, + Err(InspectionError::Refused) + )); cluster .store .fail_directory_reads @@ -427,10 +634,14 @@ async fn native_directory_shutdown_joins_paused_metadata() -> Result<()> { } }) .await?; - tokio::time::timeout(Duration::from_secs(5), cluster.close()).await??; - assert_eq!(store.active_view_reads.load(Ordering::Acquire), 0); + // The directory joins an entered read before returning from shutdown. + let mut closing = Box::pin(cluster.close()); + ensure!(futures_util::poll!(closing.as_mut()).is_pending()); + ensure!(store.active_view_reads.load(Ordering::Acquire) > 0); store.hold_view_reads.store(false, Ordering::Release); store.view_reads_resume.notify_waiters(); + tokio::time::timeout(Duration::from_secs(5), closing).await??; + assert_eq!(store.active_view_reads.load(Ordering::Acquire), 0); Ok(()) } @@ -468,7 +679,6 @@ async fn native_self_view_ignores_old_revision_and_joins_superseded_host() -> Re .as_ref() .context("missing service")? .clone(); - let retained = cluster.cell(0, 0)?; let key = object_store::path::Path::from(format!( "directory/native/raft/{}/nodes/{}/view", hex::encode([1; 16]), @@ -503,10 +713,12 @@ async fn native_self_view_ignores_old_revision_and_joins_superseded_host() -> Re }) .await?; ensure!(matches!( - retained.status().await, + service.observe_node(TIMEOUT).await, Err(CommandFailure::Unavailable) )); - ensure!(service.close().await?.unfinished.is_empty()); + ensure!( + service.close().await?.mutation_outcomes == MutationOutcomes::NoUnresolvedLocalProducers + ); assert_eq!(cluster.write(0, 1, 1).await?.result, 1_i64.to_be_bytes()); cluster.close().await } diff --git a/crates/cellar/tests/support/raft_disaster.rs b/crates/cellar/tests/support/raft_disaster.rs index 709e500cf..78b174511 100644 --- a/crates/cellar/tests/support/raft_disaster.rs +++ b/crates/cellar/tests/support/raft_disaster.rs @@ -550,6 +550,7 @@ async fn fresh_cells( host: options.clone(), }, RaftCellsServices { + recovery: None, listeners: NodeRaftListeners { control: tokio::net::TcpListener::bind("127.0.0.1:0").await?, data: tokio::net::TcpListener::bind("127.0.0.1:0").await?, diff --git a/crates/cellar/tests/support/raft_docker_binding.rs b/crates/cellar/tests/support/raft_docker_binding.rs index 67f9f7439..9b00f78f4 100644 --- a/crates/cellar/tests/support/raft_docker_binding.rs +++ b/crates/cellar/tests/support/raft_docker_binding.rs @@ -4,6 +4,7 @@ //! provisioning boundary. This does not claim container shutdown evidence. use super::*; use anyhow::Context; +use cellar::application::consensus::verification::cell_groups::RangeDirectory; use cellar::docker::{ContainerId, Docker}; use object_store::ObjectStore; use serde_json::json; @@ -36,7 +37,10 @@ async fn binding(local_observer: bool) -> Result<()> { catalog.clone(), Docker::new(&socket), "native-binding", - vec![capacity_tests::namespace(&f).await?], + vec![RangeDirectory::new( + capacity_tests::namespace(&f).await?, + Default::default(), + )?], )?; let node = NodeId::new(boot.node)?; let container = ContainerId::new(id)?; @@ -107,7 +111,10 @@ async fn binding(local_observer: bool) -> Result<()> { Fixture::open_catalog(&f.domain, f.store.clone())?, Docker::new(&socket), "native-binding", - vec![capacity_tests::namespace(&f).await?], + vec![RangeDirectory::new( + capacity_tests::namespace(&f).await?, + Default::default(), + )?], )?; reopened .bind_node(node, &container, TIMEOUT) diff --git a/crates/cellar/tests/support/raft_enrollment_native.rs b/crates/cellar/tests/support/raft_enrollment_native.rs index 28331c26f..ccd9cb6d3 100644 --- a/crates/cellar/tests/support/raft_enrollment_native.rs +++ b/crates/cellar/tests/support/raft_enrollment_native.rs @@ -1022,6 +1022,7 @@ impl Fixture { host, }, RaftCellsServices { + recovery: None, listeners, tls: public_node::tls()?, store: self.store.clone(), diff --git a/crates/cellar/tests/support/raft_first_group_operator_cli.rs b/crates/cellar/tests/support/raft_first_group_operator_cli.rs new file mode 100644 index 000000000..df7917699 --- /dev/null +++ b/crates/cellar/tests/support/raft_first_group_operator_cli.rs @@ -0,0 +1,228 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! A fixture admin with the same routed control and CLI as an operator. +use anyhow::{ensure, Context as _, Result}; +use cellar::{ + asyncrt::TaskHandle, + bucket::Bucket, + sdk::{consensus::RaftManagedFirstGroupRequest, fleet_operations::*, ExecutionDomain}, +}; +use cellar_admin::{AuditStore, Console, Fleet, FleetInfo, Permission, Principal}; +use serde_json::Value; +use std::time::Duration; +use std::{collections::BTreeMap, path::Path, sync::Arc}; +use tokio::sync::oneshot; + +pub const FLEET: &str = "first-group-fixture"; +pub const OWNER: &str = "fixture-first-group-owner-credential-000"; +pub const READER: &str = "fixture-first-group-reader-credential-000"; + +pub struct OperatorCli { + pub url: String, + pub catalog: String, + stop: Option>, + joined: Option>>, +} + +impl OperatorCli { + pub async fn start( + domain: &Arc, + bucket: Bucket, + control: RaftFleetControl, + root: &Path, + owner_id: &str, + ) -> Result { + let catalog = control.catalog_id().to_owned(); + let router = Self::router(domain, bucket, control, root, owner_id)?; + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await?; + let url = format!("http://{}", listener.local_addr()?); + let (stop, stopped) = oneshot::channel(); + let server = domain.spawn(async move { + axum::serve(listener, router) + .with_graceful_shutdown(async { + let _ = stopped.await; + }) + .await + })?; + Ok(Self { + url, + catalog, + stop: Some(stop), + joined: Some(server), + }) + } + + pub fn connect(url: String, catalog: String) -> Self { + Self { + url, + catalog, + stop: None, + joined: None, + } + } + + pub fn router( + domain: &Arc, + bucket: Bucket, + control: RaftFleetControl, + root: &Path, + owner_id: &str, + ) -> Result { + let principal = |id: &str, token: &str, grants| { + Principal::new(id.into(), token, BTreeMap::from([(FLEET.into(), grants)])) + }; + Ok(cellar_admin::router(Console { + domain: domain.clone(), + fleets: BTreeMap::from([( + FLEET.into(), + Fleet { + info: FleetInfo { + id: FLEET.into(), + name: FLEET.into(), + environment: "fixture".into(), + }, + client: cellar::management::FleetClient::new(bucket, false) + .with_raft_fleet(control), + }, + )]), + principals: vec![ + principal( + owner_id, + OWNER, + vec![ + Permission::MetadataRead, + Permission::FleetCapacity, + Permission::FleetRegister, + Permission::FleetTopology, + Permission::FleetMaintenance, + Permission::FleetStop, + ], + )?, + principal("reader", READER, vec![Permission::MetadataRead])?, + ], + audit: AuditStore::open(&root.join("first-group-audit.sqlite"))?, + assets: root.join("assets"), + origins: vec![], + })?) + } + + pub async fn call(&self, args: &[&str], token: &str) -> Result { + self.call_with_timeout(args, token, Duration::from_secs(5)) + .await + } + + pub async fn call_with_timeout( + &self, + args: &[&str], + token: &str, + timeout: Duration, + ) -> Result { + let executable = std::env::var_os("CELLAR_RAFT_OPERATOR_CLI") + .unwrap_or_else(|| env!("CARGO_BIN_EXE_cellar").into()); + let mut command = tokio::process::Command::new(executable); + command + .args(["raft", "fleet"]) + .args(args) + .args([ + "--fleet", + FLEET, + "--admin-url", + &self.url, + "--catalog-id", + &self.catalog, + "--json", + ]) + .env_remove("AWS_ACCESS_KEY_ID") + .env_remove("AWS_SECRET_ACCESS_KEY") + .env_remove("AWS_SESSION_TOKEN") + .env("CELLAR_ADMIN_TOKEN", token) + .kill_on_drop(true); + let output = tokio::time::timeout(timeout, command.output()) + .await + .with_context(|| { + format!( + "managed CLI invocation deadline for {:?}", + &args[..args.len().min(2)] + ) + })??; + for bytes in [&output.stdout, &output.stderr] { + ensure!(!String::from_utf8_lossy(bytes).contains(token)); + } + let value = + serde_json::from_slice(&output.stdout).context("first group CLI JSON response")?; + Ok(CliReply { + code: output.status.code().context("CLI exited by signal")?, + value, + }) + } + + pub async fn revision(&self) -> Result { + let status: RaftFleetStatus = + serde_json::from_value(self.call(&["status"], OWNER).await?.success()?)?; + Ok(status.observed_revision) + } + + pub async fn close(mut self) -> Result<()> { + self.signal_stop(); + if let Some(joined) = self.joined.take() { + joined.await??; + } + Ok(()) + } + + fn signal_stop(&mut self) { + if let Some(stop) = self.stop.take() { + let _ = stop.send(()); + } + } +} + +impl Drop for OperatorCli { + fn drop(&mut self) { + self.signal_stop(); + } +} + +pub struct CliReply { + pub code: i32, + pub value: Value, +} + +impl CliReply { + pub fn success(self) -> Result { + ensure!(self.code == 0, "managed CLI refused: {}", self.value); + Ok(self.value) + } +} + +pub fn first_group_plan( + catalog: &str, + setup: RaftManagedFirstGroupRequest, +) -> Result { + let history = setup.group.history; + let voters = setup + .group + .voters + .into_iter() + .map(|member| RaftFleetFirstGroupVoter { + node_id: hex::encode(member.node.bytes()), + member_id: member.member.get(), + }) + .collect::>() + .try_into() + .ok() + .context("expected RF3")?; + Ok(RaftFleetFirstGroupPlan { + catalog_id: catalog.into(), + namespace_id: hex::encode(history.namespace.bytes()), + standby: hex::encode(setup.standby), + request_id: hex::encode(setup.reservation.id), + operation_id: hex::encode(setup.reservation.operation), + expected_revision: setup.reservation.expected_revision, + publication_id: hex::encode(setup.publication.bytes()), + group_id: hex::encode(history.group.bytes()), + incarnation_id: hex::encode(history.incarnation.bytes()), + creation_id: hex::encode(setup.group.creation.bytes()), + voters, + }) +} diff --git a/crates/cellar/tests/support/raft_fleet_inventory.rs b/crates/cellar/tests/support/raft_fleet_inventory.rs index 9147e21c1..b1a673542 100644 --- a/crates/cellar/tests/support/raft_fleet_inventory.rs +++ b/crates/cellar/tests/support/raft_fleet_inventory.rs @@ -90,65 +90,6 @@ async fn fleet_inventory_namespace_coverage() -> Result<()> { f.close().await } -#[tokio::test(flavor = "multi_thread", worker_threads = 4)] -async fn fleet_inventory_registry_retries_and_scope() -> Result<()> { - let mut f = ImportFixture::new(false).await?; - let other = another_namespace(&f, 88).await?; - let root = f.f.catalog.read(TIMEOUT).await?; - let request = f.request(root.revision()); - f.store.fail_fleet_after.store(true, Ordering::Release); - f.store - .fail_fleet_confirmation - .store(true, Ordering::Release); - ensure!(f - .f - .catalog - .declare_namespace(&root, &request, &other, "Counter", TIMEOUT) - .await - .is_err()); - f.store.fail_fleet_after.store(false, Ordering::Release); - f.store - .fail_fleet_confirmation - .store(false, Ordering::Release); - let third = another_namespace(&f, 89).await?; - declare(&mut f, &third).await?; - let reopened = Fixture::open_catalog(&f.f.domain, f.store.clone())?; - let now = reopened.read(TIMEOUT).await?; - let reply = reopened - .declare_namespace(&now, &request, &other, "Counter", TIMEOUT) - .await?; - ensure!(reply.applied_revision < now.revision() && reply.snapshot.revision() == now.revision()); - ensure!(matches!( - reopened - .declare_namespace(&now, &request, &other, "Different", TIMEOUT) - .await, - Err(FleetCatalogError::PayloadConflict) - )); - let conflict = f.request(now.revision()); - ensure!(matches!( - reopened - .declare_namespace(&now, &conflict, &other, "Different", TIMEOUT) - .await, - Err(FleetCatalogError::Conflict) - )); - let wrong = Catalog::new( - f.store.clone(), - CatalogOptions { - prefix: "foreign-data/".into(), - ..Default::default() - }, - )? - .open_namespace(other.config().clone()) - .await?; - ensure!(matches!( - reopened - .declare_namespace(&now, &conflict, &wrong, "Counter", TIMEOUT) - .await, - Err(FleetCatalogError::Conflict) - )); - f.close().await -} - #[tokio::test(flavor = "multi_thread", worker_threads = 4)] async fn fleet_inventory_retains_offline_and_missing_births() -> Result<()> { let mut f = ImportFixture::new(false).await?; @@ -226,72 +167,3 @@ async fn fleet_inventory_namespace_bound() -> Result<()> { )); f.close().await } - -#[tokio::test(flavor = "multi_thread", worker_threads = 4)] -async fn fleet_inventory_registry_no_reset() -> Result<()> { - let mut f = ImportFixture::new(false).await?; - let root = f.f.catalog.read(TIMEOUT).await?; - let original = - f.f.catalog - .get(&root, FleetTable::Policy, "compiled-namespaces", TIMEOUT) - .await? - .context("registry")?; - let request = f.request(root.revision()); - let deletion = FleetMutation { - id: request.id, - operation: request.operation, - principal: request.principal, - expected_revision: request.expected_revision, - changes: vec![FleetChange { - table: FleetTable::Policy, - key: "compiled-namespaces".into(), - value: None, - }], - }; - ensure!(matches!( - f.f.catalog.commit(&root, &deletion, TIMEOUT).await, - Err(FleetCatalogError::Invalid(_)) - )); - ensure!(f.f.catalog.read(TIMEOUT).await?.revision() == root.revision()); - let value: serde_json::Value = serde_json::from_slice(&original[13..original.len() - 32])?; - let index: Vec = serde_json::from_value(value["index"].clone())?; - let page = Path::from(format!( - "coordination/raft/{}/fleet/objects/{}", - hex::encode(f.namespace.config().cluster.bytes()), - hex::encode(index) - )); - let saved = f.store.inner.get(&page).await?.bytes().await?; - f.store.inner.delete(&page).await?; - let other = another_namespace(&f, 88).await?; - let request = f.request(root.revision()); - ensure!(matches!( - f.f.catalog - .declare_namespace(&root, &request, &other, "Counter", TIMEOUT) - .await, - Err(FleetCatalogError::Corrupt(_)) - )); - ensure!(f.f.catalog.read(TIMEOUT).await?.revision() == root.revision()); - f.store.inner.put(&page, saved.into()).await?; - f.f.catalog - .declare_namespace(&root, &request, &other, "Counter", TIMEOUT) - .await?; - let root = f.f.catalog.read(TIMEOUT).await?; - let request = f.request(root.revision()); - let rollback = FleetMutation { - id: request.id, - operation: request.operation, - principal: request.principal, - expected_revision: request.expected_revision, - changes: vec![FleetChange { - table: FleetTable::Policy, - key: "compiled-namespaces".into(), - value: Some(original), - }], - }; - ensure!(matches!( - f.f.catalog.commit(&root, &rollback, TIMEOUT).await, - Err(FleetCatalogError::Invalid(_)) - )); - ensure!(f.f.catalog.read(TIMEOUT).await?.revision() == root.revision()); - f.close().await -} diff --git a/crates/cellar/tests/support/raft_fleet_permits_shared.rs b/crates/cellar/tests/support/raft_fleet_permits_shared.rs new file mode 100644 index 000000000..d23440300 --- /dev/null +++ b/crates/cellar/tests/support/raft_fleet_permits_shared.rs @@ -0,0 +1,587 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Managed voter lifecycle and exact typed permit operations for the native suite. +use super::*; +use futures_util::TryStreamExt as _; +use object_store::ObjectStore as _; + +pub(super) const SPARE: usize = 3; +pub(super) const TIMEOUT: Duration = Duration::from_secs(10); +pub(super) const SHORT: Duration = Duration::from_millis(500); + +pub(super) struct Fixture { + pub(super) cluster: ManagedCluster, + pub(super) primary: usize, + nodes: Vec>, + calls: Vec>, + cell: CellHistory, + retry_until_ms: u64, +} + +pub(super) struct Reserved { + pub(super) group: FleetGroupAuthorization, + pub(super) permit: FleetPermitAuthorization, + pub(super) plan: RepairPlan, + pub(super) expected: FleetMembershipPermit, + execution: FleetMovementExecutionRequest, +} + +pub(super) async fn qualify( + rooted: bool, + check: impl AsyncFnOnce(&mut Fixture) -> Result<()>, +) -> Result<()> { + let mut fixture = Box::pin(Fixture::start(rooted)).await?; + let result = Box::pin(check(&mut fixture)).await; + let joined = fixture.close().await; + result.with_context(|| format!("joined permit fixture: {joined:?}"))?; + joined +} + +impl Fixture { + async fn start(rooted: bool) -> Result { + let calls = (0..=SPARE) + .map(|_| Arc::new(AtomicUsize::new(0))) + .collect::>(); + let classes = calls + .iter() + .map(|count| { + let count = count.clone(); + managed_fixture::counter_class_with_callback(move || { + count.fetch_add(1, Ordering::SeqCst); + }) + }) + .collect::>>()?; + let first = + managed_fixture::Fixture::with_class(rooted, managed_fixture::counter_class()?).await?; + let applications = classes + .into_iter() + .map(|class| { + Ok(Arc::new( + Registry::new(|_, _| Box::pin(async { Ok(Output::Done) })).raft_class(class)?, + ) as Arc) + }) + .collect::>>()?; + let mut cluster = Box::pin(ManagedCluster::start_with_applications( + first, + &[7], + ArchiveSetup::Initialized, + false, + Some(applications), + FirstGroupSetup::Normal, + )) + .await?; + let nodes = std::mem::take(&mut cluster.nodes) + .into_iter() + .map(Some) + .collect(); + let cell = CellHistory { + cluster: cluster.history.cluster, + namespace: cluster.history.namespace, + cell: CellId::new("Counter:fleet")?, + incarnation: IncarnationId::new([25; 16])?, + }; + let retry_until_ms = u64::try_from( + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH)? + .as_millis(), + )? + .checked_add(60_000) + .context("retry deadline")?; + let mut fixture = Self { + cluster, + nodes, + primary: 0, + calls, + cell, + retry_until_ms, + }; + let initialized = async { + fixture.primary = fixture.leader().await?.0; + fixture.cluster.hosts[fixture.primary] + .cell(fixture.cell.clone())? + .enroll( + CellEnrollmentRequest { + creation: ProposalId::new([181; 16])?, + durability: Durability::Quorum, + }, + TIMEOUT, + ) + .await?; + ensure!(fixture.write(1, 5, Durability::Quorum).await?.result == 5_i64.to_be_bytes()); + Ok::<_, anyhow::Error>(()) + } + .await; + if let Err(error) = initialized { + let joined = fixture.close().await; + return Err(error).context(format!("joined failed permit fixture: {joined:?}")); + } + Ok(fixture) + } + + pub(super) fn catalog(&self) -> &FleetCatalog { + &self.cluster.fixtures[0].catalog + } + + pub(super) fn group(&self, index: usize) -> Result { + Ok(self.cluster.hosts[index].group(self.cluster.history)?) + } + + pub(super) fn counts(&self) -> Vec { + self.calls + .iter() + .map(|count| count.load(Ordering::SeqCst)) + .collect() + } + + pub(super) async fn leader(&self) -> Result<(usize, Term)> { + Ok(tokio::time::timeout(TIMEOUT, async { + loop { + let mut term = 0; + let mut leader = None; + for (index, node) in self.nodes.iter().enumerate() { + if node.is_some() { + if let Ok(group) = self.group(index) { + if let Ok(status) = group.status().await { + if status.term.get() > term { + term = status.term.get(); + leader = None; + } + if status.term.get() == term + && status.ready + && status.leader == Some(index as u64 + 1) + { + leader = Some((index, status.term)); + } + } + } + } + } + if let Some(leader) = leader { + return leader; + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await?) + } + + async fn discover(&self) -> Result<()> { + let cut = self.catalog().read(TIMEOUT).await?; + for (source, host) in self.cluster.hosts.iter().enumerate() { + if self.nodes[source].is_none() { + continue; + } + for (target, fixture) in self.cluster.fixtures.iter().enumerate() { + if self.nodes[target].is_some() { + inspect_current(host, fixture.spec.node, cut.revision()).await?; + } + } + } + Ok(()) + } + + pub(super) async fn reserve(&mut self, id: u8) -> Result { + self.primary = self.leader().await?.0; + self.discover().await?; + let before = self.group(self.primary)?.quorum_membership(TIMEOUT).await?; + let previous = status(&before)?; + let sequence = previous + .consumed_sequence + .checked_add(1) + .context("permit sequence")?; + drop(before); + let (_, execution) = movement::admit_replacement( + &self.cluster, + movement::ReplacementTarget { + source: self.primary, + obsolete: (self.primary + 1) % SPARE, + spare: SPARE, + }, + [id, id + 1], + false, + ) + .await?; + self.discover().await?; + let cut = self.catalog().read(TIMEOUT).await?; + let moves = self.catalog().pending_movements(&cut, TIMEOUT).await?; + let selected = moves + .into_iter() + .find(|item| item.reservation == execution.reservation) + .context("admitted movement")?; + let group = self + .catalog() + .authorize_group(&cut, &self.cluster.history, TIMEOUT) + .await?; + let permit = self + .catalog() + .authorize_membership(&cut, &self.cluster.history, TIMEOUT) + .await?; + let expected = FleetMembershipPermit::new( + previous.binding, + sequence, + ProposalId::new(execution.reservation)?, + ProposalId::new(request(id, cut.revision()).operation)?, + &selected.plan, + )?; + Ok(Reserved { + group, + permit, + plan: selected.plan, + expected, + execution, + }) + } + + pub(super) async fn catalog_permit( + &self, + ) -> Result { + let cut = self.catalog().read(TIMEOUT).await?; + self.catalog() + .authorize_membership(&cut, &self.cluster.history, TIMEOUT) + .await + } + + pub(super) async fn ensure_pending(&self, reserved: &Reserved) -> Result<()> { + let cut = self.catalog().read(TIMEOUT).await?; + let pending = self.catalog().pending_reservations(&cut, TIMEOUT).await?; + ensure!( + pending.len() == 1 + && pending[0].reservation == reserved.execution.reservation + && pending[0].boot == reserved.execution.attestation.boot, + "active movement lost its exact debit: {pending:?}" + ); + Ok(()) + } + + pub(super) async fn ensure_spare_unclaimed(&self) -> Result<()> { + let history = self.cluster.history; + let claim = format!( + "/claims/{}/{}/0000000000000004", + hex::encode(history.group.bytes()), + hex::encode(history.incarnation.bytes()), + ); + let mut objects = self.cluster.fixtures[SPARE].store.list(None); + while let Some(object) = objects.try_next().await? { + ensure!( + !object.location.as_ref().ends_with(&claim), + "unguarded startup claimed a managed learner" + ); + } + Ok(()) + } + + pub(super) async fn settle( + &self, + reserved: &Reserved, + id: u8, + phase: FleetPermitPhase, + ) -> Result<()> { + self.discover().await?; + let cut = self.catalog().read(TIMEOUT).await?; + let completion = FleetMovementCompletionRequest { + identity: request(id, cut.revision()), + reservation: reserved.execution.reservation, + attestation: reserved.execution.attestation.clone(), + }; + let applied = match phase { + FleetPermitPhase::Fenced => self + .catalog() + .settle_fenced_movement( + &cut, + &completion, + &self.cluster.hosts[self.primary], + TIMEOUT, + ) + .await? + .applied_revision(), + FleetPermitPhase::Aborted => self + .catalog() + .settle_aborted_movement( + &cut, + &completion, + &self.cluster.hosts[self.primary], + TIMEOUT, + ) + .await? + .applied_revision(), + _ => anyhow::bail!("unexpected permit settlement {phase:?}"), + }; + let settled = self.catalog().read(TIMEOUT).await?; + ensure!( + applied == settled.revision() + && self + .catalog() + .pending_reservations(&settled, TIMEOUT) + .await? + .is_empty() + ); + Ok(()) + } + + pub(super) async fn write( + &self, + sequence: u64, + delta: i64, + durability: Durability, + ) -> Result { + let handle = self.cluster.hosts[self.primary].cell(self.cell.clone())?; + let mut options = CommandOptions::new( + CommandId::new(ClientId::new([7; 16])?, sequence)?, + self.retry_until_ms, + ); + options.durability = durability; + Ok(handle + .command( + &handle.contract("Add")?, + delta.to_be_bytes().to_vec(), + options, + ) + .await?) + } + + pub(super) async fn read( + &self, + index: usize, + consistency: QueryConsistency, + timeout: Duration, + ) -> Result { + let handle = self.cluster.hosts[index].cell(self.cell.clone())?; + Ok(handle + .query( + ProposalId::new([119; 16])?, + &handle.contract("Get")?, + vec![], + QueryOptions { + consistency, + timeout, + routing: QueryRouting::Local, + ..Default::default() + }, + ) + .await?) + } + + pub(super) async fn strong(&self, index: usize) -> Result { + Ok(i64::from_be_bytes( + self.read(index, QueryConsistency::Linearizable, TIMEOUT) + .await? + .result + .as_slice() + .try_into()?, + )) + } + + pub(super) async fn begin( + &self, + index: usize, + reserved: &Reserved, + ) -> Result { + Ok(self + .group(index)? + .begin_managed_repair( + reserved.group.clone(), + reserved.permit.clone(), + reserved.plan.clone(), + TIMEOUT, + ) + .await?) + } + + pub(super) async fn advance( + &mut self, + reserved: &Reserved, + step: RepairStep, + phase: FleetPermitPhase, + ) -> Result { + let deadline = Instant::now() + TIMEOUT; + loop { + self.primary = self.leader().await?.0; + let group = self.group(self.primary)?; + let expected = group.observe(TIMEOUT).await?.membership_revision; + let remaining = deadline.saturating_duration_since(Instant::now()); + ensure!(!remaining.is_zero(), "managed repair {step:?} deadline"); + match group + .advance_managed_repair( + reserved.group.clone(), + reserved.permit.clone(), + step, + expected, + remaining.min(Duration::from_secs(2)), + ) + .await + { + Ok(observed) => { + verify_phase(&observed, reserved, phase)?; + return Ok(status(&observed)?); + } + Err(CellGroupControlError::Refused( + RuntimeError::NotLeader | RuntimeError::NotReady | RuntimeError::Overloaded, + )) + | Err(CellGroupControlError::Unconfirmed( + RuntimeError::DeadlineExceeded | RuntimeError::NotReady, + )) => { + tokio::time::sleep(Duration::from_millis(10)).await; + } + Err(CellGroupControlError::Unconfirmed(RuntimeError::NotFound)) + if step == RepairStep::FinishJoint => + { + // A leader can apply its own removal before it returns + // the terminal proof. Observe the same permit on the + // successor, with the original invocation deadline. + tokio::time::sleep(Duration::from_millis(10)).await; + } + Err(error) => return Err(error).context(format!("managed repair {step:?}")), + } + } + } + + pub(super) async fn wait_phase( + &self, + index: usize, + reserved: &Reserved, + phase: FleetPermitPhase, + ) -> Result { + let group = self.group(index)?; + let deadline = Instant::now() + TIMEOUT; + loop { + let remaining = deadline.saturating_duration_since(Instant::now()); + ensure!( + !remaining.is_zero(), + "member {index} did not observe {phase:?}" + ); + if let Ok(proof) = group.quorum_membership(remaining.min(SHORT)).await { + if status(&proof)?.phase == Some(phase) { + verify_phase(&proof, reserved, phase)?; + return Ok(status(&proof)?); + } + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + } + + pub(super) fn activation(&self, source: usize, from_s3: bool) -> Result { + Ok(LearnerActivation { + member: MemberId::new(4)?, + source: MemberId::new(source as u64 + 1)?, + source_node: self.cluster.fixtures[source].spec.node, + from_s3, + timeout: TIMEOUT, + }) + } + + pub(super) async fn activate( + &self, + reserved: &Reserved, + source: usize, + from_s3: bool, + ) -> Result { + self.discover().await?; + Ok(self.cluster.hosts[SPARE] + .activate_managed_learner_group( + reserved.group.clone(), + reserved.permit.clone(), + self.activation(source, from_s3)?, + ) + .await?) + } + + pub(super) async fn stop(&mut self, index: usize) -> Result<()> { + let node = self.nodes[index] + .take() + .context("running permit fixture node")?; + ensure!(matches!( + node.shutdown(ShutdownMode::Preserve).await?, + NodeOutcome::Stopped(_) + )); + let report = self.cluster.hosts[index].close().await?; + ensure!(report.mutation_outcomes == MutationOutcomes::NoUnresolvedLocalProducers); + Ok(()) + } + + pub(super) async fn reopen(&mut self, index: usize, generation: u8) -> Result<()> { + ensure!( + self.nodes[index].is_none(), + "engine reopened before joining prior node" + ); + let fixture = &self.cluster.fixtures[index]; + let node = start_node( + self.cluster.roots[index].path(), + fixture.application.clone(), + &fixture.spec.label, + ) + .await?; + let started = async { + let (mut config, services) = fixture + .config(&node, LogOpenMode::Resume, generation) + .await?; + archives::configure(&mut config); + node.start_raft_cells(config, services).await + } + .await; + match started { + Ok(host) => { + self.nodes[index] = Some(node); + self.cluster.hosts[index] = host; + Ok(()) + } + Err(error) => { + let joined = node.shutdown(ShutdownMode::Preserve).await; + Err(error).context(format!("reopen member {}: {joined:?}", index + 1)) + } + } + } + + pub(super) async fn restart_voters(&mut self, generation: u8) -> Result<()> { + for index in 0..SPARE { + self.stop(index).await?; + } + for index in 0..SPARE { + self.reopen(index, generation + index as u8).await?; + } + let first = &mut self.cluster.fixtures[0]; + first.catalog = managed_fixture::Fixture::open_catalog( + &first.domain, + first.store.clone(), + self.cluster.history.cluster, + )?; + self.discover().await?; + self.primary = self.leader().await?.0; + Ok(()) + } + + async fn close(mut self) -> Result<()> { + let mut errors = Vec::new(); + for index in 0..self.nodes.len() { + if self.nodes[index].is_some() { + if let Err(error) = self.stop(index).await { + errors.push(format!("node {index}: {error:#}")); + } + } + } + self.cluster.fixtures[0].domain.shutdown().await; + ensure!(errors.is_empty(), "{errors:?}"); + Ok(()) + } +} + +pub(super) fn status(proof: &CellGroupQuorumMembership) -> Result { + proof + .membership() + .fleet_status() + .context("durable fleet guard") +} + +pub(super) fn verify_phase( + proof: &CellGroupQuorumMembership, + reserved: &Reserved, + phase: FleetPermitPhase, +) -> Result<()> { + let status = status(proof)?; + ensure!( + status.permit.as_ref() == Some(&reserved.expected) + && status.consumed_sequence == reserved.expected.sequence() + && status.phase == Some(phase), + "unexpected permit guard: {status:?}" + ); + Ok(()) +} diff --git a/crates/cellar/tests/support/raft_ltx_fault.rs b/crates/cellar/tests/support/raft_ltx_fault.rs index 2e896b525..c81aea2c4 100644 --- a/crates/cellar/tests/support/raft_ltx_fault.rs +++ b/crates/cellar/tests/support/raft_ltx_fault.rs @@ -27,6 +27,7 @@ pub struct Faults { pub directory: PathBuf, pub mode: AtomicU8, pub fail_reads: AtomicBool, + pub all_directory_files: AtomicBool, pub collision: AtomicBool, pub reads: AtomicUsize, pub image_creates: AtomicUsize, @@ -50,6 +51,7 @@ impl Faults { directory, mode: AtomicU8::new(0), fail_reads: AtomicBool::new(false), + all_directory_files: AtomicBool::new(false), collision: AtomicBool::new(false), reads: AtomicUsize::new(0), image_creates: AtomicUsize::new(0), @@ -136,6 +138,12 @@ impl Drop for File { impl FaultFileSystem { fn wrap(&self, path: &Path, inner: HostFile, image_output: bool) -> HostFile { + let image_output = image_output + || (self.faults.all_directory_files.load(Ordering::Acquire) + && path.parent() == Some(self.faults.directory.as_path()) + && path + .file_name() + .is_some_and(|name| name != "LOCK" && name != "images.lock")); HostFile::from_io(File { inner, path: path.into(), @@ -256,6 +264,10 @@ impl FileSystem for FaultFileSystem { DirectFileSystem.metadata(path) } + fn symlink_metadata(&self, path: &Path) -> io::Result { + DirectFileSystem.symlink_metadata(path) + } + fn create_read_write(&self, path: &Path) -> io::Result { let image_output = path.extension().is_some_and(|p| p == "stage") || path.extension().is_some_and(|p| p == "sqlite") diff --git a/crates/cellar/tests/support/raft_metrics_native.rs b/crates/cellar/tests/support/raft_metrics_native.rs index c8d5bd385..a0e28d3de 100644 --- a/crates/cellar/tests/support/raft_metrics_native.rs +++ b/crates/cellar/tests/support/raft_metrics_native.rs @@ -251,11 +251,12 @@ async fn cluster(groups: usize) -> Result> { let observed = service.performance(); ensure!(observed.available && !observed.saturated && observed.source.len() == 32); ensure!( - observed.operations.len() == 44 && observed.counters.len() == 42, + observed.operations.len() == 45 && observed.counters.len() == 42, "performance inventory changed: {} operations, {} counters", observed.operations.len(), observed.counters.len() ); ensure!(observed.operations.iter().any(|operation| operation.name == "engine_purge")); + ensure!(observed.operations.iter().any(|operation| operation.name == "transport_query_reply_write")); let snapshot_operations = observed.operations.iter().map(|operation| operation.name) .filter(|name| name.starts_with("snapshot_")).collect::>(); ensure!(snapshot_operations == BTreeSet::from(["snapshot_encode", "snapshot_encode_queue", "snapshot_seed_encode", "snapshot_checkpoint_encode", "snapshot_transfer_encode"])); @@ -307,7 +308,7 @@ async fn cluster(groups: usize) -> Result> { // Two gauges, 42 counters, and 29 samples per operation (five outcome // counters, 22 buckets, a sum and a count). This remains independent // of the observed inventory and of the group population. - ensure!(perf_ids.len() == 1320); + ensure!(perf_ids.len() == 1349); let expected = observed.prometheus().lines().filter(|line| !line.starts_with('#') && !line.is_empty()).map(|line| line.split_whitespace().next().unwrap().to_owned()).collect::>(); ensure!(perf_ids == expected && perf_ids.is_disjoint(&identities)); identities.extend(perf_ids); diff --git a/crates/cellar/tests/support/raft_node.rs b/crates/cellar/tests/support/raft_node.rs index 6fa8ada76..596039f4c 100644 --- a/crates/cellar/tests/support/raft_node.rs +++ b/crates/cellar/tests/support/raft_node.rs @@ -312,6 +312,7 @@ pub async fn cells_many( host, }, RaftCellsServices { + recovery: None, listeners: NodeRaftListeners { control: tokio::net::TcpListener::bind("127.0.0.1:0").await?, data: tokio::net::TcpListener::bind("127.0.0.1:0").await?, @@ -367,47 +368,9 @@ pub fn archived_options(sequence: u64) -> Result { Ok(options) } -pub struct Gate { - open: std::sync::Mutex, - changed: std::sync::Condvar, - entered: tokio::sync::mpsc::UnboundedSender<()>, -} - -impl Gate { - pub fn new() -> (Arc, tokio::sync::mpsc::UnboundedReceiver<()>) { - let (entered, receive) = tokio::sync::mpsc::unbounded_channel(); - ( - Arc::new(Self { - open: std::sync::Mutex::new(false), - changed: std::sync::Condvar::new(), - entered, - }), - receive, - ) - } - - pub fn wait(&self) -> Result<()> { - self.entered.send(())?; - let mut open = self.open.lock().unwrap_or_else(|e| e.into_inner()); - while !*open { - open = self.changed.wait(open).unwrap_or_else(|e| e.into_inner()); - } - Ok(()) - } - - pub fn release(&self) { - *self.open.lock().unwrap_or_else(|e| e.into_inner()) = true; - self.changed.notify_all(); - } -} - -pub struct Release(pub Arc); - -impl Drop for Release { - fn drop(&mut self) { - self.0.release(); - } -} +#[path = "raft_read_gate.rs"] +mod read_gate; +pub use read_gate::{Gate, Release}; pub struct Counter { pub application: Arc, @@ -425,7 +388,7 @@ pub fn counter_with_blobs( gate: Arc, blobs: bool, ) -> Result { - let observe = move |_: &GroupScope| { + let observe = move |_: &cellar_logic::cell_group::CellHistory| { calls.fetch_add(1, std::sync::atomic::Ordering::SeqCst); Ok(()) }; @@ -438,7 +401,7 @@ pub fn counter_with_blobs( /// Attribute private execution to its group without changing counter SQL or gates. pub fn counter_observed( - observe: impl Fn(&GroupScope) -> Result<()> + Send + Sync + 'static, + observe: impl Fn(&cellar_logic::cell_group::CellHistory) -> Result<()> + Send + Sync + 'static, gate: Arc, ) -> Result { counter_definition(observe, gate, false, None) @@ -455,7 +418,7 @@ pub fn counter_with_seed_payload( for block in 0_u64..8192 { bytes.extend_from_slice(&sha2::Sha256::digest(block.to_be_bytes())); } - let observe = move |_: &GroupScope| { + let observe = move |_: &cellar_logic::cell_group::CellHistory| { calls.fetch_add(1, std::sync::atomic::Ordering::SeqCst); Ok(()) }; @@ -463,7 +426,7 @@ pub fn counter_with_seed_payload( } fn counter_definition( - observe: impl Fn(&GroupScope) -> Result<()> + Send + Sync + 'static, + observe: impl Fn(&cellar_logic::cell_group::CellHistory) -> Result<()> + Send + Sync + 'static, gate: Arc, blobs: bool, padding: Option>, diff --git a/crates/cellar/tests/support/raft_outbox_loopback.rs b/crates/cellar/tests/support/raft_outbox_loopback.rs index 518b329af..f2032c4d2 100644 --- a/crates/cellar/tests/support/raft_outbox_loopback.rs +++ b/crates/cellar/tests/support/raft_outbox_loopback.rs @@ -45,7 +45,7 @@ pub async fn self_delivery() -> Result<()> { digest: [98; 32], }; let target = RaftEffectTarget::new( - scope.clone(), + cellar_logic::cell_group::CellHistory::try_from(&scope.clone())?, definition, contract.clone(), receiver.verification_key(), diff --git a/crates/cellar/tests/support/raft_outbox_native.rs b/crates/cellar/tests/support/raft_outbox_native.rs index 4f7459680..482175fbc 100644 --- a/crates/cellar/tests/support/raft_outbox_native.rs +++ b/crates/cellar/tests/support/raft_outbox_native.rs @@ -180,6 +180,7 @@ impl Cluster { }, }, RaftCellsServices { + recovery: None, listeners, tls: public_node::tls()?, store: cluster.store.clone(), @@ -452,7 +453,9 @@ impl Fixture { ) .await?; let target = RaftEffectTarget::new( - destination.registration.birth.scope.clone(), + cellar_logic::cell_group::CellHistory::try_from( + &destination.registration.birth.scope.clone(), + )?, destination.definitions[0].definition, destination.definitions[0].write.clone(), receiver_key.verification_key(), diff --git a/crates/cellar/tests/support/raft_peer_inspection_native.rs b/crates/cellar/tests/support/raft_peer_inspection_native.rs index 42cc41188..b2e1fe8ba 100644 --- a/crates/cellar/tests/support/raft_peer_inspection_native.rs +++ b/crates/cellar/tests/support/raft_peer_inspection_native.rs @@ -1,95 +1,59 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. -//! Physical RPCs exercise actual Node services, including an unclaimed standby. +//! Authenticated physical metadata for current voters and an empty standby. use super::*; -async fn observe( - cluster: &Cluster, - source: usize, - target: usize, - group: usize, -) -> Result { - let deadline = tokio::time::Instant::now() + Duration::from_secs(10); - loop { - let result = cluster - .cell(source, group)? - .inspect_peer(cluster.peers[target].node, Duration::from_secs(1)) - .await; - match result { - Ok(view) if view.node.identity.process == cluster.peers[target].process => { - return Ok(view) - } - Ok(_) if tokio::time::Instant::now() < deadline => { - tokio::time::sleep(Duration::from_millis(10)).await - } - Ok(_) => anyhow::bail!("inspection never observed the current process"), - Err(_) if tokio::time::Instant::now() < deadline => { - // During restart the caller may still hold the old process's - // directory view. Its stopped address really is unreachable; - // wait for independently timed discovery of the new endpoint. - tokio::time::sleep(Duration::from_millis(10)).await; - } - Err(error) => return Err(error.into()), - } - } +async fn revision(cluster: &Cluster) -> Result { + let managed = cluster + .managed + .as_ref() + .context("managed inspection fixture")?; + Ok(managed + .domain + .run(managed.fleet.read(TIMEOUT)) + .await?? + .revision()) } #[tokio::test(flavor = "multi_thread", worker_threads = 8)] async fn native_peer_inspection_binds_member_and_dormant_physical_state() -> Result<()> { - let cluster = Cluster::start_with_inspection(true).await?; + let cluster = Cluster::open(true).await?; let receipt = cluster.write(0, 1, 3).await?; - // No query-origin grant exists for source zero at node one. - let member = observe(&cluster, 0, 1, 0).await?; - assert_eq!(member.scope, cluster.registrations[0].birth.scope); - assert_eq!(member.node.identity.node, cluster.peers[1].node); - assert_eq!(member.node.identity.process, cluster.peers[1].process); - assert_eq!(member.node.groups, 2); - assert_eq!( - member.occupancy, - ReplicaOccupancy::Member(MemberId::new(2)?) - ); - ensure!(member.member.is_some()); - let dormant = observe(&cluster, 0, 3, 0).await?; - assert_eq!(dormant.occupancy, ReplicaOccupancy::Standby); - assert!(dormant.member.is_none()); - assert_eq!(dormant.node.groups, 0); - assert_eq!(dormant.node.reserved_bytes, 0); - // A dormant physical source carries no invented member identity either. - let from_standby = observe(&cluster, 3, 0, 0).await?; - assert_eq!( - from_standby.occupancy, - ReplicaOccupancy::Member(MemberId::new(1)?) - ); - assert_eq!( - cluster - .cell(3, 0)? - .inspect_peer(cluster.peers[3].node, Duration::from_secs(1)) - .await? - .occupancy, - ReplicaOccupancy::Standby - ); - // The independent read grant from node two to node zero cannot inspect. + let revision = revision(&cluster).await?; + // The Fleet lane authenticates exact boots without any namespace read grant. + let member = cluster.inspect(0, 1, revision).await?; + assert_eq!(member.view.node.identity.node, cluster.peers[1].node); + assert_eq!(member.view.node.identity.process, cluster.peers[1].process); + assert_eq!(member.view.node.groups, 2); + let inventory = member.inventory.context("complete member inventory")?; + assert_eq!(inventory.histories().len(), GROUPS); + for birth in &cluster.births { + let row = inventory + .group_history(&birth.history) + .context("registered group absent")?; + assert_eq!(row.member(), MemberId::new(2)?); + ensure!(row.in_configuration()); + } + for (source, target) in [(0, 3), (3, 3)] { + let dormant = cluster.inspect(source, target, revision).await?; + assert_eq!(dormant.view.node.groups, 0); + assert_eq!(dormant.view.node.reserved_bytes, 0); + let inventory = dormant.inventory.context("complete standby inventory")?; + ensure!(inventory.histories().is_empty() && inventory.initializing().is_none()); + } + let from_standby = cluster.inspect(3, 0, revision).await?; assert_eq!( - cluster - .cell(2, 0)? - .inspect_peer(cluster.peers[0].node, Duration::from_secs(1)) - .await, - Err(InspectionError::Refused) + from_standby + .inventory + .context("standby observation")? + .group_history(&cluster.births[0].history) + .context("member inventory")? + .member(), + MemberId::new(1)? ); - for reader in 0..3 { + for reader in 0..VOTERS { assert_eq!( - cluster - .cell(reader, 0)? - .query( - cluster.definitions[reader].get.clone(), - vec![], - QueryOptions { - consistency: QueryConsistency::AtLeast(receipt.version_token.clone()), - ..Default::default() - } - ) - .await? - .result, + cluster.read(reader, 0, &receipt).await?.result, 3_i64.to_be_bytes() ); } @@ -106,68 +70,90 @@ async fn native_peer_inspection_binds_member_and_dormant_physical_state() -> Res #[tokio::test(flavor = "multi_thread", worker_threads = 8)] async fn native_peer_inspection_refreshes_process_and_joins_retained_waiters() -> Result<()> { - let mut cluster = Cluster::start_with_inspection(true).await?; - let previous = observe(&cluster, 0, 1, 0).await?; + let mut cluster = Cluster::open(true).await?; + let previous = cluster.inspect(0, 1, revision(&cluster).await?).await?; cluster.restart(1).await?; - let current = observe(&cluster, 0, 1, 0).await?; - assert_ne!( - previous.node.identity.process, - current.node.identity.process - ); - assert_eq!(previous.node.identity.disk, current.node.identity.disk); - assert_eq!( - previous.node.storage_instance, - current.node.storage_instance - ); - let leader = cluster.leader(0).await?; - // Admission is synchronous; none of these futures needs to be polled for - // the node to own and then cancel/join its actual network operations. + let revision = revision(&cluster).await?; + let current = cluster.inspect(0, 1, revision).await?; + let old = previous.view.fleet_boot.context("original boot")?; + let new = current.view.fleet_boot.context("retained boot")?; + assert_ne!(old.process, new.process); + assert_ne!(old.boot, new.boot); + assert!(old.epoch < new.epoch); + assert_eq!(old.disk, new.disk); + assert_eq!(old.engine, new.engine); + assert_eq!(previous.inventory, current.inventory); + // Invocation admits the network work before the first response poll. Keep + // these waiters unpolled while a real business write completes independently. + let source = cluster.service(0)?.clone(); + let before = source.resource_usage().control; let retained = (0..16) .map(|_| { - cluster - .cell(0, 0) - .map(|cell| cell.inspect_peer(cluster.peers[1].node, Duration::from_secs(30))) - }) - .collect::>>()?; - // Business producers keep their independent admission while metadata - // results remain retained by these unpolled control waiters. - assert_eq!( - cluster - .cell(leader, 0)? - .command( - cluster.definitions[leader].increment.clone(), - 1_i64.to_be_bytes().to_vec(), - public_node::options(1)? + source.inspect_peer_physical_inventory( + cluster.peers[1].node, + revision, + Duration::from_secs(30), ) - .await? - .result, - 1_i64.to_be_bytes() - ); - tokio::time::timeout(Duration::from_secs(5), cluster.close()).await??; + }) + .collect::>(); + let charged = source.resource_usage().control; + ensure!(charged.items > before.items && charged.bytes > before.bytes); + assert_eq!(cluster.write(0, 1, 1).await?.result, 1_i64.to_be_bytes()); + tokio::time::timeout(TIMEOUT, cluster.close()).await??; for waiter in retained { - let _ = waiter.await; + let _ = tokio::time::timeout(TIMEOUT, waiter).await?; } + ensure!(source.is_closed() && source.resource_usage().control.items == 0); Ok(()) } #[tokio::test(flavor = "multi_thread", worker_threads = 8)] async fn native_complete_node_load_scan_covers_every_registered_history() -> Result<()> { - let cluster = Cluster::start_with_inspection(true).await?; + let cluster = Cluster::open(true).await?; cluster.write(0, 1, 1).await?; cluster.write(1, 1, 1).await?; - let mut expected = [0; 4]; - for group in 0..2 { + let mut expected = [0; VOTERS]; + for group in 0..GROUPS { expected[cluster.leader(group).await?] += 1; } - let caller = cluster.cell(0, 0)?; - let deadline = tokio::time::Instant::now() + Duration::from_secs(5); + let managed = cluster + .managed + .as_ref() + .context("managed inspection fixture")?; + let (revision, authorization) = managed + .domain + .run(async { + let cut = managed.fleet.read(TIMEOUT).await?; + let authorization = managed + .fleet + .authorize_group(&cut, &cluster.births[0].history, TIMEOUT) + .await?; + Ok::<_, anyhow::Error>((cut.revision(), authorization)) + }) + .await??; + let observer = cluster.service(3)?; + let deadline = Instant::now() + TIMEOUT; for (target, leaders) in expected.into_iter().enumerate() { loop { - let observed = caller - .inspect_peer(cluster.peers[target].node, Duration::from_millis(500)) + let remaining = deadline.saturating_duration_since(Instant::now()); + ensure!( + !remaining.is_zero(), + "node-wide leader count did not converge for node {target}" + ); + let result = observer + .observe_peer_fleet_membership( + cluster.peers[target].node, + authorization.clone(), + revision, + remaining.min(Duration::from_millis(500)), + ) .await; - if let Ok(observed) = observed { - if let Some(load) = observed.leader_load { + if let Ok(proof) = result { + ensure!( + proof.progress().history == cluster.births[0].history + && proof.node().node.groups == 2 + ); + if let Some(load) = &proof.progress().leader_load { assert_eq!(load.capacity, 2); if load.leaders == leaders { ensure!(load.sample_age_ms < 5000, "complete load sample was stale"); @@ -175,13 +161,16 @@ async fn native_complete_node_load_scan_covers_every_registered_history() -> Res } } } - ensure!( - tokio::time::Instant::now() < deadline, - "node-wide leader count did not converge for node {target}" - ); tokio::time::sleep(Duration::from_millis(10)).await; } } + ensure!(cluster + .inspect(0, 3, revision) + .await? + .inventory + .context("standby inventory")? + .histories() + .is_empty()); assert_eq!( cluster .calls diff --git a/crates/cellar/tests/support/raft_process_recovery/worker.rs b/crates/cellar/tests/support/raft_process_recovery/worker.rs index d022d9be7..242cbad9b 100644 --- a/crates/cellar/tests/support/raft_process_recovery/worker.rs +++ b/crates/cellar/tests/support/raft_process_recovery/worker.rs @@ -115,6 +115,7 @@ pub(super) async fn run(config: Config) -> Result<()> { .start_raft_cells( native, RaftCellsServices { + recovery: None, listeners, tls: public_node::tls()?, store: store(&config.prefix)?, diff --git a/crates/cellar/tests/support/raft_read_gate.rs b/crates/cellar/tests/support/raft_read_gate.rs new file mode 100644 index 000000000..00c2845f3 --- /dev/null +++ b/crates/cellar/tests/support/raft_read_gate.rs @@ -0,0 +1,47 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Hold an entered synchronous SQL callback until the test releases it. +use anyhow::Result; +use std::sync::Arc; + +pub struct Gate { + open: std::sync::Mutex, + changed: std::sync::Condvar, + entered: tokio::sync::mpsc::UnboundedSender<()>, +} + +impl Gate { + pub fn new() -> (Arc, tokio::sync::mpsc::UnboundedReceiver<()>) { + let (entered, receive) = tokio::sync::mpsc::unbounded_channel(); + ( + Arc::new(Self { + open: std::sync::Mutex::new(false), + changed: std::sync::Condvar::new(), + entered, + }), + receive, + ) + } + + pub fn wait(&self) -> Result<()> { + self.entered.send(())?; + let mut open = self.open.lock().unwrap_or_else(|e| e.into_inner()); + while !*open { + open = self.changed.wait(open).unwrap_or_else(|e| e.into_inner()); + } + Ok(()) + } + + pub fn release(&self) { + *self.open.lock().unwrap_or_else(|e| e.into_inner()) = true; + self.changed.notify_all(); + } +} + +pub struct Release(pub Arc); + +impl Drop for Release { + fn drop(&mut self) { + self.0.release(); + } +} diff --git a/crates/cellar/tests/support/raft_repair_load.rs b/crates/cellar/tests/support/raft_repair_load.rs deleted file mode 100644 index dae9fb525..000000000 --- a/crates/cellar/tests/support/raft_repair_load.rs +++ /dev/null @@ -1,357 +0,0 @@ -// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. - -//! Competing native snapshot repairs while an independent cell serves traffic. -#![allow(clippy::disallowed_methods)] -use super::{native_cluster as cluster, public_node}; -use anyhow::{ensure, Context as _, Result}; -use cellar::sdk::consensus::*; -use std::{ - sync::{ - atomic::{AtomicUsize, Ordering}, - Arc, - }, - time::Duration, -}; - -async fn leader(cells: &[RaftCellHandle], excluded: Option) -> Result { - tokio::time::timeout(Duration::from_secs(10), async { - loop { - for (node, cell) in cells.iter().enumerate() { - if Some(node) != excluded && cell.status().await?.command_ready { - return Ok(node); - } - } - tokio::time::sleep(Duration::from_millis(10)).await; - } - }) - .await? -} - -async fn write( - cells: &[RaftCellHandle], - definitions: &[public_node::Counter], - excluded: Option, - sequence: u64, - increment: i64, -) -> Result { - // Retry only the same identity, digest and deadline, including after a lost - // response. A retry may discover the original committed receipt. - let options = public_node::options(sequence)?; - tokio::time::timeout(Duration::from_secs(10), async { - loop { - let primary = leader(cells, excluded).await?; - match cells[primary] - .command( - definitions[primary].increment.clone(), - increment.to_be_bytes().to_vec(), - options.clone(), - ) - .await - { - Ok(receipt) => { - ensure!(receipt.result == (sequence as i64 * increment).to_be_bytes()); - return Ok(receipt); - } - Err( - CommandFailure::Unavailable - | CommandFailure::Overloaded - | CommandFailure::NotLeader { .. } - | CommandFailure::Ambiguous { .. }, - ) => tokio::time::sleep(Duration::from_millis(10)).await, - Err(error) => return Err(error.into()), - } - } - }) - .await? -} - -fn bounded(service: &RaftCells) -> Result<(usize, usize)> { - let usage = service.resource_usage(); - let transport = usage.transport.context("live transport resource view")?; - for budget in [ - transport.snapshot_source, - transport.snapshot_receive, - transport.seed_metadata, - usage.business, - usage.control, - usage.archive_capture, - ] { - ensure!(budget.items <= budget.max_items && budget.bytes <= budget.max_bytes); - } - ensure!(transport.snapshot_source.max_items == 1); - ensure!(transport.snapshot_receive.max_items == 1); - // Physical snapshots retain bounded wire/codec buffers, independently of - // the complete encoded image. With this fixture's 257-byte chunks, the - // source reservation must fit within 8 MiB. - ensure!(transport.snapshot_source.max_bytes <= 8 * 1024 * 1024); - // A receive additionally admits the configured archive recovery set, - // including containing bundles and physical extraction buffers. It no - // longer reserves the old complete-image encoder/decoder allocations. - ensure!( - transport.snapshot_receive.max_bytes - == transport.snapshot_source.max_bytes - + TransportOptions::default().max_seed_recovery_bytes - ); - let sql = usage.sql.context("live SQL resource view")?; - ensure!(sql.work_reservations <= sql.work_capacity); - Ok(( - transport.snapshot_source.items, - transport.snapshot_receive.items, - )) -} - -pub async fn run() -> Result<()> { - let (gate, _entered) = public_node::Gate::new(); - gate.release(); - let calls: Vec<_> = (0..3).map(|_| Arc::new(AtomicUsize::new(0))).collect(); - let definitions = calls - .iter() - .map(|calls| public_node::counter(calls.clone(), gate.clone())) - .collect::>>()?; - let namespace = NamespaceConfig::new( - ClusterId::new([1; 16])?, - NamespaceId::new([2; 16])?, - NamespaceMode::Raft, - definitions[0].schema, - ); - let registrations = (0..3) - .map(|group| { - Ok(RaftCellRegistration { - placement: RaftCellPlacement::Replica, - namespace: namespace.clone(), - class: "Counter".into(), - birth: BirthProposal { - scope: GroupScope { - cluster: namespace.cluster, - namespace: namespace.namespace, - cell: format!("Counter:repair-load-{group}"), - group: GroupId::new([80 + group; 16])?, - incarnation: IncarnationId::new([3; 16])?, - }, - voters: (1..=3) - .map(|node| { - Ok(InitialMember { - member: MemberId::new(node)?, - node: NodeId::new([node as u8; 16])?, - }) - }) - .collect::>>()?, - learners: vec![], - }, - }) - }) - .collect::>>()?; - let cluster = cluster::Cluster::start_with_transport( - &definitions, - registrations.clone(), - 20, - RaftHostOptions { - checkpoint_entries: 4, - ..Default::default() - }, - TransportOptions { - snapshot_chunk_bytes: 257, - snapshot_timeout_ms: 30_000, - io_timeout_ms: 10_000, - ..Default::default() - }, - ) - .await?; - let cells = registrations - .iter() - .map(|registration| { - cluster - .services - .iter() - .map(|service| service.cell(namespace.namespace, ®istration.birth.scope.cell)) - .collect::, _>>() - }) - .collect::, _>>()?; - let sources = [ - leader(&cells[0], None).await?, - leader(&cells[1], None).await?, - ]; - let follower = (0..3) - .find(|node| !sources.contains(node)) - .context("common follower")?; - // The fault below holds every response on the repairing node's outgoing - // data connections, including ordinary Append acknowledgements. Place the - // traffic leader elsewhere so this is snapshot-budget contention, not a - // partition of the healthy leader's replication acknowledgements. Elections - // are independent per group; the two repair leaders do not establish this. - let healthy = leader(&cells[2], None).await?; - if healthy != sources[0] { - let status = cells[2][healthy].status().await?; - cells[2][healthy] - .transfer_leadership( - LeadershipTransfer { - target: MemberId::new(sources[0] as u64 + 1)?, - expected_term: status.term, - expected_revision: status.membership_revision, - }, - Duration::from_secs(2), - ) - .await?; - } - tokio::time::timeout(Duration::from_secs(2), async { - while !cells[2][sources[0]].status().await?.command_ready { - tokio::time::sleep(Duration::from_millis(2)).await; - } - Ok::<_, anyhow::Error>(()) - }) - .await - .context("healthy group leadership outside the held data lane")??; - - cluster.isolate(follower, false).await?; - let mut receipts = Vec::new(); - let mut cuts = Vec::new(); - for (group, group_cells) in cells.iter().take(2).enumerate() { - let before = group_cells[follower].status().await?.checkpoint; - let mut last = None; - for sequence in 1..=12 { - last = Some( - write( - group_cells, - &definitions, - Some(follower), - sequence, - group as i64 + 1, - ) - .await?, - ); - } - receipts.push(last.context("repair workload receipt")?); - cuts.push( - tokio::time::timeout(Duration::from_secs(10), async { - loop { - let primary = leader(group_cells, Some(follower)).await?; - let status = group_cells[primary].status().await?; - if status.checkpoint.index > before.index { - return Ok::<_, anyhow::Error>(status.checkpoint); - } - tokio::time::sleep(Duration::from_millis(10)).await; - } - }) - .await??, - ); - } - // Retain actual TLS response chunks on every possible snapshot source. The - // receiver must reserve its one slot before trying to read these bytes. - let holds = (0..3) - .filter(|node| *node != follower) - .map(|source| cluster.hold_data_responses_after(follower, source, 4096)) - .collect::>>()?; - cluster.heal(); - tokio::time::timeout(Duration::from_secs(10), async { - loop { - let held: usize = (0..3) - .filter(|node| *node != follower) - .map(|source| cluster.held_data_response_bytes(follower, source)) - .sum(); - if held > 0 && bounded(&cluster.services[follower])?.1 == 1 { - return Ok::<_, anyhow::Error>(()); - } - tokio::time::sleep(Duration::from_millis(2)).await; - } - }) - .await - .context("real snapshot response hold")??; - - let traffic = async { - for sequence in 1..=24 { - let receipt = write(&cells[2], &definitions, Some(follower), sequence, 7) - .await - .with_context(|| format!("healthy group command sequence={sequence}"))?; - let read = cells[2][follower] - .query( - definitions[follower].get.clone(), - vec![], - QueryOptions { - consistency: QueryConsistency::AtLeast(receipt.version_token), - timeout: Duration::from_secs(5), - ..Default::default() - }, - ) - .await - .with_context(|| format!("healthy group query sequence={sequence}"))?; - ensure!(read.served_by == MemberId::new(follower as u64 + 1)?); - ensure!(read.result == (sequence as i64 * 7).to_be_bytes()); - } - Ok::<_, anyhow::Error>(()) - }; - tokio::pin!(traffic); - let mut samples = 0; - loop { - let mut source_items = 0; - for service in &cluster.services { - source_items += bounded(service)?.0; - } - ensure!( - source_items > 0, - "held snapshot lost its source reservation" - ); - ensure!( - bounded(&cluster.services[follower])?.1 == 1, - "held snapshot receive reservation ended before traffic completed" - ); - samples += 1; - cellar::asyncrt::select! { - done = &mut traffic => { done.context("healthy commands/queries during repair")?; break; } - _ = tokio::time::sleep(Duration::from_millis(2)) => (), - } - } - for group in 0..2 { - ensure!( - cells[group][follower].status().await?.checkpoint.index < cuts[group].index, - "unfinished snapshot exposed a checkpoint" - ); - } - drop(holds); - for group in 0..2 { - let read = cells[group][follower] - .query( - definitions[follower].get.clone(), - vec![], - QueryOptions { - consistency: QueryConsistency::AtLeast(receipts[group].version_token.clone()), - timeout: Duration::from_secs(15), - ..Default::default() - }, - ) - .await?; - ensure!(read.result == (12 * (group as i64 + 1)).to_be_bytes()); - ensure!( - cells[group][follower].status().await?.checkpoint.index >= cuts[group].index, - "repair must install the compacted history" - ); - } - tokio::time::timeout(Duration::from_secs(10), async { - loop { - let idle = cluster - .services - .iter() - .map(bounded) - .collect::>>()?; - if idle.iter().all(|usage| *usage == (0, 0)) { - return Ok::<_, anyhow::Error>(()); - } - tokio::time::sleep(Duration::from_millis(10)).await; - } - }) - .await - .context("snapshot reservations physically released")??; - ensure!( - calls[follower].load(Ordering::SeqCst) == 0, - "follower replayed business logic" - ); - ensure!( - calls - .iter() - .map(|calls| calls.load(Ordering::SeqCst)) - .sum::() - == 48, - "each of 24 repair and 24 traffic commands must execute once" - ); - cellar::note!("repair budget samples={samples}; repaired_cells=2; concurrent_commands=24; concurrent_queries=24; snapshot_receive_peak=1"); - cluster.close().await -} diff --git a/crates/cellar/tests/support/raft_retained_disk.rs b/crates/cellar/tests/support/raft_retained_disk.rs new file mode 100644 index 000000000..db8e3fed6 --- /dev/null +++ b/crates/cellar/tests/support/raft_retained_disk.rs @@ -0,0 +1,42 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use anyhow::{ensure, Result}; +use sha2::{Digest, Sha256}; +use std::path::Path; + +// The stopped engine must remain byte-for-byte retained. Only hash the Raft +// subtree, never the surrounding application's credentials or live fleet data. +pub fn disk_digest( + root: &Path, +) -> Result> { + fn visit( + root: &Path, + directory: &Path, + files: &mut std::collections::BTreeMap, + ) -> Result<()> { + let mut entries = std::fs::read_dir(directory)?.collect::>>()?; + entries.sort_by_key(|entry| entry.file_name()); + for entry in entries { + let path = entry.path(); + let kind = entry.file_type()?; + ensure!( + !kind.is_symlink(), + "unexpected link in retained voter engine" + ); + if kind.is_dir() { + visit(root, &path, files)?; + } else { + let bytes = std::fs::read(&path)?; + files.insert( + path.strip_prefix(root)?.to_owned(), + Sha256::digest(bytes).into(), + ); + } + } + Ok(()) + } + let mut files = std::collections::BTreeMap::new(); + visit(root, root, &mut files)?; + ensure!(!files.is_empty(), "missing retained engine files"); + Ok(files) +} diff --git a/crates/cellar/tests/support/raft_shared_read_cluster.rs b/crates/cellar/tests/support/raft_shared_read_cluster.rs new file mode 100644 index 000000000..e394cabda --- /dev/null +++ b/crates/cellar/tests/support/raft_shared_read_cluster.rs @@ -0,0 +1,341 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Public shared-group SDK startup with directional real TCP faults. +use super::{public_node, Definition}; +use anyhow::{ensure, Context as _, Result}; +use cellar::sdk::{consensus::*, Node, NodeRaftConfig, NodeRaftListeners, ShutdownMode}; +use std::{ + sync::{atomic::Ordering, Arc}, + time::Duration, +}; +use tokio::net::TcpListener; + +#[allow(dead_code)] +#[path = "raft_tcp_fault.rs"] +mod tcp_fault; +pub(super) use tcp_fault::HeldResponses; +use tcp_fault::Proxy; + +const TIMEOUT: Duration = Duration::from_secs(5); +const VOTERS: usize = 3; +pub(super) const STALE_SOURCE_TIMEOUT: Duration = Duration::from_millis(200); + +#[derive(Clone, Copy, PartialEq, Eq)] +enum Lane { + Control, + Data, + Query, +} + +struct Link { + source: usize, + target: usize, + lane: Lane, + proxy: Proxy, +} + +pub(super) struct Cluster { + pub(super) services: Vec, + nodes: Vec, + links: Vec, + _roots: Vec, + pub(super) history: GroupHistory, +} + +impl Cluster { + pub(super) async fn start( + definitions: &[Definition; VOTERS], + mut shared: CellGroupNodeLimits, + mut host: RaftHostOptions, + ) -> Result { + shared.controller.max_groups = 1; + shared.controller.max_parallel_apply = 1; + shared.directory.max_groups = 1; + shared.entries.envelope = host.execution.envelope; + host.max_groups = 1; + host.max_input_bytes = shared.dispatch.max_input_bytes; + host.stale_source_timeout_ms = u64::try_from(STALE_SOURCE_TIMEOUT.as_millis())?; + let cluster = ClusterId::new([1; 16])?; + let namespace = NamespaceId::new([2; 16])?; + let history = GroupHistory { + cluster, + namespace, + group: GroupId::new([3; 16])?, + incarnation: IncarnationId::new([4; 16])?, + }; + let roots = (0..VOTERS) + .map(|_| tempfile::tempdir()) + .collect::, _>>()?; + let mut nodes = Vec::new(); + let mut listeners = Vec::new(); + let mut peers = Vec::new(); + for (index, root) in roots.iter().enumerate() { + let id = u8::try_from(index + 1)?; + let label = format!("shared-read-{id}"); + nodes.push( + public_node::node_named( + root.path(), + definitions[index].application.clone(), + &label, + ) + .await?, + ); + let bound = NodeRaftListeners { + control: TcpListener::bind("127.0.0.1:0").await?, + data: TcpListener::bind("127.0.0.1:0").await?, + query: TcpListener::bind("127.0.0.1:0").await?, + }; + peers.push(RaftPeerNode { + node: NodeId::new([id; 16])?, + disk: DiskLineage::new([id + 10; 16])?, + process: ProcessGeneration::new([id + 20; 16])?, + label, + server_name: "localhost".into(), + control_address: bound.control.local_addr()?, + data_address: bound.data.local_addr()?, + query_address: bound.query.local_addr()?, + }); + listeners.push(bound); + } + let birth = CellGroupBirthSpec { + history, + range: CellRange::all(), + epoch: RouteEpoch::new(1)?, + creation: ProposalId::new([5; 16])?, + voters: peers + .iter() + .enumerate() + .map(|(index, peer)| { + Ok(InitialMember { + member: MemberId::new(index as u64 + 1)?, + node: peer.node, + }) + }) + .collect::>()?, + learners: vec![], + }; + let store = Arc::new(object_store::memory::InMemory::new()); + let mut links = Vec::new(); + let mut services = Vec::new(); + for (source, listeners) in listeners.into_iter().enumerate() { + let mut routes = peers.clone(); + for (target, peer) in routes.iter_mut().enumerate() { + if source == target { + continue; + } + for (lane, address) in [ + (Lane::Control, &mut peer.control_address), + (Lane::Data, &mut peer.data_address), + (Lane::Query, &mut peer.query_address), + ] { + let proxy = Proxy::start(*address).await?; + *address = proxy.address; + links.push(Link { + source, + target, + lane, + proxy, + }); + } + } + let peer = &peers[source]; + let definition = &definitions[source]; + let mut log = ReplicaLogOptions::new( + nodes[source].info().data_dir.join("raft"), + peer.node, + peer.disk, + peer.process, + LogOpenMode::Create, + ); + log.max_groups = 1; + services.push( + nodes[source] + .start_raft_cells( + RaftCellsConfig { + infrastructure: NodeRaftConfig { + log, + consensus: RaftRuntimeOptions { + tick_interval_ms: Some(20), + max_groups: 1, + ..Default::default() + }, + transport: TransportOptions::default(), + }, + catalog: CatalogOptions { + prefix: "partition/shared-reads".into(), + ..Default::default() + }, + namespaces: vec![RaftNamespaceConfig { + namespace: NamespaceConfig::new( + cluster, + namespace, + NamespaceMode::Raft, + definition.schema, + ), + class: "Counter".into(), + initialize: if source == 0 { + Some(RaftNamespaceInitialization { + operation: ProposalId::new([6; 16])?, + groups: vec![birth.clone()], + }) + } else { + None + }, + }], + groups: vec![RaftGroupAssignment { + history, + placement: RaftGroupPlacement::Replica, + }], + peers: routes, + query_origins: vec![], + command_origins: vec![], + host: host.clone(), + shared, + }, + RaftCellsServices { + recovery: None, + listeners, + tls: tls()?, + store: store.clone(), + application: definition.application.clone(), + versions: VersionKeys::new( + VersionSigningKey::new([31; 32], u64::MAX)?, + vec![], + Duration::from_secs(86400), + )?, + }, + ) + .await?, + ); + } + Ok(Self { + services, + nodes, + links, + _roots: roots, + history, + }) + } + + pub(super) async fn leader(&self, excluded: Option) -> Result { + tokio::time::timeout(Duration::from_secs(15), async { + loop { + for (index, service) in self.services.iter().enumerate() { + if excluded != Some(index) + && service + .group(self.history)? + .status() + .await + .is_ok_and(|status| { + status.ready && status.leader == Some(index as u64 + 1) + }) + { + return Ok(index); + } + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await? + } + + pub(super) async fn isolate(&self, node: usize) -> Result<()> { + self.cut(|link| link.target == node || link.source == node) + .await + } + + pub(super) async fn isolate_consensus(&self, node: usize) -> Result<()> { + self.cut(|link| (link.target == node || link.source == node) && link.lane != Lane::Query) + .await + } + + pub(super) async fn block_control(&self, source: usize, target: usize) -> Result<()> { + self.cut(|link| { + link.source == source && link.target == target && link.lane == Lane::Control + }) + .await + } + + async fn cut(&self, select: impl Fn(&Link) -> bool) -> Result<()> { + for link in self.links.iter().filter(|link| select(link)) { + link.proxy.block(true); + } + tokio::time::timeout(TIMEOUT, async { + while self + .links + .iter() + .filter(|link| select(link)) + .any(|link| link.proxy.active.load(Ordering::SeqCst) != 0) + { + tokio::time::sleep(Duration::from_millis(1)).await; + } + }) + .await + .context("join cut TCP connections")?; + Ok(()) + } + + fn query_link(&self, source: usize, target: usize) -> Result<&Proxy> { + self.links + .iter() + .find(|link| link.source == source && link.target == target && link.lane == Lane::Query) + .map(|link| &link.proxy) + .context("directional query link") + } + + pub(super) fn hold_query_responses( + &self, + source: usize, + target: usize, + ) -> Result { + let proxy = self.query_link(source, target)?; + proxy.state.send_modify(|fault| fault.hold_response = true); + Ok(HeldResponses(proxy.state.clone())) + } + + pub(super) fn held_query_response_bytes(&self, source: usize, target: usize) -> Result { + Ok(self + .query_link(source, target)? + .held_response_bytes + .load(Ordering::SeqCst)) + } + + pub(super) fn heal(&self) { + for link in &self.links { + link.proxy.block(false); + } + } + + pub(super) async fn close(mut self) -> Result<()> { + for service in &self.services { + let report = service.close().await?; + ensure!( + report.mutation_outcomes == MutationOutcomes::NoUnresolvedLocalProducers + && report.retained_calls_at_shutdown == 0, + "unresolved request after healed partition" + ); + } + for node in self.nodes { + node.shutdown(ShutdownMode::Preserve).await?; + } + for link in &mut self.links { + link.proxy.close().await?; + } + Ok(()) + } +} + +fn tls() -> Result { + Ok(RaftTls::new( + vec![include_bytes!("../../transport/tcp/fixtures/ca.der") + .to_vec() + .into()], + vec![include_bytes!("../../transport/tcp/fixtures/server.der") + .to_vec() + .into()], + rustls::pki_types::PrivatePkcs8KeyDer::from( + include_bytes!("../../transport/tcp/fixtures/key.der").to_vec(), + ) + .into(), + )?) +} diff --git a/crates/cellar/tests/support/raft_storage_fault.rs b/crates/cellar/tests/support/raft_storage_fault.rs index cf192e8fb..cff16c519 100644 --- a/crates/cellar/tests/support/raft_storage_fault.rs +++ b/crates/cellar/tests/support/raft_storage_fault.rs @@ -27,6 +27,11 @@ pub const HOLD_LOCK_DROP: u8 = 10; pub const HOLD_IMAGE_READ: u8 = 11; pub const HOLD_SNAPSHOT_TEMP: u8 = 12; pub const HOLD_EXTENT_READ: u8 = 13; +pub const FAIL_AFTER_SYNC: u8 = 14; +pub const HOLD_DEPENDENCY_PREFIX: u8 = 15; +pub const HOLD_DEPENDENCY_SYNC: u8 = 16; +pub const HOLD_FAIL_PRIVATE_CLEANUP: u8 = 17; +pub const HOLD_SELECTED_DIRECTORY_SYNC: u8 = 18; pub struct Gate { pub mode: AtomicU8, @@ -36,6 +41,9 @@ pub struct Gate { signal: Option, pub sync_path: Mutex>, pub image_read_paths: Mutex>, + pub dependency_cut: Mutex>, + /// Optional sync selection, installed before arming and cleared on release. + pub sync_when: Mutex bool + Send + Sync>>>, } impl Gate { @@ -48,6 +56,8 @@ impl Gate { signal, sync_path: Mutex::new(None), image_read_paths: Mutex::new(vec![]), + dependency_cut: Mutex::new(None), + sync_when: Mutex::new(None), }) } @@ -77,6 +87,7 @@ pub struct Release(Arc); impl Drop for Release { fn drop(&mut self) { self.0.mode.store(OFF, Ordering::Release); + self.0.sync_when.lock().unwrap().take(); *self.0.released.lock().unwrap() = true; self.0.release.notify_all(); } @@ -113,6 +124,27 @@ struct FaultFile { image: bool, } +impl FaultFile { + fn dependency(&self) -> bool { + self.path == Path::new("snapshot.scratch") + || self + .path + .parent() + .and_then(Path::file_name) + .is_some_and(|name| name == "extents") + && self + .path + .file_name() + .is_some_and(|name| name.to_string_lossy().starts_with("e-")) + } + + fn pause_dependency(&mut self) -> io::Result<()> { + *self.gate.dependency_cut.lock().unwrap() = + Some((self.path.clone(), self.inner.file_len()?)); + self.gate.pause() + } +} + struct TrackedLock { _lock: HostFileLock, gate: Arc, @@ -137,6 +169,28 @@ impl HostFileIo for FaultFile { } fn write_all_at(&mut self, offset: u64, bytes: &[u8]) -> io::Result<()> { + if self.dependency() + && offset == 0 + && bytes.len() > 1 + && self + .gate + .mode + .compare_exchange( + HOLD_DEPENDENCY_PREFIX, + OFF, + Ordering::AcqRel, + Ordering::Acquire, + ) + .is_ok() + { + let prefix = bytes.len() / 2; + self.inner.write_all_at(offset, &bytes[..prefix])?; + self.inner.sync_all()?; + self.pause_dependency()?; + return self + .inner + .write_all_at(offset + prefix as u64, &bytes[prefix..]); + } if self.log && self .gate @@ -179,7 +233,24 @@ impl HostFileIo for FaultFile { } fn sync_all(&mut self) -> io::Result<()> { - if self.log { + if self.dependency() + && self + .gate + .mode + .compare_exchange( + HOLD_DEPENDENCY_SYNC, + OFF, + Ordering::AcqRel, + Ordering::Acquire, + ) + .is_ok() + { + self.inner.sync_all()?; + self.pause_dependency()?; + return Ok(()); + } + let sync_when = self.gate.sync_when.lock().unwrap().clone(); + if self.log && sync_when.is_none_or(|when| when()) { let mode = self.gate.mode.load(Ordering::Acquire); let mode = if matches!( mode, @@ -187,6 +258,7 @@ impl HostFileIo for FaultFile { | FAIL_SYNC | CRASH_BEFORE_SYNC | CRASH_AFTER_SYNC + | FAIL_AFTER_SYNC | FAIL_SYNC_HOLD_LOCK_DROP ) && self .gate @@ -201,6 +273,10 @@ impl HostFileIo for FaultFile { match mode { HOLD_SYNC | CRASH_BEFORE_SYNC => self.gate.pause()?, FAIL_SYNC => return Err(io::Error::other("injected Raft sync failure")), + FAIL_AFTER_SYNC => { + self.inner.sync_all()?; + return Err(io::Error::other("injected failure after durable Raft sync")); + } FAIL_SYNC_HOLD_LOCK_DROP => { self.gate.mode.store(HOLD_LOCK_DROP, Ordering::Release); return Err(io::Error::other( @@ -302,6 +378,23 @@ impl FileSystem for FaultFileSystem { } fn sync_all(&self, path: &Path) -> io::Result<()> { + let selected = self.gate.sync_path.lock().unwrap().as_deref() == Some(path); + if selected + && self + .gate + .mode + .compare_exchange( + HOLD_SELECTED_DIRECTORY_SYNC, + OFF, + Ordering::AcqRel, + Ordering::Acquire, + ) + .is_ok() + { + DirectFileSystem.sync_all(path)?; + self.gate.pause()?; + return Ok(()); + } if self.gate.sync_path.lock().unwrap().as_deref() == Some(path) && self .gate @@ -333,6 +426,15 @@ impl FileSystem for FaultFileSystem { } fn remove_file(&self, path: &Path) -> io::Result<()> { + if path + .file_name() + .is_some_and(|name| name == "working.sqlite") + && self.gate.mode.load(Ordering::Acquire) == HOLD_FAIL_PRIVATE_CLEANUP + && DirectFileSystem.metadata(path).is_ok() + { + self.gate.pause()?; + return Err(io::Error::other("injected private scratch cleanup failure")); + } DirectFileSystem.remove_file(path)?; if path .parent() diff --git a/crates/cellar/tests/support/raft_tcp_fault.rs b/crates/cellar/tests/support/raft_tcp_fault.rs new file mode 100644 index 000000000..ed5e53748 --- /dev/null +++ b/crates/cellar/tests/support/raft_tcp_fault.rs @@ -0,0 +1,224 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Fault injection between embedded nodes' real TLS listeners. +#![allow(clippy::disallowed_methods)] +use anyhow::{ensure, Result}; +use std::{ + net::SocketAddr, + sync::{ + atomic::{AtomicUsize, Ordering}, + Arc, + }, +}; +use tokio::{ + io::{AsyncRead, AsyncReadExt, AsyncWrite, AsyncWriteExt}, + net::{TcpListener, TcpStream}, + sync::watch, + task::{JoinHandle, JoinSet}, +}; + +#[derive(Clone, Copy, Default)] +pub(super) struct Fault { + pub(super) blocked: bool, + pub(super) stopped: bool, + pub(super) hold_response: bool, + pub(super) hold_after: Option, + pub(super) hold_once: bool, +} + +pub(super) struct Proxy { + pub(super) address: SocketAddr, + pub(super) state: watch::Sender, + pub(super) active: Arc, + pub(super) attempts: Arc, + pub(super) held_response_bytes: Arc, + pub(super) held_connection: Arc, + task: Option>>, +} + +// This shared fixture is also compiled by targets which only cut connections. +#[allow(dead_code)] +pub struct HeldResponses(pub(super) watch::Sender); + +impl Drop for HeldResponses { + fn drop(&mut self) { + self.0.send_modify(|fault| { + fault.hold_response = false; + fault.hold_after = None; + fault.hold_once = false; + }); + } +} + +async fn connection_cut(mut faults: watch::Receiver) { + loop { + let fault = *faults.borrow_and_update(); + if fault.blocked || fault.stopped || faults.changed().await.is_err() { + return; + } + } +} + +async fn forward_responses( + mut reader: impl AsyncRead + Unpin, + mut writer: impl AsyncWrite + Unpin, + mut faults: watch::Receiver, + held: Arc, + selected: Arc, + connection: usize, +) -> std::io::Result<()> { + let mut buffer = [0; 16 * 1024]; + let mut forwarded = 0usize; + loop { + let count = reader.read(&mut buffer).await?; + if count == 0 { + return Ok(()); + } + let mut offset = 0; + let mut counted = false; + while offset < count { + let mut fault = *faults.borrow_and_update(); + let chosen = selected.load(Ordering::SeqCst); + if fault.hold_once && chosen != 0 && chosen != connection { + fault.hold_response = false; + fault.hold_after = None; + } + let allowance = if fault.hold_response { + 0 + } else { + fault + .hold_after + .map_or(count - offset, |after| after.saturating_sub(forwarded)) + }; + if allowance == 0 { + if fault.hold_once + && selected + .compare_exchange(0, connection, Ordering::SeqCst, Ordering::SeqCst) + .is_err_and(|chosen| chosen != connection) + { + continue; + } + if !counted { + held.fetch_add(count - offset, Ordering::SeqCst); + counted = true; + } + if faults.changed().await.is_err() { + return Ok(()); + } + continue; + } + let end = count.min(offset.saturating_add(allowance)); + writer.write_all(&buffer[offset..end]).await?; + forwarded += end - offset; + offset = end; + } + } +} + +struct Active(Arc); + +impl Drop for Active { + fn drop(&mut self) { + self.0.fetch_sub(1, Ordering::SeqCst); + } +} + +impl Proxy { + pub(super) async fn start(destination: SocketAddr) -> Result { + let listener = TcpListener::bind("127.0.0.1:0").await?; + let address = listener.local_addr()?; + let (state, mut changes) = watch::channel(Fault::default()); + let active = Arc::new(AtomicUsize::new(0)); + let attempts = Arc::new(AtomicUsize::new(0)); + let held_response_bytes = Arc::new(AtomicUsize::new(0)); + let held_connection = Arc::new(AtomicUsize::new(0)); + let held = held_response_bytes.clone(); + let selected = held_connection.clone(); + let counted = active.clone(); + let attempted = attempts.clone(); + let task = tokio::spawn(async move { + let mut connections = JoinSet::new(); + loop { + cellar::asyncrt::select! { + changed = changes.changed() => { + if changed.is_err() || changes.borrow().stopped { break; } + } + done = connections.join_next(), if !connections.is_empty() => { if let Some(done) = done { done?; } } + accepted = listener.accept(), if connections.len() < 8 => { + let (mut source, _) = accepted?; + // Match the native transport: fault injection must not + // add Nagle/delayed-ACK stalls to small snapshot chunks. + source.set_nodelay(true)?; + let connection=attempted.fetch_add(1, Ordering::SeqCst)+1; + if changes.borrow().blocked { continue; } + let cut = changes.clone(); + let held = held.clone(); + let selected = selected.clone(); + let active = counted.clone(); active.fetch_add(1, Ordering::SeqCst); + let guard = Active(active); + connections.spawn(async move { + let _guard = guard; + let fault = *cut.borrow(); + if fault.blocked || fault.stopped { return; } + // A cut drops both sockets. A response hold instead + // retains one bounded ciphertext chunk until release + // or either connection direction closes. + cellar::asyncrt::select_biased! { "test fault transitions close sockets before more bytes are forwarded"; + _ = connection_cut(cut.clone()) => (), + _ = async { + let target = TcpStream::connect(destination).await?; + target.set_nodelay(true)?; + let (mut from_source, to_source) = source.split(); + let (from_target, mut to_target) = target.into_split(); + cellar::asyncrt::select! { + _ = tokio::io::copy(&mut from_source, &mut to_target) => (), + _ = forward_responses(from_target, to_source, cut, held, selected, connection) => (), + } + Ok::<_, std::io::Error>(()) + } => (), + } + }); + } + } + } + connections.abort_all(); + while let Some(result) = connections.join_next().await { + if let Err(error) = result { + ensure!(error.is_cancelled(), "proxy connection panicked: {error}"); + } + } + Ok(()) + }); + Ok(Self { + address, + state, + active, + attempts, + held_response_bytes, + held_connection, + task: Some(task), + }) + } + + pub(super) fn block(&self, blocked: bool) { + self.state.send_if_modified(|fault| { + let changed = fault.blocked != blocked; + fault.blocked = blocked; + changed + }); + } + + pub(super) async fn close(&mut self) -> Result<()> { + self.state.send_modify(|fault| fault.stopped = true); + if let Some(task) = self.task.take() { + task.await??; + } + Ok(()) + } +} + +impl Drop for Proxy { + fn drop(&mut self) { + self.state.send_modify(|fault| fault.stopped = true); + } +} diff --git a/docs/catalog.json b/docs/catalog.json index 3cf113be9..d3ad798d9 100644 --- a/docs/catalog.json +++ b/docs/catalog.json @@ -637,6 +637,102 @@ ], "category": "Design and evidence" }, + { + "path": "design/multi-raft/cell-groups/README.md", + "title": "Cell group delivery package", + "summary": "Shared cell-group delivery status, execution commands and completion boundaries.", + "roles": [ + "builder", + "platform", + "operations", + "team" + ], + "category": "Design and evidence" + }, + { + "path": "design/multi-raft/cell-groups/design.md", + "title": "Cell group design", + "summary": "Physical group consensus over independent SQLite cells, identities and hard cutover.", + "roles": [ + "builder", + "platform", + "operations", + "team" + ], + "category": "Design and evidence" + }, + { + "path": "design/multi-raft/cell-groups/handoff.md", + "title": "Cell-group implementation handoff", + "summary": "Current implementation checkpoint, evidence transfer and next executor steps.", + "roles": [ + "builder", + "platform", + "operations", + "team" + ], + "category": "Design and evidence" + }, + { + "path": "design/multi-raft/cell-groups/implementation-plan.md", + "title": "Implementation order", + "summary": "Dependency order and status of the sixteen cell-group implementation tasks.", + "roles": [ + "builder", + "platform", + "operations", + "team" + ], + "category": "Design and evidence" + }, + { + "path": "design/multi-raft/cell-groups/progress.md", + "title": "Cell-group implementation progress", + "summary": "Source-pinned cell-group implementation checkpoints and qualification limits.", + "roles": [ + "builder", + "platform", + "operations", + "team" + ], + "category": "Design and evidence" + }, + { + "path": "design/multi-raft/cell-groups/protocols.md", + "title": "Durable protocols and crash cuts", + "summary": "Durable enrollment, apply, topology, hibernation and disaster-recovery protocols.", + "roles": [ + "builder", + "platform", + "operations", + "team" + ], + "category": "Design and evidence" + }, + { + "path": "design/multi-raft/cell-groups/qualification.md", + "title": "Qualification and evidence contract", + "summary": "Failure oracles, workload profiles and evidence required for release.", + "roles": [ + "builder", + "platform", + "operations", + "team" + ], + "category": "Design and evidence" + }, + { + "path": "design/multi-raft/cell-groups/remaining-work.md", + "title": "Remaining cell-group work and priorities", + "summary": "Prioritize the remaining cell-group MVP, merge gates and full architecture deliverables.", + "roles": [ + "builder", + "platform", + "operations", + "team" + ], + "category": "Design and evidence" + }, { "path": "design/multi-raft/control-window-performance.md", "title": "Multi-Raft control-window performance", @@ -649,6 +745,18 @@ ], "category": "Design and evidence" }, + { + "path": "design/multi-raft/density-decision.md", + "title": "Shared cell groups: architecture decision", + "summary": "TiKV investigation and the decision to share physical Raft groups across whole cells.", + "roles": [ + "builder", + "platform", + "operations", + "team" + ], + "category": "Design and evidence" + }, { "path": "design/multi-raft/disaster-recovery.md", "title": "Zero-quorum disaster recovery", diff --git a/docs/design/multi-raft/README.md b/docs/design/multi-raft/README.md index 5628ebf7f..640c32c85 100644 --- a/docs/design/multi-raft/README.md +++ b/docs/design/multi-raft/README.md @@ -10,6 +10,16 @@ Purpose: Navigate consensus contracts, ordered implementation packages, recovery Leaders execute private commands; replicas publish committed SQLite/LTX images. Quorum and S3 requests require their respective durability proofs. S3 recovery does not restore votes, leadership, or quorum. +## Shared cell-group replacement + +The [density decision](density-decision.md) selects multiple independent SQLite +cells per Raft group. The [cell-group package](cell-groups/README.md) specifies +16 dependent tasks for complete manifests, shared execution, split/merge, +hibernation and fenced recovery. Contracts/storage/routing gates have passed; +shared execution is in progress. Production SDK cutover and final qualification +remain required. A dedicated hot cell uses the same machinery at occupancy one. +The MR evidence below retains the earlier one-cell implementation's scope. + ## Delivery status The [task inventory](tasks.json) records these package boundaries: diff --git a/docs/design/multi-raft/application-contract.md b/docs/design/multi-raft/application-contract.md index fe9cfa9e8..3276e5584 100644 --- a/docs/design/multi-raft/application-contract.md +++ b/docs/design/multi-raft/application-contract.md @@ -51,13 +51,14 @@ Register operations by `(class, operation, kind, contract_digest)`. Ingress bind The internal MR-06 transaction facade supplies a captured Unix-millisecond value and 32 captured random bytes per attempt. Its SQL authorizer refuses ambient clock/random functions; use these captured values as SQL parameters. Additional input facilities require explicit host adapters before becoming SDK capabilities. - `RaftClass::schema_digest()` first binds the class name, explicit schema version and supplied schema-definition digest under the `cellar-raft-schema-1` SHA-256 domain. -- It then binds that base digest and the sorted required command names and contract digests under `cellar-raft-class-1`, with explicit count/name lengths. +- It then binds that base digest and the sorted required command names and contract digests under `cellar-cell-class-1`, with explicit count/name lengths. - Read this identity after registering commands and use it for the namespace root and physical update application digest. - Omitting, renaming or changing a command cannot preserve voting compatibility by repeating the schema version/digest. - Delivery targets use `RaftClass::definition_digest()` plus an exact operation contract. - That declaration identity binds the class name, schema version and supplied definition digest before adding bindings, allowing self-delivery and cyclic class declarations without recursive manifest hashes. - It does not replace the complete `schema_digest()` check at namespace admission. -- Full target history selects the immutable namespace and its required compiled manifest. +- Full logical target history (`CellHistory`) selects the immutable namespace and its required compiled manifest. Physical group placement is absent from command and effect identity; see the [shared command boundary](../../../crates/cellar/durability/raft/README.md#shared-group-registered-command-boundary). +- Registered shared queries expose their exact `CellVersionObservation` through `QueryVersion::Cell`; `QueryContext::identity()` is logical cell history. The temporary old host uses a distinct `GroupCut` observation with no mutation sequence. CG-16 removes that staging branch; see the [registered query boundary](../../../crates/cellar/durability/raft/README.md#shared-group-registered-query-boundary). - Declaration order does not affect the identity. - Query contracts remain separate: a replica with incompatible/missing query code can still vote with the matching required commands, but refuses that query before executing it. - Changing required command contracts requires a fresh immutable namespace identity; this is a marker-1 hard cutover without a compatibility reader or in-place conversion. @@ -67,6 +68,11 @@ The internal MR-06 transaction facade supplies a captured Unix-millisecond value Queries receive no general `Context`, writable `Storage`, normal cell constructor, alarm setter or outbound-network capability. SQLite authorizer/read-only flags must reject writes, schema changes, ATTACH, mutating PRAGMA and writable extension paths even if they arrive through raw SQL. Query functions use stateless services or immutable configuration; application instance fields cannot substitute for replicated state. +Shared-group `QueryOptions::default()` requests a linearizable read. It requires +a fresh quorum, so a group that has lost its majority refuses the request. +Use `AtLeast` with a receipt's version token to require a specific committed +version, or explicitly select `Stale` to permit an older image. + Application flow with an explicit read consistency: ```text @@ -84,6 +90,84 @@ answer = counter.query(Get, consistency=AtLeast(receipt.version_token)) The framework wraps the command with its receipt transaction. Applications must not independently commit early. Errors and panics roll back private changes. The encoded result is bounded and stored with the mutation before LTX capture. +## Namespace and archive setup + +`RaftNamespaceConfig` declares a compiled namespace capability; physical +`RaftGroupAssignment` values declare complete group placement. Logical handles +bind `CellHistory`, and `enroll` replicates the registered initializer under an +explicit immutable creation ID. A handle does not create a Raft machine or infer +a cell incarnation from a physical group. + +`RaftNamespaceProvisioning` is the public operator setup path. It runs catalog +work on the supplied execution domain and shares the host's immutable range +initializer. `initialize: Some(plan)` selects an explicit, retry-stable initial +range plan; `None` only opens existing namespace/directory metadata. Managed +namespaces still require their separate authority protocol and refuse this path. +This setup creates no local member or serving capability. + +`prepare_archive` returns a one-use `RaftGroupArchiveInitialization`. Persist and +sync its exact `encoded_intent()` before `commit`; uncertain completion requires +retaining that intent and recovering rather than allocating another initializer. +`open_archive` verifies only the catalog-bound identity and existing registry +counters. Missing binding, identity or counters refuse; startup never replaces +them. Neither a prepared intent nor an opened archive supplies a learner seed, +fresh quorum, local reclamation proof or disaster fencing authority. + +`RaftCells::identity` exposes the immutable node/disk/process tuple as metadata. +For example, an ingress can combine its complete process generation with a +checked sequence to allocate query invocation IDs shared by native and Axum +handlers. Identity metadata does not authorize a stale process to serve or restart. + +## Managed shared startup ownership + +Managed startup checks the actual namespace declarations against the compiled +classes, rooted host capabilities and complete declared namespace registry. It +opens existing namespace metadata; group birth and range publication remain +separate operator protocols. An enrolled standby can boot before its first range +is published, without inventing an empty directory or serving a cell. This is not +capacity readiness, complete inventory or group activation. + +Cold operators can publish an explicit standby directory genesis before capacity +admission. Its pin, journal and operation index prove zero selected groups; a +missing head or pin refuses. The exact standby observation is then the CAS base +for the first complete range map, after the fleet has reserved every birth. +Standby has no cell route and cannot replace an existing partition. Persist and +retry the same encoded intent after an uncertain publication. Import freezing +also journals its head transition, including for a real zero-group inventory. +Initial birth admission revalidates each voter's opaque capacity receipt at the +exact fleet cut. An ordinary retained boot invalidates the previous receipt even +if its descriptor remains Active. Existing groups retain ownership; new births +require fresh capacity qualification and activation for that boot. + +The host pins its physical engine before binding a boot. The ordinary boot CAS +or retained maintenance rejoin protocol must settle, and the exact lifecycle +guard must sync before endpoints are published or groups attach. Retained routes +use durable current membership. Joined members still require their authenticated +install receipt, permanent engine claim and current catalog/boot checks. Initial +members use the managed initial-member permit. No metadata read grants a vote or +current-term quorum authority. + +The log owner and immutable rooted node descriptor must match before a storage +pin or boot mutation. Advertised control/data/query endpoints remain explicit +configuration and may differ from listener bind addresses, including wildcard +listeners. Endpoint discovery cannot substitute for those owner checks. + +One bounded directory service owns periodic rooted trust and endpoint refresh. +Its independent catalog read turn cannot occupy the foreground mutation turn. +Each batch retains one original deadline and at most eight observations; its +metadata reservation remains owned until the worker exits. Missing observations +preserve accepted routes. A superseded local boot or rooted conflict closes Raft +admission. Shared shutdown signals this service and joins entered work before +consensus and storage close. Internal observers do not retain the last external +SDK lifetime. + +Node installs its existing stop reporter from that same bound authorization, +synced guard and actual probe signer. A dropped stop waiter cannot cancel an +admitted lifecycle update. Stop acceptance, metadata discovery and local producer +quiescence remain distinct from joined-owner proof and safe disruption accounting. +The complete managed lifecycle and cold capacity/inventory bootstrap require their +own public qualification; these ownership contracts do not mark CG-07 complete. + ## Receipt semantics For v1, use `(authenticated_client_id, sequence)` as the command ID, scoped to the cell and incarnation. Each client identity is non-reusable within that incarnation. The caller allocates increasing per-cell sequences and persists its pending command before sending; `CommandOptions` requires that explicit ID. Different client identities can submit concurrently; one identity submits its commands in sequence. @@ -111,7 +195,7 @@ Persist `retry_until`/receipt retention policy. Capacity pressure cannot silentl | Outcome | Meaning and retry behavior | | --- | --- | | `NotLeader` | Redirect hint; server still validates authority after reroute | -| `Unavailable` / `Overloaded` | Refused before proposal submission; retry within policy | +| `Unavailable` / `Overloaded` | Availability refusal; inspect the command error's ambiguity flag before deciding whether submission may have occurred. Preserve the original ID and input. | | `StorageFull` | Private SQLite execution ran out of storage before proposal submission; free capacity before retrying the same identity. The private image is discarded and rebuilt; a failed write cannot establish a reusable transaction. | | `Ambiguous(command_id)` | Proposal may have committed; reconcile using the same ID | | `CommandConflict` | ID reused with different content; do not retry with altered content | @@ -123,7 +207,14 @@ Persist `retry_until`/receipt retention policy. Capacity pressure cannot silentl | `ArchiveUnavailable` | S3 requirement not yet proven; a submitted command remains ambiguous | | `CorruptState` | Stop that replica's serving/proof role and recover through the protocol | -`RaftCellHandle::command`, `query` and `status` admit synchronously before returning a future. Dropping the future cancels its waiter. An unpolled waiter does not suspend its deadline: the node checks expiry before starting queued private work and before submission. A readiness observation is a hint; every command still checks live authority and can be refused if it changed. +`RaftCellHandle::command`, `query` and `receipt` admit synchronously before returning a future. Dropping the future cancels its waiter. An unpolled waiter does not suspend its deadline: the node checks expiry before starting queued private work and before submission. A readiness observation is a hint; every command still checks live authority and can be refused if it changed. + +`CellCommandError` preserves the original command ID, a conservative ambiguity +flag and its typed cause. `CellDispatchError::refusal()` exposes a local or +authenticated remote application refusal without importing verification interfaces. +A refusal does not clear the enclosing command's ambiguity. HTTP projection must +retain both the command identity/ambiguity and the cause; it cannot infer rollback +from an availability or archive error. Cancellation before submission rolls back private work. Cancellation after submission cancels the waiter, not the consensus decision. Never convert a lost reply into a fresh mutation ID. Read retries retain their original deadline and consistency requirement. Cross-cell workflows remain application coordination. @@ -168,29 +259,46 @@ Keep separate command and query contract digests. A follower can hold the correc Perform current authentication and authorization at ingress and destination. Authorization derived from a stale database image is itself stale; operations requiring latest revocation checks must use a fresh authoritative check. Version tokens prove requested history, not access permission. -- The current SDK exposes a diagnostic `CommittedVersion` in `version` and an opaque authenticated observation in `version_token`. -- `AtLeast` accepts the opaque token and validates it on the destination before any index comparison. -- Tokens bind cluster, namespace, cell, group, incarnation, application digest, index, term, issuance and expiry. -- They survive ordinary leadership changes; cross-history comparisons fail. -- The returned position names the cut captured with the pinned image, including verified protocol-only coverage of that image. - -- `QueryOptions::routing` defaults to local execution. -- `QueryRouting::Replicas` accepts up to eight distinct member IDs in priority order, a per-attempt timeout between one millisecond and sixty seconds, and an explicit leader-fallback flag. -- The total query deadline is fixed on initial admission. -- Each attempt preserves its contract/input, consistency/token and stale policy. -- A fallback uses an untried leader hint and obtains the same read proof at its destination. -- `QueryResult::served_by` reports the responding member. -- A forwarding origin need not have compatible query code; the destination must check it before execution. - -- Each `RaftCellRegistration` explicitly selects `RaftCellPlacement::Replica` or `QueryRouter { leader_hint }`. -- A routing-only binding requires an existing namespace/birth and cannot create a member or SQLite image. -- It rejects command, local-query and local-status calls; choose explicit replica routing. -- Its optional leader hint is discovery metadata and never supplies a read barrier. -- `RaftCellsConfig::query_origins` installs trusted namespace read grants for exact cluster/namespace/node/disk/process endpoints. -- These grants do not authorize Raft messages or change membership. -- Live revocation cancels old query waits; revision tombstones prevent stale updates/regrants from reviving those requests. -- The embedding supplies trusted grants on restart; MR-11 distributes current process views and implements bounded dynamic placement. -- See the [read audit](read-audit.md) for scope and [counter HTTP contract](https://github.com/crabbuild/cellar/blob/8f18c29a0f28977eb8a4cae51848982ad7d43739/examples/raft-counter/README.md) for the native/Axum consistency, token and bounded-routing headers. +The canonical shared SDK query result carries CellVersion, CommitWitness, +CellVersionToken and the serving NodeIdentity. The logical version remains +separate from physical group term/index. Tokens authenticate the complete logical +history, schema, version, witness, issuance and expiry. They prove no access +permission. AtLeast validates the token before comparing the selected cell's +version; another history or schema refuses. Linearizable needs a fresh safe +ReadIndex and the corresponding applied root. Stale supplies neither a quorum +barrier nor a wall-clock age promise. + +QueryOptions supplies consistency, timeout, bounded max_attempts and QueryRouting. +Local uses the same shared controller. Replicas captures one to eight distinct +member preferences, a positive attempt_timeout no larger than the original timeout, +and an optional leader_fallback. Candidates run in order, at most once; one final +known leader may be tried only if it was not already a candidate. An absent leader +hint does not authorize a guessed destination. Each attempted read consumes the +same global attempt bank. Every ticket deadline can only shorten the original +invocation deadline, which starts before the waiter returns. + +Retries preserve the contract, input, token, logical history, routing fingerprint +and node admission. Expired physical SQL and transport work retain their permits +until they join, even while a later attempt runs. Results retain the actual local +or authenticated remote replica and response reservations. Wrong-history/schema, +registration and permission failures do not become replica fallback opportunities. +Incoming query RPCs execute locally and cannot recursively forward. + +Physical group discovery and leader hints are observations; destination admission +still checks current namespace/process grants, route, membership and the selected +cell read proof. The retired per-cell QueryRouter registration path is not used. +The shared routing component passes its recorded native gates; complete public +SDK, ingress and managed lifecycle qualification remains tracked by the +[cell-group implementation record](cell-groups/progress.md). + +RaftCellsConfig::query_origins installs trusted namespace read grants for exact +cluster/namespace/node/disk/process endpoints. Every authenticated cell route +binds a physical group and a separate logical cell. A grant does not authorize +Raft messages, commands, enrollment or receipt mutations. Revocation cancels old +query waits; revision tombstones prevent stale updates or regrants from reviving +those requests. The embedding supplies trusted grants on restart. The +[read audit](read-audit.md) retains historical qualification scope; its old SDK +fixtures must be converted before they can qualify the shared runtime. - `RaftCellsServices::versions` supplies explicit managed Ed25519 signing and verification keys. - Configured token lifetime is between one millisecond and seven days. @@ -205,4 +313,4 @@ Perform current authentication and authorization at ingress and destination. Aut - Expired keys may be removed. - Private signing material is never written to this registry, and registry contents cannot add a key to a server's configured trust roots. -Public constructors remain centralized in the application builder. Registration validation rejects an ordinary mutable `Cell` factory for a Raft class. Background workers and maintenance must use the same mode-aware dispatch boundary so they cannot bypass this restriction. \ No newline at end of file +Public constructors remain centralized in the application builder. Registration validation rejects an ordinary mutable `Cell` factory for a Raft class. Background workers and maintenance must use the same mode-aware dispatch boundary so they cannot bypass this restriction. diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md new file mode 100644 index 000000000..0542e0a5b --- /dev/null +++ b/docs/design/multi-raft/cell-groups/README.md @@ -0,0 +1,340 @@ + + +# Cell group delivery package + +Purpose: Shared cell-group delivery status, execution commands and completion boundaries. + +Status: executable design and implementation plan delivered on 2026-10-29; +end-to-end implementation remains in progress. CG-01–06 and CG-08 have passed +recorded component gates. CG-07 canonical SDK/fleet cutover is IN_PROGRESS; +CG-09–16 remain required. PR #160 is a draft checkpoint until its public/build +and current-source backend gates pass. + +Start with [remaining work and priorities](remaining-work.md) for the MVP order +and merge criteria. [Progress](progress.md) records source-scoped validation; +[handoff](handoff.md) identifies the current implementation and continuation. +Historical Docker/RustFS results cover their recorded local component boundaries, +including all-voter-disk loss and effects. They do not qualify the latest combined +SDK/fleet runtime, independent hosts, sustained load or release readiness. + +This delivery is the executable design and implementation plan. End-to-end +implementation and hard cutover remain tracked as separate task and release gates. See the [executor handoff](handoff.md) for the current +source, qualification boundary, reviewed main integration and commands. Plan +delivery or an intermediate task gate does not qualify the complete runtime. + +This package implements the direction selected in the +[density decision](../density-decision.md). The target is TiKV-style partitioned +multi-Raft: many cells in a group, many groups on a node, group replication across +nodes, dynamic split/merge, placement and coordinated hibernation. Cellar keeps +independent SQLite/LTX cell state and its Quorum/S3 durability contracts. +It does not implement TiKV's KV API, MVCC, transactions, wire compatibility or PD. + +## Read and execute + +The plan runner requires Python 3.11 or newer and Git; it has no Python package dependencies. + +1. Read [design](design.md), [protocols](protocols.md) and + [qualification](qualification.md). These define the target, including crash + behavior and final release gates; they supersede fixed assignment as the final + target. Fixed assignment is an intermediate milestone only. +2. Run the commands below from the repository root. `ready` reports dependency + eligibility; `show CG-07` emits a complete executor brief, source excerpts, + scope, steps, exact test commands and completion rules. It never runs them. +3. Implement each task in dependency order on the active PR branch or an explicitly + assigned branch. Preserve working intermediate commits. Do not expose a second + production mode or reinterpret existing data as the new format. +4. Record actual test outputs and source/artifact hashes. A task can become DONE + only with verified evidence and completed dependencies. Final qualification + repeats the full capability matrix at one release revision. + +Current source-scoped records live under the ignored +`.cellar/cell-groups-evidence/CG-01-06-gates-e32115aa-02/` directory, with one +`CG-NN-record.json` per CG-01–06 task. Logs, six copied executables and all source +hashes are retained locally and backed up on the workspace volume. A fresh +checkout must receive the same repository-relative evidence paths, or rerun the +unchanged exact gates before full package validation. The separately scoped local +Docker/RustFS record is retained under +`.cellar/cell-groups-evidence/CG-06-docker-f8aacd05/`. Historical CG-01–05 raw +records remain unavailable; their earlier source-scoped results are documented in +progress.md and are superseded by the fresh native records. Missing or changed +evidence still fails validation. +CG-08 current-source evidence lives under +`.cellar/cell-groups-evidence/CG-08-gates-4ea70e48-01/`; same-executable RustFS +evidence lives under the `rustfs-4ea70e48/` effects checkpoint described in progress.md. +The design, task briefs and validator unit tests are committed in this package. + +```sh +python3 docs/design/multi-raft/cell-groups/plan.py validate +python3 docs/design/multi-raft/cell-groups/plan.py preflight +python3 docs/design/multi-raft/cell-groups/plan.py ready +python3 docs/design/multi-raft/cell-groups/plan.py show CG-07 +python3 -m unittest discover -s docs/design/multi-raft/cell-groups -p 'test_*.py' +``` + +`validate` checks the task graph, coverage, source inventory, links and evidence +for DONE tasks. `preflight` checks source drift against `baseline.json`; it does +not silently refresh hashes. After a dependency changes an input, review that +change and update the baseline with its new revision and review note before +executing the dependent task. `show` labels planned test targets that are absent; +an absent target is never a passing test. `release` exits nonzero until every +task and required deployment gate is evidenced at the final source revision. + +The runner accepts no arbitrary shell commands and launches no processes or +containers apart from read-only Git source inspection. Commands displayed in +executor briefs are explicitly run by the implementation agent. Destructive +qualification belongs only to the harness's recorded, owned disposable resources. + +### Current executor handoff + +CG-01 through CG-06 and CG-08 are DONE at their declared scoped boundaries. +CG-07 is IN_PROGRESS in the spare checkout, with incomplete canonical startup changes. +Read [progress](progress.md) and [handoff](handoff.md), then run `plan.py show CG-07` +for canonical SDK/host/fleet cutover. Keep complete group closure, logical cell +identity, uncertainty, managed admission and physical ownership intact. The public +cutover and all CG-09 through CG-16 capabilities and release profiles remain +required. No intermediate gate proves the final combined runtime. + +### Historical implementation checkpoints + +These checkpoints describe earlier source-scoped work. Current records above +supersede unavailable historical raw artifacts; the current handoff determines +what remains. Earlier “next” or IN_PROGRESS statements below are historical. + +Physical group genesis, typed replica handles, permanent initial-member claims, +complete group inventory and invocation-bounded catalog-confirmed redirects are +implemented through `0df009a2`; see the latest [checkpoint](progress.md) for +evidence and limits. CG-03 has been audited at its preparation boundary. Its +record is at +`.cellar/cell-groups-evidence/CG-03-routing-final/record.json`; transfer that entire +directory along with the earlier DONE-task evidence to the implementation +executor. CG-04 now retains creation identity with the selected image and +prepares ordered enrollment and exact retries. Its physical group owner consumes +the catalog startup permit and verified selected image/manifest closure, and +obtains current-term quorum before protocol admission. The shared node scheduler now multiplexes these physical owners. The per-cell SQL +producer and exact receipts now pass their partial regression gate, followed by +fixed-pool SQL ownership and admission at `51dff0e5`. Applied-barrier receipt +reconciliation is implemented at `8a0cf640`, followed by read-only receipt lookup +and saturated-capacity/expiry/conflict cases at `91c3ecfc`. Initialization and +concurrent enrollment now run on that pool at `6e09bbc7`, including original +creation retries, follower resolution and atomic preparation-term proposal checks. +Bounded proposal acceptance and configured enrollment capacity are implemented at +`446dd905`, including pending creations, exact acceptance retries, stale-base +refusal and a shared node metadata budget. Prepared submission sealing and exact +rejection cleanup are implemented at `0cf89c40`; see the latest checkpoint for +qualification. Coherent shared read selection and immutable queries now exist +at `88c4717d`, repaired and qualified at `3da4481e`: 204 selected Rust cases +and all twelve verification commands pass. Durable cell-version witnesses and +query/receipt observations are implemented and qualified at `ed84df5d`: 214 +selected Rust cases and thirteen verification commands pass. See the handoff +for the exact scope and retained evidence. +Committed image preparation now runs through the shared SQL workers at `2085cc96`, +with its fixture repair and full partial gate at `c9100ec7`: 254 selected Rust +cases and fifteen commands pass. Transfer both +`.cellar/cell-groups-evidence/CG-04-apply-final/` and the retained failed +`.cellar/cell-groups-evidence/CG-04-apply/` directory with the prior evidence. +Selected-image recovery and physical apply fault coverage are implemented at +`1ec2eeaa`. Both image modes cover cancelled preparation/close while real image +or manifest directory sync is held, failed-sync old-root preservation, and cold +selected-root recovery with committed replay and no renewed serving authority. +All 260 selected Rust cases and fifteen commands pass; transfer the retained +`.cellar/cell-groups-evidence/CG-04-apply-recovery/` directory with prior evidence. +Whole-process kill and power-loss cuts remain unqualified. See the newest +[progress checkpoint](progress.md) for the committed-source gate and evidence. +Registered command callbacks now run on shared SQL workers at `b28078af` with +logical cell identity, pre-receipt validation and exact receipt settlement. +Command/effect identities exclude the physical group; old signed effect layouts +refuse under the marker-1 cutover. At that checkpoint shared query context, effect scheduling and +production catalog/SDK dispatch were incomplete. All 295 selected Rust cases +and seventeen commands pass; transfer +`.cellar/cell-groups-evidence/CG-04-registered/` with the previous evidence. Its +checkpoint binds 221 source inputs and fifteen selected executables. See the newest +progress record for scope and retained failures. +Registered shared query callbacks now receive exact logical cell observations at +`7a79511d`, preserving selected witnesses, read-only SQL and retained physical +ownership through cancellation and joined shutdown. The old adapters expose a +distinct temporary group cut that CG-16 must delete. See [progress](progress.md) +for this checkpoint's qualification: all 319 selected Rust cases and seventeen +commands pass. Transfer `.cellar/cell-groups-evidence/CG-04-registered-queries/` +with prior evidence; its checkpoint binds 225 source inputs and sixteen selected +executables. Registered SDK/catalog routing, production +retry/compaction integration and cache lifecycle remain required. Managed +fleet integration remains required by CG-07. A birth +metadata observation alone cannot authorize voter startup. + +Local query invocations now connect the catalog to registered execution at +`8723db2d`: namespace/class bindings, immutable request fingerprints, original +deadlines/attempt budgets, authenticated AtLeast tokens and destination authority +checks share existing workers. All 103 native cases across four affected suites +and eight commands pass. Transfer +`.cellar/cell-groups-evidence/CG-04-routed-queries/` with prior evidence; it binds +227 source inputs and four selected executables. Command/enrollment routing and +production controller/SDK integration are the next critical gaps. Metadata-only +redirect tests do not qualify live transfer. See [progress](progress.md). + +Catalog-bound command phases now retain original SQL ownership through cancelled +preparation, recover repeated acceptance and settle exact receipts at `1f014f42`. +A real leader-transfer retry preserves captured inputs; S3 minima explicitly refuse +until archive proof integration. All 105 native cases and eight commands pass. +Transfer `.cellar/cell-groups-evidence/CG-04-routed-commands-qualified/` and the failed +`CG-04-routed-commands/` and `CG-04-routed-commands-final/` gates with prior evidence. The newest record binds 229 +source inputs and four executables. Production controller ownership after ingress +cancellation and physical-group transport integration are the next critical gaps, +followed by routed enrollment and the public SDK cutover. See [progress](progress.md). + +Physical-group peer transport now starts directly on the shared runtime at +`eb99a242` and reuses existing TLS pools, pipelines, budgets and shutdown ownership. +Typed group histories refuse old untagged frames; unsupported group snapshots +refuse. All 192 native cases, 91 transport-filtered library cases and nine +commands pass. Transfer `.cellar/cell-groups-evidence/CG-04-physical-transport/` +with prior evidence (241 source inputs, eight executables). At that checkpoint, +production controller ownership and reconciliation after caller disconnect remained +required, followed by routed enrollment and SDK/shared-query RPC integration. +Those fixtures still drive runtime events. See [progress](progress.md) for exact limits. + +At `f77d0d93`, registered initialization and catalog-routed local enrollment +run on the shared controller. Commands and creations share bounded admission and +response-independent ownership. Exact creation records survive later mutations; +conflicts do not replay initialization. The partial gate passes 90 Rust cases +and ten commands; transfer +`.cellar/cell-groups-evidence/CG-04-enrollment-controller-qualified/` (254 source inputs, +four executables) with prior evidence. Next connect public handles and remote +shared dispatch, including safe receipt routing. Lazy birth/dynamic roster, +per-group failure isolation and configuration/cache lifecycle remain gaps. +See [progress](progress.md); CG-04 is not complete. + +At `da79a696`, attached-peer shared query RPC uses the existing bounded TLS lane +with physical group/logical cell identity and actual cell tokens. Host-owned +registration, original deadlines/retry budgets and physical SQL cleanup remain +bound. The partial gate passes 195 Rust cases and eleven commands; transfer +`.cellar/cell-groups-evidence/CG-04-remote-queries-final/` (260 source inputs, +4 executables). Nonmember shared discovery, bounded destination route caching +(the gateway currently resolves the catalog per RPC), write-authorized mutation +RPC and public SDK cutover remain gaps. See [progress](progress.md); CG-04 remains +IN_PROGRESS and no shared-group RustFS/load qualification is claimed. + +At `c25f0d97`, explicit nonmember read discovery uses that same query lane without +local group storage or voting identity. Four-endpoint TLS fixtures cover read +permission/process binding, revocation during held SQL, stale discovery replies, +route bounds and cleanup in both image modes. The partial gate passes 198 Rust +cases and eleven commands; transfer +`.cellar/cell-groups-evidence/CG-04-nonmember-queries/` (262 source inputs, +4 executables). Bounded destination caching, write-authorized RPC and public-host +automatic discovery/SDK binding remain open. CG-04 remains IN_PROGRESS. + +At `06652e36`, dispatcher clones share bounded range hints across reads, commands +and enrollment. Verified head changes invalidate hints; superseded lookups cannot +refill them. Three catalog cases and a both-image runtime case qualify eviction, +metadata-read reduction and catalog-outage behavior without weakening destination +admission. The partial gate passes 202 Rust cases and eleven commands; transfer +`.cellar/cell-groups-evidence/CG-04-dispatch-route-cache-final/` (265 source inputs, +4 executables). Write-authorized RPC and safe receipt lookup are next; removing +destination catalog reads requires replicated transfer seals. CG-04 stays IN_PROGRESS. + +At `5fd44437`, remote commands use explicit namespace/process write grants and +an authenticated command purpose on the bounded TLS RPC lane. Existing controller +owners retain admitted work after disconnect/revocation, and committed duplicates +return the same receipt without rerunning SQL. The partial gate passes 209 Rust +cases and eleven commands; transfer `.cellar/cell-groups-evidence/CG-04-command-rpc/` +(273 source inputs, 4 executables). Remote enrollment, standalone receipt lookup +and the public SDK cutover remain open; CG-04 remains IN_PROGRESS. + +At `0fe665d0`, remote enrollment shares the bounded command exchange, explicit +write grants and exact controller registration. Creation receipts survive later +mutations and retries; admitted initialization survives disconnect/revocation and +publication deadline. The partial gate passes 216 Rust cases and eleven commands; +transfer `.cellar/cell-groups-evidence/CG-04-enrollment-rpc/` (276 source inputs, +4 executables). Safe standalone receipt lookup and public SDK cutover remain; +CG-04 remains IN_PROGRESS. + +At `fbc15053`, standalone receipt lookup uses a quorum image without invoking +callbacks or releasing unresolved producers. It requires explicit command grants, +returns signed observations and keeps absence ambiguous. The partial gate passes +227 Rust cases and twelve commands; transfer +`.cellar/cell-groups-evidence/CG-04-receipt-rpc/` (279 source inputs, +4 executables). Dynamic group lifecycle/lazy birth and per-group failure +isolation are next; CG-04 remains IN_PROGRESS and public cutover is still pending. + +At `4326421a`, controllers admit bounded dynamic registrations and may start empty. +Catalog/root/page ownership, fresh quorum, response-independent attachment and +joined shutdown remain enforced. The partial gate passes 231 Rust cases and twelve +commands; transfer `.cellar/cell-groups-evidence/CG-04-dynamic-registration/` +(281 source inputs, 4 executables). Per-group failure isolation and +request-driven lazy birth are next; CG-04 and public cutover remain incomplete. + +At `41121a7d`, group-local failures quarantine only the owning group while retaining +its capacity and physical work. Real shared-engine faults still stop the node. +The partial gate passes 233 Rust cases and twelve commands; transfer +`.cellar/cell-groups-evidence/CG-04-group-quarantine-final/` (283 source inputs, +4 executables). Bounded request-driven provisioning is next; CG-04 remains open. + +At `69e41e86`, the controller provisions selected initial-member ranges with bounded +coalesced recovery, retained prepared owners and joined shutdown. Actual commands +and all-voter joined engine/page-store recovery pass in both image modes. The +partial gate passes 238 Rust cases and twelve commands; transfer +`.cellar/cell-groups-evidence/CG-04-local-provisioning-final/` (288 source inputs, +4 executables). The full scoped CG-04 gate below supplements this incremental +checkpoint. Production integration and shared RustFS/load remain unqualified. + +CG-04 is now DONE at `69e41e86` after all 83 declared cases pass individually and +the plan runner validates every required command. Transfer +`.cellar/cell-groups-evidence/CG-04-qualified/` and its supporting evidence +(98 distinct command records, 288 source inputs). CG-05 complete-group +snapshots and membership recovery is next. Public SDK/fleet cutover belongs to +CG-07; complete release qualification still requires CG-16. + +The implementation executor should deliver code, updated component contracts and +the required retained evidence for each task on the active PR. The final handoff +requires CG-16 and the release gate, including the real RustFS and independent +failure-domain profiles. Successful plan validation alone satisfies this design +deliverable, not those implementation or deployment gates. + +## Deliverables and source of truth + +| Artifact | Responsibility | +| --- | --- | +| `design.md` | Identities, routing, storage, APIs, compatibility and bounded ownership | +| `protocols.md` | Enrollment, read RPC, apply, snapshot/archive, transfer, hibernation and disaster contracts | +| `tasks.json` | Ordered implementation tasks, file ownership, steps, tests and invariant mapping | +| `implementation-plan.md` | Human index generated from the same tasks; `show` supplies complete briefs | +| `qualification.md`, `profiles.json` | Fault oracles, fixed workload contracts, numerical candidate gates and evidence format | +| `baseline.json` | Reviewed Cellar source hashes/excerpts and TiKV investigation pin | +| `plan.py`, `test_plan.py` | Executable package validator, drift checks, task briefs and negative validation tests | + +The complete scope includes managed fleet integration, effect identity preservation +through split/merge, and disaster recovery while a transfer is incomplete. These +cannot be dropped to make a fixed-group demonstration count as full delivery. +One-cell groups remain available through the same implementation for hot cells. + +The initial independent review corrected three delivery hazards: effects must be +adapted before public cutover; catalog admission tests precede real enrollment +integration in the host task; and remotely verified transfer closure is mandatory +before publishing split/merge ownership. The validator also binds benchmark cohorts +to exact binaries, controls and profiles instead of accepting aggregate pass flags. + +No supported density, performance improvement or hibernation latency is claimed +until the corresponding profile passes. Local containers do not prove independent +host or zone resilience. Historical one-cell measurements remain baseline evidence. + +## Execution conventions + +Use Rust 1.98.1 and locked Cargo builds at this baseline. Read current root and +subtree AGENTS.md instructions. This checkout uses a private workspace-volume +target directory, for example: + +```sh +export PATH="$HOME/.cargo/bin:$PATH" +export CARGO_TARGET_DIR="$HOME/Workspace/crabbuild-target/cellar-3617-cell-groups" +``` + +Keep `cellar-logic` free of I/O, async and dependencies. Use execution-domain +services and join owners on shutdown. Use a component `mod.rs` with named children, +or a leaf with sibling test files; never combine `feature.rs` and a Rust-filled +`feature/` directory. Match `native_raft_snapshot.rs` and `support/raft_cluster.rs` +for real-engine, native SQLite fixtures, rather than scheduler tests with stub +application markers. Do not enable `cellar_internal_tests` without its external +harness. Preserve license notices and keep credentials and raw cell data private. + +The parent `docs/design/multi-raft/plan.py validate` already rejects a repository-relative +component link in `010-large-cell-recovery-time.md`. This package has its own +repository-aware link validation; it does not suppress or claim to fix that +unrelated parent failure. Run normal repository docs checks as well. diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json new file mode 100644 index 000000000..477c29a7a --- /dev/null +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -0,0 +1,153 @@ +{ + "format": 1, + "source_revision": "bcf10c76458e97cc1696bddbebf85d9dfb7d32aa", + "runtime_revision": "bcf10c76458e97cc1696bddbebf85d9dfb7d32aa", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate. Reviewed successor 096cfb132397d9588ba9138b1cf474210fc062f3: verified group checkpoints now enter RawNode only through explicit step_checkpoint after envelope, destination, history, base, membership, bounds and engine/page-owner checks. Selected snapshots bind to the exact Ready and existing persistence ticket/lease; matching engine completion advances applied state and releases acknowledgement/CheckpointInstalled. Matching retained logs commit/replay without installing received roots. Five new native cases cover both image forms, held sync, shutdown join, invalid/overflow refusal and ignored snapshot replay; all twelve checkpoint cases passed before the final gate. Ordinary peer snapshot ingress remains closed. Local source checkpoint publication, bounded snapshot RPC, membership/disk replacement and full CG-05 qualification remain open. All anchored bytes and excerpt positions are unchanged. Reviewed successor 742b1e6d4e7155e16fb1b4eba0cce3b9056944ae: local group checkpoints use a distinct captured capability and SnapshotChange::CellGroupCheckpoint. Exact applied root/engine/configuration/membership checks reject mixed, stale and foreign writes. Storage compacts only the covered prefix and preserves hard/applied state and the entire suffix; existing retention pins protect current/pending/checkpoint roots through the engine fence. The owner persistence lane emits CheckpointPublished after matching completion; local compaction does not increment applied-state metrics. Exact outbound capture selects the retained Raft descriptor/closure after current advances. Three native cases passed for both image forms, actual RawNode-generated catch-up and suffix replay, cold leader engine/root recovery and shutdown during held publication. Materialized source recovery uses its fixture filename resolver; no RPC, automated policy, membership or full CG-05 claim. The CachedGroup excerpt is unchanged; reviewed cache.rs adds the typed local operation and preserves receiver/legacy branches. Only this anchor hash/line changed. Reviewed successor ceb7ed3149202660855d32df1ec5feed7775e068: dedicated bounded authenticated TLS checkpoint streams bind physical group/member/process identity and exact retained snapshots. Controller registrations revoke endpoints; physical source/receiver owners retain admission through cancellation. Verified receiver closure enters the existing exact Ready/persistence path. Reserved coalesced snapshot status admission prevents ordinary queue pressure from silently dropping feedback; actual failed transfer retry is exercised. Both image forms, suffix/cold recovery, invalid wire input, route revocation, dropped delivery and unrelated-group progress are covered by five native TLS cases. Existing anchored excerpts and hashes are unchanged. Automatic checkpoint policy, full receiver fault/pressure qualification, whole-group membership and disk replacement remain CG-05 work. Reviewed successor 1814b27896c3c21166186fe8fbd1a1a6bdbb721a: whole-group V2 configuration apply now requires an exact committed configuration entry and complete-root preparation in CachedGroup::changed. Pure replay independently validates the resulting ConfState and permanent bindings; the root stays unchanged and its term/membership revision match the entry. Existing legacy configuration/install checks and snapshot paths remain intact. Only the manifest anchor hash changed; its excerpt/line and all other source anchors are unchanged. Shared owner/runtime applies live Raft configuration only after sync, drives automatic joint-leave replication, supports caught-up voter handoff and fences demoted producers. Three new native cases and the 233-case regression gate pass; fresh learner seeding, lost-disk replacement, remaining CG-05 fault/pressure cuts and public cutover remain open. Reviewed successor ab2a308a937f349d3482a825d4b669d042cb03c3: authenticated fresh-group seed exchange binds exact physical identities and fresh ReadIndex source authority. Permanent node/engine/member claims stage a non-serving mapping; only an authenticated complete physical install atomically persists a local receipt with configuration, membership and current/retained roots. The CachedGroup anchor adds this local receipt; the required marker-1 group application record changes to manifest plus seed, with no old-layout reader. Generic snapshot imports cannot mint a receipt. Joined bootstrap revalidates permanent claims, active membership and receipt; same-owner physical recovery reconciles lost results without reopening pinned pages. Engine-queued work retains the grant and lease through actual sync. Joint-leave completion compares validated canonical configurations because live RawNode and pure replay can enumerate equivalent sets in different orders. Native tests delete voter 3 disk, refuse old identity reuse and unproven source/staged/generic startup, hold sync and cancel a waiter, seed/catch up/promote member 4, retire 3, transfer leadership, commit new state and cold recover both closures in both image forms. Test setup is split to bound debug future stack; no stack limit or safety assertion is relaxed. The seed-install fixture is added to CG-05 scope. All other source anchors are unchanged. Full CG-05 fault/pressure cases, managed fleet, RustFS/load/power-loss and zero-quorum qualification remain open. Reviewed successor df0ba1f67c0253a57127a215540f08bb13135279: namespace-owned group archive binding fixes the expected registry identity and canonical storage location before initialization; one-use authority and missing-counter refusal remain intact. Stable runtime checkpoint capture and upload_cut preserve the original committed root/configuration/membership across object-store I/O while quorum advances. The catalog file changes only the public range-directory reexports; the NamespaceConfig and BirthProposal excerpts are unchanged and each moves down one line. No other anchored file, excerpt, case inventory, oracle or admission quota changed. The first final plan unittest detected this unrefreshed catalog anchor after all Rust gates passed; its source-bound failed output is retained. CG-06 remains IN_PROGRESS and controller publication, S3 settlement/outage admission, local Raft file purge and fenced fresh-incarnation serving remain unqualified. Reviewed successor 0247fb8b951a9366bbabef0b0257280d358834e3: node controllers now own one fair, deduplicated dirty-group archive lane, exact-intent disk journals and recovery of uncertain publications. Abort plus second reconciliation fences delayed CAS before reallocation. New heads are journaled before predecessor hold retirement. Completion observes both current applied state and consumed apply events so recovering an old publication cannot lose an idle tail. A sealed execution capability uses captured tokens without shutdown authority and drains orphaned physical closures before verifier/journal/runtime teardown. Checkpoint cut races are typed NotReady; admission overload remains retryable. No anchored file/excerpt/hash changed. Shared S3 output settlement, unarchived-byte/disk admission, local Raft file purge, fenced fresh-incarnation serving, RustFS/load/process-kill qualification and public SDK/fleet cutover remain pending. CG-06 stays IN_PROGRESS; no original case, oracle or existing quota changed. Successor a4f37541cdc979252ea422d3dd5e224eb8b0ac2a removes only a duplicate integration-fixture module declaration caught by all-target Clippy. Runtime, assertions, oracles, anchored source bytes and quotas are unchanged; the failed source-bound gate output is retained. Successor 7914177d6f17e3f6d647a39dc496122157283591 preserves strict public checkpoint refusal after the unchanged prospective/stale-source regression exposed overbroad NotReady classification. Only the owner-private capture path can classify an already sealed root as transient when the engine has published a newer applied root. Prospective, stale public and foreign preparations remain Invalid. The full failed execution gate and executable are retained; its original assertion and all quotas remain unchanged. Reviewed successor dba41bda8d405264af2632c53c2d0d32894bdb8c: local command S3 settlement uses the existing controller archive lane and exact independently replayed receipts, including later cell images. A separate bounded S3 stage acquires retention before releasing resolved SQL output admission; original SQL and command quotas remain unchanged. Responses retain complete physical coverage, archive hold use and the exclusive journal lock, then recheck catalog birth and live leader authority. Original deadlines and callback identity remain fixed. A node budget of one qualifies deadline/cancellation isolation; missing sibling dependencies and foreign runtimes refuse. Receipt-lookup, enrollment and remote S3 output integration, unarchived-byte/disk admission, physical Raft purge, fenced fresh-incarnation serving and the public cutover remain pending. No anchored file/hash/excerpt or original acceptance oracle changed; CG-06 remains IN_PROGRESS. Successor 2b524f2a49978c3ac32acfbc4e08792a69f297fb boxes the retained private S3 proof bundle after Clippy identified large command preparation/settlement enums. Physical ownership, assertions, quotas and anchored source remain unchanged; the failed source-bound Clippy output is retained. Successor db78d771b5f3336bdebff4b5336e0d94067f08d4 shares immutable directory/birth authority per archive context instead of deep-cloning membership records into each response. Per-response S3 metadata remains fixed-size under its existing byte reservation; physical coverage, journal locking, output identity and all quotas/oracles are unchanged. The preceding six-case RustFS qualification and all-target Clippy pass at 2b524f2a remain retained; this successor is requalified separately. Successor 2e04d7509e2f9436d573adb656dfb911e07c20e2 adds positive namespace-minimum S3 qualification in native and real RustFS fixtures. Namespace setup now carries its actual configured minimum through catalog reopening; original Quorum defaults, production source and all original cases/quotas remain unchanged. Successor 2bf0dd2a6992954af41ff68722536064e4465a88 passes namespace minimum explicitly through the integration fixture after Clippy compilation rejected access to a crate-private getter. Production APIs, original defaults, assertions and quotas remain unchanged; the source-bound failed output is retained. Formatting-only successor 3d564f3db2c2eefc87119e7b3395f1bca17b99fa wraps the namespace archive fixture call using cargo format; production source, behavior, anchored bytes and quotas are unchanged. Local enrollment S3 responses now use the existing bounded archive lane and retained complete coverage: exact original creation provenance, namespace minimums, pre-initialization runtime ownership, post-proof catalog/leader/record checks, SQL-to-S3 retention transfer and deadline-safe resolved ownership. Remote enrollment still refuses unsupported S3 transmission. Existing anchor file hashes are unchanged; scoped native enrollment tests pass in both image forms. Full CG-06 acceptance remains pending. The source-bound all-target lint refusal at cc7171d3 retained its output; c2801b65 removes only two redundant fixture literal allocations. Production runtime bytes remain those of cc7171d3; cases, assertions and quotas are unchanged. Local S3 receipt lookup now seals a quorum selection, transfers SQL retention before queueing, reads the actual later archived observation, preserves typed failures/ambiguous absence and reserves payload bytes before fetch by growing the same S3 item. Remote requests/stored-minimum upgrades remain guarded on both ends, including provider-configured TLS gateways. The reproduced direct-caller domain panic at 7e1d93a0 retains its source/executable/backtrace; 214a1a13 scopes the recheck and passes all four new cases. The shared proof recheck now captures its domain and original deadline for all output kinds at 59619e0a. Original node SQL/S3 quotas and required cases/oracles are unchanged; the new expiry case deliberately fills a two-receipt ceiling. Existing anchor hashes are unchanged; full CG-06 acceptance remains pending. The source-bound macro lint refusal at 59619e0a is retained. cde0eea2 changes only the new cancellation fixture: poll the response and observe an archive attempt before cancellation, with unchanged node quotas and retained assertions. Reviewed successor 9eb958eda58841ebbc52c3b9d1df5debdc2cf96f: remote command/enrollment/receipt payloads preserve requested and achieved durability. A bounded owned reply retains the native S3 output through physical TLS flush; stream destruction precedes proof release on every cancellation path. Quorum output release, original deadlines, namespace/stored minima, exact receiver runtime ownership and mutation ambiguity remain enforced. Marker-1 hard cutover adds no legacy reader. New TLS fixtures reduce only real socket buffers and use the original 64 KiB S3 envelope, without changing original task cases/oracles or existing node quotas. Existing anchored files, excerpts and hashes are unchanged. Qualification remains pending. The initial source-bound Clippy refusal at 9eb958ed is retained. Successor 000e737ebc2a7517ee94722a532ea9159ceefd86 fixes only the new reply module logical import paths and waits for resolved ordinary controller admission before asserting same-cell progress with a held wire proof. Quotas, proof assertions and original cases are unchanged; qualification remains pending. The source-bound all-target refusal at 000e737e is retained. Successor 7d433bdbf6639d752aa692e8d0dd3a0bb993c2b1 fixes only two SQL-fixture router module references. Production runtime, assertions, original cases and quotas remain unchanged; qualification remains pending. Real RustFS source 7d433bdb passes remote output semantics and archive-wait deadlines/cancellation in both forms; its physical-delivery case and exact-executable fresh-prefix reproduction refuse before the response header. The independent raw probe encoded its operation string length as u32 while the production codec requires u16, decoding an empty operation. Successor 566e2e06884dfb44b3878f39a277e341800d113e corrects only that probe width. Production code, proof assertions, timeouts and all node quotas remain unchanged; the failed run/executable/backtrace and byte-level diagnosis are retained. Full remote qualification remains pending. The source-bound diagnostic at 79d6d1ae confirms that reading the TLS header prefetched the entire valid 29,226-byte S3 reply: transport query admission was zero and the body validated in 82 microseconds. The source 32bdcd2640f2f71dd38ea6b65b79b064bd324dd8 removes all temporary diagnostics and observes a partial encrypted socket write through the existing performance counter before any client read. Proof, 64 KiB admission, coverage, SQL release, same-cell Quorum progress, overload and cancellation assertions remain required. No production hook, deadline or quota changed; physical delivery qualification remains pending. The retained 32bdcd26 and 7b69250f RustFS probes confirm that a single undrained 30,422-byte encrypted reply fits kernel buffers even without a reader. Successor 25938fd86d13a2755066660797b07aff955ec1ad replaces that ineffective pressure trigger with a bounded window of sequential exact retries, waiting for each prior physical flush/refund before sending the next. A fixed host-lifetime physical query-reply write metric distinguishes write pressure from archive settlement and records success/failure/abandonment without durability authority. Original S3/SQL/command quotas, payload size, timeouts and proof assertions remain unchanged. All temporary DEBUG instrumentation is removed. Original task oracles and anchored source hashes/excerpts remain unchanged; qualification is pending. Runtime source 25938fd8 passes sixteen real RustFS cases in both image forms and 231 native/library regressions. The added exact metrics gate exposes a pre-existing main inventory mismatch: PR #159 added archive_current and archive_unconfirmed, while the test and API comment still expected 111 resource samples. The copied executable reproduces 113 versus 111. Validation successor debdc30c8327aad35757c6b849e029c1bbdab691 changes only the strict expected count to 113, explicitly requires both names, preserves 1-versus-32-group identity equality and corrects that documentation count. No runtime behavior, quota or original oracle changes. Unaffected source-bound passes are carried with their original runtime SHA and hashes; repaired metrics and source checks qualify separately. The remote S3 output runtime at 25938fd86d13a2755066660797b07aff955ec1ad with validation-only inventory successor debdc30c8327aad35757c6b849e029c1bbdab691 passes sixteen exact real RustFS cases in both image forms, seven signed protocol probes, twenty regression/check commands with 232 Rust cases across seven copied executables, strict runtime all-target Clippy, focused fixture Clippy and the independent source/artifact/selection/credential audit. The bounded exact-retry window proves actual held TLS writes without payload, timeout or original quota changes. Original task acceptance cases and oracles are unchanged; four original CG-06 cases and its full gate remain pending. Reviewed successor 66aa95e38fc30b0010bee70e7f6539745f99face: configured archival now persists an irreversible local archive_required bit before object-store I/O. Recovery restores that requirement but no verified frontier; independent complete coverage under the journaled remote hold reopens local compaction. The scheduler skips blocked captures while apply and sibling groups progress. Owner and storage preflight enforce the exact cut, including direct writes. Marker 1 now requires the local field, with no legacy decoder or migration; peer snapshots cannot clear it. The CachedGroup anchor adds that bool and its reviewed file hash/line are updated. No original task case, oracle or quota changes. Explicit unarchived-byte/disk admission, physical file purge, peer-install deployment policy, fenced fresh-quorum recovery and the public cutover remain pending. Qualification is pending. The source-bound all-target compile refusal at 66aa95e3 is retained: its new fixture passed a marker where capture needs the prepared capability and omitted OperationId. Successor ea995319971d023be37e4446c7b87eb467bff2c8 uses runtime capture, imports the explicit identity and orders the sibling checkpoint before retaining the unchanged one-source slot. Archive admission also checks the actual local birth before persisting policy. No original case, quota or assertion is relaxed; qualification remains pending. The retained all-target refusal at ea995319 identifies the new recovery test read guard despite explicit drop; successor 93b51065410952389f9345a4afa795d2d1971151 uses a lexical block before awaits. Review also found activation could retract an already admitted checkpoint while retaining its source slot. Owner and physical-storage admission guards now defer policy activation until that work drains; only this capacity refusal is retried at the activation boundary. The recovery test proves refusal before durable activation and success after the guard drains. Original quotas and cases are unchanged; all qualification remains pending. Native qualification at 93b51065 and a86f29e1 refused the invalid private metadata operation zero; copied reproduction and typed status identify the cause. Corrected private operation is nonzero, with public validation and physical sync sequencing preserved; qualification remains pending. Scoped archive-aware local compaction at 8dfe9ff3a316d0579b656af4818dc57d839a6765 passes seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 260 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes. The durable requirement and fresh verified frontier protect local compaction; admission guards drain earlier physical work before activation. Explicit retention admission, physical file purge, peer-install deployment policy and fenced fresh-quorum/public cutover remain pending. No original task case, oracle or quota was weakened. An earlier native slow-reader timeout remains retained and unexplained; unchanged isolated, ordered-controller and complete reruns pass. Activation fairness under sustained checkpoint traffic needs an adversarial gate before managed capture/admission. Reviewed successor ab0abf26dea4df436b5ced926f28a9ede4f360fd: the deterministic 79323600 regression shows pending activation can be overtaken by another checkpoint admission. Registration now owns an activation barrier across retries; owner and storage share an exact group/cut-bound lease so earlier work can finish. Entered metadata I/O owns activation after waiter loss; only synced cache/root publication makes the requirement permanent. Unentered rejection/shutdown releases intent, while existing durable policy remains irreversible. Frontier reset occurs once before I/O to avoid late duplicate completion discarding newer coverage. The existing 4 KiB checkpoint metadata charge bounds the cut lease. Serialized marker-1 records, original cases/oracles and quotas are unchanged; only the manifest anchor file hash/line changes with private GroupWrite owners. New physical-storage tests cover wrong lease bindings, old work draining, unentered cancellation and a held-sync lost waiter. Qualification is pending. Explicit retained-log admission, full-backend outage, physical purge, fresh-quorum recovery and hard cutover remain incomplete. Owned activation and cut-bound physical checkpoint drain at ab0abf26dea4df436b5ced926f28a9ede4f360fd pass three exact storage ownership cases, seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 262 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes; the original overtaking failure is retained. Sustained load fairness, explicit retained-log admission, full-backend outage, physical file purge and fenced fresh-quorum/public cutover remain pending. Original task cases/oracles, quotas and marker-1 serialization are unchanged. Reviewed candidate c2759d74fe652fccff52e6f3f5878a89263f4161: shared-group retained-entry admission owns node/group byte and entry reservations before SQL callbacks and Raft appends. A selected physical write atomically transfers exact entry-bound leases into actual durable debt; reopen reconstructs debt. Empty election/automatic joint-leave work has bounded separate progress capacity, while reads and existing completion/checkpoint work retain independent lanes. Callback envelopes shrink after encoding; sealed rejection and authoritative reconciliation retire unused preparation. Marker-1 serialization and original task oracles/quotas are unchanged. The new log limits are explicit and do not establish physical file amplification or full-backend outage authority. The manifest anchor excerpt is unchanged; its file hash/line reflect GroupWrite/GroupLog owners. New storage and native both-image boundary cases are not yet qualified. Full CG06/outage/purge/fresh-quorum/cutover requirements remain pending. The source-bound initial compile at c2759d74 refuses two private tracker-field accesses. Successor 2df640c995f1d9e478792ac64ebebc592909130c uses the public auto_leave() accessor in both checks. No limits, behavior, assertions, original cases or anchored hashes/excerpts change; all qualification remains pending. The retained successor compile at2df640c9 identifies the generated public getter returning &bool. Successor e090757f3c09ad3d90cd50a73609d3f171e81ea6 dereferences that flag in both checks. Limits, assertions and anchored source hashes remain unchanged; qualification is still pending. Retained all-target failure at e090757f identifies the new GroupLog pointer increasing the existing SQL Action enum past its size bound. Successor 1a8970840e2214bb88186e23dbd37effcbe80229 shares the ledger reference through the existing engine-local storage-policy allocation and retains compact GroupLog handles, without boxing unrelated SQL actions or suppressing Clippy. Group/cache ownership, entry binding, limits, original test assertions and serialized records are unchanged. The manifest anchor file hash/line is refreshed; qualification remains pending. Locked all-target Clippy passes at1a897084 with compact handles. Review successor b36ab33d10be7edf150882929c56cf43504660f1 restores execution-fence refresh immediately after four RawNode campaign/tick/step mutations, before new hash/accounting work. This preserves prompt role revocation to concurrent SQL workers; no limit, serialized record, assertion or anchored source hash changes. Runtime qualification and final-source Clippy remain pending. The exact retained-entry storage/reopen case passes atb36ab33d in the actual engine (0.09 s runtime; source-bound build/executable retained). Test successor 4e05c21430216ab12e5341244ec59c7c355f76fc drops every caller lease during held sync and uses Weak to prove physical-only ownership, then verifies consumed-cap refusal and no double debit with a surviving client lease. Both native image scenarios now fill ordinary admission with a valid auto-joint configuration and require automatic leave through the unchanged single progress slot. Production source, original limits and task oracles are unchanged; these stronger cases and final-source qualification remain pending. The retained4e05c214 wrong-term lease negative reproduces from its copied executable: its proposed term2 entry still used hardstate term1, so earlier cache validation refuses before lease checking. Successor 5238b9ac3e7cb07e3fb113642bc8f664d227f37a makes that proposed hardstate coherent, keeps the WrongIdentity assertion and adds selected-state invariance checks plus typed failure context. A new real-engine case isolates group/node byte ceilings with slack entry counts and verifies selected suffix replacement releases bytes. Production source and original task oracles/quotas are unchanged; these cases still need execution. Native fixture successor 462db264cac1a4e3f499b16227b905abc03329e0 replaces the impossible leader-only readiness wait on followers with exact selected checkpoint identity, zero retained-entry debt and released source quota on every voter. The unchanged copied5238b9ac native binary reproduces its deadline. Production code, quotas, deadline and original task oracles are unchanged. Three exact storage cases pass at5238b9ac; successor native and final-source qualification remain pending. Snapshot aggregate successor ad73f03f278f5845b311fe929d184cc947382b2d uses the existing non-inlined fixture constructor for TCP publication and install futures. The broad462db264 run and its unchanged copied binary reproduce a default-stack overflow in snapshot_empty_cell_and_extent_restart, while both constituent scenarios pass in the159-case execution suite. Stack size, quotas, deadlines, production code and every assertion/oracle are unchanged. The six-case focus, final-source Clippy, fresh RustFS and complete broad gate are pending; earlier passing focus/Clippy/RustFS evidence and original failure remain retained. Retained-entry admission at ad73f03f278f5845b311fe929d184cc947382b2d passes three real-engine focus cases, both SQL image-mode focus cases including ordinary-full automatic joint leave, seventeen real RustFS cases in both image forms, seven signed probes and twenty-two regression/check commands with 267 Rust cases. Independent source/artifact/selection/credential audit passes. Copied fixture failures remain retained. Logical byte/count debt admission is qualified at these boundaries; physical file bounds, full-backend outage authority, fenced fresh-quorum serving and public hard cutover remain pending. Original task cases/oracles/evidence array and marker-1 records are unchanged. Physical-reclamation qualification candidate 0ecd34db504104354adc034c2959e268696790b7 adds the original group_reclaim_restart_keeps_history case in both image modes and a real RustFS counterpart. Explicit test geometry uses 4KiB Raft files, 8KiB purge threshold and manual purge only in this scenario; existing fixture defaults/quotas remain unchanged. Actual files and length reduction, an uncheckpointed sibling, retained suffix, old reader, checkpoint streaming, bounded image/page collection, all3 retained-disk reopen/fresh quorum and exact receipt/effect recovery are asserted. No production behavior, marker or original acceptance oracle changes; execution and qualification remain pending. Successor 2d9be1b52d39223f9ec71d85b42fb3c16276c004 corrects test-helper visibility to its enclosing runtime fixture and adds two fresh post-restart TLS commands after all old receipt retries, checking every voter image/effect. The source-bound0ecd34db compile failure is retained; no runtime case executed at that revision. Production code, original oracle and resource limits remain unchanged. Execution and final qualification are pending. Bounded-reader successor fae2f9e341e490de841eed070c68b9080c0b52a9 preserves the complete image oracle but reads materialized images in at-most64KiB chunks. The exact2d9be1b5 case and its copied executable reproduce GenerationError::Limit; GenerationReader::read_exact_at rejects any larger read. No read budget, payload, receipt count, physical assertion, resource limit or production code changed. All qualification remains pending. Checkpoint-target successor 51a846be978c296c491509a997e8a91703b0f838 verifies post-purge retained checkpoint data through the private archive receiver, which grants no engine/vote/serving capability, and explicitly asserts that an ahead live voter still refuses the older cut. Separate one-shot source transfers retain both original one-slot pool limits. The copiedfae2f9e3 binary reproduces the earlier inappropriate live-target refusal before dependency bytes. Original rollback guard, complete data assertions, quotas and production behavior are unchanged; full qualification remains pending. Verification-lifetime successor 58d6559956eb7d37082a7bf066a21cda0df70945 retains independent coverage through actual purge/checkpoint checks, asserts its occupied one-slot archive receiver budget, then drops that finished result before waiting for a new archived cut. The journal retains the remote hold. The exact51a846be case and copied executable reproduce the prior circular wait; ownership types show the held verification permit. No quota, deadline, original assertion, payload or production behavior changes; full qualification remains pending. Selection-parser successor 9e7246b322b45e1d0f2d03a53ba387df4d2a0511 normalizes only the two validated structured observation lines that interrupt libtest progress, preserving exact case matching and the one-pass/zero-failure/zero-ignore summary requirement. The actual58d65599 Rust case passes both image modes, but its wrapper reported selection failure with exitcode0 because those notes split the result line. Raw output, executable and original record stay immutable. No Rust code or test assertion changes. Final qualification is pending. Diagnostic-output successor ea2717b62457f5038fedd6bf4dc54c22f464af16 uses the existing cellar::note API for native qualification observations instead of forbidden stdprintln, following the metrics fixture convention. The exact9e7246b3 native case passes in both modes; its all-target Clippy failure remains source-bound. No physical/recovery behavior, assertion, payload, resource limit or deadline changes. Final-source qualification remains pending. Shared-module successor 37f3def8e06c4c05ff1bd605d77db0062feaa3bb declares the archive-store fixture once in each integration target and reuses it inside the controller harness. Adding runtime support to the archive target had loaded the same file twice; all-target Clippy at ea2717b6 refused duplicate_mod. No lint suppression, runtime/resource change or original assertion changes. The source-bound passing exact ea case and lint failure remain retained; final-source qualification is pending. Owning-module successor ea2f802475aa849a1b00f0990eff6929beb4f207 places the archive-only reclamation scenario under the existing archive harness and reexports its boxed off-frame constructor, preserving target-specific module ownership. All-target Clippy at37f3def8 had refused the unused standalone helper in execution-only targets. No production code, existing target lint policy, original assertion, quota or deadline changes. Final-source qualification remains pending. Visibility successor 7741e4049704f3e1c6519b736e62deaf263c39a2 gives the SQL fixture reexport crate visibility, as required to expose the archive scenario through the runtime fixture to its integration target. The ea2f8024 compiler refusal and exact source/output remain retained. No scenario behavior or production code changed. Final-source qualification remains pending. Physical-reclamation qualification at 7741e4049704f3e1c6519b736e62deaf263c39a2 passes the original native case in both image forms, eighteen real RustFS cases/seven signed probes and twenty-two checks with268 Rust cases, seven copied executables and2575 source hashes. Actual old files disappear and aggregate file length falls while readers, checkpoints, suffixes and siblings survive; all3 retained voters reopen, all24 receipts resolve without callbacks and two new TLS commands commit. Independent source/artifact/selection/credential audit passes. Original task cases/oracles/evidence array, production behavior and marker1 records are unchanged. Three original CG06 cases/full gate, full-backend outage authority, fenced fresh-incarnation serving and public hard cutover remain pending. Reviewed complete-group dependency successor dc0ad3c245a2f9e3dd4f261003c868e9a43f19b8: adds the original CG06 dependency acknowledgement gate and realRustFS counterpart with delayed/corrupt actual B image bytes unique from A, actual local quorum successors, persisted independent recovery holds, complete recovery refusal and exact receipt checks. The recorded CG06 scope extension adds the owning sibling fixture plus existing runtime/SQL/S3 test export paths. No production source, anchored excerpt/hash, marker, original oracle/invariant, quota or deadline changes. Final-source qualification remains pending. Complete-group dependency qualification at dc0ad3c245a2f9e3dd4f261003c868e9a43f19b8 passes the original delayed/corrupt B image oracle in both image modes, nineteen real RustFS cases/seven probes and twenty-two checks with269 Rust cases/seven copied executables/2576 sources. Actual B data differs from A, later A S3 results wait or fail, Quorum progresses, stored selection stays before A, independent held recovery blocks or refuses the exact digest failure, and exact receipts never replay callbacks. Independent source/artifact/selection/credential audit passes. Production behavior, existing quotas/deadlines, original case oracles/invariants and marker1 records remain unchanged. Two original CG06 cases/full task gate, public hard cutover and full end-to-end goal remain pending. Reviewed full-backend outage successor cfbeaab04be0df734128668402e7388bc34b949f: adds the original bounded outage gate in both image modes, a real RustFS stop/start counterpart and an owner-checked Python controller with fixed operations, pinned container/resource/mount checks and joined child/cleanup. Existing fixture constructors retain default settings while the new scenario selects nine entries with one progress slot. Local cached quorum, fresh remote refusal, exact retained invocation inputs/deadline/budget, S3 minima, admission before callbacks/initializers, complete recovery and checkpoint debt retirement are asserted. The recorded scope extension adds the owning outage/control fixtures and runner/controller tests. Production source, anchored excerpts/hashes, marker1, original oracles/invariants and existing quotas/deadlines are unchanged. Final-source qualification remains pending. Inventory successor b0e2dbafec4dab9cdc91345bc9ff847635e4ebf5 replaces an incorrect borrowed four-entry fixture assumption with exact retained indexes2,3,4 and equality to ledger debt on every voter. Registered setup enrolls two cells; the preceding private retention fixture enrolls three. The copied cfbeaab0 executable reproduces the initial mismatch. The scenario now fills six registered commands to the unchanged entry ceiling9 and refuses command7 before callback, preserving all timeouts, quotas, payloads, actual history/data and original oracle. Production behavior is unchanged. Qualification remains pending. Data-readiness successor 1159e9358c106fe1260c3479759e909e413257a4 handles the observed first post-start S3 GET503 after container health200 by retrying only idempotent Generic read failures under the existing retained remote invocation absolute deadline. Missing/auth failures still refuse and exact probe data must match; noRetry outage adapter, mutating retry rules, original deadlines/limits/oracle and production source are unchanged. The original b0e2dbaf realRustFS failure and copied-binary reproduction on a fresh owned backend both retain exact503 output, container transitions and joined cleanup with stopped backend data. Native/Clippy passed atb0; final-source full qualification remains pending. Full-backend outage qualification at 1159e9358c106fe1260c3479759e909e413257a4 passes the original case in both modes, twenty real RustFS cases/seven signed probes/four pinned owner-checked stop-start transitions and twenty-three checks with270 Rust cases/seven executables/2580 source hashes, 37 plan tests and five controller tests. Warm local Quorum reaches the fixed entry9/progress1 boundary; fresh remote admission refuses beforecallback; original inputs/deadline/minima persist; complete recovery permits exact held checkpoints, all-voter debt retirement and resumed writes without replay. Independent source/artifact/selection/credential/backend-transition audit passes. The failed cfbeaab0 inventory assumption and copied reproduction remain retained. Production source, original oracles/invariants, existing quotas/timeouts and marker1 records remain unchanged. Fenced fresh-incarnation serving/fullCG06, public cutover and remaining full goal are incomplete. 2026-11-01: reviewed the 1159e935 runtime and 8e5ca2fc handoff for the next CG-06 recovery boundary. The successor adds a typed whole-group exact intent, mandatory marker-1 nullable recovery lock in the directory head, fleet-import exclusion and independently verified durable fencing. This is a hard record-shape cutover, with no legacy reader. Static unmanaged fencing/data proofs grant no replacement bootstrap or serving authority; fresh history/image/receipt rebinding, installation, activation and Docker deleted-voter qualification remain required. Original task oracles/invariants/status evidence are unchanged. 2026-11-01: reviewed the recovery-fence successor at 65c5176975907fb52a53deca58e1ae1f5ba7adec. Whole-group intent/fence barriers remain data-only, with fresh-genesis serving still required. The broad retained learner drill exposed a deterministic default-stack regression relative to the copied 1159e935 executable. Boxing rare recovery-lock metadata restores its exact default-stack test with identical canonical record bytes. Final broad/RustFS qualification is recorded separately; no task oracle or status evidence is relaxed. 2026-11-01: reviewed 65c51769 runtime / 9f3b5b86 published fence barrier for whole-group fresh-genesis restoration. SQLite receipts/client floors contain no old version authority; CellCommittedReceipt derives its covering version/witness from the selected leaf. The successor must preserve SQLite bytes, create fresh leaf histories and a durable pending/activated recovery lifecycle, and keep every application admission path closed before committed activation. Canonical recovery-intent/source holds retain original witnesses as provenance. Marker-1 root shape is a hard cutover, without legacy readers. Original CG-06 oracle/status and all sixteen tasks remain unchanged. 2026-11-01: Reviewed the candidate recovery lifecycle cache guards from 9f3b5b86. Applied roots preserve the exact recovery lineage; authenticated empty learner installation is the only snapshot acquisition boundary. The anchored CachedGroup shape is unchanged. Mandatory nullable recovery metadata stays at marker 1 with no legacy default. Candidate import/lifecycle qualification remains pending. Recovery import and lifecycle source committed at 7052d218d60181d7b0bb5a4c12686e80e294a61c; focused recovery tests pass. Full scoped native and RustFS qualification is pending. Scoped qualification at 7052d218d60181d7b0bb5a4c12686e80e294a61c now passes: seven native suites (246 cases), 36 recovery-related unit tests, eleven legacy disaster cases, real RustFS import in both image forms and all recorded static checks. Evidence CG-06-recovery-import/qualification/record.json SHA256 1c7ea62c6bd5c1e81357961c4acba22bb2820d89bc3556ec3bcaf82a43e1c0a3. The original fresh-serving oracle/full CG-06 gate and all later tasks remain pending. MVP continuation from 973cad34/7052d218: reviewed node storage/genesis, immutable mappings, physical retention and directory birth reservation before edits. Seed genesis must bind the actual imported root and local node/disk/process/engine through a catalog permit. The global reservation now records mandatory nullable recovery origin, so missing optional seed metadata cannot fall through to ordinary startup. New wire kinds remain marker1 with no legacy reader; candidate qualification is pending. Reviewed fresh-genesis/activation successor from 973cad34: actual sealed engine seed grants, atomic complete mapping/root publication, permanent recovery-origin and physical seed claims, locked topology publication, recovered startup and catalog-bound replicated activation. The CachedGroup anchor now includes immutable mapping-derived recovery genesis shared behind the existing cache; GroupLog handles retain their previous size. Cache/mapping equality preserves this lineage through writes and reopen. Ordinary startup refuses recovery origins; pending reads refuse before wait admission. The owner independently derives activation from the committed entry and preserves lifecycle on other commands. Marker1 envelopes use hard cutovers only. Source/test review and all-target Clippy pass; final-source native/RustFS qualification remains pending. Original task oracles/status/evidence are unchanged. Runtime source pinned at 12944b762631c300ec6da2a8f6528394a19e4eda after all-target Clippy passed. Final-source native/RustFS regression is pending; this is not the original fifth oracle or completed CG06. Reviewed successor 2c4326d5cf29f3935d4859e97a52af0ee08b170a: a deterministic stale-B observation regression on unchanged 12944 runtime showed reconciliation returning terminal Unavailable. Only reconciliation NotReady now returns Uncertain, retaining the original producer and requiring a fresh proof; identity/closed-engine/newer-term absence checks remain intact. The existing stale-proof assertion is strengthened to exact Uncertain plus retained producers. Registered-query qualification joins actual physical/queue cancellation ownership before another one-slot admission, without changing original quotas, deadlines or result assertions. Four reconciliation, two registered-query and one receipt-ingress cases pass at these exact source bytes, as do all-target Clippy, formatting and layout. The two earlier intermittent full-suite failure locations are not causally proven; their copied-executable isolated passes and deterministic red regression remain retained. Final-source broad native/RustFS and regenerated original CG01-05 gates remain pending. No original task oracle, evidence requirement or status is weakened. Reviewed 9a0bad9124f0089055a0ba63b2952e5ef5e77853: physical genesis/activation and registered receipt recovery from archived LTX tails pass the scoped native/RustFS checks; Docker fencing remains pending. Integrated main 81cbbc2f72f302751dd471cf06cf51394146ac63: legacy message-delivery ownership now lasts through transport completion before membership apply, and CI uses macOS 15. The host/runtime/ticks/CI excerpts are unchanged; reviewed file hashes and runtime line offsets are refreshed. No cell-group protocol anchor or task oracle changed. See progress.md for scope and evidence availability. Reviewed successor 51d136fbea6732cb93553c1a57cd45547ca9aa59: physical engine close now releases only derived manifest pins after the closed fence and engine join, before publishing its terminal result. Unwind cleanup releases these pins too; external readers retain their own owners. Both retained-closed-handle recovery regressions fail at 5e7aadb6 with image/extent directory-lock refusal, then pass with this fix. Twelve checks pass, including full execution/archive/effects/namespace-host/controller-archive native suites and strict all-target Clippy; no new RustFS qualification is claimed. CachedGroup fields/excerpt are unchanged and move by 25 lines. Canonical SDK/fleet cutover remains uncompiled local work and CG-07 remains IN_PROGRESS. Reviewed successor 66d5ea6aa4ff5f2b2753f17c2cf4f7bf3c706136: bounded physical range-directory enumeration uses immutable authenticated tree paths and an exact-view cursor, without cell scans, S3 listing, mutation, voter or serving authority. The six-file checkpoint passes its exact native case, all nineteen catalog cases, strict all-target Clippy, formatting/layout, native dependency and docs checks. The missing initial catalog re-export failure is retained before any test execution. The only anchored file change is the public cursor/page re-export in catalog.rs; NamespaceConfig and BirthProposal excerpts remain unchanged and unique, with reviewed line shifts. This is metadata inventory only: no canonical SDK/fleet cutover, managed runtime, new RustFS, load or final release qualification is claimed. Native record SHA256 8061d03d7902d21f291fb82b16bcf04c91d20716465ef5292eecd7c754dcd91a. Reviewed successor bcf10c76458e97cc1696bddbebf85d9dfb7d32aa on 2026-11-06. Canonical host startup takes namespace capabilities and physical group assignments. The Fleet root coordinates transition admission, parking and managed recovery. The durable cache retains local park metadata and preserves seal and transfer proofs through apply and snapshot. Namespace descriptors are selected before managed publication, and retained startup uses the rooted voter set. Predecessor protocol, runtime and archive excerpts remain as cutover context. The harness now charges and observes physical groups by occupancy. Native and SDK campaigns retain separate gates. The updated baseline records reviewed source inputs. Task evidence, acceptance oracles and release gates remain separate. The host anchor keeps the reviewed host options path shared by the recorded component gates. The current startup implementation lives in group_host.", + "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", + "anchors": { + "identity": { + "path": "crates/cellar-logic/consensus.rs", + "sha256": "624ccbc21e7c83a8039444fa243d9e9a9eca12da8e32bf14e46893f8a44a83a8", + "line": 83, + "excerpt": "pub struct GroupScope {\n pub cluster: ClusterId,\n pub namespace: NamespaceId,\n pub cell: String,\n pub group: GroupId,\n pub incarnation: IncarnationId,\n}\n\nimpl GroupScope {" + }, + "codec": { + "path": "crates/cellar/application/consensus/protocol.rs", + "sha256": "a2796d1c2af263cf5c8ccd80f98da54121016a91daf2f0bf60064f421d498aca", + "line": 160, + "excerpt": "pub struct CellUpdate {\n pub scope: GroupScope,\n pub application_digest: [u8; 32],\n pub proposal: ProposalId,\n pub base: LtxImage,\n pub result: LtxImage,\n pub files: Vec,\n}\n" + }, + "manifest": { + "path": "crates/cellar/durability/raft/cache.rs", + "sha256": "cedd77aaa0b0f0125fbbc233d923aba6fba681cfaef43c4d4ac2e213f59ec005", + "line": 501, + "excerpt": "pub(crate) struct CachedGroup {\n pub archive_required: bool,\n pub hard_state: HardState,\n pub conf_state: ConfState,\n pub membership: GroupMembership,\n pub snapshot: Snapshot,\n pub entries: BTreeMap,\n pub applied: DurableApply,\n pub group_manifest: Option,\n /// Derived from the immutable mapping; never supplied by a write/snapshot.\n pub recovery_genesis: Option>,\n pub manifest_pins: Option,\n pub checkpoint_manifest: Option,\n /// Local receipt; never imported from another node's checkpoint.\n pub group_seed: Option,\n pub(crate) local_park: Option,\n}" + }, + "apply": { + "path": "crates/cellar/durability/raft/apply.rs", + "sha256": "22dacc21f18449945d61ef2553621500512691faaf2da87c697262f6b4dcc3a1", + "line": 80, + "excerpt": "pub struct ImageApplier {\n group: GroupLog,\n images: GenerationStore,\n application_digest: [u8; 32],\n limits: EnvelopeLimits,\n current: Option,\n snapshot_pin: Option,\n pending: Option,\n}" + }, + "extent": { + "path": "crates/cellar/durability/raft/rooted/marker.rs", + "sha256": "b354ad8c64957569d21e8002b4f26082c41581140513fd31827e703c15076d43", + "line": 59, + "excerpt": "pub(super) fn adopt(\n group: &GroupLog,\n store: &NodeExtentStore,\n schema: [u8; 32],\n marker: &CommittedImage,\n max_image_bytes: u64,\n buffer: &mut [u8],\n) -> Result {\n marker.validate().map_err(ImageApplyError::Log)?;" + }, + "host": { + "path": "crates/cellar/durability/raft/host.rs", + "sha256": "7a7e2c538ba026459dc58b1bef4bbb622dc2e64b1fd0c87d33477af0447414cf", + "line": 71, + "excerpt": "pub struct RaftHostOptions {\n /// Shared inventory and caches for namespaces selecting extent images.\n pub extents: Option,\n pub close_report: super::operations::RaftCloseReportLimits,\n /// Explicit committed alarm/outbox scheduling. Definitions refuse activation when\n /// this capability is absent; ordinary classes do not start a scheduler.\n pub effects: Option,\n /// Opt in to bounded automatic repair and optional leadership balancing.\n pub controller: Option,\n /// Bounded complete node-wide resource scans. Peer inspection still requires\n /// its independent explicit profile and grants.\n pub inspection_scan: RaftInspectionScanOptions," + }, + "sql": { + "path": "crates/cellar/durability/raft/host_sql/mod.rs", + "sha256": "bb822ccd381c8d5532641047aeb61f4b04872fabbcaa2611dcc891cd71473640", + "line": 871, + "excerpt": "struct Cell {\n group: GroupLog,\n class: Arc,\n images: ImageOwner,\n snapshot: Option<(ApplyTicket, PreparedSnapshot, Permit)>,\n private: Option,\n active: Option,\n}\n" + }, + "runtime": { + "path": "crates/cellar/durability/raft/runtime.rs", + "sha256": "dd0ff874a3767998112d1713f712146d9cbe2102590a6727ec02588d9e017c37", + "line": 759, + "excerpt": "struct Group {\n replica: Replica,\n scheduled: bool,\n write_scheduled: bool,\n deferred: VecDeque,\n inbound: Arc,\n hints: BTreeMap,\n tick: bool,\n failed: Option," + }, + "ticks": { + "path": "crates/cellar/durability/raft/runtime.rs", + "sha256": "dd0ff874a3767998112d1713f712146d9cbe2102590a6727ec02588d9e017c37", + "line": 1084, + "excerpt": " fn tick(&mut self) {\n for (scope, group) in &mut self.groups {\n group.tick = true;\n group.wake(scope, &mut self.order);\n }\n }\n\n fn progress(&mut self) -> Result {\n let mut made = false;" + }, + "catalog": { + "path": "crates/cellar/durability/raft/catalog.rs", + "sha256": "f00771120f7712a9d9e342bdfbf56a9fb65f797d48ee6aaa6acc7e8a23bf99e6", + "line": 144, + "excerpt": "pub struct NamespaceConfig {\n pub cluster: ClusterId,\n pub namespace: NamespaceId,\n pub mode: NamespaceMode,\n pub schema_digest: [u8; 32],\n pub minimum_durability: Durability,\n pub voters: VoterPolicy,\n /// Immutable birth-admission owner. Managed namespaces refuse the ordinary\n /// creation API; each birth needs an inventoried fleet reservation." + }, + "birth": { + "path": "crates/cellar/durability/raft/catalog.rs", + "sha256": "f00771120f7712a9d9e342bdfbf56a9fb65f797d48ee6aaa6acc7e8a23bf99e6", + "line": 213, + "excerpt": "pub struct BirthProposal {\n pub scope: GroupScope,\n pub voters: Vec,\n pub learners: Vec,\n}\n\n/// Constructed only after a canonical immutable record and its reservation have\n/// been read back from the selected namespace. Its placement never overrides\n/// committed Raft configuration once a member has initialized." + }, + "version": { + "path": "crates/cellar/durability/raft/version.rs", + "sha256": "1b5294e02c9a5299e752fa5656ee153876d1b76fa5ffaa06c7da2d53defe53fe", + "line": 73, + "excerpt": "pub enum VersionError {\n Invalid,\n Expired,\n WrongHistory,\n WrongSchema,\n KeyRetention,\n Limit,\n}\n" + }, + "snapshot": { + "path": "crates/cellar/durability/raft/snapshot.rs", + "sha256": "1882395d832cc76b2233d3e9d223105b2ee03da44e5c241f04f339238aa145a3", + "line": 296, + "excerpt": "pub struct SnapshotPackage {\n group: GroupLog,\n snapshot: Snapshot,\n bytes: Option,\n root: Option,\n pin: Option,\n limits: SnapshotLimits,\n membership: super::member_state::GroupMembership,\n}" + }, + "archive": { + "path": "crates/cellar/durability/raft/archive/mod.rs", + "sha256": "a1c600e10b44c93b19fbeb14b278f5300cb908ed290f12299651f6dbb5350ef5", + "line": 343, + "excerpt": "pub struct CellArchive {\n coordination: Option,\n activity: Option,\n snapshot_scratch: stream_snapshot::ScratchPool,\n scratch_directory: Option,\n store: Arc,\n prefix: String,\n bundle_prefix: String,\n scope: GroupScope," + }, + "disaster": { + "path": "crates/cellar/durability/raft/disaster.rs", + "sha256": "8214156072b8fe4296869223596defbf75660f676635a85071b41d843a71c81c", + "line": 17, + "excerpt": "pub struct S3DisasterRecovery {\n namespace: NamespaceCatalog,\n host: cellar_ltx::LtxHost,\n limits: ArchiveLimits,\n}\n\nimpl S3DisasterRecovery {\n pub async fn open(\n store: Arc," + }, + "membership": { + "path": "crates/cellar/durability/raft/host_membership.rs", + "sha256": "ece36f2f8d1f0f099cfa3b56cc6214d60070bc9e9bc42469cdbc90c42b15521d", + "line": 15, + "excerpt": "pub(super) enum Goal {\n Fleet {\n binding: Box,\n fence: Option>,\n },\n Learner(JoiningMember),\n Repair {\n plan: RepairPlan,\n phase: RepairPhase," + }, + "effects": { + "path": "crates/cellar/durability/raft/host_effects.rs", + "sha256": "4d1b90c9dde246847b74c0b4479604f6e4f0de7ae33ace04ee1d6ac8772109ff", + "line": 8, + "excerpt": "pub(super) struct Scheduler {\n options: RaftEffectOptions,\n cursor: Option,\n next_at: u64,\n sequence: u64,\n busy: Option,\n enabled: bool,\n memory: Arc,\n}" + }, + "placement": { + "path": "crates/cellar/durability/raft/placement.rs", + "sha256": "f0ccb5cb44b7d98e8b5a6df507196cb8f5bc4da078f8176edcefa8e91ea9e7d4", + "line": 11, + "excerpt": "pub(super) struct Context<'a> {\n pub cells: &'a BTreeMap,\n pub peers: &'a BTreeMap,\n pub local: NodeId,\n pub rng: &'a mut crate::asyncrt::rng::Stream,\n}\n\nstruct Seen<'a> {\n view: &'a CellInspection," + }, + "fleet": { + "path": "crates/cellar/durability/raft/fleet_catalog/mod.rs", + "sha256": "7b20f0c1b58b3aa57d32c7e63bbc916a43e8851dde9b376ede13b2836809227f", + "line": 24, + "excerpt": "mod managed_transition_admission;\npub(crate) use managed_transition_admission::TransitionOperatorStep;\nmod managed_recovery_fence;\npub(crate) use managed_recovery_fence::PreparedManagedRecoveryFence;\npub use managed_recovery_fence::{\n FleetFencedTransitionRecovery, FleetTransitionRecoveryFenceRequest,\n};\nmod managed_recovery_intent;\npub use managed_recovery_intent::{FleetTransitionRecoveryBarrier, FleetTransitionRecoveryIntent};\npub(crate) use managed_recovery_intent::{ManagedRecoveryIntentBasis, ManagedRecoveryIntentInput};" + }, + "public": { + "path": "crates/cellar/durability/raft/host_node.rs", + "sha256": "91f1c8edf32faf1772ed42d7f3325f6f0549509f3215b83998bc2581fc6a3c12", + "line": 39, + "excerpt": "pub struct RaftHostObservation {\n pub node: super::super::inspection::NodeInspection,\n /// Synced physical boot, independent of a caller-provided process hint.\n pub fleet_boot: Option,\n pub startup_bounds: RaftHostStartupBounds,\n /// Rooted observation used by this boot, never a fresh eligibility proof.\n pub startup_enrollment: Option,\n /// Enrollment in the installed current trust cut. It may differ from boot.\n pub current_enrollment: Option," + }, + "fixture": { + "path": "crates/cellar/tests/native_raft_snapshot.rs", + "sha256": "f7f38fe3bee61c932fec7743eccece156eb44d9046d5422856127263e95e9721", + "line": 91, + "excerpt": "async fn native_extent_fresh_learner_peer_seed_resumes_and_catches_up() -> Result<()> {\n native_seed_drill(\n false,\n false,\n false,\n SeedScenario::Ordinary,\n RaftImageStorage::Extents,\n )\n .await" + }, + "harness": { + "path": "tools/perf/raft_fleet.py", + "sha256": "9a3d81a1083b63f7374264e27d510a0cb13d85d5e9928f440602688e74e2629b", + "line": 258, + "excerpt": "def main():\n parser = argparse.ArgumentParser(description=__doc__)\n commands = parser.add_subparsers(dest='mode', required=True)\n setup = commands.add_parser('provision')\n setup.add_argument('--context', default='colima-cellar-raft-perf')\n setup.add_argument('--image', required=True)\n setup.add_argument('--policy', type=Path, required=True)\n setup.add_argument('--output', type=Path, required=True)\n setup.add_argument('--backing-disk', type=Path, default=Path.home()" + }, + "recovery": { + "path": "tools/perf/process_recovery.py", + "sha256": "f95cc6a69ef88f214314a50a59428e98140258a0d9644476eb0e78da3af9c9c4", + "line": 130, + "excerpt": "def main():\n parser = argparse.ArgumentParser(description=__doc__)\n parser.add_argument('--scenario', choices=['owner-loss', 'quorum-loss'], default='owner-loss')\n parser.add_argument('--profile', choices=['272mib', '10gb'], required=True)\n parser.add_argument('--output', type=Path, required=True)\n parser.add_argument('--work-directory', type=Path, required=True)\n parser.add_argument('--store-data-directory', type=Path, required=True)\n parser.add_argument('--fixture', type=Path)\n parser.add_argument('--binary', type=Path)" + }, + "ci": { + "path": ".github/workflows/native-sdk.yml", + "sha256": "561e170bfcb193754a6166aedfbd04678f58370aef07057b0013e20cda0a6c0d", + "line": 67, + "excerpt": " RUST_VERSION: 1.98.1\n # Process tests publish executable artifacts; DWARF makes those fixtures huge.\n # Keep debug assertions and test behavior while omitting symbol information.\n CARGO_PROFILE_DEV_DEBUG: \"0\"\n CARGO_PROFILE_TEST_DEBUG: \"0\"\n\njobs:\n filesystem_release:\n if: github.event_name != 'workflow_dispatch' || inputs.mode == 'qualification'" + } + } +} diff --git a/docs/design/multi-raft/cell-groups/design.md b/docs/design/multi-raft/cell-groups/design.md new file mode 100644 index 000000000..b11d981da --- /dev/null +++ b/docs/design/multi-raft/cell-groups/design.md @@ -0,0 +1,260 @@ + + +# Cell group design + +Purpose: Physical group consensus over independent SQLite cells, identities and hard cutover. + +Status: target contract for CG-01 through CG-16; implementation in progress. +See [implementation progress](progress.md); no runtime cutover is qualified yet. +Cellar baseline: `1be27d234d4674d4d014785c60cecbfe394bdab9`. +TiKV investigation: `b3995dab28308f09cbb7c125bf32f789dbc203fe` in the supplied +checkout. The [decision](../density-decision.md) records the inspected source. + +## Architecture and scope + +A cell group owns one Raft state machine on each of its replicas. The group +contains whole cells; every serving replica has each cell's committed SQLite +state. Only the leader executes cell commands. Followers apply cell-addressed LTX +updates. Independent groups share node workers, persistent log batches, transport +and resource budgets. Three voting replicas remain the production default. + +```mermaid +flowchart LR + Route[Cell router] --> L[Group 7 leader on node A] + L --> A[Cell X SQLite] + L --> B[Cell Y SQLite] + L --> C[Cell Z SQLite] + L <-->|one group log| F[Group 7 follower on node B] + L <-->|one group log| G[Group 7 follower on node C] + F --> FC[SQLite replicas X Y Z] + G --> GC[SQLite replicas X Y Z] + L --> S3[Complete group checkpoint and tail in S3] +``` + +Cells share placement, leadership and log order, but keep separate transactions, +receipt tables, LTX chains, schema checks, authorization and query versions. +Neither cross-cell transactions nor intra-cell sharding are introduced. One hot +cell can occupy one group; all group sizes use identical consensus machinery. +The feature-off Fleet/bucket protocol and non-Raft applications keep their contract. + +Full delivery includes split/merge, automatic placement, coordinated hibernation, +materialized and extent images, managed and unmanaged namespaces, native/Axum +ingress, alarms/outbox/inbox, group S3 reclamation and fenced disaster recovery. +TiKV transaction/MVCC APIs, SQL over multiple cells, geographic replication, online +conversion of old one-cell storage and a claim of TiKV-equivalent performance are +outside this package. No new consensus algorithm or raft-rs fork is needed. + +## Identities and versions + +| Type | Fields and lifetime | +| --- | --- | +| `CellHistory` | Cluster, namespace, validated cell ID, cell incarnation; stable through elections, replica moves, split and merge | +| `GroupHistory` | Cluster, namespace, opaque group ID, group incarnation; one physical log history | +| `GroupRoute` | Group history, half-open cell-key range, routing epoch, birth/schema binding | +| `GroupMembership` | Group history, committed membership revision, member/node/disk identities; independent of routing epoch | +| `CellVersion` | Cell history, nonzero monotone mutation sequence, LTX epoch/txid/checksum and image commitment | +| `CommitWitness` | Group history, term/index and proposal identity; locates the decision without defining the cell's lifetime | +| `TransitionId` | Non-reusable operation ID plus pinned source/destination histories and expected catalog root | + +Increment the cell mutation sequence for every committed cell image update, +including internal receipt/effect maintenance. Group no-op or membership entries +do not increment it. Overflow refuses before proposal. Enrollment creates sequence +1. Preserve the sequence and `CellHistory` through normal group transfer. +Disaster recovery creates new group and cell incarnations, retaining receipt +contents/client floors but rejecting tokens from the prior cell history. + +Version tokens authenticate cell history/version, schema, expiry/key identity and +the commit witness. They grant no access. AtLeast after normal transfer compares +the same cell history's mutation sequence, using installed transfer provenance; +it does not compare log indexes from unrelated groups. A token for another cell, +incarnation or schema refuses. Preserve original command-result bytes and receipt +identity through transfer. A recovery-seeded duplicate returns the retained result +without callback execution and a version covering that effect in the new history; +the old witness remains provenance, never new-incarnation authority. + +Image roots bind logical cell history, schema and LTX position. Group manifests +bind those roots to physical group history and a committed prefix. Extent payloads +remain content addressed. A normal transfer can reuse verified immutable cell +roots locally; it cannot borrow another group's authority or an unsealed local +reference. Disaster recovery rebinds the cell root's history, retaining payload +objects only after full verification. Materialized images follow the same rules. + +## Partitioning and directory + +Use a stable ordered cell routing key: + +```text +digest = SHA256("cellar-cell-route-1\0" || namespace_id[16] + || cell_id_length_u32_be || cell_id_utf8) +CellKey = (digest[32], cell_id_utf8) +``` + +Compare the tuple lexicographically by bytes. The full cell ID breaks digest ties; +a hash collision never aliases cells. Validate IDs before hashing, without locale +normalization. Ranges are `[lower, upper)` with explicit negative/positive infinity +sentinels. Split boundaries always lie between whole cell keys. A namespace's +active range map covers the complete domain without overlap or gaps. + +This hash ordering distributes application IDs and differs intentionally from +TiKV's user-key byte ordering. Cellar exposes no cross-cell ordered scan contract. +A dedicated hot-cell interval can use actual neighbor boundaries; it need not +split a cell or special-case the consensus engine. + +The namespace root conditionally publishes a digest of an immutable paged range +tree and its mutation journal. Pages have fixed byte/count limits; readers descend +only the relevant path and cache hints. No per-request full namespace scan, no +monolithic million-cell map and no environment-selected alternate routing hash. +Updates copy bounded tree paths and CAS one root, serializing namespace topology +changes only. Independent namespaces can progress independently. The final load +profile must measure this catalog write bottleneck. + +Managed cold setup may first publish an explicit standby genesis with a namespace +pin, immutable journal and operation index, but no selected range tree. This +durable zero-group state lets complete inventory precede boot capacity admission +and the first managed birth. Missing metadata is never a standby observation. +Standby serves no cells. The first complete range map requires permanent birth +reservations and a CAS against the exact standby root. After selection, topology +cannot return to standby; retries reconcile their original operation journal. +Birth admission checks every initial voter against its current boot-bound +capacity receipt. An Active descriptor alone is insufficient after an ordinary +restart. Retained group ownership survives that restart; new placement requires +the new boot to qualify and activate capacity again. + +Initially provision a bounded explicit set of non-overlapping ranges; an empty +range record may remain unmaterialized until first enrollment. Lazy group birth +uses the existing conditional creation and permanent member claims. A node move +changes peer placement, not cell partitioning. An enrollment request goes to the +current group leader, which commits the cell's initial image. Concurrent creators +converge on that log decision and receipt. The previous per-cell reservation idea +is replaced by unique range ownership plus replicated enrollment; do not build two +assignment authorities. While a range is sealed, enrollment is refused/retried. + +Cached routes carry group history and routing epoch; every destination verifies +the complete binding. Relocation hints are authenticated by exact confirmation +against the configured trusted range directory, bounded by the original invocation deadline and a finite +total attempt count. Known cached routes need no per-request catalog read. This +mapping confirmation does not replace transport peer authentication or destination +admission checks; a peer cannot introduce an arbitrary redirect URL or catalog. +An ambiguous submitted command is retried only with its original cell-scoped +command identity. An unavailable catalog may prevent births +and topology changes; it does not turn a known group's write path into S3 quorum. + +## Runtime and persistence + +Split the current host's combined `Cell` owner into a group owner and bounded cell +execution owners. Only the group owns RawNode, hard state, membership, read barriers, +apply prefix, snapshot/archive jobs and topology transitions. A cell owns receipt +admission, private execution, pending proposal, image/query pins and effect state. +The node owns log engine, extent store, transport, SQL worker pool and global quotas. + +One unresolved data proposal per cell is sufficient initially. Prepare different +cells concurrently through fair queues; serialize publication by group log index. +No slow callback or image preparation runs on the consensus tick loop. Group/node +byte quotas include queued, prepared, in-flight and ambiguous work; charge physical +owners until they release it, even after the request waiter is cancelled. + +Data entries carry cell history, route binding, schema, mutation sequence, +base/result image, receipt/proposal identity and bounded LTX payload. Administrative +entries are explicitly typed: enroll, membership, checkpoint, seal, install, +activate, park. Unknown tags, unsupported domains or inconsistent lengths refuse +before allocation. Batch independent proposals where useful, retaining each +cell's identity and base chain; batching is never a cross-cell transaction. + +The durable group root contains group history, applied position, routing/transition +state, membership binding and an authenticated map of all cell image leaves. +Each leaf contains cell history/version, immutable original enrollment identity, +last group installation/mutation position, complete original `CommitWitness` +and effect summary. Keep original decision provenance separate from local +installation position through transfers. Bind each new witness to the actual +applied enrollment/mutation entry, retain it on unchanged-cell progress, and +recover it from the authenticated manifest without requiring compacted log +entries. Query and receipt observations carry this selected-version witness; +never manufacture a proposal ID from group progress. Enrollment identity binds the +creation operation, original cell history, schema and initial image commitment; +it is retained even after that original image is reclaimed. Use a content-addressed paged tree with bounded copy-on-write paths, not +an O(cell-count) full manifest rewrite for every update. An engine transaction +publishes the durable root and contiguous apply position after referenced image +and manifest pages are synced. Readers pin a published root and exact cell image. + +Configuration/no-op entries advance the group prefix without changing cell versions. +A missing earlier cell update prevents later group publication. Parallel staging +may finish out of order but cannot advance a hole in the durable applied prefix. +Retain exact term/index proof for each pending output; an unrelated cell's progress +cannot settle it. Storage sync failure quarantines the owning engine as today. + +An unresolved SQL producer can reconcile the original request against the exact +cell's committed receipt after leadership changes or log compaction. Select the +cell pin from a verified durable current-term applied root; follower inspection +is permitted but grants no write or strong-read authority. A matching receipt +can resolve the request from a later image. Its returned position describes the +inspected image, not the original proposal. Absence retires the old producer only +after a strictly newer term's applied barrier excludes that proposal; a persisted +term, timeout or sampled leader role is insufficient. Receipt expiry remains an +explicit refusal with unknown historical outcome. Reject changed route/history, +foreign engine and stale pin proofs. Close the private writer before releasing +cell admission; retained responses keep their byte reservations. Lost resolution +responses require receipt lookup, never an inferred failure or blind callback replay. +Receipt inspection preserves identity/floor/sequence and allocation checks but does +not admit a new client or receipt. Full admission tables cannot obscure absence; +subsequent new commands still obey their client and receipt ceilings. + +## Reads, caches and effects + +Linearizable reads obtain a fresh current-term group ReadIndex, wait for contiguous +apply through it, then select the target cell from a coherent root. An unchanged +cell's last mutation can precede the barrier. AtLeast uses cell version lineage; +Stale explicitly permits an older coherent image. No clock lease optimization is +introduced. Transfer-sealed groups refuse new strong reads and mutations for their +range; requests already in progress retain their original deadlines/proof rules. + +Idle cell eviction closes disposable execution/SQLite caches only when no proposal, +effect callback or unsafe pin is active. Reopening reconstructs the private base +from the committed cell root. Group membership, data and receipt floors remain. +File-descriptor and cache byte limits must apply to thousands of enrolled cells, +not just an active-cell metric. Cache misses, image downloads and application work +are separately measured from consensus wake latency. + +Alarms and outbox/inbox IDs bind `CellHistory`, not current group or node. Transfers +preserve delivery identity and receiver dedupe; they must drain/journal already +dispatched work and never mint new IDs for an unknown outcome. Durable summaries +let node schedulers find the next due effect without opening every SQLite DB. +Control ownership follows the current group leader. A pending old-origin delivery +after disaster recovery still needs reconciliation; the existing refusal remains. + +## Resource policy and operational API + +Expose distinct limits for cells/group, group logical bytes, resident cell bytes, +queued LTX bytes, unarchived bytes, snapshot/transition slots, open readers and +group replicas/node. Existing validated envelope/image limits remain in force. +Initial candidate occupancies are 8, 32, 64 and 128; no numeric default becomes a +capacity claim before qualification. Oversized single cells use dedicated groups +and explicit per-cell quotas. Never raise a hard cap to hide a failed test. + +Scheduler inputs include cell count, bytes, write/apply rate, queue age, archive +lag, leader CPU and failure-domain placement. Split/merge use hysteresis, cooldown +and bounded concurrent work. Only compatible adjacent ranges can merge. Replica +movement uses learners, verified catch-up and committed configuration changes; +membership changes cannot race a seal, park or disaster operation. + +The SDK continues to address cells. Add operator group list/inspect, split/merge, +rebalance, park/wake and transition status/resume operations. Status is read-only; +waiting does not advance a mutation. CLI/admin/native and Axum enforce the same +identity and authorization contract. Metrics use bounded labels and distinguish +cells, group replicas, leaders, parked groups, unresolved operations and archive +lag. Shutdown seals admission, drains/records ambiguity, joins all owned work and +preserves transition pins/debt for restart. + +## Cutover and delivery boundaries + +Use the existing raft-rs and Raft Engine integration; no TiKV source copying or +dependency replacement. Add private contracts during staged development, switch +all production consumers together, then delete superseded one-cell assumptions. +Do not ship a toggle selecting legacy per-cell consensus. Occupancy one exercises +the same group path as larger occupancies. + +All Cellar-owned markers stay at 1, with new semantic domain/protocol digests and +fresh namespace/storage roots. Old wire, image and archive layouts refuse. There +is no automatic data migration or rollback by pointing an old binary at new data. +Retain existing disks/archives; any developer reset is an explicit separate action. +The final release sweep covers catalog, tokens, public SDK, CLI/examples, effects, +managed fleet and destructive recovery, not just the new host fixture. diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md new file mode 100644 index 000000000..037622c3b --- /dev/null +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -0,0 +1,2581 @@ + + +# Cell-group implementation handoff + +Purpose: Current implementation checkpoint, evidence transfer and next executor steps. + +## CG-07 cold router return after birth-voter replacement + +The router returns after three rooted moves and joined retirement of the birth +processes. Preserve the prompt refusal of quorum reads on durably retired +voters and the physical settlement proof. Complete the original public +lifecycle and authority gates and qualify constrained voters. +[Source and test records](progress.md) + +## CG-07 catalog-driven startup and public request qualification + +The combined fresh RustFS prefix case now uses public provisioning and archive +commands, SDK voter readiness and enrollment, and native and Axum ingress. +Qualify retained cold router return after all birth voters retire and finish +the remaining managed lifecycle, public gates and constrained voter runs. +[Source and test records](progress.md) + +## CG-07 daemon-owned first-group readiness + +The daemon callback verifies the rooted archive, attaches the authorized local +voter and establishes fresh quorum before binding ingress. Replace the fixture +archive-ready rendezvous with bounded observation of the exact rooted intent and +registry. Preserve capacity attestation and boot trust checks. +[Source and test records](progress.md) + +## CG-07 rooted first-group archive initialization + +Archive setup runs through the CLI and admin API. Preserve the exact rooted +intent and verify it before startup. Move physical activation into the daemon +readiness path and establish fresh quorum before ingress. +[Source and test records](progress.md) + +## CG-07 public first-group operator workflow + +The CLI and admin path observe standby and reserve and publish the first +group using the exact rooted plan. Both RustFS process modes use the CLI +for capacity and metadata. Keep one-use archive initialization and daemon +activation under explicit ownership while moving them out of the fixture. +[Source and test records](progress.md) + +## CG-07 independent managed daemon voters on RustFS + +The real-backend process case now passes with both materialized images and +extents. It retains exact request and recovery ledgers. The fixture still +supplies capacity and the operator steps. Move these steps into public +management while preserving the exact standby, reservation, confirmation +and archive intent. Qualify constrained voter processes and the wider +lifecycle gates separately. +[Source and test records](progress.md) + +## CG-07 managed daemon startup and request path + +The normal native daemon starts three independent managed voters from standby +through a fixture operator. Their shared group serves two cells after fresh +quorum and survives a leader hard kill and retained restart. The daemon installs +shutdown ownership before Raft startup and joins the owner if asynchronous +readiness is cancelled. Next run the process gate against real RustFS and move +the fixture operator's capacity and first-group orchestration into the public +operator path. Preserve the original failures and request bounds. +[Source and test records](progress.md) + +## CG-07 managed group archive provisioning + +The SDK prepares the one-use archive initializer for a selected managed history +and verifies a saved intent without writes after setup. The managed native and +RustFS request fixture uses that path. Port the original daemon process gate +from per-cell registration to shared groups. Have the operator persist exact +setup, let each daemon attach its authorized local voter, and gate ingress on +a fresh quorum observation. Preserve every original refusal and restart oracle. +[Source and test records](progress.md) + +## CG-07 managed first-group provisioning + +The SDK reserves, publishes and confirms the first group from a declared +standby. The test keeps the Live birth decision across cancellation, verifies +conflicting and lost confirmation outcomes and reopens the exact reservation. +The managed request path passes on native storage and real RustFS. +[Source and test records](progress.md) + +Next, integrate the public first-group operations into ordinary CLI and daemon +setup. Keep the archive creation intent distinct from retained open and require +actual voter activation and quorum before public serving. + +## CG-07 managed namespace standby provisioning + +The public SDK checks the compiled schema, publishes the standby and confirms +its fleet declaration. Exact retries recover lost replies. A new fleet request +at the new revision can reuse the same standby operation after a root race. +The managed native/Axum path starts from that standby and passes on native +storage and real RustFS. Existing registry replay and reset oracles also pass +on the shared fixture. [Source and test records](progress.md) + +Next, wire CLI and daemon setup to the first-group reservation, exact range +publication, archive initialization and voter activation. Preserve the existing +capacity proofs and immutable operation identities through each boundary. + +## CG-07 managed router membership refresh + +The managed directory learns actual replacement membership for configured +routers. The native and real RustFS cases verify rejection of the retired +route, the exact old receipt, a fresh write through the replacement leader +and read-back from the joined voter. They also rerun the native/Axum request +path. [Source and test records](progress.md) + +Next, wire ordinary namespace provisioning and exercise the public startup +and request path from an empty managed deployment. Cold router return after +all birth voters retire and the complete public lifecycle gate remain open. + +## CG-07 routed SDK enrollment + +The existing ingress path now creates cells through the SDK. It preserves +creation identity and achieved durability, including exact creation replay at +S3 durability. Eight native cases and the real RustFS managed case pass. +[Source and test records](progress.md) + +Ordinary namespace provisioning, the complete public authority and lifecycle +gates, reviewed source preflight and the remaining cell-group tasks stay open. + +## CG-07 nonmember ingress and idle connection retirement + +The configured ingress path passes the seven scoped native cases and the +managed request case against real RustFS. It serves both read and write +requests through an ingress with zero local groups. The archive denial case +separates quorum acknowledgement from explicit S3 coverage. The sender now +retires a connection after the receiver's idle timeout before sending the +next queued request. The focused expiry regression and original counter +process case pass with their existing operation bounds. + +Next, qualify SDK enrollment through this path and connect ordinary namespace +provisioning. Refresh router membership across replacement, complete the +public lifecycle and authority cases, and keep the earlier RustFS successor +readiness failure open. [Source and test records](progress.md) + +## CG-07 movement discovery and basic request priorities + +The lost-observation regression proves durable movement Begin within the +original deadline. It preserves the debit and leaves the learner inactive. +The full managed native target passes. Real RustFS confirms the regression +and archived maintenance, and reproduces the successor readiness timeout. +The [progress record](progress.md) retains all three backend outcomes. + +The [basic request acceptance sequence](remaining-work.md) starts with +trusted route discovery from ingress hosting no replica, then exercises normal +namespace provisioning, failover, retained restart and quorum-loss refusal. +Keep the original successor deadline and staged-target close oracle while +isolating the RustFS failure. + +## Managed request routing checkpoint + +A public SDK command sent to a follower previously attempted local SQL and +returned NotLeader despite an observed healthy leader. The pre-fix managed HTTP +case reproduced this twice, with failed binaries/logs and voter directories +retained. Queued originating SDK commands now retain one controller invocation, +identity, captured inputs, admission and original deadline through one forwarding +hop. Incoming RPCs and the lower-level controller command API stay local-only. +The destination independently checks its route, namespace process permission and +current execution authority. Possible transmission keeps an error ambiguous; +there is no automatic uncertain mutation retry or new identity. + +Trusted embedders can grant/revoke command origins after managed discovery has +verified the actual peer boot/endpoints. Host namespace capabilities and current +transport identity/revision checks are mandatory. Membership grants no permission; +revocation retains a revision tombstone without undoing admitted work. Permissions +belong to the running transport and require trusted policy after restart. + +Command output now distinguishes local inspected SQL receipts from authenticated +remote results. Common output metadata is available for both; local_receipt is +present only for the local variant. Wire encoding, effects and physical oracles +retain local proof requirements. Existing tests make those requirements explicit. +Temporary diagnostic logs are removed and no assertion or lint is suppressed. + +The new managed native/Axum case passes in both image forms: cold capacity and +group birth, replicated enrollment of two independent cells, follower writes, +remote minimum-version/linearizable reads with actual responder identity, +conflicting retries, cross-cell tokens, absent/revoked/stale-process/foreign- +namespace permission refusals, forwarded S3 acknowledgement, joined leader-stop +failover, retained restart with original receipts/floors and new writes, then +quorum-loss write/read refusals. The counter declares explicit namespace origins; +its public process gate now requires original receipt success through every one +of six native/Axum addresses before and after retained restart. + +The complete eleven-target final-01 native campaign is **FAILED: 308 passed, +2 failed, 39 explicit skips**. The new managed HTTP case and public SDK/process +gate pass; the public gate takes 140.55s. The managed +target fails eight-turn maintenance replacement settlement and fresh successor +readiness after closing an S3 source service, including an Unobserved close +report. Original bounds/assertions and failure disks/logs remain unwaived. These +are previously recorded failure classes; the forwarding fix does not establish +their causes. No narrower component pass makes the full campaign green. + +RustFS comparison: managed request run1: PASS (53.37s); managed request run2: PASS (50.61s); archived maintenance: PASS (88.53s); automatic S3 recovery: PASS (45.41s). +Matrix status is **PASSED**. Both request invocations cover both image +forms and pass; every comparison case retains its own result. Signed readiness +and conditional probes pass. The immutable-ID/owner/resource/mount-verified, +pinned one-CPU/two-GiB backend is reused with fresh buckets/prefixes; backend and +dedicated VM are stopped with data retained. Voters remain host-native embedded +Nodes with fixture capacity/isolation. Joined stop is not a process-crash drill; +independent constrained voters, permanent fencing, partitions and load remain +unqualified. + +Source-final-01 binds 2,726 Rust/Python/manifest inputs above +3ccc3e51f03458a2a5715401b8e5f6b963146276 to SHA256 +da3514ed9ea2dd614a45f3150f92d2280f75795287b7469a5883f72564b1a87c. +FAILED native record SHA256 is +6a4f24b43031c9b9cab3a996edd3b3bf280ecf494a2a231d997912b0e75098e7. +Its individual managed request proof is extracted explicitly from that failed +campaign (SHA256 8f6e418e605c0d56a635b245ded5cc423d7657cf8f548bdc40109cdb3bdac1a8). +RustFS record SHA256 is +646ebe709b457b540a97a12e85e43aade6aee1fbbb445a5d37d84a2650330695. +Default stack, opt-level 0, assertions/overflow checks, quotas and original +operation/movement bounds remain intact. Code inputs are checked unchanged. + +Format/layout/dependencies/docs, native-only Cellar/Python-library and diff checks +pass. Strict library Clippy and plan tests/validate/preflight remain red, with no +Clippy diagnostic-key additions or silent source-anchor refresh. Earlier unseeded +maintenance/ambiguous-write/transfer/TCP/enrollment/inspection failures and older +disk-erasure qualification keep their own source and scope. CG-07 remains +IN_PROGRESS; four public gates, default/capability integration and CG-09–16 remain +open. Automatic command selection still needs local group status: route-only +ingress requires bounded leader discovery. Queries still default stale. + +## Member observation renewal checkpoint + +A deterministic managed regression now reproduces one lost local-member +observation after leadership handoff. Its foreground catalog client is independent +of the controller and background trust refresh, over the same actual backend. +Before the fix, that read consumes the entire original five-second renewal budget; +local snapshots show a healthy quorum and empty work banks. The peer observation +now receives a share of the remaining time, reserving time for other members and +final validation. A transient refusal leaves the history pending; retry obtains +all three fresh safe ReadIndex/durable-apply observations. Both image forms pass +without resuming the lost read. Root, boot, configuration and proof-age checks +remain intact. No write retry, stale proof or longer deadline is introduced. + +The lifecycle harness also reconciles actual leadership before/after maintenance +using fresh term/revision guards, rather than treating a historical survivor and +one admitted handoff as completed election. It observes actual target leadership +inside one original five-second envelope. The eight movement turns and all +durability/capacity assertions remain unchanged. Bounded local status/admission +snapshots are recorded on failure before cleanup; the original error still fails. +Temporary tagged tracing and its focused prototype are removed. + +Source-final-02 binds 2,722 Rust/Python/manifest inputs above +6b70a5357a470f75b7e36ec201f44f31ea7da536 to SHA256 +7ad7649846ca9405e6be33738b6d851f256a887e4af08c4f86004c590039554b. +The complete affected managed target passes **17 cases with 6 explicit backend +skips**, in 307.21s. Native record SHA256 is +e589e438c251e71d8ddba76abf8eb9055b63b7a08abec06526a402b431424441. + +Four same-source RustFS invocations pass: the lost-observation regression +32.19s, the unchanged full archived maintenance case +twice (85.71s and 100.80s), +and automatic S3 voter recovery through terminal removal +40.46s. Both image forms run; maintenance additionally +covers initial and joined voters. Record SHA256 is +d3c69d70f2e2fe99a840582cdf7248f5c833abbeb939a54b125ea95fa18e4040. +The immutable-ID/owner/resource/mount-verified one-CPU/two-GiB backend is reused +with fresh buckets/prefixes because retained Docker networks exhausted default +subnets. No retained resource is deleted; backend and dedicated VM are stopped +with data retained. Nodes remain host-native with fixture capacity/isolation; +independent constrained Linux voters and production fencing/load remain unqualified. + +The earlier unseeded renewal timeout was not captured at a causal low-level seam. +An instrumented full case also failed an ambiguous sequence-4 S3 command before +owned resume; another diagnostic run missed restored-target leadership. Those +failures and the initial final-source 15-pass/2-fail campaign (NotReady maintenance +and eight-turn peer replacement) are retained and unwaived. The new regression +proves recovery from a lost observation, not every prior failure's root cause. + +Format/layout/dependencies/docs, native-only Cellar/Python-library and diff checks +pass. Strict library Clippy and plan tests/validate/preflight remain red; diagnostic +comparison adds no key. Reviewed source anchors are not refreshed and no lint is +suppressed. The earlier seven-target/SDK and disk-erasure results remain bound to +their recorded source; this checkpoint qualifies the affected managed boundary. +CG-07 stays IN_PROGRESS; four public gates, default/capability cutover, CI/load, +unexplained failures and CG-09–16 remain open. + +Next: deliver the public managed lifecycle and ingress-authority gates; investigate +the unseeded command/transfer/replacement failures using the retained snapshots +and original bounds. Finish actual Docker isolation/resource-provider qualification, +then default/capability wiring and the remaining CG-09–16 objective. + +## Automatic recovery checkpoint + +The Node-owned automatic failure service now passes the complete local +shared-group path in both image forms: default failure detection, rooted +exclusion, exact-boot fixture isolation, S3 replacement, fresh stable RF3, +independent archive verification, finalization and journal-confirmed removal. +The fixture first holds isolation and proves repair has not started. Shutdown +joins that callback and releases control admission. A duplicate service refuses +without another charge. Two service restarts preserve the rooted operation; +archive payload refusal blocks terminal progression, and restored reads require +new physical S3 GETs before removal. The old engine remains byte-identical and +retained, active movements/reservations clear, and both cells retain original +receipts and accept new explicit-S3 writes. No business mutation drives repair. + +This new path first aborted on the unchanged default stack. Sanitized poll-frame +evidence and repeated red binaries are retained. Control turns, recovery/removal +phases and live verification now retain boxed futures under the existing worker +owner, admission and original deadline. A subsequent receiver refusal hid the +actual missing-spool I/O error. Archive download now joins and returns the typed +consumer failure; the malformed-checkpoint assertion is tightened accordingly. +A real-RustFS repeat exposed about 1.96 MiB of application poll frames before +the HTTP client. Catalog turns now own boxed work before their future is +constructed, shrinking nested control frames without moving lock or deadline +checks. Managed replay creates and syncs its private scratch directory before receiving +materialized image bytes. No stack, quota, grace or operation deadline is raised. + +Source-automatic-qualified binds 2,721 Rust/Python/manifest inputs +above 29f277ddf228f33068c229583efb487e01749f07 to SHA256 +e8c8f4379f8587ee70c5b011dcaf11442104fde90915da5359199f95ebfaaa3b. Seven native targets pass **267 cases with 36 explicit skips**; +the public SDK/native/Axum process/restart gate passes in 64.65s. +Native/public record SHA256 is +b5d4f68e12ad56e0676e6f29974ce9035584579e46edf0b8c470816359ef151c. + +Three exact lifecycle cases pass against pinned real RustFS in both image +forms: automatic recovery, isolated replacement and normal replacement/joined +restart. Their durations are 43.82s, 26.58s, 37.99s. +The same campaign then FAILS archived joined-voter maintenance after accepting +leadership handoff: the original five-second survivor-proof renewal expires. +The 38.54-second failed fixture and all disks are retained; no deadline is +raised and no later pass waives it. The disk-loss case was not reached there. +Campaign record SHA256 is +13b1f1f4a6b88cfd6362e5ad07fe79a6b5bd15e4555b0a580fecf7ce61dc0243. + +A separate same-source exact public fresh-incarnation recovery case passes in +both image forms after every old voter directory and verifier cache is erased, +in 20.67s. Its record SHA256 is +f739ea9c2027c50ecfa1c1c02176a59a8a9415f2447ade8128000c24ae836298. +Signed readiness/conditional probes pass for both owned backends; each one-CPU/ +two-GiB backend and VM is stopped with data retained. These are host-native +embedded Nodes and fixture isolation, not independent constrained voter containers, +permanent Docker launch revocation, sustained load or the full public lifecycle. +The backend matrix is not green. + +Format/layout/dependencies/docs, native-only Cellar/Python-library checks and +diff checks pass. Strict library Clippy and plan tests/validate/preflight remain +red; the saved diagnostic comparison records 0 added diagnostic +keys. No lint suppression or source-anchor refresh is applied. Earlier TCP, +bounded readiness/maintenance, 128-cell enrollment and inspection failures remain +unexplained and unwaived despite later passes. CG-07 remains IN_PROGRESS; +CG-09–16, the public/default/capability cutover and the full goal remain unfinished. + +Next: diagnose the current RustFS joined-maintenance proof timeout under the +original five-second bound, then deliver the public managed lifecycle/ingress +gates and orchestrate retained debt on surviving original voters, including +joined cancellation and uncertain outcomes. +Qualify actual Docker isolation/resource providers over independent constrained +voters. Then finish namespace/SDK/CLI/daemon/admin/example defaults, capability +coverage and the other public gates, followed by CG-09–16. Preserve live original- +voter physical closure and quorum admission; no weaker fallback is allowed. + +## Quarantined voter checkpoint + +Shared managed startup now verifies current host authority and prepared namespace +declarations against one root before opening mutable engine state. It still +requires the later engine-bound boot CAS. A saved pre-isolation capability cannot +modify an already quarantined log. Shared physical route refresh discards a +retained initial endpoint only with rooted permanent isolation, so an unusable +old endpoint cannot prevent publication of an admitted learner's route. Durable +votes, seed owners and resource debt are unchanged until their existing proofs. + +The new managed isolation case joins the complete obsolete Node, rejects premature +and wrong-process evidence, and verifies typed startup refusal without fleet/log +mutation. S3 replacement must restore both cells, obtain fresh stable RF3, preserve +receipts, reconcile a lost settlement confirmation and attribute the isolated +obsolete source. The retained log stays byte-identical and the replacement keeps +its physical capacity floor. Two log-mutation reproducers and the eight-turn +blocked-route failure are preserved. Diagnostic probes are removed from source. + +Source-isolated-final SHA256 is +cc2e83093bfe91439dce852e25e3340108b885f3b79b3a05e1453a9596f3feb1 +above 9070eb15, for 2,719 Rust/Python/manifest inputs. Seven native targets pass +266 cases/35 explicit skips; public SDK/native/Axum process/restart also passes. +Native/public record SHA256 is +60c4918d707c26a95efefa112dfa8400b194c67faa8643b71edd20586b812732. +Four same-source real RustFS cases pass: isolated replacement, normal replacement +and joined restart, archived maintenance and all-old-voter-disks/verifier-cache +erased public recovery. Record SHA256 is +3ea848910a5843bded56eb0140933cb2bd4120545bc32be37d6c9bee838e0d16. +Backend/VM are stopped and data retained. Both image forms run with original +quotas/bounds. Host-native embedded Nodes and fixture fencing do not qualify +permanent Docker launch revocation, independent constrained voters or load. + +Format/layout/dependencies/docs and native-only library checks pass. Strict +library Clippy still reports 220 errors/202 dead-code reports. The only changed +message removes the now-used `is_isolated` from a grouped unused-method warning; +expanded item/count comparison adds no diagnostic. Plan tests retain 35 pass, +one index/Purpose failure and one reviewed-protocol drift error. No suppression +or automatic source-anchor refresh is applied. + +Next: complete automatic failed-voter and retained surviving-voter-debt +orchestration, terminal survivor/archive verification and the full public managed +lifecycle gate. Preserve the live physical-closure requirement for surviving +original voters. Finish four missing public gates, capability/default cutover, +performance/inspection/lint/plan/CI modes and CG-09–16. Earlier unexplained +failures remain open despite current passes. The full goal is unfinished and +PR #160 remains draft. + +## Idle/restart seed retirement checkpoint + +A bounded node-owned sweep now retires authenticated retained seed journals during +idle operation and restart. It validates history/purpose/engine/archive identity, +shares live sources' group locks, reconciles the exact intent and confirms abort +before syncing empty state. Unknown/missing/corrupt state never initializes a +replacement or permits retirement of an unknown owner. Cleanup can reopen an old +immutable catalog binding without serving or membership authority. Physical joins +precede lock/admission release, including errors/panics. Joined close retains an +entered remote read even after its waiter is cancelled. One 272-KiB workspace is +charged to existing provisioning capacity; ordinary S3 admission is per journal. +The source and cleanup modules live under `controller/archive/seed/`; public Rust +paths and configured ceilings/deadlines/profiles remain unchanged. + +CG-07-seed-cleanup-01/source-final SHA256 is +9d8d111ff2121fe9802d186b04710039e480c478366b0de9f0d372719e995603 +for 2,718 inputs above acd56a9f. The complete native campaign passes 265 cases/34 +explicit skips. A test-only inventory-before-close adjustment reran managed; +production and seven other binaries are unchanged. Two journal cases and focused +cleanup pass. Current public SDK/native/Axum process/restart passes in 132.07s. +Four real RustFS cases pass in both image forms, including idle/solo-restart debt +retirement, cancelled close and all-voter-disks/verifier-cache-erased recovery. +Backend/VM are stopped; data and exact sources/binaries/failures are retained. +See [progress](progress.md) for provenance and timings. + +Strict library Clippy remains 220 diagnostics/206 normalized keys without additions. +Plan index/Purpose and reviewed-protocol drift remain red. Preserve earlier TCP +overload, five-second readiness, eight-turn maintenance, enrollment and inspection +failures; passes supply no causal fix or SLA. Next: fenced failed-voter/retained +survivor-debt orchestration, four remaining public gates, capability/default +cutover, collection/performance/inspection/lint/plan/release modes and CG-09–16. +Lost ownership still needs proper recovery authority. Independent constrained +voters, permanent Docker fencing and load remain unqualified. The full hard +cutover is incomplete and PR #160 stays draft. + +## Managed S3 learner checkpoint at acd56a9f + +Managed `from_s3` activation now captures a fresh-quorum complete checkpoint, +journals a separate upload generation and authenticates its ticket/reference in +the seed reply. The pending generation and inherited graph protect every object. +The destination independently verifies the complete closure through its serving +receiver before READY1; live source checks precede End and the existing engine +boot/catalog/install fence remains authoritative. Transfer data supplies neither +a disaster-recovery head nor an S3 command acknowledgement. Marker-1 seed requests +require `from_s3` and replies require a tagged data mode; old shapes refuse. + +One owned source job retains the original transport/read/metadata/checkpoint/S3 +resources through upload, handoff and cleanup. Joined close stops new admission +and drives transport and source cleanup together. A failed retirement retains its +durable intent and an observable error; a subsequent seed reconciles that intent. +Automatic background/restart cleanup remains required. + +A reproduced source-failover regression found that managed coalescing included +source member/node hints in destination identity. The comparator now retains the +exact destination boot/member/permit and mode, while the next completed attempt +may reauthorize a new source. An entered attempt retains its original hint and +deadline. A mode change cannot adopt the retained preparation. + +The new fault case verifies actual +target-only S3 payload refusal, a retained nonserving preparation and resource +debit, no peer fallback, source-service stop and fresh election. Full membership +repair must obtain a fresh quorum. Catalog settlement separately requires live +physical inventory from surviving original voters, so their outage retains the +debit; it cannot be converted into an offline proof by a timeout. The normal full +settlement/replay gates remain required. Explicit RustFS variants cover S3 +replacement/restart, archived maintenance and this failover/debt case. + +Candidate source SHA256 is +65cbd074ab06749b4849ef06d38c23e0df396385c63e3b74bc6fb48159cdf5e7 +for 2,717 inputs above 41a67844. Native results are 264 pass/one fail/34 explicit +skips. Execution's checkpoint TCP retry reports `Overloaded`; its unchanged +focused repeat passes, without clearing the failed full run. An earlier extent +successor-readiness timeout remains unresolved despite three isolated and later +complete managed passes. The original archived-maintenance eight-turn failure +also remains preserved. Bounded failure-only observations cannot extend a gate. + +Four same-source real RustFS cases pass: S3 replacement/restart 34.15s, archived +maintenance 85.93s, denied-payload/failover/debt 33.04s, and public all-voter-disks +plus verifier-cache-erased recovery 17.56s. Both image forms run; the backend and +dedicated VM are stopped with data retained. An initial signed S3 probe returned +503 before tests; keep that failed setup too. The passing setup checks signed-read +readiness before conditional writes. Embedded Nodes/contract fencing supply no +independent-voter, permanent Docker fencing or load qualification. Strict library +Clippy is 220 errors, down one with identical production inputs and no added keys; +plan/index/protocol drift remains red. See [progress](progress.md) for boundaries. + +Next: diagnose the checkpoint retry and bounded readiness/maintenance failures. +Preserve the earlier failed runs and all original five-second calls/eight-turn +reconciliation bounds. Finish abandoned hold cleanup, +fenced failed-voter/public lifecycle ownership, default/capability cutover, +manifest-page collection and the existing performance/inspection/lint/plan gates. +Permanent Docker launch fencing, independent constrained voters, sustained load +and CG-09–16 remain required. The complete goal is unfinished. + +## Dynamic archive ownership checkpoint at 41a67844 + +Canonical startup now retains node-wide pools and independent verifier resources +for groups added after boot. Preparation opens existing archive identity/counters +before bootstrap, charges both store bounds to the existing provisioning bank and +retains the private owner through uncertain attachment/learner installation. +Successful physical attachment hands it directly to the actor's bounded archive +lane. Joined node close releases preparations, archive jobs and then immutable +configuration; cancelled waiters cannot release an entered archive owner. + +CG-07-dynamic-archives-01/source-final SHA256 is +9cb0776c7a699f6e7af5f1aa6ed03d3bd09f277967a6bcbb5d70051a12d39f7d, +above 8c0a42f02eb78060adaa5c87472ab34f95b6e5c7. Twelve managed cases pass, +including dynamic RF3 S3 writes, missing identity refusal followed by explicit +synced initialization/retry, and joined close while an archive GET is held. Both +image forms run through native Nodes/SQLite/TLS with an in-memory fault adapter. +Seven native targets pass 263 cases with 31 explicit skips and unchanged source. +The same binaries pass public fresh-incarnation recovery against real pinned, +constrained RustFS in 30.33 seconds for both image forms after erasing all source +voter directories and verifier cache. Backend and dedicated VM are stopped. +This is embedded-Node/contract-fence coverage, not managed archive qualification +against RustFS, permanent Docker fencing, independent voter containers or load. +Public SDK process, full performance and full library gates were not rerun here; +the preceding performance/inspection/release gaps remain open. + +Format/layout/dependency/docs and native-only library checks pass. Strict Cellar +library Clippy remains red with 221 errors, one fewer than before and no new keys; +202 are dead-code errors. Plan tests remain 35 pass/one fail/one error, with +generated index/Purpose and reviewed-protocol drift refusals preserved. + +Next: qualify archived managed replacement and maintenance, then implement the +currently refused exact-reference S3 learner path. Preserve complete manifest, +image and receipt closure, uncertain outcomes, actual destination boot/catalog +authorization, physical installation and fresh-quorum catch-up. Do not treat a +seed or remote object as activation authority. Finish archive hold retirement, +public/default-backend gates, live manifest-page collection, permanent Docker +fencing/load qualification and CG-09–16. The full goal remains incomplete. + +At that checkpoint, the peer seed exchange lived in `group_snapshot_transport/seed/`: `client.rs` +authenticates the source checkpoint and stages the exact managed grant; +`install.rs` receives peer chunks and reauthorizes before durable installation. +The authenticated reply carried no archive reference. The S3 path needed to +bind and retain an exact archive cut alongside that grant, check the complete +destination against the granted marker and preserve the same install/deadline +fences. Removing the `from_s3` refusal alone cannot provide that contract. + +## Owned workspace barrier qualification checkpoint + +This checkpoint supersedes the validation and performance boundaries below. +CG-07-wide-enrollment-01/source-08 SHA256 is +f9449fbd8937d6a2b5f9a871859bf1e7469b767df51f86f8720a2fb8fff61c57, +above d268cf0bfc78c49f8c2d48f17a4a735f42028d81. Workspace creation syncs its +new child and owned parent, reusing the open inventory's ancestor barriers. +Rooted private executors retain and validate the exact workspace directory owner; +materialized paths keep their full preparation. Read-cache directories are +disposable. No deadline, quota, profile, stack, authority fence or original +assertion changes. Child/parent sync failure retains charge and quarantine; +physical close/deletion still precedes refunds. + +Five LTX targets pass 66 cases, eight shared-group native targets pass 262, and +the two affected legacy execution targets pass 71. Their explicit process/backend +skips remain recorded. Public SDK/native/Axum restart passes in 155.92 seconds. +Same-source public recovery passes both image forms against real constrained, +pinned RustFS in 19.30 seconds after erasing every source voter directory and the +old verifier cache. Backend and dedicated VM are stopped afterward. The drill +uses embedded native Nodes and a contract fence; permanent Docker launch fencing, +independent voter containers, managed recovery and sustained load remain open. + +The original performance suite still has eighteen passes and one failure: +occupancy-128 enrollment reaches 58 cells, then returns ambiguous DeadlineExceeded +under its original sixty-second population budget and five-second call cap. +Both 32-cell cases and reclamation pressure pass. Preserved probes locate more +time in physical apply preparation than publication or collection in that sample; +the image/page split still needs measurement. Two startup-pipeline experiments +failed and were removed, along with every temporary diagnostic log/test. The +original scheduler and applied-equals-committed enrollment fence remain. Exact +failed sources, executables and fixture disks are retained. + +Feature/build/static checks and strict LTX library Clippy pass. Strict Cellar +library Clippy retains the same 222 diagnostic keys; plan index/Purpose and +reviewed-protocol drift failures remain. Full-library inspection was not rerun. +CG-07 remains IN_PROGRESS; PR #160 stays draft and the goal remains active. + +Next: retain node-owned archive resources for groups activated after startup, +including joined stop/drain ownership, then implement the currently refused +exact-reference S3 learner path. `group_host/archive.rs` registers only the initial +prepared roster; `controller/provision/mod.rs` explicitly refuses S3 transfer. +Keep archive initialization explicit and journaled, reauthorize the actual +destination boot/catalog and require physical install, catch-up and fresh quorum. +Live manifest-page collection, remaining public/default capability gates, +deployment/read policy and CG-09–16 remain required. + +## Archive metadata/cache qualification checkpoint + +This checkpoint supersedes the performance and validation boundary below. +CG-07-wide-group-diagnosis-01/source-08 SHA256 is +4b55360d8d064379763d69f3d57b13b34074bef86b9328568da773b636b775e0, +above 3dc6e25cb23a87a263edf865d24c626d087ebee4. The open owner may reuse its +completed immutable root/page barriers, while still checking bytes; reopen and +failed reads invalidate that observation. A charged verifier cache stays below +one quarter of each existing ceiling until ordinary collection pressure. Serving +collection remains eager. Incoming buffered bytes are checked even when a valid +destination exists. No deadline, quota, optimization or assertion is relaxed. + +Forty-five LTX and 261 native cases pass, with their owned process/backend skips +recorded. The public native/Axum retained-restart case passes in 179.65 seconds. +Same-source public recovery passes both image forms against real constrained, +pinned RustFS in 33.02 seconds after erasing all source voter disks and verifier +cache; backend and dedicated VM are stopped afterward. This is an embedded-Node, +contract-fence boundary, not permanent Docker launch fencing or independent voter +containers. Sources, exact executables, failed runs and fixture disks are retained. + +The full performance gate now has eighteen passes and one failure. Occupancy 32 +passes the matrix and isolated S3 diagnostic; occupancy 128 is reached but fails +enrollment after 44 observed cells under the original 60-second population budget +and five-second per-call cap. Measure enrollment, complete checkpoint and +collector interference next without raising bounds or assuming a fixed cell-count +limit. Strict Cellar library Clippy retains the same 222 diagnostic keys; plan +index/Purpose and reviewed-protocol drift failures remain. Full-library inspection +was not rerun. CG-07 remains IN_PROGRESS and PR #160 stays draft. + +Next implementation priorities are managed dynamic archive attachment and exact +S3 learner activation, owned live manifest-page collection, complete public +lifecycle/ingress/default-backend gates and whole-group Docker-fenced recovery. +Capabilities, deployment/read policy and CG-09–16 remain required. See +[progress](progress.md) and [remaining work](remaining-work.md) for exact limits. + +## Owned live extent collection checkpoint + +The node-owned collector now covers the serving and separate archive-verifier +extent inventories after their complete reference adoption. It uses the existing +control bank and service registry, with sixteen deletions per physical turn; +reader, manifest, receive and uncertain-publication pins remain authoritative. +A cancelled close waiter cannot release admitted physical work or advertise +storage closure early. Advisory resource samples expose progress, deferrals and +failures. This does not add live manifest-page collection or weaken the offline +engine collector's joined-ownership guard. + +Capture CG-07-live-collection-01/source-01 SHA256 is +a1ea8aa8ab265debeef983589f385eb8cf4e398327fd4ee22aaa0eb7af05a931. +Initial locked compilation, LTX six apply/36 store (one child helper ignored), +held-deletion close and RF3 pressure checks pass. The pressure case configures +sixteen roots for each image role and runs eighty additional commands with the +original request budgets, final S3 proof, exact duplicate receipt and every-voter +minimum-version read. Exact source/executables are preserved before broader +regression. Final seven-target native regression passes 254 cases with 29 explicit +backend/process skips; three startup cases and the service-registry unit pass. +Native-only and LTX feature modes, strict LTX library Clippy and static checks pass. +The exact current-source public recovery case passes against real pinned RustFS +in both image forms in 24.47 seconds. Its source voter disks and verifier cache are +erased; fresh activation, receipts, new writes and retained restart pass. It uses +embedded native Nodes and a contract fence, with the backend and dedicated VM +stopped afterward; permanent Docker launch fencing and independent voter containers +are not qualified. Source-after comparisons are unchanged. + +The full native performance gate is still red: sixteen pass, including the new +collector pressure case, and two wider-group cases refuse during enrollment after +31 cells. The matrix reports ambiguous Runtime(DeadlineExceeded); isolated +32-cell diagnostics reports an elapsed deadline with a changed leader/not-ready +sample. The original budgets remain; 128 and the prior source's S3 failure point +are not reached. Preserve both earlier and current failures. Strict library +Clippy still has 222 diagnostics/202 dead_code with unchanged code/message/primary +file keys. Full-library inspection, plan and remaining CI gates stay open. See +[progress](progress.md) for exact scope; the MVP is incomplete. + +Next: resolve the wider-group S3 deadline and repeated complete verification, +finish owned live manifest-page reclamation, dynamic managed archive attachment +and exact-reference S3 learner activation. Preserve the old failed sources and +bounds. The whole-group disaster operator, permanent Docker launch fence, +remaining public gates, default capabilities/deployment/read policy and CG-09–16 +remain required. The goal is active and PR #160 stays draft. + +## Executable consumer and image ownership checkpoint + +This section supersedes the historical consumer checkpoints below. The restored +consumer port uses canonical shared ranges, explicit replicated enrollment, +logical versions and physical commit witnesses. Native and Python routing vectors +agree at occupancies 1, 32 and 128. Resource observations count actual groups and +logical cells separately. Configured archive ceilings remain separate from the +effective serial publication lane. Wider groups fit the original replay bank +using smaller tails. Local source 14 all-binary compilation clears the original +27 performance errors; final combined-source qualification is separate. + +Capture CG-07-perf-resume-01 preserves the earlier failures and exact source23 +executable. Python source 20 passes 282 tests with one environment skip. Native 23 +has fifteen passes and two failures: both RF3 scenarios reach the one-cell +second group's S3 command (logical cell 6) and return an ambiguous DeadlineExceeded +under the original five-second command budget. Occupancy 128 is not reached. +The instrumented isolated 32 run 24 passes in 83.44 seconds, but does not clear +those failures. Its full 32-cell checkpoint verification takes roughly 4–5 seconds; +a complete background job reaches 5.206 seconds. Investigate repeated complete +checkpoint verification and cross-group archive-lane interference next. Keep +all original deadlines, quotas and explicit S3 assertions. + +The materialized retention failure is corrected locally. Exact terminal command +and enrollment reconciliation retires private previews while keeping their pins +through physical completion. A fresh applied image transfers its pin into the +complete manifest graph without retaining an extra permanent host reference. +Startup recovers every physical group's current/checkpoint closure before +releasing conservative filename references, with no routes reachable yet. +Uncertain publication, checkpoint and reader pins still protect their closures. +The public native/Axum process drill 19 passes 80 additional commands before and +after retained restart using the original 64-image bank, in 166.22 seconds. Its +exact source/artifacts are retained. Extent preparation similarly makes only +fresh roots transient and preserves existing retention; all six LTX apply cases +pass, including live-reader and exact existing-retention collection checks. +This supplies correct eligibility, not an owned live extent collector or a +sustained-load qualification. The offline reclaim_cell_storage guard remains. + +Qualification 25 is terminal: all seven targets pass 254 cases with 29 explicit +backend/process skips. Execution 179, storage 15, archive 30/controller 16, managed 10, +host 3 and public 1 pass. The extended public drill finishes in 135.69 seconds. +Postchecks 26 pass all binary targets, three startup cases, native-only libraries, +LTX feature-off/all-feature builds and static checks. Strict library Clippy keeps +222 diagnostics (202 dead_code), with unchanged message/primary-file keys. +Source 25's 2710-input manifest SHA256 is +8c2d708074d6c265dd69fe9edb62694495998a517900700a1cc3e97e97eefb92; +both final comparisons have no changed inputs. The failed full performance gate, +full-library inspection oracle, plan drift and remaining CI are still open. +No current-source Docker/RustFS or release claim. + +Managed setup persists standby/full-range intents, reserves all physical births +before selecting ranges, initializes exact archive bindings, attaches RF3 and +enrolls logical cells. These adapters still need deployed managed qualification, +actual-leader enrollment routing, dynamic node-owned archive attachment and +exact-reference S3 learner startup. The disaster operator remains the older +per-cell workflow and is not shared-group recovery. Complete disjoint replacement +identities, whole-group seeding into public Node paths, serial namespace activation, +explicit partial inventory and permanent Docker launch revocation before its +all-voter-disks-deleted RustFS run. Preserve every original refusal, state, +receipt and restart oracle. Then finish the four missing public gates and CG-09–16. + +The goal remains active and PR #160 draft. Preserve both requested MR basename +exclusions, the eight user diagrams and all failed source/artifact captures. + +## Latest combined public recovery checkpoint + +Read progress and the ignored public_recovery_work continuation first. Final +capture CG-07-public-recovery-14 and startup-13 have identical 2136-file Rust +mappings. Archive thirty, managed ten, archive-controller sixteen, host three, +public SDK one, counter/native-only and static checks pass. All three startup +unit cases pass; full library is 1062 passed, the same retired inspection failure +and two Docker skips. Strict lint retains 222/202 dead_code with unchanged keys; +plan and 27 performance-build errors remain. All failures and original names +are preserved. + +Real-RustFS capture CG-07-public-recovery-rustfs-15 passes public recovered +startup/activation, receipts/floors, actual inventory, new writes and retained +restart in both image forms. It uses fixture fencing and three embedded Nodes +in one process, not the Docker permanent launch-fencing drill or load gate. +The owned backend is stopped with its volume retained. Inspect preservation and +publication records for exact commit, backup and any later live handles. + +Next restore/adapt the 21-file executable-consumer WIP recorded below. The new +public recovery authority/activation/close owner is ready for that integration. +Finish disjoint replacement members, serial catalog activation/partial inventory, +managed archive ownership/enrollment, strict Python oracles and 128-cell budgets. +Then complete all original public gates, exact managed S3 learner activation and +source-qualified Docker/RustFS load/recovery. Keep the full CG-01–16 goal active +and PR #160 draft. Preserve all eight diagrams and both MR basename exclusions. + +## Public recovered-node startup in qualification + +Read the new progress checkpoint and ignored public_recovery_work continuation +first. Capture CG-07-public-recovery-11 passes the complete native archive target +(30 passed, six backend cases ignored), including public Node recovery after +source disk deletion, both image forms, held-verification close, restored S3 +receipts/floors, actual physical inventory, new writes and retained restart. +The component checks at capture 11 are terminal: managed ten, archive-controller +sixteen, host three, public SDK one, counter and native-only pass. Static checks +pass; strict lint/plan/package retain recorded failures. The later joined-startup +cleanup fix passes all three startup unit cases at startup-13. Full library is +1062 passed, the same retired inspection failure and two Docker skips. Final capture CG-07-public-recovery-14 uses the identical Rust mapping as +startup-13. Archive thirty, managed ten, archive-controller sixteen, host three, +public SDK one, counter and native-only are green again. Static/lint/plan checks +and real-RustFS public recovery are still being finalized; inspect terminal +records before a final qualification claim. Keep the goal active and PR #160 draft. + +The 21-file performance consumer WIP is preserved at the path recorded in +perf_cutover_work, with manifest SHA256 +56a5989ea7e61312e0c265a89bbe745877e839b549ede39d9f413603680e6366. +It is not qualified or in the proposed checkpoint. Restore/adapt it after this +public recovery checkpoint; preserve the new services.recovery field and fixed +transport_query_reply_write Python metric inventory. The intermediate bin check +cleared 27 errors, but that did not complete Python/managed/disaster consumers. +Finish disjoint recovery members, serial catalog activation/partial inventory, +managed dynamic archive ownership/enrollment, Python fixtures/oracles and coherent +128-cell budgets. Do not publish compile-only legacy disaster behavior. + +## Latest public physical-group controls checkpoint + +Capture CG-07-group-controls-02 passes all ten managed cases, retaining all nine +parent names. The added public SDK case covers both image forms: actual physical +identity/occupancy, guarded transfer refusals and deadlines, real election, +receipt replay, old-token reads and new writes. The original membership-floor +case now holds real engine sync to prove a queued transfer expires before mutation. +Public counter inventory observes all three actual group owners. + +The SDK exposes physical log observations and separate historical archive status. +Neither grants quorum, serving or compaction permission; retained entry count is +not disk usage. Guarded transfer preserves caller term/revision/target and checks +the invocation deadline at owner execution. Lost/expired replies remain uncertain +about admission. There is no extra worker, bank or raised resource ceiling. + +The Rust manifest has 2133 inputs, SHA256 +09f5447c42949fb573c90a8d28fc0d5fc7ecedaa3bf10efbaef705d6c19029cd. +Shared execution passes 179 cases with two ignored fixtures. The three host +cases, public SDK case, counter build, native-only library compilation and five +static checks pass. All 181 execution names remain. Strict Clippy retains 222 +diagnostics, 202 dead_code, with unchanged keys; package builds retain 27 +performance errors. Plan validation retains its index/Purpose and reviewed-anchor +failures; all seven recorded inputs match 824e0a20. Do not refresh baselines merely +to pass. The first failed capture and corrected source/artifacts are retained. +Full library and RustFS were not rerun; the dedicated backend VM is stopped. + +Next feature priority remains the original performance/fleet consumer cutover: +physical group layout/placement, replicated cell enrollment, cell versions/tokens, +all HTTP/read/retry options and whole-group recovery. Complete the original four +remaining public gates, lifecycle failure/cancellation and retained nonvoting debt. +Then implement exact-reference S3 learner activation/archive-hold retirement and +refresh same-source RustFS recovery/load. Keep CG-07 IN_PROGRESS, CG-09–16 TODO, +the goal active and PR #160 draft. Consult group_controls_work in both ignored +continuations for preservation/publication and any later live handles. Preserve +the eight diagrams and exclude both requested MR progress basenames. + +## Latest managed maintenance checkpoint + +This section supersedes the historical checkpoints below. Source +CG-07-managed-maintenance-16 passes all nine managed Node cases (102.06 seconds), +retaining all eight earlier names. Both image forms complete repair, maintenance stop submitted by the survivor leader, actual Node join/signed offline proof, retained paused return, +fresh resume, terminal capacity/disruption settlement and returned-leader writes +for initial and joined members. Duplicate receipts and old tokens survive. Direct +paused writes, old-boot stop and wrong-child same-new-boot resume refuse; the wrong +child preserves PayloadConflict, an unchanged root and paused admission. + +Implementation adds bounded complete physical-group leader-load sampling, exact +returned-boot retained startup without genesis/missing-claim creation, and a +barrier-captured temporary application refusal. Effect scheduling handles the +resume race without quarantining a temporary refusal. Peer maintenance stop/resume +release their transport waiters after exact rooted preflight and bounded worker +enqueue, returning Unconfirmed while fresh proof and guard sync remain Node-owned. +This breaks the three-RPC/two-slot metadata dependency without raising capacity +or accepting queued work. Local/non-maintenance semantics stay intact. Reconcile +uncertainty through actual joined closure or fresh resumed proof, using the same +rooted intent; a submission storm competes with the worker's proof budget. + +The final Rust manifest has 2130 files, SHA256 +55e9323085d42f58d0fa77d5eb4b806eb8412e3528c7e129aab14a74b04b789d. +Broader qualification is terminal. Catalog 23 and shared execution 179 pass; +two original owned process fixtures remain ignored and all 181 names remain. +Unit build, scan five, admission three, fleet 28 (two existing Docker Engine +cases ignored), fleet-action wire six and query-wire twenty pass. Full library +is 1061 passed, the same retired inspection-protocol failure and two ignored. +All 1059 parent names remain with five scan cases added. Host three, public SDK +one, counter/effects builds, retained effects smoke, locked native-only library +compilation and five static checks pass. Strict Clippy remains 222/202 dead_code +with unchanged diagnostic keys; package builds retain 27 performance errors. +Plan validation remains 35 passed, one generated index/Purpose failure and one +reviewed-anchor error, with all seven recorded inputs unchanged from 004d1051. +No baseline, assertion, timeout, quota or stack was relaxed. Local opt-level-zero +profiles keep assertions/overflow checks; effects smoke uses the SQLite store. + +Consult managed_maintenance_work in both ignored continuations for final checks, +immutable evidence, publication and any later owned handles. All current checks +are terminal. Preserve failed captures, source/artifacts and the eight diagrams. Both requested MR progress basenames +remain excluded. No final-source RustFS, independent-host, load or release claim. + +Next feature priority is original executable consumers and public lifecycle/ +ingress/occupancy/feature-off acceptance, including the 27 performance compilation +errors, retained nonvoting inventory and lifecycle failure/cancellation +ownership. Then implement exact-reference +S3 learner activation/archive-hold retirement and refresh same-source RustFS +recovery plus replication/reclamation load. The retired inspection oracle, strict +lint and plan anchor/index gates remain required. CG-09–16 retain their complete +scope. Keep the goal active and PR #160 draft; do not merge. + +## Latest managed peer replacement checkpoint + +The follow-up on e8f9352d exports the nested startup cause through the SDK and +adds managed_group_replacement_and_joined_restart. All eight managed cases pass; +all seven original cases remain. Both image forms qualify actual peer seeding, +RawNode catch-up, joint finish, durable settlement replay, retained removed-member +physical debt, actual leader handoff, survivor writes and canonical retained +joined-member startup. Duplicate receipts, old tokens and new writes survive; +the old target-boot grant refuses before submission with no root mutation. +No runtime guard, timeout or quota was changed. The fault drill reproduces a +committed terminal CAS with lost write/confirmation replies. Provider I/O from +that settlement now becomes Unconfirmed at the movement reconciler boundary; +the next invocation resolves its permanent receipt. Corruption/authority errors +retain their typed failures. Initial promotion NotReady is also uncertain; +reconcile the same admitted intent and retain its debit. + +Final checks pass the library build, 28 fleet unit and six wire cases (two existing Docker Engine cases ignored), native-only +library compilation and five static checks. Strict Clippy remains 222/202 dead_code +with the same diagnostic keys; package compilation retains 27 performance errors. +The added plan-validator run exposes two pre-existing failures: reviewed anchor +drift and the generated index/Purpose mismatch. Their inputs match e8f9352d. +Review the anchors and generator contract; never refresh hashes merely to pass. + +This remains a local TLS/SQLite component gate with fixture catalog/attestations. +It does not complete managed_group_lifecycle. Next implement/qualify the rooted +maintenance gate, readiness, owned stop, offline proof, retained return and +fresh resume/capacity settlement for a joined member, including failure and +cancellation ownership. Then port original fleet/performance consumers, repair +the retired inspection oracle and strict lint/build gates, implement exact S3 +learner seeding/archive holds and refresh same-source RustFS recovery/load. +CG-09–16 remain required. Keep the PR draft and goal active. Consult +managed_lifecycle_work in both ignored continuations for captures, immutable +artifacts, preservation, publication and any owned live handles. Preserve all +eight user diagrams and exclude both requested MR progress basenames. + +## Latest cold managed group checkpoint + +The follow-up on b9be4de7 implements explicit standby inventory and exact-root +first range selection, journaled import freeze, catalog-owned namespace execution +and boot-bound capacity at initial birth admission. An Active hint cannot reuse +old-boot capacity. The negative admission case preserves the root; fresh actual +Ready/Active admission permits the identical proposal. Unselected reservations +retain their physical references. + +Final source CG-07-cold-managed-group-06 passes all seven managed Node cases and +23 catalog cases. Both image forms complete RF3 genesis, two independent cells, +joined close, canonical retained startup, duplicate receipts, old-token reads and +new quorum writes. Deployment bounds are explicit fixture attestations. Full +shared execution is 179 passed, two owned process fixtures ignored. Twenty wire +cases, three host cases and one public SDK case pass, along with counter/effects +builds, the original retained smoke, native-only library compilation and five +static checks. Full library has 1056 passed, one retired inspection failure and +two ignored. Package builds still fail on 27 performance errors; strict lint remains 222 diagnostics, 202 dead_code. + +All qualification jobs are terminal. Consult cold_bootstrap_work in the ignored +continuation for immutable artifacts, source hashes, failed attempts, preservation +and publication. Next feature priority is joined-member startup and whole-group +repair/drain/maintenance/stop/return/resume with real debt/accounting, then original +consumers/public gates, exact-reference S3 learner activation and current-source +RustFS recovery/load. The full managed lifecycle and CG-09–16 remain required. +Keep PR #160 draft and the end-to-end goal incomplete. Preserve all eight user +diagrams and exclude both requested MR progress basenames. + +## Latest shared membership and managed ownership work + +The user-ordered merged-source checkpoint is published to draft PR #160 at +2e400133 on origin/main 981a5417. This follow-up implements the shared +finite Ready/send fence, exact physical message admission retention, committed +configuration announcements and joined managed boot/directory/Node stop owners. +The entered-socket regression reproduced a send-fence/unreachable-hint cycle; +source-04 releases the source permit after physical exchange before advisory +feedback, retaining each actual checkpoint producer independently. + +Source-04 passes full shared execution179/0/2, managed component3, membership9/0/1, +wire20, host3, public SDK1, counter/effects builds, original effects retained smoke +and five static checks. Full library1055/1/2 retains only the retired inspection +protocol failure. Strict Clippy222/202dead remains a failing gate, with no newly +unused methods; performance compilation still has27 errors. No release/MVP claim. + +Final review reproduced rejected advertised endpoints and an invalid log owner +consuming a rooted boot. Source-06 validates node/disk before engine opening and +all immutable descriptor fields before pin/boot mutation, preserves advertised +addresses separately from bind addresses, and passes all five managed Node cases +plus the membership selection (nine passed, one owned child fixture ignored). +The final broader chain is terminal: 179 execution cases passed, two child +fixtures ignored; library 1055 passed, one retired inspection-protocol failure +and two ignored; wire 20, host three and public SDK one passed. Counter/effects +builds, the original retained effects smoke and five static checks pass. The +locked native-only cellar/cellar-python library check passes with default +features disabled and no Python runtime. Strict Clippy retains 222 diagnostics, +202 dead_code, with the same code/message/primary-file set as source-04. +Component Cargo builds still fail on 27 performance-binary errors. +Consult managed_fence_work in the ignored continuation for exact immutable +artifacts, failed attempts, backups, publication and any remaining owned handles. + +Next feature priority: nonempty managed group lifecycle and legitimate cold +range/inventory/capacity/birth bootstrap, then original consumers/inspection, +exact-reference S3 learner activation, remaining public gates and current-source +RustFS recovery/load. Active-node fixture input and metadata-only standby startup +cannot establish that full gate. CG-09–16 remain required; keep PR draft and do +not merge. Preserve eight untracked diagrams and exclude the requested MR JSONs. + +## Latest merged-source qualification and requested next work + +Local merge 9df1389f contains counter checkpoint 41ab8b4d and origin/main +981a5417 (PR #165). Capture CG-07-public-counter-03 passes the rebuilt counter, +original public SDK case, twenty query-wire cases, three namespace-host cases +(both image forms), effects build/original retained smoke and all five static +checks. Full normal library is 1055 passed, one failed and two ignored; the +failure remains the retired inspection-cancellation protocol. Performance binary +compilation and strict lint remain separate failing gates; see progress for +final source-bound diagnostics. No current combined RustFS/CI matrix claim. + +The user requests finishing this qualification and updating draft PR #160 before +implementing the shared membership fence, managed boot, owned directory refresh +and Node stop services. The upstream fence changes the retired owner only; the +shared runtime must retain prior messages through physical sends and freeze a +finite Ready boundary before membership publication. Managed startup must bind +and sync the exact boot before group attachment/publication, use managed group +permits, join directory work on shutdown, and install Node's existing stop owner. +Keep all original guards until their real replacements and tests exist. Preserve +the remaining public cases, S3 learner activation and CG-09–16 scope. + +Newest state: local counter checkpoint41ab8b4d and merge9df1389f include newly +fetched main981a5417 (PR165). PR160 remains at published fc4a8d31; do not describe +the new merged source as published or qualified. Three merge conflicts are +resolved without restoring the predecessor dispatcher or weakening assertions. +Post-merge source03 qualification is next. Record/poll its exact owned handle; +do not edit Rust while live. Source02/earlier evidence retains its original base. +Audit the upstream membership-send fence against the separate CellGroupOwner +Ready/effect/send ownership before qualifying managed moves. The next complete +feature remains actual managed boot/directory/stop integration and dependent +consumer ports; four original public gates and all backend/CG09–16 scope remain. + +Latest unpublished work on fc4a8d31 ports the public counter to one shared group +and two independent cells per node, with explicit public namespace/archive setup, +replicated enrollment and typed native/Axum projection. No production verification +imports or per-cell registration remain in that executable. Its original smoke +passes three full drills; the extra HTTP contract and missing-catalog guard pass. +The native_cell_groups_public target now exists and passes the original +sdk_many_cells_share_group case. The other four public cases remain missing; +do not replace their managed/admin/feature-off oracles with the counter result. +Source-02's overall native Cargo invocation still fails on twenty-seven old +performance API errors despite emitting its fresh public test executable. +Current-source checks: wire20 pass; host3 pass across both image forms; normal +library build and original two-run effects smoke pass. Full library1049/1/2 retains +only the old inspection-cancellation failure. Strict library Clippy234/214dead_code +is unchanged by code/message/primary-file comparison. All five static checks pass. +The next feature dependency is actual managed startup: registration refuses +managed namespaces/node-directory options; Replica bootstrap is unmanaged; Node +startup returns no Fleet stop services. Preserve capability digests, exact +catalog/node/disk/process/storage-instance boot binding, synced lifecycle guards, +permanent claims and fresh group authorization when wiring those owners. Do not +make fleet/performance callers compile by dropping their managed behavior. Read +public_counter_work in both ignored continuations for exact source/artifact hashes, +retained setup failure, completed cases and owned live handles. Do not edit Rust +while an owned build/test/format job remains live. No timeout/quota/stack increase +or assertion weakening is permitted. This newest section governs over historical +checkpoints below; the goal and CG-07 remain incomplete. + +PR #160 is the authorized draft checkpoint; do not merge. Preserve/exclude the +user's eight diagram files and both MR progress basenames. Keep the canonical PR +checkout and isolated effects checkout aligned after publication. All CG-01–16 +requirements remain intact, and the end-to-end goal remains incomplete. + +Start with [remaining work and priorities](remaining-work.md). The next MVP work +is the focused public SDK surface and counter/native/Axum/performance/fleet +consumer ports, then managed directory/startup and whole-group lifecycle, +exact-reference S3 learner activation/archive holds, and all original public, +build and current-source RustFS/DR/load gates. Transfers, split/merge, placement, +hibernation, managed recovery and full load/release qualification remain required. + +Latest source adds controller-owned public replica candidates, per-attempt +limits and an untried observed leader fallback under the original invocation, +input, minimum token, deadline and attempt bank. Authenticated reply advice is +not authority. Incoming RPCs execute locally. Queries retain physical SQL and +transport ownership after waiter expiry; responses expose the actual replica. +A reproduced SQL deadline-envelope race is fixed without increasing timeouts or +changing ordinary storage refusals. + +Fixed-08 passes twenty-one focused cases and the complete native execution suite: +177 passed, zero failed, two ignored. Fixed-07's forty routing repetitions apply +to unchanged production/new cases. The library has 1049 passed, one failed and +two ignored; all twenty wire cases pass. Host-fixed-09 passes all three host cases +across both image forms. Strict library Clippy still has 234 diagnostics, with no +additional code/message/primary-file diagnostic versus the prior checkpoint. +The overall native Cargo command still fails on unconverted counter/performance +APIs. Do not present the emitted component artifact as a package pass. + +The effects example now retries only definite-unsubmitted readiness refusals +under the same command ID/input/deadline, and lets Node drain before inspecting +Raft's final report. Original close assertions and the original smoke are intact; +close diagnostics are removed. All six original smoke trials pass at +drain-fixed-12, twelve process runs including retained-disk restart. +Production/library Rust matches fixed-08; the only later Rust +changes are the fixture's owning Entry import and this example. Effects smoke +uses the SQLite development object store. Current combined RustFS, managed DR, +load and public orchestration remain unqualified. + +Read public_replica_routing_work in the ignored continuation for exact source +hashes, immutable executables, retained failures, backup and owned handles. Finish +every owned job before staging Rust. An observation timeout is not failure; never +restart a live build/test merely because output has not arrived. Historical +sections below describe scoped checkpoints; this newest section governs. + +Latest unpublished WIP fixes retained-preparation reuse and the completion phase +of an already synced seed. Boundary probes reproduced the previous three native +failures: repeated sync of the same retained page root, and a source-process +watch rejecting completion after persistence succeeds. The probes are removed. +The original LOCK, no-extra-sync, queued-revocation, uncertainty, recovery and +fresh-quorum assertions remain. An added entered-fence expiry case preserves the +original deadline and checks cold recovery of the actual completed write. + +CG-07-seed-retained-fixed-02 passes all three formerly failing native cases, +including both image forms and the added entered-fence expiry cut. Only that new +case has a two-second operation deadline inside the unchanged peer five-second +guard; original cases retain ten seconds. Its complete native execution suite +passes: 167 passed, zero failed and two ignored in 452.06 seconds. All owned +build/test/check jobs are terminal. The full WIP, source inputs and immutable +executables are preserved under the normal checkout-specific qualification target. +Fixed-01's production/library and example Rust are identical to fixed-02. Its +normal unit target and effects binary build; original two-run effects smoke and +five static checks pass. Full library: 1048 passed, one failed, two ignored +(retired inspection protocol). Strict Clippy: 235 diagnostics, including 215 +dead_code, with none in the six production files changed for these fixes. +The overall native Cargo command still fails on fifteen counter-binary API errors. + +The earlier gateway-backend-fixed-03 checkpoint has 164 passed, three failed and +two ignored; all 21 targeted gateway/cold cases and 39 lifecycle cases pass. +Fixed-02 resolves those remaining three failures and passes the complete native +suite. This component result does not replace the failing package/library and +strict Clippy gates. + +Canonical archive startup creates/syncs its scratch parent before archive work. +The effects example retries typed busy/overload with the same command identity, +input and original deadline. Eight retained runs pass at the earlier fixed-02 +source and two more at gateway-backend-fixed-03. These are SQLite development +object-store smoke runs, not RustFS, managed recovery or load qualification. +Fixture backend declarations and cold oracles retain exact immutable namespace +metadata. Runtime/domain/schema/backend guards remain unchanged. + +Next port public counter/performance binaries and real managed shared inspection +cancellation, +finish canonical managed directory/startup and exact-reference S3 learner +activation. All five original CG-07 public gates, refreshed real Docker-fenced +RustFS recovery/load qualification, CG-09 through CG-16 and the combined hard +cutover remain required. Strict Clippy and the old inspection cancellation test +are open. At that earlier checkpoint the WIP remained unpublished; no final +system/MVP completion was claimed. +Historical sections below describe scoped checkpoints, not combined qualification. + +The executable design and implementation plan is delivered. This handoff scopes +the remaining end-to-end implementation and hard cutover, including the full task +and release gates, for subsequent execution. +This handoff does not claim completion of the +shared runtime, a production cutover or measured TiKV +performance parity. The target is the same partitioned multi-Raft pattern: many +whole cells per group, one Raft machine per group replica, and many groups sharing +node resources. A group spans voter nodes; it is not all cells on one node. + +## Starting point + +Reuse codex/shared-cell-groups and [PR 160](https://github.com/crabbuild/cellar/pull/160). +Keep the isolated effects checkout and PR checkout aligned at each published checkpoint. +Preserve the user's untracked diagram and both excluded MR progress basenames. +The current goal remains incomplete: the SDK now uses the shared host, while +managed startup, remaining public consumers and the final combined release/load +envelope are unqualified. Historical sections below describe earlier checkpoints; +the newest progress section governs current implementation and validation. + +CG-01 through CG-06 are DONE at their scoped native boundaries. Their refreshed +records remain under .cellar/cell-groups-evidence/CG-01-06-gates-e32115aa-02/. +CG-06 also passes the actual Docker-fenced, all-voter-disks-deleted RustFS drill +in both image forms at f8aacd05. Fresh history/genesis/quorum activation, receipt +restoration without replay, S3 minima, old-launch refusal, new writes and retained +restart are qualified at that unmanaged local boundary. Its Docker record is +.cellar/cell-groups-evidence/CG-06-docker-f8aacd05/qualification.json. + +CG-08 is now DONE at 4ea70e483e84ca88d77ca062278edb5ff2389646. All twenty-two current-source commands +pass, including all ten original required commands, ten exact native effects +cases, twenty-four library regressions and the full shared execution target +(164 passed, 2 ignored), and the affected archive-controller suite +(16 passed, 20 explicit RustFS ignores). The same-source/executable real RustFS case passes +both image modes with seven signed probes and independent full archive replay. +The backend and dedicated VM are stopped, data retained and default context +preserved. Source preflight and unchanged task evidence validation pass. +Record: .cellar/cell-groups-evidence/CG-08-gates-4ea70e48-01/record.json +SHA256: a072fb6fd8957b6afd5fdf4dffc88c312a0e34c2e1f5c4cfd81b4d2456ea9d0c +Executable SHA256: 85ceb50508a519e2b1fd5198f47969185438f77c30d09e9de6ab884fd6a7cce6 +All source hashes, real streams, original capture record and executable are +retained locally and backed up. See [progress](progress.md) for scope and failures. + +The namespace-host component now passes its ten-command checkpoint. It has +bounded namespace/class registrations, history-derived SQL paths, logical cell +handles, controller-owned queries and original enrollment deadlines. All three +new host cases pass both image forms, including eight cells in one physical group +on every voter and held-query shutdown. Execution/archive/effects regressions and +strict checks pass. Evidence is under +.cellar/cell-groups-evidence/CG-07-namespace-host-01/record.json. +This binds already bootstrapped services; canonical startup, public SDK routing, +ingress and managed whole-group lifecycle remain unqualified. Move to those +consumers next, using this component rather than another per-cell binding owner. + +CG-07's synchronous command-admission checkpoint now passes all nine focused +commands: the exact new regression, full execution/effects/archive suites, +all-target strict Clippy, format/layout and dependency/docs checks. The record is +.cellar/cell-groups-evidence/CG-07-command-admission-fixed-02/record.json. +Its source patch, 4680 source hashes, three executables and actual streams are +retained and independently backed up. This is a shared-runtime prerequisite; +the public SDK/fleet cutover and all original CG-07 acceptance cases remain open. +Next implement the canonical namespace/class registry and shared host with logical +cell handles, then complete managed whole-group lifecycle and accounting. + +The latest managed routing/repair work converts Begin/Promote/Finish to actual +physical-group controls (Abort is also implemented). Rooted history, permit/plan, +phase/revision and full occupant placement checks precede normal configuration; +actual durable apply plus a fresh quorum precede success. Native RawNode catch-up +and original deadline/uncertain-result rules remain. At that historical checkpoint Activate was still the old per-cell consumer. +The later peer learner work has converted it; S3 exact-reference transfer and +current-source qualification remain open, without peer fallback for S3. + +Managed route discovery projects committed joining bindings through rooted node +views. Configuration apply/checkpoint install trigger reconciliation; ordinary +known-peer traffic and cell writes keep their hot path. One missing-target retry +is bounded before frame admission. Stable physical/checkpoint factory lifetimes +survive advisory refresh; individual transfers retain exact route generations. +Explicit replacement/removal still revokes old owners and stale refreshes. New +factory views are selected once per transfer, not per chunk. Two new parked-engine +registration regressions are uncompiled/unexecuted. The existing query-auth test +body is byte-identical after moving its fixture to a reusable sibling. + +Final CG-07-managed-routing-advance-04 retains 4726 inputs (SHA256 +bfdbc3c3ff638d7cfd20b8034626e825e33c9fd09e3e9be78beb8713e6f14053) +and the same 36 inherited library errors, with no new route/repair diagnostic or +non-vendor warning. Intermediate type errors and their fixes remain recorded. +The standard unit-test compile attempt, CG-07-managed-routing-test-01, stopped at +that ordinary-library failure before test modules compiled; it ran no tests. +Read the newest progress section and exact owned handles before editing. Full +managed startup, all five original gates, fresh native/RustFS/load qualification, +archive hold retirement and CG-09–16 remain open. Do not publish unqualified WIP. + +The latest managed-genesis implementation replaces the receiver's old cell +registration/activation with activate_managed_initial_group. Catalog I/O enters +the installed execution domain even when polled elsewhere. The SDK rechecks its +namespace capability, rooted birth and actual initial voter plus synced boot, +then delegates whole-group recovery/attachment to the existing provisioning +owner. It returns only the actual durable fleet guard, never a quorum/serving +certificate. Original deadlines, late-reply suppression, owned physical joins, +retained-preparation retry and current-roster/shared-engine failure checks remain. +Attached retries validate without reopening pages or adopting the existing +registration lifecycle. Immutable namespace/directory/backend profiles share the +node image inventories and a single preparation/parallel bank. Their caches and +fixed peer metadata are charged within the original total. Canonical startup +configures these explicit resources; the managed startup gate is not lifted. + +The first two locked builds report 36 errors rather than 40, retiring all four +genesis receiver errors with no direct new code diagnostics or non-vendor +warnings. Final capture CG-07-managed-genesis-advance-04 includes original +unconsumed-reply expiry, the existing 60-second maximum and typed catalog-error +preservation. Its terminal result also has 36 remaining errors and no direct +new diagnostic/non-vendor warning. All 4723 final source inputs are retained +(SHA256 25b5a30439963dea58e34d16208b4f2572f37d6fa257c1c35dc2c8afca4ae3c6), +and all captured Rust inputs match at validation closure. All four compiler +handles are terminal. Final formatting, layout, dependency isolation, +documentation and diff checks pass; all owned validation processes are terminal. +Static review retains all 43 original assertions plus seven new profile refusal +assertions, without executing or compiling them. Read its review-resolution.json +and the newest progress section before editing; no Rust edit while an owned +build/check is live. Existing provisioning +behavioral assertions remain, with namespace profile/backend registration and +new early-owner/budget/duplicate refusals. The tests are still uncompiled and +unexecuted. Next use these profiles/owners for managed peer/S3 learner activation, +real group repair and trusted target routes, then remaining legacy retirement, +node-directory/managed startup and all five original public gates. No managed +native/RustFS qualification or full-goal completion is claimed. + +The latest path correction makes canonical startup, controller provisioning and +independent archive replay share ManifestPageStore::group_pages_path. Serving +owners use groups/////pages under the +same node base; the verifier still has an independent base. Existing physical +fault/restart test bodies are byte-identical outside their independent expected +path helper. Their original assertions remain unexecuted and uncompiled. The +workspace-volume locked library check still has the same 40 errors, with no new +primary path-code error or non-vendor warning. CG-07-provision-path-advance-01 +retains 4722 source inputs (SHA256 +87cdfd5273f790446ac1753f5593bdfe88d3e5ee8f0a6fb2a095298e5496bad3). +Final formatting, layout, dependency isolation, documentation and diff checks +pass; all owned validation processes are terminal. The new path is implemented +but unqualified. Continue namespace-bound image +profiles and existing controller-owned managed activation/registration, followed +by real repair/genesis consumers, old caller retirement and the original gates. + +The newest seed implementation adds prepare_managed_group_seed on the existing +shared complete-checkpoint stream. Fresh root admission binds the active target +reservation and the selected source's member/node/disk/process boot. Authenticated +metadata must contain the exact Seeding permit/plan; revisions cannot regress and +catalog time consumes the original deadline. The serialized engine checks its +last synced destination boot at stage and before install fence. Late delivery +cannot erase durable receipt/debt or report late startup success. Managed joined +bootstrap/revalidation requires the actual receipt, permanent claim, rooted Live +group and exact running boot. Both genesis-guarded and later-enrolled source +origins retain normal successor checks. Unmanaged entry points check namespace +ownership. Catalog preflight/staging/joined delivery use the original transport +deadline plus process/shutdown cancellation; catalog and handshake time is deducted +before forwarding, and ownership reads stay inside directory operation bounds. +All three checks report the same 40 outstanding +legacy errors, with no direct new seed/storage/bootstrap/source-auth diagnostic +or non-vendor warning. All 4722 captured source inputs are retained and all Rust +inputs matched the final CG-07-managed-seed-advance-03 at its validation point. The workspace volume is writable +again; the reviewed build/checks return to its normal checkout-specific target. +The final formatting, layout, dependency, documentation and diff checks pass; +all owned validation processes are terminal. +Five new descriptive profile/origin/guard regressions are unexecuted and grant no +physical/quorum proof. No managed/native/RustFS qualification follows. Read the +newest progress section and invariant-source-review.json. Next use existing +controller provision/registration ownership for managed learner activation, add +actual group repair begin/advance and managed genesis consumers, finish managed +S3 learner restore, remove old seed/inspection/host callers and pass all five +original CG-07 cases. Hold retirement, refreshed native/backend gates and CG-09–16 +remain open. The subsequent path correction described above unifies provisioning +and canonical startup. Its one image-backend profile still needs immutable +namespace-bound profiles sharing node admissions. Retain/extract the common +seed_transport deadline/process/shutdown helpers when removing old seed RPCs. +Do not enable managed canonical startup or publish before qualification. + +The newest query cutover removes retired per-cell SDK query admission/types, +host query execution, routing loop and request/reply body codec. QueryRoute now +accepts only a physical group plus logical cell, refusing the old Cell variant. +Current query/command/enrollment/receipt framing remains purpose separated. Common +MAC/framing, exchange, origin permission and deadline/cancellation/revocation regions +are byte-identical to baseline; the source review is not executed qualification. +Wire security oracles now use logical tokens/physical routes. The original grant +oracle uses synced shared engines/start_shared transport and retains all 18 +assertions; a new regression requires old route/request/reply refusal. Tests are +unexecuted. The sequential locked check falls from 65 to 40 errors, with no primary +query-path error or non-vendor warning. Remaining host.rs errors concern predecessor +startup services. All 4719 captured source inputs are retained and all Rust inputs +still match the reviewed final CG-07-query-cutover-advance-03 capture. All three +sequential checks report 40 errors. Review restored exact returned-token equality +in the ported wire oracle before the final formatted source; tests remain unrun. +Format/layout, native dependency, +documentation and diff checks pass; all owned validation jobs are terminal. +The workspace target remains EPERM; +only this checkout's ignored temporary system-disk target is reused. No new tests, +Clippy, RustFS or managed qualification is claimed. See the newest progress section. +Next finish actual complete-group managed repair/learner seed and genesis owners, +remove old seed/inspection/host consumers, install directory/startup ownership and +pass all five original CG-07 cases. Convert old public read/Axum fixtures without +weakening oracles. Hold retirement, refreshed native/backend gates, CG-09–16 and +the release matrix remain open. Do not publish unqualified WIP or mark the goal done. + +The newest action work routes fleet execution through the installed shared SDK, +with synchronous original-deadline/control admission, exact synced boot checks, +internal observers and unchanged authenticated remote receiver checks. Restart +handoff/resume now address GroupHistory. Actual serialized group owners recheck +RF3 term/revision/committed and survivor matched/active progress, plus the exact +synced quiescence selector for transfer. Expiry is tightened to the original call; +late entered work stays uncertain. Obsolete per-cell fleet action dispatch is +removed and restart reporting carries physical histories. The first diagnostic +falls from 90 to 70 errors; after fixing stale report types the final rebuilt +library reports 65. All captured Rust inputs match; no primary diagnostic is in +the converted action/readiness/handoff/resume paths. Four new regressions are +unexecuted and no runtime/backend qualification follows. Format/layout, native +dependency, documentation and diff checks pass; all owned jobs are terminal. Read +CG-07-group-actions-advance-01/-02 and the newest progress section. The workspace +volume still returned EPERM after cleanup; the final rebuild uses a fresh ignored +checkout-specific system-disk target, as disclosed and recorded in environment.json. +Preserve the original evidence and user diagrams. Next remove legacy query +consumers while retaining shared authenticated framing/security oracles; finish +actual whole-group managed learner/repair/genesis and installed directory/startup. +All five original CG-07 cases, explicit hold retirement, refreshed native/backend +qualification and CG-09–16 remain required. The goal is incomplete. + +The newest fleet decision core uses physical GroupHistory throughout reservation, +topology and isolated repair, with GroupRepairPlacement results and no cell-name +identity. One replica costs one engine slot while full image/staging/memory debt +remains charged. Namespace-qualified group IDs stay distinct; two current +incarnations cannot coexist in topology, and a new incarnation cannot reuse or +erase old reservation debt. Docker recovery now takes RangeDirectory handles and +checks rooted group authorization before repair observations. All 96 enabled core +tests and strict all-target Clippy pass, including three new identity/debt cases; +the first missing-import build failure is retained. Eight binaries and 4718 source +inputs are hashed under CG-07-group-model-advance-02. The runtime diagnostic falls +from 100 to 90 errors, with no primary error/warning in the converted paths and no +captured Rust input changes. Format/layout, dependency, documentation and diff +checks pass; all owned validation jobs are terminal. No public SDK, managed, +Docker or RustFS qualification follows from pure core tests. Preserve shared TLS query/command/enrollment/receipt framing while +removing the old per-cell query consumers, then complete group action RPCs, +directory ownership, startup and all original CG-07 gates. Persistent hold +retirement, refreshed native/backend gates and CG-09–16 remain unfinished. + +The newest implementation replaces both managed removal verifier callers with +complete group replay and explicit private physical inventories/shared pools. +A persistent acquire journal chooses one intent before S3 mutation, restores the +same fixed cut after restart, and reconciles its exact allocation after lost +responses. A new intent needs a fresh registry proof that the previous revision +can no longer commit. Receiver admission precedes page scans and follows physical +ownership through cancellation. Format passes. The locked diagnostic falls from +112 to 100 errors, with no primary error/warning in the new paths. All captured +Rust inputs still match. Read CG-07-managed-archive-advance-01 and the newest +progress section. Format/layout, dependency isolation, documentation and diff +checks pass in the corrected toolchain environment; all owned jobs are terminal. +Five new regressions and Clippy are unexecuted while the library fails. Persistent hold retirement remains explicit and +unfinished; do not release on drop or borrow a publisher hold. Next close remaining +group models/action RPCs/legacy consumers/directory/startup, convert old verifier +fixtures, and pass all original CG-07 and refreshed native/backend gates. CG-09–16 +and full release qualification remain open. No runtime changes are published. + +The newest work cuts fleet quorum RPCs over to the actual shared SDK gateway +and complete physical group roots. Published cuts commit full membership plus +configuration; authenticated subtree mutation summaries distinguish data changes +from protocol-only progress. Fleet archive verification now uses complete group +checkpoint/tail replay and exact persistent holds. Live receipt payloads are +hash-addressed under the existing page bound while index rows and batch byte +ceilings remain unchanged. Per-cell fleet quorum requests/read variants and the +old host RPC implementation are removed. The first library check has 120 errors; +the second and final checks each have 112, with no primary error in the new +metadata/coverage/summary/receipt modules. The final check includes configuration +order and descriptor ownership review. All 4714 source inputs are captured and +all captured Rust inputs still match. Read CG-07-fleet-root-advance-01/-02/-03 +source captures and the newest progress section. Format/layout, dependency, +documentation and diff checks pass. Regressions and Clippy are unexecuted while +the library fails. All owned jobs are terminal. No native/backend qualification +or publishable runtime is claimed. +Next implement the owned hold-acquire journal and full replay receivers for both +managed verification adapters; do not reintroduce a single-image archive or fake +scope. Then group placement/repair, action RPCs, node directory, managed startup +and all original CG-07 gates. The changed marker-1 root shape needs fresh full +qualification; earlier CG-06/08 backend evidence remains historical/source-bound. + +The newest SDK work adds physical node inventory and authenticated inspection +from the real shared engine/roster, preserving pending provisioning debt, +quarantined/unattached slots and original deadline/budget ownership. Transport +routing uses a weak shared-SDK gateway and the actual installed trust cut. Local +node profiles can now be configured; managed startup is still gated. The first +sequential diagnostic reports 124 errors with no primary error in the new node +metadata modules. The second locked check also reports 124 errors with all +captured Rust bytes unchanged. Follower-safe membership metadata is implemented: its distinct +ReadIndex context binds the origin member and accepts replies only from the +current leader, while execution/seed contexts and follower execution refusal +remain unchanged. The new two-image, three-engine regression is unexecuted while +the library fails. Read CG-07-node-inventory-advance-01/-02 captures and the newest +progress section. Next convert the whole-group fleet observation/publication +codec and complete-root archive/removal proofs, then repair/restart RPCs, the +installed directory owner, managed startup and all five original CG-07 cases. +This is local unqualified implementation, not a native/backend release gate. + +The newest SDK work adds guarded physical-group enrollment and permit fencing +with typed refused/uncertain outcomes and a fresh quorum observation after durable +apply. Minimum membership reads defer ReadIndex until apply and re-arm after a +configuration overtakes the barrier, retaining their original budget/deadline. +A fixed control-service registry joins actual movement/restart worker lifetimes +before consensus/storage shutdown; internal observers do not retain the last +external SDK lifetime. Startup checks installed host/boot/lifecycle identity. +Managed canonical startup remains gated. The first diagnostic reports 165 +errors; the final captured check reports 157 and no direct diagnostic in the new control/read/service modules. The new +SDK now retains the real transport and sends exact-boot stop/fence RPCs on its +existing authenticated lane, with original deadline and control admission +captured before returning the waiter. The receiver owns admitted mutations and +no cell/group alias is chosen. Captures are CG-07-physical-controls-advance-01 and +-02. The new real-sync and service ownership +regressions are unrun while the library is failing; read the source capture under +CG-07-physical-controls-advance-01 and newest progress section. Do not treat this +as native or RustFS qualification. Next implement the physical RPC/inspection +owners and whole-group archive/removal proofs, then managed startup and all five +original CG-07 acceptance cases. + +The latest spare work replaces namespace import's cell inventory with an +admitted physical-group set and an immutable frozen topology cut. It settles +unselected pending births without dropping debt, verifies whole-group quorum +coverage and every selected range, and performs exact-owner adoption plus a +revision-advancing topology unlock. Managed restart birth settlement and the +maintenance/resource/operator inventories now take physical directories. Public +movement/debt views omit a cell field; standalone connection requires explicit +group_directory limits. The first locked diagnostic has 194 errors; its joined +method mismatch and unused re-exports were corrected before the follow-up. +The second diagnostic has 177 errors and no own warning; the import modules +emit no direct diagnostic. Resource/movement/topology/repair models now use +GroupHistory, with final capture CG-07-group-import-advance-03. The new real-CAS +freeze regression remains unrun until the canonical crate builds. Read the three +captures and newest progress section. No +new managed runtime, RustFS, load or release qualification is claimed. Continue +with physical SDK/control RPC owners and whole-group archive/removal proofs, +then managed startup and all five original CG-07 acceptance cases. + +Review found that several admitted candidates can share a logical range and +epoch. The fleet range-debt key now includes physical group/incarnation identity; +the directory retains exclusive selected-range ownership. A new invariant test +covers distinct pending keys at the same range/epoch. The third sequential check +reported 174 errors with no own warning before that fix; the final reviewed +source capture is CG-07-group-import-advance-04. The final locked library check reports 174 remaining integration errors; +all captured Rust bytes remain unchanged. The new admission, freeze, import and +inventory modules emit no direct diagnostic, which is not qualification. Both +new regressions remain unexecuted until the canonical library builds. + +The newest spare work converts fleet bindings/birth records to physical group +histories and ranges, adds guarded managed genesis and exact local-boot startup +ordering, and adds a fresh ReadIndex membership observation independent of cells. +Inventory traverses complete physical range pages and retained reservation debt. +Earlier cell-bearing records refuse under the new marker-1 protocol identity. +The first diagnostic reported 202 errors and overlapped formatting; it is retained +without qualification. The latest sequential check reports 198 remaining errors +after the later inventory, quorum, boot, typed-error and runtime-history edits. +All new core assertions remain unexecuted, managed startup is +still gated, and old import/cancellation/inspection/movement/restart consumers +must be converted. The separate primary directory pager passes all seven checks +at 66d5ea6a, including nineteen native catalog cases. Its reviewed baseline +passes all four post-gate checks; PR 160 and both checkouts are at c54307c5. +All WIP and diagrams are preserved; managed cutover changes remain unqualified. See the newest progress section and ignored +CG-07-managed-group-advance-01 through -04 records before editing. + +The spare SDK is now a group_client component with close, drain and resource +owners. Node shutdown/stop proofs use complete group counts and explicit producer +quiescence; one unfinished command per group is no longer assumed. Transfer uses +actual owner progress with term/revision fencing. Namespace stores are released +by the controller's joined owner, independently of response cancellation. The +latest spare library check has 114 remaining fleet/query/predecessor-host errors; +no canonical acceptance case is qualified. See the newest progress section. +The minimal closed-manifest-pin fix passes twelve checks in the primary checkout +and is published on PR 160 at 110b7a4e (runtime 51d136fb). Both checkouts share +that base with all WIP and eight diagrams preserved. Keep its source-scoped evidence separate from the +unqualified cutover. Its recovery regression retains closed log/group handles +through fresh recovery in both image forms. + +CG-07 is IN_PROGRESS in the spare PR checkout. Canonical startup now assembles +namespace registrations, whole-group placements, selected-root recovery, shared +SQL/image resources, controller archival/effects and logical SDK handles. Public +admission and remote ingress are being connected to the same physical owners. +These changes are uncommitted and intentionally incomplete: the first locked +library check reported 132 unconverted SDK/fleet/query/shutdown integration +errors; the second reports 117 after the physical-query and native-effect edits. +No executable or acceptance case is qualified by either run. Preserve all +dirty/new Rust files and the user's diagrams. Do not restore +legacy startup, reinterpret GroupScope.cell or add a mode flag. See the latest +progress section and the diagnostic artifacts in +.cellar/cell-groups-evidence/CG-07-canonical-startup-advance-01/. +The exact source state and remaining work are recorded in the spare checkout's +.cellar/cell-groups-evidence/CG-07-cutover-progress.json. +The previous source preflight is not evidence for these new runtime inputs. +Read plan.py show CG-07 and the existing production callers before editing. + +1. Replace canonical host/startup/registration with group placement and cell + enrollment; do not add a legacy/shared toggle. Use the existing shared runtime, + controller, coalesced provisioning, directory and registered dispatch owners. +2. Replace per-cell binding capacity with bounded namespace/class registrations + and separate group/cell occupancy. SDK handles stay cell-addressed and carry + logical cell versions with their original commit witnesses. Preserve typed + ambiguity, caller command identities, strong reads and ingress authorization. +3. Convert managed import, capacity, repair, drain, maintenance, stop/return/resume + and topology to complete group replicas while retaining CAS journals, + permissions and disruption debt. Update native/Axum/admin/CLI/examples together. +4. Qualify all five original CG-07 cases and its complete source/evidence gate. + Occupancy one must use the same path. Feature-off Fleet contracts remain intact. + +The source-bound investigation record in the spare PR checkout is ignored: +.cellar/cell-groups-evidence/CG-07-startup-inspection-e8edfd12.json. +It records integration gaps, not implementation evidence. CG-09 through CG-16 +remain TODO: transfers, topology, placement, hibernation, managed recovery, +workload/crash qualification and deletion of superseded consumers. +The full goal stays active. Final CG-16 repeats every required capability and +profile at one release source; earlier isolated passes cannot qualify it. + +Required nullable recovery metadata stays an internal marker-1 hard cutover. +There is no legacy reader or state migration. Prepared recovery data alone is +never engine/serving authority. Keep unknown outcomes and physical joins owned. + +The prior whole-group fencing-barrier source is +65c5176975907fb52a53deca58e1ae1f5ba7adec, with 211 native cases and four real RustFS +resume/race scenarios. Its evidence remains at +`.cellar/cell-groups-evidence/CG-06-recovery-fence/`. Historical qualifications +remain scoped to their recorded sources. + +The prior bounded-outage qualification source is +1159e9358c106fe1260c3479759e909e413257a4. CG-06 remains IN_PROGRESS. Its +original bounded full-S3-backend outage case now passes in both image forms. The +normal case faults the adapter shared by catalog and archive. The real RustFS +case stops and restarts the owning runner's actual backend container, preserving +its pinned identity, owner, image, volume, resource limits and endpoint. Actual +S3 probe I/O fails during stop and recovers after start; no real-mode wrapper +fault substitutes for that outage. + +Warm local Quorum commands progress to the explicit nine-entry retention ceiling +while fresh remote discovery refuses without executing a callback. S3 commands +may commit locally but cannot return S3, and stored minima survive downgrade +attempts. At capacity, callbacks/initialization refuse before execution and +known receipts remain available. Independent complete archive recovery precedes +publication of the held exact checkpoint. Every voter then selects its complete +cut and retires actual debt, allowing new work. A failed remote invocation +retries with unchanged captured inputs, identity, deadline and budget. All three +voter images and logical versions remain exact; eight callbacks occur once. + +Twenty exact real RustFS cases pass in both image forms, with seven signed +probes and four owner-checked backend stop/start transitions. Twenty-three +checks pass with 270 Rust cases, seven copied executables and 2580 source +hashes. All four native shared-group suites, all 22 storage cases, locked +all-target Clippy, format/layout, dependency/docs/preflight/plan validation, 37 +plan tests, five controller tests, system-Python hashing and independent +source/artifact/selection/credential/backend-transition audit pass. Evidence is +under .cellar/cell-groups-evidence/CG-06-full-backend-outage/. The first-source +inventory mismatch and copied executable reproduction remain retained; the +correction verifies actual indexes 2, 3 and 4 on all voters and fills the +unchanged entry ceiling with one additional real command. Earlier dependency and +physical-reclamation qualifications and the unresolved 8dfe9ff3 slow-reader +timing limitation remain retained. + +Four original CG-06 cases pass: whole-group dependencies, bounded S3 outage, +archive tails and physical reclaim/restart. The remaining original case and full +task gate require fenced unmanaged recovery into fresh group/cell incarnations +and a serving replacement quorum. Inventory every possible old voter, including +joins after the selected cut. Use the accepted fencing-authority API plus Docker +qualification, permanently prevent old-host restart, preserve selected-cut +results/receipts, reject old tokens, initialize new histories and require fresh +quorum before serving. + +CG-08 effect/cache/result lifetimes precede CG-07 public SDK/fleet cutover. +Irreversible source sealing and activation, split/merge, placement, hibernation, +managed recovery, real load/crash qualification and final legacy removal follow. +Do not add a new cached-authority shortcut: live Seal/Install/Activate controls +still need their durable transition lifecycle and exact publication proofs. +Current outage evidence covers existing static groups, warm local routes and +fresh remote refusals. Host-native voters share one machine and RustFS one VM; +global physical disk bounds, sustained throughput, independent domains and +deleted-voter fresh-incarnation disaster serving remain unqualified. Preserve +all predecessor evidence, excluded MR-13/MR-14 progress basenames and the user's +untracked diagram. The full sixteen-task goal remains active and incomplete. + +CG-01 through CG-05 are DONE at their declared native task boundaries. CG-05 +closes at `e0550acd` with all five exact snapshot cases and eighteen passing +commands, backed by 205 Rust cases, 337 source hashes and five executables. +Its source-bound record is +`.cellar/cell-groups-evidence/CG-05-completion-admission/qualification/checkpoint.json`. +Independent hashes, full `check_evidence` and the closure audit in [progress](progress.md) +pass. CG-04's earlier 83 exact-case/238-regression gate remains retained at +`69e41e86`; no task requirement or acceptance oracle was removed. + +The final CG-05 slice fixes wrong-size wake starvation and transient item charges +from byte-rejected budget attempts. It exposes advisory runtime lane usage and +the existing exact-ticket checkpoint publication boundary. Both image forms fill +all eight completion slots with actual fsynced work, retain the ninth checkpoint +through admission and a held real engine fence, then finish its exact successor. +Two concurrent writers share one checkpoint slot through a bounded 64-enrollment +trace, inspect both newer checkpoints before the writers finish, and verify every +cell and both selected closures on all three nodes. Every new native scenario runs +directly inside the exact resource-bound acceptance case. + +Continue with CG-06 complete group S3 archival, verified checkpoint/tail coverage, +safe log reclamation and fenced unmanaged recovery. Follow with CG-08 effects/cache +lifetime, then CG-07 public SDK/fleet cutover. Idle-node orphan maintenance must be +invoked by managed startup with the complete durable-reference inventory. The +entry-count policy retains the idle tail; idle/time/byte checkpoint policy and +coordinated wake remain future work. Real RustFS/load, broader crash schedules, +independent domains, managed disaster recovery and final removal of the old host +remain required later gates. The full end-to-end goal is not complete. + +The following checkpoints retain their historical source and partial boundaries. + +The preceding checkpoint at `bad0387c` removes checkpoint scheduling's +dependence on free SQL apply slots. The bounded pending scan still visits other +groups when SQL is full, starts checkpoint work within its own capacity, then +defers SQL work without increasing its limit. Two physical image-barrier cases +hold node-zero apply in both image forms while another group's preceding complete +checkpoint publishes. Healthy voters apply its successor; node zero retains its +current root, suffix and SQL ownership until release. Both new cases also run +inside the exact resource-bound snapshot case. + +All five exact snapshot cases, 152 execution cases and nineteen storage cases +pass, along with all sixteen commands including the prescribed gate. Its record +under `.cellar/cell-groups-evidence/CG-05-checkpoint-apply-scheduling/qualification/` +binds 335 source hashes and three executables. Both original and diagnostic +failures retain their source/output/executable hashes; independent verification +and `check_evidence` pass. No timeout, assertion or physical fence was weakened. +CG-05 remains IN_PROGRESS pending completion-lane saturation and sustained capture +fairness qualification. S3 archival, public cutover and deployment gates stay open. + +The preceding checkpoint at `5cfa9fb6` adds production idle-node orphan +maintenance and authenticated retry on the same interrupted learner disk. All five +exact snapshot cases, 150 execution cases, nineteen Raft-storage cases and 38 +relevant LTX cases pass. All nineteen commands pass, including the complete CG-05 +prescribed gate and all-target Clippy for the affected crates. Its record under +`.cellar/cell-groups-evidence/CG-05-node-orphan-retry/` binds 335 source hashes, +six executables and frozen inputs; independent hashes and `check_evidence` pass. +CG-05 remains IN_PROGRESS pending broader lifecycle/pressure qualification. + +`ReplicaLog::reclaim_cell_storage` requires joined runtime/SQL ownership and one +exact applier per physical engine group, plus every additional durable retained +root. It verifies current/checkpoint closures before bounded deletion, merges +references across shared image stores, preserves live pins and holds both engine +claims through cancelled work. Missing/duplicate/foreign/oversized/busy inventories +refuse. Managed orchestration must supply the complete inventory and call this +idle-node boundary; it is not a serving-node collector or Raft log truncation API. + +Four new SIGKILL scenarios interrupt prefix/synced dependencies in both forms. +Original voter disks resume, establish fresh quorum and commit a newer cell; +the same learner disk collects unselected debris and receives a fresh TLS seed. +Ordinary routing refuses until the authenticated install receipt exists. The +learner catches a live suffix; another cold reopen verifies all four independent +cell sequences and current/checkpoint closures. Original voter disks remain intact. +Shared image tests preserve both groups and an independent reader, collect one +file per request and join cancellation. Continue capture fairness, remaining +controller crash/completion-budget and retained-root pressure qualification. + +The preceding controller-publication checkpoint at `370dcf34` retains the actual +held publication-sync and cancelled-close cuts in both forms. The healthy two-voter +quorum commits another group's writes while the stalled node retains its lifecycle +slot, captured source, bytes and engine/SQL ownership. Release selects the preceding +checkpoint and joins shutdown. Its source-bound record remains under +`.cellar/cell-groups-evidence/CG-05-controller-checkpoint-shutdown-final/`. + +The earlier exact-acceptance checkpoint at `f076731a` delivers +`native_cell_groups_snapshot` with all five required native oracles. Each passes +individually with `--exact --test-threads=1`, and all ten prescribed task commands +pass. Fifteen total commands include 144 execution regression cases and package +checks. The record under `.cellar/cell-groups-evidence/CG-05-snapshot-acceptance/` +binds 332 source hashes and two executables; both independent hashes and the +plan's full `check_evidence` validator pass. CG-05 remains IN_PROGRESS pending the +controller/lifecycle and orphan-retry audit; the new gate is not a release claim. + +Shared root-level engine helpers now live in `cell_group_engine_fixture.rs`; +the existing execution-only restart method stays in the original target. Exact +cases compose the real fixtures and preserve owned-child names. Writes now commit +in the replacement and unrelated groups while a learner install fence is held; +cold restart explicitly checks empty and populated cell sequences/images. The +composite membership test constructs futures outside its poll frame after phase +probes reproduced a debug stack overflow. The standalone case passed and the +fixed complete composite passes; no stack limits, timeouts or assertions changed. + +The pending-publication cancelled-shutdown and idle-node orphan/retry cuts +now pass above. Continue remaining controller lifecycle/saturation and capture +fairness qualification. Preserve fresh quorum, physical-owner retention and +complete selected-root checks. RustFS/load, independent-domain and public host +cutover qualification remain open. + +The earlier interrupted-transfer checkpoint at `6e56f21f` passes all 144 +shared-execution tests and nineteen native Raft-storage tests, with eight scoped commands. +Its record under `.cellar/cell-groups-evidence/CG-05-seed-transfer-cuts/` binds +330 source hashes and two executables. The seed crash matrix now includes +12 owned SIGKILL scenarios: four new cases hold partial or synced dependency bytes +in both image modes. Cold recovery preserves the empty stage and refuses startup +without a local install receipt. The production TLS source and receiver perform +these interrupted transfers; the parent checks reached cuts and retained extent +lengths before cold reopen. + +Two new revocation tests replace the source process while a physical image write +is held. A refused network waiter does not refund checkpoint/transport admission; +physical release drains it without publishing any seed. Cold recovery again +refuses startup. These revocation cases use an independently authenticated captured +peer. Production runtime code is unchanged. No automatic orphan cleanup/retry, +all-wire-boundary, power-loss, independent-domain or RustFS/load claim is made. +That checkpoint preceded the exact acceptance target delivered above; its +controller/lifecycle and retry-availability limits remain explicit. + +The earlier starting fixture mapping for the exact target is below. The newer +checkpoint above implements that target and records its exact five-case gate. Reuse the +real fixture owners and direct assertions, without launching another test binary +as an oracle or replacing any required case. + +| Required case | Existing behavior to audit and reuse | +| --- | --- | +| `snapshot_contains_every_cell_at_cut` | `checkpoint_fixture::coherent` captures both forms while quorum writes advance. | +| `partial_snapshot_never_serves` | `refuse_checkpoint`, seed receiver faults and seed crash cuts reject partial/corrupt/foreign state. | +| `whole_group_learner_and_joint_change` | `seed_install`, `membership_lifecycle` and `membership_crash` cover learner promotion, replacement, quorum loss and explicit joint restart. | +| `snapshot_empty_cell_and_extent_restart` | `tcp_publication` transfers the complete populated/empty closure and reopens retained checkpoint plus current suffix in both forms. | +| `snapshot_memory_and_pin_bounds` | `transfer_pressure` and `local_shutdown` retain bounded physical work and old reader images through concurrent apply/shutdown. | + +The shared imports, `Voter`, `Cluster` and image/apply helpers are now extracted +without semantic changes into `cell_group_engine_fixture.rs`. Both integration +targets include them at their root, preserving support-module paths and exact +ignored child names used by owned-process tests. + +The earlier pressure checkpoint at `a2036e50` passes all 142 native +shared-execution tests and eight scoped commands. Its record under +`.cellar/cell-groups-evidence/CG-05-checkpoint-pressure/` binds +327 source hashes and one executable. Two new cases transfer 32 cells with +bounded chunks and scratch in both image forms. A held receiver does not prevent +two original voters from advancing other cell versions. Separate item/byte limits +and materialized scratch contention refuse without refunding live owners. Complete +received digests remain at the captured cut and do not publish before installation. + +The old source tree becomes reclaimable after transfer release. A unique old cell +image stays readable through its independent physical pin, then is actually deleted +after that reader releases. Production runtime is unchanged. The pressure fixture +uses direct physical streaming in one process; it is not a measured RSS/TLS/load or +RustFS qualification. The complete execution run also retains the earlier seed and +explicit joint-consensus crash coverage. Next finish missing mid-stream and +controller-lifecycle faults and all five exact CG-05 acceptance cases. + +The earlier joint-restart checkpoint at `5f30cc6b` passes 16 selected +seed/membership tests and nine scoped commands. Its record under +`.cellar/cell-groups-evidence/CG-05-joint-process-restart/` binds 326 source +hashes and one executable. Four owned SIGKILL scenarios cover replicated explicit +joint entry and exit in both image forms. Three retained replica disks reopen +with new process identities; complete roots, membership revisions and incoming/ +outgoing voter and learner sets match before admission. Production TLS/controllers +regain quorum, commit a cell, finish the pending transition, commit another cell +and satisfy linearizable reads for all four cells at unchanged logical sequences. + +Production runtime code is unchanged. Shared crash/catalog fixture helpers keep +the seed regressions passing. These are replicated-boundary cuts with all original +replicas in one killed child; every in-flight/automatic-transition crash boundary, +independent domains, power loss, RustFS/load and zero-quorum remain unqualified. +Next address large-group transfer/pin pressure and consolidate all five exact +CG-05 acceptance cases, retaining the remaining mid-stream and lifecycle faults. + +The earlier seed-process checkpoint at `b997a48b` passes 11 selected seed tests +and eight scoped commands. The record under +`.cellar/cell-groups-evidence/CG-05-seed-process-crash/` binds 324 source hashes +and one executable. Two parent tests perform eight owned SIGKILL runs across both +image forms: durable stage, before engine sync, after sync and after publication. +Production TLS source/receiver transfer real quorum-authorized seeds. Cold reopen +checks the complete selected/retained root and local receipt; a stage refuses +startup and a recovered learner refuses valid writes without quorum. + +The frozen synthetic catalog preserves pre-fault external claims only for the +fixture. It is not S3 durability or a recovery API. All source voters and the +learner share the killed child; independent voter failure domains, power loss, +mid-stream cuts, joint restart, RustFS/load and zero-quorum recovery remain open. +The newer joint-restart checkpoint above covers retained-disk explicit transition +restart/completion. Bounded pressure and the exact CG-05 gate remain open. + +The earlier persistence checkpoint at `3ea90fb5` passes 9 selected seed tests +and eight scoped commands. Its record under +`.cellar/cell-groups-evidence/CG-05-seed-persistence/` binds 323 source hashes +and one executable. In both image forms, queued source-process revocation refuses +before the seed fence; revocation after entering the fence retains physical owners +and reconciles the complete seed. Sync errors before/after the underlying sync +close all engine admission and cold recovery verifies the complete root and local +receipt together. Recovered learners still refuse valid proposals without quorum. +Production receiver/storage code was unchanged. These are native one-process +fixtures, not process-kill, power-loss, managed fleet or RustFS qualification. + +The newer seed-process checkpoint above covers the staged and engine-boundary +process-kill cases. Joint-transition recovery, mid-stream revocation/crash, signed +minority receipts, pressure and the full CG-05 gate remain open. + +The earlier receiver-fault checkpoint at `3db498ea` passes 167 selected Rust +cases across shared execution/catalog/storage and eight gate commands. Its record +under `.cellar/cell-groups-evidence/CG-05-seed-receiver-faults/` binds 322 source +hashes and three executables. Two new cases cover ten malformed reply and eleven +chunk/end scenarios in both image forms, followed by successful same-engine install. +A newer published NodeView revokes an admitted pre-install grant and stale source +hints. Current and retained roots remain unchanged. Both seed and ordinary +checkpoint loops now observe physical verifier failure while network input is +pending; reproduced held-open offset failures no longer retain admission until +timeout. Unchanged legacy snapshot/storage suites were not rerun in this scoped gate. + +The newer persistence checkpoint above adds queued/entered revocation and sync +errors. Transfer-time revocation, process-kill outcomes, joint-transition restart, +signed minority receipts, pressure and all five exact CG-05 cases remain open. + +The earlier seed-fault/reconnect checkpoint at `de552f6d` passes 238 Rust +cases and eight commands. Its record under +`.cellar/cell-groups-evidence/CG-05-seed-fault-reconnect/` binds 321 source hashes +and five copied executables. Both image forms refuse eleven invalid seed requests, +five post-reply source faults and revoked/expired destination sessions without +minting startup permission. After retained cold recovery, fresh runtime/transport/ +controller owners reconnect over TLS with a new process generation, regain quorum +and commit. Isolation cannot advance applied state or satisfy a linearizable read; +after stepdown new proposals refuse. This is one OS process with explicit routes +and clock advancement, not managed fleet or process-kill qualification. + +The newer receiver checkpoint above adds reply/chunk faults and pre-install +process revocation. The remaining queued/entered persistence and full-task gaps +are listed above. Source capture fairness and controller pressure remain open. + +The earlier learner-install checkpoint at `ab2a308a` passes 238 Rust cases, +eight gate commands and 37 validator tests. Its record under +`.cellar/cell-groups-evidence/CG-05-learner-install-final/` binds 320 source hashes and +five copied executables. Authenticated seed exchange, permanent claims, non-serving +staging, atomic local receipt, joined-member bootstrap and same-owner unknown-result +reconciliation are implemented. Both image forms replace a deleted voter disk, +catch up the post-capture suffix, promote/retire through joint consensus, transfer +leadership and commit new state. Cold recovery checks both retained and current +closures. Held-sync receipt refusal and materialized caller cancellation pass. +The tests caught and fixed noncanonical configuration equality after joint leave. + +The newer checkpoint above extends this installation slice with source wire faults, +revoked/expired destination sessions, retained-peer reconnection and minority +commit/read refusal. Preserve the remaining acceptance requirements listed above. + +The earlier seed-source checkpoint at `0c3769ec` passes 236 selected Rust +cases, eight gate commands and 37 validator tests. Its record under +`.cellar/cell-groups-evidence/CG-05-seed-source/` binds 312 source hashes and five +copied executables. `seed_authority` uses a fresh group ReadIndex, including for +empty groups, and binds a complete applied root to a committed joining learner. +Capture requires that exact root; the source then tolerates ordinary writes while +revoking on membership/leadership/shutdown/deadline changes. Read and checkpoint +budgets remain owned through physical work. Both image forms, receiver verification, +stale/invalid targets, bounded admission, held-sink revocation, final-chunk expiry, +isolated ready-leader refusal and independent group progress pass local tests. + +That source-only checkpoint did not authorize destination startup. The newer +learner-install checkpoint above supplies authenticated exchange, permanent claims, +installation, catch-up/promotion and native deleted-disk replacement. Continuous-write +capture fairness and the full five-case CG-05/RustFS/load/crash gates remain open. + +The receiver checkpoint at `cb164e02` passes 226 selected Rust cases, +9 gate commands and 37 validator tests. Its evidence binds 295 source hashes +and 6 copied executables under +`.cellar/cell-groups-evidence/CG-05-checkpoint-receiver/`. Three native cases cover +complete receive/install, both representations and empty cells, held engine sync, +transfer refusal/bounds, later real quorum writes and joined storage reopen. +A missing checkpoint-only page refuses recovery while the current tree remains +complete. A reproduced cached-destination corruption gap is fixed by independent +destination reread/sync. Preserve failed and interrupted diagnostic evidence. + +The latest owner/runtime checkpoint at `096cfb13` passes 146 selected Rust +cases, 8 gate commands and 37 validator tests. Its record under +`.cellar/cell-groups-evidence/CG-05-snapshot-owner/` binds 296 source hashes +and 3 copied executables. Explicit `step_checkpoint` admits a verified +closure, binds the exact RawNode Ready to engine persistence, and advances applied +state and acknowledgement only after the matching completion. Matching retained +logs commit/replay without installing received roots. Both image forms, held +sync, invalid/overflow refusal, subsequent quorum writes and joined runtime close +are covered. The tests construct the envelope from a captured source; they do +not qualify Raft-generated outbound snapshots or snapshot RPC. + +The latest local-publication checkpoint at `742b1e6d` passes 222 selected +Rust cases, 8 gate commands and 37 validator tests. Its record under +`.cellar/cell-groups-evidence/CG-05-local-checkpoint/` binds 298 source hashes +and 5 copied executables. The owner admits an exact captured current cut, +preserves every later log entry and applied/hard state through publication, and +emits `CheckpointPublished` after its persistence ticket completes. Outbound capture +pins the retained checkpoint named by Raft's exact snapshot, after current apply +has advanced. Tests prove actual RawNode-generated catch-up, suffix replay, cold +leader engine/root recovery and re-stream, plus held publication shutdown in both +image formats. Materialized source recovery uses its fixture filename resolver; +extent recovery and destination replay use production adapters. + +The latest transport checkpoint at `ceb7ed31` passes 227 selected Rust cases, +8 gate commands and 37 validator tests. Its record under +`.cellar/cell-groups-evidence/CG-05-snapshot-transport/` binds 306 source hashes +and 5 copied executables. Five TLS cases qualify both physical image forms, +real Raft-generated catch-up and retry after refusal, suffix/cold recovery, +dropped delivery ownership, negative authenticated framing, revocation/replacement +and unrelated-group progress during an admitted receive. Controller registrations +retain exact appliers/routes. Reserved coalesced snapshot feedback survives +ordinary request pressure; transport ownership lasts through feedback admission +or shutdown. The wire acknowledgement remains admission only. See +[progress](progress.md) for exact limits and retained intermediate diagnostics. + +The latest membership checkpoint at `1814b278` passes 233 selected Rust +cases, 8 gate commands and 37 validator tests. Its record under +`.cellar/cell-groups-evidence/CG-05-group-membership/` binds 310 source hashes +and 5 copied executables. Route/revision-bound configuration entries atomically +sync membership/configuration and the unchanged complete cell root before live +Raft changes. Learner bindings, promotion checks, explicit/automatic joint changes, +leader handoff, demotion fencing and retryable discovery lag are implemented. +Two real-TCP controller cases cover both image forms; a third holds configuration +sync through continued operation and cancelled/joined shutdown. Automatic joint +leave now replicates immediately without another write or tick. + +That earlier membership fixture intentionally did not start learner 4. The newer +learner-install fixture above covers that integration. Metadata reopen still cannot +replace complete physical recovery or serving permission. Birth-member bootstrap +retains its original contract; joined bootstrap requires the new local seed receipt. Preserve all five CG-05 cases, outstanding crash/pressure cuts and +shared RustFS/load qualification. See [progress](progress.md) for exact evidence +and the retained intermediate failures. + +The latest controller checkpoint at `c27dfb41` passes 230 selected Rust +cases, 8 gate commands and 37 validator tests. Its record under +`.cellar/cell-groups-evidence/CG-05-controller-checkpoints/` binds 308 source +hashes and 5 copied executables. The controller holds a successor apply ticket, +captures its preceding durable root on a blocking owner, and publishes through +reserved completion capacity. Exact ticket/root validation prevents capture from +racing current apply. The matching persistence event releases successor work; +the runtime admission reply alone cannot. One slot spans this lifecycle. The +transport's shared checkpoint pool and initial endpoint registrations are reused. +Busy admission backs off until a later apply boundary, allowing writes to continue. + +The three new cases prove repeated compaction across two groups in both image +forms and progress/recovery while the checkpoint pool is saturated. The policy +retains the final idle tail. Continuous-saturation fairness, exhausted apply and +completion budgets, held sync/cancelled close, checkpoint failure/quarantine capacity +and retained-root pressure remain qualification gaps. The newer learner-install +checkpoint above covers native fresh-disk replacement; receiver crash/sync-error +cuts remain required. All five declared CG-05 acceptance cases remain required. See +[progress](progress.md) for limits and intermediate failures. + +The source checkpoint at `b15da415` passes 138 Rust cases, eight gate commands +and 37 validator tests. Transfer +`.cellar/cell-groups-evidence/CG-05-checkpoint-source/` with its predecessors; +its record binds 290 source hashes and three copied executables. The four new +cases cover concurrent real quorum writes while streaming a held cut, both image +representations, empty enrollment, obsolete-page retention/release, bounds, +cancelled physical ownership, stale/foreign/unpublished roots, corruption and +closed engines. Those are the preceding embedded source tests; the receiver slice above extends +storage coverage. At that source checkpoint, snapshot RPC, membership recovery +and new process/RustFS/load qualification were still open. + +The following records describe the preceding incremental checkpoints. + +The preceding reviewed source baseline is `69e41e86`, adding bounded explicit local +provisioning from already selected initial-member assignments. The controller owns +birth/bootstrap, blocking root recovery and attachment. Exact directory/group +requests coalesce; waiter admission is independent from commands. Failed recovered +owners remain charged and retry without reopening pinned page stores. Joined close +waits for physical work. Registration refusals preserve healthy groups, while +shared engine uncertainty remains node-wide. Five new native cases cover both +image modes, actual application receipts, ownership/capacity refusals, cancelled +physical close and all-voter joined engine/page-store recovery. A missing selected +page refuses without reset; restored bytes plus fresh quorum recover the receipt. +The record `.cellar/cell-groups-evidence/CG-04-local-provisioning-final/checkpoint.json` +binds 238 Rust cases, twelve commands, 288 source inputs and 4 executables. +Transfer this ignored evidence with its predecessors. This incremental checkpoint +preceded the complete CG-04 gate above. Automatic production dispatch/voter +startup and managed fleet authorization remain in CG-07. Membership recovery, +public cutover, shared RustFS, independent-process/power-loss and load remain +unqualified under their later task gates. + +The preceding reviewed source baseline is `41121a7d`, adding retained group quarantine. +Group-local failure revokes admission and runtime execution while retaining group +capacity, apply tickets and ambiguous producers. Captured leader authority refuses; +late persistence cannot restore it. Shared engine failures and lost shared SQL +worker replies remain node-wide. The original group cause and quarantine count +are observable; recovery requires joined restart. Two native cases cover both +image modes with actual manifest/engine sync faults. The healthy group completes +controller enrollment/command/exact retry and a linearizable SQL read while the +failed producer remains retained. Capacity refusal and joined cleanup are checked. +The record `.cellar/cell-groups-evidence/CG-04-group-quarantine-final/checkpoint.json` +binds 233 Rust cases, twelve commands, 283 source inputs and 4 executables. +Transfer this ignored directory and its predecessors. CG-04 remains IN_PROGRESS. +Next wire bounded request-driven provisioning into registration, then complete +configuration lifecycle and public host/SDK integration under the plan dependencies. +Shared RustFS disaster recovery and load remain unqualified. +The preceding reviewed source baseline is `4326421a`, adding bounded dynamic group +registration and empty-controller startup. Exact engine/root/applier page ownership +and exclusive transport registration precede existing bootstrap revalidation. +The applier is registered before attachment can emit events; active membership +still requires a successful attachment and serving requires current-term quorum. +Pending groups share capacity with active groups, and queued/unconsumed work uses +the existing item/byte budget. Lost waiters cannot abandon attachment; shutdown +joins it before closing physical services. Failed slots release before response. +Four native cases cover both image modes, live TLS progress during a catalog +stall, ownership/capacity refusals, failed registration retry, cancelled close, +and all-voter joined engine/page-store restart into empty controllers with exact +receipt replay. Image-store handles remain live in one process. +The record `.cellar/cell-groups-evidence/CG-04-dynamic-registration/checkpoint.json` +binds 231 Rust cases, twelve commands, 281 source inputs and 4 executables. +Transfer this ignored evidence directory with previous checkpoints. CG-04 remains +IN_PROGRESS. Next isolate group-local failures without closing healthy groups +(shared engine sync failures still quarantine the engine), then wire +request-driven provisioning/lazy birth into registration. Configuration lifecycle, +public SDK cutover and shared RustFS/load remain unfinished. +The preceding reviewed source baseline is `fbc15053`, adding standalone local and +remote receipt inspection with distinct authenticated purpose and explicit command +permission. Registered contract/digest/identity validation precedes immutable SQL +on a fresh catalog-bound quorum image. Exact retained results and ambiguous +absence carry signed cell observations; lookup never invokes callbacks or retires +another producer. Corrupt schema, expiry and conflict refuse. SQL admission stays +bounded; cancelled queued reads retain physical ownership through completion. +Controller read service references are weak, so closed handles cannot retain the +SQL engine claim. Three native cases cover both image modes; remote cases use +four TLS endpoints in one OS process. Three wire cases and one added SQLite case +cover identity, empty results, absence, capacity and schema loss. +The record at `.cellar/cell-groups-evidence/CG-04-receipt-rpc/checkpoint.json` +binds 227 Rust cases, twelve commands, 279 source inputs and 4 executables. +Transfer the ignored evidence directory with previous checkpoints. CG-04 remains +IN_PROGRESS. Next implement bounded dynamic controller registration/lazy birth, +then isolate per-group failures. Public SDK/discovery and configuration lifecycle +remain open; shared-group RustFS recovery and load remain unqualified. +The preceding reviewed source baseline is `0fe665d0`, adding remote enrollment over +the same bounded write exchange and exact controller registration as commands. +Query, Command and Enrollment have distinct authenticated mandatory purposes. +Explicit namespace/process write grants gate initialization; discovery/read grants +and voter membership do not. Captured creation identity, inputs, preview and +original routing budget survive transport. Receiver fresh catalog/leader checks +precede controller ownership, which survives disconnect, revocation and expiry. +Exact committed creation records survive later mutations and duplicate retries. +Four both-image native cases plus three wire cases qualify these boundaries. +The record at `.cellar/cell-groups-evidence/CG-04-enrollment-rpc/checkpoint.json` +binds 216 Rust cases, eleven commands, 276 source inputs and 4 executables. +Transfer this directory with previous evidence. CG-04 remains IN_PROGRESS. Next +implement standalone safe receipt lookup: validate the registered contract and +request identity, inspect the committed cell image without invoking application +code, and keep absence ambiguous unless the existing reconciliation proof excludes +the original proposal. Public SDK/discovery, dynamic roster and per-group failure +isolation follow. Shared RustFS recovery and load remain unqualified. +The preceding reviewed source baseline is `5fd44437`, adding remote commands over +the bounded TLS application RPC lane. Mandatory authenticated command purpose and +independent namespace/process write grants prevent read/voter authority from +becoming write permission. The exact controller owns accepted work through +response loss or revocation, verifies fresh catalog binding and real leader +execution authority, and returns committed receipts with logical cell tokens. +Three both-image native cases cover nonmember ingress, grant/identity/follower +refusals, duplicates/conflicts, held-SQL disconnect/revocation and cleanup. Wire +and transport tests cover bounded independent grants and authenticated payloads. +The record is `.cellar/cell-groups-evidence/CG-04-command-rpc/checkpoint.json`: +209 Rust cases, eleven commands, 273 source inputs and 4 executables. Transfer +this directory with prior evidence. CG-04 remains IN_PROGRESS. Next implement +remote enrollment and standalone safe receipt lookup, then public-host discovery/ +SDK binding, dynamic roster and per-group isolation. Destination caching still +requires replicated transfer seals; shared RustFS/load remains unqualified. +The preceding reviewed source baseline is `06652e36`, adding bounded range hints +shared across query, command and enrollment invocations. Entry/byte reservations, +LRU eviction and verified head invalidation bound retention; a superseded in-flight +lookup cannot refill the cache. Three catalog cases measure real metadata reads; +a native case runs both image modes during a catalog outage, retaining local +quorum authority and fresh remote admission. The record is +`.cellar/cell-groups-evidence/CG-04-dispatch-route-cache-final/checkpoint.json`: 202 Rust +cases, eleven commands, 265 source inputs and 4 executables. Transfer this +directory with prior evidence. CG-04 remains IN_PROGRESS. Next implement explicit +write-authorized command/enrollment RPC and safe receipt lookup, reusing existing +controller ownership. Destination cache removal depends on replicated transfer +seals/admission invalidation; stale hints must not grant serving authority. +Public-host discovery/SDK binding, dynamic roster, per-group failure isolation, +configuration lifecycle and shared RustFS/load qualification remain incomplete. +The preceding reviewed source baseline is `c25f0d97`, adding explicit nonmember +read discovery on the shared query lane. Discovery allocates no local replica or +voting identity; namespace/process grants remain independent. New TLS fixtures +cover both image modes, foreign/missing grants, original-budget leader retries, +held-SQL revocation, stale grant revisions, late replies under replaced discovery, +route/peer bounds and cleanup. The record is +`.cellar/cell-groups-evidence/CG-04-nonmember-queries/checkpoint.json`: 198 Rust cases, +eleven commands, 262 source inputs and 4 executables. Transfer this directory +with prior evidence. CG-04 remains IN_PROGRESS. Next address bounded destination +routing-cache integration, write-authorized mutation/enrollment RPC and safe +receipt lookup. Public-host automatic discovery and SDK binding, dynamic roster, +per-group failure isolation and RustFS/load qualification remain incomplete. +The preceding reviewed source baseline is `da79a696`, adding attached-peer shared +query RPC on the existing TLS lane. Physical group and logical cell identity, +route epoch/birth, actual read authority and signed cell observations remain bound. +Host-owned weak gateway registration checks engine/domain owners; replies retain +admission and cancelled/expired reads retain physical SQL until joined shutdown. +Shared and old-host queries use one read-only grant/revocation guard. The record is +`.cellar/cell-groups-evidence/CG-04-remote-queries-final/checkpoint.json`: 195 Rust cases, +eleven commands, 260 source inputs and 4 executables. Three native cases run +both image modes; three new wire cases and existing transport/host regressions +also pass. Transfer this directory with previous evidence. CG-04 remains +IN_PROGRESS. Nonmember shared discovery, bounded destination route caching and +write-authorized command/enrollment RPC are next integration gaps; the gateway +currently resolves the catalog per RPC. Public SDK cutover, dynamic roster, +per-group failure isolation, configuration/cache lifecycle and RustFS/load remain +unqualified. +The preceding reviewed source baseline is `f77d0d93`, adding registered, +catalog-routed local enrollment to the shared controller. Initializer digests bind +voting schema; old-host validation refuses this unsupported registration. Creation +identity and captured inputs remain owned independently from response lifetime. +Commands and enrollments share admission, response deadlines and physical shutdown. +Exact enrollment records survive subsequent mutations; conflicts refuse without +reinitialization. The partial record is +`.cellar/cell-groups-evidence/CG-04-enrollment-controller-qualified/checkpoint.json`: 90 +selected Rust cases, ten commands, 254 source inputs and four executables. +Eight new native cases cover registration and lifecycle/fault boundaries, with +seven running both image modes. Transfer this directory with prior evidence. +CG-04 remains IN_PROGRESS; public SDK/remote RPC integration, lazy birth/dynamic +roster, per-group failure isolation and configuration/cache lifecycle are pending. +The public host has not cut over; shared-group RustFS/load remains unqualified. +The preceding reviewed source baseline is `8c615ac9`, adding bounded command +ownership to the shared node controller. Response disconnect/deadline cannot drop +entered SQL or ambiguous submission ownership. Only never-attempted preparations +can be sealed; exact receipts or newer applied-term proof resolve submitted work. +Received stage errors release completed ownership, while lost replies require +resolution or joined shutdown. The partial record is +`.cellar/cell-groups-evidence/CG-04-command-owner/checkpoint.json`: 83 selected Rust +cases, nine commands, 248 source inputs and three executables. Seven new +native cases each cover both image modes; three unit cases cover submission seals. +Transfer this directory with earlier evidence. CG-04 remains IN_PROGRESS; routed +enrollment, remote dispatch, SDK integration, dynamic roster and per-group failure +isolation remain gaps. The public host has not cut over; RustFS/load is unqualified. +The preceding reviewed source baseline is `ee684c33`, adding a fixed-roster node +controller that drives existing shared TLS/apply owners without fixture pumping. +Its partial record is `.cellar/cell-groups-evidence/CG-04-event-controller-qualified/checkpoint.json`: +58 shared-execution cases, eight commands, 244 source inputs and +one executable. The preceding `61aa7752` gate passed 289 Rust regressions before +Clippy found a duplicated test fixture import; only that test import changed, and +the affected execution suite was rerun. Transfer both gates with prior evidence. +Six new cases +cover two groups on one SQL worker, both image modes, exact ownership, real +manifest sync failure, cancelled close and final-handle cleanup. Commands still +require caller-retained invocations; post-response resolution is not implemented. +Dynamic roster and per-group failure isolation remain gaps. CG-04 stays IN_PROGRESS. +The preceding reviewed source baseline is `eb99a242`, connecting physical groups +to the existing production TLS transport without constructing the legacy runtime. +Its partial record is `.cellar/cell-groups-evidence/CG-04-physical-transport/checkpoint.json`: +192 native cases, 91 library cases, nine commands, 241 source inputs and +eight selected executables. Transfer that directory with prior evidence. The +three new native cases qualify shared peer pools and routed command settlement +over TLS in both image modes. They still use fixture-driven runtime events, not +a production controller after ingress cancellation. Complete-group snapshots +explicitly refuse. CG-04 remains IN_PROGRESS. +The preceding reviewed source baseline is `1f014f42`, adding retained routed +command phases over the existing catalog, consensus and SQL owners. Its partial +record is `.cellar/cell-groups-evidence/CG-04-routed-commands-qualified/checkpoint.json`: +105 native cases, eight commands, 229 source inputs and four selected executables. +Transfer that directory and the retained failed `CG-04-routed-commands` and +`CG-04-routed-commands-final` gates with prior evidence. Cancellation tests retain the invocation; production controller +ownership after ingress cancellation remains unqualified. The command phases +preserve captured inputs across proven-non-commit retry and never equate proposal +acceptance with durability. S3 minima refuse pending real shared archive proof. +CG-04 remains IN_PROGRESS; follow the next execution priorities below. +The preceding source baseline is `8723db2d`, adding catalog-bound query +invocations over the existing runtime/SQL owners. Its partial record is +`.cellar/cell-groups-evidence/CG-04-routed-queries/checkpoint.json`: 103 native +cases across four affected suites, eight successful commands, 227 source inputs +and four selected executables. Transfer the complete directory with prior evidence. +This qualifies local routed queries, including authenticated cell tokens, +unchanged deadlines/attempt budgets and cancellation ownership. It does not +qualify live movement, command/enrollment routing, remote transports or the public +SDK/controller cutover. CG-04 remains IN_PROGRESS. +The preceding source baseline is `7a79511d`, adding registered queries with exact +cell observations on the existing workers. Its partial checkpoint is +`.cellar/cell-groups-evidence/CG-04-registered-queries/checkpoint.json`: 319 selected +Rust cases, seventeen successful commands, 225 source inputs and sixteen selected +executables. Transfer its entire directory with prior evidence. That checkpoint +preceded the local catalog-routed query integration; CG-04 remains IN_PROGRESS. The +explicit temporary `QueryVersion::GroupCut` and `invoke_group_query` must be +removed with the old host by CG-16. +The preceding baseline is `b28078af`, adding registered commands on shared +SQL workers and logical command/effect identities. Its partial checkpoint is +`.cellar/cell-groups-evidence/CG-04-registered/checkpoint.json`: 295 selected Rust +cases, seventeen successful commands, 221 source inputs and fifteen selected +executables. Transfer its entire directory with prior evidence. It does not +complete CG-04, qualify shared effect scheduling or implement the public cutover. +The preceding `1ec2eeaa` +checkpoint qualifies selected-image recovery and physical apply fault coverage. +Its retained record is `.cellar/cell-groups-evidence/CG-04-apply-recovery/checkpoint.json`: +260 selected Rust cases, fifteen commands, 205 source inputs and twelve executables. +Transfer that entire directory with prior evidence. It qualifies joined cold +restart, not whole-process kill or power loss. The preceding `c9100ec7` checkpoint +qualifies committed preparation on shared workers (`2085cc96`). The preceding `ed84df5d` checkpoint retains exact +cell-version witnesses in authenticated manifests and query/receipt observations. The preceding read +checkpoint at `3da4481e` decouples pin capacity from transport limits and repairs +the fixture macro. The initial read implementation is `88c4717d`. Earlier checkpoints cover prepared submission sealing +and exact rejection cleanup (`0cf89c40`) and bounded proposal acceptance and +enrollment capacity (`446dd905`). Earlier checkpoints cover shared-worker initialization +and term fencing (`6e09bbc7`, formatted at `82ae6877`), receipt lookup (`91c3ecfc`), +applied-barrier reconciliation (`8a0cf640`), the fixed SQL pool (`51dff0e5`), +the SQL producer (`f807b2f7`) and shared-node runtime (`aec4029e`). +See the latest checkpoint in [progress](progress.md) for qualification and limits. +The branch integrates `origin/main` at `8a93ffd7` (documentation consolidation, +PR 161). The design package now lives in `docs/design/multi-raft/cell-groups/`; +use the commands below. Historical evidence keeps its original tested source, +paths and hashes. Recheck origin before publishing later changes. + +The [TiKV investigation](../density-decision.md) pins the supplied checkout at +`b3995dab28308f09cbb7c125bf32f789dbc203fe`. Follow its Region ownership and shared +worker model, adapting storage to independent SQLite/LTX cells. Do not copy KV +transactions, split a cell's SQLite history, or assume TiKV sizing thresholds. + +Read [design](design.md), [protocols](protocols.md), +[implementation order](implementation-plan.md) and +[qualification](qualification.md). `tasks.json` is the source for task ownership, +dependencies, steps, exact tests and acceptance oracles. `profiles.json` freezes +candidate workload and performance gates; those numbers are targets, not results. + +## Current implementation checkpoint + +The CG-04 node runtime is committed in +`crates/cellar/durability/raft/cell_groups/runtime/`. It multiplexes physical owners, +batches engine persistence across groups, retains event admission through apply, +and joins submitted physical work on close. Native cases use independent SQLite +images and actual node workers; see [progress](progress.md) for qualification. +The prior uncommitted runtime draft has been incorporated into this source. + +The per-cell SQL producer is now committed at `f3d48bdc`. It adds +`cell_groups/execution_authority.rs`, `execution/cell.rs`, shared receipt +transaction helpers in `execution/command.rs`, and `tests/support/cell_group_sql.rs`, +with changes to physical image capture, owner/runtime authority and native tests. +It separates a selected cell image/version from group progress and requires exact +receipt settlement in both image modes. The partial gate passes at `f807b2f7`: +133 native tests and eight verification commands. It does not complete CG-04. + +The fixed SQL pool now accepts cell-history jobs on the existing blocking workers. +Admission, pending proposals and retained outputs survive waiter cancellation; +node/group quotas and an exclusive engine lease bound their ownership. Extent +workspaces are created and closed on those workers. The partial gate passes at +`51dff0e5`: 184 selected Rust tests and ten verification commands. It does not +complete CG-04 or qualify the production dispatcher. + +Per-cell reconciliation now runs on the pool's reserved progress path. A verified +current-term applied cell pin can be inspected on a follower. Matching original +receipts resolve from later images; absence requires a strictly newer applied +term before closing the old producer. Foreign engines, wrong proposals, stale +cell pins and an unapplied new-term barrier refuse without releasing ownership. +Missing producers report uncertainty. Production retry dispatch and real +snapshot/compaction integration remain required. + +Receipt lookup now uses `receipts::lookup_identity`, preserving identity, floor, +sequence and allocation checks independently from new-client admission. Command +admission retains both ceilings. Native fixtures now cover saturated identity +tables, replicated expiry floors and conflicting identities in both image modes, +including callback suppression and later successful command admission. + +Initialization now runs through `CellGroupSqlPool::initialize` and +`CellPrivateEnrollment`, using `CellEnrollmentAuthority` from the real leader. +Logical name admission covers both commands and initialization, including competing +incarnations. `CellEnrollmentState` reads verified current-term applied creation +state on leaders or followers; `resolve_enrollment` preserves the original record, +requires a newer applied term for absent/conflicting old producers, and never +reexecutes callbacks. Prepared outputs carry their term; the guarded submission +path checks it on the actual owner before append, including after that node +regains leadership. Cached acceptance grants no new authority. +The four native cases exercise materialized/extents, concurrent creation, other +cell progress, three cells in one group, retries after mutation, cancelled close, +and absent/conflicting/committed enrollment recovery. All 191 selected Rust cases +and all-target Clippy pass at `6e09bbc7`. The retained partial checkpoint is +`.cellar/cell-groups-evidence/CG-04-initialization/checkpoint.json`; it includes the +initial formatting failure. Formatting-only successor verification is recorded in +`.cellar/cell-groups-evidence/CG-04-initialization-format/gate-commands.json`: all +six checks pass, including formatting, layout, dependencies, docs, preflight and +37 validator tests. These records do not mark CG-04 DONE. + +The current admission change counts applied cells and accepted pending creations +against `max_cells_per_group` before append. Pending acceptance is keyed by logical +cell name, binds proposal identity to exact bytes, and returns the original +position on a retry. Conflicting bytes refuse; another proposal for the same +pending name waits. Independent cells remain eligible. Once applied, stale bases +and existing enrollments refuse another append and require receipt/creation +resolution. Acceptance metadata has a separate shared node budget so saturation +cannot prevent Ready or completion work. All 195 selected Rust cases and ten +verification commands pass at `446dd905`; the source-bound checkpoint is +`.cellar/cell-groups-evidence/CG-04-proposal-admission/checkpoint.json`, with 187 +source inputs and eight copied executables. See [progress](progress.md) for the +retained intermediate failures and qualification limits. This does not complete +CG-04. + +Prepared SQL submission now shares an irreversible decision across clones. Only +an actual owner's definitive pre-append capacity refusal seals the preparation; +a generic queue error leaves its outcome unresolved. The exact seal can be +recovered locally after a lost response, and the worker accepts it only for that +same preparation and engine. It closes the private producer before releasing +name admission. A fresh preparation with the same proposal ID cannot be closed by +the old proof. Accepted preparations retain their original position/digest even +after apply; this remains provisional acceptance, never commitment. Prepared +submission also closes byte-only proposal entry points on that owner. + +Next connect catalog routing for SDK creation/commands and the production host +controller to the registered workers. Local catalog-routed query invocations now +consume exact cell-version observations. Use `CellSqlPrepared::submit` and its typed result; +never release a producer from generic overload or same-term absence. Preserve +the original request fingerprint, deadline and attempt budget across redirects. Do not synthesize a legacy `GroupScope` to +satisfy callback/effect identities. The arbitrary-callback `stage` API remains a verification boundary; use +`stage_registered` for compiled commands. Public SDK routing is still pending. Real compaction/production retries +and preview/cache lifecycle remain required. These cases do not complete CG-04. + +The remaining coupling is concrete: `execution::CommandAuthority` binds the +old `GroupLog`/`GroupStatus`; `execution::images::Published` exposes `GroupScope`; +and the old `host_sql::Cell` still keys its execution state by that scope. The +new pool path already keys by cell history and exact image pins; replace the +production dispatcher with that path while keeping group authority separate. +Reuse the existing receipt transaction and callback machinery; do not manufacture +a synthetic cell-scoped group to satisfy the old interfaces. +A later mutation of another cell must not settle the first cell's pending output. + +The registered command boundary now carries `CellHistory` through +`CommandContext`, inbox authentication, effect targets and system-client identity. +`CellGroupSqlPool::stage_registered` validates the operation, full class schema, +normalized input digest and authenticated identity before receipt lookup, then +uses the existing shared worker and exact proposal settlement path. Signed effect +wire and the class manifest use distinct marker-1 domains; there is no old-wire +reader. The temporary single-cell host projects its actual logical binding. +The shared command path derives identity directly from cell execution authority. + +Registered query contexts still expose `CommittedVersion`/`GroupScope`. Replace +that boundary with the selected logical cell version and witness before the public +cutover; never infer a mutation sequence from group progress. The catalog/SDK +adapter must select the authorized class and preserve invocation deadline, +fingerprint and retry budget. Full shared effect scheduling remains CG-08 work +before CG-07. Reuse the existing SQL workers and joined shutdown. +A cancelled caller cannot release physical callback ownership or allow a second +producer for that cell. + +Shared read selection and SQL execution now exist. Use +`CellGroupRuntime::read_authority` with the original absolute domain deadline, +then consume its pin through `CellGroupSqlPool::query`. Strong reads correlate +fresh quorum to that invocation and wait for contiguous apply; Stale/AtLeast +select logical cell versions. The immutable pin, node/group reservations and +worker-owned query survive publication and cancellation. A returned query result +borrows retained bytes and exposes its logical `CellVersion`. + +Registered query integration must still replace +`application/consensus/class.rs::QueryContext`'s `CommittedVersion`/`GroupScope` +identity with the logical cell version. Preserve operation/schema validation, +read-only callback restrictions, version-token authentication and installed +transfer lineage. `CellExecutionAuthority` and `CellReconciliationAuthority` +still grant no strong-read authority; use the new request-owned read path. +Do not synthesize a cell-scoped legacy group or treat the group barrier index +as the target cell's mutation position. SDK token issuance should consume `CellSqlQueryResult::observation()` or +`CellCommittedReceipt::observation()`. These retain the selected version's actual +cell decision witness. Real compaction integration must preserve it; never use +the latest group progress or the retried command's proposal as a substitute. + + +## First executor session + +### Design delivery check, 2026-10-30 + +The requested deliverable is the executable design and implementation plan. +The existing runtime checkpoints remain implementation evidence at their stated +boundaries; this delivery does not complete CG-04 or authorize a release claim. +Fetching origin confirmed that the branch contains `origin/main` at `8a93ffd7`. +The supplied TiKV checkout still matches the investigation pin above. + +Package validation passes: 16 tasks, 95 required cases, 14 invariants and nine +profiles. Repository documentation validation also passes. In the current working +tree, preflight deliberately refuses source drift in +`crates/cellar/durability/raft/host_sql/mod.rs`; consequently 36 of the 37 plan +validator tests pass and the actual-source preflight test errors. Do not refresh +the baseline merely to silence this refusal. + +At that design-delivery check, `cell_groups/apply/` and +`host_sql/cell_groups/apply.rs` were uncommitted drafts. Subsequent execution +completed that adapter at `2085cc96` and repaired the regression fixture at +`c9100ec7`; see the current checkpoint in [progress](progress.md). The untracked +`diagram/` directory remains separate existing work and is excluded from this +implementation checkpoint. + +### Resume implementation + +Inspect `git status` and the existing diff before editing. The physical owner in +`crates/cellar/durability/raft/cell_groups/owner.rs` and its native execution +fixture now use real three-voter Raft with complete group manifests and separate +SQLite images. The initial fixture compilation failures have been repaired. +The owner now runs on the shared node scheduler; public SDK and per-cell SQL +integration remain incomplete. Preserve its persistence, completion and physical +lease boundaries during integration; +do not mistake the low-level owner cases for the full CG-04 gate. + +The SQL checkpoint includes `native_raft_execution` and `native_raft_rooted` +as well as the shared-group execution suite. Both cell receipt cases include +the final callback/owner-close extension. Preserve authority revocation, exact +committed entry matching and duplicate callback suppression during integration. +The pool now provides applied-barrier receipt reconciliation; production retry +dispatch, real compaction integration and preview/cache retention still need the +owning host. Reuse the old executor's current-term +applied-barrier rule when replacing it: a submitted producer cannot be discarded +from a sampled leader status, timeout or missing response. Resolve the original +command against the authoritative cell receipt before admitting another producer. These regressions supplement, and do not replace, the full task +gate printed by `show CG-04`. Registered SDK dispatch remains absent; registered query callbacks now consume +the existing read-authority and immutable-query paths. + +The selected version's complete commit witness now persists in the authenticated +cell leaf and flows through query and receipt observations. The source-bound gate +at `ed84df5d` qualifies exact entry matching, unchanged-cell and older-pin +provenance, restart/image-collection recovery and token round-trips in both image +modes. Actual log-compaction and group-snapshot integration remain unqualified. + +Local query invocations now connect catalog sessions to registered query workers. +Next connect command/enrollment routing and the public host without synthesizing +legacy group identities. Preserve original invocation deadlines, +contracts and retry identities. Query callbacks must receive logical cell versions; +returned tokens must use the selected observation. `CellGroupApplier` now consumes the startup manifest-page owner, and +`CellGroupSqlPool::prepare_apply` prepares committed cell images and complete +manifests on the existing workers. Consume its retained output with +`CellGroupPreparedApply::complete`; that queues owner publication and is not a +commit acknowledgement. The new native fixture uses this adapter on all three +voters. Registered host dispatch must now consume these APIs instead of copying +fixture image preparation. Keep physical apply position separate from provenance +in later group transfers. + +Run from the repository root with Python 3.11+ and Rust 1.98.1: + +```sh +export PATH="$HOME/.cargo/bin:$PATH" +export CARGO_TARGET_DIR="$HOME/Workspace/crabbuild-target/cellar-3617-cell-groups" +python3 docs/design/multi-raft/cell-groups/plan.py validate +python3 docs/design/multi-raft/cell-groups/plan.py preflight +python3 docs/design/multi-raft/cell-groups/plan.py show CG-05 +python3 -m unittest discover -s docs/design/multi-raft/cell-groups -p 'test_*.py' +``` + +Choose a different private target directory for a different checkout. The plan +runner validates and prints commands; it does not execute Rust tests or provision +infrastructure. `ready` now returns CG-05 because CG-01 through CG-04 have passed +their scoped gates. Start CG-05 under its declared requirements. Global preflight +rejects any further unreviewed source drift; record +a reviewed successor source after reconciling each implementation change. + +The four DONE records require their retained evidence directories listed in +[README](README.md). Transfer those artifacts with the source when validating on +another machine. Missing evidence must refuse; do not replace it with a pass flag +or reset completed tasks merely to bypass validation. + +An earlier passing partial CG-04 record is +`.cellar/cell-groups-evidence/CG-04-apply-final/checkpoint.json` at `c9100ec7`: +254 selected Rust cases, fifteen successful commands, 202 source inputs and +12 copied executables. Transfer the entire directory and its failed predecessor +`CG-04-apply` with the prior evidence. It qualifies committed image preparation +on the existing SQL workers and retention through held publication, not direct +image-I/O cancellation/crash cuts or registered host serving. See the priorities +below before continuing CG-04. + +The preceding durable-witness record is +`.cellar/cell-groups-evidence/CG-04-witness/checkpoint.json` at `ed84df5d`: +214 selected Rust cases, thirteen successful commands, 194 source inputs and nine +copied executables. Transfer the full directory with prior evidence. It qualifies +durable selected-version witnesses and read/receipt observations; registered SDK +serving, real compaction, CG-04 and the release gate remain incomplete. + +The preceding coherent-read record is +`.cellar/cell-groups-evidence/CG-04-reads-final/checkpoint.json` at `3da4481e`: +204 selected Rust cases, twelve successful commands, 194 source inputs and eight +copied executables. Transfer the entire directory, including intermediate failures, +with the prior evidence. This qualifies low-level coherent reads and immutable +SQL in both image modes; it does not complete registered SDK query serving, +production compaction, CG-04 or the release gate. + +The preceding submission record is +`.cellar/cell-groups-evidence/CG-04-submission/checkpoint.json` at `0cf89c40`: +197 selected Rust cases, ten successful commands, 189 source inputs and eight +copied executables. Transfer the entire directory, including intermediate failures, +with the prior evidence. Both image modes exercise rejection cleanup for creation +and mutation, exact proof matching, remembered acceptance and delayed retries. +This remains a partial CG-04 checkpoint; it does not qualify production dispatch, +coherent reads, RustFS or the hard cutover. + +The earlier receipt-lookup partial CG-04 record is +`.cellar/cell-groups-evidence/CG-04-receipt-lookup/checkpoint.json` at `91c3ecfc`: +187 selected Rust cases, ten successful commands, 181 source inputs and eight +copied executables. Transfer its entire directory with the earlier checkpoints. +It qualifies saturated-capacity lookup, replicated expiry/conflict and existing +reconciliation paths in both image modes; it does not complete CG-04. + +The prior reconciliation record is +`.cellar/cell-groups-evidence/CG-04-reconciliation/checkpoint.json` at `8a0cf640`: +181 selected Rust cases, nine successful commands, 179 source inputs and eight +copied executables. Transfer its entire directory along with the prior records. +It qualifies the new pool reconciliation boundary in both image modes, not actual +compaction, production retry dispatch or the full task gate. See [progress](progress.md) +for the historical tested boundary and subsequent lookup/refusal qualification. + +The prior partial CG-04 evidence is retained at +`.cellar/cell-groups-evidence/CG-04-sql-pool/checkpoint.json`. Transfer that entire +directory, including copied binaries and intermediate/successful logs, with the +prior SQL-producer, runtime/owner and DONE-task evidence. The pool checkpoint +binds 178 source inputs, eight executables, 184 selected Rust cases and ten +verification commands. It cannot stand in for the complete CG-04 +contract. Three RustFS-specific cases remain explicitly unrun at this checkpoint. + +## Next execution priorities + +Deliver CG tasks in dependency order and keep the status at its actual qualified +boundary. At the end of each turn, record the completed behavior, evidence and +limits, then identify the most critical remaining gap and the concrete next step. +Do not let plan maintenance replace implementation. + +1. Implement CG-06 complete-group checkpoints plus contiguous committed tails. + Replay cell-addressed LTX, enrollments, membership and administrative entries; + validate every cell base chain and dependency before publishing coverage. +2. Keep Quorum progress separate from S3 settlement. Bound archive lag/unarchived + bytes and tie actual local truncation/purge to covering checkpoints, configured + archive policy and all live group/reader/transition pins. Verify unmanaged + fresh-incarnation seeding through the existing fencing authority. +3. Follow CG-08 then CG-07 for effect/cache scheduling and production SDK/fleet + cutover. Preserve catalog/current-term authorization, captured inputs, exact + receipt reconciliation, cancellation ownership and group-local quarantine. +4. Complete CG-09 through CG-12 for irreversible transfer, split/merge, placement + and authority-checked coordinated hibernation/wake. Retain independent cell + histories and versions; never derive them from a physical group log position. +5. Build CG-14's real-infrastructure harness, then qualify managed/in-transition + S3 recovery in CG-13 and the complete crash/load/domain-loss matrix in CG-15. + CG-16 removes the old host, `TransportRuntime::Legacy`, cell-scoped transport, + `QueryVersion::GroupCut` and `RaftClass::invoke_group_query`, and reruns all gates + at one immutable source. These are required parts of the full hard cutover. + +Start from `cell_groups/checkpoint/`, `cell_groups/reclaim.rs`, existing `archive/` +and `disaster.rs`, plus `cache.rs`/`storage.rs` for group log cuts and pins. Read +`plan.py show CG-06` before creating its planned archive module/native target. +Preserve fresh quorum, permanent fencing authority and complete physical recovery; +source metadata and uploaded objects alone cannot establish those proofs. + +The opportunity is to reuse the existing shared consensus, SQL, read, apply and +provisioning paths. Avoid adding per-cell workers or another serving mode while +integrating the remaining protocols. + +## Delivery sequence + +| Milestone | Tasks | Required outcome | +| --- | --- | --- | +| Contracts and storage | CG-01–03, scoped gates already passed | Separate cell/group history, complete durable roots, bounded range routing and startup preparation | +| Shared runtime | CG-04–06 | Real multi-cell consensus and execution, complete-group snapshots, verified S3 archive and safe reclamation | +| Public cutover | CG-08 then CG-07 | Preserve effects and idle caches; route SDK and managed fleet through the shared runtime | +| Adaptive groups | CG-09–12 | Fenced transfers, split/merge, placement and coordinated hibernation/wake | +| Recovery and qualification | CG-14, CG-13, CG-15 | Reproducible load harness, fenced fresh-incarnation recovery, real RustFS and independent-domain tests | +| Final delivery | CG-16 | Remove the superseded one-cell architecture and pass all release gates at one revision | + +Use the generated dependency order for exact scheduling. A hot cell may occupy +one group through the same runtime. Do not introduce a permanent old/new mode or +an old-format migration reader; preserve marker 1 with explicit domain separation +and refuse unsupported persisted layouts. Never delete existing user data as +part of the hard cutover. + +CG-04's completed gate remains in its source-bound evidence. CG-05's complete +brief prints the five required acceptance cases, owning paths and exact commands. +The source, receiver and owner checkpoints above establish partial boundaries; +they cannot replace full snapshot, membership and lost-disk recovery qualification. + +## Release boundary + +Preserve local quorum as the Quorum acknowledgement boundary. S3 protection +requires a verified complete checkpoint and contiguous tail; uploaded log bytes +alone do not authorize reclaiming local recovery dependencies. A failed complete +group restore refuses admission. Disaster recovery requires the chosen fencing +authority API, Docker qualification, permanent old-host restart prevention and +fresh group/cell incarnations, including incomplete topology transitions. + +Measure occupancies 1/8/32/64/128, both image modes, resource use and request +latency under the frozen profiles. Report wake latency by its actual phases and +include timeouts and refusals. No fixed cost saving or immediate hibernation +response is promised before measurement. Local Docker/RustFS results and +independent failure-domain evidence are separate gates. + +Retain source, binary and output hashes, including failed runs. Update task status +only after its full gate passes. CG-16 reruns the complete capability matrix at +one release revision; `plan.py release` must continue to refuse until then. +Keep the requested MR-13/MR-14 progress JSON files excluded from the PR. Reuse the +active PR, report qualification limits and leave merging to a separate action. diff --git a/docs/design/multi-raft/cell-groups/implementation-plan.md b/docs/design/multi-raft/cell-groups/implementation-plan.md new file mode 100644 index 000000000..5e3388235 --- /dev/null +++ b/docs/design/multi-raft/cell-groups/implementation-plan.md @@ -0,0 +1,47 @@ + + +# Implementation order + +Purpose: Dependency order and status of the sixteen cell-group implementation tasks. + +Generated from `tasks.json`. Regenerate with `plan.py index` after status changes. +**115 of 163 named cases** have task-level acceptance evidence. +Accepted cases count source-pinned evidence for DONE tasks. See the +[open case audit](remaining-work.md#open-case-audit) for implemented coverage and missing gates. +The full release requires a combined run on one source revision. +CG-07 gates the default shared-group backend for new SQL namespaces. +CG-15 and CG-16 gate the full release. + +Start with the [quorum SQL delivery priorities](remaining-work.md#first-delivery-of-quorum-sql-through-public-ingress) +for the next release gate and [progress](progress.md) for exact qualification results. + +Use `python3 docs/design/multi-raft/cell-groups/plan.py show CG-01` for a complete +executor brief. Replace the ID to select another task. Commands are displayed, +never executed by the plan runner. Read the [design](design.md), +[protocols](protocols.md) and [qualification](qualification.md) first. + +| Task | Deliverable | Dependencies | Status | Accepted / required cases | +| --- | --- | --- | --- | ---: | +| CG-01 | Separate logical cell history from physical group consensus | None | DONE | 4 / 4 | +| CG-02 | Persist complete group manifests with crash-safe cell image ownership | CG-01 | DONE | 5 / 5 | +| CG-03 | Prepare paged range routing and enrollment admission | CG-01, CG-02 | DONE | 8 / 8 | +| CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | DONE | 83 / 83 | +| CG-05 | Snapshot and rebuild complete groups through membership changes | CG-04 | DONE | 5 / 5 | +| CG-06 | Archive and reclaim the complete group recovery history | CG-04, CG-05 | DONE | 5 / 5 | +| CG-08 | Preserve per-cell effects and bounded idle execution caches | CG-04, CG-05, CG-06 | DONE | 5 / 5 | +| CG-07 | Cut public SDK routing and managed fleet over to shared groups | CG-03, CG-04, CG-05, CG-06, CG-08 | IN_PROGRESS | 0 / 5 | +| CG-09 | Implement irreversible source sealing and destination activation | CG-07, CG-08 | IN_PROGRESS | 0 / 7 | +| CG-10 | Split hot groups and merge compatible adjacent ranges | CG-09 | IN_PROGRESS | 0 / 5 | +| CG-11 | Schedule group placement and adaptive partition size | CG-07, CG-10 | IN_PROGRESS | 0 / 5 | +| CG-12 | Coordinate group hibernation and authority-checked wakeup | CG-08, CG-10, CG-11 | IN_PROGRESS | 0 / 5 | +| CG-14 | Build reproducible shared-group workload and RustFS harnesses | CG-07, CG-08 | IN_PROGRESS | 0 / 5 | +| CG-13 | Recover managed groups and incomplete transitions from S3 | CG-06, CG-09, CG-10, CG-11, CG-14 | IN_PROGRESS | 0 / 6 | +| CG-15 | Qualify crashes, split merge, wake and zero-quorum recovery | CG-10, CG-11, CG-12, CG-13, CG-14 | IN_PROGRESS | 0 / 6 | +| CG-16 | Publish the unified group runtime and qualified release envelope | CG-15 | TODO | 0 / 4 | + +CG-01 through CG-08 establish the shared runtime and public cutover. CG-09 through +CG-13 add movement, adaptive scheduling, hibernation and complete managed recovery. +CG-08 adapts effects before CG-07 public cutover. CG-14 can proceed after CG-07; +CG-13 requires its real-infrastructure harness. +CG-15 and CG-16 qualify the combined artifact. These milestones are not separate +production architectures. No full delivery claim precedes the final gate. diff --git a/docs/design/multi-raft/cell-groups/plan.py b/docs/design/multi-raft/cell-groups/plan.py new file mode 100644 index 000000000..f1807be0a --- /dev/null +++ b/docs/design/multi-raft/cell-groups/plan.py @@ -0,0 +1,576 @@ +#!/usr/bin/env python3 +# Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +"""Validate delivery contracts and print executor briefs; never execute their commands.""" + +import argparse +import hashlib +import json +import math +from pathlib import Path +import re +import shlex +import subprocess +import sys +from urllib.parse import unquote, urlsplit + +PACKAGE = Path(__file__).resolve().parent +ROOT = PACKAGE.parents[3] +STATUSES = {"TODO", "IN_PROGRESS", "BLOCKED", "DONE"} +SHA256 = re.compile(r"[0-9a-f]{64}") +REVISION = re.compile(r"[0-9a-f]{40}") +CASE = re.compile(r"[a-z][a-z0-9_]*") +TASK_IDS = {f"CG-{number:02}" for number in range(1, 17)} +INVARIANTS = {f"I{number:02}" for number in range(1, 15)} +PROFILE_IDS = {"idle_density", "mixed_quorum", "mixed_s3", "hot_cold", "split_merge", + "hibernate_wake", "rustfs_disaster", "independent_domain_loss", "soak_24h"} +# Include examples, vendored crates, build scripts and future inputs by default. +# Plan bookkeeping may advance without changing the qualified executable. +RUNTIME_PATHS = [".", ":(exclude)plans/**", ":(exclude)docs/design/**"] +SAFETY_METRICS = {"lost_acknowledged_effects", "duplicate_mutations", "stale_strong_reads", + "double_writers", "unexpected_errors", "unjoined_owners", + "old_token_acceptances", "old_host_restart_acceptances", + "restored_receipt_callbacks", "missed_due_effects", "writes_without_quorum"} + + +class PlanError(ValueError): + pass + + +def require(condition, message): + if not condition: + raise PlanError(message) + + +def nonempty(value): + return isinstance(value, str) and bool(value.strip()) + + +def read_json(path): + def unique(pairs): + result = {} + for key, value in pairs: + require(key not in result, f"Duplicate JSON key: {key}") + result[key] = value + return result + + def constant(value): + raise PlanError(f"Non-finite JSON value: {value}") + + return json.loads(path.read_text(), object_pairs_hook=unique, parse_constant=constant) + + +def within(root, name): + require(nonempty(name) and not Path(name).is_absolute(), f"Not a relative path: {name}") + path = (root / name).resolve() + require(path.is_relative_to(root.resolve()), f"Path escapes repository: {name}") + return path + + +def digest(path): + with path.open("rb") as stream: + return hashlib.file_digest(stream, "sha256").hexdigest() + + +def artifact(root, item): + require(isinstance(item, dict), "Artifact must be an object") + path = within(root, item.get("path")) + require(path.is_file() and path.stat().st_size > 0, f"Missing/empty artifact: {path}") + require(isinstance(item.get("sha256"), str) and SHA256.fullmatch(item["sha256"]), + f"Invalid artifact digest: {path}") + require(digest(path) == item["sha256"], f"Artifact hash mismatch: {path}") + return path + + +def git_bytes(root, revision, path): + require(isinstance(revision, str) and REVISION.fullmatch(revision), "Invalid source revision") + within(root, path) + result = subprocess.run(["git", "show", f"{revision}:{path}"], cwd=root, + capture_output=True, check=False) + require(result.returncode == 0, f"Source absent at {revision}: {path}") + return result.stdout + + +def ordered(tasks): + by_id = {task["id"]: task for task in tasks} + require(len(by_id) == len(tasks), "Duplicate task ID") + visiting, visited, result = set(), set(), [] + + def visit(tid): + require(tid in by_id, f"Unknown dependency: {tid}") + require(tid not in visiting, f"Dependency cycle: {tid}") + if tid in visited: + return + visiting.add(tid) + for dep in by_id[tid]["depends_on"]: + visit(dep) + visiting.remove(tid) + visited.add(tid) + result.append(tid) + + for tid in by_id: + visit(tid) + return result + + +def cargo(toolchain, *args): + return ["cargo", f"+{toolchain}", *args] + + +def task_commands(task, toolchain): + test = cargo(toolchain, "test", "-p", "cellar", "--test", task["test_target"], "--locked", "--") + commands = [cargo(toolchain, "check", "-p", "cellar", "-p", "cellar-logic", "--all-targets", "--locked"), + test + ["--list"]] + commands.extend(test + [case["name"], "--exact", "--test-threads=1"] for case in task["test_cases"]) + commands.extend([cargo(toolchain, "format", "--check"), cargo(toolchain, "check-layout"), + cargo(toolchain, "clippy", "-p", "cellar", "-p", "cellar-logic", "--all-targets", + "--locked", "--", "-D", "warnings")]) + if task["id"] == "CG-01": + commands.append(cargo(toolchain, "test", "-p", "cellar-logic", "--lib", "--locked")) + if task["id"] == "CG-14": + commands.append(["python3", "-m", "unittest", "discover", "-s", "tools/perf", "-p", "test_cell_groups.py"]) + return commands + + +def release_commands(toolchain): + return [cargo(toolchain, "format", "--check"), cargo(toolchain, "check-layout"), + ["python3", "tools/check_native_dependencies.py"], ["python3", "tools/docs/check.py"], + cargo(toolchain, "check", "--workspace", "--all-targets", "--all-features", "--locked"), + cargo(toolchain, "test", "--workspace", "--all-features", "--lib", "--tests", "--locked", "--", "--test-threads=1"), + cargo(toolchain, "test", "--workspace", "--all-features", "--doc", "--locked"), + cargo(toolchain, "clippy", "--workspace", "--all-targets", "--all-features", "--locked", "--", "-D", "warnings"), + ["python3", "-m", "unittest", "discover", "-s", "tools/perf", "-p", "test_*.py"]] + + +def commands_for(task, manifest): + tasks = manifest["tasks"] if task["id"] == "CG-16" else [task] + commands = [command for item in tasks for command in task_commands(item, manifest["toolchain"])] + if task["id"] == "CG-16": + commands.extend(release_commands(manifest["toolchain"])) + return [list(command) for command in dict.fromkeys(tuple(command) for command in commands)] + + +def validate_contract(manifest, baseline, profiles, root): + require(manifest.get("format") == baseline.get("format") == profiles.get("format") == 1, "Wrong format") + require(REVISION.fullmatch(manifest.get("planned_at", "")), "Missing planned-at revision") + require(REVISION.fullmatch(baseline.get("source_revision", "")), "Missing reviewed source revision") + require(nonempty(baseline.get("review_note")), "Missing baseline review note") + require(manifest.get("toolchain") == "1.98.1", "Review CI/toolchain contract before changing it") + require(set(manifest.get("invariants", [])) == INVARIANTS, "Incomplete invariant inventory") + anchors = baseline.get("anchors", {}) + require(isinstance(anchors, dict) and anchors, "Missing source anchors") + for name, anchor in anchors.items(): + require(within(root, anchor.get("path")).is_file(), f"Missing source anchor: {name}") + require(SHA256.fullmatch(anchor.get("sha256", "")), f"Invalid anchor digest: {name}") + require(type(anchor.get("line")) is int and anchor["line"] > 0 and nonempty(anchor.get("excerpt")), + f"Missing source excerpt: {name}") + tasks = manifest.get("tasks", []) + require(isinstance(tasks, list) and tasks, "Missing tasks") + require({task.get("id") for task in tasks} == TASK_IDS, "Incomplete task inventory") + covered, names, producers = set(), set(), {} + for task in tasks: + tid = task["id"] + require(task.get("status") in STATUSES, f"Unknown task status: {tid}") + for field in ("title", "priority", "effort", "risk", "capability"): + require(nonempty(task.get(field)), f"{tid}: missing {field}") + for field in ("depends_on", "anchors", "scope", "creates", "steps", "test_cases", "invariants", + "evidence", "stop_conditions", "out_of_scope"): + require(isinstance(task.get(field), list), f"{tid}: {field} must be a list") + require(len(set(task["depends_on"])) == len(task["depends_on"]), f"{tid}: duplicate dependency") + require(task["anchors"] and set(task["anchors"]) <= set(anchors), f"{tid}: unknown source anchor") + for path in task["scope"]: + require(within(root, path).exists(), f"{tid}: missing source scope: {path}") + require(task["scope"] and task["stop_conditions"] and task["out_of_scope"], f"{tid}: incomplete scope") + require(CASE.fullmatch(task.get("test_target", "")), f"{tid}: invalid test target") + require(f'crates/cellar/tests/{task["test_target"]}.rs' in task["creates"], f"{tid}: missing target output") + for path in task["creates"]: + within(root, path) + require(path not in producers, f"Duplicate output: {path}") + require(path not in {a["path"] for a in anchors.values()}, f"Output collides with baseline: {path}") + producers[path] = tid + local_names, local_invariants = set(), set() + for case in task["test_cases"]: + require(CASE.fullmatch(case.get("name", "")), f"{tid}: invalid case") + require(case["name"] not in names, f'Duplicate test: {case["name"]}') + require(nonempty(case.get("oracle")), f"{tid}: empty oracle") + require(case.get("invariants") and set(case["invariants"]) <= INVARIANTS, f"{tid}: invalid invariant") + names.add(case["name"]) + local_names.add(case["name"]) + local_invariants.update(case["invariants"]) + require(local_names and local_invariants == set(task["invariants"]), f"{tid}: invariant coverage differs") + step_cases = set() + for step in task["steps"]: + require(nonempty(step.get("action")) and step.get("tests"), f"{tid}: empty step") + require(set(step["tests"]) <= local_names, f"{tid}: unknown step test") + step_cases.update(step["tests"]) + require(step_cases == local_names, f"{tid}: cases missing from steps") + covered.update(local_invariants) + require(covered == INVARIANTS, "Missing invariant coverage") + order = ordered(tasks) + by_id = {task["id"]: task for task in tasks} + for task in tasks: + if task["status"] == "DONE": + require(all(by_id[d]["status"] == "DONE" for d in task["depends_on"]), f'{task["id"]}: DONE before dependency') + require(task["evidence"], f'{task["id"]}: DONE without evidence') + profile_ids = set() + require(profiles.get("profiles"), "Missing qualification profiles") + for profile in profiles["profiles"]: + pid = profile.get("id") + require(isinstance(pid, str) and CASE.fullmatch(pid) and pid not in profile_ids, "Invalid/duplicate profile") + profile_ids.add(pid) + require(profile.get("required") is True, f"{pid}: required gate was disabled") + require(profile.get("topology") in {"local_containers", "independent_hosts"}, f"{pid}: invalid topology") + require(profile.get("checks"), f"{pid}: empty profile checks") + for field in ("cells", "nodes", "voters", "duration_seconds", "repetitions"): + require(type(profile.get(field)) is int and profile[field] > 0, f"{pid}: invalid {field}") + require(profile.get("occupancies") and all(type(v) is int and v > 0 for v in profile["occupancies"]), + f"{pid}: invalid occupancy matrix") + require(set(profile.get("image_modes", [])) == {"materialized", "extents"}, f"{pid}: image mode missing") + require(profile.get("gate_occupancies") and set(profile["gate_occupancies"]) <= set(profile["occupancies"]), + f"{pid}: invalid gated cohorts") + require(type(profile.get("historical_control")) is bool, f"{pid}: missing control contract") + if profile["historical_control"]: + require(REVISION.fullmatch(profile.get("historical_revision", "")), f"{pid}: missing historical pin") + for name, ratio in profile.get("ratios", {}).items(): + require(name in profile["checks"] and nonempty(ratio.get("numerator")) and nonempty(ratio.get("denominator")), + f"{pid}: invalid ratio contract") + require("control_occupancy" not in ratio or ratio["control_occupancy"] in profile["occupancies"], + f"{pid}: missing ratio control") + for name, bounds in profile["checks"].items(): + require(bounds and set(bounds) <= {"eq", "min", "max"}, f"{pid}: invalid metric bound {name}") + require(all(type(v) in (int, float) and math.isfinite(v) for v in bounds.values()), f"{pid}: nonfinite bound") + require(profile_ids == PROFILE_IDS, "Required qualification profile inventory changed") + require(any(p["required"] and p["topology"] == "independent_hosts" for p in profiles["profiles"]), + "Release requires independent-domain qualification") + return {"tasks": len(tasks), "required_cases": len(names), "invariants": len(covered), + "profiles": len(profile_ids), "order": order} + + +def metric(value, name): + require(type(value) in (int, float) and math.isfinite(value) and value >= 0, f"Missing/invalid metric: {name}") + return value + + +def check_profile(report, profile, profile_digest, revision, root, executable): + require(report.get("format") == 1 and report.get("profile_id") == profile["id"], "Wrong profile identity") + require(report.get("source_revision") == revision, "Profile source differs from release source") + require(report.get("profiles_sha256") == profile_digest, "Profile configuration changed") + require(report.get("executable_sha256") == executable and SHA256.fullmatch(executable), "Profile executable differs") + require(nonempty(report.get("environment")), "Missing actual profile environment") + require(report.get("topology") == profile["topology"], "Profile topology differs") + metrics = report.get("metrics", {}) + for name, bounds in profile["checks"].items(): + value = metrics.get(name) + metric(value, name) + for op, bound in bounds.items(): + passed = value == bound if op == "eq" else value >= bound if op == "min" else value <= bound + require(passed, f'{profile["id"]}: {name}={value} violates {op} {bound}') + artifacts = report.get("artifacts", {}) + for role in ("ledger", "measurements", "topology", "fault_schedule", "verification"): + require(role in artifacts, f"Missing profile artifact: {role}") + artifact(root, artifacts[role]) + measured = read_json(artifact(root, artifacts["measurements"])) + require(measured.get("metrics") == metrics, "Summary differs from retained measurements") + expected = {("replacement", o, mode, rep) for o in profile["occupancies"] + for mode in profile["image_modes"] for rep in range(1, profile["repetitions"] + 1)} + if profile["historical_control"]: + expected.update(("historical", 1, mode, rep) for mode in profile["image_modes"] + for rep in range(1, profile["repetitions"] + 1)) + rows = {} + for row in measured.get("cohorts", []): + key = (row.get("variant"), row.get("occupancy"), row.get("image_mode"), row.get("repetition")) + require(key in expected and key not in rows, "Unknown/duplicate measured cohort") + source = revision if key[0] == "replacement" else profile["historical_revision"] + require(row.get("source_revision") == source, "Cohort source mismatch") + require(SHA256.fullmatch(row.get("executable_sha256", "")), "Missing cohort executable") + if key[0] == "replacement": + require(row["executable_sha256"] == executable, "Cohort executable differs") + for name in ("cells", "nodes", "voters", "offered_requests_per_second"): + require(type(row.get(name)) is int and row[name] == profile[name], f"Measured cohort changed {name}") + require(metric(row.get("measured_seconds"), "measured_seconds") >= profile["duration_seconds"], "Cohort too short") + require(row.get("ledger_sha256") == artifacts["ledger"]["sha256"], "Cohort ledger not bound") + require(isinstance(row.get("metrics"), dict), "Missing cohort metrics") + rows[key] = row + require(set(rows) == expected, "Required cohort matrix incomplete") + candidates = [(key, row) for key, row in rows.items() if key[0] == "replacement" and key[1] in profile["gate_occupancies"]] + for key, row in candidates: + values = row["metrics"] + for name, ratio in profile.get("ratios", {}).items(): + control = rows[("replacement", ratio["control_occupancy"], key[2], key[3])] if "control_occupancy" in ratio else row + numerator = metric(values.get(ratio["numerator"]), ratio["numerator"]) + denominator = metric(control["metrics"].get(ratio["denominator"]), ratio["denominator"]) + require(denominator > 0, "Ratio control denominator is zero") + require(math.isclose(metric(values.get(name), name), numerator / denominator, rel_tol=1e-9), "Ratio is not derived from matched control") + for name, bounds in profile["checks"].items(): + selected = list(rows.items()) if name in SAFETY_METRICS else candidates + values = [metric(row["metrics"].get(name), name) for _, row in selected] + for op, bound in bounds.items(): + aggregate = max(values) if op == "max" else min(values) if op == "min" else bound + require(metrics[name] == aggregate, f"Metric does not retain worst cohort: {name}") + if op == "eq": + require(all(value == bound for value in values), f"Cohort equality gate failed: {name}") + topology = read_json(artifact(root, artifacts["topology"])) + require(topology.get("kind") == profile["topology"], "Actual topology mismatch") + nodes = topology.get("nodes", []) + require(len(nodes) == profile["nodes"] and len({n["id"] for n in nodes}) == len(nodes), "Incomplete topology inventory") + if profile["topology"] == "independent_hosts": + require(len({n["physical_host"] for n in nodes}) == len(nodes), "Containers do not prove independent hosts") + require(len({n["failure_domain"] for n in nodes}) >= profile["voters"], "Independent failure domains missing") + verified = read_json(artifact(root, artifacts["verification"])) + require(verified.get("source_revision") == revision and verified.get("complete") is True, "History verification incomplete") + require(verified.get("ledger_sha256") == artifacts["ledger"]["sha256"], "History verification used another ledger") + require(verified.get("measurements_sha256") == artifacts["measurements"]["sha256"], "History verification used another measurement set") + + +def check_evidence(task, record, manifest, baseline, profiles, profile_digest, root): + tid = task["id"] + require(record.get("format") == 1 and record.get("task_id") == tid, "Wrong evidence task identity") + revision = record.get("source_revision", "") + require(REVISION.fullmatch(revision), "Missing evidence source revision") + require(SHA256.fullmatch(record.get("executable_sha256", "")), "Missing evidence executable digest") + require(nonempty(record.get("environment")) and nonempty(record.get("limitations")), "Missing evidence scope") + cohort = manifest["tasks"] if tid == "CG-16" else [task] + required_paths = {baseline["anchors"][a]["path"] for t in cohort for a in t["anchors"]} + required_paths.update(f'crates/cellar/tests/{t["test_target"]}.rs' for t in cohort) + inputs = record.get("source_inputs", []) + require(isinstance(inputs, list) and inputs, "Missing source input inventory") + paths = set() + for item in inputs: + path = item.get("path") + require(path not in paths, "Duplicate source input") + paths.add(path) + content = git_bytes(root, revision, path) + require(hashlib.sha256(content).hexdigest() == item.get("sha256"), f"Source hash mismatch: {path}") + require(required_paths <= paths, f"Evidence omits source inputs: {sorted(required_paths - paths)}") + # Evidence belongs to its tested revision. A later reviewed source baseline + # must not make historical evidence depend on future edits, or stop requiring + # earlier implementation inputs. Compare every record to the frozen plan base. + changed = subprocess.check_output(["git", "diff", "--name-only", "--diff-filter=ACMRT", "-z", + manifest["planned_at"], revision, "--", *RUNTIME_PATHS], cwd=root).decode().split("\0") + require(set(changed) - {""} <= paths, "Evidence omits changed runtime inputs") + outputs = {} + for run in record.get("commands", []): + require(type(run.get("exit_code")) is int and run["exit_code"] == 0, "Failed/unconfirmed evidence command") + argv = run.get("argv") + require(isinstance(argv, list) and all(nonempty(a) for a in argv), "Invalid recorded command") + key = tuple(argv) + require(key not in outputs, "Duplicate command evidence") + path = artifact(root, {"path": run.get("output"), "sha256": run.get("sha256")}) + outputs[key] = path + expected = commands_for(task, manifest) + require(all(tuple(command) in outputs for command in expected), "Missing required command evidence") + for item in cohort: + prefix = cargo(manifest["toolchain"], "test", "-p", "cellar", "--test", item["test_target"], "--locked", "--") + inventory = outputs[tuple(prefix + ["--list"])].read_text() + for case in item["test_cases"]: + name = case["name"] + require(re.search(rf"^{re.escape(name)}: test$", inventory, re.M), f"Case absent from inventory: {name}") + output = outputs[tuple(prefix + [name, "--exact", "--test-threads=1"])].read_text() + require(re.search(rf"^test {re.escape(name)} \.\.\. ok$", output, re.M), f"Case did not pass: {name}") + require("test result: ok. 1 passed; 0 failed; 0 ignored;" in output, f"Not exactly one passing case: {name}") + required_profiles = {p["id"] for p in profiles["profiles"] if p["required"]} if tid in {"CG-15", "CG-16"} else set() + if tid == "CG-13": + required_profiles.add("rustfs_disaster") + seen = set() + by_id = {p["id"]: p for p in profiles["profiles"]} + for item in record.get("profiles", []): + report = read_json(artifact(root, item)) + pid = report.get("profile_id") + require(pid in by_id and pid not in seen, "Unknown/duplicate profile evidence") + qualified = record.get("qualified_executable_sha256", "") + require(SHA256.fullmatch(qualified), "Missing qualified deployment executable digest") + check_profile(report, by_id[pid], profile_digest, revision, root, qualified) + seen.add(pid) + require(required_profiles <= seen, f"Missing required profile evidence: {sorted(required_profiles - seen)}") + return revision + + +def check_links(package, root): + checked = 0 + for source in package.glob("*.md"): + prose = re.sub(r"```.*?```", "", source.read_text(), flags=re.S) + for raw in re.findall(r"!?\[[^\]]*\]\(([^)]+)\)", prose): + url = urlsplit(raw.strip().strip("<>")) + if url.scheme or url.netloc: + continue + target = (source.parent / unquote(url.path)).resolve() if url.path else source + require(target.is_relative_to(root.resolve()) and target.exists(), f"Broken repository link: {source.name}: {raw}") + if url.fragment and target.suffix == ".md": + headings = re.findall(r"^#+\s+(.+)$", target.read_text(), re.M) + ids = {re.sub(r"[^\w\- ]", "", h.lower().replace("`", "")).replace(" ", "-") for h in headings} + require(unquote(url.fragment) in ids, f"Missing anchor: {raw}") + checked += 1 + return checked + + +def index_text(manifest): + tasks = manifest["tasks"] + required = sum(len(task["test_cases"]) for task in tasks) + accepted = sum(len(task["test_cases"]) for task in tasks if task["status"] == "DONE") + lines = ["", "", + "# Implementation order", "", + "Purpose: Dependency order and status of the sixteen cell-group implementation tasks.", "", + "Generated from `tasks.json`. Regenerate with `plan.py index` after status changes.", + f"**{accepted} of {required} named cases** have task-level acceptance evidence.", + "Accepted cases count source-pinned evidence for DONE tasks. See the", + "[open case audit](remaining-work.md#open-case-audit) for implemented coverage and missing gates.", + "The full release requires a combined run on one source revision.", + "CG-07 gates the default shared-group backend for new SQL namespaces.", + "CG-15 and CG-16 gate the full release.", "", + "Start with the [quorum SQL delivery priorities](remaining-work.md#first-delivery-of-quorum-sql-through-public-ingress)", + "for the next release gate and [progress](progress.md) for exact qualification results.", "", + "Use `python3 docs/design/multi-raft/cell-groups/plan.py show CG-01` for a complete", + "executor brief. Replace the ID to select another task. Commands are displayed,", + "never executed by the plan runner. Read the [design](design.md),", + "[protocols](protocols.md) and [qualification](qualification.md) first.", "", + "| Task | Deliverable | Dependencies | Status | Accepted / required cases |", + "| --- | --- | --- | --- | ---: |"] + by_id = {task["id"]: task for task in tasks} + for tid in ordered(tasks): + task = by_id[tid] + cases = len(task["test_cases"]) + accepted = cases if task["status"] == "DONE" else 0 + dependencies = ", ".join(task["depends_on"]) or "None" + lines.append(f'| {tid} | {task["title"]} | {dependencies} | {task["status"]} | {accepted} / {cases} |') + lines.extend(["", "CG-01 through CG-08 establish the shared runtime and public cutover. CG-09 through", + "CG-13 add movement, adaptive scheduling, hibernation and complete managed recovery.", + "CG-08 adapts effects before CG-07 public cutover. CG-14 can proceed after CG-07;", + "CG-13 requires its real-infrastructure harness.", + "CG-15 and CG-16 qualify the combined artifact. These milestones are not separate", + "production architectures. No full delivery claim precedes the final gate.", ""]) + return "\n".join(lines) + + +def brief(task, manifest, baseline, root): + lines = [f'# {task["id"]}: {task["title"]}', "", f'Planned at `{manifest["planned_at"]}`; reviewed source `{baseline["source_revision"]}`.', + f'Status {task["status"]}; priority {task["priority"]}; effort {task["effort"]}; risk {task["risk"]}.', + f'Dependencies: {", ".join(task["depends_on"]) or "none"}.', "", + "Implement TiKV-style groups containing whole independent SQLite cells. One physical group owns", + "consensus, placement, contiguous apply, checkpoint/archive and recovery; each cell retains", + "identity, transactions, receipt dedupe, LTX and query version. Normal transfers preserve cell", + "history; disaster recovery fences old writers and creates fresh incarnations. No cross-cell", + "transactions, old-format migration or quorum bypass is allowed.", "", + "Read this repository's AGENTS.md and docs/design/multi-raft/cell-groups/{design,protocols,qualification}.md.", + "Use Rust 1.98.1, locked Cargo and a private writable workspace-volume CARGO_TARGET_DIR.", + "Keep cellar-logic pure; use execution-domain I/O/timers, typed failures and joined owners.", + "Use leaf.rs with sibling leaf_tests.rs or component/mod.rs. Markers remain 1 with fresh domains.", + "Use native_raft_snapshot.rs/support/raft_cluster.rs real-image fixtures, not stub markers.", + "Do not enable cellar_internal_tests without its external harness. Preserve licenses and secrets.", "", + "## Drift and source context", "", "Run `python3 docs/design/multi-raft/cell-groups/plan.py preflight --task " + task["id"] + "`.", + "Expected exit 0. A changed input requires review and a recorded successor baseline, not blind rehashing.", ""] + for name in task["anchors"]: + anchor = baseline["anchors"][name] + lines.extend([f'`{anchor["path"]}:{anchor["line"]}` at the reviewed revision:', "", "```", anchor["excerpt"], "```", ""]) + lines.extend(["## Scope", "", "Edit only these owner paths/subtrees and the listed new outputs:", ""]) + lines.extend(f'- `{path}`' for path in task["scope"] + task["creates"]) + lines.extend(["", "Changes to callers outside this scope require a recorded scope update and matching regression tests.", + "Keep unrelated work. Use codex/ branches and the active PR; no merge is implied.", "", "Out of scope:", ""]) + lines.extend(f'- {item}' for item in task["out_of_scope"]) + lines.extend(["", "## Steps and verification", ""]) + target = root / f'crates/cellar/tests/{task["test_target"]}.rs' + lines.append("Test target: " + ("present; inspect its real assertions." if target.is_file() else "PLANNED OUTPUT, currently absent. Create it; absence is not a pass.")) + prefix = cargo(manifest["toolchain"], "test", "-p", "cellar", "--test", task["test_target"], "--locked", "--") + for n, step in enumerate(task["steps"], 1): + lines.extend(["", f'### Step {n}', "", step["action"], "", "After implementation, verify these exact cases (each must select and pass one test):", "", "```sh"]) + lines.extend(shlex.join(prefix + [name, "--exact", "--test-threads=1"]) for name in step["tests"]) + lines.append("```") + lines.extend(["", "## Required behavior", ""]) + for case in task["test_cases"]: + lines.append(f'- `{case["name"]}` ({", ".join(case["invariants"])}): {case["oracle"]}') + lines.extend(["", "## Complete command/evidence gate", "", "First check inventory contains every named case; no zero-test success is accepted.", "", "```sh"]) + lines.extend(shlex.join(command) for command in commands_for(task, manifest)) + lines.extend(["```", "", "Every command must exit 0 with retained hashed output at the exact source revision.", + "Record environment, limitations, executable/source hashes and required deployment profiles", + "using qualification.md. Set DONE only after evidence validation and dependency completion.", + "Regenerate implementation-plan.md with plan.py index. A plan-only result is not DONE.", "", "## Stop and reconcile", ""]) + lines.extend(f'- {item}' for item in task["stop_conditions"]) + lines.extend(["", "Review protocol changes at their owning boundary and update source anchors, tests and docs together.", + "Do not weaken a failed safety assertion or discard failed load runs. CG-16 reruns every capability", + "at one release source; earlier isolated passes do not qualify a later combined artifact.", ""]) + return "\n".join(lines) + + +def load(package=PACKAGE, root=ROOT): + manifest, baseline, profiles = (read_json(package / name) for name in ("tasks.json", "baseline.json", "profiles.json")) + stats = validate_contract(manifest, baseline, profiles, root) + return manifest, baseline, profiles, stats + + +def preflight(baseline, root, names=None): + for name, anchor in baseline["anchors"].items(): + if names is not None and name not in names: + continue + content = git_bytes(root, baseline["source_revision"], anchor["path"]) + require(hashlib.sha256(content).hexdigest() == anchor["sha256"], f"Baseline is not pinned to source: {name}") + require(anchor["excerpt"] in content.decode(), f"Excerpt differs at source: {name}") + lines = content.decode().splitlines() + start = anchor["line"] - 1 + excerpt_lines = anchor["excerpt"].split("\n") + require(lines[start:start + len(excerpt_lines)] == excerpt_lines, + f"Excerpt line differs at source: {name}") + require(digest(within(root, anchor["path"])) == anchor["sha256"], f"Source drift requires review: {anchor['path']}") + + +def require_release_inputs(root, revision): + require(REVISION.fullmatch(revision), "Invalid qualified source revision") + compared = subprocess.run(["git", "diff", "--quiet", revision, "HEAD", "--", *RUNTIME_PATHS], + cwd=root, capture_output=True, check=False) + require(compared.returncode == 0, "Current executable inputs differ from qualified source") + changed = subprocess.check_output(["git", "status", "--porcelain", "--untracked-files=all", "--", *RUNTIME_PATHS], + cwd=root, text=True) + require(not changed, "Release executable inputs are dirty") + + +def validate(package=PACKAGE, root=ROOT): + manifest, baseline, profiles, stats = load(package, root) + require((package / "implementation-plan.md").read_text() == index_text(manifest), "Implementation index is stale; regenerate with plan.py index") + stats["local_links"] = check_links(package, root) + profile_digest = digest(package / "profiles.json") + for task in manifest["tasks"]: + for path in task["evidence"]: + check_evidence(task, read_json(within(root, path)), manifest, baseline, profiles, profile_digest, root) + stats["done"] = sum(t["status"] == "DONE" for t in manifest["tasks"]) + return manifest, baseline, profiles, stats + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + sub = parser.add_subparsers(dest="command", required=True) + for name in ("validate", "ready", "index", "release"): + sub.add_parser(name) + sub.add_parser("show").add_argument("task") + sub.add_parser("preflight").add_argument("--task") + args = parser.parse_args() + if args.command == "index": + manifest, _, _, _ = load() + print(index_text(manifest), end="") + return 0 + manifest, baseline, profiles, stats = validate() + by_id = {task["id"]: task for task in manifest["tasks"]} + if args.command == "validate": + print(json.dumps(stats, indent=2)) + elif args.command == "preflight": + require(not args.task or args.task in by_id, "Unknown task") + preflight(baseline, ROOT, by_id[args.task]["anchors"] if args.task else None) + print("Reviewed source hashes and excerpts match. Runtime tasks remain pending.") + elif args.command == "show": + require(args.task in by_id, "Unknown task") + print(brief(by_id[args.task], manifest, baseline, ROOT)) + elif args.command == "ready": + ready = [t["id"] for t in manifest["tasks"] if t["status"] == "TODO" + and all(by_id[d]["status"] == "DONE" for d in t["depends_on"])] + print(json.dumps({"dependency_ready": ready, "done": stats["done"], "note": "Ready means eligible to implement, not implemented."}, indent=2)) + elif args.command == "release": + require(stats["done"] == len(manifest["tasks"]), "Release blocked: implementation tasks/evidence incomplete") + revision = read_json(within(ROOT, by_id["CG-16"]["evidence"][-1]))["source_revision"] + require_release_inputs(ROOT, revision) + print(f"Release evidence verified for {revision}; only the recorded topology/load envelope is qualified.") + return 0 + + +if __name__ == "__main__": + try: + sys.exit(main()) + except (PlanError, OSError, KeyError, TypeError, json.JSONDecodeError) as error: + print(f"ERROR: {error}", file=sys.stderr) + sys.exit(1) diff --git a/docs/design/multi-raft/cell-groups/profiles.json b/docs/design/multi-raft/cell-groups/profiles.json new file mode 100644 index 000000000..1f9aa2e16 --- /dev/null +++ b/docs/design/multi-raft/cell-groups/profiles.json @@ -0,0 +1,630 @@ +{ + "format": 1, + "status": "unqualified_candidate_gates", + "note": "All numbers are proposed acceptance targets, not measurements. Freeze profile digest before each campaign; retain misses and obtain a reviewed revision before changing targets. All expected fault refusals remain in the ledger.", + "profiles": [ + { + "id": "idle_density", + "required": true, + "description": "Matched 4096-cell idle cohorts; compare occupancy 64 with occupancy 1, including control costs and complete physical inventory.", + "topology": "local_containers", + "cells": 4096, + "nodes": 5, + "voters": 3, + "occupancies": [ + 1, + 8, + 32, + 64, + 128 + ], + "duration_seconds": 300, + "offered_requests_per_second": 0, + "repetitions": 3, + "image_modes": [ + "materialized", + "extents" + ], + "checks": { + "lost_acknowledged_effects": { + "max": 0 + }, + "duplicate_mutations": { + "max": 0 + }, + "stale_strong_reads": { + "max": 0 + }, + "double_writers": { + "max": 0 + }, + "unexpected_errors": { + "max": 0 + }, + "peak_rss_limit_ratio": { + "max": 0.85 + }, + "unjoined_owners": { + "max": 0 + }, + "control_bytes_ratio": { + "max": 0.1 + }, + "cpu_core_ratio": { + "max": 0.65 + }, + "baseline_cpu_cores": { + "min": 0.05 + }, + "baseline_control_bytes": { + "min": 1 + }, + "actual_cells": { + "eq": 4096 + }, + "measured_groups_occupancy64": { + "max": 128 + } + }, + "historical_control": true, + "historical_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", + "gate_occupancies": [ + 64 + ], + "ratios": { + "control_bytes_ratio": { + "numerator": "control_bytes_per_second", + "denominator": "control_bytes_per_second", + "control_occupancy": 1 + }, + "cpu_core_ratio": { + "numerator": "cpu_cores", + "denominator": "cpu_cores", + "control_occupancy": 1 + } + } + }, + { + "id": "mixed_quorum", + "required": true, + "description": "Fixed 100 requests/s, 80% writes and 20% strong/AtLeast reads, uniform cells, Quorum acknowledgements.", + "topology": "local_containers", + "cells": 4096, + "nodes": 5, + "voters": 3, + "occupancies": [ + 1, + 8, + 32, + 64, + 128 + ], + "duration_seconds": 600, + "offered_requests_per_second": 100, + "repetitions": 3, + "image_modes": [ + "materialized", + "extents" + ], + "checks": { + "lost_acknowledged_effects": { + "max": 0 + }, + "duplicate_mutations": { + "max": 0 + }, + "stale_strong_reads": { + "max": 0 + }, + "double_writers": { + "max": 0 + }, + "unexpected_errors": { + "max": 0 + }, + "peak_rss_limit_ratio": { + "max": 0.85 + }, + "unjoined_owners": { + "max": 0 + }, + "refused_requests": { + "max": 0 + }, + "ambiguous_requests": { + "max": 0 + }, + "latency_samples": { + "min": 60000 + }, + "p99_ratio_to_occupancy1": { + "max": 1.2 + }, + "max_archive_lag_seconds": { + "max": 60 + }, + "p99_ms": { + "max": 200 + } + }, + "historical_control": true, + "historical_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", + "gate_occupancies": [ + 8, + 32, + 64, + 128 + ], + "ratios": { + "p99_ratio_to_occupancy1": { + "numerator": "p99_ms", + "denominator": "p99_ms", + "control_occupancy": 1 + } + } + }, + { + "id": "mixed_s3", + "required": true, + "description": "Identical trace to mixed_quorum with S3 acknowledgements against real RustFS.", + "topology": "local_containers", + "cells": 4096, + "nodes": 5, + "voters": 3, + "occupancies": [ + 1, + 8, + 32, + 64, + 128 + ], + "duration_seconds": 600, + "offered_requests_per_second": 100, + "repetitions": 3, + "image_modes": [ + "materialized", + "extents" + ], + "checks": { + "lost_acknowledged_effects": { + "max": 0 + }, + "duplicate_mutations": { + "max": 0 + }, + "stale_strong_reads": { + "max": 0 + }, + "double_writers": { + "max": 0 + }, + "unexpected_errors": { + "max": 0 + }, + "peak_rss_limit_ratio": { + "max": 0.85 + }, + "unjoined_owners": { + "max": 0 + }, + "refused_requests": { + "max": 0 + }, + "ambiguous_requests": { + "max": 0 + }, + "latency_samples": { + "min": 60000 + }, + "p99_ratio_to_occupancy1": { + "max": 1.2 + }, + "max_archive_lag_seconds": { + "max": 60 + }, + "p99_ms": { + "max": 1000 + } + }, + "historical_control": true, + "historical_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", + "gate_occupancies": [ + 8, + 32, + 64, + 128 + ], + "ratios": { + "p99_ratio_to_occupancy1": { + "numerator": "p99_ms", + "denominator": "p99_ms", + "control_occupancy": 1 + } + } + }, + { + "id": "hot_cold", + "required": true, + "description": "One cell receives 50% of writes; compare adapted grouping with occupancy-one replacement at identical offered traffic.", + "topology": "local_containers", + "cells": 4096, + "nodes": 5, + "voters": 3, + "occupancies": [ + 1, + 64 + ], + "duration_seconds": 600, + "offered_requests_per_second": 100, + "repetitions": 3, + "image_modes": [ + "materialized", + "extents" + ], + "checks": { + "lost_acknowledged_effects": { + "max": 0 + }, + "duplicate_mutations": { + "max": 0 + }, + "stale_strong_reads": { + "max": 0 + }, + "double_writers": { + "max": 0 + }, + "unexpected_errors": { + "max": 0 + }, + "peak_rss_limit_ratio": { + "max": 0.85 + }, + "unjoined_owners": { + "max": 0 + }, + "cold_p99_ms": { + "max": 250 + }, + "cold_p99_ratio_to_control": { + "max": 1.25 + }, + "refused_requests": { + "max": 0 + }, + "hot_cell_isolated": { + "eq": 1 + } + }, + "historical_control": false, + "historical_revision": null, + "gate_occupancies": [ + 64 + ], + "ratios": { + "cold_p99_ratio_to_control": { + "numerator": "cold_p99_ms", + "denominator": "cold_p99_ms", + "control_occupancy": 1 + } + } + }, + { + "id": "split_merge", + "required": true, + "description": "Foreground commands/strong reads during repeated split, merge, learner movement and controller restarts.", + "topology": "local_containers", + "cells": 4096, + "nodes": 5, + "voters": 3, + "occupancies": [ + 64 + ], + "duration_seconds": 900, + "offered_requests_per_second": 50, + "repetitions": 3, + "image_modes": [ + "materialized", + "extents" + ], + "checks": { + "lost_acknowledged_effects": { + "max": 0 + }, + "duplicate_mutations": { + "max": 0 + }, + "stale_strong_reads": { + "max": 0 + }, + "double_writers": { + "max": 0 + }, + "unexpected_errors": { + "max": 0 + }, + "peak_rss_limit_ratio": { + "max": 0.85 + }, + "unjoined_owners": { + "max": 0 + }, + "completed_transitions": { + "min": 30 + }, + "max_transition_pause_ms": { + "max": 10000 + }, + "peak_transition_slot_ratio": { + "max": 1 + }, + "history_checks": { + "min": 45000 + } + }, + "historical_control": false, + "historical_revision": null, + "gate_occupancies": [ + 64 + ], + "ratios": {} + }, + { + "id": "hibernate_wake", + "required": true, + "description": "Separate healthy retained leader, cold execution cache, parked leader loss, minority wake and alarm-triggered wake cohorts.", + "topology": "local_containers", + "cells": 4096, + "nodes": 5, + "voters": 3, + "occupancies": [ + 64 + ], + "duration_seconds": 600, + "offered_requests_per_second": 10, + "repetitions": 3, + "image_modes": [ + "materialized", + "extents" + ], + "checks": { + "lost_acknowledged_effects": { + "max": 0 + }, + "duplicate_mutations": { + "max": 0 + }, + "stale_strong_reads": { + "max": 0 + }, + "double_writers": { + "max": 0 + }, + "unexpected_errors": { + "max": 0 + }, + "peak_rss_limit_ratio": { + "max": 0.85 + }, + "unjoined_owners": { + "max": 0 + }, + "healthy_wake_samples": { + "min": 1000 + }, + "healthy_cached_wake_p99_ms": { + "max": 500 + }, + "leader_loss_wake_max_ms": { + "max": 10000 + }, + "missed_due_effects": { + "max": 0 + }, + "parked_control_ratio": { + "max": 0.1 + } + }, + "historical_control": false, + "historical_revision": null, + "gate_occupancies": [ + 64 + ], + "ratios": { + "parked_control_ratio": { + "numerator": "parked_control_bytes_per_second", + "denominator": "active_control_bytes_per_second" + } + } + }, + { + "id": "rustfs_disaster", + "required": true, + "description": "Owned all-voter erasure after reclaim, before/after topology publication, managed and unmanaged group restore from real RustFS.", + "topology": "local_containers", + "cells": 256, + "nodes": 5, + "voters": 3, + "occupancies": [ + 64 + ], + "duration_seconds": 900, + "offered_requests_per_second": 20, + "repetitions": 3, + "image_modes": [ + "materialized", + "extents" + ], + "checks": { + "lost_acknowledged_effects": { + "max": 0 + }, + "duplicate_mutations": { + "max": 0 + }, + "stale_strong_reads": { + "max": 0 + }, + "double_writers": { + "max": 0 + }, + "unexpected_errors": { + "max": 0 + }, + "peak_rss_limit_ratio": { + "max": 0.85 + }, + "unjoined_owners": { + "max": 0 + }, + "erased_voter_disks": { + "min": 3 + }, + "restored_cells": { + "eq": 256 + }, + "old_token_acceptances": { + "max": 0 + }, + "old_host_restart_acceptances": { + "max": 0 + }, + "peer_seed_bytes": { + "eq": 0 + }, + "restored_receipt_callbacks": { + "eq": 0 + }, + "scenarios_completed": { + "min": 6 + } + }, + "historical_control": false, + "historical_revision": null, + "gate_occupancies": [ + 64 + ], + "ratios": {} + }, + { + "id": "independent_domain_loss", + "required": true, + "description": "Actual five-node deployment across at least three independent failure domains; isolate one domain, then demonstrate two-domain quorum refusal.", + "topology": "independent_hosts", + "cells": 4096, + "nodes": 5, + "voters": 3, + "occupancies": [ + 64 + ], + "duration_seconds": 1800, + "offered_requests_per_second": 50, + "repetitions": 1, + "image_modes": [ + "materialized", + "extents" + ], + "checks": { + "lost_acknowledged_effects": { + "max": 0 + }, + "duplicate_mutations": { + "max": 0 + }, + "stale_strong_reads": { + "max": 0 + }, + "double_writers": { + "max": 0 + }, + "unexpected_errors": { + "max": 0 + }, + "peak_rss_limit_ratio": { + "max": 0.85 + }, + "unjoined_owners": { + "max": 0 + }, + "independent_failure_domains": { + "min": 3 + }, + "actual_hosts": { + "min": 5 + }, + "successful_single_domain_failover": { + "eq": 1 + }, + "writes_without_quorum": { + "eq": 0 + } + }, + "historical_control": false, + "historical_revision": null, + "gate_occupancies": [ + 64 + ], + "ratios": {} + }, + { + "id": "soak_24h", + "required": true, + "description": "24-hour mixed traffic, effect scheduling, bounded S3 outages, transfers, hibernation and rolling process restarts.", + "topology": "local_containers", + "cells": 4096, + "nodes": 5, + "voters": 3, + "occupancies": [ + 64 + ], + "duration_seconds": 86400, + "offered_requests_per_second": 50, + "repetitions": 1, + "image_modes": [ + "materialized", + "extents" + ], + "checks": { + "lost_acknowledged_effects": { + "max": 0 + }, + "duplicate_mutations": { + "max": 0 + }, + "stale_strong_reads": { + "max": 0 + }, + "double_writers": { + "max": 0 + }, + "unexpected_errors": { + "max": 0 + }, + "peak_rss_limit_ratio": { + "max": 0.85 + }, + "unjoined_owners": { + "max": 0 + }, + "measured_seconds": { + "min": 86400 + }, + "max_archive_lag_after_recovery_seconds": { + "max": 60 + }, + "steady_rss_growth_ratio": { + "max": 1.1 + }, + "completed_history_checks": { + "min": 4320000 + } + }, + "historical_control": false, + "historical_revision": null, + "gate_occupancies": [ + 64 + ], + "ratios": {} + } + ] +} diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md new file mode 100644 index 000000000..2226b1bab --- /dev/null +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -0,0 +1,8518 @@ + + +# Cell-group implementation progress + +Purpose: Source-pinned cell-group implementation checkpoints and qualification limits. + +The executable design and implementation plan is delivered. Remaining +end-to-end implementation and hard cutover work is specified in the +[executor handoff](handoff.md). +CG-01 through CG-06 and CG-08 have passed their scoped native gates. +CG-07 and CG-09 through CG-15 are in progress. CG-16 remains pending +the combined release gate. Follow the [quorum SQL priorities](remaining-work.md#first-delivery-of-quorum-sql-through-public-ingress) +for the next delivery. + +## Named case audit and public read readiness + +The plan validator verified the source-pinned evidence records for CG-01–06 +and CG-08. Those records account for 115 of the 163 required named cases. +All five CG-07 cases are implemented in the planned public target. +CG-09 through CG-16 have component and process coverage at their recorded +revisions. Their planned acceptance targets and task records are outstanding. +The audit identified drift in 14 of 24 source anchors. The successor baseline +at `bcf10c76458e97cc1696bddbebf85d9dfb7d32aa` reviews that drift and pins +current host options, the durable cache and the Fleet transition owner. +All 37 plan tests, including preflight, pass with this baseline. + +The shared read target at `1d571a411f873fde8d3b4c1cdfcabb65259f1970` passed +all 16 cases in 60.59 seconds. With the local consensus channels cut, a voter +served the historical SQLite image and refused recent-source and linearizable +queries. Both policies recovered when the channels healed. A protocol-only +leadership transfer advanced the term and physical apply on all three voters +while the same logical cell witness and version remained readable. The suite +log SHA-256 is +`4868cb212a2a90e14ffb302eee7892c0b61e1648d903fc3352f007717db8fa30`. + +All five named CG-07 cases passed on `b51be13aa9ff6ab429882d5306bbcaf95a324e90` +in the native public target. The additional harness freshness case passed, +and the opt-in RustFS lifecycle case was skipped in that invocation. The +suite took 441.62 seconds. Its command log SHA-256 is +`98ede1eb69f26ee37916bec8ae2a90c9843705114a8f9e4eeab0d1bf4d8be026`. + +The native CG-09 operator split case passed in 15.80 seconds on the same +runtime. It lost the confirmation for the first child reservation, advanced +the Fleet revision independently, reopened the CLI controller and completed +the same plan in key order. It also verified exact cancellation replay. The +case log SHA-256 is +`98bc23f770f5de9c2272e938b95d7721c92cdb5d7a2642a5387c8caf323e200a`. +This is coverage of a split admission cut. The named transition acceptance +target still needs the restart and single-writer oracles across all phases. + +The native CG-10 split-then-merge case passed in 96.74 seconds on the same +runtime. It exercised reopened operator preparation, late-write preservation +and merged SQL. The case log SHA-256 is +`47f6c76aea5e8ee76cc66b3ea857ed5f2e5ca34586dd56c222162c2486f61e35`. + +The host-native public lifecycle at `1d571a411f873fde8d3b4c1cdfcabb65259f1970` +passed the materialized mode against RustFS. The extent mode settled the +replacement and recovered its lost terminal reply. It then hit the deadline +replaying the original command with S3 durability. The leader had applied +through index 9 and retained one archive settlement. This is a distinct +failure from the constrained daemon fixture startup. The RustFS suite record +SHA-256 is `152a321a7a3c90e0327d9bffe7f10e31ce303c3e32add4049a78a96ba10e2beb`. +The diagnostic rerun at `65a01840fc84bda9c89ebfce896ab5e3dfc9f4b1` passed both +image modes in 75.31 seconds. Its record SHA-256 is +`30e706d558904b7c293c43ae7cb4aed812721209eec737643b59c1428f564c21`. +The rerun adds each host's archive position, attempts and typed error to +the retained failure report. It did not reproduce the extent deadline, +which remains open. + +## CG-09/10 operator split through merged SQL + +At `be683a3b394c06e3ffb63aede32acdac74be75d0`, the combined native case +passed in 68.42 seconds. A lost reply leaves the first child reserved. +A fresh CLI controller resumes the same plan after an unrelated Fleet +revision advances. It recovers both reserved ranges, confirms the original +admission receipt, and reopens the exact split with the planned decision. +That split passes archive preparation, source sealing, child activation, +SQL receipt replay and source retirement. The resulting child ranges merge, +and the merged group serves the late writes. The case log SHA-256 is +`10e6b800db50052c47157fea088ef572c53c50903723fd5cc20f32ca21cc0579`. + +The same source passed the separate cancellation replay in 9.15 seconds and +the managed recovery-barrier case in 95.00 seconds. The recovery case starts +from that operator-admitted split, fences the selected old histories, erases +the fixture-owned voter disks and serves restored SQL through a fresh quorum. +It uses the native fixture authority. The RustFS rerun and permanent Docker +launch revocation remain separate qualification gates. The cancellation and +recovery log SHA-256 digests are +`164717789fdb4e43d7e6bad037a3a5220f48af1c26a349c053e05be4e82c9774` +and `ac3add4a43284ad9fff17d501e02d525038a7f0a2e1c614f207ef9bb0ad7113a`. + +## CG-09/10 and CG-13 managed serving checkpoint + +At `6e6be12d403e1d7b3341531a0b664cbdb8299753`, the managed recovery +and split/merge cases both passed against RustFS in materialized and extent +image modes. Recovery used a fixture fencing authority and host-native +voters. After the replacements formed a fresh quorum, the case verified +linearizable SQL reads, archived receipt lookup, old version-token refusal, +a new quorum write and retry, and a new S3 acknowledgement. These results +qualify the serving checkpoint. A later retirement candidate passed the native +fence and Fleet-slot release case. Its extent-backed RustFS run stopped during +replacement quorum setup when the group controller closed. + +## CG-13 cold managed recovery and retirement + +At `13fb77e92e749d4b01a9c2c272ffd1ba601eb7fb`, the managed recovery case +passed natively in 91.67 seconds and against RustFS in 245.01 seconds. The +RustFS case exercised both materialized and extent-backed images. The test +joins every source voter, checks the complete requested physical lineage, +erases the owned voter directories and refuses an old boot on an empty disk. +It then drops the pre-erasure verifier and reconstructs the selected archive +cuts in an empty workspace. The replacement quorum serves restored SQL, +original receipts, new quorum writes and explicit S3 writes. The case +interrupts replacement archive setup at registry and identity creation, then +resumes the same binding. It also verifies the permanent fence again before +retiring old histories and releasing the Fleet disruption slot. The archive +holds and physical inventory debt stay recorded. + +The ordinary managed split/merge regression also passed in both image modes in +195.97 seconds. The complete RustFS suite record SHA-256 is +`18ac3ebfb43f9484bf7eeca601abf0607ea103975be3d3b63b2c144abe1b30fa`. +The native test log SHA-256 is +`bd43761d841e306fc1d9eb66c1cb92677482cf3ae41dbcbdacf825521a8b5f92`. + +The source voters are host-native Nodes and the fence uses a fixture authority. +The managed Docker drill still requires permanent launcher revocation, +independent voter processes and deletion of all their disks. + +## CG-07 RustFS lifecycle follow-up + +The RustFS source-service failover case passed on the executable source of +PR #160 at `6e6be12d`. The joined-voter maintenance case progressed through +stop, return, resume and a successful leadership handoff. The resumed leader +reported ready with its applied and committed indices both at 18. An exact +receipt replay requesting S3 durability expired with one archive settlement +outstanding. The retained journal reached index 18 by shutdown. This narrows +the observed failure to the S3 acknowledgement path after resume. The failed +RustFS lifecycle suite record SHA-256 is +`dcd124b69d21bbc57f98801201b4141931cb674d2e41ad9026dd75c4b423c352`. +The combined public quorum gate remains open. + +## CG-07 constrained Linux SQL qualification + +The daemon qualification at `9d6cf6e9c8dc20583c4177c7896ebf0673977449` +passed against real RustFS in both image modes. Each run used three separate +Linux voter containers with one effective CPU, 3 GiB of memory, zero swap and +separate disks. The public CLI provisioned the managed namespace and first +group. Native and Axum requests covered retained restart, leader SIGKILL with +a successor elected before restart, exact receipt replay and quorum loss. +The archive fault left catalog operations available. Quorum writes and +linearizable reads continued while S3 acknowledgement refused until the +original receipt obtained archive coverage. The same run exercised operator +park and wake through a fresh quorum. + +The materialized and extent-backed runs took 64.18 and 65.84 seconds. Each +mode recorded one SIGKILL exit and three joined normal exits. The suite record +SHA-256 is `839c373ec7e52c4feb914ead588c91af05ca2a6a13765e0d2bb7d7eea101542b`. +This is a single-VM public daemon result. The combined independent-process +repair, drain and maintenance gate and sustained load qualification remain open. + +## CG-07 shared-group directory CI port + +All 16 `native_raft_directory` cases passed on `a69670357de38d4850821f81b26e9feb64110f5e` in +8.84 seconds. The suite uses two real groups, a rooted Fleet observer for +physical metadata, and the same engine across a retained restart. It verifies +that unpolled inspection calls retain their reservation while business SQL +completes, then releases every reservation on joined shutdown. The entered +metadata case pauses the read, observes pending closure, releases the gate and +checks that no read owner remains. The command log SHA-256 is `6cb4e5e275690b5390c8d797f0377e7de1ba5c7a52e1aef0e27c15def1a39c44`. +The `native_raft_reads` and other predecessor consumer targets still require +ports to the canonical SDK. + +## CG-07 retained group images during node collection + +On `6521cf5c`, all three `native_raft_extents` cases passed in 22.61 seconds. +The public host refuses extent activation without storage admission before +catalog mutation. With eight and twelve extent roots, the fixture provisions +two physical groups, enrolls cells and verifies S3 receipts against the +in-memory object store. It checks that the collector reclaims files after a +restart with one group unattached, then reattaches that group and verifies +its saved SQL state and exact receipt without running the callback again. +The run also checks persisted checkpoints, bounded inventory and joined closure. +The command log SHA-256 is +`2ad1a3150d5dfadfcea3ae7b713dd08f23c987e670b60c9aa593308625a9caa8`. +The same runtime passed all 16 `native_raft_directory` cases in 33.80 seconds +(command log SHA-256 `93a8643767ee4139061dfe41704b046e96bbbe1b5a857a0a1c5ffe2d755f5f60`). +This covers local image collection and retained restart. + +## CG-07 shared-group Fleet permit CI port + +On `c0e6c74f`, all eight `native_raft_fleet_permits` cases passed in 119.18 +seconds. They use the public managed node and resource movement APIs with real +TLS peers, retained engines and an in-memory object store. The cases preserve +the exact permit after a lost reply and a controller reopen, fence delayed Begin +requests, and refuse joint consensus progress and linearizable reads when the +old majority is absent. A returning leader can apply its own removal before +replying, so the fixture obtains the exact terminal proof from the next leader. +It verifies peer and S3 learner installs, callback suppression, and retained +capacity debt when a restarted voter cannot attest its former boot. The command +log SHA-256 is `ddfecfc85d3125365d7c80a96246b00af94ff5d1c1065c725060b71b6aa1fc55`. + +At `987a7247`, the same suite passed seven cases and timed out waiting +for an S3 command receipt in +`fleet_permit_legacy_startup_cannot_seed_managed_history`. The isolated retry +passed in 18.48 seconds. The full-suite timeout remains open. The suite and +retry log SHA-256 digests are +`217f84fe6493b4e11e0747616fa42cc5241d5b9a2a0235e82bc9f8833b3bd53b` +and `803b5ad45836b7edfc1931028a51be63ebe2592f2a1bbbbc9094f1f5c6be8c30`. + +## CG-07 constrained Linux failed-voter repair attempt + +The separate repair candidate at `c97a5be76b3a05747a52d80d8442260438c900fd` +booted a managed standby, seeded the archive on real RustFS and recorded +platform isolation of the failed voter. The materialized campaign reached +`Repairing` at Fleet revision 59. The worker recorded a blocked history +with an `Observation` error, zero execution attempts and zero settled +movements. The run timed out after 161.433 seconds. The extent-backed +campaign did not start. The run record SHA-256 is +`147d750bdc98f8cfaea45fe4ccb4065b4666101daee925360c49962f651af812`. +The failing observation still needs to be isolated before rerunning the +fresh-quorum and receipt checks. + +The instrumented candidate `c4ad4796` recorded 137 +`RecoveryUnproven` results at repair selection before the standby became +unreachable near the deadline. The standby advertised 512 MiB for the +replica. The planner also charged 128 MiB of measured-policy overhead, +256 MiB of seed headroom and a 64 MiB reserve, for a minimum of 960 MiB. +The candidate at `d95157a5` declares the complete standby budget and checks +the qualified limit before injecting the voter failure. It still needs the +real RustFS replacement and receipt checks in both image modes. +The instrumented run record SHA-256 is +`e2e20d29a7b961e24d91e9e70fd7ff261875b9570aba3c72c5a3e0b64e008b2f`. + +With that budget, `d95157a5` admitted one movement at Fleet revision 60. +The materialized campaign recorded 108 execution attempts with no settled +movement or verified replacement, and stopped after 154.992 seconds. The +extent-backed campaign did not start. The controller is a Registered boot, +while the movement receiver requires an exact origin whose rooted state +permits execution. Move the test worker to an authorized boot and preserve +the receiving guard. The corrected-budget run record SHA-256 is +`daae13a6d902a335dd29b7ac8857ed57eed297e2b1c9e46a3c56ab5973c6ec0d`. + +## CG-07 automatic recovery origin authority + +On `136474ac`, the native recovery case passed both image modes in 44.75 +seconds. An exact Registered observer fails admission before it acquires +the control service or calls the platform. The fixture verifies that the +catalog revision and the observer's resource charge do not change. An +Active voter then resumes interrupted isolation, replays from the fixture +object store, verifies the replacement quorum and reaches terminal removal. +The worker checks its exact origin boot against each recovery turn's rooted +catalog view. The command log SHA-256 is +`8e221bd5689cc5f8578a98fbaff3e27c2937a99d9dd51335f0ab57a01405d5ad`. + +The separate constrained Linux RustFS candidate at `458d6950` passed +the materialized maintenance stop, return and resume sequence, including +the resumed leader's SQL receipt checks. The parent verified the worker's +exact Active boot and joined the killed voter. The repair stayed Reserved, +with 95 execution attempts and no settled movement or verified replacement. +Worker traces repeatedly reported `Unconfirmed` during the admitted turn. +The materialized campaign stopped after 155.439 seconds. The extent-backed +campaign did not start. Trace leader observation and the Begin action +separately before rerunning the fresh-quorum and receipt checks. The run +record SHA-256 is +`8aa9f58a72869a9a85d234b4962faffc555cc3bd2e5d725831eeaa723e6f39f0`. + +The materialized candidate at `b726c0fc` traced leader observation and +movement actions separately. The authorized worker advanced to `Activate` +after the quorum consumed the repair permit. It recorded 86 unconfirmed +standby activations, 93 execution attempts and zero settled movements. +The standby exited near the deadline with no attached group, and the +campaign stopped after 216.632 seconds. The extent-backed run did not start. +Capture the receiver's typed activation cause before changing seed or +attachment behavior. The run record and stage trace SHA-256 digests are +`a5c0130bacc4447639844bc241708282bb6659555c2e217df95b0d24d5e2e6eb` +and `84bad5dfab98c2bcba045669c1c8414d5f18497a12b6ee60a8cd6a19b6127585`. + +## CG-07 retained S3 seed endpoints + +The constrained materialized RustFS candidate at `85631a1d` reported 83 +standby activation failures caused by an unexpected TLS EOF during the seed +exchange. The returned leader had reopened its retained engine. The run took +198.3 seconds, made 90 execution attempts and settled no movement. The extent +campaign did not start. The run record and standby trace SHA-256 digests are +`a774b66d59490a04721abc78151831550b53690ee1a2b31f5f92c3dcbf009a9d` +and `2c4d2d2d2875d17faa9a3b0a9e4c265773955a5c9a5fa318cde7c2771d922882`. + +Retained startup previously registered seed endpoints only when periodic local +checkpointing was enabled. The native regression reopens and requalifies the +four exact boots with a checkpoint transport pool and no periodic policy. +Before the fix it reserved the repair and then left the learner unconfirmed +for all remaining turns. At `bb63c2fa` the same replacement and joined restart +passed in both image modes, in 44.41 seconds natively and 65.2 seconds against +real RustFS. The independent native managed failure-recovery regression +passed in 44.89 seconds. The RustFS proof uses host-native voters and fixture +capacity admission. The constrained Docker rerun is a separate gate. +The failing baseline, passing native test and RustFS record SHA-256 digests are +`80cd8d7c3982128b1a4b4a917afa31a983fbdb484cb5dac517fc5701707328a2`, +`f2fd1ace93fd7ed68645939be38ca45c66cce375112694a2181d6fd01abc2ac2` +and `588d0f2b5ca6dbce3e580241eaad66bc5ad3e5c63fa32e47016d62d2b4f282d4`. + +The separate peer-seed replacement run at `bb63c2fa` reached joint consensus +and missed settlement within the eight-turn fixture. An isolated retry of the +same binary passed in 54.44 seconds. The failed and passing command log +SHA-256 digests are +`fdfba994b40a40f18f4047cd684c3b795aa26975ae16d02090579316b1116d29` +and `e5a49699531baa155794619d754b95eb822c153bca9ebc0a47869ebbd301428b`. + +The next constrained materialized run on candidate `8a6ab6caaf82` passed +the previous standby seed failure. It promoted the replacement, observed +the stable voter set `[2, 3, 4]` and checked that the removed engine's files +were retained. The fixture then refused startup of the replacement while +inspecting the original birth voters at Fleet revision 60. Movement finished +at revision 63. The fixture exited after 88.545 seconds, before the replacement +served SQL, and the extent campaign did not start. The suite record and +materialized process ledger SHA-256 digests are +`a202df3cc2ac53e243221046085526c25e7db9ad44ec9c5f84d75c22fa5bf9a5` +and `0deca8b3aba071ab2deb379c3d745380249244c8c93e90ba1c2966273314dc57`. + +Candidate `c554b2a5eb187c482cabc7a56e2202d4e2f26d56` restricts the original +birth inspection to original cold voters. The standby still checks the +selected boot, applied manifest, promotion through a fresh quorum and local +SQL readiness before ingress. The candidate passed both constrained RustFS +image modes in 102.951 and 113.427 seconds. Replacement member 4 served +native and Axum SQL as leader, replayed the archived receipt with S3 +durability and accepted fresh quorum writes. The suite record SHA-256 is +`c377e7206ef3f5194ed9e706743afcd15275c7cdf4cf512cc8ec645bcc2e464c`. + +The PR-source harness at `bcf10c76458e97cc1696bddbebf85d9dfb7d32aa` adds the +replacement leader drain, joined shutdown proof and linearizable SQL on the +surviving quorum before inducing quorum loss. The native daemon and targeted +Clippy pass. The native daemon log SHA-256 is +`4cc00ce662f271e88b11a12c06d9107178044f8f2844a1805a68a0a7fb9473a4`. +The constrained run of this revision remains pending. + +## CG-07 shared cell version key rotation + +On `d6f79a1b`, all eight `native_cell_groups_contracts` tests passed. The +rotated signer verifies the original cell version token after a lost catalog +write reply, a lost confirmation read and a reopened namespace. The reopened +catalog refuses to drop the unexpired verifier. The command log SHA-256 is +`58dd3bb6b0ba959bc24b4ed210cc66ab07d53bc45d64d695bf1ea0fdc2a13deb`. + +## CG-07 forwarded ReadIndex over a lost reply + +At `310b0f4a`, all seven ported `native_cell_groups_reads` cases passed in +20.28 seconds on three embedded voters with separate disks and real TLS TCP +links. The directional cut holds both a local follower SQL read and a +membership observation pending. Both finish after healing in the original +term, with the same leader and invocation deadlines. The cases also verify +minority refusals, the physical barrier after a new election, explicit +historical reads, local minimum-version reads, catch-up while a remote read +waits, SQLite authorization and an unpolled query's physical deadline. + +The initial port reproduced a lost ReadIndex response that expired after +healing. The first retry-enabled suite at `9d7ffefc` passed that case and +hit a five-second setup write deadline in the follower minimum-version case. +The passing run verifies committed enrollment on all three voters before the +baseline write. The passing and failed suite log SHA-256 digests are +`f7c981efd4770c7721ab725f935ed6d71e04583476404d9bce2b4ef25b07c1dd` +and `24a70fce419804efeb251b5558698c274fc4635814d8436cfd39f197f2e8bf1f`. +Both existing follower quorum cases and the managed recovery regression in +both image modes passed against the same runtime. Their log SHA-256 digests +are `d27330fb8cb3ed0866106f5aa4f384097d22e660b4245490f52defdce2c555eb` +and `b852971cfc0ea4aeff6d5fcbabdf700c478440091ede6a1ab5d0c048dab90118`. +Targeted Clippy passed with warnings. + +## CG-07 public replica routing and read budgets + +At `310da5ac`, the public read target passed all 14 cases in 63.25 seconds +on three embedded voters with separate disks and TLS TCP links. This adds +bounded replica fallback, delayed-reply isolation, a pinned historical image, +32 quorum writes during an entered follower read, typed peer saturation and +expiry, and byte retention across 16 completed unpolled replies. The first +saturation port observed deadlines because it polled replies sequentially. +The passing run polls all callers within their original deadlines. It also +waits for the former leader to apply the new term and write before testing the +healed replica's linearizable read. + +A voter with the same cell schema and no local query callback can route an +explicit read to a compatible replica. The recipient checks the exact query +contract, and an origin with a locally known contract refuses a mismatched +digest. The baseline reproduced the forwarding refusal. The read and contract +suite log SHA-256 digests are +`1519d95dc20c84db26383c5076792f71d3ab59aeb069c0c7b67c8f73e1760ad4` +and `13cb85042eca2d0243ff396bee77b6d2563ef36709bf32f23578c5f00d7dd54f`. +The contract suite passed all nine cases, including deadline bounds and +tampering at every byte of a shared cell version token. Both existing +controller replica-routing cases passed in 16.28 seconds. + +## CG-09 rooted decisions and retained source sealing + +The unmanaged source seal consumes a permit bound to the selected directory +journal, exact parent route and open replica engine. On `ad3797e1`, the native +three-voter fixture drained the accepted enrollment, rejected delayed work, +committed the seal, published its checkpoint and reopened all three retained +disks. A fresh-term no-op committed while the retained cut still refused writes +and linearizable reads. Replaying the exact permit returned the original cut. +The catalog and protocol targets passed 25 and 7 tests on the same source. + +The first fixture used a direct image helper that rejected the administrative +entry. The replay uses the production applier. A tight manual tick loop then +advanced election time faster than disk sync. The passing replay pumps the +explicit campaign with the original deadline. Managed preparation, source +archive holds, child install and publication remain open. + +## CG-09 independently verified source archive holds + +`5b1d2f6f` replays the exact archived cuts of all parents before selecting the +hold set in the transition journal. The same held set can be reopened after a +lost directory confirmation. An incomplete set, a cut from before sealing, a +released hold or a replacement hold for the same cut is refused. The native +host target passed five cases, with its explicit RustFS case run separately. +That case passed against real RustFS with both materialized and extent-backed +images. The fixture uses the unmanaged verification boundary. Managed +preparation, child installation and publication still need their own proofs. + +## CG-07 retained voter discovery and process quorum loss + +`47ce0d5d` passed the constrained Linux daemon gate against real RustFS in both +image modes. The process ledger records a successful linearizable read, two +joined voter shutdowns, typed write and linearizable-read refusals, and an +historical read on the surviving voter. The same run retains its process-kill +and retained-disk restart evidence. + +`e720f5b4` passed native initial and joined voter restart, managed ingress and +daemon process cases. The opened engine supplies retained local replica +placements within declared namespaces. Attachment checks selected birth, +boot and membership authority. The managed ingress case also passed against +real RustFS in both image modes with host-native voters. + +## CG-07 feature-off native Fleet and bucket + +`feature_off_fleet_contract_unchanged` runs the compiled native counter +through both Fleet and bucket durability modes. It verifies two-process +routing, application reload, graceful ownership handoff and retained +bucket restoration. The ordinary daemon test uses the same fixture in +bucket mode. The cases use a local development object store. + +The separate fourteen-target native regression ran with +`--no-default-features` on `7634ac0f` and recorded 172 passes, +zero failures and 4 explicit skips. It covers Actor, cells, +Node, LTX replication, storage, log, daemon, events, queue, workflow, +WebSockets and containers. The broader release matrix remains separate. + +Regression log SHA256 `036e58d8505e3d3bd45cd1a32e1f56561341ffbb4dce0aac0a7fdcddb086423d`. +Public feature-off log SHA256 `71d0c55bad7ae7e3352e13e12b67b03860e87753269a3a286de0ac93acea7c14`. + +## CG-07 public occupancy and ingress authority gates + +`native_cell_groups_public` exercises one and two enrolled SQL cells on +the same three-voter counter executable, with one observed physical group per +voter. Both occupancies use the same catalog, enrollment, archive snapshot +and retained restart paths. The native and Axum adapters can request verified +S3 coverage by retrying the original quorum command with the same identity, +payload and retry deadline. The single-cell run refuses reads and writes to +the unconfigured second cell. + +`ingress_cannot_cross_cell_authority` invokes the same managed request fixture +from the public target. The managed test inventory retains the original +case names. The authority case includes the real admin, native and Axum paths +in both image modes. The occupancy fixture uses the development object store. +These local runs use one machine and fixture credentials and do not qualify the +constrained ordinary daemon path. + +Isolated occupancy test log SHA256 `d0e6c1dcb5b5705d14e8a534f2f8871974d2fdbe5360f3373b7d93a478d07a8d`. +Standalone authority test log SHA256 `54c3f88fff11c17fb16d6d1355f604887651485cbb19948cc8d2c7f46754fe2b`. + +## CG-06 Docker replay with a retained qualification network + +Both materialized and extent recovery runs pass on +`5a97e85e6a6d75de247ab3a0b601e90d84d14bb1`. The dedicated Linux Docker profile +runs real RustFS and three constrained voter containers. The sole-launcher +controller permanently revokes the old generation and deletes its three +owned disks. Every replacement verifies restored images and exact receipts, +then forms a fresh quorum, accepts new writes and joins retained restart. +The fixture qualifies one group and three independent SQL cells in one local VM. + +The original retry on `725214bd` stops before starting voters or RustFS because +Docker cannot allocate another default network. The successor accepts an +explicit unused private subnet, retains the earlier networks, and records each +actual subnet and network ID. Both runs join all containers and stop the VM. + +Materialized and extent report SHA256 values are `a4847c10c34187ef0649d22f14fc767427082c1d69f70df2d2100b928141008f` and `a5e8e1fc84bef07b705381e089b36f71a5d9bfc921c4407ce21eb6ad7eaceb1b`. + +## CG-07 managed ingress authority + +The managed operator rejects a foreign fleet or namespace and refuses +confirmation of a different physical group. The original reservation replays +and the fleet revision stays fixed until valid confirmation. Native and Axum +requests through the zero-replica router refuse an unlisted cell and a canonical +cell ID paired with a foreign cluster or namespace. A token from another cell +cannot establish a read floor. A subsequent quorum read and the original +command replay confirm that the selected history and value are intact. + +Both native cases pass on the cold-router base with the authority patch. The +owned RustFS case passes in both image modes on candidate +`3e44504c2e803f4adfe19313303968e620fe2d47`. +The fixture uses host-native voters, explicit capacity attestations and a +development peer grant setup. The standalone public-target gate and the +constrained ordinary daemon request path remain open. RustFS and its dedicated +VM stop with data retained. + +RustFS record SHA256 `4f8d822d9ed31315a2c404c17b0fd0ca64d58f5ecd3bf2c514cd0ae3cec40691`. +Operator CLI SHA256 `89b44643dbdd2d2364c167d344b371538ea6135f9084d1b02780e253bd478a8e`. + +## CG-07 cold router return after birth-voter replacement + +A retained voter whose durable configuration excludes it refuses quorum reads +without waiting out the read deadline. The earlier single-replacement fixture +reproduced on both the published and candidate binaries. Instrumentation showed +that settlement observed the exact retired history with one outstanding work +reservation. Prompt read refusal preserves the original physical-debt check, +five-second calls and eight reconciliation turns. + +The cold router returns from its retained disk after all three birth voters +have been replaced and their processes have joined shutdown. It installs routes +from the surviving membership, reads from a joined voter and replays the +original receipt before committing a new quorum write. The router retains +zero local replicas and the stopped voter disks and capacity records remain. + +Three focused native cases and two real RustFS cases pass. Both router cases +exercise materialized and extent images. The entire managed native target +passes with 26 passed and 10 explicit skips. These are +embedded same-host voters and fixture capacity. The earlier broad campaign +retains its original failure record and source scope. The RustFS backend and +VM stop with data retained. + +Source manifest `6e912fc3b1ec80705ec0daf1800a9808b2fcaeefb843f591dc70d0a948f396dc`. +Focused native record `ae1960871364f6c176e18f0c3acb50545bd720f781cd24117508689b73469200`. +Full managed native record `03412bfc51fb49ff1c21754cede3e6728030b2ac48e09e28c7295a4fe565e845`. +RustFS record `48ab59d5a250cebe899b4d01b6b3569ff548dc6308dfdf49acbfd3da137b9ed8`. + +## CG-07 catalog-driven startup and public request qualification + +Managed startup waits on the exact rooted archive and registry under its +original deadline. A missing registry never creates counters or attaches a +voter. An election supersedes the pending ReadIndex and the same startup +deadline governs the next fresh quorum observation. The process test uses the +catalog for archive readiness and retains fixture boot ordering and attestation. + +Four native cases, both independent daemon RustFS image modes and the combined +native/Axum RustFS case pass. The request case starts in an isolated fresh +prefix, uses the public first-group and archive commands, attaches the voters +through the SDK readiness owner and then enrolls two independent cells. It +verifies remote native and Axum reads and writes, preserved receipts, retained +restart and quorum-loss refusals in both image forms. These are same-host +process and embedded cases. The RustFS backend and VM stop with data retained. + +Source manifest `9352d069e187b5c8d9e039be312609c25c5f2774caa9541c9767c56308f18105`. +Native record `1cb5c6c9fb4398f18a5e70216c23e19ec81a82fcbed3c6e25216539964a65759`. +RustFS record `339ec79daa9d92896a9a12a1bcf6178bb54308553db4461e13dfedbf7cd695e3`. +CLI binary `527fa3d65b291dc8465687f811a4b5178c2ad6713beb39f878a721e1b5fd5e9b`. + +## CG-07 daemon-owned first-group readiness + +The daemon startup callback uses the installed managed boot, directory and +archive pool. It verifies the rooted first-group archive, attaches or validates +the exact local initial voter and establishes fresh quorum under the original +startup deadline. The post-barrier check binds the membership to the current +fleet history, local boot and engine. Ingress binding follows the barrier. +The shared read gate waits for a durably applied entry in the current term +before asking for ReadIndex. The prior standalone case timed out when its +first request was issued before Raft accepted current-term reads. + +Four native cases and both real RustFS image modes pass. The native operator +case verifies that a retained attachment refuses readiness after loss of its +majority. The RustFS cases verify cold startup, leader loss and retained +restart with the daemon-owned barrier. The fixture supplies capacity attestation +and startup rendezvous. Voters share a host and use development peer +authentication. The backend and VM stop with data retained. + +Source manifest `c05e1ebae002008ebd1e84f1ebed98e4a594fe0d0790f493d403ef55c98f8cb6`. +Native record `9adde470d0947965c1b80d6d2642cdfea344fb9aa1553a91447028040cc55095`. +RustFS record `f786339f658c83094826062e2fc8df381e2e145745e78367c566729f8846c91f`. +CLI binary `0ebce06b8f6bffad7035799949a058b126a6474fff817b9acd50a4014eaa97bf`. + +## CG-07 rooted first-group archive initialization + +The authenticated archive command verifies the original first-group plan and +requires its live selected history. It journals the exact namespace binding +in the fleet root before entering the one-use initializer. An exact retry +verifies the existing registry without creating counters. The native operator +case checks permission, early refusal, conflicting request and replay after +deleting the isolated fixture registry. The retry leaves that registry absent. + +Four native cases and both real RustFS image modes pass. The daemon fixture +uses the CLI and admin API for capacity, group publication and archive setup. +Voter startup and retained restart verify the exact rooted archive intent. +The fixture supplies capacity attestation and physical voter activation. +Host-native voters share a machine and use development peer authentication. +The RustFS backend and VM stop with data retained. + +Source manifest `88725305464f1766d8eeb4ad52f8c66fa094fb20592fdac0a5577c8619e4eee5`. +Native record `16e6d5d47603355a76248d13dc03b5dbacaab8fd91e51fc52b4ff84b1c57d3d8`. +RustFS record `829fad729d5ea63ad54207daff4eda4cb98353eafdf9f7d390839c532b6157cc`. +CLI binary `7423d3d18c45402114635a9b2fbfdd440f26ebf017d160a7ef988f3be2936f6f`. + +## CG-07 public first-group operator workflow + +The fleet reservation now binds the exact first-range publication digest in +the same CAS as its admitted birth. Read-only reopen verifies the original +identity, payload, committed projection and directory operation. A changed +publication ID conflicts without advancing either root. + +The public CLI and authenticated admin adapter expose standby, reserve and +publish. The process fixture also uses the existing CLI capacity commands. +The native case rejects missing grants, a mismatched namespace, publication +before reservation and a changed plan. It retries the original confirmation +after a lost reply and activates the group for two independent cells. + +Four native cases and both independent-process RustFS image modes pass. The +process cases use the actual CLI and admin API for capacity and first-group +metadata. The fixture still supplies the deployment attestation, archive +intent and daemon activation. RustFS hosts the Raft catalog, fleet coordination +and archives. Peer authentication uses the development store. Voters are +host-native processes on one machine. The backend and VM stop with data +retained. Archive initialization and voter activation still need a public +operator workflow. + +Source manifest `bf62b059a7d2e172ec81e0549d41a22e185f99f97955bddd02f3957d00382d87`. +Native record `f3b2cde1c6d4a2d498996c9ae5ebd4cbd3bcd1179d477e9891afe25cd6ed6a95`. +RustFS record `be2ccf3175600c6c99f772af9f44aeac69365d850946f35406dc852deb523465`. +CLI binary `33420164f6daa4681ceab857f2da2481620fd93000f0baa4dfc111d42b73d7aa`. + +## CG-07 independent managed daemon voters on RustFS + +The native daemon process fixture accepts an explicit loopback RustFS endpoint, +bucket and isolated prefix. The same binary passes the development-store +baseline and both RustFS image modes. The RustFS provider hosts the Raft +catalog, fleet coordination and archive records. Daemon peer authentication +uses the fixture's development store. Voters are separate host-native +processes with fixture capacity on one machine. + +Both runs start from declared standbys, verify cold RF3 activation before +ingress, and exercise independent reads and writes for two cells in the same +group. Their separate recovery ledgers verify exact retries after a leader +hard kill, retained-disk resume and fresh writes for both cells. The dedicated +backend and VM stop with data retained. Public operator orchestration, +independent constrained voter hosts and sustained-load qualification remain +open. + +Source manifest `ef2417a12645458942d9b63bfd53e0ccc52845291e581979889d6fc1f6f20b53`. +Native record `8ecfdf060e7f7b4e03033f673a8a1f06f80b739866638253723a7e253f7aedbc`. +RustFS record `f777af781135ab70da22c164fa3791c00f07f0edf44992fc8c882d359ad53556`. + +## CG-07 managed daemon startup and request path + +The native daemon now installs its shutdown channel before the Raft host starts. +The stop reporter and HTTP driver share that channel. The asynchronous startup +callback runs after transport installation and before ingress. Cancellation +during that callback closes the installed owner and releases the engine lock. + +The process case preserves the original refusal and shutdown checks and now +uses one physical group for two independent SQLite cells. Three daemon +processes start as standbys. The fixture admits capacity using authenticated +observations, confirms the first shared group, persists and verifies the +archive intent, attaches each voter and requires fresh quorum before ingress. +The test performs real TLS replication and follower reads, hard-kills the +leader, verifies exact command retries and resumes the same voter disk. + +The first shared-process candidate stalled at enrollment. The first managed +candidate entered the development catalog outside its execution domain. The +second revealed that the managed stop reporter could start before a shutdown +sender existed. The corrected process case and its ledger rerun pass. Three +embedded regressions pass, including non-Raft daemon routing and handoff. +The process case uses the development SQLite object store and fixture capacity. +The earlier RustFS component evidence remains separate. A generic operator +CLI, RustFS with independent voter processes and the broader lifecycle and +release gates remain open. + +Source manifest `a28c5abbe7fdb6edc0a6ff858bb8a0538b6e403e6fd4a966a1a9d6bd70df3b0c`. +Process record `357d57f0b5f24c784979cbb9d02ef6ed8cd804252c1d823de92e56c101808f20`. +Recovery ledger record `f5c4927962d15d4f1b0d21ee147c6dcb5e17bda9f20f89039ffe0079c2702d48`. +Embedded regression record `6f088fbe58e22c580fefc12f227391638288c1be46a84305922dd4b5d2cd3896`. + +## CG-07 managed group archive provisioning + +The SDK now opens archive provisioning for an exact declared managed namespace. +The prepare and verification operations require a live fleet history and its +selected physical birth. The operator persists and syncs the one-use intent +before committing. Verification checks the exact saved binding, identity and +counters without creating metadata. Incomplete initialization still refuses +admission and requires archive authority recovery. + +The managed fixture initializes through the SDK, then reopens the provisioning +handle and verifies the persisted intent. The native cases cover missing +archive refusal before activation, rejection of a different intent, archive +shutdown ownership and the two-cell native/Axum request path. The same request +case passes against real RustFS in both image modes with host-native voters +and fixture capacity. Ordinary CLI and daemon startup still need to orchestrate +these operations and wait for actual voter activation and fresh quorum. + +Source manifest `32d407be21812e173179dce6541b33266792937133ee059110596017dc9ef26a`. +Native record `14462fe3d49d98c6d9500e71d5ad2516f2752b863defdb74fea4d05c08f3bbb9`. +RustFS record `5644d5020538d0e733fd64e0a38808737bd381bc075cedd433fbfa01823d0c4d`. + +## CG-07 managed first-group provisioning + +The public SDK observes the exact managed standby and reserves one full-range +RF3 group through existing capacity and separation checks. The prepared +reservation retains the original standby digest and range operation. It can +be reopened from the original request after selection or a lost response. +Publication resolves the immutable birth decision before the directory CAS. +A cancellation cannot displace that Live decision, and an unselected birth +cannot admit a voter. Confirmation and physical activation remain separate. + +The cold-start case pauses before range publication and verifies that a +concurrent cancellation sees the same Live birth. A conflicting confirmation +leaves the selected range without Live fleet authorization. The test then +loses an actual confirmation reply, reopens the original reservation and +replays confirmation. It also verifies that a missing published birth refuses +without recreating metadata. The pre-capacity refusal, inventory debt, +physical activation and retained restart assertions remain in place. + +Three native cases and the native/Axum request case against real RustFS pass. +The request fixture uses the SDK first-group path from a fresh standby. Both +image forms run in the request and cold-capacity cases. RustFS uses host-native +voters with fixture capacity, and the backend and VM stop with data retained. +The ordinary CLI and daemon workflow still needs archive setup, capacity +admission and activation wiring. + +Source manifest `442c44e9b2712f1a911ab187d9906b8301ff76ad620cb6bbdfa44be96567552d`. +Native record `414decd4610f9933539b1697c1d74d0ff7eccc8c2b5f6c22e4ebca6fb55581bf`. +RustFS record `fd4922bb25e0e88fe8c3ff436755f1f631e1659f770f05e855e0b512af87f22e`. + +## CG-07 managed namespace standby provisioning + +`FleetCatalog::provision_managed_namespace` validates the compiled class and +fleet-bound namespace before pinning the exact declaration request. The same +operation publishes the empty directory and commits the fleet declaration. +The returned registration can be used for managed host startup. Group birth +still requires the existing capacity and placement proofs. + +The tests lose the declaration reply after the actual CAS, refuse changed +payload under that identity and replay the exact declaration. A separate cut +commits an unrelated fleet update while the standby head is publishing. A new +request at the new fleet revision reuses the original logical operation and +proves that the standby root is identical. Cold inventory reads the published +standby directly. The managed native/Axum path starts from this SDK setup, then +exercises the existing RF3 birth, activation and two-cell request path. + +Six focused native cases and the managed request case against real RustFS +pass. Two existing registry cases now run on the shared fixture and retain +their original replay, scope, missing-page and rollback assertions. The older +legacy target still fails to compile against the retired per-cell host API. +The backend case runs both image formats with host-native voters and fixture +capacity. The backend and dedicated VM stop with their data retained. +Normal CLI/daemon orchestration of first-group reservation and activation +remains open. + +Source manifest `c9926108e4504b1ff9a4f220e0b1551824f02108f612f1e3878b23c558b494fe`. +Native record `2dc3943876d3a235550fc1bf76ab432980ebee49b2f98c18f9cf9d854ae10f5c`. +RustFS record `c8193186b5410d301ecbcb7bc5d31f8506cbe4a0996b00922f103e2047a299c5`. + +## CG-07 managed router membership refresh + +A configured ingress without a local group refreshes its routes through the +existing managed directory service. One bounded turn resolves group authority +against the exact rooted fleet cut. The directory and the receiving quorum RPC +each use a private bounded catalog read turn. The router verifies the authenticated +quorum membership observation and the enrolled physical peers, +including joined-member disk identity. The installed route follows the committed +membership revision and contains the current and outgoing voters. Request grants, +mutation identity, deadline and destination authority stay independent. + +The baseline completed voter replacement and timed out waiting for the new +route. The first candidate reached the replacement and exposed a test oracle +that expected the original receipt value after a later committed write. The +corrected oracle verifies the latest value, the exact original receipt, and a +fresh quorum write read back through its returned version token. The native cases +pass (21.31s replacement with a router, 29.53s original replacement +and joined restart, and 68.74s native/Axum requests). + +The earlier RustFS replacement attempt failed during settlement with +`FleetCatalog::Overloaded`. Native comparisons reproduced contention while the +receiving RPC still used the foreground catalog turn. The request case passed. +The isolated-reader revision passes both cases against +the same constrained RustFS backend (24.58s replacement and 67.93s +native/Axum requests), in both image formats. The backend and dedicated VM stop +with their data retained. Voters are host-native with fixture capacity. The +older S3-source successor-readiness failure retains its separate evidence. + +Source manifest `f2d150294cef69a9959cd0110c5c8b6af540ec82994c71d8e78133fd779284a5`. +Native record `121d71a1d89f54d07cfc17a5a488338f62780d581ef7f0b1a5aa8cae54a670c4`. +RustFS record `ed0a5cde2d751ae85132f1244f41f71a8a09b254c7fc8edbde5fa15a3ad43c37`. + +## CG-07 SDK enrollment without a local replica + +SDK enrollment shares the same trusted leader selection as commands. The +controller captures the creation identity, initializer inputs and preview before +routing. The destination still checks the route and its actual leader authority. +Remote and locally inspected creation results retain separate ownership. The +SDK returns the exact creation record and achieved durability for either result. + +The managed native and Axum case rejects enrollment before the ingress has a +write grant, then enrolls its second cell through the router. The S3 retry +returns the exact original creation record after complete archive coverage. +The ingress has no local replica. Both image formats run. All eight scoped +native cases pass, including the earlier grant, revocation, deadline and +archive checks and the original public counter process case. The managed case +takes 67.41 seconds and the public process case +takes 137.73 seconds. + +The same managed executable passes against real RustFS in +72.11 seconds. The immutable backend identity and +signed provider probes pass. The dedicated backend and VM stop with data +retained. Voters are host-native embedded nodes with fixture capacity. The +earlier RustFS successor readiness failure remains open on its recorded source. + +The source manifest binds 2,728 inputs to +`7140f54b5c12cfb08195f6cbdaba32588b6fa5ba38828bf77f4b038fe32667c7`. +The native record SHA256 is `b7f9632bafeb505027114f83f3eed9613fca1d2654bb665fda18e5806328341a`. +The RustFS record SHA256 is `71e693b794736a74e6e87969271fd2c49a4e6764af28af0d3f2fb33e10cc5bcd`. +The plan index generator preserves its Purpose line and the index test passes. +Reviewed source preflight still reports drift. + +## CG-07 ingress without a local replica + +The combined candidate adds automatic read routing and a read-only command +probe on trusted group routes. The probe uses the command grant and selects +one peer before any mutation is submitted. SDK queries preserve consistency +and tokens through bounded replica attempts. Managed hosts can grant reads +after the ingress process has published its current boot. + +The first combined candidate on 1b3eac2e passes five routing cases and the +four-process managed native/Axum request case. Both image forms run. The +managed case proves a zero-group ingress, separate read and command permissions, +quorum writes, version floor and linearizable reads, cross-cell refusal, exact +receipt replay after leader stop and retained restart, and refusals after quorum +loss. It runs in 55.56 seconds. The standalone SDK +counter process case fails at the explicit remote strong read with HTTP 503 +Unavailable. The earlier read-only candidate fails at the same assertion after +retained restart. The combined failure occurs before the first normal stop. +The runs retain their logs, disk roots and original bounds. + +The first combined record SHA256 is `a1216231b0c7128bd2c1c3e5720ade8e83cd2b9636c3bf330cfb800ed9d21114` and its source manifest +SHA256 is `19725e7b24a05bdceba9eb31eef09fd2c762a899ecc4c2a6d1b63312bc06b391`. This candidate awaits real RustFS +qualification. The source manifest and full test outcomes remain tied to this +checkpoint. + +The archive denial variant receives a quorum acknowledgement and refuses the +explicit S3 acknowledgement during denied writes. Its exact S3 retry after +restoration fails with a TLS EOF. The temporary counter trace captures that +same transport error on a phase-zero version-floor read to member 1. The +receiver bounds its idle header read, but the original sender retains its +socket while waiting for a request. A focused regression waits through that +idle interval and fails at the first exact retry with no leader selected. +The red record SHA256 is `06b8f40441466b7906a763698555afc60c5ad6a61961d28420ce20d9d599cd48`. + +The idle socket fix observes peer closure before the next queued request. +It retires the old connection before transmission and sends each mutation once. +The same expiry regression passes in both image formats. All seven native cases +pass on the final source. The managed HTTP case takes +67.83 seconds and the original counter process +drill takes 140.34 seconds. The managed case returns +a quorum receipt while archive writes are refused, withholds the explicit S3 +acknowledgement and returns that acknowledgement on the exact retry after +archive access is restored. + +The same managed binary passes against the verified real RustFS backend in +71.19 seconds. Both image formats run with the +archive-write fault limited to the archive prefix. Signed readiness and +conditional probes pass. The dedicated RustFS container and VM are stopped +with their data retained. Voters remain host-native with fixture capacity. +The earlier RustFS successor readiness failure remains open on its recorded +source. + +The final source manifest SHA256 is `be85dfd746e849ce98ed0dbe5c8b48ec5b1da83ef0bdd3c4d23e84dc05a3721c`. +The native record SHA256 is `db5a0bf906cdebf2886af58afe92d0ee1111a044050a5b6e1787bab09610be82`. The RustFS record SHA256 is +`91a4a9ff56009aed4281c587f39d1d26d48b1f765298ee881d8184866444fda4`. The subsequent CI-only commit changes no frozen build or harness +inputs. Earlier failed runs retain their original assertions, logs and disks. + +## CG-07 lost movement observation + +A managed regression drops the first foreground catalog read while the leader +and quorum stay live. The old movement discovery samples each candidate once +and misses the leader. The fixed turn allows one more read-only sweep within +the original deadline. Boot, group binding, sample age and current-root guards +still apply; no membership action is retried within the turn. The test verifies +fresh quorum evidence of durable Seeding, an unchanged physical debit and no +learner activation in both image forms. + +The exact red run fails on missing durable Begin; the green run passes in +14.37 seconds. Record SHA256 values: + +- Red: `4e141feda39aee045607e844708fcae42e64e85df408099728a95179669a491b` +- Green: `a22c8eef63fb0761e61e44c0ab7e66dd57ad9d8e97c6319c74f9333cdb2080d8` + +The complete managed native target passes 19 cases with eight explicit backend +skips in 371.39 seconds. This includes the two lifecycle cases +that failed in the earlier eleven-target campaign. The current source manifest +binds 2,727 inputs to `c719c603552528b9252773b332c601e3a5f6cbe6083bac03257ef63d5618f65d`. The managed record SHA256 +is `bec0485c1a712f0773e8d96b9ca0ead487903698efa4df7258230bead7e5e436`. + +The real RustFS matrix fails overall. The lost-observation case passes in +17.30 seconds and archived maintenance passes in +82.90 seconds. The S3 source failover case fails +in 14.21 seconds. Both surviving voters report term +2 with no leader at the original five-second readiness deadline. The staged +destination again reports Unobserved on close. The failure logs and disks +are retained. The RustFS record SHA256 is `f6443912bc12a44f2112807756d5c0fcd28afa2f38d2ba28e011796742c5b7a7`. The signed +readiness and conditional probes pass. The dedicated constrained RustFS +container and VM are stopped with data retained. Voters are host-native +embedded Nodes. + +The intermittent maintenance and successor-readiness failures passed isolated +replays on the prior source, including the temporary instrumented runs. The +RustFS recurrence leaves successor liveness and the destination close report +open. The discovery fix establishes behavior under a lost observation. +The earlier full native campaign remains failed at its recorded source. + +## CG-07 managed public command routing + +A public SDK command sent to a follower previously attempted local SQL and +returned NotLeader despite an observed healthy leader. The pre-fix managed HTTP +case reproduced this twice, with failed binaries/logs and voter directories +retained. Queued originating SDK commands now retain one controller invocation, +identity, captured inputs, admission and original deadline through one forwarding +hop. Incoming RPCs and the lower-level controller command API stay local-only. +The destination independently checks its route, namespace process permission and +current execution authority. Possible transmission keeps an error ambiguous; +there is no automatic uncertain mutation retry or new identity. + +Trusted embedders can grant/revoke command origins after managed discovery has +verified the actual peer boot/endpoints. Host namespace capabilities and current +transport identity/revision checks are mandatory. Membership grants no permission; +revocation retains a revision tombstone without undoing admitted work. Permissions +belong to the running transport and require trusted policy after restart. + +Command output now distinguishes local inspected SQL receipts from authenticated +remote results. Common output metadata is available for both; local_receipt is +present only for the local variant. Wire encoding, effects and physical oracles +retain local proof requirements. Existing tests make those requirements explicit. +Temporary diagnostic logs are removed and no assertion or lint is suppressed. + +The new managed native/Axum case passes in both image forms: cold capacity and +group birth, replicated enrollment of two independent cells, follower writes, +remote minimum-version/linearizable reads with actual responder identity, +conflicting retries, cross-cell tokens, absent/revoked/stale-process/foreign- +namespace permission refusals, forwarded S3 acknowledgement, joined leader-stop +failover, retained restart with original receipts/floors and new writes, then +quorum-loss write/read refusals. The counter declares explicit namespace origins; +its public process gate now requires original receipt success through every one +of six native/Axum addresses before and after retained restart. + +The complete eleven-target final-01 native campaign is **FAILED: 308 passed, +2 failed, 39 explicit skips**. The new managed HTTP case and public SDK/process +gate pass; the public gate takes 140.55s. The managed +target fails eight-turn maintenance replacement settlement and fresh successor +readiness after closing an S3 source service, including an Unobserved close +report. Original bounds/assertions and failure disks/logs remain unwaived. These +are previously recorded failure classes; the forwarding fix does not establish +their causes. No narrower component pass makes the full campaign green. + +RustFS comparison: managed request run1: PASS (53.37s); managed request run2: PASS (50.61s); archived maintenance: PASS (88.53s); automatic S3 recovery: PASS (45.41s). +Matrix status is **PASSED**. Both request invocations cover both image +forms and pass; every comparison case retains its own result. Signed readiness +and conditional probes pass. The immutable-ID/owner/resource/mount-verified, +pinned one-CPU/two-GiB backend is reused with fresh buckets/prefixes; backend and +dedicated VM are stopped with data retained. Voters remain host-native embedded +Nodes with fixture capacity/isolation. Joined stop is not a process-crash drill; +independent constrained voters, permanent fencing, partitions and load remain +unqualified. + +Source-final-01 binds 2,726 Rust/Python/manifest inputs above +3ccc3e51f03458a2a5715401b8e5f6b963146276 to SHA256 +da3514ed9ea2dd614a45f3150f92d2280f75795287b7469a5883f72564b1a87c. +FAILED native record SHA256 is +6a4f24b43031c9b9cab3a996edd3b3bf280ecf494a2a231d997912b0e75098e7. +Its individual managed request proof is extracted explicitly from that failed +campaign (SHA256 8f6e418e605c0d56a635b245ded5cc423d7657cf8f548bdc40109cdb3bdac1a8). +RustFS record SHA256 is +646ebe709b457b540a97a12e85e43aade6aee1fbbb445a5d37d84a2650330695. +Default stack, opt-level 0, assertions/overflow checks, quotas and original +operation/movement bounds remain intact. Code inputs are checked unchanged. + +Format/layout/dependencies/docs, native-only Cellar/Python-library and diff checks +pass. Strict library Clippy and plan tests/validate/preflight remain red, with no +Clippy diagnostic-key additions or silent source-anchor refresh. Earlier unseeded +maintenance/ambiguous-write/transfer/TCP/enrollment/inspection failures and older +disk-erasure qualification keep their own source and scope. CG-07 remains +IN_PROGRESS; four public gates, default/capability integration and CG-09–16 remain +open. Automatic command selection still needs local group status: route-only +ingress requires bounded leader discovery. Queries still default stale. + +## CG-07 bounded member observation renewal + +A deterministic managed regression now reproduces one lost local-member +observation after leadership handoff. Its foreground catalog client is independent +of the controller and background trust refresh, over the same actual backend. +Before the fix, that read consumes the entire original five-second renewal budget; +local snapshots show a healthy quorum and empty work banks. The peer observation +now receives a share of the remaining time, reserving time for other members and +final validation. A transient refusal leaves the history pending; retry obtains +all three fresh safe ReadIndex/durable-apply observations. Both image forms pass +without resuming the lost read. Root, boot, configuration and proof-age checks +remain intact. No write retry, stale proof or longer deadline is introduced. + +The lifecycle harness also reconciles actual leadership before/after maintenance +using fresh term/revision guards, rather than treating a historical survivor and +one admitted handoff as completed election. It observes actual target leadership +inside one original five-second envelope. The eight movement turns and all +durability/capacity assertions remain unchanged. Bounded local status/admission +snapshots are recorded on failure before cleanup; the original error still fails. +Temporary tagged tracing and its focused prototype are removed. + +Source-final-02 binds 2,722 Rust/Python/manifest inputs above +6b70a5357a470f75b7e36ec201f44f31ea7da536 to SHA256 +7ad7649846ca9405e6be33738b6d851f256a887e4af08c4f86004c590039554b. +The complete affected managed target passes **17 cases with 6 explicit backend +skips**, in 307.21s. Native record SHA256 is +e589e438c251e71d8ddba76abf8eb9055b63b7a08abec06526a402b431424441. + +Four same-source RustFS invocations pass: the lost-observation regression +32.19s, the unchanged full archived maintenance case +twice (85.71s and 100.80s), +and automatic S3 voter recovery through terminal removal +40.46s. Both image forms run; maintenance additionally +covers initial and joined voters. Record SHA256 is +d3c69d70f2e2fe99a840582cdf7248f5c833abbeb939a54b125ea95fa18e4040. +The immutable-ID/owner/resource/mount-verified one-CPU/two-GiB backend is reused +with fresh buckets/prefixes because retained Docker networks exhausted default +subnets. No retained resource is deleted; backend and dedicated VM are stopped +with data retained. Nodes remain host-native with fixture capacity/isolation; +independent constrained Linux voters and production fencing/load remain unqualified. + +The earlier unseeded renewal timeout was not captured at a causal low-level seam. +An instrumented full case also failed an ambiguous sequence-4 S3 command before +owned resume; another diagnostic run missed restored-target leadership. Those +failures and the initial final-source 15-pass/2-fail campaign (NotReady maintenance +and eight-turn peer replacement) are retained and unwaived. The new regression +proves recovery from a lost observation, not every prior failure's root cause. + +Format/layout/dependencies/docs, native-only Cellar/Python-library and diff checks +pass. Strict library Clippy and plan tests/validate/preflight remain red; diagnostic +comparison adds no key. Reviewed source anchors are not refreshed and no lint is +suppressed. The earlier seven-target/SDK and disk-erasure results remain bound to +their recorded source; this checkpoint qualifies the affected managed boundary. +CG-07 stays IN_PROGRESS; four public gates, default/capability cutover, CI/load, +unexplained failures and CG-09–16 remain open. + +## CG-07 automatic shared-group failure recovery + +The Node-owned automatic failure service now passes the complete local +shared-group path in both image forms: default failure detection, rooted +exclusion, exact-boot fixture isolation, S3 replacement, fresh stable RF3, +independent archive verification, finalization and journal-confirmed removal. +The fixture first holds isolation and proves repair has not started. Shutdown +joins that callback and releases control admission. A duplicate service refuses +without another charge. Two service restarts preserve the rooted operation; +archive payload refusal blocks terminal progression, and restored reads require +new physical S3 GETs before removal. The old engine remains byte-identical and +retained, active movements/reservations clear, and both cells retain original +receipts and accept new explicit-S3 writes. No business mutation drives repair. + +This new path first aborted on the unchanged default stack. Sanitized poll-frame +evidence and repeated red binaries are retained. Control turns, recovery/removal +phases and live verification now retain boxed futures under the existing worker +owner, admission and original deadline. A subsequent receiver refusal hid the +actual missing-spool I/O error. Archive download now joins and returns the typed +consumer failure; the malformed-checkpoint assertion is tightened accordingly. +A real-RustFS repeat exposed about 1.96 MiB of application poll frames before +the HTTP client. Catalog turns now own boxed work before their future is +constructed, shrinking nested control frames without moving lock or deadline +checks. Managed replay creates and syncs its private scratch directory before receiving +materialized image bytes. No stack, quota, grace or operation deadline is raised. + +Source-automatic-qualified binds 2,721 Rust/Python/manifest inputs +above 29f277ddf228f33068c229583efb487e01749f07 to SHA256 +e8c8f4379f8587ee70c5b011dcaf11442104fde90915da5359199f95ebfaaa3b. Seven native targets pass **267 cases with 36 explicit skips**; +the public SDK/native/Axum process/restart gate passes in 64.65s. +Native/public record SHA256 is +b5d4f68e12ad56e0676e6f29974ce9035584579e46edf0b8c470816359ef151c. + +Three exact lifecycle cases pass against pinned real RustFS in both image +forms: automatic recovery, isolated replacement and normal replacement/joined +restart. Their durations are 43.82s, 26.58s, 37.99s. +The same campaign then FAILS archived joined-voter maintenance after accepting +leadership handoff: the original five-second survivor-proof renewal expires. +The 38.54-second failed fixture and all disks are retained; no deadline is +raised and no later pass waives it. The disk-loss case was not reached there. +Campaign record SHA256 is +13b1f1f4a6b88cfd6362e5ad07fe79a6b5bd15e4555b0a580fecf7ce61dc0243. + +A separate same-source exact public fresh-incarnation recovery case passes in +both image forms after every old voter directory and verifier cache is erased, +in 20.67s. Its record SHA256 is +f739ea9c2027c50ecfa1c1c02176a59a8a9415f2447ade8128000c24ae836298. +Signed readiness/conditional probes pass for both owned backends; each one-CPU/ +two-GiB backend and VM is stopped with data retained. These are host-native +embedded Nodes and fixture isolation, not independent constrained voter containers, +permanent Docker launch revocation, sustained load or the full public lifecycle. +The backend matrix is not green. + +Format/layout/dependencies/docs, native-only Cellar/Python-library checks and +diff checks pass. Strict library Clippy and plan tests/validate/preflight remain +red; the saved diagnostic comparison records 0 added diagnostic +keys. No lint suppression or source-anchor refresh is applied. Earlier TCP, +bounded readiness/maintenance, 128-cell enrollment and inspection failures remain +unexplained and unwaived despite later passes. CG-07 remains IN_PROGRESS; +CG-09–16, the public/default/capability cutover and the full goal remain unfinished. + +## CG-07 quarantined voter replacement and startup refusal + +CG-07-seed-cleanup-01/source-isolated-final binds 2,719 Rust/Python/manifest +inputs above 9070eb15bff289f52f3c3faddd606575fdefcaf1 to SHA256 +cc2e83093bfe91439dce852e25e3340108b885f3b79b3a05e1453a9596f3feb1. +The complete seven-target native campaign passes **266 cases with 35 explicit +skips**; the public SDK/native/Axum process/restart gate also passes in 114.80s. +Combined native/public record SHA256 is +60c4918d707c26a95efefa112dfa8400b194c67faa8643b71edd20586b812732. +All cases use the same frozen source; local opt-level 0, debug assertions, +overflow checks, quotas, five-second calls and eight movement turns are retained. + +A new physically joined obsolete-voter fixture reproduced two distinct faults. +Saved startup authority refused only after changing the quarantined Raft log; +two fresh diagnostic fixtures identify the same `.raftlog` mutation. Startup now +checks current host authorization and namespace declarations in one catalog turn +before opening the engine. The later engine-bound boot CAS remains mandatory. +Replacement then exhausted eight turns because shared route refresh kept the +quarantined initial endpoint, preventing publication of the admitted learner's +route. Refresh now discards that retained endpoint only with rooted isolation, +including when membership is otherwise unchanged. Votes and physical debt remain. + +The new case passes both image forms, requiring exact typed startup refusal, +unchanged fleet root/engine files, S3 payload reads, fresh stable RF3, restored +cell-local images/versions and receipts, lost settlement confirmation/exact replay, +`isolated_sources` attribution and the replacement's physical capacity floor. +The old log remains byte-identical and quarantined. Surviving-original-voter +physical closure is still required; the full managed failover/debt case passes. +The earlier red runs, diagnostic sources/binaries and failed fixture disks remain. + +Four exact cases pass against pinned real RustFS in both image forms: isolated +S3 replacement 23.06s, normal replacement/joined restart 36.26s, archived +initial/joined maintenance 84.55s, and public fresh-incarnation recovery after all +old voter directories and verifier cache are erased 18.04s. Record SHA256 is +3ea848910a5843bded56eb0140933cb2bd4120545bc32be37d6c9bee838e0d16. +Signed readiness and conditional operations pass; the one-CPU/two-GiB backend +and dedicated VM are stopped with data retained. These are host-native embedded +Nodes with fixture fencing, not constrained independent voter containers, +permanent Docker launch revocation, sustained load or the full lifecycle gate. + +The current campaign passes the previously failing TCP retry, but does not +explain or waive that failure, earlier bounded readiness/maintenance failures, +128-cell enrollment or inspection failures. Format/layout/dependency/docs and +native-only Cellar/Python-library checks pass. Strict library Clippy remains red +with 220 reports (202 dead-code reports). Its only message change is that the +grouped unused-method warning no longer includes the now-used `is_isolated`; +expanded item/count comparison adds no unused item or other diagnostic. No lint +is suppressed. Plan tests retain 35 passes, one generated-index/Purpose failure +and one reviewed-protocol drift error; validate/preflight refuse. SDK CI profiles +and complete public/default/capability gates remain required. CG-07 +stays IN_PROGRESS; CG-09–16 and the full hard cutover remain unfinished. + +## CG-07 owned idle and restart seed retirement + +CG-07-seed-cleanup-01/source-final binds 2,718 Rust/Python/manifest inputs above +acd56a9fc2f14937af269e0c6e1ca54bb047ccb1 to SHA256 +9d8d111ff2121fe9802d186b04710039e480c478366b0de9f0d372719e995603. +The preceding candidate has SHA256 +7ad9ac05a2da65ade95b300b5462523369f8d847197efce23900eec95fe87480. +Only the managed test's inventory measurement moved before engine close, with an +additional sealed-admission assertion. All production inputs and seven other test +binaries are unchanged; the complete managed target was rerun. Combined record +SHA256 is c53bfd8c79acb345e3a27e64a751b9760c8c818fdb62473896eb1da49bf0433b. + +Abandoned seed-upload journals now reconcile without another learner request. +One streaming node-owned sweep validates history path, canonical purpose, engine +creation and immutable namespace archive identity. It shares live sources' group +locks, skips their entered work and reconciles only the original reserve/upload +intent. Confirmed abort precedes syncing the empty record. Backend failure keeps +the intent and observable debt; missing/corrupt/foreign metadata never initializes +a replacement or authorizes retirement of an unknown owner. Cleanup may reopen an +old immutable binding without granting current membership or serving authority. + +The existing provisioning bank charges one 272-KiB journal workspace per archived +node. Ordinary 8-KiB S3 admission is per journal. Physical joins precede release of +locks/admission, including panic/error paths. Error capacity cannot stop scanning +other entries; weak lock bookkeeping does not accumulate every past history. +Close stops new sweeps and joins entered work before engine/configuration release. +No configured ceiling, original five-second call/eight-turn bound, stack, profile, +authority fence or durability assertion is relaxed. + +The regression first failed at idle retirement: actual denied registry writes +left a pending journal/error that never retried after backend recovery. The fixed +case requires idle retirement, then another retained intent through source close +and fresh-quorum repair. After every old host joins, the source alone reopens that +intent without quorum or another learner request. A gated archive read keeps close +pending and charged after its waiter is cancelled; release permits durable +retirement and joined shutdown without changing fleet authority. Both image forms +run. Existing-only journal tests reject wrong engine/purpose and missing state or +identity without reconstruction. + +Seven native targets pass **265 cases with 34 explicit backend/process skips**: +managed 14, snapshot 5, archive-controller 16, archive/recovery 31, host 3, +storage 16 and execution 180. Two targeted library journal tests pass. The final +focused cleanup case passes in 32.52 seconds and complete managed rerun in 221.10 +seconds. Current public SDK/native/Axum process/restart passes in 132.07 seconds; +its counter executable and process evidence remain retained. + +Four cases pass against pinned real RustFS at one CPU/2 GiB after signed +conditional-operation checks: replacement/restart 36.48s, initial/joined +maintenance 80.70s, refusal/failover plus idle/solo-restart cleanup and cancelled +close 36.15s, and fresh recovery after erasing every old voter directory/verifier +cache 17.66s. Both image forms run. Record SHA256 is +1fb2db0919585b80bb19311abdf382597a21acb154f59382021c176864590d7e. +Backend/VM are stopped with data retained. These embedded-voter cases and contract +fence do not qualify constrained independent Linux voters, permanent Docker launch +revocation or sustained load. The public SDK drill separately uses native processes. + +Format/layout, dependency/docs and native-only Cellar/cellar-python checks pass, +including the unavailable Python sentinel. Strict library Clippy retains **220 +diagnostics (202 dead-code)** and the same 206 normalized diagnostic keys, with no +added keys or suppressions; the seed module path move is normalized. Plan checks +retain 35 passes, one generated-index/Purpose failure and one reviewed-protocol +drift error; validation/preflight still refuse the drift. + +TCP retry passes this complete execution target, fifteen earlier exact repeats +and 33 checkpoint cases with frame/stage error context. Its original overload +failure remains unexplained. Earlier five-second readiness, eight-turn maintenance, +128-cell enrollment and inspection failures remain open; passes do not waive them +or establish a failover SLA. Full performance/library suites were not rerun here. +Finish fenced failed-voter/survivor-debt orchestration, the remaining public gates, +capability/default cutover, remaining release profiles and CG-09–16. CG-07 remains +IN_PROGRESS and PR #160 remains draft. + +## CG-07 managed S3 learner data plane and source failover at acd56a9f + +Capture CG-07-managed-s3-seeding-01/source-candidate binds 2,717 +Rust/Python/manifest inputs above 41a67844278247df18c16df3c781a3c60bc6dd71 +to SHA256 65cbd074ab06749b4849ef06d38c23e0df396385c63e3b74bc6fb48159cdf5e7. +No original five-second call, eight-turn reconciliation bound, profile, quota, +stack setting or authority fence changes. + +Managed `from_s3` activation now uses the existing fresh ReadIndex checkpoint +authority, a separate fsynced seed-upload journal and the established namespace +archive registry. A pending generation and inherited parent graph protect the +complete checkpoint. The authenticated grant binds its ticket/reference and +snapshot. The destination independently verifies S3 bytes through its serving +receiver before READY1; live source checks precede End and the existing +boot/catalog/engine installation fence remains required. This transfer supplies +neither a disaster-recovery head nor an S3 command coverage proof. Data modes are +explicit marker-1 fields; missing/old shapes and peer-mode fallback refuse. + +Source jobs retain transport, metadata, read, checkpoint and S3 admissions through +physical completion and cleanup; close stops new admission and joins source work +with transport. An uncertain retirement retains its journal and observable debt; +a later seed reconciles the exact intent before allocating again. Automatic +abandoned-journal discovery/cleanup on idle nodes and restart remains unfinished. + +An exact red regression reproduced `Unconfirmed(Invalid("conflicting provision +binding"))` after source failure: coalescing incorrectly included source +member/node hints. The fixed comparator preserves destination boot/member/permit +and data mode while reauthorizing the next source. Entered attempts retain their +original hint/deadline. Tests require real target-only payload refusal before +attachment, retained preparation/debit, no peer fallback, source-service stop, +fresh election and completed membership repair. Replacement-local reads check +actual responder, original values and versions; new writes require S3 proof. +Catalog settlement still refuses without a live physical-inventory proof from an +offline surviving original voter and retains the debit. No completion guard is +relaxed. Normal full settlement, lost-reply replay, drain and retained restart +remain covered separately. + +The frozen-source native campaign has **264 passes, one failure and 34 explicit +backend/process skips** across seven targets: managed 14, snapshot 5, +archive-controller 16, archive/recovery 31, host 3, storage 16 and execution 179. +`checkpoint_tcp_failed_transfer_allows_raft_retry` fails with transport +`Overloaded`; its unchanged focused repeat passes in 7.17 seconds. The full +execution run remains failed. Snapshot process-kill, partial-install, ownership +and memory/pin gates pass. All source inputs remain unchanged. + +The same binaries pass four explicit cases against pinned real RustFS constrained +to one CPU and 2 GiB, after verifying signed conditional operations. S3 replacement +and retained restart take 34.15 seconds; archived maintenance's initial/joined +voter matrix takes 85.93 seconds; payload-refusal/source-failover/debt checks take +33.04 seconds; public recovery after erasing every old voter directory and verifier +cache takes 17.56 seconds. Both image forms run throughout. Backend and dedicated +VM are stopped; data is retained. Record SHA256 is +b7d35381cdc4d2a387ce4bfcca28b12c5841c02288d3625364a3891ba52c88f2. +An earlier backend setup returned S3 HTTP 503 after health readiness, before any +Cellar case; that failure is retained. The second setup first checks a bounded +signed read, then the conditional probe. These use embedded Nodes and a contract +fence, not independent voter containers, permanent Docker launch revocation or +sustained load; no five-second failover SLA is established. + +The extent failover case previously exceeded its five-second ready-successor +window in a complete managed run. Three isolated repeats and subsequent complete +managed runs pass; no failure snapshot was triggered in those repeats. Root cause +remains unresolved. Bounded owner observations now run only after a failed window +and cannot rescue/extend it. The earlier eight-turn archived-maintenance failure +is also preserved. Later passes do not waive either failure. + +Strict library Clippy retains **220 errors** versus 221 at the preceding +checkpoint, with no added diagnostic keys or production suppressions. The one +removed error is the seed client's argument count; all production inputs match +the candidate. The 202 dead-code errors remain. Plan tests still have 35 passes, +one generated-index/Purpose failure and one reviewed-source drift error. Public +SDK process, full performance and full library gates were not rerun; prior +128-cell enrollment and inspection failures remain open. CG-07 stays IN_PROGRESS +and PR #160 stays draft. Finish automatic seed debt retirement, fenced failed-voter +and public lifecycle/capability/default gates, then the remaining release profiles +and CG-09–16. See [remaining work](remaining-work.md). + +## CG-07 archive ownership for post-startup groups + +Capture CG-07-dynamic-archives-01/source-final binds 2,714 Rust/Python/manifest +inputs above 8c0a42f02eb78060adaa5c87472ab34f95b6e5c7 to SHA256 +9cb0776c7a699f6e7af5f1aa6ed03d3bd09f277967a6bcbb5d70051a12d39f7d. +No request deadline, quota, profile, authority fence or durability assertion changes. + +Canonical startup retains archive provisioning resources for groups added after +boot, including a node that starts with no groups. Preparations share the original +archive pools and independent verifier image inventory. Both serving and verifier +store bounds are charged to the existing provisioning bank. The selected birth, +boot/deadline and existing catalog archive identity/counters are checked before +local bootstrap. Missing metadata never allocates a replacement archive. + +Preparation retains the private archive store with its physical group owner +through retries and learner installation. Actual attachment transfers it directly +to the controller's bounded archive lane, avoiding a recursive wait on that actor's +public registration queue. Existing preparation and archive joins retain those +resources through node close; immutable provisioning resources release after the +worker joins. Attachment alone grants no S3 acknowledgement or fresh quorum. + +The managed target passes twelve cases. New cases cover RF3 startup from empty +nodes, independent cells and explicit S3 settlement in both image forms; missing +archive refusal without genesis/catalog revision changes; explicit synced operator +initialization and retry of the same birth/capacity slot; and a held archive GET +that keeps joined close pending after its waiter expires. Quorum writes still +complete independently of asynchronous archival. These use real native Nodes, +SQLite and TLS with an instrumented in-memory object store. They do not qualify +managed S3 learners, managed archived replacement/maintenance, permanent Docker +fencing or sustained RustFS load. + +Seven native targets pass 263 cases with 31 explicit backend/process skips: +managed 12, archive-controller 16, host 3, storage 16, archive/recovery 31, +snapshot 5 and execution 180. Sources remain unchanged after the serial campaign. +The same binaries pass the public recovery drill against pinned, constrained real +RustFS in 30.33 seconds for both image forms after erasing every source voter +directory and old verifier cache. Backend and dedicated VM are stopped afterward. +That drill uses embedded Nodes and a contract fence; it does not qualify the new +managed archive cases against RustFS, permanent Docker launch revocation, +independent voter containers or sustained load. The public SDK process drill, +full performance suite and full library suite were not rerun at this checkpoint. +The preceding 8c0a42f0 performance/inspection/release gaps remain open. + +Formatting, layout, native dependency/docs checks and the native-only +Cellar/cellar-python library check pass, including an unavailable Python sentinel. +Strict Cellar library Clippy still fails with 221 errors (202 dead-code), down +from 222: the retained preparation wrapper removes the private-interface error; +no diagnostic key was added or suppressed. Plan tests retain 35 passes, one +index/Purpose failure and one reviewed-protocol drift error; validation/preflight +still refuse the stale generated index. No source anchor was silently refreshed. + +The next functional priority is the currently refused exact-reference S3 learner +path, including retained closure/receipt proofs, destination boot/catalog checks, +physical install, catch-up and fresh quorum before promotion. Full public +lifecycle/default-backend gates, archive hold retirement, live manifest-page +collection and CG-09–16 remain required. CG-07 stays IN_PROGRESS; PR #160 stays draft. + +## CG-07 owned workspace directory barriers + +Capture CG-07-wide-enrollment-01/source-08 binds 2,712 Rust/Python/manifest +inputs above d268cf0bfc78c49f8c2d48f17a4a735f42028d81 to SHA256 +f9449fbd8937d6a2b5f9a871859bf1e7469b767df51f86f8720a2fb8fff61c57. +The original optimization, stack, quotas, deadlines and assertions remain. + +Workspace creation now reuses the complete ancestor barriers established by its +open, exclusively locked extent inventory. It still syncs the new child and +parent before returning the admitted owner. Rooted private initialization and +execution validate and retain that exact directory owner; arbitrary materialized +paths keep their full directory preparation. Anonymous read-cache directories +use ordinary creation because they supply no recovery or persistence authority. +Failed child/parent barriers still quarantine and retain the complete reservation. +Last-pin cleanup still joins physical handles and syncs deletion before refund. + +Preserved regressions fail before the fix: workspace creation performs twelve +directory syncs rather than the two required barriers, and native initialization +fails on the redundant ancestor sync. The new tests also inject child and parent +sync failures separately and verify conservative charges, quarantine and recovery. +The native case commits both initialization and a subsequent SQL mutation while +redundant ancestor syncs remain armed to fail. + +At this source, five LTX targets pass 66 cases with four owned child helpers +ignored. Eight shared-group native targets pass 262 cases: execution 180, +storage 16, archive/recovery 31, archive-controller 16, managed 10, host 3 and +public SDK/native/Axum/restart 1, plus five snapshot cases. Their 31 explicit +backend/process skips remain recorded. The public drill preserves its original assertions +and eighty additional writes on each side of restart; it passes in 155.92 seconds. +The affected legacy execution targets also pass: 21 execution and 50 rooted +cases, with five explicit backend/process skips. These protect the existing +callers during the cutover; predecessor removal remains required in CG-16. + +The original performance suite has eighteen passes and one failure in 439.35 +seconds. Both 32-cell scenarios, warm-cache checks and the sixteen-root pressure +case pass. Occupancy 128 still fails enrollment with ambiguous +Runtime(DeadlineExceeded), with 58 cells observed under the original 60-second +population budget and five-second call cap. Earlier diagnostic sources and their +exact executables remain preserved. Stage timing isolated workspace setup and +manifest preparation; publication and collection were smaller in that sample. +Two bounded-pipeline experiments failed to clear the deadline and were removed, +along with all temporary diagnostic logs and the duplicate diagnostic test. The +original scheduler and the applied-equals-committed enrollment fence remain. +These measurements do not establish a production latency or density envelope. + +All binary targets, native-only Cellar/Python compilation with an unavailable +Python sentinel, LTX feature-off/all-feature builds, strict LTX library Clippy, +formatting, layout, dependency isolation, docs and diff checks pass. Strict Cellar +library Clippy retains 222 diagnostics, including 202 dead_code, with unchanged +code/message/primary-file keys. The full library was not rerun; its prior +inspection-protocol failure remains open. The plan suite retains 35 passes, +one generated-index/Purpose mismatch failure and one reviewed-protocol drift +error; validate/preflight stop at the stale index. No suppression or automatic +baseline refresh was added. + +The exact archive executable passes public recovery against real pinned RustFS +in both image forms in 19.30 seconds. Every source voter directory and the old +verifier cache are erased before fresh-incarnation restoration. Complete images, +receipts, client floors/S3 minima, fresh-quorum activation, old-token refusal, +new writes and retained restart pass. Backend identity/resources and a signed +conditional-S3 probe are checked; its one-CPU/two-GiB container and dedicated VM +are stopped afterward with data/evidence retained. This embeds three native Nodes +and a contract fencing authority. Permanent Docker launch revocation, independent +voter containers/hosts, managed recovery and sustained load remain unqualified. +Record SHA256 is +38e947c6a5c1e5c053fd47798a4d678f95dba2f72afadf7d427566a1e4ec712e. +Source-after comparisons remain unchanged. + +CG-07 remains IN_PROGRESS and the full hard cutover is incomplete. Next functional +priorities are managed dynamic archive ownership, its stop/drain integration and +exact-reference S3 learner activation. Continue measuring physical image/page +preparation before selecting the next performance change. Live manifest-page +collection, the remaining public/default-backend gates and CG-09–16 remain required. + + +## CG-07 archive metadata and warm verifier checkpoint + +Capture CG-07-wide-group-diagnosis-01/source-08 binds 2,712 +Rust/Python/manifest inputs above 3dc6e25cb23a87a263edf865d24c626d087ebee4 to +SHA256 4b55360d8d064379763d69f3d57b13b34074bef86b9328568da773b636b775e0. +Fetched main remains 981a5417d9359d1f992e2dbf214eec7d221b6777, zero commits +behind. Original optimization, stack, deadlines, quotas and assertions remain. + +Immutable extent-root and manifest-page owners now remember their own completed +file and directory barriers. Every preparation still reads and checks the bytes; +only redundant barriers for an unchanged immutable file may be reused. Reopening +clears the observation, failed reads invalidate it, metadata-only reads establish +no persistence, and failed page barriers retain quarantine. This grants neither +an image seal nor remote coverage by itself. + +The independent verifier keeps eligible files as a charged cache below one quarter +of each existing inventory ceiling. Any dimension reaching pressure, or orphan +objects without roots, enables its ordinary owned collection. Serving collection +remains eager. No pins, banks, capacity or cached archive proofs are added; remote +GETs, complete closure and independent replay remain required. Buffered extent +imports now check supplied bytes before destination reuse. Previously a valid +warm destination could hide wrong supplied bytes of the same length. + +Preserved regressions demonstrate redundant root/page syncs, complete idle cache +eviction and acceptance of wrong buffered input before their fixes. New public +archive coverage forges incoming extent bytes while preserving valid object/index +hashes and the original expected extent identity: both cold and warm verification +refuse, while the original four-cell archive remains usable. The first attempt at +that public test failed fixture setup by assuming one chunk per extent; it is +preserved as a setup failure, not evidence of the production bug. + +The same source passes six LTX apply and 39 extent-store cases, with one owned +child helper ignored. Seven native targets pass 256 cases with 29 explicit +backend/process skips: execution 179, storage 16, archive/recovery 31, +archive-controller 16, managed 10, host 3 and public SDK/native/Axum/restart 1. +The public process drill passes in 179.65 seconds with its original assertions +and eighty additional writes on each side of restart. +Five additional snapshot cases pass, with two owned child helpers ignored: +partial-install refusal, complete multi-cell cuts, extent restart, pin/memory +bounds and whole-group learner/joint membership. All binary targets, native-only +Cellar/Python compilation with an unavailable Python sentinel, LTX feature-off +and all-feature builds, and strict LTX library Clippy pass. Strict Cellar library +Clippy still fails with 222 diagnostics, including 202 dead_code; diagnostic +code/message/primary-file keys are unchanged. No suppressions or baseline refresh +were added. The full library was not rerun; its prior inspection-protocol failure +remains open. + +The exact archive executable passes public recovery against real pinned RustFS +in both image forms in 33.02 seconds. All source voter directories and the old +verifier cache are erased before restoring complete images/receipts, client +floors and S3 minima into fresh incarnations. Fresh quorum activation, old-token +refusal, new writes and retained restart pass. The backend enforces one CPU/two +GiB with checked identity/resources and a signed conditional-S3 probe. Backend +and dedicated VM are stopped with their data retained. This uses three embedded +native Nodes and a contract fencing authority; permanent Docker launch revocation, +independent voter containers, managed recovery and sustained load remain +unqualified. Record SHA256 is +b8adf98e6f019da487c493a71c2107ec4beee255e531b17723eaf074ff4d082f. +All source-after comparisons remain unchanged. + +Formatting, layout, dependency isolation, docs and diff checks pass. The plan +suite has 35 passes, one generated-index/Purpose mismatch failure and one +reviewed-source drift error for the consensus protocol. Validate/preflight stop +at the stale generated index. These failures remain explicit; no automatic +baseline refresh or task-status changes were made. + +The complete performance suite improves to eighteen passes and one failure in +483.34 seconds. Both 32-cell scenarios now pass their original enrollment, S3, +duplicate-receipt and every-voter read checks. The 16-root pressure case and new +warm-cache case also pass. The matrix reaches occupancy 128, then fails enrollment +with ambiguous Runtime(DeadlineExceeded); the observed large group has 44 cells +under the unchanged 60-second population budget and five-second per-call cap. +Quorum and collector observations remain healthy at that sample. This does not +establish a cause, fixed cell-count boundary, density claim or a passing load gate. +Earlier source failures, exact executables and fixture disks remain preserved. + +CG-07 remains IN_PROGRESS. Measure the remaining enrollment, checkpoint and +collector work, then finish managed archive attachment, exact-reference S3 learner +activation, live manifest-page ownership, the remaining public gates and full +Docker-fenced qualification. The default backend, capability/deployment/read +decisions and CG-09–16 remain required; see [remaining work](remaining-work.md). + +## CG-07 owned live extent collection + +Capture CG-07-live-collection-01/source-01 binds the implementation to SHA256 +a1ea8aa8ab265debeef983589f385eb8cf4e398327fd4ee22aaa0eb7af05a931. +Its initial and regression checks use the same frozen Rust/Python/manifest inputs, +above d04471d8e3e30f9d64f3d6a1ffacd6234bda0333. Fetched main remains +981a5417d9359d1f992e2dbf214eec7d221b6777, with zero commits behind. + +The shared host now owns one live extent collector across the serving and +independent archive-verifier inventories. Only complete startup reference +adoption constructs its inventory grants. Complete current/checkpoint graphs, +readers, staged receives and uncertain publications retain their pins. Disposable +empty workspace roots become transient without changing an existing retention. +The collector uses the existing control bank: a fixed 4 KiB service charge plus +checked temporary marking memory, with at most sixteen physical deletions per +turn. Idle inventories require no filesystem scan; last-reader release and new +object imports rearm collection. Marks include only locally present objects, +protecting dependencies imported under a pinned, incomplete receive. Physical +calls retain their inventory handles, working memory and shutdown-registry lease +until completion. Capacity refunds follow deletion and directory sync. Typed +advisory observations distinguish deferred admission, progress and stopped/failing +collection from durability proof. + +Initial checks pass: locked library compilation, six LTX apply cases, 36 LTX +store cases (one owned child-process helper ignored), the physical close test and +the RF3 pressure case. The close test holds a real deletion barrier, cancels a +public close waiter, verifies that storage and both admission charges remain, +then releases the barrier and joins closure with zero outstanding charge. +The RF3 case uses two groups/three independent cells and a 16-root ceiling in +both physical image roles. Eighty additional commands cross that bound while +retaining the original request deadlines and S3 minimum for the last write. +Duplicate receipt/witness/result checks and minimum-version reads from every +voter survive; obsolete serving objects drain below sixteen with healthy +collection observations. It passes in 107.03 seconds excluding compilation. + +The same-source seven-target native regression passes 254 cases with 29 explicit +backend/process skips: storage 15, archive/recovery 30, archive-controller 16, +execution 179, managed 10, host 3 and public SDK/native/Axum/restart 1. The public +process drill passes in 174.22 seconds, retaining eighty additional writes on +each side of restart and every original assertion. All three startup ownership +cases pass. All binary targets, formatting, layout and dependency checks pass. +The complete performance gate still fails: sixteen cases pass, including the new +collector pressure case, and two wider-group cases fail. Both stop during +32-cell-group enrollment after the observed manifest reaches 31 cells. The matrix +case returns ambiguous Runtime(DeadlineExceeded); isolated diagnostics returns +"deadline has elapsed" with a changed leader and the sampled member not ready. +The original request budgets remain unchanged. Occupancy 128 and the earlier +explicit-S3 failure point are not reached. Preserve this earlier refusal as well +as the source 23 S3 failures; neither cause nor scalable performance is qualified. +Strict library Clippy still fails with 222 diagnostics, including 202 dead_code. +Code/message/primary-file diagnostic keys match public-recovery 14 exactly, +without line offsets; nothing is suppressed or baseline-refreshed. Native-only +cellar/cellar-python compilation with the unavailable Python-runtime sentinel, +LTX feature-off/all-feature compilation, strict LTX library Clippy and the existing +control-service registry unit case pass. Final docs checks pass. + +The exact same-source archive executable passes the public recovery case against +real pinned RustFS in 24.47 seconds, in both image forms. RustFS enforces one CPU +and two GiB; its image, owner, mounts, loopback endpoint and resource limits are +checked, alongside an independent signed conditional-S3 probe. The fixture deletes +all source voter directories and the old verifier cache, restores complete images +and receipts, obtains fresh-quorum activation, preserves S3 minima/client floors, +rejects old tokens, serves new writes and survives retained restart. Its backend +and dedicated VM are stopped, retaining owned data and evidence. The test embeds +three native Nodes with independent domains/disks/TLS and uses a contract fencing +authority. It does not qualify permanent Docker launch revocation, separate voter +containers/hosts, managed recovery or sustained replication/reclamation load. +Both source-after comparisons have no changed inputs. Exact initial and broader +regression executables, failed sources/logs and the recovery artifact are retained +in the checkout-private qualification directory. Original deadlines, quotas, +stack and optimization profiles remain unchanged. This does not collect manifest pages, +authorize Raft log truncation or bypass offline engine reclamation admission. +It does not qualify sustained load, current-source Docker permanent fencing, +managed S3 learner activation or the default-backend cutover. CG-07 stays +IN_PROGRESS, CG-09–16 TODO and PR #160 draft. + +## CG-07 shared consumer and retention checkpoint + +Capture CG-07-perf-resume-01/source-inputs-25 binds 2710 Rust, Python, +manifest and profile inputs (2142 Rust files) to SHA256 +8c2d708074d6c265dd69fe9edb62694495998a517900700a1cc3e97e97eefb92. +Checks 25 and 26 finish with no changed inputs. The source includes fetched main +981a5417d9359d1f992e2dbf214eec7d221b6777, zero commits behind. + +The performance/fleet consumer port now models frozen canonical ranges with +occupancies 1–128, distinct logical histories and physical group witnesses, +explicit RF3 initialization/enrollment and actual physical resource observations. +Managed setup journals its standby/range/archive intents. Wider groups retain +the original replay bank by reducing tail batch sizes. Configured archive ceilings +are reported separately from the effective serial lane. These are implemented +adapters, not deployed managed or density/performance qualification. + +Image ownership now releases excess permanent host references. Exact terminal +command/enrollment reconciliation retires disposable previews while retaining +their pins through physical completion. A fresh applied image transfers its pin +into the complete manifest graph. Startup recovers every physical group's current +and checkpoint closures before releasing conservative filename retention, before +routes become reachable. Existing image retention, uncertain publication and +reader/checkpoint pins remain authoritative. Fresh extent preparation follows the +same pin ownership contract and preserves an existing root's retention. This +does not add a live extent collector or bypass offline reclamation admission. + +- All Cellar binary targets compile, clearing the original 27 performance errors. +- Seven native targets pass 254 cases with 29 explicit backend/process skips: + archive/recovery 30, archive-controller 16, shared execution 179, host 3, managed 10, + public SDK/native/Axum/restart 1 and storage 15. The public drill now adds 80 real + zero-delta commands on each side of retained restart, preserving all original + receipt, token, remote-read, business-value and joined-shutdown assertions and + the original 64-image bank. It passes in 135.69 seconds at this source. +- All three startup ownership/barrier unit cases pass. Full-library execution is + not rerun here; its prior retired inspection-protocol failure remains open. +- All six LTX apply cases pass at source 22; both changed LTX Rust files are + identical in source 25. The new regression keeps live readers during collection, + reclaims a fresh unpinned root and preserves exact existing host retention. +- Native-only cellar/cellar-python libraries, LTX feature-off and all-feature + compilation, formatting, layout, dependency isolation, docs and diff checks pass. +- Python tooling passes 282 cases with one environment skip. Its Python sources + are unchanged since that source 20 run. +- Strict library Clippy still fails with 222 diagnostics, including 202 dead_code. + Message/primary-file diagnostic keys exactly match public-recovery 14; no + diagnostic is suppressed or baseline refreshed. Plan drift and full CI remain. + +The new complete native performance gate is still **failing**: source 23 has +fifteen passes and two failures, both at logical cell 6 in the one-cell second +group's explicit S3 command alongside a 32-cell first group. The original +five-second command deadline returns an ambiguous DeadlineExceeded. Occupancy 128 +is not reached. Isolated instrumented 32/source 24 passes in 83.44 seconds, which +does not clear those failures. Its complete 32-cell checkpoint verification takes +roughly 4–5 seconds, and one full background job reaches 5.206 seconds. Preserve +those failures and investigate repeated verification and archive-lane interference +without raising deadlines, changing requested durability or removing assertions. + +No current-source Docker/RustFS, permanent launch-fencing, independent-host, +managed S3 learner, sustained load or release claim follows. The older executable +disaster operator still needs its whole-group port, disjoint identities, serial +catalog activation and explicit partial inventory. Dynamic managed archive +ownership, live extent collection, default-backend capability/deployment/read +policy and four original public gates remain open. See +[remaining priorities](remaining-work.md). CG-07 remains IN_PROGRESS, CG-09–16 TODO, +the goal active and PR #160 draft. Both requested MR basenames stay excluded and +the eight user diagrams remain unchanged and unstaged. + +## CG-07 combined public recovery qualification + +Runtime commit f644d9d69b24bc44024a88f33f80dc21fbffd3d0 contains the recovered +public-node startup and joined late-error cleanup. It includes fetched main +981a5417d9359d1f992e2dbf214eec7d221b6777, with zero commits behind. Final capture +CG-07-public-recovery-14 binds 2136 Rust inputs to manifest SHA256 +009e0283f389ca2b4c09220a0c2b708e9e673cd350fa4d1701c1281530033977. +Its Rust mapping exactly matches startup-13, which ran the new ownership unit +case and the full library. Documentation publication does not change that code. + +- Native archive/recovery: 30 passed, six backend cases ignored, 72.45 seconds. +- Managed Node lifecycle: ten passed, retaining all ten parent names. +- Archive controller: sixteen passed, twenty backend cases ignored. +- Host: three passed. Existing public SDK/native/Axum process/restart gate: one + passed. Counter build and locked native-only cellar/cellar-python library + compilation pass, including the unavailable Python-runtime sentinel. +- All three startup unit cases pass. Full library: 1062 passed, the same retired + inspection-protocol oracle failed, two Docker cases ignored. The new held-worker + regression first reproduced return-before-join, then passed with explicit + public-service closure before infrastructure teardown. +- Performance-tool Python: 277 passed and one environment case skipped. + Multi-Raft tool Python: ten passed and one Docker case skipped. +- Formatting, layout, dependency isolation, docs and diff checks pass. +- Strict Clippy remains 222 diagnostics, 202 dead_code, with unchanged diagnostic + keys. Plan checks remain 35 passed, one generated-index/Purpose mismatch and + one reviewed protocol.rs anchor error. Package commands still fail on the 27 + unconverted performance-runner errors; emitted integration artifacts are not + package-build successes. No failure was suppressed or baseline refreshed. + +Capture CG-07-public-recovery-rustfs-15 passes the exact public SDK recovery case +against real pinned RustFS in 19.70 seconds, covering both image forms. It uses +the same final archive executable and Rust mapping. The owned backend enforces +one CPU/two GiB, verifies its image, label and loopback endpoint, and passes an +independent signed S3/conditional-object probe. The test removes all source voter +directories and the old verifier cache, independently verifies the fixture fence, +restores complete images and receipts, obtains fresh quorum activation, preserves +S3 minima and client floors, rejects old tokens, serves new writes and survives +retained restart. The backend is stopped with its owned data volume retained. + +This is one test process embedding three real Nodes with separate domains/disks +and TLS. Its fencing authority is a contract fixture. It does not qualify the +Docker controller's permanent launch fence, separate voter processes/hosts, +managed disaster recovery, multi-group recovery orchestration or sustained +replication/reclamation load. Those original requirements remain open. No stack, +existing request deadline, admission quota or optimization profile was raised. +Earlier source/failures, exact executables and the unqualified consumer WIP are +preserved. Keep CG-07 IN_PROGRESS, CG-09–16 TODO and PR #160 draft. + +## CG-07 public recovered-node startup checkpoint + +Capture CG-07-public-recovery-11 binds 2136 Rust inputs (manifest SHA256 +8380c9d3a06e0c54eb7942589d1fb4c6799f40611a478707f2c7c18a0085e658). +The full native archive target passes 30 cases in 62.34 seconds; six explicit +RustFS cases remain ignored. All 36 inventory names remain. Managed ten, archive-controller sixteen (twenty backend cases ignored), host +three, public SDK one, counter build and native-only cellar/cellar-python library +checks also pass at this source. Five static checks pass. Strict Clippy retains +222 diagnostics, 202 dead_code, with unchanged keys. Plan checks retain their +known index/Purpose and reviewed-anchor failures. A subsequent startup cleanup regression reproduced return-before-join under a +held admitted control worker. The startup guard now retains the public service +and awaits its close before controller/runtime/SQL/log teardown. Successful +startup relinquishes that temporary clone. All three startup unit cases pass at +CG-07-public-recovery-startup-13, including the new held-worker/physical-lock +regression and both original cases. Its 2136-input manifest SHA256 is +4bb879b357ee3f1d0e406d0f73448a70d72c9de35ca44d731ef5dc871a217441. +The full library executes 1065 cases: 1062 pass, the same retired inspection +protocol oracle fails, and two Docker cases remain ignored. All original unit +names remain; the new cleanup case adds one. Final combined component checks +must rebind this cleanup correction before publication. + +The public `RaftCellsServices::recovery` capability resumes an already seeded +unmanaged log. Startup finds the original request through the selected birth, +immutable genesis and catalog binding, independently verifies the infrastructure +fence and checks the exact local seed receipt/engine/disk. Missing authority or +rejected verification refuses. Managed recovery cannot use this unmanaged lane. +One node-owned control service uses the existing control bank and original +queued permit deadlines to request actual Raft activation, reconcile its exact +catalog intent and join entered work before consensus/storage shutdown. Startup +can return pending; that supplies no serving or quorum authority. Retained +restart verifies the original fence and pins again. Exact activation receipts +remain necessary when another recovery owns the namespace lock. + +The added public Node case runs materialized and extent images after removing +every old voter directory and the old verification cache. It restores archived +SQL state, original receipt bytes and S3 minima, client floors and fresh logical +histories. It verifies independent cell state, actual group identity/occupancy, +archive-required policy and one actual group per voter. Old valid tokens refuse +against the new incarnation; new writes use it and duplicate calls do not rerun +the callback. A retained restart repeats serving checks. A held independent +verification call proves shutdown joins that entered recovery work. Its authority +is a contract fixture; this is not Docker permanent-fencing qualification. + +Broader archive verification exposed fixture contract drift. Reopened recovery +and outage gateways now preserve the full immutable namespace configuration, +including image mode. The strict no-selected-archive assertion now expects the +exact Unconfirmed contract introduced by f50b99af rather than the retired Invalid +string; it still rejects success and all other failures. The RPC fixture now +supplies a sibling namespace scratch base instead of nesting it inside a cell's +disposable SQLite directory. The original strict cleanup refusal stays intact. +The outage case now reports an early command result instead of masking it behind +a settlement wait; all original outage/deadline, callback, capacity, archive and +reclamation assertions remain. Earlier failures and executables are retained. +The performance metric inventory also includes transport_query_reply_write; +the performance-tool campaign has 277 passed and one environment skip; +the multi-Raft tool campaign has ten passed and one Docker skip. + +This does not complete CG-07, its four missing original public gates or the +combined MVP. Package builds still fail on the 27 unconverted performance-runner +errors. The larger executable consumer port is preserved as unqualified work +outside the checkout, with its original files and diff, for the next slice. +Current-source Docker/RustFS recovery/load and full release qualification remain +open. The full library run retains the known inspection-oracle failure. See the ignored public_recovery_work/perf_cutover_work continuation +entries for exact captures, backups and any live handles. + +## CG-07 public physical-group controls checkpoint + +Capture CG-07-group-controls-02 binds 2133 Rust inputs (manifest SHA256 +09f5447c42949fb573c90a8d28fc0d5fc7ecedaa3bf10efbaef705d6c19029cd). +All ten managed Node cases pass in 118.40 seconds; all nine parent names remain. +Shared execution passes 179 cases with the two original owned child fixtures +ignored, retaining all 181 names. Host three, public SDK one, counter build, +locked native-only cellar/cellar-python compilation and five static checks pass. +The public process drill retains native/Axum ingress and restart and now checks +metadata from all three actual physical owners as well as resource counts. + +The SDK exposes local physical-group observation with identity, role, membership +revision, persisted/committed/applied positions, installed checkpoint, retained +entry count and applied cell occupancy. One owner turn and one short durable-cache +read produce the sample. Entry counts do not estimate disk bytes; these samples +grant no quorum or serving authority. Archive-controller publication is sampled +separately and cannot supply retained physical coverage or compaction permission. +No per-cell Raft owner, worker or admission bank was added. + +Guarded leadership transfer retains the caller's exact term, membership revision +and target. The deadline starts at invocation and the actual owner checks it +after queuing, before mutation. Success establishes admission, not election; +expiry or close after dispatch does not establish whether a handoff was admitted. +Both image forms pass stale term/revision, invalid target, timeout bounds, +unpolled expiry and closed-owner refusals, followed by a real election, duplicate +receipts, old-token reads and new writes through the new leader. A held real +engine sync in the existing membership-floor case proves queued expiry refuses +the transfer while retaining the original leader. All original assertions remain. + +The first capture retains two new private-method diagnostics and the explicitly +interrupted failed compiler. The corrected source completes its checks without +internal-test cfg or a raised timeout, quota, stack or optimization setting. +Package commands still fail on exactly 27 unported performance-runner errors; +emission and direct execution of an integration artifact is not a package pass. +Strict Clippy remains 222 diagnostics, 202 dead_code, with unchanged diagnostic +keys. The plan suite remains 35 passed, one index/Purpose failure and one reviewed +anchor error; all seven recorded inputs of those failures match 824e0a20. +No baseline or assertion was relaxed. Full-library and real RustFS qualification +were not rerun in this checkpoint. The dedicated RustFS VM is currently stopped. + +CG-07 stays IN_PROGRESS and PR #160 stays draft. Next port the original executable +consumers and remaining public lifecycle/ingress/occupancy/feature-off gates, +including failure/cancellation and retained nonvoting inventory. Then finish +exact-reference S3 learner activation/archive holds and refresh same-source +RustFS recovery plus bounded replication/reclamation load. These local TLS/SQLite +fixtures do not establish independent-host capacity, sustained load or release. + +## CG-07 managed maintenance checkpoint + +The follow-up on 004d1051 implements and locally qualifies catalog-owned +maintenance for repaired cell groups. Capture CG-07-managed-maintenance-16 binds +2130 Rust inputs (manifest SHA256 +55e9323085d42f58d0fa77d5eb4b806eb8412e3528c7e129aab14a74b04b789d). +All nine managed Node cases pass in 102.06 seconds; all eight earlier names remain. +The new case runs both initial and joined replacement members in both image forms. + +Each cycle starts with an actual resource-backed peer repair and settlement +reconciliation, then moves the real target leader to a survivor. Rooted readiness +covers one physical group containing two independent cells. The survivor leader +originates the owned stop. Actual Node join and signed closure precede offline +confirmation; the surviving quorum accepts writes while that member is offline. +Canonical retained startup preserves the engine and both images while admission +stays paused. Old-boot stop and same-new-boot wrong-child resume requests refuse; +the latter preserves PayloadConflict, the root and paused admission. Fresh owned +resume proof, capacity/disruption settlement and actual leadership on the returned +member precede duplicate-receipt, old-token and new-write verification. A subsequent +maintenance intent proves the disruption slot was released. + +Shared fleet metadata now samples actual RawNode leadership across the complete +physical roster, with at most sixteen owners per turn. Roster revision, physical +population and the original deadline bound the scan. Unknown, failed or incomplete +owners supply no headroom. Counts describe groups independently of cell occupancy. +Five private unit cases cover bounded complete scanning and invalid/changed cuts. + +Paused retained startup now requires the exact current returned-boot lineage, +Live history, settled voting member, retained local state and pre-existing permanent +claim. Initial-member return uses read-only claim reobservation and existing group +state; it cannot create genesis or a missing claim. Joined startup retains its seed +receipt and claim checks. The directory rechecks the synced quiescence guard after +catalog I/O. Generic new-group and learner startup gates remain in force. Temporary +maintenance refusal is captured under the registration barrier and exposed as +Unavailable without submission by that invocation. Effect scheduling defers that +captured refusal even if resume races with a later flag read; permanent close +retains its distinct cause and prior mutation uncertainty remains sticky. + +The regression with a stop submitted by the survivor leader exposed a metadata dependency cycle: the stop RPC, +its matched-progress action back to the survivor leader, and the leader's fresh +inspection require three nested slots in the same two-slot peer bank. Resume has +the same dependency. A new peer maintenance stop/resume first validates exact +rooted authority/payload, then queues the existing bounded Node worker and returns +Unconfirmed. Physical transport completion releases its metadata before the +nested proof finishes. The worker revalidates context, gathers fresh proof and +syncs the exact guard; a later exact retry can observe that effect. Queueing grants +no acceptance, closure, serving or capacity. Local and non-maintenance calls keep +their original behavior. No deadline, quota, stack or promotion check was raised. + +Failed captures retain missing leader load, retained-return refusal, unlocalized +timeouts, temporary refusal reported as Closed, and the nested proof overload. +The first queued-proof run also preserves an aggressive uncertain-resubmission +storm; the final case uses the earlier physical join/signed-closure oracle, +retrying submission overload rather than competing with its admitted worker. +The pre-fix source08 join oracle fails and the final source passes with the same +shutdown deadline. Temporary production diagnostics are absent from the final +source. Source15 also passes all nine cases before the final typed-refusal check. + +The normal unit build and the five scan, three admission, 28 fleet catalog, +six fleet-action wire and twenty query-wire cases pass. The fleet selection has +thirty names; its two existing Docker Engine cases remain ignored without their +explicit owned harness. Earlier prose counted selected cases as passed; the +preserved logs show 28 passed and two ignored. Broader final-source qualification +is terminal: all 23 catalog and 179 shared execution cases pass, with the two +original owned process fixtures ignored. All 181 execution names remain. Full +library is 1061 passed, one retired inspection-protocol failure and two ignored +Docker Engine cases; all 1059 parent names remain and the five scan tests are +added. Three host cases, the original public SDK case, counter/effects builds and +the retained effects smoke pass. The effects smoke uses the SQLite development +object store. Locked native-only cellar/cellar-python library compilation and all +five static checks pass. + +Package builds still fail only on 27 unconverted performance-runner errors. +Strict library Clippy retains 222 diagnostics, 202 dead_code, with no added or +removed code/message/primary-file keys. The plan suite remains 35 passed, one +index/Purpose failure and one reviewed source-anchor error. All seven recorded +inputs of those two failures are byte-identical to 004d1051; no baseline or +assertion was relaxed. Profiles remain opt-level zero with debug assertions and +overflow checks, without internal-test cfg. Source, exact binaries, inventories, +failures and checks are preserved in the capture and its workspace-volume backup. +The earlier full suites below retain their own source boundaries. +This remains in-process TLS/SQLite qualification with a conditional fixture +catalog and explicit fixture deployment attestations. The original public lifecycle +and failure/cancellation gate, consumer/build/lint gates, exact S3 learner activation, +current-source RustFS recovery/load, optimized CI and complete feature matrix remain +required. CG-07 stays IN_PROGRESS, CG-09–16 stay TODO and PR #160 remains draft. + +## CG-07 managed peer replacement and drain checkpoint + +The follow-up on e8f9352d adds a four-Node managed lifecycle case and exposes +`CellGroupControllerError` beside `CellGroupStartupError` through the public SDK. +Callers can match the nested refusal without a verification import. Provider I/O +during terminal settlement now returns Unconfirmed, because the CAS may have committed before its confirmation read failed. Corruption and +authority errors retain their typed failures. Deadlines, resource ceilings and +original test assertions are unchanged. The cold cluster/workload helper is shared with the existing retained-start case. + +Capture CG-07-managed-lifecycle-09 binds 2127 Rust inputs (manifest SHA256 +f699ac38509185457b41feeb04d8a51e6cad8ece1bf0021e260b254ce7ca2151). +All eight managed cases pass in 48.58 seconds; the seven original names remain. In both image forms, a capacity-qualified spare +receives a two-cell group through authenticated peer seeding, actual RawNode +catch-up, joint consensus and terminal reservation settlement. A lost terminal +reply reconciles through its permanent receipt. The removed member remains in +physical inventory with its completed permit sequence and both retained cell +images. Settlement grants no disk reclamation. + +The actual leader's Handoff shutdown reports one accepted/released group and no +unavailable groups. Its surviving quorum accepts new writes. All three surviving +disks then reopen through canonical initial/joined-member startup, preserving +exact duplicate receipts, old-token reads, independent images and new writes. +Reusing the old-boot learner permit refuses before submission without changing +the fleet root. Cleanup joins entered owners even when replacement fails. + +Initial captures preserve Seeding Unconfirmed, the missing SDK error export, +the inventory probe before rooted directory acknowledgement and the actual +post-CAS confirmation loss escaping as Storage. The final case drops that +terminal CAS reply and its confirmation read, observes Unconfirmed, and proves +receipt reconciliation from a new invocation. Temporary diagnosis found `Unconfirmed(NotReady)` at promotion while the learner +caught up; no seed failure or relaxed promotion check was required. Bounded +fresh-cut reconciliation retains the same permit and debit. Diagnostic-only +production edits are absent from the final source. + +The normal library build, 28 fleet catalog unit tests, six fleet-action wire +tests, locked native-only cellar/cellar-python library check and all five static +checks pass. Strict library Clippy retains the same 222 diagnostics, 202 dead_code, +with no added or removed code/message/primary-file keys. Package compilation +still fails on the same 27 performance-runner errors. The plan-validator suite +has 35 passed, one failure and one error: the generated index/Purpose mismatch +and reviewed source-anchor drift. Every relevant plan file and the first drifting +source file is byte-identical to e8f9352d; neither baseline nor assertion was +weakened. These checks remain failing gates. The earlier complete execution, +public SDK and full-library results retain their e8f9352d source boundary. + +This is in-process loopback qualification with real TLS and SQLite, a conditional +fixture catalog and explicit fixture deployment attestations. It is not real +RustFS, independent-host, load, optimized CI or the complete public lifecycle +gate. Catalog maintenance stop/return/resume, consumer/build/lint gates, managed +S3 learner activation and current-source RustFS remain required. CG-07 stays +IN_PROGRESS, CG-09–16 stay TODO and PR #160 remains draft. Final static/lint +results and publication are retained in this capture's final-checks directory +and the ignored managed_lifecycle_work continuation. + +## CG-07 cold managed group qualification checkpoint + +The follow-up on b9be4de7 adds an explicit standby directory genesis with its +permanent pin, immutable journal and operation index. It proves zero selected +groups before capacity admission; missing metadata still refuses. First range +selection requires rooted birth reservations and an exact standby-root CAS. +Selected ranges cannot return to standby. Missing tree fields reject. Import +freeze journals its head transition, including a real zero-group inventory. +Fleet birth identity reservation and materialization enter their owning execution +domain; public callers need not provide one. + +Initial placement now revalidates every voter's current opaque capacity receipt. +The direct post-restart regression fails before this guard: an Active descriptor +admits a new birth using stale capacity, advancing root revision 19 to 20. The +fixed path refuses without a root change; actual new-boot qualification and +activation then let the identical proposal succeed. Its unselected reservation +remains inventoried. Existing group ownership and serving survive readmission. + +Capture CG-07-cold-managed-group-06 binds 2126 Rust inputs (manifest SHA256 +4f5a778a4c2bed7db7fd9bc0c30bc4f2ee962341e2fce08a03059fe01381ff01). +All seven actual-Node managed cases pass. In both image forms, three registered standby nodes qualify +and activate capacity, reserve/select one group, reach RF3 quorum and enroll two +independent SQLite cells. Canonical retained startup, exact duplicate results, +old-token reads and new quorum writes pass. Deployment resource bounds are explicit +fixture attestations, not kernel measurements. The original missing-pin failure, +import head/journal disagreement, caller-domain backtrace, fixture compilation +errors and stale-capacity admission remain preserved with their exact sources. + +The final broader chain is terminal: all 23 catalog and 179 shared execution cases +pass, with two owned process fixtures intentionally ignored. Library builds; all +twenty wire cases pass. Full library is 1056 passed, the same retired inspection +protocol failure, and two ignored. All 1058 earlier names remain and the strict +missing-tree parser test is added. Three host cases, the original public SDK +counter drill, counter/effects builds and retained effects smoke pass. The effects +smoke uses the SQLite development object store. Locked native-only library +compilation, format, layout, native dependencies, docs and diff checks pass. + +Package builds still exit 101 on 27 unconverted performance-binary errors despite +emitting current native executables. Strict Clippy fails with 222 diagnostics, +including 202 dead_code. These are failing merge gates, not package passes. +Profiles remain local opt-level zero with debug assertions/overflow checks and +without internal test cfg, raised timeouts/capacities/stacks or lint suppressions. +No optimized CI, complete feature matrix or current RustFS/load claim follows. + +This qualifies local cold initial-group/ordinary-retained startup. Joined-member +startup, repair, drain, maintenance, stop/return/resume and physical disruption +accounting still need the original public lifecycle gate. Exact-reference S3 +learner activation, remaining consumers/public gates, current-source RustFS +recovery/load and CG-09–16 remain required; CG-07 and the full goal stay incomplete. + +## CG-07 final shared ownership qualification checkpoint + +Source-06 preserves the finite shared membership fence and physical send/hint +fix from source-04. It additionally checks authorization node/disk against log +options before namespace/engine opening, and validates the immutable node, disk, +label and server-name descriptor before permanent node pins or boot mutations. +Advertised addresses remain explicit and may differ from listener bind addresses. +The existing registration matcher reuses that descriptor predicate while keeping +its independent pinned-process/transport permission requirement. + +Both regressions fail on source-05: a wildcard listener with a loopback advertised +endpoint is rejected, and a mismatched log owner consumes a rooted boot before +startup refuses it. On source-06 all five actual-Node managed cases pass, including +both image forms, fresh/retained ordinary boots, recreated-engine refusal, +foreign-boot stop refusal, declaration and owner/descriptor mismatch before a +rooted boot mutation, explicit advertised endpoints, periodic trust refresh and +shutdown joining an entered catalog read. The membership selection remains nine +passed, zero failed and one intentionally ignored owned child fixture. +Source manifest SHA256: +aa7327d2a3ae4f1244264cb6d59a179e250c7ab24119229efd638a966d080c1c. + +Source-04's full shared execution suite passes 179 cases with two owned child +fixtures ignored. Normal library builds; twenty wire cases pass; full library is +1055 passed, one retired-protocol inspection failure and two ignored. All three +host cases, counter build/original public SDK process drill, effects build/original +retained smoke and five static checks pass. Strict Clippy reports 222 diagnostics, +including 202 dead_code, down from 234/214. Three newly grouped messages contain +only residual methods from older groups; no method becomes newly unused. Package +builds still fail on 27 unconverted performance-binary errors. Source-06 changes +only the four recorded owner/endpoint/test files from that broad checkpoint. +Its complete broader chain is now terminal: 179 shared execution cases pass with +two owned child fixtures ignored; all earlier 181 names remain. Normal library +builds, all twenty wire cases pass, and full library is 1055 passed, the same +retired inspection-protocol failure, and two ignored. All three host cases, +counter build/original public SDK process drill, effects build/original retained +smoke, and all five static checks pass. Strict Clippy is unchanged from source-04: +222 diagnostics, including 202 dead_code. Component Cargo commands still exit101 +on the same 27 unconverted performance errors despite emitting fresh executables. +The failed source-05 regressions and source-03 socket repeats remain retained. +The locked native-only library check also passes for cellar and cellar-python +with default features disabled and PYO3_PYTHON pointing to a nonexistent runtime. +This is a compile check, not the complete feature-off behavioral matrix. + +This qualifies the recorded local ownership boundaries, not the full managed +lifecycle, optimized CI matrix, current RustFS backend or a release. + +## CG-07 shared membership and managed ownership implementation + +The follow-up on 2e400133 implements the shared owner's configuration send +fence and canonical managed boot/directory/Node stop ownership. Capture +CG-07-managed-fence-04 binds 2125 Rust inputs (manifest SHA256 +87c26e4e7757623a41454c57005a3611c0e1539c512fdf7894af39da8664533d). The merged origin/main remains +981a5417d9359d1f992e2dbf214eec7d221b6777; the preceding published checkpoint and +its independent qualification are recorded below. + +The original new shared regression fails before the fix: configuration Apply +overtakes an owned prior runtime message. The shared owner now includes one +pending Ready, freezes that finite boundary, joins prior physical sends, emits +an exact persisted-term/index commit announcement and joins its deliveries before +membership publication. Later heartbeat work cannot keep extending the boundary. +The runtime's exact output admission survives a dropped Delivery waiter through +ordinary SendRequest and checkpoint producer/transfer ownership. + +The socket regression then reproduced an ownership cycle twice: a completed +failed send retained its output permit while awaiting an unreachable hint; that +hint waited for an idle owner, which was blocked by the configuration fence. +Source-04 ends the send fence after physical exchange and before advisory +feedback. A checkpoint's independent producer retains its own admission through +actual source exit. Transport metadata remains charged until its task exits. +The held-event and entered-socket cases pass in both image forms, inspect both +terminal commit announcements, and elect/apply another group while the first +is fenced. Original joint-membership, sync, follower-quorum and crash cases pass. +The selected membership result is nine passed, zero failed and one intentionally +ignored owned child fixture, executed by the crash cases. + +Managed startup validates the actual compiled/declaration/rooted capability set, +pins the physical engine, binds an ordinary or authorized maintenance boot, +syncs its exact lifecycle guard, then publishes endpoints and attaches through +managed permits. Retained route reconstruction uses durable current membership; +joined members use the existing seed-receipt/permanent-claim permit. A declared +standby may precede first range publication without fabricating an empty root. +One separately budgeted directory service owns timed rooted reads and bounded +peer batches, preserves its original deadlines, closes admission on superseded +local authority and joins entered work on shutdown. Internal observers do not +retain the last external SDK owner. Node receives its existing stop services from +the actual bound Core authorization and probe signer rather than None. + +All three new actual-Node managed component cases pass: both image forms complete +fresh/retained ordinary boots, refuse a recreated pinned engine before consuming +a boot, and reject a foreign boot through the installed Node stop reporter. +Declaration mismatch refuses before a rooted boot mutation. Independent timed +trust refresh and shutdown joining an entered catalog read also pass. This does +not qualify nonempty managed startup/repair/maintenance, capacity activation, +complete inventory or the original managed_group_lifecycle public gate. + +Production cargo check passes. The source-04 broader checks are complete as +recorded above; source-06 repeats them after the additional owner/endpoint +regressions. The newest checkpoint +governs the exact final source and validation boundary. The component Cargo builds +still emit their current executables but fail the package gate on 27 unconverted +performance-binary errors. Full public consumers, exact S3 learner activation, +combined-source RustFS recovery/load and CG-09–16 remain required. Failed initial +schema/endpoint/group-event fixture attempts and the physical ownership regression +are retained, with no timeout/capacity/stack increases or production suppressions. + +## CG-07 current-main qualification checkpoint + +Counter checkpoint 41ab8b4d is committed with the source-02 qualification +below. While its checks ran, origin/main advanced to +981a5417d9359d1f992e2dbf214eec7d221b6777 (PR #165). Merge 9df1389f includes that +main plus the counter work. The three conflicts preserve the shared-only host +configuration surface, upstream incompressible seed data with CellHistory +callbacks, and every exact metrics assertion. The predecessor host dispatcher is +not restored. Upstream shutdown, snapshot, pool and qualification changes remain. + +Merged-source capture CG-07-public-counter-03 binds Rust inputs to merge +9df1389f (source manifest SHA256 +32adfacc6ad54f9760928487e2b5ab018c1cbef5c3a691792f42db89f86dc8b0). +The counter builds and the original sdk_many_cells_share_group public case passes +with the actual rebuilt counter (SHA256 +a5ebebc8086dcc17892bf706962a2539cb08589b20ebeabef58a581ba95453d8). +All three replicas observe one physical group and two independent cells. The +normal library builds: all twenty query-wire cases pass; full library has +1055 passed, one failed and two ignored. All 1052 earlier test names remain, +with six upstream additions. The sole failure is the retired-protocol inspection +cancellation fixture. All three namespace-host cases pass across both image +forms; the effects build and original two-process retained smoke pass using the +SQLite development object store. Format, layout, native dependencies, docs and +diff checks pass. Normal local profiles use opt-level zero and no internal test +cfg; this does not qualify the optimized CI campaign or feature matrix. + +The public/host Cargo commands emit fresh component executables but still exit +101 on 27 unconverted performance-binary errors. They are not package passes. +Strict library Clippy fails with 234 diagnostics, including 214 dead_code. +The one changed diagnostic reflects upstream renaming targets to yield_for_seed +in snapshot transport; the remaining code/message/primary-file set is unchanged. +All owned qualification jobs are terminal. Keep +PR #160 a draft: the complete public target, managed orchestration, combined-source +RustFS recovery/load and CG-09–16 remain unqualified. + +The upstream membership send-fence fix changes the retained legacy Replica. +Audit/reproduce its applicability to CellGroupOwner before qualifying managed +membership transitions: that owner has a separate Ready/effect/transport lifetime. +Merging the old Replica fix alone does not establish the shared owner's finite +configuration fence or final commit delivery to a removed peer. Preserve all +original deadlines, ownership reservations and destination/catalog authority. + +## CG-07 public counter and provisioning in progress + +Counter checkpoint 41ab8b4d on fc4a8d31 ports the counter from two per-cell registrations to +one physical group with independent logical histories, SQLite state and receipts. +The node's log/runtime/controller capacities are explicitly one group. Fresh +public namespace/archive provisioning persists the exact archive intent before +creating counters; retained setup opens only existing metadata. Missing catalog +with retained voter disks refuses rather than reconstructing a namespace. +The public provisioning facade reuses the host's immutable range initializer; +it exposes no legacy catalog or verification-only interface and grants no serving +or activation authority. + +The HTTP adapter uses CellVersionToken, one original query deadline/attempt bank, +explicit one/two ingress authorization, full-process/checked-sequence query IDs, +and actual replica metadata. Logical version fields and physical commit witness +fields stay separate. Typed application refusals remain accessible without +verification imports, and ambiguous command errors retain the original ID plus +cause. Node drain precedes the unchanged final quiescence predicate. The original +counter smoke and its assertions/timeouts are unchanged. + +The locked counter build passes at CG-07-public-counter-01. Three original smoke +drills pass, each forcing late startup failure and then completing both retained +process phases on all six HTTP endpoints. All replicas report one controller +group and two manifest cells. One immediate repeat cannot bind the fixture's +non-reusing port before process startup; no listener remains. The retained repeat +wrapper checks the original documented free-port precondition, observes 29.10 +seconds of socket cleanup, then passes both repeats without altering the smoke. + +An additional real native/Axum contract run passes 581 requests, including +seventeen invalid option sets per endpoint, duplicate recognized headers, +authentication, the fixed cell allowlist, forged/cross-cell minima, contract +validation before duplicate lookup, typed conflicts/sequence gaps and unchanged +cell values/receipts. This covers counter ingress; admin and forged tenant/group +acceptance remain unqualified. A separate missing-catalog refusal with three +copied voter disks returns before creating a catalog or archive intent; every +voter file hash is unchanged. This is a startup guard test, not disaster recovery. +Counter executable SHA256: +a6af26253a601a38298c29568fc008e2be92fa3c254e5b139ff4d0ccce5a6ad5. + +Source-02 creates native_cell_groups_public with the original +sdk_many_cells_share_group case. The emitted fresh executable passes that exact +case, retaining its process evidence and observed group/cell inventory. The +remaining four original public cases are not stubbed or marked passed. Source-02 +changes only this new integration target's Rust inputs relative to source-01; +production/example Rust is identical. Source manifest SHA256: +e93ddd25b82771d7f1c8199567033eaa4cb734de95cec3c160161bd386f13104. +Public test executable SHA256: +9b2987f67f9fca1515311da94c7f3520eef3a2bb44d170bd6e6983b376765825. +The overall Cargo command still exits 101 on twenty-seven unconverted performance +binary errors; a fresh component executable is not a package pass. + +The normal library target builds; all twenty wire cases pass. Full library: +1049 passed, one failed, two ignored. The sole failure remains +inspection_revocation_and_process_refresh_cancel_an_admitted_inbound_call, whose +fixture still sends the retired per-cell inspection protocol. All three current +namespace-host cases pass across both image forms. The effects target builds and +its original two-run retained smoke passes with the SQLite development store. +Format, layout, dependency, documentation and diff checks pass. Strict library +Clippy still reports 234 diagnostics, including 214 dead_code; code/message/primary +file comparison finds no added or resolved diagnostic against host-fixed-09. +No suppression, assertion weakening or timeout/capacity/stack increase is used to +obtain a library/package pass. + +No current combined-source RustFS, full public target or package qualification is +claimed. The next feature dependency is canonical managed startup: registration +still refuses managed namespaces/node-directory options, Replica attachment uses +only unmanaged bootstrap, and Node startup returns no Fleet stop services. Wire +those owners to the existing exact-boot/group authorizations and synced lifecycle +guard before qualifying managed/performance consumers. Keep every admission guard +until its real replacement is active. Original public fixture ports, remaining +consumers, S3 learner activation and all CG-09–16 scope remain required. + +## CG-07 bounded replica routing checkpoint + +The shared controller now owns bounded public replica preferences: one to eight +explicit candidates, per-attempt deadlines and an optional untried observed leader +fallback all share one invocation, fingerprint, input, token, global deadline and +attempt bank. Incoming RPCs execute locally. Authenticated leader advice supplies +no authority; every destination retains namespace/process grants and exact +cell/group admission. Remote responses expose the actual authenticated replica. +Wrong-history/schema minima fail preflight before destination retries. + +Expired physical SQL retains the original dispatcher and controller admission +until its callback exits. Outbound work retains dispatcher admission through the +physical exchange, and a successful remote result retains its transport charges. +Releasing a waiter cannot refund resources still owned by physical work. + +The new routing fixture reproduced a timer race: SQL reports its inner deadline +as Execution(Storage(RuntimeError::DeadlineExceeded)), while the invocation's +outer timer reports a runtime deadline. Fixed-06 normalizes only that exact typed +SQL deadline at dispatch; ordinary storage errors stay fatal. Probe-05's failure +and exact source/executable are retained. No timeout or capacity was increased. +The new fixture then exposed an early cleanup observation; fixed-07 waits for +actual joined transport/controller cleanup before requiring exactly zero. Held +physical callbacks still require their exact item count and charged bytes. + +Fixed-07 passes nineteen focused cases and forty routing repetitions, twenty per +image form. Its full suite has 175 passed, two failed and two ignored: original +cancellation oracles expected metadata admission to vanish after waiter abort +while SQL still ran. Fixed-08 preserves their SQL, cancellation and blocked-close +checks, requires the exact pre-abort reservation while SQL remains held, and +requires zero after joined physical shutdown. Production/new routing cases are +unchanged. Fixed-08 passes twenty-one focused cases and the complete execution +suite: **177 passed, zero failed, two ignored**, in 406.47 seconds. Its source +manifest SHA256 is +0f65bed56b6632a843b0e3e3bc5cfac03ad4145dbc9b14d75844a48c7feaf2e6; +native executable SHA256 is +b35cdf1d25e57d34476d2e27a1fd4a0944c781996f7e05014b31d94458025462. +Fixed-07's forty-repeat evidence applies to the unchanged production and cases. + +The normal unit target builds; all twenty query-wire cases pass. The full library +has **1049 passed, one failed, two ignored**. The remaining failure is the old +managed inspection fixture using its retired per-cell protocol. Host-fixed-09 +adds the owning raft::prelude::Entry import to a shared integration fixture; all +three namespace-host cases pass, including both image forms. The native Cargo +command still exits 101 on unconverted counter/performance APIs despite emitting +the fresh test executable. This is component qualification, not a package pass. +Strict library Clippy retains 234 diagnostics, including 214 dead_code; comparison +of code/message/primary file finds no additional diagnostic against the previous +shared-read checkpoint. No lint suppression is added. + +Effects qualification retains baseline/current failures rather than attributing +them to routing. Three baseline/current comparisons capture definite-unsubmitted +NotReady after failover in both executables, plus a current incomplete-shutdown +report after the complete business result. Probe-10 samples unchanged close +semantics: SQL is idle, no groups failed, but one call can remain and the outcome +is Unobserved. Readiness-fixed-11 retries only definite-unsubmitted NotReady or +NotLeader with the same command ID/input and original deadline; five of six +fresh smoke trials pass both retained runs, while one still completes the +business work then refuses shutdown quiescence. This isolates close ordering +from failover readiness. + +Drain-fixed-12 lets Node seal admission and drain its producers before inspecting +the final joined Raft report. The exact NoUnresolvedLocalProducers/no-failed-group +predicate remains; unknown mutation outcomes are never accepted as quiescence. +All temporary close diagnostics are removed. Its effects build passes, followed +by all six original smoke trials: twelve successful process runs including a +retained-disk restart per trial, with unchanged assertions and deadlines. Its +executable SHA256 is +1b86a52b35b3d6c38abb808430eeca2f053a6d5b0f22d7e249abf85caccc2297. +Its source manifest is +af29d82f0446507a4f4a089d5806a9e39537af065fd79460acba2a7a77f26f2e. +Relative to fixed-08, Rust changes are confined to the owning test import and the +effects example; production/library Rust is identical. The effect store here is +the SQLite development backend, not RustFS qualification. + +PR #160 remains a draft. Public SDK/consumer ports, managed startup, exact-reference +S3 learner activation, all five original public acceptance gates, refreshed +combined-source RustFS/DR/load gates and CG-09–16 remain required. See +[remaining work and priorities](remaining-work.md). Historical captures preserve +all earlier failures and their source/binary boundaries. + +## CG-07 shared reads and PR #160 MVP scope + +PR #160 publishes the canonical shared SDK/fleet work as a draft checkpoint. +[Remaining work and priorities](remaining-work.md) orders the MVP deliverables +and names the original public merge gates. CG-07 and the end-to-end goal remain +incomplete; all CG-09–16 requirements remain in scope. + +The counter port exposed missing public read behavior. Two real follower/SQLite +cases reproduce NotReady before production changes in +CG-07-shared-read-follower-before-01, in both image forms. Core-fixed-01 adds +origin-bound safe ReadIndex forwarding in a separate private context domain. +Selection requires current-term quorum and contiguous local apply through the +correlated barrier. Source generations revoke retained follower proofs on a +term/leader change, failure or shutdown. Seed and metadata contexts retain their +own admission rules; read proofs confer no mutation or seed authority. + +Recent-source reads now have an explicit age constraint in the SDK and wire. +Only valid current-term leader traffic refreshes follower source observations. +The captured expiry caps the physical SQL deadline, including entered callbacks; +later traffic cannot renew an old proof. Default SDK stale reads use the injected +host setting; disconnected historical reads require explicit selection. Results +expose the actual selecting member and node. Namespace/process grants, bounded +attempts, original invocation identity and joined physical cleanup remain enforced. + +The source-age fixture initially pumped fresh leader traffic while selecting its +first proof. Tagged probes reproduced that race; fixed-03 selects the non-barrier +proof directly, preserving every expiry assertion and deadline. Forty repetitions +pass, twenty per image form. Debug probes are removed. Fixed-04 adds actual TLS +follower queries with minimum-version, linearizable and recent-source constraints, +exact observations/responder metadata and transport-permit release. These are +three engines in one process, not independent-host qualification. + +Fixed-04's full execution run has 172 passed, one failed and two ignored. Its only +failure is an older router oracle expecting an authorized follower linearizable +read to refuse. Fixed-05 updates that oracle to check exact successful follower +and leader observations, callback counts and released permits, retaining namespace/ +process denial, global retry bounds and the router's absence of local replicas. +All fifteen focused cases pass on fixed-05. The complete native execution suite passes: 173 passed, zero failed and two +ignored, in 408.03 seconds on the normal default stack. +Source manifest SHA256: +64fa92f41bb3e29fb7873ebf0ae5a532ba4c007049c6e1ae47ffcb6039e2f565. + +Core-fixed-01's unit and effects targets build. Its full library suite has 1048 +passed, one failed and two ignored; the failed inspection-cancellation fixture +still sends the retired per-cell protocol. The original two-run effects smoke +passes using the SQLite development object store. Strict library Clippy fails +with 234 diagnostics, including 214 dead_code. Production and example Rust are +unchanged through fixed-05; only integration fixtures changed after core-fixed-01. +Format, layout, native dependency, documentation and diff checks pass at fixed-04; +publication checks cover the final documentation and router fixture separately. + +The overall fixed-05 native Cargo command fails on 34 unconverted performance- +binary API errors despite emitting the fresh native executable used above. Counter +and performance consumer ports, public replica routing, managed directory/startup, +exact-reference S3 learner activation and all five public acceptance cases remain +open. No full-package, strict Clippy, current-source managed RustFS/DR/load, release +or MVP pass is claimed. Historical real RustFS results retain their earlier scope. + +## CG-07 retained preparation and completed seed installation + +The preceding gateway/backend checkpoint (CG-07-gateway-backend-fixed-03) passes +all 21 targeted cases and has 164 passed, three failed and two ignored in the +full native execution suite. Its normal unit-test target and effects executable +build. The original two-run effects smoke and 39 focused lifecycle cases pass on +that same source. Format, layout, native dependency, documentation and diff checks +pass. This effects smoke uses the local SQLite-backed development object store; +it is not current RustFS or managed disaster-recovery qualification. + +The three remaining native failures are reproduced independently in +CG-07-seed-retained-probe-01. Tagged boundary probes show that seed persistence +succeeds before a post-sync source watch rejects completion in both image forms. +The retained-owner retry adds exactly one sync of the same page directory. These +probes have been removed and the original assertions remain unchanged. + +CG-07-seed-retained-fixed-01 separates source liveness before submission from +local completion after successful engine persistence. The queued engine still +checks the complete grant and exact managed boot before entering its fence. +After successful persistence, completion retains destination engine/member/disk/ +process identity, current boot, shutdown and original deadline checks, followed +by fresh catalog bootstrap. It no longer requires the source process to remain +alive after the bytes and receipt have been durably installed. This grants no +quorum or serving authority. Sync errors remain uncertain and fail closed. + +Provisioning retries now reuse an already verified physical closure only with +the same live page and engine owners, exact selected marker, complete checkpoint +retention and no pending/quarantined publication. A changed selection recovers +on the retained applier. Marker equality cannot clear uncertainty. The original +LOCK and no-additional-sync assertions remain. The seed persistence cases also +hold an entered fence through its unchanged original deadline and require a +refused late response while cold recovery preserves the actual synced root. + +Fixed-01 builds the normal unit target and effects executable; its original +retained-disk smoke passes both runs. The full ordinary library suite has 1048 +passed, one failed and two ignored. Its remaining inspection-cancellation test +still sends the retired per-cell protocol and must be ported to the actual shared +managed protocol. Format, layout, native dependency, docs and diff checks pass. +Strict library Clippy reports the same 235 diagnostics (215 dead_code); none have +a primary span in the six production files changed for these two fixes. + +The new expiry cut initially held work past the fixture peer's unchanged +five-second disconnect guard. Fixed-02 gives only that new case a two-second +operation deadline; all other cases retain their original ten-second deadline. +It passes retained-owner reuse and both complete seed-persistence cases, including +all queued/entered revocation, expiry, pre/post-sync error and cold-reopen cuts. +Production/library and example Rust match fixed-01 byte for byte; only the new +expiry fixture deadline changes. The complete native execution suite now passes: +167 passed, zero failed and two ignored, in 452.06 seconds on the normal default +stack. Its three focused regressions also pass. Capture source manifest SHA256: +1a60ff1d210b4f7c3432fabc6f4a669a88539d71271bf6faaed391f6e07c29d6. +The overall native Cargo command still fails on fifteen unconverted counter-binary +API errors even when it emits this fresh native test executable. No full-package, +strict Clippy, managed/RustFS, load or release pass is claimed. All owned jobs are +terminal; the WIP, source inputs, immutable executables and passing/failed evidence +are preserved in the checkout-specific target's qualification-evidence directory. +CG-07 and the end-to-end goal remain incomplete; CG-09 through CG-16 are required. + +The next critical MVP work is the public counter/performance and actual shared +managed inspection cutover, followed by canonical managed directory/startup and +exact-reference S3 learner activation. Preserve all original public acceptance +behaviors, then refresh real Docker-fenced RustFS recovery and replication/ +reclamation under load on the combined source before publication. + +## CG-07 canonical S3 scratch ownership and gateway backend declarations + +Canonical S3 enrollment's Image(Io(NotFound)) failure is now diagnosed and fixed. +The checkpoint receiver's anonymous materialized image spool required a parent +that canonical startup never created. An unchanged source-12 executable reproduced +it on fresh data. Creating only the three archive-spools parents made both original +smoke runs pass. A tagged image-reader probe emitted no failed-image path, and is +removed; cellar-ltx generation.rs is restored byte for byte to its prior source. +Canonical archive startup now creates and syncs this node-owned directory on its +execution domain before registering any archive owner. S3 proof, scratch quotas, +original timeouts and all physical receipt/fencing checks remain. + +The fix also exposed intermittent typed Execution(Busy) in the finite effects +example. Its caller now backs off only busy/overload under the same command ID, +contract, input and original deadline, including ambiguous outcomes. It does not +claim rollback or create a fresh business command. CG-07-canonical-s3-fixed-02 +builds the normal library unit target and effects binary. The original two-run +retained-disk smoke passes, and three additional original two-run trials pass: +eight retained runs total with executable SHA256 +4106a1a1c819f1335e62dffa91417646498bbf19821778baa8e18d69d3797995. +Capture source manifest SHA256: +231fa577c83a1d06ccfe1694e6e8a0e45ef5d259988684b9dc45b21f4e59cce1. +This is the public SDK/effects path with a SQLite-backed development object store, +not RustFS, managed fleet, zero-quorum recovery or a load envelope. All 39 prior +focused boundary cases also pass with the scratch fix (fixed-01); the subsequent +fixed-02 change is example retry policy only. + +The sixteen native gateway failures were caused by extent fixture setup declaring +a materialized namespace. Registered, custom-retention, effects and reclamation +fixtures now declare the actual backend at birth. Archive reopen forwards the +same immutable backend; a new fixture assertion refuses a contradictory setup. +The read-only RangeDirectory namespace configuration accessor is public for +embedding resources; it neither refreshes routing nor grants startup authority. +Production schema/backend/domain/runtime identity guards remain unchanged. + +A standalone original gateway case reproduces on the immutable source-11 binary. +CG-07-gateway-backend-fixed-01 emits a fresh native execution test artifact while +its overall Cargo command still fails on fifteen old counter binary API errors. +All sixteen original failing gateway cases now pass, retaining every original +assertion and both backend variants. Source manifest SHA256: +a0460dd39dc55b4401073c370110c64198ded0030dbfb40ece11ecb07018497e. +The later fixed-03 full shared execution suite has 164 passed, three failed and +two ignored; its 21 targeted cases pass. Retained-owner retry, seed revocation and the +old library inspection cancellation test remain open until new evidence proves +otherwise. No combined/full-package or strict Clippy pass is claimed. + +Next fix remaining owner/seed regressions, port public counter/performance callers +and managed shared-protocol cancellation, finish managed node-directory/startup +and exact-reference S3 learner activation, then all five original public gates. +Current-source real Docker-fenced RustFS qualification, persistent hold retirement, +CG-09 through CG-16 and combined load/release gates remain required. The full goal +is still incomplete and this WIP remains unpublished. + +## CG-07 predecessor retirement and executable qualification + +The unpublished cutover now compiles the production library and its ordinary +unit-test target. The previous 32 integration errors are resolved by removing +the predecessor per-cell host dispatcher, its managed birth/registration API +and inbound per-cell seed/inspection dispatch. Used host options, observation +contracts and storage-policy regression assertions remain. Unreachable +predecessor helpers still require the CG-16 deletion sweep; no warning +suppression or compatibility dispatcher was added. + +Canonical startup now joins physical infrastructure on explicit failure even +with retained caller handles. Cancellation schedules the same owned cleanup. +Peer/query routes are installed only after every local root attaches and the +retained inventory is checked. Both existing startup tests use real shared-group +engines, directory births and TLS peers. A default-stack overflow in permanent +claim revalidation was reproduced, traced through the macOS crash stack and +fixed by boxing catalog work before constructing nested timeout futures, with +an additional heap boundary around canonical initialization. Startup boxing +alone was insufficient (test-05); the catalog boundary fixes the original test +(test-07), and both original tests pass without diagnostic probes (test-11). +No stack limit, timeout, resource ceiling or assertion was relaxed. + +The existing poisoned-accounting regression requires close to return its +corruption error, retains old handles, and reopens the physical engine before +dropping them. This verifies release while preserving the corruption oracle. +The old storage-family tests do not prove managed startup or serving authority. + +The effects executable now builds with explicit namespace capabilities/group +placement, separate logical cell histories, enrollment initializers and public +request/error values. Development provisioning persists one-use archive intents +before committing counters; retained startup only opens them. A fresh harmless +S3 command after observed settlement replaces the advisory archive-position wait. +This executable is not yet qualified: its local SQLite-backed object-store smoke +first refused missing initializers (test-11), then, after that fix, failed S3 +enrollment during checkpoint capture with Image(Io(NotFound)) (test-12). +The failed run's voter disks, catalog, creation intents and logs are retained. +Do not reduce durability or recreate archive counters to pass this case. + +Current evidence: + +- CG-07-legacy-retirement-test-11 retains 4730 source inputs (manifest SHA256 + a48f8733addb9559a02e1b4e61398b7432edfc304393afb5034aeb731326aa70). + All 39 focused cases pass, including both startup tests, 30 host policy cases, + four low-level storage genesis, two route-lifetime and one query-auth case. +- The full ordinary library suite has 1048 passed, one failed and two ignored. + The failure sends the retired per-cell inspection protocol; its cancellation + oracle must be ported to the real shared managed protocol, not suppressed. +- A fresh native execution artifact was emitted before Cargo failed on 15 old-API + counter executable errors. The compiler event (fresh:false), source hashes and + immutable executable are retained. Its full suite has 148 passed, 19 failed and + two ignored. Sixteen failures stop at gateway Identity validation; the other + failures are retained-page-owner retry and seed persistence WrongIdentity in + both image modes. These are open regressions, not passing qualification. +- Test-12 changes only example initializers. Library and native-fixture Rust + sources match test-11 byte for byte. The normal locked unit build and effects + executable build pass. It retains 4730 inputs (manifest SHA256 + af3055a1e8692bf0a384f2c31d1baaf6a65a13c1a070c10f6701c47116b12e16). + Format, layout, native dependency, documentation and diff checks pass. + Strict library Clippy fails with 235 diagnostics (215 dead_code and 20 other + lints); the complete classification is in static/clippy-audit.json. No Clippy + or combined release pass is claimed. + +Next resolve canonical S3 enrollment's missing checkpoint image using the actual +smoke repro. Diagnose gateway identity without weakening namespace/backend/domain +checks; existing extent fixtures may advertise a materialized namespace. Preserve +all owner reuse and seed revocation assertions. Port the counter example to many +cells in one explicit group, then the remaining executable consumers. Complete +managed node-directory/startup integration, exact-reference S3 learner transfer +and all five original CG-07 public gates. Refresh real Docker-fenced RustFS +qualification on current source before publication. CG-09 through CG-16 and final +combined/load gates remain required. No new RustFS backend or recovery drill ran +in this checkpoint; historical scoped CG-06 evidence remains separate. + +## CG-07 controller-owned managed peer learner activation + +The Activate movement receiver now selects its physical history and enters the +same node provisioning manager as initial startup. Its public SDK call uses the +installed execution domain, original bounded deadline, exact namespace owner, +rooted birth and current physical boot. Pre-submission refusals remain distinct +from outcomes after controller submission. Its returned local membership receipt +is descriptive; promotion still requires actual catch-up and fresh quorum. + +Managed staging compares the expected permit and target boot with the fresh +catalog admission before creating a mapping. The source exchange still requires +its own fresh-quorum grant. Source endpoints come from rooted trust and current +node views. After complete checkpoint verification and persistence, attachment +uses actual committed membership and physical bindings, without a cell alias. +S3 learner activation explicitly refuses before submission; the exact-reference +archive transfer remains unimplemented and never falls back to a peer. + +The manager saves the applier/page owner before installation. Cancelled, expired +or lost replies retain that physical preparation and its capacity. The parallel +permit spans recovery and transfer. A completed install whose response was lost +is resumed using the durable seed receipt and permanent claim, with selected-root +recovery on the same page owner. It is not seeded again or given blank genesis. +Entered work is joined before closing transport/runtime dependencies. Retries +coalesce only for the same directory, route, boot, exact permit and source. + +Locked library capture CG-07-managed-learner-advance-01 reports 32 remaining +legacy errors, removing all four errors in the Activate receiver. No new primary +error or non-vendor warning appears in the delivered learner implementation. +This does not establish successful runtime compilation or execution. The capture +contains 4728 source inputs (manifest SHA256 +a4291efcce24441e5ebd48cea89ad3dae028a3dec322396b254b4b663492bea9). +No managed lifecycle, native race, backend or load test ran on this source. +Final format, module layout, native dependency, documentation and diff checks +passed. All owned compiler and static-check processes are terminal. + +Next remove the remaining legacy host/catalog/seed/inspection consumers while +retaining common physical deadline/process/shutdown services. Finish managed +node-directory/startup and exact-reference S3 learner transfer, then execute all +five original CG-07 public cases and the real RustFS recovery drill on current +source. Persistent archive hold retirement, CG-09–16 and the complete release +matrix remain open. The complete goal remains active and WIP is unpublished. + +## CG-07 managed group repair controls and peer-route lifetimes + +Begin, Promote and Finish movement consumers now call physical RaftGroupHandle +repair APIs. The controls require the rooted group and exact admitted permit, +check immutable birth/schema, actual current occupants and target boot/trust/ +physical-host separation, then use ordinary configuration proposals. Exact +permit/plan/phase and expected revision govern retries; aborted/fenced/later +permits cannot masquerade as progress. Successful mutation requires durable +apply and a fresh post-apply quorum, with original deadline and closure checks. +Actual RawNode readiness and catch-up enforcement is unchanged. Abort is also a +real group control. Learner Activate still uses its old consumer and remains a +required conversion; no S3 fallback or fabricated cell alias is added. + +Managed advisory routing can resolve committed joining-member/disk bindings +through rooted node views. It changes no configuration or serving authority. +Discovery lag leaves known voters usable and unresolved peers unavailable; +initial identities are taken from the authorized birth route, not guessed from +node IDs. Ordinary cell apply does not scan routes. Reconciliation runs after +configuration apply/checkpoint install and ordinary frames retry once for an +unknown target, before frame allocation/admission. Existing queue, peer, snapshot +and physical worker limits are unchanged. + +A bounded per-group lifetime token separates physical registration from advisory +route generations. Refresh and node-view changes retain the existing checkpoint +factory. Each transfer still captures an exact immutable route and rejects old +generations; per-chunk checking allocates no selected endpoint. Explicit route +replacement/removal revokes the old factory, and a stale refresh or old guard +cannot adopt/remove its successor. Group/cache guards are released before the +registry lock, and peer authorization runs after its validation guard drops. +These are implementation invariants, not executed qualification claims. + +Two new registration/generation regressions use parked real synced engines, +filesystem stores and transport/TLS listeners. They cover factory continuity, +old selection rejection, physical-owner cleanup and explicit successor retention. +They do not establish a fresh quorum, an active network checkpoint stream or +RustFS behavior. The existing query authentication test body is byte-identical; +only its shared fixture moved to a sibling file, retaining embedded TLS paths. + +Final library capture CG-07-managed-routing-advance-04 has the same 36 inherited +errors as the prior goal turn, including four old activation errors. No primary +error or non-vendor warning is added by the delivered route/repair code. The +intermediate control build's five type/import errors were corrected; its real +stream is retained with the other captures. Final source includes 4726 inputs +(manifest SHA256 +bfdbc3c3ff638d7cfd20b8034626e825e33c9fd09e3e9be78beb8713e6f14053), +with every captured Rust input matching at closure. The standard locked unit-test +compile (--lib --no-run) also stopped at the same 36 ordinary-library errors +before test modules compiled. No tests ran. All builds use the required restored +workspace-volume target. No managed/native/backend/load qualification follows, +and the WIP remains unpublished with the full goal incomplete. +Final format, module layout, native dependency, documentation and diff checks +all passed. All owned compile and static-check processes are terminal. + +Next finish controller-owned managed peer/S3 learner staging, complete-image/ +receipt install and registration using the existing immutable namespace profiles. +Preserve exact-reference S3 semantics and original fencing/retained debt. Then +remove remaining legacy host/catalog/seed/inspection consumers, retain common +physical transport deadline/process services, finish node-directory/managed +startup and all five original CG-07 public cases. Qualify these controls, trusted +route discovery and both regression modules using actual native/race/RustFS +harnesses. Persistent archive hold retirement, CG-09–16 and the full release +matrix remain required. + +## CG-07 controller-owned managed genesis and namespace profiles + +The restart-genesis receiver now calls activate_managed_initial_group on the +canonical SDK, replacing its retired cell lookup/registration/activation path. +The invocation enters its installed execution domain for catalog I/O, even if +polled on another runtime. It uses the exact registered namespace capability, actual rooted +Live birth, admitted initial voter and last-synced local boot. It reauthorizes +before submission and after attachment, checks the selected root and actual +membership, and returns a descriptive persisted fleet guard. That receipt grants +no fresh quorum, seed, serving or physical-release authority. Existing quorum +readiness and permanence/lost-disk checks remain separate and unchanged. + +The controller owns managed initial preparation, applier, selected pages, roster +reservation and transport attachment. Running attempts retain their original +deadline; another waiter cannot extend it. A completed attempt can be retried +with a fresh invocation while reusing its retained physical owner. Entered work +is joined after cancellation/expiry. Real late attachment remains registered, +but delivery cannot report late success. Both queued delivery and consuming an +unread successful reply check its original deadline; managed admission retains +the existing 60-second maximum. An attached retry performs fresh boot/catalog +checks under shared preparation/parallel admission without reopening its page +store, reattaching consensus or adopting another registration's lifecycle. +Current roster/quarantine and shared-engine fatal-error checks remain in force. +Typed startup refusal and unconfirmed outcomes stay distinct; catalog pressure, +conflict, corruption and owning causes are preserved at the receiving boundary. + +Provisioning now has immutable CellGroupProvisionNamespace profiles. Each binds +an exact directory owner and declared image backend; duplicate namespaces, +foreign owners and representation mismatches refuse. Canonical startup clones +its single node image inventories into these profiles and configures explicit +CellGroupProvisionLimits. All namespaces share one preparation bank and parallel +limit. Fixed peer metadata and advertised directory-cache capacities are charged +once against the same total byte ceiling, then subtracted from available active +preparation capacity. No resource limit is enlarged. Usage/metrics expose that +configuration retention separately, and existing joined shutdown drops it after +physical work has joined. + +Existing provisioning fixtures register their actual namespace owners and image +representation for both backends. Their coalescing, pressure, retained-owner, +joined-cancellation, missing-page restart and healthy-group assertions remain. +Additional checks refuse duplicate profiles, insufficient configuration budget +and an unconfigured foreign owner before its first materialization. These test +modules have not compiled or executed while the runtime library fails. They +provide no managed lifecycle, sync, quorum or backend qualification yet. + +The first two locked workspace-volume library builds reduce errors from 40 to +36, removing all four retired genesis receiver errors without new primary +startup/provision/profile diagnostics or non-vendor warnings. The final reviewed +capture is CG-07-managed-genesis-advance-04, also terminal with the same 36 +remaining errors. All four real compiler streams and formatter results are +retained. The final 4723 source inputs have manifest SHA256 +25b5a30439963dea58e34d16208b4f2572f37d6fa257c1c35dc2c8afca4ae3c6; +all captured Rust inputs match at this validation point. Static review records +11 unchanged native invariant owners, including permanent claims, serialized +boot checks, RawNode catch-up, complete-root attachment and common physical +transport cancellation. This is not executed qualification. The restored +workspace-volume target is used for all four captures. Formatting, module +layout, native dependency isolation, documentation and diff checks pass. All +owned validation processes are terminal. Static assertion review retains all +43 original provisioning assertions (with explicit profile arguments) and seven +additional refusals; the tests are still uncompiled and unexecuted. Latest +origin/main is an ancestor of the working branch. Canonical managed startup +remains gated and unqualified WIP is not published. + +Next finish controller-owned managed learner activation, exact-reference S3 +restore, real group repair begin/advance and trusted replacement peer routes. +Retire the remaining old host/catalog/seed/inspection consumers while retaining +common physical transport deadline/process services. Installed node-directory +ownership, managed startup sequencing and all five original CG-07 public cases +remain open, followed by current-source native/race/RustFS qualification, explicit +archive hold retirement, CG-09–16 and the complete release matrix. + +## CG-07 shared physical page path for provisioning and restart + +Canonical startup, controller provisioning and independent archive replay now +use ManifestPageStore::group_pages_path. Serving owners use the same configured +node base and groups/////pages inventory. +The archive verifier retains its independent base. This removes provisioning's +flat path, which could reopen a different page store after canonical restart. +The ordinary exclusive lock, selected-root validation and failure behavior stay +unchanged; no flat-path reader or migration is added. + +The existing provisioning fault and restart fixtures now independently expect +that canonical directory. Static comparison confirms every test body and +assertion outside the path helper is byte-identical, including missing-page +refusal, retained-owner retry and joined cancellation. Both materialized and +extent cases remain. These tests have not executed or compiled while the runtime +library fails; static retention is not physical recovery qualification. + +The workspace-volume locked build is terminal with the same 40 error-message/path +pairs as the managed-seed capture, with no primary diagnostic or non-vendor +warning in this change. CG-07-provision-path-advance-01 captures 4722 source inputs +(manifest SHA256 +87cdfd5273f790446ac1753f5593bdfe88d3e5ee8f0a6fb2a095298e5496bad3). +All captured Rust inputs match at this validation point. Formatting, module +layout, native dependency isolation, documentation and diff checks pass. All +owned validation processes are terminal. The new layout and managed peer-seed +implementation remain unqualified and unpublished. + +Next configure immutable namespace-bound provisioning profiles using shared node +image admissions, wire controller-owned managed learner activation/registration, +and add real group repair/genesis consumers. Remove remaining legacy callers +while retaining shared transport deadline/process services. Managed canonical +startup, all five original CG-07 gates, physical race tests, fresh RustFS, explicit +archive hold retirement, CG-09–16 and the release matrix remain required. + +## CG-07 managed complete-group peer seed admission and fencing + +The shared transport now has prepare_managed_group_seed, using the same complete +checkpoint stream, receiver pools and retained engine work as unmanaged seeding. +The destination must have a fresh active physical reservation for its exact boot. +The selected source must be a rooted seed owner with the exact current member, +node, disk and process boot, and a voter in the authenticated checkpoint. The +source descriptor must carry this exact consumed permit/repair in Seeding phase. +Subsequent authorization cannot regress its original catalog revision. Catalog and handshake time are deducted before forwarding the remaining original +timeout. Review also put catalog preflight, staging and joined-permit delivery +under the existing transport deadline, process revocation and shutdown wrapper. +Namespace ownership reads count inside directory operation bounds. Queued engine +work retains its grant even after the metadata waiter times out. + +Managed staging and installation retain the permanent member/engine claim and +check the exact last-synced local boot in the serialized storage worker. A queued +quiescence, stop or fence refuses further installation. Once accepted, physical +work remains owned through actual sync; expiry/close afterward suppresses late +success without erasing a durable receipt. Managed joined-member bootstrap and +permit revalidation require that receipt, the permanent claim, the rooted Live +group and the current exact boot. An old active permit is not restart authority. +Unmanaged joined/seed entry points also check actual namespace ownership. + +The staged base preserves the source's real enrollment origin. Genesis guards +remain genesis guards; an imported group's committed enrollment is installed by +ordinary snapshot successor validation. Normal membership successors, recovery +lineage, checkpoint encoding and RawNode promotion checks are unchanged, as +recorded in static source review. A staged learner remains outside the birth +configuration and cannot acquire startup/serving authority from this projection. +No marker bump, old reader, cell alias, no-op action or resource increase is added. + +The previous goal turn removed the legacy query path and reduced the diagnostic +to 40 errors. All three managed-seed captures also report 40. The reviewed final +CG-07-managed-seed-advance-03 includes deadline composition and has the same error-message/path multiset as the query +capture, with no primary error or non-vendor warning in the new seed, storage, +joined-bootstrap or source-authorization code. Its 4722 source inputs are bound +by manifest SHA256 +7746dd2c6a4902499291d91f15f3bbcb09bebb1bbc534b16ec5985cf09efa0d5; +all captured Rust inputs matched at that validation point. The first capture used the temporary target. +Workspace-volume writes are now restored; the reviewed build and follow-up checks +return to the normal checkout-specific target required by AGENTS.md. +Formatting, module layout, native dependency isolation, documentation and diff +checks pass for the final capture; all owned validation processes are terminal. + +Five new profile/origin/boot-guard regressions remain unexecuted while the library +fails. Their metadata/guard fixtures grant no catalog, sync, quorum or installation +proof. No new runtime, Clippy, managed lifecycle or RustFS qualification is claimed. +This implements the lower managed peer-seed boundary; SDK/controller-owned learner +activation and managed canonical startup are still gated. Next wire this path into +the existing provision/registration owner, add group repair begin/advance and +managed genesis consumers, finish managed S3 learner restore and remove old +seed/inspection/host callers. All five original CG-07 cases, physical race/native/ +RustFS qualification, explicit hold retirement, CG-09–16 and the release matrix +remain required. Unqualified WIP is not published and the goal stays incomplete. + +## CG-07 query hard cutover and retained authentication + +The retired per-cell SDK no longer admits business queries. Its query options, +routing/result types, pending query operation, host execution branch and remote +routing loop are removed. The old query_route module and request/reply body codec +are deleted. QueryRoute accepts only a physical-group plus logical-cell identity; +the Cell history variant and its reader are gone. Current cell queries, commands, +enrollments and receipt lookups keep their separate purpose-bound bodies on the +same authenticated peer query lane. Marker 1 remains unchanged, with no old reader, +translation, production switch or increased resource bound. + +The common framing/MAC helpers, exchange, origin-grant validator and deadline/ +cancellation/revocation controller are byte-identical to their baseline regions. +This is recorded as static source review, not executed security qualification. +Existing query codec oracles now use logical cell tokens and physical group routes, +preserving channel/exporter, digest, sequence, response correlation, process, +non-voting origin, bounds and typed-failure checks. The original grant test moves +to real synced shared engines and start_shared transport with all 18 assertions +retained: namespace/process isolation, tombstone pressure, in-flight revocation, +regrant refusal and physical pool retention/retirement. Its parked runtimes grant +no serving or quorum proof. A new regression supplies complete superseded route, +request and reply inputs and requires refusal. These tests remain unexecuted. + +The previous goal turn implemented shared fleet actions and restart readiness, +reducing the locked library diagnostic from 90 to 65 errors. The first check +reports 40 errors; the reviewed, formatted final capture +CG-07-query-cutover-advance-03 also reports 40, with no primary error or non-vendor +warning in the converted query code. Review restored exact token equality in a +ported wire oracle before the final capture; the tests remain unexecuted. All +three sequential captures and their real streams are retained. Two remaining +host.rs errors concern unchanged predecessor startup service types, rather than +query execution. Its 4719 source inputs are retained (manifest SHA256 +86d951269719dd948dcd2e0849eb9efa7c0d0d63fde87edb2fcf501b21db56dc), +and every captured Rust input still matches. Formatting, module layout, native +dependency isolation, documentation and diff checks pass; all owned validation +jobs are terminal. The system-disk target remains +checkout-specific and ignored because the normal workspace-volume probe still +returns EPERM. No other checkout artifacts were reused or cleaned. + +This is hard-cutover implementation progress, not runtime qualification. No new +runtime tests, Clippy or RustFS drill ran while the canonical library fails. +Earlier core/native/backend records retain their original source scope. Remaining +legacy SDK/Axum/read examples and low-level SQL fixtures must be converted without +weakening their behavioral oracles. Complete-group managed repair/learner seeding, +registration/genesis, old seed/inspection callers, installed directory ownership +and managed startup are the critical next work. All five original CG-07 public +gates, explicit persistent hold retirement, refreshed native/backend qualification, +CG-09 through CG-16 and the complete release matrix remain open. + +## CG-07 shared fleet actions and restart readiness + +Fleet movement/restart action dispatch now uses the installed shared SDK gateway. +Local execution checks the exact synced boot and installed runtime owner; remote +execution retains the existing authenticated action lane, rooted receiver checks +and exact target. Invocation captures its original deadline and control admission +before polling, with the existing 128 KiB reservation. Internal observers do not +retain the external SDK lifetime. Late delivery or an interrupted entered action +remains unconfirmed; it does not establish rollback. Quiescing hosts can perform +their restart handoff, while fenced/stopped boots refuse action admission. + +Restart handoff and resume callers now address complete GroupHistory values. The +actual group owner checks term, membership revision, stable RF3 configuration, +committed floor and each required survivor's matched/active progress before a +transfer. The runtime rechecks the exact synced quiescence selector after queueing +and tightens its evidence expiry to the original call deadline. The shared and +remaining legacy owners use one extracted predicate, preserving the existing +checks and actual caught-up leadership transfer. Replication metadata alone +grants no quorum, health or seed authority. Restart progress reports carry the +physical history. The obsolete per-cell fleet action dispatch modules are removed. + +The first locked library check reports 70 errors, down from the prior 90. The +reviewed final source fixes five remaining restart-report type mismatches and +reports 65 errors. There is no primary diagnostic in the new action, shared-owner, +restart-gate or converted handoff/resume paths. All 4719 captured source inputs +are retained in CG-07-group-actions-advance-02 (manifest SHA256 +248e011a2bdaa4ec19a2f93ee2b80cae44f14da43d06b70517bf76684edd6f40), +and every captured Rust input still matches. This is compiler progress, not runtime +qualification. Four new readiness/identity/quiescence regressions are unexecuted +while the library fails. No new runtime tests, Clippy or RustFS qualification ran. +The previous 96 enabled pure-core test results remain scoped to their capture. +Formatting, module layout, native dependency isolation, documentation and diff +checks pass. All owned build/formatter/check jobs are terminal. + +After the user cleaned the workspace volume, a new lock open and independent +checkout-specific probes still returned EPERM there. The failed formatter attempt +is retained. The existing repository formatter completed directly; the final +locked rebuild used a fresh ignored checkout-specific target on the system disk. +No other checkout's artifacts were reused or cleaned. The capture's environment +record documents this temporary deviation from the normal workspace-volume rule. + +The next critical cutover removes the old query routing/body codec while preserving +the authenticated framing, node/process and grant/revocation checks shared by +cell queries, commands, enrollments and receipts. Complete-group learner seeding +already has authenticated grants and joined-member bootstrap; managed repair, +registration/genesis and activation must use those real owners. Installed directory +ownership, managed startup and all five original CG-07 public cases remain open. +Persistent hold retirement, refreshed native/backend gates and CG-09 through CG-16 +are still required. No unqualified runtime changes are published. + +## CG-07 physical-group resource, repair and topology decisions + +The pure fleet reservation ledger, topology view/targets/moves and isolated-repair +view/results now identify complete GroupHistory values. Cell names no longer enter +these models. A physical replica still costs one engine slot; its combined image, +staging and memory footprint stays charged under the unchanged resource rules. +Resource reservations retain exact namespace, group and incarnation identity. +Old-incarnation reservations cannot be reused or silently erased by a new history. +Topology rejects two current incarnations of one namespace/group ID, while equal +opaque group IDs in different namespaces remain distinct. The isolated repair +result is GroupRepairPlacement, carrying the same revision/member fences for every +cell. Its deterministic tie-break uses physical history, with no cell alias. + +The Docker failure-recovery adapter now supplies complete RangeDirectory handles. +Before sampling isolated-repair resources it checks current rooted group +authorization. This neither infers failure from absence nor grants membership, +release, recovery or deletion authority. Existing physical debt, exact boot, +capacity/domain checks, work bounds and RF3 assertions remain unchanged. + +The previous goal turn made implementation progress on complete-group replay and +persistent acquire journals, reducing the captured library diagnostic to 100 +errors with static checks passing. This turn changes the authoritative group +models. Its first test compilation caught a missing BTreeSet import in a new +regression; that failed attempt is retained. The corrected capture passes all 96 +enabled cellar-logic tests and strict all-target Clippy. Three new executed +regressions cover namespace-qualified group IDs, conflicting current incarnations +and retained old-incarnation debt. No existing planner oracle was weakened. + +The eight executed pure-core binaries are retained and hashed on the workspace +volume. Capture .cellar/cell-groups-evidence/CG-07-group-model-advance-02/ binds +4718 source inputs (manifest SHA256 +e974a366a32ad5982c07d1de70ae79f250934d69c8e9b98cd9319c14a1691b62). +The locked runtime check now reports 90 errors, down from 100, with no primary +error or warning in the converted fleet models or Docker adapter. All captured +Rust inputs still match. Pure-core results do not qualify public SDK/fleet startup, +Docker isolation, RustFS or managed lifecycle execution. The library and all five +original CG-07 public gates still need to pass. Formatting, layout, dependency, +documentation and diff checks pass; all owned validation jobs are terminal. + +Investigation confirms that the old per-cell query transport shares framing with +logical-cell queries, commands, enrollments and receipts. Keep that authenticated +shared infrastructure while removing legacy query routing/codecs/host consumers. +Then finish group-native learner/restart/leadership control RPCs, installed node +directory ownership and managed startup. Explicit persistent hold retirement after +fencing/joining, old fixture cutover and refreshed native/backend gates remain +required. CG-09 through CG-16 and the complete feature-off/load/crash/release matrix +remain open. No unqualified runtime changes are published. + +## CG-07 managed complete-group replay and durable acquire ownership + +Automatic failed-node removal and management lifecycle verification now use one +shared FleetGroupArchiveVerifier contract. It opens the exact namespace-owned +physical group archive, selects survivor candidates from rooted fleet membership, +freezes the complete quorum-proven cut, acquires independent persistent retention +and restores the full checkpoint plus contiguous tail. The former per-cell birth, +cell archive and single-image verifier factories are removed from both callers. +Public configuration now supplies explicit private materialized/extent image +inventories, page bounds and node-shared archive/checkpoint/tail/scratch pools. + +An acquire journal is keyed by the exact verification basis and pending group, +independently of later business writes. Before the registry sees a mutation, the +fleet catalog conditionally stores the complete intent and observes its winner. +Concurrent controllers send only that persisted winner. Restart restores its +original fixed cut even before a live receipt was appended. Unknown responses +retain the same allocation; a fresh registry read must prove its revision was +consumed without its hold before a conditional journal successor is prepared. +An independently confirmed hold still reconciles after unrelated allocations. +Each method retains its original deadline and rechecks rooted basis before and +after work. No publisher-owned hold, new allocation on timeout, one-cell alias, +format bump or compatibility reader is used. + +Receiver admission now precedes the blocking page-inventory scan. The existing +checkpoint byte/item bank reserves receiver workspace plus bounded page inventory +metadata. Its reservation follows the entered worker, receiver and recovered +closure; cancellation cannot release it while physical work still owns it. +Pressure and limit failures remain typed. No configured quotas are increased. +Persistent holds are deliberately not released on drop or timeout. Explicit +terminal retirement still needs a fence for every user and joined physical work; +the existing per-registry hold ceiling is unchanged. + +The previous goal turn completed the captured 112-error integration diagnostic, +static checks and preservation of the prior complete-root work. This turn changes +the authoritative implementation. Formatting passes. The locked library check +now reports 100 errors, down from 112; no primary error or warning is reported in +the new verifier, acquire journal, admission code or converted lifecycle callers. +This is a failing compiler diagnostic, not qualification of those modules. +The capture under .cellar/cell-groups-evidence/CG-07-managed-archive-advance-01/ +binds 4718 source inputs (manifest SHA256 +87c1ddfc914865321c39cd67a7aff912c55044180340fb7fd7811bb85e1d4d3b). +All captured Rust inputs still match. A dependency-check attempt lacked Cargo on +PATH; its failing stderr is retained separately. With the required toolchain +environment, format/layout, native dependency isolation, documentation and diff +checks all pass. All owned formatter, compiler and static jobs are terminal. +No tests or Clippy ran because the library still fails. Origin was fetched and remains +81cbbc2f72f302751dd471cf06cf51394146ac63, with the branch zero commits behind. +Five new regressions cover lost-response allocation reconciliation, consumed +revision and foreign-purpose refusal, and receiver inventory pressure/lifetimes. +They remain unexecuted while the canonical library fails. No native, managed, +RustFS, load or release qualification is claimed for this work. + +Next finish the remaining group-native topology/repair consumers, authenticated +learner/restart/leadership RPCs and installed node-directory owner, and convert +legacy query/seed/host consumers without restoring the one-cell runtime. Managed +startup and all five original CG-07 public cases remain required, including the +hard cutover of old verifier fixtures. Then qualify explicit hold retirement and +refresh complete native/backend gates for the changed marker-1 root shape. +CG-09 through CG-16 and the combined feature-off/load/crash/release matrix remain +open; the original end-to-end goal is not complete. + +## CG-07 complete-group fleet publication and archive receipts + +Fleet membership RPCs now use the shared SDK gateway and actual physical group +owner. Both local and remote paths reauthorize the exact rooted fleet history, +retain the original deadline and use fresh safe ReadIndex plus durable apply. +The body carries the complete GroupManifest, full membership/configuration and +same-owner progress. No cell, SQL image alias or per-cell query is selected. +Published cuts contain a complete root and exact membership commitment; the +zero-cell root is explicit and missing publication cannot construct a cut. +Configuration set order is canonicalized without changing joint roles. Delivery +age extends once through the original SDK invocation. Metadata descriptors move +their membership ledger into the returned proof rather than cloning it across +response retention; complete RPC/capture work reserves the existing 1 MiB +physical metadata envelope from the unchanged configured control bank. + +Manifest page references now authenticate the maximum physical cell mutation +position throughout each subtree. Updates recompute it along existing AVL paths; +metadata reads obtain it directly from the selected root without scanning cells. +Full tree verification recomputes the summary from actual pages and leaves. +Complete-root prefix comparison distinguishes metadata-only progress from later +cell changes, rejects a conflicting root at the same position, and rejects a +changed root whose latest mutation predates the fixed cut. It also checks exact +route, configuration commitment and recovery lineage. Physical group positions +remain distinct from logical cell versions and original receipt witnesses. + +Fleet archive coverage now independently restores every checkpoint dependency +and replays the contiguous group tail under an exact persistent archive hold. +It compares the complete root and configuration commitment. A locator or uploaded +single image cannot supply this proof. The caller must own the hold and replay +receiver, preserve its original deadline, and retain entered physical work. +The fleet verifier suppresses late delivery and keeps pressure, timeout and +conflict failures distinct. This proof supplies no current health, membership, +write acknowledgement or physical release authority. + +Receipt index rows retain their original 4 KiB bound. Complete live evidence is +stored in an immutable, content-hashed object under the existing catalog page +bound; one append batch retains the original aggregate encoded-byte ceiling. +Readers require the exact object/hash and revalidate the full root/hold shape. +Orphan objects or index pages grant no receipt. Marker-1 readers refuse the old +inline/single-image shape; no fallback reader, format bump or resource quota +increase is added. The superseded per-cell fleet observation requests, read +variant and host RPC implementation are removed. Other legacy adapters remain +in the unfinished cutover. + +The previous goal turn made progress on physical node inventory and follower +quorum metadata, with a captured 124-error library diagnostic. This turn changes +the authoritative implementation. The first locked check reports 120 errors; +the second and final checks each report 112, with no primary diagnostic in the +new group metadata, complete-root coverage, manifest-summary or receipt modules. +The final check includes configuration-order review and captures 4714 source +inputs (manifest SHA256 +fb11c46a225c6859ef9ac482740721b2efbea260fa4dbf762265e4d907ec008b). +Every captured Rust input still matches the current source. This is a failing +compiler diagnostic, not qualification of those modules. Captures and actual +streams are under +.cellar/cell-groups-evidence/CG-07-fleet-root-advance-01/, -02/ and -03/. +New codec, complete-root prefix, authenticated subtree mutation and bounded +receipt-shape regressions are unexecuted while the canonical library fails. +No native, managed, RustFS, load or release qualification is claimed. Existing +CG-06/08 backend results remain scoped to their original artifacts; the new +marker-1 manifest shape requires fresh qualification before publication. +Formatting, module layout, native dependency isolation, documentation and diff +checks all pass. No new tests or Clippy ran because the library still fails. +The final compiler owner and watcher, all formatter jobs and the static runner +are terminal; no owned build or backend job remains live. + +Critical next work is the managed verifier's owned archive-hold intent journal +and complete replay-receiver integration in both automatic removal and management +control. Do not borrow a publisher's disposable hold or retry an uncertain acquire +with a new allocation. Then finish group-native placement/repair models, +learner/restart/leadership RPCs, installed node-directory ownership and managed +startup. All five original CG-07 acceptance cases, CG-09 through CG-16 and the +combined feature-off/load/crash/release matrix remain required. + +## CG-07 physical SDK inventory and follower quorum metadata + +The SDK now obtains local and authenticated remote node observations from the +shared controller and actual cell-group engine catalog. A complete engine scan +retains unattached/quarantined histories and checks that the roster and durable +slot count did not change during capture. Pending attachment/provisioning debt is +reported independently of durable slots; a zero engine count cannot erase it. +Startup bounds come from the actual controller/runtime/log capacities. Current +fleet enrollment is the exact trust cut installed in the transport, not a separate +catalog refresh. The SDK keeps original invocation deadlines and response +reservations; internal metadata observers do not retain external node lifetime. + +The transport has a weak gateway to the real shared SDK and validates its runtime +and execution-domain owner. Node probes no longer upgrade an old cell gateway or +invent a cell identity. The marker-1 node body requires the new +pending_group_registrations field and refuses the old single-history spelling. +Its new regression preserves concurrent debt independently of empty disk slots, +refuses the old body and checks scalar overflow. Local profile observation can +now be configured; managed startup, node-directory publication and mutation +controllers remain gated until their whole-group protocols are complete. + +Review found that the prior shared metadata reader required a local leader, +which cannot supply the three voter observations required by fleet recovery. +Metadata-only safe ReadIndex now forwards from followers. Its distinct context +binds the origin member and fresh invocation nonce; incoming requests must match +that authenticated member and responses must come from the current leader. +Execution/seed contexts remain local. Selection waits for the actual local +durable apply/root/configuration, records same-owner progress and rejects a term +change. The original byte/item/deadline banks and leader-only execution readiness +are unchanged. A new three-engine TCP regression covers both image forms, two +cells, joint configuration on every voter, forged nonleader replies, released +read banks and continued refusal of follower execution/transfer authority. + +The prior goal turn only reverified CG-06 status and is classified as no progress +for the full cutover. This turn changes the authoritative implementation. The +first locked diagnostic reports 124 remaining errors, down from 157. The +second locked diagnostic also reports 124 after the follower metadata changes, +with no primary diagnostic in the new inventory/read-owner/gateway modules. All +captured Rust bytes remain unchanged. Captures and real +streams are under .cellar/cell-groups-evidence/CG-07-node-inventory-advance-01/ and +-02/. Format check, layout (2026 Rust files), native dependency isolation (496 reachable +packages), documentation links and whitespace checks all pass. No new regression +has executed while the canonical library is failing. No managed runtime, RustFS, +load, or release qualification is claimed here. + +Critical next work is the authenticated whole-group fleet observation/publication +body and complete-root archive/removal proofs. Then finish repair/learner and +restart/leadership RPCs, installed node-directory ownership and managed startup, +and run all five original CG-07 acceptance cases. CG-09 through CG-16 and the +combined feature-off/release matrix remain required. Existing CG-06/08 backend +evidence stays scoped to its previously qualified source artifacts. + +## CG-07 physical SDK controls and joined service ownership + +Local SDK group handles now expose guarded fleet enrollment, exact permit +fencing and fresh physical membership observation. Configuration admission is +not completion: the call waits for its durable apply floor, then obtains a fresh +safe ReadIndex under the original invocation deadline. Exact already committed +guards/fences reconcile through a new observation. Refused means this invocation +submitted no configuration; an error after a possible send remains Unconfirmed. +It never authorizes automatic retry, reservation release or node removal. + +The bounded read owner can defer its quorum barrier until the requested minimum +is durably applied, without choosing an application cell. If a later configuration +overtakes a pending barrier, the same invocation re-arms within its original +reservation and deadline. A held minimum consumes the existing fixed node/group +read banks and does not busy-poll or enlarge a quota. The new native regression +holds actual engine sync, checks non-completion and retained admission, then +requires the real current configuration and quorum barrier in both image forms. +It also checks an unreachable floor's deadline and release of its reservations. + +The new SDK has a fixed two-slot control-service registry sharing the configured +control byte/item bank. Movement/restart workers hold their registration through +actual destructor completion; independent done signals occur after retained +worker fields are dropped. Cancelling a close waiter keeps admission closed and +does not erase the live worker. Node closure signals and joins these services +before controller/transport/storage shutdown. Internal observers retain metadata +without retaining the final external SDK lifetime, so a service cannot self-own +an otherwise abandoned node. Actual control admission is included in SDK resource +observations. Service startup refreshes the installed host and checks its exact +node/disk/engine/process boot and synced lifecycle guard; managed canonical +startup remains gated while the remaining consumers are unconverted. + +The previous goal turn made progress: physical namespace import and inventory +interfaces advanced, with an unchanged-source library diagnostic of 174 errors +and full preservation of 136 changed/new files and all eight user diagrams. +Current source is local and unqualified. The first sequential locked library +check reports 165 remaining errors; the final captured check reports 157, with no direct diagnostic in the new SDK, +read-owner or service-worker modules. Actual transport is now retained by the +new SDK, which submits persisted exact-boot lifecycle requests on the existing +authenticated stop/fence lane. It captures deadline and control admission before +returning the waiter; the receiver independently owns admitted guard mutations. +No local business group is required. A lost/expired reply remains Unconfirmed. +Formatting passes over 2025 Rust files; native dependency isolation passes with +496 reachable packages. Source captures and actual streams are retained under +.cellar/cell-groups-evidence/CG-07-physical-controls-advance-01/ and -02/. The new native and +service-lifetime regressions remain unrun until the canonical library compiles. +No RustFS, sustained-load, managed runtime or release qualification is claimed. + +Reflection: connect the physical SDK/control RPC and inspection owners, convert +whole-group archive/restart/removal verification and occupancy, and finish managed +canonical startup. Then run all five original CG-07 acceptance cases and the full +affected feature matrix. CG-07 remains IN_PROGRESS; CG-09 through CG-16 and final +combined qualification remain required. + +## CG-07 physical-group import and lifecycle inventory cutover + +The new import path accepts RangeDirectory and an opaque frozen physical-group +inventory. Shared birth reservation now enters the namespace's CAS admission +index before topology publication. Admission freeze arbitrates against disaster +recovery, then a second CAS locks the directory. The exact locked head is retained +by digest. Delayed admissions and stale topology writers cannot enter that cut; +unknown responses retain the same operation and permanent identity claims. + +Inventory settles every physical birth, including unselected pending groups. +Cancellation competes with materialization at one immutable decision key and +retains the group identity and fleet/node references. Selected lazy births stay +live; absent voter disks do not authorize imported genesis. Coverage consumes a +fresh cell-independent quorum membership capability and verifies the exact birth, +guard, stable RF3 roles, node storage pins and permanent group member/engine +claims. Handoff independently walks every frozen selected range and requires +its exact admission and coverage witness, in addition to the complete intent set. +The adopted owner is checked by actual managed birth/bootstrap effects. Topology +unlock advances the head again, so a predecessor expected head cannot reappear. + +Restart birth settlement now uses physical group decisions and exact opaque +fleet reservations. Maintenance, capacity, evacuation, resource and platform +interfaces retain RangeDirectory handles. Standalone management connection takes +explicit group directory limits; movement/debt projections no longer contain a +cell field. The marker-1 protocol adds group-import-1 with a hard cutover. + +All of this remains local and unqualified. The first sequential locked library +check reports 194 remaining integration errors, including one new joined-method +name mismatch corrected after the check ended. Its two unused predecessor +re-exports were also removed. The new native regression holds an actual admission +CAS while freeze wins, drops a freeze reply, and asserts exact inventory count, +stale topology refusal and retained reopen without object listing. It has not +executed: the canonical library still fails compilation. The follow-up check and +its exact source capture are under CG-07-group-import-advance-02. That check +ends with 177 remaining errors and no own-crate warning. The new admission, +freeze, import and inventory modules emit no direct diagnostics. Resource, +movement, topology, repair and service models were then converted to GroupHistory; +the final capture is CG-07-group-import-advance-03. None of these diagnostics +qualifies the canonical runtime or replaces the earlier directory pager checkpoint. + +Review found that several admitted candidates can share a logical range and +epoch. The fleet range-debt key now includes physical group/incarnation identity; +the directory retains exclusive selected-range ownership. A new invariant test +covers distinct pending keys at the same range/epoch. The third sequential check +reported 174 errors with no own warning before that fix; the final reviewed +source capture is CG-07-group-import-advance-04. The final locked library check reports 174 remaining integration errors; +all captured Rust bytes remain unchanged. The new admission, freeze, import and +inventory modules emit no direct diagnostic, which is not qualification. Both +new regressions remain unexecuted until the canonical library builds. + +Reflection: finish the physical SDK/control RPC owners and complete-group archive +verification used by movement, restart and removal. Then enable managed canonical +startup and run all five original CG-07 public acceptance cases. Existing per-cell +host/query/control consumers still need removal or conversion. CG-07 remains +IN_PROGRESS, and CG-09 through CG-16 and the combined release matrix remain open. + +## CG-07 group-native managed authority work + +Unqualified cutover work now removes the application cell from FleetGroupBinding. +Fleet histories carry the exact physical GroupHistory, range, route epoch and +original creation identity. Range reservations and permanent group IDs retain +the existing rooted CAS journal, node references and unknown-outcome behavior. +Namespace/class declarations are checked explicitly, without deriving a class +from a synthetic group cell. The marker stays 1; the protocol identity adds +cell-group-fleet-1 and earlier cell-bearing bindings refuse decoding. + +Managed group genesis now writes its configuration-owned fleet guard in the +first engine sync. Existing bare histories refuse retrofitting. Imported histories +can only resume an existing matching guard. Catalog bootstrap checks the exact +host/node/disk/engine/process boot, and the engine worker rechecks the synced +lifecycle guard before creating or resuming a managed group. Quiescence, accepted +stop and fencing block this work; a matching durable resume can reopen admission. +Membership envelopes retain full route binding and guarded permit sequencing +through enrollment, fence, learner repair and joint-consensus steps. Checkpoint +capture/decoding checks the guard's physical history. + +The shared runtime has a new cell-independent quorum membership observation. +It sends a fresh safe ReadIndex, waits for contiguous durable apply, and selects +one coherent configuration behind that barrier. Node/group metadata reservations +are acquired before cloning membership and remain charged through response +retention. The SDK captures the original observation deadline before polling. +This metadata grants no eligibility, archive, deletion or release lease. + +Fleet inventory now traverses physical range pages and checks both directions: +each selected route needs a rooted history, and every retained reservation must +keep its exact namespace/birth/node references, including pending births outside +the selected tree. Import and cancellation require complete group-native terminal +proofs; a predecessor cell inventory or a caller-supplied Cancelled value refuses. +The additive range pager passes its separate primary checkpoint at 66d5ea6a. +It bounds each response, pins its continuation to an immutable directory cut, +and uses authenticated tree paths without listing object-store keys. + +The first managed-authority library diagnostic ended with 202 errors in unconverted +consumers. It overlapped an owned formatter; pre/post-format snapshots and the +complete output are retained as diagnostic only. No new core authority module +emitted a diagnostic in that run, which is not qualification. The successive +sequential library checks report 205, then 198, then 198 remaining integration +errors. The last run includes typed fleet-authority failures, a definite refusal +for missing imported genesis without quarantining an otherwise healthy engine, +and public execution-domain clocks in the new external native assertions. +All new core assertions remain unexecuted. CG-07 remains IN_PROGRESS and +managed canonical startup remains closed until its real consumers are converted. + +Scope addition: native_cell_groups_catalog.rs and its existing directory fixture +cover the physical inventory pager; no original CG-07 case or oracle is replaced. +The first pager build caught a missing catalog re-export before any test executed; +that failure is retained. The corrected checkpoint passes all seven commands: +the exact new case, nineteen catalog cases, strict all-target Clippy, formatting, +layout, native dependencies and docs. No case is ignored. Native record SHA256: +8061d03d7902d21f291fb82b16bcf04c91d20716465ef5292eecd7c754dcd91a; +executable SHA256: 43f6e2f3654ca92b623b3960c04bbd443a17128a1391728f35c725bdb20b8d48. +The six-file source snapshot, exact executable and actual streams are retained +and independently backed up. Both checkouts share runtime base 66d5ea6a with all +92 preserved WIP files and eight diagrams unchanged across that fast-forward. +The reviewed source-baseline document passes all four post-gate checks and is +published with this component on PR 160 through c54307c5. Both checkouts now +share that head; managed cutover changes remain local and unqualified. +Records live under .cellar/cell-groups-evidence/CG-07-managed-group-advance-01/ +through -04/ in the spare checkout, and CG-07-directory-inventory-01/ and +CG-07-directory-inventory-fixed-02/ in the primary checkout. These are metadata +fixture results, with no new RustFS, managed runtime, load or release qualification. + +Reflection: finish group-native import/cancellation and physical inspection/RPC +consumers, connect the actual movement/restart owners and managed canonical +startup, then qualify the original five public SDK/ingress/lifecycle cases. +The full cutover, CG-09–16 and final combined release/load/crash matrix remain open. + +## CG-07 whole-group shutdown, drain and resource ownership + +The spare cutover now connects Node shutdown to the logical SDK's physical close +owner. The bounded close report contains whole GroupHistory replicas, complete +aggregate group/archive counts and separate unavailable coverage. Retained calls +are not interpreted as one unfinished command per group. Producer quiescence is +observed only behind sealed public admission, an idle application-work registry, +healthy group/provisioning state and idle physical SQL. Unobserved outcomes never +become a clean stop because a counter happens to be zero. Controller, inventory +and engine close failures remain distinct, and no report grants disk deletion. + +Shared owners expose real replication progress and recheck term, membership +revision, stable incoming voters, recent activity and catch-up on leadership +transfer. Node drain gives each physical group one bounded turn under the original +absolute shutdown deadline. Offered attempts, owner acceptance and observed +successor leadership remain separate observations, including timeout races. +No cell receives a worker or timer. Stale-term/revision assertions were added to +the real membership fixture; they are not yet executed in the uncompiled cutover. + +Resource observations now describe shared namespace/controller/runtime/SQL +admissions and actual capacities. Serving and independent archive-verifier image +stores have separate passive accounting; cached manifest counts are metadata, +not serving authority. The SDK implementation is split by responsibility under +application/consensus/group_client/. The controller's existing join owner releases +namespace image resources even when a close waiter is cancelled. Unpolled +enrollment and receipt futures defer cloning those physical resources. + +A retained-closed-metadata regression exposed cached manifest pins surviving +physical engine closure. Both image forms fail at 5e7aadb6 with directory-lock +refusal. The minimal cache/storage fix releases only derived pins after the +closed fence and physical engine join, before terminal completion; independent +readers retain their own pins. Both regressions pass with the fix. Its twelve +checks also pass full execution (165, two fixture ignores), archive data (29, +five RustFS ignores), effects (ten, one RustFS ignore), namespace host (three) +and controller-archive suites, plus all-target strict Clippy and repository checks. +The first controller target invocation used an incorrect name and was rejected +before execution; the corrected target passes. Actual streams and five executable +copies are retained under +.cellar/cell-groups-evidence/CG-07-closed-manifest-pins-fixed-01/ in the primary +checkout and its workspace-volume backup. This fix is committed at 51d136fb +and published on PR 160 through 110b7a4e, including the reviewed baseline. +Both checkouts share that base; every preserved WIP file and all eight user +diagrams match their pre-sync hashes. It does not qualify the new Node/SDK/fleet +pipeline or another RustFS drill. + +The cutover's latest locked library check still fails with 114 unconverted fleet, +query-wire and predecessor-host integration errors. Its new Node shutdown, +resource and transfer modules emit no diagnostics in that run, which is not an +executable qualification. The earlier missing import and recursive cleanup-type +graph were corrected without increasing recursion, stack, quotas or deadlines. +Layout, native dependency and documentation checks pass after the lifecycle +changes. The new lifecycle/stop-proof assertions remain unrun. CG-07 stays IN_PROGRESS; +finish whole-group fleet authority and public routing/ingress, convert their real +consumers, then run all five original acceptance cases. CG-09–16 and the final +combined release matrix remain required. + +## CG-07 canonical startup and logical SDK cutover work + +CG-07 remains IN_PROGRESS. The unpublished spare checkout now replaces the +canonical startup's per-cell birth loop with namespace capabilities, an explicit +initial range plan and whole-group assignments. Assignments resolve permanent +group reservations and recheck directory selection before bootstrap. Startup +recovers complete selected roots through the existing appliers before attaching +shared owners, and refuses retained groups missing from its placement inventory. +There is no legacy/shared startup flag. The published PR checkpoint is +110b7a4e; this incomplete cutover has not been pushed. + +The new SDK uses logical CellHistory, CellVersion and CommitWitness values and +separates administrative group handles. It owns joined closure independently of +response cancellation and explicitly reports unobserved mutation outcomes rather +than manufacturing clean shutdown or deletion proof. Materialized and extent +namespaces share node image inventories and SQL workers; independent archive +verification uses separate shared inventories. Startup opens only existing +namespace-owned archive bindings and refuses missing identities or counters. + +Public admission and application-work registration now share the Node's short +maintenance/drain barrier. Commands, enrollments and queries retain their work +leases through queued ownership, unresolved completion and undelivered replies. +Remote query ingress is being moved onto the same controller queue. Gateway +private paths derive from logical history; namespace/schema and exact controller +ownership remain checked. Native effect endpoints now use weak logical handles +instead of looking up per-cell physical bindings. + +Inspection found a further query-expiry gap: async timeout can precede physical +callback completion. The new work transfers upstream admission into that SQL +reader and tracks its physical lifetime separately. The held-query fixture now +also exercises an expired response, global admission retention, work visibility, +cancelled close and release after callback exit. Directory regressions additionally +cover history-based placement lookup, wrong namespace/incarnation and a superseded +range. These new assertions have not run while the canonical library is broken. + +The first locked library check failed with 132 compiler errors. They primarily +identify unconverted fleet, maintenance, query-wire, resource and shutdown +consumers of the old SDK types. A second check after the physical-query and +native-effect changes failed with 117 errors; the effect adapter's fourteen +old-handle errors are removed. Neither run qualifies a build or executable. +The retained streams, source manifests and new +source copies are under +`.cellar/cell-groups-evidence/CG-07-canonical-startup-advance-01/`. +This is diagnostic evidence, not qualification. Subsequent runtime edits require +another check and all five original CG-07 acceptance cases. Managed namespaces +currently refuse with ManagedAdmissionRequired before catalog writes; whole-group +managed lifecycle is unfinished. No CG task status or release gate was relaxed. + +The next critical work is actual Node shutdown/resource/drain conversion and +whole-group fleet authority, followed by public remote routing and ingress, +canonical native fixtures, Axum/admin/CLI/examples and the unchanged CG-07 gate. +CG-09 through CG-16 remain required for the end-to-end hard cutover. + +## CG-07 namespace host component + +CG-07 remains IN_PROGRESS. CellGroupHost now binds compiled namespace/class +registrations to an exact existing shared controller, runtime, transport and SQL +engine. Its immutable registry has explicit namespace and byte limits; the byte +reservation includes configured invocation, route-cache and directory-cache +capacities. Cell handles contain logical CellHistory and share that registry. +Obtaining or cloning them neither enrolls cells nor adds permanent cell bindings. +The component does not create another consensus algorithm or a legacy mode. + +Namespace SQL resources validate schema, image backend, execution limits and +absolute private paths. Materialized directories and extent caches derive their +suffix from the full logical history digest, retaining that identity independently +of group placement. Only admitted workers open SQL. The host refuses reentrant +callback host calls and requires configured matching effect ownership for classes +with private background commands. These bindings grant no leader or bootstrap +authority; every operation still obtains runtime/catalog authority. + +Commands and queries enter the existing bounded controller queue synchronously. +Unpolled or disconnected query responses retain admission and physical callback +ownership; close seals all handles and joins the controller's SQL/transport/runtime +owners. Enrollment now captures its original deadline when construction is called, +so delayed polling cannot renew it. Receipt lookup preserves the exact command +binding and returns ambiguous absence without releasing a producer. + +All ten checkpoint commands pass. Three new native host cases pass both image +forms over real three-voter engines/TLS and independent SQLite state. They enroll +and mutate eight cells under a one-group limit, verify every replica's values and +logical observations, reject wrong tenant/history/token bindings, retain exact +receipts without callback replay and keep registry bytes constant after obtaining +more handles. A held query exercises dropped response admission, global overload, +cancelled close and engine-replacement refusal until the physical callback exits. +Duplicate namespace, foreign controller and metadata exhaustion refuse without +closing the healthy controller. Delayed enrollment expires before initialization. + +The full execution target passes 165 cases (two child-process fixture entries +ignored by the ordinary run), archive-controller passes sixteen (twenty explicit +RustFS ignores), and effects pass ten (one explicit RustFS ignore). Locked host +check, all-target strict Clippy, format/layout, dependency and documentation checks +pass. One SQL worker, one queued item and one pending cell per group remain in the +native fixture; its existing node/group byte ceilings are unchanged. The fixture +now explicitly declares its namespace image backend for the new host boundary. + +Record: .cellar/cell-groups-evidence/CG-07-namespace-host-01/record.json +(SHA256 61132e3338e6a729205708ae9cc9043156c51455fecf0ae16df6689939c135d5). It retains all 4685 source hashes, including +five new Rust files, the tracked source patch, copied changed/new files, actual +streams and four executables. An independent complete workspace-volume backup is +retained. No acceptance command, oracle, quota, timeout or validator was weakened. +This is native component qualification, without a new real RustFS, load, managed +lifecycle or complete public SDK/fleet claim. + +The host currently binds already bootstrapped services. The critical next step is +to replace canonical Node startup and SDK consumers: separate namespace/class +registration from group placement/provisioning and replicated cell enrollment; +carry logical versions/witnesses and explicit consistency through local and remote +routing, native/Axum ingress, CLI and examples. Managed capacity, repair, drain, +maintenance, stop/return/resume and inventory must operate on complete group +replicas while preserving CAS journals, permissions and physical debt. Keep the +spare checkout's two canonical startup edits and user diagrams. Do not restore +legacy startup, add a legacy/shared toggle or disguise groups as GroupScope.cell. +All five original native_cell_groups_public cases and the complete CG-07 gate +remain required. CG-09 through CG-16 and the full goal remain unfinished. + +## CG-07 synchronous shared-command admission + +CG-07 remains IN_PROGRESS. The shared controller now captures registered command +admission synchronously and queues construction before returning its response +waiter. Its existing node worker owns preparation, routing, SQL and publication; +an unpolled or dropped response cannot remove that work. Capture retains the +original absolute deadline before queueing. The dispatcher also reserves its +invocation budget when command construction is called, and an unpolled +construction cannot defer its timeout or bypass its configured capacity. + +Preparation failures before SQL have no unresolved producer to quarantine. +Once SQL enters, the existing exact reconciliation, sticky ambiguity, receipt, +physical join and group quarantine rules remain intact. No worker per cell or +second execution queue is introduced. All original limits and lifecycle assertions +remain unchanged. Private effect commands still use their sealed registration. + +All nine focused-checkpoint commands pass with 4680 source hashes and three +retained executables. The new exact regression passes both image modes. The full +execution target passes 165 cases, with zero failures and two ignored child-process +entry points controlled by their parent crash tests. Effects pass ten cases with +one explicit RustFS ignore; archive-controller passes sixteen with twenty explicit +RustFS ignores. All-target locked strict Clippy, format, layout, dependency and +documentation checks pass. These native suites do not establish a new real-backend, +public SDK/fleet, managed lifecycle, load or release qualification. + +Focused record: .cellar/cell-groups-evidence/CG-07-command-admission-fixed-02/record.json +(SHA256 5a44b6f76da74b72eddf45e34cd93c7ec1eb9cf12b67e41acc43fe49331920f6). The record binds the reviewed base revision plus its exact source +patch, all source hashes, actual streams and executables. Artifacts and an +independent complete backup are retained on the workspace volume. + +The valid baseline fails because unpolled construction retains zero invocation +slots instead of one. Its exact source and failing executable are preserved under +CG-07-command-admission-baseline-02. The earlier baseline-01 omitted a key-helper +argument and executed no case. Fixed-01 passed the first mode's constructor +assertions, then its fixture attempted a second SQL owner and correctly received +AlreadyExists. Fixed-02 inspects the existing controller pool with the same +zero-active-cell assertion. Those failures and their analyses remain retained; +no acceptance oracle, timeout, ownership check or resource cap was weakened. + +The critical next vertical slice is canonical shared host/SDK startup: bounded +namespace/class registrations, group range placement and provisioning, replicated +cell enrollment, logical cell handles/versions/witnesses and explicit consistency. +Then convert managed capacity, repair, drain, maintenance, stop/return/resume and +inventory to whole group replicas with separate cell/image occupancy. Preserve +CAS journals, permissions and disruption debt. The spare checkout's two incomplete +canonical node-service/startup edits remain owned work; never restore legacy +startup or add a legacy/shared flag. All five original public CG-07 cases and its +complete gate remain required before DONE. + +## CG-08 complete effects, cache and RustFS gate + +CG-08 is DONE at its bounded shared-runtime effects/cache boundary. Qualified +source: 4ea70e483e84ca88d77ca062278edb5ff2389646, including origin/main +81cbbc2f72f302751dd471cf06cf51394146ac63. All twenty-two commands pass: the unchanged +ten-command task gate, five additional exact effects cases, twenty-four library +regressions, the complete shared execution target (164 passed, 2 ignored), +the affected archive-controller suite (16 passed, 20 explicit RustFS ignores), +and dependency/documentation checks. The task's reviewed source anchors still +match. Locked all-target check/Clippy, format and layout pass; no source edits +occurred during the run. + +The ten exact native effects cases cover both image forms where applicable: +logical delivery/proof identity across physical bindings, exact cold reopening, +due discovery without client traffic, hot/cold fairness, cancellation and actual +leader transfer, lost receiver replies, held callback/delivery shutdown, S3 claim +and completion coverage, and real materialized cleanup failure. The cleanup-fault +case retains ambiguous outcome and one-group capacity, refuses another group's +registration, and returns the original receipt on the new leader without replay. +No worker, timer or SQL connection is retained for every idle cell. + +Serial RustFS at the preceding 8a source exposed a committed-output race despite +its passing native gate. SQL had retired before fresh output authority became +available; retrying SQL reconciliation then produced missing-origin. The current +pending-output phase retains the exact receipt and archive proof, waits only for +a fresh serving fence under the original deadline, and preserves typed terminal +errors. It cannot resurrect a SQL producer, replay a callback or quarantine a +healthy group. The deterministic physical manifest-sync regression fails before +the fix and passes both image modes for new and duplicate receipt paths. Failed +setup attempts, the valid failing baseline and corrected passing snapshots are +retained under the CG-08-output-repro/fixed checkpoint directories. + +The explicit real RustFS case passes at this same source and exact effects +executable in both image modes. Seven signed S3 probes pass. Claim settlement +precedes delivery; final complete image/receipt closure is independently restored. +This is a host-native TLS fixture with the RustFS container in one local VM, +not a measured workload or independent-domain qualification. The owned backend +and dedicated VM are stopped, data retained; default Docker context stays colima. + +Task record: .cellar/cell-groups-evidence/CG-08-gates-4ea70e48-01/record.json +(SHA256 a072fb6fd8957b6afd5fdf4dffc88c312a0e34c2e1f5c4cfd81b4d2456ea9d0c). Effects executable SHA256: +85ceb50508a519e2b1fd5198f47969185438f77c30d09e9de6ab884fd6a7cce6. All 4680 tracked source hashes, stdout/stderr, +the original runner record and executable are retained and backed up on the +workspace volume. Commands with empty stdout reference their real nonempty +stderr as the generic output artifact; exact case outputs remain untouched. +The unchanged task evidence validator passes. Current RustFS evidence is under +.cellar/cell-groups-evidence/CG-08-effects-progress-783c270b/rustfs-4ea70e48/. +Earlier failing fixture runs and predecessor qualifications remain recorded. + +Recorded scope update: shared native fixture helpers under crates/cellar/tests/support/ +and the repeatable tools/multi-raft/ RustFS effects selector support the authorized +local verification. The original archive selector's twenty cases are unchanged. +Acceptance names, oracles, commands, limits and validator rules are unchanged. + +The next critical implementation is CG-07: replace canonical per-cell birth/host +startup and per-cell binding capacity with group placement and replicated +enrollment; carry logical cell versions/witnesses through SDK/ingress; convert +managed lifecycle and accounting to complete groups. Live transfers, topology, +hibernation, managed recovery, load/crash qualification and final legacy removal +remain required. This task gate does not qualify the public SDK/fleet cutover. + +## Earlier CG-08 implementation checkpoints + +The complete successor gate passes all twenty commands at 8a4c5ae0, including +164 execution cases with zero failures and two explicit RustFS ignores. The +subsequent fully serial real RustFS case exposed a production output-phase race: +SQL had committed and retired, but a later group entry temporarily prevented +fresh serving authority after archive settlement. The driver retried SQL +reconciliation with no remaining origin and returned an ambiguous missing-origin +error. The passing native gate and the serial RustFS failure remain retained; +the owned backend and VM stopped normally, with data and default context intact. + +The correction retains the exact receipt and archive proof in a distinct pending +output phase. Progress retries only current authority under the original deadline; +it cannot reopen SQL, replay a callback or turn completed SQL into quarantine +debt. Both newly committed commands and duplicate receipts use this phase. The +regression holds a real manifest-directory sync on another prepared cell under +the original one-item SQL limit. It establishes committed greater than applied, +requires the first S3 response to remain pending, then checks its exact result, +receipt retry, one callback execution, healthy group and every replica's applied +successor. The valid baseline fails before the correction; the focused corrected +case passes materialized and extent modes for both entry paths in 2.29 seconds. +Its snapshots, executable and real streams are retained under +.cellar/cell-groups-evidence/CG-08-output-repro-04/ and +.cellar/cell-groups-evidence/CG-08-output-fixed-02/. Earlier fixture construction +failures remain recorded separately. No production refusal, deadline or quota +was weakened. The new runtime requires a fresh complete gate and serial RustFS +qualification before CG-08 can become DONE. + +At e8edfd12, all ten unchanged required commands pass, along with four extra +effects cases and twenty-four library regressions. The same-source/executable +real RustFS effects case passes both image modes with all seven signed probes. +Its record, logs and executable are retained under +.cellar/cell-groups-evidence/CG-08-effects-progress-783c270b/rustfs-e8edfd12/. +The owned backend and dedicated VM are stopped; data and default context remain. + +The expanded execution suite passes 162 cases, fails two and explicitly ignores +two RustFS cases. Both failures use an old summary fixture whose outbox table +contains only next_at; lane-head discovery now also requires lane and sequence. +The fixture correction adds those identities and a third, immediately due row +blocked behind its lane's older sequence. Exact summary assertions still require +the other lane's time 40 and now all three pending rows. Both original failed +cases pass after this test-only correction, in both image forms. Production +validation, resource limits, deadlines and acceptance oracles stay intact. +The failed full gate, source inputs, both regression executables and real streams +remain under .cellar/cell-groups-evidence/CG-08-gates-e8edfd12-01/ and are backed up. +The corrected partial run lives under +.cellar/cell-groups-evidence/CG-08-apply-summary-fixture-e8edfd12-01/. +CG-08 stays IN_PROGRESS until the complete gate passes at the successor source. + +The successor implementation is under qualification on codex/shared-cell-groups. +The committed due-index foundation is 783c270b. The node controller now owns fair +group/cell discovery, exact committed-image scans, private alarm/outbox commands, +claim durability, fenced delivery and joined shutdown. No cell owns a timer or +resident execution worker. The expanded predecessor compile check passes. + +All five predecessor execution cases now pass in both image modes: alarms, lost +receiver replies, actual leader transfer, delivery shutdown and held alarm +shutdown. The successful native-10 run passes five cases in 7.11 seconds after an +18.44-second build. Its retained executable SHA256 is +e9fa57084a95fe02e0af8d8ddaceda5ccd0b8b9efd00c1b1ee92a61dcc06a8b7. +The preceding run passes four cases but times out on an ordinary follower +pre-vote under the healthy leader lease. The test-only correction uses the real +TimeoutNow transfer and caught-up target check inside the original deadline, +preserving every custody/state oracle. Both source-bound records, executables +and real logs remain under +.cellar/cell-groups-evidence/CG-08-effects-progress-783c270b/. + +The native_cell_groups_effects target now passes eight cases with zero failures +and one explicitly ignored RustFS case, in 33.05 seconds after a 3m45s build. Each +case exercises materialized and extent modes, including all five original CG-08 +case names. The source-bound native-11 record and executable are retained locally; +executable SHA256: +54d1af2d19bba8aacf206f23d04ca3fb1f3b23f1dafc554bbd49dc079760ffb9. + +Cold reopening checks the committed base, exact receipt retry, an older coherent +read and an unchanged Raft term. Discovery leaves 32 cold cells unopened; a cell +re-arming through four alarm generations rotates behind the other due cell. +Identity qualification boots two independent physical group bindings, comparing +exact signed deliveries and receiver proofs while suppressing duplicate execution +under each. This does not qualify live CG-09 transfer. The normal S3 adapter case +requires archived claim coverage before send and independently verifies the final +complete image/receipt closure. Its explicit local RustFS variant remains unrun. + +The cache review found materialized settlement closed SQLite while retaining its +private database and LTX files. Shared-cell close now discards those bounded +scratch paths before releasing the private lock, including failed opens and +post-open authority revocation. Sealed images and uncertain proposal custody are +separate. The passing cold-reopen case asserts only the lock inode remains before +injecting a bogus disposable database and checking exact reconstruction. Affected +enrollment regressions remain pending. + +The cleanup I/O failure case now passes. It holds the real private cleanup after +commit, transfers leadership through TimeoutNow, returns a typed ambiguous storage +error, and retains the failed invocation and quarantined group's capacity. With +the group/apply limits tightened to one, an actual fresh registration refuses +with Overloaded. The new leader returns the original receipt without callback +replay; both serving replicas retain the value and joined shutdown releases work. +The first test attempted to count retained capacity using the active-group metric; +its failure remains as native-12. Native-13 corrects the active/quarantined +observation and proves quota refusal directly without raising any limit. + +The real RustFS effects case passes in both image forms at e4901835, with seven +signed S3 probes, exact source/backend/executable checks and independently restored +final images/receipts. Its executable SHA256 remains +54d1af2d19bba8aacf206f23d04ca3fb1f3b23f1dafc554bbd49dc079760ffb9. +The backend and dedicated VM are stopped, data retained, and default Docker context +remains colima. The fixed --suite effects selection preserves the original twenty +archive cases. Twenty-four source-matched library regressions also pass. Strict +Clippy initially identified sibling-only fixture functions as unused in the +execution target; its shared fixture now carries the same scoped annotation as +other native targets. Production lint, test oracles and qualification commands +remain unchanged. These source-scoped records are retained under +.cellar/cell-groups-evidence/CG-08-effects-progress-783c270b/. + +Private-operation discovery now bounds its SQLite projection before allocating +the scheduling preference string. Its corruption/permit regression and strict +Clippy pass; the complete successor CG-08 evidence gate remains required. The later CG-07 +public SDK/fleet cutover is still unqualified. CG-06's prior +source-scoped closure remains intact; the full hard cutover is unfinished. + +## CG-06 complete native and local disaster-recovery gate + +CG-06 is DONE at its declared unmanaged archive/reclamation/recovery boundary. +Qualified native source: e32115aa3218dc6eaa7ed53e134f413f26305938, containing +origin/main 81cbbc2f72f302751dd471cf06cf51394146ac63. Its executable inputs match +the Docker-qualified f8aacd05 source; their only differences are plan documents. +CG-07–16 and the complete hard cutover remain unfinished. + +All 123 unique commands in the unchanged CG-01–06 gate refresh pass: 110 exact +native cases, nineteen pure-logic unit tests, all-target locked check/Clippy, +format/layout over 1990 Rust files, native dependency and documentation checks. +Six copied test executables and 3257 source hashes bind the new records. The +unavailable historical CG-01–05 raw records have been superseded by fresh exact +native gate evidence at this source; their earlier results remain historical. +No acceptance command, oracle or validator was weakened. + +Current records live under +.cellar/cell-groups-evidence/CG-01-06-gates-e32115aa-02/ and are backed up on the +workspace volume. CG-06-record.json SHA256: +29cfff144bb38f5cda6968e5a8d2cf163ff440df0562ec8948059176ab7827b8. +Its native executable SHA256: +8b18b5677e07862c892dcea3a02fceb36a11234a2636b20d6efae87f4bddd5ac. +Each task record independently passes the unchanged source/artifact/command +validator. Full package validation passes with six DONE tasks, all source-bound +records and 44 local links; no unavailable-record exemption is used. + +The reclaim case originally passed but its stderr observations interrupted the +libtest result line in the merged log. That original output is retained. The +unchanged case passes again with stdout and stderr retained and hashed separately; +the strict one-case result requirement stays intact. The original gate runner's +initial unsupported ls-tree exclude-pathspec failure is also retained. + +Both complete Docker/RustFS disk-loss drills below pass, including exact physical +recovery and retained restart. These establish local correctness only; managed +recovery, external effects, independent failure domains, sustained load and final +SDK/fleet cutover belong to later gates. The dedicated VM is stopped and data +retained. CG-08's bounded effect scheduler and safe cache lifetimes now have +priority, followed by CG-07 public SDK/fleet cutover. + +## CG-06 Docker-fenced, all-voter-disks-deleted recovery + +Source: f8aacd052591b7d3a31ea7ca426bef10d9d845cf, containing freshly fetched +origin/main 81cbbc2f72f302751dd471cf06cf51394146ac63. At this source CG-06 +remained IN_PROGRESS; the complete gate above closes it. + +The actual shared-group Docker/RustFS drill now passes in both materialized and +extent modes. Three voter processes replicate three independent SQLite cells in +one group. Four registered source commands receive S3 acknowledgements. The +trusted sole-launcher controller fsyncs permanent generation revocation, confirms +all three immutable old containers and voter volumes are absent, and only then +signs the exact recovery request. Reloading the controller refuses the revoked +generation; missing/forged certificates and recreated old disks are refused. +Its private key and launcher ledger never enter a voter or operator container. + +Three cold S3 restorations select the same complete genesis binding under fresh +group/cell/node/member/disk/engine identities. Pending application writes and all +read modes refuse. Actual TLS Raft replication commits and applies activation +before serving. Exact old receipt results preserve their original incarnation +without callbacks, Quorum retries retain stored S3 minimums, expired client floors +and conflicts refuse, and old tokens remain valid only for their old history. +New S3 writes succeed. Every fresh voter stops and resumes its retained disk with +new process identities; imported and new receipts still avoid replay, another new +write succeeds, and all three replicas' values, versions and incarnations match. +Every remaining owned process joins normally; both runs retain their object and +new voter volumes. All voter and operator containers capture one CPU/3 GiB with +no extra swap; each RustFS captures one CPU/2 GiB. + +The runner and native node live in tools/multi-raft/group_disaster.py and +tools/multi-raft/node/. Configurations are copied through the Docker API with +exact byte readback before startup. A real Docker regression verifies snapshot +isolation from later host replacements and exclusion of the controller key. +A separate 150-read probe reproduced 38 truncated JSON reads through mutable +virtiofs bind mounts; that transport is no longer used. Retained failed runs also +cover the initial unshared output path, wildcard transport endpoint mismatch and +an oracle that incorrectly expected genesis version after a later write. +The corrected oracle keeps exact original results and checks the current cell +observation (alpha version 2 after the first new write, beta version 1), with zero +replayed callbacks. No production assertion or protocol was weakened. + +Scoped record: .cellar/cell-groups-evidence/CG-06-docker-f8aacd05/qualification.json +(SHA256 08cf2112d4f91519c00d481a6cea3f8c0e378a242ecef258ac67a182c265e77d). +Deployed executable SHA256: +ea5e554d781cdd8004891d28ec8ecfb0abb0664ac5e372a135f0585842045be8. +The record binds both reports, source manifest, raw ledger/log hashes and actual +container/VM resources. Private credentials/signing material stay outside Git; +sanitized scoped evidence is copied locally and backed up on the workspace volume. +This is a local correctness result, without managed recovery, external effects, +sustained load, independent hosts, Amazon S3 or a recovery-time SLA claim. + +The original five-case CG-06 native gate and all-target checks still need a fresh +complete record. Historical CG-01–05 raw records remain unavailable; package +validation refuses the missing CG-01 record. Restore or rerun the original gates, +without changing validation or historical task boundaries. Then prioritize CG-08 +bounded effect discovery/execution and exact-root cold-cache reopening, followed +by CG-07 public SDK/fleet cutover. + +## Earlier CG-06 native quorum and receipt checkpoint + +Source: 9a0bad9124f0089055a0ba63b2952e5ef5e77853, including origin/main 81cbbc2f72f302751dd471cf06cf51394146ac63. CG-06 remains IN_PROGRESS. +Fresh recovery now installs the complete imported root and physical Raft genesis +in one synced engine publication. Permanent seed records bind the exact recovery +request, genesis commitment, node, member, disk and engine. Ordinary bootstrap +refuses recovered births; recovered startup revalidates the seed and external +fencing authority. Replacement publication retains the recovery lock until a +catalog-bound activation is committed and applied through the fresh quorum. + +The original `fenced_group_seed_new_incarnations` oracle now exercises registered +commands. It writes actual receipts and an expired client floor after the base +checkpoint, archives their LTX tails, and deletes every old native voter directory, +private SQL files, extent store and the previous +verification cache, then seeds and activates three fresh voters. Production TLS +transports, controllers, SQL workers and the replacement group's archive owner +serve retries. Exact opaque results survive without callback replay; Quorum +retries honor stored S3 minimums through actual replacement-history coverage. +Expired sequences and conflicting inputs refuse. Old tokens remain valid for +their original history at the same wall time and fail with WrongHistory for the +fresh history. A new registered write and retained-quorum restart preserve the +imported receipts and client floors. + +At this source, archive/catalog suites pass 47 cases; five explicitly ignored +RustFS cases pass when selected against the owned real backend, each in both +materialized and extent modes. Seven signed S3 operations verify conditional +creation, CAS and readback behavior. All-target locked Cellar/logic Clippy, +format/layout over 1985 Rust files, native dependency and documentation checks +pass. The RustFS backend and dedicated VM are stopped with data retained; the +default Docker context remains colima. + +Scoped record: .cellar/cell-groups-evidence/CG-06-recovery-genesis-settlement/receipt-recovery-qualification.json (SHA256 eb0efe6af7d4e6bed3a9801d821e0ea8e48058f7ce949cdc01912a74642fe25d). Records bind the tested source +inputs, copied test artifact, exact selections, logs and backend cleanup. This is +a physical/receipt recovery checkpoint, not a full CG-06 DONE or release record. +Earlier failed fixture builds, capture-admission refusal, extent-schema mismatch +and cache-admission failure remain recorded beside the corrected passing run. +The catalog golden now verifies the required recovery-origin wrapper and refuses +the older bare birth record; no compatibility reader was added. + +The remaining feature gate is actual Docker fencing. The current native and +RustFS drills use an explicit contract-test authority; they do not qualify +permanent infrastructure launch/restart revocation. The existing Docker harness +uses the older per-cell runtime. Replace its voter path with actual shared-group +nodes, retain its ownership checks/signing flow, revoke the old deployment in a +durable sole-launcher policy, erase all owned voter containers and volumes, then +cold-recover through a fresh quorum. Reopening the launcher must still refuse +the old deployment. Complete that drill and the original CG-06 gate before DONE. + +Historical CG-01–05 raw gate directories are currently unavailable in this +checkout. Their prior source-scoped results remain historical; complete package +validation requires restoring those records or rerunning the unchanged exact +gates. This checkpoint does not replace them or weaken the validator. The 37 validator +unit tests pass and all 24 reviewed source anchors match; complete validation +currently refuses the missing CG-01 record. +After CG-06, prioritize CG-08 bounded due-work scheduling and qualification of +the current close-on-settle SQL policy, then CG-07 public SDK/fleet cutover. + +## CG-06 fresh-history data import and pending admission + +Source: 7052d218d60181d7b0bb5a4c12686e80e294a61c. CG-06 remains IN_PROGRESS. A fenced, +complete source cut can now be prepared into dedicated replacement stores using +`CellGroupRecoveryPool::prepare`. Every cell receives the replacement group's +fresh incarnation, sequence 1, a new enrollment record and a genesis witness. +SQLite bytes stay exact; materialized generation identity is rederived and extent +history metadata is rebound while verified payload identities are reused. The +source intent and its retained remote hold preserve original-witness provenance. +No application callback or Raft engine is created by this data preparation. + +The replacement manifest carries mandatory nullable recovery lifecycle metadata. +Pending roots refuse writes, enrollment, stale/strong reads, receipt reconciliation, +membership changes and learner seed authority. No-ops preserve pending state. +Snapshot receive and selected-root reopen retain the full marker; storage refuses +forgetting or replacing recovery lineage. Deterministic apply checks the exact +activation operation, proposal and seed commitment, preserves the first activation +on an identical retry and rejects conflicting activation. A catalog-bound leader +activation admission API and actual replacement-quorum activation remain required. + +One node-wide import pool separately bounds cells, restored bytes, metadata and +image workspace. Result/root clones retain import admission. Cancellation during +actual disk sync retains source pins and permits until entered physical work +finishes; shutdown joins it. Clones of the fenced data proof share one verified +cut and its existing admission when preparing multiple destinations. + +At this source, seven native shared-group suites pass 246 cases; 36 recovery-related +unit cases and eleven legacy disaster cases also pass. The exact real RustFS +import case passes in both image forms with seven signed S3 probes. It checks +all-cell byte equality, fresh identities/witnesses, old-token rejection, admission +limits and stable retries. All-target locked Cellar/logic Clippy, format/layout +(1971 Rust files), native dependency/docs/preflight/plan checks, 37 plan tests and +five backend-controller tests pass. The owned RustFS backend/profile are stopped +with data retained and the default Docker context preserved. Independent audit +checks 2588 source inputs, Git blobs/symlinks, copied executables, exact case +selection and credential exclusion. + +Evidence: `.cellar/cell-groups-evidence/CG-06-recovery-import/qualification/record.json` +(SHA256 `1c7ea62c6bd5c1e81357961c4acba22bb2820d89bc3556ec3bcaf82a43e1c0a3`); independent audit +`1eae08c2bd03eb6fc42d81ae665533716aff5c90cc9217e92221733119d5b75d`. The initial test-fixture private-import compilation error and +its interrupted compiler are retained in `focus-01/`; corrected source passes. +The initial uncommitted-baseline preflight refusal is retained and the pinned +source preflight passes. Earlier socket/deadline/host-pressure and slow-reader +limitations remain historical; no causal fix is claimed for them. + +This follows the repository's internal marker-1 hard cutover: older intermediate +cell-group manifest/archive shapes have no compatibility reader. Current native +and RustFS fixtures use the complete new shape. Qualification here covers data +preparation and admission barriers. The replacement still has no engine genesis, +node/member/disk/engine seed binding, fenced publication or serving permission. +Docker voter fencing, fresh-quorum serving after erasing every voter disk, actual +registered-command duplicate receipt recovery and the original +`fenced_group_seed_new_incarnations` oracle remain pending. + +Next, consume the complete typed preparation through atomic fresh-genesis storage +and durable physical identity claims. Ordinary bootstrap must refuse an empty or +missing replacement seed after publication. Publish exactly once under the +existing recovery lock, then admit activation only with a catalog-bound permit +and prove its actual commit/apply through a fresh quorum. Reverify fencing on +resume and keep the source hold while its provenance is referenced. Complete the +accepted Docker authority drill and the original full CG-06 gate before marking +this task DONE. Then CG-08 due effects and safe cold-cache eviction precede CG-07 +public SDK/fleet cutover; transfer, split/merge, placement, hibernation, load/crash +qualification and final legacy removal remain required. + +## CG-06 exact whole-group recovery intents and fencing barrier + +Source: 65c5176975907fb52a53deca58e1ae1f5ba7adec. CG-06 remains IN_PROGRESS. The new +`RangeDirectory::fence_group_recovery` boundary independently verifies an exact +held complete-group archive before claiming the namespace or asking infrastructure +to fence it. Canonical operator bytes bind the source birth, archive hold, +replacement identities and original topology mutation. Unknown outcomes retain +that exact request; no latest-head substitution, abort or unfence is supplied. + +The recovery lock shares the directory-head CAS with topology publication and +the namespace lifecycle CAS with fleet import. Old source bootstrap and competing +topology mutations refuse once selected. The existing fencing-authority contract +accepts a typed whole-group request, requires permanent external inventory/launch +fencing and independently verifies the winning bounded receipt, including after +reopening the catalog. Archive corruption prevents any fencing call. The current +static unmanaged boundary refuses joint, repair and fleet membership records. + +Both image forms pass exact resume and topology-race scenarios against memory +and real RustFS. The actual S3 case covers four scenarios and seven signed probes, +including lost CAS/fence responses, failed independent verification, immutable +intent resume, all-cell image checks and startup/import/publication refusals. +All five native shared-group suites pass (211 cases), as do eleven legacy +disaster cases, locked all-target Cellar/logic Clippy, formatting/layout over 1966 +Rust files, native dependency/docs/preflight/plan checks, 37 plan tests and five +backend-controller tests. Independent audit verifies 2583 +source hashes, copied executables, exact selection and credential exclusion. +Owned RustFS backends/profiles are stopped with data retained. + +Two stack failures are retained with copied executables and diagnosis. The new +fixture's recovery adapter is boxed; the existing extent helper uses its actual +schema contract. A deterministic learner-replacement regression from by-value +recovery-lock metadata is corrected by boxing that rare metadata in the common +directory head. The exact default-stack test passes; canonical record bytes, +stack limits and test assertions are unchanged by the layout correction. + +Initial controller-suite socket-address/command-deadline failures and a checkpoint +fixture-driver deadline remain retained, not claimed fixed. Eight controller and +four checkpoint comparisons pass on both copied source revisions. A contemporaneous +extreme host-load snapshot is context, not proof of causation. The full unchanged +source repeat passes; the earlier 8dfe9ff3 slow-reader timing limitation remains. +This is not sustained-load, independent-domain, Docker voter-fence or deleted-voter +replacement-serving qualification. + +Evidence: `.cellar/cell-groups-evidence/CG-06-recovery-fence/`. Qualification record +SHA256: `0c94cfd5071d49e59bdefd0d1fae78aff15237602eebcfc4ade48283d50b254c`. Independent audit SHA256: `f2b13f7590220b06a7c23368563b176ce2abcf5d9506869b21538dde20db2c12`. +Prior outage/dependency/reclamation evidence remains unchanged. + +The critical next feature is fresh-history image/receipt/client-floor restoration, +fresh engine genesis, one fenced replacement publication, committed activation and +fresh-quorum admission. The returned `FencedCellGroupRecovery` still contains +original-history data and grants no engine, voting or serving permission. Qualify +Docker permanent restart/launch revocation including later/removed voters, delete +all owned voter disks, restore from real RustFS, reject old tokens, prove no +callback replay, perform new writes and restart the retained replacement quorum. +Only then run the full original CG-06 gate. CG-08 effects/cache/result lifetimes +precede CG-07 SDK/fleet cutover; CG-09–16 remain mandatory. + +## CG-06 bounded Quorum progress during a full S3 backend outage + +Source: 1159e9358c106fe1260c3479759e909e413257a4. CG-06 remains IN_PROGRESS. The +original quorum_continues_bounded_s3_outage case now runs in both image forms, +alongside an exact real RustFS counterpart. It exercises existing static-group +local-quorum durability, routing, owned SQL, archive settlement and reclamation +without changing production behavior or introducing a cached-authority shortcut. + +Each scenario explicitly selects a nine-entry group ceiling with one progress +slot. All three voters first verify the actual retained indexes 2, 3 and 4: one +election no-op and two enrollments after genesis snapshot index 1. A warm local +command and remote receipt retry establish current routes and archived state. +The fixture then disables the adapter shared by catalog and archive; the RustFS +counterpart instead requests actual Docker stop/start from its owning runner, +never toggling that adapter's outage flag. Actual S3 probe I/O fails while the +backend is stopped and returns the same probe bytes after restart. + +Warm local commands continue through the actual quorum. A later S3 command +commits locally but cannot settle; four subsequent Quorum mutations reach the +exact retained-entry ceiling, preserving every A/B value and logical version on +all three voters. The next command and initialization refuse before their +callbacks. A known Quorum receipt remains available at capacity. S3 output +expires with its original deadline, and a retry asking for Quorum still obeys +the stored S3 minimum. The healthy group is not quarantined. + +A separate, already-warmed remote receiver still performs fresh catalog GETs +during the outage and refuses before its callback. The same originating +invocation survives that error and later retries with its exact captured inputs, +identity, original deadline and attempt budget. No failed response becomes a +new-command or non-commit proof. + +An exact prepared checkpoint cannot publish beyond verified coverage while S3 is +stopped; the selected checkpoint, retained prefix and debt remain unchanged. +Restart uses the same RustFS container, image, volume, endpoint and resource +limits. The durable journal catches up, independent complete recovery verifies +both A and B, and the original failed S3 command returns its exact result +without callback replay. The completed verifier releases its one-source pool +before subsequent settlement. The same prepared checkpoint then selects +successfully; every voter selects its exact complete cut, retires actual debt to +zero and releases physical source quota before reuse. New local work and the +retained remote invocation commit, with eight total callbacks and the expected +voter state. + +The narrow Python controller accepts only four ordered stop/start transitions +for the two image modes. It checks the captured container ID, owner label, +pinned image, CPU/memory, port bindings and mounts before and after every +operation, uses explicit Docker context, and retains actual state observations. +Requests/replies are bounded, atomic local files. The native fixture performs no +Docker process calls. Failed children are joined and preserve output; cleanup +restores only the same verified owned backend. Five protocol/ownership tests +reject foreign or substituted containers and unexpected actions, verify the +fixed transitions, and retain failed-child output. + +Twenty exact real RustFS cases pass in both image forms with seven signed +protocol probes and four recorded actual backend transitions. Twenty-three +regression/check commands pass with 270 Rust cases, seven copied executables and +2580 source hashes. All four shared-group native suites, all 22 storage cases, +locked all-target Cellar/logic/LTX Clippy, format/layout, +dependency/docs/preflight/plan checks, 37 plan tests, five controller tests and +system-Python hashing pass. Source/artifact/selection/credential and +backend-transition audit passes under +.cellar/cell-groups-evidence/CG-06-full-backend-outage/. + +The original first-source failure and copied-binary reproduction remain +retained: an expected count of four came from the older three-cell fixture, +whereas registered setup enrolls two cells. The successor asserts the physical +log inventory on every voter and adds one real mutation to reach the unchanged +nine-entry ceiling. No original oracle, quota, deadline, payload or production +behavior was weakened. The first real stop/start run and a copied-binary run on +a fresh backend both return503 for the first probe GET after health200. The +successor retries only idempotent Generic read failures within the retained +remote invocation original deadline; missing/authentication failures still +refuse and exact data must match before proceeding. The outage adapter remains +noRetry and no mutation retry or production behavior changes. Both failed real +traces, container observations, copied executable and stopped backend data +remain retained. Earlier qualifications and the unresolved 8dfe9ff3 slow-reader +timing limitation remain retained. + +Reflection: four original CG-06 cases now pass: complete-group dependencies, +bounded S3 outage, archive tails and physical reclaim/restart. The remaining +original case is fenced_group_seed_new_incarnations: permanent old-host fencing, +complete old-voter inventory including later joins, fresh group/cell histories, +exact selected-cut results, old-token rejection and a serving replacement +quorum. Use the accepted fencing-authority API plus Docker qualification. The +full CG-06 gate is still pending. CG-08 lifetimes precede CG-07 production +SDK/fleet cutover, followed by irreversible transitions, split/merge, placement, +hibernation, managed recovery, measured load and final legacy removal. This is +bounded static-group outage qualification, not a throughput benchmark, +independent-domain proof, global physical disk bound, voter-disk erasure or +completed hard cutover. + +## CG-06 complete-group dependency acknowledgement + +Source: `dc0ad3c245a2f9e3dd4f261003c868e9a43f19b8`. CG-06 remains IN_PROGRESS. +The original `group_archive_requires_all_dependencies` case and an exact real +RustFS counterpart now run delay and corruption phases in both image forms. This +qualifies the existing protected-object reuse, complete recovery and S3 +settlement paths; production behavior, markers, deadlines and existing resource +limits are unchanged. The recorded scope extension adds the owning +archive-dependency fixture and its shared test exports without changing any +original case or acceptance oracle. + +Each phase commits a real registered B mutation before selecting the complete +group archive. B now has different SQLite data from A. The fixture finds an +actual materialized image chunk or extent unique to B, outside A's entire image +closure, and checks its content digest. A separate recovery client persists and +fsyncs its exact acquire intent before obtaining a durable hold. It owns a +bounded two-operation I/O pool, one checkpoint receiver and one tail-operation +pool; the controller's original pools remain unchanged. + +The delay phase holds GETs for that exact B image object. A later registered A +command commits to the local quorum but its S3 output stays pending. Another A +command receives a Quorum acknowledgement, and all three voters expose the exact +A/B data and versions. The actual remote registry selection stays at the +preceding cut. Independent whole-group recovery is also pending with its one +receiver occupied. Releasing the GET permits complete historical recovery and +A's exact S3 result. Receivers and tail reservations return to zero before +subsequent work, and the known command resolves locally without callback replay +while preserving its stored S3 minimum. + +The corruption phase overwrites that actual stored B image object with +same-length, different-digest bytes and reads them back. Independent recovery +refuses with the exact object-digest failure. A later A entry still commits, but +its S3 output fails ambiguously through the typed archive error; the archive +owner stops and its selected head cannot advance. Another A Quorum command +succeeds, all voter data remains correct and the group is healthy. Repairing +only the injected bytes permits complete recovery through a fresh physical +receiver. It does not silently restart the stopped archive owner or lower the +original command's S3 minimum. Internal exact receipt lookup observes the +original result without callback replay and supplies no external S3 +acknowledgement. + +Nineteen exact real RustFS controller cases pass in both image forms, with seven +signed protocol probes. Twenty-two regression/check commands pass with 269 +distinct Rust cases across seven copied executables and 2576 source inputs. All +four native shared-group suites, all 22 storage cases, locked all-target +Cellar/logic/LTX Clippy, format/layout, dependency/docs/preflight/plan +validation, 37 Python tests and system-Python hashing pass. Independent +source/artifact/selection and credential-exclusion audit passes. Evidence under +`.cellar/cell-groups-evidence/CG-06-archive-dependencies/` binds raw outputs, +copied executables, orchestration, reviewed plan inputs and the preceding +physical-reclamation qualification. The physical purge/restart case also passes +at this source. Earlier retained failures and the unresolved `8dfe9ff3` +slow-reader timing limitation remain recorded at their original source +boundaries. + +Reflection: three of the five original CG-06 cases now pass: complete-group +dependencies, archive tails and physical reclaim/restart. The next critical work +is the original full-backend S3 outage case: exercise local cached and remote +fresh-discovery paths separately, establish the exact bounded Quorum admission +boundary, and refuse S3 outputs until complete coverage recovers. Preserve +catalog/epoch fencing; durable source sealing and activation are required before +introducing any new cached-authority shortcut. Fenced fresh-incarnation serving +from a replacement quorum remains the other original case. The accepted +fencing-authority API plus Docker qualification must inventory every possible +old voter, including later joins. CG-08 lifetimes precede CG-07 public SDK/fleet +cutover, followed by transitions, split/merge, placement, hibernation, managed +recovery, real load and final legacy removal. No full task, production cutover, +independent-domain qualification or zero-quorum disaster-serving claim is made. + +## CG-06 physical log reclamation and retained-disk recovery + +Source: `7741e4049704f3e1c6519b736e62deaf263c39a2`. CG-06 remains IN_PROGRESS. +The original `group_reclaim_restart_keeps_history` case now runs in both image +forms, together with an exact real RustFS counterpart. It uses the production +purge, checkpoint, image/page maintenance, SQL and recovery APIs. Production +behavior and marker-1 records are unchanged. The new scenario explicitly selects +4 KiB target log files, an 8 KiB purge threshold and manual purge so filesystem +observations cannot race automatic maintenance; all prior fixture defaults and +quotas remain unchanged. + +Two real groups share each engine. The primary commits 24 registered commands +across two SQLite cells, retaining exact results and alarm state. Independently +verified complete S3 coverage permits its exact checkpoint. Eight subsequent +entries remain in the local suffix during archive lag. The sibling retains its +entire uncheckpointed history. Purge must remove actual original Raft files and +reduce their aggregate file length, while both histories remain byte-for-byte +intact and an old image reader remains readable. A retained checkpoint streams +into an independent private receiver after purge; an ahead live voter still +rejects rollback. Quiesced node services then perform bounded image/page +maintenance over the complete two-group inventory, preserving reader, current +and checkpoint closures. + +All three voter disks close and reopen under new process generations, preserving +their group/cell identities. Recovery verifies every selected image, effect +summary, checkpoint and log suffix, including the sibling. Fresh quorum resolves +all 24 exact receipts without callback replay, refuses local compaction without +a newly verified archive frontier and commits two new commands over TLS. This is +retained-disk restart, not deletion of all voter disks or fresh-incarnation +disaster recovery. No cached status grants archive or serving authority. + +Real RustFS observations from this controlled test geometry: + +| Image mode | Raft file bytes before → after purge | Original files removed | Retained suffix entries | +| --- | ---: | ---: | ---: | +| materialized | 474445 → 256891 | 42 | 8 | +| extents | 474469 → 256817 | 42 | 8 | + +These measure actual file lengths at the purge boundary, not allocated +filesystem blocks, a global disk bound, throughput or production sizing. +Reader/transition holds and sustained workloads need their broader gates. + +Eighteen exact RustFS controller cases pass in both image forms with seven +signed S3 protocol probes. Twenty-two regression/check commands pass with 268 +distinct Rust cases across seven copied executables, including all four native +shared-group suites and 22 storage cases. Final-source locked all-target +Cellar/logic/LTX Clippy, format/layout, dependency/docs/preflight/plan checks, +37 Python tests and system-Python hashing pass. Source/artifact/selection and +credential-exclusion audit binds 2575 source inputs, copied +outputs/binaries/orchestration and frozen plan inputs under +`.cellar/cell-groups-evidence/CG-06-physical-log-reclaim/`. + +Source-bound failures remain retained. The initial helper visibility failure did +not run a case. A copied binary reproduces the subsequent oracle-reader failure: +`GenerationReader` refuses reads above 64 KiB. The helper now reads every byte +in bounded chunks without raising the reader limit, shrinking payloads or +dropping retained owners. A second copied-binary reproduction confirms that an +ahead live voter must reject the older retained cut. Positive data verification +now uses the existing private archive receiver, while an explicit negative +transfer preserves the live rollback guard and one-slot pool limits. A further +copied-binary timeout exposes the test retaining the archive pool's sole +verification slot past its completed checks; it now drops that result before +awaiting a new cut, while the durable journal retains the remote hold. The quota +and deadline remain unchanged. The first successful Rust run emitted +observations inside the libtest progress line; its wrapper incorrectly refused +selection. Raw output and the original record stay unchanged. Validated +observation normalization now preserves exact matching and the zero-failure +summary; five checks reject wrong, failed, empty and incomplete selections. +Final-source Clippy also requires the project diagnostic macro instead of std +println; the structured observations use that existing API without suppressing +the lint. The archive-store fixture is declared once per integration target, and +the archive-only scenario lives under the owning archive harness with its +existing target-specific exports. Source-bound duplicate-module, unused-helper +and reexport-visibility compile/lint failures remain retained; no production +lint setting or test assertion changed. Original assertions, task cases and +acceptance oracles remain intact. Earlier qualifications and the unexplained +`8dfe9ff3` slow-reader timing limitation remain retained. + +Reflection: two original CG-06 cases now pass: archive tails and physical +reclaim/restart. Complete archive-dependency acknowledgement, full S3-backend +outage authority and fenced fresh-incarnation serving remain required, together +with the full task gate. Investigation confirms that Seal/Install/Activate wire +entries exist but live apply still rejects them and manifests lack the durable +transition lifecycle. Before introducing a cache-based outage shortcut, +cancellation/authorization must share one durable decision, sources must seal, +and children must install and activate under an exact publication. Existing +local cached and remote fresh-discovery paths must first be qualified separately +against a stopped backend; the next independent deliverable is the original +complete-group dependency acknowledgement gate. The independent reclamation case +does not waive this dependency. Finish CG-08 effect/cache/result lifetimes +before CG-07 public SDK/fleet cutover, then source transitions, topology, +placement, hibernation, managed recovery, measured load and final legacy +removal. The full goal remains active and incomplete. + +## CG-06 retained-log admission and progress capacity + +Source: `ad73f03f278f5845b311fe929d184cc947382b2d`. CG-06 remains IN_PROGRESS. +Shared-group logs now account for logical retained entries and outstanding +reservations against explicit node and group byte/count ceilings. The ledger +reconstructs selected durable debt on reopen. Preexisting debt is preserved when +a restart uses smaller limits, and new admission waits until it fits. +Encoded-entry bytes include bounded accounting overhead; these are logical +limits, not physical Raft Engine file-byte bounds. + +New SQL callbacks and initializers reserve a worst-case command envelope before +execution. Known receipt reconciliation remains available at the ceiling. A +preparation carries an exact engine/group/entry lease through submission and +uncertain outcomes. Only sealed pre-append rejection or authoritative +reconciliation releases unused preparation capacity. Entered storage retains the +lease after waiter loss, then transfers its charge into selected durable entries +after sync/cache publication without a release gap or double debit. Consumed +leases cannot authorize another append. + +Local, inherited follower, election and configuration entries use the same +ledger. Empty election and automatic configuration entries have separately +reserved progress capacity. A joint transition promises its automatic leave +before it can enter RawNode, avoiding capacity acquisition after collected apply +work changes configuration. Capacity refusal on a deferred tick is nonfatal. +Role changes revoke concurrent execution authority before accounting. Actual +selected checkpoints and suffix replacement retire debt; archive status alone +does not. Existing archive-required compaction still needs independently +verified complete coverage. + +Three exact real-engine cases pass for entry ceilings/reopen, independent +node/group byte ceilings with slack counts, suffix replacement, exact lease +identity and held physical sync after all caller owners disappear. Both native +SQL image forms pass before-callback and initializer refusal, known receipt +recovery, sibling progress, exact local checkpoint selection/source release and +resumed writes. They fill ordinary admission before a real automatic joint +transition and require its automatic leave through the unchanged single progress +entry. The local checkpoint scenario does not claim a full S3 backend outage or +physical file purge. + +All seventeen exact real RustFS controller cases pass in both image forms, with +seven signed S3 protocol probes. Twenty-two regression/check commands pass with +267 distinct Rust cases and seven copied executables; all 22 storage cases and +four native shared-group suites run. Locked all-target Cellar/logic/LTX Clippy, +format/layout, native dependency/docs/preflight/plan checks, 37 Python tests and +system-Python hashing pass. Source/artifact/selection and credential-exclusion +audit passes across 2574 source inputs. Records, copied binaries/outputs, +orchestration and frozen plan inputs are under +`.cellar/cell-groups-evidence/CG-06-log-retention/`. + +Source-bound failures remain retained: initial accessor/type/layout compile +checks, the coherent-hardstate correction to a wrong-term lease negative, the +native fixture's impossible leader-only wait on followers, and the aggregate +snapshot test's default-stack overflow. Copied binaries reproduce these fixture +failures. The successor keeps the original identity assertions, quotas and +deadlines and waits for the exact selected cut, retired debt and released +checkpoint source on each voter. The aggregate snapshot case uses the existing +non-inlined fixture constructor to keep large async construction off its poll +frame; the exact case and complete suite pass on the original default stack. No +temporary DEBUG instrumentation remains. The earlier `8dfe9ff3` slow-reader +timeout remains unexplained despite unchanged passing reruns; this change is not +claimed to fix it. The RustFS controller matrix does not delete voter disks. +Original CG-06 cases, acceptance oracles and evidence array remain unchanged; +four original cases/full task gate remain incomplete. + +Reflection: next qualify a full backend S3 outage with fresh catalog/epoch +authority, without caching ownership past fencing. Complete and qualify physical +shared-engine file reclamation constrained by every +checkpoint/archive/reader/transfer and sibling hold; logical admission alone +does not establish a disk bound. Complete fenced fresh-group/cell-incarnation +recovery and replacement-quorum serving with the complete inventory of possible +old voters, including joins after the selected cut, using the accepted +fencing-authority API plus Docker qualification. Then finish CG-08 +effect/cache/result lifetimes before CG-07 public SDK/fleet cutover. Topology, +placement, hibernation, managed recovery, measured load and final legacy removal +follow. Voters remain host-native tasks and RustFS one local VM; independent +failure domains and the full hard cutover remain unqualified. + +## CG-06 owned archive activation and checkpoint drain + +Source: `ab0abf26dea4df436b5ced926f28a9ede4f360fd`. CG-06 remains IN_PROGRESS. A +deterministic regression at `79323600` reproduced a checkpoint overtaking +pending archive activation. Registration now owns an activation intent that +closes later checkpoint admission across retries. Rejected or unentered work +releases its intent. An already admitted checkpoint shares one exact +group/cut-bound lease between its owner and physical storage, so that work can +drain while new admission is closed. Storage rejects foreign-group, wrong-cut +and non-checkpoint lease use. The existing 4 KiB metadata charge bounds the +lease; quotas and serialized marker-1 records are unchanged. + +Once metadata persistence enters, physical work retains the activation even +after the waiter disappears. The irreversible local requirement becomes +effective only after sync and cache/root publication. Failed or uncertain writes +cannot publish the intent. Frontier reset happens once before I/O, so a delayed +duplicate completion cannot clear newer verified coverage. No mutex spans I/O or +an await. Cancellation before entry releases the temporary barrier; it never +clears an existing durable policy. + +Three exact storage cases pass. The original recovery case now checks refusal of +later admission and ownership across cloned physical leases. The new checkpoint +case uses actual persisted entries and a prepared checkpoint, tests invalid +lease bindings, then drains an earlier checkpoint into the real engine while +activation waits. The new held-sync case drops the waiter and client activation, +proves physical work still owns the closed barrier, releases the actual +filesystem gate and joins the storage queue before checking durable policy. +These establish the owning boundaries; sustained load fairness is still pending. + +All seventeen exact RustFS cases pass in both image forms with seven signed +protocol probes. Twenty-two regression/check commands pass with 262 distinct +Rust cases across seven copied executables, including all nineteen storage cases +and the four native shared-group suites. Locked all-target Cellar/logic/LTX +Clippy, format/layout, dependency/docs/preflight/plan checks, 37 Python tests +and system-Python hashing pass. Records under +`.cellar/cell-groups-evidence/CG-06-retention-admission/` bind 2571 source +hashes, selections, copied binaries/outputs, orchestration and frozen plan +inputs. Independent source/artifact/selection and credential-exclusion audit +passes. The original failing regression is retained with its source and +executable. Original CG-06 cases, oracles and quotas remain unchanged; four +original cases/full gate remain pending. The earlier `8dfe9ff3` native +slow-reader timeout remains retained and unexplained despite unchanged +successful reruns; this ownership change is not claimed to fix it. The RustFS +controller matrix does not delete voter disks. + +Reflection: implement explicit retained-log byte/entry admission across node and +group before new callbacks, preserving known receipt reconciliation. Include +inherited follower entries, control entries, suffix replacement, restart and +uncertain physical outcomes, with reserved progress capacity so collected Ready, +apply and checkpoint work can drain. Then qualify a full S3 backend outage, +including fresh catalog authority without weakening ownership checks. Physical +shared-engine file purge must honor every checkpoint, archive, reader, transfer +and sibling constraint. Fenced fresh-incarnation serving needs complete +possible-old-voter lineage and the accepted fencing-authority API plus Docker +qualification. CG-08 effect/cache/result lifetimes precede CG-07 public +SDK/fleet cutover. Topology, placement, hibernation, managed recovery, measured +load and legacy removal follow. Voters remain host-native tasks and RustFS one +local VM; independent failure domains and the full hard cutover remain +unqualified. + +## CG-06 archive-aware local checkpoint compaction + +Source: `8dfe9ff3a316d0579b656af4818dc57d839a6765`. CG-06 remains IN_PROGRESS. +Archive registration now syncs an irreversible local `archive_required` bit +before object-store I/O. The marker-1 group application record requires this +field, without a legacy decoder or migration. Ordinary writes and peer snapshot +installation preserve the local requirement. Reopen reconstructs the requirement +with no trusted in-memory frontier; the journaled current remote hold must be +independently verified again before local compaction. + +Each engine-local group shares one bounded frontier across its handles. Only +independently verified complete archive coverage advances it. The route/history +and local retained term must match; equal-index coverage must match the exact +root. Older conflicting evidence cannot replace a newer cut. A reconciled intent +behind the retained checkpoint grants no frontier and waits for a newer complete +cut. Archive status remains advisory. The frontier grants neither S3 output nor +serving authority, and retains no replay workspace per idle group. The durable +journal continues to own the current remote hold. + +The automatic scheduler checks the frontier before capture, allowing quorum +apply and a sibling group's checkpoint to continue while archival is held. The +actual Raft owner checks its exact captured cut, and the storage worker repeats +admission before any local checkpoint compaction batch. A prepared source +remains a data capability and cannot bypass either boundary. Already admitted +checkpoints retain owner and physical-storage guards until completion; archive +activation retries their drain before syncing policy. This avoids retracting a +checkpoint while retaining its sole source slot. No mutex spans an await or +physical I/O. Registration also checks the actual local birth before policy +changes. + +The new native and real RustFS scenario holds actual registry publication in +both image forms. Quorum adds another cell while the configured group's +checkpoint and log prefix stay intact. A sibling publishes through the original +one-source pool and one controller checkpoint slot. Direct owner admission and +raw persistence of the prepared local checkpoint both refuse. After actual +complete coverage, the same preparation publishes; another archive outage +preserves later committed log entries without quarantine. Independent recovery +checks every selected cell. The storage regression checks activation refusal +while an earlier admission is held, durable activation after drain, +missing-field codec refusal, inability of ordinary writes to clear policy and +recovered admission remaining closed without fresh coverage. + +All seventeen exact RustFS cases pass in both image forms, with seven signed S3 +probes. All twenty-two regression/check commands pass with 260 distinct Rust +cases across seven copied executables. These include the four native +shared-group suites, all seventeen storage tests, nine snapshot unit cases, +cache/receipt/budget/journal/physical-join primitives, eighteen wire cases and +fixed metric cardinality. Locked all-target Cellar/logic/LTX Clippy, +format/layout, dependency/docs/preflight/plan checks, 37 Python tests and +system-Python hashing pass. Records under +`.cellar/cell-groups-evidence/CG-06-archive-compaction/` bind 2571 source +hashes, exact selections, frozen plan inputs and executable/output hashes. +Independent source/artifact/selection and credential-exclusion audit passes. The +initial capture/import compile refusal, recovery-test lock-scope lint and native +zero-operation refusal remain source-bound. A copied executable reproduced the +native failure; typed status identified the invalid private metadata correlation +ID. The correction uses a valid nonzero ID while preserving public zero-ID +rejection and the independent physical sync sequence. The fixture uses runtime +capture and orders sibling work before retaining its original source slot. +Original cases, oracles and quotas remain unchanged. The first broad regression +attempt timed out in the existing native slow-reader case; unchanged +copied-executable isolated and full ordered-controller reruns passed. The +passing complete rerun retains that failure and its unresolved timing limitation +without changing code, timeouts, quotas or assertions. Four original CG-06 +cases/full gate remain pending. + +Reflection: add explicit node-wide unarchived-byte/entry admission, including +inherited peer and control work, pre-callback reservations and capacity for +checkpoint/physical completion progress. Qualify a full S3 backend outage, +including the catalog-binding dependency without weakening ownership checks. +This compaction gate alone does not establish that disk bound. Follow with +physical shared-engine Raft file purge under every +checkpoint/archive/reader/transfer/sibling constraint, and fenced +fresh-incarnation serving with complete old-voter lineage inventory. +Peer-install integration of deployment archive policy, controller process-kill +and multi-head/physical-read shutdown cuts still need qualification. Archive +activation fairness under sustained checkpoint traffic also needs an adversarial +gate. CG-08 lifetimes precede CG-07 SDK/fleet cutover. Topology, placement, +hibernation, managed recovery, measured load and legacy removal remain. Voters +are host-native and RustFS is one local VM; independent failure domains and the +production hard cutover remain unqualified. + +## CG-06 remote S3 output ownership through physical delivery + +Runtime source: `25938fd86d13a2755066660797b07aff955ec1ad`. Validation-only +metric inventory correction: `debdc30c8327aad35757c6b849e029c1bbdab691`. CG-06 +remains IN_PROGRESS. Commands, enrollment and receipt lookups now transmit their +effective S3 requirement and return authenticated achieved durability. Unknown +durability codes and replies below the original requirement refuse before result +use. Existing exporter-bound MACs bind the body digest to the exact request, +purpose, peer process and complete group/cell history. Command/receipt tokens +retain their signed observation checks; enrollment returns the original creation +record, including after later mutations. Marker-1 hard cutover adds no legacy +reader or runtime mode flag. + +The receiver still needs an explicit provider binding to its exact runtime and +active archive group. It uses the independently recovered complete-cut proofs, +namespace/stored minima, current catalog/leader checks and original deadlines +already required locally. An unbound receiver refuses before mutation when S3 is +required. A stored-minimum duplicate resolves SQL before refusing an unbound +owner. Possible transmission keeps mutation failures ambiguous. No transport +retry replays application callbacks, and receipt absence remains ambiguous. + +A bounded owned reply now retains the native S3 output with its encoded bytes. +That output owns physical coverage, S3 admission and journal/remote-hold +lifetime. The existing transport reservation accounts for encoded bytes. Quorum +outputs release resolved SQL as before. During writing, an aggregate owns both +the TLS stream and reply: stream destruction precedes proof release, even if the +entire serving future is destroyed. Successful flush precedes consumption. +Cancellation, revocation and timeout cannot leave an acknowledged proof-free +response in the writer. No per-cell task, timer or metadata-only proof cache is +introduced. + +All sixteen exact RustFS cases pass in both image forms, plus seven signed S3 +protocol probes. Three new cases cover all remote output kinds and namespace/ +stored minima, original creation retries, unbound receiver refusal, archive-wait +deadlines/cancellation and actual slow command/receipt TLS delivery. Independent +wire clients use a bounded sequence of exact retries to fill small real kernel +buffers without reading replies, holding a 28 KiB body within the unchanged 64 +KiB S3 envelope. TLS header reads prefetch and a single reply fits the buffers. +The new fixed physical-write observation identifies the held phase without +client reads. Proof/coverage and byte admission remain retained while SQL is +free and a same-cell Quorum successor commits. Another S3 operation overloads +within the original one-slot limit. Drain succeeds; disconnect, grant revocation +and timeout join physical release. Truncated bodies cannot become successful +replies; exact retries return the original result without callback replay. The +earlier thirteen archive/local-output scenarios also pass. + +Twenty regression/check commands pass with 232 distinct Rust cases across seven +copied executables, including all eighteen wire codec/authentication cases, +receipt/ budget primitives, fixed performance outcomes/cardinality, the four +native shared-group suites and LTX clones. Runtime all-target Clippy, +format/layout, dependency/docs/preflight/plan checks, 37 Python tests and +system-Python hashing pass. Runtime Clippy and 231 passing regressions retain +their original source and hashes. The validation successor fixes the two +archive-readiness gauge names/count already added on main, plus the matching API +comment. Its exact metrics fixture, focused Clippy, format and repository checks +pass. An independently checked two-file delta excludes runtime behavior or quota +changes. Records under `.cellar/cell-groups-evidence/CG-06-s3-remote/` bind 2570 +source hashes, exact selections, logs/executables, frozen plan inputs and +predecessors. Independent source/artifact/selection and credential-exclusion +audit passes. The initial import-path failures retain their exact sources and +outputs. The failed raw probe and copied-executable reproductions also remain +retained: its operation length needed u16, and reading a TLS header prefetched +the full body, while a single undrained reply still fit the socket buffers. The +final probe fills a bounded retry window and observes the physical write phase. +The initial native metrics inventory failure and copied-executable reproduction +remain retained. PR #159 added two fixed readiness gauges while its inventory +expected 111. The correction requires both names and exactly 113 resource +samples, preserving duplicate/type/size checks and one-versus-32-group identity +equality. An explicit released-command admission barrier identifies the +successor phase. No original case or existing node quota was weakened. Four +original CG-06 cases and its full gate remain open. + +Reflection: next implement explicit unarchived-byte/disk admission and qualify +bounded operation during a full S3 outage; the archive-wrapper faults above are +not that gate. Fresh destination catalog binding can still depend on S3 until +replicated transfer seals are integrated; resolve that dependency without +weakening ownership admission. Add safe physical shared-engine Raft file purge +with complete checkpoint/archive/reader/transfer constraints, then fenced +fresh-incarnation serving with every possible old-voter lineage, including joins +after the cut. Fencing-authority API plus Docker qualification remains accepted. +Multi-head hold retention, process-kill restart and physical-read shutdown cuts +still need broader qualification. Incremental committed tails and smaller sealed +attestations backed by actual remote holds remain performance opportunities. +CG-08 lifetimes precede CG-07 SDK/fleet cutover; topology, placement, +hibernation, managed recovery, load and final legacy removal remain. Voters +share one host and RustFS one local VM; independent hosts, measured load and +fenced replacement serving remain unqualified. + +## CG-06 local receipt S3 observations and bounded result reads + +Source: `cde0eea2f516486578fdee7a4cc0b8be9ba8f72a`. CG-06 remains IN_PROGRESS. Local command, +enrollment and receipt outputs now share the explicitly bound controller archive +owner. `CellReceiptInvocation::select` returns an opaque quorum selection; its +consuming `response` settles required durability. Existing `execute` runs both. +S3 admission takes the selected bytes before SQL read admission releases. The +selection queues no archive work until response; dropping it refunds admission. +Request, namespace and stored receipt minima are never lowered. + +The complete independently recovered group must cover the quorum cell history/ +version and equal-version decision. Known receipts remain exact across later +images. An initially absent receipt is read again from the actual archive, which +may now contain the command. Return that image's actual observation and token; +never attest an old absence against a changed image. None remains ambiguous and +never releases a producer. Empty results remain distinct. Lookup runs no command, +initializer or receipt maintenance; expiry, conflict and sequence gaps stay typed. + +Result length is inspected before payload fetch. The same S3 permit grows for +actual bytes without another item slot or increased node ceilings. A legitimate +40 KiB result exceeds the unchanged 64 KiB fixture S3 envelope with duplicate-read +headroom: S3 returns typed Overloaded before allocation, then Quorum still reads +the intact result. Original SQL/S3 node quotas remain unchanged. The new expiry +scenario deliberately fills a two-receipt ceiling so normal replicated maintenance +prunes an expired prefix while retaining the latest receipt and permanent floor. + +S3 outputs retain complete physical coverage, shared pool admission and the journal +lock. Catalog proof rechecks enter the captured execution domain and remain bounded +by the original deadline for all three output kinds. Receipt output additionally +checks a fresh correlated quorum read's current leader/route and version floor. +That read fence tolerates later cell mutations without retaining a writer's base. +Remote S3 delivery remains guarded on both ends, including provider-configured +receivers that discover a stored S3 minimum. A rejected committed command duplicate +clears SQL ownership first and remains ambiguous to its remote caller. + +All thirteen exact RustFS cases pass in both image formats, plus seven signed S3 +probes. The four new cases cover later receipts replacing initial absence, known +results at newer cuts, signed observations, empty/absent results, namespace/stored +minima, unsubmitted-phase drops, foreign providers, typed conflict/expiry, payload +admission, read deadlines/cancellation and actual TLS receiver refusal without +callback replay or quarantine. The earlier archive/command/enrollment cases pass. + +All sixteen check/regression commands pass: 205 distinct Rust cases across six +copied executables, locked all-target Cellar/logic/LTX Clippy, format/layout, +dependency/docs/preflight/plan checks, 37 Python tests and system-Python hashing. +Records under `.cellar/cell-groups-evidence/CG-06-s3-receipts/` bind +2567 source hashes, exact selections, frozen plan inputs, executable/output +hashes and predecessors. Independent source/hash/selection and credential-exclusion +audit passes. The initial direct-caller domain panic retains its exact source, +executable, output and minimized backtrace. Production scoping fixes it; no global +host handle or assertion workaround was added. The intermediate four-case success +and final deadline-bound source remain distinguishable. The fixture macro-lint +refusal is retained; cancellation now polls the response and observes an actual +archive attempt before dropping it. No original task oracle +or existing quota was weakened. Four original CG-06 cases/full gate remain pending. + +Reflection: implement remote S3 output proofs through physical delivery, preserving +authenticated purpose, achieved durability, route/version and original deadlines. +Then explicit unarchived-byte/disk admission, physical shared-engine Raft purge +and fenced fresh-incarnation serving with every possible old-voter lineage. +Qualify multi-head leases, process-kill restart and physical-read shutdown cuts. +Incremental committed tails and smaller sealed attestations backed by real remote +hold leases remain performance opportunities. CG-08 lifetimes/maintenance precede +CG-07 SDK/fleet cutover; topology, placement, hibernation, managed recovery, load +and final legacy removal remain. Voters share one host and RustFS one local VM; +independent hosts, measured load and fenced replacement serving remain unqualified. + +## CG-06 local enrollment S3 response settlement + +Source: `c2801b6586ffc8b3b765312e0042986e881d773a`. CG-06 remains IN_PROGRESS. The explicit +`CellGroupDispatcher::with_s3` binding now also settles local enrollment responses. +The existing fair archive lane owns both output kinds under independent bounded +S3 admission. Enrollment reserves its bounded original creation record before +releasing resolved SQL output retention; no original SQL/command quota changes. + +Required durability is the maximum of request and immutable namespace minimum, +captured in the invocation fingerprint. Exact runtime ownership and active archive +registration precede initialization. After commitment, producer/preparation +ownership clears before the archive wait. A complete independently recovered group +must contain the exact history, creation ID, schema and initial image commitment. +An older cut waits; conflicting creation refuses. Subsequent mutations preserve +that original provenance without retaining superseded initial bytes. Proof and +exact retry never rerun initialization. + +The response rechecks catalog birth, actual leader authority and its exact current +creation record under the original deadline. `CellSqlEnrollment::durability` +reports S3 only while retaining its private complete-coverage proof, pool admission +and journal lock. Active outputs backpressure the shared proof pools until release; +idle groups retain no replay workspace. Remote enrollment explicitly refuses S3 +before transmission until its wire contract is integrated. + +Nine exact RustFS cases pass in both image formats. The two new cases also exercise +request and namespace minimums, a foreign runtime refusing before initialization, +publication-held output, a later same-cell Quorum mutation during the S3 wait, +retained proof slots, original-record duplicates and conflicting creation IDs. +Deadline and dropped-reply branches keep the one-command/one-SQL-slot quotas, +refund admission, allow a Quorum successor during archive outage and retry the +same creation with exactly one initializer invocation. Seven signed S3 probes +and all earlier local command/archive/refusal cases pass. + +All fourteen regression/check commands pass: 195 distinct Rust cases across six +copied executables, locked all-target Clippy for Cellar/logic/LTX, format/layout, +dependency/docs/preflight/plan checks, 37 Python tests and system-Python hashing. +The retained records under `.cellar/cell-groups-evidence/CG-06-s3-enrollment/` +bind 2566 source hashes, frozen plan inputs, exact selections, executable +and output hashes, orchestration sources and prior qualification. Independent +source/hash/selection and credential-exclusion audit passes. No original task +case, oracle or pre-existing quota was weakened. The initial fixture-allocation +Clippy refusal and its exact source/log remain retained; the correction changes +only two literals. Four original CG-06 cases and its full gate remain pending; +this is partial task evidence. + +Reflection: finish local receipt lookup, preserving typed expiry/conflict and the +ambiguity of an absent row, then remote command/enrollment/receipt output proofs +with retention through physical delivery. Add explicit unarchived-byte/disk +admission, physical shared-engine Raft file purge and fenced new-incarnation +serving with complete old-voter lineage inventory. Qualify multi-head proof +retention, controller process-kill restart and physical-read shutdown cuts. +A performance opportunity is incremental committed-tail scheduling and smaller +sealed output attestations backed by real hold leases, qualified under load. +CG-08 lifetimes/maintenance precede CG-07 public SDK/fleet cutover; topology, +placement, hibernation, managed recovery, load and final legacy removal remain. +Voters share one host and RustFS one local VM; independent hosts, measured load, +fenced replacement-quorum serving and the full hard cutover remain unqualified. + +## CG-06 local command S3 response settlement + +Source: `3d564f3db2c2eefc87119e7b3395f1bca17b99fa`. CG-06 remains IN_PROGRESS. An explicit +`CellGroupDispatcher::with_s3` binding now settles local commands through the +existing node archive lane. It checks the exact runtime owner and active group +registration. Required durability is the maximum of request, namespace and stored +receipt minimums. Unbound dispatchers retain their refusal; receipt lookup, +enrollment and remote S3 response gates remain unfinished. + +SQL resolution clears producer ownership before archival. A separate bounded S3 +stage acquires admission before taking over the committed result bytes; resolved +preparations no longer retain duplicate SQL admission. Existing SQL/command quotas +are unchanged. The first same-cell successor test reproduced Overloaded because +the waiting receipt still held the SQL pool's only operation permit. Its source +snapshot, executable and output are retained. The fixed test passes with that same +one-slot SQL quota, including an explicit zero retained-SQL-operation assertion. + +The fair archive lane takes at most 32 receipt proofs per group turn. A published +complete cut is independently replayed and its exact immutable receipt is read +without callbacks. History/version/position, equal-version decision and every +stored receipt field/result must match. A later image can prove an older receipt; +an advisory index or sibling progress cannot. The response rechecks catalog birth +and live leader authority before its token. It retains physical coverage/admission, +S3 byte/item retention and the exclusive journal lock. Weak owner uses block +predecessor-hold retirement until consumers release the coverage. Idle groups keep +no replay workspace; held outputs intentionally backpressure those shared pools. + +Seven exact RustFS cases pass in both image modes, including positive S3 responses, +a later same-cell quorum mutation during archive outage, publication-held output, +Quorum retries honoring a stored S3 minimum, proof-slot retention, callback-once +retries, one-command-slot expiry/cancellation isolation and missing-B dependency +refusal while A's receipt and quorum remain healthy. Foreign runtimes refuse +before SQL. A namespace S3 minimum overrides a Quorum request, persists that +minimum and keeps its response pending until publication. The three earlier setup/publication/refusal scenarios still pass. +The actual S3 client performs catalog/archive I/O and conditional writes. + +All fourteen regression/check commands pass: 193 distinct Rust cases across six +executables, locked all-target Clippy for Cellar/logic/LTX, format/layout, +dependency/docs/preflight/plan checks, 37 Python tests and system-Python hashing. +The records under `.cellar/cell-groups-evidence/CG-06-s3-commands/` bind +2564 source hashes, frozen inputs, copied executables, exact selections, +output hashes and predecessors. Independent source/hash/selection/credential +exclusion audit passes. The source-bound Clippy large-enum failure is retained; +boxing the private proof bundle fixes it without public variant or contract changes. +No original required case/oracle or pre-existing quota was weakened. Four original +CG-06 cases and its full task gate remain pending; this is partial evidence. + +Reflection: integrate S3 enrollment, receipt lookup and remote output contracts +using this retained owner, then explicit unarchived-byte/disk admission. Qualify +multi-head retention, controller process-kill restart and physical-read shutdown +cuts, physical shared-engine Raft log purge and fenced recovery into fresh group/ +cell incarnations. Keep the complete old-voter lineage inventory, including later +joins. CG-08 cache/effect/result lifetime work precedes CG-07 public SDK/fleet +cutover. Native voters share one host and RustFS one local VM; no independent +failure-domain, load, complete disaster recovery or full hard cutover is claimed. + +## CG-06 local RustFS archive and response-boundary qualification + +Source: `d739bf795f7e58c1c27b2c6c496f9f57355e3c02`. Production runtime bytes are unchanged from `7914177d`. +CG-06 remains IN_PROGRESS. The local qualification backend is real RustFS, as +requested. [The runner](../../../../tools/multi-raft/cell_groups.py) verifies the +owned backend's image, running state, endpoint and actual cgroup limits, then +creates a fresh bucket and isolated scenario/image prefixes. Explicit S3 clients +use ETag conditional writes without a memory or ambient-credential fallback. + +All three exact ignored cases pass in both materialized and extent modes: +controller publication while quorum advances, lost publication response/readback +with exact-intent reconciliation and joined close, and refusal of unproved S3 +command/receipt/enrollment responses, including a namespace S3 minimum. Catalog +setup and archive recovery use RustFS. Lost-response injection follows a real +successful conditional write. The refusal case does not qualify successful S3 +acknowledgements; shared dispatch still refuses those until exact operation proofs +are connected. Existing assertions and required CG-06 cases remain unchanged. + +Seven signed S3 protocol operations and all ten regression commands pass, including +183 native execution/archive/controller/snapshot cases, all-target Cellar Clippy, +format/layout, dependency/docs/preflight/plan checks and 37 Python tests. System +Python 3.9 evidence hashing passes. The record under +`.cellar/cell-groups-evidence/CG-06-rustfs-controller/` binds 2561 +source hashes, copied executables, exact selections and output hashes, plus prior +qualification and retained setup/runner/Clippy failures. Independent source/hash, +selection and credential-exclusion audit passes. No credentials enter evidence. + +The initial health check reached an unrelated VM's port forward because Docker +allocated its port inside the guest. The helper now selects a free host port +explicitly; signed requests then verify the credential-bound service. The access +key uses SDK CI's 20-character profile. Source-bound corrections retain the system +Python hashing failure and the target-specific shared-fixture dead-code failure. + +Reflection: extend this executable RustFS path with positive S3 command, receipt +and enrollment cases while adding exact cell-version/result/current-route proof +ownership. Preserve captured identity, original deadlines and callback-once +behavior. Then bound unarchived-byte/disk admission, physically reclaim Raft files +and qualify fenced fresh-incarnation serving. These host-native voters and a +single local RustFS VM do not qualify container node limits, load, process-kill +restart, independent hosts or disaster recovery. Four original CG-06 cases and +its full gate remain pending; CG-08 lifetimes precede CG-07 public cutover. + +## CG-06 controller-owned archive publication + +Source: `7914177d6f17e3f6d647a39dc496122157283591`. CG-06 remains IN_PROGRESS. +The existing node controller now owns one fair archive lane, using a deduplicated +dirty-group queue and its existing timer. Clean groups need no roster scan. Apply +and checkpoint-install progress coalesce while object-store work runs outside the +SQL lane. Registration opens only the namespace-bound expected registry, with a +private verifier store and node-shared source/I/O/upload/replay admission. + +An exclusively locked journal binds the node engine creation and archive identity. +It fsyncs canonical reservation/publication intents before remote mutation, using +an atomic file replacement plus directory sync/readback. Missing established +identity/state refuses rather than recreating metadata. Attempts reload selected +state after uncertain local writes. Restart/retry reconciles an exact publication; +if absent, it abort-fences the ticket and reconciles again before reallocation. +A new head and predecessor retirement are journaled before the old hold is released. +Published markers remain advisory metadata; physical root/replay admission does +not escape the job or remain reserved for every registered group. + +The sealed archive execution capability accepts the controller's captured domain +token without ownership of domain shutdown. Scoped blocking-work accounting spans +abandoned async waiters; close joins logical jobs and their physical closures, +then closes private stores and releases journal locks before runtime/SQL teardown. +Typed checkpoint `NotReady` and admission overload retry after progress; corrupt +metadata and journal loss stop archival. They do not become S3 acknowledgements. + +Both image modes qualify held publication while the real quorum continues, lost +publication response plus failed readback, persisted exact-intent reconciliation, +cancelled registration response, joined close, a single current remote hold and +independent recovery of every cell afterward. The first uncertain-publication +scenario found a lost idle-tail wake: recovery of M could clear dirty progress for +already-applied N. Completion now observes both runtime applied state and consumed +apply events, and the unchanged regression proves N archives without another request. +Unit cases qualify actual journal reopen/missing-state refusal and physical drain +after dropping a blocking waiter. This does not qualify a process-kill controller +restart or a bounded S3 outage ledger. + +All 40 gate commands pass: nineteen exact archive cases, two exact controller cases, +two exact lifecycle/journal unit cases, 157 execution cases, five snapshot cases and +four LTX generation cases (189 distinct Rust cases across six copied executables). +Locked all-target build/Clippy for Cellar/logic/LTX, formatting/layout, dependencies, +docs, preflight, plan validation and all 37 Python tests pass. The partial record at +`.cellar/cell-groups-evidence/CG-06-controller-archive/qualification/checkpoint.json` +binds 344 source hashes, frozen plan inputs, hashed logs/binaries and the preceding +catalog-cut checkpoint. Independent source/artifact/exact-selection verification +passes. Source snapshots retain the initial error-wrapper compile failure, native +idle-tail regression, prospective-root refusal regression and Clippy reply-type +and duplicate-module failures. Prospective/stale public roots still refuse as +invalid; only trusted owner-current capture can return typed retryable NotReady. No acceptance case, assertion, +oracle or existing quota was weakened. This is not full CG-06 task evidence. + +Reflection and next priorities: connect pending S3 outputs to retained independent +coverage proofs, verifying exact command/enrollment/receipt identity, cell version, +result and current routing authority. Preserve captured inputs and original deadlines; +never rerun callbacks. Add explicit unarchived-byte/disk admission before claiming +bounded Quorum progress through S3 outages. Then qualify local shared-engine Raft +file purge and fenced recovery under fresh group/cell incarnations. The source fence +inventory must cover every possible old voter lineage, including later joins. +Automatic remote/private-cache maintenance and controller process-kill restart still +need qualification. The controller currently uploads complete coalesced cuts; +incremental tail scheduling and performance require the measured-load gates. + +CG-08 effects/cache lifetime work precedes CG-07 public SDK/fleet cutover, followed +by topology, placement, hibernation, managed recovery, RustFS/load/fault qualification +and final legacy removal. Shared dispatch still refuses S3 durability. The other +four original CG-06 cases and its full task gate remain pending. These scenarios +use the actual in-memory object-store adapter; no real RustFS, Docker fence, +fresh-quorum serving, load or full hard cutover is claimed. Transfer ignored evidence +and its predecessors for audit. + +## CG-06 namespace archive identity and stable cut capture + +Source: `df0ba1f67c0253a57127a215540f08bb13135279`. CG-06 remains IN_PROGRESS. +The namespace catalog now +owns the expected archive registry identity and its object-store/data location. +A one-use initializer binds the namespace root and exact group birth route before +registry creation; persist its canonical bytes before mutation. A fresh catalog +owner can reopen only that immutable identity. Competing initializers, missing +bindings and missing counters refuse rather than replacing or repairing state. +Interrupted initialization remains fail-closed and requires authority recovery. + +`CellGroupRuntime::capture_checkpoint` performs bounded metadata capture on the +existing worker, using its owner and node checkpoint pool. Physical I/O remains +outside that worker. A held source consumes its quota, permits subsequent quorum +writes, and refunds on final release. New capture refuses after joined shutdown. +`upload_cut` retains the captured root/configuration/permanent membership before +upload starts and writes a zero-entry tail for that same cut. It never recaptures +a moving source after remote I/O. A held B upload allows A to commit while the old +cut still uploads, independently verifies and publishes in both image forms. +The same single-source checkpoint pool can admit the later independent receiver. + +The all-three-voter erasure scenario now also deletes every local archive intent +journal and drops the uploader/registry owners. It reconstructs the expected +registry identity through a fresh catalog owner from remote namespace metadata +before physically recovering the exact cut. No local intent journal or voter +metadata supplies that identity. This remains data-only recovery with externally +retained untrusted checkpoint/tail locators, without a fence or replacement voters. + +All 32 gate commands pass: nineteen exact archive cases, 157 execution cases, five +snapshot cases and four LTX generation regressions (185 distinct Rust cases across +four executables), all-target build/Clippy for Cellar/logic/LTX, formatting/layout, +dependency isolation, docs, preflight, plan validation and 37 Python tests. +The partial record at +`.cellar/cell-groups-evidence/CG-06-catalog-cut/qualification/checkpoint.json` +binds 336 source hashes, four copied executables, frozen plan inputs, hashed +outputs and the preceding protected-publication checkpoint. Independent source, +artifact and exact-selection verification passes. No original required case, +oracle or pre-existing quota was weakened. The first final Python run caught the +unrefreshed catalog source anchor: only reexports changed, moving the unchanged +NamespaceConfig/BirthProposal excerpts by one line. That failed source-bound output +and its old baseline are retained under `plan-baseline-failure/`. After reviewing +and pinning all anchors to the qualified source, docs, preflight, plan validation +and all 37 Python tests pass. Rust source stayed fixed; its passing checks were +retained. This record is not full task evidence. + +Reflection and next priorities: compose controller-owned publication with durable +journaling and exact unknown-outcome reconciliation, joined cancellation/shutdown, +verified-result lifetime ownership and fair bounded retry. Then implement actual +S3 command/enrollment/receipt output settlement against exact cell versions/results +and current routing authority, preserving captured inputs without callback replay. +Add explicit unarchived-byte/disk admission so bounded Quorum progress does not +require S3. Follow with physical local shared-engine Raft file purge and fenced +unmanaged recovery under fresh group/cell incarnations. The permanent fence inventory +must cover every possible old voting lineage, including joins after the archived cut. + +CG-08 effect/cache lifetime composition precedes CG-07 public SDK/fleet cutover; +then topology, placement, hibernation, managed recovery, RustFS/load/fault qualification +and final legacy removal remain. Shared dispatch still refuses S3 durability. The +other four original CG-06 acceptance cases and full task gate remain pending. +These native scenarios use the actual in-memory object-store adapter. No real +RustFS, Docker fence, fresh-quorum serving, measured load or full DR is claimed. +Preserve and transfer ignored evidence and its predecessors for audit. + +## CG-06 protected publication and remote object collection + +Source: `873484774c1ad5fb261257e4890291f8ad956f6e`. CG-06 remains IN_PROGRESS. +A complete immutable checkpoint/tail can now be published with a persistent remote hold, using a sealed +committed-source witness and independent physical replay. This is historical +complete-cut coverage; command S3 settlement and local Raft file purge remain +separate unfinished boundaries. + +Immutable keys now contain a registry ID and never-reused allocation generation. +Marker 1 makes a hard cutover, with no legacy reader or deletion of old namespaces. +Pending generations retain their exact selected parent, and uploads reuse only +objects in that protected closure. Fresh creation is a one-use capability, with +its intent durably recorded before mutation. Opening requires the owner's exact +persisted creation identity. Missing/corrupt established state and uncertain +identity creation without complete state refuse; neither resets counters. + +Tail capture retains a typed committed-source witness before upload. New source +writes may advance while B is delayed, and the older sealed cut can still publish +after every dependency verifies. Publication atomically consumes the reservation, +selects a non-regressing descriptor and creates an explicit remote hold. An exact +persisted intent reconciles a lost response/readback cancellation without another +allocation or publication, including after the source engines close. Serialized +locators/intents cannot create source authority. Retired/aborted intents never +resurrect; reader holds and pending parents remain independent of newer heads. + +Collection fully expands and hashes every protected closure before any DELETE. +It excludes marked keys, complete pending generations and newer generations. +Actual remote deletions preserve current recovered data; an old-generation DELETE +held while identical bytes are reuploaded and published cannot erase the new copy. +Corrupt B blocks both new coverage and complete collection. A physically valid +old checkpoint outside the new upload's protected parent refuses publication. +Compact marks/candidates and source witnesses have explicit node admission. +A two-owner test retains a witness through clones, refuses another capture while +quorum writes continue, and refunds only after the final witness drops. + +All 30 commands pass: seventeen exact archive cases, 156 execution cases, five +snapshot cases and four LTX generation regressions (182 distinct Rust cases across +four executables), all-target build/Clippy for Cellar/logic/LTX, formatting/layout, +dependency isolation, docs, preflight, plan validation and 37 Python tests. +The partial record at +`.cellar/cell-groups-evidence/CG-06-protected-head/qualification/checkpoint.json` +binds 334 source hashes, four copied executables, frozen inputs, hashed outputs +and the preceding independent-verifier record. Independent source/artifact/exact- +selection verification passes. The first qualification build at `a5d506a6` caught one +missed unboxing branch after the Clippy enum-layout fix; its source-bound failed +output is retained under `build-failure/`. The correction and full gate use the +source above. No required case/oracle or pre-existing quota was weakened. + +Reflection and next priorities: wire protected publication into the node controller +and persist the expected registry identity in authoritative namespace/owner state. +Then settle actual S3 command/enrollment/receipt outputs against exact cell versions, +results and current routing authority; preserve captured identities/inputs and +never rerun callbacks. Replace the Quorum-only fingerprint/guards only when those +proofs exist. Add explicit unarchived-byte/disk admission while Quorum can continue +without S3. Follow with local shared-engine file purge and fenced unmanaged seeding +under fresh group/cell incarnations. Persist source birth/provenance and fence all +possible old voting lineages, including joins after the selected archived cut. + +CG-08 still owns controller/cache lifetime composition and private-cache maintenance; +retain verified result admission alongside derived raw manifest roots. Then CG-07 +cuts public SDK/fleet routing over, followed by topology, placement, hibernation, +managed recovery, RustFS/load/domain qualification and final legacy removal. +The native fixture uses the actual in-memory object-store adapter; no real RustFS, +Docker fence, fresh-incarnation serving or performance claim follows. The other +four original CG-06 cases and full task gate remain pending. Shared dispatch still +refuses S3 durability. Transfer the ignored evidence and its predecessors for audit. + +## CG-06 archive verification without voter metadata + +Source: `14955f2dc2b01a6f6ab6b75b0122c812e5de6765`. CG-06 remains IN_PROGRESS. +The archive verifier owns dedicated physical stores and needs no GroupLog, bootstrap/member claim, +replacement genesis or surviving voter metadata. Its typed checkpoint is distinct +from the engine-bound snapshot output. Shared physical decoding preserves the +live receiver's selected-base, membership, owner and installation checks; private +archive roots have a freshly minted owner token that live engines refuse. + +Both image forms recover the exact checkpoint/tail cut, all four cells including +empty enrollment, and permanent membership after closing all three source engines, +dropping their physical owners and deleting every voter log/page/image directory +and source extent directory. Destination stores are opened only after erasure. +A separate native scenario holds a remote dependency while another execution +domain advances the healthy source quorum, then verifies newer and older archive +cuts independently. Exact-cut engine publication and source capture refuse the +private archive root; engine-selected state stays unchanged. Closed stores refuse +both outstanding and new receives. New private materialized replay results release +retention after proof pins close, and complete remote recovery succeeds after +actual private-image eviction. Existing host retention is preserved. + +All 24 commands pass: eleven exact archive cases, 156 execution cases, five snapshot +cases and four LTX generation regressions (176 distinct Rust cases across four +executables), all-target build/Clippy for Cellar/logic/LTX, formatting/layout, +dependency isolation, docs, CG-06 preflight, plan validation and 37 Python tests. +The partial record at +`.cellar/cell-groups-evidence/CG-06-isolated-verifier/qualification/checkpoint.json` +binds 326 source hashes, copied executables, frozen plan inputs and command +outputs. Independent source/artifact/exact-selection verification passes. Transfer +this ignored evidence with its committed-tail predecessor to reproduce the audit. +No required task case, oracle, quota or readiness assertion was weakened. + +Reflection and next priorities: data integrity alone cannot authorize S3 settlement +or truncation. Implement conditionally published, GC-protected coverage of the +exact complete committed cut, then bounded outage admission/S3 output settlement, +archive-aware shared-engine file purge and fenced unmanaged recovery into fresh +group/cell incarnations. Preserve source birth/provenance and an authoritative +inventory of all possible old voting lineages, including voters joined after the +selected archive cut; fencing only selected-cut members is insufficient. Follow +with CG-08 effects/cache lifetimes and CG-07 public SDK/fleet cutover, then topology, +placement, hibernation, managed recovery and final legacy removal. Controller +composition must retain the verified archive result alongside derived manifest +roots: a raw `PreparedGroupManifest` clone does not carry its parent archive quota. +Qualify that ownership propagation in CG-08 before managed integration. + +The erasure drill uses real local Raft Engine/SQLite/LTX and the actual in-memory +object-store adapter. It does not qualify RustFS, Docker fencing, a new serving +incarnation, process/machine failure isolation, measured load or the production +hard cutover. The other four original CG-06 acceptance cases and full task gate +remain pending. Shared dispatch still refuses S3 durability. The preceding +committed-tail source/evidence is preserved below. + +## CG-06 committed-tail replay and extension after compaction + +Partial source checkpoint: `7a29c2a9bff5c2daddf37f53c50bc5d575e0b069`. +CG-06 remains IN_PROGRESS. A complete immutable tail now binds its original group +checkpoint, every contiguous entry and the final root/configuration/permanent +membership. Capture copies only the exact committed/applied suffix under the +engine read lock; encoding and object-store work happen afterward. Independent +verification recovers the checkpoint and replays actual cell-addressed LTX, +enrollment, no-op, joint-membership and leave entries without business callbacks, +synthetic cells or engine installation. Final roots, images and membership must +verify against the advertised complete cut. + +`extend_tail` reuses existing entry objects and captures only the newer local +suffix. Both image forms publish a real local checkpoint and assert the old +prefix is compacted before extension. Starting again from the original local +cut refuses; extension succeeds with exactly two PUTs for the new entry and +replacement index. The old tail index is deleted before independent recovery, +proving that the flattened replacement directly names its complete closure. +This is logical prefix compaction, not qualification of physical log-file purge. + +Live apply retains its original ticket/commit/owner checks around shared physical +transition code. The new membership-root preparation API also lets the lower-level +fixture handle real configuration entries. Materialized archive replay recomputes +incoming LTX in admitted scratch when its destination is cached, then independently +verifies/syncs that destination. A warm seal cannot substitute for the supplied +delta. Extent replay and every final closure retain their independent checks. +Replay admission and pins stay charged with the result and through cancelled +physical work. A materialized case holds the exact warm result-image read, drops +the caller, commits another cell on the healthy quorum, and proves domain shutdown +waits for release before every admission charge refunds. + +All 22 commands pass at the committed source: nine archive cases selected +individually, 156 execution cases, five snapshot cases and four generation +regressions, plus all-target build/Clippy, formatting/layout, dependency isolation, +docs, source preflight, plan validation and 37 Python tests. The 174 distinct Rust +cases run across four executables. The retained record +`.cellar/cell-groups-evidence/CG-06-committed-tail/qualification/checkpoint.json` +binds 323 source hashes, four binaries, frozen plan inputs, hashed outputs, both +failed fixture stages and the preceding checkpoint. Independent source/artifact/ +exact-selection verification passes. The original +`archive_tails_cover_enrollment_and_controls` requirement passes with four real +enrollments, later independent mutations, explicit joint demotion/leave and a +real transferred-leader term no-op. The other four original CG-06 cases remain +pending; no oracle, resource limit, readiness assertion, exact entry count or +timeout was relaxed. + +The retained failures first show a fixture decoding configuration bytes as cell +commands, then a direct campaign refused by a healthy leader lease. The fixture +now shares canonical membership-root preparation and uses production leadership +transfer. Seven other cases already passed before that final setup correction. +Warm wrong-LTX tests positively observe the ordinary cached `Exists` shortcut, +then require archive replay to reject a correctly hashed but wrong SQLite delta. +Missing/corrupt base or entry bytes, omitted/noncontiguous ranges, forged final roots +and damaged warm destinations also refuse without changing selected engine roots. + +**Reflection and next step:** verification still takes an existing destination +engine/applier. Build an isolated data-verification store independent of a serving +engine's advancing cut and surviving voter disks, then implement published, +GC-protected coverage and S3 settlement with bounded outage admission. Continue +archive-aware physical log reclamation and fenced fresh-incarnation seeding. +Shared dispatch still refuses S3 durability. Real RustFS/load and zero-quorum +host/disk-loss qualification, managed recovery, public SDK/fleet cutover and final +legacy removal remain open; this native data-recovery gate does not qualify them. + +## CG-06 independently recovered object-store checkpoints + +Partial source checkpoint: `650fc63892eab59e6fd5023fa90d5f361a8c7302`. +CG-06 is IN_PROGRESS. The group-owned archive now stores complete checkpoints +without synthetic cell identities. One node pool bounds upload/download jobs +and retained metadata across all group handles. One-chunk channels connect +domain-owned physical sources/receivers to object-store I/O; source and receiver +pools separately retain closure and scratch admission through cancellation. + +An immutable bounded index binds the complete group marker and every ordered, +cell-addressed chunk. Existing objects are GET-verified and reused without PUT; +conditional creates require exact read-back even after an ambiguous PUT response. +Corrupt existing payloads refuse overwrite. Upload returns an untrusted locator. +Independent verification downloads every dependency, runs the physical receiver +and checks its complete marker against the index. Preparation leaves the selected +engine roots unchanged and grants no serving authority. + +All fifteen gate commands pass: four new archive cases selected individually, +all five existing snapshot acceptance cases, all-target build/Clippy, +formatting/layout, native dependency isolation, docs, CG-06 source preflight, +plan validation and 37 Python validator tests. Nine distinct Rust cases run across +two executables; ignored snapshot child entry points are exercised by their +process-crash parents. The source-bound record +`.cellar/cell-groups-evidence/CG-06-checkpoint-archive/qualification/checkpoint.json` +binds 317 source hashes, both executables, hashed logs, frozen plan inputs and +the preceding CG-05 checkpoint. An independent hash/source/selection audit passes. + +The new cases use real three-voter Raft Engine and SQLite/LTX with both materialized +and extent images, empty enrollment, and the actual in-memory object-store adapter +with instrumented faults. They verify unchanged-cut reuse without repeat PUTs, +recovery after source-engine closure, a missing/corrupt cell despite another +cell's later cut, ambiguous PUT confirmation, shared cross-group admission, +chunk/index bounds, cancellation, complete-GET timeout and malformed/rehashed +index refusal. Omitting a cell or substituting the advertised cut cannot produce +a verified checkpoint. The five original CG-06 required cases remain pending; +these narrower checkpoint tests do not replace them or their oracles. + +**Reflection and next step:** the critical gap is a complete checkpoint plus +contiguous committed cell/control tail and a published, GC-protected coverage +owner. Shared dispatch still refuses S3 durability. Only confirmed complete +coverage may release S3 outputs or feed the archive retention/reclamation policy. +Then implement bounded outage admission, actual shared-engine log reclamation +and fenced unmanaged seeding under fresh group/cell incarnations. Current objects +follow bounded source chunks; request coalescing and large-group limits need +qualification before any cost/throughput claim. Follow with CG-08 effects/cache +lifetime and CG-07 public SDK/fleet cutover. Real RustFS/load, managed recovery, +independent domains and final removal of the old architecture remain later gates. + +## CG-05 complete native snapshot/rebuild gate + +CG-05 is DONE at `e0550acd95a78cdf1852a408620eb054ee505669` after the complete native gate +and a requirement-by-requirement closure audit. All five required cases pass +individually with exact one-test selection. The expanded gate passes 156 shared +execution cases, nineteen storage cases, six budget cases and nineteen legacy runtime +regressions: 205 distinct Rust cases across five executables. All eighteen +commands pass, including all-target build/Clippy, formatting/layout, dependency +isolation, docs, source preflight and 37 plan-validator tests. No required case, +oracle, command, deadline, configured resource limit or profile was weakened. + +The record +`.cellar/cell-groups-evidence/CG-05-completion-admission/qualification/checkpoint.json` +binds 337 source hashes, five executables, frozen plan inputs, both budget +reproductions, intermediate trials and the preceding checkpoint. Independent +verification and `check_evidence` pass. Task status and the generated implementation +index now select CG-06 as the next dependency-ready task. + +| CG-05 requirement | Direct current native evidence | +| --- | --- | +| One complete same-cut root, membership/configuration and all cells, while later writes continue | `snapshot_contains_every_cell_at_cut`: both image forms, exact selected closure, concurrent cell advances and stale/foreign/corrupt refusal. | +| Verify all receiver dependencies before atomic installation; interrupted state never serves | `partial_snapshot_never_serves`: actual TLS/engine receivers, physical byte verification, missing/corrupt/foreign/budget refusal, revocation/sync cuts, owned seed SIGKILL and fresh authenticated same-disk retry. | +| Whole-group learner, joint membership, handoff and disk replacement preserve permanent claims/quorum | `whole_group_learner_and_joint_change`: complete learner install/catch-up before promotion, explicit/implicit joint changes, leadership handoff, deleted-voter replacement, joint restart and minority refusal. | +| Empty/populated cells and retained checkpoint/current suffix survive physical recovery | `snapshot_empty_cell_and_extent_restart`: Raft-generated TLS catch-up, both representations, independent sequence-1 cells, cold restart and missing-closure refusal. | +| Bounded transfer, live pins, cancellation, joined owners and unrelated-group progress | `snapshot_memory_and_pin_bounds`: 32-cell streaming/scratch pressure, old-reader retention/deletion, shared-store orphan reconciliation, held publication/install shutdown, full SQL apply capacity, eight-slot completion saturation and two concurrent writers through one checkpoint slot. | + +The final implementation fixes two admission faults. A partial refund used to +wake only the first registered waiter, even when its larger request still could +not fit. Release/shrink now wake registered waiters and retain a stored signal for +late worker registration. Separate item and byte atomics could also expose a +temporary item charge from a byte-rejected attempt. Both dimensions now reserve +and refund in one short mutex scope; notifications run after unlocking. Four new +regressions plus the existing two accounting cases pass. The original two wake +failures and subsequent concurrent-reservation failure retain exact source, +output and executable hashes. The notification-only probe passes the former +cases before demonstrating the second fault; the final Cargo gate validates both +fixes together. + +`CellGroupRuntime::usage` exposes advisory lane counters without request admission. +`checkpoint_before_apply` exposes the controller's existing exact-ticket publication +boundary for embedding owners. Eight real fsynced apply completions fill its fixed +admission; a ninth checkpoint retains its source while waiting. Admission refunds +before the actual storage fence, while the source remains charged. Releasing that +fence selects the complete checkpoint; the successor then applies. Both image +forms run directly in the resource acceptance case on explicitly owned, joined +current-thread runtimes. + +Two independent writers additionally enroll 32 real cells per group. While one +receiver holds node-zero capture capacity, both groups continue applying. After +release, both have newer checkpoints at the 25-entry cut, before either writer +finishes its remaining seven entries. All three nodes recover the selected current +and checkpoint closures and every independent sequence-1 cell. This is a bounded +native progress trace, not a measured starvation or latency bound for arbitrary +traffic. + +**Reflection and next step:** the snapshot/rebuild task's declared native +requirements now have implementation and current-source evidence. The next +critical feature task is CG-06: complete group S3 checkpoint plus contiguous +cell-addressed committed tail, independently verified coverage, safe local log +reclamation and fenced unmanaged seeding under fresh incarnations. CG-08 effects/ +cache lifetime and CG-07 SDK/fleet cutover follow. Entry-count policy retains its +idle final tail; idle/time/byte policy and coordinated wake remain future work. +Real RustFS, measured load, remaining crash/deployment schedules, independent +failure domains, managed zero-quorum recovery and removal of the old architecture +remain required in their later task/release gates. CG-06 through CG-16 remain TODO. + +## CG-05 checkpoint progress while SQL apply capacity is full + +Qualified source: `bad0387cb025fbba62f37aa42ba80b81d75366dd`. The production controller now +scans its bounded pending queue even when every SQL apply slot is occupied. +Checkpoint capture/publication uses independent capacity and can start before +SQL work is deferred. This removes a reproduced dependency between unrelated +groups without increasing SQL concurrency or releasing physical ownership early. + +All five required snapshot cases, 152 shared-execution cases and nineteen +Raft-storage cases pass: 176 distinct Rust cases. All sixteen commands pass, +including the complete prescribed CG-05 native gate, all-target build/Clippy, +formatting/layout, dependency isolation, docs, source preflight and 37 plan-validator +tests. The record +`.cellar/cell-groups-evidence/CG-05-checkpoint-apply-scheduling/qualification/checkpoint.json` +binds 335 source hashes, three executables, frozen plan inputs and the prior +checkpoint. Independent hashes and `check_evidence` pass. CG-05 stays IN_PROGRESS. + +Two new cases hold actual node-zero image-directory persistence, once with +materialized apply and once with extent apply. The sole SQL apply slot and its +retained completion stay charged. A second group commits a successor, which the +other two voters apply. Node zero publishes that group's preceding complete +checkpoint while preserving its unchanged current root and unapplied suffix. +After physical release both groups apply, independent sequence-1 cells verify, +and controller shutdown joins all owners. The exact resource-bound snapshot case +directly runs both scenarios. + +Both cases failed before the production change at the same five-second checkpoint +deadline. A second reproduction first verified the other voters' applied successor; +diagnostics showed one active and one pending apply, zero active checkpoints and +zero checkpoint sources. The worker stopped scanning solely because SQL capacity +was full. Moving that capacity check after checkpoint admission makes both cases +pass. The initial and diagnostic reproductions retain their exact source copies, +outputs and executables; an intermediate diagnostic-format compiler failure is +also retained. No timeout, assertion, resource bound or persistence fence changed. + +**Reflection and next step:** checkpoint capture is now independent of occupied +SQL apply capacity. Publication with an exhausted completion lane and capture +fairness under sustained traffic still need direct qualification before closing +CG-05. Then deliver CG-06 complete S3 checkpoint/tail coverage and safe Raft log +reclamation, followed by effects/cache lifetime and SDK/fleet cutover. Real RustFS, +load, independent failure domains, zero-quorum recovery and final legacy removal +remain open. This gate does not combine earlier LTX binaries into its case count. + +## CG-05 idle-node orphan reclamation and authenticated cold retry + +Qualified source: `5cfa9fb652c32cc75934ef3d0a32e22ffc009f35`. `ReplicaLog::reclaim_cell_storage` +now reconciles the complete physical engine inventory and collects bounded batches +of unselected image/page files. Both materialized and extent images are supported. +The complete prescribed CG-05 command gate passes at this source: all five exact +snapshot cases, all-target build and Clippy, formatting and layout. The expanded +gate also passes 150 shared-execution cases, nineteen Raft-storage cases and +38 relevant LTX cases, plus dependency isolation, docs, source preflight and +37 plan-validator tests. All nineteen command logs pass. + +The record +`.cellar/cell-groups-evidence/CG-05-node-orphan-retry/checkpoint.json` +binds 335 source hashes, six copied executables, frozen plan inputs, the prior +checkpoint and intermediate trial outputs. Independent hashes and the plan's +`check_evidence` pass. CG-05 remains IN_PROGRESS; the native gate does not close +the remaining lifecycle/pressure obligations or qualify the production cutover. + +Maintenance requires joined runtime and SQL ownership and holds both exclusive +claims through physical completion. It runs on the existing engine worker, where +group creation/publication cannot interleave. It refuses missing, duplicate, +foreign, oversized or busy inventories. Before deletion it verifies every group's +current and retained checkpoint closures, merges image references across shared +stores and retains caller-supplied durable archive/transition roots. The node must +exclusively own those image stores. Existing live reader pins and file/directory +sync barriers still govern deletion and capacity refunds. This is idle-node orphan +maintenance; it supplies no archive coverage, Raft log truncation or serving authority. + +Two new shared-inventory cases use two groups with one image store in each form. +They retain both groups' current/checkpoint images, collect an actual orphan with +`max_files=1`, preserve and read an independent old image, then collect it after +its reader releases. A queued request retains engine/SQL claims after its waiter +is aborted, and cancelled engine close waits for physical completion. + +Two new retry cases run four additional owned SIGKILL scenarios: partial and synced +dependency writes in both image forms. The original voter disks reopen with fresh +process identities, establish a new term/quorum and commit a newer cell before +retry. The same learner disk collects its unselected transfer debris while its +empty stage still refuses bootstrap. Fresh TLS seed installation supplies the +authenticated local receipt; ordinary routing refuses before that receipt. The +learner then applies a live Raft suffix. A second cold reopen verifies all four +independent sequence-1 cells and both current/checkpoint closures. These cases +retain the voter disks; they do not exercise zero-quorum S3 recovery. + +The supplemental `generation` library filter selected zero tests. Its output is +retained as non-qualifying diagnostic evidence; the actual `private_clone` and +`extent_store` integration targets supply the relevant image-store regressions. +Compiler failures during fixture wiring and a correctly refused pre-seed route +are retained in the intermediate logs. Context probes identified the premature +ordinary route installation. The fixtures now assert that refusal and install the +route only after the authenticated receipt; source checkpoint registration follows +the fresh route. No production transport/fencing check or assertion was relaxed. + +**Reflection and next step:** same-disk orphan cleanup and authenticated retry are +implemented at the idle-node API boundary. Managed orchestration must eventually +invoke this boundary with every durable reference. Finish sustained-write capture +fairness and remaining controller/completion-budget/root-pressure qualification +before closing CG-05. Then implement CG-06 complete S3 checkpoint plus contiguous +tail coverage and safe local Raft log reclamation. CG-08 effects/cache lifetime and +CG-07 SDK/fleet cutover follow; topology, placement, hibernation, fenced recovery +under fresh incarnations, real RustFS/load and final legacy removal remain open. + +## CG-05 controller publication ownership through cancelled shutdown + +Qualified source: `370dcf3465b3161c9e69e8b1f398bc9d49450c0c`. Both new controller publication tests pass +in the complete native execution suite: 146 cases, plus nineteen native +Raft-storage cases and the updated exact `snapshot_memory_and_pin_bounds` case. +Ten scoped commands pass, including Clippy for all three integration targets, +formatting, layout, dependency isolation, docs, source preflight and 37 plan +validator tests. The other four snapshot acceptance cases and all-target checks +were not rerun at this source; their complete earlier gate remains below. + +The record +`.cellar/cell-groups-evidence/CG-05-controller-checkpoint-shutdown-final/checkpoint.json` +binds 332 source hashes, three copied executables, frozen plan inputs and +intermediate trial outputs. Independent source/output/executable hashes pass. +This partial record is not a combined full task gate and keeps CG-05 IN_PROGRESS. +Production runtime behavior is unchanged. The first scoped gate passed all +selected Rust cases but failed Clippy on an unused helper in the shared legacy +storage target. That method was replaced with fixture-local predicate setup. +The failed source, outputs and three executables remain hash-bound in +`previous_attempt`; no assertions or fault scenarios were removed. + +The filesystem fault now supports an opt-in sync predicate. Earlier ordinary +append/commit syncs pass. Once the controller owns its checkpoint lifecycle slot +and captured source, the next actual log sync holds. The test waits for all voters +to persist the other group's election before arming this cut; it does not guess a +sync count. Both materialized and extent cases check the following directly: + +- Source count, retained bytes and the one controller slot stay charged. Another + receiver refuses with `Busy`; the prior root and applied position stay selected. +- Four writes to the other group commit and apply on the two healthy voters while + this node's shared engine is stalled. No checkpoint completion is credited early. +- Cancelled close and wait retain physical ownership. Replacement runtime and SQL + pool ownership refuse until the real sync returns. +- Release joins controller shutdown, selects the captured complete checkpoint, + keeps the committed successor unapplied and drains checkpoint admission. A new + runtime can then acquire the engine. The exact resource-bound acceptance case + directly composes both scenarios. + +This qualifies pending publication through cancelled controller shutdown in both +image forms. It does not close every controller crash, completion-budget, capture +fairness or retained-root pressure schedule. Native process tests do not qualify +independent domains, power loss, real RustFS/load or the SDK/fleet cutover. + +**Reflection and next step:** interrupted-seed retry is now the critical recovery +availability gap. The extent inventory deliberately keeps partial known files +charged after reopen, and immutable import verifies existing names instead of +silently overwriting them. Retry needs complete node-reference reconciliation +before orphan collection, preserving all selected current/checkpoint roots and +live pins across groups. Do not add group-local deletion of shared objects or +weaken corruption refusal. Implement and test that reconciliation and a fresh +authenticated retry before closing CG-05 and proceeding to CG-06 archive/reclaim. + +## CG-05 exact native snapshot acceptance target + +Qualified source: `f076731a2d384bff65914828409c01506438c4a2`. The declared +`native_cell_groups_snapshot` target now contains all five required named cases, +plus two ignored child entry points used by its actual process-kill parents. +Every required case passes separately with `--exact --test-threads=1`; inventory +and exact one-test results pass the plan's evidence validator. The +execution target's 144 regression cases also pass. Fifteen commands pass: all ten +prescribed CG-05 commands, the execution regression suite, dependency isolation, +docs, source preflight and 37 plan-validator tests. The prescribed commands include +all-target checks and Clippy for cellar/cellar-logic, formatting and layout. + +The record `.cellar/cell-groups-evidence/CG-05-snapshot-acceptance/checkpoint.json` +binds 332 source hashes, two executables, frozen plan inputs and intermediate +logs. Independent hash verification and `check_evidence` pass. No required case +or command was removed or renamed. This record does not mark the task complete; +remaining lifecycle/availability gaps below keep CG-05 IN_PROGRESS. + +The cases directly compose existing real-engine, SQLite/LTX, TLS and owned-crash +fixtures. They do not infer an oracle from another test process's success. The +engine helper extraction preserves its body byte-for-byte after keeping the +execution-only import and restart method in the original target. The process-kill +children retain their exact names and run the same snapshot executable. Fixture +progress messages use ordinary test capture, leaving the validator's required +per-case pass lines unambiguous while retaining failure diagnostics. + +| Exact case | Direct native coverage | +| --- | --- | +| `snapshot_contains_every_cell_at_cut` | Both image forms, two concurrent cell advances, exact original closure, stale/foreign cut refusal and corrupt physical bytes. | +| `partial_snapshot_never_serves` | Receiver corruption/partial/foreign/budget failures, exact Raft persistence, forged TLS seed data, revoked source, sync failures and twelve owned seed SIGKILL scenarios. | +| `whole_group_learner_and_joint_change` | Fresh quorum seed authority, deleted-voter replacement, complete learner install and catch-up before promotion, explicit/implicit joint changes, leader handoff, four explicit joint SIGKILL scenarios, sync ownership and minority refusal. | +| `snapshot_empty_cell_and_extent_restart` | Raft-generated TLS catch-up, empty and populated enrollment, retained checkpoint/current suffix cold recovery and missing closure refusal in both forms. | +| `snapshot_memory_and_pin_bounds` | 32-cell transfers, scratch/metadata admission, unique old reader preservation and actual deletion after release, cancellation, joined publication/installation shutdown and controller pool contention. | + +The audit strengthened two fixture oracles. Replacement-group and unrelated-group +writes now commit while the receiver is physically held at its install fence; +the learner later obtains those cell changes through the Raft suffix. Cold +checkpoint restart explicitly verifies empty enrollment remains at sequence 1, +transaction 0 and an empty image, while alpha/beta/suffix retain populated images +at sequence 1. + +The initial composite membership test overflowed its stack even with boxed +futures. Phase probes located the failure entering seed installation, whose +standalone counterpart passed. A non-inlined future factory releases construction +temporaries before polling nested fixtures; the complete original case then +passes in both image modes. No stack limit, timeout, assertion or fault was +relaxed. Temporary phase probes were removed and their logs retained. Production +runtime behavior is unchanged. + +**Reflection and next step:** the exact test target and its prescribed command +gate are delivered. The next concrete test should hold controller checkpoint capture, fill its +lifecycle capacity, then cancel close and verify that physical owners remain +charged until release and shutdown joins them. Follow with staged-orphan cleanup +and authenticated retry on the same disk; safe +refusal of interrupted state alone does not prove availability. Retain explicit +gaps for continuous capture fairness, minority signed receipts and retained-root +pressure. This is native qualification, not real RustFS/load, independent domains +or production SDK/fleet cutover. Complete the remaining CG-05 obligations before +CG-06 archive/reclamation; the full hard cutover remains pending. + +## CG-05 interrupted seed dependency writes and process revocation + +Qualified source: `6e56f21f2209ce3f02d78ffeeb3150a763a1476e`. All 144 native shared-execution tests +and nineteen native Raft-storage tests pass. Three ignored child fixtures are invoked +by their process-crash parents and are not standalone passes. Eight scoped commands +pass: Clippy for both targets, both complete suites, formatting, layout, dependency +isolation, docs, source preflight and 37 plan-validator tests. The record +`.cellar/cell-groups-evidence/CG-05-seed-transfer-cuts/checkpoint.json` binds +330 source hashes, two executables, frozen plan inputs and intermediate logs. +Independent hash verification passes. Other suites were not rerun at this source. + +The authenticated seed crash matrix now runs six cuts in each image form: staged, +partial dependency write, synced dependency, before engine sync, after engine sync +and publication. Four new SIGKILL scenarios use the production TLS source/receiver +and hold either an anonymous materialized-image spool or an extent payload. The +partial cut writes and syncs a prefix; the second cut finishes the file sync but +stops before complete group installation. The child observes nonzero physical +bytes, the unchanged empty stage, no checkpoint and refused bootstrap before +signaling the parent to kill it. The parent joins the killed child, checks the +recorded extent file length against its encoded full length, then cold-opens the +original learner disk with a new process identity. Both new cuts select the empty +stage with no authenticated local receipt and refuse startup. + +Two new transfer-revocation cases hold a physical dependency write and replace +the source process. The network waiter returns stale/closed authority while the +physical receiver remains admitted: both checkpoint metadata and transport data +reservations remain charged. Releasing the write lets the receiver observe the +revocation and finish cleanup; admission drains without publishing a root or +checkpoint. Cold recovery independently confirms the empty stage and refused +bootstrap. These cases use an independently authenticated captured peer; the +crash cases use the production source server. + +The shared filesystem fixture adds two opt-in dependency faults. Default behavior +and existing fault modes are unchanged. Production runtime code is unchanged. +All original voters and the learner share the killed child; this is not device +power-loss or independent-domain qualification. These cuts do not cover every +wire/manifest boundary, automatic cleanup and fresh-seed retry after a partial +orphan, measured load, RustFS or zero-quorum recovery. Existing pressure, joint +restart and seed installation cases remain green in the complete execution run. + +**Reflection and next step:** interrupted dependency writes now have direct native +failure and restart evidence. The next critical step is to audit the five exact +CG-05 acceptance oracles against these fixtures, implement their required test +target and close remaining controller/lifecycle saturation gaps. Keep unverified +orphan cleanup/retry, continuous capture fairness, minority signed receipts and +retained-checkpoint pressure explicit. Broader fault/load requirements remain in +the release matrix. Complete CG-05 before CG-06 verified S3 archive/reclamation; +production SDK/fleet cutover and final legacy removal remain pending. + +## CG-05 bounded checkpoint transfer and reader retention + +Qualified source: `a2036e5074066eb7585ea82a2eb62331368ccca5`. All 142 native shared-execution tests +pass, including two new pressure cases; two ignored child fixtures are invoked by +their process-crash parents and are not standalone passes. Eight scoped commands +pass: execution-target Clippy, the complete execution suite, formatting, layout, +dependency isolation, docs, source preflight and 37 plan-validator tests. The +record `.cellar/cell-groups-evidence/CG-05-checkpoint-pressure/checkpoint.json` +binds 327 source hashes, one executable, frozen plan inputs and intermediate +logs. Independent hash verification passes. Other integration suites were not +rerun at this source. + +Both image forms transfer 32 cells, including empty enrollment, through the +production source and independent receiver. The aggregate payload exceeds the +source reservation; emitted chunks stay within 4093 bytes. Materialized reception +uses one image-sized scratch file across the entire group. A second valid stream +refuses with `ScratchFull` without refunding the first receiver's reservation or +publishing partial state. Extent reception uses no materialized scratch file. +Independent receiver item and byte saturation refuse without leaking admission. + +While the transfer is held inside its first image, two original voters commit +changes to two cells and attempt physical reclamation. The captured tree remains +present until the source releases it. The receiver verifies every original cell's +digest and sequence at the captured cut; it never substitutes the newer images. +A completed but unpublished receiver still leaves the prior durable root selected +and retains its admission. Source, receiver and scratch usage return to zero only +after their physical owners finish. + +An independent old physical reader remains valid after the source and its tree +pin are gone. The pressure cohort has unique cell contents, so other cells cannot +accidentally preserve that obsolete image. Reclamation keeps it while the reader +exists, and deletes it after the reader releases. This proves actual retention +and release in both materialized and extent stores. + +Production runtime behavior is unchanged. This is a direct physical-stream test +with three real engines in one process, not a TLS pressure benchmark or measured +RSS ceiling. The independent read pin is a physical view, not new quorum authority. +It does not qualify continuous-write capture fairness, all controller saturation +paths, independent failure domains, RustFS or zero-quorum recovery. Earlier seed +and explicit joint-consensus process-crash cases also pass in the full suite. + +**Reflection and next step:** bounded 32-cell transfer and reader-image retention +now have native evidence. The critical remaining CG-05 work is the missing +mid-stream revocation/crash and controller-lifecycle pressure coverage, followed by +all five exact acceptance cases at one source. Keep minority signed-receipt refusal, +continuous capture fairness and retained-checkpoint lifecycle pressure explicit. +Complete that task gate before CG-06 verified S3 archive/reclamation; production +SDK/fleet cutover and final legacy removal remain pending. + +## CG-05 explicit joint-consensus restart and completion + +Qualified source: `5f30cc6b28ae01c2e6411507ea11716ef94e6456`. All 16 selected seed/membership +tests pass. Two new parent cases cover four owned process kills: after all replicas +apply explicit joint entry and after all apply joint exit, in both image forms. +Two ignored child fixtures are invoked by their parents and are not standalone +passes. Nine scoped commands pass: Clippy for the execution target, seed tests, +membership tests, formatting, layout, dependency isolation, docs, source preflight +and 37 plan-validator tests. The record +`.cellar/cell-groups-evidence/CG-05-joint-process-restart/checkpoint.json` binds +326 source hashes, one executable, frozen plan inputs and intermediate logs. +Independent hash verification passes; broader predecessor suites remain historical. + +The original group has three voters and two independent SQLite cells. Explicit +joint entry demotes member 3: incoming voters are 1/2, outgoing voters remain +1/2/3 and learner-next is 3. The child is killed with SIGKILL after the selected +replicated boundary. Every original replica disk is reopened with a new process +identity, preserving its permanent claims. Before runtime admission, recovery +checks the exact complete root, membership revision and both configuration sets. +After joint exit, the recovered configuration instead contains voters 1/2 and +learner 3, with no outgoing voters or learner-next entries. + +Production controllers and TLS transport establish a fresh leader. The recovered +group commits a new cell under its restored configuration, explicitly completes +any pending joint transition, and commits another cell after completion. All three +replicas apply the four-cell state. Fresh linearizable reads verify both original +and both new cell histories with per-cell sequence 1; group log progress does not +change those logical sequences. The demoted member remains non-serving. + +The seed and membership suites share owned-child SIGKILL/join and bounded frozen +synthetic-catalog helpers. Seed crash regression cases remain green. Production +runtime behavior was unchanged. All original replicas share the killed child and +all recovered replicas share the parent; the frozen catalog is test input, not a +production recovery backend. These cuts occur after replicated entry/exit, not +at every in-flight configuration fence or automatic-transition crash boundary. +Independent failure domains, device power loss, RustFS/load and zero-quorum +recovery remain unqualified. + +**Reflection and next step:** explicit joint membership now survives process loss +and completes with fresh quorum authority. The next critical gap is bounded +large-group transfer and retained-pin pressure, followed by consolidating the +existing evidence into all five exact CG-05 acceptance cases. Mid-stream fault, +minority receipt and controller-pressure requirements remain open. Complete CG-05 +before CG-06 verified S3 archive/reclamation; the full hard cutover remains pending. + +## CG-05 seed installation across process kills + +Qualified source: `b997a48ba686fceb2531cd96ae2734c9bed84fc9`. All 11 selected seed tests pass, +including two new parent tests covering four process-kill cuts in each image form. +One ignored child fixture is invoked explicitly by each parent; it is not counted +as a standalone pass. Eight scoped commands pass: Clippy for the execution target, +seed tests, formatting, layout, dependency isolation, docs, source preflight and +37 plan-validator tests. The source-bound record +`.cellar/cell-groups-evidence/CG-05-seed-process-crash/checkpoint.json` binds +324 source hashes, one copied executable, frozen plan inputs and intermediate +logs. Independent hash verification passes. Broader predecessor suites remain +historical results, not additional current-source passes. + +Each owned child starts a real three-voter shared group with two SQLite cells, +commits a fresh learner and transfers its seed using the production TLS sender and +receiver. The parent waits for a complete reached-cut signal and kills only that +child, checks its Unix exit signal is SIGKILL, and joins it. Cuts occur after durable +staging, before the destination engine sync, after that sync and after publication. +The child and parent execute the same test binary, retained with this checkpoint. + +Cold recovery opens the original learner disk with a new process identity and +verifies the selected root plus physical closure. The staged-only cut has no +checkpoint or authenticated local receipt and refuses startup. The pre-sync cut +allows either complete old or complete new state; both observed runs retained the +new seed. Post-sync and publication cuts require the complete seed, both cells, +retained checkpoint, learner membership and local receipt. Attaching a recovered +learner still refuses a valid enrollment without quorum and cannot advance its root. + +The parent restores a bounded frozen copy of this fixture's synthetic catalog, +including the permanent destination claim captured before the fault. It never +manufactures a new claim from recovered disk contents. This fixture catalog is +not an object-store durability implementation or S3 disaster-recovery protocol. +All source voters and the destination share the killed child process. The tests +qualify these learner-install crash cuts, not independent voter domains, device +power loss, mid-stream crash cuts, joint-transition restart, RustFS or zero-quorum +recovery. Production runtime behavior was unchanged. + +**Reflection and next step:** seed engine-boundary process kills now have direct +native evidence. The next critical gap is restarting and completing joint consensus +from retained disks. Follow with mid-stream cancellation/crash and pressure coverage, +map the existing proof to all five exact CG-05 cases, and complete that gate before +CG-06 verified S3 archive/reclamation. CG-05 remains IN_PROGRESS and the full hard +cutover remains unfinished. + +## CG-05 seed persistence revocation and sync outcomes + +Qualified source: `3ea90fb5d67d7c66aa842781788c403aa2afc80a`. All 9 selected seed tests pass, +including two new tests covering four persistence cuts in each image form. +Eight scoped commands pass: Clippy for the execution integration target, selected +seed tests, formatting, layout, dependency isolation, docs, source preflight and +37 plan-validator tests. The record +`.cellar/cell-groups-evidence/CG-05-seed-persistence/checkpoint.json` binds +323 source hashes, one copied executable, frozen plan inputs and retained +intermediate logs. Independent hash verification passes. Broader predecessor +results are historical; this checkpoint does not rerun the full shared suites. + +A held node-revision fence and the actual engine queue metric establish that the +seed install is queued. Reopening the source engine with a fresh process and +publishing its NodeView through the catalog revokes the admitted grant. The +engine refuses it with WrongIdentity before publication. Both live and cold +recovered destination roots remain the empty stage, with no checkpoint or startup +receipt. The pending install retains physical receive admission until the held +engine work completes. + +A separate cut revokes the source after the seed enters the filesystem fence. +The install retains its owners until sync completes and preserves the complete +seed despite late revocation. The receipt cannot appear before the fence. +Cold recovery revalidates the selected and retained roots, both cell images and +learner membership. A recovered learner refuses a valid enrollment without quorum. + +Injected failures immediately before and after the underlying sync return typed +I/O errors. The uncertain engine refuses metadata reads, further writes and +bootstrap; joined runtime/storage shutdown reports the failure. Reopening verifies +the complete root and local receipt together. The pre-sync cut permits either +complete old or complete new selection; the post-sync error requires the new +seed. Both observed pre-sync runs recovered the seed, illustrating why failure +cannot imply rollback. This uses an independently authenticated TLS fixture fed +by a real quorum-authorized cut; production receiver/storage behavior was unchanged. + +**Reflection and next step:** queued and entered process revocation plus sync-error +reconciliation now have direct coverage in both image forms. The most critical +remaining gap is actual process-kill installation and joint-transition recovery. +Next add child-process cuts around the seed fence and verify complete recovery, +then finish transfer-time revocation, signed minority receipts, large/empty-group +pressure and all five exact CG-05 acceptance cases. No OS process kill, power loss, +managed fleet, shared-group RustFS/load or zero-quorum recovery is qualified here. +CG-05 remains IN_PROGRESS; archive/reclamation and production hard cutover remain +required downstream work. + +## CG-05 receiver faults and physical verifier completion + +Qualified source: `3db498eacf4bd0621b7a0ecdf11ee68fe2a3ed32`. All 167 selected Rust cases pass across shared +execution, catalog and storage. All eight commands pass: all-target Clippy for +Cellar/core/LTX, those three native suites, formatting/layout, dependency isolation, +docs, source preflight and 37 plan-validator tests. The source-bound record is +`.cellar/cell-groups-evidence/CG-05-seed-receiver-faults/checkpoint.json`, with +322 source hashes, three copied executables, frozen plan inputs and retained +intermediate logs. Independent hash verification passes. The unchanged legacy +snapshot/storage suites were not repeated in this scoped gate; their earlier +results remain in the predecessor record. CG-05 remains IN_PROGRESS. + +Adversarial receive tests exposed a production resource-lifetime defect in both +seed and ordinary group-checkpoint transport: a physical worker could reject a +chunk while the network loop waited for the peer's next header. The connection +and network admission survived until timeout. Both loops now observe physical +completion throughout network reads and bounded-channel sends. Failure closes the +exchange promptly. Explicit End, complete closure verification and the real engine +fence remain necessary for installation. Work already admitted to persistence is +still owned until completion. Both original failures are retained in the evidence. + +Two new cases use independent TLS test peers in both image modes. Their payload +comes from a real quorum-authorized complete source; the deliberately small fixture +buffers less than 2 MiB of chunk payload and is not a production streaming +benchmark. Ten faulty +replies cover exporter authentication, correlation, nonce, source process, payload +digest, impossible/noncanonical barrier, unknown fields, invalid descriptor and +oversized payload. They cannot create a destination stage. Eleven chunk/end faults +cover wrong binding/sequence/digest, unknown fields, bad physical offset, premature +End, oversized chunk, disconnect after the first part, replay, and malformed End +offset/body. They cannot change the staged root, install a checkpoint or authorize +bootstrap. Physical and wire admission release after every case. A valid transfer +then installs both cells on the same engine and identities. + +The fixture captures a later cut and admits its signed descriptor, then closes and +reopens the source engine with a new process generation. Normal catalog/storage +publication produces a newer NodeView. Accepting that view revokes the pending +session; both current and retained destination roots stay unchanged and an old +source hint explicitly returns StalePeer. This is revocation before install begins, +not a process-kill or revocation during queued/entered engine persistence. The +ordinary checkpoint protocol also has a held-open malformed-offset regression. + +**Reflection and next step:** seed reply/chunk refusal and pre-install process +revocation now have direct native coverage, and verifier failure no longer retains +network admission until timeout. Next qualify revocation during transfer/queued +persistence, sync-error/process-kill installation outcomes and joint-transition +restart. Signed minority receipt refusal, large/empty group pressure, retained pins, +sustained capture fairness and all five exact CG-05 cases remain required. No +managed fleet, independent-process, shared-group RustFS/load/power-loss/zero-quorum +qualification is claimed. Complete CG-05 before verified S3 archive/reclamation +and the effects/cache plus SDK/fleet cutover tasks. + +## CG-05 seed refusal and restarted voter reconnection + +Qualified source: `de552f6daabf786c14fe9df97565538ba5d302b4`. All 238 Rust cases and eight gate commands +pass, including all-target Clippy for Cellar/core/LTX, five native suites, +formatting/layout, dependency isolation, docs, source preflight and 37 plan-validator +tests. `.cellar/cell-groups-evidence/CG-05-seed-fault-reconnect/checkpoint.json` +binds 321 source hashes, five copied executables, frozen plan inputs and +retained intermediate logs. Independent hash verification passes. These strengthen +two existing replacement cases; the test count is unchanged. CG-05 remains IN_PROGRESS. + +Both image modes now exercise independent wire probes against the production TLS +seed source. Eleven invalid requests cover exporter authentication, stale process, +wrong disk/incarnation/schema, zero and noncanonical nonce, zero and excessive +timeout, unknown fields and an ineligible voter. Five post-reply scenarios cover +wrong Ready sequence, a previous exchange's binding, unknown Ready fields, source +endpoint revocation and receiver disconnect. The probe independently verifies reply +framing, exporter-bound MAC, correlation and digest. Every scenario releases the +source pin and wire admission. A normal complete transfer later succeeds in the +same fixture, so refusal is not inferred from an unusable source. + +Production destination sessions also refuse source revocation after the signed +descriptor and expiry while the user retains the session. Neither changes the +staged root, installs a checkpoint or grants joined-member bootstrap. The same +engine and identities then complete installation and replacement. + +After cold recovery, the replacement now starts a new runtime, transport and +controller with a new process generation. Explicit physical routes remove retired +voter 3 and advertise the new endpoint. Survivors reconnect over TLS, the replacement +regains quorum and commits another cell. A linearizable read succeeds. Isolating +that group then leaves a deliberately stale ready flag: a proposal is admitted, +but cannot advance applied state or the selected root, and a fresh linearizable +read expires. After check-quorum stepdown, new proposals refuse. Admission remains +distinct from commitment. The fixture uses no event pumping after controller start. + +Two intermediate failures were fixture mistakes, retained in the evidence: the +first route still advertised retired voter 3; the second advanced only the restarted +voter's clock while survivor leader leases remained frozen. The final fixture +advances every voter and uses normal caught-up leadership handoff if a survivor +wins first. No production check or safety assertion was weakened. + +**Reflection and next step:** native reconnect and minority commit/read refusal are +now covered. Next test malformed/replayed seed replies and chunks at the receiver, +source process-generation revocation during admitted work, and sync failures or +process kills across seed installation and joint configuration. Signed receipt +refusal, large/empty group pressure, retained pins, sustained capture fairness and +the five exact CG-05 acceptance cases remain open. These tests restart runtime +owners in one OS process; they do not qualify managed fleet orchestration, +independent-process failure, power loss, shared-group RustFS/load or zero-quorum +recovery. Complete CG-05 before CG-06 archive/reclamation and CG-08/CG-07 cutover. + +## CG-05 authenticated learner installation and replacement + +Qualified source: `ab2a308a937f349d3482a825d4b669d042cb03c3`. All 238 selected Rust cases and eight +gate commands pass, including all-target Clippy for Cellar/core/LTX, five native +suites, formatting/layout, dependency isolation, docs, source preflight and 37 +plan-validator tests. `.cellar/cell-groups-evidence/CG-05-learner-install-final/checkpoint.json` +binds 320 source hashes, five copied executables, frozen plan inputs and logs. +Transfer this ignored evidence with its predecessors. CG-05 remains IN_PROGRESS. + +A dedicated authenticated seed exchange now binds the exact physical identities, +fresh group ReadIndex barrier and canonical complete checkpoint. Destination +staging reconciles permanent node/disk/engine and member claims but cannot serve. +Only a verified complete transfer followed by the actual engine fence installs a +local seed receipt atomically with configuration, membership and both root cuts. +The receipt never travels in snapshots; generic snapshot installation cannot mint +it. Joined-member bootstrap requires this receipt, the unchanged permanent claim +and active membership. Startup still needs physical recovery and fresh quorum. +The internal marker-1 group application record now requires `manifest` and `seed`; +the previous bare-manifest shape refuses. No legacy reader or data deletion is added. + +Two new tests use production controllers, real TLS, independent native engines +and SQLite/LTX in both image forms. They delete voter 3's fixture disk before +transfer and prove its old identity cannot recreate storage. An isolated source +cannot create a stage. Staged cancellation retries retain exact identities and +the original deadline; transient source-slot ownership is allowed to finish. +Initial-member bootstrap and generic-snapshot bootstrap refuse the staged learner. +A held engine sync cannot publish the receipt. Cancelling the materialized caller +retains physical owners through sync; `recover_selected` reconciles using the same +page store without bypassing its lock. The new learner installs every prior cell, +catches up a later Raft suffix while another group progresses, joins the voter set +through joint consensus, retires member 3, takes leadership and commits another +cell. Cold reopen verifies current and retained checkpoints, then starts a new +process owner that is correctly not quorum-ready. + +The tests also exposed a production joint-leave bug: RawNode's historical hash +iteration order can differ from pure configuration replay. Owner completion now +compares validated canonical configurations, preserving all sets and auto-leave +state. Fixture setup was split to avoid an oversized combined debug future; no +stack limit or safety assertion was relaxed. Intermediate failures and the final +passing runs are retained, and temporary debug probes were removed. The first +broad gate hit eight macOS socket-address allocation errors; all ten remote RPC +cases passed unchanged in a targeted recheck, followed by the complete final gate. +Host samples showed thousands of transient loopback sockets. The failed broad +run and diagnostics are retained; no global network settings were changed. + +**Reflection and next step:** the fresh-disk replacement path is implemented and +locally exercised. The immediate remaining CG-05 work is adversarial seed-frame +and process-revocation coverage, sync-error/process-kill and joint-transition crash +cuts, restarted-peer reconnection, minority write/receipt refusal, and the complete +five-case gate including empty-cell and pressure/pin scenarios. Sustained-write +capture fairness and controller pressure cuts remain open. Complete those gates, +then deliver CG-06 verified S3 archive/reclamation, CG-08 effects/caches and CG-07 +SDK/fleet cutover. Managed fleet orchestration, shared-group RustFS/load, power-loss +and zero-quorum DR remain unqualified; native disk deletion is not those proofs. + +## CG-05 live-quorum learner seed source + +Qualified source: `0c3769eccbb845a5943ef60e771c107f1800d9f2`. All 236 selected Rust cases and eight +gate commands pass, including all-target Clippy for Cellar/core/LTX, five native +suites, formatting/layout, dependency isolation, docs, source preflight and 37 +plan-validator tests. `.cellar/cell-groups-evidence/CG-05-seed-source/checkpoint.json` +binds 312 source hashes, five copied executables, frozen plan inputs and logs. +Transfer this ignored evidence with its predecessors. CG-05 remains IN_PROGRESS. + +A fresh correlated group ReadIndex now authorizes a complete applied seed root, +including a group with no cells. Only a committed new learner with the exact +node/disk binding outside both voter sets qualifies. Pending requests and retained +roots share existing node/group read budgets. No per-cell worker or new serving +mode is introduced. Capture consumes a non-cloneable authority and requires the +exact still-current root; if apply advanced first, a fresh selection is required. +Once captured, ordinary writes may continue while the immutable cut streams. +Membership changes, leadership loss/handoff, shutdown and deadline expiry revoke +the source. Checks before and after sink calls and final completion prevent a +revoked stream from reporting a successful End. + +Three new cases use production controllers and real TCP quorum traffic. Both +image forms stream complete cuts into existing receiver verification. Tests cover +uncommitted/wrong/retired member refusal, follower refusal, stale capture, empty +groups, per-group admission exhaustion, independent group progress, leadership +transfer and shutdown. Removing membership while a source blocks in its sink +revokes completion but retains pool admission until physical work exits. Expiry +during an empty group's final header chunk also refuses completion. An isolated +leader still reporting ready cannot issue a fresh seed; another group continues +real writes. Initial and strengthened targeted runs passed and are retained. + +**Reflection and next step:** source authorization is now implemented, but no new +destination is admitted by this API. The requested process identity is bound, not +authenticated. The next feature-critical slice is authenticated seed RPC with +process revocation, permanent destination node/disk/engine claims, complete atomic +installation and joined-member runtime attachment. Coordinate capture with the +controller apply boundary or bound fresh-selection retries under sustained writes; +source refusal is safe but does not establish capture fairness. Then demonstrate +real catch-up, promotion and replacement after deleting a voter disk, one-voter-loss tolerance +and minority write refusal. Preserve all five CG-05 cases and remaining receiver, +joint-transition and pressure/crash cuts. CG-06 verified archive/reclamation follows, +then CG-08 effects/caches and CG-07 public SDK/fleet cutover. Shared-group RustFS, +load, power-loss and zero-quorum DR remain unqualified. + +## CG-05 durable membership and leadership handoff + +Qualified source: `1814b27896c3c21166186fe8fbd1a1a6bdbb721a`. All 233 selected Rust cases and 8 +gate commands pass, including all-target Clippy for Cellar/core/LTX, five native +suites, format/layout, dependency isolation, docs, source preflight and 37 validator +tests. `.cellar/cell-groups-evidence/CG-05-group-membership/checkpoint.json` binds +310 source hashes, 5 copied executables, frozen plan inputs and logs. +Transfer this ignored evidence with its predecessors. CG-05 remains IN_PROGRESS. + +The owner/runtime now accepts canonical, bounded V2 membership intents bound to +one complete route and observed membership revision. Pure replay computes the +configuration without changing the live RawNode. Ordered apply preserves every +cell leaf while advancing the group position and membership revision. The engine +independently verifies the committed entry and atomically syncs configuration, +permanent member bindings and the complete root. Only its matching completion +changes live Raft progress and advances apply. Explicit and implicit joint changes +use this same boundary. Raft-generated automatic leave starts replication immediately; +it does not wait for a later write or tick. + +Fresh IDs first enter as learners with node/disk bindings. The permanent high-water +mark prevents retired-ID reuse. Promotion requires leader-observed replication +through its durable committed cut. Handoff targets another caught-up incoming voter +and revokes captured producer authority while it proceeds. Synced voter checks +also fence demoted/removed leaders before the owner consumes cache publication. +New learner routes may lag committed membership: ordinary and checkpoint sends +return retryable stale-peer failures without closing the node. + +Three new tests cover production controllers and real TCP Raft in both physical +image forms, plus held configuration sync and cancelled/joined runtime close. +Lifecycle cases preserve cell identity/version, exercise explicit and automatic +joint changes, stale/foreign/malformed refusal, learner binding, unsafe-promotion +refusal, retired-ID refusal, real leadership transfer, new-leader writes, +self-demotion fencing, unrelated-group writes and automatic checkpoints. Joined +engine metadata reopen retains the exact final configuration and revision. No +fixture event pumping follows controller startup in these two lifecycle cases. +The separate held-sync case proves old root/configuration/applied state remain +selected until sync and that shutdown joins the accepted write. + +Initial private Raft accessor and missing-import compile failures are retained. +The lifecycle timeout reproduced delayed automatic joint-leave replication; the +owner now broadcasts the Raft-generated entry through normal Ready persistence. +The same lifecycle assertions pass. Diagnostic logs, including the timeout +backtrace, remain alongside the final committed-source gate. Preflight also caught +the changed storage-engine anchor; review pinned its new validation to the tested +revision. Its initial refusal and passing rerun are retained; no gate was weakened. + +**Reflection and next step:** fresh learner 4 is deliberately not started in this +slice. The feature-critical gap is a seed/bootstrap capability tied to committed +learner membership, its permanent node/disk claim and a complete verified root. +Implement actual new-host installation/catch-up/promotion, then replace a voter +after deleting its disk. Metadata reopen alone is not physical recovery or serving +permission. Prove one-voter-loss tolerance and minority refusal and complete all +five declared CG-05 cases. Remaining receiver crash/sync-error and controller +pressure/fairness cuts stay required; they do not replace fresh-member recovery +as the next feature priority. Then deliver CG-06 verified archive/reclamation, +CG-08 effects/caches and CG-07 public SDK/fleet cutover. No shared-group RustFS, +load, power-loss or zero-quorum recovery qualification is claimed by this gate. + +## CG-05 automatic checkpoints at controller apply boundaries + +Qualified source: `c27dfb416784804b30a38b1a90812c607ed3b00f`. All 230 selected Rust cases and 8 +gate commands pass, including all-target Clippy for Cellar/core/LTX, the five +native suites, format/layout, dependency isolation, docs, preflight and 37 validator +tests. `.cellar/cell-groups-evidence/CG-05-controller-checkpoints/checkpoint.json` +binds 308 source hashes, 5 copied executables, frozen plan inputs and logs. +Transfer this ignored evidence with its predecessors. CG-05 remains IN_PROGRESS. + +An explicit controller policy selects an applied cut before its successor starts +physical apply. One node slot spans blocking capture, reserved publication admission +and persistence. The controller holds the exact successor ticket, so a hot group's +current root cannot advance during capture. A private completion-lane operation +checks that ticket, engine and exact current root while preserving any later log +suffix. Only the matching persisted `CheckpointPublished` event releases successor +apply; admission/event ordering is handled either way. The existing public manual +publication checks remain unchanged. + +The policy uses the transport's configured node pool and owns initial checkpoint +endpoint registrations. Busy pool/admission retries back off until a later apply +boundary, letting apply proceed. Capture joins precede engine-claim release; +accepted publications remain runtime-owned through shutdown. Hard failures retain +tickets and quarantine the group; shared log uncertainty closes the node. Usage +reports pending checkpoints, actual publications and transient retries. + +Three new cases use production controllers and real TCP Raft traffic after startup, +with no fixture event pumping. Repeated interleaved writes to two groups compact +successive prefixes in both image formats. A held receiver saturates the leader's +checkpoint pool while both groups continue applying; releasing it permits later +checkpoint publication. Every voter retains the applied suffix and advances its +checkpoint; joined close releases all pool admission. Initial polling borrow-conflict +and duplicate-TCP fixture failures are retained; the same assertions passed after +repair. No assertion was weakened. + +**Reflection and next step:** this entry-count policy retains the idle final tail. +The critical next gap is scheduler/persistence pressure qualification: continuous +saturation fairness, exhausted apply/completion budgets, held sync and cancelled +close, checkpoint failure/quarantine capacity, and retained-root slot pressure. +Add deterministic coverage at those boundaries before time/byte-trigger expansion. +Then complete receiver crash/sync-error cuts, whole-group learner/joint membership +and lost-disk replacement, and run all five declared CG-05 acceptance cases. CG-06 +archive/reclamation follows; upload completion alone cannot authorize reclaim. +No new shared-group RustFS, load, power-loss or zero-quorum DR qualification is +claimed. Public cutover and legacy removal remain open. + +## CG-05 authenticated checkpoint transport + +Qualified source: `ceb7ed3149202660855d32df1ec5feed7775e068`. All 227 selected Rust cases pass across shared +execution/catalog/storage and the existing single-cell storage/snapshot suites. +All 8 gate commands pass, including all-target Clippy for Cellar/core/LTX, +formatting/layout, dependency isolation, docs, source preflight and 37 +plan-validator tests. The partial record at +`.cellar/cell-groups-evidence/CG-05-snapshot-transport/checkpoint.json` binds +306 source hashes, 5 copied executables, frozen plan inputs and command logs. +Transfer this ignored directory with its predecessors. CG-05 remains IN_PROGRESS. + +Dedicated TLS data streams now carry complete checkpoints between physical group +peers. Start and chunk purposes bind the exact history/incarnation, node/disk/process, +member, schema, snapshot envelope, ordered part identity and byte digest to the +TLS exporter and start tag. Bounded headers and <=64 KiB chunks precede allocation; +an authenticated End is mandatory. Extent identities resolve from the receiver's +verified manifest. The exact retained source cut streams on a blocking owner; +receiver verification and sealing run on another. Wire, physical workspace and +scratch reservations stay with their owners through cancellation and shutdown. + +Controller registrations retain the applier/route endpoint. Removing/replacing a +route or dropping its registration revokes stream admission. The independently +verified closure enters `step_checkpoint`; the wire reply is admission only. +The existing exact Ready/persistence completion still controls Raft's acknowledgement. +A node explicitly enables the shared checkpoint pool; automatic checkpoint +selection/publication is not yet wired into the controller. + +Snapshot status feedback now has a separate 256-slot/1-MiB node budget and reserved +queue space. It coalesces by known peer while retaining admission, then mutates +Raft progress after the current Ready settles. Transport ownership lasts through +feedback admission or shutdown, so ordinary queue contention or an I/O reply timeout +cannot silently discard required snapshot failure feedback. Immediate controller +snapshot refusals also retain feedback; ordinary unreachable hints remain advisory. + +Five new real-TLS cases cover actual RawNode-generated catch-up in both image +representations, suffix replay/cold source reopen, a dropped delivery waiter, and a +refused first transfer followed by Raft's own successful retry. The harness does +not replay that failed message. Independent wire probes refuse invalid exporter, +process, body digest, transfer binding, sequence, chunk size and premature End; +revocation during an admitted receive leaves the old root selected. Replacing the +old registration guard preserves its replacement. Disconnect releases receive +admission; a different group commits and applies a real cell write while that +admission is occupied. Materialized source cold recovery still uses the fixture's +generation-ID resolver; extent recovery and destination replay use production adapters. + +The initial visibility and fixture trait-import compile diagnostics are retained. +The first all-target Clippy run was deliberately interrupted when review identified +the snapshot-feedback admission gap; only the final committed-source gate counts. +No assertion was weakened. Existing native held-sync/owner tests remain regression +coverage, rather than an independent TLS receiver crash or power-loss qualification. + +The next gap recorded at this transport checkpoint was automatic controller-owned +checkpoint selection/publication: coalesce work, coordinate with advancing apply, and bound +capture/publication retries so hot groups can checkpoint without starving unrelated +ones. Then qualify source/receiver pressure, stalled physical I/O and hint saturation, +receiver process-kill/sync-error cuts, and retained-root slot exhaustion. Complete +whole-group learner/joint configuration and lost-disk replacement before the full +CG-05 gate. All five declared acceptance cases remain required. CG-06 archive/reclaim +and later public cutover/legacy removal remain open. No new shared-group RustFS, +load, performance-parity or zero-quorum DR qualification is claimed. + +## CG-05 local publication and native Raft snapshot catch-up + +Qualified source: `742b1e6d4e7155e16fb1b4eba0cce3b9056944ae`. All 222 selected Rust cases pass across shared +execution/catalog/storage and the existing single-cell storage/snapshot suites. +All 8 gate commands pass, including all-target Clippy for Cellar/core/LTX, +formatting/layout, dependency isolation, docs, source preflight and 37 +plan-validator tests. The partial record at +`.cellar/cell-groups-evidence/CG-05-local-checkpoint/checkpoint.json` binds +298 source hashes, 5 copied executables, frozen plan inputs and command logs. +Transfer this ignored directory with its predecessors. CG-05 remains IN_PROGRESS. + +An unstreamed source becomes a bounded local publication capability. The owner +requires its exact applied cut, engine and page owner, and runs publication on the +existing persistence lane. The distinct local engine operation preserves hard +state, applied state and every later log entry while compacting the represented +prefix. Mixed, foreign and stale preparations refuse. `CheckpointPublished` +arrives only after the matching engine completion; local compaction does not +advance the controller's applied-state counter. Admission and physical pins remain +owned through queued sync and joined runtime shutdown. + +`capture_snapshot` selects the retained complete closure and original metadata +named by the exact snapshot from Raft Storage. Later current-root updates cannot +substitute a newer cut. A concurrent replacement may refuse capture; a captured +source owns the old immutable closure until streaming ends. + +Three new cases cover both image formats, a real uncommitted suffix surviving +held publication and later quorum commit, and a partitioned existing voter restored +by an actual RawNode-generated snapshot followed by suffix replay. The leader's +engine is then closed and reopened: the retained snapshot and exact tail must +match, both physical roots are recovered, and the older checkpoint streams again. +Materialized source recovery uses the fixture's explicit generation-ID resolver; +extent recovery and destination suffix replay use production adapters. The initial +materialized cold-recovery failure from using the wrong resolver is retained with +the compile diagnostics. No assertion was weakened. Joined runtime shutdown during +accepted local publication is covered in both representations. + +The gap recorded at this local-publication checkpoint was authenticated RPC and its controller-owned +source/publication lifecycle. These cases use native streaming callbacks; they do +not qualify network framing/authentication, transfer cancellation/retries, failed +snapshot reporting, automated checkpoint policy, learners/joint membership or disk +replacement. Receiver process-kill/sync-error/cancelled-waiter cuts and root-slot +pressure also remain open. All five full CG-05 acceptance cases remain required. +There is no new shared-group RustFS, load or zero-quorum DR qualification. CG-06 +archival/reclamation and the later public cutover and legacy removal remain open. + +## CG-05 Raft snapshot acceptance and persistence + +Qualified source: `096cfb132397d9588ba9138b1cf474210fc062f3`. All 146 selected Rust cases pass across shared +execution/catalog/storage. All 8 gate commands pass, including all-target +Clippy for Cellar/core/LTX, formatting/layout, dependency isolation, docs, +source preflight and 37 plan-validator tests. The partial record at +`.cellar/cell-groups-evidence/CG-05-snapshot-owner/checkpoint.json` binds +296 source hashes, 3 copied executables, frozen plan inputs and command logs. +Transfer this ignored directory with its predecessors. CG-05 remains IN_PROGRESS. + +The explicit owner/runtime `step_checkpoint` API admits only a complete verified +receiver closure matching the snapshot envelope, route, selected base, engine, +page-store owner and permanent replica membership. Bounds and overflowing cuts +are checked before RawNode mutation. Ordinary peer ingress still rejects snapshot +messages. Admission reports whether Raft selected the snapshot or retained its +matching log; neither result is a persistence receipt or serving authority. + +A selected snapshot binds to the exact Ready and existing persistence ticket, +exclusive owner lease and shared engine fence. The old applied view remains until +the matching completion is consumed. Only then does the owner advance to the +verified cut and emit the acknowledgement and `CheckpointInstalled` event. +Controller publication observes that event. The prepared receiver and its budget +remain owned through queued sync and joined runtime shutdown. If Raft already has +the matching log, it persists the commit advance and replays entries without +selecting the received checkpoint. + +Five new native cases cover a partitioned existing voter, both image formats, +invalid envelopes and overflowing cuts, held sync with no early acknowledgement, +subsequent real quorum writes, matching-log replay and shutdown during accepted +snapshot persistence. The fixture builds the snapshot envelope from captured +source state and uses the native streaming callback. It does not exercise a +source RawNode-generated snapshot or authenticated snapshot RPC. + +The gap recorded at this owner checkpoint was source-side local publication that preserves +the unapplied log suffix, followed by bounded snapshot transport and actual Raft +peer catch-up. The receiver-install write must not be reused as a local leader +checkpoint: it replaces the received state and truncates prior entries. Then +qualify receiver process-kill/sync-error/cancellation and root-slot boundaries, +whole-group learners, joint membership and lost-disk recovery. All five full CG-05 +acceptance cases remain required. No new shared-group RustFS, load or zero-quorum +DR qualification is claimed. CG-06 archival/reclamation and the later public +cutover and legacy-removal tasks remain open. + +## CG-05 verified receiver and atomic installation + +Qualified source: `cb164e025e129330f576450d214cef0e8fe92aa1`. All 226 selected Rust cases pass across shared +execution/catalog/storage, existing single-cell storage/snapshot suites and the +LTX library. All 9 gate commands pass, including all-target Clippy for +Cellar, its pure core and LTX, formatting/layout, dependency isolation, docs, +source preflight and all 37 plan-validator tests. The partial record at +`.cellar/cell-groups-evidence/CG-05-checkpoint-receiver/checkpoint.json` binds +295 source hashes, 6 copied executables, frozen plan inputs and command logs. +Transfer this ignored directory with its predecessors. CG-05 remains IN_PROGRESS. + +A receiver now stages ordered bounded parts against the destination engine's +selected base. It checks route/history/cut/membership, every manifest page and +independently sealed image, then verifies and syncs the complete tree. Flat images +use one admitted scratch spool; extent objects use bounded buffers. Repeated flat +transfers reread and sync the existing destination even when its seal is cached. +A regression reproduced acceptance of a corrupt cached destination before this +fix; that failure and the interrupted candidate gate are retained separately. + +The prepared checkpoint owns the complete destination closure and receive +reservation. The engine's group snapshot variant atomically selects the snapshot, +configuration, membership and applied root behind the shared durability fence. +Mixed contracts, stale/conflicting bases and incomplete transfers refuse. A +separate retained checkpoint root survives later ordinary apply; recovery must +verify both it and the current applied tree before startup or reclamation. + +Three new cases use a lagging existing voter of the real embedded three-voter +fixture. Both image forms and canonical empty enrollment survive byte-for-byte +transfer, a held engine sync, subsequent quorum writes through the production +apply pool, joined shutdown of all prior owners, and cold reopen of the receiver. Before +publication, the selected root stays unchanged and owner startup refuses. The +receiver reservation remains charged after the caller drops its prepared handle +while engine sync is held. A checkpoint-only page is proved absent from the +current tree, then deleted: recovery refuses despite a complete current tree. +Restoring it permits storage startup without renewing quorum authority. Truncated, +noncontiguous, corrupt, foreign-history, oversized and scratch-limited transfers +leave the prior marker selected; existing cached destination corruption also +refuses. Extent receiving does not require a materialized-image scratch spool. + +The tests invoke the verified engine installation boundary directly. They do not +qualify Raft Ready snapshot acceptance or snapshot RPC, fresh learners, joint +configuration, disk replacement, receiver process-kill/sync-error/cancelled-waiter +cuts, checkpoint root-slot exhaustion, shared-group RustFS, load or zero-quorum DR. +No full CG-05 acceptance case has been replaced with these narrower cases. + +Next work recorded at this receiver checkpoint was to bind its result to the exact Raft snapshot +effect, integrate bounded snapshot transport and completion, and qualify peer +catch-up plus crash/cancellation/root-slot boundaries. Then deliver whole-group +learner/joint membership and lost-disk recovery and run all five declared CG-05 +acceptance cases. CG-06 archival reclamation follows that task; public host/SDK +cutover and the later topology, hibernation, DR and legacy-removal gates remain open. + +## CG-05 complete-root capture and bounded streaming + +Qualified source: `b15da415c9c9c0663274d0aedd520feb37d90d2d`. All 138 selected Rust cases pass across +shared execution, catalog and manifest storage, including the four new checkpoint +source cases. Eight gate commands pass: all-target Clippy, those regression suites, +formatting, layout, dependency isolation, docs, source preflight and 37 plan-validator +tests. `.cellar/cell-groups-evidence/CG-05-checkpoint-source/checkpoint.json` binds +290 source hashes, three copied executables and retained command/diagnostic logs. +The plan inputs are frozen alongside the evidence. Transfer this ignored local +directory with its predecessor evidence; this is a partial CG-05 gate. + +The first source-side snapshot slice adds a shared checkpoint pool and a +non-cloneable physical source. Capture checks the exact selected group root, +engine owner, committed/applied cut and membership revision. It copies membership +and configuration under that same cache lock, and retains the complete immutable +closure. The source streams canonical manifest pages, including empty enrollment, +plus materialized image bytes or extent manifests and verified objects. It does +not aggregate group images or consult a later mutable manifest during transfer. + +Admission accounts for source slots, retained metadata, bounded stream workspace, +cell count and total transfer size. Borrowed chunks are at most 64 KiB, one extent +buffer is at most 4 MiB, and the source keeps its charge until its physical worker +releases it. Missing/corrupt data, a closed engine or a sink error fails the stream; +failed sources cannot be reused. Streaming grants no receiver or voting authority. + +The CG-05 test scope now includes the existing three-voter execution fixture and +`support/cell_group_checkpoint.rs` so this slice exercises real Raft commits, +SQLite/LTX, both image stores and engine-selected roots without duplicating that +fixture. The five full snapshot acceptance cases remain unchanged and outstanding. +The shared fixture additionally handles canonical empty enrollment; existing +execution regression checks cover that helper change. + +Gaps recorded at this source checkpoint: build the receiver's bounded staged closure, +verify/sync every dependency, and publish snapshot metadata plus the group root +atomically through the engine. Then connect whole-group transport, learner +catch-up, joint membership and lost-disk replacement. The current header denotes +active groups only; topology/park state must become part of the durable root before +those operations are enabled. CG-05 remains IN_PROGRESS. No RustFS/load, receiver +install, membership-change, independent-process or power-loss qualification is +claimed by this capture slice. + +## CG-04 scoped execution gate complete + +CG-04 is DONE at `69e41e865c267dff1dec5bce6f2b3d0966be09db`. All 83 declared acceptance cases pass +individually, with the required test inventory, all-target check, formatting, +layout and all-target Clippy commands. The evidence record +`.cellar/cell-groups-evidence/CG-04-qualified/checkpoint.json` validates through the +plan runner and retains 98 distinct command records, 288 source hashes +and 4 copied executables, including the supporting 238-case regression gate. +Transfer it together with the local-provisioning and predecessor evidence. + +The completed scope is native shared-group execution: one Raft owner per group +replica, independent cell SQLite histories, bounded node services, durable apply, +cell receipts and reads, authenticated dispatch, response-independent ownership, +group failure isolation and explicit local provisioning/recovery. All acceptance +cases remain intact. Completion does not claim production SDK/fleet cutover or +independent-process, power-loss, shared RustFS or performance qualification. + +The critical next task is CG-05: complete-group snapshots, atomic installation, +learner catch-up and membership recovery. It unlocks CG-06 archive/reclamation and +then effect/fleet integration. Automatic production request/voter orchestration +and public SDK binding belong to CG-07; final legacy removal belongs to CG-16. +Keep those dependencies explicit in subsequent execution. + +## CG-04 bounded local provisioning + +Qualified source: `69e41e865c267dff1dec5bce6f2b3d0966be09db`. + +`CellGroupController::configure_provisioning` installs explicit immutable node +filesystem, image-store, manifest, peer and preparation limits. `provision` +accepts an already selected range assignment; it materializes the immutable birth, +checks configured initial members, acquires permanent member/engine claims, +recovers the complete selected root on the owning blocking execution domain and +attaches through the existing runtime and exclusive transport registration. It +neither assigns ranges nor creates a Raft group for an individual cell. Managed +fleet bootstrap still refuses pending CG-07 authorization. + +Exact directory-owner/group requests share one preparation. Pending and active +groups share the controller capacity; recovery items, bytes and parallelism have +explicit bounds. Response waiters have separate admission, including queued and +unconsumed replies, so preparation does not recursively consume command permits. +Active stores remain bounded by the configured per-group limits and maximum +roster size; image stores keep their independent quotas. Settings reject invalid +physical identities, duplicate endpoints and unbounded paths before I/O. + +Failure before recovery frees its preparation and group reservations. After +recovery, an attachment refusal retains the exact page/root/applier owners and +their preparation budget. A retry revalidates catalog/bootstrap and reuses these +owners rather than reopening an engine-pinned directory. The applier enters the +event roster before attachment can emit events. Registration identity/capacity +refusals preserve healthy groups; closed services or uncertain shared-engine +failures remain node-wide. Shutdown joins admitted catalog/filesystem/attachment +work before transport, SQL and runtime shutdown, even when waiters disconnect. +Success means locally attached; serving still requires fresh current-term quorum. + +Five native cases cover both image modes with real engines, filesystem operations +and TLS. They exercise same-group coalescing, dropped waiters, bounds and foreign +bindings, single-command-slot operation, actual enrollment/mutation/exact retry, +pre-recovery catalog failure and post-recovery attachment refusal with page-owner +reuse. A blocked physical directory sync retains engine ownership across cancelled +close. Missing initial peers refuse before genesis; a wrong configured local +endpoint retains preparation while the existing group still executes a command. +All voter engines and manifest stores reopen into empty controllers; removing a +selected page refuses without resetting the marker, restoration permits recovery, +and fresh quorum returns the original receipt and SQLite value. Image-store +handles stay in the process, so this is not independent-process/power-loss evidence. + +All 238 selected Rust cases and twelve gate commands pass, including all-target +Clippy, formatting, layout, dependency isolation, docs, source preflight and 37 +validator cases. `.cellar/cell-groups-evidence/CG-04-local-provisioning-final/checkpoint.json` +binds 288 source inputs and 4 copied executables. Diagnostic failures and +reruns are retained separately; transfer this ignored directory with predecessors. + +The provisioning checkpoint below the full gate is scoped to explicit local +startup. CG-07 integrates automatic request-driven provisioning and assigned-voter +startup, preserving original operation identity, +deadline and captured inputs while using this bounded owner from local and remote +dispatch. Read/receipt requests must never implicitly enroll missing cells. +Configuration/membership lifecycle, managed fleet authorization and public +host/SDK integration remain unfinished. Later CG tasks still own snapshots, +archive/reclamation, effects, topology, hibernation, fenced recovery, load and +legacy removal. Shared-group RustFS and load remain unqualified. + +## CG-04 retained group quarantine + +Qualified source: `41121a7d08f283db6a83982ebfcba2feced036ca`. + +A group-local apply/runtime failure now quarantines the group instead of closing +healthy groups on the node. The controller first refuses application admission, +then revokes runtime execution through reserved control capacity that bypasses a +waiting apply. Captured leader authority becomes invalid. Pending apply tickets, +unresolved command/enrollment producers and the group slot remain retained until +joined shutdown. Quarantine cannot be cleared by re-registration; a late physical +persistence completion cannot restore the revoked execution fence. + +`group_failure` reports the original cause; usage distinguishes active, pending +and quarantined groups. Shared log-engine I/O/corruption/closure and lost shared +SQL worker replies still close the node. Unknown group events and controller +ownership violations also retain the fail-closed boundary. Quarantine does not +promise capacity beyond the configured node/group budgets; retained work remains +charged rather than discarded to make another request fit. + +Both image modes pass real manifest directory-sync faults while an ambiguous +producer remains owned. A dynamically registered healthy group then completes +controller enrollment, command, exact retry and a linearizable SQLite query on +the same engine, single SQL worker and TLS peers. The failed group's selected root +stays unchanged, captured authority and retries refuse, callback counts exclude +replay, and a third registration cannot reuse its occupied capacity. Joined close +releases the producer. A separate real Raft-engine sync fault still stops both +groups and refuses replacement runtime admission. Existing physical cancellation +and lost-worker-reply tests remain in the gate. + +All 233 selected Rust cases and twelve gate commands pass, including all-target +Clippy, formatting, layout, dependency isolation, docs, source preflight and 37 +validator cases. `.cellar/cell-groups-evidence/CG-04-group-quarantine-final/checkpoint.json` +binds 283 source inputs and 4 copied executables. Initial fixture compilation +errors and diagnostic reruns are retained with this ignored local evidence; +transfer it with the predecessor checkpoints. + +CG-04 remains IN_PROGRESS. The next critical gap is bounded request-driven group +provisioning: catalog selection/birth and physical root preparation must transfer +into the existing registration owner, coalesced by physical group rather than +cell. Configuration lifecycle and public host/SDK integration remain unfinished. +Later CG tasks still own snapshots, archive/reclamation, effects, topology, +hibernation, fenced recovery, load qualification and removal of the legacy host. +This checkpoint is not independent-process/power-loss, RustFS or load qualification. + +## CG-04 dynamic controller registration + +Qualified source: `4326421a94c4f9a257bf6ceb8e0f7e518eacf3de`. + +`CellGroupController::register` now adds a group without replacing the controller +or creating per-cell workers. Controllers may start empty. Pending and active +groups share a capacity bound; queued attachment and unconsumed replies retain +the existing item/byte budget. Command, enrollment and receipt dispatch consult +the live roster under short locks rather than copying it for each invocation. + +Admission binds the bootstrap permit, selected root, applier and physical route +to the same engine/history/schema. The applier must own the exact manifest page +store that verified the root. Its event registration precedes polling attachment, +so early consensus and apply events remain owned. Runtime attachment revalidates +the catalog permit before active membership is published. Registration alone +never grants application authority; current-term quorum remains required. + +The transport registration refuses adoption of an existing route. Its cleanup +compares exact route ownership so it cannot erase a replacement. A disconnected +waiter cannot abandon attachment or release its group slot. Definite failures +release group capacity before responding. Shutdown joins entered catalog/runtime +attachment before closing physical dependencies. Unexpected competing runtime +owners or runtime-wide closure keep the existing conservative node shutdown. + +Four new native cases run both image modes over real engines and TLS. Held +catalog reads leave an already active group serving. Duplicate and pending/active +capacity checks, foreign engines/roots/page owners, failed catalog retry and +cancelled close exercise ownership boundaries. Recovery joins all controllers, +closes and reopens all voter engines under a new process generation, reopens page +stores and registers selected roots into empty controllers. Fresh quorum returns +the prior exact receipt without another SQL mutation. Image-store handles remain +in the same process; this does not qualify independent-process or power loss. +The first diagnostic run exposed an obsolete catalog stall key and an attempted +page-store reopen while the engines correctly retained page pins. The corrected +fixtures preserve those safeguards and are retained with the passing evidence. + +All 231 selected Rust cases and twelve commands pass, including all-target Clippy, +formatting, layout, dependency isolation, docs, source preflight and 37 validator +cases. `.cellar/cell-groups-evidence/CG-04-dynamic-registration/checkpoint.json` +binds 281 source inputs and 4 copied executables. Transfer this ignored +local directory with prior checkpoints; it is not included in the PR. + +CG-04 remains IN_PROGRESS. The critical next gap is per-group failure isolation: +a group-local apply/command failure currently closes the node controller. +Quarantine that group, revoke serving authority and retain its physical owners +through cleanup, while proving that a healthy group continues. Shared log-engine +sync failure must still quarantine every group on the affected engine. +Request-driven provisioning/lazy birth must still connect catalog selection and +bounded local preparation to this registration API. Configuration lifecycle and public SDK discovery/cutover remain +open, followed by the CG-05–16 dependencies. Shared-group RustFS recovery and load +remain unqualified; no throughput or latency gain is claimed. + +## CG-04 standalone receipt inspection + +Qualified source: `fbc15053f8dcfd3b46078abad3cbbae7b68da8d6`. + +Local and remote receipt inspection now select an immutable committed cell image +through a correlated quorum read. The registered command contract, payload digest +and authenticated identity are validated before lookup. Inspection neither stages +a producer nor invokes the command or initializer; it does not install receipt +schema or run receipt maintenance. Initial images may lack receipt tables; +partial schema or schema loss after mutation refuses as corruption. + +The existing bounded RPC exchange carries a distinct authenticated Receipt purpose. +Explicit namespace/process command permission and the exact controller/SQL owner +are required; read permission alone cannot inspect command results. Fresh receiver +catalog binding checks group, schema, epoch and birth. Success binds the original +command identity and signed cell observation, distinguishing a retained empty +result from absence. Absence remains ambiguous even after a quorum read: the +original command may still arrive or commit. Lookup cannot release its producer. +Replicated expiry, conflicts and sequence gaps remain typed refusals. + +Three native cases run both image modes. Real TLS ingress exercises permission, +identity, retry/deadline, consumed-attempt, reply-admission and gateway-drop checks. +A prepared writer remains owned while lookup sees the old image, while the shared +SQL reservation budget is saturated, and after its receipt becomes committed. +Cancellation of a queued remote lookup preserves an entered controller writer. +A closed controller uses a weak SQL handle, allowing a replacement pool after +joined shutdown even when the closed controller handle remains retained. Three +wire cases and a real SQLite schema-loss case qualify serialization and inspection +boundaries. The initial fixture failures and their corrected bounded admission +setup are retained as diagnostic logs; production limits were not loosened. + +All 227 selected Rust cases and twelve gate commands pass, including all-target +Clippy, formatting, layout, native dependency isolation, docs, source preflight +and 37 plan-validator cases. The record at +`.cellar/cell-groups-evidence/CG-04-receipt-rpc/checkpoint.json` binds 279 +source inputs and 4 copied executables. Transfer the entire directory with +prior checkpoints; local ignored evidence is not included in the PR. + +CG-04 remains IN_PROGRESS. The critical next gap is dynamic group lifecycle: +replace the fixed controller roster with bounded registration and lazy birth, +while preserving exact consensus, transport, apply and SQL ownership. Qualify +attach/close races, cancellation and recovery before publishing serving authority. +Per-group failure isolation is the next companion gap: a failed group must stop +serving while unrelated groups continue, and retained physical work must join. +Public SDK/discovery cutover, configuration handling, complete-group snapshots, +archive/reclamation and shared RustFS/load qualification remain open. No latency, +throughput, independent-process failure or production-cutover claim is made. + +## CG-04 remote enrollment through the shared write owner + +Qualified source: `0fe665d0267f1b831b172aad5e8d435f5bf0cd39`. + +`CellEnrollmentInvocation` now consumes catalog attempts over the same bounded +write exchange as commands. An authenticated mandatory purpose distinguishes +Query, Command and Enrollment; marker 1 has no reader for the old boolean purpose. +The existing exact controller registration and namespace/process write grants +admit compiled initializers. Read permission, discovery and voter membership +cannot authorize creation. The router still allocates no local group replica. + +Creation identity, captured clock/random inputs and preview identity are frozen +before discovery and retained through retries. Original attempt/deadline budgets +remain in force. The receiver checks fresh catalog group/schema/epoch/birth and +actual leader authority before handing initialization to the controller. Followers +return NotLeader; a leader awaiting apply still returns NotReady internally. +Disconnect, permission revocation and response expiry cannot discard accepted +controller ownership. A possibly transmitted invocation cannot be reused as fresh +local enrollment. + +Success returns the exact committed creation record with bounded reply admission. +The original enrollment image survives later mutations; duplicate retries return +that record without initialization or reverting current cell state. The record +is not a current-version token. Creation/incarnation conflicts remain typed. +Command and enrollment requests share physical TLS exchange and reservation code, +so there is no second per-peer write connection or independent budget. + +Four new native cases run four TLS endpoints in one OS process in both image +modes. They cover missing/read-only write permission, voter/follower refusal, +exact creation receipts, duplicate creation after a later remote command, +creation/incarnation conflicts, consumed attempts, original deadlines, reply +reservations, gateway drop and refusal to treat routed work as fresh local work. +Held initialization tests disconnect and revocation. A held committed manifest +publication tests response expiry followed by exact creation recovery. Three +wire cases verify captured inputs/preview, bounded payloads, authenticated purpose +separation and creation/cell/schema/image reply binding. + +All 216 selected Rust cases and eleven gate commands pass, including all-target +Clippy, formatting, layout, native dependency isolation, docs, source preflight +and 37 plan-validator cases. The record at +`.cellar/cell-groups-evidence/CG-04-enrollment-rpc/checkpoint.json` binds 276 +source inputs and 4 copied executables. Transfer this entire directory with +prior checkpoints. Targeted runs remain diagnostic evidence. + +CG-04 remains IN_PROGRESS. The next request-path gap is standalone safe receipt +lookup after an ambiguous response. Public SDK/discovery, dynamic roster and +lazy group birth, per-group failure isolation and configuration lifecycle remain. +The shared lane still serializes exchanges per peer. Complete-group snapshots, +archive/reclamation, shared RustFS recovery, independent-process faults and load +remain unqualified; no throughput or latency improvement is claimed. + +## CG-04 write-authorized remote commands + +Qualified source: `5fd4443796a6e990fbe70c16acb7fa5578b7e763`. + +`CellCommandInvocation::execute_remote` sends one consumed catalog attempt over +the bounded application RPC lane. The authenticated route has a mandatory command +purpose; command payloads/results have distinct marker-1 domains. Read grants and +voter membership never create write permission. `NamespaceCommandOrigin` grants +one namespace to an exact physical process, with independently bounded revision +tombstones. Endpoint checks, rooted process refresh, fleet revocation and peer-pool +retention include command grants. Discovery still allocates no local replica. + +A host-owned `CellCommandGateway` binds the exact controller/runtime owner. The +request carries the caller's captured time/random inputs, immutable command +identity, operation contract, retry horizon and remaining deadline. The receiver +controller checks the current catalog group/schema/epoch/birth and actual leader +authority before SQL. It retains admitted work independently from the connection; +disconnect, timeout or write-grant revocation cannot undo an admitted mutation or +release its physical producer. Final registration drop refuses new commands. + +Success returns the exact committed command identity/result and authenticated +logical cell observation/token. Reply admission remains charged until consumption. +Original deadlines/attempt budgets survive retries. Conflict, expiry, sequence-gap +and not-leader refusals remain typed. Once remote transmission may occur, errors +remain conservatively ambiguous; neither admission nor response loss proves +non-commit. Committed duplicate retries return the stored result without another +application callback. + +Three new native cases run both image modes over four TLS endpoints in one OS +process. A nonmember router with no group storage tests missing/read-only, +wrong-process and foreign-namespace write permission, voter non-authority, follower +refusal, exact controller binding, committed duplicate/conflict handling, reply +reservations and registration drop. Held physical SQL tests disconnect and grant +revocation: the controller keeps ownership, then resolves the command and returns +its committed receipt on retry without a second callback. Three wire cases cover +captured inputs, payload/purpose separation, exact command/history/token binding +and typed failures. A transport case checks bounded tombstones, stale revision +refusal, independent read grants and pool cleanup. + +All 209 selected Rust cases and eleven gate commands pass, including all-target +Clippy, formatting, layout, native dependency isolation, docs, source preflight and +37 plan-validator cases. The record at +`.cellar/cell-groups-evidence/CG-04-command-rpc/checkpoint.json` binds 273 source +inputs and 4 copied executables. Transfer that entire directory with prior +checkpoints. Targeted compile/test logs are retained as diagnostic evidence. +The initial follower assertion ran without a follower gateway and failed before +leadership admission. After adding the gateway, diagnostics exposed follower +authority requests returning NotReady and becoming generic Unavailable remotely. +The owner now reports NotLeader for that role, with leader readiness and stale +authority checks preserved. Direct and remote tests assert the precise refusal. + +CG-04 remains IN_PROGRESS. Remote enrollment and standalone safe receipt lookup +are the next request-path gaps; public-host automatic discovery/SDK binding, lazy +birth/dynamic roster, per-group isolation and configuration lifecycle remain. +The RPC lane serializes exchanges per peer; no load or latency gain is claimed. +Destination catalog reads remain until replicated transfer sealing and serving +invalidation are integrated. Complete-group snapshots/archive, shared RustFS +recovery, independent-process failure, power loss and load remain unqualified. + +## CG-04 bounded invocation route cache + +Qualified source: `06652e36869144624e0ca4d860782ef512018791`. + +`CellGroupDispatcher` clones now share bounded namespace range hints across query, +command and enrollment invocations. One cached range serves many cells; no +per-cell cache, worker or Raft owner is allocated by routing. Explicit entry and +reserved-byte limits bound retention, with LRU eviction. The conservative 8 KiB +reservation per slot remains charged even while the slot is empty. + +Hints come only from a verified directory lookup. `refresh_routes` observes the +catalog in the owning execution domain; a newer head invalidates older hints. +An older in-flight lookup cannot refill the cache after that observation, and a +same-revision/different-digest observation refuses. Failed refresh retains only +historical hints. Redirect confirmation, original deadlines, attempt budgets, +exact cell identities and submitted-command ambiguity remain enforced. + +Three catalog cases disable page caching and measure actual object-store calls. +After warming one range, 64 other cells and a command route with zero metadata +reads. Byte-limited LRU eviction, successor-head invalidation and a deliberately +held stale lookup pass. A fourth native case runs both physical image modes over +real TLS quorum owners: after removing the catalog head, local reads, a command +and enrollment succeed using the hint plus actual runtime authority. Remote +admission refuses because the destination still requires a fresh catalog check. +The fixture restores metadata and joins all physical owners. + +All 202 selected Rust cases and eleven gate commands pass, including all-target +Clippy, formatting, layout, native dependency isolation, docs, source preflight and +37 plan-validator cases. The source-bound record at +`.cellar/cell-groups-evidence/CG-04-dispatch-route-cache-final/checkpoint.json` binds +265 source inputs and 4 copied executables. Transfer the complete directory +with previous checkpoints, including the initial `CG-04-dispatch-route-cache` +Clippy failure. That fixture called Tokio timeout directly; the successor uses +the execution-domain timeout without changing assertions or production behavior. +Targeted diagnostic logs are retained separately. + +CG-04 remains IN_PROGRESS. This removes repeat caller-side mapping reads; it does +not remove destination catalog admission or establish end-to-end latency gains. +Destination caching needs replicated transfer sealing and admission invalidation, +not a timeout that treats a stale hint as permission. The next critical delivery +is write-authorized command/enrollment RPC with response-independent producer +ownership and safe receipt lookup. Public-host discovery/SDK binding, lazy birth, +dynamic roster, per-group failure isolation and configuration lifecycle remain. +Complete-group snapshots/archive, shared RustFS recovery, independent-process +failures, power loss and load remain unqualified. + +## CG-04 nonmember read discovery + +Qualified source: `c25f0d97d5c98c6111ea2165b38a11a08613c0ca`. + +`RaftTransport::install_cell_query_group` installs explicit trusted discovery for +physical groups absent from the router. It creates no local group storage, +RawNode, voting identity or mutation permission. The shared query path uses that +view or an attached group's view; destination namespace grants, live membership, +catalog binding and actual read authority are still required. Removing discovery +invalidates captured routes and retires unused peer pools; it does not revoke the +destination's independently managed read grant. + +Read-only routes share the node's route cap, endpoint consistency checks and peer +pools with consensus and other queries. They participate in process-view refresh +and router-route resource accounting. Route revisions cannot move backwards or +change schema, and existing member identities cannot silently change node/disk. +The sender revalidates captured route identity before sending and after receiving; +a late authenticated reply under a replaced route cannot become a success. + +Three new native cases run both materialized and extent images over four real TLS +endpoints in one OS process. The fourth node has no stored cell groups or attached +Raft owner before or after reads. Discovery alone, a foreign-namespace grant and a +wrong-process grant refuse before SQL. Explicit permission permits leader strong +reads and follower stale/AtLeast reads. The current strong-read contract refuses +a follower; the test verifies that retrying the leader preserves the original +invocation deadline and attempt budget. Discovery shares three physical pools +across multiple routes and enforces its configured capacity. + +Revocation while a callback is held ends the remote read waiter and drains +transport reservations while physical SQL remains owned. A stale grant revision +cannot reopen admission; a newer explicit grant permits a fresh read after cleanup. +Replacing discovery while SQL is held refuses the old reply and permits a fresh +attempt. Neither mechanism grants consensus authority or creates a router replica. + +All 198 selected Rust cases and eleven gate commands pass, including all-target +Clippy, formatting, layout, native dependency isolation, docs, source preflight and +37 plan-validator cases. The source-bound record at +`.cellar/cell-groups-evidence/CG-04-nonmember-queries/checkpoint.json` contains 262 +source inputs and 4 copied executables. Transfer that complete directory with +prior checkpoints. Diagnostic targeted runs are retained separately: the first +used a local-only default Raft message before the intended absent-group assertion; +the fixture now sends a valid heartbeat. The next assumed follower strong reads; +the corrected fixture asserts the refusal and preserved-budget leader retry. + +CG-04 remains IN_PROGRESS. Discovery is explicitly installed by a trusted caller; +public-host automatic discovery and SDK binding are still missing. The destination +still resolves the catalog per RPC: bounded, correctly invalidated routing state +is the next cost/latency opportunity. Write-authorized command/enrollment RPC and +safe receipt lookup, lazy birth/dynamic roster, per-group failure isolation and +configuration/cache lifecycle remain. Complete-group snapshots, shared RustFS +recovery, independent-process failures, power loss and load remain unqualified. + +## CG-04 shared query RPC + +Qualified source: `da79a69604d58be1c7d9663ebb69c3f6ded173f6`. + +Catalog-routed `CellQueryInvocation::execute_remote` now uses the existing +node-owned TLS query lane. The authenticated marker-1 route separates physical +`GroupHistory` from logical `CellHistory`; its payload also binds the catalog +route epoch and birth. There is no reader for the former untagged query route. +The temporary old-host query arm remains explicit until CG-16. Request/reply +channel binding, sequence checks, node/disk/process/member identity and bounded +framing remain shared with the existing transport. + +`RaftTransport::install_cell_query_gateway` registers a namespace dispatcher on +the exact runtime/SQL engine and execution domain. The host owns the registration; +the transport stores a weak reference. A separate admission owner refuses new +reads after its final host handle is dropped, even while an accepted query retains +the gateway data. Destination catalog and coherent runtime +read authority are revalidated before registered SQL runs. Replies carry the +actual selected cell observation and signed cell token. Foreign histories, +incarnations and signing keys cannot produce a successful result. The caller +also enforces its requested AtLeast minimum. + +Both query paths use the same read-only origin grant/revocation guard, peer pools, +connection exchange and byte/item banks. These permissions cannot authorize a +mutation or consensus message. A fully authenticated query refusal keeps the +connection reusable; transport/protocol failures discard it. The single peer +sender assigns wire sequence numbers after dequeueing, so concurrent ingress +cannot invert them and cause false replay refusals. Each attempt is +consumed once; retries and catalog-confirmed redirects retain the original caller +deadline and total attempt budget. Returned replies keep transport admission until +dropped. Disconnect and expiry cancel read waiters without releasing physically +entered SQL or image ownership; controller shutdown still joins those workers. + +Three new native cases each run both materialized and extent images on three +real TLS endpoints in one process. They cover two cells sharing a group, strong/ +stale/AtLeast reads, exact cell observations, wrong tokens/keys/incarnations, +foreign SQL ownership, duplicate/dropped gateway registrations, concurrent-cell +requests, retained responses, +retry budgets and cancellation/deadline shutdown. A fresh linearizable barrier +still completes while a query callback is held. A catalog replacement after route +capture refuses at the old destination before callback execution; a confirmed +redirect cannot invent its new physical owner. That fixture changes metadata only +and does not qualify live movement. Three new wire cases cover bounded parsing, +physical/logical identity and channel binding, token authentication, typed failures +and payload-domain separation. + +All 195 selected Rust cases and all eleven gate commands pass. This includes the +wire tests, full transport-module regressions, shared execution/catalog suites and +existing host suite, plus all-target Clippy for Cellar/cellar-logic/cellar-ltx with +warnings denied, formatting, layout, native dependency isolation, docs, source +preflight and 37 validator cases. The record at +`.cellar/cell-groups-evidence/CG-04-remote-queries-final/checkpoint.json` binds 260 +source inputs and 4 copied executables. Transfer the complete evidence directory +with prior checkpoints. The first diagnostic compile caught a cleanup hook in the +wrong owner; it was fixed before tests. The first committed gate passed Clippy, +eight wire cases and 63 transport cases, then was stopped during native testing +after review found the registration lifetime gap. The separate admission owner +and held-callback assertions fix that gap; this successor repeats the full gate. +A second candidate was stopped during Clippy to move sequence assignment into +send order and add concurrent-cell coverage. All diagnostic and interrupted gates +remain separate from this final qualification. + +CG-04 remains IN_PROGRESS. This checkpoint routes between attached group peers; +nonmember shared discovery and its real-TLS revocation qualification remain open. +The destination currently resolves the catalog per RPC, so bounded route-cache +integration is a critical cost/latency opportunity. Public SDK binding and explicit +write-authorized command/enrollment RPC, including safe follower receipt lookup, +are still missing. Lazy birth/dynamic roster, per-group failure isolation and +configuration/cache lifecycle also remain. Shared-group RustFS recovery, separate +process failure, power loss and load are unqualified; the public host has not cut +over. Continue with these concrete integration gaps before claiming CG-04 DONE. + +## CG-04 controller-owned enrollment + +Qualified source: `f77d0d939347ead7129b39e72f76fcff11b99635`. + +`RaftClass::initializer` registers one compiled initialization callback with an +explicit contract digest. That digest participates in the voting schema identity; +changing or omitting it cannot claim the same namespace schema. Declaration order +with commands does not change the resulting identity. Existing old-host class +validation refuses this registration rather than silently ignoring the callback. +Shared initialization receives the logical cell history and captured host inputs +through its private transaction on the existing bounded SQL workers. + +`CellGroupDispatcher::enroll` freezes creation identity, schema and host inputs +before catalog discovery. `CellGroupController::enroll` takes fresh local +invocations into the same queue and byte/item budget used by commands. Both use +one response-deadline implementation and one physical shutdown owner. A dropped +or expired response cannot discard entered initialization or ambiguous submission. +Unconsumed creation replies remain charged. The SQL pool distinguishes completed +initialization errors from lost worker replies, preserving unresolved ownership +through controller failure and joined cleanup. + +The driver seals only never-attempted late preparations. Submitted creations +resolve through their original immutable enrollment record or the existing newer +applied-term absence/conflict proof, without automatic initializer replay. A +retry after later cell mutations returns the original creation record and never +restores the initial image. A different creation ID or logical incarnation is a +conflict. Request/namespace S3 minima continue to refuse without archive proof. + +Eight new native cases cover registration refusal plus lifecycle/fault behavior; +seven run both materialized and extent modes over three in-process TLS endpoints. +They hold initialization or committed manifest sync, disconnect/expire responses, +verify command/enrollment shared admission, preserve exact creation identity after +mutation, refuse conflicts without callbacks, and cancel shutdown while physical +work is held. Completed initializer errors permit an explicit retry; a physical +capture panic exercises a lost worker reply and joined cleanup. A unit case checks +initializer digest compatibility, declaration order, invalid/duplicate registration +and old-host validation refusal. The existing command-manifest compatibility case +also passes. The initial lifecycle fixture incorrectly began a new cell's client +sequence at 2; production correctly refused it. The fixture now starts at 1. +A subsequent compile caught a missing Box constructor in the common completion +envelope; that was corrected before qualification. The `bd3069d7` gate passed all +90 Rust cases, then Clippy rejected a redundant unit expression. That expression +was removed and the complete gate repeated at this successor. Both gates and +diagnostic logs are retained. + +All 90 selected Rust cases pass: 73 shared execution, 15 catalog, one initializer +unit case and one existing host compatibility case. All ten gate commands pass, +including all-target Clippy for Cellar/cellar-logic/cellar-ltx with warnings denied, +formatting, layout, dependency isolation, docs, source preflight and 37 validator +cases. The partial record at +`.cellar/cell-groups-evidence/CG-04-enrollment-controller-qualified/checkpoint.json` binds +254 source inputs and four copied executables. Transfer its complete directory +with prior evidence. The gate does not qualify shared-group RustFS, separate-process +crashes, power loss or load behavior. + +CG-04 remains IN_PROGRESS. The critical gap is now integration of public handles +and remote command/query/enrollment dispatch with these shared owners, including +catalog-confirmed retries and safe follower receipt lookup. Enrollment currently +requires a group already attached to the fixed controller roster; lazy group birth +and dynamic registration remain required. Per-group failure isolation and +configuration/cache lifecycle are also incomplete. Complete-group snapshots, +archive/reclamation, effects, topology, RustFS/load and legacy removal remain later +gates. The public host has not cut over. + +## CG-04 controller-owned commands + +Qualified source: `8c615ac91bc3c13e6c25da4f4ab584a73e609207`. + +`CellGroupController::command` transfers a fresh local invocation into the +existing node worker. The separate response handle preserves the original +deadline; dropping or timing out that handle does not cancel SQL preparation, +submission or receipt resolution. Count/byte admission covers queued commands, +active resolution and unconsumed responses, including errors. SQL capture/apply +and result bytes keep their separate pool budgets. Admission refusal returns the +original invocation, preserving any recovery state. One shared progress timer +and applied-event notifications drive retries; continuous traffic cannot reset +the timer and indefinitely postpone progress. + +A late preparation that has never attempted submission is sealed across all +clones and closed through its exact rejection capability. Entering, accepted and +uncertain attempts cannot produce that seal. An attempted submission error stays +uncertain; a later capacity refusal cannot prove the earlier attempt absent. +After submission, resolution requires the exact receipt or the existing newer +applied-term absence proof. It never replays a callback automatically. Retention +is bounded by admission, not by a timeout that discards uncertain state. On a +fatal unresolved command error, the controller retains the invocation through +physical SQL/runtime joins before releasing ownership. + +SQL staging now distinguishes a received completed error from a lost worker +reply. The former cannot hide an installed producer and releases this invocation +without closing the node; the latter remains ambiguous and retained. The public +stage API keeps its existing error type. Physical image capture, callbacks and +manifest publication continue to run on the existing SQL workers. + +Seven new native cases each run in both materialized and extent modes over three +real in-process TLS endpoints. They hold callbacks or committed manifest sync, +disconnect or expire responses, assert bounded ownership and exact identity retry, +verify unconsumed-response charging, and cancel shutdown while physical work is +held. Completed callback errors permit an explicit successful retry. A physical +image-capture panic loses the stage reply and requires joined cleanup. Three unit +cases prove clone sealing, entering/accepted refusal and uncertainty after an +attempted error. The initial fixture used an invalid retry horizon; the corrected +fixture preserves one valid horizon. An initial callback-panic injection was +contained by Cellar's transaction and therefore did not simulate a lost reply; +the final injection occurs during subsequent physical capture. Failed logs remain +with the checkpoint and are not counted as qualification. + +The gate passes 83 selected Rust cases (65 execution, 15 catalog, three submission) +and nine commands: affected tests, all-target Clippy for Cellar/cellar-logic/ +cellar-ltx with warnings denied, formatting, layout, dependencies, docs, source +preflight and 37 plan-validator cases. The partial record at +`.cellar/cell-groups-evidence/CG-04-command-owner/checkpoint.json` binds 248 +source inputs and three copied test binaries. Transfer its complete directory +with prior evidence. This qualifies local shared-controller command ownership, +not real RustFS, separate-process crash, power-loss or load behavior. + +CG-04 remains IN_PROGRESS. The critical next delivery is catalog-routed enrollment +with stable creation identity, then remote shared command/query dispatch and the +public SDK binding to this owner. Missing origin producers on followers still +need a safe leader receipt-lookup route. Dynamic roster, per-group failure +isolation and configuration/cache lifecycle are unresolved runtime gaps. +Complete-group snapshots/archive, effects, topology, RustFS/load qualification and +legacy removal remain later gates; the public host has not cut over. + +## CG-04 autonomous event controller + +Qualified source: `ee684c3394ffccbfa84bfc5d94876de281f68b0b`. + +`CellGroupController` owns a fixed roster's exclusive runtime event stream and +drives existing physical TLS transport, SQL preparation and runtime publication +without fixture pumping. Exact runtime/event/transport and engine/pool/applier +bindings refuse foreign owners even when they share an execution domain. One +node cleanup task runs bounded apply jobs, serializes each group's work and +rotates pending groups. Retained tickets preserve event and physical admission +through SQL backpressure; a shared node timer backs off retries. Only runtime +post-sync `Applied` events advance published progress. Transport queue admission +is not commit evidence. Immediate transient send refusals produce bounded advisory +hints; fatal transport/runtime/apply errors close the node and remain observable. + +Close requests and final-public-handle drop stop the controller. Cleanup joins +transport and physical SQL work before releasing the runtime's engine claim, +retaining outstanding tickets/appliers until all joins complete. Cancelling a +close waiter cannot release a blocked physical producer or permit replacement. +This controller has a fixed startup roster and currently fails the entire node +on a fatal group apply error; dynamic registration and per-group failure isolation +remain required. It does not yet own command invocations or replace the SDK host. + +Six new native cases qualify autonomous TLS replication in both image modes, +two groups sharing peer pools and one SQL worker, admission retry while that +worker is blocked, exact receipt settlement and same-domain foreign-owner refusal. +Real manifest directory-sync faults leave the engine-selected root unchanged. +Cancelled close and final-handle drop while physical sync is blocked retain +ownership until release; no interrupted preparation is published. The initial +shutdown test failed because the harness retained old pool handles while attempting +replacement. It now releases those handles after verifying physical joins; the +production lease checks and assertions remain intact. + +All 58 shared-execution cases and eight gate commands pass at this revision, +including all-target Clippy for Cellar/cellar-logic/cellar-ltx with warnings denied, +format (1,829 Rust files), layout, dependencies, docs, source preflight and 37 +plan-validator cases. The preceding `61aa7752` run passed 289 Rust cases (198 +native across seven suites and 91 transport-filtered library cases), then Clippy +rejected duplicate loading of the physical fault fixture. This successor reuses +the existing module; production code and assertions are unchanged. The affected +execution suite was rerun; other regressions retain their original revision. +The partial record +`.cellar/cell-groups-evidence/CG-04-event-controller-qualified/checkpoint.json` +binds 244 source inputs and the execution test binary. Transfer it with the +retained `CG-04-event-controller/` gate and prior evidence. Four ignored process +helpers ran through their parents in the earlier regression run; its RustFS seed +case was skipped. Local TLS endpoints share a process. This is not RustFS, +independent-failure-domain, power-loss or load qualification. + +CG-04 remains IN_PROGRESS. The most critical next gap is command ownership after +caller disconnect: move fresh invocations into this controller and distinguish +response deadlines from bounded resolution ownership. Preserve exact origin and +inputs through uncertainty; never replay ambiguous callbacks or treat missing +local state as non-commit. Before adding cancellation cleanup, seal only a +preparation that has never attempted submission: the current submission state +resets to Open after an error, which cannot establish that append did not occur. +Retain a distinct uncertain-attempt state and qualify timeout/disconnect after SQL +entry and provisional acceptance. Then complete routed enrollment, shared query +RPC and SDK integration. Configuration/cache lifecycle, snapshots/archive, +effects, topology, real RustFS/load and final legacy removal remain later gates. + +## CG-04 physical-group TLS transport + +Qualified source: `eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa`. + +`RaftTransport::start_shared` starts directly on the physical-group runtime. +`GroupRoute` and typed peers share the existing node-owned TLS +connections, control/data pipelines, byte/item budgets and joined shutdown. +Route installation requires an existing selected group, its schema, committed +member/disk binding and exact local process/listeners. Authorization rechecks +current routes and membership through queueing, I/O and response delivery. +Physical routes participate in node-view refresh, pool revocation/pruning and +combined group limits. Conflicting query/inspection endpoints refuse. + +Marker-1 frames use typed histories and a new protocol digest. Physical consensus +has no cell key; old untagged wire layouts refuse. Requests and responses bind +full replica identity, schema, sequence, payload and TLS channel. The exclusive +runtime transport claim survives until physical tasks exit. Delivery is only +receiver admission, not a Raft persistence or commit proof. Unsupported +complete-group snapshot messages refuse. The temporary legacy runtime/frame arms +are explicitly scheduled for deletion with the old host in CG-16. + +Three new native cases use real TLS endpoints. Two groups replicate three +independent SQLite cells each on the same peer pools. Unknown groups, wrong +schema/local process, conflicting grants and unsupported snapshots refuse. +Removing one group's route retains the other group and its peer pools; restoring +it delivers again. A second transport refuses until joined close releases its +claim. Existing routed command cancellation, exact receipt and leader-transfer +assertions also pass over TLS in both image modes. A new frame case verifies +history/incarnation binding and refusal of old wire/protocol layouts. + +All 283 selected Rust cases pass: 192 native cases across seven suites and +91 library cases selected by the transport filter. All nine commands pass, +including all-target Clippy for Cellar/cellar-logic/cellar-ltx with warnings denied, +formatting (1,826 Rust files), layout, dependency isolation, docs, source preflight +and all 37 plan-validator cases. The partial record at +`.cellar/cell-groups-evidence/CG-04-physical-transport/checkpoint.json` binds +241 source inputs and eight selected executables. Initial private-import and +legacy-membership-validator compile failures, the corrected targeted run and +formatter log are retained. Four process helpers run through their parent cases; +the RustFS-specific seed case is skipped. Prior checkpoints retain their original +source/scope. These local endpoints share a process; they do not qualify independent failure +domains, RustFS, power loss or load. Latest fetched `origin/main` is included and +the requested MR-13/MR-14 progress JSON files remain excluded. + +CG-04 remains IN_PROGRESS. The critical next gap is the production controller: +retain command invocations independently from callers, drive existing shared +transport/apply/completion events, and maintain bounded reconciliation after the +response deadline. The fixtures still pump events, so a cancelled preparation +future is not evidence of controller ownership after ingress disconnect. Reuse +the now-working physical transport and SQL/apply owners; qualify disconnects after +SQL entry and provisional acceptance without replaying ambiguous callbacks. Then +complete routed enrollment and public SDK/shared-query RPC integration. +Configuration/cache lifecycle, complete-group snapshots/archive, effects, +topology, real RustFS/load and final legacy removal remain later requirements. + +## CG-04 catalog-routed command phases + +Qualified source: `1f014f42264c73b1d5e3cd387f65ffed45140ecb`. + +`CellGroupDispatcher::command` freezes logical command identity, input, namespace +minimum durability, original routing budget and domain-captured time/randomness. +Preparation retains its original runtime, SQL pool, exact route and physical +preparation IDs before entering SQL. A cancelled response or routing retry resumes +that producer instead of executing a second destination's callback. Submission +reuses exact shared acceptance and pre-append seals. Acceptance is provisional; +settlement requires the exact receipt, while absence requires the existing newer +applied-term barrier. Errors preserve command identity and conservative ambiguity. +Returned receipts retain their output admission and sign the cell observation. +Both request and namespace S3 minima refuse until real shared archive proof exists. + +Two new three-voter cases cover both image modes, held-callback cancellation, +retry through a follower while retaining the original producer, duplicate proposal +acceptance, same-term uncertainty, exact receipt recovery after another cell +advances the group, and a real leader transfer. A proven non-commit can retry with +the same captured time/randomness. S3 refusal precedes callback execution. +The leadership fixture checks the caller's actual cell and uses the production +applier where installed; the held newer-term barrier assertion remains unchanged. + +All 105 native cases pass across execution, catalog, read and runtime suites; +one runtime helper is invoked by its parent. All eight commands pass, including +all-target Clippy for Cellar/cellar-logic/cellar-ltx with warnings denied, +formatting, layout, dependency isolation, docs, source preflight and all 37 +plan-validator cases. The partial record at +`.cellar/cell-groups-evidence/CG-04-routed-commands-qualified/checkpoint.json` binds +229 source inputs and four selected executables. The failed gate at +`CG-04-routed-commands` is retained: all native cases passed, then Clippy rejected +the large command error. Its typed cause is now boxed, preserving all assertions. +A second gate at `CG-04-routed-commands-final` retained the passing native tests +and fixture lint failure; the direct Tokio select now uses Cellar's required +fair-polling macro. Earlier compile and fixture failures are retained too. No RustFS +or performance qualification was run. Latest fetched `origin/main` at `8a93ffd7` is included; +the requested MR-13/MR-14 JSON files remain excluded. + +CG-04 remains IN_PROGRESS. The critical next gap is production controller ownership +of these phases after ingress cancellation, including an explicit bounded +reconciliation lifecycle beyond the response deadline. The current tests retain +the invocation while cancelling its preparation future; they do not qualify a +controller after the caller drops. Adapt the existing transport's cell-scoped +identity to physical group history, drive transport/apply/completion events, and +qualify exact resolution after caller disconnect. Then complete routed enrollment +and public SDK binding. Missing-origin follower receipt routing, configuration and +cache lifecycle remain unfinished. Reuse existing workers and resource owners; +complete-group snapshots/archive, effects, topology, real RustFS/load and the final +hard cutover remain later task gates. + +## CG-04 catalog-routed query invocations + +Qualified source: `8723db2df7436478cbc5efc9147d65d3b7651247`. + +`CellGroupDispatcher` binds a registered class to its configured namespace, +version keys and execution domain. Queries freeze bounded input, operation and +consistency before discovery. Clones share invocation admission, and the canonical +fingerprint includes logical history and routing limits. Deadline capture precedes +validation and hashing. Existing catalog sessions perform lookup, cached routing, +member retries and exact redirect confirmation without replenishing deadline or +attempt count. Each attempt is consumed before asynchronous destination work. + +Local execution authenticates AtLeast tokens, checks the destination's full route +and logical history, and reuses existing runtime read authority and registered SQL +workers. Foreign domain clocks and engines refuse. The response token signs the +selected cell observation. Dropped routing releases only invocation admission; +entered SQL retains its independent input, image and worker ownership through +joined shutdown. Directory metadata cannot create a physical group owner. + +The two new three-voter cases cover both image modes, real directory lookup, +linearizable execution, a lost response retry, follower AtLeast, foreign +cell/domain/class/engine rejection, invalid redirect ticket invalidation, original +timeout/attempt exhaustion, a confirmed replacement with no physical owner, and +cancelled routing while a callback is held. The replacement case tests only +metadata confirmation and refusal; it does not qualify live group transfer. + +All 103 native cases pass across the shared-group execution, catalog, existing +read and runtime suites. The runtime process helper is invoked by its parent. +This targeted gate complements the prior 319-case registered-query checkpoint at +its original source; it does not rerun the complete previous selection. +All eight commands pass, including all-target Clippy for +Cellar/cellar-logic/cellar-ltx with warnings denied, formatting (1,821 Rust files), +layout, dependency isolation, docs, source preflight and all 37 plan-validator +cases. The partial record is +`.cellar/cell-groups-evidence/CG-04-routed-queries/checkpoint.json`, binding +227 source inputs and four selected executables. The initial missing type +qualification, corrected targeted run and formatter logs are retained separately. +No RustFS or performance qualification was run. The branch contains latest fetched +`origin/main` at `8a93ffd7`; the requested MR-13/MR-14 JSON files remain excluded. + +CG-04 remains IN_PROGRESS. The most critical gap is routed command/enrollment +execution with retained ambiguous outcomes and exact receipt/creation settlement. +Next extend the same dispatcher and qualify enrollment, command, lost response and +linearizable query together. Production node-controller event/transport integration +and replacing the public SDK's old host binding remain required; the native +fixture still pumps shared-runtime events. Retry/cache/configuration, shared effects, +snapshots/archive, physical fault and real RustFS/load gates remain unfinished. + +## CG-04 registered queries with exact cell observations + +Qualified source: `7a79511d4b8606d11177778a3fe8adf71a8d901a`. + +`CellGroupSqlPool::query_registered` dispatches compiled queries through the +existing coherent query owner and fixed SQL workers. It validates operation +kind/contract and full schema before opening the selected image, bounds retained +operation/input storage, and preserves the original authority deadline. The +callback receives the exact `CellHistory` and `CellVersionObservation` returned +in its result, including the selected cell's decision witness. It never derives +a mutation sequence from group log progress. + +The two new three-voter cases exercise materialized and extent images. An older +A pin returns its historical state after A and B mutate; a fresh linearizable A +query returns A's sequence and witness independently of B's later group progress. +Tokens round-trip the selected observation. Invalid contract, kind, operation, +schema and input refuse before callback execution. Read-only restrictions, +contained panics and SQL deadline interruption remain enforced. Cancelled waiters +retain input/image/worker ownership, and shutdown joins entered callbacks before +releasing reservations. + +The temporary old adapters explicitly use `QueryVersion::GroupCut` and +`RaftClass::invoke_group_query`. These preserve the actual old in-memory group cut +without fabricating a cell sequence. CG-16 requires removing both with the old +host; the final context exposes the selected cell observation directly. This is +an integration checkpoint, not a second permanent production mode. + +All 319 selected Rust cases pass: 280 native cases across thirteen suites, +30 Cellar library cases, five pure cell-group core cases and four LTX image-store +cases. The existing native read suite additionally covers current-term barriers, +minority refusal, follower catch-up, delayed replies and version authentication. +Six process helpers run through parent cases; three RustFS-specific cases remain +unrun. These results do not qualify the new shared dispatcher or infrastructure. + +All seventeen commands pass, including all-target Clippy for +Cellar/cellar-logic/cellar-ltx with warnings denied, formatting (1,819 Rust files), +layout, dependency isolation, docs, source preflight and all 37 plan-validator +cases. The retained checkpoint is +`.cellar/cell-groups-evidence/CG-04-registered-queries/checkpoint.json`, binding +225 source inputs and sixteen selected executables. Pre-commit library, targeted +query, format and Clippy logs are retained separately. The branch contains latest +fetched `origin/main` at `8a93ffd7`; the requested MR-13/MR-14 JSON files remain +unchanged from main. + +CG-04 remains IN_PROGRESS. The most critical gap is catalog invocation sessions +and production SDK dispatch into registered enrollment, command and query workers, +with namespace-authorized classes, exact receipt settlement and original deadline, +fingerprint and attempt budgets. Next qualify that actual routed path under lost +responses and redirects. Reuse the existing owners instead of adding a worker +pool. Retry/cache and configuration handling, shared effects, complete-group +snapshots/archive, process-kill cuts and real RustFS/load gates remain unfinished. + +## CG-04 registered commands and logical effect identity + +Qualified source: `b28078afbd243c18ac7c671a481a3e369c086ab2`. + +`CellGroupSqlPool::stage_registered` now dispatches a real compiled `RaftClass` +on the existing fixed workers. Its retained input is bounded by the producer +budget. The worker verifies operation kind/contract, full class schema, normalized +input digest and authenticated inbox/reserved client identity before receipt +lookup, including duplicates. It uses the existing private receipt transaction, +prepared submission and exact committed settlement paths. The shared path obtains +logical identity from `CellExecutionAuthority`; it never constructs `GroupScope`. + +`CommandContext::identity()` and `RaftEffectTarget` carry `CellHistory`. System +clients and signed effect lanes/requests bind the logical cell, so normal physical +group reassignment preserves identity and a fresh cell incarnation separates it. +Canonical marker-1 records omit physical groups and use distinct signature/lane +and compiled-class domains. Old group-bearing payloads and old signatures refuse; +there is no compatibility reader. The temporary old host projects its actual +logical binding into the same command/effect interfaces. + +The new real three-voter tests use two cells and production image preparation in +both storage modes. Callbacks see the exact cell history and persist alarm state. +A's receipt survives B advancing group progress; duplicates do not run callbacks. +Wrong contract, input digest, operation and schema produce their specific typed +errors even on duplicate requests, without retaining cell admission. Unit cases +cover placement-independent signed identity, new-incarnation separation, and old +wire/domain refusal. This does not qualify live transfers or shared effect dispatch. + +All 295 selected Rust cases pass: 256 native cases across twelve suites, 30 Cellar +library cases, five pure cell-group core cases and four LTX image-store cases. +The native gate includes the existing alarm/outbox/inbox, extent-host and host +regressions. All seventeen commands pass, including all-target Clippy for +Cellar/cellar-logic/cellar-ltx with warnings denied, formatting (1,818 Rust files), +layout, dependency isolation, docs, source preflight and all 37 plan-validator +cases. The retained checkpoint is +`.cellar/cell-groups-evidence/CG-04-registered/checkpoint.json`, binding 221 source +inputs and fifteen selected executables. Failed compile, missing test delimiter, +Clippy type-alias and omitted test submission-transition attempts are retained +with the corrected targeted runs. Six process helpers run through parent tests; +three RustFS-specific cases remain unrun. The branch contains latest fetched +`origin/main` at `8a93ffd7`. + +CG-04 remains IN_PROGRESS and CG-08 remains TODO. The most critical next gap is +registered query context and version observations, followed by catalog/SDK routing +into these workers with original deadline, fingerprint and retry budgets. Query +contexts still carry the old committed group version; never invent a cell sequence +from that index. Production cache/retry and configuration handling, process-kill +cuts, shared effects and later snapshot/archive/RustFS/performance gates remain. +The opportunity is to complete one dispatcher over the existing consensus, SQL, +read and recovery owners, then remove the superseded production host. + +## CG-04 physical apply faults and selected-image recovery + +Qualified source: `1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1`. + +`CellGroupApplier::recover` reconstructs only the engine-selected complete root. +It verifies every selected physical image, logical binding and geometry, and +retains the same manifest-page owner for subsequent apply. Materialized IDs use +the existing deterministic derivation; filenames supply no authority. Extent +roots verify the full cell history and schema. Missing or corrupt selected closure +refuses. Startup must still consume the catalog permit and establish current-term +quorum before serving. + +Six new native cases exercise both storage modes. At actual image and manifest +directory-sync cuts, cancelling preparation and then cancelling close retains +worker, budget and completion ownership. A replacement pool refuses until cleanup +joins; exact preparation can then retry. Injected directory-sync failure preserves +the old selected root and releases preparation reservations. Cold restart closes +all prior engines, stores and execution domains, selects the published cell A, +ignores prepared but unpublished B, and replays B from the durable committed log. +It refuses serving authority without fresh quorum and refuses a missing selected +image directory without changing the marker. + +This is joined cold restart, not whole-process kill or power-loss qualification. +The first recovery run hit an ownership refusal because the fixture still held +engine-owned manifest pins. Closing the complete owner before reopening fixed the +fixture; ownership rules and assertions were preserved. + +All 260 selected Rust cases pass: 233 native cases across ten suites, 23 Cellar +library cases and four LTX image-store cases. Fifteen commands pass, including +all-target Clippy for Cellar/cellar-logic/cellar-ltx with warnings denied, formatting +(1,815 Rust files), layout, dependency isolation, docs, source preflight and all 37 +plan-validator tests. The partial checkpoint is +`.cellar/cell-groups-evidence/CG-04-apply-recovery/checkpoint.json`, binding 205 +source inputs and twelve copied executables. Intermediate failures and passing +targeted runs are retained. Six process helpers run via parent cases; three +RustFS-specific cases remain unrun. The branch contains latest fetched +`origin/main` at `8a93ffd7`; the requested MR-13/MR-14 JSON files remain excluded. + +CG-04 remains IN_PROGRESS. The most critical gap is the registered SDK/catalog +dispatcher: callback contexts still expose legacy cell-scoped group identity. +Next carry logical cell identity and selected-version observations through command, +query, receipt and effect validation, then use the existing shared worker path with +original deadlines, request fingerprints and retry budgets. Production cache/retry +lifecycle, configuration-entry handling and process-kill cuts remain open. Group +snapshot/compaction, RustFS recovery, performance and public cutover remain later +gates. The opportunity is to connect the existing owners, reads and recovery into +one production dispatcher rather than build another runtime. + +## CG-04 committed image preparation on shared workers + +Qualified source: `c9100ec774d66f30c14eae009f9a9c76d2bc0c33`. Runtime implementation: `2085cc96`. + +`CellGroupApplier` owns the same manifest-page store used to verify startup. It +checks the exact persisted committed entry, engine, group and preceding root +before preparing cell-addressed images. Enrollment and mutation work support +materialized and extent storage, derive bounded alarm/outbox hints from the actual +SQLite image, preserve logical versions and exact commit witnesses, and refuse +malformed scheduling state. Membership/topology administration remains refused +pending its owner integration. + +`CellGroupSqlPool::prepare_apply` uses the existing fixed SQL workers and their +reserved progress lane. Apply has a separate byte/item budget, so retained SQL +producers cannot consume its admission. `CellSqlLimits::node_bytes` bounds each +of the producer and apply budgets independently; account for their combined +ceiling. Usage exposes both budgets separately. The returned preparation retains +its reservation across clones, delivery and owner completion. Completion queues +publication; only durable owner apply establishes the new selected root. + +Materialized duplicate preparation now reopens the exact verified image through +`GenerationStore::pin_prepared`. It does not reset retirement or infer publication. +Cold images require independent byte verification and persistence barriers. +Regression tests cover restart, corruption, mismatched expected state, reader pins +and reclamation. Both real three-voter cases cover enrollment, mutation, derived +hints, forged/foreign/stale work, duplicate preparation, saturated command admission, +old-root selection before publication and retention during a held log sync. + +All 254 selected Rust cases pass: 227 native cases across ten suites, 23 +Cellar library cases and four LTX image-store cases. All-target Clippy for +Cellar/cellar-logic/cellar-ltx with warnings denied, formatting (1,812 Rust files), +layout, dependency isolation, docs, source preflight and all 37 plan-validator +cases pass. The partial checkpoint at +`.cellar/cell-groups-evidence/CG-04-apply-final/checkpoint.json` binds 202 source +inputs, 12 copied executables and fifteen successful commands. The failed gate +at `CG-04-apply` is retained: the newly inlined apply branch overflowed an existing +recovery fixture's stack. The branch is now heap allocated; normal stack limits +and assertions remain unchanged. Earlier fixture, materialized retry and Clippy +failures are retained too. Six process helpers run through parent tests; three +RustFS-specific cases remain unrun. + +CG-04 remains IN_PROGRESS. The most critical next validation is cancellation, +shutdown and crash injection while this new image/manifest preparation is held; +the held publication sync does not qualify those earlier cuts. Then integrate the +registered SDK/catalog dispatcher with logical cell identities and observations. +Production retry/cache lifecycle and configuration-entry integration remain open. +No complete shared-group snapshot, compaction, RustFS or performance claim is made. + +## CG-04 durable selected-version commit witnesses + +Qualified source: `ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2`. + +Authenticated cell manifest pages now require a complete selected-version +`CommitWitness`: original group history, term/index and proposal identity. The +local installation/mutation position remains separate for future group transfers. +Marker-1 pages without this required field refuse; there is no legacy decoder or +proposal inferred from later group progress. Shape checks bound IDs, namespace, +positions and initial enrollment identity. Ordinary update preparation requires +the current group and exact mutation position; the physical Raft owner additionally +matches the witness to the actual committed entry's proposal before publication. +Repeated enrollment and unchanged-cell progress preserve the selected witness. + +Immutable-query results expose their exact `CellVersionObservation`; committed +receipt results expose the selected-version observation, which can cover an older +command receipt. Both use the authenticated leaf rather than the entry cache. +Native tests reject a forged witness despite otherwise correct apply/image data, +preserve older pinned observations after a newer cell mutation, round-trip signed +tokens, and recover exact witnesses after restart and image collection in both +physical image modes. Codec tests reject missing/malformed witness fields. + +All 214 selected Rust cases pass: 193 native cases across eight suites and 21 +library cases covering SQL workers, receipts, budgets, options and manifest pages. +All-target Clippy for Cellar/cellar-logic with warnings denied, formatting (1,806 +Rust files), layout, dependency isolation, docs, source preflight and all 37 +plan-validator tests pass. The retained partial checkpoint at +`.cellar/cell-groups-evidence/CG-04-witness/checkpoint.json` binds 194 source inputs, +nine copied executables and thirteen successful command logs. Pre-commit compile +and formatting logs are retained separately. Four process helpers run through +parent tests; three RustFS-specific cases remain unrun. + +CG-04 remains IN_PROGRESS. This proves witness persistence and observations at the +owning storage/apply/read boundaries, not actual log compaction, group snapshot +installation, registered SDK serving or S3 recovery. Next integrate registered +callback identity, query/version-token dispatch and catalog invocation sessions; +complete the production apply adapter, retry/compaction and cache lifecycle. + +## CG-04 read-pin capacity and repaired qualification + +Qualified source: `3da4481e09c3767face30469ffc855d302a7c142`. + +Read-image retention now uses `RaftRuntimeOptions::max_read_pin_bytes`, independently +of transport request capacity. The default remains 64 MiB. Node and group read +reservations use the explicit pin ceiling; ordinary peer/apply admission remains +separate. Validation bounds configured capacity by the native image-size envelope +and pending-read count using wide arithmetic. The new unit case admits the full +native image limit without raising transport limits, accepts the minimum metadata +reservation and upper envelope, and refuses out-of-range settings. The native +read fixture now uses `cellar::asyncrt::select!`. + +All 204 selected Rust cases pass: 186 native cases across seven suites, 11 SQL +queue/extent-worker tests, four receipt tests, two budget tests and one runtime +configuration test. All-target Clippy for Cellar/cellar-logic with warnings denied, +formatting (1,806 Rust files), layout, dependency isolation, docs, source preflight +and all 37 plan-validator tests pass. The checkpoint at +`.cellar/cell-groups-evidence/CG-04-reads-final/checkpoint.json` binds 194 source +inputs, eight copied executables and twelve successful command logs. It retains +the failed full gate at `88c4717d` and earlier targeted compile/stack diagnostics +separately. No test limits or assertions were relaxed. Four process helpers run +through parent tests; three RustFS-specific cases remain unrun. + +This qualifies the low-level coherent-read and immutable-query boundary in both +image modes. CG-04 remains IN_PROGRESS. Registered SDK/catalog dispatch, callback +identity and version-token integration, real compaction/retry dispatch and cache +lifecycle remain required. In particular, the selected leaf retains its last +mutation position but not the corresponding mutation proposal ID: preserve a +complete authenticated cell commit witness before issuing SDK version observations +from that image after log compaction. Group progress is never that witness. + +## CG-04 coherent cell reads and immutable SQL queries + +Implementation source: `88c4717d`. + +The shared node runtime now owns bounded read requests with original absolute +domain deadlines. Linearizable requests use unique nonce/counter contexts and +current-term ReadIndex, then wait for contiguous group apply before pinning the +target cell. Raw or forwarded read requests cannot borrow those internal contexts. +A minority leader cannot substitute a cached commit position. A read of unchanged +A waits behind committed B while retaining A's own version. Stale/AtLeast can +select committed follower images; AtLeast compares logical cell history and +sequence/image, never unrelated group indexes. + +Separate node/group admission retains waiting reads and delivered pins. Image +selection grows the existing item reservation to its logical byte size without +consuming ordinary peer/apply capacity. The pin is consumed once by an immutable +query on the existing SQL workers. Queries can coexist with an unresolved writer +and read only committed images; an older pin remains coherent across a later +publication of the same cell. SQL output retains its own reservation through +borrowed result accessors. Wrong engines, routes, histories, expired deadlines +and revoked strong-read terms refuse. + +SQL progress hooks check the domain deadline, waiter cancellation and shutdown. +Callbacks retain worker and pin ownership until they return; callback panics are +contained and close joins entered work. Deadline checks also cover queued replies, +so a ready result cannot bypass expiry when the receiver resumes late. + +Both image modes pass targeted strong-read and lifecycle cases. The first native +compile used an unavailable clock accessor; fixtures now run clock reads inside +the owning domain. Expanding the strong cases with a pending AtLeast read and +an older same-cell pin exposed unnecessary async forwarding layers overflowing +the default fixture stack. Reexports remove those layers; a visibility error in +that intermediate change was corrected. The enhanced cases pass on the normal +stack without weakening assertions. + +The retained gate at `.cellar/cell-groups-evidence/CG-04-reads/gate-commands.json` +completed four test commands: 203 selected Rust cases passed. The fifth command, +all-target Clippy, failed because `tests/support/cell_group_sql_reads.rs` uses the +forbidden `tokio::select!` macro. Subsequent commands in that gate did not run. +This source was unqualified at that point; the repaired source and full gate are +recorded above. No task status advances. + +The repair required replacing that fixture macro with +`cellar::asyncrt::select!` and separating the logical read-image pin budget from +`RaftRuntimeOptions::max_request_bytes`: the current transport ceiling is 512 MiB, +whereas valid native images can reach 10,000,000,000 bytes. The repair above adds +the explicit bounded pin-byte option and validation test, preserves node/group accounting, count limits +and cancellation ownership, and retains this failed gate separately. + +Registered SDK dispatch, callback identity and version-token integration, catalog +sessions, production retry/compaction and cache lifecycle remain required. This +is partial CG-04 implementation, not the production cutover or release gate. + +## CG-04 prepared submission and rejection cleanup + +Qualified source: `0cf89c403f4a500d18116499b8fc0f219195cd54`. + +A prepared SQL attempt now carries its engine, group and preparation term, with +one submission decision shared by all clones. Exact accepted retries return the +original provisional position/digest, including after apply. Only an actual +owner's definitive pre-append capacity refusal seals the preparation and returns +a rejection proof. A queue overload remains uncertain. A lost rejection response +can be recovered from the local seal without contacting a closed runtime. +The first prepared submission closes raw byte-only proposal paths on that owner; +an uncertain append observation fails the owner closed rather than permitting +another append or manufacturing a rejection proof. + +SQL workers match rejection proofs to the exact preparation, not merely the +business proposal ID. Foreign engines, other cells and replacement producers +refuse old proofs. Cleanup closes the writer before releasing logical-name +admission. Queued submissions, prepared outputs and retained proofs keep their +original byte/operation reservations through cancellation and lost responses. +The real three-voter tests cover initialization and mutation in both image modes, +including a fresh preparation with the same proposal identity after rejection. + +Intermediate failures exposed two fixture accounting issues. A one-slot runtime +could refuse a read-only status poll while retaining admitted peer work; the +fixture now retries that poll while pumping the cluster under its existing +deadline. Completed enrollment outputs then correctly exhausted the SQL pool's +four retained-operation reservations. The fixture now asserts that accounting, +drops the completed phase's outputs, and asserts zero reservations/bytes before +preparing mutations. Resource limits and rejection assertions remain unchanged. +All 197 selected Rust cases pass at this source: 182 native cases across seven +suites, 11 SQL queue/extent-worker unit tests and four receipt unit tests. +All-target Clippy with warnings denied, formatting (1,802 Rust files), layout, +dependency isolation, docs, source preflight and all 37 plan-validator tests pass. +The checkpoint at `.cellar/cell-groups-evidence/CG-04-submission/checkpoint.json` +binds 189 source inputs, ten successful command logs and eight copied executables. +It retains the initial failures, diagnostic backtraces and repaired targeted run +separately. Four process helpers run through parent tests; three RustFS-specific +cases remain unrun. + +The two new cases raise the CG-04 contract to 12 required cases and the package +to 92. No task status advances. Registered SDK/catalog dispatch, coherent reads, +production retry/compaction integration and cache lifecycle remain required; +this checkpoint does not qualify the complete CG-04 or release gate. + +## CG-04 bounded proposal acceptance and enrollment capacity + +Qualified source: `446dd9057f10c8221122d0d3a9a21254adeb448f`. + +The actual group owner now counts pending accepted creations alongside applied +cells before appending a new +enrollment. Existing cells continue accepting mutations at the configured limit. +Leader-local acceptance records bind a proposal to its exact bytes and original +position; retries cannot append a pending mutation twice. Changed bytes under the +same pending identity refuse, and independent proposals for a pending logical +name wait. Mutation bases are checked against the selected cell image. After +apply, callers resolve receipts or creation records instead of resubmitting stale +images; a refusal does not prove that the original request failed. + +Acceptance retains a separate shared node budget until durable apply, role/term +loss, terminal failure or owner destruction. It retains no LTX payload or image pin and cannot +consume Ready/completion admission. A current-term applied barrier remains +mandatory before a new leader admits work, so inherited enrollments count even +when the previous process's pending map is gone. + +The preliminary native run passed 25 cases and exposed a restart-fixture issue: +restarting only the prior leader leaves live followers' leader leases intact, so +its immediate pre-vote need not elect it. The fixture now restarts all voters from +their retained disks before requiring inherited apply. A later run exposed an +extra async constructor layer overflowing the fixture's default stack; that +layer was removed. A larger-stack diagnostic is retained only for diagnosis, +not qualification. It also exposed the fixture transport discarding overloaded +peer admission. The fixture now retains and retries those exact frames under its +existing deadlines, as the direct-owner fixture already does. All 27 shared-group +execution cases now pass on the normal stack. + +All 195 selected Rust cases pass at the qualified source: 180 native cases across +seven suites, 11 SQL queue/extent-worker unit tests and four receipt unit tests. +All-target Clippy with warnings denied, formatting (1,800 Rust files), layout, +dependency isolation, docs, source preflight and all 37 plan-validator tests pass. +The checkpoint at +`.cellar/cell-groups-evidence/CG-04-proposal-admission/checkpoint.json` binds 187 +source inputs, ten successful command logs and eight copied executables, with +intermediate failures and the larger-stack diagnostic retained separately. Four +process helpers run through parent tests; three RustFS cases remain unrun. + +Four distinct admission cases are now required by the CG-04 executor gate, +bringing the package to 90 required cases. Their oracles are separate from the +existing SQL fairness and receipt-settlement cases. Registered dispatch, catalog +sessions, coherent reads, production retry/compaction and cache lifecycle remain +required. No task status or release claim advances. + +## CG-04 shared-worker initialization and enrollment + +Implementation source: `6e09bbc7`. + +`CellEnrollmentAuthority` now binds an actual leader, current-term applied prefix, +route and logical cell name before a cell image exists. Initialization uses the +existing fixed SQL workers and admission budgets. One logical name reservation +covers commands and initializers, so a competing incarnation cannot bypass a +running or unresolved producer. Worker-owned SQLite initialization captures and +verifies its complete initial LTX image; followers apply those bytes without +running the callback. Root workspace preparation is shared with ordinary commands. + +Prepared SQL outputs now carry their preparation term. `propose_at_term` checks +that fence on the real RawNode owner in the same turn as proposal admission. +Retained output cannot be appended under a newer term even after the original +node becomes leader again. The raw unfenced protocol API remains a verification +boundary; registered dispatch must use the fenced path. + +Committed retries return the immutable creation record without opening SQLite or +rerunning initialization after later mutations. `CellEnrollmentState` observes +verified current-term applied creation state on followers as well as leaders. +Exact records prove commitment; missing/conflicting records retire old producers +only after a strictly newer applied term. Resolution closes the writer before +releasing name admission, while retained outputs keep their byte reservation. +Cancelled request and close waiters cannot release an entered initializer. + +Four native cases cover both image modes: competing histories of one name, +cancelled initialization and exact resume, another cell committing while its peer +initializer is blocked, three independent cells under one physical group on each +voter, creation retries after mutation, conflicting creation refusal, cancelled +close and workspace release, follower resolution of absent/conflicting/committed +creations, and rejection of old prepared output after leadership returns. + +All 191 selected Rust cases pass at `6e09bbc7`: 176 native cases across seven +suites, 11 SQL queue/extent-worker unit tests and four receipt unit tests. +All-target Clippy passes with warnings denied. The retained partial checkpoint +at `.cellar/cell-groups-evidence/CG-04-initialization/checkpoint.json` binds 184 +source inputs, eight copied executables and the command logs. The gate stopped +at two formatting differences; that failure is retained. `82ae6877` changes only +line wrapping in those two authority checks. Successor verification is recorded +separately in `CG-04-initialization-format/gate-commands.json`: formatting +(1,798 Rust files), layout, dependency isolation, documentation, source preflight +and all 37 plan-validator tests pass. No Rust behavior changed and the Rust tests +were not repeated for line wrapping. Four process helpers execute through parent +tests; three RustFS cases remain explicitly unrun. No task status advances. + +Configured group capacity policy, catalog routing sessions, registered SDK +creation/command callbacks, coherent reads, +production retry/compaction integration and preview/cache lifecycle remain open. +This is partial CG-04 implementation, not the final hard cutover or release gate. + +## CG-04 receipt lookup independent from admission + +Qualified source: `91c3ecfc020e0d42738e4a6395045eb8a44c51f9`. + +Reconciliation now uses a read-only identity lookup that preserves sequence/floor, +request identity and result-allocation checks independently from capacity for new +clients or retained receipts. The command path still enforces both ceilings and +uses the same identity inspection; no extra client lookup is added to normal +admission. A full identity table can no longer leave an absent older-term +producer unresolved merely because it cannot admit a new client. + +Both native image-mode fixtures fill the selected cell's identity capacity with +another committed client after leadership changes, then reconcile the original +absent request. The changed cell image forces actual receipt lookup. A subsequent +new-client command still refuses at the original limit; increasing its explicit +capacity allows a fresh command from the committed base. Separate cases replicate +bounded receipt maintenance through the real group and return CommandExpired for +the pruned original receipt, and CommandConflict for a different request that won +the same client sequence. They prove callback suppression, producer admission +release, retained-output accounting and a valid next command after the refusals. +Private SQLite tests preserve capacity admission, sequence gaps, floors and result +allocation bounds. + +All 187 selected Rust cases pass: 172 native cases across seven suites, 11 SQL +queue/extent-worker unit tests and four receipt unit tests. All-target Clippy +with warnings denied, formatting (1,795 Rust files), layout, dependency isolation, +docs, source preflight and all 37 plan-validator tests pass. The checkpoint at +`.cellar/cell-groups-evidence/CG-04-receipt-lookup/checkpoint.json` binds 181 source +inputs, ten successful command logs and eight copied executables. The preliminary +four-case native pass is retained separately. Four process helpers execute through +parent tests; three RustFS cases remain explicitly unrun. No task status advances. + +This remains partial CG-04 implementation. Concurrent initialization, coherent +reads, registered dispatch, production retry and actual compaction integration, +and preview/cache lifecycle remain required before its complete gate. + +## CG-04 applied-barrier receipt reconciliation + +Implementation source: `8a0cf640ae70f3280f02e281f81b55a8823205fd`. + +The existing SQL pool now reconciles the exact pending proposal against the +original request identity in a selected immutable cell image. The new +`CellReconciliationAuthority` is issued from a verified durable current-term +applied group root, including on followers. It grants no write, strong-read or +S3 authority. A matching receipt may resolve from a later cell image without +requiring the original log entry. Its position identifies the inspected image. +A missing receipt releases the old producer only behind a strictly newer applied +term. Expiry/conflict remain distinct refusals; missing producers report uncertainty. + +The worker validates engine, route, cell history and proposal identity, rechecks +pin freshness after receipt inspection, and closes the disposable writer before +releasing cell admission. Reconciliation uses reserved progress capacity; retained +receipt outputs keep their existing byte reservation. + +Both image-mode tests use real three-voter elections and SQLite/LTX. They hold +the new-term apply on the old leader and reject premature reconciliation; advance +B again on the new leader while its first producer remains on the old follower; +then recover B's original receipt and retire A's absent older-term proposal. +They also cover same-term absence refusal, stale and foreign proofs, mismatched +proposals, unknown producer uncertainty, retained-output budgets, duplicate +callback suppression and reopening A from its committed base. + +All 181 selected Rust tests pass: 170 native cases across seven suites and 11 +SQL queue/extent-worker unit tests. All-target Clippy with warnings denied, +formatting (1,795 Rust files), layout, dependency isolation, docs, reviewed source +preflight and all 37 plan-validator tests pass. The partial checkpoint at +`.cellar/cell-groups-evidence/CG-04-reconciliation/checkpoint.json` binds 179 +source inputs, nine successful command logs and eight copied executables. +The initial fixture failure is retained: `MsgTransferLeader` is a local +administrative request, correctly rejected by the peer boundary. The corrected +fixture delivers the current leader's `MsgTimeoutNow` directive and drives the +real election, persistence and apply. Four process helpers run through parent +tests; three RustFS cases remain explicitly unrun. This remains partial CG-04 +implementation; no task status advances. + +Next, split read-only receipt lookup from new-client capacity admission and +qualify saturated identity tables, expiry floors and conflicting identities. +Then integrate real snapshot/compaction, production retry dispatch, concurrent +initialization, coherent query serving and preview/cache lifecycle. No full +CG-04, RustFS, performance or hard-cutover qualification is claimed. + +## CG-04 fixed node SQL pool + +Qualified source: `51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1`. + +`CellGroupSqlPool` now uses the existing fixed blocking worker pool and fairness +queue. Requests have typed logical cell keys; the old host adapter remains +separate during the staged cutover. The node creates no worker per enrolled cell. +A cell's admission lease remains with queued work, its physical callback and its +unresolved proposal, including when a caller cancels. A lost stage response can +resume the retained proposal without running another callback. Exact settlement +releases that cell; unrelated group progress cannot do so. + +Node/group quotas reserve capture capacity before queueing. Once capture ends, +the reservation shrinks to the retained entry/result bound while keeping its +operation slot. Prepared-response clones and receipt delivery retain that charge. +A separate progress slot per worker permits settlement without needing ordinary +queue admission. Extent workspaces are created and destroyed on the admitted +blocking worker, so rejected requests cannot release their last physical workspace +on an async executor. + +The pool validates the exact engine behind execution authority and claims an +exclusive SQL lease on that engine. Its cleanup task owns every worker join; +cancelling a close waiter cannot release a blocked callback. The engine lease +also survives through retained response admission, preventing a replacement pool +from bypassing live output reservations. Queue/stage/publication timers use the +existing node performance registry. + +All 184 selected Rust tests pass: 168 native cases across seven suites, 11 SQL +queue/extent-worker unit tests and five budget tests. Both new pool cases use +actual three-voter replication and cover blocked/cancelled callbacks, independent +cell progress, group quota refusal, exact resume, duplicate receipts, engine +fencing, cancelled close and retained-response ownership. Four process helpers +are invoked by parent tests; three RustFS-specific cases remain explicitly unrun. + +All-target Clippy with warnings denied, formatting (1,794 Rust files), module +layout, dependency isolation, docs, reviewed source preflight and all 37 plan +validator tests pass. The partial checkpoint at +`.cellar/cell-groups-evidence/CG-04-sql-pool/checkpoint.json` binds 178 source inputs, +ten command logs and eight copied executables. Intermediate logs retain the +initial verification-export compilation error and direct-Tokio-select Clippy +failure, as well as passes preceding the final ownership refinements. Only the +final gate qualifies the committed source; no task status advances here. + +Registered SDK dispatch, concurrent enrollment, strong-read serving, preview/cache +lifecycle and receipt reconciliation after leadership changes or compaction +still need integration. +Closing SQL does not prove an ambiguous submitted command failed; the owning host +must reconcile against committed receipts before admitting a new producer. CG-04 +and the full hard-cutover goal remain incomplete. + +## CG-04 per-cell SQL producer + +Implementation source: `f3d48bdc`; qualified successor: +`f807b2f70e01e3cd8c741557889e1965fd14ec49`. + +The synchronous `CellPrivateExecutor` executes one cell's private SQLite and +receipt transaction, captures its physical LTX update and encodes the actual +cell-addressed group entry. Materialized and extent-backed images share the +existing receipt admission, maintenance and transaction implementation. Physical +capture no longer requires a synthetic cell-scoped group identity. + +The real leader issues an authority pin for the selected cell, route and term. +Another cell can advance without invalidating that base. Leadership loss, owner +failure or owner drop revokes the observation; the producer checks again after +capture. Settlement requires fresh authority, the exact committed entry bytes +and expected cell image/version. Duplicate receipts come from the selected +immutable cell image without another callback. An unresolved proposal remains +busy; closing its writer does not establish a failed proposal outcome. + +The two new real-three-voter cases use the same client/sequence in two distinct +cells, commit B before A, refuse settling A with B's authority, then verify each +receipt and independent SQLite value on all voters. Both image modes also stop +the consensus owner while a SQL callback is blocked, then require a typed +Unavailable refusal and unchanged selected cell version after release. + +All 133 selected native cases pass: 13 shared execution, 15 catalog, 15 active +storage, 21 prior execution, 50 rooted and 19 prior runtime cases. Four ignored +process helpers are invoked by their parent tests; three explicitly ignored +RustFS cases were not run. All-target Clippy with warnings denied, formatting +(1,790 Rust files), layout, native dependency isolation, docs, reviewed source +preflight and all 37 plan-validator tests pass. The partial record at +`.cellar/cell-groups-evidence/CG-04-cell-sql/checkpoint.json` binds 173 source inputs, +eight successful command logs and six copied executables. It retains preliminary +runs and the initial two-fixture Clippy failure; the successor removes only an +unnecessary result wrapper and borrow, without changing assertions. + +The fixed SQL pool, fair per-cell admission, concurrent initialization, +registered SDK dispatch and coherent read serving still require integration. +The producer currently reopens after settlement and conservatively refuses +exact-entry settlement after compaction; receipt reconciliation, preview +retention and cache lifecycle must be integrated with the owning host. CG-04 +remains IN_PROGRESS. No full cutover, RustFS drill or performance qualification +is claimed by this checkpoint. + +## Earlier plan delivery and SQL draft handoff + +The implementation package contains 16 dependency-ordered tasks, 86 required +cases, 14 invariants and nine qualification profiles. Each task specifies owning +paths, implementation steps, acceptance oracles and executable test commands. +The plan runner prints those commands and checks evidence; it does not execute +the implementation or provision infrastructure. + +At plan delivery the working tree retained an unfinished per-cell SQL producer, +leader-issued execution authority, shared receipt helpers and tests for both +image formats. That draft has since been committed and qualified at the partial +SQL checkpoint above. The plan delivery itself advanced no task status or +runtime qualification. + +Plan delivery checks pass: package validation, reviewed source preflight, all +37 validator unit tests and repository documentation validation (113 documents, +2,314 local references). `show CG-04` emits the full executor brief. The release +command correctly refuses incomplete implementation evidence. The branch +contains the latest fetched `origin/main` at `8a93ffd7`; neither requested +MR-13/MR-14 progress JSON exclusion differs from main. + +## CG-04 attachment admission and main integration + +Source: `aec4029e77124848dc6b155c098f8eec9ccdacc4`. + +Attachment now reserves ordinary node capacity before catalog revalidation and +RawNode construction, retaining it through dispatch or failure. A public-runtime +regression holds a real vote's engine sync and fills the sole request slot; a +second attachment refuses with overload before claiming an owner. Releasing sync +allows the queued protocol work and shutdown to finish at the same limits. +The first fixture attempted the normal multi-peer pump with one slot and refused +transport overload before the target assertion; the final fixture isolates the +startup boundary without relaxing admission. + +The branch includes `origin/main` at `8a93ffd7`. Its documentation consolidation +moves this package to `docs/design/multi-raft/cell-groups/`. Commands, links, +catalog entries, source-root discovery and the plan-only release-input exclusion +now follow that location. Validator tests exercise both the new design path and +historical `plans/` bookkeeping. Existing evidence retains its original source, +paths and hashes; moved documentation is not fabricated runtime qualification. + +All 60 selected Rust cases pass: 11 execution, 15 catalog, 15 active storage +and 19 active existing-runtime cases. All-target Clippy with warnings denied, +formatting (1,786 Rust files), layout, dependency isolation, documentation, +source preflight and all 37 plan-validator tests pass. The partial checkpoint at +`.cellar/cell-groups-evidence/CG-04-runtime-admission/checkpoint.json` retains +165 source inputs, eight command logs, four executables and intermediate failure +logs. The larger source inventory includes main's documentation/tooling changes. +CG-04 remains IN_PROGRESS; this does not qualify per-cell SQL integration, SDK +serving, real RustFS recovery, performance or the final hard cutover. + +## CG-04 shared-node runtime + +Source: `865f253da0a14d730b62f77b17cefc6d1497890b`. + +One execution-domain task now multiplexes physical cell-group owners on each +node. It uses the existing engine, mailbox and admission primitives, with a +single in-flight persistence batch containing writes from multiple group +histories. Exact engine receipts complete the corresponding owner tickets. +Group turns and deferred transport/proposal work remain bounded; ticks coalesce. +An admitted transport frame is not a persistence or application acknowledgement. + +Apply events retain admission through image preparation and physical root +publication. Completion requests have reserved capacity. Already-started physical +phases can finish when ordinary event delivery is full; one reserved settlement +slot permits releasing an apply reservation without admitting more ordinary +output. A two-slot test exposed the original circular wait, and the fix changes +phase progress rather than increasing the limit. + +Shutdown closes admission and joins submitted engine work before releasing node +runtime ownership. Cancelling the caller's close waiter cannot release that +ownership. The existing runtime shares the terminal completion primitive with +the new worker; its behavior remains covered by its native regression suite. + +All 59 selected Rust cases pass: ten execution, 15 catalog, 15 active storage +and 19 active existing-runtime regressions. Two process helpers are ignored by +direct execution and exercised by their parent tests. All-target Clippy for +`cellar`/`cellar-logic` with warnings denied, formatting (1,786 Rust files), layout, +native dependency isolation, docs, source preflight and all 37 plan-validator +tests pass. The retained partial record is +`.cellar/cell-groups-evidence/CG-04-runtime/checkpoint.json`, binding 85 source +inputs, eight command logs and four native executables. The initial two-slot +failure and successful intermediate repeats are retained without claiming a +source-bound intermediate qualification. CG-04 remains IN_PROGRESS. + +This is the shared consensus/persistence scheduler. Per-cell SQL preparation and +fair admission, concurrent initialization, exact receipt settlement and SDK read +serving still need integration. The full CG-04 gate remains required. Complete +group snapshots, membership/topology, managed startup, RustFS qualification and +the final hard cutover remain unfinished. + +## CG-04 physical consensus owner checkpoint + +Source: `51e30eabde1ca76bf0b23963dd06d0e8130f8ee7` (successor to `0951440e`). +CG-04 remains IN_PROGRESS. The physical owner now attaches one real RawNode to +each group replica using the catalog's initial-member permit and a completely +verified selected root. The permit is revalidated before attachment. Each +GroupLog shares an exclusive owner lease; persistence requests retain it until +physical engine work finishes, even if the owner and its awaiting future are +dropped. A duplicate attachment cannot race an outstanding write. + +Ready and LightReady persistence completes before dependent messages, committed +application or read barriers are released. Image preparation is an external +effect and cannot advance Raft apply. Completion must match the exact owner, +entry digest/position, preceding root, engine and page-store owner. The new root +is published through the existing engine transaction before advancing the +contiguous prefix. No-ops preserve all cell versions. ReadIndex output remains +a barrier, not SDK serving authority. + +The local three-voter harness replicates three independent SQLite histories and +interleaved LTX updates. It holds both followers' first apply while two entries +commit, rejects foreign and incorrect-image completions, then restarts one +follower and replays its committed tail. The restart joins the engine; it is not +a process-kill or power-loss test. Separate tests exercise minority read refusal, +cancellation while a real log sync is held, and 40 versions with a backpressured +follower that later reaches the exact final SQLite bytes. A peer counter-boundary +regression rejects input before changing the owner's term or applied state. + +Ready collection accounts for small-entry overhead and the two in-flight batches' +maximum entries. A saturated apply queue backpressures new protocol requests. +Ticks coalesce behind pending Ready work so a stalled sync does not accumulate +unbounded heartbeat/election messages. Large physical effect and phase payloads +are boxed, reducing the space paid by every lightweight event. + +All 36 selected Rust cases pass: six physical-owner cases, 15 catalog cases and +15 active storage cases. The ignored process fixture is run by its parent crash +test. All-target Clippy for `cellar`/`cellar-logic` with warnings denied, +formatting (1,783 Rust files), layout, dependency isolation, docs, source preflight +and all 37 plan-validator tests pass. The retained partial record is +`.cellar/cell-groups-evidence/CG-04-owner/checkpoint.json`, binding 82 source inputs, +eight successful command logs and three executables. It is not a CG-04 DONE record. + +The initial owner fixture compilation and pre-vote persistence +expectation failed and are retained. Pre-vote has no durable vote; the corrected +sync test holds an actual data append after election. An initial Clippy run +rejected oversized enum variants; the implementation fixes their layout without +suppressing the lint. That source-bound failed gate remains at +`.cellar/cell-groups-evidence/CG-04-owner-failed-0951440e/gate-commands.json`; +earlier uncommitted fixture failure logs are retained under the final checkpoint's +`intermediate/` directory and are not treated as source-bound qualification. + +The existing shared scheduler and SQL worker still consume the prior single-cell +owner. Next, integrate this physical owner with those node resources, bounded +per-cell producers, fair admission, exact receipt settlement and coherent reads. +The complete CG-04 six-case gate remains required, including real enrollment +concurrency and callback ownership. Snapshot/configuration/topology requests +currently refuse; their complete-group protocols, managed startup, RustFS, +independent failure domains and final hard cutover remain unfinished. + +## Plan delivery and working-state audit — 2026-10-29 + +The package has 16 tasks, 86 required cases, 14 invariants and nine qualification +profiles. Its contract/evidence validation and CG-04 brief generation pass. +Freshly fetched `origin/main` is `1be27d234d4674d4d014785c60cecbfe394bdab9`, +already an ancestor of the delivery branch. The supplied TiKV checkout still +matches the investigation pin. Neither excluded MR-13/MR-14 progress JSON file +appears in the branch diff. + +There is additional uncommitted CG-04 owner work in this checkout. It is not +part of the reviewed baseline or completed evidence. The new execution test +target currently fails compilation: `GroupRoute` is ambiguous through glob +imports and the fixture calls the private `ImageCommitment::sha256` method. +The global preflight and its unit test refuse the changed `cache.rs`; the other +36 validator tests pass. Preserve this refusal until the implementation is +reviewed and a successor baseline is recorded. Do not reset those edits or +silently update hashes to make the handoff appear qualified. + +In a separate checkout of `fde3e6f9` with this plan update and the three retained +DONE evidence directories, source preflight and all 37 validator tests pass. +The release command exits nonzero with `implementation tasks/evidence incomplete`, +as required. Repository docs validation passes with 869 local references. These +checks qualify the committed plan handoff; they do not qualify the dirty runtime +prototype. An initial export-based validation attempt had incorrect Git isolation; +its temporary local metadata changes were restored before this independent rerun. + +The [handoff](handoff.md) separates this working state from the committed +starting point and lists the remaining CG-04 acceptance cases. No runtime task +is advanced by this plan delivery. + +## CG-04 durable enrollment checkpoint + +Source: `c187c47773fd22d4658de1d912b6096f92a8e26d` (successor to `9f937f22`). +CG-04 remains IN_PROGRESS. + +Each authenticated cell leaf now requires the original creation operation, +logical cell history, schema and initial image commitment. Publication includes +that record in the same complete root as the current image. Mutations refuse a +changed record; sequence one must match the initial image and later versions +advance its LTX position. Page decoding rejects missing/foreign creation records +at marker 1. No old-layout reader or inferred enrollment is introduced. The +physical pin-tree budget includes the record's additional cell identity. + +Ordered enrollment preparation checks the route, preceding complete root, +contiguous apply position and committed lifecycle phase before invoking a physical +image producer. Exact retries keep the latest image, receipts and cell version +and advance only the group prefix. This unchanged-root path reuses image/page +pins without page I/O and refuses quarantined stores. Engine publication still +requires the exact preceding branch and committed position. + +Native recovery exercises actual SQLite/LTX and Raft Engine in materialized and +extent modes. It mutates a cell, restarts, reclaims the original image, then +replays enrollment without invoking the image producer. The latest values and +receipts survive another restart, while the neighboring cell remains unchanged. +Sealed/retired ranges, competing creations and invalid apply positions refuse. +Existing process-kill cases also verify creation identity after recovery. + +The new page-wire tests preserve canonical JSON ordering when corrupting identity +fields, so format ordering cannot mask a missing identity check. A quarantine +fixture fails the actual manifest directory-sync barrier and checks fault +consumption. Its initial run selected an image-file-only fault and failed because +no page I/O was interrupted; that failed run is retained separately. + +All 32 selected Rust cases pass at this source: 15 catalog cases, 15 active +storage cases and two page-wire unit cases. The ignored process fixture is run +by its parent crash test. All-target Clippy for `cellar`/`cellar-logic` with warnings +denied, formatting (1,781 Rust files), layout, dependency isolation, repository +docs, source preflight and all 37 plan-validator tests pass. The retained partial +record is `.cellar/cell-groups-evidence/CG-04-enrollment/checkpoint.json`, binding +79 source inputs, nine successful command logs and three executables. +The failed fault-injection run remains under +`.cellar/cell-groups-evidence/CG-04-enrollment-failed-9f937f22/`. + +This is an ordered preparation/storage checkpoint. The existing `Replica` and +shared worker still consume the staged single-cell log/apply types. The next +implementation must connect physical `GroupHistory` and complete-root application +to that real consensus owner, then wire independent SQLite execution, fair +admission, receipts and reads. The six CG-04 real-Raft acceptance cases remain +required. No live shared owner, RustFS, load or hard-cutover qualification is +claimed by this checkpoint. + +## CG-03 completion audit and CG-04 continuation + +The audit checks the four CG-03 steps against retained evidence at +`0df009a24d88408a8efc40e356f794ca35513a95`: + +- Paged directory tests exercise exact coverage, bounded discovery, conditional + publication and recovery from lost replies using real conditional operations. +- Enrollment tests check pure creation identity and sealed/retired admission; + the task explicitly assigns real SQLite enrollment to CG-04. +- Native member bootstrap exercises synced physical genesis, permanent claims, + restart, unknown outcomes and refusal after claimed local records are removed. + Storage boundary tests separately verify physical identities and inventory. +- Route tests check trusted-catalog confirmation, cached operation during catalog + failure, original deadlines and attempt budgets, cancellation and command ambiguity. + +The full required command gate, source/log/artifact hashes and dependency evidence +validate. CG-03 is DONE only at that catalog/routing/admission-preparation boundary. +The record retains 77 source inputs, 20 successful commands, three executables and +44 distinct Rust cases. It does not qualify live shared dispatch, network peer +authentication, managed startup, current-term serving authority or RustFS. + +CG-04 now addresses original enrollment identity in the durable manifest, followed +by shared group ownership and execution. Its full six-case real-Raft execution +gate remains required; storage-only tests cannot substitute for that gate. + +## Plan handoff at b3b7eabd + +The package contains 16 dependency-ordered tasks, 86 required cases, 14 invariants +and nine qualification profiles. Each task supplies source anchors, owned paths, +implementation steps, exact test commands and acceptance oracles. CG-01 and CG-02 +retain their historical DONE evidence. CG-03 remains IN_PROGRESS pending the +implementation executor's completion audit; CG-04 through CG-16 remain TODO. +This handoff does not advance runtime status merely because a plan validates. + +The reviewed source is `0df009a24d88408a8efc40e356f794ca35513a95`. Existing routing +sessions now retain the original logical request, command identity, deadline and +attempt budget across retries. Relocation requires exact confirmation against the +configured trusted catalog; cached requests need no catalog read. Cancelled +lookup cannot refund an attempt, and a possibly submitted command retains its +ambiguous outcome. Catalog confirmation does not authenticate a network peer or +grant serving authority. Actual shared dispatch remains CG-04/CG-07 work. + +The retained record at +`.cellar/cell-groups-evidence/CG-03-routing-final/record.json` binds 77 source +inputs, 20 successful command logs, three test executables and 44 distinct Rust +test cases to that revision. Its gate includes all-target check and Clippy, +formatting, layout, dependency isolation and docs checks. All 37 plan-validator +tests pass. The earlier Clippy rejection of a forbidden test macro is retained; +the final source replaces it with a permitted selector without suppressing the +lint. These are existing local qualification results, not a RustFS drill or a +shared runtime performance claim. + +For this handoff, package validation, reviewed-source preflight, validator tests, +executor-brief generation and repository docs checks are rerun. The release gate +must still refuse incomplete implementation. The branch includes freshly fetched +`origin/main` at `1be27d234d4674d4d014785c60cecbfe394bdab9`; the supplied TiKV checkout +matches the investigation pin. The requested MR-13/MR-14 progress JSON files +remain excluded from the PR. + +## CG-03 physical bootstrap and inventory checkpoint + +Source: `5d0d8e5ede3b4efedc211c40122a1a26b43968b4`. CG-03 remains IN_PROGRESS. + +Physical handles now carry `GroupHistory` independently of application cells. +Real-engine genesis atomically syncs the mapping, initial membership and complete +empty group root. The same persistence worker owns both typed storage paths, +quotas, sync and cache publication. Required mapping application domains refuse +old untagged records; an engine cannot mix legacy and cell-group application +families. Existing single-cell consumers remain staged until the final cutover. + +The initial unmanaged-member bootstrap API verifies the current selected birth, +permanent node-storage pin and exact member claim after actual genesis sync. +Claims bind group history, birth, member, node, disk and engine instance. Unknown +confirmation grants no permit; same-identity restart reconciles it. Tests remove +one owned group's records while retaining the original engine identity, then +verify that the permanent claim refuses fresh genesis. A separate blank-engine +fixture exercises the node pin. Foreign nodes/catalog owners and retired routes +also refuse; managed admission still requires its later rooted fleet integration. + +Physical inventory now captures every shared history, including unattached or +retained members, and an optional initializing slot. A domain-separated digest +binds all four physical IDs without a cell name. The bounded existing wire checks +the complete occupied count; an already durable pending slot counts once. Legacy +cell-scoped lookup cannot match a group-native row. Real-engine tests verify +inventory across restart and while genesis sync is held, including wire refusal +when the node count omits a slot. + +All 216 distinct Rust cases pass across selected genesis, physical inventory, +membership, core, directory, catalog, manifest, host, snapshot and storage suites. +The shared manifest recovery fixtures now use actual cell-group genesis for both +image representations and process-kill cuts. The RustFS-specific snapshot case +remains explicitly ignored; child fixtures are invoked by their parent tests. +These results do not qualify a live shared-group host or independent failure +resilience. Source preflight, all 37 plan-validator tests, formatting, layout, +dependency isolation and repository docs checks pass. The dependency script's +initial missing-PATH failure is retained separately from its successful rerun. + +All-target Clippy for `cellar` and `cellar-logic` also passes with warnings denied. +The partial evidence record is retained at +`.cellar/cell-groups-evidence/CG-03-bootstrap/checkpoint.json`, binding 76 source +inputs, 14 successful command logs and eight executables to the source revision. +This is a checkpoint, not a CG-03 DONE record. + +A catalog bootstrap permit is not complete image recovery or serving authority. +The runtime still must verify the selected image/manifest closure, revalidate its +startup permit and obtain current-term quorum. Authenticated bounded redirects, +the shared runtime, managed fleet integration and CG-04 through CG-16 remain +required. No RustFS disaster drill, load result, performance improvement or full +CG-03/release qualification is claimed by this checkpoint. + +## Plan handoff at 98aff95a — 2026-10-29 + +The package supplies the target architecture, protocol state machines, dependency +ordered executor briefs, exact test commands and source-bound acceptance gates. +CG-01 and CG-02 retain their historical DONE evidence, CG-03 is IN_PROGRESS, and +CG-04 through CG-16 remain TODO. The final release gate remains closed. + +The branch includes freshly fetched `origin/main` at +`1be27d234d4674d4d014785c60cecbfe394bdab9`. The supplied TiKV checkout matches the +investigation pin `b3995dab28308f09cbb7c125bf32f789dbc203fe`. + +Uncommitted physical bootstrap, replica-handle and permanent member-claim changes +remain in this checkout, including two new Rust files and native test changes. +They are preserved for the next implementation executor and are not qualified by +this documentation handoff. Review their full diff and test results before +committing them or advancing the source baseline. Do not restart this work from +the earlier metadata-only checkpoint. + +Package structural validation passes: 16 tasks, 83 required cases, 14 invariants +and nine profiles. Working-tree source preflight correctly refuses drift in +`crates/cellar-logic/consensus.rs`; 34 of 37 validator tests pass, with the three +source-sensitive checks failing against that unfinished source. These failures +are not waived. The unchanged committed source is checked separately from the +draft runtime. + +Handoff verification: a temporary read-only export of committed source +`aead5cc04a9788ab2f2ee5f3fba3d6468fde5b75` passes the source preflight function and +all 37 plan-validator unit tests. This export excludes the draft runtime and +does not requalify historical DONE evidence. Working-tree structural validation, +CG-03 brief generation, repository documentation checks and `git diff --check` +pass. `release` exits nonzero for incomplete implementation evidence, as required. +The requested MR-13/MR-14 progress JSON files have no diff from `origin/main`. + +## Plan handoff at 462aa2eb — 2026-10-29 + +The package describes the complete replacement architecture, ordered implementation +briefs and source-bound qualification gates. CG-01 and CG-02 retain their existing +DONE evidence; CG-03 remains IN_PROGRESS and CG-04 through CG-16 remain TODO. +The next executor starts with physical group identity, real genesis/member-claim +integration and bounded redirects, then follows the task graph. This handoff adds +no runtime implementation or new runtime qualification claim. + +The branch includes fetched `origin/main` at +`1be27d234d4674d4d014785c60cecbfe394bdab9`. The supplied TiKV checkout remains at +`b3995dab28308f09cbb7c125bf32f789dbc203fe`. Existing local DONE evidence must +accompany a fresh checkout to validate those checkpoints. Final release remains +closed; a validated plan does not prove shared-group performance or recovery. + +Handoff checks pass: package validation (16 tasks, 83 required cases, 14 invariants, +nine profiles), reviewed source preflight, all 37 validator unit tests, repository +documentation checks and `git diff --check`. The CG-03 executor brief renders +successfully. `release` exits nonzero as expected because implementation evidence +is incomplete. Runtime suites were not rerun for these plan-only edits. + +## Historical plan handoff — 2026-10-29 + +This section records the state at `dcc52840`; the directory checkpoint below +supersedes its working-diff and source-preflight limitations. + +The package validator passes: 16 tasks, 83 required cases, 14 invariants and nine +profiles. Task briefs include source excerpts, file scope, dependencies, exact +commands and evidence requirements. The TiKV investigation pin still matches the +supplied checkout. The branch includes freshly fetched `origin/main` at +`1be27d234d4674d4d014785c60cecbfe394bdab9`. + +Uncommitted CG-03 directory work remains in this checkout: immutable range pages, +root publication and mutation journals, with native catalog tests. The observed +native run passes seven tests. This is not a complete CG-03 qualification and is +not included in a DONE record. Preserve the working diff for its next executor. + +The source preflight correctly refuses the draft `catalog.rs` module exports +against the recorded baseline. Consequently the current-checkout validator suite +passes 36 of 37 tests; its actual-source preflight test refuses that drift. Do not +blindly refresh the baseline or mark CG-03 complete. Review and qualify the draft, +commit its source, then record a successor baseline with updated excerpt lines. +Birth/disk-claim integration and bounded redirects remain outstanding. The final +release gate remains closed. + +## CG-01 — complete at the contract boundary + +Qualified source: `ee7f4a8d2dec220904f5a6860041f12c14db1cf5`. + +- Dependency-free cell/group histories, validated cell IDs, monotone counters, + canonical routing hashes and collision-safe half-open ranges. +- Logical cell versions and signed tokens independent of physical log position. + Normal transfer preserves history; disaster recovery changes incarnation. +- Materialized and extent commitment adapters with geometry validation. +- A bounded, domain-separated entry codec for cell-addressed mutations and typed + enrollment, membership, checkpoint, seal/install/activate, park and retire. + Enrollment carries replayable initial LTX bytes; decoding borrows payloads. + +The [entry codec contract](../../../../crates/cellar/application/consensus/group_protocol/README.md) +documents wire layout and checks that remain the runtime owner's responsibility. +The contracts are exposed through the unstable verification surface. Production +consumers still use the old contracts until CG-07; legacy removal follows in CG-16. + +The retained source-bound record is +`.cellar/cell-groups-evidence/CG-01-ee7f4a8d2dec/record.json`. It binds the tested +Git revision, source inputs, native test executable and all 13 command logs by +hash. The complete gate passed: + +- All-target checks and Clippy for `cellar` and `cellar-logic`, with locked builds + and Clippy warnings denied. +- All four required CG-01 cases selected individually by exact name, plus test + inventory verification to reject an empty selection. +- All 7 native cell-group contract cases, 6 existing native Raft contract cases + and 18 core library cases: 31 distinct Rust tests. +- Repository formatting, module layout, native dependency isolation and docs. + +Tests include SQLite reconstruction from enrollment bytes, an independent wire +vector, truncated/oversized metadata, cross-cell/history refusal, token tampering +and transfer versus disaster identity. These are contract tests, not proof of a +running multi-cell group, a RustFS drill or a performance claim. + +The earlier short-ID token-bound failure and intermediate logs remain under +ignored local evidence. The corrected regression passes. Code review also caught +and removed a root-only enrollment gap before the final gate: nonempty enrollment +must include its initial replayable LTX chain. + +## Remaining implementation + +CG-01 through CG-03 have passed their scoped task gates. CG-04 is IN_PROGRESS; +CG-05 through CG-16 remain TODO. Historical entries below retain the intermediate +checkpoints and their limitations. + +The reviewed source baseline is the CG-04 enrollment checkpoint at `c187c477`. +A new executor must receive all three ignored DONE-task evidence directories +before validating those historical checkpoints. Current CG-04 source changes +remain subject to their own verification and recorded review. + +No shared runtime, split/merge, hibernation latency, real RustFS recovery, measured +density or final hard cutover is qualified. The release gate remains closed until +all task and deployment evidence passes at one final revision. + + +## CG-02 — manifest pages in progress + +The first storage slice implements a bounded immutable AVL manifest and an +exclusively owned filesystem page store. Leaves bind logical versions, geometry, +last mutation and effect summaries; local image filenames do not affect roots. +Updates compare the old cell version and copy only the changed tree path. +Page publication syncs files and directories and quarantines uncertain writes. + +The new native storage target uses real SQLite/LTX and sealed generation pins. +All three current cases pass: path-copy bounds across 64 independently sealed +SQLite cells, unchanged image commitments and old-root reads; restart and +malformed/missing page refusal; conflicts/capacity, replica-independent roots and +injected page-directory sync failure. These are page-store checks; the complete CG-02 +gate has not run. The first build failed on an unavailable error derive and a +follow-up build caught argument mismatches. The first running suite exposed a +missing link-aware metadata method in the fault fixture; it now delegates to the +real filesystem instead of weakening the page-store inventory check. + +The engine root marker, image adapters for both representations, physical root +pins/reclamation and actual engine crash-cut tests remain outstanding. No +manifest can yet grant shared-group serving authority. The [component contract](../../../../crates/cellar/durability/raft/cell_groups/README.md) +records this boundary. Checkpoint logs are retained under +`.cellar/cell-groups-evidence/CG-02-manifest-pages/`. The CG-02 scope includes the consensus module's unstable +verification exports so the native tests exercise the actual component. + +Clippy passes for the library and new native target with warnings denied. The +36 plan-validator tests, source preflight, native dependency isolation and docs +checks also pass. These checks do not substitute for the still-missing CG-02 +engine publication, pin reclamation and both-image-mode qualification cases. + + +### CG-02 engine publication checkpoint + +Complete-root preparation now checks each leaf against an actual image pin, +syncs the manifest closure and holds image pins through engine persistence. The +root binds group history, range/epoch, birth/schema, membership revision and the +exact applied position. Raft Engine atomically selects that root with log/hard +state and publishes its cache only after sync. Legacy image accessors refuse a +shared-group root, and normal apply cannot skip an index or change its route. + +The local application record uses explicit marker-1 image/group domains; old +bare-image record shapes refuse. No disk migration or deletion is performed. +The existing image variant remains a staging contract until the public cutover. + +New native tests exercise a stalled real-engine sync, restart, missing-cell-image +refusal and rejection of preparation from a previous engine instance. Five +manifest cases and the 19-case existing storage suite pass, including the +engine-instance check. All-target Clippy passes after boxing the larger metadata +variant; the failed lint output is retained. Final checks are recorded under +`.cellar/cell-groups-evidence/CG-02-engine-root/`. +The standalone storage child-process fixture is intentionally ignored and is +launched by its parent persistence tests. + +Incremental apply preparation, durable root/pin reclamation, root-specific process +kill cuts and full extent-mode recovery tests remain CG-02 work. The production +shared-group host, topology and final release gates remain pending. + + +The engine-root checkpoint passes all-target Clippy, formatting and module layout, +24 directly selected native tests, dependency isolation and docs checks. The +reviewed source baseline is `4387fedd`. All 37 plan-validator tests pass. Updating +the baseline exposed a verifier bug: historical evidence was compared backward +against the moving review baseline. Evidence now covers every runtime change +since the frozen plan starting revision. A regression test ensures later baseline +reviews neither invalidate historical inputs nor waive earlier changed files. +This correction does not make CG-02 DONE or relax the final release-source gate. + +### CG-02 incremental preparation checkpoint + +Ordinary cell mutations now prepare against a complete prior root without scanning +or copying all cell pins. The physical pin tree mirrors the authenticated AVL +manifest, shares unchanged subtrees and copies changed paths/rotations. Each +mutation checks the changed image, expected cell version and next group position. +Page reads and allocations are logarithmic; lookup comparisons are bounded by +O(log² cells). Root clones are constant-time and memory admission charges the +complete owned tree, including node and cell-ID allocations. + +The engine checks the exact base root, so a staged successor from a competing +branch cannot replace the selected branch merely because its index is next. +Identical already-selected roots remain retryable. Capabilities also bind the +page-store instance; reopening requires complete preparation. A coherent root can +return a selected cell leaf plus an independent physical image pin. + +Native cases cover one update among 64 real SQLite cells, bounded page reads and +writes, competing-branch refusal, identical retry, page-owner replacement, and +96 incremental enrollments across AVL rotations. Reclamation checks keep the old +image while its reader survives, reclaim it after release, and retain all 64 +selected images through the engine's root owner after caller handles are dropped. +The first incremental run exposed an overly strict retry base check; the corrected +run is retained beside that failure under ignored local evidence. + +This is not the complete pin/reclamation gate. Page reclamation, authoritative +retention across uncertain engine outcomes, root-specific process-kill cuts and +full extent-mode recovery remain CG-02 work. The production shared-group host and +the full hard-cutover goal remain pending; CG-02 stays IN_PROGRESS. + +Runtime checkpoint `b71f62b5` passes all 7 native manifest cases and 19 existing +Raft storage cases, all-target Clippy with warnings denied, formatting, layout, +native dependency isolation and docs checks. All 37 plan-validator tests and the +reviewed source preflight pass. The initial source-drift refusal correctly required +review of the changed cache anchor before advancing the baseline. Source and +executable hashes, failed/corrected test output and final check logs are retained +under `.cellar/cell-groups-evidence/CG-02-incremental-pins/`. + +### CG-02 reclamation and uncertain publication checkpoint + +Prepared pin-tree nodes now lease their physical manifest pages and keep the +directory lock alive. Bounded collection scans an explicit inventory window, +removes only unleased pages, and refunds page/byte quotas only after the directory +sync. Remove or sync failure quarantines the owner without speculative refunds. +Every reopened store requires root inventory reconciliation before collection. + +The engine now reserves old/candidate closures before writing and selects the new +retention reference only after sync and cache publication. Dropping an armed +reservation on failure keeps both page/image closures and blocks collection until +recovery; pre-write refusals cancel unarmed reservations. The page owner retains +uncertain images even after request and engine handles disappear. Complete +recovery automatically retains a root only when it matches the selected engine +marker. Additional snapshot/archive/transition roots have an explicit bounded +retained set, independent of disposable caller handles. + +The owning shared host must still supply the complete catalog/checkpoint/transition +inventory to `reconcile_retained`. This is a trusted owner contract; page filenames +cannot discover authority, and these storage fixtures do not qualify that future +host integration. CG-02 also still requires the root-specific process-kill matrix +and full extent-mode recovery qualification. It remains IN_PROGRESS. + +Runtime checkpoint `f41f470f` passes 11 native manifest cases and 19 existing Raft +storage cases. New checks cover independent snapshot/transition retention, +collection during stalled apply, quotas after removal/barrier failures, and +two-group batch refusal before any write. A real failure injected after fsync +retains both images after request/engine handles drop; restart selects the new +durable root before reclaiming the superseded image/page. + +All-target Clippy with warnings denied, formatting, layout, native dependency +isolation, docs, source preflight and all 37 plan-validator tests pass. The first +test build required owned fixture paths; the first lint run required replacing +an unnecessary clone used to construct a one-element slice. Neither fix weakened +an assertion. Source hashes, copied test executables and failed/corrected logs +are retained under `.cellar/cell-groups-evidence/CG-02-reclamation/`. This is a +storage checkpoint, not the complete CG-02 evidence gate or a RustFS qualification. + + +### CG-02 — complete storage qualification + +Qualified source: `ec9d277e50416f786fe3c1cd851b4147f9cd4b43`. +The retained record is `.cellar/cell-groups-evidence/CG-02-ec9d277e5041/record.json`. +It binds 45 source inputs, 16 successful command logs and three copied native +executables. All five required cases were present in the inventory and each +passed when selected individually. The complete gate includes all-target checks +and Clippy, formatting/layout, dependency isolation, docs and 37 validator tests. + +The 14-case manifest suite covers both materialized and extent-backed SQLite +images. Thirty actual child-process kills span image and manifest writes/sync, +engine sync, publication and reclamation. Recovery selects a complete old or new +root, checks both cells' independent values and receipt rows, and verifies that +selected data remains readable after orphan reclamation. Missing or corrupt +manifest pages, sibling images, extent roots and non-header extents refuse +complete recovery. Extent updates also preserve an unchanged shared tail extent. + +All 19 existing Raft storage cases and the generation-store image sync regression +pass. The latter exercises five additional child-process cuts with the shared +fault adapter's original behavior. Standalone child fixtures remain intentionally +ignored by direct suite execution and are launched by their parent tests. + +This completes CG-02 at its storage boundary. Process kills do not simulate power +loss. Receipt rows are physical-state checks, not SDK deduplication qualification. +The shared host must still integrate authoritative root inventories, admission, +quorum, snapshots and archives. No RustFS drill, split/merge, hibernation latency, +density claim or final hard cutover is qualified by these tests. CG-03 is next. + + +## CG-03 — enrollment admission in progress + +The first CG-03 slice adds deterministic admission using the existing bounded +cell-addressed entry codec. Requests verify group history, route epoch, birth, +schema and the cell's half-open range. Sealed or retired lifecycle state refuses +delayed enrollment. Concurrent preparations must be checked again in committed +log order; only the selected original creation record can resolve a retry. + +The bounded canonical `EnrollmentRecord` binds logical cell history, original +creation operation, schema and initial image commitment. It survives normal +transfer without binding to the old physical group. The host must retain it +atomically with enrollment, preserve it through mutations and include it in +snapshots/archives. Returning an existing record grants no quorum/read authority +and must never restore the initial image over newer state. + +Three native admission cases pass: creation conflict/retry and route identity, +sealed/retired refusal, and bounded canonical record decoding. Negative record +checks preserve field order so rejection cannot be attributed only to JSON map +reordering. Empty records require the canonical empty image and zero checksum. +The initial build required explicit image wire fields; an intermediate running +suite exposed the absent empty-image checksum check. Failed and corrected logs +are retained with the checkpoint rather than suppressed. + +This is a pure admission boundary. The paged range tree and conditional namespace +root journal, lost birth/root response reconciliation, permanent blank-disk +refusal integration and bounded route discovery remain CG-03 work. CG-04 must +connect admission and durable creation records to the real shared group host. +CG-03 remains IN_PROGRESS; the production hard cutover is not complete. + + +Enrollment checkpoint `b1d27582` passes its three native cases, all-target Clippy +with warnings denied, formatting/layout, native dependency isolation, docs, +source preflight and all 37 plan-validator tests. Evidence under +`.cellar/cell-groups-evidence/CG-03-enrollment/checkpoint.json` binds 40 source +inputs, nine successful checks and a copied native executable. The failed initial +build/test outputs and a dependency-check environment failure remain retained; +the latter passed after restoring the Cargo path. This is a partial checkpoint, +not a CG-03 DONE record. + +### CG-03 range-directory checkpoint + +Source `a0c11969763edbe86ee08fc7d1650129058c8ee0` adds bounded immutable AVL range +pages, path-copy replacement, a conditional head selecting the exact mutation +journal and operation/group indexes, and permanent genesis pinning. Historical +retry confirmation survives later root updates. Duplicate IDs, stale parents, +changed coverage, wrong epochs, reused operations and missing/corrupt selected +dependencies refuse. Retired group IDs remain reserved. A cached discovery path +does not refresh S3 metadata or list the namespace. + +All eight native cell-group catalog tests pass. These include concurrent CAS, +lost response/restart, initialization racing a later mutation, 128-range bounded +lookup/path-copy checks, and 80 alternating split/merge metadata transitions. An +independent flat partition checks every resulting boundary plus unrelated cell +keys; original views remain readable throughout churn. The 13 existing native +fleet-catalog regressions also pass with the shared fault adapter. The adapter's +new barriers are inactive unless explicitly armed for directory operations. + +The directory has no serving authority. Birth/disk claims, managed birth +authorization, deadline/hop-bounded client redirects and the real shared host +remain outstanding. No live split/merge, Raft quorum or RustFS qualification is +claimed. CG-03 remains IN_PROGRESS and CG-04 through CG-16 remain TODO. The full +hard-cutover goal is active; the earlier plan-only handoff does not satisfy it. + +The partial record is retained under +`.cellar/cell-groups-evidence/CG-03-directory/checkpoint.json`. It includes source +and executable hashes, command output and the corrected initial fixture compile +failure. This checkpoint is separate from the still-incomplete CG-03 task gate. +The record binds 63 source inputs, nine successful command logs and two copied +native executables. All-target Clippy with warnings denied, formatting/layout, +native dependency isolation, docs, source preflight and all 37 plan-validator +tests pass at this checkpoint. + +### CG-03 cell-independent birth checkpoint + +Validation source: `40c3c0b143f72496f0345be890413ed855c241ae` (declaration-spacing successor). + +Source `9db4ba8fc91e6d67de244c1a684001393bcc9e56` binds a group birth to physical +history, range/epoch, creation operation, namespace policy and canonical initial +members. It has no single-cell ownership field; range boundaries still carry full +cell routing keys. The bounded catalog encoding uses +the distinct kind 186; the provisional overlapping tag was corrected before this +checkpoint. No older birth layout is decoded or migrated by the new API. + +Global group-ID reservation uses the existing permanent key and the entire +canonical birth. Different namespaces, incarnations, creation IDs or placements +cannot reuse that key. The range directory now verifies every new route against +its exact reserved birth before publication. Losing a topology CAS retains the +reservation. Selected ranges can remain unmaterialized until their first use. + +Lazy materialization checks current range selection and the reservation before +conditional creation, then checks selection again. Unknown outcomes reconcile +with the original encoded request. A verified birth observation remains metadata: +it refuses wrong catalog owners, missing or changed immutable data and retired +routes, and grants no permission to initialize a voter. Managed namespaces refuse +the unmanaged APIs before I/O; their rooted fleet authorization remains to be +integrated. + +All ten native catalog tests and all 16 existing node-directory regressions pass, +including the distinct wire-tag assertion. The new cases cover canonical placement, +lazy creation, missing confirmations after actual writes, exact retry after reopen, +global cross-namespace conflicts, retired-range refusal and managed-policy refusal. +Existing bounded routing, races and the 80-transition churn oracle still pass. + +Physical `GroupBootstrap`, engine mappings and replica identities still carry the +old cell-scoped identity. The next step is to make that durable log path explicitly +group-scoped, then connect the verified birth to real synced genesis, node storage +pins and permanent member claims. A synthetic cell name is not an acceptable +adapter. The required `birth_unknown_and_claimed_blank_refuse` integration case +remains absent and CG-03 remains IN_PROGRESS. Redirect budgets, the shared host and +all later task/release gates remain outstanding. + +The partial evidence record at +`.cellar/cell-groups-evidence/CG-03-birth/checkpoint.json` binds 67 source inputs, +nine successful command logs and two copied native executables. All-target Clippy +with warnings denied, formatting/layout, native dependency isolation, docs, source +preflight and all 37 plan-validator tests pass. The failed declaration-spacing +check is retained with its source revision and corrected result. This is not a +CG-03 DONE record or proof of shared-group disk-loss recovery. + + +### CG-03 physical engine registry checkpoint + +Source: `d72ea068a527d2027e806ca78ab390256e8634fa`. + +The engine catalog now keys histories by `GroupHistory`, excluding the application +cell name. Synced mapping records use the explicit `cellar-group-mapping-1` domain +and four physical identity fields. The old bare `scope` mapping shape refuses on +recovery; no compatibility decoder or disk migration is introduced. + +The staged single-cell host retains its real cell binding separately as application +metadata. Lookup, create retries, seeded attachment, pending-slot accounting and +physical inventory reject a different cell binding for an existing physical +history. These checks prevent an adapter caller from turning another cell name +into a second handle or phantom slot. The inventory wire still uses its existing +cell-scoped digest until its consumers undergo the later cutover. + +This is a storage prerequisite, not the full physical bootstrap conversion. +`GroupBootstrap`, replica handles, node pins and permanent member claims still +need the cell-independent path before the new birth API may start a voter. +The required claimed-blank-disk integration case remains absent. CG-03 stays +IN_PROGRESS; CG-04 through CG-16 and the final release gate remain pending. + + +Validation at this source passes 19 pure-core cases, 27 selected library cases, +all four genesis cases, 14 native manifest cases, 19 native storage cases, +16 node-directory cases, 10 cell-group catalog cases, 33 host cases and +54 snapshot cases. The new engine tests inspect the stored physical mapping, +restart the actual engine, reject cell aliases at lookup, creation, pending-slot +and inventory boundaries, and rewrite both catalog and group records to prove old-domain, unknown-domain, +zero-history and invalid-cell refusal. Existing managed genesis sync-barrier and +bootstrap claim tests still pass. The snapshot suite includes seeded restart, +lost-claim confirmation and selected-payload refusal; its explicitly ignored +RustFS case was not run. Parent-launched process fixtures remain intentionally +ignored by direct suite execution. + +All-target Clippy with warnings denied, formatting/layout, native dependency +isolation, docs, source preflight and all 37 plan-validator tests pass. The partial +record is retained under +`.cellar/cell-groups-evidence/CG-03-physical-identity/checkpoint.json` with source, +executable and command-output hashes: 72 source inputs, 13 successful command logs, +eight executables and 194 distinct Rust cases. An interrupted build using the machine's +older default compiler and a command naming a nonexistent test target are retained +as non-qualifying attempts. Final checks use Rust 1.98.1 and actual test targets. +These results do not qualify the shared host, new group member claims, RustFS +recovery, hibernation, performance or the final hard cutover. diff --git a/docs/design/multi-raft/cell-groups/protocols.md b/docs/design/multi-raft/cell-groups/protocols.md new file mode 100644 index 000000000..2f9e3b491 --- /dev/null +++ b/docs/design/multi-raft/cell-groups/protocols.md @@ -0,0 +1,1678 @@ + + +# Durable protocols and crash cuts + +Purpose: Durable enrollment, apply, topology, hibernation and disaster-recovery protocols. + +Status: required target behavior. Names below are contracts to implement. +The [design](design.md) defines identity, partitioning and storage. A catalog +record is never a vote, a majority proof or infrastructure fencing evidence. + +## Invariants + +| ID | Required invariant | +| --- | --- | +| I01 | Cell, group, incarnation, route and schema identities are checked independently. | +| I02 | Every acknowledged Quorum output has a real persisted majority and a published exact cell effect. | +| I03 | Receipts and cell versions cannot be borrowed across cells; retries preserve identity through movement. | +| I04 | Durable group apply is contiguous and references a complete synced image/manifest closure. | +| I05 | Strong reads use current-term quorum authority and a coherent target cell image. | +| I06 | S3 acknowledgements require a retained complete group checkpoint/tail closure. | +| I07 | Reclamation preserves every live group, snapshot, reader, archive and transition dependency. | +| I08 | Membership changes preserve quorum intersection, disk lineage and failure-domain constraints. | +| I09 | Split/merge admits at most one writable owner of every cell, including after lost replies. | +| I10 | Hibernation never manufactures authority or bypasses a fresh wake/read barrier. | +| I11 | Zero-quorum recovery fences all possible old writers and creates fresh incarnations. | +| I12 | Work, memory, file descriptors, catalog scans and shutdown remain bounded and owned. | +| I13 | Alarms/outbox/inbox preserve committed intent and stable effect identity across movement. | +| I14 | Release claims are tied to exact source, actual tests, workloads and infrastructure. | + +## Physical group observations and handoff + +`RaftCells::group(GroupHistory)` selects a local physical log; application +commands and queries continue to use independent `CellHistory` handles. +`RaftGroupHandle::observe(timeout)` samples the actual owner and its durable +cache in one serialized turn. It reports replica identity, role, membership +revision, persisted/committed/applied progress, installed checkpoint, retained +entry count and applied-manifest cell occupancy. A group containing two cells +still contributes one group replica. Retained entries are not a disk-byte +measurement. This metadata grants no quorum, serving or compaction authority. + +`archive_status()` reports the archive controller's last verified publication +independently of the owner sample. Absence means no installed archive owner; +a publication is historical metadata, not a retained coverage proof. Local +compaction continues to require the existing physical dependency verification. + +`transfer_leadership(LeadershipTransfer, timeout)` preserves the caller's exact +term, membership revision and target. The actual owner rechecks those guards, +incoming-voter membership, readiness and matched progress before admission. +Its deadline starts at invocation, including time before polling, and is checked +again after queuing before mutation. Success establishes admission only; +observe the actual election and obtain fresh application authority afterward. +Expiry or close after dispatch does not establish whether a handoff was admitted. +Observations and guarded transfer reject zero or greater-than-60-second budgets +and closed owners. They add no worker, per-cell Raft instance or admission bank. + +## Enrollment + +Resolve the current range, materialize its group birth if absent, and obtain +current-term leadership. Propose `Enroll(CellHistory, schema, initial_image)` +under that route epoch. The group checks the key is in its active range and no +other live incarnation has enrolled that ID. The command is idempotent by its +stable creation identity. Only committed/published enrollment becomes visible. + +Persist the original creation operation, logical cell history, schema and initial +image commitment atomically with enrollment. Retain this creation record through +later mutations and normal transfers; comparing a retry to the latest image would +confuse a successful earlier creation with a conflict. The initial commitment is +provenance, not a requirement to retain the superseded initial image forever. +Snapshots and archives carry the record with the selected current cell state. +Initialization runs on the bounded node SQL workers under actual-leader authority. +The compiled initializer has an explicit contract digest in the class voting +schema. It receives the logical cell history and captured host inputs through its +private transaction. A host that cannot execute that contract must refuse the +registration instead of silently omitting initialization. +Reserve the logical cluster/namespace/cell name before entering the callback; +competing incarnations must not obtain independent producer leases. Capture initial +SQLite/LTX bytes once, carry the preparation term with the output, and check that +term on the RawNode owner atomically with proposal admission. A node that loses +leadership and later regains it must reject retained output from its older term. +Cancellation retains physical ownership until the callback and cleanup complete. + +Before append, reserve group capacity against applied cells plus accepted pending +creations, keyed by logical name. Retrying one creation consumes no extra slot. +Retain bounded leader-local acceptance records independently from Ready/completion +budgets. An exact pending proposal retry returns its original acceptance, never a +second log entry; changed bytes under that identity refuse. After apply or leader +replacement, resolve the receipt/creation record rather than append an old base. +A stale-base refusal does not prove non-commitment. New leaders must apply their +current-term barrier before admitting creations against recovered capacity. + +A definitive owner capacity refusal before append may retire a preparation only +with an exact preparation-bound rejection proof. Issuing that proof irreversibly +seals every clone against later submission, even after capacity returns. A generic +queue error, lost response or same-term receipt absence proves nothing about an +earlier attempt. Remember an accepted preparation's original position/digest; +returning that acceptance again grants no commit or current authority. Keep raw +byte submission closed once the owner receives prepared SQL work so queued raw +retries cannot bypass the seal. A proof for another engine, cell or replacement +preparation refuses even when the business proposal ID matches. Cleanup closes +the private writer on its worker before releasing logical-name admission; queued +submissions and retained outputs/proofs keep their byte reservations. + +Admission decisions must be checked again in committed log order against the +current lifecycle state. Recognizing an exact retry grants neither current-term +authority nor permission to restore the initial image over newer mutations. + +| Crash/collision | Recovery | +| --- | --- | +| Birth CAS result unknown | Read the canonical birth/mutation journal; never allocate competing voter history. | +| Schema image synced, enrollment uncommitted | Keep it unqueryable; replay/reconcile the same request or collect an orphan after all pins expire. | +| Enrollment committed, response lost | Return the recorded cell history/result on retry; do not run initialization twice. | +| Concurrent creators | One committed enrollment wins; mismatch refuses rather than changing class/incarnation. | +| Split seals during preparation | Revalidate before proposal; an accepted-before-seal entry belongs to the frozen prefix. | + +## Routed read RPC + +Use the existing node-owned query lane, with independent control/data progress, +bounded peer/node admission and joined physical shutdown. Authenticate both the +physical group and logical cell histories, schema, route epoch, birth identity, +destination member and node/disk/process identities. Query origins identify a +node process; a namespace read grant never authorizes a mutation or Raft message. +Grant revocation cancels its in-flight read waiters. A destination still requires +live membership and the actual cell's read authority after routing checks. +Separate host registration admission from data retained by accepted queries. +Dropping the final host registration must refuse new reads even while an earlier +request still owns its gateway data and physical SQL work. + +Allocate authenticated wire sequence numbers in the single peer sender after +dequeueing. Concurrent callers can enqueue in a different order from admission; +that must not cause a valid request to be refused as a replay. +When the peer closes its bounded idle header read, the sender retires the +connection before sending the next queued request. An idle connection has +no valid response bytes. Once an exchange starts, a transport error still +preserves the original mutation's ambiguity. +Consume each catalog attempt once. Carry its remaining duration across hosts; +never compare unrelated monotonic clocks or replenish the original caller's +deadline/attempt budget. Confirm relocation hints through the trusted catalog. +Return the selected cell observation and authenticated cell token, never the +group's newer log cut. The caller independently enforces an AtLeast minimum. +Retain result admission until consumption, and retain entered SQL/image owners +through physical cleanup after disconnect, timeout or cancelled shutdown. + +Invocation routing shares a bounded namespace range-hint cache across query, +command and enrollment dispatch. Configure both entry and reserved-byte bounds; +clones share the same cache, and cell count does not allocate one entry per cell. +Hints are learned only from verified directory lookups. An observed successor +head clears older hints; a lookup completing from an older head cannot refill +that cache. Same-revision/different-digest observations refuse. Explicit refresh +runs in the owning execution domain; failed refresh preserves only historical +hints, never a freshness or serving claim. Redirects still require exact catalog +confirmation under the original deadline and attempt budget. + +The current shared gateway resolves the destination catalog route for each RPC. +Removing that destination read depends on replicated transfer sealing and +admission invalidation. The invocation cache does not relax the current check; +this intermediate behavior does not establish steady-state request latency. +Attached-peer and explicitly discovered nonmember reads share the same query +lane. Nonmember discovery creates no local replica and grants no voting or write +authority. It consumes the same bounded route/peer registries, participates in +node-process refresh and endpoint consistency, and releases unused pools when +removed. A replaced discovery view refuses captured requests and late replies; +removal does not revoke the destination's independently managed namespace grant. +Grant revocation cancels read waiters without releasing physically entered SQL. +The public host supplies trusted discovery and explicit permissions. Automatic +SDK reads use the local replica when present and otherwise try configured members +within the original budget. Dynamic membership refresh for ingress remains +integration work. + +Remote commands use the same bounded application RPC lane with a mandatory, +MAC-bound command purpose and distinct payload/result markers. A read grant or +voter membership never grants write admission. `NamespaceCommandOrigin` supplies +an explicit namespace/process permission with bounded revision tombstones; +endpoint checks, rooted process refresh and fleet revocation include these grants. +A host-owned command gateway binds the exact node controller. Dropping its final +registration closes new admission without rolling back accepted work. + +An ingress process can probe configured peers for a ready leader before +submitting a command. Probes use command permission and bind the complete route. +They open no SQL work. The origin retains time for a single mutation exchange +after destination selection. + +Carry the caller's captured time/random inputs, command identity, request contract, +retry horizon and remaining deadline to the destination. The controller verifies +a fresh catalog route against the transmitted group, schema, epoch and birth, +then checks actual leader authority before entering SQL. The SQL worker checks +the original invocation deadline when it dequeues a new command, before opening +its write workspace. It owns work that has started independently from the +connection. Disconnect, revocation and timeout can suppress +a reply but cannot prove non-commit or release a producer. Success contains the +exact committed command identity/result and an authenticated logical cell token; +conflict, expiration and sequence-gap refusals remain typed. Retain caller reply +byte admission through consumption. Retry the same identity and input under the +original budget; a committed duplicate must not re-execute application code. +Originating SDK enrollment uses the same bounded leader selection as commands. +Incoming enrollment always executes locally. The SDK retains a remote creation +record as authenticated transport output and preserves local SQL and archive +ownership separately. Remote enrollment uses the same bounded write exchange, +namespace and process grants, and exact controller registration. The mandatory authenticated purpose distinguishes +Query, Command and Enrollment under marker 1. A command payload cannot be decoded +as an initializer call. Capture creation identity, host inputs and preview identity +before discovery; retries retain them and the original route/deadline budget. +Receiver enrollment verifies a fresh catalog binding and actual leader authority +before initialization. Its admitted controller owner survives disconnect, +revocation and response deadline. Return only the exact committed creation record, +including the original enrollment image after later mutations. That record is not +a current-version token. Conflicting creation/incarnation refuses without replay. + +Standalone receipt lookup has its own authenticated purpose and requires the +explicit command-origin grant. Read permission alone cannot inspect command +results. The exact registered controller supplies the runtime and SQL pool; +receiver admission checks fresh catalog binding and selects a correlated quorum +read. Registered contract, payload digest and authenticated command identity +validation precede receipt inspection. Immutable reads use the existing worker, +read admission, cancellation and deadline owners without a private writer, +initializer, command callback or receipt maintenance. + +A retained receipt returns its exact result/retry horizon with a signed cell +observation. An absent receipt remains ambiguous, even after a quorum read: +original delayed or unresolved work may still commit. Lookup never releases a +producer or produces a NotCommitted proof. Preserve replicated expiry/conflict +and sequence-gap failures. Only a sequence-1 image can lack both receipt tables; +partial schema or later schema loss refuses as corruption. Missing and an empty +committed result have distinct wire encodings and API results. + +The controller accepts bounded dynamic group registrations, including startup +with an empty roster. A registration supplies the exact bootstrap permit, verified +selected root, applier owning the same manifest page store, and physical peer +route. Pending and active groups share the group limit; pending work and unconsumed +replies also retain the existing item/byte budget. A pre-existing physical route +cannot be silently adopted. Cleanup removes only the exact registered route and +cannot erase a replacement view. + +The worker installs the applier before polling runtime attachment so early +consensus events cannot escape ownership. Attachment revalidates catalog claims; +only successful completion publishes application roster membership, and serving +still requires fresh current-term quorum. Dropping a registration waiter cannot +cancel this work. Definite failures release their slot before responding. Shutdown +joins entered catalog/attachment futures before closing transport, SQL and the +runtime. A competing external runtime owner or runtime-wide failure still closes +the controller conservatively. A group-local apply/runtime failure quarantines +that group: admission refuses, execution authority is revoked through reserved +runtime control capacity, and unresolved producers/apply tickets remain owned. +Quarantined groups retain their roster capacity until joined restart. Late shared +persistence completion cannot restore their serving fence. Shared log-engine +I/O/corruption/closure and lost shared SQL worker replies still close the node. +Request-driven provisioning/lazy birth and configuration lifecycle remain open. + +The current lane serializes RPC exchanges per peer; this is not load qualification. + +## Whole-group membership + +The shared owner accepts `CellGroupMembershipChange` through bounded runtime +admission. Its marker-1 domain binds the complete route and the observed membership +revision. Explicit changes use V2 entries; legacy V1 and unbound changes refuse. +Decoding bounds wire bytes, repeated changes and context before protobuf allocation. +New member IDs must exceed the permanent high-water mark and first enter as +learners with node/disk bindings. A removed ID cannot be reused. Promotion requires +the leader to observe that learner replicated through its durable committed cut. +These checks do not provision a fresh learner or grant it serving authority. + +Configuration entries occupy the same ordered apply lane as cell mutations. Pure +replay computes the next Raft configuration and membership without changing the +live RawNode. The complete group root advances its applied position and membership +revision while preserving every cell leaf and version. Engine persistence validates +the committed entry and atomically syncs configuration, membership and root. Only +the matching persistence completion changes live Raft progress and advances apply. +Explicit and implicit joint changes are supported; Raft's context-free automatic +leave is accepted only while the persisted configuration requests auto-leave. + +Admission returns a log position and digest, not a committed membership proof. +After a lost reply, inspect committed membership instead of rebinding a new revision +and blindly repeating the operation. A new binding may precede discovery of its +physical process route; missing peer routes are retryable stale-peer failures. +Authentication and active membership still gate transport admission. + +`transfer_leader` admits a handoff only to another caught-up incoming voter after +local committed apply settles. It revokes captured producer authority during the +handoff; observe the new leader's current-term readiness for completion. Removed +or demoted leaders lose producer and leader-bound read authority at synced cache +publication, even before the owner consumes its completion. Membership alone never +allows a learner to produce writes. + +`CellGroupRuntime::seed_authority` selects a complete applied root only after a +fresh correlated group ReadIndex. It works without an enrolled cell. The target +must be a committed new learner with the exact node/disk binding, outside both +voter sets. The selected cut covers the barrier and its membership revision. +Pending requests and retained roots share the bounded node/group read budgets. + +The non-cloneable authority captures the exact selected root through the node's +checkpoint pool on a blocking owner. If apply advances before capture, the caller +must obtain a fresh authority. After capture, ordinary cell writes may continue +while the source streams its immutable cut. Membership changes, term changes, +leadership transfer, shutdown and deadline expiry revoke the source. Every sink +call has checks before and after it; the final completion also checks authority. +An incomplete or revoked stream cannot produce a successful End. The source API +binds the requested process identity; `prepare_group_seed` authenticates it through +a separate TLS-exporter-bound seed exchange. Requests, grants, readiness and chunks +have distinct marker-1 domains. The reply binds the request tag, nonce, complete +physical route, fresh barrier and canonical checkpoint descriptor. Chunk sequence, +digests and explicit End are authenticated within that exchange. Peer replacement, +transport shutdown and the original deadline revoke pending destination work. + +The marker-1 request requires an explicit `from_s3` flag and the authenticated +reply requires a tagged `data` choice: peer bytes or an archive upload ticket plus +exact checkpoint reference. Missing fields and a choice different from the request +refuse before staging. The S3 mode opens the established namespace archive registry; +a peer cannot supply a backend or override its identity. It never falls back to +peer checkpoint bytes. + +Managed startup validates its saved host capability and prepared namespace +declarations against one current catalog root before opening mutable engine +state. An already quarantined/finalized host refuses without reopening its log; +the later engine-bound boot CAS still orders admission against concurrent fences. +Managed physical route refresh discards a retained initial-voter endpoint only +with rooted permanent isolation, even when durable membership has not changed. +This permits an admitted learner's route to replace an unusable old endpoint +without removing votes or seed-owner/resource debt. Movement completion may use +that isolation proof only for the exact obsolete boot, after fresh stable RF3 +excludes it. Surviving original voters still need live physical-inventory and +worker-exit proofs; settlement leaves the obsolete disk and disruption retained. + +Managed live voting observations share one original deadline. Each read-only +member attempt receives a share of the remaining time, reserving time for other +members and final validation. A lost reply retains the unverified history and +allows the owning walk to retry fresh observations. No partial member set or +historical sample can satisfy the final proof; all three safe ReadIndex/durable +apply observations and current rooted inputs remain required. Mutation ambiguity +and retry rules are unchanged. + +The explicitly started Node-owned failure service can drive exclusion, exact-boot +platform isolation, S3 replacement, fresh RF3 and terminal survivor/archive +verification. Its private verifier initializes durable scratch storage before +receiving image bytes; serving pins cannot satisfy this proof. Recovery/removal +phase futures remain owned under the original admission, stop and turn deadline. +Shutdown joins a pending platform callback. Restart reconciles the rooted +operation; archive-read failure blocks progression and retains the intent. +A later controller repeats physical S3 replay before +the terminal CAS. Removed records retain the old engine and disk; they do not +authorize reclamation or prove a production platform fence. + +An S3 source captures the same fresh-quorum, exact-root checkpoint before catalog +I/O. A separate purpose-bound, exclusively locked local journal syncs the reserve +intent before allocating a generation and then syncs the returned upload ticket. +The pending generation and inherited parent graph protect every checkpoint object +through handoff. This transfer publishes neither a disaster-recovery head nor S3 +command coverage. The destination checks that protection, independently fetches and +verifies all objects and feeds the existing physical checkpoint receiver. Only a +complete prepared checkpoint matching the authenticated snapshot permits READY1. +The source rechecks live authority before authenticated End. The ordinary engine +installation, fresh destination boot/catalog checks and promotion/catch-up fences +remain required. + +The source retains its admitted transport, read, checkpoint and S3 resources until +physical upload and handoff cleanup finish. The destination's prepared stores own +their local pins before READY1. Caller cancellation cannot release entered work; +node close stops new source admission and joins transport with source cleanup. +An uncertain ticket retirement preserves its durable journal and observable cleanup +error. A later seed attempt reconciles that exact intent before allocating again. +A node-owned background sweep also discovers existing seed journals during idle +operation and retained restart. It streams one directory entry at a time and +checks the canonical history path, journal purpose, engine creation and exact +immutable namespace archive binding. Cleanup may outlive the selected birth; +reopening that binding grants no current membership or serving authority. Missing, +corrupt, foreign-engine or wrong-purpose state refuses without initialization. + +Live sources and cleanup share a per-history lock. A sweep skips an entered +source, never expires its protection by age, and retains its lock through entered +physical I/O, including panic/error joins. It resolves the original reserve intent +or upload ticket idempotently, confirms abort, then syncs the empty local record. +An unavailable registry or unconfirmed abort preserves the intent and observable +debt for another pass. Bounded error tracking does not stop other journals from +being examined. Empty journals need no remote request. A lost journal or engine +cannot authorize retirement of an unknown remote owner. + +The immutable provisioning configuration charges one 272-KiB journal workspace +to the existing node preparation budget. One sweep runs per node, with one +ordinary 8-KiB S3 admission per journal rather than for the entire inventory. +Configured ceilings remain unchanged. Complete passes rest for at least one +second, or the configured retry interval if longer. Close stops new sweeps and +joins entered work before releasing engine/configuration resources; cancelling a +close waiter cannot release a blocked archive read. Controller usage separately +reports active sources, the active sweep and retained cleanup/discovery errors. + +Managed retries retain the exact destination boot, member, repair permit and data +mode. Source member/node are attempt hints, independently reauthorized against a +fresh source quorum; leader failure may change them without replacing the retained +destination owner. An entered attempt keeps its original deadline and hint. A mode +change cannot adopt a retained S3 preparation. + +The destination reconciles the namespace node pin and permanent member claim +against the exact node/disk/engine. Its synced staging mapping retains the original +birth configuration and grants no startup permission. An existing remote claim +cannot recreate a missing mapping. `CellGroupSeedSession` exposes the staged group +only to construct its physical stores; dropping an uninstalled session abandons +the exchange and leaves a non-serving stage that needs a fresh authenticated retry. + +The receiver verifies and syncs the entire physical closure before a single engine +fence installs configuration, membership, complete root, retained checkpoint and a +local authenticated-install receipt. That receipt is required in the marker-1 +`cellar-group-apply-1` storage record (`manifest` plus `seed`); the old bare-manifest +shape refuses decoding. The receipt never travels in a snapshot and generic +snapshot installation cannot mint it. No old-format reader or user-data deletion +is introduced. Live grant and exact engine/process checks run before entering the +fence. Once physical persistence begins, cancellation and close retain the grant, +image owners, task reservation and consensus lease until actual completion. +A queued install rechecks the grant on the engine worker before its fence and +refuses a newly revoked source without publishing the prepared root or seed receipt. +Revocation after entering the fence cannot roll back already issued writes. A +successful fence may retain the complete authenticated seed despite that late +revocation; the local receipt still grants neither voting promotion nor current +quorum authority. A sync error leaves the live cache unpublished and closes the +engine to further mutation. Cold recovery must reconcile the selected complete +root and local receipt together; an error response alone does not prove rollback. + +`CellGroupApplier::recover_selected` reconciles an unknown install outcome using +the existing physical store owner; selected pins prevent reopening that directory +while it is still owned. It verifies current and retained checkpoint closures +without introducing catalog or quorum authority. + +`ReplicaLog::reclaim_cell_storage` performs idle-node orphan maintenance after +runtime and SQL shutdown have joined. It reserves both exclusive claims until +physical completion and serializes on the existing engine worker. Callers supply +one exact applier for every physical group, including unattached/retired groups, +and every additional durable archive/transition root. The image stores must belong +exclusively to this engine. Missing, duplicate, foreign, oversized or busy +inventories refuse. It verifies all selected current and checkpoint closures and +merges references by exact shared-store owner before any deletion. Live pins +remain independent protection; file and directory barriers precede quota refund. +Limits bound selected cells, reclaimed files and page scans. Cancellation retains +queued work and claims; engine close joins completion. This API grants neither +Raft log truncation nor S3 coverage and does not run during normal serving. + +Normal shared-group startup separately adopts the serving extent inventory after +recovering every local group's current and checkpoint graph. Those complete graph +pins replace conservative filename retention before routes become reachable. +Unmanaged startup pins the images of retained groups in declared namespaces +even when the replica assignment is omitted. These groups stay out of the +runtime roster and transport until an authorized attachment. Startup refuses +when a retained group's namespace is undeclared. +The independent archive verifier adopts its separately locked cache before +registering the first receive/replay job; cached filenames provide no recovery +proof. Disposable empty workspace roots are transient and pinned before sealing. +Existing retained roots and uncertain publications keep their protection. + +An admitted private workspace retains its exact directory and locked extent-store +owner through all SQLite/VFS handles. The store's open established its ancestor +barriers; creating a workspace still syncs the child and parent before returning. +Rooted private opens validate that owner/directory binding and do not repeat the +ancestor walk. Failed child/parent barriers keep the full charge and quarantine; +reopen establishes new barriers before another workspace can be issued. Anonymous +read-cache directories remain disposable and supply no persistence, image or +recovery proof. Materialized private paths keep their full directory preparation. + +The verifier keeps eligible files as an opportunistic, charged cache while each +inventory dimension remains below one quarter of its existing ceiling. Pressure +in roots, objects, root references or bytes enables ordinary collection; an +inventory without roots also collects orphan objects. Serving inventories collect +eligible files without this delay. Cache entries add no pins or capacity and +cannot replace archive GETs, source validation or independent local replay. +Buffered imports check the supplied extent digest even when a valid destination +already exists, so a warm cache cannot hide corrupt incoming bytes. + +Only these adopted inventories enter the node-owned live extent collector. One +physical turn runs at a time across the two node roles, with at most sixteen +deleted files. Checked working memory and the service's fixed metadata charge +come from the existing control bank. Busy admission defers work. The idle hint +uses metadata and notices a last-reader release without another mutation; marking +preserves locally imported dependencies of pinned, incomplete receives. Physical +deletion and directory sync precede capacity refund. Stop and cancelled observers +cannot release the registry lease, inventories or working memory before an +entered physical call returns. Inventory or worker failure stops collection and +appears in advisory resource observations; those observations supply no durability +proof. This service does not collect manifest pages, invoke the offline engine +collector or authorize log truncation. Long-running replication and reclamation +still require their workload qualification. + +After interrupted dependency writes, collect only unselected debris, then obtain +a fresh seed from a fresh quorum. The empty stage still has no startup permission. +Install its ordinary process route only after authenticated seed installation +produces the local receipt; a staged grant is not ordinary membership authority. +Register source checkpoint endpoints after installing their current physical +routes so the captured transport route matches the fresh source process. + +`bootstrap_joined_group_member` resumes only a locally receipted install with the +same permanent remote claim and active node/disk-bound membership. Full physical +recovery still verifies both current and retained checkpoint closures before +attachment. Birth-member bootstrap continues to admit only original members. +Neither API grants current-term quorum authority. A new learner must catch up +through ordinary Raft replication before promotion; a lost install response is +reconciled from durable local state instead of assuming failure or resetting disk. + +Native transfer/replacement qualification is recorded in progress.md. Managed +fleet orchestration, sustained-write capture fairness, the complete CG-05 fault +matrix and deployment qualification remain open. + +## Apply, checkpoint and archive + +For entry N, verify the cell base/sequence, stage the result, sync its immutable +image closure, sync changed manifest pages, then atomically persist the group root +and applied=N in the engine. Publish and acknowledge only after the marker succeeds. +Old roots remain pinned until no recovery/reader/transfer owner needs them. + +| Crash cut | Required outcome | +| --- | --- | +| Before image sync | Recover prior root; no output credited. | +| Image durable, manifest incomplete | Prior root remains authoritative; staged image is not visible. | +| Manifest durable, engine marker absent | Prior root remains authoritative; replay the same log entry without callback execution. | +| Marker durable, response/publication lost | Recover the new complete root; receipt retry returns the committed effect. | +| Any referenced image/page missing | Refuse group admission; never substitute an empty cell. | +| Later cell staged before earlier cell finishes | No applied-prefix hole and no early output settlement. | + +A checkpoint pins the whole root at K, including empty enrollments, membership, +range/transition state, receipt floors and effect summaries. Stream bounded chunks +and verify the receiver's own synced closure before atomic install. A successful +transfer is not proof that another replica has installed it. A new voter receives +fresh member/disk identity and joins only through committed membership changes. + +The native checkpoint source now captures only an exact engine-selected complete +root. It snapshots configuration and membership under the same cache lock, retains +all immutable manifest/image pins, and streams canonical pages plus materialized +bytes or extent manifests/objects. Empty enrollment remains an authenticated leaf. +The source pool charges both retained metadata and a bounded streaming workspace; +its non-cloneable source and borrowed chunks prevent overlapping streams on one +reservation. Run capture and streaming on a joined blocking owner. A cancelled +waiter does not release the source's physical pins or quota. Flat-image integrity +is final only at EOF, so partial output must never be selected by a receiver. + +This source contract currently represents active groups only: the shared owner +still refuses topology and park entries. Those owning tasks must extend the durable +root and checkpoint transition binding together. Receiver decoding/staging, atomic +engine installation, retained-checkpoint selection, transport admission and +membership catch-up are not implemented by this source API. Successful streaming +supplies no receiver-install, voter, serving, archive-coverage or reclamation proof. + +The shared transport has a dedicated checkpoint stream on its TLS data listener. +Configure one node `CellGroupCheckpointPool` with `enable_group_checkpoints`, shared +with local publication. Controller registration retains the exact applier and +physical route; route replacement/removal or registration drop revokes subsequent +stream admission. Ordinary consensus frames still reject snapshot payloads. + +Marker-1 start frames bind the physical group/incarnation, permanent member, +node/disk/process identities, schema and snapshot envelope to the TLS exporter. +Each chunk has a separate authenticated purpose, binds the start tag, and carries +a monotonic sequence, part identity, offset, length and body digest. Header and +payload limits are checked before allocation; chunks are at most 64 KiB. Extent +identities resolve against the receiver's already verified manifest. An explicit +authenticated End is required. EOF or cancellation before End cannot finish a +receive; mixed cuts and incomplete closures refuse. After verified admission, +connection loss does not establish whether installation completed. Source capture +pins the exact engine-retained snapshot even when the current applied root advances. + +Both seed and checkpoint receivers observe the physical verifier while reading +network headers/bodies and feeding its bounded input channel. A rejected part +closes the exchange even if the peer sends no more bytes; it must not retain +network admission until the peer deadline. Successful verification still requires +the authenticated End and the matching closure. This does not cancel engine +persistence after it has been admitted or replace its actual durability fence. + +Two bounded channel windows connect the network to blocking physical owners. +Node/peer data budgets and source/receive slots remain owned through actual I/O, +even after a connection or delivery waiter disappears. The checkpoint pool +separately charges physical closure metadata/workspace; flat images use the log's +shared scratch pool. Transport close joins these owners. Membership and exact +registration checks run throughout the stream and again before runtime admission. +The verified closure enters `step_checkpoint`; its stream reply proves admission +only. Raft releases its acknowledgement after the matching persistence completion. + +Snapshot delivery feedback has 256 reserved node-wide slots and 1 MiB, separate +from ordinary calls and apply completions. The runtime coalesces one status per +known peer while retaining its reservation, then applies it after the in-flight +Ready has settled. A completed transfer remains owned until feedback admission or +transport shutdown; an I/O response timeout cannot silently discard required +failure feedback. Immediate snapshot queue refusals also report failure, allowing +Raft to leave its paused snapshot state and generate a retry. Whole-group +membership orchestration remains separate integration work. No transfer reply grants new voter, serving, archival +or reclamation authority. + +The controller's optional `CellGroupCheckpointPolicy` selects a checkpoint before +starting a successor apply when the configured entry interval has elapsed. It +uses the transport's already configured pool and owns initial checkpoint endpoint +registrations. One node slot spans blocking capture, admission and durable +publication; each selected group retains its exact successor ticket. The current +root therefore cannot advance during capture. The reserved completion-lane request +checks that ticket and the current root, and can publish despite ordinary request +or apply pressure. Pending Raft work may still append a suffix. Only the matching +`CheckpointPublished` event releases the successor to physical apply; an admission +reply alone does not. Reply/event ordering is handled in either order. + +Checkpoint selection precedes the SQL apply-capacity check in the controller's +bounded pending scan. An occupied SQL slot therefore does not block another +group's capture/publication. SQL work stays queued when its capacity is full; +checkpoint lifecycle, source and completion admission remain independently bounded. +Native tests hold actual image-directory persistence in both image forms, publish +an unrelated complete checkpoint, retain its unapplied suffix, then release and +join the original physical apply. + +`CellGroupRuntime::checkpoint_before_apply` exposes this same exact-ticket +boundary for embedding owners. It waits for completion admission while retaining +the captured source; owner/engine validation still binds the current root and +ticket. Cancelling an unadmitted request queues no work. Admitted requests remain +runtime-owned through physical completion and joined shutdown. `usage()` reports +advisory ordinary/completion/feedback/read-pin charges and immutable limits without +request admission. Completion charges refund at owner admission, while physical +source/root/storage ownership continues. Gauge observations grant no authority. + +The shared budget reserves and refunds byte/item dimensions in one short lock +scope, preventing rejected byte attempts from exposing temporary item charges. +Release/shrink notify all registered request sizes and retain a stored signal for +workers that register after checking capacity. Native budget tests cover mixed- +size waiters, late registration and concurrent rejected/fitting attempts. + +Pool contention or transient admission overload releases the successor and backs +off until a later apply boundary using the controller's node timer. There is no +per-cell task or unbounded retry loop. Capture jobs are joined before releasing the +runtime's engine claim; accepted writes remain owned by the runtime through its +join. Hard checkpoint failures retain the ticket and quarantine the group, while +shared log uncertainty closes the node. Quarantined groups can retain checkpoint +capacity until joined restart, like their registration capacity. + +Native controller tests hold publication at the real engine sync after lifecycle +and checkpoint-source admission. Cancelled close and wait retain both owners and +the engine/SQL claims through physical completion. The healthy two-voter quorum +continues another group's writes during the stalled node's engine sync. Release +joins shutdown, selects the captured checkpoint and permits a replacement runtime. +Both image forms run this cut. Further native cases fill all eight completion +slots with real completed images, retain a ninth checkpoint through admission and +a held engine sync, then complete its exact successor. Two concurrent 32-entry +writers compete for one shared source slot; both advance checkpoints before the +writers finish, and all selected current/checkpoint closures recover on every node. +These are bounded functional traces; broader deployment/fault/load schedules remain +in the later qualification gates. + +This is an entry-count policy for active groups. It retains the final idle tail; +time/byte triggers and coordinated hibernation require their later explicit cut +reservation/wake contracts. It supplies no S3 archive coverage proof. Manual +publication remains available to lower-level owners; this policy does not introduce a second serving +runtime or weaken exact-cut publication checks. + +An archive publishes a complete checkpoint plus a contiguous committed tail to K. +The tail addresses cells and validates each base chain; administrative entries +also replay. Do not rerun commands. Publish coverage only after all dependencies +are durable and independently verifiable. A dependency failure in one cell blocks +group coverage. Unchanged immutable cell objects can be reused between cuts. + +The initial CG-06 checkpoint archive boundary is implemented in +`raft/cell_groups/archive/`. A node owns one `CellGroupArchivePool`; group handles +and their clones share its item/byte admission across uploads and downloads. +Group history has its own archive namespace and marker-1 domain, without a +synthetic cell identity. One admitted physical producer streams a pinned complete +checkpoint through a one-chunk channel. An immutable bounded index names each +cell-addressed chunk, offset, length and SHA-256. Unchanged chunks are GET-verified +and reused without PUT. New conditional creates require exact GET confirmation, +including an ambiguous PUT response. Corrupt existing objects refuse overwrite. + +`CellGroupArchiveReference` is an untrusted persistent locator. Only +`verify_checkpoint` can produce `VerifiedCellGroupArchiveCheckpoint`: it downloads +and hashes every indexed object, checks continuity and limits, runs the shared +physical checkpoint decoder, and binds its complete recovered marker to the +index. Missing/corrupt cells or a substituted cut refuse the whole preparation. +`CellGroupArchiveStore` owns private pages/images for an explicit expected route; +open those stores on the execution domain, separately from serving storage. +`receive_archive` needs no GroupLog, member claim, selected applied cut or surviving +voter disk. Its checkpoint type is distinct from the engine-bound receiver output +accepted by SnapshotChange. Archive roots carry a privately minted owner token +that no Raft engine accepts for publication. No replacement genesis or voter +identity is fabricated. Serving still requires fencing, fresh-incarnation seeding +and a new quorum. Verifying older and newer archive cuts never changes an engine's +selected root or the eligibility of a later archive verification. The engine-bound +receiver preserves its selected-base and membership/install checks. + +Source/receiver pools separately charge physical closures and scratch. Archive +admission charges bounded encoded/decoded metadata and I/O windows. Complete +individual PUT/GET operations, including body EOF, have domain-owned deadlines. +Ordinary failures join entered physical workers; caller cancellation closes their +bounded channels and retains permits until those workers exit. Domain shutdown +joins entered work. The pool must be shared by node orchestration when it is wired +in; constructing separate pools per group would defeat the node-wide bound. + +The checkpoint capability has no conditionally published head or remote GC lease. +It grants no group S3 settlement, archive coverage, reclamation or disaster-recovery +authority. CG-06 still requires those owners, retention admission and fenced +fresh-incarnation seeding. Its executable scope records the production modules +and native archive qualification without changing any required case or oracle. + +Quorum writes may proceed during an S3 outage within explicit disk/unarchived-byte +budgets. S3-mode outputs wait for confirmed coverage. Log reclamation requires the +covering local checkpoint and configured archive retention proofs, plus transfer +pins. Shared engine purge must preserve other groups' live ranges. Readers and +in-flight installs retain object pins until their physical owners finish. + +CG-06 additionally implements committed-tail capture and independent data replay. +`upload_tail` copies the exact contiguous suffix to the engine-selected applied +root under the cache read lock, then encodes/uploads without that lock. Entries +have bounded canonical framing, including context and sync metadata. The complete +tail index names the original checkpoint, every immutable entry object, and the +final root/configuration/permanent membership. Missing ranges, unsupported types, +invalid cuts and configured byte/entry/cell limits refuse. Upload returns a typed +`CellGroupArchiveUploadedTail` retaining the exact committed-source observation. +Its persistent reference remains an untrusted locator. `into_reference` discards +the local source witness; deserializing a locator cannot recreate that witness. + +`extend_tail` reuses an existing archived prefix and captures only the local +suffix after its cut. Its flattened index directly names the original checkpoint +and all entry objects, so the previous tail index is not a recovery dependency. +This permits continued archival after local prefix compaction. Combined limits +are checked before new uploads; a full bounded tail requires a new checkpoint. +Physical shared-engine file reclamation remains a separate unqualified boundary. + +`verify_tail` recovers every checkpoint dependency and replays all entries through +the physical transition code shared with live apply. Live apply retains its +original ticket/commit/owner checks before and after preparation. Canonical joint +membership, leave and no-op entries advance group metadata without synthetic +cells or independent cell-sequence changes. Replayed final root, configuration +and membership must match the index; every final page/image is independently +verified and synced. No application callback or engine installation occurs. + +Materialized replay uses `GenerationStore::prepare_verified`: an existing cached +result cannot replace validation of the incoming LTX. The delta is recomputed in +admitted anonymous scratch before the destination is independently verified and +synced. `GenerationPin::verify_and_sync` also preserves the receiver's existing +independent-destination checks. New private replay images become reclaimable after +their returned pins close; verifying an existing image preserves its retention. +Private-store quotas remain enforced and live proofs retain their physical data. +Extent replay applies every delta and rechecks +the complete final image. A valid but wrong LTX delta or warm destination damage +must refuse, even when metadata and object hashes are well formed. + +Immutable page and extent-root stores remember completed file and directory +barriers only on their exact open owner. Warm preparation still reads and checks +the bytes; it may reuse those completed barriers for an unchanged immutable file. +Reopening clears the observation, and a failed read invalidates that file's +observation. Metadata reads and inventory filenames cannot restore it. Both +barriers must complete before reuse, and a page persistence failure keeps its +owner quarantined. This supplies neither a missing image nor remote coverage; +archive GET, digest, complete closure and replay validation remain required. + +A node-wide `CellGroupArchiveTailPool` separately bounds copied suffixes, decoded +entries, image work and retained base/predecessor/successor metadata. Its limits +are explicit and do not change the ordinary archive/checkpoint pool limits. The +returned `VerifiedCellGroupArchiveRecovery` retains this admission with physical +pins. Cancellation stops further replay work and retains entered physical owners; +domain shutdown joins them. Typed contention and poisoned-owner failures differ. +Orchestration must keep the verified archive result alongside any derived raw +manifest roots: cloning `PreparedGroupManifest` does not clone its parent archive +admission. Controller ownership propagation and private-cache maintenance require +CG-08 qualification before managed integration. + +The physical verifier remains data-only. CG-06's separate publication owner now +combines that proof with the exact committed source and persistent remote ownership. +S3 command settlement/outage admission, local Raft file reclamation and fenced +fresh-incarnation seeding remain unfinished; no required task oracle is changed. + +### Protected publication and remote collection + +A node shares one `CellGroupArchiveUploadPool` for upload maps, source witnesses +and collector marks, alongside the existing I/O, checkpoint and tail pools. Its +explicit byte/object/owner bounds include inherited-key lookup maps and compact +collection candidates. Clones retain their admission. Source-witness admission is +held until its final clone closes, independently of the asynchronous upload job. + +Fresh registry initialization uses a one-use, non-deserializable +`PreparedCellGroupArchiveCreation`. Persist its entire `intent()` in the durable +authoritative owner before commit. Opening requires that exact expected creation +identity; never discover and adopt a replacement mutable identity on restart. +Established identity/state loss refuses admission. A lost identity-create reply +with no complete state requires authority recovery; it cannot reset counters. +`RangeDirectory::prepare_group_archive_binding` supplies that immutable namespace +binding. Persist its canonical `encode()` bytes before committing the one-use +initializer. It checks the selected birth before binding, before initialization +and before return. The marker-1 catalog record binds the namespace root and exact +creation route/identity; its key is specific to group/incarnation. The namespace +chooses the object store and group-archive data prefix. A competing initializer +cannot replace the binding or initialize a different registry beneath it. +`open_group_archive` requires a freshly verified birth observation belonging to +the same catalog owner, reads the immutable binding, checks its namespace/route, +and opens only that expected registry. Missing binding/identity/counters refuse +without repair. An incomplete initialization requires authority recovery; replaying +serialized bytes does not restore initialization authority. These APIs do not +make controller ownership durable or grant serving/disaster-recovery authority. + +Persist each allocation intent before CAS and retry only that exact intent after +an uncertain outcome. Its registry, next revision, nonce and complete operation +form its identity. Definite conflicts can expire it. Live exact results reconcile +a lost reply even after unrelated registry changes; retired/aborted results never +reappear. Revision exhaustion refuses. Registry records are bounded to 64 KiB, +eight pending generations and 64 explicit holds. These are simultaneous bounds; +retirement permits continued lifetime churn without growing tombstone lists. + +A reservation protects its never-reused upload generation and its exact selected +parent, atomically at allocation. New object keys include both the registry ID +and generation. Reuse is restricted to exact objects in that protected parent +closure, with GET verification; an arbitrary old content address cannot be +resurrected. Binding checks every inherited key even after successful physical +replay. Dropping a local handle never releases a remote reservation or hold. No +unknown owner expires by wall time. A replaced process must be fenced before its +persisted allocation can be shared or retired. + +Tail capture seals a local committed-source witness with the real engine owner, +root, configuration and permanent membership before asynchronous upload. Source +writes and local checkpoint compaction may advance afterward. Publication +rechecks that sealed prefix against the live source, independently recovers the +entire immutable checkpoint/tail and matches its complete cut to the witness. +Neither a raw locator, a private archive root nor a serialized publication intent +can replace the source witness for a new publication. + +`CellGroupRuntime::capture_checkpoint` asks the existing worker to capture its +owner's exact durable applied root. It performs bounded metadata capture only; +physical streaming and remote I/O remain outside the worker. Owner/cache mismatch +during concurrent persistence returns typed `NotReady`, to retry after apply. +`upload_cut` consumes that sealed checkpoint, retains its original configuration +and permanent membership under the upload-owner budget, uploads the checkpoint, +and writes a zero-entry tail for that same cut. It never recaptures a live root +after object-store I/O. Healthy quorum can advance while a dependency is held. +The source checkpoint slot is released before independent recovery starts, so a +single-source checkpoint pool can serve upload and verification sequentially. + +`prepare_publication` writes an immutable root descriptor in the pending generation +and prepares one exact publication intent. Persist it before `commit`. One CAS +consumes the reservation, selects the non-regressing complete root and allocates +an explicit remote hold; independent readback confirms the exact result. Equal +indices require the same complete marker, and terms cannot regress. A later peer +mutation does not invalidate a still-live exact result. Lost readback/cancellation +leaves its intent for explicit reconciliation; it never allocates another hold +or silently substitutes a newer mutable head. Reconciliation only finds an +already-published hold and independently recovers that exact immutable root. + +Readers acquire and persist their own independent hold before downloading a +selected root. `verify_hold` reads that exact closure and rechecks its hold before +returning `CellGroupArchiveCoverage`. Clones keep local pins/admission; remote +retirement is explicit, after the durable owner retires/fences all uses and joins +entered work. A predecessor's late release cannot revoke a replacement's distinct +hold. Coverage is historical data durability, without current quorum, serving, +membership or disaster-recovery authorization. S3 command output settlement +still needs its operation-specific receipt/version and current routing checks. + +Remote collection snapshots one conditional registry revision, selected roots, +explicit holds, pending parents and pending generations. It expands every root's +exact dependency keys and downloads/hashes every marked object before issuing +any DELETE. Missing/corrupt protected data or metadata refuses the entire mark. +Collection excludes marked keys, complete pending generations and generations +newer than its watermark. New reservations can inherit only already-protected +parents; retired generations cannot be rebound. Thus concurrent publication +cannot resurrect an unmarked object, and a delayed DELETE cannot erase a new +copy of identical content under another generation. Unknown DELETE outcomes may +be retried against the same permanently obsolete key. Control identities, +registries, other groups and legacy namespaces are never collection candidates. +Scans/deletions and compact candidate inventories have explicit limits. + +This remote object collection does not authorize local shared-engine Raft file +purge. Controller ownership reconciliation, private-cache maintenance, S3 +settlement with bounded unarchived bytes, immutable source birth/fence inventory, +fresh group/cell incarnations and fresh-quorum serving remain required. The +fencing inventory must cover every possible old voting lineage, including joins +after the selected archived cut. Real RustFS, Docker fencing and measured-load +qualification remain later gates. + +### Controller-owned archive publication + +`CellGroupController::archive` installs archival for an already attached group. +The namespace binding must already be initialized. Registration checks the exact +birth/store route, then opens only the namespace's expected registry. One fair +archive lane on the existing node worker rotates a deduplicated dirty-group +queue, independently of SQL apply capacity. Clean groups require no roster scan. Apply/checkpoint-install events mark a group dirty. Transient +I/O/admission failures use the existing node timer; malformed metadata and lost +journal state refuse archival. Completions compare the covered cut with both the +owner applied index and consumed apply events; recovery of an older publication +cannot lose an idle final tail. This does not enable S3 command acknowledgements +or authorize local log reclamation. + +Each group has an exclusively locked node journal bound to the engine creation +identity and expected archive creation. Its immutable identity guard precedes its +initial state; missing identity/state never reconstructs an interrupted journal. +Updates write an exclusive temporary, sync it, rename, sync the directory and +read back the exact bytes. Every attempt reloads the selected state, so an error +after rename cannot roll back to stale in-memory intent. Record size is bounded +to 32 KiB; the roster and single active lane bound retained control metadata. + +| Selected journal phase | Permitted next action | +| --- | --- | +| None | Verify an already sufficient held cut, or prepare and persist one reservation intent. | +| Reserve | Reconcile/reserve that exact intent, then persist its ticket before any object upload. Definite expiry permits a newly persisted intent. | +| Upload | Capture and upload a sealed complete cut using that pending generation; physically verify it and persist the exact publication intent before CAS. | +| Publish | Reconcile only the exact existing publication. If absent, abort its pending ticket to fence a delayed CAS, then reconcile again before allocating anything new. | + +Selecting a new head first persists its explicit hold and the predecessor to +retire. Only afterward may the old hold be released; both release and the journal +clear are idempotent. Source callbacks never run in this workflow. Physical +coverage/root admission stays scoped to the job, rather than retaining a replay +workspace for every registered group. `archive_status` exposes historical metadata +and errors only; it is neither a physical coverage capability nor an S3 receipt. + +The sealed archive execution capability accepts either the full execution domain +or a controller's captured token, without acquiring shutdown rights. A controller +tracks its admitted blocking closures independently of their async waiters. +Shutdown joins logical archive work, drains those physical closures, closes private +verifier stores and releases journal locks before closing runtime/SQL ownership. +Fresh verifier stores are required on restart. No unrelated domain work is joined +through this archive-specific drain. + +### Local command S3 response gate + +`CellGroupDispatcher::with_s3` binds local command settlement to a controller in +the same execution domain. Execution also requires its exact runtime owner and +an active archive registration for the selected group/route. Effective durability +is the maximum of the request, namespace minimum and existing receipt minimum. +Remote execution uses the same receiver-side proof gate and retains its output +through physical delivery as specified below. An unbound provider still refuses. + +A committed SQL receipt resolves its producer before any archive wait. The owner +acquires independent S3 byte/item admission before transferring output retention +from SQL; it releases the resolved preparation's duplicate SQL permit. The S3 +lane uses the node's `max_commands` and `max_command_bytes` as separate bounds, +including fixed metadata and additional receipt-read bytes. It never recursively +acquires ordinary command slots. Held or uncertain S3 work cannot keep the SQL +producer unresolved or quarantine an otherwise healthy group. + +The existing fair archive lane takes at most 32 pending command proofs per group +turn. It publishes or reopens a protected complete cut, then reads each exact +receipt from that independently recovered immutable image. Coverage must contain +the same cell history at or beyond the selected committed version/position. An +equal version must carry the same decision witness. All stored receipt fields, +including operation, contract/input digests, result, durability minimum and retry +expiry, must match. A later image can prove an earlier retained receipt; a group +index or sibling progress cannot replace that read. Missing/corrupt dependencies +anywhere in the selected group refuse the output. No callback runs during proof. + +The response rechecks the selected catalog birth/route and live leader authority +before issuing its version token. Original deadlines and captured inputs remain +unchanged. Deadline/caller cancellation does not release an entered physical read; +its cancelled flag interrupts SQL, and the archive-specific owner joins the closure. +Pending requests and unconsumed responses retain bounded S3 admission. + +A successful response retains the complete physical coverage, its pool admission +and the exclusive journal lock. Immutable route authority is shared per group, +so membership records are not copied into every response. The journal owner tracks weak uses and refuses +predecessor-hold retirement while a response still retains that coverage. A +replacement owner cannot acquire that journal lock before the response drops. +Idle registrations do not retain replay workspaces. Active consumers can occupy +those shared workspaces and backpressure subsequent archival until consumption; +this is explicit pool admission, without an unaccounted metadata-only proof cache. + +Explicit unarchived-byte/disk admission, physical shared-engine Raft file purge, fenced fresh-incarnation serving, +load and process-kill qualification remain required gates. This local command +boundary is not the complete CG-06 or public SDK/fleet cutover gate. + +### Local enrollment S3 response gate + +The same explicit dispatcher binding and bounded archive lane also settle local +enrollments. Required durability is the maximum of the request and immutable +namespace minimum, captured in the invocation fingerprint. The exact runtime +owner and active archive registration are checked before entering initialization. +Remote enrollment carries the effective request requirement unchanged and returns +authenticated achieved durability with the original creation record. + +SQL resolves and closes the initializer before archival. A separately admitted +S3 output takes retention of the bounded original creation record before SQL +releases its bytes and operation slot. The complete independently recovered root +must contain that exact history, creation ID, schema and initial image commitment. +A cut preceding creation waits; a different creation refuses. Later mutations do +not replace the original record or require retaining superseded initial bytes. +No initializer runs during proof or exact retry. + +Before returning, recheck the current catalog birth, actual leader authority and +its exact committed creation record under the original deadline. The returned +`CellSqlEnrollment::durability` derives S3 only from its private retained proof. +That output retains the same physical coverage, pool admission and journal lock +as a command response. A deadline or dropped reply keeps commitment ambiguous, +without retaining an unresolved SQL producer or quarantining its healthy group. + +### Local receipt lookup S3 response gate + +`CellReceiptInvocation::select` binds an opaque quorum selection to the original +request, route and deadline. Required durability is the maximum of request, +namespace and stored receipt minimum. A configured S3 provider must own that exact +runtime and active group registration. S3 admission takes the selected bytes +before SQL read admission releases; no producer is created or retired. Selection +queues no archive work until its consuming `response` call. The existing `execute` +method runs both phases. Dropping an unsubmitted selection refunds admission. + +A complete published cut must be independently recovered. Its cell history and +logical version must cover the quorum selection; an equal version must retain the +same decision and receipt outcome. A known receipt must match every stored field +and result, even in a later image. An initially absent receipt is read again from +the actual later archive: it may return a newly committed receipt or an ambiguous +absence, with that archived image's observation and signed token. No absence, +including an S3-protected one, proves non-commit or releases original work. + +The shared receipt reader checks row length before fetching payload bytes. The +existing S3 reservation grows atomically for actual result size without another +item slot, within the unchanged node byte/item ceilings. Failure returns typed +Overloaded before payload allocation; expiry, conflict and sequence gaps remain +typed rather than becoming None. Receipt table loss remains corruption except +for legitimate initial sequence-1 images. Empty results remain distinct from +absence. No application callback or receipt maintenance runs during lookup. + +Before output, recheck current catalog birth and a fresh correlated quorum read's +leader/route fence under the original deadline. The current cell cannot precede +the archived observation; equal versions must retain the same decision. This read +fence permits later concurrent mutations without retaining a private writer's +base. A successful response retains full physical coverage, S3 admission and the +journal lock, and reports achieved durability from that private proof. + +### Remote S3 response delivery gate + +Command, enrollment and receipt RPCs carry the effective request durability. Each +purpose retains its separate authenticated route and marker-1 payload. Successful +payloads also carry achieved durability, including an ambiguous receipt absence. +The exporter-bound MAC covers the entire response digest and the exact outstanding +request tag, purpose, node/disk/process, group/cell history and schema. Decoders +reject unknown durability codes and any achieved level below the original request. +Version tokens bind command/receipt observations; enrollment binds the original +creation/history/schema/image record and does not become a current-version token. + +The receiver applies the same local proof gates above, including namespace and +stored receipt minima. A configured provider must own its exact runtime and active +archive registration. An unbound receiver refuses before mutation when S3 is +already required; a stored-minimum duplicate resolves SQL before refusing. The +sender retains its original invocation, captured inputs and deadline. Possible +transmission keeps mutation failures ambiguous; a lost or rejected reply never +proves non-commit and no transport retry invokes callbacks automatically. + +Encoding returns a bounded owned reply. S3 replies retain the native output, +complete physical coverage, S3 admission and journal/remote-hold ownership alongside +the encoded bytes; the existing transport reservation admits those bytes. Quorum +replies release their resolved SQL output as before. The physical write stage owns +the TLS stream and reply together, with stream destruction ordered before proof +release. Flush completes before normal consumption; cancellation, revocation, +deadline or task destruction closes the stream before refunding its proof. No new +per-cell task, timer or proof cache is introduced. Client outputs retain their +existing node/peer byte admission through consumption. + +The fixed `transport_query_reply_write` observation measures only response +header/body/flush, independently of archive settlement. Its bounded host-lifetime +histogram is advisory and grants no durability authority. Qualification fills a +bounded socket window with exact retries, sending each only after the preceding +physical flush/refund, without reading replies or changing node quotas. + +Qualification must observe an actual blocked TLS write, retained S3/coverage slots, +SQL release, bounded overload, a same-cell Quorum successor and physical refund on +drain, disconnect, timeout and grant revocation. Exact retries must return the +original result or creation record without callback replay. This delivery gate +does not establish outage disk bounds, local Raft file purge, fresh-incarnation +serving, process-kill recovery, independent hosts or load performance. + +## Split and merge + +Use the same transition state machine for one parent to multiple child ranges +(split) and compatible adjacent parents to one child (merge). Create fresh child +group IDs and retire every parent; this avoids reusing a partially changed parent +as a second activation path. Initially seed children on the parents' qualified +voter set where possible; ordinary learner movement can rebalance afterward. +Pre-copy immutable images while parents are live, then send the final delta after +sealing. Hibernation and membership changes must be resolved before sealing. + +Every transition pins its exact parents, range epochs, child births, final cell +partition and resource reservations in a durable catalog journal. There is a +namespace topology CAS, but no catalog I/O on an ordinary already-routed command. +The directory exposes explicit split and merge preparation over whole-cell +bounds. Both forms check the exact union and successor epoch before the fleet +admits a transition. A Live group created by an earlier transfer can reopen +its selected range as a sealed source during a later merge. Its synced seal +continues to fence application work across that restart. + +`cellar raft fleet transition basis NAMESPACE_HEX --catalog-id DIGEST` reports +the Fleet revision and directory digest for operator admission. Submit +`cellar raft fleet transition admit NAMESPACE_HEX --plan REQUEST.json --catalog-id DIGEST` +with that basis, the exact parent histories and the child reservation IDs, +incarnations, creation IDs and ordered voters. A split names a pivot cell and +two children in key order. A merge names two adjacent parents in key order and +one child. The controller stages the exact request digest and plan before the +first reservation. Retain the original request file to reconcile each child +and the final admission after a lost response. A replacement controller checks +an existing child's exact route and committed reservation receipt. For a +missing child, the attempt is bound to the current Fleet revision and original +step. An intervening Fleet update fences an older uncommitted attempt. Admission +still requires the original directory basis and rechecks live capacity, +namespace authority and the disruption slot before committing. + +Managed planning stages the canonical directory request before the fleet selects +its digest. A replacement controller can reopen the exact admitted plan by +operation ID. If the fleet selected the plan before the directory CAS, the +controller can replay the original request from the reopened record. A cancelled +operation retains its historical decision and plan. +The fleet can authorize a seal only for the exact active request and plan +after verified child preparation. Authorization and cancellation contend on +the same directory CAS. Once authorization wins, retrying the original request +returns the recorded decision and cancellation leaves the fleet slot owned. +Use `cellar raft fleet transition seal-source NAMESPACE_HEX --plan REQUEST.json --catalog-id DIGEST` +against a managed endpoint with a voter for the selected parent. The request +binds the admitted plan, exact parent group and incarnation. The handle verifies +the running boot, selected birth and applied voter membership before minting a +local permit. The Raft owner rechecks the permit and drains the accepted prefix. +A response without an applied cut confirms proposal acceptance. Retry the same +request before publication to reconcile the durably applied index, term and +authorization digest. A merge uses one request per parent. +The managed archive owner can publish the applied cut while business admission +is sealed. The fleet verifies each held cut independently against the exact +authorized plan and rechecks the complete hold set before selecting Sealed. +After the parent archive covers its applied seal, use +`cellar raft fleet transition capture-source NAMESPACE_HEX --plan REQUEST.json --catalog-id DIGEST` +on an archive-enabled management endpoint. Use a distinct request ID from the +seal request. The plan's parent slot selects the acquire intent before the +registry allocates a hold. Independent physical replay verifies the exact +authorized cut before the slot records the retained hold. A replacement +controller reopens the same intent or verified slot and captures any remaining +parents. Holds remain retained through retirement. +`FleetCatalog::replay_managed_sealed_sources` verifies all staged cuts again +before selecting the complete sealed set and preparing non-serving child genesis +images. Once every child has an installed quorum proof, +`resume_managed_transfer_archive` rebuilds the child partition in the +controller's private verification stores and records the exact archive closure. +The verifier needs bounded capacity to retain one complete cut per parent. + +| State | Durable action and admission rule | +| --- | --- | +| Planned | CAS a unique intent against the exact current range tree; reserve child/transfer resources. Parents still serve. | +| Prepared | Children are non-serving learners of transfer state; copy immutable bytes. No child has business admission. | +| SealAuthorized | CAS the journal from Prepared to an irreversible seal decision before issuing any parent Seal request. The exact retained decision grants permission to ask the parents to seal, not a quorum proof. Cancellation can no longer win. | +| Sealing | Every parent commits `Seal(transition, children, ranges)` after all earlier accepted entries. Seal covers enrollment, commands and strong reads. | +| Sealed | All parent quorums have durably applied their seal cuts. Capture and pin the exact cell/effect/receipt closure. The operation remains roll-forward only. | +| Installed | Every child's own quorum commits and applies its seed from the sealed cuts, with fresh Raft term/vote state. Children remain non-serving. | +| Archived | Publish and independently verify an immutable remote transfer closure containing every sealed parent cut, child partition/seed root and all cell dependencies. Pin it in the transition journal before range publication. | +| Published | CAS the namespace root from all exact parent ranges to all exact child ranges at once. Validate complete equal union and no gaps/overlap. Unknown CAS results reconcile by transition ID. | +| Active | Each child commits activation binding the retained publication certificate. Serving requires that activation plus current-term quorum authority. | +| Retired | Parents persist retirement/redirect tombstones. Release old data only after child archive coverage and all dependency pins are safe. | + +Use `cellar raft fleet transition setup-child NAMESPACE_HEX --plan REQUEST.json --catalog-id DIGEST` +against the management endpoint on each reserved destination voter. The +`install` step replays the complete selected parent archive set using the +server's configured verifier and syncs the pending child. Its receipt identifies +the exact member, disk and engine. After a majority of durable install receipts, +send a separate request with step `attach` to each voter. Attachment establishes +the pending Raft roster. Send step `quorum` to each child's current leader +to select its durable fresh-term witness. A replacement controller can reopen +selected witnesses by child. The directory advances to Installed only when the +complete ordered set has been verified against current Fleet storage claims. + +After Installed, use +`cellar raft fleet transition verify-archive NAMESPACE_HEX --plan REQUEST.json --catalog-id DIGEST` +on a verifier-configured management endpoint. It independently rebuilds every +child from the sealed parent cuts and compares them with the installed proofs. +Use the resulting archive digest in a separate publication request. + +The final sealed-source partition builds a non-serving child genesis marker. +It binds the exact transition, plan and sealed-source set. An active-source +pre-copy carries image pins without this genesis. Ordinary log apply and +checkpoint install preserve the pending marker and its exact image root. +The local install binds one child genesis across all its voters and persists a +member/node/disk/engine intent before syncing the mapping. Managed installs +also require the exact transition's reserved child, a current capacity proof +for the running destination boot, and the same local boot guard at the engine +write. The controller prepares only its assigned child in the retained node +stores. A retry can reconcile the exact synced mapping while the node retains +its pins. Startup parks an exact pending child after checking the rooted +transition and install intent. It verifies and pins the retained pages and +images under the new engine owner before allowing image collection. Its +physical mapping remains charged to the node without joining the Raft roster +or publishing a route. Receipt reconciliation reuses that closure and requires +capacity evidence from the current boot. +The child retains its fleet binding while pending. A matching receipt +follows the sync. A read-only check confirms a majority of reserved voters +from matching intents and receipts on distinct pinned disks and engines. For +a managed child it verifies the original active fleet plan and reserved route +and matches the physical receipts to the fleet's permanent storage claims. +A pending voter can attach after rechecking this majority and its own receipt. +The local applier uses the exact page and image owner handed to its engine. +Managed attachment also rechecks the current boot and admitted capacity and +installs only authenticated voter routes. The controller retains the same +physical page owner across an uncertain install reply and hands it to the +pending Raft roster. After a fresh-term barrier commits and applies, the +pending leader can issue a quorum witness while the child remains inactive. +Raft can elect a leader and commit in a fresh term while the pending manifest +fences cell commands and reads. Only the child leader can attest that its +stable initial voter configuration committed and durably applied a fresh-term +position with the exact pending manifest. The directory verifies the witness +against the installed genesis and the leader's physical receipt. It roots the +whole ordered child set in one CAS and can reopen the selected set after an +ambiguous response. The managed catalog rechecks each reserved child and its +voters' permanent fleet storage claims before that CAS. Installed children +remain fenced and the parent routes stay selected. Directory publication and +activation require their own proofs. + +Child install receipts bind the source seals, own committed seed, membership and +manifest digest. Archive verification independently replays every selected parent +hold and rebuilds the child partition in dedicated stores. Every rebuilt root and +transfer origin must match its installed child. The directory rechecks the live +holds and selects an immutable closure that binds the two exact evidence sets. +For a managed namespace, the fleet catalog checks the active plan and the +reserved children against a single fleet snapshot before replay. The directory +binds the closure to the same installed set at its CAS. The closure can be +reopened from remote metadata after a lost response or voter disk loss. Recovery +replays the retained holds and reconstructs the same roots. +Publication requires this closure so losing every voter immediately afterward +still leaves a recovery path for prior S3-protected effects under the child +partition. A controller observation without +those proofs cannot publish. S3 outage may stall topology changes; it does not +change ordinary Quorum acknowledgement behavior. +Publication verifies that the original archive holds are still retained. A single +directory CAS selects all child ranges, the Published decision and the immutable +certificate binding the archive closure. An exact retry can recover that +certificate after a lost response. The same CAS roots a permanent birth fence +for every child. Ordinary initial bootstrap refuses those children even when a +separate install receipt is unavailable. Managed publication rechecks the +active fleet plan and the exact storage of the reserved child voters. + +A managed pre-copy controller stages its exact active source holds before +opening physical replay receivers. The directory selects their digest on the +same head used for preparation and cancellation. A replacement controller +reopens those descriptors, independently verifies their archived cuts, and +rebuilds the non-serving child partition in its own inventory. Preparation +compares the selected digest again at its CAS. A cancelled plan retains the +descriptors so their owner can reconcile and release the holds after joining +physical work. + +For operation-owned preparation, the controller first selects a plan-ordered +acquire intent set in the directory CAS. It sends only the selected intents +and stages the resulting holds against that same set. It can replace an +intent only after the registry proves that its revision was consumed and its +nonce owns no hold. Cancellation fences new intent selection. Once physical +pre-copy work has joined, cleanup reconciles each exact acquire. It releases +an acquired hold or consumes the pending revision before a delayed acquire +can create one. A replacement controller can repeat that cleanup using the +cancelled journal. + +Use `cellar raft fleet transition prepare NAMESPACE_HEX --plan REQUEST.json --catalog-id DIGEST` +with the admitted operation ID and plan digest. The admitted principal needs the +Fleet topology grant. A trusted management server supplies a private archive +verifier and bounded restore pool through `RaftFleetControl::with_transition_preparer`. +Preparation allocates or reopens the operation's held archive cuts, verifies them, +and returns the durable preparation digest. Retain the exact request across +timeouts. After preparation commits, an exact retry can return the selected +digest from the journal. Physical replay does not authorize a source seal. + +Use `cellar raft fleet transition authorize-seal NAMESPACE_HEX --plan REQUEST.json --catalog-id DIGEST` +after the transition reaches Prepared. The request binds the admitted plan and +the digest returned by preparation. The admitted principal +needs the Fleet topology grant. Authorization and cancellation compete on the +same directory head. An exact retry observes the selected seal decision after +a lost reply. Each source still commits its own seal under its live voter +authority. + +Use `cellar raft fleet transition publish NAMESPACE_HEX --plan REQUEST.json --catalog-id DIGEST` +after the transition reaches Archived. The request binds the admitted plan +and the exact transfer archive digest reported by transition status. The +operation's principal needs the Fleet topology grant. An exact retry returns +the original publication certificate and directory revision, including after +the transfer has retired. Children +validate their exact transition and refuse replay after later retirement. The +managed activation permit binds the selected route and install receipt to the +current boot and initial voter configuration. The child Raft owner admits the +exact certificate and seed commitment only after its fresh-term barrier. Committed +apply retains the certificate, seed commitment and activation position in the +child manifest. Subsequent mutations and checkpoints preserve that evidence. +During an active managed transfer, restart reopens the exact activated voter +from its permanently claimed engine, revalidates the publication and current +boot, and uses its existing placement. Until the child archive is initialized, +the held source closure remains available for recovery. Explicit S3 +acknowledgement requires child archive coverage. A selected child can serve +only after activation applies. The +directory selects Active after receiving one durable acknowledgement for every +child in plan order. Each applied acknowledgement can be staged independently +against the exact publication and installed seed. A replacement controller can +reconcile the stored acknowledgements and select the complete set after an +ambiguous CAS. + +Use `cellar raft fleet transition activate NAMESPACE_HEX --plan REQUEST.json --catalog-id DIGEST` +against a managed voter endpoint after publication. The request binds the child +group and incarnation to the admitted plan. An accepted proposal is provisional. +Retry the exact request to obtain its durable applied position. The operator's +Fleet topology grant and the original plan's principal are required. The topology lane +remains owned until parent retirement. After the route switch, a retained +parent voter reopens from its historical birth only if its synced seal matches +the exact selected transfer. This keeps the parent fenced and lets it +participate in the source quorum for retirement. The parent leader can commit +its tombstone only after the directory selects the complete Active set. The +entry binds the replacement publication to the original seal, and the manifest +and checkpoint preserve both. This tombstone leaves the topology lane and parent +archive holds pinned while child archive coverage is qualified. Once the +transition is Active, run `cellar raft fleet transition setup-child NAMESPACE_HEX --plan REQUEST.json` +with the selected child and the `initialize_archive` step. The operator +endpoint records one archive identity for that plan and child before creating +the registry counters. A retry verifies the selected binding and counters. +An incomplete registry requires archive authority recovery. A retained voter +opens the initialized registry and publishes coverage from its applied state. +Run `cellar raft fleet transition verify-child-archives NAMESPACE_HEX --plan REQUEST.json` +after all active children publish archive cuts. The controller persists one +acquire intent per child and independently replays each held cut. Each cut +must retain the installed seed and the selected publication and activation +position. The directory rechecks all holds before selecting the complete set. +A retry can reopen the selected digest after a lost response. The parent holds +and topology lane remain pinned for retirement settlement. +Use `cellar raft fleet transition retire-source NAMESPACE_HEX --plan REQUEST.json` +for each parent once child coverage is selected. A voter commits the tombstone +against the selected replacement publication. The controller records each +parent's position separately so a fresh controller can reopen a completed +step. The directory selects the Retired decision only after every parent +position and the live child holds are verified. The fleet +owner can root the Retired decision only after rechecking the live child holds +and binding every committed parent tombstone to the coverage and publication. +During cleanup, previously activated child voters may still reopen on their +claimed engines. New install and activation remain fenced. Child confirmation +preserves all parent references. The fleet owner confirms every +child Live in a single rooted commit after rechecking installed storage, +retirement and live child coverage. A Live child reopens using its fleet +membership, permanent engine install receipt, and the selected directory's +retirement and activation binding, independent of the transition slot. Archive +setup is required for that startup when archiving is configured. The owner +can reopen a retired parent disk in parked form after a joined process stop. +Startup checks the exact local tombstone against the immutable retirement +set and keeps that history out of the Raft and transport rosters. Authenticated +physical inventory identifies the parked source by history, birth, member and +boot. The disk and controller slot remain charged. The retirement record keeps +the final committed member and node bindings, including the disks assigned to +joined members. The fleet checks every member against its node's current boot +and parked inventory before releasing the selected preparation and sealed holds. +All parents on the same node must match the same authenticated inventory sample. +It rechecks that the children are Live and their archive holds remain valid. +The directory releases only the original transfer holds and confirms +each release before clearing topology ownership in a separate CAS. The fleet +then records every parent as retired and clears its disruption slot in one +root update. The retired row binds the operation and retirement set, keeps the +original birth identity, and stays in inventory. Retained engine slots still +consume capacity, while membership work and topology scheduling use Live +groups. An +interrupted cleanup resumes the same immutable decision. Later transitions +preserve historical publication certificates and reject stale child startup. +Source tombstones remain effective across restart, delayed messages and catalog +unavailability. The whole transition is one fenced owner operation; concurrent +controllers reconcile the same journal instead of selecting different children. + +`cellar raft fleet transition status NAMESPACE_HEX --operation-id OP_HEX --catalog-id DIGEST` +reads the admitted plan digest, directory phase and Fleet slot at its observed +revision. Use the Fleet catalog ID from `cellar raft fleet status`. Directory +cleanup can finish while the Fleet slot is still held, so the phase and slot +are reported separately. + +For a retired transition, submit +`cellar raft fleet transition resume NAMESPACE_HEX --plan REQUEST.json --catalog-id DIGEST` +as the principal that admitted it. The request binds the catalog, namespace, +operation, plan digest, expected Fleet revision and request ID to either +`confirm_children` or `complete`. Confirmation promotes the verified children +together. Completion claims the exact request, checks all parked parent replicas, +releases the archived holds and commits Fleet retirement. Retain the request +file and use the same request ID after an uncertain reply. An authenticated +node observer supplies the parked proofs while the holds are outstanding. + +| Failure | Required behavior | +| --- | --- | +| Controller lost before seal authorization | Resume or CAS Prepared to Cancelled; seal authorization and cancellation compete on the same journal state. Revoke child activation permits before releasing reservations. | +| Seal reply lost | Treat outcome as unknown; reconcile committed log/receipts. Do not reopen a parent based on timeout. | +| One merge parent sealed, another unavailable | Sealed parent remains fenced; unsealed parent's earlier writes join its eventual seal cut. No destination admission. Resume when quorum returns or use the disaster protocol. | +| Child install incomplete | Keep parents sealed and all retained pins. Repair/finish the same child or use authorized disaster recovery. | +| Transfer closure upload/verification incomplete | Do not publish child ranges. Keep parent and child pins; retry the same immutable closure. | +| Directory CAS response lost | Read journal/root and confirm the exact winner. Never roll back a possibly published transition. | +| Publication succeeds, some children not active | Only activated children serve their disjoint ranges; finish activation. Parent never resumes. | +| All disks lost immediately after publication | Recover from the journal-pinned remote transfer closure plus any archived child tails; no newly acknowledged S3 effect may disappear. | +| Delayed old proposal/read/activation | Check history, epoch, transition and local sealed/retired state; refuse stale authority. | +| Parent disks fail after publication | Children serve from installed state. Parent deletion does not establish archive safety. | + +After SealAuthorized, automatic cancellation is forbidden, even if no Seal reply +has arrived. Parent Seal admission requires that exact authorization certificate. +Availability can pause while completing a transition; preserving single-writer +ownership takes precedence. Pre-seal cancellation also needs a durable controller +decision in the same journal. Cancelled intents can never mint seal/activation +permits. Unknown authorization-versus-cancellation CAS results reconcile the +recorded winner before issuing work; timeout cannot establish that cancellation won. + +Normal transfer preserves cell incarnation, mutation sequence, exact receipt +contents/client floors and effect identities. Seed provenance authenticates the +original commit witnesses. New group positions do not replace per-cell ordering. +In-flight outbox delivery is settled or journaled with the original ID; a lost +delivery response never authorizes a new identity. The target may retry that ID +under the receiver's existing dedupe contract. + +## Adaptive partition observations + +Partition observations read cell count, image bytes and whole-cell pivots +from the selected manifest. The group owner reports newly admitted writes, +durably applied cell writes and the age of its oldest committed apply entry. +Rate calculations require two samples from the same owner, route, term and +membership. A four-key sketch tracks exact cell names within one-second +windows. Its counts are lower bounds. A rate sample uses a hot-writer window +only when it contains the entire window. A two-second sampling cadence allows +a complete window at every offset. + +The controller supplies the current directory digest and measured placement +pressure. The resulting recommendation goes through managed transition +admission. The Fleet owner rechecks the source routes, membership and term +with quorum observations before staging new reservations. + +## Coordinated hibernation + +Closing an execution cache is independent from parking the Raft group. For the +first hibernation implementation require all configured voters to participate; +a healthy majority can continue active work if another voter is unavailable, but +cannot park. This is a conservative availability tradeoff, not a Raft safety rule. +Learners must have an explicit quiesced/catch-up state; no unbounded transfer may +be abandoned merely to park. + +1. The current leader checks no unresolved proposal/read, transition, membership + change, snapshot install or unsafe effect dispatch exists. Freeze new admission + while negotiating. Publish a bounded next-due-effect summary to a node wake owner. + If the negotiation expires and the leader still holds its term, it commits + and applies a no-op before reopening admission. This advances the cut and + closes any local acknowledgement for the expired generation. +2. Commit `PreparePark(generation, term, membership_revision)` and apply through it. + The committed entry index is the generation. The applied marker preserves the + cut across snapshots and archive replay. Every voter syncs a local acknowledgement + for the exact generation and cut before a park certificate can be selected. +3. The leader assembles all acknowledgements against the current physical voter + roster. Each acknowledgement binds the member, node, disk, process, cut and + sync sequence. The leader syncs one immutable certificate for the generation + and its own parked record before delivering the certificate over authenticated + transport. Each recipient checks the exact physical roster and persists its + parked generation before acknowledging delivery. During partial delivery, an + active peer's append or election requests wake parked peers. Delayed replies + from the last active term carry no work that can wake a parked voter. +4. API request, peer message, due effect, topology operation or local restart + wakes the group. Persist closure of that park generation before processing + queued protocol or application work. The closure retains a wake floor so + restart and delayed certificate delivery cannot bypass it. +5. Resume ordinary Raft using retained term/vote/log. Clear soft leadership and + require a current-term applied position beyond the wake floor before serving. + Send wake hints to peers. Missing a quorum refuses or waits. + +Managed negotiation takes the fleet's rooted disruption slot at the exact +leader boot, applied position, term and membership revision. A later request +must still match that base. Settlement releases the slot after authenticated +durable delivery to all voters, or after a quorum observation proves a later +term or an applied position beyond the reserved park cut. A timeout alone +does not settle the rooted operation. + +On a managed voter, use `cellar raft fleet group park NAMESPACE_HEX --plan REQUEST.json` +with the fleet catalog ID and an exact group and incarnation. The plan supplies +distinct admission and settlement request IDs and the fleet revision to reserve. +`group park-status` reads the rooted operation without contacting voters. +`group wake` targets the same generation and waits for a fresh quorum read. +Use the same plan and principal for retries. An interrupted admission can be +settled by wake after the leader durably advances beyond its reserved cut. + +The node keeps parked voters out of the Raft tick roster. Its deadline index +holds one earliest effect per parked history and releases due wakes in bounded +batches on the execution-domain timer. A restart may conservatively wake all locally parked groups +in bounded batches. Never reset term/vote or initialize a missing disk. Include +park state/tombstones in local recovery and checkpoint contracts. The scheduler +must reconcile alarm changes after leader loss, and an overdue alarm wakes the +group immediately rather than waiting for a client. + +Required fault cuts include lost prepare/ack/park/wake messages, delayed duplicate +park after a new command, leader loss while parked, minority wake, mismatched +membership revision, alarm changes during negotiation and all-voter restart. +Measure healthy retained-state wake separately from election, image-cache restore +and S3 disaster recovery. Do not claim wake percentiles from a few successful trials. + +## Zero-quorum recovery and transition closure + +Recovering a live range changes authority, so serialize it with namespace topology +and fleet lifecycle operations. Pin an exact recovery set; verify its full remote +closure before requesting infrastructure fencing. Never choose a newer head after +the operator request was admitted. Existing Quorum versus S3 recovery-point limits +continue to apply. + +If the group participates in an incomplete split/merge, recover the connected +transition component, including every parent/child that could still activate. +Resolve the durable publication journal first. Fence the launch authority and all +possible voting lineages in that component, including removed/replacement hosts +whose restart could violate the selected boundary. Partial fencing cannot publish +new ownership. A list of current online nodes is not a complete fencing inventory. + +The Fleet SDK's `transition_recovery_basis` reads the active disruption and +the exact directory publication against a pinned root. It identifies the archive +source groups and reports both parent and child histories for inventory collection. +The trusted recovery verifier independently replays the retained head for +each selected range and checks the Fleet and directory roots again. A retained +older hold cannot cover a newer S3 acknowledgement. The verified result +includes required lineages from both sides of the transition, including initial +members, retained membership bindings and permanent movement admissions. Fencing +must revalidate the exact cuts and account for the platform's launch history +before changing authority. `stage_transition_recovery_intent` saves the exact +roots, holds and required lineages under a separate recovery operation ID. +A restarted controller can reopen that selection even after a topology change. +The recovery barrier selects the intent in the same directory CAS that orders +transfer publication. It closes startup for both sides of the transfer and +retains the Fleet disruption slot. Infrastructure fencing still has to revoke +launch authority and stop all old writers before replacement publication. +The managed fence request binds the intent digest and the selected directory +barrier. Before contacting the authority, the controller reconstructs the +locked transition scope and independently replays every selected cut. A +conditional create retains the independently verified authority receipt. +A retry reopens that receipt and verifies it with the authority again. +The controller checks the barrier and cuts again before returning proof. +If a late archive write advanced a cut, the receipt survives but that +selection cannot authorize restoration. +The replacement plan retains a fresh birth and placement for each selected +range, bound to that receipt and the exact directory mutation. A controller +can reopen the plan after losing the write confirmation. Each source image +is independently copied into its selected replacement store. The new cell +history starts at sequence one, with the original SQLite bytes and fresh +genesis witnesses. Fleet reserves each replacement under the existing +disruption slot. Its rooted receipt binds the exact original voter boots. +The reservation excludes old node, disk and physical-host identities and +charges tracked history and movement slots against each current capacity +attestation. The directory reserves the group identity with its recovery +origin so ordinary birth cannot initialize it. The voter seed turn checks +the exact reserved boot and running engine bounds. Each voter prepares data +in its configured page and image stores under that engine identity. The writer rechecks the +boot guard, installs the fresh Raft genesis with managed membership, and +verifies the synced image closure before creating a durable seed receipt. +An exact retry reopens the same receipt and requires the same engine state. +Before publication, Fleet reopens the exact reservation for every range and +verifies a majority of durable seed receipts against the genesis binding, +storage pins, and current qualified voter boots. It retains the selected +receipt set so a later voter completion cannot change a retried decision. +The recovery publication replaces every selected range in one directory CAS. +The new head binds the exact seed set and keeps the startup fence on the old +and replacement groups. A lost response reopens that publication without +selecting another archive cut. The node's recovery attachment entry point +reopens the selected publication by durable IDs. It checks the voter's original +seed receipt, the current qualified boot on the same disk and engine, and the +complete retained page and image closure before starting Raft. Managed startup +can reopen a synced seed into its parked image set after process loss. It checks +the recovery lock, reserved history, seed intent and pinned disk and engine. +The parked owner keeps its images and capacity charge outside the transport +roster. A new boot must qualify before its disk's seed can count for publication +or attach after publication. The voter can then replicate a fresh-term barrier +while application admission remains closed. Each range's leader must commit +activation after applying a fresh-term barrier. A voter records the exact +committed position against the selected publication and seed commitment. +A retry reconciles a lost receipt confirmation against the same position. +Fleet verifies every selected range's activation receipt before promoting +any replacement history. It materializes the exact directory births for +each batch before one Fleet CAS advances up to 15 histories and their +protected cursor. A retry reconstructs the last batch from the rooted +histories and its commit journal. After the final batch, the directory +checks the rooted Fleet settlement and every replacement history. One +directory CAS records the completion receipt and supersedes the original +transition. Operator status exposes that receipt alongside the held Fleet +slot. The release step first selects or opens the exact archive binding for +each replacement. Under that fence it creates the empty registry before +publishing the immutable archive identity. A retried controller can finish +either step while the history is still fenced. An established identity with +missing counters refuses to open. A separate CAS selects the exact +completion receipt for serving and removes the directory recovery lock. A controller can reopen that decision +after a lost response. Existing fenced owners remain parked for application +work until they restart. Retained original voters recheck the current route +and Fleet boot, the same disk and engine, their durable seed, and the exact +committed activation before joining the serving runtime. Raft must still +establish current-term quorum authority. Fleet then verifies the exact +serving release, the superseded routes, and the permanent infrastructure +fence before retiring each batch of old histories. The final Fleet CAS +retires the last batch and releases the original disruption slot. Retried +completion reopens the recorded cursor and original revision. Retained +physical occupancy and source archive holds keep their independent charges. + +For an unpublished transition, choose complete archived parent cuts and reconstruct +their non-overlapping domain under new groups. For a published transition, require +child recovery sets or a retained transfer closure plus each child's archived +tail; a parent checkpoint alone cannot cover child acknowledgements. Refuse if the +selected closure cannot account for every S3-acknowledged effect in the retained +recovery horizon. An intentionally older rollback needs explicit operator scope. + +Create fresh group and cell incarnations, node/member/disk/process identities and +Raft genesis state. Restore SQLite, receipts and client floors without callbacks. +Publish replacement ranges through one fenced topology mutation, then require a +fresh quorum and committed activation. Managed fleet import/maintenance/recovery +must share that same mutation journal and resource debt; no unmanaged bypass. +Unknown fencing/publication outcomes resume the identical request. They never +unfence old hosts or select another cut. + +The mandatory RustFS drill deletes all owned voter disks after independently +checking the ledger and archive cut. It verifies every cell, old-token rejection, +receipt retries, new writes and retained-disk restart. Repeat during split/merge +crash cuts and after physical log reclamation. This remains group/component +recovery, not a globally consistent snapshot of unrelated groups or external effects. + +The unmanaged recovery fence barrier is implemented separately from replacement +installation. `RangeDirectory::prepare_group_recovery` binds the exact source +birth, dedicated archive hold, replacement birth and expected topology mutation +in canonical marker-1 bytes. Persist those bytes and retain the hold before +`fence_group_recovery`; resume the identical bytes after an unknown outcome. +The barrier independently verifies every held archive dependency before claiming +the namespace or asking infrastructure to fence it. It refuses joint, repair +and fleet membership records at this static unmanaged boundary. + +A recovery claim shares the directory-head CAS with ordinary topology publication +and the namespace lifecycle CAS with fleet import. Once selected, its source +cannot bootstrap again and competing topology operations refuse. The existing +`DisasterFenceAuthority` contract accepts a typed `CellGroupDisasterFenceRequest`: +the deployment controller must permanently stop/revoke all possible old writers, +including later joins and removed hosts, and independently verify evidence bound +to the complete request digest. Cellar stores the winning bounded receipt and +independently verifies it again on resume. No permissive controller is supplied. + +`FencedCellGroupRecovery` retains verified data under the original histories. +It grants no engine, voting or serving permission and does not release the +topology lock or publish a replacement. Fresh-history image/receipt rebinding, +fresh-genesis installation, activation and fresh-quorum admission must complete +the recovery protocol above. The contract fixture and real RustFS CAS tests do +not qualify Docker host fencing or the deleted-voter serving drill. + +## Archive-aware local checkpoint compaction + +An archive registration first makes the local group's retention requirement +irreversible through an engine-synced metadata write. The required marker-1 +`archive_required` field belongs to the local group application record; a peer +snapshot neither grants a frontier nor clears this policy. Missing-field records +refuse; no legacy decoder, migration or marker increment is introduced. + +After opening a retained disk, that requirement blocks local checkpoint +compaction until the journaled remote hold has been independently verified again. +The controller advances a private engine-local frontier only from complete archive +coverage. Public `archive_status` remains advisory. Equal-index coverage must +match the exact local root and term; newer coverage cannot be replaced with older +or conflicting evidence. An intent reconciled behind the retained checkpoint +keeps compaction closed until the next verified cut reaches local history. + +An accepted registration owns a pending activation intent across retries. It +closes later checkpoint admission while earlier work drains. The Raft phase and +its storage batch share an exact engine/group/cut-bound lease; storage may finish +that earlier work while the intent is pending. Raw persistence cannot manufacture +or substitute the lease. The requirement is synced only after all older leases +drain, preserving the checkpoint's sole source slot without letting new captures +overtake activation. Capacity refusal is retried at this boundary. + +An unentered intent releases its barrier when registration is rejected or closed. +An entered metadata write retains the intent independently of its waiter and +publishes the irreversible requirement after physical sync and cache/root +publication. Unknown I/O closes the engine rather than completing the intent. +An existing durable bit follows the same drain and frontier-reset boundary. +Frontier reset happens once before I/O; a delayed duplicate completion cannot +discard subsequently verified coverage. No mutex guard crosses an await or +physical I/O. These owners are local capabilities and add no serialized field. + +The automatic checkpoint scheduler checks the frontier before capture, letting +quorum apply and another group's checkpoint lane progress during archive delay. +The actual owner checks the exact captured cut again; the storage worker repeats +the check before any snapshot compaction batch. An already prepared local +checkpoint cannot bypass the configured requirement through direct persistence. +The frontier retains only bounded marker metadata; the durable archive journal +owns the current remote hold, and no idle per-group replay workspace is kept. + +This gate covers local checkpoint publication. Full-backend outage retention +admission, Raft file purge, peer-install integration of deployment archive policy, +fresh-quorum recovery and the public hard cutover remain separate requirements. + + +### Engine-local retained entry ledger + +The shared-group storage engine owns one node ledger and one small group record +per physical history. `ReplicaLogOptions.cell_group_retention` supplies explicit +node/group byte and entry ceilings and bounded progress sublimits. Ordinary data +cannot consume the progress allowance for Raft's empty election/automatic leave +entries. Per-entry charges include fixed bookkeeping. Advisory debt observations +are not physical file measurements, durability proofs or admission capabilities. + +Duplicate receipts resolve before new callback admission. A new command or +initializer must obtain an owned worst-envelope reservation before executing user +code. Encoded preparation shrinks that reservation; delayed submission, ambiguous +outcomes and entered physical work retain it. Definitively sealed pre-append +rejection and authoritative receipt reconciliation may release unused leases. + +Before RawNode mutation, the owner admits local proposals, inherited peer entries +and possible election no-ops. Automatic joint leave receives promised capacity +before the configuration can require it; apply must not seek new admission after +its durable configuration is selected. Read-index envelopes are not log appends. +Heartbeat/read/completion work preserves its independent lanes at saturation. + +Persistence validates exact engine, group and entry bindings. Its selected cache +and root state precede an atomic reservation-to-debt transfer. Failed preflight +cannot change durable debt. Unknown physical outcomes close the engine; reopen +reconstructs actual retained debt without trusting old in-memory reservations. +Conflicting suffixes and local checkpoints retire only actual selected entries. +Archive coverage/status alone never releases ledger debt. A smaller restart limit +preserves selected entries and closes new admission until a reservation fits. + +This contract is being implemented and qualified. It does not by itself prove +physical disk bounds, full-backend S3 outage authority, fenced fresh-quorum serving +or the complete CG-06 gate. Those original requirements remain unchanged. diff --git a/docs/design/multi-raft/cell-groups/qualification.md b/docs/design/multi-raft/cell-groups/qualification.md new file mode 100644 index 000000000..315ca9cea --- /dev/null +++ b/docs/design/multi-raft/cell-groups/qualification.md @@ -0,0 +1,357 @@ + + +# Qualification and evidence contract + +Purpose: Failure oracles, workload profiles and evidence required for release. + +Status: proposed workload contracts and release gates, not measured results. +`profiles.json` freezes candidate requirements. Change a missed target only through +an explicit reviewed profile revision, retaining the old run and its failure. +Do not silently raise budgets, remove failures, retry mutations with new IDs, or +compare different logical cell populations to manufacture an improvement. + +## Tests and release levels + +Every task has exact named native cases. Its executor brief first lists tests, +then runs each required case with `--exact --test-threads=1`. The evidence validator +requires the name in the inventory and a successful one-test result; Cargo exiting +zero after selecting no tests is a failure. These test targets are planned outputs, +not available commands until their owning task creates them. + +Use real SQLite/LTX and Raft Engine for state assertions. Scheduler-only fixtures +with stub image markers can test fairness but cannot qualify durability. Both +materialized and extent state, empty enrollment, managed and unmanaged groups, +all read modes and mixed Quorum/S3 commands must appear in the matrix. Native +test execution is serial between fixtures, concurrent within each fixture. + +| Level | Evidence | Claim allowed | +| --- | --- | --- | +| Package | Python validator/tests, source preflight and local links | Executable plan only | +| Native | All task cases on real engines plus injected crash/process cuts | Exact tested functional boundaries | +| Local deployment | Five owned Cellar containers/processes, actual RustFS, enforced resource limits and complete ledgers | Local measured workload only | +| Independent domains | Actual separate machines/failure domains and observed physical isolation under load | Only the measured topology/domain-loss envelope | +| Release | All tasks, regressions and required profiles at one immutable source | Complete cell-group capability within that envelope | + +The final release requires all profiles marked required in `profiles.json`. +Container counts alone cannot satisfy the independent-domain profile. Million-cell +and Q2–Q4 scale claims remain separate larger experiments from the parent roadmap. + +## Managed S3 component verification + +The managed native target now implements these explicit backend cases: + +- `bootstrap::lifecycle::rustfs_managed_s3_group_replacement_and_joined_restart` +- `bootstrap::lifecycle::maintenance::rustfs_managed_archive_maintenance_stop_return_and_resume` +- `bootstrap::lifecycle::maintenance::loss::rustfs_managed_maintenance_recovers_one_lost_membership_observation` +- `bootstrap::lifecycle::s3::rustfs_managed_s3_seed_refusal_and_source_failover` +- `bootstrap::lifecycle::isolation::rustfs_managed_isolated_voter_s3_replacement_retains_disk_and_fence` +- `bootstrap::lifecycle::recovery::rustfs_managed_failure_recovery_restarts_and_verifies_s3_before_removal` +- `bootstrap::requests::rustfs_managed_native_axum_read_write_and_retained_restart` + +Run each by its exact name with `--ignored --exact --test-threads=1` in +`native_cell_groups_managed`. Configure an owned local backend with +`CELLAR_CELL_GROUPS_RUSTFS_ENDPOINT`, `CELLAR_CELL_GROUPS_RUSTFS_BUCKET`, +`CELLAR_CELL_GROUPS_RUSTFS_PREFIX`, `AWS_ACCESS_KEY_ID` and +`AWS_SECRET_ACCESS_KEY`. The helper accepts only a loopback HTTP endpoint, +requires an isolated prefix and supplies explicit credentials; absence fails. +Use a fresh prefix for every run. Never copy the private credential file into +evidence or a PR. Build first, record and copy the executables, then run cases +serially after writeback. Record and verify the pinned backend image, actual +resource limits and signed conditional-write/read-after-write probe separately. + +Both image forms run. Replacement checks real S3 payload transfer, independent +SQLite histories/receipts, joint finish, lost settlement replies, owned drain and +retained joined-member restart. Maintenance covers initial and joined voters, +owned stop, offline confirmation, paused return and fresh resume. The fault case +denies target payload GETs before attachment, requires retained nonserving owner +and debit, refuses a peer-mode change, stops the source service and requires a +fresh election and completed repair. Replacement-local queries verify actual +responder, values and versions; new writes require S3 proof. A surviving original +voter's missing physical inventory must block catalog settlement and retain its +debit even after membership repair completes. + +The isolated-voter case first joins the complete obsolete Node, then supplies +fixture isolation evidence. Reject evidence before rooted exclusion and reject +the wrong process binding without advancing the root. A saved pre-isolation +startup capability must return the typed catalog conflict, with both fleet root +and retained engine files unchanged. Replace that voter from S3, require fresh +stable RF3 and the exact `isolated_sources` receipt, lose the actual settlement +confirmation and reconcile the same request. Check both restored cell +images/receipts through replacement-local queries, explicit S3 writes, the +replacement's physical capacity floor and byte-identical retained obsolete log. +This fixture does not supply permanent Docker launch fencing or terminal node +removal. Keep the original eight reconciliation turns and five-second calls. + +The managed HTTP case uses the counter's existing native and Axum adapters with +independently bound applications and catalog clients on three actual managed +Nodes. Cold startup must qualify capacity, reserve/publish/confirm group birth, +activate initial voters and replicate enrollment before HTTP admission. Check +independent cell state, immutable retry receipts, conflicting-input refusal and +cross-cell token refusal. Require a follower command refusal before explicit +write-origin admission, reject foreign namespaces and stale processes, revoke +one current process and require no mutation, then restore permission with a +newer revision and retry the same command. Require explicit S3 acknowledgement +through the follower as well as ordinary quorum responses. Force remote +minimum-version and linearizable reads +through every adapter/node, requiring the actual responder identity. Join the +leader, require successor service, then restart retained disks with new process +generations and require old receipts/version floors plus new writes. After two +joined voter stops, observe actual quorum loss and require both adapters to +refuse writes and linearizable reads. Shutdown joins every server and Node and +accounts for unresolved work; failures retain all voter directories. Both image +forms run. The same fixture supplies `ingress_cannot_cross_cell_authority` in +`native_cell_groups_public`, with the managed exact test names retained. These +local embedded Nodes and fixture capacity attestations are separate from the +complete lifecycle, independent voter-process, permanent fencing, +crash/partition and load gates. + +The automatic recovery case uses the Node-owned failure service with the original +failure grace, probe interval and five-second turn limits. Hold an exact-boot +isolation callback, verify repair has not begun, refuse a duplicate worker, then +shut down and require +joined callback cancellation and released control admission. Restart the service +through S3 replacement while the independent verifier refuses payload GETs. +Require fresh replacement inventory, no terminal receipt and unchanged obsolete +disk. Stop again, restore reads and restart: require additional physical S3 GETs, +exact-operation terminal removal, no repeated committed isolation, empty movement +and reservation books, original receipts/new S3 writes and unchanged retained +obsolete disk. Both image forms start with cold private verifier inventories; +no business mutation drives the repair. These fixture attestations do not qualify +Docker launch revocation or the complete public lifecycle gate. + +The lost-observation regression uses independently owned catalog clients over the +same actual backend. After handoff, hold one foreground local-member catalog read +without resuming it; background trust refresh and the controller's client remain +unaffected. The walk must return a transient refusal, retry that pending history, +and verify all three members within the original five-second renewal envelope. +Both image forms require one handoff and exactly one verified history. This case +ends at renewal; the unchanged full lifecycle case separately qualifies owned +stop, return, resume and capacity settlement. On failure, bounded read-only local +status and admission snapshots are recorded before cleanup; the original error +still fails the test. + +Initial/final test handoffs reconcile actual leadership under one original +five-second envelope. Re-observe the current leader, capture its term and +membership revision, and use guarded transfer admission. NotReady or a lost +admission response requires observing actual state again; acceptance alone is +not election completion. Successful handoff still requires the target's actual +ready leader state. The original eight movement turns remain unchanged. + +The fault case also blocks actual registry mutations after target payload +refusal. Require a witnessed denied write, retained pending journal and observable +cleanup error. Restore writes without issuing another learner request: the node's +idle owner must confirm retirement and sync the empty record. A second failed +retirement remains pending through source-service close; restore writes only after +that source joins so the surviving quorum can repair. Preserve the shared-backend +effect when the first fixture owns the underlying store. + +After every old host joins shutdown, reopen only that source on its retained disk. +With registry writes denied, background discovery must find the original engine- +and archive-bound intent without a live quorum or another learner request. Restore +writes and require durable retirement without changing fleet authority. Gate an +actual archive read, start close, verify sealed admission and retained S3/sweep +ownership, cancel its waiter, then release the read and require joined shutdown. +Sample retained cell inventory before close seals engine I/O admission. Both image +forms run. Separate journal tests reject a foreign engine, ordinary archive-journal +purpose, missing state and missing identity; existing-only open cannot reconstruct +those records. + +These are embedded native Nodes with separate domains, disks and TLS. Real RustFS +verifies the S3 adapter; it does not supply independent or resource-constrained +voter hosts. A source-service stop is distinct from process SIGKILL and permanent +platform fencing. Retain every missed five-second readiness window and failed +run; later passes cannot establish a fixed failover SLA or waive that failure. +Five-second calls and eight reconciliation turns remain unchanged. See +[progress](progress.md) for source-bound results and unresolved qualification. + +## Planned harness interface + +CG-14 implements `tools/perf/cell_groups.py`; it does not exist at plan delivery. +The following is its required CLI, not a command claimed to work today: + +```text +python3 tools/perf/cell_groups.py preflight --config PRIVATE_CONFIG +python3 tools/perf/cell_groups.py run --profile PROFILE --config PRIVATE_CONFIG --output FRESH_RUN_DIRECTORY +python3 tools/perf/cell_groups.py verify --run RUN_DIRECTORY +python3 tools/perf/cell_groups.py stop --run RUN_DIRECTORY +``` + +The private configuration must bind a run UUID, explicit Docker context or process +host inventory, immutable Cellar/RustFS image digests or executable manifest, +source revision, dedicated data roots, credentials-file paths, port allocation, +node CPU/memory/swap/disk limits, failure domains and fencing-authority endpoint. +Validate all required fields; missing values never fall back to the current Docker +context or a default data directory. Credentials and unrestricted Docker inspect +output stay out of reports. Unsupported resources fail before provisioning. + +Use existing `tools/perf/raft_fleet.py` for managed-node provisioning patterns and +`tools/perf/process_recovery.py` for native process SIGKILL/erasure proof. Extend +their shared owners; do not implement a competing fake object-store or voter +fixture. Own resources by immutable IDs, run labels and exact paths. The destructive +profile must confirm ownership and join stopped processes before removing only +the recorded voter roots. Keep RustFS data and retained evidence outside voter +roots, build cleanup and global prune paths. `stop` retains data for inspection. + +## Workload accounting + +Build once before timing. Record source, compiler/features, binary digest, actual +CPU/memory limits, disk/filesystem, host activity, RustFS digest and TLS topology. +Run cohorts serially with fresh identical initial data and a frozen arrival trace. +Compare the old one-cell baseline against the replacement, and compare group +occupancies 1/8/32/64/128 within the replacement. Capture measured actual groups, +not only configured target occupancy. Each logical cell has a separate SQLite DB. + +The offered-load ledger records stable cell/client/sequence, input digest, request +start, deadline, response/timeout/cancel/refusal, returned result, cell version, +group witness and requested/achieved durability. Keep every offered request in +throughput/error denominators. Reconciliation is a separate phase and cannot +retroactively turn a timed-out request into a successful timed sample. + +Independently verify all acknowledged effects and receipt results on every serving +replica and the selected recovery cut. Check per-cell real-time strong-read order, +at-most-once mutation by command identity, allowed ambiguous outcomes and no +overlapping writable owners. Deliberately corrupt oracle inputs to prove it detects +lost effects, borrowed versions, double execution and stale strong reads. + +Measure CPU cores, process RSS, descriptors, group/cell counts, outgoing control +bytes below/above TLS with the exact boundary stated, log/image/manifest write +bytes, fsyncs, S3 PUT/GET bytes/requests and retained/archive lag. Report p50/p95/p99 +and sample counts by durability, hot/cold cell and fault phase. Warmup, setup, +election, cache restore, archive drain and joined shutdown are separate intervals. + +For ratios, require a positive meaningful matched baseline; a zero denominator +cannot pass a reduction gate. Resource peaks use actual OS/engine observations, +not configured limits or synthetic host metrics. A recovered follower is not proof +that every replica or every cell recovered. Selected-cut Quorum suffix loss is +reported separately from missing S3-protected history. + +## Required fault schedule + +Kill/restart at image sync, manifest sync, marker commit, enrollment, snapshot +install, archive publication, reclaim, seal authorization, parent seal, child seed, +root publication, activation, retirement, park and wake generation closure. +Inject corrupt/missing objects, duplicate/delayed messages, stale routes, unknown +CAS responses, leader/controller loss, a slow/minority follower, disk-full/sync +failure, network partitions and bounded S3 outage. Run foreground writes and +strong reads while splitting/merging and moving replicas. + +For zero-quorum tests, verify the archive/ledger boundary first, fence all possible +old voting lineages and demonstrate launch revocation. Delete all voter disks, +restore complete groups from real RustFS with no surviving peer images, verify +every cell and receipt, reject old tokens/old-host restart, acknowledge new writes, +and restart the fresh quorum on its new disks. Repeat before/after topology +publication and after reclamation. Managed inventory/transition journal failures +must fail closed. Unreconciled old-origin outbox work must remain refused. + +## Task evidence + +Store local raw evidence under an ignored repository-relative directory, for +example `.cellar/cell-groups-evidence/RUN/`; CI may download sanitized artifacts +there before validation. Do not commit credentials, cell data, opaque tokens or +the previously excluded MR-13/MR-14 progress JSON files. Add a relative record path +to a task's `evidence` only after its implementation and checks actually pass. + +A task record uses this shape (example values are placeholders, not evidence): + +```json +{ + "format": 1, + "task_id": "CG-01", + "source_revision": "FULL_40_HEX_COMMIT", + "executable_sha256": "64_HEX", + "qualified_executable_sha256": "DEPLOYED_CELLAR_BINARY_64_HEX_WHEN_PROFILES_ARE_PRESENT", + "environment": "Actual compiler, features, OS and resource observations", + "limitations": "Exact scope and untested boundaries", + "source_inputs": [{"path": "crates/...", "sha256": "64_HEX"}], + "commands": [{"argv": ["cargo", "+1.98.1", "test", "..."], "exit_code": 0, + "output": ".cellar/.../stdout.log", "sha256": "64_HEX"}], + "profiles": [{"path": ".cellar/.../profile.json", "sha256": "64_HEX"}] +} +``` + +`source_inputs` must include the task's pinned source anchor paths and its test +target; the validator verifies each hash against the named Git revision. Include +all runtime sources changed since the frozen `tasks.json` planned-at revision in +the record as well. Later source-baseline reviews do not rewrite this evidence +boundary or waive earlier implementation inputs. Test inventory and exact +one-test outputs must be hashed and retained. Every command in `plan.py show` +verification must appear with exit zero. Earlier failed runs stay in the run +directory and are described in the final report; do not aggregate passes across +different binaries into one task record. + +For CG-16, one record must contain all new tasks' verification commands at the same +release revision, plus the repository checks listed below and all required profile +reports. Dependencies' earlier passes alone cannot qualify a changed release. +The task record must not claim it verified different executable inputs. Later +plan-only status/evidence commits are permitted; the release command compares all +other repository inputs against the qualified source and rejects dirty files, +including examples, vendor code, build configuration and newly added input paths. + +Profile reports bind `profile_id`, `source_revision`, `executable_sha256`, the +SHA-256 of the entire frozen `profiles.json`, actual environment observations, +numeric `metrics`, and hashed artifacts for `ledger`, `measurements`, `topology`, +`fault_schedule` and `verification`. The profile must retain its controls, +repetitions and parameter matrix in those artifacts. `plan.py` checks every numeric +gate, required topology and artifact hash. A bare `passed: true` is insufficient. +Values must be finite and nonnegative; null, missing, NaN and boolean measurements +refuse. The report and every replacement cohort must match the task record's +`qualified_executable_sha256`. The primary `executable_sha256` identifies the native +test artifact; these separate fields avoid equating it with the deployed server. + +The measurements artifact is JSON with `metrics` equal to the report summary and +`cohorts` containing exactly the declared occupancy × image mode × repetition +matrix. Each cohort records: + +```text +variant: replacement | historical +occupancy, image_mode, repetition +source_revision, executable_sha256, ledger_sha256 +cells, nodes, voters, offered_requests_per_second, measured_seconds +metrics: numeric observations for that cohort +``` + +Historical controls are required only where `historical_control` is true, at +occupancy one and the pinned historical revision. Replacement controls and +candidates use the same qualified binary. Every cohort must meet its frozen +population, resources, arrival rate and minimum duration. Correctness/error gates +apply to every cohort, including historical and occupancy-one controls. Performance +gates apply to `gate_occupancies`. Summaries use the worst eligible cohort: maximum +for a max gate, minimum for a min gate, and equality in every cohort for an eq gate. +Never average a slow or failed cohort into a passing result. + +Ratio observations must equal the numerator divided by the matching control's +denominator, with the same image mode and repetition; `ratios` defines the exact +metric names and control occupancy. A missing/zero control refuses the ratio. +Hibernation's active and parked rates are separate phases of the same cohort. +The native harness derives the other measurements from actual ledgers/OS samples; +the plan validator does not replace the workload oracle or establish physical +infrastructure truth from a self-reported number. + +The topology artifact is JSON with `kind` and `nodes`, each containing `id`, +`physical_host` and `failure_domain`. Independent-domain runs require a different +physical host for each node and at least three failure domains. The verification +artifact records `source_revision`, `complete: true`, `ledger_sha256` and +`measurements_sha256`, binding its full history oracle to the actual retained +artifacts. Retain the detailed checks and physical isolation observations there, +not just these mandatory header fields. The fault schedule records requested and +observed events, timing, exact owned resource IDs and phase boundaries for every +cohort. Infrastructure evidence must still be reviewed before advertising a tier. + +## Repository release checks + +Run from the repository root with its own writable workspace target directory: + +```sh +cargo +1.98.1 format --check +cargo +1.98.1 check-layout +python3 tools/check_native_dependencies.py +python3 tools/docs/check.py +cargo +1.98.1 check --workspace --all-targets --all-features --locked +cargo +1.98.1 test --workspace --all-features --lib --tests --locked -- --test-threads=1 +cargo +1.98.1 test --workspace --all-features --doc --locked +cargo +1.98.1 clippy --workspace --all-targets --all-features --locked -- -D warnings +python3 -m unittest discover -s tools/perf -p 'test_*.py' +``` + +Feature-specific and external consumer checks in the SDK CI workflow remain +required. The new package runner supplements CI; it is not a substitute for it. diff --git a/docs/design/multi-raft/cell-groups/remaining-work.md b/docs/design/multi-raft/cell-groups/remaining-work.md new file mode 100644 index 000000000..57a5fe5f8 --- /dev/null +++ b/docs/design/multi-raft/cell-groups/remaining-work.md @@ -0,0 +1,316 @@ + + +# Remaining cell-group work and priorities + +Purpose: Prioritize the remaining cell-group MVP, merge gates and full architecture deliverables. + +PR [#160](https://github.com/crabbuild/cellar/pull/160) contains the shared runtime +and the canonical SDK/fleet cutover in progress. This ordering preserves the full +CG-01–16 objective in [tasks.json](tasks.json), [design](design.md) and +[qualification](qualification.md). See [progress](progress.md) for exact source +and validation boundaries and [handoff](handoff.md) for executor context. + +## Current boundary + +CG-01–06 and CG-08 passed their recorded component gates. Shared group owners, +cell-specific SQLite histories/receipts/effects/versions, complete checkpoints, +archives/reclamation and fenced fresh-history recovery have scoped coverage. +Managed startup, directory refresh, membership send fences, capacity admission, +peer/S3 replacement, stop/return/resume and retained initial/joined restart pass +component cases. Automatic S3 voter recovery reaches terminal removal while +retaining the obsolete engine and disk. These are fixture isolation/capacity +results, not production platform fencing or a completed public lifecycle gate. + +The latest complete eleven-target campaign has 308 passes, 2 failures and 39 +explicit skips. The new managed native/Axum request case and strengthened public +SDK/process gate pass. The full campaign remains FAILED: eight-turn maintenance +replacement settlement and fresh successor readiness after closing an S3 source +service fail, with retained disks/logs and an Unobserved close report. No bound or +assertion is relaxed and later passes do not waive these failures. + +The request case qualifies follower writes, explicit namespace/process grant and +revocation, remote minimum-version/linearizable reads, forwarded S3 acknowledgement, +joined leader-stop failover, retained restart/deduplication and quorum-loss refusals +in both image forms. The public counter requires all six native/Axum addresses to +serve original retry receipts before and after retained restart. Current-source +RustFS results and all exact source/binary limits are recorded in +[progress](progress.md). Earlier disk-erasure and other checkpoints retain their +recorded sources and are not relabeled as this campaign. + +Ingress without a local replica can serve reads and writes through trusted group +routes. SDK command forwarding retains the original owner and deadline and +requires namespace write permission. Local SQL, effect and archive proof remain +distinct from authenticated remote output. The managed acceptance exercises +native and Axum ingress with zero local groups. SDK enrollment now uses that +path and preserves the exact creation record at both durability minima. +A configured managed router now refreshes its voter routes after a replacement. +The native and real RustFS cases cover the retired route, joined-voter reads, +original receipt replay and a fresh quorum write. Cold return after all birth +voters are replaced and stopped passes the focused native and RustFS cases. +The public SDK provisions a declared managed standby and reserves, publishes +and confirms its first full-range group. The CLI and admin API admit capacity +and initialize the rooted archive. The managed SQL daemon discovers the group +and waits for the exact archive and fresh quorum before opening application +ingress. At `cd1d8315`, the native process gate and constrained Linux RustFS +gate passed in both image formats. The latter uses separate voter containers +and disks in one VM with fixture capacity. It covers native and Axum reads and +writes, voter SIGKILL, receipt replay, retained restart and quorum-loss refusal. +Production capacity admission and the combined lifecycle, outage and load +gates remain open. + +The managed native and real RustFS request cases now cover foreign cluster, +namespace and cell histories through both HTTP hosts, and foreign fleet, +namespace and physical group requests through the admin API. Valid retries +preserve the original receipt after those refusals. The named public authority +and occupancy-one cases pass their recorded native scope. + +CG-07 and CG-09 through CG-15 are in progress. CG-16 is pending the combined +release gate. Individual checkpoints cover operator-driven split and merge, +placement recommendations, fenced parking and wake, managed recovery and +parts of the qualification matrix. The task-level evidence remains open. + +The managed cold-recovery case now erases every owned source voter disk and +reopens from RustFS in both image modes. It verifies restored SQL and receipts +through a fresh quorum, resumes interrupted archive setup, and retires the old +histories. Independent voter processes and permanent Docker launch revocation +remain separate gates. In the public lifecycle work, the latest RustFS follow-up +passed source-service failover and hit an S3 receipt replay deadline after the +joined voter resumed as a ready leader. See [progress](progress.md) for both +source boundaries. The constrained Linux public SQL gate passed at +`9d6cf6e9` in both image modes, including leader SIGKILL, retained restart, +quorum refusal and archive-write outage. The remaining CG-07 process gate +combines these paths with managed repair, drain and maintenance. + +## Open case audit + +The [implementation index](implementation-plan.md) counts accepted cases from +completed task records. Component and process checkpoints retain their recorded +source revisions. Each open task still needs the exact target, case oracles +and evidence specified by `plan.py show `. + +| Task | Checked-in coverage | Acceptance still required | +| --- | --- | --- | +| CG-07 | All five named cases are in `native_cell_groups_public`. All five passed in the latest native run, covering SDK multi-cell and occupancy-one serving, cross-authority refusal, managed lifecycle and existing Bucket/Fleet behavior. | Run the public cases and constrained RustFS daemon campaign at one source with repair, drain, maintenance, retained restart and outage. | +| CG-09 | `seal_authorization_cancel_race` and `source_seal_blocks_delayed_work` run in the catalog and host targets. Component checkpoints also cover rooted archive holds and managed serving after transfer. | Build the planned transition target. Exercise publication only after destination quorum install and remote closure, restart with one writer, receipt/token/effect continuity, and archive pins through retirement. | +| CG-10 | Managed split/merge serving has run against RustFS in both image modes. | Qualify exact union and adjacency, incompatible placement/durability, partial reservations across restart, writes during movement and bounded precopy. | +| CG-11 | Placement observations and advisory recommendations exist. | Qualify hysteresis, hot-cell isolation, whole-group scale out and in, failure-domain reserve and bounded controller fanout. | +| CG-12 | The constrained daemon has parked and woken a group through fresh quorum. | Qualify all-voter eligibility, partial/stale certificates, restart and leader loss, and alarm and peer wake without a client. | +| CG-13 | Host-native voters have recovered after all owned voter disks were erased against RustFS with a fixture fence. | Qualify interrupted transition closure selection, permanent revocation of every possible old writer, fresh histories and retry under independent Docker voter processes. Exercise the outbox ambiguity refusal. | +| CG-14 | The daemon harness retains scoped evidence from real voters and RustFS. | Build the planned harness acceptance target for real occupancies, ownership, complete ledgers, evidence drift rejection and measured profiles. | +| CG-15 | Checkpoints cover individual public faults and archive recovery. | Run the history oracle, full crash and transition matrix, archive reclamation under load, wake profiles, soak and independent-domain loss. | +| CG-16 | Public shared-group SQL has scoped coverage. | Remove predecessor consensus paths, prove full public operations and joined shutdown in all modes, and bind release evidence to one source. | + +## First delivery of quorum SQL through public ingress + +An explicitly configured Raft namespace defaults to three voters and a +quorum durability minimum. Public Raft commands default to quorum durability +and queries default to linearizable reads. A quorum acknowledgement requires +a committed command and its applied receipt. Explicit S3 durability waits for +archive proof. The release should define which SQL capabilities are available +through the shared backend and preserve typed refusals for the rest. + +1. **Close CG-07 failed-voter repair and managed lifecycle.** The corrected + constrained RustFS candidate passed both image modes. It promoted the + standby, retained the failed disk, replayed the archived receipt and + served fresh quorum SQL through both ingress adapters. Run repair, joined + drain, maintenance and retained inventory together on the public PR + executable source. +2. **Pass the combined CG-07 RF3 SQL gate.** Use one source revision for + onboarding, enrollment, ingress-only SDK routing, native and Axum reads and + writes, leader loss, exact retries, retained restart and quorum refusal. + Verify continued quorum writes during archive-payload outage with catalog + access available, plus explicit S3 refusal and recovery. The directory + suite passed all 16 cases. The Fleet permit port passed all eight cases on + its original source. A later full-suite rerun hit an S3 receipt deadline and + passed the case in isolation. Sixteen shared-group read ports pass, + including follower ReadIndex recovery, peer saturation and routing from a + voter without a query callback. Resolve the extent-backed RustFS S3 + receipt deadline after repair, finish the + remaining multi-namespace router and learner readiness ports, and pass public CI before + making shared groups the default for supported SQL in new namespaces. +3. **Finish CG-09/10 operator split and merge.** One native case now resumes + the operator-admitted split after a partial reservation, serves SQL on the + activated children and verifies late writes after merging. Requalify this + path in both RustFS image modes. Complete the named transition targets, + including restart after partial seals and lost publication replies. +4. **Finish CG-11–13 managed operations.** Qualify adaptive placement, fenced + hibernation and wakeup, and S3 recovery of interrupted transitions. Run the + Docker-fenced disk-loss drill with all old voter disks deleted and a fresh + incarnation serving through quorum. Measure the difference between the + last acknowledged quorum write and the recoverable archive cut. +5. **Qualify and ship through CG-14–16.** Run the constrained mixed-load, + reclamation, crash, partition and disk-loss matrix. Reclaim only + archive-covered local history. Complete the release cutover and remove the + predecessor Raft paths after the combined gate passes. + +## P0: finish the MVP and its merge gates + +The public group-control prerequisite now has an implementation: +`RaftGroupHandle::observe` samples actual physical log/owner metadata and cell +occupancy; `archive_status` keeps publication separate from coverage; guarded +`transfer_leadership` preserves the caller's term/revision and checks the original +deadline at owner execution. The counter inventory consumes the observation API. +The performance/fleet consumer port now uses shared layout, replicated enrollment, +logical versions and physical witnesses locally; its full native RF3 gate remains +failing. The owned-workspace checkpoint passes occupancy 32 in the matrix +and isolated diagnostic case, including the original explicit S3 assertions. +The matrix reaches occupancy 128 and refuses enrollment after 58 cells under +the original 60-second setup budget and five-second per-call cap. The full suite +has eighteen passes and one failure. Earlier 28/31/44-cell enrollment and explicit +S3 failures remain preserved; this result establishes no fixed cell-count limit +or cause for the remaining refusal. Workspace creation now syncs its child and +owned parent rather than repeating the open store's ancestor barriers; rooted +private opens retain that exact directory owner. Disposable read-cache directories +use ordinary creation. Child/parent failures retain their charge and quarantine. +Preserved probes place more time in physical apply preparation than publication +or collection in that sample. Measure image/page preparation and sync costs next +without raising bounds or weakening the applied-equals-committed enrollment fence. +The node-owned live extent collector now has focused shutdown and RF3 pressure +coverage, including the independent archive-verifier store. It keeps existing +control admission through physical completion and preserves reader/manifest pins. +That does not supply manifest-page collection or sustained-load qualification. +Warm metadata barriers and a bounded verifier cache reduce repeated physical +work without supplying archive proof; incoming extent bytes remain independently +checked even when their destination is cached. Finish repeated physical +preparation, checkpoint work and cross-group interference, live manifest-page +ownership, then finish the whole-group disaster operator. Retain +the original HTTP/read/retry options, workload ledger and independent oracles; +report measured physical group counts separately from logical cell counts. + +Public recovered-node startup now has its owned activation path: an explicit +fencing authority plus exact retained seed/engine pins, actual fresh quorum +activation, catalog completion and joined shutdown. The public Node drill restores original S3 receipts/state after all old voter +directories are erased, then verifies new writes and retained restart in both +image forms, including the recorded real-RustFS run. Its three embedded Nodes +and contract fence do not qualify the Docker permanent launch-fencing drill, +separate voter processes/hosts or multi-group load. +Use this API in the executable whole-group recovery port next. Preserve complete +archive cuts before fencing, use disjoint replacement member/node identities, +and sequence each group against the current catalog root through activation; +preplanning every group against one stale root is invalid. Keep partial recovered +inventory explicit and fail closed for unseeded groups. + +| Order | Deliverable | Required result | +| --- | --- | --- | +| 1 | Managed S3 learner activation and archive holds (CG-06/07) | Expand the scoped native/RustFS results to the full failure/cancellation and fenced failed-voter lifecycle. Preserve complete manifest/image/receipt closure, destination boot/catalog checks, physical install, catch-up and fresh quorum before promotion. Retained-journal discovery and retirement now pass native/RustFS idle, outage, solo-restart and cancelled-close cases. Complete fenced handling of unknown/lost ownership and the remaining failure/cancellation lifecycle. A surviving original voter still needs a live physical-inventory proof for catalog settlement; its outage must retain the debit. Finish orchestration for that retained debt and qualify stop/offline ownership; a seed or remote object supplies no activation authority. | +| 2 | Public SDK surface and executable consumers (CG-07) | Port the original daemon/native/Axum fixtures and performance/fleet runners to namespace capabilities, physical group assignment and replicated cell enrollment. The local all-binary check clears the original 27 compilation errors; qualify the final combined source. Clear the remaining 128-cell enrollment deadline, qualify S3 at every planned occupancy, finish live manifest-page ownership and qualify owned extent collection under the original workload bounds. Retain every original HTTP/read/retry option, exact term/revision transfer guard, actual responder metadata and all five original public gates. Incorporate the implemented cold birth, repair, maintenance stop/return/resume and physical debt into the complete public lifecycle case, including retained nonvoting inventory and failure/cancellation ownership. | +| 3 | Public, build and backend qualification (CG-07 and MVP portions of CG-14/15/16) | Diagnose the original checkpoint TCP retry `Overloaded`, intermittent five-second successor-readiness timeout and earlier eight-turn maintenance failure; current passes cannot waive failed runs. Deliver the four remaining public cases below while retaining the SDK case; port `native_raft_reads` and the remaining per-cell test callers, and clear strict lint failures without suppressions. Keep the reviewed plan anchors and generated index current. Run affected SDK CI feature modes. Extend current-source real RustFS component recovery to constrained independent voters and replication/reclamation under bounded mixed load, retaining exact binaries, ledgers, failures and resource/latency measurements. | + +The public counter and first SDK gate are qualified at their recorded local +boundary. Managed boot/directory/stop ownership is implemented at its recorded +component boundary. Cold inventory/capacity and nonempty initial-group startup +pass the recorded local gate. Peer replacement, actual handoff and retained +joined-member restart pass the focused managed case. Basic catalog maintenance +stop/return/resume now passes its local role/image matrix. Original consumers and +public acceptance, lifecycle failure/cancellation, complete S3 qualification and backend +qualification still block the complete MVP. A successful metadata-only standby boot supplies no +complete-inventory or capacity proof; an Active-node fixture cannot replace the +original public lifecycle gate. The currently refused managed paths remain +closed until actual authorization, permanent claims, resource accounting and joined service ownership +are wired; removing guards or dropping consumer options is not a cutover. + +Public replica routing belongs in one controller-owned invocation. Local and +remote attempts must retain the same routing session and admission; a timed-out +physical read remains owned while another permitted attempt runs. Incoming RPCs +execute locally rather than recursively forwarding. Peer discovery and leader +hints choose a destination but never supply authority. Do not implement retries +by creating fresh independent deadlines or attempt banks. + +The original public merge gates are: + +- `sdk_many_cells_share_group`: public startup/handles, independent SQLite cells, + one configured group and all replicas; observe actual group/RawNode counts. +- `managed_group_lifecycle`: repair, drain, stop/return/resume and capacity move + complete groups while preserving cell counts and physical debt. +- `ingress_cannot_cross_cell_authority`: native/Axum/admin tenant, cell and group + binding refusals, with immutable command identity across retries. +- `occupancy_one_uses_shared_path`: one-cell and multi-cell configurations use + the same host, manifest, snapshot and archive implementation. +- `feature_off_fleet_contract_unchanged`: existing non-Raft ownership/durability + modes pass their actual regression matrix. + +Run the original counter process/restart drill after its port, including late +startup failure, both HTTP hosts, remote minimum-version/linearizable reads, +permission and cross-cell token refusal, exact retained receipts and joined close. +Do not delete options or weaken assertions to make consumers compile. + +The RustFS recovery gate deletes every old voter disk behind the accepted +fencing-authority/Docker contract, then restores a new incarnation from complete +S3 evidence. Verify restored images/receipts/client floors, original S3 minima, +permanent old-launch/process refusal, fresh quorum activation, new writes and +retained restart in both image forms. Under load, verify archive coverage before +local reclamation and account for accepted, refused and ambiguous outcomes. +Local loopback qualification must remain labeled as such. + +## Source-sealing implementation boundary + +The unmanaged verification directory roots an exact split or merge plan in the +range head CAS. It retains Planned, Prepared, SealAuthorized and Cancelled +metadata across owner restart. Authorization and cancellation compete on the +same prepared head. Ordinary range publication, import and disaster recovery +must acquire the topology head without an active transition. Routing continues +to select the original parent ranges until the verified transfer publishes. + +The managed transition holds the fleet disruption slot from admission and +binds source seal and child installation to the exact plan and physical +replicas. The parent drains accepted proposals before committing the seal. +The child install, fresh-term quorum set and verified source archive closure +pass their focused native and real RustFS gates. Atomic publication and +replicated child activation have a focused native test that also checks +retained cell values, original receipt replay and a new quorum write on each +transferred cell. Complete activated-child and sealed-parent restart, the +child archive and retirement gates, then run the whole transition on the +same qualified source. + +The sealed-source journal accepts one explicit archive hold per planned parent. +Each hold must belong to the namespace-bound archive and independently restore +the matching replicated seal and its complete image closure. The directory +records the exact ordered set only after verification. The transition owner +retains those holds until retirement and re-verifies them after restart. + +## Default backend, capabilities and deployment + +Quorum is already the command API default, and the namespace declaration +already defaults to quorum minimum durability and three voters. Those defaults +do not select the shared-group backend for newly created cells. Complete the +namespace factory, SDK, CLI, daemon, native/Axum/admin ingress and examples as +one public cutover. Missing quorum must refuse admission without selecting a +weaker durability mode. The intended deployment contract is three-replica shared +groups, quorum writes and asynchronous S3 archives, with measured archive lag/RPO. +An explicit S3 write still requires its separate complete coverage proof. + +Resolve the default backend's capability matrix before enabling it broadly. +Workflow/Queue callbacks, WebSockets and mutable-container behavior need equivalent +adapters or an explicit supported product scope; the native SQL counter cases +do not establish their coverage. Preserve capability refusals while these remain +unimplemented. Existing Fleet data requires a separate deployment decision: +this plan supplies no online Fleet-to-Raft conversion and must not reinterpret +old data as a new Raft history. + +The shared-group SDK and counter ingress default to linearizable reads. +Callers can select a receipt-based version floor or explicitly permit stale +reads. Qualify the default and explicit policies through the public request +path independently of the storage backend cutover. + +## P1: complete the requested architecture + +These are required follow-ups, not removed scope or optional substitutes for the +full hard cutover. + +| Task | Deliverable and dependency | +| --- | --- | +| CG-09 | Irreversible source sealing and whole-cell transfer, with cancellation/seal serialization, quorum installation and atomic range publication. Preserve cell histories, receipts, effect IDs and remote recovery closure. | +| CG-10 | Split hot groups and merge compatible adjacent ranges using CG-09; recover partially completed transitions without reopening a sealed source. | +| CG-11 | Group placement, failure-domain/reserve accounting, load observations, hysteresis and hot-cell isolation over the same transition/membership owners. | +| CG-12 | Fenced group parking and authority-checked wakeup. Resume actual membership/quorum under the original request deadline; measure cold response latency rather than assuming immediate readiness. | +| CG-13 | Managed S3 recovery of complete groups and incomplete transfers/splits/merges, including zero-quorum fencing, fresh incarnations and unfinished lifecycle journals. | +| CG-14 | Reproducible process/RustFS harnesses with real independent cells/groups, source/binary/profile hashes, owned resources and complete outcome ledgers. | +| CG-15 | Full load/crash, transition, wakeup and disaster-recovery matrix. Measure the documented profiles and failure schedules; no capacity or TiKV performance-parity claim follows from correctness tests. | +| CG-16 | Delete remaining predecessor consumers/codecs, finish feature-off/full release checks, publish the unified API/runbooks and qualified operating envelope. No one-cell special path, compatibility reader or production mode toggle. | + +The next functional slices are the combined public SQL lifecycle and retained +restart through an in-progress managed transfer. Preserve the existing bounds +while diagnosing eight-turn replacement, successor readiness after source close +and retained debt on surviving original voters. Complete the SQL capability +scope and release gates, then topology, placement, wake and the managed +recovery and release matrix. diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json new file mode 100644 index 000000000..00a142e6f --- /dev/null +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -0,0 +1,3164 @@ +{ + "format": 1, + "planned_at": "70b6022c8baba3d0a52de6bd9fa11866bdbd3b3f", + "toolchain": "1.98.1", + "invariants": [ + "I01", + "I02", + "I03", + "I04", + "I05", + "I06", + "I07", + "I08", + "I09", + "I10", + "I11", + "I12", + "I13", + "I14" + ], + "tasks": [ + { + "id": "CG-01", + "title": "Separate logical cell history from physical group consensus", + "status": "DONE", + "priority": "P1", + "effort": "M", + "risk": "HIGH", + "depends_on": [], + "anchors": [ + "identity", + "codec", + "version", + "ci" + ], + "scope": [ + "crates/cellar-logic/", + "crates/cellar/application/consensus/", + "crates/cellar/durability/raft/version.rs", + "crates/cellar/lib.rs" + ], + "creates": [ + "crates/cellar-logic/cell_group.rs", + "crates/cellar-logic/cell_group_tests.rs", + "crates/cellar/tests/native_cell_groups_contracts.rs", + "crates/cellar/application/consensus/group_contracts.rs", + "crates/cellar/application/consensus/group_protocol/mod.rs", + "crates/cellar/application/consensus/group_protocol/types.rs", + "crates/cellar/application/consensus/group_protocol/codec.rs" + ], + "test_target": "native_cell_groups_contracts", + "steps": [ + { + "action": "Add dependency-free CellHistory, GroupHistory, CellKey, CellVersion and route/membership epoch types. Specify canonical hash bytes and infinity bounds exactly as design.md. Add private adapters while existing consumers still compile; do not introduce a public legacy/shared mode flag.", + "tests": [ + "identity_and_hash_boundaries" + ] + }, + { + "action": "Encode cell-addressed data and typed administrative entries under new marker-1 semantic domains. Reject unknown tags, duplicate identity fields, oversized declarations and wrong cell/schema/route before allocation or application.", + "tests": [ + "envelope_rejects_wrong_cell_and_domain" + ] + }, + { + "action": "Define version tokens and receipt provenance around stable cell incarnation and monotone mutation sequence. Preserve transfer compatibility within one cell history; refuse old-incarnation and cross-cell tokens. Wire validation through owning Result types.", + "tests": [ + "version_survives_transfer_not_disaster" + ] + }, + { + "action": "Add golden byte/negative tests and bounded decoding fixtures. Map the new module through existing lib.rs paths; keep raft-rs, I/O and serde out of cellar-logic.", + "tests": [ + "codec_limits_before_allocation" + ] + } + ], + "test_cases": [ + { + "name": "identity_and_hash_boundaries", + "oracle": "Empty/control IDs, hash ties, infinity endpoints and sequence overflow reject or order exactly as the canonical specification.", + "invariants": [ + "I01" + ] + }, + { + "name": "envelope_rejects_wrong_cell_and_domain", + "oracle": "Valid LTX under another cell/route/schema and old protocol domains cannot be decoded as authorized data.", + "invariants": [ + "I01", + "I04" + ] + }, + { + "name": "version_survives_transfer_not_disaster", + "oracle": "Same cell incarnation/sequence token remains comparable after a group move; another cell or disaster incarnation refuses.", + "invariants": [ + "I01", + "I03", + "I05" + ] + }, + { + "name": "codec_limits_before_allocation", + "oracle": "Truncated, duplicate and excessive metadata/payload declarations fail within configured memory bounds.", + "invariants": [ + "I01", + "I12" + ] + } + ], + "invariants": [ + "I01", + "I03", + "I04", + "I05", + "I12" + ], + "evidence": [ + ".cellar/cell-groups-evidence/CG-01-06-gates-e32115aa-02/CG-01-record.json" + ], + "stop_conditions": [ + "A current source excerpt no longer matches: review and record the successor baseline before edits.", + "An acknowledged effect, receipt, proof boundary or resource owner would be weakened to make a test pass.", + "The implementation requires an unrecorded wire/storage compatibility path or bypass of quorum/fencing." + ], + "out_of_scope": [ + "TiKV KV/MVCC APIs and cross-cell transactions", + "Online migration or deletion of existing user disks/archives", + "Feature-off Fleet/bucket durability semantics" + ], + "capability": "contracts" + }, + { + "id": "CG-02", + "title": "Persist complete group manifests with crash-safe cell image ownership", + "status": "DONE", + "priority": "P1", + "effort": "L", + "risk": "HIGH", + "depends_on": [ + "CG-01" + ], + "anchors": [ + "manifest", + "apply", + "extent", + "sql" + ], + "scope": [ + "crates/cellar/durability/raft/", + "crates/cellar-ltx/src/extent/", + "crates/cellar-ltx/src/generation.rs", + "crates/cellar/application/consensus/mod.rs", + "crates/cellar/tests/support/raft_ltx_fault.rs", + "crates/cellar/tests/support/raft_storage_fault.rs" + ], + "creates": [ + "crates/cellar/durability/raft/cell_groups/mod.rs", + "crates/cellar/durability/raft/cell_groups/manifest.rs", + "crates/cellar/tests/support/cell_group_recovery.rs", + "crates/cellar/tests/native_cell_groups_storage.rs", + "crates/cellar/durability/raft/cell_groups/page_store.rs", + "crates/cellar/durability/raft/cell_groups/page_wire.rs", + "crates/cellar/durability/raft/cell_groups/README.md", + "crates/cellar/durability/raft/cell_groups/root.rs", + "crates/cellar/durability/raft/cell_groups/pins.rs", + "crates/cellar/durability/raft/cell_groups/retention.rs", + "crates/cellar/durability/raft/cell_groups/image_pin.rs" + ], + "test_target": "native_cell_groups_storage", + "steps": [ + { + "action": "Implement authenticated paged copy-on-write cell manifests and a group root. Leaves bind cell history, image/version, last group mutation and effect summary. Bound page fanout, decoded bytes, tree depth and cell count; use explicit execution-domain storage owners.", + "tests": [ + "manifest_updates_only_changed_path" + ] + }, + { + "action": "Separate logical-cell image bindings from group ownership in both materialized and extent paths. Sync changed images and manifest paths before one engine marker publishes applied position/root. Preserve old roots and all physical pins through failure.", + "tests": [ + "manifest_marker_crash_cuts" + ] + }, + { + "action": "Recover only complete durable roots; reject a missing/corrupt leaf or referenced image. Treat staged unmarked objects as non-serving orphans and reclaim only after pin/marker reconciliation.", + "tests": [ + "missing_cell_image_refuses_group" + ] + }, + { + "action": "Inject failures before/after image sync, manifest sync, marker commit and publication. Verify independent SQLite files, receipt floors and bounded copy-on-write work for one changed cell.", + "tests": [ + "manifest_pin_reclamation", + "both_image_representations_share_contract" + ] + } + ], + "test_cases": [ + { + "name": "manifest_updates_only_changed_path", + "oracle": "An update rewrites bounded manifest paths and the target cell, with unchanged cell image commitments preserved.", + "invariants": [ + "I04", + "I12" + ] + }, + { + "name": "manifest_marker_crash_cuts", + "oracle": "Every crash cut recovers exactly the old or new durable root without partial cells or callback reexecution.", + "invariants": [ + "I02", + "I04" + ] + }, + { + "name": "missing_cell_image_refuses_group", + "oracle": "Removing/corrupting any referenced cell or tree page prevents serving the group.", + "invariants": [ + "I04" + ] + }, + { + "name": "manifest_pin_reclamation", + "oracle": "Reader/snapshot/apply/transition pins prevent physical deletion until their owners release them.", + "invariants": [ + "I07", + "I12" + ] + }, + { + "name": "both_image_representations_share_contract", + "oracle": "Materialized and extent images retain identical history, publication and receipt safety.", + "invariants": [ + "I01", + "I03", + "I04" + ] + } + ], + "invariants": [ + "I01", + "I02", + "I03", + "I04", + "I07", + "I12" + ], + "evidence": [ + ".cellar/cell-groups-evidence/CG-01-06-gates-e32115aa-02/CG-02-record.json" + ], + "stop_conditions": [ + "A current source excerpt no longer matches: review and record the successor baseline before edits.", + "An acknowledged effect, receipt, proof boundary or resource owner would be weakened to make a test pass.", + "The implementation requires an unrecorded wire/storage compatibility path or bypass of quorum/fencing." + ], + "out_of_scope": [ + "TiKV KV/MVCC APIs and cross-cell transactions", + "Online migration or deletion of existing user disks/archives", + "Feature-off Fleet/bucket durability semantics" + ], + "capability": "storage" + }, + { + "id": "CG-03", + "title": "Prepare paged range routing and enrollment admission", + "status": "DONE", + "priority": "P1", + "effort": "L", + "risk": "HIGH", + "depends_on": [ + "CG-01", + "CG-02" + ], + "anchors": [ + "catalog", + "birth", + "host", + "fleet" + ], + "scope": [ + "crates/cellar/durability/raft/", + "crates/cellar/application/consensus/", + "crates/cellar/node/", + "crates/cellar/tests/support/raft_catalog_fault.rs", + "crates/cellar-logic/" + ], + "creates": [ + "crates/cellar/durability/raft/cell_groups/enrollment.rs", + "crates/cellar/tests/native_cell_groups_catalog.rs", + "crates/cellar/durability/raft/cell_groups/directory/mod.rs", + "crates/cellar/durability/raft/cell_groups/directory/wire.rs", + "crates/cellar/durability/raft/cell_groups/directory/tree.rs", + "crates/cellar/durability/raft/cell_groups/directory/store.rs", + "crates/cellar/tests/support/cell_group_directory.rs", + "crates/cellar/durability/raft/cell_groups/directory/birth.rs", + "crates/cellar/durability/raft/cell_groups/directory/bootstrap.rs", + "crates/cellar/durability/raft/storage_cell_genesis_tests.rs", + "crates/cellar/durability/raft/cell_groups/directory/routing.rs" + ], + "test_target": "native_cell_groups_catalog", + "steps": [ + { + "action": "Implement immutable paged range-tree records and one conditional namespace-root mutation journal. Verify complete range coverage, no overlap and exact epochs. Initial explicit ranges can be lazily materialized using existing birth and permanent disk-claim rules.", + "tests": [ + "range_directory_has_single_owner" + ] + }, + { + "action": "Define the pure enrollment decision and codec around the current range, cell creation identity and schema. Concurrent proposals must converge on one cell history when committed. This task validates admission decisions only; CG-04 connects those decisions to the actual multi-cell group owner and independent SQLite initialization.", + "tests": [ + "enrollment_decision_checks_identity" + ] + }, + { + "action": "Make physical replica/log identity, engine registry and durable mappings, node storage inventory and permanent member claims group-scoped, without a synthetic cell name. Connect the exact verified birth to synced real-engine genesis before permitting voter startup. Reject old storage domains rather than adding a legacy decoder. Reconcile lost birth/root/enrollment replies by canonical operation identity. Keep unconfirmed outcomes typed; a blank claimed voter cannot bootstrap under a new path. Refuse enrollment under sealed or retired ranges. Metadata-only birth tests do not satisfy this integration boundary.", + "tests": [ + "birth_unknown_and_claimed_blank_refuse" + ] + }, + { + "action": "Add bounded tree traversal/cache tests, restart races and stale-route redirects with original deadlines/hop budgets. Ordinary known-group commands must not require an S3 catalog request.", + "tests": [ + "bounded_catalog_discovery", + "sealed_range_rejects_enrollment", + "routing_redirect_requires_catalog_confirmation", + "routing_cached_requests_preserve_identity_without_catalog_reads", + "routing_attempt_deadline_survives_redirect_and_cancellation" + ] + } + ], + "test_cases": [ + { + "name": "range_directory_has_single_owner", + "oracle": "Duplicate/overlapping/gapped trees and stale root CAS cannot publish ownership.", + "invariants": [ + "I01", + "I09" + ] + }, + { + "name": "enrollment_decision_checks_identity", + "oracle": "Pure admission decisions reject wrong range/schema, preserve creation identity on retries and select one cell history; no native publication qualification is claimed here.", + "invariants": [ + "I01", + "I03" + ] + }, + { + "name": "birth_unknown_and_claimed_blank_refuse", + "oracle": "Real-engine genesis binds the selected group birth and permanent member claim without a cell owner; restart preserves that exact identity. Unknown creation reconciles its winner, identity substitution and old storage domains refuse, and deleting an owned claimed voter fixture cannot authorize fresh genesis under another path. No metadata observation alone grants voter-start authority.", + "invariants": [ + "I01", + "I08" + ] + }, + { + "name": "bounded_catalog_discovery", + "oracle": "Lookup reads a bounded tree path without namespace-wide listing; cache bounds hold under churn.", + "invariants": [ + "I12" + ] + }, + { + "name": "sealed_range_rejects_enrollment", + "oracle": "No delayed enrollment can create a cell beyond a committed seal boundary.", + "invariants": [ + "I09" + ] + }, + { + "name": "routing_redirect_requires_catalog_confirmation", + "oracle": "An untrusted relocation is followed only after an exact route lookup in the configured trusted catalog; stale tickets, foreign namespace/schema and non-advancing epochs refuse. The logical cell, original command identity, request fingerprint and invocation deadline survive a confirmed split redirect.", + "invariants": [ + "I01", + "I03", + "I09" + ] + }, + { + "name": "routing_cached_requests_preserve_identity_without_catalog_reads", + "oracle": "Cached route admission and same-group retries perform no object-store reads during a catalog outage. The original finite attempt budget covers retries and redirects; cached bindings cannot cross cells, incarnations or catalog owners. A cached route never grants serving authority.", + "invariants": [ + "I01", + "I03", + "I12" + ] + }, + { + "name": "routing_attempt_deadline_survives_redirect_and_cancellation", + "oracle": "A real held catalog read expires under the captured invocation deadline, preserving command ambiguity. Dropped discovery cannot refund an attempt, and a closed execution domain refuses even cached admission.", + "invariants": [ + "I03", + "I12" + ] + } + ], + "invariants": [ + "I01", + "I03", + "I08", + "I09", + "I12" + ], + "evidence": [ + ".cellar/cell-groups-evidence/CG-01-06-gates-e32115aa-02/CG-03-record.json" + ], + "stop_conditions": [ + "A current source excerpt no longer matches: review and record the successor baseline before edits.", + "An acknowledged effect, receipt, proof boundary or resource owner would be weakened to make a test pass.", + "The implementation requires an unrecorded wire/storage compatibility path or bypass of quorum/fencing." + ], + "out_of_scope": [ + "TiKV KV/MVCC APIs and cross-cell transactions", + "Online migration or deletion of existing user disks/archives", + "Feature-off Fleet/bucket durability semantics" + ], + "capability": "catalog" + }, + { + "id": "CG-04", + "title": "Execute many independent SQLite cells through one real Raft group", + "status": "DONE", + "priority": "P1", + "effort": "L", + "risk": "HIGH", + "depends_on": [ + "CG-01", + "CG-02", + "CG-03" + ], + "anchors": [ + "host", + "sql", + "runtime", + "identity", + "version" + ], + "scope": [ + "crates/cellar/durability/raft/", + "crates/cellar/application/consensus/", + "crates/cellar-logic/", + "crates/cellar-ltx/src/generation.rs", + "crates/cellar-ltx/tests/private_clone.rs", + "crates/cellar/tests/native_cell_groups_catalog.rs", + "crates/cellar/tests/support/cell_group_directory.rs" + ], + "creates": [ + "crates/cellar/durability/raft/cell_groups/owner.rs", + "crates/cellar/tests/native_cell_groups_execution.rs", + "crates/cellar/durability/raft/cell_groups/enrollment_apply.rs", + "crates/cellar/durability/raft/cell_groups/page_wire_tests.rs", + "crates/cellar/durability/raft/cell_groups/runtime/mod.rs", + "crates/cellar/durability/raft/cell_groups/runtime/worker.rs", + "crates/cellar/tests/support/cell_group_runtime.rs", + "crates/cellar/durability/raft/cell_groups/execution_authority.rs", + "crates/cellar/durability/raft/execution/cell.rs", + "crates/cellar/durability/raft/execution/command.rs", + "crates/cellar/tests/support/cell_group_sql.rs", + "crates/cellar/durability/raft/host_sql/cell_groups/mod.rs", + "crates/cellar/durability/raft/host_sql/cell_groups/admission.rs", + "crates/cellar/durability/raft/host_sql/cell_groups/worker.rs", + "crates/cellar/tests/support/cell_group_sql_pool.rs", + "crates/cellar/tests/support/cell_group_sql_reconcile.rs", + "crates/cellar/durability/raft/cell_groups/enrollment_authority.rs", + "crates/cellar/durability/raft/execution/enrollment.rs", + "crates/cellar/tests/support/cell_group_sql_enrollment.rs", + "crates/cellar/durability/raft/cell_groups/proposal_admission.rs", + "crates/cellar/tests/support/cell_group_admission.rs", + "crates/cellar/durability/raft/execution/submission.rs", + "crates/cellar/tests/support/cell_group_sql_submission.rs", + "crates/cellar/durability/raft/cell_groups/read_authority.rs", + "crates/cellar/durability/raft/cell_groups/runtime/reads.rs", + "crates/cellar/durability/raft/host_sql/cell_groups/query.rs", + "crates/cellar/tests/support/cell_group_sql_reads.rs", + "crates/cellar/durability/raft/cell_groups/apply/mod.rs", + "crates/cellar/durability/raft/cell_groups/apply/images.rs", + "crates/cellar/durability/raft/cell_groups/apply/summary.rs", + "crates/cellar/durability/raft/cell_groups/apply/summary_tests.rs", + "crates/cellar/durability/raft/host_sql/cell_groups/apply.rs", + "crates/cellar/tests/support/cell_group_sql_apply.rs", + "crates/cellar/durability/raft/cell_groups/apply/recovery.rs", + "crates/cellar/tests/support/cell_group_sql_apply_faults.rs", + "crates/cellar/tests/support/cell_group_sql_apply_recovery.rs", + "crates/cellar/durability/raft/execution/registered.rs", + "crates/cellar/durability/raft/effect_history.rs", + "crates/cellar/tests/support/cell_group_sql_registered.rs", + "crates/cellar/tests/support/cell_group_sql_registered_queries.rs", + "crates/cellar/durability/raft/cell_groups/dispatch/mod.rs", + "crates/cellar/tests/support/cell_group_sql_dispatch.rs", + "crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs", + "crates/cellar/tests/support/cell_group_sql_command_dispatch.rs", + "crates/cellar/durability/raft/transport_runtime.rs", + "crates/cellar/durability/raft/transport_shared.rs", + "crates/cellar/tests/support/cell_group_tcp.rs", + "crates/cellar/durability/raft/cell_groups/dispatch/command/owned.rs", + "crates/cellar/durability/raft/cell_groups/controller/mod.rs", + "crates/cellar/durability/raft/cell_groups/controller/worker.rs", + "crates/cellar/durability/raft/cell_groups/controller/commands.rs", + "crates/cellar/durability/raft/execution/submission_tests.rs", + "crates/cellar/tests/support/cell_group_controller.rs", + "crates/cellar/tests/support/cell_group_controller_commands.rs", + "crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs", + "crates/cellar/durability/raft/cell_groups/controller/enrollments.rs", + "crates/cellar/durability/raft/cell_groups/controller/calls.rs", + "crates/cellar/durability/raft/cell_groups/controller/requests.rs", + "crates/cellar/application/consensus/class_initializer_tests.rs", + "crates/cellar/tests/support/cell_group_controller_enrollment.rs", + "crates/cellar/durability/raft/query_cell_transport.rs", + "crates/cellar/durability/raft/query_cell_wire.rs", + "crates/cellar/tests/support/cell_group_remote_queries.rs", + "crates/cellar/durability/raft/query_cell_discovery.rs", + "crates/cellar/tests/support/cell_group_query_origins.rs", + "crates/cellar/durability/raft/cell_groups/directory/route_cache.rs", + "crates/cellar/tests/support/cell_group_routing_cache.rs", + "crates/cellar/tests/support/cell_group_dispatch_cache_execution.rs", + "crates/cellar/durability/raft/command_origins.rs", + "crates/cellar/durability/raft/command_cell_wire.rs", + "crates/cellar/durability/raft/command_cell_transport.rs", + "crates/cellar/durability/raft/cell_groups/dispatch/command/remote.rs", + "crates/cellar/tests/support/cell_group_remote_commands.rs", + "crates/cellar/durability/raft/enrollment_cell_wire.rs", + "crates/cellar/durability/raft/cell_groups/dispatch/enrollment_remote.rs", + "crates/cellar/tests/support/cell_group_remote_enrollment.rs", + "crates/cellar/durability/raft/cell_groups/dispatch/receipt.rs", + "crates/cellar/durability/raft/receipt_cell_wire.rs", + "crates/cellar/tests/support/cell_group_receipt_lookup.rs", + "crates/cellar/durability/raft/cell_groups/controller/registration.rs", + "crates/cellar/tests/support/cell_group_registration.rs", + "crates/cellar/durability/raft/cell_groups/controller/failure.rs", + "crates/cellar/tests/support/cell_group_quarantine.rs", + "crates/cellar/durability/raft/cell_groups/controller/provision/mod.rs", + "crates/cellar/durability/raft/cell_groups/controller/provision/config.rs", + "crates/cellar/durability/raft/cell_groups/controller/provision/prepare.rs", + "crates/cellar/durability/raft/cell_groups/controller/provision/manager.rs", + "crates/cellar/tests/support/cell_group_provision.rs" + ], + "test_target": "native_cell_groups_execution", + "steps": [ + { + "action": "Split the combined host Cell owner into group consensus state and per-cell execution state. One GroupHistory owns exactly one RawNode per voter; cell count does not allocate additional RawNodes. Reuse shared persistence, transport and SQL pools. Integrate CG-03 enrollment decisions with replicated enrollment and independent SQLite initialization before exposing cell commands. Atomically retain the immutable enrollment creation record with the selected cell state; preserve it through mutations and transfer so retries never reinitialize or restore the initial image over newer state. Consume the prepared CellRoutingSession attempts when integrating dispatch: preserve the logical request fingerprint, original domain deadline and total attempt budget across member retries and group redirects. Catalog-confirmed mappings are hints; destination admission must validate the full binding and current authority. Command producers remain domain-owned after a routing waiter is cancelled.", + "tests": [ + "three_cells_one_rawnode_per_voter", + "concurrent_enrollment_one_cell" + ] + }, + { + "action": "Admit one unresolved proposal per cell with fair cell queues and group/node byte quotas. Allow independent cells to prepare concurrently, then publish committed entries through one contiguous group prefix with per-cell base/version checks. Enforce configured group enrollment capacity on the actual owner before append, counting applied cells and pending accepted logical creations. Exact creation retries consume no additional slot; competing creations cannot exceed the limit before apply. Verify capacity refusal leaves existing cells available and release reservations only with proven resolution.", + "tests": [ + "shared_group_fair_admission", + "pending_enrollments_respect_group_capacity", + "node_runtime_bounds_proposal_acceptance_across_groups" + ] + }, + { + "action": "Implement exact per-cell output settlement and receipt retry. Group no-ops/configuration entries advance group apply without changing cell version. Reject stale term/activation/completion tickets and preserve unknown proposed outcomes. Check the prepared term atomically with proposal admission on the actual Raft owner. Bind same-term acceptance retries to the original proposal identity and bytes; a lost acceptance response must recover retained acceptance or reconcile the original receipt without appending the same mutation again. Reject changed payloads under a reused identity, and retain uncertainty across elections until an applied barrier proves resolution. A definitive pre-append owner refusal must irreversibly seal all clones of that preparation before returning an identity-bound rejection proof. Generic queue errors cannot retire a submitted SQL producer. Close rejected producers on their owning worker before releasing name admission; delayed retries and proofs from another preparation must not affect a replacement producer.", + "tests": [ + "cell_receipt_not_group_progress", + "proposal_acceptance_retries_do_not_append_twice", + "inherited_enrollments_count_after_applied_term_barrier", + "sql_submission_rejection_seals_delayed_retries", + "rooted_sql_submission_rejection_seals_delayed_retries" + ] + }, + { + "action": "Implement Stale, AtLeast and current-term ReadIndex for the target cell. Pin coherent manifest/image views and include cross-cell noninterference and slow-apply cases in the real-engine fixture. Correlate each strong read to its own runtime-owned request context and absolute domain deadline; retain separate bounded node/group pin admission through delivery and physical SQL execution. Configure logical image-pin bytes independently of transport message/request bytes; validate overflow and supported image bounds, and test independent capacity and reservation cleanup. Persist each selected version's full commit witness in the authenticated manifest and require exact proposal matching at apply. Keep original decision provenance separate from local installation position, retain it across unchanged-cell progress/restart and carry it through query and receipt observations. Missing witness fields must refuse at marker 1. Query callbacks consume immutable pins on existing workers, preserve read-only SQL policy and contain panics. Deadline/cancellation must interrupt SQLite work while arbitrary callback ownership remains retained until it finishes; shutdown joins that work.", + "tests": [ + "strong_read_at_group_barrier", + "apply_prefix_never_skips_slow_cell", + "rooted_strong_read_at_group_barrier", + "cell_query_pins_deadline_and_cancelled_cleanup", + "rooted_cell_query_pins_deadline_and_cancelled_cleanup" + ] + }, + { + "action": "Run physical group apply on the existing node SQL workers with independent bounded progress admission. Transfer the startup manifest-page owner, verify the exact committed entry/base, prepare either cell image representation and derive bounded scheduling summaries. Retain preparation through response cancellation and physical publication. Materialized retries may pin only the exact verified prepared image; they cannot infer publication or reset retirement. Test the physical image store retry across restart and corruption, and reject malformed effect tables rather than reporting idle. Keep membership/topology administrative apply explicitly refused until its owning tasks implement it.", + "tests": [ + "shared_workers_prepare_committed_cell_images", + "rooted_shared_workers_prepare_committed_cell_images" + ] + }, + { + "action": "Qualify image and manifest I/O ownership at actual held/failing directory barriers. Recover complete cell-group images through the same stable physical naming and verified extent bindings used by apply; accept only the engine-selected root and retain its exact cell versions/witnesses. Cold restart may replay a committed unpublished entry but never substitutes selected files or inherited commit state for fresh serving authority. Missing closure refuses before owner admission. Keep whole-process kill/power-loss qualification distinct from joined cold restart.", + "tests": [ + "shared_apply_cancelled_close_joins_preparation", + "rooted_shared_apply_cancelled_close_joins_preparation", + "shared_apply_sync_failure_preserves_selected_root", + "rooted_shared_apply_sync_failure_preserves_selected_root", + "shared_apply_recovers_selected_images_before_replay", + "rooted_shared_apply_recovers_selected_images_before_replay" + ] + }, + { + "action": "Use logical CellHistory at registered command and signed-effect identity boundaries. Dispatch namespace-authorized classes on the existing shared SQL workers; validate operation, schema, input and authenticated inbox identity before receipt lookup. Preserve exact proposal submission/settlement and cancellation ownership. Group placement is absent from effect identity; fresh cell incarnations remain fenced. Marker-1 old group-bearing effect wire and old signature domains refuse. Catalog/SDK dispatch remains separate integration work.", + "tests": [ + "registered_commands_use_cell_identity_and_receipts", + "rooted_registered_commands_use_cell_identity_and_receipts" + ] + }, + { + "action": "Dispatch compiled queries through the existing coherent query worker and callback guard with the actual CellVersionObservation, bounded operation/input admission and original read deadline. Keep legacy group cuts explicitly typed during integration; never fabricate a cell sequence or witness. Remove the temporary group-cut observation and old constructor with CG-16; final serving uses the common cell-group path.", + "tests": [ + "registered_queries_keep_cell_observation_and_ownership", + "rooted_registered_queries_keep_cell_observation_and_ownership" + ] + }, + { + "action": "Connect catalog routing sessions to compiled query execution with a namespace-bound class and captured input/consistency. Share bounded invocation admission across dispatcher clones, preserve the original domain deadline and total retry budget, consume each attempt before entering asynchronous work, authenticate AtLeast cell tokens and return a token for the selected observation. Exact destination route and engine checks remain mandatory; a confirmed catalog redirect cannot invent a physical owner. Dropping the routing invocation must leave entered SQL owned until joined cleanup. Command/enrollment dispatch and final public SDK replacement remain required.", + "tests": [ + "catalog_routed_queries_preserve_invocation_and_authority", + "rooted_catalog_routed_queries_preserve_invocation_and_authority" + ] + }, + { + "action": "Route immutable registered command invocations through the same dispatcher and catalog sessions. Capture host time/randomness once, retain the original runtime/SQL producer across cancelled preparation, and resume it before considering a new destination. Submission acceptance is provisional; settle only an exact committed receipt or the existing newer-term absence proof. Preserve command identity, deadline and attempt budget across retries, and refuse S3 requests or S3 namespace minima until a real archive proof is integrated. The production controller must retain active invocations and drive publication; enrollment and the public SDK/controller cutover remain required.", + "tests": [ + "catalog_routed_commands_preserve_producer_and_receipts", + "rooted_catalog_routed_commands_preserve_producer_and_receipts" + ] + }, + { + "action": "Connect physical group histories to the existing node-owned peer transport without constructing the legacy runtime. Share TLS pools, lanes, byte/item banks, revocation and joined shutdown. Bind typed marker-1 history and full replica identity to authenticated requests/responses; refuse old untagged wire layouts and unsupported group snapshots. Include physical routes in all resource and endpoint-conflict checks. The temporary legacy runtime/frame adapter must be deleted with the old host in CG-16. Qualify actual multi-group replication and routed command settlement over TLS before integrating the production controller.", + "tests": [ + "physical_group_transport_multiplexes_real_raft", + "catalog_commands_replicate_over_physical_tls", + "rooted_catalog_commands_replicate_over_physical_tls" + ] + }, + { + "action": "Own the exclusive shared event stream and bounded per-group apply jobs in one node controller. Preserve exact owner bindings and retain physical tickets through backpressure and joined shutdown. Admit fresh command invocations independently from response waiters; retain bounded ownership beyond disconnect/deadline until exact receipt, proven non-commit or joined shutdown. Seal only never-attempted preparations across all clones. Distinguish completed stage errors from lost worker replies and never automatically replay uncertain callbacks.", + "tests": [ + "group_controller_drives_tls_apply_and_bounded_retry", + "rooted_group_controller_drives_tls_apply_and_bounded_retry", + "group_controller_checks_exact_owners", + "group_controller_cancelled_close_joins_sql", + "group_controller_apply_failure_and_shutdown_preserve_selected_root", + "rooted_group_controller_apply_failure_and_shutdown_preserve_selected_root", + "controller_owned_commands_survive_sql_disconnect", + "controller_owned_commands_seal_late_preparations", + "controller_owned_commands_survive_accepted_disconnect", + "controller_owned_commands_resolve_after_response_deadline", + "controller_owned_commands_join_cancelled_shutdown", + "controller_owned_commands_release_completed_callback_errors", + "controller_owned_commands_join_a_lost_sql_reply" + ] + }, + { + "action": "Register initialization with an explicit digest in the class voting schema. Route immutable creation IDs and captured host inputs through the existing catalog session into the node controller. Share command/enrollment queues and admission, preserve original response deadlines, seal only never-attempted preparations and resolve the original immutable creation record after commitment or later mutations. Distinguish completed initialization errors from lost replies, join physical shutdown, and refuse unsupported S3 durability. The old public host must refuse registered initialization until its shared-runtime cutover; do not silently ignore the callback.", + "tests": [ + "controller_enrollment_survives_initializer_disconnect", + "controller_enrollment_seals_late_preparation", + "controller_enrollment_survives_accepted_disconnect", + "controller_enrollment_resolves_after_deadline", + "controller_enrollment_joins_cancelled_shutdown", + "controller_enrollment_releases_completed_initializer_errors", + "controller_enrollment_joins_lost_worker_reply", + "controller_enrollment_requires_registered_initializer" + ] + }, + { + "action": "Route consumed shared query attempts through the existing physical peer query lane. Bind physical history, logical cell, schema, route epoch and birth to the authenticated request; use the same read-only namespace grant/revocation guard as existing queries. Destination catalog and runtime authority remain mandatory. Return signed cell observations while retaining bounded result admission, and preserve original deadlines and attempts across retries/redirects. Cancellation must leave entered SQL physically owned until joined cleanup. Complete nonmember shared discovery and public SDK binding separately; mutation RPC must use explicit write authorization, never query-origin grants. Remove temporary legacy query framing and serving with CG-16. Separate public registration admission lifetime from data retained by accepted queries; the last host registration drop must refuse new requests even while an earlier callback is held. Assign query wire sequence numbers in the single peer sender after dequeueing so concurrent ingress cannot cause a false replay refusal.", + "tests": [ + "shared_remote_queries_bind_cell_versions_and_gateway_owners", + "shared_remote_queries_retain_cancelled_sql_until_join", + "shared_remote_queries_retain_expired_sql_until_join" + ] + }, + { + "action": "Install bounded physical read discovery for nonmember routers without creating a local group or borrowing voting authority. Include it in shared peer-pool lifetime, endpoint consistency, node-process refresh and route admission/observability. Revalidate captured route identity before sending and after receiving. Destination namespace grants remain independently authenticated, revocable and strictly read-only. Qualify absent, foreign-namespace and wrong-process grants and held-callback revocation over real TLS, preserving physical SQL ownership. Integrate trusted discovery publication and selection into the public host in CG-07; this API does not authorize automatic membership or mutation RPC.", + "tests": [ + "shared_nonmember_queries_require_grants_without_local_replicas", + "shared_nonmember_query_revocation_retains_physical_sql", + "shared_nonmember_query_route_replacement_refuses_stale_reply" + ] + }, + { + "action": "Share bounded namespace range hints across query, command and enrollment invocations, with explicit entry/byte reservations and LRU eviction. Cache ranges rather than cells; a failed metadata refresh does not invent a new serving proof. Verified successor heads invalidate prior hints and older in-flight observations cannot refill them. Preserve fresh destination RPC admission until replicated transfer-seal authority is integrated. Run the full native_cell_groups_catalog suite, including dispatcher_route_cache_shares_ranges_across_operations, dispatcher_route_cache_bounds_and_refreshes_hints and dispatcher_route_cache_refuses_superseded_inflight_refill, alongside the real runtime case below. Cached mappings never replace runtime ownership, exact cell/version checks, current-term quorum or producer ownership.", + "tests": [ + "shared_cached_dispatch_preserves_authority_during_catalog_outage" + ] + }, + { + "action": "Connect remote shared commands to the existing bounded RPC lane and controller. Authenticate command purpose separately from read payloads, require independent bounded namespace/process write grants even for voters, and include those grants in endpoint consistency, process refresh, fleet revocation and pool lifecycle. Preserve captured inputs, immutable command identity, remaining timeout and attempt budget. Destination fresh catalog binding precedes actual leader/SQL authority; only committed exact receipts and signed cell observations return success. Disconnect/revocation cannot cancel already admitted ownership. Also run all query_wire tests, including shared_command_wire_preserves_inputs_and_refuses_query_payloads, shared_command_purpose_is_authenticated_and_cannot_reuse_read_reply and shared_command_results_bind_identity_history_and_typed_refusals. Remote enrollment and standalone receipt lookup are specified in later steps. Run command_grants_bound_tombstones_without_changing_read_permission in the full transport suite to qualify independent read grants, write tombstone caps and shared-pool cleanup. Distinguish follower execution-authority requests as NotLeader from leaders awaiting durable apply (NotReady); preserve stale captured-authority checks.", + "tests": [ + "shared_remote_commands_require_write_grants_and_exact_receipts", + "shared_remote_commands_survive_disconnected_sql_waiters", + "shared_remote_command_revocation_preserves_admitted_ownership" + ] + }, + { + "action": "Route enrollment through the same bounded write RPC exchange, explicit namespace/process grants and exact controller registration as commands. Authenticate Query, Command and Enrollment as distinct mandatory purposes under marker 1, with no legacy reader. Preserve creation identity, captured inputs, preview identity and original routing budget. Recheck fresh destination catalog and leader authority before transferring initialization to the controller; return only the exact committed creation record, never treat the initial image as a current version token. Reuse shared reply budgets and preserve ownership after response disconnect, revocation and publication deadline. Run all query_wire tests including shared_enrollment_wire_preserves_captured_identity_and_inputs, shared_enrollment_purpose_cannot_be_reinterpreted_as_command_or_read and shared_enrollment_reply_binds_creation_cell_schema_and_initial_image, plus all existing command/query and controller enrollment regressions. Standalone receipt lookup follows below; dynamic roster and public SDK cutover remain open.", + "tests": [ + "shared_remote_enrollment_requires_write_grants_and_exact_creation", + "shared_remote_enrollment_survives_disconnected_initializer", + "shared_remote_enrollment_revocation_retains_admitted_creation", + "shared_remote_enrollment_resolves_after_publication_deadline" + ] + }, + { + "action": "Add standalone local and remote receipt inspection with a distinct authenticated purpose on the bounded existing RPC exchange. Require explicit command-origin permission and exact controller/SQL ownership; validate command contract, payload digest and authenticated identity before reading receipts. Select a fresh catalog-bound quorum image and use immutable SQL without staging a producer, installing schema, maintaining receipts, or invoking a command/initializer. Return exact retained result/identity and signed cell observation, or ambiguous absence; absence must never retire an unresolved producer. Reuse physical read cancellation, deadline, byte accounting and joined cleanup. Run full query_wire and receipts suites, including shared_receipt_wire_refuses_command_payloads_and_preserves_identity, shared_receipt_reply_distinguishes_unknown_from_empty_committed_result, shared_receipt_purpose_cannot_authorize_mutation and cell_lookup_distinguishes_initial_absence_from_lost_schema. Keep corrupt/partial schema and replicated expiry distinct from absence. Dynamic roster/lazy group birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete.", + "tests": [ + "shared_receipt_lookup_requires_grants_and_never_executes_commands", + "shared_receipt_lookup_keeps_unresolved_producers_at_capacity", + "shared_receipt_lookup_cancellation_preserves_active_writer" + ] + }, + { + "action": "Replace the fixed controller roster with bounded dynamic registration, including empty-controller startup. Reserve pending plus active group slots and shared command item/byte admission before transfer. Bind exact engine/history/schema, selected root and applier page owner; install an exclusively owned physical route without adopting a prior registration. Register the applier before polling attachment so early consensus/apply events remain owned. Revalidate the bootstrap catalog through the existing runtime attach path; expose application roster membership only after success, with fresh quorum still required. Retain attachment through response cancellation and join catalog/runtime attachment before closing physical dependencies. Release definite failures before responding; unexpected competing runtime owners or runtime closure keep the conservative node shutdown boundary. Complete-group membership changes follow CG-05; production request/voter orchestration and SDK/fleet binding follow CG-07. Run the full transport and shared runtime suites.", + "tests": [ + "dynamic_registration_preserves_live_groups_and_disconnected_waiters", + "dynamic_registration_refuses_foreign_owners_and_retries_failed_catalog", + "dynamic_registration_joins_attachment_before_cancelled_shutdown", + "dynamic_registration_recovers_selected_roots_into_an_empty_controller" + ] + }, + { + "action": "Isolate group-local controller apply/runtime/ambiguous command failures through a bounded retained quarantine lifecycle. Reserve a control lane to revoke execution while apply is waiting, refuse new group admission, preserve captured producer/apply owners until joined shutdown, and prevent late persistence completion from restoring authority. Keep failed group capacity charged. Shared engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Expose original group failure and quarantine counts; no in-place replacement or automatic callback replay. Qualify healthy-group SQL progress and node-wide storage failure separately. Explicit local provisioning uses the lifecycle below; production configuration and public cutover are integrated in CG-07 and CG-16.", + "tests": [ + "group_quarantine_retains_producer_and_serves_healthy_group", + "shared_engine_sync_failure_still_stops_all_groups" + ] + }, + { + "action": "Provision already selected initial-member ranges through the controller with immutable explicit filesystem/image/peer settings. Bound preparation bytes, parallelism, resident failed preparations and response waiters independently from commands. Coalesce by exact physical group and directory owner; preserve recovered page owners across attachment retries. Register appliers before polling attachment and join entered catalog/filesystem work before physical shutdown. This explicit local API does not allocate ranges, orchestrate followers, authorize managed fleet admission or wire automatic request-driven/public SDK startup. CG-07 integrates those production paths after its snapshot, archive and effect dependencies.", + "tests": [ + "provisioning_coalesces_and_serves", + "provisioning_retries_retained_owner", + "provisioning_joins_cancelled_close", + "provisioning_recovers_selected_roots", + "provisioning_refusals_preserve_healthy_group" + ] + } + ], + "test_cases": [ + { + "name": "three_cells_one_rawnode_per_voter", + "oracle": "Three separate SQLite histories are served by exactly three RawNodes total across three voters; interleaved values agree on all replicas.", + "invariants": [ + "I01", + "I02", + "I04" + ] + }, + { + "name": "shared_group_fair_admission", + "oracle": "Concurrent cells receive bounded fair admission; cancellation retains physical accounting and busy SQL never blocks consensus ticks.", + "invariants": [ + "I12" + ] + }, + { + "name": "cell_receipt_not_group_progress", + "oracle": "A later update to B cannot settle or replace A; identical client/sequence in A and B remains independent.", + "invariants": [ + "I02", + "I03" + ] + }, + { + "name": "strong_read_at_group_barrier", + "oracle": "ReadIndex through B serves unchanged A coherently; stale leader/minority/incorrect incarnation cannot satisfy a strong read.", + "invariants": [ + "I01", + "I05" + ] + }, + { + "name": "apply_prefix_never_skips_slow_cell", + "oracle": "Out-of-order staging cannot publish beyond a missing earlier cell entry after crash or election.", + "invariants": [ + "I04" + ] + }, + { + "name": "concurrent_enrollment_one_cell", + "oracle": "Actual simultaneous enrollment through the real group owner creates one SQLite history per cell and one initialization effect on all voters, including a lost response and leader restart.", + "invariants": [ + "I01", + "I03", + "I04" + ] + }, + { + "name": "pending_enrollments_respect_group_capacity", + "oracle": "Before any apply, accepted distinct creations fill the configured group capacity; exact retries reserve once, conflicting incarnations refuse and a further creation is rejected before append. Existing cells can still mutate at capacity.", + "invariants": [ + "I01", + "I04", + "I12" + ] + }, + { + "name": "proposal_acceptance_retries_do_not_append_twice", + "oracle": "An exact pending mutation retry returns its original log position and digest at saturated acceptance capacity; changed bytes under that identity refuse. After apply, a stale base cannot append again, the original image advances once on every voter and released capacity admits another cell.", + "invariants": [ + "I02", + "I03", + "I04", + "I12" + ] + }, + { + "name": "inherited_enrollments_count_after_applied_term_barrier", + "oracle": "Restart all voters from retained disks with an uncommitted enrollment present only on the longest log. The recovered leader refuses admission until inherited history and its current-term barrier apply, then counts that cell against capacity. Old prepared terms refuse.", + "invariants": [ + "I01", + "I04", + "I12" + ] + }, + { + "name": "node_runtime_bounds_proposal_acceptance_across_groups", + "oracle": "Pending acceptance in one physical group exhausts the shared node budget and refuses a different group, while exact retries retain their original position. Peer persistence and durable apply still progress under saturation, releasing capacity for the other group on all voters.", + "invariants": [ + "I04", + "I12" + ] + }, + { + "name": "sql_submission_rejection_seals_delayed_retries", + "oracle": "Actual three-voter SQL initialization and mutation distinguish queue overload from definitive owner refusal. Lost rejection responses are recovered from the sealed preparation; foreign-engine, wrong-cell and same-proposal replacement proofs refuse. Old clones remain rejected after capacity returns, remembered acceptance never re-appends after apply, and worker cleanup releases names while retained outputs keep bounded reservations. Both image modes return all retained resources on close.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "rooted_sql_submission_rejection_seals_delayed_retries", + "oracle": "Actual three-voter SQL initialization and mutation distinguish queue overload from definitive owner refusal. Lost rejection responses are recovered from the sealed preparation; foreign-engine, wrong-cell and same-proposal replacement proofs refuse. Old clones remain rejected after capacity returns, remembered acceptance never re-appends after apply, and worker cleanup releases names while retained outputs keep bounded reservations. Both image modes return all retained resources on close.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "rooted_strong_read_at_group_barrier", + "oracle": "Extent-backed strong reads require fresh quorum and contiguous apply behind another cell, preserve the target cell version, wait for AtLeast publication, retain older coherent pins and original cell witnesses for authenticated version tokens, and refuse wrong histories, expired deadlines and revoked leadership.", + "invariants": [ + "I01", + "I05", + "I12" + ] + }, + { + "name": "cell_query_pins_deadline_and_cancelled_cleanup", + "oracle": "Real group read pins enforce capacity and engine binding. Immutable SQL refuses writes, interrupts deadline-bound work, contains callback panics and retains entered work after waiter cancellation; close joins it and returns all SQL reservations.", + "invariants": [ + "I01", + "I05", + "I12" + ] + }, + { + "name": "rooted_cell_query_pins_deadline_and_cancelled_cleanup", + "oracle": "Real group read pins enforce capacity and engine binding. Immutable SQL refuses writes, interrupts deadline-bound work, contains callback panics and retains entered work after waiter cancellation; close joins it and returns all SQL reservations.", + "invariants": [ + "I01", + "I05", + "I12" + ] + }, + { + "name": "shared_workers_prepare_committed_cell_images", + "oracle": "The existing fixed SQL workers prepare committed enrollment and mutation images on all three voters. Wrong engines and forged entries refuse; exact duplicate preparations yield the same manifest; stale bases refuse after publication. Business admission saturation does not prevent apply. Derived scheduling hints match actual SQLite state. Preparation leaves the old root selected, and its independent reservation survives clones and blocked physical publication until the owner completion is consumed.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "rooted_shared_workers_prepare_committed_cell_images", + "oracle": "The existing fixed SQL workers prepare committed enrollment and mutation images on all three voters. Wrong engines and forged entries refuse; exact duplicate preparations yield the same manifest; stale bases refuse after publication. Business admission saturation does not prevent apply. Derived scheduling hints match actual SQLite state. Preparation leaves the old root selected, and its independent reservation survives clones and blocked physical publication until the owner completion is consumed.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "shared_apply_cancelled_close_joins_preparation", + "oracle": "Hold actual image and manifest directory barriers separately. Cancel the apply receiver and a close waiter; entered work, bytes, progress slot and exclusive SQL lease remain retained until release. Joined close refunds the reservations without publishing; a fresh pool repeats exact preparation and the real Raft owner publishes it.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "rooted_shared_apply_cancelled_close_joins_preparation", + "oracle": "Hold actual image and manifest directory barriers separately. Cancel the apply receiver and a close waiter; entered work, bytes, progress slot and exclusive SQL lease remain retained until release. Joined close refunds the reservations without publishing; a fresh pool repeats exact preparation and the real Raft owner publishes it.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "shared_apply_sync_failure_preserves_selected_root", + "oracle": "Inject actual image and manifest directory-sync failures separately. Preparation fails, refunds its admission, and never advances the selected root or applied index despite the entry being committed.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "rooted_shared_apply_sync_failure_preserves_selected_root", + "oracle": "Inject actual image and manifest directory-sync failures separately. Preparation fails, refunds its admission, and never advances the selected root or applied index despite the entry being committed.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "shared_apply_recovers_selected_images_before_replay", + "oracle": "Close all previous runtime, engine, manifest and image owners. Recover the engine-selected manifest and physical image closure in a new process generation, ignoring an already prepared but unpublished cell. Replay that committed entry without inventing fresh quorum/serving authority. After another cold restart, missing selected image files must refuse recovery without changing the persisted marker.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "rooted_shared_apply_recovers_selected_images_before_replay", + "oracle": "Close all previous runtime, engine, manifest and image owners. Recover the engine-selected manifest and physical image closure in a new process generation, ignoring an already prepared but unpublished cell. Replay that committed entry without inventing fresh quorum/serving authority. After another cold restart, missing selected image files must refuse recovery without changing the persisted marker.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "registered_commands_use_cell_identity_and_receipts", + "oracle": "Dispatch a real registered class through the shared SQL pool and production image applier on three voters. Two logical cells in one group receive their exact histories, mutate independently and retain alarm hints. After B advances group progress, A returns its original receipt without rerunning its callback. Wrong contract, input digest, operation and class schema refuse even on a receipt retry, releasing physical admission.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "rooted_registered_commands_use_cell_identity_and_receipts", + "oracle": "Dispatch a real registered class through the shared SQL pool and production image applier on three voters. Two logical cells in one group receive their exact histories, mutate independently and retain alarm hints. After B advances group progress, A returns its original receipt without rerunning its callback. Wrong contract, input digest, operation and class schema refuse even on a receipt retry, releasing physical admission.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "registered_queries_keep_cell_observation_and_ownership", + "oracle": "Run a registered read-only callback on the existing worker with a real selected cell observation. An older A pin remains readable after A and B mutate; a fresh linearizable A query returns A's sequence and decision witness, not B's group progress, and round-trips its signed token. Invalid contract/kind/operation/schema/input refuses before callback execution. Panic and deadline interruption leave the worker usable. Cancelled query/close retain input, image and physical callback ownership until joined cleanup.", + "invariants": [ + "I01", + "I05", + "I12" + ] + }, + { + "name": "rooted_registered_queries_keep_cell_observation_and_ownership", + "oracle": "Run a registered read-only callback on the existing worker with a real selected cell observation. An older A pin remains readable after A and B mutate; a fresh linearizable A query returns A's sequence and decision witness, not B's group progress, and round-trips its signed token. Invalid contract/kind/operation/schema/input refuses before callback execution. Panic and deadline interruption leave the worker usable. Cancelled query/close retain input, image and physical callback ownership until joined cleanup.", + "invariants": [ + "I01", + "I05", + "I12" + ] + }, + { + "name": "catalog_routed_queries_preserve_invocation_and_authority", + "oracle": "Resolve a real catalog route and execute a registered linearizable query on three voters. A dropped response retries the same fingerprint/deadline, an authenticated AtLeast token reads on a follower, and a foreign cell token/domain/class refuses before callback execution. Invocation admission is shared across clones and total attempts/deadlines cannot reset. A catalog-confirmed replacement preserves request identity but refuses execution without its physical owner. Cancelled routing releases only its own reservation; held SQL remains charged and joined.", + "invariants": [ + "I01", + "I05", + "I12" + ] + }, + { + "name": "rooted_catalog_routed_queries_preserve_invocation_and_authority", + "oracle": "Resolve a real catalog route and execute a registered linearizable query on three voters. A dropped response retries the same fingerprint/deadline, an authenticated AtLeast token reads on a follower, and a foreign cell token/domain/class refuses before callback execution. Invocation admission is shared across clones and total attempts/deadlines cannot reset. A catalog-confirmed replacement preserves request identity but refuses execution without its physical owner. Cancelled routing releases only its own reservation; held SQL remains charged and joined.", + "invariants": [ + "I01", + "I05", + "I12" + ] + }, + { + "name": "catalog_routed_commands_preserve_producer_and_receipts", + "oracle": "A real catalog-routed registered command on three voters resumes a held/cancelled preparation without another callback or a foreign-node producer. Repeated submission retains the original acceptance; same-term receipt absence stays ambiguous and charged. Exact publication returns the original cell receipt and token, including a duplicate after another cell advances group progress. Real leader transfer permits NotCommitted only behind the newer applied term, then retry preserves captured time/randomness and command identity. S3 request and namespace minima refuse without weakening durability.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "rooted_catalog_routed_commands_preserve_producer_and_receipts", + "oracle": "A real catalog-routed registered command on three voters resumes a held/cancelled preparation without another callback or a foreign-node producer. Repeated submission retains the original acceptance; same-term receipt absence stays ambiguous and charged. Exact publication returns the original cell receipt and token, including a duplicate after another cell advances group progress. Real leader transfer permits NotCommitted only behind the newer applied term, then retry preserves captured time/randomness and command identity. S3 request and namespace minima refuse without weakening durability.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "physical_group_transport_multiplexes_real_raft", + "oracle": "Two physical Raft groups on three real TLS endpoints replicate three independent SQLite cells each using the same bounded per-peer connection pools. Unknown groups, wrong schema/local process, conflicting query/inspection endpoints and unsupported snapshots refuse. Removing one group preserves the other group and peer pools; restored routing delivers again. Exclusive transport ownership refuses a second service until joined shutdown.", + "invariants": [ + "I01", + "I03", + "I12" + ] + }, + { + "name": "catalog_commands_replicate_over_physical_tls", + "oracle": "Run the catalog-routed registered command cancellation, duplicate acceptance, exact receipt and real leader-transfer retry assertions over production physical-group TLS delivery in this image mode. Preserve captured command inputs and original producer ownership; no fixture-direct peer delivery occurs after transport attachment. The fixture still drives runtime apply and completion events, so this does not qualify production controller ownership.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "rooted_catalog_commands_replicate_over_physical_tls", + "oracle": "Run the catalog-routed registered command cancellation, duplicate acceptance, exact receipt and real leader-transfer retry assertions over production physical-group TLS delivery in this image mode. Preserve captured command inputs and original producer ownership; no fixture-direct peer delivery occurs after transport attachment. The fixture still drives runtime apply and completion events, so this does not qualify production controller ownership.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "group_controller_drives_tls_apply_and_bounded_retry", + "oracle": "Two groups replicate over real TLS using one SQL worker per node. The controller exclusively drives events, preserves per-group apply order and retries SQL capacity without fixture pumping.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "rooted_group_controller_drives_tls_apply_and_bounded_retry", + "oracle": "Repeat autonomous multi-group TLS replication, bounded SQL backoff and exact receipt publication with extent images.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "group_controller_checks_exact_owners", + "oracle": "Refuse foreign runtime/event/transport and engine/SQL/applier combinations even within the same execution domain.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "group_controller_cancelled_close_joins_sql", + "oracle": "Cancel a close waiter or drop the final public controller handle while physical work is blocked. Retain claims and tickets until physical SQL and runtime joins complete.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "group_controller_apply_failure_and_shutdown_preserve_selected_root", + "oracle": "A real manifest sync failure quarantines the group without advancing the selected root. Captured authority refuses; explicit shutdown still joins physical ownership. Interrupted preparation cannot publish or allow premature replacement.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "rooted_group_controller_apply_failure_and_shutdown_preserve_selected_root", + "oracle": "Repeat group quarantine after manifest sync failure and joined controller shutdown with extent images; preserve the engine-selected root and physical ownership.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "controller_owned_commands_survive_sql_disconnect", + "oracle": "Drop the response while a registered SQL callback is physically held. Retain bounded controller/producer/invocation admission, then settle the exact receipt without callback replay in both image modes.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "controller_owned_commands_seal_late_preparations", + "oracle": "Expire the response while a callback is held. Seal the eventual never-submitted preparation, join private cleanup and leave the selected cell unchanged. An explicit retry may execute once, in both image modes.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "controller_owned_commands_survive_accepted_disconnect", + "oracle": "Drop the response after commitment while manifest sync is held. Preserve command ownership until exact publication, and return the original receipt on identity retry without replay in both image modes.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "controller_owned_commands_resolve_after_response_deadline", + "oracle": "Expire the original response deadline after commitment while manifest sync is held. Continue bounded resolution and settle the exact receipt without replay in both image modes.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "controller_owned_commands_join_cancelled_shutdown", + "oracle": "Cancel a close waiter during a physically blocked callback. Retain the invocation and engine/SQL claims until joined shutdown; release all command admission only afterward, in both image modes.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "controller_owned_commands_release_completed_callback_errors", + "oracle": "A received callback error discards its private mutation and releases command ownership without closing the node. Explicit identity retry succeeds exactly once; both image modes.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "controller_owned_commands_join_a_lost_sql_reply", + "oracle": "An injected physical image-capture panic loses the stage reply. Preserve ambiguous invocation ownership through controller failure and physical SQL joins, then release admission; both image modes.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "controller_enrollment_survives_initializer_disconnect", + "oracle": "Drop the response while the registered initializer is held on a real SQL worker. Retain shared command/enrollment admission, settle the original creation record after release, and return it unchanged after a later mutation. Conflicting creation IDs/incarnations refuse without callback replay in both image modes.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "controller_enrollment_seals_late_preparation", + "oracle": "Expire the response while initialization is held. Seal the eventual never-attempted preparation, join private cleanup and leave the cell absent. An explicit creation retry may initialize once; both image modes.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "controller_enrollment_survives_accepted_disconnect", + "oracle": "Disconnect after commitment while manifest publication is held. Retain creation ownership until exact publication, then return the same creation record without reinitialization, including after mutation; both image modes.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "controller_enrollment_resolves_after_deadline", + "oracle": "Expire the response after commitment while manifest sync is held. Continue bounded creation resolution independently of the waiter, and preserve the original creation record on retry in both image modes.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "controller_enrollment_joins_cancelled_shutdown", + "oracle": "Cancel controller close during a held initializer. Preserve SQL admission and engine claims until physical worker joins, then release all invocation/command retention in both image modes.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "controller_enrollment_releases_completed_initializer_errors", + "oracle": "A completed initializer error rolls back private schema changes and releases invocation ownership without closing the node. Explicit same-creation retry commits once in both image modes.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "controller_enrollment_joins_lost_worker_reply", + "oracle": "A physical capture panic loses the initialization reply. Keep ambiguous creation ownership through failed-controller cleanup and physical worker joins, releasing admission only afterward; both image modes.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "controller_enrollment_requires_registered_initializer", + "oracle": "Refuse enrollment before routing/SQL when the namespace-bound class has no registered initializer, and release invocation admission.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "shared_remote_queries_bind_cell_versions_and_gateway_owners", + "oracle": "Two cells share one physical group and production query TLS lane in both image modes. Linearizable, stale and AtLeast reads preserve actual cell observations and signed tokens. Retries retain deadlines/attempt limits; unconsumed replies retain transport admission. Foreign tokens, logical incarnations, signing keys, SQL owners and duplicate/missing gateway registrations refuse. A catalog replacement after route capture refuses at the old destination without callbacks; a confirmed redirect cannot invent the new physical owner. Concurrent callers query different cells through the same peer lane without replay refusals.", + "invariants": [ + "I01", + "I05", + "I12" + ] + }, + { + "name": "shared_remote_queries_retain_cancelled_sql_until_join", + "oracle": "Disconnect a remote query while its compiled SQL callback is physically held, in both image modes. Raft still establishes another linearizable read barrier. Routing/transport reservations release after cancellation, but SQL input/image/work ownership remains charged. Cancelling controller close cannot release the worker before its physical join. Dropping the final host gateway registration while the original callback is held refuses new reads before any additional callback; in-flight data retention cannot extend admission.", + "invariants": [ + "I01", + "I05", + "I12" + ] + }, + { + "name": "shared_remote_queries_retain_expired_sql_until_join", + "oracle": "Expire the original remote-query deadline while SQL is held, in both image modes. Node/peer query admission drains while the SQL owner remains retained. Raft control progress continues and cancelled shutdown waits for physical cleanup. Dropping the final host gateway registration while the original callback is held refuses new reads before any additional callback; in-flight data retention cannot extend admission.", + "invariants": [ + "I01", + "I05", + "I12" + ] + }, + { + "name": "shared_nonmember_queries_require_grants_without_local_replicas", + "oracle": "A fourth real TLS endpoint queries two cells in both image modes without local group storage or a RawNode. Discovery alone and wrong-namespace/process grants refuse before SQL. Explicit grants permit leader strong reads and follower stale/AtLeast reads with authenticated logical observations. A strong attempt at a follower refuses before SQL; retrying the leader preserves the original deadline and total attempt budget. Consensus submission remains refused. Read discovery shares route/peer limits, rejects conflicting endpoints and stale revisions, and releases unused pools on removal.", + "invariants": [ + "I01", + "I05", + "I12" + ] + }, + { + "name": "shared_nonmember_query_revocation_retains_physical_sql", + "oracle": "Revoke the nonmember namespace grant while its remote SQL callback is physically held, in both image modes. The waiter refuses and transport reservations drain while physical SQL remains owned. Stale grant revision cannot reopen admission and new requests cannot enter SQL. A newer explicit grant permits reads after physical cleanup; no local replica appears.", + "invariants": [ + "I01", + "I05", + "I12" + ] + }, + { + "name": "shared_nonmember_query_route_replacement_refuses_stale_reply", + "oracle": "Replace the router discovery revision while remote SQL is held, in both image modes. The complete authenticated reply under the captured old route is refused as stale. A new attempt under the replacement succeeds, without leaking invocation/transport ownership or creating a router replica.", + "invariants": [ + "I01", + "I05", + "I12" + ] + }, + { + "name": "shared_cached_dispatch_preserves_authority_during_catalog_outage", + "oracle": "Warm one range hint, remove the actual catalog head, then query another cell, commit a command and enroll a new cell through real controller/SQL/TLS owners in both image modes without fresh mapping reads. Cache occupancy stays one range. Strong local reads still use real quorum. Explicit refresh fails; remote gateway admission remains fresh and refuses while the catalog head is absent. Restore metadata and join every owner.", + "invariants": [ + "I01", + "I05", + "I12" + ] + }, + { + "name": "shared_remote_commands_require_write_grants_and_exact_receipts", + "oracle": "Over real TLS and both image modes, a router without local group storage cannot invoke commands using discovery, a read grant, a wrong-process or foreign-namespace write grant, or voter membership. A fully registered follower with write permission still refuses before SQL. Explicit namespace/process write permission at the leader permits a committed receipt with authenticated cell observation. Same-identity retries return that receipt without another callback; conflicting input refuses. Original attempt deadlines, exact controller ownership, response byte admission and host-registration drop remain enforced.", + "invariants": [ + "I01", + "I03", + "I05", + "I12" + ] + }, + { + "name": "shared_remote_commands_survive_disconnected_sql_waiters", + "oracle": "Hold entered command SQL behind a physical callback gate, disconnect the remote waiter, and verify the destination controller retains the producer. Release SQL, observe ownership resolve, then retry the exact command and obtain one committed effect in both image modes. No fixture event pumping or local router replica is used.", + "invariants": [ + "I01", + "I03", + "I05", + "I12" + ] + }, + { + "name": "shared_remote_command_revocation_preserves_admitted_ownership", + "oracle": "During a held admitted command, revoke the explicit write grant. The remote response terminates ambiguously while controller ownership remains. Stale grant revisions refuse; after physical completion a newer grant allows the original committed receipt to be returned without another callback. Run both image modes with no read grant.", + "invariants": [ + "I01", + "I03", + "I05", + "I12" + ] + }, + { + "name": "shared_remote_enrollment_requires_write_grants_and_exact_creation", + "oracle": "Over real TLS in both image modes, nonmember ingress without write grants, read-only ingress and voter membership cannot enroll. A fully registered follower with a write grant returns NotLeader before initialization. Leader admission returns the exact creation identity and original image; duplicate enrollment after a later remote command preserves the original record without rerunning initialization. Creation/incarnation conflicts refuse. Consumed attempts, original deadlines, response admission and gateway drop remain enforced; routed work cannot masquerade as fresh local work.", + "invariants": [ + "I01", + "I03", + "I05", + "I12" + ] + }, + { + "name": "shared_remote_enrollment_survives_disconnected_initializer", + "oracle": "Hold entered initialization, disconnect the remote waiter, and verify controller ownership survives while network admission drains. Release SQL and retry the exact creation identity; one initialized cell and the committed creation record survive in both image modes.", + "invariants": [ + "I01", + "I03", + "I05", + "I12" + ] + }, + { + "name": "shared_remote_enrollment_revocation_retains_admitted_creation", + "oracle": "Revoke explicit write permission while initialization is physically held. The remote error remains ambiguous and controller ownership remains until the creation resolves. Renew permission and recover the exact record without rerunning initialization, in both image modes.", + "invariants": [ + "I01", + "I03", + "I05", + "I12" + ] + }, + { + "name": "shared_remote_enrollment_resolves_after_publication_deadline", + "oracle": "Hold committed manifest publication until the remote deadline expires. The caller receives an ambiguous error while the controller retains accepted creation. Release publication and retry the same identity; the exact original record returns with one initialization in both image modes.", + "invariants": [ + "I01", + "I03", + "I05", + "I12" + ] + }, + { + "name": "shared_receipt_lookup_requires_grants_and_never_executes_commands", + "oracle": "Over four real TLS endpoints in one process and both image modes, receipt inspection requires explicit command-origin permission; discovery and read grants refuse. Initial receipt absence returns an authenticated cell observation without invoking the command. After a normal commit, lookup returns the exact retained result and retry horizon without a second callback. Conflicting payload and sequence gaps remain typed. Original deadlines, one-use attempts, result admission and gateway drop remain enforced; the router has no local group replica.", + "invariants": [ + "I01", + "I03", + "I05", + "I12" + ] + }, + { + "name": "shared_receipt_lookup_keeps_unresolved_producers_at_capacity", + "oracle": "With a prepared command retaining its producer, quorum lookup observes only the committed image and reports ambiguous absence. Holding that lookup result fills the two-reservation SQL budget; another lookup refuses overload without releasing the producer. It neither exposes the uncommitted receipt nor releases the producer. Contract/digest mismatch and non-quorum read authority refuse. After the original command publishes, lookup returns its receipt while leaving producer cleanup to its actual owner; no callback reruns. Closed controller handles cannot retain the engine claim after the SQL pool closes; a replacement pool starts. Run both image modes.", + "invariants": [ + "I01", + "I03", + "I05", + "I12" + ] + }, + { + "name": "shared_receipt_lookup_cancellation_preserves_active_writer", + "oracle": "Hold a local controller command inside SQL, queue a remote lookup on a bounded two-slot fixture worker, then disconnect the lookup. Network admission drains while the active writer remains controller-owned. Release SQL, join the cancelled read, and inspect the committed receipt without a second command callback in both image modes.", + "invariants": [ + "I01", + "I03", + "I05", + "I12" + ] + }, + { + "name": "dynamic_registration_preserves_live_groups_and_disconnected_waiters", + "oracle": "Three real voter controllers register a second group while its catalog reads are held; the first group continues replicated SQL apply. Pending and active slots enforce the same capacity, duplicate attachment refuses, and disconnected registration waiters cannot abandon ownership. After revalidation the new group establishes quorum and runs registered enrollment and exact creation retry through the shared SQL workers and TLS pools, in both image modes.", + "invariants": [ + "I01", + "I03", + "I05", + "I12" + ] + }, + { + "name": "dynamic_registration_refuses_foreign_owners_and_retries_failed_catalog", + "oracle": "Reject foreign engines, roots and a different page owner on the same engine. Refuse adoption of an existing transport route. A real bootstrap catalog-read failure releases only the failed registration slot/route; an already serving group continues. Retry the same prepared history successfully and enroll after real quorum, in both image modes.", + "invariants": [ + "I01", + "I03", + "I05", + "I12" + ] + }, + { + "name": "dynamic_registration_joins_attachment_before_cancelled_shutdown", + "oracle": "Hold real catalog revalidation after registration admission, drop the response and cancel the close waiter. The controller retains the pending slot and runtime engine claim until attachment joins. After release, close finishes and a replacement runtime may start; run both image modes.", + "invariants": [ + "I01", + "I03", + "I05", + "I12" + ] + }, + { + "name": "dynamic_registration_recovers_selected_roots_into_an_empty_controller", + "oracle": "After registering and mutating a group, join all controllers, close and reopen all engines from the existing voter disks under a new process generation, reopen manifest page owners and recover selected roots into fresh empty controllers. Attachment alone refuses leader authority; fresh quorum then returns the prior exact receipt without another SQL mutation. Verify the persisted value and cell sequence in both image modes. Image-store handles remain in the same process. This is joined engine/runtime/page-store restart, not independent-process or power-loss qualification.", + "invariants": [ + "I01", + "I03", + "I05", + "I12" + ] + }, + { + "name": "group_quarantine_retains_producer_and_serves_healthy_group", + "oracle": "A real manifest directory-sync failure quarantines one group in both image modes. Preserve its selected root, revoke captured leader authority and new reads/writes, refuse callback replay, and retain its ambiguous producer and group capacity. Dynamically register another group on the same engine/SQL/TLS resources and prove controller enrollment, command, exact retry and linearizable SQLite query remain usable with per-group producer admission. Joined close releases retained command admission.", + "invariants": [ + "I01", + "I03", + "I05", + "I12" + ] + }, + { + "name": "shared_engine_sync_failure_still_stops_all_groups", + "oracle": "A real shared Raft-engine sync failure remains node-wide in both image modes: both groups refuse serving, controller/physical services join, and replacement runtime admission refuses the uncertain engine.", + "invariants": [ + "I01", + "I03", + "I05", + "I12" + ] + }, + { + "name": "provisioning_coalesces_and_serves", + "oracle": "Starting from selected reserved ranges with no genesis or page owners, production provisioning materializes birth, verifies configured members, bootstraps all three engines, recovers roots and attaches real TLS peers. Same-owner concurrent calls share one preparation; dropped waiters cannot cancel it. Waiter, pending and active group limits refuse excess requests; conflicting assignments refuse. A single command slot remains usable. Both image modes then enroll, mutate and return the exact retry without another cell mutation.", + "invariants": [ + "I01", + "I03", + "I05", + "I12" + ] + }, + { + "name": "provisioning_retries_retained_owner", + "oracle": "A catalog failure before recovery releases preparation and group capacity. A real filesystem hook injects a catalog failure after physical recovery but before attachment; the exact page owner remains locked and charged. Retrying the same assignment succeeds without reopening or syncing the page store again. Both image modes retain the same selected-root owner.", + "invariants": [ + "I01", + "I03", + "I05", + "I12" + ] + }, + { + "name": "provisioning_joins_cancelled_close", + "oracle": "Hold a real manifest directory sync inside execution-domain recovery, cancel close and drop the response. Preparation admission and the runtime engine claim remain held; a replacement runtime refuses. Release the physical operation, then observe joined close and released preparation capacity in both image modes.", + "invariants": [ + "I01", + "I03", + "I05", + "I12" + ] + }, + { + "name": "provisioning_recovers_selected_roots", + "oracle": "After actual controller enrollment and mutation through provisioned groups, join all controllers and reopen all three voter engines under a new process generation. Empty controllers recover using the same production provisioner. Removing the engine-selected manifest page refuses recovery and leaves the selected marker unchanged; restoring its exact bytes permits recovery. Attachment alone refuses leader authority; fresh quorum then returns the prior exact receipt and a linearizable query verifies value 8 and cell sequence 2. Run both image modes. Image-store handles remain in process; this is not independent-process or power-loss qualification.", + "invariants": [ + "I01", + "I03", + "I05", + "I12" + ] + }, + { + "name": "provisioning_refusals_preserve_healthy_group", + "oracle": "In both image modes, configured peers missing an immutable birth member refuse before genesis and free preparation capacity. A configured but wrong local endpoint fails physical registration after recovery, retains the prepared owner and its budget, and leaves healthy groups serving actual controller commands. Neither refusal grants membership or authority. Joined close releases all owners.", + "invariants": [ + "I01", + "I03", + "I05", + "I12" + ] + } + ], + "invariants": [ + "I01", + "I02", + "I03", + "I04", + "I05", + "I12" + ], + "evidence": [ + ".cellar/cell-groups-evidence/CG-01-06-gates-e32115aa-02/CG-04-record.json" + ], + "stop_conditions": [ + "A current source excerpt no longer matches: review and record the successor baseline before edits.", + "An acknowledged effect, receipt, proof boundary or resource owner would be weakened to make a test pass.", + "The implementation requires an unrecorded wire/storage compatibility path or bypass of quorum/fencing." + ], + "out_of_scope": [ + "TiKV KV/MVCC APIs and cross-cell transactions", + "Online migration or deletion of existing user disks/archives", + "Feature-off Fleet/bucket durability semantics" + ], + "capability": "execution" + }, + { + "id": "CG-05", + "title": "Snapshot and rebuild complete groups through membership changes", + "status": "DONE", + "priority": "P1", + "effort": "L", + "risk": "HIGH", + "depends_on": [ + "CG-04" + ], + "anchors": [ + "snapshot", + "membership", + "manifest", + "extent" + ], + "scope": [ + "crates/cellar/durability/raft/", + "crates/cellar-ltx/src/", + "crates/cellar/tests/native_cell_groups_execution.rs", + "crates/cellar/tests/support/cell_group_checkpoint.rs", + "crates/cellar/tests/support/cell_group_checkpoint_receive.rs", + "crates/cellar/tests/support/cell_group_runtime.rs", + "crates/cellar/tests/support/cell_group_sql.rs", + "crates/cellar/tests/support/cell_group_sql_apply.rs", + "crates/cellar/tests/support/cell_group_snapshot_owner.rs", + "crates/cellar/durability/raft/cell_groups/checkpoint/local.rs", + "crates/cellar/tests/support/cell_group_checkpoint_local.rs", + "crates/cellar/application/consensus/mod.rs", + "crates/cellar/tests/support/cell_group_network_probes.rs", + "crates/cellar/tests/support/cell_group_controller.rs", + "crates/cellar/tests/support/cell_group_controller_checkpoints.rs", + "crates/cellar/tests/support/cell_group_membership.rs", + "crates/cellar/tests/support/cell_group_seed.rs", + "crates/cellar/tests/support/cell_group_seed_install.rs", + "crates/cellar/tests/support/cell_group_seed_probes.rs", + "crates/cellar/tests/support/cell_group_seed_receiver.rs", + "crates/cellar/tests/support/raft_storage_fault.rs" + ], + "creates": [ + "crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs", + "crates/cellar/tests/native_cell_groups_snapshot.rs", + "crates/cellar/tests/support/cell_group_seed_persistence.rs", + "crates/cellar/tests/support/cell_group_seed_crash.rs", + "crates/cellar/tests/support/cell_group_crash_fixture.rs", + "crates/cellar/tests/support/cell_group_membership_crash.rs", + "crates/cellar/tests/support/cell_group_checkpoint_pressure.rs", + "crates/cellar/tests/support/cell_group_engine_fixture.rs", + "crates/cellar/durability/raft/cell_groups/reclaim.rs", + "crates/cellar/tests/support/cell_group_node_reclaim.rs", + "crates/cellar/tests/support/cell_group_seed_retry.rs", + "crates/cellar/tests/support/cell_group_completion_pressure.rs", + "crates/cellar/tests/support/cell_group_checkpoint_fairness.rs" + ], + "test_target": "native_cell_groups_snapshot", + "steps": [ + { + "action": "Capture one group root at cut K with every enrolled cell, membership, range and transition state. Pin immutable dependencies while cells continue preparing later writes; stream bounded chunks rather than aggregate all images in memory.", + "tests": [ + "snapshot_contains_every_cell_at_cut" + ] + }, + { + "action": "Verify and sync every receiver dependency before atomic group install. Incomplete, mixed-cut or wrong-history snapshots fail closed. Match materialized and extent representations, including an enrolled empty cell.", + "tests": [ + "partial_snapshot_never_serves" + ] + }, + { + "action": "Adapt learner admission, joint configuration, leadership handoff and disk replacement to whole groups. Preserve permanent disk lineage and learner catch-up proofs; snapshots do not authorize new voters.", + "tests": [ + "whole_group_learner_and_joint_change" + ] + }, + { + "action": "Run real peer catch-up, process restart and lost-disk scenarios while a different cell writes. Reclaim transferred scratch only after physical owner release.", + "tests": [ + "snapshot_empty_cell_and_extent_restart", + "snapshot_memory_and_pin_bounds" + ] + } + ], + "test_cases": [ + { + "name": "snapshot_contains_every_cell_at_cut", + "oracle": "A concurrent writer cannot produce a snapshot mixing cells from different group cuts.", + "invariants": [ + "I04", + "I07" + ] + }, + { + "name": "partial_snapshot_never_serves", + "oracle": "Interrupted/corrupt/foreign dependency install leaves the prior root or refuses admission.", + "invariants": [ + "I01", + "I04" + ] + }, + { + "name": "whole_group_learner_and_joint_change", + "oracle": "Fresh learner installs all cells before promotion; one failed voter remains tolerable and minority writes refuse.", + "invariants": [ + "I02", + "I08" + ] + }, + { + "name": "snapshot_empty_cell_and_extent_restart", + "oracle": "Empty enrolled cells and extent/materialized populated cells survive checkpoint and retained restart.", + "invariants": [ + "I03", + "I04" + ] + }, + { + "name": "snapshot_memory_and_pin_bounds", + "oracle": "Large group transfer stays within declared scratch/buffer slots and preserves live reader pins.", + "invariants": [ + "I07", + "I12" + ] + } + ], + "invariants": [ + "I01", + "I02", + "I03", + "I04", + "I07", + "I08", + "I12" + ], + "evidence": [ + ".cellar/cell-groups-evidence/CG-01-06-gates-e32115aa-02/CG-05-record.json" + ], + "stop_conditions": [ + "A current source excerpt no longer matches: review and record the successor baseline before edits.", + "An acknowledged effect, receipt, proof boundary or resource owner would be weakened to make a test pass.", + "The implementation requires an unrecorded wire/storage compatibility path or bypass of quorum/fencing." + ], + "out_of_scope": [ + "TiKV KV/MVCC APIs and cross-cell transactions", + "Online migration or deletion of existing user disks/archives", + "Feature-off Fleet/bucket durability semantics" + ], + "capability": "snapshot" + }, + { + "id": "CG-06", + "title": "Archive and reclaim the complete group recovery history", + "status": "DONE", + "priority": "P1", + "effort": "L", + "risk": "HIGH", + "depends_on": [ + "CG-04", + "CG-05" + ], + "anchors": [ + "archive", + "snapshot", + "disaster", + "manifest" + ], + "scope": [ + "crates/cellar/durability/raft/", + "crates/cellar-ltx/src/", + "crates/cellar/tests/support/cell_group_checkpoint.rs", + "crates/cellar/tests/support/cell_group_engine_fixture.rs", + "crates/cellar/tests/support/cell_group_runtime.rs", + "crates/cellar/tests/support/cell_group_sql.rs", + "crates/cellar/tests/support/cell_group_s3_commands.rs" + ], + "creates": [ + "crates/cellar/durability/raft/cell_groups/archive/mod.rs", + "crates/cellar/durability/raft/cell_groups/archive/checkpoint.rs", + "crates/cellar/durability/raft/cell_groups/archive/objects.rs", + "crates/cellar/tests/native_cell_groups_archive.rs", + "crates/cellar/tests/support/cell_group_archive_store.rs", + "crates/cellar/durability/raft/cell_groups/archive/tail.rs", + "crates/cellar/durability/raft/cell_groups/archive/tail_admission.rs", + "crates/cellar/durability/raft/cell_groups/archive/tail_codec.rs", + "crates/cellar/durability/raft/cell_groups/archive/replay.rs", + "crates/cellar/durability/raft/cell_groups/apply/transition.rs", + "crates/cellar/tests/support/cell_group_archive_tail.rs", + "crates/cellar/durability/raft/cell_groups/archive/data_store.rs", + "crates/cellar/durability/raft/cell_groups/checkpoint/receive_archive.rs", + "crates/cellar/tests/support/cell_group_archive_isolation.rs", + "crates/cellar/durability/raft/cell_groups/archive/collection.rs", + "crates/cellar/durability/raft/cell_groups/archive/graph.rs", + "crates/cellar/durability/raft/cell_groups/archive/publication.rs", + "crates/cellar/durability/raft/cell_groups/archive/upload.rs", + "crates/cellar/durability/raft/cell_groups/archive/ownership/mod.rs", + "crates/cellar/durability/raft/cell_groups/archive/ownership/state.rs", + "crates/cellar/durability/raft/cell_groups/archive/ownership/wire.rs", + "crates/cellar/tests/support/cell_group_archive_publication.rs", + "crates/cellar/durability/raft/cell_groups/directory/archive.rs", + "crates/cellar/tests/support/cell_group_archive_binding.rs", + "crates/cellar/durability/raft/cell_groups/archive/execution.rs", + "crates/cellar/durability/raft/cell_groups/archive/execution_tests.rs", + "crates/cellar/durability/raft/cell_groups/controller/archive/mod.rs", + "crates/cellar/durability/raft/cell_groups/controller/archive/job.rs", + "crates/cellar/durability/raft/cell_groups/controller/archive/journal.rs", + "crates/cellar/durability/raft/cell_groups/controller/archive/journal_tests.rs", + "crates/cellar/tests/native_cell_groups_archive_controller.rs", + "crates/cellar/tests/support/cell_group_controller_archive.rs", + "crates/cellar/tests/support/cell_group_rustfs.rs", + "crates/cellar/tests/support/cell_group_s3_response.rs", + "tools/multi-raft/cell_groups.py", + "crates/cellar/durability/raft/cell_groups/archive/receipt.rs", + "crates/cellar/durability/raft/cell_groups/controller/archive/settlement.rs", + "crates/cellar/tests/support/cell_group_s3_commands.rs", + "crates/cellar/durability/raft/cell_groups/archive/enrollment.rs", + "crates/cellar/tests/support/cell_group_s3_enrollment.rs", + "crates/cellar/tests/support/cell_group_s3_receipts.rs", + "crates/cellar/durability/raft/query_reply.rs", + "crates/cellar/tests/support/cell_group_s3_remote.rs", + "crates/cellar/tests/support/cell_group_s3_remote_wire.rs", + "crates/cellar/durability/raft/cell_groups/archive_compaction.rs", + "crates/cellar/tests/support/cell_group_archive_dependencies.rs", + "crates/cellar/tests/support/cell_group_s3_outage.rs", + "crates/cellar/tests/support/cell_group_backend_control.rs", + "tools/multi-raft/backend_control.py", + "tools/multi-raft/test_backend_control.py", + "crates/cellar/durability/raft/cell_groups/directory/recovery/mod.rs", + "crates/cellar/durability/raft/cell_groups/directory/recovery/request.rs", + "crates/cellar/tests/support/cell_group_disaster_fence.rs", + "crates/cellar/durability/raft/cell_groups/recovery/mod.rs", + "crates/cellar/durability/raft/cell_groups/recovery/state.rs", + "crates/cellar/durability/raft/cell_groups/recovery/admission.rs", + "crates/cellar/durability/raft/cell_groups/recovery/import.rs", + "crates/cellar/durability/raft/cell_groups/recovery/tests.rs", + "crates/cellar/durability/raft/cell_groups/recovery/genesis.rs", + "crates/cellar/durability/raft/cell_groups/directory/recovery/seed.rs", + "crates/cellar/tests/support/cell_group_disaster_seed.rs", + "crates/cellar/durability/raft/cell_groups/directory/recovery/publication.rs", + "crates/cellar/durability/raft/cell_groups/directory/recovery/bootstrap.rs", + "crates/cellar/durability/raft/cell_groups/directory/recovery/activation.rs", + "crates/cellar/tests/support/cell_group_disaster_activation.rs" + ], + "test_target": "native_cell_groups_archive", + "steps": [ + { + "action": "Replace single-cell archive envelopes with complete group checkpoint and cell-addressed tail closures. Verify membership, administrative entries and every cell base chain before publishing group coverage; reuse unchanged immutable payloads.", + "tests": [ + "group_archive_requires_all_dependencies" + ] + }, + { + "action": "Keep Quorum acknowledgement independent from asynchronous S3 publication; S3-mode settlement requires complete confirmed group coverage. Bound unarchived bytes and expose group lag without crediting uploaded-but-unverified objects.", + "tests": [ + "quorum_continues_bounded_s3_outage" + ] + }, + { + "action": "Tie local truncation and physical shared-engine purge to covering snapshots, archive policy and all live group/reader/transition pins. Preserve lagging-peer snapshot availability and prevent deleting another group range.", + "tests": [ + "group_reclaim_restart_keeps_history" + ] + }, + { + "action": "Implement unmanaged fresh-incarnation whole-group S3 seeding through the existing fencing authority. Preserve selected-cut receipts, reject old tokens and prove no callback replay; later CG-13 adds managed and in-transition recovery.", + "tests": [ + "fenced_group_seed_new_incarnations", + "archive_tails_cover_enrollment_and_controls" + ] + } + ], + "test_cases": [ + { + "name": "group_archive_requires_all_dependencies", + "oracle": "A delayed/corrupt image for B blocks coverage and S3 acknowledgement of a later A entry.", + "invariants": [ + "I06" + ] + }, + { + "name": "quorum_continues_bounded_s3_outage", + "oracle": "Quorum can progress without S3 until explicit retention admission limits; S3 outputs cannot.", + "invariants": [ + "I02", + "I06", + "I12" + ] + }, + { + "name": "group_reclaim_restart_keeps_history", + "oracle": "Actual log files can be reclaimed without losing any acknowledged cell effect/receipt on retained restart.", + "invariants": [ + "I03", + "I07" + ] + }, + { + "name": "fenced_group_seed_new_incarnations", + "oracle": "Complete archived group restores under fresh identities with exact results, old-token rejection and fresh-quorum admission.", + "invariants": [ + "I06", + "I11" + ] + }, + { + "name": "archive_tails_cover_enrollment_and_controls", + "oracle": "Enrollment, no-ops and membership/administrative cuts replay without holes or synthetic cells.", + "invariants": [ + "I04", + "I06" + ] + } + ], + "invariants": [ + "I02", + "I03", + "I04", + "I06", + "I07", + "I11", + "I12" + ], + "evidence": [ + ".cellar/cell-groups-evidence/CG-01-06-gates-e32115aa-02/CG-06-record.json" + ], + "stop_conditions": [ + "A current source excerpt no longer matches: review and record the successor baseline before edits.", + "An acknowledged effect, receipt, proof boundary or resource owner would be weakened to make a test pass.", + "The implementation requires an unrecorded wire/storage compatibility path or bypass of quorum/fencing." + ], + "out_of_scope": [ + "TiKV KV/MVCC APIs and cross-cell transactions", + "Online migration or deletion of existing user disks/archives", + "Feature-off Fleet/bucket durability semantics" + ], + "capability": "archive" + }, + { + "id": "CG-07", + "title": "Cut public SDK routing and managed fleet over to shared groups", + "status": "IN_PROGRESS", + "priority": "P1", + "effort": "L", + "risk": "HIGH", + "depends_on": [ + "CG-03", + "CG-04", + "CG-05", + "CG-06", + "CG-08" + ], + "anchors": [ + "public", + "host", + "fleet", + "placement", + "version" + ], + "scope": [ + "crates/cellar/durability/raft/", + "crates/cellar/application/consensus/", + "crates/cellar/node/", + "crates/cellar/cli/", + "crates/cellar-axum/", + "crates/cellar-admin/", + "examples/", + "crates/cellar/sdk.rs", + "crates/cellar/tests/native_cell_groups_execution.rs", + "crates/cellar/tests/support/", + "crates/cellar/tests/native_cell_groups_host.rs" + ], + "creates": [ + "crates/cellar/tests/native_cell_groups_public.rs" + ], + "test_target": "native_cell_groups_public", + "steps": [ + { + "action": "Switch the existing public host/startup/registration APIs to group placement plus cell enrollment. Preserve cell-addressed application calls and explicit query consistency. Remove production creation of one RawNode for each ordinary cell.", + "tests": [ + "sdk_many_cells_share_group" + ] + }, + { + "action": "Adapt managed namespace import, capacity accounting, repair, drain, maintenance and topology inventory to group replicas plus separate cell/image occupancy. Reuse existing CAS journal, permissions and disruption debt; do not bypass managed admission.", + "tests": [ + "managed_group_lifecycle" + ] + }, + { + "action": "Update native/Axum dispatch, CLI configuration, admin/operator views and examples. Validate tenant/cell authorization independently from group route; give stale hints bounded redirects and typed ambiguous mutation results.", + "tests": [ + "ingress_cannot_cross_cell_authority" + ] + }, + { + "action": "Audit all GroupScope.cell consumers, codecs, receipt/token paths and example configs. Delete superseded single-image/one-cell host assumptions after all consumers use shared groups; occupancy-one must use the exact same code.", + "tests": [ + "occupancy_one_uses_shared_path", + "feature_off_fleet_contract_unchanged" + ] + } + ], + "test_cases": [ + { + "name": "sdk_many_cells_share_group", + "oracle": "Public startup and cell handles serve independent SQLite cells under one configured group on all replicas.", + "invariants": [ + "I01", + "I02", + "I03" + ] + }, + { + "name": "managed_group_lifecycle", + "oracle": "Repair, drain, stop/return/resume and capacity charges move complete groups while preserving cell count and physical debt.", + "invariants": [ + "I08", + "I12" + ] + }, + { + "name": "ingress_cannot_cross_cell_authority", + "oracle": "Native/Axum/admin routes reject forged tenant/cell/group bindings; retries preserve caller command identity.", + "invariants": [ + "I01", + "I03" + ] + }, + { + "name": "occupancy_one_uses_shared_path", + "oracle": "Single-cell and multi-cell configurations exercise one host/manifest/snapshot/archive implementation with no legacy fallback.", + "invariants": [ + "I04", + "I06" + ] + }, + { + "name": "feature_off_fleet_contract_unchanged", + "oracle": "Existing non-Raft feature modes retain their ownership and durability behavior through the full regression suite.", + "invariants": [ + "I14" + ] + } + ], + "invariants": [ + "I01", + "I02", + "I03", + "I04", + "I06", + "I08", + "I12", + "I14" + ], + "evidence": [], + "stop_conditions": [ + "A current source excerpt no longer matches: review and record the successor baseline before edits.", + "An acknowledged effect, receipt, proof boundary or resource owner would be weakened to make a test pass.", + "The implementation requires an unrecorded wire/storage compatibility path or bypass of quorum/fencing." + ], + "out_of_scope": [ + "TiKV KV/MVCC APIs and cross-cell transactions", + "Online migration or deletion of existing user disks/archives", + "Feature-off Fleet/bucket durability semantics" + ], + "capability": "public" + }, + { + "id": "CG-08", + "title": "Preserve per-cell effects and bounded idle execution caches", + "status": "DONE", + "priority": "P1", + "effort": "L", + "risk": "HIGH", + "depends_on": [ + "CG-04", + "CG-05", + "CG-06" + ], + "anchors": [ + "effects", + "sql", + "host", + "manifest" + ], + "scope": [ + "crates/cellar/durability/raft/", + "crates/cellar-ltx/src/", + "crates/cellar/tests/support/", + "tools/multi-raft/" + ], + "creates": [ + "crates/cellar/durability/raft/cell_groups/effect_index.rs", + "crates/cellar/tests/native_cell_groups_effects.rs" + ], + "test_target": "native_cell_groups_effects", + "steps": [ + { + "action": "Move alarm/outbox/inbox identity to CellHistory and retain existing receiver dedupe/client floors. Record effect summaries in the cell manifest and maintain a bounded next-due index owned by node services.", + "tests": [ + "effect_identity_stable_across_group_binding" + ] + }, + { + "action": "Close idle execution/SQLite caches only after unresolved proposals, dispatches and unsafe pins settle. Reopen against the exact committed cell root without resetting ownership or acknowledging based on cached state.", + "tests": [ + "idle_cache_reopens_exact_cell_base" + ] + }, + { + "action": "Make fair effect discovery independent of opening every cell database. Serialize control decisions with cell commands and preserve unknown dispatched outcomes through leadership change and future transfer seals.", + "tests": [ + "due_effect_without_client_request" + ] + }, + { + "action": "Integrate all cache/effect workers with shutdown, resource observation and physical permit lifetimes. Cover due alarms with no client traffic and mixed hot/cold group cells.", + "tests": [ + "pending_command_or_dispatch_pins_cache", + "cache_and_effect_shutdown_joins" + ] + } + ], + "test_cases": [ + { + "name": "effect_identity_stable_across_group_binding", + "oracle": "Changing placement metadata never allocates a new delivery identity or loses receiver duplicate suppression.", + "invariants": [ + "I03", + "I13" + ] + }, + { + "name": "idle_cache_reopens_exact_cell_base", + "oracle": "Eviction saves disposable resources and the next command starts from the verified committed image without an election requirement.", + "invariants": [ + "I04", + "I12" + ] + }, + { + "name": "due_effect_without_client_request", + "oracle": "Durable due summaries schedule alarms/outbox work without scanning or hydrating every cell.", + "invariants": [ + "I12", + "I13" + ] + }, + { + "name": "pending_command_or_dispatch_pins_cache", + "oracle": "Uncertain proposal/delivery cannot be evicted and forgotten after cancellation or leader loss.", + "invariants": [ + "I02", + "I03", + "I13" + ] + }, + { + "name": "cache_and_effect_shutdown_joins", + "oracle": "All retained cache/pin/timer workers finish or preserve durable debt before joined stop reports success.", + "invariants": [ + "I12" + ] + } + ], + "invariants": [ + "I02", + "I03", + "I04", + "I12", + "I13" + ], + "evidence": [ + ".cellar/cell-groups-evidence/CG-08-gates-4ea70e48-01/record.json" + ], + "stop_conditions": [ + "A current source excerpt no longer matches: review and record the successor baseline before edits.", + "An acknowledged effect, receipt, proof boundary or resource owner would be weakened to make a test pass.", + "The implementation requires an unrecorded wire/storage compatibility path or bypass of quorum/fencing." + ], + "out_of_scope": [ + "TiKV KV/MVCC APIs and cross-cell transactions", + "Online migration or deletion of existing user disks/archives", + "Feature-off Fleet/bucket durability semantics" + ], + "capability": "effects" + }, + { + "id": "CG-09", + "title": "Implement irreversible source sealing and destination activation", + "status": "IN_PROGRESS", + "priority": "P1", + "effort": "L", + "risk": "HIGH", + "depends_on": [ + "CG-07", + "CG-08" + ], + "anchors": [ + "fleet", + "membership", + "archive", + "version" + ], + "scope": [ + "crates/cellar/durability/raft/", + "crates/cellar-logic/" + ], + "creates": [ + "crates/cellar/durability/raft/cell_groups/transition.rs", + "crates/cellar/durability/raft/cell_groups/transition_tests.rs", + "crates/cellar/tests/native_cell_groups_transitions.rs" + ], + "test_target": "native_cell_groups_transitions", + "steps": [ + { + "action": "Implement Planned/Prepared/SealAuthorized/Sealed/Installed/Published/Active/Retired journal states from protocols.md. Seal authorization and pre-seal cancellation CAS the same state; no source Seal request is allowed before authorization wins.", + "tests": [ + "seal_authorization_cancel_race" + ] + }, + { + "action": "Commit parent seals after accepted mutations, block new enrollment/writes/strong reads, and pin the exact cell/effect/receipt cuts. After SealAuthorized only roll forward, including unknown replies and partially sealed merges.", + "tests": [ + "source_seal_blocks_delayed_work" + ] + }, + { + "action": "Seed fresh non-serving child group histories with preserved CellHistory/versions/receipts/effect IDs. Require child quorum install receipts and an atomic range-tree publication certificate before committed child activation. Publish and verify a complete remote transfer closure before changing active ranges; pin its digest in the publication certificate so all-voter loss immediately after publication remains recoverable.", + "tests": [ + "child_requires_install_and_publication", + "publication_requires_remote_transfer_closure" + ] + }, + { + "action": "Persist stale-authority tombstones and transition resources across controller/leader/node restart. Release source data only after all child baseline archives and physical pins are safe; no timer may reopen the parent.", + "tests": [ + "transition_restart_retains_single_writer", + "transfer_preserves_tokens_receipts_effects", + "retirement_keeps_required_archive_pins" + ] + } + ], + "test_cases": [ + { + "name": "seal_authorization_cancel_race", + "oracle": "Exactly one root decision wins; cancelled intents cannot seal and authorized intents cannot roll back after unknown responses.", + "invariants": [ + "I09" + ] + }, + { + "name": "source_seal_blocks_delayed_work", + "oracle": "Accepted-before-seal work appears in the cut; delayed/new mutation/enrollment/strong read cannot pass the seal.", + "invariants": [ + "I01", + "I05", + "I09" + ] + }, + { + "name": "child_requires_install_and_publication", + "oracle": "Copied bytes or a catalog hint alone cannot activate a child; quorum seed plus exact publication and activation are required.", + "invariants": [ + "I02", + "I04", + "I09" + ] + }, + { + "name": "transition_restart_retains_single_writer", + "oracle": "Crash at every phase and lost CAS replies preserve a single writable owner and resume the same transition.", + "invariants": [ + "I09", + "I12" + ] + }, + { + "name": "transfer_preserves_tokens_receipts_effects", + "oracle": "Old cell-history versions and duplicate commands remain valid at the child while effect IDs and receiver dedupe survive.", + "invariants": [ + "I03", + "I05", + "I13" + ] + }, + { + "name": "retirement_keeps_required_archive_pins", + "oracle": "Source cleanup cannot orphan any child baseline, archived acknowledgement or unresolved physical owner.", + "invariants": [ + "I06", + "I07" + ] + }, + { + "name": "publication_requires_remote_transfer_closure", + "oracle": "Missing remote transfer dependencies block publication. Erase all local copies immediately after publication in the native fixture; independent archive replay of the pinned closure retains every prior S3-acknowledged cell effect. CG-13/15 qualify fenced live recovery through the public disaster API.", + "invariants": [ + "I06", + "I09", + "I11" + ] + } + ], + "invariants": [ + "I01", + "I02", + "I03", + "I04", + "I05", + "I06", + "I07", + "I09", + "I11", + "I12", + "I13" + ], + "evidence": [], + "stop_conditions": [ + "A current source excerpt no longer matches: review and record the successor baseline before edits.", + "An acknowledged effect, receipt, proof boundary or resource owner would be weakened to make a test pass.", + "The implementation requires an unrecorded wire/storage compatibility path or bypass of quorum/fencing." + ], + "out_of_scope": [ + "TiKV KV/MVCC APIs and cross-cell transactions", + "Online migration or deletion of existing user disks/archives", + "Feature-off Fleet/bucket durability semantics" + ], + "capability": "transitions" + }, + { + "id": "CG-10", + "title": "Split hot groups and merge compatible adjacent ranges", + "status": "IN_PROGRESS", + "priority": "P1", + "effort": "L", + "risk": "HIGH", + "depends_on": [ + "CG-09" + ], + "anchors": [ + "placement", + "fleet", + "host", + "catalog" + ], + "scope": [ + "crates/cellar/durability/raft/", + "crates/cellar-logic/", + "crates/cellar-admin/", + "crates/cellar/cli/" + ], + "creates": [ + "crates/cellar/durability/raft/cell_groups/partition.rs", + "crates/cellar/tests/native_cell_groups_split_merge.rs" + ], + "test_target": "native_cell_groups_split_merge", + "steps": [ + { + "action": "Build split decisions over whole ordered CellKeys with exact equal-union range proofs, including empty edges and hash collisions. Allocate fresh child histories; parent group retirement uses CG-09 instead of in-place identity reuse.", + "tests": [ + "split_equal_union_whole_cells" + ] + }, + { + "action": "Build merge decisions for adjacent compatible namespace/schema/durability/failure-domain ranges. Seal all parents, install one fresh child and publish one atomic root replacement; partial parent failure cannot admit the child.", + "tests": [ + "merge_requires_adjacent_compatible_ranges" + ] + }, + { + "action": "Pre-copy immutable state and apply final sealed deltas with bounded parallelism and resource reservations. Define explicit operator split/merge/status/resume APIs; waiting remains read-only and ambiguous requests retain their IDs.", + "tests": [ + "partial_merge_failure_no_double_writer" + ] + }, + { + "action": "Exercise repeated split/merge under foreground writes, stale client routes, controller loss and skew. A dedicated hot cell is a range with one cell using the common runtime.", + "tests": [ + "split_merge_under_writes", + "transition_precopy_resource_bounds" + ] + } + ], + "test_cases": [ + { + "name": "split_equal_union_whole_cells", + "oracle": "Children partition the parent exactly, never split a SQLite database, and tolerate hash ties without aliasing.", + "invariants": [ + "I01", + "I09" + ] + }, + { + "name": "merge_requires_adjacent_compatible_ranges", + "oracle": "Non-adjacent or policy/schema-incompatible sources refuse before sealing or allocating unsafe authority.", + "invariants": [ + "I01", + "I09" + ] + }, + { + "name": "partial_merge_failure_no_double_writer", + "oracle": "A sealed source stays fenced while another lacks quorum; no child admits prematurely.", + "invariants": [ + "I02", + "I09" + ] + }, + { + "name": "split_merge_under_writes", + "oracle": "Every acknowledged effect and receipt reconciles exactly through repeated transitions and cached stale routes.", + "invariants": [ + "I02", + "I03", + "I05", + "I09" + ] + }, + { + "name": "transition_precopy_resource_bounds", + "oracle": "Bounded precopy/delta streaming cannot starve live groups or release transfer debt before completion.", + "invariants": [ + "I07", + "I12" + ] + } + ], + "invariants": [ + "I01", + "I02", + "I03", + "I05", + "I07", + "I09", + "I12" + ], + "evidence": [], + "stop_conditions": [ + "A current source excerpt no longer matches: review and record the successor baseline before edits.", + "An acknowledged effect, receipt, proof boundary or resource owner would be weakened to make a test pass.", + "The implementation requires an unrecorded wire/storage compatibility path or bypass of quorum/fencing." + ], + "out_of_scope": [ + "TiKV KV/MVCC APIs and cross-cell transactions", + "Online migration or deletion of existing user disks/archives", + "Feature-off Fleet/bucket durability semantics" + ], + "capability": "split_merge" + }, + { + "id": "CG-11", + "title": "Schedule group placement and adaptive partition size", + "status": "IN_PROGRESS", + "priority": "P1", + "effort": "L", + "risk": "HIGH", + "depends_on": [ + "CG-07", + "CG-10" + ], + "anchors": [ + "placement", + "fleet", + "membership", + "public" + ], + "scope": [ + "crates/cellar-logic/", + "crates/cellar/durability/raft/", + "crates/cellar-admin/", + "crates/cellar/cli/" + ], + "creates": [ + "crates/cellar-logic/cell_group_placement.rs", + "crates/cellar-logic/cell_group_placement_tests.rs", + "crates/cellar/tests/native_cell_groups_scheduler.rs" + ], + "test_target": "native_cell_groups_scheduler", + "steps": [ + { + "action": "Define dependency-free scheduler observations/actions using group bytes, cell count, write/apply rate, queue age, archive lag and failure-domain load. Keep observations advisory; actual owners revalidate membership, route and resource permits.", + "tests": [ + "scheduler_hysteresis_and_hot_isolation" + ] + }, + { + "action": "Add bounded split/merge hysteresis and cooldown with deterministic reason codes. Isolate an oversized/hot cell without claiming intra-cell parallelism; refuse unsafe merge and prevent repeated split/merge oscillation.", + "tests": [ + "scheduler_cannot_grant_authority" + ] + }, + { + "action": "Integrate scale-out/in, leader balance, learner movement, maintenance and repair with whole-group inventory. Serialize membership with transition/park/disaster ownership; reserve recovery capacity and keep exact physical charges.", + "tests": [ + "whole_group_scale_out_in" + ] + }, + { + "action": "Expose bounded metrics/status and qualify node loss under mixed load. Avoid broadcasting all cell/group metadata to every node and record topology-journal contention.", + "tests": [ + "repair_preserves_failure_domains_and_reserve", + "control_plane_fanout_bounded" + ] + } + ], + "test_cases": [ + { + "name": "scheduler_hysteresis_and_hot_isolation", + "oracle": "Stable decisions split hot/large ranges, merge eligible small neighbors and do not oscillate at thresholds.", + "invariants": [ + "I12" + ] + }, + { + "name": "scheduler_cannot_grant_authority", + "oracle": "Stale observed plans cannot bypass current route, membership, permit or quorum checks.", + "invariants": [ + "I01", + "I08", + "I09" + ] + }, + { + "name": "whole_group_scale_out_in", + "oracle": "Adding/removing nodes transfers group replicas without remapping or reinitializing individual cells.", + "invariants": [ + "I03", + "I08" + ] + }, + { + "name": "repair_preserves_failure_domains_and_reserve", + "oracle": "One-domain loss repairs groups within bounded slots; a second loss cannot fabricate quorum.", + "invariants": [ + "I02", + "I08", + "I12" + ] + }, + { + "name": "control_plane_fanout_bounded", + "oracle": "Paged inventory and targeted updates remain bounded under many enrolled cells and groups.", + "invariants": [ + "I12" + ] + } + ], + "invariants": [ + "I01", + "I02", + "I03", + "I08", + "I09", + "I12" + ], + "evidence": [], + "stop_conditions": [ + "A current source excerpt no longer matches: review and record the successor baseline before edits.", + "An acknowledged effect, receipt, proof boundary or resource owner would be weakened to make a test pass.", + "The implementation requires an unrecorded wire/storage compatibility path or bypass of quorum/fencing." + ], + "out_of_scope": [ + "TiKV KV/MVCC APIs and cross-cell transactions", + "Online migration or deletion of existing user disks/archives", + "Feature-off Fleet/bucket durability semantics" + ], + "capability": "scheduler" + }, + { + "id": "CG-12", + "title": "Coordinate group hibernation and authority-checked wakeup", + "status": "IN_PROGRESS", + "priority": "P1", + "effort": "L", + "risk": "HIGH", + "depends_on": [ + "CG-08", + "CG-10", + "CG-11" + ], + "anchors": [ + "ticks", + "runtime", + "effects", + "membership" + ], + "scope": [ + "crates/cellar-logic/", + "crates/cellar/durability/raft/" + ], + "creates": [ + "crates/cellar/durability/raft/cell_groups/hibernate.rs", + "crates/cellar-logic/cell_group_hibernation.rs", + "crates/cellar/tests/native_cell_groups_hibernate.rs" + ], + "test_target": "native_cell_groups_hibernate", + "steps": [ + { + "action": "Implement PreparePark and generation-bound local durable park/closed records. Require all configured voters to acknowledge the committed applied cut, with explicit learner state and no unresolved proposal/read/topology work.", + "tests": [ + "park_requires_all_voters_and_no_pending_work" + ] + }, + { + "action": "Pause routine group ticks only after the exact park certificate is durable. Keep node transport/control/effect wake owners active and bounded; partial park delivery must be safe and stale certificates must never re-park a woken group.", + "tests": [ + "partial_park_and_stale_certificate" + ] + }, + { + "action": "Wake on API/peer/effect/topology activity or restart, durably close the park generation, resume retained Raft and obtain fresh current-term commit/read authority. Never reset votes, assume retained leadership or replace a missing quorum with S3.", + "tests": [ + "wake_requires_fresh_quorum_barrier" + ] + }, + { + "action": "Integrate alarm due indexes, membership serialization, cancellation and shutdown. Measure healthy cached wake, image-cache restore and leader-loss wake separately with enough samples.", + "tests": [ + "alarm_and_peer_wake_no_client", + "parked_restart_and_leader_loss" + ] + } + ], + "test_cases": [ + { + "name": "park_requires_all_voters_and_no_pending_work", + "oracle": "A missing voter or unresolved command/effect/topology operation prevents parking but not ordinary safe quorum operation.", + "invariants": [ + "I02", + "I10", + "I13" + ] + }, + { + "name": "partial_park_and_stale_certificate", + "oracle": "Lost/late messages, generation replay and new commands cannot cause unsafe re-parking or lost work.", + "invariants": [ + "I01", + "I10" + ] + }, + { + "name": "wake_requires_fresh_quorum_barrier", + "oracle": "Retained leader and minority wake cannot serve strong reads/writes without current quorum proof.", + "invariants": [ + "I02", + "I05", + "I10" + ] + }, + { + "name": "alarm_and_peer_wake_no_client", + "oracle": "Due effects and peer traffic wake parked groups without client traffic or per-cell threads.", + "invariants": [ + "I10", + "I12", + "I13" + ] + }, + { + "name": "parked_restart_and_leader_loss", + "oracle": "All-voter retained restart and parked leader failure preserve term/vote/history and eventually serve through ordinary elections.", + "invariants": [ + "I04", + "I08", + "I10" + ] + } + ], + "invariants": [ + "I01", + "I02", + "I04", + "I05", + "I08", + "I10", + "I12", + "I13" + ], + "evidence": [], + "stop_conditions": [ + "A current source excerpt no longer matches: review and record the successor baseline before edits.", + "An acknowledged effect, receipt, proof boundary or resource owner would be weakened to make a test pass.", + "The implementation requires an unrecorded wire/storage compatibility path or bypass of quorum/fencing." + ], + "out_of_scope": [ + "TiKV KV/MVCC APIs and cross-cell transactions", + "Online migration or deletion of existing user disks/archives", + "Feature-off Fleet/bucket durability semantics" + ], + "capability": "hibernate" + }, + { + "id": "CG-13", + "title": "Recover managed groups and incomplete transitions from S3", + "status": "IN_PROGRESS", + "priority": "P1", + "effort": "L", + "risk": "HIGH", + "depends_on": [ + "CG-06", + "CG-09", + "CG-10", + "CG-11", + "CG-14" + ], + "anchors": [ + "disaster", + "fleet", + "recovery", + "archive" + ], + "scope": [ + "crates/cellar/durability/raft/", + "tools/cellar-recovery-tests/", + "crates/cellar/tests/support/raft_process_recovery/" + ], + "creates": [ + "crates/cellar/durability/raft/cell_groups/disaster.rs", + "crates/cellar/tests/native_cell_groups_disaster.rs" + ], + "test_target": "native_cell_groups_disaster", + "steps": [ + { + "action": "Extend recovery admission to the exact managed topology journal and transition component. Resolve publication outcome and enumerate all parent/child/replacement voting lineages that could still activate; preserve fleet debt/import exclusion.", + "tests": [ + "managed_disaster_blocks_old_lifecycle" + ] + }, + { + "action": "Verify complete selected recovery closures before fencing. For unpublished operations recover parent domain; for published operations require child sets or transfer closure plus child tails. A parent checkpoint alone cannot cover child S3 acknowledgements.", + "tests": [ + "transition_disaster_selects_complete_closure" + ] + }, + { + "action": "Use the explicit fencing-authority API to permanently revoke old launch authority and confirm isolation before publishing any new range. Create fresh cell/group/node/member/disk/process identities, seed receipts without callbacks, and require a fresh quorum.", + "tests": [ + "fence_all_possible_old_writers" + ] + }, + { + "action": "Test exact-request retry after unknown fencing/catalog results, old-host restart attempts, outbox ambiguity and retained-disk restart. Refuse partial fences, missing dependencies, unmanaged bypasses and unsupported rollback.", + "tests": [ + "all_disks_deleted_restore_all_cells", + "disaster_retry_and_old_token_refusal", + "disaster_outbox_ambiguity_refuses" + ] + } + ], + "test_cases": [ + { + "name": "managed_disaster_blocks_old_lifecycle", + "oracle": "Import, maintenance and failure recovery cannot concurrently activate old or replacement group histories.", + "invariants": [ + "I08", + "I09", + "I11" + ] + }, + { + "name": "transition_disaster_selects_complete_closure", + "oracle": "Published children require their acknowledged tails; unpublished parents recover an exact complete domain without duplication or gaps.", + "invariants": [ + "I06", + "I09", + "I11" + ] + }, + { + "name": "fence_all_possible_old_writers", + "oracle": "Partial inventory, incomplete launch revocation or a restartable old voter prevents new incarnation publication.", + "invariants": [ + "I11" + ] + }, + { + "name": "all_disks_deleted_restore_all_cells", + "oracle": "Real RustFS restore after erasing every owned voter disk preserves all selected-cut values/receipts and admits fresh writes only after quorum.", + "invariants": [ + "I02", + "I03", + "I06", + "I11" + ] + }, + { + "name": "disaster_retry_and_old_token_refusal", + "oracle": "Lost replies resume the same cut/identities; old tokens refuse and duplicate results return without callback replay.", + "invariants": [ + "I01", + "I03", + "I11" + ] + }, + { + "name": "disaster_outbox_ambiguity_refuses", + "oracle": "Old-origin uncertain external effects cannot be silently reissued under a fresh identity.", + "invariants": [ + "I11", + "I13" + ] + } + ], + "invariants": [ + "I01", + "I02", + "I03", + "I06", + "I08", + "I09", + "I11", + "I13" + ], + "evidence": [], + "stop_conditions": [ + "A current source excerpt no longer matches: review and record the successor baseline before edits.", + "An acknowledged effect, receipt, proof boundary or resource owner would be weakened to make a test pass.", + "The implementation requires an unrecorded wire/storage compatibility path or bypass of quorum/fencing." + ], + "out_of_scope": [ + "TiKV KV/MVCC APIs and cross-cell transactions", + "Online migration or deletion of existing user disks/archives", + "Feature-off Fleet/bucket durability semantics" + ], + "capability": "disaster" + }, + { + "id": "CG-14", + "title": "Build reproducible shared-group workload and RustFS harnesses", + "status": "IN_PROGRESS", + "priority": "P1", + "effort": "M", + "risk": "HIGH", + "depends_on": [ + "CG-07", + "CG-08" + ], + "anchors": [ + "harness", + "recovery", + "fixture", + "ci" + ], + "scope": [ + "tools/perf/", + "tools/cellar-recovery-tests/", + "crates/cellar/tests/support/" + ], + "creates": [ + "tools/perf/cell_groups.py", + "tools/perf/test_cell_groups.py", + "crates/cellar/tests/native_cell_groups_harness.rs" + ], + "test_target": "native_cell_groups_harness", + "steps": [ + { + "action": "Extend existing native workload and process-recovery fixtures with logical-cell/group population, mixed operations and per-cell independent ledgers. Observe actual RawNode/group/cell counts; do not substitute counter rows in one database.", + "tests": [ + "harness_counts_real_cells_and_groups" + ] + }, + { + "action": "Implement the planned harness CLI in qualification.md with explicit private fixture config, owned run directories, source/binary/image/profile hashes and resource preflight. Never change global Docker context or touch unowned resources.", + "tests": [ + "harness_owned_resources_only" + ] + }, + { + "action": "Separate build/setup/warmup/measurement and run equal-cell cohorts serially. Record all offered/accepted/refused/ambiguous operations, latencies, control bytes, CPU/RSS/FD, image/log writes, S3 requests/lag and cleanup/join outcomes.", + "tests": [ + "harness_ledger_preserves_all_outcomes" + ] + }, + { + "action": "Implement strict result validation for profiles.json and replayable fault schedules. Retain failed attempts; refuse unsupported profiles, zero selected tests, missing artifacts, source drift and fake multi-node topology claims.", + "tests": [ + "harness_rejects_missing_or_drifted_evidence", + "harness_profile_gates_measured" + ] + } + ], + "test_cases": [ + { + "name": "harness_counts_real_cells_and_groups", + "oracle": "Metrics distinguish independent SQLite cells, RawNodes, groups and replicas and match physical inventory.", + "invariants": [ + "I14" + ] + }, + { + "name": "harness_owned_resources_only", + "oracle": "Destructive actions refuse wrong context, label, directory, container identity or run UUID.", + "invariants": [ + "I11", + "I12", + "I14" + ] + }, + { + "name": "harness_ledger_preserves_all_outcomes", + "oracle": "Refusals/timeouts/cancellations remain in offered-load and correctness accounting; no silent retry improves benchmark results.", + "invariants": [ + "I02", + "I03", + "I14" + ] + }, + { + "name": "harness_rejects_missing_or_drifted_evidence", + "oracle": "Changed sources/binaries/profiles, zero-test runs and absent logs cannot produce complete reports.", + "invariants": [ + "I14" + ] + }, + { + "name": "harness_profile_gates_measured", + "oracle": "Numeric metrics and source-bound artifacts, not a passed boolean alone, determine benchmark acceptance.", + "invariants": [ + "I12", + "I14" + ] + } + ], + "invariants": [ + "I02", + "I03", + "I11", + "I12", + "I14" + ], + "evidence": [], + "stop_conditions": [ + "A current source excerpt no longer matches: review and record the successor baseline before edits.", + "An acknowledged effect, receipt, proof boundary or resource owner would be weakened to make a test pass.", + "The implementation requires an unrecorded wire/storage compatibility path or bypass of quorum/fencing." + ], + "out_of_scope": [ + "TiKV KV/MVCC APIs and cross-cell transactions", + "Online migration or deletion of existing user disks/archives", + "Feature-off Fleet/bucket durability semantics" + ], + "capability": "harness" + }, + { + "id": "CG-15", + "title": "Qualify crashes, split merge, wake and zero-quorum recovery", + "status": "IN_PROGRESS", + "priority": "P1", + "effort": "L", + "risk": "HIGH", + "depends_on": [ + "CG-10", + "CG-11", + "CG-12", + "CG-13", + "CG-14" + ], + "anchors": [ + "fixture", + "recovery", + "harness", + "runtime" + ], + "scope": [ + "crates/cellar/durability/raft/", + "crates/cellar/tests/", + "tools/perf/", + "tools/cellar-recovery-tests/" + ], + "creates": [ + "crates/cellar/tests/native_cell_groups_faults.rs" + ], + "test_target": "native_cell_groups_faults", + "steps": [ + { + "action": "Run the full native history oracle over interleaved cells, both image modes and all new durable cuts. Include deterministic exhaustive transition/park event schedules and process SIGKILL at persistence boundaries; finite tests complement rather than prove the protocol.", + "tests": [ + "history_oracle_rejects_cross_cell_loss" + ] + }, + { + "action": "Run real RustFS campaigns with physical log reclamation, S3 outage, slow apply/minority, leader/controller restart and every split/merge cut. Verify per-cell values, versions, receipts, effect IDs and no two writable owners.", + "tests": [ + "all_new_durable_crash_cuts" + ] + }, + { + "action": "Delete all owned voter disks in standalone and in-transition scenarios after fencing; verify new incarnations, old-host restart rejection, every selected-cut cell and retained restart. Record quorum-only suffix loss separately from S3-protected effects.", + "tests": [ + "rustfs_reclaim_and_transition_recovery" + ] + }, + { + "action": "Run candidate density, mixed/skew, transition, hibernation, independent-domain and 24-hour profiles. Compare identical logical-cell populations and retain misses. Do not declare a physical tier from containers on one host.", + "tests": [ + "rustfs_zero_quorum_transition_matrix", + "load_and_wake_profiles_complete", + "soak_and_domain_loss_history" + ] + } + ], + "test_cases": [ + { + "name": "history_oracle_rejects_cross_cell_loss", + "oracle": "Mutated histories with a lost acknowledged effect, borrowed version or duplicate receipt execution are rejected by the oracle.", + "invariants": [ + "I02", + "I03", + "I05", + "I14" + ] + }, + { + "name": "all_new_durable_crash_cuts", + "oracle": "Native process kills cover manifest, enrollment, seed, publication, seal, activation, park and wake boundaries without partial admission.", + "invariants": [ + "I04", + "I09", + "I10" + ] + }, + { + "name": "rustfs_reclaim_and_transition_recovery", + "oracle": "Real object-store closure plus actual log-file reclamation survives group restart and transition interruption.", + "invariants": [ + "I06", + "I07", + "I09" + ] + }, + { + "name": "rustfs_zero_quorum_transition_matrix", + "oracle": "Every specified parent/child disaster case fences old histories and restores complete selected state under fresh quorum.", + "invariants": [ + "I06", + "I09", + "I11" + ] + }, + { + "name": "load_and_wake_profiles_complete", + "oracle": "All required profile metrics, errors, samples and topology attestations meet the frozen candidate gates.", + "invariants": [ + "I10", + "I12", + "I14" + ] + }, + { + "name": "soak_and_domain_loss_history", + "oracle": "24-hour and independent-domain runs preserve history, bounded resources and joined lifecycle under foreground load.", + "invariants": [ + "I08", + "I12", + "I13", + "I14" + ] + } + ], + "invariants": [ + "I02", + "I03", + "I04", + "I05", + "I06", + "I07", + "I08", + "I09", + "I10", + "I11", + "I12", + "I13", + "I14" + ], + "evidence": [], + "stop_conditions": [ + "A current source excerpt no longer matches: review and record the successor baseline before edits.", + "An acknowledged effect, receipt, proof boundary or resource owner would be weakened to make a test pass.", + "The implementation requires an unrecorded wire/storage compatibility path or bypass of quorum/fencing." + ], + "out_of_scope": [ + "TiKV KV/MVCC APIs and cross-cell transactions", + "Online migration or deletion of existing user disks/archives", + "Feature-off Fleet/bucket durability semantics" + ], + "capability": "faults" + }, + { + "id": "CG-16", + "title": "Publish the unified group runtime and qualified release envelope", + "status": "TODO", + "priority": "P1", + "effort": "M", + "risk": "HIGH", + "depends_on": [ + "CG-15" + ], + "anchors": [ + "ci", + "host", + "catalog", + "version", + "fixture" + ], + "scope": [ + "crates/", + "examples/", + "docs/", + "tools/", + ".github/workflows/" + ], + "creates": [ + "crates/cellar/tests/native_cell_groups_release.rs" + ], + "test_target": "native_cell_groups_release", + "steps": [ + { + "action": "Audit every production registration, scope codec, image marker, archive, SDK and operator path for the old cell-equals-group assumption. Remove temporary staging adapters and legacy runtime choices; leave explicit one-cell occupancy on the common code. Remove QueryVersion::GroupCut and RaftClass::invoke_group_query along with the old query adapters; final QueryContext exposes the selected cell observation directly. Remove TransportRuntime::Legacy, the typed cell-history transport frame arm and old transport entry points; the final shared transport must not retain a legacy serving mode.", + "tests": [ + "no_legacy_per_cell_consensus_path" + ] + }, + { + "action": "Update public docs, examples, CLI/admin error messages and operator runbooks for group placement, split/merge, park/wake and fenced S3 recovery. Document fresh-root cutover and rollback limits without deleting existing state.", + "tests": [ + "public_group_operations_end_to_end" + ] + }, + { + "action": "Run the complete workspace/feature-off/native/Axum regression matrix and all new task tests at the same immutable release source. Repeat deployment gates at that source; evidence from different revisions cannot be added into one release pass.", + "tests": [ + "release_all_modes_and_shutdown" + ] + }, + { + "action": "Publish source/artifact/profile hashes, exact supported topology/load envelope and every limitation. plan.py release must refuse until all tasks, invariant cases and mandatory profile reports pass with retained evidence.", + "tests": [ + "release_evidence_single_source" + ] + } + ], + "test_cases": [ + { + "name": "no_legacy_per_cell_consensus_path", + "oracle": "Public configuration cannot select a legacy one-cell host; physical group counts follow grouping, including occupancy one. No temporary GroupCut query observation or invoke_group_query constructor remains; registered queries expose the actual selected cell version and witness.", + "invariants": [ + "I01", + "I14" + ] + }, + { + "name": "public_group_operations_end_to_end", + "oracle": "SDK/native/Axum/admin/CLI share authorization and identity for enrollment, strong reads, split/merge, park/wake and status.", + "invariants": [ + "I01", + "I03", + "I05", + "I09", + "I10" + ] + }, + { + "name": "release_all_modes_and_shutdown", + "oracle": "Materialized/extents, managed/unmanaged, feature-off applications and owned shutdown pass the complete repository regression suite.", + "invariants": [ + "I04", + "I08", + "I12", + "I13", + "I14" + ] + }, + { + "name": "release_evidence_single_source", + "oracle": "All capabilities and numerical profiles are verified at the published source with no missing, stale or synthetic evidence.", + "invariants": [ + "I14" + ] + } + ], + "invariants": [ + "I01", + "I03", + "I04", + "I05", + "I08", + "I09", + "I10", + "I12", + "I13", + "I14" + ], + "evidence": [], + "stop_conditions": [ + "A current source excerpt no longer matches: review and record the successor baseline before edits.", + "An acknowledged effect, receipt, proof boundary or resource owner would be weakened to make a test pass.", + "The implementation requires an unrecorded wire/storage compatibility path or bypass of quorum/fencing." + ], + "out_of_scope": [ + "TiKV KV/MVCC APIs and cross-cell transactions", + "Online migration or deletion of existing user disks/archives", + "Feature-off Fleet/bucket durability semantics" + ], + "capability": "release" + } + ] +} diff --git a/docs/design/multi-raft/cell-groups/test_plan.py b/docs/design/multi-raft/cell-groups/test_plan.py new file mode 100644 index 000000000..897aad312 --- /dev/null +++ b/docs/design/multi-raft/cell-groups/test_plan.py @@ -0,0 +1,392 @@ +# Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +"""Negative tests for the delivery gate, not tests of the proposed Raft runtime.""" + +import copy +import hashlib +import importlib.util +import json +from pathlib import Path +import subprocess +import tempfile +import unittest +from unittest.mock import patch + +SPEC = importlib.util.spec_from_file_location("cell_group_plan", Path(__file__).with_name("plan.py")) +plan = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(plan) + + +class PlanTests(unittest.TestCase): + def setUp(self): + self.manifest = plan.read_json(plan.PACKAGE / "tasks.json") + self.actual_manifest = copy.deepcopy(self.manifest) + for task in self.manifest["tasks"]: + task.update(status="TODO", evidence=[]) + self.baseline = plan.read_json(plan.PACKAGE / "baseline.json") + self.profiles = plan.read_json(plan.PACKAGE / "profiles.json") + self.temp = tempfile.TemporaryDirectory(prefix="cell-group-plan-test-") + self.addCleanup(self.temp.cleanup) + self.root = Path(self.temp.name) + self.revision = "a" * 40 + self.executable = "b" * 64 + self.profile_digest = "c" * 64 + + def contract(self): + return plan.validate_contract(self.manifest, self.baseline, self.profiles, plan.ROOT) + + def put(self, name, value): + path = self.root / name + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(value if isinstance(value, str) else json.dumps(value)) + return {"path": name, "sha256": plan.digest(path)} + + def evidence(self): + task = self.manifest["tasks"][0] + paths = {self.baseline["anchors"][a]["path"] for a in task["anchors"]} + paths.add(f'crates/cellar/tests/{task["test_target"]}.rs') + record = {"format": 1, "task_id": task["id"], "source_revision": self.revision, + "executable_sha256": self.executable, "environment": "Synthetic validator fixture only", + "limitations": "Not runtime evidence", "source_inputs": [], "commands": [], "profiles": []} + for path in sorted(paths): + record["source_inputs"].append({"path": path, "sha256": hashlib.sha256(path.encode()).hexdigest()}) + for index, command in enumerate(plan.commands_for(task, self.manifest)): + output = "Command exited successfully.\n" + if command[-1] == "--list": + output = "\n".join(c["name"] + ": test" for c in task["test_cases"]) + "\n" + elif "--exact" in command: + name = command[command.index("--exact") - 1] + output = f"running 1 test\ntest {name} ... ok\ntest result: ok. 1 passed; 0 failed; 0 ignored; 3 filtered out; finished in 0.01s\n" + item = self.put(f"commands/{index}.log", output) + record["commands"].append({"argv": command, "exit_code": 0, "output": item["path"], "sha256": item["sha256"]}) + return record + + def verify_evidence(self, record, changed=b""): + with patch.object(plan, "git_bytes", side_effect=lambda root, rev, path: path.encode()), \ + patch.object(plan.subprocess, "check_output", return_value=changed): + return plan.check_evidence(self.manifest["tasks"][0], record, self.manifest, self.baseline, + self.profiles, self.profile_digest, self.root) + + def profile_fixture(self, pid="mixed_quorum"): + profile = next(p for p in self.profiles["profiles"] if p["id"] == pid) + metrics = {name: next(iter(bounds.values())) for name, bounds in profile["checks"].items()} + for name in profile["ratios"]: + metrics[name] = 1.0 + ledger = self.put("ledger.jsonl", '{"validator_fixture_only": true}\n') + rows = [] + for variant, occupancies in [("replacement", profile["occupancies"]), ("historical", [1] if profile["historical_control"] else [])]: + for occupancy in occupancies: + for mode in profile["image_modes"]: + for rep in range(1, profile["repetitions"] + 1): + row = {"variant": variant, "occupancy": occupancy, "image_mode": mode, "repetition": rep, + "source_revision": self.revision if variant == "replacement" else profile["historical_revision"], + "executable_sha256": self.executable, "ledger_sha256": ledger["sha256"], + "measured_seconds": profile["duration_seconds"], "metrics": copy.deepcopy(metrics)} + row.update({k: profile[k] for k in ("cells", "nodes", "voters", "offered_requests_per_second")}) + for ratio in profile["ratios"].values(): + row["metrics"].setdefault(ratio["numerator"], 100) + row["metrics"].setdefault(ratio["denominator"], 100) + rows.append(row) + measured = {"metrics": metrics, "cohorts": rows} + measurements = self.put("measurements.json", measured) + topology = {"kind": profile["topology"], "nodes": [ + {"id": str(n), "physical_host": f"host{n}", "failure_domain": f"domain{n % 3}"} + for n in range(profile["nodes"])]} + verification = {"source_revision": self.revision, "complete": True, "ledger_sha256": ledger["sha256"], + "measurements_sha256": measurements["sha256"]} + report = {"format": 1, "profile_id": pid, "source_revision": self.revision, + "profiles_sha256": self.profile_digest, "executable_sha256": self.executable, + "environment": "Synthetic validator fixture only", "topology": profile["topology"], + "metrics": metrics, "artifacts": {"ledger": ledger, "measurements": measurements, + "topology": self.put("topology.json", topology), + "fault_schedule": self.put("faults.json", {"fixture": True}), + "verification": self.put("verification.json", verification)}} + return profile, report + + def edit_artifact(self, report, role, change): + item = report["artifacts"][role] + content = plan.read_json(self.root / item["path"]) + change(content) + report["artifacts"][role] = self.put(item["path"], content) + if role == "measurements": + self.edit_artifact(report, "verification", lambda obj: obj.update(measurements_sha256=report["artifacts"][role]["sha256"])) + + def verify_profile(self, profile, report): + plan.check_profile(report, profile, self.profile_digest, self.revision, self.root, self.executable) + + def test_current_contract_and_index(self): + stats = self.contract() + self.assertEqual(stats["tasks"], 16) + self.assertGreaterEqual(stats["required_cases"], 83) + self.assertEqual(stats["invariants"], 14) + self.assertEqual((plan.PACKAGE / "implementation-plan.md").read_text(), plan.index_text(self.actual_manifest)) + + def test_evidence_coverage_keeps_frozen_base_after_baseline_advances(self): + record = self.evidence() + self.baseline["source_revision"] = "f" * 40 + with patch.object(plan, "git_bytes", side_effect=lambda root, rev, path: path.encode()), \ + patch.object(plan.subprocess, "check_output", return_value=b"") as diff: + plan.check_evidence(self.manifest["tasks"][0], record, self.manifest, + self.baseline, self.profiles, self.profile_digest, self.root) + argv = diff.call_args.args[0] + self.assertEqual(argv[5:7], [self.manifest["planned_at"], self.revision]) + # Earlier implementation inputs stay required even when a later review + # baseline already includes them. Advancing the baseline cannot waive them. + with self.assertRaisesRegex(plan.PlanError, "omits changed runtime inputs"): + self.verify_evidence(record, b"crates/cellar/earlier-input.rs\0") + + def test_dependency_order_covers_effects_and_harness(self): + order = plan.ordered(self.manifest["tasks"]) + self.assertLess(order.index("CG-08"), order.index("CG-07")) + self.assertLess(order.index("CG-14"), order.index("CG-13")) + + def test_cycle_refuses(self): + self.manifest["tasks"][0]["depends_on"] = ["CG-16"] + with self.assertRaisesRegex(plan.PlanError, "cycle"): + self.contract() + + def test_unknown_dependency_refuses(self): + self.manifest["tasks"][0]["depends_on"] = ["CG-99"] + with self.assertRaisesRegex(plan.PlanError, "Unknown dependency"): + self.contract() + + def test_duplicate_task_refuses(self): + self.manifest["tasks"].append(copy.deepcopy(self.manifest["tasks"][0])) + with self.assertRaises(plan.PlanError): + self.contract() + + def test_done_without_evidence_refuses(self): + self.manifest["tasks"][0]["status"] = "DONE" + with self.assertRaisesRegex(plan.PlanError, "without evidence"): + self.contract() + + def test_done_before_dependency_refuses(self): + self.manifest["tasks"][1].update(status="DONE", evidence=["fake.json"]) + with self.assertRaisesRegex(plan.PlanError, "before dependency"): + self.contract() + + def test_missing_case_oracle_refuses(self): + self.manifest["tasks"][0]["test_cases"][0]["oracle"] = "" + with self.assertRaisesRegex(plan.PlanError, "empty oracle"): + self.contract() + + def test_uncovered_step_test_refuses(self): + self.manifest["tasks"][0]["steps"][0]["tests"] = ["imaginary_test"] + with self.assertRaisesRegex(plan.PlanError, "unknown step test"): + self.contract() + + def test_profile_removal_or_disable_refuses(self): + saved = copy.deepcopy(self.profiles) + self.profiles["profiles"].pop() + with self.assertRaisesRegex(plan.PlanError, "profile inventory"): + self.contract() + self.profiles = saved + self.profiles["profiles"][0]["required"] = False + with self.assertRaisesRegex(plan.PlanError, "disabled"): + self.contract() + + def test_path_escape_and_symlink_refuse(self): + with self.assertRaises(plan.PlanError): + plan.within(self.root, "../outside") + with tempfile.TemporaryDirectory() as outside: + (self.root / "link").symlink_to(outside, target_is_directory=True) + with self.assertRaises(plan.PlanError): + plan.within(self.root, "link/secret") + + def test_duplicate_json_and_nan_refuse(self): + for text in ('{"format":1,"format":2}', '{"metric":NaN}'): + self.put("bad.json", text) + with self.assertRaises(plan.PlanError): + plan.read_json(self.root / "bad.json") + + def test_actual_reviewed_source_preflight(self): + plan.preflight(self.baseline, plan.ROOT) + + def test_incorrect_excerpt_line_refuses(self): + self.baseline["anchors"]["codec"]["line"] += 1 + with self.assertRaisesRegex(plan.PlanError, "Excerpt line differs"): + plan.preflight(self.baseline, plan.ROOT) + + def test_source_drift_refuses(self): + anchor = copy.deepcopy(self.baseline["anchors"]["identity"]) + content = (plan.ROOT / anchor["path"]).read_bytes() + path = self.root / anchor["path"] + path.parent.mkdir(parents=True) + path.write_bytes(content + b"\n// changed\n") + baseline = {"source_revision": self.revision, "anchors": {"identity": anchor}} + with patch.object(plan, "git_bytes", return_value=content), self.assertRaisesRegex(plan.PlanError, "Source drift"): + plan.preflight(baseline, self.root) + + def test_evidence_accepts_complete_exact_test_records(self): + self.assertEqual(self.verify_evidence(self.evidence()), self.revision) + + def test_missing_command_evidence_refuses(self): + record = self.evidence() + record["commands"].pop() + with self.assertRaisesRegex(plan.PlanError, "Missing required command"): + self.verify_evidence(record) + + def test_zero_selected_or_ignored_test_refuses(self): + for text in ("test result: ok. 0 passed; 0 failed; 0 ignored;", "test result: ok. 0 passed; 0 failed; 1 ignored;"): + record = self.evidence() + run = next(x for x in record["commands"] if "--exact" in x["argv"]) + item = self.put(run["output"], text) + run["sha256"] = item["sha256"] + with self.assertRaisesRegex(plan.PlanError, "Case did not pass"): + self.verify_evidence(record) + + def test_false_inventory_refuses(self): + record = self.evidence() + run = next(x for x in record["commands"] if x["argv"][-1] == "--list") + item = self.put(run["output"], "some_other_test: test\n") + run["sha256"] = item["sha256"] + with self.assertRaisesRegex(plan.PlanError, "absent from inventory"): + self.verify_evidence(record) + + def test_tampered_log_or_failed_exit_refuses(self): + record = self.evidence() + (self.root / record["commands"][0]["output"]).write_text("tampered") + with self.assertRaisesRegex(plan.PlanError, "hash mismatch"): + self.verify_evidence(record) + record = self.evidence() + record["commands"][0]["exit_code"] = 1 + with self.assertRaisesRegex(plan.PlanError, "Failed/unconfirmed"): + self.verify_evidence(record) + + def test_missing_or_wrong_source_input_refuses(self): + record = self.evidence() + record["source_inputs"].pop() + with self.assertRaisesRegex(plan.PlanError, "omits source inputs"): + self.verify_evidence(record) + record = self.evidence() + record["source_inputs"][0]["sha256"] = "0" * 64 + with self.assertRaisesRegex(plan.PlanError, "Source hash mismatch"): + self.verify_evidence(record) + + def test_unrecorded_changed_runtime_source_refuses(self): + with self.assertRaisesRegex(plan.PlanError, "changed runtime inputs"): + self.verify_evidence(self.evidence(), b"crates/unrecorded.rs\0") + + def test_release_commands_repeat_every_exact_case(self): + commands = plan.commands_for(self.manifest["tasks"][-1], self.manifest) + selected = {x[x.index("--exact") - 1] for x in commands if "--exact" in x} + expected = {c["name"] for task in self.manifest["tasks"] for c in task["test_cases"]} + self.assertEqual(selected, expected) + self.assertIn(["python3", "tools/docs/check.py"], commands) + + def test_complete_profile_matrix_passes(self): + self.verify_profile(*self.profile_fixture()) + + def test_profile_source_binary_or_config_drift_refuses(self): + for field, value in (("source_revision", "d" * 40), ("executable_sha256", "d" * 64), ("profiles_sha256", "d" * 64)): + profile, report = self.profile_fixture() + report[field] = value + with self.assertRaises(plan.PlanError): + self.verify_profile(profile, report) + + def test_missing_or_duplicate_cohort_refuses(self): + for mutate in (lambda x: x["cohorts"].pop(), lambda x: x["cohorts"].append(copy.deepcopy(x["cohorts"][0]))): + profile, report = self.profile_fixture() + self.edit_artifact(report, "measurements", mutate) + with self.assertRaisesRegex(plan.PlanError, "cohort"): + self.verify_profile(profile, report) + + def test_short_or_wrong_population_cohort_refuses(self): + for field, value in (("measured_seconds", 1), ("cells", 1), ("nodes", 1), ("source_revision", "d" * 40), ("executable_sha256", "d" * 64)): + profile, report = self.profile_fixture() + self.edit_artifact(report, "measurements", lambda x: x["cohorts"][0].update({field: value})) + with self.assertRaises(plan.PlanError): + self.verify_profile(profile, report) + + def test_numeric_failure_negative_boolean_or_nan_refuses(self): + for value in (1, -1, True, float("nan")): + profile, report = self.profile_fixture() + report["metrics"]["lost_acknowledged_effects"] = value + with self.assertRaises(plan.PlanError): + self.verify_profile(profile, report) + + def test_better_aggregate_cannot_hide_slow_cohort(self): + profile, report = self.profile_fixture() + def slow(measured): + row = next(x for x in measured["cohorts"] if x["variant"] == "replacement" and x["occupancy"] == 64) + row["metrics"]["p99_ms"] = 400 + row["metrics"]["p99_ratio_to_occupancy1"] = 2 + self.edit_artifact(report, "measurements", slow) + with self.assertRaisesRegex(plan.PlanError, "worst cohort"): + self.verify_profile(profile, report) + + def test_ratio_must_match_control_and_nonzero_denominator(self): + profile, report = self.profile_fixture() + self.edit_artifact(report, "measurements", lambda x: x["cohorts"][0]["metrics"].update(p99_ms=0)) + with self.assertRaisesRegex(plan.PlanError, "denominator"): + self.verify_profile(profile, report) + + def test_safety_failure_in_control_or_historical_cohort_refuses(self): + for variant in ("replacement", "historical"): + profile, report = self.profile_fixture() + def lose(measured): + row = next(x for x in measured["cohorts"] if x["variant"] == variant and x["occupancy"] == 1) + row["metrics"]["lost_acknowledged_effects"] = 1 + self.edit_artifact(report, "measurements", lose) + with self.assertRaisesRegex(plan.PlanError, "worst cohort"): + self.verify_profile(profile, report) + + def test_independent_hosts_cannot_be_faked_by_container_count(self): + profile, report = self.profile_fixture("independent_domain_loss") + self.verify_profile(profile, report) + self.edit_artifact(report, "topology", lambda x: [n.update(physical_host="same-host") for n in x["nodes"]]) + with self.assertRaisesRegex(plan.PlanError, "independent hosts"): + self.verify_profile(profile, report) + + def test_verification_must_bind_actual_ledger(self): + profile, report = self.profile_fixture() + self.edit_artifact(report, "verification", lambda x: x.update(ledger_sha256="d" * 64)) + with self.assertRaisesRegex(plan.PlanError, "another ledger"): + self.verify_profile(profile, report) + + def test_verification_cannot_omit_artifact(self): + profile, report = self.profile_fixture() + del report["artifacts"]["fault_schedule"] + with self.assertRaisesRegex(plan.PlanError, "Missing profile artifact"): + self.verify_profile(profile, report) + + def test_brief_labels_unimplemented_target_and_exact_commands(self): + text = plan.brief(self.manifest["tasks"][0], self.manifest, self.baseline, self.root) + self.assertIn("PLANNED OUTPUT, currently absent", text) + self.assertIn("--exact --test-threads=1", text) + self.assertIn("A plan-only result is not DONE", text) + + def test_release_covers_all_executable_paths_and_plan_only_commits(self): + def git(*args): + return subprocess.check_output(["git", "-c", "core.hooksPath=/dev/null", "-c", "commit.gpgsign=false", + "-c", "user.name=Plan Validator Test", "-c", "user.email=plan-test@example.invalid", + *args], cwd=self.root, text=True, stderr=subprocess.DEVNULL) + git("init", "--quiet") + inputs = ["crates/lib.rs", "examples/main.rs", "vendor/lib.rs", ".cargo/config.toml", "build.rs", + "rust-toolchain.toml", "docs/api.md", "future-runtime-input.txt"] + for name in inputs + ["plans/status.json", "docs/design/status.json"]: + self.put(name, "original\n") + git("add", ".") + git("commit", "--quiet", "-m", "Synthetic validator fixture") + revision = git("rev-parse", "HEAD").strip() + plan.require_release_inputs(self.root, revision) + self.put("plans/status.json", "updated plan\n") + self.put("docs/design/status.json", "updated plan\n") + plan.require_release_inputs(self.root, revision) + git("add", "plans", "docs/design") + git("commit", "--quiet", "-m", "Plan-only status") + plan.require_release_inputs(self.root, revision) + for name in inputs: + self.put(name, "changed\n") + with self.assertRaisesRegex(plan.PlanError, "dirty"): + plan.require_release_inputs(self.root, revision) + self.put(name, "original\n") + self.put("examples/main.rs", "changed\n") + git("add", "examples") + git("commit", "--quiet", "-m", "Changed executable") + with self.assertRaisesRegex(plan.PlanError, "differ from qualified"): + plan.require_release_inputs(self.root, revision) + + +if __name__ == "__main__": + unittest.main() diff --git a/docs/design/multi-raft/density-decision.md b/docs/design/multi-raft/density-decision.md new file mode 100644 index 000000000..c864f99f0 --- /dev/null +++ b/docs/design/multi-raft/density-decision.md @@ -0,0 +1,272 @@ + + +# Shared cell groups: architecture decision + +Purpose: TiKV investigation and the decision to share physical Raft groups across whole cells. + +Status: selected replacement architecture; runtime conversion and qualification +are pending. This document does not claim that Cellar supports shared groups yet. +It supersedes one-cell-per-group as the target architecture, while the existing +technical design and qualification records describe the implemented baseline. + +The [executable delivery package](cell-groups/README.md) expands this decision into +the complete replacement, including dynamic split/merge and coordinated hibernation. +Its detailed protocols supersede the initial assignment sketch below: range-based +ownership and replicated enrollment replace per-cell reservation as the single +assignment authority, and stable cell history is separate from group placement. +Fixed assignment is an intermediate delivery milestone, not the final scope. + +## Decision + +Make a **cell group** the unit of Raft consensus, placement, membership, snapshot +and disaster recovery. A group contains multiple cells. Each cell retains its own +SQLite database, LTX chain, command receipts, query versions and application +identity. Every replica of a group owns the group's complete committed state. + +Use one implementation for every group size. A large or hot cell can occupy a +group alone; it does not select a second consensus architecture. Remove the +assumption that creating a cell creates a Raft instance. Keep the shared node log +engine, worker pools, transport and resource limits already implemented. + +Groups are independent of physical nodes. A three-voter group spans three +placement domains; one node can lead several groups and follow others. Grouping +all cells currently on a node would bind ownership to a host and produce an +unbounded failure and scheduling unit. It is not the intended design. + +The decision targets the fixed consensus overhead of many small cells. It does +not promise lower latency for every workload, remove per-cell storage work, or +provide cross-cell transactions. Busy cells in the same group compete for a +leader, log bandwidth and the contiguous apply prefix. + +Grouping also does not split a hot cell's SQLite transaction stream across leaders. +The application retains its existing per-cell atomicity boundary. + +## TiKV verification + +Inspected the user-provided TiKV checkout at commit +`b3995dab28308f09cbb7c125bf32f789dbc203fe` (2026-10-22). No TiKV files were changed +and no TiKV benchmark was run. The paths below are relative to that checkout. + +| Source | Verified behavior | Cellar consequence | +| --- | --- | --- | +| `components/raftstore/src/store/peer.rs`, `Peer` | A Region peer owns a `RawNode>`. | One Raft machine per group replica, rather than per application object. | +| `components/raftstore-v2/src/raft/peer.rs`, `Peer` | The tablet path also owns one `RawNode>` per peer. | The consensus boundary survives a change of storage layout. | +| `components/raftstore/src/store/fsm/store.rs`, `StoreMeta` | Maps Region IDs and key-range boundaries to Region metadata. | A node hosts many groups and routes to the selected group. | +| `components/raftstore/src/store/util.rs`, `check_key_in_region`; `store/fsm/apply.rs`, `handle_put` | Writes validate their key against the Region's half-open key range. Many distinct keys can use the same Region. | Every Cellar entry must identify and validate its target cell inside its group. | +| `components/raftstore/src/store/fsm/apply.rs`, `exec_batch_split`, `exec_prepare_merge`, `exec_commit_merge` | Splits and merges change Region boundaries and epochs through replicated administrative operations. | Moving a cell between groups requires a protocol and routing fencing, not a local map edit. | +| `components/raftstore/src/store/async_io/write.rs`, `WriteTaskBatch::add_write_task` | Batches log writes while retaining their Region IDs. | Shared physical persistence and shared consensus groups are separate optimizations. Cellar already has the former. | +| `components/raftstore/src/store/fsm/peer.rs`, `quorum_agree_to_hibernate`, `on_hibernate_request`, tick handling | Coordinates hibernation votes and checks pending history before stopping ticks. | Group hibernation is a separate future protocol; an idle cell cannot stop a busy group's Raft ticks. | + +TiKV's [multi-Raft explanation](https://tikv.org/deep-dive/scalability/multi-raft/) +and [current Region glossary](https://docs.pingcap.com/tidb/stable/glossary/#regionpeerraft-group) +confirm that a Region holds a range of keys and its peers form one Raft group. +Cellar's analogous partition contains whole cells, not individual SQLite keys or +pages. Keep the name **cell group** to avoid confusing it with a geographic +region. Do not copy TiKV's byte thresholds or historical event-loop defaults: +Cellar has different image, transaction and recovery costs. + +## Evidence and expected savings + +Cellar source inspected: `1be27d234d4674d4d014785c60cecbfe394bdab9`. +The local idle experiment at that source measured 32 and 128 three-voter groups: +1.437 and 5.794 MB/s outgoing control traffic, respectively, or approximately +44 KiB/s per logical group across the three embedded nodes. Settled ten-second +windows recorded no engine writes or object-store reads/puts. These are debug +build diagnostics on one M2 Max with TLS loopback proxies and an in-memory object +store, not production costs, RustFS results or a shared-group benchmark. +Probe sources, exact environment and run metadata remain in the local ignored +`.cellar/cell-cost-report.md` and `.cellar/cell-cost-metadata.json`. + +For C cells with approximately K cells per group, G is approximately ceil(C/K). +With R voters, no learners and heartbeat interval H, baseline leader-to-follower +logical deliveries are G*(R-1)/H per second, before replies and batching. + +| Cells | Target cells/group | Groups at ideal packing | Group replicas with three voters | +| ---: | ---: | ---: | ---: | +| 1,000,000 | 1 | 1,000,000 | 3,000,000 | +| 1,000,000 | 32 | 31,250 | 93,750 | +| 1,000,000 | 64 | 15,625 | 46,875 | +| 1,000,000 | 128 | 7,813 | 23,439 | + +The number of cell database replicas remains 3,000,000 in every row. Grouping +reduces consensus state, timers, election and membership bookkeeping. It does not +divide cell data, receipts, LTX bytes, SQLite apply work or archive payload by K. +These counts are arithmetic, not a supported fleet size or a measured speedup. +Partially filled groups, dedicated cells and placement constraints increase G. + +Benchmark occupancies 1, 8, 32, 64 and 128 before choosing a default. Limit group +size by total bytes, write rate, apply latency and recovery budget as well as cell +count. Cell count alone cannot distinguish 64 tiny counters from 64 large cells. + +## Identity, assignment and admission + +Separate physical consensus history from the target cell: + +| Contract | Scope | +| --- | --- | +| `GroupScope` | Cluster, namespace, group ID, group incarnation; no cell name | +| `CellScope` | Group scope, cell identity and immutable assignment identity | +| Group birth / membership | Voter identities, disk claims, placement and initial protocol/schema binding | +| Cell enrollment | Cell scope, schema/application digest and verified initial image | +| Command / receipt / served version | Cell scope plus the exact group term/index and cell image identity | + +The names above describe the replacement contracts, not types already delivered. +Start with cells from one namespace and its immutable application/schema contract +in a group. Do not combine tenants or incompatible durability/placement policies +merely because they currently reside on the same node. + +Initially, assignment is immutable for an enrolled cell. A directory selects a +group with capacity for a new cell; adding nodes or groups does not rehash existing +cells. Concurrent creation uses a single conditional cell-assignment reservation. +Only its confirmed winner may enroll the cell, and the replicated enrollment binds +that reservation and the group incarnation. An uncertain reservation is reconciled +before proceeding. Route publication follows committed, durable enrollment; a +stale route cannot grant authority. Reservations and retries are bounded and +recoverable across controller/leader crashes. + +New groups can take new cells without moving existing ones. Group capacity is +rechecked at enrollment, so concurrent reservations cannot exceed its hard bound. +A refused reservation cannot be silently reassigned after it might have enrolled. +Birth records and permanent voter-disk claims occur per group. Cell-directory and +enrollment metadata still scale with the cell population. + +Online cell transfer, split and merge are separate work. They require source +quiescence committed through its quorum, an exact transfer cut, destination +installation and a fenced routing transition. No destination becomes writable +while the source might still accept commands. Receipts and effect identities must +survive transfer without replaying acknowledged work. Until that protocol passes +fault tests, operations move whole groups or create new groups for new cells. + +## Write, apply and read invariants + +1. Admit a command under current group leadership and the exact cell assignment. + Prepare it in that cell's private SQLite transaction, including its receipt. + Preserve one unresolved data proposal per cell in the first implementation. +2. Propose an envelope containing the group history, cell identity/assignment, + proposal identity, application digest and that cell's base/result LTX images. + Never propose an unscoped LTX update or replay application callbacks on followers. +3. Different cells may prepare concurrently through bounded shared SQL workers. + Fair cell queues feed group and node byte budgets. A shared log still orders + all entries; initial apply publication remains ordered by group index. +4. Apply an entry only to its addressed cell. Fsync immutable image data before + atomically advancing the durable group manifest and contiguous applied prefix. + Publish the cell view and settle output only after that marker is durable. +5. Quorum and S3 acknowledgements retain their existing meanings. Timeouts remain + ambiguous after proposal; duplicate receipt lookup remains scoped to the cell. + +If entries 40 and 41 update A and B, group applied=41 means both entries are +applied. A's latest image still corresponds to entry 40. B's progress cannot satisfy +an A receipt, replace A's base image or manufacture a newer A version. Advancing a +global applied index past an unfinished earlier entry is forbidden. + +Linearizable reads obtain a current-term group barrier, wait for the group apply +prefix to cover it, then pin the target cell's image from a coherent published +manifest. The cell need not have a mutation at the barrier index. AtLeast compares +the target cell's verified version within the same assignment and incarnation. +Stale reads remain explicitly stale. No read treats directory ownership or another +cell's newer image as a quorum proof. + +## Durable group manifest and crash recovery + +Replace the single-image `DurableApply` with a bounded committed manifest root. +The root authenticates the group history, applied position, enrollment state and +each cell's image commitment, LTX position and latest mutation position. Include +deletions/tombstones if deletion is exposed; absence must not resurrect a cell. +Use incremental immutable manifest storage so updating one cell does not copy +every SQLite image or serialize an unbounded group map on each command. + +Recover only a complete manifest selected by the durable engine marker. A crash +before the marker may leave orphan image/manifest objects; it cannot publish them. +A crash after the marker must recover every referenced cell without repeating its +callback. Missing, corrupt or mismatched referenced state refuses group admission. +Retention must protect the previous recoverable root until the new root is durable, +plus all snapshot, archive, reader and in-flight apply pins. + +Raft hard state, membership and apply state remain group scoped. Preserve the +existing shared-engine failure quarantine and permanent voter identity checks. +Closing one idle execution cache leaves its committed image and group membership +intact. It does not stop elections, change ownership or erase its cell manifest. + +## Snapshots, S3 and reclamation + +A group checkpoint at index K contains the complete cell manifest as of K, +membership/enrollment state and all referenced image data or extent roots. Cell +images may have different last mutation indexes, each no greater than K. Pin that +consistent set before streaming. Install it atomically at the group boundary; +receiving only a subset cannot make the group queryable or writable. + +S3 recovery sets contain a complete checkpoint plus an optional contiguous, +cell-addressed committed log tail through a declared group cut. Verify every +dependency, checksum, base/result chain and membership transition before publishing +coverage. Subsequent checkpoints can reuse unchanged immutable cell objects; they +need not upload every database again. A hot cell's archive failure can delay the +group's archive prefix, so track group lag and bound retained bytes. + +An S3 acknowledgement for entry N requires a retained complete group recovery set +covering N. Archive coverage for one cell cannot cover unrelated earlier entries +in the group. Keep quorum-only acknowledgements distinct: a zero-quorum disaster +can lose their unarchived suffix even though the live quorum had acknowledged it. + +Local log truncation requires a valid covering group snapshot and the existing +recovery/retention conditions. Where reclamation relies on S3, require confirmed +complete group coverage too. Uploaded bytes alone establish neither condition. +Physical shared-engine file reclamation must respect every group's live log ranges. +Reclaiming entries never deletes current cell images or dependencies pinned by a +retained recovery set. Preserve snapshot availability for lagging peers. + +Zero-quorum recovery fences every old voter of the group using the existing +fencing-authority contract, then creates one new group incarnation with fresh voter +identities. Restore every cell from the selected group recovery set, preserve its +receipts and require a fresh quorum before admission. Old tokens/assignments cannot +be used to write in the new incarnation. Old-origin outbox ambiguity still refuses +admission unless explicitly reconciled; grouping does not solve external effects +or make different groups' recovery cuts a fleet-wide consistent snapshot. + +## Replacement and delivery + +Replace the current one-cell/group path instead of adding another production mode. +Keep format markers at 1, with distinct protocol/domain bindings that reject old +layouts and mismatched peers. Use a fresh development namespace/storage root for +the hard cutover. Do not reinterpret a one-cell image as a multi-cell manifest, +silently migrate existing state, or delete existing disks/archives. A group with +one enrolled cell uses the exact same replacement code as a larger group. + +| Step | Owning code | Required evidence before completion | +| --- | --- | --- | +| 1. Separate identities and enrollment | `cellar-logic/consensus.rs`; `application/consensus/protocol.rs`; Raft catalog, routing, version and receipt adapters | Cross-cell/namespace/assignment tokens and LTX updates refuse; concurrent enrollment has one winner; old wire/storage bindings refuse. | +| 2. Group manifest and ordered apply | Raft cache/storage, apply, extent roots, host and SQL owners | Three distinct SQLite cells in one actual `RawNode` group; interleaved writes; durable marker/image crash cuts; duplicate receipts; missing manifest dependencies refuse. | +| 3. Group snapshots and peer rebuild | Snapshot/checkpoint transfer, membership and startup | Follower catch-up and lost-disk replacement restore every cell at one cut; no partial group admission; bounded streaming and pins. | +| 4. Group S3 archive and reclamation | Archive codec/publisher/recovery, extent objects, log purge | Real RustFS checkpoint/tail restore; delayed cell dependency blocks coverage; acknowledged history survives actual file reclamation and restart. | +| 5. Fenced group disaster recovery | Catalog disaster API, fencing authority and Docker fixture | Delete all voter disks; restore all cells into one new incarnation; reject old tokens; preserve receipts and new writes; quorum-only unarchived suffix reported separately. | +| 6. Fleet and application integration | Public SDK, native/Axum ingress, fleet planner, alarms, outbox and operator views | One group leader/placement for many cells; per-cell authorization, schema, effects and fairness; whole-group repair/drain; all owners join on shutdown. | +| 7. Remove baseline assumptions and qualify | CLI/examples/docs, all native Raft harnesses, capacity profiles | No production fallback allocating a group for each ordinary cell; occupancy-one uses shared code; equal-workload performance and fault evidence below. | + +These are dependent implementation slices of one replacement, not separate +supported architectures. They are not marked complete by adding types, changing +an ID, grouping rows in one SQLite database, or counting fewer catalog records. +Existing MR-16 physical-tier release gates remain open. + +## Qualification matrix + +Compare the existing revision against the replacement at equal logical cell +population, per-cell data sizes, offered arrival rate, active fraction, request +mix, durability and hardware. Within the replacement, compare occupancies +1/8/32/64/128. Use independent SQLite files and LTX chains in every cohort. + +Measure idle CPU/RSS/FD/control bytes, real Raft instance count, fsyncs, log and +image write amplification, archive requests/bytes/lag, p50/p95/p99, refusal/error +counts and recovery time to successful business reads/writes. Include a hot cell +beside cold cells, concurrent cells in one group, separate groups, large images, +slow apply, a slow minority, leader loss, object-store outage and mixed durability. +Record shared-group head-of-line delay rather than averaging it away. + +Fault tests must cover interleaved cell history across every new durable boundary, +stale assignment/term completions, enrollment crash/retry, snapshots while another +cell writes, corrupt/missing cell dependencies, reclamation with lagging peers, +retained-disk restart and the destructive real RustFS drill above. Strong-read and +receipt assertions must use the individual cell's history. Preserve failure and +ambiguous outcomes in the workload ledger. + +Group hibernation follows as separate work with safe wake/read authority, retained +state, peer-triggered wake, pending work, alarms/outbox and membership coverage. +No full-group wake latency or production density claim follows from this decision. diff --git a/docs/design/multi-raft/fleet-operations/operator-workflows.md b/docs/design/multi-raft/fleet-operations/operator-workflows.md index e201c14ca..2115e77ca 100644 --- a/docs/design/multi-raft/fleet-operations/operator-workflows.md +++ b/docs/design/multi-raft/fleet-operations/operator-workflows.md @@ -23,6 +23,7 @@ flowchart LR Start with these sections; subsequent entries retain their original source and evidence scope: - [Delivered keep-set workflow](#delivered-keep-set-workflow) +- [First managed group workflow](#first-managed-group-workflow) - [Delivered add-capacity workflow](#delivered-add-capacity-workflow) - [Failure-observation SDK workflow](#failure-observation-sdk-workflow) - [Automatic failure-recovery SDK workflow](#automatic-failure-recovery-sdk-workflow) @@ -101,6 +102,103 @@ The API routes use `/api/v1/fleets/{fleet}/raft`: - CLI exit codes are 0 for the requested result, 2 invalid input, 3 conflict, 4 blocked/cancelled/authentication/authorization, 5 unavailable or unknown result, and 6 wait timeout. - Mutations are never retried automatically; exact caller retries are reconciled against the catalog. +## First managed group workflow + +Provision a new native SQL namespace with +`FleetCatalog::provision_managed_sql_namespace` and its compiled `RaftClass`. +The SDK binds the fleet cluster and class schema, and declares a shared Raft +standby with three voters, quorum writes and materialized images. Use +`provision_managed_namespace` for an explicit namespace policy such as extent +images. Configure the admin fleet with the resulting exact declaration. Bring +up the three enrolled hosts and qualify and activate their capacity through +the workflow below. Read the current catalog generation and revision before +preparing the first group. + +A compiled application's one-shot bootstrap command can provision the same +default standby before starting any voter. Run it with write access to the +selected storage scope and an existing fleet catalog. + +```sh +application --provision-sql-namespace sql-namespace.json --bucket "$FLEET_BUCKET" +``` + +The JSON plan contains `catalog_id`, `cluster_id`, `incarnation_id`, +`coordination_prefix`, `catalog_prefix`, `namespace_id`, `script`, `version`, +`class`, `principal`, `request_id`, `operation_id` and `expected_revision`. +Use the exact compiled script version and Raft class. The optional +`image_storage` field accepts `materialized` (the default) or `extents`. +The receipt includes the selected image policy and schema digest for the +admin and voter configuration. +Retain the plan and retry it byte for byte after an unknown result. + +```sh +cellar raft fleet first-group standby "$NAMESPACE" --fleet F --catalog-id "$CATALOG" --json +cellar raft fleet first-group reserve "$NAMESPACE" --fleet F --catalog-id "$CATALOG" \ + --plan first-group.json --json +cellar raft fleet first-group publish "$NAMESPACE" --fleet F --catalog-id "$CATALOG" \ + --plan first-group.json --request-id "$CONFIRMATION" --expected-revision "$CURRENT_REVISION" --json +cellar raft fleet first-group archive "$NAMESPACE" --fleet F --catalog-id "$CATALOG" \ + --plan first-group.json --request-id "$ARCHIVE_REQUEST" --expected-revision "$PUBLISHED_REVISION" --json +``` + +Keep `first-group.json` with the catalog and namespace IDs, the returned +`standby` digest, and the reservation's `request_id`, `operation_id` and +`expected_revision`. Include `publication_id`, `group_id`, +`incarnation_id` and `creation_id` identities and exactly three `voters`. +Each voter has a registered `node_id` and a nonzero numeric `member_id`. +List voters in increasing member ID order. Hex identities use lowercase and +the same lengths as the node and fleet operations below. + +For an uninterrupted first-group setup, persist a single `setup.json` with +`reservation` set to the full `first-group.json` object. Add `publication` and +`archive`, each containing its own `request_id` and `expected_revision`. With +exclusive fleet mutation during setup, use the reservation's revision plus one +for publication and plus two for archive initialization. + +```sh +cellar raft fleet first-group apply "$NAMESPACE" --fleet F --catalog-id "$CATALOG" \ + --plan setup.json --json +``` + +The CLI validates all three identities and the catalog pin before submission. +It runs the existing authenticated operations in order under one 30-second +deadline and returns the three verified receipts. An error from a specific +stage describes that stage's outcome. Earlier stages may already be committed. +Replay the saved plan after an unknown outcome. If a concurrent fleet change +causes a definite revision conflict, use the individual stage commands to +reconcile with the latest revision. Daemon readiness still requires the local +member claim and a fresh quorum. + +Reserve checks live capacity and replica separation and binds the exact range +mutation. Confirm with a distinct request ID at the observed fleet revision. +Publish uses the original reservation and plan. Retry an uncertain reply with +the same confirmation ID and revision. After a definite revision conflict, +retain the plan and use a new confirmation request at the new revision. A +published receipt describes catalog admission. Read the current revision again +before submitting the archive request with its own ID. The archive command roots +the exact initializer intent before creating registry state. Reconcile an unknown +reply with the same request. An incomplete archive refuses verification and +requires authority recovery. An archive receipt supplies no coverage or voter +authority. For a compiled SQL application on its initial voters, use +`NativeDaemon::managed_sql_namespace` with the cluster and namespace IDs, the +startup timeout, resource preparation and the application binding. The daemon +discovers the first full-range group, waits for the rooted archive and complete +registry, validates or attaches the local voter and establishes fresh quorum +under the same deadline. The binding receives the selected history after that +barrier. The same entry point can reopen a retained initial voter while the +first group and its membership remain selected. +Managed retained boot discovers local replicas from its opened engine within +the declared namespaces. Each replica still requires its selected catalog +history, exact boot and membership authorization before attachment. Configure +router placements explicitly. + +| Method and suffix under `/api/v1/fleets/{fleet}/raft` | Permission and result | +| --- | --- | +| `GET /namespaces/{namespace}/first-group/standby` | `metadata_read`; exact empty standby digest | +| `POST /namespaces/{namespace}/first-group/reserve` | `fleet_topology`; exact plan and historical reservation receipt | +| `POST /namespaces/{namespace}/first-group/publish` | `fleet_topology`; `plan`, `request_id`, `expected_revision` and historical publication receipt | +| `POST /namespaces/{namespace}/first-group/archive` | `fleet_topology`; `plan`, `request_id`, `expected_revision` and exact archive initialization receipt | + ## Delivered add-capacity workflow The SDK, admin API and CLI expose explicit registration, qualification and activation. `fleet_register` authorizes a deployment descriptor; `fleet_capacity` separately trusts the operator to attest actual platform limits. Neither `operate`, `configure`, `fleet_topology` nor either new grant implies the other. Use a separately granted `fleet_topology` principal for redistribution. Human authentication supplies the principal; JSON cannot choose it. @@ -744,4 +842,4 @@ The following JSON is an illustrative future response, not a measured run. Group | `PlannerSearchExhausted` | Narrow the requested change or increase a reviewed search budget; no unsafe partial plan is admitted | | `StaleMaintenanceReceipt` | Revalidate the original maintenance operation against the current process and group state | -Pause optional balancing to reduce movement pressure. Cancellation prevents further unadmitted steps, reconciles existing actions and may finish with partial committed effects. Never kill background workers or delete CAS records to cancel. \ No newline at end of file +Pause optional balancing to reduce movement pressure. Cancellation prevents further unadmitted steps, reconciles existing actions and may finish with partial committed effects. Never kill background workers or delete CAS records to cancel. diff --git a/docs/design/multi-raft/implementation-plan.md b/docs/design/multi-raft/implementation-plan.md index d5c84897e..bae1eb8c4 100644 --- a/docs/design/multi-raft/implementation-plan.md +++ b/docs/design/multi-raft/implementation-plan.md @@ -1194,7 +1194,7 @@ Stop conditions: Stop if full-image publication dominates the declared SLO; redu | `harness_rejects_stale_binary` | Benchmark cannot attribute results to a different executable | | `archived_progress_wakes_commands_without_maintenance_poll` | Committed images and accepted archive coverage wake eight sequential S3 commands while preserving normal proof checks | -Verification target: `native_raft_multigroup`. Run the exact list/test commands shown by `plan.py show MR-14`; the manifest is authoritative for arguments. +The `native_raft_multigroup` target and its `plan.py show MR-14` commands apply to the former per-cell runtime at the revisions recorded in its evidence. The [cell-group plan](cell-groups/implementation-plan.md) governs the shared runtime. Current archive qualification uses independently verified group holds, and the benchmark freshness test runs in `native_cell_groups_public`. ### MR-15 Qualify failure histories and release the isolated mode @@ -1228,6 +1228,12 @@ Verification target: `native_raft_qualification`. Run the exact list/test comman Dependencies: MR-15. Phase: scale. Initial status: TODO. +The [density decision](density-decision.md) selects shared cell groups as the +replacement for one-cell consensus. The [cell-group plan](cell-groups/README.md) +tracks runtime, manifests, archive, recovery and SDK conversion. This does not +complete MR-16 or relax its MR-15 dependency and physical-tier release gates. + + Modify: `crates/cellar/durability/raft/runtime.rs`, `crates/cellar/durability/raft/placement.rs`, `crates/cellar/durability/raft/apply.rs`, `tools/perf/raft_profiles.json`. Create: `docs/design/multi-raft/density-decision.md`, `crates/cellar/tests/native_raft_density.rs`. diff --git a/docs/design/multi-raft/scaling.md b/docs/design/multi-raft/scaling.md index 78d5cca22..bd1cd8e36 100644 --- a/docs/design/multi-raft/scaling.md +++ b/docs/design/multi-raft/scaling.md @@ -18,7 +18,7 @@ flowchart LR B --> C["Measured qualification tier"] ``` -Multi-Raft distributes many small consensus groups. The first release uses one cell per group and three voters. A node may lead one cell and follow another; adding nodes does not enlarge every group's quorum. Aggregate capacity grows only while storage, network, routing, apply and workload distribution permit it. +Multi-Raft distributes many small consensus groups. The current production runtime uses one cell per group and three voters. The [density decision](density-decision.md) selects shared cell groups as its replacement; cutover and qualification remain pending. A node may lead one group and follow another; adding nodes does not enlarge every group's quorum. Aggregate capacity grows only while storage, network, routing, apply and workload distribution permit it. ## Capacity dimensions @@ -82,16 +82,27 @@ Limit concurrent snapshot generation/transfers and archive backfill. Reserve sto LTX logs replicate changed pages, which may exceed logical command size. Measure logical bytes, encoded LTX bytes, replicated bytes, SQLite copy/apply bytes and S3 bytes separately. Full-image publication is expected to constrain larger cells; do not extrapolate a small-counter benchmark to large databases. -## Later group and storage refinements - -MR-16 first measures which resource limits density. It then selects and qualifies one improvement at a time: page-versioned committed images, safe group hibernation, or several small cells in one group. Every choice requires an updated protocol and crash tests before implementation. - -- Shared groups retain separate SQLite databases and LTX chains. -- Their log entries identify a cell; leadership/membership and consensus ordering are shared. -- This reduces group count but couples placement, failure recovery and throughput. -- It does not create cross-cell atomicity. -- Large/hot cells may merit dedicated groups, but an online transfer must fence the source, commit a transfer boundary, install the destination and atomically change routing without admitting two writers. -- Until that protocol is qualified, group assignment is immutable. +## Shared group replacement and later refinements + +The [density decision](density-decision.md) selects shared cell groups and specifies +their identity, apply, snapshot, archive and recovery boundaries before runtime +conversion. The [executable delivery package](cell-groups/README.md) defines the +complete implementation, including safe dynamic split/merge and hibernation. +Separate SQLite databases and LTX chains remain per cell. Leadership, +membership, placement and the contiguous apply/archive prefixes become group +scoped. Dedicated hot cells use the same implementation at occupancy one. + +With C cells and average occupancy K, group count is approximately C/K; physical +cell database replicas remain C*R. The capacity profiles above still describe +the one-cell baseline. Shared groups reduce fixed consensus overhead, while +adding queueing and recovery coupling that must be measured under equal logical +cell workloads. They provide no cross-cell transaction contract. + +Initial cell assignment is immutable. Online transfer, split and merge require +source fencing, a committed transfer cut, destination installation and a fenced +routing transition. Group hibernation is also a separate protocol and cannot be +inferred from closing an idle cell's execution cache. MR-16's physical-tier gates +remain unchanged; the architecture decision is not scale qualification. ## Federation for very large fleets diff --git a/docs/design/multi-raft/technical-design.md b/docs/design/multi-raft/technical-design.md index fbc21c8af..75690e32a 100644 --- a/docs/design/multi-raft/technical-design.md +++ b/docs/design/multi-raft/technical-design.md @@ -12,6 +12,11 @@ Purpose: Define consensus authority, commit, persistence, read proofs, archival, Status: proposed implementation contract. Baseline and source hashes are in [baseline.json](baseline.json); execution order is in [implementation-plan.md](implementation-plan.md). +The [shared cell-group decision](density-decision.md) supersedes the one-cell +consensus target below. This document retains the earlier runtime's contracts; +[the replacement](cell-groups/design.md) separates cell identity from group +consensus and requires complete-group recovery and qualification before cutover. + ## 1 Scope and architectural decision - Introduce an opt-in Raft namespace. diff --git a/docs/index.html b/docs/index.html index ab7639cc7..c3bb476e5 100644 --- a/docs/index.html +++ b/docs/index.html @@ -63,7 +63,7 @@

The document library

Choose your r