diff --git a/CHANGELOG.md b/CHANGELOG.md index 83cf4c08..633561b2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,10 @@ after its public API and format compatibility policies are established. ## [Unreleased] +- Completed migration recovery now verifies the version-two namespace before reporting success, refusing unknown reserved GC/recovery entries without effects while preserving published retention state (#111). Recovery storage implementors must supply the new read-only `verify_complete` capability. + +- Migration restart laws preserve complete filesystem witnesses when rejecting damaged records and pools, conflicting or substituted stages, invalid ordering, copied-root identity, changed inventory, foreign receipts, unknown names and wrong kinds (#111). + - Reader-fence process scenarios retain migration writer authority through collector preparation, removing a release/reacquire gap without changing production lock semantics (#174). - Linux public snapshot process laws verify reader-death fence release, persistent lock identity and exclusion of new readers during collection with kernel-observed ordering (#113). diff --git a/docs/formats/segment-store-v2/migration-recovery.md b/docs/formats/segment-store-v2/migration-recovery.md index 033a183d..ea4124d3 100644 --- a/docs/formats/segment-store-v2/migration-recovery.md +++ b/docs/formats/segment-store-v2/migration-recovery.md @@ -44,8 +44,6 @@ only version-2 migration recovery may continue. The migration recovery boundary admits only these ordered prefixes: - - | State | Required response | | --- | --- | | no migration artifact | admit exact version 1 | @@ -56,8 +54,6 @@ The migration recovery boundary admits only these ordered prefixes: | marker without receipt | reopen full v2 view and publish receipt | | exact receipt with optional exact receipt stage | clean the stage and admit complete migration | - - Every row also requires the admission checks in [Executable recovery boundary](#executable-recovery-boundary). An intent stage surviving a namespace effect, or a marker stage surviving a receipt effect, @@ -99,7 +95,8 @@ Restart compares device and inode identity; mount identity is same-process evidence. Whenever an exact intent survives, recovery continues with its persisted bytes. -The filesystem laws cover every forward prefix, every strict byte-prefix -truncation of all three stages, unchanged version-1 bytes, and refusal before -mutation for corrupt intent and unexpected nested residue. The complete -restart corruption matrix remains tracked separately in #111. +The filesystem laws cover every forward prefix, every strict byte-prefix truncation of all three stages and unchanged version-1 bytes. The [restart ambiguity matrix](../../testing-evidence/migration-restart-matrix.md) covers corrupt records, contradictory and byte-equal substituted stages, invalid ordering, copied-root identity, immutable-pool and current-head damage, changed inventory, foreign receipts, unknown names and wrong kinds. Each new refusal retains a complete before/after witness of names, file identities and bytes and checks the existing typed failure boundary. The evidence record distinguishes kernel/filesystem behavior from platform-admission, process-death and power-loss claims. + +## Completed namespace admission + +After a plan selects `Complete`, recovery calls `StoreMigrationRecoveryStorage::verify_complete` before returning its receipt. The filesystem implementation applies existing version-two namespace admission: reserved GC/recovery entries must match the current protocol, while owned retention state is permitted. A refusal retains the original cause under `StoreMigrationRecoveryError::Observation` and `FilesystemMigrationRecoveryRefusal::NamespacePreflight`, before any recovery effect. This is namespace admission, not verification or recovery of retained content; retention remains the owner of those records and stages. diff --git a/docs/formats/segment-store-v2/rationale.md b/docs/formats/segment-store-v2/rationale.md index 47f563f5..4f8388df 100644 --- a/docs/formats/segment-store-v2/rationale.md +++ b/docs/formats/segment-store-v2/rationale.md @@ -117,3 +117,9 @@ filesystem UUID as the device coordinate in this change. That would require separate platform admission, compatibility, and format decisions. Device renumbering remains a refusal; this change does not introduce re-admission or silently substitute a different identity coordinate. + +## Verify completed migration before reporting completion + +An exact receipt proves the migration records agree; it does not prove the current reserved namespace still admits. Recovery therefore invokes the read-only `StoreMigrationRecoveryStorage::verify_complete` capability after planning selects `Complete` and before returning success. The filesystem adapter reuses version-two namespace admission, retaining the original cause through `Observation` and `NamespacePreflight`. Existing corruption/planning refusals retain priority because this check follows planning. No namespace effect, synchronization or retention recovery is initiated. + +Completed stores may contain published retention roots, manifests and heads. Reusing the partial-migration empty-directory preflight would reject valid post-migration state, so completion uses the existing version-two admission policy instead. Migration completion verifies its root/reserved directory contract; retention owns the interpretation of retention artifacts and stages. This adds a required method to the public recovery storage port; external implementations must implement equivalent effect-free admission. On-disk formats and identities are unchanged. diff --git a/docs/formats/segment-store-v2/requirements.md b/docs/formats/segment-store-v2/requirements.md index 3b9920cb..69e9f524 100644 --- a/docs/formats/segment-store-v2/requirements.md +++ b/docs/formats/segment-store-v2/requirements.md @@ -32,7 +32,7 @@ case is not evidence. | `KEEP-MIGRATION-002` | Format marker, intent, and receipt have complete fixed byte tables, named domains, bounds, checksums, deterministic store identity, and exact initial-state digests | exact admission in `tests/store_format_marker.rs`, `tests/store_migration_intent.rs`, and `tests/store_migration_receipt.rs`; canonical construction in `tests/store_migration_intent_encoding.rs` and `tests/store_migration_receipt_encoding.rs`; seeded `migration_format` fuzz target | Implemented | | `KEEP-MIGRATION-003` | Migration revalidates version-1 head, catalog, pools, root identity (all three coordinates within the migrating process; device and file across restart), and writer authority before mutation | bounded canonical pool inventory in `tests/store_migration_inventory.rs`; writer-locked filesystem pool admission in `filesystem_inventory_*_tests`; exact authority observation and drift refusal in `filesystem_migration_authority_tests`; verification-first execution in `tests/store_migration_execution.rs`; fresh filesystem integration and post-publication drift refusal in `filesystem_migration_storage_tests`; a version-one store still holding a retained stage refuses before the intent is observed in `filesystem_migration_storage_tests` | Implemented | | `KEEP-MIGRATION-004` | Every partial migration prefix continues idempotently under writer authority, comparing only the restart-stable root coordinates (device and file) against the persisted intent | restart-stable coordinate laws in `filesystem_version_two_admission_tests` and complete reopen laws in `filesystem_migration_remount_tests`; `tests/store_migration_recovery.rs` and `tests/store_migration_recovery_order.rs` freeze planner and ordering laws; `src/adapters/store_migration/filesystem_migration_recovery_tests.rs` covers every forward prefix; `src/adapters/store_migration/filesystem_migration_recovery_truncation_tests.rs` covers every strict fixed-stage truncation | Implemented in #108 | -| `KEEP-MIGRATION-005` | Unknown, out-of-order, substituted, corrupt, conflicting, or changed evidence is unrecoverable ambiguity | forward-execution stage preservation, byte-equal inode-substitution, out-of-order-prefix, and post-publication drift laws in `filesystem_migration_storage_tests`; unknown `retention` entries, non-digest namespace directories, and noncanonical pool names refuse before any retention stage is written in `filesystem_retention_namespace_tests`; restart corruption and mutation matrix remains | In progress in #111 | +| `KEEP-MIGRATION-005` | Unknown, out-of-order, substituted, corrupt, conflicting, or changed evidence is unrecoverable ambiguity | Forward-execution laws remain. Filesystem restart record, pair, ordering, root, pool and namespace laws assert exact existing refusal boundaries and preserve complete names, device/inode identities and bytes; see [restart matrix](../../testing-evidence/migration-restart-matrix.md) for scenarios, calibration, diagnostic limits and validation ownership. | Implemented in #111 candidate, including reserved complete-state namespace refusal; final review and integration pending | | `KEEP-MIGRATION-006` | Migration never rewrites or deletes admitted version-1 immutable bytes | exact segment, catalog, and head witnesses in `src/adapters/store_migration/filesystem_migration_storage_tests.rs`, `src/adapters/store_migration/filesystem_migration_recovery_tests.rs`, and `src/adapters/store_migration/filesystem_migration_recovery_truncation_tests.rs`; subprocess restart witnesses in `cargo xtask durability-crash-matrix --sequence migration` | Implemented in #108 | | `KEEP-MIGRATION-007` | Process death around every intent stage, canonical link, namespace prefix, marker stage, receipt stage, cleanup, and synchronization boundary reaches a documented lawful state | ordered phases and capabilities in `tests/store_migration_phase.rs` and `tests/store_migration_storage.rs`; exact phase-failure execution in `tests/store_migration_execution.rs`; production 21-phase forward execution in `filesystem_migration_storage_tests`; `cargo xtask durability-crash-matrix --sequence migration` runs 68 production subprocess cases at `KEEP-CRASH-053..=073`, debug and release | Implemented in #108 | | `KEEP-MIGRATION-008` | Version-1 admission refuses every version-2 or partial-migration artifact after migration begins | `FORMAT` refusal before mutation in `filesystem_migration_authority_tests`; exact version-1 reopen refusal of a migrated root and separate version-2 namespace admission in `filesystem_initialization_namespace`; version-2 reopen returns a distinct `FilesystemVersionTwoAdmission` that no version-1 publisher can consume (pinned by `tests/version_two_admission_contract.rs`), admits every version-2 protocol directory under the Linux profile, and jointly admits the exact marker, intent, and receipt before returning writer authority, with aliased-directory, corrupt, oversized, and mutually inconsistent record refusals in `filesystem_version_two_admission_tests` and `filesystem_platform_profile_tests`; remaining compatibility and fuzz matrix | In progress in #112 | diff --git a/docs/testing-evidence/migration-restart-matrix.md b/docs/testing-evidence/migration-restart-matrix.md new file mode 100644 index 00000000..a8f7c5ad --- /dev/null +++ b/docs/testing-evidence/migration-restart-matrix.md @@ -0,0 +1,69 @@ +# Migration restart ambiguity matrix + +This closure ledger records #111 and KEEP-MIGRATION-005. Change kinds: missing runtime verification and the completed-namespace refusal bug fix documented below. The original test matrix alone changed no production behavior. Owner: `@flyingrobots`. Base: main `6051abb25a9fd33ae7ee0de5614514b709a4d82a`. Implementation and verification in this branch do not constitute mainline integration; the PR records the exact candidate, required checks and integration status. + +## Contract coverage + +| Invariant | Filesystem restart evidence | Disposition | +| --- | --- | --- | +| Corrupt intent, marker and receipt refuse | `filesystem_migration_restart_record_tests` damages each staged/canonical record checksum and requires the exact record-specific decoder variant and expected/observed checksum bytes. | Implemented. Existing codec corruption/fuzz owners retain exhaustive field grammar coverage. | +| Overlong stages are not disposed of as partial records | Record laws require exact `StageOverlong` stage and length for all three stages. | Implemented; existing strict-prefix success tests remain. | +| A receipt must bind its own intent | A valid checksummed receipt from another migrated root refuses with `ReceiptUndecodable` / `IntentDigestMismatch` and both digest coordinates. | Implemented. | +| Canonical/stage pairs bind one inode and identical bytes | `filesystem_migration_restart_pair_tests` replaces each linked stage with equal bytes at another inode, requiring `Adoption` / `ExactRecordRefusal::KindLengthOrIdentity`; contradictory bytes require exact `StageDiffers`. | Implemented; removing inode comparison incorrectly admits the substituted stage and fails the law. | +| Effects require durable intent and prior stage cleanup | `filesystem_migration_restart_order_tests` pins `EffectBeforeIntent` and `StageAfterEffect` with exact stage/effect coordinates through reopened authority. | Implemented. | +| Namespace holes, marker before namespace and receipt before marker refuse | Ordering laws cover filesystem-realizable holes with exact absent/present coordinates and both record-order refusals. | Implemented. An absent parent with a present child is not a realizable filesystem case. | +| Persisted root identity is binding | `filesystem_migration_restart_root_tests` copies the complete interrupted store into a different root and requires `IntentDiffers`. | Implemented; removing the root-file comparison incorrectly recovers the copy. | +| Immutable pools and current HEAD remain authoritative | `filesystem_migration_restart_pool_tests` damages segment/catalog/HEAD magic with exact decoder coordinates, substitutes a valid segment under the wrong digest name, and adds a valid orphan that changes the inventory bound by intent. | Implemented, including precise segment digest mismatch and `IntentDiffers`. | +| Unknown names refuse | Pool laws require exact pool, raw name and canonical-width refusal; namespace laws cover root and every nested migration directory. | Implemented; original root `Namespace` / `InvalidData` and nested `Adoption` / `NamespacePreflight` boundaries remain. | +| Wrong kinds and payload-bearing fences refuse | Namespace laws replace each fixed record/fence with directories and symbolic links, replace protocol directories with files, and require the exact nonempty fence kind/length refusal. | Implemented; existing diagnostic shapes are preserved. | +| Refusal preserves evidence | Every new scenario records root and descendants before reopening and after refusal: names, device/inode, file bytes, directory presence and link targets. Unknown witness shapes fail. | Implemented; deleting HEAD during observation fails the preservation assertion. | +| Success coverage and definition of done remain intact | Existing forward-prefix recovery, strict stage truncation, remount and process-death owners remain. The requirements ledger and normative recovery page link this matrix. | Full validation and exact-head hosted results belong to the PR receipt. Original roadmap checkboxes remain unchanged; merge is separately required. | + +Root namespace admission currently exposes a typed `FilesystemMigrationAuthorityError::Namespace` boundary with `InvalidData`, while its deepest namespace diagnostic is a textual I/O payload. Nested preflight exposes the existing typed `NamespacePreflight` wrapper. These tests pin those existing public failure shapes; this change does not claim to add finer typed namespace coordinates or complete the separate #110 diagnostic audit. + +## Calibration and observed results + +The covered runtime matrix passes in debug and release against the unmodified product. Separately copied source/build mutations demonstrate meaningful RED outcomes: omitted root-file comparison admits a copied root; omitted exact-record inode comparison admits a substituted stage; deleting HEAD during observation fails the preservation witness; skipping recovery namespace preflight changes an unknown-entry refusal from `Adoption` to the later `Resumption` boundary; omitting inventory and derived store-identifier comparison incorrectly recovers changed inventory. None of these mutations is included in the branch. + +Removing only the direct inventory comparison survives because the independently checked store identifier also binds that inventory. This survivor is retained as evidence of redundant protection, not reported as a weak test or hidden behind a mutation score. Removing both parts of that binding makes the orphan law fail with `ambiguous migration restarted successfully`. + +These are calibrations of missing verification, not claimed production bug fixes with fabricated parent RED. The existing single corrupt-intent test was removed under the recorded deletion criterion “subsumed by stronger evidence”: the new record matrix preserves its exact checksum refusal, adds checksum coordinates, and replaces its entry-count-only witness with all names, identities and bytes. Remaining risk lives in the new record law and the complete witness, not in a count assertion. + +## Execution and limits + +Replay `cargo test --lib --all-features --locked filesystem_migration_restart` and its `--release` variant. Tests run in copied Docker sources with pinned Rust 1.96.0 on Linux aarch64. The laws are medium-size real-filesystem experiments using repository-only platform admission to isolate migration semantics. They execute a named forward prefix, drop writer authority, alter owned evidence, then reopen and derive current intent before recovery. This is deterministic restart semantics, not a process-death or physical power-loss campaign; existing crash campaigns retain those claims. + +Schedules are the named forward phase and scenario tuple in each law; no random seed is involved. Each scenario owns fresh storage. Raw execution and calibration logs are retained in the author's issue-111 scratch evidence. The PR records immutable candidate coordinates and actual full validation results; compiler/setup failures are not behavioral RED evidence. + +Before/after equality does not independently prove absence of a transient write restored before observation. The tests combine preserved-state witnesses with the exact refusal boundary; source inspection places observation/planning/adoption refusal before resumed mutation. They do not establish isolation against arbitrary concurrent out-of-band writers. Ordinary-test resource ceilings and exhaustive mutation adequacy are not claimed; the enforcement profile's existing gaps remain. Retire these laws only if the corresponding contract disappears or stronger restart evidence subsumes it. + +## Historical landing counterexample: complete-plan namespace admission + +Landing review on `ff6f5be4b98e985e00601ab3e854a95e87aa9b34` (tree `de06577722b63ba09d425f31dfa7a1acd647e128`) exposed a limit that prevented closing #111 at that head; the correction and remaining acceptance gates are recorded below. After `RemoveReceiptStage`, adding `gc/unexpected` still permits recovery to return success. The new nested-namespace matrix currently tests the earlier `AdmitNamespacePrefix` state. Resumed recovery's pre-mutation refusal remains supported; complete-plan admission is a separate unfulfilled obligation. + +The [isolated probe patch](migration-restart-matrix/complete-namespace-probe.patch) selects only `gc` and the complete prefix in the existing namespace law, leaving production code and its refusal expectation unchanged. Its [runtime RED receipt](migration-restart-matrix/complete-namespace-probe-red.txt) records successful compilation and the executed law failing with `ambiguous migration restarted successfully` (exit 101). Replay in a fresh copy using `git apply --unidiff-zero`, then `cargo test --lib --all-features --locked unknown_root_and_nested_entries_preserve_restart_evidence -- --nocapture`. Execution used Rust 1.96.0, Linux aarch64, actual ext4 scratch and isolated source/build directories. Committed output normalizes only container path prefixes and trailing empty lines; original output and full variant are retained by the author. + +The `Complete` branch returns before `adopt_residue` and nested namespace preflight. Residue observation checks directory presence and kinds, not unknown nested membership. This evidence shows an accepted namespace contradiction, not a recovery mutation or byte substitution. Closure requires complete-plan admission consistent with its owning contract, permanent refusal coverage and preservation of lawful post-migration retention state. Do not indiscriminately apply the incomplete-prefix empty-pool rule to completed stores. A narrower acceptance scope needs explicit maintainer approval; changing prose alone does not close the original requirement. + +## Complete-plan correction under validation + +Regression commit `6351c7c` adds `completed_migration_refuses_unknown_reserved_entries_without_effects`. Its [observed RED](migration-restart-matrix/complete-regression-red.txt) ran with production code from `a8fda647bc67271ec7895577eab55f4b3a5e0085` and only the regression test added; it failed because recovery succeeded. The first source-copy command had a shell variable naming error and ran no test; the corrected isolated copy ran the regression and exited 101. Neither setup trouble nor an absent API is counted as RED. + +The correction invokes the new read-only recovery-port capability `verify_complete` only after the planner accepts the records as `Complete`. The filesystem implementation uses existing version-two namespace admission, with precise `Observation` / `NamespacePreflight` refusal before effects. The new law covers reserved GC, recovery and disposition names with the complete preserved-state witness. `completed_migration_preserves_published_retention_state` performs real retention publication, retries migration, requires a Complete receipt with no forward phases, and reads the exact retained bytes through a public snapshot. The [focused GREEN receipt](migration-restart-matrix/complete-fix-green.txt) records both new laws passing in debug/release and the existing restart matrix passing with the correction. + +This changes production refusal behavior and adds a required method for external `StoreMigrationRecoveryStorage` implementors. Retention content/stage semantics remain owned by retention; complete migration admission neither disposes stages nor certifies their content. No on-disk format, identity or durability ordering changes. The earlier counterexample remains historical evidence, not an unresolved claim about the corrected code. Full stable-candidate validation, the independent review's diagnostic calibration batch and exact-head approval are still required; KEEP-MIGRATION-005 remains in progress until those gates close. + +## Diagnostic calibration closure + +On fixed source `a3ea6c3e2ac58ca82a5c26f43d902ba1ed883be7`, tree `13a6148d8b4ca1fc28b8342cd6a06bc46b9bb2a5`, one finite negative-control batch demonstrates the distinct diagnostic assertions identified by independent review. The real restart runs first, and the unchanged complete witness comparison must pass. Only then does the copied fixture replace the observed error; expected values and assertion code are unchanged. This is oracle calibration at the observation boundary, not a claim that production emitted these artificial errors or that every production guard was separately mutated. + +| Control | Intended failing observation | Evidence | +| --- | --- | --- | +| Wrong diagnostics after real restart | Record checksum/length/foreign receipt, contradictory pair, ordering, HEAD/pool corruption, substituted segment, canonical pool name and root kind/no-follow refusal assertions reach their own failures. The pool identity is preserved here so its deeper decoder assertion executes. | [Patch](migration-restart-matrix/diagnostic.patch), [RED](migration-restart-matrix/diagnostic-red.txt) | +| Preserved outer recovery boundary, wrong inner cause | Root admission remains unchanged so nested cases execute. `Adoption` and `Observation` remain correctly typed while their inner causes change, failing the nested namespace, directory-kind, reader-fence and new completed-state diagnostic checks. | [Patch](migration-restart-matrix/nested-diagnostic.patch), [RED](migration-restart-matrix/nested-diagnostic-red.txt) | +| Wrong immutable-pool identity | Preserve the underlying artifact failure but report the other pool, reaching the pool-identity assertion before decoder checks. | [Patch](migration-restart-matrix/pool-diagnostic.patch), [RED](migration-restart-matrix/pool-diagnostic-red.txt) | +| Incorrect empty-pool completion policy | Substitute partial-migration preflight for completed namespace admission; the new law refuses genuine published retention state instead of returning Complete. | [Patch](migration-restart-matrix/lawful-retention.patch), [RED](migration-restart-matrix/lawful-retention-red.txt) | + +Each accepted control compiled, executed its named runtime law and exited 101. The initial pool control failed compilation on unused imports and is excluded as RED evidence; its corrected isolated build reaches the intended assertion. The first nested control stopped at the outer boundary, so the corrected control preserves that boundary and reaches the inner checks. Original variants and all attempted logs remain retained with the traced launchers; neither rejected attempt is represented as accepted calibration. + +Replay each patch independently in a fresh copied source with `git apply --unidiff-zero` and a separate build directory. Use `cargo test --lib --all-features --locked filesystem_migration_restart -- --nocapture` for the diagnostic control; replace the filter with `filesystem_migration_restart_namespace_tests`, `damaged_immutable_pool_bytes_refuse_restart_before_effects` or `completed_migration_preserves_published_retention_state` respectively for the remaining controls. The [restored GREEN receipt](migration-restart-matrix/calibration-restored-green.txt) records the unchanged fixed tree and both debug/release execution of the restart and completion laws. Logs normalize only isolated container path prefixes and trailing empty lines. Prior root/inode/inventory/preflight/preservation calibrations remain credited; this batch does not create a per-coordinate or per-row adequacy claim. diff --git a/docs/testing-evidence/migration-restart-matrix/calibration-restored-green.txt b/docs/testing-evidence/migration-restart-matrix/calibration-restored-green.txt new file mode 100644 index 00000000..4e67177b --- /dev/null +++ b/docs/testing-evidence/migration-restart-matrix/calibration-restored-green.txt @@ -0,0 +1,81 @@ ++ git rev-parse 'HEAD^{tree}' +13a6148d8b4ca1fc28b8342cd6a06bc46b9bb2a5 ++ cargo test --lib --all-features --locked filesystem_migration_restart + Finished `test` profile [unoptimized + debuginfo] target(s) in 0.03s + Running unittests src/lib.rs (/debug/deps/keep-09abf1fe30bdfc46) + +running 22 tests +test adapters::store_migration::filesystem_migration_restart_pool_tests::a_valid_segment_substitution_reports_both_digest_coordinates ... ok +test adapters::store_migration::filesystem_migration_restart_pool_tests::a_damaged_current_head_preserves_all_restart_evidence ... ok +test adapters::store_migration::filesystem_migration_restart_pool_tests::adding_a_valid_orphan_cannot_reuse_the_previous_migration_intent ... ok +test adapters::store_migration::filesystem_migration_restart_namespace_tests::a_nonempty_reader_fence_preserves_its_exact_refusal ... ok +test adapters::store_migration::filesystem_migration_restart_order_tests::effects_without_durable_intent_preserve_restart_evidence ... ok +test adapters::store_migration::filesystem_migration_restart_pool_tests::damaged_immutable_pool_bytes_refuse_restart_before_effects ... ok +test adapters::store_migration::filesystem_migration_restart_pool_tests::unknown_immutable_pool_names_refuse_with_the_exact_name ... ok +test adapters::store_migration::filesystem_migration_restart_root_tests::a_byte_equal_store_copy_refuses_the_persisted_root_identity ... ok +test adapters::store_migration::filesystem_migration_restart_order_tests::a_marker_before_namespace_completion_preserves_evidence ... ok +test adapters::store_migration::filesystem_migration_restart_order_tests::a_receipt_without_its_marker_preserves_evidence ... ok +test adapters::store_migration::filesystem_migration_restart_order_tests::stages_surviving_later_effects_refuse_restart ... ok +test adapters::store_migration::filesystem_migration_restart_pair_tests::conflicting_stage_bytes_preserve_both_records_on_restart ... ok +test adapters::store_migration::filesystem_migration_restart_pair_tests::byte_equal_stage_substitution_refuses_before_restart_effects ... ok +test adapters::store_migration::filesystem_migration_restart_record_tests::overlong_stages_refuse_restart_without_disposal ... ok +test adapters::store_migration::filesystem_migration_restart_record_tests::a_valid_foreign_receipt_refuses_its_conflicting_intent_digest ... ok +test adapters::store_migration::filesystem_migration_restart_namespace_tests::completed_migration_refuses_unknown_reserved_entries_without_effects ... ok +test adapters::store_migration::filesystem_migration_restart_order_tests::namespace_holes_refuse_restart_at_the_exact_coordinates ... ok +test adapters::store_migration::filesystem_migration_restart_namespace_tests::regular_files_at_protocol_directory_names_refuse_restart ... ok +test adapters::store_migration::filesystem_migration_restart_record_tests::checksum_damage_in_each_migration_record_preserves_restart_evidence ... ok +test adapters::store_migration::filesystem_migration_restart_namespace_tests::unknown_root_and_nested_entries_preserve_restart_evidence ... ok +test adapters::store_migration::filesystem_migration_restart_namespace_tests::directory_substitution_at_each_fixed_file_refuses_restart ... ok +test adapters::store_migration::filesystem_migration_restart_namespace_tests::symbolic_links_at_fixed_record_names_refuse_without_following_them ... ok + +test result: ok. 22 passed; 0 failed; 0 ignored; 0 measured; 354 filtered out; finished in 0.32s + ++ cargo test --lib --all-features --release --locked filesystem_migration_restart + Finished `release` profile [optimized] target(s) in 0.01s + Running unittests src/lib.rs (/release/deps/keep-ab9e64645c5a4507) + +running 22 tests +test adapters::store_migration::filesystem_migration_restart_pool_tests::a_damaged_current_head_preserves_all_restart_evidence ... ok +test adapters::store_migration::filesystem_migration_restart_pool_tests::a_valid_segment_substitution_reports_both_digest_coordinates ... ok +test adapters::store_migration::filesystem_migration_restart_pool_tests::adding_a_valid_orphan_cannot_reuse_the_previous_migration_intent ... ok +test adapters::store_migration::filesystem_migration_restart_namespace_tests::a_nonempty_reader_fence_preserves_its_exact_refusal ... ok +test adapters::store_migration::filesystem_migration_restart_order_tests::effects_without_durable_intent_preserve_restart_evidence ... ok +test adapters::store_migration::filesystem_migration_restart_pool_tests::damaged_immutable_pool_bytes_refuse_restart_before_effects ... ok +test adapters::store_migration::filesystem_migration_restart_pool_tests::unknown_immutable_pool_names_refuse_with_the_exact_name ... ok +test adapters::store_migration::filesystem_migration_restart_root_tests::a_byte_equal_store_copy_refuses_the_persisted_root_identity ... ok +test adapters::store_migration::filesystem_migration_restart_order_tests::a_marker_before_namespace_completion_preserves_evidence ... ok +test adapters::store_migration::filesystem_migration_restart_order_tests::a_receipt_without_its_marker_preserves_evidence ... ok +test adapters::store_migration::filesystem_migration_restart_order_tests::stages_surviving_later_effects_refuse_restart ... ok +test adapters::store_migration::filesystem_migration_restart_pair_tests::conflicting_stage_bytes_preserve_both_records_on_restart ... ok +test adapters::store_migration::filesystem_migration_restart_pair_tests::byte_equal_stage_substitution_refuses_before_restart_effects ... ok +test adapters::store_migration::filesystem_migration_restart_record_tests::overlong_stages_refuse_restart_without_disposal ... ok +test adapters::store_migration::filesystem_migration_restart_record_tests::a_valid_foreign_receipt_refuses_its_conflicting_intent_digest ... ok +test adapters::store_migration::filesystem_migration_restart_namespace_tests::completed_migration_refuses_unknown_reserved_entries_without_effects ... ok +test adapters::store_migration::filesystem_migration_restart_order_tests::namespace_holes_refuse_restart_at_the_exact_coordinates ... ok +test adapters::store_migration::filesystem_migration_restart_namespace_tests::regular_files_at_protocol_directory_names_refuse_restart ... ok +test adapters::store_migration::filesystem_migration_restart_record_tests::checksum_damage_in_each_migration_record_preserves_restart_evidence ... ok +test adapters::store_migration::filesystem_migration_restart_namespace_tests::unknown_root_and_nested_entries_preserve_restart_evidence ... ok +test adapters::store_migration::filesystem_migration_restart_namespace_tests::directory_substitution_at_each_fixed_file_refuses_restart ... ok +test adapters::store_migration::filesystem_migration_restart_namespace_tests::symbolic_links_at_fixed_record_names_refuse_without_following_them ... ok + +test result: ok. 22 passed; 0 failed; 0 ignored; 0 measured; 354 filtered out; finished in 0.24s + ++ cargo test --lib --all-features --locked completed_migration + Finished `test` profile [unoptimized + debuginfo] target(s) in 0.01s + Running unittests src/lib.rs (/debug/deps/keep-09abf1fe30bdfc46) + +running 2 tests +test adapters::retention::filesystem_retention_migration_completion_tests::completed_migration_preserves_published_retention_state ... ok +test adapters::store_migration::filesystem_migration_restart_namespace_tests::completed_migration_refuses_unknown_reserved_entries_without_effects ... ok + +test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 374 filtered out; finished in 0.08s + ++ cargo test --lib --all-features --release --locked completed_migration + Finished `release` profile [optimized] target(s) in 0.01s + Running unittests src/lib.rs (/release/deps/keep-ab9e64645c5a4507) + +running 2 tests +test adapters::retention::filesystem_retention_migration_completion_tests::completed_migration_preserves_published_retention_state ... ok +test adapters::store_migration::filesystem_migration_restart_namespace_tests::completed_migration_refuses_unknown_reserved_entries_without_effects ... ok + +test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 374 filtered out; finished in 0.08s diff --git a/docs/testing-evidence/migration-restart-matrix/complete-fix-green.txt b/docs/testing-evidence/migration-restart-matrix/complete-fix-green.txt new file mode 100644 index 00000000..5c6b4eb9 --- /dev/null +++ b/docs/testing-evidence/migration-restart-matrix/complete-fix-green.txt @@ -0,0 +1,125 @@ ++ cargo test --lib --all-features --locked completed_migration + Compiling rustix v1.1.4 + Compiling io-lifetimes v3.0.1 + Compiling proc-macro2 v1.0.107 + Compiling linux-raw-sys v0.12.1 + Compiling io-lifetimes v2.0.4 + Compiling bitflags v2.13.1 + Compiling io-extras v0.19.0 + Compiling quote v1.0.47 + Compiling unicode-ident v1.0.24 + Compiling shlex v2.0.1 + Compiling cap-primitives v4.0.2 + Compiling find-msvc-tools v0.1.9 + Compiling once_cell v1.21.4 + Compiling ambient-authority v0.0.2 + Compiling libc v0.2.186 + Compiling anstyle v1.0.14 + Compiling cap-std v4.0.2 + Compiling ipnet v2.12.0 + Compiling clap_lex v1.1.0 + Compiling maybe-owned v0.3.4 + Compiling cc v1.3.0 + Compiling cfg-if v1.0.4 + Compiling clap_builder v4.6.2 + Compiling cap-fs-ext v4.0.2 + Compiling condtype v1.3.0 + Compiling arrayvec v0.7.8 + Compiling arrayref v0.3.9 + Compiling constant_time_eq v0.4.2 + Compiling regex-lite v0.1.9 + Compiling allocation-counter v0.8.1 + Compiling syn v2.0.119 + Compiling blake3 v1.8.5 + Compiling clap v4.6.4 + Compiling fs-set-times v0.20.3 + Compiling rustix-linux-procfs v0.1.1 + Compiling divan-macros v0.1.21 + Compiling divan v0.1.21 + Compiling keep v0.0.0 () + Finished `test` profile [unoptimized + debuginfo] target(s) in 5.51s + Running unittests src/lib.rs (/debug/deps/keep-09abf1fe30bdfc46) + +running 2 tests +test adapters::retention::filesystem_retention_migration_completion_tests::completed_migration_preserves_published_retention_state ... ok +test adapters::store_migration::filesystem_migration_restart_namespace_tests::completed_migration_refuses_unknown_reserved_entries_without_effects ... ok + +test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 374 filtered out; finished in 0.21s + ++ cargo test --lib --all-features --release --locked completed_migration + Compiling rustix v1.1.4 + Compiling bitflags v2.13.1 + Compiling linux-raw-sys v0.12.1 + Compiling io-lifetimes v2.0.4 + Compiling io-lifetimes v3.0.1 + Compiling proc-macro2 v1.0.107 + Compiling quote v1.0.47 + Compiling io-extras v0.19.0 + Compiling unicode-ident v1.0.24 + Compiling cap-primitives v4.0.2 + Compiling shlex v2.0.1 + Compiling find-msvc-tools v0.1.9 + Compiling once_cell v1.21.4 + Compiling anstyle v1.0.14 + Compiling clap_lex v1.1.0 + Compiling maybe-owned v0.3.4 + Compiling ambient-authority v0.0.2 + Compiling libc v0.2.186 + Compiling ipnet v2.12.0 + Compiling cap-std v4.0.2 + Compiling cc v1.3.0 + Compiling clap_builder v4.6.2 + Compiling cap-fs-ext v4.0.2 + Compiling cfg-if v1.0.4 + Compiling condtype v1.3.0 + Compiling constant_time_eq v0.4.2 + Compiling arrayref v0.3.9 + Compiling arrayvec v0.7.8 + Compiling regex-lite v0.1.9 + Compiling allocation-counter v0.8.1 + Compiling syn v2.0.119 + Compiling blake3 v1.8.5 + Compiling clap v4.6.4 + Compiling fs-set-times v0.20.3 + Compiling rustix-linux-procfs v0.1.1 + Compiling divan-macros v0.1.21 + Compiling divan v0.1.21 + Compiling keep v0.0.0 () + Finished `release` profile [optimized] target(s) in 6.05s + Running unittests src/lib.rs (/release/deps/keep-ab9e64645c5a4507) + +running 2 tests +test adapters::retention::filesystem_retention_migration_completion_tests::completed_migration_preserves_published_retention_state ... ok +test adapters::store_migration::filesystem_migration_restart_namespace_tests::completed_migration_refuses_unknown_reserved_entries_without_effects ... ok + +test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 374 filtered out; finished in 0.09s + ++ cargo test --lib --all-features --locked filesystem_migration_restart + Finished `test` profile [unoptimized + debuginfo] target(s) in 0.01s + Running unittests src/lib.rs (/debug/deps/keep-09abf1fe30bdfc46) + +running 22 tests +test adapters::store_migration::filesystem_migration_restart_pool_tests::adding_a_valid_orphan_cannot_reuse_the_previous_migration_intent ... ok +test adapters::store_migration::filesystem_migration_restart_pool_tests::a_valid_segment_substitution_reports_both_digest_coordinates ... ok +test adapters::store_migration::filesystem_migration_restart_pool_tests::a_damaged_current_head_preserves_all_restart_evidence ... ok +test adapters::store_migration::filesystem_migration_restart_namespace_tests::a_nonempty_reader_fence_preserves_its_exact_refusal ... ok +test adapters::store_migration::filesystem_migration_restart_order_tests::effects_without_durable_intent_preserve_restart_evidence ... ok +test adapters::store_migration::filesystem_migration_restart_pool_tests::unknown_immutable_pool_names_refuse_with_the_exact_name ... ok +test adapters::store_migration::filesystem_migration_restart_pool_tests::damaged_immutable_pool_bytes_refuse_restart_before_effects ... ok +test adapters::store_migration::filesystem_migration_restart_root_tests::a_byte_equal_store_copy_refuses_the_persisted_root_identity ... ok +test adapters::store_migration::filesystem_migration_restart_order_tests::a_marker_before_namespace_completion_preserves_evidence ... ok +test adapters::store_migration::filesystem_migration_restart_order_tests::a_receipt_without_its_marker_preserves_evidence ... ok +test adapters::store_migration::filesystem_migration_restart_order_tests::stages_surviving_later_effects_refuse_restart ... ok +test adapters::store_migration::filesystem_migration_restart_pair_tests::byte_equal_stage_substitution_refuses_before_restart_effects ... ok +test adapters::store_migration::filesystem_migration_restart_pair_tests::conflicting_stage_bytes_preserve_both_records_on_restart ... ok +test adapters::store_migration::filesystem_migration_restart_record_tests::overlong_stages_refuse_restart_without_disposal ... ok +test adapters::store_migration::filesystem_migration_restart_record_tests::a_valid_foreign_receipt_refuses_its_conflicting_intent_digest ... ok +test adapters::store_migration::filesystem_migration_restart_namespace_tests::completed_migration_refuses_unknown_reserved_entries_without_effects ... ok +test adapters::store_migration::filesystem_migration_restart_order_tests::namespace_holes_refuse_restart_at_the_exact_coordinates ... ok +test adapters::store_migration::filesystem_migration_restart_record_tests::checksum_damage_in_each_migration_record_preserves_restart_evidence ... ok +test adapters::store_migration::filesystem_migration_restart_namespace_tests::regular_files_at_protocol_directory_names_refuse_restart ... ok +test adapters::store_migration::filesystem_migration_restart_namespace_tests::unknown_root_and_nested_entries_preserve_restart_evidence ... ok +test adapters::store_migration::filesystem_migration_restart_namespace_tests::symbolic_links_at_fixed_record_names_refuse_without_following_them ... ok +test adapters::store_migration::filesystem_migration_restart_namespace_tests::directory_substitution_at_each_fixed_file_refuses_restart ... ok + +test result: ok. 22 passed; 0 failed; 0 ignored; 0 measured; 354 filtered out; finished in 0.26s diff --git a/docs/testing-evidence/migration-restart-matrix/complete-namespace-probe-red.txt b/docs/testing-evidence/migration-restart-matrix/complete-namespace-probe-red.txt new file mode 100644 index 00000000..6c1b076b --- /dev/null +++ b/docs/testing-evidence/migration-restart-matrix/complete-namespace-probe-red.txt @@ -0,0 +1,53 @@ + Compiling rustix v1.1.4 + Compiling bitflags v2.13.1 + Compiling io-lifetimes v3.0.1 + Compiling linux-raw-sys v0.12.1 + Compiling proc-macro2 v1.0.107 + Compiling io-lifetimes v2.0.4 + Compiling unicode-ident v1.0.24 + Compiling quote v1.0.47 + Compiling io-extras v0.19.0 + Compiling find-msvc-tools v0.1.9 + Compiling once_cell v1.21.4 + Compiling cap-primitives v4.0.2 + Compiling shlex v2.0.1 + Compiling ambient-authority v0.0.2 + Compiling maybe-owned v0.3.4 + Compiling libc v0.2.186 + Compiling ipnet v2.12.0 + Compiling anstyle v1.0.14 + Compiling clap_lex v1.1.0 + Compiling cap-std v4.0.2 + Compiling cc v1.3.0 + Compiling clap_builder v4.6.2 + Compiling cap-fs-ext v4.0.2 + Compiling cfg-if v1.0.4 + Compiling arrayvec v0.7.8 + Compiling arrayref v0.3.9 + Compiling regex-lite v0.1.9 + Compiling constant_time_eq v0.4.2 + Compiling condtype v1.3.0 + Compiling allocation-counter v0.8.1 + Compiling syn v2.0.119 + Compiling blake3 v1.8.5 + Compiling clap v4.6.4 + Compiling fs-set-times v0.20.3 + Compiling rustix-linux-procfs v0.1.1 + Compiling divan-macros v0.1.21 + Compiling divan v0.1.21 + Compiling keep v0.0.0 () + Finished `test` profile [unoptimized + debuginfo] target(s) in 5.47s + Running unittests src/lib.rs (/debug/deps/keep-09abf1fe30bdfc46) + +running 1 test +Error: "ambiguous migration restarted successfully" +test adapters::store_migration::filesystem_migration_restart_namespace_tests::unknown_root_and_nested_entries_preserve_restart_evidence ... FAILED + +failures: + +failures: + adapters::store_migration::filesystem_migration_restart_namespace_tests::unknown_root_and_nested_entries_preserve_restart_evidence + +test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 373 filtered out; finished in 0.03s + +error: test failed, to rerun pass `--lib` diff --git a/docs/testing-evidence/migration-restart-matrix/complete-namespace-probe.patch b/docs/testing-evidence/migration-restart-matrix/complete-namespace-probe.patch new file mode 100644 index 00000000..a7b6db16 --- /dev/null +++ b/docs/testing-evidence/migration-restart-matrix/complete-namespace-probe.patch @@ -0,0 +1,17 @@ +diff --git a/src/adapters/store_migration/filesystem_migration_restart_namespace_tests.rs b/src/adapters/store_migration/filesystem_migration_restart_namespace_tests.rs +--- a/src/adapters/store_migration/filesystem_migration_restart_namespace_tests.rs ++++ b/src/adapters/store_migration/filesystem_migration_restart_namespace_tests.rs +@@ -15,9 +15 @@ fn unknown_root_and_nested_entries_preserve_restart_evidence() -> Result<(), Box +- for parent in [ +- "", +- "retention", +- "retention/roots", +- "retention/manifests", +- "gc", +- "recovery", +- "recovery/dispositions", +- ] { ++ for parent in ["gc"] { +@@ -26 +18 @@ fn unknown_root_and_nested_entries_preserve_restart_evidence() -> Result<(), Box +- Phase::AdmitNamespacePrefix, ++ Phase::RemoveReceiptStage, diff --git a/docs/testing-evidence/migration-restart-matrix/complete-regression-red.txt b/docs/testing-evidence/migration-restart-matrix/complete-regression-red.txt new file mode 100644 index 00000000..ab59e698 --- /dev/null +++ b/docs/testing-evidence/migration-restart-matrix/complete-regression-red.txt @@ -0,0 +1,53 @@ + Compiling rustix v1.1.4 + Compiling io-lifetimes v2.0.4 + Compiling bitflags v2.13.1 + Compiling linux-raw-sys v0.12.1 + Compiling proc-macro2 v1.0.107 + Compiling io-lifetimes v3.0.1 + Compiling io-extras v0.19.0 + Compiling unicode-ident v1.0.24 + Compiling quote v1.0.47 + Compiling cap-primitives v4.0.2 + Compiling once_cell v1.21.4 + Compiling shlex v2.0.1 + Compiling find-msvc-tools v0.1.9 + Compiling libc v0.2.186 + Compiling anstyle v1.0.14 + Compiling maybe-owned v0.3.4 + Compiling cap-std v4.0.2 + Compiling ipnet v2.12.0 + Compiling ambient-authority v0.0.2 + Compiling cc v1.3.0 + Compiling clap_lex v1.1.0 + Compiling cap-fs-ext v4.0.2 + Compiling cfg-if v1.0.4 + Compiling constant_time_eq v0.4.2 + Compiling condtype v1.3.0 + Compiling regex-lite v0.1.9 + Compiling arrayref v0.3.9 + Compiling arrayvec v0.7.8 + Compiling clap_builder v4.6.2 + Compiling allocation-counter v0.8.1 + Compiling syn v2.0.119 + Compiling blake3 v1.8.5 + Compiling clap v4.6.4 + Compiling rustix-linux-procfs v0.1.1 + Compiling fs-set-times v0.20.3 + Compiling divan-macros v0.1.21 + Compiling divan v0.1.21 + Compiling keep v0.0.0 () + Finished `test` profile [unoptimized + debuginfo] target(s) in 5.63s + Running unittests src/lib.rs (/debug/deps/keep-09abf1fe30bdfc46) + +running 1 test +test adapters::store_migration::filesystem_migration_restart_namespace_tests::completed_migration_refuses_unknown_reserved_entries_without_effects ... FAILED + +failures: + +failures: + adapters::store_migration::filesystem_migration_restart_namespace_tests::completed_migration_refuses_unknown_reserved_entries_without_effects + +test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 374 filtered out; finished in 0.03s + +Error: "ambiguous migration restarted successfully" +error: test failed, to rerun pass `--lib` diff --git a/docs/testing-evidence/migration-restart-matrix/diagnostic-red.txt b/docs/testing-evidence/migration-restart-matrix/diagnostic-red.txt new file mode 100644 index 00000000..32a8d7cf --- /dev/null +++ b/docs/testing-evidence/migration-restart-matrix/diagnostic-red.txt @@ -0,0 +1,163 @@ + Compiling rustix v1.1.4 + Compiling proc-macro2 v1.0.107 + Compiling bitflags v2.13.1 + Compiling linux-raw-sys v0.12.1 + Compiling io-lifetimes v2.0.4 + Compiling io-lifetimes v3.0.1 + Compiling io-extras v0.19.0 + Compiling quote v1.0.47 + Compiling unicode-ident v1.0.24 + Compiling shlex v2.0.1 + Compiling cap-primitives v4.0.2 + Compiling once_cell v1.21.4 + Compiling find-msvc-tools v0.1.9 + Compiling ipnet v2.12.0 + Compiling anstyle v1.0.14 + Compiling maybe-owned v0.3.4 + Compiling clap_lex v1.1.0 + Compiling ambient-authority v0.0.2 + Compiling libc v0.2.186 + Compiling cap-std v4.0.2 + Compiling cap-fs-ext v4.0.2 + Compiling cc v1.3.0 + Compiling clap_builder v4.6.2 + Compiling cfg-if v1.0.4 + Compiling constant_time_eq v0.4.2 + Compiling arrayvec v0.7.8 + Compiling arrayref v0.3.9 + Compiling regex-lite v0.1.9 + Compiling condtype v1.3.0 + Compiling allocation-counter v0.8.1 + Compiling syn v2.0.119 + Compiling blake3 v1.8.5 + Compiling clap v4.6.4 + Compiling rustix-linux-procfs v0.1.1 + Compiling fs-set-times v0.20.3 + Compiling divan-macros v0.1.21 + Compiling divan v0.1.21 + Compiling keep v0.0.0 () + Finished `test` profile [unoptimized + debuginfo] target(s) in 7.37s + Running unittests src/lib.rs (/debug/deps/keep-09abf1fe30bdfc46) + +running 22 tests + +thread 'adapters::store_migration::filesystem_migration_restart_order_tests::effects_without_durable_intent_preserve_restart_evidence' (1987485) panicked at src/adapters/store_migration/filesystem_migration_restart_order_tests.rs:26:9: +reader.lock: Ambiguity { source: StageOverlong { stage: Intent, observed: 0 } } +note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace +test adapters::store_migration::filesystem_migration_restart_order_tests::effects_without_durable_intent_preserve_restart_evidence ... FAILED + +thread 'adapters::store_migration::filesystem_migration_restart_order_tests::stages_surviving_later_effects_refuse_restart' (1987487) panicked at src/adapters/store_migration/filesystem_migration_restart_order_tests.rs:65:9: +reader.lock: Ambiguity { source: StageOverlong { stage: Intent, observed: 0 } } + +thread 'adapters::store_migration::filesystem_migration_restart_pair_tests::byte_equal_stage_substitution_refuses_before_restart_effects' (1987488) panicked at src/adapters/store_migration/filesystem_migration_restart_pair_tests.rs:26:9: +migration.intent.next: Ambiguity { source: StageOverlong { stage: Intent, observed: 0 } } +test adapters::store_migration::filesystem_migration_restart_order_tests::stages_surviving_later_effects_refuse_restart ... FAILED +test adapters::store_migration::filesystem_migration_restart_pair_tests::byte_equal_stage_substitution_refuses_before_restart_effects ... FAILED + +thread 'adapters::store_migration::filesystem_migration_restart_pair_tests::conflicting_stage_bytes_preserve_both_records_on_restart' (1987489) panicked at src/adapters/store_migration/filesystem_migration_restart_pair_tests.rs:53:9: +migration.intent.next: Ambiguity { source: StageOverlong { stage: Intent, observed: 0 } } +test adapters::store_migration::filesystem_migration_restart_pair_tests::conflicting_stage_bytes_preserve_both_records_on_restart ... FAILED + +thread 'adapters::store_migration::filesystem_migration_restart_pool_tests::a_damaged_current_head_preserves_all_restart_evidence' (1987490) panicked at src/adapters/store_migration/filesystem_migration_restart_pool_tests.rs:74:5: +HEAD refusal lost its decoder coordinate: Head { source: InvalidMagic { observed: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] } } + +thread 'adapters::store_migration::filesystem_migration_restart_pool_tests::a_valid_segment_substitution_reports_both_digest_coordinates' (1987491) panicked at src/adapters/store_migration/filesystem_migration_restart_pool_tests.rs:120:5: +Inventory { source: Artifact { pool: Segments, name: RecoveryEntryName { bytes: [98, 55, 53, 52, 50, 100, 99, 101, 100, 50, 97, 98, 55, 55, 48, 56, 57, 52, 97, 49, 52, 100, 49, 100, 48, 52, 98, 48, 54, 54, 101, 51, 97, 56, 57, 57, 57, 52, 50, 54, 48, 50, 99, 53, 57, 56, 54, 100, 51, 53, 98, 97, 54, 100, 102, 54, 99, 49, 97, 51, 53, 99, 102, 99, 46, 115, 101, 103] }, source: Catalog { source: InvalidMagic { observed: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] } } } } +test adapters::store_migration::filesystem_migration_restart_pool_tests::a_damaged_current_head_preserves_all_restart_evidence ... FAILED +test adapters::store_migration::filesystem_migration_restart_pool_tests::a_valid_segment_substitution_reports_both_digest_coordinates ... FAILED + +thread 'adapters::store_migration::filesystem_migration_restart_pool_tests::damaged_immutable_pool_bytes_refuse_restart_before_effects' (1987493) panicked at src/adapters/store_migration/filesystem_migration_restart_pool_tests.rs:46:9: +segments: exact decoder failure lost: Inventory { source: Artifact { pool: Segments, name: RecoveryEntryName { bytes: [98, 55, 53, 52, 50, 100, 99, 101, 100, 50, 97, 98, 55, 55, 48, 56, 57, 52, 97, 49, 52, 100, 49, 100, 48, 52, 98, 48, 54, 54, 101, 51, 97, 56, 57, 57, 57, 52, 50, 54, 48, 50, 99, 53, 57, 56, 54, 100, 51, 53, 98, 97, 54, 100, 102, 54, 99, 49, 97, 51, 53, 99, 102, 99, 46, 115, 101, 103] }, source: Catalog { source: InvalidMagic { observed: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] } } } } + +thread 'adapters::store_migration::filesystem_migration_restart_pool_tests::unknown_immutable_pool_names_refuse_with_the_exact_name' (1987494) panicked at src/adapters/store_migration/filesystem_migration_restart_pool_tests.rs:183:9: +segments: Inventory { source: Name { pool: Segments, name: RecoveryEntryName { bytes: [117, 110, 107, 110, 111, 119, 110] }, source: WrongLength { expected: 0, observed: 0 } } } +test adapters::store_migration::filesystem_migration_restart_pool_tests::damaged_immutable_pool_bytes_refuse_restart_before_effects ... FAILED +test adapters::store_migration::filesystem_migration_restart_pool_tests::unknown_immutable_pool_names_refuse_with_the_exact_name ... FAILED + +thread 'adapters::store_migration::filesystem_migration_restart_pool_tests::adding_a_valid_orphan_cannot_reuse_the_previous_migration_intent' (1987492) panicked at src/adapters/store_migration/filesystem_migration_restart_pool_tests.rs:152:5: +changed immutable inventory must not inherit old intent: Ambiguity { source: StageOverlong { stage: Intent, observed: 0 } } +test adapters::store_migration::filesystem_migration_restart_pool_tests::adding_a_valid_orphan_cannot_reuse_the_previous_migration_intent ... FAILED + +thread 'adapters::store_migration::filesystem_migration_restart_record_tests::overlong_stages_refuse_restart_without_disposal' (1987497) panicked at src/adapters/store_migration/filesystem_migration_restart_record_tests.rs:127:9: +migration.intent.next: Ambiguity { source: StageOverlong { stage: Intent, observed: 0 } } +test adapters::store_migration::filesystem_migration_restart_record_tests::overlong_stages_refuse_restart_without_disposal ... FAILED + +thread 'adapters::store_migration::filesystem_migration_restart_record_tests::checksum_damage_in_each_migration_record_preserves_restart_evidence' (1987496) panicked at src/adapters/store_migration/filesystem_migration_restart_record_tests.rs:36:9: +assertion `left == right` failed: migration.intent.next: Ambiguity { source: StageOverlong { stage: Intent, observed: 0 } } + left: None + right: Some(([182, 113, 5, 102, 219, 211, 120, 126, 63, 118, 217, 159, 102, 152, 55, 32, 172, 64, 162, 43, 161, 87, 105, 166, 106, 171, 31, 54, 66, 194, 148, 89], [182, 113, 5, 102, 219, 211, 120, 126, 63, 118, 217, 159, 102, 152, 55, 32, 172, 64, 162, 43, 161, 87, 105, 166, 106, 171, 31, 54, 66, 194, 148, 88])) +test adapters::store_migration::filesystem_migration_restart_record_tests::checksum_damage_in_each_migration_record_preserves_restart_evidence ... FAILED + +thread 'adapters::store_migration::filesystem_migration_restart_namespace_tests::a_nonempty_reader_fence_preserves_its_exact_refusal' (1987477) panicked at src/adapters/store_migration/filesystem_migration_restart_namespace_tests.rs:121:5: +Ambiguity { source: StageOverlong { stage: Intent, observed: 0 } } +test adapters::store_migration::filesystem_migration_restart_namespace_tests::a_nonempty_reader_fence_preserves_its_exact_refusal ... FAILED + +thread 'adapters::store_migration::filesystem_migration_restart_root_tests::a_byte_equal_store_copy_refuses_the_persisted_root_identity' (1987498) panicked at src/adapters/store_migration/filesystem_migration_restart_root_tests.rs:20:5: +copied root must not inherit migration authority: Ambiguity { source: StageOverlong { stage: Intent, observed: 0 } } +test adapters::store_migration::filesystem_migration_restart_root_tests::a_byte_equal_store_copy_refuses_the_persisted_root_identity ... FAILED + +thread 'adapters::store_migration::filesystem_migration_restart_namespace_tests::unknown_root_and_nested_entries_preserve_restart_evidence' (1987482) panicked at src/adapters/store_migration/filesystem_migration_restart_namespace_tests.rs:132:5: +namespace refusal lost its typed boundary: Namespace { source: Custom { kind: PermissionDenied, error: "calibrated wrong namespace cause" } } + +thread 'adapters::store_migration::filesystem_migration_restart_namespace_tests::regular_files_at_protocol_directory_names_refuse_restart' (1987480) panicked at src/adapters/store_migration/filesystem_migration_restart_namespace_tests.rs:132:5: +namespace refusal lost its typed boundary: Namespace { source: Custom { kind: PermissionDenied, error: "calibrated wrong namespace cause" } } +test adapters::store_migration::filesystem_migration_restart_namespace_tests::unknown_root_and_nested_entries_preserve_restart_evidence ... FAILED +test adapters::store_migration::filesystem_migration_restart_namespace_tests::regular_files_at_protocol_directory_names_refuse_restart ... FAILED + +thread 'adapters::store_migration::filesystem_migration_restart_order_tests::namespace_holes_refuse_restart_at_the_exact_coordinates' (1987486) panicked at src/adapters/store_migration/filesystem_migration_restart_order_tests.rs:98:9: +reader.lock: Ambiguity { source: StageOverlong { stage: Intent, observed: 0 } } +test adapters::store_migration::filesystem_migration_restart_order_tests::namespace_holes_refuse_restart_at_the_exact_coordinates ... FAILED + +thread 'adapters::store_migration::filesystem_migration_restart_order_tests::a_marker_before_namespace_completion_preserves_evidence' (1987483) panicked at src/adapters/store_migration/filesystem_migration_restart_order_tests.rs:117:5: +Ambiguity { source: StageOverlong { stage: Intent, observed: 0 } } +test adapters::store_migration::filesystem_migration_restart_order_tests::a_marker_before_namespace_completion_preserves_evidence ... FAILED +test adapters::store_migration::filesystem_migration_restart_order_tests::a_receipt_without_its_marker_preserves_evidence ... FAILED + +thread 'adapters::store_migration::filesystem_migration_restart_order_tests::a_receipt_without_its_marker_preserves_evidence' (1987484) panicked at src/adapters/store_migration/filesystem_migration_restart_order_tests.rs:137:5: +Ambiguity { source: StageOverlong { stage: Intent, observed: 0 } } + +thread 'adapters::store_migration::filesystem_migration_restart_namespace_tests::symbolic_links_at_fixed_record_names_refuse_without_following_them' (1987481) panicked at src/adapters/store_migration/filesystem_migration_restart_namespace_tests.rs:132:5: +namespace refusal lost its typed boundary: Namespace { source: Custom { kind: PermissionDenied, error: "calibrated wrong namespace cause" } } +test adapters::store_migration::filesystem_migration_restart_namespace_tests::symbolic_links_at_fixed_record_names_refuse_without_following_them ... FAILED + +thread 'adapters::store_migration::filesystem_migration_restart_namespace_tests::completed_migration_refuses_unknown_reserved_entries_without_effects' (1987478) panicked at src/adapters/store_migration/filesystem_migration_restart_namespace_tests.rs:181:9: +completed migration admitted invalid gc: Ambiguity { source: StageOverlong { stage: Intent, observed: 0 } } +test adapters::store_migration::filesystem_migration_restart_namespace_tests::completed_migration_refuses_unknown_reserved_entries_without_effects ... FAILED +test adapters::store_migration::filesystem_migration_restart_namespace_tests::directory_substitution_at_each_fixed_file_refuses_restart ... FAILED + +thread 'adapters::store_migration::filesystem_migration_restart_namespace_tests::directory_substitution_at_each_fixed_file_refuses_restart' (1987479) panicked at src/adapters/store_migration/filesystem_migration_restart_namespace_tests.rs:132:5: +namespace refusal lost its typed boundary: Namespace { source: Custom { kind: PermissionDenied, error: "calibrated wrong namespace cause" } } + +thread 'adapters::store_migration::filesystem_migration_restart_record_tests::a_valid_foreign_receipt_refuses_its_conflicting_intent_digest' (1987495) panicked at src/adapters/store_migration/filesystem_migration_restart_record_tests.rs:154:5: +Ambiguity { source: StageOverlong { stage: Intent, observed: 0 } } +test adapters::store_migration::filesystem_migration_restart_record_tests::a_valid_foreign_receipt_refuses_its_conflicting_intent_digest ... FAILED + +failures: + +failures: + adapters::store_migration::filesystem_migration_restart_namespace_tests::a_nonempty_reader_fence_preserves_its_exact_refusal + adapters::store_migration::filesystem_migration_restart_namespace_tests::completed_migration_refuses_unknown_reserved_entries_without_effects + adapters::store_migration::filesystem_migration_restart_namespace_tests::directory_substitution_at_each_fixed_file_refuses_restart + adapters::store_migration::filesystem_migration_restart_namespace_tests::regular_files_at_protocol_directory_names_refuse_restart + adapters::store_migration::filesystem_migration_restart_namespace_tests::symbolic_links_at_fixed_record_names_refuse_without_following_them + adapters::store_migration::filesystem_migration_restart_namespace_tests::unknown_root_and_nested_entries_preserve_restart_evidence + adapters::store_migration::filesystem_migration_restart_order_tests::a_marker_before_namespace_completion_preserves_evidence + adapters::store_migration::filesystem_migration_restart_order_tests::a_receipt_without_its_marker_preserves_evidence + adapters::store_migration::filesystem_migration_restart_order_tests::effects_without_durable_intent_preserve_restart_evidence + adapters::store_migration::filesystem_migration_restart_order_tests::namespace_holes_refuse_restart_at_the_exact_coordinates + adapters::store_migration::filesystem_migration_restart_order_tests::stages_surviving_later_effects_refuse_restart + adapters::store_migration::filesystem_migration_restart_pair_tests::byte_equal_stage_substitution_refuses_before_restart_effects + adapters::store_migration::filesystem_migration_restart_pair_tests::conflicting_stage_bytes_preserve_both_records_on_restart + adapters::store_migration::filesystem_migration_restart_pool_tests::a_damaged_current_head_preserves_all_restart_evidence + adapters::store_migration::filesystem_migration_restart_pool_tests::a_valid_segment_substitution_reports_both_digest_coordinates + adapters::store_migration::filesystem_migration_restart_pool_tests::adding_a_valid_orphan_cannot_reuse_the_previous_migration_intent + adapters::store_migration::filesystem_migration_restart_pool_tests::damaged_immutable_pool_bytes_refuse_restart_before_effects + adapters::store_migration::filesystem_migration_restart_pool_tests::unknown_immutable_pool_names_refuse_with_the_exact_name + adapters::store_migration::filesystem_migration_restart_record_tests::a_valid_foreign_receipt_refuses_its_conflicting_intent_digest + adapters::store_migration::filesystem_migration_restart_record_tests::checksum_damage_in_each_migration_record_preserves_restart_evidence + adapters::store_migration::filesystem_migration_restart_record_tests::overlong_stages_refuse_restart_without_disposal + adapters::store_migration::filesystem_migration_restart_root_tests::a_byte_equal_store_copy_refuses_the_persisted_root_identity + +test result: FAILED. 0 passed; 22 failed; 0 ignored; 0 measured; 354 filtered out; finished in 0.10s + +error: test failed, to rerun pass `--lib` diff --git a/docs/testing-evidence/migration-restart-matrix/diagnostic.patch b/docs/testing-evidence/migration-restart-matrix/diagnostic.patch new file mode 100644 index 00000000..1be2c68b --- /dev/null +++ b/docs/testing-evidence/migration-restart-matrix/diagnostic.patch @@ -0,0 +1,53 @@ +diff --git a/src/adapters/store_migration/filesystem_migration_restart_test_fixture.rs b/src/adapters/store_migration/filesystem_migration_restart_test_fixture.rs +--- a/src/adapters/store_migration/filesystem_migration_restart_test_fixture.rs ++++ b/src/adapters/store_migration/filesystem_migration_restart_test_fixture.rs +@@ -41 +41 @@ pub(super) fn refusal(root: &Path) -> Result, Box> { +- Ok(error) ++ Ok(wrong_diagnostic(error)) +@@ -89,0 +90,46 @@ fn visit(root: &Path, relative: &Path, entries: &mut Witness) -> Result<(), Box< ++ ++// Calibration only: replace the observed refusal after the real restart and ++// unchanged complete witness check, preserving relevant outer error boundaries. ++fn wrong_diagnostic(error: Box) -> Box { ++ use super::{ ++ FilesystemMigrationAuthorityError as Authority, FilesystemMigrationInventoryError as Inventory, ++ StoreMigrationFixedStage, StoreMigrationRecoveryAmbiguity, StoreMigrationRecoveryError, ++ }; ++ use crate::adapters::{CatalogDecodeError, CatalogRestartError, PublicationHeadDecodeError, RecoveryPoolNameError}; ++ let error = match error.downcast::() { ++ Ok(_) => return Box::new(StoreMigrationRecoveryError::Ambiguity { ++ source: StoreMigrationRecoveryAmbiguity::StageOverlong { ++ stage: StoreMigrationFixedStage::Intent, observed: 0, ++ }, ++ }), ++ Err(error) => error, ++ }; ++ let authority = match error.downcast::() { ++ Ok(authority) => *authority, ++ Err(error) => return error, ++ }; ++ Box::new(match authority { ++ Authority::Namespace { .. } => Authority::Namespace { ++ source: std::io::Error::new(std::io::ErrorKind::PermissionDenied, "calibrated wrong namespace cause"), ++ }, ++ Authority::Head { .. } => Authority::Head { ++ source: PublicationHeadDecodeError::InvalidMagic { observed: [0; 16] }, ++ }, ++ Authority::Inventory { source } => Authority::Inventory { ++ source: match source { ++ Inventory::Artifact { pool, name, .. } => Inventory::Artifact { ++ pool, name, ++ source: Box::new(CatalogRestartError::Catalog { ++ source: CatalogDecodeError::InvalidMagic { observed: [0; 16] }, ++ }), ++ }, ++ Inventory::Name { pool, name, .. } => Inventory::Name { ++ pool, name, ++ source: RecoveryPoolNameError::WrongLength { expected: 0, observed: 0 }, ++ }, ++ source => source, ++ }, ++ }, ++ authority => authority, ++ }) ++} diff --git a/docs/testing-evidence/migration-restart-matrix/lawful-retention-red.txt b/docs/testing-evidence/migration-restart-matrix/lawful-retention-red.txt new file mode 100644 index 00000000..dff17138 --- /dev/null +++ b/docs/testing-evidence/migration-restart-matrix/lawful-retention-red.txt @@ -0,0 +1,53 @@ + Compiling rustix v1.1.4 + Compiling bitflags v2.13.1 + Compiling io-lifetimes v2.0.4 + Compiling io-lifetimes v3.0.1 + Compiling proc-macro2 v1.0.107 + Compiling linux-raw-sys v0.12.1 + Compiling quote v1.0.47 + Compiling unicode-ident v1.0.24 + Compiling io-extras v0.19.0 + Compiling once_cell v1.21.4 + Compiling cap-primitives v4.0.2 + Compiling shlex v2.0.1 + Compiling find-msvc-tools v0.1.9 + Compiling ipnet v2.12.0 + Compiling libc v0.2.186 + Compiling cap-std v4.0.2 + Compiling maybe-owned v0.3.4 + Compiling clap_lex v1.1.0 + Compiling anstyle v1.0.14 + Compiling cc v1.3.0 + Compiling ambient-authority v0.0.2 + Compiling cfg-if v1.0.4 + Compiling clap_builder v4.6.2 + Compiling cap-fs-ext v4.0.2 + Compiling constant_time_eq v0.4.2 + Compiling regex-lite v0.1.9 + Compiling arrayref v0.3.9 + Compiling arrayvec v0.7.8 + Compiling condtype v1.3.0 + Compiling allocation-counter v0.8.1 + Compiling syn v2.0.119 + Compiling blake3 v1.8.5 + Compiling clap v4.6.4 + Compiling fs-set-times v0.20.3 + Compiling rustix-linux-procfs v0.1.1 + Compiling divan-macros v0.1.21 + Compiling divan v0.1.21 + Compiling keep v0.0.0 () + Finished `test` profile [unoptimized + debuginfo] target(s) in 5.77s + Running unittests src/lib.rs (/debug/deps/keep-09abf1fe30bdfc46) + +running 1 test +Error: Observation { source: Custom { kind: InvalidData, error: NamespacePreflight { source: Custom { kind: InvalidData, error: "migration namespace contains an unknown entry" } } } } +test adapters::retention::filesystem_retention_migration_completion_tests::completed_migration_preserves_published_retention_state ... FAILED + +failures: + +failures: + adapters::retention::filesystem_retention_migration_completion_tests::completed_migration_preserves_published_retention_state + +test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 375 filtered out; finished in 0.08s + +error: test failed, to rerun pass `--lib` diff --git a/docs/testing-evidence/migration-restart-matrix/lawful-retention.patch b/docs/testing-evidence/migration-restart-matrix/lawful-retention.patch new file mode 100644 index 00000000..56c7483d --- /dev/null +++ b/docs/testing-evidence/migration-restart-matrix/lawful-retention.patch @@ -0,0 +1,6 @@ +diff --git a/src/adapters/store_migration/filesystem_migration_recovery.rs b/src/adapters/store_migration/filesystem_migration_recovery.rs +--- a/src/adapters/store_migration/filesystem_migration_recovery.rs ++++ b/src/adapters/store_migration/filesystem_migration_recovery.rs +@@ -87 +87 @@ impl StoreMigrationRecoveryStorage for FilesystemStoreMigrationAuthority { +- filesystem_initialization_namespace::admit_version_two(self.root()) ++ super::filesystem_migration_namespace::preflight_recovery_directories(self.root()) diff --git a/docs/testing-evidence/migration-restart-matrix/nested-diagnostic-red.txt b/docs/testing-evidence/migration-restart-matrix/nested-diagnostic-red.txt new file mode 100644 index 00000000..8f096ddb --- /dev/null +++ b/docs/testing-evidence/migration-restart-matrix/nested-diagnostic-red.txt @@ -0,0 +1,73 @@ + Compiling rustix v1.1.4 + Compiling proc-macro2 v1.0.107 + Compiling linux-raw-sys v0.12.1 + Compiling io-lifetimes v2.0.4 + Compiling bitflags v2.13.1 + Compiling io-lifetimes v3.0.1 + Compiling io-extras v0.19.0 + Compiling unicode-ident v1.0.24 + Compiling quote v1.0.47 + Compiling once_cell v1.21.4 + Compiling find-msvc-tools v0.1.9 + Compiling cap-primitives v4.0.2 + Compiling shlex v2.0.1 + Compiling maybe-owned v0.3.4 + Compiling ambient-authority v0.0.2 + Compiling anstyle v1.0.14 + Compiling cap-std v4.0.2 + Compiling ipnet v2.12.0 + Compiling clap_lex v1.1.0 + Compiling libc v0.2.186 + Compiling cc v1.3.0 + Compiling cfg-if v1.0.4 + Compiling cap-fs-ext v4.0.2 + Compiling regex-lite v0.1.9 + Compiling condtype v1.3.0 + Compiling arrayref v0.3.9 + Compiling arrayvec v0.7.8 + Compiling clap_builder v4.6.2 + Compiling constant_time_eq v0.4.2 + Compiling allocation-counter v0.8.1 + Compiling syn v2.0.119 + Compiling blake3 v1.8.5 + Compiling clap v4.6.4 + Compiling rustix-linux-procfs v0.1.1 + Compiling fs-set-times v0.20.3 + Compiling divan-macros v0.1.21 + Compiling divan v0.1.21 + Compiling keep v0.0.0 () + Finished `test` profile [unoptimized + debuginfo] target(s) in 5.61s + Running unittests src/lib.rs (/debug/deps/keep-09abf1fe30bdfc46) + +running 6 tests + +thread 'adapters::store_migration::filesystem_migration_restart_namespace_tests::a_nonempty_reader_fence_preserves_its_exact_refusal' (1998679) panicked at src/adapters/store_migration/filesystem_migration_restart_namespace_tests.rs:121:5: +Observation { source: Custom { kind: Other, error: "calibrated wrong observation cause" } } +note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace +test adapters::store_migration::filesystem_migration_restart_namespace_tests::a_nonempty_reader_fence_preserves_its_exact_refusal ... FAILED + +thread 'adapters::store_migration::filesystem_migration_restart_namespace_tests::completed_migration_refuses_unknown_reserved_entries_without_effects' (1998680) panicked at src/adapters/store_migration/filesystem_migration_restart_namespace_tests.rs:181:9: +completed migration admitted invalid gc: Observation { source: Custom { kind: Other, error: "calibrated wrong observation cause" } } +test adapters::store_migration::filesystem_migration_restart_namespace_tests::completed_migration_refuses_unknown_reserved_entries_without_effects ... FAILED + +thread 'adapters::store_migration::filesystem_migration_restart_namespace_tests::unknown_root_and_nested_entries_preserve_restart_evidence' (1998684) panicked at src/adapters/store_migration/filesystem_migration_restart_namespace_tests.rs:39:13: +retention: Adoption { source: Custom { kind: Other, error: "calibrated wrong adoption cause" } } +test adapters::store_migration::filesystem_migration_restart_namespace_tests::unknown_root_and_nested_entries_preserve_restart_evidence ... FAILED + +thread 'adapters::store_migration::filesystem_migration_restart_namespace_tests::regular_files_at_protocol_directory_names_refuse_restart' (1998682) panicked at src/adapters/store_migration/filesystem_migration_restart_namespace_tests.rs:100:13: +retention/roots: Observation { source: Custom { kind: Other, error: "calibrated wrong observation cause" } } +test adapters::store_migration::filesystem_migration_restart_namespace_tests::regular_files_at_protocol_directory_names_refuse_restart ... FAILED +test adapters::store_migration::filesystem_migration_restart_namespace_tests::directory_substitution_at_each_fixed_file_refuses_restart ... ok +test adapters::store_migration::filesystem_migration_restart_namespace_tests::symbolic_links_at_fixed_record_names_refuse_without_following_them ... ok + +failures: + +failures: + adapters::store_migration::filesystem_migration_restart_namespace_tests::a_nonempty_reader_fence_preserves_its_exact_refusal + adapters::store_migration::filesystem_migration_restart_namespace_tests::completed_migration_refuses_unknown_reserved_entries_without_effects + adapters::store_migration::filesystem_migration_restart_namespace_tests::regular_files_at_protocol_directory_names_refuse_restart + adapters::store_migration::filesystem_migration_restart_namespace_tests::unknown_root_and_nested_entries_preserve_restart_evidence + +test result: FAILED. 2 passed; 4 failed; 0 ignored; 0 measured; 370 filtered out; finished in 0.79s + +error: test failed, to rerun pass `--lib` diff --git a/docs/testing-evidence/migration-restart-matrix/nested-diagnostic.patch b/docs/testing-evidence/migration-restart-matrix/nested-diagnostic.patch new file mode 100644 index 00000000..417e6eaf --- /dev/null +++ b/docs/testing-evidence/migration-restart-matrix/nested-diagnostic.patch @@ -0,0 +1,58 @@ +diff --git a/src/adapters/store_migration/filesystem_migration_restart_test_fixture.rs b/src/adapters/store_migration/filesystem_migration_restart_test_fixture.rs +--- a/src/adapters/store_migration/filesystem_migration_restart_test_fixture.rs ++++ b/src/adapters/store_migration/filesystem_migration_restart_test_fixture.rs +@@ -41 +41 @@ pub(super) fn refusal(root: &Path) -> Result, Box> { +- Ok(error) ++ Ok(wrong_diagnostic(error)) +@@ -89,0 +90,51 @@ fn visit(root: &Path, relative: &Path, entries: &mut Witness) -> Result<(), Box< ++ ++// Calibration only: replace the observed refusal after the real restart and ++// unchanged complete witness check, preserving relevant outer error boundaries. ++fn wrong_diagnostic(error: Box) -> Box { ++ use super::{ ++ FilesystemMigrationAuthorityError as Authority, FilesystemMigrationInventoryError as Inventory, ++ StoreMigrationFixedStage, StoreMigrationRecoveryAmbiguity, StoreMigrationRecoveryError, ++ }; ++ use crate::adapters::{CatalogDecodeError, CatalogRestartError, PublicationHeadDecodeError, RecoveryPoolNameError}; ++ let error = match error.downcast::() { ++Ok(recovery) => return match *recovery { ++ StoreMigrationRecoveryError::Adoption { .. } => Box::new(StoreMigrationRecoveryError::Adoption { source: std::io::Error::other("calibrated wrong adoption cause") }), ++ StoreMigrationRecoveryError::Observation { .. } => Box::new(StoreMigrationRecoveryError::Observation { source: std::io::Error::other("calibrated wrong observation cause") }), ++ _ => Box::new(StoreMigrationRecoveryError::Ambiguity { ++ source: StoreMigrationRecoveryAmbiguity::StageOverlong { ++ stage: StoreMigrationFixedStage::Intent, observed: 0, ++ }, ++ }), ++ }, ++ Err(error) => error, ++ }; ++ if error.is::() { return error; } ++ let authority = match error.downcast::() { ++ Ok(authority) => *authority, ++ Err(error) => return error, ++ }; ++ Box::new(match authority { ++ Authority::Namespace { .. } => Authority::Namespace { ++ source: std::io::Error::new(std::io::ErrorKind::PermissionDenied, "calibrated wrong namespace cause"), ++ }, ++ Authority::Head { .. } => Authority::Head { ++ source: PublicationHeadDecodeError::InvalidMagic { observed: [0; 16] }, ++ }, ++ Authority::Inventory { source } => Authority::Inventory { ++ source: match source { ++ Inventory::Artifact { pool, name, .. } => Inventory::Artifact { ++ pool, name, ++ source: Box::new(CatalogRestartError::Catalog { ++ source: CatalogDecodeError::InvalidMagic { observed: [0; 16] }, ++ }), ++ }, ++ Inventory::Name { pool, name, .. } => Inventory::Name { ++ pool, name, ++ source: RecoveryPoolNameError::WrongLength { expected: 0, observed: 0 }, ++ }, ++ source => source, ++ }, ++ }, ++ authority => authority, ++ }) ++} diff --git a/docs/testing-evidence/migration-restart-matrix/pool-diagnostic-red.txt b/docs/testing-evidence/migration-restart-matrix/pool-diagnostic-red.txt new file mode 100644 index 00000000..74471459 --- /dev/null +++ b/docs/testing-evidence/migration-restart-matrix/pool-diagnostic-red.txt @@ -0,0 +1,58 @@ + Compiling rustix v1.1.4 + Compiling proc-macro2 v1.0.107 + Compiling io-lifetimes v2.0.4 + Compiling bitflags v2.13.1 + Compiling io-lifetimes v3.0.1 + Compiling linux-raw-sys v0.12.1 + Compiling io-extras v0.19.0 + Compiling unicode-ident v1.0.24 + Compiling quote v1.0.47 + Compiling shlex v2.0.1 + Compiling find-msvc-tools v0.1.9 + Compiling once_cell v1.21.4 + Compiling cap-primitives v4.0.2 + Compiling cap-std v4.0.2 + Compiling libc v0.2.186 + Compiling ambient-authority v0.0.2 + Compiling maybe-owned v0.3.4 + Compiling anstyle v1.0.14 + Compiling clap_lex v1.1.0 + Compiling ipnet v2.12.0 + Compiling cc v1.3.0 + Compiling cap-fs-ext v4.0.2 + Compiling clap_builder v4.6.2 + Compiling cfg-if v1.0.4 + Compiling constant_time_eq v0.4.2 + Compiling regex-lite v0.1.9 + Compiling arrayvec v0.7.8 + Compiling arrayref v0.3.9 + Compiling condtype v1.3.0 + Compiling allocation-counter v0.8.1 + Compiling syn v2.0.119 + Compiling blake3 v1.8.5 + Compiling clap v4.6.4 + Compiling fs-set-times v0.20.3 + Compiling rustix-linux-procfs v0.1.1 + Compiling divan-macros v0.1.21 + Compiling divan v0.1.21 + Compiling keep v0.0.0 () + Finished `test` profile [unoptimized + debuginfo] target(s) in 5.62s + Running unittests src/lib.rs (/debug/deps/keep-09abf1fe30bdfc46) + +running 1 test + +thread 'adapters::store_migration::filesystem_migration_restart_pool_tests::damaged_immutable_pool_bytes_refuse_restart_before_effects' (1999606) panicked at src/adapters/store_migration/filesystem_migration_restart_pool_tests.rs:42:9: +assertion `left == right` failed: refusal must identify the damaged immutable pool + left: Catalogs + right: Segments +note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace +test adapters::store_migration::filesystem_migration_restart_pool_tests::damaged_immutable_pool_bytes_refuse_restart_before_effects ... FAILED + +failures: + +failures: + adapters::store_migration::filesystem_migration_restart_pool_tests::damaged_immutable_pool_bytes_refuse_restart_before_effects + +test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 375 filtered out; finished in 0.01s + +error: test failed, to rerun pass `--lib` diff --git a/docs/testing-evidence/migration-restart-matrix/pool-diagnostic.patch b/docs/testing-evidence/migration-restart-matrix/pool-diagnostic.patch new file mode 100644 index 00000000..548e1d97 --- /dev/null +++ b/docs/testing-evidence/migration-restart-matrix/pool-diagnostic.patch @@ -0,0 +1,50 @@ +diff --git a/src/adapters/store_migration/filesystem_migration_restart_test_fixture.rs b/src/adapters/store_migration/filesystem_migration_restart_test_fixture.rs +--- a/src/adapters/store_migration/filesystem_migration_restart_test_fixture.rs ++++ b/src/adapters/store_migration/filesystem_migration_restart_test_fixture.rs +@@ -41 +41 @@ pub(super) fn refusal(root: &Path) -> Result, Box> { +- Ok(error) ++ Ok(wrong_diagnostic(error)) +@@ -89,0 +90,43 @@ fn visit(root: &Path, relative: &Path, entries: &mut Witness) -> Result<(), Box< ++ ++// Calibration only: replace the observed refusal after the real restart and ++// unchanged complete witness check, preserving relevant outer error boundaries. ++fn wrong_diagnostic(error: Box) -> Box { ++ use super::{ ++ FilesystemMigrationAuthorityError as Authority, FilesystemMigrationInventoryError as Inventory, ++ StoreMigrationFixedStage, StoreMigrationRecoveryAmbiguity, StoreMigrationRecoveryError, ++ }; ++ use crate::adapters::{PublicationHeadDecodeError, RecoveryPoolNameError}; ++ let error = match error.downcast::() { ++ Ok(_) => return Box::new(StoreMigrationRecoveryError::Ambiguity { ++ source: StoreMigrationRecoveryAmbiguity::StageOverlong { ++ stage: StoreMigrationFixedStage::Intent, observed: 0, ++ }, ++ }), ++ Err(error) => error, ++ }; ++ let authority = match error.downcast::() { ++ Ok(authority) => *authority, ++ Err(error) => return error, ++ }; ++ Box::new(match authority { ++ Authority::Namespace { .. } => Authority::Namespace { ++ source: std::io::Error::new(std::io::ErrorKind::PermissionDenied, "calibrated wrong namespace cause"), ++ }, ++ Authority::Head { .. } => Authority::Head { ++ source: PublicationHeadDecodeError::InvalidMagic { observed: [0; 16] }, ++ }, ++ Authority::Inventory { source } => Authority::Inventory { ++ source: match source { ++ Inventory::Artifact { pool, name, source } => Inventory::Artifact { ++ pool: match pool { super::MigrationInventoryPool::Segments => super::MigrationInventoryPool::Catalogs, super::MigrationInventoryPool::Catalogs => super::MigrationInventoryPool::Segments }, name, source, ++ }, ++ Inventory::Name { pool, name, .. } => Inventory::Name { ++ pool, name, ++ source: RecoveryPoolNameError::WrongLength { expected: 0, observed: 0 }, ++ }, ++ source => source, ++ }, ++ }, ++ authority => authority, ++ }) ++} diff --git a/src/adapters/retention.rs b/src/adapters/retention.rs index 54f5afc0..9a7d92dc 100644 --- a/src/adapters/retention.rs +++ b/src/adapters/retention.rs @@ -274,3 +274,6 @@ pub use transition_preflight::{RetentionTransitionPreflight, preflight_retention pub use transition_preflight_error::RetentionTransitionPreflightError; pub use transition_readiness::RetentionTransitionReadiness; pub use verified_closure::VerifiedRetentionClosure; + +#[cfg(all(test, feature = "repository-tasks"))] +mod filesystem_retention_migration_completion_tests; diff --git a/src/adapters/retention/filesystem_retention_migration_completion_tests.rs b/src/adapters/retention/filesystem_retention_migration_completion_tests.rs new file mode 100644 index 00000000..1b5ed751 --- /dev/null +++ b/src/adapters/retention/filesystem_retention_migration_completion_tests.rs @@ -0,0 +1,48 @@ +//! Completed migration admits live retention state without taking its ownership. + +use std::error::Error; + +use super::filesystem_retention_test_fixture::{ + ROOT_HEX, fixture, initial_preparation, open_authority, +}; +use crate::{ + CatalogRestartByteLimit, CatalogRestartPolicy, FilesystemRetentionSnapshot, + FilesystemStoreMigrationAuthority, ReaderAttemptLimit, SegmentReadPolicy, + StoreMigrationRecoveryPlan, execute_retention_publication, recover_store_migration, +}; + +// Size: medium. Oracle: migration completion preserves a successfully published retained root. +// Delete only if migration completion cannot be retried or stronger admission laws subsume it. +#[test] +fn completed_migration_preserves_published_retention_state() -> Result<(), Box> { + let (store, mut publisher) = open_authority("completed-migration-retention")?; + let root_bytes = fixture(ROOT_HEX)?; + let preparation = initial_preparation(&root_bytes)?; + let _published = execute_retention_publication(&mut publisher, &preparation)?; + drop(publisher); + let mut migration = + FilesystemStoreMigrationAuthority::reopen_for_recovery_unchecked_for_repository_tasks( + store.path(), + SegmentReadPolicy::MAXIMUM, + )?; + let intent = migration.observe_intent()?; + let receipt = recover_store_migration(&mut migration, &intent)?; + assert_eq!(receipt.plan(), StoreMigrationRecoveryPlan::Complete); + assert_eq!(receipt.executed_phases().next(), None); + drop(migration); + let policy = CatalogRestartPolicy::new( + SegmentReadPolicy::MAXIMUM, + CatalogRestartByteLimit::new(1_048_576)?, + ); + let view = + FilesystemRetentionSnapshot::load(store.path(), policy, ReaderAttemptLimit::DEFAULT)?; + let namespace = super::AdmittedRetentionRoot::decode(&root_bytes)? + .root() + .namespace() + .digest(); + let retained = view + .retained_root(namespace)? + .ok_or("published root disappeared")?; + assert_eq!(&*retained, root_bytes.as_slice()); + Ok(()) +} diff --git a/src/adapters/store_migration.rs b/src/adapters/store_migration.rs index 0f33fec5..7a1b5dee 100644 --- a/src/adapters/store_migration.rs +++ b/src/adapters/store_migration.rs @@ -66,6 +66,20 @@ mod filesystem_migration_repository_tasks; mod filesystem_migration_residue; #[cfg(test)] mod filesystem_migration_residue_kind_tests; +#[cfg(all(test, unix))] +mod filesystem_migration_restart_namespace_tests; +#[cfg(all(test, unix))] +mod filesystem_migration_restart_order_tests; +#[cfg(all(test, unix))] +mod filesystem_migration_restart_pair_tests; +#[cfg(all(test, unix))] +mod filesystem_migration_restart_pool_tests; +#[cfg(all(test, unix))] +mod filesystem_migration_restart_record_tests; +#[cfg(all(test, unix))] +mod filesystem_migration_restart_root_tests; +#[cfg(all(test, unix))] +mod filesystem_migration_restart_test_fixture; mod filesystem_migration_storage; #[cfg(test)] mod filesystem_migration_storage_tests; diff --git a/src/adapters/store_migration/filesystem_migration_recovery.rs b/src/adapters/store_migration/filesystem_migration_recovery.rs index faf00abc..23151f43 100644 --- a/src/adapters/store_migration/filesystem_migration_recovery.rs +++ b/src/adapters/store_migration/filesystem_migration_recovery.rs @@ -83,6 +83,11 @@ impl StoreMigrationRecoveryStorage for FilesystemStoreMigrationAuthority { filesystem_migration_residue::observe(self.root()) } + fn verify_complete(&mut self) -> io::Result<()> { + filesystem_initialization_namespace::admit_version_two(self.root()) + .map_err(|source| io::Error::new(source.kind(), Refusal::NamespacePreflight { source })) + } + fn adopt_residue( &mut self, residue: &StoreMigrationResidue, diff --git a/src/adapters/store_migration/filesystem_migration_recovery_tests.rs b/src/adapters/store_migration/filesystem_migration_recovery_tests.rs index 3dbd367f..95cc1c8b 100644 --- a/src/adapters/store_migration/filesystem_migration_recovery_tests.rs +++ b/src/adapters/store_migration/filesystem_migration_recovery_tests.rs @@ -9,8 +9,7 @@ use super::filesystem_migration_test_fixture::{maximum_policy, open_authority}; use super::migration_resumption::{MigrationRecords, execute_phase}; use super::{ AdmittedStoreFormatMarker, AdmittedStoreMigrationIntent, AdmittedStoreMigrationReceipt, - FilesystemStoreMigrationAuthority, StoreMigrationFixedStage, StoreMigrationIntentDecodeError, - StoreMigrationPhase, StoreMigrationRecoveryAmbiguity, StoreMigrationRecoveryError, + FilesystemStoreMigrationAuthority, StoreMigrationFixedStage, StoreMigrationPhase, StoreMigrationRecoveryPlan, StoreMigrationStorage, recover_store_migration, }; @@ -116,47 +115,6 @@ fn a_truncated_intent_stage_is_discarded_and_the_migration_completes() -> Result Ok(()) } -#[test] -fn a_corrupt_durable_intent_refuses_recovery_before_any_mutation() -> Result<(), Box> { - let (sandbox, mut authority) = open_authority("filesystem-migration-recovery-corrupt")?; - let intent = authority.observe_intent()?; - StoreMigrationStorage::verify_current(&mut authority, &intent)?; - let records = MigrationRecords::for_intent(&intent); - for phase in StoreMigrationPhase::ALL.iter().take(6) { - execute_phase(&mut authority, *phase, &records)?; - } - drop(authority); - let canonical = sandbox.path().join("migration.intent"); - let mut bytes = fs::read(&canonical)?; - let last = bytes.last_mut().ok_or("intent is empty")?; - *last ^= 1; - fs::write(&canonical, &bytes)?; - let before = fs::read_dir(sandbox.path())?.count(); - - let mut recovered = FilesystemStoreMigrationAuthority::reopen_for_recovery_unchecked_for_tests( - sandbox.path(), - maximum_policy(), - )?; - let expected = recovered.observe_intent()?; - let error = recover_store_migration(&mut recovered, &expected) - .err() - .ok_or("a corrupt durable intent was recovered")?; - drop(recovered); - - assert!(matches!( - error, - StoreMigrationRecoveryError::Ambiguity { - source: StoreMigrationRecoveryAmbiguity::IntentUndecodable { - source: StoreMigrationIntentDecodeError::ChecksumMismatch { .. } - } - } - )); - assert_eq!(fs::read_dir(sandbox.path())?.count(), before); - assert!(!sandbox.path().join("reader.lock").exists()); - sandbox.remove()?; - Ok(()) -} - pub(super) fn assert_complete_migration( root: &Path, intent: &super::CanonicalStoreMigrationIntent, diff --git a/src/adapters/store_migration/filesystem_migration_restart_namespace_tests.rs b/src/adapters/store_migration/filesystem_migration_restart_namespace_tests.rs new file mode 100644 index 00000000..cdc32415 --- /dev/null +++ b/src/adapters/store_migration/filesystem_migration_restart_namespace_tests.rs @@ -0,0 +1,190 @@ +//! Restart refuses unknown names and incorrect filesystem kinds without cleanup. + +use super::filesystem_migration_restart_test_fixture::{prefix, refusal}; +use super::{ + FilesystemMigrationAuthorityError as Authority, FilesystemMigrationRecoveryRefusal as Refusal, + FilesystemMigrationResidueKind as Kind, StoreMigrationPhase as Phase, + StoreMigrationRecoveryError as Recovery, +}; +use std::{error::Error, fs, io}; + +// Size: medium. Oracle: recovery admits only the closed migration namespace. +// Delete only if unknown names become explicitly admitted by a new protocol. +#[test] +fn unknown_root_and_nested_entries_preserve_restart_evidence() -> Result<(), Box> { + for parent in [ + "", + "retention", + "retention/roots", + "retention/manifests", + "gc", + "recovery", + "recovery/dispositions", + ] { + let store = prefix( + &format!("restart-unknown-{}", parent.replace('/', "-")), + Phase::AdmitNamespacePrefix, + )?; + fs::write( + store.path().join(parent).join("unexpected"), + b"retain this evidence", + )?; + let error = refusal(store.path())?; + if parent.is_empty() { + require_namespace(error.as_ref()); + } else { + let Some(Recovery::Adoption { source }) = error.downcast_ref::() else { + return Err(format!("{parent}: wrong refusal boundary: {error:?}").into()); + }; + assert!( + matches!(source.get_ref().and_then(|source| source.downcast_ref::()), + Some(Refusal::NamespacePreflight { source }) if source.kind() == io::ErrorKind::InvalidData), + "{parent}: {error:?}" + ); + } + store.remove()?; + } + Ok(()) +} + +// Size: medium. Oracle: fixed record and fence names are regular files, never directories. +// Delete only if the named filesystem kinds change by protocol decision. +#[test] +fn directory_substitution_at_each_fixed_file_refuses_restart() -> Result<(), Box> { + for name in [ + "migration.intent", + "migration.intent.next", + "FORMAT", + "FORMAT.next", + "migration.receipt", + "migration.receipt.next", + "reader.lock", + ] { + let store = prefix( + &format!("restart-wrong-file-kind-{name}"), + Phase::RemoveReceiptStage, + )?; + let path = store.path().join(name); + if path.exists() { + fs::remove_file(&path)?; + } + fs::create_dir(path)?; + let error = refusal(store.path())?; + require_namespace(error.as_ref()); + store.remove()?; + } + Ok(()) +} + +// Size: medium. Oracle: all protocol directories must remain real directories. +// Delete only if migration directory requirements are removed. +#[test] +fn regular_files_at_protocol_directory_names_refuse_restart() -> Result<(), Box> { + for name in [ + "retention", + "gc", + "recovery", + "retention/roots", + "retention/manifests", + "recovery/dispositions", + ] { + let store = prefix( + &format!("restart-wrong-dir-kind-{}", name.replace('/', "-")), + Phase::AdmitNamespacePrefix, + )?; + let path = store.path().join(name); + fs::remove_dir_all(&path)?; + fs::write(path, b"not a directory")?; + let error = refusal(store.path())?; + if name.contains('/') { + assert!( + matches!(error.downcast_ref::(), Some(Recovery::Observation { source }) + if matches!(source.get_ref().and_then(|source| source.downcast_ref::()), + Some(Refusal::NamespaceKind { observed_kind: Kind::RegularFile }))), + "{name}: {error:?}" + ); + } else { + require_namespace(error.as_ref()); + } + store.remove()?; + } + Ok(()) +} + +// Size: medium. Oracle: a reader fence carries no payload; observation names its exact length. +// Delete only if reader-lock payloads acquire defined protocol semantics. +#[test] +fn a_nonempty_reader_fence_preserves_its_exact_refusal() -> Result<(), Box> { + let store = prefix("restart-nonempty-reader-fence", Phase::AdmitReaderFence)?; + fs::write(store.path().join("reader.lock"), b"x")?; + let error = refusal(store.path())?; + assert!( + matches!(error.downcast_ref::(), Some(Recovery::Observation { source }) + if matches!(source.get_ref().and_then(|source| source.downcast_ref::()), + Some(Refusal::ReaderFence { observed_kind: Kind::RegularFile, observed_length: 1 }))), + "{error:?}" + ); + store.remove()?; + Ok(()) +} + +fn require_namespace(error: &(dyn Error + 'static)) { + assert!( + matches!(error.downcast_ref::(), Some(Authority::Namespace { source }) + if source.kind() == io::ErrorKind::InvalidData), + "namespace refusal lost its typed boundary: {error:?}" + ); +} + +// Size: medium. Oracle: migration fixed names never follow substituted symbolic links. +// Delete only if no-follow admission is explicitly replaced by another protocol. +#[test] +fn symbolic_links_at_fixed_record_names_refuse_without_following_them() -> Result<(), Box> +{ + for name in [ + "migration.intent", + "migration.intent.next", + "FORMAT", + "FORMAT.next", + "migration.receipt", + "migration.receipt.next", + "reader.lock", + ] { + let store = prefix( + &format!("restart-linked-record-{name}"), + Phase::RemoveReceiptStage, + )?; + let path = store.path().join(name); + if path.exists() { + fs::remove_file(&path)?; + } + std::os::unix::fs::symlink("HEAD", path)?; + let error = refusal(store.path())?; + require_namespace(error.as_ref()); + store.remove()?; + } + Ok(()) +} + +// Size: medium. Oracle: completed migration still admits only the reserved v2 namespace. +// Delete only if that namespace contract is removed or stronger restart laws subsume this law. +#[test] +fn completed_migration_refuses_unknown_reserved_entries_without_effects() +-> Result<(), Box> { + for parent in ["gc", "recovery", "recovery/dispositions"] { + let store = prefix( + &format!("restart-complete-unknown-{}", parent.replace('/', "-")), + Phase::RemoveReceiptStage, + )?; + fs::write(store.path().join(parent).join("unexpected"), b"preserve")?; + let error = refusal(store.path())?; + assert!( + matches!(error.downcast_ref::(), Some(Recovery::Observation { source }) + if matches!(source.get_ref().and_then(|source| source.downcast_ref::()), + Some(Refusal::NamespacePreflight { source }) if source.kind() == io::ErrorKind::InvalidData)), + "completed migration admitted invalid {parent}: {error:?}" + ); + store.remove()?; + } + Ok(()) +} diff --git a/src/adapters/store_migration/filesystem_migration_restart_order_tests.rs b/src/adapters/store_migration/filesystem_migration_restart_order_tests.rs new file mode 100644 index 00000000..4706ae15 --- /dev/null +++ b/src/adapters/store_migration/filesystem_migration_restart_order_tests.rs @@ -0,0 +1,148 @@ +//! Filesystem restart refusal at impossible migration orderings. + +use super::filesystem_migration_restart_test_fixture::{prefix, refusal}; +use super::{ + StoreMigrationEffect as Effect, StoreMigrationFixedStage as Stage, + StoreMigrationPhase as Phase, StoreMigrationRecoveryAmbiguity as Ambiguity, + StoreMigrationRecoveryError as Recovery, +}; +use std::{error::Error, fs}; + +// Size: medium. Oracle: no namespace, marker or receipt effect may precede durable intent. +// Delete only if the migration ordering contract disappears or stronger coverage subsumes it. +#[test] +fn effects_without_durable_intent_preserve_restart_evidence() -> Result<(), Box> { + for (name, effect) in [ + ("reader.lock", Effect::Namespace), + ("FORMAT", Effect::Marker), + ("migration.receipt", Effect::Receipt), + ] { + let store = prefix( + &format!("restart-before-intent-{name}"), + Phase::WriteIntentStage, + )?; + fs::write(store.path().join(name), [])?; + let error = refusal(store.path())?; + assert!( + matches!(error.downcast_ref::(), Some(Recovery::Ambiguity { + source: Ambiguity::EffectBeforeIntent { effect: observed } + }) if *observed == effect), + "{name}: {error:?}" + ); + store.remove()?; + } + Ok(()) +} + +// Size: medium. Oracle: later effects require cleanup of the previous stage. +// Delete only if the cleanup ordering requirement disappears. +#[test] +fn stages_surviving_later_effects_refuse_restart() -> Result<(), Box> { + for (phase, name, stage, effect) in [ + ( + Phase::LinkIntent, + "reader.lock", + Stage::Intent, + Effect::Namespace, + ), + (Phase::LinkIntent, "FORMAT", Stage::Intent, Effect::Marker), + ( + Phase::LinkIntent, + "migration.receipt", + Stage::Intent, + Effect::Receipt, + ), + ( + Phase::LinkMarker, + "migration.receipt", + Stage::Marker, + Effect::Receipt, + ), + ] { + let store = prefix(&format!("restart-stage-after-{phase:?}-{name}"), phase)?; + fs::write(store.path().join(name), [])?; + let error = refusal(store.path())?; + assert!( + matches!(error.downcast_ref::(), Some(Recovery::Ambiguity { + source: Ambiguity::StageAfterEffect { stage: observed_stage, effect: observed_effect } + }) if *observed_stage == stage && *observed_effect == effect), + "{name}: {error:?}" + ); + store.remove()?; + } + Ok(()) +} + +// Size: medium. Oracle: filesystem-realizable holes cannot be filled over later names. +// Delete only if the ordered namespace prefix is removed. +#[test] +fn namespace_holes_refuse_restart_at_the_exact_coordinates() -> Result<(), Box> { + for (name, absent, present) in [ + ("reader.lock", 0, 1), + ("retention", 1, 4), + ("retention/roots", 2, 3), + ("retention/manifests", 3, 4), + ("gc", 4, 5), + ] { + let store = prefix( + &format!("restart-namespace-hole-{absent}"), + Phase::AdmitNamespacePrefix, + )?; + let path = store.path().join(name); + if path.is_dir() { + fs::remove_dir_all(path)?; + } else { + fs::remove_file(path)?; + } + let error = refusal(store.path())?; + assert!( + matches!(error.downcast_ref::(), Some(Recovery::Ambiguity { + source: Ambiguity::NamespaceOutOfOrder { absent: observed_absent, present: observed_present } + }) if *observed_absent == absent && *observed_present == present), + "{name}: {error:?}" + ); + store.remove()?; + } + Ok(()) +} + +// Size: medium. Oracle: a marker cannot precede the complete namespace. +// Delete only if migration stops requiring the namespace before marker publication. +#[test] +fn a_marker_before_namespace_completion_preserves_evidence() -> Result<(), Box> { + let store = prefix("restart-marker-before-namespace", Phase::WriteMarkerStage)?; + fs::remove_dir_all(store.path().join("recovery"))?; + fs::remove_dir(store.path().join("gc"))?; + let error = refusal(store.path())?; + assert!( + matches!( + error.downcast_ref::(), + Some(Recovery::Ambiguity { + source: Ambiguity::MarkerBeforeNamespace + }) + ), + "{error:?}" + ); + store.remove()?; + Ok(()) +} + +// Size: medium. Oracle: a receipt requires the durable marker it binds. +// Delete only if receipts cease to bind the migration marker. +#[test] +fn a_receipt_without_its_marker_preserves_evidence() -> Result<(), Box> { + let store = prefix("restart-receipt-before-marker", Phase::WriteReceiptStage)?; + fs::remove_file(store.path().join("FORMAT"))?; + let error = refusal(store.path())?; + assert!( + matches!( + error.downcast_ref::(), + Some(Recovery::Ambiguity { + source: Ambiguity::ReceiptBeforeMarker + }) + ), + "{error:?}" + ); + store.remove()?; + Ok(()) +} diff --git a/src/adapters/store_migration/filesystem_migration_restart_pair_tests.rs b/src/adapters/store_migration/filesystem_migration_restart_pair_tests.rs new file mode 100644 index 00000000..4876788a --- /dev/null +++ b/src/adapters/store_migration/filesystem_migration_restart_pair_tests.rs @@ -0,0 +1,62 @@ +//! Restart laws for fixed-stage identity and contradictory paired evidence. + +use super::filesystem_migration_restart_test_fixture::{prefix, refusal}; +use super::{ + StoreMigrationFixedStage as Stage, StoreMigrationPhase as Phase, + StoreMigrationRecoveryAmbiguity as Ambiguity, StoreMigrationRecoveryError as Recovery, +}; +use crate::adapters::filesystem_exact_record::{ExactRecordError, ExactRecordRefusal}; +use std::{error::Error, fs}; + +// Size: medium. Oracle: stage and canonical record must share one inode, not only bytes. +// Delete only if the fixed-stage protocol disappears or stronger restart laws subsume it. +#[test] +fn byte_equal_stage_substitution_refuses_before_restart_effects() -> Result<(), Box> { + for (name, phase) in [ + ("migration.intent.next", Phase::LinkIntent), + ("FORMAT.next", Phase::LinkMarker), + ("migration.receipt.next", Phase::LinkReceipt), + ] { + let store = prefix(&format!("restart-substitution-{name}"), phase)?; + let path = store.path().join(name); + let bytes = fs::read(&path)?; + fs::remove_file(&path)?; + fs::write(path, bytes)?; + let error = refusal(store.path())?; + assert!( + matches!(error.downcast_ref::(), Some(Recovery::Adoption { source }) + if matches!(source.get_ref().and_then(|source| source.downcast_ref::()), + Some(ExactRecordError::Refused(ExactRecordRefusal::KindLengthOrIdentity)))), + "{name}: {error:?}" + ); + store.remove()?; + } + Ok(()) +} + +// Size: medium. Oracle: contradictory stage/target bytes refuse rather than choose a winner. +// Delete only if paired migration evidence is no longer part of the protocol. +#[test] +fn conflicting_stage_bytes_preserve_both_records_on_restart() -> Result<(), Box> { + for (name, phase, stage) in [ + ("migration.intent.next", Phase::LinkIntent, Stage::Intent), + ("FORMAT.next", Phase::LinkMarker, Stage::Marker), + ("migration.receipt.next", Phase::LinkReceipt, Stage::Receipt), + ] { + let store = prefix(&format!("restart-conflict-{name}"), phase)?; + let path = store.path().join(name); + let mut bytes = fs::read(&path)?; + *bytes.last_mut().ok_or("empty stage")? ^= 1; + fs::remove_file(&path)?; + fs::write(path, bytes)?; + let error = refusal(store.path())?; + assert!( + matches!(error.downcast_ref::(), Some(Recovery::Ambiguity { + source: Ambiguity::StageDiffers { stage: observed } + }) if *observed == stage), + "{name}: {error:?}" + ); + store.remove()?; + } + Ok(()) +} diff --git a/src/adapters/store_migration/filesystem_migration_restart_pool_tests.rs b/src/adapters/store_migration/filesystem_migration_restart_pool_tests.rs new file mode 100644 index 00000000..a0caff57 --- /dev/null +++ b/src/adapters/store_migration/filesystem_migration_restart_pool_tests.rs @@ -0,0 +1,192 @@ +//! Restart admission verifies immutable pools and current HEAD before recovery. + +use super::filesystem_migration_restart_test_fixture::{prefix, refusal}; +use super::{ + FilesystemMigrationAuthorityError as Authority, FilesystemMigrationInventoryError as Inventory, + MigrationInventoryPool as Pool, StoreMigrationPhase as Phase, +}; +use crate::adapters::{ + CatalogDecodeError, CatalogRestartError, PublicationHeadDecodeError, SegmentHeaderError, + SegmentReadError, +}; +use std::{ + error::Error, + fs, + path::{Path, PathBuf}, +}; + +// Size: medium. Oracle: malformed magic retains its exact decoder coordinates; +// restart cannot mutate a store whose immutable segment or catalog is corrupt. +// Delete only if immutable-pool verification is removed or stronger restart laws subsume it. +#[test] +fn damaged_immutable_pool_bytes_refuse_restart_before_effects() -> Result<(), Box> { + for (directory, pool) in [("segments", Pool::Segments), ("catalogs", Pool::Catalogs)] { + let store = prefix( + &format!("restart-damaged-{directory}"), + Phase::RemoveIntentStage, + )?; + let path = only_entry(&store.path().join(directory))?; + let (expected, observed) = damage_magic(&path)?; + let error = refusal(store.path())?; + let Some(Authority::Inventory { + source: + Inventory::Artifact { + pool: found, + source, + .. + }, + }) = error.downcast_ref::() + else { + return Err(format!("{directory}: wrong restart failure: {error:?}").into()); + }; + assert_eq!( + *found, pool, + "refusal must identify the damaged immutable pool" + ); + assert!( + match (pool, source.as_ref()) { + (Pool::Segments, CatalogRestartError::Segment { source, .. }) => + matches!(source.as_ref(), + SegmentReadError::Header { source: SegmentHeaderError::InvalidMagic { expected: found_expected, observed: found_observed } } + if *found_expected == expected && *found_observed == observed), + ( + Pool::Catalogs, + CatalogRestartError::Catalog { + source: CatalogDecodeError::InvalidMagic { observed: found }, + }, + ) => *found == observed, + _ => false, + }, + "{directory}: exact decoder failure lost: {error:?}" + ); + store.remove()?; + } + Ok(()) +} + +// Size: medium. Oracle: the freshly reopened current HEAD must decode before recovery. +// Delete only if migration no longer binds the version-one head. +#[test] +fn a_damaged_current_head_preserves_all_restart_evidence() -> Result<(), Box> { + let store = prefix("restart-damaged-current-head", Phase::RemoveIntentStage)?; + let (_expected, observed) = damage_magic(&store.path().join("HEAD"))?; + let error = refusal(store.path())?; + assert!( + matches!(error.downcast_ref::(), Some(Authority::Head { + source: PublicationHeadDecodeError::InvalidMagic { observed: found } + }) if *found == observed), + "HEAD refusal lost its decoder coordinate: {error:?}" + ); + store.remove()?; + Ok(()) +} + +fn only_entry(directory: &Path) -> Result> { + let mut entries = fs::read_dir(directory)?; + let path = entries.next().ok_or("fixture pool is empty")??.path(); + if entries.next().is_some() { + return Err("fixture pool is not a single artifact".into()); + } + Ok(path) +} + +type MagicCoordinates = ([u8; 16], [u8; 16]); + +fn damage_magic(path: &Path) -> Result> { + let mut bytes = fs::read(path)?; + let expected = bytes.get(..16).ok_or("no magic")?.try_into()?; + *bytes.first_mut().ok_or("empty record")? ^= 1; + let observed = bytes.get(..16).ok_or("no magic")?.try_into()?; + fs::write(path, bytes)?; + Ok((expected, observed)) +} + +// Size: medium. Oracle: valid alternate content cannot occupy another segment's name. +// Delete only if physical segment names no longer bind exact content. +#[test] +fn a_valid_segment_substitution_reports_both_digest_coordinates() -> Result<(), Box> { + use crate::adapters::{AdmittedSegment, SegmentReadPolicy}; + let store = prefix( + "restart-valid-segment-substitution", + Phase::RemoveIntentStage, + )?; + let path = only_entry(&store.path().join("segments"))?; + let original = fs::read(&path)?; + let replacement = super::filesystem_inventory_catalogs_test_fixture::empty_segment_bytes()?; + let expected = AdmittedSegment::decode(&original, SegmentReadPolicy::MAXIMUM)?.digest(); + let observed = AdmittedSegment::decode(&replacement, SegmentReadPolicy::MAXIMUM)?.digest(); + fs::write(&path, replacement)?; + let error = refusal(store.path())?; + assert!( + matches!(error.downcast_ref::(), Some(Authority::Inventory { + source: Inventory::Artifact { pool: Pool::Segments, source, .. } + }) if matches!(source.as_ref(), CatalogRestartError::SegmentCoordinate { expected: found_expected, observed: found_observed } + if *found_expected == expected && *found_observed == observed)), + "{error:?}" + ); + store.remove()?; + Ok(()) +} + +// Size: medium. Oracle: persisted intent binds the entire immutable inventory, including orphans. +// Delete only if inventory binding is explicitly removed from the migration protocol. +#[test] +fn adding_a_valid_orphan_cannot_reuse_the_previous_migration_intent() -> Result<(), Box> +{ + use super::{StoreMigrationRecoveryAmbiguity, StoreMigrationRecoveryError}; + use crate::adapters::{AdmittedSegment, SegmentReadPolicy, physical_pool_name}; + let store = prefix( + "restart-valid-orphan-inventory-change", + Phase::RemoveIntentStage, + )?; + let orphan = super::filesystem_inventory_catalogs_test_fixture::empty_segment_bytes()?; + let digest = AdmittedSegment::decode(&orphan, SegmentReadPolicy::MAXIMUM)?.digest(); + fs::write( + store + .path() + .join("segments") + .join(physical_pool_name::segment(digest)), + orphan, + )?; + let error = refusal(store.path())?; + assert!( + matches!( + error.downcast_ref::(), + Some(StoreMigrationRecoveryError::Ambiguity { + source: StoreMigrationRecoveryAmbiguity::IntentDiffers + }) + ), + "changed immutable inventory must not inherit old intent: {error:?}" + ); + store.remove()?; + Ok(()) +} + +// Size: medium. Oracle: canonical pool filenames have 68/85 bytes in segment-store/v1. +// Delete only if immutable pool naming changes by protocol decision. +#[test] +fn unknown_immutable_pool_names_refuse_with_the_exact_name() -> Result<(), Box> { + use crate::adapters::RecoveryPoolNameError; + for (directory, pool, width) in [ + ("segments", Pool::Segments, 68), + ("catalogs", Pool::Catalogs, 85), + ] { + let store = prefix( + &format!("restart-unknown-pool-{directory}"), + Phase::RemoveIntentStage, + )?; + fs::write( + store.path().join(directory).join("unknown"), + b"retained evidence", + )?; + let error = refusal(store.path())?; + assert!( + matches!(error.downcast_ref::(), Some(Authority::Inventory { + source: Inventory::Name { pool: found_pool, name, source: RecoveryPoolNameError::WrongLength { expected, observed: 7 } } + }) if *found_pool == pool && name.as_bytes() == b"unknown" && *expected == width), + "{directory}: {error:?}" + ); + store.remove()?; + } + Ok(()) +} diff --git a/src/adapters/store_migration/filesystem_migration_restart_record_tests.rs b/src/adapters/store_migration/filesystem_migration_restart_record_tests.rs new file mode 100644 index 00000000..b3a9b432 --- /dev/null +++ b/src/adapters/store_migration/filesystem_migration_restart_record_tests.rs @@ -0,0 +1,163 @@ +//! Exact restart diagnostics and preserved filesystem evidence for damaged records. + +use super::filesystem_migration_restart_test_fixture::{prefix, refusal}; +use super::{ + StoreFormatMarkerDecodeError as MarkerDecode, StoreMigrationFixedStage as Stage, + StoreMigrationIntentDecodeError as IntentDecode, StoreMigrationPhase as Phase, + StoreMigrationReceiptDecodeError as ReceiptDecode, + StoreMigrationRecoveryAmbiguity as Ambiguity, StoreMigrationRecoveryError as Recovery, + StoreMigrationStageDecodeError as StageDecode, +}; +use std::{error::Error, fs}; + +// Size: medium. Oracle: the documented restart boundary retains the exact decoder +// checksum coordinates and preserves all names, inode identities and bytes. +// Delete only if migration records disappear or stronger restart laws subsume this matrix. +#[test] +fn checksum_damage_in_each_migration_record_preserves_restart_evidence() +-> Result<(), Box> { + for (name, phase) in [ + ("migration.intent.next", Phase::WriteIntentStage), + ("migration.intent", Phase::RemoveIntentStage), + ("FORMAT.next", Phase::WriteMarkerStage), + ("FORMAT", Phase::RemoveMarkerStage), + ("migration.receipt.next", Phase::WriteReceiptStage), + ("migration.receipt", Phase::RemoveReceiptStage), + ] { + let store = prefix(&format!("restart-checksum-{name}"), phase)?; + let path = store.path().join(name); + let mut bytes = fs::read(&path)?; + let start = bytes.len().checked_sub(32).ok_or("no record checksum")?; + let expected: [u8; 32] = bytes.get(start..).ok_or("missing checksum")?.try_into()?; + *bytes.last_mut().ok_or("empty record")? ^= 1; + let observed: [u8; 32] = bytes.get(start..).ok_or("missing checksum")?.try_into()?; + fs::write(path, bytes)?; + let error = refusal(store.path())?; + assert_eq!( + checksum_coordinates(error.as_ref(), name), + Some((expected, observed)), + "{name}: {error:?}" + ); + store.remove()?; + } + Ok(()) +} + +fn checksum_coordinates(error: &(dyn Error + 'static), name: &str) -> Option<([u8; 32], [u8; 32])> { + let Recovery::Ambiguity { source } = error.downcast_ref::()? else { + return None; + }; + match (name, source) { + ( + "migration.intent", + Ambiguity::IntentUndecodable { + source: IntentDecode::ChecksumMismatch { expected, observed }, + }, + ) + | ( + "FORMAT", + Ambiguity::MarkerUndecodable { + source: MarkerDecode::ChecksumMismatch { expected, observed }, + }, + ) + | ( + "migration.receipt", + Ambiguity::ReceiptUndecodable { + source: ReceiptDecode::ChecksumMismatch { expected, observed }, + }, + ) + | ( + "migration.intent.next", + Ambiguity::StageUndecodable { + stage: Stage::Intent, + source: + StageDecode::Intent { + source: IntentDecode::ChecksumMismatch { expected, observed }, + }, + }, + ) + | ( + "FORMAT.next", + Ambiguity::StageUndecodable { + stage: Stage::Marker, + source: + StageDecode::Marker { + source: MarkerDecode::ChecksumMismatch { expected, observed }, + }, + }, + ) + | ( + "migration.receipt.next", + Ambiguity::StageUndecodable { + stage: Stage::Receipt, + source: + StageDecode::Receipt { + source: ReceiptDecode::ChecksumMismatch { expected, observed }, + }, + }, + ) => Some((*expected, *observed)), + _ => None, + } +} + +// Size: medium. Oracle: an overlong stage is not a discardable partial record. +// Delete only if stage framing disappears or stronger restart coverage subsumes it. +#[test] +fn overlong_stages_refuse_restart_without_disposal() -> Result<(), Box> { + for (name, phase, stage) in [ + ( + "migration.intent.next", + Phase::WriteIntentStage, + Stage::Intent, + ), + ("FORMAT.next", Phase::WriteMarkerStage, Stage::Marker), + ( + "migration.receipt.next", + Phase::WriteReceiptStage, + Stage::Receipt, + ), + ] { + let store = prefix(&format!("restart-overlong-{name}"), phase)?; + let path = store.path().join(name); + let mut bytes = fs::read(&path)?; + bytes.push(0); + let length = bytes.len(); + fs::write(path, bytes)?; + let error = refusal(store.path())?; + assert!( + matches!(error.downcast_ref::(), Some(Recovery::Ambiguity { + source: Ambiguity::StageOverlong { stage: observed_stage, observed } + }) if *observed_stage == stage && *observed == length), + "{name}: {error:?}" + ); + store.remove()?; + } + Ok(()) +} + +// Size: medium. Oracle: a checksummed receipt from another root cannot bind this intent. +// Delete only if receipts no longer bind the migration intent. +#[test] +fn a_valid_foreign_receipt_refuses_its_conflicting_intent_digest() -> Result<(), Box> { + use super::AdmittedStoreMigrationIntent; + let store = prefix("restart-foreign-receipt-local", Phase::RemoveReceiptStage)?; + let foreign = prefix("restart-foreign-receipt-other", Phase::RemoveReceiptStage)?; + let own_intent = fs::read(store.path().join("migration.intent"))?; + let other_intent = fs::read(foreign.path().join("migration.intent"))?; + let expected = AdmittedStoreMigrationIntent::decode(&own_intent)?.digest(); + let observed = AdmittedStoreMigrationIntent::decode(&other_intent)?.digest(); + fs::copy( + foreign.path().join("migration.receipt"), + store.path().join("migration.receipt"), + )?; + let error = refusal(store.path())?; + assert!( + matches!(error.downcast_ref::(), Some(Recovery::Ambiguity { + source: Ambiguity::ReceiptUndecodable { source: ReceiptDecode::IntentDigestMismatch { expected: found_expected, observed: found_observed } } + }) if found_expected == expected.as_bytes() && found_observed == observed.as_bytes()), + "{error:?}" + ); + store.remove()?; + foreign.remove()?; + Ok(()) +} diff --git a/src/adapters/store_migration/filesystem_migration_restart_root_tests.rs b/src/adapters/store_migration/filesystem_migration_restart_root_tests.rs new file mode 100644 index 00000000..56ce69ec --- /dev/null +++ b/src/adapters/store_migration/filesystem_migration_restart_root_tests.rs @@ -0,0 +1,46 @@ +//! Restart-stable root identity remains binding even when every byte agrees. + +use super::filesystem_migration_restart_test_fixture::{prefix, refusal}; +use super::{StoreMigrationPhase, StoreMigrationRecoveryAmbiguity, StoreMigrationRecoveryError}; +use crate::adapters::filesystem_test_sandbox::TestDirectory; +use std::{error::Error, fs, path::Path}; + +// Size: medium. Oracle: persisted migration intent binds the original root inode; +// copying the entire store must not authorize continuation under a new root. +// Delete only if root binding is explicitly removed or stronger restart evidence subsumes it. +#[test] +fn a_byte_equal_store_copy_refuses_the_persisted_root_identity() -> Result<(), Box> { + let original = prefix( + "restart-original-root", + StoreMigrationPhase::RemoveIntentStage, + )?; + let copied = TestDirectory::create("restart-substituted-root")?; + copy_directory(original.path(), copied.path())?; + let error = refusal(copied.path())?; + assert!( + matches!( + error.downcast_ref::(), + Some(StoreMigrationRecoveryError::Ambiguity { + source: StoreMigrationRecoveryAmbiguity::IntentDiffers + }) + ), + "copied root must not inherit migration authority: {error:?}" + ); + copied.remove()?; + original.remove()?; + Ok(()) +} + +fn copy_directory(source: &Path, destination: &Path) -> Result<(), Box> { + for entry in fs::read_dir(source)? { + let entry = entry?; + let target = destination.join(entry.file_name()); + if entry.file_type()?.is_dir() { + fs::create_dir(&target)?; + copy_directory(&entry.path(), &target)?; + } else { + fs::copy(entry.path(), target)?; + } + } + Ok(()) +} diff --git a/src/adapters/store_migration/filesystem_migration_restart_test_fixture.rs b/src/adapters/store_migration/filesystem_migration_restart_test_fixture.rs new file mode 100644 index 00000000..7bbd259c --- /dev/null +++ b/src/adapters/store_migration/filesystem_migration_restart_test_fixture.rs @@ -0,0 +1,89 @@ +//! Filesystem restart fixtures and complete evidence witnesses for migration refusal. + +use std::collections::BTreeMap; +use std::error::Error; +use std::fs; +use std::os::unix::fs::MetadataExt; +use std::path::{Path, PathBuf}; + +use super::filesystem_migration_test_fixture::{maximum_policy, open_authority}; +use super::migration_resumption::{MigrationRecords, execute_phase}; +use super::{FilesystemStoreMigrationAuthority, StoreMigrationPhase, recover_store_migration}; +use crate::adapters::filesystem_test_sandbox::TestDirectory; + +pub(super) fn prefix( + name: &str, + last: StoreMigrationPhase, +) -> Result> { + let (sandbox, mut authority) = open_authority(name)?; + let intent = authority.observe_intent()?; + authority.verify_current(&intent)?; + let records = MigrationRecords::for_intent(&intent); + for phase in StoreMigrationPhase::ALL { + execute_phase(&mut authority, phase, &records)?; + if phase == last { + return Ok(sandbox); + } + } + Err("requested migration prefix was not executed".into()) +} + +pub(super) fn refusal(root: &Path) -> Result, Box> { + let before = witness(root)?; + let error = restart(root) + .err() + .ok_or("ambiguous migration restarted successfully")?; + assert_eq!( + witness(root)?, + before, + "recovery refusal must preserve every name, inode and byte" + ); + Ok(error) +} + +fn restart(root: &Path) -> Result<(), Box> { + let mut authority = FilesystemStoreMigrationAuthority::reopen_for_recovery_unchecked_for_tests( + root, + maximum_policy(), + )?; + let expected = authority.observe_intent()?; + let _receipt = recover_store_migration(&mut authority, &expected)?; + Ok(()) +} + +#[derive(Debug, Eq, PartialEq)] +pub(super) enum Contents { + Directory, + File(Vec), + Link(PathBuf), +} + +type Witness = BTreeMap; + +fn witness(root: &Path) -> Result> { + let mut entries = BTreeMap::new(); + visit(root, Path::new(""), &mut entries)?; + Ok(entries) +} + +fn visit(root: &Path, relative: &Path, entries: &mut Witness) -> Result<(), Box> { + let path = root.join(relative); + let metadata = fs::symlink_metadata(&path)?; + let contents = if metadata.is_dir() { + for entry in fs::read_dir(&path)? { + visit(root, &relative.join(entry?.file_name()), entries)?; + } + Contents::Directory + } else if metadata.is_file() { + Contents::File(fs::read(&path)?) + } else if metadata.is_symlink() { + Contents::Link(fs::read_link(&path)?) + } else { + return Err("unexpected evidence kind in owned fixture".into()); + }; + entries.insert( + relative.to_path_buf(), + (metadata.dev(), metadata.ino(), contents), + ); + Ok(()) +} diff --git a/src/adapters/store_migration/migration_recovery_execution.rs b/src/adapters/store_migration/migration_recovery_execution.rs index 1f79ce9b..fc3d5044 100644 --- a/src/adapters/store_migration/migration_recovery_execution.rs +++ b/src/adapters/store_migration/migration_recovery_execution.rs @@ -74,7 +74,7 @@ pub enum StoreMigrationRecoveryError { /// Preserved current-state refusal or operational failure. source: io::Error, }, - /// The residue could not be observed. + /// The residue or completed namespace could not be admitted without effects. Observation { /// Preserved storage failure. source: io::Error, @@ -117,6 +117,10 @@ pub enum StoreMigrationRecoveryError { /// /// Returns [`StoreMigrationRecoveryError`] at the exact boundary that refused. /// +/// A complete plan verifies the completed version-two namespace before returning, +/// without requiring the retention pools to remain empty. A completion refusal +/// retains its source under [`StoreMigrationRecoveryError::Observation`]. +/// /// Resumption is internal: external callers cannot bypass recovery admission. /// /// ```compile_fail @@ -140,6 +144,11 @@ pub fn recover_store_migration( })?; let plan = plan_store_migration_recovery(&expected_admitted, &residue) .map_err(|source| StoreMigrationRecoveryError::Ambiguity { source })?; + if plan == StoreMigrationRecoveryPlan::Complete { + storage + .verify_complete() + .map_err(|source| StoreMigrationRecoveryError::Observation { source })?; + } let observed_prefix = super::StoreMigrationNamespacePrefix::observe(&residue) .map_err(|source| StoreMigrationRecoveryError::Ambiguity { source })?; let persisted = persisted_intent(&residue, expected)?; diff --git a/src/adapters/store_migration/migration_recovery_storage.rs b/src/adapters/store_migration/migration_recovery_storage.rs index 11b6133b..4b356243 100644 --- a/src/adapters/store_migration/migration_recovery_storage.rs +++ b/src/adapters/store_migration/migration_recovery_storage.rs @@ -24,6 +24,16 @@ pub trait StoreMigrationRecoveryStorage: StoreMigrationStorage { /// kind, a link, or an unknown entry. fn observe_residue(&mut self) -> io::Result; + /// Verifies completed migration namespace admission without effects. + /// + /// Owned post-migration retention state remains permitted. This does not + /// certify retention content or perform retention recovery. + /// + /// # Errors + /// + /// Returns the original namespace refusal or filesystem failure. + fn verify_complete(&mut self) -> io::Result<()>; + /// Reopens, verifies, and retains every exact stage and canonical record /// the residue holds, so later phases find the handles the forward /// protocol would have retained.