diff --git a/CHANGELOG.md b/CHANGELOG.md index 034b40ec..4167bfa0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,8 @@ after its public API and format compatibility policies are established. ## [Unreleased] +- Strengthened writer-lock replacement evidence with a private after-lock scheduling checkpoint, exact refusal and preserved file bytes; production lock ordering and public behavior are unchanged (#169). + - Migration compatibility laws preserve version-one bytes and reject version-one authority at every forward prefix; public version/flag refusal tests and bounded seeded recovery-planner fuzzing extend transition evidence (#112). - Completed migration recovery now verifies the version-two namespace before reporting success, refusing unknown reserved GC/recovery entries without effects while preserving published retention state (#111). Recovery storage implementors must supply the new read-only `verify_complete` capability. diff --git a/docs/formats/segment-store-v1/requirements.md b/docs/formats/segment-store-v1/requirements.md index 4254ddd6..35f59bae 100644 --- a/docs/formats/segment-store-v1/requirements.md +++ b/docs/formats/segment-store-v1/requirements.md @@ -118,7 +118,7 @@ device fault evidence. | `KEEP-RECOVERY-001` | Crash identifiers `KEEP-CRASH-001` through `KEEP-CRASH-035` form one contiguous typed vocabulary, map to the exact owning protocol sequence, and admit an occurrence counter only for record append | Ordered identifier-and-sequence ledger | `xtask/tests/durability_crash_point_contract.rs` | Implemented in #17 | | `KEEP-RECOVERY-002` | Initialization admits the platform before mutation, opens and locks the writer file, admits `staging`, `segments`, and `catalogs` in order, and returns a receipt only after root synchronization; every failed operation retains its exact phase and prevents later transitions | Fault-recording initialization port | `tests/store_initialization.rs` | Implemented in #17 | | `KEEP-RECOVERY-003` | Production initialization admits only one writable, non-casefolded Linux ext4 store profile; every existing protocol directory must independently satisfy that profile and share the root's device and mount identity; initialization refuses any noncanonical root entry before mutation, completes an empty or partial canonical namespace without replacing evidence, excludes a second initializer, and retains writer authority through the synchronized receipt; published-store reopen performs no mutation, reacquires the writer lock, and requires the complete initialized root plus regular `HEAD` | Capability-relative initialization, published-reopen, child-profile, and exact platform-profile matrix | `src/adapters/filesystem_store_initializer_tests.rs`, `src/adapters/filesystem_catalog_publisher_tests.rs`, `src/adapters/filesystem_platform_profile.rs`, `tests/store_initialization.rs` | Implemented in #17 | -| `KEEP-RECOVERY-004` | Writer authority is returned only when the locked handle still has the exact device and inode resolved by the canonical `writer.lock` entry after kernel acquisition | Deterministic lock-entry replacement fixture | `src/adapters/filesystem_writer_lock_tests.rs` | Implemented in #17 | +| `KEEP-RECOVERY-004` | Writer authority is returned only when the locked handle still has the exact device and inode resolved by the canonical `writer.lock` entry after kernel acquisition | Deterministic replacement through the shared authority-producing acquisition boundary; calibrated ignored-refusal mutant | `src/adapters/filesystem_writer_lock_tests.rs`; [acquisition evidence](../../testing-evidence/writer-acquisition-identity.md) | Implemented in #17; acquisition-boundary evidence strengthened in #169 | | `KEEP-RECOVERY-005` | Recovery counts the root and three protocol directories in fixed order before retaining names, refuses at the configured or protocol entry ceiling with the exact observed-at-least count, then returns one duplicate-free inventory sorted by namespace and raw name bytes | Fault-recording inventory port | `tests/recovery_inventory.rs` | Implemented in #17 | | `KEEP-RECOVERY-006` | Filesystem inventory pins the admitted root and protocol directories without following links, verifies child-directory identity before and after scanning, stops each count at the remaining global budget plus one, preserves raw Linux entry-name bytes, and performs no protocol mutation | Capability-relative filesystem fixture | `src/adapters/filesystem_recovery_inventory_tests.rs`, `tests/recovery_inventory.rs` | Implemented in #17 | | `KEEP-RECOVERY-007` | Name classification requires the four initialized root entries, admits only fixed protocol names and canonical pool coordinates in their owning namespaces, refuses simultaneous fixed recovery stages before artifact reads, and moves a refused raw name without duplicating its allocation | Canonical-name matrix and allocation counter | `tests/recovery_name_classification.rs`, `tests/recovery_name_classification_memory.rs` | Implemented in #17 | diff --git a/docs/testing-evidence/writer-acquisition-identity.md b/docs/testing-evidence/writer-acquisition-identity.md new file mode 100644 index 00000000..8a59985a --- /dev/null +++ b/docs/testing-evidence/writer-acquisition-identity.md @@ -0,0 +1,27 @@ +# Writer acquisition identity evidence + +Change kind: test-oracle correction with a private deterministic scheduling seam for [#169](https://github.com/flyingrobots/keep/issues/169), discovered while auditing T-13.1 under #131. Main `6051abb25a9fd33ae7ee0de5614514b709a4d82a` already propagates identity refusal correctly. This is not an ordinary runtime bug fix: unmutated parent production is not claimed to violate the contract. + +## Promise and boundary + +`KEEP-RECOVERY-004` requires the authority-producing acquisition path to reject an opened lock file whose canonical pathname now names a different device/inode after kernel acquisition. The old helper test did not exercise that authority path. The first replacement test exercised acquisition but replaced the entry before locking; it detected an ignored refusal yet survived moving verification before locking. Neither is credited as sufficient evidence for the final ordering claim. + +The final law opens and retains the real root lock and original lock file, invokes the shared acquisition boundary, and replaces the canonical pathname at a private synchronous checkpoint after the kernel file lock and before identity verification. An independently opened handle probes the original file nonblockingly inside the checkpoint; the law requires `TryLockError::WouldBlock` before interpreting the replacement refusal. The ordinary and initialization paths invoke that same body with a no-op checkpoint. The checkpoint runs under root-then-file lock ordering, exposes no public callback API and never waits for another lock. It exists to control the observed transition without sleeps, stress or global hooks. + +The test requires no guard to escape, exact `VerifyFileIdentity/InvalidData`, and unchanged bytes in both files. Existing public acquisition laws separately cover ordinary success, exclusion, missing-file refusal and no-follow behavior. This test enters the shared module boundary after outer pathname opening; it does not claim to explore every public-entry-point or raw namespace race. + +## Reproducible calibration + +[Checked-in receipts and replay commands](writer-acquisition-identity/README.md) provide the pinned parent, toolchain, features, profiles, exact production mutation patches, actual RED output and restored GREEN output. The resulting final candidate SHA and required hosted checks are recorded on PR #170 because a document cannot contain its own commit hash. + +Ignoring identity refusal survives the old helper/public-lock/initialization tests. Moving verification before locking survives the initial stronger test. Both fail the final after-lock law with `replacement received writer authority`. Separate wrong-phase and destructive-original/replacement mutations fail the precise diagnostic and persisted-byte assertions. The earlier removed-call mutant hit dead-code lint and is excluded from runtime evidence. Source timestamps are invalidated after restoration to avoid stale binaries. + +Deletion criterion: the helper-only test is subsumed by a stronger test of the actual authority-producing transition. No accepted behavior or refusal expectation is relaxed; production lock ordering, identity checks and guard construction remain intact around the private checkpoint. + +A later review isolated a separate schedule-oracle gap: moving the checkpoint itself before locking survived the law at `1b27e4c6d2e791b8088123793befc219a97937c2`. The independent contention witness now detects that reorder with `Some(Ok(()))` instead of the required lock contention. The prior verification-order calibration remains historical evidence for its distinct defect; the new receipt records this checkpoint-order calibration separately. + +## Execution and limits + +The medium test uses one owned filesystem sandbox and real kernel locks in copied Linux Docker source with a dedicated build target. Initial library/mutation runs used overlay; public integration fixtures used ext4. The first broader all-feature attempt correctly refused overlay in unrelated production-platform laws and remains preserved as a setup failure, not a regression RED. Corrected broad validation and final calibration bind both library and integration scratch roots to ext4 without bypassing platform admission. + +There is no new network dependency, random schedule, persistent format, benchmark, allocator experiment or physical power-loss claim. Per-test resource enforcement gaps remain those in the binding testing enforcement ledger. The replay page names both the focused proof and its limits; earlier green CI is never substituted for checks on a changed head. diff --git a/docs/testing-evidence/writer-acquisition-identity/README.md b/docs/testing-evidence/writer-acquisition-identity/README.md new file mode 100644 index 00000000..65b0dd0a --- /dev/null +++ b/docs/testing-evidence/writer-acquisition-identity/README.md @@ -0,0 +1,56 @@ +# Acquisition calibration receipts + +These captured outputs and production mutation patches support [#169's evidence](../writer-acquisition-identity.md). Absolute container source/target prefixes in output are normalized to ``, and redundant trailing blank lines are removed; assertion messages, outcomes and diagnostics are otherwise retained. They are execution receipts, not golden expectations or tests of test-count totals. + +## Coordinates and environment + +The old-oracle survivor uses main `6051abb25a9fd33ae7ee0de5614514b709a4d82a` with the identity-refusal result ignored. The early-order survivor uses the initial strengthened test at `6a5958b9f27b81be70b25e1c680398258818e2f8` with verification moved before locking. The final RED receipts use the after-lock checkpoint and the current strengthened law. The resulting committed head and final hosted check results are recorded in [PR #170](https://github.com/flyingrobots/keep/pull/170), separately from these focused receipts. + +Execution used copied Linux Docker source/build trees, Rust `1.96.0 (ac68faa20 2026-05-25)`, Cargo `1.96.0 (30a34c682 2026-05-25)`, host target `aarch64-unknown-linux-gnu`, the committed lockfile and default Cargo features for focused tests. Mutant RED runs use the debug test profile. Restored tests use debug and release; Clippy explicitly enables every feature and target. The final calibration uses ext4 scratch for both library and integration fixtures. Earlier old-oracle/early-order library runs used overlay; the complete profile correction and excluded setup failures are recorded in the parent evidence page. + +Per-test resource ceilings and network-denial enforcement were not supplied by this execution profile; the medium test owns its scratch state and does not call the network. The container itself is not claimed as complete hermeticity enforcement. Maintainer `@flyingrobots` owns the contract; the author supplies these receipts. + +## Replay inside a copied Docker checkout + +Run the following only inside the repository's Docker validation environment, with an isolated copy of the candidate source and a dedicated `CARGO_TARGET_DIR`. Library fixtures use source-local `target/tmp` when `CARGO_TARGET_TMPDIR` is absent; integration fixtures use the target scratch directory. Bind both to ext4 before running the broader production-platform suite. Do not mutate a running validation tree or mount the host repository writable. + +For each patch below, start with the unmutated candidate, apply the patch, touch the production file to invalidate timestamp caches, execute the focused command, retain the failing output, reverse only that patch, and touch the restored source before GREEN. Successful compilation followed by the named runtime failure is required; an exit code alone is insufficient. + +```sh +git apply docs/testing-evidence/writer-acquisition-identity/early-order.patch +touch src/adapters/filesystem_writer_lock.rs +cargo test --locked --lib filesystem_writer_lock +git apply -R docs/testing-evidence/writer-acquisition-identity/early-order.patch +touch src/adapters/filesystem_writer_lock.rs +``` + +Substitute each other listed patch for `early-order.patch`. The captured assertions use pre-formatting line coordinates; the named law, assertion expression and literal expected bytes identify the current check. + +| Mutation | Patch | Actual RED output | Named failure | +| --- | --- | --- | --- | +| Verify before taking the file lock | [patch](early-order.patch) | [receipt](early-order-red.txt) | `replacement received writer authority` | +| Run checkpoint before taking the file lock | [patch](early-checkpoint.patch) | [receipt](early-checkpoint-red.txt) | `replacement checkpoint must observe the acquired kernel lock` | +| Ignore the identity refusal | [patch](ignored-refusal.patch) | [receipt](ignored-refusal-red.txt) | `replacement received writer authority` | +| Return the wrong phase | [patch](wrong-phase.patch) | [receipt](wrong-phase-red.txt) | `replacement must retain the identity-refusal boundary` | +| Truncate displaced evidence | [patch](lost-original.patch) | [receipt](lost-original-red.txt) | Original bytes differ from empty observed bytes | +| Truncate replacement evidence | [patch](lost-replacement.patch) | [receipt](lost-replacement-red.txt) | Replacement bytes differ from empty observed bytes | + +The [old-oracle survivor](old-oracle-survived.txt) executes `cargo test --locked --lib filesystem_writer_lock`, `cargo test --locked --test catalog_writer_lock`, `cargo test --locked --test store_initialization`, and `cargo test --locked --lib filesystem_store_initializer_tests` on the pinned main mutation. The [early-order survivor](early-order-survived.txt) executes only the first command on the initial strengthened test. Their survival establishes the precise gaps; neither is claimed as full-suite mutant survival. + +The [restored GREEN receipt](restored-green.txt) runs the following after every production mutation is removed: + +```sh +cargo test --locked --lib filesystem_writer_lock +cargo test --locked --release --lib filesystem_writer_lock +cargo test --locked --test catalog_writer_lock +cargo test --locked --release --test catalog_writer_lock +cargo test --locked --test store_initialization +cargo test --locked --release --test store_initialization +cargo test --locked --lib filesystem_store_initializer_tests +cargo test --locked --release --lib filesystem_store_initializer_tests +cargo clippy --workspace --all-targets --all-features --locked -- -D warnings +``` + +The helper-only test is retired because the stronger authority-boundary law subsumes its refusal claim. Complete generated schedule exploration, unrelated recovery paths and physical durability remain outside these receipts. + +The [checkpoint-order survivor](early-checkpoint-survived.txt) runs the focused debug law at `1b27e4c6d2e791b8088123793befc219a97937c2` with the checkpoint moved before locking. The [checkpoint-order RED](early-checkpoint-red.txt) adds the independent contention witness to that mutation; the [restored checkpoint GREEN](checkpoint-green.txt) records formatting, the focused law in debug/release and all-feature workspace Clippy after restoring production order. These runs use the same copied Docker toolchain and ext4 scratch profile above. They supplement the earlier receipts rather than changing their historical coordinates. diff --git a/docs/testing-evidence/writer-acquisition-identity/checkpoint-green.txt b/docs/testing-evidence/writer-acquisition-identity/checkpoint-green.txt new file mode 100644 index 00000000..350ba0ef --- /dev/null +++ b/docs/testing-evidence/writer-acquisition-identity/checkpoint-green.txt @@ -0,0 +1,22 @@ + Compiling keep v0.0.0 () + Finished `test` profile [unoptimized + debuginfo] target(s) in 1.20s + Running unittests src/lib.rs (/debug/deps/keep-882caa9da157737f) + +running 1 test +test adapters::filesystem_writer_lock::tests::replaced_lock_entry_refuses_authority_after_kernel_acquisition ... ok + +test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 352 filtered out; finished in 0.00s + + Compiling keep v0.0.0 () + Finished `release` profile [optimized] target(s) in 4.03s + Running unittests src/lib.rs (/release/deps/keep-8ca1b588e6dfe3ea) + +running 1 test +test adapters::filesystem_writer_lock::tests::replaced_lock_entry_refuses_authority_after_kernel_acquisition ... ok + +test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 352 filtered out; finished in 0.00s + + Checking keep v0.0.0 () + Checking xtask v0.0.0 (/xtask) + Checking keep-benchmark v0.0.0 (/benchmark) + Finished `dev` profile [unoptimized + debuginfo] target(s) in 2.60s diff --git a/docs/testing-evidence/writer-acquisition-identity/early-checkpoint-red.txt b/docs/testing-evidence/writer-acquisition-identity/early-checkpoint-red.txt new file mode 100644 index 00000000..3c37f75c --- /dev/null +++ b/docs/testing-evidence/writer-acquisition-identity/early-checkpoint-red.txt @@ -0,0 +1,22 @@ + Compiling keep v0.0.0 () + Finished `test` profile [unoptimized + debuginfo] target(s) in 1.79s + Running unittests src/lib.rs (/debug/deps/keep-882caa9da157737f) + +running 1 test +test adapters::filesystem_writer_lock::tests::replaced_lock_entry_refuses_authority_after_kernel_acquisition ... FAILED + +failures: + +---- adapters::filesystem_writer_lock::tests::replaced_lock_entry_refuses_authority_after_kernel_acquisition stdout ---- + +thread 'adapters::filesystem_writer_lock::tests::replaced_lock_entry_refuses_authority_after_kernel_acquisition' (1771617) panicked at src/adapters/filesystem_writer_lock_tests.rs:37:5: +replacement checkpoint must observe the acquired kernel lock: Some(Ok(())) +note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace + + +failures: + adapters::filesystem_writer_lock::tests::replaced_lock_entry_refuses_authority_after_kernel_acquisition + +test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 352 filtered out; finished in 0.00s + +error: test failed, to rerun pass `--lib` diff --git a/docs/testing-evidence/writer-acquisition-identity/early-checkpoint-survived.txt b/docs/testing-evidence/writer-acquisition-identity/early-checkpoint-survived.txt new file mode 100644 index 00000000..c557ad2d --- /dev/null +++ b/docs/testing-evidence/writer-acquisition-identity/early-checkpoint-survived.txt @@ -0,0 +1,8 @@ + Compiling keep v0.0.0 () + Finished `test` profile [unoptimized + debuginfo] target(s) in 1.38s + +running 1 test + Running unittests src/lib.rs (/debug/deps/keep-882caa9da157737f) +test adapters::filesystem_writer_lock::tests::replaced_lock_entry_refuses_authority_after_kernel_acquisition ... ok + +test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 352 filtered out; finished in 0.00s diff --git a/docs/testing-evidence/writer-acquisition-identity/early-checkpoint.patch b/docs/testing-evidence/writer-acquisition-identity/early-checkpoint.patch new file mode 100644 index 00000000..9f8bc207 --- /dev/null +++ b/docs/testing-evidence/writer-acquisition-identity/early-checkpoint.patch @@ -0,0 +1,12 @@ +--- a/src/adapters/filesystem_writer_lock.rs ++++ b/src/adapters/filesystem_writer_lock.rs +@@ -118,8 +118,8 @@ + } + let expected_identity = FileIdentity::from(&metadata); + let lock_file = lock_file.into_std(); +- acquire_lock(&lock_file, WriterLockAcquirePhase::Acquire)?; + after_acquire(); ++ acquire_lock(&lock_file, WriterLockAcquirePhase::Acquire)?; + verify_current_identity(&directory, expected_identity)?; + Ok(Self { + directory, diff --git a/docs/testing-evidence/writer-acquisition-identity/early-order-red.txt b/docs/testing-evidence/writer-acquisition-identity/early-order-red.txt new file mode 100644 index 00000000..58aa2cb8 --- /dev/null +++ b/docs/testing-evidence/writer-acquisition-identity/early-order-red.txt @@ -0,0 +1,19 @@ + Compiling keep v0.0.0 () + Finished `test` profile [unoptimized + debuginfo] target(s) in 2.04s + Running unittests src/lib.rs (/debug/deps/keep-882caa9da157737f) + +running 1 test +test adapters::filesystem_writer_lock::tests::replaced_lock_entry_refuses_authority_after_kernel_acquisition ... FAILED + +failures: + +---- adapters::filesystem_writer_lock::tests::replaced_lock_entry_refuses_authority_after_kernel_acquisition stdout ---- +Error: "replacement received writer authority" + + +failures: + adapters::filesystem_writer_lock::tests::replaced_lock_entry_refuses_authority_after_kernel_acquisition + +test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 352 filtered out; finished in 0.00s + +error: test failed, to rerun pass `--lib` diff --git a/docs/testing-evidence/writer-acquisition-identity/early-order-survived.txt b/docs/testing-evidence/writer-acquisition-identity/early-order-survived.txt new file mode 100644 index 00000000..06a6af4f --- /dev/null +++ b/docs/testing-evidence/writer-acquisition-identity/early-order-survived.txt @@ -0,0 +1,8 @@ + Compiling keep v0.0.0 () + Finished `test` profile [unoptimized + debuginfo] target(s) in 1.52s + Running unittests src/lib.rs (/debug/deps/keep-882caa9da157737f) + +running 1 test +test adapters::filesystem_writer_lock::tests::replaced_lock_entry_refuses_authority_after_kernel_acquisition ... ok + +test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 352 filtered out; finished in 0.00s diff --git a/docs/testing-evidence/writer-acquisition-identity/early-order.patch b/docs/testing-evidence/writer-acquisition-identity/early-order.patch new file mode 100644 index 00000000..8fba2f2a --- /dev/null +++ b/docs/testing-evidence/writer-acquisition-identity/early-order.patch @@ -0,0 +1,13 @@ +--- a/src/adapters/filesystem_writer_lock.rs ++++ b/src/adapters/filesystem_writer_lock.rs +@@ -117,9 +117,9 @@ + } + let expected_identity = FileIdentity::from(&metadata); + let lock_file = lock_file.into_std(); ++ verify_current_identity(&directory, expected_identity)?; + acquire_lock(&lock_file, WriterLockAcquirePhase::Acquire)?; + after_acquire(); +- verify_current_identity(&directory, expected_identity)?; + Ok(Self { + directory, + _root_lock_file: root_lock_file, diff --git a/docs/testing-evidence/writer-acquisition-identity/ignored-refusal-red.txt b/docs/testing-evidence/writer-acquisition-identity/ignored-refusal-red.txt new file mode 100644 index 00000000..6c03690f --- /dev/null +++ b/docs/testing-evidence/writer-acquisition-identity/ignored-refusal-red.txt @@ -0,0 +1,19 @@ + Compiling keep v0.0.0 () + Finished `test` profile [unoptimized + debuginfo] target(s) in 1.78s + Running unittests src/lib.rs (/debug/deps/keep-882caa9da157737f) + +running 1 test +test adapters::filesystem_writer_lock::tests::replaced_lock_entry_refuses_authority_after_kernel_acquisition ... FAILED + +failures: + +---- adapters::filesystem_writer_lock::tests::replaced_lock_entry_refuses_authority_after_kernel_acquisition stdout ---- +Error: "replacement received writer authority" + + +failures: + adapters::filesystem_writer_lock::tests::replaced_lock_entry_refuses_authority_after_kernel_acquisition + +test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 352 filtered out; finished in 0.00s + +error: test failed, to rerun pass `--lib` diff --git a/docs/testing-evidence/writer-acquisition-identity/ignored-refusal.patch b/docs/testing-evidence/writer-acquisition-identity/ignored-refusal.patch new file mode 100644 index 00000000..ba9e345b --- /dev/null +++ b/docs/testing-evidence/writer-acquisition-identity/ignored-refusal.patch @@ -0,0 +1,11 @@ +--- a/src/adapters/filesystem_writer_lock.rs ++++ b/src/adapters/filesystem_writer_lock.rs +@@ -119,7 +119,7 @@ + let lock_file = lock_file.into_std(); + acquire_lock(&lock_file, WriterLockAcquirePhase::Acquire)?; + after_acquire(); +- verify_current_identity(&directory, expected_identity)?; ++ let _ = verify_current_identity(&directory, expected_identity); + Ok(Self { + directory, + _root_lock_file: root_lock_file, diff --git a/docs/testing-evidence/writer-acquisition-identity/lost-original-red.txt b/docs/testing-evidence/writer-acquisition-identity/lost-original-red.txt new file mode 100644 index 00000000..41b24eb8 --- /dev/null +++ b/docs/testing-evidence/writer-acquisition-identity/lost-original-red.txt @@ -0,0 +1,23 @@ + Compiling keep v0.0.0 () + Finished `test` profile [unoptimized + debuginfo] target(s) in 1.65s + Running unittests src/lib.rs (/debug/deps/keep-882caa9da157737f) + +running 1 test +error: test failed, to rerun pass `--lib` +test adapters::filesystem_writer_lock::tests::replaced_lock_entry_refuses_authority_after_kernel_acquisition ... FAILED + +failures: + +---- adapters::filesystem_writer_lock::tests::replaced_lock_entry_refuses_authority_after_kernel_acquisition stdout ---- + +thread 'adapters::filesystem_writer_lock::tests::replaced_lock_entry_refuses_authority_after_kernel_acquisition' (1751542) panicked at src/adapters/filesystem_writer_lock_tests.rs:45:5: +assertion `left == right` failed + left: [] + right: [111, 114, 105, 103, 105, 110, 97, 108, 32, 101, 118, 105, 100, 101, 110, 99, 101] +note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace + + +failures: + adapters::filesystem_writer_lock::tests::replaced_lock_entry_refuses_authority_after_kernel_acquisition + +test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 352 filtered out; finished in 0.00s diff --git a/docs/testing-evidence/writer-acquisition-identity/lost-original.patch b/docs/testing-evidence/writer-acquisition-identity/lost-original.patch new file mode 100644 index 00000000..0afa1039 --- /dev/null +++ b/docs/testing-evidence/writer-acquisition-identity/lost-original.patch @@ -0,0 +1,10 @@ +--- a/src/adapters/filesystem_writer_lock.rs ++++ b/src/adapters/filesystem_writer_lock.rs +@@ -182,6 +182,7 @@ + if observed == expected { + return Ok(()); + } ++ directory.open_with("displaced.lock", &lock_options()).and_then(|file| file.set_len(0)).map_err(|source| WriterLockAcquireError::io(WriterLockAcquirePhase::VerifyFileIdentity, source))?; + Err(WriterLockAcquireError::io( + WriterLockAcquirePhase::VerifyFileIdentity, + io::Error::new( diff --git a/docs/testing-evidence/writer-acquisition-identity/lost-replacement-red.txt b/docs/testing-evidence/writer-acquisition-identity/lost-replacement-red.txt new file mode 100644 index 00000000..40e0c04a --- /dev/null +++ b/docs/testing-evidence/writer-acquisition-identity/lost-replacement-red.txt @@ -0,0 +1,24 @@ + Compiling keep v0.0.0 () + Finished `test` profile [unoptimized + debuginfo] target(s) in 1.12s + Running unittests src/lib.rs (/debug/deps/keep-882caa9da157737f) + +running 1 test +test adapters::filesystem_writer_lock::tests::replaced_lock_entry_refuses_authority_after_kernel_acquisition ... FAILED + +failures: + +---- adapters::filesystem_writer_lock::tests::replaced_lock_entry_refuses_authority_after_kernel_acquisition stdout ---- + +thread 'adapters::filesystem_writer_lock::tests::replaced_lock_entry_refuses_authority_after_kernel_acquisition' (1751834) panicked at src/adapters/filesystem_writer_lock_tests.rs:49:5: +assertion `left == right` failed + left: [] + right: [114, 101, 112, 108, 97, 99, 101, 109, 101, 110, 116, 32, 101, 118, 105, 100, 101, 110, 99, 101] +note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace + + +failures: + adapters::filesystem_writer_lock::tests::replaced_lock_entry_refuses_authority_after_kernel_acquisition + +test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 352 filtered out; finished in 0.00s + +error: test failed, to rerun pass `--lib` diff --git a/docs/testing-evidence/writer-acquisition-identity/lost-replacement.patch b/docs/testing-evidence/writer-acquisition-identity/lost-replacement.patch new file mode 100644 index 00000000..8f82fabd --- /dev/null +++ b/docs/testing-evidence/writer-acquisition-identity/lost-replacement.patch @@ -0,0 +1,10 @@ +--- a/src/adapters/filesystem_writer_lock.rs ++++ b/src/adapters/filesystem_writer_lock.rs +@@ -182,6 +182,7 @@ + if observed == expected { + return Ok(()); + } ++ directory.open_with("writer.lock", &lock_options()).and_then(|file| file.set_len(0)).map_err(|source| WriterLockAcquireError::io(WriterLockAcquirePhase::VerifyFileIdentity, source))?; + Err(WriterLockAcquireError::io( + WriterLockAcquirePhase::VerifyFileIdentity, + io::Error::new( diff --git a/docs/testing-evidence/writer-acquisition-identity/old-oracle-survived.txt b/docs/testing-evidence/writer-acquisition-identity/old-oracle-survived.txt new file mode 100644 index 00000000..547bac50 --- /dev/null +++ b/docs/testing-evidence/writer-acquisition-identity/old-oracle-survived.txt @@ -0,0 +1,44 @@ + Compiling keep v0.0.0 () + Finished `test` profile [unoptimized + debuginfo] target(s) in 1.47s + Running unittests src/lib.rs (/debug/deps/keep-882caa9da157737f) + +running 1 test +test adapters::filesystem_writer_lock::tests::replaced_lock_entry_cannot_authorize_the_opened_handle ... ok + +test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 352 filtered out; finished in 0.00s + + Compiling keep v0.0.0 () + Finished `test` profile [unoptimized + debuginfo] target(s) in 0.77s + Running tests/catalog_writer_lock.rs (/debug/deps/catalog_writer_lock-ef4d33dbd86a9529) + +running 4 tests +test lock_acquisition_never_follows_a_symbolic_link ... ok +test replacing_the_lock_entry_cannot_split_live_writer_authority ... ok +test missing_lock_evidence_is_never_created_by_acquisition ... ok +test one_persistent_lock_excludes_every_second_writer ... ok + +test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s + + Compiling keep v0.0.0 () + Finished `test` profile [unoptimized + debuginfo] target(s) in 0.23s + Running tests/store_initialization.rs (/debug/deps/store_initialization-6a5a96730e6049bf) + +running 2 tests +test initialization_admits_platform_before_every_namespace_transition ... ok +test initialization_stops_at_and_preserves_every_exact_failure_phase ... ok + +test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s + + Finished `test` profile [unoptimized + debuginfo] target(s) in 0.01s + Running unittests src/lib.rs (/debug/deps/keep-882caa9da157737f) + +running 7 tests +test adapters::filesystem_store_initializer_tests::unknown_namespace_refuses_before_writer_file_creation ... ok +test adapters::filesystem_store_initializer_tests::empty_namespace_is_admitted_only_with_the_complete_root_shape ... ok +test adapters::filesystem_store_initializer_tests::partial_canonical_namespace_is_completed_without_replacing_evidence ... ok +test adapters::filesystem_store_initializer_tests::published_reopen_refuses_unknown_root_evidence ... ok +test adapters::filesystem_store_initializer_tests::linux_initializer_synchronizes_an_opath_root ... ok +test adapters::filesystem_store_initializer_tests::published_reopen_requires_a_complete_root_namespace ... ok +test adapters::filesystem_store_initializer_tests::retained_initializer_authority_excludes_a_second_initializer ... ok + +test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 346 filtered out; finished in 0.04s diff --git a/docs/testing-evidence/writer-acquisition-identity/restored-green.txt b/docs/testing-evidence/writer-acquisition-identity/restored-green.txt new file mode 100644 index 00000000..b72052f5 --- /dev/null +++ b/docs/testing-evidence/writer-acquisition-identity/restored-green.txt @@ -0,0 +1,94 @@ + Compiling keep v0.0.0 () + Finished `test` profile [unoptimized + debuginfo] target(s) in 1.93s + Running unittests src/lib.rs (/debug/deps/keep-882caa9da157737f) + +running 1 test +test adapters::filesystem_writer_lock::tests::replaced_lock_entry_refuses_authority_after_kernel_acquisition ... ok + +test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 352 filtered out; finished in 0.00s + + Compiling keep v0.0.0 () + Finished `release` profile [optimized] target(s) in 4.71s + Running unittests src/lib.rs (/release/deps/keep-8ca1b588e6dfe3ea) + +running 1 test +test adapters::filesystem_writer_lock::tests::replaced_lock_entry_refuses_authority_after_kernel_acquisition ... ok + +test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 352 filtered out; finished in 0.00s + + Compiling keep v0.0.0 () + Finished `test` profile [unoptimized + debuginfo] target(s) in 1.42s + Running tests/catalog_writer_lock.rs (/debug/deps/catalog_writer_lock-ef4d33dbd86a9529) + +running 4 tests +test lock_acquisition_never_follows_a_symbolic_link ... ok +test replacing_the_lock_entry_cannot_split_live_writer_authority ... ok +test missing_lock_evidence_is_never_created_by_acquisition ... ok +test one_persistent_lock_excludes_every_second_writer ... ok + +test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s + + Compiling keep v0.0.0 () + Finished `release` profile [optimized] target(s) in 2.46s + Running tests/catalog_writer_lock.rs (/release/deps/catalog_writer_lock-54abf0417acf05a7) + +running 4 tests +test lock_acquisition_never_follows_a_symbolic_link ... ok +test one_persistent_lock_excludes_every_second_writer ... ok +test replacing_the_lock_entry_cannot_split_live_writer_authority ... ok +test missing_lock_evidence_is_never_created_by_acquisition ... ok + +test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s + + Compiling keep v0.0.0 () + Finished `test` profile [unoptimized + debuginfo] target(s) in 0.26s + Running tests/store_initialization.rs (/debug/deps/store_initialization-6a5a96730e6049bf) + +running 2 tests +test initialization_admits_platform_before_every_namespace_transition ... ok +test initialization_stops_at_and_preserves_every_exact_failure_phase ... ok + +test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s + + Compiling keep v0.0.0 () + Finished `release` profile [optimized] target(s) in 0.14s + Running tests/store_initialization.rs (/release/deps/store_initialization-1b3d949930a31f32) + +running 2 tests +test initialization_stops_at_and_preserves_every_exact_failure_phase ... ok +test initialization_admits_platform_before_every_namespace_transition ... ok + +test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s + + Finished `test` profile [unoptimized + debuginfo] target(s) in 0.01s + Running unittests src/lib.rs (/debug/deps/keep-882caa9da157737f) + +running 7 tests +test adapters::filesystem_store_initializer_tests::unknown_namespace_refuses_before_writer_file_creation ... ok +test adapters::filesystem_store_initializer_tests::published_reopen_requires_a_complete_root_namespace ... ok +test adapters::filesystem_store_initializer_tests::linux_initializer_synchronizes_an_opath_root ... ok +test adapters::filesystem_store_initializer_tests::partial_canonical_namespace_is_completed_without_replacing_evidence ... ok +test adapters::filesystem_store_initializer_tests::empty_namespace_is_admitted_only_with_the_complete_root_shape ... ok +test adapters::filesystem_store_initializer_tests::published_reopen_refuses_unknown_root_evidence ... ok +test adapters::filesystem_store_initializer_tests::retained_initializer_authority_excludes_a_second_initializer ... ok + +test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 346 filtered out; finished in 0.04s + + Finished `release` profile [optimized] target(s) in 0.01s + Running unittests src/lib.rs (/release/deps/keep-8ca1b588e6dfe3ea) + +running 7 tests +test adapters::filesystem_store_initializer_tests::unknown_namespace_refuses_before_writer_file_creation ... ok +test adapters::filesystem_store_initializer_tests::empty_namespace_is_admitted_only_with_the_complete_root_shape ... ok +test adapters::filesystem_store_initializer_tests::partial_canonical_namespace_is_completed_without_replacing_evidence ... ok +test adapters::filesystem_store_initializer_tests::published_reopen_refuses_unknown_root_evidence ... ok +test adapters::filesystem_store_initializer_tests::published_reopen_requires_a_complete_root_namespace ... ok +test adapters::filesystem_store_initializer_tests::linux_initializer_synchronizes_an_opath_root ... ok +test adapters::filesystem_store_initializer_tests::retained_initializer_authority_excludes_a_second_initializer ... ok + +test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 346 filtered out; finished in 0.01s + + Checking keep v0.0.0 () + Checking xtask v0.0.0 (/xtask) + Checking keep-benchmark v0.0.0 (/benchmark) + Finished `dev` profile [unoptimized + debuginfo] target(s) in 3.24s diff --git a/docs/testing-evidence/writer-acquisition-identity/wrong-phase-red.txt b/docs/testing-evidence/writer-acquisition-identity/wrong-phase-red.txt new file mode 100644 index 00000000..423a80e6 --- /dev/null +++ b/docs/testing-evidence/writer-acquisition-identity/wrong-phase-red.txt @@ -0,0 +1,22 @@ + Compiling keep v0.0.0 () + Finished `test` profile [unoptimized + debuginfo] target(s) in 1.56s + Running unittests src/lib.rs (/debug/deps/keep-882caa9da157737f) + +running 1 test +test adapters::filesystem_writer_lock::tests::replaced_lock_entry_refuses_authority_after_kernel_acquisition ... FAILED + +failures: + +---- adapters::filesystem_writer_lock::tests::replaced_lock_entry_refuses_authority_after_kernel_acquisition stdout ---- + +thread 'adapters::filesystem_writer_lock::tests::replaced_lock_entry_refuses_authority_after_kernel_acquisition' (1751250) panicked at src/adapters/filesystem_writer_lock_tests.rs:35:5: +replacement must retain the identity-refusal boundary: Io { phase: Acquire, source: Custom { kind: InvalidData, error: "writer.lock changed identity during acquisition" } } +note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace + + +failures: + adapters::filesystem_writer_lock::tests::replaced_lock_entry_refuses_authority_after_kernel_acquisition + +test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 352 filtered out; finished in 0.00s + +error: test failed, to rerun pass `--lib` diff --git a/docs/testing-evidence/writer-acquisition-identity/wrong-phase.patch b/docs/testing-evidence/writer-acquisition-identity/wrong-phase.patch new file mode 100644 index 00000000..99c0228b --- /dev/null +++ b/docs/testing-evidence/writer-acquisition-identity/wrong-phase.patch @@ -0,0 +1,11 @@ +--- a/src/adapters/filesystem_writer_lock.rs ++++ b/src/adapters/filesystem_writer_lock.rs +@@ -183,7 +183,7 @@ + return Ok(()); + } + Err(WriterLockAcquireError::io( +- WriterLockAcquirePhase::VerifyFileIdentity, ++ WriterLockAcquirePhase::Acquire, + io::Error::new( + io::ErrorKind::InvalidData, + "writer.lock changed identity during acquisition", diff --git a/src/adapters/filesystem_writer_lock.rs b/src/adapters/filesystem_writer_lock.rs index c3a98447..3b43f6a1 100644 --- a/src/adapters/filesystem_writer_lock.rs +++ b/src/adapters/filesystem_writer_lock.rs @@ -97,6 +97,18 @@ impl FilesystemWriterLock { directory: Dir, root_lock_file: File, lock_file: cap_std::fs::File, + ) -> Result { + Self::acquire_with(directory, root_lock_file, lock_file, || {}) + } + + // The private synchronous seam runs under root-then-file locks. Production + // supplies a no-op; tests may probe contention nonblockingly and replace + // the entry here, but must never wait for another lock. + fn acquire_with( + directory: Dir, + root_lock_file: File, + lock_file: cap_std::fs::File, + after_acquire: F, ) -> Result { let metadata = lock_file.metadata().map_err(|source| { WriterLockAcquireError::io(WriterLockAcquirePhase::InspectFile, source) @@ -107,6 +119,7 @@ impl FilesystemWriterLock { let expected_identity = FileIdentity::from(&metadata); let lock_file = lock_file.into_std(); acquire_lock(&lock_file, WriterLockAcquirePhase::Acquire)?; + after_acquire(); verify_current_identity(&directory, expected_identity)?; Ok(Self { directory, diff --git a/src/adapters/filesystem_writer_lock_tests.rs b/src/adapters/filesystem_writer_lock_tests.rs index 0343ca8a..ff6e371d 100644 --- a/src/adapters/filesystem_writer_lock_tests.rs +++ b/src/adapters/filesystem_writer_lock_tests.rs @@ -6,35 +6,59 @@ use std::fs; use cap_std::ambient_authority; use cap_std::fs::Dir; -use super::{FileIdentity, LOCK_FILE_NAME, open_existing, verify_current_identity}; +use super::{FilesystemWriterLock, LOCK_FILE_NAME, acquire_root, open_existing}; use crate::adapters::filesystem_test_sandbox::TestDirectory; use crate::adapters::{WriterLockAcquireError, WriterLockAcquirePhase}; +// Size: medium. Oracle: KEEP-RECOVERY-004 forbids returning mismatched authority. +// This shared acquisition boundary performs the real kernel lock and returns the guard. +// Delete if a stronger scheduled public acquisition law subsumes this transition. #[test] -fn replaced_lock_entry_cannot_authorize_the_opened_handle() -> Result<(), Box> { +fn replaced_lock_entry_refuses_authority_after_kernel_acquisition() -> Result<(), Box> { let sandbox = TestDirectory::create("writer-lock-identity")?; - fs::write(sandbox.path().join(LOCK_FILE_NAME), [])?; + fs::write(sandbox.path().join(LOCK_FILE_NAME), b"original evidence")?; let directory = Dir::open_ambient_dir(sandbox.path(), ambient_authority())?; + let root_lock = acquire_root(&directory)?; let opened = open_existing(&directory)?; - let expected = FileIdentity::read(&opened)?; + let contender = fs::File::open(sandbox.path().join(LOCK_FILE_NAME))?; - fs::rename( - sandbox.path().join(LOCK_FILE_NAME), - sandbox.path().join("displaced.lock"), - )?; - fs::write(sandbox.path().join(LOCK_FILE_NAME), [])?; - - let error = verify_current_identity(&directory, expected) + let mut replacement = Ok(()); + let mut lock_observation = None; + let result = FilesystemWriterLock::acquire_with(directory, root_lock, opened, || { + lock_observation = Some(contender.try_lock()); + replacement = fs::rename( + sandbox.path().join(LOCK_FILE_NAME), + sandbox.path().join("displaced.lock"), + ) + .and_then(|()| fs::write(sandbox.path().join(LOCK_FILE_NAME), b"replacement evidence")); + }); + drop(contender); + replacement?; + assert!( + matches!(lock_observation, Some(Err(fs::TryLockError::WouldBlock))), + "replacement checkpoint must observe the acquired kernel lock: {lock_observation:?}" + ); + let error = result .err() - .ok_or("replacement was admitted as the opened lock file")?; - assert!(matches!( - error, - WriterLockAcquireError::Io { - phase: WriterLockAcquirePhase::VerifyFileIdentity, - ref source, - } if source.kind() == std::io::ErrorKind::InvalidData - )); - drop(opened); + .ok_or("replacement received writer authority")?; + assert!( + matches!( + error, + WriterLockAcquireError::Io { + phase: WriterLockAcquirePhase::VerifyFileIdentity, + ref source, + } if source.kind() == std::io::ErrorKind::InvalidData + ), + "replacement must retain the identity-refusal boundary: {error:?}" + ); + assert_eq!( + fs::read(sandbox.path().join("displaced.lock"))?, + b"original evidence" + ); + assert_eq!( + fs::read(sandbox.path().join(LOCK_FILE_NAME))?, + b"replacement evidence" + ); sandbox.remove()?; Ok(()) }