Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@ after its public API and format compatibility policies are established.

## [Unreleased]

- Strengthened writer-lock replacement evidence with a private after-lock scheduling checkpoint, exact refusal and preserved file bytes; production lock ordering and public behavior are unchanged (#169).

- Migration compatibility laws preserve version-one bytes and reject version-one authority at every forward prefix; public version/flag refusal tests and bounded seeded recovery-planner fuzzing extend transition evidence (#112).

- Completed migration recovery now verifies the version-two namespace before reporting success, refusing unknown reserved GC/recovery entries without effects while preserving published retention state (#111). Recovery storage implementors must supply the new read-only `verify_complete` capability.
Expand Down
2 changes: 1 addition & 1 deletion docs/formats/segment-store-v1/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,7 @@ device fault evidence.
| `KEEP-RECOVERY-001` | Crash identifiers `KEEP-CRASH-001` through `KEEP-CRASH-035` form one contiguous typed vocabulary, map to the exact owning protocol sequence, and admit an occurrence counter only for record append | Ordered identifier-and-sequence ledger | `xtask/tests/durability_crash_point_contract.rs` | Implemented in #17 |
| `KEEP-RECOVERY-002` | Initialization admits the platform before mutation, opens and locks the writer file, admits `staging`, `segments`, and `catalogs` in order, and returns a receipt only after root synchronization; every failed operation retains its exact phase and prevents later transitions | Fault-recording initialization port | `tests/store_initialization.rs` | Implemented in #17 |
| `KEEP-RECOVERY-003` | Production initialization admits only one writable, non-casefolded Linux ext4 store profile; every existing protocol directory must independently satisfy that profile and share the root's device and mount identity; initialization refuses any noncanonical root entry before mutation, completes an empty or partial canonical namespace without replacing evidence, excludes a second initializer, and retains writer authority through the synchronized receipt; published-store reopen performs no mutation, reacquires the writer lock, and requires the complete initialized root plus regular `HEAD` | Capability-relative initialization, published-reopen, child-profile, and exact platform-profile matrix | `src/adapters/filesystem_store_initializer_tests.rs`, `src/adapters/filesystem_catalog_publisher_tests.rs`, `src/adapters/filesystem_platform_profile.rs`, `tests/store_initialization.rs` | Implemented in #17 |
| `KEEP-RECOVERY-004` | Writer authority is returned only when the locked handle still has the exact device and inode resolved by the canonical `writer.lock` entry after kernel acquisition | Deterministic lock-entry replacement fixture | `src/adapters/filesystem_writer_lock_tests.rs` | Implemented in #17 |
| `KEEP-RECOVERY-004` | Writer authority is returned only when the locked handle still has the exact device and inode resolved by the canonical `writer.lock` entry after kernel acquisition | Deterministic replacement through the shared authority-producing acquisition boundary; calibrated ignored-refusal mutant | `src/adapters/filesystem_writer_lock_tests.rs`; [acquisition evidence](../../testing-evidence/writer-acquisition-identity.md) | Implemented in #17; acquisition-boundary evidence strengthened in #169 |
| `KEEP-RECOVERY-005` | Recovery counts the root and three protocol directories in fixed order before retaining names, refuses at the configured or protocol entry ceiling with the exact observed-at-least count, then returns one duplicate-free inventory sorted by namespace and raw name bytes | Fault-recording inventory port | `tests/recovery_inventory.rs` | Implemented in #17 |
| `KEEP-RECOVERY-006` | Filesystem inventory pins the admitted root and protocol directories without following links, verifies child-directory identity before and after scanning, stops each count at the remaining global budget plus one, preserves raw Linux entry-name bytes, and performs no protocol mutation | Capability-relative filesystem fixture | `src/adapters/filesystem_recovery_inventory_tests.rs`, `tests/recovery_inventory.rs` | Implemented in #17 |
| `KEEP-RECOVERY-007` | Name classification requires the four initialized root entries, admits only fixed protocol names and canonical pool coordinates in their owning namespaces, refuses simultaneous fixed recovery stages before artifact reads, and moves a refused raw name without duplicating its allocation | Canonical-name matrix and allocation counter | `tests/recovery_name_classification.rs`, `tests/recovery_name_classification_memory.rs` | Implemented in #17 |
Expand Down
27 changes: 27 additions & 0 deletions docs/testing-evidence/writer-acquisition-identity.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Writer acquisition identity evidence

Change kind: test-oracle correction with a private deterministic scheduling seam for [#169](https://github.com/flyingrobots/keep/issues/169), discovered while auditing T-13.1 under #131. Main `6051abb25a9fd33ae7ee0de5614514b709a4d82a` already propagates identity refusal correctly. This is not an ordinary runtime bug fix: unmutated parent production is not claimed to violate the contract.

## Promise and boundary

`KEEP-RECOVERY-004` requires the authority-producing acquisition path to reject an opened lock file whose canonical pathname now names a different device/inode after kernel acquisition. The old helper test did not exercise that authority path. The first replacement test exercised acquisition but replaced the entry before locking; it detected an ignored refusal yet survived moving verification before locking. Neither is credited as sufficient evidence for the final ordering claim.

The final law opens and retains the real root lock and original lock file, invokes the shared acquisition boundary, and replaces the canonical pathname at a private synchronous checkpoint after the kernel file lock and before identity verification. An independently opened handle probes the original file nonblockingly inside the checkpoint; the law requires `TryLockError::WouldBlock` before interpreting the replacement refusal. The ordinary and initialization paths invoke that same body with a no-op checkpoint. The checkpoint runs under root-then-file lock ordering, exposes no public callback API and never waits for another lock. It exists to control the observed transition without sleeps, stress or global hooks.

The test requires no guard to escape, exact `VerifyFileIdentity/InvalidData`, and unchanged bytes in both files. Existing public acquisition laws separately cover ordinary success, exclusion, missing-file refusal and no-follow behavior. This test enters the shared module boundary after outer pathname opening; it does not claim to explore every public-entry-point or raw namespace race.

## Reproducible calibration

[Checked-in receipts and replay commands](writer-acquisition-identity/README.md) provide the pinned parent, toolchain, features, profiles, exact production mutation patches, actual RED output and restored GREEN output. The resulting final candidate SHA and required hosted checks are recorded on PR #170 because a document cannot contain its own commit hash.

Ignoring identity refusal survives the old helper/public-lock/initialization tests. Moving verification before locking survives the initial stronger test. Both fail the final after-lock law with `replacement received writer authority`. Separate wrong-phase and destructive-original/replacement mutations fail the precise diagnostic and persisted-byte assertions. The earlier removed-call mutant hit dead-code lint and is excluded from runtime evidence. Source timestamps are invalidated after restoration to avoid stale binaries.

Deletion criterion: the helper-only test is subsumed by a stronger test of the actual authority-producing transition. No accepted behavior or refusal expectation is relaxed; production lock ordering, identity checks and guard construction remain intact around the private checkpoint.

A later review isolated a separate schedule-oracle gap: moving the checkpoint itself before locking survived the law at `1b27e4c6d2e791b8088123793befc219a97937c2`. The independent contention witness now detects that reorder with `Some(Ok(()))` instead of the required lock contention. The prior verification-order calibration remains historical evidence for its distinct defect; the new receipt records this checkpoint-order calibration separately.

## Execution and limits

The medium test uses one owned filesystem sandbox and real kernel locks in copied Linux Docker source with a dedicated build target. Initial library/mutation runs used overlay; public integration fixtures used ext4. The first broader all-feature attempt correctly refused overlay in unrelated production-platform laws and remains preserved as a setup failure, not a regression RED. Corrected broad validation and final calibration bind both library and integration scratch roots to ext4 without bypassing platform admission.

There is no new network dependency, random schedule, persistent format, benchmark, allocator experiment or physical power-loss claim. Per-test resource enforcement gaps remain those in the binding testing enforcement ledger. The replay page names both the focused proof and its limits; earlier green CI is never substituted for checks on a changed head.
56 changes: 56 additions & 0 deletions docs/testing-evidence/writer-acquisition-identity/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# Acquisition calibration receipts

These captured outputs and production mutation patches support [#169's evidence](../writer-acquisition-identity.md). Absolute container source/target prefixes in output are normalized to `<isolated-build>`, and redundant trailing blank lines are removed; assertion messages, outcomes and diagnostics are otherwise retained. They are execution receipts, not golden expectations or tests of test-count totals.

## Coordinates and environment

The old-oracle survivor uses main `6051abb25a9fd33ae7ee0de5614514b709a4d82a` with the identity-refusal result ignored. The early-order survivor uses the initial strengthened test at `6a5958b9f27b81be70b25e1c680398258818e2f8` with verification moved before locking. The final RED receipts use the after-lock checkpoint and the current strengthened law. The resulting committed head and final hosted check results are recorded in [PR #170](https://github.com/flyingrobots/keep/pull/170), separately from these focused receipts.

Execution used copied Linux Docker source/build trees, Rust `1.96.0 (ac68faa20 2026-05-25)`, Cargo `1.96.0 (30a34c682 2026-05-25)`, host target `aarch64-unknown-linux-gnu`, the committed lockfile and default Cargo features for focused tests. Mutant RED runs use the debug test profile. Restored tests use debug and release; Clippy explicitly enables every feature and target. The final calibration uses ext4 scratch for both library and integration fixtures. Earlier old-oracle/early-order library runs used overlay; the complete profile correction and excluded setup failures are recorded in the parent evidence page.

Per-test resource ceilings and network-denial enforcement were not supplied by this execution profile; the medium test owns its scratch state and does not call the network. The container itself is not claimed as complete hermeticity enforcement. Maintainer `@flyingrobots` owns the contract; the author supplies these receipts.

## Replay inside a copied Docker checkout

Run the following only inside the repository's Docker validation environment, with an isolated copy of the candidate source and a dedicated `CARGO_TARGET_DIR`. Library fixtures use source-local `target/tmp` when `CARGO_TARGET_TMPDIR` is absent; integration fixtures use the target scratch directory. Bind both to ext4 before running the broader production-platform suite. Do not mutate a running validation tree or mount the host repository writable.

For each patch below, start with the unmutated candidate, apply the patch, touch the production file to invalidate timestamp caches, execute the focused command, retain the failing output, reverse only that patch, and touch the restored source before GREEN. Successful compilation followed by the named runtime failure is required; an exit code alone is insufficient.

```sh
git apply docs/testing-evidence/writer-acquisition-identity/early-order.patch
touch src/adapters/filesystem_writer_lock.rs
cargo test --locked --lib filesystem_writer_lock
git apply -R docs/testing-evidence/writer-acquisition-identity/early-order.patch
touch src/adapters/filesystem_writer_lock.rs
```

Substitute each other listed patch for `early-order.patch`. The captured assertions use pre-formatting line coordinates; the named law, assertion expression and literal expected bytes identify the current check.

| Mutation | Patch | Actual RED output | Named failure |
| --- | --- | --- | --- |
| Verify before taking the file lock | [patch](early-order.patch) | [receipt](early-order-red.txt) | `replacement received writer authority` |
| Run checkpoint before taking the file lock | [patch](early-checkpoint.patch) | [receipt](early-checkpoint-red.txt) | `replacement checkpoint must observe the acquired kernel lock` |
| Ignore the identity refusal | [patch](ignored-refusal.patch) | [receipt](ignored-refusal-red.txt) | `replacement received writer authority` |
| Return the wrong phase | [patch](wrong-phase.patch) | [receipt](wrong-phase-red.txt) | `replacement must retain the identity-refusal boundary` |
| Truncate displaced evidence | [patch](lost-original.patch) | [receipt](lost-original-red.txt) | Original bytes differ from empty observed bytes |
| Truncate replacement evidence | [patch](lost-replacement.patch) | [receipt](lost-replacement-red.txt) | Replacement bytes differ from empty observed bytes |

The [old-oracle survivor](old-oracle-survived.txt) executes `cargo test --locked --lib filesystem_writer_lock`, `cargo test --locked --test catalog_writer_lock`, `cargo test --locked --test store_initialization`, and `cargo test --locked --lib filesystem_store_initializer_tests` on the pinned main mutation. The [early-order survivor](early-order-survived.txt) executes only the first command on the initial strengthened test. Their survival establishes the precise gaps; neither is claimed as full-suite mutant survival.

The [restored GREEN receipt](restored-green.txt) runs the following after every production mutation is removed:

```sh
cargo test --locked --lib filesystem_writer_lock
cargo test --locked --release --lib filesystem_writer_lock
cargo test --locked --test catalog_writer_lock
cargo test --locked --release --test catalog_writer_lock
cargo test --locked --test store_initialization
cargo test --locked --release --test store_initialization
cargo test --locked --lib filesystem_store_initializer_tests
cargo test --locked --release --lib filesystem_store_initializer_tests
cargo clippy --workspace --all-targets --all-features --locked -- -D warnings
```

The helper-only test is retired because the stronger authority-boundary law subsumes its refusal claim. Complete generated schedule exploration, unrelated recovery paths and physical durability remain outside these receipts.

The [checkpoint-order survivor](early-checkpoint-survived.txt) runs the focused debug law at `1b27e4c6d2e791b8088123793befc219a97937c2` with the checkpoint moved before locking. The [checkpoint-order RED](early-checkpoint-red.txt) adds the independent contention witness to that mutation; the [restored checkpoint GREEN](checkpoint-green.txt) records formatting, the focused law in debug/release and all-feature workspace Clippy after restoring production order. These runs use the same copied Docker toolchain and ext4 scratch profile above. They supplement the earlier receipts rather than changing their historical coordinates.
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
Compiling keep v0.0.0 (<isolated-build>)
Finished `test` profile [unoptimized + debuginfo] target(s) in 1.20s
Running unittests src/lib.rs (<isolated-build>/debug/deps/keep-882caa9da157737f)

running 1 test
test adapters::filesystem_writer_lock::tests::replaced_lock_entry_refuses_authority_after_kernel_acquisition ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 352 filtered out; finished in 0.00s

Compiling keep v0.0.0 (<isolated-build>)
Finished `release` profile [optimized] target(s) in 4.03s
Running unittests src/lib.rs (<isolated-build>/release/deps/keep-8ca1b588e6dfe3ea)

running 1 test
test adapters::filesystem_writer_lock::tests::replaced_lock_entry_refuses_authority_after_kernel_acquisition ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 352 filtered out; finished in 0.00s

Checking keep v0.0.0 (<isolated-build>)
Checking xtask v0.0.0 (<isolated-build>/xtask)
Checking keep-benchmark v0.0.0 (<isolated-build>/benchmark)
Finished `dev` profile [unoptimized + debuginfo] target(s) in 2.60s
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
Compiling keep v0.0.0 (<isolated-build>)
Finished `test` profile [unoptimized + debuginfo] target(s) in 1.79s
Running unittests src/lib.rs (<isolated-build>/debug/deps/keep-882caa9da157737f)

running 1 test
test adapters::filesystem_writer_lock::tests::replaced_lock_entry_refuses_authority_after_kernel_acquisition ... FAILED

failures:

---- adapters::filesystem_writer_lock::tests::replaced_lock_entry_refuses_authority_after_kernel_acquisition stdout ----

thread 'adapters::filesystem_writer_lock::tests::replaced_lock_entry_refuses_authority_after_kernel_acquisition' (1771617) panicked at src/adapters/filesystem_writer_lock_tests.rs:37:5:
replacement checkpoint must observe the acquired kernel lock: Some(Ok(()))
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
adapters::filesystem_writer_lock::tests::replaced_lock_entry_refuses_authority_after_kernel_acquisition

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 352 filtered out; finished in 0.00s

error: test failed, to rerun pass `--lib`
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
Compiling keep v0.0.0 (<isolated-build>)
Finished `test` profile [unoptimized + debuginfo] target(s) in 1.38s

running 1 test
Running unittests src/lib.rs (<isolated-build>/debug/deps/keep-882caa9da157737f)
test adapters::filesystem_writer_lock::tests::replaced_lock_entry_refuses_authority_after_kernel_acquisition ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 352 filtered out; finished in 0.00s
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
--- a/src/adapters/filesystem_writer_lock.rs
+++ b/src/adapters/filesystem_writer_lock.rs
@@ -118,8 +118,8 @@
}
let expected_identity = FileIdentity::from(&metadata);
let lock_file = lock_file.into_std();
- acquire_lock(&lock_file, WriterLockAcquirePhase::Acquire)?;
after_acquire();
+ acquire_lock(&lock_file, WriterLockAcquirePhase::Acquire)?;
verify_current_identity(&directory, expected_identity)?;
Ok(Self {
directory,
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
Compiling keep v0.0.0 (<isolated-build>)
Finished `test` profile [unoptimized + debuginfo] target(s) in 2.04s
Running unittests src/lib.rs (<isolated-build>/debug/deps/keep-882caa9da157737f)

running 1 test
test adapters::filesystem_writer_lock::tests::replaced_lock_entry_refuses_authority_after_kernel_acquisition ... FAILED

failures:

---- adapters::filesystem_writer_lock::tests::replaced_lock_entry_refuses_authority_after_kernel_acquisition stdout ----
Error: "replacement received writer authority"


failures:
adapters::filesystem_writer_lock::tests::replaced_lock_entry_refuses_authority_after_kernel_acquisition

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 352 filtered out; finished in 0.00s

error: test failed, to rerun pass `--lib`
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
Compiling keep v0.0.0 (<isolated-build>)
Finished `test` profile [unoptimized + debuginfo] target(s) in 1.52s
Running unittests src/lib.rs (<isolated-build>/debug/deps/keep-882caa9da157737f)

running 1 test
test adapters::filesystem_writer_lock::tests::replaced_lock_entry_refuses_authority_after_kernel_acquisition ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 352 filtered out; finished in 0.00s
Loading
Loading