Retention recovery, crash-matrix evidence, reader fence, and model-based transitions (item 6)

CHANGELOG.md

@@ -10,6 +10,49 @@ after its public API and format compatibility policies are established.
 
 ### Added
 
+- Model-based retention evidence: every three-operation sequence over initial
+  publications of two namespaces, a successor, a byte-identical retry, and a
+  stale initial (125 sequences, each in a fresh migrated store) agrees with a
+  deterministic namespace-to-(generation, anchor-set) map and liveness after
+  every step, observed through the fenced reader view; a source contract
+  keeps clocks, paths, environment, and identity out of the retention core.
+- `FilesystemRetentionSnapshot` is the version-two reader view: it admits the
+  root as version two, acquires a shared `ReaderFence` on `reader.lock`,
+  double-collects the catalog and retention heads around loading through
+  `collect_retention_view` (bounded by `ReaderAttemptLimit`, refusing an
+  exhausted limit or an absent catalog), binds the catalog snapshot, the
+  retention head, and its manifest, and verifies each selected root against
+  the manifest on demand while the fence is held.
+- Storage-independent retention recovery planning: `assess_root_stage`,
+  `assess_manifest_stage`, and `assess_head_stage` classify each fixed stage
+  as absent, complete, truncated, or corrupt through the decoders' own
+  truncation laws; `plan_retention_recovery` turns that evidence, the observed
+  current state, and pool-entry observations into an ordered
+  `RetentionRecoveryPlan` (discard a pre-effect truncated stage, link and
+  protect complete orphans, finalize a complete head over linked stages, clean
+  up stages the published head already names) or a typed
+  `RetentionRecoveryRefusal`. `RetentionRecoveryStorage` names one blocking
+  capability per step and `execute_retention_recovery` runs a plan in order,
+  stopping at the first refused step with the completed prefix named in
+  `RetentionRecoveryError`. `FilesystemRetentionPublicationAuthority::recover`
+  observes the stages within their format bounds, reopens complete stages
+  bound to their identity, and executes the plan under the retained writer
+  lock, so a crash after the head stage is synchronized finalizes on restart
+  and a byte-identical retry is already committed. Laws drive every
+  publication prefix from 0 through 18 phases, truncate each stage mid-write,
+  and replay successor prefixes over a published generation; each recovers to
+  its documented state, recovery is idempotent, and the forward retry reports
+  the predicted outcome. Publication runs that recovery as its first step, so
+  an interrupted publication no longer waits for a human unless it left a
+  complete orphan; `RecoveryRefused` and `RecoveryStepRefused` carry
+  recovery's own errors through `RetentionCurrentStateRefusal`. The crash
+  matrix gains `KEEP-CRASH-036` through `052`: a child migrates a golden
+  bundle store, publishes retention generation one, and is killed before,
+  during, or after each of the seventeen phases; restart reopens the store,
+  runs recovery, and requires the documented steps, outcome, and forward
+  retry. `FilesystemVersionTwoAdmission::reopen_unchecked_for_repository_tasks`
+  and `FilesystemStoreMigrationAuthority::open_unchecked_for_repository_tasks`
+  give repository tools the same bypass version one already had.
 - `FilesystemRetentionPublicationAuthority` executes the 17 ordered retention
   publication phases against a completely migrated version-2 root. It stages
   `root.next`, `manifest.next`, and `head.next` exclusively, verifies device

README.md

@@ -51,10 +51,10 @@ Keep is required to refuse all three, before mutating anything.
   generation-versioned catalogs, and a fixed-width `HEAD` are published
   through an ordered protocol whose every step is a named crash point.
   Platform admission is Linux ext4, non-casefolded, one writer.
-- **Proven restart recovery for version 1.** The crash matrix kills real
-  writer processes at 105 before/during/after coordinates
-  (`KEEP-CRASH-001`–`035`) and verifies the store lands in exactly one
-  documented lawful state each time.
+- **Proven restart recovery.** The crash matrix kills real writer processes
+  at 156 before/during/after coordinates (`KEEP-CRASH-001`–`052`) and
+  verifies the store lands in exactly one documented lawful state each time,
+  for version-1 publication and for version-2 retention publication.
 - **Version-2 retention and migration, forward path.** Explicit retention
   roots, deterministic closure verification, a one-way 21-phase migration,
   and a 17-phase retention publication — all with production filesystem
@@ -69,17 +69,21 @@ Keep is required to refuse all three, before mutating anything.
 
 ## What it does not do yet
 
-Version 2 writes correctly from a clean start and, if it finds the residue of
-an interrupted publication, refuses rather than guesses. Nothing yet recovers
-that residue, and readers have no fence, so **an interrupted version-2
-publication waits for a human until #19 lands.** A version-1 store stays
-admitted until its owner migrates it; migrate only if you accept that wait.
+Version 2 writes correctly from a clean start, and the next publication
+recovers the residue of an interrupted one: a stage cut mid-write is
+discarded, a head already synchronized is finalized, and a byte-identical
+retry reports already committed. The one state that waits for a human is a
+complete orphan, a crash between the root link and the head finalization,
+which stays recovery-protected until explicit disposition lands with garbage
+collection (#21). The crash matrix proves that recovery by killing real
+writer processes at all 51 retention coordinates. Readers hold a shared
+fence and double-collect both heads, so a view never straddles a
+publication. A version-1 store stays admitted until its owner migrates it.
 
 | Gap | Tracked |
 | --- | --- |
 | Restart recovery for retention publication and migration | [#19](https://github.com/flyingrobots/keep/issues/19) |
 | Restart-stable root identity coordinate in the migration intent | [#97](https://github.com/flyingrobots/keep/issues/97) |
-| Reader fence binding one consistent catalog + retention snapshot | [#19](https://github.com/flyingrobots/keep/issues/19) |
 | Precise verification reports at explicit depths | [#20](https://github.com/flyingrobots/keep/issues/20) |
 | Garbage collection and identity-preserving compaction | [#21](https://github.com/flyingrobots/keep/issues/21) |
 | Bounded production ingestion through the durable store | [#82](https://github.com/flyingrobots/keep/issues/82) |

docs/formats/segment-store-v2/README.md

@@ -94,10 +94,14 @@ head and the catalog it selects, and refuses superseded candidates, retained
 stages, replaced protocol directories, and every namespace or capacity
 violation before mutation, each as a typed `RetentionCurrentStateRefusal`.
 
-Not implemented: retention publication recovery and `KEEP-CRASH-036..052`
-process-death evidence, partial-prefix migration recovery and
-`KEEP-CRASH-053..073`, the reader fence, model-based transition evidence, and
-garbage collection. Issue #19 owns the first four and issue #21 the last;
+Retention publication recovery is implemented and proven both in-process for
+every crash prefix and by the crash matrix, which kills a real writer before,
+during, and after `KEEP-CRASH-036` through `052`.
+Readers bind one consistent catalog, retention head, and manifest view under a
+shared `ReaderFence` and verify selected roots on demand. Every three-operation
+transition sequence agrees with a deterministic namespace-to-anchor-set model.
+Not implemented: partial-prefix migration recovery and `KEEP-CRASH-053..073`,
+and garbage collection. Issue #19 owns the first four and issue #21 the last;
 issue #97 owns the restart-stable root identity coordinate. A version-1 store
 remains admitted until its owner migrates it, and the
 [requirements ledger](requirements.md) is the authority on which requirements

docs/formats/segment-store-v2/recovery.md

@@ -232,7 +232,10 @@ The retention crash points are:
 | `KEEP-CRASH-052` | retention cleanup synchronization |
 
 `RetentionPublicationPhase::ALL` freezes this exact order as a typed public
-vocabulary. Storage execution and process-death evidence remain unimplemented.
+vocabulary. `FilesystemRetentionPublicationAuthority::recover` implements the
+classification above and its effects, and the crash matrix kills a real
+writer before, during, and after every point and requires restart to recover
+to the documented state.
 
 Each point requires before, during, and after process-death evidence. Restart
 must establish exact catalog visibility, retention head, namespace generation,

docs/formats/segment-store-v2/requirements.md

@@ -15,10 +15,10 @@ case is not evidence.
 | `KEEP-RETENTION-004` | Retain and release compare expected and observed generations and publish exact successors only | unforgeable readiness and preflight proofs in `tests/retention_transition.rs` and `tests/retention_preflight.rs`; exact successor preparation and complete receipt evidence in `tests/retention_publication_preparation.rs` and `tests/retention_publication_execution.rs`; writer-locked initial filesystem publication in `filesystem_retention_storage_tests`; observed-head successor publication, exact predecessor binding, and absent-head refusal in `filesystem_retention_successor_tests`; the store's catalog head must name the closure's catalog generation and digest before any forward write in `filesystem_retention_catalog_tests`; a head whose predecessor disagrees with its manifest refuses in `filesystem_retention_current_tests`; a successor reopens and decodes the manifest-selected predecessor root and refuses an absent or changed one in `filesystem_retention_expectation_tests` | Implemented |
 | `KEEP-RETENTION-005` | Closure derivation is deterministic, bounded, cycle-safe, fail-closed, and verifies complete blob reconstruction | exact accounting, reconstruction, adversarial-catalog, and exhaustive model laws in `tests/retention_closure.rs`; corrupt members refuse through the inherited segment-record admission laws and seeded `segment_format` fuzz target routed by `closure-corruption.md` | Implemented |
 | `KEEP-RETENTION-006` | Publication follows the exact ordered durability protocol, including new namespace-directory admission and retention of fixed-stage evidence until head commit, and returns only after cleanup synchronization | typed vocabulary and blocking port in `tests/retention_publication_phase.rs` and `tests/retention_publication_storage.rs`; ordered execution, conditional namespace sync, and all 17 exact storage-fault boundaries in `tests/retention_publication_execution.rs`; production 17-phase forward filesystem execution, exclusive staging, byte-equal inode-substitution refusal, and retained-stage recovery refusal in `filesystem_retention_storage_tests`; orphan namespace directories count against the 4,096 ceiling and refuse a new namespace before any stage is written in `filesystem_retention_capacity_tests`; crash injection remains | In progress in #19 |
-| `KEEP-RETENTION-007` | Restart resolves every fixed-stage crash prefix to one documented lawful state or typed ambiguity | recovery-required refusals before any mutation in `filesystem_retention_expectation_tests`: an absent head over populated pools, a non-initial head prepared against an absent head, an orphan directory for a namespace expected absent, and an absent directory for a namespace expected current; debug and release crash matrix remains; replaced protocol directories, an absent or changed head-selected catalog, an over-full census, zero-generation pool names, and a stage retained by a failed write refuse in `filesystem_retention_*_tests` | In progress in #19 |
-| `KEEP-RETENTION-008` | Readers double-collect catalog and retention heads and bind one complete catalog, manifest, and root-generation view under a `ReaderFence` | immutable snapshot and concurrency tests | Planned in #19 |
+| `KEEP-RETENTION-007` | Restart resolves every fixed-stage crash prefix to one documented lawful state or typed ambiguity | recovery-required refusals before any mutation in `filesystem_retention_expectation_tests`: an absent head over populated pools, a non-initial head prepared against an absent head, an orphan directory for a namespace expected absent, and an absent directory for a namespace expected current; replaced protocol directories, an absent or changed head-selected catalog, an over-full census, zero-generation pool names, and a stage retained by a failed write refuse in `filesystem_retention_*_tests`; storage-independent classification of every fixed-stage crash prefix (discard, link and protect, finalize, clean up, or typed refusal) in `recovery_planner_tests`; every publication prefix 0 through 18, each mid-write truncation, and successor prefixes recover in-process to the documented state, idempotently, with the forward retry reporting the predicted outcome, in `filesystem_retention_recovery_prefix_tests`; `cargo xtask durability-crash-matrix` kills a real writer before, during, and after `KEEP-CRASH-036` through `052` and requires restart recovery to reach the documented state and the forward retry to report the predicted outcome | Implemented |
+| `KEEP-RETENTION-008` | Readers double-collect catalog and retention heads and bind one complete catalog, manifest, and root-generation view under a `ReaderFence` | `ReaderFence` holds a shared kernel lock on a verified zero-length `reader.lock`; `collect_retention_view` accepts a view only when both head coordinates agree before and after loading and refuses an exhausted attempt limit (`retention_view_collector_tests`); `FilesystemRetentionSnapshot` binds the catalog snapshot, retention head, and manifest under the fence and verifies each selected root on demand while the fence is held, refusing a substituted root and a replaced fence, and two readers share the fence while an exclusive lock waits (`filesystem_retention_snapshot_tests`) | Implemented |
 | `KEEP-RETENTION-009` | Exact already-committed retry is idempotent only while its successor remains current | byte-identical planning in `tests/retention_transition.rs`; authority-revalidated zero-mutation retry receipt in `tests/retention_publication_execution.rs`; exact already-committed filesystem retry with a byte-identical retention witness in `filesystem_retention_storage_tests`; superseded-candidate filesystem refusal with zero mutation in `filesystem_retention_successor_tests`; committed retry reopens the head-selected manifest entry and root pool bytes, refusing absent, changed, or corrupt evidence in `filesystem_retention_current_tests`; every refusal is a typed `RetentionCurrentStateRefusal` source, with superseded, committed-root-absent, committed-root-changed, and head-absent-with-artifacts pinned by downcast | Implemented |
-| `KEEP-RETENTION-010` | Model operation sequences agree with a deterministic namespace-to-anchor-set map and never admit caller identity, paths, clocks, or application policy | model-based and source-architecture tests | Planned in #19 |
+| `KEEP-RETENTION-010` | Model operation sequences agree with a deterministic namespace-to-anchor-set map and never admit caller identity, paths, clocks, or application policy | every three-operation sequence over initial publications of two namespaces, a successor, a byte-identical retry, and a stale initial (125 sequences, each in a fresh migrated store) agrees with a deterministic namespace-to-(generation, anchor-set) map plus liveness after every step, observed through the fenced reader view, in `retention_model_tests`; `tests/retention_core_architecture_contract.rs` refuses any clock, path, environment, or identity token in the retention core | Implemented |
 
 <!-- markdownlint-enable MD013 -->
 
@@ -60,10 +60,11 @@ case is not evidence.
 - A fresh forward writer is not proof that version 2 is restart-safe or
   production-admitted; partial-prefix recovery and crash evidence remain
   mandatory. This applies to retention publication exactly as it applies to
-  migration: the filesystem publication writer refuses every retained stage
-  instead of continuing it. A stage left behind by a failed write is recovery
-  evidence like any crash residue; it is never unlinked, and the next
-  publication refuses until recovery classifies it.
+  migration: the filesystem publication writer never continues a retained
+  stage; it recovers it first, discarding a pre-effect truncated stage,
+  finalizing a complete head, and refusing a complete orphan until explicit
+  disposition. A stage left behind by a failed write is recovery evidence like
+  any crash residue and is classified the same way.
 - Publication binds this store's catalog `HEAD` to the verified closure and
   reopens the head-selected catalog pool entry under authority, but it does
   not re-read closure-member segments: every read authenticates them, and

docs/formats/segment-store-v2/retention.md

@@ -165,8 +165,9 @@ implements root, manifest, and head codecs with a typed verified anchor-set
 digest, expected-state transition planning, deterministic closure verification,
 a blocking publication storage capability port, and ordered storage-port
 orchestration. `FilesystemRetentionPublicationAuthority` publishes initial and
-successor generations against its observed head and refuses superseded
-candidates and retained stages; recovery, fencing, and collection remain absent.
+successor generations against its observed head, recovers retained stages
+first, and refuses superseded candidates and protected orphans; fencing and
+collection remain absent.
 
 ## Global retention manifest
 

src/adapters/filesystem_exact_record.rs

@@ -205,7 +205,8 @@ pub(super) fn link_without_replacement(
     }
 }
 
-fn open_read(directory: &Dir, name: &str) -> io::Result<File> {
+/// Opens `name` read-only without following links or blocking.
+pub(super) fn open_read(directory: &Dir, name: &str) -> io::Result<File> {
     let mut options = OpenOptions::new();
     options.read(true).follow(FollowSymlinks::No).nonblock(true);
     directory.open_with(name, &options)

src/adapters/filesystem_version_two_admission.rs

@@ -3,7 +3,7 @@
 use std::path::Path;
 
 use cap_fs_ext::DirExt;
-#[cfg(test)]
+#[cfg(any(test, feature = "repository-tasks"))]
 use cap_std::ambient_authority;
 use cap_std::fs::Dir;
 
@@ -64,6 +64,25 @@ impl FilesystemVersionTwoAdmission {
     }
 
     /// Releases the writer lock and the three pinned retention capabilities.
+    /// Reopens a migrated root without platform admission for repository tasks.
+    ///
+    /// The crash matrix and other repository tools run on hosts outside the
+    /// admitted Linux profile; every namespace, record, and identity law still
+    /// applies. Production callers use [`Self::reopen`].
+    ///
+    /// # Errors
+    ///
+    /// Returns [`FilesystemPlatformAdmissionError`] exactly as [`Self::reopen`]
+    /// does for every boundary after platform admission.
+    #[cfg(feature = "repository-tasks")]
+    pub fn reopen_unchecked_for_repository_tasks(
+        store_root: &Path,
+    ) -> Result<Self, FilesystemPlatformAdmissionError> {
+        let root = Dir::open_ambient_dir(store_root, ambient_authority())
+            .map_err(|source| FilesystemPlatformAdmissionError::Platform { source })?;
+        Self::admit(root)
+    }
+
     pub(super) fn into_parts(self) -> (FilesystemWriterLock, Dir, Dir, Dir) {
         (self.lock, self.retention, self.roots, self.manifests)
     }