diff --git a/CHANGELOG.md b/CHANGELOG.md index 5a8ba91..1a1d892 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,8 @@ after its public API and format compatibility policies are established. ## [Unreleased] +- Current durable-surface documentation distinguishes delivered recovery, fenced authenticated reads and explicit verification from pending ingestion, GC, compaction and general candidate-catalog retained-closure admission (#130). + - Retention verification refuses observed file or symlink substitutions of a selected namespace directory as typed corruption, preserving the original selected root evidence (#114). - Verification preserves typed canonical-namespace and no-follow entry-kind contradictions as corruption while leaving inconclusive observation failures operational (#114). diff --git a/README.md b/README.md index dbe311a..7f75e8f 100644 --- a/README.md +++ b/README.md @@ -73,15 +73,25 @@ Keep is required to refuse all three, before mutating anything. is discarded and rebuilt from freshly verified current intent. Its crash matrix kills real writer processes at 68 before/during/after coordinates (`KEEP-CRASH-053`–`073`), preserving - every version-1 byte. Broader hostile restart combinations remain in #111. + every version-1 byte. The [restart matrix](docs/testing-evidence/migration-restart-matrix.md) records additional hostile-prefix evidence and its limits. -## What it does not do yet - -Version-2 retention recovery, fenced reader snapshots and model-based transitions are implemented in this branch but still require the correctness corrections and independent acceptance review tracked in PR #99. +Version-2 complete-stage retention recovery, fenced retention snapshots and model-based transitions are implemented on main through [PR #99](https://github.com/flyingrobots/keep/pull/99). The retention process-death sequence checks the recovered head generation and exact selected-root bytes before retry; its [evidence receipt](docs/testing-evidence/retention-crash-reader-oracle.md) bounds that claim to the declared initial-publication crash coordinates. -Incomplete retention stages are preserved and block publication pending explicit disposition; automatic disposal is deferred. The [landing ledger](docs/testing-evidence/retention-landing.md) tracks execution-failure reporting and final acceptance under the [approved recovery contract](docs/formats/segment-store-v2/retention-recovery.md). +Recovery failures retain their typed cause and report known effects separately from uncertain effects or durability under the [approved recovery contract](docs/formats/segment-store-v2/retention-recovery.md). + +Writer authority coordinates cooperating writers in a managed namespace; it does not isolate arbitrary concurrent out-of-band filesystem mutation. + +`DurableStore` supplies fenced authenticated whole-blob reconstruction and exact-range reads, delivered through [PR #164](https://github.com/flyingrobots/keep/pull/164); its [read contract](docs/invariants/authenticated-reconstruction/README.md) distinguishes complete-blob and range evidence and separate allocation limits. + +[Explicit verification reports](docs/invariants/verification/README.md) name the subject, requested depth and evidence actually established, delivered through [PR #165](https://github.com/flyingrobots/keep/pull/165). Unsupported depths refuse; a report grants no live retention authority. + +[Reader-fence process-death evidence](docs/testing-evidence/reader-fence-process.md) verifies the lock lifecycle and preserved fence bytes; it is not physical power-loss evidence. + +## What it does not do yet + +Incomplete retention stages are preserved and block publication pending explicit disposition; automatic disposal remains deferred in [#155](https://github.com/flyingrobots/keep/issues/155). Complete orphans remain recovery-protected until explicit disposition lands with garbage collection (#21). @@ -91,9 +101,7 @@ A version-1 store stays admitted until its owner migrates it. | Gap | Tracked | | --- | --- | -| Retention recovery correctness remediation and broader migration corruption coverage | [PR #99](https://github.com/flyingrobots/keep/pull/99), [#111](https://github.com/flyingrobots/keep/issues/111) | -| Fenced reader correctness remediation and independent acceptance | [PR #99](https://github.com/flyingrobots/keep/pull/99) | -| Precise verification reports at explicit depths | [#20](https://github.com/flyingrobots/keep/issues/20) | +| Candidate-catalog preservation of every retained closure | [#125](https://github.com/flyingrobots/keep/issues/125) | | Garbage collection and identity-preserving compaction | [#21](https://github.com/flyingrobots/keep/issues/21) | | Bounded production ingestion through the durable store | [#82](https://github.com/flyingrobots/keep/issues/82) | | Encrypted representations | [#86](https://github.com/flyingrobots/keep/issues/86) | diff --git a/docs/audits/114-durable-verification-scope.md b/docs/audits/114-durable-verification-scope.md index 3d2fc0f..419734c 100644 --- a/docs/audits/114-durable-verification-scope.md +++ b/docs/audits/114-durable-verification-scope.md @@ -1,6 +1,10 @@ # Durable verification landing scope -Status: implementation candidate for [#114](https://github.com/flyingrobots/keep/issues/114), under verification parent [#20](https://github.com/flyingrobots/keep/issues/20). +Status: implemented on main for [#114](https://github.com/flyingrobots/keep/issues/114), under verification parent [#20](https://github.com/flyingrobots/keep/issues/20). + +Delivered in [PR #165](https://github.com/flyingrobots/keep/pull/165), merged as `2efc131e8466b458088eaf5de0a5981e636d8f85`. [Independent review](https://github.com/flyingrobots/keep/pull/165#issuecomment-5974504658), [Code Lawyer closure](https://github.com/flyingrobots/keep/pull/165#issuecomment-5974510981) and all four [candidate checks](https://github.com/flyingrobots/keep/actions/runs/37160522753) cover exact head `1f3991f86fa66783d88b9ac8dbb79ecd0d9a9554`; the signed merge preserves that tree. + +The sections below preserve chronological implementation and review records. Earlier pending gates and open-finding tables describe their named intermediate heads; final closure above supersedes those statuses without expanding their historical evidence claims. This ledger reconciles the requested verification outcome with the code available at the branch baseline; current closure dispositions and their evidence are recorded below. diff --git a/docs/formats/segment-store-v2/README.md b/docs/formats/segment-store-v2/README.md index ddad4ee..4ef7b92 100644 --- a/docs/formats/segment-store-v2/README.md +++ b/docs/formats/segment-store-v2/README.md @@ -6,7 +6,11 @@ catalog, and publication-head byte while adding explicit retention state, reader fences, migration evidence, and reserved GC and recovery-disposition namespaces. -ADR-0009 owns the cross-cutting retention and liveness decision. These pages own its durable representation. The one-way migration, version-two reopen, forward retention publication, partial-prefix migration recovery, and the 68-case migration process-death matrix are implemented with executable evidence. Retention recovery and reader fencing are implemented in this branch; correctness remediation and independent acceptance remain tracked in PR #99. Collection remains planned in #21. The [requirements ledger](requirements.md) records requirements and their evidence status. A version-1 store remains admitted until its owner migrates it. +ADR-0009 owns the cross-cutting retention and liveness decision. These pages own its durable representation. + +The one-way migration, version-two reopen, forward retention publication, partial-prefix migration recovery, complete-stage retention recovery, and fenced retention snapshots are implemented on main through [PR #99](https://github.com/flyingrobots/keep/pull/99). + +The [requirements ledger](requirements.md) records evidence and remaining obligations; the laws below are normative requirements, not a claim that every corresponding runtime surface exists. ## Core laws @@ -89,19 +93,31 @@ head and the catalog it selects, and refuses superseded candidates, retained stages, replaced protocol directories, and every namespace or capacity violation before mutation, each as a typed `RetentionCurrentStateRefusal`. -Retention publication recovery, fenced reader snapshots and model-based transition evidence are implemented in this branch. +Retention publication recovery, fenced retention snapshots and model-based transition evidence are implemented on main. + +The [landing ledger](../../testing-evidence/retention-landing.md) records the accepted scope and evidence for merged PR #99. + +Incomplete retention stages are preserved pending explicit disposition; automatic disposal remains deferred in [#155](https://github.com/flyingrobots/keep/issues/155). + +Execution failures preserve the typed cause and report known effects separately from uncertain effects or durability, as specified by the [retention recovery contract](retention-recovery.md). + +Writer authority coordinates cooperating writers in a managed namespace; it does not isolate arbitrary out-of-band filesystem mutation or make pathname unlink conditional on inode identity. -Their bounded landing and independent acceptance remain tracked in PR #99 and the [landing ledger](../../testing-evidence/retention-landing.md). Incomplete retention stages are preserved pending explicit disposition; automatic disposal is deferred. Execution-failure reporting remains part of the [retention recovery contract](retention-recovery.md). +`FilesystemRetentionSnapshot` binds a catalog, retention head and manifest under a shared reader fence and verifies selected roots on demand. `DurableStore` composes that view with authenticated whole-blob and exact-range reads, delivered in [PR #164](https://github.com/flyingrobots/keep/pull/164); the [read contract](../../invariants/authenticated-reconstruction/README.md) records proof and allocation limits. + +General version-two catalog publication still needs the candidate-catalog retained-closure admission gate tracked in [#125](https://github.com/flyingrobots/keep/issues/125); verifying a new retention root against the current catalog is a different operation. + +[Durable verification reports](../../invariants/verification/README.md) are delivered in [PR #165](https://github.com/flyingrobots/keep/pull/165), with explicit subject/depth evidence and typed non-success outcomes. + +Production durable ingestion ([#82](https://github.com/flyingrobots/keep/issues/82)), garbage collection and compaction ([#21](https://github.com/flyingrobots/keep/issues/21)) remain absent from main; those prepared portions of unmerged PR #107 are not delivered APIs here. The `KEEP-CRASH-036..052` initial-publication process-death sequence includes independent recovered-reader checks; its [evidence receipt](../../testing-evidence/retention-crash-reader-oracle.md) records the assertions, calibration, and scope. Partial-prefix migration recovery and the 68-case `KEEP-CRASH-053..073` -process-death matrix are implemented. Broader migration restart corruption -and compatibility coverage remain in #111 and #112; issue #21 owns garbage -collection. Reopen compares only the restart-stable root coordinates, device -and inode, against the intent; see -[root identity across restart](recovery.md#root-identity-across-restart). A -version-1 store -remains admitted until its owner migrates it, and the -[requirements ledger](requirements.md) is the authority on which requirements -are proven. +process-death matrix are implemented. + +The [migration restart matrix](../../testing-evidence/migration-restart-matrix.md) and [compatibility and fuzz evidence](../../testing-evidence/migration-compatibility-fuzz.md) record the additional merged #111/#112 evidence and its remaining limits. + +Reopen compares restart-stable root coordinates, device and inode, against the intent; see [root identity across restart](recovery.md#root-identity-across-restart). + +A version-1 store remains admitted until its owner migrates it. The [requirements ledger](requirements.md) records implementation and evidence gaps; planned cases are not proof. diff --git a/docs/formats/segment-store-v2/closure.md b/docs/formats/segment-store-v2/closure.md index 3e0e79b..5e4c191 100644 --- a/docs/formats/segment-store-v2/closure.md +++ b/docs/formats/segment-store-v2/closure.md @@ -1,10 +1,6 @@ # Closure Verification -- Status: Normative version-2 protocol; storage-independent verifier - implemented; publication binds this store's catalog head and the catalog it - selects to the verified closure; member re-verification under filesystem - authority is planned in issue - [#19](https://github.com/flyingrobots/keep/issues/19) +- Status: Normative version-2 protocol; the storage-independent verifier and live member re-verification under filesystem retention authority are implemented. General candidate-catalog retained-closure admission remains in [#125](https://github.com/flyingrobots/keep/issues/125). - Format coordinate: `keep.segment-store/v2` - Requirement: [`KEEP-RETENTION-005`](requirements.md#retention-transitions) - Decision record: diff --git a/docs/formats/segment-store-v2/requirements.md b/docs/formats/segment-store-v2/requirements.md index 8cda0a9..09057b0 100644 --- a/docs/formats/segment-store-v2/requirements.md +++ b/docs/formats/segment-store-v2/requirements.md @@ -15,7 +15,7 @@ case is not evidence. | `KEEP-RETENTION-004` | Retain and release compare expected and observed generations and publish exact successors only | unforgeable readiness and preflight proofs in `tests/retention_transition.rs` and `tests/retention_preflight.rs`; exact successor preparation and complete receipt evidence in `tests/retention_publication_preparation.rs` and `tests/retention_publication_execution.rs`; writer-locked initial filesystem publication in `filesystem_retention_storage_tests`; observed-head successor publication, exact predecessor binding, and absent-head refusal in `filesystem_retention_successor_tests`; the store's catalog head must name the closure's catalog generation and digest before any forward write in `filesystem_retention_catalog_tests`; a head whose predecessor disagrees with its manifest refuses in `filesystem_retention_current_tests`; a successor reopens and decodes the manifest-selected predecessor root and refuses an absent or changed one in `filesystem_retention_expectation_tests` | Implemented | | `KEEP-RETENTION-005` | Closure derivation is deterministic, bounded, cycle-safe, fail-closed, and verifies complete blob reconstruction | exact accounting, reconstruction, adversarial-catalog, and exhaustive model laws in `tests/retention_closure.rs`; corrupt members refuse through the inherited segment-record admission laws and seeded `segment_format` fuzz target routed by `closure-corruption.md` | Implemented | | `KEEP-RETENTION-006` | Publication follows the exact ordered durability protocol, including new namespace-directory admission and retention of fixed-stage evidence until head commit, and returns only after cleanup synchronization | typed vocabulary and blocking port in `tests/retention_publication_phase.rs` and `tests/retention_publication_storage.rs`; ordered execution, conditional namespace sync, and all 17 exact storage-fault boundaries in `tests/retention_publication_execution.rs`; production 17-phase forward filesystem execution, exclusive staging, byte-equal inode-substitution refusal, and retained-stage recovery refusal in `filesystem_retention_storage_tests`; orphan namespace directories count against the 4,096 ceiling and refuse a new namespace before any stage is written in `filesystem_retention_capacity_tests`; crash injection remains | In progress in #19 | -| `KEEP-RETENTION-007` | Complete-stage recovery preserves canonical history; incomplete stages require disposition before mutation | Complete publication-prefix recovery and reader-state laws remain; incomplete direct/publication recovery and process-death laws now require typed refusal and preserved bytes. See [landing ledger](../../testing-evidence/retention-landing.md). | Bounded landing in #99; automatic incomplete-stage disposition explicitly deferred | +| `KEEP-RETENTION-007` | Complete-stage recovery preserves canonical history; incomplete stages require disposition before mutation | Complete publication-prefix recovery and reader-state laws remain; incomplete direct/publication recovery and process-death laws now require typed refusal and preserved bytes. See [landing ledger](../../testing-evidence/retention-landing.md). | Implemented under the bounded scope merged in #99; automatic incomplete-stage disposition deferred to #155 | | `KEEP-RETENTION-008` | Readers double-collect catalog and retention heads and bind one complete catalog, manifest, and root-generation view under a `ReaderFence` | `ReaderFence` holds a shared kernel lock on a verified zero-length `reader.lock`; `collect_retention_view` accepts a view only when both head coordinates agree before and after loading and refuses an exhausted attempt limit (`retention_view_collector_tests`); `FilesystemRetentionSnapshot` binds the catalog snapshot, retention head, and manifest under the fence and verifies each selected root on demand while the fence is held, refusing a substituted root and a replaced fence, and two readers share the fence while an exclusive lock waits (`filesystem_retention_snapshot_tests`); Linux public snapshot process laws prove live-reader exclusion, SIGKILL fence release with persistent inode preservation, and kernel-queued new-reader exclusion until collection releases the fence (`tests/reader_fence_process.rs`; [evidence](../../testing-evidence/reader-fence-process.md)) | Implemented | | `KEEP-RETENTION-009` | Exact already-committed retry is idempotent only while its successor remains current | byte-identical planning in `tests/retention_transition.rs`; authority-revalidated zero-mutation retry receipt in `tests/retention_publication_execution.rs`; exact already-committed filesystem retry with a byte-identical retention witness in `filesystem_retention_storage_tests`; superseded-candidate filesystem refusal with zero mutation in `filesystem_retention_successor_tests`; committed retry reopens the head-selected manifest entry and root pool bytes, refusing absent, changed, or corrupt evidence in `filesystem_retention_current_tests`; every refusal is a typed `RetentionCurrentStateRefusal` source, with superseded, committed-root-absent, committed-root-changed, and head-absent-with-artifacts pinned by downcast | Implemented | | `KEEP-RETENTION-010` | Model operation sequences agree with a deterministic namespace-to-anchor-set map and never admit caller identity, paths, clocks, or application policy | all three-operation histories over initial publications of two namespaces, successor, release, restore, byte-identical retry and stale initial (343 histories, each in a fresh migrated store) compare fenced namespace generations, anchor sets and liveness against an operation-derived model after every step; exact typed refusals remain checked. The count describes exploration, not correctness. See [release/restore evidence](../../testing-evidence/retention-release-restore-model.md). Core architecture checks remain separate static evidence. | Implemented; release/restore model coverage added for #128 | @@ -32,7 +32,7 @@ case is not evidence. | `KEEP-MIGRATION-002` | Format marker, intent, and receipt have complete fixed byte tables, named domains, bounds, checksums, deterministic store identity, and exact initial-state digests | exact admission in `tests/store_format_marker.rs`, `tests/store_migration_intent.rs`, and `tests/store_migration_receipt.rs`; canonical construction in `tests/store_migration_intent_encoding.rs` and `tests/store_migration_receipt_encoding.rs`; seeded `migration_format` fuzz target | Implemented | | `KEEP-MIGRATION-003` | Migration revalidates version-1 head, catalog, pools, root identity (all three coordinates within the migrating process; device and file across restart), and writer authority before mutation | bounded canonical pool inventory in `tests/store_migration_inventory.rs`; writer-locked filesystem pool admission in `filesystem_inventory_*_tests`; exact authority observation and drift refusal in `filesystem_migration_authority_tests`; verification-first execution in `tests/store_migration_execution.rs`; fresh filesystem integration and post-publication drift refusal in `filesystem_migration_storage_tests`; a version-one store still holding a retained stage refuses before the intent is observed in `filesystem_migration_storage_tests` | Implemented | | `KEEP-MIGRATION-004` | Every partial migration prefix continues idempotently under writer authority, comparing only the restart-stable root coordinates (device and file) against the persisted intent | restart-stable coordinate laws in `filesystem_version_two_admission_tests` and complete reopen laws in `filesystem_migration_remount_tests`; `tests/store_migration_recovery.rs` and `tests/store_migration_recovery_order.rs` freeze planner and ordering laws; `src/adapters/store_migration/filesystem_migration_recovery_tests.rs` covers every forward prefix; `src/adapters/store_migration/filesystem_migration_recovery_truncation_tests.rs` covers every strict fixed-stage truncation | Implemented in #108 | -| `KEEP-MIGRATION-005` | Unknown, out-of-order, substituted, corrupt, conflicting, or changed evidence is unrecoverable ambiguity | Forward-execution laws remain. Filesystem restart record, pair, ordering, root, pool and namespace laws assert exact existing refusal boundaries and preserve complete names, device/inode identities and bytes; see [restart matrix](../../testing-evidence/migration-restart-matrix.md) for scenarios, calibration, diagnostic limits and validation ownership. | Implemented in #111 candidate, including reserved complete-state namespace refusal; final review and integration pending | +| `KEEP-MIGRATION-005` | Unknown, out-of-order, substituted, corrupt, conflicting, or changed evidence is unrecoverable ambiguity | Forward-execution laws remain. Filesystem restart record, pair, ordering, root, pool and namespace laws assert exact existing refusal boundaries and preserve complete names, device/inode identities and bytes; see [restart matrix](../../testing-evidence/migration-restart-matrix.md) for scenarios, calibration, diagnostic limits and validation ownership. | Evidence delivered in merged PR #161 (#111), including complete-state namespace refusal; see the restart matrix for bounded coverage and separate diagnostic obligations | | `KEEP-MIGRATION-006` | Migration never rewrites or deletes admitted version-1 immutable bytes | exact segment, catalog, and head witnesses in `src/adapters/store_migration/filesystem_migration_storage_tests.rs`, `src/adapters/store_migration/filesystem_migration_recovery_tests.rs`, and `src/adapters/store_migration/filesystem_migration_recovery_truncation_tests.rs`; subprocess restart witnesses in `cargo xtask durability-crash-matrix --sequence migration` | Implemented in #108 | | `KEEP-MIGRATION-007` | Process death around every intent stage, canonical link, namespace prefix, marker stage, receipt stage, cleanup, and synchronization boundary reaches a documented lawful state | ordered phases and capabilities in `tests/store_migration_phase.rs` and `tests/store_migration_storage.rs`; exact phase-failure execution in `tests/store_migration_execution.rs`; production 21-phase forward execution in `filesystem_migration_storage_tests`; `cargo xtask durability-crash-matrix --sequence migration` runs 68 production subprocess cases at `KEEP-CRASH-053..=073`, debug and release | Implemented in #108 | | `KEEP-MIGRATION-008` | Version-1 admission refuses every version-2 or partial-migration artifact after migration begins | Every forward-prefix compatibility law preserves v1 HEAD/catalog/segment bytes and refuses v1 authority after migration effects; public decoder laws pin unsupported versions and every mandatory flag bit; bounded seeded migration parser/recovery-planner fuzzing explores valid and malformed transitions. Existing jointly bound version-two admission and root-identity laws remain. See [compatibility evidence](../../testing-evidence/migration-compatibility-fuzz.md) for coverage, calibration and limits. | Implemented | diff --git a/docs/formats/segment-store-v2/retention-publication.md b/docs/formats/segment-store-v2/retention-publication.md index d432c0f..e696fa2 100644 --- a/docs/formats/segment-store-v2/retention-publication.md +++ b/docs/formats/segment-store-v2/retention-publication.md @@ -20,9 +20,9 @@ the current manifest and derives exact canonical successors. ordered durability phases, and returns the complete receipt only after cleanup; exact already-committed retry revalidates authority and performs no mutation. -Version-2 catalog publication holds the same writer authority and proves every -current retained closure against its candidate catalog before replacing the -catalog `HEAD`. +Version-2 catalog publication must hold the same writer authority and prove every current retained closure against its candidate catalog before replacing the catalog `HEAD`. + +That general candidate-catalog gate remains unimplemented and is tracked in [#125](https://github.com/flyingrobots/keep/issues/125). The implemented retention publication and complete-stage retention recovery paths reverify a root's closure against the current catalog; this does not establish that an arbitrary successor catalog preserves every retained root. ## Generation transition diff --git a/docs/invariants/authenticated-reconstruction/README.md b/docs/invariants/authenticated-reconstruction/README.md index 2d9e8e2..7030352 100644 --- a/docs/invariants/authenticated-reconstruction/README.md +++ b/docs/invariants/authenticated-reconstruction/README.md @@ -1,6 +1,6 @@ # Authenticated Reconstruction Contract -**Status:** Normative for every Keep operation that claims authenticated reconstruction. The public non-durable `ReferenceStore` implements the complete-object and exact-range forms. Linux `DurableStore` and `DurableSnapshot` compose those read cores with fenced version-two catalog and retained-root admission; final #109 acceptance remains recorded in the [evidence ledger](../../testing-evidence/durable-authenticated-reads.md). +**Status:** Normative for every Keep operation that claims authenticated reconstruction. The public non-durable `ReferenceStore` implements the complete-object and exact-range forms. Linux `DurableStore` and `DurableSnapshot` compose those read cores with fenced version-two catalog and retained-root admission; delivery through merged [PR #164](https://github.com/flyingrobots/keep/pull/164) and its #109 acceptance evidence are recorded in the [evidence ledger](../../testing-evidence/durable-authenticated-reads.md). The [rationale](rationale.md) records the governed decisions and rejected alternatives. The [requirement ledger](requirements.md) maps each law to its diff --git a/docs/invariants/authenticated-reconstruction/requirements.md b/docs/invariants/authenticated-reconstruction/requirements.md index ee08a7a..3be1f9f 100644 --- a/docs/invariants/authenticated-reconstruction/requirements.md +++ b/docs/invariants/authenticated-reconstruction/requirements.md @@ -14,7 +14,7 @@ gap, not implementation evidence. | `KEEP-RECONSTRUCT-006` | Authenticated success, evidenced content refusal, and operational failure remain distinct outcomes; operational failure supports no content conclusion. | Typed outcome classification | Public API integration tests and contract inspection | Implemented typed outcomes; persisted refusal receipts remain separate | `tests/streaming_cas/refusal_laws.rs`, `tests/range_read_failures.rs`, `tests/golden_file_worldline/durable_refusal_laws.rs`, `tests/golden_file_worldline/durable_writer_failures.rs` | | `KEEP-RECONSTRUCT-007` | Whole-object and range receipts bind target, exact layout, proof scope, and exact emitted coordinates without granting retention or application authority. | Receipt type inspection | Public API contract tests | Implemented | `src/authenticated_read/reconstruction_receipt.rs`, `src/authenticated_read/range_read_receipt.rs`, `tests/range_read_contract.rs` | | `KEEP-RECONSTRUCT-008` | Automatic layout choice is deterministic; an exact requested layout never falls back. | Canonically ordered layout set | Unit and public API integration tests | Implemented | `src/reference/store_tests.rs`, `tests/streaming_cas/reconstruction_laws.rs` | -| `KEEP-RECONSTRUCT-009` | A durable read pins one immutable view and prevents required evidence from being garbage-collected, deleted, or invalidated through completion. | Pinned-generation and retained-closure model | Recovery, concurrency, corruption, and crash-injection tests | Implemented for cooperating managed-store operations; final #109 acceptance pending | `src/adapters/retention/durable_view_law_tests.rs`, `tests/golden_file_worldline/durable_assertions.rs`; [scope and evidence](../../testing-evidence/durable-authenticated-reads.md) | -| `KEEP-RECONSTRUCT-010` | Durable reconstruction names its view and returns either authenticated success, evidenced refusal, or operational failure without hidden whole-blob allocation. | Durable consumer conformance model | Public API integration, memory, recovery, and crash-injection tests | Implemented; final #109 acceptance pending | `tests/golden_file_worldline/durable_read_memory.rs`, `tests/golden_file_worldline/durable_corruption_laws.rs`, `tests/golden_file_worldline/durable_output_laws.rs`; [scope and evidence](../../testing-evidence/durable-authenticated-reads.md) | +| `KEEP-RECONSTRUCT-009` | A durable read pins one immutable view and prevents required evidence from being garbage-collected, deleted, or invalidated through completion. | Pinned-generation and retained-closure model | Recovery, concurrency, corruption, and crash-injection tests | Implemented for cooperating managed-store operations; delivered through merged #164 | `src/adapters/retention/durable_view_law_tests.rs`, `tests/golden_file_worldline/durable_assertions.rs`; [scope and evidence](../../testing-evidence/durable-authenticated-reads.md) | +| `KEEP-RECONSTRUCT-010` | Durable reconstruction names its view and returns either authenticated success, evidenced refusal, or operational failure without hidden whole-blob allocation. | Durable consumer conformance model | Public API integration, memory, recovery, and crash-injection tests | Implemented; delivered through merged #164 | `tests/golden_file_worldline/durable_read_memory.rs`, `tests/golden_file_worldline/durable_corruption_laws.rs`, `tests/golden_file_worldline/durable_output_laws.rs`; [scope and evidence](../../testing-evidence/durable-authenticated-reads.md) | -The durable implementation status describes the current candidate, not a merged release or completed independent review. Snapshot admission materializes catalog-selected segment bytes under the caller's aggregate segment-byte limit; catalog bytes have a separate format maximum, while decoded indexes and retention records allocate additionally under format and record-count limits. The segment limit is not a total snapshot-memory cap; the memory witness measures additional reconstruction allocation. Managed-namespace cooperation is required for the fence guarantee. The #109 ledger distinguishes fresh reopen from process death and kernel exclusion from absent production GC. +The durable implementation is delivered on main through [PR #164](https://github.com/flyingrobots/keep/pull/164), with exact-head independent review and validation recorded in the linked evidence; mainline delivery is not a packaged-release claim. Snapshot admission materializes catalog-selected segment bytes under the caller's aggregate segment-byte limit; catalog bytes have a separate format maximum, while decoded indexes and retention records allocate additionally under format and record-count limits. The segment limit is not a total snapshot-memory cap; the memory witness measures additional reconstruction allocation. Managed-namespace cooperation is required for the fence guarantee. The #109 ledger distinguishes fresh reopen from process death and kernel exclusion from absent production GC. diff --git a/docs/invariants/verification/README.md b/docs/invariants/verification/README.md index 27db61b..7fcd8b1 100644 --- a/docs/invariants/verification/README.md +++ b/docs/invariants/verification/README.md @@ -1,6 +1,6 @@ # Verification reports -Status: durable verification candidate under [#114](https://github.com/flyingrobots/keep/issues/114); final acceptance is tracked in the [closure ledger](../../audits/114-durable-verification-scope.md). +Status: implemented on main through [PR #165](https://github.com/flyingrobots/keep/pull/165) for [#114](https://github.com/flyingrobots/keep/issues/114); final acceptance and historical scope are recorded in the [closure ledger](../../audits/114-durable-verification-scope.md). ## Contract diff --git a/docs/invariants/verification/requirements.md b/docs/invariants/verification/requirements.md index c927c50..276720f 100644 --- a/docs/invariants/verification/requirements.md +++ b/docs/invariants/verification/requirements.md @@ -4,4 +4,4 @@ The [original task](../../audits/114-durable-verification-scope.md) remains auth | ID | Requirement | Status | Evidence and remaining work | | --- | --- | --- | --- | -| `KEEP-VERIFY-006` | Durable verification at explicit subject-specific achieved depths, with precise refusals, immutable reports and bounded costs. | Implemented on the PR branch | The candidate implements subject-specific catalog, segment, record, blob and retained-namespace reports, typed durable ingress outcomes and bounded conflict evidence; focused runtime/calibration and catalog-ceiling evidence are recorded. Exact-head validation, independent acceptance and mainline integration are recorded on [PR #165](https://github.com/flyingrobots/keep/pull/165); implementation does not imply merged delivery. See the [closure ledger](../../audits/114-durable-verification-scope.md) and [execution evidence](../../testing-evidence/durable-verification.md). | +| `KEEP-VERIFY-006` | Durable verification at explicit subject-specific achieved depths, with precise refusals, immutable reports and bounded costs. | Implemented on main through #165 | The implementation provides subject-specific catalog, segment, record, blob and retained-namespace reports, typed durable ingress outcomes and bounded conflict evidence; focused runtime/calibration and catalog-ceiling evidence are recorded. Exact-head validation, independent acceptance and mainline integration are recorded on [PR #165](https://github.com/flyingrobots/keep/pull/165); the signed merge preserves the independently reviewed candidate tree. See the [closure ledger](../../audits/114-durable-verification-scope.md) and [execution evidence](../../testing-evidence/durable-verification.md). | diff --git a/docs/testing-evidence/current-durable-surfaces.md b/docs/testing-evidence/current-durable-surfaces.md new file mode 100644 index 0000000..cceee73 --- /dev/null +++ b/docs/testing-evidence/current-durable-surfaces.md @@ -0,0 +1,35 @@ +# Current durable surface documentation (#130) + +Change kind: documentation correction; no runtime, public signature, identifier, codec, format byte, or test expectation changes. + +## Source boundary + +The initial reconciliation used main integration `6051abb25a9fd33ae7ee0de5614514b709a4d82a`, which merged PR #99. This landing updates the current claims to main `2efc131e8466b458088eaf5de0a5981e636d8f85`, including delivered authenticated reads (#164/#109), verification (#165/#114), migration restart/compatibility evidence (#161/#111 and #162/#112), and reader-fence process-death evidence (#160/#113). Prepared implementation on `feature/roadmap-m4-tasks` and unmerged audit PRs do not establish delivery on this baseline. The source links below locate each claim; this is a reviewed documentation record, not a source-string assertion masquerading as runtime evidence. + +| Claim | Owning source or evidence | Documentation disposition | +| --- | --- | --- | +| Migration can resume admitted residue under writer authority | [Migration recovery](../../src/adapters/store_migration/filesystem_migration_recovery.rs), [requirements](../formats/segment-store-v2/requirements.md#migration) | Preserve the implemented migration claim and bound additional coverage by the [restart matrix](migration-restart-matrix.md) and [compatibility/fuzz evidence](migration-compatibility-fuzz.md), now merged through #161/#162. These finite receipts do not establish exhaustive hostile-state coverage. | +| Retention recovery executes complete-stage plans and preserves incomplete stages | [Filesystem recovery](../../src/adapters/retention/filesystem_retention_recovery.rs), [planner](../../src/adapters/retention/recovery_planner.rs), [accepted landing evidence](retention-landing.md) | Remove obsolete absence and pending-#99 claims; keep incomplete-stage disposition deferred to #155. | +| Execution failure is distinct from pre-effect refusal | [Execution](../../src/adapters/retention/recovery_execution.rs), [progress](../../src/adapters/retention/retention_storage_progress.rs), [contract](../formats/segment-store-v2/retention-recovery.md) | Preserve typed causes, known/uncertain effects, and durability distinctions. | +| Retention snapshots bind a fenced view and admit selected roots | [Snapshot](../../src/adapters/retention/filesystem_retention_snapshot.rs), [collector](../../src/adapters/retention/retention_view_collector.rs) | Keep the snapshot contract distinct from the now-delivered `DurableStore` composition in #164. [Reader-fence process evidence](reader-fence-process.md) records the merged #160 lifecycle laws, not power-loss guarantees. | +| Forward retention publication and complete-root recovery reverify live closure | [Closure admission](../../src/adapters/retention/filesystem_retention_closure_admission.rs), [forward verification](../../src/adapters/retention/filesystem_retention_catalog.rs) | Remove the stale planned member-reverification claim. | +| General candidate-catalog admission must preserve every retained root | [Catalog preflight](../../src/adapters/filesystem_catalog_current.rs), [catalog storage](../../src/adapters/filesystem_catalog_storage.rs), [publication requirement](../formats/segment-store-v2/retention-publication.md#closure-admission) | Keep the normative requirement; explicitly identify its implementation gap in #125. Current-root verification against the current catalog does not satisfy this different requirement. | +| Durable authenticated reads are delivered | [Durable store](../../src/adapters/durable/store.rs), [snapshot](../../src/adapters/durable/snapshot.rs), [read contract](../invariants/authenticated-reconstruction/README.md), [evidence](durable-authenticated-reads.md) | Describe whole-blob and exact-range proof boundaries, stable locator, fence lifetime and separate allocation limits; #164 supplies mainline integration. | +| Explicit verification reports are delivered | [Report](../../src/verification/report.rs), [durable ingress](../../src/adapters/verification_ingress.rs), [subject/depth matrix](../invariants/verification/README.md), [evidence](durable-verification.md) | Describe named subject/depth evidence and precise non-success outcomes; #165 supplies mainline integration. Reports grant no retention authority or future SnapshotBinding proof. | +| Production durable ingestion, GC and compaction are not delivered | [Crate public surface](../../src/lib.rs), [GC ledger](../formats/segment-store-v2/requirements.md#garbage-collection-reservation) | Track #82 and #21; do not import PR #107's remaining prepared implementations. | + +The managed-namespace concurrency contract remains the [accepted decision](../adr/retention-bounded-recovery-landing.md): cooperating writers under Keep authority, without a guarantee against arbitrary concurrent raw namespace mutation. + +## Validation and limitations + +The oracle is correspondence between public documentation and the source at the named integration, reviewed at each owning boundary above. Documentation integrity/refusal/link checks, Markdown lint, rustdoc generation and doctests validate document structure, references and compiling examples. Formatting and Clippy remain required. Their success does not prove storage behavior. + +No runtime assertion was introduced or materially changed, so RED-on-parent, runtime mutation calibration, fault injection, generated-case reduction and new test resource ceilings are not applicable to this documentation-only change under the [enforcement profile](../testing/enforcement.md). Existing runtime evidence remains attributed to its original source and scope; this documentation change does not expand its behavioral claims. The final PR receipt records commands, source SHA and actual check results. + +No acceptance criterion is reduced. No original ROADMAP task is newly checked. The #130 request for a source-based documentation regression is satisfied by this explicit claim/source reconciliation and existing documentation checks; a permanent test searching for prose or implementation tokens would assert document structure, not Keep runtime behavior. + +## Landing integration + +The normal merge retains both CHANGELOG histories, main's expanded reader-fence evidence, all public exports and runtime implementations, and #130's bounded recovery correction. Crate-level edits affect rustdoc only. The old assertions that #109/#114 and #111/#112/#113 remain undelivered are removed from current status; their historical receipts retain their original source and limits. + +Final validation and independent review are recorded on the resulting PR head. Main's signed #165 integration tree is the runtime reference; documentation checks and compiling doctests cannot prove new storage behavior, and this PR introduces none. diff --git a/docs/testing-evidence/durable-authenticated-reads.md b/docs/testing-evidence/durable-authenticated-reads.md index b8bb17d..4f2fbc9 100644 --- a/docs/testing-evidence/durable-authenticated-reads.md +++ b/docs/testing-evidence/durable-authenticated-reads.md @@ -1,6 +1,8 @@ # Durable authenticated reads (#109) -Change kind: new read API with a shared-core extraction. The baseline is main `6051abb25a9fd33ae7ee0de5614514b709a4d82a`. No write protocol or on-disk format is changed; #125's candidate-catalog publication gate is not added or bypassed. This is an in-progress implementation ledger, not acceptance of #109. +Change kind: new read API with a shared-core extraction. The baseline is main `6051abb25a9fd33ae7ee0de5614514b709a4d82a`. No write protocol or on-disk format is changed; #125's candidate-catalog publication gate is not added or bypassed. The sections below preserve chronological implementation and review evidence; earlier pending gates describe their historical heads and are superseded by this final landing receipt. + +Delivered in [PR #164](https://github.com/flyingrobots/keep/pull/164), merged as `1079551bc6b331eb9847823e7d22b22ea4c47b62`. [Independent review](https://github.com/flyingrobots/keep/pull/164#issuecomment-5974246110), [Code Lawyer closure](https://github.com/flyingrobots/keep/pull/164#issuecomment-5974261856), all four [candidate checks](https://github.com/flyingrobots/keep/actions/runs/37158962636) and [post-merge checks](https://github.com/flyingrobots/keep/actions/runs/37159373928) pass; reviewed head `9d19e2e0c3184efd5bc08c1f8cc12edd15421a93` and signed merge have the same tree. ## Review after draft readiness diff --git a/docs/testing-evidence/durable-verification.md b/docs/testing-evidence/durable-verification.md index cc66265..15355e5 100644 --- a/docs/testing-evidence/durable-verification.md +++ b/docs/testing-evidence/durable-verification.md @@ -1,6 +1,8 @@ # Durable verification evidence -Status: the scoped implementation and post-readiness review corrections for [#114](https://github.com/flyingrobots/keep/issues/114) are implemented on the PR branch. Final exact-head independent review and required checks remain acceptance gates in the [scope ledger](../audits/114-durable-verification-scope.md); mainline delivery is not claimed. The sections below preserve chronological slice evidence, including superseded intermediate limitations. +Status: Delivered in [PR #165](https://github.com/flyingrobots/keep/pull/165), merged as `2efc131e8466b458088eaf5de0a5981e636d8f85`. [Independent review](https://github.com/flyingrobots/keep/pull/165#issuecomment-5974504658), [Code Lawyer closure](https://github.com/flyingrobots/keep/pull/165#issuecomment-5974510981) and all four [candidate checks](https://github.com/flyingrobots/keep/actions/runs/37160522753) cover exact head `1f3991f86fa66783d88b9ac8dbb79ecd0d9a9554`; the signed merge preserves that tree. + +The sections below preserve chronological slice and intermediate review evidence. Earlier pending acceptance statements describe those historical heads and are superseded by the final landing receipt above; no broader runtime or power-loss guarantee is inferred. ## Catalog report slice diff --git a/src/lib.rs b/src/lib.rs index e22dea9..74ef5b5 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -40,11 +40,25 @@ //! namespace transitions while retaining version-1 immutable bytes. //! Partial-prefix recovery now plans and resumes lawful migration residue, //! returning typed refusals and an ordered execution receipt. Filesystem -//! retention publication, bounded restart recovery and fenced snapshots are -//! available. [`DurableStore`] composes a fenced snapshot with authenticated -//! reconstruction and exact-range reads; its snapshot allocation policy is -//! explicit. Garbage collection remains absent. Complete durable read-law and -//! Worldline acceptance remains tracked in issue #109. +//! retention publication and complete-stage restart recovery are available. +//! +//! Incomplete retention stages require explicit disposition before recovery +//! effects; automatic disposal remains deferred. Execution failures preserve +//! their typed cause and distinguish known effects from uncertain effects or +//! durability. Writer authority coordinates cooperating writers in a managed +//! namespace; it does not isolate arbitrary out-of-band filesystem mutation. +//! +//! [`FilesystemRetentionSnapshot`] binds a catalog, retention head and manifest +//! under a shared reader fence and verifies selected roots on demand. +//! [`DurableStore`] composes that fenced view with authenticated reconstruction +//! and exact-range reads under explicit snapshot allocation limits. +//! [`VerificationReport`] names the subject, requested depth and established +//! evidence; unsupported depths refuse without downgrade or a partial report. +//! +//! Production durable ingestion, garbage collection and compaction remain +//! unimplemented here. General version-2 catalog publication still needs a +//! retained-closure gate; retention publication's live closure verification +//! does not establish that an arbitrary successor catalog preserves every root. #[cfg(test)] extern crate self as keep;