Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
945f06c
docs: reconcile durable verification landing scope (#114)
flyingrobots Oct 2, 2026
92f9208
feat: report subject-specific admitted catalog evidence (#114)
flyingrobots Oct 2, 2026
d65c845
feat: report admitted segment and logical-record evidence (#114)
flyingrobots Oct 3, 2026
b204f83
feat: verify complete blob and retained root evidence (#114)
flyingrobots Oct 3, 2026
6504c86
feat: preserve durable verification outcomes and view evidence (#114)
flyingrobots Oct 3, 2026
28c9417
test: map existing corruption laws through verification outcomes (#114)
flyingrobots Oct 3, 2026
b33c7da
test: reject corruption labels for verification resource limits (#114)
flyingrobots Oct 3, 2026
26d3522
fix: restrict root catalog provenance to verified closure (#114)
flyingrobots Oct 3, 2026
665ffb3
fix: retain corruption classification at store admission (#114)
flyingrobots Oct 3, 2026
2b28c4a
fix: prevent ordinal comparisons of verification depths (#114)
flyingrobots Oct 3, 2026
6802644
refactor: centralize exhaustive verification failure classes (#114)
flyingrobots Oct 3, 2026
cc1e37b
test: calibrate exact admission diagnostic assertions (#114)
flyingrobots Oct 3, 2026
c003c89
fix: reject unsupported retention requests before reads (#114)
flyingrobots Oct 3, 2026
27d5934
fix: preserve selected-root wrong-kind corruption (#114)
flyingrobots Oct 3, 2026
80afd11
merge: preserve durable read contracts in verification integration (#…
flyingrobots Oct 3, 2026
90b9af3
fix: retain missing selected segment identity in verification (#114)
flyingrobots Oct 3, 2026
c544e2d
fix: classify evidenced namespace contradictions precisely (#114)
flyingrobots Oct 3, 2026
f2098ef
fix: retain typed selected namespace kind refusals (#114)
flyingrobots Oct 3, 2026
1f3991f
test: reconcile selected-segment diagnostic expectations for #114
flyingrobots Oct 3, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,30 @@ after its public API and format compatibility policies are established.

## [Unreleased]

- Retention verification refuses observed file or symlink substitutions of a selected namespace directory as typed corruption, preserving the original selected root evidence (#114).

- Verification preserves typed canonical-namespace and no-follow entry-kind contradictions as corruption while leaving inconclusive observation failures operational (#114).

- Filesystem verification names the missing catalog-selected segment and preserves its digest, failing I/O phase and original cause instead of labeling the present catalog missing (#114).

- Selected-root verification preserves observed non-regular file evidence as typed corruption instead of an inconclusive no-follow-open error (#114).

- Published retention verification rejects unsupported depths before reading selected root evidence, even when that evidence is missing (#114).

- Removed ordinal comparison traits from verification depths; callers use each subject’s explicit supported-depth set (#114).

- Verification classifies typed migration-record and root-identity contradictions as corruption while preserving admission causes and keeping resource or unclassified I/O failures operational (#114).

- Shallow retention-root verification reports no longer claim unused catalog provenance; catalog coordinates are attached only after successful closure verification (#114).

- Added read-only durable verification ingress, exact moving-view conflict witnesses, selected-namespace root reports, and typed selected-root diagnostic sources (#114).

- Added complete-blob and admitted-retention-root verification reports, with catalog provenance for checked blob evidence and successful root closure, preserving exact missing members, identity/profile contradictions, resource failures, and original typed causes (#114).

- Added allocation-free verification reports for already-admitted physical segments and logical chunk/layout records, with explicit subject-specific supported depths (#114).

- Admitted catalog snapshots can report explicit framing, checksum, or catalog-reachability evidence with immutable subject coordinates; unsupported requests refuse without claiming blob completeness or retention closure (#114).

- Durable read diagnostics render their boundary once while preserving each original typed source for error-chain reporters (#109).

- Caller-supplied malformed durable layout records now preserve the reconstruction or range input-decode boundary instead of reporting committed-layout corruption (#109).
Expand Down
133 changes: 133 additions & 0 deletions docs/audits/114-durable-verification-scope.md

Large diffs are not rendered by default.

86 changes: 86 additions & 0 deletions docs/invariants/verification/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
# Verification reports

Status: durable verification candidate under [#114](https://github.com/flyingrobots/keep/issues/114); final acceptance is tracked in the [closure ledger](../../audits/114-durable-verification-scope.md).

## Contract

A report names the exact subject, the caller's requested depth and the evidence established for that subject.

`VerificationDepth` has equality but no `Ord` or `PartialOrd`: a catalog membership check cannot certify a blob, and a layout identity cannot certify its missing chunks.

Report fields and construction are private; callers can inspect or copy established evidence but cannot construct or deepen it.

Reports contain no plaintext, keys or filesystem paths and convey no publication, retention authority, reader fence or assurance that physical bytes still exist later.

## Current subject and depth matrix

| Entry point | Subject | Supported depths | Scope of the proof |
| --- | --- | --- | --- |
| `AdmittedSegment::verify` | Exact physical segment digest | `Framing`, `Checksum` | The admitted immutable segment's physical representation; logical claims require record-specific reports. |
| `AdmittedSegmentRecord::verify` for a chunk | Exact `ChunkId` | `Framing`, `Checksum`, `ChunkIdentity` | That record's complete chunk bytes; no blob or profile-boundary claim. |
| `AdmittedSegmentRecord::verify` for a layout | Exact `LayoutId` | `Framing`, `Checksum`, `LayoutIdentity` | Canonical layout bytes and identity; no requirement that referenced chunks exist. |
| `CatalogSnapshot::verify_blob` | Named logical blob in one catalog | `Framing`, `Checksum`, `ChunkIdentity`, `LayoutIdentity`, `CompleteBlobIdentity` | Canonical layout discovery; chunk presence at chunk depth; full profile replay and logical hash at complete depth. |
| `AdmittedRetentionRoot::verify` | Exact namespace, root generation and root digest | `Framing`, `Checksum`, `RetentionClosure` | Closure depth checks every anchor against the supplied catalog and its admitted limits. |
| `CatalogSnapshot::verify` | Selected catalog generation and digest | `Framing`, `Checksum`, `CatalogReachability` | Exact catalog-to-record bindings; no complete logical or retained closure claim. |

Every other depth returns `VerificationRefusal::Unsupported` (wrapped by `VerificationError` for blob/root operations) with the exact subject, request and supported set; the operation neither downgrades the request nor returns a success report.

`FilesystemRetentionSnapshot::verify_retention` accepts the same depths as direct root verification. Unsupported requests refuse with the requested namespace before reading its selected root or re-admitting the catalog, including when the namespace or root evidence is absent. Loading the fenced snapshot is a separate operation with its own admission failures.

Selected-root observation rejects a non-regular file, including a symlink to valid root bytes, with a typed kind refusal classified as corruption. Subsequent opens still follow no links and check the opened file; the preliminary kind observation provides no isolation guarantee against concurrent raw namespace substitution.

`SnapshotBinding` remains unsupported until its separate protocol exists; catalog/retention coordinates must not be mislabeled as that future proof.

## Costs and admission boundary

Reporting physical segment, logical record and catalog evidence is constant time and allocation-free; shallow root reporting has the same costs. Blob discovery decodes catalogued layouts in canonical identity order, retaining at most one decoded layout at a time. Chunk verification looks up every referenced member; complete blob verification additionally streams every selected byte through profile replay and complete identity calculation. Retention closure uses its existing checked limits, an ordered member index bounded by the root node limit, and one decoded layout at a time. These operations perform no I/O, mutate no persistent bytes, and synchronize nothing.

These costs exclude prerequisite admission: segment admission verifies all records, checksums and identities with bounded duplicate-detection allocation; layout record admission may allocate bounded layout metadata; catalog admission binds its entries to admitted segment records.

A framing request on already admitted evidence still requires that stronger admission to have succeeded first; these APIs are not shallow raw-byte scans that tolerate deeper corruption.

Records prepared for publication provide the same logical proof over their canonical representation without asserting that the record has been written or made durable.

## Durable ingress and view collection

`verify_segment` admits raw segment bytes before reporting; `verify_catalog_bytes` admits the supplied publication head, catalog and selected segments before reporting.

`FilesystemCatalogSnapshot::load_for_verification` reads the exact selected artifacts under `CatalogRestartPolicy`, and its `verify` and `verify_blob` methods re-admit owned bytes before reporting.

`FilesystemRetentionSnapshot::load_for_verification` holds the existing shared fence and uses bounded before/load/after collection; `verify_retention` reads and verifies only the manifest-selected root for the supplied namespace digest, checks namespace/generation/digest, and establishes the requested root evidence against that same catalog.

Filesystem loading blocks on reads and fence acquisition; the owner retains caller-bounded segment bytes plus protocol-bounded catalog/manifest data, while reporting may rebuild the bounded catalog indexes.

Selected-root verification holds one bounded root buffer, decoded anchors and catalog indexes; closure adds its checked node-bounded member index and one decoded layout at a time, with no whole-blob output buffer.

These operations do not publish, repair, run recovery, or acquire writer authority; a retained incomplete stage is not disposed of by verification. Opening a retention verification snapshot uses the shared production filesystem admission path, including its fallible root-directory synchronization probe; verification over an already admitted snapshot performs no synchronization.

Exhausted moving-view collection returns `Ambiguous` with the actual last before/after catalog and retention coordinates; no partial view or report is returned.

A failed observation is operational unless its retained typed cause establishes a precise missing artifact or content contradiction; classification never parses error prose.

Each named original interface verifies one requested subject and returns one `VerifiedSubject`; traversal of a blob's chunks or a root's anchors establishes that subject's depth, without manufacturing separate reports for its dependencies.

This satisfies the original per-subject interface contract; it is not a whole-store enumeration or aggregate-report API, and the earlier work-in-progress references to a required aggregate operation were broader than the original named interfaces.

## Catalog-ceiling memory boundary

The catalog-ceiling runtime law supplies 1,048,576 distinct chunk records and requires exact sample lookups plus a `CatalogReachability` report within 1 GiB (1,073,741,824 bytes) of incremental tracked live allocations during catalog/head admission, lookups and reporting.

This bound excludes caller-owned encoded segment/catalog buffers, fixture construction, segment admission, allocator bookkeeping and process RSS; it is not a total-process memory promise.

Filesystem owners additionally retain the selected segment bytes up to their explicit `CatalogRestartByteLimit`, protocol-bounded catalog bytes and admission indexes; those owners must be included when sizing a verification process.

No serialization, repair, quarantine, GC execution or new durable report format is introduced here.

[Evidence and calibration](../../testing-evidence/durable-verification.md) distinguish runtime laws, static/API restrictions and final acceptance checks.

## Logical refusal contract

Blob and root verification distinguish missing catalog members, demonstrated contradictions, unsupported requests, and operational failures without returning partial reports. Original layout or closure causes retain their typed coordinates. Resource exhaustion is operational, not evidence of corruption.

The report preserves the catalog generation/digest used by catalog and blob operations or successful root closure; root framing/checksum reports carry no catalog provenance; this provenance is not a live fence. Multiple valid layouts for a blob are representations, not automatically ambiguity: discovery selects the first canonical identity.

Immutable admitted-view operations have no moving observation to classify; ambiguity is produced by the durable collection path, retaining at most the last conflicting coordinate pair and the original attempt-limit cause.

Raw layout/root decoder errors also convert into `VerificationError` without losing their typed causes; these conversions name unadmitted input subjects and cannot manufacture a report.
5 changes: 5 additions & 0 deletions docs/invariants/verification/rationale.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# Verification boundary rationale

## Typed namespace observations

Canonical membership and no-follow entry-kind observations belong to filesystem namespace admission, including the earlier platform directory traversal. Their typed source records a demonstrated contradiction; an I/O error alone does not. Verification consumes that evidence rather than parsing messages or broadly equating InvalidData, ELOOP or NotADirectory with corruption. Failed directory iteration remains operational before any membership inference. The shared guard leaves existing no-follow opens and opened-file checks intact and does not make later pathname operations conditional on inode identity. Catalog-selected segment I/O similarly retains its known digest and original cause, so absence identifies the missing evidence without renaming the present catalog. These additive diagnostic types enrich the public error surface; downstream exhaustive matches may need new arms, without changing durable formats or successful behavior.
7 changes: 7 additions & 0 deletions docs/invariants/verification/requirements.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Verification requirements

The [original task](../../audits/114-durable-verification-scope.md) remains authoritative; partial subject reporting does not satisfy the full durable requirement.

| ID | Requirement | Status | Evidence and remaining work |
| --- | --- | --- | --- |
| `KEEP-VERIFY-006` | Durable verification at explicit subject-specific achieved depths, with precise refusals, immutable reports and bounded costs. | Implemented on the PR branch | The candidate implements subject-specific catalog, segment, record, blob and retained-namespace reports, typed durable ingress outcomes and bounded conflict evidence; focused runtime/calibration and catalog-ceiling evidence are recorded. Exact-head validation, independent acceptance and mainline integration are recorded on [PR #165](https://github.com/flyingrobots/keep/pull/165); implementation does not imply merged delivery. See the [closure ledger](../../audits/114-durable-verification-scope.md) and [execution evidence](../../testing-evidence/durable-verification.md). |
Loading
Loading