Skip to content

Fix: validate completed migration namespaces and restart evidence (#111) - #161

Merged
flyingrobots merged 9 commits into
mainfrom
test/111-migration-restart-ambiguity
Oct 3, 2026
Merged

flyingrobots merged 9 commits into
mainfrom
test/111-migration-restart-ambiguity

Conversation

@flyingrobots

@flyingrobots flyingrobots commented Oct 2, 2026 •

Copy link
Copy Markdown
Owner

Problem and outcome

Migration restart lacked runtime evidence for several ambiguity classes. The expanded matrix also exposed a production defect: a completed migration returned success with an unexpected entry in a reserved namespace. This PR adds exact refusal and preserved-evidence laws, then requires effect-free namespace admission before returning Complete.

Change kind: missing runtime verification plus a demonstrated bug fix. The permanent regression was observed RED on unfixed production in 6351c7c; a3ea6c3 fixes it. The weaker entry-count-only corrupt-intent test is replaced by filesystem observations of names, identities, bytes, directories and symbolic-link targets. No test-count assertion is introduced.

Invariant and approach

A completed migration must still satisfy its reserved namespace contract. MigrationRecoveryStorage::verify_complete is a required read-only capability; the filesystem adapter uses existing version-two namespace admission and preserves the typed Observation → NamespacePreflight cause. Planning refusals retain precedence. Lawful published retention state remains admissible and has a real filesystem success law.

Using the incomplete-migration empty-retention-pool policy for completion was rejected because it would refuse lawful later retention state. Blind acceptance and automatic cleanup were rejected because they cannot establish the invariant. The #99 incomplete-retention-stage disposition deferral remains unchanged.

Evidence and current gate

The consolidated evidence maps the record, identity, ordering, namespace, immutable-pool and root-authority laws, historical controls, new regression and finite diagnostic calibration batch. Controls exercise the actual restart and unchanged preservation witness before perturbing observed diagnostics. Compilation failures and controls that failed at an earlier assertion are explicitly excluded from their intended calibration claims.

The full required copied-Docker validation chain and hosted checks passed at code candidate a3ea6c3e2ac58ca82a5c26f43d902ba1ed883be7. Receipt-only successor e20629852f402e129d889e398014781d20d6ff0b received independent APPROVE, closing both original review findings, but its hosted Rust check failed at the existing reader-fence setup with Busy. The first failure is preserved. Independently reviewed PR #175 repaired that fixture handoff and merged as 1c2b9d788fd4029d2469d2651faf0f8db2e0869e after all final checks passed. Current candidate 2f22d0d9097820503d2a81085bb748273de9f56d normally merges that mainline repair, preserving both changelog entries. Its full copied-Docker validation passes at exact tracked tree 658e4fc920a4584ec1153c4c30f45af1b295427d, and all four required jobs in final run 37154563204 pass on that exact candidate. Independent integration review verifies both-parent merge semantics and is recorded in the final review comment. Earlier approvals/checks are not substituted for these final integration gates.

Compatibility, failure modes and limits

The new required recovery-port method is a source compatibility change for external trait implementors. On-disk bytes, formats, dependencies and existing partial-migration recovery policy are unchanged. Completion now refuses invalid reserved namespace membership without initiating recovery mutations. Namespace admission is not certification of every retained artifact's content; existing retention recovery owns that evidence.

Tests use repository-only platform admission, real filesystem restart boundaries and deterministic fault inputs. They do not establish production platform eligibility, physical power-loss behavior or isolation from arbitrary concurrent raw namespace changes. Before/after equality does not exclude transient changes restored before observation. Existing process-death and forward-success campaigns remain separate evidence owners. Ordinary-test resource enforcement limitations are recorded.

No benchmark improvement is claimed. Completion adds bounded namespace admission I/O; no content identity or security boundary is weakened. Broader diagnostic redesign under #110 remains separate. Original roadmap checkboxes are unchanged; mainline delivery requires integration.

Closes #111. Refs #131, #132.

@coderabbitai

coderabbitai Bot commented Oct 2, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: 0f1576fa-a5f5-4742-84ad-29502db0a240
📥 Commits

Reviewing files that changed from the base of the PR and between 5cb758a and 2f22d0d.

📒 Files selected for processing (24)
  • CHANGELOG.md
  • docs/formats/segment-store-v2/migration-recovery.md
  • docs/formats/segment-store-v2/rationale.md
  • docs/formats/segment-store-v2/requirements.md
  • docs/testing-evidence/migration-restart-matrix.md
  • docs/testing-evidence/migration-restart-matrix/calibration-restored-green.txt
  • docs/testing-evidence/migration-restart-matrix/complete-fix-green.txt
  • docs/testing-evidence/migration-restart-matrix/complete-namespace-probe-red.txt
  • docs/testing-evidence/migration-restart-matrix/complete-namespace-probe.patch
  • docs/testing-evidence/migration-restart-matrix/complete-regression-red.txt
  • docs/testing-evidence/migration-restart-matrix/diagnostic-red.txt
  • docs/testing-evidence/migration-restart-matrix/diagnostic.patch
  • docs/testing-evidence/migration-restart-matrix/lawful-retention-red.txt
  • docs/testing-evidence/migration-restart-matrix/lawful-retention.patch
  • docs/testing-evidence/migration-restart-matrix/nested-diagnostic-red.txt
  • docs/testing-evidence/migration-restart-matrix/nested-diagnostic.patch
  • docs/testing-evidence/migration-restart-matrix/pool-diagnostic-red.txt
  • docs/testing-evidence/migration-restart-matrix/pool-diagnostic.patch
  • src/adapters/retention.rs
  • src/adapters/retention/filesystem_retention_migration_completion_tests.rs
  • src/adapters/store_migration/filesystem_migration_recovery.rs
  • src/adapters/store_migration/filesystem_migration_restart_namespace_tests.rs
  • src/adapters/store_migration/migration_recovery_execution.rs
  • src/adapters/store_migration/migration_recovery_storage.rs

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

📜 Recent review details
⏰ Context from checks skipped due to timeout. (4)
  • GitHub Check: Runtime fuzz smoke
  • GitHub Check: Dependency policy
  • GitHub Check: Rust quality gates
  • GitHub Check: Documentation and workflow integrity
🧰 Additional context used
🪛 LanguageTool
CHANGELOG.md

[grammar] ~11-~11: Ensure spelling is correct
Context: ...etention state (#111). Recovery storage implementors must supply the new read-only `verify_c...

(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)

docs/testing-evidence/migration-restart-matrix.md

[grammar] ~22-~22: Use a hyphen to join words.
Context: ... this change does not claim to add finer typed namespace coordinates or complete ...

(QB_NEW_EN_HYPHEN)


[style] ~30-~30: ‘new record’ might be wordy. Consider a shorter alternative.
Context: ...on “subsumed by stronger evidence”: the new record matrix preserves its exact checksum ref...

(EN_WORDINESS_PREMIUM_NEW_RECORD)


[style] ~30-~30: ‘new record’ might be wordy. Consider a shorter alternative.
Context: ... and bytes. Remaining risk lives in the new record law and the complete witness, not in a ...

(EN_WORDINESS_PREMIUM_NEW_RECORD)


[grammar] ~54-~54: Ensure spelling is correct
Context: ...xternal StoreMigrationRecoveryStorage implementors. Retention content/stage semantics rema...

(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)

🔇 Additional comments (9)
src/adapters/store_migration/migration_recovery_storage.rs (1)

27-35: LGTM!

src/adapters/store_migration/filesystem_migration_recovery.rs (1)

86-89: LGTM!

src/adapters/store_migration/migration_recovery_execution.rs (1)

77-77: LGTM!

Also applies to: 120-123, 147-151

src/adapters/store_migration/filesystem_migration_restart_namespace_tests.rs (1)

14-190: LGTM!

src/adapters/retention.rs (1)

278-279: LGTM!

docs/formats/segment-store-v2/rationale.md (1)

120-125: LGTM!

docs/formats/segment-store-v2/migration-recovery.md (1)

47-47: LGTM!

Also applies to: 56-56, 98-102

CHANGELOG.md (1)

11-11: LGTM!

Also applies to: 13-13

src/adapters/retention/filesystem_retention_migration_completion_tests.rs (1)

18-18: 📐 Maintainability & Code Quality

The concern is refuted. TestDirectory::create includes the process ID in the sandbox path, removes stale state at that path before creating it, and Drop attempts recursive cleanup. The test does not need an explicit store.remove() call.


Summary by CodeRabbit

  • New Features
    • Completed migration recovery now verifies the version-two namespace before reporting success, refuses unknown reserved entries without making changes, and preserves valid published retention state.
  • Documentation
    • Updated migration recovery guidance and added a reference matrix covering restart scenarios, refusal boundaries, and evidence limitations.
  • Tests
    • Expanded filesystem migration restart coverage for invalid records, unexpected entries, ordering issues, and mismatched roots or inventories.
    • Tests verify that refusals preserve filesystem evidence and report relevant error details.

Walkthrough

Completed migration recovery now verifies version-two namespace admission before reporting success. The storage interface requires a read-only verify_complete operation. Unix filesystem tests cover restart refusals and preservation of filesystem evidence. Documentation and test records describe the contract and validation results.

Changes

Migration recovery and restart verification

Layer / File(s) Summary
Completed namespace admission
src/adapters/store_migration/migration_recovery_storage.rs, src/adapters/store_migration/migration_recovery_execution.rs, src/adapters/store_migration/filesystem_migration_recovery.rs, src/adapters/store_migration/filesystem_migration_restart_namespace_tests.rs, src/adapters/retention/*, docs/formats/segment-store-v2/rationale.md, CHANGELOG.md
Completed recovery plans call the new read-only verify_complete operation. Namespace failures become observation errors. Tests cover unknown reserved entries and preservation of published retention state.
Restart refusal checks and filesystem witnesses
src/adapters/store_migration.rs, src/adapters/store_migration/filesystem_migration_restart_*, src/adapters/store_migration/filesystem_migration_recovery_tests.rs
Adds Unix-only tests for stage and record changes, ordering, namespace and pool contents, and root identity. Shared fixtures compare filesystem witnesses before and after refusals. Removes the earlier corrupt-intent test.
Recovery contract and test evidence
docs/formats/segment-store-v2/*, docs/testing-evidence/migration-restart-matrix*, docs/testing-evidence/migration-restart-matrix/*, CHANGELOG.md
Updates migration requirements and recovery documentation. Adds a restart evidence matrix and test logs covering validation results, calibration, and evidence limitations.

Priority: ⚪ Not assessed

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Bug fix

Merge Risk: ⚪ Minimal · up to 2f22d

No actionable issue or failing Rust check is established for this head. Normal merge checks can proceed.

Security Architecture Review

Security architecture risk: 🔵 Low · up to 2f22d

The change strengthens completed-migration validation without expanding filesystem authority. Invalid reserved entries now prevent success, while lawful retention state remains permitted. Remaining uncertainty concerns downstream storage implementations and behavior outside the documented writer-coordination and restart guarantees.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • inferred — The added check operates on the migration authority's existing root capability and accepts no new caller-supplied path. Its directly evidenced exposure is the already-authorized store root and reserved descendants, not newly granted cross-store or service authority. Raw concurrent mutation outside writer coordination remains outside the established guarantee.

Security Findings and Attack Paths

  • observed — The completed-state regression inserts unexpected entries into gc, recovery, and recovery/dispositions and expects Observation containing NamespacePreflight with InvalidData. The new gate turns those invalid namespaces into refusal rather than successful completion, without adopting or cleaning the unexpected entries.

Trust Boundaries and Controls

  • observed — Production recovery acquires the existing writer lock and captures root identity before constructing its authority. The added verification reuses that root. Namespace enforcement checks entry kinds with symlink metadata, opens protocol directories without following links, and rejects noncanonical membership.

Resilience and Maintainability Implications

  • observed — The filesystem implementation performs admission without mutation and returns the original refusal through the existing error wrappers. Refusal therefore preserves recovery evidence instead of attempting automatic cleanup; the documented restart laws pair typed failure boundaries with filesystem-state witnesses.
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 64.10% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 39 functions across 14 files. (18 skipped… Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly identifies the main change: validating completed migration namespaces and improving restart evidence.
Description check ✅ Passed The description covers the problem, invariant, approach, rejected alternatives, failure modes, testing evidence, compatibility, recovery, security, and benchmark impact. It omits the template’s explic…
Full details: Docstring Coverage

Explanation

Docstring coverage is 64.10% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 39 functions across 14 files. (18 skipped: 18 unsupported.)

  • Fix all pre-merge checks with AI
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

The stages line up, or recovery says no
A witness keeps names, bytes, and links in view
The namespace gets checked before “complete”
Retention stays recorded through the test
Green logs mark the paths that passed
The restart rules are written down at last

Comment @coderabbitai help to get the list of available commands.

@flyingrobots
flyingrobots marked this pull request as ready for review October 2, 2026 20:38
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@flyingrobots

Copy link
Copy Markdown
Owner Author

Code Lawyer: complete-plan namespace acceptance remains unproved

Severity Location Verified scenario Required disposition
P2 completion/contract gap migration_recovery_execution.rs:149-155; filesystem_migration_residue.rs:54-86; docs/formats/segment-store-v2/requirements.md:35 On exact candidate ff6f5be4b98e985e00601ab3e854a95e87aa9b34, execute migration through RemoveReceiptStage, add gc/unexpected, then reopen and recover. Recovery succeeds. The new namespace law covers only the earlier AdmitNamespacePrefix state; its complete-state counterpart fails with ambiguous migration restarted successfully. Do not mark KEEP-MIGRATION-005 / #111 complete from this evidence. Resolve complete-plan namespace admission under the existing contract, preserve lawful post-migration retention state and precise refusal ordering, and add a permanent runtime regression. A scope reduction requires explicit maintainer acceptance; prose alone cannot establish the original criterion.

This is a compiled runtime probe in an isolated source/build copy, with unchanged production code. It changes only the new namespace law's selected directory to gc and forward prefix to RemoveReceiptStage. Exit 101 names the intended law and successful recovery, not a compiler or setup error. Candidate source/tree: ff6f5be4b98e985e00601ab3e854a95e87aa9b34 / de06577722b63ba09d425f31dfa7a1acd647e128.

The complete fast path returns before adoption/preflight. Nested residue observation checks expected directory kinds/presence, not unknown membership. The documented pre-mutation namespace guarantee for resumed recovery remains supported; this finding does not allege mutation during Complete. It concerns the broader unknown-evidence completion claim and #111 acceptance scope. Do not indiscriminately require empty retention pools on completed stores or reopen #99's separate retention-stage disposition decision.

PR #161 remains unmerged pending this obligation and the independent review's finite calibration assessment. @codex — independent confirmation welcome.

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create an environment for this repo.

@flyingrobots

Copy link
Copy Markdown
Owner Author

Independent ULTRA STRICT review — PR #161

Repository: flyingrobots/keep. Branch: test/111-migration-restart-ambiguity. Reviewed head: ff6f5be4b98e985e00601ab3e854a95e87aa9b34. Reviewed tracked tree: de06577722b63ba09d425f31dfa7a1acd647e128. Target: main 34d70909b0cd93f6b020a59d4d40f43b07971cd8. Original topic head: 5cb758a857d88308214f40578b46776b3267895e; original base: 6051abb25a9fd33ae7ee0de5614514b709a4d82a.

This is the user-authorized independent Codex fallback, applying the mandatory agy-review protocol; it is not represented as an agy execution. Review was read-only in the isolated candidate checkout. No host Rust execution, source changes, publication, configuration changes, or subagents. Only this report was written. Source head/tree and clean working state were rechecked before disposition.

Findings

P2 — KEEP-MIGRATION-005 is marked implemented despite a reproduced completed-restart unknown-entry gap

Changed locations: docs/formats/segment-store-v2/requirements.md:35, docs/testing-evidence/migration-restart-matrix.md:17; relevant test coverage: src/adapters/store_migration/filesystem_migration_restart_namespace_tests.rs:14-46.

The new unknown-entry law inserts residue after AdmitNamespacePrefix, when recovery will adopt and resume. It does not exercise completed receipt state. At completed receipt state, migration_recovery_execution.rs:149-155 returns a successful Complete receipt before adopt_residue reaches the nested preflight at filesystem_migration_recovery.rs:91-94. The observer at filesystem_migration_residue.rs:54-86 checks directory presence/kinds rather than nested membership, while root admission at filesystem_initialization_namespace.rs:213-235 admits those directories without checking their children. Thus adding gc/unexpected to an otherwise exact completed store is accepted by migration recovery. The normative closed namespace expressly calls every other protocol-directory entry unrecoverable ambiguity (docs/formats/segment-store-v2/recovery.md:31-35).

This is reproduced, not a hypothetical hardening request. The parent supplied an isolated copied-source probe that changes only the unknown-entry test's parent list to ["gc"] and forward phase to RemoveReceiptStage. Read-only recursive comparison of /build/keep161-landing-source and /build/keep161-complete-probe-source, excluding .git/target, found only that test module different. The compiled test actually executed and failed with ambiguous migration restarted successfully: landing-evidence/161-complete-namespace-probe.log:38-53; variant: 161-complete-namespace-probe.rs:14-24. No product mutation was used. The existing new partial-prefix law can remain green while this promised completed-restart refusal is absent.

This production omission predates the topic; the introducing defect in this diff is declaring its broader ambiguity requirement complete without covering this state. The probe establishes successful admission of unknown evidence, not resumed mutation, returned wrong content, or physical durability damage. The existing normative wording that nested membership is checked before mutating recovery (migration-recovery.md:78) remains accurate for the partial path, but cannot justify the new broad completion disposition.

Required resolution: keep this closure unaccepted until completed-restart admission enforces the applicable canonical namespace and the regression checks the exact refusal and complete preserved witness. A narrow implementation is a read-only verify_complete capability on the recovery port, called only after the planner selects Complete, before returning that receipt. Its filesystem implementation can reuse the existing version-two namespace admission semantics, which permits owned post-migration retention state and checks empty gc and canonical recovery/dispositions (filesystem_initialization_namespace.rs:100-135). Preserve the exact underlying cause through a distinct completion-verification boundary if needed. Add lawful completed-store and lawful post-migration retention-state acceptance checks beside the unknown-gc refusal. State that this verifies the migration completion/namespace contract rather than pretending to verify all retention bytes; retention semantic admission remains owned by retention recovery/publication. Applying partial-migration empty-directory preflight indiscriminately to completed stores would reject legitimate retention artifacts. If the intended contract is to delegate completed-state validation elsewhere, that requires an explicit maintainer-approved contract/scope resolution, an observable delegated admission boundary, and honest remaining requirement ownership; simply narrowing prose cannot certify the original #111 completion requirement.

P2 — Recorded falsification covers common preservation and four guards, but not the distinct new refusal/diagnostic outcomes

Changed location: docs/testing-evidence/migration-restart-matrix.md:26-30. Binding rule: docs/Testing Standards.md:46-50; required evidence record: docs/testing/enforcement.md:22.

Five useful calibration families are recorded: root identity, exact-record inode identity, filesystem preservation, nested preflight, and inventory/store-identifier binding. All were verified against the retained mutant source differences and raw runtime failure output. However, the HEAD-deletion mutation makes record checksum and overlong tests fail at the shared preservation assertion (filesystem_migration_restart_test_fixture.rs:36) before their own exact refusal assertions run. It therefore calibrates preservation, not checksum/overlong diagnostics. The root/inode/inventory guard-removal mutations fail at the shared unexpected-success check, likewise supplying behavioral guard evidence rather than evidence that unrelated refusal assertions can fail. Passing debug/release receipts are not substitutes for that missing falsification.

One complete finite calibration batch is required for these existing distinct promises; no per-byte, per-coordinate, per-matrix-row, or mutation-percentage campaign is requested:

Promise family requiring falsification Existing assertion locations
Record corruption, overlong-stage classification, foreign receipt binding filesystem_migration_restart_record_tests.rs:36-40, :127-132, :154-159
Contradictory paired bytes filesystem_migration_restart_pair_tests.rs:53-58
Effect-before-intent, stage-after-effect, namespace hole and marker/receipt prerequisite refusal filesystem_migration_restart_order_tests.rs:26-31, :65-70, :98-103, :117-125, :137-145
Immutable-pool/HEAD corruption, valid wrong-name segment substitution and canonical pool-name refusal filesystem_migration_restart_pool_tests.rs:42-61, :74-79, :120-126, :183-188
Fixed-file no-follow/kind admission, directory-kind observation, nonempty fence refusal, and root unknown-name admission filesystem_migration_restart_namespace_tests.rs:33-43, :73, :100-108, :121-126, :163; common exact namespace assertion :131-137

A bounded common-boundary negative control that preserves storage and substitutes an incorrect refusal after the real restart attempt can exercise these diagnostic assertions in one batch, provided each named assertion actually executes and fails for that intended reason. Label such an experiment oracle calibration, not evidence of a production bug or mutation of every production guard. Reuse existing successful root/inode/inventory/preflight/preservation receipts; do not redo them merely for additional counts. Record immutable source/tree, variant or patch, exact commands, runtime RED output, and restored GREEN. Add any regression from the completed-state gap above with its real baseline RED; do not fabricate RED for the original missing-verification-only change.

Verification Checklist

Entire topic diff and evidence classification

  • All 13 files in the target-to-head diff inspected: CHANGELOG.md; docs/formats/segment-store-v2/{migration-recovery,requirements}.md; docs/testing-evidence/migration-restart-matrix.md; src/adapters/store_migration.rs; filesystem_migration_recovery_tests.rs; all six filesystem_migration_restart_{record,pair,order,namespace,pool,root}_tests.rs; and filesystem_migration_restart_test_fixture.rs.
  • Diff is 924 insertions/52 deletions in 13 files. Production surface is only #[cfg(all(test, unix))] module registration (store_migration.rs:69-82). No product implementation, dependency, API, canonical format, allocation policy, or durable publication order changes in the topic diff. The declared missing-verification change is appropriate; tests are product-boundary evidence, mutations are calibration, and registration/structure/Markdown checks are static evidence.
  • Every new test names an oracle and medium size plus retirement criterion. Controlled loops enumerate declared scenarios rather than count harness entries. The six groups fit the stated file thresholds. The shared witness uses sorted BTreeMap comparison and checks nonempty root plus every descendant with no-follow metadata; unknown object kinds fail (restart_test_fixture.rs:63-89). No hidden test dependence on directory iteration order; pool_tests.rs:84-90 checks the pool contains exactly one fixture artifact before selecting it.
  • Root copying is over the owned fixture, containing only known files/directories (root_tests.rs:34-45); it is not a general-purpose unchecked copy of hostile filesystem content. Fresh fixture creation has test-specific names and PID isolation (tests/segment_filesystem_stage/sandbox.rs:19-32). remove and Drop are actually present at :49-57, resolving CodeRabbit's unavailable-fixture context. Failed-law logs retain the actual assertion even though sandbox Drop removes owned scratch.

Every changed runtime contract path

Paths below use src/adapters/store_migration/ unless another path is given.

Path / observed promise Traced production path and source checks
Forward-prefix fixture followed by restart restart_test_fixture.rs:14-28 → real migration_resumption.rs:59-94 phase dispatch → filesystem_migration_storage.rs:14-121; local authority drops on return. restart_test_fixture.rs:44-50 reopens and freshly derives intent before calling recovery.
Public versus private/repository recovery admission Public filesystem_migration_recovery.rs:42-48 checks production platform; private test route :52-58 and repository-task route filesystem_migration_repository_tasks.rs:63-69 intentionally bypass only that profile. All join recover_root at recovery.rs:61-77: writer lock, root namespace, root identity, pinned pools. The evidence explicitly does not claim private admission proves production eligibility.
Fresh current authority filesystem_migration_authority.rs:96-116: namespace → root identity → HEAD decoder → complete inventory → selected catalog → HEAD reread → namespace/root recheck → intent. :129-137 compares fresh current intent. :149-175 requires version-one staging empty. Same-process roots compare device/mount/file (:178-185).
Recovery refusal before effects migration_recovery_execution.rs:129-145: verification → observation → decode expected intent → pure planner → ordered-prefix observation → persisted intent. Adoption/discard/resumption only at :157-174; source ordering supports no initiated resumed effects for the tested Observation/Ambiguity/Adoption refusals. Returned error sources preserved at :224-233. Complete early return separately checked and flagged above.
Checksum damage in all staged/canonical migration records restart_record_tests.rs:19-40,46-99 → bounded observer filesystem_migration_residue.rs:22-39 → planner canonical intent migration_recovery_planner.rs:36-39, intent stage :93-113, marker :158-193, receipt :210-230 → exact codecs: migration_intent_decoder.rs:72-81, format_marker_decoder.rs:75-84, migration_receipt_decoder.rs:71-80. Trailer-only XOR leaves checksum preimage unchanged, so captured original trailer is an appropriate expected checksum.
Overlong stage classification restart_record_tests.rs:106-132 → observer bound length+1 at residue.rs:35-39 → planner.rs:247-254 refuses Greater; only Less is incomplete. This preserves rather than disposes the extra-byte records.
Foreign receipt's own intent binding restart_record_tests.rs:141-159 obtains two independently rooted real migrations and installs the foreign valid receipt → planner.rs:228-230 → migration_receipt_decoder.rs:84-93. Public intent digests provide the exact encoded-record coordinates; shared decoder foundations are separate format evidence, not claimed independent reproof here.
Byte-equal inode substitution restart_pair_tests.rs:14-33 unlinks stage while canonical hard link keeps original inode alive, creates byte-equal replacement → planner byte equality → adoption recovery.rs:160-168 → fixed-record verify-linked → src/adapters/filesystem_exact_record.rs:265-277, exact KindLengthOrIdentity cause under Recovery::Adoption.
Contradictory stage/target bytes restart_pair_tests.rs:40-58 unlinks stage before modifying/replacing it, preserving canonical bytes → planner :57-64, :185-192, :231-238 exact StageDiffers, before adoption.
Effects before intent and uncleared predecessor stages restart_order_tests.rs:14-70 → planner :75-88, :42-55, :179-183. Inserted empty effect records intentionally exercise prerequisite refusal ordering before record decoding; each claimed stage/effect coordinate is checked.
Namespace holes and marker/receipt prerequisites restart_order_tests.rs:79-145 → namespace array order migration_recovery_residue.rs:6-15,86-96 → planner :123-131,149-152. Reader absent reports slots 0/1; removing retention removes its children and next present gc is slot4; roots/manifests/gc holes map 2/3,3/4,4/5. Parent absent/child present is rightly excluded as unrealizable. Marker/receipt cases remove predecessors while leaving their complete stage evidence.
Unknown root and nested names restart_namespace_tests.rs:14-46 → root admission src/adapters/filesystem_initialization_namespace.rs:213-235,180-202, typed Authority::Namespace/InvalidData; nested membership → adoption recovery.rs:91-94 → filesystem_migration_namespace.rs:231-245,279-305, typed NamespacePreflight. This law uses a partial prefix. The completed-prefix bypass is finding 1, not silently covered by the partial law.
Wrong kinds and substituted symlinks restart_namespace_tests.rs:53-110,142-166 → initialization namespace no-follow optional/required metadata src/adapters/filesystem_initialization_namespace.rs:154-176,213-231 for fixed names/top directories; nested directory kind → residue.rs:79-86, exact NamespaceKind/RegularFile under Observation. Symlink witness uses link target, not followed HEAD bytes (fixture.rs:71-80).
Payload-bearing reader fence restart_namespace_tests.rs:117-126 → residue.rs:42-50, exact regular-file length1 refusal; migration does not convert it into a lock merely because the name exists.
Damaged segments/catalogs/current HEAD restart_pool_tests.rs:22-61,70-79 → authority HEAD decode authority.rs:99-101; inventory filesystem_inventory_reader.rs:110-143 → segments filesystem_inventory_segments.rs:99-123 and catalogs filesystem_inventory_catalogs.rs admission. First-byte XOR preserves length and names, selects exact magic decoder cause; segment error also checks canonical expected magic.
Valid segment under wrong physical digest name restart_pool_tests.rs:107-126 → inventory segment filesystem_inventory_segments.rs:104-122 checks physical name against admitted digest, returns both digest coordinates. Original one-record segment and independent existing empty-segment fixture are different valid contents.
Valid orphan changes full inventory restart_pool_tests.rs:134-159 → inventory scan/sort/hash filesystem_inventory_reader.rs:110-179, filesystem_inventory_names.rs:11-46 → canonical intent inventory/store identifier → planner.rs:263-271, exact IntentDiffers. Inventory-only mutant survivor is correct redundant protection, not a percentage gap.
Canonical unknown pool-name refusal restart_pool_tests.rs:168-188 → names→segment/catalog parser src/adapters/recovery/recovery_pool_name.rs:7-40,44-51; wrong width is retained with exact pool and raw unknown name.
Copied root identity restart_root_tests.rs:12-29 → fresh-root-derived current intent → persisted-intent restart comparison planner.rs:258-271, binding device/file but ignoring historical mount. No identity-preserving restore claim is made.
Refusal's complete final-state witness Every new scenario → restart_test_fixture.rs:31-41 → :63-89: root/descendant names, directories, device/inode identities, file bytes, symlink target. Equality is a final-state guarantee; it cannot alone prove no transient restored write. Exact refusal ordering and cooperating-writer scope supply the documented additional constraints.
Existing success and removed-test coverage Existing forward-prefix recovery filesystem_migration_recovery_tests.rs:20-82, strict intent-prefix success :85-115, existing all-three-stage truncation and remount owners retained. Deleted corrupt-intent checksum test from original base was inspected: exact IntentUndecodable/ChecksumMismatch remains in restart_record_tests.rs:51-56; its count-only witness is replaced by stronger full witness. Deletion criterion expressly recorded at matrix :30.

Every merge audited against both parents

Both-parent changed-file sets and combined diffs inspected. All six mainline PR integration merges below are exact topic-parent trees (zero delta to second parent). Earlier normal merges preserve both change streams; combined resolution changes are documentation only. No migration production file differs between original base and current main, or between original topic head and current head. Hence imported production changes cannot reroute this migration implementation; shared admission/authority invariants still needed explicit checking below.

Merge SHA Parents / integration checks
ff6f5be4b98e985e00601ab3e854a95e87aa9b34 5cb758a857d88308214f40578b46776b3267895e + 34d70909b0cd93f6b020a59d4d40f43b07971cd8; 81/13 changed-file sets. Only CHANGELOG and v2 requirements differ from both parents. Every parent CHANGELOG bullet survives verbatim, no conflict markers; combined CHANGELOG adds #111 plus all six incoming entries. V2 requirement005 changes remain alongside incoming reader/model rows.
34d70909b0cd93f6b020a59d4d40f43b07971cd8 (#160) d08fafb2280a480a3c7d9460d13d53bbed4ae46b + c51e231e4ab456fbc8fa84f0fc2ca9609bb19ae3; exact second-parent tree. Reader process evidence does not alter migration or certify platform eligibility.
ee21b01d7b7740eaa56116809630534ea7caa05b 8e436b2db7e8afe2a1942df19bd4c7473e414eaf + d08fafb2280a480a3c7d9460d13d53bbed4ae46b; 70/6 files; combined CHANGELOG/v2 requirements preserve reader claim and independent model343 update.
d08fafb2280a480a3c7d9460d13d53bbed4ae46b (#159) 1325841cbcd27f4c504870728e3ca87d87a2c4cc + e768c8fcf769f25c422762cf67fa93384363ea68; exact second-parent tree. Operation-derived expected retention state preserved.
64bbbf915d87e43fa5c902ddeb0d893f246f9af5 f8ab8128c0b591e24110dfc883905d6995bde6c3 + 1325841cbcd27f4c504870728e3ca87d87a2c4cc; 60/4 files; only CHANGELOG differs from both.
1325841cbcd27f4c504870728e3ca87d87a2c4cc (#156) 64fafe3ddcc92bcc45a461a0161030b87559070d + 05658799fb259a445f68c8bd434135483d491e40; exact second-parent tree. Public-stage evidence retained as distinct platform/public API evidence.
05658799fb259a445f68c8bd434135483d491e40 5f90f22bde1582ecaa7215d85e7ce9db62975d25 + 64fafe3ddcc92bcc45a461a0161030b87559070d; 58/4 files; combined CHANGELOG/v1 requirements preserve sealed-stage prohibition plus public integration laws.
64fafe3ddcc92bcc45a461a0161030b87559070d (#157) 182e49520f98c6035828a739dcf3c224df535b84 + 3626f6e2677a1d6d3af08b88245584800596ff55; exact second-parent tree. Catalog locked-root route now admits profile; migration private-test profile isolation is explicitly different, not misrepresented as catalog/public proof.
657593fe50c824dd31dc328bf9e696183ef20767 fa06adfde89b70be5aaf7356206b7d8c1fbce169 + 182e49520f98c6035828a739dcf3c224df535b84; 49/12 files; combined CHANGELOG/v1 rationale/requirements preserve catalog platform and sealed-stage/partial-seal corrections.
182e49520f98c6035828a739dcf3c224df535b84 (#158) d0cff10d7c911d33d615c3aa2246ae2b4497432a + e781c0b276ec4d1f66a76668bd31893261a2e6dd; exact second-parent tree. Sealed receipt writable capability escape remains removed.
e781c0b276ec4d1f66a76668bd31893261a2e6dd 15aa976a77d5b65a4b6e8f8a2f9e01d2ba3b0d58 + d0cff10d7c911d33d615c3aa2246ae2b4497432a; 36/15 files; combined CHANGELOG/v1 requirements preserve both stage-observer and framing protections.
d0cff10d7c911d33d615c3aa2246ae2b4497432a (#172) 6051abb25a9fd33ae7ee0de5614514b709a4d82a + 07e6cc4875c05592b71bb1f8b9c80631a31bcda5; exact second-parent tree. Partial-seal contradiction refusal remains before truncated classification.

Imported production invariants checked at current head: partial-seal framing validation at src/adapters/recovery/recovery_segment_classifier.rs:68-75 → recovery_segment_seal_framing.rs:9-67; immutable sealed authority at sealed_segment.rs:7-21,69 and specialized hidden-stage conversion observed_segment_stage.rs:73-84; actual locked-root catalog profile admission at filesystem_catalog_publisher.rs:89-104 → filesystem_platform_admission.rs:34-40 → filesystem_platform_profile.rs:61-73. Topic prefix fixtures publish known complete immutable v1 segment/catalog data and do not touch the retired sealed callback or version-one truncated-stage assessment.

#99 binding decisions inspected in docs/testing-evidence/retention-landing.md:9-13,54-75: incomplete retention stages require disposition, cooperating writers delimit raw-path-race isolation, and execution failure is not rollback. This topic neither changes retention recovery nor transfers its automatic-discard prohibition to the separate fixed migration-record protocol. Existing migration incomplete-pre-effect-stage behavior remains explicit and unchanged; no automatic retention disposal is reopened. The witness limitations at matrix :38 are consistent with those approved concurrency/effects constraints.

Constants, numeric claims, and documentary dispositions

Constant / claim Evidence and disposition
Six law groups, Unix registration, no production behavior/API change store_migration.rs:69-82, exact13file target diff; six scenario modules plus one fixture module, 21 law functions. Changed production compilation consists only of test-gated registration.
32-byte checksum trailer; 16-byte magic mutation; XOR1; extra byte restart_record_tests.rs:30-33,123; pool :95-101; canonical record checksum source migration_intent_decoder.rs:72-81, receipt :71-80, marker :75-84, and format tables in recovery.md:39-50,89-107,178-193. These are protocol/controlled-corruption constants, not measured limits.
Pool widths68/85 and observed unknown-name width7 restart_pool_tests.rs:165-186 ↔ recovery_pool_name.rs:7-8,14-26; physical format is 64 hex + .seg, or 16 hex + hyphen +64hex+.cat; literal unknown is seven bytes.
Empty reader fence and observed payload1 restart_namespace_tests.rs:117-126 ↔ filesystem_migration_residue.rs:42-50 and normative recovery.md:71; b"x" supplies one byte.
Namespace slots0..6 / test hole pairs migration_recovery_residue.rs:6-15,86-96; five filesystem-realizable removal cases checked in path table above. Numeric coordinates derive from normative order, not runtime timing.
Current fixture HEAD width128; three fixed stages filesystem_migration_authority.rs:25,188-196; exact HEAD format fixture; migration stages intent256, marker96, receipt256 in normative recovery tables and production record constants. Only one extra byte is read by bounded observer.
Existing21 phases, crash053..073,68 migration subprocess cases Phase order is migration_phase.rs:57-78 and migration recovery docs point to the existing crash owner. They are unchanged historical/exploration coordinates, not new #111 evidence. Fresh copied validation traces debug/optimized crash commands; this reviewer does not assert a terminal campaign pass from an in-progress log.
Existing strict-prefix/forward success claims and count20 filesystem_migration_recovery_tests.rs:21-79 covers0..=ALL.len; complete receipt possible after phase20, stage removal, with finalsync phase21. New diff does not edit those expectations. All-three-stage truncation/remount/crash owners remain separate.
New matrix debug/release GREEN claim Historical complete-validation.log:325-373 and :2306-2354 show all21 named restart laws passing in both profiles; fresh candidate 161-validation.log:423-459 and :2470-2517 repeats named law results at tracked tree pinned at log:1-2. Passing counts do not prove calibration or completed-state unknown refusal.
Five mutation outcomes and inventory-only survivor Retained raw root/identity/namespace/inventory-binding logs each :38-56; effect log :43-69; inventory-only log ends with one passed orphan law. Retained Docker source diffs verify exact omitted comparisons/preflight and HEAD deletion. Effect RED is preservation only, explicitly not unrelated diagnostics.
Rust1.96.0, Linuxaarch64, actualext4 scratch Candidate validation log:1-10 records tree, rustc version, architecture, ext4 /dev/loop0; negative platform fixture reports tmpfs. rust-toolchain.toml is pinned. Manifest describes separate source/build directories and both scratch routes. Tests' profile bypass is explicitly acknowledged at matrix:34.
No claimed timing/rate/buffer/performance improvement in topic None added. Medium classification admits real filesystem I/O; ordinary per-test memory/deadline/network-isolation and suite-SLO gaps explicitly remain at matrix:38/enforcement profile. Those declarations are limitations, not invented ceilings or approved global compliance.
Incoming retention343 histories retention_model_tests.rs:54-62 has seven operations; nested second/third enumeration plus seven first-operation tests gives7³=343. Expected state derives from operation at:270-307; evidence retention-release-restore-model.md:7-11,23-35 distinguishes no-ops, independent reader calibrations, fixed depth and pinned historical receipts. No current migration parity inferred from that count.
Incoming reader watchdog20seconds tests/reader_fence_process/reader.rs:20; evidence reader-fence-process.md:13 calls it a failure watchdog, not passing timing oracle/latency promise. Kernel-observed wait and SIGKILL schedules remain at tests/reader_fence_process.rs:23-77; recorded landing controls reader-fence-process.md:27-35 are pinned separately.
New requirements005 Implemented disposition Not supported for the complete-receipt unknown-entry example; finding1. All other new matrix examples have the named source laws, with diagnostic calibration missing as finding2. No false claim of original roadmap checkbox updates or already-merged status was added.
Markdown long paragraphs / removedMD013 suppression .markdownlint-cli2.yaml:10 disablesMD013 repository-wide. New prose uses one physical line per paragraph; removed redundant table-specific suppression is consistent with configured policy, not a defect.

Review feedback reconciliation

  • Full supplied queue inspected: two top-level comments, one CodeRabbit APPROVED review on old 5cb758a857d88308214f40578b46776b3267895e, zero inline threads. Pagination implementation inspected at landing-evidence/fetch_queue.rb:13-49, including nested thread pagination; the parent owns live final queue refresh. No review approval for the current merge head was inferred from the old approval.
  • CodeRabbit global feedback says no actionable comments, but its checks timed out and sandbox source was unavailable to that reviewer. Sandbox cleanup source was found via src/adapters/mod.rs:130-132 path mapping and inspected directly; missing-context claim is resolved.
  • CodeRabbit docstring warning66.67% against80%,33 functions/4 skipped is a vendor metric on the old head, not a Keep quality target. Binding policy explicitly rejects project percentage gates (Testing Standards.md, coverage section) and requires actual public item documentation. This PR adds private test-module helpers, not an external public API; existing recovery public functions have explicit invariants/errors/docs. No docstring percentage finding issued.
  • Connector comment reports review usage limits; it is not an independent code verdict. CodeRabbit old approval does not rebut the reproduced completed-state omission or supply missing Rule4 evidence.

Checks executed, inspected, and unavailable

Executed by this reviewer: read-only Git head/tree/status/log/diff/diff-check; complete13file source inspection; both-parent/combined merge examination; deterministic parent CHANGELOG-entry preservation comparison; queue and pagination inspection; read-only Docker discovery/differences of retained mutation sources and copied completion probe. git diff --check passed. No host or Docker Rust tests were launched by this reviewer.

Inspected execution receipts: historical issue111 focused/full debug/release validations, initial compile/setup failures (excluded as behavioralRED), preserved five calibration families and inventory-only survivor; fresh exact tracked-tree copied Docker validation log and manifest; parent completion probe with runtime failure. Fresh validation log shows commands advancing through full debug/release tests, docs, MSRV and fuzz formatting/check/Clippy, but terminal session status and final hosted checks remain the parent's gate. After inspection, the parent reported terminal session10260 exit0 for the full exact-head chain. This report distinguishes that parent-attested terminal result from its own receipt inspection; it does not infer completion from intermediate output. Dependency/workflow/runtime-fuzz hosted results are not independently attested by this report.

Coverage limitations remain explicit: private/repository-only admission isolates migration semantics; in-process drop/reopen is deterministic restart evidence, not process death; process-death campaigns are not physical power-loss evidence; no arbitrary external raw-writer isolation; witness equality alone cannot rule out a transient restored write; ordinary per-test resource ceilings/suite budgets are not established. No parser or production transformer changed, so this topic does not need a fabricated new fuzz or benchmark claim. Imported historical receipts are preserved as their pinned evidence and not relabeled measurements of this exact head.

Exact-head verdict

ff6f5be4b98e985e00601ab3e854a95e87aa9b34 / tree de06577722b63ba09d425f31dfa7a1acd647e128: two finite blockers remain, a reproduced completion-claim gap and a distinct missing-falsification batch. Current-head required hosted checks remain a separate gate; the parent reported the copied full validation terminal exit0. Subsequent checkpoint/source heads require their own disposition and cannot inherit this exact-head verdict. This review does not authorize a merge.

REQUEST CHANGES

@flyingrobots flyingrobots changed the title Test: complete migration restart ambiguity evidence (#111) Fix: validate completed migration namespaces and restart evidence (#111) Oct 3, 2026
@flyingrobots

Copy link
Copy Markdown
Owner Author

Independent bounded delta review — PR #161

Reviewed exact head e20629852f402e129d889e398014781d20d6ff0b, tree d0e7e4b447495d428608a89e2142023b8e8c1781, targeting main 34d70909b0cd93f6b020a59d4d40f43b07971cd8. Read-only checkout the isolated candidate checkout was clean and head/tree were checked before disposition. This is the authorized independent Codex fallback under the agy-review protocol. No source changes, host Rust execution, publication, configuration changes, or delegation occurred. Only this report was written.

Both prior finite P2 findings are CLOSED for the reviewed changes and evidence. No new verified defect in this bounded delta was found. Required hosted CI is currently failed and still blocks merging; this review is not a waiver of that gate.

Disposition of both prior findings

Completed restart namespace gap — CLOSED. The actual bug fix adds verify_complete to the semantic recovery port (src/adapters/store_migration/migration_recovery_storage.rs:27-35). recover_store_migration first verifies current authority, observes residue, and successfully plans recovery; only for Complete does it call the new capability before returning a receipt (migration_recovery_execution.rs:133-164). The filesystem implementation performs the existing read-only version-two namespace admission (filesystem_migration_recovery.rs:86-88 → src/adapters/filesystem_initialization_namespace.rs:100-135). This rejects reserved unknown GC/recovery/disposition evidence without applying partial-migration empty-retention-pool policy to lawful completed stores. Typed failure retains its original cause under Observation and NamespacePreflight; planner corruption/order refusals remain earlier and retain priority. The implementation has no write, unlink, synchronization, publication, or retention-recovery operation.

The permanent regression completed_migration_refuses_unknown_reserved_entries_without_effects covers gc, recovery, and recovery/dispositions at RemoveReceiptStage, uses the existing complete before/after witness, and checks the exact nested refusal (filesystem_migration_restart_namespace_tests.rs:169-190). Its committed runtime RED receipt records successful compilation and an executed test failing because the unchanged product returned success (docs/testing-evidence/migration-restart-matrix/complete-regression-red.txt:38-53), against unfixed production a8fda647bc67271ec7895577eab55f4b3a5e0085. This is actual baseline RED, not a fabricated regression for the earlier verification-only matrix.

The lawful-retention law publishes a real retention root, drops publication authority, obtains fresh migration authority, requires Complete with no forward phases, then reads the exact retained bytes through the public snapshot (src/adapters/retention/filesystem_retention_migration_completion_tests.rs:17-46). The deliberately wrong empty-pool completion policy compiles and fails this law with Observation/NamespacePreflight/InvalidData (161-calibration/lawful-retention.log:38-53). Fixed debug/release GREEN receipts cover both new laws and the original matrix. The public trait addition and external-implementor compatibility obligation are explicitly documented in CHANGELOG, migration recovery API docs, and docs/formats/segment-store-v2/rationale.md:119-125. No no-op default lets an existing implementation silently skip admission. Retention content and stage semantics remain owned by retention; this does not certify arbitrary retained content or reopen #99 automatic disposition.

Distinct refusal/diagnostic falsification gap — CLOSED. The finite batch implements the previously requested boundary negative controls without changing expectations. diagnostic.patch calls wrong_diagnostic only after the real restart returned a refusal and the unchanged complete witness equality passed (restart_test_fixture.rs:31-41, patch replacement at line41). The raw diagnostic-red.log:42-163 shows 22 executed runtime laws failing at their own diagnostic assertions, rather than at fixture setup or preservation. It reaches record checksum/overlong/foreign receipt, paired conflict, all ordering law families, HEAD/pool decoder and segment-coordinate failures, pool naming, root namespace kind/no-follow/error-kind checks, and the new completed-state check. Counts here identify observed executions, not a correctness score.

Two corrected controls close the earlier masked comparisons. nested-diagnostic.patch leaves root Authority refusals untouched and preserves outer Adoption/Observation wrappers while replacing their inner cause. nested-diagnostic-corrected.log:42-73 then reaches nested membership at namespace test line39, nested directory kind at line100, nonempty fence at line121, and completed namespace at line181; root-only kind/link laws remain green intentionally. pool-diagnostic.patch preserves the artifact cause while swapping its semantic pool; pool-diagnostic-corrected.log:42-58 reaches the pool-identity assertion at pool test line42, observing Catalogs instead of Segments. The original diagnostic batch preserves that pool identity and reaches the deeper decoder assertion separately. This distinguishes those qualitative promises without requiring one mutation per coordinate or scenario row.

The initial pool control's unused-import compilation failure and initial nested control's earlier-boundary failure were inspected and are excluded from admitted evidence, as the ledger states. Original failure attempts remain in scratch. Four committed RED receipts were independently compared to their raw scratch logs after the documented path normalization/trailing-empty-line handling: all match exactly. The restored GREEN receipt likewise exactly matches its raw log, pins fixed source tree 13a6148d8b4ca1fc28b8342cd6a06bc46b9bb2a5, and records 22 restart laws plus two completion laws passing in debug/release. Existing root/inode/inventory/preflight/preservation calibration from the original review remains credited. This closes the finite requested batch; no exhaustive mutation score or per-field adequacy is inferred.

Mandatory Verification Checklist

  • Prior exhaustive scope adopted after checking unchanged coordinates. The original report landing-evidence/161-independent-review.md contains the complete 13-file path, both-parent merge, constants, documentary claims, and historical feedback checklist at ff6f5be4b98e985e00601ab3e854a95e87aa9b34. The current-to-original diff was inspected in full. The only source changes are six files: retention test registration; new lawful-retention test; filesystem recovery implementation; appended completed-state namespace law; recovery executor; and recovery storage port. Original record/pair/order/pool/root laws, shared witness, authority observation, codecs, inventory, existing truncation/remount success laws, imported mainline invariants, AGENTS.md, Testing Standards, enforcement profile, dependencies, toolchain, and all other paths remain unchanged. Their prior review is adopted, not represented as repeated execution.
  • Every commit/merge. The four commits after the prior review are linear, with no new merges: a8fda647bc67271ec7895577eab55f4b3a5e0085 preserves the counterexample; 6351c7cb4db10e39e096fb43be3e10c3c69cf626 adds the permanent regression; a3ea6c3e2ac58ca82a5c26f43d902ba1ed883be7 fixes completion and adds lawful-state evidence/docs; e20629852f402e129d889e398014781d20d6ff0b adds receipts/docs only. Git comparison of a3ea6c3 to final head confirms no src, tests, Cargo, lockfile, or toolchain change. All original both-parent merge resolutions and Fix: refuse corrupt partial segment seals during recovery (#171) #172/Fix: keep sealed stage authority private during crash observation (#146) #158/Fix: preserve platform admission for catalog publisher authority (#150) #157/Test: verify public admitted filesystem stages (#147) #156/Test: verify retention release and restore against independent model (#128) #159/Test: prove reader-fence process death and collector exclusion (#113) #160 invariants remain covered by the adopted checklist and unaffected files.
  • Complete runtime path. Public profile-aware, private test, and repository-task recovery constructors still join filesystem_migration_recovery.rs:61-77. Current intent observation at filesystem_migration_authority.rs:96-116 and current verification feed migration_recovery_execution.rs:133-146; migration_recovery_planner.rs:228-238 chooses Complete only after exact receipt admission with no receipt stage. Complete verification now executes at executor :147-150 before success :157-164. Root admission checks file/directory kinds and exact fixed root membership; nested full-v2 admission at filesystem_initialization_namespace.rs:125-135 permits required retention directories, rejects nonempty gc, admits only dispositions under recovery, and requires dispositions empty. Directory opens use no-follow capabilities. Existing bounded root membership scan and fixed required directories introduce no unbounded whole-retention-pool materialization.
  • Parallel paths/ports. Repository search finds one StoreMigrationRecoveryStorage implementation, the filesystem authority. The public trait is re-exported through store_migration.rs:183 and src/lib.rs:164; new capability has documented synchronous effect-free semantics and errors, with no boolean parameter, serializer-owned values, async, unsafe, or adapter-specific wire type in the port. All existing consumers call the same executor; no second completion path bypasses the new check. Incomplete-stage discard and resumed-stage adoption branches remain unchanged except line offsets. No action is added for VersionOne or Resume plans.
  • Errors/refusal priority/effects. Planner failure still exits before completion admission. Completion I/O/refusal mapping preserves source kind and NamespacePreflight cause, then executor wraps it as Observation. Error::source remains unchanged. On Complete, fresh authority and pure planning precede read-only admission, then a receipt with no executed phases; no mutation, sync, or rollback is claimed. Existing resumed failure-effects contracts and Retention recovery, crash-matrix evidence, reader fence, and model-based transitions (item 6) #99 cooperating-writer scope remain adopted. Full witnesses observe final persisted state; absence of arbitrary raw-writer interference or transient restored writes is not newly claimed.
  • New tests/registration/oracles. Appended namespace regression has specified reserved-name refusal and full semantic witness; lawful-retention test has independently fixture-named exact public root bytes after real publication and reopened migration. Both carry medium size, named oracle, and retirement criterion. Retention registration is gated test + repository-tasks (retention.rs:278-279); namespace tests remain Unix gated. Owned scratch and unique names retain prior isolation. The copied controls are calibration, permanent filesystem laws are product evidence, and compile/docs/format/patch checks are static evidence. No source-string/count test is substituted for runtime behavior.
  • All numeric/constant changes. No production timing, rate, allocation, parser-length, phase-count, or persistence constant changes. Reserved parent list has three entries corresponding to the existing v2 reserved directories. Retention reader bound1,048,576 at new test :35 matches existing retention fixture/model policy; it is fixture admission, not a measured performance claim. Ledger Rust1.96.0/Linuxaarch64/ext4 coordinates match fixed validation log :1-10. Diagnostic22, nested6 executions with4 intended failures/2 intentional passes, pool1, lawful-retention1, and restored22+2 counts match the inspected raw receipts. Exit101 is observed in traced launchers, not inferred from a compilation failure. Old measurements remain pinned as historical evidence. Added docs explicitly distinguish original verification-only change from the production bug fix and required trait API compatibility; identities and on-disk formats are unchanged.
  • Every new evidence/doc claim. All 24 delta files were inspected: six source files, five prose/change-ledger documents, eight raw text receipts (probe, regression, focused GREEN, four admitted control REDs, restored GREEN), and five zero-context patches. migration-restart-matrix.md:40-54 is explicitly historical counterexample then correction; :58-69 describes admitted controls, excluded attempts, replay commands and normalized receipts. Requirements005 identifies a corrected candidate with review/integration pending rather than already-merged completion. Rationale rejects empty-pool completion policy and scopes retention ownership. New paragraph layout follows existing one-physical-line convention and configured MD013 policy. No new performance optimization, parser, codec, or durable write protocol requires fabricated benchmark/fuzz/power-loss evidence.
  • Replay/static verification executed by reviewer. git diff --check passed. All four new calibration patches independently passed git apply --check --unidiff-zero against the unchanged fixed source at final head; no patch was applied. Raw-to-committed normalized receipt equality passed for each accepted control and restored GREEN. Git status stayed clean and final head/tree stayed pinned. No Rust test, calibration, or mutation was executed by this reviewer; runtime results are inspected receipts with their exact sources, not reviewer-generated outcomes.
  • Full local validation inspected. landing-evidence/161-fixed-validation.log pins fixed tree 13a6148d8b4ca1fc28b8342cd6a06bc46b9bb2a5, Rust1.96.0, Linuxaarch64, actualext4 and negative tmpfs. It traces golden/conformance/structure, both production process-death campaigns, formatting, all/minimal checks and Clippy-Dwarnings, full debug/release tests, docs/doctests/MSRV and fuzz formatting/check/Clippy. New laws pass at lines245/426 and2297/2475; final fuzz Clippy completes at log end. Parent supplied terminal exit0. Final e206298 changes only evidence/docs, so that runtime code remains byte-identical; this does not infer that final hosted checks passed.
  • Hosted failure and final gates. Parent reported final run37153036962 failure; raw landing-evidence/161-hosted-rust-failure.log:1115-1132 confirms collection_excludes_a_new_snapshot_until_release fails with Error: Busy, while the companion reader-death law passes. The earlier hosted pass is not substituted for this first preserved failure. Reader-fence process sources have no change in this delta, and no cause has been demonstrated by this review; the inherited-file-description hypothesis is unverified. Diagnose and resolve the failed required check before merge. Do not retry-to-green or treat this source approval as permission to waive it. Parent owns refreshed feedback/required-protection checks; old CodeRabbit approval and optional percentage warning remain historical rather than final-head certification.

Exact-head verdict and limits

e20629852f402e129d889e398014781d20d6ff0b / tree d0e7e4b447495d428608a89e2142023b8e8c1781: APPROVE the bounded source and evidence delta. Both original finite findings are resolved with the intended contract, real baseline regression evidence, and the requested bounded calibration batch. This adopts the verified unchanged original checklist plus the delta checklist above.

Merging remains blocked by the failed required hosted reader-fence test and final repository gates. Approval is an independent review disposition, not a clean-CI assertion, flake diagnosis, merge authorization, or risk waiver. Existing private platform-admission, fixed-schedule, resource-ceiling, raw-writer and physical-power-loss limitations remain exactly as scoped in the prior review and ledger.

APPROVE

@flyingrobots

Copy link
Copy Markdown
Owner Author

Code Lawyer activity — source findings closed, CI blocked

Obligation Disposition and evidence
P2 completed migration admits unknown reserved entries Regression 6351c7cb4db10e39e096fb43be3e10c3c69cf626 observed RED on unfixed production. Fix a3ea6c3e2ac58ca82a5c26f43d902ba1ed883be7 adds read-only completion admission with precise existing causes; lawful published retention remains admissible.
P2 distinct diagnostic calibration missing e20629852f402e129d889e398014781d20d6ff0b commits four finite controls and restored GREEN. Corrected nested/pool controls reach intended assertions; invalid earlier attempts are excluded.
Independent review Exact-head APPROVE closes both original findings with the mandatory checklist. This does not waive CI.
Local validation Full copied-Docker chain exits zero on unchanged runtime code at a3ea6c3; final successor changes evidence/docs only.
Final hosted checks Three jobs pass, Rust fails at the existing collector-exclusion process test with Error: Busy in run 37153036962. No rerun-to-green. Original log preserved.
Focused prerequisite repair #174 / PR #175 retains writer authority through collector preparation. Runtime RED/GREEN confirms that setup contract; the exact untraced hosted schedule remains unproved. Land that independently after its gates, merge main normally here, then review the integration delta and check the resulting exact head.

MERGE GATE: LOCKED. The two source-review findings are resolved; failed required CI remains the specific blocker. No automatic retention-stage disposition, broader diagnostic redesign, platform isolation or power-loss guarantee is added. CodeRabbit's rate limit is not an approval; the authorized independent review is the source-review gate. Requirements remain candidate-only until integration.

@flyingrobots

Copy link
Copy Markdown
Owner Author

Independent integration delta review — PR #161

Reviewed head 2f22d0d9097820503d2a81085bb748273de9f56d, tree 658e4fc920a4584ec1153c4c30f45af1b295427d, targeting main 1c2b9d788fd4029d2469d2651faf0f8db2e0869e, in the clean read-only the isolated candidate checkout checkout. This is the authorized independent Codex fallback under the agy-review protocol. No host Rust execution, source changes, commits, publication, configuration changes or delegation occurred. Only this report was written.

No verified integration defect was found. Both earlier finite P2 findings remain CLOSED. The approved migration completion implementation and calibration evidence are unchanged; the merge imports the independently reviewed reader-fixture correction without changing production locking. This review approves this exact head, not a later successor or a waiver of repository gates.

Mandatory Verification Checklist

Scope and both-parent integration

Every changed path and semantic interaction

Outcome File:line path verified Integration result
Continuous initialization/migration writer ownership tests/reader_fence_process/fixture.rs:13-15 initializes; :32-37 transfers initialization writer authority into migration and executes the forward protocol; :41 returns the same authority. src/adapters/store_migration/filesystem_migration_authority.rs:39-46 owns inventory; filesystem_inventory_reader.rs:28-34,83-89 owns the actual writer lock. Tuple handoff moves the existing guard. No drop/reacquire gap or new product API. Forward migration remains migration_execution.rs:19-32,104-131; holding the completed authority does not execute additional writes.
Both fixture consumers reject a competing writer tests/reader_fence_process.rs:27-34,70-77 → src/adapters/filesystem_writer_lock.rs:72-83,96-115,123-155. Both paths require exact WriterLockAcquireError::Busy, immediately after handoff and before their own child spawn. Successful contention admission or an unrelated error fails. The asserted contract is fixture authority, not reproduction of an untraced inheritance schedule.
Live public snapshot excludes collection and real process death releases it First law reader_fence_process.rs:35-49 → reader.rs:55-82,26-40 → src/adapters/retention/filesystem_retention_snapshot.rs:122-161 → reader_fence.rs:35-69; parent readiness precedes exclusive EWOULDBLOCK; reader.rs:119-127 verifies SIGKILL and reaps before positive exclusive acquisition. Snapshot load takes no writer authority. Its shared reader fence is independent of the migration writer guard, so earlier/longer writer ownership does not introduce a lock cycle or suppress legitimate snapshot admission. Existing exact errno, process death and positive acquisition assertions remain unchanged.
Persistent fence and release ordering First law reader_fence_process.rs:36,50-57 compares device/inode/zero length, then explicitly drops collector and writer. Second law :78-88 acquires exclusive fence before spawn, observes the exact kernel queue through reader.rs:92-116,180-189, drops collector then writer, and requires readiness/normal completion. The reader fence remains the mechanism that excludes the child. No sleep, retry, elapsed-time-only negative oracle, global serialization, inode replacement or production unlocking change. Actual guards release at explicit schedule points.
Normal/error teardown reader.rs:68-81,149-154 kills/reaps children and removes sockets; tests/segment_filesystem_stage/sandbox.rs:49-58 owns cleanup. The longer-lived writer is an owned guard, not a durability action on Drop. Failure cannot make a dead child or empty wait pass. The fixture has only the two inspected consumers.
Completed migration refuses reserved unknown evidence migration_recovery_execution.rs:133-146 verifies current state and plans first; :147-150 calls migration_recovery_storage.rs:27-35; filesystem implementation filesystem_migration_recovery.rs:86-88 → src/adapters/filesystem_initialization_namespace.rs:100-135; success returns only at executor :157-164. The imported fixture uses forward execution; it does not bypass or reroute recovery. Completion still uses the same v2 namespace admission as public snapshot load (filesystem_retention_snapshot.rs:127-129). Planner corruption priority, original Observation/NamespacePreflight source and effect-free completion are unchanged.
Completed migration permits lawful retention Namespace admission filesystem_initialization_namespace.rs:125-135 allows retention-owned publication state while refusing nonempty reserved GC/disposition directories. src/adapters/retention/filesystem_retention_migration_completion_tests.rs:17-46 publishes real retained bytes, reopens migration, requires Complete/no phases, then loads exact retained bytes through the public snapshot. The continuous process fixture publishes no retention root and leaves complete migration artifacts unchanged. Its retained guard and the separate lawful-retention law do not impose partial-migration empty-pool policy. Retention content/recovery remains retention-owned; #99 is not reopened.

Prior findings, regression and calibration evidence

  • P2 completed-state namespace gap remains CLOSED. Required public verify_complete, correct read-only filesystem admission, precise errors, corruption precedence and documented external-implementor compatibility are unchanged. Permanent completed_migration_refuses_unknown_reserved_entries_without_effects at filesystem_migration_restart_namespace_tests.rs:169-190 still covers the three reserved parents at completed RemoveReceiptStage, using the complete witness. Its real baseline RED, fixed GREEN and wrong-empty-pool retention control remain exactly the approved receipt/patch blobs.
  • P2 distinct diagnostic falsification gap remains CLOSED. The admitted common-boundary diagnostic, corrected nested-error, corrected semantic-pool and wrong-empty-pool controls plus restored GREEN remain identical to e206298. The prior review verified their runtime assertion reachability, declared normalization and patch application checks. The original compilation failure and masked earlier nested assertion remain retained and excluded from admitted evidence. No new diagnostic promise or per-field mutation obligation arises from this merge.
  • Incoming deterministic runtime RED at docs/testing-evidence/reader-fence-process/continuous-authority-red.txt:42-59 executes the collector law alone and fails the continuous-exclusion assertion before any child spawn; one companion law is filtered. Fixed GREEN at continuous-authority-green.txt:5-11,52-56 executes both original process schedules in debug and release, two passes in each profile. This is falsification of the shared fixture boundary, not an invented product-lock regression or a claimed hosted-race reproduction.
  • Independently compared both committed reader RED/GREEN receipts with retained raw 174-red.log and 174-focused-green.log: byte-equal after only the declared source/target substitutions and trailing-empty-tail normalization. The adopted PR175 review additionally verified copied-source archive blobs against exact regression/runtime-fix revisions; mainline retains those reviewed blobs.
  • hosted-busy-failure.txt independently matches an exact contiguous diagnostic excerpt of retained 161-hosted-rust-failure.log after declared line-end whitespace trimming. The prior e206298 failure with bare Busy remains in the record; it is not converted into a pass. Evidence prose reader-fence-process.md:41-47 correctly labels inherited lock descriptions as a source-backed possible schedule and the passing delayed-exec trace as inconclusive, not proved original causality. The imported fixture removes the demonstrated release/reacquire contract gap regardless of that hypothesis.

Constants, figures, documentation and review feedback

  • No production limit, timing, buffer, retry, parser, phase, format, identity or synchronization constant changes. Incoming harness constants and original calibrated reader observations are unchanged: 20-second fail-only watchdog (reader.rs:20), 1,048,576-byte catalog admission (:32), generation one (:37), signal nine (:123), zero-length persistent fence (reader_fence_process.rs:52) and default reader-attempt policy. These are existing fixture/admission observations, not new measured throughput or enforced resource budgets.
  • New numeric claims match evidence: one selected RED/one filtered test; two GREEN laws in each profile; Rust1.96.0/Linux aarch64/owned ext4. Historical source/run/job coordinates remain pinned; original reader/migration counts and constants are unchanged and retain the adopted raw-evidence audit. No percentage is substituted for correctness. The current integration log pins the exact candidate tree and environment at lines1-10.
  • CHANGELOG and incoming evidence accurately describe continuous fixture authority, unchanged production lock semantics, real deterministic RED, preserved reader schedules, replay commands and bounded causal claims. Source documentation and test size/oracle/retirement declarations remain applicable; paragraphs preserve one physical line. No new parser/format/production write protocol warrants fabricated fuzz vectors, benchmarks or power-loss claims.
  • Inspected landing-evidence/161-integration-queue.json: seven global comments, one CodeRabbit approval on old 5cb758a..., no threads. Historical published findings/closure match the adopted reviews. Current provider processing is not approval. Its inconclusive description check relies on stale test-only summaries; actual required completion method and production behavior change are present and documented. The optional 66.67%/80% docstring metric remains nonbinding; actual public-item documentation was checked. Earlier sandbox uncertainty was resolved by the real mapped sandbox owner, not ignored. Parent owns current pagination/feedback refresh and effective approval admission; this review does not claim to have independently fetched GitHub state.

Execution and final gate boundary

  • Executed by this reviewer: read-only Git status/head/tree/history/diffs, parent blob/mode comparisons, CHANGELOG preservation/marker checks, raw-to-committed reader receipt comparisons, hosted-failure excerpt comparison, target-relative whitespace check and the exact whole-tree CI whitespace command from .github/workflows/ci.yml:122-123. All passed. No Rust tests or source mutations were run by this reviewer.
  • Inspected exact-tree copied-Docker landing-evidence/161-integration-validation.log: tree 658e4fc..., pinned Rust1.96.0, Linux aarch64, actual ext4 /dev/loop0 scratch and tmpfs negative fixture (:1-10). Trace covers golden, both debug/release durability campaigns, conformance, structure, formatting, all/minimal feature checks and Clippy with warnings denied, complete debug/release workspace suites, doctests/docs/MSRV and fuzz formatting/check/Clippy (:11-17,78-94,2137,4185,4225,4232-4248). Final fuzz Clippy completes successfully. Parent supplied terminal session73048 exit0; the exit receipt is reported by the parent rather than inferred solely from log text.
  • Both integration-sensitive outcomes execute successfully in that fresh exact-tree run: completed migration preservation/refusal laws at lines248/426 and2297/2475; both reader process laws at786-787 and2834-2835. This is stronger than adopting isolated passes from separate trees, while remaining bounded Linux process-death evidence rather than physical power-loss proof.
  • Parent reports all four required hosted jobs successful for final candidate run37154563204 at 2f22d0d..., and all four successful for mainline PR175 run37154541889. This current status is parent-verified, not independently retrieved by this read-only reviewer. Earlier Fix: validate completed migration namespaces and restart evidence (#111) #161 Busy and Fix: retain writer authority through reader-fence test setup (#174) #175 infrastructure/whitespace failures stay preserved and excluded from green runtime claims. Optional provider processing is not counted as an effective approval. Branch protections, refreshed queue and merge authorization remain the parent's final gate responsibilities.

Existing admitted limits remain explicit: untraced original hosted inheritance schedule, arbitrary fork/exec interleavings, physical power loss, hostile noncooperating writers, full GC and fully enforced resource ceilings are not newly established. They neither reopen resolved findings nor justify speculative hardening in this bounded integration review.

Reviewed SHA: 2f22d0d9097820503d2a81085bb748273de9f56d.

APPROVE

@flyingrobots

Copy link
Copy Markdown
Owner Author

Code Lawyer closure — integrated candidate 2f22d0d

Obligation Severity / source Disposition and evidence
Completed recovery accepts unknown reserved entries P2 / self and independent review Permanent regression 6351c7cb4db10e39e096fb43be3e10c3c69cf626 fails on unfixed production. Fix a3ea6c3e2ac58ca82a5c26f43d902ba1ed883be7 adds read-only complete namespace admission preserving typed source errors and planner priority. Lawful published retention remains admitted. Closed.
Distinct refusal assertions lack calibration P2 / independent review Receipt commit e20629852f402e129d889e398014781d20d6ff0b contains finite diagnostic, nested-source, pool-identity and lawful-retention controls with restored GREEN. Compile failure/earlier-boundary attempts excluded. Both P2s closed by independent review 5973444466.
Existing reader-fence setup fails with Busy Required hosted runtime gate Original run 37153036962 preserved. Separate #174/#175 retains continuous collector authority, independently reviewed and merged as 1c2b9d788fd4029d2469d2651faf0f8db2e0869e. Fixture-contract RED/GREEN does not claim a proven trace of the original hosted interleaving. Current merge 2f22d0d integrates the repair; only CHANGELOG conflicted, both entries preserved.
Final local validation Required Full copied-Docker chain exits zero at exact tracked tree 658e4fc920a4584ec1153c4c30f45af1b295427d: golden/conformance/structure, both crash campaigns, formatting, all/minimal check and Clippy, full debug/release tests, docs/doctests/MSRV and fuzz compilation/Clippy.
Final hosted validation Required All four jobs in run 37154563204 pass on exact 2f22d0d9097820503d2a81085bb748273de9f56d. Earlier green heads and failed attempts are not substituted for these results.
Review queue Complete paginated review bodies/comments/threads Original independent findings closed with evidence; no unresolved actionable inline findings or active changes-requested reviews. Hosted review limits/optional incomplete CodeRabbit review supply no approval. Authorized independent final integration review is the source gate.

Delivered: complete migration namespace refusal before recovery effects, preserved lawful retention state, exact diagnostic and full filesystem restart witnesses. Compatibility: public recovery-port implementors must supply the new required read-only verify_complete capability. No format or identity changes. Deferred/unchanged: #99 automatic incomplete-retention-stage disposition, #110 broader diagnostic redesign, platform admission evidence, arbitrary raw-writer isolation and physical-power-loss claims. Before/after witnesses do not rule out transient restored writes; ordinary test resource enforcement remains a stated limitation.

Final integration review separately records exact-head approval and the mandatory both-parent merge/path/evidence checklist. The target is 1c2b9d788fd4029d2469d2651faf0f8db2e0869e. Recheck candidate and repository gates immediately before a normal signed merge.

MERGE GATE: OPEN under the maintainer's existing authorization.

@flyingrobots
flyingrobots merged commit 80d23f5 into main Oct 3, 2026
4 of 5 checks passed
@flyingrobots
flyingrobots deleted the test/111-migration-restart-ambiguity branch October 3, 2026 21:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Complete migration restart corruption and ambiguity matrix

1 participant