Skip to content

Repository navigation

Tracker: P1-RECOVERY, Gateway recovery while an independent Runner survives #101

Closed
Closed
Tracker: P1-RECOVERY, Gateway recovery while an independent Runner survives#101
Labels
documentationImprovements or additions to documentation

Description

@novelKR
novelKR
opened on Oct 14, 2026

What this issue is

The CodeSpace tracker for P1-RECOVERY, started on the owner's direction of 2026-10-14. It is a living tracker: when the state changes, add a comment (date, what changed, evidence link) and tick the boxes below.

Scope

In an operator-selected mode, an independent Runner keeps its processes, I/O and original deadlines when the Gateway exits, and a new Gateway takes over the same executions. Existing modes keep their termination behavior. InProcess, restoring I/O after the Runner itself restarts, and replaying argv are out of scope (ADR-002).

Units

One CodeSpace PR per unit, merged in order. Each unit starts on the owner's approval after the previous one is merged and its post-merge CI is green.

Unit What it adds After
P1R-C01 Opt-in independent Runner lifecycle: serial connections, exit intents (detach or stop), Runner identity in Hello; a re-attached Gateway may only observe, write stdin, resize and terminate DevGuard ledger start
P1R-C02 Process records in the operations database, with a schema version and a single-writer lock P1R-C01
P1R-C03 Claim on the existing connection, with an epoch fence on every mutating request P1R-C02
P1R-C04 Reconciliation of records, Runner inventory and DevGuard attempts; a per-workspace barrier P1R-C03
P1R-C05 Output gaps, uncertainty and Runner loss; the operator runbook P1R-C04
P1R-C06 Qualification harness and the macOS runs, then the report P1R-C05

Rules

  • The default mode does not change. The new mode is opt-in and is not a supported mode until P1R-C06 qualifies it.
  • No Codex pin, DevGuard pin, DevGuard release, installed service or policy change without the owner's approval. A Codex fix is a recorded patch file only (upstream patches).
  • MCP contract changes (new error codes, gap and uncertainty reasons, mode-specific tool text) are listed in each PR's "Contract changes" and need the owner's approval at merge.
  • The known defects OX-01 and OX-02 (Handoff: CS-RG managed-execution suspension (#75) and the DevGuard boundary revalidation (2026-09-27) #76) are not fixed here; P1-RECOVERY does not change when a PTY handle is dropped.

Checklist

Activity

  1. novelKR
    added
    documentationImprovements or additions to documentation
    on Oct 14, 2026

  2. novelKR commented on Oct 14, 2026

    @novelKR
    novelKR
    on Oct 14, 2026 – with Claude
    OwnerAuthor

    2026-10-14, DevGuard ledger started. novelKR/DevGuard#36 was merged on the owner's approval at head 9b19078, as DevGuard main 165cb84 (documentation only; the merge tree equals the head). Post-merge CI run 38573601167 is green.

    The ledger now records P1-RECOVERY as in-progress with P1R-C01 active, the plan carries the dated re-baseline, the integration specification's recovery mode section is excepted from the suspension for P1-RECOVERY, and ADR-001 has the amendment for the independent Runner's credential file. CodeSpace's DevGuard pin stays 6e7e065.

    P1R-C01 may ask for its merge once its pull request is green at its exact head.

    Generated by Claude Code

  3. novelKR commented on Oct 14, 2026

    @novelKR
    novelKR
    on Oct 14, 2026 – with Claude
    OwnerAuthor

    2026-10-14, P1R-C01 merged and verified; P1R-C02 started. #102 was merged on the owner's approval at head c22e823, as main feb5f52. Post-merge CI run 38592385183 passed all 17 jobs, hosted macOS included. The approval also covered the contract changes listed in #102. With it the owner started P1R-C02. The DevGuard pin stays 6e7e065.

    The DevGuard ledger update, P1R-C01 merged-verified and P1R-C02 active, is novelKR/DevGuard#37 (head a623275, CI green), waiting for the owner's merge approval.

    Generated by Claude Code

  4. novelKR commented on Oct 14, 2026

    @novelKR
    novelKR
    on Oct 14, 2026 – with Claude
    OwnerAuthor

    2026-10-14, P1R-C02 merged and verified; P1R-C03 started. #103 was merged on the owner's approval at head a4f362c, as main 5976df0. Post-merge CI run 38612003338 passed all 17 jobs, hosted macOS included. The approval also covered the contract changes listed in #103. With it the owner started P1R-C03. The DevGuard pin stays 6e7e065.

    DevGuard ledger: novelKR/DevGuard#37 (P1R-C01 merged-verified, P1R-C02 active) was merged on the owner's approval as DevGuard main 0f4bf73, post-merge CI green. The next update, novelKR/DevGuard#38 (P1R-C02 merged-verified, P1R-C03 active), was merged on the owner's approval at head bd078f5 as DevGuard main 77293a2; post-merge CI run 38616346872 is green.

    Generated by Claude Code

  5. novelKR commented on Oct 15, 2026

    @novelKR
    novelKR
    on Oct 15, 2026 – with Claude
    OwnerAuthor

    2026-10-15, P1R-C03 merged and verified; P1R-C04 started. #104 was merged on the owner's approval at head 4e54309, as main 1fda789. Post-merge CI run 38686514388 passed all 17 jobs, hosted macOS included. In its first attempt the macOS contracts job failed in its "macOS contracts and dependencies" step, on a tree whose PR run 38631018229 had passed; the one re-run of that job passed. The approval also covered the contract changes listed in #104. With it the owner started P1R-C04. The DevGuard pin stays 6e7e065.

    DevGuard ledger: novelKR/DevGuard#39 (P1R-C03 merged-verified, P1R-C04 active) was merged on the owner's approval at head 760041a as DevGuard main 7cd86e4; post-merge CI run 38708338123 is green.

    Generated by Claude Code

  6. novelKR commented on Oct 15, 2026

    @novelKR
    novelKR
    on Oct 15, 2026 – with Claude
    OwnerAuthor

    2026-10-15, P1R-C04 merged and verified; P1R-C05 started. #105 was merged on the owner's approval at head 7cabcba, as main 385e177. Post-merge CI run 38708302887 passed, with Linux isolation skipped by the plan as on the PR. In its first attempt the macOS contracts job failed in the runner's DevGuard tests (cargo test -p codespace-runner --features devguard --lib -- registration wire admission launch independent lifecycle exited 101), on a tree whose PR run 38704449082 had passed; the one re-run of that job passed. Which test failed is in that attempt's artifact only. The approval also covered the contract changes listed in #105. With it the owner started P1R-C05. The DevGuard pin stays 6e7e065.

    Follow-up, tests only: #106 (head 41db94c), opened while looking for that failure. A P1R-C03 test in the same command, an_independent_runner_answers_registration_before_the_next_claim, failed 1 run in 25 locally; #106 fixes it and makes five tests from #105 wait for a process's end rather than its exit. It is not a unit; it was merged on the owner's approval as main 950e2b2. Its post-merge CI run 38718331468 passed after one re-run: in the first attempt only the Linux integration job failed (cargo test --locked --workspace exited 101; the failing test is in that attempt's artifact only).

    DevGuard ledger: novelKR/DevGuard#40 (P1R-C04 merged-verified, P1R-C05 active) was merged on the owner's approval at head 7b6ad73 as DevGuard main 494793a; post-merge CI run 38718406648 is green.

    Generated by Claude Code

  7. novelKR commented on Oct 15, 2026

    @novelKR
    novelKR
    on Oct 15, 2026 – with Claude
    OwnerAuthor

    2026-10-15: what P1R-C05 found outside its unit

    While driving #108, the P1R-C05 work found:

    1. A race in a P1R-C04 test. mcp::independent_tests::an_independent_gateway_blocks_a_workspace_whose_records_and_runner_disagree failed runner, server, store: answer for a lost Runner's processes and let an operator release them (P1R-C05) #108's Linux Integration job on 5a41aaf (run 38728457392), whose log named it thanks to scripts: name a failed stage's tests in the job log #107. It fails on main 3fac934 too (5 of 64 runs at 8 in parallel) and is probably main 950e2b2's Integration failure (run 38718331468). The test terminates its sandboxed command while the sandbox is still being set up. A test-only fix, terminating once the command has printed a line, passed 80 of 80 runs on main. Details: comment on #108.
    2. A race in P1R-C04's reclaim. a_gateway_that_started_work_claims_its_runner_again_and_reconciles_it fails about 1 in 100 runs at 16 in parallel, on main and on runner, server, store: answer for a lost Runner's processes and let an operator release them (P1R-C05) #108's branch. An exit that the Runner sends right after the new Claim's answer reaches the records' exit-following loop (crates/server/src/records.rs) before reclaim_loop (crates/runner/src/uds.rs) makes the link current. The status query is then refused as reconnecting, and the record ends as "exited, how not known", which reconciliation cannot correct. In use, this loses the exit code of a process that exits around a reconnection; nothing runs twice.
    3. Terminating a sandboxed command during sandbox setup on Linux leaves it running outside the Runner's reach: runner: a sandboxed command terminated during its sandbox setup keeps running (Linux) #109.
    4. Open from runner, server, store: answer for a lost Runner's processes and let an operator release them (P1R-C05) #108: a read-only DevGuard admission status command, since --open reopens admission.

    Plan: 1 and 2 go into one follow-up PR after #108 merges, as #106 did for P1R-C04, so P1R-C06's Mac qualification runs on a main that has them. #109 stays outside the units unless the owner adds it.

    Generated by Claude Code

  8. novelKR commented on Oct 15, 2026

    @novelKR
    novelKR
    on Oct 15, 2026 – with Claude
    OwnerAuthor

    2026-10-15, P1R-C05 merged and verified; P1R-C06 started. #108 was merged on the owner's approval at head 5deb102, as main dadb58c. Post-merge CI run 38739506757 passed all 17 jobs on the first attempt, macOS contracts and Linux isolation included, and the documentation site run 38739507537 passed. The approval also covered the contract changes listed in #108. With it the owner started P1R-C06. The DevGuard pin stays 6e7e065.

    CI tooling, not a unit: #107 and #110 make a failed CI stage list its failed tests and cargo's rerun targets in the job log, never their output. Both were merged on the owner's approval, #107 at head dea6c47 as main 3fac934 and #110 at head dfe9144 as main e947e80; their post-merge CI runs 38728182277 and 38739632977 passed on the first attempt.

    Follow-up for findings 1 and 2 above: #111 (head 57908bc), from the P1R-C05 work, was merged on the owner's approval as main 345864d; its post-merge CI run 38743295778 passed on the first attempt. P1R-C06's macOS qualification runs on a main that includes it.

    DevGuard ledger: novelKR/DevGuard#41 (P1R-C05 merged-verified, P1R-C06 active) was merged on the owner's approval at head cf3e206 as DevGuard main ec66b12; post-merge CI run 38743349871 is green.

    Generated by Claude Code

  9. novelKR commented on Oct 15, 2026

    @novelKR
    novelKR
    on Oct 15, 2026 – with Claude
    OwnerAuthor

    2026-10-15, P1R-C06's harness and protocol merged. #112 was merged on the owner's approval at head 8de5fdb, as main b2cb042. Post-merge CI run 38767662399 passed all 17 jobs on the first attempt, and the documentation site run 38767664121 passed. It adds the recovery-* functional suites and the recovery restart measurement to scripts/qualify-devguard.py, the Gateway recovery qualification protocol in English and Korean, and the independent Runner's adoption and rollback to the operations guide. It changes no MCP tool, error code or status value, and no pin. The independent mode stays unsupported until the macOS run qualifies a combination.

    Next, in P1R-C06's thread: the macOS check and candidate freeze on b2cb042 or a later main, the measurement, then a docs-only report PR. The P1R-C06 box is ticked when that report merges, and the DevGuard ledger then records P1-RECOVERY's result.

    Generated by Claude Code

  10. novelKR commented on Oct 17, 2026

    @novelKR
    novelKR
    on Oct 17, 2026 – with Claude
    OwnerAuthor

    2026-10-17, harness fix for P1R-C06's macOS run. The first recovery run on candidate b2cb042 stopped in its rehearsal as harness_failed: the harness kept open files past Terminal's default limit of 256, so every functional scenario from the 27th on failed with Errno 24. Nothing failed in CodeSpace or DevGuard. #113 (scripts only) keeps the harness within that limit; it was merged on the owner's approval at head 4297da4 as main 41f83b6, and its post-merge CI run 39032408626 passed all 17 jobs on the first attempt, as did the documentation site run 39032408913. The macOS run now uses candidate 41f83b6; the b2cb042 candidate and its evidence are kept and will be listed in the report as superseded.

    Generated by Claude Code

  11. novelKR commented on Oct 18, 2026

    @novelKR
    novelKR
    on Oct 18, 2026 – with Claude
    OwnerAuthor

    2026-10-18, second harness fix for P1R-C06's macOS run. The full recovery run on candidate 41f83b6 failed on three harness timing races right after gateway restarts: the harness sent new work a few milliseconds before the new gateway had reconciled the workspace, or before it had freed a workspace after an exit, and counted CodeSpace's documented refusals (RECOVERY_PENDING, WORKSPACE_BUSY) as failures. No command ran twice and no refused command ran. #114 (the harness, its tests and the protocol text only) sends such work again for a bounded time; it was merged on the owner's approval at head 727e45f as main b5ddd59, and its post-merge CI run 39091754442 passed all 17 jobs on the first attempt, as did the documentation site run 39091754677. The candidate is frozen again at b5ddd59; b2cb042 and 41f83b6 and their evidence are kept and will be listed in the report as superseded.

    Generated by Claude Code

  12. novelKR commented on Oct 18, 2026

    @novelKR
    novelKR
    on Oct 18, 2026 – with Claude
    OwnerAuthor

    2026-10-18, third harness fix for P1R-C06's macOS run. The full recovery run on candidate b5ddd59 failed on one termination sample whose release the harness could not watch: the harness's reader of a gateway's log wrote each line to the evidence before parsing it, and with the evidence disk busy under the load, the launch's attempt ID reached the harness after its 5-second wait had run out. DevGuard released that attempt itself (scope_terminated), and none of the three races of the 41f83b6 run came back. #115 (the harness, its tests and the protocol text only) indexes each log line first and writes the log file on a thread of its own, and counts a log it cannot write as raw evidence that could not be written; it was merged on the owner's approval at head 80dc2eb as main 1c0eadd, and its post-merge CI run 39114783991 passed all 17 jobs on the first attempt, as did the documentation site run 39114784286. The next run freezes the candidate again at 1c0eadd; b2cb042, 41f83b6 and b5ddd59 and their evidence are kept and will be listed in the report as superseded.

    Generated by Claude Code

  13. novelKR commented on Oct 19, 2026

    @novelKR
    novelKR
    on Oct 19, 2026 – with Claude
    OwnerAuthor

    2026-10-19, P1R-C06 qualified and reported. The runs on the candidate 1c0eadd qualified the independent Runner on macOS for one exact combination: uds with --runner-lifecycle independent and a Runner the operator starts, pipe and PTY executions, participations off and required, and both in one gateway, with DevGuard's client 6e7e065 and the release 0.1.0-6e7e065-94d85365 under the current policy. The five check suites, the freeze, the rehearsal and the full recovery run passed; the run made three repetitions of a 10-minute idle interval and a 30-minute load interval, with 308 gateway restarts in all. CS-RG's candidate, matrix, faults and authority passed again on the same candidate. Linux off is CI functional verification only, and Linux required and restoring processes after the Runner itself restarts stay unsupported.

    The report, #116 (documentation only, the code tree unchanged), was merged on the owner's approval at head 30165c9 as main d0f2e49: the qualification report. Its post-merge CI run 39197574489 passed, as did the documentation site run 39197574724, which deployed the site. Next, DevGuard's ledger records P1-RECOVERY as implemented with this qualification.

    Generated by Claude Code

  14. novelKR commented on Oct 19, 2026

    @novelKR
    novelKR
    on Oct 19, 2026 – with Claude
    OwnerAuthor

    2026-10-19, P1-RECOVERY recorded as implemented and qualified on macOS. DevGuard's ledger now records P1-RECOVERY as implemented, qualification macos-qualified, with its qualified_combination (CodeSpace 1c0eadd, DevGuard's client 6e7e065, the release 0.1.0-6e7e065-94d85365, Codex 6b9826e with the recorded patch, Rust 1.95.0, uds with --runner-lifecycle independent), and P1R-C06 as merged-verified; the dated notes, V-P1 and the R3 gate cite the qualification report. novelKR/DevGuard#45 (documentation and ledger only) was merged on the owner's approval at head 4044575 as DevGuard main 4b50958, and its post-merge CI run 39215131839 passed.

    Every box above is ticked, and on the owner's decision this tracker is closed as completed. The independent mode stays opt-in; making it the default, Linux required (DG-LINUX) and restoring processes after the Runner itself restarts stay outside P1-RECOVERY. #109 and the read-only DevGuard admission status command noted here on 2026-10-15 stay open outside it.

    Generated by Claude Code

  15. novelKR
    closed this as completedon Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentation

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions