[novelKR]

Issue

Part of #101: P1R-C06. The full recovery run on the qualification host failed on three timing races in the harness, so this fixes the harness and its protocol text before the candidate is frozen again. It follows #113 and does not close an issue.

Item	Value
Base	main at 41f83b6 (the merge of #113)
Failed run	p1r-c06-recovery-2026-10-17-41f83b6 on the qualification host, MANIFEST.json SHA-256 1b2b57fbea66ae3a08b78dcd76d86f992dc4709a21e22a51343b97cf0f0b22a2; verify accepts it (recorded and recomputed failed). Its candidate (37ed6aa3…) passed and its rehearsal (aad4c41e…) was inconclusive as designed
Pins	Codex gitlink with its recorded patch, and DevGuard 6e7e065; both unchanged

Summary

Before: right after a gateway restart, the harness sometimes sent new work a few milliseconds too early and counted CodeSpace's documented answer as a failure. In the full run, which otherwise passed (53 of 55 functional scenarios; 5 of 6 intervals; every pressure sample normal):

• A termination sample of repetition 1's load interval was sent 5 ms after the 40th gateway's Claim and 64 ms before that gateway had reconciled the term workspace. It was refused with RECOVERY_PENDING and recorded as an error, and with fewer than 100 samples one error fails the termination p99.
• recovery-reconcile required approval held the executions of its 80 ms and 160 ms rounds about 12 ms after the eighth gateway started, before it had reconciled confirm: that gateway had answered the previous round's two resumes from its record at once. Both were refused with RECOVERY_PENDING, so those two rounds never ran.
• recovery-reconcile required pipe kill-during-terminate started its next round, and then its fresh command, within 0.6 ms of process_status reporting the terminated process exited. Both were refused with WORKSPACE_BUSY: the Runner reports a process exited once it has reaped it, and the gateway frees the workspace only when the Runner's exit event follows, once the process's output has closed.

No command ran twice and no refused command ran.

After: where the harness needs new work started, it sends it again while it meets one of these two refusals, for a bounded time, and the protocol says so. A scenario whose row expects one of these refusals still checks it, and a refusal that lasts is still a failure.

How.

• once_free (scripts/qualify-devguard.py) sends new work again while it is refused with RECOVERY_PENDING or WORKSPACE_BUSY, for up to 30 seconds. once_reconciled keeps its meaning (only RECOVERY_PENDING) and now takes the codes to wait through.
• Functional scenarios: reconcile running and kill-during-terminate and observation retention start new work through once_free where a process just exited; store writer-lock and schema and the approval scenario's held execution wait for the reconciliation through once_reconciled. kill-during-terminate records the refusals each round's start met.
• Measurement samples: a termination's target and a refusal's execution are sent again, each time under a new tag, while the serving gateway refuses them with RECOVERY_PENDING, for up to 10 seconds (RECONCILE_WAIT_S). A target still refused then is an error, as before; a capacity refusal is still not a sample. Each row records the refusals it met (pending_refusals), and every attempt stays in execs.jsonl, whose markers check still finds a refused command that ran.
• Protocol (docs/recovery-qualification.md and the Korean page): the functional suites state both refusals and the rule, and the measurement states the samples' wait. The registry hashes are recorded.

The protocol freezes the harness with it, and a change after the first run is a new protocol, reviewed like any other change, whose affected runs are repeated: on this merge the candidate is frozen again and the whole recovery run is repeated.

Contract changes

None. The product is unchanged; only the harness, its tests and the protocol text change. Termination and refusal sample rows gain pending_refusals, and kill-during-terminate rounds gain start_refusals and start_waited_ms; verify recomputes the same metrics from them.

A product note for the report: process_status reports a process exited once the Runner has reaped it, a little before the gateway frees its workspace on the Runner's exit event. The server's instructions for the independent lifetime say that new work in the workspace of a process found running "returns WORKSPACE_BUSY until the process exits or is terminated", which reads as the same moment. On the host both refusals came within 0.6 ms of the exit report.

Tests

Verified here at the head of this branch, in a Linux cloud container, with debug builds of codespace-mcp and codespace-codex-runtime (the product is unchanged since b2cb042):

• python3 -B -m unittest discover -s scripts/tests: 124 passed, 2 of them new. once_reconciled waits for the reconciliation only, while once_free also waits while the workspace is still held; both answer at once to any other refusal, and at their limit to one that lasts. A termination target refused with RECOVERY_PENDING is sent again under a new tag and timed once it started; one refused for capacity is not a sample and is not sent again; one still refused when the wait ends is an error. The refusal sample times the authority's refusal, not the reconciliation's.
• pyflakes on the four scripts; scripts/check_docs.py, and the docs site's test, build and check as the docs workflow runs them.
• recovery --rehearsal --participation off under a soft limit of 256 (ulimit -Sn 256): functional 34 passed, 2 gaps not run; idle passed; load inconclusive by design here ("no load ran: there is no DevGuard service to run it"); the run inconclusive (exit 2), verify verified; at most 43 open descriptors. It ran at da8383a, whose harness and tests are this head's (one Korean word of the protocol differs); MANIFEST.json SHA-256 4cc876606194596aa9aaa3df1ad75922bbc247619b20e881ffc431fa80a4b78b.
• The races do not reproduce in this container, where an off gateway reconciles within a few milliseconds. No scenario or sample of that rehearsal met either refusal (each records the refusals it waited through, all 0). The unfixed harness at the protocol's durations (one off repetition: functional, a 10-minute idle and a 30-minute load interval, 105 restarts) met neither refusal either: functional 34 passed, idle passed, load inconclusive by design, verify verified.
• With every CPU saturated (12 busy loops on 4 CPUs), recovery-reconcile and recovery-store passed twice with each harness. Gateways then did refuse some requests with RECOVERY_PENDING, but only where the unfixed harness already waited (the kill-during-exec probes and the approval resumes).

Not verified here: required, which runs only on macOS, where reconciling asks the authority and took about 70 ms under load; and the measurement against the installed service. On the merge, the qualification host builds the merged main, and the owner's paste runs the five checks, freezes the candidate, and repeats the rehearsal and the full run.

Security scenarios

• Rollback honesty: the waits apply only to the two refusals the gateway documents as transient, and only where the harness needs new work started; each is bounded, and a refusal that outlasts it fails the scenario or is an error sample. Every attempt is kept as evidence, and the markers still catch a refused command that ran.
• Credentials: unchanged; the harness passes only the credential file's path and reads no credential.

Out of scope

• The repeated qualification run and its report, which follow on the candidate frozen at this merge.
• Changing when the gateway reports a process exited or frees its workspace; that is product behavior, noted for the report.
• Opening the harness's serving gate only once a gateway has reconciled every workspace: the samples would then no longer cover the time right after a Claim.