Skip to content

Fix Raft membership fence and CI qualification failures - #165

Merged
forhappy merged 44 commits into
mainfrom
codex/ci-163-followup
Nov 3, 2026
Merged

forhappy merged 44 commits into
mainfrom
codex/ci-163-followup

Conversation

@forhappy

@forhappy forhappy commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Fix durability and CI failures inherited from #163 while preserving acknowledgement proofs, ownership and epoch checks, FULL-synchronous durability, correctness assertions and timeout budgets.

  • Drain persisted Raft configuration notifications before membership revocation. Retire unclaimed snapshot offers only after authenticated fresh-seed authorization, retaining active transfers and physical reservations through cancellation.
  • Retain one next snapshot event with its original admission while the preceding durably installed image finishes SQL publication. Exact ticket/image read fences remain enforced; pre-install overlap and a second deferred event still fail closed.
  • Preserve completed worker reclamation during concurrent inspection of an empty worker mutex. Publish the monotonic proof only after worker destruction, retaining active-work, cell, facet and slot-reuse guards.
  • Return typed cancellation when shutdown listener or drain-token task admission encounters a closed execution domain; retain owned joins and refuse graceful completion without proof.
  • Make recovery fault tests interrupt the actual bulk transfer and verify the recovered incompressible SQL payload. Preserve signed closure, physical process joins and operation-specific retry rules. The Python recovery read validates the exact unknown-response envelope and never replays a mutation.
  • Give durability fixtures one coherent FULL-sync namespace with independent node-key writers. Prebuild and drain build writes before live lease qualification; retain debug assertions, overflow checks and every original qualification command and aggregate gate. Restore RPC runner writeback policy afterward.

Validation on published head 300df5ceb64d70f796d811f7c565e1232bcb1954 is complete: all six original workflows succeeded; 33 checks passed, all four original Rust framework/SDK aggregate gates passed, and the same six optional contexts were skipped. All four native campaigns passed the strengthened retained-route regression, 970 library tests, all 292 fleet cases and 54 snapshot cases, retaining the existing ignores. All four ABI/SDK jobs, JavaScript qualification, both Python example workflows, all release application jobs and every Typed RPC job passed. All four RPC jobs restored the runner writeback policy successfully. No correctness assertion or timeout budget was relaxed.

Additional regression evidence:

  • Pool reclamation qualification: the baseline fails both the native drain-proof assertion and the forced-contention retained-route test at its original 35-second retirement deadline. The identical published candidate passes both, the full 970-test library, 30 forced-contention route cases in 30.010–30.023 seconds, strict Clippy and repository checks. All original 2/35/45-second deadlines and epoch/cancellation/retired-route assertions remain intact.
  • Snapshot publication overlap reproduced in three fresh baseline cases. The candidate passes 200 fresh balance cases and six snapshot/crash suites totaling 324 tests; added ownership tests preserve event charges and reject invalid overlap phases. Separate balance and strict-lint campaign.
  • Closed-domain shutdown admission reproduces the original panic; the new boundary tests, full library, 109 native-node tests and 200 fresh management-stop receipt cases pass.
  • Both full RPC ABI matrices, five unchanged saturation campaigns per ABI, ten alternating complete Axum package runs, all-features tests and strict relevant Clippy pass.

Measured Linux qualification gains: dense proof p95 408 → 103 ms and median first post-recovery write 369 → 120 ms; unchanged 144-case saturation campaign median 28.08 → 16.04 seconds with regular Python and 28.40 → 22.76 seconds free-threaded; sampled Axum lease-write commit p95 7.25 → 3.91 ms. These are qualification workload measurements, not production TPS.

Limits: the earlier Node delivery timeout at the original 250 ms budget did not recur in 2,000 original-Cargo diagnostic cases and 67 traced cases, a separate complete 200-case syscall campaign, or the 100-case original package qualification on the current head. No Node root-cause fix is claimed. Direct artifact execution shortened campaign launch time but worsened Worker-side restart POST p95/p99, so that experiment was discarded. Two earlier isolated fleet refusals also remain unexplained after the full fleet suite and 100 fresh targeted cases each. Focused failure context remains; assertions, refusal handling and deadlines were not relaxed.

Private combined diagnostic jobs were intentionally stopped before their unchanged 60-minute ceilings after preserving completed scopes; their cancelled workflows are not reported as whole-run successes. Production capacity, failover qualification and global Cloudflare latency remain separate work.

@forhappy
forhappy marked this pull request as ready for review November 2, 2026 03:49
A paused append stream can miss the final commit broadcast. Drain prior sends, announce the committed index and term under durable leader authority, and drain that finite notification set before revoking membership. Preserve receiver log matching and physical closure proofs.
Use the canonical FULL-synchronous namespace for both live worker cases and recovery. Keep ordinary reads, CAS and listing coherent while separating node authority from application-data writers. Preserve all shutdown and receipt assertions.
Keep evidence, service startup, CLI, SQL and finalizing-route failures distinguishable without changing observation deadlines or success predicates.
Start the dormant repair participant, route the same permit at the highest observed term, and await complete restart proofs within existing bounds. Match the two archive-readiness gauges in the exact metrics inventory. Qualify changes to the shared durable fixture source.
A queued seed worker can start after the warm inventory. Keep the same\ncompletion identity and require its genuine receipt within the existing\nfive-second readiness bound. Preserve independent API probes and floors.
On the same x64 runner, 100 fresh cases had restarted-write p99 142.579 ms and max 178.144 ms. Two drained cohorts (200 cases) retained p95 near 19 ms while reducing p99 to 20.498 ms and max to 21.267 ms. All cases passed.

Prebuild the unchanged test artifact and sync its filesystem before the existing 100-case campaign. Preserve FULL sync, all assertions, and every timeout. Kernel writeback tuning added no gain and is excluded.
On x64, fresh lease-commit p95 was 1.101 ms with a 421.319 ms maximum. Three drained 128-case cohorts passed with p95 0.836-0.906 ms and maximum 1.845-2.212 ms. ARM reproduced a fresh 692.225 ms commit and failure; all three drained cohorts passed.

Build the existing full-feature qualifier and flush completed compiler output before live campaigns. Retain every request/lease deadline, FULL sync, original command, and cleanup assertion. Kernel tuning did not consistently improve on drainage and is excluded.
A full FT matrix still encountered simultaneous 1.08-1.14 second COMMIT stalls on three independent authority files after the build flush; nodes correctly fenced.

Use the previously measured 4/16 MiB dirty limits and 1s/500ms expiry/writeback policy on the disposable x64 RPC runner. Paired x64 cohorts reduced median commit p95 from 873 to 765 us (12.4%); the bounded full matrix passed with max commit 4.528 ms. Restore the original policy in an always-run step.

Keep FULL sync, filesystem barriers, every lease/request deadline, matrix case and cleanup assertion unchanged. This is runner isolation, not production storage tuning.
An ignored voter offer could hold the only source slot for a learner's
entire deadline. Atomically retire unclaimed offers while protecting
active transfers and retaining physical I/O reservations.

Select partial-body faults from actual bulk bytes, verify incompressible
SQL state after recovery, and match the native TCP_NODELAY policy so the
proxy cannot add delayed-ACK stalls. Keep all deadlines and correctness
assertions.
Shared data transactions delayed FULL-sync lease CAS past the one-second
authority bound. Reuse the canonical fixture namespace so authority keys
keep independent durable writers and coherent ordinary reads. The outage
wraps the entire namespace before either owner starts.

Keep metrics, fencing, pre-replay audit and closure assertions intact.
Expose the exact expired-readiness stop error without changing its
accepted outcomes.
Bound read-only proof reset and exact-request stop reconciliation by
their existing deadlines. Preserve mutation attempt and closure checks.

Prebuild dense qualification with optimized code, debug assertions,
and overflow checks; drain build writeback before live proof.
Return cancellation when forced closure refuses listener or drain-token
admission. Keep listeners owned and joined; never fabricate graceful proof.

The old closed-domain panic reproduces. Candidate boundary, 968 library,
109 native-node and 200 unchanged management receipt cases pass, together
with strict Clippy and repository checks.
Reuse the coherent FULL-sync namespace so node lease CAS writes cannot
queue behind application-data SQLite transactions. Keep held-result,
cleanup, lease and resource assertions unchanged.

Both full RPC ABI matrices pass. The same 144 saturation cases complete
43% faster with regular Python and 20% faster with free-threaded Python
across the retained comparisons; this is qualification campaign throughput.
Prebuild the no-default-features artifact and finish build writeback before
the original one-second lease checks. Keep test concurrency, assertions,
feature commands and timeout budgets unchanged.

Ten alternating full-package runs pass; sampled lease-write commit p95
falls from 7.25 ms to 3.91 ms. All-features tests and strict Clippy pass.
Report setup, activation, capacity, observation, repair and join context
without changing requests, retries, assertions or timeout budgets.

The original Overloaded refusal did not reproduce in the instrumented
292-case fleet suite or 100 fresh minority/majority repair cases. Retain
phase context so the complete SDK campaign can identify a recurrence.
Retain one runtime event with its admission until the preceding durable snapshot finishes SQL publication. Earlier overlaps and a second deferred event still close admission; read version and image proof gates remain unchanged.
A concurrent inspection of a freed worker mutex could revoke the observed drain proof and make generation retirement miss its maintenance tick. Publish reclamation after destruction; retain active-work and slot-reuse checks and the original retirement deadlines.
@forhappy
forhappy merged commit 981a541 into main Nov 3, 2026
39 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant