Repository navigation
Fix Raft membership fence and CI qualification failures - #165
Merged
Merged
Conversation
Debug codec work exhausted fixed protocol windows. Keep assertions and overflow checks enabled at opt-level 1.
Reuse the canonical durable fixture namespace so node lease CAS does not queue behind application data writers. Preserve the lease, handler, audit, and shutdown contracts.
Retain the last physical copies and topology service state on failure. Keep exact completion predicates and the original observation deadlines.
forhappy
marked this pull request as ready for review
November 2, 2026 03:49
A paused append stream can miss the final commit broadcast. Drain prior sends, announce the committed index and term under durable leader authority, and drain that finite notification set before revoking membership. Preserve receiver log matching and physical closure proofs.
Use the canonical FULL-synchronous namespace for both live worker cases and recovery. Keep ordinary reads, CAS and listing coherent while separating node authority from application-data writers. Preserve all shutdown and receipt assertions.
Keep evidence, service startup, CLI, SQL and finalizing-route failures distinguishable without changing observation deadlines or success predicates.
Start the dormant repair participant, route the same permit at the highest observed term, and await complete restart proofs within existing bounds. Match the two archive-readiness gauges in the exact metrics inventory. Qualify changes to the shared durable fixture source.
A queued seed worker can start after the warm inventory. Keep the same\ncompletion identity and require its genuine receipt within the existing\nfive-second readiness bound. Preserve independent API probes and floors.
On the same x64 runner, 100 fresh cases had restarted-write p99 142.579 ms and max 178.144 ms. Two drained cohorts (200 cases) retained p95 near 19 ms while reducing p99 to 20.498 ms and max to 21.267 ms. All cases passed. Prebuild the unchanged test artifact and sync its filesystem before the existing 100-case campaign. Preserve FULL sync, all assertions, and every timeout. Kernel writeback tuning added no gain and is excluded.
On x64, fresh lease-commit p95 was 1.101 ms with a 421.319 ms maximum. Three drained 128-case cohorts passed with p95 0.836-0.906 ms and maximum 1.845-2.212 ms. ARM reproduced a fresh 692.225 ms commit and failure; all three drained cohorts passed. Build the existing full-feature qualifier and flush completed compiler output before live campaigns. Retain every request/lease deadline, FULL sync, original command, and cleanup assertion. Kernel tuning did not consistently improve on drainage and is excluded.
A full FT matrix still encountered simultaneous 1.08-1.14 second COMMIT stalls on three independent authority files after the build flush; nodes correctly fenced. Use the previously measured 4/16 MiB dirty limits and 1s/500ms expiry/writeback policy on the disposable x64 RPC runner. Paired x64 cohorts reduced median commit p95 from 873 to 765 us (12.4%); the bounded full matrix passed with max commit 4.528 ms. Restore the original policy in an always-run step. Keep FULL sync, filesystem barriers, every lease/request deadline, matrix case and cleanup assertion unchanged. This is runner isolation, not production storage tuning.
An ignored voter offer could hold the only source slot for a learner's entire deadline. Atomically retire unclaimed offers while protecting active transfers and retaining physical I/O reservations. Select partial-body faults from actual bulk bytes, verify incompressible SQL state after recovery, and match the native TCP_NODELAY policy so the proxy cannot add delayed-ACK stalls. Keep all deadlines and correctness assertions.
Shared data transactions delayed FULL-sync lease CAS past the one-second authority bound. Reuse the canonical fixture namespace so authority keys keep independent durable writers and coherent ordinary reads. The outage wraps the entire namespace before either owner starts. Keep metrics, fencing, pre-replay audit and closure assertions intact. Expose the exact expired-readiness stop error without changing its accepted outcomes.
Bound read-only proof reset and exact-request stop reconciliation by their existing deadlines. Preserve mutation attempt and closure checks. Prebuild dense qualification with optimized code, debug assertions, and overflow checks; drain build writeback before live proof.
Return cancellation when forced closure refuses listener or drain-token admission. Keep listeners owned and joined; never fabricate graceful proof. The old closed-domain panic reproduces. Candidate boundary, 968 library, 109 native-node and 200 unchanged management receipt cases pass, together with strict Clippy and repository checks.
Reuse the coherent FULL-sync namespace so node lease CAS writes cannot queue behind application-data SQLite transactions. Keep held-result, cleanup, lease and resource assertions unchanged. Both full RPC ABI matrices pass. The same 144 saturation cases complete 43% faster with regular Python and 20% faster with free-threaded Python across the retained comparisons; this is qualification campaign throughput.
Prebuild the no-default-features artifact and finish build writeback before the original one-second lease checks. Keep test concurrency, assertions, feature commands and timeout budgets unchanged. Ten alternating full-package runs pass; sampled lease-write commit p95 falls from 7.25 ms to 3.91 ms. All-features tests and strict Clippy pass.
Report setup, activation, capacity, observation, repair and join context without changing requests, retries, assertions or timeout budgets. The original Overloaded refusal did not reproduce in the instrumented 292-case fleet suite or 100 fresh minority/majority repair cases. Retain phase context so the complete SDK campaign can identify a recurrence.
Retain one runtime event with its admission until the preceding durable snapshot finishes SQL publication. Earlier overlaps and a second deferred event still close admission; read version and image proof gates remain unchanged.
A concurrent inspection of a freed worker mutex could revoke the observed drain proof and make generation retirement miss its maintenance tick. Publish reclamation after destruction; retain active-work and slot-reuse checks and the original retirement deadlines.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix durability and CI failures inherited from #163 while preserving acknowledgement proofs, ownership and epoch checks, FULL-synchronous durability, correctness assertions and timeout budgets.
Validation on published head
300df5ceb64d70f796d811f7c565e1232bcb1954is complete: all six original workflows succeeded; 33 checks passed, all four original Rust framework/SDK aggregate gates passed, and the same six optional contexts were skipped. All four native campaigns passed the strengthened retained-route regression, 970 library tests, all 292 fleet cases and 54 snapshot cases, retaining the existing ignores. All four ABI/SDK jobs, JavaScript qualification, both Python example workflows, all release application jobs and every Typed RPC job passed. All four RPC jobs restored the runner writeback policy successfully. No correctness assertion or timeout budget was relaxed.Additional regression evidence:
Measured Linux qualification gains: dense proof p95 408 → 103 ms and median first post-recovery write 369 → 120 ms; unchanged 144-case saturation campaign median 28.08 → 16.04 seconds with regular Python and 28.40 → 22.76 seconds free-threaded; sampled Axum lease-write commit p95 7.25 → 3.91 ms. These are qualification workload measurements, not production TPS.
Limits: the earlier Node delivery timeout at the original 250 ms budget did not recur in 2,000 original-Cargo diagnostic cases and 67 traced cases, a separate complete 200-case syscall campaign, or the 100-case original package qualification on the current head. No Node root-cause fix is claimed. Direct artifact execution shortened campaign launch time but worsened Worker-side restart POST p95/p99, so that experiment was discarded. Two earlier isolated fleet refusals also remain unexplained after the full fleet suite and 100 fresh targeted cases each. Focused failure context remains; assertions, refusal handling and deadlines were not relaxed.
Private combined diagnostic jobs were intentionally stopped before their unchanged 60-minute ceilings after preserving completed scopes; their cancelled workflows are not reported as whole-run successes. Production capacity, failover qualification and global Cloudflare latency remain separate work.