Skip to content

Repair patch generation and add honest, matched evaluation controls - #4

Merged
YeDonS merged 6 commits into
mainfrom
fix/evidence-patches-and-controls
Oct 8, 2026
Merged

YeDonS merged 6 commits into
mainfrom
fix/evidence-patches-and-controls

Conversation

@YeDonS

@YeDonS YeDonS commented Oct 8, 2026 •

Copy link
Copy Markdown
Owner

The original 100-task run never invoked the test specialist, and 17 generated candidates were withheld after failing applicability checks. Public documentation now states that scope explicitly, removes automated causal agreement from headline performance, corrects the passing requests-2674 example, and documents sample-distribution and inference-portability limits.

This adds source reads during synthesis, source-priority evidence handoffs, bounded git-apply-error repair with complete attempt records, regular text-file creation/deletion, shared model/tool call ceilings, and a single-conversation control. A predeclared 20-task three-arm experiment and a separate repair study of all 17 invalid candidates preserve the original 48/100 rather than changing it retrospectively.

The selected post-hoc repair follow-up recovered 17/17 applicable patches and resolved 11/17 in official tests (six functional failures). The development run is retained separately: 14 applicable, 9 resolved, three provider errors. Neither run changes the original 48/100 or pools the best patches across attempts.

The predeclared 20-task controls resolved 11/20 with a single conversation and 10/20 under both routed window policies. All three arms share the same frozen source/model and ten model/eighteen tool call ceilings, with repairs and tests disabled. Actual token use differs; single used fewer calls/reported tokens. These exploratory results do not establish a routed advantage. A deterministic replay of the original 100 ledgers found 13 partial source-loss cases and no case losing all source; source retention improved without a measured resolution gain in the paired pilot.

Validation: 107 offline tests passed; total source coverage 84.9%, CLI coverage 72.2% (previously zero). Ruff lint/format and wheel build pass. All 94 new official outcomes (60 controls + two independent repair sets of 17) are complete and bound to exact patch/source hashes, with durable reports, image digests, excerpts and raw-log manifests. Current code also hardens new-file citation boundaries and clears generated stale checker notices. Review is a source/trace/hash self-review; no human causal review or live native-provider API call is claimed.

Remaining limits are explicit: historical inference did not run the test specialist; the architecture control covers 20 tasks with call ceilings rather than exact token-cost matching; public-data training contamination, account-specific CLI availability, and trusted local execution without an OS sandbox remain. Independent inference uses provider APIs; native API credentials were unavailable for live verification.

@YeDonS
YeDonS marked this pull request as ready for review October 8, 2026 22:23
@YeDonS
YeDonS merged commit 0490b07 into main Oct 8, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant