Repository navigation
Repair patch generation and add honest, matched evaluation controls - #4
Merged
Merged
Conversation
YeDonS
marked this pull request as ready for review
October 8, 2026 22:23
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The original 100-task run never invoked the test specialist, and 17 generated candidates were withheld after failing applicability checks. Public documentation now states that scope explicitly, removes automated causal agreement from headline performance, corrects the passing requests-2674 example, and documents sample-distribution and inference-portability limits.
This adds source reads during synthesis, source-priority evidence handoffs, bounded git-apply-error repair with complete attempt records, regular text-file creation/deletion, shared model/tool call ceilings, and a single-conversation control. A predeclared 20-task three-arm experiment and a separate repair study of all 17 invalid candidates preserve the original 48/100 rather than changing it retrospectively.
The selected post-hoc repair follow-up recovered 17/17 applicable patches and resolved 11/17 in official tests (six functional failures). The development run is retained separately: 14 applicable, 9 resolved, three provider errors. Neither run changes the original 48/100 or pools the best patches across attempts.
The predeclared 20-task controls resolved 11/20 with a single conversation and 10/20 under both routed window policies. All three arms share the same frozen source/model and ten model/eighteen tool call ceilings, with repairs and tests disabled. Actual token use differs; single used fewer calls/reported tokens. These exploratory results do not establish a routed advantage. A deterministic replay of the original 100 ledgers found 13 partial source-loss cases and no case losing all source; source retention improved without a measured resolution gain in the paired pilot.
Validation: 107 offline tests passed; total source coverage 84.9%, CLI coverage 72.2% (previously zero). Ruff lint/format and wheel build pass. All 94 new official outcomes (60 controls + two independent repair sets of 17) are complete and bound to exact patch/source hashes, with durable reports, image digests, excerpts and raw-log manifests. Current code also hardens new-file citation boundaries and clears generated stale checker notices. Review is a source/trace/hash self-review; no human causal review or live native-provider API call is claimed.
Remaining limits are explicit: historical inference did not run the test specialist; the architecture control covers 20 tasks with call ceilings rather than exact token-cost matching; public-data training contamination, account-specific CLI availability, and trusted local execution without an OS sandbox remain. Independent inference uses provider APIs; native API credentials were unavailable for live verification.