Repository navigation
Publish real LLM benchmark and 48/100 official patch resolutions - #2
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The project now has a completed real LLM experiment and official functional patch results for the same fixed 100 historical issues. This replaces the previous credential-blocked status with inspectable predictions, causal explanations, source citations, traces, and per-instance official reports.
The actor used
codex-cli:gpt-6.1-solwith medium reasoning and HTTPS through the CLI's existing local login. No authentication files or API credentials enter the repository or CI. Inference source stayed frozen at3dc26ac14fe7573bb84ef851262645353842ef58; reference patches and hidden tests were only used after each prediction froze. Historical inference is source-only; separate real-model CLI/HTTP checks actually exercise all three roles on the trusted synthetic fixture.Validation: 75 local tests passed; Ruff lint/format passed; sdist/wheel build and clean wheel installation reproduced the packaged fixture and found all 100 tasks. Verified all outcome IDs, frozen batch identities, patch/report/log hashes, source digest, JSON, documentation links, and credential-pattern scan. CI checks three Python versions and both Docker images. Official evaluation runs: batch 1, batch 2, batch 3, batch 4.
Limitations are retained: training contamination on public historical issues, same-model review bias, model alias drift, 17 invalid candidates, 10 abstentions, and 2 test timeouts. No outcome-based repair, exclusion, human accuracy claim, or controlled orchestration ablation is implied.
Two post-hoc maintainer-patch controls also passed official required regressions with the same image digests (control workflow). Incidental fixture/import errors still occurred elsewhere in their output, so those messages alone do not invalidate the grade. Both original model failures remain unresolved; controls are separate from 48/100.