Skip to content

Publish real LLM benchmark and 48/100 official patch resolutions - #2

Merged
YeDonS merged 7 commits into
mainfrom
feat/model-benchmark-results
Oct 8, 2026
Merged

YeDonS merged 7 commits into
mainfrom
feat/model-benchmark-results

Conversation

@YeDonS

@YeDonS YeDonS commented Oct 8, 2026 •

Copy link
Copy Markdown
Owner

The project now has a completed real LLM experiment and official functional patch results for the same fixed 100 historical issues. This replaces the previous credential-blocked status with inspectable predictions, causal explanations, source citations, traces, and per-instance official reports.

  • Real inference: 100 complete, 90 cited diagnoses, 10 abstentions; file Hit@1 87%, Hit@5 88%, MRR@5 0.875.
  • Official SWE-bench 5.0.2: 48/100 resolved, 23 unresolved, 27 empty patches, 2 test timeouts (600 seconds), no missing outcomes. Four frozen batches preserve original task order and exact patch hashes.
  • Automated post-freeze causal comparison: 87 correct, 1 partial, 2 incorrect, 10 unscorable. This is labeled automated agreement; human accuracy remains null.
  • Added durable official reports, image digests, test excerpts, raw-log hashes, invalid candidates, failure analysis, real authenticated HTTP evidence, and updated English/Chinese README and reproduction documentation.

The actor used codex-cli:gpt-6.1-sol with medium reasoning and HTTPS through the CLI's existing local login. No authentication files or API credentials enter the repository or CI. Inference source stayed frozen at 3dc26ac14fe7573bb84ef851262645353842ef58; reference patches and hidden tests were only used after each prediction froze. Historical inference is source-only; separate real-model CLI/HTTP checks actually exercise all three roles on the trusted synthetic fixture.

Validation: 75 local tests passed; Ruff lint/format passed; sdist/wheel build and clean wheel installation reproduced the packaged fixture and found all 100 tasks. Verified all outcome IDs, frozen batch identities, patch/report/log hashes, source digest, JSON, documentation links, and credential-pattern scan. CI checks three Python versions and both Docker images. Official evaluation runs: batch 1, batch 2, batch 3, batch 4.

Limitations are retained: training contamination on public historical issues, same-model review bias, model alias drift, 17 invalid candidates, 10 abstentions, and 2 test timeouts. No outcome-based repair, exclusion, human accuracy claim, or controlled orchestration ablation is implied.

Two post-hoc maintainer-patch controls also passed official required regressions with the same image digests (control workflow). Incidental fixture/import errors still occurred elsewhere in their output, so those messages alone do not invalidate the grade. Both original model failures remain unresolved; controls are separate from 48/100.

@YeDonS
YeDonS merged commit 6e74922 into main Oct 8, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant