Publish real LLM benchmark and 48/100 official patch resolutions

.github/workflows/ci.yml

@@ -17,8 +17,8 @@ jobs:
           enable-cache: true
           python-version: ${{ matrix.python }}
       - run: uv sync --locked --all-extras
-      - run: uv run ruff check src tests
-      - run: uv run ruff format --check src tests
+      - run: uv run ruff check src tests scripts
+      - run: uv run ruff format --check src tests scripts
       - run: uv run pytest --cov=code_assistant --cov-report=xml
       - name: Run real offline reproduction
         run: |

README.md

@@ -13,18 +13,20 @@ registered source citations. Patches are proposed without changing the repositor
 |---|---|
 | Search / test / synthesis agents and conditional routing | Implemented and covered by offline integration tests |
 | Search, source reads, pytest execution, log inspection | Real tools with budgets and structured events |
-| FastAPI, bearer authentication, OpenAPI docs | Locally tested |
-| Docker service, restricted runner, Compose | Build and execution checks passed in GitHub Actions |
+| FastAPI, bearer authentication, OpenAPI docs | Real-model HTTP check passed: 401 without token, 200 with token |
+| Docker service and restricted runner | Both image builds and execution checks passed in GitHub Actions |
 | 100 historical issues across 12 repositories | Pinned, reproducible SWE-bench Lite subset |
 | Retrieval baseline on all 100 pre-fix snapshots | **100 completed; Hit@1 40%; Hit@5 71%; MRR@5 0.5202** |
-| Real-model root-cause accuracy | Integration and review workflow implemented; **not evaluated** |
-| Generated-patch resolution | Official harness export/scoring implemented; **not evaluated** |
+| Real LLM run on all 100 issues | **90 cited diagnoses, 10 abstentions; file Hit@1 87%, Hit@5 88%** |
+| Automated root-cause comparison | **87 correct, 1 partial, 2 incorrect, 10 unscorable**; human accuracy remains unmeasured |
+| Generated-patch resolution | **48/100 resolved** in official SWE-bench tests; 23 failed, 27 empty, 2 timed out |
 
 Baseline numbers measure **file localization against maintainer patches**, not bug-fix accuracy.
-The first three historical tasks have now been diagnosed with a real model through a locally
-authenticated Codex CLI. All three patches apply; official functional evaluation is separate.
+The real run uses `gpt-6.1-sol` through a locally authenticated Codex CLI, with a frozen
+configuration and source fingerprint. Automated causal comparison is a labeled LLM judgment
+with possible same-model bias. Functional resolution comes from the official SWE-bench harness.
 The scripted demo is synthetic and excluded from the 100 historical tasks.
-[Results and traces](reports/README.md).
+[Results and traces](reports/README.md) · [Reproduce real-model evaluation](docs/MODEL_EVALUATION.md).
 
 ## Quick start
 
@@ -120,11 +122,15 @@ and set `ASSISTANT_TEST_IMAGE` when needed. Historical patch evaluation uses off
 ```bash
 # Explicit local execution for the trusted bundled fixture:
 ASSISTANT_TEST_EXECUTOR=local uv run code-assistant serve
-curl http://127.0.0.1:8000/v1/debug \
+curl --noproxy 127.0.0.1 http://127.0.0.1:8000/v1/debug \
   -H 'Content-Type: application/json' \
   -d '{"repository":"checkout","issue":"Passing discount=0 to calculate_total(100, discount=0) returns 90 instead of 100."}'
 ```
 
+Set `ASSISTANT_MODE=model` when serving with the real-model configuration above.
+The [authenticated real-model HTTP check](reports/real-model-api/validation.json)
+completed search → test → search → synthesis on the trusted synthetic fixture.
+
 Open [interactive API docs](http://127.0.0.1:8000/docs). For other repositories, configure
 `ASSISTANT_WORKSPACE_ROOT` and pass a relative directory. Configure `ASSISTANT_API_TOKEN` and
 send `Authorization: Bearer <token>` before exposing beyond loopback. No wildcard CORS or
@@ -170,8 +176,8 @@ responses contain source excerpts and are ignored by Git.
 index truncation, invalid synthesis, and incomplete traces. [Logging walkthrough](docs/INTERVIEW.md).
 
 ```bash
-uv run ruff check src tests
-uv run ruff format --check src tests
+uv run ruff check src tests scripts
+uv run ruff format --check src tests scripts
 uv run pytest --cov=code_assistant
 uv build
 ```

README.zh-CN.md

@@ -12,17 +12,19 @@
 |---|---|
 | 搜索、测试、综合三个 agent 及条件路由 | 已实现，离线集成测试覆盖 |
 | 仓库搜索、源码读取、pytest 执行、日志读取 | 真实工具，具有调用预算和事件日志 |
-| FastAPI、Bearer 认证、交互式接口文档 | 本地已验证 |
-| Docker 服务、受限测试镜像、Compose | GitHub Actions 构建及运行验证均通过 |
+| FastAPI、Bearer 认证、交互式接口文档 | 真实模型 HTTP 验证通过：无令牌 401，有令牌 200 |
+| Docker 服务及受限测试镜像 | 两种镜像的 GitHub Actions 构建及运行验证均通过 |
 | 12 个仓库的 100 条历史问题 | 固定版本 SWE-bench Lite 子集，可重复选样 |
 | 100 个修复前快照的检索基线 | **全部完成；Hit@1 40%，Hit@5 71%，MRR@5 0.5202** |
-| 真实模型根因正确率 | 接入与人工评分流程已实现；**尚未评测** |
-| 生成补丁的回归测试通过率 | 官方评测导出与评分已实现；**尚未评测** |
+| 100 条真实模型运行 | **90 条带证据诊断，10 条证据不足；文件 Hit@1 87%，Hit@5 88%** |
+| 自动根因核对 | **87 条正确、1 条部分正确、2 条错误、10 条不可评分**；人工准确率未测 |
+| 生成补丁的官方测试 | **48/100 解决**；23 条未通过、27 条空补丁、2 条超时，全部计入分母 |
 
-这些数字衡量“候选文件是否命中维护者补丁修改的文件”，不是修复成功率。
-已通过本地 Codex CLI 登录对前三条历史问题运行真实模型，三个补丁均通过应用检查；
-功能测试由官方评测另行验证。固定演示使用人工构造缺陷，排除在 100 条历史任务之外。
-[真实运行结果与日志](reports/README.md)。
+文件定位指标衡量“候选文件是否命中维护者补丁修改的文件”，与修复成功率分别计算。
+完整模型运行通过本地 Codex CLI 登录调用 `gpt-6.1-sol`，配置和源码指纹已冻结。
+自动根因核对由 LLM 对照维护者补丁进行，存在同模型评审偏差；功能解决率由官方测试
+计算。固定演示使用人工构造缺陷，排除在 100 条历史任务之外。
+[真实运行结果与日志](reports/README.md) · [复现模型评测](docs/MODEL_EVALUATION.md)。
 
 ## 快速运行
 
@@ -116,11 +118,15 @@ uv run code-assistant debug /你的仓库路径 "问题描述" --executor docker
 ```bash
 # 显式允许执行可信的内置演示测试。
 ASSISTANT_TEST_EXECUTOR=local uv run code-assistant serve
-curl http://127.0.0.1:8000/v1/debug \
+curl --noproxy 127.0.0.1 http://127.0.0.1:8000/v1/debug \
   -H 'Content-Type: application/json' \
   -d '{"repository":"checkout","issue":"Passing discount=0 to calculate_total(100, discount=0) returns 90 instead of 100."}'
 ```
 
+使用上面的真实模型配置启动服务时，另设 `ASSISTANT_MODE=model`。
+[真实模型 HTTP 验证](reports/real-model-api/validation.json)已通过认证并完成可信示例的
+“搜索 → 测试 → 再搜索 → 综合”。
+
 打开[交互式接口文档](http://127.0.0.1:8000/docs)。其他仓库通过
 `ASSISTANT_WORKSPACE_ROOT` 指定父目录，请求传相对目录。向回环地址之外开放前应配置
 `ASSISTANT_API_TOKEN`，请求发送 `Authorization: Bearer <token>`。服务限制路径范围，
@@ -165,8 +171,8 @@ token 用量。事件日志不记录提示词、源码正文和隐藏推理。
 及未完成运行。[面试讲解](docs/INTERVIEW.md)。
 
 ```bash
-uv run ruff check src tests
-uv run ruff format --check src tests
+uv run ruff check src tests scripts
+uv run ruff format --check src tests scripts
 uv run pytest --cov=code_assistant
 uv build
 ```

docs/ARCHITECTURE.md

@@ -40,6 +40,21 @@ binary data, and files larger than 500 KB. It caps 20,000 files / 80 MB and logs
 Each evidence record has an immutable ID for that run. Source reads and search snippets have
 distinct kinds: a search snippet alone cannot satisfy the final source-read requirement.
 
+## Local-login model adapter
+
+[`codex_model.py`](../src/code_assistant/codex_model.py) implements LangChain's chat-model
+interface over non-interactive Codex CLI inference. Application tool schemas and messages
+travel over stdin; schema-validated JSON becomes `AIMessage.tool_calls` or a structured
+diagnosis. Authentication stays with the CLI's supported login.
+
+Each call runs in an empty temporary directory with user configuration/rules ignored and
+built-in execution, file operations, web search, connectors, plugins, and extra agents
+disabled. CLI events are checked and unexpected built-in activity fails the call. Only the
+application tools have the actor repository path. Time/output limits and safe error handling
+bound the subprocess. The model still receives the selected source excerpts in its input.
+The pinned run records CLI version, model identifier, reasoning effort, transport, usage,
+and reconnect counts. [Reproduction](MODEL_EVALUATION.md).
+
 ## Output and verification
 
 Synthesis produces a Pydantic `Diagnosis`: conclusion, causal explanation, affected files,
@@ -52,6 +67,18 @@ Confidence is model-reported, not a calibrated probability. Semantic review and
 regression tests are required for correctness claims. The service never applies patches;
 `patch_verified` remains false. Project tests separately verify the synthetic example's patch.
 
+[`patches.py`](../src/code_assistant/patches.py) separately validates diff paths against
+affected files that were actually read, mechanically recounts hunk lengths, and checks
+applicability on a temporary source copy. Invalid candidates are withheld with their original
+hashes preserved. It does not run regression tests. After inference freezes, official
+SWE-bench containers evaluate the submitted patch and retain separate functional results.
+
+Each historical issue has an isolated snapshot and trace. Concurrent workers share no actor
+state; the main thread persists complete predictions and restores task order at completion.
+Resumption rejects task, model, source, environment, budget, and worker-count mismatches.
+Four frozen batches preserve the full 100-task denominator and patch identities during
+official evaluation. Automated gold-based review occurs only after each prediction freezes.
+
 ## Execution and service boundaries
 
 Local execution is disabled by default. Opting into local pytest executes repository code

docs/BENCHMARK.md

@@ -2,12 +2,15 @@
 
 ## Decision and scope
 
-The future model experiment asks whether the assistant identifies the causal mechanism in
-real bug reports, as confirmed by merged maintainer fixes. Its primary diagnosis metric is
-**human-adjudicated root-cause accuracy**. Functional patch resolution is measured separately
-by the official SWE-bench harness. Both are currently unmeasured.
-
-The completed experiment is a **deterministic file-localization baseline**. It checks data
+The real-model experiment asks whether the assistant identifies the causal mechanism in
+real bug reports, as confirmed by merged maintainer fixes. The 100-task run is complete:
+90 cited diagnoses, 10 abstentions, Hit@1 87%, Hit@5 88%. An explicitly automated causal
+comparison scores 87 correct, 1 partial, 2 incorrect, and 10 unscorable. **Human-adjudicated
+root-cause accuracy remains unmeasured.** Functional resolution is measured separately by
+the pinned official harness: **48/100 resolved**, 23 unresolved, 27 empty patches,
+2 test timeouts (600 seconds). See [published results](../reports/README.md).
+
+The earlier completed experiment is a **deterministic file-localization baseline**. It checks data
 preparation, snapshot isolation, search/read tools, scoring, and per-issue telemetry. It does
 not test LLM reasoning or establish that multi-agent orchestration improves outcomes.
 
@@ -79,8 +82,8 @@ raw durations, trace conservation checks, and manifests are published in [report
 ## Model evaluation and causal rubric
 
 First run three pilot issues for environment/protocol debugging, then freeze the model,
-prompts, tools, budgets, and code before the 100-issue run. A full run requires provider
-credentials and incurs that provider's usage charges. Record actual token usage when supplied;
+prompts, tools, budgets, and code before the 100-issue run. A full run requires a provider
+key or supported local CLI login and consumes that account's allowance. Record token usage when supplied;
 unknown usage stays null. Do not alter the task set after viewing results.
 
 ```bash
@@ -116,6 +119,21 @@ benefits from the retrieval baseline versus an LLM treatment.
 
 ## Official patch evaluation
 
+The runnable evaluation path is [MODEL_EVALUATION.md](MODEL_EVALUATION.md), pinned to
+`swebench==5.0.2`. The 100 inference outputs are also frozen in four consecutive 25-task
+batches so Docker tests can overlap later inference. Batch selection follows the original
+task order and includes abstentions and invalid patches; no outcome-based filtering is used.
+Every nonempty evaluated patch has a SHA-256 checked against the final predictions. The
+four batch denominators sum to 100. The original dataset checksum and actual image digests
+are retained. The full run resolved 48/100; 23 nonempty patches failed tests, 27 predictions
+had empty patches, and 2 tests timed out. There are no missing outcomes or image-pull failures.
+The three-task pilot separately resolved all three cases; it does not enlarge the denominator.
+
+Two post-hoc maintainer-patch controls also passed official required regressions with the
+same image digests, despite incidental fixture/import errors in the larger test output.
+Those errors alone therefore do not justify excluding the original model failures.
+[Control evidence](../reports/evaluation-controls/README.md). The agent score remains 48/100.
+
 Use a separate environment with a recorded, pinned SWE-bench harness version and working Docker.
 Export the **same pinned source records** for official evaluation, instead of silently loading
 whatever the upstream `main` revision becomes:
@@ -157,12 +175,17 @@ and selected dataset hash before making a patch-correctness claim.
 
 ## 中文说明
 
-已完成的是 100 条真实历史问题的**文件定位基线**，不是 100 条大模型根因诊断或补丁修复。
-固定数据版本、2026 种子和按仓库轮转的选样规则均可核验，12 个仓库全部有覆盖。
+已完成 100 条历史问题的检索基线和真实模型运行。真实模型给出 90 条带证据诊断，
+10 条主动判断证据不足；文件 Hit@1 为 87%，Hit@5 为 88%。固定数据版本、2026 种子和
+按仓库轮转的选样规则均可核验，12 个仓库全部有覆盖。
 
 agent 只能访问修复前快照，参考补丁和隐藏测试放在外部；但公开历史问题可能存在训练
 污染，因此不能宣称完全不存在泄漏。文件命中率与因果正确性严格区分。
 
 人工根因评分要求说明触发条件、错误机制及责任代码，并与维护者补丁一致；相关文件命中
 但解释不全只能算部分正确。每条需要复核人和理由，全部复核后才生成根因正确率。
-功能修复率必须由官方 SWE-bench 环境运行回归测试确认。当前两者都没有结果，不填写数字。
+自动评审已对照维护者补丁产生 87 条正确、1 条部分正确、2 条错误及 10 条不可评分；
+这不是人工准确率。官方 SWE-bench 环境确认 **48/100** 修复；23 条未通过、27 条空补丁、
+2 条超时均保留在分母中，逐条报告、日志摘录与哈希已发布。100 条推理阶段关闭历史
+仓库测试执行，专职测试 agent 已在真实模型的可信
+示例中实际运行；官方隐藏测试仅在预测冻结后执行。