Repair patch generation and add honest, matched evaluation controls

.env.example

@@ -18,3 +18,10 @@ ASSISTANT_MODEL=anthropic:claude-sonnet-4-5-20250929
 # Docker execution is available from the host CLI; do not mount a Docker socket into the API.
 # ASSISTANT_TEST_EXECUTOR=docker
 ASSISTANT_TEST_IMAGE=code-assistant-test:local
+# Optional evidence/format-repair controls (new runs only; never resume old settings):
+ASSISTANT_EVIDENCE_POLICY=source_priority
+ASSISTANT_PATCH_REPAIR_ATTEMPTS=2
+ASSISTANT_MAX_MODEL_CALLS=18
+ASSISTANT_MAX_AGENT_STEPS=7
+ASSISTANT_MAX_TOOL_CALLS=18
+# ASSISTANT_WORKFLOW=single  # use ASSISTANT_PATCH_REPAIR_ATTEMPTS=0 for a pure baseline

README.md

@@ -4,8 +4,14 @@
 [English](README.md) · [简体中文](README.zh-CN.md)
 
 An evidence-first developer assistant built with **LangChain, LangGraph, FastAPI, and Docker**.
-Three specialist agents locate code, reproduce failures, and synthesize a diagnosis with
-registered source citations. Patches are proposed without changing the repository.
+The measured historical configuration is a **code-search → synthesis pipeline** with
+registered source citations. A test specialist and failure-to-search route are available
+when execution is explicitly enabled for trusted repositories. Patches are proposed without
+changing the repository.
+
+**The frozen 48/100 result did not exercise the test agent.** All 100 traces used search →
+synthesis; testing happened afterward in official SWE-bench containers. The tiny checkout
+fixture checks plumbing and cannot establish real-world multi-agent benefit.
 
 ## Implementation and evidence
 
@@ -18,13 +24,23 @@ registered source citations. Patches are proposed without changing the repositor
 | 100 historical issues across 12 repositories | Pinned, reproducible SWE-bench Lite subset |
 | Retrieval baseline on all 100 pre-fix snapshots | **100 completed; Hit@1 40%; Hit@5 71%; MRR@5 0.5202** |
 | Real LLM run on all 100 issues | **90 cited diagnoses, 10 abstentions; file Hit@1 87%, Hit@5 88%** |
-| Automated root-cause comparison | **87 correct, 1 partial, 2 incorrect, 10 unscorable**; human accuracy remains unmeasured |
 | Generated-patch resolution | **48/100 resolved** in official SWE-bench tests; 23 failed, 27 empty, 2 timed out |
+| Selected post-hoc patch repair | **17/17 applicable; 11/17 officially resolved** after bounded correction; separate from original 48/100 |
+| Matched 20-task controls | **Single 11/20; both routed windows 10/20** with equal model/tool call ceilings; [protocol and results](docs/EXPERIMENTS.md) |
+
+The single arm used fewer calls and reported tokens; this pilot does not show a routed
+advantage. The two window policies each resolved the same 10/20 tasks. Source
+priority retains more source but has not demonstrated higher resolution. Repair results
+use additional inference on previously invalid candidates and are not a new pass@1 score.
 
 Baseline numbers measure **file localization against maintainer patches**, not bug-fix accuracy.
 The real run uses `gpt-6.1-sol` through a locally authenticated Codex CLI, with a frozen
-configuration and source fingerprint. Automated causal comparison is a labeled LLM judgment
-with possible same-model bias. Functional resolution comes from the official SWE-bench harness.
+configuration and source fingerprint. BM25 is a retrieval reference, **not a matched LLM control**.
+Automated root-cause judgments are retained for auditing, excluded from headline performance:
+40 explanations labeled correct did not resolve, while one labeled incorrect did resolve.
+Named human causal accuracy remains unmeasured. Functional resolution comes from official tests.
+The repository-balanced 100-task subset differs from Lite's full distribution; **48% is not
+comparable to its leaderboard**. Public historical fixes may occur in model training data.
 The scripted demo is synthetic and excluded from the 100 historical tasks.
 [Results and traces](reports/README.md) · [Reproduce real-model evaluation](docs/MODEL_EVALUATION.md).
 
@@ -69,17 +85,40 @@ agents have separate prompts, tool lists, and contexts, can share one model, and
 |---|---|
 | Code-search | `search_repository`, `read_file` |
 | Test-runner | `read_file`, `run_tests`, `inspect_logs` |
-| Synthesis | No execution tools; validated JSON diagnosis |
+| Synthesis / bounded patch repair | `read_file`; validated JSON diagnosis; no execution tools |
+
+The default handoff retains up to 24 distinct observations, prioritizing source reads over
+tests/logs over search snippets. The full citation ledger is preserved. Applicability errors
+can trigger two format-repair attempts within the shared 18-model-call / 18-tool-call ceilings.
+All attempts and original candidates are retained. New/deleted text files are supported;
+renames, binaries and symlinks are rejected. `ASSISTANT_WORKFLOW=single` uses one investigation
+conversation; set `ASSISTANT_PATCH_REPAIR_ATTEMPTS=0` for a pure single-conversation control.
 
-Affected files must occur in registered source-read citations. This verifies provenance, not
+Existing affected files must occur in registered source-read citations; declared new files
+still require a genuine existing-source citation. This verifies provenance, not
 semantic correctness. Missing evidence or model abstention returns `insufficient_evidence`.
 Search passes, model turns, tool calls, reads, test duration, and output size are capped.
 [Architecture and limitations](docs/ARCHITECTURE.md).
 
 ## Real model and repository
 
-You can use a local [Codex CLI login](https://learn.chatgpt.com/docs/non-interactive-mode)
-instead of configuring a separate provider API key (tested with CLI 0.162.0-alpha.2):
+For independent inference and service deployments, configure a provider API key. OpenAI and
+Anthropic adapters use LangChain directly; choose a tool-capable model available to your account:
+
+```bash
+cp .env.example .env
+# Set ASSISTANT_MODE=model, ASSISTANT_MODEL=provider:model-id,
+# and ANTHROPIC_API_KEY or OPENAI_API_KEY in .env.
+uv run code-assistant doctor
+uv run code-assistant debug /path/to/repository "Observed bug and expected behavior"
+```
+
+`doctor` checks configuration, not inference. No provider API key was available for a live
+API-provider run here; adapter configuration is tested offline. Changing provider/model
+creates a new experiment and cannot reproduce the published model's score.
+
+An optional local [Codex CLI adapter](https://learn.chatgpt.com/docs/non-interactive-mode)
+was used for the published research runs (CLI 0.162.0-alpha.2, account-specific model alias):
 
 ```bash
 codex login
@@ -95,14 +134,9 @@ disables built-in execution/search/connectors, and exchanges application tool ca
 structured JSON. Only the application's allowlisted tools see the repository. Authentication
 stays with the CLI; never copy its login files into public CI or Docker images.
 
-Alternatively, use a provider API key:
-
-```bash
-cp .env.example .env
-# Edit ASSISTANT_MODE=model, ASSISTANT_MODEL=provider:model-id,
-# and ANTHROPIC_API_KEY or OPENAI_API_KEY in .env.
-uv run code-assistant debug /path/to/repository "Observed bug and expected behavior"
-```
+CLI flags and model availability can change. Verify account terms and supported use before
+using subscription authentication in a service; this repository makes no eligibility or
+terms-compliance claim. Frozen-patch replay is independent of CLI access.
 
 `debug` always uses model mode. Choose a model available to your account that supports tools.
 The model name in `.env.example` is configurable. Keys load from environment variables or `.env`.

README.zh-CN.md

@@ -3,8 +3,13 @@
 [![CI](https://github.com/YeDonS/agentic-code-search/actions/workflows/ci.yml/badge.svg)](https://github.com/YeDonS/agentic-code-search/actions/workflows/ci.yml)
 [English](README.md) · [简体中文](README.zh-CN.md)
 
-基于 **LangChain、LangGraph、FastAPI 和 Docker** 的开发助手。三个专职 agent
-分别定位代码、复现失败、综合诊断，输出附带源码引用和修复建议，不直接修改仓库。
+基于 **LangChain、LangGraph、FastAPI 和 Docker** 的开发助手。历史成绩来自
+**代码搜索 → 综合诊断** 两步流程，输出带源码引用的诊断与补丁，不直接修改仓库。
+对显式允许执行的可信仓库，还提供测试 agent 及“测试失败后回到搜索”的路由。
+
+**原始 48/100 没有测试 agent 参与推理。** 100 条 trace 全部是搜索 → 综合，官方
+容器测试在补丁冻结后单独执行。小型 checkout 示例只能验证流程连通，不能证明真实
+项目上的多 agent 优势。
 
 ## 实现与验证状态
 
@@ -17,13 +22,21 @@
 | 12 个仓库的 100 条历史问题 | 固定版本 SWE-bench Lite 子集，可重复选样 |
 | 100 个修复前快照的检索基线 | **全部完成；Hit@1 40%，Hit@5 71%，MRR@5 0.5202** |
 | 100 条真实模型运行 | **90 条带证据诊断，10 条证据不足；文件 Hit@1 87%，Hit@5 88%** |
-| 自动根因核对 | **87 条正确、1 条部分正确、2 条错误、10 条不可评分**；人工准确率未测 |
 | 生成补丁的官方测试 | **48/100 解决**；23 条未通过、27 条空补丁、2 条超时，全部计入分母 |
+| 原无效补丁的事后补救 | **17/17 可应用、11/17 官方解决**；有限纠错，独立于原始 48/100 |
+| 20 题配对对照 | **单 agent 11/20，两种两步流程均 10/20**；同模型、相同调用上限，[协议与结果](docs/EXPERIMENTS.md) |
+
+单 agent 本次实际调用和报告 token 更少，小样本未支持路由流程的优势。两种证据窗口
+均解决同一组 10/20 题。源码优先确实减少源码丢失，但尚未
+观察到修复率提升。17 题补救使用了额外推理预算，不是新的首次成功率。
 
 文件定位指标衡量“候选文件是否命中维护者补丁修改的文件”，与修复成功率分别计算。
 完整模型运行通过本地 Codex CLI 登录调用 `gpt-6.1-sol`，配置和源码指纹已冻结。
-自动根因核对由 LLM 对照维护者补丁进行，存在同模型评审偏差；功能解决率由官方测试
-计算。固定演示使用人工构造缺陷，排除在 100 条历史任务之外。
+BM25 仅作检索参考，不能充当同条件的 LLM 对照。自动根因评审保留作审计，不列为主成绩：
+被判正确的解释有 40 条没有解决，另有 1 条被判错误的补丁实际通过了官方测试。
+人工根因准确率仍未测。功能解决率依据官方测试。100 条按仓库均衡抽取，与 Lite 完整
+分布不同，**48% 不与其排行榜横比**；公开历史修复也可能已进入模型训练数据。
+固定演示排除在历史任务之外。
 [真实运行结果与日志](reports/README.md) · [复现模型评测](docs/MODEL_EVALUATION.md)。
 
 ## 快速运行
@@ -68,16 +81,38 @@ flowchart TD
 |---|---|
 | 代码搜索 | `search_repository`、`read_file` |
 | 测试 | `read_file`、`run_tests`、`inspect_logs` |
-| 综合 | 无执行工具，输出经校验的 JSON 诊断 |
+| 综合 / 有限补丁纠错 | `read_file`；输出经校验的 JSON，无执行工具 |
+
+默认窗口保留最多 24 条不同观察，优先源码读取，其次测试/日志，最后搜索片段；完整
+证据账本不丢弃。应用检查失败后最多纠错两次，共享每题 18 次模型、18 次工具调用上限，
+保留原始候选及每次尝试。支持文本文件新建/删除，拒绝重命名、二进制和符号链接。
+设置 `ASSISTANT_WORKFLOW=single` 可使用单一调查对话；纯单对话对照还需设置
+`ASSISTANT_PATCH_REPAIR_ATTEMPTS=0`，禁用后续独立纠错。
 
-受影响文件必须出现在真实登记的源码读取引用中。这能验证出处，不能自动证明因果解释
+已存在的受影响文件必须有登记的源码读取引用；声明新建文件也须引用真实的已有源码。
+这能验证出处，不能自动证明因果解释
 正确。证据不足或模型主动放弃时返回 `insufficient_evidence`。检索轮数、模型轮数、
 工具次数、读取范围、测试时长及输出大小均有限制。[架构及限制](docs/ARCHITECTURE.md)。
 
 ## 接入真实模型与仓库
 
-也可以复用本地 [Codex CLI 登录](https://learn.chatgpt.com/docs/non-interactive-mode)，
-无需单独导出 API Key（已验证 CLI 0.162.0-alpha.2）：
+独立推理及部署服务优先使用模型 API Key，经 LangChain 直接调用 OpenAI 或 Anthropic，
+选择自己账号可用、支持工具调用的模型：
+
+```bash
+cp .env.example .env
+# 配置 ASSISTANT_MODE=model、ASSISTANT_MODEL=provider:model-id，
+# 以及对应的 ANTHROPIC_API_KEY 或 OPENAI_API_KEY。
+uv run code-assistant doctor
+uv run code-assistant debug /你的仓库路径 "描述实际行为、预期行为及复现条件"
+```
+
+`doctor` 仅检查配置，不代表推理通过。本次环境没有独立模型 API Key，原生 API
+适配器做了离线配置测试，尚无真实 API 调用验证。更换模型/提供商属于新实验，不能据此
+复现已发布模型的同一成绩。
+
+另提供可选的本地 [Codex CLI 适配器](https://learn.chatgpt.com/docs/non-interactive-mode)。
+历史实验使用 CLI 0.162.0-alpha.2 和账号特定的模型别名：
 
 ```bash
 codex login
@@ -92,14 +127,8 @@ uv run code-assistant debug /你的仓库路径 "描述实际行为、预期行
 通过结构化 JSON 请求项目工具；仓库只向项目的白名单工具开放。登录凭证由 CLI 自行
 管理，不复制到公开 CI 或 Docker 镜像。
 
-也可以使用独立模型 API Key：
-
-```bash
-cp .env.example .env
-# 在 .env 中配置 ASSISTANT_MODE=model、ASSISTANT_MODEL=provider:model-id，
-# 以及对应的 ANTHROPIC_API_KEY 或 OPENAI_API_KEY。
-uv run code-assistant debug /你的仓库路径 "描述实际行为、预期行为及复现条件"
-```
+CLI 参数和模型可用性可能变化。将订阅登录用于服务前，应核对适用的账户条款和支持范围；
+本仓库不作合规或资格承诺。冻结补丁的官方复跑不依赖 CLI 登录或模型别名。
 
 CLI 的 `debug` 始终使用真实模型。请选择账号可用、支持工具调用的模型；配置示例中的
 模型名称可修改。密钥从环境变量或 `.env` 读取，相关源码片段会发送给配置的模型服务。

docs/ARCHITECTURE.md

@@ -19,6 +19,13 @@ and tool list. Evidence and short findings cross role boundaries. This is sequen
 orchestration, not parallel independent model processes. The supervisor's rules are deliberate:
 they are cheap to audit and guarantee an evidence-gathering opportunity before synthesis.
 
+**Measured scope:** the original 100 historical tasks disabled test execution. Every trace
+was code_search → synthesis; no test specialist or failure-to-search loop participated.
+The full route was exercised only on the tiny synthetic checkout fixture. Official patch
+testing runs after freezing and is not an inference-time agent call. A `single` workflow
+provides a continuous-conversation control; [paired experiments](EXPERIMENTS.md) use equal
+model-call and tool-call ceilings and report actual tokens separately.
+
 ## Tool protocol
 
 [`agents.py`](../src/code_assistant/agents.py) uses LangChain `BaseChatModel.bind_tools`,
@@ -62,15 +69,23 @@ evidence IDs, proposed change, optional unified diff, confidence, and limitation
 or affected files absent from cited reads are rejected. The response includes the full route
 trace, real baseline test status, mode, and run ID.
 
+Synthesis has `read_file` to verify exact context before writing a diff. Existing affected
+files require cited source reads; explicitly declared new-file headers are allowed with at
+least one genuine source citation. The full ledger is retained. A bounded handoff defaults
+to 24 distinct records with source reads first, tests/logs next, and search snippets last.
+
 Citation checks establish provenance only. A model can still misunderstand genuine evidence.
 Confidence is model-reported, not a calibrated probability. Semantic review and official
 regression tests are required for correctness claims. The service never applies patches;
 `patch_verified` remains false. Project tests separately verify the synthetic example's patch.
 
 [`patches.py`](../src/code_assistant/patches.py) separately validates diff paths against
 affected files that were actually read, mechanically recounts hunk lengths, and checks
-applicability on a temporary source copy. Invalid candidates are withheld with their original
-hashes preserved. It does not run regression tests. After inference freezes, official
+applicability on a temporary source copy. Text-file additions/deletions are supported;
+renames, linked paths, hidden paths, binaries and special modes are rejected. Up to two
+pre-evaluation repair requests feed back the apply error, share the global model/tool budget,
+and preserve every attempt. Invalid final candidates are withheld with original hashes.
+It does not run regression tests. After inference freezes, official
 SWE-bench containers evaluate the submitted patch and retain separate functional results.
 
 Each historical issue has an isolated snapshot and trace. Concurrent workers share no actor

docs/BENCHMARK.md

@@ -4,9 +4,9 @@
 
 The real-model experiment asks whether the assistant identifies the causal mechanism in
 real bug reports, as confirmed by merged maintainer fixes. The 100-task run is complete:
-90 cited diagnoses, 10 abstentions, Hit@1 87%, Hit@5 88%. An explicitly automated causal
-comparison scores 87 correct, 1 partial, 2 incorrect, and 10 unscorable. **Human-adjudicated
-root-cause accuracy remains unmeasured.** Functional resolution is measured separately by
+90 cited diagnoses, 10 abstentions, Hit@1 87%, Hit@5 88%. **Human-adjudicated
+root-cause accuracy remains unmeasured.** Automated judgments are audit data, not a primary
+performance metric. Functional resolution is measured separately by
 the pinned official harness: **48/100 resolved**, 23 unresolved, 27 empty patches,
 2 test timeouts (600 seconds). See [published results](../reports/README.md).
 
@@ -28,6 +28,31 @@ not test LLM reasoning or establish that multi-agent orchestration improves outc
 - `instance_id` ends in a **solution PR number**, not necessarily the original issue number.
   `maintainer_fix_url` correctly links to `/pull/<number>`.
 
+The pinned data's exact repository allocation is below. Small repositories are fully included
+while Django and SymPy are heavily subsampled. Thus **48/100 cannot be compared directly
+with SWE-bench Lite leaderboard percentages**, nor corrected into one using naive reweighting.
+
+| Repository | Selected | Pinned Lite total |
+|---|---:|---:|
+| astropy/astropy | 6 | 6 |
+| django/django | 12 | 114 |
+| matplotlib/matplotlib | 12 | 23 |
+| mwaskom/seaborn | 4 | 4 |
+| pallets/flask | 3 | 3 |
+| psf/requests | 6 | 6 |
+| pydata/xarray | 5 | 5 |
+| pylint-dev/pylint | 6 | 6 |
+| pytest-dev/pytest | 12 | 17 |
+| scikit-learn/scikit-learn | 12 | 23 |
+| sphinx-doc/sphinx | 11 | 16 |
+| sympy/sympy | 11 | 77 |
+| Total | 100 | 300 |
+
+Counts are computed from our pinned parquet, rather than inferred from a current website.
+Public historical fixes can have entered later model training; no contamination-free claim
+is made. The [official Lite overview](https://www.swebench.com/lite.html) describes the complete
+benchmark. [New matched pilot and separate repair study](EXPERIMENTS.md).
+
 [`tasks.jsonl`](../benchmarks/tasks.jsonl) contains actor inputs. Separate
 [`gold/reference.jsonl`](../benchmarks/gold/reference.jsonl) contains changed-file labels,
 test IDs, and patch hashes. Full patches remain in the upstream cached parquet and can be