Agentic Code-Search & Debugging Assistant

English · 简体中文

An evidence-first developer assistant built with LangChain, LangGraph, FastAPI, and Docker. The measured historical configuration is a code-search → synthesis pipeline with registered source citations. A test specialist and failure-to-search route are available when execution is explicitly enabled for trusted repositories. Patches are proposed without changing the repository.

The frozen 48/100 result did not exercise the test agent. All 100 traces used search → synthesis; testing happened afterward in official SWE-bench containers. The tiny checkout fixture checks plumbing and cannot establish real-world multi-agent benefit.

Implementation and evidence

Component	Status
Search / test / synthesis agents and conditional routing	Implemented and covered by offline integration tests
Search, source reads, pytest execution, log inspection	Real tools with budgets and structured events
FastAPI, bearer authentication, OpenAPI docs	Real-model HTTP check passed: 401 without token, 200 with token
Docker service and restricted runner	Both image builds and execution checks passed in GitHub Actions
100 historical issues across 12 repositories	Pinned, reproducible SWE-bench Lite subset
Retrieval baseline on all 100 pre-fix snapshots	100 completed; Hit@1 40%; Hit@5 71%; MRR@5 0.5202
Real LLM run on all 100 issues	90 cited diagnoses, 10 abstentions; file Hit@1 87%, Hit@5 88%
Generated-patch resolution	48/100 resolved in official SWE-bench tests; 23 failed, 27 empty, 2 timed out
Selected post-hoc patch repair	17/17 applicable; 11/17 officially resolved after bounded correction; separate from original 48/100
Matched 20-task controls	Single 11/20; both routed windows 10/20 with equal model/tool call ceilings; protocol and results

The single arm used fewer calls and reported tokens; this pilot does not show a routed advantage. The two window policies each resolved the same 10/20 tasks. Source priority retains more source but has not demonstrated higher resolution. Repair results use additional inference on previously invalid candidates and are not a new pass@1 score.

Baseline numbers measure file localization against maintainer patches, not bug-fix accuracy. The real run uses gpt-6.1-sol through a locally authenticated Codex CLI, with a frozen configuration and source fingerprint. BM25 is a retrieval reference, not a matched LLM control. Automated root-cause judgments are retained for auditing, excluded from headline performance: 40 explanations labeled correct did not resolve, while one labeled incorrect did resolve. Named human causal accuracy remains unmeasured. Functional resolution comes from official tests. The repository-balanced 100-task subset differs from Lite's full distribution; 48% is not comparable to its leaderboard. Public historical fixes may occur in model training data. The scripted demo is synthetic and excluded from the 100 historical tasks. Results and traces · Reproduce real-model evaluation.

Quick start

Requirements: Python 3.11+, uv, Git. The demo needs no model credentials or Docker.

git clone https://github.com/YeDonS/agentic-code-search.git
cd agentic-code-search
uv sync --locked --all-extras
uv run code-assistant demo

The demo investigates calculate_total(100, discount=0), which incorrectly returns 90. It searches source, reads the function, runs actual tests (1 failed, 2 passed), reads the failure log, refines the search, and returns a cited diagnosis and unified diff. mode=demo is a scripted LangChain fixture, not an LLM; it accepts only this exact scenario. A regression test applies the suggestion to a temporary copy and confirms 3 passed. The assistant itself runs baseline tests only, so its response retains patch_verified=false.

Routing and tools

flowchart TD
    I[Bug report and repository] --> R{State-based router}
    R -->|First pass or missing source| S[Code-search agent]
    S -->|Search and source reads| R
    R -->|Source found and execution enabled| T[Test-runner agent]
    T -->|Focused test and log inspection| R
    R -->|Failure with search budget remaining| S
    R -->|Evidence ready or budget reached| Y[Synthesis agent]
    Y --> V[Citation validation]
    V --> O[Diagnosis or insufficient evidence]

Loading

The supervisor uses deterministic rules. Models choose tool calls within each role. The three agents have separate prompts, tool lists, and contexts, can share one model, and run sequentially.

Agent	Tools
Code-search	search_repository, read_file
Test-runner	read_file, run_tests, inspect_logs
Synthesis / bounded patch repair	read_file; validated JSON diagnosis; no execution tools

The default handoff retains up to 24 distinct observations, prioritizing source reads over tests/logs over search snippets. The full citation ledger is preserved. Applicability errors can trigger two format-repair attempts within the shared 18-model-call / 18-tool-call ceilings. All attempts and original candidates are retained. New/deleted text files are supported; renames, binaries and symlinks are rejected. ASSISTANT_WORKFLOW=single uses one investigation conversation; set ASSISTANT_PATCH_REPAIR_ATTEMPTS=0 for a pure single-conversation control.

Existing affected files must occur in registered source-read citations; declared new files still require a genuine existing-source citation. This verifies provenance, not semantic correctness. Missing evidence or model abstention returns insufficient_evidence. Search passes, model turns, tool calls, reads, test duration, and output size are capped. Architecture and limitations.

Real model and repository

For independent inference and service deployments, configure a provider API key. OpenAI and Anthropic adapters use LangChain directly; choose a tool-capable model available to your account:

cp .env.example .env
# Set ASSISTANT_MODE=model, ASSISTANT_MODEL=provider:model-id,
# and ANTHROPIC_API_KEY or OPENAI_API_KEY in .env.
uv run code-assistant doctor
uv run code-assistant debug /path/to/repository "Observed bug and expected behavior"

doctor checks configuration, not inference. No provider API key was available for a live API-provider run here; adapter configuration is tested offline. Changing provider/model creates a new experiment and cannot reproduce the published model's score.

An optional local Codex CLI adapter was used for the published research runs (CLI 0.162.0-alpha.2, account-specific model alias):

codex login
export ASSISTANT_MODEL=codex-cli:gpt-6.1-sol  # select a model available to your account
export ASSISTANT_CODEX_REASONING_EFFORT=medium
export ASSISTANT_CODEX_TRANSPORT=https       # optional, useful on this evaluation network
uv run code-assistant doctor
uv run code-assistant debug /path/to/repository "Observed bug and expected behavior"

This consumes your Codex account allowance. The adapter uses an empty temporary directory, disables built-in execution/search/connectors, and exchanges application tool calls through structured JSON. Only the application's allowlisted tools see the repository. Authentication stays with the CLI; never copy its login files into public CI or Docker images.

CLI flags and model availability can change. Verify account terms and supported use before using subscription authentication in a service; this repository makes no eligibility or terms-compliance claim. Frozen-patch replay is independent of CLI access.

debug always uses model mode. Choose a model available to your account that supports tools. The model name in .env.example is configurable. Keys load from environment variables or .env. Source excerpts are sent to the configured provider. Test execution defaults to disabled; opt in with --executor local only for trusted code, or use the host's Docker runner:

docker build --target test-runner -t code-assistant-test:local .
uv run code-assistant debug /path/to/repository "Bug description" --executor docker

The generic runner includes pytest, not arbitrary project dependencies. Build a suitable image and set ASSISTANT_TEST_IMAGE when needed. Historical patch evaluation uses official SWE-bench images.

FastAPI and Docker

# Explicit local execution for the trusted bundled fixture:
ASSISTANT_TEST_EXECUTOR=local uv run code-assistant serve
curl --noproxy 127.0.0.1 http://127.0.0.1:8000/v1/debug \
  -H 'Content-Type: application/json' \
  -d '{"repository":"checkout","issue":"Passing discount=0 to calculate_total(100, discount=0) returns 90 instead of 100."}'

Set ASSISTANT_MODE=model when serving with the real-model configuration above. The authenticated real-model HTTP check completed search → test → search → synthesis on the trusted synthetic fixture.

Open interactive API docs. For other repositories, configure ASSISTANT_WORKSPACE_ROOT and pass a relative directory. Configure ASSISTANT_API_TOKEN and send Authorization: Bearer <token> before exposing beyond loopback. No wildcard CORS or arbitrary shell-command endpoint is provided. At most two API runs execute concurrently.

docker compose up --build
docker compose run --rm assistant code-assistant demo

Compose binds loopback, runs as non-root with a read-only filesystem, persists artifacts in a named volume, and disables API test execution by default. No Docker socket is mounted. For real repositories, add a read-only bind mount and configure its container workspace path.

Historical benchmark

uv run code-assistant benchmark build  # optional: rebuild the checked-in, pinned task set
uv run code-assistant benchmark run runs/my-retrieval-100 --mode retrieval
uv run code-assistant logs runs/my-retrieval-100
# Requires provider credentials; pilot before scaling:
uv run code-assistant benchmark run runs/model-pilot --mode model --limit 3
uv run code-assistant benchmark run runs/model-100 --mode model --workers 6
# Resume only the exact same source/environment/model/task configuration:
uv run code-assistant benchmark run runs/model-100 --mode model --workers 6 --resume
uv run code-assistant benchmark export runs/model-100/predictions.jsonl runs/model-100/swebench.jsonl

Actors receive issue text and a source archive at base_commit, with no Git history, gold patch, hidden test patch, or scoring labels. Scoring runs afterward. Failures remain in the denominator. Localization, human causal review, and official patch tests are separate metrics. Selection, review rubric, leakage risks, and official harness commands.

Logs and checks

Runs write runs/<id>/events.jsonl and response.json. Benchmarks add manifests, snapshot hashes, predictions, scores, and per-issue traces. Events capture routing reasons, roles, tools, outcomes, duration, evidence counts, exit codes, and provider-reported token usage when available. Event logs omit prompts, source text, and hidden reasoning. Detailed local responses contain source excerpts and are ignored by Git.

code-assistant logs <directory> identifies empty searches, tool/environment errors, timeouts, index truncation, invalid synthesis, and incomplete traces. Trace audit.

uv run ruff check src tests scripts
uv run ruff format --check src tests scripts
uv run pytest --cov=code_assistant
uv build

CI tests Python 3.11/3.12/3.13 and checks both Docker images. Deliberately failing demo tests run in a subprocess and are excluded from the project's normal test collection.

src/code_assistant/ contains implementation; examples/ the synthetic demo; benchmarks/ the pinned tasks and separate labels; tests/ behavioral tests; docs/ architecture and reproduction guides; reports/ curated public evidence. Project code is MIT; upstream historical data and source remain attributable to their authors. Notices.