DarwinPatch

A budgeted repair-search controller for reliable coding agents.

DarwinPatch is a runnable technical demo for a practical agent-reliability problem: a coding agent should not merely generate a patch and retry when it fails. It should verify every candidate through hard gates, convert failures into bounded evidence, route the next attempt deliberately, and leave behind an audit trail explaining why each candidate was rejected or promoted.

DarwinPatch is not a replacement for a coding agent. It is the reliability layer around candidate patches.

candidate patch
  -> hard gates
  -> bounded EvidencePacket on failure
  -> route-aware next candidate selection
  -> candidate archive + lineage + fingerprints
  -> benchmark and static HTML report

Why this matters

Modern coding agents can produce impressive patches, but long-horizon repair loops are often too opaque:

• visible tests can pass while hidden behavior regresses,
• retries can repeat the same failure mode,
• patches can touch unsafe or irrelevant files,
• raw logs can leak too much context into the next attempt,
• users are left reconstructing what happened after the fact.

DarwinPatch demonstrates a controlled alternative: verifier-grounded self-correction with bounded evidence and auditable promotion.

Current controlled result

The default benchmark is deterministic and offline: 40 repair cases across four small Python task families, with curated candidate pools and explicit candidate metadata. This keeps the evidence reproducible while isolating the repair-controller behavior.

single_shot:           solve@budget = 0.2
linear_retry:          solve@budget = 0.6
clean_context_review:  solve@budget = 0.6
archive_no_routing:    solve@budget = 0.6
evidence_aware_review: solve@budget = 0.9
full_darwinpatch:      solve@budget = 0.9

The important interpretation is precise:

• ordered retry is not enough;
• bounded evidence plus route-compatible candidate metadata provides the main repair-selection lift;
• full DarwinPatch turns that lift into an auditable reliability system: hard gates, evidence packets, archive records, lineage, failure fingerprints, policy selection, and reports.

This is intentionally an honest MVP result. full_darwinpatch does not claim to beat evidence_aware_review on solve rate in this controlled benchmark; it systematizes the same selection lift into reproducible, inspectable engineering artifacts.

Quick start

python3 -m pip install -r requirements-dev.txt
python3 -m darwinpatch demo --run-dir .darwinpatch_runs/demo

Expected output shape:

DarwinPatch demo complete
LLM required: no
Mode: offline_curated_patches
Hero search: promoted after 2 attempts
Repair path: visible_tests -> behavior_repair -> promoted
Hero report: darwinpatch_runs/demo/hero_report.html
Benchmark: single_shot solve@budget=0.2 vs full_darwinpatch solve@budget=0.9
Benchmark report: darwinpatch_runs/demo/benchmark_report.html
Policy selected: two_attempt_budgeted
Policy summary: darwinpatch_runs/demo/policy/policy_experiment_summary.md
Demo summary: darwinpatch_runs/demo/demo_summary.json

Open the generated reports:

.darwinpatch_runs/demo/hero_report.html
.darwinpatch_runs/demo/benchmark_report.html
.darwinpatch_runs/demo/demo_summary.json
.darwinpatch_runs/demo/policy/policy_experiment_summary.md

These reports show candidate cards, hard-gate outcomes, evidence IDs, fingerprints, route decisions, patch diffs, benchmark tables, confidence intervals, and claim-to-artifact links.

You can see my generated reports at: Benchmark report and Hero report.

System design

DarwinPatch is organized around a small repair loop:

controller.py   repair-search controller
runner.py       isolated task execution
gates.py        hard verification gates
evidence.py     bounded EvidencePacket construction
critic.py       failure-to-route mapping
archive.py      candidate archive
benchmark.py    baselines and evaluation summaries
policy.py       train/validation/test policy experiment
report.py       static HTML reports
llm.py          optional LLM backed hero candidate generation

Each candidate must pass:

1. scope guard,
2. patch application,
3. Python AST parsing,
4. secret scan,
5. visible developer tests,
6. release-gate regression tests.

Visible test failures expose bounded stdout/stderr tails. Release-gate regression failures deliberately withhold full failure output from the repair loop, simulating a hidden-test boundary.

(Note: I have used OpenRouter's free gpt-oss-120b model for optional LLM generation, but the demo is designed to be deterministic and reproducible without an API key. The controlled benchmark uses curated candidate pools rather than live generation.)

Candidate metadata, not filename guessing

Route-aware selection uses declared candidate metadata such as:

{
  "patch": "candidate.patch",
  "label": "spec_complete_repair",
  "generator": "synthetic_generator",
  "intent": "complete_spec_repair",
  "compatible_routes": ["behavior_repair", "regression_repair", "scope_repair"]
}

DarwinPatch does not inspect patch filenames to decide which candidate is likely to be correct. Path-only inputs remain supported for simple CLI use, but without route metadata they fall back to original order rather than filename heuristics.

Baselines

The controlled benchmark includes six baselines:

Baseline	What it tests
single_shot	One candidate, no correction loop.
linear_retry	Ordered retry up to the same candidate budget.
clean_context_review	Ordered retry without DarwinPatch route-aware selection or evidence archive.
archive_no_routing	Archive-compatible retry without route-aware selection.
evidence_aware_review	Bounded evidence plus candidate metadata; no archive, lineage, parent scoring, failure-fingerprint stop logic, or policy machinery.
full_darwinpatch	Evidence packets, failure fingerprints, archive, route-aware selection, hard gates, reports, and policy experiment support.

Benchmark summaries include solve@budget, 95% Wilson confidence intervals, average attempts, regressions blocked, invalid patches rejected, evidence packets admitted, and route metadata matches.

Optional LLM mode

The default demo does not require an LLM or API key. That is intentional: the benchmark is deterministic and reproducible.

For a live hero demonstration, you can optionally use OpenRouter or any other LLM to generate candidate patches before applying the same gates/search/report pipeline:

OpenRouter example with gpt-oss-120b:

export OPENROUTER_API_KEY="..."
python3 -m darwinpatch demo --run-dir .darwinpatch_runs/demo --llm

If a provider has trouble with reasoning payloads:

python3 -m darwinpatch demo --run-dir .darwinpatch_runs/demo --llm --no-llm-reasoning
# equivalent:
python3 -m darwinpatch demo --run-dir .darwinpatch_runs/demo --llm --llm-reasoning false

What this is not

DarwinPatch is a controlled MVP, not a full autonomous software-engineering agent.

It does not claim:

• to solve large multi-file product changes yet,
• to benchmark live LLM generation across the full task suite yet,
• to beat evidence_aware_review on controlled solve rate,
• or to replace the coding model that proposes candidate patches.

The claim is narrower and stronger:

DarwinPatch shows that a coding-agent repair loop can be wrapped with hard verification, bounded evidence, route-aware selection, and auditable reporting. In the controlled benchmark, bounded evidence plus route metadata lifts solve@budget from ordered retry's 0.6 to 0.9, while the full DarwinPatch controller turns that behavior into a reproducible reliability system.

Individual commands

Run the smoke task through deterministic gates:

python3 -m darwinpatch run tasks/smoke_markdown_parser --run-dir .darwinpatch_runs/smoke

Run repair search with candidate archive and routing:

python3 -m darwinpatch search tasks/smoke_markdown_parser --run-dir .darwinpatch_runs/search

Run benchmark ablations:

python3 -m darwinpatch benchmark benchmarks/phase3_smoke_suite.json --run-dir .darwinpatch_runs/benchmark

Generate static HTML reports:

python3 -m darwinpatch report .darwinpatch_runs/search --output .darwinpatch_runs/search_report.html
python3 -m darwinpatch report .darwinpatch_runs/benchmark --output .darwinpatch_runs/benchmark_report.html

Run the guarded outer-loop policy experiment:

python3 -m darwinpatch policy-experiment benchmarks/phase5_policy_experiment.json --run-dir .darwinpatch_runs/policy

The policy experiment evaluates policy variants on train and validation splits, selects one policy using validation metrics, then runs only that selected policy on the held-out test split. Test cases are final-report-only and are not used for tuning or policy selection.

Run tests

python3 -m pytest -q tests