[steadhac]

Summary

Adds test creation agents for finbot: a pre-merge check finds which changes have no tests, and a set of guides and agents helps contributors write those tests the way this repo already does. CI runs the check with no model and no secrets, so it works on pull requests from forks.

No application code changes.

Why

• CI runs four test files today. A pull request can change a detector or an agent without any related test running.
• Real example: three tests in tests/unit/agents/test_specialized_agents.py have failed on main since they were merged, and nothing caught it, because CI never runs that file.
• Contributions are increasingly AI-assisted, so code arrives faster than it can be reviewed by hand. The check makes each pull request show its own test gaps.

What's in this pull request

Part	What it does
scripts/verify_change.py	Finds the tests that cover each changed file, runs them, validates changed challenge YAML (schema, detector registered, config accepted, prerequisites exist), and with --coverage lists the changed lines no test runs. Verdicts: CLEARED, NEEDS TESTS, BLOCKED, NO TESTS APPLY
tests/unit/scripts/test_verify_change.py	26 tests for the check, including seven end-to-end runs in a throwaway git repository
CI job verify-change	Pull requests only, read-only token, LLM_PROVIDER=mock, no secrets. Writes the test gaps to the job summary with a ready-made prompt for each, and uploads a JSON report
AGENTS.md	Testing instructions that Codex, Copilot, Cursor and Gemini CLI read automatically
.claude/skills/	Testing guides for detectors and agents, a pre-merge workflow, acceptance criteria, and the repo's bug-ticket format
.claude/agents/	test-writer, bug-hunter and requirements-reviewer, each pinned to a model

How it fits together

flowchart LR
    dev([Contributor]) -->|/verify-change| VC[verify-change skill]
    VC -->|runs| S[[scripts/verify_change.py]]
    S -->|untested files and lines| VC
    VC -->|one agent per gap, in parallel| TW
    VC -->|a test failed| QT

    subgraph agents [Agents: the workers]
        TW[test-writer<br/>Sonnet]
        BH[bug-hunter<br/>Opus]
        RR[requirements-reviewer<br/>Sonnet]
    end

    subgraph skills [Skills: the how-to guides]
        DT[detector-tests]
        AT[agent-tests]
        AC[acceptance-criteria]
        QT[qa-ticket]
    end

    TW -->|testing a detector| DT
    TW -->|testing an agent| AT
    BH -->|writes tests meant to break it| DT
    BH -->|writes tests meant to break it| AT
    BH -->|confirmed bugs| QT
    RR -->|turns an issue into criteria| AC

    other([Codex, Copilot, Cursor, Gemini]) -->|AGENTS.md| DT
    other -->|AGENTS.md| AT

Loading

Agents write tests on the contributor's laptop. CI only runs the check and the committed tests, with no model and no key.

Tests are written locally and CI only checks them. The finished tests mock the model, so they run in CI at no cost.

Try it

uv sync
uv run python scripts/verify_change.py --base origin/main --coverage

Example output, run against an earlier change on main:

Changed lines run by tests: 4/18 (22%, minimum 80%)
  finbot/ctf/detectors/implementations/guardrail_prevention.py: 4/18; not run: 82-83, 123-135

With Claude Code, run /verify-change. With another assistant, AGENTS.md points it at the same guides.

How it was verified

• The check's own tests: 26 tests pass. In a mutation run that planted 12 bugs in the script, the tests caught all 12.
• Against main: it maps changes to their tests, validates challenge definitions and reports uncovered lines, as shown above.
• The agents, on code from main:
  • test-writer wrote 30 tests for communication.py. After two assertions were tightened, they caught 10 of 10 planted bugs.
  • bug-hunter found 5 issues in guardrail_prevention.py, including a webhook that blocks every call completing Carte Noire. These will be filed separately.
  • requirements-reviewer wrote acceptance criteria for CD001-WEB-001: Auth routes unit tests #132 and flagged that the issue names a file that doesn't exist.
• Independent review: two other models reviewed the change, one with the repository and one with the diff only. Confirmed findings are fixed in 7c79808 and 6db4c52.

Decisions worth a look

• The CI job doesn't block merges yet (continue-on-error) because of the three failing tests above. A follow-up fixes them so it can block.
• No model runs in CI. Fork pull requests get no secrets, and an agent reading pull request text is exposed to prompt injection. File names and errors from a pull request are neutralised before they reach the job summary or a prompt.
• .claude/ is a project-level choice. If you'd rather not have it, the script, its tests, the CI job and AGENTS.md stand on their own.
• Before running Claude Code on someone else's pull request, read its changes to .claude/ and AGENTS.md. .claude/README.md says so.

Follow-ups

• Fix the three failing tests in test_specialized_agents.py, then make the job blocking
• Browser tests on phones and desktops (separate pull request)
• Guides for MCP tools and HTTP routes
• File the five guardrail_prevention.py findings

[steadhac]

@saikishu The new Changed code is tested job passes. The red test job is a pre-existing break on main: CI installs unpinned packages and now gets Starlette 1.x, which fails test_vendor_isolation.py on every PR. Fixed in #616 (install from uv.lock); once that merges, this PR's CI should go green.