docs(main): ATR-only baseline on Garak + PINT (per #66)

[eeee2345]

@hiskudin — as you asked in #66, here it is as a PR rather than a comment: the reproducible ATR-only baseline on Garak and PINT.

benchmarks/atr-baseline.md contains:

• Garak — in-scope recall 74.4% (1,321/1,775 across 21 families), with the two out-of-scope families (snowball, harmbench) called out and excluded from the headline number.
• PINT — precision 99.7% / recall 63.6% / F1 77.7% (TP=287, FP=1, TN=398, FN=164); PINT is the only corpus here with a real measured precision, since it carries benign samples.
• Reproduce — pinned to agent-threat-rules commit 1831d0d5, with the exact scripts to regenerate both reports.

The Defender-side column is intentionally left blank — this is the ATR half of a like-for-like table, for your team to fill in on the same corpora, per your offer to review on your side.

Happy to adjust the file's location or format, or to align on the tool-result-focused subset you raised in #66 as a starting point for a bundled pack.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[Copilot]

Pull request overview

Adds a reproducible benchmark note capturing an ATR-only baseline on two corpora (Garak and Lakera PINT) to support the ongoing discussion in issue #66 about potential ATR rule bundling and like-for-like comparison against Defender.

Changes:

• Documented pinned ATR baseline results for Garak (recall) and PINT (precision/recall/F1), including scope caveats.
• Added step-by-step reproduction commands pinned to a specific agent-threat-rules commit for repeatability.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[hiskudin]

Defender column (same corpora as this baseline)

Ran locally against the same Garak + PINT inputs pinned in this PR (agent-threat-rules@1831d0d5 corpora). Criterion: @stackone/defender defendToolResult with blockHighRisk: true → count as positive when allowed === false.

Engine	PINT P	PINT R	PINT F1	PINT confusion	Garak in-scope R (1,775)	Garak overall R (3,475)
ATR (this PR)	99.7%	63.6%	77.7%	TP=287 FP=1 TN=398 FN=164	74.4% (1,321/1,775)	38.2% (1,329/3,475)
Defender Tier 1	100%	12.9%	22.8%	TP=58 FP=0 TN=399 FN=393	33.5% (595/1,775)	17.1% (595/3,475)
Defender Tier 1+2	82.1%	87.6%	84.8%	TP=395 FP=86 TN=313 FN=56	76.8% (1,363/1,775)	41.7% (1,448/3,475)

Notes for reading this honestly:

• Reporting both T1 and T1+T2 — a single “Defender” cell would be misleading (T1 is a small curated regex pack; T2 carries most recall on these chat/jailbreak sets).
• ATR still leads on PINT precision; Defender T1+T2 leads on PINT recall/F1 and is roughly tied with ATR on Garak in-scope recall.
• These corpora are jailbreak/PINT-shaped, not tool-result traffic — still want an FP check on tool-like benign data before deciding on an opt-in ATR pack (ATR rule-pack as a Tier 1 supplement (462 patterns, MIT) + cross-benchmark offer #66).

Happy to fold this into benchmarks/atr-baseline.md once this PR lands (or as a follow-up docs commit).