Repository navigation
fix(oie): align OWASP LLM Top10 / AIX B2 gold to hub AI CREs #1124
Description
Problem
The AIX B2 arm (run_full_pipeline.py → fixture owasp_llm_top10_2025) scores 0% exact on the d1-winner combo, while the librarian already returns AI Exchange–topic CREs.
Canonical gold in application/tests/fixtures/owasp_mappings/owasp_llm_top10_2025.json still uses classic AppSec CREs (often via gold_remap: related_expand_orphan). Hub Links gold under tmp/oie_owasp_eval/hub_link_gold/owasp_llm_top10_2025.json is the AI set.
Example (LLM01 Prompt Injection):
- B2 gold: Input validation / Injection protection / Output encoding…
- Hub:
686-110Direct prompt injection,012-625Indirect prompt injection - Pred (d1-winner): includes
686-110→ hub hit, B2 miss
Re-score of d1-winner preds vs nonempty hub gold: 3/6 = 50% (LLM06–09 hub Links empty today).
Evidence
tmp/oie_owasp_eval/experiments/miss_slice_d1_winner.json(worktreeOpenCRE-wt-oie-1088-dual)tmp/oie_owasp_eval/experiments/product_metric_d1_winner.json— aix exact 0%, d1 10%- Unit test already encodes the intended contract and fails on current fixture:
application.tests.owasp_mapping_fixtures_test.TestOwaspMappingFixtures.test_llm_top10_uses_ai_topic_cres_not_classic
Proposed fix
- Rewrite
owasp_llm_top10_2025.jsoncre_idsto hub AI CREs (LLM01–05, LLM10 from hub Links / AI Exchange). - For LLM06–09 (empty hub today): add provisional AI-topic gold (same provenance pattern as other provisional fixtures) — do not leave classic AppSec stand-ins.
- Drop or invert
related_expand_orphanfor this fixture so classic neighbors are not treated as exact gold. - Make
test_llm_top10_uses_ai_topic_cres_not_classicgreen; re-run AIX B2 arm (Module C env unchanged).
Out of scope
- Librarian toggles / Module C ablations (will not fix this mismatch).
- Expecting AIX exact lift from
SHORTLIST_JUDGE/ leaf-drill env alone.
Success
- Fixture test green
- AIX B2 exact ≫ 0% on same d1-winner librarian defaults (target: match hub-proxy ballpark on nonempty sections)
- Docs note in
docs/oie/full-pipeline.mdthat AIX arm gold is hub AI CREs, not classic expand
Activity
northdpole commentedon Oct 2, 2026
Implemented on
stack/oie-p7-full-pipeline-docs(worktree):application/tests/fixtures/owasp_mappings/owasp_llm_top10_2025.jsonto hub AI CREs for LLM01–05/10.gold_provenance.kind=provisional) — hub Links were empty.related_expand_orphanfor this fixture.test_llm_top10_uses_ai_topic_cres_not_classicand parser test green.Also shipping Module C defaults:
FORCE_RESOURCES=api,SHORTLIST_JUDGE_MAX_PICKS=3. Cold-start full verify in progress.northdpole commentedon Oct 2, 2026
Cold-start after hub gold +
MAX_PICKS=3+ experimentalFORCE_RESOURCES=api:Shipping: hub/provisional gold +
SHORTLIST_JUDGE_MAX_PICKS=3. LeavingFORCE_RESOURCESoff by default (API exact regression). Evidence:tmp/oie_owasp_eval/experiments/product_metric_cold_start_post60.json.bhushan4work commentedon Oct 4, 2026
I would like to work on this issue , could u please assign it to me
V-2007-P commentedon Oct 19, 2026
Hey! @northdpole ...
I’m interested in contributing to this issue. I noticed that the fixture updates and tests have already been reported as completed. Is there any remaining part of this issue that would be useful for me to take on? I’d be happy to help once the scope is clear.
Bornunique911 commentedon Nov 2, 2026
Hey @northdpole — verified that the fixture-alignment requirements in this issue are already present on
staging: AI-topic hub/provisional gold, norelated_expand_orphanon the LLM fixture, a passing AI-topic fixture test, and the AIX gold documentation note.PR #1197 is complementary rather than the implementation of the fixture fix. It adds B2 report provenance (
gold,gold_original,gold_remap,gold_is_expanded, and remap summary) so future scoring can be audited for expanded/remapped evidence without changing scoring or fixture content.The only acceptance evidence not independently re-run by #1197 is the AIX B2 measurement; the issue’s existing report records 50% exact after the gold fix.