feat(oie): RQ A→B→C fan-out and Admin OIE queue tab by northdpole · Pull Request #1142 · OWASP/OpenCRE

northdpole · 2026-10-09T08:36:59Z

Summary

Add RQ queue oie fan-out for Module A (per repo) → B/C (per artifact_id), with --rq / CRE_OIE_RQ, start_oie_workers, and harvest_input.artifact_id migration.
Admin OIE queue tab + shared RQ dashboard snapshot; wire import-all / workers for the oie queue.
Fix B2 eval harvester LangGraph repos_yaml TypeError so top10/llm/api score arms can run; keep promoted librarian/safety knobs usable on RQ workers.

Test plan

python -m unittest application.tests.oie_rq.fanout_test application.tests.oie_rq.stage_filter_test application.tests.oie_b2_gold_loader_test.TestOieB2HarvesterLanggraphKwargs
make alembic-guardrail (new revision a9b0c1d2e3f4)
Local: make start-oie-workers + scripts/run_oie_pipeline.py --rq (or full_pipeline --rq) drains queue without silent B2 TypeError
Admin OIE tab loads /admin/oie/rq/status with queue depth

Stacked on #1137 (admin-panel-showcase).

Made with Cursor

coderabbitai · 2026-10-09T08:37:10Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review: 1

main

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

Source mode: Auto
Repository: OWASP/OpenCRE
Branch: feat/oie-rq-fanout
File: .coderabbit.yml
Commit: 8161414
Layers: Repository YAML
Review profile: CHILL
Plan: Advanced
Run ID: 75b80a9e-cd95-44fc-a502-e0af5ed52742

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

Trigger review

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Dual-index retrieval, CRE summaries, prior cage, hub seed, edition remap, and related librarian defaults. Rewrites #1088 as the reviewable base; later stacked PRs add harvester, B2 eval, fixtures, mappings, and full-pipeline docs.

- Refuse plain-text fetch redirects that leave GitHub raw hosts (SSRF). - Re-check expected embedding dimension on query-cache hits. - Fill phrase-only oie blocks in apply_oie_fill_if_missing without losing phrases. - Expose last_preferred_cre_ids on StandardHopRetriever and honour its allowlist. - Commit each metadata enrichment so a later failure cannot roll back earlier work. - Load .env from the repo root in the enrichment scripts. - Parse hostnames for CRE hub sync SSL and loopback checks; batch SQLite IN lookups; stop echoing database credentials.

Optional requirement extract (off/auto/on) plus GitHub tarball/URL ingest wiring for Module B → C. Stacked on #1088 librarian base.

Harness runners, hop-distance scoring, and gold loaders without bulky source fixtures (those land in follow-up stacked PRs).

Wire LlmSafetyGuard (LiteLLM) as the live factory default so runs evaluate safety instead of NullSafetyGuard. Apply the guard on every decision path, prefer safety reason codes over empty shortlists, and cover a synthetic adversarial corpus in unit tests.

- Do not fall back to main/master when the user named a branch. - Download tarballs with a timeout and a size cap; cap per-member and total markdown bytes read from archives. - Validate owner, repo and branch and URL-encode them for codeload. - Delete prior ingest rows only after download and chunking succeed, and never rows from other runs. - Pass the model override through a local config instead of os.environ.

- Redact database passwords in orchestrator, B2 and setup_oie output; write tmp/oie.env with mode 600. - Keep the existing --db for --skip-a and a user-supplied --repos-yaml. - Create the artifact dir before writing orchestrator reports. - Clear per-model knowledge rows between Qwen and Gemini classify runs. - Key the switch-grid embed cache by model, honour --force with stale-lock recovery, write explicit zero flags to stack_winners.yaml, and tolerate LibrarianConfig fields that are env-only. - Match B1 sheet-grain gold exactly instead of by substring.

Large fixture slice for ASVS eval arms only — split from NIST/other sources so reviewers can treat this as mechanical data review.

NIST OSCAL sample plus Top10/API/K8s/LLM/AISVS/agentic fixture sources and gold JSON. Stacked after the ASVS-only fixture PR.

Update OWASP mapping fixtures and loader helpers so orphan umbrella Related targets become canonical gold for eval and product mappings.

- Forward leaf_drilldown_min_sections=0 instead of replacing it with 20. - Make queue wiping in run_full_pipeline opt-in via --wipe-queues and add the flag to the detached clone-DB runner, which now reports its real exit code. - Fall back to sys.executable when venv/bin/python is missing. - Fix the B2 report gate string (top-3) and drop the nonexistent make target from the docs. - Make the denied-resource drill-down test actually reach the allowlist gate.

Prior-caged retrieval embeds the same header and body several times per chunk. Prefetch the unique strings for a queue batch so those Gemini calls collapse into one list request.

* trim/1127: chore(owasp-agent): type embed_sync and its test chore: drop generated frontend bundle (rebuild at deploy) refactor(owasp-agent): move the meta-embedding push out of sync_all into embed_sync # Conflicts: # application/frontend/www/bundle.js

…a-embedding push sync_all no longer embeds; call embed_meta_if_enabled after it so deployments that set OWASP_AGENT_EMBED_META=1 keep embedding.

* trim/1137: fix(owasp-agent): admin sync and scheduler stage apply the opt-in meta-embedding push chore(owasp-agent): type embed_sync and its test chore: drop generated frontend bundle (rebuild at deploy) refactor(owasp-agent): move the meta-embedding push out of sync_all into embed_sync # Conflicts: # application/tests/oie_scheduler/runner_test.py

…EST_EVENTS and OIE_EMBED_FILED_NODES Embedding auto-filed nodes spends provider tokens and writes the embeddings table, so it is opt-in (OIE_EMBED_FILED_NODES=1) like OIE_HARVEST_EVENTS.

* trim/1136: fix(oie-scheduler): embed_filed_nodes defaults off, document OIE_HARVEST_EVENTS and OIE_EMBED_FILED_NODES fix(owasp-agent): admin sync and scheduler stage apply the opt-in meta-embedding push chore(owasp-agent): type embed_sync and its test chore: drop generated frontend bundle (rebuild at deploy) refactor(owasp-agent): move the meta-embedding push out of sync_all into embed_sync # Conflicts: # application/frontend/www/bundle.js # application/tests/admin_panel_test.py # docs/oie/feature-flags.md

…base

* trim/1127: test(owasp-agent): sync and concepts must not import application.database

* trim/1137: test(owasp-agent): sync and concepts must not import application.database

* trim/1136: test(owasp-agent): sync and concepts must not import application.database

…and counts real vectors

…d of patch.dict in the embed_sync tests

* trim/1127: test(owasp-agent): restore the environment with a plain helper instead of patch.dict in the embed_sync tests

…embed_fn instead of patching embed_sync

* trim/1137: test(owasp-agent): exercise the meta-embedding push through injected embed_fn instead of patching embed_sync test(owasp-agent): restore the environment with a plain helper instead of patch.dict in the embed_sync tests # Conflicts: # application/tests/oie_scheduler/runner_test.py # application/utils/oie_scheduler/jobs.py

…he new opt-in assertions

* trim/1136: test(oie-scheduler): restore the environment with a plain helper in the new opt-in assertions test(owasp-agent): exercise the meta-embedding push through injected embed_fn instead of patching embed_sync test(owasp-agent): restore the environment with a plain helper instead of patch.dict in the embed_sync tests # Conflicts: # application/tests/admin_panel_test.py # application/utils/admin_panel/service.py

…vironment with a plain helper in the #1142 tests

* trim/1142: trim(#1142): admin upstream-sync-and-ingest no longer replaces existing CRE embeddings trim(#1142): no credentialed CORS without an allowlist, OIE workers opt-in in import-all, guard levers-matrix queue reset trim(#1137): drop brevity-bot skills and stale handover doc, restore README/dev setup, loopback guard for run_admin_local.sh trim(#1127): drop adversarial asker/batch scripts and canned CRE follow-ups, set source_channel in raw embedding writers trim(#1120): fix merge-conflict markers, guard full pipeline, drop dead experiments style: black-format llm_safety_guard.py chore(oie): trim #1118 for the staging rollout chore(oie): trim #1117 for the staging rollout chore(oie): trim #1116 and guard destructive eval scripts chore(harvester): trim #1115 for the staging rollout chore(oie): trim #1088 for the staging rollout test: embedding-failure scheduler test needs graph filing enabled now that it defaults off style: format files flagged by CI (black, prettier) Make event-repo harvesting opt-in (OIE_HARVEST_EVENTS) Make event page parsing linear-time and clean Playwright fallback text Embed auto-filed graph nodes in the CRE expansion job Parse OWASP event pages and harvest event repos # Conflicts: # scripts/owasp_agent_adversarial_asker.py # scripts/owasp_agent_adversarial_batch_run.py

* trim/1155: trim(#1155): drop laptop-local edge probe scripts, make node metadata seeding opt-in, fix empty-key blurb match trim(#1142): admin upstream-sync-and-ingest no longer replaces existing CRE embeddings trim(#1142): no credentialed CORS without an allowlist, OIE workers opt-in in import-all, guard levers-matrix queue reset trim(#1137): drop brevity-bot skills and stale handover doc, restore README/dev setup, loopback guard for run_admin_local.sh trim(#1127): drop adversarial asker/batch scripts and canned CRE follow-ups, set source_channel in raw embedding writers trim(#1120): fix merge-conflict markers, guard full pipeline, drop dead experiments style: black-format llm_safety_guard.py chore(oie): trim #1118 for the staging rollout chore(oie): trim #1117 for the staging rollout chore(oie): trim #1116 and guard destructive eval scripts chore(harvester): trim #1115 for the staging rollout chore(oie): trim #1088 for the staging rollout test: embedding-failure scheduler test needs graph filing enabled now that it defaults off style: format files flagged by CI (black, prettier) Make event-repo harvesting opt-in (OIE_HARVEST_EVENTS) Make event page parsing linear-time and clean Playwright fallback text Embed auto-filed graph nodes in the CRE expansion job Parse OWASP event pages and harvest event repos

* trim/1159: trim(#1159): drop cambridge->boston alias, dead locals, test env/tempfile leaks trim(#1155): drop laptop-local edge probe scripts, make node metadata seeding opt-in, fix empty-key blurb match trim(#1142): admin upstream-sync-and-ingest no longer replaces existing CRE embeddings trim(#1142): no credentialed CORS without an allowlist, OIE workers opt-in in import-all, guard levers-matrix queue reset trim(#1137): drop brevity-bot skills and stale handover doc, restore README/dev setup, loopback guard for run_admin_local.sh trim(#1127): drop adversarial asker/batch scripts and canned CRE follow-ups, set source_channel in raw embedding writers trim(#1120): fix merge-conflict markers, guard full pipeline, drop dead experiments style: black-format llm_safety_guard.py chore(oie): trim #1118 for the staging rollout chore(oie): trim #1117 for the staging rollout chore(oie): trim #1116 and guard destructive eval scripts chore(harvester): trim #1115 for the staging rollout chore(oie): trim #1088 for the staging rollout test: embedding-failure scheduler test needs graph filing enabled now that it defaults off style: format files flagged by CI (black, prettier) Make event-repo harvesting opt-in (OIE_HARVEST_EVENTS) Make event page parsing linear-time and clean Playwright fallback text Embed auto-filed graph nodes in the CRE expansion job Parse OWASP event pages and harvest event repos

* trim/1158: trim(#1158): strip_injection no longer truncates legitimate questions; correct default-off claims trim(#1159): drop cambridge->boston alias, dead locals, test env/tempfile leaks trim(#1155): drop laptop-local edge probe scripts, make node metadata seeding opt-in, fix empty-key blurb match trim(#1142): admin upstream-sync-and-ingest no longer replaces existing CRE embeddings trim(#1142): no credentialed CORS without an allowlist, OIE workers opt-in in import-all, guard levers-matrix queue reset trim(#1137): drop brevity-bot skills and stale handover doc, restore README/dev setup, loopback guard for run_admin_local.sh trim(#1127): drop adversarial asker/batch scripts and canned CRE follow-ups, set source_channel in raw embedding writers trim(#1120): fix merge-conflict markers, guard full pipeline, drop dead experiments style: black-format llm_safety_guard.py chore(oie): trim #1118 for the staging rollout chore(oie): trim #1117 for the staging rollout chore(oie): trim #1116 and guard destructive eval scripts chore(harvester): trim #1115 for the staging rollout chore(oie): trim #1088 for the staging rollout test: embedding-failure scheduler test needs graph filing enabled now that it defaults off style: format files flagged by CI (black, prettier) Make event-repo harvesting opt-in (OIE_HARVEST_EVENTS) Make event page parsing linear-time and clean Playwright fallback text Embed auto-filed graph nodes in the CRE expansion job Parse OWASP event pages and harvest event repos # Conflicts: # application/mcp/ask.py # application/web/web_main.py

* trim/1162: (27 commits) docs(env): document CHAT_BLOCK_UNVERIFIED (default on, changes public chat answers) trim(#1162): drop prune_noise_chunks (unguarded DELETE on embeddings), validate authority weights before use, document ANSWER_LOG_*, no host paths in sync_check_staging trim(#1158): strip_injection no longer truncates legitimate questions; correct default-off claims trim(#1159): drop cambridge->boston alias, dead locals, test env/tempfile leaks trim(#1155): drop laptop-local edge probe scripts, make node metadata seeding opt-in, fix empty-key blurb match trim(#1142): admin upstream-sync-and-ingest no longer replaces existing CRE embeddings trim(#1142): no credentialed CORS without an allowlist, OIE workers opt-in in import-all, guard levers-matrix queue reset trim(#1137): drop brevity-bot skills and stale handover doc, restore README/dev setup, loopback guard for run_admin_local.sh trim(#1127): drop adversarial asker/batch scripts and canned CRE follow-ups, set source_channel in raw embedding writers trim(#1120): fix merge-conflict markers, guard full pipeline, drop dead experiments style: black-format llm_safety_guard.py chore(oie): trim #1118 for the staging rollout chore(oie): trim #1117 for the staging rollout chore(oie): trim #1116 and guard destructive eval scripts chore(harvester): trim #1115 for the staging rollout chore(oie): trim #1088 for the staging rollout fix(auth): harden web sign-in found by adversarial review fix(auth): /auth/user needs any role, not admin fix(auth): carry the PKCE code_verifier from login to callback fix(auth): build a fresh OAuth flow per login instead of a shared singleton ...

* trim/1142: docs(staging): use placeholder tester addresses; test: restore the environment with a plain helper in the #1142 tests test(oie-scheduler): restore the environment with a plain helper in the new opt-in assertions test(owasp-agent): exercise the meta-embedding push through injected embed_fn instead of patching embed_sync test(owasp-agent): restore the environment with a plain helper instead of patch.dict in the embed_sync tests

* trim/1155: docs(staging): use placeholder tester addresses; test: restore the environment with a plain helper in the #1142 tests test(oie-scheduler): restore the environment with a plain helper in the new opt-in assertions test(owasp-agent): exercise the meta-embedding push through injected embed_fn instead of patching embed_sync test(owasp-agent): restore the environment with a plain helper instead of patch.dict in the embed_sync tests

* trim/1159: test(owasp-agent): restore the environment with a plain helper in index_routing_test docs(staging): use placeholder tester addresses; test: restore the environment with a plain helper in the #1142 tests test(oie-scheduler): restore the environment with a plain helper in the new opt-in assertions test(owasp-agent): exercise the meta-embedding push through injected embed_fn instead of patching embed_sync test(owasp-agent): restore the environment with a plain helper instead of patch.dict in the embed_sync tests

* trim/1158: test(owasp-agent): restore the environment with a plain helper in index_routing_test docs(staging): use placeholder tester addresses; test: restore the environment with a plain helper in the #1142 tests test(oie-scheduler): restore the environment with a plain helper in the new opt-in assertions test(owasp-agent): exercise the meta-embedding push through injected embed_fn instead of patching embed_sync test(owasp-agent): restore the environment with a plain helper instead of patch.dict in the embed_sync tests

* trim/1162: test(auth): inject the Google flow and id-token verifier into the web callback instead of patching them; plain env helper; placeholder addresses in .env.example test(owasp-agent): restore the environment with a plain helper in index_routing_test docs(staging): use placeholder tester addresses; test: restore the environment with a plain helper in the #1142 tests test(oie-scheduler): restore the environment with a plain helper in the new opt-in assertions test(owasp-agent): exercise the meta-embedding push through injected embed_fn instead of patching embed_sync test(owasp-agent): restore the environment with a plain helper instead of patch.dict in the embed_sync tests

northdpole requested review from Pa04rth, paoga87 and robvanderveer as code owners October 9, 2026 08:37

northdpole force-pushed the admin-panel-showcase branch from e114292 to 4178ecf Compare October 11, 2026 23:04

northdpole force-pushed the feat/oie-rq-fanout branch from 723d697 to cba71ba Compare October 11, 2026 23:05

northdpole mentioned this pull request Oct 11, 2026

Fix OWASP agent probe gaps (chapters, people, membership, CRE facts) #1127

Merged

4 tasks

northdpole force-pushed the admin-panel-showcase branch from 4178ecf to aad929a Compare October 12, 2026 07:05

northdpole force-pushed the feat/oie-rq-fanout branch from cba71ba to 2a1c75d Compare October 12, 2026 07:05

northdpole force-pushed the admin-panel-showcase branch from aad929a to 475945f Compare October 12, 2026 07:17

northdpole force-pushed the feat/oie-rq-fanout branch from 2a1c75d to 545522b Compare October 12, 2026 07:17

northdpole force-pushed the admin-panel-showcase branch from 475945f to 48e9d4f Compare October 12, 2026 07:19

northdpole force-pushed the feat/oie-rq-fanout branch 3 times, most recently from d68ddbc to 7aac2f5 Compare October 12, 2026 22:19

northdpole force-pushed the admin-panel-showcase branch from 48e9d4f to 30a2a40 Compare October 12, 2026 22:19

northdpole mentioned this pull request Oct 13, 2026

Admin panel redesign (CRE blues) on OIE RQ fanout tip #1146

Merged

4 tasks

northdpole added 13 commits October 14, 2026 09:39

feat(harvester): requirement extract and GitHub URL ingest

70af687

Optional requirement extract (off/auto/on) plus GitHub tarball/URL ingest wiring for Module B → C. Stacked on #1088 librarian base.

feat(oie): B2 eval harness and CRE neighborhood scorer

583e987

Harness runners, hop-distance scoring, and gold loaders without bulky source fixtures (those land in follow-up stacked PRs).

chore(oie): ASVS 5.0 provisional B2 sources and gold

955528d

Large fixture slice for ASVS eval arms only — split from NIST/other sources so reviewers can treat this as mechanical data review.

chore(oie): NIST 800-53 and remaining B2 sources/gold

86d320a

NIST OSCAL sample plus Top10/API/K8s/LLM/AISVS/agentic fixture sources and gold JSON. Stacked after the ASVS-only fixture PR.

feat(mappings): promote Related-orphan remaps into canonical fixtures

62cedd4

Update OWASP mapping fixtures and loader helpers so orphan umbrella Related targets become canonical gold for eval and product mappings.

feat(librarian): batch Module C query embeddings

53562bf

Prior-caged retrieval embeds the same header and body several times per chunk. Prefetch the unique strings for a queue batch so those Gemini calls collapse into one list request.

fix(librarian): keep LLM SafetyGuard when restacking leaf-drilldown

657284b

northdpole added 17 commits November 1, 2026 12:23

fix(owasp-agent): admin sync and scheduler stage apply the opt-in met…

81fae82

…a-embedding push sync_all no longer embeds; call embed_meta_if_enabled after it so deployments that set OWASP_AGENT_EMBED_META=1 keep embedding.

fix(oie-scheduler): embed_filed_nodes defaults off, document OIE_HARV…

a600ae0

…EST_EVENTS and OIE_EMBED_FILED_NODES Embedding auto-filed nodes spends provider tokens and writes the embeddings table, so it is opt-in (OIE_EMBED_FILED_NODES=1) like OIE_HARVEST_EVENTS.

test(owasp-agent): sync and concepts must not import application.data…

cf93371

…base

Merge branch 'trim/1127' into trim/1137

f1c5640

* trim/1127: test(owasp-agent): sync and concepts must not import application.database

Merge branch 'trim/1137' into trim/1136

1dfabfd

* trim/1137: test(owasp-agent): sync and concepts must not import application.database

Merge branch 'trim/1136' into trim/1142

7a85643

* trim/1136: test(owasp-agent): sync and concepts must not import application.database

fix(admin): CRE embedding guard only treats a missing table as empty …

29865ca

…and counts real vectors

test(owasp-agent): restore the environment with a plain helper instea…

823027a

…d of patch.dict in the embed_sync tests

Merge branch 'trim/1127' into trim/1137

da23c54

* trim/1127: test(owasp-agent): restore the environment with a plain helper instead of patch.dict in the embed_sync tests

test(owasp-agent): exercise the meta-embedding push through injected …

ad96712

…embed_fn instead of patching embed_sync

test(oie-scheduler): restore the environment with a plain helper in t…

d670774

…he new opt-in assertions

docs(staging): use placeholder tester addresses; test: restore the en…

8161414

…vironment with a plain helper in the #1142 tests

northdpole changed the base branch from admin-panel-showcase to staging November 1, 2026 22:51

northdpole merged commit 989384f into staging Nov 1, 2026
11 checks passed

northdpole deleted the feat/oie-rq-fanout branch November 1, 2026 22:51

Navigation Menu

Repository navigation

feat(oie): RQ A→B→C fan-out and Admin OIE queue tab - #1142

feat(oie): RQ A→B→C fan-out and Admin OIE queue tab#1142
northdpole merged 166 commits into
stagingfrom
feat/oie-rq-fanout

northdpole commented Oct 9, 2026

Uh oh!

coderabbitai Bot commented Oct 9, 2026 •

edited

Loading

Review skipped

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

northdpole commented Oct 9, 2026

Summary

Test plan

Uh oh!

coderabbitai Bot commented Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

coderabbitai Bot commented Oct 9, 2026 •

edited

Loading