Repository navigation
Constrain Smart Approve judge results to the operation reviewed #447
Description
Audit observation
The optional SMART_APPROVE_JUDGE path is disabled by default. Source review and a deterministic scripted-provider test show that the judge receives only tool names, and a returned name can be cached as a tool-wide AlwaysAllow permission. A scripted response also included a tool name that was not requested. The test measured an approval decision only; it did not dispatch a tool, use a live model, or establish an unauthorized action in a real session.
This is a configuration-specific design concern that limits what the current audit can claim about Smart Approve.
Recommendation
Preserve no-prompt behavior for tools with trustworthy, operation-appropriate read-only metadata. For unclassified tools, bind any automated decision to the exact operation reviewed, reject unsolicited results, and avoid persisting a broad tool-name AlwaysAllow from an LLM judgment. If the operation cannot be scoped safely, retain the existing user approval prompt. This keeps common registered read-only workflows convenient while requiring confirmation only for ambiguous calls.
Scope
No code change is included for this concern in the current remediation patch. The judge is opt-in and disabled by default; its live-model behavior was outside the audit.
Activity
Broccolito commentedon Oct 23, 2026
Defensive audit remediation update (local source branch; not released)
The optional judge now receives each request ID, tool name and complete arguments. Returned ID/name pairs must match the current candidate set; malformed or unsolicited pairs and ambiguous duplicate input IDs fail closed. An accepted verdict applies only to that invocation and creates no persistent tool-wide grant. Legacy automatic allows are ignored centrally, while restrictive legacy preferences and explicit human grants remain effective.
Verification: all four new deterministic judge tests and all 12 Smart Approve policy regressions passed on macOS/arm64. These cover same-name read/write distinctions, argument inclusion, malformed/unsolicited responses, provider failure, duplicate IDs, no approval reuse or persistence, and migration behavior. The separate two-test public scope/configuration suite also passed. The audit's scripted observation suite confirmed fresh approval is required for a later write and unsolicited verdicts are rejected; no tool side effect was dispatched.
Recommendations preserving convenience: continue using registered risk annotations for known tools, keep unknown tools subject to confirmation by default, preserve explicit user-created grants, and keep the LLM judge opt-in and invocation-scoped. The judge remains disabled by default and still depends on model classification quality. These tests do not demonstrate live-model resistance to deceptive content or make a model verdict a deterministic authorization boundary.
Status: source changes and focused regressions are complete locally on
codex/security-reviewer-assessment(basef73ebf74c578717a7af64b221e57358a63dda8e3). No merge or fixed distribution is claimed. Leaving this issue open for maintainer review and integration.Broccolito commentedon Oct 24, 2026
Issue #447: Constrain Smart Approve judge results to the reviewed invocation
Issue · Host PR #454 · merged main source
Historical desktop/browser/CLI observations used f227 artifacts, accepted via the reviewed production-equivalence comparison; exact-main CI/build/smoke is attributed separately below.
Problem. A model-derived read-only classification could apply too broadly to later invocations of a tool.
Fix and UX. The optional model-based judge receives the actual request ID, tool name and arguments. Returned ID/name pairs must match the inspected set; malformed, unsolicited and out-of-scope decisions fail closed. Duplicate input request IDs are rejected; repeated matching output verdicts are deduplicated. Inspector allowance stays invocation-local and creates no persistent tool-wide judge grant. Legacy Smart AlwaysAllow does not silently bypass fresh decisions; explicit human/restrictive policy retains precedence. Only the optional model-based judge defaults off; annotation risk grading remains enabled by default. Under the default policy, ambiguity returns to normal human confirmation.
Validation. Exact merged-main Ubuntu CI passed
smart_approve_judge_tests(4/4) andsmart_approve_tests(12/12): actual arguments/same-tool distinction, invocation nonreuse, malformed/unsolicited decisions, duplicate input IDs and legacy-policy migration. Merged-main Rust CI and UI CI remain separately attributed. Deterministic judge responses validate authority handling, not live-model accuracy.Limits. The earlier draft's “duplicate returned candidates rejected” wording was too broad; matching duplicates are deduplicated without extra authority. A judge can misclassify instructions and regex filters can miss malicious input. Arguments may reach the configured provider when enabled. No measured live-model jailbreak resistance or comprehensive approval guarantee is claimed. Relevant reviewer label: Reviewer 1 Specific Minor Comment 3.
Recommendation. Keep judge decisions invocation-local and recover ambiguity through normal confirmation; do not turn model verdicts into persistent name-wide grants.
Merged-main verification. PR454 is merged at
0580110; main Rust CI, frontend CI and related checks finished 20 successes/one skipped (17 push successes plus three landing/deployment checks; these are not security-test counts). Normal main build manifest SHA256:38cc2979c7da104cb623975bac7f6922a085e834a1be5c18e961d7f26ea811fc; shipping receipt:e6e1d8957d8d0b22a9aae814011a1e028c5d279848ffabbf4c6d0c4edf3c0112. Actual main01 smoke separately recorded four pairs—three reads/one typed Deny—and three unchanged canaries; it does not rerun/recount the historical 81-pair cohort.Submission record. Manuscript v1.1 local Supplementary Data S1: Supplementary Data S1 - BioRouter Security Audit - 2026-10-24.zip; manifest SHA256 4f0c0c110ca0b07a29d5281870227f7b36c744a0f37ccb499b8ef1283dea116e. This is a user/submission package, not a claim of public upload. Public PR/source/CI links above provide the public implementation proof.