Repository navigation
Run the live Claude Code tests only when asked - #478
bar-capsule wants to merge 1 commit into
Conversation
The live tests run only with ACS_LIVE_CLAUDE=1, and with the flag set a missing claude CLI is an error. Signed-off-by: Bar Kaduri <bar@capsule.security>
drk-ram
left a comment
There was a problem hiding this comment.
Reviewed at head e4a8763b58d4710f6eb799fbfce2942da2f5e221.
Small, focused change and I like the direction: gating the live Claude Code round-trip test behind ACS_LIVE_CLAUDE=1 instead of the old shutil.which("claude") is not None check is a real fix, not just cosmetic. The old gate meant anyone who happened to have the claude CLI on PATH in CI or on a dev box would silently start running a test that spawns a real claude --print subprocess against a live Guardian - that's a surprising thing to have happen to you without opting in.
What I checked
LiveClaudeCodeRoundTripstill raisesRuntimeErrorinsetUpClassifACS_LIVE_CLAUDE=1is set butclaudeisn't actually on PATH, rather than silently skipping - so a misconfigured opt-in fails loudly instead of reading as a pass.run_conformance.py'sEXPECTED_SKIPSreason strings were updated to matchtest_live.LIVE_SKIP_REASONexactly, andtest_run_conformance.py's newLiveClaudeIsOptIntest asserts that equality directly (test_allowlisted_reasons_match_the_test) plus the behavioral claim that havingclaudeon PATH alone does not enable the tests (via a real subprocess with a fakeclaudeshim on PATH andACS_LIVE_CLAUDEstripped from the environment) - good, this is testing the actual regression this PR fixes rather than just the new code path.- Docs (
adapters/README.md,docs/claude_code/README.md) were updated consistently with the new opt-in flag.
No issues found.
Approving.
|
Closing this slice because PR #22 (the reference adapters for Claude Code, Cursor and NAT) merged into integration as 2630ec2, and that merge carries every slice in this stack. The top of the stack had the same files as PR #22's final commit, so nothing in this slice was left behind. Thanks, Bar, for cutting the adapters into pieces this size, because it made the review possible. |
Stacked on #477. Next: #479. The full stack is listed in #22.
What changed
claudeCLI on the logged-in account wheneverclaudewas on PATH. They now run only withACS_LIVE_CLAUDE=1, and with the flag set, a missing CLI is an error.run_conformance.pyallowlists the opt-in skip by its exact reason, and a test checks thatclaudeon PATH alone leaves them skipped.Findings this answers. Raised while reviewing this stack. Every tip below this one was tested with
clauderemoved from PATH.Which issue does this implement
Part of #134
Base branch
Stacked on #477, not on
integration. It lands through #22; see the stack in #22.Type of change
I tested this
At this slice's top (
e4a8763b), withclauderemoved from PATH andACS_LIVE_CLAUDEandACS_SPEC_DIRunset: 315 passed and 3 expected skips with the NAT extra, 277 passed and 3 expected skips without it. The skips are the two opt-in live Claude Code tests and the manual Cursor procedure. CI passes on this PR, including the installed-wheel check on Python 3.11 and 3.12.Checklist
git commit -s(required by the DCO)Security
Unchanged adapters; the live Claude Code tests run only when asked.