Skip to content

Run the live Claude Code tests only when asked - #478

Closed
bar-capsule wants to merge 1 commit into
refimpl/adapters-02-core-clientfrom
refimpl/adapters-03-live-tests-opt-in
Closed

bar-capsule wants to merge 1 commit into
refimpl/adapters-02-core-clientfrom
refimpl/adapters-03-live-tests-opt-in

Conversation

@bar-capsule

@bar-capsule bar-capsule commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Stacked on #477. Next: #479. The full stack is listed in #22.

What changed

  • The live Claude Code tests drove the real claude CLI on the logged-in account whenever claude was on PATH. They now run only with ACS_LIVE_CLAUDE=1, and with the flag set, a missing CLI is an error.
  • run_conformance.py allowlists the opt-in skip by its exact reason, and a test checks that claude on PATH alone leaves them skipped.

Findings this answers. Raised while reviewing this stack. Every tip below this one was tested with claude removed from PATH.

Which issue does this implement

Part of #134

Base branch

Stacked on #477, not on integration. It lands through #22; see the stack in #22.

Type of change

  • Reference implementation or adapter

I tested this

At this slice's top (e4a8763b), with claude removed from PATH and ACS_LIVE_CLAUDE and ACS_SPEC_DIR unset: 315 passed and 3 expected skips with the NAT extra, 277 passed and 3 expected skips without it. The skips are the two opt-in live Claude Code tests and the manual Cursor procedure. CI passes on this PR, including the installed-wheel check on Python 3.11 and 3.12.

Checklist

  • Commits are signed off with git commit -s (required by the DCO)
  • Prose follows STYLE.md
  • No secrets, tokens, or internal URLs in the diff

Security

Unchanged adapters; the live Claude Code tests run only when asked.

The live tests run only with ACS_LIVE_CLAUDE=1, and with the flag set a
missing claude CLI is an error.

Signed-off-by: Bar Kaduri <bar@capsule.security>

@drk-ram drk-ram left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at head e4a8763b58d4710f6eb799fbfce2942da2f5e221.

Small, focused change and I like the direction: gating the live Claude Code round-trip test behind ACS_LIVE_CLAUDE=1 instead of the old shutil.which("claude") is not None check is a real fix, not just cosmetic. The old gate meant anyone who happened to have the claude CLI on PATH in CI or on a dev box would silently start running a test that spawns a real claude --print subprocess against a live Guardian - that's a surprising thing to have happen to you without opting in.

What I checked

  • LiveClaudeCodeRoundTrip still raises RuntimeError in setUpClass if ACS_LIVE_CLAUDE=1 is set but claude isn't actually on PATH, rather than silently skipping - so a misconfigured opt-in fails loudly instead of reading as a pass.
  • run_conformance.py's EXPECTED_SKIPS reason strings were updated to match test_live.LIVE_SKIP_REASON exactly, and test_run_conformance.py's new LiveClaudeIsOptIn test asserts that equality directly (test_allowlisted_reasons_match_the_test) plus the behavioral claim that having claude on PATH alone does not enable the tests (via a real subprocess with a fake claude shim on PATH and ACS_LIVE_CLAUDE stripped from the environment) - good, this is testing the actual regression this PR fixes rather than just the new code path.
  • Docs (adapters/README.md, docs/claude_code/README.md) were updated consistently with the new opt-in flag.

No issues found.

Approving.

@rocklambros

Copy link
Copy Markdown
Collaborator

Closing this slice because PR #22 (the reference adapters for Claude Code, Cursor and NAT) merged into integration as 2630ec2, and that merge carries every slice in this stack. The top of the stack had the same files as PR #22's final commit, so nothing in this slice was left behind. Thanks, Bar, for cutting the adapters into pieces this size, because it made the review possible.

@rocklambros rocklambros closed this Nov 5, 2026
@github-project-automation github-project-automation Bot moved this from In progress to Done in ACS Project Tracker Nov 5, 2026
@bar-capsule
bar-capsule deleted the refimpl/adapters-03-live-tests-opt-in branch November 8, 2026 09:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants