Skip to content

Test the Grok ACP fallback against a fake CLI and fix two bugs - #969

Merged
Finesssee merged 4 commits into
mainfrom
test/grok-acp-fake-server
Oct 21, 2026
Merged

Finesssee merged 4 commits into
mainfrom
test/grok-acp-fake-server

Conversation

@Finesssee

Copy link
Copy Markdown
Collaborator

Summary

Adds end-to-end tests for the Grok ACP billing fallback (#958) against a scripted fake grok agent stdio. The fallback had only in-memory pipe tests and a single missing-binary check, and it had never been run through a real child process. This PR runs it through one. It also fixes two bugs the fake exposed. Branch test/grok-acp-fake-server, based on origin/main 546b49e3d.

  • Fake CLI (rust/examples/fake_grok_acp.rs): a cargo example, so there is no new dependency and no node or python in CI. It speaks newline-delimited JSON-RPC 2.0 on stdio and takes a scenario argument. It writes its pid and whether it has a console window to a pid file before the handshake.
  • Real-process tests (rust/src/providers/grok/acp_process_tests.rs): they go through the production spawn_child, so the stdio, framing, handshake, billing request, parsing, kill and reap all run for real. Scenarios: credits answer in the upstream config shape with subscription_tier; the legacy monthlyLimit shape; noise (non-JSON, notifications, reply to another id, server request reusing the pending id); truncated frames; -32601; auth errors on initialize and on billing; hang on initialize; hang on billing; early exit; half a frame then EOF; a 300 KiB line.
  • Lifecycle checks: every test asserts the child is no longer running after the call returns (killed and reaped). The probe is itself tested: it must report a live child as running.
  • Hidden spawn: the child reports whether it has a console window. The production child reports none. A control child started without CREATE_NO_WINDOW reports one, which shows the probe can detect an inherited console.
  • Fallback path: the missing and expired login cases reach the real child through fetch_cli_acp_without_login. A failing child keeps the original login error. An isolated account never starts the child.
  • Test-only injection: CliBillingRunner::Process (#[cfg(test)]) takes a program path, arguments and budgets. The production runner is unchanged apart from the resolver fix below.

Bugs fixed

  1. Plain grok from PATH. The production runner spawned Command::new("grok"). The native installer puts the CLI under ~/.grok/bin/grok.exe, which is often not on PATH. The fallback then reported NotInstalled even with the CLI installed. It now uses accounts::login::executable(), the resolver the login flow already uses: the native path first, then which. Found by reading the code, not reproduced with a real install.
  2. Server requests taken as replies. reply_for matched on id alone. A message with a method is a request or notification, never a reply. A server request that reused the pending id ended the wait with Parse("Grok CLI reply had no result"). Reproduced: the noise test fails without the fix. Upstream Swift has the same id-only match, so this goes beyond upstream parity.

Windows differences and not ported

  • No real grok CLI was run. Per the user's choice, the tests use the scripted fake only.
  • detect_cli_version (providers/grok/mod.rs, Command::new("grok") --version) still uses PATH only. It is outside the ACP fallback, so it is unchanged here. Follow-up candidate.
  • The resolver change has no unit test: it depends on the user's home directory.

Commands run

  • cargo fmt --all -- --check: clean
  • cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings: clean
  • cargo clippy --manifest-path apps/desktop-tauri/src-tauri/Cargo.toml --all-targets -- -D warnings: clean
  • cargo test --manifest-path rust/Cargo.toml --lib providers::grok: 174 passed, 0 failed (156 existing, 18 new; the probe-control test is Windows-only)
  • cargo test --manifest-path rust/Cargo.toml: 5212 passed, 0 failed, 2 ignored (run before the final test-only edits; the grok filter was rerun after them)
  • cargo test --manifest-path apps/desktop-tauri/src-tauri/Cargo.toml: 756 passed, 0 failed
  • Hidden-console run: the process_tests binary was started with Start-Process -WindowStyle Hidden, which gives the test process a console. The inherited-console control ran (not skipped) and passed, and all 17 process tests in that build passed (the liveness test was added after this run and passed in the later grok run). This is the only run where the hidden-spawn assertion is meaningful, because a shell without a console makes the control skip.
  • Negative check: the noise test without the method filter fails with Parse("Grok CLI reply had no result").

Note: the fake is a cargo example. Run the full cargo test, or cargo build --manifest-path rust/Cargo.toml --example fake_grok_acp first. A --lib filter alone does not build it, and the process tests then fail with a message that says so.

No UI proof: no frontend file changed. Backend only.

Coordinator follow-ups

  • The liveness-probe test now spawns its child with CREATE_NO_WINDOW. Without it, a test runner that has no console would give the child a visible console window.
  • detect_cli_version now uses the same CLI resolver as login and ACP (~/.grok/bin/grok.exe first, then PATH) instead of a bare grok.
  • Re-ran: cargo test --lib providers::grok 174 passed; clippy -D warnings on rust/ clean.

Factory Agent added 4 commits October 21, 2026 15:52
The ACP fallback spawned plain grok from PATH. The native installer puts it under ~/.grok/bin, which is often not on PATH, so the fallback reported NotInstalled. It now uses the same resolver as the login flow.

A message with a method is a server request or notification, never a reply. Matching on id alone let a server request that reused the pending id end the wait with no result.
examples/fake_grok_acp stands in for grok agent stdio. The end-to-end tests spawn it through the production spawn path and cover the handshake, billing, the upstream credits and legacy shapes, noise and garbled frames, JSON-RPC errors, auth errors, early exit, partial frames, oversized lines, and timeouts. Each test checks that the child was started with no console window and is not running afterwards. Provider-level tests cover the missing and expired login fallback and the isolated-account guard. The CLI program is injectable for tests only.
@coderabbitai

coderabbitai Bot commented Oct 21, 2026

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 32 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
  • Configuration used: Repository: nesszer/Win-CodexBar/.coderabbit.yaml
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: 7304d16f-a89e-4313-bec5-2b250f05108f
📥 Commits

Reviewing files that changed from the base of the PR and between 2af7700 and b646e9a.

📒 Files selected for processing: 7
  • docs/PROVIDERS.md
  • rust/examples/fake_grok_acp.rs
  • rust/src/providers/grok/accounts.rs
  • rust/src/providers/grok/acp.rs
  • rust/src/providers/grok/acp_process_tests.rs
  • rust/src/providers/grok/acp_tests.rs
  • rust/src/providers/grok/mod.rs
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@Finesssee
Finesssee merged commit c2ca52a into main Oct 21, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant