From a1c5bba699877977d99efde58e52556885d041bf Mon Sep 17 00:00:00 2001 From: Codev contributors Date: Fri, 9 Oct 2026 20:02:37 -0400 Subject: [PATCH] Exercise MCP Auto calls in Linux and macOS previews --- DESIGN.md | 2 + docs/release-plan-2026-10.md | 2 +- scripts/smoke-packaged-linux-app.sh | 85 ++++++++++++++++++- scripts/smoke-packaged-macos-app.sh | 123 +++++++++++++++++++++++++++- 4 files changed, 209 insertions(+), 3 deletions(-) diff --git a/DESIGN.md b/DESIGN.md index 75f53dcf..6a7d14da 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -596,6 +596,8 @@ Findings that change the backlog: - **V4b Auto-mode disclosure review (2026-10-03, `1729220`).** Adversarial review confirmed that Auto mode also approves configured MCP tool calls, not only shell commands. The footer tooltip, command-permissions explainer, and Code task system prompt now state that MCP tools run without approval unless an exact project server/tool deny rule blocks them, and warn that they may affect external services. Regression checks cover the tooltip and prompt wording. Windows CI [#37148187560](https://github.com/jay23606/codev/actions/runs/37148187560) passed 936 portable tests (2 opt-in skips), 35 Avalonia headless UI tests on Windows/Linux/macOS, 8 Windows UI tests, and native credential-vault smoke checks. Desktop preview [#37148187564](https://github.com/jay23606/codev/actions/runs/37148187564) passed app/CLI build, launch, and packaging on all four RIDs; publishing was skipped for the PR. This review found no additional policy-routing bypass, but it does not satisfy Q3's full manual security review. - **V4c. Real-use log (first pass, 2026-10-09).** The first Codev dogfood reports have been read and triaged in [`docs/real-use-log-release-2.md`](docs/real-use-log-release-2.md). They map to B1 (Auto prompts), B15 (local-model cold load), P2/P6 (saved hosted-key reconnect), collapsed activity/readability, sidebar ordering, and the existing Windows hidden-process follow-up. The [PR #62 Windows package preview #37991522656](https://github.com/jay23606/codev/actions/runs/37991522656) ran the reported deletion-plus-Git-status command in Auto without a prompt, then verified an exact Deny after switching back to Auto; PR #65 records the evidence. Confirm the behavior again after the public Release 2 install; native manual items remain open. +- **V4d. Release 2 cross-platform MCP preview coverage (2026-10-09, packaged run pending).** The existing Linux and macOS packaged-app scripts now configure a loopback Streamable HTTP MCP server and the deterministic mock-model MCP turn. They assert tool discovery, the expected echo call, the bounded untrusted result path, and a completed Auto turn; cleanup tracks the mock server process. Git Bash syntax validation, embedded Python compilation (32 blocks), Node syntax checks, and `git diff --check` pass. The scheduled October 11 four-RID preview is the first runtime check of these additions. Until that run passes, this is coverage added, not a verified cross-platform runtime result; native macOS/Linux interaction remains separately open. + **Done when:** V4a stays green, the checklist is committed and has been run at least once, and the first real-use log has been read and its items triaged. #### V5. Measure whether Codev's own features help on real, large projects diff --git a/docs/release-plan-2026-10.md b/docs/release-plan-2026-10.md index 1c703e82..1994860d 100644 --- a/docs/release-plan-2026-10.md +++ b/docs/release-plan-2026-10.md @@ -42,7 +42,7 @@ The Release 1 ledger leaves 65 manual workflows open, including native macOS/Lin **Release 2 checkpoint (verified 2026-10-09 at main `80c911e`; product code through PR #77, documentation through PR #80).** Subsequent product changes include the exact-Deny Auto regression, persisted sidebar reorder, profile and hosted-key fixes, and a clarification in the Auto/MCP tooltips when saved MCP permissions cannot be read. PR #77's Windows/Linux/macOS CI [#38003412218](https://github.com/jay23606/codev/actions/runs/38003412218) and post-merge CI [#38003676335](https://github.com/jay23606/codev/actions/runs/38003676335) passed. PR #79 documented the Auto/profile/MCP denial-composition review and focused test evidence in `DESIGN.md`; its Windows/Linux/macOS CI [#38004723785](https://github.com/jay23606/codev/actions/runs/38004723785) and post-merge CI [#38004977342](https://github.com/jay23606/codev/actions/runs/38004977342) passed. PR #80 refreshed this checkpoint; its main-branch Windows/Linux/macOS CI [#38005530614](https://github.com/jay23606/codev/actions/runs/38005530614) passed. PR #74 added the eight-archive prepublish gate; PR #76 changed the release preview workflow to run packaged interaction checks only on manual dispatch or version-tag pushes. Their PR CI [#38001121514](https://github.com/jay23606/codev/actions/runs/38001121514) [#38002345137](https://github.com/jay23606/codev/actions/runs/38002345137) and post-merge CI [#38001372222](https://github.com/jay23606/codev/actions/runs/38001372222) [#38002651235](https://github.com/jay23606/codev/actions/runs/38002651235) passed. The last exact-head packaged preview remains PR #62 [#37991522656](https://github.com/jay23606/codev/actions/runs/37991522656); run one new four-target preview by manual dispatch against the frozen October 11 candidate. Do not run packaged previews for ordinary PRs or local desktop smokes during routine work. Clean-machine installation, physical accessibility, and native macOS/Linux interaction remain explicit limitations. Keep the October 11 freeze and October 13 Release 2 target below. -Target Release 2 for **Tuesday, October 13, 2026** (Day 5 after the October 8 publication; acceptable window: Day 3–5). On October 9–10 (Days 1–2), collect install reports, triage reproducible issues, and fix release blockers. On October 11 (Day 3), freeze the candidate and run exact-head Windows/Linux/macOS CI plus all four packaged desktop/CLI checks by manually dispatching the desktop-preview workflow once against the frozen main commit. The workflow runs only on manual dispatch and version-tag pushes; pull requests and ordinary merges use the Windows/Linux/macOS CI workflow without packaged interaction smokes. Use a local desktop smoke only if a release-critical interaction remains uncovered by hosted checks and focused tests; routine local runs stay disabled. Publish on October 12–13 only if exact-head checks pass, update the release notes for verified changes and remaining limitations, and confirm all eight README latest-download links. If a blocker moves the candidate, move Release 2 to Day 3–5 after the verified fix. +Target Release 2 for **Tuesday, October 13, 2026** (Day 5 after the October 8 publication; acceptable window: Day 3–5). On October 9–10 (Days 1–2), collect install reports, triage reproducible issues, and fix release blockers. On October 11 (Day 3), freeze the candidate and run exact-head Windows/Linux/macOS CI plus all four packaged desktop/CLI checks by manually dispatching the desktop-preview workflow once against the frozen main commit. The four-RID preview will also run the new Linux/macOS Streamable HTTP MCP Auto scenarios, which were added to the existing packaged checks without changing the preview cadence. The workflow runs only on manual dispatch and version-tag pushes; pull requests and ordinary merges use the Windows/Linux/macOS CI workflow without packaged interaction smokes. Use a local desktop smoke only if a release-critical interaction remains uncovered by hosted checks and focused tests; routine local runs stay disabled. Publish on October 12–13 only if exact-head checks pass, update the release notes for verified changes and remaining limitations, and confirm all eight README latest-download links. If a blocker moves the candidate, move Release 2 to Day 3–5 after the verified fix. **Earlier Release 2 candidate evidence (October 9, code head `e729b85`).** The four-target preview [#37885221048](https://github.com/jay23606/codev/actions/runs/37885221048) passed before the later Release 2 fixes. Subsequent PR #35 removed packaged previews from regular pushes to `main`; PR #76 later removed them from pull requests as well. This evidence is historical and does not validate current main `7cb7d2b`; the exact current candidate still needs the one manual four-target preview scheduled above. The prior main CI's intermittent Linux headless review-test failure was fixed layout-synchronously in PR #34. Treat Release 2 scope as stabilization and recheck [`docs/release-notes.md`](release-notes.md) against the exact release commit before tagging. diff --git a/scripts/smoke-packaged-linux-app.sh b/scripts/smoke-packaged-linux-app.sh index 6975db7b..fae67604 100644 --- a/scripts/smoke-packaged-linux-app.sh +++ b/scripts/smoke-packaged-linux-app.sh @@ -15,6 +15,7 @@ export CODEV_DATA_ROOT="$smoke_root/data" log_path="$smoke_root/app.log" app_pid='' mock_pid='' +mcp_http_pid='' cleanup() { if [[ -n "$app_pid" ]] && kill -0 "$app_pid" 2>/dev/null; then kill "$app_pid" 2>/dev/null || true @@ -26,6 +27,7 @@ cleanup() { fi if [[ -n "$app_pid" ]]; then wait "$app_pid" 2>/dev/null || true; fi if [[ -n "$mock_pid" ]]; then kill "$mock_pid" 2>/dev/null || true; wait "$mock_pid" 2>/dev/null || true; fi + if [[ -n "$mcp_http_pid" ]]; then kill "$mcp_http_pid" 2>/dev/null || true; wait "$mcp_http_pid" 2>/dev/null || true; fi rm -rf -- "$smoke_root" } trap cleanup EXIT @@ -38,7 +40,7 @@ assert_search_focus() { mock_port_path="$smoke_root/mock-ollama.port" mock_request_log="$smoke_root/mock-ollama-requests.jsonl" -node ./scripts/mock-ollama-server.js --activity-summary --port-file "$mock_port_path" --request-log "$mock_request_log" >"$smoke_root/mock-ollama.log" 2>&1 & +node ./scripts/mock-ollama-server.js --activity-summary --mcp-http --port-file "$mock_port_path" --request-log "$mock_request_log" >"$smoke_root/mock-ollama.log" 2>&1 & mock_pid=$! for _ in {1..200}; do if [[ -s "$mock_port_path" ]]; then break; fi @@ -54,8 +56,40 @@ for _ in {1..200}; do done if [[ ! -s "$mock_port_path" ]]; then cat "$smoke_root/mock-ollama.log" >&2; echo 'Mock Ollama did not report its loopback port within 20 seconds.' >&2; exit 1; fi mock_port="$(cat "$mock_port_path")" +mcp_http_port_path="$smoke_root/mock-mcp-http.port" +mcp_http_call_log="$smoke_root/mock-mcp-http-calls.jsonl" +node ./scripts/mock-mcp-http-server.js --port-file "$mcp_http_port_path" --call-log "$mcp_http_call_log" >"$smoke_root/mock-mcp-http.log" 2>&1 & +mcp_http_pid=$! +for _ in {1..200}; do + if [[ -s "$mcp_http_port_path" ]]; then break; fi + mcp_http_state="$(ps -p "$mcp_http_pid" -o stat= 2>/dev/null || true)" + if [[ -z "$mcp_http_state" || "$mcp_http_state" == Z* ]]; then + if wait "$mcp_http_pid"; then mcp_http_status=0; else mcp_http_status=$?; fi + mcp_http_pid='' + cat "$smoke_root/mock-mcp-http.log" >&2 + echo "Mock MCP HTTP server exited before startup completed (exit $mcp_http_status)." >&2 + exit 1 + fi + sleep 0.1 +done +if [[ ! -s "$mcp_http_port_path" ]]; then cat "$smoke_root/mock-mcp-http.log" >&2; echo 'Mock MCP HTTP server did not report its loopback port within 20 seconds.' >&2; exit 1; fi +mcp_http_port="$(cat "$mcp_http_port_path")" mkdir -p "$CODEV_DATA_ROOT/Codev" printf '{"Theme":"dark","OllamaEndpoint":"http://127.0.0.1:%s/","DefaultProjectCommandPermissionMode":"Auto"}\n' "$mock_port" >"$CODEV_DATA_ROOT/Codev/avalonia-settings.json" +python3 - "$CODEV_DATA_ROOT/Codev/mcp-servers.json" "$mcp_http_port" <<'PY' +import json +import sys + +with open(sys.argv[1], "w", encoding="utf-8") as target: + json.dump([{ + "id": "smoke-http", + "name": "Packaged smoke HTTP MCP", + "transport": "Http", + "enabled": True, + "url": f"http://127.0.0.1:{sys.argv[2]}/mcp", + "oauthEnabled": False, + }], target) +PY "$app_path" >"$log_path" 2>&1 & app_pid=$! @@ -422,6 +456,55 @@ PY echo 'Linux packaged Code task read, searched, created a file, and verified a command in one Auto turn.' +# Exercise configured Streamable HTTP MCP discovery and execution in Auto. +mcp_prompt='Run the packaged MCP Streamable HTTP smoke.' +before_mcp_count="$(jq --arg id "$conversation_id" '[.[] | select(.Id == $id) | .Messages[]] | length' "$conversations_path")" +xdotool windowfocus --sync "$window_id" +xdotool key --clearmodifiers ctrl+l +xdotool type --clearmodifiers --delay 1 "$mcp_prompt" +xdotool key --clearmodifiers Return + +for _ in {1..300}; do + if jq -e --arg id "$conversation_id" --argjson before "$before_mcp_count" --arg prompt "$mcp_prompt" \ + '.[] | select(.Id == $id) | .Messages as $messages | + ($messages | length) >= ($before + 2) and + $messages[-2].Content == $prompt and + $messages[-1].Role == "assistant" and + $messages[-1].Content == "Packaged MCP Streamable HTTP call passed."' \ + "$conversations_path" >/dev/null 2>&1; then break; fi + if ! kill -0 "$app_pid" 2>/dev/null; then cat "$log_path" >&2; echo 'Packaged Avalonia app exited during the Linux MCP HTTP smoke.' >&2; exit 1; fi + sleep 0.1 +done + +python3 - "$conversations_path" "$conversation_id" "$mock_request_log" "$mcp_http_call_log" <<'PY' +import json +import sys + +prompt = "Run the packaged MCP Streamable HTTP smoke." +with open(sys.argv[1], encoding="utf-8") as source: + conversations = json.load(source) +conversation = next((item for item in conversations if item.get("Id") == sys.argv[2]), None) +messages = conversation.get("Messages", []) if conversation else [] +if not any(message.get("Role") == "user" and message.get("Content") == prompt for message in messages): + raise SystemExit("The packaged Linux MCP prompt was not persisted.") +if not messages or messages[-1].get("Role") != "assistant" or messages[-1].get("Content") != "Packaged MCP Streamable HTTP call passed.": + raise SystemExit(f"The packaged Linux MCP call did not finish: {messages[-4:]!r}") +with open(sys.argv[3], encoding="utf-8") as source: + requests = [json.loads(line) for line in source if line.strip()] +turn = [request for request in requests if request.get("last_user_message") == prompt] +if len(turn) != 2 or turn[0].get("last_role") != "user" or \ + len([name for name in turn[0].get("tool_names", []) if name.startswith("mcp_smoke-http_echo_")]) != 1 or \ + turn[1].get("last_role") != "tool" or not turn[1].get("last_tool_name", "").startswith("mcp_smoke-http_echo_") or \ + turn[1].get("keep_alive") != "30m": + raise SystemExit(f"The packaged Linux MCP discovery/result sequence was unexpected: {turn!r}") +with open(sys.argv[4], encoding="utf-8") as source: + calls = [json.loads(line) for line in source if line.strip()] +if calls != [{"name": "echo", "message": "packaged HTTP marker"}]: + raise SystemExit(f"The packaged Linux HTTP MCP fixture received an unexpected call: {calls!r}") +PY + +echo 'Linux packaged Code task discovered and called the Streamable HTTP MCP echo tool in Auto without approval.' + # Verify the platform primary-modifier shortcut creates and activates a fresh chat. previous_conversation_id="$(jq -r '.' "$active_path")" xdotool windowfocus --sync "$window_id" diff --git a/scripts/smoke-packaged-macos-app.sh b/scripts/smoke-packaged-macos-app.sh index 726ba7cf..650be361 100644 --- a/scripts/smoke-packaged-macos-app.sh +++ b/scripts/smoke-packaged-macos-app.sh @@ -15,6 +15,7 @@ export CODEV_DATA_ROOT="$smoke_root/data" log_path="$smoke_root/app.log" app_pid='' mock_pid='' +mcp_http_pid='' cleanup() { if [[ -n "$app_pid" ]] && kill -0 "$app_pid" 2>/dev/null; then kill "$app_pid" 2>/dev/null || true @@ -26,6 +27,7 @@ cleanup() { fi if [[ -n "$app_pid" ]]; then wait "$app_pid" 2>/dev/null || true; fi if [[ -n "$mock_pid" ]]; then kill "$mock_pid" 2>/dev/null || true; wait "$mock_pid" 2>/dev/null || true; fi + if [[ -n "$mcp_http_pid" ]]; then kill "$mcp_http_pid" 2>/dev/null || true; wait "$mcp_http_pid" 2>/dev/null || true; fi rm -rf -- "$smoke_root" } trap cleanup EXIT @@ -34,7 +36,7 @@ mock_port_path="$smoke_root/mock-ollama.port" mock_request_log="$smoke_root/mock-ollama-requests.jsonl" model_enabled_path="$smoke_root/mock-ollama-model.enabled" printf '0' >"$model_enabled_path" -node ./scripts/mock-ollama-server.js --activity-summary --port-file "$mock_port_path" --request-log "$mock_request_log" --model-enabled-file "$model_enabled_path" >"$smoke_root/mock-ollama.log" 2>&1 & +node ./scripts/mock-ollama-server.js --activity-summary --mcp-http --port-file "$mock_port_path" --request-log "$mock_request_log" --model-enabled-file "$model_enabled_path" >"$smoke_root/mock-ollama.log" 2>&1 & mock_pid=$! for _ in {1..200}; do if [[ -s "$mock_port_path" ]]; then break; fi @@ -50,8 +52,40 @@ for _ in {1..200}; do done if [[ ! -s "$mock_port_path" ]]; then cat "$smoke_root/mock-ollama.log" >&2; echo 'Mock Ollama did not report its loopback port within 20 seconds.' >&2; exit 1; fi mock_port="$(cat "$mock_port_path")" +mcp_http_port_path="$smoke_root/mock-mcp-http.port" +mcp_http_call_log="$smoke_root/mock-mcp-http-calls.jsonl" +node ./scripts/mock-mcp-http-server.js --port-file "$mcp_http_port_path" --call-log "$mcp_http_call_log" >"$smoke_root/mock-mcp-http.log" 2>&1 & +mcp_http_pid=$! +for _ in {1..200}; do + if [[ -s "$mcp_http_port_path" ]]; then break; fi + mcp_http_state="$(ps -p "$mcp_http_pid" -o stat= 2>/dev/null || true)" + if [[ -z "$mcp_http_state" || "$mcp_http_state" == Z* ]]; then + if wait "$mcp_http_pid"; then mcp_http_status=0; else mcp_http_status=$?; fi + mcp_http_pid='' + cat "$smoke_root/mock-mcp-http.log" >&2 + echo "Mock MCP HTTP server exited before startup completed (exit $mcp_http_status)." >&2 + exit 1 + fi + sleep 0.1 +done +if [[ ! -s "$mcp_http_port_path" ]]; then cat "$smoke_root/mock-mcp-http.log" >&2; echo 'Mock MCP HTTP server did not report its loopback port within 20 seconds.' >&2; exit 1; fi +mcp_http_port="$(cat "$mcp_http_port_path")" mkdir -p "$CODEV_DATA_ROOT/Codev" printf '{"Theme":"dark","OllamaEndpoint":"http://127.0.0.1:%s/","DefaultProjectCommandPermissionMode":"Auto"}\n' "$mock_port" >"$CODEV_DATA_ROOT/Codev/avalonia-settings.json" +python3 - "$CODEV_DATA_ROOT/Codev/mcp-servers.json" "$mcp_http_port" <<'PY' +import json +import sys + +with open(sys.argv[1], "w", encoding="utf-8") as target: + json.dump([{ + "id": "smoke-http", + "name": "Packaged smoke HTTP MCP", + "transport": "Http", + "enabled": True, + "url": f"http://127.0.0.1:{sys.argv[2]}/mcp", + "oauthEnabled": False, + }], target) +PY "$app_path" >"$log_path" 2>&1 & app_pid=$! @@ -894,6 +928,93 @@ PY echo 'macOS packaged Code task read, searched, created a file, and verified a command in one Auto turn.' +# Exercise configured Streamable HTTP MCP discovery and execution in Auto. +mcp_prompt='Run the packaged MCP Streamable HTTP smoke.' +before_mcp_count="$(python3 - "$conversations_path" "$conversation_id" <<'PY' +import json +import sys + +with open(sys.argv[1], encoding="utf-8") as source: + conversations = json.load(source) +conversation = next(item for item in conversations if item.get("Id") == sys.argv[2]) +print(len(conversation.get("Messages", []))) +PY +)" +osascript - "$app_pid" "$mcp_prompt" <<'APPLESCRIPT' +on run argv + set targetPid to item 1 of argv as integer + set promptText to item 2 of argv as text + tell application "System Events" + set targetProcess to first process whose unix id is targetPid + set frontmost of targetProcess to true + delay 0.25 + key code 37 using {command down} + delay 0.5 + keystroke promptText + end tell +end run +APPLESCRIPT +await_saved_draft "$mcp_prompt" "the MCP Streamable HTTP prompt" +osascript - "$app_pid" <<'APPLESCRIPT' +on run argv + set targetPid to item 1 of argv as integer + tell application "System Events" + set targetProcess to first process whose unix id is targetPid + set frontmost of targetProcess to true + key code 36 + end tell +end run +APPLESCRIPT + +for _ in {1..300}; do + if python3 - "$conversations_path" "$conversation_id" "$before_mcp_count" "$mcp_prompt" <<'PY' +import json +import sys + +with open(sys.argv[1], encoding="utf-8") as source: + conversations = json.load(source) +conversation = next((item for item in conversations if item.get("Id") == sys.argv[2]), None) +messages = conversation.get("Messages", []) if conversation else [] +before = int(sys.argv[3]) +success = (len(messages) >= before + 2 and messages[-2].get("Content") == sys.argv[4] and + messages[-1].get("Role") == "assistant" and + messages[-1].get("Content") == "Packaged MCP Streamable HTTP call passed.") +raise SystemExit(0 if success else 1) +PY + then break; fi + if ! kill -0 "$app_pid" 2>/dev/null; then cat "$log_path" >&2; echo 'Packaged Avalonia app exited during the macOS MCP HTTP smoke.' >&2; exit 1; fi + sleep 0.1 +done + +python3 - "$conversations_path" "$conversation_id" "$mock_request_log" "$mcp_http_call_log" <<'PY' +import json +import sys + +prompt = "Run the packaged MCP Streamable HTTP smoke." +with open(sys.argv[1], encoding="utf-8") as source: + conversations = json.load(source) +conversation = next((item for item in conversations if item.get("Id") == sys.argv[2]), None) +messages = conversation.get("Messages", []) if conversation else [] +if not any(message.get("Role") == "user" and message.get("Content") == prompt for message in messages): + raise SystemExit("The packaged macOS MCP prompt was not persisted.") +if not messages or messages[-1].get("Role") != "assistant" or messages[-1].get("Content") != "Packaged MCP Streamable HTTP call passed.": + raise SystemExit(f"The packaged macOS MCP call did not finish: {messages[-4:]!r}") +with open(sys.argv[3], encoding="utf-8") as source: + requests = [json.loads(line) for line in source if line.strip()] +turn = [request for request in requests if request.get("last_user_message") == prompt] +if len(turn) != 2 or turn[0].get("last_role") != "user" or \ + len([name for name in turn[0].get("tool_names", []) if name.startswith("mcp_smoke-http_echo_")]) != 1 or \ + turn[1].get("last_role") != "tool" or not turn[1].get("last_tool_name", "").startswith("mcp_smoke-http_echo_") or \ + turn[1].get("keep_alive") != "30m": + raise SystemExit(f"The packaged macOS MCP discovery/result sequence was unexpected: {turn!r}") +with open(sys.argv[4], encoding="utf-8") as source: + calls = [json.loads(line) for line in source if line.strip()] +if calls != [{"name": "echo", "message": "packaged HTTP marker"}]: + raise SystemExit(f"The packaged macOS HTTP MCP fixture received an unexpected call: {calls!r}") +PY + +echo 'macOS packaged Code task discovered and called the Streamable HTTP MCP echo tool in Auto without approval.' + # Exercise Command+Shift+F with a long transcript, choose the earlier # Auto-command prompt, and close the search panel with Escape. find_query='Run the packaged Auto mode command smoke.'