| Wall | What actually happens | |
|---|---|---|
| A full-screen app hides everything else | measured | A full-screen window gets its own Space, and both the accessibility API and the screenshot see only the Space in front. Measured: five windows on screen, all belonging to one full-screen editor, while four other apps reported zero. This cost us half a day and it looks exactly like a broken lookup. How to tell. |
| Electron apps | measured | Chromium builds its tree lazily and an app can ship without enabling it - a bare accessibility call returned 8 nodes and no buttons at depth 14, 30 and 60 alike. This server switches the tree on itself before it looks. Measured on a VS Code fork with two windows open: 0 buttons before, 728 after, same windows, same moment. Not re-measurable on a live process, because the switch cannot be turned off again - use CMCP_INGEN_ELECTRON=1 on a fresh one. Chrome is a separate case: the attribute is an Electron API and Chrome does not implement it. Measured: 29 buttons before and after. You get the browser window, not the page in it. |
| Electron apps | measured | Chromium builds its tree lazily and an app can ship without enabling it - a bare accessibility call returned 8 nodes and no buttons at depth 14, 30 and 60 alike. This server switches the tree on itself before it looks. Measured on a VS Code fork with two windows open: 0 buttons before, 728 after, same windows, same moment. Not re-measurable on a live process, because the switch cannot be turned off again - use CMCP_INGEN_ELECTRON=1 on a fresh one. Chrome is a separate case: the attribute is an Electron API and Chrome does not implement it (measured: 29 buttons before and after). Chrome's own switch is the one a screen reader sets, and this server sets that one: measured 5 Nov on Chrome 156, a test page went from 0 page elements to 6, its password field among them and blacked out. Chrome builds the page 3 to 15 seconds after it is first switched on; until then an empty search says tree_building and screenshots black that window out whole. Only Google Chrome is measured; Chrome app windows (PWAs) are not covered. |
| A window on another Space | not yet | computer_screenshot with an app name fails when that app's window is on a different Space. The error now says so instead of claiming the app is not running. |
| App-scoped screenshots | measured | Measured 19/9: asking for one app's window fails for four of five apps with a ScreenCaptureKit streaming error, in the same second that a full-screen capture succeeds. Root cause not found. Full-screen capture works; use it and crop. |
| Several displays | measured | Measured 19 Sep 2026 on a Mac with three. A screenshot captures one display, and until that day it always captured display 0 without saying so, which meant two thirds of the desktop was invisible with no error. Fixed: computer_screenshot takes a display index, and every answer now reports how many displays exist and which one it took, so an agent that cannot find a window knows where else to look. The earlier row here said "this machine has one" - it had three, and nobody had checked. |
| Wall | What actually happens | |
|---|---|---|
| Redacting secrets | measured | Only what macOS marks as secure is painted out. Measured on a real web page: a password input carries role AXTextField with subrole AXSecureTextField and is redacted; a plain text field is not. A password in a text editor, or a token printed in a terminal, carries no marker and will not be hidden. Use readonly when the screen holds something the system cannot know about. |
| Working without taking over the screen | measured | Three kinds, and the difference is how the action is delivered, not what the tool is called. Quiet by delivery: computer_press, computer_set_value and computer_menu fire the element's own accessibility action, so they work on a window behind the one you are in and never touch your pointer. The app can still bring itself forward in response - measured: File > New Finder Window did - and then press and menu answer took_screen and hand the front straight back. Quiet when you name the app: computer_click, computer_type, computer_key and computer_scroll are delivered into that app's own event queue - measured: the text arrived, the pointer stayed where it was left, and the front window did not change. computer_launch joins them with background: true - measured on a clean Mac, six of eight apps brought themselves forward as they started, and the front is handed straight back. Without an app they go through the system input tap and land wherever your keyboard focus is, so in background mode they are refused. One measured limit: keystrokes do not land in a Chromium window that is not focused, so for anything Chromium-based use computer_press and computer_set_value. Held back in background mode: computer_move, computer_drag, computer_paste, computer_activate, computer_quit, computer_space and computer_window. Each either moves your pointer, brings something to the front, or changes what you see - a window moved on a screen you are looking at is a quiet channel with a very visible result. computer_ask_user left this list on 29 September: in background mode its question waits in the menu bar icon instead of a dialog. And the held-back seven are no longer out of reach: computer_request_screen asks you to lend this one agent the screen, approved with Touch ID in the menu bar, for at most 15 minutes. While it has it, each step waits while you are using the keyboard or mouse (a step already running finishes), and one click in the menu takes the screen back and stops a step still running. Set CMCP_BACKGROUND and nothing can borrow it. |
| Working without taking over the screen | measured | Three kinds, and the difference is how the action is delivered, not what the tool is called. Quiet by delivery: computer_press, computer_set_value and computer_menu fire the element's own accessibility action, so they work on a window behind the one you are in and never touch your pointer. The app can still bring itself forward in response - measured: File > New Finder Window did - and then press and menu answer took_screen and hand the front straight back - gave_back says whether that worked (once in 65 moments on GitHub’s Mac it did not). Quiet when you name the app: computer_click, computer_type, computer_key and computer_scroll are delivered into that app's own event queue - measured: the text arrived, the pointer stayed where it was left, and the front window did not change. computer_launch joins them with background: true - measured on a clean Mac, six of eight apps brought themselves forward as they started, and the front is handed straight back - gave_back says whether that worked. Without an app they go through the system input tap and land wherever your keyboard focus is, so in background mode they are refused. One measured limit: keystrokes do not land in a Chromium window that is not focused, so for anything Chromium-based use computer_press and computer_set_value. Held back in background mode: computer_move, computer_drag, computer_paste, computer_activate, computer_quit, computer_space and computer_window. Each either moves your pointer, brings something to the front, or changes what you see - a window moved on a screen you are looking at is a quiet channel with a very visible result. computer_ask_user left this list on 29 September: in background mode its question waits in the menu bar icon instead of a dialog. And the held-back seven are no longer out of reach: computer_request_screen asks you to lend this one agent the screen, approved with Touch ID in the menu bar, for at most 15 minutes. While it has it, each step waits while you are using the keyboard or mouse (a step already running finishes), and one click in the menu takes the screen back and stops a step still running. Set CMCP_BACKGROUND and nothing can borrow it. |
| Several agents at once | measured | Each MCP client runs its own server. Measured: two servers, 50 interleaved writes, zero torn lines, and every line carries a session mark. Writes in the same app take turns: each holds a lock for that app, so keys and clicks never interleave within an action. Between actions they still share one pointer and one focused window; give background work to computer_press. |
| Knowing where the cursor is | measured | computer_focused reports the focused element and whether it is secure. The system-wide lookup alone returns nothing - measured - so it goes through the focused application, then the frontmost app. |
| A Save / Don't Save sheet | measured | The sheet you meet every time you close something with unsaved work. Measured: the sheet is found as AXSheet, all three buttons come back with a center and a pressable flag, and computer_find with contains: "Don't Save" returns exactly that one button - an agent looks for the words, not for a position. It is also the clearest case for the gate: a sheet can throw away work on your behalf, so in read-only mode pressing it is refused and nothing reaches the machine. Measured against a sheet the test puts up itself, in a fully transparent window, so proving it costs nobody their screen. |
| maxSeconds | start only. It stops by itself after this long. Default 600, at most 3600. | |
| displayId | start only. Which display, by the id from computer_displays. Default: the main one. |
Records one display to a video file, for a person to watch afterwards - a tutorial, a demo, proof of what was done. Starting asks you every time: it is your screen, for minutes. In read-only mode it is refused. Stopping and asking for the status never ask. Only password managers are left out of the recording, including ones opened while it runs (the list is checked about four times a second, so one opened mid-recording can be visible for a moment; if the check cannot be made for two seconds, or a password manager cannot be kept out, the recording stops rather than keep filming). Everything else is recorded as it looks: password fields are not blacked out the way they are in screenshots, and anything typed into an ordinary field is visible. The answer is the file's path - this server never reads the file back and never sends it to the model. It stops on its own at maxSeconds, and when the server that started it ends - if your MCP client quits, that can take up to about half a minute. A server that hangs rather than ends does not count: the recording then runs to maxSeconds, and macOS's own recording indicator is what shows it. Needs macOS 15, and macOS shows its own recording indicator in the menu bar while it runs.
Records one display to a video file, for a person to watch afterwards - a tutorial, a demo, proof of what was done. Starting asks you every time: it is your screen, for minutes. In read-only mode it is refused. Stopping and asking for the status never ask. Only password manager apps are left out of the recording, including ones opened while it runs (the list is checked about four times a second, so one opened mid-recording can be visible for a moment; if the check cannot be made for two seconds, or a password manager cannot be kept out, the recording stops rather than keep filming). Everything else is recorded as it looks: a password manager page inside a browser is recorded, password fields are not blacked out the way they are in screenshots, and anything typed into an ordinary field is visible. The answer is the file's path - this server never reads the file back and never sends it to the model. It stops on its own at maxSeconds, and when the server that started it ends - if your MCP client quits, that can take up to about half a minute. A server that hangs rather than ends does not count: the recording then runs to maxSeconds, and macOS's own recording indicator is what shows it. Needs macOS 15, and macOS shows its own recording indicator in the menu bar while it runs.
Wait until an element appears, then return. The alternative is a screenshot loop, and the arithmetic is stark: twenty polls cost one tool call here and twenty images the other way - twenty pictures of a progress bar, in the model's context.
diff --git a/gemini-extension.json b/gemini-extension.json index 6805b233..c78cae35 100644 --- a/gemini-extension.json +++ b/gemini-extension.json @@ -1,6 +1,6 @@ { "name": "computer-mcp", - "version": "0.4.0", + "version": "0.4.1", "description": "Drive macOS from any MCP client with the guardrails on by default: password fields blacked out in memory before the screenshot is written, password managers that are refused and terminals that ask once per session, and a log the server only appends to that never stores typed text in clear. 32 tools, 14 read-only. No API key, MIT.", "mcpServers": { "computer": { diff --git a/helper/Sources/cmcp-helper/Accessibility.swift b/helper/Sources/cmcp-helper/Accessibility.swift index 948094f5..f802f653 100644 --- a/helper/Sources/cmcp-helper/Accessibility.swift +++ b/helper/Sources/cmcp-helper/Accessibility.swift @@ -57,17 +57,172 @@ enum AX { // Standarden er uaendret: traeet taendes. static let taendTraeSlaaetFra = ProcessInfo.processInfo.environment["CMCP_INGEN_ELECTRON"] == "1" + /// ⛔ MAALT 4/11-2026: Google Chrome KENDER IKKE `AXManualAccessibility`. + /// System Events: «Kan ikke hente attribute "AXManualAccessibility" of + /// application process "Google Chrome"» (-1728). Kontakten ovenfor er en + /// Electron-opfindelse; vi satte den paa Chrome ved hvert kald, og Chrome + /// ignorerede den. Resultatet var et vindue uden web-omraade: 185 elementer + /// laest i to Chrome-vinduer - faner, adresselinje, knapper - og NUL fra selve + /// siderne. Saa kunne vi hverken taste i en side (computer_type: focus-unknown) + /// eller se et adgangskodefelt i den, og det sidste betyder at sloeringen + /// af kodeord i Chrome-sider ikke kunne holde, hvad forsiden lover. + /// + /// Chromes egen kontakt er `AXEnhancedUserInterface` - den VoiceOver saetter. + /// Maalt paa en engangs-profil af Chrome 156 med en testside: + /// foer: computer_find «CMCPAXTEST» -> 0 traeffere (titlen fandtes, siden ikke) + /// efter: 6 - overskrift, link, tekstfelt MED sin vaerdi, og kodefeltet som + /// AXSecureTextField, secure: true + /// Og kun dér: den anden Chrome-proces paa maskinen stod uroert paa false. + /// + /// Den saettes KUN paa Chromium-browsere. Paa almindelige AppKit-programmer har + /// den en kendt bivirkning (vinduesflytninger animeres), og dem ser vi i + /// forvejen uden den. Kun Google Chrome er maalt; de andre er samme motor. + static let chromiumBrowsere: Set${M}-TEKST
+`); +writeFileSync(join(mappe, 'side.html'), `hovedside
`); +// Startes i BAGGRUNDEN (open -g): en direkte start hentede Chrome frem foran personen (5/11). +const chrome = await startChrome([`--user-data-dir=${join(mappe, 'profil')}`, '--no-first-run', '--no-default-browser-check', + '--disable-extensions', '--disable-popup-blocking', '--disable-session-crashed-bubble', '--window-position=60,90', + '--window-size=600,420', `file://${join(mappe, 'side.html')}`]); +const ryd = () => { chrome.stop(); try { rmSync(mappe, { recursive: true, force: true }); } catch {} }; +process.on('exit', ryd); + +try { + // Vent paa popup-vinduet (titlen, Chromes egen brugerflade) UDEN at taende traeet. + let pop = null; + for (let i = 0; i < 60 && !pop; i++) { + await vent(250); + // `find` (ikke `windows`) giver underrollen med. + const w = hj(['find', '--app', 'com.google.Chrome', '--role', 'AXWindow', '--contains', `${M}-POP`, '--depth', '3'], { CMCP_INGEN_ELECTRON: '1' }); + pop = (w.matches || [])[0] || null; + } + check('0. popup-vinduet er aabnet', !!pop, 'Chrome viste intet popup paa 15 s'); + // 5/11: maalt MENS proevevinduet er aabent (efter lukning ville fokus vende tilbage og skjule det). + // Paa processen (ASN), ikke navnet - personens Chrome og proevens hedder begge «Google Chrome». + // 0b (oplysning, ikke et tjek): MAALT 5/11 - Chrome tager forreste plads trods open -g; derfor inaktivitetsvagten. + console.log(` forreste proces foer/efter start: ${chrome.forrestFoer} / ${chrome.forrestEfter()} (${chrome.forrestNavn()})`); + if (pop) { + const f = pop.frame || {}; + console.log(` popup-vinduets underrolle: ${pop.subrole ?? '(ikke i svaret)'} · ramme ${JSON.stringify(f)}`); + const iPop = (r) => r.x >= f.x - 4 && r.y >= f.y - 4 && r.x + r.w <= f.x + f.w + 4 && r.y + r.h <= f.y + f.h + 4; + const helePop = (r) => Math.abs(r.w - f.w) <= 8 && Math.abs(r.h - f.h) <= 8; + + const r1 = hj(['secure-rects', '--app', 'com.google.Chrome']); + check('1. kaldet der taender Chrome, svaerter popup-vinduet helt', r1.rects.filter(iPop).some(helePop), + `rekt ${JSON.stringify(r1.rects.filter(iPop))} grunde ${JSON.stringify(r1.whole_window)}`); + + let fnd = { count: 0 }; + for (let i = 0; i < 60 && !fnd.count; i++) { await vent(500); fnd = hj(['find', '--app', 'com.google.Chrome', '--contains', `${M}-NAVN`, '--depth', '60']); } + check('2. popup-sidens felt ses', fnd.count > 0, 'ikke fundet paa 30 s'); + + const r3 = hj(['secure-rects', '--app', 'com.google.Chrome']).rects.filter(iPop); + check('3. kun kodefeltet i popup\'en svaertes, ikke hele vinduet', + r3.length >= 1 && !r3.some(helePop) && r3.some((r) => r.w < 400 && r.h < 60), JSON.stringify(r3)); + } +} finally { + check('Z. proevens Chrome er lukket igen', chrome.stop(), 'den lever stadig - naeste proeve ville maale dens vinduer'); + ryd(); +} +process.exit(slutDom(fejl, 5, START_MS)); diff --git a/test/chrome-side-live.mjs b/test/chrome-side-live.mjs new file mode 100644 index 00000000..57f01f8c --- /dev/null +++ b/test/chrome-side-live.mjs @@ -0,0 +1,127 @@ +// Chrome-sider gennem den SENDTE hjaelper - adfaerd, ikke tekst. +// +// ⛔ REVIEW 5/11-2026 (Opus): claims-offline 8b-8e tjekker at strengene STAAR i kilden. Syv af +// otte meningsfulde mutationer forblev groenne - ogsaa «porten svaerter intet» og «kontakten +// saettes aldrig». En proeve der ikke kan blive roed af det den hedder, beviser ingenting. +// Denne kalder hjaelperen mod en rigtig Chrome og maaler det der skal vaere sandt: +// +// 1. kaldet der TAENDER Chromes trae, svarer tree_building (ikke et stumt 0) +// 2. mens siden bygges, svaertes HELE vinduet (fail-closed) - kodefeltet kan ikke ses endnu +// 3. naar siden er bygget, ses den: tekstfeltet findes +// 4. og saa svaertes KUN kodefeltet, ikke hele vinduet +// 5. skrivning i tekstfeltet bekraeftes, laest tilbage +// 6. skrivning i kodefeltet naegtes (secure-field) +// +// Koeres bevidst: `node test/chrome-side-live.mjs`. Den roerer hverken mus eller tastatur, men den +// aabner et lille Chrome-vindue med en engangs-profil i et par sekunder, og hjaelperen taender +// Chromes trae ogsaa i dit eget Chrome (det er produktets adfaerd). Uden Google Chrome: UMAALT. +import { execFileSync } from 'node:child_process'; +import { startChrome, inaktivSek, MIN_INAKTIV, slutDom } from './chrome-start.mjs'; +const START_MS = Date.now(); +import { existsSync, mkdtempSync, writeFileSync, rmSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join, dirname } from 'node:path'; +import { fileURLToPath } from 'node:url'; + +const ROOT = join(dirname(fileURLToPath(import.meta.url)), '..'); +const H = process.env.CMCP_HELPER || join(ROOT, 'mcp-server', 'vendor', 'cmcp-helper'); +const CHROME = '/Applications/Google Chrome.app/Contents/MacOS/Google Chrome'; +if (!existsSync(CHROME)) { console.log('SPRUNGET OVER: 8 (Google Chrome er ikke installeret her) - bevist intet, ikke bestaaet'); process.exit(0); } +// ⛔ MAALT 5/11 kl. 02.09: mens Mac'en er LAAST, bygger Chrome ikke siders tilgaengeligheds-trae. +// Et maaleprogram uden hjaelperen fandt 0 web-omraader efter 25,6 s paa en frisk Chrome (laast), +// mod 2,1 s kl. 01.20 (ulaast). Proeven maaler da laasen, ikke koden - saa den siger UMAALT. +{ + const io = execFileSync('ioreg', ['-n', 'Root', '-d1'], { encoding: 'utf8' }); + if (/CGSSessionScreenIsLocked"?\s*=\s*(Yes|true|1)/.test(io)) { console.log('SPRUNGET OVER: 8 (skaermen er laast - Chrome bygger ingen sider imens) - bevist intet, ikke bestaaet'); process.exit(0); } +} +// ⛔ MAALT 5/11 08.21: Chrome henter sig selv frem trods `open -g`. Kun naar personen ikke arbejder. +if (inaktivSek() < MIN_INAKTIV) { console.log(`SPRUNGET OVER: 8 (personen arbejder lige nu - inaktiv ${Math.round(inaktivSek())} s; en ny Chrome henter sig selv frem foran dem) - bevist intet, ikke bestaaet`); process.exit(0); } + +let fejl = 0; +const check = (navn, ok, hvorfor) => { console.log(`${ok ? 'OK ' : 'DUMP'} ${navn}${ok ? '' : ' - ' + hvorfor}`); if (!ok) fejl++; }; +const hj = (args, stdin, env) => JSON.parse(execFileSync(H, args, { input: stdin, encoding: 'utf8', timeout: 60000, env: env ? { ...process.env, ...env } : process.env })); +const vent = (ms) => new Promise((r) => setTimeout(r, ms)); + +const mappe = mkdtempSync(join(tmpdir(), 'cmcp-chrome-')); +const side = join(mappe, 'side.html'); +writeFileSync(side, `