Filter accessibility snapshots by role and accessible name

CHANGELOG.md

@@ -4,6 +4,7 @@
 - Fixed a task tab with a "Leave site?" prompt stalling the extension. Chrome's `tabs.remove` waits until the tab is gone, and a page's beforeunload prompt, which a background tab shows only once activated, can hold the tab open indefinitely. `browser_finish` and task close waited on that removal inside the ownership queue, so every later ownership change, `browser_open`, `browser_tabs`, tab event, and native reconnect stalled behind it until the extension was reloaded. Task cleanup still removes ownership before closing tabs, now waits at most two seconds for Chrome to close them, and leaves a tab still waiting on its prompt out of both `closed_tab_ids` and `retained_tab_ids`; the tab closes if the user confirms leaving. A `browser_act` `close` waits the same two seconds and then returns `close_pending: true` instead of holding its request open.
 - No single extension operation can stall the operations queued behind it any more. An ownership change or attention-notice change still running after 10 seconds, or a background tab or alarm event after 30 seconds, keeps running and is logged, but the next operation starts. Ten seconds covers the two-second tab removal wait with room for normal Chrome calls and stays well under the host's 30-second command deadline, so a command queued behind a stuck operation can still answer.
 - `browser_finish` and task close now report an extension that does not answer in time as `extension_timeout`, telling the agent that the task is still open and to reload the extension if it stays unresponsive, then finish or close again. They previously reported `state_store_error` and suggested repairing `~/.agenttab` or freeing disk space. When the extension refuses cleanup, such as for a `keep_tab_ids` entry the task does not own, the error now carries the extension's own code and message instead of `native_protocol_error`. Disconnects and protocol or transport failures during cleanup keep their own error codes too.
+- Added accessibility-only `browser_snapshot.query` filters for exact case-insensitive AX roles and case-insensitive accessible-name substrings (not descriptions or values). Filtering preserves actionable refs and frame IDs, composes with `root_ref`/`frame_id`, and applies `max_nodes`/`truncated` to matching nodes, including DOM editable fallbacks and the scripting outline. Core RPC, MCP, OMP, Pi, and the TypeScript SDK expose the same optional query.
 - `scripts/reload_unpacked_extension.sh` now reaches the Chrome instance and profile that have the extension. It previously handed the wake page to whichever same-named Chrome instance macOS picked, so a second Chrome (such as a headless test browser) could receive it, nothing reloaded, and `dev:deploy` still reported success. The script now finds the profiles whose preferences list the extension in the application's default user data directory, opens the wake page in each through Chrome's own instance handoff, and exits with an error when no profile has the extension. Use `--user-data-dir` or `--profile-directory` (or `AGENTTAB_CHROME_USER_DATA_DIR` and `AGENTTAB_CHROME_PROFILE_DIRECTORY`) to override the detection.
 - `context_click` no longer opens Chrome's native context menu. On macOS that menu stalled every AgentTab request until someone dismissed it. Page `contextmenu` handlers still run, though they see the event as already handled. The suppression applies only to that one right click and expires on its own after five seconds.
 - Snapshot refs now survive a service worker restart or extension reload. The extension stores each snapshot's refs before returning it, drops them when their tab closes or navigates to a newer page revision, and clears them when Chrome starts. A ref still resolves only while its page document is the same one the snapshot saw. A ref the extension never stored fails with `stale_ref`, saying it is unknown or expired and asking for a new snapshot.

README.md

@@ -49,7 +49,7 @@ Standard mode exposes exactly ten tools:
 | Tool | Purpose |
 |---|---|
 | `browser_open` | Create a task tab, create an unfocused window for a new task, or explicitly adopt the active tab. Reports whether the resulting tab supports `full` or `tab_only` automation. |
-| `browser_snapshot` | Read an accessibility tree, bounded text or HTML, or a screenshot from a full-route task tab. |
+| `browser_snapshot` | Read an accessibility tree, bounded text or HTML, or a screenshot from a full-route task tab. Accessibility mode can filter actionable refs with `query: { role: "button", name: "More message options" }` (case-insensitive exact role and accessible-name substring). |
 | `browser_act` | Run typed actions against one task tab and expected page revision. Restricted-origin task tabs retain only navigation, history, reload, and close actions. |
 | `browser_wait` | Wait for load, URL, text, selector, network-idle, task-attributed download, or challenge-cleared conditions supported by the tab's route. |
 | `browser_tabs` | List only tabs owned by the current task, including each tab's automation route. |

docs/mcp.md

@@ -63,7 +63,7 @@ For MCP, the capability store namespace is `mcp`; OMP uses `omp`; Pi uses `pi`.
 | Tool | Required input and behavior |
 |---|---|
 | `browser_open` | `mode: "create"` optionally accepts an `http`, `https`, or `about` URL, `background`, and `placement`. The default `placement: "task"` creates a tab in the task's existing window when possible. `placement: "new_window"` creates the first tab of an otherwise empty task in a separate unfocused normal window and rejects `background: false`. `mode: "adopt_active"` explicitly adopts only the currently active tab. The result includes task, tab, window, page-revision, and `automation_route` identifiers. It returns after the tab's first load completes, up to 10 seconds, so its `page_revision` and `url` describe the loaded page and can be passed straight to `browser_act`. |
-| `browser_snapshot` | Requires `tab_id`. Modes are `accessibility`, `text`, `html`, and `screenshot`; each may scope to an opaque CDP `frame_id`. Only accessibility snapshots return revisioned node references, each bound to its originating frame; full-tree accessibility snapshots also append `dom_fallback` textbox nodes for editable elements that never entered Chrome's accessibility tree, so those refs still address the existing typed actions. Screenshot requests may select `png`, `jpeg`, or `webp`, set JPEG/WebP `quality`, bound dimensions with `max_width`/`max_height`, and cap compressed image size with `max_bytes`. Snapshots require the `full` automation route. |
+| `browser_snapshot` | Requires `tab_id`. Modes are `accessibility`, `text`, `html`, and `screenshot`; each may scope to an opaque CDP `frame_id`. Only accessibility snapshots return revisioned node references, each bound to its originating frame; full-tree accessibility snapshots also append `dom_fallback` textbox nodes for editable elements that never entered Chrome's accessibility tree, so those refs still address the existing typed actions. Accessibility mode optionally accepts `query: { role?: string, name?: string }`: `role` is an exact case-insensitive AX role (1–256 characters), and `name` is a case-insensitive substring of the accessible name (1–2048 characters), not the description or value. Both predicates must match when supplied; an empty query leaves the snapshot unfiltered. Filtering works with `root_ref` and `frame_id`, preserves actionable refs and frame IDs, and applies `max_nodes` to matching nodes, setting `truncated` when more matches exist. For example, `{ tab_id: 1, mode: "accessibility", query: { role: "button", name: "More message options" }, max_nodes: 10 }` locates controls without reading the entire returned tree. Other modes reject `query`, as they reject `root_ref`/`max_nodes`. Screenshot requests may select `png`, `jpeg`, or `webp`, set JPEG/WebP `quality`, bound dimensions with `max_width`/`max_height`, and cap compressed image size with `max_bytes`. Snapshots require the `full` automation route. |
 | `browser_act` | Requires `tab_id`, `expected_page_revision`, and one to 64 typed actions. Actions are click, hover, double-click, context-click, press, type, fill, select, scroll, drag, navigate, history movement, reload, close, dialog decision, and staged file upload. Element-target actions accept exactly one `ref` or `selector` and optional `frame_id`; `press` accepts the Standard navigation/editing keys plus optional unique `Alt`, `Control`, `Meta`, and `Shift` modifiers. Lowercase `a`, `z`, and `y` require exactly one of Control or Meta, optionally Shift, never Alt. No coordinate action, generic key injection, clipboard shortcut, or cross-frame target is exposed in Standard mode. When a step fails, the error `details` carry `failed_action_index` (zero-based) and `completed_actions`, the results of the steps that ran first; in a batch of more than one action, the message also names the failing step. A failure after at least one completed step has `outcome: "unknown"`, because those steps already changed the page. A `close` releases the tab from the task and returns `completed: true` once Chrome closes it, or `started: true, close_pending: true` when the tab is still open two seconds later, as when a "Leave site?" prompt holds it; the tab then closes if the user confirms leaving. An optional `intent`, one line of 1 to 120 characters, replaces the agent pointer's generated caption for every action in the batch, for example "Open the bookmarks folder". It is visible on screen, so it must never carry secrets or field values. |
 | `browser_wait` | Requires `tab_id` and one load, URL, text, selector, value, network-idle, download, or `challenge_cleared` condition. Selector waits may set `attached`, `detached`, `visible`, `hidden`, or `enabled` state and scope to a `frame_id`; value waits require selector and expected value and may scope to a frame. A download wait may resume after only its opaque tab-owned `after` cursor. A `challenge_cleared` wait takes the `challenge_id` of an active challenge episode on the tab and completes with `matched: true` once the episode clears or is superseded; if the episode reports it no longer expects automatic clearing, the wait returns early with outcome `needs_user` and result `status: "challenge_needs_user"`; an unknown, ended, or other-task episode fails with `challenge_not_found` and outcome `not_started`, and a timeout carries the latest event in `error.details.challenge`. `timeout_ms` is at most 120 seconds. Load and URL waits also follow Chrome's top-frame navigation events, with polling as a fallback: a load condition matches a top-frame load that completes after the wait starts, and a URL condition also matches committed navigations, same-document history changes (`pushState` and `replaceState`), and fragment changes seen during the wait. A top-frame navigation error during a load or URL wait fails with `navigation_failed`, `outcome: "unknown"`, and Chrome's network error name, never the URL; a superseded or cancelled navigation does not fail the wait. A `tab_only` route accepts load and URL conditions only; frame-scoped reads, value/selector state, network-idle, and download attribution, and challenge waits require the tab-scoped debugger connection available on the `full` route. |
 | `browser_tabs` | Takes an empty object and lists only the current task's tabs, including each tab's `automation_route`. |

host-rs/crates/agenttab-protocol/src/lib.rs

@@ -422,6 +422,40 @@ fn default_true() -> bool {
     true
 }
 
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub struct AccessibilityQuery {
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub role: Option<String>,
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub name: Option<String>,
+}
+
+// Serde's derived struct visitor also accepts sequences; the wire query must be an object.
+fn deserialize_accessibility_query<'de, D>(
+    deserializer: D,
+) -> Result<Option<AccessibilityQuery>, D::Error>
+where
+    D: serde::Deserializer<'de>,
+{
+    struct QueryObject;
+    impl<'de> serde::de::Visitor<'de> for QueryObject {
+        type Value = AccessibilityQuery;
+
+        fn expecting(&self, formatter: &mut std::fmt::Formatter) -> std::fmt::Result {
+            formatter.write_str("an accessibility query object")
+        }
+
+        fn visit_map<M>(self, map: M) -> Result<Self::Value, M::Error>
+        where
+            M: serde::de::MapAccess<'de>,
+        {
+            AccessibilityQuery::deserialize(serde::de::value::MapAccessDeserializer::new(map))
+        }
+    }
+    deserializer.deserialize_map(QueryObject).map(Some)
+}
+
 #[derive(Debug, Clone, Serialize, Deserialize)]
 #[serde(tag = "mode", rename_all = "snake_case", deny_unknown_fields)]
 pub enum BrowserSnapshotParams {
@@ -435,6 +469,12 @@ pub enum BrowserSnapshotParams {
         max_depth: Option<u16>,
         #[serde(default, skip_serializing_if = "Option::is_none")]
         max_nodes: Option<u32>,
+        #[serde(
+            default,
+            deserialize_with = "deserialize_accessibility_query",
+            skip_serializing_if = "Option::is_none"
+        )]
+        query: Option<AccessibilityQuery>,
     },
     Text {
         tab_id: u64,
@@ -1132,6 +1172,7 @@ impl MethodParams {
                 root_ref,
                 max_depth,
                 max_nodes,
+                query,
                 ..
             }) => {
                 validate_optional_frame_id(method, frame_id)?;
@@ -1152,6 +1193,14 @@ impl MethodParams {
                         "max_nodes must be between 1 and 5000",
                     )?;
                 }
+                if let Some(query) = query {
+                    if let Some(role) = &query.role {
+                        require_len(method, role, 1, 256, "query.role")?;
+                    }
+                    if let Some(name) = &query.name {
+                        require_len(method, name, 1, 2_048, "query.name")?;
+                    }
+                }
             }
             Self::Snapshot(BrowserSnapshotParams::Text {
                 frame_id,
@@ -2966,6 +3015,56 @@ mod tests {
         }
     }
 
+    #[test]
+    fn snapshot_accessibility_queries_are_bounded_and_mode_specific() {
+        for query in [
+            json!({}),
+            json!({"role": "BUTTON"}),
+            json!({"name": "message"}),
+            json!({"role": "button", "name": "More message options"}),
+        ] {
+            let params = json!({
+                "tab_id": 1, "mode": "accessibility", "frame_id": "frame-1",
+                "root_ref": "rt1.root", "max_nodes": 1, "query": query,
+            });
+            let (_, parsed) =
+                RpcRequest::parse(request("browser_snapshot", params.clone(), false)).unwrap();
+            assert_eq!(parsed.value(), params);
+        }
+        for query in [
+            json!(null),
+            json!([]),
+            json!(["button", "options"]),
+            json!("button"),
+            json!(1),
+            json!({"role": null}),
+            json!({"name": null}),
+            json!({"role": 1}),
+            json!({"name": true}),
+            json!({"role": ""}),
+            json!({"name": ""}),
+            json!({"role": "x".repeat(257)}),
+            json!({"name": "x".repeat(2049)}),
+            json!({"description": "message"}),
+            json!({"role": "button", "extra": true}),
+        ] {
+            assert!(RpcRequest::parse(request(
+                "browser_snapshot",
+                json!({"tab_id": 1, "mode": "accessibility", "query": query}),
+                false,
+            ))
+            .is_err());
+        }
+        for mode in ["text", "html", "screenshot"] {
+            assert!(RpcRequest::parse(request(
+                "browser_snapshot",
+                json!({"tab_id": 1, "mode": mode, "query": {"role": "button"}}),
+                false,
+            ))
+            .is_err());
+        }
+    }
+
     #[test]
     fn runtime_constraints_reject_schema_invalid_limits() {
         for (method, params, mutation) in [

packages/extension/src/accessibility-query.ts

@@ -0,0 +1,13 @@
+export interface AccessibilityQuery {
+  role?: string;
+  name?: string;
+}
+
+/** Match the exposed accessible name only, not description or value. */
+export function accessibilityMatcher(query: AccessibilityQuery | undefined): (role: string, name: string) => boolean {
+  const role = query?.role?.toLowerCase();
+  const name = query?.name?.toLowerCase();
+  return (nodeRole, nodeName) =>
+    (role === undefined || nodeRole.toLowerCase() === role) &&
+    (name === undefined || nodeName.toLowerCase().includes(name));
+}

packages/extension/src/browser.ts

@@ -1,4 +1,5 @@
 import { RevisionTracker } from "./revisions";
+import { accessibilityMatcher, type AccessibilityQuery } from "./accessibility-query";
 import {
   dispatchDomClick,
   dispatchDomFill,
@@ -1076,6 +1077,7 @@ export class StandardBrowserRuntime {
       max_nodes: maxNodes,
       max_depth: maxDepth,
       ...(root !== undefined ? { root_id: root.backendNodeId } : {}),
+      ...(params.query !== undefined ? { query: params.query as AccessibilityQuery } : {}),
     }, root?.documentId);
     await this.assertScriptingRevision(tabId, pageRevision, "Page changed while the accessibility outline was captured");
     const outline = isRecord(value) ? value : {};
@@ -1411,13 +1413,22 @@ export class StandardBrowserRuntime {
         ...(frameId !== undefined ? { frameId } : {}),
       }, false, frameDocument.sessionId);
     const nodes = (Array.isArray(result.nodes) ? result.nodes.filter(isRecord) : []) as AxNode[];
+    const matches = accessibilityMatcher(params.query as AccessibilityQuery | undefined);
+    const matchingNodes = params.query === undefined ? nodes : nodes.filter((node) => matches(
+      typeof node.role?.value === "string" ? node.role.value : "unknown",
+      typeof node.name?.value === "string" ? node.name.value : "",
+    ));
     const axBackendIds = new Set<number>();
     for (const node of nodes) {
       if (typeof node.backendDOMNodeId === "number") axBackendIds.add(node.backendDOMNodeId);
     }
-    const domFallbackNodes = rootTarget === undefined
+    const fallbackNodes = rootTarget === undefined
       ? await this.#domEditableFallbackNodes(tabId, pageRevision, axBackendIds, frameDocument, taskId)
       : [];
+    const domFallbackNodes = params.query === undefined ? fallbackNodes : fallbackNodes.filter((node) => matches(
+      typeof node.role === "string" ? node.role : "unknown",
+      typeof node.name === "string" ? node.name : "",
+    ));
     const after = await this.frameDocument(tabId, frameDocument.frameId);
     if (frameDocument.documentId !== after.documentId) {
       const currentPageRevision = isTopFrame
@@ -1436,7 +1447,7 @@ export class StandardBrowserRuntime {
         pageRevision,
         taskId,
       });
-    const encoded = nodes.slice(0, Math.max(0, maxNodes - domFallbackNodes.length)).map((node) => ({
+    const encoded = matchingNodes.slice(0, Math.max(0, maxNodes - domFallbackNodes.length)).map((node) => ({
       ...(typeof node.backendDOMNodeId === "number" ? { ref: mintRef(node.backendDOMNodeId) } : {}),
       ...(typeof node.backendDOMNodeId === "number" ? { frame_id: frameDocument.frameId } : {}),
       role: typeof node.role?.value === "string" ? node.role.value : "unknown",
@@ -1475,7 +1486,7 @@ export class StandardBrowserRuntime {
       ...(pendingFrames.length > 0 ? { frames_truncated: true, frames_limit: 256 } : {}),
       mode,
       nodes: combined.slice(0, maxNodes),
-      truncated: nodes.length + domFallbackNodes.length > maxNodes,
+      truncated: matchingNodes.length + domFallbackNodes.length > maxNodes,
       ...(domFallbackNodes.length > 0 ? { dom_fallback_nodes: domFallbackNodes.length } : {}),
     });
   }