Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Added

- KDE Plasma Wayland (KWin 6.3 and later) gets the virtual pen and touchscreen for `click`, `drag` and `scroll` with `delivery: "background"`, as routes `kwin-pen` and `kwin-touch`: the user's mouse pointer stays where it is. KWin maps the pen to the whole workspace and the touchscreen to the output under the point through `org.kde.KWin.InputDevice`, read back before input, and the server removes what KWin wrote for these devices into `kcminputrc` once they are gone. The point is checked through KWin scripting against KWin's stacking order (panels, popups and notifications included), on the current desktop and activity; Show Desktop and full-screen effects such as the Overview refuse the call. KWin's `SyncWithMouse` tablet option and `KWIN_WAYLAND_EMULATE_TABLET`, which move or click the user's pointer, leave no route. Measured on KWin 6.6.7 on a DRM output with GTK 4.

### Fixed

- A background right click on GNOME 48 and later arrived as a middle click and a middle click as a right click: GNOME 48 changed the stylus button defaults (`button-action` to `right`, `secondary-button-action` to `middle`), and mutter rewrites the pen's buttons by them. The server now sets both actions for its own pen (the settings path keyed by its vendor and product id) before every action, so mutter hands clients the button the pen pressed, and resets them when the pen goes away. Measured with GNOME 48's defaults on GNOME Shell 46, with GTK 4 and with native Wayland and Xwayland Chromium; mutter 48 and later read the same explicit values (from mutter's source).
Expand Down
4 changes: 3 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -181,7 +181,7 @@ On Hyprland with `COMPUTER_USE_LINUX_HYPRLAND_BACKGROUND=1`, a window target wit

Where there is no agent seat, a window-targeted background call has no route (the notice, then focused input); element, browser tab and terminal targets still work. For full isolation, with its own keyboard and pointer, run the app in an [agent-workspace-linux](https://github.com/agent-sh/agent-workspace-linux) workspace; the notice points there. `doctor` lists the routes this machine has in `capabilities.background_input` (`seat` when the agent seat is available) and the seat's route, pointer support or the reason it has none in `capabilities.background_seat`. `capabilities.background_pointer` shows the GNOME pen route and its limits, or why there is none.

On GNOME Wayland, background `click`, `drag` and `scroll` go through a virtual pen and a virtual touchscreen instead of the user's pointer ([#480](https://github.com/agent-sh/computer-use-linux/issues/480); the idea and the tested approach are [@Ondra-Zik](https://github.com/Ondra-Zik)'s). Mutter gives a tablet tool its own cursor and sends its events through the tablet protocol, not `wl_pointer`, so the user's mouse stays where it is. Clicks are pen taps (left), the upper barrel button (right) and the lower one (middle); `click_count` repeats them. GNOME 48 and later default the lower button to a right click and the upper one to a middle click, so before each action the server sets its own pen's button actions to match the buttons it presses, and resets them when the pen goes away. A drag is the pen tip pressed at the start, moved and lifted at the end. The pen leaves proximity after each action: libinput pairs a pen with a touchscreen and ignores that screen's touches while the pen is near, and that screen can be the user's. A scroll is a one-finger touch pan, 60 logical pixels per wheel notch (5 notches per page), split into strokes that stay inside the window and 48 pixels from the monitor's edges, where GNOME Shell starts its edge swipes; the finger rests before it lifts, so the app does not fling further. Before any input, the server checks through the GNOME Shell extension that the point is on the target window, on the workspace the user sees, and that no window stacked above it covers the point; both ends of a drag and every pan stroke must be on the window, and a pan always starts at the point. The results report the route as `gnome-pen` or `gnome-touch`. The devices need write access to `/dev/uinput`, as the uinput pointer does; without it, or without the extension, the call gets the notice and then focused input.
On GNOME Wayland (and KDE Plasma, below), background `click`, `drag` and `scroll` go through a virtual pen and a virtual touchscreen instead of the user's pointer ([#480](https://github.com/agent-sh/computer-use-linux/issues/480); the idea and the tested approach are [@Ondra-Zik](https://github.com/Ondra-Zik)'s). Mutter gives a tablet tool its own cursor and sends its events through the tablet protocol, not `wl_pointer`, so the user's mouse stays where it is. Clicks are pen taps (left), the upper barrel button (right) and the lower one (middle); `click_count` repeats them. GNOME 48 and later default the lower button to a right click and the upper one to a middle click, so before each action the server sets its own pen's button actions to match the buttons it presses, and resets them when the pen goes away. A drag is the pen tip pressed at the start, moved and lifted at the end. The pen leaves proximity after each action: libinput pairs a pen with a touchscreen and ignores that screen's touches while the pen is near, and that screen can be the user's. A scroll is a one-finger touch pan, 60 logical pixels per wheel notch (5 notches per page), split into strokes that stay inside the window and 48 pixels from the monitor's edges, where GNOME Shell starts its edge swipes; the finger rests before it lifts, so the app does not fling further. Before any input, the server checks through the GNOME Shell extension that the point is on the target window, on the workspace the user sees, and that no window stacked above it covers the point; both ends of a drag and every pan stroke must be on the window, and a pan always starts at the point. The results report the route as `gnome-pen` or `gnome-touch`. The devices need write access to `/dev/uinput`, as the uinput pointer does; without it, or without the extension, the call gets the notice and then focused input.

Limits:

Expand All @@ -193,6 +193,8 @@ Limits:
- Mutter maps an unconfigured tablet across the whole layout, and an unconfigured touchscreen to the builtin panel. When a device maps elsewhere than the point needs, the server writes that device's `output` setting (`org.gnome.desktop.peripherals.tablet` or `.touchscreen`, at the path keyed by the devices' own vendor and product ids, `6375:7065` and `6375:7463`), waits for mutter's `org.gnome.Mutter.InputMapping` to confirm the new mapping, and resets the setting when the device goes away. No other device's setting is touched.
- The devices appear on first use and go away after ten idle minutes or when the server exits.

On KDE Plasma Wayland with KWin 6.3 or later the same route runs as `kwin-pen` and `kwin-touch`. KWin gives a tablet its own cursor from 6.3 (it tracks the tablet cursor itself and moves it whether or not the client under it supports tablets). KWin maps a tablet with no setting to the active output, which follows the user's pointer, and a touchscreen to the internal panel or the first output. So before input the server sets the pen's `mapToWorkspace` and the touchscreen's `outputName` (the output under the point) through `org.kde.KWin.InputDevice`, reads them back, and once the devices are gone removes what KWin wrote for them into `kcminputrc` (the groups keyed by these devices' vendor, product and name) with `kwriteconfig6`. The window check goes through KWin scripting: the point must be on the target, on the desktop and activity the user sees, not minimized or hidden, with no window, panel, popup or notification stacked above it. The call is refused while KWin shows the desktop or runs a full-screen effect such as the Overview (`org.kde.kwin.Effects.activeEffects`), and when another setting for the device moves where input lands (calibration, input or output area, rotation). A scroll reads the output's rotation with `kscreen-doctor` and refuses a rotated or flipped output; without `kscreen-doctor` there is no touch route. There is no route at all when KWin's `SyncWithMouse` tablet option (`kcminputrc` `[Tablet]`) is on, since it moves the user's pointer to where the pen was, when KWin runs with `KWIN_WAYLAND_EMULATE_TABLET`, which sends pen input over apps without tablet support through the user's pointer, or without `kreadconfig6` and `kwriteconfig6`.

Some clients mishandle a second `wl_seat` (Zed, [zed-industries/zed#66735](https://github.com/zed-industries/zed/issues/66735)); that is one more reason the sway seat exists only during a call. A menu the agent opens on sway closes when its call ends, since the seat that opened it goes away. While sway is in a binding mode other than `default`, background keys have no route, since the mode's bindings would take them.

### On-screen indicator
Expand Down
12 changes: 6 additions & 6 deletions pi/extension/generated-tools.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@ export interface GeneratedMcpToolDefinition {
}

export const GENERATED_SERVER_VERSION = "0.7.58";
export const GENERATED_TOOL_CATALOG_HASH = "e447f071f2ccf6c438c5fca9c80dbbe642661b1e9afa6239aad6534b2815ebfb";
export const GENERATED_SHELL_TOOL_CATALOG_HASH = "edaac05bdf1eaca9f3ae95f7e0e7d9f41797c920ec7331c39e6864c04841fa51";
export const GENERATED_TOOL_CATALOG_HASH = "de04450abc876a3712849d85daad7b80ed6656e1ddf21e2bfd16e94712b4247c";
export const GENERATED_SHELL_TOOL_CATALOG_HASH = "26a00c32301af2704aeaf97fe3cb58f9429e255305ad4e17efd50db506cd6717";
export const GENERATED_MCP_TOOLS =
[
{
Expand Down Expand Up @@ -279,7 +279,7 @@ export const GENERATED_MCP_TOOLS =
}
],
"default": null,
"description": "\"background\" sends the input through the agent's own pointer, so the\nuser's pointer and keyboard focus stay where they are: on X11 without\na window manager (an XInput 2 master pointer) and on sway 1.10 or later\n(a transient seat). On GNOME Wayland a virtual pen (a touch pan for\nscroll) leaves the user's pointer alone, but a click still focuses the\nwindow, and apps without tablet support ignore it. Needs x and y in\nscreenshot capture pixels over the window the call targets, without\nrelative or an element. Elsewhere the\nfirst such call in a server session sends nothing and returns a\nnotice; repeated, and for every later such call, it goes through the\nuser's pointer as with \"focus\". Default: \"focus\"."
"description": "\"background\" sends the input through the agent's own pointer, so the\nuser's pointer and keyboard focus stay where they are: on X11 without\na window manager (an XInput 2 master pointer) and on sway 1.10 or later\n(a transient seat). On GNOME and KDE Plasma Wayland a virtual pen (a touch pan for\nscroll) leaves the user's pointer alone, but a click still focuses the\nwindow, and apps without tablet support ignore it. Needs x and y in\nscreenshot capture pixels over the window the call targets, without\nrelative or an element. Elsewhere the\nfirst such call in a server session sends nothing and returns a\nnotice; repeated, and for every later such call, it goes through the\nuser's pointer as with \"focus\". Default: \"focus\"."
},
"element_index": {
"default": null,
Expand Down Expand Up @@ -501,7 +501,7 @@ export const GENERATED_MCP_TOOLS =
"openWorldHint": true,
"readOnlyHint": false
},
"description": "Drag with the left button from one point to another in screenshot capture pixels. Takes no relative coordinates; a window target matters only with delivery=\"background\", which on GNOME Wayland drags with a virtual pen inside that window and leaves the user's pointer where it is (elsewhere the first such call returns a one-time notice, then later calls drag through the user's pointer). Pass screenshot_after (true, \"screen\", or \"window\") or include_tree=true to return post-action observation in the same result.",
"description": "Drag with the left button from one point to another in screenshot capture pixels. Takes no relative coordinates; a window target matters only with delivery=\"background\", which on GNOME and KDE Plasma Wayland drags with a virtual pen inside that window and leaves the user's pointer where it is (elsewhere the first such call returns a one-time notice, then later calls drag through the user's pointer). Pass screenshot_after (true, \"screen\", or \"window\") or include_tree=true to return post-action observation in the same result.",
"inputSchema": {
"$defs": {
"KeyboardDelivery": {
Expand Down Expand Up @@ -547,7 +547,7 @@ export const GENERATED_MCP_TOOLS =
}
],
"default": null,
"description": "\"background\" drags with a virtual pen on GNOME Wayland, so the\nuser's pointer stays where it is; both points must be on the target\nwindow, with no window stacked above them. Elsewhere the first such\ncall in a server session sends nothing and returns a notice;\nrepeated, and for every later such call, the drag goes through the\nuser's pointer as with \"focus\". Default: \"focus\"."
"description": "\"background\" drags with a virtual pen on GNOME and KDE Plasma Wayland, so the\nuser's pointer stays where it is; both points must be on the target\nwindow, with no window stacked above them. Elsewhere the first such\ncall in a server session sends nothing and returns a notice;\nrepeated, and for every later such call, the drag goes through the\nuser's pointer as with \"focus\". Default: \"focus\"."
},
"end_x": {
"description": "End x in raw screenshot capture pixels.",
Expand Down Expand Up @@ -2094,7 +2094,7 @@ export const GENERATED_MCP_TOOLS =
}
],
"default": null,
"description": "\"background\" sends the input through the agent's own pointer, so the\nuser's pointer and keyboard focus stay where they are: on X11 without\na window manager (an XInput 2 master pointer) and on sway 1.10 or later\n(a transient seat). On GNOME Wayland a virtual pen (a touch pan for\nscroll) leaves the user's pointer alone, but a click still focuses the\nwindow, and apps without tablet support ignore it. Needs x and y in\nscreenshot capture pixels over the window the call targets, without\nrelative or an element. Elsewhere the\nfirst such call in a server session sends nothing and returns a\nnotice; repeated, and for every later such call, it goes through the\nuser's pointer as with \"focus\". Default: \"focus\"."
"description": "\"background\" sends the input through the agent's own pointer, so the\nuser's pointer and keyboard focus stay where they are: on X11 without\na window manager (an XInput 2 master pointer) and on sway 1.10 or later\n(a transient seat). On GNOME and KDE Plasma Wayland a virtual pen (a touch pan for\nscroll) leaves the user's pointer alone, but a click still focuses the\nwindow, and apps without tablet support ignore it. Needs x and y in\nscreenshot capture pixels over the window the call targets, without\nrelative or an element. Elsewhere the\nfirst such call in a server session sends nothing and returns a\nnotice; repeated, and for every later such call, it goes through the\nuser's pointer as with \"focus\". Default: \"focus\"."
},
"direction": {
"description": "up, down, left or right (case-insensitive).",
Expand Down
Loading
Loading