diff --git a/CHANGELOG.md b/CHANGELOG.md
index 54940bcf..e1d0a6f8 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
+### Added
+
+- KDE Plasma Wayland (KWin 6.3 and later) gets the virtual pen and touchscreen for `click`, `drag` and `scroll` with `delivery: "background"`, as routes `kwin-pen` and `kwin-touch`: the user's mouse pointer stays where it is. KWin maps the pen to the whole workspace and the touchscreen to the output under the point through `org.kde.KWin.InputDevice`, read back before input, and the server removes what KWin wrote for these devices into `kcminputrc` once they are gone. The point is checked through KWin scripting against KWin's stacking order (panels, popups and notifications included), on the current desktop and activity; Show Desktop and full-screen effects such as the Overview refuse the call. KWin's `SyncWithMouse` tablet option and `KWIN_WAYLAND_EMULATE_TABLET`, which move or click the user's pointer, leave no route. Measured on KWin 6.6.7 on a DRM output with GTK 4.
+
### Fixed
- A background right click on GNOME 48 and later arrived as a middle click and a middle click as a right click: GNOME 48 changed the stylus button defaults (`button-action` to `right`, `secondary-button-action` to `middle`), and mutter rewrites the pen's buttons by them. The server now sets both actions for its own pen (the settings path keyed by its vendor and product id) before every action, so mutter hands clients the button the pen pressed, and resets them when the pen goes away. Measured with GNOME 48's defaults on GNOME Shell 46, with GTK 4 and with native Wayland and Xwayland Chromium; mutter 48 and later read the same explicit values (from mutter's source).
diff --git a/README.md b/README.md
index 26f5cbdf..d98f535b 100644
--- a/README.md
+++ b/README.md
@@ -181,7 +181,7 @@ On Hyprland with `COMPUTER_USE_LINUX_HYPRLAND_BACKGROUND=1`, a window target wit
Where there is no agent seat, a window-targeted background call has no route (the notice, then focused input); element, browser tab and terminal targets still work. For full isolation, with its own keyboard and pointer, run the app in an [agent-workspace-linux](https://github.com/agent-sh/agent-workspace-linux) workspace; the notice points there. `doctor` lists the routes this machine has in `capabilities.background_input` (`seat` when the agent seat is available) and the seat's route, pointer support or the reason it has none in `capabilities.background_seat`. `capabilities.background_pointer` shows the GNOME pen route and its limits, or why there is none.
-On GNOME Wayland, background `click`, `drag` and `scroll` go through a virtual pen and a virtual touchscreen instead of the user's pointer ([#480](https://github.com/agent-sh/computer-use-linux/issues/480); the idea and the tested approach are [@Ondra-Zik](https://github.com/Ondra-Zik)'s). Mutter gives a tablet tool its own cursor and sends its events through the tablet protocol, not `wl_pointer`, so the user's mouse stays where it is. Clicks are pen taps (left), the upper barrel button (right) and the lower one (middle); `click_count` repeats them. GNOME 48 and later default the lower button to a right click and the upper one to a middle click, so before each action the server sets its own pen's button actions to match the buttons it presses, and resets them when the pen goes away. A drag is the pen tip pressed at the start, moved and lifted at the end. The pen leaves proximity after each action: libinput pairs a pen with a touchscreen and ignores that screen's touches while the pen is near, and that screen can be the user's. A scroll is a one-finger touch pan, 60 logical pixels per wheel notch (5 notches per page), split into strokes that stay inside the window and 48 pixels from the monitor's edges, where GNOME Shell starts its edge swipes; the finger rests before it lifts, so the app does not fling further. Before any input, the server checks through the GNOME Shell extension that the point is on the target window, on the workspace the user sees, and that no window stacked above it covers the point; both ends of a drag and every pan stroke must be on the window, and a pan always starts at the point. The results report the route as `gnome-pen` or `gnome-touch`. The devices need write access to `/dev/uinput`, as the uinput pointer does; without it, or without the extension, the call gets the notice and then focused input.
+On GNOME Wayland (and KDE Plasma, below), background `click`, `drag` and `scroll` go through a virtual pen and a virtual touchscreen instead of the user's pointer ([#480](https://github.com/agent-sh/computer-use-linux/issues/480); the idea and the tested approach are [@Ondra-Zik](https://github.com/Ondra-Zik)'s). Mutter gives a tablet tool its own cursor and sends its events through the tablet protocol, not `wl_pointer`, so the user's mouse stays where it is. Clicks are pen taps (left), the upper barrel button (right) and the lower one (middle); `click_count` repeats them. GNOME 48 and later default the lower button to a right click and the upper one to a middle click, so before each action the server sets its own pen's button actions to match the buttons it presses, and resets them when the pen goes away. A drag is the pen tip pressed at the start, moved and lifted at the end. The pen leaves proximity after each action: libinput pairs a pen with a touchscreen and ignores that screen's touches while the pen is near, and that screen can be the user's. A scroll is a one-finger touch pan, 60 logical pixels per wheel notch (5 notches per page), split into strokes that stay inside the window and 48 pixels from the monitor's edges, where GNOME Shell starts its edge swipes; the finger rests before it lifts, so the app does not fling further. Before any input, the server checks through the GNOME Shell extension that the point is on the target window, on the workspace the user sees, and that no window stacked above it covers the point; both ends of a drag and every pan stroke must be on the window, and a pan always starts at the point. The results report the route as `gnome-pen` or `gnome-touch`. The devices need write access to `/dev/uinput`, as the uinput pointer does; without it, or without the extension, the call gets the notice and then focused input.
Limits:
@@ -193,6 +193,8 @@ Limits:
- Mutter maps an unconfigured tablet across the whole layout, and an unconfigured touchscreen to the builtin panel. When a device maps elsewhere than the point needs, the server writes that device's `output` setting (`org.gnome.desktop.peripherals.tablet` or `.touchscreen`, at the path keyed by the devices' own vendor and product ids, `6375:7065` and `6375:7463`), waits for mutter's `org.gnome.Mutter.InputMapping` to confirm the new mapping, and resets the setting when the device goes away. No other device's setting is touched.
- The devices appear on first use and go away after ten idle minutes or when the server exits.
+On KDE Plasma Wayland with KWin 6.3 or later the same route runs as `kwin-pen` and `kwin-touch`. KWin gives a tablet its own cursor from 6.3 (it tracks the tablet cursor itself and moves it whether or not the client under it supports tablets). KWin maps a tablet with no setting to the active output, which follows the user's pointer, and a touchscreen to the internal panel or the first output. So before input the server sets the pen's `mapToWorkspace` and the touchscreen's `outputName` (the output under the point) through `org.kde.KWin.InputDevice`, reads them back, and once the devices are gone removes what KWin wrote for them into `kcminputrc` (the groups keyed by these devices' vendor, product and name) with `kwriteconfig6`. The window check goes through KWin scripting: the point must be on the target, on the desktop and activity the user sees, not minimized or hidden, with no window, panel, popup or notification stacked above it. The call is refused while KWin shows the desktop or runs a full-screen effect such as the Overview (`org.kde.kwin.Effects.activeEffects`), and when another setting for the device moves where input lands (calibration, input or output area, rotation). A scroll reads the output's rotation with `kscreen-doctor` and refuses a rotated or flipped output; without `kscreen-doctor` there is no touch route. There is no route at all when KWin's `SyncWithMouse` tablet option (`kcminputrc` `[Tablet]`) is on, since it moves the user's pointer to where the pen was, when KWin runs with `KWIN_WAYLAND_EMULATE_TABLET`, which sends pen input over apps without tablet support through the user's pointer, or without `kreadconfig6` and `kwriteconfig6`.
+
Some clients mishandle a second `wl_seat` (Zed, [zed-industries/zed#66735](https://github.com/zed-industries/zed/issues/66735)); that is one more reason the sway seat exists only during a call. A menu the agent opens on sway closes when its call ends, since the seat that opened it goes away. While sway is in a binding mode other than `default`, background keys have no route, since the mode's bindings would take them.
### On-screen indicator
diff --git a/pi/extension/generated-tools.ts b/pi/extension/generated-tools.ts
index fc691251..ba447a08 100644
--- a/pi/extension/generated-tools.ts
+++ b/pi/extension/generated-tools.ts
@@ -8,8 +8,8 @@ export interface GeneratedMcpToolDefinition {
}
export const GENERATED_SERVER_VERSION = "0.7.58";
-export const GENERATED_TOOL_CATALOG_HASH = "e447f071f2ccf6c438c5fca9c80dbbe642661b1e9afa6239aad6534b2815ebfb";
-export const GENERATED_SHELL_TOOL_CATALOG_HASH = "edaac05bdf1eaca9f3ae95f7e0e7d9f41797c920ec7331c39e6864c04841fa51";
+export const GENERATED_TOOL_CATALOG_HASH = "de04450abc876a3712849d85daad7b80ed6656e1ddf21e2bfd16e94712b4247c";
+export const GENERATED_SHELL_TOOL_CATALOG_HASH = "26a00c32301af2704aeaf97fe3cb58f9429e255305ad4e17efd50db506cd6717";
export const GENERATED_MCP_TOOLS =
[
{
@@ -279,7 +279,7 @@ export const GENERATED_MCP_TOOLS =
}
],
"default": null,
- "description": "\"background\" sends the input through the agent's own pointer, so the\nuser's pointer and keyboard focus stay where they are: on X11 without\na window manager (an XInput 2 master pointer) and on sway 1.10 or later\n(a transient seat). On GNOME Wayland a virtual pen (a touch pan for\nscroll) leaves the user's pointer alone, but a click still focuses the\nwindow, and apps without tablet support ignore it. Needs x and y in\nscreenshot capture pixels over the window the call targets, without\nrelative or an element. Elsewhere the\nfirst such call in a server session sends nothing and returns a\nnotice; repeated, and for every later such call, it goes through the\nuser's pointer as with \"focus\". Default: \"focus\"."
+ "description": "\"background\" sends the input through the agent's own pointer, so the\nuser's pointer and keyboard focus stay where they are: on X11 without\na window manager (an XInput 2 master pointer) and on sway 1.10 or later\n(a transient seat). On GNOME and KDE Plasma Wayland a virtual pen (a touch pan for\nscroll) leaves the user's pointer alone, but a click still focuses the\nwindow, and apps without tablet support ignore it. Needs x and y in\nscreenshot capture pixels over the window the call targets, without\nrelative or an element. Elsewhere the\nfirst such call in a server session sends nothing and returns a\nnotice; repeated, and for every later such call, it goes through the\nuser's pointer as with \"focus\". Default: \"focus\"."
},
"element_index": {
"default": null,
@@ -501,7 +501,7 @@ export const GENERATED_MCP_TOOLS =
"openWorldHint": true,
"readOnlyHint": false
},
- "description": "Drag with the left button from one point to another in screenshot capture pixels. Takes no relative coordinates; a window target matters only with delivery=\"background\", which on GNOME Wayland drags with a virtual pen inside that window and leaves the user's pointer where it is (elsewhere the first such call returns a one-time notice, then later calls drag through the user's pointer). Pass screenshot_after (true, \"screen\", or \"window\") or include_tree=true to return post-action observation in the same result.",
+ "description": "Drag with the left button from one point to another in screenshot capture pixels. Takes no relative coordinates; a window target matters only with delivery=\"background\", which on GNOME and KDE Plasma Wayland drags with a virtual pen inside that window and leaves the user's pointer where it is (elsewhere the first such call returns a one-time notice, then later calls drag through the user's pointer). Pass screenshot_after (true, \"screen\", or \"window\") or include_tree=true to return post-action observation in the same result.",
"inputSchema": {
"$defs": {
"KeyboardDelivery": {
@@ -547,7 +547,7 @@ export const GENERATED_MCP_TOOLS =
}
],
"default": null,
- "description": "\"background\" drags with a virtual pen on GNOME Wayland, so the\nuser's pointer stays where it is; both points must be on the target\nwindow, with no window stacked above them. Elsewhere the first such\ncall in a server session sends nothing and returns a notice;\nrepeated, and for every later such call, the drag goes through the\nuser's pointer as with \"focus\". Default: \"focus\"."
+ "description": "\"background\" drags with a virtual pen on GNOME and KDE Plasma Wayland, so the\nuser's pointer stays where it is; both points must be on the target\nwindow, with no window stacked above them. Elsewhere the first such\ncall in a server session sends nothing and returns a notice;\nrepeated, and for every later such call, the drag goes through the\nuser's pointer as with \"focus\". Default: \"focus\"."
},
"end_x": {
"description": "End x in raw screenshot capture pixels.",
@@ -2094,7 +2094,7 @@ export const GENERATED_MCP_TOOLS =
}
],
"default": null,
- "description": "\"background\" sends the input through the agent's own pointer, so the\nuser's pointer and keyboard focus stay where they are: on X11 without\na window manager (an XInput 2 master pointer) and on sway 1.10 or later\n(a transient seat). On GNOME Wayland a virtual pen (a touch pan for\nscroll) leaves the user's pointer alone, but a click still focuses the\nwindow, and apps without tablet support ignore it. Needs x and y in\nscreenshot capture pixels over the window the call targets, without\nrelative or an element. Elsewhere the\nfirst such call in a server session sends nothing and returns a\nnotice; repeated, and for every later such call, it goes through the\nuser's pointer as with \"focus\". Default: \"focus\"."
+ "description": "\"background\" sends the input through the agent's own pointer, so the\nuser's pointer and keyboard focus stay where they are: on X11 without\na window manager (an XInput 2 master pointer) and on sway 1.10 or later\n(a transient seat). On GNOME and KDE Plasma Wayland a virtual pen (a touch pan for\nscroll) leaves the user's pointer alone, but a click still focuses the\nwindow, and apps without tablet support ignore it. Needs x and y in\nscreenshot capture pixels over the window the call targets, without\nrelative or an element. Elsewhere the\nfirst such call in a server session sends nothing and returns a\nnotice; repeated, and for every later such call, it goes through the\nuser's pointer as with \"focus\". Default: \"focus\"."
},
"direction": {
"description": "up, down, left or right (case-insensitive).",
diff --git a/plugins/computer-use-linux/skills/computer-use-linux/SKILL.md b/plugins/computer-use-linux/skills/computer-use-linux/SKILL.md
index 4567672f..0686ff05 100644
--- a/plugins/computer-use-linux/skills/computer-use-linux/SKILL.md
+++ b/plugins/computer-use-linux/skills/computer-use-linux/SKILL.md
@@ -236,7 +236,7 @@ directly to a window-relative click.
- `mcp --listen ADDR:PORT` serves streamable HTTP at `/mcp` and requires a token from `COMPUTER_USE_LINUX_MCP_TOKEN_FILE` or `COMPUTER_USE_LINUX_MCP_TOKEN`. A non-loopback bind needs `--allowed-host NAME`. The port is plain HTTP; put TLS in front. Do not retry a 401 by putting the token in a URL. Over HTTP, `get_tree_page` and index and selector calls need `snapshot_id`; if refused, call `get_app_state` again. Idle sessions expire after an hour; on `404 Session not found`, initialize again. The cached tree and its `snapshot_id` survive.
- On GNOME Wayland, the remote-control portal asks on each new `computer-use-linux` process. `COMPUTER_USE_LINUX_PERSIST_REMOTE_DESKTOP=1` in the server environment asks the portal to remember the grant (GNOME starts with the remember box checked). Later processes reuse a single-use restore token stored mode 0600 in the user state directory. Unset is the default and prompts every time. The flag does nothing when ydotool, not the portal, is the input backend.
- On Hyprland, `COMPUTER_USE_LINUX_HYPRLAND_BACKGROUND=1` lets targeted `type_text`, `press_key`, `get_app_state` and `screenshot` work on a window without focusing it, including one on a hidden workspace. Pass `window_id` (or another target) on every call. An AT-SPI action can move focus inside the app, so re-focus a field with its `activate` action before typing into it. The active Hyprland keymap must be US, or background `type_text` and `press_key` calls are refused; background `get_app_state` and `screenshot` still capture. Text must be printable ASCII, tab or newline. An unfocused Chromium ignores browser accelerators such as Ctrl+R, and Chromium joins AT-SPI only when started with `ACCESSIBILITY_ENABLED=1` if the accessibility bus reports itself disabled.
-- To act without taking the user's keyboard, pass `delivery: "background"` to `type_text` or `press_key` (naming an element or `browser_tab` implies it). Name an element from `get_app_state` (text is inserted and read back; Enter or space runs a button's, link's, check box's or single-line field's action), a `browser_tab` when `COMPUTER_USE_LINUX_DEVTOOLS` names the port of a browser the user started with `--remote-debugging-port`, or a terminal by `tty` or terminal selectors (kitty remote control, wezterm, tmux or GNU screen). On X11 and sway 1.10+ a plain window target also works: the agent gets its own keyboard focus (an XInput 2 master or a transient seat), and `click`/`scroll` with `delivery: "background"`, `x`/`y` and a window target use the agent's own pointer (X11 only with no window manager). On sway, Xwayland, GTK and Chromium windows and Alt/Super chords have no background route; on X11, window manager shortcuts have none. `doctor` shows the seat in `capabilities.background_seat`. On GNOME Wayland, `click`, `drag` and `scroll` with `delivery: "background"`, a window target and points on that window use a virtual pen and touchscreen: the user's mouse stays put, but a click focuses the window, apps without tablet support ignore the pen (re-observe after the call), a drag stays in one window, and a scroll is a touch pan (`capabilities.background_pointer`). When no route serves a call, the first one in the session sends nothing and returns a notice (`background.retry_to_continue: true`) with the reason; repeating the call sends it through normal input, which moves the user's keyboard focus and pointer, and later calls with no route do that without asking again (`background.used: false`). Repeat only if taking the user's focus is acceptable; otherwise use the setting the reason names, or agent-workspace-linux (https://github.com/agent-sh/agent-workspace-linux), which gives the agent its own session with its own keyboard and pointer. A wrong or ambiguous target is still an error.
+- To act without taking the user's keyboard, pass `delivery: "background"` to `type_text` or `press_key` (naming an element or `browser_tab` implies it). Name an element from `get_app_state` (text is inserted and read back; Enter or space runs a button's, link's, check box's or single-line field's action), a `browser_tab` when `COMPUTER_USE_LINUX_DEVTOOLS` names the port of a browser the user started with `--remote-debugging-port`, or a terminal by `tty` or terminal selectors (kitty remote control, wezterm, tmux or GNU screen). On X11 and sway 1.10+ a plain window target also works: the agent gets its own keyboard focus (an XInput 2 master or a transient seat), and `click`/`scroll` with `delivery: "background"`, `x`/`y` and a window target use the agent's own pointer (X11 only with no window manager). On sway, Xwayland, GTK and Chromium windows and Alt/Super chords have no background route; on X11, window manager shortcuts have none. `doctor` shows the seat in `capabilities.background_seat`. On GNOME Wayland and KDE Plasma Wayland (KWin 6.3+), `click`, `drag` and `scroll` with `delivery: "background"`, a window target and points on that window use a virtual pen and touchscreen: the user's mouse stays put, but a click focuses the window, apps without tablet support ignore the pen (re-observe after the call), a drag stays in one window, and a scroll is a touch pan (`capabilities.background_pointer`). When no route serves a call, the first one in the session sends nothing and returns a notice (`background.retry_to_continue: true`) with the reason; repeating the call sends it through normal input, which moves the user's keyboard focus and pointer, and later calls with no route do that without asking again (`background.used: false`). Repeat only if taking the user's focus is acceptable; otherwise use the setting the reason names, or agent-workspace-linux (https://github.com/agent-sh/agent-workspace-linux), which gives the agent its own session with its own keyboard and pointer. A wrong or ambiguous target is still an error.
- Desktop input is stateful. Avoid concurrent tool calls against this MCP server.
- Pi serializes the native Computer Use tools and keeps one process for the session. If that process exits, do not replay an ambiguous mutating call; obtain a fresh `get_app_state` before another element-based action.
- `click`, `drag`, `press_key`, `type_text`, `perform_action`, and `set_value` can change real application state.
diff --git a/scripts/virtual_pen_kwin.Dockerfile b/scripts/virtual_pen_kwin.Dockerfile
new file mode 100644
index 00000000..6d8312e3
--- /dev/null
+++ b/scripts/virtual_pen_kwin.Dockerfile
@@ -0,0 +1,12 @@
+# SPDX-License-Identifier: MIT
+# KWin 6.6 for scripts/virtual_pen_kwin_test.py: kwin_wayland on a DRM
+# device, GTK 4 for the target window, and the KDE config tools the
+# virtual pen uses.
+FROM ubuntu:26.04
+ENV DEBIAN_FRONTEND=noninteractive
+RUN apt-get update \
+ && apt-get install -y --no-install-recommends \
+ kwin-wayland kwin-common kwin-style-breeze breeze-cursor-theme \
+ dbus libglib2.0-bin qdbus-qt6 libkscreen-bin qt6-wayland libkf6config-bin \
+ python3 python3-gi gir1.2-gtk-4.0 libgtk-4-1 fonts-dejavu-core \
+ && rm -rf /var/lib/apt/lists/*
diff --git a/scripts/virtual_pen_kwin_test.py b/scripts/virtual_pen_kwin_test.py
new file mode 100644
index 00000000..7f33f012
--- /dev/null
+++ b/scripts/virtual_pen_kwin_test.py
@@ -0,0 +1,352 @@
+#!/usr/bin/python3
+# SPDX-License-Identifier: MIT
+"""delivery="background" pointer input through the virtual pen on KWin.
+
+KWin 6.6 (Ubuntu 26.04, in a container) runs on a vkms output with its DRM
+backend, so libinput reads the machine's input devices, the virtual pen's
+among them; KWin's virtual backend reads none. Opt-in with
+COMPUTER_USE_LINUX_TEST_KWIN_SEAT=1: it needs docker, passwordless sudo
+(modprobe vkms) and no graphical session on seat0, since uinput devices are
+machine-wide and KWin opens every input device. Without the variable it
+skips.
+
+In the container a GTK 4 window logs every event with its device's source.
+A relative mouse nudge shows where the user's pointer is. Background left, right,
+middle and double clicks, a drag and two scrolls go through the pen and the
+touchscreen: the window sees pen and touch input with the buttons named and
+no mouse event, and a second nudge shows the user's pointer where it was. A point under another window is refused before
+input, and so is a call while KWin shows the desktop or on a minimized
+target. While the pen runs, kcminputrc holds its workspace mapping; after the
+server exits, no virtual device and none of its kcminputrc groups are left.
+With KWin's SyncWithMouse tablet option on, there is no route.
+"""
+import contextlib
+import json
+import os
+from pathlib import Path
+import shutil
+import subprocess
+import sys
+import time
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+from agent_seat_test import Server, check, stop, wait_for
+from socket_fixture import private_bus_config
+import virtual_pen_test as gnome
+
+HERE = Path(__file__).resolve().parent
+SHOW_DESKTOP = """
+import time
+from gi.repository import Gio, GLib
+bus = Gio.bus_get_sync(Gio.BusType.SESSION)
+bus.call_sync('org.kde.KWin', '/KWin', 'org.kde.KWin', 'showDesktop', GLib.Variant('(b)', (True,)),
+ None, Gio.DBusCallFlags.NONE, 2000)
+time.sleep(600)
+"""
+MINIMIZED = """
+import gi
+gi.require_version('Gtk', '4.0')
+from gi.repository import GLib, Gtk
+app = Gtk.Application(application_id='dev.avifenesh.CulPenMinimized')
+def activate(app):
+ window = Gtk.ApplicationWindow(application=app, title='cul-pen-minimized')
+ window.set_default_size(300, 300)
+ window.present()
+ GLib.timeout_add(1000, lambda: window.minimize() and False)
+app.connect('activate', activate)
+app.run([])
+"""
+SCREENSHOT = """
+import struct, sys, zlib
+from gi.repository import Gio, GLib
+width, height = int(sys.argv[1]), int(sys.argv[2])
+def png(path):
+ def chunk(kind, data):
+ return struct.pack('>I', len(data)) + kind + data + struct.pack('>I', zlib.crc32(kind + data))
+ raw = b''.join(b'\\0' + b'\\0' * (3 * width) for _ in range(height))
+ open(path, 'wb').write(b'\\x89PNG\\r\\n\\x1a\\n' + chunk(b'IHDR', struct.pack('>IIBBBBB', width, height, 8, 2, 0, 0, 0))
+ + chunk(b'IDAT', zlib.compress(raw)) + chunk(b'IEND', b''))
+XML = '''
+
+'''
+def call(connection, sender, path, interface, method, parameters, invocation):
+ filename = parameters.unpack()[2]
+ png(filename)
+ invocation.return_value(GLib.Variant('(bs)', (True, filename)))
+def acquired(connection, name):
+ connection.register_object('/org/gnome/Shell/Screenshot', Gio.DBusNodeInfo.new_for_xml(XML).interfaces[0],
+ call, None, None)
+Gio.bus_own_name(Gio.BusType.SESSION, 'org.gnome.Shell.Screenshot', Gio.BusNameOwnerFlags.NONE, acquired, None, None)
+GLib.MainLoop().run()
+"""
+IMAGE = 'computer-use-linux-kwin-pen:local'
+SOCKET = 'cul-kwin-0'
+
+
+def guest(binary):
+ failures = []
+ out = Path('/out')
+ runtime = Path('/run/user/0')
+ runtime.mkdir(parents=True, exist_ok=True)
+ os.chmod(runtime, 0o700)
+ env = dict(os.environ, XDG_RUNTIME_DIR=str(runtime), XDG_SESSION_TYPE='wayland', XDG_CURRENT_DESKTOP='KDE',
+ COMPUTER_USE_LINUX_INDICATOR='0', AT_SPI_BUS_ADDRESS='unix:path=/nonexistent', NO_AT_BRIDGE='1')
+ env.pop('DISPLAY', None)
+ with contextlib.ExitStack() as cleanup:
+ config = private_bus_config(out / 'bus.conf', runtime)
+ bus = subprocess.Popen(['dbus-daemon', '--config-file', str(config), '--nofork', '--print-address=1'],
+ stdout=subprocess.PIPE, text=True, env=env)
+ cleanup.callback(stop, bus)
+ env['DBUS_SESSION_BUS_ADDRESS'] = bus.stdout.readline().strip()
+ kwin = subprocess.Popen(['kwin_wayland', '--drm', '--socket', SOCKET, '--no-lockscreen',
+ '--no-global-shortcuts'],
+ env=dict(env, KWIN_DRM_DEVICES=os.environ['CARD']),
+ stdout=(out / 'kwin.log').open('w'), stderr=subprocess.STDOUT)
+ cleanup.callback(stop, kwin)
+ wait_for(lambda: (runtime / SOCKET).exists(), 'the KWin socket', 30)
+ env['WAYLAND_DISPLAY'] = SOCKET
+ # Plasma's screenshots come from the desktop portal, which asks the
+ # user; here a stand-in answers the server's first screenshot route
+ # with a blank image of the output's size, so capture pixels are
+ # layout pixels as on a 100% Plasma desktop.
+ mode = next(Path('/sys/class/drm').glob(Path(os.environ['CARD']).name + '-*/modes')).read_text().split()[0]
+ width, height = (int(side) for side in mode.split('x'))
+ shot = subprocess.Popen(['/usr/bin/python3', '-c', SCREENSHOT, str(width), str(height)], env=env)
+ cleanup.callback(stop, shot)
+ owner = ['gdbus', 'call', '--session', '--dest', 'org.freedesktop.DBus', '--object-path',
+ '/org/freedesktop/DBus', '--method', 'org.freedesktop.DBus.NameHasOwner', 'org.gnome.Shell.Screenshot']
+ wait_for(lambda: 'true' in subprocess.run(owner, env=env, capture_output=True, text=True).stdout,
+ 'the screenshot stand-in', 10)
+ root = out / 'gtk'
+ root.mkdir(exist_ok=True)
+ app = subprocess.Popen(['/usr/bin/python3', str(HERE / 'virtual_pen_test.py'), '--gtk', str(root)],
+ env=dict(env, GDK_BACKEND='wayland', GTK_A11Y='none'),
+ stdout=(out / 'app.log').open('w'), stderr=subprocess.STDOUT)
+ cleanup.callback(stop, app)
+ geometry = json.loads(wait_for(lambda: (root / 'geometry.json').exists()
+ and (root / 'geometry.json').read_text(), 'GTK window geometry', 20))
+ server = Server(binary, env)
+ cleanup.callback(server.close)
+ def background_pointer(server):
+ return (server.call('doctor', {}).get('capabilities') or {}).get('background_pointer') or {}
+ report = background_pointer(server)
+ print('doctor', json.dumps(report)[:300], flush=True)
+ check(failures, report.get('routes') == ['kwin-pen', 'kwin-touch'], f'doctor offers the KWin routes: {report}')
+ target = next(w for w in server.call('list_windows', {})['windows'] if w.get('title') == 'cul-pen-target')
+ wid = target['window_id']
+ check(failures, server.call('activate_window', dict(window_id=wid)).get('ok'), 'activate the GTK window')
+ time.sleep(1)
+ shot = server.call('screenshot', dict(max_width=64))
+ check(failures, shot.get('coordinate_width'), f'screenshot: {json.dumps(shot)[:1500]}')
+ bounds = next(w for w in server.call('list_windows', {})['windows'] if w['window_id'] == wid)['bounds']
+ origin = (bounds['x'], bounds['y'])
+ print(f'window at {bounds}, geometry {geometry}', flush=True)
+ px, py, _, _ = geometry['pad']
+ sx, sy, sw, sh = geometry['scroller']
+
+ def at(local):
+ return dict(x=int(origin[0] + local[0]), y=int(origin[1] + local[1]))
+
+ def events(since=0.0):
+ return gnome.events(root, since)
+ # Where the user's pointer is: a relative nudge shows it.
+ since = time.monotonic()
+ gnome.nudge_mouse()
+ user_point = wait_for(lambda: [(e['x'], e['y']) for e in events(since)
+ if e['kind'] == 'motion' and e['source'] == 'mouse'],
+ 'mouse motion before the pen', 5)[-1]
+ print('user pointer', user_point, flush=True)
+ pen_start = time.monotonic()
+
+ def background(tool, **arguments):
+ result = server.call(tool, dict(arguments, window_id=wid, delivery='background'))
+ report = result.get('background') or {}
+ check(failures, result.get('ok') and report.get('used') is True,
+ f'background {tool} {arguments}: {result.get("message")} {report}')
+ return result
+
+ def pen_presses(since, button, point):
+ return [e for e in events(since) if e['kind'] == 'press' and e['source'] == 'pen'
+ and e.get('button') == button and gnome.near(e, point)]
+ rc = Path('/root/.config/kcminputrc')
+ for button, code, offset in (('left', 1, 100), ('right', 3, 160), ('middle', 2, 220)):
+ point = (px + offset, py + 120)
+ since = time.monotonic()
+ result = background('click', **at(point), button=button)
+ check(failures, (result.get('background') or {}).get('route') == 'kwin-pen',
+ f'{button} route: {result.get("background")}')
+ time.sleep(.4)
+ presses = pen_presses(since, code, point)
+ check(failures, len(presses) == 1,
+ f'{button} click: one pen press with button {code} at {point}: {events(since)[:8]}')
+ check(failures, presses and [e for e in events(presses[0]['t']) if e['kind'] == 'proximity-out'],
+ f'{button} click: the pen left proximity after it')
+ if button == 'left':
+ text = rc.read_text() if rc.exists() else ''
+ check(failures, 'computer-use-linux virtual pen]' in text and 'MapToWorkspace=true' in text,
+ f'the pen maps to the whole workspace: {text}')
+ point = (px + 280, py + 120)
+ since = time.monotonic()
+ background('click', **at(point), click_count=2)
+ time.sleep(.4)
+ check(failures, len(pen_presses(since, 1, point)) == 2, f'double click: two pen presses at {point}')
+ start, end = (px + 100, py + 200), (px + 300, py + 260)
+ since = time.monotonic()
+ background('drag', start_x=at(start)['x'], start_y=at(start)['y'], end_x=at(end)['x'], end_y=at(end)['y'])
+ time.sleep(.4)
+ seen = events(since)
+ check(failures, any(e['kind'] == 'press' and e['source'] == 'pen' and gnome.near(e, start) for e in seen)
+ and any(e['kind'] == 'release' and e['source'] == 'pen' and gnome.near(e, end) for e in seen)
+ and sum(e['kind'] == 'motion' and e['source'] == 'pen' for e in seen) >= 5,
+ f'drag: pen press at {start}, motion, release at {end}: {seen[:3]} ... {seen[-3:]}')
+ point = (sx + sw // 2, sy + sh // 2)
+ before = 0
+ for name in ('scroll', 'scroll after a pen click'):
+ if name != 'scroll':
+ background('click', **at((px + 200, py + 400)))
+ since = time.monotonic()
+ result = background('scroll', **at(point), direction='down', pages=1)
+ check(failures, (result.get('background') or {}).get('route') == 'kwin-touch',
+ f'{name} route: {result.get("background")} {result.get("message")}')
+ time.sleep(1)
+ seen = events(since)
+ scrolled = [e['value'] for e in seen if e['kind'] == 'scrolled']
+ check(failures, any(e['kind'] == 'touch-begin' and e['source'] == 'touch' for e in seen)
+ and scrolled and max(scrolled) > before + 100, f'{name}: touch pan moved the list: {scrolled[-3:]}')
+ before = max(scrolled) if scrolled else before
+ # KWin re-sends the pointer's position when the surface under it
+ # changes focus; the pointer itself must not move or click.
+ mouse = [e for e in events(pen_start) if e.get('source') == 'mouse'
+ and (e['kind'] != 'motion' or abs(e['x'] - user_point[0]) > 2 or abs(e['y'] - user_point[1]) > 2)]
+ check(failures, not mouse, f'no mouse movement or button during pen input: {mouse[:5]}')
+ since = time.monotonic()
+ gnome.nudge_mouse()
+ moved = wait_for(lambda: [e for e in events(since) if e['kind'] == 'motion' and e['source'] == 'mouse'],
+ 'mouse motion after the nudge', 5)
+ print('pointer after the nudge', moved[:2], flush=True)
+ check(failures, abs(moved[-1]['x'] - user_point[0]) <= 2 and abs(moved[-1]['y'] - user_point[1]) <= 2,
+ f'user pointer still at {user_point}: {moved[-2:]}')
+ cover = subprocess.Popen(['/usr/bin/python3', '-c', gnome.COVER], env=dict(env, GDK_BACKEND='wayland'),
+ stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+ cleanup.callback(stop, cover)
+ cover_window = wait_for(lambda: next((w for w in server.call('list_windows', {})['windows']
+ if w.get('title') == 'cul-pen-cover' and w.get('bounds')), None),
+ 'the cover window', 20)
+ time.sleep(1)
+ b = cover_window['bounds']
+ covered = (b['x'] + b['width'] // 2, b['y'] + b['height'] // 2)
+ since = time.monotonic()
+ refused = server.call('click', dict(x=covered[0], y=covered[1], window_id=wid, delivery='background'))
+ check(failures, refused.get('ok') is False and 'stacked above' in refused.get('message', ''),
+ f'a covered point is refused: {refused.get("message")}')
+ time.sleep(.5)
+ check(failures, not [e for e in events(since) if e['kind'] == 'press'], 'refused click sent no press')
+ stop(cover)
+ # Show Desktop hides windows without minimizing them.
+ # KWin ends Show Desktop when the client that asked leaves the bus.
+ shower = subprocess.Popen(['/usr/bin/python3', '-c', SHOW_DESKTOP], env=env)
+ cleanup.callback(stop, shower)
+ time.sleep(1.5)
+ since = time.monotonic()
+ refused = server.call('click', dict(at((px + 100, py + 300)), window_id=wid, delivery='background'))
+ check(failures, refused.get('ok') is False and 'showing the desktop' in refused.get('message', ''),
+ f'Show Desktop is refused: {refused.get("message")}')
+ stop(shower)
+ time.sleep(1)
+ check(failures, not [e for e in events(since) if e['kind'] == 'press'], 'Show Desktop click sent no press')
+ # A minimized window is no target.
+ hidden = subprocess.Popen(['/usr/bin/python3', '-c', MINIMIZED], env=dict(env, GDK_BACKEND='wayland'),
+ stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+ cleanup.callback(stop, hidden)
+ minimized = wait_for(lambda: next((w for w in server.call('list_windows', {})['windows']
+ if w.get('title') == 'cul-pen-minimized' and w.get('hidden')), None),
+ 'the minimized window', 20)
+ refused = server.call('click', dict(at((px + 100, py + 300)), window_id=minimized['window_id'],
+ delivery='background'))
+ check(failures, refused.get('ok') is False and 'minimized' in refused.get('message', ''),
+ f'a minimized target is refused: {refused.get("message")}')
+ stop(hidden)
+ # The on-screen indicator's overlay (its pill sits over the window's
+ # top rows) takes no input and must not count as cover, on the
+ # second call too, once the overlay is up.
+ if (Path(binary).parent / 'computer-use-linux-indicator').exists():
+ shown = Server(binary, {key: value for key, value in env.items() if key != 'COMPUTER_USE_LINUX_INDICATOR'})
+ cleanup.callback(shown.close)
+ under_pill = (px + 400, py + 3)
+ for attempt in (1, 2):
+ since = time.monotonic()
+ result = shown.call('click', dict(at(under_pill), window_id=wid, delivery='background'))
+ time.sleep(.6)
+ check(failures, result.get('ok') and pen_presses(since, 1, under_pill),
+ f'click {attempt} under the indicator overlay: {result.get("message")}')
+ shown.close()
+ else:
+ print('SKIP indicator overlay: no computer-use-linux-indicator next to the server', flush=True)
+ server.close()
+ time.sleep(1)
+ check(failures, not gnome.leftover_devices(), f'no virtual device after exit: {gnome.leftover_devices()}')
+ text = rc.read_text() if rc.exists() else ''
+ check(failures, 'computer-use-linux virtual' not in text, f'no kcminputrc group left: {text}')
+ # The tablet cursor following the mouse would move the user's pointer.
+ subprocess.run(['kwriteconfig6', '--file', 'kcminputrc', '--group', 'Tablet', '--key', 'SyncWithMouse',
+ 'true'], env=env, check=True)
+ synced = Server(binary, env)
+ cleanup.callback(synced.close)
+ report = background_pointer(synced)
+ check(failures, report.get('available') is False and 'SyncWithMouse' in report.get('reason', ''),
+ f'SyncWithMouse leaves no route: {report}')
+ synced.close()
+ return failures
+
+
+def host(binary):
+ if not os.environ.get('COMPUTER_USE_LINUX_TEST_KWIN_SEAT'):
+ print('SKIP kwin pen: set COMPUTER_USE_LINUX_TEST_KWIN_SEAT=1 on a machine with docker and no desktop session')
+ return
+ missing = [tool for tool in ('docker', 'sudo') if not shutil.which(tool)]
+ if missing:
+ sys.exit(f'kwin pen fixture: missing {missing}')
+ if subprocess.run(['sudo', '-n', 'true'], capture_output=True).returncode != 0:
+ sys.exit('kwin pen fixture: needs passwordless sudo')
+ busy = gnome.seat0_graphical_sessions()
+ if busy:
+ sys.exit(f'kwin pen fixture: seat0 has graphical sessions {busy}; uinput input would reach them')
+ subprocess.run(['docker', 'build', '-q', '-t', IMAGE, '-f', str(HERE / 'virtual_pen_kwin.Dockerfile'), str(HERE)],
+ check=True, stdout=subprocess.DEVNULL)
+ cache = Path(os.environ.get('XDG_CACHE_HOME', Path.home() / '.cache'))
+ out = Path(os.environ.get('CUL_PEN_TEST_KEEP_LOGS') or cache / f'cul-kwin-pen-{os.getpid()}')
+ out.mkdir(parents=True, exist_ok=True)
+ loaded = Path('/sys/module/vkms').exists()
+ subprocess.run(['sudo', '-n', 'modprobe', 'vkms'], check=True)
+ try:
+ subprocess.run(['sudo', '-n', 'udevadm', 'settle'], check=True)
+ card = str(Path('/dev/dri/by-path/platform-vkms-card').resolve())
+ binary = Path(binary).resolve()
+ # --network host: libinput learns of new devices through udev's
+ # netlink group, which a separate network namespace never hears.
+ status = subprocess.run(['docker', 'run', '--rm', '--privileged', '--network', 'host',
+ '-v', '/dev:/dev', '-v', '/run/udev:/run/udev:ro',
+ '-v', f'{HERE}:/scripts:ro', '-v', f'{binary.parent}:/cul:ro', '-v', f'{out}:/out',
+ '-e', f'CARD={card}', IMAGE, 'python3', '/scripts/virtual_pen_kwin_test.py',
+ '--guest', f'/cul/{binary.name}'], timeout=600).returncode
+ finally:
+ # The GNOME fixture loads vkms under its own udev rule, and udev keeps
+ # a device's tags until it goes away: leave none behind.
+ if not loaded and not gnome.unload_vkms():
+ print('kwin pen fixture: vkms is still in use and stays loaded', flush=True)
+ if not os.environ.get('CUL_PEN_TEST_KEEP_LOGS'):
+ subprocess.run(['docker', 'run', '--rm', '-v', f'{out}:/out', IMAGE, 'sh', '-c', 'rm -rf /out/*'],
+ capture_output=True)
+ out.rmdir()
+ if status != 0:
+ sys.exit('kwin pen fixture failed')
+ print('kwin pen: all checks passed', flush=True)
+
+
+if __name__ == '__main__':
+ if sys.argv[1] == '--guest':
+ failures = guest(sys.argv[2])
+ if failures:
+ sys.exit(f'{len(failures)} kwin pen check(s) failed')
+ else:
+ host(sys.argv[1])
diff --git a/scripts/virtual_pen_test.py b/scripts/virtual_pen_test.py
index 15fb355b..73cc8137 100755
--- a/scripts/virtual_pen_test.py
+++ b/scripts/virtual_pen_test.py
@@ -184,6 +184,18 @@ def nudge_mouse():
fcntl.ioctl(uinput, 0x5502)
+def unload_vkms(timeout=10):
+ """Removes vkms, retrying while its last user lets go; whether it is gone."""
+ deadline = time.monotonic() + timeout
+ while True:
+ subprocess.run(['sudo', '-n', 'modprobe', '-r', 'vkms'], capture_output=True)
+ if not Path('/sys/module/vkms').exists():
+ return True
+ if time.monotonic() > deadline:
+ return False
+ time.sleep(.5)
+
+
def seat0_graphical_sessions():
sessions = []
listing = subprocess.run(['loginctl', 'list-sessions', '--no-legend'], capture_output=True, text=True).stdout
@@ -237,10 +249,15 @@ def run(binary, root, tty):
rule = Path('/run/udev/rules.d/61-mutter.rules')
subprocess.run(['sudo', '-n', 'tee', str(rule)], input=UDEV_RULE, text=True, check=True,
stdout=subprocess.DEVNULL)
- subprocess.run(['sudo', '-n', 'modprobe', '-r', 'vkms'], capture_output=True)
+ # udevd notices new rules through inotify, which a bind mount of
+ # /run/udev (the KWin fixture's container) can leave deaf.
+ subprocess.run(['sudo', '-n', 'udevadm', 'control', '--reload'], check=True)
+ if not unload_vkms():
+ sys.exit('gnome pen fixture: vkms is loaded and in use, so it cannot be reloaded under the rule')
subprocess.run(['sudo', '-n', 'modprobe', 'vkms'], check=True)
- cleanup.callback(subprocess.run, ['sudo', '-n', 'modprobe', '-r', 'vkms'], capture_output=True)
+ cleanup.callback(unload_vkms)
cleanup.callback(lambda: (subprocess.run(['sudo', '-n', 'rm', '-f', str(rule)]),
+ subprocess.run(['sudo', '-n', 'udevadm', 'control', '--reload']),
subprocess.run(['sudo', '-n', 'udevadm', 'trigger', '-s', 'drm', '--settle'])))
subprocess.run(['sudo', '-n', 'udevadm', 'trigger', '-s', 'drm', '--settle'], check=True)
user = subprocess.run(['id', '-un'], capture_output=True, text=True, check=True).stdout.strip()
diff --git a/skills/computer-use-linux/SKILL.md b/skills/computer-use-linux/SKILL.md
index 4567672f..0686ff05 100644
--- a/skills/computer-use-linux/SKILL.md
+++ b/skills/computer-use-linux/SKILL.md
@@ -236,7 +236,7 @@ directly to a window-relative click.
- `mcp --listen ADDR:PORT` serves streamable HTTP at `/mcp` and requires a token from `COMPUTER_USE_LINUX_MCP_TOKEN_FILE` or `COMPUTER_USE_LINUX_MCP_TOKEN`. A non-loopback bind needs `--allowed-host NAME`. The port is plain HTTP; put TLS in front. Do not retry a 401 by putting the token in a URL. Over HTTP, `get_tree_page` and index and selector calls need `snapshot_id`; if refused, call `get_app_state` again. Idle sessions expire after an hour; on `404 Session not found`, initialize again. The cached tree and its `snapshot_id` survive.
- On GNOME Wayland, the remote-control portal asks on each new `computer-use-linux` process. `COMPUTER_USE_LINUX_PERSIST_REMOTE_DESKTOP=1` in the server environment asks the portal to remember the grant (GNOME starts with the remember box checked). Later processes reuse a single-use restore token stored mode 0600 in the user state directory. Unset is the default and prompts every time. The flag does nothing when ydotool, not the portal, is the input backend.
- On Hyprland, `COMPUTER_USE_LINUX_HYPRLAND_BACKGROUND=1` lets targeted `type_text`, `press_key`, `get_app_state` and `screenshot` work on a window without focusing it, including one on a hidden workspace. Pass `window_id` (or another target) on every call. An AT-SPI action can move focus inside the app, so re-focus a field with its `activate` action before typing into it. The active Hyprland keymap must be US, or background `type_text` and `press_key` calls are refused; background `get_app_state` and `screenshot` still capture. Text must be printable ASCII, tab or newline. An unfocused Chromium ignores browser accelerators such as Ctrl+R, and Chromium joins AT-SPI only when started with `ACCESSIBILITY_ENABLED=1` if the accessibility bus reports itself disabled.
-- To act without taking the user's keyboard, pass `delivery: "background"` to `type_text` or `press_key` (naming an element or `browser_tab` implies it). Name an element from `get_app_state` (text is inserted and read back; Enter or space runs a button's, link's, check box's or single-line field's action), a `browser_tab` when `COMPUTER_USE_LINUX_DEVTOOLS` names the port of a browser the user started with `--remote-debugging-port`, or a terminal by `tty` or terminal selectors (kitty remote control, wezterm, tmux or GNU screen). On X11 and sway 1.10+ a plain window target also works: the agent gets its own keyboard focus (an XInput 2 master or a transient seat), and `click`/`scroll` with `delivery: "background"`, `x`/`y` and a window target use the agent's own pointer (X11 only with no window manager). On sway, Xwayland, GTK and Chromium windows and Alt/Super chords have no background route; on X11, window manager shortcuts have none. `doctor` shows the seat in `capabilities.background_seat`. On GNOME Wayland, `click`, `drag` and `scroll` with `delivery: "background"`, a window target and points on that window use a virtual pen and touchscreen: the user's mouse stays put, but a click focuses the window, apps without tablet support ignore the pen (re-observe after the call), a drag stays in one window, and a scroll is a touch pan (`capabilities.background_pointer`). When no route serves a call, the first one in the session sends nothing and returns a notice (`background.retry_to_continue: true`) with the reason; repeating the call sends it through normal input, which moves the user's keyboard focus and pointer, and later calls with no route do that without asking again (`background.used: false`). Repeat only if taking the user's focus is acceptable; otherwise use the setting the reason names, or agent-workspace-linux (https://github.com/agent-sh/agent-workspace-linux), which gives the agent its own session with its own keyboard and pointer. A wrong or ambiguous target is still an error.
+- To act without taking the user's keyboard, pass `delivery: "background"` to `type_text` or `press_key` (naming an element or `browser_tab` implies it). Name an element from `get_app_state` (text is inserted and read back; Enter or space runs a button's, link's, check box's or single-line field's action), a `browser_tab` when `COMPUTER_USE_LINUX_DEVTOOLS` names the port of a browser the user started with `--remote-debugging-port`, or a terminal by `tty` or terminal selectors (kitty remote control, wezterm, tmux or GNU screen). On X11 and sway 1.10+ a plain window target also works: the agent gets its own keyboard focus (an XInput 2 master or a transient seat), and `click`/`scroll` with `delivery: "background"`, `x`/`y` and a window target use the agent's own pointer (X11 only with no window manager). On sway, Xwayland, GTK and Chromium windows and Alt/Super chords have no background route; on X11, window manager shortcuts have none. `doctor` shows the seat in `capabilities.background_seat`. On GNOME Wayland and KDE Plasma Wayland (KWin 6.3+), `click`, `drag` and `scroll` with `delivery: "background"`, a window target and points on that window use a virtual pen and touchscreen: the user's mouse stays put, but a click focuses the window, apps without tablet support ignore the pen (re-observe after the call), a drag stays in one window, and a scroll is a touch pan (`capabilities.background_pointer`). When no route serves a call, the first one in the session sends nothing and returns a notice (`background.retry_to_continue: true`) with the reason; repeating the call sends it through normal input, which moves the user's keyboard focus and pointer, and later calls with no route do that without asking again (`background.used: false`). Repeat only if taking the user's focus is acceptable; otherwise use the setting the reason names, or agent-workspace-linux (https://github.com/agent-sh/agent-workspace-linux), which gives the agent its own session with its own keyboard and pointer. A wrong or ambiguous target is still an error.
- Desktop input is stateful. Avoid concurrent tool calls against this MCP server.
- Pi serializes the native Computer Use tools and keeps one process for the session. If that process exits, do not replay an ambiguous mutating call; obtain a fresh `get_app_state` before another element-based action.
- `click`, `drag`, `press_key`, `type_text`, `perform_action`, and `set_value` can change real application state.
diff --git a/src/diagnostics.rs b/src/diagnostics.rs
index 812790aa..1712e778 100644
--- a/src/diagnostics.rs
+++ b/src/diagnostics.rs
@@ -102,7 +102,7 @@ pub struct CapabilityMap {
/// or the reason it has none.
pub background_seat: serde_json::Value,
/// Background click, drag and scroll on GNOME Wayland: a virtual pen
- /// (gnome-pen) and touchscreen (gnome-touch) that leave the user's mouse
+ /// (gnome-pen or kwin-pen) and touchscreen (gnome-touch or kwin-touch) that leave the user's mouse
/// pointer where it is, with their limits, or the reason there is none.
pub background_pointer: serde_json::Value,
/// The backend the tool will use by default for each selectable layer.
diff --git a/src/server.rs b/src/server.rs
index 619fe653..2a149149 100644
--- a/src/server.rs
+++ b/src/server.rs
@@ -2918,7 +2918,7 @@ impl ComputerUseLinux {
#[tool(
name = "drag",
output_schema = rmcp::handler::server::tool::schema_for_output::(),
- description = "Drag with the left button from one point to another in screenshot capture pixels. Takes no relative coordinates; a window target matters only with delivery=\"background\", which on GNOME Wayland drags with a virtual pen inside that window and leaves the user's pointer where it is (elsewhere the first such call returns a one-time notice, then later calls drag through the user's pointer). Pass screenshot_after (true, \"screen\", or \"window\") or include_tree=true to return post-action observation in the same result.",
+ description = "Drag with the left button from one point to another in screenshot capture pixels. Takes no relative coordinates; a window target matters only with delivery=\"background\", which on GNOME and KDE Plasma Wayland drags with a virtual pen inside that window and leaves the user's pointer where it is (elsewhere the first such call returns a one-time notice, then later calls drag through the user's pointer). Pass screenshot_after (true, \"screen\", or \"window\") or include_tree=true to return post-action observation in the same result.",
annotations(
read_only_hint = false,
destructive_hint = true,
@@ -2948,8 +2948,9 @@ impl ComputerUseLinux {
let received = Some(serde_json::json!(params.clone()));
let input_guard = Arc::clone(&self.input_operation_lock).lock_owned().await;
let outcome = match crate::virtual_pen::availability() {
- Ok(()) => {
+ Ok(desktop) => {
self.background_pen(
+ desktop,
"drag",
PenRequest::Drag {
start: (params.start_x, params.start_y),
@@ -2964,9 +2965,7 @@ impl ComputerUseLinux {
Err(reason) => {
drop(input_guard);
BackgroundOutcome::Unavailable {
- reason: format!(
- "a background drag goes through the GNOME virtual pen, and {reason}"
- ),
+ reason: format!("a background drag goes through the virtual pen, and {reason}"),
fallback: FocusFallback::default(),
}
}
@@ -5488,7 +5487,7 @@ struct ClickParams {
/// "background" sends the input through the agent's own pointer, so the
/// user's pointer and keyboard focus stay where they are: on X11 without
/// a window manager (an XInput 2 master pointer) and on sway 1.10 or later
- /// (a transient seat). On GNOME Wayland a virtual pen (a touch pan for
+ /// (a transient seat). On GNOME and KDE Plasma Wayland a virtual pen (a touch pan for
/// scroll) leaves the user's pointer alone, but a click still focuses the
/// window, and apps without tablet support ignore it. Needs x and y in
/// screenshot capture pixels over the window the call targets, without
@@ -5770,7 +5769,7 @@ struct ScrollParams {
/// "background" sends the input through the agent's own pointer, so the
/// user's pointer and keyboard focus stay where they are: on X11 without
/// a window manager (an XInput 2 master pointer) and on sway 1.10 or later
- /// (a transient seat). On GNOME Wayland a virtual pen (a touch pan for
+ /// (a transient seat). On GNOME and KDE Plasma Wayland a virtual pen (a touch pan for
/// scroll) leaves the user's pointer alone, but a click still focuses the
/// window, and apps without tablet support ignore it. Needs x and y in
/// screenshot capture pixels over the window the call targets, without
@@ -5855,7 +5854,7 @@ struct DragParams {
/// Window title substring, ignoring ASCII case.
#[serde(default)]
window_title: Option,
- /// "background" drags with a virtual pen on GNOME Wayland, so the
+ /// "background" drags with a virtual pen on GNOME and KDE Plasma Wayland, so the
/// user's pointer stays where it is; both points must be on the target
/// window, with no window stacked above them. Elsewhere the first such
/// call in a server session sends nothing and returns a notice;
@@ -10682,8 +10681,8 @@ struct BackgroundReport {
/// would go, through the user's keyboard focus and pointer.
used: bool,
/// The background route used: atspi-element, devtools-tab, terminal,
- /// x11-master, sway-transient-seat, hyprland-send-key-state, gnome-pen
- /// or gnome-touch.
+ /// x11-master, sway-transient-seat, hyprland-send-key-state, gnome-pen,
+ /// gnome-touch, kwin-pen or kwin-touch.
#[serde(skip_serializing_if = "Option::is_none")]
route: Option,
/// False on the one-time notice: nothing was sent.
@@ -10784,14 +10783,42 @@ enum PenRequest {
},
}
-/// What a pen or touch action did to the user's keyboard focus: mutter
-/// focuses the window a tip press or a touch lands on.
+/// The windows the virtual pen checks a point against, bottom to top.
+async fn pen_windows(
+ desktop: crate::virtual_pen::Desktop,
+) -> anyhow::Result> {
+ match desktop {
+ crate::virtual_pen::Desktop::Gnome => {
+ crate::windowing::backends::gnome::extension_windows_stacked().await
+ }
+ // The indicator's overlay surfaces take no input (an empty input
+ // region), so they cover nothing.
+ crate::virtual_pen::Desktop::Kwin => {
+ Ok(crate::windowing::backends::kwin::windows_stacked()
+ .await?
+ .into_iter()
+ .filter(|window| !window.pid.is_some_and(is_indicator_process))
+ .collect())
+ }
+ }
+}
+
+/// What a pen or touch action did to the user's keyboard focus: the
+/// compositor focuses the window a tip press or a touch lands on.
+/// Whether pid runs this server's on-screen indicator.
+fn is_indicator_process(pid: u32) -> bool {
+ std::fs::read_link(format!("/proc/{pid}/exe")).is_ok_and(|exe| {
+ exe.file_name()
+ .is_some_and(|name| name == "computer-use-linux-indicator")
+ })
+}
+
fn pen_focus_sentence(before: Option, after: Option