diff --git a/CHANGELOG.md b/CHANGELOG.md index 54940bcf..e1d0a6f8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Added + +- KDE Plasma Wayland (KWin 6.3 and later) gets the virtual pen and touchscreen for `click`, `drag` and `scroll` with `delivery: "background"`, as routes `kwin-pen` and `kwin-touch`: the user's mouse pointer stays where it is. KWin maps the pen to the whole workspace and the touchscreen to the output under the point through `org.kde.KWin.InputDevice`, read back before input, and the server removes what KWin wrote for these devices into `kcminputrc` once they are gone. The point is checked through KWin scripting against KWin's stacking order (panels, popups and notifications included), on the current desktop and activity; Show Desktop and full-screen effects such as the Overview refuse the call. KWin's `SyncWithMouse` tablet option and `KWIN_WAYLAND_EMULATE_TABLET`, which move or click the user's pointer, leave no route. Measured on KWin 6.6.7 on a DRM output with GTK 4. + ### Fixed - A background right click on GNOME 48 and later arrived as a middle click and a middle click as a right click: GNOME 48 changed the stylus button defaults (`button-action` to `right`, `secondary-button-action` to `middle`), and mutter rewrites the pen's buttons by them. The server now sets both actions for its own pen (the settings path keyed by its vendor and product id) before every action, so mutter hands clients the button the pen pressed, and resets them when the pen goes away. Measured with GNOME 48's defaults on GNOME Shell 46, with GTK 4 and with native Wayland and Xwayland Chromium; mutter 48 and later read the same explicit values (from mutter's source). diff --git a/README.md b/README.md index 26f5cbdf..d98f535b 100644 --- a/README.md +++ b/README.md @@ -181,7 +181,7 @@ On Hyprland with `COMPUTER_USE_LINUX_HYPRLAND_BACKGROUND=1`, a window target wit Where there is no agent seat, a window-targeted background call has no route (the notice, then focused input); element, browser tab and terminal targets still work. For full isolation, with its own keyboard and pointer, run the app in an [agent-workspace-linux](https://github.com/agent-sh/agent-workspace-linux) workspace; the notice points there. `doctor` lists the routes this machine has in `capabilities.background_input` (`seat` when the agent seat is available) and the seat's route, pointer support or the reason it has none in `capabilities.background_seat`. `capabilities.background_pointer` shows the GNOME pen route and its limits, or why there is none. -On GNOME Wayland, background `click`, `drag` and `scroll` go through a virtual pen and a virtual touchscreen instead of the user's pointer ([#480](https://github.com/agent-sh/computer-use-linux/issues/480); the idea and the tested approach are [@Ondra-Zik](https://github.com/Ondra-Zik)'s). Mutter gives a tablet tool its own cursor and sends its events through the tablet protocol, not `wl_pointer`, so the user's mouse stays where it is. Clicks are pen taps (left), the upper barrel button (right) and the lower one (middle); `click_count` repeats them. GNOME 48 and later default the lower button to a right click and the upper one to a middle click, so before each action the server sets its own pen's button actions to match the buttons it presses, and resets them when the pen goes away. A drag is the pen tip pressed at the start, moved and lifted at the end. The pen leaves proximity after each action: libinput pairs a pen with a touchscreen and ignores that screen's touches while the pen is near, and that screen can be the user's. A scroll is a one-finger touch pan, 60 logical pixels per wheel notch (5 notches per page), split into strokes that stay inside the window and 48 pixels from the monitor's edges, where GNOME Shell starts its edge swipes; the finger rests before it lifts, so the app does not fling further. Before any input, the server checks through the GNOME Shell extension that the point is on the target window, on the workspace the user sees, and that no window stacked above it covers the point; both ends of a drag and every pan stroke must be on the window, and a pan always starts at the point. The results report the route as `gnome-pen` or `gnome-touch`. The devices need write access to `/dev/uinput`, as the uinput pointer does; without it, or without the extension, the call gets the notice and then focused input. +On GNOME Wayland (and KDE Plasma, below), background `click`, `drag` and `scroll` go through a virtual pen and a virtual touchscreen instead of the user's pointer ([#480](https://github.com/agent-sh/computer-use-linux/issues/480); the idea and the tested approach are [@Ondra-Zik](https://github.com/Ondra-Zik)'s). Mutter gives a tablet tool its own cursor and sends its events through the tablet protocol, not `wl_pointer`, so the user's mouse stays where it is. Clicks are pen taps (left), the upper barrel button (right) and the lower one (middle); `click_count` repeats them. GNOME 48 and later default the lower button to a right click and the upper one to a middle click, so before each action the server sets its own pen's button actions to match the buttons it presses, and resets them when the pen goes away. A drag is the pen tip pressed at the start, moved and lifted at the end. The pen leaves proximity after each action: libinput pairs a pen with a touchscreen and ignores that screen's touches while the pen is near, and that screen can be the user's. A scroll is a one-finger touch pan, 60 logical pixels per wheel notch (5 notches per page), split into strokes that stay inside the window and 48 pixels from the monitor's edges, where GNOME Shell starts its edge swipes; the finger rests before it lifts, so the app does not fling further. Before any input, the server checks through the GNOME Shell extension that the point is on the target window, on the workspace the user sees, and that no window stacked above it covers the point; both ends of a drag and every pan stroke must be on the window, and a pan always starts at the point. The results report the route as `gnome-pen` or `gnome-touch`. The devices need write access to `/dev/uinput`, as the uinput pointer does; without it, or without the extension, the call gets the notice and then focused input. Limits: @@ -193,6 +193,8 @@ Limits: - Mutter maps an unconfigured tablet across the whole layout, and an unconfigured touchscreen to the builtin panel. When a device maps elsewhere than the point needs, the server writes that device's `output` setting (`org.gnome.desktop.peripherals.tablet` or `.touchscreen`, at the path keyed by the devices' own vendor and product ids, `6375:7065` and `6375:7463`), waits for mutter's `org.gnome.Mutter.InputMapping` to confirm the new mapping, and resets the setting when the device goes away. No other device's setting is touched. - The devices appear on first use and go away after ten idle minutes or when the server exits. +On KDE Plasma Wayland with KWin 6.3 or later the same route runs as `kwin-pen` and `kwin-touch`. KWin gives a tablet its own cursor from 6.3 (it tracks the tablet cursor itself and moves it whether or not the client under it supports tablets). KWin maps a tablet with no setting to the active output, which follows the user's pointer, and a touchscreen to the internal panel or the first output. So before input the server sets the pen's `mapToWorkspace` and the touchscreen's `outputName` (the output under the point) through `org.kde.KWin.InputDevice`, reads them back, and once the devices are gone removes what KWin wrote for them into `kcminputrc` (the groups keyed by these devices' vendor, product and name) with `kwriteconfig6`. The window check goes through KWin scripting: the point must be on the target, on the desktop and activity the user sees, not minimized or hidden, with no window, panel, popup or notification stacked above it. The call is refused while KWin shows the desktop or runs a full-screen effect such as the Overview (`org.kde.kwin.Effects.activeEffects`), and when another setting for the device moves where input lands (calibration, input or output area, rotation). A scroll reads the output's rotation with `kscreen-doctor` and refuses a rotated or flipped output; without `kscreen-doctor` there is no touch route. There is no route at all when KWin's `SyncWithMouse` tablet option (`kcminputrc` `[Tablet]`) is on, since it moves the user's pointer to where the pen was, when KWin runs with `KWIN_WAYLAND_EMULATE_TABLET`, which sends pen input over apps without tablet support through the user's pointer, or without `kreadconfig6` and `kwriteconfig6`. + Some clients mishandle a second `wl_seat` (Zed, [zed-industries/zed#66735](https://github.com/zed-industries/zed/issues/66735)); that is one more reason the sway seat exists only during a call. A menu the agent opens on sway closes when its call ends, since the seat that opened it goes away. While sway is in a binding mode other than `default`, background keys have no route, since the mode's bindings would take them. ### On-screen indicator diff --git a/pi/extension/generated-tools.ts b/pi/extension/generated-tools.ts index fc691251..ba447a08 100644 --- a/pi/extension/generated-tools.ts +++ b/pi/extension/generated-tools.ts @@ -8,8 +8,8 @@ export interface GeneratedMcpToolDefinition { } export const GENERATED_SERVER_VERSION = "0.7.58"; -export const GENERATED_TOOL_CATALOG_HASH = "e447f071f2ccf6c438c5fca9c80dbbe642661b1e9afa6239aad6534b2815ebfb"; -export const GENERATED_SHELL_TOOL_CATALOG_HASH = "edaac05bdf1eaca9f3ae95f7e0e7d9f41797c920ec7331c39e6864c04841fa51"; +export const GENERATED_TOOL_CATALOG_HASH = "de04450abc876a3712849d85daad7b80ed6656e1ddf21e2bfd16e94712b4247c"; +export const GENERATED_SHELL_TOOL_CATALOG_HASH = "26a00c32301af2704aeaf97fe3cb58f9429e255305ad4e17efd50db506cd6717"; export const GENERATED_MCP_TOOLS = [ { @@ -279,7 +279,7 @@ export const GENERATED_MCP_TOOLS = } ], "default": null, - "description": "\"background\" sends the input through the agent's own pointer, so the\nuser's pointer and keyboard focus stay where they are: on X11 without\na window manager (an XInput 2 master pointer) and on sway 1.10 or later\n(a transient seat). On GNOME Wayland a virtual pen (a touch pan for\nscroll) leaves the user's pointer alone, but a click still focuses the\nwindow, and apps without tablet support ignore it. Needs x and y in\nscreenshot capture pixels over the window the call targets, without\nrelative or an element. Elsewhere the\nfirst such call in a server session sends nothing and returns a\nnotice; repeated, and for every later such call, it goes through the\nuser's pointer as with \"focus\". Default: \"focus\"." + "description": "\"background\" sends the input through the agent's own pointer, so the\nuser's pointer and keyboard focus stay where they are: on X11 without\na window manager (an XInput 2 master pointer) and on sway 1.10 or later\n(a transient seat). On GNOME and KDE Plasma Wayland a virtual pen (a touch pan for\nscroll) leaves the user's pointer alone, but a click still focuses the\nwindow, and apps without tablet support ignore it. Needs x and y in\nscreenshot capture pixels over the window the call targets, without\nrelative or an element. Elsewhere the\nfirst such call in a server session sends nothing and returns a\nnotice; repeated, and for every later such call, it goes through the\nuser's pointer as with \"focus\". Default: \"focus\"." }, "element_index": { "default": null, @@ -501,7 +501,7 @@ export const GENERATED_MCP_TOOLS = "openWorldHint": true, "readOnlyHint": false }, - "description": "Drag with the left button from one point to another in screenshot capture pixels. Takes no relative coordinates; a window target matters only with delivery=\"background\", which on GNOME Wayland drags with a virtual pen inside that window and leaves the user's pointer where it is (elsewhere the first such call returns a one-time notice, then later calls drag through the user's pointer). Pass screenshot_after (true, \"screen\", or \"window\") or include_tree=true to return post-action observation in the same result.", + "description": "Drag with the left button from one point to another in screenshot capture pixels. Takes no relative coordinates; a window target matters only with delivery=\"background\", which on GNOME and KDE Plasma Wayland drags with a virtual pen inside that window and leaves the user's pointer where it is (elsewhere the first such call returns a one-time notice, then later calls drag through the user's pointer). Pass screenshot_after (true, \"screen\", or \"window\") or include_tree=true to return post-action observation in the same result.", "inputSchema": { "$defs": { "KeyboardDelivery": { @@ -547,7 +547,7 @@ export const GENERATED_MCP_TOOLS = } ], "default": null, - "description": "\"background\" drags with a virtual pen on GNOME Wayland, so the\nuser's pointer stays where it is; both points must be on the target\nwindow, with no window stacked above them. Elsewhere the first such\ncall in a server session sends nothing and returns a notice;\nrepeated, and for every later such call, the drag goes through the\nuser's pointer as with \"focus\". Default: \"focus\"." + "description": "\"background\" drags with a virtual pen on GNOME and KDE Plasma Wayland, so the\nuser's pointer stays where it is; both points must be on the target\nwindow, with no window stacked above them. Elsewhere the first such\ncall in a server session sends nothing and returns a notice;\nrepeated, and for every later such call, the drag goes through the\nuser's pointer as with \"focus\". Default: \"focus\"." }, "end_x": { "description": "End x in raw screenshot capture pixels.", @@ -2094,7 +2094,7 @@ export const GENERATED_MCP_TOOLS = } ], "default": null, - "description": "\"background\" sends the input through the agent's own pointer, so the\nuser's pointer and keyboard focus stay where they are: on X11 without\na window manager (an XInput 2 master pointer) and on sway 1.10 or later\n(a transient seat). On GNOME Wayland a virtual pen (a touch pan for\nscroll) leaves the user's pointer alone, but a click still focuses the\nwindow, and apps without tablet support ignore it. Needs x and y in\nscreenshot capture pixels over the window the call targets, without\nrelative or an element. Elsewhere the\nfirst such call in a server session sends nothing and returns a\nnotice; repeated, and for every later such call, it goes through the\nuser's pointer as with \"focus\". Default: \"focus\"." + "description": "\"background\" sends the input through the agent's own pointer, so the\nuser's pointer and keyboard focus stay where they are: on X11 without\na window manager (an XInput 2 master pointer) and on sway 1.10 or later\n(a transient seat). On GNOME and KDE Plasma Wayland a virtual pen (a touch pan for\nscroll) leaves the user's pointer alone, but a click still focuses the\nwindow, and apps without tablet support ignore it. Needs x and y in\nscreenshot capture pixels over the window the call targets, without\nrelative or an element. Elsewhere the\nfirst such call in a server session sends nothing and returns a\nnotice; repeated, and for every later such call, it goes through the\nuser's pointer as with \"focus\". Default: \"focus\"." }, "direction": { "description": "up, down, left or right (case-insensitive).", diff --git a/plugins/computer-use-linux/skills/computer-use-linux/SKILL.md b/plugins/computer-use-linux/skills/computer-use-linux/SKILL.md index 4567672f..0686ff05 100644 --- a/plugins/computer-use-linux/skills/computer-use-linux/SKILL.md +++ b/plugins/computer-use-linux/skills/computer-use-linux/SKILL.md @@ -236,7 +236,7 @@ directly to a window-relative click. - `mcp --listen ADDR:PORT` serves streamable HTTP at `/mcp` and requires a token from `COMPUTER_USE_LINUX_MCP_TOKEN_FILE` or `COMPUTER_USE_LINUX_MCP_TOKEN`. A non-loopback bind needs `--allowed-host NAME`. The port is plain HTTP; put TLS in front. Do not retry a 401 by putting the token in a URL. Over HTTP, `get_tree_page` and index and selector calls need `snapshot_id`; if refused, call `get_app_state` again. Idle sessions expire after an hour; on `404 Session not found`, initialize again. The cached tree and its `snapshot_id` survive. - On GNOME Wayland, the remote-control portal asks on each new `computer-use-linux` process. `COMPUTER_USE_LINUX_PERSIST_REMOTE_DESKTOP=1` in the server environment asks the portal to remember the grant (GNOME starts with the remember box checked). Later processes reuse a single-use restore token stored mode 0600 in the user state directory. Unset is the default and prompts every time. The flag does nothing when ydotool, not the portal, is the input backend. - On Hyprland, `COMPUTER_USE_LINUX_HYPRLAND_BACKGROUND=1` lets targeted `type_text`, `press_key`, `get_app_state` and `screenshot` work on a window without focusing it, including one on a hidden workspace. Pass `window_id` (or another target) on every call. An AT-SPI action can move focus inside the app, so re-focus a field with its `activate` action before typing into it. The active Hyprland keymap must be US, or background `type_text` and `press_key` calls are refused; background `get_app_state` and `screenshot` still capture. Text must be printable ASCII, tab or newline. An unfocused Chromium ignores browser accelerators such as Ctrl+R, and Chromium joins AT-SPI only when started with `ACCESSIBILITY_ENABLED=1` if the accessibility bus reports itself disabled. -- To act without taking the user's keyboard, pass `delivery: "background"` to `type_text` or `press_key` (naming an element or `browser_tab` implies it). Name an element from `get_app_state` (text is inserted and read back; Enter or space runs a button's, link's, check box's or single-line field's action), a `browser_tab` when `COMPUTER_USE_LINUX_DEVTOOLS` names the port of a browser the user started with `--remote-debugging-port`, or a terminal by `tty` or terminal selectors (kitty remote control, wezterm, tmux or GNU screen). On X11 and sway 1.10+ a plain window target also works: the agent gets its own keyboard focus (an XInput 2 master or a transient seat), and `click`/`scroll` with `delivery: "background"`, `x`/`y` and a window target use the agent's own pointer (X11 only with no window manager). On sway, Xwayland, GTK and Chromium windows and Alt/Super chords have no background route; on X11, window manager shortcuts have none. `doctor` shows the seat in `capabilities.background_seat`. On GNOME Wayland, `click`, `drag` and `scroll` with `delivery: "background"`, a window target and points on that window use a virtual pen and touchscreen: the user's mouse stays put, but a click focuses the window, apps without tablet support ignore the pen (re-observe after the call), a drag stays in one window, and a scroll is a touch pan (`capabilities.background_pointer`). When no route serves a call, the first one in the session sends nothing and returns a notice (`background.retry_to_continue: true`) with the reason; repeating the call sends it through normal input, which moves the user's keyboard focus and pointer, and later calls with no route do that without asking again (`background.used: false`). Repeat only if taking the user's focus is acceptable; otherwise use the setting the reason names, or agent-workspace-linux (https://github.com/agent-sh/agent-workspace-linux), which gives the agent its own session with its own keyboard and pointer. A wrong or ambiguous target is still an error. +- To act without taking the user's keyboard, pass `delivery: "background"` to `type_text` or `press_key` (naming an element or `browser_tab` implies it). Name an element from `get_app_state` (text is inserted and read back; Enter or space runs a button's, link's, check box's or single-line field's action), a `browser_tab` when `COMPUTER_USE_LINUX_DEVTOOLS` names the port of a browser the user started with `--remote-debugging-port`, or a terminal by `tty` or terminal selectors (kitty remote control, wezterm, tmux or GNU screen). On X11 and sway 1.10+ a plain window target also works: the agent gets its own keyboard focus (an XInput 2 master or a transient seat), and `click`/`scroll` with `delivery: "background"`, `x`/`y` and a window target use the agent's own pointer (X11 only with no window manager). On sway, Xwayland, GTK and Chromium windows and Alt/Super chords have no background route; on X11, window manager shortcuts have none. `doctor` shows the seat in `capabilities.background_seat`. On GNOME Wayland and KDE Plasma Wayland (KWin 6.3+), `click`, `drag` and `scroll` with `delivery: "background"`, a window target and points on that window use a virtual pen and touchscreen: the user's mouse stays put, but a click focuses the window, apps without tablet support ignore the pen (re-observe after the call), a drag stays in one window, and a scroll is a touch pan (`capabilities.background_pointer`). When no route serves a call, the first one in the session sends nothing and returns a notice (`background.retry_to_continue: true`) with the reason; repeating the call sends it through normal input, which moves the user's keyboard focus and pointer, and later calls with no route do that without asking again (`background.used: false`). Repeat only if taking the user's focus is acceptable; otherwise use the setting the reason names, or agent-workspace-linux (https://github.com/agent-sh/agent-workspace-linux), which gives the agent its own session with its own keyboard and pointer. A wrong or ambiguous target is still an error. - Desktop input is stateful. Avoid concurrent tool calls against this MCP server. - Pi serializes the native Computer Use tools and keeps one process for the session. If that process exits, do not replay an ambiguous mutating call; obtain a fresh `get_app_state` before another element-based action. - `click`, `drag`, `press_key`, `type_text`, `perform_action`, and `set_value` can change real application state. diff --git a/scripts/virtual_pen_kwin.Dockerfile b/scripts/virtual_pen_kwin.Dockerfile new file mode 100644 index 00000000..6d8312e3 --- /dev/null +++ b/scripts/virtual_pen_kwin.Dockerfile @@ -0,0 +1,12 @@ +# SPDX-License-Identifier: MIT +# KWin 6.6 for scripts/virtual_pen_kwin_test.py: kwin_wayland on a DRM +# device, GTK 4 for the target window, and the KDE config tools the +# virtual pen uses. +FROM ubuntu:26.04 +ENV DEBIAN_FRONTEND=noninteractive +RUN apt-get update \ + && apt-get install -y --no-install-recommends \ + kwin-wayland kwin-common kwin-style-breeze breeze-cursor-theme \ + dbus libglib2.0-bin qdbus-qt6 libkscreen-bin qt6-wayland libkf6config-bin \ + python3 python3-gi gir1.2-gtk-4.0 libgtk-4-1 fonts-dejavu-core \ + && rm -rf /var/lib/apt/lists/* diff --git a/scripts/virtual_pen_kwin_test.py b/scripts/virtual_pen_kwin_test.py new file mode 100644 index 00000000..7f33f012 --- /dev/null +++ b/scripts/virtual_pen_kwin_test.py @@ -0,0 +1,352 @@ +#!/usr/bin/python3 +# SPDX-License-Identifier: MIT +"""delivery="background" pointer input through the virtual pen on KWin. + +KWin 6.6 (Ubuntu 26.04, in a container) runs on a vkms output with its DRM +backend, so libinput reads the machine's input devices, the virtual pen's +among them; KWin's virtual backend reads none. Opt-in with +COMPUTER_USE_LINUX_TEST_KWIN_SEAT=1: it needs docker, passwordless sudo +(modprobe vkms) and no graphical session on seat0, since uinput devices are +machine-wide and KWin opens every input device. Without the variable it +skips. + +In the container a GTK 4 window logs every event with its device's source. +A relative mouse nudge shows where the user's pointer is. Background left, right, +middle and double clicks, a drag and two scrolls go through the pen and the +touchscreen: the window sees pen and touch input with the buttons named and +no mouse event, and a second nudge shows the user's pointer where it was. A point under another window is refused before +input, and so is a call while KWin shows the desktop or on a minimized +target. While the pen runs, kcminputrc holds its workspace mapping; after the +server exits, no virtual device and none of its kcminputrc groups are left. +With KWin's SyncWithMouse tablet option on, there is no route. +""" +import contextlib +import json +import os +from pathlib import Path +import shutil +import subprocess +import sys +import time + +sys.path.insert(0, str(Path(__file__).resolve().parent)) +from agent_seat_test import Server, check, stop, wait_for +from socket_fixture import private_bus_config +import virtual_pen_test as gnome + +HERE = Path(__file__).resolve().parent +SHOW_DESKTOP = """ +import time +from gi.repository import Gio, GLib +bus = Gio.bus_get_sync(Gio.BusType.SESSION) +bus.call_sync('org.kde.KWin', '/KWin', 'org.kde.KWin', 'showDesktop', GLib.Variant('(b)', (True,)), + None, Gio.DBusCallFlags.NONE, 2000) +time.sleep(600) +""" +MINIMIZED = """ +import gi +gi.require_version('Gtk', '4.0') +from gi.repository import GLib, Gtk +app = Gtk.Application(application_id='dev.avifenesh.CulPenMinimized') +def activate(app): + window = Gtk.ApplicationWindow(application=app, title='cul-pen-minimized') + window.set_default_size(300, 300) + window.present() + GLib.timeout_add(1000, lambda: window.minimize() and False) +app.connect('activate', activate) +app.run([]) +""" +SCREENSHOT = """ +import struct, sys, zlib +from gi.repository import Gio, GLib +width, height = int(sys.argv[1]), int(sys.argv[2]) +def png(path): + def chunk(kind, data): + return struct.pack('>I', len(data)) + kind + data + struct.pack('>I', zlib.crc32(kind + data)) + raw = b''.join(b'\\0' + b'\\0' * (3 * width) for _ in range(height)) + open(path, 'wb').write(b'\\x89PNG\\r\\n\\x1a\\n' + chunk(b'IHDR', struct.pack('>IIBBBBB', width, height, 8, 2, 0, 0, 0)) + + chunk(b'IDAT', zlib.compress(raw)) + chunk(b'IEND', b'')) +XML = ''' + +''' +def call(connection, sender, path, interface, method, parameters, invocation): + filename = parameters.unpack()[2] + png(filename) + invocation.return_value(GLib.Variant('(bs)', (True, filename))) +def acquired(connection, name): + connection.register_object('/org/gnome/Shell/Screenshot', Gio.DBusNodeInfo.new_for_xml(XML).interfaces[0], + call, None, None) +Gio.bus_own_name(Gio.BusType.SESSION, 'org.gnome.Shell.Screenshot', Gio.BusNameOwnerFlags.NONE, acquired, None, None) +GLib.MainLoop().run() +""" +IMAGE = 'computer-use-linux-kwin-pen:local' +SOCKET = 'cul-kwin-0' + + +def guest(binary): + failures = [] + out = Path('/out') + runtime = Path('/run/user/0') + runtime.mkdir(parents=True, exist_ok=True) + os.chmod(runtime, 0o700) + env = dict(os.environ, XDG_RUNTIME_DIR=str(runtime), XDG_SESSION_TYPE='wayland', XDG_CURRENT_DESKTOP='KDE', + COMPUTER_USE_LINUX_INDICATOR='0', AT_SPI_BUS_ADDRESS='unix:path=/nonexistent', NO_AT_BRIDGE='1') + env.pop('DISPLAY', None) + with contextlib.ExitStack() as cleanup: + config = private_bus_config(out / 'bus.conf', runtime) + bus = subprocess.Popen(['dbus-daemon', '--config-file', str(config), '--nofork', '--print-address=1'], + stdout=subprocess.PIPE, text=True, env=env) + cleanup.callback(stop, bus) + env['DBUS_SESSION_BUS_ADDRESS'] = bus.stdout.readline().strip() + kwin = subprocess.Popen(['kwin_wayland', '--drm', '--socket', SOCKET, '--no-lockscreen', + '--no-global-shortcuts'], + env=dict(env, KWIN_DRM_DEVICES=os.environ['CARD']), + stdout=(out / 'kwin.log').open('w'), stderr=subprocess.STDOUT) + cleanup.callback(stop, kwin) + wait_for(lambda: (runtime / SOCKET).exists(), 'the KWin socket', 30) + env['WAYLAND_DISPLAY'] = SOCKET + # Plasma's screenshots come from the desktop portal, which asks the + # user; here a stand-in answers the server's first screenshot route + # with a blank image of the output's size, so capture pixels are + # layout pixels as on a 100% Plasma desktop. + mode = next(Path('/sys/class/drm').glob(Path(os.environ['CARD']).name + '-*/modes')).read_text().split()[0] + width, height = (int(side) for side in mode.split('x')) + shot = subprocess.Popen(['/usr/bin/python3', '-c', SCREENSHOT, str(width), str(height)], env=env) + cleanup.callback(stop, shot) + owner = ['gdbus', 'call', '--session', '--dest', 'org.freedesktop.DBus', '--object-path', + '/org/freedesktop/DBus', '--method', 'org.freedesktop.DBus.NameHasOwner', 'org.gnome.Shell.Screenshot'] + wait_for(lambda: 'true' in subprocess.run(owner, env=env, capture_output=True, text=True).stdout, + 'the screenshot stand-in', 10) + root = out / 'gtk' + root.mkdir(exist_ok=True) + app = subprocess.Popen(['/usr/bin/python3', str(HERE / 'virtual_pen_test.py'), '--gtk', str(root)], + env=dict(env, GDK_BACKEND='wayland', GTK_A11Y='none'), + stdout=(out / 'app.log').open('w'), stderr=subprocess.STDOUT) + cleanup.callback(stop, app) + geometry = json.loads(wait_for(lambda: (root / 'geometry.json').exists() + and (root / 'geometry.json').read_text(), 'GTK window geometry', 20)) + server = Server(binary, env) + cleanup.callback(server.close) + def background_pointer(server): + return (server.call('doctor', {}).get('capabilities') or {}).get('background_pointer') or {} + report = background_pointer(server) + print('doctor', json.dumps(report)[:300], flush=True) + check(failures, report.get('routes') == ['kwin-pen', 'kwin-touch'], f'doctor offers the KWin routes: {report}') + target = next(w for w in server.call('list_windows', {})['windows'] if w.get('title') == 'cul-pen-target') + wid = target['window_id'] + check(failures, server.call('activate_window', dict(window_id=wid)).get('ok'), 'activate the GTK window') + time.sleep(1) + shot = server.call('screenshot', dict(max_width=64)) + check(failures, shot.get('coordinate_width'), f'screenshot: {json.dumps(shot)[:1500]}') + bounds = next(w for w in server.call('list_windows', {})['windows'] if w['window_id'] == wid)['bounds'] + origin = (bounds['x'], bounds['y']) + print(f'window at {bounds}, geometry {geometry}', flush=True) + px, py, _, _ = geometry['pad'] + sx, sy, sw, sh = geometry['scroller'] + + def at(local): + return dict(x=int(origin[0] + local[0]), y=int(origin[1] + local[1])) + + def events(since=0.0): + return gnome.events(root, since) + # Where the user's pointer is: a relative nudge shows it. + since = time.monotonic() + gnome.nudge_mouse() + user_point = wait_for(lambda: [(e['x'], e['y']) for e in events(since) + if e['kind'] == 'motion' and e['source'] == 'mouse'], + 'mouse motion before the pen', 5)[-1] + print('user pointer', user_point, flush=True) + pen_start = time.monotonic() + + def background(tool, **arguments): + result = server.call(tool, dict(arguments, window_id=wid, delivery='background')) + report = result.get('background') or {} + check(failures, result.get('ok') and report.get('used') is True, + f'background {tool} {arguments}: {result.get("message")} {report}') + return result + + def pen_presses(since, button, point): + return [e for e in events(since) if e['kind'] == 'press' and e['source'] == 'pen' + and e.get('button') == button and gnome.near(e, point)] + rc = Path('/root/.config/kcminputrc') + for button, code, offset in (('left', 1, 100), ('right', 3, 160), ('middle', 2, 220)): + point = (px + offset, py + 120) + since = time.monotonic() + result = background('click', **at(point), button=button) + check(failures, (result.get('background') or {}).get('route') == 'kwin-pen', + f'{button} route: {result.get("background")}') + time.sleep(.4) + presses = pen_presses(since, code, point) + check(failures, len(presses) == 1, + f'{button} click: one pen press with button {code} at {point}: {events(since)[:8]}') + check(failures, presses and [e for e in events(presses[0]['t']) if e['kind'] == 'proximity-out'], + f'{button} click: the pen left proximity after it') + if button == 'left': + text = rc.read_text() if rc.exists() else '' + check(failures, 'computer-use-linux virtual pen]' in text and 'MapToWorkspace=true' in text, + f'the pen maps to the whole workspace: {text}') + point = (px + 280, py + 120) + since = time.monotonic() + background('click', **at(point), click_count=2) + time.sleep(.4) + check(failures, len(pen_presses(since, 1, point)) == 2, f'double click: two pen presses at {point}') + start, end = (px + 100, py + 200), (px + 300, py + 260) + since = time.monotonic() + background('drag', start_x=at(start)['x'], start_y=at(start)['y'], end_x=at(end)['x'], end_y=at(end)['y']) + time.sleep(.4) + seen = events(since) + check(failures, any(e['kind'] == 'press' and e['source'] == 'pen' and gnome.near(e, start) for e in seen) + and any(e['kind'] == 'release' and e['source'] == 'pen' and gnome.near(e, end) for e in seen) + and sum(e['kind'] == 'motion' and e['source'] == 'pen' for e in seen) >= 5, + f'drag: pen press at {start}, motion, release at {end}: {seen[:3]} ... {seen[-3:]}') + point = (sx + sw // 2, sy + sh // 2) + before = 0 + for name in ('scroll', 'scroll after a pen click'): + if name != 'scroll': + background('click', **at((px + 200, py + 400))) + since = time.monotonic() + result = background('scroll', **at(point), direction='down', pages=1) + check(failures, (result.get('background') or {}).get('route') == 'kwin-touch', + f'{name} route: {result.get("background")} {result.get("message")}') + time.sleep(1) + seen = events(since) + scrolled = [e['value'] for e in seen if e['kind'] == 'scrolled'] + check(failures, any(e['kind'] == 'touch-begin' and e['source'] == 'touch' for e in seen) + and scrolled and max(scrolled) > before + 100, f'{name}: touch pan moved the list: {scrolled[-3:]}') + before = max(scrolled) if scrolled else before + # KWin re-sends the pointer's position when the surface under it + # changes focus; the pointer itself must not move or click. + mouse = [e for e in events(pen_start) if e.get('source') == 'mouse' + and (e['kind'] != 'motion' or abs(e['x'] - user_point[0]) > 2 or abs(e['y'] - user_point[1]) > 2)] + check(failures, not mouse, f'no mouse movement or button during pen input: {mouse[:5]}') + since = time.monotonic() + gnome.nudge_mouse() + moved = wait_for(lambda: [e for e in events(since) if e['kind'] == 'motion' and e['source'] == 'mouse'], + 'mouse motion after the nudge', 5) + print('pointer after the nudge', moved[:2], flush=True) + check(failures, abs(moved[-1]['x'] - user_point[0]) <= 2 and abs(moved[-1]['y'] - user_point[1]) <= 2, + f'user pointer still at {user_point}: {moved[-2:]}') + cover = subprocess.Popen(['/usr/bin/python3', '-c', gnome.COVER], env=dict(env, GDK_BACKEND='wayland'), + stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + cleanup.callback(stop, cover) + cover_window = wait_for(lambda: next((w for w in server.call('list_windows', {})['windows'] + if w.get('title') == 'cul-pen-cover' and w.get('bounds')), None), + 'the cover window', 20) + time.sleep(1) + b = cover_window['bounds'] + covered = (b['x'] + b['width'] // 2, b['y'] + b['height'] // 2) + since = time.monotonic() + refused = server.call('click', dict(x=covered[0], y=covered[1], window_id=wid, delivery='background')) + check(failures, refused.get('ok') is False and 'stacked above' in refused.get('message', ''), + f'a covered point is refused: {refused.get("message")}') + time.sleep(.5) + check(failures, not [e for e in events(since) if e['kind'] == 'press'], 'refused click sent no press') + stop(cover) + # Show Desktop hides windows without minimizing them. + # KWin ends Show Desktop when the client that asked leaves the bus. + shower = subprocess.Popen(['/usr/bin/python3', '-c', SHOW_DESKTOP], env=env) + cleanup.callback(stop, shower) + time.sleep(1.5) + since = time.monotonic() + refused = server.call('click', dict(at((px + 100, py + 300)), window_id=wid, delivery='background')) + check(failures, refused.get('ok') is False and 'showing the desktop' in refused.get('message', ''), + f'Show Desktop is refused: {refused.get("message")}') + stop(shower) + time.sleep(1) + check(failures, not [e for e in events(since) if e['kind'] == 'press'], 'Show Desktop click sent no press') + # A minimized window is no target. + hidden = subprocess.Popen(['/usr/bin/python3', '-c', MINIMIZED], env=dict(env, GDK_BACKEND='wayland'), + stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + cleanup.callback(stop, hidden) + minimized = wait_for(lambda: next((w for w in server.call('list_windows', {})['windows'] + if w.get('title') == 'cul-pen-minimized' and w.get('hidden')), None), + 'the minimized window', 20) + refused = server.call('click', dict(at((px + 100, py + 300)), window_id=minimized['window_id'], + delivery='background')) + check(failures, refused.get('ok') is False and 'minimized' in refused.get('message', ''), + f'a minimized target is refused: {refused.get("message")}') + stop(hidden) + # The on-screen indicator's overlay (its pill sits over the window's + # top rows) takes no input and must not count as cover, on the + # second call too, once the overlay is up. + if (Path(binary).parent / 'computer-use-linux-indicator').exists(): + shown = Server(binary, {key: value for key, value in env.items() if key != 'COMPUTER_USE_LINUX_INDICATOR'}) + cleanup.callback(shown.close) + under_pill = (px + 400, py + 3) + for attempt in (1, 2): + since = time.monotonic() + result = shown.call('click', dict(at(under_pill), window_id=wid, delivery='background')) + time.sleep(.6) + check(failures, result.get('ok') and pen_presses(since, 1, under_pill), + f'click {attempt} under the indicator overlay: {result.get("message")}') + shown.close() + else: + print('SKIP indicator overlay: no computer-use-linux-indicator next to the server', flush=True) + server.close() + time.sleep(1) + check(failures, not gnome.leftover_devices(), f'no virtual device after exit: {gnome.leftover_devices()}') + text = rc.read_text() if rc.exists() else '' + check(failures, 'computer-use-linux virtual' not in text, f'no kcminputrc group left: {text}') + # The tablet cursor following the mouse would move the user's pointer. + subprocess.run(['kwriteconfig6', '--file', 'kcminputrc', '--group', 'Tablet', '--key', 'SyncWithMouse', + 'true'], env=env, check=True) + synced = Server(binary, env) + cleanup.callback(synced.close) + report = background_pointer(synced) + check(failures, report.get('available') is False and 'SyncWithMouse' in report.get('reason', ''), + f'SyncWithMouse leaves no route: {report}') + synced.close() + return failures + + +def host(binary): + if not os.environ.get('COMPUTER_USE_LINUX_TEST_KWIN_SEAT'): + print('SKIP kwin pen: set COMPUTER_USE_LINUX_TEST_KWIN_SEAT=1 on a machine with docker and no desktop session') + return + missing = [tool for tool in ('docker', 'sudo') if not shutil.which(tool)] + if missing: + sys.exit(f'kwin pen fixture: missing {missing}') + if subprocess.run(['sudo', '-n', 'true'], capture_output=True).returncode != 0: + sys.exit('kwin pen fixture: needs passwordless sudo') + busy = gnome.seat0_graphical_sessions() + if busy: + sys.exit(f'kwin pen fixture: seat0 has graphical sessions {busy}; uinput input would reach them') + subprocess.run(['docker', 'build', '-q', '-t', IMAGE, '-f', str(HERE / 'virtual_pen_kwin.Dockerfile'), str(HERE)], + check=True, stdout=subprocess.DEVNULL) + cache = Path(os.environ.get('XDG_CACHE_HOME', Path.home() / '.cache')) + out = Path(os.environ.get('CUL_PEN_TEST_KEEP_LOGS') or cache / f'cul-kwin-pen-{os.getpid()}') + out.mkdir(parents=True, exist_ok=True) + loaded = Path('/sys/module/vkms').exists() + subprocess.run(['sudo', '-n', 'modprobe', 'vkms'], check=True) + try: + subprocess.run(['sudo', '-n', 'udevadm', 'settle'], check=True) + card = str(Path('/dev/dri/by-path/platform-vkms-card').resolve()) + binary = Path(binary).resolve() + # --network host: libinput learns of new devices through udev's + # netlink group, which a separate network namespace never hears. + status = subprocess.run(['docker', 'run', '--rm', '--privileged', '--network', 'host', + '-v', '/dev:/dev', '-v', '/run/udev:/run/udev:ro', + '-v', f'{HERE}:/scripts:ro', '-v', f'{binary.parent}:/cul:ro', '-v', f'{out}:/out', + '-e', f'CARD={card}', IMAGE, 'python3', '/scripts/virtual_pen_kwin_test.py', + '--guest', f'/cul/{binary.name}'], timeout=600).returncode + finally: + # The GNOME fixture loads vkms under its own udev rule, and udev keeps + # a device's tags until it goes away: leave none behind. + if not loaded and not gnome.unload_vkms(): + print('kwin pen fixture: vkms is still in use and stays loaded', flush=True) + if not os.environ.get('CUL_PEN_TEST_KEEP_LOGS'): + subprocess.run(['docker', 'run', '--rm', '-v', f'{out}:/out', IMAGE, 'sh', '-c', 'rm -rf /out/*'], + capture_output=True) + out.rmdir() + if status != 0: + sys.exit('kwin pen fixture failed') + print('kwin pen: all checks passed', flush=True) + + +if __name__ == '__main__': + if sys.argv[1] == '--guest': + failures = guest(sys.argv[2]) + if failures: + sys.exit(f'{len(failures)} kwin pen check(s) failed') + else: + host(sys.argv[1]) diff --git a/scripts/virtual_pen_test.py b/scripts/virtual_pen_test.py index 15fb355b..73cc8137 100755 --- a/scripts/virtual_pen_test.py +++ b/scripts/virtual_pen_test.py @@ -184,6 +184,18 @@ def nudge_mouse(): fcntl.ioctl(uinput, 0x5502) +def unload_vkms(timeout=10): + """Removes vkms, retrying while its last user lets go; whether it is gone.""" + deadline = time.monotonic() + timeout + while True: + subprocess.run(['sudo', '-n', 'modprobe', '-r', 'vkms'], capture_output=True) + if not Path('/sys/module/vkms').exists(): + return True + if time.monotonic() > deadline: + return False + time.sleep(.5) + + def seat0_graphical_sessions(): sessions = [] listing = subprocess.run(['loginctl', 'list-sessions', '--no-legend'], capture_output=True, text=True).stdout @@ -237,10 +249,15 @@ def run(binary, root, tty): rule = Path('/run/udev/rules.d/61-mutter.rules') subprocess.run(['sudo', '-n', 'tee', str(rule)], input=UDEV_RULE, text=True, check=True, stdout=subprocess.DEVNULL) - subprocess.run(['sudo', '-n', 'modprobe', '-r', 'vkms'], capture_output=True) + # udevd notices new rules through inotify, which a bind mount of + # /run/udev (the KWin fixture's container) can leave deaf. + subprocess.run(['sudo', '-n', 'udevadm', 'control', '--reload'], check=True) + if not unload_vkms(): + sys.exit('gnome pen fixture: vkms is loaded and in use, so it cannot be reloaded under the rule') subprocess.run(['sudo', '-n', 'modprobe', 'vkms'], check=True) - cleanup.callback(subprocess.run, ['sudo', '-n', 'modprobe', '-r', 'vkms'], capture_output=True) + cleanup.callback(unload_vkms) cleanup.callback(lambda: (subprocess.run(['sudo', '-n', 'rm', '-f', str(rule)]), + subprocess.run(['sudo', '-n', 'udevadm', 'control', '--reload']), subprocess.run(['sudo', '-n', 'udevadm', 'trigger', '-s', 'drm', '--settle']))) subprocess.run(['sudo', '-n', 'udevadm', 'trigger', '-s', 'drm', '--settle'], check=True) user = subprocess.run(['id', '-un'], capture_output=True, text=True, check=True).stdout.strip() diff --git a/skills/computer-use-linux/SKILL.md b/skills/computer-use-linux/SKILL.md index 4567672f..0686ff05 100644 --- a/skills/computer-use-linux/SKILL.md +++ b/skills/computer-use-linux/SKILL.md @@ -236,7 +236,7 @@ directly to a window-relative click. - `mcp --listen ADDR:PORT` serves streamable HTTP at `/mcp` and requires a token from `COMPUTER_USE_LINUX_MCP_TOKEN_FILE` or `COMPUTER_USE_LINUX_MCP_TOKEN`. A non-loopback bind needs `--allowed-host NAME`. The port is plain HTTP; put TLS in front. Do not retry a 401 by putting the token in a URL. Over HTTP, `get_tree_page` and index and selector calls need `snapshot_id`; if refused, call `get_app_state` again. Idle sessions expire after an hour; on `404 Session not found`, initialize again. The cached tree and its `snapshot_id` survive. - On GNOME Wayland, the remote-control portal asks on each new `computer-use-linux` process. `COMPUTER_USE_LINUX_PERSIST_REMOTE_DESKTOP=1` in the server environment asks the portal to remember the grant (GNOME starts with the remember box checked). Later processes reuse a single-use restore token stored mode 0600 in the user state directory. Unset is the default and prompts every time. The flag does nothing when ydotool, not the portal, is the input backend. - On Hyprland, `COMPUTER_USE_LINUX_HYPRLAND_BACKGROUND=1` lets targeted `type_text`, `press_key`, `get_app_state` and `screenshot` work on a window without focusing it, including one on a hidden workspace. Pass `window_id` (or another target) on every call. An AT-SPI action can move focus inside the app, so re-focus a field with its `activate` action before typing into it. The active Hyprland keymap must be US, or background `type_text` and `press_key` calls are refused; background `get_app_state` and `screenshot` still capture. Text must be printable ASCII, tab or newline. An unfocused Chromium ignores browser accelerators such as Ctrl+R, and Chromium joins AT-SPI only when started with `ACCESSIBILITY_ENABLED=1` if the accessibility bus reports itself disabled. -- To act without taking the user's keyboard, pass `delivery: "background"` to `type_text` or `press_key` (naming an element or `browser_tab` implies it). Name an element from `get_app_state` (text is inserted and read back; Enter or space runs a button's, link's, check box's or single-line field's action), a `browser_tab` when `COMPUTER_USE_LINUX_DEVTOOLS` names the port of a browser the user started with `--remote-debugging-port`, or a terminal by `tty` or terminal selectors (kitty remote control, wezterm, tmux or GNU screen). On X11 and sway 1.10+ a plain window target also works: the agent gets its own keyboard focus (an XInput 2 master or a transient seat), and `click`/`scroll` with `delivery: "background"`, `x`/`y` and a window target use the agent's own pointer (X11 only with no window manager). On sway, Xwayland, GTK and Chromium windows and Alt/Super chords have no background route; on X11, window manager shortcuts have none. `doctor` shows the seat in `capabilities.background_seat`. On GNOME Wayland, `click`, `drag` and `scroll` with `delivery: "background"`, a window target and points on that window use a virtual pen and touchscreen: the user's mouse stays put, but a click focuses the window, apps without tablet support ignore the pen (re-observe after the call), a drag stays in one window, and a scroll is a touch pan (`capabilities.background_pointer`). When no route serves a call, the first one in the session sends nothing and returns a notice (`background.retry_to_continue: true`) with the reason; repeating the call sends it through normal input, which moves the user's keyboard focus and pointer, and later calls with no route do that without asking again (`background.used: false`). Repeat only if taking the user's focus is acceptable; otherwise use the setting the reason names, or agent-workspace-linux (https://github.com/agent-sh/agent-workspace-linux), which gives the agent its own session with its own keyboard and pointer. A wrong or ambiguous target is still an error. +- To act without taking the user's keyboard, pass `delivery: "background"` to `type_text` or `press_key` (naming an element or `browser_tab` implies it). Name an element from `get_app_state` (text is inserted and read back; Enter or space runs a button's, link's, check box's or single-line field's action), a `browser_tab` when `COMPUTER_USE_LINUX_DEVTOOLS` names the port of a browser the user started with `--remote-debugging-port`, or a terminal by `tty` or terminal selectors (kitty remote control, wezterm, tmux or GNU screen). On X11 and sway 1.10+ a plain window target also works: the agent gets its own keyboard focus (an XInput 2 master or a transient seat), and `click`/`scroll` with `delivery: "background"`, `x`/`y` and a window target use the agent's own pointer (X11 only with no window manager). On sway, Xwayland, GTK and Chromium windows and Alt/Super chords have no background route; on X11, window manager shortcuts have none. `doctor` shows the seat in `capabilities.background_seat`. On GNOME Wayland and KDE Plasma Wayland (KWin 6.3+), `click`, `drag` and `scroll` with `delivery: "background"`, a window target and points on that window use a virtual pen and touchscreen: the user's mouse stays put, but a click focuses the window, apps without tablet support ignore the pen (re-observe after the call), a drag stays in one window, and a scroll is a touch pan (`capabilities.background_pointer`). When no route serves a call, the first one in the session sends nothing and returns a notice (`background.retry_to_continue: true`) with the reason; repeating the call sends it through normal input, which moves the user's keyboard focus and pointer, and later calls with no route do that without asking again (`background.used: false`). Repeat only if taking the user's focus is acceptable; otherwise use the setting the reason names, or agent-workspace-linux (https://github.com/agent-sh/agent-workspace-linux), which gives the agent its own session with its own keyboard and pointer. A wrong or ambiguous target is still an error. - Desktop input is stateful. Avoid concurrent tool calls against this MCP server. - Pi serializes the native Computer Use tools and keeps one process for the session. If that process exits, do not replay an ambiguous mutating call; obtain a fresh `get_app_state` before another element-based action. - `click`, `drag`, `press_key`, `type_text`, `perform_action`, and `set_value` can change real application state. diff --git a/src/diagnostics.rs b/src/diagnostics.rs index 812790aa..1712e778 100644 --- a/src/diagnostics.rs +++ b/src/diagnostics.rs @@ -102,7 +102,7 @@ pub struct CapabilityMap { /// or the reason it has none. pub background_seat: serde_json::Value, /// Background click, drag and scroll on GNOME Wayland: a virtual pen - /// (gnome-pen) and touchscreen (gnome-touch) that leave the user's mouse + /// (gnome-pen or kwin-pen) and touchscreen (gnome-touch or kwin-touch) that leave the user's mouse /// pointer where it is, with their limits, or the reason there is none. pub background_pointer: serde_json::Value, /// The backend the tool will use by default for each selectable layer. diff --git a/src/server.rs b/src/server.rs index 619fe653..2a149149 100644 --- a/src/server.rs +++ b/src/server.rs @@ -2918,7 +2918,7 @@ impl ComputerUseLinux { #[tool( name = "drag", output_schema = rmcp::handler::server::tool::schema_for_output::(), - description = "Drag with the left button from one point to another in screenshot capture pixels. Takes no relative coordinates; a window target matters only with delivery=\"background\", which on GNOME Wayland drags with a virtual pen inside that window and leaves the user's pointer where it is (elsewhere the first such call returns a one-time notice, then later calls drag through the user's pointer). Pass screenshot_after (true, \"screen\", or \"window\") or include_tree=true to return post-action observation in the same result.", + description = "Drag with the left button from one point to another in screenshot capture pixels. Takes no relative coordinates; a window target matters only with delivery=\"background\", which on GNOME and KDE Plasma Wayland drags with a virtual pen inside that window and leaves the user's pointer where it is (elsewhere the first such call returns a one-time notice, then later calls drag through the user's pointer). Pass screenshot_after (true, \"screen\", or \"window\") or include_tree=true to return post-action observation in the same result.", annotations( read_only_hint = false, destructive_hint = true, @@ -2948,8 +2948,9 @@ impl ComputerUseLinux { let received = Some(serde_json::json!(params.clone())); let input_guard = Arc::clone(&self.input_operation_lock).lock_owned().await; let outcome = match crate::virtual_pen::availability() { - Ok(()) => { + Ok(desktop) => { self.background_pen( + desktop, "drag", PenRequest::Drag { start: (params.start_x, params.start_y), @@ -2964,9 +2965,7 @@ impl ComputerUseLinux { Err(reason) => { drop(input_guard); BackgroundOutcome::Unavailable { - reason: format!( - "a background drag goes through the GNOME virtual pen, and {reason}" - ), + reason: format!("a background drag goes through the virtual pen, and {reason}"), fallback: FocusFallback::default(), } } @@ -5488,7 +5487,7 @@ struct ClickParams { /// "background" sends the input through the agent's own pointer, so the /// user's pointer and keyboard focus stay where they are: on X11 without /// a window manager (an XInput 2 master pointer) and on sway 1.10 or later - /// (a transient seat). On GNOME Wayland a virtual pen (a touch pan for + /// (a transient seat). On GNOME and KDE Plasma Wayland a virtual pen (a touch pan for /// scroll) leaves the user's pointer alone, but a click still focuses the /// window, and apps without tablet support ignore it. Needs x and y in /// screenshot capture pixels over the window the call targets, without @@ -5770,7 +5769,7 @@ struct ScrollParams { /// "background" sends the input through the agent's own pointer, so the /// user's pointer and keyboard focus stay where they are: on X11 without /// a window manager (an XInput 2 master pointer) and on sway 1.10 or later - /// (a transient seat). On GNOME Wayland a virtual pen (a touch pan for + /// (a transient seat). On GNOME and KDE Plasma Wayland a virtual pen (a touch pan for /// scroll) leaves the user's pointer alone, but a click still focuses the /// window, and apps without tablet support ignore it. Needs x and y in /// screenshot capture pixels over the window the call targets, without @@ -5855,7 +5854,7 @@ struct DragParams { /// Window title substring, ignoring ASCII case. #[serde(default)] window_title: Option, - /// "background" drags with a virtual pen on GNOME Wayland, so the + /// "background" drags with a virtual pen on GNOME and KDE Plasma Wayland, so the /// user's pointer stays where it is; both points must be on the target /// window, with no window stacked above them. Elsewhere the first such /// call in a server session sends nothing and returns a notice; @@ -10682,8 +10681,8 @@ struct BackgroundReport { /// would go, through the user's keyboard focus and pointer. used: bool, /// The background route used: atspi-element, devtools-tab, terminal, - /// x11-master, sway-transient-seat, hyprland-send-key-state, gnome-pen - /// or gnome-touch. + /// x11-master, sway-transient-seat, hyprland-send-key-state, gnome-pen, + /// gnome-touch, kwin-pen or kwin-touch. #[serde(skip_serializing_if = "Option::is_none")] route: Option, /// False on the one-time notice: nothing was sent. @@ -10784,14 +10783,42 @@ enum PenRequest { }, } -/// What a pen or touch action did to the user's keyboard focus: mutter -/// focuses the window a tip press or a touch lands on. +/// The windows the virtual pen checks a point against, bottom to top. +async fn pen_windows( + desktop: crate::virtual_pen::Desktop, +) -> anyhow::Result> { + match desktop { + crate::virtual_pen::Desktop::Gnome => { + crate::windowing::backends::gnome::extension_windows_stacked().await + } + // The indicator's overlay surfaces take no input (an empty input + // region), so they cover nothing. + crate::virtual_pen::Desktop::Kwin => { + Ok(crate::windowing::backends::kwin::windows_stacked() + .await? + .into_iter() + .filter(|window| !window.pid.is_some_and(is_indicator_process)) + .collect()) + } + } +} + +/// What a pen or touch action did to the user's keyboard focus: the +/// compositor focuses the window a tip press or a touch lands on. +/// Whether pid runs this server's on-screen indicator. +fn is_indicator_process(pid: u32) -> bool { + std::fs::read_link(format!("/proc/{pid}/exe")).is_ok_and(|exe| { + exe.file_name() + .is_some_and(|name| name == "computer-use-linux-indicator") + }) +} + fn pen_focus_sentence(before: Option, after: Option>) -> String { match after { None => "The user's keyboard focus could not be read back.".to_string(), Some(after) if after == before => "The user's keyboard focus did not move.".to_string(), Some(Some(after)) => format!( - "GNOME gave window_id {after} keyboard focus{}, so the user's typing now goes there.", + "The compositor gave window_id {after} keyboard focus{}, so the user's typing now goes there.", before .map(|before| format!(" (it was on window_id {before})")) .unwrap_or_default() @@ -11171,7 +11198,7 @@ impl ComputerUseLinux { "a background {action} needs x and y in screenshot capture pixels" )); }; - if crate::virtual_pen::availability().is_ok() { + if let Ok(desktop) = crate::virtual_pen::availability() { let request = match input(f64::from(x), f64::from(y)) { seat::PointerInput::Click { button, count, .. } => PenRequest::Click { at: (x, y), @@ -11183,7 +11210,7 @@ impl ComputerUseLinux { } }; return self - .background_pen(action, request, target, received, input_guard) + .background_pen(desktop, action, request, target, received, input_guard) .await; } let kind = match seat::availability() { @@ -11253,6 +11280,7 @@ impl ComputerUseLinux { /// with no window stacked above it there. async fn background_pen( &self, + desktop: crate::virtual_pen::Desktop, action: &'static str, request: PenRequest, target: Option, @@ -11272,18 +11300,42 @@ impl ComputerUseLinux { }; // Before the extension: GNOME Shell turns extensions off on the lock // screen, and focused input must not stand in there either. - if pen::screen_locked().await { + if pen::screen_locked(desktop).await { return fail("Nothing was sent: the screen is locked.".to_string()); } - let windows = match crate::windowing::backends::gnome::extension_windows_stacked().await { + if desktop == pen::Desktop::Kwin { + if let Some(reason) = pen::kwin_blocker().await { + return fail(format!("Nothing was sent: {reason}.")); + } + } + let windows = match pen_windows(desktop).await { Ok(windows) => windows, Err(error) => { - return unavailable(format!("the virtual pen checks which window is under the point through the computer-use-linux GNOME Shell extension, which did not answer ({error:#}); run setup_window_targeting")) + return unavailable(match desktop { + pen::Desktop::Gnome => format!("the virtual pen checks which window is under the point through the computer-use-linux GNOME Shell extension, which did not answer ({error:#}); run setup_window_targeting"), + pen::Desktop::Kwin => format!("the virtual pen checks which window is under the point through KWin scripting, which did not answer ({error:#})"), + }) } }; let window = match resolve_window_target(&windows, &target) { Ok(window) => window.clone(), - Err(error) => return fail(format!("Nothing was sent: {error:#}")), + Err(error) => { + // KWin's list holds only what the user can see; say why a + // window KWin has is not in it. + if desktop == pen::Desktop::Kwin { + if let Ok(all) = crate::windowing::backends::kwin::list_windows().await { + if let Ok(window) = resolve_window_target(&all, &target) { + if !windows + .iter() + .any(|shown| shown.window_id == window.window_id) + { + return fail(format!("Nothing was sent: window_id {} is minimized, hidden, or not on the desktop and activity the user sees.", window.window_id)); + } + } + } + } + return fail(format!("Nothing was sent: {error:#}")); + } }; let cached = self.desktop_size.lock().ok().and_then(|guard| *guard); let capture = match cached { @@ -11299,21 +11351,31 @@ impl ComputerUseLinux { ), }, }; - let monitors = match pen::current_monitors().await { + let monitors = match desktop { + pen::Desktop::Gnome => pen::current_monitors().await, + pen::Desktop::Kwin => pen::kwin_monitors().await, + }; + let monitors = match monitors { Ok(monitors) => monitors, Err(error) => { return unavailable(format!( - "mutter's monitor layout could not be read: {error:#}" + "the compositor's monitor layout could not be read: {error:#}" )) } }; + let check = |windows: &[crate::windowing::types::WindowInfo], point| match desktop { + pen::Desktop::Gnome => pen::check_point_on_target(windows, window.window_id, point), + pen::Desktop::Kwin => { + pen::check_point_on_visible_target(windows, window.window_id, point) + } + }; let layout = match pen::Layout::new(monitors, capture) { Ok(layout) => layout, Err(error) => return fail(format!("Nothing was sent: {}", as_sentence(&error))), }; let on_target = |(x, y): (i32, i32)| { let point = layout.logical_point(x, y); - pen::check_point_on_target(&windows, window.window_id, point) + check(&windows, point) .map(|rect| (point, rect)) .map_err(|error| format!("Nothing was sent: {}", as_sentence(&error))) }; @@ -11334,7 +11396,7 @@ impl ComputerUseLinux { button, count, }, - pen::PEN_ROUTE, + desktop.pen_route(), format!("the {action} at ({}, {}) through a virtual pen", at.0, at.1), ) } @@ -11356,7 +11418,7 @@ impl ComputerUseLinux { .collect(); ( Gesture::Drag { path }, - pen::PEN_ROUTE, + desktop.pen_route(), format!( "the drag from ({}, {}) to ({}, {}) through a virtual pen", start.0, start.1, end.0, end.1 @@ -11373,6 +11435,9 @@ impl ComputerUseLinux { }; // A touchscreen maps to one monitor, through that monitor's // rotation, which the device values here do not apply. + if monitor.transform == u32::MAX { + return fail("Nothing was sent: a background scroll pans with a virtual touchscreen, and this monitor's rotation could not be read (kscreen-doctor -j).".to_string()); + } if monitor.transform != 0 { return fail("Nothing was sent: a background scroll pans with a virtual touchscreen, and this monitor is rotated or flipped.".to_string()); } @@ -11390,8 +11455,7 @@ impl ComputerUseLinux { }; // Every stroke ends on the target too, with nothing above it. for (_, end) in &strokes { - if let Err(error) = pen::check_point_on_target(&windows, window.window_id, *end) - { + if let Err(error) = check(&windows, *end) { return fail(format!( "Nothing was sent: the pan would end off the target: {}", as_sentence(&error) @@ -11405,7 +11469,7 @@ impl ComputerUseLinux { let count = strokes.len(); ( Gesture::Pan { strokes }, - pen::TOUCH_ROUTE, + desktop.touch_route(), format!( "the scroll at ({}, {}) as a {distance:.0} px touch pan in {count} stroke{} through a virtual touchscreen", at.0, @@ -11422,7 +11486,7 @@ impl ComputerUseLinux { .map(|window| window.window_id); let sent = tokio::task::spawn_blocking(move || { let _hold = input_guard; - pen::perform(&gesture, &layout) + pen::perform(&gesture, &layout, desktop) }) .await; match sent { @@ -11437,7 +11501,7 @@ impl ComputerUseLinux { as_sentence(&message) )), Ok(Ok(())) => { - let focused_after = crate::windowing::backends::gnome::extension_windows_stacked() + let focused_after = pen_windows(desktop) .await .ok() .map(|windows| { @@ -11447,7 +11511,7 @@ impl ComputerUseLinux { .map(|window| window.window_id) }); let focus = pen_focus_sentence(focused_before, focused_after); - let limits = if route == pen::TOUCH_ROUTE { + let limits = if route == desktop.touch_route() { "A touch pan scrolls only where the app pans on touch; re-observe to confirm." } else { "An app without tablet support does not see pen input; re-observe to confirm." diff --git a/src/virtual_pen.rs b/src/virtual_pen.rs index 3006e125..41b0d53c 100644 --- a/src/virtual_pen.rs +++ b/src/virtual_pen.rs @@ -50,6 +50,20 @@ //! leave another's pen on the defaults. Never write "default" for this: //! mutter 48 and later turn it into a left click. //! +//! KDE Plasma: KWin 6.3 and later give a tablet tool its own cursor too +//! (TabletInputRedirection tracks it, and moves it whether or not the client +//! under it supports tablets), and send its events through the tablet +//! protocol. KWin maps a tablet with no setting to the active output, which +//! follows the user's pointer, and a touchscreen to the internal panel or +//! the first output. So the pen sets its own device's "mapToWorkspace" and +//! the touchscreen its "outputName" through org.kde.KWin.InputDevice, reads +//! them back before any event, and the server removes what KWin wrote into +//! kcminputrc for them (the groups keyed by these devices' vendor, product +//! and name) when they go away. KWin's "SyncWithMouse" tablet option moves +//! the user's pointer to where the pen was, and KWIN_WAYLAND_EMULATE_TABLET +//! sends pen input over clients without tablet support through it; with +//! either there is no route. +//! //! Lifecycle: created on first use, and both devices go away after //! [crate::agent_seat::IDLE_LIFETIME] without use, or when the server exits. @@ -63,10 +77,37 @@ use std::path::{Path, PathBuf}; use std::sync::Mutex; use std::time::{Duration, Instant}; -/// The route name results report for pen input. -pub(crate) const PEN_ROUTE: &str = "gnome-pen"; -/// The route name results report for touch scrolling. -pub(crate) const TOUCH_ROUTE: &str = "gnome-touch"; +/// The desktop the pen and touchscreen run on. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum Desktop { + Gnome, + Kwin, +} + +impl Desktop { + /// The route name results report for pen input. + pub(crate) fn pen_route(self) -> &'static str { + match self { + Desktop::Gnome => "gnome-pen", + Desktop::Kwin => "kwin-pen", + } + } + + /// The route name results report for touch scrolling. + pub(crate) fn touch_route(self) -> &'static str { + match self { + Desktop::Gnome => "gnome-touch", + Desktop::Kwin => "kwin-touch", + } + } +} + +/// KWin 6.3 moved the tablet cursor into TabletInputRedirection and moves it +/// for clients without tablet support (4681e9819f, 663f7342d6). +const KWIN_MIN: (u32, u32) = (6, 3); +const KWIN_DEVICES: &str = "/org/kde/KWin/InputDevice"; +/// The kcminputrc keys KWin writes for a tablet or touchscreen mapping. +const KWIN_MAPPING_KEYS: [&str; 3] = ["MapToWorkspace", "OutputName", "OutputUuid"]; /// The pan distance of one wheel notch, in logical pixels. pub(crate) const PAN_PER_NOTCH: f64 = 60.0; /// A pan stays this far inside the window and the monitor edges: GNOME @@ -94,24 +135,24 @@ const TOUCH_NAME: &str = "computer-use-linux virtual touch"; /// Why the pen has no route in a session, or None when it has one. Pure: /// the facts come from [SessionFacts::from_env]. -pub(crate) fn classify(facts: &SessionFacts) -> std::result::Result<(), String> { +pub(crate) fn classify(facts: &SessionFacts) -> std::result::Result { let desktop = facts.current_desktop.as_str(); let has = |name: &str| desktop.split(':').any(|part| part.trim() == name); if facts.native_x11 { return Err("the session runs X11, where a tablet moves the core pointer".to_string()); } if desktop.contains("gnome") && facts.wayland { - return Ok(()); + return Ok(Desktop::Gnome); } - if has("kde") { - return Err("KWin gives a tablet its own cursor too (Plasma 6.2 and later), but maps a tablet without a configured output to the active output, which follows the user's pointer, and needs a configured output for touch, so the virtual pen is GNOME only".to_string()); + if has("kde") && facts.wayland { + return Ok(Desktop::Kwin); } - Err("the virtual pen needs GNOME on Wayland: on wlroots compositors, Hyprland, niri and COSMIC a tablet moves the one seat cursor".to_string()) + Err("the virtual pen needs GNOME or KDE Plasma on Wayland: on wlroots compositors, Hyprland, niri and COSMIC a tablet moves the one seat cursor".to_string()) } /// The pen route here, or why there is none. -pub(crate) fn availability() -> std::result::Result<(), String> { - classify(&SessionFacts::from_env())?; +pub(crate) fn availability() -> std::result::Result { + let desktop = classify(&SessionFacts::from_env())?; let writable = std::ffi::CString::new("/dev/uinput") .ok() // SAFETY: path is a NUL-terminated CString that outlives the call, @@ -120,16 +161,99 @@ pub(crate) fn availability() -> std::result::Result<(), String> { if !writable { return Err("/dev/uinput is not writable for this user, so no virtual pen can be made (the same access the uinput pointer needs)".to_string()); } + if desktop == Desktop::Kwin { + kwin_ready()?; + } + Ok(desktop) +} + +fn on_path(tool: &str) -> bool { + std::env::var_os("PATH") + .is_some_and(|path| std::env::split_paths(&path).any(|dir| dir.join(tool).is_file())) +} + +/// The installed KWin's major and minor version, read once. +fn kwin_version() -> Option<(u32, u32)> { + static VERSION: std::sync::OnceLock> = std::sync::OnceLock::new(); + *VERSION.get_or_init(|| { + let output = std::process::Command::new("kwin_wayland") + .arg("--version") + .stdin(std::process::Stdio::null()) + .stderr(std::process::Stdio::null()) + .output() + .ok()?; + parse_kwin_version(&String::from_utf8_lossy(&output.stdout)) + }) +} + +/// "kwin 6.6.7" to (6, 6). +fn parse_kwin_version(text: &str) -> Option<(u32, u32)> { + let version = text.split_whitespace().nth(1)?; + let mut parts = version.split('.'); + Some((parts.next()?.parse().ok()?, parts.next()?.parse().ok()?)) +} + +/// A kcminputrc boolean as KConfig reads it (every spelling it accepts): +/// kreadconfig6 --type bool answers in its exit status. +fn kcminputrc_bool(groups: &[&str], key: &str) -> bool { + let mut command = std::process::Command::new("kreadconfig6"); + command.args(["--file", "kcminputrc"]); + for group in groups { + command.args(["--group", group]); + } + command + .args(["--key", key, "--type", "bool", "--default", "false"]) + .stdin(std::process::Stdio::null()) + .stdout(std::process::Stdio::null()) + .stderr(std::process::Stdio::null()) + .status() + .is_ok_and(|status| status.success()) +} + +/// What KWin needs for the pen route, or why it falls short. +fn kwin_ready() -> std::result::Result<(), String> { + match kwin_version() { + Some(version) if version >= KWIN_MIN => {} + Some((major, minor)) => { + return Err(format!("KWin {major}.{minor} moves the user's pointer with a tablet; the virtual pen needs KWin 6.3 or later, where a tablet has its own cursor")); + } + None => { + return Err( + "kwin_wayland --version did not answer, so KWin's version is unknown".to_string(), + ); + } + } + for tool in ["kreadconfig6", "kwriteconfig6"] { + if !on_path(tool) { + return Err(format!("{tool} is not installed; the virtual pen uses it to read KWin's tablet option and to remove its own devices' KWin settings afterwards")); + } + } + if kcminputrc_bool(&["Tablet"], "SyncWithMouse") { + return Err("KWin's tablet cursor follows the mouse here (kcminputrc [Tablet] SyncWithMouse), so the user's pointer would jump to where the pen was".to_string()); + } Ok(()) } +/// The routes the doctor offers: on KWin a scroll needs kscreen-doctor for +/// the output's rotation. +fn routes(desktop: Desktop) -> Vec<&'static str> { + if desktop == Desktop::Kwin && !on_path("kscreen-doctor") { + vec![desktop.pen_route()] + } else { + vec![desktop.pen_route(), desktop.touch_route()] + } +} + /// Doctor line for the pen route. pub(crate) fn doctor_summary() -> serde_json::Value { match availability() { - Ok(()) => serde_json::json!({ + Ok(desktop) => serde_json::json!({ "available": true, - "routes": [PEN_ROUTE, TOUCH_ROUTE], - "note": "Background click and drag use a virtual pen, background scroll a one-finger touch pan; the user's mouse pointer does not move. A pen tap or a touch gives the window keyboard focus. Apps without tablet support never see the pen, scrolling needs touch panning in the app, and a drag stays inside one window. Needs the computer-use-linux GNOME Shell extension to check the window under the point.", + "routes": routes(desktop), + "note": format!("Background click and drag use a virtual pen, background scroll a one-finger touch pan; the user's mouse pointer does not move. A pen tap or a touch gives the window keyboard focus. Apps without tablet support never see the pen, scrolling needs touch panning in the app, and a drag stays inside one window. {}", match desktop { + Desktop::Gnome => "Needs the computer-use-linux GNOME Shell extension to check the window under the point.", + Desktop::Kwin => "KWin scripting checks the window under the point; Show Desktop, an open Overview or another full-screen effect refuses the call.", + }), }), Err(reason) => serde_json::json!({ "available": false, "reason": reason }), } @@ -287,7 +411,7 @@ impl Layout { .map(|monitor| monitor.rect) .reduce(|a, b| a.union(&b)) .filter(|bounds| bounds.width > 0.0 && bounds.height > 0.0) - .ok_or_else(|| "mutter reported no monitor".to_string())?; + .ok_or_else(|| "the compositor reported no monitor".to_string())?; if capture_width == 0 || capture_height == 0 { return Err("the screenshot size is unknown; take a screenshot first".to_string()); } @@ -414,6 +538,54 @@ pub(crate) fn check_point_on_target( Ok(rect) } +/// The check for a window list that holds only the windows the user can +/// see, bottom to top (KWin's): the point must be on the target, and no +/// window above it may cover the point. +pub(crate) fn check_point_on_visible_target( + windows: &[crate::windowing::types::WindowInfo], + target_id: u64, + point: (f64, f64), +) -> std::result::Result { + let target = windows + .iter() + .find(|window| window.window_id == target_id) + .ok_or_else(|| { + format!("window_id {target_id} is not on the desktop the user sees, or is minimized or hidden") + })?; + // workspace is Some(-1) where KWin did not say (kwin::windows_stacked). + if target.workspace.is_some() { + return Err(format!( + "KWin did not say whether window_id {target_id} is on the desktop and activity the user sees" + )); + } + let rect = window_rect(target) + .ok_or_else(|| format!("window_id {target_id} has no known position"))?; + if !rect.contains(point) { + return Err(format!( + "({:.0}, {:.0}) in layout pixels is outside window_id {target_id}, which spans ({:.0}, {:.0}) to ({:.0}, {:.0})", + point.0, + point.1, + rect.x, + rect.y, + rect.x + rect.width, + rect.y + rect.height + )); + } + let top = windows + .iter() + .rev() + .find(|window| window_rect(window).is_some_and(|rect| rect.contains(point))) + .expect("the target contains the point"); + if top.window_id != target_id { + return Err(format!( + "window_id {} ({}) is stacked above window_id {target_id} at that point", + top.window_id, + top.title.as_deref().unwrap_or("untitled") + )); + } + Ok(rect) +} + /// Finger strokes that pan by motion (logical pixels, along one axis), each /// starting at start and staying inside bounds. A point outside bounds, or /// one with too little room in the pan's direction, has no strokes. @@ -714,6 +886,81 @@ impl Kind { } } +impl Kind { + /// The kcminputrc groups KWin keeps this device's settings in. + fn kcminputrc_groups(self) -> [String; 4] { + let (product, name) = match self { + Kind::Pen => (PEN_PRODUCT, PEN_NAME), + Kind::Touch => (TOUCH_PRODUCT, TOUCH_NAME), + }; + [ + "Libinput".to_string(), + VENDOR.to_string(), + product.to_string(), + name.to_string(), + ] + } +} + +/// Removes the mapping keys KWin wrote into kcminputrc for kind's device. +/// Run once the device is gone: KWin writes a setting back when it changes +/// on a live device. +fn kcminputrc_delete(kind: Kind) { + for key in KWIN_MAPPING_KEYS { + let mut command = std::process::Command::new("kwriteconfig6"); + command.args(["--file", "kcminputrc"]); + for group in kind.kcminputrc_groups() { + command.arg("--group").arg(group); + } + let _ = command + .args(["--key", key, "--delete", "--notify"]) + .stdin(std::process::Stdio::null()) + .stdout(std::process::Stdio::null()) + .stderr(std::process::Stdio::null()) + .status(); + } +} + +/// What a device changed in the desktop's settings, undone when it drops. +/// It is the last field of [Pen] and [Touch], so the device is gone first. +struct Settings { + desktop: Desktop, + kind: Kind, + wrote_output: bool, +} + +impl Settings { + /// Clears what an earlier server may have left for kind, before the + /// device exists. + fn prepare(desktop: Desktop, kind: Kind) -> Self { + match desktop { + Desktop::Gnome => reset_output(kind), + Desktop::Kwin => kcminputrc_delete(kind), + } + Self { + desktop, + kind, + wrote_output: false, + } + } +} + +impl Drop for Settings { + fn drop(&mut self) { + match self.desktop { + Desktop::Gnome => { + if self.wrote_output { + reset_output(self.kind); + } + if self.kind == Kind::Pen { + reset_stylus_actions(); + } + } + Desktop::Kwin => kcminputrc_delete(self.kind), + } + } +} + fn axis(code: AbsoluteAxisCode, max: i32, resolution: i32) -> UinputAbsSetup { UinputAbsSetup::new(code, AbsInfo::new(0, 0, max, 0, 0, resolution)) } @@ -913,39 +1160,29 @@ fn gsettings_reset(settings: &str, key: &str) { .status(); } +// Fields drop in order: the device goes away before its settings. struct Pen { events: PenEvents, node: PathBuf, - wrote_output: bool, /// Whether libinput has seen the pen leave proximity on its own. primed: bool, + settings: Settings, } impl Drop for Pen { fn drop(&mut self) { let _ = self.events.leave(); - if self.wrote_output { - reset_output(Kind::Pen); - } - reset_stylus_actions(); } } struct Touch { events: TouchEvents, node: PathBuf, - wrote_output: bool, -} - -impl Drop for Touch { - fn drop(&mut self) { - if self.wrote_output { - reset_output(Kind::Touch); - } - } + settings: Settings, } struct Devices { + desktop: Desktop, pen: Option, touch: Option, last_use: Instant, @@ -1043,10 +1280,14 @@ fn covering_area( connection: &zbus::blocking::Connection, kind: Kind, node: &Path, - wrote_output: &mut bool, + settings: &mut Settings, point: (f64, f64), layout: &Layout, ) -> Result { + if settings.desktop == Desktop::Kwin { + return kwin_covering_area(connection, kind, node, point, layout); + } + let wrote_output = &mut settings.wrote_output; let current = wait_mapping(connection, node, |mapping| mapping != Mapping::Unknown)?; match (kind, current) { (_, Mapping::Layout) => return Ok(layout.bounds), @@ -1088,21 +1329,197 @@ fn covering_area( } } +fn kwin_path(node: &Path) -> Result { + let name = node + .file_name() + .and_then(|name| name.to_str()) + .ok_or_else(|| anyhow!("device node has no name"))?; + zbus::zvariant::OwnedObjectPath::try_from(format!("{KWIN_DEVICES}/{name}")) + .context("KWin input device path") +} + +fn kwin_get( + connection: &zbus::blocking::Connection, + path: &str, + interface: &str, + property: &str, +) -> Result { + let reply = connection + .call_method( + Some("org.kde.KWin"), + path, + Some("org.freedesktop.DBus.Properties"), + "Get", + &(interface, property), + ) + .with_context(|| format!("KWin's {interface}.{property} did not answer"))?; + reply + .body() + .deserialize() + .with_context(|| format!("KWin's {interface}.{property} returned an unexpected reply")) +} + +/// The sysnames (event7) of the input devices KWin has. +fn kwin_devices(connection: &zbus::blocking::Connection) -> Result> { + let value = kwin_get( + connection, + KWIN_DEVICES, + "org.kde.KWin.InputDeviceManager", + "devicesSysNames", + )?; + Vec::::try_from(value).context("KWin's devicesSysNames is not a string list") +} + +fn kwin_device_get( + connection: &zbus::blocking::Connection, + path: &str, + property: &str, +) -> Result { + kwin_get(connection, path, "org.kde.KWin.InputDevice", property) +} + +fn kwin_device_set( + connection: &zbus::blocking::Connection, + path: &str, + property: &str, + value: zbus::zvariant::Value<'_>, +) -> Result<()> { + connection + .call_method( + Some("org.kde.KWin"), + path, + Some("org.freedesktop.DBus.Properties"), + "Set", + &("org.kde.KWin.InputDevice", property, value), + ) + .with_context(|| format!("KWin refused {property} for the virtual device"))?; + Ok(()) +} + +/// Waits for accept to hold, MAPPING_TIMEOUT at most. +fn wait_kwin(what: &str, mut accept: impl FnMut() -> Result) -> Result<()> { + let deadline = Instant::now() + MAPPING_TIMEOUT; + loop { + if accept()? { + return Ok(()); + } + if Instant::now() >= deadline { + bail!("KWin did not {what}"); + } + std::thread::sleep(Duration::from_millis(50)); + } +} + +/// The area a device covers on KWin: the pen maps to the whole workspace, +/// the touchscreen to the output under point. Both are set on the device +/// and read back before any event. +fn kwin_covering_area( + connection: &zbus::blocking::Connection, + kind: Kind, + node: &Path, + point: (f64, f64), + layout: &Layout, +) -> Result { + let path = kwin_path(node)?; + let name = path + .as_str() + .rsplit('/') + .next() + .unwrap_or_default() + .to_string(); + wait_kwin("list the virtual device", || { + Ok(kwin_devices(connection)?.contains(&name)) + })?; + // Settings an earlier user left for a device of this name would move + // where the input lands. + for (property, default) in [ + ("calibrationMatrix", "defaultCalibrationMatrix"), + ("outputArea", "defaultOutputArea"), + ("inputArea", "defaultInputArea"), + ("rotation", "defaultRotation"), + ] { + if let (Ok(current), Ok(default)) = ( + kwin_device_get(connection, path.as_str(), property), + kwin_device_get(connection, path.as_str(), default), + ) { + if current != default { + bail!("KWin applies a {property} setting to the virtual device (kcminputrc [Libinput][{VENDOR}][..])"); + } + } + } + match kind { + Kind::Pen => { + // A relative tool moves from its last position, not to the point. + if let Ok(relative) = kwin_device_get(connection, path.as_str(), "tabletToolIsRelative") + { + if bool::try_from(&relative).unwrap_or(true) { + bail!("KWin runs the virtual pen in relative mode"); + } + } + let mapped = || -> Result { + bool::try_from(kwin_device_get( + connection, + path.as_str(), + "mapToWorkspace", + )?) + .context("mapToWorkspace is not a boolean") + }; + if !mapped()? { + kwin_device_set(connection, path.as_str(), "mapToWorkspace", true.into())?; + wait_kwin("map the pen to the whole workspace", mapped)?; + } + Ok(layout.bounds) + } + Kind::Touch => { + let monitor = layout + .monitor_at(point) + .ok_or_else(|| anyhow!("no monitor holds the point"))?; + let connector = monitor.spec.connector.clone(); + let output = || -> Result { + String::try_from(kwin_device_get(connection, path.as_str(), "outputName")?) + .context("outputName is not a string") + }; + if output()? != connector { + kwin_device_set( + connection, + path.as_str(), + "outputName", + connector.as_str().into(), + )?; + wait_kwin(&format!("map the touchscreen to {connector}"), || { + Ok(output()? == connector) + })?; + } + Ok(monitor.rect) + } + } +} + /// Sends gesture through the pen or the touchscreen, creating them on /// first use. Blocking: run it off the async runtime. -pub(crate) fn perform(gesture: &Gesture, layout: &Layout) -> std::result::Result<(), PenError> { - let connection = zbus::blocking::Connection::session() - .map_err(|error| PenError::NoRoute(format!("no session bus to reach mutter: {error}")))?; - // A device node no device has: mutter answers "does not exist" when its - // InputMapping service runs. - mapping( - &connection, - Path::new("/dev/input/computer_use_linux_probe"), - ) +pub(crate) fn perform( + gesture: &Gesture, + layout: &Layout, + desktop: Desktop, +) -> std::result::Result<(), PenError> { + let connection = zbus::blocking::Connection::session().map_err(|error| { + PenError::NoRoute(format!("no session bus to reach the compositor: {error}")) + })?; + match desktop { + // A device node no device has: mutter answers "does not exist" when + // its InputMapping service runs. + Desktop::Gnome => mapping( + &connection, + Path::new("/dev/input/computer_use_linux_probe"), + ) + .map(|_| ()), + Desktop::Kwin => kwin_devices(&connection).map(|_| ()), + } .map_err(|error| PenError::NoRoute(format!("{error:#}")))?; let _process = process_lock(Duration::from_secs(5))?; let mut guard = lock_devices(); let devices = guard.get_or_insert_with(|| Devices { + desktop, pen: None, touch: None, last_use: Instant::now(), @@ -1140,16 +1557,16 @@ fn run_pen( layout: &Layout, ) -> std::result::Result<(), PenError> { if devices.pen.is_none() { - reset_output(Kind::Pen); + // A failure below drops settings, which undoes them. + let settings = Settings::prepare(devices.desktop, Kind::Pen); let (device, node) = create(Kind::Pen).map_err(|error| { - reset_stylus_actions(); PenError::NoRoute(format!("the virtual pen could not be made: {error:#}")) })?; devices.pen = Some(Pen { events: PenEvents::new(device), node, - wrote_output: false, primed: false, + settings, }); } let pen = devices.pen.as_mut().expect("pen was just made"); @@ -1157,7 +1574,7 @@ fn run_pen( connection, Kind::Pen, &pen.node.clone(), - &mut pen.wrote_output, + &mut pen.settings, first, layout, ) @@ -1177,11 +1594,13 @@ fn run_pen( // Before every action, under the process lock and right before the pen // comes near: mutter reads them as the pen enters proximity, and another // server's pen going away resets the shared path. - set_stylus_actions().map_err(|error| { - PenError::NotSent(format!( - "the pen's button actions could not be set: {error:#}" - )) - })?; + if pen.settings.desktop == Desktop::Gnome { + set_stylus_actions().map_err(|error| { + PenError::NotSent(format!( + "the pen's button actions could not be set: {error:#}" + )) + })?; + } if !pen.primed { let value = device_value(first, &area); pen.events @@ -1242,7 +1661,7 @@ fn run_touch( layout: &Layout, ) -> std::result::Result<(), PenError> { if devices.touch.is_none() { - reset_output(Kind::Touch); + let settings = Settings::prepare(devices.desktop, Kind::Touch); let (device, node) = create(Kind::Touch).map_err(|error| { PenError::NoRoute(format!( "the virtual touchscreen could not be made: {error:#}" @@ -1251,10 +1670,10 @@ fn run_touch( devices.touch = Some(Touch { events: TouchEvents::new(device), node, - wrote_output: false, + settings, }); - // mutter lists a new touchscreen before it has mapped it, and the - // first pan sent at once never reached the window. + // mutter lists a new touchscreen before it has mapped it; a pan + // sent at once never reached the window. std::thread::sleep(TOUCH_SETTLE); } // libinput pairs a tablet with the first touchscreen it sees and, for a @@ -1275,7 +1694,7 @@ fn run_touch( connection, Kind::Touch, &touch.node.clone(), - &mut touch.wrote_output, + &mut touch.settings, first, layout, ) @@ -1321,24 +1740,185 @@ fn run_touch( /// Whether GNOME Shell's lock screen is up (org.gnome.ScreenSaver). The /// window list cannot show it, and a tap there would reach the lock screen. -pub(crate) async fn screen_locked() -> bool { +pub(crate) async fn screen_locked(desktop: Desktop) -> bool { let Ok(connection) = zbus::Connection::session().await else { return false; }; + // Plasma's screen locker answers on the freedesktop name. + let (name, path) = match desktop { + Desktop::Gnome => ("org.gnome.ScreenSaver", "/org/gnome/ScreenSaver"), + Desktop::Kwin => ("org.freedesktop.ScreenSaver", "/ScreenSaver"), + }; let reply = tokio::time::timeout( Duration::from_secs(2), - connection.call_method( - Some("org.gnome.ScreenSaver"), - "/org/gnome/ScreenSaver", - Some("org.gnome.ScreenSaver"), - "GetActive", - &(), - ), + connection.call_method(Some(name), path, Some(name), "GetActive", &()), ) .await; matches!(reply, Ok(Ok(message)) if message.body().deserialize::().unwrap_or(false)) } +/// Why KWin's state rules out pen input now, checked before every call: +/// Show Desktop hides windows without minimizing them, a full-screen effect +/// such as the Overview takes the input, and KWIN_WAYLAND_EMULATE_TABLET +/// in KWin's environment sends pen input over clients without tablet +/// support through the user's pointer. Some(reason) is a refusal; None +/// means nothing stands in the way. +pub(crate) async fn kwin_blocker() -> Option { + const FULL_SCREEN: [&str; 6] = [ + "overview", + "windowview", + "desktopgrid", + "expo", + "tileseditor", + "cube", + ]; + let Ok(connection) = zbus::Connection::session().await else { + return Some("no session bus to reach KWin".to_string()); + }; + let get = |path: &'static str, interface: &'static str, property: &'static str| { + let connection = connection.clone(); + async move { + let reply = tokio::time::timeout( + Duration::from_secs(2), + connection.call_method( + Some("org.kde.KWin"), + path, + Some("org.freedesktop.DBus.Properties"), + "Get", + &(interface, property), + ), + ) + .await + .ok()? + .ok()?; + reply + .body() + .deserialize::() + .ok() + } + }; + match get("/KWin", "org.kde.KWin", "showingDesktop").await { + Some(value) if bool::try_from(&value).unwrap_or(false) => { + return Some( + "KWin is showing the desktop, which hides windows without minimizing them" + .to_string(), + ); + } + Some(_) => {} + None => return Some("KWin's showingDesktop did not answer".to_string()), + } + let active = get("/Effects", "org.kde.kwin.Effects", "activeEffects") + .await + .and_then(|value| Vec::::try_from(value).ok()); + match active { + Some(effects) => { + if let Some(effect) = effects + .iter() + .find(|effect| FULL_SCREEN.contains(&effect.as_str())) + { + return Some(format!( + "KWin's {effect} effect is open and takes the input" + )); + } + } + None => return Some("KWin's activeEffects did not answer".to_string()), + } + let pid = async { + tokio::time::timeout( + Duration::from_secs(2), + connection.call_method( + Some("org.freedesktop.DBus"), + "/org/freedesktop/DBus", + Some("org.freedesktop.DBus"), + "GetConnectionUnixProcessID", + &("org.kde.KWin",), + ), + ) + .await + .ok()? + .ok()? + .body() + .deserialize::() + .ok() + } + .await; + let Some(environ) = pid.and_then(|pid| std::fs::read(format!("/proc/{pid}/environ")).ok()) + else { + return Some( + "KWin's environment could not be read to rule out KWIN_WAYLAND_EMULATE_TABLET" + .to_string(), + ); + }; + let emulated = environ.split(|byte| *byte == 0).any(|entry| { + entry + .strip_prefix(b"KWIN_WAYLAND_EMULATE_TABLET=") + .is_some_and(|value| !value.is_empty() && value != b"0") + }); + emulated.then(|| "KWin runs with KWIN_WAYLAND_EMULATE_TABLET, which sends pen input over clients without tablet support through the user's pointer".to_string()) +} + +/// The logical layout from KWin's outputs. KWin's scripting gives no +/// rotation, so kscreen-doctor supplies it; a monitor whose rotation is +/// unknown gets transform u32::MAX, which a touch pan refuses. +pub(crate) async fn kwin_monitors() -> Result> { + let outputs = crate::windowing::backends::kwin::named_outputs().await?; + let rotations = kscreen_rotations().await.unwrap_or_default(); + Ok(outputs + .into_iter() + .map(|(name, monitor)| Monitor { + rect: Rect { + x: f64::from(monitor.x), + y: f64::from(monitor.y), + width: f64::from(monitor.width), + height: f64::from(monitor.height), + }, + // kscreen's rotation 1 is none. + transform: match rotations.get(&name) { + Some(1) => 0, + Some(other) => *other, + None => u32::MAX, + }, + spec: MonitorSpec { + connector: name, + vendor: String::new(), + product: String::new(), + serial: String::new(), + }, + }) + .collect()) +} + +async fn kscreen_rotations() -> Option> { + #[derive(serde::Deserialize)] + struct Config { + outputs: Vec, + } + #[derive(serde::Deserialize)] + struct Output { + name: String, + rotation: u32, + } + let output = tokio::time::timeout( + Duration::from_secs(5), + tokio::process::Command::new("kscreen-doctor") + .arg("-j") + .stdin(std::process::Stdio::null()) + .stderr(std::process::Stdio::null()) + .output(), + ) + .await + .ok()? + .ok()?; + let config: Config = serde_json::from_slice(&output.stdout).ok()?; + Some( + config + .outputs + .into_iter() + .map(|output| (output.name, output.rotation)) + .collect(), + ) +} + /// The logical layout from mutter's DisplayConfig.GetCurrentState. pub(crate) async fn current_monitors() -> Result> { use std::collections::HashMap; @@ -1639,19 +2219,23 @@ mod tests { } #[test] - fn only_gnome_on_wayland_gets_the_pen() { + fn gnome_and_kde_on_wayland_get_the_pen() { let facts = |desktop: &str, wayland: bool, native_x11: bool| SessionFacts { native_x11, wayland, current_desktop: desktop.to_string(), ..SessionFacts::default() }; - assert!(classify(&facts("ubuntu:gnome", true, false)).is_ok()); + assert_eq!( + classify(&facts("ubuntu:gnome", true, false)), + Ok(Desktop::Gnome) + ); assert!(classify(&facts("gnome", false, true)).is_err()); - assert!(classify(&facts("kde", true, false)) - .unwrap_err() - .contains("KWin")); + assert_eq!(classify(&facts("kde", true, false)), Ok(Desktop::Kwin)); + assert!(classify(&facts("kde", false, true)).is_err()); assert!(classify(&facts("sway", true, false)).is_err()); + assert_eq!(parse_kwin_version("kwin 6.6.7\n"), Some((6, 6))); + assert_eq!(parse_kwin_version("kwin_wayland"), None); } #[derive(Clone, Default)] diff --git a/src/windowing/backends/kwin.rs b/src/windowing/backends/kwin.rs index 640d0e73..f0c83991 100644 --- a/src/windowing/backends/kwin.rs +++ b/src/windowing/backends/kwin.rs @@ -56,6 +56,66 @@ pub async fn list_windows() -> Result> { Ok(windows) } +/// The windows on the desktop and activity the user sees, bottom to top in +/// KWin's stacking order, with docks, popups and notifications: whatever can +/// cover a point. The desktop window and minimized or hidden windows are left +/// out. workspace is None for a window KWin places on the current desktop and +/// activity, and Some(-1) where it did not say. +pub(crate) async fn windows_stacked() -> Result> { + parse_kwin_windows_stacked(&call_kwin_window_script().await?) +} + +/// Named logical outputs, in KWin's order. +pub(crate) async fn named_outputs() -> Result> { + let json = call_kwin_window_script().await?; + let names = KwinElementClickSnapshot::from_json(&json)? + .outputs + .into_iter() + .map(|output| output.name) + .collect::>(); + Ok(names.into_iter().zip(kwin_output_layout(&json)?).collect()) +} + +fn parse_kwin_windows_stacked(json: &str) -> Result> { + let snapshot = parse_kwin_snapshot(json)?; + let order = snapshot + .stacking + .context("KWin did not expose its stacking order")?; + let mut raw: std::collections::HashMap = snapshot + .windows + .into_iter() + .filter_map(|window| Some((window.kwin_uuid()?, window))) + .collect(); + let flag = |value: &Option| json_value_as_bool(value.as_ref()); + let mut windows = Vec::new(); + for uuid in order + .iter() + .filter_map(|uuid| normalize_kwin_uuid(uuid.as_deref()?)) + { + let Some(window) = raw.remove(&uuid) else { + continue; + }; + if flag(&window.desktop_window) == Some(true) + || flag(&window.minimized) == Some(true) + || flag(&window.hidden) == Some(true) + || flag(&window.on_current_desktop) == Some(false) + || flag(&window.on_current_activity) == Some(false) + { + continue; + } + // A window whose desktop or activity KWin did not say still covers + // what is below it, but is no target. + let unknown = flag(&window.on_current_desktop).is_none() + || flag(&window.on_current_activity).is_none(); + let Ok(mut info) = WindowInfo::try_from(window) else { + continue; + }; + info.workspace = unknown.then_some(-1); + windows.push(info); + } + Ok(windows) +} + pub(crate) async fn logical_desktop_rect() -> Result<(i32, i32, i32, i32)> { let json = call_kwin_window_script().await?; parse_kwin_logical_desktop_rect(&json) @@ -713,6 +773,26 @@ fn kwin_window_script_source( return []; }} + function onCurrentActivity(window) {{ + try {{ + var activities = window.activities; + if (activities === null || activities === undefined) {{ + return null; + }} + if (activities.length === 0) {{ + return true; + }} + var current = workspace.currentActivity; + // "" when no activity manager runs. + if (typeof current !== "string" || current === "") {{ + return null; + }} + return Array.prototype.indexOf.call(activities, current) >= 0; + }} catch (error) {{ + return null; + }} + }} + var activeWindow = null; try {{ activeWindow = "activeWindow" in workspace ? workspace.activeWindow : workspace.activeClient; @@ -738,6 +818,8 @@ fn kwin_window_script_source( workspace: firstDesktop(window), minimized: read(window, "minimized"), onCurrentDesktop: onCurrentDesktop(window), + onCurrentActivity: onCurrentActivity(window), + hidden: read(window, "hidden"), active: read(window, "active") || window === activeWindow, clientType: clientType(window), normalWindow: read(window, "normalWindow"), @@ -755,12 +837,25 @@ fn kwin_window_script_source( }}; }}); + function stackingOrder() {{ + try {{ + var order = workspace.stackingOrder; + if (order && typeof order.length === "number") {{ + return Array.prototype.map.call(order, function(window) {{ + return read(window, "uuid") || read(window, "internalId"); + }}); + }} + }} catch (error) {{}} + return null; + }} + callDBus(serviceName, objectPath, iface, "ReceiveWindows", JSON.stringify({{ backend: "kwin", pluginName: pluginName, desktopGeometry: workspaceGeometry(), outputs: activeOutputs(), - windows: windows + windows: windows, + stacking: stackingOrder() }})); }})(); "# @@ -1010,6 +1105,9 @@ struct KwinWindowSnapshot { #[serde(default)] outputs: Vec, windows: Vec, + /// Window uuids bottom to top. + #[serde(default)] + stacking: Option>>, } impl KwinWindowSnapshot { @@ -1069,6 +1167,10 @@ struct KwinRawWindow { workspace: Option, minimized: Option, on_current_desktop: Option, + #[serde(default)] + on_current_activity: Option, + #[serde(default)] + hidden: Option, active: Option, client_type: Option, normal_window: Option, @@ -2777,3 +2879,73 @@ mod transaction_tests { drop(bus); } } + +#[cfg(test)] +mod pen_stacking_tests { + use super::parse_kwin_windows_stacked; + + fn window(uuid: &str, extra: &str) -> String { + let mut window = serde_json::json!({ + "uuid": format!("{{{uuid}}}"), "caption": uuid, "x": 0, "y": 0, "width": 100, + "height": 100, "minimized": false, "onCurrentDesktop": true, + "onCurrentActivity": true, "hidden": false, + }); + if !extra.is_empty() { + let more: serde_json::Value = + serde_json::from_str(&format!("{{{}}}", &extra[1..])).unwrap(); + for (key, value) in more.as_object().unwrap() { + window[key] = value.clone(); + } + } + window.to_string() + } + + #[test] + fn the_stacked_list_holds_only_visible_windows_bottom_to_top() { + let windows = [ + window("00000000-0000-0000-0000-000000000001", ""), + window("00000000-0000-0000-0000-000000000002", r#","hidden":true"#), + window( + "00000000-0000-0000-0000-000000000003", + r#","onCurrentActivity":false"#, + ), + window( + "00000000-0000-0000-0000-000000000004", + r#","onCurrentDesktop":false"#, + ), + window( + "00000000-0000-0000-0000-000000000005", + r#","minimized":true"#, + ), + window( + "00000000-0000-0000-0000-000000000006", + r#","onCurrentActivity":null"#, + ), + window( + "00000000-0000-0000-0000-000000000007", + r#","desktopWindow":true"#, + ), + ]; + // Listed in reverse, stacked in order: the stacking order wins. + let stacking = (1..=7) + .map(|n| format!(r#""{{00000000-0000-0000-0000-00000000000{n}}}""#)) + .collect::>() + .join(","); + let json = format!( + r#"{{"windows":[{}],"stacking":[{stacking}]}}"#, + windows.iter().rev().cloned().collect::>().join(",") + ); + let stacked = parse_kwin_windows_stacked(&json).unwrap(); + let titles: Vec<_> = stacked + .iter() + .map(|window| (window.title.clone().unwrap(), window.workspace)) + .collect(); + assert_eq!( + titles, + [ + ("00000000-0000-0000-0000-000000000001".to_string(), None), + ("00000000-0000-0000-0000-000000000006".to_string(), Some(-1)), + ] + ); + } +} diff --git a/tests/virtual_pen.rs b/tests/virtual_pen.rs index 7704810b..fc64e4e2 100644 --- a/tests/virtual_pen.rs +++ b/tests/virtual_pen.rs @@ -1,17 +1,24 @@ // SPDX-License-Identifier: MIT -//! delivery="background" click, drag and scroll through the GNOME virtual pen -//! and touchscreen, on a real GNOME Shell that owns a real seat. Opt-in: see -//! scripts/virtual_pen_test.py. +//! delivery="background" click, drag and scroll through the virtual pen and +//! touchscreen, on a real GNOME Shell and a real KWin that own a real seat. +//! Opt-in: see scripts/virtual_pen_test.py and +//! scripts/virtual_pen_kwin_test.py. -#[test] -fn gnome_pen_clicks_drags_and_scrolls_without_moving_the_user_pointer() { +fn fixture(script: &str) { let status = std::process::Command::new("/usr/bin/python3") - .arg(concat!( - env!("CARGO_MANIFEST_DIR"), - "/scripts/virtual_pen_test.py" - )) + .arg(format!("{}/scripts/{script}", env!("CARGO_MANIFEST_DIR"))) .arg(env!("CARGO_BIN_EXE_computer-use-linux")) .status() .expect("run the virtual pen fixture"); - assert!(status.success(), "virtual pen fixture failed"); + assert!(status.success(), "{script} failed"); +} + +#[test] +fn gnome_pen_clicks_drags_and_scrolls_without_moving_the_user_pointer() { + fixture("virtual_pen_test.py"); +} + +#[test] +fn kwin_pen_clicks_drags_and_scrolls_without_moving_the_user_pointer() { + fixture("virtual_pen_kwin_test.py"); }