diff --git a/CHANGELOG.md b/CHANGELOG.md index 204c9e55..54940bcf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Fixed + +- A background right click on GNOME 48 and later arrived as a middle click and a middle click as a right click: GNOME 48 changed the stylus button defaults (`button-action` to `right`, `secondary-button-action` to `middle`), and mutter rewrites the pen's buttons by them. The server now sets both actions for its own pen (the settings path keyed by its vendor and product id) before every action, so mutter hands clients the button the pen pressed, and resets them when the pen goes away. Measured with GNOME 48's defaults on GNOME Shell 46, with GTK 4 and with native Wayland and Xwayland Chromium; mutter 48 and later read the same explicit values (from mutter's source). +- A background scroll within 8 seconds of a background click or drag sent nothing: libinput pairs a pen with the first touchscreen it sees and drops that screen's touches while the pen is in proximity, and for 90 ms after it leaves. The same pairing could silence the user's own touchscreen while the pen hovered. The pen now leaves proximity at the end of each action instead of hovering for 8 seconds, and a scroll waits out the 90 ms. + ## [0.7.58] - 2026-11-20 ### Added diff --git a/README.md b/README.md index 2e06ad6c..26f5cbdf 100644 --- a/README.md +++ b/README.md @@ -181,7 +181,7 @@ On Hyprland with `COMPUTER_USE_LINUX_HYPRLAND_BACKGROUND=1`, a window target wit Where there is no agent seat, a window-targeted background call has no route (the notice, then focused input); element, browser tab and terminal targets still work. For full isolation, with its own keyboard and pointer, run the app in an [agent-workspace-linux](https://github.com/agent-sh/agent-workspace-linux) workspace; the notice points there. `doctor` lists the routes this machine has in `capabilities.background_input` (`seat` when the agent seat is available) and the seat's route, pointer support or the reason it has none in `capabilities.background_seat`. `capabilities.background_pointer` shows the GNOME pen route and its limits, or why there is none. -On GNOME Wayland, background `click`, `drag` and `scroll` go through a virtual pen and a virtual touchscreen instead of the user's pointer ([#480](https://github.com/agent-sh/computer-use-linux/issues/480); the idea and the tested approach are [@Ondra-Zik](https://github.com/Ondra-Zik)'s). Mutter gives a tablet tool its own cursor and sends its events through the tablet protocol, not `wl_pointer`, so the user's mouse stays where it is. Clicks are pen taps (left), the upper barrel button (right) and the lower one (middle); `click_count` repeats them. A drag is the pen tip pressed at the start, moved and lifted at the end. A scroll is a one-finger touch pan, 60 logical pixels per wheel notch (5 notches per page), split into strokes that stay inside the window and 48 pixels from the monitor's edges, where GNOME Shell starts its edge swipes; the finger rests before it lifts, so the app does not fling further. Before any input, the server checks through the GNOME Shell extension that the point is on the target window, on the workspace the user sees, and that no window stacked above it covers the point; both ends of a drag and every pan stroke must be on the window, and a pan always starts at the point. The results report the route as `gnome-pen` or `gnome-touch`. The devices need write access to `/dev/uinput`, as the uinput pointer does; without it, or without the extension, the call gets the notice and then focused input. +On GNOME Wayland, background `click`, `drag` and `scroll` go through a virtual pen and a virtual touchscreen instead of the user's pointer ([#480](https://github.com/agent-sh/computer-use-linux/issues/480); the idea and the tested approach are [@Ondra-Zik](https://github.com/Ondra-Zik)'s). Mutter gives a tablet tool its own cursor and sends its events through the tablet protocol, not `wl_pointer`, so the user's mouse stays where it is. Clicks are pen taps (left), the upper barrel button (right) and the lower one (middle); `click_count` repeats them. GNOME 48 and later default the lower button to a right click and the upper one to a middle click, so before each action the server sets its own pen's button actions to match the buttons it presses, and resets them when the pen goes away. A drag is the pen tip pressed at the start, moved and lifted at the end. The pen leaves proximity after each action: libinput pairs a pen with a touchscreen and ignores that screen's touches while the pen is near, and that screen can be the user's. A scroll is a one-finger touch pan, 60 logical pixels per wheel notch (5 notches per page), split into strokes that stay inside the window and 48 pixels from the monitor's edges, where GNOME Shell starts its edge swipes; the finger rests before it lifts, so the app does not fling further. Before any input, the server checks through the GNOME Shell extension that the point is on the target window, on the workspace the user sees, and that no window stacked above it covers the point; both ends of a drag and every pan stroke must be on the window, and a pan always starts at the point. The results report the route as `gnome-pen` or `gnome-touch`. The devices need write access to `/dev/uinput`, as the uinput pointer does; without it, or without the extension, the call gets the notice and then focused input. Limits: @@ -189,7 +189,7 @@ Limits: - The window check sees windows, not GNOME Shell's own UI. The call is refused while the screen is locked, while no window has focus (the overview may be open), and when the target is not on the workspace of the focused window, but a Shell dialog, notification banner or open menu over the point is not seen. - Mutter sends tablet events only to clients that support the tablet protocol (GTK 3 and 4, Qt, Chromium, Firefox and Xwayland do). Other clients never see the pen, and the call cannot tell: re-observe the window. This is also why the pen is not the default route: the uinput pointer reaches every client. - A drag stays inside one window. A touch pan scrolls only where the app pans on touch. -- The pen hovers at its last point, with its own cursor, for 8 seconds after an action, then leaves proximity; hover effects such as tooltips end then. +- The pen leaves proximity right after each action, so hover effects such as tooltips end with it. - Mutter maps an unconfigured tablet across the whole layout, and an unconfigured touchscreen to the builtin panel. When a device maps elsewhere than the point needs, the server writes that device's `output` setting (`org.gnome.desktop.peripherals.tablet` or `.touchscreen`, at the path keyed by the devices' own vendor and product ids, `6375:7065` and `6375:7463`), waits for mutter's `org.gnome.Mutter.InputMapping` to confirm the new mapping, and resets the setting when the device goes away. No other device's setting is touched. - The devices appear on first use and go away after ten idle minutes or when the server exits. diff --git a/plugins/computer-use-linux/skills/computer-use-linux/SKILL.md b/plugins/computer-use-linux/skills/computer-use-linux/SKILL.md index 6fc7f387..4567672f 100644 --- a/plugins/computer-use-linux/skills/computer-use-linux/SKILL.md +++ b/plugins/computer-use-linux/skills/computer-use-linux/SKILL.md @@ -236,7 +236,7 @@ directly to a window-relative click. - `mcp --listen ADDR:PORT` serves streamable HTTP at `/mcp` and requires a token from `COMPUTER_USE_LINUX_MCP_TOKEN_FILE` or `COMPUTER_USE_LINUX_MCP_TOKEN`. A non-loopback bind needs `--allowed-host NAME`. The port is plain HTTP; put TLS in front. Do not retry a 401 by putting the token in a URL. Over HTTP, `get_tree_page` and index and selector calls need `snapshot_id`; if refused, call `get_app_state` again. Idle sessions expire after an hour; on `404 Session not found`, initialize again. The cached tree and its `snapshot_id` survive. - On GNOME Wayland, the remote-control portal asks on each new `computer-use-linux` process. `COMPUTER_USE_LINUX_PERSIST_REMOTE_DESKTOP=1` in the server environment asks the portal to remember the grant (GNOME starts with the remember box checked). Later processes reuse a single-use restore token stored mode 0600 in the user state directory. Unset is the default and prompts every time. The flag does nothing when ydotool, not the portal, is the input backend. - On Hyprland, `COMPUTER_USE_LINUX_HYPRLAND_BACKGROUND=1` lets targeted `type_text`, `press_key`, `get_app_state` and `screenshot` work on a window without focusing it, including one on a hidden workspace. Pass `window_id` (or another target) on every call. An AT-SPI action can move focus inside the app, so re-focus a field with its `activate` action before typing into it. The active Hyprland keymap must be US, or background `type_text` and `press_key` calls are refused; background `get_app_state` and `screenshot` still capture. Text must be printable ASCII, tab or newline. An unfocused Chromium ignores browser accelerators such as Ctrl+R, and Chromium joins AT-SPI only when started with `ACCESSIBILITY_ENABLED=1` if the accessibility bus reports itself disabled. -- To act without taking the user's keyboard, pass `delivery: "background"` to `type_text` or `press_key` (naming an element or `browser_tab` implies it). Name an element from `get_app_state` (text is inserted and read back; Enter or space runs a button's, link's, check box's or single-line field's action), a `browser_tab` when `COMPUTER_USE_LINUX_DEVTOOLS` names the port of a browser the user started with `--remote-debugging-port`, or a terminal by `tty` or terminal selectors (kitty remote control, wezterm, tmux or GNU screen). On X11 and sway 1.10+ a plain window target also works: the agent gets its own keyboard focus (an XInput 2 master or a transient seat), and `click`/`scroll` with `delivery: "background"`, `x`/`y` and a window target use the agent's own pointer (X11 only with no window manager). On sway, Xwayland, GTK and Chromium windows and Alt/Super chords have no background route; on X11, window manager shortcuts have none. `doctor` shows the seat in `capabilities.background_seat`. On GNOME Wayland, `click`, `drag` and `scroll` with `delivery: "background"`, a window target and points on that window use a virtual pen and touchscreen: the user's mouse stays put, but a click focuses the window, apps without tablet support ignore the pen (re-observe after the call), a drag stays in one window, a scroll is a touch pan, and hover lasts 8 s (`capabilities.background_pointer`). When no route serves a call, the first one in the session sends nothing and returns a notice (`background.retry_to_continue: true`) with the reason; repeating the call sends it through normal input, which moves the user's keyboard focus and pointer, and later calls with no route do that without asking again (`background.used: false`). Repeat only if taking the user's focus is acceptable; otherwise use the setting the reason names, or agent-workspace-linux (https://github.com/agent-sh/agent-workspace-linux), which gives the agent its own session with its own keyboard and pointer. A wrong or ambiguous target is still an error. +- To act without taking the user's keyboard, pass `delivery: "background"` to `type_text` or `press_key` (naming an element or `browser_tab` implies it). Name an element from `get_app_state` (text is inserted and read back; Enter or space runs a button's, link's, check box's or single-line field's action), a `browser_tab` when `COMPUTER_USE_LINUX_DEVTOOLS` names the port of a browser the user started with `--remote-debugging-port`, or a terminal by `tty` or terminal selectors (kitty remote control, wezterm, tmux or GNU screen). On X11 and sway 1.10+ a plain window target also works: the agent gets its own keyboard focus (an XInput 2 master or a transient seat), and `click`/`scroll` with `delivery: "background"`, `x`/`y` and a window target use the agent's own pointer (X11 only with no window manager). On sway, Xwayland, GTK and Chromium windows and Alt/Super chords have no background route; on X11, window manager shortcuts have none. `doctor` shows the seat in `capabilities.background_seat`. On GNOME Wayland, `click`, `drag` and `scroll` with `delivery: "background"`, a window target and points on that window use a virtual pen and touchscreen: the user's mouse stays put, but a click focuses the window, apps without tablet support ignore the pen (re-observe after the call), a drag stays in one window, and a scroll is a touch pan (`capabilities.background_pointer`). When no route serves a call, the first one in the session sends nothing and returns a notice (`background.retry_to_continue: true`) with the reason; repeating the call sends it through normal input, which moves the user's keyboard focus and pointer, and later calls with no route do that without asking again (`background.used: false`). Repeat only if taking the user's focus is acceptable; otherwise use the setting the reason names, or agent-workspace-linux (https://github.com/agent-sh/agent-workspace-linux), which gives the agent its own session with its own keyboard and pointer. A wrong or ambiguous target is still an error. - Desktop input is stateful. Avoid concurrent tool calls against this MCP server. - Pi serializes the native Computer Use tools and keeps one process for the session. If that process exits, do not replay an ambiguous mutating call; obtain a fresh `get_app_state` before another element-based action. - `click`, `drag`, `press_key`, `type_text`, `perform_action`, and `set_value` can change real application state. diff --git a/scripts/virtual_pen_test.py b/scripts/virtual_pen_test.py index eff972cf..15fb355b 100755 --- a/scripts/virtual_pen_test.py +++ b/scripts/virtual_pen_test.py @@ -8,7 +8,7 @@ output, with libinput reading the seat's devices: a headless mutter reads none, so it never sees a uinput device. The fixture needs passwordless sudo for that (modprobe vkms, an ACL on /dev/uinput, a login session on the -terminal, chvt), and refuses to run while any graphical session is on seat0: +terminal, its getty masked for the run, chvt), and refuses to run while any graphical session is on seat0: uinput devices are machine-wide, so this must never run on a desktop someone uses. Without the variable it skips. @@ -18,14 +18,22 @@ scroll go through the pen and the touchscreen: the window sees pen and touch events at the points, and no mouse event, so the user's pointer stays where the normal click left it. A point under another window is refused before -input. After the server exits no virtual device and no tablet or touchscreen -setting is left behind. The pen's proximity ends HOVER seconds after its last -action, and wev, which binds no tablet, shows what an app without tablet -support sees of a pen click. +input. The session's schemas carry GNOME 48's stylus defaults, which swap the +barrel buttons, and a right click must still be a right click, also after a +second server's pen came and went (they share one settings path). The pen +leaves proximity right after each action, and a scroll right after a pen +click must still pan (libinput drops touches while a paired pen is near). +When Chromium or Google Chrome is installed, the same clicks, a double +click, a drag and a scroll go to a page in it, native Wayland and under +Xwayland, which must see pen and touch input with the buttons named. After +the server exits no virtual device and no tablet, stylus or touchscreen +setting is left behind. wev, which binds no tablet, shows what an app +without tablet support sees of a pen click. """ import contextlib import json import os +import re from pathlib import Path import shutil import subprocess @@ -39,7 +47,6 @@ UUID = 'computer-use-linux@avifenesh.dev' EXTENSION = Path(__file__).resolve().parent.parent / 'gnome-shell-extension' / UUID DISPLAY_NAME = 'cul-pen-0' -HOVER = 8 def gtk_app(root): @@ -239,13 +246,36 @@ def run(binary, root, tty): user = subprocess.run(['id', '-un'], capture_output=True, text=True, check=True).stdout.strip() subprocess.run(['sudo', '-n', 'setfacl', '-m', f'u:{user}:rw', '/dev/uinput'], check=True) cleanup.callback(subprocess.run, ['sudo', '-n', 'setfacl', '-x', f'u:{user}', '/dev/uinput']) + # Without --no-x11 mutter listens for X clients and starts Xwayland + # for the first one: the Chromium check runs under it too. + xwayland = shutil.which('Xwayland') + # GNOME 48's stylus defaults (gsettings-desktop-schemas f25d3e7c) as + # schema overrides: the lower button a right click, the upper one a + # middle click. GNOME 46 passes the codes through. + schemas = root / 'schemas' + schemas.mkdir() + for source in Path('/usr/share/glib-2.0/schemas').iterdir(): + if source.suffix in ('.xml', '.override'): + shutil.copy(source, schemas / source.name) + (schemas / 'zz_cul-gnome48-stylus.gschema.override').write_text( + "[org.gnome.desktop.peripherals.tablet.stylus]\nbutton-action='right'\nsecondary-button-action='middle'\n") + subprocess.run(['glib-compile-schemas', str(schemas)], check=True) + env['GSETTINGS_SCHEMA_DIR'] = str(schemas) launcher = root / 'shell.sh' exports = ' '.join(f"{key}='{env[key]}'" for key in ( 'DBUS_SESSION_BUS_ADDRESS', 'XDG_RUNTIME_DIR', 'XDG_CONFIG_HOME', 'XDG_DATA_HOME', 'XDG_CACHE_HOME', - 'XDG_SESSION_TYPE', 'XDG_CURRENT_DESKTOP')) + 'XDG_SESSION_TYPE', 'XDG_CURRENT_DESKTOP', 'GSETTINGS_SCHEMA_DIR')) launcher.write_text(f'#!/bin/bash\nexport {exports}\nexec gnome-shell --wayland --display-server ' - f'--no-x11 --wayland-display={DISPLAY_NAME} > {root}/shell.log 2>&1\n') + f'{"" if xwayland else "--no-x11 "}--wayland-display={DISPLAY_NAME} > {root}/shell.log 2>&1\n') launcher.chmod(0o755) + # logind starts a getty on the active terminal when no session holds + # it (autovt), and one left from an earlier run takes the terminal + # back and hangs up the shell's session. Masked until the end of the + # run (--runtime: nothing survives a reboot either). + gettys = [f'getty@{tty}.service', f'autovt@{tty}.service'] + subprocess.run(['sudo', '-n', 'systemctl', 'mask', '--runtime', '--now', *gettys], capture_output=True) + cleanup.callback(subprocess.run, ['sudo', '-n', 'systemctl', 'unmask', '--runtime', *gettys], + capture_output=True) unit = f'cul-pen-shell-{os.getpid()}' subprocess.run(['sudo', '-n', 'systemd-run', '--quiet', f'--unit={unit}', '--collect', '-p', 'PAMName=login', '-p', f'TTYPath=/dev/{tty}', '-p', 'StandardInput=tty-force', '-p', f'User={user}', @@ -258,6 +288,14 @@ def run(binary, root, tty): 'dev.avifenesh.ComputerUseLinux.WindowControl'] wait_for(lambda: 'true' in subprocess.run(owner, env=env, capture_output=True, text=True).stdout, 'the GNOME Shell extension on the bus', 30) + x_display = None + if xwayland: + # mutter names the display it took: "Using public X11 display :0". + x_display = wait_for(lambda: (re.search(r'public X11 display (:\d+)', (root / 'shell.log').read_text()) + or [None, None])[1], 'the Xwayland display', 20) + # mutter writes the X authority it gives Xwayland's clients into + # its runtime directory. + x_authority = wait_for(lambda: sorted(runtime.glob('.mutter-Xwaylandauth.*')), 'the Xwayland authority', 20)[0] app = subprocess.Popen(['/usr/bin/python3', __file__, '--gtk', str(root)], env=dict(env, GTK_A11Y='none'), stdout=(root / 'app.log').open('w'), stderr=subprocess.STDOUT) cleanup.callback(stop, app) @@ -311,9 +349,18 @@ def pen_presses(since, button, point): presses = pen_presses(since, code, point) check(failures, len(presses) == 1, f'{button} click: one pen press with button {code} at {point}: {events(root, since)[:8]}') - # libinput's 50 ms proximity timeout is off: the pen still hovers. - check(failures, presses and not [e for e in events(root, presses[0]['t']) if e['kind'] == 'proximity-out'], - f'{button} click: the pen still hovers 0.4 s later') + # The pen leaves proximity after the action. + check(failures, presses and [e for e in events(root, presses[0]['t']) if e['kind'] == 'proximity-out'], + f'{button} click: the pen left proximity after it') + if button == 'left': + # A second server's pen comes and goes; its reset of the + # shared stylus settings must not swap this server's buttons. + other = Server(binary, env) + theirs = other.call('click', dict(at((px + 40, py + 500)), window_id=wid, delivery='background')) + check(failures, theirs.get('ok') and (theirs.get('background') or {}).get('used') is True, + f'a second server clicks through its own pen: {theirs.get("message")}') + other.close() + time.sleep(.5) point = (px + 280, py + 120) since = time.monotonic() background('click', **at(point), click_count=2) @@ -338,6 +385,18 @@ def pen_presses(since, button, point): check(failures, any(e['kind'] == 'touch-begin' and e['source'] == 'touch' for e in seen) and scrolled and max(scrolled) > 100, f'scroll: touch pan moved the list down: {scrolled[-3:]}') + # libinput drops touches while a paired pen is in proximity and for + # 90 ms after: a scroll right after a pen click must still pan. + background('click', **at((px + 200, py + 400))) + before = max(scrolled) if scrolled else 0 + since = time.monotonic() + background('scroll', **at(point), direction='down', pages=1) + time.sleep(1) + seen = events(root, since) + scrolled = [e['value'] for e in seen if e['kind'] == 'scrolled'] + check(failures, any(e['kind'] == 'touch-begin' and e['source'] == 'touch' for e in seen) + and scrolled and max(scrolled) > before + 100, + f'scroll after a pen click: touch pan moved the list further: {scrolled[-3:]}') # The user's pointer never moved: no mouse event after the pen began, # and GTK still has it where the normal click left it. mouse = [e for e in events(root, pen_start) if e.get('source') == 'mouse'] @@ -360,14 +419,25 @@ def pen_presses(since, button, point): stop(cover) time.sleep(.5) check(failures, not [e for e in events(root, since) if e['kind'] == 'press'], 'refused click sent no press') - # Hover: the pen leaves proximity HOVER seconds after its last action. + # The pen leaves proximity right after the release. last = time.monotonic() background('click', **at((px + 120, py + 300))) - press = wait_for(lambda: [e for e in events(root, last) if e['kind'] == 'press'], 'hover click press', 5)[0]['t'] - out = wait_for(lambda: [e for e in events(root, press) if e['kind'] == 'proximity-out'], 'pen proximity out', HOVER + 6) - hover = out[0]['t'] - press - print(f'pen hovered {hover:.1f} s', flush=True) - check(failures, HOVER - 1 <= hover <= HOVER + 3, f'hover ended after {hover:.1f} s') + press = wait_for(lambda: [e for e in events(root, last) if e['kind'] == 'press'], 'click press', 5)[0]['t'] + out = wait_for(lambda: [e for e in events(root, press) if e['kind'] == 'proximity-out'], 'pen proximity out', 3) + print(f'pen left proximity {out[0]["t"] - press:.2f} s after the press', flush=True) + check(failures, out[0]['t'] - press < .5, f'pen left proximity {out[0]["t"] - press:.2f} s after the press') + chrome = chrome_binary() + if chrome: + logger = chrome_logger(root) + cleanup.callback(logger.shutdown) + for platform in ('wayland', 'x11'): + if platform == 'x11' and not x_display: + print('SKIP chrome x11: no Xwayland', flush=True) + continue + chrome_checks(failures, server, root, dict(env, DISPLAY=x_display, XAUTHORITY=str(x_authority)) if x_display else env, + chrome, platform, logger.server_address[1], cleanup) + else: + print('SKIP chrome: no chromium or google-chrome', flush=True) # An app without tablet support: wev logs every pointer and touch event. if shutil.which('wev'): wev = subprocess.Popen(['wev'], env=env, stdout=(root / 'wev.log').open('w'), stderr=subprocess.STDOUT) @@ -389,9 +459,170 @@ def pen_presses(since, button, point): check(failures, not leftover_devices(), f'no virtual device after exit: {leftover_devices()}') dump = subprocess.run(['dconf', 'dump', '/org/gnome/desktop/peripherals/'], env=env, capture_output=True, text=True).stdout - check(failures, 'tablets' not in dump and 'touchscreens' not in dump, f'no tablet or touch setting left: {dump}') + check(failures, 'tablets' not in dump and 'touchscreens' not in dump and 'stylus' not in dump, + f'no tablet, stylus or touch setting left: {dump}') return failures +CHROME_PAGE = """cul-pen-chrome + +
+""" + + +def chrome_binary(): + return next((path for path in (shutil.which(name) for name in ('chromium', 'chromium-browser', 'google-chrome')) + if path), None) + + +def chrome_logger(root): + """Serves the page and appends what it posts to chrome.jsonl.""" + import http.server + import threading + log = (root / 'chrome.jsonl').open('a', buffering=1) + + class Handler(http.server.BaseHTTPRequestHandler): + def do_GET(self): + body = CHROME_PAGE.encode() + self.send_response(200) + self.send_header('Content-Type', 'text/html') + self.send_header('Content-Length', str(len(body))) + self.end_headers() + self.wfile.write(body) + + def do_POST(self): + entry = json.loads(self.rfile.read(int(self.headers['Content-Length']))) + entry['t'] = time.monotonic() + log.write(json.dumps(entry) + '\n') + self.send_response(204) + self.end_headers() + + def log_message(self, *_): + pass + server = http.server.ThreadingHTTPServer(('127.0.0.1', 0), Handler) + threading.Thread(target=server.serve_forever, daemon=True).start() + return server + + +def chrome_events(root, since=0.0): + path = root / 'chrome.jsonl' + if not path.exists(): + return [] + return [entry for entry in (json.loads(line) for line in path.read_text().splitlines() if line.strip()) + if entry['t'] >= since] + + +def chrome_checks(failures, server, root, env, chrome, platform, port, cleanup): + """Background clicks, a drag and a scroll on Chromium, native Wayland or + under Xwayland: each must reach the page as the pen or touch input it is, + with the button the call named.""" + name = f'chrome {platform}' + profile = root / f'chrome-{platform}' + chrome_env = dict(env) + if platform == 'x11': + chrome_env.pop('WAYLAND_DISPLAY', None) + process = subprocess.Popen([chrome, f'--user-data-dir={profile}', '--no-first-run', '--no-default-browser-check', + '--password-store=basic', '--disable-gpu', f'--ozone-platform={platform}', + '--window-size=1000,700', f'--app=http://127.0.0.1:{port}/?{platform}'], + env=chrome_env, stdout=(root / f'chrome-{platform}.log').open('w'), + stderr=subprocess.STDOUT) + cleanup.callback(stop, process) + started = time.monotonic() + ready = wait_for(lambda: [e for e in chrome_events(root, started) if e['kind'] == 'ready'], f'{name} page', 40)[-1] + window = wait_for(lambda: next((w for w in server.call('list_windows', {})['windows'] + if w.get('title') == 'cul-pen-chrome'), None), f'{name} window', 20) + wid = window['window_id'] + check(failures, server.call('activate_window', dict(window_id=wid)).get('ok'), f'{name}: activate') + time.sleep(1.5) + bounds = next(w for w in server.call('list_windows', {})['windows'] if w['window_id'] == wid)['bounds'] + print(f'{name} window {window.get("app_id")} at {bounds}, page {ready}', flush=True) + pad, scroller = ready['pad'], ready['list'] + # The page's client area ends at the window's bottom right, inside any + # frame; the first click calibrates the exact offset. + guess = (bounds['x'] + bounds['width'] - (pad[2] + scroller[2]), bounds['y'] + bounds['height'] - pad[3]) + + def screen(local, offset): + return dict(x=int(offset[0] + local[0]), y=int(offset[1] + local[1])) + + def call(tool, **arguments): + result = server.call(tool, dict(arguments, window_id=wid, delivery='background')) + report = result.get('background') or {} + check(failures, result.get('ok') and report.get('used') is True, + f'{name} background {tool} {arguments}: {result.get("message")} {report}') + return result + + def seen(since, kind, **match): + return [e for e in chrome_events(root, since) if e['kind'] == kind + and all(e.get(key) == value for key, value in match.items())] + since = time.monotonic() + point = (pad[0] + 150, pad[1] + 150) + result = call('click', **screen(point, guess)) + check(failures, (result.get('background') or {}).get('route') == 'gnome-pen', f'{name} left route: {result.get("background")}') + clicks = wait_for(lambda: seen(since, 'click'), f'{name} left click', 5) + downs = seen(since, 'pointerdown') + print(f'{name} left click: {downs} {clicks}', flush=True) + offset = (guess[0] + point[0] - clicks[0]['x'], guess[1] + point[1] - clicks[0]['y']) + check(failures, downs and downs[0]['button'] == 0 and downs[0]['type'] == 'pen', + f'{name} left click: a pen pointerdown with button 0: {downs}') + + def near_page(entry, local, slack=3): + return abs(entry['x'] - local[0]) <= slack and abs(entry['y'] - local[1]) <= slack + for button, dom_button, kind, offset_x in (('right', 2, 'contextmenu', 250), ('middle', 1, 'auxclick', 350)): + point = (pad[0] + offset_x, pad[1] + 150) + since = time.monotonic() + call('click', **screen(point, offset), button=button) + time.sleep(.6) + downs = seen(since, 'pointerdown') + check(failures, len(downs) == 1 and downs[0]['button'] == dom_button and downs[0]['type'] == 'pen' + and near_page(downs[0], point) and seen(since, kind), + f'{name} {button} click: one pen pointerdown with button {dom_button} at {point} and a {kind}: ' + f'{chrome_events(root, since)[:6]}') + if button == 'right': + check(failures, not seen(since, 'auxclick', button=1), f'{name} right click is not a middle click') + point = (pad[0] + 150, pad[1] + 300) + since = time.monotonic() + call('click', **screen(point, offset), click_count=2) + time.sleep(.8) + check(failures, len(seen(since, 'pointerdown', button=0)) == 2 and seen(since, 'dblclick'), + f'{name} double click: two pointerdowns and a dblclick: {chrome_events(root, since)[:8]}') + start, end = (pad[0] + 100, pad[1] + 400), (pad[0] + 300, pad[1] + 480) + since = time.monotonic() + call('drag', start_x=screen(start, offset)['x'], start_y=screen(start, offset)['y'], + end_x=screen(end, offset)['x'], end_y=screen(end, offset)['y']) + time.sleep(.6) + downs, ups = seen(since, 'pointerdown'), seen(since, 'pointerup') + moves = [e for e in seen(since, 'pointermove') if e['buttons'] == 1] + check(failures, downs and near_page(downs[0], start) and ups and near_page(ups[-1], end) and len(moves) >= 5 + and all(e['type'] == 'pen' for e in downs + ups + moves), + f'{name} drag: pen down at {start}, {len(moves)} moves, up at {end}: {downs[:1]} {ups[-1:]}') + point = (scroller[0] + scroller[2] / 2, scroller[1] + scroller[3] / 2) + since = time.monotonic() + result = call('scroll', **screen(point, offset), direction='down', pages=1) + check(failures, (result.get('background') or {}).get('route') == 'gnome-touch', f'{name} scroll route: {result.get("background")}') + time.sleep(1.2) + scrolled = [e['value'] for e in seen(since, 'scrolled')] + check(failures, scrolled and max(scrolled) > 100, f'{name} scroll: the list scrolled down: {scrolled[-3:]}') + mouse = [e for e in chrome_events(root, started) if e.get('type') == 'mouse' and e['kind'] != 'pointermove'] + check(failures, not mouse, f'{name}: no mouse button events: {mouse[:4]}') + stop(process) + UDEV_RULE = 'SUBSYSTEM=="drm", KERNEL=="card[0-9]*", ENV{ID_PATH}!="platform-vkms", TAG+="mutter-device-ignore"\n' @@ -436,7 +667,8 @@ def main(): keep = os.environ.get('CUL_PEN_TEST_KEEP_LOGS') if keep: Path(keep).mkdir(parents=True, exist_ok=True) - for name in ('shell.log', 'app.log', 'events.jsonl', 'wev.log'): + for name in ('shell.log', 'app.log', 'events.jsonl', 'wev.log', 'chrome.jsonl', + 'chrome-wayland.log', 'chrome-x11.log'): if (root / name).exists(): shutil.copy(root / name, Path(keep) / name) if failures: diff --git a/skills/computer-use-linux/SKILL.md b/skills/computer-use-linux/SKILL.md index 6fc7f387..4567672f 100644 --- a/skills/computer-use-linux/SKILL.md +++ b/skills/computer-use-linux/SKILL.md @@ -236,7 +236,7 @@ directly to a window-relative click. - `mcp --listen ADDR:PORT` serves streamable HTTP at `/mcp` and requires a token from `COMPUTER_USE_LINUX_MCP_TOKEN_FILE` or `COMPUTER_USE_LINUX_MCP_TOKEN`. A non-loopback bind needs `--allowed-host NAME`. The port is plain HTTP; put TLS in front. Do not retry a 401 by putting the token in a URL. Over HTTP, `get_tree_page` and index and selector calls need `snapshot_id`; if refused, call `get_app_state` again. Idle sessions expire after an hour; on `404 Session not found`, initialize again. The cached tree and its `snapshot_id` survive. - On GNOME Wayland, the remote-control portal asks on each new `computer-use-linux` process. `COMPUTER_USE_LINUX_PERSIST_REMOTE_DESKTOP=1` in the server environment asks the portal to remember the grant (GNOME starts with the remember box checked). Later processes reuse a single-use restore token stored mode 0600 in the user state directory. Unset is the default and prompts every time. The flag does nothing when ydotool, not the portal, is the input backend. - On Hyprland, `COMPUTER_USE_LINUX_HYPRLAND_BACKGROUND=1` lets targeted `type_text`, `press_key`, `get_app_state` and `screenshot` work on a window without focusing it, including one on a hidden workspace. Pass `window_id` (or another target) on every call. An AT-SPI action can move focus inside the app, so re-focus a field with its `activate` action before typing into it. The active Hyprland keymap must be US, or background `type_text` and `press_key` calls are refused; background `get_app_state` and `screenshot` still capture. Text must be printable ASCII, tab or newline. An unfocused Chromium ignores browser accelerators such as Ctrl+R, and Chromium joins AT-SPI only when started with `ACCESSIBILITY_ENABLED=1` if the accessibility bus reports itself disabled. -- To act without taking the user's keyboard, pass `delivery: "background"` to `type_text` or `press_key` (naming an element or `browser_tab` implies it). Name an element from `get_app_state` (text is inserted and read back; Enter or space runs a button's, link's, check box's or single-line field's action), a `browser_tab` when `COMPUTER_USE_LINUX_DEVTOOLS` names the port of a browser the user started with `--remote-debugging-port`, or a terminal by `tty` or terminal selectors (kitty remote control, wezterm, tmux or GNU screen). On X11 and sway 1.10+ a plain window target also works: the agent gets its own keyboard focus (an XInput 2 master or a transient seat), and `click`/`scroll` with `delivery: "background"`, `x`/`y` and a window target use the agent's own pointer (X11 only with no window manager). On sway, Xwayland, GTK and Chromium windows and Alt/Super chords have no background route; on X11, window manager shortcuts have none. `doctor` shows the seat in `capabilities.background_seat`. On GNOME Wayland, `click`, `drag` and `scroll` with `delivery: "background"`, a window target and points on that window use a virtual pen and touchscreen: the user's mouse stays put, but a click focuses the window, apps without tablet support ignore the pen (re-observe after the call), a drag stays in one window, a scroll is a touch pan, and hover lasts 8 s (`capabilities.background_pointer`). When no route serves a call, the first one in the session sends nothing and returns a notice (`background.retry_to_continue: true`) with the reason; repeating the call sends it through normal input, which moves the user's keyboard focus and pointer, and later calls with no route do that without asking again (`background.used: false`). Repeat only if taking the user's focus is acceptable; otherwise use the setting the reason names, or agent-workspace-linux (https://github.com/agent-sh/agent-workspace-linux), which gives the agent its own session with its own keyboard and pointer. A wrong or ambiguous target is still an error. +- To act without taking the user's keyboard, pass `delivery: "background"` to `type_text` or `press_key` (naming an element or `browser_tab` implies it). Name an element from `get_app_state` (text is inserted and read back; Enter or space runs a button's, link's, check box's or single-line field's action), a `browser_tab` when `COMPUTER_USE_LINUX_DEVTOOLS` names the port of a browser the user started with `--remote-debugging-port`, or a terminal by `tty` or terminal selectors (kitty remote control, wezterm, tmux or GNU screen). On X11 and sway 1.10+ a plain window target also works: the agent gets its own keyboard focus (an XInput 2 master or a transient seat), and `click`/`scroll` with `delivery: "background"`, `x`/`y` and a window target use the agent's own pointer (X11 only with no window manager). On sway, Xwayland, GTK and Chromium windows and Alt/Super chords have no background route; on X11, window manager shortcuts have none. `doctor` shows the seat in `capabilities.background_seat`. On GNOME Wayland, `click`, `drag` and `scroll` with `delivery: "background"`, a window target and points on that window use a virtual pen and touchscreen: the user's mouse stays put, but a click focuses the window, apps without tablet support ignore the pen (re-observe after the call), a drag stays in one window, and a scroll is a touch pan (`capabilities.background_pointer`). When no route serves a call, the first one in the session sends nothing and returns a notice (`background.retry_to_continue: true`) with the reason; repeating the call sends it through normal input, which moves the user's keyboard focus and pointer, and later calls with no route do that without asking again (`background.used: false`). Repeat only if taking the user's focus is acceptable; otherwise use the setting the reason names, or agent-workspace-linux (https://github.com/agent-sh/agent-workspace-linux), which gives the agent its own session with its own keyboard and pointer. A wrong or ambiguous target is still an error. - Desktop input is stateful. Avoid concurrent tool calls against this MCP server. - Pi serializes the native Computer Use tools and keeps one process for the session. If that process exits, do not replay an ambiguous mutating call; obtain a fresh `get_app_state` before another element-based action. - `click`, `drag`, `press_key`, `type_text`, `perform_action`, and `set_value` can change real application state. diff --git a/src/server.rs b/src/server.rs index 1aecffa5..619fe653 100644 --- a/src/server.rs +++ b/src/server.rs @@ -11450,7 +11450,7 @@ impl ComputerUseLinux { let limits = if route == pen::TOUCH_ROUTE { "A touch pan scrolls only where the app pans on touch; re-observe to confirm." } else { - "The pen hovers at its last point for up to 8 s. An app without tablet support does not see pen input; re-observe to confirm." + "An app without tablet support does not see pen input; re-observe to confirm." }; BackgroundOutcome::Done( successful_action_with_focus( diff --git a/src/virtual_pen.rs b/src/virtual_pen.rs index 3d900cb7..3006e125 100644 --- a/src/virtual_pen.rs +++ b/src/virtual_pen.rs @@ -19,7 +19,9 @@ //! (GTK 3 and 4, Qt, Chromium, Firefox and Xwayland bind it); //! - scrolling is a one-finger touch pan, which only scrolls where the app //! pans on touch; -//! - a hovering pen leaves proximity after [HOVER_LIFETIME]. +//! - the pen leaves proximity right after each action: libinput pairs a +//! tablet with the first touchscreen it sees and drops that screen's +//! touches while the pen is in proximity, which can be the user's own. //! //! Mapping: mutter maps a tablet or touchscreen to one monitor when its //! "output" setting (org.gnome.desktop.peripherals.tablet or .touchscreen, @@ -35,8 +37,20 @@ //! device goes away. Those settings paths belong to this server's own //! vendor and product ids; no other device's setting is touched. //! -//! Lifecycle: created on first use, the pen leaves proximity after -//! [HOVER_LIFETIME], and both devices go away after +//! Buttons: mutter rewrites a stylus button by the tool's button-action +//! settings before it reaches a client (meta-seat-impl.c), at a path keyed +//! by the device's ids for a tool with no serial. GNOME 48 changed their +//! defaults from passing the evdev code through to "right" for the lower +//! button (BTN_STYLUS) and "middle" for the upper one (BTN_STYLUS2; +//! gsettings-desktop-schemas f25d3e7c), the opposite of what GTK, Xwayland, +//! Chromium and KWin make of the bare codes. So the pen writes both actions +//! for its own tool (lower: middle, upper: right) before every action, which +//! gives every client the code the pen pressed, and resets them when it goes +//! away. Every server's pen shares that path, so one server's reset must not +//! leave another's pen on the defaults. Never write "default" for this: +//! mutter 48 and later turn it into a left click. +//! +//! Lifecycle: created on first use, and both devices go away after //! [crate::agent_seat::IDLE_LIFETIME] without use, or when the server exits. use crate::agent_seat::SessionFacts; @@ -53,8 +67,6 @@ use std::time::{Duration, Instant}; pub(crate) const PEN_ROUTE: &str = "gnome-pen"; /// The route name results report for touch scrolling. pub(crate) const TOUCH_ROUTE: &str = "gnome-touch"; -/// How long the pen hovers after an action before it leaves proximity. -pub(crate) const HOVER_LIFETIME: Duration = Duration::from_secs(8); /// The pan distance of one wheel notch, in logical pixels. pub(crate) const PAN_PER_NOTCH: f64 = 60.0; /// A pan stays this far inside the window and the monitor edges: GNOME @@ -71,6 +83,11 @@ const AXIS_MAX: i32 = 65535; const PRESSURE_MAX: i32 = 4095; const MAPPING_TIMEOUT: Duration = Duration::from_secs(3); const STEP: Duration = Duration::from_millis(12); +/// libinput ignores a touchscreen paired with a pen for 90 ms after the pen +/// leaves proximity, counted from the event's own time. +const TOUCH_AFTER_PEN: Duration = Duration::from_millis(200); +/// A new touchscreen's wait before its first touch. +const TOUCH_SETTLE: Duration = Duration::from_millis(300); const PEN_NAME: &str = "computer-use-linux virtual pen"; const TOUCH_NAME: &str = "computer-use-linux virtual touch"; @@ -112,7 +129,7 @@ pub(crate) fn doctor_summary() -> serde_json::Value { Ok(()) => serde_json::json!({ "available": true, "routes": [PEN_ROUTE, TOUCH_ROUTE], - "note": "Background click and drag use a virtual pen, background scroll a one-finger touch pan; the user's mouse pointer does not move. A pen tap or a touch gives the window keyboard focus. Apps without tablet support never see the pen, scrolling needs touch panning in the app, a drag stays inside one window, and the pen hovers for up to 8 s after an action. Needs the computer-use-linux GNOME Shell extension to check the window under the point.", + "note": "Background click and drag use a virtual pen, background scroll a one-finger touch pan; the user's mouse pointer does not move. A pen tap or a touch gives the window keyboard focus. Apps without tablet support never see the pen, scrolling needs touch panning in the app, and a drag stays inside one window. Needs the computer-use-linux GNOME Shell extension to check the window under the point.", }), Err(reason) => serde_json::json!({ "available": false, "reason": reason }), } @@ -462,8 +479,9 @@ pub(crate) fn pan_bounds(window: &Rect, monitor: &Rect) -> Rect { pub(crate) enum PenButton { /// The tip. Tip, - /// BTN_STYLUS, the lower barrel button: a middle click (GTK and - /// Xwayland map it to button 2, as xf86-input-wacom does). + /// BTN_STYLUS, the lower barrel button: a middle click. GTK, Xwayland, + /// Chromium and KWin map it to button 2, as xf86-input-wacom does; the + /// pen's button actions keep mutter from rewriting it. Lower, /// BTN_STYLUS2, the upper barrel button: a right click (button 3). Upper, @@ -529,6 +547,8 @@ pub(crate) struct PenEvents { events: E, in_proximity: bool, tip_down: bool, + /// When the pen last left proximity. + left_at: Option, } impl PenEvents { @@ -537,6 +557,7 @@ impl PenEvents { events, in_proximity: false, tip_down: false, + left_at: None, } } @@ -618,6 +639,7 @@ impl PenEvents { ])?; self.in_proximity = false; self.tip_down = false; + self.left_at = Some(Instant::now()); Ok(()) } } @@ -814,10 +836,56 @@ fn gvariant_string(value: &str) -> String { format!("'{}'", value.replace('\\', "\\\\").replace('\'', "\\'")) } +/// The settings of the pen's tool: a uinput pen has no serial, so mutter +/// keys them by the device's vendor and product id. +fn stylus_settings() -> String { + format!( + "org.gnome.desktop.peripherals.tablet.stylus:/org/gnome/desktop/peripherals/stylus/default-{VENDOR:04x}:{PEN_PRODUCT:04x}/" + ) +} + +/// The pen's button actions, so mutter hands clients the evdev code the pen +/// pressed (BTN_STYLUS: middle, BTN_STYLUS2: right) whatever the GNOME +/// version's defaults are. Written before the device exists: mutter reads +/// them when the tool first enters proximity. +const STYLUS_ACTIONS: [(&str, &str); 2] = [ + ("button-action", "middle"), + ("secondary-button-action", "right"), +]; + +/// Writes the actions that differ, so an unchanged setting sends no change +/// signal to every settings listener. +fn set_stylus_actions() -> Result<()> { + for (key, value) in STYLUS_ACTIONS { + let current = std::process::Command::new("gsettings") + .args(["get", &stylus_settings(), key]) + .stdin(std::process::Stdio::null()) + .stderr(std::process::Stdio::null()) + .output() + .ok() + .filter(|output| output.status.success()) + .map(|output| String::from_utf8_lossy(&output.stdout).trim().to_string()); + if current.as_deref() != Some(gvariant_string(value).as_str()) { + gsettings_set(&stylus_settings(), key, value)?; + } + } + Ok(()) +} + +fn reset_stylus_actions() { + for (key, _) in STYLUS_ACTIONS { + gsettings_reset(&stylus_settings(), key); + } +} + fn set_output(kind: Kind, output: [&str; 4]) -> Result<()> { let value = format!("[{}]", output.map(gvariant_string).join(", ")); + gsettings_set(&kind.settings(), "output", &value) +} + +fn gsettings_set(settings: &str, key: &str, value: &str) -> Result<()> { let status = std::process::Command::new("gsettings") - .args(["set", &kind.settings(), "output", &value]) + .args(["set", settings, key, value]) .stdin(std::process::Stdio::null()) .stdout(std::process::Stdio::null()) .stderr(std::process::Stdio::piped()) @@ -825,8 +893,7 @@ fn set_output(kind: Kind, output: [&str; 4]) -> Result<()> { .context("failed to run gsettings")?; if !status.status.success() { bail!( - "gsettings set {} output failed: {}", - kind.settings(), + "gsettings set {settings} {key} failed: {}", String::from_utf8_lossy(&status.stderr).trim() ); } @@ -834,8 +901,12 @@ fn set_output(kind: Kind, output: [&str; 4]) -> Result<()> { } fn reset_output(kind: Kind) { + gsettings_reset(&kind.settings(), "output"); +} + +fn gsettings_reset(settings: &str, key: &str) { let _ = std::process::Command::new("gsettings") - .args(["reset", &kind.settings(), "output"]) + .args(["reset", settings, key]) .stdin(std::process::Stdio::null()) .stdout(std::process::Stdio::null()) .stderr(std::process::Stdio::null()) @@ -856,6 +927,7 @@ impl Drop for Pen { if self.wrote_output { reset_output(Kind::Pen); } + reset_stylus_actions(); } } @@ -888,8 +960,7 @@ fn lock_devices() -> std::sync::MutexGuard<'static, Option> { .unwrap_or_else(|poisoned| poisoned.into_inner()) } -/// Lifts the pen out of proximity after [HOVER_LIFETIME] and removes the -/// devices after [crate::agent_seat::IDLE_LIFETIME]. One at a time; it exits +/// Removes the devices after [crate::agent_seat::IDLE_LIFETIME]. One at a time; it exits /// once the devices are gone. fn start_reaper() { let spawned = std::thread::Builder::new() @@ -903,18 +974,15 @@ fn start_reaper() { }; let idle = devices.last_use.elapsed(); if idle >= crate::agent_seat::IDLE_LIFETIME { - let _process = process_lock(Duration::from_secs(2)); - *guard = None; - REAPER.store(false, std::sync::atomic::Ordering::SeqCst); - return; - } - if idle >= HOVER_LIFETIME { - if let Some(pen) = devices.pen.as_mut() { - let _ = pen.events.leave(); + // Removing resets settings every server's pen shares: + // not while another server is using its pen. + if let Ok(_process) = process_lock(Duration::from_secs(2)) { + *guard = None; + REAPER.store(false, std::sync::atomic::Ordering::SeqCst); + return; } } - // A later action can start a new hover: look again soon. - Duration::from_millis(500) + crate::agent_seat::IDLE_LIFETIME.saturating_sub(idle) + Duration::from_millis(500) }; std::thread::sleep(wait); }); @@ -925,7 +993,7 @@ fn start_reaper() { /// Removes both devices now. Called when the server exits. pub(crate) fn shutdown() { - let _process = process_lock(Duration::from_secs(2)); + let _process = process_lock(Duration::from_secs(10)); *lock_devices() = None; } @@ -1074,6 +1142,7 @@ fn run_pen( if devices.pen.is_none() { reset_output(Kind::Pen); let (device, node) = create(Kind::Pen).map_err(|error| { + reset_stylus_actions(); PenError::NoRoute(format!("the virtual pen could not be made: {error:#}")) })?; devices.pen = Some(Pen { @@ -1104,8 +1173,15 @@ fn run_pen( let started = |error: anyhow::Error| PenError::Partial(format!("{error:#}")); // libinput lifts a tablet out of proximity 50 ms after its last event // until the tablet sends a proximity out of its own. One in and out at - // the first point, once mutter has the device, shows that it does, so - // the pen then hovers until HOVER_LIFETIME lifts it. + // the first point, once mutter has the device, shows that it does. + // Before every action, under the process lock and right before the pen + // comes near: mutter reads them as the pen enters proximity, and another + // server's pen going away resets the shared path. + set_stylus_actions().map_err(|error| { + PenError::NotSent(format!( + "the pen's button actions could not be set: {error:#}" + )) + })?; if !pen.primed { let value = device_value(first, &area); pen.events @@ -1149,6 +1225,12 @@ fn run_pen( } Gesture::Pan { .. } => unreachable!("pans go through the touchscreen"), } + // A hovering pen would silence the touchscreen libinput paired it with, + // and does for 90 ms after it leaves; hold the process lock past that, + // so no server's scroll starts inside it. + std::thread::sleep(Duration::from_millis(30)); + events.leave().map_err(started)?; + std::thread::sleep(TOUCH_AFTER_PEN); Ok(()) } @@ -1171,6 +1253,22 @@ fn run_touch( node, wrote_output: false, }); + // mutter lists a new touchscreen before it has mapped it, and the + // first pan sent at once never reached the window. + std::thread::sleep(TOUCH_SETTLE); + } + // libinput pairs a tablet with the first touchscreen it sees and, for a + // pen with no tilt, drops every touch while the pen is in proximity + // (evdev-tablet.c, tablet_toggle_touch_device), and for 90 ms after it + // leaves (evdev-fallback.c, the arbitration timer). The pen leaves at the + // end of each action; wait out the rest of that. + if let Some(pen) = devices.pen.as_mut() { + pen.events + .leave() + .map_err(|error| PenError::NotSent(format!("{error:#}")))?; + if let Some(left) = pen.events.left_at { + std::thread::sleep(TOUCH_AFTER_PEN.saturating_sub(left.elapsed())); + } } let touch = devices.touch.as_mut().expect("touch was just made"); let area = covering_area(