Skip to content

feat: background pointer input on KDE Plasma through the virtual pen - #494

Merged
avifenesh merged 4 commits into
mainfrom
feat/kwin-pen
Nov 20, 2026
Merged

avifenesh merged 4 commits into
mainfrom
feat/kwin-pen

Conversation

@avifenesh

@avifenesh avifenesh commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Follow-up to #489 and #480: the KDE Plasma half, and the answer on hover.

What changes

KDE Plasma Wayland with KWin 6.3 or later gets the virtual pen and touchscreen for click, drag and scroll with delivery: "background", as routes kwin-pen and kwin-touch. The user's mouse pointer stays where it is.

  • Why KWin 6.3: KWin tracks the tablet cursor itself from 6.3 (4681e9819f) and moves it whether or not the client under it supports tablets (663f7342d6). Both commits are in v6.3.0 and not in v6.2.0. kwin_wayland --version is read once.
  • Mapping: KWin maps a tablet with no setting to the active output, which follows the user's pointer, and a touchscreen to the internal panel or the first output (backends/libinput/connection.cpp, applyScreenToDevice). Before input the server sets the pen's mapToWorkspace (KWin then maps it across workspace()->geometry(), tabletToolPosition) and the touchscreen's outputName (the output under the point) through org.kde.KWin.InputDevice on the devices' own /org/kde/KWin/InputDevice/eventN. It reads each value back before any event.
  • Cleanup: KWin writes those settings into kcminputrc ([Libinput][25461][28773][computer-use-linux virtual pen] and the touch group). Setting them back to defaults leaves MapToWorkspace=false and a stale OutputUuid, so once the devices are gone the server deletes the keys with kwriteconfig6 --delete --notify. The settings guard is the last field of each device, so it runs after the device is destroyed: KWin writes a setting back when it changes on a live device. No other device's group is touched.
  • Window check: KWin scripting now also returns workspace.stackingOrder, each window's hidden flag and whether it is on the current activity. The point must be on the target, and the target must be on the desktop and activity the user sees, not minimized or hidden. No window above it may cover the point: normal windows, docks, popups and notifications all count. A target whose desktop or activity KWin does not report is refused.
  • Rotation: KWin scripting has no output rotation, so a scroll reads it from kscreen-doctor -j and refuses a rotated output, or one whose rotation it cannot read, as on GNOME.
  • No route: KWin's [Tablet] SyncWithMouse option (92b48ae5df) moves the user's pointer to where the pen was, so with it on there is no route. The same holds without kreadconfig6/kwriteconfig6. The lock screen is read from org.freedesktop.ScreenSaver.
  • Refusals before input: the call is refused while KWin shows the desktop (showingDesktop) or runs a full-screen effect (org.kde.kwin.Effects.activeEffects: Overview, Window View and the like). It is also refused when the device carries a calibration, input or output area or rotation other than its default. KWIN_WAYLAND_EMULATE_TABLET in KWin's environment leaves no route, since it sends pen input over apps without tablet support through the user's pointer.

Hover (item 3 of the follow-up)

No hover tool. The server has no pointer-move tool on any route, so a background-only hover would be a new tool surface (schema, Pi catalog, plugins, safety classification) for an effect that reaches tablet-aware clients only. #493 also removed the 8 s lingering hover: a hovering pen makes libinput drop touches on the touchscreen it paired with, which can be the user's.

AWSL

agent-workspace-linux's backends cannot exercise uinput tablets. Its KWin runs --virtual, and KWin creates a libinput backend only for its DRM backend (DrmBackend::createInputBackend); the other backends return none. GNOME runs --headless, which reads no input devices. Hyprland and COSMIC run nested or headless too. So the KWin test runs KWin's DRM backend itself. KWin falls back to its Noop session (a plain open()) when no logind is reachable, so it runs as root in a privileged container on a vkms output, with no logind or VT. An AWSL change would have meant a DRM-backed workspace mode, which is not a small change.

Tests

scripts/virtual_pen_kwin_test.py (tests/virtual_pen.rs, opt-in with COMPUTER_USE_LINUX_TEST_KWIN_SEAT=1) runs KWin 6.6.7 from Ubuntu 26.04 (scripts/virtual_pen_kwin.Dockerfile) with --drm on vkms in a privileged container. It uses --network host, because libinput hears about new devices through udev's netlink group, which a separate network namespace never receives. A GTK 4 window logs its events. Plasma's screenshots go through the desktop portal, which asks the user, so a stand-in org.gnome.Shell.Screenshot answers the server's first screenshot route with a blank image of the output's size.

The checks:

  • doctor offers kwin-pen/kwin-touch.
  • Left, right and middle clicks arrive as pen presses with buttons 1/3/2 at the points, and the pen leaves proximity after each.
  • kcminputrc holds MapToWorkspace=true for the pen while it runs.
  • A double click gives two pen presses.
  • A drag gives a pen press, moves and a release at the ends.
  • Two scrolls (the second right after a pen click) are touch pans that move the list.
  • No mouse movement or button event happens, and a relative nudge shows the user's pointer where it was.
  • A point under another window is refused with nothing sent.
  • After exit there is no virtual device and no computer-use-linux virtual group in kcminputrc.
  • With SyncWithMouse=true, doctor reports no route and names the option.
  • Show Desktop is refused, and so is a minimized target.
  • Not covered: two outputs. This kernel's vkms has one, so the run cannot tell mapToWorkspace from KWin's active-output default; it checks the setting KWin persisted.

Results on the dev desk:

The GNOME seat fixture now reloads udev's rules after writing its own, since the container's bind mount of /run/udev left udevd missing the new rule. It also unloads vkms with retries. Both fixtures pass back to back.

fmt, clippy -D warnings, cargo test --no-fail-fast, mcp_safety_check.py, check_plugins.sh and the Pi catalog --check pass on the dev desk. The drag and delivery schema descriptions name KDE Plasma, and the Pi catalog is regenerated.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 100bfab. Configure here.

Comment thread src/server.rs
}
}
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Focus message names GNOME on Plasma

Low Severity

A successful KWin pen or touch action that moves keyboard focus still reports that GNOME gave the window focus. pen_focus_sentence is shared by both desktops and hard-codes GNOME, so Plasma results tell the agent the wrong compositor took the user's typing.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 100bfab. Configure here.

@avifenesh

Copy link
Copy Markdown
Collaborator Author

Fresh-context review: last commit only (100bfab, KDE Plasma virtual pen on top of #493). KWin claims checked against KDE/kwin master and v6.3.0.

Confirmed upstream: mapToWorkspace maps across workspace()->geometry() (backends/libinput/connection.cpp tabletToolPosition); a tablet with no output falls back to workspace()->activeOutput(); the kcminputrc group is [Libinput][vendor][product][name] (applyDeviceConfig); setOutputName writes OutputUuid; KWin re-applies config on a notify only to pointer devices (slotKGlobalSettingsNotifyChange), so a delete never changes a live pen or touchscreen; SyncWithMouse is read from [Tablet] in input.cpp. The drop order (device first, then the settings guard) is right.

High

  1. The window check accepts a target the user cannot see, so input lands somewhere else. src/windowing/backends/kwin.rs:96-100 drops only desktop, minimized and onCurrentDesktop == false windows, and src/virtual_pen.rs:533 (check_point_on_visible_target) treats everything left as visible. Three cases get through. In each, the pen lands on a window below the target or on the Plasma desktop window, which the list leaves out, so a double click there can open a desktop icon:

    • Show Desktop. KWin hides every window with setHiddenByShowDesktop, not minimized, and isShown() (window.cpp) checks that flag. Scripting does not expose it, but org.kde.KWin /KWin has a showingDesktop D-Bus property (dbusinterface.h:49).
    • Activities. A window on another activity is in workspace.windowList() and passes onCurrentDesktop. Compare window.activities (empty means all) with workspace.currentActivity.
    • hidden. Window::hidden is a Q_PROPERTY and is not read.

    A target whose onCurrentDesktop is unknown (null) is also kept (flag(...) == Some(false)). For the target, unknown should refuse. For windows that might cover the point, keeping them is the safe choice.

Medium

  1. KWIN_WAYLAND_EMULATE_TABLET=1 moves and clicks the user's pointer. With it set, input.cpp emulateTabletEvent sends a tablet event over a client without tablet support through input()->pointer()->processMotionAbsolute and processButton. That code is in v6.3.0 and in master. kwin_ready (src/virtual_pen.rs:213) does not check it. The variable lives in kwin_wayland's own environment (/proc/<pid>/environ, same user), not the server's.

  2. The SyncWithMouse check only matches the exact string "true" (src/virtual_pen.rs:230). KConfig reads any bool that is not false/no/off/0 as true, so True, 1 or yes turn the option on in KWin while the server still offers the route. Use kreadconfig6 --type bool.

  3. An open Overview can be detected. The known-limit note says no reading exists, and that is true of KWin scripting. But org.kde.KWin /Effects exports org.kde.kwin.Effects.activeEffects (src/org.kde.kwin.Effects.xml). Overview, Window View and the other full-screen QuickSceneEffects are listed there while they run. A pen tap in the Overview picks or closes windows, so this is worth closing rather than documenting.

  4. The test does not prove the mapping change. The fixture has one vkms output, so the active output is the whole workspace. The pen lands in the same place with or without mapToWorkspace, and the touchscreen's default output is the only output. The kcminputrc assertion shows the setting was written, not that it decides where input goes. Other gaps:

    • Nothing covers the stacking filter beyond one normal window above the target: no minimized target, other desktop, dock, popup, Show Desktop or activity case.
    • There is no unit test for parse_kwin_windows_stacked / check_point_on_visible_target. That is pure logic, and only a unit test reaches the filter branches.
    • --no-lockscreen means the org.freedesktop.ScreenSaver path never runs.

    A second vkms output (configfs, kernel 6.x), or an output offset with the user's pointer on the other output, would make the pen part fail without this commit.

Low

  1. Possible 1 px error at the far edge. KWin scales the tablet axis to size - 1 (backends/libinput/events.cpp:348, (size.width() - 1) * outputArea.width()), while device_value assumes size. A point lands up to about 1 logical px toward the workspace origin at the far edge, so a point within a pixel of a target's left or top edge, far from the origin, can reach the neighbouring window. Scale by size - 1 for KWin, or keep points 1 px inside the target.
  2. Only mapToWorkspace and outputName are read back. applyDeviceConfig falls back to [Libinput][Defaults] for keys the device group lacks, so a default OutputArea, TabletToolRelativeMode, Rotation or Calibration changes the mapping without any check. All four are D-Bus properties on the same device object. Reading back outputArea == (0,0,1,1), tabletToolIsRelative == false and rotation == 0 costs one call each.
  3. Blocking calls on the async runtime. availability() runs kreadconfig6 on the tokio worker for every background call and every doctor run. The first call also runs kwin_wayland --version, with no timeout. That version is the installed binary's, which can differ from the running KWin after an upgrade with no re-login.
  4. The doctor can promise a touch route it will refuse. When kscreen-doctor is missing, the doctor still lists kwin-touch, but every scroll is refused (src/server.rs:11403). Either report the limit or drop the route.
  5. Nits:
    • pen_windows is inserted between pen_focus_sentence's doc comment and the function (src/server.rs:10786-10802), so that doc comment now belongs to pen_windows.
    • Layout::new still says "mutter reported no monitor" on KWin (src/virtual_pen.rs:403).

@revuto-review revuto-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an auto review done by revuto.


Two findings in the KWin window check; both are in the new stacking path, not in the device or kcminputrc handling.

  1. The server's own indicator overlay blocks background actions on KDE. The KWin check treats every window above the target as covering the point, and the indicator's click-through layer-shell surfaces are such windows. With the indicator on (the default), points under them are refused. The KWin fixture runs with COMPUTER_USE_LINUX_INDICATOR='0', so it never exercised this.
  2. The target's visibility check fails open. A target whose desktop state reads as unknown, or that is on another Plasma Activity, passes as "on the desktop the user sees".

I read the code only; nothing was built or run, and the KWin behaviour for layer-shell windows and Activities is from KWin's documented scripting API, not from a live session.

Comment thread src/virtual_pen.rs
Comment on lines +557 to +568
let top = windows
.iter()
.rev()
.find(|window| window_rect(window).is_some_and(|rect| rect.contains(point)))
.expect("the target contains the point");
if top.window_id != target_id {
return Err(format!(
"window_id {} ({}) is stacked above window_id {target_id} at that point",
top.window_id,
top.title.as_deref().unwrap_or("untitled")
));
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an auto review done by revuto.


On KDE this counts the server's own indicator overlay as a covering window, so background actions under it are refused with "window_id N (untitled) is stacked above …".

computer-use-linux-indicator is on by default and draws on Layer::Overlay layer-shell surfaces (src/bin/computer-use-linux-indicator/main.rs:507). KWin lists those in workspace.stackingOrder like the panels and notifications this check is meant to see. parse_kwin_windows_stacked only drops desktop, minimized and other-desktop windows, so the overlay's surfaces stay in the list, above every normal window.

The surfaces have an empty input region (main.rs:503-505), so they never take the pen; the refusal is a false positive. While the overlay is up (8 s after any tool call, IDLE_BEFORE_FADE), the refused areas are:

  • a 26 px strip along every edge of every output (EDGE), e.g. the scrollbar of a maximized window;
  • the 560×56 status pill at top centre, y 34–90 (PILL_SIZE, PILL_TOP), e.g. the tab strip or toolbar of a maximized window;
  • the 112×112 cursor sprite at the last focused click, scroll or drag point.

GNOME is not affected: the extension draws Shell actors, which are not in its window list. scripts/virtual_pen_kwin_test.py:71 sets COMPUTER_USE_LINUX_INDICATOR='0', so the fixture never saw this.

Suggested fix: leave the overlay's surfaces out of the stack (the script already reads each window's pid, which can be matched to the overlay process), or hide the overlay around the check as hold_for_capture does for screenshots. A fixture run with the indicator on and a click near an output edge would cover it.

Comment on lines +97 to +102
if flag(&window.desktop_window) == Some(true)
|| flag(&window.minimized) == Some(true)
|| flag(&window.on_current_desktop) == Some(false)
{
continue;
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an auto review done by revuto.


This filter fails open for the target window in two cases, so the pen can be sent at a point where the target is not actually on screen.

  • Unknown desktop state. onCurrentDesktop() in the script returns null when it cannot tell, and only Some(false) is dropped here. A target with null passes as visible. The existing visible_element_window requires == Some(true) for the same decision (and has the unknown_or_other_desktop_visibility_never_dispatches test). Keeping unknown windows is right for the windows that may cover the point, but the target itself should need Some(true).
  • Activities. The script reads neither window.activities nor workspace.currentActivity. A window on another Activity is hidden by KWin but is not minimized, can share the current desktop id, and stays in the stacking order. If it is the target and nothing listed above it contains the point, the tap goes to whatever the user actually sees there. The same gap makes hidden other-Activity windows count as covering, which only causes refusals.

Suggested fix: carry an onCurrentActivity flag from the script (empty activities, or it contains currentActivity), drop windows where it is false, and have check_point_on_visible_target refuse a target whose desktop or activity state is not known to be current.

KWin 6.3 and later give a tablet tool its own cursor, as mutter does, so
the virtual pen and touchscreen now serve background click, drag and
scroll on KDE Plasma Wayland too, as routes kwin-pen and kwin-touch.

KWin maps a tablet with no setting to the active output, which follows the
user's pointer, and a touchscreen to the internal panel or the first
output. Before input the server sets the pen's mapToWorkspace and the
touchscreen's outputName through org.kde.KWin.InputDevice and reads them
back; once the devices are gone it removes what KWin wrote for them into
kcminputrc with kwriteconfig6. The point is checked against KWin's
stacking order through KWin scripting, panels, popups and notifications
included; a scroll reads the output's rotation with kscreen-doctor. KWin's
SyncWithMouse tablet option, which moves the user's pointer to the pen,
leaves no route.

scripts/virtual_pen_kwin_test.py runs KWin 6.6 from Ubuntu 26.04 in a
container on a vkms output with its DRM backend (KWin's virtual backend
reads no input devices), opt-in with COMPUTER_USE_LINUX_TEST_KWIN_SEAT=1.
- The window list leaves out hidden windows and windows on another
  activity; a target whose desktop or activity KWin did not report is
  refused, and a minimized or hidden target is named as such.
- Show Desktop and full-screen effects (org.kde.kwin.Effects.activeEffects:
  Overview, Window View and the like) refuse the call.
- KWIN_WAYLAND_EMULATE_TABLET in KWin's environment leaves no route.
- SyncWithMouse is read as KConfig reads a boolean (kreadconfig6 --type
  bool answers in its exit status).
- A calibration, input or output area or rotation other than the default
  on the device refuses the call before any event.
- doctor offers kwin-touch only with kscreen-doctor installed.
- The fixture adds Show Desktop and a minimized target.
@avifenesh

Copy link
Copy Markdown
Collaborator Author

Fresh-context review addressed in e93a454 (on main now that #493 landed at 3a0cd24):

  1. High, visibility: the stacked list drops hidden windows (hidden) and windows on another activity (activities vs workspace.currentActivity), as well as minimized ones and those on other desktops. A target whose desktop or activity KWin did not report is refused. Show Desktop (org.kde.KWin.showingDesktop) refuses the call. A minimized or hidden target gets its own message. The fixture now covers Show Desktop, holding the D-Bus client, because KWin ends Show Desktop when the asking client leaves, and a minimized target.
  2. KWIN_WAYLAND_EMULATE_TABLET (non-empty and not 0) in the environment of org.kde.KWin's owner refuses the call.
  3. SyncWithMouse is read with kreadconfig6 --type bool --default false, which answers in its exit status (checked: True and 1 read as on). The fixture's check failed on the string reading and passes now.
  4. Full-screen effects: activeEffects containing overview, windowview, desktopgrid, expo, tileseditor or cube refuses the call. Not run in the fixture: this container's KWin lacks the QML modules the Overview needs.
  5. Tests: Show Desktop and the minimized target are added. Two outputs are not covered: this kernel's vkms has one output, so the fixture cannot tell mapToWorkspace from the active-output default. It does check the setting KWin persisted.
  6. Low, edge rounding: left as is. The error is at most 1 logical px at the far edge, and both ends of a stroke are checked inside the window.
  7. Calibration, input or output area or rotation differing from the device's default refuses the call before any event.
  8. Blocking calls: kreadconfig6 is a few ms per call. kwin_wayland --version runs once per server and reports the installed KWin, which is the running one unless KWin was upgraded without a restart.
  9. doctor offers kwin-touch only when kscreen-doctor is installed.
  10. Nits: fixed.

KWin fixture, GNOME fixture, fmt, clippy, tests, MCP safety, plugins and the Pi catalog pass on the dev desk.

@avifenesh

Copy link
Copy Markdown
Collaborator Author

Fresh-context re-review: latest commit only (e93a454, "fix: review findings on the KWin pen"). KWin claims checked against KDE/kwin master and v6.3.0.

Verified:

  • org.kde.KWin /KWin showingDesktop and org.kde.kwin.Effects.activeEffects exist as used.
  • QuickSceneEffect::isActive() is true only while the effect runs (src/effect/quickeffect.cpp:453), so the names list matches an open Overview or Window View and not merely a loaded one. For master, overview, windowview and tileseditor are the in-tree full-screen effects; the other names do no harm.
  • The default* device properties used for the calibration, output area, input area and rotation check exist (device.h, defaultRotation in master only, which the if let (Ok, Ok) skip handles).
  • kreadconfig6 --type bool closes the SyncWithMouse spelling gap.
  • An unknown desktop or activity now refuses as a target and still counts as cover.
  • Show Desktop and the effects check refuse when KWin does not answer.

Earlier items 1, 3, 4, 9 and 10 are resolved. Item 6 is left as the author argued: both stroke ends are checked inside the window, so the remaining risk is a sub-pixel edge case. Item 8 is accepted as replied.

Medium

  1. The emulation check passes when it cannot read KWin's environment (src/virtual_pen.rs, kwin_blocker). If GetConnectionUnixProcessID fails or times out, or /proc/<pid>/environ cannot be read, the ? returns None, which means "nothing stands in the way". The other two checks in the same function refuse when there is no answer. An unread environment should refuse too, or at least be reported in the result: with the variable set, the pen moves and clicks the user's pointer.
  2. Relative tablet mode is not read back (src/virtual_pen.rs, kwin_covering_area, the new settings loop). KWin master has tabletToolIsRelative (device.h:196, config key TabletToolRelativeMode, which can come from [Libinput][Defaults] like the others). In relative mode the pen moves the tablet cursor from its last position, so input lands away from the checked point. There is no default property for it, so compare it with false when the property exists.

Low

  1. The new tests prove only part of the fix. The Show Desktop case fails before this commit and passes after it, so it does. The minimized-target case does not: the old check_point_on_visible_target message already said "or is minimized", and minimized windows were already filtered, so it passed before. Nothing exercises the hidden filter, activities, unknown visibility, the effects refusal, the environment check or the device settings refusal. A unit test over parse_kwin_windows_stacked with hand-written JSON (hidden, other activity, null activity, desktop window) is pure logic that no fixture reaches, and it would cover four of these cheaply.
  2. Activities when KWin reports no current activity. onCurrentActivity in the KWin script treats workspace.currentActivity === "" (activity manager not running) as a real activity. Every window with a non-empty activity list then reads as off the activity and is dropped from the cover list. A window with an empty list (all activities) can still be a target, so a window above it would no longer count. Treat "" as unknown (null), which keeps such windows as cover and refuses them as targets.
  3. The check runs long before the input. kwin_blocker runs before the window script, the monitor read and the device mapping wait (up to MAPPING_TIMEOUT each), so Show Desktop or the Overview can start in between. A second showingDesktop/activeEffects read inside perform, after the mapping read-back and just before the first event, would narrow the gap to milliseconds. This is the same class as the GNOME path, so it is not a blocker.
  4. Nit: WindowInfo.workspace = Some(-1) now means "visibility unknown" in windows_stacked only. A named field or a comment where check_point_on_visible_target reads it would keep later callers from reading it as a desktop number.

No High findings.

- The KWIN_WAYLAND_EMULATE_TABLET check refuses when KWin's environment
  cannot be read, as the other checks refuse when KWin does not answer.
- A pen KWin runs in relative mode refuses the call.
- An empty current activity (no activity manager) counts as unknown.
- A unit test pins the stacked list: hidden, other-activity, other-desktop,
  minimized and desktop windows are left out, a window of unknown activity
  stays as cover only, and stacking order wins over listing order.
@avifenesh

Copy link
Copy Markdown
Collaborator Author

Re-review addressed in fbf1f55:

  1. The KWIN_WAYLAND_EMULATE_TABLET check now refuses when the session bus, KWin's pid or /proc/<pid>/environ cannot be read.
  2. When the pen's tabletToolIsRelative property exists and is true (or not a boolean), the call is refused before any event.
  3. parse_kwin_windows_stacked has a unit test with hand-written windows. It checks that hidden, other-activity, other-desktop, minimized and desktop windows are dropped, that a window of unknown activity stays as cover only (workspace: Some(-1)), and that stacking order beats listing order. It fails on the commit before the visibility fix. The effects, environment and device-settings refusals still have no test: this container's KWin cannot open the Overview, and the other two need KWin restarted with other settings. They are short reads next to the tested Show Desktop check.
  4. A "" current activity counts as unknown.
  5. Left for now, as you noted for GNOME too: the checks run right before the window list and mapping reads. A second pass inside perform would need the async D-Bus reads on the blocking thread.
  6. Nit: there is now a comment at the check site.

KWin and GNOME fixtures, fmt, clippy, tests, MCP safety, plugins and the Pi catalog pass on the dev desk.

@revuto-review revuto-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an auto review done by revuto.


Two findings, both in the KWin window check. Read-only review: nothing was built or run.

  1. The indicator overlay still counts as a covering window (open from the earlier review). With the default indicator on, a background action under its edge strips, pill or cursor sprite is refused as "stacked above", though the overlay is click-through. The KWin fixture runs with COMPUTER_USE_LINUX_INDICATOR=0, so it cannot see this.
  2. The new "minimized, hidden, or not on the desktop" message also fires when the target was only ambiguous. A pid target whose app has a popup open is the common case: the stacked list holds both windows, the filtered list_windows holds one, and the real error with the candidate ids is dropped.

What I checked and found sound:

  • The e93a454 filter changes: a target whose desktop or activity KWin did not report is refused, and hidden and other-activity windows are dropped.
  • The GNOME path through the Settings refactor: the same resets run, now after the device is gone.
  • No tool parameter, dependency or version changes.

Not verified here (no KWin source in the sandbox): the upstream claims about SyncWithMouse, KWIN_WAYLAND_EMULATE_TABLET, and KWin applying outputName to a live touchscreen.

Comment thread src/virtual_pen.rs
Comment on lines +573 to +577
let top = windows
.iter()
.rev()
.find(|window| window_rect(window).is_some_and(|rect| rect.contains(point)))
.expect("the target contains the point");

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an auto review done by revuto.


Still open at e93a454: the server's own indicator overlay counts as a covering window here, so background actions under it are refused.

On Plasma the indicator is a set of wlr-layer-shell surfaces on the overlay layer, click-through by an empty input region (src/bin/computer-use-linux-indicator/main.rs:503-511). KWin's stacking order holds layer-shell surfaces, which is why this check sees Plasma's panels. parse_kwin_windows_stacked drops only the desktop window and minimized, hidden or other-desktop windows, so the overlay's surfaces stay in the list.

While the overlay is up (from any tool call's indicator.action(..) until 8 s idle, IDLE_BEFORE_FADE), each output carries:

  • four 26 px edge strips (EDGE);
  • a 560×56 pill at top centre, 34 px down (PILL_SIZE, PILL_TOP), which sits over the toolbar or tab row of a maximized window;
  • the cursor sprite where it last rested.

A click, a drag end or a pan stroke under any of these fails with "window_id N (untitled) is stacked above…", though the pen would pass through. N is not in list_windows, so the agent cannot tell what blocks it. The first call in a session passes, because the snapshot is taken before self.indicator.action(action) (server.rs:11463); the same call a moment later is refused.

The KWin fixture sets COMPUTER_USE_LINUX_INDICATOR=0, so it never exercises this.

Two ways to fix it:

  • leave the overlay's windows out of the stack, matching the overlay process by the pid KWin reports for each window; or
  • take the indicator's capture hold around the window snapshot, so its surfaces are gone when the stack is read.

Comment thread src/server.rs
Comment on lines +11311 to +11316
if desktop == pen::Desktop::Kwin {
if let Ok(all) = crate::windowing::backends::kwin::list_windows().await {
if let Ok(window) = resolve_window_target(&all, &target) {
return fail(format!("Nothing was sent: window_id {} is minimized, hidden, or not on the desktop and activity the user sees.", window.window_id));
}
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an auto review done by revuto.


This fallback reports "minimized, hidden, or not on the desktop and activity the user sees" for every resolution failure, including one where the target is visible but ambiguous in the stacked list.

windows_stacked() keeps popups, panels and notifications, while kwin::list_windows() filters them out through element_window. A pid target goes through unique_window_match, so an app with a popup open has two matches in the stacked list and one in the filtered list. The common case is the context menu a background right click just opened. The sequence is then:

  1. The first resolve_window_target fails with "pid N matched multiple windows (ids)…".
  2. The retry against list_windows() succeeds on the main window.
  3. The call returns that the main window is minimized or hidden. It is neither.

The discarded error carried the candidate ids, including the popup's, which list_windows never shows and which the agent needs to target the menu.

Suggested fix: claim this only when the window resolved from the full list is absent from windows by window_id; otherwise return the original error.

@revuto-review revuto-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an auto review done by revuto.


The desktop/activity visibility finding is resolved. Two earlier correctness findings remain open in the KWin window check and target-resolution fallback.

Comment thread src/virtual_pen.rs
let top = windows
.iter()
.rev()
.find(|window| window_rect(window).is_some_and(|rect| rect.contains(point)))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an auto review done by revuto.


[P2] Still open: exclude the indicator's click-through surfaces from this check. The default indicator creates Layer::Overlay surfaces with an empty input region (src/bin/computer-use-linux-indicator/main.rs:503–511), but parse_kwin_windows_stacked does not exclude them. This rectangle-only lookup therefore refuses background actions under the 26 px output-edge strips, status pill, or cursor sprite while the indicator is visible, although those surfaces cannot receive input. The snapshot precedes indicator.action, so a subsequent call within the 8-second visibility interval can fail where the first succeeded. Exclude the indicator process's surfaces or hold the indicator hidden while reading the stack. The KWin fixture disables the indicator and misses this case.

Comment thread src/server.rs Outdated
if desktop == pen::Desktop::Kwin {
if let Ok(all) = crate::windowing::backends::kwin::list_windows().await {
if let Ok(window) = resolve_window_target(&all, &target) {
return fail(format!("Nothing was sent: window_id {} is minimized, hidden, or not on the desktop and activity the user sees.", window.window_id));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an auto review done by revuto.


[P2] Still open: preserve ambiguity errors when the resolved window is present in the visible stack. A PID target can match both a visible main window and its popup in windows_stacked, causing unique_window_match to return candidate IDs. list_windows filters out popups through element_window, so the retry succeeds on the main window and this branch incorrectly reports it as minimized or hidden, discarding the IDs needed to disambiguate. Only return this visibility explanation if the retry's window ID is absent from windows; otherwise return the original error.

- The indicator's overlay surfaces (layer-shell, empty input region) no
  longer count as cover: a second background call under its pill was
  refused. The fixture runs a server with the indicator on and clicks
  twice under the pill; the second click failed before.
- The minimized-or-hidden message is given only when the window resolved
  from the full list is absent from the visible stack, so an ambiguous
  target keeps its error and candidate ids.
- The focus sentence names the compositor, not GNOME.
@avifenesh

Copy link
Copy Markdown
Collaborator Author

revuto and Bugbot findings addressed in 6173193:

  • Indicator overlay: windows whose process is computer-use-linux-indicator are left out of the KWin cover list, since its layer-shell surfaces have an empty input region. The fixture now runs a server with the indicator on and clicks twice under its pill. On the previous commit the second click was refused ("window_id ... (untitled) is stacked above ..."); now both reach the window as pen presses.
  • Fallback message: "minimized, hidden, or not on the desktop" is returned only when the window resolved from the full list is missing from the visible stack. Otherwise the original error, with its candidate ids, comes back.
  • Bugbot: the focus sentence now says "The compositor gave window_id ...".
  • Unknown visibility and activities were already fixed in e93a454 and fbf1f55.

KWin and GNOME fixtures, fmt, clippy, tests, MCP safety, plugins and the Pi catalog pass on the dev desk.

@avifenesh
avifenesh merged commit 6173193 into main Nov 20, 2026
24 of 25 checks passed
@avifenesh
avifenesh deleted the feat/kwin-pen branch November 20, 2026 03:37
@avifenesh avifenesh mentioned this pull request Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant