Repository navigation
feat: background pointer input on KDE Plasma through the virtual pen - #494
Conversation
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 100bfab. Configure here.
| } | ||
| } | ||
| } | ||
|
|
There was a problem hiding this comment.
Focus message names GNOME on Plasma
Low Severity
A successful KWin pen or touch action that moves keyboard focus still reports that GNOME gave the window focus. pen_focus_sentence is shared by both desktops and hard-codes GNOME, so Plasma results tell the agent the wrong compositor took the user's typing.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 100bfab. Configure here.
|
Fresh-context review: last commit only (100bfab, KDE Plasma virtual pen on top of #493). KWin claims checked against KDE/kwin master and v6.3.0. Confirmed upstream: High
Medium
Low
|
There was a problem hiding this comment.
This is an auto review done by revuto.
Two findings in the KWin window check; both are in the new stacking path, not in the device or kcminputrc handling.
- The server's own indicator overlay blocks background actions on KDE. The KWin check treats every window above the target as covering the point, and the indicator's click-through layer-shell surfaces are such windows. With the indicator on (the default), points under them are refused. The KWin fixture runs with
COMPUTER_USE_LINUX_INDICATOR='0', so it never exercised this. - The target's visibility check fails open. A target whose desktop state reads as unknown, or that is on another Plasma Activity, passes as "on the desktop the user sees".
I read the code only; nothing was built or run, and the KWin behaviour for layer-shell windows and Activities is from KWin's documented scripting API, not from a live session.
| let top = windows | ||
| .iter() | ||
| .rev() | ||
| .find(|window| window_rect(window).is_some_and(|rect| rect.contains(point))) | ||
| .expect("the target contains the point"); | ||
| if top.window_id != target_id { | ||
| return Err(format!( | ||
| "window_id {} ({}) is stacked above window_id {target_id} at that point", | ||
| top.window_id, | ||
| top.title.as_deref().unwrap_or("untitled") | ||
| )); | ||
| } |
There was a problem hiding this comment.
This is an auto review done by revuto.
On KDE this counts the server's own indicator overlay as a covering window, so background actions under it are refused with "window_id N (untitled) is stacked above …".
computer-use-linux-indicator is on by default and draws on Layer::Overlay layer-shell surfaces (src/bin/computer-use-linux-indicator/main.rs:507). KWin lists those in workspace.stackingOrder like the panels and notifications this check is meant to see. parse_kwin_windows_stacked only drops desktop, minimized and other-desktop windows, so the overlay's surfaces stay in the list, above every normal window.
The surfaces have an empty input region (main.rs:503-505), so they never take the pen; the refusal is a false positive. While the overlay is up (8 s after any tool call, IDLE_BEFORE_FADE), the refused areas are:
- a 26 px strip along every edge of every output (
EDGE), e.g. the scrollbar of a maximized window; - the 560×56 status pill at top centre, y 34–90 (
PILL_SIZE,PILL_TOP), e.g. the tab strip or toolbar of a maximized window; - the 112×112 cursor sprite at the last focused click, scroll or drag point.
GNOME is not affected: the extension draws Shell actors, which are not in its window list. scripts/virtual_pen_kwin_test.py:71 sets COMPUTER_USE_LINUX_INDICATOR='0', so the fixture never saw this.
Suggested fix: leave the overlay's surfaces out of the stack (the script already reads each window's pid, which can be matched to the overlay process), or hide the overlay around the check as hold_for_capture does for screenshots. A fixture run with the indicator on and a click near an output edge would cover it.
| if flag(&window.desktop_window) == Some(true) | ||
| || flag(&window.minimized) == Some(true) | ||
| || flag(&window.on_current_desktop) == Some(false) | ||
| { | ||
| continue; | ||
| } |
There was a problem hiding this comment.
This is an auto review done by revuto.
This filter fails open for the target window in two cases, so the pen can be sent at a point where the target is not actually on screen.
- Unknown desktop state.
onCurrentDesktop()in the script returnsnullwhen it cannot tell, and onlySome(false)is dropped here. A target withnullpasses as visible. The existingvisible_element_windowrequires== Some(true)for the same decision (and has theunknown_or_other_desktop_visibility_never_dispatchestest). Keeping unknown windows is right for the windows that may cover the point, but the target itself should needSome(true). - Activities. The script reads neither
window.activitiesnorworkspace.currentActivity. A window on another Activity is hidden by KWin but is not minimized, can share the current desktop id, and stays in the stacking order. If it is the target and nothing listed above it contains the point, the tap goes to whatever the user actually sees there. The same gap makes hidden other-Activity windows count as covering, which only causes refusals.
Suggested fix: carry an onCurrentActivity flag from the script (empty activities, or it contains currentActivity), drop windows where it is false, and have check_point_on_visible_target refuse a target whose desktop or activity state is not known to be current.
KWin 6.3 and later give a tablet tool its own cursor, as mutter does, so the virtual pen and touchscreen now serve background click, drag and scroll on KDE Plasma Wayland too, as routes kwin-pen and kwin-touch. KWin maps a tablet with no setting to the active output, which follows the user's pointer, and a touchscreen to the internal panel or the first output. Before input the server sets the pen's mapToWorkspace and the touchscreen's outputName through org.kde.KWin.InputDevice and reads them back; once the devices are gone it removes what KWin wrote for them into kcminputrc with kwriteconfig6. The point is checked against KWin's stacking order through KWin scripting, panels, popups and notifications included; a scroll reads the output's rotation with kscreen-doctor. KWin's SyncWithMouse tablet option, which moves the user's pointer to the pen, leaves no route. scripts/virtual_pen_kwin_test.py runs KWin 6.6 from Ubuntu 26.04 in a container on a vkms output with its DRM backend (KWin's virtual backend reads no input devices), opt-in with COMPUTER_USE_LINUX_TEST_KWIN_SEAT=1.
- The window list leaves out hidden windows and windows on another activity; a target whose desktop or activity KWin did not report is refused, and a minimized or hidden target is named as such. - Show Desktop and full-screen effects (org.kde.kwin.Effects.activeEffects: Overview, Window View and the like) refuse the call. - KWIN_WAYLAND_EMULATE_TABLET in KWin's environment leaves no route. - SyncWithMouse is read as KConfig reads a boolean (kreadconfig6 --type bool answers in its exit status). - A calibration, input or output area or rotation other than the default on the device refuses the call before any event. - doctor offers kwin-touch only with kscreen-doctor installed. - The fixture adds Show Desktop and a minimized target.
|
Fresh-context review addressed in e93a454 (on main now that #493 landed at 3a0cd24):
KWin fixture, GNOME fixture, fmt, clippy, tests, MCP safety, plugins and the Pi catalog pass on the dev desk. |
100bfab to
e93a454
Compare
|
Fresh-context re-review: latest commit only (e93a454, "fix: review findings on the KWin pen"). KWin claims checked against KDE/kwin master and v6.3.0. Verified:
Earlier items 1, 3, 4, 9 and 10 are resolved. Item 6 is left as the author argued: both stroke ends are checked inside the window, so the remaining risk is a sub-pixel edge case. Item 8 is accepted as replied. Medium
Low
No High findings. |
- The KWIN_WAYLAND_EMULATE_TABLET check refuses when KWin's environment cannot be read, as the other checks refuse when KWin does not answer. - A pen KWin runs in relative mode refuses the call. - An empty current activity (no activity manager) counts as unknown. - A unit test pins the stacked list: hidden, other-activity, other-desktop, minimized and desktop windows are left out, a window of unknown activity stays as cover only, and stacking order wins over listing order.
|
Re-review addressed in fbf1f55:
KWin and GNOME fixtures, fmt, clippy, tests, MCP safety, plugins and the Pi catalog pass on the dev desk. |
There was a problem hiding this comment.
This is an auto review done by revuto.
Two findings, both in the KWin window check. Read-only review: nothing was built or run.
- The indicator overlay still counts as a covering window (open from the earlier review). With the default indicator on, a background action under its edge strips, pill or cursor sprite is refused as "stacked above", though the overlay is click-through. The KWin fixture runs with
COMPUTER_USE_LINUX_INDICATOR=0, so it cannot see this. - The new "minimized, hidden, or not on the desktop" message also fires when the target was only ambiguous. A
pidtarget whose app has a popup open is the common case: the stacked list holds both windows, the filteredlist_windowsholds one, and the real error with the candidate ids is dropped.
What I checked and found sound:
- The e93a454 filter changes: a target whose desktop or activity KWin did not report is refused, and hidden and other-activity windows are dropped.
- The GNOME path through the
Settingsrefactor: the same resets run, now after the device is gone. - No tool parameter, dependency or version changes.
Not verified here (no KWin source in the sandbox): the upstream claims about SyncWithMouse, KWIN_WAYLAND_EMULATE_TABLET, and KWin applying outputName to a live touchscreen.
| let top = windows | ||
| .iter() | ||
| .rev() | ||
| .find(|window| window_rect(window).is_some_and(|rect| rect.contains(point))) | ||
| .expect("the target contains the point"); |
There was a problem hiding this comment.
This is an auto review done by revuto.
Still open at e93a454: the server's own indicator overlay counts as a covering window here, so background actions under it are refused.
On Plasma the indicator is a set of wlr-layer-shell surfaces on the overlay layer, click-through by an empty input region (src/bin/computer-use-linux-indicator/main.rs:503-511). KWin's stacking order holds layer-shell surfaces, which is why this check sees Plasma's panels. parse_kwin_windows_stacked drops only the desktop window and minimized, hidden or other-desktop windows, so the overlay's surfaces stay in the list.
While the overlay is up (from any tool call's indicator.action(..) until 8 s idle, IDLE_BEFORE_FADE), each output carries:
- four 26 px edge strips (
EDGE); - a 560×56 pill at top centre, 34 px down (
PILL_SIZE,PILL_TOP), which sits over the toolbar or tab row of a maximized window; - the cursor sprite where it last rested.
A click, a drag end or a pan stroke under any of these fails with "window_id N (untitled) is stacked above…", though the pen would pass through. N is not in list_windows, so the agent cannot tell what blocks it. The first call in a session passes, because the snapshot is taken before self.indicator.action(action) (server.rs:11463); the same call a moment later is refused.
The KWin fixture sets COMPUTER_USE_LINUX_INDICATOR=0, so it never exercises this.
Two ways to fix it:
- leave the overlay's windows out of the stack, matching the overlay process by the
pidKWin reports for each window; or - take the indicator's capture hold around the window snapshot, so its surfaces are gone when the stack is read.
| if desktop == pen::Desktop::Kwin { | ||
| if let Ok(all) = crate::windowing::backends::kwin::list_windows().await { | ||
| if let Ok(window) = resolve_window_target(&all, &target) { | ||
| return fail(format!("Nothing was sent: window_id {} is minimized, hidden, or not on the desktop and activity the user sees.", window.window_id)); | ||
| } | ||
| } |
There was a problem hiding this comment.
This is an auto review done by revuto.
This fallback reports "minimized, hidden, or not on the desktop and activity the user sees" for every resolution failure, including one where the target is visible but ambiguous in the stacked list.
windows_stacked() keeps popups, panels and notifications, while kwin::list_windows() filters them out through element_window. A pid target goes through unique_window_match, so an app with a popup open has two matches in the stacked list and one in the filtered list. The common case is the context menu a background right click just opened. The sequence is then:
- The first
resolve_window_targetfails with "pid N matched multiple windows (ids)…". - The retry against
list_windows()succeeds on the main window. - The call returns that the main window is minimized or hidden. It is neither.
The discarded error carried the candidate ids, including the popup's, which list_windows never shows and which the agent needs to target the menu.
Suggested fix: claim this only when the window resolved from the full list is absent from windows by window_id; otherwise return the original error.
There was a problem hiding this comment.
This is an auto review done by revuto.
The desktop/activity visibility finding is resolved. Two earlier correctness findings remain open in the KWin window check and target-resolution fallback.
| let top = windows | ||
| .iter() | ||
| .rev() | ||
| .find(|window| window_rect(window).is_some_and(|rect| rect.contains(point))) |
There was a problem hiding this comment.
This is an auto review done by revuto.
[P2] Still open: exclude the indicator's click-through surfaces from this check. The default indicator creates Layer::Overlay surfaces with an empty input region (src/bin/computer-use-linux-indicator/main.rs:503–511), but parse_kwin_windows_stacked does not exclude them. This rectangle-only lookup therefore refuses background actions under the 26 px output-edge strips, status pill, or cursor sprite while the indicator is visible, although those surfaces cannot receive input. The snapshot precedes indicator.action, so a subsequent call within the 8-second visibility interval can fail where the first succeeded. Exclude the indicator process's surfaces or hold the indicator hidden while reading the stack. The KWin fixture disables the indicator and misses this case.
| if desktop == pen::Desktop::Kwin { | ||
| if let Ok(all) = crate::windowing::backends::kwin::list_windows().await { | ||
| if let Ok(window) = resolve_window_target(&all, &target) { | ||
| return fail(format!("Nothing was sent: window_id {} is minimized, hidden, or not on the desktop and activity the user sees.", window.window_id)); |
There was a problem hiding this comment.
This is an auto review done by revuto.
[P2] Still open: preserve ambiguity errors when the resolved window is present in the visible stack. A PID target can match both a visible main window and its popup in windows_stacked, causing unique_window_match to return candidate IDs. list_windows filters out popups through element_window, so the retry succeeds on the main window and this branch incorrectly reports it as minimized or hidden, discarding the IDs needed to disambiguate. Only return this visibility explanation if the retry's window ID is absent from windows; otherwise return the original error.
- The indicator's overlay surfaces (layer-shell, empty input region) no longer count as cover: a second background call under its pill was refused. The fixture runs a server with the indicator on and clicks twice under the pill; the second click failed before. - The minimized-or-hidden message is given only when the window resolved from the full list is absent from the visible stack, so an ambiguous target keeps its error and candidate ids. - The focus sentence names the compositor, not GNOME.
|
revuto and Bugbot findings addressed in 6173193:
KWin and GNOME fixtures, fmt, clippy, tests, MCP safety, plugins and the Pi catalog pass on the dev desk. |


Follow-up to #489 and #480: the KDE Plasma half, and the answer on hover.
What changes
KDE Plasma Wayland with KWin 6.3 or later gets the virtual pen and touchscreen for
click,dragandscrollwithdelivery: "background", as routeskwin-penandkwin-touch. The user's mouse pointer stays where it is.kwin_wayland --versionis read once.applyScreenToDevice). Before input the server sets the pen'smapToWorkspace(KWin then maps it acrossworkspace()->geometry(),tabletToolPosition) and the touchscreen'soutputName(the output under the point) throughorg.kde.KWin.InputDeviceon the devices' own/org/kde/KWin/InputDevice/eventN. It reads each value back before any event.kcminputrc([Libinput][25461][28773][computer-use-linux virtual pen]and the touch group). Setting them back to defaults leavesMapToWorkspace=falseand a staleOutputUuid, so once the devices are gone the server deletes the keys withkwriteconfig6 --delete --notify. The settings guard is the last field of each device, so it runs after the device is destroyed: KWin writes a setting back when it changes on a live device. No other device's group is touched.workspace.stackingOrder, each window'shiddenflag and whether it is on the current activity. The point must be on the target, and the target must be on the desktop and activity the user sees, not minimized or hidden. No window above it may cover the point: normal windows, docks, popups and notifications all count. A target whose desktop or activity KWin does not report is refused.kscreen-doctor -jand refuses a rotated output, or one whose rotation it cannot read, as on GNOME.[Tablet] SyncWithMouseoption (92b48ae5df) moves the user's pointer to where the pen was, so with it on there is no route. The same holds withoutkreadconfig6/kwriteconfig6. The lock screen is read fromorg.freedesktop.ScreenSaver.showingDesktop) or runs a full-screen effect (org.kde.kwin.Effects.activeEffects: Overview, Window View and the like). It is also refused when the device carries a calibration, input or output area or rotation other than its default.KWIN_WAYLAND_EMULATE_TABLETin KWin's environment leaves no route, since it sends pen input over apps without tablet support through the user's pointer.Hover (item 3 of the follow-up)
No hover tool. The server has no pointer-move tool on any route, so a background-only hover would be a new tool surface (schema, Pi catalog, plugins, safety classification) for an effect that reaches tablet-aware clients only. #493 also removed the 8 s lingering hover: a hovering pen makes libinput drop touches on the touchscreen it paired with, which can be the user's.
AWSL
agent-workspace-linux's backends cannot exercise uinput tablets. Its KWin runs
--virtual, and KWin creates a libinput backend only for its DRM backend (DrmBackend::createInputBackend); the other backends return none. GNOME runs--headless, which reads no input devices. Hyprland and COSMIC run nested or headless too. So the KWin test runs KWin's DRM backend itself. KWin falls back to its Noop session (a plainopen()) when no logind is reachable, so it runs as root in a privileged container on a vkms output, with no logind or VT. An AWSL change would have meant a DRM-backed workspace mode, which is not a small change.Tests
scripts/virtual_pen_kwin_test.py(tests/virtual_pen.rs, opt-in withCOMPUTER_USE_LINUX_TEST_KWIN_SEAT=1) runs KWin 6.6.7 from Ubuntu 26.04 (scripts/virtual_pen_kwin.Dockerfile) with--drmon vkms in a privileged container. It uses--network host, because libinput hears about new devices through udev's netlink group, which a separate network namespace never receives. A GTK 4 window logs its events. Plasma's screenshots go through the desktop portal, which asks the user, so a stand-inorg.gnome.Shell.Screenshotanswers the server's first screenshot route with a blank image of the output's size.The checks:
doctorofferskwin-pen/kwin-touch.kcminputrcholdsMapToWorkspace=truefor the pen while it runs.computer-use-linux virtualgroup inkcminputrc.SyncWithMouse=true,doctorreports no route and names the option.mapToWorkspacefrom KWin's active-output default; it checks the setting KWin persisted.Results on the dev desk:
The GNOME seat fixture now reloads udev's rules after writing its own, since the container's bind mount of
/run/udevleft udevd missing the new rule. It also unloads vkms with retries. Both fixtures pass back to back.fmt, clippy
-D warnings,cargo test --no-fail-fast,mcp_safety_check.py,check_plugins.shand the Pi catalog--checkpass on the dev desk. The drag anddeliveryschema descriptions name KDE Plasma, and the Pi catalog is regenerated.