🖱️ Guide Cursor

A cursor that hovers over your screen and talks. Point at anything, explain it out loud, and teach a task step by step — without ever taking the mouse, the focus, or a click away from the person using the machine.

---

What it is

Two small scripts:

• overlay.py — an always-on-top, fully transparent, click-through drawing layer. It draws a ring or a highlight box anywhere on screen and never touches the real mouse or the keyboard focus.
• guide.py — a lesson player. It speaks each step with edge-tts and moves the ring to match, from a plain JSON lesson file.

It exists so an assistant can teach a task on a real desktop — "click here, then this, and here is why" — with a voice and a pointer, while the human keeps working underneath.

How it works

lesson.json  ──►  guide.py  ──►  %TEMP%\guide_cursor_cmd.json  ──►  overlay.py
                     │                                                  │
                     └──►  edge-tts  ──►  ffplay (speech)               └──►  ring / box on screen

The two processes never talk directly. guide.py writes a one-line JSON command to a file, and overlay.py polls that file and redraws. Any program that can write a file can therefore drive the cursor.

Requirements

• Windows for the overlay (it uses Windows transparency and DPI APIs); the lesson player itself is portable
• Python 3.10+ with tkinter (bundled with python.org builds)
• edge-tts — pip install -r requirements.txt
• ffmpeg / ffplay on PATH for audio playback

Speech defaults to en-US-ChristopherNeural. Change it with --voice.

Install and run

pip install -r requirements.txt

pythonw overlay.py                     # start the overlay (no console window)

python guide.py --point 1280 800 --say "this is the file you need"
python guide.py --point 900 700 --label "Click here"
python guide.py examples/example_lesson.json
python guide.py --hide                 # clear the ring
python guide.py --quit                 # stop the overlay

Coordinates are screen pixels, measured from the top-left corner of the primary display.

Lesson format

{
  "title": "Find the thing you need",
  "steps": [
    {"say": "This is the toolbar.",
     "box": [240, 90, 2560, 140], "color": "#00b3ff", "label": "Toolbar", "hold": 1.0},
    {"say": "And the file you came for is right here.",
     "ring": [806, 1017], "r": 34, "color": "#ff00ff", "label": "Start here", "hold": 1.5},
    {"hide": true}
  ]
}

key	meaning
say	spoken with TTS before the hold
ring	[x, y] centre of the ring, in screen pixels
box	[x1, y1, x2, y2] rectangle to highlight instead
r	ring radius (default 44)
color	hex colour (default #ff2d55)
label	small caption drawn with the ring or box
hold	extra seconds to wait after the speech finishes
hide	clear the overlay and wait

Speech is cached per line in audio/, keyed by a hash of the text, so re-running a lesson is instant.

Pointing at real UI elements

Hard-coded pixel guesses break the moment a window moves. With cua-driver installed you can resolve real elements instead:

cua-driver call list_windows                                            # window bounds, in screen pixels
cua-driver call get_window_state '{"pid":27392,"window_id":1443794}'    # every element's label + frame
cua-driver call bring_to_front '{"pid":27392,"window_id":1443794}'      # raise the target window

cua_targets.py wraps that into one call:

python cua_targets.py --app "File Explorer" --label "README" --point

It prints the element's screen coordinates and, with --point, moves the ring onto it. Any automation tool that can report element frames will do; cua-driver is simply the one this was built against.

Notes from building it

• DPI awareness is mandatory. Without SetProcessDpiAwareness(2) before the window is created, the overlay is scaled by the display's DPI factor and every command lands that many times too far from the origin (1.5× on a 150% display). It is the first thing overlay.py does, and it must stay above mainloop().
• The ring is measured, not assumed. Commanded to (900, 700), a composited screenshot shows the ring centred at (897, 703) — within 3 px, tested at two positions.
• Verify with a composited grab. A layered, colour-keyed window is invisible to some capture paths (cua-driver's own get_desktop_state, for one) while perfectly visible to others. If a ring looks missing, check the capture method before the code.
• Keep it click-through. WS_EX_TRANSPARENT | WS_EX_NOACTIVATE | WS_EX_TOOLWINDOW mean no focus theft, no blocked clicks, and no taskbar entry.

Built with

Guide Cursor was built and tested on Hermes Agent, Nous Research's agent harness. An agent in that harness decided what to point at, narrated the steps, and used this very cursor to debug its own rendering while it was being written — which is where the DPI and capture-path notes above come from.

The scripts themselves are harness-agnostic plain Python: any assistant, script or person that can write a JSON command file can drive the cursor.

Licence

MIT — see LICENSE.

Built by Benjamin Tia, with the very cursor it draws pointing at the parts of itself that needed fixing.