AgentCursor

Local, free, human-like cursor for AI agents: any Mac app or browser tab, over MCP.

AgentCursor gives you (and any coding agent or automation script) a real browser driven with visible, convincingly human cursor movement and timing.

Use it as:

• A powerful MCP tool for Claude, Cursor, Grok, custom agents, etc.
• A realistic E2E / acceptance testing tool that works on actual production sites (human paths + timing are more resilient than robotic Playwright clicks).
• A workflow automation engine for complex multi-step processes (logins, form flows, data entry, admin tasks) with natural hover, move, type, and scroll behavior.
• A debugging / demo automation tool (the cursor is visible so you can watch exactly what the automation did).

All local. All free. MIT licensed. No cloud, no paywalled stealth.

The major browser automation MCPs often make realistic movement a cloud-only feature. AgentCursor brings the realistic cursor to your local machine for agents and traditional testing/automation use cases.

Status: phase 1 (Chrome extension) and phase 2 (macOS OS-cursor for genuinely trusted events) are both implemented. See docs/DESIGN.md.

Quick start

git clone https://github.com/kumard3/agentcursor.git
cd agentcursor && pnpm install && pnpm build
node dist/index.js setup

setup starts the local service, lists the AI apps it finds on your machine (Claude Code, Cursor, VS Code, Codex, Windsurf, Claude Desktop, Gemini CLI) and opens a setup page at http://127.0.0.1:8931. From there you connect each app with one click, grant the two macOS permissions, load the optional Chrome extension, and press Test the cursor. Prefer the terminal? node dist/index.js setup --all connects every detected app (or --client=cursor,codex).

Then ask your AI app something like: "Use agentcursor: open Notes, create a new note and write a 3 item shopping list."

How it fits together: every AI app launches agentcursor as an ordinary stdio MCP server. The first launch starts one shared background service (MCP over HTTP at 127.0.0.1:8931/mcp, the extension bridge on 8930) and every later launch reuses it, so several apps can use AgentCursor at once without fighting over a port. After a rebuild, the next launch replaces an older running service; an idle service exits after 10 minutes. The service only accepts requests from this machine and rejects other websites (Host and Origin checks).

Changelog (key updates)

• Unreleased: E2E testing. AgentCursor.launch() starts a private Chrome (throwaway profile, its own copy of the extension on a free port) with its own visible cursor, so tests never touch your mouse, your browser or the MCP server on 8930, and several can run side by side. Works headed or headless: true. Computer use reaches the SDK too: Desktop.open("Notes") drives any Mac app with the real cursor and text-first reads. New agent CLI: every tool is a shell command (agentcursor read_page, agentcursor desktop_click --text Save) sharing one background service, so no tool schemas ever enter the model's context. Refs now stay with an element across reads, so read_page --changes and desktop_read --changes return only what changed: a repeat read of an unchanged window costs ~8 estimated tokens instead of ~397. Page reads got ~43% smaller, and the stdio tool list dropped from about 3,178 to 2,735 estimated tokens. New expect() with auto-retrying matchers (toBeVisible, toBeHidden, toHaveText, toContainText, toHaveCount, toHaveURL, .not). navigate now waits for the page to load, commands wait briefly for the content script after a navigation, isVisible()/count() answer immediately instead of waiting 5s, and headless screenshots no longer return the previous frame.

• 0.4.0: Desktop control for any Mac app: desktop_read returns the window as compact text with [dN] refs from the macOS accessibility tree (a small Swift helper), then desktop_click / desktop_type / desktop_key / desktop_scroll drive the real cursor and keyboard with the same persona-based human motion. desktop_screenshot is a cropped, downscaled fallback. Chromium and Electron apps get their accessibility tree switched on automatically. New onboarding: agentcursor setup plus a local setup page that connects seven AI apps, checks permissions and tests the cursor. All AI apps now share one background service (stdio launches proxy to it), which removes the port 8930 already in use failure. The extension WebSocket now rejects connections from web pages.

• 0.3.0: Programmatic SDK — import { AgentCursor } from "agentcursor" with a Playwright-shaped locator API (getByRole/getByText/getByLabel/getByPlaceholder/getByTestId/css + chaining + filter/nth + click/type/fill/hover/dragTo/press/scrollIntoView + boundingBox/isVisible/count/waitFor), every action driven by the human cursor. connect() and os() lifecycles. Library entry split from the MCP bin so importing the package no longer boots a server; built with tsup (ships .d.ts). Locator resolution uses @testing-library/dom in the content script.

• 0.2.9: Active-tab resolution no longer requires Chrome to be the OS-focused window — it falls back to the active tab in any window, then any open http(s) tab. Fixes No active tab found when an agent drives the browser while you're in your editor/terminal (the normal case).

• 0.2.8: Interaction: press_key — press Enter / Escape / Tab / arrows / Home / End / etc. on the focused element, content or stealth (trusted CDP key event). Rounds out the Comet-style action set: Navigation, Identification, Interaction.

• 0.2.7: Identification tools — find (locate elements by visible text / accessible name, shadow-DOM aware) and click_text (find the best text match, then human-move + click, content or stealth, with a re-read retry). Target by what the element says, not by ref or pixel coords.

• 0.2.6: Stealth typing inserts the whole string in one Input.insertText call. Per-character insertion landed at a reset caret on controlled editors (X's Draft.js) and typed text backward.

• 0.2.5: Content-script drag reports the held button (buttons mask) during the move, matching the stealth and OS drivers, so JS drag handlers see a real drag. Added a Known Limitations section.

• 0.2.4: read_page resolves multi-ID aria-labelledby names (it was passing the whole space-separated list to getElementById as one ID, so those labels came back empty); now shadow-DOM-tree-scope aware.

• 0.2.3: drag now performs a real drag (press at the start, move with the button held, release at the end) in the stealth (chrome.debugger) and OS-cursor drivers — previously a move-then-click. pnpm smoke asserts the screenshot image content.

• 0.2.2: drag tool. screenshot now returns a viewport-scaled image (1 image pixel = 1 click coordinate) for a vision loop — see the page, then click/move_to by x/y. Stealth (chrome.debugger) moves also animate the visible overlay, so the cursor stays on screen. Ships as a Claude Code plugin. Internal snapshot refresh resolves refs past the 60th element. pnpm build no longer mutates version files (use pnpm reload for the extension dev loop).

• 0.2.0: Added screenshot, hover, status MCP tools. Deep shadow DOM traversal in read_page / snapshot (critical for X.com, Reddit, modern SPAs). Library re-exports for programmatic use. Repositioned as general local automation/testing/workflow tool over MCP. Version bumps and packaging polish.

• 0.1.0: Initial MCP server, human path engine, extension bridge, OS cursor driver, basic tools (read_page, click, type, etc.).

How it works

Three layers, with one shared wire contract (src/protocol):

coding agent ──MCP/stdio──▶ MCP server ──localhost WebSocket──▶ Chrome extension ──▶ your real tab
                            (src/server)                        (extension/)
                                 │
                                 ▼
                          human-path engine (src/path-engine)
            from + to → timed cursor samples with overshoot, log-normal
            velocity, jitter, off-center landing, dwell — fresh every call

The MCP server generates the cursor sample stream; the extension is a thin replayer. The same stream works for the content-script driver, the chrome.debugger stealth driver, and (phase 2) the OS cursor — they all implement one BrowserDriver interface.

Why human-like movement is hard

Modern detectors (DataDome, Castle, reCAPTCHA v3, PerimeterX) flag overly smooth Bézier paths, constant velocity, dead-center clicks, zero dwell, teleporting jumps, and replayed identical paths. The engine addresses each:

• Fitts's law sets per-move duration from distance and target size.
• Asymmetric, eased velocity — not a symmetric min-jerk bell.
• Overshoot-and-correct on long moves.
• Sub-pixel Gaussian jitter, zero at the endpoints.
• Off-center landing inside the target.
• Right-skewed dwell before the press.
• Per-call entropy — paths are never cached or replayed.

Realism is necessary but not sufficient: content-script events are isTrusted=false, and chrome.debugger still leaks CDP tells. The real evasion endgame is the phase-2 OS cursor (genuine, trusted OS events).

Install

git clone https://github.com/kumard3/agentcursor.git
cd agentcursor
pnpm install
pnpm build      # builds dist/index.js + extension/dist/*

Install as a Claude Code plugin (one step)

AgentCursor ships as a Claude Code plugin that registers the MCP server for you:

claude plugin marketplace add kumard3/agentcursor
claude plugin install agentcursor

That registers the agentcursor MCP server automatically (no manual claude mcp add). You still load the extension once (step 1 below) if you want browser tabs. If you previously registered it by hand, remove that to avoid duplicate tools: claude mcp remove agentcursor.

1. Load the extension

1. Open chrome://extensions, enable Developer mode.
2. Load unpacked → select the extension/ folder.
3. Keep a normal http(s) tab open and focused (not chrome:// or the Web Store — content scripts can't run there).

2. Connect via MCP (agents, Cursor, Claude, custom tools, etc.)

The easy way is node dist/index.js setup (see Quick start). To do it by hand:

Claude Code:

claude mcp add --scope user agentcursor -- node /absolute/path/to/agentcursor/dist/index.js

Cursor, Windsurf, or any MCP-capable coding environment:

Add to your MCP servers config (exact format depends on the host):

{
  "mcpServers": {
    "agentcursor": {
      "command": "node",
      "args": ["/absolute/path/to/agentcursor/dist/index.js"]
    }
  }
}

Any other MCP client (including future Grok harnesses, custom agents, test runners that speak MCP) — just point it at the stdio server the same way.

HTTP-capable clients can skip the stdio launcher and point straight at http://127.0.0.1:8931/mcp while the service runs (node dist/index.js serve keeps it in the foreground).

The service exposes the WebSocket bridge on ws://127.0.0.1:8930 (override with AGENTCURSOR_WS_PORT; the HTTP port defaults to that plus one, or set AGENTCURSOR_HTTP_PORT). The extension auto-reconnects. AGENTCURSOR_TOOLS=desktop or =browser registers only one tool family, which keeps unused tool definitions out of your context.

3. Programmatic SDK (import { AgentCursor })

Drive the human cursor from your own Node/TS code with a Playwright-shaped locator API — no MCP client needed. Same engine, same stealth; every action moves a real cursor.

import { AgentCursor } from "agentcursor";

// Attach to a running Chrome that has the extension loaded.
// Pass { stealth: true } for trusted CDP events; use AgentCursor.os() for the nut-js OS cursor.
const ac = await AgentCursor.connect();

await ac.navigate("https://example.com");
await ac.getByRole("button", { name: "Buy now" }).click();
await ac.getByLabel("Email").fill("a@b.com");
await ac.getByText("Submit").click();
await ac.getByLabel("Email").press("Enter");

// chaining + filtering, just like Playwright
await ac.locator(".row").filter({ hasText: "Pro" }).nth(0).getByText("Edit").click();

if (await ac.getByTestId("checkout").isVisible()) {
  await ac.getByTestId("checkout").click();
}

await ac.screenshot({ path: "out.png" });
await ac.close();

Lifecycles

• AgentCursor.launch({ headless?, userDataDir?, args?, executablePath?, seed?, stealth? }): start a browser with its own cursor. executablePath picks the binary (Chrome for Testing, Brave, Edge, BrowserOS); userDataDir drives a real, logged-in profile with no copy and leaves it intact; headless runs it in the background. See Attach any browser and E2E tests.
• AgentCursor.connect({ port?, stealth?, timeoutMs? }): attach to a running Chrome with the extension loaded (works with your real, logged-in profile).
• AgentCursor.os({ stealth?, ... }): same locator API, but the real OS cursor is moved via nut-js (genuinely trusted events); page sensing still goes through the extension.

Run JS in the page: ac.evaluate(fn, ...args), Playwright-shaped, runs in the page realm via CDP (page cookies/session, awaits promises, ignores the page CSP):

const title = await ac.evaluate(() => document.title);
const res = await ac.evaluate(async (id) => {
  const r = await fetch(`/api/item/${id}`, { method: "POST", credentials: "include" });
  return { status: r.status, body: await r.text() };
}, 42);

Locators (lazy, chainable, Playwright-shaped)

• Find: locator(css), getByRole(role, { name }), getByText, getByLabel, getByPlaceholder, getByTestId.
• Refine: .filter({ hasText }), .nth(i), .first(), .last(), and chaining (a.locator(b)).
• Act: .click(), .dblclick(), .hover(), .type(), .fill(), .press(key), .dragTo(other), .scrollIntoView().
• Query: .boundingBox(), .textContent(), .isVisible(), .count(), .waitFor({ state }).

Each action resolves the locator in the page (role/label/etc. via @testing-library/dom, css/text via the DOM), then drives the human-path engine to the element. ac.actions exposes the lower-level ActionService (move by coords, find, clickText, scroll) as an escape hatch.

A full runnable example is in examples/sdk-quickstart.mjs; pnpm smoke:sdk runs the end-to-end pipeline against a simulated browser.

Note: in-page (stealth: false) events are isTrusted=false. Use stealth: true (CDP) or AgentCursor.os() when you need trusted events.

4. CLI for coding agents (agentcursor <command>)

Every MCP tool is also a shell command. One background service holds the page, the [refs] and the frontmost app, so commands chain like a session, and an agent that uses the CLI loads no tool schemas at all.

agentcursor help                        # one line per command (agentcursor tools for full descriptions)

# browser
agentcursor navigate https://example.com
agentcursor read_page --includeText false
agentcursor click_text "Buy now"
agentcursor read_page --changes         # only what changed since your last read
agentcursor find Submit                 # a handful of tokens instead of a whole page
agentcursor screenshot                  # writes a file, prints the path

# any Mac app (computer use)
agentcursor desktop_open Notes
agentcursor desktop_read --max 40       # the window as text, not pixels
agentcursor desktop_click --text "New Note"
agentcursor desktop_type "hello" --submit

Positionals fill a command's parameters in the order agentcursor help lists them, and anything can be given as --flag value. Dashes work too (click-text). AGENTCURSOR_OUT sets where screenshots are written.

Reading only the changes. A [ref] stays with the same element for the life of the page (or until you switch apps), so a ref you learned earlier keeps working and reads can be compared. Pass --changes to read_page or desktop_read and you get the added and removed lines plus a count of the ones that merely moved:

- [e5] text "Email" <input> @293,300
+ [e5] text "Email" value="a@b.com" <input> @293,300
(5 moved, 2 unchanged, 10 total)

Measured on a Finder window: a repeat read costs 29 chars (~8 estimated tokens) against 1390 (~397) for the full read. The first read, or a diff that would be bigger than the full read, returns the full read instead.

5. Computer use from the SDK (Desktop)

import { Desktop } from "agentcursor";

const d = await Desktop.open("TextEdit");     // seed: 7 for reproducible motion
console.log(await d.read());                  // window as compact text with [dN] refs
await d.type("Dear team,", { clear: true });
await d.click("Save");                        // a [dN] ref or any visible label
await d.key("cmd+s");
if (await d.waitForText("Saved")) console.log("done");

find() returns only matching elements (tens of tokens), view() gives the structured elements, and screenshot({ path }) is the fallback when the text is not enough. macOS only, and it needs Accessibility permission for whichever app starts it.

Background runs, and a cursor per session. By default computer use moves the one real pointer, which takes your machine over. background: true posts input straight to the target process instead: your pointer never moves, the app is never raised, and every Desktop keeps a cursor of its own, so runs happen while you work and several can run at once.

const alice = await Desktop.open("Notes", { background: true, showCursor: { color: "#4ade80", label: "alice" } });
const bob = await Desktop.open("Reminders", { background: true, showCursor: { color: "#60a5fa", label: "bob" } });
await Promise.all([alice.type("from alice"), bob.type("from bob")]);

showCursor is off unless you ask for it: it draws a click-through cursor above every window, coloured and named, so a background run is watchable. For MCP and the CLI, set AGENTCURSOR_BACKGROUND=1 and AGENTCURSOR_SHOW_CURSOR=1.

Measured on a background TextEdit while the terminal stayed frontmost: the text landed, the frontmost app did not change, and the pointer sat at 471,628 before and after.

The limit worth knowing: this works for native (AppKit) apps. Chromium and Electron apps ignore process-posted events, measured on Chrome, which ignored them even while frontmost, so it covers Notes, Mail, Finder, TextEdit and friends but not Chrome, Arc, VS Code, Slack or Discord. For browsers use AgentCursor.launch({ headless: true }), which is fully background anyway and draws its cursor in the page.

Driving a browser with computer use instead of the DOM. A page can also be read and clicked as an app, with no extension and no DOM: the clicks are real OS clicks, so the page sees isTrusted=true. Chrome only builds its accessibility tree for a screen reader, so ask for it at launch:

const ac = await AgentCursor.launch({ accessibility: true });   // --force-renderer-accessibility
await ac.goto("http://localhost:3000");

const d = await Desktop.open();
console.log(await d.find("Menu"));     // [d23] button "Menu" @87,481
await d.click("Menu");                  // real cursor, trusted click

Without that option a desktop read of Chrome sees the toolbar and no page content (16 elements against 61 on the same window, measured). For your own Chrome, start it with --force-renderer-accessibility.

Which path to pick:

	Extension (DOM)	Computer use (accessibility tree)
Events	synthetic, or trusted with stealth: true	always real OS input
Reads	~112 est tokens for a page	~609 for the same window, browser chrome included
Runs headless / in parallel	yes	no: needs a visible, frontmost window and the real mouse
Works outside the browser	no	any Mac app, including Electron

find costs about 8 tokens on either path, and --changes works for desktop reads too.

Attach any browser (real profile, in the background)

AgentCursor drives any Chromium browser (Chrome, Chrome for Testing, Brave, Edge, BrowserOS, and mostly Arc) using your real, logged-in profile, in the background (it dispatches synthetic or CDP events, so the window need not be focused and your physical mouse is never touched). This is "computer use, but token-cheap and headless": read_page / find / evaluate cost tens to a few hundred tokens each versus a screenshot vision loop. os() is the only mode that runs in the foreground (it moves the real OS cursor).

Three ways to attach:

Mode	Browser	Profile	Background	Setup
Extension (connect() / MCP, default)	any Chromium	your real one	yes	Load unpacked once
agentcursor launch / AgentCursor.launch({ userDataDir })	any Chromium binary	real (no copy) or throwaway	yes (--headless)	one command
os()	frontmost	real	no (moves your mouse)	Accessibility grant

A. Extension mode (recommended for agents). Load extension/ unpacked into the browser you want (its real profile), point your MCP client at the stdio server, and drive it. See Load the extension and Connect via MCP. Nothing is copied; the browser you already use is the one being driven.

B. agentcursor launch (one command, real profile, no manual extension load). Starts a browser wired to the running service and holds it open until Ctrl-C:

# real, logged-in profile, in the background:
agentcursor launch \
  --user-data-dir "$HOME/Library/Application Support/Google/Chrome" \
  --chrome "/Applications/Google Chrome for Testing.app/Contents/MacOS/Google Chrome for Testing" \
  --headless

# throwaway profile, visible window (the default):
agentcursor launch

Flags: --user-data-dir DIR (real profile, left intact; omit for a throwaway copy), --chrome PATH (any Chromium binary; also AGENTCURSOR_CHROME), --headless, --port (WS port, default 8930). Then drive it with the normal tools (agentcursor read_page, click_text, evaluate, ...).

Caveats:

• Profile lock: with --user-data-dir, the browser must be fully quit first. Chromium will not share a running profile's data dir.
• Arc: Chromium underneath, but it wraps its own launch, so --user-data-dir + the CDP pipe is flaky. For Arc, prefer extension mode (A).
• Same-origin fetch inside evaluate carries that profile's cookies, so authenticated requests work against the logged-in session.

Tools (MCP)

Tool	What it does
read_page	Interactive elements with stable [ref] handles, roles, rects, visible text. Call this first in almost every test or workflow.
find	Identification: locate elements by their visible text / accessible name (shadow-DOM aware). Returns ranked [ref] matches — target by what the element says, no ref needed.
click_text	Identification + interaction in one step: find the best text match and human-move + click it (re-reads if needed). Supports nth, double, stealth.
move_to	Human-like path to a [ref] or x/y (no click).
click	Full human move + click (supports button, double, stealth mode for trusted events).
hover	Human approach + hover events (mouseover/mouseenter). Critical for dropdowns, tooltips, nav, and realistic workflows.
drag	Human path drag from ref/coords to target while holding button (sliders, reorder, canvas).
type	Human-timed keystrokes (auto human-clicks ref to focus if provided).
press_key	Press a single key (Enter, Escape, Tab, arrows, Home/End, PageUp/Down, Space, or a character) on the focused element. content or stealth.
scroll	Eased, human-stepped scrolling.
screenshot	Capture the visible tab as an image, scaled so 1 image pixel = 1 click coordinate — see the page, then click/move_to by x/y (the vision loop). Also for visual assertions and agent grounding.
navigate	Load a URL in the active tab.
get_url	Current tab URL.
evaluate	Run a JS function in the page (CDP Runtime.evaluate): page realm, uses the page's own cookies/session, awaits promises, ignores the page CSP. For reads and requests with no button (() => document.title, async () => (await fetch('/api/x', { method: 'POST', credentials: 'include' })).status). Returns the JSON result. Shows the debugger banner while it runs.
wait_for	Wait for element ref or visible text (up to timeout). Use for resilient testing flows.
status	Health / connection status, driver, active URL, port. Great for CI, long-running workflows, and monitoring.

Any driving action accepts stealth: true to deliver trusted events through the chrome.debugger driver (this shows Chrome's "debugging this browser" banner).

Desktop tools (any Mac app)

No extension needed. Grant Accessibility (and Screen Recording, only for screenshots) to the app your AI runs in; the setup page has buttons for both.

Tool	What it does
desktop_apps	Running apps, frontmost marked.
desktop_open	Open or switch to an app by name and bring it to the front.
desktop_read	The window as compact text: buttons, fields, links, menus, list items and visible text, each with a [dN] ref and center point. find returns only the best matches for a label.
desktop_click	Human path + click on a [dN] ref, a visible label (text), or x/y. Double and right click supported.
desktop_move	Move there without clicking (hover menus, tooltips).
desktop_type	Persona-timed typing into the focused field, or click a field first (ref, into label, x/y). clear replaces, submit presses Enter.
desktop_key	Keys and shortcuts: enter, esc, tab, cmd+s, cmd+shift+t.
desktop_scroll	Scroll by pixels over a target or where the cursor is.
desktop_screenshot	One window (or the area around a ref), downscaled JPEG, with the formula to turn image pixels into screen x/y. For canvases and custom-drawn UI.

Why text first. Measured on a MacBook (text tokens estimated at 3.5 characters per token; image tokens from Anthropic's width x height / 750):

App	desktop_read (whole window)	desktop_read find / click by label	Screenshot at 1024px wide
Finder	~454	~9	662
System Settings	~543	~8	1181
Arc	~705	~9	852

The bigger saving is the loop: clicking by ref or label needs no screenshot to find coordinates and none to check where the click landed.

E2E tests: a Playwright alternative with a visible cursor

AgentCursor.launch() opens its own Chrome with its own cursor. Your mouse stays yours, your normal browser and the MCP agent's cursor keep working, and every launch is independent, so two launches give you two cursors (two users in a chat app, a buyer and a seller). Use any runner; this is plain node:test:

import { test } from "node:test";
import { AgentCursor, expect } from "agentcursor";

test("checkout", async () => {
  const ac = await AgentCursor.launch({ seed: 7 }); // headless: true for CI
  try {
    await ac.goto("http://localhost:3000");
    await ac.getByRole("button", { name: "Buy now" }).click();
    await ac.getByLabel("Email").fill("agent@cursor.dev");
    await ac.getByRole("button", { name: "Submit" }).click();
    await expect(ac.getByText("Order placed")).toBeVisible();
    await expect(ac).toHaveURL(/\/thanks$/);
  } finally {
    await ac.close();
  }
});

test("two cursors at once", async () => {
  const [alice, bob] = await Promise.all([
    AgentCursor.launch({ args: ["--window-position=0,0", "--window-size=760,900"] }),
    AgentCursor.launch({ args: ["--window-position=780,0", "--window-size=760,900"] }),
  ]);
  // alice and bob act in parallel, each with a visible cursor in its own window
  await Promise.all([alice.close(), bob.close()]);
});

• expect(locator): toBeVisible, toBeHidden, toHaveText (string or RegExp), toContainText, toHaveCount, and expect(ac).toHaveURL. Each retries until it passes or { timeout } (default 5s) runs out; .not inverts.
• Chrome is found automatically (or executablePath / AGENTCURSOR_CHROME). The extension is loaded over --remote-debugging-pipe, so stable Chrome works.
• Same seed means the same motion and typing, so runs (and recorded demos) are reproducible.
• Not there yet: multiple tabs per launch, network mocking, traces, and non-Chromium browsers. Default events are isTrusted=false; pass stealth: true for trusted CDP input.

pnpm e2e runs test/e2e/showcase.e2e.mjs (HEADLESS=1 pnpm e2e for CI).

Using as a Testing & Workflow Automation Tool

AgentCursor is not only for agents — it's a practical local browser automation primitive you can use directly in tests and scripts via MCP or by importing the core.

Why it shines for testing/automation on real sites:

• Human cursor paths + dwell + jitter + off-center clicks make interactions look like a real person (useful when sites have light behavioral signals).
• The visible cursor + overlay makes it excellent for demo videos, manual review of automation, and debugging failing flows.
• screenshot + read_page + wait_for + hover give you the primitives for visual + functional checks.
• Works against your real Chrome profile (cookies, extensions, logins) — perfect for realistic E2E that headless tools struggle with.

Example flow an agent or a test script might do:

read_page
hover "nav-menu"
click "Products"
wait_for text:"Featured"
screenshot
type {ref: "search", text: "laptop"}
click "search button"
...

Direct / programmatic use (API style): The core ActionService, path engine, and drivers are designed to be importable. See "Programmatic Use" below.

Example: Using with Claude Code to post on X.com / Reddit

With the MCP integration, you can tell Claude Code (or Cursor) to use agentcursor for realistic posting/automation on real sites:

1. Have a logged-in tab open on x.com (or reddit.com).
2. Start the server (ideally with OS driver on mac for best results).
3. In Claude: "Add agentcursor MCP if not present, then use the tools to navigate to x.com if needed, read the page, hover and click the compose area, type a test post, screenshot for verification, and click the post button. Use human-like actions and wait_for as needed. Report status often."

The shadow DOM support (added in 0.2.0) helps surface elements inside X's web components. Combine with screenshot + status + loops of read_page / wait_for for resilience on SPAs.

See the testing section above for general flow patterns. Always start with status and read_page, use screenshot to ground the agent.

Trusted OS cursor (phase 2, macOS)

Content-script events are isTrusted=false, and chrome.debugger still leaks CDP tells. For genuinely trusted, indistinguishable input, switch to the OS-cursor driver, which moves the real macOS system cursor along the same human path:

pnpm add @nut-tree-fork/nut-js        # optional native dependency
AGENTCURSOR_DRIVER=os node dist/index.js

It still reads the page through the extension (keep a normal tab focused), but every move/click/scroll becomes a real OS event. Requires the Chrome window visible and foregrounded at 100% zoom, and Accessibility permission for your terminal/Node in System Settings → Privacy & Security. Coordinate mapping for multi-monitor / fractional-scaling setups is still rough.

Known limitations

• Desktop control is macOS only for now. It reads the accessibility tree, so apps that draw their own UI without accessibility (games, some canvases) need desktop_screenshot plus x/y clicks.

• macOS grants permissions to the app that launched AgentCursor (Terminal, Cursor, Claude...), not to AgentCursor itself. If the shared service was first started from a different app, grant that one, or stop the service (curl -X POST http://127.0.0.1:8931/shutdown) and let your main app start it.

• Desktop typing does not render typo corrections. The persona's timing applies, but only the final characters are typed.

• Content-script events are isTrusted=false. For detection-sensitive sites pass stealth: true (the chrome.debugger driver, trusted events) or use the OS-cursor driver. The visible overlay cursor shows in every mode.

• React-controlled inputs (X's composer, some design systems) can ignore content-script typing, which sets value directly. Use stealth: true (CDP Input.insertText) or the OS driver there.

• wait_for by text and the snapshot text field use innerText, which does not pierce shadow DOM. read_page's element list does traverse shadow roots, so prefer waiting on a [ref] over page text on web-component-heavy sites (X, Reddit).

• The OS-cursor driver assumes 100% browser zoom and a single display; multi-monitor and fractional scaling can be off.

• hover dispatches its hover events through the content script (the approach move is trusted under stealth, the explicit mouseover/mouseenter are not).

• Content-mode press_key carries key/code but not legacy keyCode — a constructed KeyboardEvent always reports keyCode: 0. Modern handlers read key; for sites that still check keyCode/which, use stealth: true (the CDP key event sets the real virtual key code).

Measuring realism

Serve the detector over http (the extension's content script only runs on http(s), not file://):

python3 -m http.server 8080 --directory test-detector

Open http://localhost:8080, click the targets by hand, then drive them with the agent. Each click is scored on straightness, velocity variance, dwell, off-center landing, overshoot, and isTrusted — the same features detectors use. Use it to tune the engine.

Development

pnpm dev         # run the server with tsx (no build)
pnpm typecheck    # tsc --noEmit
pnpm test             # vitest (path-engine + coord-map unit tests)
pnpm build:ext    # rebuild just the extension (no version change)
pnpm reload       # rebuild the extension AND patch-bump the version, so a chrome://extensions reload is visibly new
pnpm smoke        # end-to-end run: real MCP client + server, simulated browser (now covers screenshot/hover/status too)
pnpm build:native # rebuild the macOS accessibility helper (dist/native/agentcursor-ax)
AGENTCURSOR_HTTP_PORT=8931 node scripts/desktop-live.mjs   # live desktop run: reads Finder, types in TextEdit, closes it without saving

The smoke script is also a good template for writing your own automation or test runners that drive AgentCursor over MCP.

Credits

The path engine builds on the ghost-cursor lineage (Bézier + Fitts) and the mouse-dynamics literature — WindMouse, SapiAgent, BeCAPTCHA-Mouse, and the vendor write-ups from DataDome and Castle on what makes synthetic movement detectable. See docs/DESIGN.md.

License

MIT