armtrack

Ways to drive a rover arm by moving your own hand instead of a gamepad or a scale-model controller, built for the UMD Loop team's arm in the University Rover Challenge. Each method tracks the operator differently and feeds the same arm model, so they can be compared on one 3-D view.

Tracking	How it measures the operator	Code	Status
Stereo hand	two ordinary cameras, MediaPipe hand landmarks, stereo calibration and triangulation	tracking/stereo.py	works; depth noisy on fast motion
Mono hand	one camera, depth from the hand's apparent size (solvePnP on MediaPipe's metric hand model)	tracking/mono.py	works; depth only as good as the hand-size estimate
Phone (Arm Pilot)	a held iPhone tracked by ARKit, thumb pad for clutch and gripper	tracking/phone.py, ios/ArmPilot	recommended; drives the simulated arm (demo above)
Marker wand	a printed ArUco cube on a handle, one webcam, a gyro in the grip	tracking/wand.py, cad/, firmware/	verified in simulation; hardware being built

Layout

src/armtrack/
  tracking/   one module per way of tracking the operator (above)
  vision/     shared camera plumbing: capture, ChArUco boards, calibration, hand landmarks, geometry
  arm/        the arm: kinematics and IK, the three.js viewer, a scripted demo
ios/ArmPilot/ the iPhone app for phone tracking
cad/, firmware/  the wand's printable cube and grip, and its gyro firmware
ros/          the bridge to the rover's joint_trajectory_controller, tested on mock hardware
docs/         the wand design, the input-device comparison, and what driving the real arm needs

Every mode is a subcommand of one CLI: uv run armtrack --help.

Stereo hand tracking

How it works

1. Calibrate once. Wave a printed ChArUco board through both views. Each camera's lens is solved first, then where the cameras sit relative to each other (calib.json). ChArUco, not a plain checkerboard, so partial views still count and the two cameras never disagree on which corner is which. The cameras are not hardware-synchronised, so a pair is only captured while the board is held still.
2. Track. Both frames run through MediaPipe in parallel. Every combination of hand-in-camera-1 with hand-in-camera-2 is triangulated and the pairing whose 3-D points reproject best into both images wins. That is more reliable than trusting left/right labels from a side view. A One Euro filter smooths the result without adding lag when you move fast.
3. Frame. Output starts in camera-1 coordinates. Lay the board flat on the desk and press b to switch to a desk frame (x along the board, y across it, z up), and press z to zero at the current hand position so the output is a displacement: the natural input for an arm (a clutch, in teleop terms).

Setup (macOS)

curl -LsSf https://astral.sh/uv/install.sh | sh    # if uv is not installed
git clone <this repo> && cd armtrack
uv sync

iPhone as a camera. Continuity Camera needs macOS 13+, iOS 16+, the same Apple ID on both, Wi-Fi and Bluetooth on. Put the phone on a stand in landscape, rear camera toward you, screen locked. It then appears to the Mac as an ordinary camera. (Fallback: any iPhone "IP camera" app, and pass its stream URL as --cam2.)

Camera permission. Run from Terminal on the Mac itself, not over SSH; the first run asks to let Terminal use the camera. If it never asks: System Settings > Privacy & Security > Camera.

Run

uv run armtrack cameras                  # a labelled tile per camera: note the indices
uv run armtrack board                    # board.png: print at 100% scale
uv run armtrack calibrate --cam1 0 --cam2 1 --square-mm 35
uv run armtrack track --cam1 0 --cam2 1
uv run armtrack listen                   # in a second terminal: what an arm would receive

--square-mm is the side of one square as measured with a ruler on your print; printers scale.

No printer? Show the board on any screen that is not one of the two cameras. armtrack board --screen 2556x1179 --out board-phone.png fills a screen of that many pixels (that one is an iPhone 15, 15 Pro or 16 in landscape; use the device's own resolution). Open it full screen at full brightness, measure a square on the glass, and pass that as --square-mm. A laptop or tablet is better than a phone: bigger squares detect from further away. Keep glare off the screen, and expect to hold a phone closer, 25 to 50 cm from each camera.

Calibration tips. Aim for a stereo error under 1 px. Hold the board still for a beat at each pose; tilt it 30 to 45 degrees in different directions; cover near and far and every corner of both frames (the green overlay shows coverage). Pairs only count when both cameras see the board, so spend most of the time in the overlap.

Placement. 40 to 70 cm apart with 45 to 90 degrees between their views works well. Wider angles give better depth but less overlap. After calibrating, nothing may move, including the laptop lid angle. If either camera moves, recalibrate.

Track keys

key
z	zero here: output becomes displacement from this point
b	set the desk frame from the board lying flat where cam1 sees it (saved to calib.json)
r	record both raw streams to recordings/
a d w s	spin and tilt the 3-D view; + - zoom
q	quit

Output packet (UDP, default 127.0.0.1:5005)

{"t": 1790730000.1, "hand": "Right", "frame": "desk", "reproj_px": 2.1,
 "wrist_mm": [x, y, z], "relative_mm": [dx, dy, dz],
 "palm_center_mm": [...], "rotation": [[...]], "quat_wxyz": [...],
 "pinch_mm": 42.0, "grip_open": 0.35, "palm_length_mm": 95.0,
 "landmarks_mm": [[x, y, z], ...21]}

rotation columns are the hand's axes: x toward the index side, y from wrist toward the fingers, z out of the palm. grip_open is 0 when pinched and 1 when wide open, normalised by palm size.

Mono hand tracking

uv run armtrack mono --cam 1 --palm-mm 95

One camera cannot measure depth, so mono infers it from how big the hand looks. MediaPipe supplies a metric model of the hand's shape, and solvePnP finds where that model must sit for it to line up with the landmarks in the image. Left/right and up/down are about as good as stereo. Depth is only as good as the hand-size assumption: an unmeasured hand 10% larger than average reads 10% closer. --palm-mm (wrist crease to the base of the middle finger) removes most of that. Depth is also noisier, because a 1 px error in apparent size is a few mm at arm's length. If calib.json exists, its lens for that camera is reused (--calib-cam); otherwise the lens is guessed from --hfov. On a Mac, turn off Center Stage (Control Center > Video Effects): it crops and zooms on its own, which changes the lens under the solver's feet.

Marker wand

A printed ArUco cube on a handle, seen by one calibrated webcam, with a gyro in the grip for fast rotation. uv run armtrack wand-markers writes the printable markers, armtrack lens calibrates the camera, and armtrack wand runs the live loop. Design, measurements and parts: docs/wand.md.

The rover arm

Every mode opens a 3-D view in the browser (http://localhost:8765, opened for you): the Loop rover on its rocker-bogie suspension with the arm (turret, shoulder motors, actuator-driven upper arm, forearm, pitch and roll wrist, wrist camera, two-finger gripper), lit and shadowed. Drag to orbit, right-drag or shift-drag to pan, scroll to zoom; buttons jump to side, front and top views, follow the gripper, or draw its trail. Anyone on the network can open the same page, so it can go on a projector.

uv run armtrack demo runs a scripted pick and place with no tracker, for showing the arm on its own.

The control path, in arm.py:

• Target. Your motion since engaging, scaled 2.5x, moves a target for the gripper tip plus a wrist pitch and roll. Targets are kept above the ground, out of the chassis and mast, and within reach.
• IK. A damped least-squares solver, warm-started from the last answer and clamped to joint limits every step, so the arm never flips between elbow configurations, stays stable at full stretch and over its base, and settles at the nearest pose when a target is out of reach.
• Motion. Every joint is capped at 0.5 rad/s (about 29 deg/s), the limit in the team's URDF, with an acceleration limit on top, and brakes in time to stop on its goal. The arm therefore lags a fast hand the way the real one would, and a tracking glitch cannot fling it. The gripper takes 0.8 s end to end. The view shows the commanded target as a green ghost and the joint speeds as amber bars.

Upper arm and forearm lengths (0.582 m each), joint speed and the wrist roll range come from the team's URDF in umdloop/software-urc4. The wrist and gripper lengths, the pitch joints' ranges and the accelerations in ArmSpec are still estimates. The joint angles go out in every UDP packet under arm.

Phone tracking (Arm Pilot, recommended)

Camera tracking of a bare hand is jittery in depth. Holding an iPhone avoids that: ARKit tracks the phone's position and rotation by fusing its camera with its motion sensors, so it neither drifts like integrated accelerometer data nor jitters like a hand's apparent size.

1. On the Mac: uv run armtrack phone. It prints this Mac's IP address and opens the 3-D view.
2. Open ios/ArmPilot/ArmPilot.xcodeproj in Xcode, plug in the iPhone, pick it as the run destination, and press Run. (First time: Developer Mode on the phone, and trust the developer.)
3. In Arm Pilot, type the Mac's IP. Phone and Mac must be able to reach each other: the same home Wi-Fi, the phone's hotspot, or Tailscale on both. Large shared networks (campus, hotel) usually block device-to-device traffic.
4. Hold the phone screen-up with the top edge pointing where the arm should reach. Hold your thumb on the big pad to move the arm and lift it to let go (the arm holds). While holding, slide your thumb up to open the gripper and down to close it, as far or as little as you want. Forward is wherever the phone points when your thumb goes down.

FINE switches the arm from 2.5x your motion to 0.5x, for lining up on a key or a button; switching mid-hold does not move the arm. HOME sends it back to its rest pose. When you push the arm out of reach or onto a joint limit, the pad turns red and the phone buzzes: the Mac reports it back over the same connection ten times a second.

The back camera needs to see an ordinary room for tracking to stay "normal"; the app shows the tracking state.

Tests

uv run pytest tests

These render the board into two simulated cameras 40 cm apart, calibrate with the real code, and check that the rig, a triangulated hand, and the desk frame come back within about 1%. They also run MediaPipe on a sample image.

Limits, and where this goes next

• Sync. The cameras free-run, so a fast hand is seen at slightly different instants. The skew is shown in the view; it costs accuracy on fast motion.
• Hand only. An arm controller probably wants the whole arm (shoulder, elbow, wrist) to map joint-for-joint. MediaPipe Pose gives those landmarks and triangulates the same way; alternatively, drive the rover arm's inverse kinematics from the hand pose alone, which is closer to how it is used.
• Latency and dropouts. Before this goes near real hardware it needs a deadman (no hand for N frames means hold position), rate and workspace limits, and a scale factor between hand motion and arm motion.