Add Model Ghost Preview extension (VIZ-3444)

.gitignore

@@ -1 +1,5 @@
-.vscode/*
+.vscode/*
+node_modules/
+dist/
+*.foxe
+demo.mcap

.utils/generate_readme.py

@@ -14,6 +14,7 @@
 
 CATEGORY_NAMES = {
     "datasets": "Datasets",
+    "extensions": "Foxglove Extensions",
     "foxglove_sdk": "Foxglove SDK",
     "integrations": "Integrations",
     "jupyter_notebooks": "Jupyter Notebooks"
@@ -43,6 +44,7 @@ def extract_metadata(readme_path):
 def scan_tutorials():
     tutorials = []
     for root, dirs, files in os.walk(ROOT_DIR):
+        dirs[:] = [name for name in dirs if name not in {"node_modules", "dist", ".git"}]
         if "README.md" in files and root != ROOT_DIR:
             readme_path = os.path.join(root, "README.md")
             rel_path = os.path.relpath(readme_path, ROOT_DIR)

README.md

@@ -29,6 +29,10 @@ Below is a list of all tutorials available in this repository:
 - 🎥 [Video](https://youtu.be/jJej6aT1jKg)
 - 📊 [Visualize](https://app.foxglove.dev/~/view?ds=foxglove-sample-stream&ds.recordingId=vqKKQcot421Kwg84&ds.overrideLayoutId=b7513959-1d46-4a89-bc24-1584d9677ca1&ds.start=2023-09-01T13:19:45.047438263Z&ds.end=2023-09-01T13:20:15.047438263Z)
 
+## Foxglove Extensions
+### [Model Ghost Preview extension](extensions/model_ghost_preview/README.md)
+- 📝 Preview a 3D model as a ghost at the hovered timeline time, even while paused
+
 ## Foxglove SDK
 ### [Using Foxglove to Visualize Ethernet/IP data](foxglove_sdk/ethernet_ip_integration/README.md)
 - 📝 Using Foxglove data, it's easier than ever to stream time series data. In this project, we show you how.

extensions/model_ghost_preview/README.md

@@ -0,0 +1,109 @@
+---
+title: "Model Ghost Preview extension"
+short_description: "Preview a 3D model as a ghost at the hovered timeline time, even while paused"
+---
+
+# Model Ghost Preview extension
+
+While you hover the playback bar, this panel keeps the model at the current time and draws a second copy — by default a light-blue wireframe — at the hovered time. A path through every pose shows where the ghost sits along the trajectory.
+
+![Solid haul truck at the current time](media/current-truck.png)
+
+![Wireframe ghost further along the path](media/ghost-preview.png)
+
+The ghost only appears when preview is enabled, a preview time is set, and that pose is different from the current one. Hovering the path inside the panel also sets the preview time (so Map and Plot can follow it); clicking the path seeks playback.
+
+## Why a custom panel
+
+VIZ-3444 asks for this on the built-in 3D panel, the way the Map panel already previews GNSS while you skim the timeline. A message converter cannot do it: converters run when a new input message arrives, and global variables do not reprocess the current frame. While playback is paused, nothing arrives, so the ghost cannot move with the hover. Built-in 3D panel internals are not extensible either. This extension is a working prototype of the interaction — its own three.js scene — until the 3D panel supports preview natively.
+
+## Install
+
+From `extensions/model_ghost_preview/extension`:
+
+```sh
+npm ci && npm run local-install
+```
+
+That installs the extension into Foxglove desktop. Refresh the app (`Ctrl-R`) and add a **Model Ghost Preview** panel.
+
+To share a build instead, package it and drag the `.foxe` into Foxglove:
+
+```sh
+npm run package
+```
+
+## Demo data
+
+```sh
+cd extensions/model_ghost_preview/demo
+pip install -r requirements.txt
+python generate_demo_mcap.py --output demo.mcap
+```
+
+The file is about 60 seconds at 20 Hz. A truck drives a winding haul road a few hundred meters long.
+
+| Topic | Schema | What it is |
+| --- | --- | --- |
+| `/truck/pose` | `foxglove.PoseInFrame` | Pose in the `map` frame. This is the ghost panel input. |
+| `/tf` | `foxglove.FrameTransforms` | `map` → `truck`, so the built-in 3D panel can follow the same motion. |
+| `/scene/road` | `foxglove.SceneUpdate` | Road ribbon, center line, and a few berms, published once. |
+
+Open the MCAP in Foxglove, then import [`foxglove_layouts/model_ghost_preview.json`](foxglove_layouts/model_ghost_preview.json) from the layout menu. The layout places this panel next to a 3D panel and a Raw Messages panel on `/truck/pose`. If a panel shows up as unknown after import, add **Model Ghost Preview** from the panel list and pick the pose topic (or add the panel yourself, then export a layout).
+
+Poses are drawn in the topic's own frame. That frame is the panel's fixed world frame — there is no TF tree. The message **receive time** (log time) is the timeline key, because that is what the playback bar and `currentTime` use.
+
+## Settings
+
+| Group | Setting | Notes |
+| --- | --- | --- |
+| General | Pose topic | Supported schemas only. |
+| General | Child frame | Shown for TF-like schemas. Empty locks onto the first `child_frame_id` seen in the range; the field suggests frames observed on the topic. |
+| General | Interpolation | `interpolate` (lerp position, slerp rotation) or `previous`. |
+| Model | Source | Procedural haul truck, or a `.glb` / `.gltf` URL. |
+| Model | Scale, yaw, pitch, roll | Degrees, Euler ZYX. |
+| Model | Truck color | Body color of the procedural truck. |
+| Preview | Visibility | The Preview ON / OFF toggle. |
+| Preview | Style | `wireframe` (default), `transparent`, or `solid`. |
+| Preview | Color, opacity | Wireframe uses an edge overlay plus a faint fill. |
+| Preview | Time label | `+Δt s` relative to the current time. |
+| Path | Visibility, color | Polyline of every pose. |
+| Path | Highlight segment | Brighter, thicker span between current and preview time. |
+| View | Follow | `off`, `current`, or `ghost`. Orbit offset is kept. |
+| View | Grid | Ground grid. The scene is Z-up. |
+
+glTF assets are Y-up. The loader applies +90° about X before the yaw/pitch/roll offsets so the model stands Z-up. Set roll to -90 to cancel that correction. If the URL fails to load, the panel shows the error and falls back to the truck.
+
+Once the pose range has loaded, the camera frames the road around the current pose from above and to one side, pulled in so a long haul road does not shrink the truck to a speck. Hovering a path vertex sets the preview time; a click without a drag seeks. Orbit dragging is left alone.
+
+## Supported schemas
+
+- `foxglove.PoseInFrame`
+- `foxglove.PosesInFrame` (first pose)
+- `foxglove.FrameTransform` and `foxglove.FrameTransforms`
+- `geometry_msgs/PoseStamped`, `geometry_msgs/msg/PoseStamped`
+- `nav_msgs/Odometry`, `nav_msgs/msg/Odometry` (`pose.pose`)
+- `tf2_msgs/TFMessage`, `tf2_msgs/msg/TFMessage`
+- `geometry_msgs/TransformStamped`, `geometry_msgs/msg/TransformStamped`
+
+## Limitations
+
+- One fixed frame. Transforms are not composed through a TF tree.
+- `subscribeMessageRange` is best-effort. Very large recordings can be truncated by memory limits, and only the extracted poses are kept.
+- Live sources (Foxglove WebSocket, rosbridge, native ROS) do not provide a future range, so there is nothing to preview ahead of the current time. Recorded data is required.
+- The built-in 3D panel does not show this ghost. Native preview there is what VIZ-3444 asks for; this panel is the stop-gap.
+
+## Development
+
+From `extensions/model_ghost_preview/extension`:
+
+```sh
+npm ci
+npm test
+npm run build
+npm run lint
+npx tsc --noEmit
+npm run dev:harness
+```
+
+`npm test` covers pose extraction, timeline interpolation, preview-time normalization, and the settings reducer. `npm run dev:harness` mounts the real panel on synthetic truck poses with a fake playback bar: click seeks, hover sets an absolute preview time, and Play/Pause runs the clock. Screenshots in `media/` were captured from that harness.

extensions/model_ghost_preview/demo/generate_demo_mcap.py

@@ -0,0 +1,211 @@
+#!/usr/bin/env python3
+"""Write a short haul-road MCAP for the Model Ghost Preview panel."""
+
+import argparse
+import math
+from pathlib import Path
+
+import foxglove
+from foxglove.channels import FrameTransformsChannel, PoseInFrameChannel, SceneUpdateChannel
+from foxglove.messages import (
+    Color,
+    CubePrimitive,
+    FrameTransform,
+    FrameTransforms,
+    LinePrimitive,
+    LinePrimitiveLineType,
+    Point3,
+    Pose,
+    PoseInFrame,
+    Quaternion,
+    SceneEntity,
+    SceneUpdate,
+    Timestamp,
+    TriangleListPrimitive,
+    Vector3,
+)
+
+EPOCH_SEC = 1_700_000_000
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description="Generate a haul-truck ghost-preview demo MCAP")
+    parser.add_argument("--output", default="demo.mcap", help="Output .mcap path")
+    parser.add_argument("--duration", type=float, default=60.0, help="Recording length in seconds")
+    parser.add_argument("--rate", type=float, default=20.0, help="Pose sample rate in Hz")
+    args = parser.parse_args()
+    if args.duration <= 0 or args.rate <= 0:
+        raise SystemExit("--duration and --rate must be positive")
+
+    output = Path(args.output)
+    samples = [sample_pose(index / args.rate, args.duration) for index in range(int(args.duration * args.rate))]
+    with foxglove.open_mcap(output, allow_overwrite=True):
+        pose_channel = PoseInFrameChannel("/truck/pose")
+        tf_channel = FrameTransformsChannel("/tf")
+        scene_channel = SceneUpdateChannel("/scene/road")
+        for sample in samples:
+            log_time = int(round(sample["time_sec"] * 1e9))
+            stamp = timestamp(sample["time_sec"])
+            pose = Pose(
+                position=Vector3(x=sample["x"], y=sample["y"], z=sample["z"]),
+                orientation=Quaternion(x=sample["qx"], y=sample["qy"], z=sample["qz"], w=sample["qw"]),
+            )
+            pose_channel.log(
+                PoseInFrame(timestamp=stamp, frame_id="map", pose=pose),
+                log_time=log_time,
+            )
+            tf_channel.log(
+                FrameTransforms(
+                    transforms=[
+                        FrameTransform(
+                            timestamp=stamp,
+                            parent_frame_id="map",
+                            child_frame_id="truck",
+                            translation=Vector3(x=sample["x"], y=sample["y"], z=sample["z"]),
+                            rotation=Quaternion(
+                                x=sample["qx"], y=sample["qy"], z=sample["qz"], w=sample["qw"]
+                            ),
+                        )
+                    ]
+                ),
+                log_time=log_time,
+            )
+        scene_channel.log(road_scene(samples), log_time=int(EPOCH_SEC * 1e9))
+    print(f"wrote {output} ({len(samples)} poses)")
+
+
+def sample_pose(t_sec: float, duration: float) -> dict[str, float]:
+    length = 280.0
+    s = 0.0 if duration == 0 else t_sec / duration
+    x = s * length
+    y = 35.0 * math.sin(s * math.pi * 1.5) + 8.0 * math.sin(s * math.pi * 4.0)
+    z = 6.0 * s + 1.5 * math.sin(s * math.pi * 2.0)
+    ds = 1.0 / duration
+    dx = length * ds
+    dy = (
+        35.0 * math.pi * 1.5 * ds * math.cos(s * math.pi * 1.5)
+        + 8.0 * math.pi * 4.0 * ds * math.cos(s * math.pi * 4.0)
+    )
+    dz = 6.0 * ds + 1.5 * math.pi * 2.0 * ds * math.cos(s * math.pi * 2.0)
+    yaw = math.atan2(dy, dx)
+    pitch = math.atan2(dz, math.hypot(dx, dy))
+    qx, qy, qz, qw = heading_quaternion(yaw, -pitch)
+    return {
+        "time_sec": EPOCH_SEC + t_sec,
+        "x": x,
+        "y": y,
+        "z": z,
+        "qx": qx,
+        "qy": qy,
+        "qz": qz,
+        "qw": qw,
+        "yaw": yaw,
+    }
+
+
+def heading_quaternion(yaw: float, pitch: float) -> tuple[float, float, float, float]:
+    half_yaw = yaw * 0.5
+    half_pitch = pitch * 0.5
+    q_yaw = (0.0, 0.0, math.sin(half_yaw), math.cos(half_yaw))
+    q_pitch = (0.0, math.sin(half_pitch), 0.0, math.cos(half_pitch))
+    return multiply_quat(q_yaw, q_pitch)
+
+
+def multiply_quat(
+    a: tuple[float, float, float, float], b: tuple[float, float, float, float]
+) -> tuple[float, float, float, float]:
+    ax, ay, az, aw = a
+    bx, by, bz, bw = b
+    return (
+        aw * bx + ax * bw + ay * bz - az * by,
+        aw * by - ax * bz + ay * bw + az * bx,
+        aw * bz + ax * by - ay * bx + az * bw,
+        aw * bw - ax * bx - ay * by - az * bz,
+    )
+
+
+def timestamp(time_sec: float) -> Timestamp:
+    sec = int(math.floor(time_sec))
+    nsec = int(round((time_sec - sec) * 1e9))
+    if nsec >= 1_000_000_000:
+        sec += 1
+        nsec -= 1_000_000_000
+    return Timestamp(sec=sec, nsec=nsec)
+
+
+def road_scene(samples: list[dict[str, float]]) -> SceneUpdate:
+    stride = 5
+    center = samples[::stride]
+    if len(center) < 2:
+        center = samples
+    half_width = 7.0
+    points: list[Point3] = []
+    for index, sample in enumerate(center):
+        if index + 1 < len(center):
+            nxt = center[index + 1]
+            dx = nxt["x"] - sample["x"]
+            dy = nxt["y"] - sample["y"]
+        else:
+            prev = center[index - 1]
+            dx = sample["x"] - prev["x"]
+            dy = sample["y"] - prev["y"]
+        norm = math.hypot(dx, dy) or 1.0
+        px = -dy / norm
+        py = dx / norm
+        points.append(
+            Point3(x=sample["x"] + px * half_width, y=sample["y"] + py * half_width, z=sample["z"])
+        )
+        points.append(
+            Point3(x=sample["x"] - px * half_width, y=sample["y"] - py * half_width, z=sample["z"])
+        )
+    indices: list[int] = []
+    rows = len(center)
+    for index in range(rows - 1):
+        base = index * 2
+        indices.extend([base, base + 1, base + 2, base + 1, base + 3, base + 2])
+    line_points = [Point3(x=sample["x"], y=sample["y"], z=sample["z"] + 0.05) for sample in center]
+    berms: list[CubePrimitive] = []
+    for index, sample in enumerate(center[::8]):
+        side = -1.0 if index % 2 == 0 else 1.0
+        berms.append(
+            CubePrimitive(
+                pose=Pose(
+                    position=Vector3(
+                        x=sample["x"] + math.cos(sample["yaw"] + math.pi / 2.0) * 9.0 * side,
+                        y=sample["y"] + math.sin(sample["yaw"] + math.pi / 2.0) * 9.0 * side,
+                        z=sample["z"] + 0.6,
+                    ),
+                    orientation=Quaternion(x=0.0, y=0.0, z=0.0, w=1.0),
+                ),
+                size=Vector3(x=3.2, y=1.4, z=1.2),
+                color=Color(r=0.55, g=0.48, b=0.32, a=1.0),
+            )
+        )
+    stamp = timestamp(samples[0]["time_sec"] if samples else EPOCH_SEC)
+    entity = SceneEntity(
+        timestamp=stamp,
+        frame_id="map",
+        id="haul-road",
+        frame_locked=False,
+        triangles=[
+            TriangleListPrimitive(
+                points=points,
+                indices=indices,
+                color=Color(r=0.35, g=0.37, b=0.4, a=1.0),
+            )
+        ],
+        lines=[
+            LinePrimitive(
+                type=LinePrimitiveLineType.LineStrip,
+                thickness=0.35,
+                points=line_points,
+                color=Color(r=0.17, g=0.28, b=0.95, a=1.0),
+            )
+        ],
+        cubes=berms,
+    )
+    return SceneUpdate(entities=[entity])
+
+
+if __name__ == "__main__":
+    main()

extensions/model_ghost_preview/demo/requirements.txt

@@ -0,0 +1,2 @@
+foxglove-sdk>=0.28,<0.29
+mcap>=1.2.0