From 444b3ccfe82542ef4646e3d3ef254b8d5908861a Mon Sep 17 00:00:00 2001 From: Dmitry Slepichev Date: Wed, 14 Oct 2026 03:34:36 +0400 Subject: [PATCH 1/3] [clean] document one capture instant per track() call Feeding track() one camera per call looked allowed, since partial image sets are supported, but cuVSLAM matches cameras only within a call. State the rule in the Odometry::Track and Python track() docs and the README, add a short TROUBLESHOOTING section with remedies, and fix skill docs that suggested a wider sync threshold or described image_jitter_threshold_ms as a tolerance between cameras. Also correct the timestamp source (images[0]) and the nonexistent is_valid in Odometry.track. Co-Authored-By: Claude Opus 5.5 --- README.md | 2 +- TROUBLESHOOTING.md | 20 +++++++++++++++++-- cuvslam-skills/cuvslam-troubleshoot/SKILL.md | 4 ++-- .../cuvslam-troubleshoot/commands/bag2edex.md | 2 +- .../references/isaac-ros-visual-slam-usage.md | 4 ++-- libs/cuvslam/cuvslam2.h | 6 +++--- python/cuvslam2.cpp | 8 +++++--- 7 files changed, 32 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index 2968ff5a..ee2da3c8 100644 --- a/README.md +++ b/README.md @@ -118,7 +118,7 @@ The accuracy and robustness of cuVSLAM can be influenced by several factors. If - **Intrinsic and Extrinsic Calibration**: Accurate camera calibration is crucial. Ensure your calibration parameters are precise. For more details, refer to our guide on [image undistortion](examples/euroc/README.md#distortion-models). If you're new to calibration, consider working with [experienced vendors](https://nvidia-isaac-ros.github.io/v/release-3.2/getting_started/hardware_setup/sensors/amr_extrinsic_calibration.html#extrinsic-calibration-of-sensors-in-custom-locations) -- **Synchronization and Timestamps**: Accurate synchronization significantly impacts cuVSLAM performance. Make sure multi-camera images are captured simultaneously—ideally through hardware synchronization—and verify correct relative timestamps across cameras. Refer to our [multi-camera hardware assembly guide](examples/realsense/multicamera_hardware_assembly.md) for building a rig with synchronized RealSense cameras +- **Synchronization and Timestamps**: Accurate synchronization significantly impacts cuVSLAM performance. Make sure multi-camera images are captured simultaneously—ideally through hardware synchronization—and verify correct relative timestamps across cameras. Refer to our [multi-camera hardware assembly guide](examples/realsense/multicamera_hardware_assembly.md) for building a rig with synchronized RealSense cameras. Each `track()` call must carry images captured at one instant (within 1 ms); cycling through cameras or camera groups across calls is not supported - **Frame Rate**: Frame rate significantly affects performance. The ideal frame rate depends on translational and rotational velocities. Typically, 30 FPS is suitable for most "human-speed" motions. Adjust accordingly for faster movements diff --git a/TROUBLESHOOTING.md b/TROUBLESHOOTING.md index 786fe20b..47c35bd1 100644 --- a/TROUBLESHOOTING.md +++ b/TROUBLESHOOTING.md @@ -41,6 +41,7 @@ See [Requirements](README.md#performance). - [Try to use rectified images if possible](#try-to-use-rectified-images-if-possible) - [Disable SLAM and ground constraint](#disable-slam-and-ground-constraint) - [Step 6: Verify synchronization](#step-6-verify-synchronization) + - [Cameras that are not synchronized with each other](#cameras-that-are-not-synchronized-with-each-other) - [Step 7: Run stereo odometry tracking](#step-7-run-stereo-odometry-tracking) - [Step 8: Adjust odometry settings](#step-8-adjust-odometry-settings) - [Improve input images](#improve-input-images) @@ -150,8 +151,9 @@ cuvslam::Odometry::Config::max_frame_delta_s image_jitter_threshold_ms ``` -In the Isaac ROS cuVSLAM node, there is also a synchronizer that can align multiple image streams with non-equal -timestamps. Check Isaac ROS documentation and tune: +In the Isaac ROS cuVSLAM node, there is also a synchronizer that groups images whose timestamps are within +`sync_matching_threshold_ms` into one frame; it cannot synchronize free-running cameras. Check Isaac ROS documentation +and tune: ``` num_cameras @@ -351,6 +353,20 @@ Example: Left and right eyes out of sync ![Example: Left and right eyes out of sync](doc/images/troubleshooting_desync.gif) +### Cameras that are not synchronized with each other + +Each `track()` call must carry images captured at one instant (within 1 ms). A camera may be missing from a call, +for example after a frame drop, but cycling through cameras or camera groups across calls is not supported, and neither +is filling an empty slot with an older image or giving images from different instants one timestamp. cuVSLAM matches +features across cameras only within a call. `track()` rejects a call whose image timestamps differ by 1 ms or more, +but when such input passes this check, the pose freezes, drifts or, with an IMU, diverges without an error. + +If your cameras are not synchronized: + +1. Synchronize them in hardware (external trigger). +2. If they mostly line up, pass only frame sets whose images are within 1 ms of each other and drop the rest. +3. Otherwise, run one tracker per camera (or per synchronized stereo pair) and fuse their poses yourself. + ## Step 7: Run stereo odometry tracking If mono left and mono right both work, run stereo tracking. Stereo reuses extrinsic calibration via the diff --git a/cuvslam-skills/cuvslam-troubleshoot/SKILL.md b/cuvslam-skills/cuvslam-troubleshoot/SKILL.md index 0885b8e8..42743194 100644 --- a/cuvslam-skills/cuvslam-troubleshoot/SKILL.md +++ b/cuvslam-skills/cuvslam-troubleshoot/SKILL.md @@ -90,7 +90,7 @@ Key parameters to note once you have the config: cuNLS availability, `depth_camera_ids`, and pinhole calibration. Knowing this up front directs the diagnosis toward vision, depth, or IMU root causes. - `rectified_images` (Isaac ROS) / `rectified_stereo_camera` (C++ API) — wrong value flips the entire stereo pipeline -- `image_jitter_threshold_ms` — tolerated timestamp jitter between left/right images; too tight a value causes frames to be dropped silently +- `image_jitter_threshold_ms` — tolerated gap between consecutive frames (exceeding it only logs a warning), not a tolerance between cameras - `Tracker.Mode` plus the `async_sba` / `sync_mode` that must match it — affects reproducibility - `enable_slam` / loop closure settings — SLAM should only be tuned after odometry is solid - `denoise_input_images`, `imu_from_left` — sensor-specific flags @@ -169,7 +169,7 @@ With the config and dataset in hand, verify these before touching the diagnostic 1. **Accurate intrinsic calibration** — verify distortion model and params match the actual camera. See `references/edex-calibration-pitfalls.md` for common EDEX mistakes (e.g. `"fisheye"` with zero params ≠ `"pinhole"`). 2. **Accurate extrinsic calibration** (stereo/multi-cam) — verify baseline axis and magnitude match the physical rig. See `references/edex-calibration-pitfalls.md` for wrong-axis baseline diagnosis. -3. **Proper image synchronization** — left/right timestamps must match per frame. +3. **Proper image synchronization** — each `track()` call carries images captured at one instant (within 1 ms), never cameras cycled across calls. 4. **Sharp images, low noise** — frame rate matters more than resolution. 5. **Textured, rigid scenes** — avoid repetitive patterns and dynamic objects. 6. **Correct parameters and coordinate frames** — check `rectified_stereo_camera` flag. diff --git a/cuvslam-skills/cuvslam-troubleshoot/commands/bag2edex.md b/cuvslam-skills/cuvslam-troubleshoot/commands/bag2edex.md index 081539b3..99c13f36 100644 --- a/cuvslam-skills/cuvslam-troubleshoot/commands/bag2edex.md +++ b/cuvslam-skills/cuvslam-troubleshoot/commands/bag2edex.md @@ -225,7 +225,7 @@ If any topic prints a frame_id belonging to a different camera, set `camera_opti **Wrong intrinsics** — intrinsics are read from `camera_info` messages in the bag. Verify the published `camera_info` matches the actual camera before converting. -**Mismatched frame counts** — only synchronized frame pairs (within `sync_threshold_ns`) are saved. Increase `sync_threshold_ns` in the config if too many frames are dropped. +**Mismatched frame counts** — only frame sets whose image topics are all within `sync_threshold_ns` (default 1 ms) are saved. Don't raise it to keep more frames: images more than 1 ms apart are not one capture instant. **No synchronized frames found** — the tool exits with an error listing per-topic frame counts. Check that image topics are correct and timestamps overlap within `sync_threshold_ns`. diff --git a/cuvslam-skills/cuvslam-troubleshoot/references/isaac-ros-visual-slam-usage.md b/cuvslam-skills/cuvslam-troubleshoot/references/isaac-ros-visual-slam-usage.md index 35624de9..a0f41518 100644 --- a/cuvslam-skills/cuvslam-troubleshoot/references/isaac-ros-visual-slam-usage.md +++ b/cuvslam-skills/cuvslam-troubleshoot/references/isaac-ros-visual-slam-usage.md @@ -65,8 +65,8 @@ ros2 launch isaac_ros_visual_slam isaac_ros_visual_slam_realsense_bag.launch.py | `enable_localization_n_mapping` | `true` | `false` = odometry only, no loop closure | | `enable_image_denoising` | `false` | Enable for low-light/noisy cameras | | `enable_ground_constraint_in_slam` | `false` | Constrain to horizontal plane for wheeled robots | -| `image_jitter_threshold_ms` | — | Tolerated timestamp jitter between left/right images | -| `sync_matching_threshold_ms` | — | Tolerated timestamp offset across cameras for sync | +| `image_jitter_threshold_ms` | `34.0` | Tolerated gap between consecutive frames; exceeding it only logs a warning | +| `sync_matching_threshold_ms` | `5.0` | Grouping window only; `Odometry::Track` rejects images differing by 1 ms or more, so keep grouped timestamps under 1 ms | | `base_frame` | `base_link` | Robot base frame for TF | | `camera_optical_frames` | — | List of camera optical frame names (must match TF tree) | | `imu_frame` | `imu` | IMU frame (VIO mode only) | diff --git a/libs/cuvslam/cuvslam2.h b/libs/cuvslam/cuvslam2.h index 3a626244..5baec1ba 100644 --- a/libs/cuvslam/cuvslam2.h +++ b/libs/cuvslam/cuvslam2.h @@ -593,13 +593,13 @@ class CUVSLAM_API Odometry { * * The track will output poses in the same coordinate system until a loss of tracking. * - * Image timestamps have to match. cuVSLAM will use timestamp from the camera 0 image. + * Image timestamps have to match. cuVSLAM will use the timestamp of images[0]. * If a camera rig provides "almost synchronized" frames, the timestamps should be within 1 millisecond. * The number of images for tracker should not exceed rig->num_cameras. * * @param[in] images an array of synchronized images no more than rig->num_cameras. - * Must use ImageData::DataType::UINT8. Partial ImageSet is supported, for example due to frame drops. Corresponding - * cameras are identified by Image::camera_index. + * Must use ImageData::DataType::UINT8. Partial ImageSet is supported, for example due to frame drops, but cycling + * through cameras or camera groups across calls is not. Corresponding cameras are identified by Image::camera_index. * @param[in] masks (Optional) an array of corresponding masks no more than rig->num_cameras. * Must use ImageData::DataType::UINT8 and ImageData::Encoding::MONO. Non-zero pixels are excluded: no features are * selected or kept there, and in OdometryMode::RGBD and OdometryMode::Multisensor the depth at those pixels is diff --git a/python/cuvslam2.cpp b/python/cuvslam2.cpp index 7e7414fb..f36cc6fc 100644 --- a/python/cuvslam2.cpp +++ b/python/cuvslam2.cpp @@ -716,7 +716,8 @@ NB_MODULE(pycuvslam, m) { "images etc. can still be either CPU or GPU arrays/tensors. The reverse does not hold:\n" "odometry with `use_gpu` disabled only accepts CPU arrays/tensors.\n\n" "The images etc. must be in the same order as cameras in the rig.\n" - "If data for a camera is not available, pass empty array or tensor for that camera image.\n\n" + "If data for a camera is not available, pass empty array or tensor for that camera image.\n" + "Cycling through cameras or camera groups across calls is not supported.\n\n" "Parameters:\n" " timestamp: Image timestamp in nanoseconds\n" " images: List of numpy arrays containing the camera images\n" @@ -726,7 +727,7 @@ NB_MODULE(pycuvslam, m) { " depths: Optional list of numpy arrays containing depth images\n" " options: Optional per-frame options that override default settings for this call only\n\n" "Returns:\n" - " :class:`PoseEstimate` object with the computed pose. If tracking fails, `is_valid` will be False.") + " :class:`PoseEstimate` object with the computed pose. If tracking fails, `world_from_rig` is `None`.") .def( "register_imu_measurement", [](Odometry& self, uint32_t sensor_index, const ImuMeasurement& imu_measurement) { @@ -1287,7 +1288,8 @@ NB_MODULE(pycuvslam, m) { "images etc. can still be either CPU or GPU arrays/tensors. The reverse does not hold:\n" "odometry with `use_gpu` disabled only accepts CPU arrays/tensors.\n\n" "The images etc. must be in the same order as cameras in the rig.\n" - "If data for a camera is not available, pass empty array or tensor for that camera image.\n\n" + "If data for a camera is not available, pass empty array or tensor for that camera image.\n" + "Cycling through cameras or camera groups across calls is not supported.\n\n" "Parameters:\n" " timestamp: Images timestamp in nanoseconds\n" " images: List of numpy arrays or tensors containing the camera images\n" From 44c45ee83d451c7798232a0e77c60ac4092f4389 Mon Sep 17 00:00:00 2001 From: Dmitry Slepichev Date: Wed, 14 Oct 2026 19:08:11 +0400 Subject: [PATCH 2/3] [clean] state exact sync boundaries, drop untested desync outcomes bag2edex keeps a frame set only when its timestamps differ by less than sync_threshold_ns, and track() rejects images 1 ms or more apart, so say so exactly. No contract or test backs the claim that such input freezes, drifts or, with an IMU, diverges the pose, and the outcome varies by odometry mode, so state only that tracking accuracy is not guaranteed. Co-Authored-By: Claude Opus 5.5 --- TROUBLESHOOTING.md | 2 +- cuvslam-skills/cuvslam-troubleshoot/commands/bag2edex.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/TROUBLESHOOTING.md b/TROUBLESHOOTING.md index 47c35bd1..91b337dc 100644 --- a/TROUBLESHOOTING.md +++ b/TROUBLESHOOTING.md @@ -359,7 +359,7 @@ Each `track()` call must carry images captured at one instant (within 1 ms). A c for example after a frame drop, but cycling through cameras or camera groups across calls is not supported, and neither is filling an empty slot with an older image or giving images from different instants one timestamp. cuVSLAM matches features across cameras only within a call. `track()` rejects a call whose image timestamps differ by 1 ms or more, -but when such input passes this check, the pose freezes, drifts or, with an IMU, diverges without an error. +but when such input passes this check, tracking accuracy is not guaranteed. If your cameras are not synchronized: diff --git a/cuvslam-skills/cuvslam-troubleshoot/commands/bag2edex.md b/cuvslam-skills/cuvslam-troubleshoot/commands/bag2edex.md index 99c13f36..9aef446d 100644 --- a/cuvslam-skills/cuvslam-troubleshoot/commands/bag2edex.md +++ b/cuvslam-skills/cuvslam-troubleshoot/commands/bag2edex.md @@ -225,7 +225,7 @@ If any topic prints a frame_id belonging to a different camera, set `camera_opti **Wrong intrinsics** — intrinsics are read from `camera_info` messages in the bag. Verify the published `camera_info` matches the actual camera before converting. -**Mismatched frame counts** — only frame sets whose image topics are all within `sync_threshold_ns` (default 1 ms) are saved. Don't raise it to keep more frames: images more than 1 ms apart are not one capture instant. +**Mismatched frame counts** — a frame set is saved only if its image timestamps differ by less than `sync_threshold_ns` (default 1 ms); a difference equal to or greater than the threshold is rejected. Don't raise it to keep more frames: images 1 ms or more apart are not one capture instant. **No synchronized frames found** — the tool exits with an error listing per-topic frame counts. Check that image topics are correct and timestamps overlap within `sync_threshold_ns`. From d3c64de01782b84dda3c0abd7f9bdce8147e1b71 Mon Sep 17 00:00:00 2001 From: Dmitry Slepichev Date: Fri, 23 Oct 2026 06:52:37 +0400 Subject: [PATCH 3/3] [clean] note unsynchronized cameras in README, check node defaults Review asked the README to say unsynchronized cameras are not supported, suggest separate trackers and link the TROUBLESHOOTING section; the sync bullet now does that. Review also asked whether the Isaac ROS defaults come from the node and whether the skill can check them. They are the declare_parameter values in release-4.3 visual_slam_node.cpp, except imu_frame, which defaults to "" there, not imu. The reference now names that source and tells the skill to check it for the user's release. Co-Authored-By: Claude Opus 5.5 --- README.md | 2 +- .../references/isaac-ros-visual-slam-usage.md | 5 ++++- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index ee2da3c8..efe55a60 100644 --- a/README.md +++ b/README.md @@ -118,7 +118,7 @@ The accuracy and robustness of cuVSLAM can be influenced by several factors. If - **Intrinsic and Extrinsic Calibration**: Accurate camera calibration is crucial. Ensure your calibration parameters are precise. For more details, refer to our guide on [image undistortion](examples/euroc/README.md#distortion-models). If you're new to calibration, consider working with [experienced vendors](https://nvidia-isaac-ros.github.io/v/release-3.2/getting_started/hardware_setup/sensors/amr_extrinsic_calibration.html#extrinsic-calibration-of-sensors-in-custom-locations) -- **Synchronization and Timestamps**: Accurate synchronization significantly impacts cuVSLAM performance. Make sure multi-camera images are captured simultaneously—ideally through hardware synchronization—and verify correct relative timestamps across cameras. Refer to our [multi-camera hardware assembly guide](examples/realsense/multicamera_hardware_assembly.md) for building a rig with synchronized RealSense cameras. Each `track()` call must carry images captured at one instant (within 1 ms); cycling through cameras or camera groups across calls is not supported +- **Synchronization and Timestamps**: Accurate synchronization significantly impacts cuVSLAM performance. Make sure multi-camera images are captured simultaneously—ideally through hardware synchronization—and verify correct relative timestamps across cameras. Refer to our [multi-camera hardware assembly guide](examples/realsense/multicamera_hardware_assembly.md) for building a rig with synchronized RealSense cameras. Each `track()` call must carry images captured at one instant (within 1 ms); cycling through cameras or camera groups across calls is not supported. Unsynchronized cameras are not supported either; see [Cameras that are not synchronized with each other](TROUBLESHOOTING.md#cameras-that-are-not-synchronized-with-each-other) for workarounds such as one tracker per camera - **Frame Rate**: Frame rate significantly affects performance. The ideal frame rate depends on translational and rotational velocities. Typically, 30 FPS is suitable for most "human-speed" motions. Adjust accordingly for faster movements diff --git a/cuvslam-skills/cuvslam-troubleshoot/references/isaac-ros-visual-slam-usage.md b/cuvslam-skills/cuvslam-troubleshoot/references/isaac-ros-visual-slam-usage.md index a0f41518..ec866631 100644 --- a/cuvslam-skills/cuvslam-troubleshoot/references/isaac-ros-visual-slam-usage.md +++ b/cuvslam-skills/cuvslam-troubleshoot/references/isaac-ros-visual-slam-usage.md @@ -56,6 +56,9 @@ ros2 launch isaac_ros_visual_slam isaac_ros_visual_slam_realsense_bag.launch.py ## Key Parameters +Defaults are the `declare_parameter` values in release-4.3 `visual_slam_node.cpp`; check them there for the user's +release before relying on one. + | Parameter | Default | Notes | |---|---|---| | `num_cameras` | `2` | Number of input cameras (max 32) | @@ -69,7 +72,7 @@ ros2 launch isaac_ros_visual_slam isaac_ros_visual_slam_realsense_bag.launch.py | `sync_matching_threshold_ms` | `5.0` | Grouping window only; `Odometry::Track` rejects images differing by 1 ms or more, so keep grouped timestamps under 1 ms | | `base_frame` | `base_link` | Robot base frame for TF | | `camera_optical_frames` | — | List of camera optical frame names (must match TF tree) | -| `imu_frame` | `imu` | IMU frame (VIO mode only) | +| `imu_frame` | `""` | IMU frame (VIO mode only) | | `save_map_folder_path` | `""` | Path to save SLAM map | | `load_map_folder_path` | `""` | Path to load a saved SLAM map |