diff --git a/README.md b/README.md index 2968ff5a..efe55a60 100644 --- a/README.md +++ b/README.md @@ -118,7 +118,7 @@ The accuracy and robustness of cuVSLAM can be influenced by several factors. If - **Intrinsic and Extrinsic Calibration**: Accurate camera calibration is crucial. Ensure your calibration parameters are precise. For more details, refer to our guide on [image undistortion](examples/euroc/README.md#distortion-models). If you're new to calibration, consider working with [experienced vendors](https://nvidia-isaac-ros.github.io/v/release-3.2/getting_started/hardware_setup/sensors/amr_extrinsic_calibration.html#extrinsic-calibration-of-sensors-in-custom-locations) -- **Synchronization and Timestamps**: Accurate synchronization significantly impacts cuVSLAM performance. Make sure multi-camera images are captured simultaneously—ideally through hardware synchronization—and verify correct relative timestamps across cameras. Refer to our [multi-camera hardware assembly guide](examples/realsense/multicamera_hardware_assembly.md) for building a rig with synchronized RealSense cameras +- **Synchronization and Timestamps**: Accurate synchronization significantly impacts cuVSLAM performance. Make sure multi-camera images are captured simultaneously—ideally through hardware synchronization—and verify correct relative timestamps across cameras. Refer to our [multi-camera hardware assembly guide](examples/realsense/multicamera_hardware_assembly.md) for building a rig with synchronized RealSense cameras. Each `track()` call must carry images captured at one instant (within 1 ms); cycling through cameras or camera groups across calls is not supported. Unsynchronized cameras are not supported either; see [Cameras that are not synchronized with each other](TROUBLESHOOTING.md#cameras-that-are-not-synchronized-with-each-other) for workarounds such as one tracker per camera - **Frame Rate**: Frame rate significantly affects performance. The ideal frame rate depends on translational and rotational velocities. Typically, 30 FPS is suitable for most "human-speed" motions. Adjust accordingly for faster movements diff --git a/TROUBLESHOOTING.md b/TROUBLESHOOTING.md index 786fe20b..91b337dc 100644 --- a/TROUBLESHOOTING.md +++ b/TROUBLESHOOTING.md @@ -41,6 +41,7 @@ See [Requirements](README.md#performance). - [Try to use rectified images if possible](#try-to-use-rectified-images-if-possible) - [Disable SLAM and ground constraint](#disable-slam-and-ground-constraint) - [Step 6: Verify synchronization](#step-6-verify-synchronization) + - [Cameras that are not synchronized with each other](#cameras-that-are-not-synchronized-with-each-other) - [Step 7: Run stereo odometry tracking](#step-7-run-stereo-odometry-tracking) - [Step 8: Adjust odometry settings](#step-8-adjust-odometry-settings) - [Improve input images](#improve-input-images) @@ -150,8 +151,9 @@ cuvslam::Odometry::Config::max_frame_delta_s image_jitter_threshold_ms ``` -In the Isaac ROS cuVSLAM node, there is also a synchronizer that can align multiple image streams with non-equal -timestamps. Check Isaac ROS documentation and tune: +In the Isaac ROS cuVSLAM node, there is also a synchronizer that groups images whose timestamps are within +`sync_matching_threshold_ms` into one frame; it cannot synchronize free-running cameras. Check Isaac ROS documentation +and tune: ``` num_cameras @@ -351,6 +353,20 @@ Example: Left and right eyes out of sync ![Example: Left and right eyes out of sync](doc/images/troubleshooting_desync.gif) +### Cameras that are not synchronized with each other + +Each `track()` call must carry images captured at one instant (within 1 ms). A camera may be missing from a call, +for example after a frame drop, but cycling through cameras or camera groups across calls is not supported, and neither +is filling an empty slot with an older image or giving images from different instants one timestamp. cuVSLAM matches +features across cameras only within a call. `track()` rejects a call whose image timestamps differ by 1 ms or more, +but when such input passes this check, tracking accuracy is not guaranteed. + +If your cameras are not synchronized: + +1. Synchronize them in hardware (external trigger). +2. If they mostly line up, pass only frame sets whose images are within 1 ms of each other and drop the rest. +3. Otherwise, run one tracker per camera (or per synchronized stereo pair) and fuse their poses yourself. + ## Step 7: Run stereo odometry tracking If mono left and mono right both work, run stereo tracking. Stereo reuses extrinsic calibration via the diff --git a/cuvslam-skills/cuvslam-troubleshoot/SKILL.md b/cuvslam-skills/cuvslam-troubleshoot/SKILL.md index 0885b8e8..42743194 100644 --- a/cuvslam-skills/cuvslam-troubleshoot/SKILL.md +++ b/cuvslam-skills/cuvslam-troubleshoot/SKILL.md @@ -90,7 +90,7 @@ Key parameters to note once you have the config: cuNLS availability, `depth_camera_ids`, and pinhole calibration. Knowing this up front directs the diagnosis toward vision, depth, or IMU root causes. - `rectified_images` (Isaac ROS) / `rectified_stereo_camera` (C++ API) — wrong value flips the entire stereo pipeline -- `image_jitter_threshold_ms` — tolerated timestamp jitter between left/right images; too tight a value causes frames to be dropped silently +- `image_jitter_threshold_ms` — tolerated gap between consecutive frames (exceeding it only logs a warning), not a tolerance between cameras - `Tracker.Mode` plus the `async_sba` / `sync_mode` that must match it — affects reproducibility - `enable_slam` / loop closure settings — SLAM should only be tuned after odometry is solid - `denoise_input_images`, `imu_from_left` — sensor-specific flags @@ -169,7 +169,7 @@ With the config and dataset in hand, verify these before touching the diagnostic 1. **Accurate intrinsic calibration** — verify distortion model and params match the actual camera. See `references/edex-calibration-pitfalls.md` for common EDEX mistakes (e.g. `"fisheye"` with zero params ≠ `"pinhole"`). 2. **Accurate extrinsic calibration** (stereo/multi-cam) — verify baseline axis and magnitude match the physical rig. See `references/edex-calibration-pitfalls.md` for wrong-axis baseline diagnosis. -3. **Proper image synchronization** — left/right timestamps must match per frame. +3. **Proper image synchronization** — each `track()` call carries images captured at one instant (within 1 ms), never cameras cycled across calls. 4. **Sharp images, low noise** — frame rate matters more than resolution. 5. **Textured, rigid scenes** — avoid repetitive patterns and dynamic objects. 6. **Correct parameters and coordinate frames** — check `rectified_stereo_camera` flag. diff --git a/cuvslam-skills/cuvslam-troubleshoot/commands/bag2edex.md b/cuvslam-skills/cuvslam-troubleshoot/commands/bag2edex.md index 081539b3..9aef446d 100644 --- a/cuvslam-skills/cuvslam-troubleshoot/commands/bag2edex.md +++ b/cuvslam-skills/cuvslam-troubleshoot/commands/bag2edex.md @@ -225,7 +225,7 @@ If any topic prints a frame_id belonging to a different camera, set `camera_opti **Wrong intrinsics** — intrinsics are read from `camera_info` messages in the bag. Verify the published `camera_info` matches the actual camera before converting. -**Mismatched frame counts** — only synchronized frame pairs (within `sync_threshold_ns`) are saved. Increase `sync_threshold_ns` in the config if too many frames are dropped. +**Mismatched frame counts** — a frame set is saved only if its image timestamps differ by less than `sync_threshold_ns` (default 1 ms); a difference equal to or greater than the threshold is rejected. Don't raise it to keep more frames: images 1 ms or more apart are not one capture instant. **No synchronized frames found** — the tool exits with an error listing per-topic frame counts. Check that image topics are correct and timestamps overlap within `sync_threshold_ns`. diff --git a/cuvslam-skills/cuvslam-troubleshoot/references/isaac-ros-visual-slam-usage.md b/cuvslam-skills/cuvslam-troubleshoot/references/isaac-ros-visual-slam-usage.md index 35624de9..ec866631 100644 --- a/cuvslam-skills/cuvslam-troubleshoot/references/isaac-ros-visual-slam-usage.md +++ b/cuvslam-skills/cuvslam-troubleshoot/references/isaac-ros-visual-slam-usage.md @@ -56,6 +56,9 @@ ros2 launch isaac_ros_visual_slam isaac_ros_visual_slam_realsense_bag.launch.py ## Key Parameters +Defaults are the `declare_parameter` values in release-4.3 `visual_slam_node.cpp`; check them there for the user's +release before relying on one. + | Parameter | Default | Notes | |---|---|---| | `num_cameras` | `2` | Number of input cameras (max 32) | @@ -65,11 +68,11 @@ ros2 launch isaac_ros_visual_slam isaac_ros_visual_slam_realsense_bag.launch.py | `enable_localization_n_mapping` | `true` | `false` = odometry only, no loop closure | | `enable_image_denoising` | `false` | Enable for low-light/noisy cameras | | `enable_ground_constraint_in_slam` | `false` | Constrain to horizontal plane for wheeled robots | -| `image_jitter_threshold_ms` | — | Tolerated timestamp jitter between left/right images | -| `sync_matching_threshold_ms` | — | Tolerated timestamp offset across cameras for sync | +| `image_jitter_threshold_ms` | `34.0` | Tolerated gap between consecutive frames; exceeding it only logs a warning | +| `sync_matching_threshold_ms` | `5.0` | Grouping window only; `Odometry::Track` rejects images differing by 1 ms or more, so keep grouped timestamps under 1 ms | | `base_frame` | `base_link` | Robot base frame for TF | | `camera_optical_frames` | — | List of camera optical frame names (must match TF tree) | -| `imu_frame` | `imu` | IMU frame (VIO mode only) | +| `imu_frame` | `""` | IMU frame (VIO mode only) | | `save_map_folder_path` | `""` | Path to save SLAM map | | `load_map_folder_path` | `""` | Path to load a saved SLAM map | diff --git a/libs/cuvslam/cuvslam2.h b/libs/cuvslam/cuvslam2.h index 3a626244..5baec1ba 100644 --- a/libs/cuvslam/cuvslam2.h +++ b/libs/cuvslam/cuvslam2.h @@ -593,13 +593,13 @@ class CUVSLAM_API Odometry { * * The track will output poses in the same coordinate system until a loss of tracking. * - * Image timestamps have to match. cuVSLAM will use timestamp from the camera 0 image. + * Image timestamps have to match. cuVSLAM will use the timestamp of images[0]. * If a camera rig provides "almost synchronized" frames, the timestamps should be within 1 millisecond. * The number of images for tracker should not exceed rig->num_cameras. * * @param[in] images an array of synchronized images no more than rig->num_cameras. - * Must use ImageData::DataType::UINT8. Partial ImageSet is supported, for example due to frame drops. Corresponding - * cameras are identified by Image::camera_index. + * Must use ImageData::DataType::UINT8. Partial ImageSet is supported, for example due to frame drops, but cycling + * through cameras or camera groups across calls is not. Corresponding cameras are identified by Image::camera_index. * @param[in] masks (Optional) an array of corresponding masks no more than rig->num_cameras. * Must use ImageData::DataType::UINT8 and ImageData::Encoding::MONO. Non-zero pixels are excluded: no features are * selected or kept there, and in OdometryMode::RGBD and OdometryMode::Multisensor the depth at those pixels is diff --git a/python/cuvslam2.cpp b/python/cuvslam2.cpp index 7e7414fb..f36cc6fc 100644 --- a/python/cuvslam2.cpp +++ b/python/cuvslam2.cpp @@ -716,7 +716,8 @@ NB_MODULE(pycuvslam, m) { "images etc. can still be either CPU or GPU arrays/tensors. The reverse does not hold:\n" "odometry with `use_gpu` disabled only accepts CPU arrays/tensors.\n\n" "The images etc. must be in the same order as cameras in the rig.\n" - "If data for a camera is not available, pass empty array or tensor for that camera image.\n\n" + "If data for a camera is not available, pass empty array or tensor for that camera image.\n" + "Cycling through cameras or camera groups across calls is not supported.\n\n" "Parameters:\n" " timestamp: Image timestamp in nanoseconds\n" " images: List of numpy arrays containing the camera images\n" @@ -726,7 +727,7 @@ NB_MODULE(pycuvslam, m) { " depths: Optional list of numpy arrays containing depth images\n" " options: Optional per-frame options that override default settings for this call only\n\n" "Returns:\n" - " :class:`PoseEstimate` object with the computed pose. If tracking fails, `is_valid` will be False.") + " :class:`PoseEstimate` object with the computed pose. If tracking fails, `world_from_rig` is `None`.") .def( "register_imu_measurement", [](Odometry& self, uint32_t sensor_index, const ImuMeasurement& imu_measurement) { @@ -1287,7 +1288,8 @@ NB_MODULE(pycuvslam, m) { "images etc. can still be either CPU or GPU arrays/tensors. The reverse does not hold:\n" "odometry with `use_gpu` disabled only accepts CPU arrays/tensors.\n\n" "The images etc. must be in the same order as cameras in the rig.\n" - "If data for a camera is not available, pass empty array or tensor for that camera image.\n\n" + "If data for a camera is not available, pass empty array or tensor for that camera image.\n" + "Cycling through cameras or camera groups across calls is not supported.\n\n" "Parameters:\n" " timestamp: Images timestamp in nanoseconds\n" " images: List of numpy arrays or tensors containing the camera images\n"