usls

usls is a cross-platform Rust library powered by ONNX Runtime for efficient inference of SOTA vision and vision-language models (typically under 1B parameters).

(Generated by Seedream4.5)

🌟 Highlights

• ⚡ High Performance: Multi-threading, SIMD, and CUDA-accelerated processing
• ✨ Cross-Platform: Linux, macOS, Windows with ONNX Runtime execution providers (CUDA, TensorRT, CoreML, OpenVINO, DirectML, etc.)
• 🎯 Precision Support: FP32, FP16, INT8, UINT8, Q4, Q4F16, BNB4, and more
• 🛠️ Full-Stack Suite: DataLoader, Annotator, and Viewer for complete workflows
• 🏗️ Unified API: Single Model trait inference with run()/forward()/encode_images()/encode_texts() and unified Y output
• 📥 Auto-Management: Automatic model download (HuggingFace/GitHub), caching and path resolution
• 📦 Multiple Inputs: Image, directory, video, webcam, stream and combinations
• 🌱 Model Ecosystem: 50+ SOTA vision and VLM models

🚀 Quick Start

Run the YOLO-Series demo to explore models with different tasks, precision and execution providers:

• Tasks: detect, segment, pose, classify, obb
• Versions: v5, v6, v7, v8, v9, v10, 11, 12, v13, 26
• Scales: n, s, m, l, x
• Precision: fp32, fp16, q8, int8, q4, q4f16, bnb4, and more
• Execution Providers: CPU, CUDA, TensorRT, TensorRT-RTX, CoreML, OpenVINO, and more

CPU

cargo run -r --example yolo -- --task detect --ver 26 --scale n --dtype fp16

Nvidia CUDA + CUDA Image Processor

cargo run -r -F cuda --example yolo -- --task segment --ver 11 --scale m --device cuda:0 --processor-device cuda:0

Nvidia TensorRT + CUDA Image Processor

cargo run -r -F tensorrt-full --example yolo -- --device tensorrt:0 --processor-device cuda:0

Nvidia TensorRT-RTX + CUDA Image Processor

cargo run -r -F nvrtx-full --example yolo -- --device nvrtx:0 --processor-device cuda:0

Apple Silicon CoreML

cargo run -r -F coreml --example yolo -- --device coreml

Intel OpenVINO (CPU/GPU/VPU)

cargo run -r -F openvino -F ort-load-dynamic --example yolo -- --device openvino:CPU

📊 Performance Benchmarks

Environment: NVIDIA RTX 3060Ti (TensorRT-10.11.0.33, CUDA 12.8, TensorRT-RTX-1.3.0.35) / Intel i5-12400F

Setup: YOLO26 Detection, COCO2017-val (5,000 images), 640x640, Conf thresholds: [0.35, 0.3, ..]

Results are for rough reference only.

Scale	EP	Image Processor	DType	Batch	Preprocess	Inference	Postprocess	Total
n	TensorRT	CUDA	FP16	1	~233µs	~1.3ms	~14µs	~1.55ms
n	TensorRT-RTX	CUDA	FP32	1	~233µs	~2.0ms	~10µs	~2.24ms
n	TensorRT-RTX	CUDA	FP16	1	❓	❓	❓	❓
n	CUDA	CUDA	FP32	1	~233µs	~5.0ms	~17µs	~5.25ms
n	CUDA	CUDA	FP16	1	~233µs	~3.6ms	~17µs	~3.85ms
n	CUDA	CPU	FP32	1	~800µs	~6.5ms	~14µs	~7.31ms
n	CUDA	CPU	FP16	1	~800µs	~5.0ms	~14µs	~5.81ms
n	CPU	CPU	FP32	1	~970µs	~20.5ms	~14µs	~21.48ms
n	CPU	CPU	FP16	1	~970µs	~25.0ms	~14µs	~25.98ms
n	TensorRT	CUDA	FP16	8	~1.2ms	~6.0ms	~55µs	~7.26ms
n	TensorRT	CPU	FP16	8	~18.0ms	~25.5ms	~55µs	~43.56ms
m	TensorRT	CUDA	FP16	1	~233µs	~3.6ms	~14µs	~3.85ms
m	TensorRT	CUDA	Int8	1	~233µs	~2.6ms	~14µs	~2.84ms
m	CUDA	CUDA	FP32	1	~233µs	~16.1ms	~17µs	~16.35ms
m	CUDA	CUDA	FP16	1	~233µs	~8.8ms	~17µs	~9.05ms

🔍 What's Next?

• 📖 Online Documentation
• 📚 API Reference
• 📦 Model Zoo
• 🚀 Examples

📦 Model Zoo

Note

Status: ✅ Supported  |  ❓ Unknown  |  ❌ Not Supported For Now

🔍 All ONNX models are available from the ONNX Models Repository

🔥 YOLO-Series

Model	Task / Description	Demo	Dynamic Batch	TensorRT	FP32	FP16	Q8	Q4f16	BNB4
YOLOv5	Image Classification Object Detection Instance Segmentation	demo	✅	✅	✅	✅	✅	❌	❌
YOLOv6	Object Detection	demo	✅	✅	✅	✅	✅	❌	❌
YOLOv7	Object Detection	demo	✅	✅	✅	✅	✅	❌	❌
YOLOv8	Object Detection Instance Segmentation Image Classification Oriented Object Detection Keypoint Detection	demo	✅	✅	✅	✅	✅	❌	❌
YOLO11	Object Detection Instance Segmentation Image Classification Oriented Object Detection Keypoint Detection	demo	✅	✅	✅	✅	✅	❌	❌
YOLOv9	Object Detection	demo	✅	✅	✅	✅	✅	❌	❌
YOLOv10	Object Detection	demo	✅	✅	✅	✅	✅	❌	❌
YOLOv12	Image Classification Object Detection Instance Segmentation	demo	✅	✅	✅	✅	✅	✅	✅
YOLOv13	Object Detection	demo	✅	✅	✅	✅	✅	✅	✅
YOLO26	Object Detection Instance Segmentation Image Classification Oriented Object Detection Keypoint Detection	demo	✅	✅	✅	✅	✅	✅	✅

🏷️ Image Classification & Tagging

Model	Task / Description	Demo	Dynamic Batch	TensorRT	FP32	FP16	Q8	Q4f16	BNB4
BEiT	Image Classification	demo	✅	✅	✅	✅	❌	❌	❌
ConvNeXt	Image Classification	demo	✅	✅	✅	✅	❌	❌	❌
FastViT	Image Classification	demo	✅	✅	✅	✅	❌	❌	❌
MobileOne	Image Classification	demo	✅	✅	✅	✅	❌	❌	❌
DeiT	Image Classification	demo	✅	✅	✅	✅	❌	❌	❌
RAM	Image Tagging	demo	✅	❓	✅	✅	✅	✅	✅
RAM++	Image Tagging	demo	✅	❓	✅	✅	✅	✅	✅

🎯 Object Detection

Model	Task / Description	Demo	Dynamic Batch	TensorRT	FP32	FP16	Q8	Q4f16	BNB4
RT-DETRv1	Object Detection	demo	✅	✅	✅	✅	✅	✅	✅
RT-DETRv2	Object Detection	demo	✅	✅	✅	✅	✅	✅	✅
RT-DETRv4	Object Detection	demo	✅	✅	✅	✅	✅	✅	✅
RF-DETR	Object Detection	demo	✅	✅	✅	✅	✅	✅	✅
PP-PicoDet	Object Detection	demo	❌	❓	✅	❌	❌	❌	❌
D-FINE	Object Detection	demo	✅	❓	✅	❌	❌	❌	❌
DEIM	Object Detection	demo	✅	❓	✅	❌	❌	❌	❌
DEIMv2	Object Detection	demo	✅	❓	✅	✅	✅	✅	✅

🎨 Image Segmentation

Model	Task / Description	Demo	Dynamic Batch	TensorRT	FP32	FP16	Q8	Q4f16	BNB4
SAM	Segment Anything	demo	✅	❓	✅	❌	❌	❌	❌
SAM-HQ	Segment Anything	demo	✅	❓	✅	❌	❌	❌	❌
MobileSAM	Segment Anything	demo	✅	❓	✅	❌	❌	❌	❌
EdgeSAM	Segment Anything	demo	✅	❓	✅	❌	❌	❌	❌
YOLOE-v8/11-Prompt-Free	Open-Set Detection And Segmentation	demo	✅	✅	✅	✅	✅	✅	✅
YOLOE-26-Prompt-Free	Open-Set Detection And Segmentation	demo	✅	✅	✅	✅	✅	✅	✅
FastSAM	Instance Segmentation	demo	✅	✅	✅	✅	✅	✅	✅
EdgeCrafter (ECDetSeg)	Instance Segmentation	demo	❓	✅	✅	✅	❌	❌	❌
SAM2	Segment Anything	demo	✅	❓	✅	❌	❌	❌	❌
SAM3-Tracker	Segment Anything	demo	✅	✅	✅	✅	✅	✅	✅
BiRefNet - COD	Camouflaged Object Detection	demo	✅	❓	✅	✅	✅	✅	✅
BiRefNet - DIS	Dichotomous Image Segmentation	demo	✅	❓	✅	✅	✅	✅	✅
BiRefNet - HRSOD	High-Resolution Salient Object Detection	demo	✅	❓	✅	✅	✅	✅	✅
BiRefNet - Massive	Multi-Dataset Robust Segmentation	demo	✅	❓	✅	✅	✅	✅	✅

✨ Background Removal

Model	Task / Description	Demo	Dynamic Batch	TensorRT	FP32	FP16	Q8	Q4f16	BNB4
RMBG	Image Segmentation Background Removal	demo	✅	❓	✅	✅	✅	✅	✅
BEN2	Image Segmentation Background Removal	demo	✅	❓	✅	✅	❌	❌	❌

👀 Gaze Estimation

Model	Task / Description	Demo	Dynamic Batch	TensorRT	FP32	FP16	Q8	Q4f16	BNB4
MobileGaze	Eye Gaze Estimation	demo	✅	❓	✅	✅	✅	✅	✅

✂️ Image Matting & Portrait Segmentation

Model	Task / Description	Demo	Dynamic Batch	TensorRT	FP32	FP16	Q8	Q4f16	BNB4
MODNet	Image Matting	demo	✅	❓	✅	✅	✅	❌	❌
MediaPipe Selfie	Image Segmentation	demo	✅	❓	✅	✅	✅	❌	❌
BiRefNet - Portrait	Portrait Background Removal	demo	✅	❓	✅	✅	✅	✅	✅
BiRefNet - Matting	Portrait Matting & Background Removal	demo	✅	❓	✅	✅	✅	✅	✅
BiRefNet - HR Matting	High-Resolution Portrait Matting	demo	✅	❓	✅	✅	✅	✅	✅
BiRefNet - General	General Purpose Segmentation	demo	✅	❓	✅	✅	✅	✅	✅
BiRefNet - HR General	High-Resolution General Segmentation	demo	✅	❓	✅	✅	✅	✅	✅
BiRefNet - Lite General	Lightweight General Segmentation (2K)	demo	✅	❓	✅	✅	✅	✅	✅
BiRefNet - General Tiny	Lightweight General Segmentation with Swin-V1-Tiny	demo	✅	❓	✅	✅	✅	✅	✅

🗺️ Open-Set Detection & Segmentation

Model	Task / Description	Demo	Dynamic Batch	TensorRT	FP32	FP16	Q8	Q4f16	BNB4
GroundingDINO	Open-Set Detection With Language	demo	✅	❓	✅	✅	✅	✅	✅
MM-GDINO	Open-Set Detection With Language	demo	✅	❓	✅	✅	✅	✅	✅
LLMDet	Open-Set Detection With Language	demo	✅	❓	✅	✅	✅	✅	✅
OWLv2	Open-Set Object Detection	demo	✅	❓	✅	✅	❌	❌	❌
YOLO-World	Open-Set Detection With Language	demo	✅	✅	✅	✅	✅	✅	✅
YOLOE-Prompt-Based	Open-Set Detection And Segmentation	demo	✅	✅	✅	✅	✅	✅	✅
YOLOE-26-Prompt-Based	Open-Set Detection And Segmentation	demo	✅	✅	✅	✅	✅	✅	✅
SAM3-Image	Open-Set Detection And Segmentation	demo	✅	✅	✅	✅	✅	✅	✅

🏃 Multi-Object Tracking

Model	Task / Description	Demo	Dynamic Batch	TensorRT	FP32	FP16	Q8	Q4f16	BNB4
ByteTrack	Multi-Object Tracking	demo	❌	❌	❌	❌	❌	❌	❌

💎 Image Super-Resolution

Model	Task / Description	Demo	Dynamic Batch	TensorRT	FP32	FP16	Q8	Q4f16	BNB4
Swin2SR	Image Restoration	demo	✅	❓	✅	✅	✅	✅	✅
APISR	Anime Super-Resolution	demo	✅	❓	✅	✅	✅	✅	✅

🤸 Pose Estimation

Model	Task / Description	Demo	Dynamic Batch	TensorRT	FP32	FP16	Q8	Q4f16	BNB4
RTMPose	Keypoint Detection	demo	✅	❓	✅	✅	✅	✅	✅
DWPose	Keypoint Detection	demo	✅	❓	✅	✅	✅	✅	✅
RTMW	Keypoint Detection	demo	✅	❓	✅	✅	✅	✅	✅
RTMO	Keypoint Detection	demo	✅	❓	✅	✅	✅	✅	❌
ECPose	Keypoint Detection	demo	✅	❓	✅	✅	❌	❌	❌

🔍 OCR & Document Understanding

Model	Task / Description	Demo	Dynamic Batch	TensorRT	FP32	FP16	Q8	Q4f16	BNB4
DB (PaddleOCR det v4 / v5)	Text Detection	demo	✅	❓	✅	✅	❌	❌	❌
FAST	Text Detection	demo	✅	❓	✅	✅	❌	❌	❌
LinkNet	Text Detection	demo	✅	❓	✅	✅	❌	❌	❌
SVTR (PaddleOCR rec v4 / v5)	Text Recognition	demo	✅	❓	✅	✅	❌	❌	❌
TrOCR	Text Recognition	demo	✅	❓	✅	✅	❌	❌	❌
SLANet (PaddleOCR tab v4 / v5)	Table Recognition	demo	✅	❓	✅	✅	❌	❌	❌
DocLayout-YOLO	Object Detection	demo	✅	❓	✅	✅	✅	❌	❌
PP-DocLayout-v1-Plus-L	Object Detection	demo	✅	✅	✅	✅	✅	❌	❌
PP-DocLayout-v2	Object Detection	demo	✅	❓	✅	✅	✅	✅	✅
PP-DocLayout-v3	Object Detection	demo	✅	✅	✅	✅	✅	✅	✅

🧩 Vision-Language Models (VLM)

Model	Task / Description	Demo	Dynamic Batch	TensorRT	FP32	FP16	Q8	Q4f16	BNB4
BLIP	Image Captioning	demo	✅	❓	✅	❓	❌	❌	❌
Florence2	A Variety of Vision Tasks	demo	✅	❓	✅	✅	❌	❌	❌
Moondream2	Open-Set Object Detection Open-Set Keypoints Detection Image Captioning Visual Question Answering	demo	✅	❓	❌	❌	✅	✅	❌
SmolVLM	Visual Question Answering	demo	✅	❓	✅	❓	❓	❓	❓
SmolVLM2	Visual Question Answering	demo	✅	❓	✅	❓	❓	❓	❓
FastVLM	Vision Language Models	demo	✅	❓	✅	✅	✅	✅	✅

🧬 Embedding Model

Model	Task / Description	Demo	Dynamic Batch	TensorRT	FP32	FP16	Q8	Q4f16	BNB4
CLIP	Vision-Language Embedding	demo	✅	❓	✅	✅	✅	✅	✅
jina-clip-v1	Vision-Language Embedding	demo	✅	❓	✅	✅	✅	✅	✅
jina-clip-v2	Vision-Language Embedding	demo	✅	❓	✅	✅	✅	✅	✅
mobileclip	Vision-Language Embedding	demo	✅	❓	✅	✅	✅	✅	✅
SigLIP	Vision-Language Embedding	demo	✅	❓	✅	✅	✅	✅	✅
SigLIPv2	Vision-Language Embedding	demo	✅	❓	✅	✅	✅	✅	✅
DINOv2	Vision Embedding	demo	✅	❓	✅	❌	❌	❌	❌
DINOv3	Vision Embedding	demo	✅	❓	✅	✅	✅	✅	✅

📐 Depth Estimation

Model	Task / Description	Demo	Dynamic Batch	TensorRT	FP32	FP16	Q8	Q4f16	BNB4
DepthAnything v1	Monocular Depth Estimation	demo	✅	❓	✅	✅	✅	✅	✅
DepthAnything v2	Monocular Depth Estimation	demo	✅	❓	✅	✅	✅	✅	✅
DepthPro	Monocular Depth Estimation	demo	✅	❓	✅	✅	✅	✅	✅
Depth-Anything-3	Monocular Metric Multi-View	demo	✅	❓	✅	✅	✅	✅	✅

🌌 Others

Model	Task / Description	Demo	Dynamic Batch	TensorRT	FP32	FP16	Q8	Q4f16	BNB4
Sapiens	Foundation for Human Vision Models	demo	✅	❓	✅	✅	✅	✅	✅
YOLOPv2	Panoptic Driving	demo	✅	❓	✅	❌	❌	❌	❌

🤝 Contributing

This is a personal project maintained in spare time, so progress on performance optimization and new model support may vary.

We highly welcome PRs for model optimization! If you have expertise in specific models and can help optimize their interfaces or post-processing, your contributions would be invaluable. Feel free to open an issue or submit a pull request for suggestions, bug reports, or new features.

🙏 Acknowledgments

• This project is built on top of ort (ONNX Runtime for Rust), which provides seamless Rust bindings for ONNX Runtime. Special thanks to the ort maintainers.

• Special thanks to @kadu-v for the jamtrack-rs project, which inspired our ByteTracker implementation

Thanks to all the open-source libraries and their maintainers that make this project possible. See Cargo.toml for a complete list of dependencies.

📜 License

This project is licensed under LICENSE.