Qwen 3 ASR

.gitignore

@@ -94,3 +94,8 @@ audio/
 TODO
 frames/
 .run/
+
+# Local Qwen3 validation and development helpers
+asr/qwen3/internal/
+asr/qwen3/model_export/_internal/
+asr/qwen3/tests/

CMakeLists.txt

@@ -100,6 +100,7 @@ add_subdirectory(image_gen/flux2)
 add_subdirectory(asr/rnnt)
 add_subdirectory(vision/sam2)
 add_subdirectory(asr/whisper)
+add_subdirectory(asr/qwen3)
 
 if(DIN_BUILD_TESTING)
     set(_din_cli_test_audio "${DIN_DEPLOY_ARTIFACTS_ROOT}/audio/de/schakaleundaraber_elli.mp3")

README.md

@@ -19,6 +19,8 @@ DIN Deploy is a collection of practical samples for exporting and running local
 | OpenAI Whisper | `openai/whisper-tiny`<br>`openai/whisper-base`<br>`openai/whisper-small`<br>`openai/whisper-medium`<br>`openai/whisper-large-v3`<br>`openai/whisper-large-v3-turbo` | [Whisper](asr/whisper/README.md) |
 | NVIDIA Parakeet TDT 0.6B v3 | `nvidia/parakeet-tdt-0.6b-v3` | [RNNT](asr/rnnt/README.md) |
 | NVIDIA Nemotron 3.5 ASR Streaming 0.6B | `nvidia/nemotron-3.5-asr-streaming-0.6b` | [RNNT](asr/rnnt/README.md) |
+| Qwen3 ASR | `Qwen/Qwen3-ASR-0.6B-hf`<br>`Qwen/Qwen3-ASR-1.7B-hf` | [Qwen3](asr/qwen3/README.md) |
+| Qwen3 Forced Aligner | `Qwen/Qwen3-ForcedAligner-0.6B-hf` | [Qwen3](asr/qwen3/README.md) |
 
 ### Computer Vision
 

THIRD_PARTY_NOTICES.md

@@ -8,12 +8,16 @@ DIN Deploy is distributed under the Apache License, Version 2.0. The project sou
 | miniaudio | MIT: <https://github.com/mackron/miniaudio/blob/0.11.23/LICENSE> |
 | lodepng | zlib: <https://github.com/lvandeve/lodepng/blob/master/LICENSE> |
 | nlohmann/json | MIT: <https://github.com/nlohmann/json/blob/develop/LICENSE.MIT> |
+| PCRE2 | BSD-3-Clause WITH PCRE2-exception ([notice](#pcre2-notice)): <https://github.com/PCRE2Project/pcre2/blob/pcre2-10.46/LICENCE.md> |
+| utf8proc | MIT and Unicode data license ([notices](#utf8proc-notices)): <https://github.com/JuliaStrings/utf8proc/blob/v2.11.0/LICENSE.md> |
 | NVIDIA NVTX | Apache-2.0 with LLVM exception: <https://github.com/NVIDIA/NVTX/blob/master/LICENSE> |
 | ONNX Runtime | MIT: <https://github.com/microsoft/onnxruntime/blob/main/LICENSE> |
 | ONNX Runtime TensorRT RTX EP ABI | Apache-2.0: <https://github.com/NVIDIA/TensorRT-RTX-EP-ABI/blob/main/LICENSE> |
 | Slang | Apache-2.0 with LLVM exception: <https://github.com/shader-slang/slang/blob/master/LICENSE> |
 | Vulkan-Headers and Vulkan-Loader | Apache-2.0: <https://github.com/KhronosGroup/Vulkan-Headers/blob/main/LICENSE.txt> and <https://github.com/KhronosGroup/Vulkan-Loader/blob/main/LICENSE.txt> |
 | nanobind | BSD-3-Clause: <https://github.com/wjakob/nanobind/blob/main/LICENSE> |
+| Qwen3-ASR inference utilities | Apache-2.0, copyright 2026 The Alibaba Qwen team: <https://github.com/QwenLM/Qwen3-ASR/blob/main/LICENSE> |
+| Transformers Qwen3-ASR processing | Apache-2.0, copyright Hugging Face: <https://github.com/huggingface/transformers/blob/v5.13.0/LICENSE> |
 | Python dependencies | See the package license links and inventory below. |
 | Distributed models and model artifacts | See the model license inventory below. Model terms may differ from this project's license. |
 
@@ -28,6 +32,8 @@ For components supplied through an SDK or binary package, the corresponding vend
 - `lodepng`: zlib
 - `nanobind` v2.9.2: BSD-3-Clause
 - `nlohmann/json` v3.11.3: MIT
+- `PCRE2` 10.46: BSD-3-Clause WITH PCRE2-exception
+- `utf8proc` 2.11.0: MIT and Unicode data license
 - NVIDIA NVTX v3.5.0 C/C++: Apache-2.0 WITH LLVM-exception
 - ONNX Runtime SDK 1.27.0: MIT
 - ONNX Runtime TensorRT RTX Execution Provider ABI: Apache-2.0
@@ -90,3 +96,213 @@ For components supplied through an SDK or binary package, the corresponding vend
 - `openai/whisper-large-v3-turbo`: Apache-2.0
 - `openai/whisper-medium`: Apache-2.0
 - `openai/whisper-small`: Apache-2.0
+
+- `Qwen/Qwen3-ASR-0.6B-hf`: Apache-2.0
+- `Qwen/Qwen3-ASR-1.7B-hf`: Apache-2.0
+- `Qwen/Qwen3-ForcedAligner-0.6B-hf`: Apache-2.0
+
+## PCRE2 notice
+
+```text
+PCRE2 License
+=============
+
+| SPDX-License-Identifier: | BSD-3-Clause WITH PCRE2-exception |
+|---------|-------|
+
+PCRE2 is a library of functions to support regular expressions whose syntax
+and semantics are as close as possible to those of the Perl 5 language.
+
+Releases 10.00 and above of PCRE2 are distributed under the terms of the "BSD"
+licence, as specified below, with one exemption for certain binary
+redistributions. The documentation for PCRE2, supplied in the "doc" directory,
+is distributed under the same terms as the software itself. The data in the
+testdata directory is not copyrighted and is in the public domain.
+
+The basic library functions are written in C and are freestanding. Also
+included in the distribution is a just-in-time compiler that can be used to
+optimize pattern matching. This is an optional feature that can be omitted when
+the library is built.
+
+
+COPYRIGHT
+---------
+
+### The basic library functions
+
+    Written by:       Philip Hazel
+    Email local part: Philip.Hazel
+    Email domain:     gmail.com
+
+    Retired from University of Cambridge Computing Service,
+    Cambridge, England.
+
+    Copyright (c) 1997-2007 University of Cambridge
+    Copyright (c) 2007-2024 Philip Hazel
+    All rights reserved.
+
+### PCRE2 Just-In-Time compilation support
+
+    Written by:       Zoltan Herczeg
+    Email local part: hzmester
+    Email domain:     freemail.hu
+
+    Copyright (c) 2010-2024 Zoltan Herczeg
+    All rights reserved.
+
+### Stack-less Just-In-Time compiler
+
+    Written by:       Zoltan Herczeg
+    Email local part: hzmester
+    Email domain:     freemail.hu
+
+    Copyright (c) 2009-2024 Zoltan Herczeg
+    All rights reserved.
+
+### All other contributions
+
+Many other contributors have participated in the authorship of PCRE2. As PCRE2
+has never required a Contributor Licensing Agreement, or other copyright
+assignment agreement, all contributions have copyright retained by each
+original contributor or their employer.
+
+
+THE "BSD" LICENCE
+-----------------
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+* Redistributions of source code must retain the above copyright notices,
+  this list of conditions and the following disclaimer.
+
+* Redistributions in binary form must reproduce the above copyright
+  notices, this list of conditions and the following disclaimer in the
+  documentation and/or other materials provided with the distribution.
+
+* Neither the name of the University of Cambridge nor the names of any
+  contributors may be used to endorse or promote products derived from this
+  software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+POSSIBILITY OF SUCH DAMAGE.
+
+
+EXEMPTION FOR BINARY LIBRARY-LIKE PACKAGES
+------------------------------------------
+
+The second condition in the BSD licence (covering binary redistributions) does
+not apply all the way down a chain of software. If binary package A includes
+PCRE2, it must respect the condition, but if package B is software that
+includes package A, the condition is not imposed on package B unless it uses
+PCRE2 independently.
+
+End
+```
+
+## utf8proc notices
+
+```text
+## utf8proc license ##
+
+**utf8proc** is a software package originally developed
+by Jan Behrens and the rest of the Public Software Group, who
+deserve nearly all of the credit for this library, that is now maintained by the Julia-language developers.  Like the original utf8proc,
+whose copyright and license statements are reproduced below, all new
+work on the utf8proc library is licensed under the [MIT "expat"
+license](http://opensource.org/licenses/MIT):
+
+*Copyright &copy; 2014-2021 by Steven G. Johnson, Jiahao Chen, Tony Kelman, Jonas Fonseca, and other contributors listed in the git history.*
+
+Permission is hereby granted, free of charge, to any person obtaining a
+copy of this software and associated documentation files (the "Software"),
+to deal in the Software without restriction, including without limitation
+the rights to use, copy, modify, merge, publish, distribute, sublicense,
+and/or sell copies of the Software, and to permit persons to whom the
+Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+DEALINGS IN THE SOFTWARE.
+
+## Original utf8proc license ##
+
+*Copyright (c) 2009, 2013 Public Software Group e. V., Berlin, Germany*
+
+Permission is hereby granted, free of charge, to any person obtaining a
+copy of this software and associated documentation files (the "Software"),
+to deal in the Software without restriction, including without limitation
+the rights to use, copy, modify, merge, publish, distribute, sublicense,
+and/or sell copies of the Software, and to permit persons to whom the
+Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+DEALINGS IN THE SOFTWARE.
+
+## Unicode data license ##
+
+This software contains data (`utf8proc_data.c`) derived from processing
+the Unicode data files. The following license applies to that data:
+
+**COPYRIGHT AND PERMISSION NOTICE**
+
+*Copyright (c) 1991-2007 Unicode, Inc. All rights reserved. Distributed
+under the Terms of Use in http://www.unicode.org/copyright.html.*
+
+Permission is hereby granted, free of charge, to any person obtaining a
+copy of the Unicode data files and any associated documentation (the "Data
+Files") or Unicode software and any associated documentation (the
+"Software") to deal in the Data Files or Software without restriction,
+including without limitation the rights to use, copy, modify, merge,
+publish, distribute, and/or sell copies of the Data Files or Software, and
+to permit persons to whom the Data Files or Software are furnished to do
+so, provided that (a) the above copyright notice(s) and this permission
+notice appear with all copies of the Data Files or Software, (b) both the
+above copyright notice(s) and this permission notice appear in associated
+documentation, and (c) there is clear notice in each modified Data File or
+in the Software as well as in the documentation associated with the Data
+File(s) or Software that the data or software has been modified.
+
+THE DATA FILES AND SOFTWARE ARE PROVIDED "AS IS", WITHOUT WARRANTY OF ANY
+KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OF
+THIRD PARTY RIGHTS. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR HOLDERS
+INCLUDED IN THIS NOTICE BE LIABLE FOR ANY CLAIM, OR ANY SPECIAL INDIRECT OR
+CONSEQUENTIAL DAMAGES, OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF
+USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER
+TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
+PERFORMANCE OF THE DATA FILES OR SOFTWARE.
+
+Except as contained in this notice, the name of a copyright holder shall
+not be used in advertising or otherwise to promote the sale, use or other
+dealings in these Data Files or Software without prior written
+authorization of the copyright holder.
+
+Unicode and the Unicode logo are trademarks of Unicode, Inc., and may be
+registered in some jurisdictions. All other trademarks and registered
+trademarks mentioned herein are the property of their respective owners.
+```

asr/qwen3/CMakeLists.txt

@@ -0,0 +1,17 @@
+# Independent ASR and forced-alignment APIs, sharing the model runtime.
+add_din_shared_library(din_asr_qwen3 STATIC qwen3.cpp forced_aligner.cpp)
+target_include_directories(din_asr_qwen3 PUBLIC "${CMAKE_CURRENT_SOURCE_DIR}")
+target_link_libraries(din_asr_qwen3 PUBLIC din_common_ort din_common_io nlohmann_json::nlohmann_json)
+find_package(CUDAToolkit QUIET)
+if(CUDAToolkit_FOUND)
+    target_link_libraries(din_asr_qwen3 PRIVATE CUDA::cudart)
+    target_compile_definitions(din_asr_qwen3 PRIVATE DIN_QWEN3_CUDA=1)
+endif()
+add_din_executable(din_asr_qwen3_cli main.cpp)
+target_link_libraries(din_asr_qwen3_cli PRIVATE din_asr_qwen3 argparse)
+add_din_executable(din_asr_qwen3_aligner_cli aligner_main.cpp)
+target_link_libraries(din_asr_qwen3_aligner_cli PRIVATE din_asr_qwen3 argparse)
+
+if(BUILD_TESTING AND EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/tests/CMakeLists.txt")
+    add_subdirectory(tests)
+endif()

asr/qwen3/README.md

@@ -0,0 +1,90 @@
+# Qwen3 ASR and forced alignment
+
+ONNX Runtime inference on CPU or TensorRT RTX, with independent ASR and alignment APIs.
+
+## Supported models
+
+| Model | Hugging Face ID |
+|---|---|
+| ASR 0.6B | `Qwen/Qwen3-ASR-0.6B-hf` |
+| ASR 1.7B | `Qwen/Qwen3-ASR-1.7B-hf` |
+| Forced Aligner 0.6B | `Qwen/Qwen3-ForcedAligner-0.6B-hf` |
+
+All models support BF16 (original precision, default), FP16 and FP32 exports.
+Use FP32 for CPU inference.
+
+## Supported capabilities
+
+| Model / upstream toolkit capability | C++ sample |
+|---|:---:|
+| Offline, single stream | ✓ |
+| Online / streaming, single utterance | ✓ |
+| Batched inference | — |
+| Long-form ASR | ✓ |
+| Automatic language detection / language hint | ✓ |
+| ASR: 30 languages and 22 Chinese dialects | ✓ |
+| Standalone alignment / alignment of any ASR output | ✓ |
+| Long-form alignment with timed transcript segments (C++ API) | ✓ |
+| Long-form alignment of unsegmented text | — |
+| Word timestamps: en, de, es, fr, it, pt, ru, ko | ✓ |
+| Chinese / Cantonese character timestamps | ✓ |
+| Automatic Japanese word timestamps | — |
+| Character timestamps: all 11 alignment languages (sample extension) | ✓ |
+| Caller-supplied alignment units | ✓ |
+
+See [upstream language support](https://github.com/QwenLM/Qwen3-ASR).
+Japanese alignment requires character mode or supplied units.
+
+## Export and validate
+
+Run commands from the repository root; install the export dependencies first.
+
+```bash
+pip install -r asr/qwen3/requirements.txt
+python asr/qwen3/model_export/export_qwen3_asr.py --output models/qwen3-asr
+python asr/qwen3/model_export/export_qwen3_asr.py --task aligner --output models/qwen3-aligner
+```
+
+HF downloads checkpoints automatically. Use `--size 1.7B` for the larger ASR model,
+or `--dtype fp16` / `--dtype fp32` to change precision; use a separate output directory.
+Each ASR export contains one encoder, one decoder and the shared log-mel graph.
+
+```bash
+python asr/qwen3/model_export/validate_qwen3_asr.py --onnx-dir models/qwen3-asr --audio audio.wav
+python asr/qwen3/model_export/validate_qwen3_asr.py --task aligner --onnx-dir models/qwen3-aligner --audio audio.wav --transcript transcript.txt --language English
+```
+
+## Build and run
+
+Follow the [repository build setup](../../README.md). Replace `<build>` and
+`<build/bin>` with your build and executable directories; append `.exe` on Windows.
+
+```text
+cmake --build <build> --config Release --target din_asr_qwen3_cli din_asr_qwen3_aligner_cli
+<build/bin>/din_asr_qwen3_cli audio.wav --model-dir models/qwen3-asr
+<build/bin>/din_asr_qwen3_aligner_cli audio.wav --model-dir models/qwen3-aligner --transcript transcript.txt --lang-id en
+```
+
+Both CLIs accept `--provider cpu|trt-rtx` (default: trt-rtx).
+Use `--granularity characters` for character alignment or `--units` for one supplied
+alignment unit per transcript line. Character timestamps have 80 ms resolution.
+Japanese word segmentation can be an optional external preprocessing pass supplied through `AlignUnits` / `--units`.
+
+## Long-form and streaming
+
+Long recordings are split automatically. `--max-new-tokens` defaults to 1024 per
+chunk; `reached_eos=false` means the transcript is incomplete. Increase the budget
+or reduce `--max-chunk-seconds`. Export-time `--cache-capacity` defaults to 8192 tokens.
+
+Add `--stream` for streaming transcription; `- --stream` reads mono 16 kHz float32
+PCM from stdin. Outputs are replacement hypotheses, not incremental text.
+Streaming reprocesses the current utterance, so latency grows with its length;
+start a new stream before exceeding the exported context capacity.
+
+## C++ integration
+
+Use `Qwen3Pipeline` ([qwen3.h](qwen3.h)) for transcription and
+`Qwen3ForcedAligner` ([forced_aligner.h](forced_aligner.h)) for text from any ASR.
+`AlignSegments` accepts timed transcript segments and returns recording-relative
+timestamps. Alignment is limited to 180 seconds, 2048 units and 8192 context tokens
+per segment; use `--max-chunk-seconds 175` when transcribing for subsequent alignment.