[krishung5]

Description

Add audio inputs to the OpenEngine gRPC Generate path.

• Advertise audio only when the loaded model and checkpoint support it, including the audio-encoder requirement for Gemma4.
• Map audio URLs, data URIs, and raw bytes while preserving media order and forwarding caller UUIDs.
• Count decoded audio sample storage toward the aggregate media memory limit.
• Cover request mapping, capability gates, media loading, session cleanup, and real-weight Whisper transcription.

Stacked on video PR #20654. The image-input prerequisite #20446 is merged into main. This PR does not depend on image performance #21037 or image routing #21125.

UUID forwarding is covered by mapper tests. Engine-side UUID use depends on the model supporting multimodal hashing; Whisper disables that path, so its conformance test does not claim UUID/cache-identity coverage.

Test Coverage

Local validation of branch refresh 8fb1e663d:

• 361 focused CPU tests passed, with no skips.
• Both real-weight Whisper GPU conformance cases passed: default task and explicit transcription prompt, including the expected transcript and input-capability checks.
• Targeted pre-commit checks passed.

These runs used a TensorRT-LLM 1.4.0rc0 release container with the merged Python sources installed. They do not replace matching-build upstream validation. Confirm the A10 conformance stage and shared model/audio fixtures against the published PR head before merge.

Related Issues

• Relates to DIS-2971; audio passthrough is one part of its image/video/audio scope.

PR Checklist

• Reviewed the checklist; the change is backward compatible and includes regression coverage.

Dev Engineer Review

• Adds model-aware image, video, and audio support to OpenEngine gRPC Generate.
• Preserves media order and caller UUIDs. Enforces media-count, decoded-byte, and concurrency limits.
• Adds asynchronous decoding and DNS-safe remote fetching. Maps retryable fetch failures to UNAVAILABLE.
• Updates model-type resolution, token-ID capability reporting, request cancellation, and media capability discovery.
• Review severity counts are unavailable.
• Focused tests passed. Post-refresh GPU conformance needs follow-up because LLM_MODELS_ROOT was unavailable.

QA Engineer Review

• Adds unit coverage for capability discovery, media mapping, ordering, UUIDs, placeholders, limits, cancellation, concurrency, validation, decoding, and error mapping.
• Adds GPU conformance coverage for Qwen3-VL image and video generation and Whisper audio transcription.
• Updates async media, URL validation, model configuration, Whisper, video decoding, Control, and server lifecycle tests.
• Lists test_multimodal.py in l0_cpu.yml.
• Lists test_multimodal_conformance.py in l0_a10.yml.
• Coverage verdict: needs follow-up because post-refresh model-backed conformance was not rerun.

Per-File QA Perspective

• tensorrt_llm/grpc/openengine/capabilities.py: Adds model-aware media capability resolution and placeholder formatting. Verify unsupported modalities and malformed templates.
• tensorrt_llm/grpc/openengine/control.py: Reports supported modalities and token-ID support through GetModelInfo. Verify default and multimodal responses.
• tensorrt_llm/grpc/openengine/errors.py: Adds MediaUnavailableError. Verify exports and gRPC status mapping.
• tensorrt_llm/grpc/openengine/request_mapping.py: Adds media loading, validation, ordering, limits, cancellation, and token-ID checks. Verify all input-source paths and cleanup behavior.
• tensorrt_llm/grpc/openengine/servicer.py: Adds capability caching, request registration, duplicate detection, and media error handling. Verify abort and cleanup paths.
• tensorrt_llm/grpc/openengine/server.py: Closes the media HTTP session during shutdown, including shutdown failures. Verify lifecycle cleanup.
• tensorrt_llm/inputs/media_io.py: Adds byte loading, DNS-safe fetching, retry classification, and decoded-memory reservations. Verify limits and reservation cleanup.
• tensorrt_llm/inputs/model_config.py: Adds shared model-type resolution. Verify class-level and instance-level fallbacks.
• tensorrt_llm/inputs/utils.py: Routes media loading through safe HTTP fetching. Verify image, video, and audio regressions.
• tensorrt_llm/serve/chat_utils.py: Uses the shared model-type resolver. Verify existing chat behavior.
• tensorrt_llm/_torch/models/modeling_whisper.py: Declares that Whisper does not support token-ID input. Verify Whisper validation.
• tests/integration/test_lists/test-db/l0_a10.yml: Adds test_multimodal_conformance.py to the A10 list.
• tests/integration/test_lists/test-db/l0_cpu.yml: Adds test_multimodal.py to the CPU list.
• tests/unittest/grpc/openengine/test_control.py: Covers multimodal model-info and token-ID capability reporting. The file has no direct test-list entry shown.
• tests/unittest/grpc/openengine/test_multimodal.py: Covers media mapping, limits, cancellation, validation, and gRPC errors. Listed in l0_cpu.yml.
• tests/unittest/grpc/openengine/test_multimodal_conformance.py: Covers real image, video, and audio generation. Listed in l0_a10.yml.
• tests/unittest/inputs/test_async_media_loading.py: Covers raw-byte loading, session lifecycle, and video decoder arguments. The file has no direct test-list entry shown.
• tests/unittest/inputs/test_model_config.py: Covers model-type resolution. The file has no direct test-list entry shown.
• tests/unittest/inputs/test_url_validation.py: Covers DNS, timeout, HTTP, redirect, and response-size behavior. The file has no direct test-list entry shown.
• tests/unittest/_torch/modeling/test_modeling_whisper.py: Covers Whisper processor capabilities. The file has no direct test-list entry shown.
• tests/unittest/inputs/test_video_decode.py: Covers decoded-video limits and reservation cleanup. The file has no direct test-list entry shown.
• tests/unittest/grpc/openengine/test_openengine.py: Verifies HTTP session cleanup after server shutdown. The file has no direct test-list entry shown.

[krishung5]

/bot run

[tensorrt-cicd]

PR_Github #80682 [ run ] triggered by Bot. Commit: 68b0ef2 Link to invocation

[tensorrt-cicd]

PR_Github #80682 [ run ] completed with state SUCCESS. Commit: 68b0ef2
/LLM/main/L0_MergeRequest_PR pipeline #66432 completed with status: 'UNSTABLE'

CI Report

⚠️ Multi-GPU Label Required:
Multi-GPU tests require the ci: full pre-merge approved label on this PR. Either:

• Wait for the PR to be fully approved — the label is added automatically once approval is complete. Having unresolved open comments is fine, or
• If needed, ask a member of NVIDIA/trt-llm-ci-approvers to add the label manually.
  Then re-trigger CI with the same bot command (no rebase needed).

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[krishung5]

/bot run

[tensorrt-cicd]

PR_Github #80993 [ run ] triggered by Bot. Commit: 8fb1e66 Link to invocation

[tensorrt-cicd]

PR_Github #80993 [ run ] completed with state SUCCESS. Commit: 8fb1e66
/LLM/main/L0_MergeRequest_PR pipeline #66715 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[krishung5]

/bot run

[github-actions]

👎 Promotion blocked, new vulnerability found

Vulnerability report

Component	Vulnerability	Description	Severity
py-filelock	CVE-2026-112906	filelock is a platform-independent file lock for Python. From 4.1.0 until 4.2.1, SoftFileLock versions from 4.1.0 can reuse a stale observation when an attacker pre-dates and removes its own marker, at least two lock-user processes are waiting, coarse or matching modification times preserve the observation, and an honest contender creates a fresh marker on the reused inode, causing that live marker to be removed as stale. This issue is fixed in version 4.2.1.	MEDIUM
py-filelock	CVE-2026-112914	filelock is a platform-independent file lock for Python. From 4.1.0 until 4.2.1, SoftFileLock can restore a stale marker by name after moving it aside, allowing another local user with rename access in a non-sticky lock directory to swap in a symbolic link and, on a runtime that follows the link and a same-filesystem target, make the lock user hard-link one of its files into the lock directory. This issue is fixed in version 4.2.1.	MEDIUM
py-filelock	CVE-2026-112937	filelock is a platform-independent file lock for Python. From 4.1.0 until 4.2.1, ReadWriteLock can remove a randomized .filelock-* private-hard-link directory during cleanup, allowing another local user with rename access in a non-sticky parent to observe a live name or forge a dead-owner sweep name and rename a sibling directory onto it for deletion. This issue is fixed in version 4.2.1.	MEDIUM
py-filelock	CVE-2026-112940	filelock is a platform-independent file lock for Python. From 3.20.4 until 4.2.1, ReadWriteLock let SQLite reopen the database and side files by name; on POSIX systems an attacker with rename access in a non-sticky parent can swap those names to cause overwrite or unlink operations, while a writable adjacent hot journal is additionally required for attacker-chosen content overwrites. This issue is fixed in version 4.2.1.	HIGH

[krishung5]

/bot run

[tensorrt-cicd]

PR_Github #81087 [ run ] triggered by Bot. Commit: 6a07a16 Link to invocation

[kaiyux]

Approving on behalf of trt-llm-models-devs, thx

[github-actions]

Automatically added "ci: full pre-merge approved" because this PR has satisfied the required GitHub review approvals. Unresolved review conversations and other required checks remain independent merge requirements.

[tensorrt-cicd]

PR_Github #81087 [ run ] completed with state SUCCESS. Commit: 6a07a16
/LLM/main/L0_MergeRequest_PR pipeline #66798 completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

CI Report

Link to invocation

[krishung5]

/bot run

[tensorrt-cicd]

PR_Github #81211 [ run ] triggered by Bot. Commit: b422f01 Link to invocation