Skip to content

Clarify raw vLLM video sampling requests - #349

Open
sylvesterkaczmarek wants to merge 2 commits into
NVIDIA:mainfrom
sylvesterkaczmarek:docs/reasoner-raw-vllm-video-sampling-20261031
Open

sylvesterkaczmarek wants to merge 2 commits into
NVIDIA:mainfrom
sylvesterkaczmarek:docs/reasoner-raw-vllm-video-sampling-20261031

Conversation

@sylvesterkaczmarek

@sylvesterkaczmarek sylvesterkaczmarek commented Oct 31, 2026 •

Copy link
Copy Markdown

Resubmits NVIDIA/cosmos#337 at the new Cosmos Framework cookbook path after NVIDIA/cosmos#384 moved the file to .agents/skills/.

The Reasoner quickstart and prompt guide now show video frame sampling via media_io_kwargs={"video": {"num_frames": -1, "fps": 4}} in both OpenAI client extra_body and direct /v1/chat/completions JSON. This matches run_with_vllm.ipynb (and avoids suggesting two independent sampling paths at once). The raw JSON uses the model ID nvidia/Cosmos3-Super, consistent with the documented vLLM launch. Addresses NVIDIA/cosmos#190.

Validation: Documentation-only; checked the actual notebook cells, model-serving instructions and guide for consistency, parsed the Python examples, and passed git diff --check. No live vLLM inference or model-serving claim. Maintainer verification on a running server is still welcome.

DCO signed off. Thanks @yaoxu-crypto for the migration guidance and invitation to resubmit.

Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>

@pengcuo pengcuo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review notes (docs only; I did not run anything).

  1. The raw HTTP example uses a model id that vLLM will not serve. payload = {"model": "Cosmos3-Super", ...}. The setup guide serves with vllm serve nvidia/Cosmos3-Super (no --served-model-name), so the served id is nvidia/Cosmos3-Super and a request for Cosmos3-Super returns "model does not exist". The OpenAI snippet above it correctly uses client.models.list().data[0].id. Use nvidia/Cosmos3-Super here (or nvidia/Cosmos3-Nano, since the quickstart uses Nano on port 8000).

  2. It contradicts the notebook this README documents. run_with_vllm.ipynb samples video with extra_body={"media_io_kwargs": {"video": {"num_frames": -1, "fps": 4}}}. It does not use mm_processor_kwargs. The PR says no live vLLM request was run, so it is unverified which shape is right, and the two docs now describe different mechanisms. Please pick one and check it against a running server. If both are valid, say what each controls. If media_io_kwargs already samples at 4 fps, adding do_sample_frames: True may sample twice.

  3. Note reasoner_prompt_guide.md already says "the notebook uses frame sampling through extra_body ... mm_processor_kwargs". That sentence is also not what the notebook does, so it should be fixed together with this.

Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
@sylvesterkaczmarek

Copy link
Copy Markdown
Author

Thanks for the detailed review, @pengcuo. You were right about the raw model ID and the mismatch with the actual notebook. Commit 7bd3b8e aligns the README and prompt guide with the notebook’s media_io_kwargs.video sampling, fixes the raw model ID to nvidia/Cosmos3-Super, and avoids recommending overlapping sampling mechanisms. I checked the notebook cells and Python example syntax. I have not run a live vLLM server, so the server-side validation you requested remains open.

@yaoxu-crypto yaoxu-crypto added the Skill New skill or updating existing skill label Nov 3, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Skill New skill or updating existing skill

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants