From defade539e1c43631f6fd50f1e8301e80c0cebf2 Mon Sep 17 00:00:00 2001 From: padatta <103029195+padatta@users.noreply.github.com> Date: Mon, 2 Nov 2026 09:55:44 +0530 Subject: [PATCH 1/3] Add Intel XPU support for Cosmos3-Edge Diffusers Signed-off-by: padatta <103029195+padatta@users.noreply.github.com> --- .agents/skills/cookbooks/cosmos3/README.md | 27 +++++++++- .../skills/cosmos3-generate-media/SKILL.md | 8 ++- .../references/diffusers.md | 51 +++++++++++-------- 3 files changed, 62 insertions(+), 24 deletions(-) diff --git a/.agents/skills/cookbooks/cosmos3/README.md b/.agents/skills/cookbooks/cosmos3/README.md index 7be6256c7..429e635f0 100644 --- a/.agents/skills/cookbooks/cosmos3/README.md +++ b/.agents/skills/cookbooks/cosmos3/README.md @@ -23,7 +23,8 @@ local Framework environment below with ModelOpt, datasets, and notebook tooling. ## Prerequisites -- Linux with NVIDIA GPU access. +- Linux with NVIDIA GPU access, or an Intel GPU supported by PyTorch XPU for the + Cosmos3-Edge Diffusers backend. - [`uv`](https://docs.astral.sh/uv/getting-started/installation/), `git`, and `git-lfs` installed. - Hugging Face access to the gated Cosmos3 model repos. Generator also requires access to the gated @@ -158,7 +159,9 @@ for a one-off run, pass `--no-guardrails` to `cosmos_framework.scripts.inference Direct generation with `Cosmos3OmniPipeline` (Generator · Audiovisual). Create a venv and install the backend, choosing `--torch-backend` to match your driver -(see [CUDA driver and the `cuXXX` backend](#cuda-driver-and-the-cuxxx-backend)): +(see [CUDA driver and the `cuXXX` backend](#cuda-driver-and-the-cuxxx-backend)). +XPU is supported for Cosmos3-Edge generation only; other backends retain their +documented NVIDIA GPU requirements. ```bash uv venv --python 3.13 --seed --managed-python @@ -177,6 +180,26 @@ uv pip install --torch-backend=cu130 \ transformers ``` +For Cosmos3-Edge on an Intel Arc GPU, install the same dependencies with the +PyTorch XPU backend: + +```bash +uv pip install --torch-backend=xpu \ + "diffusers @ git+https://github.com/huggingface/diffusers.git" \ + accelerate \ + av \ + cosmos_guardrail \ + huggingface_hub \ + imageio \ + imageio-ffmpeg \ + torch \ + torchvision \ + transformers +``` + +This Cosmos3-Edge path was validated on Intel Arc Graphics with PyTorch +`2.13.0.dev20260530+xpu` and Diffusers `0.35.2`. + ## TensorRT-LLM Generator OpenAI-compatible **VisualGen** server for Generator audiovisual text-to-image, diff --git a/.agents/skills/cosmos3-generate-media/SKILL.md b/.agents/skills/cosmos3-generate-media/SKILL.md index ca299da1b..529492ded 100644 --- a/.agents/skills/cosmos3-generate-media/SKILL.md +++ b/.agents/skills/cosmos3-generate-media/SKILL.md @@ -97,10 +97,14 @@ the first command (Quickstart; § 7 `get_pipe` / `run_diffusers_payload`; use-ca The notebook's venv, with the torch backend matching the driver's CUDA major version (13.x → `cu130`, 12.x → `cu128`; check `nvidia-smi`), plus the setup guide's -headless OpenCV swap: +headless OpenCV swap. For Cosmos3-Edge on Intel Arc, set +`COSMOS3_TORCH_BACKEND=xpu`; this path was validated with PyTorch +`2.13.0.dev20260530+xpu` and Diffusers `0.35.2`. Other models and backends remain +on their documented NVIDIA GPU paths. ```bash export COSMOS3_DIFFUSERS_VENV="${COSMOS3_DIFFUSERS_VENV:-$PWD/.venv-cosmos3-diffusers}" +export COSMOS3_TORCH_BACKEND="${COSMOS3_TORCH_BACKEND:-cu130}" # This skill sits inside the cosmos-framework checkout, whose pyproject.toml pins uv ==0.11.29 for the # framework venv; --no-config (or UV_NO_CONFIG=1) keeps that pin, and the framework's other [tool.uv] # settings, away from this standalone venv. UV_* environment variables still apply. Running the two @@ -117,7 +121,7 @@ source "$COSMOS3_DIFFUSERS_VENV/bin/activate" # cosmos_guardrail pulls the GUI opencv-python wheel (needs libxcb/libGL). The headless swap (setup guide, # TensorRT-LLM section) is part of this install step — one compound command, so the import check below cannot # run before it (eval run 2026-10-24: an agent ran the check before the separate swap lines -> libxcb ImportError). -uv pip install --no-config --python "$COSMOS3_DIFFUSERS_VENV" --torch-backend=cu130 \ +uv pip install --no-config --python "$COSMOS3_DIFFUSERS_VENV" --torch-backend="$COSMOS3_TORCH_BACKEND" \ "diffusers @ git+https://github.com/huggingface/diffusers.git" \ accelerate av cosmos_guardrail huggingface_hub imageio imageio-ffmpeg \ torch torchvision transformers \ diff --git a/.agents/skills/cosmos3-generate-media/references/diffusers.md b/.agents/skills/cosmos3-generate-media/references/diffusers.md index 325a1f2d7..f4e805fad 100644 --- a/.agents/skills/cosmos3-generate-media/references/diffusers.md +++ b/.agents/skills/cosmos3-generate-media/references/diffusers.md @@ -9,7 +9,7 @@ README section "Run with Diffusers". Its relative links (`../../README.md#diffus ### Quickstart Set up the environment: [Diffusers setup](../../cookbooks/cosmos3/README.md#diffusers). -Run a text-to-video generation with `Cosmos3OmniPipeline`: +Run a Cosmos3-Edge text-to-video generation with `Cosmos3OmniPipeline`: ```python import json @@ -20,36 +20,36 @@ from diffusers.utils import export_to_video prompt = json.load(open("assets/prompts/text2video/robot_kitchen.json")) negative = json.load(open("assets/negative_prompts/text2video/neg_prompt.json")) +device = "xpu" if hasattr(torch, "xpu") and torch.xpu.is_available() else "cuda" pipe = Cosmos3OmniPipeline.from_pretrained( - "nvidia/Cosmos3-Nano", torch_dtype=torch.bfloat16, device_map="cuda" + "nvidia/Cosmos3-Edge", torch_dtype=torch.bfloat16 ) -pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config, flow_shift=10.0) +pipe.to(device) +pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config, flow_shift=8.0) result = pipe( prompt=json.dumps(prompt), negative_prompt=json.dumps(negative), image=None, - num_frames=189, - height=720, - width=1280, + num_frames=121, + height=480, + width=832, fps=24, - num_inference_steps=35, - guidance_scale=6.0, + num_inference_steps=20, + guidance_scale=5.0, enable_sound=False, add_resolution_template=False, add_duration_template=False, - generator=torch.Generator(device="cuda").manual_seed(1234), + generator=torch.Generator(device=device).manual_seed(1234), ) -export_to_video(result.video, "/tmp/cosmos3_t2v_diffusers.mp4", fps=24) +export_to_video(result.video, "/tmp/cosmos3_edge_t2v_diffusers.mp4", fps=24) ``` -To run **Cosmos3-Super** instead, load the larger checkpoint: -`Cosmos3OmniPipeline.from_pretrained("nvidia/Cosmos3-Super", ...)`. - -To run **Cosmos3-Edge** instead, load `nvidia/Cosmos3-Edge` and use its -single-GPU, no-audio settings: `height=480`, `width=832`, `num_frames=121`, -`num_inference_steps=20`, `guidance_scale=5.0`, and `flow_shift=8.0`. +The XPU path was validated on Intel Arc Graphics with PyTorch +`2.13.0.dev20260530+xpu` and Diffusers `0.35.2`. XPU support is limited to +Cosmos3-Edge generation through Diffusers. Nano and Super examples remain on +their documented CUDA paths. ### Notebook walkthrough @@ -259,6 +259,16 @@ def release_pipe() -> None: print(f"released previous pipeline; cuda allocated {cuda_allocated_gib():.1f} GiB") +def inference_device(model: str) -> str: + if ( + resolve_model_id(model) == "nvidia/Cosmos3-Edge" + and hasattr(torch, "xpu") + and torch.xpu.is_available() + ): + return "xpu" + return "cuda" + + def get_pipe(model: str): global _pipe, _pipe_model model_id = resolve_model_id(model) @@ -289,7 +299,7 @@ def get_pipe(model: str): token=os.environ.get("HF_TOKEN") or None, ) pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config, flow_shift=FIXED_SAMPLING["shift"]) - pipe.to("cuda") + pipe.to(inference_device(model)) _pipe = pipe _pipe_model = model_id print(f"loaded pipeline in {time.time() - t0:.1f}s") @@ -306,7 +316,7 @@ def run_diffusers_payload(payload_path: Path, output_dir: str | Path, *, model: pipe = get_pipe(model) if not distilled: pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config, flow_shift=payload["shift"]) - generator = torch.Generator(device="cuda").manual_seed(payload["seed"]) + generator = torch.Generator(device=inference_device(model)).manual_seed(payload["seed"]) vision_path = resolve_payload_path(payload_path, payload["vision_path"]) if "vision_path" in payload else None image = load_image(str(vision_path)) if payload["model_mode"] == "image2video" else None video = load_video(str(vision_path)) if payload["model_mode"] == "video2video" else None @@ -496,11 +506,12 @@ from diffusers.schedulers.scheduling_unipc_multistep import UniPCMultistepSchedu from diffusers.utils import export_to_video compact = lambda p: json.dumps(json.load(open(p)), ensure_ascii=True, separators=(",", ":")) # run from this skill directory (the one with assets/) +device = "xpu" if hasattr(torch, "xpu") and torch.xpu.is_available() else "cuda" pipe = Cosmos3OmniPipeline.from_pretrained("nvidia/Cosmos3-Edge", torch_dtype=torch.bfloat16, safety_checker=None, enable_safety_checker=False) pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config, flow_shift=8.0) -pipe.to("cuda") -gen = torch.Generator(device="cuda").manual_seed(1234) # FIXED_SAMPLING seed, as in the Nano snippet +pipe.to(device) +gen = torch.Generator(device=device).manual_seed(1234) # FIXED_SAMPLING seed, as in the Nano snippet ``` ```python From a658b3fa0617c5ea56a7b8c09b286b0ec4c1c5c3 Mon Sep 17 00:00:00 2001 From: padatta <103029195+padatta@users.noreply.github.com> Date: Tue, 3 Nov 2026 11:35:43 +0530 Subject: [PATCH 2/3] Address review: correct validated stack, keep Nano quickstart, scope XPU to Edge - Replace the incorrect Diffusers 0.35.2 / torch nightly claim with the stack actually validated from a fresh install (torch 2.15.0+xpu, Diffusers 0.42.0.dev0 @ 86ef420), stated once in the setup guide. - Restore the Nano quickstart and add a separate Cosmos3-Edge XPU block. - State guardrail status: XPU block validated with the guardrail off. - Revert the notebook-verbatim runner to CUDA and mark CUDA-only cells. - Use one rule: XPU applies to Cosmos3-Edge only. - Document the 480p VAE decode limit (>=33 frames) on the Arc iGPU. Signed-off-by: padatta <103029195+padatta@users.noreply.github.com> --- .agents/skills/cookbooks/cosmos3/README.md | 18 +++- .../skills/cosmos3-generate-media/SKILL.md | 7 +- .../references/diffusers.md | 95 ++++++++++++++----- 3 files changed, 86 insertions(+), 34 deletions(-) diff --git a/.agents/skills/cookbooks/cosmos3/README.md b/.agents/skills/cookbooks/cosmos3/README.md index 429e635f0..daacc866a 100644 --- a/.agents/skills/cookbooks/cosmos3/README.md +++ b/.agents/skills/cookbooks/cosmos3/README.md @@ -180,8 +180,8 @@ uv pip install --torch-backend=cu130 \ transformers ``` -For Cosmos3-Edge on an Intel Arc GPU, install the same dependencies with the -PyTorch XPU backend: +For Cosmos3-Edge on an Intel GPU, install the same dependencies with the +PyTorch XPU backend, then verify the device: ```bash uv pip install --torch-backend=xpu \ @@ -195,10 +195,20 @@ uv pip install --torch-backend=xpu \ torch \ torchvision \ transformers + +python -c 'import torch; print(torch.xpu.is_available(), torch.xpu.get_device_name(0))' ``` -This Cosmos3-Edge path was validated on Intel Arc Graphics with PyTorch -`2.13.0.dev20260530+xpu` and Diffusers `0.35.2`. +Validated for Cosmos3-Edge text-to-video (480p, 17 frames, 20 steps, guardrail +off) on an Intel Core Ultra 9 285H with Intel Arc Graphics (iGPU), using a +fresh install from the command above: Python 3.13.18, PyTorch `2.15.0+xpu`, +torchvision `0.29.2+xpu`, Diffusers `0.42.0.dev0` (git commit `86ef420`), and +Transformers `5.22.1`. On this iGPU, VAE decode at 480p with 33 or more frames, +including Edge's default 121, did not complete. + +If `import torch` fails with `undefined symbol: urGraphGetIdExp`, a system oneAPI +install on `LD_LIBRARY_PATH` is shadowing the runtime shipped with the XPU wheel; +remove the oneAPI entries from `LD_LIBRARY_PATH` for this venv. ## TensorRT-LLM Generator diff --git a/.agents/skills/cosmos3-generate-media/SKILL.md b/.agents/skills/cosmos3-generate-media/SKILL.md index 529492ded..c2e6c96bb 100644 --- a/.agents/skills/cosmos3-generate-media/SKILL.md +++ b/.agents/skills/cosmos3-generate-media/SKILL.md @@ -97,10 +97,9 @@ the first command (Quickstart; § 7 `get_pipe` / `run_diffusers_payload`; use-ca The notebook's venv, with the torch backend matching the driver's CUDA major version (13.x → `cu130`, 12.x → `cu128`; check `nvidia-smi`), plus the setup guide's -headless OpenCV swap. For Cosmos3-Edge on Intel Arc, set -`COSMOS3_TORCH_BACKEND=xpu`; this path was validated with PyTorch -`2.13.0.dev20260530+xpu` and Diffusers `0.35.2`. Other models and backends remain -on their documented NVIDIA GPU paths. +headless OpenCV swap. For Cosmos3-Edge on an Intel GPU, set +`COSMOS3_TORCH_BACKEND=xpu` (validated stack: setup guide, Diffusers section). XPU +is Edge-only; Nano, Super, and the distilled checkpoints stay on CUDA. ```bash export COSMOS3_DIFFUSERS_VENV="${COSMOS3_DIFFUSERS_VENV:-$PWD/.venv-cosmos3-diffusers}" diff --git a/.agents/skills/cosmos3-generate-media/references/diffusers.md b/.agents/skills/cosmos3-generate-media/references/diffusers.md index f4e805fad..e138f0418 100644 --- a/.agents/skills/cosmos3-generate-media/references/diffusers.md +++ b/.agents/skills/cosmos3-generate-media/references/diffusers.md @@ -9,7 +9,7 @@ README section "Run with Diffusers". Its relative links (`../../README.md#diffus ### Quickstart Set up the environment: [Diffusers setup](../../cookbooks/cosmos3/README.md#diffusers). -Run a Cosmos3-Edge text-to-video generation with `Cosmos3OmniPipeline`: +Run a text-to-video generation with `Cosmos3OmniPipeline`: ```python import json @@ -20,19 +20,67 @@ from diffusers.utils import export_to_video prompt = json.load(open("assets/prompts/text2video/robot_kitchen.json")) negative = json.load(open("assets/negative_prompts/text2video/neg_prompt.json")) -device = "xpu" if hasattr(torch, "xpu") and torch.xpu.is_available() else "cuda" pipe = Cosmos3OmniPipeline.from_pretrained( - "nvidia/Cosmos3-Edge", torch_dtype=torch.bfloat16 + "nvidia/Cosmos3-Nano", torch_dtype=torch.bfloat16, device_map="cuda" ) -pipe.to(device) +pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config, flow_shift=10.0) + +result = pipe( + prompt=json.dumps(prompt), + negative_prompt=json.dumps(negative), + image=None, + num_frames=189, + height=720, + width=1280, + fps=24, + num_inference_steps=35, + guidance_scale=6.0, + enable_sound=False, + add_resolution_template=False, + add_duration_template=False, + generator=torch.Generator(device="cuda").manual_seed(1234), +) +export_to_video(result.video, "/tmp/cosmos3_t2v_diffusers.mp4", fps=24) +``` + +To run **Cosmos3-Super** instead, load the larger checkpoint: +`Cosmos3OmniPipeline.from_pretrained("nvidia/Cosmos3-Super", ...)`. + +To run **Cosmos3-Edge** instead, load `nvidia/Cosmos3-Edge` and use its +single-GPU, no-audio settings: `height=480`, `width=832`, `num_frames=121`, +`num_inference_steps=20`, `guidance_scale=5.0`, and `flow_shift=8.0`. + +### Cosmos3-Edge on Intel XPU (skill addition, not in the README) + +XPU applies to Cosmos3-Edge only; Nano, Super, and the distilled checkpoints stay on CUDA. +Install with `--torch-backend=xpu` ([Diffusers setup](../../cookbooks/cosmos3/README.md#diffusers)), then run Edge +on `xpu`. This block was validated with the guardrail **off**. The Edge checkpoint's `model_index.json` already +sets `enable_safety_checker: false`, and the guardrail (`cosmos_guardrail`) has not been run on XPU yet; it is passed +explicitly below so the setting is visible. Set it to `True` (as `get_pipe`, § 7, does) only after checking the guardrail +models on your XPU setup: + +```python +import json +import torch +from diffusers import Cosmos3OmniPipeline +from diffusers.schedulers.scheduling_unipc_multistep import UniPCMultistepScheduler +from diffusers.utils import export_to_video + +prompt = json.load(open("assets/prompts/text2video/robot_kitchen.json")) +negative = json.load(open("assets/negative_prompts/text2video/neg_prompt.json")) + +pipe = Cosmos3OmniPipeline.from_pretrained( + "nvidia/Cosmos3-Edge", torch_dtype=torch.bfloat16, enable_safety_checker=False +) +pipe.to("xpu") pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config, flow_shift=8.0) result = pipe( prompt=json.dumps(prompt), negative_prompt=json.dumps(negative), image=None, - num_frames=121, + num_frames=17, height=480, width=832, fps=24, @@ -41,15 +89,15 @@ result = pipe( enable_sound=False, add_resolution_template=False, add_duration_template=False, - generator=torch.Generator(device=device).manual_seed(1234), + generator=torch.Generator(device="xpu").manual_seed(1234), ) -export_to_video(result.video, "/tmp/cosmos3_edge_t2v_diffusers.mp4", fps=24) +export_to_video(result.video, "/tmp/cosmos3_edge_t2v_xpu.mp4", fps=24) ``` -The XPU path was validated on Intel Arc Graphics with PyTorch -`2.13.0.dev20260530+xpu` and Diffusers `0.35.2`. XPU support is limited to -Cosmos3-Edge generation through Diffusers. Nano and Super examples remain on -their documented CUDA paths. +The block uses `num_frames=17`. On the validated Intel Arc iGPU, VAE decode at 480p with 33 or more frames +(including Edge's default 121) did not complete within 10 minutes, with or without `vae.enable_tiling()`, so +longer clips are not supported on XPU yet. Validated package versions and hardware: +[Diffusers setup](../../cookbooks/cosmos3/README.md#diffusers). ### Notebook walkthrough @@ -166,6 +214,8 @@ Jupyter-only step (the install cell registered a `Cosmos3 Diffusers (Python 3.13 ## 5. Verify GPU and Python Environment +This cell checks CUDA only. On Intel XPU (Cosmos3-Edge only), check `torch.xpu.is_available()` and `torch.xpu.get_device_name(0)` instead. + ```python import torch import diffusers @@ -194,7 +244,7 @@ Shared definitions, verbatim in payloads.md: - `create_payload` (`backend="diffusers"`): § 9.2 (`run_with_diffusers.ipynb` block; differences from the other notebooks' versions are tabulated there). - `MODEL_IDS`, `DISTILLED_MODELS`, `resolve_model_id`, `is_distilled_model`: § 5. -The Diffusers runner (verbatim; the kernel/venv guard that preceded the imports is covered by § 4, and the definitions above are pointed to where they sat in the cell): +The Diffusers runner (verbatim; the kernel/venv guard that preceded the imports is covered by § 4, and the definitions above are pointed to where they sat in the cell). It is CUDA-only (`pipe.to("cuda")`, CUDA generator, `torch.cuda` cleanup in `release_pipe`); for Cosmos3-Edge on Intel XPU use the Quickstart's XPU block or § 8.4: ```python import json @@ -259,16 +309,6 @@ def release_pipe() -> None: print(f"released previous pipeline; cuda allocated {cuda_allocated_gib():.1f} GiB") -def inference_device(model: str) -> str: - if ( - resolve_model_id(model) == "nvidia/Cosmos3-Edge" - and hasattr(torch, "xpu") - and torch.xpu.is_available() - ): - return "xpu" - return "cuda" - - def get_pipe(model: str): global _pipe, _pipe_model model_id = resolve_model_id(model) @@ -299,7 +339,7 @@ def get_pipe(model: str): token=os.environ.get("HF_TOKEN") or None, ) pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config, flow_shift=FIXED_SAMPLING["shift"]) - pipe.to(inference_device(model)) + pipe.to("cuda") _pipe = pipe _pipe_model = model_id print(f"loaded pipeline in {time.time() - t0:.1f}s") @@ -316,7 +356,7 @@ def run_diffusers_payload(payload_path: Path, output_dir: str | Path, *, model: pipe = get_pipe(model) if not distilled: pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config, flow_shift=payload["shift"]) - generator = torch.Generator(device=inference_device(model)).manual_seed(payload["seed"]) + generator = torch.Generator(device="cuda").manual_seed(payload["seed"]) vision_path = resolve_payload_path(payload_path, payload["vision_path"]) if "vision_path" in payload else None image = load_image(str(vision_path)) if payload["model_mode"] == "image2video" else None video = load_video(str(vision_path)) if payload["model_mode"] == "video2video" else None @@ -461,7 +501,8 @@ Text-to-image as the notebook runs it (`get_pipe` + the `text2image` branch of `run_diffusers_payload`; `FIXED_SAMPLING`: 35 steps, guidance 6.0, shift 10.0, 720 × 1280, seed 1234; `get_pipe` loads with `enable_safety_checker=True` — `False` is the § 1 switch; output path per § 4.2). Save the snippet as a file and run it with the venv's interpreter, `"$COSMOS3_DIFFUSERS_VENV/bin/python" .py`, -from this skill directory (the one with `assets/`) — `python` alone is the venv's only in a shell that activated it: +from this skill directory (the one with `assets/`) — `python` alone is the venv's only in a shell that activated it. +Nano runs on CUDA only; on Intel XPU use the Edge snippet of § 8.4: ```python import json, os, torch @@ -496,7 +537,9 @@ then verify with ffprobe or the venv's `av` (see SKILL.md Verify the output). Edge, composed from the README sentence above and the text2image / video branches of `run_diffusers_payload` (diffusers.md § 7; the diffusers notebook has no `t2i_edge` / `t2v_edge_noaudio` case, payloads.md § 8.2). Run the setup block, then ONE mode block — the one the task asks for — in the -same Python process; text-to-video needs more GPU time and memory than a single frame: +same Python process; text-to-video needs more GPU time and memory than a single frame. The `device` line selects Intel XPU +when available (Edge is the only model with an XPU path) and CUDA otherwise; like the Nano snippet, it disables the guardrail +for a smoke run, as in the Quickstart's XPU block: ```python # setup — shared by both modes From e50e0aae25a33141f616ed80b7bddce4d9a8a71d Mon Sep 17 00:00:00 2001 From: padatta <103029195+padatta@users.noreply.github.com> Date: Tue, 3 Nov 2026 13:17:41 +0530 Subject: [PATCH 3/3] Document cosmos_guardrail XPU blocker Enabling the guardrail fails on hosts without CUDA because cosmos_guardrail 0.3.2 loads the CUDA-saved face-blur checkpoint with torch.load and no map_location. With that load mapped to CPU, the guardrail ran on XPU (text and video checks on xpu:0), so state that in place of 'not run on XPU yet'. Signed-off-by: padatta <103029195+padatta@users.noreply.github.com> --- .agents/skills/cookbooks/cosmos3/README.md | 5 ++++- .../cosmos3-generate-media/references/diffusers.md | 10 ++++++---- 2 files changed, 10 insertions(+), 5 deletions(-) diff --git a/.agents/skills/cookbooks/cosmos3/README.md b/.agents/skills/cookbooks/cosmos3/README.md index daacc866a..ef369e020 100644 --- a/.agents/skills/cookbooks/cosmos3/README.md +++ b/.agents/skills/cookbooks/cosmos3/README.md @@ -204,7 +204,10 @@ off) on an Intel Core Ultra 9 285H with Intel Arc Graphics (iGPU), using a fresh install from the command above: Python 3.13.18, PyTorch `2.15.0+xpu`, torchvision `0.29.2+xpu`, Diffusers `0.42.0.dev0` (git commit `86ef420`), and Transformers `5.22.1`. On this iGPU, VAE decode at 480p with 33 or more frames, -including Edge's default 121, did not complete. +including Edge's default 121, did not complete. The guardrail cannot be enabled +on XPU with `cosmos_guardrail` 0.3.2, which loads a CUDA-saved checkpoint +without `map_location`; see +[the Diffusers skill reference](../../cosmos3-generate-media/references/diffusers.md#cosmos3-edge-on-intel-xpu-skill-addition-not-in-the-readme). If `import torch` fails with `undefined symbol: urGraphGetIdExp`, a system oneAPI install on `LD_LIBRARY_PATH` is shadowing the runtime shipped with the XPU wheel; diff --git a/.agents/skills/cosmos3-generate-media/references/diffusers.md b/.agents/skills/cosmos3-generate-media/references/diffusers.md index e138f0418..ced757a4c 100644 --- a/.agents/skills/cosmos3-generate-media/references/diffusers.md +++ b/.agents/skills/cosmos3-generate-media/references/diffusers.md @@ -55,10 +55,12 @@ single-GPU, no-audio settings: `height=480`, `width=832`, `num_frames=121`, XPU applies to Cosmos3-Edge only; Nano, Super, and the distilled checkpoints stay on CUDA. Install with `--torch-backend=xpu` ([Diffusers setup](../../cookbooks/cosmos3/README.md#diffusers)), then run Edge -on `xpu`. This block was validated with the guardrail **off**. The Edge checkpoint's `model_index.json` already -sets `enable_safety_checker: false`, and the guardrail (`cosmos_guardrail`) has not been run on XPU yet; it is passed -explicitly below so the setting is visible. Set it to `True` (as `get_pipe`, § 7, does) only after checking the guardrail -models on your XPU setup: +on `xpu`. This block runs with the guardrail **off**; the Edge checkpoint's `model_index.json` already sets +`enable_safety_checker: false`, and the block passes it explicitly so the setting is visible. Turning it on fails on +XPU (and on any host without CUDA) with `cosmos_guardrail` 0.3.2: `cosmos_utils.load_model` calls `torch.load` without +`map_location` on the CUDA-saved face-blur checkpoint. With that load mapped to CPU, the guardrail ran on XPU (text and +video checks on `xpu:0`); switch to `enable_safety_checker=True`, as `get_pipe` (§ 7) does, once a fixed +`cosmos_guardrail` is available: ```python import json