feat(packaging): ship PAI converter in nvidia-ncore wheel - #171
feat(packaging): ship PAI converter in nvidia-ncore wheel#171eattia-nvidia wants to merge 1 commit into
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
| ], | ||
| }, | ||
| extra_requires = { | ||
| "pai": [ |
There was a problem hiding this comment.
The pai extra is missing runtime dependencies, so pip install "nvidia-ncore[pai]" gives an import error rather than a working converter. tools/data_converter/pai/pai_remote/downloader.py imports rich, which isn't listed, installing the extra and importing that module fails with ModuleNotFoundError: No module named 'rich'. Comparing against the dependency set needed to build the converter as a bazel target, scipy and universal-pathlib also look absent. Worth being clear this isn't introduced here: rich is undeclared in every deps/pip/requirements_*.in on main today. It's just that this MR is what makes the module installable by external users, so a latent gap becomes a user-visible break at exactly the moment the extra ships.
There was a problem hiding this comment.
Agreed that the extra has to match what the installed converter actually imports.
scipy and universal_pathlib are already required by the base nvidia-ncore wheel, so pip install "nvidia-ncore[pai]" gets them without listing them again on the extra.
rich is only used by pai_remote/downloader.py (and the pai-clip-dl CLI). Those are not part of ncore-convert: streaming conversion uses config / index / remote / streaming, and local conversion reads pai-clip-dl output from disk. downloader.py is no longer packaged, so the extra does not need rich. pai-clip-dl stays a Bazel-only tool in this repo.
| version = "{BUILD_EMBED_LABEL}", | ||
| deps = [ | ||
| ":ncore_pkg", | ||
| "//tools/data_converter/pai:pai_converter_pkg", |
There was a problem hiding this comment.
Adding //tools/data_converter/pai:pai_converter_pkg to the wheel deps means the wheel installs a top-level tools/ package, the built wheel's top-level entries are ['ncore', 'nvidia_ncore-.dist-info', 'tools']. tools is about as generic a name as exists on PyPI, so nvidia-ncore[pai] would collide with anything else installing a tools module in the same environment, and would shadow it depending on install order. Since this is the release that first publishes it, it's much cheaper to namespace now (e.g. under ncore/) than after users depend on the current layout.
There was a problem hiding this comment.
Good catch — a top-level tools package on PyPI is too generic.
The published wheel now installs the converter under ncore.converters (ncore.converters.pai.converter:cli). There is no top-level tools in the wheel. In-repo Bazel targets are unchanged (//tools/data_converter/pai:convert); only the packaged import path is namespaced.
Package the PAI data converter under ncore.converters so PhysicalAI-AV clips can be converted from an installed wheel without a source checkout. Add the ncore-convert console script and a [pai] extra for converter deps.
e66d26b to
365cb9d
Compare
| @@ -0,0 +1,85 @@ | |||
| # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | |||
There was a problem hiding this comment.
What do you think about moving stuff into ncore/converters to avoid namespace remapping ? @janickm
|
I have some big reservations on this change and the whole repo and wheel structure is explicitly designed to not ship (the wheel is for APIs, not for tools - the converters are merely "samples" of how to convert common data) - there might be different options here, let's discuss this offline first, I'll reach out (making draft for now) |
|
closing as discussed offline with the alternative bazel-based http_archive approach |
3 participants
Description
Package the PAI data converter in the
nvidia-ncorewheel so PhysicalAI-AV clips can be converted without a source checkout or Bazel.pai_converter_pkgto//ncore:ncore_wheel(converter +pai_remote+ CLI).ncore-convert→tools.data_converter.pai.converter:cli.nvidia-ncore[pai]for converter deps (Click, DracoPy, imageio, pandas, pyarrow, PyNvVideoCodec, remotezip, requests, tqdm, debugpy).ncore/PYPI_README.mdandtools/data_converter/pai/README.md.//ncore:pytest_wheel_packaging_3_11.pai-clip-dlis not shipped as its own console script in this PR; streaming (pai-stream-v4) is the primary no-checkout path. Camera decode via PyNvVideoCodec needs an NVIDIA GPU (Turing+).Type of Change
Checklist
bazel test //...)bazel run //:format)