Repository navigation
feat(plugins): local plugin catalogs in shared folders - #399
Merged
Merged
Conversation
Organizations can offer plugins from folders, typically network shares, in addition to the online DataLab plugin catalog: - A folder provides its catalog.json file if any, otherwise the plugin wheels it holds. It is read as files, so UNC paths work. - Local catalogs are listed before the online catalog and hide the same plugins in it. - Folders are set in the Plugin settings tab (plugins_catalog_path_list option) or with the DATALAB_PLUGINS_CATALOG environment variable. - A catalog that cannot be read is reported without hiding the plugins of the other catalogs. - "datalab-plugin catalog build" writes catalog.json from the wheels of a folder, keeping the edited fields. "datalab-plugin catalog check" reports wheels that are missing, changed, refused or not listed. Includes tests, documentation, release notes and French translations.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lets organizations distribute their own plugins, or the versions of public plugins they have approved, from local plugin catalogs. A local catalog is a folder, typically a network share, read in addition to the online DataLab plugin catalog.
Why
The Available plugins tab read a single catalog, set by the
plugins_catalog_urloption (configuration file only). Afile:URL was accepted, but:Path(r"\\server\share\catalog.json").as_uri()givesfile://server/share/..., whichurllibrefuses. Onlyfile:////server/...works, which users are unlikely to guess.What changes
Local catalogs (
datalab/plugins/catalog.py, Qt-free)catalog.jsonfile if any, with release URLs relative to the folder and kept inside it. Otherwise DataLab lists the plugin wheels the folder holds (pure Python,datalab.pluginsentry point, versions grouped). Other files are ignored and reported.load_catalogs()reads every source independently. A source that cannot be read (disconnected share, no Internet access) is reported without hiding the plugins of the other sources.DATALAB_PLUGINS_CATALOGfolders, then configured folders, then the online catalog. A plugin listed by a local catalog hides the same plugin (same distribution or ID) in the following catalogs. When a hidden online entry is revoked, a warning is shown.Configuration
plugins_catalog_path_listoption.DATALAB_PLUGINS_CATALOGenvironment variable (os.pathsep-separated). It is read when catalogs are loaded, and missing folders are kept: a share unavailable at startup is reported, not silently dropped.User interface
Command line (
datalab/plugins/catalogtool.py, Qt-free)datalab-plugin catalog build <folder>writescatalog.jsonfrom the folder wheels. Each wheel is checked as DataLab does before installing it, for one of the Python versions DataLab supports. Edited fields (ID, name, tier, status and reason, repository, keywords,yanked…) are kept, and revoked plugins stay listed. The file is replaced atomically, and nothing is written if a wheel is refused.datalab-plugin catalog check <folder>reports missing, changed, refused or unlisted wheels. Exit code 1 on problems.Other
store.get_available_distributions()is extracted as a module function and reused by the CLI.SearchPathItemWidgetnow takes the name of the environment variable providing a path (env_var) instead of afrom_envflag.Documentation and translations
doc/features/advanced/plugins.rst: new Local plugin catalogs subsection. The paragraph recommendingfile:URLs for network folders is replaced.doc/release_notes/release_1.04.md: three user-facing bullets in Plugin installation.build/libsources, and only the entries related to this feature were updated.Tests
plugin_local_catalog_unit_test.py: folder scan, localcatalog.json(relative paths, escaping the folder refused), copy with digest check, precedence and per-source errors, environment variable.plugin_catalogtool_unit_test.py:buildandcheck, curated fields kept, refused wheels, exit codes.plugin_available_dialog_test.py: local catalog takes precedence and installs with the expected origin, failing online source, settings section persisted without reload prompt.catalogtooladded to the Qt-free import contract test.The plugin and configuration test suites pass locally (230 tests).
Notes for reviewers
catalog.json, the SHA-256 of every wheel is computed at each refresh. This is fine for a few dozen wheels on a share.catalog.json, newly copied wheels appear only afterdatalab-plugin catalog build(documented, and reported bycheck).DESKTOP_PYTHONS(3.9–3.14) must be updated when DataLab supports a new Python version.