diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index eab1b6b..2e92e97 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,16 +1,47 @@ # Contributing a plugin to the catalog -Listing a plugin takes a public GitHub repository, a GitHub release holding the wheel, and a pull request adding one YAML file. No PyPI account is needed. +Listing a plugin takes a public GitHub repository, a GitHub release holding a wheel or a single Python file, and a pull request adding one YAML entry. No PyPI account is needed. Single-file artifacts are supported by the updated Desktop client only, not DataLab-Web. ## Requirements -- The source code is public on GitHub and licensed under an OSI-approved license, declared in the wheel metadata (`license = "..."` in `pyproject.toml` with setuptools 77 or later). Accepted licenses: 0BSD, AFL-3.0, Apache-2.0, BSD-2-Clause, BSD-3-Clause, BSL-1.0, CECILL-2.1, EPL-2.0, EUPL-1.2, GPL-2.0 and 3.0, ISC, LGPL-2.1 and 3.0, MIT, MPL-2.0, PSF-2.0, Unlicense, Zlib. -- The plugin is distributed as a single pure-Python wheel (`*-py3-none-any.whl`), without compiled code. -- The wheel declares its plugin class in the `datalab.plugins` entry-point group (DataLab desktop) and/or `datalab.web_plugins` (DataLab-Web). +- The source code is public on GitHub and licensed under an OSI-approved license. A wheel declares its license in its metadata (`license = "..."` in `pyproject.toml` with setuptools 77 or later); a module declares its SPDX license identifier in the catalog entry and must have corresponding source licensing. Accepted licenses: 0BSD, AFL-3.0, Apache-2.0, BSD-2-Clause, BSD-3-Clause, BSL-1.0, CECILL-2.1, EPL-2.0, EUPL-1.2, GPL-2.0 and 3.0, ISC, LGPL-2.1 and 3.0, MIT, MPL-2.0, PSF-2.0, Unlicense, Zlib. +- The artifact is either a pure-Python wheel (`*-py3-none-any.whl`) or one importable `datalab_.py` source file (maximum 4 MiB), without compiled plugin code. +- A wheel declares its plugin class in the `datalab.plugins` (Desktop) and/or `datalab.web_plugins` (Web) entry-point group. A module uses Desktop's existing file-name convention and plugin API; it does not need an entry-point file or packaging metadata. - Its requirements must be satisfied by DataLab (for example `datalab-platform`, `sigima`, `numpy`, `scipy`) or, for Desktop, by explicitly listed dependency wheels. Existing host package versions cannot be replaced. Additional wheels may contain native libraries but must cover the intended Python/ABI/platform; no compilation or system installation is performed. Web requirements remain restricted to packages supplied by the Web host. -- The plugin ID is the `id` of the plugin's `PluginInfo`. Prefer a reverse-domain name that you control, such as `io.github..`. The `org.datalab.` prefix is reserved for DataLab-Platform plugins. +- The entry's ID agrees with a statically readable `PluginInfo.id` when present. For an unchanged historical module with unavailable metadata or several plugin classes, the entry identifies the entire module without rewriting its runtime identities. Prefer a reverse-domain name that you control, such as `io.github..`. The `org.datalab.` prefix is reserved for DataLab-Platform plugins. -Projects created with `datalab-plugin create` already follow these rules. +The generators `datalab-plugin create` and `create --project` provide a starting point for the two source forms. Catalog metadata must still be supplied when it is not in the artifact. + +## Source organization is not the distribution format + +A plugin may be developed as a single `datalab_.py` module or as a larger Python package. Both use the same plugin API and may expose the same functional capabilities. The layered project generated by `datalab-plugin create --project` is one convenient organization, not a catalog requirement. + +The catalog accepts both wheels and raw single-file Desktop plugins. A wheel may contain just one module: packaging it does not require adopting the layered project. A raw `.py` entry avoids that packaging step entirely. + +## Listing a single-file Desktop plugin + +Attach the unchanged `datalab_.py` file to a versioned GitHub release, then add a YAML entry such as: + +```yaml +id: io.github.alice.spectrum-tools +name: Spectrum Tools +kind: module +summary: Spectral processing in one Python file +license: BSD-3-Clause +repository: https://github.com/alice/spectrum-tools +capabilities: [processing] +releases: + - version: "1.0.0" + filename: datalab_spectrum_tools.py + sha256: + requires_python: ">=3.9" +``` + +`filename` selects an exact release asset and stays stable between versions; do not add version numbers or hyphens that make the Python module name invalid. `summary`, `license` and a release version are required in the public entry. Literal source identities and versions, when readable, must agree with the entry. Unknown metadata is not evaluated: constructors, dynamic expressions and plugin code are never executed during inspection. A syntactically valid module is not thereby certified as a working or safe plugin. + +Modules use already available Desktop libraries. They cannot declare dependency wheel installation, use PyPI as their artifact source or advertise the Web target. The catalog mirrors their exact bytes under `modules//`; copying that tree and `catalog.json` provides an offline mirror. Local catalogs may also scan `.py` and `.whl` files directly at the root, without an index. + +For a local module whose version is unavailable, the generated index keeps an empty version. Desktop compares digests and asks for explicit replacement when content changes, rather than inventing a newer version. Local `catalog build/check/prepare` commands include modules, but only wheels participate in dependency preparation. Descriptive metadata supplied in an existing local JSON index is preserved when it is rebuilt. ## Additional Desktop dependencies @@ -32,6 +63,8 @@ The CI checks and mirrors these artifacts without executing them. The Desktop cl ## Submitting a plugin +The following is the wheel workflow; single-file authors use the release asset and entry described above instead. + 1. Build the wheel (`python -m build`) and attach it to a GitHub release of your repository, tagged `v` (for example `v1.0.0`). 2. Fork this repository and add `plugins/.yaml`: diff --git a/README.md b/README.md index 1881bb5..4e0021c 100644 --- a/README.md +++ b/README.md @@ -8,13 +8,15 @@ In DataLab 1.4 or later, open **Plugins > Configure plugins...**, select the **A To install a downloaded file instead (also with the standalone version of DataLab): -1. Open and download the wheel (`.whl`) of the plugin. +1. Open and download the plugin artifact: a wheel (`.whl`) or, for a **Python file** entry, the unchanged `datalab_.py` file. 2. In DataLab, open **Plugins > Configure plugins...**, go to the **Install plugins** tab and click **Install from file...**. If the plugin lists **Additional Desktop packages**, download the wheels for your Python version and platform into a `dependencies` folder next to the plugin wheel. A complete local catalog with these files can be installed without Internet access, including with the standalone application. DataLab shows the additional packages before confirmation, never replaces existing package versions, and activates the added dependencies after a restart. Uninstalling a plugin keeps its shared dependencies. A plugin runs with the same rights as DataLab: it may read and modify your files. Install only plugins from authors you trust. Being listed here does not mean that a plugin was audited. +Single-file plugins use libraries already available in Desktop and do not install additional packages. A module may be distributed without a packaging project; source organization does not determine its functional capabilities. Known versions can be updated; local scripts without a version use their digest and offer explicit replacement when changed. + ## Publishing a plugin Any author may list a plugin by opening a pull request that adds one small YAML file to the [plugins](plugins) folder. See [CONTRIBUTING.md](CONTRIBUTING.md). @@ -22,8 +24,9 @@ Any author may list a plugin by opening a pull request that adds one small YAML ## How the catalog works - Each plugin is described by `plugins/.yaml`, validated against [schema/plugin-entry.schema.json](schema/plugin-entry.schema.json). -- For each release, the CI downloads the wheel from the GitHub release (or from PyPI), checks its SHA-256 digest against the entry, and inspects it without running its code: pure-Python plugin, `datalab.plugins` (desktop) and/or `datalab.web_plugins` (web) entry points, declared requirements and OSI-approved license. Desktop releases may also list reviewed dependency wheels, including native wheels, which are verified and mirrored with the plugin. Web requirements must still be provided by its host. Python compatibility is checked per target: a desktop plugin must install on at least one Python version supported by DataLab (3.9 to 3.14), a web plugin on the Python of the Pyodide release used by DataLab-Web (3.12). DataLab resolves the complete dependency graph against the actual installation; publication metadata checks are not a functional certification of every platform. -- When a pull request is merged, the catalog is rebuilt and published: `catalog.json`, a copy of every wheel in `wheels//`, and a human-readable `index.html`. Wheels are downloaded again at each build and must still match the reviewed digests. +- Desktop entries may use `kind: module` with an exact `.py` GitHub release asset, summary, license and version. The source can remain unchanged: available metadata is read statically, and the entry supplies what cannot be determined without execution. The page labels these artifacts **Python file**, and does not include them in the Web filter. +- For each wheel release, the CI downloads the wheel from the GitHub release (or from PyPI), checks its SHA-256 digest against the entry, and inspects it without running its code: pure-Python plugin, `datalab.plugins` (desktop) and/or `datalab.web_plugins` (web) entry points, declared requirements and OSI-approved license. Desktop wheels may also list reviewed dependency wheels, including native wheels, which are verified and mirrored with the plugin. Web requirements must still be provided by its host. Python compatibility is checked per target: a desktop plugin must install on at least one Python version supported by DataLab (3.9 to 3.14), a web plugin on the Python of the Pyodide release used by DataLab-Web (3.12). DataLab resolves the complete dependency graph against the actual installation; publication metadata checks are not a functional certification of every platform. +- When a pull request is merged, the catalog is rebuilt and published: `catalog.json`, copies of wheels in `wheels//` and modules in `modules//`, and a human-readable `index.html`. All artifacts are downloaded again and must still match the reviewed digests. - The page uses content-versioned URLs for its stylesheet and script, so browsers fetch updated assets instead of reusing an older cached version. - Plugins from DataLab-Platform repositories are marked **official**, others **community**. IDs starting with `org.datalab.` are reserved for official plugins. @@ -33,6 +36,8 @@ Any author may list a plugin by opening a pull request that adds one small YAML A release may additionally list `dependency_wheels`, each with `filename`, mirrored `url`, `sha256` and `size`, plus upstream and metadata information. The field is optional: existing catalogs and self-contained plugins keep their format and behavior. Dependency artifacts belong to Desktop only; this does not add a package-index installer to DataLab-Web. +For modules, a plugin record adds `kind: "module"` and its real import name in `module`, without a fictitious `distribution`. Its releases still use `filename`, `url`, `sha256`, `size`, version, optional Python constraint and the `desktop` target, but cannot include dependency wheels. Existing wheel records need no new fields. An older wheel-only client skips module artifacts as non-installable. + Plugins may also declare an optional `icon` HTTPS URL, used only by the web page. The image is loaded directly by the browser, not downloaded or mirrored by the catalog build. Initials remain the fallback if the image is absent or unavailable, including on a private mirror without access to the image host. See [CONTRIBUTING.md](CONTRIBUTING.md#optional-icon). ## Development @@ -45,3 +50,5 @@ python tools/catalog.py build --output site ``` [tools/wheels.py](tools/wheels.py) is a verbatim copy of `datalab/plugins/wheels.py` from DataLab: update both together. A test compares them when a sibling DataLab checkout is available. Update `TARGET_PYTHONS` in [tools/catalog.py](tools/catalog.py) when DataLab supports a new Python version or DataLab-Web moves to another Pyodide release. + +[tools/modules.py](tools/modules.py) is likewise copied from `datalab/plugins/modules.py`, with an equality test. The module inspector has no Qt dependency and never imports the artifact. The optional Desktop mirror integration test requires the updated client; the remaining publication tests also run against the latest published DataLab reference environment. diff --git a/schema/plugin-entry.schema.json b/schema/plugin-entry.schema.json index 18a4b63..dc2f82f 100644 --- a/schema/plugin-entry.schema.json +++ b/schema/plugin-entry.schema.json @@ -18,6 +18,12 @@ "minLength": 1, "maxLength": 80 }, + "kind": { + "description": "Distribution format, not a functional plugin category. Defaults to wheel.", + "enum": ["wheel", "module"] + }, + "summary": {"type": "string", "minLength": 1}, + "license": {"type": "string", "minLength": 1}, "repository": { "description": "Public source repository, also used to find GitHub releases.", "type": "string", @@ -56,6 +62,23 @@ } }, "allOf": [ + { + "if": {"properties": {"kind": {"const": "module"}}, "required": ["kind"]}, + "then": { + "required": ["summary", "license"], + "properties": { + "releases": { + "items": { + "required": ["filename"], + "properties": { + "source": {"const": "github"}, + "dependency_wheels": {"maxItems": 0} + } + } + } + } + } + }, { "if": { "properties": {"status": {"enum": ["deprecated", "revoked"]}}, @@ -71,6 +94,15 @@ "required": ["version"], "properties": { "version": {"type": "string", "minLength": 1}, + "filename": { + "description": "Exact GitHub release asset for a single-file module.", + "type": "string", + "pattern": "^datalab_[A-Za-z0-9_]+\\.py$" + }, + "requires_python": { + "description": "Optional Python compatibility constraint for a module release.", + "type": "string" + }, "sha256": { "description": "SHA-256 digest of the wheel. The catalog check prints it when missing.", "type": "string", diff --git a/tests/test_catalog.py b/tests/test_catalog.py index c8a947f..b36a9f5 100644 --- a/tests/test_catalog.py +++ b/tests/test_catalog.py @@ -5,10 +5,12 @@ from __future__ import annotations import hashlib +import importlib.util import io import json import sys import zipfile +from importlib.machinery import ModuleSpec, PathFinder from pathlib import Path import pytest @@ -168,6 +170,186 @@ def test_missing_digest_reports_the_value_to_declare( catalog.build_catalog(entries, HOST) +def module_entry(served: dict, tmp_path: Path, source: bytes) -> tuple[Path, str]: + """Publish an unchanged Python file as an exact GitHub release asset.""" + repository = "https://github.com/someone/single-file-plugin" + filename = "datalab_single_probe.py" + digest = publish_on_github(served, repository, "v1.0", (filename, source)) + entries = tmp_path / "plugins" + write_entry( + entries, + "io.github.someone.single", + f"""id: io.github.someone.single +name: Single-file example +kind: module +summary: An unchanged historical script +license: BSD-3-Clause +repository: {repository} +capabilities: [processing] +releases: + - version: "1.0" + filename: {filename} + sha256: {digest} +""", + ) + return entries, filename + + +def test_single_file_release_is_mirrored_without_execution( + served: dict, tmp_path: Path +) -> None: + """A public artifact is usable offline without building a wheel.""" + marker = tmp_path / "must-not-run" + source = f"open({str(marker)!r}, 'w').write('executed')\n".encode() + entries, filename = module_entry(served, tmp_path, source) + site = tmp_path / "site" + result = catalog.build_catalog(entries, HOST, site) + (plugin,) = result["plugins"] + assert plugin["kind"] == "module" + assert plugin["module"] == filename[:-3] + assert "distribution" not in plugin + (release,) = plugin["releases"] + assert release["targets"] == ["desktop"] + assert (site / release["url"]).read_bytes() == source + assert not marker.exists() + page = (site / "index.html").read_text("utf-8") + assert "Python file" in page + assert 'data-targets="desktop"' in page + + +def test_single_file_published_mirror_installs_with_the_desktop_client( + served: dict, tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + host = importlib.util.find_spec("datalab") + plugins = ( + PathFinder.find_spec("datalab.plugins", host.submodule_search_locations) + if host is not None and host.submodule_search_locations is not None + else None + ) + if ( + plugins is None + or plugins.submodule_search_locations is None + or PathFinder.find_spec( + "datalab.plugins.modules", plugins.submodule_search_locations + ) + is None + ): + pytest.skip("Requires the updated Desktop client, not the legacy Qt host") + marker = tmp_path / "must-not-run" + source = f"open({str(marker)!r}, 'w').write('executed')\n".encode() + entries, _filename = module_entry(served, tmp_path, source) + site = tmp_path / "site" + catalog.build_catalog(entries, HOST, site) + # Exercise the actual Desktop client on the generated offline mirror. + from datalab.plugins.catalog import download_release, read_folder_catalog + from datalab.plugins.store import InstalledPluginStore + + monkeypatch.setattr( + "urllib.request.urlopen", + lambda *_args, **_kwargs: pytest.fail("Offline mirror used the network"), + ) + (offline,), warnings = read_folder_catalog(str(site)) + assert not warnings + candidate = offline.get_installable_release() + assert candidate is not None + download = tmp_path / "download" + download.mkdir() + artifact = download_release(candidate, str(download)) + store = InstalledPluginStore(str(tmp_path / "installed")) + installed = store.install_module( + artifact, + expected_sha256=candidate.sha256, + catalog_id=offline.id, + catalog_name=offline.name, + catalog_version=candidate.version, + ).plugin + assert installed.version == "1.0" + assert Path(store.get_path(installed)).read_bytes() == source + assert not marker.exists() + + +def test_optional_integration_does_not_import_the_legacy_qt_host( + served: dict, tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """DataLab 1.3 has plugins.py, which imports Qt before a child lookup.""" + original = PathFinder.find_spec + legacy = ModuleSpec("datalab.plugins", loader=None, is_package=False) + + def find_spec(name, path=None, target=None): + if name == "datalab.plugins": + return legacy + return original(name, path, target) + + monkeypatch.setattr(PathFinder, "find_spec", find_spec) + with pytest.raises(pytest.skip.Exception, match="updated Desktop client"): + test_single_file_published_mirror_installs_with_the_desktop_client( + served, tmp_path, monkeypatch + ) + + +@pytest.mark.parametrize( + ("source", "detail"), + [ + (b"", "empty"), + (b"class Broken(", "Invalid Python"), + ( + b"from datalab.plugins import PluginInfo\n" + b"class Example:\n" + b' PLUGIN_INFO = PluginInfo(id="org.example.other", version="1.0")', + "plugin_id", + ), + ( + b"from datalab.plugins import PluginInfo\n" + b"class Example:\n" + b' PLUGIN_INFO = PluginInfo(id="io.github.someone.single", ' + b'version="2.0")', + "version", + ), + ], +) +def test_single_file_invalid_source_or_conflicting_metadata( + served: dict, tmp_path: Path, source: bytes, detail: str +) -> None: + entries, _filename = module_entry(served, tmp_path, source) + with pytest.raises(catalog.CatalogError, match=detail): + catalog.build_catalog(entries, HOST) + + +def test_single_file_requires_exact_asset_and_has_no_dependency_installation( + served: dict, tmp_path: Path +) -> None: + entries, filename = module_entry(served, tmp_path, b"VALUE = 1\n") + path = entries / "io.github.someone.single.yaml" + original = path.read_text("utf-8") + path.write_text(original.replace(filename, "datalab_missing.py"), encoding="utf-8") + with pytest.raises(catalog.CatalogError, match="expected the module asset"): + catalog.build_catalog(entries, HOST) + path.write_text( + original.replace( + f" filename: {filename}", + f" filename: {filename}\n source: pypi\n distribution: anything", + ), + encoding="utf-8", + ) + with pytest.raises(catalog.CatalogError, match="github"): + catalog.build_catalog(entries, HOST) + + +def test_module_inspector_copy_matches_desktop() -> None: + """The shared module contract must remain byte-for-byte identical.""" + desktop = ( + Path(__file__).resolve().parents[2] + / "DataLab" + / "datalab" + / "plugins" + / "modules.py" + ) + if desktop.is_file(): + assert ( + catalog.ROOT / "tools" / "modules.py" + ).read_bytes() == desktop.read_bytes() + + def test_additional_desktop_dependencies_are_verified_and_mirrored( served: dict, tmp_path: Path ) -> None: diff --git a/tools/catalog.py b/tools/catalog.py index a7a011d..7869a23 100644 --- a/tools/catalog.py +++ b/tools/catalog.py @@ -37,10 +37,12 @@ import jsonschema import yaml +from packaging.specifiers import SpecifierSet from packaging.utils import canonicalize_name from packaging.version import InvalidVersion, Version sys.path.insert(0, str(Path(__file__).resolve().parent)) +import modules # noqa: E402 (verbatim copy of datalab/plugins/modules.py) import wheels # noqa: E402 (verbatim copy of datalab/plugins/wheels.py) ROOT = Path(__file__).resolve().parents[1] @@ -176,6 +178,13 @@ def resolve_wheel(entry: dict, release: dict) -> tuple[str, str]: (asset["name"], asset["browser_download_url"]) for asset in info["assets"] ] where = f"GitHub release {tag} of {entry['repository']}" + if entry.get("kind") == "module": + selected = [item for item in files if item[0] == release["filename"]] + if len(selected) != 1: + raise CatalogError( + f"{where}: expected the module asset {release['filename']}" + ) + return selected[0] pure_wheels = [item for item in files if item[0].endswith("-py3-none-any.whl")] if len(pure_wheels) != 1: raise CatalogError( @@ -328,6 +337,53 @@ def check_release( f"{label}: the wheel digest must be declared, add to this release: " f"sha256: {digest}" ) + if entry.get("kind") == "module": + try: + manifest = modules.inspect_module(data, filename=filename) + manifest = modules.apply_catalog_metadata( + manifest, + plugin_id=entry["id"], + name=entry["name"], + version=release["version"], + summary=entry["summary"], + ) + if release.get("requires_python"): + supported = SpecifierSet(release["requires_python"]) + if not any( + Version(version) in supported + for version in TARGET_PYTHONS["desktop"] + ): + raise ValueError("Module has no supported Python version") + except ValueError as exc: + raise CatalogError(f"{label}: {exc}") from exc + if entry["license"] not in ALLOWED_LICENSES: + raise CatalogError( + f"{label}: license is not an accepted OSI-approved license" + ) + if release.get("dependency_wheels"): + raise CatalogError( + f"{label}: module dependencies must be provided by the host" + ) + if release.get("source", "github") != "github": + raise CatalogError(f"{label}: modules must use GitHub release assets") + record = { + "version": manifest["version"], + "filename": filename, + "url": f"modules/{digest}/{filename}", + "sha256": digest, + "size": len(data), + "requires_python": release.get("requires_python"), + "targets": ["desktop"], + "source_url": url, + } + if "yanked" in release: + record["yanked"] = release["yanked"] + package = { + "module": manifest["name"], + "summary": entry["summary"], + "license": entry["license"], + } + return record, package, data, {} groups, metadata = _read_wheel_metadata(data) dependency_records, dependency_files, dependency_versions = check_dependency_wheels( release @@ -406,6 +462,14 @@ def check_entry( "capabilities": entry.get("capabilities", []), "keywords": entry.get("keywords", []), } + if entry.get("kind") == "module": + plugin["kind"] = "module" + names = { + modules.module_name(release["filename"]) for release in entry["releases"] + } + if len(names) != 1: + raise CatalogError(f"{path.name}: module file names must remain stable") + plugin["module"] = next(iter(names)) if "icon" in entry: plugin["icon"] = entry["icon"] if "status_reason" in entry: @@ -429,9 +493,11 @@ def check_entry( files.update(dependency_files) records.sort(key=lambda item: item[0], reverse=True) distributions = { - canonicalize_name(package["distribution"]) for _v, _r, package in records + canonicalize_name(package["distribution"]) + for _v, _r, package in records + if "distribution" in package } - if len(distributions) != 1: + if entry.get("kind", "wheel") == "wheel" and len(distributions) != 1: raise CatalogError(f"{path.name}: releases come from different distributions") plugin.update(records[0][2]) plugin["releases"] = [record for _version, record, _package in records] @@ -463,7 +529,16 @@ def build_catalog( plugins.append(plugin) files.update(plugin_files) owners: dict[str, str] = {} + module_owners: dict[str, str] = {} for plugin in plugins: + if plugin.get("module"): + name = plugin["module"].casefold() + if name in module_owners: + errors.append( + f"{plugin['id']}: module {name} is already listed by " + f"{module_owners[name]}" + ) + module_owners[name] = plugin["id"] if "distribution" not in plugin: continue key = canonicalize_name(plugin["distribution"]) @@ -556,6 +631,8 @@ def _render_card(plugin: dict) -> str: f'{esc(status.title())}' ) badges += [f'{esc(label)}' for label in capabilities] + if plugin.get("kind") == "module": + badges.append('Python file') badges += [ f'{esc(target.title())}' for target in targets ] diff --git a/tools/modules.py b/tools/modules.py new file mode 100644 index 0000000..dbd88b8 --- /dev/null +++ b/tools/modules.py @@ -0,0 +1,249 @@ +# Copyright (c) DataLab Platform Developers, BSD 3-Clause license, see LICENSE file. + +"""Inspect single-file plugins without importing or executing their code. + +The public catalog keeps a verbatim copy of this Qt-free module. Only literal +strings and references to previously declared string constants are read. +""" + +from __future__ import annotations + +import ast +import hashlib +import os.path as osp +from typing import Any + +from packaging.version import InvalidVersion, Version + +MAX_MODULE_BYTES = 4 * 1024 * 1024 +_FIELDS = { + "id": "plugin_id", + "name": "display_name", + "version": "version", + "description": "summary", +} + + +class ModuleInspectionError(ValueError): + """A source file is not an acceptable single-file plugin artifact.""" + + +def module_name(filename: str) -> str: + """Return the importable name of a ``datalab_*.py`` file.""" + name, extension = osp.splitext(filename) + if ( + "/" in filename + or "\\" in filename + or extension != ".py" + or not name.startswith("datalab_") + or len(name) <= len("datalab_") + or not name.isidentifier() + ): + raise ModuleInspectionError( + "A plugin module file name must look like datalab_.py" + ) + return name + + +def _string(node: ast.expr, constants: dict[str, str]) -> str | None: + if isinstance(node, ast.Constant) and isinstance(node.value, str): + return node.value + if isinstance(node, ast.Name): + return constants.get(node.id) + return None + + +def _assignment(node: ast.stmt) -> tuple[str, ast.expr] | None: + if ( + isinstance(node, ast.Assign) + and len(node.targets) == 1 + and isinstance(node.targets[0], ast.Name) + ): + return node.targets[0].id, node.value + if ( + isinstance(node, ast.AnnAssign) + and isinstance(node.target, ast.Name) + and node.value is not None + ): + return node.target.id, node.value + return None + + +def _metadata(tree: ast.Module) -> tuple[dict[str, str], list[str]]: + constants: dict[str, str] = {} + constructors: set[str] = set() + namespaces: set[str] = set() + records: list[dict[str, str]] = [] + for statement in tree.body: + if isinstance(statement, (ast.Import, ast.ImportFrom)): + for alias in statement.names: + bound = alias.asname or alias.name.split(".", maxsplit=1)[0] + constants.pop(bound, None) + constructors.discard(bound) + namespaces.discard(bound) + if isinstance(statement, ast.ImportFrom) and statement.module in ( + "datalab.plugins", + "datalab.plugins.base", + ): + constructors.update( + alias.asname or alias.name + for alias in statement.names + if alias.name == "PluginInfo" + ) + elif isinstance(statement, ast.Import): + namespaces.update( + alias.asname or alias.name + for alias in statement.names + if alias.name == "datalab.plugins" and alias.asname + ) + assignment = _assignment(statement) + if assignment is not None: + name, value = assignment + text = _string(value, constants) + constants.pop(name, None) + constructors.discard(name) + namespaces.discard(name) + if text is not None: + constants[name] = text + if not isinstance(statement, ast.ClassDef): + if assignment is None and not isinstance( + statement, (ast.Import, ast.ImportFrom) + ): + for node in ast.walk(statement): + if isinstance(node, ast.Name) and isinstance(node.ctx, ast.Store): + constants.pop(node.id, None) + constructors.discard(node.id) + namespaces.discard(node.id) + continue + local = dict(constants) + local_constructors = set(constructors) + local_namespaces = set(namespaces) + for member in statement.body: + assignment = _assignment(member) + if assignment is None: + for node in ast.walk(member): + if isinstance(node, ast.Name) and isinstance(node.ctx, ast.Store): + local.pop(node.id, None) + local_constructors.discard(node.id) + local_namespaces.discard(node.id) + continue + name, value = assignment + if name != "PLUGIN_INFO": + text = _string(value, local) + local.pop(name, None) + local_constructors.discard(name) + local_namespaces.discard(name) + if text is not None: + local[name] = text + continue + record: dict[str, str] = {} + if isinstance(value, ast.Call): + function = value.func + recognized = ( + isinstance(function, ast.Name) and function.id in local_constructors + ) or ( + isinstance(function, ast.Attribute) + and function.attr == "PluginInfo" + and isinstance(function.value, ast.Name) + and function.value.id in local_namespaces + ) + if recognized and not any( + keyword.arg is None for keyword in value.keywords + ): + for keyword in value.keywords: + if keyword.arg in _FIELDS: + text = _string(keyword.value, local) + if text is not None: + record[_FIELDS[keyword.arg]] = text + records.append(record) + warnings: list[str] = [] + if len(records) != 1: + warnings.append( + "Plugin metadata is not unique or not statically readable; " + "the catalog describes the module as a whole" + ) + return {}, warnings + record = records[0] + if "version" in record and record["version"]: + try: + Version(record["version"]) + except InvalidVersion: + warnings.append("The declared version is not comparable; using the digest") + record.pop("version") + if not all(record.get(field) for field in ("plugin_id", "display_name", "version")): + warnings.append( + "Some plugin metadata is unavailable without executing the file" + ) + return record, warnings + + +def inspect_module(path_or_bytes: str | bytes, *, filename: str) -> dict[str, Any]: + """Return file facts and safely readable plugin metadata. + + Valid Python is not necessarily a working or trustworthy plugin. Missing + metadata stays unknown; callers may supply it explicitly in a catalog. + """ + name = module_name(filename) + if isinstance(path_or_bytes, str): + with open(path_or_bytes, "rb") as stream: + data = stream.read(MAX_MODULE_BYTES + 1) + else: + data = path_or_bytes + if not data: + raise ModuleInspectionError("Plugin module is empty") + if len(data) > MAX_MODULE_BYTES: + raise ModuleInspectionError( + f"Plugin module exceeds the {MAX_MODULE_BYTES} byte size limit" + ) + try: + tree = ast.parse(data, filename=filename) + # Parsing alone accepts e.g. a return outside a function. + compile(tree, filename, "exec") + except (SyntaxError, ValueError, UnicodeError, RecursionError) as exc: + raise ModuleInspectionError(f"Invalid Python plugin {filename}: {exc}") from exc + metadata, warnings = _metadata(tree) + return { + "filename": filename, + "name": name, + "sha256": hashlib.sha256(data).hexdigest(), + "size_bytes": len(data), + "version": metadata.get("version", ""), + "plugin_id": metadata.get("plugin_id", ""), + "display_name": metadata.get("display_name", ""), + "summary": metadata.get("summary", ""), + "warnings": warnings, + } + + +def apply_catalog_metadata( + manifest: dict[str, Any], + *, + plugin_id: str, + name: str, + version: str, + summary: str = "", +) -> dict[str, Any]: + """Supplement unknown metadata, refusing a known identity/version conflict.""" + if version: + Version(version) + for field, expected in (("plugin_id", plugin_id), ("version", version)): + actual = manifest[field] + if ( + actual + and expected + and ( + Version(actual) != Version(expected) + if field == "version" + else actual != expected + ) + ): + raise ModuleInspectionError( + f"Catalog {field} {expected!r} differs from the file: {actual!r}" + ) + return { + **manifest, + "plugin_id": plugin_id, + "display_name": name, + "version": version or manifest["version"], + "summary": summary or manifest["summary"], + }