From 490c09074f0ecc81ecb0c207a3205c2911b6f508 Mon Sep 17 00:00:00 2001 From: Harrison O'Connor-Hoover Date: Wed, 30 Sep 2026 20:21:57 -0400 Subject: [PATCH 1/4] Isolate plugin commands and share durable run identity --- HANDOFF.md | 29 ++-- docs/decisions.md | 8 ++ docs/quality-review-2026-09-30.md | 61 ++++++++ src/dander/cli/entrypoint.py | 12 +- src/dander/cli/main.py | 117 +-------------- src/dander/cli/plugins_command.py | 136 ++++++++++++++++++ src/dander/control/orchestration.py | 39 ++++- src/dander/control/postgresql_run_store.py | 37 +---- src/dander/control/s3_run_store.py | 37 +---- tests/cli/test_plugins_cli.py | 17 +-- tests/cli/test_plugins_entrypoint.py | 70 +++++++++ tests/control/test_postgresql_run_store.py | 4 + tests/control/test_s3_run_store.py | 3 + ...R-285-quality-review-command-boundaries.md | 31 ++++ 14 files changed, 392 insertions(+), 209 deletions(-) create mode 100644 docs/quality-review-2026-09-30.md create mode 100644 src/dander/cli/plugins_command.py create mode 100644 tests/cli/test_plugins_entrypoint.py create mode 100644 tickets/DANDER-285-quality-review-command-boundaries.md diff --git a/HANDOFF.md b/HANDOFF.md index 1d10507b..06b983df 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -2,31 +2,34 @@ ## Finished -- Reconciled retained August 27–28 Control/Fargate/Spark successes and corrected the obsolete DANDER-251 non-execution claim. -- Updated five completed tickets, Druff hosted-mode documentation, and Salesforce plugin write-command boundaries. -- Prepared Salesforce 0.3.2, ServiceNow 0.2.3, and Dander catalog/install-check patches outside this documentation change. +- Scanned repository structure, duplication, import boundaries, error handling, and test health; recorded scope and remaining priorities in `docs/quality-review-2026-09-30.md`. +- Isolated plugin commands so help and catalog search avoid bootstrap, deployment, and provider modules. +- Shared the existing durable-run identity rule between S3 and PostgreSQL, with storage behavior preserved. ## Try It -Read `docs/control-execution-history.md`, then `docs/support-status.md`. Prepared package and catalog patches are in `/tmp/dander-curated-compat-20260930/`. +Run `uv run dander plugins --help` and `uv run dander plugins search incident`. Read the quality review for findings and the limits of the scan. ## Checks -- Documentation: relative links, retained output equality, source/image identities, and `git diff --check` passed. -- Prepared fixes: 93 Salesforce and 23 ServiceNow tests passed on public Dander 0.7.1 and 0.9.0rc20, current 0.9.0rc32, and each package's locked environment. Package lint, format, and typing passed. -- Prepared Dander patch: 37 focused tests, full Ruff lint/format, canonical strict typing (507 files), contract drift, and outside-checkout wheel resolution/metadata/entry-point checks passed using local candidate wheels. +- Full suite with disposable local PostgreSQL 17: **2,419 passed**, one existing Starlette/httpx deprecation warning. +- Focused CLI/storage/lifecycle suite: **75 passed**; final CLI test-import adjustment: **10 passed**. +- Ruff lint/format, canonical strict typing (**509 files**), Control contract drift, documentation links, and diff checks passed. +- Compared moved command bodies and identity comparisons structurally against the original code; behavior-bearing expressions match. Protected CI and merge results are available on the task's attached quality-review PR. ## Decisions -- Retained live evidence qualifies only its named revisions and workloads; no new cloud runs or support promotion. -- Do not publish catalog pins until the corresponding packages exist on PyPI. +- Use the existing lightweight command-dispatch pattern; preserve command options, installer pins, and storage schemas. +- Keep provider-specific SQL and recovery explicit; defer broader extraction until changes demonstrate a shared responsibility. ## Remaining -- Await the requested narrow exception to AGENTS.md's single-writable-repository rule for the two existing connector repositories. -- Then apply the prepared connector patches, complete protected CI/publication, verify PyPI, apply the Dander catalog patch, and merge with exact-main CI. +- Large CLI/bootstrap and warehouse/deployment modules remain the next maintenance targets; see the review's specific recommendations. +- Curated connector releases remain separately blocked on the requested exception to the single-writable-repository rule. Prepared patches remain in `/tmp/dander-curated-compat-20260930/` and the preserved connector branch. +- Existing test-client deprecation warning remains; no live cloud qualification was rerun. ## Review First -- `docs/control-execution-history.md` and corrected support boundaries. -- `/tmp/dander-curated-compat-20260930/*-compatibility.patch` for the pending package work. +- `docs/quality-review-2026-09-30.md` for the assessment and scope. +- `src/dander/cli/plugins_command.py` and `src/dander/cli/entrypoint.py` for command isolation. +- `src/dander/control/orchestration.py` for the shared run identity rule. diff --git a/docs/decisions.md b/docs/decisions.md index cefe62d7..8c6e4478 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -1,5 +1,13 @@ # Engineering Decisions +## 2026-09-30 — Keep command loading and run identity at their owning boundaries + +- Plugin commands live in `cli/plugins_command.py`. The console dispatcher selects their group + directly, like hosted Control; project configuration is loaded only by installation so help and + catalog search do not depend on deployment modules. CLI options and outputs remain compatible. +- `same_run_identity` lives beside the provider-neutral `RunStore` contract. S3 and PostgreSQL use + the same existing 14-field comparison; storage schemas and lifecycle transitions are unchanged. + ## 2026-09-08 — Recover pending PostgreSQL runs without reading completed history - PostgreSQL Control schema v4 derives `needs_reconciliation` from canonical run records and diff --git a/docs/quality-review-2026-09-30.md b/docs/quality-review-2026-09-30.md new file mode 100644 index 00000000..757dc0db --- /dev/null +++ b/docs/quality-review-2026-09-30.md @@ -0,0 +1,61 @@ +# Code quality review — September 30, 2026 + +Reviewed from `425045ff2bbd5ba8b6d6d8eec3c28fb0f59bf525`, with changes tracked in +[DANDER-285](../tickets/DANDER-285-quality-review-command-boundaries.md). + +## Assessment and scope + +The codebase has a sound core: typed domain contracts, explicit provider boundaries, conditional +storage updates, and substantial tests. Its main maintenance pressure is concentrated in large +command, orchestration, and provider modules. A wholesale rewrite is not justified by this review. + +The inventory covered 1,409 tracked files. Python structural analysis covered the tracked Python +sources and tests; the runtime inventory contained 270 modules and approximately 80,000 lines, +excluding scaffold templates. Searches examined duplicate function bodies, large modules/functions, +broad exception handlers, unfinished markers, typing suppressions, and subprocess boundaries. + +Manual review followed selected paths through CLI dispatch, plugin loading, Control HTTP and run +storage, graph mutation, pipeline execution, ingestion, warehouse staging, assertions, BigQuery +telemetry/fencing, and CI scope selection. This combines a whole-repository scan with targeted +reading; it is not a line-by-line review of every file or new live-cloud qualification. + +## Changes + +| Finding | Change | Evidence | +| --- | --- | --- | +| `plugins search` loaded 13 bootstrap, 3 deployment, and 55 provider modules via the root CLI. Offline discovery depended on unrelated command code. | Move install/scaffold/search into `cli/plugins_command.py`; dispatch the plugin group directly and load project configuration only for installation. | New subprocess tests failed before the change and pass afterward. Help/search import none of those module families. Existing CLI tests retain exact pins, version-2 manifests, scaffold behavior, and catalog filtering. | +| S3 and PostgreSQL stores duplicated the same 14-field durable-identity comparison. A future change could update only one implementation. | Put `same_run_identity` beside the `RunStore` contract and remove the duplicated implementations and redundant run-ID checks. | Both backend tests reject a changed plan revision, retain the original stored snapshot, and still accept lifecycle transitions with conditional revisions. | + +No dependency, schema, provider payload, or connector package pin changed. + +## Existing protections confirmed + +- Object-store graph mutation rules and generic assertion planning already have shared owners. +- BigQuery ingestion and graph work already use the shared job-measurement collector. +- The inspected Control mutation routes offload synchronous storage work from the async handler. +- PostgreSQL recovery has a pending-run query; it does not rely on scanning all completed history. +- CI scope detection includes deleted files and both sides of renames. +- Reviewed broad exception handlers at plugin and file boundaries convert external failures or + clean temporary files; they do not justify a blanket exception-handling rewrite. + +## Remaining maintenance priorities + +1. **CLI and bootstrap composition:** `cli/main.py` still owns several command groups and wide + bootstrap signatures. When those commands next change, extract one cohesive group or pass the + existing typed options object through the relevant boundary. Preserve flags and defaults. +2. **Large provider implementations:** warehouse writers and deployment renderers remain large. + Extract demonstrated shared semantics; keep SQL dialects, transactions, and provider-specific + recovery explicit. Similar-looking SQL is not enough reason to add a generic writer framework. +3. **Support evidence:** passing tests do not close the separately tracked curated-package + compatibility release or prove new cloud configurations. Use [support status](support-status.md) + and [execution history](control-execution-history.md) for those boundaries. + +## Verification + +- Baseline: 2,416 tests passed with a disposable local PostgreSQL 17 database; Ruff lint/format, + canonical strict typing, and generated Control contract drift checks passed. +- Changed-path validation: 75 plugin CLI, console dispatch, S3/PostgreSQL storage, and lifecycle + tests passed. The new import tests demonstrated the pre-change coupling before passing. +- Final full-suite and protected CI results are recorded in the handoff and pull request. +- The baseline and focused runs emitted one existing Starlette/httpx test-client deprecation + warning. No live provider workload was submitted by this review. diff --git a/src/dander/cli/entrypoint.py b/src/dander/cli/entrypoint.py index 33861262..3f71612b 100644 --- a/src/dander/cli/entrypoint.py +++ b/src/dander/cli/entrypoint.py @@ -1,4 +1,4 @@ -"""Lightweight console dispatcher that keeps hosted Control startup provider-free.""" +"""Load hosted Control and plugin commands without the full deployment CLI.""" from __future__ import annotations @@ -25,6 +25,16 @@ def dispatch(arguments: Sequence[str]) -> None: hosted_app(args=args, prog_name="dander") return + if args[:1] == ["plugins"]: + import typer + + from dander.cli.plugins_command import plugins_app + + plugin_app = typer.Typer() + plugin_app.add_typer(plugins_app, name="plugins") + plugin_app(args=args, prog_name="dander") + return + from dander.cli.main import app app(args=args, prog_name="dander") diff --git a/src/dander/cli/main.py b/src/dander/cli/main.py index fd815cdc..c6b7cdca 100644 --- a/src/dander/cli/main.py +++ b/src/dander/cli/main.py @@ -6,9 +6,6 @@ import os import re import shlex -import shutil as shutil -import subprocess as subprocess -import sys from pathlib import Path from typing import TYPE_CHECKING @@ -55,6 +52,7 @@ ) from dander.cli.kubernetes_command import register_kubernetes_commands from dander.cli.oci_command import register_oci_commands +from dander.cli.plugins_command import plugins_app from dander.cli.provider_runtime import build_catalog_publisher, build_secret_store from dander.cli.qualification_command import register_qualification_command from dander.cli.run_command import ( @@ -88,16 +86,12 @@ from dander.pipeline.graph_service import GraphDocumentError, serve_graph_file from dander.plugins import ( ConnectorPluginError, - PluginScaffoldError, load_connector_plugins, - scaffold_connector_plugin, - search_connector_catalog, ) from dander.project import ( ProjectConfigError, ProjectScaffoldError, load_project_config, - load_project_plugins, scaffold_project, ) from dander.sandbox import GuardedFreeTierVerifier, SandboxSafetyError @@ -129,7 +123,6 @@ verify_app = typer.Typer(help="Verify deployed resources with read-only checks.") metadata_app = typer.Typer(help="Inspect the durable metadata spine and run ledger.") graph_app = typer.Typer(help="Open validated pipeline graphs to local visual editors.") -plugins_app = typer.Typer(help="Install and inspect explicitly pinned connector plugins.") connector_app = typer.Typer(help="Inspect and check configured connector capabilities.") app.add_typer(config_app, name="config") app.add_typer(control_app, name="control") @@ -218,114 +211,6 @@ def validate_project( console.print(f"[green]{summary}[/green]") -@plugins_app.command("install") -def install_plugins( - project_config: Path = typer.Option(_DEFAULT_PROJECT_CONFIG, "--config"), # noqa: B008 - platforms_config: Path | None = typer.Option(None, "--platforms-config"), # noqa: B008 - deployment: str | None = typer.Option(None, "--deployment"), -) -> None: - """Install the manifest's exact connector-plugin package pins.""" - try: - plugins = load_project_plugins(project_config) - except ProjectConfigError as error: - raise ClickException(str(error)) from error - requirements = [ - f"{plugin.distribution}=={plugin.version}" for _, plugin in sorted(plugins.items()) - ] - if not requirements: - console.print("No connector plugins are declared in dander.yaml.") - return - plugin_count = len(requirements) - # Keep the package running this command in the resolver transaction. Plugin - # compatibility constraints must fail clearly instead of silently replacing - # Dander with an older release inside a source-free runtime image. - requirements.append(f"dander-platform=={__version__}") - uv_executable = shutil.which("uv") - command = ( - (uv_executable, "pip", "install", "--python", sys.executable, *requirements) - if uv_executable is not None - else (sys.executable, "-m", "pip", "install", *requirements) - ) - try: - completed = subprocess.run( # noqa: S603 - command, - check=False, - ) - except OSError as error: - raise ClickException("Could not start the Python package installer") from error - if completed.returncode != 0: - raise ClickException("Connector plugin installation failed") - try: - load_connector_plugins(plugins) - except ConnectorPluginError as error: - raise ClickException(f"Installed connector plugins are incompatible: {error}") from error - console.print(f"[green]Installed {plugin_count} connector plugin(s).[/green]") - - -@plugins_app.command("scaffold") -def scaffold_plugin( - plugin_id: str = typer.Argument( # noqa: B008 - ..., - help="Lowercase connector identifier, for example acme_crm.", - ), - directory: Path | None = typer.Option( # noqa: B008 - None, - "--directory", - help="New destination directory (defaults to dander-connector-).", - ), - display_name: str | None = typer.Option( # noqa: B008 - None, - "--display-name", - help="Human-readable connector name shown in Druff.", - ), -) -> None: - """Create a tested generic-REST connector plugin without overwriting a path.""" - destination = directory or Path(f"dander-connector-{plugin_id.replace('_', '-')}") - try: - created = scaffold_connector_plugin( - plugin_id, - destination, - display_name=display_name, - ) - except PluginScaffoldError as error: - raise ClickException(str(error)) from error - console.print(f"[green]Created connector plugin at {created}.[/green]") - console.print(f"Next: cd {created} && uv sync --extra dev && uv run pytest") - - -@plugins_app.command("search") -def search_plugins( - query: str = typer.Argument( # noqa: B008 - "", - help="Optional connector name, package, or capability to search for.", - ), -) -> None: - """Search Dander's small curated connector catalog.""" - connectors = search_connector_catalog(query) - if not connectors: - console.print(f"No curated connectors match {query!r}.") - return - - table = Table(title="Dander connector catalog") - table.add_column("Connector") - table.add_column("Package pin") - table.add_column("Dander") - table.add_column("Support") - table.add_column("Validation") - for connector in connectors: - table.add_row( - connector.display_name, - f"{connector.distribution}=={connector.version}", - connector.dander_specifier, - connector.support_status, - connector.validation_status, - ) - console.print(table) - console.print("Exact package pins:") - for connector in connectors: - console.print(f" {connector.distribution}=={connector.version}") - - @connector_app.command("inspect") def inspect_connector( source_or_pipeline: str = typer.Argument( # noqa: B008 diff --git a/src/dander/cli/plugins_command.py b/src/dander/cli/plugins_command.py new file mode 100644 index 00000000..2ae74b97 --- /dev/null +++ b/src/dander/cli/plugins_command.py @@ -0,0 +1,136 @@ +"""Connector plugin commands independent of deployment and runtime setup.""" + +from __future__ import annotations + +import shutil +import subprocess +import sys +from pathlib import Path + +import typer +from click import ClickException +from rich.console import Console +from rich.table import Table + +from dander import __version__ +from dander.plugins import ( + ConnectorPluginError, + PluginScaffoldError, + load_connector_plugins, + scaffold_connector_plugin, + search_connector_catalog, +) + +plugins_app = typer.Typer(help="Install and inspect explicitly pinned connector plugins.") +console = Console() +_DEFAULT_PROJECT_CONFIG = Path("dander.yaml") + + +@plugins_app.command("install") +def install_plugins( + project_config: Path = typer.Option(_DEFAULT_PROJECT_CONFIG, "--config"), # noqa: B008 + platforms_config: Path | None = typer.Option(None, "--platforms-config"), # noqa: B008 + deployment: str | None = typer.Option(None, "--deployment"), +) -> None: + """Install the manifest's exact connector-plugin package pins.""" + from dander.project import ProjectConfigError, load_project_plugins + + try: + plugins = load_project_plugins(project_config) + except ProjectConfigError as error: + raise ClickException(str(error)) from error + requirements = [ + f"{plugin.distribution}=={plugin.version}" for _, plugin in sorted(plugins.items()) + ] + if not requirements: + console.print("No connector plugins are declared in dander.yaml.") + return + plugin_count = len(requirements) + # Keep the package running this command in the resolver transaction. Plugin + # compatibility constraints must fail clearly instead of silently replacing + # Dander with an older release inside a source-free runtime image. + requirements.append(f"dander-platform=={__version__}") + uv_executable = shutil.which("uv") + command = ( + (uv_executable, "pip", "install", "--python", sys.executable, *requirements) + if uv_executable is not None + else (sys.executable, "-m", "pip", "install", *requirements) + ) + try: + completed = subprocess.run( # noqa: S603 + command, + check=False, + ) + except OSError as error: + raise ClickException("Could not start the Python package installer") from error + if completed.returncode != 0: + raise ClickException("Connector plugin installation failed") + try: + load_connector_plugins(plugins) + except ConnectorPluginError as error: + raise ClickException(f"Installed connector plugins are incompatible: {error}") from error + console.print(f"[green]Installed {plugin_count} connector plugin(s).[/green]") + + +@plugins_app.command("scaffold") +def scaffold_plugin( + plugin_id: str = typer.Argument( # noqa: B008 + ..., + help="Lowercase connector identifier, for example acme_crm.", + ), + directory: Path | None = typer.Option( # noqa: B008 + None, + "--directory", + help="New destination directory (defaults to dander-connector-).", + ), + display_name: str | None = typer.Option( # noqa: B008 + None, + "--display-name", + help="Human-readable connector name shown in Druff.", + ), +) -> None: + """Create a tested generic-REST connector plugin without overwriting a path.""" + destination = directory or Path(f"dander-connector-{plugin_id.replace('_', '-')}") + try: + created = scaffold_connector_plugin( + plugin_id, + destination, + display_name=display_name, + ) + except PluginScaffoldError as error: + raise ClickException(str(error)) from error + console.print(f"[green]Created connector plugin at {created}.[/green]") + console.print(f"Next: cd {created} && uv sync --extra dev && uv run pytest") + + +@plugins_app.command("search") +def search_plugins( + query: str = typer.Argument( # noqa: B008 + "", + help="Optional connector name, package, or capability to search for.", + ), +) -> None: + """Search Dander's small curated connector catalog.""" + connectors = search_connector_catalog(query) + if not connectors: + console.print(f"No curated connectors match {query!r}.") + return + + table = Table(title="Dander connector catalog") + table.add_column("Connector") + table.add_column("Package pin") + table.add_column("Dander") + table.add_column("Support") + table.add_column("Validation") + for connector in connectors: + table.add_row( + connector.display_name, + f"{connector.distribution}=={connector.version}", + connector.dander_specifier, + connector.support_status, + connector.validation_status, + ) + console.print(table) + console.print("Exact package pins:") + for connector in connectors: + console.print(f" {connector.distribution}=={connector.version}") diff --git a/src/dander/control/orchestration.py b/src/dander/control/orchestration.py index 39447466..d92a684b 100644 --- a/src/dander/control/orchestration.py +++ b/src/dander/control/orchestration.py @@ -919,7 +919,8 @@ class RunStore(Protocol): Implementations own deterministic keys, idempotency lookup, bounded pagination, and compare-and-swap revisions. ``append_attempt`` must replay an identical record and reject a - different record at the same ``attempt_id``. + different record at the same ``attempt_id``. ``save`` must preserve the submission identity + defined by ``same_run_identity`` while permitting lifecycle and result updates. """ def claim(self, record: RunRecord) -> RunClaim: ... @@ -955,6 +956,41 @@ class PendingRunStore(Protocol): def list_pending(self, *, cursor: str | None, limit: int) -> StoredRunPage: ... +def same_run_identity(first: RunRecord, second: RunRecord) -> bool: + """Compare the original submission fields that snapshot updates must preserve.""" + return ( + first.run_id, + first.environment, + first.project, + first.graph, + first.graph_revision, + first.graph_content_sha256, + first.plan_id, + first.plan_revision, + first.trigger, + first.idempotency_key_sha256, + first.submission_sha256, + first.requested_at, + first.requested_deadline_seconds, + first.created_at, + ) == ( + second.run_id, + second.environment, + second.project, + second.graph, + second.graph_revision, + second.graph_content_sha256, + second.plan_id, + second.plan_revision, + second.trigger, + second.idempotency_key_sha256, + second.submission_sha256, + second.requested_at, + second.requested_deadline_seconds, + second.created_at, + ) + + def run_needs_reconciliation(record: RunRecord) -> bool: """Keep a run eligible until execution, outcome, results, and cleanup are resolved.""" return ( @@ -1449,6 +1485,7 @@ def _require_optional_summary(value: str | None, *, label: str) -> None: "parse_placement_candidate_spec", "parse_size_class_candidate_spec", "run_needs_reconciliation", + "same_run_identity", "transition_run", "validate_submission_plan", ] diff --git a/src/dander/control/postgresql_run_store.py b/src/dander/control/postgresql_run_store.py index 647d2564..da1a357c 100644 --- a/src/dander/control/postgresql_run_store.py +++ b/src/dander/control/postgresql_run_store.py @@ -29,6 +29,7 @@ StoredRun, StoredRunPage, run_needs_reconciliation, + same_run_identity, ) from dander.control.orchestration_serialization import ( OrchestrationSerializationError, @@ -175,7 +176,7 @@ def find_idempotency( def save(self, stored: StoredRun, record: RunRecord) -> StoredRun: """Replace one snapshot only when its opaque revision still matches durable state.""" - if record.run_id != stored.record.run_id or not _same_run_identity(stored.record, record): + if not same_run_identity(stored.record, record): raise RunStoreConflictError("A run snapshot cannot change its durable identity.") expected_revision = _checked_revision(stored.revision) next_revision = _checked_revision(self._revision_factory()) @@ -449,40 +450,6 @@ def _attempt_from_row( return attempt -def _same_run_identity(first: RunRecord, second: RunRecord) -> bool: - return ( - first.run_id, - first.environment, - first.project, - first.graph, - first.graph_revision, - first.graph_content_sha256, - first.plan_id, - first.plan_revision, - first.trigger, - first.idempotency_key_sha256, - first.submission_sha256, - first.requested_at, - first.requested_deadline_seconds, - first.created_at, - ) == ( - second.run_id, - second.environment, - second.project, - second.graph, - second.graph_revision, - second.graph_content_sha256, - second.plan_id, - second.plan_revision, - second.trigger, - second.idempotency_key_sha256, - second.submission_sha256, - second.requested_at, - second.requested_deadline_seconds, - second.created_at, - ) - - def _canonical_mutation_result(value: object) -> bytes: data = _stored_bytes(value, _MAX_MUTATION_BYTES, "mutation result") try: diff --git a/src/dander/control/s3_run_store.py b/src/dander/control/s3_run_store.py index be1b6a74..2a74f0c6 100644 --- a/src/dander/control/s3_run_store.py +++ b/src/dander/control/s3_run_store.py @@ -23,6 +23,7 @@ RunStoreIdempotencyConflictError, StoredRun, StoredRunPage, + same_run_identity, ) from dander.control.orchestration_serialization import ( OrchestrationSerializationError, @@ -365,7 +366,7 @@ def find_idempotency( def save(self, stored: StoredRun, record: RunRecord) -> StoredRun: """Replace one run snapshot only at the caller's opaque S3 revision.""" - if record.run_id != stored.record.run_id or not _same_run_identity(stored.record, record): + if not same_run_identity(stored.record, record): raise RunStoreConflictError("A run snapshot cannot change its durable identity.") revision = self._write_object( self._run_name(record.run_id), @@ -646,40 +647,6 @@ def _validate_snapshot_replay(entry: _IdempotencyEntry, record: RunRecord) -> No raise RunStoreCorruptionError("An S3 run snapshot contradicts its idempotency claim.") -def _same_run_identity(first: RunRecord, second: RunRecord) -> bool: - return ( - first.run_id, - first.environment, - first.project, - first.graph, - first.graph_revision, - first.graph_content_sha256, - first.plan_id, - first.plan_revision, - first.trigger, - first.idempotency_key_sha256, - first.submission_sha256, - first.requested_at, - first.requested_deadline_seconds, - first.created_at, - ) == ( - second.run_id, - second.environment, - second.project, - second.graph, - second.graph_revision, - second.graph_content_sha256, - second.plan_id, - second.plan_revision, - second.trigger, - second.idempotency_key_sha256, - second.submission_sha256, - second.requested_at, - second.requested_deadline_seconds, - second.created_at, - ) - - def _canonical_json(value: object) -> bytes: return json.dumps( value, diff --git a/tests/cli/test_plugins_cli.py b/tests/cli/test_plugins_cli.py index 6674404c..4300831f 100644 --- a/tests/cli/test_plugins_cli.py +++ b/tests/cli/test_plugins_cli.py @@ -2,13 +2,14 @@ from __future__ import annotations +import shutil import subprocess import sys from typing import TYPE_CHECKING from typer.testing import CliRunner -import dander.cli.main as cli_module +import dander.cli.plugins_command as plugins_module from dander import __version__ from dander.cli.main import app @@ -51,9 +52,9 @@ def run(command: tuple[str, ...], *, check: bool) -> subprocess.CompletedProcess captured.append(command) return subprocess.CompletedProcess(command, 0) - monkeypatch.setattr(cli_module.shutil, "which", lambda _: None) - monkeypatch.setattr(cli_module.subprocess, "run", run) - monkeypatch.setattr(cli_module, "load_connector_plugins", lambda _: None) + monkeypatch.setattr(shutil, "which", lambda _: None) + monkeypatch.setattr(subprocess, "run", run) + monkeypatch.setattr(plugins_module, "load_connector_plugins", lambda _: None) result = CliRunner().invoke(app, ["plugins", "install", "--config", str(manifest)]) @@ -81,7 +82,7 @@ def test_plugins_install_is_a_noop_without_declarations( def unexpected_run(*args: object, **kwargs: object) -> None: raise AssertionError((args, kwargs)) - monkeypatch.setattr(cli_module.subprocess, "run", unexpected_run) + monkeypatch.setattr(subprocess, "run", unexpected_run) result = CliRunner().invoke(app, ["plugins", "install", "--config", str(manifest)]) @@ -115,9 +116,9 @@ def run(command: tuple[str, ...], *, check: bool) -> subprocess.CompletedProcess captured.append(command) return subprocess.CompletedProcess(command, 0) - monkeypatch.setattr(cli_module.shutil, "which", lambda _: None) - monkeypatch.setattr(cli_module.subprocess, "run", run) - monkeypatch.setattr(cli_module, "load_connector_plugins", lambda _: None) + monkeypatch.setattr(shutil, "which", lambda _: None) + monkeypatch.setattr(subprocess, "run", run) + monkeypatch.setattr(plugins_module, "load_connector_plugins", lambda _: None) result = CliRunner().invoke(app, ["plugins", "install", "--config", str(manifest)]) diff --git a/tests/cli/test_plugins_entrypoint.py b/tests/cli/test_plugins_entrypoint.py new file mode 100644 index 00000000..f90a98f1 --- /dev/null +++ b/tests/cli/test_plugins_entrypoint.py @@ -0,0 +1,70 @@ +"""The installed plugin CLI stays independent of deployment and runtime commands.""" + +from __future__ import annotations + +import subprocess +import sys +import textwrap +from typing import TYPE_CHECKING + +import pytest + +if TYPE_CHECKING: + from pathlib import Path + + +@pytest.mark.parametrize("arguments", [("plugins", "--help"), ("plugins", "search", "incident")]) +def test_plugins_console_does_not_load_deployment_commands(arguments: tuple[str, ...]) -> None: + script = textwrap.dedent( + """ + import sys + from dander.cli.entrypoint import dispatch + + try: + dispatch(sys.argv[1:]) + except SystemExit as error: + assert error.code == 0 + + forbidden = ( + "dander.cli.main", "dander.bootstrap", "dander.deployment", "dander.providers", + "azure.identity", "boto3", "google.auth", "google.api_core", "dlt", "oci", + ) + loaded = sorted( + name for name in sys.modules + if any(name == prefix or name.startswith(prefix + ".") for prefix in forbidden) + ) + assert loaded == [], loaded + """ + ) + result = subprocess.run( + [sys.executable, "-c", script, *arguments], + check=False, + capture_output=True, + text=True, + timeout=30, + ) + + assert result.returncode == 0, result.stdout + result.stderr + assert "plugins" in result.stdout or "dander-connector-servicenow==" in result.stdout + + +def test_plugins_console_reports_manifest_errors_without_traceback(tmp_path: Path) -> None: + result = subprocess.run( + [ + sys.executable, + "-c", + "from dander.cli.entrypoint import main; main()", + "plugins", + "install", + "--config", + str(tmp_path / "missing.yaml"), + ], + check=False, + capture_output=True, + text=True, + timeout=30, + ) + + assert result.returncode == 1 + assert "Error:" in result.stderr + assert "Traceback" not in result.stdout + result.stderr diff --git a/tests/control/test_postgresql_run_store.py b/tests/control/test_postgresql_run_store.py index 846e170a..95ef7d9f 100644 --- a/tests/control/test_postgresql_run_store.py +++ b/tests/control/test_postgresql_run_store.py @@ -266,6 +266,10 @@ def test_postgresql_run_store_conforms_across_restart( with pytest.raises(RunStoreIdempotencyConflictError): restarted.claim(replace(first, submission_sha256="c" * 64)) + with pytest.raises(RunStoreConflictError, match="durable identity"): + restarted.save(claimed.stored, replace(first, plan_revision="f" * 64)) + assert restarted.get(first.run_id) == claimed.stored + canceling = transition_run( claimed.stored.record, HostedRunState.CANCELING, diff --git a/tests/control/test_s3_run_store.py b/tests/control/test_s3_run_store.py index 1be0ee4f..8ea3ad43 100644 --- a/tests/control/test_s3_run_store.py +++ b/tests/control/test_s3_run_store.py @@ -496,6 +496,9 @@ def test_snapshot_compare_and_swap_and_attempt_immutability() -> None: backend = FakeS3Backend() store = _store(backend) claimed = store.claim(create_run_record(_submission())).stored + with pytest.raises(RunStoreConflictError, match="durable identity"): + store.save(claimed, replace(claimed.record, plan_revision="f" * 64)) + assert store.get(claimed.record.run_id) == claimed canceling = transition_run( claimed.record, HostedRunState.CANCELING, diff --git a/tickets/DANDER-285-quality-review-command-boundaries.md b/tickets/DANDER-285-quality-review-command-boundaries.md new file mode 100644 index 00000000..744ab4f4 --- /dev/null +++ b/tickets/DANDER-285-quality-review-command-boundaries.md @@ -0,0 +1,31 @@ +--- +id: DANDER-285 +title: Isolate plugin commands and share durable run identity +status: in-code +component: python +depends_on: [] +created: 2026-09-30 +--- + +## Context + +A repository-wide structural scan found that offline plugin discovery loaded deployment commands, +and the S3 and PostgreSQL run stores each maintained the same submission-identity comparison. +Both add unnecessary coupling when changing otherwise independent features. + +## Acceptance Criteria + +- [x] Move plugin commands to their own module while preserving options, output, and installation. +- [x] Dispatch plugin help/search without importing bootstrap, deployment, or provider modules. +- [x] Have both run stores use one identity rule with the existing compared fields unchanged. +- [x] Verify rejected identity changes leave persisted records unchanged and lifecycle saves work. +- [x] Record review scope, checks, and remaining maintenance priorities. + +## Implementation Notes + +The existing lightweight Control dispatcher pattern also selects `plugins`. Manifest loading stays +inside installation; help and catalog discovery need neither deployment configuration nor SDKs. +`same_run_identity` belongs to the provider-neutral orchestration contract. Storage serialization, +schema versions, installer pins, and public CLI syntax are unchanged. + +See [the quality review](../docs/quality-review-2026-09-30.md) for findings and validation. From 6f753e822577213bc610e44f0debb475d90e6727 Mon Sep 17 00:00:00 2001 From: Harrison O'Connor-Hoover Date: Wed, 30 Sep 2026 20:29:46 -0400 Subject: [PATCH 2/4] Refresh dependencies flagged by the runtime security audit --- HANDOFF.md | 2 ++ docs/decisions.md | 3 +++ docs/quality-review-2026-09-30.md | 13 +++++++++- pyproject.toml | 2 +- ...R-285-quality-review-command-boundaries.md | 6 ++++- uv.lock | 26 +++++++++---------- 6 files changed, 36 insertions(+), 16 deletions(-) diff --git a/HANDOFF.md b/HANDOFF.md index 06b983df..c9fdcae6 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -5,6 +5,7 @@ - Scanned repository structure, duplication, import boundaries, error handling, and test health; recorded scope and remaining priorities in `docs/quality-review-2026-09-30.md`. - Isolated plugin commands so help and catalog search avoid bootstrap, deployment, and provider modules. - Shared the existing durable-run identity rule between S3 and PostgreSQL, with storage behavior preserved. +- Updated four dependencies after protected CI identified 20 published advisories; the strict runtime dependency audit is now clean locally. ## Try It @@ -14,6 +15,7 @@ Run `uv run dander plugins --help` and `uv run dander plugins search incident`. - Full suite with disposable local PostgreSQL 17: **2,419 passed**, one existing Starlette/httpx deprecation warning. - Focused CLI/storage/lifecycle suite: **75 passed**; final CLI test-import adjustment: **10 passed**. +- Updated dependencies: **139 authentication/ingestion tests passed**; strict `runtime-all` dependency audit reports **no known vulnerabilities**. - Ruff lint/format, canonical strict typing (**509 files**), Control contract drift, documentation links, and diff checks passed. - Compared moved command bodies and identity comparisons structurally against the original code; behavior-bearing expressions match. Protected CI and merge results are available on the task's attached quality-review PR. diff --git a/docs/decisions.md b/docs/decisions.md index 8c6e4478..3380a4fc 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -7,6 +7,9 @@ catalog search do not depend on deployment modules. CLI options and outputs remain compatible. - `same_run_identity` lives beside the provider-neutral `RunStore` contract. S3 and PostgreSQL use the same existing 14-field comparison; storage schemas and lifecycle transitions are unchanged. +- Refresh the four dependencies flagged by protected CI without audit exclusions. Raise the direct + PyJWT minimum to 2.15.1 so package consumers also receive its fixes; keep transitive updates in + the lockfile and verify the full runtime requirements with the existing strict audit. ## 2026-09-08 — Recover pending PostgreSQL runs without reading completed history diff --git a/docs/quality-review-2026-09-30.md b/docs/quality-review-2026-09-30.md index 757dc0db..29a586e9 100644 --- a/docs/quality-review-2026-09-30.md +++ b/docs/quality-review-2026-09-30.md @@ -25,8 +25,19 @@ reading; it is not a line-by-line review of every file or new live-cloud qualifi | --- | --- | --- | | `plugins search` loaded 13 bootstrap, 3 deployment, and 55 provider modules via the root CLI. Offline discovery depended on unrelated command code. | Move install/scaffold/search into `cli/plugins_command.py`; dispatch the plugin group directly and load project configuration only for installation. | New subprocess tests failed before the change and pass afterward. Help/search import none of those module families. Existing CLI tests retain exact pins, version-2 manifests, scaffold behavior, and catalog filtering. | | S3 and PostgreSQL stores duplicated the same 14-field durable-identity comparison. A future change could update only one implementation. | Put `same_run_identity` beside the `RunStore` contract and remove the duplicated implementations and redundant run-ID checks. | Both backend tests reject a changed plan revision, retain the original stored snapshot, and still accept lifecycle transitions with conditional revisions. | +| Protected CI found 20 published dependency advisories in the existing lockfile. | Update only AnyIO 4.14.1 → 4.14.2, urllib3 2.7.0 → 2.8.0, PyJWT 2.13.0 → 2.15.1, and OAuthLib 3.3.1 → 4.0.0. Raise Dander's direct PyJWT minimum to 2.15.1. | The strict audit of the exported `runtime-all` requirements reports no known vulnerabilities; 139 authentication and ingestion tests passed with the updates. | -No dependency, schema, provider payload, or connector package pin changed. +No new dependency, schema, provider payload, or connector package pin was introduced. The four +dependency updates address the audit findings; they do not imply that every advisory was exploitable +through Dander. The failing audit is retained in [the initial PR run](https://github.com/harrisonoconnorhover/dander/actions/runs/36795849992). + +Compatibility review used the upstream [AnyIO advisory](https://github.com/agronholm/anyio/security/advisories/GHSA-82r6-8w77-94w6), +[urllib3 release notes](https://github.com/urllib3/urllib3/releases/tag/2.8.0), +[PyJWT release](https://github.com/jpadilla/pyjwt/releases/tag/2.15.1), and +[OAuthLib release notes](https://github.com/oauthlib/oauthlib/releases/tag/v4.0.0). +OAuthLib's documented breaking changes concern server grant validation and token revocation; +Dander receives it through the Google authentication client's dependency chain. Dander's OIDC +decoder already supplies fresh verification options on every call. ## Existing protections confirmed diff --git a/pyproject.toml b/pyproject.toml index baa4eee4..0bc1ced1 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -31,7 +31,7 @@ dependencies = [ "pyyaml>=6.0", "fastapi>=0.115", # hosted Control API transport "uvicorn>=0.30", # hosted Control API server - "pyjwt[crypto]>=2.13,<3", # hosted OIDC access-token verification + "pyjwt[crypto]>=2.15.1,<3", # hosted OIDC access-token verification ] [project.urls] diff --git a/tickets/DANDER-285-quality-review-command-boundaries.md b/tickets/DANDER-285-quality-review-command-boundaries.md index 744ab4f4..f36c6b54 100644 --- a/tickets/DANDER-285-quality-review-command-boundaries.md +++ b/tickets/DANDER-285-quality-review-command-boundaries.md @@ -1,6 +1,6 @@ --- id: DANDER-285 -title: Isolate plugin commands and share durable run identity +title: Isolate plugin commands, share run identity, and refresh vulnerable dependencies status: in-code component: python depends_on: [] @@ -12,6 +12,7 @@ created: 2026-09-30 A repository-wide structural scan found that offline plugin discovery loaded deployment commands, and the S3 and PostgreSQL run stores each maintained the same submission-identity comparison. Both add unnecessary coupling when changing otherwise independent features. +Protected CI also identified published advisories in four existing dependencies. ## Acceptance Criteria @@ -20,6 +21,7 @@ Both add unnecessary coupling when changing otherwise independent features. - [x] Have both run stores use one identity rule with the existing compared fields unchanged. - [x] Verify rejected identity changes leave persisted records unchanged and lifecycle saves work. - [x] Record review scope, checks, and remaining maintenance priorities. +- [x] Update the four affected dependencies and pass the strict runtime dependency audit. ## Implementation Notes @@ -27,5 +29,7 @@ The existing lightweight Control dispatcher pattern also selects `plugins`. Mani inside installation; help and catalog discovery need neither deployment configuration nor SDKs. `same_run_identity` belongs to the provider-neutral orchestration contract. Storage serialization, schema versions, installer pins, and public CLI syntax are unchanged. +The dependency patch updates AnyIO, urllib3, PyJWT, and OAuthLib, with the direct PyJWT minimum +raised to 2.15.1. No audit exclusions or new dependencies were added. See [the quality review](../docs/quality-review-2026-09-30.md) for findings and validation. diff --git a/uv.lock b/uv.lock index 9b399a8a..f4ff8da3 100644 --- a/uv.lock +++ b/uv.lock @@ -170,15 +170,15 @@ wheels = [ [[package]] name = "anyio" -version = "4.14.1" +version = "4.14.2" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "idna" }, { name = "typing-extensions", marker = "python_full_version < '3.13'" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/3b/72/5562aabb8dd7181e8e860622a38bea08d17842b99ecd4c91f84ac95251b0/anyio-4.14.1.tar.gz", hash = "sha256:8d648a3544c1a700e3ff78615cd679e4c5c3f149904287e73687b2596963629e", size = 254831, upload-time = "2026-06-24T20:56:06.017Z" } +sdist = { url = "https://files.pythonhosted.org/packages/61/cc/a381afa6efea9f496eff839d4a6a1aed3bfafc7b3ab4b0d1b243a12573dd/anyio-4.14.2.tar.gz", hash = "sha256:cfa139f3ed1a23ee8f88a145ddb5ac7605b8bbfd8592baacd7ce3d8bb4313c7f", size = 260176, upload-time = "2026-07-12T20:29:07.082Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/b0/7b/90df4a0a816d98d6ea26f559d87836d494a2cf1fcf063be67df50a7bcc30/anyio-4.14.1-py3-none-any.whl", hash = "sha256:4e5533c5b8ff0a24f5d7a176cbe6877129cd183893f66b537f8f227d10527d72", size = 124875, upload-time = "2026-06-24T20:56:04.413Z" }, + { url = "https://files.pythonhosted.org/packages/da/35/f2287558c17e29fafc8ef3daf819bb9834061cfa43bff8014f7df7f63bdc/anyio-4.14.2-py3-none-any.whl", hash = "sha256:9f505dda5ac9f0c8309b5e8bd445a8c2bf7246f3ce950121e45ea15bc41d1494", size = 125813, upload-time = "2026-07-12T20:29:05.763Z" }, ] [[package]] @@ -826,7 +826,7 @@ requires-dist = [ { name = "pyarrow", marker = "extra == 'snowflake'", specifier = ">=19,<26" }, { name = "pydantic", specifier = ">=2.12" }, { name = "pydantic-settings", specifier = ">=2.3" }, - { name = "pyjwt", extras = ["crypto"], specifier = ">=2.13,<3" }, + { name = "pyjwt", extras = ["crypto"], specifier = ">=2.15.1,<3" }, { name = "pytest", marker = "extra == 'dev'", specifier = ">=8.3" }, { name = "pytest-cov", marker = "extra == 'dev'", specifier = ">=5.0" }, { name = "pyyaml", specifier = ">=6.0" }, @@ -1971,11 +1971,11 @@ wheels = [ [[package]] name = "oauthlib" -version = "3.3.1" +version = "4.0.0" source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/0b/5f/19930f824ffeb0ad4372da4812c50edbd1434f678c90c2733e1188edfc63/oauthlib-3.3.1.tar.gz", hash = "sha256:0f0f8aa759826a193cf66c12ea1af1637f87b9b4622d46e866952bb022e538c9", size = 185918, upload-time = "2025-06-19T22:48:08.269Z" } +sdist = { url = "https://files.pythonhosted.org/packages/7a/d8/a1bcc8ba112a627f8ffbdc212a78ce18d3ac07e91a5ca65d27918eee25a1/oauthlib-4.0.0.tar.gz", hash = "sha256:efb274799819440f95b4ab3b818869f1ce9ae26c5beacba0201d1a1b76b54f86", size = 187232, upload-time = "2026-09-28T06:01:18.77Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/be/9c/92789c596b8df838baa98fa71844d84283302f7604ed565dafe5a6b5041a/oauthlib-3.3.1-py3-none-any.whl", hash = "sha256:88119c938d2b8fb88561af5f6ee0eec8cc8d552b7bb1f712743136eb7523b7a1", size = 160065, upload-time = "2025-06-19T22:48:06.508Z" }, + { url = "https://files.pythonhosted.org/packages/d9/f4/78229a1066068ca14fc60fb26cf7381cabe4382261392b90e5f9552722d4/oauthlib-4.0.0-py3-none-any.whl", hash = "sha256:624c28c13a0a59cabf9747dfa52af63be3e512a7f2714df16e91b5b3a145e6cd", size = 159715, upload-time = "2026-09-28T06:01:17.008Z" }, ] [[package]] @@ -2558,11 +2558,11 @@ wheels = [ [[package]] name = "pyjwt" -version = "2.13.0" +version = "2.15.1" source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/3b/81/58d0ac84e1ef3a3843791d6954d94c0b33d526c75eeb1efbce9d0a4c4077/pyjwt-2.13.0.tar.gz", hash = "sha256:41571c89ca91598c79e8ef18a2d07367d4810fbbd6f637794879baf1b7703423", size = 107515, upload-time = "2026-05-21T19:54:36.618Z" } +sdist = { url = "https://files.pythonhosted.org/packages/43/ea/5194e52748b0da83d71e082d75496eaec6e58f419f5e184786ded517e6a9/pyjwt-2.15.1.tar.gz", hash = "sha256:4f259e80cdfb6b3fc18a7de51fd1ef9ec79652f25019bae68975ca2468a34df8", size = 121252, upload-time = "2026-09-28T18:40:42.598Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/a3/5e/ecf12fdb62546d64385c158514e9b2b671f7832108ef2ecd2020ce0af2d1/pyjwt-2.13.0-py3-none-any.whl", hash = "sha256:66adcc2aff09b3f1bbd95fc1e1577df8ac8723c978552fd43304c8a290ac5728", size = 31274, upload-time = "2026-05-21T19:54:35.362Z" }, + { url = "https://files.pythonhosted.org/packages/50/ca/44de4e75f8aadc457f0634be3b542815078ded46dca30efb960edeecad6e/pyjwt-2.15.1-py3-none-any.whl", hash = "sha256:42d59d631f7768a1028a64c7ff581a9bf7519804daf91fc5b6c56e30eec5e193", size = 33860, upload-time = "2026-09-28T18:40:41.429Z" }, ] [package.optional-dependencies] @@ -3226,11 +3226,11 @@ wheels = [ [[package]] name = "urllib3" -version = "2.7.0" +version = "2.8.0" source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/53/0c/06f8b233b8fd13b9e5ee11424ef85419ba0d8ba0b3138bf360be2ff56953/urllib3-2.7.0.tar.gz", hash = "sha256:231e0ec3b63ceb14667c67be60f2f2c40a518cb38b03af60abc813da26505f4c", size = 433602, upload-time = "2026-05-07T16:13:18.596Z" } +sdist = { url = "https://files.pythonhosted.org/packages/e3/05/b17359e1cefb4f909b5e40b1b90a496d987258916dbbf88e842c729f510e/urllib3-2.8.0.tar.gz", hash = "sha256:63bf2ead4c879426ebf22ef2a781eeb4aa3b4ae798a0435506f8687fd5bb9b63", size = 458972, upload-time = "2026-09-15T19:29:36.253Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/7f/3e/5db95bcf282c52709639744ca2a8b149baccf648e39c8cc87553df9eae0c/urllib3-2.7.0-py3-none-any.whl", hash = "sha256:9fb4c81ebbb1ce9531cce37674bbc6f1360472bc18ca9a553ede278ef7276897", size = 131087, upload-time = "2026-05-07T16:13:17.151Z" }, + { url = "https://files.pythonhosted.org/packages/92/9d/c4e665119135114480843e7ab388fa94d8480650450e6f8e26b70d323a4c/urllib3-2.8.0-py3-none-any.whl", hash = "sha256:0cf3cae568d36aa9576b28dfb35f11328f1cb974ca7647d9475ebb86c75ac6e3", size = 135717, upload-time = "2026-09-15T19:29:34.577Z" }, ] [[package]] From b24ecf25bd1147ed2833558b98ca0daf0c84c0ec Mon Sep 17 00:00:00 2001 From: Harrison O'Connor-Hoover Date: Wed, 30 Sep 2026 20:39:03 -0400 Subject: [PATCH 3/4] Refresh vulnerable runtime image system packages --- Dockerfile | 4 ++++ HANDOFF.md | 2 ++ docs/quality-review-2026-09-30.md | 2 ++ src/dander/templates/project/Dockerfile | 4 ++++ tickets/DANDER-285-quality-review-command-boundaries.md | 2 ++ 5 files changed, 14 insertions(+) diff --git a/Dockerfile b/Dockerfile index 5fe37468..7603d96b 100644 --- a/Dockerfile +++ b/Dockerfile @@ -26,14 +26,18 @@ RUN apt-get update \ && apt-get install --yes --no-install-recommends \ libpq5 \ bsdutils \ + gzip \ libblkid1 \ liblastlog2-2 \ libmount1 \ + libpcre2-8-0 \ libsmartcols1 \ + libsqlite3-0 \ libuuid1 \ login \ mount \ openssl \ + perl-base \ util-linux \ && rm -rf /var/lib/apt/lists/* RUN pip install --no-cache-dir uv==0.12.0 diff --git a/HANDOFF.md b/HANDOFF.md index c9fdcae6..d48e94bf 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -6,6 +6,7 @@ - Isolated plugin commands so help and catalog search avoid bootstrap, deployment, and provider modules. - Shared the existing durable-run identity rule between S3 and PostgreSQL, with storage behavior preserved. - Updated four dependencies after protected CI identified 20 published advisories; the strict runtime dependency audit is now clean locally. +- Refreshed four inherited system packages in runtime/starter Dockerfiles after 13 container findings; the updated OS layer passes a local vulnerability scan. ## Try It @@ -16,6 +17,7 @@ Run `uv run dander plugins --help` and `uv run dander plugins search incident`. - Full suite with disposable local PostgreSQL 17: **2,419 passed**, one existing Starlette/httpx deprecation warning. - Focused CLI/storage/lifecycle suite: **75 passed**; final CLI test-import adjustment: **10 passed**. - Updated dependencies: **139 authentication/ingestion tests passed**; strict `runtime-all` dependency audit reports **no known vulnerabilities**. +- Runtime OS layer: linux/amd64 build and patched-package inspection passed; Trivy found **zero fixable high/critical findings**. Scaffold/release/import checks: **15 passed**. - Ruff lint/format, canonical strict typing (**509 files**), Control contract drift, documentation links, and diff checks passed. - Compared moved command bodies and identity comparisons structurally against the original code; behavior-bearing expressions match. Protected CI and merge results are available on the task's attached quality-review PR. diff --git a/docs/quality-review-2026-09-30.md b/docs/quality-review-2026-09-30.md index 29a586e9..93f3ae39 100644 --- a/docs/quality-review-2026-09-30.md +++ b/docs/quality-review-2026-09-30.md @@ -26,6 +26,7 @@ reading; it is not a line-by-line review of every file or new live-cloud qualifi | `plugins search` loaded 13 bootstrap, 3 deployment, and 55 provider modules via the root CLI. Offline discovery depended on unrelated command code. | Move install/scaffold/search into `cli/plugins_command.py`; dispatch the plugin group directly and load project configuration only for installation. | New subprocess tests failed before the change and pass afterward. Help/search import none of those module families. Existing CLI tests retain exact pins, version-2 manifests, scaffold behavior, and catalog filtering. | | S3 and PostgreSQL stores duplicated the same 14-field durable-identity comparison. A future change could update only one implementation. | Put `same_run_identity` beside the `RunStore` contract and remove the duplicated implementations and redundant run-ID checks. | Both backend tests reject a changed plan revision, retain the original stored snapshot, and still accept lifecycle transitions with conditional revisions. | | Protected CI found 20 published dependency advisories in the existing lockfile. | Update only AnyIO 4.14.1 → 4.14.2, urllib3 2.7.0 → 2.8.0, PyJWT 2.13.0 → 2.15.1, and OAuthLib 3.3.1 → 4.0.0. Raise Dander's direct PyJWT minimum to 2.15.1. | The strict audit of the exported `runtime-all` requirements reports no known vulnerabilities; 139 authentication and ingestion tests passed with the updates. | +| The runtime image scan found 13 high/critical advisories in inherited Debian packages. | Add `gzip`, `libpcre2-8-0`, `libsqlite3-0`, and `perl-base` to the existing package-refresh step in the runtime and starter Dockerfiles. | Built the actual runtime OS layer for linux/amd64 and confirmed patched package versions. Trivy reports zero fixable high/critical findings in that layer; 15 scaffold, release-metadata, and import-isolation tests pass. Full image checks remain part of protected CI. | No new dependency, schema, provider payload, or connector package pin was introduced. The four dependency updates address the audit findings; they do not imply that every advisory was exploitable @@ -68,5 +69,6 @@ decoder already supplies fresh verification options on every call. - Changed-path validation: 75 plugin CLI, console dispatch, S3/PostgreSQL storage, and lifecycle tests passed. The new import tests demonstrated the pre-change coupling before passing. - Final full-suite and protected CI results are recorded in the handoff and pull request. +- The full 2,419-test suite and strict typing passed again after the Python dependency updates. - The baseline and focused runs emitted one existing Starlette/httpx test-client deprecation warning. No live provider workload was submitted by this review. diff --git a/src/dander/templates/project/Dockerfile b/src/dander/templates/project/Dockerfile index ad9e88fb..c794ea1e 100644 --- a/src/dander/templates/project/Dockerfile +++ b/src/dander/templates/project/Dockerfile @@ -24,13 +24,17 @@ RUN apt-get update \ && apt-get install --yes --no-install-recommends \ libpq5 \ bsdutils \ + gzip \ libblkid1 \ liblastlog2-2 \ libmount1 \ + libpcre2-8-0 \ libsmartcols1 \ + libsqlite3-0 \ libuuid1 \ login \ mount \ + perl-base \ util-linux \ && rm -rf /var/lib/apt/lists/* RUN pip install --no-cache-dir "dander-platform[runtime-all]==${DANDER_VERSION}" \ diff --git a/tickets/DANDER-285-quality-review-command-boundaries.md b/tickets/DANDER-285-quality-review-command-boundaries.md index f36c6b54..8d56f287 100644 --- a/tickets/DANDER-285-quality-review-command-boundaries.md +++ b/tickets/DANDER-285-quality-review-command-boundaries.md @@ -13,6 +13,7 @@ A repository-wide structural scan found that offline plugin discovery loaded dep and the S3 and PostgreSQL run stores each maintained the same submission-identity comparison. Both add unnecessary coupling when changing otherwise independent features. Protected CI also identified published advisories in four existing dependencies. +The image scan subsequently found 13 advisories in four inherited Debian packages. ## Acceptance Criteria @@ -22,6 +23,7 @@ Protected CI also identified published advisories in four existing dependencies. - [x] Verify rejected identity changes leave persisted records unchanged and lifecycle saves work. - [x] Record review scope, checks, and remaining maintenance priorities. - [x] Update the four affected dependencies and pass the strict runtime dependency audit. +- [x] Refresh the four affected system packages in runtime and starter images and verify the OS layer. ## Implementation Notes From 780a9ab00e4b9bb0bb9f2338346bc0d30c0031df Mon Sep 17 00:00:00 2001 From: Harrison O'Connor-Hoover Date: Wed, 30 Sep 2026 20:46:06 -0400 Subject: [PATCH 4/4] Refresh flagged Spark and OCI image libraries --- HANDOFF.md | 4 ++-- docs/quality-review-2026-09-30.md | 2 +- infra/oci/controller/Dockerfile | 2 +- infra/spark/Dockerfile | 1 + tickets/DANDER-285-quality-review-command-boundaries.md | 4 ++-- 5 files changed, 7 insertions(+), 6 deletions(-) diff --git a/HANDOFF.md b/HANDOFF.md index d48e94bf..88a3e6fd 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -6,7 +6,7 @@ - Isolated plugin commands so help and catalog search avoid bootstrap, deployment, and provider modules. - Shared the existing durable-run identity rule between S3 and PostgreSQL, with storage behavior preserved. - Updated four dependencies after protected CI identified 20 published advisories; the strict runtime dependency audit is now clean locally. -- Refreshed four inherited system packages in runtime/starter Dockerfiles after 13 container findings; the updated OS layer passes a local vulnerability scan. +- Refreshed flagged system packages in runtime/starter, Spark, and OCI Dockerfiles; the runtime OS layer and complete Spark/OCI images pass local scans. ## Try It @@ -17,7 +17,7 @@ Run `uv run dander plugins --help` and `uv run dander plugins search incident`. - Full suite with disposable local PostgreSQL 17: **2,419 passed**, one existing Starlette/httpx deprecation warning. - Focused CLI/storage/lifecycle suite: **75 passed**; final CLI test-import adjustment: **10 passed**. - Updated dependencies: **139 authentication/ingestion tests passed**; strict `runtime-all` dependency audit reports **no known vulnerabilities**. -- Runtime OS layer: linux/amd64 build and patched-package inspection passed; Trivy found **zero fixable high/critical findings**. Scaffold/release/import checks: **15 passed**. +- Runtime OS layer and Spark/OCI images: linux/amd64 builds and Trivy scans passed with **zero fixable high/critical findings**. User/driver/handler checks passed. Scaffold/release/import tests: **15 passed**; OCI image/publication tests: **10 passed**. - Ruff lint/format, canonical strict typing (**509 files**), Control contract drift, documentation links, and diff checks passed. - Compared moved command bodies and identity comparisons structurally against the original code; behavior-bearing expressions match. Protected CI and merge results are available on the task's attached quality-review PR. diff --git a/docs/quality-review-2026-09-30.md b/docs/quality-review-2026-09-30.md index 93f3ae39..f10f0fb2 100644 --- a/docs/quality-review-2026-09-30.md +++ b/docs/quality-review-2026-09-30.md @@ -26,7 +26,7 @@ reading; it is not a line-by-line review of every file or new live-cloud qualifi | `plugins search` loaded 13 bootstrap, 3 deployment, and 55 provider modules via the root CLI. Offline discovery depended on unrelated command code. | Move install/scaffold/search into `cli/plugins_command.py`; dispatch the plugin group directly and load project configuration only for installation. | New subprocess tests failed before the change and pass afterward. Help/search import none of those module families. Existing CLI tests retain exact pins, version-2 manifests, scaffold behavior, and catalog filtering. | | S3 and PostgreSQL stores duplicated the same 14-field durable-identity comparison. A future change could update only one implementation. | Put `same_run_identity` beside the `RunStore` contract and remove the duplicated implementations and redundant run-ID checks. | Both backend tests reject a changed plan revision, retain the original stored snapshot, and still accept lifecycle transitions with conditional revisions. | | Protected CI found 20 published dependency advisories in the existing lockfile. | Update only AnyIO 4.14.1 → 4.14.2, urllib3 2.7.0 → 2.8.0, PyJWT 2.13.0 → 2.15.1, and OAuthLib 3.3.1 → 4.0.0. Raise Dander's direct PyJWT minimum to 2.15.1. | The strict audit of the exported `runtime-all` requirements reports no known vulnerabilities; 139 authentication and ingestion tests passed with the updates. | -| The runtime image scan found 13 high/critical advisories in inherited Debian packages. | Add `gzip`, `libpcre2-8-0`, `libsqlite3-0`, and `perl-base` to the existing package-refresh step in the runtime and starter Dockerfiles. | Built the actual runtime OS layer for linux/amd64 and confirmed patched package versions. Trivy reports zero fixable high/critical findings in that layer; 15 scaffold, release-metadata, and import-isolation tests pass. Full image checks remain part of protected CI. | +| Image scans found 13 high/critical findings in the runtime's inherited Debian packages, 3 in Spark, and 16 in OCI. | Refresh `gzip`, `libpcre2-8-0`, `libsqlite3-0`, and `perl-base` in runtime/starter Dockerfiles; refresh PCRE in Spark and `expat`, `libevent`, and `libxml2` in OCI. Keep existing bases and scan rules. | The linux/amd64 runtime OS layer and complete Spark/OCI images build and scan with zero fixable high/critical findings. Patched versions, runtime users, Spark driver identity, and OCI handler imports were checked. The 15 scaffold/release/import and 10 OCI image/publication tests pass. Protected CI also checks the complete runtime image. | No new dependency, schema, provider payload, or connector package pin was introduced. The four dependency updates address the audit findings; they do not imply that every advisory was exploitable diff --git a/infra/oci/controller/Dockerfile b/infra/oci/controller/Dockerfile index 47ebcee5..d25b892c 100644 --- a/infra/oci/controller/Dockerfile +++ b/infra/oci/controller/Dockerfile @@ -14,7 +14,7 @@ RUN pip3 install --target /python --no-cache-dir --no-deps /tmp/dander_platform- FROM fnproject/python:3.12@sha256:fa298b94e95a54e6d10c54aa5691fefdb01c76c5c38b3da352a2f9dc0fbdf739 -RUN microdnf update -y curl libcurl python3.12 python3.12-libs sqlite-libs \ +RUN microdnf update -y curl expat libcurl libevent libxml2 python3.12 python3.12-libs sqlite-libs \ && microdnf clean all WORKDIR /function diff --git a/infra/spark/Dockerfile b/infra/spark/Dockerfile index 1d3f3fdf..b08e4ca9 100644 --- a/infra/spark/Dockerfile +++ b/infra/spark/Dockerfile @@ -16,6 +16,7 @@ RUN apt-get update \ && apt-get install --yes --no-install-recommends \ ca-certificates \ libjemalloc2 \ + libpcre2-8-0 \ procps \ tini \ && groupadd --gid 1099 spark \ diff --git a/tickets/DANDER-285-quality-review-command-boundaries.md b/tickets/DANDER-285-quality-review-command-boundaries.md index 8d56f287..e4063e10 100644 --- a/tickets/DANDER-285-quality-review-command-boundaries.md +++ b/tickets/DANDER-285-quality-review-command-boundaries.md @@ -13,7 +13,7 @@ A repository-wide structural scan found that offline plugin discovery loaded dep and the S3 and PostgreSQL run stores each maintained the same submission-identity comparison. Both add unnecessary coupling when changing otherwise independent features. Protected CI also identified published advisories in four existing dependencies. -The image scan subsequently found 13 advisories in four inherited Debian packages. +Image scans subsequently found affected inherited libraries in runtime, Spark, and OCI images. ## Acceptance Criteria @@ -23,7 +23,7 @@ The image scan subsequently found 13 advisories in four inherited Debian package - [x] Verify rejected identity changes leave persisted records unchanged and lifecycle saves work. - [x] Record review scope, checks, and remaining maintenance priorities. - [x] Update the four affected dependencies and pass the strict runtime dependency audit. -- [x] Refresh the four affected system packages in runtime and starter images and verify the OS layer. +- [x] Refresh affected system packages in runtime/starter, Spark, and OCI Dockerfiles; build and scan the runtime OS layer and complete Spark/OCI images. ## Implementation Notes