diff --git a/HANDOFF.md b/HANDOFF.md index 8817cf1e..41c9682c 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -2,36 +2,37 @@ ## Finished -- Added a read-only comparison of an unsaved graph with its exact saved revision, including downstream outputs and write effects. -- Added readable explanations of recorded run outcomes and eligible next actions; unavailable measurements remain unknown. -- Exposed both operations through existing Control authentication and regenerated their public contracts. +- Merged read-only graph change preview and recorded run explanations through protected PR #547. +- Implemented selected-date output repair from retained raw data with single-task Cloud Run and BigQuery. +- Reused durable run identity, cancellation, history, replay, lease fencing, and conditional cleanup. +- Added preview/start contracts, publication measurements, and operator documentation with explicit support limits. ## Try It -With Control running and a saved graph selected, POST candidate graph JSON to `/v1/projects/{project}/graphs/{graph}/change-preview` with its saved `If-Match` ETag. GET `/v1/runs/{run_id}/explanation` for a configured lifecycle's run. See `docs/control-contracts.md` for semantics and client compatibility. +Use the API examples in `docs/control-contracts.md` and `docs/output-repair.md`. Preview a saved graph's UTC date interval with its `If-Match` ETag, then start `/repairs` with the same revision and an `Idempotency-Key`. A repair-capable runtime and existing partitioned outputs are required. Published clients do not yet expose this flow. ## Checks -- 87 focused comparison, explanation, HTTP, OIDC, and contract tests passed. -- Canonical strict typing passed across 513 files. -- Ruff lint/format, generated Control contract drift, and diff checks passed. -- Protected CI and exact-main verification remain pending for this implementation step. No live cloud workload was run. +- Full local run: 2,429 passed, 56 skipped; all 28 loopback-server failures passed on retry with socket permission. The additional explanation test passed in its 17-test file. +- Focused Control/runtime/executor/deployment selection: 538 passed, 21 skipped. +- Strict typing passed across 516 files; Ruff lint/format, generated contracts, and diff checks passed. +- PR #547 protected CI passed and merged as `099fb52`; exact-main CI is running. Repair protected CI and live provider qualification remain pending. ## Decisions -- Keep static graph comparison separate from infrastructure deployment preview; no new persistence or provider reads. -- Explain existing run evidence deterministically, with no new AI service or fabricated diagnosis. -- Deliver real date-scoped output repair and the guided interface as subsequent steps of the same active objective. +- Preview stays read-only; explanations use collected evidence and retain unknown measurements. +- Repair leaves source extraction progress untouched and replaces only the selected dates, atomically per output. +- Deliver one guided workflow in Druff; additional providers and broader graph execution are separate work. ## Remaining -- Implement bounded output-partition repair from retained raw data, preserving normal extraction progress and unrelated output rows. -- Integrate connect, configure, preview, run, and outcome in Druff; its repository exception is awaiting the user's answer. -- Publish and consume a verified immutable producer artifact for the paired Druff client; existing published clients use their original bundle. -- Complete protected PR, merge, and exact-main CI for each runtime change. +- Complete the repair protected PR, merge, and exact-main CI. +- Complete the bounded live repair check after the existing AWS billing session is restored; current aggregate cash exposure is unverified. +- Integrate Druff's guided flow once the requested repository exception is answered. +- Publish and consume the matching immutable contract/runtime artifacts; the complete guided journey is not delivered yet. ## Review First -- `src/dander/control/change_preview.py` for graph comparison and write explanations. -- `src/dander/control/run_explanation.py` for evidence-based outcome handling. -- `docs/control-contracts.md` for the API and publication boundaries. +- `src/dander/providers/bigquery/graph.py` and `tests/pipeline/test_date_repair.py` for scoped publication. +- `src/dander/control/cloud_run_execution_backend.py` for execution binding and cleanup. +- `docs/output-repair.md` for practical behavior and limitations. diff --git a/docs/control-contracts.md b/docs/control-contracts.md index 7895d3f2..4457b04e 100644 --- a/docs/control-contracts.md +++ b/docs/control-contracts.md @@ -188,12 +188,15 @@ Run explanations only show counters when the runtime result summary has been col execution can succeed before those measurements arrive; missing measurements are not zero rows. Explanations distinguish skipped work, pending cancellation, failed execution, and confirmed results without guessing an error cause. Suggested actions honor the current run controls; the -explanation does not perform them. Replay is another execution, not rollback or date-range repair. +explanation does not perform them. Ordinary replay is another full execution, not rollback or +date-range repair; replaying a repair run retains its recorded date window. Neither operation requires provider-specific browser logic or a new service dependency. Editors can request change previews; viewers can read run explanations when a lifecycle is wired. Regenerate consumers from the immutable producer artifact after its release. The guided Druff -surface and date-scoped repair are separate follow-on work, not delivered by these two routes. +surface is separate work, not delivered by these two routes. Current source also exposes +`graph.repair-preview` and `run.repair` for eligible execution plans; see +[selected output date repair](output-repair.md) for the request, supported scope, and evidence limits. ## Hosted Control service diff --git a/docs/decisions.md b/docs/decisions.md index fa730e90..9c89b101 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -1,5 +1,18 @@ # Engineering Decisions +## 2026-10-01 — Repair output dates from retained raw data + +- Keep a repair window in the existing durable run trigger, preserving cancellation, replay, + history, and request identity. Preview reads only the saved graph and current execution plans. + The initial runtime is single-task Cloud Run with BigQuery; other backends reject repair. +- Skip extraction and its saved progress. Compile current retained raw data, then replace only + the selected UTC dates in each existing output under the normal lease fence. Reject keys + crossing the interval boundary; publication is atomic per output, not across all outputs. +- Bind the exact graph hash and dates to the native execution. A fixed runtime argument rejects + older images before work; conditional cleanup restores the Job configuration only while the + repair still owns it. Live provider qualification and the paired Druff client remain separate + delivery requirements, not claims implied by local tests. + ## 2026-10-01 — Explain pipeline changes and runs through the existing Control boundary - Compare an unsaved graph with the exact saved revision in a pure, provider-neutral operation. diff --git a/docs/output-repair.md b/docs/output-repair.md new file mode 100644 index 00000000..53ee9501 --- /dev/null +++ b/docs/output-repair.md @@ -0,0 +1,61 @@ +# Repair selected output dates + +Current source implements date-scoped output repair for direct Cloud Run execution with a +BigQuery warehouse. It rebuilds selected dates from the raw data **already stored** in the +warehouse. It does not fetch source records, advance ingestion progress, or reconstruct a past +source snapshot. The published RC19/RC20 client/runtime pair does not include this operation. + +## Preview, then start + +1. Read the saved graph and retain its `ETag`. +2. POST the date window to `/v1/projects/{project}/graphs/{graph}/repair-preview` with that + `If-Match` value. Review the outputs, UTC date boundary, available execution environments, + and limitations. This preview does not read warehouse rows or start a workload. +3. POST the same window to `/v1/projects/{project}/graphs/{graph}/repairs` with `If-Match` and + a new `Idempotency-Key`. When the preview offers more than one environment, select one through + `?environment=...`. Reuse the same key and body if a response is lost. +4. Poll the returned run through `/v1/runs/{run_id}` and `/v1/runs/{run_id}/explanation`. + The status retains `repair_window`. Existing cancellation, history, and replay controls apply; + replay retains the window but uses the raw data available when it runs. + +For September 1 and 2, send: + +```json +{"start_date": "2026-09-01", "end_date": "2026-09-03"} +``` + +The start date is included and the end date is excluded. For `TIMESTAMP` fields, dates are +interpreted in UTC. An editor can preview; starting a repair requires the operator role. + +## Eligible outputs and write behavior + +Every configured output must declare a `DATE` or `TIMESTAMP` partition field and a scalar business +key. Output tables must already exist. The first slice retains the graph's existing replace +configuration, but repair publication replaces only the selected dates. It requires the normal +BigQuery lease and fencing; sandbox execution and other warehouse/execution backends reject repair. +Outputs may not overlap each other or their raw input tables. + +The runtime stages the current graph result before publishing. It checks selected keys and rejects +a key that moves across the date boundary in either direction, including an undated counterpart. +That failure leaves the affected output unchanged; choose a wider range or do a full rebuild. +Rows outside the interval remain untouched. Empty selected results remove only selected dates. + +Each output publishes in one fenced transaction. If a later output fails, earlier successful outputs +remain; retrying converges on the current raw data. Publication counts appear in run telemetry and +the explanation. The existing ingestion counters stay zero because extraction was skipped. +Computing the result can scan more data than the dates being replaced; the preview cannot quote +row counts or cost without warehouse work. + +## Runtime compatibility and verification + +Control binds the exact dates and graph hash to the deterministic Cloud Run execution. The fixed +repair contract argument makes an old runtime reject the request before pipeline work, rather +than silently executing a full replacement. Adoption checks the exact execution payload. A +different ordinary execution ignores a well-formed inherited selection; Control also restores +the base Job settings after terminal repair while they still belong to that repair. + +Local checks cover date boundaries, unchanged surrounding rows, collision rejection, transaction +rollback, lease fencing, no source/watermark access, replay identity, and Cloud Run adoption and +cleanup. The row checks execute translated predicates against SQLite; they do not establish live +BigQuery or Cloud Run qualification. A bounded synthetic live check and the paired Druff interface +remain required before claiming the complete guided workflow is delivered. diff --git a/docs/support-status.md b/docs/support-status.md index 46dd7402..b770fd1b 100644 --- a/docs/support-status.md +++ b/docs/support-status.md @@ -1,6 +1,6 @@ # Current implementation and support status -Updated September 30, 2026. This page is the current overview; dated decisions and acceptance +Updated October 1, 2026. This page is the current overview; dated decisions and acceptance records preserve what happened on their named revisions. Update this page when a capability is integrated or a named release/profile changes status. @@ -27,6 +27,8 @@ the final paused/no-drift state. These are dated observations, not a continuous | Control execution through Fargate, Cloud Run, or Dataproc | Implemented with typed PostgreSQL or S3 run storage | Experimental; one Control process per run-store schema. AWS compatibility flags remain available. | | Distributed Spark graphs | Bounded linear, keyed-join, and two-transform shapes implemented; retained August 27–28 runs include exact Fargate/Dataproc output parity | Experimental; the [execution history](control-execution-history.md) identifies each tested revision and bounded shape. | | Druff hosted authoring | Hosted Control connections, OIDC/PKCE, persisted graphs, and run lifecycle controls implemented | Experimental; retained browser checks used synthetic OIDC. They do not establish production identity-provider or HA qualification. | +| Graph change preview and run explanation | Read-only comparison against the saved revision and readable recorded outcomes in current Control source | Requires a newly generated client from the matching immutable artifact; published Druff clients do not expose these operations. | +| Date-scoped output repair | Current source implements retained-raw repair with single-task Cloud Run and BigQuery; local transaction, lifecycle, and transport tests | Experimental and not live-qualified or published. See [output repair](output-repair.md) for existing-table, date-boundary, and per-output transaction limits. | | PostgreSQL, Snowflake, Redshift warehouse adapters | Local conformance and named bounded live evidence | Experimental; the runtime compatibility matrix determines allowed state/warehouse pairs. | | Azure, OCI, Kubernetes profiles | Named lifecycle/qualification records | No general support promotion; consult each exact profile's evidence. | | PostgreSQL Control graphs/runs/schedules | Database conformance plus one real Cloud Run graph with crash recovery, cancellation, output, and history checks | Experimental; one local Control process, no supported hosted or multiple-replica topology. | diff --git a/src/dander/cli/run_command.py b/src/dander/cli/run_command.py index 2eaebdea..5dda9cc7 100644 --- a/src/dander/cli/run_command.py +++ b/src/dander/cli/run_command.py @@ -81,6 +81,7 @@ from rich.console import Console from dander.core.interfaces import SecretStoreProvider + from dander.pipeline.repair import GraphRepairWindow _SOURCE_NAME = re.compile(r"^[A-Za-z_][A-Za-z0-9_-]*$") @@ -108,6 +109,8 @@ class RunOptions: catalog_output: Path | None publish_dataplex: bool dataplex_location: str + repair_window: GraphRepairWindow | None = None + repair_graph_content_sha256: str | None = None @dataclass(frozen=True, slots=True) @@ -324,6 +327,8 @@ def _resolve_run(options: RunOptions) -> _ResolvedRun: selected_models=selected_models, catalog_output=options.catalog_output, publish_catalog=publish_catalog, + repair_window=options.repair_window, + repair_graph_content_sha256=options.repair_graph_content_sha256, ) return _ResolvedRun( pipeline_id=options.pipeline_or_source, @@ -387,8 +392,14 @@ def _resolve_graph_plan( selected_models: Sequence[str] | None, catalog_output: Path | None, publish_catalog: bool, + repair_window: GraphRepairWindow | None = None, + repair_graph_content_sha256: str | None = None, ) -> GraphExecutionPlan | None: + if (repair_window is None) != (repair_graph_content_sha256 is None): + raise ClickException("Date repair requires both a date window and an exact graph hash") if graph_file is None: + if repair_window is not None: + raise ClickException("Date repair requires a graph pipeline") return None if build_models or selected_models is not None: raise ClickException("Graph pipelines do not accept --build-models or --select-model") @@ -396,10 +407,17 @@ def _resolve_graph_plan( raise ClickException("Graph metadata publication is not supported yet") try: graph = load_graph_for_execution(graph_file) + if repair_window is not None: + from dander.control.graph_store import canonicalize_graph_document + + canonical = canonicalize_graph_document(graph.model_dump(mode="json", by_alias=True)) + if canonical.content_sha256 != repair_graph_content_sha256: + raise GraphRuntimeError("Date repair graph hash does not match the loaded graph") return plan_graph_execution( graph, config, endpoint_relations=endpoint_relations, + repair_window=repair_window, ) except GraphRuntimeError as error: raise ClickException(str(error)) from error @@ -423,7 +441,13 @@ def _verify_safety(options: RunOptions, resolved: _ResolvedRun) -> None: def _require_provider_compatible_options(options: RunOptions, resolved: _ResolvedRun) -> None: """Reject BigQuery-only operations before any external client can be constructed.""" + if options.repair_window is not None and options.sandbox: + raise ClickException("Date repair requires BigQuery lease fencing and cannot use --sandbox") if resolved.warehouse_provider != "bigquery": + if options.repair_window is not None: + raise ClickException( + "Date repair is available only with direct BigQuery graph execution" + ) if options.sandbox: raise ClickException("--sandbox is available only with a BigQuery warehouse") if options.guarded_free_tier: @@ -433,19 +457,21 @@ def _require_provider_compatible_options(options: RunOptions, resolved: _Resolve def _build_executor(options: RunOptions, resolved: _ResolvedRun) -> PipelineExecutor: - secrets = build_secret_store( - "environment" if options.sandbox else resolved.secret_provider, - None if options.sandbox else resolved.secret_config, - ) - auth = build_auth(resolved.source_config, secrets) - try: - source_adapter = build_source_adapter( - resolved.source_config, - auth, - plugin_registry=resolved.plugin_registry, + source_adapter = None + if options.repair_window is None: + secrets = build_secret_store( + "environment" if options.sandbox else resolved.secret_provider, + None if options.sandbox else resolved.secret_config, ) - except ConnectorPluginError as error: - raise ClickException(str(error)) from error + auth = build_auth(resolved.source_config, secrets) + try: + source_adapter = build_source_adapter( + resolved.source_config, + auth, + plugin_registry=resolved.plugin_registry, + ) + except ConnectorPluginError as error: + raise ClickException(str(error)) from error stores = _build_control_stores(options, resolved) catalog_publisher = ( @@ -460,7 +486,11 @@ def _build_executor(options: RunOptions, resolved: _ResolvedRun) -> PipelineExec else None ) warehouse = _build_warehouse_runtime(resolved) - ingestion = _build_ingestion_runner(options, resolved, source_adapter, warehouse, stores) + ingestion = ( + _build_ingestion_runner(options, resolved, source_adapter, warehouse, stores) + if source_adapter is not None + else None + ) transform_runner = _build_transform_runner(resolved, warehouse) return PipelineExecutor( pipeline_id=resolved.pipeline_id, @@ -478,6 +508,7 @@ def _build_executor(options: RunOptions, resolved: _ResolvedRun) -> PipelineExec registry_output=options.catalog_output, catalog_publisher=catalog_publisher, leases=stores.leases, + repair_window=options.repair_window, ) @@ -711,16 +742,24 @@ def _render_dry_run(options: RunOptions, resolved: _ResolvedRun, *, console: Con missing_bigquery_catalog = ( resolved.warehouse_provider == "bigquery" and not resolved.gcp_project ) - _print_plan( - resolved.source_config.name, - resolved.endpoint_relations, - _selected_endpoints(resolved.source_config, resolved.graph_plan), - console=console, - catalog_label="" if missing_bigquery_catalog else None, - sandbox=options.sandbox, - guarded_free_tier=options.guarded_free_tier, - batch_rows=options.batch_rows, - ) + if options.repair_window is not None: + window = options.repair_window + console.print( + f"Retained-raw repair: {window.start_date} inclusive to {window.end_date} exclusive " + "(UTC). No source extraction or watermark changes. Current raw data is recompiled; " + "staging may scan outside the selected dates." + ) + else: + _print_plan( + resolved.source_config.name, + resolved.endpoint_relations, + _selected_endpoints(resolved.source_config, resolved.graph_plan), + console=console, + catalog_label="" if missing_bigquery_catalog else None, + sandbox=options.sandbox, + guarded_free_tier=options.guarded_free_tier, + batch_rows=options.batch_rows, + ) if resolved.graph_plan is not None: _print_graph_plan( resolved.graph_plan, diff --git a/src/dander/cli/runtime_command.py b/src/dander/cli/runtime_command.py index 71aea6da..3dbddbba 100644 --- a/src/dander/cli/runtime_command.py +++ b/src/dander/cli/runtime_command.py @@ -5,6 +5,7 @@ import json import logging import os +import re import tempfile import time from pathlib import Path @@ -31,6 +32,7 @@ PhysicalPlanError, deserialize_physical_plan, ) +from dander.pipeline.repair import GraphRepairWindow from dander.runtime_contract import ( RUNTIME_CONTRACT, LauncherContext, @@ -60,6 +62,8 @@ _CONSOLE = Console() _LOGGER = logging.getLogger(__name__) _PROJECTED_PLATFORMS_ENV = "DANDER_PLATFORMS_CONFIG_JSON" +_REPAIR_ENV = "DANDER_GRAPH_REPAIR_JSON" +_REPAIR_CONTRACT = "io.dander.graph-repair/v1" class RuntimePlatformConfigError(RuntimeError): @@ -191,6 +195,7 @@ def execute_runtime( batch_rows: int = typer.Option(10_000, "--batch-rows", min=1, max=100_000), budget_name: str = typer.Option("dander-sbx-cap", "--budget-name", hidden=True), physical_plan: str | None = typer.Option(None, "--physical-plan", hidden=True), + graph_repair_contract: str | None = typer.Option(None, "--graph-repair-contract", hidden=True), ) -> None: """Execute one pipeline and emit only versioned, non-sensitive JSON Lines.""" try: @@ -198,6 +203,9 @@ def execute_runtime( validate_runtime_identifier(pipeline, label="pipeline id") validate_runtime_identifier(platform, label="platform") context = LauncherContext.from_environment() + repair_window, repair_graph_content_sha256 = _load_repair_selection( + graph_repair_contract, context=context + ) selected_physical_plan = _load_physical_plan_argument( physical_plan, pipeline_id=pipeline, @@ -245,6 +253,8 @@ def execute_runtime( catalog_output=catalog_output, publish_dataplex=False, dataplex_location="us", + repair_window=repair_window, + repair_graph_content_sha256=repair_graph_content_sha256, ) with ( graceful_signal_handlers(), @@ -373,6 +383,52 @@ def execute_runtime( ) +def _load_repair_selection( + contract: str | None, *, context: LauncherContext +) -> tuple[GraphRepairWindow | None, str | None]: + raw = os.environ.get(_REPAIR_ENV) + if contract is None and raw is None: + return None, None + if contract != _REPAIR_CONTRACT or raw is None or len(raw) > 1024: + raise RuntimeContractError("Date repair requires its supported contract and selection.") + if context.launcher != "cloud_run": + raise RuntimeContractError("Date repair requires Cloud Run execution context.") + try: + payload = json.loads(raw) + if not isinstance(payload, dict) or set(payload) != { + "start_date", + "end_date", + "graph_content_sha256", + "execution_name", + }: + raise ValueError("invalid selection") + content_sha256 = payload["graph_content_sha256"] + if ( + not isinstance(content_sha256, str) + or re.fullmatch(r"[0-9a-f]{64}", content_sha256) is None + ): + raise ValueError("invalid graph identity") + execution_name = os.environ.get("CLOUD_RUN_EXECUTION") + if not execution_name or not isinstance(payload["execution_name"], str): + raise ValueError("missing execution identity") + validate_runtime_identifier(payload["execution_name"], label="repair execution") + window = GraphRepairWindow.model_validate( + {"start_date": payload["start_date"], "end_date": payload["end_date"]} + ) + except (TypeError, ValueError) as error: + raise RuntimeContractError( + "Date repair selection is invalid for this execution." + ) from error + # Cloud Run snapshots Job configuration for every execution. An independently scheduled + # normal run may inherit a previous repair selection before Control restores the Job; + # only the named execution is allowed to apply that otherwise valid date window. + if payload["execution_name"] != execution_name: + return None, None + if context.shard_count != 1: + raise RuntimeContractError("Date repair requires a single Cloud Run task.") + return window, content_sha256 + + def _load_physical_plan_argument( value: str | None, *, diff --git a/src/dander/control/application.py b/src/dander/control/application.py index 4f3235be..77d7738f 100644 --- a/src/dander/control/application.py +++ b/src/dander/control/application.py @@ -3,9 +3,9 @@ from __future__ import annotations import threading -from dataclasses import dataclass +from dataclasses import dataclass, replace from datetime import UTC, datetime -from typing import TYPE_CHECKING, Protocol +from typing import TYPE_CHECKING, Protocol, runtime_checkable from dander import __version__ from dander.control.bundle import BUNDLE_ID, packaged_bundle_digest @@ -44,12 +44,14 @@ RunPageResponse, RunStatusResponse, ) +from dander.control.repair import GraphRepairPreviewResponse, preview_graph_repair from dander.control.run_explanation import RunExplanationResponse, explain_run if TYPE_CHECKING: from collections.abc import Callable from dander.control.orchestration import PlacementMode, RunSubmission + from dander.pipeline.repair import GraphRepairWindow from dander.plugins import InstalledConnectorPlugin MAX_LOG_RECORDS = 500 @@ -123,6 +125,7 @@ def find_api_start( environments: tuple[str, ...], placement_mode: PlacementMode, idempotency_key: str, + repair_window: GraphRepairWindow | None = None, ) -> RunStatusResponse | None: ... def list(self, *, cursor: str | None, limit: int) -> RunPageResponse: ... @@ -165,6 +168,18 @@ def idempotency_lookup( ) -> tuple[tuple[str, ...], PlacementMode]: ... +@runtime_checkable +class RepairSubmissionResolver(Protocol): + """Optional repair admission supported by selected hosted execution plans.""" + + @property + def supports_repairs(self) -> bool: ... + + def repair_environments(self, record: GraphRecord) -> tuple[str, ...]: ... + + def require_repair_submission(self, submission: RunSubmission) -> None: ... + + class CanonicalGraphValidator: """Reapply Dander's canonical graph semantics without selecting a provider.""" @@ -230,6 +245,11 @@ def capabilities(self) -> CapabilitiesResponse: operations.extend( ["run.start", "run.read", "run.explain", "run.logs", "run.cancel", "run.replay"] ) + if ( + isinstance(self.submission_resolver, RepairSubmissionResolver) + and self.submission_resolver.supports_repairs + ): + operations.extend(["graph.repair-preview", "run.repair"]) return CapabilitiesResponse( dander_version=__version__, contract=ContractIdentity(id=BUNDLE_ID, sha256=packaged_bundle_digest()), @@ -317,6 +337,29 @@ def preview_graph_changes( baseline = self.require_graph_revision(project, graph, expected_revision) return compare_graph_changes(baseline.document, candidate) + def preview_repair( + self, + project: str, + graph: str, + window: GraphRepairWindow, + *, + expected_revision: str, + ) -> GraphRepairPreviewResponse: + """Validate declared output eligibility and show currently available routes.""" + record = self.require_graph_revision(project, graph, expected_revision) + resolver = self._require_repair_resolver() + environments = resolver.repair_environments(record) + if not environments: + raise ControlOperationUnavailableError( + "No current execution plan supports output repair for this graph." + ) + return preview_graph_repair( + record.document, + content_sha256=record.content_sha256, + window=window, + environments=environments, + ) + def start_run( self, project: str, @@ -327,10 +370,25 @@ def start_run( environment: str | None = None, size_class: str | None = None, estimated_input_bytes: int | None = None, + repair_window: GraphRepairWindow | None = None, ) -> RunStatusResponse: lifecycle = self._require_lifecycle() record = self.require_graph_revision(project, graph, expected_revision) resolver = self._require_submission_resolver() + if repair_window is not None: + preview = self.preview_repair( + project, graph, repair_window, expected_revision=expected_revision + ) + if environment is None: + if len(preview.environments) != 1: + raise ControlOperationConflictError( + "Choose an execution environment from the repair preview." + ) + environment = preview.environments[0] + elif environment not in preview.environments: + raise ControlOperationUnavailableError( + "The selected environment does not support output repair." + ) with self._run_start_lock: if size_class is None and estimated_input_bytes is None: idempotency_environments, placement_mode = resolver.idempotency_lookup( @@ -342,6 +400,7 @@ def start_run( environments=idempotency_environments, placement_mode=placement_mode, idempotency_key=idempotency_key, + repair_window=repair_window, ) if existing is not None: return existing @@ -355,6 +414,12 @@ def start_run( ) if submission.graph != record or submission.idempotency_key != idempotency_key: raise RuntimeError("The submission resolver changed validated request identity.") + if repair_window is not None: + submission = replace( + submission, + trigger=replace(submission.trigger, repair_window=repair_window), + ) + self._require_repair_resolver().require_repair_submission(submission) return lifecycle.start(submission) def get_run(self, address: RunAddress) -> RunStatusResponse: @@ -417,6 +482,14 @@ def _require_submission_resolver(self) -> RunSubmissionResolver: ) return self.submission_resolver + def _require_repair_resolver(self) -> RepairSubmissionResolver: + resolver = self._require_submission_resolver() + if not isinstance(resolver, RepairSubmissionResolver) or not resolver.supports_repairs: + raise ControlOperationUnavailableError( + "Output repair is unavailable for the selected execution profile." + ) + return resolver + def graph_resource_response(record: GraphRecord) -> GraphResourceResponse: """Project one internal record without exposing its provider-native revision in JSON.""" diff --git a/src/dander/control/bundle.py b/src/dander/control/bundle.py index aa603cf5..c20291d8 100644 --- a/src/dander/control/bundle.py +++ b/src/dander/control/bundle.py @@ -17,6 +17,7 @@ ConnectorCatalogResponse, ControlBootstrapDescriptor, DeploymentPreviewResponse, + DestinationDocument, GraphCreateRequest, GraphPageResponse, GraphResourceResponse, @@ -32,8 +33,10 @@ RunState, RunStatusResponse, ) +from dander.control.repair import GraphRepairOutput, GraphRepairPreviewResponse from dander.control.run_explanation import RunExplanationResponse, explain_run from dander.pipeline.graph import graph_to_payload +from dander.pipeline.repair import GraphRepairWindow BUNDLE_ID: Final = "io.dander.control.contracts/v1" MANIFEST_SCHEMA: Final = "io.dander.control.contracts-manifest/v1" @@ -49,6 +52,8 @@ "deployment-preview": DeploymentPreviewResponse, "graph-create": GraphCreateRequest, "graph-change-preview": GraphChangePreviewResponse, + "graph-repair-preview": GraphRepairPreviewResponse, + "graph-repair-window": GraphRepairWindow, "graph-page": GraphPageResponse, "graph-resource": GraphResourceResponse, "graph-validation": GraphValidationResponse, @@ -661,6 +666,34 @@ def _fixtures() -> dict[str, tuple[str, dict[str, Any]]]: ) ).model_dump(mode="json"), ), + "graph-repair-window": ( + "graph-repair-window", + {"start_date": "2026-09-01", "end_date": "2026-09-03"}, + ), + "graph-repair-preview": ( + "graph-repair-preview", + GraphRepairPreviewResponse( + graph_content_sha256="a" * 64, + window=GraphRepairWindow.model_validate( + {"start_date": "2026-09-01", "end_date": "2026-09-03"} + ), + outputs=( + GraphRepairOutput( + node_id="daily_output", + name="Daily output", + destination=DestinationDocument( + dataset="analytics", + table="daily_output", + business_key=["id"], + ), + partition_field="updated_at", + partition_type="TIMESTAMP", + business_key=("id",), + ), + ), + environments=("gcp",), + ).model_dump(mode="json"), + ), "run-status": ( "run-status", { diff --git a/src/dander/control/cloud_run_execution_backend.py b/src/dander/control/cloud_run_execution_backend.py index 17d021c8..45b06847 100644 --- a/src/dander/control/cloud_run_execution_backend.py +++ b/src/dander/control/cloud_run_execution_backend.py @@ -11,6 +11,8 @@ import json import re from collections.abc import Mapping +from contextlib import suppress +from copy import deepcopy from datetime import UTC, datetime from typing import TYPE_CHECKING, Protocol, cast @@ -30,12 +32,14 @@ ResultsState, RunOutcome, RunTrigger, + TriggerKind, ) from dander.identity.aws_google import FargateIdentityError from dander.identity.control_google import ( GoogleControlIdentityError, prepare_control_google_identity, ) +from dander.pipeline.repair import GraphRepairWindow from dander.providers.cloud_run import CloudRunBinding, CloudRunOperationError if TYPE_CHECKING: @@ -55,6 +59,8 @@ _MAX_CURSOR_LENGTH = 2_048 _MAX_LOG_MESSAGE_LENGTH = 16_384 _DEFAULT_TIMEOUT_SECONDS = 30.0 +_REPAIR_ENV = "DANDER_GRAPH_REPAIR_JSON" +_REPAIR_ARGUMENTS = ("--graph-repair-contract", "io.dander.graph-repair/v1") class _Response(Protocol): @@ -126,7 +132,6 @@ def submit_or_adopt( trigger: RunTrigger, ) -> BackendHandle: """Start or adopt the deterministic Cloud Run execution for one attempt.""" - del trigger binding = self._binding_for(plan) token = _execution_token(run_id, attempt_id) try: @@ -134,34 +139,26 @@ def submit_or_adopt( except CloudRunOperationError as error: raise ExecutionBackendError("Cloud Run execution identity is invalid.") from error handle = BackendHandle(backend_id=_BACKEND_ID, execution_id=execution_resource) + repair_payload = _repair_payload(plan, trigger, execution_resource) execution = self._try_get_execution(execution_resource) if execution is not None: - self._validate_execution(binding, execution, execution_resource) + self._validate_submission_execution( + plan, binding, execution, execution_resource, repair_payload + ) return handle job = self._get_job(binding) self._validate_job(plan, binding, job) if job.get("startExecutionToken") == token: + _require_repair_payload( + _job_task(job), repair_payload, _execution_arguments(plan, repair_payload) + ) return handle etag = job.get("etag") if not isinstance(etag, str) or not etag: raise ExecutionBackendError("Cloud Run Job does not expose a concurrency token.") - update = { - key: value - for key, value in job.items() - if key - in { - "name", - "labels", - "annotations", - "launchStage", - "binaryAuthorization", - "template", - "client", - "clientVersion", - "etag", - } - } + update = _job_update(job) + _set_repair_payload(update, plan, repair_payload) update["startExecutionToken"] = token try: self._request_json( @@ -174,11 +171,18 @@ def submit_or_adopt( except _GoogleCallError as start_error: execution = self._try_get_execution(execution_resource) if execution is not None: - self._validate_execution(binding, execution, execution_resource) + self._validate_submission_execution( + plan, binding, execution, execution_resource, repair_payload + ) return handle reconciled_job = self._get_job(binding) self._validate_job(plan, binding, reconciled_job) if reconciled_job.get("startExecutionToken") == token: + _require_repair_payload( + _job_task(reconciled_job), + repair_payload, + _execution_arguments(plan, repair_payload), + ) return handle raise ExecutionBackendError( "Cloud Run execution could not be created or adopted." @@ -214,6 +218,7 @@ def observe(self, plan: ExecutionPlan, handle: BackendHandle) -> BackendObservat outcome = RunOutcome.FAILED stage = "failed" failure_code = _execution_failure_code(execution) + cleanup_state = self._restore_repair_job(plan, binding, execution, handle.execution_id) result_summary = None if outcome is RunOutcome.SUCCEEDED: try: @@ -238,13 +243,61 @@ def observe(self, plan: ExecutionPlan, handle: BackendHandle) -> BackendObservat if outcome is RunOutcome.SUCCEEDED else ResultsState.UNAVAILABLE ), - cleanup_state=CleanupState.CONFIRMED, + cleanup_state=cleanup_state, observed_at=self._now(), stage=stage, failure_code=failure_code, result_summary=result_summary, ) + def _restore_repair_job( + self, + plan: ExecutionPlan, + binding: CloudRunBinding, + execution: Mapping[str, object], + resource: str, + ) -> CleanupState: + """Restore only this completed execution's still-owned Job configuration.""" + repair_payload = _execution_repair_payload(plan, execution, resource) + if repair_payload is None: + return CleanupState.CONFIRMED + self._validate_submission_execution(plan, binding, execution, resource, repair_payload) + token = resource.rsplit("-", maxsplit=1)[-1] + try: + job = self._get_job(binding) + except ExecutionBackendError: + return CleanupState.PENDING + if not _job_has_repair_selection(job, token=token, payload=repair_payload, plan=plan): + return CleanupState.CONFIRMED + # A new deployment or launch supersedes ownership. Do not restore an old plan over it. + try: + self._validate_job(plan, binding, job) + except ExecutionBackendError: + return CleanupState.CONFIRMED + if not isinstance(job.get("etag"), str) or not job["etag"]: + return CleanupState.PENDING + update = _job_update(job) + _set_repair_payload(update, plan, None) + # Lost responses and ETag conflicts are reconciled below before another patch. + with suppress(_GoogleCallError, ExecutionBackendError): + # Omitting both execution-token fields updates configuration without starting work. + self._request_json( + "restore job", + "PATCH", + f"{_API_ROOT}/{binding.job_resource}", + json=update, + expected=(200,), + ) + try: + current = self._get_job(binding) + except ExecutionBackendError: + return CleanupState.PENDING + return ( + CleanupState.PENDING + if _job_has_repair_selection(current, token=token, payload=repair_payload, plan=plan) + else CleanupState.CONFIRMED + ) + def logs( self, plan: ExecutionPlan, @@ -363,6 +416,8 @@ def _binding_for(self, plan: ExecutionPlan) -> CloudRunBinding: or image.group("project") != binding.project_id or image.group("region") != binding.region or template.workload_identity != binding.runtime_service_account + or _REPAIR_ARGUMENTS[0] in template.command + or any(name == _REPAIR_ENV for name, _ in template.environment) ): raise ExecutionBackendError("The execution plan does not match its Cloud Run binding.") return binding @@ -430,7 +485,8 @@ def _validate_job( job.get("name") != binding.job_resource or not isinstance(container, Mapping) or container.get("image") != plan.image - or container.get("args") != list(plan_template.command) + or container.get("args") + not in [list(plan_template.command), list(plan_template.command + _REPAIR_ARGUMENTS)] or template.get("taskCount") != plan_template.schedule.task_count or template.get("parallelism") != plan_template.schedule.maximum_parallelism or task_template.get("serviceAccount") != binding.runtime_service_account @@ -448,6 +504,31 @@ def _validate_execution( if execution.get("name") != resource or execution.get("job") != binding.job_name: raise ExecutionBackendError("Cloud Run returned an unexpected execution.") + @classmethod + def _validate_submission_execution( + cls, + plan: ExecutionPlan, + binding: CloudRunBinding, + execution: Mapping[str, object], + resource: str, + repair_payload: str | None, + ) -> None: + cls._validate_execution(binding, execution, resource) + task = execution.get("template") + containers = task.get("containers") if isinstance(task, Mapping) else None + container = containers[0] if isinstance(containers, list) and len(containers) == 1 else None + if ( + not isinstance(task, Mapping) + or not isinstance(container, Mapping) + or task.get("serviceAccount") != binding.runtime_service_account + or container.get("image") != plan.image + or container.get("args") != _execution_arguments(plan, repair_payload) + ): + raise ExecutionBackendError( + "The Cloud Run execution does not match its immutable execution plan." + ) + _require_repair_payload(task, repair_payload, _execution_arguments(plan, repair_payload)) + def _request_json( self, operation: str, @@ -499,6 +580,143 @@ def _validate_binding(binding: CloudRunBinding) -> None: raise ExecutionBackendError("A Cloud Run plan binding is invalid.") +def _repair_payload(plan: ExecutionPlan, trigger: RunTrigger, resource: str) -> str | None: + if trigger.repair_window is None: + return None + if plan.execution_template.schedule.task_count != 1: + raise ExecutionBackendError("Date repair requires a single Cloud Run task.") + return json.dumps( + { + **trigger.repair_window.model_dump(mode="json"), + "graph_content_sha256": plan.graph_content_sha256, + "execution_name": resource.rsplit("/", maxsplit=1)[-1], + }, + sort_keys=True, + separators=(",", ":"), + ) + + +def _execution_arguments(plan: ExecutionPlan, repair_payload: str | None) -> list[str]: + suffix = _REPAIR_ARGUMENTS if repair_payload is not None else () + return list(plan.execution_template.command + suffix) + + +def _execution_repair_payload( + plan: ExecutionPlan, execution: Mapping[str, object], resource: str +) -> str | None: + task = execution.get("template") + containers = task.get("containers") if isinstance(task, Mapping) else None + container = containers[0] if isinstance(containers, list) and len(containers) == 1 else None + environment = container.get("env", []) if isinstance(container, Mapping) else [] + if not isinstance(environment, list): + raise ExecutionBackendError("Cloud Run execution environment is invalid.") + selected = [ + item + for item in environment + if isinstance(item, Mapping) and item.get("name") == _REPAIR_ENV + ] + if not selected: + return None + raw = selected[0].get("value") + try: + if len(selected) != 1 or not isinstance(raw, str) or len(raw) > 1024: + raise ValueError("invalid repair selection") + payload = json.loads(raw) + if not isinstance(payload, dict): + raise ValueError("invalid repair selection") + window = GraphRepairWindow.model_validate( + {"start_date": payload["start_date"], "end_date": payload["end_date"]} + ) + trigger = RunTrigger(kind=TriggerKind.API, trigger_id="control-api", repair_window=window) + if raw != _repair_payload(plan, trigger, resource): + raise ValueError("repair selection does not match execution") + except (KeyError, TypeError, ValueError) as error: + raise ExecutionBackendError("Cloud Run execution repair selection is invalid.") from error + return raw + + +def _job_has_repair_selection( + job: Mapping[str, object], *, token: str, payload: str, plan: ExecutionPlan +) -> bool: + if job.get("startExecutionToken") != token: + return False + try: + _require_repair_payload(_job_task(job), payload, _execution_arguments(plan, payload)) + except ExecutionBackendError: + return False + return True + + +def _job_update(job: Mapping[str, object]) -> dict[str, object]: + return { + key: deepcopy(value) + for key, value in job.items() + if key + in { + "name", + "labels", + "annotations", + "launchStage", + "binaryAuthorization", + "template", + "client", + "clientVersion", + "etag", + } + } + + +def _job_task(job: Mapping[str, object]) -> Mapping[str, object]: + template = job.get("template") + task = template.get("template") if isinstance(template, Mapping) else None + if not isinstance(task, Mapping): + raise ExecutionBackendError("Cloud Run Job task template is invalid.") + return cast("Mapping[str, object]", task) + + +def _require_repair_payload( + task: Mapping[str, object], expected: str | None, arguments: list[str] +) -> None: + containers = task.get("containers") + container = containers[0] if isinstance(containers, list) and len(containers) == 1 else None + if not isinstance(container, Mapping): + raise ExecutionBackendError("Cloud Run task container is invalid.") + if container.get("args") != arguments: + raise ExecutionBackendError("Cloud Run execution has a different date-repair command.") + environment = container.get("env", []) + if not isinstance(environment, list) or any( + not isinstance(item, Mapping) for item in environment + ): + raise ExecutionBackendError("Cloud Run task environment is invalid.") + selected = [item for item in environment if item.get("name") == _REPAIR_ENV] + wanted = [{"name": _REPAIR_ENV, "value": expected}] if expected is not None else [] + if selected != wanted: + raise ExecutionBackendError("Cloud Run execution has a different date-repair selection.") + + +def _set_repair_payload( + update: dict[str, object], plan: ExecutionPlan, payload: str | None +) -> None: + # The patch has already been deep-copied. Preserve every unrelated container setting, + # and always clear a previous selection when a normal run follows a repair. + task = _job_task(update) + containers = task.get("containers") + container = containers[0] if isinstance(containers, list) and len(containers) == 1 else None + if not isinstance(container, dict): + raise ExecutionBackendError("Cloud Run task container is invalid.") + environment = container.get("env", []) + if not isinstance(environment, list) or any( + not isinstance(item, Mapping) for item in environment + ): + raise ExecutionBackendError("Cloud Run task environment is invalid.") + remaining = [item for item in environment if item.get("name") != _REPAIR_ENV] + if payload is not None: + remaining.append({"name": _REPAIR_ENV, "value": payload}) + if remaining or "env" in container: + container["env"] = remaining + container["args"] = _execution_arguments(plan, payload) + + def _execution_token(run_id: str, attempt_id: str) -> str: return hashlib.sha256(f"{run_id}\0{attempt_id}".encode()).hexdigest()[:16] diff --git a/src/dander/control/contracts/v1/fixtures/graph-repair-preview.json b/src/dander/control/contracts/v1/fixtures/graph-repair-preview.json new file mode 100644 index 00000000..8434b4bd --- /dev/null +++ b/src/dander/control/contracts/v1/fixtures/graph-repair-preview.json @@ -0,0 +1 @@ +{"environments":["gcp"],"estimated_cost_usd":null,"estimated_rows":null,"graph_content_sha256":"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa","limitations":["Start date is included; end date is excluded. Timestamp fields use UTC calendar dates.","Source extraction and its normal saved progress are unchanged. This is not a historical snapshot.","Only selected output dates are published; computing them can scan data outside that range.","A record moving across the date boundary stops publication for that output. Choose a wider range or rebuild the full output.","Each output publishes atomically. Earlier successful outputs remain if a later output fails; repeating the same repair converges on the current raw data.","Execution requires a current repair-capable runtime with a BigQuery warehouse. Runtime validation and warehouse checks still apply."],"outputs":[{"business_key":["id"],"destination":{"business_key":["id"],"dataset":"analytics","project":null,"table":"daily_output"},"name":"Daily output","node_id":"daily_output","partition_field":"updated_at","partition_type":"TIMESTAMP"}],"source":"retained_raw","summary":"Rebuild the selected output dates from the raw data currently stored in your warehouse.","window":{"end_date":"2026-09-03","start_date":"2026-09-01"}} diff --git a/src/dander/control/contracts/v1/fixtures/graph-repair-window.json b/src/dander/control/contracts/v1/fixtures/graph-repair-window.json new file mode 100644 index 00000000..eb398ba7 --- /dev/null +++ b/src/dander/control/contracts/v1/fixtures/graph-repair-window.json @@ -0,0 +1 @@ +{"end_date":"2026-09-03","start_date":"2026-09-01"} diff --git a/src/dander/control/contracts/v1/manifest.json b/src/dander/control/contracts/v1/manifest.json index 3ea99965..465c79c6 100644 --- a/src/dander/control/contracts/v1/manifest.json +++ b/src/dander/control/contracts/v1/manifest.json @@ -1 +1 @@ -{"bundle_id":"io.dander.control.contracts/v1","bundle_sha256":"03fdac1a19a98f10f1af6fa7821b705291833f88a23e723300abace660130d5b","files":[{"contract":"api-error","kind":"fixture","path":"fixtures/api-error.json","sha256":"ed11f73d577b895d9cb6c1167e696b95a30b8ffa2c2d784ff83b7c6346b6ca0d"},{"contract":"capabilities","kind":"fixture","path":"fixtures/capabilities.json","sha256":"91dff64c697c6a527c9e71f2b67e167eea434e19b28a884bce83e390ce94c7dd"},{"contract":"connector-catalog","kind":"fixture","path":"fixtures/connector-catalog.json","sha256":"c5d8c023bebdc7f39d445bbf8e54f8b3c2f0a72f57aaff3e5c8334274843dc25"},{"contract":"control-bootstrap","kind":"fixture","path":"fixtures/control-bootstrap.json","sha256":"4dac4acea0153c63bf6fbefc6a2f25a4354c1f61f448af3330adf2d1a95d9c80"},{"contract":"deployment-preview","kind":"fixture","path":"fixtures/deployment-preview.json","sha256":"aa9703ab7e8e1ef91a1f8324798709b01bbc5580ba14e3b936fe882820fba539"},{"contract":"graph-change-preview","kind":"fixture","path":"fixtures/graph-change-preview.json","sha256":"4c108460b9567164472b9f1cad035b23b0a07da31f2062a0de5cfe1f91b79333"},{"contract":"graph-create","kind":"fixture","path":"fixtures/graph-create.json","sha256":"c576c1e3109e9dbbdb3f9c81aec477e84639458dbfc2846e1f48f8f3331cc529"},{"contract":"graph-page","kind":"fixture","path":"fixtures/graph-page.json","sha256":"95c2133bb9b2c5aa18de13b73e91322450fcdbf958bee619b603276ac88ba816"},{"contract":"graph-resource","kind":"fixture","path":"fixtures/graph-resource.json","sha256":"68b523de7a34f8383f6d32e47587669091e1292dc93d5afe6b978c7b01dc811d"},{"contract":"graph-validation","kind":"fixture","path":"fixtures/graph-validation.json","sha256":"64a5f8a1f39bde3ef30f5401cf285a9de6091aad9771591b1817168e646dc3e0"},{"contract":"log-page","kind":"fixture","path":"fixtures/log-page.json","sha256":"481c46c6365cfdcaf184dacd53f59cb34e894ec9a11675e876bd220aa609d76c"},{"contract":"mutation-result","kind":"fixture","path":"fixtures/mutation-result.json","sha256":"aa0eb0a3e48f03102c1165aff9933e2be1c54f80a2e29bb67a0aa3bc1fae9898"},{"contract":"operation-catalog","kind":"fixture","path":"fixtures/operation-catalog.json","sha256":"adb73dac2061e1b00a85148cfaf810040aeafb32fb7fd670c077596853f05508"},{"contract":"pipeline-graph","kind":"fixture","path":"fixtures/pipeline-graph-alias-input.json","sha256":"de73bc8e172dfa8daba536fe14fef13a272dd61e8f1bcfef286fe043477ca451"},{"contract":"pipeline-graph","kind":"fixture","path":"fixtures/pipeline-graph.json","sha256":"65c6fa7b88221b1e9a1e51ec773fb9c374533d0ed54e2f2a5c8c7a1242b1b1bc"},{"contract":"plugin-catalog","kind":"fixture","path":"fixtures/plugin-catalog.json","sha256":"039cf19f5ce4c4d5b35e8b1801ff151cf0912b7f90792196dce76e2fce7726df"},{"contract":"project-list","kind":"fixture","path":"fixtures/project-list.json","sha256":"157e70424be88bfaf0429d66dd3a8103a4e402e3f9067685bfe78e2faadf56d3"},{"contract":"run-explanation","kind":"fixture","path":"fixtures/run-explanation.json","sha256":"e0ccfd75142882f91fcc0e352b09c5ecf92193646c3a98107f56fa6482b15cd4"},{"contract":"run-page","kind":"fixture","path":"fixtures/run-page.json","sha256":"cae943a95060651795160b980d8c25b99ff92440cf905b115b96282cf624fd87"},{"contract":"run-request","kind":"fixture","path":"fixtures/run-request.json","sha256":"36013392b347194d13be6747840c93edb9c8dbf19468a6e19032f26899824f71"},{"contract":"run-status","kind":"fixture","path":"fixtures/run-status.json","sha256":"c2d76c2b9879b45d954027247da0849f216db44d8f449ea4a0c24443a8283556"},{"contract":"api-error","kind":"schema","path":"schemas/api-error.schema.json","sha256":"de86daa1616237146b0666f3c69bf7c22d5679595d756ca980becb69b1613164"},{"contract":"capabilities","kind":"schema","path":"schemas/capabilities.schema.json","sha256":"3be0992ebd1cbbb0d4dfb2c1205b715a7e4c6131b8a4f2ebf47564b4c354ae0c"},{"contract":"connector-catalog","kind":"schema","path":"schemas/connector-catalog.schema.json","sha256":"132f169721b81569c4934d03c34e1a276176b40a45711de61264bb4928420dc5"},{"contract":"control-bootstrap","kind":"schema","path":"schemas/control-bootstrap.schema.json","sha256":"013a30d4d4c6f417a898ce0c81fc00dcbfc138e76b79d49694d1af7a6f8b4096"},{"contract":"deployment-preview","kind":"schema","path":"schemas/deployment-preview.schema.json","sha256":"67449ad10f78b972ff3fbda98948f3679dcdecf908ae468fe8ddf843158dfe81"},{"contract":"graph-change-preview","kind":"schema","path":"schemas/graph-change-preview.schema.json","sha256":"e45220e046d17c5b608da11e59bbeec6160f56f910b568bd15c8651465e585bf"},{"contract":"graph-create","kind":"schema","path":"schemas/graph-create.schema.json","sha256":"15cefbe92bdc171916d0118bfa33c665ac39a409661b222861a6f72a23331b41"},{"contract":"graph-page","kind":"schema","path":"schemas/graph-page.schema.json","sha256":"204018767f05e10a110f7caa177312ffea1266cfbd7eb841ed8a8ff90fbe1e09"},{"contract":"graph-resource","kind":"schema","path":"schemas/graph-resource.schema.json","sha256":"dc3ad0085768407720c825c95a5ff7f1ea01e85f6377a8f32445e926a9e88c61"},{"contract":"graph-validation","kind":"schema","path":"schemas/graph-validation.schema.json","sha256":"e835cd4ed7f698420a643305116f46644d54c8514548b849e34c23d398123d54"},{"contract":"log-page","kind":"schema","path":"schemas/log-page.schema.json","sha256":"bdc4059aecaeb7e18c93b3173726b256119306137a0988ddf1673bb50d260bc6"},{"contract":"mutation-result","kind":"schema","path":"schemas/mutation-result.schema.json","sha256":"078ca417acce9d4cd61c89607e5174156e986f432055c07509b9282fe2f537b7"},{"contract":"operation-catalog","kind":"schema","path":"schemas/operation-catalog.schema.json","sha256":"3ed2d3fd7defc677f7841b4e252a31dffd739922829d572ba9d5a7a635fae000"},{"contract":"pipeline-graph","kind":"schema","path":"schemas/pipeline-graph.schema.json","sha256":"3c2aa99c1ab5a93f2c590c0634fd1b39b8d1e84c56fbe8e42e0ff3051ee40377"},{"contract":"plugin-catalog","kind":"schema","path":"schemas/plugin-catalog.schema.json","sha256":"d934e93c333123284d96e2c9fb19080c952c6e353a721207d36228347923eb83"},{"contract":"project-list","kind":"schema","path":"schemas/project-list.schema.json","sha256":"a61e8c0dff565792079c1954584d7424a90fad35de85745682492a60e8067243"},{"contract":"run-explanation","kind":"schema","path":"schemas/run-explanation.schema.json","sha256":"e00ca3f82bfca14e77d61f00b14bc2c314820b680ff357b3c45e988eae6a8324"},{"contract":"run-page","kind":"schema","path":"schemas/run-page.schema.json","sha256":"6eb580e2803181117c7e52b61d46d183f291865c8d147369cba0488de06797dc"},{"contract":"run-request","kind":"schema","path":"schemas/run-request.schema.json","sha256":"0a8aa50cf4e82b573476e446e1c9eb6fba1482625abb300b719199ce68866eba"},{"contract":"run-status","kind":"schema","path":"schemas/run-status.schema.json","sha256":"d29fbb9d1614a5ea27a762dab58089c7b352a4cd2f8d27e019e8896237359b07"}],"json_schema_dialect":"https://json-schema.org/draft/2020-12/schema","reference_convention":"self-contained internal #/$defs references","schema":"io.dander.control.contracts-manifest/v1"} +{"bundle_id":"io.dander.control.contracts/v1","bundle_sha256":"a28316b7e5158e0520fe1c24d59885083714f47b67aa396892fa9742060fb279","files":[{"contract":"api-error","kind":"fixture","path":"fixtures/api-error.json","sha256":"ed11f73d577b895d9cb6c1167e696b95a30b8ffa2c2d784ff83b7c6346b6ca0d"},{"contract":"capabilities","kind":"fixture","path":"fixtures/capabilities.json","sha256":"91dff64c697c6a527c9e71f2b67e167eea434e19b28a884bce83e390ce94c7dd"},{"contract":"connector-catalog","kind":"fixture","path":"fixtures/connector-catalog.json","sha256":"c5d8c023bebdc7f39d445bbf8e54f8b3c2f0a72f57aaff3e5c8334274843dc25"},{"contract":"control-bootstrap","kind":"fixture","path":"fixtures/control-bootstrap.json","sha256":"4dac4acea0153c63bf6fbefc6a2f25a4354c1f61f448af3330adf2d1a95d9c80"},{"contract":"deployment-preview","kind":"fixture","path":"fixtures/deployment-preview.json","sha256":"aa9703ab7e8e1ef91a1f8324798709b01bbc5580ba14e3b936fe882820fba539"},{"contract":"graph-change-preview","kind":"fixture","path":"fixtures/graph-change-preview.json","sha256":"4c108460b9567164472b9f1cad035b23b0a07da31f2062a0de5cfe1f91b79333"},{"contract":"graph-create","kind":"fixture","path":"fixtures/graph-create.json","sha256":"c576c1e3109e9dbbdb3f9c81aec477e84639458dbfc2846e1f48f8f3331cc529"},{"contract":"graph-page","kind":"fixture","path":"fixtures/graph-page.json","sha256":"95c2133bb9b2c5aa18de13b73e91322450fcdbf958bee619b603276ac88ba816"},{"contract":"graph-repair-preview","kind":"fixture","path":"fixtures/graph-repair-preview.json","sha256":"1616eb1cc4944f5da6289e98084bd7744504a2892bdea17b77de91357d7dd536"},{"contract":"graph-repair-window","kind":"fixture","path":"fixtures/graph-repair-window.json","sha256":"2e31c98ace07994d1bed9d848cee474177bb42d723d2c18a5474098136bb3a9d"},{"contract":"graph-resource","kind":"fixture","path":"fixtures/graph-resource.json","sha256":"68b523de7a34f8383f6d32e47587669091e1292dc93d5afe6b978c7b01dc811d"},{"contract":"graph-validation","kind":"fixture","path":"fixtures/graph-validation.json","sha256":"64a5f8a1f39bde3ef30f5401cf285a9de6091aad9771591b1817168e646dc3e0"},{"contract":"log-page","kind":"fixture","path":"fixtures/log-page.json","sha256":"481c46c6365cfdcaf184dacd53f59cb34e894ec9a11675e876bd220aa609d76c"},{"contract":"mutation-result","kind":"fixture","path":"fixtures/mutation-result.json","sha256":"aa0eb0a3e48f03102c1165aff9933e2be1c54f80a2e29bb67a0aa3bc1fae9898"},{"contract":"operation-catalog","kind":"fixture","path":"fixtures/operation-catalog.json","sha256":"adb73dac2061e1b00a85148cfaf810040aeafb32fb7fd670c077596853f05508"},{"contract":"pipeline-graph","kind":"fixture","path":"fixtures/pipeline-graph-alias-input.json","sha256":"de73bc8e172dfa8daba536fe14fef13a272dd61e8f1bcfef286fe043477ca451"},{"contract":"pipeline-graph","kind":"fixture","path":"fixtures/pipeline-graph.json","sha256":"65c6fa7b88221b1e9a1e51ec773fb9c374533d0ed54e2f2a5c8c7a1242b1b1bc"},{"contract":"plugin-catalog","kind":"fixture","path":"fixtures/plugin-catalog.json","sha256":"039cf19f5ce4c4d5b35e8b1801ff151cf0912b7f90792196dce76e2fce7726df"},{"contract":"project-list","kind":"fixture","path":"fixtures/project-list.json","sha256":"157e70424be88bfaf0429d66dd3a8103a4e402e3f9067685bfe78e2faadf56d3"},{"contract":"run-explanation","kind":"fixture","path":"fixtures/run-explanation.json","sha256":"e0ccfd75142882f91fcc0e352b09c5ecf92193646c3a98107f56fa6482b15cd4"},{"contract":"run-page","kind":"fixture","path":"fixtures/run-page.json","sha256":"cae943a95060651795160b980d8c25b99ff92440cf905b115b96282cf624fd87"},{"contract":"run-request","kind":"fixture","path":"fixtures/run-request.json","sha256":"36013392b347194d13be6747840c93edb9c8dbf19468a6e19032f26899824f71"},{"contract":"run-status","kind":"fixture","path":"fixtures/run-status.json","sha256":"c2d76c2b9879b45d954027247da0849f216db44d8f449ea4a0c24443a8283556"},{"contract":"api-error","kind":"schema","path":"schemas/api-error.schema.json","sha256":"de86daa1616237146b0666f3c69bf7c22d5679595d756ca980becb69b1613164"},{"contract":"capabilities","kind":"schema","path":"schemas/capabilities.schema.json","sha256":"726c775f8b9c87b068ac57f42e011fede45a9af3d46feb9a56d80ab31d65308d"},{"contract":"connector-catalog","kind":"schema","path":"schemas/connector-catalog.schema.json","sha256":"132f169721b81569c4934d03c34e1a276176b40a45711de61264bb4928420dc5"},{"contract":"control-bootstrap","kind":"schema","path":"schemas/control-bootstrap.schema.json","sha256":"013a30d4d4c6f417a898ce0c81fc00dcbfc138e76b79d49694d1af7a6f8b4096"},{"contract":"deployment-preview","kind":"schema","path":"schemas/deployment-preview.schema.json","sha256":"67449ad10f78b972ff3fbda98948f3679dcdecf908ae468fe8ddf843158dfe81"},{"contract":"graph-change-preview","kind":"schema","path":"schemas/graph-change-preview.schema.json","sha256":"e45220e046d17c5b608da11e59bbeec6160f56f910b568bd15c8651465e585bf"},{"contract":"graph-create","kind":"schema","path":"schemas/graph-create.schema.json","sha256":"15cefbe92bdc171916d0118bfa33c665ac39a409661b222861a6f72a23331b41"},{"contract":"graph-page","kind":"schema","path":"schemas/graph-page.schema.json","sha256":"204018767f05e10a110f7caa177312ffea1266cfbd7eb841ed8a8ff90fbe1e09"},{"contract":"graph-repair-preview","kind":"schema","path":"schemas/graph-repair-preview.schema.json","sha256":"2c738cccf9c4e14ae5ad683ab590e9d9c682f630d497de796fd8c0f09bbf6ccf"},{"contract":"graph-repair-window","kind":"schema","path":"schemas/graph-repair-window.schema.json","sha256":"cf447ee4441b22db5e994e2ecf27469ad0c8d17a2cf31817142cee401827439b"},{"contract":"graph-resource","kind":"schema","path":"schemas/graph-resource.schema.json","sha256":"dc3ad0085768407720c825c95a5ff7f1ea01e85f6377a8f32445e926a9e88c61"},{"contract":"graph-validation","kind":"schema","path":"schemas/graph-validation.schema.json","sha256":"e835cd4ed7f698420a643305116f46644d54c8514548b849e34c23d398123d54"},{"contract":"log-page","kind":"schema","path":"schemas/log-page.schema.json","sha256":"bdc4059aecaeb7e18c93b3173726b256119306137a0988ddf1673bb50d260bc6"},{"contract":"mutation-result","kind":"schema","path":"schemas/mutation-result.schema.json","sha256":"078ca417acce9d4cd61c89607e5174156e986f432055c07509b9282fe2f537b7"},{"contract":"operation-catalog","kind":"schema","path":"schemas/operation-catalog.schema.json","sha256":"3ed2d3fd7defc677f7841b4e252a31dffd739922829d572ba9d5a7a635fae000"},{"contract":"pipeline-graph","kind":"schema","path":"schemas/pipeline-graph.schema.json","sha256":"3c2aa99c1ab5a93f2c590c0634fd1b39b8d1e84c56fbe8e42e0ff3051ee40377"},{"contract":"plugin-catalog","kind":"schema","path":"schemas/plugin-catalog.schema.json","sha256":"d934e93c333123284d96e2c9fb19080c952c6e353a721207d36228347923eb83"},{"contract":"project-list","kind":"schema","path":"schemas/project-list.schema.json","sha256":"a61e8c0dff565792079c1954584d7424a90fad35de85745682492a60e8067243"},{"contract":"run-explanation","kind":"schema","path":"schemas/run-explanation.schema.json","sha256":"e00ca3f82bfca14e77d61f00b14bc2c314820b680ff357b3c45e988eae6a8324"},{"contract":"run-page","kind":"schema","path":"schemas/run-page.schema.json","sha256":"b9caffd70b6f5fe6a14b6666e3934aa18d7934318e6db3c268c8d9598979d4fd"},{"contract":"run-request","kind":"schema","path":"schemas/run-request.schema.json","sha256":"0a8aa50cf4e82b573476e446e1c9eb6fba1482625abb300b719199ce68866eba"},{"contract":"run-status","kind":"schema","path":"schemas/run-status.schema.json","sha256":"e391062817d4f4153c341d17f99a8363dc7516c32eaf73785baf65f8231969bf"}],"json_schema_dialect":"https://json-schema.org/draft/2020-12/schema","reference_convention":"self-contained internal #/$defs references","schema":"io.dander.control.contracts-manifest/v1"} diff --git a/src/dander/control/contracts/v1/schemas/capabilities.schema.json b/src/dander/control/contracts/v1/schemas/capabilities.schema.json index 0555601f..17452738 100644 --- a/src/dander/control/contracts/v1/schemas/capabilities.schema.json +++ b/src/dander/control/contracts/v1/schemas/capabilities.schema.json @@ -1 +1 @@ -{"$defs":{"CompatibilityRange":{"additionalProperties":false,"properties":{"maximum_druff_contract":{"title":"Maximum Druff Contract","type":"string"},"minimum_druff_contract":{"title":"Minimum Druff Contract","type":"string"}},"required":["minimum_druff_contract","maximum_druff_contract"],"title":"CompatibilityRange","type":"object"},"ContractIdentity":{"additionalProperties":false,"properties":{"id":{"const":"io.dander.control.contracts/v1","title":"Id","type":"string"},"sha256":{"pattern":"^[0-9a-f]{64}$","title":"Sha256","type":"string"}},"required":["id","sha256"],"title":"ContractIdentity","type":"object"},"ControlLimits":{"additionalProperties":false,"properties":{"max_graph_bytes":{"exclusiveMinimum":0,"title":"Max Graph Bytes","type":"integer"},"max_log_records":{"exclusiveMinimum":0,"title":"Max Log Records","type":"integer"},"max_page_size":{"exclusiveMinimum":0,"title":"Max Page Size","type":"integer"}},"required":["max_graph_bytes","max_page_size","max_log_records"],"title":"ControlLimits","type":"object"}},"$id":"urn:dander:control:contracts:v1:capabilities","$schema":"https://json-schema.org/draft/2020-12/schema","additionalProperties":false,"properties":{"api_version":{"const":"v1","default":"v1","title":"Api Version","type":"string"},"compatibility":{"$ref":"#/$defs/CompatibilityRange"},"contract":{"$ref":"#/$defs/ContractIdentity"},"dander_version":{"title":"Dander Version","type":"string"},"limits":{"$ref":"#/$defs/ControlLimits"},"operations":{"items":{"enum":["graph.read","graph.edit","graph.delete","graph.validate","graph.change-preview","deployment.preview","run.start","run.read","run.explain","run.logs","run.cancel","run.replay"],"type":"string"},"title":"Operations","type":"array"}},"required":["dander_version","contract","compatibility","operations","limits"],"title":"CapabilitiesResponse","type":"object"} +{"$defs":{"CompatibilityRange":{"additionalProperties":false,"properties":{"maximum_druff_contract":{"title":"Maximum Druff Contract","type":"string"},"minimum_druff_contract":{"title":"Minimum Druff Contract","type":"string"}},"required":["minimum_druff_contract","maximum_druff_contract"],"title":"CompatibilityRange","type":"object"},"ContractIdentity":{"additionalProperties":false,"properties":{"id":{"const":"io.dander.control.contracts/v1","title":"Id","type":"string"},"sha256":{"pattern":"^[0-9a-f]{64}$","title":"Sha256","type":"string"}},"required":["id","sha256"],"title":"ContractIdentity","type":"object"},"ControlLimits":{"additionalProperties":false,"properties":{"max_graph_bytes":{"exclusiveMinimum":0,"title":"Max Graph Bytes","type":"integer"},"max_log_records":{"exclusiveMinimum":0,"title":"Max Log Records","type":"integer"},"max_page_size":{"exclusiveMinimum":0,"title":"Max Page Size","type":"integer"}},"required":["max_graph_bytes","max_page_size","max_log_records"],"title":"ControlLimits","type":"object"}},"$id":"urn:dander:control:contracts:v1:capabilities","$schema":"https://json-schema.org/draft/2020-12/schema","additionalProperties":false,"properties":{"api_version":{"const":"v1","default":"v1","title":"Api Version","type":"string"},"compatibility":{"$ref":"#/$defs/CompatibilityRange"},"contract":{"$ref":"#/$defs/ContractIdentity"},"dander_version":{"title":"Dander Version","type":"string"},"limits":{"$ref":"#/$defs/ControlLimits"},"operations":{"items":{"enum":["graph.read","graph.edit","graph.delete","graph.validate","graph.change-preview","graph.repair-preview","deployment.preview","run.start","run.read","run.explain","run.repair","run.logs","run.cancel","run.replay"],"type":"string"},"title":"Operations","type":"array"}},"required":["dander_version","contract","compatibility","operations","limits"],"title":"CapabilitiesResponse","type":"object"} diff --git a/src/dander/control/contracts/v1/schemas/graph-repair-preview.schema.json b/src/dander/control/contracts/v1/schemas/graph-repair-preview.schema.json new file mode 100644 index 00000000..d895da7a --- /dev/null +++ b/src/dander/control/contracts/v1/schemas/graph-repair-preview.schema.json @@ -0,0 +1 @@ +{"$defs":{"DestinationDocument":{"additionalProperties":false,"properties":{"business_key":{"items":{"type":"string"},"title":"Business Key","type":"array"},"dataset":{"minLength":1,"title":"Dataset","type":"string"},"project":{"anyOf":[{"type":"string"},{"type":"null"}],"default":null,"title":"Project"},"table":{"minLength":1,"title":"Table","type":"string"}},"required":["dataset","table"],"title":"DestinationDocument","type":"object"},"GraphRepairOutput":{"additionalProperties":false,"properties":{"business_key":{"items":{"type":"string"},"title":"Business Key","type":"array"},"destination":{"$ref":"#/$defs/DestinationDocument"},"name":{"title":"Name","type":"string"},"node_id":{"title":"Node Id","type":"string"},"partition_field":{"title":"Partition Field","type":"string"},"partition_type":{"enum":["DATE","TIMESTAMP"],"title":"Partition Type","type":"string"}},"required":["node_id","name","destination","partition_field","partition_type","business_key"],"title":"GraphRepairOutput","type":"object"},"GraphRepairWindow":{"additionalProperties":false,"description":"UTC calendar dates: start is inclusive and end is exclusive.","properties":{"end_date":{"format":"date","title":"End Date","type":"string"},"start_date":{"format":"date","title":"Start Date","type":"string"}},"required":["start_date","end_date"],"title":"GraphRepairWindow","type":"object"}},"$id":"urn:dander:control:contracts:v1:graph-repair-preview","$schema":"https://json-schema.org/draft/2020-12/schema","additionalProperties":false,"properties":{"environments":{"items":{"type":"string"},"title":"Environments","type":"array"},"estimated_cost_usd":{"default":null,"title":"Estimated Cost Usd","type":"null"},"estimated_rows":{"default":null,"title":"Estimated Rows","type":"null"},"graph_content_sha256":{"title":"Graph Content Sha256","type":"string"},"limitations":{"default":["Start date is included; end date is excluded. Timestamp fields use UTC calendar dates.","Source extraction and its normal saved progress are unchanged. This is not a historical snapshot.","Only selected output dates are published; computing them can scan data outside that range.","A record moving across the date boundary stops publication for that output. Choose a wider range or rebuild the full output.","Each output publishes atomically. Earlier successful outputs remain if a later output fails; repeating the same repair converges on the current raw data.","Execution requires a current repair-capable runtime with a BigQuery warehouse. Runtime validation and warehouse checks still apply."],"items":{"type":"string"},"title":"Limitations","type":"array"},"outputs":{"items":{"$ref":"#/$defs/GraphRepairOutput"},"title":"Outputs","type":"array"},"source":{"const":"retained_raw","default":"retained_raw","title":"Source","type":"string"},"summary":{"default":"Rebuild the selected output dates from the raw data currently stored in your warehouse.","title":"Summary","type":"string"},"window":{"$ref":"#/$defs/GraphRepairWindow"}},"required":["graph_content_sha256","window","outputs","environments"],"title":"GraphRepairPreviewResponse","type":"object"} diff --git a/src/dander/control/contracts/v1/schemas/graph-repair-window.schema.json b/src/dander/control/contracts/v1/schemas/graph-repair-window.schema.json new file mode 100644 index 00000000..60f61cca --- /dev/null +++ b/src/dander/control/contracts/v1/schemas/graph-repair-window.schema.json @@ -0,0 +1 @@ +{"$id":"urn:dander:control:contracts:v1:graph-repair-window","$schema":"https://json-schema.org/draft/2020-12/schema","additionalProperties":false,"description":"UTC calendar dates: start is inclusive and end is exclusive.","properties":{"end_date":{"format":"date","title":"End Date","type":"string"},"start_date":{"format":"date","title":"Start Date","type":"string"}},"required":["start_date","end_date"],"title":"GraphRepairWindow","type":"object"} diff --git a/src/dander/control/contracts/v1/schemas/run-page.schema.json b/src/dander/control/contracts/v1/schemas/run-page.schema.json index 96b4a876..2d9d7351 100644 --- a/src/dander/control/contracts/v1/schemas/run-page.schema.json +++ b/src/dander/control/contracts/v1/schemas/run-page.schema.json @@ -1 +1 @@ -{"$defs":{"RunPlacementDecision":{"additionalProperties":false,"description":"Fixed-size explanation of the execution-plan selection used by Control.","properties":{"decision_schema":{"const":"io.dander.control.placement-decision/v1","title":"Decision Schema","type":"string"},"eligible_plan_count":{"maximum":100,"minimum":1,"title":"Eligible Plan Count","type":"integer"},"estimated_cost_microusd":{"anyOf":[{"maximum":9223372036854775807,"minimum":0,"type":"integer"},{"type":"null"}],"default":null,"title":"Estimated Cost Microusd"},"max_cost_microusd":{"anyOf":[{"maximum":9223372036854775807,"minimum":0,"type":"integer"},{"type":"null"}],"default":null,"title":"Max Cost Microusd"},"mode":{"enum":["automatic","manual_override","configured_default","scheduled","replay"],"title":"Mode","type":"string"},"preferred_locality":{"anyOf":[{"type":"string"},{"type":"null"}],"default":null,"title":"Preferred Locality"},"selected_environment":{"title":"Selected Environment","type":"string"},"selected_locality":{"anyOf":[{"type":"string"},{"type":"null"}],"default":null,"title":"Selected Locality"}},"required":["decision_schema","mode","selected_environment","eligible_plan_count"],"title":"RunPlacementDecision","type":"object"},"RunSizeClassDecision":{"additionalProperties":false,"description":"Fixed-size explanation of the selected execution resource class.","properties":{"cpu_millis":{"minimum":1,"title":"Cpu Millis","type":"integer"},"decision_schema":{"const":"io.dander.control.size-class-decision/v2","title":"Decision Schema","type":"string"},"eligible_plan_count":{"maximum":100,"minimum":1,"title":"Eligible Plan Count","type":"integer"},"ephemeral_storage_mib":{"anyOf":[{"minimum":1,"type":"integer"},{"type":"null"}],"default":null,"title":"Ephemeral Storage Mib"},"estimate_observed_at":{"anyOf":[{"type":"string"},{"type":"null"}],"default":null,"title":"Estimate Observed At"},"estimate_source":{"anyOf":[{"type":"string"},{"type":"null"}],"default":null,"title":"Estimate Source"},"estimated_input_bytes":{"anyOf":[{"maximum":9223372036854775807,"minimum":0,"type":"integer"},{"type":"null"}],"default":null,"title":"Estimated Input Bytes"},"max_input_bytes":{"maximum":9223372036854775807,"minimum":0,"title":"Max Input Bytes","type":"integer"},"memory_mib":{"minimum":1,"title":"Memory Mib","type":"integer"},"mode":{"enum":["automatic_input","manual_override","configured_default","scheduled","replay"],"title":"Mode","type":"string"},"selected_size_class":{"title":"Selected Size Class","type":"string"}},"required":["decision_schema","mode","selected_size_class","max_input_bytes","cpu_millis","memory_mib","eligible_plan_count"],"title":"RunSizeClassDecision","type":"object"},"RunState":{"enum":["queued","running","succeeded","failed","canceling","canceled","retrying"],"title":"RunState","type":"string"},"RunStatusResponse":{"additionalProperties":false,"properties":{"affected":{"default":0,"minimum":0,"title":"Affected","type":"integer"},"assertions":{"default":0,"minimum":0,"title":"Assertions","type":"integer"},"assets":{"default":0,"minimum":0,"title":"Assets","type":"integer"},"can_cancel":{"default":false,"title":"Can Cancel","type":"boolean"},"can_replay":{"default":false,"title":"Can Replay","type":"boolean"},"endpoints":{"default":0,"minimum":0,"title":"Endpoints","type":"integer"},"extracted":{"default":0,"minimum":0,"title":"Extracted","type":"integer"},"failure_code":{"anyOf":[{"type":"string"},{"type":"null"}],"default":null,"title":"Failure Code"},"failure_summary":{"anyOf":[{"type":"string"},{"type":"null"}],"default":null,"title":"Failure Summary"},"finished_at":{"anyOf":[{"type":"string"},{"type":"null"}],"default":null,"title":"Finished At"},"logs_available":{"default":false,"title":"Logs Available","type":"boolean"},"models":{"default":0,"minimum":0,"title":"Models","type":"integer"},"placement":{"anyOf":[{"$ref":"#/$defs/RunPlacementDecision"},{"type":"null"}],"default":null},"result_schema":{"anyOf":[{"const":"io.dander.control.execution-result-summary/v1","type":"string"},{"type":"null"}],"default":null,"title":"Result Schema"},"run_id":{"title":"Run Id","type":"string"},"sizing":{"anyOf":[{"$ref":"#/$defs/RunSizeClassDecision"},{"type":"null"}],"default":null},"skipped":{"default":false,"title":"Skipped","type":"boolean"},"stage":{"anyOf":[{"type":"string"},{"type":"null"}],"default":null,"title":"Stage"},"started_at":{"anyOf":[{"type":"string"},{"type":"null"}],"default":null,"title":"Started At"},"state":{"$ref":"#/$defs/RunState"},"telemetry":{"anyOf":[{"$ref":"#/$defs/RunTelemetrySummary"},{"type":"null"}],"default":null}},"required":["run_id","state"],"title":"RunStatusResponse","type":"object"},"RunTelemetrySummary":{"additionalProperties":false,"description":"Fixed-size telemetry totals; provider operation details remain in logs.","properties":{"bytes_billed":{"maximum":9223372036854775807,"minimum":0,"title":"Bytes Billed","type":"integer"},"bytes_processed":{"maximum":9223372036854775807,"minimum":0,"title":"Bytes Processed","type":"integer"},"bytes_read":{"maximum":9223372036854775807,"minimum":0,"title":"Bytes Read","type":"integer"},"bytes_written":{"maximum":9223372036854775807,"minimum":0,"title":"Bytes Written","type":"integer"},"duration_ms":{"maximum":9223372036854775807,"minimum":0,"title":"Duration Ms","type":"integer"},"execution_duration_ms":{"maximum":9223372036854775807,"minimum":0,"title":"Execution Duration Ms","type":"integer"},"operation_count":{"maximum":9223372036854775807,"minimum":0,"title":"Operation Count","type":"integer"},"queue_duration_ms":{"maximum":9223372036854775807,"minimum":0,"title":"Queue Duration Ms","type":"integer"},"retry_count":{"maximum":9223372036854775807,"minimum":0,"title":"Retry Count","type":"integer"},"rows_affected":{"maximum":9223372036854775807,"minimum":0,"title":"Rows Affected","type":"integer"},"rows_read":{"maximum":9223372036854775807,"minimum":0,"title":"Rows Read","type":"integer"},"rows_written":{"maximum":9223372036854775807,"minimum":0,"title":"Rows Written","type":"integer"},"spill_bytes":{"maximum":9223372036854775807,"minimum":0,"title":"Spill Bytes","type":"integer"}},"required":["duration_ms","operation_count","retry_count","rows_read","rows_written","rows_affected","bytes_read","bytes_written","bytes_processed","bytes_billed","queue_duration_ms","execution_duration_ms","spill_bytes"],"title":"RunTelemetrySummary","type":"object"}},"$id":"urn:dander:control:contracts:v1:run-page","$schema":"https://json-schema.org/draft/2020-12/schema","additionalProperties":false,"description":"A bounded page of normalized, non-sensitive run summaries.","properties":{"items":{"items":{"$ref":"#/$defs/RunStatusResponse"},"maxItems":100,"title":"Items","type":"array"},"next_cursor":{"anyOf":[{"type":"string"},{"type":"null"}],"default":null,"title":"Next Cursor"}},"required":["items"],"title":"RunPageResponse","type":"object"} +{"$defs":{"GraphRepairWindow":{"additionalProperties":false,"description":"UTC calendar dates: start is inclusive and end is exclusive.","properties":{"end_date":{"format":"date","title":"End Date","type":"string"},"start_date":{"format":"date","title":"Start Date","type":"string"}},"required":["start_date","end_date"],"title":"GraphRepairWindow","type":"object"},"RunPlacementDecision":{"additionalProperties":false,"description":"Fixed-size explanation of the execution-plan selection used by Control.","properties":{"decision_schema":{"const":"io.dander.control.placement-decision/v1","title":"Decision Schema","type":"string"},"eligible_plan_count":{"maximum":100,"minimum":1,"title":"Eligible Plan Count","type":"integer"},"estimated_cost_microusd":{"anyOf":[{"maximum":9223372036854775807,"minimum":0,"type":"integer"},{"type":"null"}],"default":null,"title":"Estimated Cost Microusd"},"max_cost_microusd":{"anyOf":[{"maximum":9223372036854775807,"minimum":0,"type":"integer"},{"type":"null"}],"default":null,"title":"Max Cost Microusd"},"mode":{"enum":["automatic","manual_override","configured_default","scheduled","replay"],"title":"Mode","type":"string"},"preferred_locality":{"anyOf":[{"type":"string"},{"type":"null"}],"default":null,"title":"Preferred Locality"},"selected_environment":{"title":"Selected Environment","type":"string"},"selected_locality":{"anyOf":[{"type":"string"},{"type":"null"}],"default":null,"title":"Selected Locality"}},"required":["decision_schema","mode","selected_environment","eligible_plan_count"],"title":"RunPlacementDecision","type":"object"},"RunSizeClassDecision":{"additionalProperties":false,"description":"Fixed-size explanation of the selected execution resource class.","properties":{"cpu_millis":{"minimum":1,"title":"Cpu Millis","type":"integer"},"decision_schema":{"const":"io.dander.control.size-class-decision/v2","title":"Decision Schema","type":"string"},"eligible_plan_count":{"maximum":100,"minimum":1,"title":"Eligible Plan Count","type":"integer"},"ephemeral_storage_mib":{"anyOf":[{"minimum":1,"type":"integer"},{"type":"null"}],"default":null,"title":"Ephemeral Storage Mib"},"estimate_observed_at":{"anyOf":[{"type":"string"},{"type":"null"}],"default":null,"title":"Estimate Observed At"},"estimate_source":{"anyOf":[{"type":"string"},{"type":"null"}],"default":null,"title":"Estimate Source"},"estimated_input_bytes":{"anyOf":[{"maximum":9223372036854775807,"minimum":0,"type":"integer"},{"type":"null"}],"default":null,"title":"Estimated Input Bytes"},"max_input_bytes":{"maximum":9223372036854775807,"minimum":0,"title":"Max Input Bytes","type":"integer"},"memory_mib":{"minimum":1,"title":"Memory Mib","type":"integer"},"mode":{"enum":["automatic_input","manual_override","configured_default","scheduled","replay"],"title":"Mode","type":"string"},"selected_size_class":{"title":"Selected Size Class","type":"string"}},"required":["decision_schema","mode","selected_size_class","max_input_bytes","cpu_millis","memory_mib","eligible_plan_count"],"title":"RunSizeClassDecision","type":"object"},"RunState":{"enum":["queued","running","succeeded","failed","canceling","canceled","retrying"],"title":"RunState","type":"string"},"RunStatusResponse":{"additionalProperties":false,"properties":{"affected":{"default":0,"minimum":0,"title":"Affected","type":"integer"},"assertions":{"default":0,"minimum":0,"title":"Assertions","type":"integer"},"assets":{"default":0,"minimum":0,"title":"Assets","type":"integer"},"can_cancel":{"default":false,"title":"Can Cancel","type":"boolean"},"can_replay":{"default":false,"title":"Can Replay","type":"boolean"},"endpoints":{"default":0,"minimum":0,"title":"Endpoints","type":"integer"},"extracted":{"default":0,"minimum":0,"title":"Extracted","type":"integer"},"failure_code":{"anyOf":[{"type":"string"},{"type":"null"}],"default":null,"title":"Failure Code"},"failure_summary":{"anyOf":[{"type":"string"},{"type":"null"}],"default":null,"title":"Failure Summary"},"finished_at":{"anyOf":[{"type":"string"},{"type":"null"}],"default":null,"title":"Finished At"},"logs_available":{"default":false,"title":"Logs Available","type":"boolean"},"models":{"default":0,"minimum":0,"title":"Models","type":"integer"},"placement":{"anyOf":[{"$ref":"#/$defs/RunPlacementDecision"},{"type":"null"}],"default":null},"repair_window":{"anyOf":[{"$ref":"#/$defs/GraphRepairWindow"},{"type":"null"}],"default":null},"result_schema":{"anyOf":[{"const":"io.dander.control.execution-result-summary/v1","type":"string"},{"type":"null"}],"default":null,"title":"Result Schema"},"run_id":{"title":"Run Id","type":"string"},"sizing":{"anyOf":[{"$ref":"#/$defs/RunSizeClassDecision"},{"type":"null"}],"default":null},"skipped":{"default":false,"title":"Skipped","type":"boolean"},"stage":{"anyOf":[{"type":"string"},{"type":"null"}],"default":null,"title":"Stage"},"started_at":{"anyOf":[{"type":"string"},{"type":"null"}],"default":null,"title":"Started At"},"state":{"$ref":"#/$defs/RunState"},"telemetry":{"anyOf":[{"$ref":"#/$defs/RunTelemetrySummary"},{"type":"null"}],"default":null}},"required":["run_id","state"],"title":"RunStatusResponse","type":"object"},"RunTelemetrySummary":{"additionalProperties":false,"description":"Fixed-size telemetry totals; provider operation details remain in logs.","properties":{"bytes_billed":{"maximum":9223372036854775807,"minimum":0,"title":"Bytes Billed","type":"integer"},"bytes_processed":{"maximum":9223372036854775807,"minimum":0,"title":"Bytes Processed","type":"integer"},"bytes_read":{"maximum":9223372036854775807,"minimum":0,"title":"Bytes Read","type":"integer"},"bytes_written":{"maximum":9223372036854775807,"minimum":0,"title":"Bytes Written","type":"integer"},"duration_ms":{"maximum":9223372036854775807,"minimum":0,"title":"Duration Ms","type":"integer"},"execution_duration_ms":{"maximum":9223372036854775807,"minimum":0,"title":"Execution Duration Ms","type":"integer"},"operation_count":{"maximum":9223372036854775807,"minimum":0,"title":"Operation Count","type":"integer"},"queue_duration_ms":{"maximum":9223372036854775807,"minimum":0,"title":"Queue Duration Ms","type":"integer"},"retry_count":{"maximum":9223372036854775807,"minimum":0,"title":"Retry Count","type":"integer"},"rows_affected":{"maximum":9223372036854775807,"minimum":0,"title":"Rows Affected","type":"integer"},"rows_read":{"maximum":9223372036854775807,"minimum":0,"title":"Rows Read","type":"integer"},"rows_written":{"maximum":9223372036854775807,"minimum":0,"title":"Rows Written","type":"integer"},"spill_bytes":{"maximum":9223372036854775807,"minimum":0,"title":"Spill Bytes","type":"integer"}},"required":["duration_ms","operation_count","retry_count","rows_read","rows_written","rows_affected","bytes_read","bytes_written","bytes_processed","bytes_billed","queue_duration_ms","execution_duration_ms","spill_bytes"],"title":"RunTelemetrySummary","type":"object"}},"$id":"urn:dander:control:contracts:v1:run-page","$schema":"https://json-schema.org/draft/2020-12/schema","additionalProperties":false,"description":"A bounded page of normalized, non-sensitive run summaries.","properties":{"items":{"items":{"$ref":"#/$defs/RunStatusResponse"},"maxItems":100,"title":"Items","type":"array"},"next_cursor":{"anyOf":[{"type":"string"},{"type":"null"}],"default":null,"title":"Next Cursor"}},"required":["items"],"title":"RunPageResponse","type":"object"} diff --git a/src/dander/control/contracts/v1/schemas/run-status.schema.json b/src/dander/control/contracts/v1/schemas/run-status.schema.json index a78b60c7..f5d536ca 100644 --- a/src/dander/control/contracts/v1/schemas/run-status.schema.json +++ b/src/dander/control/contracts/v1/schemas/run-status.schema.json @@ -1 +1 @@ -{"$defs":{"RunPlacementDecision":{"additionalProperties":false,"description":"Fixed-size explanation of the execution-plan selection used by Control.","properties":{"decision_schema":{"const":"io.dander.control.placement-decision/v1","title":"Decision Schema","type":"string"},"eligible_plan_count":{"maximum":100,"minimum":1,"title":"Eligible Plan Count","type":"integer"},"estimated_cost_microusd":{"anyOf":[{"maximum":9223372036854775807,"minimum":0,"type":"integer"},{"type":"null"}],"default":null,"title":"Estimated Cost Microusd"},"max_cost_microusd":{"anyOf":[{"maximum":9223372036854775807,"minimum":0,"type":"integer"},{"type":"null"}],"default":null,"title":"Max Cost Microusd"},"mode":{"enum":["automatic","manual_override","configured_default","scheduled","replay"],"title":"Mode","type":"string"},"preferred_locality":{"anyOf":[{"type":"string"},{"type":"null"}],"default":null,"title":"Preferred Locality"},"selected_environment":{"title":"Selected Environment","type":"string"},"selected_locality":{"anyOf":[{"type":"string"},{"type":"null"}],"default":null,"title":"Selected Locality"}},"required":["decision_schema","mode","selected_environment","eligible_plan_count"],"title":"RunPlacementDecision","type":"object"},"RunSizeClassDecision":{"additionalProperties":false,"description":"Fixed-size explanation of the selected execution resource class.","properties":{"cpu_millis":{"minimum":1,"title":"Cpu Millis","type":"integer"},"decision_schema":{"const":"io.dander.control.size-class-decision/v2","title":"Decision Schema","type":"string"},"eligible_plan_count":{"maximum":100,"minimum":1,"title":"Eligible Plan Count","type":"integer"},"ephemeral_storage_mib":{"anyOf":[{"minimum":1,"type":"integer"},{"type":"null"}],"default":null,"title":"Ephemeral Storage Mib"},"estimate_observed_at":{"anyOf":[{"type":"string"},{"type":"null"}],"default":null,"title":"Estimate Observed At"},"estimate_source":{"anyOf":[{"type":"string"},{"type":"null"}],"default":null,"title":"Estimate Source"},"estimated_input_bytes":{"anyOf":[{"maximum":9223372036854775807,"minimum":0,"type":"integer"},{"type":"null"}],"default":null,"title":"Estimated Input Bytes"},"max_input_bytes":{"maximum":9223372036854775807,"minimum":0,"title":"Max Input Bytes","type":"integer"},"memory_mib":{"minimum":1,"title":"Memory Mib","type":"integer"},"mode":{"enum":["automatic_input","manual_override","configured_default","scheduled","replay"],"title":"Mode","type":"string"},"selected_size_class":{"title":"Selected Size Class","type":"string"}},"required":["decision_schema","mode","selected_size_class","max_input_bytes","cpu_millis","memory_mib","eligible_plan_count"],"title":"RunSizeClassDecision","type":"object"},"RunState":{"enum":["queued","running","succeeded","failed","canceling","canceled","retrying"],"title":"RunState","type":"string"},"RunTelemetrySummary":{"additionalProperties":false,"description":"Fixed-size telemetry totals; provider operation details remain in logs.","properties":{"bytes_billed":{"maximum":9223372036854775807,"minimum":0,"title":"Bytes Billed","type":"integer"},"bytes_processed":{"maximum":9223372036854775807,"minimum":0,"title":"Bytes Processed","type":"integer"},"bytes_read":{"maximum":9223372036854775807,"minimum":0,"title":"Bytes Read","type":"integer"},"bytes_written":{"maximum":9223372036854775807,"minimum":0,"title":"Bytes Written","type":"integer"},"duration_ms":{"maximum":9223372036854775807,"minimum":0,"title":"Duration Ms","type":"integer"},"execution_duration_ms":{"maximum":9223372036854775807,"minimum":0,"title":"Execution Duration Ms","type":"integer"},"operation_count":{"maximum":9223372036854775807,"minimum":0,"title":"Operation Count","type":"integer"},"queue_duration_ms":{"maximum":9223372036854775807,"minimum":0,"title":"Queue Duration Ms","type":"integer"},"retry_count":{"maximum":9223372036854775807,"minimum":0,"title":"Retry Count","type":"integer"},"rows_affected":{"maximum":9223372036854775807,"minimum":0,"title":"Rows Affected","type":"integer"},"rows_read":{"maximum":9223372036854775807,"minimum":0,"title":"Rows Read","type":"integer"},"rows_written":{"maximum":9223372036854775807,"minimum":0,"title":"Rows Written","type":"integer"},"spill_bytes":{"maximum":9223372036854775807,"minimum":0,"title":"Spill Bytes","type":"integer"}},"required":["duration_ms","operation_count","retry_count","rows_read","rows_written","rows_affected","bytes_read","bytes_written","bytes_processed","bytes_billed","queue_duration_ms","execution_duration_ms","spill_bytes"],"title":"RunTelemetrySummary","type":"object"}},"$id":"urn:dander:control:contracts:v1:run-status","$schema":"https://json-schema.org/draft/2020-12/schema","additionalProperties":false,"properties":{"affected":{"default":0,"minimum":0,"title":"Affected","type":"integer"},"assertions":{"default":0,"minimum":0,"title":"Assertions","type":"integer"},"assets":{"default":0,"minimum":0,"title":"Assets","type":"integer"},"can_cancel":{"default":false,"title":"Can Cancel","type":"boolean"},"can_replay":{"default":false,"title":"Can Replay","type":"boolean"},"endpoints":{"default":0,"minimum":0,"title":"Endpoints","type":"integer"},"extracted":{"default":0,"minimum":0,"title":"Extracted","type":"integer"},"failure_code":{"anyOf":[{"type":"string"},{"type":"null"}],"default":null,"title":"Failure Code"},"failure_summary":{"anyOf":[{"type":"string"},{"type":"null"}],"default":null,"title":"Failure Summary"},"finished_at":{"anyOf":[{"type":"string"},{"type":"null"}],"default":null,"title":"Finished At"},"logs_available":{"default":false,"title":"Logs Available","type":"boolean"},"models":{"default":0,"minimum":0,"title":"Models","type":"integer"},"placement":{"anyOf":[{"$ref":"#/$defs/RunPlacementDecision"},{"type":"null"}],"default":null},"result_schema":{"anyOf":[{"const":"io.dander.control.execution-result-summary/v1","type":"string"},{"type":"null"}],"default":null,"title":"Result Schema"},"run_id":{"title":"Run Id","type":"string"},"sizing":{"anyOf":[{"$ref":"#/$defs/RunSizeClassDecision"},{"type":"null"}],"default":null},"skipped":{"default":false,"title":"Skipped","type":"boolean"},"stage":{"anyOf":[{"type":"string"},{"type":"null"}],"default":null,"title":"Stage"},"started_at":{"anyOf":[{"type":"string"},{"type":"null"}],"default":null,"title":"Started At"},"state":{"$ref":"#/$defs/RunState"},"telemetry":{"anyOf":[{"$ref":"#/$defs/RunTelemetrySummary"},{"type":"null"}],"default":null}},"required":["run_id","state"],"title":"RunStatusResponse","type":"object"} +{"$defs":{"GraphRepairWindow":{"additionalProperties":false,"description":"UTC calendar dates: start is inclusive and end is exclusive.","properties":{"end_date":{"format":"date","title":"End Date","type":"string"},"start_date":{"format":"date","title":"Start Date","type":"string"}},"required":["start_date","end_date"],"title":"GraphRepairWindow","type":"object"},"RunPlacementDecision":{"additionalProperties":false,"description":"Fixed-size explanation of the execution-plan selection used by Control.","properties":{"decision_schema":{"const":"io.dander.control.placement-decision/v1","title":"Decision Schema","type":"string"},"eligible_plan_count":{"maximum":100,"minimum":1,"title":"Eligible Plan Count","type":"integer"},"estimated_cost_microusd":{"anyOf":[{"maximum":9223372036854775807,"minimum":0,"type":"integer"},{"type":"null"}],"default":null,"title":"Estimated Cost Microusd"},"max_cost_microusd":{"anyOf":[{"maximum":9223372036854775807,"minimum":0,"type":"integer"},{"type":"null"}],"default":null,"title":"Max Cost Microusd"},"mode":{"enum":["automatic","manual_override","configured_default","scheduled","replay"],"title":"Mode","type":"string"},"preferred_locality":{"anyOf":[{"type":"string"},{"type":"null"}],"default":null,"title":"Preferred Locality"},"selected_environment":{"title":"Selected Environment","type":"string"},"selected_locality":{"anyOf":[{"type":"string"},{"type":"null"}],"default":null,"title":"Selected Locality"}},"required":["decision_schema","mode","selected_environment","eligible_plan_count"],"title":"RunPlacementDecision","type":"object"},"RunSizeClassDecision":{"additionalProperties":false,"description":"Fixed-size explanation of the selected execution resource class.","properties":{"cpu_millis":{"minimum":1,"title":"Cpu Millis","type":"integer"},"decision_schema":{"const":"io.dander.control.size-class-decision/v2","title":"Decision Schema","type":"string"},"eligible_plan_count":{"maximum":100,"minimum":1,"title":"Eligible Plan Count","type":"integer"},"ephemeral_storage_mib":{"anyOf":[{"minimum":1,"type":"integer"},{"type":"null"}],"default":null,"title":"Ephemeral Storage Mib"},"estimate_observed_at":{"anyOf":[{"type":"string"},{"type":"null"}],"default":null,"title":"Estimate Observed At"},"estimate_source":{"anyOf":[{"type":"string"},{"type":"null"}],"default":null,"title":"Estimate Source"},"estimated_input_bytes":{"anyOf":[{"maximum":9223372036854775807,"minimum":0,"type":"integer"},{"type":"null"}],"default":null,"title":"Estimated Input Bytes"},"max_input_bytes":{"maximum":9223372036854775807,"minimum":0,"title":"Max Input Bytes","type":"integer"},"memory_mib":{"minimum":1,"title":"Memory Mib","type":"integer"},"mode":{"enum":["automatic_input","manual_override","configured_default","scheduled","replay"],"title":"Mode","type":"string"},"selected_size_class":{"title":"Selected Size Class","type":"string"}},"required":["decision_schema","mode","selected_size_class","max_input_bytes","cpu_millis","memory_mib","eligible_plan_count"],"title":"RunSizeClassDecision","type":"object"},"RunState":{"enum":["queued","running","succeeded","failed","canceling","canceled","retrying"],"title":"RunState","type":"string"},"RunTelemetrySummary":{"additionalProperties":false,"description":"Fixed-size telemetry totals; provider operation details remain in logs.","properties":{"bytes_billed":{"maximum":9223372036854775807,"minimum":0,"title":"Bytes Billed","type":"integer"},"bytes_processed":{"maximum":9223372036854775807,"minimum":0,"title":"Bytes Processed","type":"integer"},"bytes_read":{"maximum":9223372036854775807,"minimum":0,"title":"Bytes Read","type":"integer"},"bytes_written":{"maximum":9223372036854775807,"minimum":0,"title":"Bytes Written","type":"integer"},"duration_ms":{"maximum":9223372036854775807,"minimum":0,"title":"Duration Ms","type":"integer"},"execution_duration_ms":{"maximum":9223372036854775807,"minimum":0,"title":"Execution Duration Ms","type":"integer"},"operation_count":{"maximum":9223372036854775807,"minimum":0,"title":"Operation Count","type":"integer"},"queue_duration_ms":{"maximum":9223372036854775807,"minimum":0,"title":"Queue Duration Ms","type":"integer"},"retry_count":{"maximum":9223372036854775807,"minimum":0,"title":"Retry Count","type":"integer"},"rows_affected":{"maximum":9223372036854775807,"minimum":0,"title":"Rows Affected","type":"integer"},"rows_read":{"maximum":9223372036854775807,"minimum":0,"title":"Rows Read","type":"integer"},"rows_written":{"maximum":9223372036854775807,"minimum":0,"title":"Rows Written","type":"integer"},"spill_bytes":{"maximum":9223372036854775807,"minimum":0,"title":"Spill Bytes","type":"integer"}},"required":["duration_ms","operation_count","retry_count","rows_read","rows_written","rows_affected","bytes_read","bytes_written","bytes_processed","bytes_billed","queue_duration_ms","execution_duration_ms","spill_bytes"],"title":"RunTelemetrySummary","type":"object"}},"$id":"urn:dander:control:contracts:v1:run-status","$schema":"https://json-schema.org/draft/2020-12/schema","additionalProperties":false,"properties":{"affected":{"default":0,"minimum":0,"title":"Affected","type":"integer"},"assertions":{"default":0,"minimum":0,"title":"Assertions","type":"integer"},"assets":{"default":0,"minimum":0,"title":"Assets","type":"integer"},"can_cancel":{"default":false,"title":"Can Cancel","type":"boolean"},"can_replay":{"default":false,"title":"Can Replay","type":"boolean"},"endpoints":{"default":0,"minimum":0,"title":"Endpoints","type":"integer"},"extracted":{"default":0,"minimum":0,"title":"Extracted","type":"integer"},"failure_code":{"anyOf":[{"type":"string"},{"type":"null"}],"default":null,"title":"Failure Code"},"failure_summary":{"anyOf":[{"type":"string"},{"type":"null"}],"default":null,"title":"Failure Summary"},"finished_at":{"anyOf":[{"type":"string"},{"type":"null"}],"default":null,"title":"Finished At"},"logs_available":{"default":false,"title":"Logs Available","type":"boolean"},"models":{"default":0,"minimum":0,"title":"Models","type":"integer"},"placement":{"anyOf":[{"$ref":"#/$defs/RunPlacementDecision"},{"type":"null"}],"default":null},"repair_window":{"anyOf":[{"$ref":"#/$defs/GraphRepairWindow"},{"type":"null"}],"default":null},"result_schema":{"anyOf":[{"const":"io.dander.control.execution-result-summary/v1","type":"string"},{"type":"null"}],"default":null,"title":"Result Schema"},"run_id":{"title":"Run Id","type":"string"},"sizing":{"anyOf":[{"$ref":"#/$defs/RunSizeClassDecision"},{"type":"null"}],"default":null},"skipped":{"default":false,"title":"Skipped","type":"boolean"},"stage":{"anyOf":[{"type":"string"},{"type":"null"}],"default":null,"title":"Stage"},"started_at":{"anyOf":[{"type":"string"},{"type":"null"}],"default":null,"title":"Started At"},"state":{"$ref":"#/$defs/RunState"},"telemetry":{"anyOf":[{"$ref":"#/$defs/RunTelemetrySummary"},{"type":"null"}],"default":null}},"required":["run_id","state"],"title":"RunStatusResponse","type":"object"} diff --git a/src/dander/control/dataproc_serverless_execution_backend.py b/src/dander/control/dataproc_serverless_execution_backend.py index 55c3d37a..69179355 100644 --- a/src/dander/control/dataproc_serverless_execution_backend.py +++ b/src/dander/control/dataproc_serverless_execution_backend.py @@ -128,7 +128,10 @@ def submit_or_adopt( trigger: RunTrigger, ) -> BackendHandle: """Create or adopt one deterministic batch for the logical attempt.""" - del trigger + if trigger.repair_window is not None: + raise ExecutionBackendError( + "Date repair is not supported by the Managed Spark backend." + ) binding = self._binding_for(plan) batch_id = _batch_id(run_id, attempt_id) batch_resource = binding.batch_resource(batch_id) diff --git a/src/dander/control/fargate_execution_backend.py b/src/dander/control/fargate_execution_backend.py index 8da0dac7..e94d6343 100644 --- a/src/dander/control/fargate_execution_backend.py +++ b/src/dander/control/fargate_execution_backend.py @@ -144,6 +144,8 @@ def submit_or_adopt( trigger: RunTrigger, ) -> BackendHandle: """Start or adopt the one deterministic Standard Workflow execution for an attempt.""" + if trigger.repair_window is not None: + raise ExecutionBackendError("Date repair is not supported by the Fargate backend.") binding = self._binding_for(plan) name = _execution_name(run_id, attempt_id) execution_arn = _execution_arn(binding, name) diff --git a/src/dander/control/http.py b/src/dander/control/http.py index 51bc1b4d..8f2d334e 100644 --- a/src/dander/control/http.py +++ b/src/dander/control/http.py @@ -55,6 +55,7 @@ GraphCreateRequest, PipelineGraphDocument, ) +from dander.pipeline.repair import GraphRepairError, GraphRepairWindow if TYPE_CHECKING: from collections.abc import AsyncIterator, Callable @@ -579,6 +580,58 @@ def preview_graph(project: str, graph: str, if_match: _IF_MATCH_HEADER) -> objec project, graph, expected_revision=decode_revision_etag(if_match) ) + @app.post( + "/v1/projects/{project}/graphs/{graph}/repair-preview", + dependencies=auth_dependencies(ControlCapability.VALIDATE_PREVIEW), + ) + async def preview_repair( + project: str, graph: str, request: Request, if_match: _IF_MATCH_HEADER + ) -> object: + window = await _read_repair_window(request) + try: + return await run_in_threadpool( + application.preview_repair, + project, + graph, + window, + expected_revision=decode_revision_etag(if_match), + ) + except GraphRepairError as error: + raise GraphBodyError( + "repair_invalid", str(error), HTTPStatus.UNPROCESSABLE_ENTITY + ) from error + + @app.post( + "/v1/projects/{project}/graphs/{graph}/repairs", + status_code=HTTPStatus.ACCEPTED, + dependencies=auth_dependencies(ControlCapability.RUN), + ) + async def start_repair( + project: str, + graph: str, + request: Request, + if_match: _IF_MATCH_HEADER, + idempotency_key: _IDEMPOTENCY_HEADER, + environment: Annotated[ + str | None, Query(pattern=r"^[a-z0-9][a-z0-9_-]{0,62}$", max_length=63) + ] = None, + ) -> object: + window = await _read_repair_window(request) + try: + return await run_in_threadpool( + application.start_run, + project, + graph, + expected_revision=decode_revision_etag(if_match), + idempotency_key=idempotency_key, + environment=environment, + repair_window=window, + ) + except GraphRepairError as error: + raise GraphBodyError( + "repair_invalid", str(error), HTTPStatus.UNPROCESSABLE_ENTITY + ) from error + @app.post( "/v1/projects/{project}/graphs/{graph}/runs", status_code=HTTPStatus.ACCEPTED, @@ -665,6 +718,18 @@ async def replay_run( return app +async def _read_repair_window(request: Request) -> GraphRepairWindow: + payload = await _read_bounded_json(request, 1024, subject="repair") + try: + return GraphRepairWindow.model_validate(payload) + except ValidationError as error: + raise GraphBodyError( + "repair_invalid", + "Use YYYY-MM-DD start_date and end_date; the end date is exclusive and must be later.", + HTTPStatus.UNPROCESSABLE_ENTITY, + ) from error + + def _resource_response(record: GraphRecord, status: HTTPStatus = HTTPStatus.OK) -> JSONResponse: projected = graph_resource_response(record) response = JSONResponse(projected.model_dump(mode="json"), status_code=status) @@ -695,9 +760,14 @@ def _error_response( def _operation_capability(operation: str) -> ControlCapability: if operation in {"graph.edit"}: return ControlCapability.EDIT - if operation in {"graph.validate", "graph.change-preview", "deployment.preview"}: + if operation in { + "graph.validate", + "graph.change-preview", + "graph.repair-preview", + "deployment.preview", + }: return ControlCapability.VALIDATE_PREVIEW - if operation in {"run.start", "run.cancel", "run.replay"}: + if operation in {"run.start", "run.repair", "run.cancel", "run.replay"}: return ControlCapability.RUN if operation == "graph.delete": return ControlCapability.ADMIN diff --git a/src/dander/control/models.py b/src/dander/control/models.py index 4083b395..4f361c43 100644 --- a/src/dander/control/models.py +++ b/src/dander/control/models.py @@ -18,6 +18,7 @@ from dander.pipeline.graph_ops import validate_field_wiring from dander.pipeline.node_config import ExecutableJoinType, PartitioningType from dander.pipeline.operations import ComparisonOperator, MatchLogic +from dander.pipeline.repair import GraphRepairWindow # noqa: TC001 - Pydantic resolves this model from dander.pipeline.request_spec import HttpMethod from dander.warehouse import ProviderExtension # noqa: TC001 - Pydantic resolves this enum from dander.writer import SchemaEvolution, WriteMode @@ -702,6 +703,7 @@ class RunStatusResponse(ControlModel): can_cancel: bool = False can_replay: bool = False logs_available: bool = False + repair_window: GraphRepairWindow | None = None class RunPageResponse(ControlModel): @@ -798,10 +800,12 @@ class CapabilitiesResponse(ControlModel): "graph.delete", "graph.validate", "graph.change-preview", + "graph.repair-preview", "deployment.preview", "run.start", "run.read", "run.explain", + "run.repair", "run.logs", "run.cancel", "run.replay", diff --git a/src/dander/control/orchestration.py b/src/dander/control/orchestration.py index d92a684b..9b4ef111 100644 --- a/src/dander/control/orchestration.py +++ b/src/dander/control/orchestration.py @@ -16,6 +16,7 @@ from typing import TYPE_CHECKING, Protocol, runtime_checkable from dander.physical_plan import PhysicalExecutionMode, PhysicalPlan, serialize_physical_plan +from dander.pipeline.repair import GraphRepairWindow if TYPE_CHECKING: from datetime import datetime @@ -563,6 +564,7 @@ class RunTrigger: trigger_id: str scheduled_occurrence: datetime | None = None replay_of_run_id: str | None = None + repair_window: GraphRepairWindow | None = None def __post_init__(self) -> None: _require_portable_id(self.trigger_id, label="trigger") @@ -578,6 +580,11 @@ def __post_init__(self) -> None: ) if self.replay_of_run_id is not None: _require_opaque_id(self.replay_of_run_id, label="replayed run") + if self.repair_window is not None: + if not isinstance(self.repair_window, GraphRepairWindow): + raise OrchestrationContractError("repair window is invalid") + if self.kind not in {TriggerKind.API, TriggerKind.MANUAL}: + raise OrchestrationContractError("date repair requires an explicit run trigger") @dataclass(frozen=True, slots=True) @@ -623,6 +630,14 @@ def __post_init__(self) -> None: def fingerprint(self) -> str: """Identity of logical input excluding request time and the idempotency key itself.""" occurrence = self.trigger.scheduled_occurrence + trigger_payload: dict[str, object] = { + "kind": self.trigger.kind.value, + "trigger_id": self.trigger.trigger_id, + "scheduled_occurrence": occurrence.isoformat() if occurrence else None, + "replay_of_run_id": self.trigger.replay_of_run_id, + } + if self.trigger.repair_window is not None: + trigger_payload["repair_window"] = self.trigger.repair_window.model_dump(mode="json") payload = { "environment": self.environment, "project": self.project, @@ -631,12 +646,7 @@ def fingerprint(self) -> str: "graph_content_sha256": self.graph.content_sha256, "plan_id": self.plan_id, "plan_revision": self.plan_revision, - "trigger": { - "kind": self.trigger.kind.value, - "trigger_id": self.trigger.trigger_id, - "scheduled_occurrence": occurrence.isoformat() if occurrence else None, - "replay_of_run_id": self.trigger.replay_of_run_id, - }, + "trigger": trigger_payload, "requested_deadline_seconds": self.requested_deadline_seconds, } encoded = json.dumps(payload, sort_keys=True, separators=(",", ":")).encode() @@ -1155,6 +1165,10 @@ def attempt_identity(run_id: str, attempt_number: int) -> str: def validate_submission_plan(submission: RunSubmission, plan: ExecutionPlan) -> None: """Reject plan selection drift before a provider request can occur.""" + if submission.trigger.repair_window is not None and ( + plan.backend_id != "cloud_run" or plan.execution_template.schedule.task_count != 1 + ): + raise OrchestrationContractError("date repair requires a single-task Cloud Run plan") expected = ( submission.environment, submission.project, diff --git a/src/dander/control/orchestration_serialization.py b/src/dander/control/orchestration_serialization.py index 602083c9..2be7c85d 100644 --- a/src/dander/control/orchestration_serialization.py +++ b/src/dander/control/orchestration_serialization.py @@ -43,6 +43,7 @@ SecretReference, ) from dander.physical_plan import deserialize_physical_plan +from dander.pipeline.repair import GraphRepairWindow RUN_RECORD_SCHEMA_V1 = "io.dander.control.run-record/v1" RUN_RECORD_SCHEMA_V2 = "io.dander.control.run-record/v2" @@ -746,7 +747,7 @@ def _deserialize_execution_result_summary_value(value: object) -> ExecutionResul def _trigger_payload(trigger: RunTrigger) -> dict[str, object]: - return { + payload: dict[str, object] = { "kind": trigger.kind.value, "trigger_id": trigger.trigger_id, "scheduled_occurrence": ( @@ -754,6 +755,9 @@ def _trigger_payload(trigger: RunTrigger) -> dict[str, object]: ), "replay_of_run_id": trigger.replay_of_run_id, } + if trigger.repair_window is not None: + payload["repair_window"] = trigger.repair_window.model_dump(mode="json") + return payload def _trigger(value: object) -> RunTrigger: @@ -765,6 +769,11 @@ def _trigger(value: object) -> RunTrigger: values["scheduled_occurrence"], "scheduled_occurrence" ), replay_of_run_id=_optional_string(values["replay_of_run_id"], "replay_of_run_id"), + repair_window=( + GraphRepairWindow.model_validate(values["repair_window"]) + if "repair_window" in values + else None + ), ) diff --git a/src/dander/control/repair.py b/src/dander/control/repair.py new file mode 100644 index 00000000..7fa4b1f4 --- /dev/null +++ b/src/dander/control/repair.py @@ -0,0 +1,80 @@ +"""Read-only presentation of an output repair against a saved graph.""" + +from __future__ import annotations + +from typing import Literal + +from dander.control.models import ( + ControlModel, + DestinationDocument, + PipelineGraphDocument, + TargetNodeDocument, +) +from dander.pipeline.repair import GraphRepairWindow, plan_graph_repair + + +class GraphRepairOutput(ControlModel): + node_id: str + name: str + destination: DestinationDocument + partition_field: str + partition_type: Literal["DATE", "TIMESTAMP"] + business_key: tuple[str, ...] + + +class GraphRepairPreviewResponse(ControlModel): + graph_content_sha256: str + window: GraphRepairWindow + outputs: tuple[GraphRepairOutput, ...] + environments: tuple[str, ...] + source: Literal["retained_raw"] = "retained_raw" + summary: str = ( + "Rebuild the selected output dates from the raw data currently stored in your warehouse." + ) + limitations: tuple[str, ...] = ( + "Start date is included; end date is excluded. Timestamp fields use UTC calendar dates.", + "Source extraction and its normal saved progress are unchanged. " + "This is not a historical snapshot.", + "Only selected output dates are published; computing them can scan data " + "outside that range.", + "A record moving across the date boundary stops publication for that output. " + "Choose a wider range or rebuild the full output.", + "Each output publishes atomically. Earlier successful outputs remain if a later output " + "fails; repeating the same repair converges on the current raw data.", + "Execution requires a current repair-capable runtime with a BigQuery warehouse. " + "Runtime validation and warehouse checks still apply.", + ) + estimated_rows: None = None + estimated_cost_usd: None = None + + +def preview_graph_repair( + document: PipelineGraphDocument, + *, + content_sha256: str, + window: GraphRepairWindow, + environments: tuple[str, ...], +) -> GraphRepairPreviewResponse: + """Describe declared eligibility without reading rows or starting an execution.""" + targets = plan_graph_repair(document.to_domain(), window) + nodes = {node.id: node for node in document.nodes} + outputs: list[GraphRepairOutput] = [] + for target in targets: + node = nodes[target.node_id] + assert isinstance(node, TargetNodeDocument) and node.config.writer is not None + outputs.append( + GraphRepairOutput( + node_id=node.id, + name=node.name, + destination=node.config.writer.destination, + partition_field=target.partition_field, + partition_type=target.partition_type, + business_key=target.business_key, + ) + ) + return GraphRepairPreviewResponse( + graph_content_sha256=content_sha256, + window=window, + outputs=tuple(outputs), + environments=environments, + ) diff --git a/src/dander/control/run_explanation.py b/src/dander/control/run_explanation.py index 190c29cf..082b17b9 100644 --- a/src/dander/control/run_explanation.py +++ b/src/dander/control/run_explanation.py @@ -68,14 +68,35 @@ def explain_run(status: RunStatusResponse) -> RunExplanationResponse: caveats: list[str] = [] summary = _summary(status, results_available=results_available) + if status.repair_window is not None: + window = status.repair_window + details.append( + f"Output repair from retained raw data: {window.start_date.isoformat()} included " + f"through {window.end_date.isoformat()} excluded (UTC)." + ) + caveats.append( + "This repair does not extract source records or advance normal source progress." + ) + if status.state in _ACTIVE_STATES and status.stage in _STAGE_DETAILS: details.append(_STAGE_DETAILS[status.stage]) if results_available: + output_detail = ( + f"Recorded output: {status.affected:,} affected rows and {status.models:,} models." + ) + if status.repair_window is not None: + output_detail = "Output repair measurements have not been collected." + if status.telemetry is not None: + output_detail = ( + f"Recorded repair: {status.telemetry.rows_written:,} rows written and " + f"{status.telemetry.rows_affected:,} inserted/deleted row operations " + f"across {status.models:,} outputs." + ) details.extend( ( f"Recorded extraction: {status.extracted:,} rows " f"from {status.endpoints:,} endpoints.", - f"Recorded output: {status.affected:,} affected rows and {status.models:,} models.", + output_detail, f"Recorded validation and catalog: {status.assertions:,} assertions evaluated " f"and {status.assets:,} assets.", ) @@ -114,7 +135,10 @@ def explain_run(status: RunStatusResponse) -> RunExplanationResponse: ) if status.can_replay: caveats.append( - "Replay starts another execution; it does not undo prior writes or repair only " + "Replay repeats this repair window against the raw data available at that time; " + "it does not restore a historical snapshot." + if status.repair_window is not None + else "Replay starts another execution; it does not undo prior writes or repair only " "a selected date range." ) diff --git a/src/dander/control/run_lifecycle.py b/src/dander/control/run_lifecycle.py index 1ecf3958..363e6b9d 100644 --- a/src/dander/control/run_lifecycle.py +++ b/src/dander/control/run_lifecycle.py @@ -71,6 +71,7 @@ from dander.control.graph_store import GraphRecord, GraphStore from dander.control.input_size_estimator import InputSizeEstimator from dander.control.orchestration import ExecutionBackend, RunStore + from dander.pipeline.repair import GraphRepairWindow _IDEMPOTENCY_KEY = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:-]{7,127}$") _PORTABLE_ID = re.compile(r"^[a-z0-9][a-z0-9_-]{0,62}$") @@ -297,6 +298,31 @@ class PlanRunSubmissionResolver: default_size_class: str | None = None input_size_estimators: tuple[tuple[str, InputSizeEstimator], ...] = () + @property + def supports_repairs(self) -> bool: + """Whether a registered route can transport bounded retained-raw repair.""" + return any(_supports_date_repair(plan) for plan in self.plans.plans) + + def repair_environments(self, record: GraphRecord) -> tuple[str, ...]: + """Current graph environments with single-task Cloud Run transport.""" + return tuple( + sorted( + { + plan.environment + for plan in self.plans.current_for_graph(record) + if _supports_date_repair(plan) + } + ) + ) + + def require_repair_submission(self, submission: RunSubmission) -> None: + """Check exact plan eligibility before admitting a repair to the lifecycle.""" + plan = self.plans.for_submission(submission) + if submission.trigger.repair_window is None or not _supports_date_repair(plan): + raise ControlOperationUnavailableError( + "Date repair requires a single-task Cloud Run execution plan." + ) + def __post_init__(self) -> None: revisions = [candidate.plan_revision for candidate in self.placement_candidates] if len(revisions) != len(set(revisions)): @@ -794,6 +820,7 @@ def find_api_start( environments: tuple[str, ...], placement_mode: PlacementMode, idempotency_key: str, + repair_window: GraphRepairWindow | None = None, ) -> RunStatusResponse | None: """Replay one unambiguous no-sizing API request before mutable estimation.""" if ( @@ -847,6 +874,7 @@ def find_api_start( and durable.graph_content_sha256 == record.content_sha256 and durable.trigger.kind is TriggerKind.API and durable.trigger.trigger_id == "control-api" + and durable.trigger.repair_window == repair_window and placement_matches and ( durable.size_class_decision is None @@ -1027,6 +1055,7 @@ def _replay(self, address: RunAddress, *, idempotency_key: str) -> MutationResul kind=TriggerKind.API, trigger_id="control-api", replay_of_run_id=source.run_id, + repair_window=source.trigger.repair_window, ), placement_decision=PlacementDecision( mode=PlacementMode.REPLAY, @@ -1351,6 +1380,10 @@ def _now(self) -> datetime: return now.astimezone(UTC) +def _supports_date_repair(plan: ExecutionPlan) -> bool: + return plan.backend_id == "cloud_run" and plan.execution_template.schedule.task_count == 1 + + def _run_state(record: RunRecord) -> RunState: if record.run_state is HostedRunState.QUEUED: return RunState.QUEUED @@ -1387,6 +1420,7 @@ def _status_response(record: RunRecord) -> RunStatusResponse: assertions=summary.assertions if summary else 0, assets=summary.assets if summary else 0, result_schema=summary.schema if summary else None, + repair_window=record.trigger.repair_window, skipped=summary.skipped if summary else False, telemetry=( RunTelemetrySummary( diff --git a/src/dander/executor.py b/src/dander/executor.py index 658e3707..7b136fd2 100644 --- a/src/dander/executor.py +++ b/src/dander/executor.py @@ -29,6 +29,7 @@ from dander.catalog import CatalogAsset, CatalogPublisher, MetadataStore from dander.concurrency import OwnershipGuard from dander.ingestion import SourceConfig + from dander.pipeline.repair import GraphRepairWindow from dander.state import LeaseStore, RunHistoryStore _LOGGER = logging.getLogger(__name__) @@ -77,7 +78,7 @@ def __init__( *, pipeline_id: str, source_config: SourceConfig, - ingestion: _IngestionRunner, + ingestion: _IngestionRunner | None, history: RunHistoryStore, catalog: str | None = None, project: str | None = None, @@ -92,7 +93,12 @@ def __init__( catalog_publisher: CatalogPublisher | None = None, dataplex_publisher: CatalogPublisher | None = None, leases: LeaseStore | None = None, + repair_window: GraphRepairWindow | None = None, ) -> None: + if repair_window is not None and (ingestion is not None or not build_models): + raise ValueError("Date repair must build graph outputs without an ingestion runner") + if ingestion is None and repair_window is None: + raise ValueError("Ordinary pipeline execution requires an ingestion runner") if build_models and transform_runner is None: raise ValueError("A transform runner is required when build_models is enabled") if catalog is None: @@ -155,6 +161,7 @@ def __init__( self._registry_output = registry_output self._catalog_publisher = catalog_publisher or dataplex_publisher self._leases = leases + self._repair_window = repair_window def execute( self, @@ -165,7 +172,7 @@ def execute( """Execute every enabled stage and record one truthful terminal outcome.""" started_ns = time.monotonic_ns() run_id = run_id or uuid4().hex - stage = RunStage.INGEST + stage = RunStage.TRANSFORM if self._repair_window is not None else RunStage.INGEST endpoints = extracted = affected = models = assertions = assets = 0 start_history = self._history.restart_retryable if retry else self._history.start start_history(run_id, self._source_config.name, pipeline_id=self._pipeline_id) @@ -209,10 +216,13 @@ def execute( self._pipeline_id, current_run_id=run_id, ) - ingestion_result = self._ingestion.run( - run_id=run_id, - ownership=heartbeat, - ) + if self._repair_window is None: + assert self._ingestion is not None + ingestion_result = self._ingestion.run(run_id=run_id, ownership=heartbeat) + else: + ingestion_result = PipelineRunResult( + run_id=run_id, source=self._source_config.name, endpoints=() + ) endpoints = len(ingestion_result.endpoints) extracted = sum(result.extracted for result in ingestion_result.endpoints) affected = sum(result.affected for result in ingestion_result.endpoints) diff --git a/src/dander/pipeline/repair.py b/src/dander/pipeline/repair.py new file mode 100644 index 00000000..82f22bf4 --- /dev/null +++ b/src/dander/pipeline/repair.py @@ -0,0 +1,124 @@ +"""Retained-raw date repair contracts and credential-free eligibility checks.""" + +from __future__ import annotations + +import re +from dataclasses import dataclass +from datetime import date +from typing import TYPE_CHECKING, Literal, Self + +from pydantic import BaseModel, ConfigDict, field_validator, model_validator + +from dander.pipeline.graph_ops import validate_field_wiring +from dander.pipeline.node_config import TargetNodeConfig +from dander.writer import WriteMode + +if TYPE_CHECKING: + from dander.pipeline.graph import PipelineGraph + +_DATE = re.compile(r"^[0-9]{4}-[0-9]{2}-[0-9]{2}$") +_KEY_TYPES = frozenset( + { + "STRING", + "INT64", + "INTEGER", + "BOOL", + "BOOLEAN", + "DATE", + "DATETIME", + "TIMESTAMP", + "BYTES", + "NUMERIC", + "BIGNUMERIC", + } +) + + +class GraphRepairError(ValueError): + """The graph cannot safely repair a date range from retained raw data.""" + + +class GraphRepairWindow(BaseModel): + """UTC calendar dates: start is inclusive and end is exclusive.""" + + model_config = ConfigDict(extra="forbid", frozen=True) + + start_date: date + end_date: date + + @field_validator("start_date", "end_date", mode="before") + @classmethod + def strict_date(cls, value: object) -> date: + if type(value) is date: + return value + if not isinstance(value, str) or not _DATE.fullmatch(value): + raise ValueError("Repair dates must use YYYY-MM-DD") + return date.fromisoformat(value) + + @model_validator(mode="after") + def ordered_window(self) -> Self: + if self.end_date <= self.start_date: + raise ValueError("Repair end_date must be after start_date (exclusive)") + return self + + +@dataclass(frozen=True, slots=True) +class GraphTargetRepair: + node_id: str + partition_field: str + partition_type: Literal["DATE", "TIMESTAMP"] + business_key: tuple[str, ...] + + +def plan_graph_repair( + graph: PipelineGraph, + window: GraphRepairWindow, +) -> tuple[GraphTargetRepair, ...]: + """Validate every declared output before any source or warehouse client is constructed. + + This checks the authored write contract. Ordinary executable graph compilation still runs + later, and publication checks actual partition values and key collisions transactionally. + """ + GraphRepairWindow.model_validate(window.model_dump()) + validate_field_wiring(graph) + targets: list[GraphTargetRepair] = [] + for node in graph.nodes: + if node.type != "target": + continue + config = node.config + if not isinstance(config, TargetNodeConfig) or config.writer is None: + raise GraphRepairError(f"Repair target {node.id!r} needs a declared writer") + writer = config.writer + if writer.write_mode is not WriteMode.REPLACE: + raise GraphRepairError(f"Repair target {node.id!r} must use replace write mode") + partition = writer.partitioning + if partition is None or partition.field is None: + raise GraphRepairError( + f"Repair target {node.id!r} needs a DATE/TIMESTAMP partition field" + ) + fields = {field.name: (field.cast_to or field.type).upper() for field in node.fields} + partition_type = fields.get(partition.field) + if partition_type not in {"DATE", "TIMESTAMP"}: + raise GraphRepairError( + f"Repair target {node.id!r} partition field must be DATE or TIMESTAMP" + ) + keys = tuple(writer.destination.business_key) + if ( + not keys + or len(keys) != len(set(keys)) + or any(fields.get(key) not in _KEY_TYPES for key in keys) + ): + raise GraphRepairError( + f"Repair target {node.id!r} needs distinct, declared scalar business-key fields" + ) + targets.append( + GraphTargetRepair( + node_id=node.id, + partition_field=partition.field, + partition_type="DATE" if partition_type == "DATE" else "TIMESTAMP", + business_key=keys, + ) + ) + if not targets: + raise GraphRepairError("Date repair requires at least one configured output") + return tuple(targets) diff --git a/src/dander/pipeline/runtime.py b/src/dander/pipeline/runtime.py index 28ee8066..be8fdf2c 100644 --- a/src/dander/pipeline/runtime.py +++ b/src/dander/pipeline/runtime.py @@ -18,6 +18,12 @@ from dander.pipeline.errors import GraphValidationError from dander.pipeline.graph_ops import validate_field_wiring from dander.pipeline.node_config import SourceNodeConfig, TargetNodeConfig +from dander.pipeline.repair import ( + GraphRepairError, + GraphRepairWindow, + GraphTargetRepair, + plan_graph_repair, +) from dander.warehouse import BigQuerySchemaCompatibilityError, RelationRef from dander.writer.base import WriteMode @@ -66,6 +72,8 @@ class GraphExecutionPlan: bindings: GraphSourceBindings targets: tuple[CompiledTarget, ...] + repair_window: GraphRepairWindow | None = None + repair_targets: tuple[GraphTargetRepair, ...] = () def plan_graph_execution( @@ -75,6 +83,7 @@ def plan_graph_execution( endpoint_relations: Mapping[str, RelationRef] | None = None, project: str | None = None, dataset: str | None = None, + repair_window: GraphRepairWindow | None = None, ) -> GraphExecutionPlan: """Bind graph sources to connector endpoints and compile every executable target.""" try: @@ -194,6 +203,16 @@ def plan_graph_execution( target=replace(compiled.target, schema=(), declared_schema=schema), ) ) + try: + repair_targets = plan_graph_repair(graph, repair_window) if repair_window else () + except GraphRepairError as error: + raise GraphRuntimeError(str(error)) from error + if repair_window is not None: + destinations = [target.target.relation_ref for target in targets] + if len(destinations) != len(set(destinations)): + raise GraphRuntimeError("Repair outputs must have distinct destinations") + if any(destination in relations.values() for destination in destinations): + raise GraphRuntimeError("Repair outputs cannot overwrite their retained raw inputs") return GraphExecutionPlan( bindings=GraphSourceBindings( connector=source_config.name, @@ -201,4 +220,6 @@ def plan_graph_execution( source_relations=relations, ), targets=tuple(targets), + repair_window=repair_window, + repair_targets=repair_targets, ) diff --git a/src/dander/providers/bigquery/graph.py b/src/dander/providers/bigquery/graph.py index 1eed889f..450ac800 100644 --- a/src/dander/providers/bigquery/graph.py +++ b/src/dander/providers/bigquery/graph.py @@ -38,11 +38,12 @@ ) if TYPE_CHECKING: - from collections.abc import Iterable + from collections.abc import Iterable, Mapping from pathlib import Path from dander.concurrency import OwnershipGuard from dander.pipeline.graph import Node + from dander.pipeline.repair import GraphRepairWindow, GraphTargetRepair from dander.writer.bigquery import _BigQueryClient as _WriterClient _IDENTIFIER = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$") @@ -66,6 +67,38 @@ def delete_table(self, table: str, *, not_found_ok: bool = False) -> None: """Delete one run-scoped staging table.""" +class _RepairQueryJob: + """Keep actual committed row counts from the repair script's final SELECT.""" + + def __init__(self, job: _QueryJob) -> None: + self._job = job + self.output_rows = 0 + self.num_dml_affected_rows = 0 + + def __getattr__(self, name: str) -> object: + return getattr(self._job, name) + + def result(self) -> object: + result = self._job.result() + rows = iter(cast("Iterable[Mapping[str, object]]", result)) + row = next(rows, None) + if row is None: + raise GraphRuntimeError("Date repair did not return its committed row counts") + written, affected = row["rows_written"], row["rows_affected"] + if ( + not isinstance(written, int) + or isinstance(written, bool) + or not isinstance(affected, int) + or isinstance(affected, bool) + or written < 0 + or affected < written + ): + raise GraphRuntimeError("Date repair returned invalid committed row counts") + self.output_rows = written + self.num_dml_affected_rows = affected + return result + + class BigQueryGraphRunner: """Materialize compiled graph targets inside the transform stage.""" @@ -78,6 +111,10 @@ def __init__( ) -> None: for target in plan.targets: _validate_target(target, project) + if plan.repair_window is not None and {item.node_id for item in plan.repair_targets} != { + target.node_id for target in plan.targets + }: + raise GraphRuntimeError("Date repair requires an eligible partition for every output") self._plan = plan self._project = project self._client = client or cast( @@ -93,6 +130,10 @@ def build( ownership: OwnershipGuard | None = None, ) -> TransformRunResult: """Stage and transactionally publish selected replace-mode targets.""" + if self._plan.repair_window is not None and ( + ownership is None or ownership.fence is None or ownership.fence.lease_table is None + ): + raise GraphRuntimeError("Date repair requires active BigQuery lease fencing") selected_ids = set(selected) if selected is not None else None known = {target.node_id for target in self._plan.targets} if selected_ids is not None and (unknown := sorted(selected_ids - known)): @@ -143,13 +184,14 @@ def _materialize( ), operation=TelemetryOperation.TRANSFORM, ) - telemetry.run( - lambda: self._client.query( - f"CREATE TABLE IF NOT EXISTS `{target_id}` AS " - f"SELECT {quoted_columns} FROM `{staging_id}` WHERE FALSE" - ), - operation=TelemetryOperation.TRANSFORM, - ) + if self._plan.repair_window is None: + telemetry.run( + lambda: self._client.query( + f"CREATE TABLE IF NOT EXISTS `{target_id}` AS " + f"SELECT {quoted_columns} FROM `{staging_id}` WHERE FALSE" + ), + operation=TelemetryOperation.TRANSFORM, + ) if ownership is not None: ownership.verify() replacement = ( @@ -157,6 +199,13 @@ def _materialize( f"INSERT INTO `{target_id}` ({quoted_columns})\n" f"SELECT {quoted_columns} FROM `{staging_id}`" ) + if self._plan.repair_window is not None: + repair = next( + item for item in self._plan.repair_targets if item.node_id == compiled.node_id + ) + replacement = _repair_dml( + target_id, staging_id, columns, repair, self._plan.repair_window + ) fence = ownership.fence if ownership is not None else None if fence is not None: script = fenced_dml(replacement, fence) @@ -165,11 +214,78 @@ def _materialize( else: script = f"BEGIN TRANSACTION;\n{replacement};\nCOMMIT TRANSACTION;" submit = partial(self._client.query, script) - telemetry.run(submit, operation=TelemetryOperation.TRANSFORM, retry_mutation=True) + if self._plan.repair_window is not None: + script = ( + "DECLARE dander_repair_deleted INT64 DEFAULT 0;\n" + "DECLARE dander_repair_inserted INT64 DEFAULT 0;\n" + + script + + "\nSELECT dander_repair_inserted AS rows_written, " + "dander_repair_deleted + dander_repair_inserted AS rows_affected;" + ) + repair_config = fencing_job_config(fence) if fence is not None else None + telemetry.run( + lambda: _RepairQueryJob(self._client.query(script, job_config=repair_config)), + operation=TelemetryOperation.TRANSFORM, + retry_mutation=True, + ) + else: + telemetry.run(submit, operation=TelemetryOperation.TRANSFORM, retry_mutation=True) finally: self._client.delete_table(staging_id, not_found_ok=True) +def _repair_dml( + target_id: str, + staging_id: str, + columns: tuple[str, ...], + repair: GraphTargetRepair, + window: GraphRepairWindow, +) -> str: + """Validate the selected keys against both retained sides before changing any target row.""" + + def selected(alias: str) -> str: + value = f"{alias}.`{repair.partition_field}`" + if repair.partition_type == "TIMESTAMP": + value = f"DATE({value}, 'UTC')" + return ( + f"({value} >= DATE '{window.start_date.isoformat()}' " + f"AND {value} < DATE '{window.end_date.isoformat()}')" + ) + + invalid = " OR ".join(f"`{column}` IS NULL" for column in repair.business_key) + keys = ", ".join(f"`{key}`" for key in repair.business_key) + join = " AND ".join(f"s.`{key}` = t.`{key}`" for key in repair.business_key) + quoted_columns = ", ".join(f"`{column}`" for column in columns) + validations: list[str] = [] + for relation in (staging_id, target_id): + validations.extend( + ( + f"ASSERT NOT EXISTS (SELECT 1 FROM `{relation}` s " + f"WHERE {selected('s')} AND ({invalid})) " + "AS 'Selected repair business keys must not be null';", + f"ASSERT NOT EXISTS (SELECT 1 FROM `{relation}` s " + f"WHERE {selected('s')} GROUP BY {keys} " + "HAVING COUNT(*) > 1) AS 'Repair business keys must be unique';", + ) + ) + validations.extend( + ( + f"ASSERT NOT EXISTS (SELECT 1 FROM `{staging_id}` s JOIN `{target_id}` t ON {join} " + f"WHERE {selected('s')} AND NOT COALESCE({selected('t')}, FALSE)) " + "AS 'Repair staged keys overlap target rows outside the selected dates';", + f"ASSERT NOT EXISTS (SELECT 1 FROM `{target_id}` t JOIN `{staging_id}` s ON {join} " + f"WHERE {selected('t')} AND NOT COALESCE({selected('s')}, FALSE)) " + "AS 'Repair target keys moved outside the selected dates in retained raw data';", + f"DELETE FROM `{target_id}` t WHERE {selected('t')};", + "SET dander_repair_deleted = @@row_count;", + f"INSERT INTO `{target_id}` ({quoted_columns})\n" + f"SELECT {quoted_columns} FROM `{staging_id}` s WHERE {selected('s')};", + "SET dander_repair_inserted = @@row_count", + ) + ) + return "\n".join(validations) + + def prepare_bigquery_target_writer( target_node: Node, *, diff --git a/src/dander/providers/postgresql/transform.py b/src/dander/providers/postgresql/transform.py index 80ce7a7c..71f64038 100644 --- a/src/dander/providers/postgresql/transform.py +++ b/src/dander/providers/postgresql/transform.py @@ -187,6 +187,8 @@ def __init__( target_fence: PostgreSQLTargetFence, timeouts: PostgreSQLTimeouts, ) -> None: + if plan.repair_window is not None: + raise GraphRuntimeError("Date repair is unavailable for PostgreSQL graph execution") self._plan = plan self._database = database self._pool = pool diff --git a/src/dander/providers/redshift/transform.py b/src/dander/providers/redshift/transform.py index b7ef90a8..3614284a 100644 --- a/src/dander/providers/redshift/transform.py +++ b/src/dander/providers/redshift/transform.py @@ -216,6 +216,8 @@ def __init__( target_fence: RedshiftTargetFence, statement_timeout_ms: int, ) -> None: + if plan.repair_window is not None: + raise GraphRuntimeError("Date repair is unavailable for Redshift graph execution") self._plan = plan self._database = database self._connection_factory = connection_factory diff --git a/src/dander/providers/snowflake/transform.py b/src/dander/providers/snowflake/transform.py index b823b45b..51aa528c 100644 --- a/src/dander/providers/snowflake/transform.py +++ b/src/dander/providers/snowflake/transform.py @@ -212,6 +212,8 @@ def __init__( target_fence: SnowflakeTargetFence, warehouse: str | None = None, ) -> None: + if plan.repair_window is not None: + raise GraphRuntimeError("Date repair is unavailable for Snowflake graph execution") self._plan = plan self._database = database self._connection_factory = connection_factory diff --git a/tests/cli/test_run_command.py b/tests/cli/test_run_command.py index 45603aab..c8ca8d9a 100644 --- a/tests/cli/test_run_command.py +++ b/tests/cli/test_run_command.py @@ -4,7 +4,7 @@ import subprocess import sys -from dataclasses import dataclass +from dataclasses import dataclass, replace from pathlib import Path from types import SimpleNamespace from typing import TYPE_CHECKING, Any, Protocol, cast @@ -12,11 +12,14 @@ import pytest from click import ClickException from rich.console import Console +from tests.pipeline.test_date_repair import repair_graph, repair_source from typer.testing import CliRunner import dander.cli.run_command as run_module from dander.cli.main import app +from dander.control.graph_store import canonicalize_graph_document from dander.executor import PipelineExecutionResult +from dander.pipeline.repair import GraphRepairWindow from dander.runtime import EndpointRunResult, PipelineRunResult from dander.security import NoAuth from dander.state import StateCapabilities, StateMigration, StateRuntime @@ -386,6 +389,7 @@ def test_postgresql_rejects_bigquery_safety_before_external_clients( sandbox=sandbox, guarded_free_tier=guarded_free_tier, dry_run=False, + repair_window=None, ) resolved = SimpleNamespace( warehouse_provider="postgresql", @@ -408,7 +412,9 @@ def __init__(self) -> None: def test_postgresql_dataplex_publication_fails_before_executor( monkeypatch: MonkeyPatch, ) -> None: - options = SimpleNamespace(sandbox=False, guarded_free_tier=False, dry_run=False) + options = SimpleNamespace( + sandbox=False, guarded_free_tier=False, dry_run=False, repair_window=None + ) resolved = SimpleNamespace( warehouse_provider="postgresql", publish_catalog=True, @@ -583,3 +589,108 @@ def test_only_unsupported_postgresql_state_bigquery_warehouse_pair_fails_closed( state_provider="postgresql", warehouse_provider="redshift", ) + + +def _repair_options(tmp_path: Path) -> run_module.RunOptions: + (tmp_path / "connectors").mkdir() + (tmp_path / "connectors/records.yaml").write_text(repair_source().model_dump_json()) + (tmp_path / "graph.json").write_text(repair_graph().model_dump_json(by_alias=True)) + (tmp_path / "dander.yaml").write_text( + "version: 1\npipelines:\n repair:\n source: records\n graph: graph.json\n" + " models: []\n build_models: false\n publish_dataplex: false\n" + ) + return run_module.RunOptions( + pipeline_or_source="repair", + project="unit-project", + dataset="raw", + connectors_dir=tmp_path / "connectors", + project_config=tmp_path / "dander.yaml", + platforms_config=None, + deployment=None, + dry_run=False, + sandbox=False, + guarded_free_tier=False, + batch_rows=10_000, + budget_name="unused", + state_path=tmp_path / "state.sqlite3", + build_models=False, + models_dir=tmp_path, + selected_models=None, + catalog_output=None, + publish_dataplex=False, + dataplex_location="us-central1", + repair_window=GraphRepairWindow.model_validate( + {"start_date": "2026-09-01", "end_date": "2026-09-03"} + ), + repair_graph_content_sha256=canonicalize_graph_document( + repair_graph().model_dump(mode="json", by_alias=True) + ).content_sha256, + ) + + +def test_repair_composition_does_not_build_source_auth_extraction_or_touch_watermarks( + tmp_path: Path, + monkeypatch: MonkeyPatch, +) -> None: + from dander.state import SqliteLeaseStore, SqliteRunHistoryStore + from dander.transform import TransformRunResult + + options = _repair_options(tmp_path) + resolved = run_module._resolve_run(options) + + def forbidden(*args: object, **kwargs: object) -> None: + pytest.fail("repair must not contact source or watermarks") + + for name in ( + "build_secret_store", + "build_auth", + "build_source_adapter", + "_build_ingestion_runner", + ): + monkeypatch.setattr(run_module, name, forbidden) + + class _Watermarks: + def __getattr__(self, name: str) -> object: + pytest.fail(f"repair must not access watermark method {name}") + + class _Transform: + def build(self, *args: object, **kwargs: object) -> TransformRunResult: + assert kwargs["ownership"] is not None + return TransformRunResult(models=("target",), assertions=0) + + stores = SimpleNamespace( + history=SqliteRunHistoryStore(options.state_path), + leases=SqliteLeaseStore(options.state_path), + watermarks=_Watermarks(), + metadata=None, + ) + monkeypatch.setattr(run_module, "_build_control_stores", lambda *args: stores) + monkeypatch.setattr(run_module, "_build_warehouse_runtime", lambda *args: object()) + monkeypatch.setattr(run_module, "_build_transform_runner", lambda *args: _Transform()) + + result = run_module._build_executor(options, resolved).execute(run_id="repair-test") + + assert result.ingestion.endpoints == () + assert result.models == ("target",) + + +@pytest.mark.parametrize("failure", ["hash_changed", "hash_missing", "unsupported_provider"]) +def test_repair_fails_before_clients_for_stale_graph_or_unsupported_backend( + tmp_path: Path, + monkeypatch: MonkeyPatch, + failure: str, +) -> None: + options = _repair_options(tmp_path) + if failure == "hash_changed": + options = replace(options, repair_graph_content_sha256="0" * 64) + elif failure == "hash_missing": + options = replace(options, repair_graph_content_sha256=None) + else: + resolved = replace(run_module._resolve_run(options), warehouse_provider="postgresql") + monkeypatch.setattr(run_module, "_resolve_run", lambda *args: resolved) + monkeypatch.setattr( + run_module, "_build_executor", lambda *args: pytest.fail("no clients allowed") + ) + + with pytest.raises(ClickException, match="hash|direct BigQuery"): + run_module.execute_run(options, console=Console()) diff --git a/tests/cli/test_runtime_cli.py b/tests/cli/test_runtime_cli.py index 3bb8eacc..ff5fb4ff 100644 --- a/tests/cli/test_runtime_cli.py +++ b/tests/cli/test_runtime_cli.py @@ -5,9 +5,11 @@ import json import logging from contextlib import contextmanager +from datetime import date from pathlib import Path from typing import TYPE_CHECKING, cast +import pytest from click import ClickException from typer.testing import CliRunner @@ -49,6 +51,7 @@ def _invoke( extra_env: dict[str, str] | None = None, platform: str = "gcp", physical_plan: PhysicalPlan | None = None, + graph_repair_contract: str | None = None, ) -> Result: monkeypatch.setattr(runtime_module, "execute_run", execute) arguments = [ @@ -71,6 +74,8 @@ def _invoke( arguments.extend( ("--physical-plan", serialize_physical_plan(physical_plan).decode("utf-8")) ) + if graph_repair_contract is not None: + arguments.extend(("--graph-repair-contract", graph_repair_contract)) return CliRunner().invoke( app, arguments, @@ -83,6 +88,127 @@ def _invoke( ) +def test_runtime_repair_passes_exact_window_and_graph_identity_to_execution( + monkeypatch: MonkeyPatch, +) -> None: + captured: list[RunOptions] = [] + + def execute(options: RunOptions, **_: object) -> PipelineExecutionResult: + captured.append(options) + return PipelineExecutionResult( + run_id="repair", + pipeline_id="greenhouse_jobs", + ingestion=PipelineRunResult(run_id="repair", source="retained_raw", endpoints=()), + models=("output",), + assertions=0, + assets=1, + ) + + payload = { + "start_date": "2026-09-01", + "end_date": "2026-09-03", + "graph_content_sha256": "a" * 64, + "execution_name": "job-123", + } + result = _invoke( + monkeypatch, + execute, + graph_repair_contract="io.dander.graph-repair/v1", + extra_env={ + "CLOUD_RUN_EXECUTION": "job-123", + "DANDER_GRAPH_REPAIR_JSON": json.dumps(payload), + }, + ) + + assert result.exit_code == RuntimeExitCode.SUCCESS, result.output + assert len(captured) == 1 + window = captured[0].repair_window + assert window is not None + assert window.start_date == date(2026, 9, 1) + assert window.end_date == date(2026, 9, 3) + assert captured[0].repair_graph_content_sha256 == "a" * 64 + + +@pytest.mark.parametrize( + ("marker", "payload_change", "execution_name"), + [ + (None, {}, "job-123"), + ("io.dander.graph-repair/v0", {}, "job-123"), + ("io.dander.graph-repair/v1", {}, ""), + ("io.dander.graph-repair/v1", {"end_date": "2026-08-01"}, "job-123"), + ("io.dander.graph-repair/v1", {"graph_content_sha256": "invalid"}, "job-123"), + ("io.dander.graph-repair/v1", {"unknown": "extra"}, "job-123"), + ("io.dander.graph-repair/v1", {"end_date": "invalid"}, "another-execution"), + ], +) +def test_runtime_rejects_invalid_repair_before_execution( + monkeypatch: MonkeyPatch, + marker: str | None, + payload_change: dict[str, str], + execution_name: str, +) -> None: + def execute(options: RunOptions, **_: object) -> PipelineExecutionResult: + raise AssertionError("An invalid repair must not execute any pipeline work") + + payload = { + "start_date": "2026-09-01", + "end_date": "2026-09-03", + "graph_content_sha256": "a" * 64, + "execution_name": "job-123", + **payload_change, + } + result = _invoke( + monkeypatch, + execute, + graph_repair_contract=marker, + extra_env={ + "CLOUD_RUN_EXECUTION": execution_name, + "DANDER_GRAPH_REPAIR_JSON": json.dumps(payload), + }, + ) + + assert result.exit_code == RuntimeExitCode.INVALID_INVOCATION + assert "runtime.started" not in result.output + + +def test_independently_scheduled_run_ignores_valid_stale_repair_selection( + monkeypatch: MonkeyPatch, +) -> None: + captured: list[RunOptions] = [] + + def execute(options: RunOptions, **_: object) -> PipelineExecutionResult: + captured.append(options) + return PipelineExecutionResult( + run_id="normal", + pipeline_id="greenhouse_jobs", + ingestion=PipelineRunResult(run_id="normal", source="greenhouse_jobs", endpoints=()), + models=("output",), + assertions=0, + assets=1, + ) + + payload = { + "start_date": "2026-09-01", + "end_date": "2026-09-03", + "graph_content_sha256": "a" * 64, + "execution_name": "prior-repair", + } + result = _invoke( + monkeypatch, + execute, + graph_repair_contract="io.dander.graph-repair/v1", + extra_env={ + "CLOUD_RUN_EXECUTION": "scheduled-normal", + "DANDER_GRAPH_REPAIR_JSON": json.dumps(payload), + }, + ) + + assert result.exit_code == RuntimeExitCode.SUCCESS, result.output + assert len(captured) == 1 + assert captured[0].repair_window is None + assert captured[0].repair_graph_content_sha256 is None + + def test_runtime_execute_uses_launcher_run_id_and_emits_json_lines( monkeypatch: MonkeyPatch, ) -> None: diff --git a/tests/control/test_cloud_run_execution_backend.py b/tests/control/test_cloud_run_execution_backend.py index 9dfec7af..b7099f2c 100644 --- a/tests/control/test_cloud_run_execution_backend.py +++ b/tests/control/test_cloud_run_execution_backend.py @@ -3,8 +3,10 @@ from __future__ import annotations import hashlib +import json +from copy import deepcopy from dataclasses import dataclass, field, replace -from datetime import UTC, datetime +from datetime import UTC, date, datetime import pytest @@ -29,6 +31,7 @@ ResourceProjection, ScheduleProjection, ) +from dander.pipeline.repair import GraphRepairWindow from dander.providers.cloud_run import CloudRunBinding from dander.runtime_contract import RUNTIME_CONTRACT @@ -90,6 +93,8 @@ class _Transport: calls: list[tuple[str, str, dict[str, object]]] = field(default_factory=list) log_response: dict[str, object] = field(default_factory=dict) lose_patch_response: bool = False + reject_restore: bool = False + supersede_before_restore: bool = False fail_job_read: bool = False fail_log_read: bool = False close_count: int = 0 @@ -117,9 +122,22 @@ def request(self, method: str, url: str, **kwargs: object) -> _Response: if method == "PATCH" and "/jobs/" in resource: payload = kwargs["json"] assert isinstance(payload, dict) + if "startExecutionToken" not in payload: + if self.supersede_before_restore: + self.job["startExecutionToken"] = "f" * 16 + self.job["etag"] = "new-launch-etag" + return _Response(409, {}) + if self.reject_restore: + return _Response(503, {}) + self.job.update(deepcopy(payload)) + if self.lose_patch_response: + raise OSError("provider transport secret") + return _Response(200, {"name": "operations/restore"}) token = payload["startExecutionToken"] assert isinstance(token, str) - self.job["startExecutionToken"] = token + self.job.update(deepcopy(payload)) + template = payload["template"] + assert isinstance(template, dict) execution_resource = ( f"projects/{PROJECT}/locations/{REGION}/jobs/{JOB}/executions/{JOB}-{token}" ) @@ -127,6 +145,7 @@ def request(self, method: str, url: str, **kwargs: object) -> _Response: "name": execution_resource, "job": JOB, "taskCount": 1, + "template": deepcopy(template["template"]), "startTime": "2026-08-26T18:00:01Z", } if self.lose_patch_response: @@ -250,12 +269,18 @@ def _backend( ) -def _start(backend: CloudRunExecutionBackend, plan: ExecutionPlan) -> BackendHandle: +def _start( + backend: CloudRunExecutionBackend, + plan: ExecutionPlan, + *, + trigger: RunTrigger | None = None, + run_id: str = "run-hosted-001", +) -> BackendHandle: return backend.submit_or_adopt( plan, - run_id="run-hosted-001", + run_id=run_id, attempt_id="attempt-1-hosted", - trigger=RunTrigger(kind=TriggerKind.API, trigger_id="control-api"), + trigger=trigger or RunTrigger(kind=TriggerKind.API, trigger_id="control-api"), ) @@ -317,6 +342,186 @@ def test_submit_rejects_deployed_job_drift_before_mutation() -> None: assert not [call for call in transport.calls if call[0] == "PATCH"] +def _repair_trigger() -> RunTrigger: + return RunTrigger( + kind=TriggerKind.API, + trigger_id="control-api", + repair_window=GraphRepairWindow(start_date=date(2026, 9, 1), end_date=date(2026, 9, 3)), + ) + + +def _execution_container(transport: _Transport, handle: BackendHandle) -> dict[str, object]: + task = transport.executions[handle.execution_id]["template"] + assert isinstance(task, dict) + containers = task["containers"] + assert isinstance(containers, list) and isinstance(containers[0], dict) + return containers[0] + + +def test_repair_is_execution_bound_and_adopted_after_lost_response() -> None: + backend, plan, transport = _backend( + transport=_Transport(_job(_plan()), lose_patch_response=True) + ) + + handle = _start(backend, plan, trigger=_repair_trigger()) + adopted = _start(backend, plan, trigger=_repair_trigger()) + + assert adopted == handle + assert len([call for call in transport.calls if call[0] == "PATCH"]) == 1 + container = _execution_container(transport, handle) + assert container["args"] == [ + *plan.execution_template.command, + "--graph-repair-contract", + "io.dander.graph-repair/v1", + ] + environment = container["env"] + assert isinstance(environment, list) + payload = json.loads(environment[0]["value"]) + assert payload == { + "start_date": "2026-09-01", + "end_date": "2026-09-03", + "graph_content_sha256": plan.graph_content_sha256, + "execution_name": handle.execution_id.rsplit("/", maxsplit=1)[-1], + } + + +@pytest.mark.parametrize("execution_visible", [True, False]) +def test_adoption_rejects_another_repair_window_without_second_patch( + execution_visible: bool, +) -> None: + backend, plan, transport = _backend() + _start(backend, plan, trigger=_repair_trigger()) + if not execution_visible: + transport.executions.clear() + changed = replace( + _repair_trigger(), + repair_window=GraphRepairWindow(start_date=date(2026, 9, 2), end_date=date(2026, 9, 3)), + ) + + with pytest.raises(ExecutionBackendError, match="different date-repair selection"): + _start(backend, plan, trigger=changed) + + assert len([call for call in transport.calls if call[0] == "PATCH"]) == 1 + + +def test_normal_run_clears_repair_marker_and_window_without_changing_prior_execution() -> None: + backend, plan, transport = _backend() + repair = _start(backend, plan, trigger=_repair_trigger()) + + normal = _start(backend, plan, run_id="next-normal-run") + + assert _execution_container(transport, normal)["args"] == list(plan.execution_template.command) + assert _execution_container(transport, normal)["env"] == [] + assert _execution_container(transport, repair)["env"] + assert _start(backend, plan, trigger=_repair_trigger()) == repair + + +def test_repair_adoption_requires_the_guard_that_makes_old_images_fail_closed() -> None: + backend, plan, transport = _backend() + handle = _start(backend, plan, trigger=_repair_trigger()) + _execution_container(transport, handle)["args"] = list(plan.execution_template.command) + + with pytest.raises(ExecutionBackendError, match="immutable execution plan"): + _start(backend, plan, trigger=_repair_trigger()) + + assert len([call for call in transport.calls if call[0] == "PATCH"]) == 1 + + +def _fail_execution(transport: _Transport, handle: BackendHandle) -> None: + transport.executions[handle.execution_id].update( + {"completionTime": "2026-08-26T18:01:00Z", "failedCount": 1} + ) + + +@pytest.mark.parametrize("lost_response", [False, True]) +def test_terminal_repair_restores_job_without_starting_work_and_reconciles_lost_response( + lost_response: bool, +) -> None: + backend, plan, transport = _backend() + handle = _start(backend, plan, trigger=_repair_trigger()) + _fail_execution(transport, handle) + transport.lose_patch_response = lost_response + + observed = backend.observe(plan, handle) + backend.observe(plan, handle) + + assert observed.outcome is RunOutcome.FAILED + assert observed.cleanup_state is CleanupState.CONFIRMED + patches = [call[2]["json"] for call in transport.calls if call[0] == "PATCH"] + assert len(patches) == 2 + cleanup = patches[1] + assert isinstance(cleanup, dict) + assert cleanup["etag"] == "job-etag-1" + assert "startExecutionToken" not in cleanup + assert "runExecutionToken" not in cleanup + template = cleanup["template"] + assert isinstance(template, dict) + task = template["template"] + assert isinstance(task, dict) + containers = task["containers"] + assert isinstance(containers, list) + assert containers[0]["args"] == list(plan.execution_template.command) + assert containers[0]["env"] == [] + assert len(transport.executions) == 1 + + +def test_cleanup_conflict_preserves_newer_launch_configuration() -> None: + backend, plan, transport = _backend() + handle = _start(backend, plan, trigger=_repair_trigger()) + _fail_execution(transport, handle) + transport.supersede_before_restore = True + + observed = backend.observe(plan, handle) + + assert observed.cleanup_state is CleanupState.CONFIRMED + assert transport.job["startExecutionToken"] == "f" * 16 + assert transport.job["etag"] == "new-launch-etag" + assert len([call for call in transport.calls if call[0] == "PATCH"]) == 2 + backend.observe(plan, handle) + assert len([call for call in transport.calls if call[0] == "PATCH"]) == 2 + + +def test_failed_cleanup_remains_pending_until_existing_reconciliation_retries() -> None: + backend, plan, transport = _backend() + handle = _start(backend, plan, trigger=_repair_trigger()) + _fail_execution(transport, handle) + transport.reject_restore = True + + observed = backend.observe(plan, handle) + + assert observed.outcome is RunOutcome.FAILED + assert observed.cleanup_state is CleanupState.PENDING + transport.reject_restore = False + assert backend.observe(plan, handle).cleanup_state is CleanupState.CONFIRMED + assert len(transport.executions) == 1 + + +def test_old_repair_cleanup_does_not_change_a_later_normal_launch() -> None: + backend, plan, transport = _backend() + repair = _start(backend, plan, trigger=_repair_trigger()) + normal = _start(backend, plan, run_id="new-normal") + _fail_execution(transport, repair) + current = deepcopy(transport.job) + + assert backend.observe(plan, repair).cleanup_state is CleanupState.CONFIRMED + + assert transport.job == current + assert len([call for call in transport.calls if call[0] == "PATCH"]) == 2 + assert _execution_container(transport, normal)["env"] == [] + + +def test_normal_terminal_observation_does_not_patch_job_configuration() -> None: + backend, plan, transport = _backend() + handle = _start(backend, plan) + _fail_execution(transport, handle) + transport.calls.clear() + + assert backend.observe(plan, handle).cleanup_state is CleanupState.CONFIRMED + + assert len(transport.calls) == 1 + assert transport.calls[0][0] == "GET" + + def test_observe_normalizes_start_success_failure_and_cancellation() -> None: backend, plan, transport = _backend() handle = _start(backend, plan) diff --git a/tests/control/test_dataproc_serverless_execution_backend.py b/tests/control/test_dataproc_serverless_execution_backend.py index e78a799d..72b74c95 100644 --- a/tests/control/test_dataproc_serverless_execution_backend.py +++ b/tests/control/test_dataproc_serverless_execution_backend.py @@ -5,7 +5,7 @@ import copy import json from dataclasses import dataclass, field, replace -from datetime import UTC, datetime +from datetime import UTC, date, datetime import pytest @@ -42,6 +42,7 @@ PhysicalStage, serialize_physical_plan, ) +from dander.pipeline.repair import GraphRepairWindow from dander.providers.dataproc_serverless import ( DataprocServerlessBinding, DataprocServerlessOperationError, @@ -304,6 +305,20 @@ def _backend( ) +def test_repair_rejects_before_any_managed_spark_api_call() -> None: + backend, plan, transport = _backend() + trigger = RunTrigger( + kind=TriggerKind.API, + trigger_id="control-api", + repair_window=GraphRepairWindow(start_date=date(2026, 9, 1), end_date=date(2026, 9, 2)), + ) + + with pytest.raises(ExecutionBackendError, match="Date repair is not supported"): + backend.submit_or_adopt(plan, run_id="repair-run", attempt_id="attempt-1", trigger=trigger) + + assert not transport.calls + + def _start( backend: DataprocServerlessExecutionBackend, plan: ExecutionPlan, diff --git a/tests/control/test_fargate_execution_backend.py b/tests/control/test_fargate_execution_backend.py index e8120305..48b777f8 100644 --- a/tests/control/test_fargate_execution_backend.py +++ b/tests/control/test_fargate_execution_backend.py @@ -4,7 +4,7 @@ import json from dataclasses import replace -from datetime import UTC, datetime +from datetime import UTC, date, datetime from typing import TYPE_CHECKING, cast import pytest @@ -31,6 +31,7 @@ ScheduleProjection, SecretReference, ) +from dander.pipeline.repair import GraphRepairWindow from dander.providers.fargate import FargateBinding from dander.runtime_contract import RUNTIME_CONTRACT @@ -317,6 +318,23 @@ def _backend( ) +def test_repair_rejects_before_any_fargate_api_call(tmp_path: Path) -> None: + backend, plan, step_functions, logs, ecs = _backend(tmp_path) + trigger = RunTrigger( + kind=TriggerKind.API, + trigger_id="control-api", + repair_window=GraphRepairWindow(start_date=date(2026, 9, 1), end_date=date(2026, 9, 2)), + ) + + with pytest.raises(ExecutionBackendError, match="Date repair is not supported"): + backend.submit_or_adopt(plan, run_id="repair-run", attempt_id="attempt-1", trigger=trigger) + + assert not step_functions.describe_calls + assert not step_functions.start_calls + assert not logs.calls + assert not ecs.calls + + def _start( backend: FargateExecutionBackend, plan: ExecutionPlan, diff --git a/tests/control/test_hosted_control.py b/tests/control/test_hosted_control.py index 778d731c..2c85bb8c 100644 --- a/tests/control/test_hosted_control.py +++ b/tests/control/test_hosted_control.py @@ -25,6 +25,7 @@ RunStatusResponse, ) from dander.control.orchestration import PlacementMode, RunSubmission, RunTrigger, TriggerKind +from dander.pipeline.repair import GraphRepairWindow if TYPE_CHECKING: from datetime import datetime @@ -288,6 +289,7 @@ def find_api_start( environments: tuple[str, ...], placement_mode: PlacementMode, idempotency_key: str, + repair_window: GraphRepairWindow | None = None, ) -> RunStatusResponse | None: del record, environments, placement_mode, idempotency_key return None @@ -378,6 +380,118 @@ def _status(run_id: str) -> RunStatusResponse: return RunStatusResponse(run_id=run_id, state=RunState.QUEUED) +@dataclass(frozen=True) +class _RepairResolver(_SubmissionResolver): + supports_repairs: bool = True + + def repair_environments(self, record: GraphRecord) -> tuple[str, ...]: + return ("gcp",) + + def require_repair_submission(self, submission: RunSubmission) -> None: + assert submission.trigger.repair_window is not None + + +def _repair_graph() -> dict[str, object]: + fields = [{"name": "id", "type": "STRING"}, {"name": "updated_at", "type": "TIMESTAMP"}] + return { + "name": "repairable", + "nodes": [ + { + "id": "records", + "type": "source", + "name": "Records", + "fields": fields, + "config": {"connector": "records", "endpoint": "records"}, + }, + { + "id": "output", + "type": "target", + "name": "Daily output", + "fields": fields, + "config": { + "writer": { + "write_mode": "replace", + "destination": { + "dataset": "analytics", + "table": "daily", + "business_key": ["id"], + }, + "partitioning": {"field": "updated_at"}, + } + }, + }, + ], + "edges": [{"from": "records", "to": "output"}], + } + + +def test_repair_preview_and_start_bind_window_to_saved_graph_without_source_access() -> None: + lifecycle = _Lifecycle() + application = ControlApplication( + InMemoryGraphStore(), + lifecycle=cast("RunLifecyclePort", lifecycle), + submission_resolver=cast("RunSubmissionResolver", _RepairResolver(environment="gcp")), + projects=("demo-project",), + ) + window = {"start_date": "2026-09-01", "end_date": "2026-09-03"} + with TestClient(create_control_app(application)) as client: + created = _create(client, "alpha-graph", _repair_graph()) + assert created.status_code == 201 + route = "/v1/projects/demo-project/graphs/alpha-graph" + headers = {"If-Match": created.headers["etag"]} + preview = client.post(f"{route}/repair-preview", json=window, headers=headers) + assert preview.status_code == 200 + assert preview.json()["window"] == window + assert preview.json()["source"] == "retained_raw" + assert preview.json()["outputs"][0]["partition_field"] == "updated_at" + assert preview.json()["environments"] == ["gcp"] + assert lifecycle.starts == [] + assert client.get(route).headers["etag"] == created.headers["etag"] + started = client.post( + f"{route}/repairs", + json=window, + headers={**headers, "Idempotency-Key": "repair-key-0001"}, + ) + assert started.status_code == 202 + assert lifecycle.starts[0].graph.content_sha256 == created.json()["content_sha256"] + assert lifecycle.starts[0].trigger.repair_window == GraphRepairWindow.model_validate(window) + assert lifecycle.starts[0].idempotency_key == "repair-key-0001" + assert lifecycle.starts[0].environment == "gcp" + assert "run.repair" in client.get("/v1/capabilities").json()["operations"] + + unavailable = client.post( + f"{route}/repairs?environment=aws", + json=window, + headers={**headers, "Idempotency-Key": "repair-key-0002"}, + ) + assert unavailable.status_code == 501 + assert len(lifecycle.starts) == 1 + stale = client.post( + f"{route}/repair-preview", json=window, headers={"If-Match": '"c3RhbGU"'} + ) + assert stale.status_code == 412 + + +@pytest.mark.parametrize( + "body", + [ + {"start_date": "2026-09-03", "end_date": "2026-09-01"}, + {"start_date": "2026-09-01", "end_date": "2026-09-03", "source_token": "no-echo"}, + ], +) +def test_invalid_repair_body_is_rejected_before_dispatch(client: TestClient, body: object) -> None: + with client: + created = _create(client, "alpha-graph") + for suffix in ("repair-preview", "repairs"): + response = client.post( + f"/v1/projects/demo-project/graphs/alpha-graph/{suffix}", + json=body, + headers={"If-Match": created.headers["etag"], "Idempotency-Key": "repair-key-0001"}, + ) + assert response.status_code == 422 + assert "no-echo" not in response.text + + def test_normalized_lifecycle_receives_decoded_revision_and_explicit_idempotency() -> None: lifecycle = _Lifecycle() resolver = _SubmissionResolver() diff --git a/tests/control/test_orchestration_contracts.py b/tests/control/test_orchestration_contracts.py index 36a90a04..fc49eb1e 100644 --- a/tests/control/test_orchestration_contracts.py +++ b/tests/control/test_orchestration_contracts.py @@ -3,7 +3,7 @@ from __future__ import annotations from dataclasses import dataclass, field, replace -from datetime import UTC, datetime, timedelta +from datetime import UTC, date, datetime, timedelta import pytest @@ -42,6 +42,7 @@ transition_run, validate_submission_plan, ) +from dander.control.orchestration_serialization import deserialize_run_record, serialize_run_record from dander.deployment.projection import ( EXECUTION_PROJECTION_SCHEMA, ExecutionTemplate, @@ -50,6 +51,7 @@ ResourceProjection, ScheduleProjection, ) +from dander.pipeline.repair import GraphRepairWindow from dander.runtime_contract import RUNTIME_CONTRACT NOW = datetime(2026, 8, 25, 12, tzinfo=UTC) @@ -345,6 +347,50 @@ def test_placement_evidence_does_not_change_exact_plan_idempotency() -> None: assert create_run_record(placed).run_id == create_run_record(submission).run_id +def test_repair_window_is_durable_and_preserves_old_records() -> None: + original = _submission() + repair = replace( + original, + trigger=replace( + original.trigger, + repair_window=GraphRepairWindow(start_date=date(2026, 9, 1), end_date=date(2026, 9, 3)), + ), + ) + another_window = replace( + repair, + trigger=replace( + repair.trigger, + repair_window=GraphRepairWindow(start_date=date(2026, 9, 2), end_date=date(2026, 9, 3)), + ), + ) + record = create_run_record(repair) + + assert len({original.fingerprint, repair.fingerprint, another_window.fingerprint}) == 3 + assert deserialize_run_record(serialize_run_record(record)) == record + assert b'"repair_window"' not in serialize_run_record(create_run_record(original)) + assert ( + deserialize_run_record(serialize_run_record(create_run_record(original))).trigger + == original.trigger + ) + + +def test_non_cloud_run_repair_is_rejected_before_store_or_provider_effects() -> None: + submission = _submission( + trigger=RunTrigger( + kind=TriggerKind.API, + trigger_id="control-api", + repair_window=GraphRepairWindow(start_date=date(2026, 9, 1), end_date=date(2026, 9, 2)), + ) + ) + store = _FakeRunStore() + backend = _FakeBackend() + + with pytest.raises(OrchestrationContractError, match="single-task Cloud Run"): + dispatch_run_attempt(store, backend, submission, _plan(), now=NOW) + + assert not store.runs + + def test_transition_preserves_independent_outcome_results_and_cleanup_truth() -> None: queued = create_run_record(_submission()) attempt_id = attempt_identity(queued.run_id, 1) diff --git a/tests/control/test_run_explanation.py b/tests/control/test_run_explanation.py index 8a67cd2a..da4fd41e 100644 --- a/tests/control/test_run_explanation.py +++ b/tests/control/test_run_explanation.py @@ -7,10 +7,73 @@ from dander.control.models import RunState, RunStatusResponse, RunTelemetrySummary from dander.control.run_explanation import RunExplanationResponse, explain_run +from dander.pipeline.repair import GraphRepairWindow _RESULT_SCHEMA = "io.dander.control.execution-result-summary/v1" +def test_repair_explanation_preserves_window_and_distinguishes_replay_from_history() -> None: + result = explain_run( + RunStatusResponse( + run_id="repair", + state=RunState.SUCCEEDED, + result_schema=_RESULT_SCHEMA, + can_replay=True, + repair_window=GraphRepairWindow.model_validate( + { + "start_date": "2026-09-01", + "end_date": "2026-09-03", + } + ), + ) + ) + assert any( + "2026-09-01 included through 2026-09-03 excluded" in value for value in result.details + ) + assert any("does not extract" in value for value in result.caveats) + assert any("Replay repeats this repair window" in value for value in result.caveats) + assert not any("does not undo prior writes or repair only" in value for value in result.caveats) + assert "Output repair measurements have not been collected." in result.details + assert not any("0 affected rows" in value for value in result.details) + + +def test_repair_explains_output_telemetry_instead_of_zero_ingestion_counts() -> None: + result = explain_run( + RunStatusResponse( + run_id="measured-repair", + state=RunState.SUCCEEDED, + result_schema=_RESULT_SCHEMA, + affected=0, + models=1, + repair_window=GraphRepairWindow.model_validate( + {"start_date": "2026-09-01", "end_date": "2026-09-03"} + ), + telemetry=RunTelemetrySummary( + duration_ms=2500, + operation_count=2, + retry_count=0, + rows_read=0, + rows_written=1234, + rows_affected=2456, + bytes_read=0, + bytes_written=0, + bytes_processed=0, + bytes_billed=0, + queue_duration_ms=0, + execution_duration_ms=0, + spill_bytes=0, + ), + ) + ) + + assert any( + "1,234 rows written" in detail and "2,456 inserted/deleted row operations" in detail + for detail in result.details + ) + assert not any("0 affected rows" in detail for detail in result.details) + assert not any("measurements have not been collected" in detail for detail in result.details) + + def test_success_explains_collected_counts_without_claiming_a_data_diff() -> None: result = explain_run( RunStatusResponse( diff --git a/tests/control/test_run_lifecycle.py b/tests/control/test_run_lifecycle.py index 2ed6e63e..8fdd2910 100644 --- a/tests/control/test_run_lifecycle.py +++ b/tests/control/test_run_lifecycle.py @@ -6,7 +6,7 @@ import threading from concurrent.futures import ThreadPoolExecutor from dataclasses import dataclass, field, replace -from datetime import UTC, datetime, timedelta +from datetime import UTC, date, datetime, timedelta from typing import TYPE_CHECKING, cast import pytest @@ -93,10 +93,12 @@ PhysicalStage, serialize_physical_plan, ) +from dander.pipeline.repair import GraphRepairWindow from dander.providers.cloud_run import CloudRunBinding from dander.providers.dataproc_serverless import DataprocServerlessBinding from dander.providers.fargate import FargateBinding from dander.runtime_contract import RUNTIME_CONTRACT +from tests.pipeline.test_date_repair import repair_graph if TYPE_CHECKING: from collections.abc import Callable @@ -1140,6 +1142,49 @@ def test_replay_creates_a_new_durable_run_and_rejects_graph_drift() -> None: lifecycle.replay(RunAddress(source.run_id), idempotency_key="replay-key-0002") +def test_repair_replay_and_history_preserve_window_and_reject_normal_key_reuse() -> None: + graph_store, graph = _graph_store() + plan = _gcp_plan(graph) + store = _Store() + backend = _Backend() + lifecycle = _lifecycle(graph_store, plan, store, backend) + selection = PlanRunSubmissionResolver(ExecutionPlanRegistry((plan,)), "gcp") + original = selection.resolve(graph, idempotency_key="repair-key-0001", requested_at=NOW) + window = GraphRepairWindow(start_date=date(2026, 9, 1), end_date=date(2026, 9, 3)) + repair = replace(original, trigger=replace(original.trigger, repair_window=window)) + + assert selection.supports_repairs + assert selection.repair_environments(graph) == ("gcp",) + selection.require_repair_submission(repair) + source = lifecycle.start(repair) + assert source.repair_window == window + with pytest.raises(ControlOperationIdempotencyConflictError): + lifecycle.find_api_start( + graph, + environments=("gcp",), + placement_mode=PlacementMode.CONFIGURED_DEFAULT, + idempotency_key="repair-key-0001", + ) + handle = next(iter(backend.effects.values())) + backend.observations[handle.execution_id] = _terminal(RunOutcome.FAILED) + lifecycle.reconcile_once() + + replayed = lifecycle.replay(RunAddress(source.run_id), idempotency_key="repair-replay-0001") + + assert replayed.resulting_run_id is not None + assert store.runs[replayed.resulting_run_id].record.trigger.repair_window == window + assert store.runs[replayed.resulting_run_id].record.trigger.replay_of_run_id == source.run_id + assert lifecycle.get(RunAddress(replayed.resulting_run_id)).repair_window == window + + +def test_repair_resolver_excludes_non_cloud_run_routes() -> None: + _, graph = _graph_store() + resolver = PlanRunSubmissionResolver(ExecutionPlanRegistry((_plan(graph),)), "production") + + assert not resolver.supports_repairs + assert resolver.repair_environments(graph) == () + + def test_same_lifecycle_interface_selects_a_non_aws_backend_without_pipeline_changes() -> None: graph_store, graph = _graph_store() plan = _plan(graph, backend_id="gcp") @@ -1724,6 +1769,105 @@ def test_idempotent_retry_replays_before_mutable_metadata_is_reestimated() -> No ) +def test_repair_retry_preserves_sizing_and_rejects_changed_window_or_normal_key_reuse() -> None: + graph_store = InMemoryGraphStore() + graph = graph_store.create( + "demo", + "hosted-graph", + PipelineGraphDocument.from_domain(repair_graph()), + idempotency_key="repair-graph-key", + ) + small = _gcp_plan(graph) + large = replace( + small, + plan_id="gcp-bigquery-large", + execution_template=replace( + small.execution_template, + resources=replace(small.execution_template.resources, memory_mib=8192), + ), + ) + estimator = _InputEstimator(500) + store = _Store() + backend = _Backend() + composition = compose_run_control( + graph_store=graph_store, + store=cast("RunStore", store), + plans=(small, large), + backends={"cloud_run": cast("ExecutionBackend", backend)}, + environment="gcp", + size_class_candidates=( + SizeClassCandidate(small.revision, "small", 1_000), + SizeClassCandidate(large.revision, "large", 10_000), + ), + default_size_class="small", + input_size_estimators=(("gcp", estimator),), + start_reconciler=False, + clock=lambda: datetime.now(UTC) + timedelta(seconds=1), + ) + application = ControlApplication( + graph_store, + lifecycle=composition.lifecycle, + submission_resolver=composition.resolver, + projects=("demo",), + ) + window = GraphRepairWindow(start_date=date(2026, 9, 1), end_date=date(2026, 9, 3)) + first = application.start_run( + "demo", + "hosted-graph", + expected_revision=graph.revision, + idempotency_key="stable-repair-key", + repair_window=window, + environment="gcp", + ) + estimator.estimated_input_bytes = 5_000 + + replayed = application.start_run( + "demo", + "hosted-graph", + expected_revision=graph.revision, + idempotency_key="stable-repair-key", + repair_window=window, + environment="gcp", + ) + + assert replayed.run_id == first.run_id + assert store.runs[first.run_id].record.plan_revision == small.revision + assert len(backend.submissions) == 1 + assert estimator.calls == 1 + for changed_window in ( + None, + GraphRepairWindow(start_date=date(2026, 9, 1), end_date=date(2026, 9, 4)), + ): + with pytest.raises(ControlOperationIdempotencyConflictError): + application.start_run( + "demo", + "hosted-graph", + expected_revision=graph.revision, + idempotency_key="stable-repair-key", + repair_window=changed_window, + environment="gcp", + ) + assert estimator.calls == 1 + + application.start_run( + "demo", + "hosted-graph", + expected_revision=graph.revision, + idempotency_key="normal-run-key", + environment="gcp", + ) + with pytest.raises(ControlOperationIdempotencyConflictError): + application.start_run( + "demo", + "hosted-graph", + expected_revision=graph.revision, + idempotency_key="normal-run-key", + repair_window=window, + environment="gcp", + ) + assert estimator.calls == 2 + + def test_auto_retry_preserves_placement_and_rejects_cross_environment_key_collisions() -> None: graph_store, graph = _graph_store() fargate = _plan(graph) diff --git a/tests/fixtures/kubernetes-control-values.yaml b/tests/fixtures/kubernetes-control-values.yaml index e3572a8e..21c72c83 100644 --- a/tests/fixtures/kubernetes-control-values.yaml +++ b/tests/fixtures/kubernetes-control-values.yaml @@ -37,8 +37,8 @@ control: image: 127.0.0.1:5001/dander@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa druff: port: 8080 - bootstrap: '{"api_audience":"dander-kubernetes-control","api_url":"https://dander.127.0.0.1.sslip.io:8443","compatibility":{"maximum_druff_contract":"1.x","minimum_druff_contract":"1.0.0"},"contract":{"id":"io.dander.control.contracts/v1","sha256":"03fdac1a19a98f10f1af6fa7821b705291833f88a23e723300abace660130d5b"},"issuer":"https://issuer.127.0.0.1.sslip.io:8443/default","logout_uri":"https://dander.127.0.0.1.sslip.io:8443/signed-out","public_client_id":"druff-kubernetes-spa","redirect_uri":"https://dander.127.0.0.1.sslip.io:8443/auth/callback","schema_version":1}' - rolloutDigest: aa5f974008827a19774aa39b0ebf9bc953f947012db47ed3e8e72b65ad3f90b1 + bootstrap: '{"api_audience":"dander-kubernetes-control","api_url":"https://dander.127.0.0.1.sslip.io:8443","compatibility":{"maximum_druff_contract":"1.x","minimum_druff_contract":"1.0.0"},"contract":{"id":"io.dander.control.contracts/v1","sha256":"a28316b7e5158e0520fe1c24d59885083714f47b67aa396892fa9742060fb279"},"issuer":"https://issuer.127.0.0.1.sslip.io:8443/default","logout_uri":"https://dander.127.0.0.1.sslip.io:8443/signed-out","public_client_id":"druff-kubernetes-spa","redirect_uri":"https://dander.127.0.0.1.sslip.io:8443/auth/callback","schema_version":1}' + rolloutDigest: f5761bbe9132380e45a1cc48dfab2ba686eb47692a5f7dc90927b118cc576341 resources: requests: cpu: 100m diff --git a/tests/pipeline/test_date_repair.py b/tests/pipeline/test_date_repair.py new file mode 100644 index 00000000..08960e5f --- /dev/null +++ b/tests/pipeline/test_date_repair.py @@ -0,0 +1,437 @@ +"""Repair SQL executes against rows locally; live BigQuery qualification remains separate.""" + +from __future__ import annotations + +import re +import sqlite3 +from datetime import date, datetime +from pathlib import Path +from typing import TYPE_CHECKING, Any + +import pytest +from pydantic import ValidationError + +from dander.concurrency import FencingToken +from dander.ingestion import Endpoint, RawField, SourceConfig +from dander.pipeline.graph import PipelineGraph +from dander.pipeline.repair import GraphRepairError, GraphRepairWindow, plan_graph_repair +from dander.pipeline.runtime import GraphRuntimeError, plan_graph_execution +from dander.providers.bigquery.graph import BigQueryGraphRunner + +if TYPE_CHECKING: + from collections.abc import Sequence + + from dander.pipeline.runtime import GraphExecutionPlan + +_START = date(2026, 9, 1) +_END = date(2026, 9, 3) +_RAW = "unit-project.raw.records_events" +_TARGET = "unit-project.curated.events" +_LEASE = "unit-project.state.leases" + + +class _Ownership: + fence = FencingToken(lease_table=_LEASE, pipeline_id="repair", run_id="repair-run", token=1) + + def verify(self) -> None: + pass + + +def repair_graph() -> PipelineGraph: + fields = [ + {"name": "id", "type": "INT64"}, + {"name": "event_date", "type": "DATE"}, + {"name": "value", "type": "STRING"}, + ] + return PipelineGraph.model_validate( + { + "name": "repair_fixture", + "nodes": [ + { + "id": "source", + "name": "Events", + "type": "source", + "config": {"connector": "records", "endpoint": "events"}, + "fields": fields, + }, + { + "id": "target", + "name": "Events", + "type": "target", + "fields": fields, + "config": { + "writer": { + "write_mode": "replace", + "destination": { + "dataset": "curated", + "table": "events", + "business_key": ["id"], + }, + "partitioning": {"field": "event_date"}, + } + }, + }, + ], + "edges": [ + { + "from": "source", + "to": "target", + "mappings": [ + {"source": field["name"], "target": field["name"]} for field in fields + ], + } + ], + } + ) + + +def repair_source() -> SourceConfig: + return SourceConfig( + name="records", + base_url="https://unused.example.test", + auth_strategy="none", + endpoints=[ + Endpoint( + name="events", + path="/events", + primary_key=["id"], + raw_schema=[ + RawField(name="id", data_type="INT64"), + RawField(name="event_date", data_type="DATE"), + RawField(name="value", data_type="STRING"), + ], + ) + ], + ) + + +def _plan() -> GraphExecutionPlan: + return plan_graph_execution( + repair_graph(), + repair_source(), + project="unit-project", + dataset="raw", + repair_window=GraphRepairWindow(start_date=_START, end_date=_END), + ) + + +class _Job: + def __init__(self, rows: list[dict[str, int]] | None = None) -> None: + self.rows = rows or [] + + def result(self) -> object: + return self.rows + + +class _SqliteClient: + """Execute generated SELECT/ASSERT/DML with only dialect syntax translated. + + This exercises the actual row predicates, cross-window joins, and rollback boundaries. + It is deliberately not evidence of BigQuery dialect or provider behavior. + """ + + def __init__( + self, raw: Sequence[tuple[object, ...]], target: Sequence[tuple[object, ...]] + ) -> None: + self.db = sqlite3.connect(":memory:", isolation_level=None) + self.db.execute( + f"CREATE TABLE `{_LEASE}` (pipeline_id TEXT, run_id TEXT, fencing_token INTEGER, " + "lease_expires_at TEXT, heartbeat_at TEXT)" + ) + self.db.execute( + f"INSERT INTO `{_LEASE}` VALUES ('repair', 'repair-run', 1, '2099-01-01 00:00:00', '')" + ) + for table, rows in ((_RAW, raw), (_TARGET, target)): + self.db.execute(f"CREATE TABLE `{table}` (id INT64, event_date DATE, value STRING)") + self.db.executemany(f"INSERT INTO `{table}` VALUES (?, ?, ?)", rows) + self.queries: list[str] = [] + self.deleted: list[str] = [] + + def query(self, query: str, *, job_config: object | None = None) -> _Job: + parameters = { + parameter.name: parameter.value + for parameter in getattr(job_config, "query_parameters", []) + } + self.queries.append(query) + query = re.sub(r"OPTIONS \(expiration_timestamp=.*?\)\)\n", "", query) + query = re.sub(r"`([^`]+)`\.`([^`]+)`\.`([^`]+)`", r"`\1.\2.\3`", query) + query = re.sub(r"DATE '([0-9-]+)'", r"'\1'", query) + query = query.replace("CURRENT_TIMESTAMP()", "CURRENT_TIMESTAMP") + deleted = inserted = last_count = 0 + try: + for statement in query.split(";"): + statement = statement.strip() + if not statement or statement.startswith("DECLARE "): + continue + if statement.startswith("ASSERT "): + if statement == "ASSERT @@row_count = 1 AS 'Dander pipeline lease lost'": + if last_count != 1: + raise GraphRuntimeError("Dander pipeline lease lost") + continue + assertion = re.fullmatch( + r"ASSERT NOT EXISTS \((.*)\) AS '(.*)'", statement, re.S + ) + assert assertion is not None, statement + if self.db.execute(assertion[1]).fetchone() is not None: + raise GraphRuntimeError(assertion[2]) + elif statement.startswith("SET dander_repair_deleted"): + deleted = last_count + elif statement.startswith("SET dander_repair_inserted"): + inserted = last_count + elif statement.startswith("SELECT dander_repair_inserted"): + return _Job([{"rows_written": inserted, "rows_affected": deleted + inserted}]) + else: + # SQLite requires AS for DELETE aliases; BigQuery permits either spelling. + statement = re.sub( + r"(DELETE FROM `[^`]+`) t WHERE", r"\1 AS t WHERE", statement + ) + last_count = self.db.execute(statement, parameters).rowcount + except Exception: + if self.db.in_transaction: + self.db.rollback() + raise + return _Job() + + def delete_table(self, table: str, *, not_found_ok: bool = False) -> None: + assert not_found_ok + self.db.execute(f"DROP TABLE IF EXISTS `{table}`") + self.deleted.append(table) + + def rows(self, table: str = _TARGET) -> list[tuple[Any, ...]]: + return list(self.db.execute(f"SELECT * FROM `{table}` ORDER BY id")) + + +def test_repair_replaces_only_selected_dates_from_retained_raw_and_reports_real_counts() -> None: + raw = [ + (1, "2026-08-31", "raw-before"), + (2, "2026-09-01", "fixed"), + (4, "2026-09-03", "raw-end"), + ] + old = [ + (1, "2026-08-31", "keep-before"), + (2, "2026-09-01", "old"), + (3, "2026-09-02", "deleted"), + (4, "2026-09-03", "keep-end"), + ] + client = _SqliteClient(raw, old) + + result = BigQueryGraphRunner(plan=_plan(), project="unit-project", client=client).build( + Path("."), ownership=_Ownership() + ) + + assert client.rows() == [ + (1, "2026-08-31", "keep-before"), + (2, "2026-09-01", "fixed"), + (4, "2026-09-03", "keep-end"), + ] + assert client.rows(_RAW) == raw + assert sum(item.rows_written for item in result.telemetry) == 1 + assert sum(item.rows_affected for item in result.telemetry) == 3 + assert len(client.deleted) == 1 + + +@pytest.mark.parametrize( + ("raw", "target", "message"), + [ + ([(1, "2026-09-01", "new")], [(1, "2026-08-31", "old")], "overlap target rows"), + ([(1, "2026-09-03", "new")], [(1, "2026-09-01", "old")], "moved outside"), + ( + [(1, "2026-09-01", "a"), (1, "2026-09-02", "b")], + [(1, "2026-09-01", "old")], + "must be unique", + ), + ([(None, "2026-09-01", "new")], [(1, "2026-09-01", "old")], "must not be null"), + ([(1, None, "new")], [(1, "2026-09-01", "old")], "moved outside"), + ([(1, "2026-09-01", "new")], [(1, None, "old")], "overlap target rows"), + ], +) +def test_unsafe_window_repair_rolls_back_every_target_row_and_cleans_stage( + raw: list[tuple[object, ...]], + target: list[tuple[object, ...]], + message: str, +) -> None: + client = _SqliteClient(raw, target) + + with pytest.raises(GraphRuntimeError, match=message): + BigQueryGraphRunner(plan=_plan(), project="unit-project", client=client).build( + Path("."), ownership=_Ownership() + ) + + assert client.rows() == target + assert len(client.deleted) == 1 + + +def test_empty_selected_raw_window_deletes_only_selected_target_dates() -> None: + client = _SqliteClient([], [(1, "2026-09-01", "remove"), (2, "2026-09-03", "keep")]) + result = BigQueryGraphRunner(plan=_plan(), project="unit-project", client=client).build( + Path("."), ownership=_Ownership() + ) + assert client.rows() == [(2, "2026-09-03", "keep")] + assert sum(item.rows_written for item in result.telemetry) == 0 + assert sum(item.rows_affected for item in result.telemetry) == 1 + + +def test_unrelated_null_dates_null_keys_and_duplicate_keys_outside_window_are_preserved() -> None: + outside = [ + (None, "2026-08-31", "null-key"), + (1, None, "null-date"), + (3, "2026-09-03", "duplicate-a"), + (3, "2026-09-04", "duplicate-b"), + ] + client = _SqliteClient( + [(2, "2026-09-01", "fixed"), (5, None, "unused"), (5, None, "unused-duplicate")], + [*outside, (2, "2026-09-01", "old")], + ) + + BigQueryGraphRunner(plan=_plan(), project="unit-project", client=client).build( + Path("."), ownership=_Ownership() + ) + + assert client.rows() == [outside[0], outside[1], (2, "2026-09-01", "fixed"), *outside[2:]] + + +def test_repair_requires_fencing_and_rejects_a_stale_lease_transactionally() -> None: + original = [(1, "2026-09-01", "original")] + client = _SqliteClient([(1, "2026-09-01", "fixed")], original) + runner = BigQueryGraphRunner(plan=_plan(), project="unit-project", client=client) + with pytest.raises(GraphRuntimeError, match="lease fencing"): + runner.build(Path(".")) + assert client.queries == [] + + client.db.execute(f"UPDATE `{_LEASE}` SET fencing_token = 2") + with pytest.raises(GraphRuntimeError, match="lease lost"): + runner.build(Path("."), ownership=_Ownership()) + assert client.rows() == original + assert len(client.deleted) == 1 + + +def test_insert_failure_rolls_back_the_preceding_date_delete() -> None: + original = [(1, "2026-09-01", "original"), (2, "2026-09-03", "keep")] + client = _SqliteClient([(1, "2026-09-01", "reject")], []) + client.db.execute(f"DROP TABLE `{_TARGET}`") + client.db.execute( + f"CREATE TABLE `{_TARGET}` " + "(id INT64, event_date DATE, value STRING CHECK(value != 'reject'))" + ) + client.db.executemany(f"INSERT INTO `{_TARGET}` VALUES (?, ?, ?)", original) + + with pytest.raises(sqlite3.IntegrityError): + BigQueryGraphRunner(plan=_plan(), project="unit-project", client=client).build( + Path("."), ownership=_Ownership() + ) + + assert client.rows() == original + assert len(client.deleted) == 1 + + +def test_timestamp_selection_uses_utc_calendar_boundaries() -> None: + graph = repair_graph() + for node in graph.nodes: + node.fields[1].type = "TIMESTAMP" + source = repair_source() + source.endpoints[0].raw_schema[1].data_type = "TIMESTAMP" + plan = plan_graph_execution( + graph, + source, + project="unit-project", + dataset="raw", + repair_window=GraphRepairWindow(start_date=_START, end_date=_END), + ) + before = "2026-09-01T00:30:00+01:00" # August 31 in UTC. + inside = "2026-09-01T00:00:00Z" + end = "2026-09-03T00:00:00Z" + client = _SqliteClient( + [(1, before, "new-before"), (2, inside, "fixed"), (3, end, "new-end")], + [(1, before, "keep-before"), (2, inside, "old"), (3, end, "keep-end")], + ) + + BigQueryGraphRunner(plan=plan, project="unit-project", client=client).build( + Path("."), ownership=_Ownership() + ) + + assert client.rows() == [(1, before, "keep-before"), (2, inside, "fixed"), (3, end, "keep-end")] + + +@pytest.mark.parametrize("provider", ["postgresql", "redshift", "snowflake"]) +def test_other_graph_runners_reject_repair_before_constructing_provider_clients( + provider: str, +) -> None: + import importlib + + module = importlib.import_module(f"dander.providers.{provider}.transform") + runner = getattr( + module, + {"postgresql": "PostgreSQL", "redshift": "Redshift", "snowflake": "Snowflake"}[provider] + + "GraphRunner", + ) + kwargs: dict[str, object] = {"plan": _plan(), "database": "unused", "target_fence": None} + if provider == "postgresql": + kwargs.update(pool=None, timeouts=None) + else: + kwargs["connection_factory"] = None + if provider == "redshift": + kwargs["statement_timeout_ms"] = 10 + + with pytest.raises(GraphRuntimeError, match="unavailable"): + runner(**kwargs) + + +@pytest.mark.parametrize( + "invalid", + [ + "20260901", + "2026-9-01", + "2026-02-30", + "2026-09-01T00:00:00Z", + 1788220800, + datetime(2026, 9, 1), + ], +) +def test_repair_dates_are_strict_calendar_dates(invalid: object) -> None: + with pytest.raises(ValidationError): + GraphRepairWindow.model_validate({"start_date": invalid, "end_date": "2026-09-03"}) + + +@pytest.mark.parametrize("end", ["2026-09-01", "2026-08-31"]) +def test_repair_window_must_be_nonempty(end: str) -> None: + with pytest.raises(ValidationError, match="after start_date"): + GraphRepairWindow.model_validate({"start_date": "2026-09-01", "end_date": end}) + + +def test_repair_eligibility_requires_declared_partition_and_business_key() -> None: + graph = repair_graph() + window = GraphRepairWindow(start_date=_START, end_date=_END) + target = graph.nodes[-1] + payload = graph.model_dump(mode="json", by_alias=True) + payload["nodes"][-1]["config"]["writer"]["destination"]["business_key"] = [] + with pytest.raises(GraphRepairError, match="business-key"): + plan_graph_repair(PipelineGraph.model_validate(payload), window) + target.fields[1].type = "STRING" + with pytest.raises(GraphRepairError, match="DATE or TIMESTAMP"): + plan_graph_repair(graph, window) + + +@pytest.mark.parametrize("invalid_destination", ["raw_input", "duplicate_output"]) +def test_repair_rejects_destinations_that_could_overwrite_retained_inputs_or_each_other( + invalid_destination: str, +) -> None: + payload = repair_graph().model_dump(mode="json", by_alias=True) + if invalid_destination == "raw_input": + payload["nodes"][-1]["config"]["writer"]["destination"].update( + dataset="raw", table="records_events" + ) + message = "retained raw" + else: + duplicate = dict(payload["nodes"][-1], id="another_target") + payload["nodes"].append(duplicate) + payload["edges"].append(dict(payload["edges"][0], to="another_target")) + message = "distinct destinations" + + with pytest.raises(GraphRuntimeError, match=message): + plan_graph_execution( + PipelineGraph.model_validate(payload), + repair_source(), + project="unit-project", + dataset="raw", + repair_window=GraphRepairWindow(start_date=_START, end_date=_END), + ) diff --git a/tests/test_executor.py b/tests/test_executor.py index 122e0ef3..460d1e63 100644 --- a/tests/test_executor.py +++ b/tests/test_executor.py @@ -11,6 +11,7 @@ from dander.catalog import MetadataSnapshot, MetadataStore from dander.executor import PipelineExecutor from dander.ingestion import Endpoint, SourceConfig +from dander.pipeline.repair import GraphRepairWindow from dander.runtime import EndpointRunResult, PipelineRunResult from dander.state import ( LeaseHandle, @@ -103,6 +104,7 @@ def __init__(self) -> None: self.failure: tuple[str | None, str | None] | None = None self.reconciled: tuple[str, str] | None = None self.restarted = False + self.counts: tuple[int, int, int] | None = None def start(self, run_id: str, source: str, *, pipeline_id: str | None = None) -> None: assert pipeline_id is not None @@ -144,6 +146,7 @@ def finish( ) -> None: assert self.started is not None and run_id == self.started[0] self.finished = (status, failure_stage, models, assertions, assets) + self.counts = (endpoints, extracted, affected) self.failure = (failure_code, failure_summary) def reconcile_interrupted(self, pipeline_id: str, *, current_run_id: str) -> None: @@ -279,6 +282,44 @@ def _executor( ) +def test_date_repair_keeps_history_and_lease_without_ingestion_or_watermarks( + tmp_path: Path, +) -> None: + history = _History() + leases = _Leases(available=True) + executor = PipelineExecutor( + pipeline_id="example_pipeline", + source_config=SourceConfig( + name="example", + base_url="https://unused.example.test", + auth_strategy="none", + endpoints=[Endpoint(name="widgets", path="/widgets", primary_key=["id"])], + ), + ingestion=None, + history=history, + project="valid-project-123", + models_dir=tmp_path, + selected_models=("stg_widgets",), + build_models=True, + transform_runner=_Transform(), + leases=leases, + repair_window=GraphRepairWindow.model_validate( + {"start_date": "2026-09-01", "end_date": "2026-09-03"} + ), + ) + + result = executor.execute(run_id="repair-run") + + assert result.ingestion.endpoints == () + assert result.models == ("stg_widgets",) + assert history.counts == (0, 0, 0) + assert sum(item.rows_affected for item in result.telemetry.operations) == 3 + assert history.finished == (RunStatus.SUCCEEDED, None, 1, 2, 0) + assert history.checkpoints == [RunStage.TRANSFORM] + assert history.reconciled == ("example_pipeline", "repair-run") + assert leases.released + + def test_executor_records_one_truthful_complete_lifecycle( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, diff --git a/tickets/DANDER-287-date-scoped-output-repair.md b/tickets/DANDER-287-date-scoped-output-repair.md new file mode 100644 index 00000000..5467ee81 --- /dev/null +++ b/tickets/DANDER-287-date-scoped-output-repair.md @@ -0,0 +1,37 @@ +--- +id: DANDER-287 +title: Repair selected output dates from retained raw data +status: in-code +component: python +depends_on: [DANDER-286] +created: 2026-10-01 +--- + +## Context + +A full replay is too broad when an operator wants to rebuild a known date interval. Add that +bounded operation to the existing Control lifecycle and direct BigQuery graph runtime. + +## Acceptance Criteria + +- [x] Preview the exact saved graph, UTC interval, eligible routes, and publication limits. +- [x] Retain the window in durable identity, history, replay, and Cloud Run execution settings. +- [x] Skip source extraction and watermark changes; require the exact graph and lease fence. +- [x] Preserve surrounding rows, reject crossing keys, and publish atomically per output. +- [x] Reject incompatible backends and older runtimes before pipeline work. +- [x] Reconcile terminal Job cleanup without overwriting a newer deployment or launch. +- [x] Explain measured publication counts separately from the zero ingestion counts. +- [ ] Verify a bounded synthetic BigQuery and Cloud Run run from an immutable current artifact. +- [ ] Publish the paired contract artifact and integrate the guided Druff controls. + +## Verification and boundary + +Focused runtime, API, lifecycle, Cloud Run, explanation, and transaction tests passed. The full +local suite passed 2,429 tests and skipped 56; its 28 loopback-server failures all passed when +rerun with socket permission. The additional explanation test passed in its 17-test file. +Canonical strict typing passed across 516 files; Ruff and generated contract checks passed. + +Row-level tests execute the generated predicates through a narrow SQLite translation. This is +not live BigQuery proof. Current raw data is recompiled, so staging can scan outside the selected +dates. Existing outputs are required; earlier outputs remain published if a later output fails. +See [output repair](../docs/output-repair.md) for operator instructions and compatibility.