Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,12 @@ capabilities enter through the next minor release.

## Unreleased

## 0.9.0rc33 — 2026-10-01 (integration candidate)

### Changed

- Curated discovery and the Salesforce starter pin compatible Salesforce `0.3.2` and ServiceNow
`0.2.3` packages, with their exact published minimums and a shared upper bound below Dander 0.10.
- The base installation and `postgres` extra no longer install Google/BigQuery SDKs. Install
`dander-platform[bigquery,gcp]` for the GCP stack or `dander-platform[runtime-all]` for every
provider. Runtime Dockerfiles already select `runtime-all`; public RC20 remains unchanged.
Expand All @@ -16,10 +20,22 @@ capabilities enter through the next minor release.

### Added

- Read-only graph change previews and deterministic explanations of recorded run outcomes.
- Date-scoped output repair from retained raw data for single-task Cloud Run and BigQuery,
preserving source extraction progress and rows outside the selected UTC date interval.
- Versioned preview, explanation, and repair contracts for the guided Druff integration.
- Let one always-on AWS-hosted Control select immutable AWS Fargate/Redshift or GCP Cloud
Run/BigQuery execution plans through the same run API and durable lifecycle, using keyless
AWS-to-Google workload identity while preserving direct single-container execution.

### Known limitations

- RC33 is a GitHub integration candidate for matching consumer contracts and qualification.
It does not promote the public PyPI beta beyond RC20 or close the Phase 8 support gates.
- Repair requires existing partitioned outputs with scalar business keys. Publication is atomic
per output; earlier outputs can remain committed if a later output fails. Retained raw data
is current data, not a historical snapshot. Live repair qualification remains outstanding.

## 0.9.0rc32 — 2026-08-23 (beta)

### Fixed
Expand Down
12 changes: 8 additions & 4 deletions HANDOFF.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
- Merged selected-date output repair from retained raw data with single-task Cloud Run and BigQuery through [PR #548](https://github.com/harrisonoconnorhover/dander/pull/548).
- Reused durable run identity, cancellation, history, replay, lease fencing, and conditional cleanup.
- Added preview/start contracts, publication measurements, and operator documentation with explicit support limits.
- Updated the curated catalog, Salesforce starter, and setup docs for Salesforce `0.3.2` and ServiceNow `0.2.3`; producer merge waits for verified publication.

## Try It

Expand All @@ -17,6 +18,9 @@ Use the API examples in `docs/control-contracts.md` and `docs/output-repair.md`.
- Strict typing passed across 516 files; Ruff, generated contracts, distribution installation, infrastructure validation, and container/secret scans passed.
- Exact-main CI passed for preview/explanation at `099fb52` and for repair at [`79aa1cb`](https://github.com/harrisonoconnorhover/dander/actions/runs/36849004513).
- No live repair workload or paid qualification ran.
- RC33 preparation: 4 release metadata tests, Ruff lint/format, metadata consistency, and contract drift checks passed; lockfile dependencies stayed fixed.
- CI exposed 5 historical Redshift fixture failures after the version bump. Fixtures now explicitly bind RC32; all 32 focused benchmark/metadata tests pass. Candidate CI must rerun.
- The corrected version candidate passed full protected CI. Catalog/starter changes passed 42 focused checks plus all 10 catalog cases, Ruff, strict typing, metadata, and unchanged regenerated contracts; final combined CI remains required.

## Decisions

Expand All @@ -26,10 +30,10 @@ Use the API examples in `docs/control-contracts.md` and `docs/output-repair.md`.

## Remaining

- Complete the bounded live repair check after the existing AWS billing session is restored; current aggregate cash exposure is unverified.
- Integrate Druff's guided flow once the requested repository exception is answered.
- Publish and consume the matching immutable contract/runtime artifacts; the complete guided journey is not delivered yet.
- Curated connector releases await their requested repository exception; advertised Salesforce/ServiceNow pins still target Dander 0.7.
- Complete current aggregate billing reconciliation and bounded live repair; AWS sign-in now works.
- Finish Druff's guided flow. The user approved its repository work and protected delivery.
- Publish and consume the verified RC33 GitHub integration candidate after protected merge and exact-main CI; public PyPI RC20 stays unchanged.
- Finish the approved Salesforce/ServiceNow compatibility releases; advertised pins still target Dander 0.7.

## Review First

Expand Down
10 changes: 10 additions & 0 deletions docs/connector-plugins.md
Original file line number Diff line number Diff line change
Expand Up @@ -165,6 +165,16 @@ CLI results include an exact public package pin, its compatible Dander range, su
and provider-validation status. Catalog API metadata also includes documentation links. The
catalog ships with Dander; PyPI remains the package store and source of the actual distribution.

The RC33 catalog pins Salesforce `0.3.2` (`dander-platform>=0.6.0rc1,<0.10`) and ServiceNow
`0.2.3` (`dander-platform>=0.5.0,<0.10`). Both accept the public Dander RC20 beta and RC33
integration candidate. Older Dander installations retain their bundled catalog; use these exact
pins in the project's `plugins` configuration, then run `dander plugins install` and
`dander validate` before rebuilding the runtime image. Installing a package alone does not
activate it or replace a deployed image.

Compatibility checks cover clean package installations and each connector's existing behavior
suite. They do not add a new live Salesforce or ServiceNow acceptance result.

When `dander graph serve` opens a project, `GET /v1/plugin-catalog` exposes the same non-secret
catalog and marks only validated, manifest-declared plugins as installed. `GET /v1/connectors`
continues to describe the runtime connectors that are actually active. Authoring tools may present
Expand Down
5 changes: 5 additions & 0 deletions docs/control-contracts.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,11 @@
Published client baseline: Dander `0.9.0rc19`. The current source bundle also contains unreleased
additions described below; their presence in this checkout does not publish a new client artifact.

RC33 prepares those additions for a GitHub integration prerelease. Its bundle SHA-256 is
`a28316b7e5158e0520fe1c24d59885083714f47b67aa396892fa9742060fb279`.
Consumers must wait for the verified release wheel and pin its separate SHA-256 as well as this
bundle digest. This candidate does not change the public PyPI beta or qualify a runtime profile.

Dander is the authority for data crossing the future Control API boundary. The deterministic
`io.dander.control.contracts/v1` bundle lives in `src/dander/control/contracts/v1` and is included
in the `dander-platform` wheel and source distribution. Druff must generate its client from a
Expand Down
10 changes: 10 additions & 0 deletions docs/decisions.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,15 @@
# Engineering Decisions

## 2026-10-01 — Publish a fixed integration candidate before consuming its contracts

- Publish RC33 as a canonical GitHub prerelease after protected merge, exact-main CI, and
distribution checks. Upload the wheel, source archive, and checksums to a draft, then publish
with repository release immutability enabled. Consumers pin the wheel and contract hashes.
- Keep public PyPI RC20 and its support documentation unchanged. The existing publication
metadata check still prevents promoting an unqualified candidate through the PyPI workflow.
- The user approved this task's Druff, Salesforce, and ServiceNow repository work, including
protected PRs, merges, and verified releases. Upstream WagnerJ-Dev repositories remain read-only.

## 2026-10-01 — Repair output dates from retained raw data

- Keep a repair window in the existing durable run trigger, preserving cancellation, replay,
Expand Down
5 changes: 3 additions & 2 deletions docs/release-audit.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,9 @@ cloud path remains outside the live proof.

## Current release and deployment record

- Public Dander beta: `0.9.0rc20`; public Salesforce connector: `0.3.1`; public ServiceNow connector:
`0.2.2`.
- Public Dander beta: `0.9.0rc20`; public Salesforce connector: `0.3.2`; public ServiceNow connector:
`0.2.3`. The October 1 connector releases widen package compatibility through Dander 0.9 with
clean-install behavior checks; the historical live evidence below keeps its recorded versions.
- Public Dander `0.9.0rc20` was published from protected-main commit
`75c5654e95439eaf18e90fbacc849799f4fe42b6` and immutable tag `v0.9.0rc20` by trusted-publishing
run `31815063258`. Its public wheel and source-distribution hashes matched the workflow
Expand Down
5 changes: 3 additions & 2 deletions docs/salesforce.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,14 +53,15 @@ add its governed projection to the relevant model. Undeclared fields fail before

## Project configuration

With `dander-connector-salesforce 0.3.1`, a Dander `0.7` project can use:
With `dander-connector-salesforce 0.3.2`, projects using Dander `>=0.6.0rc1,<0.10` can use the
following exact pin, including the public RC20 beta and RC33 integration candidate:

```yaml
version: 1
plugins:
salesforce:
distribution: dander-connector-salesforce
version: 0.3.1
version: 0.3.2
pipelines:
salesforce_crm:
source: salesforce
Expand Down
14 changes: 14 additions & 0 deletions docs/servicenow.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,20 @@ Dander's first ServiceNow slice reads the `incident` table through the official
deliberately read-only: the connector performs one OAuth token operation and one paged GET. The
simulator's create, update, and delete operations exist only to prepare synthetic acceptance data.

For the curated package, pin `dander-connector-servicenow==0.2.3` through the project's manifest:

```yaml
plugins:
servicenow:
distribution: dander-connector-servicenow
version: 0.2.3
```

This package accepts Dander `>=0.5.0,<0.10`, including the public RC20 beta and RC33 integration
candidate. Run `dander plugins install` and `dander validate`, then rebuild the runtime image.
See [connector discovery](connector-plugins.md#discover-curated-connectors) for the distinction
between package compatibility and live provider evidence.

The tracked contract is
[`contracts/servicenow-table-simulator.openapi.yaml`](../contracts/servicenow-table-simulator.openapi.yaml).
Its seven operations are:
Expand Down
8 changes: 6 additions & 2 deletions docs/session-resume.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,17 @@
# Session Resume — 2026-08-16

This is a dated operational snapshot. Use [current support status](support-status.md) and
`HANDOFF.md` for today's deployment and integration state, including paused schedules.

Read `HANDOFF.md`, `docs/decisions.md`, `docs/spec-alignment.md`, and
`docs/release-audit.md` before changing code or cloud resources.

## Public releases

- Dander `0.9.0rc20` is the current public beta.
- Salesforce `0.3.1` and ServiceNow `0.2.2` are the current stable connector releases for Dander
`0.7.x`; their accepted release candidates remain recorded in the Phase 1 evidence.
- Salesforce `0.3.2` and ServiceNow `0.2.3` are the current stable connector releases and accept
Dander 0.9 candidates. These compatibility releases preserve the existing runtime behavior;
accepted earlier provider runs remain recorded with their original versions in Phase 1 evidence.
- Druff's fork contains Josh's reconciled graph-client ancestry and the later persistence,
execution, catalog, operation-authoring, and deployment-preview work.

Expand Down
3 changes: 2 additions & 1 deletion docs/support-status.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ integrated or a named release/profile changes status.
| Revision or artifact | State | What it means |
|---|---|---|
| Public prerelease `0.9.0rc20` | Published August 14 | The package installed by the hosted quickstart; later main-source features are not automatically included. |
| Current source (`0.9.0rc32`) | Unreleased integration | Includes PostgreSQL Control, startup profiles, and later planning/Spark work. This is not a new public release. |
| Current source (`0.9.0rc33`) | Prepared GitHub integration candidate | Includes PostgreSQL Control, planning/Spark work, graph previews, run explanations, and scoped repair. Consumer integration requires the exact verified artifact; this does not promote the public PyPI beta. |
| Retained GCP private RC22 | Operator trial closed | The August 2–September 1 observation and final seven clean days passed; all five schedules were subsequently paused. |
| Local `codex/hdfs-enterprise-foundations` at `e187fdd` | Remaining enterprise implementation | PostgreSQL durability/startup has been integrated here. HDFS/YARN, Hive interfaces, semantics, locality, and enterprise packaging remain on the preserved branch. |

Expand All @@ -23,6 +23,7 @@ the final paused/no-drift state. These are dated observations, not a continuous
|---|---|---|
| GCP Cloud Run + BigQuery pipeline | Released named profile and retained operator evidence | GCP remains the supported compatibility baseline within the documented beta limits. |
| Direct single-container execution | Existing default path | Does not require hosted Control, Spark, Kubernetes, or Hadoop. |
| Curated connector packages | RC33 catalog pins Salesforce `0.3.2` and ServiceNow `0.2.3`; clean-install behavior checks cover the declared minimums, Dander `0.7.1`, and public RC20 | Compatibility metadata now accepts Dander 0.9 candidates. Existing provider evidence keeps its recorded versions; this is not a new live provider qualification. |
| Hosted Control API, graph persistence, OIDC | Implemented with named local/cloud acceptance records | Experimental; each profile keeps its own identity, storage, and lifecycle evidence. |
| Control execution through Fargate, Cloud Run, or Dataproc | Implemented with typed PostgreSQL or S3 run storage | Experimental; one Control process per run-store schema. AWS compatibility flags remain available. |
| Distributed Spark graphs | Bounded linear, keyed-join, and two-transform shapes implemented; retained August 27–28 runs include exact Fargate/Dataproc output parity | Experimental; the [execution history](control-execution-history.md) identifies each tested revision and bounded shape. |
Expand Down
2 changes: 1 addition & 1 deletion examples/salesforce/dander.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ platform:
plugins:
salesforce:
distribution: dander-connector-salesforce
version: 0.3.1
version: 0.3.2

pipelines:
salesforce_crm:
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[project]
name = "dander-platform"
version = "0.9.0rc32"
version = "0.9.0rc33"
description = "Opinionated, GCP-native, self-hosted data platform: ingest + transform + catalog behind one CLI."
readme = "README.md"
requires-python = ">=3.12"
Expand Down
12 changes: 6 additions & 6 deletions src/dander/plugins/catalog.py
Original file line number Diff line number Diff line change
Expand Up @@ -84,31 +84,31 @@ def as_dict(
"ingestion with replay cursors."
),
distribution="dander-connector-salesforce",
version="0.3.1",
dander_specifier=">=0.6.0,<0.8",
version="0.3.2",
dander_specifier=">=0.6.0rc1,<0.10",
support_status="first-party-beta",
validation_status="provider-validated",
documentation_url=(
"https://github.com/harrisonoconnorhover/"
"dander-connector-salesforce#dander-salesforce-connector"
),
pypi_url="https://pypi.org/project/dander-connector-salesforce/0.3.1/",
pypi_url="https://pypi.org/project/dander-connector-salesforce/0.3.2/",
repository_url=("https://github.com/harrisonoconnorhover/dander-connector-salesforce"),
),
CatalogConnector(
connector_id="servicenow",
display_name="ServiceNow",
description="Read-only ServiceNow Table API incident ingestion with stable paging.",
distribution="dander-connector-servicenow",
version="0.2.2",
dander_specifier=">=0.6.0,<0.8",
version="0.2.3",
dander_specifier=">=0.5.0,<0.10",
support_status="first-party-beta",
validation_status="provider-validated",
documentation_url=(
"https://github.com/harrisonoconnorhover/"
"dander-connector-servicenow#dander-servicenow-connector"
),
pypi_url="https://pypi.org/project/dander-connector-servicenow/0.2.2/",
pypi_url="https://pypi.org/project/dander-connector-servicenow/0.2.3/",
repository_url=("https://github.com/harrisonoconnorhover/dander-connector-servicenow"),
),
)
Expand Down
6 changes: 3 additions & 3 deletions tests/cli/test_plugins_cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -166,16 +166,16 @@ def test_plugins_search_lists_curated_exact_package_pins() -> None:
result = CliRunner().invoke(app, ["plugins", "search"])

assert result.exit_code == 0, result.output
assert "dander-connector-salesforce==0.3.1" in result.output
assert "dander-connector-servicenow==0.2.2" in result.output
assert "dander-connector-salesforce==0.3.2" in result.output
assert "dander-connector-servicenow==0.2.3" in result.output


def test_plugins_search_filters_and_reports_no_match() -> None:
filtered = CliRunner().invoke(app, ["plugins", "search", "incident"])
missing = CliRunner().invoke(app, ["plugins", "search", "not-a-connector"])

assert filtered.exit_code == 0, filtered.output
assert "dander-connector-servicenow==0.2.2" in filtered.output
assert "dander-connector-servicenow==0.2.3" in filtered.output
assert "dander-connector-salesforce" not in filtered.output
assert missing.exit_code == 0, missing.output
assert "No curated connectors match" in missing.output
33 changes: 24 additions & 9 deletions tests/plugins/test_catalog.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@

from typing import Any, cast

import pytest

from dander.plugins import (
CURATED_CONNECTORS,
ConnectorPlugin,
Expand All @@ -27,15 +29,19 @@ def _installed_salesforce(version: str = "0.2.0") -> InstalledConnectorPlugin:
)


def test_curated_catalog_has_exact_stable_pins_and_dander_07_compatibility() -> None:
catalog = build_plugin_catalog(dander_version="0.7.0rc1")
@pytest.mark.parametrize("version", ["0.7.0rc1", "0.7.1", "0.9.0rc20", "0.9.0rc33"])
def test_curated_catalog_has_exact_pins_and_current_dander_compatibility(version: str) -> None:
catalog = build_plugin_catalog(dander_version=version)
connectors = cast("list[dict[str, object]]", catalog["connectors"])

assert catalog["schema_version"] == 1
assert catalog["dander_version"] == "0.7.0rc1"
assert catalog["dander_version"] == version
assert {connector["id"] for connector in connectors} == {"salesforce", "servicenow"}
assert {connector["version"] for connector in connectors} == {"0.3.1", "0.2.2"}
assert all(connector["dander_specifier"] == ">=0.6.0,<0.8" for connector in connectors)
assert {connector["version"] for connector in connectors} == {"0.3.2", "0.2.3"}
assert {connector["id"]: connector["dander_specifier"] for connector in connectors} == {
"salesforce": ">=0.6.0rc1,<0.10",
"servicenow": ">=0.5.0,<0.10",
}
assert all(connector["compatible"] is True for connector in connectors)
assert all(connector["support_status"] == "first-party-beta" for connector in connectors)
assert all(connector["validation_status"] == "provider-validated" for connector in connectors)
Expand Down Expand Up @@ -68,14 +74,23 @@ def test_catalog_installation_status_uses_only_validated_manifest_plugins() -> N


def test_catalog_marks_unsupported_dander_version_incompatible() -> None:
catalog = build_plugin_catalog(dander_version="0.8.0")
catalog = build_plugin_catalog(dander_version="0.10.0")
connectors = cast("list[dict[str, object]]", catalog["connectors"])

assert all(connector["compatible"] is False for connector in connectors)


def test_catalog_marks_dander_05_incompatible_with_current_connectors() -> None:
catalog = build_plugin_catalog(dander_version="0.5.1")
@pytest.mark.parametrize(
("version", "salesforce", "servicenow"),
[("0.4.9", False, False), ("0.5.0", False, True), ("0.6.0rc1", True, True)],
)
def test_catalog_preserves_each_published_minimum(
version: str, salesforce: bool, servicenow: bool
) -> None:
catalog = build_plugin_catalog(dander_version=version)
connectors = cast("list[dict[str, object]]", catalog["connectors"])

assert all(connector["compatible"] is False for connector in connectors)
assert {connector["id"]: connector["compatible"] for connector in connectors} == {
"salesforce": salesforce,
"servicenow": servicenow,
}
Loading
Loading