Skip to content

deployment-scoped artifacts and reporting epic - #551

Open
dannash100 wants to merge 29 commits into
mainfrom
epic/deployment-artefacts
Open

deployment-scoped artifacts and reporting epic#551
dannash100 wants to merge 29 commits into
mainfrom
epic/deployment-artefacts

Conversation

@dannash100

Copy link
Copy Markdown
Contributor

Epic branch for deployment-scoped artifacts. Implementation PRs merge here rather than into main.

  • Merges feat(platform): D2: spec deployment-scoped artefacts #529's spec onto current main. That branch sat 247 commits behind, and two of its links pointed at files main has since renamed (device-identity.md, servers/products.md).
  • Settles the review points still open on it: the digest is verified as the bytes arrive as well as when they are served, and artefact is spelled artifact to match the code.
  • First implementation branch is feat/group-scoped-artifacts.

@dannash100 dannash100 changed the title feat(platform): D2: deployment-scoped artifacts feat(platform): D2: deployment-scoped artifacts and reporting Sep 6, 2026
@dannash100 dannash100 changed the title feat(platform): D2: deployment-scoped artifacts and reporting deployment-scoped artifacts and reporting epic Sep 6, 2026
@dannash100

Copy link
Copy Markdown
Contributor Author

🤖 # What the reporting-schema pipeline needs from the dbt repos

Canopy now works out which pairs of group and Tamanu version need a reporting
schema, hands pgro a restored replica of that group's data migrated to that
version, and takes the built schema back as a group-scoped artifact it offers to
that group's servers. bestool reads what a server has, compares it against what
canopy offers, and applies the difference.

The one piece nobody owns yet is the build in the middle. pgro runs it as a
container against the replica it has just migrated, so build_reporting_assets.py
needs to be callable that way.

What pgro passes in

Environment on the build container:

Variable What it is
TAMANU_DL_DB_URL Host of the restored replica
TAMANU_DL_DB_USER, TAMANU_DL_DB_PASSWORD, TAMANU_DL_DB_DATABASE Its credentials and database
TAMANU_VERSION The Tamanu version the replica was migrated to, and the version to build for
TAMANU_DEPLOYMENT The group the snapshot came from
SCHEMA_CALLBACK_URL Where to POST the built SQL

The first four match the replica target already in each deployment repo's
config/profiles.yml, so the connection side needs no change beyond selecting
that target.

What it needs to change

  1. Take the version as an input. get_deployment_version()
    (scripts/utils/dbt_utils.py:82) reads the version out of dbt_project.yml, so a
    build is currently for whatever version the checkout sits at. Canopy names the
    version, and it will often not be the one the repo is on.

  2. Take the deployment as an input. get_deployment_name() reads it from the
    repo the same way. It is not cosmetic:
    generate_reporting_schema_script (scripts/utils/report_utils.py:299) excludes
    restricted nodes unless has_sensitive_facility is set, which is exactly the
    group configuration a schema follows from.

  3. Run without cutting a release. Following from the first two: the build has to
    run against an arbitrary version without create_version_branch.

  4. POST the result to SCHEMA_CALLBACK_URL. The build already writes
    compiled/v{VERSION}/reporting-schema-v{VERSION}-{DEPLOYMENT}.sql. pgro cannot
    read a file out of a finished container, and a Job's termination message is
    capped at 4 KiB, so the SQL comes back over HTTP.

  5. Stamp the version on the schema. The generated SQL does
    drop schema if exists reporting cascade; create schema reporting; and then
    creates views, with nothing recording which version it was built for. bestool
    needs to read that back to know whether a server is on the right schema.
    bestool currently reads it as a comment on the schema:

    COMMENT ON SCHEMA reporting IS '<version>';

    If a one-row table is preferred instead, that is fine, it just needs agreeing:
    it is a one-function change on bestool's side and a guess until then.

Separately, worth fixing

publish-artifacts.yml collapses v2.60.2 into v2.60.x, so a patch release
overwrites the previous bundle at the same key. Nothing is then left to grade
currency against. Not a blocker for the pipeline, but it undermines the figure
canopy shows.

The deployment repos also have no version bump path: neither
create-version-branch.yml nor tamanu-upgrade.yml.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant