Skip to content

[HDX-12246] migrate: support customer-required clusters (6.4) - #74

Merged
barottomartin merged 4 commits into
devfrom
HDX-12246/migrate-customer-support
Sep 10, 2026
Merged

[HDX-12246] migrate: support customer-required clusters (6.4)#74
barottomartin merged 4 commits into
devfrom
HDX-12246/migrate-customer-support

Conversation

@barottomartin

@barottomartin barottomartin commented Sep 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What

Retrofits the global hdxcli migrate command for turbine-api 6.4's multi-org breaking changes.

Since turbine-api 6.3 (HDX-11681), project creation requires an existing Customer on the target cluster, and tables may only reference storages/credentials that are members of the project's customer (HDX-11643). migrate previously forwarded the source project body verbatim (carrying the source cluster's customer UUID), which fails against 6.4 targets — exactly the break flagged on the ticket.

Changes

  • New migrate/customer.py
    • Detects whether the target requires a customer from the projects OPTIONS metadata, so pre-6.3 targets are untouched (no version checks).
    • Resolves the target customer from --target-customer (name or UUID) → an existing target project's immutable customer → an interactive picker; offers to create the customer when missing, mapping 401/403 to an actionable insufficient-permissions message.
    • Registers the migrated table's non-default storages and autoingest credentials with the customer (add_storage/add_credential), always behind a confirmation, running before any resource is created so a decline or 403 never leaves a half-migrated project.
  • resources.py — strip the source customer/org, inject the resolved customer on project create, run the membership pre-flight.
  • commands.py — add --target-customer / -tc and document it.
  • Bumps hdxcli to 1.0.84.

Scope

Per the ticket comment, this covers only the global migrate command, which is the only hdxcli command maintained going forward. The config-only project migrate (and the shared migrate_resource_config engine) and hdxcli project create have the same 6.3+ break; they are legacy commands and are intentionally not being updated. There is no follow-up planned for them.

Testing

  • 21 unit tests in tests/test_migrate_customer.py (resolution precedence, OPTIONS gating, collectors, default-candidate rule). Full suite green.
  • Live, cross-version:
    • 6.4 AIO (v6.3.0-rc.1-193): flag resolution, interactive picker, customer creation offer, existing-project inheritance, conflicting-flag warning, and the storage-membership pre-flight (decline aborts before any creation; accept runs add_storage and the table create passes validation).
    • Local docker on turbine v6.0.17 (genuine pre-customer release): old→new upgrade (v6.0 source → 6.4 target, customer resolved + injected) and the new→old regression guard (6.4 source → v6.0 target: "Target customer → Not required", all customer/membership logic skipped, project created as legacy).

Notes

  • CI: run_tests.yml now runs only the hermetic unit tests on push (pytest tests --ignore=tests/command_line_interface). The live-cluster acceptance suite covers the legacy CRUD commands, has no migrate cases, and on 6.x has known failures unrelated to this PR, so it moves to an opt-in workflow_dispatch job. actions/checkout and actions/setup-python are bumped to v7 (node24).
  • v1 nested URLs still work on 6.4 (no redirects/deprecation), so no URL changes were needed.
  • No PyPI publish is triggered by merge (repo has no publish workflow); the version bump is unspent until a manual tag + upload.

🤖 Generated with Claude Code

barottomartin and others added 2 commits September 2, 2026 12:58
@barottomartin @claude
[HDX-12246] migrate: support customer-required clusters (6.4)
03c415e 
Since turbine-api 6.3 (HDX-11681), project creation requires an existing
Customer on the target cluster, and tables may only reference storages and
credentials that are members of the project's customer (HDX-11643). The
`migrate` command previously forwarded the source project body verbatim,
which fails against 6.4 targets.

Add a customer-resolution and membership pre-flight step to the global
`migrate` command:

- New `migrate/customer.py`:
  - Detects whether the target requires a customer from the projects
    OPTIONS metadata, so pre-6.3 targets are untouched (no version checks).
  - Resolves the target customer from `--target-customer` (name or UUID),
    an existing target project's immutable customer, or an interactive
    picker; offers to create the customer when missing, mapping 401/403 to
    an actionable insufficient-permissions message.
  - Registers the migrated table's non-default storages and autoingest
    credentials with the customer (add_storage / add_credential), always
    behind a confirmation, running before any resource is created so a
    decline or 403 never leaves a half-migrated project.
- `resources.py`: strip the source customer/org, inject the resolved
  customer on project create, and run the membership pre-flight.
- `commands.py`: add `--target-customer` / `-tc` and document it.

Verified live across versions: 6.4 AIO (flag, picker, creation offer,
inheritance, conflicting-flag warning, storage membership accept/decline)
and a local v6.0.17 pre-customer cluster (old->new upgrade injects the
customer; new->old skips all customer logic).

Bumps hdxcli to 1.0.84.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@barottomartin @claude
[HDX-12246] ci: run only hermetic unit tests on push
177392a 
Only the migrate command is actively maintained and the live-cluster
acceptance suite (tests/command_line_interface) has no migrate cases,
so pushes now run `pytest tests --ignore=tests/command_line_interface`
with no cluster secrets. The acceptance suite moves to an opt-in
workflow_dispatch job that still receives the HDXCLI_TESTS_CLUSTER_*
secrets.

Also bump actions/checkout and actions/setup-python to v7 (node24),
matching the runtime GitHub already forces on this workflow.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@hdx-heatherblairs
hdx-heatherblairs self-requested a review September 8, 2026 21:17
barottomartin and others added 2 commits September 9, 2026 14:25
@barottomartin @claude
[HDX-12246] migrate: address review — customer detection, creation, m…
9181e83 
…emberships

Follow-up fixes from code review:

- Customer feature detection now reads the projects OPTIONS `required` flag
  instead of mere field presence. Field-present-but-optional clusters
  (6.1-6.3.x, which auto-assign the default customer) no longer get pushed
  into the customer picker: with no --target-customer they are left to the
  server's default (previously this aborted a non-interactive run and forced
  interaction on a migration that used to just work). An explicit
  --target-customer is still honored on such clusters; the picker is only
  entered when the target actually requires a customer.
- Customer creation trusts the POST response body for the created uuid/name
  instead of re-reading the list by the typed name, which broke for names the
  server slugifies (e.g. spaces to dashes) — previously it errored after
  mutating the target. `find_customer` also matches the slugified form.
- ensure_storage_memberships / ensure_credential_memberships continue past a
  resource without a `customers` field instead of returning, so one such
  resource no longer abandons registration of the rest.
- Strip `hdx_deployment_id` from the copied project body: some cluster
  versions (e.g. released 6.3.x) expose it as writable in the projects
  metadata, so the adapter would copy the source's value and the create fails
  as a duplicate. The target derives its own.

Verified live on released v6.3.1 (optional customer field): a full
`migrate --only resources` with no --target-customer completes
non-interactively, the target project gets its own deployment_id and the
server-assigned default customer.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@barottomartin @claude
[HDX-12246] migrate: tolerate missing add_storage/add_credential on 6…
dde238e 
….1-6.3

Cross-version review found that 6.1, 6.2 and 6.3 clusters expose a
`customers` field on storage/credential serializers but do not yet provide
the customer `add_storage`/`add_credential` actions (those arrive in 6.4
together with membership enforcement). So with --target-customer against such
a cluster, a non-member non-default storage or autoingest credential made
hdxcli POST to a nonexistent endpoint and abort on the raw 404.

Treat 404/405 from the membership-registration call as "the target manages
membership itself" and skip it. Correct on 6.1-6.3, which attach the
storage/credential to the customer server-side at table-create time and do
not enforce membership; harmless on 6.4+ where the action exists.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@barottomartin
barottomartin merged commit 8b81af6 into dev Sep 10, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@hdx-heatherblairs hdx-heatherblairs hdx-heatherblairs approved these changes

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@barottomartin @hdx-heatherblairs