Skip to content

feat: add multi_env_ml_deployment example (ACA-5889) - #125

Merged
p3ck merged 24 commits into
redhat-cop:mainfrom
p3ck:ACA-5889
Sep 25, 2026
Merged

p3ck merged 24 commits into
redhat-cop:mainfrom
p3ck:ACA-5889

Conversation

@p3ck

@p3ck p3ck commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds a new multi_env_ml_deployment example that automates a multi-environment (dev → staging → prod) Azure Machine Learning deployment workflow, including gated model promotion with an immutable audit trail (ACA-5889).

What's included

  • Role multi_env_ml_deployment — provisions shared infrastructure (ACR, shared services) and per-environment Azure ML workspaces/compute, promotes models across environments through an approval gate, records immutable audit events, and tears everything down cleanly.
  • Example playbook playbooks/multi_env_ml_deployment.yml + vars, registered in the README and playbook docs.
  • Docs — role README, playbook guide, AAP survey/workflow definition, and design + implementation planning docs.
  • Integration test azure_ops_test_multi_env_ml_deployment covering create/validate and teardown against live Azure.
  • Collection version bump in galaxy.yml.

Notable fixes folded in

  • Unique compute cluster name per environment (AmlCompute names must be unique within the RG).
  • Register the initial model from a local artifact rather than an empty datastore path.
  • Read the audit blob list from the correct return key (.blob).
  • Best-effort teardown using ignore_errors so partial failures don't block cleanup.

Testing

  • Integration test added under tests/integration/targets/azure_ops_test_multi_env_ml_deployment (runs against live Azure, self-contained resource group).

🤖 Generated with Claude Code

p3ck and others added 24 commits September 22, 2026 13:12
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
…l docs

- Move approval nodes BEFORE promote jobs in workflow graph (was after)
- Add azure_ml_menv_promotion_approved to survey (boolean, default false)
- Fix ML registry name description (max 28 chars, not 20)
- Correct resource name char counts to match actual derivations

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
- Make playbook operation-selectable via menv_operation (fixes AAP job template conflict)
- Stabilize correlation_id per run/invocation (freeze in playbook and tasks/main.yml)
- Audit rejected promotions before failing (validate_promotion_gate records promote_rejected)
- Improve audit blob uniqueness (add environment and random token to blob name)
- Soften audit trail immutability claims (tamper-evident vs tamper-proof; add WORM guidance)
- Carry source accuracy forward through promotion (gate reads real value)
- Fix tempdir cleanup leak when use_registry=false (move cleanup out of when block)
- Document DNS-safe resource group naming requirement
- Update design spec variable names to match shipped implementation
- Update docs to use menv_operation selector throughout

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Azure rejects duplicate managed compute names within a resource group
(ClusterNameAlreadyTaken), but azure_ml_menv_compute_name was the static
'training-cluster' for dev/staging/prod. Derive it per environment from the
env code (uniqueness across stages) plus a sanitized resource-group token
(uniqueness across deployments), within the 16-char AmlCompute name limit.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…path

The initial model registration pointed at
azureml://datastores/workspaceblobstore/paths/model, a datastore path with
no blobs, so Azure ML failed validation with ModelAssetPathNotFoundInStorage.
Register from a local tempdir artifact (uploaded by the module) instead,
mirroring the working promotion step. Fixed in both the playbook and the
integration test.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
azure_rm_storageblob_info returns the blob list under .blob, not .blobs,
so the assert crashed with 'object of type dict has no attribute blobs'.
Also tighten the search to '-promote-' so it counts successful promotions
without matching '-promote_rejected-' records.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
failed_when is not a valid attribute for include_role, so the teardown
task errored at parse time. Use ignore_errors: true, which is valid on
include_role and is inherited by the role's included tasks, to keep
cleanup best-effort.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
failed_when is not valid on include_tasks (TaskInclude), so recording the
environment-deletion audit event errored at parse time. Use ignore_errors
to keep the audit write best-effort during teardown. The failed_when: false
entries on the surrounding azure_rm_* module tasks are valid and unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Replace ignore_errors on the audit include in delete_environment with a
  block/rescue (failed_when does not suppress errors inside included tasks,
  hence the earlier ignore_errors; block/rescue is the lint-clean equivalent
  that actually works with include_tasks).
- Move mid-name Jinja to a single trailing expression in the promote_model
  task and the playbook Promote model task to satisfy name[template].

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
These are development-process artifacts (plan + design spec) generated
during development, not documentation for consumers of the collection.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
playbooks/vars/multi_env_ml_deployment_vars.yml re-declared nearly the
entire role defaults verbatim. Reduce it to only what the playbook needs
at play scope (the model name/version referenced by the register-model
block, which is not in role scope when that stage runs standalone), and
document that everything else comes from the role defaults. Same cleanup
applied to the other example playbooks in prior PRs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@p3ck
p3ck requested review from mikemorency and ronger4 September 25, 2026 16:27

@mikemorency mikemorency left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i skimmed all three of the markdown docs and theres some overlap. Might be worth fixing if you wind up pushing other changes

Comment thread galaxy.yml
namespace: cloud
name: azure_ops
version: 6.1.2
version: 6.2.0

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

did you mean to include this?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think AI is being too clever and noticing this is the last issue in my epic for this collection...

@p3ck
p3ck merged commit 51028ee into redhat-cop:main Sep 25, 2026
24 checks passed
@p3ck p3ck mentioned this pull request Sep 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants