diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/.itsm/azurelocal-itsm.yml b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/.itsm/azurelocal-itsm.yml index 5a174742..5a206160 100644 --- a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/.itsm/azurelocal-itsm.yml +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/.itsm/azurelocal-itsm.yml @@ -48,3 +48,21 @@ triggers: raiseTicket: false NotReady: raiseTicket: false + + # ----- Update: 4 - Monitor In-Flight Updates statuses (v0.8.87) ----- + # Emitted by Export-AzLocalUpdateRunMonitorReport in update-monitor.xml. + # Wire monitor-updates.yml with raise_itsm_ticket=true to open tickets for + # these. Dedupe key = SHA256(ClusterResourceId|UpdateName|TriggerCategory), + # so a stuck/attempt-gap cluster opens at most ONE incident until it clears. + AttemptWithoutRun: + raiseTicket: true + severity: 3 + category: "Update attempt with no observable updateRun" # UpdateLastAttempt tag set, but no updateRun materialised (commonly a URP package pre-install health-check failure) + StepError: + raiseTicket: true + severity: 2 + category: "Update step stuck in error" # progress.status == Error while state is still InProgress + LongRunningOverall: + raiseTicket: false # informational backstop; opt-in if you want a ticket per run exceeding the overall-elapsed threshold + LongRunningStep: + raiseTicket: false # informational; opt-in if you want a ticket per run whose current step exceeds the per-step threshold diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/README.md b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/README.md index 6a4cb5c0..18637ff6 100644 --- a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/README.md +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/README.md @@ -56,49 +56,50 @@ It is written in the same step-by-step style as [`ITSM/README.md`](../ITSM/READM By the end of this guide you will have: - A federated identity (no client secrets) wired into your CI/CD platform with the **minimum** Azure RBAC needed for cluster update management. -- Config/Fleet workflows committed to your repo and visible in the Actions / Pipelines UI: - - **Config: 01 - Validate Auth and Inventory Clusters** (GitHub) - merged auth + inventory flow with a clear setup-first summary (including collapsible subscription details) and cluster inventory export. - - **Config: 02 - Manage UpdateRing Tags** - bulk-apply `UpdateRing`, `UpdateStartWindow`, `UpdateExclusionsWindow`, `UpdateExcluded` tags from CSV. - - **Config: 03 - Apply-Updates Schedule Coverage Audit** - read-only audit that validates your schedule against `UpdateStartWindow` tags. - - **Fleet: 01 - Assess Update Readiness** - pre-flight readiness + blocking-health gate report. - - **Fleet: 02 - Fleet Connectivity Status** - Arc connectivity, NIC health, and ARB status snapshot. - - **Fleet: 03 - Sideload Updates (opt-in)** - on-prem media pre-stage workflow for disconnected environments. - - **Fleet: 04 - Apply Updates** - ring-scoped update execution with WhatIf support and schedule-aware gating. - - **Fleet: 05 - Monitor In-Flight Updates** - active-run progress and stuck-run diagnostics. - - **Fleet: 06 - Fleet Update Status** - daily fleet-wide update-state summary. - - **Fleet: 07 - Fleet Health Status** - daily 24-hour health-failure summary. +- Config/Monitor/Update workflows committed to your repo and visible in the Actions / Pipelines UI: + - **Config: 1 - Validate Auth and Inventory Clusters** (GitHub) - merged auth + inventory flow with a clear setup-first summary (including collapsible subscription details) and cluster inventory export. + - **Config: 2 - Manage UpdateRing Tags** - bulk-apply `UpdateRing`, `UpdateStartWindow`, `UpdateExclusionsWindow`, `UpdateExcluded` tags from CSV. + - **Config: 3 - Apply-Updates Schedule Coverage Audit** - read-only audit that validates your schedule against `UpdateStartWindow` tags. + - **Monitor: 1 - Fleet Connectivity Status** - Arc connectivity, NIC health, and ARB status snapshot. + - **Monitor: 2 - Fleet Health Status** - daily 24-hour health-failure summary. + - **Monitor: 3 - Fleet Update Status** - daily fleet-wide update-state summary. + - **Update: 1 - Assess Update Readiness** - pre-flight readiness + blocking-health gate report. + - **Update: 2 - Sideload Updates (opt-in)** - on-prem media pre-stage workflow for disconnected environments. + - **Update: 3 - Apply Updates** - ring-scoped update execution with WhatIf support and schedule-aware gating. + - **Update: 4 - Monitor In-Flight Updates** - active-run progress and stuck-run diagnostics. - An end-to-end "ring-based" rollout pattern: Pilot -> Wave2 -> Production, with each ring gated on the previous wave's success. - **Optional**: a ServiceNow integration that opens deduped incidents for clusters whose run status indicates the module's own retries cannot recover (failures, blocking health checks, sideloaded payload missing) - see [section 7](#7-optional-open-itsm-tickets-for-clusters-needing-operator-action). The pipelines are **fully opt-in additive layers** over the module. The PowerShell functions also work without any pipeline at all - see [section 10](#10-standalone-html-report-no-pipeline) for the ad-hoc / desktop story. -### 1.1 Why the pipelines are named `Config: NN` and `Fleet: NN` +### 1.1 Why the pipelines are named `Config: N`, `Monitor: N`, and `Update: N` -The active workflow model uses two clear groups: +The active workflow model uses three clear groups: -- `Config: 01-03` for onboarding and configuration. -- `Fleet: 01-07` for day-2 operational monitoring and update execution. +- `Config: 1-3` for onboarding and configuration. +- `Monitor: 1-3` for day-2 fleet monitoring (connectivity, update status, health). +- `Update: 1-4` for the update lifecycle (readiness, sideload, apply, in-flight monitor). | Group | Workflow name | GH Actions | Azure DevOps | |---|---|---|---| -| Config | Config: 01 - Validate Auth and Inventory Clusters | `setup-validate-and-inventory.yml` | `authentication-test.yml` + `inventory-clusters.yml` | -| Config | Config: 02 - Manage UpdateRing Tags | `manage-updatering-tags.yml` | `manage-updatering-tags.yml` | -| Config | Config: 03 - Apply-Updates Schedule Coverage Audit | `apply-updates-schedule-audit.yml` | `apply-updates-schedule-audit.yml` | -| Fleet | Fleet: 01 - Assess Update Readiness | `assess-update-readiness.yml` | `assess-update-readiness.yml` | -| Fleet | Fleet: 02 - Fleet Connectivity Status | `fleet-connectivity-status.yml` | `fleet-connectivity-status.yml` | -| Fleet | Fleet: 03 - Sideload Updates (Opt-in) | `sideload-updates.yml` | `sideload-updates.yml` | -| Fleet | Fleet: 04 - Apply Updates | `apply-updates.yml` | `apply-updates.yml` | -| Fleet | Fleet: 05 - Monitor In-Flight Updates | `monitor-updates.yml` | `monitor-updates.yml` | -| Fleet | Fleet: 06 - Fleet Update Status | `fleet-update-status.yml` | `fleet-update-status.yml` | -| Fleet | Fleet: 07 - Fleet Health Status | `fleet-health-status.yml` | `fleet-health-status.yml` | +| Config | Config: 1 - Validate Auth and Inventory Clusters | `setup-validate-and-inventory.yml` | `authentication-test.yml` + `inventory-clusters.yml` | +| Config | Config: 2 - Manage UpdateRing Tags | `manage-updatering-tags.yml` | `manage-updatering-tags.yml` | +| Config | Config: 3 - Apply-Updates Schedule Coverage Audit | `apply-updates-schedule-audit.yml` | `apply-updates-schedule-audit.yml` | +| Monitor | Monitor: 1 - Fleet Connectivity Status | `fleet-connectivity-status.yml` | `fleet-connectivity-status.yml` | +| Monitor | Monitor: 2 - Fleet Health Status | `fleet-health-status.yml` | `fleet-health-status.yml` | +| Monitor | Monitor: 3 - Fleet Update Status | `fleet-update-status.yml` | `fleet-update-status.yml` | +| Update | Update: 1 - Assess Update Readiness | `assess-update-readiness.yml` | `assess-update-readiness.yml` | +| Update | Update: 2 - Sideload Updates (Opt-in) | `sideload-updates.yml` | `sideload-updates.yml` | +| Update | Update: 3 - Apply Updates | `apply-updates.yml` | `apply-updates.yml` | +| Update | Update: 4 - Monitor In-Flight Updates | `monitor-updates.yml` | `monitor-updates.yml` | -- **GitHub Actions**: the Actions sidebar sorts workflows alphabetically by the `name:` field. Prefixing names with `Config: NN` and `Fleet: NN` keeps the sidebar in intended execution order (the `Config:` group sorts ahead of `Fleet:` so onboarding pipelines appear first). +- **GitHub Actions**: the Actions sidebar sorts workflows alphabetically by the `name:` field. Prefixing names with `Config: N`, `Monitor: N`, and `Update: N` keeps the sidebar grouped (the `Config:` onboarding group sorts first, then `Monitor:` day-2 reporting, then the `Update:` lifecycle pipelines). - ![GitHub Actions sidebar showing Config and Fleet workflows in execution order](../docs/images/github-actions-10-pipelines-view.png) + ![GitHub Actions sidebar showing Config, Monitor, and Update workflows grouped](../docs/images/github-actions-10-pipelines-view.png) - *The Config/Fleet numeric prefixes keep the GitHub Actions sidebar in practical execution order rather than a purely alphabetical scatter.* + *The Config/Monitor/Update numeric prefixes keep the GitHub Actions sidebar grouped by purpose rather than a purely alphabetical scatter.* -- **Azure DevOps**: the Pipelines list sorts by the pipeline **definition name** chosen at import time (not by filename). Use the same `Config: NN` / `Fleet: NN` naming when you import so the list stays in operational order. +- **Azure DevOps**: the Pipelines list sorts by the pipeline **definition name** chosen at import time (not by filename). Use the same `Config: N` / `Monitor: N` / `Update: N` naming when you import so the list stays grouped by purpose. If you prefer a different naming scheme (e.g. `00 - Auth`, `01 - Inventory`, ...), just change the `name:` field in each GH Actions YAML and / or pick a different prefix at ADO import time. Nothing else in the module depends on these display names. @@ -956,13 +957,13 @@ Both platforms expect the YAML files inside this folder to land in a platform-sp > Copy-AzLocalPipelineExample -Destination .\pipelines -Platform AzureDevOps > ``` > -> The function prints a short "next steps" summary pointing at the copied YAML location with the recommended workflow / pipeline to run first (GitHub: **Config: 01 - Validate Auth and Inventory Clusters**, Azure DevOps: auth validation + inventory onboarding). Supports `-Platform GitHub | AzureDevOps | All`, `-PassThru`, `-WhatIf`, `-Confirm`. +> The function prints a short "next steps" summary pointing at the copied YAML location with the recommended workflow / pipeline to run first (GitHub: **Config: 1 - Validate Auth and Inventory Clusters**, Azure DevOps: auth validation + inventory onboarding). Supports `-Platform GitHub | AzureDevOps | All`, `-PassThru`, `-WhatIf`, `-Confirm`. > > **Refusing to overwrite**: the function will refuse to overwrite any file that already exists in `-Destination`, listing the conflicts in the error message. To refresh after a module upgrade, delete the existing copies first (`Remove-Item .\.github\workflows\*.yml`) and re-run. ### 5.1 GitHub Actions -1. **Run Config: 01 first (strongly recommended).** Before exercising the Fleet workflows, validate that the App Registration, federated credentials, GitHub secrets, environments, and RBAC role assignment all line up - and capture the count + per-subscription detail of subscriptions visible to the pipeline identity - by running **`Config: 01 - Validate Auth and Inventory Clusters`**. This narrows any failure to one onboarding workflow instead of debugging multiple operational workflows simultaneously. **Re-run periodically** (recommended monthly, or after any RBAC change in the tenant) to confirm the pipeline identity's subscription scope has not silently widened or narrowed. +1. **Run Config: 1 first (strongly recommended).** Before exercising the Update and Monitor workflows, validate that the App Registration, federated credentials, GitHub secrets, environments, and RBAC role assignment all line up - and capture the count + per-subscription detail of subscriptions visible to the pipeline identity - by running **`Config: 1 - Validate Auth and Inventory Clusters`**. This narrows any failure to one onboarding workflow instead of debugging multiple operational workflows simultaneously. **Re-run periodically** (recommended monthly, or after any RBAC change in the tenant) to confirm the pipeline identity's subscription scope has not silently widened or narrowed. The onboarding workflow ships with the module at [`github-actions/setup-validate-and-inventory.yml`](./github-actions/setup-validate-and-inventory.yml). It emits the auth validation JUnit report (Authentication / Subscription Scope / Resource Graph Reachability), writes a setup-focused markdown summary (including a collapsible subscription-details section), and exports both auth and inventory artifacts for downstream workflows. @@ -987,8 +988,8 @@ Both platforms expect the YAML files inside this folder to land in a platform-sp At the `gh` CLI level, `gh run watch` shows the run summary as the steps complete (`gh` actually renders these as Unicode check marks; reproduced here in ASCII): ```text - ? Select a workflow run * Config: 01 - Validate Auth and Inventory Clusters, Config: 01 - Validate Auth and Inventory Clusters [main] 12s ago - [OK] main Config: 01 - Validate Auth and Inventory Clusters - + ? Select a workflow run * Config: 1 - Validate Auth and Inventory Clusters, Config: 1 - Validate Auth and Inventory Clusters [main] 12s ago + [OK] main Config: 1 - Validate Auth and Inventory Clusters - Triggered via workflow_dispatch less than a minute ago JOBS @@ -1001,7 +1002,7 @@ Both platforms expect the YAML files inside this folder to land in a platform-sp [OK] Post Azure login (OIDC) [OK] Complete job - [OK] Run Config: 01 - Validate Auth and Inventory Clusters () completed with 'success' + [OK] Run Config: 1 - Validate Auth and Inventory Clusters () completed with 'success' ``` Inside the `Collect Authentication and Subscription Scope Report` step, the run log shows: @@ -1022,7 +1023,7 @@ Both platforms expect the YAML files inside this folder to land in a platform-sp ``` - ![Config: 01 - Validate Auth and Inventory Clusters run, showing authentication validation output and subscription scope details](../docs/images/auth-smoke-test-validate-oidc.png) + ![Config: 1 - Validate Auth and Inventory Clusters run, showing authentication validation output and subscription scope details](../docs/images/auth-smoke-test-validate-oidc.png) You may see one informational `windows-latest` -> `windows-2025-vs2026` migration notice in the run annotations. The sample workflows pin `runs-on: windows-latest` (the module is a Windows-side PowerShell module), and GitHub will retarget the alias to the new image automatically when it becomes the default - no action required on your part. As of v0.7.60 the previously-seen Node.js 20 deprecation banner (against `actions/checkout@v4`, `azure/login@v2`, `actions/upload-artifact@v4`, `dorny/test-reporter@v1`) is gone: the sample workflows have been refreshed to Node 24-compatible majors (`@v5`, `@v3`, `@v6`, `@v3` respectively). @@ -1527,6 +1528,8 @@ Configure your CI/CD platform's alerting on the JUnit failures - GitHub Actions The connector reads the JUnit results the Apply Updates pipeline already publishes and, for each cluster whose status matches your configured trigger matrix (default: `Failed`, `Error`, `HealthCheckBlocked`, `SideloadedBlocked`), opens a deduped ServiceNow incident via the Table API. Idempotency is enforced via a SHA256 dedupe key written to a custom `u_azlocal_dedupe_key` column, so re-running the same workflow does not create duplicates. +> **v0.8.87: the Update: 4 in-flight monitor (`monitor-updates.yml`) now supports the same opt-in ITSM step.** Toggle `raise_itsm_ticket=true` (`raiseItsmTicket` on Azure DevOps) and it reads `./reports/update-monitor.xml` after publishing the JUnit. The monitor JUnit now emits per-testcase `ClusterResourceId` / `UpdateName` / `Status` properties (`Status` is `StepError` / `LongRunningStep` / `LongRunningOverall` / `InProgress` / `Failed` / `AttemptWithoutRun`); the sample matrix raises on `AttemptWithoutRun` + `StepError` and leaves `LongRunning*` opt-in. The monitor stays report-only and always green - ITSM failures never affect its result. To pick a poll cadence, the **Config: 3** schedule auditor now prints a "Recommended in-flight monitor schedule (Update: 4)" cron derived from your apply windows (`-MonitorFiresPerHour`, `-MonitorTrailingDays`). + This README does not duplicate the setup - it is a single-source-of-truth in [`../ITSM/README.md`](../ITSM/README.md). Here is the high-level wiring you'll do over there: > **Shortcut for getting the sample into your repo**: from the repo root, run @@ -1635,7 +1638,7 @@ Copy this file once per ring (e.g. `assess-update-readiness-Pilot.yml`, `assess- **Why one YAML per ring**: GitHub Actions `schedule:`-triggered runs cannot supply `inputs:` values - they always use the workflow's default `inputs:`. So a single `assess-update-readiness*.yml` with three crons in one `schedule:` block (and `update_ring` required, no default) would never actually run on cron - every cron tick would fail input validation. Splitting one YAML per ring (each with its own default + single cron) is the cleanest fix. Azure DevOps has the same constraint - `schedules:`-triggered runs use the YAML's default `parameters:` values, so the same per-ring split pattern applies. -**Known gap**: the Step.3 schedule-coverage audit (`apply-updates-schedule-audit.yml`) currently validates Step.7 cron-to-`UpdateStartWindow` coverage only - it does **not** audit whether each Step.5 cron is correctly anchored ahead of a Step.7 cron. **Always pair Step.5 + Step.7 cron edits in the same PR** so the lead-time relationship is reviewable by a human at merge time. The per-pipeline appendix entry for [Fleet: 01 - Assess Update Readiness](docs/appendix-pipelines.md#fleet-01---assess-update-readiness) repeats this guidance with the same lead-time table. +**Known gap**: the Step.3 schedule-coverage audit (`apply-updates-schedule-audit.yml`) currently validates Step.7 cron-to-`UpdateStartWindow` coverage only - it does **not** audit whether each Step.5 cron is correctly anchored ahead of a Step.7 cron. **Always pair Step.5 + Step.7 cron edits in the same PR** so the lead-time relationship is reviewable by a human at merge time. The per-pipeline appendix entry for [Update: 1 - Assess Update Readiness](docs/appendix-pipelines.md#update-1---assess-update-readiness) repeats this guidance with the same lead-time table. **Always-green caveat**: `assess-update-readiness.yml` never goes red at the pipeline level - per-cluster readiness gaps surface as JUnit `` entries in the Tests / Checks tab via `readiness.xml`. A silently-empty `readiness.xml` (e.g. an `update_ring` typo with zero clusters in scope) **will not generate a red-build email**. Either check the Tests tab after each scheduled run, or wire the JUnit reporter into the CI status surface you already monitor. @@ -2034,34 +2037,35 @@ Automation-Pipeline-Examples/ templates/ incident-body.md # - Mustache-style ticket body template. github-actions/ - setup-validate-and-inventory.yml # Config: 01. Auth + subscription-scope validation and cluster inventory (merged auth+inventory in v0.8.85; manual + weekly Sun 08:00 UTC). - manage-updatering-tags.yml # Config: 02. Apply UpdateRing / UpdateStartWindow / UpdateExclusionsWindow / UpdateExcluded tags (manual). - apply-updates-schedule-audit.yml # Config: 03. Weekly read-only audit: UpdateStartWindow tags vs apply-updates cron (Mon 05:00 UTC, v0.7.65). - assess-update-readiness.yml # Fleet: 01. Pre-flight readiness report (manual; v0.7.0). - fleet-connectivity-status.yml # Fleet: 02. Daily fleet connectivity / Arc / NIC / Resource Bridge snapshot + node-coverage reconciliation (daily 05:30 UTC, v0.7.79+; reconciliation enhanced in v0.7.85). - sideload-updates.yml # Fleet: 03. Opt-in self-hosted-runner workflow: Robocopy + WinRM sideload of solution-update media to clusters gated on UpdateSideloaded (manual; v0.8.7). - apply-updates.yml # Fleet: 04. Apply updates to one UpdateRing (with optional ITSM step, v0.7.4). - monitor-updates.yml # Fleet: 05. In-flight update monitor: per-cluster current step + elapsed duration; flags long-running runs (manual, optional cron; v0.7.90). - fleet-update-status.yml # Fleet: 06. Scheduled fleet update-status snapshot (daily 06:00 UTC). - fleet-health-status.yml # Fleet: 07. Scheduled fleet 24-hour health-check failure report (daily 07:00 UTC, v0.7.65). + apply-updates-schedule-audit.yml # Config: 3. Weekly read-only audit: UpdateStartWindow tags vs apply-updates cron (Mon 05:00 UTC, v0.7.65). + apply-updates.yml # Update: 3. Apply updates to one UpdateRing (with optional ITSM step, v0.7.4). + assess-update-readiness.yml # Update: 1. Pre-flight readiness report (manual; v0.7.0). + fleet-connectivity-status.yml # Monitor: 1. Daily fleet connectivity / Arc / NIC / Resource Bridge snapshot + node-coverage reconciliation (daily 05:30 UTC, v0.7.79+; reconciliation enhanced in v0.7.85). + fleet-health-status.yml # Monitor: 2. Scheduled fleet 24-hour health-check failure report (daily 07:00 UTC, v0.7.65). + fleet-update-status.yml # Monitor: 3. Scheduled fleet update-status snapshot (daily 06:00 UTC). + manage-updatering-tags.yml # Config: 2. Apply UpdateRing / UpdateStartWindow / UpdateExclusionsWindow / UpdateExcluded tags (manual). + monitor-updates.yml # Update: 4. In-flight update monitor: per-cluster current step + elapsed duration; flags long-running runs (manual, optional cron; v0.7.90). + setup-validate-and-inventory.yml # Config: 1. Auth + subscription-scope validation and cluster inventory (merged auth+inventory in v0.8.85; manual + weekly Sun 08:00 UTC). + sideload-updates.yml # Update: 2. Opt-in self-hosted-runner workflow: Robocopy + WinRM sideload of solution-update media to clusters gated on UpdateSideloaded (manual; v0.8.7). + azure-devops/ - setup-validate-and-inventory.yml - manage-updatering-tags.yml apply-updates-schedule-audit.yml + apply-updates.yml assess-update-readiness.yml fleet-connectivity-status.yml - sideload-updates.yml - apply-updates.yml - monitor-updates.yml - fleet-update-status.yml fleet-health-status.yml + fleet-update-status.yml + manage-updatering-tags.yml + monitor-updates.yml + setup-validate-and-inventory.yml + sideload-updates.yml ``` --- ## 14. Pipeline reference -Moved to [docs/appendix-pipelines.md](docs/appendix-pipelines.md) - one section per pipeline (`Config: 01 - ...` ... `Fleet: 07 - ...`) mapping 1:1 to the bundled `*.yml` workflows, with purpose, inputs, trigger, cmdlets invoked, dependencies, artefacts, RBAC, and exit conditions for each. Kept out-of-line to keep this README focused on the runbook. +Moved to [docs/appendix-pipelines.md](docs/appendix-pipelines.md) - one section per pipeline (`Config: 1 - ...` ... `Update: 4 - ...`) mapping 1:1 to the bundled `*.yml` workflows, with purpose, inputs, trigger, cmdlets invoked, dependencies, artefacts, RBAC, and exit conditions for each. Kept out-of-line to keep this README focused on the runbook. ## Appendix B: Release history diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/apply-updates-schedule.example.yml b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/apply-updates-schedule.example.yml index 80095dc3..98f0d7ae 100644 --- a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/apply-updates-schedule.example.yml +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/apply-updates-schedule.example.yml @@ -27,7 +27,7 @@ # A cron firing that lands on a day with NO matching schedule rows is # logged and exits 0 - no errors, no failures. # -# This file is REQUIRED for the "Fleet: 04 - Apply Updates" +# This file is REQUIRED for the "Update: 3 - Apply Updates" # (apply-updates.yml) pipeline when triggered by a schedule (not # workflow_dispatch). If the file is missing, apply-updates.yml throws # with the exact remediation command. diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/apply-updates-schedule-audit.yml b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/apply-updates-schedule-audit.yml index f3cae97b..3f1c0343 100644 --- a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/apply-updates-schedule-audit.yml +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/apply-updates-schedule-audit.yml @@ -1,5 +1,5 @@ # AZLOCAL-PIPELINE-ID: apply-updates-schedule-audit -# Config: 03 - Apply-Updates Schedule Coverage Audit Pipeline +# Config: 3 - Apply-Updates Schedule Coverage Audit Pipeline # This pipeline runs the read-only Test-AzLocalApplyUpdatesScheduleCoverage advisor # weekly to detect drift between the cron schedule(s) in your apply-updates pipeline # YAML and the UpdateStartWindow tags that operators have placed on Azure Local clusters. @@ -87,7 +87,7 @@ parameters: default: false variables: - GENERATED_AGAINST_MODULE_VERSION: '0.8.86' + GENERATED_AGAINST_MODULE_VERSION: '0.8.87' REQUIRED_MODULE_VERSION: '${{ parameters.moduleVersion }}' reportsPath: '$(Build.ArtifactStagingDirectory)/reports' # v0.8.7 sideload advisor defaults. Override at the pipeline / variable-group @@ -98,7 +98,7 @@ variables: stages: - stage: ScheduleCoverage - displayName: 'Config: 03 - Audit Apply-Updates Schedule Coverage' + displayName: 'Config: 3 - Audit Apply-Updates Schedule Coverage' jobs: - job: Audit displayName: 'Run Schedule Coverage Advisor' diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/apply-updates.yml b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/apply-updates.yml index 2ecb5cea..6b70a930 100644 --- a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/apply-updates.yml +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/apply-updates.yml @@ -1,5 +1,5 @@ # AZLOCAL-PIPELINE-ID: apply-updates -# Fleet: 04 - Apply Updates to Azure Local Clusters +# Update: 3 - Apply Updates to Azure Local Clusters # This pipeline applies updates to clusters filtered by UpdateRing tag value # Author guidance lives OUTSIDE the customise markers so module refreshes @@ -129,7 +129,7 @@ variables: # log if the YAML appears stale - prompting you to refresh via # Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5. - name: GENERATED_AGAINST_MODULE_VERSION - value: '0.8.86' + value: '0.8.87' # Resolution order for the module version pin (leave all unset to install the latest, # which is the default "fix-forward" behaviour): queue-time parameter > pipeline variable # 'REQUIRED_MODULE_VERSION' overridden at queue time > empty (latest). @@ -252,7 +252,7 @@ stages: # Stage 2: Apply Updates (with approval gate for production) - stage: ApplyUpdates - displayName: 'Fleet: 04 - Apply Updates' + displayName: 'Update: 3 - Apply Updates' dependsOn: CheckReadiness condition: and(succeeded(), gt(dependencies.CheckReadiness.outputs['ReadinessCheck.readiness.ReadyCount'], 0)) diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/assess-update-readiness.yml b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/assess-update-readiness.yml index d4458972..e7c41790 100644 --- a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/assess-update-readiness.yml +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/assess-update-readiness.yml @@ -1,5 +1,5 @@ # AZLOCAL-PIPELINE-ID: assess-update-readiness -# Fleet: 01 - Assess Update Readiness (Pre-flight go/no-go gate) +# Update: 1 - Assess Update Readiness (Pre-flight go/no-go gate) # -------------------------------------------------- # Runs Get-AzLocalClusterUpdateReadiness and Test-AzLocalClusterHealth -BlockingOnly # against a target UpdateRing (or the whole fleet) BEFORE apply-updates.yml runs. @@ -70,7 +70,7 @@ variables: # the version actually installed and to the latest on PSGallery, and emits a warning # log if the YAML appears stale - prompting you to refresh via # Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5. - GENERATED_AGAINST_MODULE_VERSION: '0.8.86' + GENERATED_AGAINST_MODULE_VERSION: '0.8.87' # Resolution order for the module version pin (leave all unset to install the latest, # which is the default "fix-forward" behaviour): queue-time parameter > pipeline variable # 'REQUIRED_MODULE_VERSION' overridden at queue time > empty (latest). @@ -82,7 +82,7 @@ pool: stages: - stage: Assess - displayName: 'Fleet: 01 - Assess Update Readiness' + displayName: 'Update: 1 - Assess Update Readiness' jobs: - job: AssessReadiness displayName: 'Readiness + blocking health gate' diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/fleet-connectivity-status.yml b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/fleet-connectivity-status.yml index 60fabc7c..dc772d32 100644 --- a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/fleet-connectivity-status.yml +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/fleet-connectivity-status.yml @@ -1,5 +1,5 @@ # AZLOCAL-PIPELINE-ID: fleet-connectivity-status -# Fleet: 02 - Fleet Connectivity Status Monitoring Pipeline +# Monitor: 1 - Fleet Connectivity Status Monitoring Pipeline # This pipeline surfaces network-connectivity issues across every Azure Local # cluster the service connection can read - four scopes in one pipeline: # @@ -49,7 +49,7 @@ # Pipeline display name carries the same Step.N - prefix as the filename so the # ADO pipelines list (sorts by `name:` field) lists pipelines in execution order. -name: 'Fleet: 02 - Fleet Connectivity Status' +name: 'Monitor: 1 - Fleet Connectivity Status' # BEGIN-AZLOCAL-CUSTOMIZE:schedule-triggers # Edits inside this block are preserved across Update-AzLocalPipelineExample @@ -110,7 +110,7 @@ variables: # the version actually installed and to the latest on PSGallery, and emits a warning # log if the YAML appears stale - prompting you to refresh via # Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5. - GENERATED_AGAINST_MODULE_VERSION: '0.8.86' + GENERATED_AGAINST_MODULE_VERSION: '0.8.87' # Resolution order for the module version pin (leave all unset to install the latest, # which is the default "fix-forward" behaviour): queue-time parameter > pipeline variable # 'REQUIRED_MODULE_VERSION' overridden at queue time > empty (latest). diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/fleet-health-status.yml b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/fleet-health-status.yml index 8e85f945..807ddb14 100644 --- a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/fleet-health-status.yml +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/fleet-health-status.yml @@ -1,5 +1,5 @@ # AZLOCAL-PIPELINE-ID: fleet-health-status -# Fleet: 07 - Fleet Health Status Monitoring Pipeline +# Monitor: 2 - Fleet Health Status Monitoring Pipeline # This pipeline surfaces 24-hour system health-check failures across every Azure Local # cluster the service connection can read - including clusters that are already # "up to date" with no available updates. The 24-hour health checks continue to run @@ -105,7 +105,7 @@ variables: # the version actually installed and to the latest on PSGallery, and emits a warning # log if the YAML appears stale - prompting you to refresh via # Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5. - GENERATED_AGAINST_MODULE_VERSION: '0.8.86' + GENERATED_AGAINST_MODULE_VERSION: '0.8.87' # Resolution order for the module version pin (leave all unset to install the latest, # which is the default "fix-forward" behaviour): queue-time parameter > pipeline variable # 'REQUIRED_MODULE_VERSION' overridden at queue time > empty (latest). diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/fleet-update-status.yml b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/fleet-update-status.yml index efb2b9e1..51d704b3 100644 --- a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/fleet-update-status.yml +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/fleet-update-status.yml @@ -1,5 +1,5 @@ # AZLOCAL-PIPELINE-ID: fleet-update-status -# Fleet: 06 - Fleet Update Status Monitoring Pipeline +# Monitor: 3 - Fleet Update Status Monitoring Pipeline # This pipeline monitors update status across all Azure Local clusters and generates reports # # USE CASES: @@ -93,7 +93,7 @@ variables: # the version actually installed and to the latest on PSGallery, and emits a warning # log if the YAML appears stale - prompting you to refresh via # Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5. - GENERATED_AGAINST_MODULE_VERSION: '0.8.86' + GENERATED_AGAINST_MODULE_VERSION: '0.8.87' # Resolution order for the module version pin (leave all unset to install the latest, # which is the default "fix-forward" behaviour): queue-time parameter > pipeline variable # 'REQUIRED_MODULE_VERSION' overridden at queue time > empty (latest). diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/manage-updatering-tags.yml b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/manage-updatering-tags.yml index 26140ab6..d0d7857f 100644 --- a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/manage-updatering-tags.yml +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/manage-updatering-tags.yml @@ -1,5 +1,5 @@ # AZLOCAL-PIPELINE-ID: manage-updatering-tags -# Config: 02 - Manage UpdateRing Tags +# Config: 2 - Manage UpdateRing Tags # This pipeline creates or updates UpdateRing tags on Azure Local clusters from a CSV file # # WORKFLOW: @@ -46,7 +46,7 @@ variables: # log if the YAML appears stale - prompting you to refresh via # Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5. - name: GENERATED_AGAINST_MODULE_VERSION - value: '0.8.86' + value: '0.8.87' # Resolution order for the module version pin (leave all unset to install the latest, # which is the default "fix-forward" behaviour): queue-time parameter > pipeline variable # 'REQUIRED_MODULE_VERSION' overridden at queue time > empty (latest). @@ -58,7 +58,7 @@ pool: stages: - stage: ManageTags - displayName: 'Config: 02 - Manage UpdateRing Tags' + displayName: 'Config: 2 - Manage UpdateRing Tags' jobs: - job: ApplyTags displayName: 'Apply UpdateRing Tags' diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/monitor-updates.yml b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/monitor-updates.yml index 2005dcc4..e85d5e28 100644 --- a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/monitor-updates.yml +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/monitor-updates.yml @@ -1,5 +1,5 @@ # AZLOCAL-PIPELINE-ID: monitor-updates -# Fleet: 05 - Monitor In-Flight Updates (Azure DevOps) +# Update: 4 - Monitor In-Flight Updates (Azure DevOps) # ----------------------------------------------------------- # Reports clusters whose latest update run is currently in flight, with the # CURRENT STEP each cluster is on and the ELAPSED DURATION of the run. Built @@ -83,10 +83,54 @@ parameters: type: string default: '' + # --- ITSM Connector (ServiceNow auto-raise on in-flight monitor failures) --- + # Set raiseItsmTicket=true to open ServiceNow incidents (v0.8.87) from each + # stuck-step / failed / attempt-without-run cluster surfaced in + # update-monitor.xml. Default false so existing schedules stay byte-identical + # until you opt in. The connector reads the JUnit file this pipeline already + # produces (update-monitor.xml) and the trigger matrix in + # ./.itsm/azurelocal-itsm.yml (defaults raise AttemptWithoutRun / StepError / + # Failed; LongRunningStep / LongRunningOverall are opt-in). + - name: raiseItsmTicket + displayName: 'Open ITSM tickets (ServiceNow) for in-flight monitor failures' + type: boolean + default: false + + - name: itsmConfigPath + displayName: 'Path to ITSM matrix config (YAML or JSON)' + type: string + default: './.itsm/azurelocal-itsm.yml' + + - name: itsmDryRun + displayName: 'ITSM: build payloads + run read-only dedupe but do NOT create tickets' + type: boolean + default: false + + - name: itsmForceCreate + displayName: 'ITSM: bypass dedupe and always create new tickets (use with caution)' + type: boolean + default: false + variables: - GENERATED_AGAINST_MODULE_VERSION: '0.8.86' + GENERATED_AGAINST_MODULE_VERSION: '0.8.87' REQUIRED_MODULE_VERSION: '${{ parameters.moduleVersion }}' reportsPath: '$(Build.ArtifactStagingDirectory)/reports' + # ITSM connector. When raiseItsmTicket=true the Raise ITSM task below + # references $(ITSM_SN_INSTANCE_URL), $(ITSM_SN_CLIENT_ID), + # $(ITSM_SN_CLIENT_SECRET). The simplest binding is an ADO variable group + # named 'AzureLocal-ITSM-Secrets'. ADO does not allow mixing mapping-form + # variables with '- group:' entries in the same block, so to enable the + # group reference convert this entire 'variables:' block to list form: + # variables: + # - name: GENERATED_AGAINST_MODULE_VERSION + # value: '0.8.87' + # - name: REQUIRED_MODULE_VERSION + # value: '${{ parameters.moduleVersion }}' + # - name: reportsPath + # value: '$(Build.ArtifactStagingDirectory)/reports' + # - group: AzureLocal-ITSM-Secrets + # When raiseItsmTicket=false the Raise ITSM task is skipped and the secrets + # are not consulted, so the conversion is only required for opt-in users. stages: - stage: MonitorInFlight @@ -187,3 +231,103 @@ stages: testRunTitle: 'In-Flight Update Monitor' mergeTestResults: false failTaskOnFailedTests: false + + # ---------------------------------------------------------------------- + # ITSM Connector (ServiceNow auto-raise on in-flight monitor failures) + # + # Fully opt-in (gated on parameters.raiseItsmTicket == true). Runs AFTER + # the JUnit file is published, so the ITSM task is strictly additive - + # failures here never affect the monitor result (which is always green; + # this is a report-only pipeline). Reads the same update-monitor.xml that + # powers the summary. The JUnit emitter writes per-testcase + # (ClusterName / ClusterResourceId / UpdateName / Status / CurrentStep / + # ClusterPortalUrl / UpdateRunPortalUrl) so New-AzLocalIncident can compute + # the SHA256 dedupe key (one incident per stuck/failed/attempt-gap cluster + # until it clears) and the Mustache body template can deep-link into the + # Azure portal. + # ---------------------------------------------------------------------- + - task: PowerShell@2 + displayName: 'Install powershell-yaml (ITSM config parser)' + condition: and(succeeded(), eq('${{ parameters.raiseItsmTicket }}', 'true')) + inputs: + targetType: 'inline' + pwsh: true + script: | + if (-not (Get-Module -ListAvailable -Name powershell-yaml)) { + Install-Module powershell-yaml -Scope CurrentUser -Force -AllowClobber + } + + - task: AzureCLI@2 + displayName: 'Raise ITSM tickets' + name: itsm + condition: and(succeeded(), eq('${{ parameters.raiseItsmTicket }}', 'true')) + env: + # BEGIN-AZLOCAL-CUSTOMIZE:itsm-secrets + # Bind your ITSM connector secrets from the variable group above. + # Defaults assume the ServiceNow OAuth client_credentials naming used + # by azurelocal-itsm.yml (env://ITSM_SN_*). If your secret names differ, + # update them here - this block is preserved by + # Update-AzLocalPipelineExample across module upgrades. + ITSM_SN_INSTANCE_URL: $(ITSM_SN_INSTANCE_URL) + ITSM_SN_CLIENT_ID: $(ITSM_SN_CLIENT_ID) + ITSM_SN_CLIENT_SECRET: $(ITSM_SN_CLIENT_SECRET) + # END-AZLOCAL-CUSTOMIZE:itsm-secrets + inputs: + azureSubscription: 'AzureLocal-ServiceConnection' # Update with your service connection name + scriptType: 'pscore' + scriptLocation: 'inlineScript' + inlineScript: | + Import-Module AzLocal.UpdateManagement -Force + + $configPath = "${{ parameters.itsmConfigPath }}" + $dryRun = [System.Convert]::ToBoolean("${{ parameters.itsmDryRun }}") + $force = [System.Convert]::ToBoolean("${{ parameters.itsmForceCreate }}") + + if (-not (Test-Path $configPath)) { + Write-Host "##vso[task.logissue type=warning]ITSM config not found at '$configPath' - skipping ticket creation." + exit 0 + } + + $cfg = Get-AzLocalItsmConfig -Path $configPath + + $junitInput = "$(reportsPath)/update-monitor.xml" + if (-not (Test-Path $junitInput)) { + Write-Host "##vso[task.logissue type=warning]No update-monitor.xml found at '$junitInput' - skipping ticket creation." + exit 0 + } + + $params = @{ + InputArtifactPath = $junitInput + Config = $cfg + RunMetadata = @{ + Platform = 'azure-devops' + RunId = $env:BUILD_BUILDID + RunUrl = "$($env:SYSTEM_COLLECTIONURI)$($env:SYSTEM_TEAMPROJECT)/_build/results?buildId=$($env:BUILD_BUILDID)" + Branch = $env:BUILD_SOURCEBRANCH + } + DryRun = $dryRun + ForceCreate = $force + ExportPath = "$(reportsPath)/itsm-results.csv" + ExportJUnitPath = "$(reportsPath)/itsm-results.xml" + } + + $results = New-AzLocalIncident @params + $results | Format-Table ClusterName, Action, TicketId, Severity -AutoSize + + - task: PublishBuildArtifacts@1 + displayName: 'Publish ITSM Artefacts' + condition: and(always(), eq('${{ parameters.raiseItsmTicket }}', 'true')) + inputs: + PathtoPublish: '$(reportsPath)' + ArtifactName: 'azlocal-step.8-update-monitor-itsm-results_$(stamp.artifactStamp)' + publishLocation: 'Container' + continueOnError: true + + - task: PublishTestResults@2 + displayName: 'Publish ITSM Test Results' + condition: and(always(), eq('${{ parameters.raiseItsmTicket }}', 'true')) + inputs: + testResultsFormat: 'JUnit' + testResultsFiles: '$(reportsPath)/itsm-results.xml' + testRunTitle: 'ITSM Tickets (Update: 4 Monitor) - $(Build.BuildNumber)' + failTaskOnFailedTests: false diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/setup-validate-and-inventory.yml b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/setup-validate-and-inventory.yml index 05944b11..80e5191c 100644 --- a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/setup-validate-and-inventory.yml +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/setup-validate-and-inventory.yml @@ -1,5 +1,5 @@ # AZLOCAL-PIPELINE-ID: setup-validate-and-inventory -# Config: 01 - Validate Auth and Inventory Azure Local Clusters +# Config: 1 - Validate Auth and Inventory Azure Local Clusters # # PURPOSE: # This is the first step in the setup workflow. It combines two operations: @@ -10,7 +10,7 @@ # UpdateRing tag status. # # Run this pipeline: -# - BEFORE importing the operational pipelines (Fleet: 01-07). It narrows +# - BEFORE importing the operational pipelines (Update: 1-4 and Monitor: 1-3). It narrows # any failure to authentication/discovery rather than seven interacting # pipelines. # - PERIODICALLY (recommended monthly, or after every RBAC change in the @@ -41,11 +41,11 @@ # cluster-inventory (ClusterUpdateRings.csv + .json + README_Instructions.txt). # # HOW TO RUN (after the file is committed and the pipeline is imported): -# Pipelines -> Config: 01 - Validate Auth and Inventory Clusters -> Run pipeline. +# Pipelines -> Config: 1 - Validate Auth and Inventory Clusters -> Run pipeline. # # See Automation-Pipeline-Examples/README.md, section 5, for the full setup story. -name: 'Config: 01 - Validate Auth and Inventory Clusters' +name: 'Config: 1 - Validate Auth and Inventory Clusters' # BEGIN-AZLOCAL-CUSTOMIZE:schedule-triggers # Add or modify `schedules:` blocks here to enable cron-driven runs. Content @@ -79,7 +79,7 @@ variables: # log if the YAML appears stale - prompting you to refresh via # Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5. - name: GENERATED_AGAINST_MODULE_VERSION - value: '0.8.86' + value: '0.8.87' # Resolution order for the module version pin (leave all unset to install the latest, # which is the default "fix-forward" behaviour): queue-time parameter > pipeline variable # 'REQUIRED_MODULE_VERSION' overridden at queue time > empty (latest). @@ -155,7 +155,7 @@ stages: inputs: testResultsFormat: 'JUnit' testResultsFiles: '$(Build.ArtifactStagingDirectory)/auth-report/auth-report.xml' - testRunTitle: '[JUnit Debug] Config: 01 - Authentication Validation' + testRunTitle: '[JUnit Debug] Config: 1 - Authentication Validation' failTaskOnFailedTests: false mergeTestResults: false diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/sideload-updates.yml b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/sideload-updates.yml index 2975fb5b..059eff6b 100644 --- a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/sideload-updates.yml +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/sideload-updates.yml @@ -1,5 +1,5 @@ # AZLOCAL-PIPELINE-ID: sideload-updates -# Fleet: 03 - Sideload Updates (Opt-in, on-prem self-hosted agent) +# Update: 2 - Sideload Updates (Opt-in, on-prem self-hosted agent) # # OPT-IN, OFF BY DEFAULT. This pipeline automates the on-prem sideloading of # Azure Local solution-update media for clusters that cannot pull updates from @@ -79,7 +79,7 @@ variables: # the version actually installed and to the latest on PSGallery, and emits a warning # log if the YAML appears stale - prompting you to refresh via # Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5. - GENERATED_AGAINST_MODULE_VERSION: '0.8.86' + GENERATED_AGAINST_MODULE_VERSION: '0.8.87' # Resolution order for the module version pin (leave all unset to install the latest, # which is the default "fix-forward" behaviour): queue-time parameter > pipeline variable # 'REQUIRED_MODULE_VERSION' overridden at queue time > empty (latest). diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/docs/appendix-pipelines.md b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/docs/appendix-pipelines.md index 2270ece2..9b759b3d 100644 --- a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/docs/appendix-pipelines.md +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/docs/appendix-pipelines.md @@ -2,7 +2,7 @@ > **What you will find here:** A comprehensive reference index for every bundled workflow (GitHub Actions + Azure DevOps twins), with a one-line summary of the job, its triggers, the cmdlets it invokes, what it depends on, and the artefacts it produces. Use this as the at-a-glance reference card after you have read the per-workflow runbook in the main pipeline [README.md](../README.md). -Each pipeline section heading below uses **`Config: NN`** or **`Fleet: NN`** naming to match the operator-facing workflow names. These identifiers appear in the workflow `name:` fields (GH Actions), ADO pipeline import names, the parent README's section 1.1 table, and these section headings. The execution order in the [end-to-end runbook](../README.md#6-end-to-end-runbook-bring-an-estate-online) is operational - read it for the recommended sequence. +Each pipeline section heading below uses **`Config: N`**, **`Monitor: N`**, or **`Update: N`** naming to match the operator-facing workflow names. These identifiers appear in the workflow `name:` fields (GH Actions), ADO pipeline import names, the parent README's section 1.1 table, and these section headings. The execution order in the [end-to-end runbook](../README.md#6-end-to-end-runbook-bring-an-estate-online) is operational - read it for the recommended sequence. --- @@ -12,26 +12,26 @@ The table below is the ground truth for what each shipped YAML does **out of the | Pipeline | GitHub Actions trigger | Azure DevOps trigger | Notes | |---|---|---|---| -| **Config: 01** - Validate Auth and Inventory Clusters | `workflow_dispatch` + weekly `schedule: cron '0 8 * * 0'` (Sun 08:00 UTC) | `trigger: none` + weekly `schedules: cron '0 8 * * 0'` (Sun 08:00 UTC) | Run on initial wiring + after every RBAC / federated-credential / subscription change. Ships a weekly Sunday 08:00 UTC re-validation + inventory refresh; edit or remove the cron in the `BEGIN/END-AZLOCAL-CUSTOMIZE:schedule-triggers` block to change the cadence. | -| **Config: 02** - Manage UpdateRing Tags | `workflow_dispatch` only | `trigger: none` (manual only) | Runs on-demand whenever you edit the CSV. | -| **Config: 03** - Apply-Updates Schedule Coverage Audit | `workflow_dispatch` + `schedule: cron '0 5 * * 1'` (Mondays 05:00 UTC) | `trigger: none` + `schedules: cron '0 5 * * 1'` | Weekly read-only drift advisor: compares apply-updates cron(s) to `UpdateStartWindow` tags. Runs before the daily fleet pipelines so its annotations are first on Monday mornings. | -| **Fleet: 01** - Assess Update Readiness | `workflow_dispatch` only | `trigger: none` (manual only) | **Recommended customisation** - schedule per ring 12-72h before the matching **Fleet: 04** `UpdateStartWindow` (see [Fleet: 01 - Assess Update Readiness](#fleet-01---assess-update-readiness) below for the per-ring pattern + lead-time table). Always-green at pipeline level; per-cluster readiness gaps surface as JUnit `` entries. | -| **Fleet: 02** - Fleet Connectivity Status | `workflow_dispatch` + `schedule: cron '30 5 * * *'` (daily 05:30 UTC) | `trigger: none` + `schedules: cron '30 5 * * *'` | Daily fleet connectivity / Arc / NIC / Resource Bridge snapshot with bidirectional node-coverage reconciliation. Runs 30 min before **Fleet: 06** so connectivity issues are visible upstream of update reporting. | -| **Fleet: 03** - Sideload Updates (Opt-in) | `workflow_dispatch` only | `trigger: none` (manual only) | Optional pre-stage on-prem update media workflow for disconnected environments. | -| **Fleet: 04** - Apply Updates | `workflow_dispatch` only | `trigger: none` (manual only) | **No schedule shipped** - see the warning in [Fleet: 04 - Apply Updates](#fleet-04---apply-updates) below. The cluster `UpdateStartWindow` / `UpdateExclusionsWindow` tags only gate updates *while the pipeline is running*; they do **not** start the pipeline. | -| **Fleet: 05** - Monitor In-Flight Updates | `workflow_dispatch` only (commented-out `schedule: cron '*/30 * * * *'` sample shipped) | `trigger: none` only (commented-out `schedules: cron '*/30 * * * *'` sample shipped) | In-flight update progress / long-running-run flag. Disabled by default to avoid 48 idle runs/day between waves. Uncomment the cron when a wave is active. | -| **Fleet: 06** - Fleet Update Status | `workflow_dispatch` + `schedule: cron '0 6 * * *'` (daily 06:00 UTC) | `trigger: none` + `schedules: cron '0 6 * * *'` | Daily fleet update snapshot. | -| **Fleet: 07** - Fleet Health Status | `workflow_dispatch` + `schedule: cron '0 7 * * *'` (daily 07:00 UTC) | `trigger: none` + `schedules: cron '0 7 * * *'` | Daily 24-hour health-check snapshot. Offset by one hour from **Fleet: 06** to avoid contention. | +| **Config: 1** - Validate Auth and Inventory Clusters | `workflow_dispatch` + weekly `schedule: cron '0 8 * * 0'` (Sun 08:00 UTC) | `trigger: none` + weekly `schedules: cron '0 8 * * 0'` (Sun 08:00 UTC) | Run on initial wiring + after every RBAC / federated-credential / subscription change. Ships a weekly Sunday 08:00 UTC re-validation + inventory refresh; edit or remove the cron in the `BEGIN/END-AZLOCAL-CUSTOMIZE:schedule-triggers` block to change the cadence. | +| **Config: 2** - Manage UpdateRing Tags | `workflow_dispatch` only | `trigger: none` (manual only) | Runs on-demand whenever you edit the CSV. | +| **Config: 3** - Apply-Updates Schedule Coverage Audit | `workflow_dispatch` + `schedule: cron '0 5 * * 1'` (Mondays 05:00 UTC) | `trigger: none` + `schedules: cron '0 5 * * 1'` | Weekly read-only drift advisor: compares apply-updates cron(s) to `UpdateStartWindow` tags. Runs before the daily fleet pipelines so its annotations are first on Monday mornings. | +| **Monitor: 1** - Fleet Connectivity Status | `workflow_dispatch` + `schedule: cron '30 5 * * *'` (daily 05:30 UTC) | `trigger: none` + `schedules: cron '30 5 * * *'` | Daily fleet connectivity / Arc / NIC / Resource Bridge snapshot with bidirectional node-coverage reconciliation. Runs 30 min before **Monitor: 3** so connectivity issues are visible upstream of update reporting. | +| **Monitor: 2** - Fleet Health Status | `workflow_dispatch` + `schedule: cron '0 7 * * *'` (daily 07:00 UTC) | `trigger: none` + `schedules: cron '0 7 * * *'` | Daily 24-hour health-check snapshot. Offset by one hour from **Monitor: 3** to avoid contention. | +| **Monitor: 3** - Fleet Update Status | `workflow_dispatch` + `schedule: cron '0 6 * * *'` (daily 06:00 UTC) | `trigger: none` + `schedules: cron '0 6 * * *'` | Daily fleet update snapshot. | +| **Update: 1** - Assess Update Readiness | `workflow_dispatch` only | `trigger: none` (manual only) | **Recommended customisation** - schedule per ring 12-72h before the matching **Update: 3** `UpdateStartWindow` (see [Update: 1 - Assess Update Readiness](#update-1---assess-update-readiness) below for the per-ring pattern + lead-time table). Always-green at pipeline level; per-cluster readiness gaps surface as JUnit `` entries. | +| **Update: 2** - Sideload Updates (Opt-in) | `workflow_dispatch` only | `trigger: none` (manual only) | Optional pre-stage on-prem update media workflow for disconnected environments. | +| **Update: 3** - Apply Updates | `workflow_dispatch` only | `trigger: none` (manual only) | **No schedule shipped** - see the warning in [Update: 3 - Apply Updates](#update-3---apply-updates) below. The cluster `UpdateStartWindow` / `UpdateExclusionsWindow` tags only gate updates *while the pipeline is running*; they do **not** start the pipeline. | +| **Update: 4** - Monitor In-Flight Updates | `workflow_dispatch` only (commented-out `schedule: cron '*/30 * * * *'` sample shipped) | `trigger: none` only (commented-out `schedules: cron '*/30 * * * *'` sample shipped) | In-flight update progress / long-running-run flag. Disabled by default to avoid 48 idle runs/day between waves. Uncomment the cron when a wave is active. | > **All times are UTC.** GitHub Actions and Azure DevOps schedules both run on UTC; convert from your local timezone when picking cron values. Both platforms can delay scheduled runs by several minutes during high-load periods - do not rely on second-precision alignment. > > **GitHub Actions only**: scheduled workflows are **automatically disabled after 60 days of repository inactivity** (no commits, PRs, or issue activity). Re-enable them via the Actions UI or run any push. Azure DevOps schedules do not have this auto-disable behaviour. -> **Common conventions for every row below**: `Cmdlets invoked` lists only AzLocal.UpdateManagement cmdlets the YAML calls directly (helper cmdlets they themselves call transitively are documented in each cmdlet's section of the module-level [`README.md`](../../README.md)). `Depends on` lists upstream pipelines whose output is consumed or whose work must be in place before this pipeline runs - `None` means the pipeline is self-contained. `Exit conditions` describes what a non-success YAML run state means in practice; per-cluster issues are always surfaced as JUnit `` entries rather than failing the whole pipeline unless noted otherwise. `ITSM` indicates whether the bundled YAMLs ship with the opt-in ServiceNow auto-raise connector (`Get-AzLocalItsmConfig` + per-failure ticket creation gated on the `raise_itsm_ticket` / `raiseItsmTicket` input); only Fleet: 02, Fleet: 04, Fleet: 06, and Fleet: 07 ship it - the matrix config lives at `./.itsm/azurelocal-itsm.yml` (see [`Automation-Pipeline-Examples/ITSM/`](../ITSM/) for the full recipe). +> **Common conventions for every row below**: `Cmdlets invoked` lists only AzLocal.UpdateManagement cmdlets the YAML calls directly (helper cmdlets they themselves call transitively are documented in each cmdlet's section of the module-level [`README.md`](../../README.md)). `Depends on` lists upstream pipelines whose output is consumed or whose work must be in place before this pipeline runs - `None` means the pipeline is self-contained. `Exit conditions` describes what a non-success YAML run state means in practice; per-cluster issues are always surfaced as JUnit `` entries rather than failing the whole pipeline unless noted otherwise. `ITSM` indicates whether the bundled YAMLs ship with the opt-in ServiceNow auto-raise connector (`Get-AzLocalItsmConfig` + per-failure ticket creation gated on the `raise_itsm_ticket` / `raiseItsmTicket` input); only Monitor: 1, Monitor: 2, Monitor: 3, and Update: 3 ship it - the matrix config lives at `./.itsm/azurelocal-itsm.yml` (see [`Automation-Pipeline-Examples/ITSM/`](../ITSM/) for the full recipe). --- -## Config: 01 - Validate Auth and Inventory Azure Local Clusters +## Config: 1 - Validate Auth and Inventory Azure Local Clusters | Aspect | Value | |---|---| @@ -45,13 +45,13 @@ The table below is the ground truth for what each shipped YAML does **out of the | **RBAC** | Whatever the pipeline identity has - the probe itself is read-only and intentionally surfaces both over- and under-grants. | | **Exit conditions** | Pipeline run is green only when every probe passes. A red run means the identity, its role assignment, or its subscription scope is wrong - downstream pipelines will give nonsense results until this passes. | | **ITSM** | Not supported - this is an identity-only probe with no per-cluster failure surface to dispatch on. | -| **Introduced** | v0.7.70 (separate auth + inventory); v0.8.85 (merged into Config: 01). | +| **Introduced** | v0.7.70 (separate auth + inventory); v0.8.85 (merged into Config: 1). | > **Note**: v0.8.85 consolidates the separate authentication and inventory pipelines into a single `setup-validate-and-inventory.yml` workflow. --- -## Config: 02 - Manage UpdateRing Tags +## Config: 2 - Manage UpdateRing Tags | Aspect | Value | |---|---| @@ -59,8 +59,8 @@ The table below is the ground truth for what each shipped YAML does **out of the | **Inputs** | `csv_path` (required). | | **Trigger** | Manual only (`workflow_dispatch` / **Run pipeline** button). No schedule shipped - this is a deliberate change-controlled operation that should follow a CSV edit + review. Add a `schedule:` / `schedules:` block if your CSV is auto-generated and you want periodic re-application. | | **Cmdlets invoked** | `Set-AzLocalClusterUpdateRingTag`. | -| **Depends on** | Config: 01 produces the input CSV (the operator typically downloads `cluster-inventory.csv`, edits ring / window / exclusion values in a PR, then re-uploads it as the input to this pipeline). | -| **Artefacts** | `UpdateRingTag_YYYYMMDD_HHmmss.csv` (per-cluster CSV log: ClusterName, ResourceGroup, SubscriptionId, ResourceId, Action, PreviousTagValue, NewTagValue, Status, Message) and `UpdateRingTag_Results.json` (the same rows in JSON form, written from `-PassThru`) inside the published log artifact. Since v0.8.0 the pipeline job summary tab also renders a **result breakdown table** (Created / Updated / Already in sync / Skipped / WhatIf / Failed) plus a collapsible **per-cluster details** block, at parity with Config: 03 / Fleet: 01 / Fleet: 04 / Fleet: 06. | +| **Depends on** | Config: 1 produces the input CSV (the operator typically downloads `cluster-inventory.csv`, edits ring / window / exclusion values in a PR, then re-uploads it as the input to this pipeline). | +| **Artefacts** | `UpdateRingTag_YYYYMMDD_HHmmss.csv` (per-cluster CSV log: ClusterName, ResourceGroup, SubscriptionId, ResourceId, Action, PreviousTagValue, NewTagValue, Status, Message) and `UpdateRingTag_Results.json` (the same rows in JSON form, written from `-PassThru`) inside the published log artifact. Since v0.8.0 the pipeline job summary tab also renders a **result breakdown table** (Created / Updated / Already in sync / Skipped / WhatIf / Failed) plus a collapsible **per-cluster details** block, at parity with Config: 3 / Monitor: 3 / Update: 1 / Update: 3. | | **When to run** | After editing the inventory CSV; whenever ring membership or maintenance windows change. | | **RBAC** | Write to tags only. The built-in **Tag Contributor** role on the cluster scope is sufficient (`Microsoft.Resources/tags/*`). Since v0.7.65, `Set-AzLocalClusterUpdateRingTag` writes via `Microsoft.Resources/tags/default` PATCH, so the broader `Microsoft.AzureStackHCI/clusters/write` is not required. | | **Exit conditions** | Pipeline run is green when every CSV row is processed (added / updated / unchanged). Per-cluster tag-write failures surface in the run log; the pipeline does not currently fail on per-row errors. Re-run after triaging is safe (the cmdlet is idempotent). | @@ -68,7 +68,7 @@ The table below is the ground truth for what each shipped YAML does **out of the --- -## Config: 03 - Apply-Updates Schedule Coverage Audit +## Config: 3 - Apply-Updates Schedule Coverage Audit | Aspect | Value | |---|---| @@ -76,7 +76,7 @@ The table below is the ground truth for what each shipped YAML does **out of the | **Inputs** | `pipeline_path` (file or folder; default `.github/workflows` on GitHub Actions, `.azure-pipelines` on Azure DevOps - the standard consumer locations for the bundled `apply-updates.yml` sample), `lead_time_minutes` (0-60, default 5), `include_untagged` (default false), `module_version` (optional). | | **Trigger** | Manual (`workflow_dispatch` / **Run pipeline** button) **plus** scheduled weekly on Mondays at 05:00 UTC (`cron '0 5 * * 1'`). Deliberately runs before the daily `fleet-connectivity-status` (05:30 UTC), `fleet-update-status` (06:00 UTC), and `fleet-health-status` (07:00 UTC) pipelines so its drift annotations land at the top of the operator's Monday-morning inbox. Edit the cron in the YAML to change cadence. | | **Cmdlets invoked** | `Test-AzLocalApplyUpdatesScheduleCoverage`, `Get-AzLocalApplyUpdatesScheduleConfig` (when a `schedule.yml` is present in the repo). | -| **Depends on** | Config: 01 (cluster inventory + `UpdateStartWindow` tags), Config: 02 (tags applied) for non-trivial output. Runs fine on an empty fleet but the audit is meaningless. | +| **Depends on** | Config: 1 (cluster inventory + `UpdateStartWindow` tags), Config: 2 (tags applied) for non-trivial output. Runs fine on an empty fleet but the audit is meaningless. | | **Artefacts** | `schedule-coverage-audit.xml` (JUnit, one `` per `(UpdateRing, UpdateStartWindow)` pair, uncovered = ``), `schedule-coverage-audit.csv` (full Audit view with `Status` / `Recommendation` columns), `schedule-coverage-matrix.csv` (every distinct `(Ring, Window)` pair with its required cron), `schedule-coverage-recommend.md` (ready-to-paste GH Actions + Azure DevOps cron blocks), markdown step summary. | | **When to run** | Hands-off scheduled. Trigger manually whenever you have just tagged a new ring or changed a maintenance window - see the [end-to-end runbook in section 8.3](../README.md#83-end-to-end-runbook-apply-updates-schedule-coverage-audit). | | **RBAC** | Read-only - `Reader` on the cluster scope plus `Microsoft.ResourceGraph/resources/read`. No write actions are ever taken. Covered by the `Azure Stack HCI Update Operator (custom)` custom role. | @@ -86,7 +86,7 @@ The table below is the ground truth for what each shipped YAML does **out of the --- -## Fleet: 02 - Fleet Connectivity Status +## Monitor: 1 - Fleet Connectivity Status | Aspect | Value | |---|---| @@ -94,9 +94,9 @@ The table below is the ground truth for what each shipped YAML does **out of the | **Inputs** | `subscription_ids` (optional - comma-separated; defaults to every subscription the federated identity can read), `module_version` (optional). v0.7.4-style ITSM toggles are also exposed: `raise_itsm_ticket`, `itsm_config_path`, `itsm_dry_run`, `itsm_force_create` (all optional, default off). | | **Trigger** | Manual (`workflow_dispatch` / **Run pipeline** button) **plus** scheduled daily at 05:30 UTC (`cron '30 5 * * *'`). Deliberately runs 30 minutes before `fleet-update-status` (06:00 UTC) so connectivity issues are visible upstream of update reporting. Edit the cron in the YAML to change cadence. | | **Cmdlets invoked** | `Get-AzLocalFleetConnectivityStatus`. When ITSM is enabled: `Get-AzLocalItsmConfig`. | -| **Depends on** | None directly. Config: 01 should be green at least once so the subscription scope is trusted. This is the upstream "can we see the fleet at all?" probe that other fleet pipelines silently rely on. | +| **Depends on** | None directly. Config: 1 should be green at least once so the subscription scope is trusted. This is the upstream "can we see the fleet at all?" probe that other fleet pipelines silently rely on. | | **Artefacts** | `fleet-connectivity-status.xml` (JUnit, one `` per cluster), `fleet-cluster-connectivity.csv` (per-cluster reconciliation), `fleet-arc-status-summary.csv` (per-cluster Arc agent counts), `fleet-arc-non-connected-machines.csv` (per-machine triage list), `fleet-physical-nics.csv` (NIC issues only), `fleet-physical-nic-all.csv` (full NIC inventory), `fleet-physical-nic-stats.csv` (histogram by `NicType` + `NicStatus`), `fleet-arb-status.csv` (Azure Resource Bridge state), markdown job summary including the *"How to interpret + act on a non-zero reconciliation"* guidance subsection. | -| **When to run** | Hands-off scheduled. Trigger manually whenever the apply-updates chain returns "zero ready clusters" or the fleet count drops unexpectedly - Fleet: 02 is the upstream "can we see the fleet at all?" probe. | +| **When to run** | Hands-off scheduled. Trigger manually whenever the apply-updates chain returns "zero ready clusters" or the fleet count drops unexpectedly - Monitor: 1 is the upstream "can we see the fleet at all?" probe. | | **RBAC** | Read-only - `Reader` plus `Microsoft.ResourceGraph/resources/read`, `Microsoft.AzureStackHCI/edgeDevices/read`, `Microsoft.HybridCompute/machines/read`, and `Microsoft.ResourceConnector/appliances/read`. All four already live in the **`Azure Stack HCI Update Operator (custom)`** custom role definition shipped in [section 3.1](../README.md#31-custom-role-azure-stack-hci-update-operator-custom). | | **Exit conditions** | Pipeline run is green when the snapshot completes. Per-cluster connectivity issues surface as JUnit `` entries; non-zero reconciliation deltas surface in the markdown summary with remediation guidance. | | **ITSM** | Supported (opt-in via `raise_itsm_ticket=true` / `raiseItsmTicket=true`). When enabled, one ServiceNow incident is raised per cluster with an Arc-disconnected machine or non-zero reconciliation delta; dedupe key is per-cluster + reason via the matrix at `./.itsm/azurelocal-itsm.yml`. `itsm_dry_run` builds payloads without creating tickets; `itsm_force_create` bypasses dedupe. | @@ -104,11 +104,11 @@ The table below is the ground truth for what each shipped YAML does **out of the --- -## Fleet: 01 - Assess Update Readiness +## Update: 1 - Assess Update Readiness > **Behaviour change in v0.8.81**: status icons in the Summary counts, Not-Ready and All-clusters tables now render via the new shared `Get-AzLocalStatusIconMap` private helper (host-aware: GitHub Markdown shortcodes on the `GitHubActions` host, Unicode glyphs everywhere else - fixes literal `:white_check_mark:` text on Azure DevOps step summaries). The Cluster column wraps in a portal deep-link via `Get-AzLocalClusterPortalLink` and the standing **Ctrl-click tip** banner is single-sourced via `Get-AzLocalCtrlClickTip`. The Up-to-Date bucket gains the readiness-cascade check icon. -> **Behaviour change in v0.8.74**: clusters that have already applied every required update are now classified as **Up to Date** (counted separately) instead of being lumped into the Not-Ready bucket. The summary's "Up to date" count is now accurate, the "Not-Ready clusters (review first)" table excludes Up-to-Date and Ready clusters, and the "All clusters detail" table shows a readable `Status` column in place of the old `Ready` boolean. Classification is shared with Fleet: 04 and Fleet: 06 via the `Get-AzLocalClusterReadinessStatus` helper. +> **Behaviour change in v0.8.74**: clusters that have already applied every required update are now classified as **Up to Date** (counted separately) instead of being lumped into the Not-Ready bucket. The summary's "Up to date" count is now accurate, the "Not-Ready clusters (review first)" table excludes Up-to-Date and Ready clusters, and the "All clusters detail" table shows a readable `Status` column in place of the old `Ready` boolean. Classification is shared with Update: 3 and Monitor: 3 via the `Get-AzLocalClusterReadinessStatus` helper. | Aspect | Value | |---|---| @@ -116,39 +116,39 @@ The table below is the ground truth for what each shipped YAML does **out of the | **Inputs** | `update_ring` (required), `throttle_limit` (optional). | | **Trigger** | **Manual only by default** (`workflow_dispatch` / **Run pipeline** button). **No schedule is shipped** because the `update_ring` input is required and there is no single ring value that would be correct for every consumer. See the **recommended customisation note below** for the per-ring scheduling pattern + lead-time guidance. | | **Cmdlets invoked** | `Get-AzLocalClusterInventory`, `Get-AzLocalClusterUpdateReadiness`, `Test-AzLocalClusterHealth`. | -| **Depends on** | Config: 01 (an `UpdateRing` tag must exist on at least one cluster to scope the readiness query) and Config: 02 (to apply that tag at scale). | +| **Depends on** | Config: 1 (an `UpdateRing` tag must exist on at least one cluster to scope the readiness query) and Config: 2 (to apply that tag at scale). | | **Artefacts** | `readiness.xml`, `readiness.csv`, `health-blocking.xml`, `health-blocking.csv`. | -| **When to run** | 12-72 hours before each Fleet: 04 wave for the same ring (12-24h for small rings, 48-72h for large rings of 50+ clusters) - long enough for an operator to triage any `` entries before the maintenance window opens, short enough that the readiness signal is still fresh. See the **recommended customisation note below** for the cron pattern. | +| **When to run** | 12-72 hours before each Update: 3 wave for the same ring (12-24h for small rings, 48-72h for large rings of 50+ clusters) - long enough for an operator to triage any `` entries before the maintenance window opens, short enough that the readiness signal is still fresh. See the **recommended customisation note below** for the cron pattern. | | **RBAC** | Read-only. Covered by the `Azure Stack HCI Update Operator (custom)` custom role. | | **Exit conditions** | Pipeline run is always green - per-cluster readiness gaps and blocking health checks surface as JUnit `` entries in the Tests tab. **Caveat**: because the pipeline never goes red, a silently-empty `readiness.xml` (e.g. ring tag typo, zero clusters in scope) will not generate a red email - check the Tests tab / `readiness.xml` artefact after each run, or wire the JUnit reporter into your existing CI status surface. | -| **ITSM** | Not supported - this is a pre-flight gate intended for operator review before the Fleet: 04 wave; readiness gaps surface as JUnit `` entries and feed the Fleet: 04 ITSM dispatch downstream rather than raising tickets here. | +| **ITSM** | Not supported - this is a pre-flight gate intended for operator review before the Update: 3 wave; readiness gaps surface as JUnit `` entries and feed the Update: 3 ITSM dispatch downstream rather than raising tickets here. | -> **RECOMMENDED CUSTOMISATION: schedule Fleet: 01 per ring, 12-72h ahead of the matching Fleet: 04 cron.** Unlike Fleet: 04 (which silently does nothing if you forget to schedule it), Fleet: 01 is *recommended* rather than *mandatory* - Fleet: 04 does its own internal `Get-AzLocalClusterUpdateReadiness` per-cluster pre-flight, so skipping Fleet: 01 will not break the apply step. What you lose by skipping it is the **human review window** - the chance to see "12 clusters in Ring2 have blocking health checks" and intervene before the maintenance window starts. +> **RECOMMENDED CUSTOMISATION: schedule Update: 1 per ring, 12-72h ahead of the matching Update: 3 cron.** Unlike Update: 3 (which silently does nothing if you forget to schedule it), Update: 1 is *recommended* rather than *mandatory* - Update: 3 does its own internal `Get-AzLocalClusterUpdateReadiness` per-cluster pre-flight, so skipping Update: 1 will not break the apply step. What you lose by skipping it is the **human review window** - the chance to see "12 clusters in Ring2 have blocking health checks" and intervene before the maintenance window starts. > -> Because `update_ring` is a required input there is no single bundled cron - **add one `schedule:` (GitHub Actions) / `schedules:` (Azure DevOps) entry per ring you intend to patch**, each setting `update_ring:` via the workflow inputs (GH Actions `with:` / ADO `parameters:`). Anchor each Fleet: 01 cron to the matching Fleet: 04 cron (same days, earlier by the lead time): +> Because `update_ring` is a required input there is no single bundled cron - **add one `schedule:` (GitHub Actions) / `schedules:` (Azure DevOps) entry per ring you intend to patch**, each setting `update_ring:` via the workflow inputs (GH Actions `with:` / ADO `parameters:`). Anchor each Update: 1 cron to the matching Update: 3 cron (same days, earlier by the lead time): > -> | Ring size | Recommended Fleet: 01 lead time | Fleet: 01 cron (UTC, anchoring on a Fleet: 04 Sat 02:00 window) | Rationale | +> | Ring size | Recommended Update: 1 lead time | Update: 1 cron (UTC, anchoring on a Update: 3 Sat 02:00 window) | Rationale | > |---|---|---|---| -> | Small (<= 10 clusters) | 12-24 hours ahead | `'0 2 * * 5'` (Fri 02:00 ahead of a Sat 02:00 Fleet: 04) | Enough to triage a handful of `` entries; fresh enough that the health signal is still meaningful at apply time. | -> | Medium (10-50 clusters) | 24-48 hours ahead | `'0 2 * * 4'` (Thu 02:00 ahead of a Sat 02:00 Fleet: 04) | Lets you raise tickets / engage cluster owners on per-cluster issues before the weekend. | -> | Large (50+ clusters) | 48-72 hours ahead | `'0 2 * * 3'` (Wed 02:00 ahead of a Sat 02:00 Fleet: 04) | Allows time to escalate, swap clusters into a deferral ring (`UpdateExcluded=True`), or stage parallel mitigation. | +> | Small (<= 10 clusters) | 12-24 hours ahead | `'0 2 * * 5'` (Fri 02:00 ahead of a Sat 02:00 Update: 3) | Enough to triage a handful of `` entries; fresh enough that the health signal is still meaningful at apply time. | +> | Medium (10-50 clusters) | 24-48 hours ahead | `'0 2 * * 4'` (Thu 02:00 ahead of a Sat 02:00 Update: 3) | Lets you raise tickets / engage cluster owners on per-cluster issues before the weekend. | +> | Large (50+ clusters) | 48-72 hours ahead | `'0 2 * * 3'` (Wed 02:00 ahead of a Sat 02:00 Update: 3) | Allows time to escalate, swap clusters into a deferral ring (`UpdateExcluded=True`), or stage parallel mitigation. | > -> **Known gap**: the Config: 03 schedule-coverage audit currently validates Fleet: 04 cron-to-`UpdateStartWindow` coverage only - it does **not** audit whether each Fleet: 01 cron is correctly anchored to a Fleet: 04 cron. Pair Fleet: 01 and Fleet: 04 cron edits in the same PR so the lead-time relationship is reviewable. The end-to-end runbook in the parent README's [section 8.1.1](../README.md#811-recommended-step5-pre-flight-schedule-per-ring) has worked examples for the most common ring layouts. +> **Known gap**: the Config: 3 schedule-coverage audit currently validates Update: 3 cron-to-`UpdateStartWindow` coverage only - it does **not** audit whether each Update: 1 cron is correctly anchored to a Update: 3 cron. Pair Update: 1 and Update: 3 cron edits in the same PR so the lead-time relationship is reviewable. The end-to-end runbook in the parent README's [section 8.1.1](../README.md#811-recommended-step5-pre-flight-schedule-per-ring) has worked examples for the most common ring layouts. -## Fleet: 03 - Sideload Updates (Opt-in) +## Update: 2 - Sideload Updates (Opt-in) -> **Behaviour change in v0.8.81 (Fleet: 04 Apply-Updates Readiness Gate)**: NOTE this callout describes the **Apply-Updates Readiness Gate** report (`Export-AzLocalClusterReadinessGateReport`), which is internal to the Fleet: 04 apply-updates pipeline (not the Sideload Updates pipeline documented in the table below). The gate report drops its inline icon switch in favour of the new shared `Get-AzLocalStatusIconMap` private helper (host-aware: fixes literal `:white_check_mark:` text on Azure DevOps step summaries); the Cluster column wraps in a portal deep-link via `Get-AzLocalClusterPortalLink`; and the standing Ctrl-click tip banner is single-sourced via `Get-AzLocalCtrlClickTip`. +> **Behaviour change in v0.8.81 (Update: 3 Apply-Updates Readiness Gate)**: NOTE this callout describes the **Apply-Updates Readiness Gate** report (`Export-AzLocalClusterReadinessGateReport`), which is internal to the Update: 3 apply-updates pipeline (not the Sideload Updates pipeline documented in the table below). The gate report drops its inline icon switch in favour of the new shared `Get-AzLocalStatusIconMap` private helper (host-aware: fixes literal `:white_check_mark:` text on Azure DevOps step summaries); the Cluster column wraps in a portal deep-link via `Get-AzLocalClusterPortalLink`; and the standing Ctrl-click tip banner is single-sourced via `Get-AzLocalCtrlClickTip`. > **OPT-IN, OFF BY DEFAULT, self-hosted runner required.** Introduced in v0.8.7. This pipeline is inert unless the repository variable `SIDELOAD_UPDATES` is the literal string `'true'`. Full architecture, the scheduled-task survival model, the shared-state / multi-runner contract, the auth-map, and the catalog format are in [sideload.md](sideload.md); robocopy throttling guidance is in [sideload-robocopy.md](sideload-robocopy.md). | Aspect | Value | |---|---| -| **Purpose** | Pre-stage Azure Local solution-update media onto clusters that **cannot pull updates from Azure directly** (dark / air-gapped / restricted-egress fabrics). It Robocopies the `CombinedSolutionBundle` (or OEM SBE package) to each cluster's infrastructure `import` SMB share, verifies the SHA256 over WinRM, runs `Add-SolutionUpdate`, and flips `UpdateSideloaded=True` so the downstream Fleet: 04 apply can proceed. | +| **Purpose** | Pre-stage Azure Local solution-update media onto clusters that **cannot pull updates from Azure directly** (dark / air-gapped / restricted-egress fabrics). It Robocopies the `CombinedSolutionBundle` (or OEM SBE package) to each cluster's infrastructure `import` SMB share, verifies the SHA256 over WinRM, runs `Add-SolutionUpdate`, and flips `UpdateSideloaded=True` so the downstream Update: 3 apply can proceed. | | **Inputs** | `update_ring` (optional - a single ring, a `;`-delimited list, or `***` for every tagged cluster; default `***`), `dry_run` (optional, default `true` - previews the plan + transitions with no staging / task / tag changes), `module_version` (optional pin). | | **Trigger** | **Manual only by default** (`workflow_dispatch` / **Run pipeline** button). **No schedule ships** because the pipeline requires an on-prem self-hosted runner labelled `azlocal-sideload` (GH) / a self-hosted agent in a pool with the `azlocal-sideload` demand (ADO) that most repos do not have. Once the runner/agent is online, uncomment the bundled `*/30 * * * *` cron inside the `BEGIN/END-AZLOCAL-CUSTOMIZE:schedule-triggers` block to drive the state machine. | | **Cmdlets invoked** | `Resolve-AzLocalSideloadPlan` (selects clusters whose next apply window is within `SIDELOAD_LEAD_DAYS`), `Invoke-AzLocalSideloadUpdate` (the re-entrant state machine), `Export-AzLocalSideloadStatusReport` + `Add-AzLocalSideloadStepSummary` (JUnit + Markdown summary). | | **Re-entrant state machine** | Each run advances every in-scope cluster by **one** transition and exits: `Planned -> Copying -> Copied -> Verified -> Imported -> SideloadFlagged`. The multi-hour copy itself runs in a **detached Windows Scheduled Task** (driven by `Tools/Invoke-AzLocalSideloadCopyTask.ps1`) so no pipeline run is ever long-lived. Re-running is always safe; a stale `Copying` heartbeat (> `SIDELOAD_HEARTBEAT_STALE_MINUTES`) is re-driven. | -| **Depends on** | Config: 01 (`UpdateRing` + sideload tags present, including `UpdateAuthAccountId`), Config: 02 (to apply those tags at scale), and a populated `sideload-auth-map.csv` + `sideload-catalog.yml`. The downstream Fleet: 04 apply consumes the `UpdateSideloaded=True` flag this pipeline sets. | +| **Depends on** | Config: 1 (`UpdateRing` + sideload tags present, including `UpdateAuthAccountId`), Config: 2 (to apply those tags at scale), and a populated `sideload-auth-map.csv` + `sideload-catalog.yml`. The downstream Update: 3 apply consumes the `UpdateSideloaded=True` flag this pipeline sets. | | **Artefacts** | `sideload-status.xml` (JUnit, one cluster per test), `sideload-status.csv`, plus per-run copy/verify logs under the shared `SIDELOAD_STATE_ROOT\logs`. | | **Shared state** | `SIDELOAD_STATE_ROOT` must be a UNC path that every runner/agent can read+write (`state\`, `logs\`, `cache\`). Verified media is cached under `SIDELOAD_CACHE_ROOT` (defaults to `\cache`) so a bundle is downloaded + hashed once and reused across clusters. | | **RBAC** | ARG fleet read + Key Vault secret read + `UpdateSideloaded` tag write for the pipeline identity; a separate Active Directory `[pscredential]` (built from two Key Vault secrets named in the auth-map row) is used for the cluster WinRM remoting and `Add-SolutionUpdate`. | @@ -157,13 +157,13 @@ The table below is the ground truth for what each shipped YAML does **out of the --- -## Fleet: 04 - Apply Updates +## Update: 3 - Apply Updates > **Behaviour change in v0.8.81**: the per-host apply-updates step summary (`Add-AzLocalApplyUpdatesStepSummary`) drops its inline `$iconSuccess` / `$iconFail` / `$iconBlock` / `$iconSkip` / `$iconWarn` definitions in favour of the new shared `Get-AzLocalStatusIconMap` private helper (fixes literal `:white_check_mark:` text on Azure DevOps step summaries). The Cluster column in both the Cluster Actions and Clusters Skipped at Readiness Gate tables now resolves each cluster's resource id from the readiness CSV (built upstream by `Get-AzLocalClusterUpdateReadiness`) and wraps the cell in a portal deep-link via `Get-AzLocalClusterPortalLink`. Adds the standing Ctrl-click tip via `Get-AzLocalCtrlClickTip`. The upstream apply-results JSON shape (`ClusterName` / `Status` / `UpdateName` / `Duration` / `Message`) is intentionally unchanged - the cluster id lookup is done in the renderer. -> **Behaviour change in v0.8.74**: the pre-apply readiness gate report (`Export-AzLocalClusterReadinessGateReport`) replaces its binary `Ready?` column with a readable `Status` column so fully-patched clusters now show **Up to Date** instead of a no-entry icon that implied failure. The header line shows a distinct `Up to Date` count and the step emits a new additive `UP_TO_DATE_COUNT` / `UpToDateCount` output (the `READY_COUNT` gate that drives the apply is unchanged). Classification is shared with Fleet: 01 and Fleet: 06 via the `Get-AzLocalClusterReadinessStatus` helper. When the gate finds no clusters ready, the "No Clusters Ready" summary (`Add-AzLocalNoReadyClustersStepSummary`) now renders an Up-to-Date vs Not-Ready breakdown - so an idle run makes clear that already-patched clusters are a healthy steady state (a notice, not a warning, when every cluster is up to date) rather than a fault. +> **Behaviour change in v0.8.74**: the pre-apply readiness gate report (`Export-AzLocalClusterReadinessGateReport`) replaces its binary `Ready?` column with a readable `Status` column so fully-patched clusters now show **Up to Date** instead of a no-entry icon that implied failure. The header line shows a distinct `Up to Date` count and the step emits a new additive `UP_TO_DATE_COUNT` / `UpToDateCount` output (the `READY_COUNT` gate that drives the apply is unchanged). Classification is shared with Update: 1 and Monitor: 3 via the `Get-AzLocalClusterReadinessStatus` helper. When the gate finds no clusters ready, the "No Clusters Ready" summary (`Add-AzLocalNoReadyClustersStepSummary`) now renders an Up-to-Date vs Not-Ready breakdown - so an idle run makes clear that already-patched clusters are a healthy steady state (a notice, not a warning, when every cluster is up to date) rather than a fault. -> **Behaviour change in v0.8.78**: per-cluster `ScheduleBlocked` / `SideloadedBlocked` / `ExcludedByTag` outcomes now render as JUnit `` instead of `` so `dorny/test-reporter` no longer flips the Step.07 check RED on by-design gate-respect outcomes - `HealthCheckBlocked` deliberately stays a `` because a critical health failure IS actionable. The Fleet: 04 Readiness KPI table also gains optional **Already Up to Date** and **Not Ready (needs attention before updating)** rows, wired in both shipped `apply-updates.yml` templates from the upstream `readiness.UpToDateCount` / `readiness.NotReadyCount` outputs (emitted since v0.8.74), so operators see the full ring breakdown - eligible / started / skipped / blocked / up-to-date / needs-attention - in one place. GitHub Actions only: `actions/download-artifact@v6 -> @v7` to silence the Node 20 deprecation warning (v7 chosen over v8 because v8 has a breaking `digest-mismatch=error` default). +> **Behaviour change in v0.8.78**: per-cluster `ScheduleBlocked` / `SideloadedBlocked` / `ExcludedByTag` outcomes now render as JUnit `` instead of `` so `dorny/test-reporter` no longer flips the Step.07 check RED on by-design gate-respect outcomes - `HealthCheckBlocked` deliberately stays a `` because a critical health failure IS actionable. The Update: 3 Readiness KPI table also gains optional **Already Up to Date** and **Not Ready (needs attention before updating)** rows, wired in both shipped `apply-updates.yml` templates from the upstream `readiness.UpToDateCount` / `readiness.NotReadyCount` outputs (emitted since v0.8.74), so operators see the full ring breakdown - eligible / started / skipped / blocked / up-to-date / needs-attention - in one place. GitHub Actions only: `actions/download-artifact@v6 -> @v7` to silence the Node 20 deprecation warning (v7 chosen over v8 because v8 has a breaking `digest-mismatch=error` default). > **New in v0.8.79 - operator break-glass override (`Force Immediate Update`)**: optional pipeline parameter on `apply-updates.yml` (`force_immediate_update` GitHub Actions choice input / `forceImmediateUpdate` Azure DevOps boolean parameter) defaulting to `false`. When set to `true` on a **manual run / queue only**, bypasses the per-cluster `UpdateStartWindow` / `UpdateExclusionsWindow` Step 3c maintenance-window gate for emergency / out-of-window patching. Defence in depth: GitHub Actions YAML collapses the flag to `'false'` for any non-`workflow_dispatch` event (`${{ github.event_name == 'workflow_dispatch' && github.event.inputs.force_immediate_update || 'false' }}`); Azure DevOps rechecks `$(Build.Reason) -eq 'Manual'` at runtime before honouring the parameter. A `WARNING:` GUI label is prepended on both pipeline hosts; a high-visibility `::warning::` (GHA) / `##vso[task.logissue type=warning]` (ADO) banner is emitted into the run log when the override fires. The override is unreachable from the `apply-updates-schedule.yml` configuration file (no `forceImmediateUpdate` field exists on `New-AzLocalApplyUpdatesScheduleConfig` / `Resolve-AzLocalPipelineUpdateRing` / `Get-AzLocalApplyUpdatesScheduleAudit`). Other readiness gates (connectivity, health, sideload status, `UpdateExcluded` operator hard-override) continue to apply. Maps to `Invoke-AzLocalReadinessGatedClusterUpdate -ForceImmediateUpdate` -> `Start-AzLocalClusterUpdate -IgnoreScheduleTags` for direct cmdlet use. @@ -173,20 +173,20 @@ The table below is the ground truth for what each shipped YAML does **out of the | **Inputs** | `update_ring` (required), `update_name` (optional - leave blank for latest), `dry_run` (optional), `throttle_limit` (optional). **v0.7.4 adds** `raise_itsm_ticket`, `itsm_config_path`, `itsm_dry_run`, `itsm_force_create` (all optional, defaults preserve existing behaviour). | | **Trigger** | **Manual only by default** (`workflow_dispatch` / **Run pipeline** button). **No schedule is shipped** - you must add one. See the **mandatory customisation note below** and the schedule-alignment guidance in [section 8](../README.md#8-scheduling-maintenance-windows-and-change-freeze-periods). | | **Cmdlets invoked** | `Get-AzLocalApplyUpdatesScheduleConfig`, `Get-AzLocalClusterUpdateReadiness`, `Start-AzLocalClusterUpdate`. When ITSM is enabled: `Get-AzLocalItsmConfig`. | -| **Depends on** | Config: 01 (`UpdateRing` tags present), Fleet: 01 (readiness reviewed for the wave). Config: 03 (schedule coverage audit) should be green so the cron(s) that start this pipeline actually fire at every tagged `UpdateStartWindow`. | +| **Depends on** | Config: 1 (`UpdateRing` tags present), Update: 1 (readiness reviewed for the wave). Config: 3 (schedule coverage audit) should be green so the cron(s) that start this pipeline actually fire at every tagged `UpdateStartWindow`. | | **Artefacts** | `update-results.xml` (JUnit, one cluster per test), `update-logs/*` (CSV + detail). When ITSM is enabled: `itsm-results.csv`, `itsm-results.xml`. | -| **When to run** | During the maintenance window for each ring, after the Fleet: 01 readiness assessment is reviewed. | +| **When to run** | During the maintenance window for each ring, after the Update: 1 readiness assessment is reviewed. | | **RBAC** | Write to clusters required. Covered by the `Azure Stack HCI Update Operator (custom)` custom role (`Microsoft.AzureStackHCI/clusters/updates/apply/action` + cluster-update reads). | -| **Exit conditions** | Pipeline run is green when every in-scope cluster either succeeds or is correctly classified as `ScheduleBlocked` / `SideloadedBlocked` / `ExcludedByTag` (these are skips, not failures - rendered as JUnit `` from v0.8.78). Per-cluster update failures and `HealthCheckBlocked` outcomes surface as JUnit `` entries; long-running runs are tracked by Fleet: 05. | +| **Exit conditions** | Pipeline run is green when every in-scope cluster either succeeds or is correctly classified as `ScheduleBlocked` / `SideloadedBlocked` / `ExcludedByTag` (these are skips, not failures - rendered as JUnit `` from v0.8.78). Per-cluster update failures and `HealthCheckBlocked` outcomes surface as JUnit `` entries; long-running runs are tracked by Update: 4. | | **ITSM** | Supported (opt-in via `raise_itsm_ticket=true` / `raiseItsmTicket=true`). When enabled, one ServiceNow incident is raised per cluster whose update run finished with a failure state; classification skips (`ScheduleBlocked` / `SideloadedBlocked` / `ExcludedByTag`) do not generate tickets. `itsm_dry_run` builds payloads without creating tickets; `itsm_force_create` bypasses dedupe. | > **MANDATORY CUSTOMISATION: the Apply Updates pipeline does not ship with a schedule.** The cluster `UpdateStartWindow` / `UpdateExclusionsWindow` tags **only gate updates *while the pipeline is already running***; they do **not** start the pipeline. If you (a) use `UpdateStartWindow` tags to define when updates may be installed and (b) leave the shipped `apply-updates.yml` with `workflow_dispatch` only (GH) / `trigger: none` (ADO), **no updates will ever be applied automatically** - the pipeline will simply never start during the window. > > Add a `schedule:` (GitHub Actions) / `schedules:` (Azure DevOps) block to `apply-updates.yml` that fires at (or a few minutes before) the start of every `UpdateStartWindow` you have tagged. One cron entry per distinct window value. Worked examples and the per-cluster scheduling model are in [section 8](../README.md#8-scheduling-maintenance-windows-and-change-freeze-periods). > -> Pipeline summaries (Fleet: 04 GH Actions + Azure DevOps) classify per-cluster skips as `ScheduleBlocked` (outside `UpdateStartWindow` or inside `UpdateExclusionsWindow`), `SideloadedBlocked` (`UpdateSideloaded=False` on a sideloaded-workflow cluster) and, new in v0.7.90, `ExcludedByTag` (`UpdateExcluded=True` operator hard override). The Actions Required callout points operators at the `UpdateExcluded` tag when one or more clusters are intentionally held out of automation. +> Pipeline summaries (Update: 3 GH Actions + Azure DevOps) classify per-cluster skips as `ScheduleBlocked` (outside `UpdateStartWindow` or inside `UpdateExclusionsWindow`), `SideloadedBlocked` (`UpdateSideloaded=False` on a sideloaded-workflow cluster) and, new in v0.7.90, `ExcludedByTag` (`UpdateExcluded=True` operator hard override). The Actions Required callout points operators at the `UpdateExcluded` tag when one or more clusters are intentionally held out of automation. -## Fleet: 05 - Monitor In-Flight Updates +## Update: 4 - Monitor In-Flight Updates > **Behaviour change in v0.8.81**: the `$stateIcon` (`State` column) and `$statusIcon` (`Current Step Status` column) switches in `Export-AzLocalUpdateRunMonitorReport` no longer hard-code GitHub-Markdown shortcodes (`:large_blue_circle:` / `:hourglass_flowing_sand:` / `:red_circle:` / etc.) which previously rendered as literal text on Azure DevOps step summaries. Both switches now read from the shared `Get-AzLocalStatusIconMap` private helper using the new `State*` (`StateInProgress` / `StateSucceeded` / `StateFailed` / `StateNotStarted` / `StateUnknown`) and `Status*` (`StatusInProgress` / `StatusCancelled`) keys. JUnit XML output and CSV columns are unchanged. @@ -196,37 +196,17 @@ The table below is the ground truth for what each shipped YAML does **out of the | **Inputs** | `scope` (`all-clusters` (default) or `by-update-ring`), `update_ring` (only used when `scope=by-update-ring`; single ring, `Prod;Ring2` semicolon-list, or `***` wildcard), `long_running_threshold_hours` (0-48, default 6), `module_version` (optional). | | **Trigger** | **Manual only by default** (`workflow_dispatch` / **Run pipeline** button). A commented-out `schedule: cron '*/30 * * * *'` (GH) / `schedules: cron '*/30 * * * *'` (ADO) sample is shipped inside the `# BEGIN-AZLOCAL-CUSTOMIZE:schedule-triggers` block - uncomment it for the duration of an active wave, then re-comment it once the wave drains, so you do not generate 48 empty runs per day between waves. | | **Cmdlets invoked** | `Get-AzLocalUpdateRuns`, `Get-AzLocalClusterInventory` (used for ring-scope mode). | -| **Depends on** | Fleet: 04 must be active (at least one `InProgress` run) for the output to be non-empty. Runs cleanly on an empty fleet (reports "no in-flight runs"). | +| **Depends on** | Update: 3 must be active (at least one `InProgress` run) for the output to be non-empty. Runs cleanly on an empty fleet (reports "no in-flight runs"). | | **Artefacts** | `update-monitor.xml` (JUnit, one `` per in-flight cluster - the failure message contains current step + progress + elapsed duration when the run is over `long_running_threshold_hours`, or the deepest-step error detail when `failureType='StepError'` fires under v0.7.96), `update-monitor.csv` (full per-cluster rows including the new `Status` + `ErrorMessage` columns), markdown job summary with the in-flight table (cluster + update names as portal links from v0.7.96) and long-running flag column. | | **Exit conditions** | Pipeline run is green when the snapshot completes. Long-running runs (over `long_running_threshold_hours`) AND stuck-step errors (`failureType='StepError'`, v0.7.96+) surface as JUnit `` entries so they appear in the Tests / Checks tab without failing the whole pipeline. v0.7.96 also adds two new `GITHUB_OUTPUT` values: `STEP_ERRORED` (count of runs that hit a step error inside the threshold) and `UNRESOLVED_FAILURES` (count of Failed runs surfaced from `Get-AzLocalUpdateRunFailures`). | -| **When to run** | While a wave is active (manually, or via the commented cron). The two daily snapshot pipelines (Fleet: 06 and Fleet: 07) remain the steady-state daily reports; this pipeline is purpose-built for the "is the apply-updates run I started two hours ago still progressing?" question that those daily snapshots cannot answer between runs. | +| **When to run** | While a wave is active (manually, or via the commented cron). The two daily snapshot pipelines (Monitor: 2 and Monitor: 3) remain the steady-state daily reports; this pipeline is purpose-built for the "is the apply-updates run I started two hours ago still progressing?" question that those daily snapshots cannot answer between runs. | | **RBAC** | Read-only - `Reader` on the cluster scope, plus the cluster-update API read paths already enumerated in the `Azure Stack HCI Update Operator (custom)` custom role. | -| **ITSM** | Not supported - in-flight monitor exists to be a fast feedback loop during an active wave; long-running runs surface as JUnit `` entries for the operator on call, and post-wave failures are picked up + ticketed by the daily Fleet: 06 snapshot. | +| **ITSM** | Not supported - in-flight monitor exists to be a fast feedback loop during an active wave; long-running runs surface as JUnit `` entries for the operator on call, and post-wave failures are picked up + ticketed by the daily Monitor: 3 snapshot. | | **Introduced** | v0.7.90. Enhanced in v0.7.96 with the `Status` + `ErrorMessage` columns, the new `StepError` JUnit failure type, the always-shown Failed-runs block, portal-linked Cluster Name / Update Name cells, and the new `STEP_ERRORED` / `UNRESOLVED_FAILURES` `GITHUB_OUTPUT` values. **Behaviour change in v0.7.98:** UX overhaul of the in-flight table - rows now sort by composite `SeverityScore` (`StepError severity x 1000 + RunSeverity x 100 + elapsed-hours bucket`) so stuck step errors and runs over 14 days bubble to the top regardless of cluster alphabetical order; each row carries per-cell `StateIcon` + `StatusIcon` icons and a horizontal chip stack of flags (`STEP-STUCK`, `RUN-STUCK`, `UNRESOLVED`, `RECENT-FAIL`); the job summary opens with a single `CRITICAL / WARN / OK` fleet status badge that collapses the worst row across the fleet into one line (e.g. `CRITICAL - 1 stuck step error(s), 1 run(s) > 14d, 1 step(s) > 4h`); each failed step's `errorMessage` is now wrapped in a collapsible `
Verbose error...
` block; and JUnit `` + `` are populated with real per-run elapsed seconds (was `time="0"` before, which made GitHub Test Reporter render `"5 tests were completed in 0ms"`). | --- -## Fleet: 06 - Fleet Update Status - -> **Behaviour change in v0.8.81**: the GitHub-Markdown shortcodes used by the Critical Health Status table, the Primary Status table, and the Version Distribution Support column are now routed through the shared `Get-AzLocalStatusIconMap` private helper using the new `Info` / `GreenCircle` / `YellowCircle` / `CycleArrows` / `GreyQuestion` / `SupportSupported` / `SupportUnsupported` / `SupportUnknown` keys (fixes literal `:green_circle:` / `:yellow_circle:` / `:information_source:` / etc. text on Azure DevOps step summaries). JUnit XML output and CSV columns are unchanged. - -| Aspect | Value | -|---|---| -| **Purpose** | Daily fleet-wide snapshot of cluster update state. Read-only. **v0.7.90 pivots the Version Distribution markdown table by YYMM** (leading column `Version` = `2511`, `2604`, ...; new `Update Versions` column lists each distinct full version installed within that YYMM as ` x ` separated by `
`; rows sorted ascending by YYMM so the oldest YYMM is at the top). The underlying `` JUnit XML is unchanged - still one `` per distinct full `CurrentVersion` - so machine-readable consumers and CI test-reporters are unaffected. **v0.7.96 reworks the Primary Status bucket cascade to align with the Azure portal Update Manager `state` filter:** `NeedsAttention` is now promoted into the **Update Failed** bucket; `PreparationFailed` lands in a new **Action Required** bucket (with its own actionable guidance in the Actions Required callout because the run never started, so re-arming is required); `PreparationInProgress` is folded into **Update In Progress**. The new bucket surfaces via (1) a new `` JUnit attribute on the primary testsuite, (2) per-testcase `failureType='PreparationFailed'` instead of the generic `UpdateFailure`, (3) a new `ACTION_REQUIRED` `GITHUB_OUTPUT` (`actionRequired` pipeline variable in ADO), (4) `Summary.UpdateFailures` + `Summary.ActionRequired` in `readiness-status.json`, and (5) a new "Action Required (PreparationFailed)" row in the Primary Status markdown table with separate actionable prose in the Actions Required callout. The `📜 Update Run History and Error Details` markdown table renders Cluster Name and Update Name cells as `` deep-links (cluster's Updates blade + the single-instance update-run history view) - same linking as the Fleet: 05 in-flight table. | -| **Inputs** | Scope (`-AllClusters` or `-ScopeByUpdateRingTag`), `throttle_limit` (optional). v0.7.4-style ITSM toggles are also exposed: `raise_itsm_ticket`, `itsm_config_path`, `itsm_dry_run`, `itsm_force_create` (all optional, default off). | -| **Trigger** | Manual (`workflow_dispatch` / **Run pipeline** button) **plus** scheduled daily at 06:00 UTC (`cron '0 6 * * *'`). Edit the cron in the YAML to change cadence. | -| **Cmdlets invoked** | `Get-AzLocalClusterInventory`, `Get-AzLocalClusterUpdateReadiness`, `Get-AzLocalUpdateSummary`, `Get-AzLocalAvailableUpdates`, `Get-AzLocalUpdateRuns`, `Get-AzLocalUpdateRunFailures`, `Get-AzLocalLatestSolutionVersion`. When ITSM is enabled: `Get-AzLocalItsmConfig`. | -| **Depends on** | None directly. Best-quality output when Fleet: 02 (connectivity) has run earlier the same morning, since clusters Arc can't see won't appear in the readiness or update-summary results. | -| **Artefacts** | `readiness-status.xml` / `.csv` / `.json` (the `.json` carries the rolled-up bucket counts under `Summary.*` - `UpdateFailures`, `ActionRequired`, `HealthFailures`, `SbeBlocked`, `InProgress`, `ReadyForUpdate`, `UpToDate`, `Other` - and the new `primaryActionRequired` field mirrors the JUnit testsuite attribute from v0.7.96 for downstream automation), `cluster-inventory.csv`, `update-summaries.csv`, `available-updates.csv`, `update-runs.csv`. | -| **When to run** | Hands-off scheduled. Trigger manually for ad-hoc reporting. | -| **RBAC** | Read-only. Covered by the `Azure Stack HCI Update Operator (custom)` custom role. | -| **Exit conditions** | Pipeline run is green when the snapshot completes. Per-cluster issues (outdated version, failed run history) surface as JUnit `` entries. v0.7.96 distinguishes `failureType='UpdateFailure'` (run executed and failed - landed in `NeedsAttention`) from `failureType='PreparationFailed'` (run never started - landed in the new `Action Required` bucket) so downstream automation can split paging behaviour per failure mode. | -| **ITSM** | Supported (opt-in via `raise_itsm_ticket=true` / `raiseItsmTicket=true`). When enabled, one ServiceNow incident is raised per unresolved failed update run picked up by the daily snapshot; dedupe key combines cluster + update version + failure reason via `./.itsm/azurelocal-itsm.yml`. `itsm_dry_run` builds payloads without creating tickets; `itsm_force_create` bypasses dedupe. | -| **Introduced** | Originally Fleet: 04 in v0.7.4 (then Step 7); renumbered to Step 8 in v0.7.90 when the in-flight monitor moved into Step 7; renumbered to Step 9 in v0.8.7 when the on-prem sideload pipeline took Step 6. v0.7.90 added the YYMM-pivoted Version Distribution table. v0.7.96 promoted `NeedsAttention` into the `Update Failed` bucket, added the new `Action Required` bucket for `PreparationFailed`, folded `PreparationInProgress` into `Update In Progress`, added the `primaryActionRequired` JUnit attribute, the `ACTION_REQUIRED` `GITHUB_OUTPUT`, `Summary.UpdateFailures` + `Summary.ActionRequired` in `readiness-status.json`, and portal-linked Cluster Name / Update Name cells in the `Update Run History and Error Details` markdown table. **Behaviour change in v0.7.98:** each `` inside the `📜 Update Run History and Error Details` testsuite now emits `time` = `DurationMinutes * 60` (`DurationMinutes` from KQL `datetime_diff('minute', EndTime, StartTime)` via `Get-AzLocalUpdateRunFailures`), and the testsuite-level `time=` is the sum of those per-row seconds. The other two testsuites (`FleetVersionDistribution`, `AzureLocalFleetUpdateStatus`) intentionally stay at `time="0"` - they are instantaneous snapshot projections and a synthetic duration would be misleading. | - ---- - -## Fleet: 07 - Fleet Health Status +## Monitor: 2 - Fleet Health Status > **Behaviour change in v0.8.81**: fixes the **"Healthy + Unhealthy != Total" KPI counting bug**. The previous KPI table summed only two buckets (Healthy + Unhealthy) and silently dropped clusters whose `HealthStatus` was `In progress` / `Unknown` / `Health check failed` AND had no failure rows. The KPI block is now split into two tables: **Cluster Counts** (Total / Healthy / Unhealthy / **Other**, where Other is computed as `Total - Healthy - Unhealthy` clamped non-negative so the three buckets always sum to Total) and **Failing Checks Breakdown** (Total / Critical / Warning / Distinct Reasons). New `other_clusters` pipeline step output; `-PassThru` PSCustomObject gains `OtherClusters`. **Detailed Results columns reordered** to put the most-specific identifier first: `Severity | Title | Failure Reason | Description | Health Check Name | Failure Remediation | Target Resource Name | Target Resource Type | Last Occurrence | Resource Group`. Description is a collapsible `
` block that surfaces drive/volume-level detail (e.g. the file path emitted by a `Microsoft.Health.FaultType.Volume.FileSystem.Corruption.Correctable` warning); Health Check Name is the raw fault-type name (preserved verbatim so operators can grep / triage by FaultType across runs). Icons in the table are wired through the new shared `Get-AzLocalStatusIconMap` private helper. @@ -236,7 +216,7 @@ The table below is the ground truth for what each shipped YAML does **out of the | **Inputs** | `severity` (optional - `Critical`, `Warning`, or `All`; default `All`), `update_ring_tag` (optional - narrow to one wave), `throttle_limit` (optional). v0.7.4-style ITSM toggles are also exposed: `raise_itsm_ticket`, `itsm_config_path`, `itsm_dry_run`, `itsm_force_create` (all optional, default off). | | **Trigger** | Manual (`workflow_dispatch` / **Run pipeline** button) **plus** scheduled daily at 07:00 UTC (`cron '0 7 * * *'`). Deliberately offset by one hour from `fleet-update-status` (06:00 UTC) to avoid agent and ARM contention. Edit the cron in the YAML to change cadence. | | **Cmdlets invoked** | `Get-AzLocalFleetHealthFailures`, `Get-AzLocalFleetHealthOverview`. When ITSM is enabled: `Get-AzLocalItsmConfig`. | -| **Depends on** | None directly. Best-quality output when Fleet: 02 (connectivity) has run earlier the same morning. | +| **Depends on** | None directly. Best-quality output when Monitor: 1 (connectivity) has run earlier the same morning. | | **Artefacts** | `fleet-health-status.xml` (JUnit, one `` per failing check, grouped under `Critical Health Failures` / `Warning Health Failures` testsuites), `fleet-health-detail.csv` (one row per failing check), `fleet-health-summary.csv` (aggregated by `FailureReason` + `Severity`), markdown job summary. | | **When to run** | Hands-off scheduled. Trigger manually for ad-hoc fleet-wide health triage outside the daily schedule, especially after operational events (capacity changes, network maintenance, certificate rotations) where health-check failures are expected to spike. | | **RBAC** | Read-only. Covered by the `Azure Stack HCI Update Operator (custom)` custom role. | @@ -245,3 +225,23 @@ The table below is the ground truth for what each shipped YAML does **out of the | **Introduced** | v0.7.65. | --- + +## Monitor: 3 - Fleet Update Status + +> **Behaviour change in v0.8.81**: the GitHub-Markdown shortcodes used by the Critical Health Status table, the Primary Status table, and the Version Distribution Support column are now routed through the shared `Get-AzLocalStatusIconMap` private helper using the new `Info` / `GreenCircle` / `YellowCircle` / `CycleArrows` / `GreyQuestion` / `SupportSupported` / `SupportUnsupported` / `SupportUnknown` keys (fixes literal `:green_circle:` / `:yellow_circle:` / `:information_source:` / etc. text on Azure DevOps step summaries). JUnit XML output and CSV columns are unchanged. + +| Aspect | Value | +|---|---| +| **Purpose** | Daily fleet-wide snapshot of cluster update state. Read-only. **v0.7.90 pivots the Version Distribution markdown table by YYMM** (leading column `Version` = `2511`, `2604`, ...; new `Update Versions` column lists each distinct full version installed within that YYMM as ` x ` separated by `
`; rows sorted ascending by YYMM so the oldest YYMM is at the top). The underlying `` JUnit XML is unchanged - still one `` per distinct full `CurrentVersion` - so machine-readable consumers and CI test-reporters are unaffected. **v0.7.96 reworks the Primary Status bucket cascade to align with the Azure portal Update Manager `state` filter:** `NeedsAttention` is now promoted into the **Update Failed** bucket; `PreparationFailed` lands in a new **Action Required** bucket (with its own actionable guidance in the Actions Required callout because the run never started, so re-arming is required); `PreparationInProgress` is folded into **Update In Progress**. The new bucket surfaces via (1) a new `` JUnit attribute on the primary testsuite, (2) per-testcase `failureType='PreparationFailed'` instead of the generic `UpdateFailure`, (3) a new `ACTION_REQUIRED` `GITHUB_OUTPUT` (`actionRequired` pipeline variable in ADO), (4) `Summary.UpdateFailures` + `Summary.ActionRequired` in `readiness-status.json`, and (5) a new "Action Required (PreparationFailed)" row in the Primary Status markdown table with separate actionable prose in the Actions Required callout. The `📜 Update Run History and Error Details` markdown table renders Cluster Name and Update Name cells as `` deep-links (cluster's Updates blade + the single-instance update-run history view) - same linking as the Update: 4 in-flight table. | +| **Inputs** | Scope (`-AllClusters` or `-ScopeByUpdateRingTag`), `throttle_limit` (optional). v0.7.4-style ITSM toggles are also exposed: `raise_itsm_ticket`, `itsm_config_path`, `itsm_dry_run`, `itsm_force_create` (all optional, default off). | +| **Trigger** | Manual (`workflow_dispatch` / **Run pipeline** button) **plus** scheduled daily at 06:00 UTC (`cron '0 6 * * *'`). Edit the cron in the YAML to change cadence. | +| **Cmdlets invoked** | `Get-AzLocalClusterInventory`, `Get-AzLocalClusterUpdateReadiness`, `Get-AzLocalUpdateSummary`, `Get-AzLocalAvailableUpdates`, `Get-AzLocalUpdateRuns`, `Get-AzLocalUpdateRunFailures`, `Get-AzLocalLatestSolutionVersion`. When ITSM is enabled: `Get-AzLocalItsmConfig`. | +| **Depends on** | None directly. Best-quality output when Monitor: 1 (connectivity) has run earlier the same morning, since clusters Arc can't see won't appear in the readiness or update-summary results. | +| **Artefacts** | `readiness-status.xml` / `.csv` / `.json` (the `.json` carries the rolled-up bucket counts under `Summary.*` - `UpdateFailures`, `ActionRequired`, `HealthFailures`, `SbeBlocked`, `InProgress`, `ReadyForUpdate`, `UpToDate`, `Other` - and the new `primaryActionRequired` field mirrors the JUnit testsuite attribute from v0.7.96 for downstream automation), `cluster-inventory.csv`, `update-summaries.csv`, `available-updates.csv`, `update-runs.csv`. | +| **When to run** | Hands-off scheduled. Trigger manually for ad-hoc reporting. | +| **RBAC** | Read-only. Covered by the `Azure Stack HCI Update Operator (custom)` custom role. | +| **Exit conditions** | Pipeline run is green when the snapshot completes. Per-cluster issues (outdated version, failed run history) surface as JUnit `` entries. v0.7.96 distinguishes `failureType='UpdateFailure'` (run executed and failed - landed in `NeedsAttention`) from `failureType='PreparationFailed'` (run never started - landed in the new `Action Required` bucket) so downstream automation can split paging behaviour per failure mode. | +| **ITSM** | Supported (opt-in via `raise_itsm_ticket=true` / `raiseItsmTicket=true`). When enabled, one ServiceNow incident is raised per unresolved failed update run picked up by the daily snapshot; dedupe key combines cluster + update version + failure reason via `./.itsm/azurelocal-itsm.yml`. `itsm_dry_run` builds payloads without creating tickets; `itsm_force_create` bypasses dedupe. | +| **Introduced** | Originally shipped in v0.7.4 (as Step 7); renumbered to Step 8 in v0.7.90 when the in-flight monitor moved into Step 7; renumbered to Step 9 in v0.8.7 when the on-prem sideload pipeline took Step 6. v0.7.90 added the YYMM-pivoted Version Distribution table. v0.7.96 promoted `NeedsAttention` into the `Update Failed` bucket, added the new `Action Required` bucket for `PreparationFailed`, folded `PreparationInProgress` into `Update In Progress`, added the `primaryActionRequired` JUnit attribute, the `ACTION_REQUIRED` `GITHUB_OUTPUT`, `Summary.UpdateFailures` + `Summary.ActionRequired` in `readiness-status.json`, and portal-linked Cluster Name / Update Name cells in the `Update Run History and Error Details` markdown table. **Behaviour change in v0.7.98:** each `` inside the `📜 Update Run History and Error Details` testsuite now emits `time` = `DurationMinutes * 60` (`DurationMinutes` from KQL `datetime_diff('minute', EndTime, StartTime)` via `Get-AzLocalUpdateRunFailures`), and the testsuite-level `time=` is the sum of those per-row seconds. The other two testsuites (`FleetVersionDistribution`, `AzureLocalFleetUpdateStatus`) intentionally stay at `time="0"` - they are instantaneous snapshot projections and a synthetic duration would be misleading. | + +--- diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/docs/sideload.md b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/docs/sideload.md index 141361c6..db747e31 100644 --- a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/docs/sideload.md +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/docs/sideload.md @@ -20,7 +20,7 @@ imported, the pipeline flips the `UpdateSideloaded=True` gate so the downstream internet-connected clusters. For the per-pipeline reference card (inputs, artefacts, RBAC, exit conditions) see -[appendix-pipelines.md - Fleet: 03](appendix-pipelines.md#fleet-03---sideload-updates-opt-in). +[appendix-pipelines.md - Update: 2](appendix-pipelines.md#update-2---sideload-updates-opt-in). For robocopy throttling guidance see [sideload-robocopy.md](sideload-robocopy.md). --- diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/apply-updates-schedule-audit.yml b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/apply-updates-schedule-audit.yml index 6df7771b..73d557de 100644 --- a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/apply-updates-schedule-audit.yml +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/apply-updates-schedule-audit.yml @@ -31,7 +31,7 @@ # Workflow name carries the same Step.N - prefix as the filename so the GitHub # Actions sidebar (which sorts workflows alphabetically by this `name:` field) # lists the eight pipelines in execution order. -name: 'Config: 03 - Apply-Updates Schedule Coverage Audit' +name: 'Config: 3 - Apply-Updates Schedule Coverage Audit' on: # BEGIN-AZLOCAL-CUSTOMIZE:schedule-triggers @@ -96,7 +96,7 @@ env: # this to the version actually installed and to the latest on PSGallery, and emits a # ::notice annotation if the YAML appears stale - prompting you to refresh via # Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5. - GENERATED_AGAINST_MODULE_VERSION: '0.8.86' + GENERATED_AGAINST_MODULE_VERSION: '0.8.87' REQUIRED_MODULE_VERSION: ${{ github.event.inputs.module_version || vars.REQUIRED_MODULE_VERSION || '' }} # v0.8.4 - opt this workflow into Node.js 24 for all JavaScript actions # (actions/checkout, actions/download-artifact, actions/upload-artifact, diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/apply-updates.yml b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/apply-updates.yml index d81ec4c2..2acce124 100644 --- a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/apply-updates.yml +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/apply-updates.yml @@ -12,7 +12,7 @@ # Workflow name carries the same Step.N - prefix as the filename so the GitHub # Actions sidebar (which sorts workflows alphabetically by this `name:` field) # lists the eight pipelines in execution order. -name: 'Fleet: 04 - Apply Updates' +name: 'Update: 3 - Apply Updates' on: # PREFERRED PATTERN - drive automated update runs from a ring-aware @@ -148,7 +148,7 @@ env: # this to the version actually installed and to the latest on PSGallery, and emits a # ::notice annotation if the YAML appears stale - prompting you to refresh via # Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5. - GENERATED_AGAINST_MODULE_VERSION: '0.8.86' + GENERATED_AGAINST_MODULE_VERSION: '0.8.87' # Resolution order for the module version pin (leave all unset to install the latest, # which is the default "fix-forward" behaviour): manual workflow_dispatch input > # repository variable 'REQUIRED_MODULE_VERSION' > empty (latest). diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/assess-update-readiness.yml b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/assess-update-readiness.yml index 924a8ca5..a8f2a444 100644 --- a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/assess-update-readiness.yml +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/assess-update-readiness.yml @@ -36,7 +36,7 @@ # Workflow name carries the same Step.N - prefix as the filename so the GitHub # Actions sidebar (which sorts workflows alphabetically by this `name:` field) # lists the eight pipelines in execution order. -name: 'Fleet: 01 - Assess Update Readiness' +name: 'Update: 1 - Assess Update Readiness' on: # BEGIN-AZLOCAL-CUSTOMIZE:schedule-triggers @@ -72,7 +72,7 @@ env: # this to the version actually installed and to the latest on PSGallery, and emits a # ::notice annotation if the YAML appears stale - prompting you to refresh via # Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5. - GENERATED_AGAINST_MODULE_VERSION: '0.8.86' + GENERATED_AGAINST_MODULE_VERSION: '0.8.87' # Resolution order for the module version pin (leave all unset to install the latest, # which is the default "fix-forward" behaviour): manual workflow_dispatch input > # repository variable 'REQUIRED_MODULE_VERSION' > empty (latest). diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/fleet-connectivity-status.yml b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/fleet-connectivity-status.yml index 0ce0ad0a..65dad3d0 100644 --- a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/fleet-connectivity-status.yml +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/fleet-connectivity-status.yml @@ -50,7 +50,7 @@ # Workflow name carries the same Step.N - prefix as the filename so the GitHub # Actions sidebar (which sorts workflows alphabetically by this `name:` field) # lists the pipelines in execution order. -name: 'Fleet: 02 - Fleet Connectivity Status' +name: 'Monitor: 1 - Fleet Connectivity Status' on: # BEGIN-AZLOCAL-CUSTOMIZE:schedule-triggers @@ -116,7 +116,7 @@ env: # PSGallery, and emits a ::notice annotation if the YAML appears stale - # prompting you to refresh via Copy-AzLocalPipelineExample -Update. See # Automation-Pipeline-Examples/README.md section 5. - GENERATED_AGAINST_MODULE_VERSION: '0.8.86' + GENERATED_AGAINST_MODULE_VERSION: '0.8.87' # Resolution order for the module version pin (leave all unset to install # the latest, the default "fix-forward" behaviour): manual workflow_dispatch # input > repository variable 'REQUIRED_MODULE_VERSION' > empty (latest). diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/fleet-health-status.yml b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/fleet-health-status.yml index 1a988bfe..bab6f10e 100644 --- a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/fleet-health-status.yml +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/fleet-health-status.yml @@ -32,7 +32,7 @@ # Workflow name carries the same Step.N - prefix as the filename so the GitHub # Actions sidebar (which sorts workflows alphabetically by this `name:` field) # lists the eight pipelines in execution order. -name: 'Fleet: 07 - Fleet Health Status' +name: 'Monitor: 2 - Fleet Health Status' on: # BEGIN-AZLOCAL-CUSTOMIZE:schedule-triggers @@ -116,7 +116,7 @@ env: # this to the version actually installed and to the latest on PSGallery, and emits a # ::notice annotation if the YAML appears stale - prompting you to refresh via # Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5. - GENERATED_AGAINST_MODULE_VERSION: '0.8.86' + GENERATED_AGAINST_MODULE_VERSION: '0.8.87' # Resolution order for the module version pin (leave all unset to install the latest, # which is the default "fix-forward" behaviour): manual workflow_dispatch input > # repository variable 'REQUIRED_MODULE_VERSION' > empty (latest). diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/fleet-update-status.yml b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/fleet-update-status.yml index d267508c..a339d85c 100644 --- a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/fleet-update-status.yml +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/fleet-update-status.yml @@ -24,7 +24,7 @@ # Workflow name carries the same Step.N - prefix as the filename so the GitHub # Actions sidebar (which sorts workflows alphabetically by this `name:` field) # lists the eight pipelines in execution order. -name: 'Fleet: 06 - Fleet Update Status' +name: 'Monitor: 3 - Fleet Update Status' on: # BEGIN-AZLOCAL-CUSTOMIZE:schedule-triggers @@ -105,7 +105,7 @@ env: # this to the version actually installed and to the latest on PSGallery, and emits a # ::notice annotation if the YAML appears stale - prompting you to refresh via # Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5. - GENERATED_AGAINST_MODULE_VERSION: '0.8.86' + GENERATED_AGAINST_MODULE_VERSION: '0.8.87' # Resolution order for the module version pin (leave all unset to install the latest, # which is the default "fix-forward" behaviour): manual workflow_dispatch input > # repository variable 'REQUIRED_MODULE_VERSION' > empty (latest). diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/manage-updatering-tags.yml b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/manage-updatering-tags.yml index c9ed17e5..dfbfa3ff 100644 --- a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/manage-updatering-tags.yml +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/manage-updatering-tags.yml @@ -22,7 +22,7 @@ # Workflow name carries the same Setup prefix as the filename so the GitHub # Actions sidebar (which sorts workflows alphabetically by this `name:` field) # lists the setup and operational pipelines in execution order. -name: 'Config: 02 - Manage UpdateRing Tags' +name: 'Config: 2 - Manage UpdateRing Tags' on: # BEGIN-AZLOCAL-CUSTOMIZE:schedule-triggers @@ -65,7 +65,7 @@ env: # this to the version actually installed and to the latest on PSGallery, and emits a # ::notice annotation if the YAML appears stale - prompting you to refresh via # Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5. - GENERATED_AGAINST_MODULE_VERSION: '0.8.86' + GENERATED_AGAINST_MODULE_VERSION: '0.8.87' # Resolution order for the module version pin (leave all unset to install the latest, # which is the default "fix-forward" behaviour): manual workflow_dispatch input > # repository variable 'REQUIRED_MODULE_VERSION' > empty (latest). diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/monitor-updates.yml b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/monitor-updates.yml index 087fa286..1c513ecd 100644 --- a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/monitor-updates.yml +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/monitor-updates.yml @@ -23,7 +23,7 @@ # Workflow name carries the same Step.N - prefix as the filename so the GitHub # Actions sidebar (which sorts workflows alphabetically by this `name:` field) # lists the pipelines in execution order. -name: 'Fleet: 05 - Monitor In-Flight Updates' +name: 'Update: 4 - Monitor In-Flight Updates' on: # BEGIN-AZLOCAL-CUSTOMIZE:schedule-triggers @@ -76,6 +76,39 @@ on: description: 'CRITICAL tier for overall-elapsed (default 3 days). In-flight runs older than this get a :rotating_light: chip; older than 2x this get a :skull: chip. Strongest visual signal that human intervention is needed - a 19-day-stuck run will no longer be hidden behind a single "step errored" chip.' required: false default: '3' + # ITSM (ServiceNow) auto-ticketing - opt-in (v0.8.87). When enabled the + # connector reads the update-monitor.xml this pipeline already produces and, + # for each cluster whose Status matches the trigger matrix (default raises + # for AttemptWithoutRun / StepError / Failed), opens a deduped incident. + # Default 'false' so existing schedules stay byte-identical until you opt in. + raise_itsm_ticket: + description: 'Open ITSM tickets (ServiceNow) for in-flight monitor failures (stuck steps, failed runs, attempt-without-run)' + required: false + default: 'false' + type: choice + options: + - 'false' + - 'true' + itsm_config_path: + description: 'Path to ITSM matrix config (YAML or JSON)' + required: false + default: './.itsm/azurelocal-itsm.yml' + itsm_dry_run: + description: 'ITSM: build payloads + run read-only dedupe but do NOT create tickets' + required: false + default: 'false' + type: choice + options: + - 'false' + - 'true' + itsm_force_create: + description: 'ITSM: bypass dedupe and always create new tickets (use with caution)' + required: false + default: 'false' + type: choice + options: + - 'false' + - 'true' module_version: description: 'Pin AzLocal.UpdateManagement version (empty = latest from PSGallery). See Automation-Pipeline-Examples/README.md section 5 "Optional configuration".' required: false @@ -86,7 +119,7 @@ env: # this to the version actually installed and to the latest on PSGallery, and emits a # ::notice annotation if the YAML appears stale - prompting you to refresh via # Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5. - GENERATED_AGAINST_MODULE_VERSION: '0.8.86' + GENERATED_AGAINST_MODULE_VERSION: '0.8.87' # Resolution order for the module version pin (leave all unset to install the latest, # which is the default "fix-forward" behaviour): manual workflow_dispatch input > # repository variable 'REQUIRED_MODULE_VERSION' > empty (latest). @@ -235,3 +268,101 @@ jobs: list-suites: failed list-tests: failed continue-on-error: true + + # ---------------------------------------------------------------------- + # ITSM Connector (ServiceNow auto-raise on in-flight monitor failures) + # + # Fully opt-in (gated on inputs.raise_itsm_ticket == 'true'). Runs AFTER + # the JUnit file is uploaded + published, so the ITSM action is strictly + # additive - failures here never affect the monitor exit code (which is + # always green; this is a report-only pipeline). + # + # Reads the same update-monitor.xml that powers the run summary. The + # JUnit emitter writes per-testcase (ClusterName / + # ClusterResourceId / UpdateName / Status / CurrentStep / + # ClusterPortalUrl / UpdateRunPortalUrl) so New-AzLocalIncident can + # compute the SHA256 dedupe key (one incident per stuck/failed/attempt-gap + # cluster until it clears) and the Mustache body template can deep-link + # into the Azure portal. Configure which monitor Status values raise a + # ticket in ./.itsm/azurelocal-itsm.yml (defaults raise AttemptWithoutRun / + # StepError / Failed; LongRunningStep / LongRunningOverall are opt-in). + # ---------------------------------------------------------------------- + - name: Install powershell-yaml (ITSM config parser) + if: ${{ github.event.inputs.raise_itsm_ticket == 'true' }} + shell: pwsh + run: | + if (-not (Get-Module -ListAvailable -Name powershell-yaml)) { + Install-Module powershell-yaml -Scope CurrentUser -Force -AllowClobber + } + + - name: Raise ITSM tickets + if: ${{ github.event.inputs.raise_itsm_ticket == 'true' }} + shell: pwsh + id: itsm + env: + # BEGIN-AZLOCAL-CUSTOMIZE:itsm-secrets + # Bind your ITSM connector secrets here. Defaults match the + # ServiceNow OAuth client_credentials naming used by azurelocal-itsm.yml. + # Preserved by Update-AzLocalPipelineExample across module upgrades. + ITSM_SN_INSTANCE_URL: ${{ secrets.ITSM_SN_INSTANCE_URL }} + ITSM_SN_CLIENT_ID: ${{ secrets.ITSM_SN_CLIENT_ID }} + ITSM_SN_CLIENT_SECRET: ${{ secrets.ITSM_SN_CLIENT_SECRET }} + # END-AZLOCAL-CUSTOMIZE:itsm-secrets + INPUT_ITSM_CONFIG_PATH: ${{ github.event.inputs.itsm_config_path }} + INPUT_ITSM_DRY_RUN: ${{ github.event.inputs.itsm_dry_run }} + INPUT_ITSM_FORCE_CREATE: ${{ github.event.inputs.itsm_force_create }} + run: | + Import-Module AzLocal.UpdateManagement -Force + + $configPath = $env:INPUT_ITSM_CONFIG_PATH + $dryRun = $env:INPUT_ITSM_DRY_RUN -eq 'true' + $force = $env:INPUT_ITSM_FORCE_CREATE -eq 'true' + + if (-not (Test-Path $configPath)) { + Write-Host "::warning::ITSM config not found at '$configPath' - skipping ticket creation." + exit 0 + } + + $cfg = Get-AzLocalItsmConfig -Path $configPath + + $junitInput = './reports/update-monitor.xml' + if (-not (Test-Path $junitInput)) { + Write-Host "::warning::No update-monitor.xml found at '$junitInput' - skipping ticket creation." + exit 0 + } + + $params = @{ + InputArtifactPath = $junitInput + Config = $cfg + RunMetadata = @{ + Platform = 'github' + RunId = $env:GITHUB_RUN_ID + RunUrl = "$env:GITHUB_SERVER_URL/$env:GITHUB_REPOSITORY/actions/runs/$env:GITHUB_RUN_ID" + Branch = $env:GITHUB_REF + } + DryRun = $dryRun + ForceCreate = $force + ExportPath = './reports/itsm-results.csv' + ExportJUnitPath = './reports/itsm-results.xml' + } + + $results = New-AzLocalIncident @params + $results | Format-Table ClusterName, Action, TicketId, Severity -AutoSize + + - name: Upload ITSM Artefacts + if: ${{ github.event.inputs.raise_itsm_ticket == 'true' }} + uses: actions/upload-artifact@v6 + with: + name: azlocal-step.8-update-monitor-itsm-results_${{ steps.artifact-stamp.outputs.timestamp }} + path: ./reports/itsm-*.* + retention-days: 30 + continue-on-error: true + + - name: Publish ITSM Test Results + if: ${{ github.event.inputs.raise_itsm_ticket == 'true' }} + uses: dorny/test-reporter@v3 + with: + name: 'ITSM Tickets (Update: 4 Monitor)' + path: ./reports/itsm-results.xml + reporter: java-junit + continue-on-error: true diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/setup-validate-and-inventory.yml b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/setup-validate-and-inventory.yml index 4aa08f3b..a9d4ce9b 100644 --- a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/setup-validate-and-inventory.yml +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/setup-validate-and-inventory.yml @@ -1,5 +1,5 @@ # AZLOCAL-PIPELINE-ID: setup-validate-and-inventory -# Config: 01 - Validate Auth and Inventory Azure Local Clusters +# Config: 1 - Validate Auth and Inventory Azure Local Clusters # # PURPOSE: # This is the first step in the setup workflow. It combines two operations: @@ -7,7 +7,7 @@ # 2. INVENTORY: Queries all Azure Local clusters and exports inventory with UpdateRing tag status # # Run this workflow: -# - BEFORE adding the operational workflows (Fleet: 01-07). It narrows any failure +# - BEFORE adding the operational workflows (Update: 1-4 and Monitor: 1-3). It narrows any failure # to authentication/discovery rather than seven interacting pipelines. # - PERIODICALLY (recommended monthly, or after every RBAC change in the tenant) # to confirm subscription scope and cluster inventory are accurate. @@ -42,7 +42,7 @@ # # See Automation-Pipeline-Examples/README.md for full setup story. -name: 'Config: 01 - Validate Auth and Inventory Clusters' +name: 'Config: 1 - Validate Auth and Inventory Clusters' on: workflow_dispatch: @@ -73,7 +73,7 @@ env: # this to the version actually installed and to the latest on PSGallery, and emits a # ::notice annotation if the YAML appears stale - prompting you to refresh via # Copy-AzLocalPipelineExample. See Automation-Pipeline-Examples/README.md section 5. - GENERATED_AGAINST_MODULE_VERSION: '0.8.86' + GENERATED_AGAINST_MODULE_VERSION: '0.8.87' # Resolution order for the module version pin (leave all unset to install the latest, # which is the default "fix-forward" behaviour): manual workflow_dispatch input > # repository variable 'REQUIRED_MODULE_VERSION' > empty (latest). @@ -171,7 +171,7 @@ jobs: if: always() uses: dorny/test-reporter@v3 with: - name: '[JUnit Debug] Config: 01 - Authentication Validation' + name: '[JUnit Debug] Config: 1 - Authentication Validation' path: ./reports/auth-report.xml reporter: java-junit fail-on-error: false diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/sideload-updates.yml b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/sideload-updates.yml index b5ee2a81..7306b3e4 100644 --- a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/sideload-updates.yml +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/sideload-updates.yml @@ -38,7 +38,7 @@ # Workflow name carries the same Step.N - prefix as the filename so the GitHub # Actions sidebar (which sorts workflows alphabetically by this `name:` field) # lists the pipelines in execution order. -name: 'Fleet: 03 - Sideload Updates (Opt-in)' +name: 'Update: 2 - Sideload Updates (Opt-in)' on: # BEGIN-AZLOCAL-CUSTOMIZE:schedule-triggers @@ -83,7 +83,7 @@ env: # this to the version actually installed and to the latest on PSGallery, and emits a # ::notice annotation if the YAML appears stale - prompting you to refresh via # Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5. - GENERATED_AGAINST_MODULE_VERSION: '0.8.86' + GENERATED_AGAINST_MODULE_VERSION: '0.8.87' # Resolution order for the module version pin (leave all unset to install the latest, # which is the default "fix-forward" behaviour): manual workflow_dispatch input > # repository variable 'REQUIRED_MODULE_VERSION' > empty (latest). diff --git a/AzLocal.UpdateManagement/AzLocal.UpdateManagement.psd1 b/AzLocal.UpdateManagement/AzLocal.UpdateManagement.psd1 index 3a3b643d..39b7cca5 100644 --- a/AzLocal.UpdateManagement/AzLocal.UpdateManagement.psd1 +++ b/AzLocal.UpdateManagement/AzLocal.UpdateManagement.psd1 @@ -3,7 +3,7 @@ RootModule = 'AzLocal.UpdateManagement.psm1' # Version number of this module. - ModuleVersion = '0.8.86' + ModuleVersion = '0.8.87' # Supported PSEditions CompatiblePSEditions = @('Desktop', 'Core') @@ -317,9 +317,11 @@ # ReleaseNotes of this module ReleaseNotes = @' -## Version 0.8.86 - Patch: renames the three onboarding pipeline templates from `Setup: 0N` to `Config: 0N` so the GitHub Actions sidebar (and the Azure DevOps Pipelines list) sorts the onboarding / configuration workflows ahead of the `Fleet: 0N` operational workflows. Both surfaces sort alphabetically by the workflow `name:` / definition name, and `C` (Config) sorts before `F` (Fleet); the previous `Setup:` prefix sorted AFTER `Fleet:`. Only the operator-facing `name:` (GitHub Actions) / `displayName:` (Azure DevOps) fields change - filenames, `AZLOCAL-PIPELINE-ID` values, aliases, prune logic, and all cmdlet behaviour are unchanged. Bundled pipeline README + appendix docs updated to match. No public API or behavioural change. Export count unchanged (still 60). `GENERATED_AGAINST_MODULE_VERSION` bumped from `'0.8.85'` to `'0.8.86'`. +## Version 0.8.87 - Renames the bundled pipeline display names into a three-group `Config: N` / `Monitor: N` / `Update: N` scheme (single-digit, e.g. `Monitor: 1 - Fleet Connectivity Status`, `Update: 4 - Monitor In-Flight Updates`), replacing the former `Setup: 0N` / `Fleet: 0N` prefixes; filenames and `AZLOCAL-PIPELINE-ID` values are unchanged. Also renames the "Orphan ARBs" section in the `New-AzLocalFleetConnectivityStatusSummary` (Monitor: 1 - Fleet Connectivity Status) output to "Non-Azure Local and/or Orphan ARB appliances" and adds a caveat that an Arc resource bridge with no matching in-scope Azure Local cluster is NOT necessarily orphaned - Arc resource bridge is also used by VMware vSphere and SCVMM. Adds investigate-before-acting guidance. KPI note, causes list, and cluster-table cross-reference updated to match. The Cluster Connectivity table is split into "Cluster with Connectivity Issues" (shown first) and "Cluster without Connectivity Issues" (Connected + ARB Running, collapsed). Output text only - no public API or behavioural change. Export count unchanged (still 60). `GENERATED_AGAINST_MODULE_VERSION` bumped to `'0.8.87'`. Also: `Export-AzLocalUpdateRunMonitorReport` (Update: 4) now writes per-`` `` (ClusterName/ClusterResourceId/UpdateName/Status/CurrentStep/portal URLs) into `update-monitor.xml`, with Status values StepError/LongRunningStep/LongRunningOverall/InProgress/Failed/AttemptWithoutRun, so `New-AzLocalIncident` can dedupe and deep-link stuck/failed/attempt-without-run clusters. The bundled `monitor-updates.yml` (GitHub + Azure DevOps) gains an opt-in ITSM ticketing step (`raise_itsm_ticket`/`raiseItsmTicket`, default off). The sample ITSM matrix gains AttemptWithoutRun/StepError (raise) and opt-in LongRunningOverall/LongRunningStep entries. `Export-AzLocalApplyUpdatesScheduleAudit` (Config: 3) gains an always-on "Recommended in-flight monitor schedule (Update: 4)" section plus `-MonitorFiresPerHour` (1-12, default 2) and `-MonitorTrailingDays` (0-14, default 3). Additive. -## Version 0.8.85 - Patch: introduces Setup/Fleet naming in bundled pipeline templates, adds merged GitHub onboarding workflow `setup-validate-and-inventory.yml`, and updates pipeline refresh tooling with optional deprecated-file pruning guarded by AZLOCAL-PIPELINE-ID verification. No public API or behavioural change. Export count unchanged (still 60). `GENERATED_AGAINST_MODULE_VERSION` bumped from `'0.8.84'` to `'0.8.85'`. +## Version 0.8.86 - Patch: renamed the three onboarding pipeline templates from `Setup: 0N` to `Config: 0N` (operator-facing display name only; filenames, `AZLOCAL-PIPELINE-ID` values, aliases, prune logic, and cmdlet behaviour unchanged). Superseded by the v0.8.87 `Config:`/`Monitor:`/`Update:` naming scheme. Export count unchanged (still 60). `GENERATED_AGAINST_MODULE_VERSION` bumped to `'0.8.86'`. + +## Version 0.8.85 - Patch: introduced Setup/Fleet pipeline naming, added the merged GitHub onboarding workflow `setup-validate-and-inventory.yml`, and added optional deprecated-file pruning guarded by AZLOCAL-PIPELINE-ID verification. Superseded by later naming. No public API or behavioural change. `GENERATED_AGAINST_MODULE_VERSION` bumped to `'0.8.85'`. ## Version 0.8.83 - Patch: fix-forward for v0.8.82 Item-5. The Step.08 `UpdateLastAttempt` reconciliation in `Export-AzLocalUpdateRunMonitorReport` reads `$inv.tags` from `Get-AzLocalClusterInventory`, but the v0.8.82 inventory projection did not carry the raw ARM `tags` bag - so the "Recent update attempts with no observable updateRun" section was silently always empty in production. v0.8.83 surfaces the raw `tags` bag on every inventory row (in-memory only; the CSV / JSON export keeps its explicit `$selectColumns` whitelist - new regression test asserts both). Also wires `attempts_without_run` into the GitHub Actions `monitor-updates.yml` `jobs.outputs:` block (ADO `Set-AzLocalPipelineOutput` auto-publishes so only an ADO docstring refresh needed), and corrects the `Export-AzLocalUpdateRunMonitorReport` "6 step outputs" docstring to "7 step outputs". No public API change. Export count unchanged (still 60). All bundled pipeline templates bump `GENERATED_AGAINST_MODULE_VERSION` from `'0.8.82'` to `'0.8.83'`. @@ -327,17 +329,13 @@ ## Version 0.8.81 - Step summary polish across Steps 05-10. Step.10 fixes the KPI bug where Healthy + Unhealthy did not sum to Total (splits into Cluster Counts + Failing Checks Breakdown tables, adds `OtherClusters` step output; Detailed Results gains Title, raw FailureName and collapsible Description columns surfacing drive/volume detail e.g. file paths from `...FileSystem.Corruption.Correctable`). Steps 05-09 adopt three shared private helpers: `Get-AzLocalStatusIconMap` (host-aware GitHub Unicode vs Azure DevOps shortcodes), `Get-AzLocalClusterPortalLink` (portal deep-link wrapper) and `Get-AzLocalCtrlClickTip` (single-source Ctrl-click banner). Fixes literal shortcode text (`:white_check_mark:` etc.) rendering on Azure DevOps step summaries in Step.08 monitor + Step.09 fleet-update-status. No public API change (still 60 exports). All bundled pipeline templates bump `GENERATED_AGAINST_MODULE_VERSION` from `'0.8.80'` to `'0.8.81'`. -## Version 0.8.80 - Minor: three additive pipeline failure-rendering improvements across Step.05 / Step.08 / Step.09 / Step.10 step summaries. Q1 - Step.09 `Get-AzLocalUpdateRunFailures` attaches a `HealthCheckEvidence` array column (same-cluster Critical health-check entries within +/-2h) to HealthCheck-category rows; new private helper `Get-AzLocalUpdateRunHealthEvidence`; `-EnrichWithHealthEvidence` opt-out. Q2 - Step.05 `Test-AzLocalClusterHealth` and Step.10 `Get-AzLocalFleetHealthFailures` add per-check `Title` + full `TargetResourceID`; `Export-AzLocalFleetHealthStatusReport` wraps TargetResourceName in a portal hyperlink. Q3 - both deepest-error walkers capture step `description` alongside `errorMessage`; new `DeepestStepDescription` / `ErrorDescription` fields on `Get-AzLocalUpdateRunFailures` / `Format-AzLocalUpdateRun` / `Get-AzLocalUpdateRuns -PassThru`; Step.08 + Step.09 renderers combine the two in markdown failure cells and JUnit bodies. No new exports (still 60). `GENERATED_AGAINST_MODULE_VERSION` bumped from `'0.8.79'` to `'0.8.80'`. See CHANGELOG.md for the full v0.8.80 entry. +## Version 0.8.80 - Minor: three additive pipeline failure-rendering improvements across Step.05 / Step.08 / Step.09 / Step.10 step summaries. Q1 - Step.09 `Get-AzLocalUpdateRunFailures` attaches a `HealthCheckEvidence` array column (same-cluster Critical health-check entries within +/-2h) to HealthCheck-category rows; new private helper `Get-AzLocalUpdateRunHealthEvidence`; `-EnrichWithHealthEvidence` opt-out. Q2 - Step.05 `Test-AzLocalClusterHealth` and Step.10 `Get-AzLocalFleetHealthFailures` add per-check `Title` + full `TargetResourceID`; `Export-AzLocalFleetHealthStatusReport` wraps TargetResourceName in a portal hyperlink. Q3 - both deepest-error walkers capture step `description` alongside `errorMessage`; new `DeepestStepDescription` / `ErrorDescription` fields on `Get-AzLocalUpdateRunFailures` / `Format-AzLocalUpdateRun` / `Get-AzLocalUpdateRuns -PassThru`; Step.08 + Step.09 renderers combine the two in markdown failure cells and JUnit bodies. No new exports (still 60). `GENERATED_AGAINST_MODULE_VERSION` bumped from `'0.8.79'` to `'0.8.80'`. ## Version 0.8.79 - Patch: operator break-glass override for Step.07. Adds `Force Immediate Update` to `Invoke-AzLocalReadinessGatedClusterUpdate` (`-ForceImmediateUpdate`) and `Start-AzLocalClusterUpdate` (`-IgnoreScheduleTags`), plus matching pipeline parameters `force_immediate_update` (GitHub Actions `workflow_dispatch` input) / `forceImmediateUpdate` (Azure DevOps `parameters:` boolean) both defaulting to `false`. When enabled, the per-cluster Step 3c maintenance-window gate (`UpdateStartWindow` / `UpdateExclusionsWindow` tags) is bypassed and updates start regardless of the current UTC time. Intended for emergency / out-of-window patching driven by an on-call operator. The override is UNREACHABLE from the apply-updates-schedule.yml configuration file - GHA wiring collapses the flag to `'false'` for any non-`workflow_dispatch` trigger (`${{ github.event_name == 'workflow_dispatch' && github.event.inputs.force_immediate_update || 'false' }}`), and ADO rechecks `$(Build.Reason) -eq 'Manual'` at runtime before honouring the parameter. A `WARNING:` GUI label is prepended on both pipeline hosts and a high-visibility `::warning::` (GHA) / `##vso[task.logissue type=warning]` (ADO) banner is emitted into the run log when the override fires. No new exports (still 60). All bundled pipeline templates bump `GENERATED_AGAINST_MODULE_VERSION` from `'0.8.78'` to `'0.8.79'`. -- **CHANGE (`Start-AzLocalClusterUpdate`)**: new `[switch]$IgnoreScheduleTags` parameter; when set, the Step 3c maintenance-window block is skipped and a per-cluster `Warning` is logged with the bypassed tag values. Does NOT bypass any other readiness gate. -- **CHANGE (`Invoke-AzLocalReadinessGatedClusterUpdate`)**: new `[switch]$ForceImmediateUpdate` parameter; forwards `-IgnoreScheduleTags` per cluster and emits a host-aware warning banner at the top of the apply. -- **CHANGE (`apply-updates.yml`)**: new `force_immediate_update` GHA choice / `forceImmediateUpdate` ADO boolean parameter; both enforce the manual-only constraint at the YAML layer so a scheduled cron firing can never honour the flag. - ## Version 0.8.78 - Patch: pipeline-summary UX polish (`ScheduleBlocked` / `SideloadedBlocked` / `ExcludedByTag` now JUnit ``; `Add-AzLocalApplyUpdatesStepSummary` gains `-UpToDateCount` / `-NotReadyCount`; `actions/download-artifact@v6` -> `@v7`). See CHANGELOG.md / docs/release-history.md. -For full v0.7.x and v0.8.x release notes see: +For full release notes see: https://github.com/NeilBird/Azure-Local/blob/main/AzLocal.UpdateManagement/CHANGELOG.md '@ diff --git a/AzLocal.UpdateManagement/AzLocal.UpdateManagement.psm1 b/AzLocal.UpdateManagement/AzLocal.UpdateManagement.psm1 index f593905f..bf79b8de 100644 --- a/AzLocal.UpdateManagement/AzLocal.UpdateManagement.psm1 +++ b/AzLocal.UpdateManagement/AzLocal.UpdateManagement.psm1 @@ -151,7 +151,7 @@ Set-StrictMode -Version 1.0 # bumps to one but not the other are caught before release. Two consumers: # - Start-AzLocalClusterUpdate emits this in the run log header. # - Get-AzLocalFleetStatusData stamps it into exported fleet-state JSON. -$script:ModuleVersion = '0.8.86' +$script:ModuleVersion = '0.8.87' $script:DefaultApiVersion = '2025-10-01' $script:DefaultLogFolder = Join-Path -Path $env:ProgramData -ChildPath 'AzLocal.UpdateManagement' diff --git a/AzLocal.UpdateManagement/CHANGELOG.md b/AzLocal.UpdateManagement/CHANGELOG.md index a591afa2..1c43dea7 100644 --- a/AzLocal.UpdateManagement/CHANGELOG.md +++ b/AzLocal.UpdateManagement/CHANGELOG.md @@ -5,6 +5,95 @@ All notable changes to the AzLocal.UpdateManagement module (renamed from AzStack The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [0.8.87] - 2026-06-16 + +Renames the bundled pipeline display names into a three-group `Config: N` / +`Monitor: N` / `Update: N` scheme (single-digit, e.g. +`Monitor: 1 - Fleet Connectivity Status`, `Update: 4 - Monitor In-Flight +Updates`), replacing the former `Setup: 0N` / `Fleet: 0N` prefixes so the +GitHub Actions and Azure DevOps lists group onboarding, day-2 monitoring, and +update-lifecycle workflows logically. Display name (`name:` / `displayName:`) +only - filenames and `AZLOCAL-PIPELINE-ID` values are unchanged. + +Also renames the "Orphan ARBs" section in the +`New-AzLocalFleetConnectivityStatusSummary` (Monitor: 1 - Fleet Connectivity +Status) markdown output to "Non-Azure Local and/or Orphan ARB appliances" and +adds an explicit caveat that an Arc resource bridge with no matching in-scope +Azure Local cluster is NOT necessarily orphaned. Azure Arc resource bridge is +also used by VMware vSphere and System Center Virtual Machine Manager (SCVMM) +Arc-enabled deployments, so a listed appliance may be a healthy, in-use bridge +for a non-Azure Local platform. Output text only - no public API change. Export +count unchanged (still 60). + +Also extends the in-flight monitor (Update: 4) and ITSM connector so stuck / +failed / attempt-without-run clusters surfaced by +`Export-AzLocalUpdateRunMonitorReport` can auto-raise deduped ServiceNow +incidents, and teaches the Config: 3 schedule auditor +(`Export-AzLocalApplyUpdatesScheduleAudit`) to recommend an Update: 4 monitor +poll cadence. New parameters are additive with backward-compatible defaults; +export count unchanged (still 60). + +### Added + +- `Export-AzLocalUpdateRunMonitorReport` now writes per-`` + `` into `update-monitor.xml` (`ClusterName`, `ClusterResourceId`, + `UpdateName`, `Status`, `CurrentStep`, `ClusterPortalUrl`, + `UpdateRunPortalUrl`). The `Status` values are `StepError`, `LongRunningStep`, + `LongRunningOverall`, `InProgress`, `Failed`, and `AttemptWithoutRun` (the + UpdateLastAttempt-reconciliation rows, which use a `(no update name)` + `UpdateName` fallback so the ITSM dedupe key stays stable per cluster). This + is what lets `New-AzLocalIncident` dedupe and deep-link from the monitor + pipeline (previously every monitor row was skipped for missing + ClusterResourceId / UpdateName). +- `Export-AzLocalApplyUpdatesScheduleAudit` (Config: 3) gains a new always-on + "Recommended in-flight monitor schedule (Update: 4)" section that derives a + `monitor-updates.yml` poll cron from the apply-window weekday(s), covering the + apply day plus a configurable trailing window for multi-day runs. Two new + parameters: `-MonitorFiresPerHour` (1-12, default 2 = every 30 min) and + `-MonitorTrailingDays` (0-14, default 3, mirroring the Update: 4 CRITICAL + elapsed tier). +- The bundled `monitor-updates.yml` (GitHub Actions and Azure DevOps) gains an + opt-in ITSM ticketing step (`raise_itsm_ticket` / `raiseItsmTicket`, + default off) that runs `New-AzLocalIncident` against the monitor JUnit after + it is published. Strictly additive - the monitor remains report-only and + always green. +- The sample ITSM trigger matrix + (`Automation-Pipeline-Examples/.itsm/azurelocal-itsm.yml`) gains entries for + the Update: 4 monitor statuses: `AttemptWithoutRun` (raise, severity 3), + `StepError` (raise, severity 2), and opt-in `LongRunningOverall` / + `LongRunningStep`. Documented in `ITSM/ITSM-Config-Reference.md`. + +### Changed + +- Renames the bundled pipeline display names (`name:` on GitHub Actions, + `displayName:` / `name:` on Azure DevOps) into the three-group + `Config: N` / `Monitor: N` / `Update: N` scheme with single-digit numbering: + `Config: 1-3` (onboarding/config), `Monitor: 1-3` (day-2 fleet monitoring), + and `Update: 1-4` (update lifecycle). `Monitor: 2` is Fleet Health Status and + `Monitor: 3` is Fleet Update Status (numbered to match alphabetical order). + Filenames, `AZLOCAL-PIPELINE-ID` values, aliases, and `-PruneDeprecated` + logic are unchanged - this is a display-name-only change. Bundled pipeline + `README.md`, `docs/appendix-pipelines.md`, and the ITSM / release docs are + updated to match. +- `New-AzLocalFleetConnectivityStatusSummary` renames the `### Orphan ARBs (no + matching cluster in scope)` section heading to `### Non-Azure Local and/or + Orphan ARB appliances`, adds an intro line ("These ARB appliances do not have + a matching Azure Local instance (cluster) in scope."), and expands the italic + caveat to call out VMware vSphere and SCVMM Arc resource bridges and to add + investigate-before-acting guidance (check resource type, custom location, and + the platform served; do not delete an ARB until confirmed genuinely orphaned). +- The KPI-table "Orphan ARBs" note, the "how to interpret" causes list (new + leading bullet about non-Azure Local resource bridges), and the cluster-table + cross-reference now point at the renamed section. +- The `### Cluster Connectivity (with ARB Status)` table is split into two: a + `### Cluster with Connectivity Issues` table (clusters that are not Connected + with a Running ARB) shown first and expanded, and a `### Cluster without + Connectivity Issues` table (Connectivity = Connected AND ARB Status = Running) + collapsed behind a `Expand to view clusters` details block. + Both tables share the same columns. +- `GENERATED_AGAINST_MODULE_VERSION` bumped from `0.8.86` to `0.8.87` across all + bundled pipeline templates. + ## [0.8.86] - 2026-06-16 Patch release. Renames the three onboarding pipeline templates from `Setup: 0N` diff --git a/AzLocal.UpdateManagement/ITSM/ITSM-Config-Reference.md b/AzLocal.UpdateManagement/ITSM/ITSM-Config-Reference.md index 19119cdc..de991813 100644 --- a/AzLocal.UpdateManagement/ITSM/ITSM-Config-Reference.md +++ b/AzLocal.UpdateManagement/ITSM/ITSM-Config-Reference.md @@ -56,6 +56,15 @@ Top-level keys are JUnit `Status` values produced by `Get-AzLocalUpdateRuns` / `Invoke-AzLocalFleetOperation` (e.g. `Failed`, `Error`, `HealthCheckBlocked`, `SideloadedBlocked`, `ExcludedByTag`, `ScheduleBlocked`, `Skipped`, `NotReady`). +From v0.8.87 the **Update: 4 - Monitor In-Flight Updates** pipeline +(`Export-AzLocalUpdateRunMonitorReport` -> `update-monitor.xml`) also emits +per-testcase `Status` values you can target: `AttemptWithoutRun` (an +`UpdateLastAttempt` tag with no observable updateRun, commonly a URP package +pre-install health-check failure), `StepError` (a step stuck in `Error` while +the run is still `InProgress`), and the informational `LongRunningOverall` / +`LongRunningStep` thresholds. Wire `monitor-updates.yml` with +`raise_itsm_ticket=true` to action them. + | Field | Type | Notes | |---|---|---| | `triggers..raiseTicket` | bool | Required. `true` opens a ticket; `false` (default) skips. | diff --git a/AzLocal.UpdateManagement/ITSM/README.md b/AzLocal.UpdateManagement/ITSM/README.md index 58b92e5b..e9854cbc 100644 --- a/AzLocal.UpdateManagement/ITSM/README.md +++ b/AzLocal.UpdateManagement/ITSM/README.md @@ -30,6 +30,8 @@ When any of the four "operator-attention" pipelines finishes - **`Step.4_fleet-c > **v0.7.76: Step.4 fleet-connectivity ticketing added.** When the `Step.4_fleet-connectivity-status` pipeline was introduced in v0.7.76 it shipped with the same opt-in ITSM wiring as Step.7 / Step.8 - gated on `raise_itsm_ticket=true`, reading `./reports/fleet-connectivity-status.xml`, and using the existing `azurelocal-itsm.yml` trigger matrix. Each row in the JUnit file emits `Status=Critical` or `Status=Warning` (Disconnected / Offline / partial-connectivity Arc-agent / ARB / NIC / cluster rows), so the same `Critical` / `Warning` entries from the trigger matrix in [Section 5](#5-author-the-trigger-matrix) drive Step.4 with no extra config. Step.4 emits a row-specific `UpdateName` per testcase (e.g. `ClusterConnectivity=Disconnected`, `ArcAgent=Disconnected []`, `PhysicalNic=Down [/]`, `ARB=Offline []`), so the SHA256 dedupe key naturally separates a cluster-level disconnect from an individual NIC / Arc-agent / ARB failure even when several rows fire for the same cluster on the same day. +> **v0.8.87: Update: 4 in-flight monitor ticketing added.** The `monitor-updates.yml` pipeline (Update: 4 - Monitor In-Flight Updates, produced by `Export-AzLocalUpdateRunMonitorReport`) now ships with the same opt-in ITSM wiring - gated on `raise_itsm_ticket=true` (`raiseItsmTicket` on Azure DevOps), reading `./reports/update-monitor.xml`, and using the existing `azurelocal-itsm.yml` trigger matrix. Until v0.8.87 the monitor JUnit carried no per-`` ``, so every monitor row was skipped by `New-AzLocalIncident` for a missing `ClusterResourceId` / `UpdateName`. The monitor now emits `ClusterName` / `ClusterResourceId` / `UpdateName` / `Status` / `CurrentStep` / `ClusterPortalUrl` / `UpdateRunPortalUrl` per row, with `Status` one of `StepError` / `LongRunningStep` / `LongRunningOverall` / `InProgress` / `Failed` / `AttemptWithoutRun`. The trigger matrix gains `AttemptWithoutRun` (raise, severity 3) and `StepError` (raise, severity 2) plus opt-in `LongRunningOverall` / `LongRunningStep` entries (see [Section 5](#5-author-the-trigger-matrix)). `AttemptWithoutRun` rows (UpdateLastAttempt tag with no observable updateRun) use a `(no update name)` `UpdateName` fallback so the SHA256 dedupe key stays stable per cluster. The monitor remains report-only and always green - ITSM failures never affect its result. + What it deliberately does **not** do in Phase 1: open Jira / ADO Work Items, send Teams / Slack notifications, or close tickets on success. See [ITSM-Connector-Plan.md Sections 2 + 9](./ITSM-Connector-Plan.md) for the phased roadmap. --- diff --git a/AzLocal.UpdateManagement/Private/Get-AzLocalPipelineManifest.ps1 b/AzLocal.UpdateManagement/Private/Get-AzLocalPipelineManifest.ps1 index 5db6430d..f1ae650b 100644 --- a/AzLocal.UpdateManagement/Private/Get-AzLocalPipelineManifest.ps1 +++ b/AzLocal.UpdateManagement/Private/Get-AzLocalPipelineManifest.ps1 @@ -82,7 +82,7 @@ function Get-AzLocalPipelineManifest { # # v0.8.85: the two standalone onboarding workflows authentication-test.yml # (DisplayStep 0) and inventory-clusters.yml (DisplayStep 1) were merged - # into a single setup-validate-and-inventory.yml (Config: 01). Its Aliases + # into a single setup-validate-and-inventory.yml (Config: 1). Its Aliases # list BOTH superseded filenames (and their Step.N_ ancestors) so # Update-AzLocalPipelineExample can rename-match a customer who still has # either old file and carry their schedule CRONs into the merged workflow. diff --git a/AzLocal.UpdateManagement/Public/Export-AzLocalApplyUpdatesScheduleAudit.ps1 b/AzLocal.UpdateManagement/Public/Export-AzLocalApplyUpdatesScheduleAudit.ps1 index b60aa2a3..3681a16c 100644 --- a/AzLocal.UpdateManagement/Public/Export-AzLocalApplyUpdatesScheduleAudit.ps1 +++ b/AzLocal.UpdateManagement/Public/Export-AzLocalApplyUpdatesScheduleAudit.ps1 @@ -136,6 +136,18 @@ function Export-AzLocalApplyUpdatesScheduleAudit { shifted back this many days). When not explicitly bound it is resolved from the SIDELOAD_LEAD_DAYS environment variable, defaulting to 7. + .PARAMETER MonitorFiresPerHour + How often the v0.8.87 "Recommended in-flight monitor schedule" section + suggests the Update: 4 monitor-updates pipeline should poll while an + update run is in flight (1-12, default 2 = every 30 minutes). Drives + the minute field of the recommended monitor cron. + + .PARAMETER MonitorTrailingDays + How many days after an apply window opens an update run may still be + in flight (0-14, default 3, mirroring the Update: 4 monitor's CRITICAL + elapsed tier). The recommended monitor cron covers the apply weekday(s) + plus this many trailing days so multi-day runs stay observed. + .PARAMETER InstalledModuleVersion Optional [string] used in the markdown footer ('Generated by AzLocal.UpdateManagement v'). @@ -225,6 +237,14 @@ function Export-AzLocalApplyUpdatesScheduleAudit { [ValidateRange(0, 365)] [int]$SideloadLeadDays = 7, + [Parameter(Mandatory = $false)] + [ValidateRange(1, 12)] + [int]$MonitorFiresPerHour = 2, + + [Parameter(Mandatory = $false)] + [ValidateRange(0, 14)] + [int]$MonitorTrailingDays = 3, + [Parameter(Mandatory = $false)] [AllowEmptyString()] [AllowNull()] @@ -279,6 +299,8 @@ function Export-AzLocalApplyUpdatesScheduleAudit { Write-Host "IncludeUntagged : $IncludeUntagged" Write-Host "ClusterCsvPath : $(if ($haveCsv) { $ClusterCsvPath } else { '(skipped - empty or missing)' })" Write-Host "Platform : $Platform" + Write-Host "MonitorFires/hr : $MonitorFiresPerHour" + Write-Host "MonitorTrailDays : $MonitorTrailingDays" Write-Host '' # ---- Audit view (rows + CSV) ------------------------------------------ @@ -859,6 +881,88 @@ function Export-AzLocalApplyUpdatesScheduleAudit { $sideloadScheduleIncluded = $true } + # ---- Recommended in-flight monitor schedule (Update: 4) ---------------- + # v0.8.87: recommend a poll cron for the 'Update: 4 - Monitor In-Flight + # Updates' (monitor-updates.yml) pipeline, scaled to how often updates are + # applied. Once an apply window fires, an update run can stay in-flight for + # hours to days (the monitor's own CRITICAL elapsed tier defaults to 3 + # days), so the monitor must poll frequently across the apply weekday(s) + # AND the trailing days an update may still be running. Cadence (fires/hour) + # is driven by -MonitorFiresPerHour; trailing coverage by -MonitorTrailingDays. + # Always emitted (independent of the sideload opt-in). + $monitorFires = $MonitorFiresPerHour + $monitorIntervalMinutes = [int][math]::Floor(60 / $monitorFires) + if ($monitorIntervalMinutes -lt 1) { $monitorIntervalMinutes = 1 } + $monitorMinuteField = if ($monitorFires -le 1) { '0' } else { ('*/{0}' -f $monitorIntervalMinutes) } + + # Distinct apply-window weekly firings (DayOfWeek) from the apply pipeline + # crons. Read-AzLocalApplyUpdatesYamlCrons uses a unary-comma return - + # direct assignment only (NEVER @() wrap). + $monitorApplyFirings = New-Object 'System.Collections.Generic.List[psobject]' + try { + $monitorCronTriggers = Read-AzLocalApplyUpdatesYamlCrons -Path $PipelineYamlPath -ErrorAction Stop + foreach ($mct in $monitorCronTriggers) { + try { + $mparsed = ConvertFrom-AzLocalCronExpression -Expression $mct.CronExpression -ErrorAction Stop + if ($mparsed.IsValid -and -not $mparsed.IsComplex) { + foreach ($mft in @($mparsed.FireTimes)) { + $monitorApplyFirings.Add([pscustomobject]@{ + DayOfWeek = [int]$mft.DayOfWeek + }) | Out-Null + } + } + } + catch { + Write-Verbose "Monitor section: cron parse failed for '$($mct.CronExpression)': $($_.Exception.Message)" + } + } + } + catch { + Write-Warning "Failed to read apply-updates crons for the monitor schedule recommendation: $($_.Exception.Message)" + } + + [void]$md.Add('### Recommended in-flight monitor schedule (Update: 4)') + [void]$md.Add('') + [void]$md.Add(('The **Update: 4 - Monitor In-Flight Updates** (`monitor-updates.yml`) pipeline should poll frequently while any update run is in flight. An update can stay in-flight for hours to days after its apply window opens, so poll **{0}x/hour** (every {1} min) across the apply weekday(s) plus the next **{2} day(s)** to catch multi-day runs.' -f $monitorFires, $monitorIntervalMinutes, $MonitorTrailingDays)) + [void]$md.Add('') + + $monitorDayNames = @('Sun', 'Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat') + $monitorApplyDows = @($monitorApplyFirings | ForEach-Object { [int]$_.DayOfWeek } | Sort-Object -Unique) + if ($monitorApplyDows.Count -gt 0) { + # Expand each apply weekday across the trailing coverage days. + $monitorCoverageSet = New-Object 'System.Collections.Generic.HashSet[int]' + foreach ($applyDow in $monitorApplyDows) { + for ($k = 0; $k -le $MonitorTrailingDays; $k++) { + [void]$monitorCoverageSet.Add((($applyDow + $k) % 7)) + } + } + $monitorCoverageDows = @($monitorCoverageSet) | Sort-Object + $monitorDayField = if ($monitorCoverageDows.Count -ge 7) { '*' } else { ($monitorCoverageDows -join ',') } + $monitorCron = ('{0} * * * {1}' -f $monitorMinuteField, $monitorDayField) + $monitorCoverageLabel = if ($monitorCoverageDows.Count -ge 7) { 'every day' } else { (($monitorCoverageDows | ForEach-Object { $monitorDayNames[$_] }) -join ', ') } + $monitorApplyLabel = ($monitorApplyDows | ForEach-Object { $monitorDayNames[$_] }) -join ', ' + + [void]$md.Add(('Apply firing weekday(s): **{0}**. With {1} trailing day(s) of coverage, the monitor should run on: **{2}**.' -f $monitorApplyLabel, $MonitorTrailingDays, $monitorCoverageLabel)) + [void]$md.Add('') + [void]$md.Add('Recommended Update: 4 monitor cron - paste into the `schedule:` block of `monitor-updates.yml`:') + [void]$md.Add('') + [void]$md.Add('```') + [void]$md.Add($monitorCron) + [void]$md.Add('```') + [void]$md.Add('') + [void]$md.Add('> GitHub Actions supports sub-hour cron (e.g. `*/30`). Azure DevOps YAML `schedules:` is hourly-only - for sub-hour cadence on ADO, trigger `monitor-updates.yml` from an external scheduler via REST. Widen the day list above if your runs routinely exceed the trailing-day coverage.') + [void]$md.Add('') + } + else { + $monitorFallbackCron = ('{0} * * * *' -f $monitorMinuteField) + [void]$md.Add('> No simple weekly apply-window firings were found in the pipeline YAML, so a focused monitor window could not be derived. Run the monitor 24x7 at the configured cadence:') + [void]$md.Add('') + [void]$md.Add('```') + [void]$md.Add($monitorFallbackCron) + [void]$md.Add('```') + [void]$md.Add('') + } + [void]$md.Add('### Reports Available') [void]$md.Add("- ``$AuditCsvFileName`` - one row per (UpdateRing, UpdateStartWindow) pair") [void]$md.Add("- ``$MatrixCsvFileName`` - full inventory + required cron per row") diff --git a/AzLocal.UpdateManagement/Public/Export-AzLocalUpdateRunMonitorReport.ps1 b/AzLocal.UpdateManagement/Public/Export-AzLocalUpdateRunMonitorReport.ps1 index 51aa570d..a0b4f316 100644 --- a/AzLocal.UpdateManagement/Public/Export-AzLocalUpdateRunMonitorReport.ps1 +++ b/AzLocal.UpdateManagement/Public/Export-AzLocalUpdateRunMonitorReport.ps1 @@ -491,6 +491,23 @@ function Export-AzLocalUpdateRunMonitorReport { # ---- JUnit XML via the shared emitter --------------------------------- $testCases = New-Object 'System.Collections.Generic.List[hashtable]' + # v0.8.87: emit per-testcase (ClusterName / ClusterResourceId / + # UpdateName / Status / CurrentStep / portal URLs) so the ITSM connector + # (New-AzLocalIncident) can compute the SHA256 dedupe key and the Mustache + # body template can deep-link into the Azure portal. The Status property is + # what Get-AzLocalItsmTriggerDecision matches against the trigger matrix. + $tcProps = { + param($row, [string]$statusValue) + [ordered]@{ + ClusterName = [string]$row.ClusterName + ClusterResourceId = [string]$row.ClusterResourceId + UpdateName = [string]$row.UpdateName + Status = $statusValue + CurrentStep = [string]$row.CurrentStep + ClusterPortalUrl = [string]$row.ClusterPortalUrl + UpdateRunPortalUrl = [string]$row.UpdateRunPortalUrl + } + } foreach ($r in ($inFlight | Sort-Object @{Expression='SeverityScore';Descending=$true}, ClusterName)) { $safeName = ($r.ClusterName -replace '[^A-Za-z0-9_.-]', '_') $caseName = "$safeName - $($r.UpdateName) - $($r.CurrentStep)" @@ -513,6 +530,7 @@ function Export-AzLocalUpdateRunMonitorReport { ClassName = 'UpdateMonitor' Time = [double]$r.RunDurationSeconds Failure = @{ Message = $msg; Type = 'StepError'; Body = $msg } + Properties = (& $tcProps $r 'StepError') }) | Out-Null } elseif ($r.ExceedsStepThreshold) { @@ -522,6 +540,7 @@ function Export-AzLocalUpdateRunMonitorReport { ClassName = 'UpdateMonitor' Time = [double]$r.RunDurationSeconds Failure = @{ Message = $msg; Type = 'LongRunningStep'; Body = $msg } + Properties = (& $tcProps $r 'LongRunningStep') }) | Out-Null } elseif ($r.ExceedsThreshold) { @@ -531,6 +550,7 @@ function Export-AzLocalUpdateRunMonitorReport { ClassName = 'UpdateMonitor' Time = [double]$r.RunDurationSeconds Failure = @{ Message = $msg; Type = 'LongRunningOverall'; Body = $msg } + Properties = (& $tcProps $r 'LongRunningOverall') }) | Out-Null } else { @@ -538,6 +558,7 @@ function Export-AzLocalUpdateRunMonitorReport { Name = $caseName ClassName = 'UpdateMonitor' Time = [double]$r.RunDurationSeconds + Properties = (& $tcProps $r 'InProgress') }) | Out-Null } } @@ -561,6 +582,7 @@ function Export-AzLocalUpdateRunMonitorReport { ClassName = 'UpdateMonitor' Time = [double]$r.RunDurationSeconds Failure = @{ Message = $msg; Type = 'RecentFailure'; Body = $msg } + Properties = (& $tcProps $r 'Failed') }) | Out-Null } foreach ($gap in ($attemptGaps | Sort-Object @{Expression='AttemptUtc';Descending=$true}, ClusterName)) { @@ -574,6 +596,17 @@ function Export-AzLocalUpdateRunMonitorReport { ClassName = 'UpdateMonitor' Time = 0.0 Failure = @{ Message = $msg; Type = 'AttemptWithoutRun'; Body = $msg } + Properties = [ordered]@{ + ClusterName = [string]$gap.ClusterName + ClusterResourceId = [string]$gap.ClusterResourceId + UpdateName = $updateLabel + Status = 'AttemptWithoutRun' + CurrentStep = '' + ClusterPortalUrl = if ($gap.ClusterResourceId) { 'https://portal.azure.com/#@/resource' + [string]$gap.ClusterResourceId + '/updates' } else { '' } + UpdateRunPortalUrl = '' + Outcome = [string]$gap.Outcome + AttemptUtc = [string]$gap.AttemptUtcText + } }) | Out-Null } $null = New-AzLocalPipelineJUnitXml -TestSuitesName 'Update Run Monitor' -Suites @( diff --git a/AzLocal.UpdateManagement/Public/New-AzLocalFleetConnectivityStatusSummary.ps1 b/AzLocal.UpdateManagement/Public/New-AzLocalFleetConnectivityStatusSummary.ps1 index 8d61a059..15967b61 100644 --- a/AzLocal.UpdateManagement/Public/New-AzLocalFleetConnectivityStatusSummary.ps1 +++ b/AzLocal.UpdateManagement/Public/New-AzLocalFleetConnectivityStatusSummary.ps1 @@ -334,7 +334,7 @@ function New-AzLocalFleetConnectivityStatusSummary { [void]$sb.AppendLine("| ARBs in scope | $arbTotal | One row per ``resourceconnector/appliances`` (multi-cluster-per-RG collapsed via summarize/make_set) |") [void]$sb.AppendLine("| Clusters with an ARB | $clustersWithArb | Clusters matched to an ARB by ClusterId |") [void]$sb.AppendLine("| Clusters without an ARB | $clustersWithoutArb | Clusters that show ``_(no ARB)_`` in the per-cluster table below |") - [void]$sb.AppendLine("| Orphan ARBs | $($orphanArbs.Count) | ARBs whose RG contains no in-scope HCI cluster (listed in the Orphan ARBs section below when > 0) |") + [void]$sb.AppendLine("| Orphan ARBs | $($orphanArbs.Count) | ARBs whose RG contains no in-scope HCI cluster (listed in the 'Non-Azure Local and/or Orphan ARB appliances' section below when > 0; may include VMware/SCVMM resource bridges) |") [void]$sb.AppendLine('') # 5c. "How to interpret + act" static prose @@ -379,59 +379,121 @@ function New-AzLocalFleetConnectivityStatusSummary { [void]$sb.AppendLine('') [void]$sb.AppendLine('**``Orphan ARBs`` > 0** _(ARBs whose RG contains no in-scope HCI cluster)_') [void]$sb.AppendLine('') - [void]$sb.AppendLine('Inspect the ''Orphan ARBs (no matching cluster in scope)'' section below for the full resource IDs. Causes:') + [void]$sb.AppendLine('Inspect the ''Non-Azure Local and/or Orphan ARB appliances'' section below for the full resource IDs. Causes:') [void]$sb.AppendLine('') + [void]$sb.AppendLine('- **ARB serves a different platform** - Azure Arc resource bridge is also used by VMware vSphere and SCVMM; the appliance may be a healthy bridge for a non-Azure Local deployment, not an orphan. Confirm the platform before acting.') [void]$sb.AppendLine('- **Cluster deleted but its ARB was not cleaned up** - delete the orphan ARB.') [void]$sb.AppendLine('- **ARB in a different sub/RG from the cluster it serves** - scope-list drift; correct the input parameter list to include the matching cluster.') [void]$sb.AppendLine('- **ARB created for a cluster excluded from this run** (e.g. environment filter) - verify the filter; no action if expected.') [void]$sb.AppendLine('') - # 5d. Cluster Connectivity (with ARB Status) per-cluster table - [void]$sb.AppendLine('### Cluster Connectivity (with ARB Status)') + # 5d. Cluster Connectivity (with ARB Status) per-cluster tables. + # Clusters are split into two tables: those WITH connectivity issues (anything + # that is not Connected with a Running ARB) shown first and expanded, and those + # WITHOUT connectivity issues (Connectivity = Connected AND ARB Status = Running) + # collapsed behind an "Expand to view clusters" details block. + $clusterIntro = '_One row per cluster, left-joined to the cluster''s Azure Resource Bridge (ARB) appliance status. Each cluster has at most one ARB. ARBs without a matching cluster in scope are listed separately under ''Non-Azure Local and/or Orphan ARB appliances''._' + $tableHeader = '| Cluster | Connectivity | Cluster Status | Nodes | ARB | ARB Status | ARB Days Since LastModified | Resource Group | Location |' + $tableDivider = '|---------|---------------|-----------------|-------|-----|-------------|------------------------------|----------------|----------|' + + # Row renderer - plain scriptblock (NOT .GetNewClosure()) so it retains the + # module SessionState (access to Get-ClusterSev / Get-StatusIcon / Get-ArbSev) + # and lexical access to $arbByClusterId. + $renderClusterRow = { + param($r) + $sev = Get-ClusterSev $r.ConnectivityStatus + $icon = Get-StatusIcon $sev + $clusterCell = if ($r.ClusterId) { '[{0}](https://portal.azure.com/#@/resource{1})' -f $r.ClusterName, $r.ClusterId } else { $r.ClusterName } + + $arb = $null + if ($r.ClusterId) { $arb = $arbByClusterId[$r.ClusterId.ToLowerInvariant()] } + + if ($arb) { + $arbSev = Get-ArbSev $arb.ArbStatus + $arbIcon = Get-StatusIcon $arbSev + $arbCell = if ($arb.ArbId) { '[{0}](https://portal.azure.com/#@/resource{1})' -f $arb.ArbName, $arb.ArbId } else { $arb.ArbName } + $arbStatusCell = '{0} {1}' -f $arbIcon, $arb.ArbStatus + $arbDaysCell = $arb.DaysSinceLastModified + } + else { + $arbCell = '_(no ARB)_' + $arbStatusCell = '-' + $arbDaysCell = '-' + } + + ('| {0} | {1} {2} | {3} | {4} | {5} | {6} | {7} | {8} | {9} |' -f $clusterCell, $icon, $r.ConnectivityStatus, $r.ClusterStatus, $r.NodeCount, $arbCell, $arbStatusCell, $arbDaysCell, $r.ResourceGroup, $r.Location) + } + + # Classify each cluster as healthy (Connected + matched ARB Running) or with issues. + $clustersWithIssues = New-Object 'System.Collections.Generic.List[object]' + $clustersHealthy = New-Object 'System.Collections.Generic.List[object]' + foreach ($r in $ClusterRows) { + $arbForRow = $null + if ($r.ClusterId) { $arbForRow = $arbByClusterId[$r.ClusterId.ToLowerInvariant()] } + $isHealthy = ($r.ConnectivityStatus -eq 'Connected') -and ($null -ne $arbForRow) -and ($arbForRow.ArbStatus -eq 'Running') + if ($isHealthy) { [void]$clustersHealthy.Add($r) } else { [void]$clustersWithIssues.Add($r) } + } + + # 5d-i. Clusters WITH connectivity issues (shown first, not collapsed). + [void]$sb.AppendLine('### Cluster with Connectivity Issues') + [void]$sb.AppendLine('') + [void]$sb.AppendLine($clusterIntro) + [void]$sb.AppendLine('') + if ($ClusterRows.Count -eq 0) { + [void]$sb.AppendLine('*No clusters returned.*') + } + elseif ($clustersWithIssues.Count -eq 0) { + [void]$sb.AppendLine('*No clusters with connectivity issues - every cluster is Connected with a Running ARB.*') + } + else { + [void]$sb.AppendLine($tableHeader) + [void]$sb.AppendLine($tableDivider) + foreach ($r in ($clustersWithIssues | Select-Object -First 100)) { + [void]$sb.AppendLine((& $renderClusterRow $r)) + } + if ($clustersWithIssues.Count -gt 100) { + [void]$sb.AppendLine('') + [void]$sb.AppendLine("*Showing first 100 of $($clustersWithIssues.Count); see ``fleet-cluster-connectivity.csv`` and ``fleet-arb-status.csv`` for the full lists.*") + } + } [void]$sb.AppendLine('') - [void]$sb.AppendLine('_One row per cluster, left-joined to the cluster''s Azure Resource Bridge (ARB) appliance status. Each cluster has at most one ARB. ARBs without a matching cluster in scope are listed separately as ''Orphan ARBs''._') + + # 5d-ii. Clusters WITHOUT connectivity issues (collapsed by default). + [void]$sb.AppendLine('### Cluster without Connectivity Issues') + [void]$sb.AppendLine('') + [void]$sb.AppendLine($clusterIntro) [void]$sb.AppendLine('') if ($ClusterRows.Count -eq 0) { [void]$sb.AppendLine('*No clusters returned.*') } + elseif ($clustersHealthy.Count -eq 0) { + [void]$sb.AppendLine('*No clusters are currently Connected with a Running ARB.*') + } else { - [void]$sb.AppendLine('| Cluster | Connectivity | Cluster Status | Nodes | ARB | ARB Status | ARB Days Since LastModified | Resource Group | Location |') - [void]$sb.AppendLine('|---------|---------------|-----------------|-------|-----|-------------|------------------------------|----------------|----------|') - foreach ($r in ($ClusterRows | Select-Object -First 100)) { - $sev = Get-ClusterSev $r.ConnectivityStatus - $icon = Get-StatusIcon $sev - $clusterCell = if ($r.ClusterId) { '[{0}](https://portal.azure.com/#@/resource{1})' -f $r.ClusterName, $r.ClusterId } else { $r.ClusterName } - - $arb = $null - if ($r.ClusterId) { $arb = $arbByClusterId[$r.ClusterId.ToLowerInvariant()] } - - if ($arb) { - $arbSev = Get-ArbSev $arb.ArbStatus - $arbIcon = Get-StatusIcon $arbSev - $arbCell = if ($arb.ArbId) { '[{0}](https://portal.azure.com/#@/resource{1})' -f $arb.ArbName, $arb.ArbId } else { $arb.ArbName } - $arbStatusCell = '{0} {1}' -f $arbIcon, $arb.ArbStatus - $arbDaysCell = $arb.DaysSinceLastModified - } - else { - $arbCell = '_(no ARB)_' - $arbStatusCell = '-' - $arbDaysCell = '-' - } - - [void]$sb.AppendLine(('| {0} | {1} {2} | {3} | {4} | {5} | {6} | {7} | {8} | {9} |' -f $clusterCell, $icon, $r.ConnectivityStatus, $r.ClusterStatus, $r.NodeCount, $arbCell, $arbStatusCell, $arbDaysCell, $r.ResourceGroup, $r.Location)) + [void]$sb.AppendLine('
') + [void]$sb.AppendLine('Expand to view clusters') + [void]$sb.AppendLine('') + [void]$sb.AppendLine($tableHeader) + [void]$sb.AppendLine($tableDivider) + foreach ($r in ($clustersHealthy | Select-Object -First 100)) { + [void]$sb.AppendLine((& $renderClusterRow $r)) } - if ($ClusterRows.Count -gt 100) { + if ($clustersHealthy.Count -gt 100) { [void]$sb.AppendLine('') - [void]$sb.AppendLine("*Showing first 100 of $($ClusterRows.Count); see ``fleet-cluster-connectivity.csv`` and ``fleet-arb-status.csv`` for the full lists.*") + [void]$sb.AppendLine("*Showing first 100 of $($clustersHealthy.Count); see ``fleet-cluster-connectivity.csv`` and ``fleet-arb-status.csv`` for the full lists.*") } + [void]$sb.AppendLine('') + [void]$sb.AppendLine('
') } # 5e. Orphan ARBs table (conditional) if ($orphanArbs.Count -gt 0) { [void]$sb.AppendLine('') - [void]$sb.AppendLine('### Orphan ARBs (no matching cluster in scope)') + [void]$sb.AppendLine('### Non-Azure Local and/or Orphan ARB appliances') + [void]$sb.AppendLine('') + [void]$sb.AppendLine('These ARB appliances do not have a matching Azure Local instance (cluster) in scope.') [void]$sb.AppendLine('') - [void]$sb.AppendLine('_ARB appliances whose resource group does not contain any HCI cluster visible to this run. May indicate a stale appliance, or a cluster outside the configured scope._') + [void]$sb.AppendLine('_ARB appliances whose resource group does not contain any HCI cluster visible to this run. This does **not** by itself mean the appliance is stale or orphaned: Azure Arc resource bridge is also used by **VMware vSphere** and **System Center Virtual Machine Manager (SCVMM)** Arc-enabled deployments, so an appliance listed here may be a healthy, in-use resource bridge for a non-Azure Local platform. Other causes include a stale appliance left behind after a cluster was deleted, or a cluster that is outside the configured scope of this run. **Fully investigate each appliance - check its resource type, associated custom location, and the platform it serves - before taking any action; do not delete an ARB until you have confirmed it is genuinely orphaned.**_') [void]$sb.AppendLine('') [void]$sb.AppendLine('| ARB | Status | Resource Group | Days Since LastModified |') [void]$sb.AppendLine('|-----|--------|----------------|--------------------------|') diff --git a/AzLocal.UpdateManagement/README.md b/AzLocal.UpdateManagement/README.md index 66990113..fb5ee94d 100644 --- a/AzLocal.UpdateManagement/README.md +++ b/AzLocal.UpdateManagement/README.md @@ -2,7 +2,7 @@ > ⚠️ **Disclaimer**: This module is **NOT** a Microsoft supported service offering or product. It is provided as example code only, with no warranty or official support. Refer to the [MIT license](https://github.com/NeilBird/Azure-Local/blob/main/LICENSE) for further information. -**Latest Version:** v0.8.86 - [Published in PowerShell Gallery](https://www.powershellgallery.com/packages/AzLocal.UpdateManagement/0.8.86) +**Latest Version:** v0.8.87 - [Published in PowerShell Gallery](https://www.powershellgallery.com/packages/AzLocal.UpdateManagement/0.8.87) > 📢 **Renamed in v0.7.3**: this module was previously published as `AzStackHci.ManageUpdates`. The new module name aligns with the Azure Local product name (_Microsoft retired the *Azure Stack HCI* brand in late 2024_). The module GUID is preserved across the rename. If you have the old name installed, run: > @@ -23,7 +23,7 @@ Azure Local REST API specification (includes update management endpoints): https **This README (overview + most-recent release notes):** - [Where to Start](#where-to-start) -- [What's New in v0.8.86](#whats-new-in-v0886) +- [What's New in v0.8.87](#whats-new-in-v0887) - [Files](#files) - [Prerequisites](#prerequisites) - [RBAC Requirements](#rbac-requirements) (summary; full reference in [docs/rbac.md](docs/rbac.md)) @@ -86,23 +86,35 @@ If you are new to this module, work through these in order from a regular PowerS > Most CI/CD pipelines in [Automation-Pipeline-Examples/](Automation-Pipeline-Examples/) are direct implementations of one of these workflows. Start there if you want a copy-pasteable end-to-end pipeline. -## What's New in v0.8.86 +## What's New in v0.8.87 -**Pipeline sidebar-ordering fix.** Renames the three onboarding pipeline templates from `Setup: 0N` to `Config: 0N` so the GitHub Actions sidebar (and the Azure DevOps Pipelines list) lists the onboarding / configuration workflows *ahead of* the `Fleet: 0N` operational workflows. Both surfaces sort alphabetically by the workflow `name:` / definition name, and `C` (Config) sorts before `F` (Fleet) - the previous `Setup:` prefix sorted *after* `Fleet:`. +**Pipeline display-name rename.** The bundled pipeline display names are renamed into a three-group `Config: N` / `Monitor: N` / `Update: N` scheme (single-digit, e.g. `Monitor: 1 - Fleet Connectivity Status`, `Update: 4 - Monitor In-Flight Updates`), replacing the former `Setup: 0N` / `Fleet: 0N` prefixes so the GitHub Actions and Azure DevOps lists group onboarding, day-2 monitoring, and update-lifecycle workflows logically. Filenames and `AZLOCAL-PIPELINE-ID` values are unchanged. + +**Fleet Connectivity Status output clarification.** Renames the "Orphan ARBs" section in the `New-AzLocalFleetConnectivityStatusSummary` (Monitor: 1 - Fleet Connectivity Status) markdown output to **"Non-Azure Local and/or Orphan ARB appliances"** and adds an explicit caveat that an Arc resource bridge with no matching in-scope Azure Local cluster is *not necessarily orphaned*. Azure Arc resource bridge is also used by **VMware vSphere** and **System Center Virtual Machine Manager (SCVMM)** Arc-enabled deployments, so a listed appliance may be a healthy, in-use resource bridge for a non-Azure Local platform. + +**In-flight monitor ITSM auto-ticketing + Config: 3 monitor-cadence recommendation.** The Update: 4 in-flight monitor and the ServiceNow ITSM connector now join up so stuck, failed, and attempt-without-run clusters auto-raise deduped incidents, and the Config: 3 schedule auditor recommends how often to poll the monitor. + +### Added + +- **`Export-AzLocalUpdateRunMonitorReport` (Update: 4)** now writes per-`` `` into `update-monitor.xml` (`ClusterName`, `ClusterResourceId`, `UpdateName`, `Status`, `CurrentStep`, `ClusterPortalUrl`, `UpdateRunPortalUrl`). `Status` is one of `StepError`, `LongRunningStep`, `LongRunningOverall`, `InProgress`, `Failed`, or `AttemptWithoutRun`. This lets `New-AzLocalIncident` dedupe (one incident per affected cluster until it clears) and deep-link into the Azure portal - previously every monitor row was skipped for a missing ClusterResourceId / UpdateName. +- **Opt-in ITSM ticketing in `monitor-updates.yml`** (GitHub Actions and Azure DevOps). A new `raise_itsm_ticket` / `raiseItsmTicket` input (default off) runs `New-AzLocalIncident` against the monitor JUnit after it is published. The monitor stays report-only and always green; ITSM failures never affect its result. The sample trigger matrix `azurelocal-itsm.yml` gains `AttemptWithoutRun` (raise, severity 3), `StepError` (raise, severity 2), and opt-in `LongRunningOverall` / `LongRunningStep` entries. +- **`Export-AzLocalApplyUpdatesScheduleAudit` (Config: 3)** gains an always-on **"Recommended in-flight monitor schedule (Update: 4)"** section that derives a `monitor-updates.yml` poll cron from the apply-window weekday(s), covering the apply day plus a configurable trailing window for multi-day runs. Two new parameters: `-MonitorFiresPerHour` (1-12, default 2 = every 30 min) and `-MonitorTrailingDays` (0-14, default 3). ### Changed -- The three onboarding workflows are renamed `Config: 01 - Validate Auth and Inventory Clusters`, `Config: 02 - Manage UpdateRing Tags`, and `Config: 03 - Apply-Updates Schedule Coverage Audit` (was `Setup: 0N`). Display-name (`name:` / `displayName:`) change only - filenames, `AZLOCAL-PIPELINE-ID` values, aliases, and `-PruneDeprecated` logic are unchanged. -- Bundled pipeline `README.md` + `docs/appendix-pipelines.md` updated to use `Config: NN` naming throughout. +- The section heading `### Orphan ARBs (no matching cluster in scope)` becomes `### Non-Azure Local and/or Orphan ARB appliances`, with an intro line and an expanded caveat calling out VMware vSphere / SCVMM resource bridges and adding investigate-before-acting guidance (check the resource type, associated custom location, and the platform served; do not delete an ARB until you have confirmed it is genuinely orphaned). +- The KPI-table "Orphan ARBs" note, the "how to interpret" causes list (new leading bullet about non-Azure Local resource bridges), and the cluster-table cross-reference now point at the renamed section. +- The `### Cluster Connectivity (with ARB Status)` table is split into a `### Cluster with Connectivity Issues` table (clusters not Connected with a Running ARB) shown first and expanded, and a `### Cluster without Connectivity Issues` table (Connectivity = Connected AND ARB Status = Running) collapsed behind an `Expand to view clusters` details block. Both tables share the same columns. ### Notes +- The Monitor: 1 connectivity changes are **output text only**. The monitor / ITSM / Config: 3 changes add new JUnit properties, two optional parameters (`-MonitorFiresPerHour`, `-MonitorTrailingDays`), and an opt-in pipeline step - all **backward-compatible** with existing defaults. - **No new exports** (count unchanged at 60). -- **`GENERATED_AGAINST_MODULE_VERSION`** bumped from `0.8.85` to `0.8.86` across bundled pipeline templates. +- **`GENERATED_AGAINST_MODULE_VERSION`** bumped from `0.8.86` to `0.8.87` across bundled pipeline templates. > Previous release notes have moved into the [Release History](#release-history) appendix at the bottom of this document. -See [CHANGELOG.md](CHANGELOG.md) for full release details. See [`What's New in v0.8.85`](#whats-new-in-v0885) in the Release History for the previous release. +See [CHANGELOG.md](CHANGELOG.md) for full release details. See [`What's New in v0.8.86`](#whats-new-in-v0886) in the Release History for the previous release. ## Files @@ -581,7 +593,11 @@ This code is provided as-is for educational and reference purposes. The full What's-New history (v0.7.81 and earlier) has moved to [docs/release-history.md](docs/release-history.md). -The most recent release notes for **v0.8.86** stay above under [`What's New in v0.8.86`](#whats-new-in-v0886). +The most recent release notes for **v0.8.87** stay above under [`What's New in v0.8.87`](#whats-new-in-v0887). + +### What's New in v0.8.86 + +**Pipeline sidebar-ordering fix.** Renames the three onboarding pipeline templates from `Setup: 0N` to `Config: 0N` so the GitHub Actions sidebar (and the Azure DevOps Pipelines list) lists the onboarding / configuration workflows *ahead of* the `Fleet: 0N` operational workflows. Both surfaces sort alphabetically by the workflow `name:` / definition name, and `C` (Config) sorts before `F` (Fleet) - the previous `Setup:` prefix sorted *after* `Fleet:`. Display-name change only - filenames, `AZLOCAL-PIPELINE-ID` values, aliases, and `-PruneDeprecated` logic are unchanged. See [CHANGELOG.md](CHANGELOG.md#0886---2026-06-16) for the full v0.8.86 entry. ### What's New in v0.8.85 diff --git a/AzLocal.UpdateManagement/Tests/AzLocal.UpdateManagement.Tests.ps1 b/AzLocal.UpdateManagement/Tests/AzLocal.UpdateManagement.Tests.ps1 index 69b929e2..6f82f67b 100644 --- a/AzLocal.UpdateManagement/Tests/AzLocal.UpdateManagement.Tests.ps1 +++ b/AzLocal.UpdateManagement/Tests/AzLocal.UpdateManagement.Tests.ps1 @@ -34,8 +34,8 @@ Describe 'Module: AzLocal.UpdateManagement' { $script:ModuleInfo | Should -Not -BeNullOrEmpty } - It 'Should have version 0.8.86' { - $script:ModuleInfo.Version | Should -Be '0.8.86' + It 'Should have version 0.8.87' { + $script:ModuleInfo.Version | Should -Be '0.8.87' } It 'Module version constants are in sync between .psm1 and .psd1' { @@ -12561,8 +12561,29 @@ Describe 'Function: New-AzLocalFleetConnectivityStatusSummary' { $script:md | Should -Match '### How to interpret \+ act on a non-zero reconciliation' } - It 'Includes the Cluster Connectivity (with ARB Status) table heading' { - $script:md | Should -Match '### Cluster Connectivity \(with ARB Status\)' + It 'Includes the Cluster with Connectivity Issues table heading' { + $script:md | Should -Match '### Cluster with Connectivity Issues' + } + + It 'Includes the Cluster without Connectivity Issues table heading' { + $script:md | Should -Match '### Cluster without Connectivity Issues' + } + + It 'Collapses only the without-issues table in a details block' { + $script:md | Should -Match '### Cluster without Connectivity Issues[\s\S]*?
\s*\r?\nExpand to view clusters' + $script:md | Should -Match '
' + } + + It 'Lists a Connected cluster with a Running ARB under the without-issues table' { + # Mobile is Connected and arb-mobile is Running, so it must appear after + # the "without Connectivity Issues" heading. + $script:md | Should -Match '### Cluster without Connectivity Issues[\s\S]*\[Mobile\]\(https://portal\.azure\.com/#@/resource/sub/abc/rg/x/p/m\.azurestackhci/clusters/Mobile\)' + } + + It 'Lists a Disconnected cluster under the with-issues table' { + # NewYork is Disconnected, so it must appear under "with Connectivity Issues" + # and before the "without Connectivity Issues" heading. + $script:md | Should -Match '### Cluster with Connectivity Issues[\s\S]*\| \[NewYork\][\s\S]*### Cluster without Connectivity Issues' } It 'Renders cluster rows with portal links when ClusterId is present' { @@ -12664,7 +12685,7 @@ Describe 'Function: New-AzLocalFleetConnectivityStatusSummary' { } It 'Does NOT include the Orphan ARBs section when there are no orphans' { - $script:mdEmpty | Should -Not -Match '### Orphan ARBs' + $script:mdEmpty | Should -Not -Match '### Non-Azure Local and/or Orphan ARB appliances' } It 'Stays well under the 21K cap with empty inputs' { @@ -12698,7 +12719,7 @@ Describe 'Function: New-AzLocalFleetConnectivityStatusSummary' { } It 'Includes the Orphan ARBs section heading' { - $script:mdOrphan | Should -Match '### Orphan ARBs \(no matching cluster in scope\)' + $script:mdOrphan | Should -Match '### Non-Azure Local and/or Orphan ARB appliances' } It 'Lists the orphan ARB by name' { @@ -12849,7 +12870,8 @@ Describe 'Function: New-AzLocalFleetConnectivityStatusSummary' { It 'Treats missing CSV files as empty (no error)' { # No fleet-arb-status.csv, no fleet-physical-nics.csv etc. # Should still produce a complete markdown document. - $script:mdCsv | Should -Match '### Cluster Connectivity \(with ARB Status\)' + $script:mdCsv | Should -Match '### Cluster with Connectivity Issues' + $script:mdCsv | Should -Match '### Cluster without Connectivity Issues' $script:mdCsv | Should -Match '\*No physical NIC issues\.' } @@ -19439,3 +19461,84 @@ Describe 'v0.8.82: Export-AzLocalUpdateRunMonitorReport reconciles UpdateLastAtt } #endregion v0.8.82 Item 5: LastUpdated column + UpdateLastAttempt audit tag + +#region v0.8.87: monitor JUnit properties + ITSM wiring + Config:03 monitor cron + +Describe 'v0.8.87: Export-AzLocalUpdateRunMonitorReport emits per-testcase ITSM Properties' { + BeforeAll { + $script:src887mon = Get-Content -LiteralPath "$PSScriptRoot/../Public/Export-AzLocalUpdateRunMonitorReport.ps1" -Raw + } + It 'Defines a $tcProps scriptblock with the ITSM dedupe/deep-link keys' { + $script:src887mon | Should -Match '\$tcProps\s*=\s*\{' + $script:src887mon | Should -Match 'ClusterResourceId\s*=\s*\[string\]\$row\.ClusterResourceId' + $script:src887mon | Should -Match 'UpdateName\s*=\s*\[string\]\$row\.UpdateName' + $script:src887mon | Should -Match 'Status\s*=\s*\$statusValue' + } + It 'Attaches Properties to each in-flight testcase with the correct Status value' { + $script:src887mon | Should -Match "Properties = \(& \`$tcProps \`$r 'StepError'\)" + $script:src887mon | Should -Match "Properties = \(& \`$tcProps \`$r 'LongRunningStep'\)" + $script:src887mon | Should -Match "Properties = \(& \`$tcProps \`$r 'LongRunningOverall'\)" + $script:src887mon | Should -Match "Properties = \(& \`$tcProps \`$r 'InProgress'\)" + $script:src887mon | Should -Match "Properties = \(& \`$tcProps \`$r 'Failed'\)" + } + It 'AttemptWithoutRun gap rows emit a Status=AttemptWithoutRun property with a stable UpdateName fallback' { + $script:src887mon | Should -Match "Status\s+=\s+'AttemptWithoutRun'" + $script:src887mon | Should -Match '\(no update name\)' + } +} + +Describe 'v0.8.87: Export-AzLocalApplyUpdatesScheduleAudit recommends an Update:4 monitor cron' { + BeforeAll { + $script:src887cfg = Get-Content -LiteralPath "$PSScriptRoot/../Public/Export-AzLocalApplyUpdatesScheduleAudit.ps1" -Raw + } + It 'Exposes -MonitorFiresPerHour (1-12, default 2) and -MonitorTrailingDays (0-14, default 3)' { + $script:src887cfg | Should -Match '\[ValidateRange\(1,\s*12\)\]' + $script:src887cfg | Should -Match '\[int\]\$MonitorFiresPerHour\s*=\s*2' + $script:src887cfg | Should -Match '\[ValidateRange\(0,\s*14\)\]' + $script:src887cfg | Should -Match '\[int\]\$MonitorTrailingDays\s*=\s*3' + } + It 'Renders an always-on "Recommended in-flight monitor schedule (Update: 4)" section' { + $script:src887cfg | Should -Match 'Recommended in-flight monitor schedule \(Update: 4\)' + } + It 'Derives the minute field from the cadence (0 for hourly, */N otherwise)' { + $script:src887cfg | Should -Match "if \(\`$monitorFires -le 1\) \{ '0' \} else \{ \('\*/\{0\}' -f \`$monitorIntervalMinutes\) \}" + } + It 'Expands apply weekday(s) across the trailing coverage window' { + $script:src887cfg | Should -Match 'for \(\$k = 0; \$k -le \$MonitorTrailingDays; \$k\+\+\)' + $script:src887cfg | Should -Match '\$monitorCoverageSet\.Add\(\(\(\$applyDow \+ \$k\) % 7\)\)' + } + It 'Falls back to a 24x7 cron when no apply firings are derivable' { + $script:src887cfg | Should -Match "\`$monitorFallbackCron = \('\{0\} \* \* \* \*' -f \`$monitorMinuteField\)" + } +} + +Describe 'v0.8.87: ITSM trigger matrix covers the Update:4 monitor statuses' { + BeforeAll { + $script:itsmCfg887 = Get-Content -LiteralPath "$PSScriptRoot/../Automation-Pipeline-Examples/.itsm/azurelocal-itsm.yml" -Raw + } + It 'Raises on AttemptWithoutRun and StepError' { + $script:itsmCfg887 | Should -Match '(?m)^\s*AttemptWithoutRun:' + $script:itsmCfg887 | Should -Match '(?m)^\s*StepError:' + } + It 'Includes opt-in (raiseTicket:false) entries for the long-running monitor statuses' { + $script:itsmCfg887 | Should -Match '(?m)^\s*LongRunningOverall:' + $script:itsmCfg887 | Should -Match '(?m)^\s*LongRunningStep:' + } +} + +Describe 'v0.8.87: monitor-updates.yml gains an opt-in ITSM ticketing step' { + It 'GitHub Actions monitor pipeline wires raise_itsm_ticket + New-AzLocalIncident' { + $gh = Get-Content -LiteralPath "$PSScriptRoot/../Automation-Pipeline-Examples/github-actions/monitor-updates.yml" -Raw + $gh | Should -Match '(?m)^\s*raise_itsm_ticket:' + $gh | Should -Match 'New-AzLocalIncident' + $gh | Should -Match 'update-monitor\.xml' + } + It 'Azure DevOps monitor pipeline wires raiseItsmTicket + New-AzLocalIncident' { + $ado = Get-Content -LiteralPath "$PSScriptRoot/../Automation-Pipeline-Examples/azure-devops/monitor-updates.yml" -Raw + $ado | Should -Match '(?m)^\s*-\s*name:\s*raiseItsmTicket' + $ado | Should -Match 'New-AzLocalIncident' + $ado | Should -Match 'update-monitor\.xml' + } +} + +#endregion v0.8.87: monitor JUnit properties + ITSM wiring + Config:03 monitor cron diff --git a/AzLocal.UpdateManagement/Tests/test-run-timings.csv b/AzLocal.UpdateManagement/Tests/test-run-timings.csv index 947cb7dd..719dea81 100644 --- a/AzLocal.UpdateManagement/Tests/test-run-timings.csv +++ b/AzLocal.UpdateManagement/Tests/test-run-timings.csv @@ -37,3 +37,9 @@ TimestampUtc,ModuleVersion,Total,Passed,Failed,Skipped,WallClockSeconds,PesterDu "2026-06-16T10:25:03Z","0.8.85","1295","38","0","0","617.07","615.99","3.31","Invoke-Tests.ps1","5.1.26100.8730" "2026-06-16T11:05:53Z","0.8.85","1305","1265","1","1","148.49","145.99","15.16","Invoke-Tests.ps1","5.1.26100.8730" "2026-06-16T11:43:15Z","0.8.86","1305","1266","0","1","77.33","75.24","3.83","Invoke-Tests.ps1","5.1.26100.8730" +"2026-06-16T12:47:00Z","0.8.87","1305","1266","0","1","116.33","113.88","4.43","Invoke-Tests.ps1","5.1.26100.8730" +"2026-06-16T13:00:02Z","0.8.87","1306","1267","0","1","96.98","95.68","3.11","Invoke-Tests.ps1","5.1.26100.8730" +"2026-06-16T13:06:07Z","0.8.87","1309","1270","0","1","76.61","75.29","2.85","Invoke-Tests.ps1","5.1.26100.8730" +"2026-06-16T13:51:12Z","0.8.87","1321","1282","0","1","84","82.59","3.47","Invoke-Tests.ps1","5.1.26100.8730" +"2026-06-16T13:54:16Z","0.8.87","1321","1282","0","1","49.15","47.69","2.78","Invoke-Tests.ps1","5.1.26100.8730" +"2026-06-16T15:05:37Z","0.8.87","1321","1282","0","1","136.51","134.21","10.79","Invoke-Tests.ps1","5.1.26100.8730" diff --git a/AzLocal.UpdateManagement/docs/release-history.md b/AzLocal.UpdateManagement/docs/release-history.md index dfa3a4e5..28b20f35 100644 --- a/AzLocal.UpdateManagement/docs/release-history.md +++ b/AzLocal.UpdateManagement/docs/release-history.md @@ -4,7 +4,15 @@ > > **For older releases**, this is the canonical reference; the main README intentionally stays slim so the most recent block is easy to find. > -> **For v0.8.86 (the current release)**, see the main [README.md](../README.md#whats-new-in-v0886) `What's New in v0.8.86` section. +> **For v0.8.87 (the current release)**, see the main [README.md](../README.md#whats-new-in-v0887) `What's New in v0.8.87` section. + +--- + +### What's New in v0.8.87 + +Renames the bundled pipeline display names into a three-group `Config: N` / `Monitor: N` / `Update: N` scheme (single-digit, e.g. `Monitor: 1 - Fleet Connectivity Status`, `Update: 4 - Monitor In-Flight Updates`), replacing the former `Setup: 0N` / `Fleet: 0N` prefixes; filenames and `AZLOCAL-PIPELINE-ID` values are unchanged. Also renames the "Orphan ARBs" section in the `New-AzLocalFleetConnectivityStatusSummary` (Monitor: 1 - Fleet Connectivity Status) markdown output to "Non-Azure Local and/or Orphan ARB appliances" and adds an explicit caveat that an Arc resource bridge with no matching in-scope Azure Local cluster is NOT necessarily orphaned. Azure Arc resource bridge is also used by **VMware vSphere** and **System Center Virtual Machine Manager (SCVMM)** Arc-enabled deployments, so an appliance listed here may be a healthy, in-use resource bridge for a non-Azure Local platform. The section now carries investigate-before-acting guidance (check the resource type, associated custom location, and the platform served; do not delete an ARB until you have confirmed it is genuinely orphaned). The KPI-table note, the "how to interpret" causes list, and the cluster-table cross-reference are updated to match. The `### Cluster Connectivity (with ARB Status)` table is split into a `### Cluster with Connectivity Issues` table (clusters not Connected with a Running ARB) shown first and expanded, and a `### Cluster without Connectivity Issues` table (Connectivity = Connected AND ARB Status = Running) collapsed behind an `Expand to view clusters` details block; both tables share the same columns. Output text only - no public API, parameter, or behavioural change. Export count unchanged (still 60). `GENERATED_AGAINST_MODULE_VERSION` bumped from `0.8.86` to `0.8.87` across all bundled pipeline templates. + +Also (additive): `Export-AzLocalUpdateRunMonitorReport` (Update: 4) now writes per-`` `` (`ClusterName` / `ClusterResourceId` / `UpdateName` / `Status` / `CurrentStep` / `ClusterPortalUrl` / `UpdateRunPortalUrl`) into `update-monitor.xml`, with `Status` one of `StepError` / `LongRunningStep` / `LongRunningOverall` / `InProgress` / `Failed` / `AttemptWithoutRun`, so `New-AzLocalIncident` can dedupe and deep-link stuck / failed / attempt-without-run clusters. The bundled `monitor-updates.yml` (GitHub Actions + Azure DevOps) gains an opt-in ITSM ticketing step (`raise_itsm_ticket` / `raiseItsmTicket`, default off), and the sample ITSM matrix `azurelocal-itsm.yml` gains `AttemptWithoutRun` / `StepError` (raise) plus opt-in `LongRunningOverall` / `LongRunningStep` entries. `Export-AzLocalApplyUpdatesScheduleAudit` (Config: 3) gains an always-on "Recommended in-flight monitor schedule (Update: 4)" section plus two new parameters `-MonitorFiresPerHour` (1-12, default 2) and `-MonitorTrailingDays` (0-14, default 3). All backward-compatible with existing defaults. ---